Can't Access ANY url from python (errno 61)

[MattB]

Hey all,

I've been working on a program that accesses my school's password
protected website and downloads directory names. I'm using mechanize.

Recently, the program has been unable to open the website, returning
the 'errno 61 connection refused' error. I presume the school's server
was blocking me because of many automated logins.

However, it turns out that I cannot now open ANY url from within
Python on my computer using mechanize (or urllib for that matter).
And I've tried in several places -- my place, a friend's place (who
also has comcast as an ISP) and the school -- but no dice, constant
errno 61's whenever I try to open a url.

The strangest thing about this is that firefox still works flawlessly
on any site.

However, at my friend's place, I was able to open url's from HIS
computer, no problem.

Can anyone think of anything I can do to solve this problem? Why would
firefox be working fine but not Python?

Any suggestions would be greatly appreciated.

Matt

 

[Jonathan Gardner]

I've been working on a program that accesses my school's password
protected website and downloads directory names. I'm using mechanize.

Recently, the program has been unable to open the website, returning
the 'errno 61 connection refused' error. I presume the school's server
was blocking me because of many automated logins.

However, it turns out that I cannot now open ANY url from within
Python on my computer using mechanize (or urllib for that matter).
And I've tried in several places -- my place, a friend's place (who
also has comcast as an ISP) and the school -- but no dice, constant
errno 61's whenever I try to open a url.

The strangest thing about this is that firefox still works flawlessly
on any site.

However, at my friend's place, I was able to open url's from HIS
computer, no problem.

Can anyone think of anything I can do to solve this problem? Why would
firefox be working fine but not Python?

Any suggestions would be greatly appreciated.

Are you running behind a firewall? See if Firefox is configured with a
proxy. You'll have to use that to talk to any website.

 

[Chris Rebert]

Hey all,

I've been working on a program that accesses my school's password
protected website and downloads directory names. I'm using mechanize.

Recently, the program has been unable to open the website, returning
the 'errno 61 connection refused' error. I presume the school's server
was blocking me because of many automated logins.

However, it turns out that I cannot now open ANY url from within
Python on my computer using mechanize (or urllib for that matter).
And I've tried in several places -- my place, a friend's place (who
also has comcast as an ISP) and the school -- but no dice, constant
errno 61's whenever I try to open a url.

The strangest thing about this is that firefox still works flawlessly
on any site.

Can anyone think of anything I can do to solve this problem? Why would
firefox be working fine but not Python?

Any suggestions would be greatly appreciated.

Based on what you've said, it's possible the school may have blocked
mechanize's User-Agent somehow.
Try spoofing mechanize as Firefox by setting the string for the
User-Agent header to that of Firefox.
See the section "Changing the automatically-added headers
(User-Agent)" on http://wwwsearch.sourceforge.net/mechanize/doc.html
for how to do that.
To see what Firefox's User-Agent is, use http://whatsmyuseragent.com/

Cheers,
Chris

 

[Martin P. Hellwig]

Hey all,

I've been working on a program that accesses my school's password
protected website and downloads directory names. I'm using mechanize.

Recently, the program has been unable to open the website, returning
the 'errno 61 connection refused' error. I presume the school's server
was blocking me because of many automated logins.

Being a former school BOFH, I can assure you that if I was annoyed by
your 'misuse' I would have tracked you down and made you aware of it.

However, it turns out that I cannot now open ANY url from within
Python on my computer using mechanize (or urllib for that matter).
And I've tried in several places -- my place, a friend's place (who
also has comcast as an ISP) and the school -- but no dice, constant
errno 61's whenever I try to open a url.

As mentioned by Jonathan Gardener, this is most likely a proxy gateway.

The strangest thing about this is that firefox still works flawlessly
on any site.

Your system might have been centrally configure so that applications are
aware of the proxy, firefox probably has been piggybacking on those
settings (as it should). Most platforms can be made aware of a proxy by
a DHCP option send by the DHCP server (that is when you automatically
get an IP address).

Any suggestions would be greatly appreciated.

Matt

Google a bit around how you can figure out (from inside your script)
whether your used platform has a proxy configured and how to use it with
your application.

Good luck!

 

[MattB]

Being a former school BOFH, I can assure you that if I was annoyed by
your 'misuse' I would have tracked you down and made you aware of it.




As mentioned by Jonathan Gardener, this is most likely a proxy gateway.




Your system might have been centrally configure so that applications are
aware of the proxy, firefox probably has been piggybacking on those
settings (as it should). Most platforms can be made aware of a proxy by
a DHCP option send by the DHCP server (that is when you automatically
get an IP address).



Google a bit around how you can figure out (from inside your script)
whether your used platform has a proxy configured and how to use it with
your application.

Good luck!

Hey all,

I've used httpfox to identify the precise headers being sent by
firefox, and then added them to my program using br.addheaders(), as
per the proper mechanize syntax. No dice. (In fact, these headers were
in the program when I ran it successfully from my friend's computer at
his apartment). So I'm pretty sure it's not a header issue.

I'll check and see whether firefox and my system are using a proxy.

Also, based on Martin's comment, I just wanted to make you all aware
that I intend no misuse, but rather am just trying to learn, as I'm a
programming noob. I am not doing anything that I can't do myself from
firefox (ie, I have an account at the school, and am allowed to sign
on with my name and password and look up information in the student
directory). If I do it for more than one student, it just becomes
repetitive, so I thought this was a first modest goal in learning to
do some programming.)

That said, I'm happy to discontinue the attempts, but I'd like to know
how it is that my computer (unless using firefox) is completely
blocked from opening urls from within python. (And how to fix it).

Thanks for the continued help.

 

[MattB]

Hey all,

I've used httpfox to identify the precise headers being sent by
firefox, and then added them to my program using br.addheaders(), as
per the proper mechanize syntax. No dice. (In fact, these headers were
in the program when I ran it successfully from my friend's computer at
his apartment). So I'm pretty sure it's not a header issue.

I'll check and see whether firefox and my system are using a proxy.

Also, based on Martin's comment, I just wanted to make you all aware
that I intend no misuse, but rather am just trying to learn, as I'm a
programming noob. I am not doing anything that I can't do myself from
firefox (ie, I have an account at the school, and am allowed to sign
on with my name and password and look up information in the student
directory). If I do it for more than one student, it just becomes
repetitive, so I thought this was a first modest goal in learning to
do some programming.)

That said, I'm happy to discontinue the attempts, but I'd like to know
how it is that my computer (unless using firefox) is completely
blocked from opening urls from within python. (And how to fix it).

Thanks for the continued help.

Breakthrough:

I tried switching from a wireless connection to my router, and instead
used an ethernet connection -- and now everything works.

Why would this make a difference? MAC address? Is it possible for an
external server to see my MAC address and block it? Clearly wasn't an
IP address issue!

 

[Chris Rebert]

Breakthrough:

I tried switching from a wireless connection to my router, and instead
used an ethernet connection -- and now everything works.

Why would this make a difference? MAC address? Is it possible for an
external server to see my MAC address and block it? Clearly wasn't an
IP address issue!

If you're using the campus network and depending on the exact network
details, yes, they very likely can know your MAC address and thus
block it.
Since your Wi-Fi card and Ethernet card have different hardware MAC
addresses, yes, switching would change your visible MAC address, thus
circumventing any blocks based on it.

Cheers,
Chris

 

[MattB]

If you're using the campus network and depending on the exact network
details, yes, they very likely can know your MAC address and thus
block it.
Since your Wi-Fi card and Ethernet card have different hardware MAC
addresses, yes, switching would change your visible MAC address, thus
circumventing any blocks based on it.

Cheers,
Chris

Chris,

I'm using the network in my own apartment. Not the campus's.
Moreover, my mac's MAC address is different from the MAC address shown
by my router, but as I said I'm also blocked when using my friend's
wireless router at his apartment.

So it must be my mac's MAC, and not the router's MAC, that's being
blocked, right?

But ALSO -- is it my ISP that's blocking the mac's MAC (and not the
school), since I can't raise ANY url's from python when I'm on
wireless?

 

[Lie Ryan]

I'm using the network in my own apartment. Not the campus's.
Moreover, my mac's MAC address is different from the MAC address shown
by my router, but as I said I'm also blocked when using my friend's
wireless router at his apartment.

So it must be my mac's MAC, and not the router's MAC, that's being
blocked, right?

But ALSO -- is it my ISP that's blocking the mac's MAC (and not the
school), since I can't raise ANY url's from python when I'm on
wireless?

MAC or IP blocking can't be the reason, as the OP stated, he can use
Firefox just fine.

Can you access, say, http://www.google.com from urllib or mechanize?

If you can't access *any website* using urllib/mechanize but you can
with a browser and you're on a unix-based machine, you probably have the
same problem as I used to have. Check whether you used the same hostname
in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves
its hostname configurations, I use Gentoo); after editing those files
reboot (there are ways to avoid reboot, but rebooting guarantees the
conf file is reread).

Check the hostname by running this python script:

import socket
hn = socket.gethostname()
print hn
print socket.gethostbyname(hn) # should be 127.0.0.1

 

[MattB]

MAC or IP blocking can't be the reason, as the OP stated, he can use
Firefox just fine.

Can you access, say,http://www.google.comfrom urllib or mechanize?

If you can't access *any website* using urllib/mechanize but you can
with a browser and you're on a unix-based machine, you probably have the
same problem as I used to have. Check whether you used the same hostname
in /etc/conf.d/hostname and /etc/hosts (or wherever your distro saves
its hostname configurations, I use Gentoo); after editing those files
reboot (there are ways to avoid reboot, but rebooting guarantees the
conf file is reread).

Check the hostname by running this python script:

import socket
hn = socket.gethostname()
print hn
print socket.gethostbyname(hn) # should be 127.0.0.1

Lie,

Wow. Strangely, that script returned 192.168.1.106. However, in Snow
Leopard's airport settings, if I click on 'advanced' and then
'proxies', the default proxy for 'http' is 127.0.0.1:4444 (and in
these settings, the 'use proxy for http' is checked).

I just tried checking the unix files you mentioned. In etc/hosts, the
following info is displayed:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##


127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost

Also found a file called ntp-restrict.conf, containing:

# Access restrictions documented in ntp.conf(5) and
# http://support.ntp.org/bin/view/Support/AccessRestrictions
# Limit network machines to time queries only

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# localhost is unrestricted
restrict 127.0.0.1
restrict -6 ::1

includefile /private/etc/ntp.conf

Not sure if these are what I'm looking for -- I'm new to unix so I may
need a bit more hand-holding here.

I appreciate your time and effort.

Matt

 

[Lie Ryan]

Lie,

Wow. Strangely, that script returned 192.168.1.106. However, in Snow
Leopard's airport settings, if I click on 'advanced' and then
'proxies', the default proxy for 'http' is 127.0.0.1:4444 (and in
these settings, the 'use proxy for http' is checked).

I just tried checking the unix files you mentioned. In etc/hosts, the
following info is displayed:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##


127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost


Not sure if these are what I'm looking for -- I'm new to unix so I may
need a bit more hand-holding here.

Try the alternatives in this site: http://movealong.org/hostname.html

Also found a file called ntp-restrict.conf, containing:

# Access restrictions documented in ntp.conf(5) and
# http://support.ntp.org/bin/view/Support/AccessRestrictions
# Limit network machines to time queries only

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# localhost is unrestricted
restrict 127.0.0.1
restrict -6 ::1

includefile /private/etc/ntp.conf

Not sure if ntp-restrict.conf is a red herring here since its name
implies it's something for NTP (Network Time Protocol, used for time
synchronization)

 

[Martin P. Hellwig]

On 02/20/10 00:20, MattB wrote:

Also, based on Martin's comment, I just wanted to make you all aware
that I intend no misuse, but rather am just trying to learn, as I'm a
programming noob.

<cut>
It wasn't my intention to imply that, rather the opposite, that if some
BOFH would see your action as misuse (some admins are pretty trigger
happy), you would expect them to contact you directly (I would).
Even though you are on a wireless there are ways to track you down.
For example I would search my log to see if I can make an association
between a login to one of my servers and your mac address.

BTW, I always followed the philosophy that learning is much more fun if
you can brake/blow something up. Thus on my networks all students had a
lot of freedom to do whatever they think was appropriate but they where
aware that every action on my network was monitored and logged.

 

[Steven D'Aprano]

On 02/20/10 00:20, MattB wrote:

<cut>
It wasn't my intention to imply that, rather the opposite, that if some
BOFH would see your action as misuse (some admins are pretty trigger
happy), you would expect them to contact you directly (I would). Even
though you are on a wireless there are ways to track you down. For
example I would search my log to see if I can make an association
between a login to one of my servers and your mac address.

This may be what you would do, but in my experience, it is more likely
that the admin would be far too busy and/or lazy to try to track you down
unless (1) they wanted to give you a warning prior to expulsion, or (2)
they needed to know who you were prior to laying charges.

In my experience they're far more likely to just hit the problem with a
hammer by blocking you as much as is technically possible.

BTW, I always followed the philosophy that learning is much more fun if
you can brake/blow something up. Thus on my networks all students had a
lot of freedom to do whatever they think was appropriate but they where
aware that every action on my network was monitored and logged.

Unfortunately this is the exception rather than the rule, I think.

 

[MattB]

This may be what you would do, but in my experience, it is more likely
that the admin would be far too busy and/or lazy to try to track you down
unless (1) they wanted to give you a warning prior to expulsion, or (2)
they needed to know who you were prior to laying charges.

In my experience they're far more likely to just hit the problem with a
hammer by blocking you as much as is technically possible.


Unfortunately this is the exception rather than the rule, I think.

All -- problem solved. Following Lie's suggestions, and the links
from those pages, I went hunting around in my /library/preferences/
SystemConfiguration/. I opened all of the 6 or 7 files that were in
there, and all looked as if they contained info directly related to my
airport card -- networks, server names, etc. So I copied them to a
backup folder and then deleted them. Then I rebooted.

Then I opened IDLE and was able to raise URLs again. I'm not sure
what it was about those files (or maybe a specific one) -- does anyone
have any ideas?

Thanks for your help and suggestions while I tried to figure this out.

 

[Lie Ryan]

All -- problem solved. Following Lie's suggestions, and the links
from those pages, I went hunting around in my /library/preferences/
SystemConfiguration/. I opened all of the 6 or 7 files that were in
there, and all looked as if they contained info directly related to my
airport card -- networks, server names, etc. So I copied them to a
backup folder and then deleted them. Then I rebooted.

Glad to hear your problem solved.

Then I opened IDLE and was able to raise URLs again. I'm not sure
what it was about those files (or maybe a specific one) -- does anyone
have any ideas?

Do the files get auto-generated? If they do, perhaps you could try
diff-ing them with your backups and get some clue of what caused it.