handling https sites

[padfoot]

Sir,
Is there any module in python to open https sites through a
proxy.I am connectyed to the net via a proxy server and i am unable to
crawl https sites.

 

[Shawn Milochik]

Sir,
Is there any module in python to open https sites through a
proxy.I am connectyed to the net via a proxy server and i am unable to
crawl https sites.

Check out the "Scrape the Web" series from the 2009 PyCon:

http://advocacy.python.org/podcasts/

 

[cgoldberg]

Is there any module in python to open https

sites through a proxy.

yes, you can use "urllib2".

from the urllib2 docs:
"The default is to read the list of proxies from the environment
variables"

So if you have a proxy setup, it should detect it from your
environment vars. If you want to specify something different, you
want to build an opener with a ProxyHandler:

http://docs.python.org/library/urllib2.html#urllib2.ProxyHandler

-Corey