cookielib and urllib2: thread-safe?

[Alex Hunsley]

I'm writing a test script in python for pulling web pages from a web
server using urllib2 and cookielib. Since the main thing I am testing is
what happens when concurrent requests are made to the web server, I need
to make several requests concurrently, which I'll do from different
threads in my python script. So the important question is: are cookielib
and urllib2 thread safe? Are there any precautions that apply to using
these libs in a multi-threaded context?

thanks!
alex

 

[John J. Lee]

I'm writing a test script in python for pulling web pages from a web
server using urllib2 and cookielib. Since the main thing I am testing
is what happens when concurrent requests are made to the web server, I
need to make several requests concurrently, which I'll do from
different threads in my python script. So the important question is:
are cookielib and urllib2 thread safe? Are there any precautions that
apply to using these libs in a multi-threaded context?

urllib2: For HTTP, yes, AFAIK. For other protocols, eg. FTP, perhaps
not.

cookielib: No. It's currently thread-broken, simply because I've
never had reason to do threaded stuff with it. There is thread
synchronization code in there, but I have little doubt that it's
broken, because it's untested (I should have removed the
synchronization code entirely, in fact, since nobody volunteered to
test & fix before release: unfortunately the first beta caught me by
surprise...).

I won't be making it threadsafe, so don't wait for me to do it. I'm
happy to help others do so. Nothing especially hard for somebody with
plenty of thread experience to do, AFAIK.

I had thought part of a patch had gone in which stated this
thread-unsafety very prominently in the cookielib module docs, but it
seems that never happened, or at least not in time for 2.4.0 -- eek!


John