urllib leaves connections/sockets waiting. BIG problem!!!

[binaryj]

hi i am using urllib2 to do some automated web thing.
basically i hit on sites and check the price what they are offering
for their product and then decide if i want to lower or increase my
pricing.so in short i have to hit hundreds of sites!!!!!

for the problem:
=============
i run 20 threads all do the same stuff. (hit and run )
after around 10-15 hits(per thread) hits the thread does nothing. it
freezes. slowely but STEADILY all the threads end up with the same
fate

i did some netstat and found out that the connecton(sockets) the
program had opened are waiting the CLOSE_WAIT state !!

netstat -t
tcp 1 0 192.168.1.2:4882 host-blabla:www
CLOSE_WAIT
tcp 1 0 192.168.1.2:4884 host-blabla:www
CLOSE_WAIT
tcp 1 0 192.168.1.2:4375 host-blabla:www
CLOSE_WAIT


OUTPUT OF PROGRAM:
THREAD: #Thread-2 getting price from webi-d 7511975 DONE !!!
THREAD: #Thread-1 getting price from webi-d 4449152 DONE !!!
THREAD: #Thread-2 getting price from webi-d 7466091 DONE !!!
THREAD: #Thread-1 getting price from webi-d 8641914 DONE !!!
THREAD: #Thread-2 getting price from webi-d 7745289 DONE !!!
THREAD: #Thread-1 getting price from webi-d 6032442 DONE !!!
THREAD: #Thread-2 getting price from webi-d 8149873 DONE !!!
no-price-on-page error
THREAD: #Thread-1 getting price from webi-d 5842934 DONE !!!
no-price-on-page error
THREAD: #Thread-2 getting price from webi-d 3385778 DONE !!!
THREAD: #Thread-1 getting price from webi-d 4610122 DONE !!!
THREAD: #Thread-2 getting price from webi-d 8641536 DONE !!!
THREAD: #Thread-1 getting price from webi-d 4219935 DONE !!!
---------and thats it, it freezes. i have waited 1hr the sockets have
not changed their states!


please help

 

[Jonathan Gardner]

hi i am using urllib2 to do some automated web thing.
basically i hit on sites and check the price what they are offering
for their product and then decide if i want to lower or increase my
pricing.so in short i have to hit hundreds of sites!!!!!

for the problem:
=============
i run 20 threads all do the same stuff. (hit and run )
after around 10-15 hits(per thread) hits the thread does nothing. it
freezes. slowely but STEADILY all the threads end up with the same
fate

i did some netstat and found out that the connecton(sockets) the
program had opened are waiting the CLOSE_WAIT state !!

netstat -t
tcp        1      0 192.168.1.2:4882        host-blabla:www
CLOSE_WAIT
tcp        1      0 192.168.1.2:4884        host-blabla:www
CLOSE_WAIT
tcp        1      0 192.168.1.2:4375        host-blabla:www
CLOSE_WAIT

OUTPUT OF PROGRAM:
THREAD: #Thread-2 getting price from webi-d  7511975 DONE !!!
THREAD: #Thread-1 getting price from webi-d  4449152 DONE !!!
THREAD: #Thread-2 getting price from webi-d  7466091 DONE !!!
THREAD: #Thread-1 getting price from webi-d  8641914 DONE !!!
THREAD: #Thread-2 getting price from webi-d  7745289 DONE !!!
THREAD: #Thread-1 getting price from webi-d  6032442 DONE !!!
THREAD: #Thread-2 getting price from webi-d  8149873 DONE !!!
no-price-on-page error
THREAD: #Thread-1 getting price from webi-d  5842934 DONE !!!
no-price-on-page error
THREAD: #Thread-2 getting price from webi-d  3385778 DONE !!!
THREAD: #Thread-1 getting price from webi-d  4610122 DONE !!!
THREAD: #Thread-2 getting price from webi-d  8641536 DONE !!!
THREAD: #Thread-1 getting price from webi-d  4219935 DONE !!!
---------and thats it, it freezes. i have waited 1hr the sockets have
not changed their states!

please help

I think we'll need more details before being able to assess what is
wrong. Can you supply some sample code that has the same bug?