gevent + urllib2 blocking

  • Thread starter 月忧茗
  • Start date
Æ

月忧茗

Hi,

One problem, thanks for help.


import gevent.monkey
gevent.monkey.match_all()

from lxml import etree
# I using xpath parse the html


def _get(p):
url = BUILD_URL(p)
html = urllib2.urlopen(url)

# RUN AT HERE AND BLOCKING

# ver1
tree = etree.parse(html, parse)

# ver2
# tree = etree.fromstring(html.read(), parse)

....


workers = [gevent.spawn(_get, i) for i in xrange(1, 100)]
gevent.joinall(workers)



When using ver1, etree.parse(html, parse)
I noticed there are only two greenlets runs at "RUN AT HERE AND BLOCKING"

others are not running!


But, when I using ver2, this code will running normally as expected.


Why this ?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,582
Members
45,057
Latest member
KetoBeezACVGummies

Latest Threads

Top