gevent + urllib2 blocking

Discussion in 'Python' started by 月忧茗, Feb 22, 2013.

  1. Hi,

    One problem, thanks for help.


    import gevent.monkey
    gevent.monkey.match_all()

    from lxml import etree
    # I using xpath parse the html


    def _get(p):
    url = BUILD_URL(p)
    html = urllib2.urlopen(url)

    # RUN AT HERE AND BLOCKING

    # ver1
    tree = etree.parse(html, parse)

    # ver2
    # tree = etree.fromstring(html.read(), parse)

    ....


    workers = [gevent.spawn(_get, i) for i in xrange(1, 100)]
    gevent.joinall(workers)



    When using ver1, etree.parse(html, parse)
    I noticed there are only two greenlets runs at "RUN AT HERE AND BLOCKING"

    others are not running!


    But, when I using ver2, this code will running normally as expected.


    Why this ?
     
    月忧茗, Feb 22, 2013
    #1
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    17,395
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    1,307
    Valentin Tihomirov
    Oct 12, 2003
  3. Josef Cihal
    Replies:
    0
    Views:
    1,288
    Josef Cihal
    Sep 5, 2005
  4. nukleus
    Replies:
    14
    Views:
    1,297
    Chris Uppal
    Jan 22, 2007
  5. self.python
    Replies:
    4
    Views:
    1,008
    Damjan
    Jul 7, 2012
  6. subin

    concurrent.futures vs gevent

    subin, Jul 29, 2012, in forum: Python
    Replies:
    0
    Views:
    262
    subin
    Jul 29, 2012
  7. Tony Shao
    Replies:
    0
    Views:
    283
    Tony Shao
    Dec 29, 2012
  8. Roy Smith

    Weird problem with UDP and gevent

    Roy Smith, Oct 18, 2013, in forum: Python
    Replies:
    3
    Views:
    218
    Grant Edwards
    Oct 19, 2013
Loading...