urllib2.urlopen taking way too much time

Discussion in 'Python' started by Phonethics Mobile Media, Apr 19, 2010.

  1. handler = urllib2.urlopen(req) is taking way too much time to retrieve
    the URL. The same code using sockets in PHP doesn't delay this long.
    I had 'Authorization':'Basic ' + base64.b64encode("username:password")
    in my header though.
    [ I didnt use HTTPPasswordMgr & HTTPPasswordMgrWithDefaultRealm
    because I was unable to send headers with the example shown here :
    http://docs.python.org/howto/urllib2.html#id6 ]

    I have data = handler.readline() in a loop - this is an endless
    network script (twitter streaming) so cant really wait for the entire
    url contents to finish loading.

    I had to socket.setdefaulttimeout to 30 to make it work.
    Is it because python is getting chunks in high number of KBs ?

    Python 2.6.2

    --
    Phonethics
    Phonethics Mobile Media, Apr 19, 2010
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris
    Replies:
    0
    Views:
    1,043
    Chris
    Jul 10, 2005
  2. Filip Gruszczyński

    urllib2 urlopen takes too much time

    Filip Gruszczyński, Jun 21, 2009, in forum: Python
    Replies:
    1
    Views:
    1,113
  3. Gregory A. Beamer

    Re: SQL Query is taking too much time

    Gregory A. Beamer, Oct 15, 2009, in forum: ASP .Net
    Replies:
    0
    Views:
    422
    Gregory A. Beamer
    Oct 15, 2009
  4. Shailendra

    Multiprocessing taking too much time

    Shailendra, Jul 29, 2010, in forum: Python
    Replies:
    1
    Views:
    274
    John Nagle
    Jul 29, 2010
  5. Dysgraphic Programmer

    cProfile taking up too much memory?

    Dysgraphic Programmer, Mar 14, 2011, in forum: Python
    Replies:
    0
    Views:
    216
    Dysgraphic Programmer
    Mar 14, 2011
Loading...

Share This Page