urllib2 and transfer-encoding = chunked

Discussion in 'Python' started by jdvolz@gmail.com, Jan 18, 2007.

  1. Guest

    I am having errors which appear to be linked to a previous bug in
    urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
    Has anyone established a standard workaround? I keep finding old
    posts about it, that basically give up and say "well it's a known bug."
    Any help would be greatly appreciated.
     
    , Jan 18, 2007
    #1
    1. Advertising

  2. <> escribió en el mensaje
    news:...
    >I am having errors which appear to be linked to a previous bug in
    > urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
    > Has anyone established a standard workaround? I keep finding old
    > posts about it, that basically give up and say "well it's a known bug."
    > Any help would be greatly appreciated.


    Perhaps if you said what the supposed error is...

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jan 18, 2007
    #2
    1. Advertising

  3. Guest

    Haha! My mistake.

    The error is that when a web server is chunking a web page only the
    first chunk appears to be acquired by the urllib2.urlopen call. If you
    check the headers, there is no 'Content-length' (as expected) and
    instead there is 'transfer-encoding' = 'chunked'. I am getting about
    the first 30Kb, and then nothing else.

    I don't get a ValueError like described at the following post:

    http://groups.google.com/group/comp...669?lnk=gst&q=chunked&rnum=1#8a3ea9ca84b28669

    Here's the code that I think is failing, it's basically textbook Python
    for accessing a url:

    file = urllib2.urlopen(url)
    contenttype = file.info().type
    if contenttype and contenttype.find('text') > -1:
    return file.read()
    #
    # I am checking the content type because I don't want to download
    ..jpegs and the like
    #

    I have typed similar commands into the interpreter, which also produces
    only about the first 30KB of the url.

    Sorry for the confusion.

    Gabriel Genellina wrote:
    > <> escribió en el mensaje
    > news:...
    > >I am having errors which appear to be linked to a previous bug in
    > > urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
    > > Has anyone established a standard workaround? I keep finding old
    > > posts about it, that basically give up and say "well it's a known bug."
    > > Any help would be greatly appreciated.

    >
    > Perhaps if you said what the supposed error is...
    >
    > --
    > Gabriel Genellina
     
    , Jan 18, 2007
    #3
  4. In article <>,
    wrote:

    > Haha! My mistake.
    >
    > The error is that when a web server is chunking a web page only the
    > first chunk appears to be acquired by the urllib2.urlopen call. If you
    > check the headers, there is no 'Content-length' (as expected) and
    > instead there is 'transfer-encoding' = 'chunked'. I am getting about
    > the first 30Kb, and then nothing else.
    >
    > I don't get a ValueError like described at the following post:


    Hi jdvolz,
    What error *do* you get? Or is it that no error is raised; you're just
    not getting all of the data? If it is the latter, then the sending
    server might be at fault for not properly following the chunked transfer
    protocol. One way to find out would be to fire up Ethereal and see
    what's coming down the wire.

    > I am having errors which appear to be linked to a previous bug in
    > urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
    > Has anyone established a standard workaround? I keep finding old
    > posts about it, that basically give up and say "well it's a known bug."


    Can you give us some pointers to some of these old posts? And tell us
    what version of Python you're using.

    --
    Philip
    http://NikitaTheSpider.com/
    Whole-site HTML validation, link checking and more
     
    Nikita the Spider, Jan 20, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. CW

    transfer encoding: chunked

    CW, Feb 15, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    573
  2. Josef Cihal
    Replies:
    0
    Views:
    873
    Josef Cihal
    Sep 5, 2005
  3. Tristan
    Replies:
    0
    Views:
    642
    Tristan
    May 29, 2009
  4. Eric

    SOAP and Chunked Transfer

    Eric, Jul 23, 2004, in forum: ASP .Net Web Services
    Replies:
    3
    Views:
    178
    Dino Chiesa [Microsoft]
    Jul 28, 2004
  5. Carlitos

    Transfer-Encoding: chunked

    Carlitos, Oct 24, 2006, in forum: Javascript
    Replies:
    0
    Views:
    111
    Carlitos
    Oct 24, 2006
Loading...

Share This Page