urllib2 and transfer-encoding = chunked

[jdvolz]

I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.

 

[Gabriel Genellina]

I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.

Perhaps if you said what the supposed error is...

 

[jdvolz]

Haha! My mistake.

The error is that when a web server is chunking a web page only the
first chunk appears to be acquired by the urllib2.urlopen call. If you
check the headers, there is no 'Content-length' (as expected) and
instead there is 'transfer-encoding' = 'chunked'. I am getting about
the first 30Kb, and then nothing else.

I don't get a ValueError like described at the following post:

http://groups.google.com/group/comp...669?lnk=gst&q=chunked&rnum=1#8a3ea9ca84b28669

Here's the code that I think is failing, it's basically textbook Python
for accessing a url:

file = urllib2.urlopen(url)
contenttype = file.info().type
if contenttype and contenttype.find('text') > -1:
return file.read()
#
# I am checking the content type because I don't want to download
..jpegs and the like
#

I have typed similar commands into the interpreter, which also produces
only about the first 30KB of the url.

Sorry for the confusion.

 

[Nikita the Spider]

Haha! My mistake.

The error is that when a web server is chunking a web page only the
first chunk appears to be acquired by the urllib2.urlopen call. If you
check the headers, there is no 'Content-length' (as expected) and
instead there is 'transfer-encoding' = 'chunked'. I am getting about
the first 30Kb, and then nothing else.

I don't get a ValueError like described at the following post:

Hi jdvolz,
What error *do* you get? Or is it that no error is raised; you're just
not getting all of the data? If it is the latter, then the sending
server might be at fault for not properly following the chunked transfer
protocol. One way to find out would be to fire up Ethereal and see
what's coming down the wire.

I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."

Can you give us some pointers to some of these old posts? And tell us
what version of Python you're using.