Iterators from urllib2

[Joshua Ginsberg]

I'm a bit baffled by something...

In a script I wrote, I have defined a function that runs
urllib2.urlopen() on a urllib2.Request object and returns the file-like
object. The code that calls this function attempts to build a
csv.DictReader object based on that file-like object, but an error is
thrown saying the file-like object is not an iterator.

I could have sworn urllib2.urlopen returned an iterator, so I tested:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close',
'fileno', 'fp', 'geturl', 'headers', 'info', 'next', 'read',
'readline', 'readlines', 'url']

Yep. But what about in my code? I modify my code to print dir(ifs)
before creating the DictReader...

['__doc__', '__init__', '__module__', '__repr__', 'close', 'fp',
'geturl', 'headers', 'info', 'read', 'readline', 'url']
Traceback (most recent call last):
File "CSVParser.py", line 144, in ?
print parseQHost(circuits[cktname], cktname)
File "CSVParser.py", line 126, in parseQHost
r = csv.DictReader(ifs, fieldlist)
File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
python2.3/csv.py", line 100, in __init__
self.reader = reader(f, dialect, *args)
TypeError: argument 1 must be an iterator

Whoa! Where did the __iter__, readlines, and next attributes go? Ideas?

-jag

 

[Michael Hoffman]

I'm a bit baffled by something...

In a script I wrote, I have defined a function that runs
urllib2.urlopen() on a urllib2.Request object and returns the file-like
object. The code that calls this function attempts to build a
csv.DictReader object based on that file-like object, but an error is
thrown saying the file-like object is not an iterator.

I could have sworn urllib2.urlopen returned an iterator, so I tested:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close',
'fileno', 'fp', 'geturl', 'headers', 'info', 'next', 'read',
'readline', 'readlines', 'url']

Yep. But what about in my code? I modify my code to print dir(ifs)
before creating the DictReader...

['__doc__', '__init__', '__module__', '__repr__', 'close', 'fp',
'geturl', 'headers', 'info', 'read', 'readline', 'url']
Traceback (most recent call last):
File "CSVParser.py", line 144, in ?
print parseQHost(circuits[cktname], cktname)
File "CSVParser.py", line 126, in parseQHost
r = csv.DictReader(ifs, fieldlist)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
python2.3/csv.py", line 100, in __init__
self.reader = reader(f, dialect, *args)
TypeError: argument 1 must be an iterator

Whoa! Where did the __iter__, readlines, and next attributes go? Ideas?

Works for me on Python 2.4.1.

 

[Andrew Dalke]

['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close',
'fileno', 'fp', 'geturl', 'headers', 'info', 'next', 'read',
'readline', 'readlines', 'url']

Yep. But what about in my code? I modify my code to print dir(ifs)
before creating the DictReader...

['__doc__', '__init__', '__module__', '__repr__', 'close', 'fp',
'geturl', 'headers', 'info', 'read', 'readline', 'url'] ...
Whoa! Where did the __iter__, readlines, and next attributes
go? Ideas?

That difference comes from this code in urllib.py:addbase

class addbase:
"""Base class for addinfo and addclosehook."""

def __init__(self, fp):
self.fp = fp
self.read = self.fp.read
self.readline = self.fp.readline
if hasattr(self.fp, "readlines"): self.readlines = self.fp.readlines
if hasattr(self.fp, "fileno"): self.fileno = self.fp.fileno
if hasattr(self.fp, "__iter__"):
self.__iter__ = self.fp.__iter__
if hasattr(self.fp, "next"):
self.next = self.fp.next

It looks like the fp for your latter code
doesn't have the additional properties. Try
adding the following debug code to figure out
what's up

print dir(ifs)
print "fp=", ifs.fp
print "dir(fp)", dir(ifs.fp)

Odds are you'll get different results.

Andrew
(e-mail address removed)