Memory errors with imaplib

[Martey]

Hi everyone,

I am trying to use imaplib to download messages, but I keep getting
memory errors when I try to download a large message (i.e. one with
attachments). Here is my test code (similar to the example in the
imaplib documentation):

import getpass, imaplib

M = imaplib.IMAP4_SSL("imap.gmail.com")
M.login("(e-mail address removed)", getpass.getpass())
M.select("[Gmail]/All Mail")
typ, data = M.search(None, 'ALL')
for num in data[0].split():
typ, data = M.fetch(num, '(RFC822)')
print 'Message %s\n%s\n' % (num, data[0][1])
M.close()
M.logout()


And here is the error message:

Python(4521) malloc: *** mmap(size=6389760) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "imaptest.py", line 8, in <module>
typ, data = M.fetch(num, '(RFC822)')
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 437, in fetch
typ, dat = self._simple_command(name, message_set, message_parts)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 1055, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 885, in _command_complete
typ, data = self._get_tagged_response(tag)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 986, in _get_tagged_response
self._get_response()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 948, in _get_response
data = self.read(size)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/imaplib.py", line 1150, in read
data = self.sslobj.read(size-read)
MemoryError

I am using Mac OS X 10.5 and Python 2.5.1 (downloaded from
python.org). I think it is possible that it is <http://bugs.python.org/
issue1092502>, but the workarounds suggested there do not seem to work
for me. Is this an actual Python bug, or am I doing something wrong?

 

[Fredrik Lundh]

I am trying to use imaplib to download messages, but I keep getting
memory errors when I try to download a large message (i.e. one with
attachments). Here is my test code (similar to the example in the
imaplib documentation):

> /.../

I am using Mac OS X 10.5 and Python 2.5.1 (downloaded from
python.org). I think it is possible that it is <http://bugs.python.org/
issue1092502>, but the workarounds suggested there do not seem to work
for me. Is this an actual Python bug, or am I doing something wrong?

looks like a known bug in imaplib:

http://bugs.python.org/issue1389051

"In a worst case scenario, you'll need some 13 gigabytes of
virtual memory to read a 15 megabyte message..."

</F>

 

[Grant Edwards]

looks like a known bug in imaplib:

http://bugs.python.org/issue1389051

"In a worst case scenario, you'll need some 13 gigabytes of
virtual memory to read a 15 megabyte message..."

The problem and the one-line soulution have been known for over
two years and it's still an open bug?

 

[Christian Heimes]

The problem and the one-line soulution have been known for over
two years and it's still an open bug?

Nobody was interested to provide a patch. I raised the level of the bug.
It's going to be fixed soonish.

Christian

 

[Grant Edwards]

Nobody was interested to provide a patch. I raised the level of the bug.
It's going to be fixed soonish.

If the solution shown in the bug report is correct, I'd be more
than happy to generate a patch.

 

[Christian Heimes]

If the solution shown in the bug report is correct, I'd be more
than happy to generate a patch.

The solution seems fine but IMO it should be fixed in the ssl socket and
not in imaplib. I like to get it *fixed* and not *worked around*.

Christian

 

[Grant Edwards]

The solution seems fine but IMO it should be fixed in the ssl
socket and not in imaplib. I like to get it *fixed* and not
*worked around*.

I've always been taught that you can't assume that a read on a
socket (ssl or otherwise) will return the number of bytes you
request. You've always got to code for the case where read()
returns smaller chunks. Maybe I've been taught wrongly, but
under Unix at least I've found that it's always a good practice
to assume that network reads will often return smaller chunks
than requested.