Finding messages in huge mboxes

  • Thread starter Bastiaan Welmers
  • Start date
B

Bastiaan Welmers

Hi,

I wondered if anyone has ever met this same mbox issue.

I'm having the following problem:

I need find messages in huge mbox files (50MB or more).
The following way is (of course?) not very usable:

fp = open("mbox", "r")
archive = mailbox.UnixMailbox(fp)
i=0
while i < message_number_needed:
i+=1
archive.next()

needed_message = archive.next()

Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

i=0
j=0
while 1:
i+=1
fp.seek(-i, SEEK_TO_END=2)
line = fp.readline()
if not line:
break
if line[:5] == 'From ':
j+=1
if j == total_messages - message_number_needed:
archive.seekp = fp.tell()
message = archive.next()
# message found

But also seems to be slow and CPU consuming.

Anyone who has a better idea?

Regards,

Bastiaan Welmers
 
D

Diez B. Roggisch

Anyone who has a better idea?

AFAIK MUAs usually use a mbox.index-file for faster access. The index is
computed once, and updated whenever a new message is added. You could
create this index quite easily yourself by looping over the mbox and
pickling a list of tell'ed positions. If you also store the creation-date
of the index and the filesize of the mbox-file, you should be able to
create a function that will update the index whenever the underlying mbox
has changed. Another approach would be to perform index-creation on regular
bases using cron.

Regards,

Diez
 
D

Donn Cave

Bastiaan Welmers said:
I need find messages in huge mbox files (50MB or more). ....
Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

readline() is not your friend here. I suggest that
you read large blocks of data, like 8192 bytes for
example, and search them iteratively. Like,
next = block.find('\nFrom ', prev + 1)

This will give you the location of each message in
the current block, so you can split the block up
into a list of messages. (There will be an extra
chunk of data at the beginning of each block, before
the first "From " - recycle that onto the end of the
next block.)

Since file object buffering is at best useless in this
application, I would use posix.open, posix.lseek and
posix.read. Taking this approach, I find that reading
the last 10 messages in a 100 Mb folder takes 0.05 sec.

Donn Cave, (e-mail address removed)
 
D

David M. Cooke

At some point said:
readline() is not your friend here. I suggest that
you read large blocks of data, like 8192 bytes for
example, and search them iteratively. Like,
next = block.find('\nFrom ', prev + 1)

Unless, of course, you read '\nFr', then 'om ' in the next block...

I can't think of a simple way around this (except for reading by
lines). Concating the last two together means having to keep track of
what you've seen in the last block. Maybe picking off the last line
from the last block (using line.rfind('\n')), and concatenating that
to the beginning of the next.
 
D

Donn Cave

Quoth (e-mail address removed) (David M. Cooke):
|> In article <[email protected]>,
|> ...
|>> I need find messages in huge mbox files (50MB or more).
|> ...
|>> Especially because I often need messages at the end
|>> of the MBOX file.
|>> So I tried the following (scanning messages backwards
|>> on found "From " lines with readline())
|>
|> readline() is not your friend here. I suggest that
|> you read large blocks of data, like 8192 bytes for
|> example, and search them iteratively. Like,
|> next = block.find('\nFrom ', prev + 1)
|
| Unless, of course, you read '\nFr', then 'om ' in the next block...
|
| I can't think of a simple way around this (except for reading by
| lines). Concating the last two together means having to keep track of
| what you've seen in the last block. Maybe picking off the last line
| from the last block (using line.rfind('\n')), and concatenating that
| to the beginning of the next.

I'm reading from the end backwards, so the fragment is block[:start].
Append that to the block before it, and each block always will end at
a message boundary. If you start in the middle, you have to deal with
an extra boundary problem. If reading forward from the beginning, it
would be about as simple.

If I have overlooked some obvious problem with this, it wouldn't be
the first time, but I think it's as simple as it could be. The only
inelegance to it is that you have to scan the fragment at least twice
(one extra time for each time it's added to a new block.)

Donn Cave, (e-mail address removed)
 
M

Miki Tebeka

Hell Bastiaan,
I need find messages in huge mbox files (50MB or more).
...
Anyone who has a better idea?
I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.

--- last.py ---
#!/usr/bin/env python
from os import popen
from sys import argv

# Find last "From:" line
last = popen("grep -n 'From:' %s | tail -1" % argv[1]).read()
last = int(last.split(":")[0])
# Find total number of lines
size = popen("wc -l %s" % argv[1]).read()
size = int(size.split()[0].strip())
# Print the message
print popen("tail -%d %s" % (size - last, argv[1])).read()
--- last.py ---
Tool less than 1sec on my computer on a 11MB mailbox.

HTH.
Miki
 
C

Cameron Laird

Hell Bastiaan,
I need find messages in huge mbox files (50MB or more).
...
Anyone who has a better idea?
I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.

--- last.py ---
#!/usr/bin/env python
from os import popen
from sys import argv

# Find last "From:" line
last = popen("grep -n 'From:' %s | tail -1" % argv[1]).read()
last = int(last.split(":")[0])
# Find total number of lines
size = popen("wc -l %s" % argv[1]).read()
size = int(size.split()[0].strip())
# Print the message
print popen("tail -%d %s" % (size - last, argv[1])).read()
--- last.py ---
Tool less than 1sec on my computer on a 11MB mailbox.
.
.
.
Absolutely.

Miki, I'd find this illustration even more compelling if it exploited
commands.getoutput(.)
in place of your triplicated
popen(.).read()
 
E

Erno Kuusela

Bastiaan Welmers said:
Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

i=0
j=0
while 1:
i+=1
fp.seek(-i, SEEK_TO_END=2)
line = fp.readline()
if not line:
break
if line[:5] == 'From ':
j+=1
if j == total_messages - message_number_needed:
archive.seekp = fp.tell()
message = archive.next()
# message found

But also seems to be slow and CPU consuming.

something like this might work. the loop below scanned a 115MB mailbox
in about 1 second on a 1.2ghz k7. extracts the next-to-last message,
but you get the idea. if you don't want to read the file into cache,
you could adapt it to start with a smaller mmapped chunk from the end
of the file and enlarge it until you find what you want.


import os, re, mmap, sys
from cStringIO import StringIO
import email

fd = os.open(sys.argv[1], os.O_RDONLY)
size = os.fstat(fd).st_size
print size
buf = mmap.mmap(fd, size, access=mmap.ACCESS_READ)
message_offsets = []
for m in re.finditer(r'(?s)\n\nFrom', buf):
message_offsets.append(m.start())

msgfp = StringIO(buf[message_offsets[-2] + 2:message_offsets[-1] + 2])
msg = email.message_from_file(msgfp)
print msg['to']

-- erno
 
B

Bastiaan Welmers

Miki said:
Hell Bastiaan,

I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.
Sounds as a very good idea. Tanks.

/Bastiaan
 
B

Bastiaan Welmers

Miklós said:
What about putting it into a database like MySQL? <pyWink>

Too much work to archieve this. It's just a Mailman archieve mbox
which has to be opened. So then I have to rewrite
pipermail archiever.

/Bastiaan
 
B

Bastiaan Welmers

Diez said:
AFAIK MUAs usually use a mbox.index-file for faster access. The index is
computed once, and updated whenever a new message is added. You could
create this index quite easily yourself by looping over the mbox and
pickling a list of tell'ed positions. If you also store the creation-date
of the index and the filesize of the mbox-file, you should be able to
create a function that will update the index whenever the underlying mbox
has changed. Another approach would be to perform index-creation on
regular bases using cron.
Also good idea. It's a mailman archieve so then I have
to hack mailman for creating an index file besides the
mbox file.

/Bastiaan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top