Size limit on compiling?

[Leif K-Brooks]

I have a Python file of 1.2MB (it's a static database, not code, so it
really does have to be that big). It doesn't seem to get compiled when
imported (there's no .pyc file created), so importing it takes a few
seconds which I would really like to avoid. Is Python not compiling it
because of the large size? Is it because the whole file is on one really
long line? How can I fix this?

 

[Aahz]

I have a Python file of 1.2MB (it's a static database, not code, so it
really does have to be that big). It doesn't seem to get compiled when
imported (there's no .pyc file created), so importing it takes a few
seconds which I would really like to avoid. Is Python not compiling it
because of the large size? Is it because the whole file is on one really
long line? How can I fix this?

Use a real database, or at least pickle.

 

[Peter Hansen]

I have a Python file of 1.2MB (it's a static database, not code, so it
really does have to be that big). It doesn't seem to get compiled when
imported (there's no .pyc file created), so importing it takes a few
seconds which I would really like to avoid. Is Python not compiling it
because of the large size? Is it because the whole file is on one really
long line? How can I fix this?

If it's the main file, it won't get compiled anyway. Only files
that are imported are compiled. In any case, see Aahz' answer for the
best approach.

-Peter

 

[Bengt Richter]

If it's the main file, it won't get compiled anyway. Only files
that are imported are compiled. In any case, see Aahz' answer for the
best approach.

If your file is highly structured (or you can separately prepare a version that is),
you might consider accessing it via mmap and struct (or even without struct,
if it's e.g., a flat array of fixed-length strings). You could enhance this
by pre-computing an index and storing it at the end or beginning of the file,
maybe data for a dict of (offset,length) values, similarly packed/retrieved
from the raw binary file (byte string) format. If you open the file read-only,
an efficient OS will not even page in or allocate swap file space for anything
you don't access (though 1.2 mb these days is not a big worry ;-).
But if your data is complicated, pretty soon you'd be reinventing pickle
and various db things, and you might as well use what's available already.

Regards,
Bengt Richter