How to read file during module import?

[Jeremy]

I have a module that, when loaded, reads and parses a supporting
file. The supporting file contains all the data for the module and
the function that reads/parses the file sets up the data structure for
the module.

How can I locate the file during the import statement. The supporting
file is located in the same directory as the module, but when I import
I get a No such file or directory error. I could hard code the path
to the filename, but that would make it only work on my machine.

A related question: Can I parse the data once and keep it somewhere
instead of reading the supporting file every time? I tried pickling
but that wouldn't work because I have custom classes. (Either that or
I just don't know how to pickle—this is a highly probable event.)

Thanks,
Jeremy

 

[Gabriel Genellina]

How can I locate the file during the import statement. The supporting
file is located in the same directory as the module, but when I import
I get a No such file or directory error. I could hard code the path
to the filename, but that would make it only work on my machine.

The directory containing the current module is:

module_dir = os.path.dirname(os.path.abspath(__file__))

so you could open your supporting file using:

fn = os.path.join(module_dir, "supporting_file_name.ext")
open(fn) ...

A related question: Can I parse the data once and keep it somewhere
instead of reading the supporting file every time? I tried pickling
but that wouldn't work because I have custom classes. (Either that or
I just don't know how to pickle—this is a highly probable event.)

What kind of "custom classes"?
An open file, or a socket, are examples of non pickleable objects; most
other basic built-in objects are pickleable. Instances of user-defined
classes are pickleable if they contain pickleable attributes. Micro recipe:

# pickle some_object
with open(filename, "wb") as f:
pickle.dump(some_object, f, -1)

# unpickle it
with open(filename, "rb") as f:
some_object = pickle.load(f)

Try again and report any problem you encounter...

 

[Jeremy]

The directory containing the current module is:

module_dir = os.path.dirname(os.path.abspath(__file__))

I didn't know about __file__ this works! Thanks.

so you could open your supporting file using:

fn = os.path.join(module_dir, "supporting_file_name.ext")
open(fn) ...


What kind of "custom classes"?

My custom classes are not very fancy. They basically are dictionaries
and lists organizing the data in the supporting file. I was actually
surprised they didn't pickle because the classes were so simple.

An open file, or a socket, are examples of non pickleable objects; most  
other basic built-in objects are pickleable. Instances of user-defined  
classes are pickleable if they contain pickleable attributes. Micro recipe:

# pickle some_object
with open(filename, "wb") as f:
   pickle.dump(some_object, f, -1)

When I did this I got the following error:

PicklingError: Can't pickle <class '__main__.element'>: it's not found
as __main__.element

Am I just being dumb?

Thanks,
Jeremy

 

[Dave Angel]

I have a module that, when loaded, reads and parses a supporting
file. The supporting file contains all the data for the module and
the function that reads/parses the file sets up the data structure for
the module.

How can I locate the file during the import statement. The supporting
file is located in the same directory as the module, but when I import
I get a No such file or directory error. I could hard code the path
to the filename, but that would make it only work on my machine.

A related question: Can I parse the data once and keep it somewhere
instead of reading the supporting file every time? I tried pickling
but that wouldn't work because I have custom classes. (Either that or
I just don't know how to pickle—this is a highly probable event.)

Thanks,
Jeremy

By supporting file, presumably you mean a text or binary file, not a
python module. That's why import won't work.

Now, if the data supports it, you could make that supporting file simply
be an importable python program. I've seen code generators to create
python code from a binary file such as a jpeg, so that you can just
embed such code into your program.

But I'll guess that this isn't reasonable for your data, that the format
is something you want to separately manipulate.

I think the secret you're looking for is the __file__ attribute of a
module. That contains the full path to the module source file (.py or
..pyc, usually). So you use dirname on that string, to get the directory
name, and os.path.join to combine with your own filename. Now you have a
string to pass to open().

DaveA

 

[HigStar]

I have had trouble with the __file__ attribute in the past, when using
py2exe (i.e. on the windows platform) and using the bundle feature
(which zips all files).
Using os.path.realpath( __file__ ) resolves to something like .../
library.zip/packageName/fileName
Then when trying to perform an open on a file, say .../library.zip/
packageName/supportingPackageName, the file can not be found.

 

[Gabriel Genellina]

I have had trouble with the __file__ attribute in the past, when using
py2exe (i.e. on the windows platform) and using the bundle feature
(which zips all files).
Using os.path.realpath( __file__ ) resolves to something like .../
library.zip/packageName/fileName
Then when trying to perform an open on a file, say .../library.zip/
packageName/supportingPackageName, the file can not be found.

Use pkgutil.get_data then:
http://docs.python.org/library/pkgutil.html#pkgutil.get_data

 

[Gabriel Genellina]

My custom classes are not very fancy. They basically are dictionaries
and lists organizing the data in the supporting file. I was actually
surprised they didn't pickle because the classes were so simple.


When I did this I got the following error:

PicklingError: Can't pickle <class '__main__.element'>: it's not found
as __main__.element

Am I just being dumb?

No, but maybe you're redefining 'element':

py> from pickle import dumps
py> class Foo(object):
.... pass
....
py> x = Foo()
py> class Foo(object):
.... pass
....
py> dumps(x)
Traceback (most recent call last):
....
pickle.PicklingError: Can't pickle <class '__main__.Foo'>: it's not the
same obj
ect as __main__.Foo

Using reload() may lead to this error too. Basically, the class says: "I
am 'DonQuijote', from module 'LaMancha'". But nobody at LaMancha knows
DonQuijote, or they say DonQuijote is a different person.

 

[HigStar]

Use pkgutil.get_data then:http://docs.python.org/library/pkgutil.html#pkgutil.get_data

Thanks Gabriel.
I should have been more specific with my example, the problem I had
was when trying to reference a non python file, so something like .../
library.zip/packageName/supportingDll.dll or .../library.zip/
packageName/supportingConfigFile.ini
For the DLL case, adding the directory that __file__ lives in to the
os.environ["PATH"] doesn't work.

Cheers

Adrian

 

[Gabriel Genellina]

Use pkgutil.get_data
then:http://docs.python.org/library/pkgutil.html#pkgutil.get_data

I should have been more specific with my example, the problem I had
was when trying to reference a non python file, so something like .../
library.zip/packageName/supportingDll.dll or .../library.zip/
packageName/supportingConfigFile.ini
For the DLL case, adding the directory that __file__ lives in to the
os.environ["PATH"] doesn't work.

pkgutil.get_data let you extract such data from the .zip file, but you
can't use it as a file name. For a config file it may be ok, but I don't
think you can load a DLL that way. At least for us mere mortals, Windows
requires the DLL to actually reside on a disk file by its own. Perhaps
some black magic might help...