distutils - distributing non-python data files

[RickMuller]

I really appreciate the ease that the distutils make distributing
Python modules. However, I have a question about using them to
distribute non-Python (i.e. text) data files that support Python
modules. Currently when I have data of this type, I parse it into
python objects and make a python module from it. In other words, given
a data file like

% cat grocery_list.txt
eggs
spam
cheese

I would make a data structure like
grocery_list = ['eggs', 'spam', 'cheese']
which can be easily imported from a python file, and which the
distutils installation programs make it easy for me to find on the
Python path.

Of course, the data I'm using is much more complex than this, but you
get the idea. I'm starting to feel like this is a Bad Thing, because if
the data file is distributed as a plain text file (e.g. data values in
columns), I'm putting a barrier to updating the data if I have to parse
a new file into Python. But I don't know whether there is a better way
to include a file like 'grocery_list.txt' in my python
distutils-distributed module so that I can get it from my other python
modules?

 

[Jorgen Grahn]

I really appreciate the ease that the distutils make distributing
Python modules. However, I have a question about using them to
distribute non-Python (i.e. text) data files that support Python
modules.

I think you forgot to explain why you cannot simply use distutil's
data_files parameter. Do you have trouble finding the file at runtime?

http://docs.python.org/dist/node12.html

/Jorgen

 

[Micah Elliott]

I really appreciate the ease that the distutils make distributing
Python modules. However, I have a question about using them to
distribute non-Python (i.e. text) data files that support Python
modules.

It's not clear from your questions whether this is user-configuration
data for your program, or just data files that your program generates.

If the latter, you might look into pickle'ing. Often the best place
to look for examples is in your Python distribution itself. My
cursory scan didn't show much this time, but the Ft module might be
worthy of note. It installs a Share directory
(.../site-packages/Ft/Share) that contains some XML files.

Currently when I have data of this type, I parse it into
python objects and make a python module from it.

Sometimes this is a reasonable approach, and you might get away with
just storing it as Python code with no conversion. Mailman (and one
of my projects) uses config files that are actually treated as Python
modules. For configuration data Python is such a readable language
that even users unfamiliar with it can often safely edit a config
file.

In other words,
given a data file like

% cat grocery_list.txt
eggs
spam
cheese

I would make a data structure like
grocery_list = ['eggs', 'spam', 'cheese']
which can be easily imported from a python file, and which the
distutils installation programs make it easy for me to find on the
Python path.

Of course, the data I'm using is much more complex than this, but
you get the idea.

Then you might consider going to an XML format, and let
SAX/DOM/whatever do the work for you. Or maybe use the Configuration

I'm starting to feel like this is a Bad Thing, because if the data
file is distributed as a plain text file (e.g. data values in
columns), I'm putting a barrier to updating the data if I have to
parse a new file into Python.

Text, binary, columnar, XML... whatever the format, if it changes you
will always be susceptible to changing your parser.

But I don't know whether there is a better way to include a file
like 'grocery_list.txt' in my python distutils-distributed module so
that I can get it from my other python modules?

If it's configuration data not stored within your distribution, you
could have some places to auto-check, e.g.: $HOME/somerc.py,
/etc. And/or you could support an environment variable.