Where to put data

[bvdp]

I'm having a disagreement with a buddy on the packaging of a program we're doing in Python. It's got a number of modules and large number of library files. The library stuff is data, not code.

I'd like to put the modules in /usr/lib/pythonX.Y/mymodules or wherever setup.py decides. And the data in /usr/share/lib/myprogram.

My buddy says, that it'll be hard to be consistant in the /usr/share/.. when we consider platforms other than linux. So, he wants:

/usr/lib/pythonX.Y/myprogram
mymodules ...
mydata ....

I've got 2 issues with this:

1. I don't know if putting data in the python tree is "legit".
2. I'd have to do a lot of rewritting. My modules currently use:

include mymodules.foobar
x=mymodules.foobar.func()

and I would need to change that to:

include myprogram.mymodules.foobar

x=myprogram.mymodules.foobar.func()


unless there is python way to drop the "myprogram" bit?

 

[Rick Johnson]

I've got 2 issues with this:

   1. I don't know if putting data in the python tree is "legit".
   2. I'd have to do a lot of rewritting. My modules currently use:

I would not put anything in the toplevel Python folder. You need to
place everything under site-packages --> "Python27\Lib\site-packages
\PackageName\blah". Of course client created files should be saved to
a more accessible place.

[...]
unless there is python way to drop the "myprogram" bit?

Considering mymodules is a valid python package, you can do:
py> from mymodules import foobar

 

[Evan Driscoll]

I would just like to make a strong plea that you make it possible to
install in places other than /usr. Bascially, 'python setup.py install
--prefix /some/alternative/place' should work.

Evan

 

[bvdp]

I would not put anything in the toplevel Python folder. You need to
place everything under site-packages --> "Python27\Lib\site-packages
\PackageName\blah". Of course client created files should be saved to
a more accessible place.

Oh. Just looking at my setup (Ubunutu 11.10) and I see that /usr/lib/python2.7 doesn't have a site-packages directory. However, /usr/local/lib/python2.7 has both dist-packages and site-packages.

So, my stuff should probably go into /usr/local/lib/python2.7/site-packages?

Interesting (?) that these are empty dirs right now?

Also, if I look at my sys.path value I see that /usr/local/lib/python2.7/dist-packages is in the path; but site-packages is not.

Considering mymodules is a valid python package, you can do:
py> from mymodules import foobar

Yes. Understand that part. And then I can just call 'foobar()'. What I was wondering is if there was a way to set something in __init__.py to shorten the calls. So, if I have:

/usr/local/lib/python2.7/dist-packages/myprogram
mymods
__init__.py
mod1.py
mod2.py
mylibs
__init__.py


Is there some magic I can put into myprogram/__init__.py which forces modules to be imported from mymods instead of myprogram/mymods?

Thanks.

 

[bvdp]

Right now my program does a search for modules in "all the normal places", which seems to work for windows, mac and linux. Once the modules are found I just insert that location into sys.path[0].

Which permits the modules to reside anywhere on the HDD. However, I have feeling that this isn't quite pythonic.

 

[bvdp]

Right now my program does a search for modules in "all the normal places", which seems to work for windows, mac and linux. Once the modules are found I just insert that location into sys.path[0].

Which permits the modules to reside anywhere on the HDD. However, I have feeling that this isn't quite pythonic.

 

[Martin P. Hellwig]

On 25/01/2012 17:26, bvdp wrote:

<cut explanation of bikeshed argument, where do I put the darn things>

Well once you think about distributing, here is the guide line I use:

- If it is meant as a library that can be 'imported' in python:

site-packages is the place to be, some linux distros are rather

creative with them so be careful.

- If it is a 'stand-alone' application that just happens to use the
python interpreter as a dependency:

/usr/local/bin for the executable script and /usr/local/lib for the

program module(s) itself, of course this is platform dependent, consult
posix first and then distribution/os specific preferences.

- If the additional (binary) data is static in nature:

/usr/local/share is the right place (or whichever preference the

distribution/os has)

- If the additional (binary) data is dynamic in nature (like databases)
and the program is run as a daemon:

/usr/local/, /var/ or /opt/

- If the additional (binary) data is dynamic and it is run per user:

$HOME/.[application name]/ (the famous dot files in your home folder).

All that is unix like of course, Windows tend to put it all in the
application folder in the Program Files folder, and user specific data
in the profiles Application Data.

Of course opinions vary so I can only say this is what I usually follow,
with the wisdom bestowed upon me by unix admins that where much more
experience then I ever will be and always had a fit when I didn't put it
in the right directory.

YMMV

 

[Michael Torrie]

Right now my program does a search for modules in "all the normal
places", which seems to work for windows, mac and linux. Once the
modules are found I just insert that location into sys.path[0].

Which permits the modules to reside anywhere on the HDD. However, I
have feeling that this isn't quite pythonic

Unless you are writing a python library that will be used by others, I
don't think that where you put your files has anything to do with being
"pythonic" or not. Just do what works for your OS.

On Linux, many packages written in many languages have to deal with the
problem of where to put their files. For example, firefox, even when
installed as a package on Fedora, puts just about everything in
/usr/lib/firefox-#.#, and then symlinks the start binary back into
/usr/bin. Other packages, such as Calibre, put things in
/usr/lib/calibre (most python modules go here), and some things like
extension scripts in /usr/share/calibre. Other packages usr
/usr/libexec/packagename/. In the case of both firefox and calibre, if
you install from tarball it makes a "firefox" or "calibre" folder in a
place of your choosing and dumps its stuff inside, sometimes with a
mini-unix directory structure (bin, share, lib etc).

On Mac of course, you can put everything in your application bundle.
That's how most standalone apps written in python work.

On Windows you could stick all your python modules in your application's
directory in Program Files.

 

[bvdp]

Unless you are writing a python library that will be used by others, I
don't think that where you put your files has anything to do with being
"pythonic" or not. Just do what works for your OS.

Yes. I agree and it's nice to have a confirmation. So far I've been putting all my program into /usr/local/share/MYPROGRAM and then simply inserting an entry into sys.path.

Then, for other systems, I check a few common locations until I find the installation.

I'm getting mangled by the debian maintainers and friends who seem to believe that python modules need to go into /usr/lib/python...

 

[bvdp]

Unless you are writing a python library that will be used by others, I
don't think that where you put your files has anything to do with being
"pythonic" or not. Just do what works for your OS.

Yes. I agree and it's nice to have a confirmation. So far I've been putting all my program into /usr/local/share/MYPROGRAM and then simply inserting an entry into sys.path.

Then, for other systems, I check a few common locations until I find the installation.

I'm getting mangled by the debian maintainers and friends who seem to believe that python modules need to go into /usr/lib/python...

 

[Michael Torrie]

Yes. I agree and it's nice to have a confirmation. So far I've been
putting all my program into /usr/local/share/MYPROGRAM and then
simply inserting an entry into sys.path.

Then, for other systems, I check a few common locations until I find
the installation.

I'm getting mangled by the debian maintainers and friends who seem to
believe that python modules need to go into /usr/lib/python...

I guess the maintainers aren't distinguishing between python apps and
their submodules and general python modules (libraries), which is pretty
silly. Even as a mere user I would not like my /usr/lib/python
directory cluttered with python code that is not useful generally but is
only for specific apps. Namespace collisions are inevitable with other
python apps (not libraries) if folks insist on doing this.

Calibre appears to be in the Ubuntu standard repositories. I just
checked and in calibre proper (not talking about dependent libraries and
things that would be useful outside of calibre), there are no python
files installed in /usr/lib/python/. Calibre modules that belong to
calibre proper are in /usr/lib/calibre. Recipes (really just python
scripts) are in /usr/share/calibre. Maybe Ubuntu is doing things
differently than Debian, but I'm hard pressed to see the logic in
forcing everything ever written in python, such as submodules, installed
to /usr/lib/python. Baffles the mind.

 

[John Nagle]

I'm having a disagreement with a buddy on the packaging of a program
we're doing in Python. It's got a number of modules and large number
of library files. The library stuff is data, not code.

How much data? Megabytes? Gigabytes?

I have some modules which contain nothing but big
constants, written by a program in Python format.

John Nagle

 

[bvdp]

I guess the maintainers aren't distinguishing between python apps and
their submodules and general python modules (libraries), which is pretty
silly. Even as a mere user I would not like my /usr/lib/python
directory cluttered with python code that is not useful generally but is
only for specific apps. Namespace collisions are inevitable with other
python apps (not libraries) if folks insist on doing this.

Well, I might be wrong in my assumptions. Never got invited to join the inner circle so I'm looking at all this from the outside.

Calibre appears to be in the Ubuntu standard repositories. I just

Yeah, I looked at that as well after your earlier post.

checked and in calibre proper (not talking about dependent libraries and
things that would be useful outside of calibre), there are no python
files installed in /usr/lib/python/. Calibre modules that belong to

But, when you dl from the calibre site the default location is /opt.

calibre proper are in /usr/lib/calibre. Recipes (really just python
scripts) are in /usr/share/calibre. Maybe Ubuntu is doing things
differently than Debian, but I'm hard pressed to see the logic in
forcing everything ever written in python, such as submodules, installed
to /usr/lib/python. Baffles the mind.

I completely agree.

Mind you, one consolation in putting things in, for example, /usr/lib/pythonX.Y are:

- you can let setup find that that magic location
- you don't need to worry about your app finding the lib (python modules).

But, I've pretty much decided that the easy way (and dare I say the correct way?) is to let my packager decide where to install the modules. My program really doesn't care (nor do I). And, then I'll end up with my program's stuff something like:

myprogram
mymodules ...
mylib ...
program-bin

And then have a link in the users path which is a link or a one line call to program-bin. With modules in a directory at the same level as program-bin I don't have to do any module searches, etc. Seems to be a simple and sort-of-elegant solution.

I've tried this with one-line callers and links and it seems to work in all cases. Any gotchas?

 

[bvdp]

I guess the maintainers aren't distinguishing between python apps and
their submodules and general python modules (libraries), which is pretty
silly. Even as a mere user I would not like my /usr/lib/python
directory cluttered with python code that is not useful generally but is
only for specific apps. Namespace collisions are inevitable with other
python apps (not libraries) if folks insist on doing this.

Well, I might be wrong in my assumptions. Never got invited to join the inner circle so I'm looking at all this from the outside.

Calibre appears to be in the Ubuntu standard repositories. I just

Yeah, I looked at that as well after your earlier post.

checked and in calibre proper (not talking about dependent libraries and
things that would be useful outside of calibre), there are no python
files installed in /usr/lib/python/. Calibre modules that belong to

But, when you dl from the calibre site the default location is /opt.

calibre proper are in /usr/lib/calibre. Recipes (really just python
scripts) are in /usr/share/calibre. Maybe Ubuntu is doing things
differently than Debian, but I'm hard pressed to see the logic in
forcing everything ever written in python, such as submodules, installed
to /usr/lib/python. Baffles the mind.

I completely agree.

Mind you, one consolation in putting things in, for example, /usr/lib/pythonX.Y are:

- you can let setup find that that magic location
- you don't need to worry about your app finding the lib (python modules).

But, I've pretty much decided that the easy way (and dare I say the correct way?) is to let my packager decide where to install the modules. My program really doesn't care (nor do I). And, then I'll end up with my program's stuff something like:

myprogram
mymodules ...
mylib ...
program-bin

And then have a link in the users path which is a link or a one line call to program-bin. With modules in a directory at the same level as program-bin I don't have to do any module searches, etc. Seems to be a simple and sort-of-elegant solution.

I've tried this with one-line callers and links and it seems to work in all cases. Any gotchas?

 

[bvdp]

How much data? Megabytes? Gigabytes?

I have some modules which contain nothing but big
constants, written by a program in Python format.

John Nagle

A couple of hundred files totaling about 2 meg. Not a lot. Gosh, I remember when this was several full floppies