Odd files; just left behind?

[Robin Becker]

I see a folder .python-eggs in my home directory on one of our servers with
various .so files

~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

are these just left behind from some install process?

 

[John J. Lee]

I see a folder .python-eggs in my home directory on one of our servers
with various .so files

~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

are these just left behind from some install process?

Ah, it's about the resource extraction feature of setuptools. It's
possible for zipped eggs to contain resources that have to be
extracted to the filesystem. It puts them in the directory you saw
(or somewhere else -- see the docstring in my other reply).

http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction



John

 

[Robin Becker]

Ah, it's about the resource extraction feature of setuptools. It's
possible for zipped eggs to contain resources that have to be
extracted to the filesystem. It puts them in the directory you saw
(or somewhere else -- see the docstring in my other reply).

http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction

......
Thanks John, but why not clean up?

 

[Bruno Desthuilliers]

Robin Becker a écrit :

.....
Thanks John, but why not clean up?

Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
extracted from the egg to be usable. Where they are extracted is IIRC
configurable...

 

[Robin Becker]

Robin Becker a écrit :
Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
extracted from the egg to be usable. Where they are extracted is IIRC
configurable...

You're right! These files aren't temporary? It seems strange that the python
files live with their associated lib/site-packages and the .so files are
somewhere entirely different.

Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright", "credits" or "license" for more information.
it would seem simpler to have the .so files inside the site-packages and there's
the question of why this folder has to be obfuscated (name starts with .). Even
if these files are "resources" why should they be assumed to belong to the user?
I could make the PYTHON folder world usable, but that won't help if I make all
other things in myhome private. Surely code resources belong in the same area as
the code that uses them.

Just another nonsense from the eggworld

 

[Bruno Desthuilliers]

Robin Becker a écrit :

You're right! These files aren't temporary?


It seems strange that the
python files live with their associated lib/site-packages and the .so
files are somewhere entirely different.

IIRC, where .so files are extracted is configurable using
PYTHON_EGG_CACHE (cf the first link above).

Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright", "credits" or "license" for more information./myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc
>/myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

it would seem simpler to have the .so files inside the site-packages and
there's the question of why this folder has to be obfuscated (name
starts with .). Even if these files are "resources" why should they be
assumed to belong to the user?

Notice that you did install your lib/python in /myhome, not in /usr[/local]

(snip)

Just another nonsense from the eggworld

Or just another example of not reading the FineManual(tm) ?

 

[Robin Becker]

Bruno Desthuilliers wrote:
.........

it would seem simpler to have the .so files inside the site-packages and
there's the question of why this folder has to be obfuscated (name
starts with .). Even if these files are "resources" why should they be
assumed to belong to the user?

Notice that you did install your lib/python in /myhome, not in /usr[/local]

yes, so should I then assume that the root installs will put all the .so files
in /usr/local/lib/.python-eggs?

In fact when I look at my root installed pythons they seem quite happy to put
the _mysql.so underneath site-packages. I suspect that if I just installed from
source I'd get that behaviour, but the egg stuff is different.

(snip)

Or just another example of not reading the FineManual(tm) ?

well I didn't write the installer or the associated egg setup. The egg things
are supposed to do the right thing, but perhaps they're too difficult for the
egg layers.

 

[Bruno Desthuilliers]

Robin Becker a écrit :

Bruno Desthuilliers wrote:
........

it would seem simpler to have the .so files inside the site-packages
and there's the question of why this folder has to be obfuscated
(name starts with .). Even if these files are "resources" why should
they be assumed to belong to the user?

Notice that you did install your lib/python in /myhome, not in
/usr[/local]

yes, so should I then assume that the root installs will put all the .so
files in /usr/local/lib/.python-eggs?

In fact when I look at my root installed pythons they seem quite happy
to put the _mysql.so underneath site-packages. I suspect that if I just
installed from source I'd get that behaviour, but the egg stuff is
different.

(snip)

Or just another example of not reading the FineManual(tm) ?

well I didn't write the installer or the associated egg setup. The egg
things are supposed to do the right thing, but perhaps they're too
difficult for the egg layers.

Can't tell. So far, I've had no problem with eggs, so...

 

[Robert Kern]

You're right! These files aren't temporary?

They're a cache.

It seems strange that the python
files live with their associated lib/site-packages and the .so files are
somewhere entirely different.

Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright", "credits" or "license" for more information.
it would seem simpler to have the .so files inside the site-packages and there's
the question of why this folder has to be obfuscated (name starts with .). Even
if these files are "resources" why should they be assumed to belong to the user?

Because they are unpacked at runtime by the user that imported the module.
Usually, they won't have write access to site-packages.

I could make the PYTHON folder world usable, but that won't help if I make all
other things in myhome private. Surely code resources belong in the same area as
the code that uses them.

Well, you can't import a .so from a zip file. Thus they need to be placed
somewhere else. Don't worry about making ~/.python-eggs world readable. When
other users import the package, they will have the .so's unpacked, too.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[John J. Lee]

Bruno Desthuilliers wrote:
........

it would seem simpler to have the .so files inside the
site-packages and there's the question of why this folder has to be
obfuscated (name starts with .). Even if these files are
"resources" why should they be assumed to belong to the user?

Notice that you did install your lib/python in /myhome, not in /usr[/local]

yes, so should I then assume that the root installs will put all the
.so files in /usr/local/lib/.python-eggs?

No no -- the extraction of these .so files happens at *runtime*, so it
has to write the files somewhere it has permission to write to. It's
like RL's zipapp resource extraction feature in fact, except the
resource extraction happens at runtime. See those web pages I pointed
you at before.

In fact when I look at my root installed pythons they seem quite happy
to put the _mysql.so underneath site-packages. I suspect that if I
just installed from source I'd get that behaviour, but the egg stuff
is different.

Not sure without all the details, but I suspect those are non-egg
installs.


John

 

[Robin Becker]

Robert Kern wrote:
........

They're a cache.

they're actually the files that get used by the installed extension in
this case (MySQLdb).

...........

Because they are unpacked at runtime by the user that imported the module.
Usually, they won't have write access to site-packages.

in this particular case the python being used was installed with a user
prefix eg /myhome/PYTHON so everything belongs to the user; extensions
installed by the same user can surely be installed in those folders.
Upon checking I find that the same installs which produced the
~/.python-eggs files also managed to put the eggs like
MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg inside the
various site-packages directories.

I see no reason why the binary resources shouldn't be unpacked to
site-packages at install time.

It seems reasonable that since install time is the only time an egg
install can reasonably expect to write to the install area then that's
when these resources should be unpacked. The zipapp install system
referred to by John Lee does it that way because in practice we find we
do need to install on CD's etc.

What happens if I as a naive idiot install an egg to a write once file
system? I suppose the assumption is that we don't mind such files being
created in the user's space. Spewing stuff all over some else's disk at
run time instead of doing it once at install is wrong and may not even
be possible. It also allows for the carefully crafted binary extensions
to be substituted by anyone with write permission to the cache folder.

 

[Robert Kern]

Robert Kern wrote:
.......

they're actually the files that get used by the installed extension in
this case (MySQLdb).

Yes. They are extracted from the zipfile at runtime and left there so they don't
have to be extracted again. That's why I called it a cache. I didn't mean
anything else by the term.

..........

in this particular case the python being used was installed with a user
prefix eg /myhome/PYTHON so everything belongs to the user; extensions
installed by the same user can surely be installed in those folders.
Upon checking I find that the same installs which produced the
~/.python-eggs files also managed to put the eggs like
MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg inside the
various site-packages directories.

I see no reason why the binary resources shouldn't be unpacked to
site-packages at install time.

Then do so: Use the -Z argument to easy_install such that it always unpacks the
zip file. If you don't, then the egg will be left zipped, and the resources
extracted at runtime. Zipped eggs have benefits over the unzipped eggs in that
they decrease the amount of import overhead of having many items in your sys.path.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco