Odd files; just left behind?

Discussion in 'Python' started by Robin Becker, Sep 19, 2007.

  1. Robin Becker

    Robin Becker Guest

    I see a folder .python-eggs in my home directory on one of our servers with
    various .so files

    ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

    are these just left behind from some install process?
    --
    Robin Becker
     
    Robin Becker, Sep 19, 2007
    #1
    1. Advertising

  2. Robin Becker

    John J. Lee Guest

    Robin Becker <> writes:

    > I see a folder .python-eggs in my home directory on one of our servers
    > with various .so files
    >
    > ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >
    > are these just left behind from some install process?


    Ah, it's about the resource extraction feature of setuptools. It's
    possible for zipped eggs to contain resources that have to be
    extracted to the filesystem. It puts them in the directory you saw
    (or somewhere else -- see the docstring in my other reply).

    http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction



    John
     
    John J. Lee, Sep 19, 2007
    #2
    1. Advertising

  3. Robin Becker

    Robin Becker Guest

    John J. Lee wrote:
    > Robin Becker <> writes:
    >
    >> I see a folder .python-eggs in my home directory on one of our servers
    >> with various .so files
    >>
    >> ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>
    >> are these just left behind from some install process?

    >
    > Ah, it's about the resource extraction feature of setuptools. It's
    > possible for zipped eggs to contain resources that have to be
    > extracted to the filesystem. It puts them in the directory you saw
    > (or somewhere else -- see the docstring in my other reply).
    >
    > http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    > http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction

    ......
    Thanks John, but why not clean up?

    --
    Robin Becker
     
    Robin Becker, Sep 19, 2007
    #3
  4. Robin Becker a écrit :
    > John J. Lee wrote:
    >
    >> Robin Becker <> writes:
    >>
    >>> I see a folder .python-eggs in my home directory on one of our servers
    >>> with various .so files
    >>>
    >>> ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>>
    >>>
    >>> are these just left behind from some install process?

    >>
    >>
    >> Ah, it's about the resource extraction feature of setuptools. It's
    >> possible for zipped eggs to contain resources that have to be
    >> extracted to the filesystem. It puts them in the directory you saw
    >> (or somewhere else -- see the docstring in my other reply).
    >>
    >> http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    >> http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction
    >>

    >
    > .....
    > Thanks John, but why not clean up?
    >

    Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
    extracted from the egg to be usable. Where they are extracted is IIRC
    configurable...
     
    Bruno Desthuilliers, Sep 20, 2007
    #4
  5. Robin Becker

    Robin Becker Guest

    Bruno Desthuilliers wrote:
    > Robin Becker a écrit :
    >> John J. Lee wrote:
    >>
    >>> Robin Becker <> writes:
    >>>
    >>>> I see a folder .python-eggs in my home directory on one of our servers
    >>>> with various .so files
    >>>>
    >>>> ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>>>
    >>>>
    >>>> are these just left behind from some install process?
    >>>
    >>> Ah, it's about the resource extraction feature of setuptools. It's
    >>> possible for zipped eggs to contain resources that have to be
    >>> extracted to the filesystem. It puts them in the directory you saw
    >>> (or somewhere else -- see the docstring in my other reply).
    >>>
    >>> http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    >>> http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction
    >>>

    >> .....
    >> Thanks John, but why not clean up?
    >>

    > Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
    > extracted from the egg to be usable. Where they are extracted is IIRC
    > configurable...

    You're right! These files aren't temporary? It seems strange that the python
    files live with their associated lib/site-packages and the .so files are
    somewhere entirely different.

    Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
    [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import MySQLdb
    >>> print MySQLdb.__file__

    /myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc
    >>> import _mysql
    >>> print _mysql.__file__

    /myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>>


    it would seem simpler to have the .so files inside the site-packages and there's
    the question of why this folder has to be obfuscated (name starts with .). Even
    if these files are "resources" why should they be assumed to belong to the user?
    I could make the PYTHON folder world usable, but that won't help if I make all
    other things in myhome private. Surely code resources belong in the same area as
    the code that uses them.

    Just another nonsense from the eggworld
    --
    Robin Becker
     
    Robin Becker, Sep 21, 2007
    #5
  6. Robin Becker a écrit :
    > Bruno Desthuilliers wrote:
    >> Robin Becker a écrit :
    >>> John J. Lee wrote:
    >>>
    >>>> Robin Becker <> writes:
    >>>>
    >>>>> I see a folder .python-eggs in my home directory on one of our servers
    >>>>> with various .so files
    >>>>>
    >>>>> ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>>>>
    >>>>>
    >>>>> are these just left behind from some install process?
    >>>>
    >>>> Ah, it's about the resource extraction feature of setuptools. It's
    >>>> possible for zipped eggs to contain resources that have to be
    >>>> extracted to the filesystem. It puts them in the directory you saw
    >>>> (or somewhere else -- see the docstring in my other reply).
    >>>>
    >>>> http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    >>>>
    >>>> http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction
    >>>>
    >>> .....
    >>> Thanks John, but why not clean up?
    >>>

    >> Does it needs to be "cleaned up" ? .so files (shared libs) *need* to
    >> be extracted from the egg to be usable. Where they are extracted is
    >> IIRC configurable...


    > You're right! These files aren't temporary?




    > It seems strange that the
    > python files live with their associated lib/site-packages and the .so
    > files are somewhere entirely different.


    IIRC, where .so files are extracted is configurable using
    PYTHON_EGG_CACHE (cf the first link above).

    > Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
    > [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
    > Type "help", "copyright", "credits" or "license" for more information.
    > >>> import MySQLdb
    > >>> print MySQLdb.__file__

    > /myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc
    >
    > >>> import _mysql
    > >>> print _mysql.__file__

    > /myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so



    > >>>

    >
    > it would seem simpler to have the .so files inside the site-packages and
    > there's the question of why this folder has to be obfuscated (name
    > starts with .). Even if these files are "resources" why should they be
    > assumed to belong to the user?


    Notice that you did install your lib/python in /myhome, not in /usr[/local]

    (snip)
    > Just another nonsense from the eggworld


    Or just another example of not reading the FineManual(tm) ?
     
    Bruno Desthuilliers, Sep 21, 2007
    #6
  7. Robin Becker

    Robin Becker Guest

    Bruno Desthuilliers wrote:
    .........
    >> >>>

    >>
    >> it would seem simpler to have the .so files inside the site-packages and
    >> there's the question of why this folder has to be obfuscated (name
    >> starts with .). Even if these files are "resources" why should they be
    >> assumed to belong to the user?

    >
    > Notice that you did install your lib/python in /myhome, not in /usr[/local]


    yes, so should I then assume that the root installs will put all the .so files
    in /usr/local/lib/.python-eggs?

    In fact when I look at my root installed pythons they seem quite happy to put
    the _mysql.so underneath site-packages. I suspect that if I just installed from
    source I'd get that behaviour, but the egg stuff is different.

    >
    > (snip)
    >> Just another nonsense from the eggworld

    >
    > Or just another example of not reading the FineManual(tm) ?


    well I didn't write the installer or the associated egg setup. The egg things
    are supposed to do the right thing, but perhaps they're too difficult for the
    egg layers.
    --
    Robin Becker
     
    Robin Becker, Sep 21, 2007
    #7
  8. Robin Becker a écrit :
    > Bruno Desthuilliers wrote:
    > ........
    >>> >>>
    >>>
    >>> it would seem simpler to have the .so files inside the site-packages
    >>> and there's the question of why this folder has to be obfuscated
    >>> (name starts with .). Even if these files are "resources" why should
    >>> they be assumed to belong to the user?

    >>
    >> Notice that you did install your lib/python in /myhome, not in
    >> /usr[/local]

    >
    > yes, so should I then assume that the root installs will put all the .so
    > files in /usr/local/lib/.python-eggs?
    >
    > In fact when I look at my root installed pythons they seem quite happy
    > to put the _mysql.so underneath site-packages. I suspect that if I just
    > installed from source I'd get that behaviour, but the egg stuff is
    > different.
    >
    >>
    >> (snip)
    >>> Just another nonsense from the eggworld

    >>
    >> Or just another example of not reading the FineManual(tm) ?

    >
    > well I didn't write the installer or the associated egg setup. The egg
    > things are supposed to do the right thing, but perhaps they're too
    > difficult for the egg layers.


    Can't tell. So far, I've had no problem with eggs, so...
     
    Bruno Desthuilliers, Sep 21, 2007
    #8
  9. Robin Becker

    Robert Kern Guest

    Robin Becker wrote:
    > Bruno Desthuilliers wrote:
    >> Robin Becker a écrit :
    >>> John J. Lee wrote:
    >>>
    >>>> Robin Becker <> writes:
    >>>>
    >>>>> I see a folder .python-eggs in my home directory on one of our servers
    >>>>> with various .so files
    >>>>>
    >>>>> ~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    >>>>>
    >>>>>
    >>>>> are these just left behind from some install process?
    >>>> Ah, it's about the resource extraction feature of setuptools. It's
    >>>> possible for zipped eggs to contain resources that have to be
    >>>> extracted to the filesystem. It puts them in the directory you saw
    >>>> (or somewhere else -- see the docstring in my other reply).
    >>>>
    >>>> http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction
    >>>> http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction
    >>>>
    >>> .....
    >>> Thanks John, but why not clean up?
    >>>

    >> Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
    >> extracted from the egg to be usable. Where they are extracted is IIRC
    >> configurable...

    > You're right! These files aren't temporary?


    They're a cache.

    > It seems strange that the python
    > files live with their associated lib/site-packages and the .so files are
    > somewhere entirely different.
    >
    > Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
    > [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
    > Type "help", "copyright", "credits" or "license" for more information.
    > >>> import MySQLdb
    > >>> print MySQLdb.__file__

    > /myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc
    > >>> import _mysql
    > >>> print _mysql.__file__

    > /myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
    > >>>

    >
    > it would seem simpler to have the .so files inside the site-packages and there's
    > the question of why this folder has to be obfuscated (name starts with .). Even
    > if these files are "resources" why should they be assumed to belong to the user?


    Because they are unpacked at runtime by the user that imported the module.
    Usually, they won't have write access to site-packages.

    > I could make the PYTHON folder world usable, but that won't help if I make all
    > other things in myhome private. Surely code resources belong in the same area as
    > the code that uses them.


    Well, you can't import a .so from a zip file. Thus they need to be placed
    somewhere else. Don't worry about making ~/.python-eggs world readable. When
    other users import the package, they will have the .so's unpacked, too.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 21, 2007
    #9
  10. Robin Becker

    John J. Lee Guest

    Robin Becker <> writes:

    > Bruno Desthuilliers wrote:
    > ........
    >>> >>>
    >>>
    >>> it would seem simpler to have the .so files inside the
    >>> site-packages and there's the question of why this folder has to be
    >>> obfuscated (name starts with .). Even if these files are
    >>> "resources" why should they be assumed to belong to the user?

    >>
    >> Notice that you did install your lib/python in /myhome, not in /usr[/local]

    >
    > yes, so should I then assume that the root installs will put all the
    > .so files in /usr/local/lib/.python-eggs?


    No no -- the extraction of these .so files happens at *runtime*, so it
    has to write the files somewhere it has permission to write to. It's
    like RL's zipapp resource extraction feature in fact, except the
    resource extraction happens at runtime. See those web pages I pointed
    you at before.


    > In fact when I look at my root installed pythons they seem quite happy
    > to put the _mysql.so underneath site-packages. I suspect that if I
    > just installed from source I'd get that behaviour, but the egg stuff
    > is different.


    Not sure without all the details, but I suspect those are non-egg
    installs.


    John
     
    John J. Lee, Sep 22, 2007
    #10
  11. Robin Becker

    Robin Becker Guest

    Robert Kern wrote:
    ........
    >
    > They're a cache.


    they're actually the files that get used by the installed extension in
    this case (MySQLdb).

    ...........
    >>
    >> it would seem simpler to have the .so files inside the site-packages and there's
    >> the question of why this folder has to be obfuscated (name starts with .). Even
    >> if these files are "resources" why should they be assumed to belong to the user?

    >
    > Because they are unpacked at runtime by the user that imported the module.
    > Usually, they won't have write access to site-packages.
    >


    in this particular case the python being used was installed with a user
    prefix eg /myhome/PYTHON so everything belongs to the user; extensions
    installed by the same user can surely be installed in those folders.
    Upon checking I find that the same installs which produced the
    ~/.python-eggs files also managed to put the eggs like
    MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg inside the
    various site-packages directories.

    I see no reason why the binary resources shouldn't be unpacked to
    site-packages at install time.

    It seems reasonable that since install time is the only time an egg
    install can reasonably expect to write to the install area then that's
    when these resources should be unpacked. The zipapp install system
    referred to by John Lee does it that way because in practice we find we
    do need to install on CD's etc.

    What happens if I as a naive idiot install an egg to a write once file
    system? I suppose the assumption is that we don't mind such files being
    created in the user's space. Spewing stuff all over some else's disk at
    run time instead of doing it once at install is wrong and may not even
    be possible. It also allows for the carefully crafted binary extensions
    to be substituted by anyone with write permission to the cache folder.
    --
    Robin Becker
     
    Robin Becker, Sep 22, 2007
    #11
  12. Robin Becker

    Robert Kern Guest

    Robin Becker wrote:
    > Robert Kern wrote:
    > .......
    >> They're a cache.

    >
    > they're actually the files that get used by the installed extension in
    > this case (MySQLdb).


    Yes. They are extracted from the zipfile at runtime and left there so they don't
    have to be extracted again. That's why I called it a cache. I didn't mean
    anything else by the term.

    > ..........
    >>> it would seem simpler to have the .so files inside the site-packages and there's
    >>> the question of why this folder has to be obfuscated (name starts with .). Even
    >>> if these files are "resources" why should they be assumed to belong to the user?

    >> Because they are unpacked at runtime by the user that imported the module.
    >> Usually, they won't have write access to site-packages.

    >
    > in this particular case the python being used was installed with a user
    > prefix eg /myhome/PYTHON so everything belongs to the user; extensions
    > installed by the same user can surely be installed in those folders.
    > Upon checking I find that the same installs which produced the
    > ~/.python-eggs files also managed to put the eggs like
    > MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg inside the
    > various site-packages directories.
    >
    > I see no reason why the binary resources shouldn't be unpacked to
    > site-packages at install time.


    Then do so: Use the -Z argument to easy_install such that it always unpacks the
    zip file. If you don't, then the egg will be left zipped, and the resources
    extracted at runtime. Zipped eggs have benefits over the unzipped eggs in that
    they decrease the amount of import overhead of having many items in your sys.path.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 23, 2007
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Guy Noir
    Replies:
    2
    Views:
    344
    Guy Noir
    Nov 2, 2006
  2. Michael Speer

    Odd behavior with odd code

    Michael Speer, Feb 16, 2007, in forum: C Programming
    Replies:
    33
    Views:
    1,155
    Richard Heathfield
    Feb 18, 2007
  3. =?iso-8859-1?q?Jean-Fran=E7ois_Michaud?=

    Help on table align on left of page vs left hanging indent

    =?iso-8859-1?q?Jean-Fran=E7ois_Michaud?=, Jul 10, 2007, in forum: XML
    Replies:
    2
    Views:
    1,047
    =?iso-8859-1?q?Jean-Fran=E7ois_Michaud?=
    Jul 16, 2007
  4. pc
    Replies:
    2
    Views:
    1,391
    crisgoogle
    Jun 8, 2011
  5. lawrence
    Replies:
    13
    Views:
    341
    Thomas 'PointedEars' Lahn
    Sep 4, 2004
Loading...

Share This Page