egg and modpython

Discussion in 'Python' started by Wensheng, Sep 8, 2006.

  1. Wensheng

    Wensheng Guest

    I installed pysqlite2 using easy_install.
    and got this when using it from modpython:
    --------------------------------------------------------------------------------------------------
    Mod_python error: "PythonHandler etc.modpython"

    Traceback (most recent call last):

    File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
    299, in HandlerDispatch
    result = object(req)

    File "/usr/lib/python2.4/site-packages/spark/proj/etc/modpython.py",
    line 21, in handler
    return ReqModPython(req).run()

    File "/usr/lib/python2.4/site-packages/spark/ReqBase.py", line 141,
    in run
    con_mod = __import__(conf,'','',[''])

    File "/var/www/hello/controllers/wiki_controller.py", line 6, in ?
    Storm.conn(driver='sqlite',db="/var/www/hello/etc/sparkdemo.db")

    File "/usr/lib/python2.4/site-packages/spark/storm.py", line 64, in
    conn
    cls.__CONN = __import__('db'+driver).DbSpark(**kwds)

    File "/usr/lib/python2.4/site-packages/spark/dbsqlite.py", line 7, in
    ?
    try: from pysqlite2 import dbapi2 as sqlite

    File "build/bdist.linux-i686/egg/pysqlite2/dbapi2.py", line 27, in ?

    File "build/bdist.linux-i686/egg/pysqlite2/_sqlite.py", line 7, in ?

    File "build/bdist.linux-i686/egg/pysqlite2/_sqlite.py", line 4, in
    __bootstrap__

    File
    "/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
    line 799, in resource_filename
    return get_provider(package_or_requirement).get_resource_filename(

    File
    "/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
    line 1229, in get_resource_filename
    return self._extract_resource(manager, zip_path)

    File
    "/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
    line 1249, in _extract_resource
    real_path = manager.get_cache_path(

    File
    "/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
    line 880, in get_cache_path
    self.extraction_error()

    File
    "/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
    line 846, in extraction_error
    raise err

    ExtractionError: Can't extract file(s) to egg cache

    The following error occurred while trying to extract file(s) to the
    Python egg
    cache:

    [Errno 13] Permission denied: '/var/www/.python-eggs'

    The Python egg cache directory is currently set to:

    /var/www/.python-eggs

    Perhaps your account does not have write access to this directory? You
    can
    change the cache directory by setting the PYTHON_EGG_CACHE environment
    variable to point to an accessible directory.
    ----------------------------------------------------------------------------

    Can peak developers fix this please?


    I know I won't have this problem with python 2.5 because it has
    sqlite3, but this affect other eggs too!
    Here's similiar occurance, I can't reply it since it's too old.
    http://groups.google.com/group/comp...read/thread/d35e5264a6b491d5/cee519cb5557ad0e

    in the mean time, I just have to use old "download, unzip/untar &&
    python setup.py install" way.

    Thanks
    Wensheng, Sep 8, 2006
    #1
    1. Advertising

  2. Wensheng a écrit :
    > I installed pysqlite2 using easy_install.
    > and got this when using it from modpython:
    > --------------------------------------------------------------------------------------------------
    > Mod_python error: "PythonHandler etc.modpython"
    >
    > Traceback (most recent call last):
    >

    (snip)
    >
    > ExtractionError: Can't extract file(s) to egg cache
    >
    > The following error occurred while trying to extract file(s) to the
    > Python egg
    > cache:
    >
    > [Errno 13] Permission denied: '/var/www/.python-eggs'
    >
    > The Python egg cache directory is currently set to:
    >
    > /var/www/.python-eggs
    >
    > Perhaps your account does not have write access to this directory? You
    > can
    > change the cache directory by setting the PYTHON_EGG_CACHE environment
    > variable to point to an accessible directory.
    > ----------------------------------------------------------------------------
    >
    > Can peak developers fix this please?


    Why should they "fix" something that's
    1/ documented
    -> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues
    2/ not a an egg specific issue
    3/ really just a configuration/permission problem

    (snip)

    > in the mean time, I just have to use old "download, unzip/untar &&
    > python setup.py install" way.


    No - you have to read the manual and fix you configuration. FWIW, the
    above traceback gives far enough clues (it's even written in all letters).

    Or you may read easy_install's doc to avoid installing zipped eggs:
    http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

    HTH
    Bruno Desthuilliers, Sep 10, 2006
    #2
    1. Advertising

  3. Wensheng

    Paul Boddie Guest

    Bruno Desthuilliers wrote:
    > Wensheng a écrit :
    > > I installed pysqlite2 using easy_install.
    > > and got this when using it from modpython:
    > > --------------------------------------------------------------------------------------------------
    > > Mod_python error: "PythonHandler etc.modpython"


    [...]

    I applaud you for studying the traceback in more depth than I can find
    the motivation for, Bruno. ;-) However, this looks like a program using
    some package installed by setuptools/easy_install needs to unpack that
    package when running.

    > > ExtractionError: Can't extract file(s) to egg cache

    >
    > > The following error occurred while trying to extract file(s) to the
    > > Python egg cache:

    >
    > > [Errno 13] Permission denied: '/var/www/.python-eggs'


    And this wants to happen apparently in the home directory of the Web
    server user, whose rights may be somewhat limited. Or perhaps one has
    to initialise setuptools for any particular user by running it first as
    that user: something that isn't a particularly common thing to do with
    non-interactive users on UNIX-like systems.

    [...]

    > > Can peak developers fix this please?

    >
    > Why should they "fix" something that's
    > 1/ documented
    > -> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues


    "The Internal Structure of Python Eggs / STOP! This is not the first
    document you should read!" A case of "Beware of the Leopard" perhaps?

    > 2/ not a an egg specific issue


    In my experience, most other package managers actually unpack their
    resources when invoked, rather than leaving the programs to attempt
    such things when running. Debian packages even compile the bytecode
    files to ensure that nothing tries (and fails) to write them to
    privileged locations afterwards.

    > 3/ really just a configuration/permission problem


    Which should have been solved when the package manager was invoked,
    especially since this was most likely done as root.

    [...]

    > > in the mean time, I just have to use old "download, unzip/untar &&
    > > python setup.py install" way.

    >
    > No - you have to read the manual and fix you configuration. FWIW, the
    > above traceback gives far enough clues (it's even written in all letters).


    The various ways to deal with this include running easy_install as the
    Web server user so that it can touch the directory and whatever else it
    needs, and perhaps to set up various permissions - something you
    wouldn't usually do; running easy_install with the --always-unzip flag
    when installing the package as root; using python setup.py install as
    usual.

    This situation looks either a lot like a use case whose popularity has
    been underestimated by the developers - intriguing given the number of
    people supposedly using setuptools for their Web application
    deployments - or a packaging mistake with the pysqlite2 .egg file.
    Either way, system packages rarely suffer from this kind of problem,
    and it's quite understandable (and justifiable) to consider this bad
    behaviour.

    > Or you may read easy_install's doc to avoid installing zipped eggs:
    > http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation


    Perhaps the developers should reverse their defaults and emulate the
    usual behaviour of package managers, unfashionable as that may seem.

    Paul
    Paul Boddie, Sep 11, 2006
    #3
  4. Wensheng

    Damjan Guest

    > I applaud you for studying the traceback in more depth than I can find
    > the motivation for, Bruno. ;-) However, this looks like a program using
    > some package installed by setuptools/easy_install needs to unpack that
    > package when running.


    See news:

    --
    damjan
    Damjan, Sep 11, 2006
    #4
  5. Wensheng

    Wensheng Guest

    Paul Boddie wrote:
    > Bruno Desthuilliers wrote:
    > > Wensheng a écrit :
    > > > I installed pysqlite2 using easy_install.
    > > > and got this when using it from modpython:
    > > > --------------------------------------------------------------------------------------------------
    > > > Mod_python error: "PythonHandler etc.modpython"

    >
    > [...]
    >
    > I applaud you for studying the traceback in more depth than I can find
    > the motivation for, Bruno. ;-) However, this looks like a program using
    > some package installed by setuptools/easy_install needs to unpack that
    > package when running.
    >
    > > > ExtractionError: Can't extract file(s) to egg cache

    > >
    > > > The following error occurred while trying to extract file(s) to the
    > > > Python egg cache:

    > >
    > > > [Errno 13] Permission denied: '/var/www/.python-eggs'

    >
    > And this wants to happen apparently in the home directory of the Web
    > server user, whose rights may be somewhat limited. Or perhaps one has
    > to initialise setuptools for any particular user by running it first as
    > that user: something that isn't a particularly common thing to do with
    > non-interactive users on UNIX-like systems.
    >
    > [...]
    >
    > > > Can peak developers fix this please?

    > >
    > > Why should they "fix" something that's
    > > 1/ documented
    > > -> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues

    >
    > "The Internal Structure of Python Eggs / STOP! This is not the first
    > document you should read!" A case of "Beware of the Leopard" perhaps?
    >
    > > 2/ not a an egg specific issue

    >
    > In my experience, most other package managers actually unpack their
    > resources when invoked, rather than leaving the programs to attempt
    > such things when running. Debian packages even compile the bytecode
    > files to ensure that nothing tries (and fails) to write them to
    > privileged locations afterwards.
    >
    > > 3/ really just a configuration/permission problem

    >
    > Which should have been solved when the package manager was invoked,
    > especially since this was most likely done as root.
    >
    > [...]
    >
    > > > in the mean time, I just have to use old "download, unzip/untar &&
    > > > python setup.py install" way.

    > >
    > > No - you have to read the manual and fix you configuration. FWIW, the
    > > above traceback gives far enough clues (it's even written in all letters).

    >
    > The various ways to deal with this include running easy_install as the
    > Web server user so that it can touch the directory and whatever else it
    > needs, and perhaps to set up various permissions - something you
    > wouldn't usually do; running easy_install with the --always-unzip flag
    > when installing the package as root; using python setup.py install as
    > usual.
    >
    > This situation looks either a lot like a use case whose popularity has
    > been underestimated by the developers - intriguing given the number of
    > people supposedly using setuptools for their Web application
    > deployments - or a packaging mistake with the pysqlite2 .egg file.
    > Either way, system packages rarely suffer from this kind of problem,
    > and it's quite understandable (and justifiable) to consider this bad
    > behaviour.
    >
    > > Or you may read easy_install's doc to avoid installing zipped eggs:
    > > http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

    >
    > Perhaps the developers should reverse their defaults and emulate the
    > usual behaviour of package managers, unfashionable as that may seem.
    >
    > Paul


    Thank you for clarification and other info/sugguestions.

    As I read the document, if package author set "zip_safe = false" in
    their setup.py, there would be no such problem.

    I also second the idea of making "--always-unzip" the default.

    It's EASY_install and EZ_setup, should be EASY so that we don't have to
    read the document:), right?

    If it's turbogears, yes, I would definitely read the document, but a
    package manager...
    Wensheng, Sep 11, 2006
    #5
  6. Paul Boddie wrote:
    > Bruno Desthuilliers wrote:
    >> Wensheng a écrit :
    >>> I installed pysqlite2 using easy_install.
    >>> and got this when using it from modpython:
    >>> --------------------------------------------------------------------------------------------------
    >>> Mod_python error: "PythonHandler etc.modpython"

    >
    > [...]
    >
    > I applaud you for studying the traceback in more depth than I can find
    > the motivation for, Bruno. ;-) However, this looks like a program using
    > some package installed by setuptools/easy_install needs to unpack that
    > package when running.


    Yes, this is explained in the doc. Pure Python eggs can run from zipped
    files, but when there's a binary extension it has to be unzipped. The
    OP's problem is that the PYTHON_EGG_CACHE env points to a directory not
    writable for the user account. The two obvious solutions are to change
    this env variable or reinstall the egg with the appropriate flag to
    avoid installing it as a zipped egg.


    >>> ExtractionError: Can't extract file(s) to egg cache
    >>> The following error occurred while trying to extract file(s) to the
    >>> Python egg cache:
    >>> [Errno 13] Permission denied: '/var/www/.python-eggs'

    >
    > And this wants to happen apparently in the home directory of the Web
    > server user, whose rights may be somewhat limited. Or perhaps one has
    > to initialise setuptools for any particular user by running it first as
    > that user:


    No, it's just a matter of specifying another value for the PYTHON_EGG_CACHE.

    >
    > [...]
    >
    >>> Can peak developers fix this please?

    >> Why should they "fix" something that's
    >> 1/ documented
    >> -> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues

    >
    > "The Internal Structure of Python Eggs / STOP! This is not the first
    > document you should read!"


    That doesn't mean you should not read it.

    > A case of "Beware of the Leopard" perhaps?


    !-)

    >> 2/ not a an egg specific issue

    >
    > In my experience, most other package managers actually unpack their
    > resources when invoked, rather than leaving the programs to attempt
    > such things when running.


    eggs are the Python's equivalent to Java's JAR, not a RPM-like. I said
    it was not an egg-specific issue (which is not totally accurate) because
    it mostly have to do with loading dynamic libs (.so, .dll etc) from
    zipped files.

    > Debian packages even compile the bytecode
    > files to ensure that nothing tries (and fails) to write them to
    > privileged locations afterwards.


    So does eggs. But once again, eggs are *not* a general-purpose package
    management system like apt, RPM, emerge etc...

    (snip)
    > This situation looks either a lot like a use case whose popularity has
    > been underestimated by the developers - intriguing given the number of
    > people supposedly using setuptools for their Web application
    > deployments - or a packaging mistake with the pysqlite2 .egg file.


    My VHO is that eggs needing to extract some of their content should not
    be packaged as zip-safe.


    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    Bruno Desthuilliers, Sep 12, 2006
    #6
  7. Wensheng

    Paul Boddie Guest

    Bruno Desthuilliers wrote:
    >
    > eggs are the Python's equivalent to Java's JAR, not a RPM-like. I said
    > it was not an egg-specific issue (which is not totally accurate) because
    > it mostly have to do with loading dynamic libs (.so, .dll etc) from
    > zipped files.


    True. It is notable that various Java application servers (to consider
    the questioner's original problem) do unpack .war files, which probably
    share more properties with .egg files than .jar files do. Indeed, I
    don't recall a single occasion where I've ever needed to unpack .jar
    files to have them successfully used by Java programs.

    > > Debian packages even compile the bytecode
    > > files to ensure that nothing tries (and fails) to write them to
    > > privileged locations afterwards.

    >
    > So does eggs. But once again, eggs are *not* a general-purpose package
    > management system like apt, RPM, emerge etc...


    However, it seems to me that the setuptools and EasyInstall ecosystem,
    whilst a remarkable achievement, has grown in scope sufficiently to
    confuse such comparisons. Some .egg files are more than just being like
    ..jar files, as I note above, and with additional dependency information
    included (as seems to be encouraged), one is suddenly required to deal
    with a system that resembles a Python-centric package and dependency
    management system rolled into one, operating in parallel with whatever
    native package manager and whatever dependency management system
    (they're typically separate things) that may be in operation.

    Paul
    Paul Boddie, Sep 12, 2006
    #7
  8. Paul Boddie wrote:
    > Bruno Desthuilliers wrote:
    >> eggs are the Python's equivalent to Java's JAR, not a RPM-like. I said
    >> it was not an egg-specific issue (which is not totally accurate) because
    >> it mostly have to do with loading dynamic libs (.so, .dll etc) from
    >> zipped files.

    >
    > True. It is notable that various Java application servers (to consider
    > the questioner's original problem) do unpack .war files, which probably
    > share more properties with .egg files than .jar files do. Indeed, I
    > don't recall a single occasion where I've ever needed to unpack .jar
    > files to have them successfully used by Java programs.


    AFAIK, jar files don't include native dynamic libs... But I'm not a Java
    expert (happily forgot most of what I used to know about this language).

    >>> Debian packages even compile the bytecode
    >>> files to ensure that nothing tries (and fails) to write them to
    >>> privileged locations afterwards.

    >> So does eggs. But once again, eggs are *not* a general-purpose package
    >> management system like apt, RPM, emerge etc...

    >
    > However, it seems to me that the setuptools and EasyInstall ecosystem,
    > whilst a remarkable achievement, has grown in scope sufficiently to
    > confuse such comparisons. Some .egg files are more than just being like
    > .jar files, as I note above, and with additional dependency information
    > included (as seems to be encouraged), one is suddenly required to deal
    > with a system that resembles a Python-centric package and dependency
    > management system rolled into one, operating in parallel with whatever
    > native package manager and whatever dependency management system
    > (they're typically separate things) that may be in operation.


    Yes, true. But I'd still say that it's more a problem of proper
    packaging (ie, an egg with dynamic libs inside should better not be
    packaged as 'zip-safe') than a bug in the eggs system itself.

    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    Bruno Desthuilliers, Sep 12, 2006
    #8
  9. Wensheng

    Paul Boddie Guest

    Bruno Desthuilliers wrote:
    >
    > AFAIK, jar files don't include native dynamic libs... But I'm not a Java
    > expert (happily forgot most of what I used to know about this language).


    I can't be bothered to read the specification to say what is allowed in
    ..jar files, but .war files frequently get unpacked by various
    application servers either for efficiency reasons or in order to access
    resources that have to be processed. Whether such software is badly
    designed or not (eg. the horror of Oracle Containers for Java), there's
    a precedent for such behaviour that needs to be reviewed continuously.

    [...]

    > Yes, true. But I'd still say that it's more a problem of proper
    > packaging (ie, an egg with dynamic libs inside should better not be
    > packaged as 'zip-safe') than a bug in the eggs system itself.


    There's always going to be an issue of how much you need to know in
    order to be able to package stuff effectively. With Debian packaging,
    there's a culture around this with a moderately steep learning curve;
    with distutils, a simple script is often enough for a Python-only
    package, but various strangely defined options for installing scripts,
    data and programs cause the learning curve to point more steeply
    upwards. I imagine that providing native code libraries can be even
    more of a hassle, but at such a point setuptools should really start to
    help out and detect the situations where compressed .egg files have to
    be unpacked.

    But returning briefly to my point about what setuptools and its
    associates should be used for, I think the following message sums up
    one side of the conflicting demands made upon it:

    http://mail.python.org/pipermail/python-dev/2004-October/049572.html

    Because one platform has no package manager, people reach for the
    closest solution, inflicting a (meta)package format on everyone else.
    Attempts to make distutils work with .deb files seem to have stalled,
    and I don't buy the "awkward Debian" argument that some suggest is the
    reason for this. It's more likely that the metapackage format in
    question isn't expressive enough to get the job done.

    Paul
    Paul Boddie, Sep 13, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Julio C?sar Carrascal

    ModPython and embedding

    Julio C?sar Carrascal, Nov 13, 2003, in forum: Python
    Replies:
    0
    Views:
    280
    Julio C?sar Carrascal
    Nov 13, 2003
  2. Sam
    Replies:
    2
    Views:
    322
    Steve Holden
    Jan 5, 2005
  3. Bob Van Zant

    Re: modpython, apache and windows

    Bob Van Zant, Jan 5, 2005, in forum: Python
    Replies:
    1
    Views:
    327
  4. Catherine
    Replies:
    2
    Views:
    317
    John Nagle
    Oct 8, 2007
  5. R. Bernstein
    Replies:
    3
    Views:
    364
    R. Bernstein
    Dec 21, 2008
Loading...

Share This Page