egg and modpython

[Wensheng]

I installed pysqlite2 using easy_install.
and got this when using it from modpython:
--------------------------------------------------------------------------------------------------
Mod_python error: "PythonHandler etc.modpython"

Traceback (most recent call last):

File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
299, in HandlerDispatch
result = object(req)

File "/usr/lib/python2.4/site-packages/spark/proj/etc/modpython.py",
line 21, in handler
return ReqModPython(req).run()

File "/usr/lib/python2.4/site-packages/spark/ReqBase.py", line 141,
in run
con_mod = __import__(conf,'','',[''])

File "/var/www/hello/controllers/wiki_controller.py", line 6, in ?
Storm.conn(driver='sqlite',db="/var/www/hello/etc/sparkdemo.db")

File "/usr/lib/python2.4/site-packages/spark/storm.py", line 64, in
conn
cls.__CONN = __import__('db'+driver).DbSpark(**kwds)

File "/usr/lib/python2.4/site-packages/spark/dbsqlite.py", line 7, in
?
try: from pysqlite2 import dbapi2 as sqlite

File "build/bdist.linux-i686/egg/pysqlite2/dbapi2.py", line 27, in ?

File "build/bdist.linux-i686/egg/pysqlite2/_sqlite.py", line 7, in ?

File "build/bdist.linux-i686/egg/pysqlite2/_sqlite.py", line 4, in
__bootstrap__

File
"/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
line 799, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(

File
"/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
line 1229, in get_resource_filename
return self._extract_resource(manager, zip_path)

File
"/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
line 1249, in _extract_resource
real_path = manager.get_cache_path(

File
"/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
line 880, in get_cache_path
self.extraction_error()

File
"/usr/lib/python2.4/site-packages/setuptools-0.6c2-py2.4.egg/pkg_resources.py",
line 846, in extraction_error
raise err

ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the
Python egg
cache:

[Errno 13] Permission denied: '/var/www/.python-eggs'

The Python egg cache directory is currently set to:

/var/www/.python-eggs

Perhaps your account does not have write access to this directory? You
can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
----------------------------------------------------------------------------

Can peak developers fix this please?


I know I won't have this problem with python 2.5 because it has
sqlite3, but this affect other eggs too!
Here's similiar occurance, I can't reply it since it's too old.
http://groups.google.com/group/comp...read/thread/d35e5264a6b491d5/cee519cb5557ad0e

in the mean time, I just have to use old "download, unzip/untar &&
python setup.py install" way.

Thanks

 

[Bruno Desthuilliers]

Wensheng a écrit :

I installed pysqlite2 using easy_install.
and got this when using it from modpython:
--------------------------------------------------------------------------------------------------
Mod_python error: "PythonHandler etc.modpython"

Traceback (most recent call last):
(snip)

ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the
Python egg
cache:

[Errno 13] Permission denied: '/var/www/.python-eggs'

The Python egg cache directory is currently set to:

/var/www/.python-eggs

Perhaps your account does not have write access to this directory? You
can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.

Why should they "fix" something that's
1/ documented
-> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues
2/ not a an egg specific issue
3/ really just a configuration/permission problem

(snip)

in the mean time, I just have to use old "download, unzip/untar &&
python setup.py install" way.

No - you have to read the manual and fix you configuration. FWIW, the
above traceback gives far enough clues (it's even written in all letters).

Or you may read easy_install's doc to avoid installing zipped eggs:
http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

HTH

 

[Paul Boddie]

Wensheng a écrit :

[...]

I applaud you for studying the traceback in more depth than I can find
the motivation for, Bruno. ;-) However, this looks like a program using
some package installed by setuptools/easy_install needs to unpack that
package when running.

ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the
Python egg cache:

[Errno 13] Permission denied: '/var/www/.python-eggs'

And this wants to happen apparently in the home directory of the Web
server user, whose rights may be somewhat limited. Or perhaps one has
to initialise setuptools for any particular user by running it first as
that user: something that isn't a particularly common thing to do with
non-interactive users on UNIX-like systems.

[...]

Why should they "fix" something that's
1/ documented
-> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues

"The Internal Structure of Python Eggs / STOP! This is not the first
document you should read!" A case of "Beware of the Leopard" perhaps?

2/ not a an egg specific issue

In my experience, most other package managers actually unpack their
resources when invoked, rather than leaving the programs to attempt
such things when running. Debian packages even compile the bytecode
files to ensure that nothing tries (and fails) to write them to
privileged locations afterwards.

3/ really just a configuration/permission problem

Which should have been solved when the package manager was invoked,
especially since this was most likely done as root.

[...]

No - you have to read the manual and fix you configuration. FWIW, the
above traceback gives far enough clues (it's even written in all letters).

The various ways to deal with this include running easy_install as the
Web server user so that it can touch the directory and whatever else it
needs, and perhaps to set up various permissions - something you
wouldn't usually do; running easy_install with the --always-unzip flag
when installing the package as root; using python setup.py install as
usual.

This situation looks either a lot like a use case whose popularity has
been underestimated by the developers - intriguing given the number of
people supposedly using setuptools for their Web application
deployments - or a packaging mistake with the pysqlite2 .egg file.
Either way, system packages rarely suffer from this kind of problem,
and it's quite understandable (and justifiable) to consider this bad
behaviour.

Or you may read easy_install's doc to avoid installing zipped eggs:
http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

Perhaps the developers should reverse their defaults and emulate the
usual behaviour of package managers, unfashionable as that may seem.

Paul

 

[Damjan]

I applaud you for studying the traceback in more depth than I can find

the motivation for, Bruno. ;-) However, this looks like a program using
some package installed by setuptools/easy_install needs to unpack that
package when running.

See

 

[Wensheng]

Wensheng a écrit :

[...]

I applaud you for studying the traceback in more depth than I can find
the motivation for, Bruno. ;-) However, this looks like a program using
some package installed by setuptools/easy_install needs to unpack that
package when running.

ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the
Python egg cache:

[Errno 13] Permission denied: '/var/www/.python-eggs'

And this wants to happen apparently in the home directory of the Web
server user, whose rights may be somewhat limited. Or perhaps one has
to initialise setuptools for any particular user by running it first as
that user: something that isn't a particularly common thing to do with
non-interactive users on UNIX-like systems.

[...]

Why should they "fix" something that's
1/ documented
-> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues

"The Internal Structure of Python Eggs / STOP! This is not the first
document you should read!" A case of "Beware of the Leopard" perhaps?

2/ not a an egg specific issue

In my experience, most other package managers actually unpack their
resources when invoked, rather than leaving the programs to attempt
such things when running. Debian packages even compile the bytecode
files to ensure that nothing tries (and fails) to write them to
privileged locations afterwards.

3/ really just a configuration/permission problem

Which should have been solved when the package manager was invoked,
especially since this was most likely done as root.

[...]

No - you have to read the manual and fix you configuration. FWIW, the
above traceback gives far enough clues (it's even written in all letters).

The various ways to deal with this include running easy_install as the
Web server user so that it can touch the directory and whatever else it
needs, and perhaps to set up various permissions - something you
wouldn't usually do; running easy_install with the --always-unzip flag
when installing the package as root; using python setup.py install as
usual.

This situation looks either a lot like a use case whose popularity has
been underestimated by the developers - intriguing given the number of
people supposedly using setuptools for their Web application
deployments - or a packaging mistake with the pysqlite2 .egg file.
Either way, system packages rarely suffer from this kind of problem,
and it's quite understandable (and justifiable) to consider this bad
behaviour.

Or you may read easy_install's doc to avoid installing zipped eggs:
http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

Perhaps the developers should reverse their defaults and emulate the
usual behaviour of package managers, unfashionable as that may seem.

Paul

Thank you for clarification and other info/sugguestions.

As I read the document, if package author set "zip_safe = false" in
their setup.py, there would be no such problem.

I also second the idea of making "--always-unzip" the default.

It's EASY_install and EZ_setup, should be EASY so that we don't have to
read the document, right?

If it's turbogears, yes, I would definitely read the document, but a
package manager...

 

[Bruno Desthuilliers]

Wensheng a écrit :

[...]

I applaud you for studying the traceback in more depth than I can find
the motivation for, Bruno. ;-) However, this looks like a program using
some package installed by setuptools/easy_install needs to unpack that
package when running.

Yes, this is explained in the doc. Pure Python eggs can run from zipped
files, but when there's a binary extension it has to be unzipped. The
OP's problem is that the PYTHON_EGG_CACHE env points to a directory not
writable for the user account. The two obvious solutions are to change
this env variable or reinstall the egg with the appropriate flag to
avoid installing it as a zipped egg.

ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the
Python egg cache:
[Errno 13] Permission denied: '/var/www/.python-eggs'

And this wants to happen apparently in the home directory of the Web
server user, whose rights may be somewhat limited. Or perhaps one has
to initialise setuptools for any particular user by running it first as
that user:

No, it's just a matter of specifying another value for the PYTHON_EGG_CACHE.

[...]

Why should they "fix" something that's
1/ documented
-> http://peak.telecommunity.com/DevCenter/EggFormats#zip-file-issues

"The Internal Structure of Python Eggs / STOP! This is not the first
document you should read!"

That doesn't mean you should not read it.

A case of "Beware of the Leopard" perhaps?
!-)


In my experience, most other package managers actually unpack their
resources when invoked, rather than leaving the programs to attempt
such things when running.

eggs are the Python's equivalent to Java's JAR, not a RPM-like. I said
it was not an egg-specific issue (which is not totally accurate) because
it mostly have to do with loading dynamic libs (.so, .dll etc) from
zipped files.

Debian packages even compile the bytecode
files to ensure that nothing tries (and fails) to write them to
privileged locations afterwards.

So does eggs. But once again, eggs are *not* a general-purpose package
management system like apt, RPM, emerge etc...

(snip)

This situation looks either a lot like a use case whose popularity has
been underestimated by the developers - intriguing given the number of
people supposedly using setuptools for their Web application
deployments - or a packaging mistake with the pysqlite2 .egg file.

My VHO is that eggs needing to extract some of their content should not
be packaged as zip-safe.

 

[Paul Boddie]

eggs are the Python's equivalent to Java's JAR, not a RPM-like. I said
it was not an egg-specific issue (which is not totally accurate) because
it mostly have to do with loading dynamic libs (.so, .dll etc) from
zipped files.

True. It is notable that various Java application servers (to consider
the questioner's original problem) do unpack .war files, which probably
share more properties with .egg files than .jar files do. Indeed, I
don't recall a single occasion where I've ever needed to unpack .jar
files to have them successfully used by Java programs.

So does eggs. But once again, eggs are *not* a general-purpose package
management system like apt, RPM, emerge etc...

However, it seems to me that the setuptools and EasyInstall ecosystem,
whilst a remarkable achievement, has grown in scope sufficiently to
confuse such comparisons. Some .egg files are more than just being like
..jar files, as I note above, and with additional dependency information
included (as seems to be encouraged), one is suddenly required to deal
with a system that resembles a Python-centric package and dependency
management system rolled into one, operating in parallel with whatever
native package manager and whatever dependency management system
(they're typically separate things) that may be in operation.

Paul

 

[Bruno Desthuilliers]

True. It is notable that various Java application servers (to consider
the questioner's original problem) do unpack .war files, which probably
share more properties with .egg files than .jar files do. Indeed, I
don't recall a single occasion where I've ever needed to unpack .jar
files to have them successfully used by Java programs.

AFAIK, jar files don't include native dynamic libs... But I'm not a Java
expert (happily forgot most of what I used to know about this language).

However, it seems to me that the setuptools and EasyInstall ecosystem,
whilst a remarkable achievement, has grown in scope sufficiently to
confuse such comparisons. Some .egg files are more than just being like
.jar files, as I note above, and with additional dependency information
included (as seems to be encouraged), one is suddenly required to deal
with a system that resembles a Python-centric package and dependency
management system rolled into one, operating in parallel with whatever
native package manager and whatever dependency management system
(they're typically separate things) that may be in operation.

Yes, true. But I'd still say that it's more a problem of proper
packaging (ie, an egg with dynamic libs inside should better not be
packaged as 'zip-safe') than a bug in the eggs system itself.

 

[Paul Boddie]

AFAIK, jar files don't include native dynamic libs... But I'm not a Java
expert (happily forgot most of what I used to know about this language).

I can't be bothered to read the specification to say what is allowed in
..jar files, but .war files frequently get unpacked by various
application servers either for efficiency reasons or in order to access
resources that have to be processed. Whether such software is badly
designed or not (eg. the horror of Oracle Containers for Java), there's
a precedent for such behaviour that needs to be reviewed continuously.

[...]

Yes, true. But I'd still say that it's more a problem of proper
packaging (ie, an egg with dynamic libs inside should better not be
packaged as 'zip-safe') than a bug in the eggs system itself.

There's always going to be an issue of how much you need to know in
order to be able to package stuff effectively. With Debian packaging,
there's a culture around this with a moderately steep learning curve;
with distutils, a simple script is often enough for a Python-only
package, but various strangely defined options for installing scripts,
data and programs cause the learning curve to point more steeply
upwards. I imagine that providing native code libraries can be even
more of a hassle, but at such a point setuptools should really start to
help out and detect the situations where compressed .egg files have to
be unpacked.

But returning briefly to my point about what setuptools and its
associates should be used for, I think the following message sums up
one side of the conflicting demands made upon it:

http://mail.python.org/pipermail/python-dev/2004-October/049572.html

Because one platform has no package manager, people reach for the
closest solution, inflicting a (meta)package format on everyone else.
Attempts to make distutils work with .deb files seem to have stalled,
and I don't buy the "awkward Debian" argument that some suggest is the
reason for this. It's more likely that the metapackage format in
question isn't expressive enough to get the job done.

Paul