Parent module not loaded error

[Ståle Undheim]

I have a pretty strange error that I can't figure out the cause off.
This is in a Django app.

I am using berkelydb, with secondary databases for indexing. The
secondary databases are associated with a callback that uses cPickle
to serialize index values. The problem is that cPickle.dumps(value)
fails when I run it through mod_wsgi or mod_python on the deployment
server (Ubuntu Linux 8.10), but works fine when when I use Djangos
runserver on the same computer. Also, on my own computer (Ubuntu Linux
8.10), it works fine through mod_python.

Each time I call on cPickle.dumps(value) I get:
SystemError("Parent module 'd4' not loaded",)

Using sys.exc_info() I get no type or traceback, just that exception.
The callstack as far as I can figure is:

django
view
bsddb.put
indexer callback
cPickle.dumps

cPickle.loads works fine within the callback, I can also use
cPickle.dumps() outside the indexer callback. But inside the callback,
I need to use normal pickle instead. I just want to know why. I am
assuming it has something to do with the fact that I go from python to
C (bsddb) back to python (indexer callback) and back to C (cPickle).
It is still strange then that cPickle.loads() work, but not
cPickle.dumps(). I am also only storing integer values when I get this
error, so no fancy list, dictionaries or objects.

 

[Graham Dumpleton]

I have a pretty strange error that I can't figure out the cause off.
This is in a Django app.

I am using berkelydb, with secondary databases for indexing. The
secondary databases are associated with a callback that uses cPickle
to serialize index values. The problem is that cPickle.dumps(value)
fails when I run it throughmod_wsgior mod_python on the deployment
server (Ubuntu Linux 8.10), but works fine when when I use Djangos
runserver on the same computer. Also, on my own computer (Ubuntu Linux
8.10), it works fine through mod_python.

Each time I call on cPickle.dumps(value) I get:
SystemError("Parent module 'd4' not loaded",)

Using sys.exc_info() I get no type or traceback, just that exception.
The callstack as far as I can figure is:

django
view
bsddb.put
indexer callback
cPickle.dumps

cPickle.loads works fine within the callback, I can also use
cPickle.dumps() outside the indexer callback. But inside the callback,
I need to use normal pickle instead. I just want to know why. I am
assuming it has something to do with the fact that I go from python to
C (bsddb) back to python (indexer callback) and back to C (cPickle).
It is still strange then that cPickle.loads() work, but not
cPickle.dumps(). I am also only storing integer values when I get this
error, so no fancy list, dictionaries or objects.

Where is module 'd4'? Is that one of yours defined inside of your
Django project?

One of the problems with Django runserver is that the parent of the
site directory and the site directory are effectively starting points
for searching for modules, even though neither is explicitly listed in
sys.path. The parent is used because runserver does momentarily add
parent to sys.path to import site package root, after which it removes
directory from sys.path. The site directory itself is used because for
runserver that is the current working directory and imports look there
automatically.

End result is that modules inside of site can be imported either via
site package root, or direct. For example, if d4.py was at same level
as settings.py file and site was called mysite, you could import it as
mysite.d4 or just d4. This is bad because means you could end up with
two copies of a module imported under the different name references.

If they way the pickle was done in application was such that reference
was via d4, and then when you deployed to mod_python or mod_wsgi you
only added parent directory of site to Python path, then it will not
be able to find d4, since at that point would only be found as
mysite.d4.

If this is the case, add the site directory to Python path by setting
PythonPath for mod_python or sys.path in WSGI script file for
mod_wsgi. Better still, fix up your code so module references in code,
ie., urls.py, or anywhere else, are always done by site package root
reference and not direct. Ie., such that it uses mysite.d4 and not
just d4.

Would that seem about right?

Graham

 

[Ståle Undheim]

Where is module 'd4'? Is that one of yours defined inside of your
Django project?

d4 is the root project name. This message occurs in d4/views.py.

Here is how I set up my wsgi.py - which is in the d4/ folder:
sys.path.append(path.join(path.dirname(__file__), '..'))

os.environ['DJANGO_SETTINGS_MODULE'] = 'd4.settings'
application = django.core.handlers.wsgi.WSGIHandler()

One of the problems with Django runserver is that the parent of the
site directory and the site directory are effectively starting points
for searching for modules, even though neither is explicitly listed in
sys.path. The parent is used because runserver does momentarily add
parent to sys.path to import site package root, after which it removes
directory from sys.path. The site directory itself is used because for
runserver that is the current working directory and imports look there
automatically.

End result is that modules inside of site can be imported either via
site package root, or direct. For example, if d4.py was at same level
as settings.py file and site was called mysite, you could import it as
mysite.d4 or just d4. This is bad because means you could end up with
two copies of a module imported under the different name references.

Everywhere in the code, the imports refer to d4.something.something.
All urls.py files refer to the full package names. The test code where
I am running this is actually quit minimal.

If they way the pickle was done in application was such that reference
was via d4, and then when you deployed to mod_python or mod_wsgi you
only added parent directory of site to Python path, then it will not
be able to find d4, since at that point would only be found as
mysite.d4.

I use "import cPickle as pickle" at the top off my views file. So I
don't reference imports from other files.

If this is the case, add the site directory to Python path by setting
PythonPath for mod_python or sys.path in WSGI script file for
mod_wsgi. Better still, fix up your code so module references in code,
ie., urls.py, or anywhere else, are always done by site package root
reference and not direct. Ie., such that it uses mysite.d4 and not
just d4.

Would that seem about right?

Unfortunatly, at least to me, it seems that all code is allready
correctly using packagenames. The views.py file where I am testing
this is a simple file that doesn't depend on anything else in the
project. It only imports a set of system modules for testing and info.
The only "dependency" is really the urls.py file, which does refer to
all the views as "d4.views.somemethod".

 

[Graham Dumpleton]

Where is module 'd4'? Is that one of yours defined inside of your
Django project?

d4 is the root project name. This message occurs in d4/views.py.

Here is how I set up my wsgi.py - which is in the d4/ folder:
  sys.path.append(path.join(path.dirname(__file__), '..'))

  os.environ['DJANGO_SETTINGS_MODULE'] = 'd4.settings'
  application = django.core.handlers.wsgi.WSGIHandler()

One of the problems with Django runserver is that the parent of the
site directory and the site directory are effectively starting points
for searching for modules, even though neither is explicitly listed in
sys.path. The parent is used because runserver does momentarily add
parent to sys.path to import site package root, after which it removes
directory from sys.path. The site directory itself is used because for
runserver that is the current working directory and imports look there
automatically.

End result is that modules inside of site can be imported either via
site package root, or direct. For example, if d4.py was at same level
as settings.py file and site was called mysite, you could import it as
mysite.d4 or just d4. This is bad because means you could end up with
two copies of a module imported under the different name references.

Everywhere in the code, the imports refer to d4.something.something.
All urls.py files refer to the full package names. The test code where
I am running this is actually quit minimal.

If they way the pickle was done in application was such that reference
was via d4, and then when you deployed to mod_python or mod_wsgi you
only added parent directory of site to Python path, then it will not
be able to find d4, since at that point would only be found as
mysite.d4.

I use "import cPickle as pickle" at the top off my views file. So I
don't reference imports from other files.

If this is the case, add the site directory to Python path by setting
PythonPath for mod_python or sys.path in WSGI script file for
mod_wsgi. Better still, fix up your code so module references in code,
ie., urls.py, or anywhere else, are always done by site package root
reference and not direct. Ie., such that it uses mysite.d4 and not
just d4.

Would that seem about right?

Unfortunatly, at least to me, it seems that all code is allready
correctly using packagenames. The views.py file where I am testing
this is a simple file that doesn't depend on anything else in the
project. It only imports a set of system modules for testing and info.
The only "dependency" is really the urls.py file, which does refer to
all the views as "d4.views.somemethod".

As a test, just prior to where unpickle is done, do:

import sys
import d4

print >> sys.stderr, "d4.__name__", dr.__name__
print >> sys.stderr, "d4.__file__", dr.__file__

This will just confirm that manual import works and what values of
those attributes are. The messages will end up in Apache error log for
that virtual host or main server error log as appropriate.

Graham

 

[Ståle Undheim]

As a test, just prior to where unpickle is done, do:

  import sys
  import d4

  print >> sys.stderr, "d4.__name__", dr.__name__
  print >> sys.stderr, "d4.__file__", dr.__file__

This will just confirm that manual import works and what values of
those attributes are. The messages will end up in Apache error log for
that virtual host or main server error log as appropriate.

d4.__name__ d4
d4.__file__ /home/petromarker/django-apps/d4/src/d4/../d4/__init__.pyc

It's the same both before and after the exception.