Import from database

[Steve Holden]

I'm trying to load module code from a database, which stores for each
module its full name, code, load date and a Boolean indicating whether
it's a package or not.

The following simple program:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
#import bsddb.db
import a.b.c.d
import bsddb

gives a traceback from its last line. The whole output is

$ python -i test.py
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 7, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path
It appears that for some reason execution of the code in
bsddb/__init__.pyc (loaded from the database using my importer) is
causing the os module to execute again although it has already run and
been cached in sys.modules.

Any explanations would be welcome. The importer module is shown below (a
posting with an attachment to avoid code folding didn't appear to make
it out)

Also if someone can suggest how associate the source file with
module/package code from the database that would make tracebacks look
more conventional, though I'd like to retain the indication the module
was loaded by dbimp somehow.

#
# Import modules from a database
#
import sys, db, marshal, imp, new
conn = db.conn()
curs = conn.cursor()

curs.execute("select modName from module")
impdict = {}
for n in [x[0] for x in curs.fetchall()]:
impdict[n] = 1


class dbimporter(object):

def __init__(self, item, *args, **kw):
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(self, fullname, path=None):
#print "find_module:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(self, modname):
print "load_module:", modname
if modname in sys.modules:
return sys.modules[modname]
curs.execute("select modCode, modPackage from module where
modName=%s", (modname, ))
row = curs.fetchone() # should only BE one ...S
if not row:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package = row
code = marshal.loads(code)
module = new.module(modname)
sys.modules[modname] = module
module.__name__ = modname
exec code in module.__dict__
module.__file__ = "db:%s" % modname
module.__loader__ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path
hook?


regards
Steve

 

[Kartic]

Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

 

[Steve Holden]

Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

I'm not sure I understand, as I am currently relying on the system
implementations of any modules that happen to be loaded before my code
picks up. As to what's in the database

SELECT *
FROM `module`
WHERE modName LIKE '%path%'

shows that I have macpathm macurlpath, ntpath, nturlpath, os2emxpath and
posixpath all there (but, as I say, I've already executed the standard
modules by the time anything of mine gets to run). If I alter my test
program to:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
k = sys.modules.keys()
k.sort()
for kk in k:
print kk
#import bsddb.db
import a.b.c.d
import bsddb

then I get as output

MySQLdb
MySQLdb.MySQLdb
...
MySQLdb.types
UserDict
__builtin__
__main__
_codecs
_locale
_mysql
_mysql_exceptions
_sre
array
cPickle
codecs
copy
copy_reg
db
dbimp
...
mx.Misc
mx.Misc.LazyModule
new
nt
ntpath
os
os.path
re
...
zipimport
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 11, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

In other words, the os module is /already/ in sys.smodules, as is
os.path, yet the interpreter is complaining (I presume) that os.path is
not a module. I don't even know *why* os is being executed a second
time. I can only assume it's being imported as some other name like
"bsddb.os" and some element of the import system is actually doing an
import rather than refusing to guess.

I presume I need to control the import process more closely to make sure
that this import attempt is rejected, but I can't see how to do that.

regards
Steve

 

[Kartic]

Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

 

[Steve Holden]

Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

Here it is, plus also the loader program I used to suck in the standard
library (in case it's that that's faulty):

CREATE TABLE `module` (
`modName` varchar(25) NOT NULL default '',
`modPackage` tinyint(1) NOT NULL default '0',
`modCompileTime` timestamp(14) NOT NULL,
`modCode` longtext,
PRIMARY KEY (`modName`)
) TYPE=MyISAM;

#### WARNING: email client may wrap some lines ...
#
# Establish standard library in database
#
import db
import os
import glob
import sys
import marshal

conn = db.conn()
curs = conn.cursor()

if sys.argv[1] == "-d":
curs.execute("delete from module")
print "All existing modules cleared"
del sys.argv[1]

def importpy(path, modname, package):
print "Importing", path, modname
c = compile(file(path).read(), "db:%s" % modname, "exec")
curs.execute("""delete from module where modName = %s""", (modname,))
curs.execute("""insert into module (modName, modCode, modPackage,
modCompileTime)
values (%s, %s, %s, now())""", (modname,
marshal.dumps(c), package))
print "Added", modname
conn.commit()

def importall(path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f):
fn = os.path.join(path, f, "__init__.py")
if os.path.exists(fn):
ml = modlist + [f]
importpy(fn, ".".join(ml), 1)
importall(os.path.join(path, f), ml)
elif f.endswith('.py') and '.' not in f[:-3] and f !=
"__init__.py":
importpy(os.path.join(path, f), ".".join(modlist+[f[:-3]]), 0)

if __name__ == "__main__":
for path in sys.argv[1:]:
importall(path, [])

regards
Steve