Import from database

Discussion in 'Python' started by Steve Holden, Jan 24, 2005.

  1. Steve Holden

    Steve Holden Guest

    I'm trying to load module code from a database, which stores for each
    module its full name, code, load date and a Boolean indicating whether
    it's a package or not.

    The following simple program:

    import dbimp, sys

    if __name__ == "__main__":
    dbimp.install()
    #import bsddb.db
    import a.b.c.d
    import bsddb

    gives a traceback from its last line. The whole output is

    $ python -i test.py
    Accepted *db*
    found a in db
    load_module: a
    a loaded: <module 'a' from 'db:a'> pkg: 1
    found a.b in db
    load_module: a.b
    a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
    found a.b.c in db
    load_module: a.b.c
    a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
    found a.b.c.d in db
    load_module: a.b.c.d
    a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
    found bsddb in db
    load_module: bsddb
    found weakref in db
    load_module: weakref
    weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
    Traceback (most recent call last):
    File "test.py", line 7, in ?
    import bsddb
    File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
    exec code in module.__dict__
    File "db:bsddb", line 62, in ?
    File "/usr/lib/python2.4/os.py", line 133, in ?
    from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
    altsep,
    ImportError: No module named path
    >>>


    It appears that for some reason execution of the code in
    bsddb/__init__.pyc (loaded from the database using my importer) is
    causing the os module to execute again although it has already run and
    been cached in sys.modules.

    Any explanations would be welcome. The importer module is shown below (a
    posting with an attachment to avoid code folding didn't appear to make
    it out)

    Also if someone can suggest how associate the source file with
    module/package code from the database that would make tracebacks look
    more conventional, though I'd like to retain the indication the module
    was loaded by dbimp somehow.

    #
    # Import modules from a database
    #
    import sys, db, marshal, imp, new
    conn = db.conn()
    curs = conn.cursor()

    curs.execute("select modName from module")
    impdict = {}
    for n in [x[0] for x in curs.fetchall()]:
    impdict[n] = 1


    class dbimporter(object):

    def __init__(self, item, *args, **kw):
    if item != "*db*":
    raise ImportError
    print "Accepted", item

    def find_module(self, fullname, path=None):
    #print "find_module:", fullname, "from", path
    if fullname not in impdict:
    #print "Bailed on", fullname
    return None
    else:
    print "found", fullname, "in db"
    return self

    def load_module(self, modname):
    print "load_module:", modname
    if modname in sys.modules:
    return sys.modules[modname]
    curs.execute("select modCode, modPackage from module where
    modName=%s", (modname, ))
    row = curs.fetchone() # should only BE one ...S
    if not row:
    #print modname, "not found in db"
    raise ImportError, "DB module %s not found in modules"
    code, package = row
    code = marshal.loads(code)
    module = new.module(modname)
    sys.modules[modname] = module
    module.__name__ = modname
    exec code in module.__dict__
    module.__file__ = "db:%s" % modname
    module.__loader__ = dbimporter
    if package:
    module.__path__ = sys.path
    exec code in module.__dict__
    print modname, "loaded:", repr(module), "pkg:", package
    return module

    def install():
    sys.path_hooks.append(dbimporter)
    sys.path_importer_cache.clear() # probably not necessary
    sys.path.insert(0, "*db*") # probably not needed with a metea-path
    hook?


    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
    Steve Holden, Jan 24, 2005
    #1
    1. Advertising

  2. Steve Holden

    Kartic Guest

    Steve,

    I believe you have to put ntpath, macpath and posixpath in the module
    database for os.path to work.

    I tried it with zipimporter builtin and I got the same traceback till I
    added ntpath.py to my zip file. (Of course, I renamed the original
    ntpath to _ntpath so that the original did not get imported)
    Thanks,
    --Kartic
    Kartic, Jan 24, 2005
    #2
    1. Advertising

  3. Steve Holden

    Steve Holden Guest

    Kartic wrote:

    > Steve,
    >
    > I believe you have to put ntpath, macpath and posixpath in the module
    > database for os.path to work.
    >
    > I tried it with zipimporter builtin and I got the same traceback till I
    > added ntpath.py to my zip file. (Of course, I renamed the original
    > ntpath to _ntpath so that the original did not get imported)
    > Thanks,
    > --Kartic
    >

    I'm not sure I understand, as I am currently relying on the system
    implementations of any modules that happen to be loaded before my code
    picks up. As to what's in the database

    SELECT *
    FROM `module`
    WHERE modName LIKE '%path%'

    shows that I have macpathm macurlpath, ntpath, nturlpath, os2emxpath and
    posixpath all there (but, as I say, I've already executed the standard
    modules by the time anything of mine gets to run). If I alter my test
    program to:

    import dbimp, sys

    if __name__ == "__main__":
    dbimp.install()
    k = sys.modules.keys()
    k.sort()
    for kk in k:
    print kk
    #import bsddb.db
    import a.b.c.d
    import bsddb

    then I get as output

    MySQLdb
    MySQLdb.MySQLdb
    ...
    MySQLdb.types
    UserDict
    __builtin__
    __main__
    _codecs
    _locale
    _mysql
    _mysql_exceptions
    _sre
    array
    cPickle
    codecs
    copy
    copy_reg
    db
    dbimp
    ...
    mx.Misc
    mx.Misc.LazyModule
    new
    nt
    ntpath
    os
    os.path
    re
    ...
    zipimport
    Accepted *db*
    found a in db
    load_module: a
    a loaded: <module 'a' from 'db:a'> pkg: 1
    found a.b in db
    load_module: a.b
    a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
    found a.b.c in db
    load_module: a.b.c
    a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
    found a.b.c.d in db
    load_module: a.b.c.d
    a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
    found bsddb in db
    load_module: bsddb
    found weakref in db
    load_module: weakref
    weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
    Traceback (most recent call last):
    File "test.py", line 11, in ?
    import bsddb
    File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
    exec code in module.__dict__
    File "db:bsddb", line 62, in ?
    File "/usr/lib/python2.4/os.py", line 133, in ?
    from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
    altsep,
    ImportError: No module named path

    In other words, the os module is /already/ in sys.smodules, as is
    os.path, yet the interpreter is complaining (I presume) that os.path is
    not a module. I don't even know *why* os is being executed a second
    time. I can only assume it's being imported as some other name like
    "bsddb.os" and some element of the import system is actually doing an
    import rather than refusing to guess.

    I presume I need to control the import process more closely to make sure
    that this import attempt is rejected, but I can't see how to do that.

    regards
    Steve
    Steve Holden, Jan 25, 2005
    #3
  4. Steve Holden

    Kartic Guest

    Steve,

    Hmmm... Yes, I see what you are saying. Could you post your create
    table statement so that I can create a db and play around with
    dbimport?

    Thanks,
    --Kartic
    Kartic, Jan 25, 2005
    #4
  5. Steve Holden

    Steve Holden Guest

    Kartic wrote:

    > Steve,
    >
    > Hmmm... Yes, I see what you are saying. Could you post your create
    > table statement so that I can create a db and play around with
    > dbimport?
    >
    > Thanks,
    > --Kartic
    >

    Here it is, plus also the loader program I used to suck in the standard
    library (in case it's that that's faulty):

    CREATE TABLE `module` (
    `modName` varchar(25) NOT NULL default '',
    `modPackage` tinyint(1) NOT NULL default '0',
    `modCompileTime` timestamp(14) NOT NULL,
    `modCode` longtext,
    PRIMARY KEY (`modName`)
    ) TYPE=MyISAM;

    #### WARNING: email client may wrap some lines ...
    #
    # Establish standard library in database
    #
    import db
    import os
    import glob
    import sys
    import marshal

    conn = db.conn()
    curs = conn.cursor()

    if sys.argv[1] == "-d":
    curs.execute("delete from module")
    print "All existing modules cleared"
    del sys.argv[1]

    def importpy(path, modname, package):
    print "Importing", path, modname
    c = compile(file(path).read(), "db:%s" % modname, "exec")
    curs.execute("""delete from module where modName = %s""", (modname,))
    curs.execute("""insert into module (modName, modCode, modPackage,
    modCompileTime)
    values (%s, %s, %s, now())""", (modname,
    marshal.dumps(c), package))
    print "Added", modname
    conn.commit()

    def importall(path, modlist):
    os.chdir(path)
    for f in glob.glob("*"):
    if os.path.isdir(f):
    fn = os.path.join(path, f, "__init__.py")
    if os.path.exists(fn):
    ml = modlist + [f]
    importpy(fn, ".".join(ml), 1)
    importall(os.path.join(path, f), ml)
    elif f.endswith('.py') and '.' not in f[:-3] and f !=
    "__init__.py":
    importpy(os.path.join(path, f), ".".join(modlist+[f[:-3]]), 0)

    if __name__ == "__main__":
    for path in sys.argv[1:]:
    importall(path, [])

    regards
    Steve
    Steve Holden, Jan 25, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Paul Opal
    Replies:
    12
    Views:
    940
    Paul Opal
    Oct 11, 2004
  2. JPractitioner
    Replies:
    13
    Views:
    20,143
    Roedy Green
    Feb 24, 2006
  3. Vitali Gontsharuk
    Replies:
    2
    Views:
    561
    Vitali Gontsharuk
    Aug 25, 2005
  4. Artur M. Piwko

    import/from import question

    Artur M. Piwko, Jun 29, 2003, in forum: Python
    Replies:
    1
    Views:
    834
    Peter Hansen
    Jul 2, 2003
  5. Dennis
    Replies:
    2
    Views:
    499
    Dennis
    Aug 18, 2003
Loading...

Share This Page