Extending the import mechanism - what is recommended?

[dbr517]

I need to extend the import mechanism to support another file type.
I've already written the necessary C library to read the file and
return a python code object.

I found one example which just sub-classed imputil.ImportManager like
this:

from myLib import pye_code as pye_code
class MyImporter(imputil.ImportManager):
def __init__(self):
imputil.ImportManager.__init__(self)
self.add_suffix('.pye', self.import_pye)
self.install()

def import_pye(self, filepath, fileinfo, filename):
data = pye_code(filepath)
return 0, data, {}

This actually works fine if the module is just a few lines of code,
but it won't chain to the "built-in" importers; if the module that I'm
importing does something as simple as 'import re', it fails.

It may be that my confusion here is because (even after reading the
code), I'm not clear on the purposes of imputil.ImportManager vs.
imputil.Importer :-(

What is the "preferred" way to do this type of extension? One other
note; at this time, I just need to import individual module files with
this extension; I don't need to import packages.

Thanks!

Dan

 

[Steve Holden]

I need to extend the import mechanism to support another file type.
I've already written the necessary C library to read the file and
return a python code object.

I found one example which just sub-classed imputil.ImportManager like
this:

from myLib import pye_code as pye_code
class MyImporter(imputil.ImportManager):
def __init__(self):
imputil.ImportManager.__init__(self)
self.add_suffix('.pye', self.import_pye)
self.install()

def import_pye(self, filepath, fileinfo, filename):
data = pye_code(filepath)
return 0, data, {}

This actually works fine if the module is just a few lines of code,
but it won't chain to the "built-in" importers; if the module that I'm
importing does something as simple as 'import re', it fails.

It may be that my confusion here is because (even after reading the
code), I'm not clear on the purposes of imputil.ImportManager vs.
imputil.Importer :-(

What is the "preferred" way to do this type of extension? One other
note; at this time, I just need to import individual module files with
this extension; I don't need to import packages.

Here's an importer I wrote some time ago to bring modules in from a
relational database. I won't trouble you with the code to compile the
modules/packages and add them into the database, but perhaps the
attached code will be enough to show you what you are doing that's not
working.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

#
# Import modules from a database
#
# NOTE: could use sys version info to select appropriate module version
# - could ask whether to install if absent ... heh, heh
#
import sys, db, marshal
VER = sys.hexversion

debug = 0

conn = db.conn()
curs = conn.cursor()

curs.execute("select modName from module where modPyVersion=%s", (sys.hexversion, ))
impdict = {}
for n in [x[0] for x in curs.fetchall()]:
impdict[n] = 1

class DBmodule(type(sys)):
def __repr__(self):
return "<DBmodule '%s' originally from '%s'>" % (self.__name__, self.__file__)

class dbimporter(object):

def __init__(self, item, *args, **kw):
if item != "*db*":
raise ImportError
if debug:
print "dbimporter: item:", id(self), item, "args:", args, "keywords:", kw
print "Accepted", item

def find_module(self, fullname, path=None):
if debug:
print "%x: find_module: %s from %s" % (id(self), fullname, path)
if fullname not in impdict:
if debug:
print "Bailed on", fullname
return None
else:
if debug:
print "found", fullname, "in db"
return self

def load_module(self, modname):
if debug:
print "%x: load_module: %s" % (id(self), modname)
if modname in sys.modules:
return sys.modules[modname]
curs.execute("select modCode, modPackage, modFilePath from module where modName=%s and modPyVersion=%s", (modname, VER))
row = curs.fetchone() # should only BE one ...S
if not row:
if debug:
print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package, path = row
code = marshal.loads(code)
module = DBmodule(modname)
sys.modules[modname] = module
module.__name__ = modname
module.__file__ = path # was "db:%s" % modname
module.__loader__ = self
if package:
module.__path__ = ["*db*"]
exec code in module.__dict__
if debug:
print modname, "loaded:", module, "pkg:", package

return module


def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?

 

[dbr517]

I need to extend the import mechanism to support another file type.
I've already written the necessary C library to read the file and
return a python code object.

I found one example which just sub-classed imputil.ImportManager like
this:

from myLib import pye_code as pye_code
class MyImporter(imputil.ImportManager):
def __init__(self):
imputil.ImportManager.__init__(self)
self.add_suffix('.pye', self.import_pye)
self.install()

def import_pye(self, filepath, fileinfo, filename):
data = pye_code(filepath)
return 0, data, {}

This actually works fine if the module is just a few lines of code,
but it won't chain to the "built-in" importers; if the module that I'm
importing does something as simple as 'import re', it fails.

It may be that my confusion here is because (even after reading the
code), I'm not clear on the purposes of imputil.ImportManager vs.
imputil.Importer :-(

What is the "preferred" way to do this type of extension? One other
note; at this time, I just need to import individual module files with
this extension; I don't need to import packages.

Here's an importer I wrote some time ago to bring modules in from a
relational database. I won't trouble you with the code to compile the
modules/packages and add them into the database, but perhaps the
attached code will be enough to show you what you are doing that's not
working.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[dbimp.py]#
# Import modules from a database
#
# NOTE: could use sys version info to select appropriate module version
# - could ask whether to install if absent ... heh, heh
#
import sys, db, marshal
VER = sys.hexversion ----------------------------------->
def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?

Steve -

Thanks! Got this to work with one interesting problem . . . if I use
sys.path.insert(0....) and insert my hook at the head of the path,
then I can't import anything EXCEPT my special modules . . . If I use
sys.path.append("*pye*") then I'm OK.

One thing I changed was that my load_module function raises
ImportError if it fails; my understanding from reading PEP302 is that
that's what's SUPPOSED to happen . . .

At any rate, I can now import my custom modules . . . thanks!

Dan

 

[Steve Holden]

I need to extend the import mechanism to support another file type.
I've already written the necessary C library to read the file and
return a python code object.
I found one example which just sub-classed imputil.ImportManager like
this:
from myLib import pye_code as pye_code
class MyImporter(imputil.ImportManager):
def __init__(self):
imputil.ImportManager.__init__(self)
self.add_suffix('.pye', self.import_pye)
self.install()
def import_pye(self, filepath, fileinfo, filename):
data = pye_code(filepath)
return 0, data, {}
This actually works fine if the module is just a few lines of code,
but it won't chain to the "built-in" importers; if the module that I'm
importing does something as simple as 'import re', it fails.
It may be that my confusion here is because (even after reading the
code), I'm not clear on the purposes of imputil.ImportManager vs.
imputil.Importer :-(
What is the "preferred" way to do this type of extension? One other
note; at this time, I just need to import individual module files with
this extension; I don't need to import packages.

Here's an importer I wrote some time ago to bring modules in from a
relational database. I won't trouble you with the code to compile the
modules/packages and add them into the database, but perhaps the
attached code will be enough to show you what you are doing that's not
working.
[...]
[dbimp.py]#
# Import modules from a database
#
# NOTE: could use sys version info to select appropriate module version
# - could ask whether to install if absent ... heh, heh
#
import sys, db, marshal
VER = sys.hexversion ----------------------------------->
def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?

Steve -

Thanks! Got this to work with one interesting problem . . . if I use
sys.path.insert(0....) and insert my hook at the head of the path,
then I can't import anything EXCEPT my special modules . . . If I use
sys.path.append("*pye*") then I'm OK.

One thing I changed was that my load_module function raises
ImportError if it fails; my understanding from reading PEP302 is that
that's what's SUPPOSED to happen . . .

At any rate, I can now import my custom modules . . . thanks!

Excellent news, thanks for letting me know. I can't say that code was
particularly well-tested, so it's possible that my version makes some
error that stops the import from running against further path elements
if the search by the custom importer fails.

May have a few minutes to look at this later, but not for a while.
Anyway, glad the code helped. Perhaps someone else with more PEP302-fu
would be able to point out my error.

Any other changes you'd like to feed back besides the ImportError?

regards
Steve