Extending the import mechanism - what is recommended?

Discussion in 'Python' started by dbr517@gmail.com, Jan 29, 2008.

  1. Guest

    I need to extend the import mechanism to support another file type.
    I've already written the necessary C library to read the file and
    return a python code object.

    I found one example which just sub-classed imputil.ImportManager like
    this:

    from myLib import pye_code as pye_code
    class MyImporter(imputil.ImportManager):
    def __init__(self):
    imputil.ImportManager.__init__(self)
    self.add_suffix('.pye', self.import_pye)
    self.install()

    def import_pye(self, filepath, fileinfo, filename):
    data = pye_code(filepath)
    return 0, data, {}

    This actually works fine if the module is just a few lines of code,
    but it won't chain to the "built-in" importers; if the module that I'm
    importing does something as simple as 'import re', it fails.

    It may be that my confusion here is because (even after reading the
    code), I'm not clear on the purposes of imputil.ImportManager vs.
    imputil.Importer :-(

    What is the "preferred" way to do this type of extension? One other
    note; at this time, I just need to import individual module files with
    this extension; I don't need to import packages.

    Thanks!

    Dan
    , Jan 29, 2008
    #1
    1. Advertising

  2. Steve Holden Guest

    wrote:
    > I need to extend the import mechanism to support another file type.
    > I've already written the necessary C library to read the file and
    > return a python code object.
    >
    > I found one example which just sub-classed imputil.ImportManager like
    > this:
    >
    > from myLib import pye_code as pye_code
    > class MyImporter(imputil.ImportManager):
    > def __init__(self):
    > imputil.ImportManager.__init__(self)
    > self.add_suffix('.pye', self.import_pye)
    > self.install()
    >
    > def import_pye(self, filepath, fileinfo, filename):
    > data = pye_code(filepath)
    > return 0, data, {}
    >
    > This actually works fine if the module is just a few lines of code,
    > but it won't chain to the "built-in" importers; if the module that I'm
    > importing does something as simple as 'import re', it fails.
    >
    > It may be that my confusion here is because (even after reading the
    > code), I'm not clear on the purposes of imputil.ImportManager vs.
    > imputil.Importer :-(
    >
    > What is the "preferred" way to do this type of extension? One other
    > note; at this time, I just need to import individual module files with
    > this extension; I don't need to import packages.
    >

    Here's an importer I wrote some time ago to bring modules in from a
    relational database. I won't trouble you with the code to compile the
    modules/packages and add them into the database, but perhaps the
    attached code will be enough to show you what you are doing that's not
    working.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/

    #
    # Import modules from a database
    #
    # NOTE: could use sys version info to select appropriate module version
    # - could ask whether to install if absent ... heh, heh :)
    #
    import sys, db, marshal
    VER = sys.hexversion

    debug = 0

    conn = db.conn()
    curs = conn.cursor()

    curs.execute("select modName from module where modPyVersion=%s", (sys.hexversion, ))
    impdict = {}
    for n in [x[0] for x in curs.fetchall()]:
    impdict[n] = 1

    class DBmodule(type(sys)):
    def __repr__(self):
    return "<DBmodule '%s' originally from '%s'>" % (self.__name__, self.__file__)

    class dbimporter(object):

    def __init__(self, item, *args, **kw):
    if item != "*db*":
    raise ImportError
    if debug:
    print "dbimporter: item:", id(self), item, "args:", args, "keywords:", kw
    print "Accepted", item

    def find_module(self, fullname, path=None):
    if debug:
    print "%x: find_module: %s from %s" % (id(self), fullname, path)
    if fullname not in impdict:
    if debug:
    print "Bailed on", fullname
    return None
    else:
    if debug:
    print "found", fullname, "in db"
    return self

    def load_module(self, modname):
    if debug:
    print "%x: load_module: %s" % (id(self), modname)
    if modname in sys.modules:
    return sys.modules[modname]
    curs.execute("select modCode, modPackage, modFilePath from module where modName=%s and modPyVersion=%s", (modname, VER))
    row = curs.fetchone() # should only BE one ...S
    if not row:
    if debug:
    print modname, "not found in db"
    raise ImportError, "DB module %s not found in modules"
    code, package, path = row
    code = marshal.loads(code)
    module = DBmodule(modname)
    sys.modules[modname] = module
    module.__name__ = modname
    module.__file__ = path # was "db:%s" % modname
    module.__loader__ = self
    if package:
    module.__path__ = ["*db*"]
    exec code in module.__dict__
    if debug:
    print modname, "loaded:", module, "pkg:", package

    return module


    def install():
    sys.path_hooks.append(dbimporter)
    sys.path_importer_cache.clear() # probably not necessary
    sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?
    Steve Holden, Jan 29, 2008
    #2
    1. Advertising

  3. Guest

    On Jan 29, 2:36 pm, Steve Holden <> wrote:
    > wrote:
    > > I need to extend the import mechanism to support another file type.
    > > I've already written the necessary C library to read the file and
    > > return a python code object.

    >
    > > I found one example which just sub-classed imputil.ImportManager like
    > > this:

    >
    > > from myLib import pye_code as pye_code
    > > class MyImporter(imputil.ImportManager):
    > > def __init__(self):
    > > imputil.ImportManager.__init__(self)
    > > self.add_suffix('.pye', self.import_pye)
    > > self.install()

    >
    > > def import_pye(self, filepath, fileinfo, filename):
    > > data = pye_code(filepath)
    > > return 0, data, {}

    >
    > > This actually works fine if the module is just a few lines of code,
    > > but it won't chain to the "built-in" importers; if the module that I'm
    > > importing does something as simple as 'import re', it fails.

    >
    > > It may be that my confusion here is because (even after reading the
    > > code), I'm not clear on the purposes of imputil.ImportManager vs.
    > > imputil.Importer :-(

    >
    > > What is the "preferred" way to do this type of extension? One other
    > > note; at this time, I just need to import individual module files with
    > > this extension; I don't need to import packages.

    >
    > Here's an importer I wrote some time ago to bring modules in from a
    > relational database. I won't trouble you with the code to compile the
    > modules/packages and add them into the database, but perhaps the
    > attached code will be enough to show you what you are doing that's not
    > working.
    >
    > regards
    > Steve
    > --
    > Steve Holden +1 571 484 6266 +1 800 494 3119
    > Holden Web LLC http://www.holdenweb.com/
    >
    > [dbimp.py]#
    > # Import modules from a database
    > #
    > # NOTE: could use sys version info to select appropriate module version
    > # - could ask whether to install if absent ... heh, heh :)
    > #
    > import sys, db, marshal
    > VER = sys.hexversion

    ----------------------------------->
    > def install():
    > sys.path_hooks.append(dbimporter)
    > sys.path_importer_cache.clear() # probably not necessary
    > sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?


    Steve -

    Thanks! Got this to work with one interesting problem . . . if I use
    sys.path.insert(0....) and insert my hook at the head of the path,
    then I can't import anything EXCEPT my special modules . . . If I use
    sys.path.append("*pye*") then I'm OK.

    One thing I changed was that my load_module function raises
    ImportError if it fails; my understanding from reading PEP302 is that
    that's what's SUPPOSED to happen . . .

    At any rate, I can now import my custom modules . . . thanks!

    Dan
    , Feb 4, 2008
    #3
  4. Steve Holden Guest

    wrote:
    > On Jan 29, 2:36 pm, Steve Holden <> wrote:
    >> wrote:
    >>> I need to extend the import mechanism to support another file type.
    >>> I've already written the necessary C library to read the file and
    >>> return a python code object.
    >>> I found one example which just sub-classed imputil.ImportManager like
    >>> this:
    >>> from myLib import pye_code as pye_code
    >>> class MyImporter(imputil.ImportManager):
    >>> def __init__(self):
    >>> imputil.ImportManager.__init__(self)
    >>> self.add_suffix('.pye', self.import_pye)
    >>> self.install()
    >>> def import_pye(self, filepath, fileinfo, filename):
    >>> data = pye_code(filepath)
    >>> return 0, data, {}
    >>> This actually works fine if the module is just a few lines of code,
    >>> but it won't chain to the "built-in" importers; if the module that I'm
    >>> importing does something as simple as 'import re', it fails.
    >>> It may be that my confusion here is because (even after reading the
    >>> code), I'm not clear on the purposes of imputil.ImportManager vs.
    >>> imputil.Importer :-(
    >>> What is the "preferred" way to do this type of extension? One other
    >>> note; at this time, I just need to import individual module files with
    >>> this extension; I don't need to import packages.

    >> Here's an importer I wrote some time ago to bring modules in from a
    >> relational database. I won't trouble you with the code to compile the
    >> modules/packages and add them into the database, but perhaps the
    >> attached code will be enough to show you what you are doing that's not
    >> working.
    >>

    [...]
    >> [dbimp.py]#
    >> # Import modules from a database
    >> #
    >> # NOTE: could use sys version info to select appropriate module version
    >> # - could ask whether to install if absent ... heh, heh :)
    >> #
    >> import sys, db, marshal
    >> VER = sys.hexversion

    > ----------------------------------->
    >> def install():
    >> sys.path_hooks.append(dbimporter)
    >> sys.path_importer_cache.clear() # probably not necessary
    >> sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?

    >
    > Steve -
    >
    > Thanks! Got this to work with one interesting problem . . . if I use
    > sys.path.insert(0....) and insert my hook at the head of the path,
    > then I can't import anything EXCEPT my special modules . . . If I use
    > sys.path.append("*pye*") then I'm OK.
    >
    > One thing I changed was that my load_module function raises
    > ImportError if it fails; my understanding from reading PEP302 is that
    > that's what's SUPPOSED to happen . . .
    >
    > At any rate, I can now import my custom modules . . . thanks!
    >

    Excellent news, thanks for letting me know. I can't say that code was
    particularly well-tested, so it's possible that my version makes some
    error that stops the import from running against further path elements
    if the search by the custom importer fails.

    May have a few minutes to look at this later, but not for a while.
    Anyway, glad the code helped. Perhaps someone else with more PEP302-fu
    would be able to point out my error.

    Any other changes you'd like to feed back besides the ImportError?

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
    Steve Holden, Feb 4, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. piyush
    Replies:
    0
    Views:
    1,838
    piyush
    Jul 14, 2004
  2. Muhammad Shuaib Anjum
    Replies:
    2
    Views:
    630
    Muhammad Shuaib Anjum
    Aug 13, 2004
  3. Nicolas Fleury
    Replies:
    4
    Views:
    429
    Greg Ewing
    Mar 21, 2005
  4. ch424
    Replies:
    3
    Views:
    620
    ch424
    Jul 25, 2005
  5. Malcolm Greene
    Replies:
    1
    Views:
    359
    Dan Bishop
    Apr 12, 2008
Loading...

Share This Page