Import a module from a non-file?

Discussion in 'Python' started by Petri Savolainen, Oct 17, 2003.

  1. I was trying to roll my own python code importer, but in the end, it seems
    that no matter what you try, it is always necessary to supply a REAL file
    object for python lower-level import machinery to work. Is this really
    true? Or did I miss something?

    Of course, it is easy to get the module source from somewhere, put it into a
    tmpfile etc. ... but I'd be nice to be able to skip that extra step.

    Thanks,

    Petri
    Petri Savolainen, Oct 17, 2003
    #1
    1. Advertising

  2. Petri Savolainen

    Peter Hansen Guest

    Petri Savolainen wrote:
    >
    > I was trying to roll my own python code importer, but in the end, it seems
    > that no matter what you try, it is always necessary to supply a REAL file
    > object for python lower-level import machinery to work. Is this really
    > true? Or did I miss something?
    >
    > Of course, it is easy to get the module source from somewhere, put it into a
    > tmpfile etc. ... but I'd be nice to be able to skip that extra step.


    You don't say what you actually tried. Doesn't imp.load_module() allow
    passing a "file-like object" containing the source?

    What about exec? I would think that if you read in the source code string,
    created a new module, and passed its dictionary in to exec, you would get
    basically the same result.

    (Neither idea is based on personal success doing this... they're just ideas.)

    -Peter
    Peter Hansen, Oct 17, 2003
    #2
    1. Advertising

  3. Petri Savolainen wrote:

    > I was trying to roll my own python code importer, but in the end, it seems
    > that no matter what you try, it is always necessary to supply a REAL file
    > object for python lower-level import machinery to work. Is this really
    > true? Or did I miss something?


    No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
    for details. Basically, you can write and install "Importer" objects,
    which may return "Loader" objects (an importer an also be a loader, if
    you wish, so importer.find_module can do a "return self" as long as the
    importer object also has a load_module method). A loader's method
    load_module must return the fully loaded module object, and it can
    built it in any way it pleases -- e.g. with new.module from standard
    module new to make a new empty module object, then (if for example it
    has obtained the Python source of the module from wherever) an exec
    of that source using the new module's dictionary as locals and globals.

    Let me give a trivial example made by overriding __import__ -- NOT the
    recommended mechanism, just simpler to explain!

    import __builtin__
    bi = __builtin__.__import__

    def __import__(name, *args):
    if name == 'foo': return makefoo()
    return bi(name, *args)
    __builtin__.__import__ = __import__

    import new
    import sys
    def makefoo():
    mod = new.module('foo')
    source = '''
    print "foo is being imported"
    def foo(): return "hi, foo here"
    '''
    exec source in mod.__dict__
    return mod

    import foo
    print foo.foo()
    import foo
    print foo.foo()

    running this emits:

    [alex@lancelot ba]$ python i.py
    foo is being imported
    hi, foo here
    foo is being imported
    hi, foo here
    [alex@lancelot ba]$

    note that each import invokes makefoo AGAIN: if you want to use
    the cache conveniently maintained for you in sys.modules['foo']
    after the first time foo is imported, you have to do that
    explicitly should you choose to use this "override __import__"
    old-but-still-there functionality (caching for the new and
    improved mechanisms of PEP 302 is different -- again, see the
    PEP itself for all details).


    Alex
    Alex Martelli, Oct 17, 2003
    #3
  4. Petri Savolainen

    David Boddie Guest

    Alex Martelli <> wrote in message news:<sAUjb.7504$>...
    > Petri Savolainen wrote:
    >
    > > I was trying to roll my own python code importer, but in the end, it seems
    > > that no matter what you try, it is always necessary to supply a REAL file
    > > object for python lower-level import machinery to work. Is this really
    > > true? Or did I miss something?

    >
    > No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
    > for details.


    That's only for Python 2.3 and later, though, isn't it? I ran into this problem
    using Python 2.2 with the ihooks and imp modules. I don't remember exactly
    which part of the infrastructure was insisting on file objects but the following
    lines from Python/import.c in the Python 2.3 source distribution would appear
    to be relevant:

    /* First check that there's an open file (if we need one) */
    switch (type) {
    case PY_SOURCE:
    case PY_COMPILED:
    if (fp == NULL) {
    PyErr_Format(PyExc_ValueError,
    "file object required for import (type code %d)",
    type);
    return NULL;
    }
    }

    I really should investigate the new import mechanism. In the meantime, it's good
    to know that the alternative import hook solutions still work with Python 2.3.

    David
    David Boddie, Oct 17, 2003
    #4
  5. David Boddie wrote:

    > Alex Martelli <> wrote in message
    > news:<sAUjb.7504$>...
    >> Petri Savolainen wrote:
    >>
    >> > I was trying to roll my own python code importer, but in the end, it
    >> > seems that no matter what you try, it is always necessary to supply a
    >> > REAL file object for python lower-level import machinery to work. Is
    >> > this really true? Or did I miss something?

    >>
    >> No, it's not true. Read PEP 302,
    >> http://www.python.org/peps/pep-0302.html , for details.

    >
    > That's only for Python 2.3 and later, though, isn't it? I ran into this


    Yes, PEP 302 and its early benefits (import-from-zip) were first implemented
    with Python 2.3.

    > problem using Python 2.2 with the ihooks and imp modules. I don't remember


    Admittedly not the cleanest architecture, though quite workable.

    > exactly which part of the infrastructure was insisting on file objects but
    > the following lines from Python/import.c in the Python 2.3 source
    > distribution would appear to be relevant:
    >
    > /* First check that there's an open file (if we need one) */
    > switch (type) {
    > case PY_SOURCE:
    > case PY_COMPILED:
    > if (fp == NULL) {
    > PyErr_Format(PyExc_ValueError,
    > "file object required for import (type code %d)",
    > type);
    > return NULL;
    > }
    > }


    That's the load_module function exposed by module imp -- and in
    any case when your type is IMP_HOOK you don't enter this test (in
    2.3 ff -- no older sources around to check, sorry).


    > I really should investigate the new import mechanism. In the meantime,
    > it's good to know that the alternative import hook solutions still work
    > with Python 2.3.


    You mean overriding the builtin __import__ ? Yes, it's kept around
    for backwards compatibility, of course (also weird cases in which you
    WANT the sys.modules cache to be systematically bypassed, though I
    admit I can't think of a use case for that right now...;-).


    Alex
    Alex Martelli, Oct 18, 2003
    #5
  6. Petri Savolainen wrote:

    > I was trying to roll my own python code importer, but in the end, it seems
    > that no matter what you try, it is always necessary to supply a REAL file
    > object for python lower-level import machinery to work. Is this really
    > true? Or did I miss something?


    Probably, because I have been importing modules from over a network
    socket for a long time now in Pyro. (http://pyro.sourceforge.net)

    The code in Pyro works back to python version 2.0, if I'm not mistaken,
    so you won't need to upgrade to 2.3 and use all new kinds of import hooks...

    Basically what I'm doing is
    * load the module's bytecode over the network
    * create a new module and put it in sys.modules using new.module(...)
    * execute the downloaded code in this new module: exec code in mod.__dict__
    * some extra magic to trap cascaded imports.

    If you need more info, have a look at the _retrieveCode method
    in the protocol.py file of Pyro, or just ask :)

    I'm not sure if my way of doing things is the 'best' way, but hey,
    it works... :)

    --Irmen de Jong
    Irmen de Jong, Oct 18, 2003
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page