Import a module from a non-file?

[Petri Savolainen]

I was trying to roll my own python code importer, but in the end, it seems
that no matter what you try, it is always necessary to supply a REAL file
object for python lower-level import machinery to work. Is this really
true? Or did I miss something?

Of course, it is easy to get the module source from somewhere, put it into a
tmpfile etc. ... but I'd be nice to be able to skip that extra step.

Thanks,

Petri

 

[Peter Hansen]

I was trying to roll my own python code importer, but in the end, it seems
that no matter what you try, it is always necessary to supply a REAL file
object for python lower-level import machinery to work. Is this really
true? Or did I miss something?

Of course, it is easy to get the module source from somewhere, put it into a
tmpfile etc. ... but I'd be nice to be able to skip that extra step.

You don't say what you actually tried. Doesn't imp.load_module() allow
passing a "file-like object" containing the source?

What about exec? I would think that if you read in the source code string,
created a new module, and passed its dictionary in to exec, you would get
basically the same result.

(Neither idea is based on personal success doing this... they're just ideas.)

-Peter

 

[Alex Martelli]

I was trying to roll my own python code importer, but in the end, it seems
that no matter what you try, it is always necessary to supply a REAL file
object for python lower-level import machinery to work. Is this really
true? Or did I miss something?

No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
for details. Basically, you can write and install "Importer" objects,
which may return "Loader" objects (an importer an also be a loader, if
you wish, so importer.find_module can do a "return self" as long as the
importer object also has a load_module method). A loader's method
load_module must return the fully loaded module object, and it can
built it in any way it pleases -- e.g. with new.module from standard
module new to make a new empty module object, then (if for example it
has obtained the Python source of the module from wherever) an exec
of that source using the new module's dictionary as locals and globals.

Let me give a trivial example made by overriding __import__ -- NOT the
recommended mechanism, just simpler to explain!

import __builtin__
bi = __builtin__.__import__

def __import__(name, *args):
if name == 'foo': return makefoo()
return bi(name, *args)
__builtin__.__import__ = __import__

import new
import sys
def makefoo():
mod = new.module('foo')
source = '''
print "foo is being imported"
def foo(): return "hi, foo here"
'''
exec source in mod.__dict__
return mod

import foo
print foo.foo()
import foo
print foo.foo()

running this emits:

[alex@lancelot ba]$ python i.py
foo is being imported
hi, foo here
foo is being imported
hi, foo here
[alex@lancelot ba]$

note that each import invokes makefoo AGAIN: if you want to use
the cache conveniently maintained for you in sys.modules['foo']
after the first time foo is imported, you have to do that
explicitly should you choose to use this "override __import__"
old-but-still-there functionality (caching for the new and
improved mechanisms of PEP 302 is different -- again, see the
PEP itself for all details).


Alex

 

[David Boddie]

No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
for details.

That's only for Python 2.3 and later, though, isn't it? I ran into this problem
using Python 2.2 with the ihooks and imp modules. I don't remember exactly
which part of the infrastructure was insisting on file objects but the following
lines from Python/import.c in the Python 2.3 source distribution would appear
to be relevant:

/* First check that there's an open file (if we need one) */
switch (type) {
case PY_SOURCE:
case PY_COMPILED:
if (fp == NULL) {
PyErr_Format(PyExc_ValueError,
"file object required for import (type code %d)",
type);
return NULL;
}
}

I really should investigate the new import mechanism. In the meantime, it's good
to know that the alternative import hook solutions still work with Python 2.3.

David

 

[Alex Martelli]

That's only for Python 2.3 and later, though, isn't it? I ran into this

Yes, PEP 302 and its early benefits (import-from-zip) were first implemented
with Python 2.3.

problem using Python 2.2 with the ihooks and imp modules. I don't remember

Admittedly not the cleanest architecture, though quite workable.

exactly which part of the infrastructure was insisting on file objects but
the following lines from Python/import.c in the Python 2.3 source
distribution would appear to be relevant:

/* First check that there's an open file (if we need one) */
switch (type) {
case PY_SOURCE:
case PY_COMPILED:
if (fp == NULL) {
PyErr_Format(PyExc_ValueError,
"file object required for import (type code %d)",
type);
return NULL;
}
}

That's the load_module function exposed by module imp -- and in
any case when your type is IMP_HOOK you don't enter this test (in
2.3 ff -- no older sources around to check, sorry).

I really should investigate the new import mechanism. In the meantime,
it's good to know that the alternative import hook solutions still work
with Python 2.3.

You mean overriding the builtin __import__ ? Yes, it's kept around
for backwards compatibility, of course (also weird cases in which you
WANT the sys.modules cache to be systematically bypassed, though I
admit I can't think of a use case for that right now...;-).


Alex

 

[Irmen de Jong]

I was trying to roll my own python code importer, but in the end, it seems
that no matter what you try, it is always necessary to supply a REAL file
object for python lower-level import machinery to work. Is this really
true? Or did I miss something?

Probably, because I have been importing modules from over a network
socket for a long time now in Pyro. (http://pyro.sourceforge.net)

The code in Pyro works back to python version 2.0, if I'm not mistaken,
so you won't need to upgrade to 2.3 and use all new kinds of import hooks...

Basically what I'm doing is
* load the module's bytecode over the network
* create a new module and put it in sys.modules using new.module(...)
* execute the downloaded code in this new module: exec code in mod.__dict__
* some extra magic to trap cascaded imports.

If you need more info, have a look at the _retrieveCode method
in the protocol.py file of Pyro, or just ask

I'm not sure if my way of doing things is the 'best' way, but hey,
it works...

--Irmen de Jong