Dynamic and lazy import

[Alexandre Badez]

Hye everyone,

I'm would like to do something a bit tricky.

I would like when I do something like to create a __init__ package's
(here calle my_package) file witch make an action when we try to
import something in this package...

Quiet like __getattribute__ work for a class, I would like this kind
of behaviour for a whole module or a package

be a bit clearer:
I've got a normal package/module/class (whathever) where I do
something like:

from my_package import my_module

(but in the my_package I've got only:
my_package / __init__.py

And in the __init__ of my my_package something like __getattribute__
(but for a module) that would return something for my_module

I know, that I can do something like charging everything "before" in
__init__ like:
my_module = the_thing_I_want

But I need it to be lazy (because I may have many module witch could
be very awful...).

Any idea ?
Is it possible ?

 

[Gabriel Genellina]

En Tue, 16 Oct 2007 10:16:50 -0300, Alexandre Badez

I would like when I do something like to create a __init__ package's
(here calle my_package) file witch make an action when we try to
import something in this package...

Quiet like __getattribute__ work for a class, I would like this kind
of behaviour for a whole module or a package

be a bit clearer:
I've got a normal package/module/class (whathever) where I do
something like:

from my_package import my_module

(but in the my_package I've got only:
my_package / __init__.py

And in the __init__ of my my_package something like __getattribute__
(but for a module) that would return something for my_module

I know, that I can do something like charging everything "before" in
__init__ like:
my_module = the_thing_I_want

But I need it to be lazy (because I may have many module witch could
be very awful...).

A module doesn't *have* to be a module... You can replace
sys.modules['your_module_name'] with a suitable object. See this classical
recipe from Alex Martelli:
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65207>

 

[Stargaming]

Hye everyone,

I'm would like to do something a bit tricky.

I would like when I do something like to create a __init__ package's
(here calle my_package) file witch make an action when we try to import
something in this package...

Quiet like __getattribute__ work for a class, I would like this kind of
behaviour for a whole module or a package

be a bit clearer:
I've got a normal package/module/class (whathever) where I do something
like:

from my_package import my_module

(but in the my_package I've got only: my_package / __init__.py

And in the __init__ of my my_package something like __getattribute__
(but for a module) that would return something for my_module

I know, that I can do something like charging everything "before" in
__init__ like:
my_module = the_thing_I_want

But I need it to be lazy (because I may have many module witch could be
very awful...).

Any idea ?
Is it possible ?

Depending on the exact complexity of your needs and exact use case just
inserting some code on top of every module could suffice. To reduce
boilerplate, if the task you're trying to realize is pretty generic for
every module, you could insert another module hiding this code and used
like this::

from .modregister import init
init(__name__)

If you really have to _create_ modules dynamically, follow the advice
given earlier in this thread.

HTH,
Stargaming

 

[Alexandre Badez]

Thanks for all your advices, but it's not really what I would like to
do.

I'm going to be more clearer for what I really want to do.

Here we have got many library for different applications. All those
library have a version and between a version and an other, there isn't
always a very good backward compatibility (I know, it's very ugly, but
it's like that...).

Moreover some library use the version 1.1 and some the version 1.2 of
lib A and version 2.1 and version 2.0 of lib B ... you know what: it's
very ugly.

My idea was to be able to use lib quiet like that.

import A (<- If I want to use the very last version)
# or
import A.1_1 (<- If I want to use the version 1.1 of the A lib)

Something else ?
Yes
I do not want to add all those path in PYTHONPATH (would be too ugly,
and "complicated").
I want it lazy (do not import every version of every lib every time)
I want it scalable: if a user or the admin add a new lib or a version
of lib it would be very very great if he had nothing else (than copy
his directory) to do.

So what I wanted to do, was to be able to control what the user really
wanted to import, and act he excepted and put the "intelligence" in a
__init__ script

 

[Diez B. Roggisch]

Thanks for all your advices, but it's not really what I would like to
do.

I'm going to be more clearer for what I really want to do.

Here we have got many library for different applications. All those
library have a version and between a version and an other, there isn't
always a very good backward compatibility (I know, it's very ugly, but
it's like that...).

Moreover some library use the version 1.1 and some the version 1.2 of
lib A and version 2.1 and version 2.0 of lib B ... you know what: it's
very ugly.

My idea was to be able to use lib quiet like that.

import A (<- If I want to use the very last version)
# or
import A.1_1 (<- If I want to use the version 1.1 of the A lib)

Something else ?
Yes
I do not want to add all those path in PYTHONPATH (would be too ugly,
and "complicated").
I want it lazy (do not import every version of every lib every time)
I want it scalable: if a user or the admin add a new lib or a version
of lib it would be very very great if he had nothing else (than copy
his directory) to do.

So what I wanted to do, was to be able to control what the user really
wanted to import, and act he excepted and put the "intelligence" in a
__init__ script

Use setuptools + pkg_resources to install sereval versions of your libraries
together and then you can require a certain version of your lib.

HOWEVER: this WON'T work for several versions of a library in ONE running
python process!!!! Because the import will then either fail silently (after
all, "import foo" is ignored if foo is already present) or pkg_resources is
so clever that it keeps tracks of requirements and if they are conflicting
will puke on you.

Diez

 

[Alexandre Badez]

Use setuptools + pkg_resources to install sereval versions of your libraries
together and then you can require a certain version of your lib.

HOWEVER: this WON'T work for several versions of a library in ONE running
python process!!!! Because the import will then either fail silently (after
all, "import foo" is ignored if foo is already present) or pkg_resources is
so clever that it keeps tracks of requirements and if they are conflicting
will puke on you.

Diez

Well, I would like to be able to use "setuptools", but the problem is
that I can't.
Cause the administrator do not want us to be able to add lib in python
dir.
So we have to create our own library directory...

Moreover, I haven't seen in distutils how it manage different version
of the same library; as far as I know, It just replace the old one by
the newest one... and that's not really what I want.

 

[Diez B. Roggisch]

Diez

Well, I would like to be able to use "setuptools", but the problem is
that I can't.
Cause the administrator do not want us to be able to add lib in python
dir.
So we have to create our own library directory...

That doesn't matter, setuptools is capable of installing anywhere - you just
have to setup _one_ PYTHONPATH (e.g. ~/.python24_packages) and then specify
that when installing the eggs.

Moreover, I haven't seen in distutils how it manage different version
of the same library; as far as I know, It just replace the old one by
the newest one... and that's not really what I want.

Well, you might consider my statement in the last post as hint that you
didn't look properly. It CAN and will install sereval versions in parallel,
and allow for previous selection. GIYF.

Diez