changing sys.path

A

Andrea Crotti

So suppose I want to modify the sys.path on the fly before running some code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.

If I take these paths and add them to site-packages/my_paths.pth
everything works, but at run-time the paths which I actually see before
importing are exactly the same.

So there is something I guess that depends on the order, but what can I
reset/reload to make these paths available (I thought I didn't need
anything in theory)?
 
J

jmfauth

So suppose I want to modify the sys.path on the fly before running some code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.

If I take these paths and add them to site-packages/my_paths.pth
everything works, but at run-time the paths which I actually see before
importing are exactly the same.

So there is something I guess that depends on the order, but what can I
reset/reload to make these paths available (I thought I didn't need
anything in theory)?
Click to expand...
Traceback (most recent call last):
fct hello in mod.py


sys.path? Probably, the most genious Python idea.

jmf
 
R

Rick Johnson

So suppose I want to modify the sys.path on the fly before running some code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.

If I take these paths and add them to site-packages/my_paths.pth
everything works, but at run-time the paths which I actually see before
importing are exactly the same.
Click to expand...

1. Is paths_to_add a nested list?
2. Have you tried inspecting the contents of sys.path AFTER calling
extend method?

Consider:
py> sys.path.__len__()
14
py> sys.path.extend([[1,2,3]])
py> sys.path.__len__()
15
 
J

John Nagle

So suppose I want to modify the sys.path on the fly before running some
code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.
Click to expand...

Do

import sys

first.

John Nagle
 
A

Andrea Crotti

I think I finally located the issue with the sys.path extension.

The problem is that I have many namespace directories, for example

lib:
- sub1
- sub2

lib:
- sub3
- sub4

But to have everything working I had lib.sub3 in easy-install.pth.
Now if I try to add something else to the path it doesn't take care of
the namespace
declaration
(every __init__.py in the packages contains:
__import__('pkg_resources').declare_namespace(__name__))
and just doesn't find the other submodules..

If I try to add manually lib.sub1, lib.sub2 changing the sys.path the
imports will only work for the first one.

Strangely if I just create a dev_main.pth in site-packages containing
the same paths, everything works perfectly.

Any suggestions now that the problem is more clear?
Thanks,
Andrea
 
D

Dave Angel

I think I finally located the issue with the sys.path extension.

The problem is that I have many namespace directories, for example

lib:
- sub1
- sub2

lib:
- sub3
- sub4

But to have everything working I had lib.sub3 in easy-install.pth.
Now if I try to add something else to the path it doesn't take care of
the namespace
declaration
(every __init__.py in the packages contains:
__import__('pkg_resources').declare_namespace(__name__))
and just doesn't find the other submodules..

If I try to add manually lib.sub1, lib.sub2 changing the sys.path the
imports will only work for the first one.

Strangely if I just create a dev_main.pth in site-packages containing
the same paths, everything works perfectly.

Any suggestions now that the problem is more clear?
Thanks,
Andrea
Click to expand...

The only code I saw in this thread was:

sys.path.extend(paths_to_add)

Can you add a print of paths_to_add, and of sys.path after you execute it?

If there's only one path, are you putting it in a list anyway? If not
then it won't do what you expect.
 
A

Andrea Crotti

Ok now it's getting really confusing, I tried a small example to see
what is the real behaviour,
so I created some package namespaces (where the __init__.py declare the
namespace package).

/home/andrea/test_ns:
total used in directory 12 available 5655372
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
-rw-r--r-- 1 andrea andrea 125 Feb 10 14:46 test.py

/home/andrea/test_ns/a.b:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a/b:
total 12
-rw-r--r-- 1 andrea andrea 25 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a/c:
total 12
-rw-r--r-- 1 andrea andrea 20 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py


So this test.py works perfectly:
import sys
sys.path.insert(0, 'a.c')
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

from a.c import api as api_ac

While just mixing the order:
import sys
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

sys.path.insert(0, 'a.c')
from a.c import api as api_ac

Doesn't work anymore

[andrea@precision test_ns]$ python2 test.py
Traceback (most recent call last):
File "test.py", line 7, in <module>
from a.c import api as api_ac
ImportError: No module named c



Am I missing something/doing something stupid?
 
D

Dave Angel

Ok now it's getting really confusing, I tried a small example to see
what is the real behaviour,
so I created some package namespaces (where the __init__.py declare the
namespace package).

/home/andrea/test_ns:
total used in directory 12 available 5655372
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
-rw-r--r-- 1 andrea andrea 125 Feb 10 14:46 test.py

/home/andrea/test_ns/a.b:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a/b:
total 12
-rw-r--r-- 1 andrea andrea 25 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a/c:
total 12
-rw-r--r-- 1 andrea andrea 20 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py


So this test.py works perfectly:
import sys
sys.path.insert(0, 'a.c')
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

from a.c import api as api_ac

While just mixing the order:
import sys
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

sys.path.insert(0, 'a.c')
from a.c import api as api_ac

Doesn't work anymore

[andrea@precision test_ns]$ python2 test.py
Traceback (most recent call last):
File "test.py", line 7, in <module>
from a.c import api as api_ac
ImportError: No module named c



Am I missing something/doing something stupid?
Click to expand...

Yes, you've got periods in your directory names. A period means
something special within python, and specifically within the import.

When you say from a.c import api

You're telling it: from package a get module c, and from there
impoort the symbol api

But package a has no module c, so it complains.


In an earlier message you asserted you were using all absolute paths in
your additions to sys.path. Here you're inserting relative ones. How come?
 
A

Andrea Crotti

Yes, you've got periods in your directory names. A period means
something special within python, and specifically within the import.

When you say from a.c import api

You're telling it: from package a get module c, and from there
impoort the symbol api

But package a has no module c, so it complains.


In an earlier message you asserted you were using all absolute paths
in your additions to sys.path. Here you're inserting relative ones.
How come?
Click to expand...

Well yes I have periods, but that's also the real-world situation.
We have many directories that are contributing to the same namespace in
the same superdirectory
which should not interfere, so it was decided to give this naming.
It would be quite hard to change I guess and I have to prove that this
is problem.

I renamed everything and this
import sys
from os import path
sys.path.insert(0, path.abspath('ab'))

from a.b import api as api_ab

sys.path.insert(0, path.abspath('ac'))
from a.c import api as api_ac

still fails, so the period in the name is not the problem.

Also absolute or relative paths in this small example doesn't make any
difference.

Adding all the paths in one go works perfectly fine anyway, so I
probably have to make sure
I add them *all* before anything is imported.

If there are better solutions I would like to hear them :)
 
P

Peter Otten

Andrea said:
Ok now it's getting really confusing, I tried a small example to see
what is the real behaviour,
so I created some package namespaces (where the __init__.py declare the
namespace package).

/home/andrea/test_ns:
total used in directory 12 available 5655372
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
-rw-r--r-- 1 andrea andrea 125 Feb 10 14:46 test.py

/home/andrea/test_ns/a.b:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a/b:
total 12
-rw-r--r-- 1 andrea andrea 25 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a/c:
total 12
-rw-r--r-- 1 andrea andrea 20 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py


So this test.py works perfectly:
import sys
sys.path.insert(0, 'a.c')
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

from a.c import api as api_ac

While just mixing the order:
import sys
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

sys.path.insert(0, 'a.c')
from a.c import api as api_ac

Doesn't work anymore

[andrea@precision test_ns]$ python2 test.py
Traceback (most recent call last):
File "test.py", line 7, in <module>
from a.c import api as api_ac
ImportError: No module named c



Am I missing something/doing something stupid?
Click to expand...

The package a will be either a.c/a/ or a.b/a/ depending on whether a.c/ or
a.b/ appears first in sys.path.

If it's a.c/a, that does not contain a c submodule or subpackage.
 
A

Andrea Crotti

The package a will be either a.c/a/ or a.b/a/ depending on whether
a.c/ or a.b/ appears first in sys.path. If it's a.c/a, that does not
contain a c submodule or subpackage.
Click to expand...


I would agree if I didn't have this declaration
__import__('pkg_resources').declare_namespace(__name__)
in each subdirectory.

And how do you explain the fact that changing the order everything works?
Namespace packages are supposed to work exactly like this, if it doesn't
resolve the
"c" instead of raising an Exception it goes forward in the sys.path and
try again, which
is what actually happens when I do this

sys.path.append(path.abspath('ab'))
sys.path.append(path.abspath('ac'))

from a.b import api as api_ab
from a.c import api as api_ac

Maybe this:
Definition: pkgutil.extend_path(path, name)
Docstring:
Extend a package's path.

Intended use is to place the following code in a package's __init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)


might come handy, from what I'm gathering is the only way to have a more
dynamic path
manipulation with namespace packages..
 
P

Peter Otten

Andrea said:
I would agree if I didn't have this declaration
__import__('pkg_resources').declare_namespace(__name__)
in each subdirectory.
Click to expand...

Sorry, you didn't mention that in the post I responded to and I didn't
follow the thread closely.

I found a description for declare_namespace() at
http://peak.telecommunity.com/DevCenter/PkgResources

but the text explaining the function is completely unintelligible to me, so
I cannot contribute anything helpful here :(
 
A

Andrea Crotti

Sorry, you didn't mention that in the post I responded to and I didn't
follow the thread closely.

I found a description for declare_namespace() at
http://peak.telecommunity.com/DevCenter/PkgResources

but the text explaining the function is completely unintelligible to me, so
I cannot contribute anything helpful here :(
Click to expand...

Well in the end I submitted a bug report
http://bugs.python.org/issue13991
I'm not sure it's really a bug and maybe I'm just doing something wrong,
but to me the behavior is at least unexpected..
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Manipulating sys.path 0
sys.path 3
Appending to sys.path during module install with distutils 0
adding a directory to sys.path 5
sys.path in python3.3 3
System default sys.path 0
FW: python import sys.path 2
sys.path and importing modules from other directories 0

Members online

No members online now.
Total: 26 (members: 2, guests: 24)
Robots: 448

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top