organizing your scripts, with plenty of re-use

B

Buck

I'd like to get to zero-installation if possible. It's easy with
simple python scripts, why not packages too? I know the technical
reasons, but I haven't heard any practical reasons.
If the reasons are purely technical, it smells like a PEP to me.

That's what I meant to say. It IS a zero-installation schema, and it also  
works if you properly install the package. Quoting Steven D'Aprano  
(changing names slightly):

"""You would benefit greatly from separating the interface from
the backend. You should arrange matters so that the users see something
like this:

project/
+-- animal
+-- mammal
+-- reptile
+-- somepackagename/
     +-- __init__.py
     +-- animals.py
     +-- mammals/
         +-- __init__.py
         +-- horse.py
         +-- otter.py
     +-- reptiles/
         +-- __init__.py
         +-- gator.py
         +-- newt.py
     +-- misc/
         +-- __init__.py
         +-- lungs.py
         +-- swimming.py

where the front end is made up of three scripts "animal", "mammal" and
"reptile", and the entire backend is in a package.""" [ignore the rest]

By example, the `animal` script would contain:

 from somepackagename import animals
animals.main()

or perhaps something more elaborate, but in any case, the script imports  
whatever it needs from the `somepackagename` package.

The above script can be run:

a) directly from the `project` directory; this could be a checked out copy  
 from svn, or a tar file extracted in /tmp, or whatever. No need to install  
anything, it just works.

b) alternatively, you may install somepackagename into site-packages (or  
the user site directory, or any other location along the Python path), and  
copy the scripts into /usr/bin (or any other location along the system  
PATH), and it still works.

The key is to put all the core functionality into a package, and place the  
package where Python can find it. Also, it's a good idea to use relative  
imports from inside the package. There is no need to juggle with sys.path  
nor even set PYTHONPATH nor import __main__ nor play any strange games; it  
Just Works (tm).

Hi Gabriel. This is very thoughtful. Thanks.

As in the OP, when I have 50 different runnable scripts, it become
necessary to arrange them in directories. How would you do that in
your scheme? Currently it looks like they're required to live directly
above the package containing their code.

--Buck
 
E

Ethan Furman

Buck wrote:

[snip]
Steven had the nicest workaround (with the location = __import__
('__main__').__file__ trick), but none of them solve the problem of
the OP: organization of runnable scripts. So far it's been required to
place all runnable scripts directly above any used packages. The
workaround that Gabriel has been touting requires this too.

[snip]

Wha? "Place all runnable scripts directly above any used packages?" I
must have missed something major in this thread. The only thing
necessary is to have the package being imported to be somewhere in
PYTHONPATH.

Previous post:
>I'd like to get to zero-installation if possible. It's easy with
>simple python scripts, why not packages too? I know the technical
>reasons, but I haven't heard any practical reasons.

I don't think we mean the same thing by "zero-installation"... seems to
me that if you have to copy it, check it out, or anything to get the
code from point A to point 'usable on your computer', then you have done
some sort of installation.

~Ethan~
 
G

Gabriel Genellina

Quoting Steven D'Aprano  
(changing names slightly):

"""You would benefit greatly from separating the interface from
the backend. You should arrange matters so that the users see something
like this:

project/
+-- animal
+-- mammal
+-- reptile
+-- somepackagename/
     +-- __init__.py
     +-- animals.py
     +-- mammals/
         +-- __init__.py
         +-- horse.py
         +-- otter.py
     +-- reptiles/
         +-- __init__.py
         +-- gator.py
         +-- newt.py
     +-- misc/
         +-- __init__.py
         +-- lungs.py
         +-- swimming.py

where the front end is made up of three scripts "animal", "mammal" and
"reptile", and the entire backend is in a package.""" [ignore the rest]

By example, the `animal` script would contain:

 from somepackagename import animals
animals.main()

or perhaps something more elaborate, but in any case, the script
imports  
whatever it needs from the `somepackagename` package.

The above script can be run:

a) directly from the `project` directory; this could be a checked out
copy  
 from svn, or a tar file extracted in /tmp, or whatever. No need to
install  
anything, it just works.

b) alternatively, you may install somepackagename into site-packages
(or  
the user site directory, or any other location along the Python path),
and  
copy the scripts into /usr/bin (or any other location along the system  
PATH), and it still works.

The key is to put all the core functionality into a package, and place
the  
package where Python can find it. Also, it's a good idea to use
relative  
imports from inside the package. There is no need to juggle with
sys.path  
nor even set PYTHONPATH nor import __main__ nor play any strange games;
it  
Just Works (tm).

As in the OP, when I have 50 different runnable scripts, it become
necessary to arrange them in directories. How would you do that in
your scheme? Currently it looks like they're required to live directly
above the package containing their code.

I'd say that, if your application contains 50 different runnable scripts
in several directories, it deserves an install script. Being able to
execute something from a checked-out copy is fine for a small application
but in your case I'd say we are off limits...

Python must be able to import the package. In case a) above, this happens
because the directory containing the script (and the package) is in
sys.path by default. In case b), because you copy the package into some
place that is already in sys.path. There is another option: add the
directory containing the package to sys.path, by setting the environment
variable PYTHONPATH. Robert Kern already explained how to do that:
<http://permalink.gmane.org/gmane.comp.python.general/639964>
 
S

Stef Mientki

[snip]
please don't get angry,
I'm not a programmer, I'm just a human ;-)

Hierarchical choices are done on todays knowledge, tomorrow we might
have different views and want/need to arrange things in another way.
An otter may become a reptile ;-)
So from the human viewpoint the following should be possible (and is
for example possible in Delphi)
- I can move the complete project anywhere I like and it should still
work without any modifications (when I move my desk I can still do my
work)

Move a complete package anywhere along the PYTHONPATH and it will
still work. Check.
- I can move any file in he project to any other place in the project
and again everything should work without any modifications ( when I
rearrange my books, I can still find a specific book)

Move any file in any directory to any other spot in that same
directory and it will still work. Check. ;-) sub ;-)

Humans are a lot smarter than computers. Even 'just humans'. ;-) If
you move your book, then can't find it on the last shelf it used to be
on, you look on other shelves, you look on your desk, you look on the
coffe table, you look in your car, etc, etc, and so forth. If you
move a file in a package to somewhere else, and you don't tell the
package where it's at, it's not going to start looking all over the
hard-drive for it.
I didn't say "whole the world";-)
If that were the case you would have to be extra careful to have
every module's name be distinct, and then what's the point of having
packages?
Yes, I still wonder !

cheers,
Stef
 
S

Stef Mientki

Stephen said:
On Mon, Oct 12, 2009 at 4:15 PM, Stef Mientki <[email protected]

Hierarchical choices are done on todays knowledge, tomorrow we
might have different views and want/need to arrange things in
another way.
An otter may become a reptile ;-)
So from the human viewpoint the following should be possible (and
is for example possible in Delphi)
- I can move the complete project anywhere I like and it should
still work without any modifications (when I move my desk I can
still do my work)


This is readily doable in Python; in fact it should just work. You
only need to ensure where you move it ends up in the PYTHONPATH-- but
that'll take at most just one change, once. You can use batch scripts
to alter it if you really need to, but that's all you would need to do.


- I can move any file in he project to any other place in the
project and again everything should work without any modifications
( when I rearrange my books, I can still find a specific book)

In my humble opinion if these actions are not possible, there must
be redundant information in the collection. The only valid reason
for redundant information is to perform self healing (or call it
error correction), and here we have a catch-22.


This is the problem; this is just incorrect thinking and if one's
programming in Python needs to be excised. Things don't work like
that, and trying to make it work like that will result in you going
against the grain of /how/ Python works, and life will become difficult.
I agree it's difficult at the start, but after that,
you can create a new script / module everywhere you like ( within the
projects), and every module is available,
and you can use (almost) any file as a self running program or as a
library module.
Packages are meaningful in Python. They aren't just directories which
you're organizing for your human-programmer's convenience and
understanding. They are a fundamental structure of objects and must be
treated as such.
Well I missed that, I can't see the "higher" meaning of packages,
so if someone has a good link that explains the "higher" meaning of
packages,
I'ld be much obliged.
Python doesn't think in terms of files. You can't just move files
around within a project because that's a semantically significant change.

It's just like you can't move a method on a class to another class,
and expect everything to just work.
No, that's like tearing a page from a book and gluing it in another book
and the story get's
So the difference in my reasoning might be a mis-concept of the largest
atomic part, which in my view is a class / function.
A package is just like a class, its a significant, structural
component in the code itself. It contains files, but those files
aren't what's significant to Python-- what's significant is that they
are modules, which are also just like a class. They're objects which
contain other objects.

Every package creates a namespace of objects it contains; those
objects are modules (and other things that may be defined in
__init__.py). Every module creates a namespace of objects it contains.
Classes create a namespace of objects they contain.

These are all objects. A package isn't a directory you're organizing
-- its an object which contains other objects.
Well I'm completely lost now, but that's probably because I'm not a
programmer,
and I don't know the exact meaning of these words.
So I don't see a class as an object, just as a definition of
"functionality".
A module, a file, a package, a directory (with it's subdirectories) is
just a collector for classes.
I can freely move a class around within any of these containers without
affecting the functionality of that class or anything that really uses
that class.
If you try to change this so that there's no object organization of an
object hierarchy, then you're going to create significant issues for
maintenance and have to jump through all kinds of hoops to try to
accomplish it. It sounds like you want a system where each file is a
unique module, and its particular location isn't important-- the
filename creates a unique identity, a top-level namespace which can be
accessed from anywhere.
yes that's about the view I've now.
Python's object model just doesn't work like that. There's nothing
wrong with how Python works in this regard, its simply different, and
if you try to program Delphi in Python, you'll run into lots of
problems. Just like if you try to program Java or C in Python. They're
different languages with different object models, and you have to
adapt to the object model of the language if you want to use the
language effectively.
I would love to hear one (and preferable more ;-) feature that a package
adds to a collection of classes.

cheers,
Stef
 
B

Buck

I don't think we mean the same thing by "zero-installation"... seems to
me that if you have to copy it, check it out, or anything to get the
code from point A to point 'usable on your computer', then you have done
some sort of installation.

I think most people would agree that installation is whatever you need
to do between downloading the software and being able to use it. For
GNU packages, it's './configure && make && make install'. For Python
packages, it's usually './setup.py install'.
Wha?  "Place all runnable scripts directly above any used packages?"  I
must have missed something major in this thread.  The only thing
necessary is to have the package being imported to be somewhere in
PYTHONPATH.

The only way to get your packages on the PYTHONPATH currently is to:
* install the packages to site-packages (I don't have access)
* edit the PYTHONPATH all users' environment (again, no access)
* create some boilerplate that edits sys.path at runtime (various
problems in previous post)
* put your scripts directly above the package (this seems best so
far, but forces a flat hierarchy of scripts)
 
G

Gabriel Genellina

The only way to get your packages on the PYTHONPATH currently is to:
* install the packages to site-packages (I don't have access)
* edit the PYTHONPATH all users' environment (again, no access)
* create some boilerplate that edits sys.path at runtime (various
problems in previous post)
* put your scripts directly above the package (this seems best so
far, but forces a flat hierarchy of scripts)

Not exactly - in short, you have to place the package under some directory
that is eventually listed in sys.path.
By default, one of such directories is site-packages, but there are also
per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages
(see PEP 370 [1]). If you don't want to copy the package there, you can
even add a .pth file and it will be processed.

[1] http://www.python.org/dev/peps/pep-0370/
 
B

bukzor

The only way to get your packages on the PYTHONPATH currently is to:
   * install the packages to site-packages  (I don't have access)
   * edit the PYTHONPATH all users' environment  (again, no access)
   * create some boilerplate that edits sys.path at runtime (various
problems in previous post)
   * put your scripts directly above the package (this seems best so
far, but forces a flat hierarchy of scripts)

Not exactly - in short, you have to place the package under some directory  
that is eventually listed in sys.path.
By default, one of such directories is site-packages, but there are also  
per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages  
(see PEP 370 [1]). If you don't want to copy the package there, you can  
even add a .pth file and it will be processed.

The .pth files are intriguing. Is this the best reference?
http://docs.python.org/library/site.html

My current solution very closely resembles the .pth system, but uses
the scripts' directory and accepts relative paths.
If I'm reading the doc correctly, the .pth system currently doesn't
support either of these. Does anyone know of the rationale for this?
 
B

bukzor

En Tue, 13 Oct 2009 17:38:44 -0300, Buck <[email protected]> escribió:
Not exactly - in short, you have to place the package under some directory  
that is eventually listed in sys.path.
By default, one of such directories is site-packages, but there are also  
per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages  
(see PEP 370 [1]). If you don't want to copy the package there, you can  
even add a .pth file and it will be processed.

The .pth files are intriguing. Is this the best reference?http://docs.python.org/library/site.html

My current solution very closely resembles the .pth system, but uses
the scripts' directory and accepts relative paths.
If I'm reading the doc correctly, the .pth system currently doesn't
support either of these. Does anyone know of the rationale for this?

I thought this was one of the least objectionable of my posts.

Does anyone know how to find a discussion of the .pth implementation?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top