Distributing applications that use 3rd party modules

Discussion in 'Python' started by eliben, May 17, 2008.

  1. eliben

    eliben Guest

    Hello,

    I'm getting into Python now after years of Perl, and as part of my
    research I must understand how to do some common tasks I need.

    I have a bunch of Windows PCs at work to which I want to distribute an
    application I've developed on my PC. All these PCs have Python 2.5
    installed.

    If my application contains only code I've developed, I simply zip its
    directory with .py files and send it to everyone, who can then use it
    by running the entry-point .py file. However, what if I've installed
    some 3rd party modules on my PC, and my application uses them (for
    example pyparsing, PiYAML and some others) ? I don't want to manually
    install all these packages (there may be dozens of them) on all those
    PCs (there may be dozens of those too). What is the best method I can
    use ? Naturally, I want all the non-standard packages my app uses to
    be detected automatically and collected into some kind of convenient
    distributable that is easy to pass around and run.

    I'm aware of py2exe - tried it and it works fine. But it creates huge
    executables, and I don't want to distribute those all the time. I much
    prefer a zipped directory of .py scripts that takes some 10s of KBs.

    Thanks in advance,
    Eli
    eliben, May 17, 2008
    #1
    1. Advertising

  2. On May 17, 4:42 am, eliben <> wrote:
    > Hello,
    >
    > I'm getting into Python now after years of Perl, and as part of my
    > research I must understand how to do some common tasks I need.
    >
    > I have a bunch of Windows PCs at work to which I want to distribute an
    > application I've developed on my PC. All these PCs have Python 2.5
    > installed.
    >
    > If my application contains only code I've developed, I simply zip its
    > directory with .py files and send it to everyone, who can then use it
    > by running the entry-point .py file. However, what if I've installed
    > some 3rd party modules on my PC, and my application uses them (for
    > example pyparsing, PiYAML and some others) ? I don't want to manually
    > install all these packages (there may be dozens of them) on all those
    > PCs (there may be dozens of those too). What is the best method I can
    > use ? Naturally, I want all the non-standard packages my app uses to
    > be detected automatically and collected into some kind of convenient
    > distributable that is easy to pass around and run.
    >
    > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > executables, and I don't want to distribute those all the time. I much
    > prefer a zipped directory of .py scripts that takes some 10s of KBs.
    >
    > Thanks in advance,
    > Eli


    You might want to check out Buildout. It allows you to run all that
    stuff in it's own virtual world, so to speak. I'm planning on playing
    with it this week.

    Also, there's this site which has collected the various means of
    distributing Python apps:

    http://www.freehackers.org/Packaging_a_python_program

    Hope that helps!

    Mike
    Mike Driscoll, May 17, 2008
    #2
    1. Advertising

  3. eliben

    John Machin Guest

    eliben wrote:
    > Hello,
    >
    > I'm getting into Python now after years of Perl, and as part of my
    > research I must understand how to do some common tasks I need.
    >
    > I have a bunch of Windows PCs at work to which I want to distribute an
    > application I've developed on my PC. All these PCs have Python 2.5
    > installed.
    >
    > If my application contains only code I've developed, I simply zip its
    > directory with .py files and send it to everyone, who can then use it
    > by running the entry-point .py file. However, what if I've installed
    > some 3rd party modules on my PC, and my application uses them (for
    > example pyparsing, PiYAML and some others) ? I don't want to manually
    > install all these packages (there may be dozens of them) on all those
    > PCs (there may be dozens of those too). What is the best method I can
    > use ? Naturally, I want all the non-standard packages my app uses to
    > be detected automatically and collected into some kind of convenient
    > distributable that is easy to pass around and run.
    >
    > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > executables, and I don't want to distribute those all the time. I much
    > prefer a zipped directory of .py scripts that takes some 10s of KBs.


    What do you call "huge"?

    In my environment the end-users don't have Python installed on their own
    PCs, but the team does have a networked shared drive; we put py2exe'd
    applications there. Less complaints about app load time than for things
    like Word/Excel/Outlook, and no complaints from me about distribution,
    and no worries about users running old versions.

    Cheers,
    John
    John Machin, May 17, 2008
    #3
  4. eliben

    eliben Guest

    On May 17, 3:23 pm, Mike Driscoll <> wrote:
    > On May 17, 4:42 am, eliben <> wrote:
    >
    >
    >
    > > Hello,

    >
    > > I'm getting into Python now after years of Perl, and as part of my
    > > research I must understand how to do some common tasks I need.

    >
    > > I have a bunch of Windows PCs at work to which I want to distribute an
    > > application I've developed on my PC. All these PCs have Python 2.5
    > > installed.

    >
    > > If my application contains only code I've developed, I simply zip its
    > > directory with .py files and send it to everyone, who can then use it
    > > by running the entry-point .py file. However, what if I've installed
    > > some 3rd party modules on my PC, and my application uses them (for
    > > example pyparsing, PiYAML and some others) ? I don't want to manually
    > > install all these packages (there may be dozens of them) on all those
    > > PCs (there may be dozens of those too). What is the best method I can
    > > use ? Naturally, I want all the non-standard packages my app uses to
    > > be detected automatically and collected into some kind of convenient
    > > distributable that is easy to pass around and run.

    >
    > > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > > executables, and I don't want to distribute those all the time. I much
    > > prefer a zipped directory of .py scripts that takes some 10s of KBs.

    >
    > > Thanks in advance,
    > > Eli

    >
    > You might want to check out Buildout. It allows you to run all that
    > stuff in it's own virtual world, so to speak. I'm planning on playing
    > with it this week.
    >


    I looked at its examples and I'm not sure it's what I need. It seems
    useful for other cases though.

    > Also, there's this site which has collected the various means of
    > distributing Python apps:
    >
    > http://www.freehackers.org/Packaging_a_python_program
    >


    This page only talks about the packagers that create .exe files that
    don't need Python installed.


    Is there a simple way to find out which packages are used by my
    script ?

    Eli
    eliben, May 17, 2008
    #4
  5. eliben

    Terry Reedy Guest

    "eliben" <> wrote in message
    news:...
    | Is there a simple way to find out which packages are used by my
    | script ?

    Ggrep for import statements?
    (But that only only gives direct dependencies.)
    Does py2exe have an option to output the list it collects?|
    I think people have mentioned such here.
    Terry Reedy, May 17, 2008
    #5
  6. eliben

    eliben Guest

    On May 17, 8:54 pm, "Terry Reedy" <> wrote:
    > "eliben" <> wrote in message
    >
    > news:...
    > | Is there a simple way to find out which packages are used by my
    > | script ?
    >
    > Ggrep for import statements?


    This isn't very comprehensive :)

    > (But that only only gives direct dependencies.)
    > Does py2exe have an option to output the list it collects?|
    > I think people have mentioned such here.


    Digging around the standard library documentation, I came upon the
    'modulefinder' module, which seems to be able to do what I want, at
    least in terms of finding all the modules used. I wonder how difficult
    it should be to wrap something around modulefinder that copies all the
    relevant modules into a local directory and modifies sys.path
    accordingly.

    Eli
    eliben, May 17, 2008
    #6
  7. On May 17, 10:13 am, eliben <> wrote:
    > On May 17, 3:23 pm, Mike Driscoll <> wrote:
    >
    >
    >
    > > On May 17, 4:42 am, eliben <> wrote:

    >
    > > > Hello,

    >
    > > > I'm getting into Python now after years of Perl, and as part of my
    > > > research I must understand how to do some common tasks I need.

    >
    > > > I have a bunch of Windows PCs at work to which I want to distribute an
    > > > application I've developed on my PC. All these PCs have Python 2.5
    > > > installed.

    >
    > > > If my application contains only code I've developed, I simply zip its
    > > > directory with .py files and send it to everyone, who can then use it
    > > > by running the entry-point .py file. However, what if I've installed
    > > > some 3rd party modules on my PC, and my application uses them (for
    > > > example pyparsing, PiYAML and some others) ? I don't want to manually
    > > > install all these packages (there may be dozens of them) on all those
    > > > PCs (there may be dozens of those too). What is the best method I can
    > > > use ? Naturally, I want all the non-standard packages my app uses to
    > > > be detected automatically and collected into some kind of convenient
    > > > distributable that is easy to pass around and run.

    >
    > > > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > > > executables, and I don't want to distribute those all the time. I much
    > > > prefer a zipped directory of .py scripts that takes some 10s of KBs.

    >
    > > > Thanks in advance,
    > > > Eli

    >
    > > You might want to check out Buildout. It allows you to run all that
    > > stuff in it's own virtual world, so to speak. I'm planning on playing
    > > with it this week.

    >
    > I looked at its examples and I'm not sure it's what I need. It seems
    > useful for other cases though.
    >
    > > Also, there's this site which has collected the various means of
    > > distributing Python apps:

    >
    > >http://www.freehackers.org/Packaging_a_python_program

    >
    > This page only talks about the packagers that create .exe files that
    > don't need Python installed.
    >
    > Is there a simple way to find out which packages are used by my
    > script ?
    >
    > Eli


    I'm fairly certain you can use that Freeze application to do the dirty
    work for you. Or you can just distribute the pyc files...I've heard
    that works somehow too.

    Mike
    Mike Driscoll, May 18, 2008
    #7
  8. On May 17, 4:42 am, eliben <> wrote:
    > Hello,
    >
    > I'm getting into Python now after years of Perl, and as part of my
    > research I must understand how to do some common tasks I need.
    >
    > I have a bunch of Windows PCs at work to which I want to distribute an
    > application I've developed on my PC. All these PCs have Python 2.5
    > installed.
    >
    > If my application contains only code I've developed, I simply zip its
    > directory with .py files and send it to everyone, who can then use it
    > by running the entry-point .py file. However, what if I've installed
    > some 3rd party modules on my PC, and my application uses them (for
    > example pyparsing, PiYAML and some others) ? I don't want to manually
    > install all these packages (there may be dozens of them) on all those
    > PCs (there may be dozens of those too). What is the best method I can
    > use ? Naturally, I want all the non-standard packages my app uses to
    > be detected automatically and collected into some kind of convenient
    > distributable that is easy to pass around and run.
    >
    > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > executables, and I don't want to distribute those all the time. I much
    > prefer a zipped directory of .py scripts that takes some 10s of KBs.
    >
    > Thanks in advance,
    > Eli


    One way I forgot to mention is to put Python on the network (i.e. the
    intranet). We do that here at work and I can develop my applications
    on my machine and then put them on there for anyone to use. That way
    they never have to install Python, let alone the bother of installing
    dependencies.

    Mike
    Mike Driscoll, May 20, 2008
    #8
  9. eliben

    castironpi Guest

    On May 20, 7:56 am, Mike Driscoll <> wrote:
    > On May 17, 4:42 am, eliben <> wrote:
    >
    >
    >
    >
    >
    > > Hello,

    >
    > > I'm getting into Python now after years of Perl, and as part of my
    > > research I must understand how to do some common tasks I need.

    >
    > > I have a bunch of Windows PCs at work to which I want to distribute an
    > > application I've developed on my PC. All these PCs have Python 2.5
    > > installed.

    >
    > > If my application contains only code I've developed, I simply zip its
    > > directory with .py files and send it to everyone, who can then use it
    > > by running the entry-point .py file. However, what if I've installed
    > > some 3rd party modules on my PC, and my application uses them (for
    > > example pyparsing, PiYAML and some others) ? I don't want to manually
    > > install all these packages (there may be dozens of them) on all those
    > > PCs (there may be dozens of those too). What is the best method I can
    > > use ? Naturally, I want all the non-standard packages my app uses to
    > > be detected automatically and collected into some kind of convenient
    > > distributable that is easy to pass around and run.

    >
    > > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > > executables, and I don't want to distribute those all the time. I much
    > > prefer a zipped directory of .py scripts that takes some 10s of KBs.

    >
    > > Thanks in advance,
    > > Eli

    >
    > One way I forgot to mention is to put Python on the network (i.e. the
    > intranet). We do that here at work and I can develop my applications
    > on my machine and then put them on there for anyone to use. That way
    > they never have to install Python, let alone the bother of installing
    > dependencies.
    >
    > Mike- Hide quoted text -
    >
    > - Show quoted text -


    Impossible and Useless here:

    I assert there's no way to generate the list.

    a= raw_input( )
    if a== 'something unknown':
    b= 'imp'
    b+= 'ort '
    b+= 'os'
    exec( b )
    castironpi, May 20, 2008
    #9
  10. eliben

    eliben Guest

    On May 17, 11:42 am, eliben <> wrote:
    > Hello,
    >
    > I'm getting into Python now after years of Perl, and as part of my
    > research I must understand how to do some common tasks I need.
    >
    > I have a bunch of Windows PCs at work to which I want to distribute an
    > application I've developed on my PC. All these PCs have Python 2.5
    > installed.
    >
    > If my application contains only code I've developed, I simply zip its
    > directory with .py files and send it to everyone, who can then use it
    > by running the entry-point .py file. However, what if I've installed
    > some 3rd party modules on my PC, and my application uses them (for
    > example pyparsing, PiYAML and some others) ? I don't want to manually
    > install all these packages (there may be dozens of them) on all those
    > PCs (there may be dozens of those too). What is the best method I can
    > use ? Naturally, I want all the non-standard packages my app uses to
    > be detected automatically and collected into some kind of convenient
    > distributable that is easy to pass around and run.
    >
    > I'm aware of py2exe - tried it and it works fine. But it creates huge
    > executables, and I don't want to distribute those all the time. I much
    > prefer a zipped directory of .py scripts that takes some 10s of KBs.
    >
    > Thanks in advance,
    > Eli


    Hello,

    If anyone is interested, I've implemented such a utility - called
    libcollect.py, downloadable at: http://eli.thegreenplace.net/files/prog_code/libcollect.py.txt

    Here's its description from the embedded docstring:

    Motivation:

    Imagine that you've written a script that uses several libraries, some
    of
    which you've written and some you've downloaded and installed (for
    example
    PyYAML). You want to distribute the script to your friends and co-
    workers,
    who already have Python installed with all the standard library. But
    your
    script won't run on their machines, because they have neither your
    personal
    libraries, nor PyYAML installed. So what can you do ?
    * You can ask them to install PyYAML and other libraries your script
    uses,
    and send them your own libraries. This is a lengthy and
    inconvenient
    process.
    * You can use a tool like py2exe to package your delivery. This has a
    downside, however. py2exe produces large files (several MBs) and you
    may
    not want that.
    * You can painstakingly collect the libraries into a directory where
    your
    script can find them, and package the directory together with the
    script.

    LibCollect makes the third option trivial, by doing all the dirty work
    for you.

    Example:

    Suppose your script is named script.py, and is located in directory
    $DIR
    (although I'm using Unix-y notation here, it is for convenience only.
    LibCollect works similarly well on Windows platforms). Follow these
    steps
    to prepare a self-contained distribution with LibCollect:

    Create a distribution setup script in the same directory. Lets assume
    you call it distrib_script.py. You can easily place it in any
    directory
    you like, I'm using the same one to make the example simpler.

    Add the following to distrib_script.py (assuming that libcollect.py is
    in your sys.path):

    **************************************************************

    import libcollect

    # Create a LibCollect object
    lc = libcollect.LibCollect()

    # Prepare arguments for do_collect
    #

    # Path to the script (can be absolute or relative)
    scriptname = 'script.py'

    # Ask the resulting distribution to be placed in
    # directory distrib
    targetdir = 'distrib'

    # Specify which libraries to exclude from the
    # distribution (because you know they're installed
    # on the target machine)
    excludes = ["wx",
    "pywin",
    "win32api",
    "win32com"]

    # Zip the libraries used by the script to reduce
    # clutter and save space
    zip_lib = True

    # This does the actual work
    lc.do_collect( scriptname,
    targetdir,
    excludes,
    zip_lib=zip_lib)

    **************************************************************

    Now run distrib_script.py.
    When it finishes running, you will see that the distrib directory
    has been created in $DIR. In $DIR/distrib you will see two files,
    script.py and distlib.zip

    * script.py is a loader that replaces your original script.py - this
    is the program your users should run. All it does (look at the
    code, it's short!) is prepare the sys.path to include the
    packaged libraries, and runs your own script.py that was also
    packaged into the .zip file
    * distlib.zip is the distribution library, containing all the code
    your script needs to run on any machine with Python installed,
    and nothing else (except the modules you specified in the exclusion
    list). You may choose to pass on the zip file creation and leave
    your distribution library as a directory by providing False
    to the zip_lib argument of LibCollect.do_collect (take a look at
    its documentation, there are some other options there)

    How to use LibCollect:

    * It is most convenient to use LibCollect in the way demonstrated
    in the example above. You may want to update your application from
    time to time, and having a distribution script handy will turn
    the preparation of a new distribution into a 5-second process.
    * If you don't want to create a distribution script, you can use
    a more direct method of invoking libcollect.py as a program on
    your script. Call it without arguments and it will print
    a usage string that will explain what you need to do.

    How it works:

    * LibCollect uses the standard modulefinder module to find out which
    libraries are used by your script. It categorizes them into two
    types: standard libraries that came with Python, and non-standard
    libraries you've installed or written.
    * Only libraries of the second type are included in the distribution
    (bar the libraries you've explicitly asked to exclude).
    * It then builds a directory with all the included libraries, in a
    way that your script will be able to find them. The script itself
    is also packaged into the same place.
    * On request, this directory can be zipped into a single file, to
    employ Python's built-in zip import facility.
    * In the distribution directory, a new file with the name of your
    script is created. It is a simple loader that uses the runpy module
    to transparently load your script from the distribution library.
    This way your script is not being modified (sys.path is rigged
    from the loader).

    Compatibility:
    Python 2.5
    Tested on Windows and Linux, but should work on other platforms
    where the standard Python distribution works.
    eliben, Jun 9, 2008
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matthew Wilson
    Replies:
    3
    Views:
    277
    Michael Hudson
    Dec 4, 2003
  2. Eduardo Bernal
    Replies:
    2
    Views:
    285
    Bob Ippolito
    Mar 5, 2004
  3. nyiann
    Replies:
    1
    Views:
    316
    John Machin
    Mar 16, 2005
  4. Michael Williams
    Replies:
    0
    Views:
    317
    Michael Williams
    Nov 14, 2005
  5. Tina I
    Replies:
    11
    Views:
    405
    Tina I
    May 17, 2007
Loading...

Share This Page