Squisher -- a lightweight, self-contained alternative to eggs?

Discussion in 'Python' started by Adam Atlas, Mar 5, 2007.

  1. Adam Atlas

    Adam Atlas Guest

    I wrote this little program called Squisher that takes a ZIP file
    containing Python modules and generates a totally self-contained .pyc
    file that imports a specified module therein. (Conveniently, Python's
    bytecode parser ignores anything after an end marker, and the
    zipimport mechanism skips any non-ZIP data at the beginning of a
    file!) For example, say you have a directory called 'foo', which
    contains an __init__.py, which contains "print 'hello'; x = 123", and
    a thing.py, which contains "y = 456". If you make a ZIP archive of
    this and run it through Squisher, you'll get a single .pyc file which
    can be imported by any Python installation anywhere just like any
    other module, without requiring users to install any supporting
    mechanisms (like setuptools), and giving all the flexibility and
    stability of zipimport. In other words, you can do this simply by
    dropping the foo.pyc file into a directory:

    >>> import foo

    hello
    >>> foo.x

    123
    >>> from foo.thing import y
    >>> y

    456

    Of course, this is a stupid and useless example, but you can do it
    with any package you could use as an egg, yet without requiring people
    to install setuptools (or any other external glue) to use it. I've
    tested it with several large packages, including SQLAlchemy.

    Right now I'm just testing and polishing up the code... in the
    meantime, any comments?
     
    Adam Atlas, Mar 5, 2007
    #1
    1. Advertising

  2. Adam Atlas

    Adam Atlas Guest

    This could be easily made into a distutils extension (which was my
    intention all along, though that's not implemented yet). That's not
    the point. This is not intended as a "way to package source code".
    It's analogous to bdist, not sdist. The convenience gain is for the
    users, not (primarily) the developers, of the package.

    To use an Egg, you have to either put it in your sys.path manually or
    install setuptools and use it to install the Egg in some global scope.
    The advantage of Squisher is that you can take a whole package, squish
    it into one .pyc file, and distribute that. Then, anyone who downloads
    it can get a simple module file that can be used anywhere a usual .py
    file can, without editing sys.path or installing it somewhere.
    (Actually, this could be combined with eggs, since they're just ZIP
    files. You could use the same file as an egg or as a squished module
    simply by changing its extension.)

    ....Or am I missing something obvious about setuptools that already
    does this and making a fool of myself? :)
     
    Adam Atlas, Mar 5, 2007
    #2
    1. Advertising

  3. Adam Atlas a écrit :
    (snip)
    > If you make a ZIP archive of
    > this and run it through Squisher, you'll get a single .pyc file which
    > can be imported by any Python installation anywhere just like any
    > other module, without requiring users to install any supporting
    > mechanisms (like setuptools),


    setuptools offers *much* more than a convenient way to package source code.
     
    Bruno Desthuilliers, Mar 5, 2007
    #3
  4. Adam Atlas

    Adam Atlas Guest

    Ah... heh, sorry, I misread your message as "a much more convenient
    way" rather than "much more than a convenient way". Anyway, I
    understand that, and I do indeed find setuptools useful and use it on
    a regular basis.

    But my other points still stand. This would be a moot point if
    setuptools were part of the standard library, but it's not, and I
    don't see why people should have to bother installing it if they
    simply want to try a small package. Look at Beautiful Soup, for
    example. It's distributed as a single .py file, and that's great. With
    most modules, all I want to do is download them and plop them into my
    project directory. You can always copy it into site-packages if you
    want to access it globally, and you can always unzip it if you need to
    see the source.

    So I *will* retract my statement that this could be an "alternative"
    to eggs -- ideally, it would be an addition, since it doesn't break
    compatibility at all. You could download an egg and rename it to .pyc,
    and import it like any other module, and at any point later, you could
    rename it back to .egg and use setuptools to install it if you wish.
     
    Adam Atlas, Mar 5, 2007
    #4
  5. Adam Atlas

    Stef Mientki Guest

    Adam Atlas wrote:
    > Ah... heh, sorry, I misread your message as "a much more convenient
    > way" rather than "much more than a convenient way". Anyway, I
    > understand that, and I do indeed find setuptools useful and use it on
    > a regular basis.
    >
    > But my other points still stand. This would be a moot point if
    > setuptools were part of the standard library, but it's not, and I
    > don't see why people should have to bother installing it if they
    > simply want to try a small package. Look at Beautiful Soup, for
    > example. It's distributed as a single .py file, and that's great. With
    > most modules, all I want to do is download them and plop them into my
    > project directory. You can always copy it into site-packages if you
    > want to access it globally, and you can always unzip it if you need to
    > see the source.
    >
    > So I *will* retract my statement that this could be an "alternative"
    > to eggs -- ideally, it would be an addition, since it doesn't break
    > compatibility at all. You could download an egg and rename it to .pyc,
    > and import it like any other module, and at any point later, you could
    > rename it back to .egg and use setuptools to install it if you wish.
    >

    Good point to make these things much easier!

    But possibly I'm the only Windows user here,
    as I still find these talks all very difficult to understand,
    and I really don't understand why all this complexity is necessary,
    setuptools ? eggs ? zips ? pythonpaths ?

    As a normal Windows user,
    I'm used to run an install file,
    and hit just 1 button.
    As a normal Windows programmer,
    I'm used to create a simple Inno-setup file,
    and my users can behave as a simplistic and happy Windows user.

    But I guess the needed complexity is all thanks to NIX ;-)

    I think I never would have started with Python,
    if I didn't bounced into the Enthought-edition.

    --
    cheers,
    Stef Mientki
    http://pic.flappie.nl
     
    Stef Mientki, Mar 5, 2007
    #5
  6. Adam Atlas

    MonkeeSage Guest

    Adam,

    Sounds like a nice idea to me. Pretty ingenious use of the zip/
    bytecode headers and all too. Post a message when you release it
    please.

    Regards,
    Jordan
     
    MonkeeSage, Mar 5, 2007
    #6
  7. Adam Atlas

    Paul Boddie Guest

    Stef Mientki wrote:
    >
    > As a normal Windows user,
    > I'm used to run an install file,
    > and hit just 1 button.
    > As a normal Windows programmer,
    > I'm used to create a simple Inno-setup file,
    > and my users can behave as a simplistic and happy Windows user.
    >
    > But I guess the needed complexity is all thanks to NIX ;-)


    "Au contraire, as they say in England." - Steve X. Citement

    Most UNIX-based (or at least GNU/Linux or BSD-based) Python users are
    likely to have Python provided by their vendor together with a
    "constellation" of packages. As long as the software they need is
    packaged by their distribution, it's a matter of selecting the package
    concerned; other packages upon which the desired software depends will
    be installed automatically. You don't usually need to click through
    some wizard ("next", "I agree", "next", "next", "next", "next",
    "next", "finish"), either.

    Paul
     
    Paul Boddie, Mar 5, 2007
    #7
  8. Adam Atlas

    MonkeeSage Guest

    Stef,

    What Adam is talking about has nothing to do with windows or *nix.
    He's talking about packing one or more .py files into a single
    archive, which can be imported just like the regular .py files. This
    means you can distribute a whole bunch of module files/dirs as a
    single .pyc file. It just makes it easier to have a single file to
    distribute rather than a whole bunch, or a zip that one has to unzip
    into a certain directory, &c. So you can have myscript.py +
    myscriptfiles.pyc vs. myscript.py + somedepdir/ + someotherdepdir/ +
    somedepfile.py...&c.

    If you still don't get it, you don't need it.

    Regards,
    Jordan
     
    MonkeeSage, Mar 5, 2007
    #8
  9. Adam Atlas a écrit :
    > Ah... heh, sorry, I misread your message as "a much more convenient
    > way" rather than "much more than a convenient way".


    !-)

    (snip)

    > But my other points still stand.


    Indeed.
     
    Bruno Desthuilliers, Mar 5, 2007
    #9
  10. Stef Mientki a écrit :
    > Adam Atlas wrote:
    >
    >> Ah... heh, sorry, I misread your message as "a much more convenient
    >> way" rather than "much more than a convenient way". Anyway, I
    >> understand that, and I do indeed find setuptools useful and use it on
    >> a regular basis.
    >>
    >> But my other points still stand. This would be a moot point if
    >> setuptools were part of the standard library, but it's not, and I
    >> don't see why people should have to bother installing it if they
    >> simply want to try a small package. Look at Beautiful Soup, for
    >> example. It's distributed as a single .py file, and that's great. With
    >> most modules, all I want to do is download them and plop them into my
    >> project directory. You can always copy it into site-packages if you
    >> want to access it globally, and you can always unzip it if you need to
    >> see the source.
    >>
    >> So I *will* retract my statement that this could be an "alternative"
    >> to eggs -- ideally, it would be an addition, since it doesn't break
    >> compatibility at all. You could download an egg and rename it to .pyc,
    >> and import it like any other module, and at any point later, you could
    >> rename it back to .egg and use setuptools to install it if you wish.
    >>

    > Good point to make these things much easier!
    >
    > But possibly I'm the only Windows user here,


    I don't think so.

    > as I still find these talks all very difficult to understand,
    > and I really don't understand why all this complexity is necessary,
    > setuptools ? eggs ? zips ? pythonpaths ?
    >
    > As a normal Windows user,
    > I'm used to run an install file,
    > and hit just 1 button.


    As a normal *n*x user, I'm used to run a single command line. Don't even
    have to hit a button !-)

    > But I guess the needed complexity is all thanks to NIX ;-)


    FUD.
     
    Bruno Desthuilliers, Mar 5, 2007
    #10
  11. Adam Atlas

    Adam Atlas Guest

    Okay, here's the prototype...
    <http://adamatlas.org/2007/03/Squisher-0.1.py>
    It's meant to be run as a command line program (pass it a directory or
    a zip file as an argument). By default it will save the new file to
    the argument's base name plus '.pyc'. You can override this with -o.

    Obviously it's very much a work in progress... try it out and let me
    know the results.
     
    Adam Atlas, Mar 6, 2007
    #11
  12. Adam Atlas

    Peter Wang Guest

    On Mar 5, 12:31 am, "Adam Atlas" <> wrote:
    > Right now I'm just testing and polishing up the code... in the
    > meantime, any comments?


    How does this work with compiled extension modules?
     
    Peter Wang, Mar 6, 2007
    #12
  13. Adam Atlas

    Adam Atlas Guest

    Doesn't seem to work. I guess zipimport doesn't support that by
    default... but if I remember correctly, Setuptools adds that. Maybe
    I'll take a look at how it does it (I think by extracting the .so to /
    tmp?) and see how easy it would be to integrate it here.
     
    Adam Atlas, Mar 6, 2007
    #13
  14. Adam Atlas

    Adam Atlas Guest

    I updated it.
    http://adamatlas.org/2007/03/Squisher-0.2.py

    New Things:
    - It supports C extensions within squished packages.
    - It supports including squished packages within other squished
    packages. (That is, you can have a package that includes a .pyc
    generated by this, and turn that whole package into one .pyc.)
    - If you import it after you import setuptools in a setup.py, it will
    override a bit of setuptools so that bdist_egg will result in an egg
    that can also be used as a squished package (if you rename it to
    something ending in .pyc).
    - It puts a helpful docstring at the top of generated .pyc files,
    mainly for the purpose of being easy to see if you open the file in a
    text editor. (It tells you what can be done with the file -- name it
    something .pyc, or unzip it, or, if applicable, use it as an egg.)
     
    Adam Atlas, Mar 7, 2007
    #14
  15. Adam Atlas a écrit :
    > Doesn't seem to work. I guess zipimport doesn't support that by
    > default... but if I remember correctly, Setuptools adds that. Maybe
    > I'll take a look at how it does it (I think by extracting the .so to /
    > tmp?)


    or to another known location, IIRC.

    > and see how easy it would be to integrate it here.
    >
     
    Bruno Desthuilliers, Mar 8, 2007
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ralf W. Grosse-Kunstleve
    Replies:
    16
    Views:
    601
    Lonnie Princehouse
    Jul 11, 2005
  2. Ralf W. Grosse-Kunstleve
    Replies:
    18
    Views:
    606
    Bengt Richter
    Jul 11, 2005
  3. Ralf W. Grosse-Kunstleve
    Replies:
    2
    Views:
    414
    Dan Sommers
    Jul 12, 2005
  4. Replies:
    9
    Views:
    487
    Paul Boddie
    Jan 4, 2006
  5. Adam Atlas

    [ANN] squisher 0.3

    Adam Atlas, Mar 17, 2007, in forum: Python
    Replies:
    0
    Views:
    234
    Adam Atlas
    Mar 17, 2007
Loading...

Share This Page