Packaging a proprietary Python library for multiple OSs

Discussion in 'Python' started by Michael Herrmann, Dec 5, 2013.

  1. Hi everyone,

    I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux &Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.

    The library is currently shipped in the form of a Zip file. This archive contains the compiled Python code for the implementation of my library, plus all dependencies. By placing the Zip file on his PYTHONPATH, the customer can use the library from his Python scripts.

    Shipping a Zip file with all dependencies included has the following advantages:

    * No internet access or administrator privileges are required to install the library.
    * The customer does not have to worry about installing / managing dependencies of my library.

    It also has the disadvantage that the customer is not (easily) able to use his own versions of my library's dependencies.

    Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.

    Unfortunately, py2exe is only available for Windows. I need to also be ableto build it on Linux & Mac, hence change the build process to not use py2exe.

    My questions are:

    1. Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies? From what I have seen, it seems to be an unusual approach, at the least. How do *you* prefer to obtain and install Python libraries?
    2. Is it possible to distribute the library in a form that allows for an offline installation without administrator privileges using other tools, such as setuptools?

    My insight into the state of the art in Python regarding these matters is limited, so I would appreciate advice from someone with more experience in the subject.

    A hard requirement is that I can only ship binary distributions of my library, as this is a proprietary product. I looked at Distutils and Setuptools,where the recommended approach seems to be to simply ship all sources.

    Many thanks!
    Michael Herrmann, Dec 5, 2013
    #1
    1. Advertising

  2. Michael Herrmann

    rusi Guest

    On Thursday, December 5, 2013 3:44:50 PM UTC+5:30, Michael Herrmann wrote:
    > Hi everyone,
    >
    > I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux & Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.


    Wheel is the upcoming standard I think.
    http://www.python.org/dev/peps/pep-0427/

    1. It would be dishonest to remove the 'upcoming'
    2. It would also be dishonest if you thought I know anything about the subject :)
    3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
    place to ask
    rusi, Dec 5, 2013
    #2
    1. Advertising

  3. Michael Herrmann

    Roy Smith Guest

    In article <>,
    Michael Herrmann <> wrote:

    > 1. Is it considered a bad idea in the Python community to ship one large Zip
    > file with all dependencies?


    Yes.

    > How do *you* prefer to obtain and install Python libraries?


    "pip install"

    > 2. Is it possible to distribute the library in a form that allows for an
    > offline installation without administrator privileges using other tools,
    > such as setuptools?


    You can use "pip --find-links" to point pip at a local repository of
    packages. That solves the offline part. And the "without admin privs"
    part is solved by setting up a virtualenv.

    > A hard requirement is that I can only ship binary distributions of my
    > library, as this is a proprietary product. I looked at Distutils and
    > Setuptools, where the recommended approach seems to be to simply ship all
    > sources.


    Keep in mind that shipping just the pyc files offers very weak
    protection against people examining your code. Google for "python
    decompile" and you'll find a number of projects. I'm looking at the
    docs for uncompyle now, which says:

    > 'uncompyle' converts Python byte-code back into equivalent Python
    > source. It accepts byte-code from Python version 2.7 only.
    >
    > The generated source is very readable: docstrings, lists, tuples and
    > hashes get pretty-printed.


    About the only thing not shipping Python source does is satisfy a
    check-box requirement that you not ship source. It may make the lawyers
    and bean-counters happy, but that's about it.
    Roy Smith, Dec 5, 2013
    #3
  4. Michael Herrmann

    Kevin Walzer Guest

    On 12/5/13, 5:14 AM, Michael Herrmann wrote:
    > Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.


    If your library and their dependencies are simply .pyc files, then I
    don't see why a zip collated via py2exe wouldn't work on other
    platforms. Obviously this point is moot if your library includes true
    compiled (C-based) extensions.

    --Kevin

    --
    Kevin Walzer
    Code by Kevin/Mobile Code by Kevin
    http://www.codebykevin.com
    http://www.wtmobilesoftware.com
    Kevin Walzer, Dec 5, 2013
    #4
  5. On Dec 5, 2013, at 2:56 AM, rusi <> wrote:

    > 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
    > place to ask


    Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches”?

    “Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"
    Travis Griggs, Dec 5, 2013
    #5
  6. On Thursday, December 5, 2013 11:56:16 AM UTC+1, rusi wrote:
    > Wheel is the upcoming standard I think.
    > http://www.python.org/dev/peps/pep-0427/


    I hadn't known of Wheel - thanks for pointing it out!
    Michael Herrmann, Dec 5, 2013
    #6
  7. On Thursday, December 5, 2013 3:09:32 PM UTC+1, Roy Smith wrote:
    > > 1. Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies?

    > Yes.


    I see. Unfortunately, the library's users may be non-technical and might not even have experience with Python. The easier the installation process, therefore, the better.

    > > How do *you* prefer to obtain and install Python libraries?

    > "pip install"


    Thanks for this input.

    > > 2. Is it possible to distribute the library in a form that allows for an
    > > offline installation without administrator privileges using other tools,
    > > such as setuptools?

    >
    > You can use "pip --find-links" to point pip at a local repository of
    > packages. That solves the offline part. And the "without admin privs"
    > part is solved by setting up a virtualenv.


    Both "pip --find-links" and "virtualenv" sound technically feasible but maybe too difficult for my users (especially virtualenv).

    > > A hard requirement is that I can only ship binary distributions of my
    > > library, as this is a proprietary product. I looked at Distutils and
    > > Setuptools, where the recommended approach seems to be to simply ship all
    > > sources.

    > Keep in mind that shipping just the pyc files offers very weak
    > protection against people examining your code. Google for "python
    > decompile" and you'll find a number of projects. I'm looking at the
    > docs for uncompyle now, which says:
    > > 'uncompyle' converts Python byte-code back into equivalent Python
    > > source. It accepts byte-code from Python version 2.7 only.


    Very interesting point. Thank you very much for pointing out uncompyle. I had always known that it was easy to decompile .pyc files, but hadn't imagined it to be that easy. I just tried uncompyle with some of our proprietary ..pyc files. It took 5 minutes to set up and the results are near-perfect. Scary... :-S We might have to look into tools such as http://www.bitboost.com/#Python_obfuscator to obfuscate our code.

    Thanks for the valuable insights!
    Michael
    Michael Herrmann, Dec 5, 2013
    #7
  8. On Thursday, December 5, 2013 4:26:40 PM UTC+1, Kevin Walzer wrote:
    > On 12/5/13, 5:14 AM, Michael Herrmann wrote:
    > If your library and their dependencies are simply .pyc files, then I
    > don't see why a zip collated via py2exe wouldn't work on other
    > platforms. Obviously this point is moot if your library includes true
    > compiled (C-based) extensions.


    As I said, I need to make my *build* platform-independent.

    Thanks,
    Michael
    Michael Herrmann, Dec 5, 2013
    #8
  9. Michael Herrmann

    Guest

    On Thu, Dec 5, 2013, at 10:49, Michael Herrmann wrote:
    > Very interesting point. Thank you very much for pointing out uncompyle. I
    > had always known that it was easy to decompile .pyc files, but hadn't
    > imagined it to be that easy. I just tried uncompyle with some of our
    > proprietary .pyc files. It took 5 minutes to set up and the results are
    > near-perfect. Scary... :-S We might have to look into tools such as
    > http://www.bitboost.com/#Python_obfuscator to obfuscate our code.


    Or you could just sue anyone who steals your code.
    , Dec 5, 2013
    #9
  10. Michael Herrmann

    Kevin Walzer Guest

    On 12/5/13, 10:50 AM, Michael Herrmann wrote:
    > As I said, I need to make my *build* platform-independent.


    cx_Freeze is platform independent, but I'm not sure if it generates
    libraries or simply executables.

    --Kevin

    --
    Kevin Walzer
    Code by Kevin/Mobile Code by Kevin
    http://www.codebykevin.com
    http://www.wtmobilesoftware.com
    Kevin Walzer, Dec 5, 2013
    #10
  11. On Fri, Dec 6, 2013 at 2:32 AM, Travis Griggs <> wrote:
    >
    > On Dec 5, 2013, at 2:56 AM, rusi <> wrote:
    >
    >> 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
    >> place to ask

    >
    > Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches�
    >
    > “Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"


    No, it's not like that. It's that there are some people who, despite
    truckloads of evidence to the contrary, still think that Google Groups
    is worth using. Rusi is one of them. Fortunately, he has defended his
    position by making his posts not look like the ridiculous junk that GG
    creates by default, but that doesn't make GG a good product. It's like
    an argument my boss and I had: I said that PHP is a bad language, and
    he said that it can't possibly be a bad language because he's able to
    write good code in it.

    I don't know what mailing list there is for virtualenv as I don't use
    it, but there's likely to be an alternative source of knowledge on it.

    ChrisA
    Chris Angelico, Dec 5, 2013
    #11
  12. Michael Herrmann

    Guest

    On Thursday, 5 December 2013 16:52:45 UTC+1, wrote:
    > Or you could just sue anyone who steals your code.


    I see your point but I don't think it's very practical. If the person who stole the code sits in some remote country with a completely different legalsystem, I think I'll have a hard time getting at this person. If I even manage to find out where the person is at all.
    , Dec 5, 2013
    #12
  13. Michael Herrmann

    Roy Smith Guest

    On Thursday, December 5, 2013 11:20:41 AM UTC-5, Chris Angelico wrote:

    > No, it's not like that. It's that there are some people who, despite
    > truckloads of evidence to the contrary, still think that Google Groups
    > > is worth using. Rusi is one of them. Fortunately, he has defended his
    > > position by making his posts not look like the ridiculous junk that GG
    > > creates by default, but that doesn't make GG a good product.


    I use GG on occasion (I'm using it now), when I don't have access to a better newsreader. Like Rusi, I take the effort to clean up the double-space mess GG produces by default. That doesn't mean GG isn't a piece of crap; itis. That fact that I, and Rusi, know enough, and take the effort, to overcome its shortcomings doesn't change that.

    I put GG it in the category of "attractive nuisance". It's like leaving cans of spray paint laying around school playgrounds and then being surprisedwhen the kids pick them up and use them to paint graffiti. It certainly violates Google's "do no harm" motto when it interacts with usenet groups.

    I keep hearing that I should use gmane as a superior interface. Well, I tried that. I went to http://dir.gmane.org/search.php, where it asks me to search for a newsgroup. I type in "comp.lang.python", and it tells me, "No matching groups". So, that seems pretty broken to me.

    > It's like an argument my boss and I had: I said that PHP is a bad language, and
    > he said that it can't possibly be a bad language because he's able to
    > write good code in it.


    PHP is a disaster of a language. But, like any bad tool, a good craftsman can produce a quality product with it. Wikipedia is written in PHP. So, apparently, is gmane :) As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have someredeeming qualities.
    Roy Smith, Dec 5, 2013
    #13
  14. On 05/12/2013 16:20, Chris Angelico wrote:
    > On Fri, Dec 6, 2013 at 2:32 AM, Travis Griggs <> wrote:
    >>
    >> On Dec 5, 2013, at 2:56 AM, rusi <> wrote:
    >>
    >>> 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
    >>> place to ask

    >>
    >> Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches�
    >>
    >> “Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"

    >
    > No, it's not like that. It's that there are some people who, despite
    > truckloads of evidence to the contrary, still think that Google Groups
    > is worth using. Rusi is one of them. Fortunately, he has defended his
    > position by making his posts not look like the ridiculous junk that GG
    > creates by default, but that doesn't make GG a good product. It's like
    > an argument my boss and I had: I said that PHP is a bad language, and
    > he said that it can't possibly be a bad language because he's able to
    > write good code in it.
    >
    > I don't know what mailing list there is for virtualenv as I don't use
    > it, but there's likely to be an alternative source of knowledge on it.
    >
    > ChrisA
    >


    gmane.comp.python.virtualenv

    --
    My fellow Pythonistas, ask not what our language can do for you, ask
    what you can do for our language.

    Mark Lawrence
    Mark Lawrence, Dec 5, 2013
    #14
  15. On Fri, Dec 6, 2013 at 4:12 AM, Roy Smith <> wrote:
    >> It's like an argument my boss and I had: I said that PHP is a bad language, and
    >> he said that it can't possibly be a bad language because he's able to
    >> write good code in it.

    >
    > PHP is a disaster of a language. But, like any bad tool, a good craftsman can produce a quality product with it. Wikipedia is written in PHP. So,apparently, is gmane :) As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have some redeeming qualities.


    Right. And the fact that Wikipedia can be written in PHP is not itself
    proof that it's a good language. You and Rusi are fighting against
    GG's faults and not entirely succeeding, as your paragraphs come out
    unwrapped; that's possibly the least of the GG woes, but it's one of
    the first clues that someone's replies are likely to be double-spaced.

    A good tool does most of your work for you. A bad tool has to be
    fought every inch of the way. Sometimes a tool is good but wrongly
    chosen (don't use DeScribe Macro Language for writing a GUI - drop to
    REXX for that!), but some tools have no good use at all.

    ChrisA
    Chris Angelico, Dec 5, 2013
    #15
  16. Michael Herrmann

    Zero Piraeus Guest

    :

    On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
    > I keep hearing that I should use gmane as a superior interface. Well,
    > I tried that. I went to http://dir.gmane.org/search.php, where it
    > asks me to search for a newsgroup. I type in "comp.lang.python", and
    > it tells me, "No matching groups". So, that seems pretty broken to
    > me.


    That's not entirely fair - Gmane presents mailing lists as newsgroups,
    not vice versa, so it doesn't know that is
    connected to comp.lang.python (or that comp.lang.python even exists).

    A search for the mailing list from the front page works just fine:

    http://gmane.org/find.php?list=

    -[]z.

    --
    Zero Piraeus: flagellum dei
    http://etiol.net/pubkey.asc
    Zero Piraeus, Dec 5, 2013
    #16
  17. On 05/12/2013 17:50, Zero Piraeus wrote:
    > :
    >
    > On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
    >> I keep hearing that I should use gmane as a superior interface. Well,
    >> I tried that. I went to http://dir.gmane.org/search.php, where it
    >> asks me to search for a newsgroup. I type in "comp.lang.python", and
    >> it tells me, "No matching groups". So, that seems pretty broken to
    >> me.

    >
    > That's not entirely fair - Gmane presents mailing lists as newsgroups,
    > not vice versa, so it doesn't know that is
    > connected to comp.lang.python (or that comp.lang.python even exists).
    >
    > A search for the mailing list from the front page works just fine:
    >
    > http://gmane.org/find.php?list=
    >
    > -[]z.
    >


    Another useful link, there are just a few python goodies there
    http://news.gmane.org/index.php?prefix=gmane.comp.python

    --
    My fellow Pythonistas, ask not what our language can do for you, ask
    what you can do for our language.

    Mark Lawrence
    Mark Lawrence, Dec 5, 2013
    #17
  18. Michael Herrmann

    Robert Kern Guest

    On 2013-12-05 17:50, Zero Piraeus wrote:
    > :
    >
    > On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
    >> I keep hearing that I should use gmane as a superior interface. Well,
    >> I tried that. I went to http://dir.gmane.org/search.php, where it
    >> asks me to search for a newsgroup. I type in "comp.lang.python", and
    >> it tells me, "No matching groups". So, that seems pretty broken to
    >> me.

    >
    > That's not entirely fair - Gmane presents mailing lists as newsgroups,
    > not vice versa, so it doesn't know that is
    > connected to comp.lang.python (or that comp.lang.python even exists).
    >
    > A search for the mailing list from the front page works just fine:
    >
    > http://gmane.org/find.php?list=


    Right. GMane is an NNTP service, but it is not part of the USENET network.
    comp.lang.python is a USENET newsgroup and requires a true USENET server (not
    just an NNTP server) to access.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Dec 5, 2013
    #18
  19. Michael Herrmann

    Kevin Walzer Guest

    On 12/5/13, 10:50 AM, Michael Herrmann wrote:
    > On Thursday, December 5, 2013 4:26:40 PM UTC+1, Kevin Walzer wrote:
    >> On 12/5/13, 5:14 AM, Michael Herrmann wrote:
    >> If your library and their dependencies are simply .pyc files, then I
    >> don't see why a zip collated via py2exe wouldn't work on other
    >> platforms. Obviously this point is moot if your library includes true
    >> compiled (C-based) extensions.

    >
    > As I said, I need to make my *build* platform-independent.


    Giving this further thought, I'm wondering how hard it would be to roll
    your own using modulefinder, Python's zip tools, and some custom code.
    Just sayin'.

    --Kevin


    --
    Kevin Walzer
    Code by Kevin/Mobile Code by Kevin
    http://www.codebykevin.com
    http://www.wtmobilesoftware.com
    Kevin Walzer, Dec 6, 2013
    #19
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joakim Persson
    Replies:
    8
    Views:
    1,367
    Paul Casteels
    Sep 21, 2005
  2. Replies:
    18
    Views:
    9,826
    AdrianC
    Sep 19, 2010
  3. Biggmatt
    Replies:
    0
    Views:
    461
    Biggmatt
    Apr 19, 2006
  4. Craig
    Replies:
    24
    Views:
    1,389
    Dennis Lee Bieber
    Mar 27, 2008
  5. Alpha Blue
    Replies:
    6
    Views:
    105
    Alpha Blue
    Feb 1, 2010
Loading...

Share This Page