Packaging a proprietary Python library for multiple OSs

[Michael Herrmann]

Hi everyone,

I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux &Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.

The library is currently shipped in the form of a Zip file. This archive contains the compiled Python code for the implementation of my library, plus all dependencies. By placing the Zip file on his PYTHONPATH, the customer can use the library from his Python scripts.

Shipping a Zip file with all dependencies included has the following advantages:

* No internet access or administrator privileges are required to install the library.
* The customer does not have to worry about installing / managing dependencies of my library.

It also has the disadvantage that the customer is not (easily) able to use his own versions of my library's dependencies.

Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.

Unfortunately, py2exe is only available for Windows. I need to also be ableto build it on Linux & Mac, hence change the build process to not use py2exe.

My questions are:

1. Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies? From what I have seen, it seems to be an unusual approach, at the least. How do *you* prefer to obtain and install Python libraries?
2. Is it possible to distribute the library in a form that allows for an offline installation without administrator privileges using other tools, such as setuptools?

My insight into the state of the art in Python regarding these matters is limited, so I would appreciate advice from someone with more experience in the subject.

A hard requirement is that I can only ship binary distributions of my library, as this is a proprietary product. I looked at Distutils and Setuptools,where the recommended approach seems to be to simply ship all sources.

Many thanks!

 

[rusi]

Hi everyone,

I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux & Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.

Wheel is the upcoming standard I think.
http://www.python.org/dev/peps/pep-0427/

1. It would be dishonest to remove the 'upcoming'
2. It would also be dishonest if you thought I know anything about the subject
3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
place to ask

 

[Roy Smith]

1. Is it considered a bad idea in the Python community to ship one large Zip
file with all dependencies?
Yes.

How do *you* prefer to obtain and install Python libraries?

"pip install"

2. Is it possible to distribute the library in a form that allows for an
offline installation without administrator privileges using other tools,
such as setuptools?

You can use "pip --find-links" to point pip at a local repository of
packages. That solves the offline part. And the "without admin privs"
part is solved by setting up a virtualenv.

A hard requirement is that I can only ship binary distributions of my
library, as this is a proprietary product. I looked at Distutils and
Setuptools, where the recommended approach seems to be to simply ship all
sources.

Keep in mind that shipping just the pyc files offers very weak
protection against people examining your code. Google for "python
decompile" and you'll find a number of projects. I'm looking at the
docs for uncompyle now, which says:

'uncompyle' converts Python byte-code back into equivalent Python
source. It accepts byte-code from Python version 2.7 only.

The generated source is very readable: docstrings, lists, tuples and
hashes get pretty-printed.

About the only thing not shipping Python source does is satisfy a
check-box requirement that you not ship source. It may make the lawyers
and bean-counters happy, but that's about it.

 

[Kevin Walzer]

Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.

If your library and their dependencies are simply .pyc files, then I
don't see why a zip collated via py2exe wouldn't work on other
platforms. Obviously this point is moot if your library includes true
compiled (C-based) extensions.

--Kevin

 

[Travis Griggs]

3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
place to ask

Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches”?

“Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"

 

[Michael Herrmann]

Wheel is the upcoming standard I think.
http://www.python.org/dev/peps/pep-0427/

I hadn't known of Wheel - thanks for pointing it out!

 

[Michael Herrmann]

Yes.

I see. Unfortunately, the library's users may be non-technical and might not even have experience with Python. The easier the installation process, therefore, the better.

"pip install"

Thanks for this input.

You can use "pip --find-links" to point pip at a local repository of
packages. That solves the offline part. And the "without admin privs"
part is solved by setting up a virtualenv.

Both "pip --find-links" and "virtualenv" sound technically feasible but maybe too difficult for my users (especially virtualenv).

Keep in mind that shipping just the pyc files offers very weak
protection against people examining your code. Google for "python
decompile" and you'll find a number of projects. I'm looking at the
docs for uncompyle now, which says:

Very interesting point. Thank you very much for pointing out uncompyle. I had always known that it was easy to decompile .pyc files, but hadn't imagined it to be that easy. I just tried uncompyle with some of our proprietary ..pyc files. It took 5 minutes to set up and the results are near-perfect. Scary... :-S We might have to look into tools such as http://www.bitboost.com/#Python_obfuscator to obfuscate our code.

Thanks for the valuable insights!
Michael

 

[Michael Herrmann]

On 12/5/13, 5:14 AM, Michael Herrmann wrote:
If your library and their dependencies are simply .pyc files, then I
don't see why a zip collated via py2exe wouldn't work on other
platforms. Obviously this point is moot if your library includes true
compiled (C-based) extensions.

As I said, I need to make my *build* platform-independent.

Thanks,
Michael

 

[random832]

Very interesting point. Thank you very much for pointing out uncompyle. I
had always known that it was easy to decompile .pyc files, but hadn't
imagined it to be that easy. I just tried uncompyle with some of our
proprietary .pyc files. It took 5 minutes to set up and the results are
near-perfect. Scary... :-S We might have to look into tools such as
http://www.bitboost.com/#Python_obfuscator to obfuscate our code.

Or you could just sue anyone who steals your code.

 

[Kevin Walzer]

As I said, I need to make my *build* platform-independent.

cx_Freeze is platform independent, but I'm not sure if it generates
libraries or simply executables.

--Kevin

 

[Chris Angelico]

Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches�

“Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"

No, it's not like that. It's that there are some people who, despite
truckloads of evidence to the contrary, still think that Google Groups
is worth using. Rusi is one of them. Fortunately, he has defended his
position by making his posts not look like the ridiculous junk that GG
creates by default, but that doesn't make GG a good product. It's like
an argument my boss and I had: I said that PHP is a bad language, and
he said that it can't possibly be a bad language because he's able to
write good code in it.

I don't know what mailing list there is for virtualenv as I don't use
it, but there's likely to be an alternative source of knowledge on it.

ChrisA

 

[mherrmann.at]

Or you could just sue anyone who steals your code.

I see your point but I don't think it's very practical. If the person who stole the code sits in some remote country with a completely different legalsystem, I think I'll have a hard time getting at this person. If I even manage to find out where the person is at all.

 

[Roy Smith]

No, it's not like that. It's that there are some people who, despite
truckloads of evidence to the contrary, still think that Google Groups

I use GG on occasion (I'm using it now), when I don't have access to a better newsreader. Like Rusi, I take the effort to clean up the double-space mess GG produces by default. That doesn't mean GG isn't a piece of crap; itis. That fact that I, and Rusi, know enough, and take the effort, to overcome its shortcomings doesn't change that.

I put GG it in the category of "attractive nuisance". It's like leaving cans of spray paint laying around school playgrounds and then being surprisedwhen the kids pick them up and use them to paint graffiti. It certainly violates Google's "do no harm" motto when it interacts with usenet groups.

I keep hearing that I should use gmane as a superior interface. Well, I tried that. I went to http://dir.gmane.org/search.php, where it asks me to search for a newsgroup. I type in "comp.lang.python", and it tells me, "No matching groups". So, that seems pretty broken to me.

It's like an argument my boss and I had: I said that PHP is a bad language, and
he said that it can't possibly be a bad language because he's able to
write good code in it.

PHP is a disaster of a language. But, like any bad tool, a good craftsman can produce a quality product with it. Wikipedia is written in PHP. So, apparently, is gmane As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have someredeeming qualities.

 

[Mark Lawrence]

No, it's not like that. It's that there are some people who, despite
truckloads of evidence to the contrary, still think that Google Groups
is worth using. Rusi is one of them. Fortunately, he has defended his
position by making his posts not look like the ridiculous junk that GG
creates by default, but that doesn't make GG a good product. It's like
an argument my boss and I had: I said that PHP is a bad language, and
he said that it can't possibly be a bad language because he's able to
write good code in it.

I don't know what mailing list there is for virtualenv as I don't use
it, but there's likely to be an alternative source of knowledge on it.

ChrisA

gmane.comp.python.virtualenv

 

[Chris Angelico]

PHP is a disaster of a language. But, like any bad tool, a good craftsman can produce a quality product with it. Wikipedia is written in PHP. So,apparently, is gmane As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have some redeeming qualities.

Right. And the fact that Wikipedia can be written in PHP is not itself
proof that it's a good language. You and Rusi are fighting against
GG's faults and not entirely succeeding, as your paragraphs come out
unwrapped; that's possibly the least of the GG woes, but it's one of
the first clues that someone's replies are likely to be double-spaced.

A good tool does most of your work for you. A bad tool has to be
fought every inch of the way. Sometimes a tool is good but wrongly
chosen (don't use DeScribe Macro Language for writing a GUI - drop to
REXX for that!), but some tools have no good use at all.

ChrisA

 

[Zero Piraeus]

:

I keep hearing that I should use gmane as a superior interface. Well,
I tried that. I went to http://dir.gmane.org/search.php, where it
asks me to search for a newsgroup. I type in "comp.lang.python", and
it tells me, "No matching groups". So, that seems pretty broken to
me.

That's not entirely fair - Gmane presents mailing lists as newsgroups,
not vice versa, so it doesn't know that (e-mail address removed) is
connected to comp.lang.python (or that comp.lang.python even exists).

A search for the mailing list from the front page works just fine:

http://gmane.org/[email protected]

-[]z.

 

[Mark Lawrence]

:

I keep hearing that I should use gmane as a superior interface. Well,
I tried that. I went to http://dir.gmane.org/search.php, where it
asks me to search for a newsgroup. I type in "comp.lang.python", and
it tells me, "No matching groups". So, that seems pretty broken to
me.

That's not entirely fair - Gmane presents mailing lists as newsgroups,
not vice versa, so it doesn't know that (e-mail address removed) is
connected to comp.lang.python (or that comp.lang.python even exists).

A search for the mailing list from the front page works just fine:

http://gmane.org/[email protected]

-[]z.

Another useful link, there are just a few python goodies there
http://news.gmane.org/index.php?prefix=gmane.comp.python

 

[Robert Kern]

:



That's not entirely fair - Gmane presents mailing lists as newsgroups,
not vice versa, so it doesn't know that (e-mail address removed) is
connected to comp.lang.python (or that comp.lang.python even exists).

A search for the mailing list from the front page works just fine:

http://gmane.org/[email protected]

Right. GMane is an NNTP service, but it is not part of the USENET network.
comp.lang.python is a USENET newsgroup and requires a true USENET server (not
just an NNTP server) to access.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Kevin Walzer]

As I said, I need to make my *build* platform-independent.

Giving this further thought, I'm wondering how hard it would be to roll
your own using modulefinder, Python's zip tools, and some custom code.
Just sayin'.

--Kevin