Maintaining a backported module

[Steven D'Aprano]

As some of you are aware, I have a module accepted into the standard
library:

http://docs.python.org/3.4/library/statistics.html

I'm now at the point where I wish to backport this module to support
versions of Python back to 3.1 at least and possibly 2.7, and put it up
on PyPI.

I'm looking for advice on best practices for doing so. Any suggestions
for managing bug fixes and enhancements to two separate code-bases
without them diverging too much?

Other than "Avoid it"

 

[Paul Rubin]

As some of you are aware, I have a module accepted into the standard
library:
http://docs.python.org/3.4/library/statistics.html

Wow, neat, I had seen something about the module and thought it looked
great, but I didn't realize you were the author. Awesome!

Any suggestions for managing bug fixes and enhancements to two
separate code-bases without them diverging too much?

I haven't touched 3.x yet, but I found with a little bit of care in
resisting use of new features, I was able to implement in the
intersection of 2.x and 1.5, so the same code worked in both. That
avoided the need for two codebases in the stuff I did. I don't know if
it would be more difficult with the stats module.

 

[Metallicow]

A useful library for this purpose is ‘six’ (as in “3 × 2”)
<URL:http://pythonhosted.org/six/>. You can use its features to do
things that are useful or better in Python 3, but which need special
implementation to work on Python 2; and the same code will just work on
both versions.

+1 for stdev Steven. Thanks for the extra legs.
Hope all goes well with introductions... I'm sure it will.
Good Job.

 

[Ethan Furman]

I'm looking for advice on best practices for doing so. Any suggestions
for managing bug fixes and enhancements to two separate code-bases
without them diverging too much?

Confining your code to the intersection of 2.7 and 3.x is probably going to be the easiest thing to do as 2.7 has a
bunch of 3.x features.

Sadly, when I backported Enum I was targeting 2.5 - 3.x because I have systems still running 2.5. That was *not* a fun
experience.

 

[Antoon Pardon]

Op 24-10-13 06:54, Steven D'Aprano schreef:

As some of you are aware, I have a module accepted into the standard
library:

http://docs.python.org/3.4/library/statistics.html

I'm now at the point where I wish to backport this module to support
versions of Python back to 3.1 at least and possibly 2.7, and put it up
on PyPI.

I'm looking for advice on best practices for doing so. Any suggestions
for managing bug fixes and enhancements to two separate code-bases
without them diverging too much?

Other than "Avoid it"

I would use only one code-base and make a branch for the 2.7 version.
You can then adapt the 2.7 version where really necessary but in general
just merge adaption made in the main branch to the 2.7 branch.

I am assuming you are using some kind of version control.

 

[Steven D'Aprano]

Ned Batchelder has managed something at least as ambitious (supporting
Python versions 2.4 through 3.3), which should be helpful in your case
<URL:http://nedbatchelder.com/blog/200910/

running_the_same_code_on_python_2x_and_3x.html>.

I too have written code that supports 2.4 - 3.3 (although only relatively
small modules). That's not the problem here. My problem is that I have
two chunks of code:

1) statistics.py in the standard library, which is written for
Python 3.3/3.4 only, and should be as clean as possible;

2) a backport of it, on PyPI, which will support older Pythons, and
may be slower/uglier if need be.

My problem is not supporting 2.7 and 3.4 in the one code base. My problem
is, how do I prevent #1 and #2 from gradually diverging?

The easy answer is "unit tests", but the unit tests for 1) are in the std
lib and target 3.4, while the unit tests for 2) will be on PyPI and
won't. So how do I keep the unit tests from diverging?

 

[Ned Batchelder]

Ned Batchelder has managed something at least as ambitious (supporting
Python versions 2.4 through 3.3), which should be helpful in your case
<URL:http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html>.

FWIW, coverage.py currently runs on 2.3 through 3.4. It mostly comes
down to:

1. avoiding newer features (decorators! generator expressions! rpartition!),
2. using a compatibility layer like "six" (though I started my own
called backward.py before six existed),
3. using some awkward syntax workarounds (sys.exc_info()[1] to get the
current exception),
4. somehow finding a way to test on all those versions (pythonz helps,
and you have to limit your dependencies).

Also, I've just started the coverage.py 4.x branch, which will run on

=2.6 and >=3.2, and it's very nice to get rid of some of that

compatibility stuff.

--Ned.

 

[Steven D'Aprano]

coverage.py currently runs on 2.3 through 3.4

You support all the way back to 2.3???

I don't know whether to admire your dedication, or back away slowly since
you're obviously a crazy person

 

[rusi]

You support all the way back to 2.3???

I don't know whether to admire your dedication, or back away slowly since
you're obviously a crazy person

Yes, in the end youve to choose between a rock and a hard place:
Either you write idiomatic, beautiful 3-code that is not 2-compatible or you write not so beautiful code putting compatibility at the highest priority.

Cant think of examples right now but Ive seen very old python code with True, False defined and inside try-block and all that. Looks ugly but is compatibility-wise correct adn appropriate

BTW saw this yesterday:
An interesting read correlating OSS projects that succeed and fail with the difference between a commitment to computer science and to computer engineering.
Consider it an early-warning system!

http://www.forbes.com/sites/danwoods/2013/10/22/why-hasnt-open-source-taken-over-storage/

 

[Ned Batchelder]

You support all the way back to 2.3???

I don't know whether to admire your dedication, or back away slowly since
you're obviously a crazy person

Yeah, it's kind of crazy. It was a boiling frog situation: the package
supported 2.3 back when that was just normal, and there was no obvious
time to drop it, so it's been carried along.

It's been fun dropping the contortions for coverage.py 4.x, though!

--Ned.

 

[Ethan Furman]

Have you or could you publish anything regarding your experiences, I
suspect it would be an enlightening read for a lot of us?

The only thing I can add to what's already been posted in this thread
(and it was advice I got from Barry -- Thanks, Barry! is when your
class structure cannot be written the same in both 2 and 3 (because, for
example, you are using metaclasses) then you have to define your
methods outside of a class, store them in a dictionary, and then use
type to create the class. You can look at Enum for an example (his or
mine .

 

[Terry Reedy]

I want to thank you for this package. I have used it when writing test
modules for idlelib modules and aiming for 100% coverage forces me to
really understand the code tested to hit all the conditional lines.

It's been fun dropping the contortions for coverage.py 4.x, though!

One request: ignore "if __name__ == '__main__':" clauses at the end of
files, which cannot be run under coverage.py, so 100% coverage is
reported as 100% instead of 9x%.

 

[Mark Lawrence]

You can do this already with current Coverage: tell Coverage to exclude
<URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
statements, and it won't count them for coverage calculations.

An alternative to Ned's great bit of kit is figleaf. This, Ned's and
many other useful tools are listed here
https://wiki.python.org/moin/PythonTestingToolsTaxonomy

 

[Ethan Furman]

You can do this already with current Coverage: tell Coverage to exclude
<URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
statements, and it won't count them for coverage calculations.

While that's neat (being able to exclude items) is there any reason to
ever count the `if __name__ == '__main__'` clause? Are there any
circumstances where it could run under Coverage? (Apologies if this is
a dumb question, I know nothing about Coverage myself -- but I'm going
to go look it up now.

 

[Terry Reedy]

You can do this already with current Coverage: tell Coverage to exclude
<URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
statements, and it won't count them for coverage calculations.

OK, I added .coveragerc and that works. In the process of verifying
this, I was reminded that there is an overt bug in the html report as
displayed by Firefox. The fonts used for line numbers
("class='linenos'") and line text ("class='text'") are slightly
different and, more importantly, with difference sizes (line numbers are
larger). So corresponding numbers and text do not line up with each
other. This makes the J,K hotkeys and missing branch notations much less
useful than intended. If I use cntl-scrollwheel to change text size,
both change in the same proportion, so the mismatch is maintained.

 

[Ned Batchelder]

While that's neat (being able to exclude items) is there any reason to
ever count the `if __name__ == '__main__'` clause? Are there any
circumstances where it could run under Coverage? (Apologies if this is
a dumb question, I know nothing about Coverage myself -- but I'm going
to go look it up now.

Sure, if that line appears in program.py, then it will be run if you
execute program.py: $ coverage run program.py

You can run coverage a number of times, even with different main
programs, then combine all the data to produce a combined report. This
way you could cover all of the __main__ clauses in a number of files.

--Ned.

 

[Terry Reedy]

OK, I added .coveragerc and that works. In the process of verifying
this, I was reminded that there is an overt bug in the html report as
displayed by Firefox. The fonts used for line numbers
("class='linenos'") and line text ("class='text'") are slightly
different and, more importantly, with difference sizes (line numbers are
larger). So corresponding numbers and text do not line up with each
other. This makes the J,K hotkeys and missing branch notations much less
useful than intended. If I use cntl-scrollwheel to change text size,
both change in the same proportion, so the mismatch is maintained.

I found the style.css file and the linenos entry that works to make
numbers and text line us is, *after correction*:

/* Source file styles */
..linenos p {
text-align: right;
margin: 0;
padding: 0 .5em 0 .5em;
color: #999999;
font-family: verdana, sans-serif;
font-size: .625em; /* 10/16 */
/* line-height: 1.6em; /* 16/10 */
}

Padding was '0 .5em', which I gather is the same as '0 .5em .5em .5em',
which adds padding to both top and bottom, instead of just one of the
two. The corresponding text padding is '0 0 0 .5em'. The extra .5 for
numbers puts padding needed on the right to separate numbers from the
vertical green bars. The commented-out line-height added to mis-alignment.

The verdana numerals *are* larger than those for the inherited page
default (font: 'inherit' at the top of the file). But they do not cause
misalignment.

 

[Oscar Benjamin]

I want to thank you for this package. I have used it when writing test

modules for idlelib modules and aiming for 100% coverage forces me to
really understand the code tested to hit all the conditional lines.

One request: ignore "if __name__ == '__main__':" clauses at the end of

files, which cannot be run under coverage.py, so 100% coverage is reported
as 100% instead of 9x%.

Coverage already does this. You have to put it in your coveragerc.
Personally I arrange to ensure that I am testing that part though. You can
enable coverage for these with the pth file and the environment variable
whose name temporarily escapes me. Or you can just invoke coverage directly
on the script.

Oscar

 

[Metallicow]

+1 for stdev Steven. Thanks for the extra legs.
Hope all goes well with introductions... I'm sure it will.
Good Job.

Well, what I am trying to get at is whether it is better as...

stddev or stdev...? 6(3standard abc) vs 5(2nonstandard abc)
How many people are going to have to change their code...

Is alphabetsoup ABC better than oppsImissedAspoonfull(AC)?

As far as readability and standard naming conventions are concerned?