The real problem with Python 3 - no business case for conversion(was "I strongly dislike Python 3")

[MRAB]

I maintain two projects that have to work from 2.5 to 3.1. On one of
them (~5kloc) we took the separate support route, and on the other
(~30kloc) I decided to keep a single codebase. IME the maintenance
burden on the former is substantially higher than the latter. Is the
difference in difficulty perhaps domain-related, or a result of a
certain style of coding? Could you give us some more details about
what you were working on that caused you to conclude this?

In my work on the regex module I use a single codebase and generate the
sources for Python 2.5-2.7 and for Python 3.1 from it. It works easily
enough for me.

 

[Fuzzyman]

The big danger is Python 2.x becoming abandonware (2.7 being the final
release) before major projects are ported. Using Python 2.x for new
projects is not advisable (at least many will think so), and using 3.x
is not possible. What to do? It's not a helpful situation for Python.

But Python 2.3, 2.4 & 2.5 are *already* abandonware and see *major*
use in many systems and businesses. Python development has always gone
ahead of what *some* people use - and they don't seem to mind that
they're using essentially abandoned versions of Python.

Now that 2.7 is out I *might* be able to persuade my current company
to migrate to 2.6 on the servers, and they're far faster at adopting
tech than many companies I know.

All the best,

Michael Foord

 

[Carl Banks]

The entire fact that 3.x was *designed* to be incompatible should tell
you that supporting 2.x and 3.x with a single code base is a bad idea,
except for the very smallest of projects.  This is the point where a
project should fork and provide two different versions.

Well, I think it could be a reasonable thing to maintain a single
codebase in 2.x and use 2to3 (and/or a custom translator) to translate
to 3.x version for quite a while.

For the humble library I maintain, I plan to release a Python 3
version as soon as a Python 3 version of numpy is released, maintain a
single codebase (translating from 2 version to 3) for awhile, then at
some point fork them and maintain them separately.

Given that I add features about once every 2 years I don't think it'll
be too much of a burden, though.


Carl Banks

 

[Paul Rubin]

The point, one more time with feeling, is that the incompatibilities
between 2.x and 3.x will *increase* over time.

The issue is less the "incompatibilities" than the -backwards-
incompatibilities. Yes, Python 3 may introduce forward
incompatibilities by adding features absent from Python 2. But it will
be possible to maintain a common codebase simply by not using those
features. On the other hand, the door appears closed for Python 3
adding more stuff that breaks Python 2 code.

 

[geremy condra]

The point, one more time with feeling, is that the incompatibilities
between 2.x and 3.x will *increase* over time.

....and? I don't get to use features from 2.7, why would I expect to
use features from 3.3?

If you now have a code base that is relatively easy to maintain for both
Python 2.x and 3.x, that is a result of much back-porting efforts and of
a new-feature moratorium that is currently in effect. Enjoy that
situation as you may, because it is guaranteed not to last.

I have to target the oldest version of Python I want to support. New
features are irrelevant. I'm not sure why I should need to explain
that to you.

Indeed, the feature moratorium is designed in part to help slow-moving
codebases migrate to Python 3.x before Python resumes its normal pace of
change again. If you're choosing to use that time to further entrench
codebases for Python 2.x, I think that's a short-sighted choice.

I welcome the day that I can stop supporting 2.x. Until then, I have to
support both and your argument is irrelevant.

Python 2.7 is the last 2.x, no further 3.x features will be back-ported.
New 3.x features will begin to appear after the moratorium ends. The
combination of those two means that *the single-codebase burden will
only increase over time* as Python 3.x diverges further from what Python
2.x can support.

See above.

Geremy Condra

 

[David Cournapeau]

One thing that would be very useful is how to maintain something that
works on 2.x and 3.x, but not limiting yourself to 2.6. Giving up
versions below 2.6 is out of the question for most projects with a
significant userbase IMHO. As such, the idea of running the python 3
warnings is not so useful IMHO - unless it could be made to work
better for python 2.x < 2.6, but I am not sure the idea even makes
sense.

This is exactly how I felt about my support for pyparsing, that I was
trying continue to provide support for 2.3 users, up through 3.x
users, with a single code base.  (This would actually have been
possible if I had been willing to introduce a performance penalty for
Python 2 users, but performance is such a critical issue for parsing I
couldn't justify it to myself.)  This meant that I had to constrain my
implementation, while trying to incorporate forward-looking support
features (such as __bool__ and __dir__), which have no effect on older
Python versions, but support additions in newer Pythons.  I just
couldn't get through on the python-dev list that I couldn't just
upgrade my code to 2.6 and then use 2to3 to keep in step across the
2-3 chasm, as this would leave behind my faithful pre-2.6 users.

Here are some of the methods I used:

- No use of sets.  Instead I defined a very simple set simulation
using dict keys, which could be interchanged with set for later
versions.

- No generator expressions, only list comprehensions.

- No use of decorators.  BUT, pyparsing includes a decorator method,
traceParseAction, which can be used by users with later Pythons as
@traceParseAction in their own code.

- No print statements.  As pyparsing is intended to be an internal
module, it does no I/O as part of its function - it only processes a
given string, and returns a data structure.

- Python 2-3 compatible exception syntax.  This may have been my
trickiest step.  The change of syntax for except from

    except ExceptionType, ex:

to:

    except ExceptionType as ex:

is completely forward and backward incompatible.  The workaround is to
rewrite as:

    except ExceptionType:
        ex = sys.exc_info()[0]

which works just fine in 2.x and 3.x.  However, there is a slight
performance penalty in doing this, and pyparsing uses exceptions as
part of its grammar success/failure signalling and backtracking; I've
used this technique everywhere I can get away with it, but there is
one critical spot where I can't use it, so I have to keep 2 code bases
with slight differences between them.

- Implement __bool__, followed by __nonzero__ = __bool__.  This will
give you boolean support for your classes in 2.3-3.1.

- Implement __dir__, which is unused by old Pythons, but supports
customization of dir() output for your own classes.

- Implement __len__, __contains__, __iter__ and __reversed__ for
container classes.

- No ternary expressions.  Not too difficult really, there are several
well-known workarounds for this, either by careful use of and's and
or's, or using the bool-as-int to return the value from
(falseValue,trueValue)[condition].

- Define a version-sensitive portion of your module, to define
synonyms for constants that changed name between versions.  Something
like:

    _PY3K = sys.version_info[0] > 2
    if _PY3K:
        _MAX_INT = sys.maxsize
        basestring = str
        _str2dict = set
        alphas = string.ascii_lowercase + string.ascii_uppercase
    else:
        _MAX_INT = sys.maxint
        range = xrange
        _str2dict = lambda strg : dict( [(c,0) for c in strg] )
        alphas = string.lowercase + string.uppercase

The main body of my code uses range throughout (for example), and with
this definition I get the iterator behavior of xrange regardless of
Python version.

In the end I still have 2 source files, one for Py2 and one for Py3,
but there is only a small and manageable number of differences between
them, and I expect at some point I will move forward to supporting Py3
as my primary target version.  But personally I think this overall
Python 2-3 migration process is moving along at a decent rate, and I
should be able to make my switchover in another 12-18 months.  But in
the meantime, I am still able to support all versions of Python NOW,
and I plan to continue doing so (albeit "support" for 2.x versions
will eventually mean "continue to offer a frozen feature set, with
minimal bug-fixing if any").

I realize that pyparsing is a simple-minded module in comparison to
others: it is pure Python, so it has no issues with C extensions; it
does no I/O, so print-as-statement vs. print-as-function is not an
issue; and it imports few other modules, so the ones it does have not
been dropped in Py3; and overall it is only a few thousand lines of
code.  But I just offer this post as a concrete data point in this
discussion.

Thanks for the helpful post.  However it looks like, other than the
except syntax, all of these are things you're already doing to support
2.3 to 2.6, so it seems suggest supporting 3.1 and 2.6 is maybe a
little more work than supporting 2.3 and 2.6.

We all know that Python is quite successful even with a history of
backward-incompatible changes and feature additions and even minor
paradigm shifts.  What I'm interested in is, what things about the 2
to 3 transition is harder than already exists?

The 2->3 transition is harder because backward incompatibilities are
much more pervasive. In Numpy, adapting the code for python 2.6 was
mostly trivial, for example (removing variables named "as" or "with").
For 3.x, it required much more work.

The question really is one of cost vs benefit: for any new version of
2.x, supporting it was very useful because new python 2.x would become
the default of popular linux distribution, and that's what most casual
windows users would get since it was the default download.

Python 3.x required much more work, for a much smaller perceived
benefit as far as I am concerned (I don't claim to represent anyone
but myself here for numpy transition to 3.x). In particular, assuming
the main argument of python 3.x is a cleaner language, it evaporates
for projects which need to support both 2.x and 3.x projects. Unless
you maintain 2 codebases, this means the project code is less clean,
not cleaner because of compatibility requirements.

As most people I guess, I find discussion about print vs print()
completely pointless - certainly, this is the simplest and most
trivial change to apply to a codebase. But I think python 3.x needs to
bring more tangible advantages to make the cost more bearable. I am
hopeful that python 3.4 or 3.5 will have features/improvements which
will make this worthwhile, but 3.1, not so much,

David

 

[Terry Reedy]

I'm not sure why I don't have this post, but I'm happy to help edit
etc if Martin
wants to put together a rough draft.

Since I wrote that, someone pointed out the the Python How-to collection
includes Porting Extension Modules to 3.0
by Benjamim Peterson. So all Pyilip or Martin need to do is read that
and suggest additions.

Since it is part of the doc set, I presume an issue could be opened on
the tracker.

 

[Terry Reedy]

On 7/7/2010 4:31 AM, Paul McGuire wrote:

[snip interesting report on how Paul suppost pyparsing for 2.3 to 3.1]

Thank you for this.

Do you think such cross-version support would have been easier or harder
if the major changes and deletions in 3.0 has been spread over several
versions, such as 2.5 - 2.7. In other words, suppose the Python 3 idea
never occurred to anyone and

2.5 dropped the old int division and finished the unification of int and
long.

2.6 dropped classic classes and switched range, filter, and map to their
iterator versions.

2.7 made unicode the text type

This is not purely a hypothetical question since the issue of spreading
or bunching changes may arise again in the future.

 

[Paul Rubin]

What gives you that idea? Can you reference a specific statement from
the PYthon developers that says that?

It's just logic. As I understand it, future versions of Python 3 are
supposed to not break programs that work under current versions of
Python 3. So any Python 2 program that is a valid Python 3 program
today has to stay valid.

 

[geremy condra]

[backward-]incompatibilities between 2.x and 3.x will *increase*
over time.

...and? I don't get to use features from 2.7, why would I expect to
use features from 3.3?

Conversely, why would you support Python 3.1?

Because in 10 years the percentage of people who have Python 2.x
installed will be the same as the percentage that have Python 1.x
installed today.

Why do you have to support Python 3 at all?

See above.

Will you expect to continue
maintaining a single code base that supports PYthon 2.x and Python 3.2,
then 3.3, and so on?

Yes, with the occasional feature or bugfix. Is there an alternate
interpretation I'm missing?

The only point being made here is to answer the question of why people
are saying that a single code base for both 2.x and 3.x is a maintenance
burden. If, in your case, at the present time, that's not the case, then
good for you! But it will get increasingly harder to do, and the reasons
why have now been explained. Apply them as you see fit.

I see you stating that it will become harder but I don't see why that
should be the case.

Geremy Condra

 

[Terry Reedy]

The point, one more time with feeling, is that the incompatibilities
between 2.x and 3.x will *increase* over time.

For the purpose of maintaining least-common-denominator multi-version
code, it is only deletions and semantic changes that matter. Feature
additions will be more common, at least for awhile. The problem of
backporting 3.x code that uses 3.x features definitely will increase.

 

[Zooko O'Whielacronx]

Dear Paul McGuire:

Thank you very much for these notes!

See also a few other notes:

Michael Foord: http://www.voidspace.org.uk/python/weblog/arch_d7_2010_03_20.shtml#e1167
Ned Batchelder:
http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html

I was wondering if it would be useful to have a library that
implements these hacks so that people like me who prefer to maintain a
single codebase instead of using a tool like 2to3 could easily adopt
them.

Oh look! Here is one:

http://pybites.blogspot.com/2010/06/six-python-23-compatibility-helpers.html



Regards,

Zooko

 

[Terry Reedy]

Yes, that's what I meant. Python 3 is deliberately under no obligation
to support code that works in Python 2. If something needs fixing, and
that fix would involve breaking Python 2 code, then that's not a
consideration any more.

Code that works in 3.1 *is* 3.1 code. Leaving aside bug fixes and
additions that makes code that once raised an exception do something
instead, I do not know that 3.1 broke and 3.0 code and I do not know of
any deprecations in 3.1 that will become removals in 3.2.

The predictable result is that Python 3 will continue to gain
backward-incompatible changes in future.

For the core syntax, not too likely. In any case, the usually 3 version
pending-deprecation, deprecation, removal process would apply. Some of
the library modules that do not work well for 3.1 will see more changes.

What gives you that idea? Can you reference a specific statement from
the PYthon developers that says that?

3.0 was stated to be a special case. I will let you look.

 

[Martin v. Loewis]

Am 07.07.2010 23:10, schrieb Brendan Abel:

The entire fact that 3.x was *designed* to be incompatible should tell
you that supporting 2.x and 3.x with a single code base is a bad idea,
except for the very smallest of projects. This is the point where a
project should fork and provide two different versions.

I completely disagree. My personal experience is that this is well
possible even for large code bases, and I would recommend having a
single code base for 2.x and 3.x *in particular* for large projects,
which probably need to support 2.x users for quite some time, but
simultaneously need to support 3.x users.

Regards,
Martin

 

[Michele Simionato]

This is a point I do not understand. My recent module plac is meant to
work from Python 2.3 to Python 3.1 and to this goal I make use of 2to3
at the *client* side.
Users of Python 2.X get the original code with no magic whatsoever;
users of Python 3.X
get the same code, but at installation time 2to3 is run by the
setup.py script. The mechanism requires distribute to be installed,
but I would say that having distribute is a must for Python 3.X users;
Python 2.X users do not need anything, so the approach is backward
compatible. I thought this was the recommended way of using 2to3 and
so far is working for me.

M. Simionato

 

[Martin v. Loewis]

Python 3.x will continue to change. The incompatibilities between 3.x

and 2.x will only become more numerous. If your goal is to support
2.x, and 3.x, you'd be best supporting them separately.

I don't think that's a particularly good approach. Having a single code
base for both likely reduced the maintenance burden significantly
compared to a code fork.

Regards,
Martin

 

[Martin v. Loewis]

The point, one more time with feeling, is that the incompatibilities

between 2.x and 3.x will *increase* over time.

I think this is unfounded, and actually false.

Instead, the incompatibilities will *decrease* over the next few years.
Suppose you support 2.x and 3.x from a single code base. Today, you
might support 2.3 up to 3.1. In a year, you might drop 2.3, and start
support 3.2. That will lead to a reduction of incompatibilities: you
can start using 2.4 features, and drop work-arounds for 2.3 limitations.
Likewise when you then, later, drop 2.4.

As a specific example, since 2.5, the modules in the email package are
in lower case, in compliance with PEP 8. 2.x has backwards compatibility
so that you can continue to use the uppercase module names, which you
need for 2.3/2.4 compat. In 3.x, you can't use these spelling anymore,
so your code might use try/except as a work-around. When you drop 2.4,
you can drop the work-around.

If you now have a code base that is relatively easy to maintain for both
Python 2.x and 3.x, that is a result of much back-porting efforts and of
a new-feature moratorium that is currently in effect. Enjoy that
situation as you may, because it is guaranteed not to last.

On the contrary: it's getting better. The back-porting efforts only
become available in 2.6, and you would have to support at least 2.5
today (at least, that's how many maintainers feel). Only when you
can drop 2.5 support, you can actually start *using* these backport
features, allowing you to make the 2.x and 3.x code more similar.

Eventually, the difference will get larger again. Hopefully, by that
time, 2.7 got out of use.

Regards,
Martin

 

[Martin v. Loewis]

I just

couldn't get through on the python-dev list that I couldn't just
upgrade my code to 2.6 and then use 2to3 to keep in step across the
2-3 chasm, as this would leave behind my faithful pre-2.6 users.

Not sure whom you had been talking to. But I would have tried to explain
that you don't *have* to port to 2.6 to use 2to3 - it will
work just as fine with 2.3 code, and earlier.

- No use of sets. Instead I defined a very simple set simulation
using dict keys, which could be interchanged with set for later
versions.

This I don't understand. IIUC, you want to support 2.3 and later.
Now, 2.3 already has a set module, even though set is not a builtin.
So you could use sets just as well.

- No generator expressions, only list comprehensions.

Ok. Not sure how this causes problems, though - just don't use them, then.

- No use of decorators. BUT, pyparsing includes a decorator method,
traceParseAction, which can be used by users with later Pythons as
@traceParseAction in their own code.

Of course, users of older Python versions could explicitly wrap
the functions with the decorator if they wanted to.

- No print statements. As pyparsing is intended to be an internal
module, it does no I/O as part of its function - it only processes a
given string, and returns a data structure.

If you don't need them, fine. If you do, I'd just let 2to3 transform them.

- Python 2-3 compatible exception syntax. This may have been my
trickiest step. The change of syntax for except from

except ExceptionType, ex:

to:

except ExceptionType as ex:

is completely forward and backward incompatible. The workaround is to
rewrite as:

except ExceptionType:
ex = sys.exc_info()[0]

Likewise, and more importantly so: use 2to3. It can be done this way,
but I find the 2to3 solution much cleaner.

But in
the meantime, I am still able to support all versions of Python NOW,
and I plan to continue doing so (albeit "support" for 2.x versions
will eventually mean "continue to offer a frozen feature set, with
minimal bug-fixing if any").

The same would have been possible if you had chosen to use 2to3.

Regards,
Martin

 

[Steven D'Aprano]

The entire fact that 3.x was *designed* to be incompatible should tell
you that supporting 2.x and 3.x with a single code base is a bad idea,
except for the very smallest of projects.

I don't see that follows at all. If the incompatibilities are designed to
be mechanically translatable (e.g. "x.keys()" -> "list(x.keys())" then
you can start with a single code-base and run the translator.

If only there were some sort of Python program that could be used to
translate code from 2 to 3...

*wink*

 

[Philip Semanchuk]

Since I wrote that, someone pointed out the the Python How-to
collection includes Porting Extension Modules to 3.0
by Benjamim Peterson. So all Pyilip or Martin need to do is read
that and suggest additions.

That document is here:
http://wiki.python.org/moin/PortingExtensionModulesToPy3k

It took me a while to find. It's a shame it's not better known; I
looked for such a document before I started porting sysv_ipc and
posix_ipc and didn't find this one.

Thanks for the pointer.

Cheers
Philip