Those two controversial 2nd & 3rd paragraphs of my ch 1

[Alf P. Steinbach]

Referring to <url: http://tinyurl.com/programmingbookP3>

Due to especially Steven D'Aprano's comments I've replaced "hopeless" with "very
hard" in paragraph 1 of section 1.1 -- I know he'll disagree with that also
but I think any more downplaying of the difficulties would be misleading.

According to Mensanator's comments I've changed paragraph 2 in section 1.1 to
point out that the latest version of Python is typically not bundled with an OS.

The two paragraphs now read:


<quote>
As of this writing two main variants of the Python language are in use, namely
Python 2.x and Python 3.x (versions 3.0 and greater). Mostly they’re the same
but the effect of e.g. the / division operator changed in 3.0, so in practice
it’s very hard to create programs that work the same – or even just work –
with both variants. The examples and discussion in this book assume Python
version 3.1.1 or later.

Python implementations, the tools etc. that you need to create and run Python
programs – at least for some earlier version of Python! – are bundled with
many operating systems such as Linux and Mac OS X, but unfortunately not with
Windows.
</quote>


It's said that the hardest part of a book is the opening sentence -- "It was a
dark and stormy night; the rain fell in torrents, except at occasional
intervals, when it was checked by a violent gust of wind which swept up the
streets (for it is in London that our scene lies), rattling along the housetops,
and fiercely agitating the scanty flame of the lamps that struggled against the
darkness" -- but in my case, not 1st sentence but the 2nd and 3rd paragraphs!

Comments welcome.


Cheers,

- Alf

PS: Now more stuff added to ch 3 but I've not updated the separate TOC since
that's a bit of work.

 

[Peter Otten]

As of this writing two main variants of the Python language are in use,
namely Python 2.x and Python 3.x (versions 3.0 and greater). Mostly
they’re the same but the effect of e.g. the / division operator changed in
3.0, so in practice it’s very hard to create programs that work the same
– or even just work – with both variants. The examples and discussion in
this book assume Python version 3.1.1 or later.

It may be hard to support Python 2 and 3 with the same script, but the
division operator is not a good example to support that statement because
you can get the 3.x behaviour in 2.x with a simple

from __future__ import division

at the top of the module.

Peter

 

[Daniel Fetchinson]

As of this writing two main variants of the Python language are in use,

It may be hard to support Python 2 and 3 with the same script, but the
division operator is not a good example to support that statement because
you can get the 3.x behaviour in 2.x with a simple

from __future__ import division

at the top of the module.

I was about to write the same!

Python 2.6.2 (r262:71600, Aug 21 2009, 12:23:57)
[GCC 4.4.1 20090818 (Red Hat 4.4.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Also, I would replace

"in practice it’s very hard to create programs"

with

"in practice it’s very hard to create complex programs"

because for small programs it's very possible to write code that will
work with both python 2 and 3. The question is of course what program
is simple/small and what program is large/complex, but without
qualifications your sentence is misleading, I think.

Cheers,
Daniel

HTH,
Daniel

 

[Stefan Behnel]

Daniel Fetchinson, 13.01.2010 16:23:

Also, I would replace

"in practice it’s very hard to create programs"

with

"in practice it’s very hard to create complex programs"

because for small programs it's very possible to write code that will
work with both python 2 and 3. The question is of course what program
is simple/small and what program is large/complex, but without
qualifications your sentence is misleading, I think.

We had the example of Cheetah over in the other thread. Would you say that
Cheetah doesn't qualify as "complex" program? There are also other
examples, like Django. Often enough it's just a couple of cleanups and tiny
try-except blocks in the program header that enables running the program in
both Py2 and Py3, If it's more, it can usually be done using 2to3. So I
would change the above statement into something more like "for some
programs, especially large existing code bases, it can be hard to get the
code to work in both Python 2 and Python 3". Nevertheless, it has been
done, more than once.

Personally, I don't see much value in deliberately trying to keep people
from porting their code to Py3 by producing underqualified statements like
the above.

Stefan

 

[Daniel Fetchinson]

Also, I would replace

We had the example of Cheetah over in the other thread. Would you say that
Cheetah doesn't qualify as "complex" program?

One code base of cheetah works under python 2 and 3? I doubt it, but I
could be wrong. What I can easily imagine is that somebody ported
cheetah to python 3. In this case there are two code bases, one for
python 2 and another for python 3. So it's not the same program that
runs under python 2 and 3.

What the sentence in Alf's book is about is having the same code base
working for both python 2 and 3.

There are also other
examples, like Django.

Again, django has been ported to python 3, that's fine, everybody
acknowledges that, but it's not the case that one code base works with
both python versions.

Often enough it's just a couple of cleanups and tiny
try-except blocks in the program header that enables running the program in
both Py2 and Py3.

Yes, this is true. But I'd say it's fair to say that with complex
programs this is usually not the case, but I definitely not want to
enter into a discussion into whether any given program is complex or
simple. It's a matter of judgement and gut feeling it's pointless to
argue about this too much.

If it's more, it can usually be done using 2to3.

Again, 2to3 helps with porting, but does not help with having a single
code base that will run unmodified on python 2 and 3, which is what
Alf was writing about.

So I
would change the above statement into something more like "for some
programs, especially large existing code bases, it can be hard to get the
code to work in both Python 2 and Python 3".

I actually agree with this sentence

Nevertheless, it has been done, more than once.

Example? Just to be clear I'm looking for an example where one given
code runs on python 2 and 3 unmodified. I think django and cheetah
doesn't count because they simply take their python 2 code, run it
through 2to3 which gives them a python 3 code (I could be wrong
though). Two codes for the two python versions.

Personally, I don't see much value in deliberately trying to keep people
from porting their code to Py3 by producing underqualified statements like
the above.

Nobody is deliberately trying to keep people from porting! I think you
misunderstand what is being said, these two statements are very
different: (1) single code base working on both python versions (2)
creating a second code from a code so that the second code works with
python 3 and the first one with python 2. Statement (2) is about
porting, statement (1) is about something else.

Having said all that I actually seriously doubt (probably in agreement
with you) that Alf is able to write good and helpful material on the
relationship between python 2 and 3, porting, migrating, etc, based on
his emails

Cheers,
Daniel

 

[André]

One code base of cheetah works under python 2 and 3? I doubt it, but I
could be wrong. What I can easily imagine is that somebody ported
cheetah to python 3. In this case there are two code bases, one for
python 2 and another for python 3. So it's not the same program that
runs under python 2 and 3.

I don't know about Cheetah, but I have read about other programs with
one code base that run under both Python 2 and Python 3. And I can
guarantee that it is the case for Crunchy.

It *is* possible to have one program working correctly under both
Python 2 *and* 3 with a single code base. It might not be the
officially recommended way... but that does not make it impossible.

André

 

[Stefan Behnel]

Daniel Fetchinson, 13.01.2010 17:30:

Again, django has been ported to python 3, that's fine, everybody
acknowledges that, but it's not the case that one code base works with
both python versions.

Well, if the port is done via 2to3, you can install the same code base in
Python 2 and Python 3, and the distutils install mechanism will run an
automated transformation over the code during the installation. If there is
no manual interaction required to run the code on both platforms, I would
say that qualifies as "one code base works with both Python versions". It's
not different from generating code or specialised config files during the
installation or similar things.

> I think django and cheetah
> doesn't count because they simply take their python 2 code, run it
> through 2to3 which gives them a python 3 code (I could be wrong
> though). Two codes for the two python versions.

But just one code base that has to be maintained. And I think the
maintenance is the main aspect here.

Just to be clear I'm looking for an example where one given
code runs on python 2 and 3 unmodified.

lxml for example. Not only the Cython compiled part (which is automatically
portable anyway), also all of its Python code base and its entire test
suite. It runs on all Python versions from 2.3 through 3.1, and it doesn't
use 2to3 or any other kind of code modification.

The regular Python code base was almost trivial to port, but porting the
test suite was actually quite involved. The main reasons for that were a)
doctests and b) the requirement to test exactly string input/output and
exactly unicode input/output on both platforms. Most code simply doesn't
have that requirement, but lxml does.

Stefan

 

[Steve Holden]

One code base of cheetah works under python 2 and 3? I doubt it, but I
could be wrong. What I can easily imagine is that somebody ported
cheetah to python 3. In this case there are two code bases, one for
python 2 and another for python 3. So it's not the same program that
runs under python 2 and 3.

What the sentence in Alf's book is about is having the same code base
working for both python 2 and 3.


Again, django has been ported to python 3, that's fine, everybody
acknowledges that, but it's not the case that one code base works with
both python versions.


Yes, this is true. But I'd say it's fair to say that with complex
programs this is usually not the case, but I definitely not want to
enter into a discussion into whether any given program is complex or
simple. It's a matter of judgement and gut feeling it's pointless to
argue about this too much.


Again, 2to3 helps with porting, but does not help with having a single
code base that will run unmodified on python 2 and 3, which is what
Alf was writing about.


I actually agree with this sentence


Example? Just to be clear I'm looking for an example where one given
code runs on python 2 and 3 unmodified. I think django and cheetah
doesn't count because they simply take their python 2 code, run it
through 2to3 which gives them a python 3 code (I could be wrong
though). Two codes for the two python versions.


Nobody is deliberately trying to keep people from porting! I think you
misunderstand what is being said, these two statements are very
different: (1) single code base working on both python versions (2)
creating a second code from a code so that the second code works with
python 3 and the first one with python 2. Statement (2) is about
porting, statement (1) is about something else.

Having said all that I actually seriously doubt (probably in agreement
with you) that Alf is able to write good and helpful material on the
relationship between python 2 and 3, porting, migrating, etc, based on
his emails

[sigh]

OK, let's review the approved strategy for porting to 3.x from 2.x.

1. Ensure you have an acceptable test suite that verifies all major
functionality.

2. Port your code to the most recent version of 2.x (retaining backward
compatibility with older versions where necessary and practical)

3. Enable Python 3 warnings with the -3 argument and fix all
incompatibilities

4. Run 2to3 on your 2.x code and observe the errors obtained when
running the result under 3.x

5. Fix by paraphrasing your 2.x code to remove translation errors and
return to step 4, repeating until there are no more errors.

In other words, you are not recommended to aim for source compatibility,
you instead aim for a 2.x source which produces a correct 3.x source
when run through the 2to3 translator.

In this way you can continue to produce both 2.x and 3.x versions of
your projects form a single source tree until you no longer care about
2.x, at which point you simply use the output of the final 2to3
conversion as your onward source.

Of course this only works for pure Python code, but that's the majority
of Python code produced.

regards
Steve

 

[Daniel Fetchinson]

One code base of cheetah works under python 2 and 3? I doubt it, but I

I don't know about Cheetah, but I have read about other programs with
one code base that run under both Python 2 and Python 3. And I can
guarantee that it is the case for Crunchy.

It *is* possible to have one program working correctly under both
Python 2 *and* 3 with a single code base. It might not be the
officially recommended way... but that does not make it impossible.

Cool, thanks, I didn't know about crunchy having this property, good to know.

Cheers,
Daniel

 

[Daniel Fetchinson]

Again, django has been ported to python 3, that's fine, everybody

Well, if the port is done via 2to3, you can install the same code base in
Python 2 and Python 3, and the distutils install mechanism will run an
automated transformation over the code during the installation. If there is
no manual interaction required to run the code on both platforms, I would
say that qualifies as "one code base works with both Python versions". It's
not different from generating code or specialised config files during the
installation or similar things.

This is all true, but what we were talking about is not exactly this,
but rather having a single code base running unmodified on python 2
and 3. Another poster actually supplied an example: crunchy.

But just one code base that has to be maintained. And I think the
maintenance is the main aspect here.

This is true again. But where it all started is Alf's sentence in his
book. And I think he has a point there, it might be possible to do
this (single code base for both versions), but it's not easy.

lxml for example. Not only the Cython compiled part (which is automatically
portable anyway), also all of its Python code base and its entire test
suite. It runs on all Python versions from 2.3 through 3.1, and it doesn't
use 2to3 or any other kind of code modification.

The regular Python code base was almost trivial to port, but porting the
test suite was actually quite involved. The main reasons for that were a)
doctests and b) the requirement to test exactly string input/output and
exactly unicode input/output on both platforms. Most code simply doesn't
have that requirement, but lxml does.

Cool, that's another example I was unaware of, thanks!

Cheers,
Daniel

 

[Terry Reedy]

Example? Just to be clear I'm looking for an example where one given
code runs on python 2 and 3 unmodified. I think django and cheetah
doesn't count because they simply take their python 2 code, run it
through 2to3 which gives them a python 3 code (I could be wrong
though). Two codes for the two python versions.

If one only edit the Py2 version and reruns 2to3 for a Py3 version,
whenever desired, there is just *one* code base. This is what the
developers mean when they say one code base for 2.x and 3.x.

One could even arrange for 2to3 to be used as a front end for running
with 3.x, though caching the 3.x version until it is out-of-date is
obviously faster.

 

[Alf P. Steinbach]

* Daniel Fetchinson:

Nobody is deliberately trying to keep people from porting! I think you
misunderstand what is being said, these two statements are very
different: (1) single code base working on both python versions (2)
creating a second code from a code so that the second code works with
python 3 and the first one with python 2. Statement (2) is about
porting, statement (1) is about something else.

Having said all that I actually seriously doubt (probably in agreement
with you) that Alf is able to write good and helpful material on the
relationship between python 2 and 3, porting, migrating, etc, based on
his emails

You're mostly absolutely right.

I think it's OK to write things like "It's possible to travel to the moon" and
"It's very difficult to travel to the moon", because those are just general
observations -- even though an astronaut may disagree with the latter stm.

But in order to write about the design of rockets, planning, economics, the
physics of it, life support, etc., all the nitty gritty details about traveling
to the moon so as to enable someone to do it, one needs to have done it.

And so far I haven't done that 2.x/3.x compatible code thing except for some
very trivial stuff. It so happened that what I wrote in Oct 2009 or whenever it
was, about "/", was verified just now by stumbling upon the "/" bug in
[wave.py]. But that apparent detail was again just a general observation based
on common sense and experience with similar differences in other languages, much
like "you can expect problems if you have a big hole where your rocket's nose
tip should be". To write about actual porting, version-compatible code, etc., so
that it's useful to the reader would or will require far more. I'm probably not
going to do that.

There are however some comments about 2.x/3.x differences, e.g. about "range"
and "/", and I'm just now progressing into territory where I'll mention "long".

And I think that those comments constitute good and helpful material on the
relationship between 2.x and 3.x?


Cheers,

- Alf

 

[Gabriel Genellina]

En Wed, 13 Jan 2010 17:11:18 -0300, Daniel Fetchinson

This is all true, but what we were talking about is not exactly this,
but rather having a single code base running unmodified on python 2
and 3. Another poster actually supplied an example: crunchy.

Having separate code bases for 2.x and 3.x would be a nightmare. I'm still
suffering because of some decision taken years ago here at work: someone
decided to make separate projects per customer, that were essentially the
very same product (with small modifications and additions for each
customer). Not even branches in the version control system: completely
separate projects. That's insane.

So I try hard not to have separate branches for 2.x and 3.x code, and
until now the strategy outlined by Steve Holden has worked fine for us.
Basically, we only consider "source" the 2.x code; the 3.x code is
"derived", not source (like compiled .pyc files or .exe executables) as
isn't even under version control (to avoid the temptation to patch the
generated 3.x code).

The Python 3.1 interpreter doesn't run the very same code as the 2.x one,
but for all practical purposes, we have a single code base for both 2.x
and 3.1

This is true again. But where it all started is Alf's sentence in his
book. And I think he has a point there, it might be possible to do
this (single code base for both versions), but it's not easy.

Some syntax constructs simply aren't compatible, like try/except. And
u"ABC" is invalid in 3.x, but "from __future__ import unicode_literals"
cannot always been used. Generating the 3.x source with 2to3 -in my
experience- is easier than restricting oneself to write code using only
the subset of compatible features, which leads to non-idiomatic code style.
It *can* be done, as some examples show -- but I prefer the 2to3 approach.

 

[Lie Ryan]

* Daniel Fetchinson:

Nobody is deliberately trying to keep people from porting! I think you
misunderstand what is being said, these two statements are very
different: (1) single code base working on both python versions (2)
creating a second code from a code so that the second code works with
python 3 and the first one with python 2. Statement (2) is about
porting, statement (1) is about something else.

Having said all that I actually seriously doubt (probably in agreement
with you) that Alf is able to write good and helpful material on the
relationship between python 2 and 3, porting, migrating, etc, based on
his emails

You're mostly absolutely right.

I think it's OK to write things like "It's possible to travel to the
moon" and "It's very difficult to travel to the moon", because those are
just general observations -- even though an astronaut may disagree
with the latter stm.

But in order to write about the design of rockets, planning, economics,
the physics of it, life support, etc., all the nitty gritty details
about traveling to the moon so as to enable someone to do it, one needs
to have done it.

And so far I haven't done that 2.x/3.x compatible code thing except for
some very trivial stuff. It so happened that what I wrote in Oct 2009 or
whenever it was, about "/", was verified just now by stumbling upon the
"/" bug in [wave.py]. But that apparent detail was again just a general
observation based on common sense and experience with similar
differences in other languages, much like "you can expect problems if
you have a big hole where your rocket's nose tip should be". To write
about actual porting, version-compatible code, etc., so that it's useful
to the reader would or will require far more. I'm probably not going to
do that.

There are however some comments about 2.x/3.x differences, e.g. about
"range" and "/", and I'm just now progressing into territory where I'll
mention "long".

And I think that those comments constitute good and helpful material on
the relationship between 2.x and 3.x?

The GNU Build System have succeeded in producing a program that can
write shell scripts (./configure) that works across any shell
implementation and any shell version on any OS (even on Windows with
Cygwin) generally without the need for manual tweaking.

It is common knowledge that a carefully written python scripts is easily
portable between different OS-es, and since there is only one "shell" to
support[1], why do you think this scenario is harder than what GNU Build
System developers were facing? They have shown that it is "possible" to
write a complex *and* portable shell script; why do you think it is
"impossible" to write a complex and portable python script?

Though keeping everything in one code base may often be difficult and
only of little practical benefit, it is not impossible. Modern version
control systems makes creating a fork not-so-difficult[2]; and
maintaining two or more codebase is much cheaper nowadays than they used
to be.


[1] we're ignoring other python implementations since they haven't
targeted 3.0; but anyway due to the much better defined standard,
alternative implementations doesn't pose much difference except for
occasional lack of the newest feature on many cases
[2] though not, by any definition, "easy"; but at least it's often
easier to fork than to keep things in one code base. Another, better,
alternative is to write a program that transform a human readable code
to portable code; which is what 2to3 is about.

 

[Alf P. Steinbach]

* Lie Ryan -> Alf P. Steinbach:

why do you think it is "impossible" to write a complex and portable
python script?

I don't. You're not quoting me.

Though keeping everything in one code base may often be difficult and
only of little practical benefit, it is not impossible. Modern version
control systems makes creating a fork not-so-difficult[2]; and
maintaining two or more codebase is much cheaper nowadays than they used
to be.
Huh.


[1] we're ignoring other python implementations since they haven't
targeted 3.0; but anyway due to the much better defined standard,
alternative implementations doesn't pose much difference except for
occasional lack of the newest feature on many cases
[2] though not, by any definition, "easy"; but at least it's often
easier to fork than to keep things in one code base. Another, better,
alternative is to write a program that transform a human readable code
to portable code; which is what 2to3 is about.

The upshot for 2.x/3.x compatible code base is to write in 2.x and convert
automatically to 3.x, that is, to write the source code in *one* language.

That's because it's very hard to write code that works directly in both
languages (or language versions, if you prefer).

The paragraph in the book is about why one, in practice, has to choose one
version for one's source code, and why one shouldn't automatically think that
what's learned about that version necessarily applies to the other version.


Cheers & hth.,

- Alf