Python 3.0 slow file IO

[thomasvangurp]

I just recently learned python, I'm using it mainly to process huge
<5GB txt files of ASCII information about DNA. I've decided to learn
3.0, but maybe I need to step back to 2.6?

I'm getting exceedingly frustrated by the slow file IO behaviour of
python 3.0. I know that a bug-report was submitted here:
http://bugs.python.org/issue4533. And a solution was posted.
However, i don't know how to apply this patch. I've searched the
forums and tried:
C:\python30> patch -p0 < fileio_buffer.patch
The patch command is not recognized..

Any help on implementing this patch, or advice on moving back to the
older version is appreciated.
Kind regards,
Thomas

 

[Ulrich Eckhardt]

C:\python30> patch -p0 < fileio_buffer.patch
The patch command is not recognized..

You need the 'patch' program first. Further, you will need a C compiler. If
you don't know how to compile from sources, I would postpone patching
sources to after learning that.

[...] advice on moving back to the older version is appreciated.

Just install the latest 2.6 release. I think you can easily install it
alongside 3.0, all you need to pay attention to is the "PATH" environment
variable.

Uli

 

[rdmurray]

I just recently learned python, I'm using it mainly to process huge
<5GB txt files of ASCII information about DNA. I've decided to learn
3.0, but maybe I need to step back to 2.6?

I'm getting exceedingly frustrated by the slow file IO behaviour of
python 3.0. I know that a bug-report was submitted here:
http://bugs.python.org/issue4533. And a solution was posted.

[snip stuff about patching]

I suggest you stick with Python 2.5 or 2.6. Python 3.0 isn't as mature
as the 2.x series. It's brand new, lot's of things have changed, too.

3.1 will come out much sooner than a normal "next dot" release python
would, and will contain very significant improvements in the IO speed.
In the meantime, in your case at least, to get real work done you'll
be better off using 2.6. You can use the 'from __future__ import' to
bring various 3.0 features into 2.6 and make 2.6 look more like 3.0,
and you can use the -3 flag to check your code for 3.0 compatibility, so
that you will be ready to move back to the 3 series when 3.1 comes out.
(Assuming you don't end up needing any 3rd party libraries that haven't
been ported yet by then, that is.)

Unless, of course, you are comfortable with compiling from source
and would like to help _test_ the 3.1 IO library....

--RDM

 

[thomasvangurp]

Thanks a lot for all the responses. I'll move back to Python 2.5 for
compatibility with SciPY and some other third party packages.
I'll leave the compilation process for some other day, for now I'm a
happy user, mayve In the future I would like to contribute to the
developmental process..