file data => array(s)

[Eric]

I'm trying to read some file data into a set of arrays. The file data
is just four columns of numbers, like so:

1.2 2.2 3.3 0.5
0.1 0.2 1.0 10.1
... and so on

I'd like to read this into four arrays, one array for each column.
Alternatively, I guess something like this is okay too:

[[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1], ... and so on]

I came up with the following for the four array option:

file = open(fileName, 'r')
for line in file.readlines():
d1, e1, d2, e2 = map(float, line.split())
data1.append(d1) # where data1, err1, data2, err2 are init-ed
as empty lists
err1.append(e1)
data2.append(d2)
err2.append(e2)
file.close()

But somehow it doesn't seem very python-esque (I'm thinking there's a
more elegant and succinct way to do it in python). I've also tried
replacing the above "map" line with:

d = d + map(float, line.split()) # where d is initialized as d
= []

But all I get is one long flat list, not what I want.

So is the map and append method the best I can do or is there a
slicker way?

One more thing, no numpy. Nothing against numpy but I'm curious to
see what can be done with just the box stock python install.

TIA,
eric

 

[Dave Angel]

I'm trying to read some file data into a set of arrays. The file data
is just four columns of numbers, like so:

1.2 2.2 3.3 0.5
0.1 0.2 1.0 10.1
... and so on

I'd like to read this into four arrays, one array for each column.
Alternatively, I guess something like this is okay too:

[[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1], ... and so on]

I came up with the following for the four array option:

file = open(fileName, 'r')
for line in file.readlines():

The readlines() call is a waste of time/space. file is already an
iterator that'll return lines for you.

d1, e1, d2, e2 = map(float, line.split())
data1.append(d1) # where data1, err1, data2, err2 are init-ed
as empty lists
err1.append(e1)
data2.append(d2)
err2.append(e2)
file.close()

But somehow it doesn't seem very python-esque (I'm thinking there's a
more elegant and succinct way to do it in python). I've also tried
replacing the above "map" line with:

d = d + map(float, line.split()) # where d is initialized as d
= []

But all I get is one long flat list, not what I want.

So is the map and append method the best I can do or is there a
slicker way?

One more thing, no numpy. Nothing against numpy but I'm curious to
see what can be done with just the box stock python install.

TIA,
eric

When I see a problem like this, I turn to zip(). It's got some powerful
uses when rows and columns need inverting.

I didn't try it on an actual file, but the following works:
linedata = [[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1] ]

data, err1, data2, err2 = zip(*linedata)

print data
print err1
print data2
print err2

So you could try (untested)

file = open(filename, "r")
linedata = [ map(float, line) for line in file]
data, err1, data2, err2 = zip(*linedata)
file.close()

Note that your code won't work (and mine probably won't either) if one
of the lines has 3 or 5 items. Or if one of the numbers isn't legal
format for a float. So you need to think about error checking, or
decide whether a partial result is important.

 

[Eric]

Note that your code won't work (and mine probably won't either) if one
of the lines has 3 or 5 items.  Or if one of the numbers isn't legal
format for a float.  So you need to think about error checking, or
decide whether a partial result is important.

Haven't tried your suggestion yet, I just wanted to comment on this
last part real quick. I have the same concern, my plan is to wrap all
that stuff up in a "try:" construct. In such cases the program only
has to kick out a simple, sensible (for non-programmers) error message
and quit.

BTW, I didn't say it originally, but this is for 2.7 and hopefully
it'll be easy to carry over to 3.2.

Thanks,
eric

 

[Steven D'Aprano]

I'm trying to read some file data into a set of arrays. The file data
is just four columns of numbers, like so:

1.2 2.2 3.3 0.5
0.1 0.2 1.0 10.1
... and so on

I'd like to read this into four arrays, one array for each column.
Alternatively, I guess something like this is okay too:

[[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1], ... and so on]

First thing: due to the fundamental nature of binary floating point
numbers, if you convert text like "0.1" to a float, you don't get 0.1,
you get 0.10000000000000001. That is because 0.1000...01 is the closest
possible combination of fractions of 1/2, 1/4, 1/8, ... that adds up to
1/10.

If this fact disturbs you, you can import the decimal module and use
decimal.Decimal instead; otherwise forget I said anything and continue
using float. I will assume you're happy with floats.

Assuming the file is small, say, less than 50MB, I'd do it like this:


# Version for Python 2.x
f = open(filename, 'r')
text = f.read() # Grab the whole file at once.
numbers = map(float, text.split())
f.close()

That gives you a single list [1.2, 2.2, 3.3, 0.5, 0.1, 0.2, ...] which
you can now split into groups of four. There are lots of ways to do this.
Here's an inefficient way which hopefully will be simple to understand:

result = []
while numbers != []:
result.append(numbers[0:4])
del numbers[0:4]


Here is a much more efficient method which is only a tad harder to
understand:

result = []
for start in range(0, len(numbers), 4):
result.append(numbers[start:start+4])


And just for completeness, here is an advanced technique using itertools:

n = len(numbers)//4
numbers = iter(numbers)
from itertools import islice
result = [list(islice(numbers, 4)) for i in range(n)]

Be warned that this version throws away any partial group left over at
the end; if you don't want that, change the line defining n to this
instead:

n = len(numbers)//4 + (len(numbers)%4 > 0)

 

[Eric]

I'm trying to read some file data into a set of arrays.  The file data
is just four columns of numbers, like so:

    1.2    2.2   3.3  0.5
    0.1   0.2    1.0  10.1
    ... and so on

I'd like to read this into four arrays, one array for each column.
Alternatively, I guess something like this is okay too:

    [[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1], ... and so on]

I came up with the following for the four array option:

    file = open(fileName, 'r')
    for line in file.readlines():

The readlines() call is a waste of time/space.  file is already an
iterator that'll return lines for you.

       d1, e1, d2, e2 = map(float, line.split())
       data1.append(d1)  # where data1, err1, data2, err2 areinit-ed
as empty lists
       err1.append(e1)
       data2.append(d2)
       err2.append(e2)
    file.close()

But somehow it doesn't seem very python-esque (I'm thinking there's a
more elegant and succinct way to do it in python).  I've also tried
replacing the above "map" line with:

       d = d + map(float, line.split())  # where d is initialized as d
= []

But all I get is one long flat list, not what I want.

So is the map and append method the best I can do or is there a
slicker way?

One more thing, no numpy.  Nothing against numpy but I'm curious to
see what can be done with just the box stock python install.

TIA,
eric

When I see a problem like this, I turn to zip().  It's got some powerful
uses when rows and columns need inverting.

I didn't try it on an actual file, but the following works:
linedata =    [[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1] ]

data, err1, data2, err2 = zip(*linedata)

print data
print err1
print data2
print err2

So you could try (untested)

file = open(filename, "r")
linedata = [ map(float, line) for line in file]
data, err1, data2, err2 = zip(*linedata)
file.close()

DaveA

Neat. This is what I had in mind for a python-esque solution. Only
thing is "map(float,line)" should be "map(float,line.split()). Looks
like it should be easy enough to weed out any funky data sets because
between map() and zip() it's fairly picky about the amount and type of
data.

Finally, the input files I'll be using for real aren't just four
columns of data. The beginning of the file may have comments
(optional) and will have two lines of text to identify the data.
Maybe I can still do it w/o readlines.

Thanks,
eric

 

[Eric]

I'm trying to read some file data into a set of arrays.  The file data
is just four columns of numbers, like so:

   1.2    2.2   3.3  0.5
   0.1   0.2    1.0  10.1
   ... and so on

I'd like to read this into four arrays, one array for each column.
Alternatively, I guess something like this is okay too:

   [[1.2, 2.2, 3.3, 0.5], [0.1, 0.2, 1.0, 10.1], ... and so on]

First thing: due to the fundamental nature of binary floating point
numbers, if you convert text like "0.1" to a float, you don't get 0.1,
you get 0.10000000000000001. That is because 0.1000...01 is the closest
possible combination of fractions of 1/2, 1/4, 1/8, ... that adds up to
1/10.

If this fact disturbs you, you can import the decimal module and use
decimal.Decimal instead; otherwise forget I said anything and continue
using float. I will assume you're happy with floats.

Yeah, I don't think it'll be a problem. As I understand it a float in
python is a double in C and all our old C programs used doubles. From
PDP-11 to MIPS3k to P2 I've seen what I think may have been rounding
or precision errors but I haven't heard any complaints because of
them.

Thanks,
eric

 

[MRAB]

On 15/12/2011 18:37, Eric wrote:
[snip]

Neat. This is what I had in mind for a python-esque solution.

[snip]
FYI, the word is "Pythonic" when talking about the programming
language. The word "Pythonesque" refers to Monty Python.