[newbie] confusion concerning fetching an element in a 2d-array

[Jean Dubois]

I'm confused by the behaviour of the following python-script I wrote:

#!/usr/bin/env python
#I first made a data file 'test.dat' with the following content
#1.0 2 3
#4 5 6.0
#7 8 9
import numpy as np
lines=[line.strip() for line in open('test.dat')]
#convert lines-list to numpy-array
array_lines=np.array(lines)
#fetch element at 2nd row, 2nd column:
print array_lines[1, 1]


When running the script I always get the following error:
IndexError: invalid index

Can anyone here explain me what I am doing wrong and how to fix it?

thanks in advance
jean

 

[Steven D'Aprano]

I'm confused by the behaviour of the following python-script I wrote:

#!/usr/bin/env python
#I first made a data file 'test.dat' with the following content
#1.0 2 3
#4 5 6.0
#7 8 9
import numpy as np
lines=[line.strip() for line in open('test.dat')]
#convert lines-list to numpy-array
array_lines=np.array(lines)
#fetch element at 2nd row, 2nd column:
print array_lines[1, 1]


When running the script I always get the following error: IndexError:
invalid index

Can anyone here explain me what I am doing wrong and how to fix it?

Yes. Inspect the array by printing it, and you'll see that it is a one-
dimensional array, not two, and the entries are strings:


py> import numpy as np
py> # simulate a text file
.... data = """1.0 2 3
.... 4 5 6.0
.... 7 8 9"""
py> lines=[line.strip() for line in data.split('\n')]
py> # convert lines-list to numpy-array
.... array_lines = np.array(lines)
py> print array_lines
['1.0 2 3' '4 5 6.0' '7 8 9']


The interactive interpreter is your friend! You never need to guess what
the problem is, Python has powerful introspection abilities, one of the
most powerful is also one of the simplest: print. Another powerful tool
in the interactive interpreter is help().

So, what to do about it? Firstly, convert your string read from a file
into numbers, then build your array. Here's one way:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]


There may be other ways to do this, but that works for me.

 

[Jean Dubois]

Op dinsdag 25 maart 2014 12:01:37 UTC+1 schreef Steven D'Aprano:

I'm confused by the behaviour of the following python-script I wrote:

#!/usr/bin/env python
#I first made a data file 'test.dat' with the following content
#1.0 2 3
#4 5 6.0
#7 8 9
import numpy as np
lines=[line.strip() for line in open('test.dat')]
#convert lines-list to numpy-array
array_lines=np.array(lines)
#fetch element at 2nd row, 2nd column:
print array_lines[1, 1]


When running the script I always get the following error: IndexError:
invalid index

Can anyone here explain me what I am doing wrong and how to fix it?

Yes. Inspect the array by printing it, and you'll see that it is a one-
dimensional array, not two, and the entries are strings:


py> import numpy as np
py> # simulate a text file
... data = """1.0 2 3
... 4 5 6.0
... 7 8 9"""
py> lines=[line.strip() for line in data.split('\n')]
py> # convert lines-list to numpy-array
... array_lines = np.array(lines)
py> print array_lines
['1.0 2 3' '4 5 6.0' '7 8 9']


The interactive interpreter is your friend! You never need to guess what
the problem is, Python has powerful introspection abilities, one of the
most powerful is also one of the simplest: print. Another powerful tool
in the interactive interpreter is help().

So, what to do about it? Firstly, convert your string read from a file
into numbers, then build your array. Here's one way:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]

Dear Steve,
Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't
assemble together to one working program (I really tried hard).
Can I tell from your comment I shouldn't use numpy?
I also don't see how to get the value an element specified by (row,
column) from a numpy_array like "array_lines" in my original code

All I need is a little python-example reading a file with e.g. three lines
with three numbers per line and putting those numbers as floats in a
3x3-numpy_array, then selecting an element from that numpy_array using
it's row and column-number.

thanks in advance and kind regards,
jean

 

[Dave Angel]

Op dinsdag 25 maart 2014 12:01:37 UTC+1 schreef Steven D'Aprano:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]

Dear Steve,
Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't
assemble together to one working program (I really tried hard).
Can I tell from your comment I shouldn't use numpy?
I also don't see how to get the value an element specified by (row,
column) from a numpy_array like "array_lines" in my original code

I don't use numpy, but I thought Steven's description was clear
enough.

Your problem was not the extraction, but the creation of the
array. Use print to prove that to yourself.

All I need is a little python-example reading a file with e.g. three lines
with three numbers per line and putting those numbers as floats in a
3x3-numpy_array, then selecting an element from that numpy_array using
it's row and column-number.

If your instructor wanted you to copy examples, he would have
given you one.

First write some code to split and convert each line into floats.
You don't even try that in your original. Then do that in a
loop. Now you have all the floats in one list or array. I presume
that it's two dimensional. Use print to check. If it's not,
you'll have to either post process it with a reshape method, or
change the way you accumulate it. I can't help with the latter.

 

[Steven D'Aprano]

On Tue, 25 Mar 2014 06:47:23 -0700, Jean Dubois wrote:

[...]

Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't assemble
together to one working program (I really tried hard). Can I tell from
your comment I shouldn't use numpy?

No, you misunderstood me. I never suggested that you avoid numpy. If you
read my code, I use numpy. That's the "import numpy as np", and then
later I refer to np.

What I suggested is that you open up the Python interactive interpreter
and use it for experimentation. Not that you avoid numpy! You will use
numpy inside the interactive interpreter.

Do you know how to start the interactive interpreter? You seem to be
using Linux, or maybe Mac. Open a console or xterm window, and type

python

then press Enter, and the interactive interpreter will start.

I also don't see how to get the
value an element specified by (row, column) from a numpy_array like
"array_lines" in my original code

Once you build the array, you get the element the same way you tried (but
failed) earlier:

print array_lines[1, 1]


will print the element at row 1, column 1. The problem you had before was
that you had a one dimensional array, not a two dimensional one.

All I need is a little python-example reading a file with e.g. three
lines with three numbers per line and putting those numbers as floats
in a 3x3-numpy_array, then selecting an element from that numpy_array
using it's row and column-number.

Yes, I'm sure you do. And you will learn much more by writing this code
yourself.

You were very, very close with your earlier attempt. In English, you had:

* import the numpy module
* read the file into a list of text lines
* store the text lines in a numpy array
* try to print a single item from the array

but it failed because the list of lines was only one dimensional. I gave
you some code that was very similar:

* import the numpy module
* fake a text file using a string
* split the string into a list of substrings
* convert each substring into a number (float)
* store the numbers in a numpy array
* change the size of the array to turn it from 1D to 2D
* print the array


Can you identify which part of code goes with each English description?
That's your first job. Then identify the parts of code you want to take
from my example and put it into your example.

Good luck, and have fun!

 

[Peter Otten]

Op dinsdag 25 maart 2014 12:01:37 UTC+1 schreef Steven D'Aprano:

I'm confused by the behaviour of the following python-script I wrote:

#!/usr/bin/env python
#I first made a data file 'test.dat' with the following content
#1.0 2 3
#4 5 6.0
#7 8 9
import numpy as np
lines=[line.strip() for line in open('test.dat')]
#convert lines-list to numpy-array
array_lines=np.array(lines)
#fetch element at 2nd row, 2nd column:
print array_lines[1, 1]


When running the script I always get the following error: IndexError:
invalid index

Can anyone here explain me what I am doing wrong and how to fix it?

Yes. Inspect the array by printing it, and you'll see that it is a one-
dimensional array, not two, and the entries are strings:


py> import numpy as np
py> # simulate a text file
... data = """1.0 2 3
... 4 5 6.0
... 7 8 9"""
py> lines=[line.strip() for line in data.split('\n')]
py> # convert lines-list to numpy-array
... array_lines = np.array(lines)
py> print array_lines
['1.0 2 3' '4 5 6.0' '7 8 9']


The interactive interpreter is your friend! You never need to guess what
the problem is, Python has powerful introspection abilities, one of the
most powerful is also one of the simplest: print. Another powerful tool
in the interactive interpreter is help().

So, what to do about it? Firstly, convert your string read from a file
into numbers, then build your array. Here's one way:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]

Dear Steve,
Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't
assemble together to one working program (I really tried hard).
Can I tell from your comment I shouldn't use numpy?
I also don't see how to get the value an element specified by (row,
column) from a numpy_array like "array_lines" in my original code

All I need is a little python-example reading a file with e.g. three lines
with three numbers per line and putting those numbers as floats in a
3x3-numpy_array, then selecting an element from that numpy_array using
it's row and column-number.

I'll try, too, but be warned that I'm using the same methology as Steven.
Try to replicate every step in the following exploration.

First let's make sure we start with the same data:

$ cat test.dat
1.0 2 3
4 5 6.0
7 8 9

Then fire up the interactve interpreter:

$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import numpy
lines = [line.strip() for line in open("test.dat")]
lines

['1.0 2 3', '4 5 6.0', '7 8 9']

As you can see lines is a list of three strings.
Let's break these strings into parts:

cells = [line.split() for line in lines]
cells

[['1.0', '2', '3'], ['4', '5', '6.0'], ['7', '8', '9']]

We now have a list of lists of strings and you can address individual items
with

cells[1][2]

'6.0'

What happens when pass this list of lists of strings to the numpy.array()
constructor?
array([['1.0', '2', '3'],
['4', '5', '6.0'],
['7', '8', '9']],
dtype='|S3')

a[1,2]

'6.0'

It sort of works, but the array entries are strings rather than floating
point numbers. Let's fix that:
array([[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 9.]])

a[1,2]

6.0

OK, now we can put the previous steps into a script:

$ cat tmp.py
import numpy
cells = [line.split() for line in open("test.dat")]
a = numpy.array(cells, dtype=float)
print a[1, 2]

Run it:
$ python tmp.py
6.0

Seems to work. But reading a 2D array from a file really looks like a common
task -- there should be a library function for that:

$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.array([[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 9.]])

 

[Mark H Harris]

If your instructor wanted you to copy examples, he would have
given you one.

{ouch}


Give them a toad they'll have warts all day; teach them to eat a toad
they'll never have warts again... or something like that.

marcus

 

[Mark H Harris]

If your instructor wanted you to copy examples, he would have
given you one.

{ouch}


Give them a toad they'll have warts all day; teach them to eat a toad
they'll never have warts again... or something like that.

marcus

 

[Jean Dubois]

Op dinsdag 25 maart 2014 17:12:12 UTC+1 schreef Peter Otten:

Op dinsdag 25 maart 2014 12:01:37 UTC+1 schreef Steven D'Aprano:

On Tue, 25 Mar 2014 03:26:26 -0700, Jean Dubois wrote:

I'm confused by the behaviour of the following python-script I wrote:

#!/usr/bin/env python
#I first made a data file 'test.dat' with the following content
#1.0 2 3
#4 5 6.0
#7 8 9
import numpy as np
lines=[line.strip() for line in open('test.dat')]
#convert lines-list to numpy-array
array_lines=np.array(lines)
#fetch element at 2nd row, 2nd column:
print array_lines[1, 1]


When running the script I always get the following error: IndexError:
invalid index

Can anyone here explain me what I am doing wrong and how to fix it?

Yes. Inspect the array by printing it, and you'll see that it is a one-
dimensional array, not two, and the entries are strings:


py> import numpy as np
py> # simulate a text file
... data = """1.0 2 3
... 4 5 6.0
... 7 8 9"""
py> lines=[line.strip() for line in data.split('\n')]
py> # convert lines-list to numpy-array
... array_lines = np.array(lines)
py> print array_lines
['1.0 2 3' '4 5 6.0' '7 8 9']


The interactive interpreter is your friend! You never need to guess what
the problem is, Python has powerful introspection abilities, one of the
most powerful is also one of the simplest: print. Another powerful tool
in the interactive interpreter is help().

So, what to do about it? Firstly, convert your string read from a file
into numbers, then build your array. Here's one way:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]

Dear Steve,
Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't
assemble together to one working program (I really tried hard).
Can I tell from your comment I shouldn't use numpy?
I also don't see how to get the value an element specified by (row,
column) from a numpy_array like "array_lines" in my original code

All I need is a little python-example reading a file with e.g. three lines
with three numbers per line and putting those numbers as floats in a
3x3-numpy_array, then selecting an element from that numpy_array using
it's row and column-number.

I'll try, too, but be warned that I'm using the same methology as Steven.
Try to replicate every step in the following exploration.
First let's make sure we start with the same data:
$ cat test.dat
1.0 2 3
4 5 6.0
7 8 9
Then fire up the interactve interpreter:
$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import numpy
lines = [line.strip() for line in open("test.dat")]
lines

['1.0 2 3', '4 5 6.0', '7 8 9']
As you can see lines is a list of three strings.
Let's break these strings into parts:

cells = [line.split() for line in lines]
cells

[['1.0', '2', '3'], ['4', '5', '6.0'], ['7', '8', '9']]
We now have a list of lists of strings and you can address individual items
with

cells[1][2]

'6.0'
What happens when pass this list of lists of strings to the numpy.array()
constructor?array([['1.0', '2', '3'],
['4', '5', '6.0'],
['7', '8', '9']],
dtype='|S3')

a[1,2]

'6.0'
It sort of works, but the array entries are strings rather than floating
point numbers. Let's fix that:array([[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 9.]])

a[1,2]

6.0
OK, now we can put the previous steps into a script:
$ cat tmp.py
import numpy
cells = [line.split() for line in open("test.dat")]
a = numpy.array(cells, dtype=float)
print a[1, 2]
Run it:
$ python tmp.py
6.0
Seems to work. But reading a 2D array from a file really looks like a common
task -- there should be a library function for that:
$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.array([[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 9.]])

Thank you very much Peter for this great lesson, you made a lot of things
clear to me.

sincerely
jean

 

[Jean Dubois]

Op dinsdag 25 maart 2014 15:42:13 UTC+1 schreef Dave Angel:

Op dinsdag 25 maart 2014 12:01:37 UTC+1 schreef Steven D'Aprano:

py> values = [float(s) for s in data.split()]
py> print values
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
py> array_lines = np.array(values)
py> array_lines = array_lines.reshape(3, 3)
py> print array_lines
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. 9.]]

Dear Steve,
Thanks for answering my question but unfortunately now I'm totally
confused.
Above I see parts from different programs which I can't
assemble together to one working program (I really tried hard).
Can I tell from your comment I shouldn't use numpy?
I also don't see how to get the value an element specified by (row,
column) from a numpy_array like "array_lines" in my original code

I don't use numpy, but I thought Steven's description was clear
enough.
Your problem was not the extraction, but the creation of the
array. Use print to prove that to yourself.

All I need is a little python-example reading a file with e.g. three lines
with three numbers per line and putting those numbers as floats in a
3x3-numpy_array, then selecting an element from that numpy_array using
it's row and column-number.

If your instructor wanted you to copy examples, he would have
given you one.

please Dave leave that belittling tone behind, there's no instructor
whatsoever involved here. I constructed this example myself as I know very
well I have to start with little pieces of code first to be able to master
larger problems later. I just wanted to figure this example out first,
and as I now learned from Peter's marvellous explanation there _is_ an
elegant solution in Python to this kind of problem.
So if you are irritated by newbie-questions in the future, just neglect
them

thanks
jean

 

[Joel Goldstick]

Jean, be aware there is also python tutor list you might like. This is
sometimes a tough crowd here. Don't be discouraged. It can be a badge of
honor sometimes

 

[Dave Angel]

Op dinsdag 25 maart 2014 15:42:13 UTC+1 schreef Dave Angel:
please Dave leave that belittling tone behind, there's no instructor
whatsoever involved here.

It wasn't my intention to belittle you; I'm sorry. But we do get a
lot of people here asking for homework help, and not saying
that's what it is. And when it's homework we do better explaining
than just writing the code.

In fact you only needed to add a line like
data= infile.read ()
to Steven's original to get a complete working example.

I constructed this example myself as I know very
well I have to start with little pieces of code first to be able to master
larger problems later. I just wanted to figure this example out first,
and as I now learned from Peter's marvellous explanation there _is_ an
elegant solution in Python to this kind of problem.
So if you are irritated by newbie-questions in the future, just neglect
them

Not irritated, just temporarily misguided.

 

[Jean Dubois]

Op dinsdag 25 maart 2014 20:58:10 UTC+1 schreef Dave Angel:

It wasn't my intention to belittle you; I'm sorry. But we do get a
lot of people here asking for homework help, and not saying
that's what it is. And when it's homework we do better explaining
than just writing the code.

In fact you only needed to add a line like
data= infile.read ()
to Steven's original to get a complete working example.


Not irritated, just temporarily misguided.

OK, no hard feelings

kind regards,
jean

 

[Jean Dubois]

Op dinsdag 25 maart 2014 20:15:27 UTC+1 schreef Joel Goldstick:

Jean, be aware there is also python tutor list you might like.  This issometimes a tough crowd here. Don't be discouraged. It can be a badge of honor sometimes

thanks for the suggestions, I already subscribed to the python tutor list

kind regards,
jean