Python arrays and sting formatting options

[Ivan Reborin]

Hello everyone,

I was wondering if anyone here has a moment of time to help me with 2
things that have been bugging me.

1. Multi dimensional arrays - how do you load them in python
For example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
-------
with "i" being the row number, "j" the column number, and "k" the ..
uhmm, well, the "group" number, how would you load this ?

If fortran90 you would just do:

do 10 k=1,2
do 20 i=1,3

read(*,*)(a(i,j,k),j=1,3)

20 continue
10 continue

How would the python equivalent go ?

2. I've read the help on the next one but I just find it difficult
understanding it.
I have;
a=2.000001
b=123456.789
c=1234.0001

How do you print them with the same number of decimals ?
(eg. 2.000, 123456.789, 1234.000)
and how do you print them with the same number of significant
decimals?
(eg. 2.000001, 123456.7, 1234.000 - always 8 decimals) ?


Is something like this possible (built-in) in python ?

Really grateful for all the help and time you can spare.

 

[Mensanator]

Hello everyone,

I was wondering if anyone here has a moment of time to help me with 2
things that have been bugging me.

1. Multi dimensional arrays - how do you load them in python
For example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
-------
with "i" being the row number, "j" the column number, and "k" the ..
uhmm, well, the "group" number, how would you load this ?

If fortran90 you would just do:

do 10 k=1,2
do 20 i=1,3

read(*,*)(a(i,j,k),j=1,3)

20 continue
10 continue

How would the python equivalent go ?

2. I've read the help on the next one but I just find it difficult
understanding it.
I have;
a=2.000001
b=123456.789
c=1234.0001

How do you print them with the same number of decimals ?
(eg. 2.000, 123456.789, 1234.000) 1234.000


and how do you print them with the same number of significant
decimals?
(eg. 2.000001, 123456.7, 1234.000 - always 8 decimals) ?

Your examples are 7 decimals (and you're not rounding).

Here's what 8 looks like (note that it's %0.7e because there
is always one digit to the left of the decimal point.)
1.2340001e+03

If you actually meant 7, then use %0.6e:

1.234000e+03



Is something like this possible (built-in) in python ?

You can do more with gmpy.

 

[Ivan Reborin]

Hello Mensanator, thank you for answering in such a short time.

If you actually meant 7, then use %0.6e:

Sorry about that; I have the habit of counting the point as a decimal
place too.

1.234000e+03

I understood the above from help, but it's not what's been bugging me.
Mea culpa, I've defined the question in a confusing way, I see that
now. What I've meant to ask was, when I have 3 numbers, how would you
print them with the same format which would apply to them 3 numbers.

for example, I have
print a,b,c

now if I print them with
print '%12.3f' %a,b,c
the format will apply only to a, and not to b and c. I could of course
write
print '%12.3f %12.3f ... 3 times
but that is just unpractical.

Is there a way to just do something like this (not normal syntax, just
my wishful thinking):
print 3*'%12.3f' %a,b,c
(meaning - use this format for the next 3 real numbers that come
along)

 

[bearophileHUGS]

Ivan Reborin:

Is there a way to just do something like this (not normal syntax, just
my wishful thinking):
print 3*'%12.3f' %a,b,c
(meaning - use this format for the next 3 real numbers that come
along)

The Python genie grants you that wish. You were almost right:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string

(Note the spaces and parentheses. Python programmers thank you if put
them improving readability a little).

Bye,
bearophile

 

[Ivan Reborin]

Hello bearophile, thank you for replying.

The Python genie grants you that wish. You were almost right:
2.000 123456.789 1234.000

Works beautifully Thank you!

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string

Just one more question - it's actually an extension to this one
(forgive my curiosity, but I really need this info, and searching
google always gives me the same stuff again and again) ...

a = 2.000001
b = 123456.789
c = 1234.0001
d = 98765.4321
# same as above except for d

print (3 * '%12.3f') % (a, b, c)
#this works beautifully

How to add d at the end but with a different format now, since I've
"used" the "format part" ?

Again, my weird wishful-thinking code:
print (3*'%12.3f', '%5.3f') %(a,b,c),d

(Note the spaces and parentheses. Python programmers thank you if put
them improving readability a little).

Yes, ok. I can agree with that - separating the format from the
variable list part sounds reasonable.

 

[Chris Rebert]

Hello bearophile, thank you for replying.

Works beautifully Thank you!


Just one more question - it's actually an extension to this one
(forgive my curiosity, but I really need this info, and searching
google always gives me the same stuff again and again) ...

a = 2.000001
b = 123456.789
c = 1234.0001
d = 98765.4321
# same as above except for d

print (3 * '%12.3f') % (a, b, c)
#this works beautifully

How to add d at the end but with a different format now, since I've
"used" the "format part" ?

Again, my weird wishful-thinking code:
print (3*'%12.3f', '%5.3f') %(a,b,c),d

Again, very close to the correct code:

print (3*'%12.3f' + '%5.3f') %(a,b,c,d)

Regards,
Chris

 

[Marc 'BlackJack' Rintsch]

a = 2.000001
b = 123456.789
c = 1234.0001
d = 98765.4321
# same as above except for d

print (3 * '%12.3f') % (a, b, c)
#this works beautifully

How to add d at the end but with a different format now, since I've
"used" the "format part" ?

Again, my weird wishful-thinking code: print (3*'%12.3f', '%5.3f')
%(a,b,c),d

Maybe you should stop that wishful thinking and programming by accident
and start actually thinking about what the code does, then it's easy to
construct something working yourself.

The ``%`` operator on strings expects a string on the left with format
strings in it and a tuple with objects to replace the format strings
with. So you want

'%12.3f%12.3f%12.3f%5.3f' % (a, b, c, d)

But without repeating the '%12.3f' literally. So you must construct that
string dynamically by repeating the '%12.3f' and adding the '%5.3f':

In [27]: 3 * '%12.3f'
Out[27]: '%12.3f%12.3f%12.3f'

In [28]: 3 * '%12.3f' + '%5.3f'
Out[28]: '%12.3f%12.3f%12.3f%5.3f'

Now you can use the ``%`` operator on that string:

In [29]: (3 * '%12.3f' + '%5.3f') % (a, b, c, d)
Out[29]: ' 2.000 123456.789 1234.00098765.432'

(I guess there should be at least a space before the last format string.)

This time you *have* to put parenthesis around the construction of the
format string BTW because ``%`` has a higher priority than ``+``. So
implicit parentheses look like this:

3 * '%12.3f' + '%5.3f' % (a, b, c, d)
<=> 3 * '%12.3f' + ('%5.3f' % (a, b, c, d))

And there are of course not enough formatting place holders for four
objects in '%5.3f'.

It's also important to learn why your wrong codes fail. In your wishful
thinking example you will get a `TypeError` saying "unsupported operand
type(s) for %: 'tuple' and 'tuple'". That's because on the left side of
the ``%`` operator you wrote a tuple:

In [34]: (3 * '%12.3f', '%5.3f')
Out[34]: ('%12.3f%12.3f%12.3f', '%5.3f')

Ciao,
Marc 'BlackJack' Rintsch

 

[Marc 'BlackJack' Rintsch]

1. Multi dimensional arrays - how do you load them in python For
example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
-------
with "i" being the row number, "j" the column number, and "k" the ..
uhmm, well, the "group" number, how would you load this ?

If fortran90 you would just do:

do 10 k=1,2
do 20 i=1,3

read(*,*)(a(i,j,k),j=1,3)

20 continue
10 continue

How would the python equivalent go ?

Well, I don't know if this qualifies as equivalent:

=====
from __future__ import with_statement
from functools import partial
from itertools import islice
from pprint import pprint


def read_group(lines, count):
return [map(int, s.split()) for s in islice(lines, count)]


def main():
result = list()

with open('test.txt') as lines:
#
# Filter empty lines.
#
lines = (line for line in lines if line.strip())
#
# Read groups until end of file.
#
result = list(iter(partial(read_group, lines, 3), list()))

pprint(result, width=30)


if __name__ == '__main__':
main()
=====

The output is:

[[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]],
[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]]]

`k` is the first index here, not the last and the code doesn't use fixed
values for the ranges of `i`, `j`, and `k`, in fact it doesn't use index
variables at all but simply reads what's in the file. Only the group
length is hard coded in the source code.

Ciao,
Marc 'BlackJack' Rintsch

 

[Aidan]

Hello everyone,

I was wondering if anyone here has a moment of time to help me with 2
things that have been bugging me.

1. Multi dimensional arrays - how do you load them in python
For example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
-------
with "i" being the row number, "j" the column number, and "k" the ..
uhmm, well, the "group" number, how would you load this ?

If fortran90 you would just do:

do 10 k=1,2
do 20 i=1,3

read(*,*)(a(i,j,k),j=1,3)

20 continue
10 continue

How would the python equivalent go ?

2. I've read the help on the next one but I just find it difficult
understanding it.
I have;
a=2.000001
b=123456.789
c=1234.0001

How do you print them with the same number of decimals ?
(eg. 2.000, 123456.789, 1234.000)
and how do you print them with the same number of significant
decimals?
(eg. 2.000001, 123456.7, 1234.000 - always 8 decimals) ?


Is something like this possible (built-in) in python ?

Really grateful for all the help and time you can spare.

I'm not sure if this is applicable to your multi-dimensional list
problem... but it sounded a bit sudoku like (with row, columns and
groups) so I thought I'd share a bit of code of developed in regards to
solving sudoku puzzles...

Given a list of 9 list elements, each with nine elements (lets call it
sudoku_grid), the following list comprehensions produce lists of indexes
into sudoku grid

vgroups = [[(x,y) for y in xrange(9)] for x in xrange(9)]
hgroups = [[(x,y) for x in xrange(9)] for y in xrange(9)]
lgroups = [[(x,y) for x in xrange(a,a+3) for y in xrange(b,b+3)]
for a in xrange(0,9,3) for b in xrange(0,9,3)]

where sudoku_grid[y][x] yields the value at position (x,y), assuming the
top left corner is indexed as (0,0)

HTH

 

[Ivan Reborin]

On 30 Sep 2008 07:07:52 GMT, Marc 'BlackJack' Rintsch <[email protected]>
wrote:

Hello Marc, thanks for answering (on both subjects). I understand now
the logic which lays behind what you were explaining in the other one.
It cleared things quite a bit.

Well, I don't know if this qualifies as equivalent:

=====
from __future__ import with_statement
from functools import partial
from itertools import islice
from pprint import pprint


def read_group(lines, count):
return [map(int, s.split()) for s in islice(lines, count)]

def main():
result = list()
with open('test.txt') as lines:
lines = (line for line in lines if line.strip())
result = list(iter(partial(read_group, lines, 3), list()))
pprint(result, width=30)
if __name__ == '__main__':
main()
=====

I'm afraid I must admit I find the code above totally uncomprehesible
(I can't even see where the array here is mentioned - "result"?) and
inpractical for any kind of engineering practice I had in mind.

Does python, perchance, have some wrapper functions or something,
which would allow one to load an array in a more natural "technical"
way ? Like something mentioned above in my post (now deleted) ?

Also, is there a way to change counter for arrays to go from 0 to 1 ?
(first element being with the index 1) ?
(probably not since that seems like a language implementation thing,
but it doesn't hurt to ask)

 

[Marc 'BlackJack' Rintsch]

=====
from __future__ import with_statement from functools import partial
from itertools import islice
from pprint import pprint


def read_group(lines, count):
return [map(int, s.split()) for s in islice(lines, count)]

def main():
with open('test.txt') as lines:
lines = (line for line in lines if line.strip())
result = list(iter(partial(read_group, lines, 3), list()))
pprint(result, width=30)

if __name__ == '__main__':
main()
=====

I'm afraid I must admit I find the code above totally uncomprehesible (I
can't even see where the array here is mentioned - "result"?) and
inpractical for any kind of engineering practice I had in mind.

Learn Python then to understand that code. ;-)

There is no array. The data type is called "list" in Python, so `result`
is a nested list. And in Python it quite unusual to build lists by
creating them with the final size filled with place holder objects and
then fill the real values in. Instead lists are typically created by
appending values to existing lists, using list comprehension or the
`list()` function with some iterable object.

Typical Python code tries to minimize the use of index variables. Python
is not Fortran (or C, or Pascal, …).

Does python, perchance, have some wrapper functions or something, which
would allow one to load an array in a more natural "technical" way ?
Like something mentioned above in my post (now deleted) ?

Also, is there a way to change counter for arrays to go from 0 to 1 ?

You can write your own sequence type but that would be odd because the
rest of the language expects zero as the first index, so you will be
constantly fighting the language by adding or subtracting 1 all the time
at the "border" between your custom sequence type and the the rest of
Python.

Ciao,
Marc 'BlackJack' Rintsch

 

[Peter Pearson]

1. Multi dimensional arrays - how do you load them in python
For example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
-------
with "i" being the row number, "j" the column number, and "k" the ..
uhmm, well, the "group" number, how would you load this ?

If fortran90 you would just do:

do 10 k=1,2
do 20 i=1,3

read(*,*)(a(i,j,k),j=1,3)

20 continue
10 continue

How would the python equivalent go ?

Since you're coming from the FORTRAN world (thank you for
that stroll down Memory Lane), you might be doing scientific
computations, and so might be interested in the SciPy
package (Google scipy), which gives you arrays and matrices.
Don't expect to be able to use it without learning some Python,
though.

 

[Marc 'BlackJack' Rintsch]

You would drag yourself out of the 1960s, install numpy, and then do
something like this:

a = read_array(open("filename.dat","r"))

In [64]: a = numpy.fromfile('test.txt', dtype=int, sep=' ')

In [65]: a
Out[65]:
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18])

In [66]: a.reshape(2, 3, 3)
Out[66]:
array([[[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9]],

[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]]])

Ciao,
Marc 'BlackJack' Rintsch

 

[Ivan Reborin]

Since you're coming from the FORTRAN world (thank you for
that stroll down Memory Lane), you might be doing scientific
computations, and so might be interested in the SciPy
package (Google scipy), which gives you arrays and matrices.
Don't expect to be able to use it without learning some Python,
though.

Actually, no (regarding memory lane . I'm helping a friend to
translate some of my old routines to python so he can use them in his
programs.
I'm still using fortran84, and mean to continue doing so as long as
something better doesn't come along.

But as I said, got a job that't got to be done, so I'm trying to
figure out how to do array operations as easily as possible in python,
which are necessary for all my calculations.

Best regards
Ivan

 

[Steven D'Aprano]

You would drag yourself out of the 1960s, install numpy, and then do
something like this:

I think that was thoughtlessly rude to somebody who is asking a perfectly
reasonable question.

 

[Steven D'Aprano]

=====
from __future__ import with_statement from functools import partial
from itertools import islice
from pprint import pprint


def read_group(lines, count):
return [map(int, s.split()) for s in islice(lines, count)]

def main():
with open('test.txt') as lines:
lines = (line for line in lines if line.strip())
result = list(iter(partial(read_group, lines, 3), list()))
pprint(result, width=30)

if __name__ == '__main__':
main()
=====

I'm afraid I must admit I find the code above totally uncomprehesible
(I can't even see where the array here is mentioned - "result"?) and
inpractical for any kind of engineering practice I had in mind.

Learn Python then to understand that code. ;-)

There is no array. The data type is called "list" in Python, so
`result` is a nested list. And in Python it quite unusual to build
lists by creating them with the final size filled with place holder
objects and then fill the real values in. Instead lists are typically
created by appending values to existing lists, using list comprehension
or the `list()` function with some iterable object.

I would weaken that claim a tad... I'd say it is "usual" to write
something like this:

alist = []
for x in some_values:
alist.append(something_from_x)


but it is not uncommon (at least not in my code) to write something like
this equivalent code instead:

alist = [None]*len(some_values)
for i, x in enumerate(some_values):
alist = something_from_x


Most often the first way is most natural, but the second way is sometimes
more natural. It will also be more familiar to somebody coming from
Fortran, and it is a micro-optimization for large lists because it
doesn't need to resize the list as it grows.

I stress the *micro*-optimization, because Python lists are highly
optimized to resize as rarely as possible and as quickly as possible, so
you're not saving much time.

And Marc, I think you're being a little unfair to the OP, who is clearly
unfamiliar with Python. I've been using Python for perhaps ten years, and
I still find your code above dense and hard to comprehend. It uses a
number of "advanced Python concepts" that a newbie is going to have
trouble with:

- the with statement acts by magic; if you don't know what it does, it's
an opaque black box.

- you re-use the same name for different uses, which can cause confusion.

- generator expressions.

- functional programming using partial.

- you call a function that uses a list comprehension with both map and
iterator slicing inside it.


No wonder the OP had trouble with it. *I* have trouble with it, and would
need to sit down at the interactive interpreter and play around with it
for a while to be sure what it actually does. If it was your intention to
make Python look as obtuse and mysterious as possible, you almost
succeeded. The one things you missed was to replace the read_group
function with a lambda in the partial.

 

[Gabriel Genellina]

En Mon, 29 Sep 2008 19:04:18 -0300, Ivan Reborin

1. Multi dimensional arrays - how do you load them in python
For example, if I had:
-------
1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18

I agree that using NumPy is the way to go if you're going to do lots of
array calculations. But a plain Python code would look like this (more
comprehensible than other posted versions, I hope):

--- begin code ---
def read_group(fin, rows_per_group):
rows = []
for i in range(rows_per_group):
line = fin.readline()
row = [float(x) for x in line.split()]
rows.append(row)
fin.readline() # skip blank line
return rows

# simulate a file using a string instead
# actual code would use: fin = open(filename)

from StringIO import StringIO
fin = StringIO("""1 2 3
4 5 6
7 8 9

10 11 12
13 14 15
16 17 18
""")

# read 2 groups of 3 lines each
matrix = [read_group(fin, 3) for k in range(2)]
print matrix
--- end code ---

A more compact version of read_group (collapsing all rows.append onto the
outer list comprehension):

--- begin code ---
def read_group(fin, rows_per_group):
rows = [[float(x) for x in fin.readline().split()]
for i in range(rows_per_group)]
fin.readline() # skip blank line
return rows
--- end code ---

 

[Paul Probert]

You would drag yourself out of the 1960s, install numpy, and
then do something like this:

a = read_array(open("filename.dat","r"))


If not full-up scipy (which provides all sorts of scientific
and numerical-analysis stuff), then at least numpy (which
provides the basic array/matrix operations:

http://numpy.scipy.org/

Though the software is free, the documentation isn't. You've
got to buy the book if you want something to read. IMO, it's
definitely worth it, and a good way to support the project even
if you don't really need something to keep your bookends apart.

clip ...
The book is free now, as of Aug 21, 08.
http://www.tramy.us/guidetoscipy.html

Paul Probert

 

[Marc 'BlackJack' Rintsch]

There is no array. The data type is called "list" in Python, so
`result` is a nested list. And in Python it quite unusual to build
lists by creating them with the final size filled with place holder
objects and then fill the real values in. Instead lists are typically
created by appending values to existing lists, using list comprehension
or the `list()` function with some iterable object.

I would weaken that claim a tad... I'd say it is "usual" to write
something like this:

alist = []
for x in some_values:
alist.append(something_from_x)


but it is not uncommon (at least not in my code) to write something like
this equivalent code instead:

alist = [None]*len(some_values)
for i, x in enumerate(some_values):
alist = something_from_x

I have never done this, except in the beginning I used Python, and --
maybe more importantly -- I've never seen this in others code. I really
looks like a construct from someone who is still programming in some
other language(s).

Most often the first way is most natural, but the second way is
sometimes more natural.

When will it be more natural to introduce an unnecessary index?

And Marc, I think you're being a little unfair to the OP, who is clearly
unfamiliar with Python. I've been using Python for perhaps ten years,
and I still find your code above dense and hard to comprehend. It uses a
number of "advanced Python concepts" that a newbie is going to have
trouble with:

- the with statement acts by magic; if you don't know what it does, it's
an opaque black box.

Everything acts by magic unless you know what it does. The Fortran

read(*,*)(a(i,j,k),j=1,3)

in the OP's first post looks like magic too. I admit that my code shows
off advanced Python features but I don't think ``with`` is one of them.
It makes it easier to write robust code and maybe even understandable
without documentation by just reading it as "English text".

- you re-use the same name for different uses, which can cause
confusion.

Do you mean `lines`? Then I disagree because the (duck) type is always
"iterable over lines". I just changed the content by filtering.

- generator expressions.

- functional programming using partial.

- you call a function that uses a list comprehension with both map and
iterator slicing inside it.


No wonder the OP had trouble with it. *I* have trouble with it, and
would need to sit down at the interactive interpreter and play around
with it for a while to be sure what it actually does. If it was your
intention to make Python look as obtuse and mysterious as possible, you
almost succeeded. The one things you missed was to replace the
read_group function with a lambda in the partial.

Well that would make the code harder to understand. ;-)

Serious, I think it should be easy to understand the code for someone who
knows Python. Yes a newbie will have trouble to understand this, but
Python is not Fortran and IMHO I haven't used something really exotic or
strange nor did I wrote convoluted and hard to understand things like
deeply nested list comprehensions.

Ciao,
Marc 'BlackJack' Rintsch

 

[Steven D'Aprano]

Sheesh. I guess I should have added a smiley face.

So much for trying to be helpful.

Oh the rest of your post was helpful. I think you were trying to be
funny, but I think you failed.