Question about PDL

[January Weiner]

Hello,

in my program, I am using a matrix which has usually a size of 2000x2000 or
similar. It contains short integers. The matrix is filled in one by one in
a loop over rows and columns.

The only calculation that I am doing with this matrix is averaging
over a window sliding along the diagonals. Finally, I need to access each
matrix element one after another in a loop to calculate something.

At first, I used a regular Perl matrix, constructed as [ [ .... ], [ ... ],
..... ]. This was quite slow and took a lot of memory (the memory footprint
of my program grews up by roughly 50MB), so I googled and found PDL -- Perl
Data Language.

PDL is supposed to be much faster and to have a smaller memory footprint.

While I can see the latter (the footprint is now negligible compared to the
whole program), it is roughly three to four times slower than the regular
perlish way. I am creating the matrix as follows [please bear with me -- I
am not posting actual code, because it is rather complex, and you will see
that answering my question doesn't require finding out whether my code is
correct]:

$matrix = short(zeroes($l1, $l2)) ;

(where $l1 and $l2 are dimensions of the matrix), and accessing / setting
the elements using at() and set():

set( $matrix, $i, $j, $value ) ;
$value = at( $matrix, $i, $j ) ;

There is another way of doing it using PDL::NiceSlice, which uses
constructs like $matrix->($i, $j), but I found it to be even slower.

QUESTION: Is this a normal behaviour? Is it normal that a standard perlish
matrix is few times faster than the PDL implementation? Or should I start
finding out where I messed things up?

Thank you in advance,

January

 

[Joost Diepenmaat]

PDL is supposed to be much faster and to have a smaller memory footprint.

While I can see the latter (the footprint is now negligible compared to the
whole program), it is roughly three to four times slower than the regular
perlish way. I am creating the matrix as follows [please bear with me -- I
am not posting actual code, because it is rather complex, and you will see
that answering my question doesn't require finding out whether my code is
correct]:

$matrix = short(zeroes($l1, $l2)) ;

(where $l1 and $l2 are dimensions of the matrix), and accessing / setting
the elements using at() and set():

set( $matrix, $i, $j, $value ) ;
$value = at( $matrix, $i, $j ) ;

There is another way of doing it using PDL::NiceSlice, which uses
constructs like $matrix->($i, $j), but I found it to be even slower.

If I'm reading you correctly, all your calculations involve getting a
value from the matrix, processing it in pure perl and then setting it
back. That is not how you get good speed from PDL. You want to move as
many calculations as you can to the PDL operators, especially those
operators that can act on groups of values.

I only have very limited knowledge of PDL, but it seems to me that you
could probably do your averaging by using the PDL projection
operators. See

<http://www.johnlapeyre.com/pdl/pdldoc/newbook/node4.html#SECTION00440000000000000000>

and the section immediately after that.

 

[January Weiner]

If I'm reading you correctly, all your calculations involve getting a
value from the matrix, processing it in pure perl and then setting it
back. That is not how you get good speed from PDL. You want to move as
many calculations as you can to the PDL operators, especially those
operators that can act on groups of values.

Yes, I do it now. The operation that I needed was conv2d from PDL::Image2D,
with a convolution matrix of the form zeroes($window,$window)->diagonal++.

However, the other big advantage from PDL is for me the reduction of the
memory footprint, which is substantial.

I'm now stuck with the old conundrum: low memory footprint OR speed.

j.

I only have very limited knowledge of PDL, but it seems to me that you
could probably do your averaging by using the PDL projection
operators. See

and the section immediately after that.