float storage

I

ivo welch

hi perl gurus: I would like to read about 1GB of floats, not doubles.
Does perl have the facility to designate internal storage of an array
to be in float (4-byte), not double (8-byte) representation?
Otherwise, I am running out of memory...

sincerely, /iaw
 
B

Ben Morrow

Quoth (e-mail address removed) (ivo welch):
hi perl gurus: I would like to read about 1GB of floats, not doubles.
Does perl have the facility to designate internal storage of an array
to be in float (4-byte), not double (8-byte) representation?
Otherwise, I am running out of memory...

No. NVs (Perl's representation of floating-point) are always NVs.

You could perhaps achieve this by writing a little XS or Inline class to
tie an array to a C float[]; something like (completely untested):

Tie/FloatArray.xs:

/* standard XS stuff */

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

typedef struct floatarray {
UV size;
float *data;
} *Tie_FloatArray;

void extend_floatarray(Tie_FloatArray this, UV size)
{
this->size = size;
Renewc(this->data, this->size ? this->size : 1, float, float *);
}

/* note that the line below has a literal tab in the middle */

TYPEMAP
Tie_FloatArray T_PTROBJ_SPECIAL

INPUT
T_PTROBJ_SPECIAL
if (sv_derived_from($arg, \"${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\")) {
IV tmp = SvIV((SV*)SvRV($arg));
$var = ($type) tmp;
}
else
croak(\"$var is not of type ${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\")

OUTPUT
T_PTROBJ_SPECIAL
sv_setref_pv($arg, \"${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\", (void*)$var);

MODULE = Tie::FloatArray PACKAGE = Tie::FloatArray

Tie_FloatArray
TIEARRAY (char *class)
CODE:
Newc(434, RETVAL, 1, struct floatarray, Tie_FloatArray);
RETVAL->size = 0;
Newc(434, RETVAL->data, 1, float, float *);
OUTPUT:
RETVAL

SV *
FETCH (Tie_FloatArray this, UV index)
CODE:
if (index < this->size) {
RETVAL = NEWSV(434, 0);
sv_setnv(RETVAL, this->data[index]);
}
else {
RETVAL = &PL_sv_undef;
}
OUTPUT:
RETVAL

float
STORE (Tie_FloatArray this, UV index, float value)
CODE:
if (index > this->size) {
croak("index %"UVf" > size %"UVf" of Tie::FloatArray",
index, this->size);
}
RETVAL = this->data[index] = value;
OUTPUT:
RETVAL

UV
FETCHSIZE (Tie_FloatArray this)
CODE:
RETVAL = this->size;
OUTPUT:
RETVAL

void
STORESIZE (Tie_FloatArray this, UV size)
CODE:
extend_floatarray(this, size);

void
EXTEND (Tie_FloatArray this, UV size)
CODE:
extend_floatarray(this, size);

void
DESTROY (Tie_FloatArray this)
CODE:
Safefree(this->data);
Safefree(this);

Note that this array *only* supports float values: no undefs.

Ben
 
C

ctcgag

hi perl gurus: I would like to read about 1GB of floats, not doubles.

What do you mean by 1 GB? 1GB the way C would store them, at 4 bytes each?
1GB of text representing floats, at ~20 bytes each?
Does perl have the facility to designate internal storage of an array
to be in float (4-byte), not double (8-byte) representation?

That's probably the least of your problems. All the reference counts and
flags and double indirection and etc. that make up perl data internally is
much larger that the 4 byte difference you are talking about. I'd
consider about 20 bytes per float to be the absolute minimum you can get
out of perl, and that is if you are very careful.
Otherwise, I am running out of memory...

PDL might have something to help you out here. I don't really know PDL, so
I can't tell you more than that.

I often pack floats into strings, then pass the string to Inline::C
subroutines, where I cast the char* back to a float*. This works
especially well if Perl doesn't even need the floats, it just has to hold
onto them between Inline::C calls.

But really, your best bet is probably to spend your time changing the
algorithm so it doesn't need all data to be in memory at once. Otherwise,
when the problem size increases by 10%, all of your micro-tweaking effort
goes out the window and you are back to OOM errors again.

Xho
 
T

Tassilo v. Parseval

Also sprach ivo welch:
hi perl gurus: I would like to read about 1GB of floats, not doubles.
Does perl have the facility to designate internal storage of an array
to be in float (4-byte), not double (8-byte) representation?
Otherwise, I am running out of memory...

Other than using Inline/XS as Ben suggested, you can try to go the
slightly cheaper path by using a packed string.

sub next_float {
# this function returns the next float
# in some way
...
}

my $floats;
while ( defined(my $f = next_float()) ) {
$floats .= pack "f", $f;
}

To get back at one of the floats, use a function:

sub get_float {
my ($floats, $idx) = @_;
return unpack "f", substr $floats, $idx*4, 4;
}

my $f = get_float($floats, 0); # returns the first one

Tassilo
 
B

Ben Morrow

Quoth (e-mail address removed)-aachen.de:
Also sprach ivo welch:


Other than using Inline/XS as Ben suggested, you can try to go the
slightly cheaper path by using a packed string.

....and of course you could write a (Perl, not XS) class to tie an array
to this packed string. Silly, I should have thought of that :)

Ben
 
I

ivo welch

(yes, 1GB is 4-byte float data size. yes, I could change the
algorithm. what I should do is use an data base---but that would mean
installing one and learning SQL for a 1-time application.)

thank you everyone. probably back to C++ for me. it's been a long
time---hope I still remember it. now that I have used perl for a
while, C++ really seems like a dinosaur. the string and array
handling is truly awkward.

regards,

/iaw
 
E

Eric Bohlman

(e-mail address removed) (ivo welch) wrote in
(yes, 1GB is 4-byte float data size. yes, I could change the
algorithm. what I should do is use an data base---but that would mean
installing one and learning SQL for a 1-time application.)

There are a number of Perl modules that provide SQL-based access to
self-contained databases that don't require installing a full-blown
database server: DBD::SQLite would probably be a good choice (it stores
each database in a single file, and the module is self-contained; all you
need are it and DBI). The amount of SQL you'd need to know for such an
application can be learned in about an hour.
 
J

John Bokma

Eric said:
(e-mail address removed) (ivo welch) wrote in


There are a number of Perl modules that provide SQL-based access to
self-contained databases that don't require installing a full-blown
database server: DBD::SQLite would probably be a good choice (it stores
each database in a single file, and the module is self-contained; all you
need are it and DBI). The amount of SQL you'd need to know for such an
application can be learned in about an hour.

If you have SQL experience, yes. But learning it from scratch takes more
then one hour, especially if you want to run your queries fast.
Installing, configuring etc. will take more. My best (educated) guess
would be 2 days, at the least.

I consider learning SQL a good investment of time though. It's like
hashes, regexps, etc, once you understand them, you find new uses.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,479
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top