float storage

Discussion in 'Perl Misc' started by ivo welch, Jun 18, 2004.

  1. ivo welch

    ivo welch Guest

    hi perl gurus: I would like to read about 1GB of floats, not doubles.
    Does perl have the facility to designate internal storage of an array
    to be in float (4-byte), not double (8-byte) representation?
    Otherwise, I am running out of memory...

    sincerely, /iaw
    ivo welch, Jun 18, 2004
    #1
    1. Advertising

  2. ivo welch

    Ben Morrow Guest

    Quoth (ivo welch):
    > hi perl gurus: I would like to read about 1GB of floats, not doubles.
    > Does perl have the facility to designate internal storage of an array
    > to be in float (4-byte), not double (8-byte) representation?
    > Otherwise, I am running out of memory...


    No. NVs (Perl's representation of floating-point) are always NVs.

    You could perhaps achieve this by writing a little XS or Inline class to
    tie an array to a C float[]; something like (completely untested):

    Tie/FloatArray.xs:

    /* standard XS stuff */

    #include "EXTERN.h"
    #include "perl.h"
    #include "XSUB.h"

    typedef struct floatarray {
    UV size;
    float *data;
    } *Tie_FloatArray;

    void extend_floatarray(Tie_FloatArray this, UV size)
    {
    this->size = size;
    Renewc(this->data, this->size ? this->size : 1, float, float *);
    }

    /* note that the line below has a literal tab in the middle */

    TYPEMAP
    Tie_FloatArray T_PTROBJ_SPECIAL

    INPUT
    T_PTROBJ_SPECIAL
    if (sv_derived_from($arg, \"${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\")) {
    IV tmp = SvIV((SV*)SvRV($arg));
    $var = ($type) tmp;
    }
    else
    croak(\"$var is not of type ${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\")

    OUTPUT
    T_PTROBJ_SPECIAL
    sv_setref_pv($arg, \"${(my $ntt=$ntype)=~s/_/::/g;\$ntt}\", (void*)$var);

    MODULE = Tie::FloatArray PACKAGE = Tie::FloatArray

    Tie_FloatArray
    TIEARRAY (char *class)
    CODE:
    Newc(434, RETVAL, 1, struct floatarray, Tie_FloatArray);
    RETVAL->size = 0;
    Newc(434, RETVAL->data, 1, float, float *);
    OUTPUT:
    RETVAL

    SV *
    FETCH (Tie_FloatArray this, UV index)
    CODE:
    if (index < this->size) {
    RETVAL = NEWSV(434, 0);
    sv_setnv(RETVAL, this->data[index]);
    }
    else {
    RETVAL = &PL_sv_undef;
    }
    OUTPUT:
    RETVAL

    float
    STORE (Tie_FloatArray this, UV index, float value)
    CODE:
    if (index > this->size) {
    croak("index %"UVf" > size %"UVf" of Tie::FloatArray",
    index, this->size);
    }
    RETVAL = this->data[index] = value;
    OUTPUT:
    RETVAL

    UV
    FETCHSIZE (Tie_FloatArray this)
    CODE:
    RETVAL = this->size;
    OUTPUT:
    RETVAL

    void
    STORESIZE (Tie_FloatArray this, UV size)
    CODE:
    extend_floatarray(this, size);

    void
    EXTEND (Tie_FloatArray this, UV size)
    CODE:
    extend_floatarray(this, size);

    void
    DESTROY (Tie_FloatArray this)
    CODE:
    Safefree(this->data);
    Safefree(this);

    Note that this array *only* supports float values: no undefs.

    Ben

    --
    I've seen things you people wouldn't believe: attack ships on fire off
    the shoulder of Orion; I watched C-beams glitter in the dark near the
    Tannhauser Gate. All these moments will be lost, in time, like tears in rain.
    Time to die.
    Ben Morrow, Jun 19, 2004
    #2
    1. Advertising

  3. ivo welch

    Guest

    (ivo welch) wrote:
    > hi perl gurus: I would like to read about 1GB of floats, not doubles.


    What do you mean by 1 GB? 1GB the way C would store them, at 4 bytes each?
    1GB of text representing floats, at ~20 bytes each?

    > Does perl have the facility to designate internal storage of an array
    > to be in float (4-byte), not double (8-byte) representation?


    That's probably the least of your problems. All the reference counts and
    flags and double indirection and etc. that make up perl data internally is
    much larger that the 4 byte difference you are talking about. I'd
    consider about 20 bytes per float to be the absolute minimum you can get
    out of perl, and that is if you are very careful.

    > Otherwise, I am running out of memory...


    PDL might have something to help you out here. I don't really know PDL, so
    I can't tell you more than that.

    I often pack floats into strings, then pass the string to Inline::C
    subroutines, where I cast the char* back to a float*. This works
    especially well if Perl doesn't even need the floats, it just has to hold
    onto them between Inline::C calls.

    But really, your best bet is probably to spend your time changing the
    algorithm so it doesn't need all data to be in memory at once. Otherwise,
    when the problem size increases by 10%, all of your micro-tweaking effort
    goes out the window and you are back to OOM errors again.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Jun 19, 2004
    #3
  4. Also sprach ivo welch:

    > hi perl gurus: I would like to read about 1GB of floats, not doubles.
    > Does perl have the facility to designate internal storage of an array
    > to be in float (4-byte), not double (8-byte) representation?
    > Otherwise, I am running out of memory...


    Other than using Inline/XS as Ben suggested, you can try to go the
    slightly cheaper path by using a packed string.

    sub next_float {
    # this function returns the next float
    # in some way
    ...
    }

    my $floats;
    while ( defined(my $f = next_float()) ) {
    $floats .= pack "f", $f;
    }

    To get back at one of the floats, use a function:

    sub get_float {
    my ($floats, $idx) = @_;
    return unpack "f", substr $floats, $idx*4, 4;
    }

    my $f = get_float($floats, 0); # returns the first one

    Tassilo
    --
    $_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
    pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
    $_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
    Tassilo v. Parseval, Jun 19, 2004
    #4
  5. ivo welch

    Ben Morrow Guest

    Quoth -aachen.de:
    > Also sprach ivo welch:
    >
    > > hi perl gurus: I would like to read about 1GB of floats, not doubles.
    > > Does perl have the facility to designate internal storage of an array
    > > to be in float (4-byte), not double (8-byte) representation?
    > > Otherwise, I am running out of memory...

    >
    > Other than using Inline/XS as Ben suggested, you can try to go the
    > slightly cheaper path by using a packed string.


    ....and of course you could write a (Perl, not XS) class to tie an array
    to this packed string. Silly, I should have thought of that :)

    Ben

    --
    "If a book is worth reading when you are six, *
    it is worth reading when you are sixty." - C.S.Lewis
    Ben Morrow, Jun 19, 2004
    #5
  6. ivo welch

    ivo welch Guest

    (yes, 1GB is 4-byte float data size. yes, I could change the
    algorithm. what I should do is use an data base---but that would mean
    installing one and learning SQL for a 1-time application.)

    thank you everyone. probably back to C++ for me. it's been a long
    time---hope I still remember it. now that I have used perl for a
    while, C++ really seems like a dinosaur. the string and array
    handling is truly awkward.

    regards,

    /iaw
    ivo welch, Jun 19, 2004
    #6
  7. ivo welch

    Eric Bohlman Guest

    (ivo welch) wrote in
    news::

    > (yes, 1GB is 4-byte float data size. yes, I could change the
    > algorithm. what I should do is use an data base---but that would mean
    > installing one and learning SQL for a 1-time application.)


    There are a number of Perl modules that provide SQL-based access to
    self-contained databases that don't require installing a full-blown
    database server: DBD::SQLite would probably be a good choice (it stores
    each database in a single file, and the module is self-contained; all you
    need are it and DBI). The amount of SQL you'd need to know for such an
    application can be learned in about an hour.
    Eric Bohlman, Jun 20, 2004
    #7
  8. ivo welch

    John Bokma Guest

    Eric Bohlman wrote:

    > (ivo welch) wrote in
    > news::
    >
    >>(yes, 1GB is 4-byte float data size. yes, I could change the
    >>algorithm. what I should do is use an data base---but that would mean
    >>installing one and learning SQL for a 1-time application.)

    >
    > There are a number of Perl modules that provide SQL-based access to
    > self-contained databases that don't require installing a full-blown
    > database server: DBD::SQLite would probably be a good choice (it stores
    > each database in a single file, and the module is self-contained; all you
    > need are it and DBI). The amount of SQL you'd need to know for such an
    > application can be learned in about an hour.


    If you have SQL experience, yes. But learning it from scratch takes more
    then one hour, especially if you want to run your queries fast.
    Installing, configuring etc. will take more. My best (educated) guess
    would be 2 days, at the least.

    I consider learning SQL a good investment of time though. It's like
    hashes, regexps, etc, once you understand them, you find new uses.

    --
    John MexIT: http://johnbokma.com/mexit/
    personal page: http://johnbokma.com/
    Experienced Perl programmer available: http://castleamber.com/
    Happy Customers: http://castleamber.com/testimonials.html
    John Bokma, Jun 20, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page