processing large numbers/values/figures

Discussion in 'Perl Misc' started by Sherm Pendley, Jul 10, 2006.

  1. Lukas Ruf <> writes:

    > How can I deal with large numbers in Perl?


    Have a look at the Math::BigInt, Math::BigFloat, and Math::BigRat modules,
    depending on the kind of numbers you're working with. All three are core.

    sherm--

    --
    Web Hosting by West Virginians, for West Virginians: http://wv-www.net
    Cocoa programming in Perl: http://camelbones.sourceforge.net
     
    Sherm Pendley, Jul 10, 2006
    #1
    1. Advertising

  2. Sherm Pendley

    Lukas Ruf Guest

    Dear all,

    for a large number of files, I must accumulate numbers found
    therein. For this accumulation, I have been running into
    number-overruns.

    As a beginner with Perl, I have been able to 'split()' lines
    into arrays of numeric-strings. However, when adding fields
    together, number-overruns do happen.

    After searching the web for a while (maybe with the wrong
    keywords) without success, I kindly ask this mailing list
    for help:

    How can I deal with large numbers in Perl?

    - adding
    - subtracting
    - multiplying
    - dividing
    - printing

    + large in the range up to 10*2^^40 (10 Tera).
    + most important: adding and printing to ASCII

    Thanks in advance for any help!

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 10, 2006
    #2
    1. Advertising

  3. Sherm Pendley

    Guest

    Lukas Ruf <> wrote:
    > Dear all,
    >
    > for a large number of files, I must accumulate numbers found
    > therein. For this accumulation, I have been running into
    > number-overruns.


    I don't get overruns until 10**307. Are you sure it is an overrun
    rather than just a loss of precision?

    >
    > As a beginner with Perl, I have been able to 'split()' lines
    > into arrays of numeric-strings. However, when adding fields
    > together, number-overruns do happen.
    >
    > After searching the web for a while (maybe with the wrong
    > keywords) without success, I kindly ask this mailing list
    > for help:
    >
    > How can I deal with large numbers in Perl?
    >
    > - adding
    > - subtracting
    > - multiplying
    > - dividing
    > - printing
    >
    > + large in the range up to 10*2^^40 (10 Tera).


    On my 32 bit machine, I get unit-level precision up to over 1000*2**40.

    > + most important: adding and printing to ASCII


    Aside from the bignum and other modules previously mentioned, have you
    looked at using printf with a higher precision format? Maybe you are
    losing precision at the print stage rather than the internal
    representation.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 10, 2006
    #3
  4. Sherm Pendley

    Lukas Ruf Guest

    > A. Sinan Unur [Mon, 10 Jul 2006 18:30:19 GMT]:
    >
    > Lukas Ruf <> wrote in news:slrneb55hc.kbb.ruf@pc-
    > 4082.ethz.ch:
    >
    > > How can I deal with large numbers in Perl?
    > >
    > > - adding
    > > - subtracting
    > > - multiplying
    > > - dividing
    > > - printing
    > >
    > > + large in the range up to 10*2^^40 (10 Tera).
    > > + most important: adding and printing to ASCII

    >
    > #!/usr/bin/perl
    >
    > use strict;
    > use warnings;
    >
    > use bignum;
    >
    > my $x = 10*(2**40);
    > print 123.55 * $x, "\n";
    >


    thanks very much!

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 10, 2006
    #4
  5. Sherm Pendley

    Lukas Ruf Guest

    > Sherm Pendley [Mon, 10 Jul 2006 14:09:10 -0400]:
    >
    > Lukas Ruf <> writes:
    >
    > > How can I deal with large numbers in Perl?

    >
    > Have a look at the Math::BigInt, Math::BigFloat, and Math::BigRat
    > modules, depending on the kind of numbers you're working with. All
    > three are core.
    >


    thank you very much!

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 10, 2006
    #5
  6. Sherm Pendley

    Lukas Ruf Guest

    > [10 Jul 2006 20:21:24 GMT]:
    >
    > Lukas Ruf <> wrote:
    > > Dear all,
    > >
    > > for a large number of files, I must accumulate numbers found
    > > therein. For this accumulation, I have been running into
    > > number-overruns.

    >
    > I don't get overruns until 10**307. Are you sure it is an overrun
    > rather than just a loss of precision?
    >


    my problem is, after a certain value, it turns to -1.

    > > + large in the range up to 10*2^^40 (10 Tera).

    >
    > On my 32 bit machine, I get unit-level precision up to over 1000*2**40.
    >


    thanks for checking.

    > > + most important: adding and printing to ASCII

    >
    > Aside from the bignum and other modules previously mentioned, have you
    > looked at using printf with a higher precision format? Maybe you are
    > losing precision at the print stage rather than the internal
    > representation.
    >


    I make use of Perl's hashes

    my %hsh = ();

    $hsh{$key} += @record{$index};

    printf("%15d should be in the range up to 2**34\n",
    $hsh{$key});


    Any hint what goes wrong?

    Thanks in advance.

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #6
  7. Sherm Pendley

    Lukas Ruf Guest

    > Abigail [10 Jul 2006 22:26:45 GMT]:
    >


    thanks for answering,

    > Lukas Ruf () wrote on MMMMDCXCVI September MCMXCIII in
    > <URL:news:>:


    >
    > A Perl using 32-bit integers will automatically convert integers to
    > floats. If you're dealing with numbers up to about 2**53, there shouldn't
    > be any loss of precision.
    >


    weird, why do I still get '-1' even when using 'bignum'?

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #7
  8. Sherm Pendley

    Lukas Ruf Guest

    > Lukas Ruf [11 Jul 2006 08:28:18 +0100]:
    >
    > > [10 Jul 2006 20:21:24 GMT]:
    > >
    > > Aside from the bignum and other modules previously mentioned, have you
    > > looked at using printf with a higher precision format? Maybe you are
    > > losing precision at the print stage rather than the internal
    > > representation.
    > >

    >
    > I make use of Perl's hashes
    >
    > my %hsh = ();
    >
    > $hsh{$key} += @record{$index};
    >
    > printf("%15d should be in the range up to 2**34\n",
    > $hsh{$key});
    >
    >
    > Any hint what goes wrong?
    >


    according to your hint, I tried the following:

    printf(tot_file "%20s $tot_key\n", $tot_val);
    printf(tot_file "%20s %20llu\n", $tot_val, $tot_key);

    with the following result:

    ==> byte_total_stat.txt <==
    byte_total 156999049890
    byte_total 4294967295

    where the former seems to be reasonable while the later is 2**32-1

    Thus my question:
    Is there an easy way to format printouts with numbers of that size?
    I.e. I would like to specify anything like '%20llu' that allows
    for printing of the correct values.

    Thanks in advance for any hint!

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #8
  9. Sherm Pendley

    Sisyphus Guest

    "Lukas Ruf" <> wrote in message
    news:...
    > > Abigail [10 Jul 2006 22:26:45 GMT]:
    > >

    >
    > thanks for answering,
    >
    > > Lukas Ruf () wrote on MMMMDCXCVI September MCMXCIII in
    > > <URL:news:>:

    >
    > >
    > > A Perl using 32-bit integers will automatically convert integers to
    > > floats. If you're dealing with numbers up to about 2**53, there

    shouldn't
    > > be any loss of precision.
    > >

    >
    > weird, why do I still get '-1' even when using 'bignum'?
    >


    Checkout 'perldoc -f sprintf' (which applies equally to the 'printf'
    function). There you'll find:

    %d a signed integer, in decimal

    Specifying "%d" is good for numbers up to 31 bits (plus the 'sign' bit) only
    on 32-bit systems (since that's the size of a signed integer). For larger
    numbers up to 2 ** 53 use "%f" (or "%.0f" if you want to avoid the decimal
    point) .... or you may find that simply using the 'print' function instead
    of the 'printf' function produces the output you want.

    It would probably help us to answer your questions about specific instances
    if you could provide a (*minimal*) script (copy'n'paste) that demonstrate
    the problem(s) - as opposed to code snippets and general descriptions.

    As a starter - does the output of the following script fail in some way to
    satisfy your requirements ?

    ---------------------------
    use warnings;
    use bignum;

    $x = 2 ** 57 + 20345;
    print $x, "\n";
    __END__
    ---------------------------

    Cheers,
    Rob
     
    Sisyphus, Jul 11, 2006
    #9
  10. Lukas Ruf <> wrote:

    > $hsh{$key} += @record{$index};



    You should always enable warnings when developing Perl code.


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Jul 11, 2006
    #10
  11. Sherm Pendley

    Guest

    Lukas Ruf <> wrote:
    > > [10 Jul 2006 20:21:24 GMT]:
    > >
    > > Lukas Ruf <> wrote:
    > > > Dear all,
    > > >
    > > > for a large number of files, I must accumulate numbers found
    > > > therein. For this accumulation, I have been running into
    > > > number-overruns.

    > >
    > > I don't get overruns until 10**307. Are you sure it is an overrun
    > > rather than just a loss of precision?
    > >

    >
    > my problem is, after a certain value, it turns to -1.


    No, it just prints as -1.

    > printf("%15d should be in the range up to 2**34\n",
    > $hsh{$key});
    >
    > Any hint what goes wrong?


    You have a floating point number which is holding something which happens
    to be an integer which doesn't fit into a 32bit integer. Your %d field is
    coercing it into a 32 bit integer for formatting purposes, which does
    overflow. Print with a floating point format, not an integet format.

    printf "%20.f", 2**50;
    1125899906842624

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 11, 2006
    #11
  12. Lukas Ruf wrote:
    >> Abigail [10 Jul 2006 22:26:45 GMT]:
    >>

    >
    > thanks for answering,
    >
    >> Lukas Ruf () wrote on MMMMDCXCVI September MCMXCIII in
    >> <URL:news:>:

    >
    >> A Perl using 32-bit integers will automatically convert integers to
    >> floats. If you're dealing with numbers up to about 2**53, there shouldn't
    >> be any loss of precision.
    >>

    >
    > weird, why do I still get '-1' even when using 'bignum'?
    >


    Failure to read the docs... :)

    perldoc -f printf:

    Don't fall into the trap of using a "printf" when a simple
    "print" would do.
    The "print" is more efficient and less error prone.

    Actually, Perl's printf is probably just passing back the underlying
    library error even though bignum produces the correct result. On
    Solaris's printf(3C) manpage, for example:

    The printf(), fprintf(), and sprintf() functions return the
    number of bytes transmitted (excluding the terminating null
    byte in the case of sprintf()).
    ...
    Each function returns a negative value if an output error
    was encountered.


    So, on 32-bit, you'll see:

    use bignum;
    my $x = 10*(2**40);
    print "x = $x";
    printf( "%d should be in the range up to 2**40\n", $x );

    --> x = 10995116277760
    -1 should be in the range up to 2**40


    --
    Charles DeRykus
     
    Charles DeRykus, Jul 11, 2006
    #12
  13. Sherm Pendley

    Lukas Ruf Guest

    > Sisyphus [Tue, 11 Jul 2006 21:23:03 +1000]:
    >
    >
    > "Lukas Ruf" <> wrote in message
    > news:...
    > > > Abigail [10 Jul 2006 22:26:45 GMT]:
    > > >

    > >
    > > thanks for answering,
    > >
    > > > Lukas Ruf () wrote on MMMMDCXCVI September MCMXCIII in
    > > > <URL:news:>:

    > >
    > > >
    > > > A Perl using 32-bit integers will automatically convert integers to
    > > > floats. If you're dealing with numbers up to about 2**53, there

    > shouldn't
    > > > be any loss of precision.
    > > >

    > >
    > > weird, why do I still get '-1' even when using 'bignum'?
    > >

    >
    > Checkout 'perldoc -f sprintf' (which applies equally to the 'printf'
    > function). There you'll find:
    >
    > %d a signed integer, in decimal
    >
    > Specifying "%d" is good for numbers up to 31 bits (plus the 'sign' bit) only
    > on 32-bit systems (since that's the size of a signed integer). For larger
    > numbers up to 2 ** 53 use "%f" (or "%.0f" if you want to avoid the decimal
    > point) .... or you may find that simply using the 'print' function instead
    > of the 'printf' function produces the output you want.
    >


    thanks. this solves my problem.

    > It would probably help us to answer your questions about specific instances
    > if you could provide a (*minimal*) script (copy'n'paste) that demonstrate
    > the problem(s) - as opposed to code snippets and general descriptions.
    >


    ok. I understand.


    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #13
  14. Sherm Pendley

    Lukas Ruf Guest

    > [11 Jul 2006 16:31:53 GMT]:
    >
    > You have a floating point number which is holding something which
    > happens to be an integer which doesn't fit into a 32bit integer.
    > Your %d field is coercing it into a 32 bit integer for formatting
    > purposes, which does overflow. Print with a floating point format,
    > not an integet format.
    >
    > printf "%20.f", 2**50; 1125899906842624
    >


    thanks! This, I have not known before.

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #14
  15. Sherm Pendley

    Lukas Ruf Guest

    > Tad McClellan [Tue, 11 Jul 2006 07:51:46 -0500]:
    >
    > Lukas Ruf <> wrote:
    >
    > > $hsh{$key} += @record{$index};

    >
    >
    > You should always enable warnings when developing Perl code.
    >


    meanwhile, I do.

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 11, 2006
    #15
  16. Lukas Ruf <> wrote:
    >> Tad McClellan [Tue, 11 Jul 2006 07:51:46 -0500]:
    >>
    >> Lukas Ruf <> wrote:
    >>
    >> > $hsh{$key} += @record{$index};

    >>
    >>
    >> You should always enable warnings when developing Perl code.
    >>

    >
    > meanwhile, I do.



    You should not ignore the warnings that perl emits when
    developing Perl code.


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Jul 11, 2006
    #16
  17. Lukas Ruf <> writes:

    > perl does not emit any warning anymore -- did you refer to my
    > mistyping of
    >
    >> >> > $hsh{$key} += @record{$index};

    >
    > instead of
    >
    > $hsh{$key} += @record[$index];


    Not that I want to be redundant, but the above is exactly why the posting
    guidelines suggest copy-and-pasting your code instead of retyping it. It
    helps avoid errors such as these.

    It also helps avoid "read the posting guidelines" responses. :)

    sherm--

    --
    Web Hosting by West Virginians, for West Virginians: http://wv-www.net
    Cocoa programming in Perl: http://camelbones.sourceforge.net
     
    Sherm Pendley, Jul 12, 2006
    #17
  18. Sherm Pendley

    Lukas Ruf Guest

    > Tad McClellan [Tue, 11 Jul 2006 16:31:01 -0500]:
    >
    > Lukas Ruf <> wrote:
    > >> Tad McClellan [Tue, 11 Jul 2006 07:51:46 -0500]:
    > >>
    > >> Lukas Ruf <> wrote:
    > >>
    > >> > $hsh{$key} += @record{$index};
    > >>
    > >>
    > >> You should always enable warnings when developing Perl code.
    > >>

    > >
    > > meanwhile, I do.

    >
    >
    > You should not ignore the warnings that perl emits when developing
    > Perl code.
    >


    perl does not emit any warning anymore -- did you refer to my
    mistyping of

    > >> > $hsh{$key} += @record{$index};


    instead of

    $hsh{$key} += @record[$index];

    ??

    But anyway: I make use of sth like the following code:

    use strict;
    use warnings;

    my $inline = "";
    my @record;
    my %hsh = ();
    my $key = 0;
    my $index = 0;

    open(INFILE, "< $FILENAME") || die "can't open $FILENAME" ;
    print(STDERR "$FILENAME opened\n");

    while (defined($inline = <INFILE>))
    {
    chomp $inline; # remove '\n'
    $inline =~ s/\s+//g;
    @record = split(/,/, $inline);

    $record[$index]++;
    $hsh{$key} += $record[$index];
    }

    My questions:
    - is it correct that all newly created hash-values are initialized
    to zero by Perl automagically?
    - or do I need to initialize them manually? If so, how?

    Thanks in advance.

    wbr,
    Lukas
    --
    Lukas Ruf <http://www.lpr.ch> | Ad Personam
    rbacs <http://wiki.lpr.ch> | Restaurants, Bars and Clubs
    Raw IP <http://www.rawip.org> | Low Level Network Programming
    <http://lists.lpr.ch/muttprint> | muttprint mailing list
     
    Lukas Ruf, Jul 12, 2006
    #18
  19. Sherm Pendley

    Dr.Ruud Guest

    Lukas Ruf schreef:

    > But anyway: I make use of sth like the following code:


    Where both $key and $index remain 0?

    --
    Affijn, Ruud

    "Gewoon is een tijger."
     
    Dr.Ruud, Jul 12, 2006
    #19
  20. Lukas Ruf wrote:
    >
    > But anyway: I make use of sth like the following code:
    >
    > use strict;
    > use warnings;
    >
    > my $inline = "";
    > my @record;
    > my %hsh = ();
    > my $key = 0;
    > my $index = 0;
    >
    > open(INFILE, "< $FILENAME") || die "can't open $FILENAME" ;


    You should include the $! variable in the error message so you know *why* the
    file could not be opened.


    > print(STDERR "$FILENAME opened\n");
    >
    > while (defined($inline = <INFILE>))


    You should declare $inline here instead of above:

    while (defined(my $inline = <INFILE>))


    > {
    > chomp $inline; # remove '\n'


    Not necessarily, chomp removes whatever $/ contains ... sometimes, and
    sometimes it removes more than $/ contains (paragraph mode), and sometimes it
    removes nothing.


    > $inline =~ s/\s+//g;


    There, you didn't need chomp() anyway because \s includes \n.


    > @record = split(/,/, $inline);
    >
    > $record[$index]++;
    > $hsh{$key} += $record[$index];
    > }
    >
    > My questions:
    > - is it correct that all newly created hash-values are initialized
    > to zero by Perl automagically?


    No. Any key without a defined value has the value undef ... which perl
    automagically converts to zero when used in the correct context.


    > - or do I need to initialize them manually? If so, how?


    To initialize the values of a hash you first have to have the keys.

    $_ = 0 for values %hash;

    Will set every value in the hash to zero.



    John
    --
    use Perl;
    program
    fulfillment
     
    John W. Krahn, Jul 12, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mike
    Replies:
    5
    Views:
    2,148
  2. Andrew Thompson

    Update figures for Plug-In size?

    Andrew Thompson, Feb 20, 2004, in forum: Java
    Replies:
    3
    Views:
    352
    Matt Parker
    Feb 21, 2004
  3. Steve
    Replies:
    5
    Views:
    42,804
    Steve
    May 17, 2004
  4. wayne
    Replies:
    3
    Views:
    380
    Andrew Thompson
    Dec 1, 2004
  5. Scott Stark
    Replies:
    1
    Views:
    122
    Scott Stark
    Aug 3, 2003
Loading...

Share This Page