fastest way to allocate memory ?

Discussion in 'Perl Misc' started by georg.heiss@gmx.de, Jan 16, 2009.

  1. Guest

    Hi, i try to allocate 10GB of memory on my box and it takes about 27
    seconds.
    Is there a faster way to do this?

    my $gras = "A" x (1024 * 1024 * 10000);
    my $needle = "B";
    print "\nAllocated " . length($gras) . " byte buffer\n";
    $gras = $gras.$needle;
    print "Allocated " . length($gras) . " byte buffer\n";
    if ($gras ~~ /$needle/) {print "found needle\n"; }
    ###

    $ time perl memtake.pl &
    Allocated 10485760000 byte buffer
    Allocated 10485760001 byte buffer
    found needle

    real 0m27.280s
    user 0m13.429s
    sys 0m13.839s
     
    , Jan 16, 2009
    #1
    1. Advertising

  2. Guest

    "" <> wrote:
    > Hi, i try to allocate 10GB of memory on my box and it takes about 27
    > seconds.


    You are actually allocating 20GB of memory, as first it builds a temporary
    string and then it copies it. And it isn't the allocation, but the
    initialization and copying that takes the time. (And then it copies it
    again, but presumable it either reuses the memory used for the first
    temporary variable, or optimizes that copy away.)

    > Is there a faster way to do this?


    Do what? Based on the code you show, we could simply reduce your
    entire program to 'print "found needle\n"', as that what the end result
    is. Which parts of the code you show are window dressing and which parts
    are the core?

    > my $gras = "A" x (1024 * 1024 * 10000);
    > my $needle = "B";
    > print "\nAllocated " . length($gras) . " byte buffer\n";
    > $gras = $gras.$needle;
    > print "Allocated " . length($gras) . " byte buffer\n";
    > if ($gras ~~ /$needle/) {print "found needle\n"; }


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Jan 16, 2009
    #2
    1. Advertising

  3. Steve Roscio Guest

    To preallocate memory, I use vec() to set a byte out at the end of the
    chunk of memory I want.

    - Steve
     
    Steve Roscio, Jan 16, 2009
    #3
  4. Uri Guttman Guest

    >>>>> "SR" == Steve Roscio <> writes:

    SR> To preallocate memory, I use vec() to set a byte out at the end of the
    SR> chunk of memory I want.

    you could just as easily use 4 arg substr (or the slower lvalue substr)
    which would be simpler and much easier to read.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Free Perl Training --- http://perlhunter.com/college.html ---------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Jan 16, 2009
    #4
  5. Hans Mulder Guest

    Uri Guttman wrote:
    >>>>>> "SR" == Steve Roscio <> writes:

    >
    > SR> To preallocate memory, I use vec() to set a byte out at the end of the
    > SR> chunk of memory I want.
    >
    > you could just as easily use 4 arg substr (or the slower lvalue substr)
    > which would be simpler and much easier to read.


    I get an error "substr outside of string" when I do that.

    I would use $var = 'x' x $size, when I need a large chunk of memory.
    (I hope the '=' operator can set $var without copying the string.)

    -- HansM
     
    Hans Mulder, Jan 16, 2009
    #5
  6. Uri Guttman Guest

    >>>>> "HM" == Hans Mulder <> writes:

    HM> Uri Guttman wrote:
    >>>>>>> "SR" == Steve Roscio <> writes:

    SR> To preallocate memory, I use vec() to set a byte out at the
    >> end of the

    SR> chunk of memory I want.
    >> you could just as easily use 4 arg substr (or the slower lvalue
    >> substr)
    >> which would be simpler and much easier to read.


    HM> I get an error "substr outside of string" when I do that.

    HM> I would use $var = 'x' x $size, when I need a large chunk of memory.
    HM> (I hope the '=' operator can set $var without copying the string.)

    interesting. i would have though extending with substr would work. it
    warns for lvalue substr as well. i may be confusing it with extending
    arrays which is warning free.

    this still begs the question as to why the OP wants a buffer that
    large. it smells to me like a poor design decision.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Free Perl Training --- http://perlhunter.com/college.html ---------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Jan 17, 2009
    #6
  7. [A complimentary Cc of this posting was sent to

    <>], who wrote in article <>:
    > Hi, i try to allocate 10GB of memory on my box and it takes about 27
    > seconds.


    You are allocating 30000MiB, not 10GB (and not 20GB, as somebody
    mentioned) + malloc() overhead.

    > my $gras = "A" x (1024 * 1024 * 10000);


    RHS is constant, thus computed at compile time ==> 10000GiB.
    LHS taks another 10000GiB.

    > my $needle = "B";
    > print "\nAllocated " . length($gras) . " byte buffer\n";
    > $gras = $gras.$needle;


    RHS takes another 10000GiB.

    > Is there a faster way to do this?


    my $gras = "A";
    $gras x= 1024 * 1024 * 10000;
    $gras .= $needle;

    would take 3x smaller memory footprint.

    But if the *allocation* is a bottleneck, make sure your perl is
    compiled with "my" malloc. It takes major care to make allocation as
    quick as possible...

    perl -V:".*malloc.*"

    Hope this helps,
    ILya
     
    Ilya Zakharevich, Jan 18, 2009
    #7
  8. On 2009-01-18 13:53, Petr Vileta "fidokomik" <> wrote:
    > "Ilya Zakharevich" <> píše v diskusním příspěvku
    > news:gkupi0$m2m$...
    >
    >> $gras x= 1024 * 1024 * 10000;

    >
    > From which Perl version this x= operator work (5.8 or 5.10)?


    At least 5.005_03:

    % /usr/bin/perl -le '$x = "a"; $x x= 5; print $x'
    aaaaa
    % /usr/bin/perl -V
    Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
    Platform:
    osname=linux, osvers=2.2.5-22smp, archname=i386-linux
    uname='linux porky.devel.redhat.com 2.2.5-22smp #1 smp wed jun 2 09:11:51 edt 1999 i686 unknown '

    hp
     
    Peter J. Holzer, Jan 18, 2009
    #8
  9. [A complimentary Cc of this posting was sent to
    Petr Vileta \"fidokomik\"
    <>], who wrote in article <gkvcgq$30kg$>:
    > "Ilya Zakharevich" <> píše v diskusním pøíspìvku
    > news:gkupi0$m2m$...
    >
    > > $gras x= 1024 * 1024 * 10000;

    >
    > From which Perl version this x= operator work (5.8 or 5.10)?


    The first I saw was v4.010. I do not remember the timeline of Perl,
    but I would expect it would be there from v2.early or somesuch...

    Yours,
    Ilya
     
    Ilya Zakharevich, Jan 19, 2009
    #9
  10. [A complimentary Cc of this posting was sent to
    Xho Jingleheimerschmidt
    <>], who wrote in article <4973797e$0$25707$>:
    > >> $gras = $gras.$needle;

    > >
    > > RHS takes another 10000GiB.

    >
    > It seems that, at least as of 5.8.8, this construct is optimized
    > to be about the same as $gras.=$needle, and as long as there is room to
    > grow $gras in-place it does not need to be copied.


    Interesting... What about

    $gras = $pre.$gras;
    $gras = $pre.$gras.$needle;

    when there is enough place on the left (and right)? This would be
    much more useful (since one can always use explicit .= for the initial
    construct)...

    Yours,
    Ilya
     
    Ilya Zakharevich, Jan 19, 2009
    #10
  11. Guest

    Ilya Zakharevich <> wrote:
    > [A complimentary Cc of this posting was sent to
    > Xho Jingleheimerschmidt
    > <>], who wrote in article
    > <4973797e$0$25707$>:
    > > >> $gras = $gras.$needle;
    > > >
    > > > RHS takes another 10000GiB.

    > >
    > > It seems that, at least as of 5.8.8, this construct is optimized
    > > to be about the same as $gras.=$needle, and as long as there is room to
    > > grow $gras in-place it does not need to be copied.

    >
    > Interesting... What about
    >
    > $gras = $pre.$gras;


    Perl doesn't seem to reserve any memory the front of the string
    automatically. If $gras was previously front-shortened, for example by
    using

    substr $gras, 0, 10, "";

    Then it seems that $gras = $pre.$gras; is optimized to edit in place rather
    than copying.

    Oddly, this works even if $pre is longer than the amount of
    front-shortening. Apparently in that case it triggers code that slides the
    contents of $gras down enough to make room at the front, without making an
    entirely new copy. It seems that it could do this same thing if no
    front-shortening had occurred, but for some reason it doesn't.

    Also, the use of 30 Gig for $gras = $pre.$gras; (when there is no front
    shortening) is transient. When the old contents of $gras are freed, my
    implementation of perl actually returns that memory to the OS and shrinks
    back to 20G


    > $gras = $pre.$gras.$needle;


    That too seems to trigger non-copying in-place editing of $gras, provided
    it had previously been front-shortened.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Jan 19, 2009
    #11
  12. On Fri, 16 Jan 2009 05:47:41 -0300, Daniel Molina Wegener wrote:
    > You can try using the mmap(2) syscall from perl. There is
    > a module that you can try from CPAN:
    >
    > http://search.cpan.org/~swalters/Sys-Mmap-0.13/Mmap.pm
    >
    > I don't know another options, since the POSIX module can't
    > do this task and mmap(2) isn't commint from POSIX, but is present in
    > SUSv3:
    >
    > http://www.opengroup.org/onlinepubs/000095399/functions/mmap.html
    >


    I would recommend Sys::Mmap::Simple instead (but since I wrote it I'm not
    exactly unbiased), in particular its function map_anonymous which is made
    exactly for this kind of situation (more reasons to prefer it are in the
    perldoc).

    Regards,

    Leon Timmermans
     
    Leon Timmermans, Jan 20, 2009
    #12
  13. [A complimentary Cc of this posting was sent to
    Xho Jingleheimerschmidt
    <>], who wrote in article <4973797e$0$25707$>:

    > >> my $needle = "B";
    > >> $gras = $gras.$needle;


    > > RHS takes another 10000GiB.


    > It seems that, at least as of 5.8.8, this construct is optimized
    > to be about the same as $gras.=$needle


    No, it is not:

    > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

    Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
    18184 free: 111 114 56 21 6 2 2 2 1 0 0 0 0 0 0 0 0
    408 21 51 30 18
    650612 used: 144 140 70 134 10 6 6 4 1 16 1 0 0 0 0 0 1
    103 149 544 138 7
    Total sbrk(): 669696/37:186. Odd ends: pad+heads+chain+tail: 0+900+0+0.

    > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; $s = $s.q(b); 1"

    Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
    17892 free: 111 114 49 20 6 2 2 2 1 0 0 0 0 0 0 0 0
    407 21 51 30 18
    1179284 used: 144 140 77 135 10 6 6 4 1 16 1 0 0 0 0 0 2
    104 149 544 138 7
    Total sbrk(): 1198080/38:187. Odd ends: pad+heads+chain+tail: 0+904+0+0.

    > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; $s = q(b); 1"

    Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
    17988 free: 111 114 52 20 6 2 2 2 1 0 0 0 0 0 0 0 0
    407 21 51 30 18
    650808 used: 144 140 74 135 10 6 6 4 1 16 1 0 0 0 0 0 1
    104 149 544 138 7
    Total sbrk(): 669696/37:186. Odd ends: pad+heads+chain+tail: 0+900+0+0.

    Hope this helps,
    Ilya
     
    Ilya Zakharevich, Jan 20, 2009
    #13
  14. Guest

    Ilya Zakharevich <> wrote:
    > [A complimentary Cc of this posting was sent to
    > Xho Jingleheimerschmidt
    > <>], who wrote in article
    > <4973797e$0$25707$>:
    >
    > > >> my $needle = "B";
    > > >> $gras = $gras.$needle;

    >
    > > > RHS takes another 10000GiB.

    >
    > > It seems that, at least as of 5.8.8, this construct is optimized
    > > to be about the same as $gras.=$needle

    >
    > No, it is not:
    >
    > > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"


    The optimization only seems to happen or lexical variables.

    What happens if you "my" $s? (None of the perls I have ready access to
    support PERL_DEBUG_MSTATS.)

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Jan 21, 2009
    #14
  15. [A complimentary Cc of this posting was sent to

    <>], who wrote in article <20090120212109.240$>:
    > Ilya Zakharevich <> wrote:
    > > [A complimentary Cc of this posting was sent to
    > > Xho Jingleheimerschmidt
    > > <>], who wrote in article
    > > <4973797e$0$25707$>:
    > >
    > > > >> my $needle = "B";
    > > > >> $gras = $gras.$needle;

    > >
    > > > > RHS takes another 10000GiB.

    > >
    > > > It seems that, at least as of 5.8.8, this construct is optimized
    > > > to be about the same as $gras.=$needle

    > >
    > > No, it is not:
    > >
    > > > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

    >
    > The optimization only seems to happen or lexical variables.


    True; it looks like a spin-off of my older "lexicals as targets"
    optimization... This is visible otherwise:

    perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s .= q(b); 1"
    ....
    a <;> nextstate(main 2 -e:1) v ->b
    d <2> concat[t3] vKS/2 ->e
    b <0> padsv[$s:1,2] sRM ->c
    c <$> const(PV "b") s ->d
    e <;> nextstate(main 2 -e:1) v ->f

    perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s = $s.q(b); 1"
    a <;> nextstate(main 2 -e:1) v ->b
    d <2> concat[$s:1,2] sK/TARGMY,2 ->e
    b <0> padsv[$s:1,2] s ->c
    c <$> const(PV "b") s ->d
    e <;> nextstate(main 2 -e:1) v ->f


    The opflags for "b" difer by "RM" (I do not know the meaning), and
    opflags for "d" are also subtly different...

    Thanks for your research,
    Ilya
     
    Ilya Zakharevich, Jan 21, 2009
    #15
  16. Guest

    On Wed, 21 Jan 2009 04:11:33 +0000 (UTC), Ilya Zakharevich <> wrote:

    >[A complimentary Cc of this posting was sent to
    >
    ><>], who wrote in article <20090120212109.240$>:
    >> Ilya Zakharevich <> wrote:
    >> > [A complimentary Cc of this posting was sent to
    >> > Xho Jingleheimerschmidt
    >> > <>], who wrote in article
    >> > <4973797e$0$25707$>:
    >> >
    >> > > >> my $needle = "B";
    >> > > >> $gras = $gras.$needle;
    >> >
    >> > > > RHS takes another 10000GiB.
    >> >
    >> > > It seems that, at least as of 5.8.8, this construct is optimized
    >> > > to be about the same as $gras.=$needle
    >> >
    >> > No, it is not:
    >> >
    >> > > env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

    >>
    >> The optimization only seems to happen or lexical variables.

    >
    >True; it looks like a spin-off of my older "lexicals as targets"
    >optimization... This is visible otherwise:
    >
    > perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s .= q(b); 1"
    >...
    >a <;> nextstate(main 2 -e:1) v ->b
    >d <2> concat[t3] vKS/2 ->e
    >b <0> padsv[$s:1,2] sRM ->c
    >c <$> const(PV "b") s ->d
    >e <;> nextstate(main 2 -e:1) v ->f
    >
    > perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s = $s.q(b); 1"
    >a <;> nextstate(main 2 -e:1) v ->b
    >d <2> concat[$s:1,2] sK/TARGMY,2 ->e
    >b <0> padsv[$s:1,2] s ->c
    >c <$> const(PV "b") s ->d
    >e <;> nextstate(main 2 -e:1) v ->f
    >
    >
    >The opflags for "b" difer by "RM" (I do not know the meaning), and
    >opflags for "d" are also subtly different...
    >
    >Thanks for your research,
    >Ilya


    All this cryptic in-between code makes me laff.
    The QuakeC compiler is exactly the same.
    You have a good niche there Ilya, a place where nobody
    but you understands. I hope they don't stop playing Quake for
    your sake.

    sln
     
    , Jan 21, 2009
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roedy Green
    Replies:
    3
    Views:
    3,257
    Roedy Green
    Aug 14, 2003
  2. sks_cpp
    Replies:
    7
    Views:
    391
    Jerry Coffin
    Jul 4, 2003
  3. Curt
    Replies:
    37
    Views:
    1,190
    Alexander Terekhov
    Jul 26, 2003
  4. Dmytro Bablinyuk
    Replies:
    4
    Views:
    382
    peter koch
    Sep 5, 2006
  5. castironpi
    Replies:
    5
    Views:
    350
    castironpi
    Aug 24, 2008
Loading...

Share This Page