fastest way to allocate memory ?

[georg.heiss]

Hi, i try to allocate 10GB of memory on my box and it takes about 27
seconds.
Is there a faster way to do this?

my $gras = "A" x (1024 * 1024 * 10000);
my $needle = "B";
print "\nAllocated " . length($gras) . " byte buffer\n";
$gras = $gras.$needle;
print "Allocated " . length($gras) . " byte buffer\n";
if ($gras ~~ /$needle/) {print "found needle\n"; }
###

$ time perl memtake.pl &
Allocated 10485760000 byte buffer
Allocated 10485760001 byte buffer
found needle

real 0m27.280s
user 0m13.429s
sys 0m13.839s

 

[xhoster]

Hi, i try to allocate 10GB of memory on my box and it takes about 27
seconds.

You are actually allocating 20GB of memory, as first it builds a temporary
string and then it copies it. And it isn't the allocation, but the
initialization and copying that takes the time. (And then it copies it
again, but presumable it either reuses the memory used for the first
temporary variable, or optimizes that copy away.)

Is there a faster way to do this?

Do what? Based on the code you show, we could simply reduce your
entire program to 'print "found needle\n"', as that what the end result
is. Which parts of the code you show are window dressing and which parts
are the core?

my $gras = "A" x (1024 * 1024 * 10000);
my $needle = "B";
print "\nAllocated " . length($gras) . " byte buffer\n";
$gras = $gras.$needle;
print "Allocated " . length($gras) . " byte buffer\n";
if ($gras ~~ /$needle/) {print "found needle\n"; }

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Steve Roscio]

To preallocate memory, I use vec() to set a byte out at the end of the
chunk of memory I want.

- Steve

 

[Uri Guttman]

SR> To preallocate memory, I use vec() to set a byte out at the end of the
SR> chunk of memory I want.

you could just as easily use 4 arg substr (or the slower lvalue substr)
which would be simpler and much easier to read.

uri

 

[Hans Mulder]

SR> To preallocate memory, I use vec() to set a byte out at the end of the
SR> chunk of memory I want.

you could just as easily use 4 arg substr (or the slower lvalue substr)
which would be simpler and much easier to read.

I get an error "substr outside of string" when I do that.

I would use $var = 'x' x $size, when I need a large chunk of memory.
(I hope the '=' operator can set $var without copying the string.)

-- HansM

 

[Uri Guttman]

HM> I get an error "substr outside of string" when I do that.

HM> I would use $var = 'x' x $size, when I need a large chunk of memory.
HM> (I hope the '=' operator can set $var without copying the string.)

interesting. i would have though extending with substr would work. it
warns for lvalue substr as well. i may be confusing it with extending
arrays which is warning free.

this still begs the question as to why the OP wants a buffer that
large. it smells to me like a poor design decision.

uri

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
(e-mail address removed)

Hi, i try to allocate 10GB of memory on my box and it takes about 27
seconds.

You are allocating 30000MiB, not 10GB (and not 20GB, as somebody
mentioned) + malloc() overhead.

my $gras = "A" x (1024 * 1024 * 10000);

RHS is constant, thus computed at compile time ==> 10000GiB.
LHS taks another 10000GiB.

my $needle = "B";
print "\nAllocated " . length($gras) . " byte buffer\n";
$gras = $gras.$needle;

RHS takes another 10000GiB.

Is there a faster way to do this?

my $gras = "A";
$gras x= 1024 * 1024 * 10000;
$gras .= $needle;

would take 3x smaller memory footprint.

But if the *allocation* is a bottleneck, make sure your perl is
compiled with "my" malloc. It takes major care to make allocation as
quick as possible...

perl -V:".*malloc.*"

Hope this helps,
ILya

 

[Peter J. Holzer]

From which Perl version this x= operator work (5.8 or 5.10)?

At least 5.005_03:

% /usr/bin/perl -le '$x = "a"; $x x= 5; print $x'
aaaaa
% /usr/bin/perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
Platform:
osname=linux, osvers=2.2.5-22smp, archname=i386-linux
uname='linux porky.devel.redhat.com 2.2.5-22smp #1 smp wed jun 2 09:11:51 edt 1999 i686 unknown '

hp

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Petr Vileta \"fidokomik\"

From which Perl version this x= operator work (5.8 or 5.10)?

The first I saw was v4.010. I do not remember the timeline of Perl,
but I would expect it would be there from v2.early or somesuch...

Yours,
Ilya

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle, and as long as there is room to
grow $gras in-place it does not need to be copied.

Interesting... What about

$gras = $pre.$gras;
$gras = $pre.$gras.$needle;

when there is enough place on the left (and right)? This would be
much more useful (since one can always use explicit .= for the initial
construct)...

Yours,
Ilya

 

[xhoster]

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle, and as long as there is room to
grow $gras in-place it does not need to be copied.

Interesting... What about

$gras = $pre.$gras;

Perl doesn't seem to reserve any memory the front of the string
automatically. If $gras was previously front-shortened, for example by
using

substr $gras, 0, 10, "";

Then it seems that $gras = $pre.$gras; is optimized to edit in place rather
than copying.

Oddly, this works even if $pre is longer than the amount of
front-shortening. Apparently in that case it triggers code that slides the
contents of $gras down enough to make room at the front, without making an
entirely new copy. It seems that it could do this same thing if no
front-shortening had occurred, but for some reason it doesn't.

Also, the use of 30 Gig for $gras = $pre.$gras; (when there is no front
shortening) is transient. When the old contents of $gras are freed, my
implementation of perl actually returns that memory to the OS and shrinks
back to 20G

$gras = $pre.$gras.$needle;

That too seems to trigger non-copying in-place editing of $gras, provided
it had previously been front-shortened.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Leon Timmermans]

You can try using the mmap(2) syscall from perl. There is
a module that you can try from CPAN:

http://search.cpan.org/~swalters/Sys-Mmap-0.13/Mmap.pm

I don't know another options, since the POSIX module can't
do this task and mmap(2) isn't commint from POSIX, but is present in
SUSv3:

http://www.opengroup.org/onlinepubs/000095399/functions/mmap.html

I would recommend Sys::Mmap::Simple instead (but since I wrote it I'm not
exactly unbiased), in particular its function map_anonymous which is made
exactly for this kind of situation (more reasons to prefer it are in the
perldoc).

Regards,

Leon Timmermans

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle

No, it is not:

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
18184 free: 111 114 56 21 6 2 2 2 1 0 0 0 0 0 0 0 0
408 21 51 30 18
650612 used: 144 140 70 134 10 6 6 4 1 16 1 0 0 0 0 0 1
103 149 544 138 7
Total sbrk(): 669696/37:186. Odd ends: pad+heads+chain+tail: 0+900+0+0.

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; $s = $s.q(b); 1"

Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
17892 free: 111 114 49 20 6 2 2 2 1 0 0 0 0 0 0 0 0
407 21 51 30 18
1179284 used: 144 140 77 135 10 6 6 4 1 16 1 0 0 0 0 0 2
104 149 544 138 7
Total sbrk(): 1198080/38:187. Odd ends: pad+heads+chain+tail: 0+904+0+0.

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; $s = q(b); 1"

Memory allocation statistics after execution: (buckets 4(4)..528380(524288)
17988 free: 111 114 52 20 6 2 2 2 1 0 0 0 0 0 0 0 0
407 21 51 30 18
650808 used: 144 140 74 135 10 6 6 4 1 16 1 0 0 0 0 0 1
104 149 544 138 7
Total sbrk(): 669696/37:186. Odd ends: pad+heads+chain+tail: 0+900+0+0.

Hope this helps,
Ilya

 

[xhoster]

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle

No, it is not:

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

The optimization only seems to happen or lexical variables.

What happens if you "my" $s? (None of the perls I have ready access to
support PERL_DEBUG_MSTATS.)

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt

my $needle = "B";
$gras = $gras.$needle;

RHS takes another 10000GiB.

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle

No, it is not:

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

The optimization only seems to happen or lexical variables.

True; it looks like a spin-off of my older "lexicals as targets"
optimization... This is visible otherwise:

perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s .= q(b); 1"
....
a <;> nextstate(main 2 -e:1) v ->b
d <2> concat[t3] vKS/2 ->e
b <0> padsv[$s:1,2] sRM ->c
c <$> const(PV "b") s ->d
e <;> nextstate(main 2 -e:1) v ->f

perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s = $s.q(b); 1"
a <;> nextstate(main 2 -e:1) v ->b
d <2> concat[$s:1,2] sK/TARGMY,2 ->e
b <0> padsv[$s:1,2] s ->c
c <$> const(PV "b") s ->d
e <;> nextstate(main 2 -e:1) v ->f


The opflags for "b" difer by "RM" (I do not know the meaning), and
opflags for "d" are also subtly different...

Thanks for your research,
Ilya

 

[sln]

[A complimentary Cc of this posting was sent to

[A complimentary Cc of this posting was sent to
Xho Jingleheimerschmidt
<[email protected]>:

my $needle = "B";
$gras = $gras.$needle;

RHS takes another 10000GiB.

It seems that, at least as of 5.8.8, this construct is optimized
to be about the same as $gras.=$needle

No, it is not:

env PERL_DEBUG_MSTATS=1 perl -e "$s=q(a); $s x= 5e5; 1"

The optimization only seems to happen or lexical variables.

True; it looks like a spin-off of my older "lexicals as targets"
optimization... This is visible otherwise:

perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s .= q(b); 1"
...
a <;> nextstate(main 2 -e:1) v ->b
d <2> concat[t3] vKS/2 ->e
b <0> padsv[$s:1,2] sRM ->c
c <$> const(PV "b") s ->d
e <;> nextstate(main 2 -e:1) v ->f

perl -MO=Concise -e "my $s=q(a); $s x= 5e5; $s = $s.q(b); 1"
a <;> nextstate(main 2 -e:1) v ->b
d <2> concat[$s:1,2] sK/TARGMY,2 ->e
b <0> padsv[$s:1,2] s ->c
c <$> const(PV "b") s ->d
e <;> nextstate(main 2 -e:1) v ->f


The opflags for "b" difer by "RM" (I do not know the meaning), and
opflags for "d" are also subtly different...

Thanks for your research,
Ilya

All this cryptic in-between code makes me laff.
The QuakeC compiler is exactly the same.
You have a good niche there Ilya, a place where nobody
but you understands. I hope they don't stop playing Quake for
your sake.

sln