P
peter pilsl
I've a perl-daemon that should run for weeks and rarely serves requests.
When it serves a request, then it needs huge amount of memory for a
short time.
My problem is that perl is not giving back this memory to the OS/other
processes/its own relatives when it finished its work and is just
waiting for the next request. Even not if the whole system is completely
running out of memory and the machine is going dead-swaping-mode.
A short example (terminal-based single-thread-deamon) is illustrating my
problem:
-------------------memd_test.pl--------
#!/usr/bin/perl -w
use strict;
use warnings;
$|=1;
sub Run {
my @x;
foreach (0..200000) {
push(@x,[33,44,55,66,77,88,99,22,11,33,44,55,66,77,88,99,22,11,33,44,55,66,77,88,99,22,11]);
}
}
my $c;
while (<STDIN>) {
print $c++;
Run();
}
------------------------
each CR on stdin will run the short subroutine that does nothing but
claiming lot of memory and ... unfortunately ... not releasing it to the
OS after.
If you start this example-programm a few times and feed it with 10 CR
then you end up with:
#top
2461 root 9 0 122m 122m 1228 T 0.0 6.0 0:02.78 0 121m
memd_test2.pl
2492 root 9 0 122m 122m 1228 T 0.0 6.0 0:02.77 0 121m
memd_test2.pl
2564 root 9 0 122m 122m 1228 T 0.0 6.0 0:00.98 0 121m
memd_test2.pl
2570 root 9 0 122m 122m 1228 T 0.0 6.0 0:01.87 0 121m
memd_test2.pl
note that my programm is not memleaking here, cause it will reuse the
same memory internally again. It simply does not give it back to the OS.
I have some experience in programming daemons and working with mod_perl
so I learned a bit about memory-stuff and I know that its a tricky topic
and things are not as easy at it seems. I always found a solution but
this time its different. I cant get the stuff under control.
One single request that needs 200m for about 2 seconds will take this
200m away from the system for the rest of its life.
I need to implement the process as daemon, cause the startup-time of the
process is very expensive and it might happen that multiple requests
will come in at the same time.
I know that this maybe is not perl's fault, but has lot to do with the
OS, but ... what can I do?
I encounter this struggle with perl 5.8.5 on linux 2.4 and 5.8.7 on
linux 2.6
thnx,
peter
# perl -VV
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=linux, osvers=2.4.24, archname=i686-linux
uname='linux goldfisch.at 2.4.24 #9 wed mar 10 22:29:04 cet 2004
i686 unknown '
config_args='-de'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='2.96 20000731 (Mandrake Linux 8.1
2.96-0.62mdk)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.2.4'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Sep 21 2004 11:47:55
@INC:
/usr/local/lib/perl5/5.8.5/i686-linux
/usr/local/lib/perl5/5.8.5
/usr/local/lib/perl5/site_perl/5.8.5/i686-linux
/usr/local/lib/perl5/site_perl/5.8.5
/usr/local/lib/perl5/site_perl/5.8.0/i686-linux
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl
When it serves a request, then it needs huge amount of memory for a
short time.
My problem is that perl is not giving back this memory to the OS/other
processes/its own relatives when it finished its work and is just
waiting for the next request. Even not if the whole system is completely
running out of memory and the machine is going dead-swaping-mode.
A short example (terminal-based single-thread-deamon) is illustrating my
problem:
-------------------memd_test.pl--------
#!/usr/bin/perl -w
use strict;
use warnings;
$|=1;
sub Run {
my @x;
foreach (0..200000) {
push(@x,[33,44,55,66,77,88,99,22,11,33,44,55,66,77,88,99,22,11,33,44,55,66,77,88,99,22,11]);
}
}
my $c;
while (<STDIN>) {
print $c++;
Run();
}
------------------------
each CR on stdin will run the short subroutine that does nothing but
claiming lot of memory and ... unfortunately ... not releasing it to the
OS after.
If you start this example-programm a few times and feed it with 10 CR
then you end up with:
#top
2461 root 9 0 122m 122m 1228 T 0.0 6.0 0:02.78 0 121m
memd_test2.pl
2492 root 9 0 122m 122m 1228 T 0.0 6.0 0:02.77 0 121m
memd_test2.pl
2564 root 9 0 122m 122m 1228 T 0.0 6.0 0:00.98 0 121m
memd_test2.pl
2570 root 9 0 122m 122m 1228 T 0.0 6.0 0:01.87 0 121m
memd_test2.pl
note that my programm is not memleaking here, cause it will reuse the
same memory internally again. It simply does not give it back to the OS.
I have some experience in programming daemons and working with mod_perl
so I learned a bit about memory-stuff and I know that its a tricky topic
and things are not as easy at it seems. I always found a solution but
this time its different. I cant get the stuff under control.
One single request that needs 200m for about 2 seconds will take this
200m away from the system for the rest of its life.
I need to implement the process as daemon, cause the startup-time of the
process is very expensive and it might happen that multiple requests
will come in at the same time.
I know that this maybe is not perl's fault, but has lot to do with the
OS, but ... what can I do?
I encounter this struggle with perl 5.8.5 on linux 2.4 and 5.8.7 on
linux 2.6
thnx,
peter
# perl -VV
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=linux, osvers=2.4.24, archname=i686-linux
uname='linux goldfisch.at 2.4.24 #9 wed mar 10 22:29:04 cet 2004
i686 unknown '
config_args='-de'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='2.96 20000731 (Mandrake Linux 8.1
2.96-0.62mdk)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.2.4'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Sep 21 2004 11:47:55
@INC:
/usr/local/lib/perl5/5.8.5/i686-linux
/usr/local/lib/perl5/5.8.5
/usr/local/lib/perl5/site_perl/5.8.5/i686-linux
/usr/local/lib/perl5/site_perl/5.8.5
/usr/local/lib/perl5/site_perl/5.8.0/i686-linux
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl