perl CGI file upload memory leak

L

Lionel Valero

Hello,

perl v5.8.0
CGI.pm-3.00
http-2.0.48

Everytime i upload a file via my perl script, the amount of bytes transfered
(size of the uploaded file) are not released from memory.

I closed the file handle of the new created file.

Any idea ?

Regards.
 
L

Lionel Valero

What else could i use ?

No i used to use mod_perl but i stopped thinking the problem comes from it.

Thanks.

Purl said:
Lionel Valero wrote:

(snipped)




Stop using Stein's module and discover if your
memory leak vanishes.

You are polling your memory usage before, during and after, yes?
Are you using mod_perl to run this script?


Purl Gurl

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

Lionel Valero

I have found something, it is very weird, when i upload a file, i see that the
system free ram deceases with the same amount of the file size uploaded, it is
released only when i delete this file via a perl script !!!!

Purl said:
Lionel Valero wrote:

(snipped)




Stop using Stein's module and discover if your
memory leak vanishes.

You are polling your memory usage before, during and after, yes?
Are you using mod_perl to run this script?


Purl Gurl

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

Lionel Valero

I have found the problem with the help of "perl -d", the file handle of STDOUT
remained opened :

while(<$file>) {
print LOCAL $_;
}
close (LOCAL);
close STDOUT; ====> what i must do


Lionel said:
I have found something, it is very weird, when i upload a file, i see
that the system free ram deceases with the same amount of the file size
uploaded, it is released only when i delete this file via a perl script
!!!!

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

Lionel Valero

Here it is :

#---------------------------------------------------------------#
# #
# sub fadd #
# #
#---------------------------------------------------------------#
sub fadd {

my $dm_sid = $_[0];
my $dm_user = $_[1];
my $dm_passwd = $_[2];
my $dm_dbh = $_[3];
my $dm_rep = $_[4];
my $dm_fic = $_[5];
my $ficgenpath;
my $nomfic;
my $file=$dm_fic;
my $bytesread;
my $maxsize=50000000;
my $OUTFILE;
my @msg;

if ($ENV{CONTENT_LENGTH} <= $maxsize)
{

......
push (@msg, 'Reception du fichier...');
if (open(OUTFILE, ">$ficgenpath"))
{
binmode(OUTFILE);
while ($bytesread = read($file, my $buffer, 1024)) {
print OUTFILE $buffer;
}
#while(<$file>) {
#print OUTFILE $_;
#}
close OUTFILE;
close STDOUT; == > I HAVE TO REMOVE IT, IT DOES NIOT WORK ANY MORE
# taille
my $inode = stat($ficgenpath);
my $size = $inode->size;
push (@msg, "$ficgenpath,size=$size");
if ($size != 0) {
push (@msg, "Creation du fichier ....
} #end fadd


Purl said:
Lionel Valero wrote:





You should not have to manually close STDOUT. I would
be interested in reading your complete code, least that
portion which would leave STDOUT open perhaps with
buffer content?

This hints of problems elsewhere.


Purl Gurl

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

Lionel Valero

sorry i did not posted the entire procedure (too long), here it is :

Thanks a lot for your analyse.

#---------------------------------------------------------------#
# #
# sub fadd #
# #
#---------------------------------------------------------------#
sub fadd {

my $dm_sid = $_[0];
my $dm_user = $_[1];
my $dm_passwd = $_[2];
my $dm_dbh = $_[3];
my $dm_rep = $_[4];
my $dm_fic = $_[5];
my $ficgenpath;
my $nomfic;
my $file=$dm_fic;
my $bytesread;
my $maxsize=50000000;
my $OUTFILE;
my @msg;

if ($ENV{CONTENT_LENGTH} <= $maxsize)
{
# test de la syntaxe du fichier : si vide
if ($dm_fic ne '')
{
#netscape + opera = sans path windows/unix
#ie = avec path windows/unix
if ( $file =~ /(\\|\/)/ )
{
#iexplorer avec path local filesystem
$dm_fic =~ s/^.*(\\|\/)//;
$nomfic=$dm_fic;
}
else
{
#netscape opera sans path local filesystem
$nomfic=$file;
}

push (@msg, "file=$file");
push (@msg, "nomfic=$nomfic");

# test de la syntaxe du fichier apres traitement du path
#if ($nomfic =~ /^[A-Za-z0-9\-_\.]+$/)
if ($nomfic)
{
$ficgenpath =
'/usr/local/apache2/htdocs/users/'.$dm_user.'/'.$dm_rep.'/'.$nomfic;
push (@msg, "Ajout de fichier $ficgenpath");
#teste si le fic existe --pb de cache web
my $sth = $dm_dbh->prepare("SELECT * FROM Files WHERE Fic_Nom='$nomfic' and
Fic_Dir_Nom='$dm_rep' and Fic_User_Id='$dm_user'");
my $retval=$sth->execute(); #retval=1=ok
$sth->finish();
if ($retval == 1)
{
push (@msg, 'Le fichier existe deja !');
}
else
{
# reception file system
push (@msg, 'Reception du fichier...');
if (open(OUTFILE, ">$ficgenpath"))
{
binmode(OUTFILE);
while ($bytesread = read($file, my $buffer, 1024)) {
print OUTFILE $buffer;
}
#while(<$file>) {
#print OUTFILE $_;
#}
close OUTFILE;
# taille
my $inode = stat($ficgenpath);
my $size = $inode->size;
push (@msg, "$ficgenpath,size=$size");
if ($size != 0) {
push (@msg, "Creation du fichier :$ficgenpath : ok...");
# ajoute le fichier
# calcul de la date date courante plus 1 mois
my $tm = localtime;
(my $cday, my $cmonth, my $cyear) = ($tm->mday, $tm->mon, $tm->year);
# bug si 31
if ($cday == 31) {$cday--;}
(my $eyear, my $emonth, my $eday) = Add_Delta_Days($cyear, $cmonth, $cday,
30);
$eyear=$eyear+1900;
$emonth=$emonth+1;
my $edate = $eyear.'-'.$emonth.'-'.$eday;
my $sql = qq/INSERT INTO Files
values('$dm_fic','$dm_rep','$dm_user','$edate','S')/;
eval { $dm_dbh->do( $sql )};
if ( $@ )
{
push (@msg, "Mysql error : Insert failed : $@");
}
else
{
push (@msg, 'Insert OK');
}
}
else
{
push (@msg, 'Erreur le fichier est vide !');
}
}
else
{
push (@msg, 'Erreur de creation de fichier !');
}
}
}
else
{
push (@msg, 'Erreur de syntaxe : le nom du fichier est incorrect !');
}
}
else
{
push (@msg, 'Erreur de syntaxe : le nom du fichier est incorrect !');
}
}
else
{
push (@msg , "Entrenoo : ATTENTION fichier trop gros : $ENV{CONTENT_LENGTH}
octets, limite de 50mo !");
}
noo::console(\@msg);
} #end fadd


Purl said:
Lionel Valero wrote:





You have not defined your $ficgenpath variable.
Your script fails at this point. It is not
possible for you to upload a file, as you
have previously indicated.


Purl Gurl

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

lionel valero

Hello,

I actually tried to write a simple code, which just handle the upload
task, i noticed the same behaviour, i tried to saturate the memory with
requests using big files, and apparently the OS manages to revover some
free mem but nor entirely, when i delete the uploaded file from a shell,
the free mem is recovered.

I will try your method and i will let you know, i assume that you deal
with the post content to recover the file, don't you ?

Regards.
 
U

Uri Guttman

lv> I actually tried to write a simple code, which just handle the upload
lv> task, i noticed the same behaviour, i tried to saturate the memory
lv> with requests using big files, and apparently the OS manages to
lv> revover some free mem but nor entirely, when i delete the uploaded
lv> file from a shell, the free mem is recovered.

do you realize that you are conversing with the resident troll in this
group? search google and you will see. the best advice is to ignore her
and seek help from someone else. just think, why would she tell you to
not use a standard module that comes with perl unless she was a moron?
now that you have been warned i won't tell you again but you should
research this and make up your own mind. and just wait until you
disagree with her or make a negative comment on her 'advice'.

uri
 
L

lionel valero

I tried your code, and after execution the system recovered the used
memory, but the parsing part of your code did not match the content of
the POST, i assume it is defined in a RFC, i will check. Thanks.
 
M

Mothra

Hello Lionel,

lionel valero said:
Hello,

I actually tried to write a simple code, which just handle the upload
task, i noticed the same behaviour, i tried to saturate the memory with
requests using big files, and apparently the OS manages to revover some
free mem but nor entirely, when i delete the uploaded file from a shell,
the free mem is recovered.

[snipped]

Ron Savage has some great tutorials on his web site. Even
one on how to upload files via CGI. Take a peek :)


http://savage.net.au/Perl-tutorials.html#tut_37

Mothra
 
L

Lionel Valero

Thanks for your help,

I tried the script, and the side effect is the same, the memory taken to cache
the upload file is not released after the end of the script.

Maybe i do not understand something about memory management !
Hello Lionel,

Hello,

I actually tried to write a simple code, which just handle the upload
task, i noticed the same behaviour, i tried to saturate the memory with
requests using big files, and apparently the OS manages to revover some
free mem but nor entirely, when i delete the uploaded file from a shell,
the free mem is recovered.


[snipped]

Ron Savage has some great tutorials on his web site. Even
one on how to upload files via CGI. Take a peek :)


http://savage.net.au/Perl-tutorials.html#tut_37

Mothra

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 
L

Lionel Valero

Hi,

I have my answer, the crunching of the free memory is a part of the behaviour of
Linux, when checking with the free command, the interesting line to read is "-/+
buffers/cache" no matter if the "free Mem:" is low.

Regards.
Hello Lionel,

Hello,

I actually tried to write a simple code, which just handle the upload
task, i noticed the same behaviour, i tried to saturate the memory with
requests using big files, and apparently the OS manages to revover some
free mem but nor entirely, when i delete the uploaded file from a shell,
the free mem is recovered.


[snipped]

Ron Savage has some great tutorials on his web site. Even
one on how to upload files via CGI. Take a peek :)


http://savage.net.au/Perl-tutorials.html#tut_37

Mothra

--
-=O=------------------------------------------=O=-
Lionel Valéro
Analyste Informatique Département Génie Chimique
École Polytechnique de Montréal
C.P. 6079, succ. centre-ville
Montréal (Québec) H3C 3A7
Tel: (514) 340 - 4711 # 4805 / C552
Fax: (514) 340 - 4159
-=O=------------------------------------------=O=-
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,766
Messages
2,569,569
Members
45,042
Latest member
icassiem

Latest Threads

Top