File handle re-use woes using pipe within a loop

M

Mark Gowans

Hi There,

I've got a perl script which very simply does something along the
lines of:

while (TRUE) {

for $i = 1 to n.. {

$rVar = "r$i"
$wVar = "w$i"
pipe($rVar, $wVar)

fork()

}

if Parent process {
close $wVar
read from $rVar
} else {
close $rVar
write to $wVar
}

sleep 10
}

For the first iteration of the while loop - all works hunky dory. The
trouble is, on the 2nd interation the pipe command fails with
something along the line of "print on closed file handle".

I'm guessing this is because for some reason, I cant re-use my
file-handle names (and because I've expressly closed them it fails).

So.. how do I get around this? I guess I'd like something that does:

....
&forget_about_all_previous_file_handle_instances()
pipe($rVar,$wVar)

...any help greatly appreciated!

Many thanks,

Mark
 
C

ctcgag

Hi There,

I've got a perl script which very simply does something along the
lines of:

while (TRUE) {

for $i = 1 to n.. {

There is probably something wrong with your code. Why don't you show
us your code?
$rVar = "r$i"
$wVar = "w$i"
pipe($rVar, $wVar)

There is probably something wrong with your code. Why don't you show
us your code?

There is probably something wrong with your code. Why don't you show
us your code?

}

if Parent process {

There is probably something wrong with your code. Why don't you show
us your code?
close $wVar
read from $rVar
} else {
close $rVar
write to $wVar

There is probably something wrong with your code. Why don't you show
us your code?
}

sleep 10
}

For the first iteration of the while loop - all works hunky dory.

I find that hard to believe. Your actual code must have very little
resemblance to what you posted.
The
trouble is, on the 2nd interation the pipe command fails with
something along the line of "print on closed file handle".

I find that hard to believe, also.

I'd recommend using strict and lexical file handles, and then if you still
need help, then try writing example code that works and demonstrates what
you says it does.

Xho
 
M

Mark Gowans

Crikey, quite the synic aren't you?! :eek:)

I recon its pretty similar to the pseudocode below, but here's a bit
of perl to demonstrate exactly the below..

#!/usr/local/bin/perl5 -w
while (TRUE) {
my $pid = $$;
my @children = ();
my $parent = 0;
@LIST = ("a", "b", "c", "d", "e",);
foreach $item(@LIST) {
$rVar = "r$item";
$wVar = "w$item";
pipe ($rVar, $wVar);
my $newPID = fork();

if (not defined $newPID) {
die "Fork didn't work\n";
} elsif ( $newPID == 0 ) {
$parent = $pid;
$pid = $$;
@children = ();
@LIST = ();
} else {
push @children, $newPID;
$childpipe{$newPID} = $item;
}
}

if ($parent) {
#child process..
close $rVar;
print $wVar $$;
exit (0);
} else {
#parent process..
while (my $child = shift @children) {
my $justDied = waitpid ($child,0);
$rVar = "r$childpipe{$child}";
$wVar = "w$childpipe{$child}";
close $wVar;
$stuff = <$rVar>;
print "$childpipe{$child}, $stuff\n";
unless ($justDied == $child)
{
print "I wonder what died?\n";
}
}
close
}

sleep 10;
}

Agreed, the problem is probably somewhere in my code - although I'll
be damned if I can find it!

Although you may find it hard to believe ;o), all works (seemingly) as
I desire if you take the while (TRUE) loop out, but as originally
stated, breaks if you put it back on the 2nd iteration.

Agreed, using pre-defined file handles would be better, although given
that @LIST is of a non-pre-defined size, I couldn't think of a better
way to ensure that I read from the children in order...

On a more serious note.. thanks in advance for your help and taking
the time to ponder this..

Many thanks,

Mark
 
G

gnari

Mark Gowans said:
Crikey, quite the synic aren't you?! :eek:)

what Xho was referring to, is that he cannot run th pseudo code so see its
behaviour.
it is not worth it to spend much time on it because who knows how similar it
is to your real code.
one cannot even run perl -c on the pseudo code to see what warnings there
are.

[rearranging top-posting ...]
(e-mail address removed) wrote in message
#!/usr/local/bin/perl5 -w
while (TRUE) {
my $pid = $$;
my @children = ();
....

I see that he will have to wait some more ...

gnari
 
B

Brian McCauley

(e-mail address removed) (Mark Gowans) rudeness spits TOFU in our faces:

[ Rudeness corrected - but remember, Mark, if you think I seem at all
hostile towards you just spat in our faces. ]

[ pseudo-code ]
[...] here's a bit of perl to demonstrate exactly the below..

[ code that doesn't use strict or lexical handles ]

I'd recommend using strict and lexical file handles, and then if you still
need help, then try writing example code that works and demonstrates what
you says it does.
Agreed, using pre-defined file handles would be better, although given
that @LIST is of a non-pre-defined size, I couldn't think of a better
way to ensure that I read from the children in order...

Nobody said anything about pre-defined handles. I think you
miss-understand what we mean by "lexical file handles". Perl doesn't
really have lexically scoped file handles but you can pretend it does.
If something is lexically scoped within a loop then each time through
the loop the same name refers to a different thing.

Actually what happens with lexical filehandles is you get hard
references to semi-anonymous symbols. Don't worry you don't need to
understand that 99% of the time. Just act as if you have lexically
scoped file handles!

You said:
$rVar = "r$item";
$wVar = "w$item";
pipe ($rVar, $wVar);

This uses sybolic handle references.

To use lexically scoped file handled you could have said:

pipe(my($rVar),my($wVar));

Note: This autovivication of handles requires 5.6.1 or later. In
earlier versions you need to initialise the variables to handles in
some way such as:

my $rVar = IO::Handle->new;

Oh, and when we said "use strict" what we really mean is:

Always delare all variables as lexically scoped in the smallest
applicable scope unless there is a reason to do so[1].

Do not use symbolic references unless there is a reson to do so.

Don't use barewords[2].

Using strict is not an end in itself but it will help you to achive
these.

Footnotes to the above:

[1] Note this is not just good advice in Perl - it is good advice in
programming in general. If the language you are using supports the
concept of lexically scoped variables and you've not followed this
advice then any question you ask in comp.lang.* is likely to be seen
as rude. This is because often your problem will result from your
failure to use appropriate scoping - and even if it does not, anyone
who tries to help you would have to waste time looking to see if
problem resulted from your failure to use appropriate scoping.

[2] i.e. do not rely on the fact that if there's no function foo()
then Perl will treat foo as 'foo'.
#!/usr/local/bin/perl5 -w
while (TRUE) {

There is no function called TRUE() in Perl.

You got no error because you "forgot" to use strict.

while (1) {
my $pid = $$;
my @children = ();

There is no need to explicitly clear a varaible declared using my().
@children will be empty already.

my @children;
my $parent = 0;

Try to use the natural representation. The natural representation of
the concept of a scalar not being defined is the special scalar value
undef. As it hapens this is also whay my() will initialise scalars
to!

my $parent;

Actually for reasons I'll get to latter you don't need $parent at all.
@LIST = ("a", "b", "c", "d", "e",);

You forgot to delare @LIST.
You got no error because you "forgot" to use strict.

my @LIST = ("a", "b", "c", "d", "e",);

foreach $item(@LIST) {

You forgot to delare $item.
You got no error because you "forgot" to use strict.

foreach my $item(@LIST) {
$rVar = "r$item";
$wVar = "w$item";
pipe ($rVar, $wVar);


You forgot to delare $rVar and $wVar.
You got no error because you "forgot" to use strict.

Are you beginning to see a pattern yet?

Since you use these variables outside the loop you'd need to decalre
them outside the loop, however I think it's better to declare them
within the loop and re-arrange the code.

$rVar and $wVar are symbolic references. You should never use symbolic
references where real references will do. If you always "use strict"
then you can tell Perl that you've decided you really need to use
symbolic references by saying "no strict 'refs'".

However there's no reason to use sumbolic refs here so don't.

pipe (my($rVar), my($wVar));

my $newPID = fork();

if (not defined $newPID) {
die "Fork didn't work\n";

You should usually include the error in your error message.
You should usually not suppress the line number from your error
message.

die "Fork didn't work: $!";

} elsif ( $newPID == 0 ) {
$parent = $pid;
$pid = $$;
@children = ();
@LIST = ();

You are clearing @LIST on the assumption that it will terminate the
for() loop. Don't do that. The documentation of foreach() stresses
the fact that it is not defined what happens if you change the value
of an array while you are interating over it. To exit a loop use
last().

However rather than exiting the loop, then having another if($parent)
to find out if you are in the child or the parent it's better to just
call the child code here then exit().

And I wouldn't bother clearing @children in the child. Memory is not
that expensive!
} else {
push @children, $newPID;
$childpipe{$newPID} = $item;
}

Rather than saving $item, you can save the handle $rVar. You don't
need to save $wVar because this is where you should be closing it
anyhow! Not that you actually need to do so explicitly - using
lexical filehandles takes care of that for you.

$childpipe{$newPID} = $rVar;
if ($parent) {
#child process..
close $rVar;
print $wVar $$;
exit (0);

This should go inside the loop - or (if it's likely to grow long)
inside a subroutine that's called from within the loop.
} else {
#parent process..
while (my $child = shift @children) {
my $justDied = waitpid ($child,0);

You are still relying on the fact that the child will die before the
parent has read the output from it. This is only going to be true if
the child's total output fits in a single pipe buffer. Anyhow, you
are not cheching the exist status so there was no point waiting at all.
$rVar = "r$childpipe{$child}";
$wVar = "w$childpipe{$child}";
close $wVar;

This close() is no longer needed, it was closed already!
$stuff = <$rVar>;
print "$childpipe{$child}, $stuff\n";
unless ($justDied == $child)
{
print "I wonder what died?\n";
}
}
close

Why are you closing STDOUT inside the loop? Once you've closed it all
further prints to it will fail.

Could this be your real problem? If so then your problem had nothing
to do with all your bad habits - but your bad habits made your code
much more difficult for us to understand your code.

Indeed I've alreay spent >30m helping you with obvious problems in
your code before I stumbled upon it.

And of course that spurious close() wasn't in the pseudo-code at all
so anybody looking at the pseudo-code was completely wasting their
time. I think you own an appology to Xho for calling him a cynic.
}

sleep 10;
}

Agreed, the problem is probably somewhere in my code - although I'll
be damned if I can find it!

Perhaps if you made your code simpler it would be easier?

Of course, simplifying you code can often fix the problem without you
ever needing to find it. Was is also what the so-called cynic was
trying to tell you.

#!/usr/bin/perl
use strict;
use warnings;

while (1) {
my $pid = $$;
my (@children,%childpipe);
my @LIST = ("a", "b", "c", "d", "e",);
foreach my $item (@LIST) {
pipe (my($rVar), my($wVar));
my $newPID = fork();

if (not defined $newPID) {
die "Fork didn't work: $!";
} elsif ( $newPID == 0 ) {
#child process..
undef $rVar; # close()
print $wVar $$;
exit (0);
} else {
push @children, $newPID;
$childpipe{$newPID} = $rVar;
}
}

#parent process..
while (my $child = shift @children) {
my $rVar = $childpipe{$child};
my $stuff = <$rVar>;
print "$child, $stuff\n";
}

sleep 10;
}
__END__

Note that for convenience you can also combine the pipe() and fork()
(and optionally exec() too) into open() thus:

my $newPID = open my $rVar, '-|';

Here you don't need the $rVar variable because STDOUT in the child
becomes connected to the pipe.

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
 
M

Matthew Braid

Mark said:
Crikey, quite the synic aren't you?! :eek:)

I recon its pretty similar to the pseudocode below, but here's a bit
of perl to demonstrate exactly the below..

#!/usr/local/bin/perl5 -w
while (TRUE) {
my $pid = $$;
my @children = ();
my $parent = 0;
@LIST = ("a", "b", "c", "d", "e",);
foreach $item(@LIST) {
$rVar = "r$item";
$wVar = "w$item";

Does it work if you change the last two lines to:

my $rVar = "r$item";
my $wVar = "w$item";

?

Come to think of it, do you even need to set these to a string anyway?
I'm not sure if it works with pipe, but don't newer versions of perl
auto-vivify lexical filehandles? Just as a check, try replacing those
same two lines with:

my ($rVar, $wVar);

If neither of those work I can't help sorry - I don't use pipe all that
often (read: at all :) )

MB
 
M

Mark Gowans

Brian McCauley said:
(e-mail address removed) (Mark Gowans) rudeness spits TOFU in our faces:

[ Rudeness corrected - but remember, Mark, if you think I seem at all
hostile towards you just spat in our faces. ]

[..large snip..]

Magic.. That works a real treat. You were, of course, quite correct in
that just removing the 'close' from the end of the code made it work.

Didn't, but now understand the concept of lexical handles, and will be
using 'use strict' in future.. You've got to forgive a newbie a few
foibles? :eek:)

Couple of quick questions for future reference if I may...

I originally had:

#parent process..
while (my $child = shift @children) {
my $justDied = waitpid($child,0);
<do stuff>
}

You commented that with the above, I'm relying on the fact that the
child will die before the parent has read the output from it. That its
only going to be true if the child's total output fits in a single
pipe buffer and that I'm not checking the exit status so there was no
point in waiting at all...

and so you suggested:

#parent process
while (my $child = shift @children) {
<do stuff>
}

Most likely my understanding of waitpid is wrong.. but I wanted to
ensure that the child had exited before I attempted to read from it..
The trouble is, my child processes take some time to complete, and I
didn't want the parent to attempt to read from them before they've
finished doing the do. Is waitpid, and then checking $? the correct
thing to do? I guess I want a blocking wait.. or is this dangerous?
<worries about zombified children>

2 Final quick questions...

Having-rearranged the code, I've now a whole load of 'my <variable>'
definitions within the while loop. Is this good practice? Should I be
undef'ing them at the end of the loop to keep tabs on memory?

Finally, I note you suggested using 'undef $variable' instead of my
'close $variable'. Whats the difference?

Many thanks for your help - much appreciated

Mark.
 
B

Ben Morrow

Couple of quick questions for future reference if I may...

I originally had:

#parent process..
while (my $child = shift @children) {
my $justDied = waitpid($child,0);
<do stuff>
}

You commented that with the above, I'm relying on the fact that the
child will die before the parent has read the output from it. That its
only going to be true if the child's total output fits in a single
pipe buffer and that I'm not checking the exit status so there was no
point in waiting at all...

and so you suggested:

#parent process
while (my $child = shift @children) {
<do stuff>
}


Most likely my understanding of waitpid is wrong.. but I wanted to
ensure that the child had exited before I attempted to read from
it.. The trouble is, my child processes take some time to complete,
and I didn't want the parent to attempt to read from them before
they've finished doing the do. Is waitpid, and then checking $? the
correct thing to do? I guess I want a blocking wait.. or is this
dangerous? <worries about zombified children>

No no no, what you want is a blocking *read*. This means that each
time you try and read something from the child, the parent will wait
until there's something there to read before returning it. As reads in
Perl are blocking by default, you simply don't need to worry: just
read assuming the data is there, and if it isn't the parent will wait
until it is.

If you try and wait till the child has exited before reading anything,
where will all the child's output go in the meanwhile? Some of it will
go in the pipe buffer, but they aren't very big...

WRT zombies, read perldoc perlipc.
2 Final quick questions...

Having-rearranged the code, I've now a whole load of 'my <variable>'
definitions within the while loop. Is this good practice? Should I be
undef'ing them at the end of the loop to keep tabs on memory?

The whole point of 'my' variables is that when they go out of scope
they are destroyed automatically. For a variable declared within the
body of a loop, each time through the loop is a separate scope, so all
the variables will be destroyed at the end of each iteration (even if
you exit the iteration with something non-local like 'next' or 'last'
or 'goto' or 'die').
Finally, I note you suggested using 'undef $variable' instead of my
'close $variable'. Whats the difference?

Err... I didn't think Brian did suggest undefing anything. I think
what he said is to use lexical FHs, and then not to ever explicitly
close them: again, they will automatically be closed when they go out
of scope.

Ben
 
B

Brian McCauley

In typically careless fashion Brian McCauley said:
pipe(my($rVar),my($wVar));

Opps, that should be:

pipe(my($rVar),my($wVar)) or die "pipe(): $!";
my $newPID = open my $rVar, '-|';

I forgot to mention that using this mechanism you should not use
waitpid() explicitly with $newPID. To get the exit status from the
child, the parent should close($rVar) then examine the value in $?.

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
 
S

Sam Holden

Most likely my understanding of waitpid is wrong.. but I wanted to
ensure that the child had exited before I attempted to read from it..
The trouble is, my child processes take some time to complete, and I
didn't want the parent to attempt to read from them before they've
finished doing the do. Is waitpid, and then checking $? the correct
thing to do? I guess I want a blocking wait.. or is this dangerous?
<worries about zombified children>

Don't do that. Just read, the system will handle all the hard stuff and
your read won't return until the child has actually written some data
(unless you specifically arrange for it not to which obviously in this
instance you won't). Just keep reading until EOF which the system will
again arrange to happen when the child finishes.

Otherwise what will happen (not always but at some point) is that the
child will be waiting for the parent to read some data (so that it can
write some more - after filling up the buffer space) and the parent will
be waiting for the child to exit before reading the data. Obviously
that's bad. In fact it is classic deadlock, and something to be avoided.
 
B

Brian McCauley

Ben Morrow said:
Err... I didn't think Brian did suggest undefing anything.

Err... actually I did. :)
I think what he said is to use lexical FHs, and then not to ever
explicitly close them: again, they will automatically be closed when
they go out of scope.

Yes most of the time that is true. But sometimes you want to close a
file handle before the variable goes out of scope. This is most often
encountered in the case (as in the OP's code) where children should
close their copy of the reading end of the FIFO to which they are
writing. If they fail to do so they would simply block without
getting a SIGPIPE in the event that the parent dies prematurely.

To explicitly close a lexical filehandle you can use close() in which
case the Perl filehandle object will continue to exist but will be in
a closed state until the variable goes out of scope. Or you can
undef() the variable. The latter will destroy the filehandle object
(assuming there are no other references left). Destroying the
filehandle object will implicitly close it.

To my mind having a closed filehandle object floating about seems
untidy so I prefer to use undef(). There are other people to whom the
idea of using undef() to mean close() is untidy. TIMTOWTDI!

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
 
C

ctcgag

Brian McCauley said:
To my mind having a closed filehandle object floating about seems
untidy so I prefer to use undef(). There are other people to whom the
idea of using undef() to mean close() is untidy. TIMTOWTDI!

I like to catch errors that result from closing a file handle with:
close $whatever or die $! # or some variant on that.

If you just let it go out of scope (or undef) will you get some
kind of warning if the close fails?

(I know in this case it wouldn't be meaningful for the close to fail,
but I like to cultivate good habits anyway)

Xho
 
B

Brian McCauley

I like to catch errors that result from closing a file handle with:
close $whatever or die $! # or some variant on that.

If you just let it go out of scope (or undef) will you get some
kind of warning if the close fails?

No, if you want to get a the error return from close() then you need
to close() explicitly.

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top