Final "Flocking" Script

[\Dandy\ Randy]

Thanx you to everyone who has been helping with my flocking issue. After
reading several perdocs from www.perdoc.com I have come up with the
following script. This script simply advances the second value of a text
file number by 1 each time it is run. The only quirk I have found is with
"use strict" and "use warnings" In my script these lines are blocked out ...
it is the only way the sript runs correctly.

#!/usr/bin/perl

#use strict;
#use warnings;
use 5.004;
use Fcntl qwDEFAULT :flock);

open (FH, "<data.txt") or die "Can't open file: $!";
flock (FH, LOCK_EX) or die "Can't lock file: $!";
$data=<FH>;
chomp ($data);
($total,$opened,$followed)=split(/\|/,$data);
close(FH);

$opened = $opened + 1;

sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
$!";
flock (FH, LOCK_EX) or die "can't lock filename: $!";
truncate (FH, 0) or die "can't truncate filename: $!";
print FH "$total|$opened|$followed\n";
close FH;

print "Content-type: text/html \n\n";
print "Done\n";
exit;

The end result, the number advances correctly. This script is basically a
counter for email ... one of my other programs sends out several emails, and
in the email code, there are instructions to run this script ... it simply
tells me how many emails were actually opened compared to how many sent. I
am hoping the flocking script i'm using above is the correct one for my
"cause". If it's not, please advise. thanx to everyone for the help;

Randy

P.S.

In case any of you are wondering about the strict or warning issues, when I
unblock these commands, the server returns the following text:

--------------------------------------
Internal Server Error
The server encountered an internal error or misconfiguration and was unable
to complete your request.
Please contact the server administrator and inform them of the time the
error occurred, and anything you might have done that may have caused the
error.

More information about this error may be available in the server error log.

 

[Tad McClellan]

sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
$!";
flock (FH, LOCK_EX) or die "can't lock filename: $!";
truncate (FH, 0) or die "can't truncate filename: $!";

There should be a seek(FH, 0, 0) here in case some other process
advanced the file pointer between your sysopen() and your flock().

In case any of you are wondering about the strict or warning issues, when I
unblock these commands, the server returns the following text:

--------------------------------------
[snip]

More information about this error may be available in the server error log.
--------------------------------------

So then, what did it say in the server error log?

 

[Tad McClellan]

Put the following at the beginning of the script:

use CGI::Carp 'fatalsToBrowser';

And be sure to *remove it* for production!

 

[Gunnar Hjalmarsson]

And be sure to *remove it* for production!

What makes you say that, Tad? The CGI::Carp docs suggests that the
carpout() routine shouldn't be used for production for *performance*
reasons, but I don't think there is such a comment as regards
fatalsToBrowser(). Or is there any other reason for your advise?

 

[Alan J. Flavell]

Tad McClellan wrote:

What makes you say that, Tad? The CGI::Carp docs suggests that the
carpout() routine shouldn't be used for production for *performance*
reasons, but I don't think there is such a comment as regards
fatalsToBrowser(). Or is there any other reason for your advise?

I don't know Tad's motivation, but: it can expose details of the
internals of your scripts to an intruder. The same goes for exposing
warnings within the browser window.

Of course, it also gives an unprofessional impression if it happens to
a bona fide user. Ideally you should cover all known failure modes at
development time, and supply user-friendly responses for them, rather
than exposing the end-users to what are _supposed_ to be your internal
diagnostics.

But indeed, during development these features can be _very_ useful.

 

[Matija Papec]

X-Ftn-To: Tad McClellan

There should be a seek(FH, 0, 0) here in case some other process
advanced the file pointer between your sysopen() and your flock().

Hi,
isn't a file pointer something that belongs to(and only to) process itself?

 

[Tad McClellan]

You wrote:

Who is "you"?

Please provide a proper attribution when you quote someone.

I always worried about having the open and lock statements separate. I had
the idea (possibly wrong) that there could be some time lag between
execution of the 2 statements, and the file could change during that time.

Eh?

The very purspose of doing file locking in the first place is
because of such a "time lag", most commonly called a
"race condition".

So your idea is perfectly right. (so far)

I usually do something like:

.............
use Fcntl ':flock'; # import LOCK_* constants
use Fcntl 'EFAULT';
use POSIX;
sub lock {
flock F, LOCK_EX;
# and, in case someone appended <-----
# while we were waiting... <------ (This is from perl docs.)
seek F, 0, 2; #<-----------
}

sub unlock {
flock F, LOCK_UN;
}

Uh oh, if this gets called, then you have _introduced_ a race,
exactly the opposite of what you're trying to do.

If you're worried about the open/flock timing why not worry
about the unlock/close timing as well?

seek()ing can overcome the former, but the later is a problem.

Do not unlock the file, just close() it. close() will unlock
it in a way that _is_ atomic.

if (sysopen(F,$file, O_WRONLY)){
lock();}
.............

That way, the open and lock is "atomic".

No they're not.

Why do you think that they are atomic in your code?

Your process could _still_ do the open() and lose its time slice
before doing the flock().

In fact, it makes the time for the race to bite you _longer_ due
to the overhead of calling a subroutine.

You made it a little worse, but that's OK since you're seek()ing anyway.

I wonder if this is actually
necessary.

If "this" means your code above, then no, it is not necessary.

The perl docs show some examples where the open and lock are
in separate statements, so I thought that should be OK.

Putting them in the same statement will not affect the possibility
of a race either.

But I once did
some tests of locking in which I appended, say, 100 times to a file with
a script using open and lock separately and then in a single statement, and
it seemed to make some difference

Must have been a statistical anomaly.

- there were occasionally some errors
when the open and lock were in separate statements. However, this could have
been for some other reason...

Like the race between the unlock and the close.

 

[Matija Papec]

X-Ftn-To: Steve Grazzini

The problem would occur when another process writes to the
file between the append-open() and the flock(). This could
invalidate your seek pointer -- i.e. you're prepared to write
at the end of the file, and the end of the file *moves* so
you end up writing in the wrong place.

This only applies to appending, though, and the FAQ suggests
that it only applies to Windows. Maybe somebody can list some

You mean faq from perldoc or some other faq?
It seems that under Linux you can freely change $file while script sleeps,


open FH, "+>>$file" or die $!;
sleep 20;
flock(FH, 2); #exclusive_lock
print FH "writing to EOF\n";

 

[Brian McCauley]

X-Ftn-To: Steve Grazzini

Unless, of course, you are not using an OS that doesn't correctly
implement append-open().

Or, to put it another way (wihtout the tripple negative)...

Only if your OS has broken append-open.

I don't know if it _only_ applies to Windows but it doesn't apply on
OSs with working append-open.

I must admit I'd thought 5.8 was going to have a work-round for this bug
by putting an a seek-to-end before each write to a filehandle opened
for append on Windows. Guess it never made it.

It seems that under Linux you can freely change $file while script sleeps,

You can but it doesn't matter, because append-open isn't broken on Linux.

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\

 

[Steve Grazzini]

You mean faq from perldoc or some other faq?

The Perl FAQ.

% perldoc -q "do i still have to use locking"

I was referring to this:

If you know you are only going to use a system that does
correctly implement appending (i.e. not Win32) then you
can omit the seek() from the above code.

 

[ctcgag]

I don't know Tad's motivation, but: it can expose details of the
internals of your scripts to an intruder. The same goes for exposing
warnings within the browser window.

In my particular case, all the users of the scripts are inside our
firewall. Any of those users could just walk into my office and shove
an icepick in my ear. So if they've gone bad and are on the attack, CGI
scripts are the least of my worries.

Of course, I'm usually one of the major users of my own scripts, so
at least part of the time that the user gets a ugly error message, the
user is someone in a position to do something about it.

Of course, it also gives an unprofessional impression if it happens to
a bona fide user. Ideally you should cover all known failure modes at
development time, and supply user-friendly responses for them, rather
than exposing the end-users to what are _supposed_ to be your internal
diagnostics.

I've had a group decide that the error messages looked unprofessional.
But they weren't willing to spend the time making user-friendly responses,
so instead they just yanked the fatalsToBrowser, and let the scripts
die silently. No ugly error messages anymore, just pretty half-rendered
web pages with no clue as to what is wrong. The veneer of professionalism,
without the substance. It just boggled my mind.

But indeed, during development these features can be _very_ useful.

Yes, and everything I do is always in development.

Xho

 

[Steve Grazzini]

[ error-checking omitted ]

Which kind of screwed my up, because you don't
use Fcntl ':flock';

Sigh.

I actually ran it that way and didn't notice -- demonstrating
either that the OP's flock() didn't do anything useful, or that
I'm a careless fool, or very possibly: both.

 

[Gunnar Hjalmarsson]

I'm afraid you misinterpreted my meaning of the term 'user' here.

Yes, so it seems.

Naturally, those who are installing the script should get these
kind of diagnostics until the scripts are working.

Okay, then we are talking about adding the recommendation to the
installation instructions that the fatalsToBrowser() routine is
commented out when the script is up and running. I take it as a
_warning_ to be _considered_.

You mentioned the risk for helping possible intruders as a reason to
do so.

I can see two reasons against it:

- Web hosts offering shared accounts often have the bad habit of
changing things without informing their customers. When that happens,
it's useful to see the resulting error messages.

- It would make the installation a little more difficult. (In my case,
fatalsToBrowser() is called from each one of 15+ *.pl scripts.)

So, basically we are talking about security vs. convenience. So far, I
have concluded that - _in my particular case_ - the convenience aspect
carries greater weight.

 

[Tad McClellan]

we had a hacker into our system for a couple of months
before we became aware of him (and, as far as we could diagnose
afterwards, we only became aware of him as an accident

The "Star Wars secrets for cocaine" hackers of the 1980s were
tripped up due to a $0.75 discrepency in billing for computer time.

The book about that incident, "The Cuckoo's Egg" (Clifford Stoll),
is a very good read. Kept me up until 3am two nights...