Final "Flocking" Script

Discussion in 'Perl Misc' started by \Dandy\ Randy, Aug 5, 2003.

  1. Thanx you to everyone who has been helping with my flocking issue. After
    reading several perdocs from www.perdoc.com I have come up with the
    following script. This script simply advances the second value of a text
    file number by 1 each time it is run. The only quirk I have found is with
    "use strict" and "use warnings" In my script these lines are blocked out ...
    it is the only way the sript runs correctly.

    #!/usr/bin/perl

    #use strict;
    #use warnings;
    use 5.004;
    use Fcntl qw:)DEFAULT :flock);

    open (FH, "<data.txt") or die "Can't open file: $!";
    flock (FH, LOCK_EX) or die "Can't lock file: $!";
    $data=<FH>;
    chomp ($data);
    ($total,$opened,$followed)=split(/\|/,$data);
    close(FH);

    $opened = $opened + 1;

    sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
    $!";
    flock (FH, LOCK_EX) or die "can't lock filename: $!";
    truncate (FH, 0) or die "can't truncate filename: $!";
    print FH "$total|$opened|$followed\n";
    close FH;

    print "Content-type: text/html \n\n";
    print "Done\n";
    exit;

    The end result, the number advances correctly. This script is basically a
    counter for email ... one of my other programs sends out several emails, and
    in the email code, there are instructions to run this script ... it simply
    tells me how many emails were actually opened compared to how many sent. I
    am hoping the flocking script i'm using above is the correct one for my
    "cause". If it's not, please advise. thanx to everyone for the help;

    Randy

    P.S.

    In case any of you are wondering about the strict or warning issues, when I
    unblock these commands, the server returns the following text:

    --------------------------------------
    Internal Server Error
    The server encountered an internal error or misconfiguration and was unable
    to complete your request.
    Please contact the server administrator and inform them of the time the
    error occurred, and anything you might have done that may have caused the
    error.

    More information about this error may be available in the server error log.
    --------------------------------------
    Apache/1.3.27 Server at www.awebsite.com Port 80
     
    \Dandy\ Randy, Aug 5, 2003
    #1
    1. Advertising

  2. \"Dandy\" Randy <> wrote:

    > sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
    > $!";
    > flock (FH, LOCK_EX) or die "can't lock filename: $!";
    > truncate (FH, 0) or die "can't truncate filename: $!";



    There should be a seek(FH, 0, 0) here in case some other process
    advanced the file pointer between your sysopen() and your flock().


    > In case any of you are wondering about the strict or warning issues, when I
    > unblock these commands, the server returns the following text:
    >
    > --------------------------------------


    [snip]

    > More information about this error may be available in the server error log.
    > --------------------------------------



    So then, what did it say in the server error log?


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Aug 5, 2003
    #2
    1. Advertising

  3. Gunnar Hjalmarsson <> wrote:

    > Put the following at the beginning of the script:
    >
    > use CGI::Carp 'fatalsToBrowser';



    And be sure to *remove it* for production!


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Aug 5, 2003
    #3
  4. Tad McClellan wrote:
    > Gunnar Hjalmarsson <> wrote:
    >>Put the following at the beginning of the script:
    >>
    >> use CGI::Carp 'fatalsToBrowser';

    >
    > And be sure to *remove it* for production!


    What makes you say that, Tad? The CGI::Carp docs suggests that the
    carpout() routine shouldn't be used for production for *performance*
    reasons, but I don't think there is such a comment as regards
    fatalsToBrowser(). Or is there any other reason for your advise?

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
     
    Gunnar Hjalmarsson, Aug 5, 2003
    #4
  5. On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:

    > Tad McClellan wrote:


    > > And be sure to *remove it* for production!

    >
    > What makes you say that, Tad? The CGI::Carp docs suggests that the
    > carpout() routine shouldn't be used for production for *performance*
    > reasons, but I don't think there is such a comment as regards
    > fatalsToBrowser(). Or is there any other reason for your advise?


    I don't know Tad's motivation, but: it can expose details of the
    internals of your scripts to an intruder. The same goes for exposing
    warnings within the browser window.

    Of course, it also gives an unprofessional impression if it happens to
    a bona fide user. Ideally you should cover all known failure modes at
    development time, and supply user-friendly responses for them, rather
    than exposing the end-users to what are _supposed_ to be your internal
    diagnostics.

    But indeed, during development these features can be _very_ useful.

    --
    "In the hardest of cases, I show them the W3C membership list: as
    soon as the customer sees that MS is a member, then everything the
    W3C says is good, and typically they even demand conformance with
    W3C written into the contract" - (Matthias Esken, freely translated)
     
    Alan J. Flavell, Aug 5, 2003
    #5
  6. \Dandy\ Randy

    Matija Papec Guest

    X-Ftn-To: Tad McClellan

    (Tad McClellan) wrote:
    >> sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
    >> $!";
    >> flock (FH, LOCK_EX) or die "can't lock filename: $!";
    >> truncate (FH, 0) or die "can't truncate filename: $!";

    >
    >There should be a seek(FH, 0, 0) here in case some other process
    >advanced the file pointer between your sysopen() and your flock().


    Hi,
    isn't a file pointer something that belongs to(and only to) process itself?


    --
    Matija
     
    Matija Papec, Aug 5, 2003
    #6
  7. Chas Friedman <> wrote:
    > You wrote:



    Who is "you"?

    Please provide a proper attribution when you quote someone.


    >>Thanx you to everyone who has been helping with my flocking issue.


    >>open (FH, "<data.txt") or die "Can't open file: $!";
    >>flock (FH, LOCK_EX) or die "Can't lock file: $!";


    > I always worried about having the open and lock statements separate. I had
    > the idea (possibly wrong) that there could be some time lag between
    > execution of the 2 statements, and the file could change during that time.



    Eh?

    The very purspose of doing file locking in the first place is
    because of such a "time lag", most commonly called a
    "race condition".

    So your idea is perfectly right. (so far)


    > I usually do something like:
    >
    > .............
    > use Fcntl ':flock'; # import LOCK_* constants
    > use Fcntl ':DEFAULT';
    > use POSIX;
    > sub lock {
    > flock F, LOCK_EX;
    > # and, in case someone appended <-----
    > # while we were waiting... <------ (This is from perl docs.)
    > seek F, 0, 2; #<-----------
    > }
    >
    > sub unlock {
    > flock F, LOCK_UN;
    > }



    Uh oh, if this gets called, then you have _introduced_ a race,
    exactly the opposite of what you're trying to do.

    If you're worried about the open/flock timing why not worry
    about the unlock/close timing as well?

    seek()ing can overcome the former, but the later is a problem.

    Do not unlock the file, just close() it. close() will unlock
    it in a way that _is_ atomic.


    > if (sysopen(F,$file, O_WRONLY)){
    > lock();}
    > .............
    >
    > That way, the open and lock is "atomic".



    No they're not.

    Why do you think that they are atomic in your code?

    Your process could _still_ do the open() and lose its time slice
    before doing the flock().

    In fact, it makes the time for the race to bite you _longer_ due
    to the overhead of calling a subroutine.

    You made it a little worse, but that's OK since you're seek()ing anyway.


    > I wonder if this is actually
    > necessary.



    If "this" means your code above, then no, it is not necessary.


    > The perl docs show some examples where the open and lock are
    > in separate statements, so I thought that should be OK.



    Putting them in the same statement will not affect the possibility
    of a race either.


    > But I once did
    > some tests of locking in which I appended, say, 100 times to a file with
    > a script using open and lock separately and then in a single statement, and
    > it seemed to make some difference



    Must have been a statistical anomaly.


    > - there were occasionally some errors
    > when the open and lock were in separate statements. However, this could have
    > been for some other reason...



    Like the race between the unlock and the close. :)


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Aug 6, 2003
    #7
  8. \Dandy\ Randy

    Matija Papec Guest

    X-Ftn-To: Steve Grazzini

    Steve Grazzini <> wrote:
    >The problem would occur when another process writes to the
    >file between the append-open() and the flock(). This could
    >invalidate your seek pointer -- i.e. you're prepared to write
    >at the end of the file, and the end of the file *moves* so
    >you end up writing in the wrong place.
    >
    >This only applies to appending, though, and the FAQ suggests
    >that it only applies to Windows. Maybe somebody can list some


    You mean faq from perldoc or some other faq?
    It seems that under Linux you can freely change $file while script sleeps,


    open FH, "+>>$file" or die $!;
    sleep 20;
    flock(FH, 2); #exclusive_lock
    print FH "writing to EOF\n";


    --
    Matija
     
    Matija Papec, Aug 6, 2003
    #8
  9. Matija Papec <> writes:

    > X-Ftn-To: Steve Grazzini
    >
    > Steve Grazzini <> wrote:
    > >The problem would occur when another process writes to the
    > >file between the append-open() and the flock(). This could
    > >invalidate your seek pointer -- i.e. you're prepared to write
    > >at the end of the file, and the end of the file *moves* so
    > >you end up writing in the wrong place.


    Unless, of course, you are not using an OS that doesn't correctly
    implement append-open().

    Or, to put it another way (wihtout the tripple negative)...

    Only if your OS has broken append-open.

    > >
    > >This only applies to appending, though, and the FAQ suggests
    > >that it only applies to Windows.


    I don't know if it _only_ applies to Windows but it doesn't apply on
    OSs with working append-open.

    I must admit I'd thought 5.8 was going to have a work-round for this bug
    by putting an a seek-to-end before each write to a filehandle opened
    for append on Windows. Guess it never made it.

    > It seems that under Linux you can freely change $file while script sleeps,


    You can but it doesn't matter, because append-open isn't broken on Linux.

    --
    \\ ( )
    . _\\__[oo
    .__/ \\ /\@
    . l___\\
    # ll l\\
    ###LL LL\\
     
    Brian McCauley, Aug 6, 2003
    #9
  10. Matija Papec <> wrote:
    > Steve Grazzini <> wrote:
    > >The problem would occur when another process writes to the
    > >file between the append-open() and the flock().
    > >
    > >This only applies to appending, though, and the FAQ suggests
    > >that it only applies to Windows.

    >
    > You mean faq from perldoc or some other faq?


    The Perl FAQ.

    % perldoc -q "do i still have to use locking"

    I was referring to this:

    If you know you are only going to use a system that does
    correctly implement appending (i.e. not Win32) then you
    can omit the seek() from the above code.

    --
    Steve
     
    Steve Grazzini, Aug 6, 2003
    #10
  11. \Dandy\ Randy

    Guest

    "Alan J. Flavell" <> wrote:
    > On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:
    >
    > > Tad McClellan wrote:

    >
    > > > And be sure to *remove it* for production!

    > >
    > > What makes you say that, Tad? The CGI::Carp docs suggests that the
    > > carpout() routine shouldn't be used for production for *performance*
    > > reasons, but I don't think there is such a comment as regards
    > > fatalsToBrowser(). Or is there any other reason for your advise?

    >
    > I don't know Tad's motivation, but: it can expose details of the
    > internals of your scripts to an intruder. The same goes for exposing
    > warnings within the browser window.


    In my particular case, all the users of the scripts are inside our
    firewall. Any of those users could just walk into my office and shove
    an icepick in my ear. So if they've gone bad and are on the attack, CGI
    scripts are the least of my worries. :)

    Of course, I'm usually one of the major users of my own scripts, so
    at least part of the time that the user gets a ugly error message, the
    user is someone in a position to do something about it.


    > Of course, it also gives an unprofessional impression if it happens to
    > a bona fide user. Ideally you should cover all known failure modes at
    > development time, and supply user-friendly responses for them, rather
    > than exposing the end-users to what are _supposed_ to be your internal
    > diagnostics.


    I've had a group decide that the error messages looked unprofessional.
    But they weren't willing to spend the time making user-friendly responses,
    so instead they just yanked the fatalsToBrowser, and let the scripts
    die silently. No ugly error messages anymore, just pretty half-rendered
    web pages with no clue as to what is wrong. The veneer of professionalism,
    without the substance. It just boggled my mind.

    >
    > But indeed, during development these features can be _very_ useful.


    Yes, and everything I do is always in development. :)

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service New Rate! $9.95/Month 50GB
     
    , Aug 6, 2003
    #11
  12. wrote:
    > Steve Grazzini <> wrote:
    >>
    >> [ error-checking omitted ]

    >
    > Which kind of screwed my up, because you don't
    > use Fcntl ':flock';


    Sigh.

    I actually ran it that way and didn't notice -- demonstrating
    either that the OP's flock() didn't do anything useful, or that
    I'm a careless fool, or very possibly: both.

    --
    Steve
     
    Steve Grazzini, Aug 6, 2003
    #12
  13. Alan J. Flavell wrote:
    > On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:
    >> Alan J. Flavell wrote:
    >>> Of course, it also gives an unprofessional impression if it
    >>> happens to a bona fide user. Ideally you should cover all known
    >>> failure modes at development time, and supply user-friendly
    >>> responses for them, rather than exposing the end-users to what
    >>> are _supposed_ to be your internal diagnostics.

    >>
    >> I'm distributing a CGI script to largely novise users (mostly on
    >> shared hosting accounts without access to the error logs...),

    >
    > I'm afraid you misinterpreted my meaning of the term 'user' here.


    Yes, so it seems.

    > Naturally, those who are installing the script should get these
    > kind of diagnostics until the scripts are working.


    Okay, then we are talking about adding the recommendation to the
    installation instructions that the fatalsToBrowser() routine is
    commented out when the script is up and running. I take it as a
    _warning_ to be _considered_. :)

    You mentioned the risk for helping possible intruders as a reason to
    do so.

    I can see two reasons against it:

    - Web hosts offering shared accounts often have the bad habit of
    changing things without informing their customers. When that happens,
    it's useful to see the resulting error messages.

    - It would make the installation a little more difficult. (In my case,
    fatalsToBrowser() is called from each one of 15+ *.pl scripts.)

    So, basically we are talking about security vs. convenience. So far, I
    have concluded that - _in my particular case_ - the convenience aspect
    carries greater weight.

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
     
    Gunnar Hjalmarsson, Aug 7, 2003
    #13
  14. Alan J. Flavell <> wrote:

    > we had a hacker into our system for a couple of months
    > before we became aware of him (and, as far as we could diagnose
    > afterwards, we only became aware of him as an accident



    The "Star Wars secrets for cocaine" hackers of the 1980s were
    tripped up due to a $0.75 discrepency in billing for computer time.

    The book about that incident, "The Cuckoo's Egg" (Clifford Stoll),
    is a very good read. Kept me up until 3am two nights...


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Aug 8, 2003
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. JFCM
    Replies:
    4
    Views:
    5,778
  2. Replies:
    5
    Views:
    543
    Chris Uppal
    Nov 17, 2006
  3. Rajat
    Replies:
    3
    Views:
    744
    Jorgen Grahn
    Jan 8, 2010
  4. \Dandy\ Randy

    Flocking Advise

    \Dandy\ Randy, Sep 9, 2003, in forum: Perl Misc
    Replies:
    8
    Views:
    94
  5. Dick Rosser

    a Post-script to flocking question

    Dick Rosser, Nov 13, 2003, in forum: Perl Misc
    Replies:
    1
    Views:
    115
    Tassilo v. Parseval
    Nov 13, 2003
Loading...

Share This Page