sysopen failures

Discussion in 'Perl Misc' started by Marc Girod, Aug 6, 2010.

  1. Marc Girod

    Marc Girod Guest

    Hello,

    A script saves mails sent by a crontab--so, there may be bursts...
    It uses sysopen, I assume to make sure it doesn't overwrite existing
    files.
    At times, we get bursts of errors (File exists), which I trace to the
    sysopen call.
    However, I cannot find that all the corresponding files would have
    existed.
    I read the doc and get to:

    In many systems the "O_EXCL" flag is available for opening
    files
    in exclusive mode. This is not locking: exclusiveness
    means here
    that if the file already exists, sysopen() fails. "O_EXCL"
    may
    not work on network filesystems, and has no effect unless
    the
    "O_CREAT" flag is set as well.

    The script does write to a network filesystems (home directory on a
    remote filer, 4 ms round-trip).

    Shoud I look for a replacement for sysopen?
    Or for an other theory to explain the problem?

    The bit of code doing the open:

    if(defined($mode)? sysopen(FILE, $file, O_EXCL | O_CREAT | O_WRONLY,
    $mode):
    sysopen(FILE, $file, O_EXCL | O_CREAT | O_WRONLY))
    {
    _dump(*FILE, @$r_lines);
    return close(FILE);
    }
    return(0);

    Thanks,
    Marc
     
    Marc Girod, Aug 6, 2010
    #1
    1. Advertising

  2. Marc Girod wrote:
    >
    > A script saves mails sent by a crontab--so, there may be bursts...
    > It uses sysopen, I assume to make sure it doesn't overwrite existing
    > files.
    > At times, we get bursts of errors (File exists), which I trace to the
    > sysopen call.
    > However, I cannot find that all the corresponding files would have
    > existed.
    > I read the doc and get to:
    >
    > In many systems the "O_EXCL" flag is available for opening
    > files
    > in exclusive mode. This is not locking: exclusiveness
    > means here
    > that if the file already exists, sysopen() fails. "O_EXCL"
    > may
    > not work on network filesystems, and has no effect unless
    > the
    > "O_CREAT" flag is set as well.
    >
    > The script does write to a network filesystems (home directory on a
    > remote filer, 4 ms round-trip).
    >
    > Shoud I look for a replacement for sysopen?
    > Or for an other theory to explain the problem?
    >
    > The bit of code doing the open:
    >
    > if(defined($mode)? sysopen(FILE, $file, O_EXCL | O_CREAT | O_WRONLY,
    > $mode):
    > sysopen(FILE, $file, O_EXCL | O_CREAT | O_WRONLY))
    > {


    Your code does not check the return value from sysopen() so the error
    message you are receiving is not related to sysopen().


    > _dump(*FILE, @$r_lines);
    > return close(FILE);
    > }
    > return(0);



    John
    --
    Any intelligent fool can make things bigger and
    more complex... It takes a touch of genius -
    and a lot of courage to move in the opposite
    direction. -- Albert Einstein
     
    John W. Krahn, Aug 7, 2010
    #2
    1. Advertising

  3. Marc Girod

    Marc Girod Guest

    On Aug 7, 7:40 am, "John W. Krahn" <> wrote:

    > Your code does not check the return value from sysopen() so the error
    > message you are receiving is not related to sysopen().


    Doesn't it?
    The block is an if block.
    If the condition (the return from sysopen) is false, the function
    returns 0 unconditionally.

    Besides, the error I get from $!, at the time of reporting in the
    calling function is, as I wrote it, 'File exists'.

    And last: it is not *my* code.
    Marc
     
    Marc Girod, Aug 7, 2010
    #3
  4. On 2010-08-07 08:55, Marc Girod <> wrote:
    > On Aug 7, 7:40 am, "John W. Krahn" <> wrote:
    >> Your code does not check the return value from sysopen() so the error
    >> message you are receiving is not related to sysopen().

    >
    > Doesn't it?
    > The block is an if block.
    > If the condition (the return from sysopen) is false, the function
    > returns 0 unconditionally.


    But you haven't shown where the error is reported. We could only guess
    that it's the next thing after “return 0â€.

    > Besides, the error I get from $!, at the time of reporting in the
    > calling function is, as I wrote it, 'File exists'.


    There may be something between the sysopen and the printing of $! which
    causes that error. Since we haven't seen that code we can't tell.
    I do agree that it's very likely that “File exists†is set by sysopen in
    this case.

    > And last: it is not *my* code.


    You posted it, so in the context of this thread it's your code. :p

    hp
     
    Peter J. Holzer, Aug 7, 2010
    #4
  5. On 2010-08-06 17:06, Marc Girod <> wrote:
    > A script saves mails sent by a crontab--so, there may be bursts...
    > It uses sysopen, I assume to make sure it doesn't overwrite existing
    > files.
    > At times, we get bursts of errors (File exists), which I trace to the
    > sysopen call.
    > However, I cannot find that all the corresponding files would have
    > existed.
    > I read the doc and get to:
    >
    > In many systems the "O_EXCL" flag is available for opening
    > files
    > in exclusive mode. This is not locking: exclusiveness
    > means here
    > that if the file already exists, sysopen() fails. "O_EXCL"
    > may
    > not work on network filesystems, and has no effect unless
    > the
    > "O_CREAT" flag is set as well.


    If the network filesystem doesn't support O_EXCL, then the open will
    succeed even though it shouldn't (or it will fail every time with
    EINVAL), it won't fail when the file doesn't exist.


    > The script does write to a network filesystems (home directory on a
    > remote filer, 4 ms round-trip).
    >
    > Shoud I look for a replacement for sysopen?


    No. sysopen is the closest you can get to the OS. If sysopen reporte
    EEXIST, then the OS really thinks the file exists at the time.

    > Or for an other theory to explain the problem?


    Yes. Most likely causes are IMHO:

    * The file really exists at the time. You have to find out why
    (maybe your script is supposed to remove the file before it
    terminates and it either doesn't do it or a previous invocation
    hasn't finished yet). If you know why the solution is probably
    obvious.
    * The file did exist and has already been removed at the time the
    script runs, but the information about the file's existence is still
    cached by the OS. In this case you should check the configuration of
    the file system (on both the client and the server).

    hp
     
    Peter J. Holzer, Aug 7, 2010
    #5
  6. Marc Girod wrote:
    > On Aug 7, 7:40 am, "John W. Krahn"<> wrote:
    >
    >> Your code does not check the return value from sysopen() so the error
    >> message you are receiving is not related to sysopen().

    >
    > Doesn't it?
    > The block is an if block.
    > If the condition (the return from sysopen) is false,


    Or true.

    > the function returns 0 unconditionally.


    Correct.

    > Besides, the error I get from $!, at the time of reporting in the
    > calling function is, as I wrote it, 'File exists'.


    By that time $! could have been changed by some other system function.

    You need to capture or print the value of $! immediately after the
    system function that sets it, for example:

    sysopen my $FILE, $file, O_EXCL | O_CREAT | O_WRONLY, $mode or do {
    warn "Cannot open '$file' $!";
    return 0;
    };


    *Note that "$mode" is really the permissions field, and the MODE field
    is actually "O_EXCL | O_CREAT | O_WRONLY".




    John
    --
    Any intelligent fool can make things bigger and
    more complex... It takes a touch of genius -
    and a lot of courage to move in the opposite
    direction. -- Albert Einstein
     
    John W. Krahn, Aug 7, 2010
    #6
  7. Marc Girod

    Marc Girod Guest

    On Aug 7, 1:10 pm, "Peter J. Holzer" <> wrote:

    > If the network filesystem doesn't support O_EXCL, then the open will
    > succeed even though it shouldn't (or it will fail every time with
    > EINVAL), it won't fail when the file doesn't exist.


    Thanks.

    > No. sysopen is the closest you can get to the OS.


    OK.

    > Yes. Most likely causes are IMHO:
    >
    >  * The file really exists at the time. You have to find out why
    >    (maybe your script is supposed to remove the file before it
    >    terminates and it either doesn't do it or a previous invocation
    >    hasn't finished yet). If you know why the solution is probably
    >    obvious.
    >  * The file did exist and has already been removed at the time the
    >    script runs, but the information about the file's existence is still
    >    cached by the OS. In this case you should check the configuration of
    >    the file system (on both the client and the server).


    Well, I have to figure out how either could be possible.
    I cannot see what in the code could remove anything...
    Those files are left to be accessible from a web page.
    There is a loop which increments a digit in the file name and tries
    100 times.
    I don't think this is very smart, but I cannot see how the 100
    different files could exist.

    The error returned in the end is of course only the one for the last
    try.
    So that I don't *know* why the 99 first attempts failed.
    But in the general case, I cannot find the 100th file either.

    I may leave this for a while: I have more urgent, and the symptom has
    stopped for now.
    But I don't doubt it will come back.

    Thanks.
    Marc
     
    Marc Girod, Aug 9, 2010
    #7
  8. Marc Girod

    Marc Girod Guest

    On Aug 7, 6:40 pm, "John W. Krahn" <> wrote:

    > Or true.


    Indeed.

    > By that time $! could have been changed by some other system function.


    Yes.

    > You need to capture or print the value of $! immediately after the
    > system function that sets it, for example:


    Yes...

    > *Note that "$mode" is really the permissions field, and the MODE field
    > is actually "O_EXCL | O_CREAT | O_WRONLY".


    I saw that. It was set to 0444.
    Thanks. I'll save a pointer to this and come back later.

    Marc
     
    Marc Girod, Aug 9, 2010
    #8
  9. On 2010-08-09 11:50, Marc Girod <> wrote:

    [sysopen(... O_EXCL ...) fails with EEXIST although the file shouldn't
    exist]

    > Well, I have to figure out how either could be possible.
    > I cannot see what in the code could remove anything...
    > Those files are left to be accessible from a web page.
    > There is a loop which increments a digit in the file name and tries
    > 100 times.
    > I don't think this is very smart, but I cannot see how the 100
    > different files could exist.
    >
    > The error returned in the end is of course only the one for the last
    > try.
    > So that I don't *know* why the 99 first attempts failed.
    > But in the general case, I cannot find the 100th file either.


    I would log each failure with all the details I can think of. In this
    case:

    * exact name of the file to be created
    * $! immediately after the failure
    * cwd at the time of the failure
    * try to lstat the file just after the failure (but after logging $!,
    or you will change it!) and log all relevant information if the
    file exists - this will help you to determine where the spurious
    files come from


    > I may leave this for a while: I have more urgent, and the symptom has
    > stopped for now.
    > But I don't doubt it will come back.


    If you add the diagnostics now you will have the information next time
    the problem occurs.

    hp
     
    Peter J. Holzer, Aug 9, 2010
    #9
  10. On Mon, 09 Aug 2010 04:50:35 -0700, Marc Girod wrote:

    > Well, I have to figure out how either could be possible. I cannot see
    > what in the code could remove anything... Those files are left to be
    > accessible from a web page. There is a loop which increments a digit in
    > the file name and tries 100 times.
    > I don't think this is very smart, but I cannot see how the 100 different
    > files could exist.
    >
    > The error returned in the end is of course only the one for the last
    > try.
    > So that I don't *know* why the 99 first attempts failed. But in the
    > general case, I cannot find the 100th file either.


    Stupid suggestion and does not match exactly what you wrote above, but:
    Are there maybe two instances of your program running and interfering
    with each other?

    M4
     
    Martijn Lievaart, Aug 9, 2010
    #10
  11. Marc Girod

    Marc Girod Guest

    On Aug 9, 6:32 pm, Martijn Lievaart <> wrote:

    > Stupid suggestion and does not match exactly what you wrote above, but:
    > Are there maybe two instances of your program running and interfering
    > with each other?


    Thanks... This script is invoked via a ~/.forward file, i.e. for every
    incoming mail. I guess it is very possible that the same mail is
    presented twice, e.g. if it contains the address twice (?)
    Again something I should log.
    But each instance generates its own filenames, and sure, the first
    would easily clash (protected only by timestamp), but it should impact
    only the first of the two instances...

    Marc
     
    Marc Girod, Aug 10, 2010
    #11
  12. Marc Girod

    C.DeRykus Guest

    On Aug 10, 12:20 am, Marc Girod <> wrote:
    > On Aug 9, 6:32 pm, Martijn Lievaart <> wrote:
    >
    > > Stupid suggestion and does not match exactly what you wrote above, but:
    > > Are there maybe two instances of your program running and interfering
    > > with each other?

    >
    > Thanks... This script is invoked via a ~/.forward file, i.e. for every
    > incoming mail. I guess it is very possible that the same mail is
    > presented twice, e.g. if it contains the address twice (?)
    > Again something I should log.
    > But each instance generates its own filenames, and sure, the first
    > would easily clash (protected only by timestamp), but it should impact
    > only the first of the two instances...
    >


    Another wild speculation: a .forward loop could bombard
    a server with the same message rapidly until the loop's
    detected and shutdown. A possible error race condition
    might ensue...Still, I suppose a .forward loop shutdown
    would've been under the microscope by now.

    The added logging/diagnostics should flush this out too.

    --
    Charles DeRykus
     
    C.DeRykus, Aug 10, 2010
    #12
  13. On Tue, 10 Aug 2010 00:20:34 -0700, Marc Girod wrote:

    > On Aug 9, 6:32 pm, Martijn Lievaart <> wrote:
    >
    >> Stupid suggestion and does not match exactly what you wrote above, but:
    >> Are there maybe two instances of your program running and interfering
    >> with each other?

    >
    > Thanks... This script is invoked via a ~/.forward file, i.e. for every
    > incoming mail. I guess it is very possible that the same mail is
    > presented twice, e.g. if it contains the address twice (?) Again
    > something I should log.
    > But each instance generates its own filenames, and sure, the first would
    > easily clash (protected only by timestamp), but it should impact only
    > the first of the two instances...


    In that case I suggest you make sure the filenames are really unique
    (hint, use the pid, available in $$), or implement some sort of locking
    mechanism.

    M4
     
    Martijn Lievaart, Aug 10, 2010
    #13
  14. On 2010-08-10 22:16, Martijn Lievaart <> wrote:
    > On Tue, 10 Aug 2010 00:20:34 -0700, Marc Girod wrote:
    >> On Aug 9, 6:32 pm, Martijn Lievaart <> wrote:
    >>> Stupid suggestion and does not match exactly what you wrote above, but:
    >>> Are there maybe two instances of your program running and interfering
    >>> with each other?

    >>
    >> Thanks... This script is invoked via a ~/.forward file, i.e. for every
    >> incoming mail. I guess it is very possible that the same mail is
    >> presented twice, e.g. if it contains the address twice (?) Again
    >> something I should log.
    >> But each instance generates its own filenames, and sure, the first would
    >> easily clash (protected only by timestamp), but it should impact only
    >> the first of the two instances...

    >
    > In that case I suggest you make sure the filenames are really unique
    > (hint, use the pid, available in $$),


    Since the files are not removed before the program exits, using the pid
    doesn't make the filenames unique. Even with a timestamp with a
    resolution of one second there is a chance that the pid wraps around too
    fast (although it's pretty unlikely).

    > or implement some sort of locking
    > mechanism.


    O_EXCL is "some kind of locking mechanism": Any already existing file is
    "locked" and won't be overwritten. It doesn't wait (wouldn't be very
    useful since files aren't deleted) but fails immediately. Then the
    program tries a new file name. This looks ok (it is also a usual method
    for creating temporary files). Maybe the "try next file filename" part
    doesn't work correctly - we haven't seen the code and it probably isn't
    well tested.

    Oh, and for everything running from a .forward file: Be sure to use the
    correct exit codes from sys_exit.h.

    hp
     
    Peter J. Holzer, Aug 12, 2010
    #14
  15. Marc Girod

    C.DeRykus Guest

    On Aug 12, 8:04 am, "Peter J. Holzer" <> wrote:
    > On 2010-08-10 22:16, Martijn Lievaart <> wrote:
    >
    >
    >
    > > On Tue, 10 Aug 2010 00:20:34 -0700, Marc Girod wrote:
    > >> On Aug 9, 6:32 pm, Martijn Lievaart <> wrote:

    ....
    > > or implement some sort of locking
    > > mechanism.

    >
    > O_EXCL is "some kind of locking mechanism": Any already existing file is
    > "locked" and won't be overwritten. It doesn't wait (wouldn't be very
    > useful since files aren't deleted) but fails immediately. Then the
    > program tries a new file name. This looks ok (it is also a usual method
    > for creating temporary files). Maybe the "try next file filename" part
    > doesn't work correctly - we haven't seen the code and it probably isn't
    > well tested.


    It's not clear to me what O_EXCL will do though
    since the files are remote and the OP's doc say:

    "O_EXCL" may not work on network filesystems ...

    This makes me wonder if "not work" might well
    manifest as a flurry of false positives for
    an error at times but work normally at other
    times.

    I don't know if there are other network locking
    mechanisms that'd be easy or reliable though.

    --
    Charles DeRykus
     
    C.DeRykus, Aug 12, 2010
    #15
  16. On 2010-08-12 19:01, C.DeRykus <> wrote:
    > It's not clear to me what O_EXCL will do though
    > since the files are remote and the OP's doc say:
    >
    > "O_EXCL" may not work on network filesystems ...
    >
    > This makes me wonder if "not work" might well
    > manifest as a flurry of false positives for
    > an error at times but work normally at other
    > times.


    I already covered this in my first posting in this thread. If you can
    add anything substantial to that, please do.

    hp
     
    Peter J. Holzer, Aug 12, 2010
    #16
  17. Marc Girod

    C.DeRykus Guest

    On Aug 12, 1:03 pm, "Peter J. Holzer" <> wrote:
    > On 2010-08-12 19:01, C.DeRykus <> wrote:
    >
    > > It's not clear to me what O_EXCL will do though
    > > since the files are remote and the OP's doc say:

    >
    > >    "O_EXCL" may not work on network filesystems ...

    >
    > > This makes me wonder if "not work" might well
    > > manifest  as a flurry of false positives for
    > > an error at times but work normally at other
    > > times.

    >
    > I already covered this in my first posting in this thread. If you can
    > add anything substantial to that, please do.
    >


    I must've missed it. I did see the remark that
    'it's very likely that “File exists” is set by
    sysopen' without explicit mention of sysopen not
    being guaranteed to work over the network.

    --
    Charles DeRykus
     
    C.DeRykus, Aug 12, 2010
    #17
  18. In article
    <>,
    "C.DeRykus" <> wrote:

    > On Aug 12, 1:03 pm, "Peter J. Holzer" <> wrote:
    > > On 2010-08-12 19:01, C.DeRykus <> wrote:
    > >
    > > > It's not clear to me what O_EXCL will do though
    > > > since the files are remote and the OP's doc say:

    > >
    > > >    "O_EXCL" may not work on network filesystems ...

    > >
    > > > This makes me wonder if "not work" might well
    > > > manifest  as a flurry of false positives for
    > > > an error at times but work normally at other
    > > > times.

    > >
    > > I already covered this in my first posting in this thread. If you can
    > > add anything substantial to that, please do.
    > >

    >
    > I must've missed it. I did see the remark that
    > 'it's very likely that ³File exists² is set by
    > sysopen' without explicit mention of sysopen not
    > being guaranteed to work over the network.
    >
    > --
    > Charles DeRykus


    I've been following this thread since it started. This is the first
    mention of the file being on a networked filesystem.

    --
    DeeDee, don't press that button! DeeDee! NO! Dee...
    [I filter all Goggle Groups posts, so any reply may be automatically ignored]
     
    Michael Vilain, Aug 13, 2010
    #18
  19. Marc Girod

    C.DeRykus Guest

    On Aug 12, 9:16 pm, Michael Vilain <> wrote:
    > In article
    > <>,
    >
    >
    >
    >  "C.DeRykus" <> wrote:
    > > On Aug 12, 1:03 pm, "Peter J. Holzer" <> wrote:
    > > > On 2010-08-12 19:01, C.DeRykus <> wrote:

    >
    > > > > It's not clear to me what O_EXCL will do though
    > > > > since the files are remote and the OP's doc say:

    >
    > > > >    "O_EXCL" may not work on network filesystems ...

    >
    > > > > This makes me wonder if "not work" might well
    > > > > manifest  as a flurry of false positives for
    > > > > an error at times but work normally at other
    > > > > times.

    >
    > > > I already covered this in my first posting in this thread. If you can
    > > > add anything substantial to that, please do.

    >
    > > I must've missed it. I did see the remark that
    > > 'it's  very likely that ³File exists² is set by
    > > sysopen' without explicit mention of sysopen not
    > > being guaranteed to work over the network.

    >


    >
    > I've been following this thread since it started.  This is the first
    > mention of the file being on a networked filesystem.


    See the original post:

    "The script does write to a network filesystems
    (home directory on a remote filer, 4 ms round-trip)."

    --
    Charles DeRykus
     
    C.DeRykus, Aug 13, 2010
    #19
  20. On 2010-08-12 22:31, C.DeRykus <> wrote:
    > On Aug 12, 1:03 pm, "Peter J. Holzer" <> wrote:
    >> On 2010-08-12 19:01, C.DeRykus <> wrote:
    >> > It's not clear to me what O_EXCL will do though
    >> > since the files are remote and the OP's doc say:

    >>
    >> >    "O_EXCL" may not work on network filesystems ...

    >>
    >> > This makes me wonder if "not work" might well
    >> > manifest  as a flurry of false positives for
    >> > an error at times but work normally at other
    >> > times.

    >>
    >> I already covered this in my first posting in this thread. If you can
    >> add anything substantial to that, please do.
    >>

    >
    > I must've missed it.


    <>

    hp
     
    Peter J. Holzer, Aug 13, 2010
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Spencer

    Re: So many failures early on

    Kevin Spencer, Aug 13, 2003, in forum: ASP .Net
    Replies:
    4
    Views:
    339
    Kevin Spencer
    Aug 27, 2003
  2. Hon Guin Lee - Web Producer - SMI Marketing

    sysopen problem

    Hon Guin Lee - Web Producer - SMI Marketing, Oct 7, 2003, in forum: Perl Misc
    Replies:
    1
    Views:
    110
  3. Michele Dondi

    problem with sysopen() on nfs

    Michele Dondi, Sep 18, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    101
    Michele Dondi
    Sep 19, 2004
  4. Michele Dondi

    Q re sysopen() and more...

    Michele Dondi, Oct 13, 2004, in forum: Perl Misc
    Replies:
    6
    Views:
    113
    Michele Dondi
    Oct 15, 2004
  5. Tomasz Chmielewski

    sysopen - die only if EBUSY?

    Tomasz Chmielewski, Oct 2, 2008, in forum: Perl Misc
    Replies:
    3
    Views:
    124
    Ben Morrow
    Oct 2, 2008
Loading...

Share This Page