fcntl problems

Discussion in 'Python' started by mhearne808[insert-at-sign-here]gmail[insert-dot-he, Aug 30, 2007.

  1. I'm having a number of problems with the fcntl module. First off, my
    system info:

    Mac OS X
    Darwin igskcicglthearn.cr.usgs.gov 8.10.1 Darwin Kernel Version
    8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386
    i386 i386
    Python 2.5.1 (built from source)

    OK, the weirdness:

    First of all, if I try this:
    file = open("counter.txt","w+")
    fcntl.flock(file.fileno(), fcntl.LOCK_NB)

    I get this:
    ---------------------------------------------------------------------------
    <type 'exceptions.IOError'> Traceback (most recent call
    last)
    /Users/mhearne/src/python/<ipython console> in <module>()
    <type 'exceptions.IOError'>: [Errno 9] Bad file descriptor

    However, if I try this:
    fcntl.flock(file.fileno(), fcntl.LOCK_EX)

    I get no errors.

    Proceeding forward with the locked file, let's say I do the above in
    Python interactive Process A. Then in python interactive Process B, I
    repeat the "open" function on the same file with the same
    permissions. Then, in each process, I write some text to the file
    using the write() method. After closing the file in both processes,
    the only text I see in the file is from Process B!

    According to my Python Cookbook:
    "Exclusive lock: This denies all _other_ processes both read and write
    access to the file."

    I seem to be experiencing the reverse of that description. Is this my
    lack of understanding, or have I discovered a bug?

    Thanks,

    Mike
    mhearne808[insert-at-sign-here]gmail[insert-dot-he, Aug 30, 2007
    #1
    1. Advertising

  2. On Aug 30, 4:19 pm, "mhearne808[insert-at-sign-here]gmail[insert-dot-
    here]com" <> wrote:
    > I'm having a number of problems with the fcntl module. First off, my
    > system info:
    >
    > Mac OS X
    > Darwin igskcicglthearn.cr.usgs.gov 8.10.1 Darwin Kernel Version
    > 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386
    > i386 i386
    > Python 2.5.1 (built from source)
    >
    > OK, the weirdness:
    >
    > First of all, if I try this:
    > file = open("counter.txt","w+")
    > fcntl.flock(file.fileno(), fcntl.LOCK_NB)
    >
    > I get this:
    > ---------------------------------------------------------------------------
    > <type 'exceptions.IOError'> Traceback (most recent call
    > last)
    > /Users/mhearne/src/python/<ipython console> in <module>()
    > <type 'exceptions.IOError'>: [Errno 9] Bad file descriptor
    >
    > However, if I try this:
    > fcntl.flock(file.fileno(), fcntl.LOCK_EX)
    >
    > I get no errors.
    >
    > Proceeding forward with the locked file, let's say I do the above in
    > Python interactive Process A. Then in python interactive Process B, I
    > repeat the "open" function on the same file with the same
    > permissions. Then, in each process, I write some text to the file
    > using the write() method. After closing the file in both processes,
    > the only text I see in the file is from Process B!
    >
    > According to my Python Cookbook:
    > "Exclusive lock: This denies all _other_ processes both read and write
    > access to the file."
    >
    > I seem to be experiencing the reverse of that description. Is this my
    > lack of understanding, or have I discovered a bug?
    >
    > Thanks,
    >
    > Mike


    I've been doing some experiments, and here are some specific examples
    to try. I'll designate the two interactive python processes as PA and
    PB. Both processes were started in the same directory. Here goes:
    PA: import fcntl
    PA: f = open("foo.txt","w+")
    PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    PA: f.write("text1\n")
    PB: f = open("foo.txt","w+")
    PB: f.write("text2\n")
    PA: f.close()
    PB: f.close()

    contents of foo.txt are:
    text2

    Second experiment:

    PA: f = open("foo.txt","w+")
    PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    PB: f = open("foo.txt","w+")
    PA: f.write("text1\n")
    PB: f.write("text2\n")
    PA: f.write("text3\n")
    PB: f.close()
    PA: f.write("text4\n")
    PA: f.close()

    contents of foo.txt are:
    text1
    text3
    text4

    Third experiment:
    PA: f = open("foo.txt","w+")
    PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
    PA: f.write("text1\n")
    PB: f = open("foo.txt","w+")
    PB: f.write("text2\n")
    PB: f.close()
    PA: f.close()

    contents of foo.txt are:
    text1

    Fourth experiment:
    PA: f = open("foo.txt","w+")
    PA: f.write("text1\n")
    PB: f = open("foo.txt","w+")
    PB: f.write("text2\n")
    PB: f.close()
    PA: f.close()

    contents of foo.txt are:
    text1

    >From these last two experiments I can only conclude that file locking

    isn't doing a durned thing.

    What's going on?

    --Mike
    mhearne808[insert-at-sign-here]gmail[insert-dot-he, Aug 31, 2007
    #2
    1. Advertising

  3. mhearne808[insert-at-sign-here]gmail[insert-dot-he

    Miles Guest

    On 8/30/07, mhearne808 wrote:
    > I'm having a number of problems with the fcntl module.


    Read this first: http://linux.die.net/man/2/flock

    > First of all, if I try this:
    > file = open("counter.txt","w+")
    > fcntl.flock(file.fileno(), fcntl.LOCK_NB)
    >
    > I get this:
    > ---------------------------------------------------------------------------
    > <type 'exceptions.IOError'> Traceback (most recent call
    > last)
    > /Users/mhearne/src/python/<ipython console> in <module>()
    > <type 'exceptions.IOError'>: [Errno 9] Bad file descriptor


    That should be:
    >>> fcntl.flock(f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)


    > Proceeding forward with the locked file, let's say I do the above in
    > Python interactive Process A. Then in python interactive Process B, I
    > repeat the "open" function on the same file with the same
    > permissions. Then, in each process, I write some text to the file
    > using the write() method. After closing the file in both processes,
    > the only text I see in the file is from Process B!


    This is due to two issues: caching and file position. When you open
    the file in both processes as 'w+', they are both positioned at the
    *current* EOF, but from that point on the offset is not externally
    influenced. The correct sequence of events should be:
    - open file in mode w+
    - obtain exclusive lock
    - f.seek(0, 2) # (to end of file)
    - write to file
    - f.flush() # or f.close()
    - release lock

    > Is this my lack of understanding, or have I discovered a bug?


    If you find yourself asking this question, it's too often the former :)

    -Miles
    Miles, Aug 31, 2007
    #3
  4. mhearne808[insert-at-sign-here]gmail[insert-dot-he

    Miles Guest

    Sorry, that last quote-only reply was accidental. :)

    On 8/30/07, mhearne808 wrote:
    > I've been doing some experiments, and here are some specific examples
    > to try.


    [snipped examples]

    > From these last two experiments I can only conclude that file locking
    > isn't doing a durned thing.
    >
    > What's going on?


    File locking isn't doing a durned thing in those cases because you're
    only obtaining the lock from a single process.

    > According to my Python Cookbook:
    > "Exclusive lock: This denies all _other_ processes both read and write
    > access to the file."


    This is only for mandatory locking; POSIX flock is advisory locking,
    which states: "Only one process may hold an exclusive lock for a given
    file at a given time." Advisory locks don't have any effect on
    processes that don't use locks. Mandatory locks are kernel enforced,
    but non-POSIX and not available in Mac OS X.

    -Miles
    Miles, Aug 31, 2007
    #4
  5. On Aug 31, 12:23 am, Miles <> wrote:
    > Sorry, that last quote-only reply was accidental. :)
    >
    > On 8/30/07, mhearne808 wrote:
    > > I've been doing some experiments, and here are some specific examples
    > > to try.

    >
    > [snipped examples]
    >
    > > From these last two experiments I can only conclude that file locking
    > > isn't doing a durned thing.

    >
    > > What's going on?

    >
    > File locking isn't doing a durned thing in those cases because you're
    > only obtaining the lock from a single process.
    >
    > > According to my Python Cookbook:
    > > "Exclusive lock: This denies all _other_ processes both read and write
    > > access to the file."

    >
    > This is only for mandatory locking; POSIX flock is advisory locking,
    > which states: "Only one process may hold an exclusive lock for a given
    > file at a given time." Advisory locks don't have any effect on
    > processes that don't use locks. Mandatory locks are kernel enforced,
    > but non-POSIX and not available in Mac OS X.
    >
    > -Miles


    I think I'm still confused. Maybe I should explain the behavior that
    I want, and then figure out if it is possible.

    I have a script that will be run from a cron job once a minute. One
    of the things this script will do is open a file to stash some
    temporary results. I expect that this script will always finish its
    work in less than 15 seconds, but I didn't want to depend on that.
    Thus I started to look into file locking, which I had hoped I could
    use in the following fashion:

    Process A opens file foo
    Process A locks file foo
    Process A takes more than a minute to do its work
    Process B wakes up
    Process B determines that file foo is locked
    Process B quits in disgust
    Process A finishes its work

    Since I couldn't figure out file locking, I decided to just have
    Process A create a "pid" file in the directory - analogous to the
    "Occupied" sign on an airplane bathroom. This works, but it seems a
    little hacky.

    --Mike
    mhearne808[insert-at-sign-here]gmail[insert-dot-he, Aug 31, 2007
    #5
  6. "mhearne808[insert-at-sign-here]gmail[insert-dot-here]com" <> writes:

    > I think I'm still confused.


    What Miles tried to tell you is that you should call fcnt.flock from
    both PA and PB. In the example you posted, you failed to call it from
    PB. No lock call, so no locking happened.

    > I have a script that will be run from a cron job once a minute. One
    > of the things this script will do is open a file to stash some
    > temporary results. I expect that this script will always finish its
    > work in less than 15 seconds, but I didn't want to depend on that.
    >
    > Thus I started to look into file locking, which I had hoped I could
    > use in the following fashion:
    >
    > Process A opens file foo
    > Process A locks file foo
    > Process A takes more than a minute to do its work
    > Process B wakes up
    > Process B determines that file foo is locked
    > Process B quits in disgust
    > Process A finishes its work


    File locking supports that scenario, as you suspected. You need to
    use flock with LOCK_EX|LOCK_NB. If the call succeeds, you got the
    lock. If you get an exception whose errno is EWOULDBLOCK, you quit in
    disgust.
    Hrvoje Niksic, Aug 31, 2007
    #6
  7. mhearne808[insert-at-sign-here]gmail[insert-dot-he

    Miles Guest

    On 8/31/07, mhearne808 wrote:
    > I have a script that will be run from a cron job once a minute. One
    > of the things this script will do is open a file to stash some
    > temporary results. I expect that this script will always finish its
    > work in less than 15 seconds, but I didn't want to depend on that.
    > Thus I started to look into file locking, which I had hoped I could
    > use in the following fashion:
    >
    > Process A opens file foo
    > Process A locks file foo
    > Process A takes more than a minute to do its work
    > Process B wakes up
    > Process B determines that file foo is locked
    > Process B quits in disgust
    > Process A finishes its work


    That would look like (untested):

    import fcntl, sys
    f = open('foo', 'w+')
    try:
    fcntl.flock(f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError, e:
    if e.args[0] == 35:
    sys.exit(1)
    else:
    raise
    f.seek(0, 2) # seek to end
    # do your thing with the file
    f.flush()
    fcntl.flock(f.fileno(), fcntl.LOCK_UN)
    f.close()

    -Miles
    Miles, Aug 31, 2007
    #7
  8. On Aug 31, 8:42 am, Miles <> wrote:
    > On 8/31/07, mhearne808 wrote:
    > > I have a script that will be run from a cron job once a minute. One
    > > of the things this script will do is open a file to stash some
    > > temporary results. I expect that this script will always finish its
    > > work in less than 15 seconds, but I didn't want to depend on that.
    > > Thus I started to look into file locking, which I had hoped I could
    > > use in the following fashion:

    >
    > > Process A opens file foo
    > > Process A locks file foo
    > > Process A takes more than a minute to do its work
    > > Process B wakes up
    > > Process B determines that file foo is locked
    > > Process B quits in disgust
    > > Process A finishes its work

    >
    > That would look like (untested):
    >
    > importfcntl, sys
    > f = open('foo', 'w+')
    > try:
    > fcntl.flock(f.fileno(),fcntl.LOCK_EX |fcntl.LOCK_NB)
    > except IOError, e:
    > if e.args[0] == 35:
    > sys.exit(1)
    > else:
    > raise
    > f.seek(0, 2) # seek to end
    > # do your thing with the file
    > f.flush()fcntl.flock(f.fileno(),fcntl.LOCK_UN)
    > f.close()
    >
    > -Miles


    I tested that, and it works! Thanks!

    Looking at my flock(3) man page, I'm guessing that "35" is the error
    code for EWOULDBLOCK. Which system header file am I supposed to look
    in to figure that magic number out?

    I would make the argument that this module could be either more
    pythonic, or simply documented more completely. The open source
    response, of course, would be "go for it!".

    --Mike
    mhearne808[insert-at-sign-here]gmail[insert-dot-he, Aug 31, 2007
    #8
  9. mhearne808[insert-at-sign-here]gmail[insert-dot-he

    Miles Guest

    On 8/31/07, mhearne808 wrote:
    > Looking at my flock(3) man page, I'm guessing that "35" is the error
    > code for EWOULDBLOCK. Which system header file am I supposed to look
    > in to figure that magic number out?


    I got the error number by looking at the IOError exception raised when
    playing with the interactive interpreter, but I really should have
    written:

    from errno import EWOULDBLOCK
    ....
    if e.args[0] == EWOULDBLOCK:
    ....

    - Miles
    Miles, Aug 31, 2007
    #9
  10. In article <>,
    "mhearne808[insert-at-sign-here]gmail[insert-dot-here]com"
    <> wrote:

    > Looking at my flock(3) man page, I'm guessing that "35" is the error
    > code for EWOULDBLOCK. Which system header file am I supposed to look
    > in to figure that magic number out?


    On a MacOS system, you can find them in /usr/include/sys/errno.h
    On a Linux system, try /usr/include/asm-generic/errno.h

    However, if you're writing in Python, you will probably have an easier
    time using the "errno" module, e.g.,

    ] import errno
    ] errno.errorcode[35]
    'EDEADLOCK'

    Note that some codes have multiple names (e.g., EAGAIN and EWOULDBLOCK)
    so that this lookup may not return exactly the name you're expecting.

    Cheers,
    -M

    --
    Michael J. Fromberger | Lecturer, Dept. of Computer Science
    http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA
    Michael J. Fromberger, Aug 31, 2007
    #10
  11. In message <>, Miles
    wrote:

    > except IOError, e:
    > if e.args[0] == 35:


    Why not

    except IOError, (ErrNo, Msg) :
    if ErrNo == errno.EAGAIN :
    Lawrence D'Oliveiro, Sep 1, 2007
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. wicur

    Fcntl problem

    wicur, Feb 2, 2004, in forum: Perl
    Replies:
    0
    Views:
    508
    wicur
    Feb 2, 2004
  2. =?ISO-8859-1?Q?J=FCrgen_Voss?=

    locking with fcntl

    =?ISO-8859-1?Q?J=FCrgen_Voss?=, Apr 21, 2004, in forum: C Programming
    Replies:
    1
    Views:
    268
    Joona I Palaste
    Apr 21, 2004
  3. Meyer, Tony

    FCNTL module deprecation warning

    Meyer, Tony, Jul 15, 2003, in forum: Python
    Replies:
    2
    Views:
    325
    Peter Hansen
    Jul 15, 2003
  4. Omid Fatemi

    fcntl and FCNTL

    Omid Fatemi, Oct 30, 2004, in forum: Python
    Replies:
    4
    Views:
    6,827
    Steve Holden
    Nov 1, 2004
  5. Mitko Haralanov

    Invalid argument with fcntl.fcntl

    Mitko Haralanov, Jun 6, 2007, in forum: Python
    Replies:
    0
    Views:
    534
    Mitko Haralanov
    Jun 6, 2007
Loading...

Share This Page