locking files on Linux

Discussion in 'Python' started by andrea crotti, Oct 18, 2012.

  1. I'm trying to understand how I can lock a file while writing on it,
    because I might have multiple processes working on it at the same time.

    I found the fcntl.lockf function but if I do this:

    In [109]: locked = open('locked.txt', 'w')

    In [110]: fcntl.lockf(locked, fcntl.LOCK_EX)

    I can happily open the file with vim from somewhere and write on it, so
    it doesn't seem to be very useful, or am I missing something?

    I can otherwise use the classic ".lock" file when working on it, but I
    would be glad to see a smarter solution (should only work on Linux)..
    andrea crotti, Oct 18, 2012
    #1
    1. Advertising

  2. On 2012-10-18, andrea crotti <> wrote:

    > I'm trying to understand how I can lock a file while writing on it,
    > because I might have multiple processes working on it at the same time.
    >
    > I found the fcntl.lockf function but if I do this:
    >
    > In [109]: locked = open('locked.txt', 'w')
    >
    > In [110]: fcntl.lockf(locked, fcntl.LOCK_EX)
    >
    > I can happily open the file with vim from somewhere and write on it, so
    > it doesn't seem to be very useful, or am I missing something?


    File locks under Unix have historically been "advisory". That means
    that programs have to _choose_ to pay attention to them. Most
    programs do not.

    Linux does support mandatory locking, but it's rarely used and must be
    manually enabled at the filesystem level. It's probably worth noting
    that in the Linux kernel docs, the document on mandatory file locking
    begins with a section titled "Why you should avoid mandatory locking".

    http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems
    http://kernel.org/doc/Documentation/filesystems/locks.txt
    http://kernel.org/doc/Documentation/filesystems/mandatory-locking.txt
    http://www.thegeekstuff.com/2012/04/linux-file-locking-types/
    http://www.hackinglinuxexposed.com/articles/20030623.html

    --
    Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
    at twin NECTARINES above
    gmail.com a MOUTH that knows no
    BOUNDS --
    Grant Edwards, Oct 18, 2012
    #2
    1. Advertising

  3. 2012/10/18 Grant Edwards <>:
    > On 2012-10-18, andrea crotti <> wrote:
    >
    >
    > File locks under Unix have historically been "advisory". That means
    > that programs have to _choose_ to pay attention to them. Most
    > programs do not.
    >
    > Linux does support mandatory locking, but it's rarely used and must be
    > manually enabled at the filesystem level. It's probably worth noting
    > that in the Linux kernel docs, the document on mandatory file locking
    > begins with a section titled "Why you should avoid mandatory locking".
    >
    > http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems
    > http://kernel.org/doc/Documentation/filesystems/locks.txt
    > http://kernel.org/doc/Documentation/filesystems/mandatory-locking.txt
    > http://www.thegeekstuff.com/2012/04/linux-file-locking-types/
    > http://www.hackinglinuxexposed.com/articles/20030623.html
    >
    > --
    > Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
    > at twin NECTARINES above
    > gmail.com a MOUTH that knows no
    > BOUNDS --
    > --
    > http://mail.python.org/mailman/listinfo/python-list



    Uhh I see thanks, I guess I'll use the good-old .lock file (even if it
    might have some problems too).

    Anyway I'm only afraid that my same application could modify the
    files, so maybe I can instruct it to check if the file is locked.

    Or maybe using sqlite would work even if writing from different
    processes?

    I would prefer to keep something human readable as INI-format though,
    rather then a sqlite file..

    Thanks
    andrea crotti, Oct 18, 2012
    #3
  4. On 2012-10-18, andrea crotti <> wrote:
    > 2012/10/18 Grant Edwards <>:
    >> On 2012-10-18, andrea crotti <> wrote:
    >>
    >> File locks under Unix have historically been "advisory". That means
    >> that programs have to _choose_ to pay attention to them. Most
    >> programs do not.
    >>
    >> Linux does support mandatory locking, but it's rarely used and must be
    >> manually enabled at the filesystem level. It's probably worth noting
    >> that in the Linux kernel docs, the document on mandatory file locking
    >> begins with a section titled "Why you should avoid mandatory locking".

    >
    > Uhh I see thanks, I guess I'll use the good-old .lock file (even if
    > it might have some problems too).
    >
    > Anyway I'm only afraid that my same application could modify the
    > files, so maybe I can instruct it to check if the file is locked.


    If what you're guarding against is multiple instances of your
    application modifying the file, then either of the advisory file
    locking schemes or the separate lock file should work fine.

    --
    Grant Edwards grant.b.edwards Yow! All this time I've
    at been VIEWING a RUSSIAN
    gmail.com MIDGET SODOMIZE a HOUSECAT!
    Grant Edwards, Oct 18, 2012
    #4
  5. On 18 October 2012 14:44, andrea crotti <> wrote:
    > 2012/10/18 Grant Edwards <>:
    >> On 2012-10-18, andrea crotti <> wrote:
    >>
    >>
    >> File locks under Unix have historically been "advisory". That means
    >> that programs have to _choose_ to pay attention to them. Most
    >> programs do not.
    >>
    >> Linux does support mandatory locking, but it's rarely used and must be
    >> manually enabled at the filesystem level. It's probably worth noting
    >> that in the Linux kernel docs, the document on mandatory file locking
    >> begins with a section titled "Why you should avoid mandatory locking".
    >>
    >> http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems
    >> http://kernel.org/doc/Documentation/filesystems/locks.txt
    >> http://kernel.org/doc/Documentation/filesystems/mandatory-locking.txt
    >> http://www.thegeekstuff.com/2012/04/linux-file-locking-types/
    >> http://www.hackinglinuxexposed.com/articles/20030623.html
    >>
    >> --
    >> Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
    >> at twin NECTARINES above
    >> gmail.com a MOUTH that knows no
    >> BOUNDS --
    >> --
    >> http://mail.python.org/mailman/listinfo/python-list

    >
    >
    > Uhh I see thanks, I guess I'll use the good-old .lock file (even if it
    > might have some problems too).


    I think you've misunderstood what Grant meant.

    >
    > Anyway I'm only afraid that my same application could modify the
    > files, so maybe I can instruct it to check if the file is locked.


    In that case fcntl will work for you. The point is that fcntl only
    locks the file if all of the applications accessing the file use
    fcntl. Any other application such as vim can simply ignore the fcntl
    lock. Have a read of the links that Grant posted.

    Did you try writing twice from the same application that uses fcntl?

    > Or maybe using sqlite would work even if writing from different
    > processes?


    That would also work.


    Oscar
    Oscar Benjamin, Oct 18, 2012
    #5
  6. 2012/10/18 Grant Edwards <>:
    >
    > If what you're guarding against is multiple instances of your
    > application modifying the file, then either of the advisory file
    > locking schemes or the separate lock file should work fine.
    >
    > --
    > Grant Edwards grant.b.edwards Yow! All this time I've
    > at been VIEWING a RUSSIAN
    > gmail.com MIDGET SODOMIZE a HOUSECAT!
    > --
    > http://mail.python.org/mailman/listinfo/python-list


    Ok so I tried a small example to see if I can make it fail, but this
    below just works perfectly fine.

    Maybe it's too fast and it release the file in time, but I would
    expect it to take some time and fail instead..

    import fcntl

    from multiprocessing import Process

    FILENAME = 'file.txt'


    def long_text():
    return ('some text' * (100 * 100))


    class Locked:
    def __init__(self, fileobj):
    self.fileobj = fileobj

    def __enter__(self):
    # any problems here?
    fcntl.lockf(self.fileobj, fcntl.LOCK_EX)
    return self.fileobj

    def __exit__(self, type, value, traceback):
    fcntl.lockf(self.fileobj, fcntl.LOCK_UN)


    def write_to_file():
    with open(FILENAME, 'w') as to_lock:
    with Locked(to_lock):
    to_lock.write(long_text())


    if __name__ == '__main__':
    Process(target=write_to_file).start()
    Process(target=write_to_file).start()
    andrea crotti, Oct 18, 2012
    #6
  7. On 18 October 2012 15:49, andrea crotti <> wrote:
    > 2012/10/18 Grant Edwards <>:
    >>
    >> If what you're guarding against is multiple instances of your
    >> application modifying the file, then either of the advisory file
    >> locking schemes or the separate lock file should work fine.

    >
    > Ok so I tried a small example to see if I can make it fail, but this
    > below just works perfectly fine.
    >
    > Maybe it's too fast and it release the file in time, but I would
    > expect it to take some time and fail instead..


    Why not come up with a test that actually shows you if it works? Here
    are two suggestions:

    1) Use time.sleep() so that you know how long the lock is held for.
    2) Write different data into the file from each process and see what
    you end up with.

    >
    > import fcntl
    >
    > from multiprocessing import Process
    >
    > FILENAME = 'file.txt'
    >
    >
    > def long_text():
    > return ('some text' * (100 * 100))
    >
    >
    > class Locked:
    > def __init__(self, fileobj):
    > self.fileobj = fileobj
    >
    > def __enter__(self):
    > # any problems here?
    > fcntl.lockf(self.fileobj, fcntl.LOCK_EX)
    > return self.fileobj
    >
    > def __exit__(self, type, value, traceback):
    > fcntl.lockf(self.fileobj, fcntl.LOCK_UN)
    >
    >
    > def write_to_file():
    > with open(FILENAME, 'w') as to_lock:


    I don't think it will work if you truncate the file like this. This
    will empty the file *before* checking for the lock. Try opening the
    file for reading and writing (without truncating).

    > with Locked(to_lock):
    > to_lock.write(long_text())
    >
    >
    > if __name__ == '__main__':
    > Process(target=write_to_file).start()
    > Process(target=write_to_file).start()



    Oscar
    Oscar Benjamin, Oct 18, 2012
    #7
  8. 2012/10/18 Oscar Benjamin <>:
    >
    > Why not come up with a test that actually shows you if it works? Here
    > are two suggestions:
    >
    > 1) Use time.sleep() so that you know how long the lock is held for.
    > 2) Write different data into the file from each process and see what
    > you end up with.
    >



    Ok thanks I will try, but I thought that what I did was the worst
    possible case, because I'm opening and writing on the same file from
    two different processes, locking the file with LOCK_EX.

    It should not open it at all as far as I understood...
    andrea crotti, Oct 18, 2012
    #8
  9. On 18 October 2012 16:08, andrea crotti <> wrote:
    > 2012/10/18 Oscar Benjamin <>:
    >>
    >> Why not come up with a test that actually shows you if it works? Here
    >> are two suggestions:
    >>
    >> 1) Use time.sleep() so that you know how long the lock is held for.
    >> 2) Write different data into the file from each process and see what
    >> you end up with.
    >>

    >
    >
    > Ok thanks I will try, but I thought that what I did was the worst
    > possible case, because I'm opening and writing on the same file from
    > two different processes, locking the file with LOCK_EX.
    >
    > It should not open it at all as far as I understood...


    I don't think you have understood. Read the link that Grant posted:
    http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems

    And my other comment:

    >> def write_to_file():
    >> with open(FILENAME, 'w') as to_lock:

    >
    > I don't think it will work if you truncate the file like this. This
    > will empty the file *before* checking for the lock. Try opening the
    > file for reading and writing (without truncating).


    The lock is cooperative. It does not prevent the file from being
    opened or overwritten. It only prevents any other process from
    obtaining the lock. Here you open the file with mode 'w' which
    truncates the file instantly (without checking for the lock).


    Oscar
    Oscar Benjamin, Oct 18, 2012
    #9
  10. 2012/10/18 Oscar Benjamin <>:
    >
    > The lock is cooperative. It does not prevent the file from being
    > opened or overwritten. It only prevents any other process from
    > obtaining the lock. Here you open the file with mode 'w' which
    > truncates the file instantly (without checking for the lock).
    >
    >
    > Oscar



    Very good thanks now I understood, actually my problem was in the
    assumption that it should fail when the lock is already taken, but by
    default lockf just blocks until the lock is released.

    It seems to work quite nicely so I'm going to use this..
    andrea crotti, Oct 19, 2012
    #10
  11. andrea crotti

    Nobody Guest

    On Thu, 18 Oct 2012 14:44:27 +0100, andrea crotti wrote:

    > Uhh I see thanks, I guess I'll use the good-old .lock file (even if it
    > might have some problems too).


    In which case, you don't see. A lock file is also advisory, i.e. it only
    affects applications which explicitly check for a lock file.

    Historically, the advantage of lock files was that they worked on NFS
    implementations which didn't implement locking (it's a long-standing Unix
    joke that "NFS" stands for "Not a FileSystem", because it failed to
    conform to established filesystem semantics).

    Nowadays, NFS implementations which don't support locking are sufficiently
    rare that they can safely be ignored. So lock files don't offer any
    advantages, and one fairly obvious disadvantage (the possibility of a
    "stale" lock file if the program terminates unexpectedly without removing
    the lock file).

    For any form of advisory locking, the one thing which matters is that all
    progams which access the file agree on the mechanism used, i.e. whether to
    use lockf(), fcntl(), flock() (locks created by one mechanism may or may
    not be recognised by the others), or lock files, and in the case of lock
    files, the naming convention which is used.

    If the file is specific to a particular program, and you just need to
    protect against multiple instances of that program, you can use whichever
    mechanism you wish, and would be strongly advised to use kernel locks
    (fcntl() is the most portable, followed by lockf(); flock() is a BSD-ism).
    Nobody, Oct 19, 2012
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Christopher Campbell

    Ghostscript locking files in ASP.NET

    Christopher Campbell, Oct 15, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    2,079
    Jacob Yang [MSFT]
    Oct 16, 2003
  2. rao
    Replies:
    3
    Views:
    762
    drippy
    Dec 16, 2004
  3. z. f.
    Replies:
    15
    Views:
    4,525
    z. f.
    Dec 2, 2004
  4. Timasmith
    Replies:
    4
    Views:
    442
    Bjorn Borud
    Nov 1, 2006
  5. Replies:
    3
    Views:
    1,429
    Rolf Magnus
    Jan 18, 2009
Loading...

Share This Page