Re: How to safely maintain a status file

Discussion in 'Python' started by John Nagle, Jul 9, 2012.

  1. John Nagle

    John Nagle Guest

    On 7/8/2012 2:52 PM, Christian Heimes wrote:
    > You are contradicting yourself. Either the OS is providing a fully
    > atomic rename or it doesn't. All POSIX compatible OS provide an atomic
    > rename functionality that renames the file atomically or fails without
    > loosing the target side. On POSIX OS it doesn't matter if the target exists.


    Rename on some file system types (particularly NFS) may not be atomic.
    >
    > You don't need locks or any other fancy stuff. You just need to make
    > sure that you flush the data and metadata correctly to the disk and
    > force a re-write of the directory inode, too. It's a standard pattern on
    > POSIX platforms and well documented in e.g. the maildir RFC.
    >
    > You can use the same pattern on Windows but it doesn't work as good.


    That's because you're using the wrong approach. See how to use
    ReplaceFile under Win32:

    http://msdn.microsoft.com/en-us/library/aa365512(VS.85).aspx

    Renaming files is the wrong way to synchronize a
    crawler. Use a database that has ACID properties, such as
    SQLite. Far fewer I/O operations are required for small updates.
    It's not the 1980s any more.

    I use a MySQL database to synchronize multiple processes
    which crawl web sites. The tables of past activity are InnoDB
    tables, which support transactions. The table of what's going
    on right now is a MEMORY table. If the database crashes, the
    past activity is recovered cleanly, the MEMORY table comes back
    empty, and all the crawler processes lose their database
    connections, abort, and are restarted. This allows multiple
    servers to coordinate through one database.

    John Nagle
     
    John Nagle, Jul 9, 2012
    #1
    1. Advertising

  2. John Nagle

    alex23 Guest

    On Jul 10, 6:24 am, John Nagle <> wrote:
    > That's because you're using the wrong approach. See how to use
    > ReplaceFile under Win32:
    >
    > http://msdn.microsoft.com/en-us/library/aa365512(VS.85).aspx


    I'm not convinced ReplaceFile is atomic:

    "The ReplaceFile function combines several steps within a single
    function. An application can call ReplaceFile instead of calling
    separate functions to save the data to a new file, rename the original
    file using a temporary name, rename the new file to have the same name
    as the original file, and delete the original file."

    About the best you can get in Windows, I think, is MoveFileTransacted,
    but you need to be running Vista or later:

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa365241(v=vs.85)..aspx

    I agree with your suggestion of using something transactional that
    isn't bound to later Window versions, though.
     
    alex23, Jul 10, 2012
    #2
    1. Advertising

  3. John Nagle

    Laszlo Nagy Guest


    > Renaming files is the wrong way to synchronize a
    > crawler. Use a database that has ACID properties, such as
    > SQLite. Far fewer I/O operations are required for small updates.
    > It's not the 1980s any more.

    I agree with this approach. However, the OP specifically asked about
    "how to update status file".
     
    Laszlo Nagy, Jul 12, 2012
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Richard Baron Penman

    How to safely maintain a status file

    Richard Baron Penman, Jul 8, 2012, in forum: Python
    Replies:
    0
    Views:
    206
    Richard Baron Penman
    Jul 8, 2012
  2. Dennis Lee Bieber

    Re: How to safely maintain a status file

    Dennis Lee Bieber, Jul 8, 2012, in forum: Python
    Replies:
    1
    Views:
    192
    Plumo
    Jul 9, 2012
  3. Laszlo Nagy

    Re: How to safely maintain a status file

    Laszlo Nagy, Jul 8, 2012, in forum: Python
    Replies:
    1
    Views:
    238
    Nobody
    Jul 9, 2012
  4. Plumo
    Replies:
    1
    Views:
    219
    Michael Hrivnak
    Jul 9, 2012
  5. Plumo
    Replies:
    1
    Views:
    218
    Laszlo Nagy
    Jul 12, 2012
Loading...

Share This Page