Re: write to the same file from multiple processes at the same time?

Discussion in 'Python' started by Jp Calderone, May 27, 2005.

  1. Jp Calderone

    Jp Calderone Guest

    On 27 May 2005 06:21:21 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote:
    >Peter Hansen <> writes:
    >> The OP was probably on the right track when he suggested that things
    >> like SQLite (conveniently wrapped with PySQLite) had already solved
    >> this problem.

    >
    >But they haven't. They depend on messy things like server processes
    >constantly running, which goes against the idea of a cgi that only
    >runs when someone calls it.


    SQLite is an in-process dbm.

    Jp
     
    Jp Calderone, May 27, 2005
    #1
    1. Advertising

  2. Jp Calderone

    Paul Rubin Guest

    Jp Calderone <> writes:
    > >But they haven't. They depend on messy things like server processes
    > >constantly running, which goes against the idea of a cgi that only
    > >runs when someone calls it.

    >
    > SQLite is an in-process dbm.


    http://www.sqlite.org/faq.html#q7

    (7) Can multiple applications or multiple instances of the same
    application access a single database file at the same time?

    Multiple processes can have the same database open at the same
    time. Multiple processes can be doing a SELECT at the same
    time. But only one process can be making changes to the database
    at once.

    But multiple processes changing the database simultaneously is
    precisely what the OP wants to do.
     
    Paul Rubin, May 27, 2005
    #2
    1. Advertising

  3. On Fri, 27 May 2005 14:17:58 GMT, Jp Calderone <>
    declaimed the following in comp.lang.python:

    Not a Python solution -- my apologies...

    > That said, I think an RDBM is a ridiculously complex solution to this simple problem. A filesystem lock, preferably using the directory or symlink trick (but flock() is fun too, if you're into that sort of thing), is clearly the solution to go with here.
    >

    About 15 years ago, on my Amiga, I'd be using ARexx and message
    ports.

    The processes wouldn't be opening a log file. They'd be on the
    nature of (psuedoARexx):


    if ~show("P", "Logger") then
    do
    say "Logger process is not available"
    /* should add code to start logger */
    exit
    end

    /* do work */

    address "Logger" "This is a message to be Logged"

    /* do work */

    -----
    The logger process is a bit more complex

    /* make sure support library is loaded */
    if ~show("L", "rexxsupport.library") then
    do
    if ~addlib("rexxsupport.library", 0, -30, 0) then
    do
    say "Unable to load library"
    exit
    end
    end

    if ~openport("Logger") then
    do
    say "Unable to create logger port -- is it already running?"
    exit
    end

    call open("logfile", "path/to/log/file", "APPEND")

    do while waitpkt("Logger")
    pkt = getpkt("Logger")
    cmd = getarg(pkt) /* depending on invocation, may have */
    /* multiple arguments: getarg(pkt, #) */
    call reply(pkt, 0) /* 0 = return code, no error */
    call writeln("logfile", cmd)
    end


    Believe it or not, the above /would/ be the simple method for
    the Amiga OS <G> Message ports were kernel ("exec") level entities
    implemented as queues, and write/read were protected operations. The
    ARexx "address" command used them with ARexx specific overhead padded
    into the message. The rexxsupport library added functions to let ARexx
    scripts create named ARexx compatible message ports and retrieve
    messages from them.

    Nearest match for message ports, that I've seen, is the VMS
    "mailbox" -- though mailboxes can be multiple readers, and have their
    own buffers (The Amiga did not have process protected memory, so the
    message ports really just had the memory address of the message buffer
    itself -- the reply(message) is needed to let the process know when the
    buffer is available for reuse).

    I don't know if WinNT "named pipes" can be used in a similar
    scheme; I do know W9x doesn't support creating "named pipes" -- one
    would be down to writing TCP/IP socket operations to emulate the
    ability... Much too complex for simple clients -- note how the clients,
    above, aren't using /any/ low-level operations, just regular ARexx
    statements ("address" is part of REXX). Only the logger itself needed
    low-level support. [Non-ARexx programs could link the ARexx system to
    gain call level access to the ports -- heck, as I recall, Irmen had
    added ARexx support to the old AmigaPython port]

    --
    > ============================================================== <
    > | Wulfraed Dennis Lee Bieber KD6MOG <
    > | Bestiaria Support Staff <
    > ============================================================== <
    > Home Page: <http://www.dm.net/~wulfraed/> <
    > Overflow Page: <http://wlfraed.home.netcom.com/> <
     
    Dennis Lee Bieber, May 27, 2005
    #3
  4. Jp Calderone

    Peter Hansen Guest

    Paul Rubin wrote:
    > http://www.sqlite.org/faq.html#q7

    [snip]
    > Multiple processes can have the same database open at the same
    > time. Multiple processes can be doing a SELECT at the same
    > time. But only one process can be making changes to the database
    > at once.
    >
    > But multiple processes changing the database simultaneously is
    > precisely what the OP wants to do.


    What isn't described in the above quote from the FAQ is how SQLite
    *protects* your data from corruption in this case, unlike the "raw"
    approach where you just use file handles.

    And PySQLite conveniently wraps the relevant calls with retries when the
    database is "locked" by the writing process, making it roughly a
    no-brainer to use SQLite databases as nice simple log files where you're
    trying to write from multiple CGI processes like the OP wanted.

    Disclaimer: I haven't actually done that myself, and have only started
    playing with pysqlite2 a day ago, but I have spent a fair bit of time
    experimenting and reading the relevant docs and I believe I've got this
    all correct.

    -Peter
     
    Peter Hansen, May 27, 2005
    #4
  5. Jp Calderone

    Paul Rubin Guest

    Peter Hansen <> writes:
    > And PySQLite conveniently wraps the relevant calls with retries when
    > the database is "locked" by the writing process, making it roughly a
    > no-brainer to use SQLite databases as nice simple log files where
    > you're trying to write from multiple CGI processes like the OP wanted.


    Oh, ok. But what kind of locks does it use?
     
    Paul Rubin, May 27, 2005
    #5
  6. Jp Calderone

    Peter Hansen Guest

    Paul Rubin wrote:
    > Peter Hansen <> writes:
    >
    >>And PySQLite conveniently wraps the relevant calls with retries when
    >>the database is "locked" by the writing process, making it roughly a
    >>no-brainer to use SQLite databases as nice simple log files where
    >>you're trying to write from multiple CGI processes like the OP wanted.

    >
    > Oh, ok. But what kind of locks does it use?


    I think the FAQ can answer that better than I can, since I'm not sure
    whether you're asking about any low-level (OS) locks it might use or
    higher-level (e.g. database-level locking) that it might use. In
    summary, however, at the database level it provides only coarse-grained
    locking on the entire database. It *is* supposed to be a relatively
    simple/lightweight solution compared to typical RDBMSes...

    (There's also an excrutiating level of detail about this whole area in
    the page at http://www.sqlite.org/lockingv3.html ).

    -Peter
     
    Peter Hansen, May 27, 2005
    #6
  7. Jp Calderone

    Paul Rubin Guest

    Peter Hansen <> writes:
    > I think the FAQ can answer that better than I can, since I'm not sure
    > whether you're asking about any low-level (OS) locks it might use or
    > higher-level (e.g. database-level locking) that it might use. In
    > summary, however, at the database level it provides only
    > coarse-grained locking on the entire database. It *is* supposed to be
    > a relatively simple/lightweight solution compared to typical RDBMSes...


    Compared to what the OP was asking for, which was a way to synchronize
    appending to a serial log file, SQlite is very complex. It's also
    much more complex than (say) the dbm module, which is what Python apps
    normally use as a lightweight db.

    > (There's also an excrutiating level of detail about this whole area in
    > the page at http://www.sqlite.org/lockingv3.html ).


    Oh ok, it says it uses some special locking system calls on Windows.
    Since those calls aren't in the Python stdlib, it must be using C
    extensions, which again means complexity. But it looks like the
    built-in msvcrt module has ways to lock parts of files in Windows.

    Really, I think the Python library is somewhat lacking in not
    providing a simple, unified interface for doing stuff like this.
     
    Paul Rubin, May 27, 2005
    #7
  8. Jp Calderone

    Mike Meyer Guest

    Re: write to the same file from multiple processes at the sametime?

    Paul Rubin <http://> writes:
    > Really, I think the Python library is somewhat lacking in not
    > providing a simple, unified interface for doing stuff like this.


    It's got one. Well, three, actually.

    The syslog module solves the problem quite nicely, but only works on
    Unix. If the OP is working on Unix systems, that may be a good
    solution.

    The logging module has a SysLogHandler that talks to syslog on
    Unix. It also has an NTEventLogHandler for use on NT. I'm not familiar
    with NT's event log, but I presume it has the same kind of
    functionality as Unix's syslog facility.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, May 28, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. darrel
    Replies:
    2
    Views:
    472
    darrel
    Apr 5, 2006
  2. Smegly
    Replies:
    1
    Views:
    1,105
    Mitchell
    May 19, 2004
  3. gabor
    Replies:
    19
    Views:
    1,132
    Steve Holden
    May 31, 2005
  4. Jp Calderone
    Replies:
    0
    Views:
    463
    Jp Calderone
    May 27, 2005
  5. Jp Calderone
    Replies:
    4
    Views:
    453
    gabor
    May 31, 2005
Loading...

Share This Page