Multiple independently started python processes and sharing of amodule

Discussion in 'Python' started by Martin P. Hellwig, Jan 14, 2011.

  1. Hi all,

    I have the following problem (which I already have a hacked around
    solution that works but I'd would like some more input on it):

    I have a situation where multiple python processes are started
    independently from each other but by the same user with the same
    environment (as happens with mod_wsgi, when not using daemon mode).

    All of these processes access a single module which needs
    synchronization for some of the commands, for example a db (MySQLdb)
    module where when a select is done, the fetchall must be done of that
    same process before another process can do anything else.

    How would I go and provide synchronization?
    Locking does not seem to work because there is no relationship between
    all the python processes except that they are started by the same user.

    Currently my solution is to wrap the module around a module that when
    used creates a directory and pipes to the process
    (multiprocessing.Connection) thus enforcing single access and within
    that I have wrapped the db function around again so that select
    statement as mentioned above is actually an execute followed by a fetchall.

    I still have the nagging feeling that I have reinvented a squared wheel
    or am totally missing the point.

    Any suggestions/comments are greatly appreciated,

    Thanks in advanced,

    Martin P. Hellwig
    Martin P. Hellwig, Jan 14, 2011
    #1
    1. Advertising

  2. Re: Multiple independently started python processes and sharing ofa module

    ----- Original message -----
    > Hi all,
    >
    > I have the following problem (which I already have a hacked around
    > solution that works but I'd would like some more input on it):
    >
    > I have a situation where multiple python processes are started
    > independently from each other but by the same user with the same
    > environment (as happens with mod_wsgi, when not using daemon mode).
    >
    > All of these processes access a single module which needs
    > synchronization for some of the commands, for example a db (MySQLdb)
    > module where when a select is done, the fetchall must be done of that
    > same process before another process can do anything else.
    >


    If the processes are independent, they are not sharing the database connection, unless you've taken steps to make it so. MySQLdb imported in one process should not interfere with MySQLdb importerd in another process.

    > <snip>


    --
    regards,
    kushal
    Kushal Kumaran, Jan 14, 2011
    #2
    1. Advertising

  3. Re: Multiple independently started python processes and sharing ofa module

    On 01/14/11 03:04, Kushal Kumaran wrote:
    > ----- Original message -----
    >> Hi all,
    >>
    >> I have the following problem (which I already have a hacked around
    >> solution that works but I'd would like some more input on it):
    >>
    >> I have a situation where multiple python processes are started
    >> independently from each other but by the same user with the same
    >> environment (as happens with mod_wsgi, when not using daemon mode).
    >>
    >> All of these processes access a single module which needs
    >> synchronization for some of the commands, for example a db (MySQLdb)
    >> module where when a select is done, the fetchall must be done of that
    >> same process before another process can do anything else.
    >>

    >
    > If the processes are independent, they are not sharing the database connection, unless you've taken steps to make it so. MySQLdb imported in one process should not interfere with MySQLdb importerd in another process.
    >
    >> <snip>

    >

    It might be a misconfiguration but, under mod_wsgi with apache it does.

    Cheers,

    Martin
    Martin P. Hellwig, Jan 14, 2011
    #3
  4. Re: Multiple independently started python processes and sharing ofa module

    On 01/14/11 10:05, Kushal Kumaran wrote:
    <cut>
    > This might help though:
    > https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
    >
    > It seems if you're not using 'daemon' mode, global data might be shared.
    >

    Yes I read that thoroughly before I started out implementing a solution.
    But in my case I wanted something that worked as expected and not be
    depending on specific configuration of underlying technology as I can
    not assure that these condition will be met.

    > You could create new connections for each request (and close them when
    > done). There won't be interference between select/fetch across
    > multiple database connections. Additionally, the documentation of
    > MySQLdb says it is a bad idea to share database connections between
    > threads.
    >


    That is a possible solution too, however the performance impact is in
    the range of 40% while doing forced synchronization and overhead of the
    singleton wrapper is around 20%. So the latter is what I have gone with.

    Thanks for bouncing off ideas though, much appreciated.

    --
    mph
    Martin P. Hellwig, Jan 14, 2011
    #4
  5. Martin P. Hellwig

    Aahz Guest

    In article <igo5a0$r48$-september.org>,
    Martin P. Hellwig <> wrote:
    >
    >Currently my solution is to wrap the module around a module that when
    >used creates a directory and pipes to the process
    >(multiprocessing.Connection) thus enforcing single access and within
    >that I have wrapped the db function around again so that select
    >statement as mentioned above is actually an execute followed by a fetchall.
    >
    >I still have the nagging feeling that I have reinvented a squared wheel
    >or am totally missing the point.


    What you want is a worker pool or connection pool. I.e. you keep around
    multiple open connections and assign them per request.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Programming language design is not a rational science. Most reasoning
    about it is at best rationalization of gut feelings, and at worst plain
    wrong." --GvR, python-ideas, 2009-03-01
    Aahz, Feb 6, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    383
  2. Replies:
    0
    Views:
    343
  3. Travis
    Replies:
    0
    Views:
    270
    Travis
    Feb 12, 2009
  4. Dennis Nedry
    Replies:
    5
    Views:
    150
    Dennis Nedry
    May 31, 2010
  5. Jake Barnes
    Replies:
    2
    Views:
    307
Loading...

Share This Page