Multiple independently started python processes and sharing of amodule

[Martin P. Hellwig]

Hi all,

I have the following problem (which I already have a hacked around
solution that works but I'd would like some more input on it):

I have a situation where multiple python processes are started
independently from each other but by the same user with the same
environment (as happens with mod_wsgi, when not using daemon mode).

All of these processes access a single module which needs
synchronization for some of the commands, for example a db (MySQLdb)
module where when a select is done, the fetchall must be done of that
same process before another process can do anything else.

How would I go and provide synchronization?
Locking does not seem to work because there is no relationship between
all the python processes except that they are started by the same user.

Currently my solution is to wrap the module around a module that when
used creates a directory and pipes to the process
(multiprocessing.Connection) thus enforcing single access and within
that I have wrapped the db function around again so that select
statement as mentioned above is actually an execute followed by a fetchall.

I still have the nagging feeling that I have reinvented a squared wheel
or am totally missing the point.

Any suggestions/comments are greatly appreciated,

Thanks in advanced,

Martin P. Hellwig

 

[Kushal Kumaran]

----- Original message -----

Hi all,

I have the following problem (which I already have a hacked around
solution that works but I'd would like some more input on it):

I have a situation where multiple python processes are started
independently from each other but by the same user with the same
environment (as happens with mod_wsgi, when not using daemon mode).

All of these processes access a single module which needs
synchronization for some of the commands, for example a db (MySQLdb)
module where when a select is done, the fetchall must be done of that
same process before another process can do anything else.

If the processes are independent, they are not sharing the database connection, unless you've taken steps to make it so. MySQLdb imported in one process should not interfere with MySQLdb importerd in another process.

 

[Martin P. Hellwig]

----- Original message -----

If the processes are independent, they are not sharing the database connection, unless you've taken steps to make it so. MySQLdb imported in one process should not interfere with MySQLdb importerd in another process.

It might be a misconfiguration but, under mod_wsgi with apache it does.

Cheers,

Martin

 

[Martin P. Hellwig]

On 01/14/11 10:05, Kushal Kumaran wrote:

This might help though:
https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading

It seems if you're not using 'daemon' mode, global data might be shared.

Yes I read that thoroughly before I started out implementing a solution.
But in my case I wanted something that worked as expected and not be
depending on specific configuration of underlying technology as I can
not assure that these condition will be met.

You could create new connections for each request (and close them when
done). There won't be interference between select/fetch across
multiple database connections. Additionally, the documentation of
MySQLdb says it is a bad idea to share database connections between
threads.

That is a possible solution too, however the performance impact is in
the range of 40% while doing forced synchronization and overhead of the
singleton wrapper is around 20%. So the latter is what I have gone with.

Thanks for bouncing off ideas though, much appreciated.

 

[Aahz]

Currently my solution is to wrap the module around a module that when
used creates a directory and pipes to the process
(multiprocessing.Connection) thus enforcing single access and within
that I have wrapped the db function around again so that select
statement as mentioned above is actually an execute followed by a fetchall.

I still have the nagging feeling that I have reinvented a squared wheel
or am totally missing the point.

What you want is a worker pool or connection pool. I.e. you keep around
multiple open connections and assign them per request.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Programming language design is not a rational science. Most reasoning
about it is at best rationalization of gut feelings, and at worst plain
wrong." --GvR, python-ideas, 2009-03-01