does python have a generic object pool like commons-pool in Java

R

Rick Lawson

Appreciate any help on this. I am porting an app from Java to python
and need generic object pooling with hooks for object initialization /
cleanup and be able to specify an object timeout.

Thanks !
Rick
 
J

Jonathan Gardner

Appreciate any help on this. I am porting an app from Java to python
and need generic object pooling with hooks for object initialization /
cleanup and be able to specify an object timeout.

Are you looking for something like a thread pool or a connection pool?
Such a thing is easy to code in Python. Might as well write one from
scratch for your particular need.

By the way, a tip for writing Python: Forget Java. If you're porting
an app, consider rewriting it from the architecture on up. You'll save
yourself time and get a better result.
 
R

Rick Lawson

Are you looking for something like a thread pool or a connection pool?
Such a thing is easy to code in Python. Might as well write one from
scratch for your particular need.

By the way, a tip for writing Python: Forget Java. If you're porting
an app, consider rewriting it from the architecture on up. You'll save
yourself time and get a better result.

Jonathan,

Thanks for the advice but would like to pick your brain a little more.
The pool I need is not a thread pool or db pool. I need to pool
objects which are a proxy for an external C process that communicates
via sockets. The pool is necessary because the initial connection/
authentication is expensive, plus the connection needs to be recycled
every so often due to instability on the C side. The existing Java
code uses commons-pool which incidentally is the base object pool for
a db connection pooling library (DBCP). Anyway, maybe this is just so
easy to do in Python that everyone rolls their own - but I have done a
lot of googling with no luck.

I just hate to re-invent the wheel here - I guess I should look at a
db connection pooling library and see if they have a generic object
pool somewhere in there. Any you can recommend that are more or less
standalone db connection pool libraries ?

Thanks,
Rick
 
D

Diez B. Roggisch

Rick said:
Jonathan,

Thanks for the advice but would like to pick your brain a little more.
The pool I need is not a thread pool or db pool. I need to pool
objects which are a proxy for an external C process that communicates
via sockets. The pool is necessary because the initial connection/
authentication is expensive, plus the connection needs to be recycled
every so often due to instability on the C side. The existing Java
code uses commons-pool which incidentally is the base object pool for
a db connection pooling library (DBCP). Anyway, maybe this is just so
easy to do in Python that everyone rolls their own - but I have done a
lot of googling with no luck.

I just hate to re-invent the wheel here - I guess I should look at a
db connection pooling library and see if they have a generic object
pool somewhere in there. Any you can recommend that are more or less
standalone db connection pool libraries ?

I don't know enough about your domain, e.g. if your pooled objects have
state or not.

Assuming that

- they are objects
- they don't have state (like database connections)
- you want to recycle them after so many calls

one of the simplest solutions would be something like this:

class AutoDestructingWrapper(object):

MAX_CALL_COUNT = 1000

def __init__(self, object_factory):
self._object_factory = object_factory
self._call_count = self.MAX_CALL_COUNT


def __getattr__(self, name):
if self._call_count >= self.MAX_CALL_COUNT:
self._delegate = self._object_factory()
return getattr(self._delegate, name)


Diez
 
D

Diez B. Roggisch

Diez said:
I don't know enough about your domain, e.g. if your pooled objects have
state or not.

Assuming that

- they are objects
- they don't have state (like database connections)
- you want to recycle them after so many calls

one of the simplest solutions would be something like this:

class AutoDestructingWrapper(object):

MAX_CALL_COUNT = 1000

def __init__(self, object_factory):
self._object_factory = object_factory
self._call_count = self.MAX_CALL_COUNT


This must be

def __getattr__(self, name):
self._call_count += 1
if self._call_count >= self.MAX_CALL_COUNT:
self._delegate = self._object_factory()
self._call_count = 0
return getattr(self._delegate, name)

of course.

Diez
 
J

John Nagle

Rick said:
Jonathan,

Thanks for the advice but would like to pick your brain a little more.
The pool I need is not a thread pool or db pool. I need to pool
objects which are a proxy for an external C process that communicates
via sockets.

"fcgi" is an option for this sort of thing. With "mod_fcgi" installed
in Apache, and "fcgi.py" used to manage the Python side of the problem, you
can have semi-persistent programs started up and shut down for you on the
server side.

I use this for "sitetruth.com". It's useful when CGI is too slow
because the cost of launching Python and loading all the modules is
far greater than the cost of processing the request.

Once you find and install a working Apache "mod_fcgi" module, and a
working "fcgi.py" module (I suggest "http://svn.saddi.com/py-lib/trunk/fcgi.py")
writing the server side app is no harder than with CGI. But the performance
is far better.

John Nagle
 
A

alex23

     "fcgi" is an option for this sort of thing.  With "mod_fcgi" installed
in Apache, and "fcgi.py" used to manage the Python side of the problem, you
can have semi-persistent programs started up and shut down for you on the
server side.

Hey John,

The environments in which I've been asked to develop webs apps using
Python have all utilised mod_wsgi. Do you have any experience with
mod_wsgi vs mod_fcgi, and if so, can you comment on any relevant
performance / capability / ease-of-use differences?

Cheers,
alex23
 
J

John Nagle

alex23 said:
Hey John,

The environments in which I've been asked to develop webs apps using
Python have all utilised mod_wsgi. Do you have any experience with
mod_wsgi vs mod_fcgi, and if so, can you comment on any relevant
performance / capability / ease-of-use differences?

Cheers,
alex23

FCGI seems to be somewhat out of favor, perhaps because it isn't
complicated enough. It's a mature technology and works reasonably
well. It's been a puzzle to me that FCGI was taken out of the
main Apache code base, because it gives production-level performance
with CGI-type simplicity.

WSGI has a mode for running Python inside the Apache process,
which is less secure and doesn't allow multiple Python processes.
That complicates mod_wsgi considerably, and ties it very closely
to specific versions of Python and Python modules. As a result,
the WSGI developers are patching at a great rate. I think the
thing has too many "l33t features".

I'd avoid "embedded mode". "Daemon mode" is the way to go if you use WSGI.
I haven't tried WSGI; I don't need the grief of a package under
heavy development.

John Nagle
 
G

Graham Dumpleton

    FCGI seems to be somewhat out of favor, perhaps because it isn't
complicated enough.

I doubt that is the reason. It is out of favour for Python web hosting
at least, because web hosting companies provide really crappy support
for using Python with it and also don't like long lived processes.
Most FASTCGI deployments are configured in a way more appropriate for
PHP because that is what the default settings favour. This works to
the detriment of Python. Even when people setup FASTCGI themselves,
they still don't play with the configuration to optimise it for their
specific Python application. Thus it can still run poorly.
It's a mature technology and works reasonably well.

But is showing its age. It really needs a refresh. Various of its
design decisions were from when network speeds and bandwidth were much
lower and this complicates the design. Making the protocol format
simpler in some areas would make it easier to implement. The protocol
could also be built on to make process management more flexible,
rather than relying on each implementation to come up with adhoc ways
of managing it.

Part of the problem with FASTCGI, or hosting of dynamic applications
in general, is that web servers primary focus is serving of static
files. Any support for dynamic web applications is more of a bolt on
feature. Thus, web applications using Python will always be at a
disadvantage. PHP manages okay because their model of operation is
closer to the one shot processing of static files. Python requires
persistent to work adequately with modern fat applications.
It's been a puzzle to me that FCGI was taken out of the
main Apache code base, because it gives production-level performance
with CGI-type simplicity.

As far as I know FASTCGI support has in the past never been a part of
the Apache code base. Both mod_fastcgi and mod_fcgid were developed by
independent people and not under the ASF.

In Apache 2.3 development versions (to become 2.4 when released),
there will however be a mod_proxy_fcgi. The ASF has also taken over
management of mod_fcgid and working out how that may be incorporated
into future Apache versions.
    WSGI has a mode for running Python inside the Apache process,
which is less secure and doesn't allow multiple Python processes.

Depending on the requirements it is more than adequate and if
configured correctly gives better performance and scalability. Not
everyone runs in a shared hosting environment. The different modes of
mod_wsgi therefore give choice.
That complicates mod_wsgi considerably,

No it doesn't. It is actual daemon mode that complicates things, not
embedded mode.
and ties it very closely
to specific versions of Python and Python modules.  As a result,
the WSGI developers are patching at a great rate.

What are you talking about when you say 'As a result, the WSGI
developers are patching at a great rate'? As with some of your other
comments, this is leaning towards being FUD and nothing more. There is
no 'patching at a great rate' going on.
 I think the thing has too many "l33t features".

It is all about choice and providing flexibility. You may not see a
need for certain features, but it doesn't mean other people don't have
a need for it.
I'd avoid "embedded mode".  "Daemon mode" is the way to go if you use WSGI.

I'd agree, but not sure you really understand the reasons for why it
would be preferred.
I haven't tried WSGI;

Then why comment on it as if you are knowledgeable on it.
I don't need the grief of a package under heavy development.

More FUD.

Graham
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,731
Messages
2,569,432
Members
44,835
Latest member
KetoRushACVBuy

Latest Threads

Top