does python have a generic object pool like commons-pool in Java

Discussion in 'Python' started by Rick Lawson, Jul 15, 2009.

  1. Rick Lawson

    Rick Lawson Guest

    Appreciate any help on this. I am porting an app from Java to python
    and need generic object pooling with hooks for object initialization /
    cleanup and be able to specify an object timeout.

    Thanks !
    Rick
    Rick Lawson, Jul 15, 2009
    #1
    1. Advertising

  2. On Jul 14, 6:34 pm, Rick Lawson <> wrote:
    > Appreciate any help on this. I am porting an app from Java to python
    > and need generic object pooling with hooks for object initialization /
    > cleanup and be able to specify an object timeout.
    >


    Are you looking for something like a thread pool or a connection pool?
    Such a thing is easy to code in Python. Might as well write one from
    scratch for your particular need.

    By the way, a tip for writing Python: Forget Java. If you're porting
    an app, consider rewriting it from the architecture on up. You'll save
    yourself time and get a better result.
    Jonathan Gardner, Jul 15, 2009
    #2
    1. Advertising

  3. Rick Lawson

    Rick Lawson Guest

    On Jul 15, 4:53 am, Jonathan Gardner <>
    wrote:
    > On Jul 14, 6:34 pm, Rick Lawson <> wrote:
    >
    > > Appreciate any help on this. I am porting an app from Java to python
    > > and need generic object pooling with hooks for object initialization /
    > > cleanup and be able to specify an object timeout.

    >
    > Are you looking for something like a thread pool or a connection pool?
    > Such a thing is easy to code in Python. Might as well write one from
    > scratch for your particular need.
    >
    > By the way, a tip for writing Python: Forget Java. If you're porting
    > an app, consider rewriting it from the architecture on up. You'll save
    > yourself time and get a better result.


    Jonathan,

    Thanks for the advice but would like to pick your brain a little more.
    The pool I need is not a thread pool or db pool. I need to pool
    objects which are a proxy for an external C process that communicates
    via sockets. The pool is necessary because the initial connection/
    authentication is expensive, plus the connection needs to be recycled
    every so often due to instability on the C side. The existing Java
    code uses commons-pool which incidentally is the base object pool for
    a db connection pooling library (DBCP). Anyway, maybe this is just so
    easy to do in Python that everyone rolls their own - but I have done a
    lot of googling with no luck.

    I just hate to re-invent the wheel here - I guess I should look at a
    db connection pooling library and see if they have a generic object
    pool somewhere in there. Any you can recommend that are more or less
    standalone db connection pool libraries ?

    Thanks,
    Rick
    Rick Lawson, Jul 15, 2009
    #3
  4. Rick Lawson wrote:

    > On Jul 15, 4:53 am, Jonathan Gardner <>
    > wrote:
    >> On Jul 14, 6:34 pm, Rick Lawson <> wrote:
    >>
    >> > Appreciate any help on this. I am porting an app from Java to python
    >> > and need generic object pooling with hooks for object initialization /
    >> > cleanup and be able to specify an object timeout.

    >>
    >> Are you looking for something like a thread pool or a connection pool?
    >> Such a thing is easy to code in Python. Might as well write one from
    >> scratch for your particular need.
    >>
    >> By the way, a tip for writing Python: Forget Java. If you're porting
    >> an app, consider rewriting it from the architecture on up. You'll save
    >> yourself time and get a better result.

    >
    > Jonathan,
    >
    > Thanks for the advice but would like to pick your brain a little more.
    > The pool I need is not a thread pool or db pool. I need to pool
    > objects which are a proxy for an external C process that communicates
    > via sockets. The pool is necessary because the initial connection/
    > authentication is expensive, plus the connection needs to be recycled
    > every so often due to instability on the C side. The existing Java
    > code uses commons-pool which incidentally is the base object pool for
    > a db connection pooling library (DBCP). Anyway, maybe this is just so
    > easy to do in Python that everyone rolls their own - but I have done a
    > lot of googling with no luck.
    >
    > I just hate to re-invent the wheel here - I guess I should look at a
    > db connection pooling library and see if they have a generic object
    > pool somewhere in there. Any you can recommend that are more or less
    > standalone db connection pool libraries ?


    I don't know enough about your domain, e.g. if your pooled objects have
    state or not.

    Assuming that

    - they are objects
    - they don't have state (like database connections)
    - you want to recycle them after so many calls

    one of the simplest solutions would be something like this:

    class AutoDestructingWrapper(object):

    MAX_CALL_COUNT = 1000

    def __init__(self, object_factory):
    self._object_factory = object_factory
    self._call_count = self.MAX_CALL_COUNT


    def __getattr__(self, name):
    if self._call_count >= self.MAX_CALL_COUNT:
    self._delegate = self._object_factory()
    return getattr(self._delegate, name)


    Diez
    Diez B. Roggisch, Jul 15, 2009
    #4
  5. Diez B. Roggisch wrote:

    > Rick Lawson wrote:
    >
    >> On Jul 15, 4:53 am, Jonathan Gardner <>
    >> wrote:
    >>> On Jul 14, 6:34 pm, Rick Lawson <> wrote:
    >>>
    >>> > Appreciate any help on this. I am porting an app from Java to python
    >>> > and need generic object pooling with hooks for object initialization /
    >>> > cleanup and be able to specify an object timeout.
    >>>
    >>> Are you looking for something like a thread pool or a connection pool?
    >>> Such a thing is easy to code in Python. Might as well write one from
    >>> scratch for your particular need.
    >>>
    >>> By the way, a tip for writing Python: Forget Java. If you're porting
    >>> an app, consider rewriting it from the architecture on up. You'll save
    >>> yourself time and get a better result.

    >>
    >> Jonathan,
    >>
    >> Thanks for the advice but would like to pick your brain a little more.
    >> The pool I need is not a thread pool or db pool. I need to pool
    >> objects which are a proxy for an external C process that communicates
    >> via sockets. The pool is necessary because the initial connection/
    >> authentication is expensive, plus the connection needs to be recycled
    >> every so often due to instability on the C side. The existing Java
    >> code uses commons-pool which incidentally is the base object pool for
    >> a db connection pooling library (DBCP). Anyway, maybe this is just so
    >> easy to do in Python that everyone rolls their own - but I have done a
    >> lot of googling with no luck.
    >>
    >> I just hate to re-invent the wheel here - I guess I should look at a
    >> db connection pooling library and see if they have a generic object
    >> pool somewhere in there. Any you can recommend that are more or less
    >> standalone db connection pool libraries ?

    >
    > I don't know enough about your domain, e.g. if your pooled objects have
    > state or not.
    >
    > Assuming that
    >
    > - they are objects
    > - they don't have state (like database connections)
    > - you want to recycle them after so many calls
    >
    > one of the simplest solutions would be something like this:
    >
    > class AutoDestructingWrapper(object):
    >
    > MAX_CALL_COUNT = 1000
    >
    > def __init__(self, object_factory):
    > self._object_factory = object_factory
    > self._call_count = self.MAX_CALL_COUNT
    >
    >



    This must be

    def __getattr__(self, name):
    self._call_count += 1
    if self._call_count >= self.MAX_CALL_COUNT:
    self._delegate = self._object_factory()
    self._call_count = 0
    return getattr(self._delegate, name)

    of course.

    Diez
    Diez B. Roggisch, Jul 15, 2009
    #5
  6. Rick Lawson

    John Nagle Guest

    Rick Lawson wrote:
    > On Jul 15, 4:53 am, Jonathan Gardner <>
    > wrote:
    >> On Jul 14, 6:34 pm, Rick Lawson <> wrote:
    >>
    >>> Appreciate any help on this. I am porting an app from Java to python
    >>> and need generic object pooling with hooks for object initialization /
    >>> cleanup and be able to specify an object timeout.

    >> Are you looking for something like a thread pool or a connection pool?
    >> Such a thing is easy to code in Python. Might as well write one from
    >> scratch for your particular need.
    >>
    >> By the way, a tip for writing Python: Forget Java. If you're porting
    >> an app, consider rewriting it from the architecture on up. You'll save
    >> yourself time and get a better result.

    >
    > Jonathan,
    >
    > Thanks for the advice but would like to pick your brain a little more.
    > The pool I need is not a thread pool or db pool. I need to pool
    > objects which are a proxy for an external C process that communicates
    > via sockets.


    "fcgi" is an option for this sort of thing. With "mod_fcgi" installed
    in Apache, and "fcgi.py" used to manage the Python side of the problem, you
    can have semi-persistent programs started up and shut down for you on the
    server side.

    I use this for "sitetruth.com". It's useful when CGI is too slow
    because the cost of launching Python and loading all the modules is
    far greater than the cost of processing the request.

    Once you find and install a working Apache "mod_fcgi" module, and a
    working "fcgi.py" module (I suggest "http://svn.saddi.com/py-lib/trunk/fcgi.py")
    writing the server side app is no harder than with CGI. But the performance
    is far better.

    John Nagle
    John Nagle, Jul 16, 2009
    #6
  7. Rick Lawson

    alex23 Guest

    On Jul 16, 2:03 pm, John Nagle <> wrote:
    >      "fcgi" is an option for this sort of thing.  With "mod_fcgi" installed
    > in Apache, and "fcgi.py" used to manage the Python side of the problem, you
    > can have semi-persistent programs started up and shut down for you on the
    > server side.


    Hey John,

    The environments in which I've been asked to develop webs apps using
    Python have all utilised mod_wsgi. Do you have any experience with
    mod_wsgi vs mod_fcgi, and if so, can you comment on any relevant
    performance / capability / ease-of-use differences?

    Cheers,
    alex23
    alex23, Jul 16, 2009
    #7
  8. Rick Lawson

    John Nagle Guest

    alex23 wrote:
    > On Jul 16, 2:03 pm, John Nagle <> wrote:
    >> "fcgi" is an option for this sort of thing. With "mod_fcgi" installed
    >> in Apache, and "fcgi.py" used to manage the Python side of the problem, you
    >> can have semi-persistent programs started up and shut down for you on the
    >> server side.

    >
    > Hey John,
    >
    > The environments in which I've been asked to develop webs apps using
    > Python have all utilised mod_wsgi. Do you have any experience with
    > mod_wsgi vs mod_fcgi, and if so, can you comment on any relevant
    > performance / capability / ease-of-use differences?
    >
    > Cheers,
    > alex23


    FCGI seems to be somewhat out of favor, perhaps because it isn't
    complicated enough. It's a mature technology and works reasonably
    well. It's been a puzzle to me that FCGI was taken out of the
    main Apache code base, because it gives production-level performance
    with CGI-type simplicity.

    WSGI has a mode for running Python inside the Apache process,
    which is less secure and doesn't allow multiple Python processes.
    That complicates mod_wsgi considerably, and ties it very closely
    to specific versions of Python and Python modules. As a result,
    the WSGI developers are patching at a great rate. I think the
    thing has too many "l33t features".

    I'd avoid "embedded mode". "Daemon mode" is the way to go if you use WSGI.
    I haven't tried WSGI; I don't need the grief of a package under
    heavy development.

    John Nagle
    John Nagle, Jul 16, 2009
    #8
  9. On Jul 16, 3:05 pm, John Nagle <> wrote:
    > alex23 wrote:
    > > On Jul 16, 2:03 pm, John Nagle <> wrote:
    > >>      "fcgi" is an option for this sort of thing.  With "mod_fcgi" installed
    > >> in Apache, and "fcgi.py" used to manage the Python side of the problem, you
    > >> can have semi-persistent programs started up and shut down for you on the
    > >> server side.

    >
    > > Hey John,

    >
    > > The environments in which I've been asked to develop webs apps using
    > > Python have all utilisedmod_wsgi. Do you have any experience with
    > >mod_wsgivs mod_fcgi, and if so, can you comment on any relevant
    > > performance / capability / ease-of-use differences?

    >
    > > Cheers,
    > > alex23

    >
    >     FCGI seems to be somewhat out of favor, perhaps because it isn't
    > complicated enough.


    I doubt that is the reason. It is out of favour for Python web hosting
    at least, because web hosting companies provide really crappy support
    for using Python with it and also don't like long lived processes.
    Most FASTCGI deployments are configured in a way more appropriate for
    PHP because that is what the default settings favour. This works to
    the detriment of Python. Even when people setup FASTCGI themselves,
    they still don't play with the configuration to optimise it for their
    specific Python application. Thus it can still run poorly.

    > It's a mature technology and works reasonably well.


    But is showing its age. It really needs a refresh. Various of its
    design decisions were from when network speeds and bandwidth were much
    lower and this complicates the design. Making the protocol format
    simpler in some areas would make it easier to implement. The protocol
    could also be built on to make process management more flexible,
    rather than relying on each implementation to come up with adhoc ways
    of managing it.

    Part of the problem with FASTCGI, or hosting of dynamic applications
    in general, is that web servers primary focus is serving of static
    files. Any support for dynamic web applications is more of a bolt on
    feature. Thus, web applications using Python will always be at a
    disadvantage. PHP manages okay because their model of operation is
    closer to the one shot processing of static files. Python requires
    persistent to work adequately with modern fat applications.

    > It's been a puzzle to me that FCGI was taken out of the
    > main Apache code base, because it gives production-level performance
    > with CGI-type simplicity.


    As far as I know FASTCGI support has in the past never been a part of
    the Apache code base. Both mod_fastcgi and mod_fcgid were developed by
    independent people and not under the ASF.

    In Apache 2.3 development versions (to become 2.4 when released),
    there will however be a mod_proxy_fcgi. The ASF has also taken over
    management of mod_fcgid and working out how that may be incorporated
    into future Apache versions.

    >     WSGI has a mode for running Python inside the Apache process,
    > which is less secure and doesn't allow multiple Python processes.


    Depending on the requirements it is more than adequate and if
    configured correctly gives better performance and scalability. Not
    everyone runs in a shared hosting environment. The different modes of
    mod_wsgi therefore give choice.

    > That complicates mod_wsgi considerably,


    No it doesn't. It is actual daemon mode that complicates things, not
    embedded mode.

    > and ties it very closely
    > to specific versions of Python and Python modules.  As a result,
    > the WSGI developers are patching at a great rate.


    What are you talking about when you say 'As a result, the WSGI
    developers are patching at a great rate'? As with some of your other
    comments, this is leaning towards being FUD and nothing more. There is
    no 'patching at a great rate' going on.

    > I think the thing has too many "l33t features".


    It is all about choice and providing flexibility. You may not see a
    need for certain features, but it doesn't mean other people don't have
    a need for it.

    > I'd avoid "embedded mode".  "Daemon mode" is the way to go if you use WSGI.


    I'd agree, but not sure you really understand the reasons for why it
    would be preferred.

    > I haven't tried WSGI;


    Then why comment on it as if you are knowledgeable on it.

    > I don't need the grief of a package under heavy development.


    More FUD.

    Graham
    Graham Dumpleton, Jul 17, 2009
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    2,116
    Irmen de Jong
    Jun 22, 2004
  2. boardrider
    Replies:
    0
    Views:
    723
    boardrider
    Aug 24, 2006
  3. Patrick Kowalzick
    Replies:
    5
    Views:
    456
    Patrick Kowalzick
    Mar 14, 2006
  4. www
    Replies:
    2
    Views:
    10,285
    bugbear
    Aug 22, 2007
  5. Gianni Galore
    Replies:
    0
    Views:
    1,253
    Gianni Galore
    Sep 8, 2010
Loading...

Share This Page