Re: Async client for PostgreSQL?

Discussion in 'Python' started by Werner Thie, Sep 1, 2012.

  1. Werner Thie

    Werner Thie Guest

    On 8/31/12 7:17 PM, Laszlo Nagy wrote:
    > Is there any extension for Python that can do async I/O for PostgreSQL
    > with tornadoweb's ioloop?
    >
    > Something like:
    >
    > class MainHandler(tornado.web.RequestHandler):
    > @tornado.web.asynchronous
    > def get(self):
    > pg_connection.(long_taking_query_sql,params,callback=self.on_query_opened)
    >
    > def on_query_opened(self, query):
    > self.write(process_rows(query))
    > self.finish()
    >
    >
    >
    > What would be an alternative?
    >
    > The theoretical problem: suppose there are 100 clients (web browsers)
    > connected to the server with keep alive connections. They are doing
    > long-polls and they are also sending/receiving events (with short
    > response times). Each web browser has an associated state stored on the
    > server side, in the memory (as an object tree). The state is bound to
    > the client with a session id. Most requests will have to be responded
    > with small amounts of data, calculated from the session state, or
    > queried from the database. Most database queries are simple, running for
    > about 100msec. But a few of them will run for 1sec or more. Number of
    > requests ending in database queries is relatively low (10/sec). Other
    > requests can be responded must faster. but they are much more frequent
    > (100/sec, that is. 1 request/sec/client). There is a big global cache
    > full of (Python) objects. Their purpose is to reduce the number of
    > database queries. These objects in the global cache are emitting events
    > to other objects found in the client sessions. Generally, it is not
    > possible to tell what request will end in a database query.
    >
    > Multi-threading is not an option because number of clients is too high
    > (running 100 threads is not good). This is why I decided to use anyc
    > I/O. Tornadoweb looks good for most requirements: async i/o, store
    > session state in objects etc. The biggest problem is that psycopg is not
    > compatible with this model. If I use blocking I/O calls inside a request
    > handler, then they will block all other requests most of the time,
    > resulting in slow response times.
    >
    > What would be a good solution for this?
    >
    > Thanks,
    >
    > Laszlo
    >


    Hi

    does running on tornado imply that you would not consider twisted
    http://twistedmatrix.com ?

    If not, twisted has exactly this capability hiding long running queries
    on whatever db's behind deferToThread().

    Brute force I would pack the db access into a twisted run web-service,
    forking work out in twisted either with deferToThread() or if you want
    to take advantage of using processes instead of threads thus being able
    to tap all cores, then have a look at ampoule at
    https://launchpad.net/ampoule - be aware though that ampoule has a 64k
    limit on what can be passed around.

    Werner
     
    Werner Thie, Sep 1, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steven
    Replies:
    0
    Views:
    394
    Steven
    Nov 30, 2005
  2. Laszlo Nagy

    Async client for PostgreSQL?

    Laszlo Nagy, Sep 1, 2012, in forum: Python
    Replies:
    2
    Views:
    318
  3. Laszlo Nagy

    Re: Async client for PostgreSQL?

    Laszlo Nagy, Sep 1, 2012, in forum: Python
    Replies:
    2
    Views:
    320
  4. Werner Thie

    Re: Async client for PostgreSQL?

    Werner Thie, Sep 1, 2012, in forum: Python
    Replies:
    0
    Views:
    294
    Werner Thie
    Sep 1, 2012
  5. Laszlo Nagy

    Re: Async client for PostgreSQL?

    Laszlo Nagy, Sep 2, 2012, in forum: Python
    Replies:
    0
    Views:
    217
    Laszlo Nagy
    Sep 2, 2012
Loading...

Share This Page