Async client for PostgreSQL?

Discussion in 'Python' started by Laszlo Nagy, Sep 1, 2012.

  1. Laszlo Nagy

    Laszlo Nagy Guest

    Is there any extension for Python that can do async I/O for PostgreSQL
    with tornadoweb's ioloop?

    Something like:

    class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
    pg_connection.(long_taking_query_sql,params,callback=self.on_query_opened)

    def on_query_opened(self, query):
    self.write(process_rows(query))
    self.finish()



    What would be an alternative?

    The theoretical problem: suppose there are 100 clients (web browsers)
    connected to the server with keep alive connections. They are doing
    long-polls and they are also sending/receiving events (with short
    response times). Each web browser has an associated state stored on the
    server side, in the memory (as an object tree). The state is bound to
    the client with a session id. Most requests will have to be responded
    with small amounts of data, calculated from the session state, or
    queried from the database. Most database queries are simple, running for
    about 100msec. But a few of them will run for 1sec or more. Number of
    requests ending in database queries is relatively low (10/sec). Other
    requests can be responded must faster. but they are much more frequent
    (100/sec, that is. 1 request/sec/client). There is a big global cache
    full of (Python) objects. Their purpose is to reduce the number of
    database queries. These objects in the global cache are emitting events
    to other objects found in the client sessions. Generally, it is not
    possible to tell what request will end in a database query.

    Multi-threading is not an option because number of clients is too high
    (running 100 threads is not good). This is why I decided to use anyc
    I/O. Tornadoweb looks good for most requirements: async i/o, store
    session state in objects etc. The biggest problem is that psycopg is not
    compatible with this model. If I use blocking I/O calls inside a request
    handler, then they will block all other requests most of the time,
    resulting in slow response times.

    What would be a good solution for this?

    Thanks,

    Laszlo
    Laszlo Nagy, Sep 1, 2012
    #1
    1. Advertising

  2. Laszlo Nagy

    jwp Guest

    On Friday, August 31, 2012 10:17:18 PM UTC-7, Laszlo Nagy wrote:
    > Is there any extension for Python that can do async I/O for PostgreSQL


    As others point out, the easiest route is using one of the blocking driverswith threads and "emulate" async operations.

    However, the low-level parts of py-postgresql (python.projects.postgresql.org) were designed with arbitrary modes in mind. That is, the protocol code is independent of the transport so that it could be used with frameworks like twisted given some effort. Much of the work that will go into py-postgresql over the next couple years will be to make it easier to integrate into arbitrary frameworks. Currently, I suspect it would require some "heavy lifting".. =\

    cheers,

    github.com/jwp
    jwp, Sep 2, 2012
    #2
    1. Advertising

  3. Laszlo Nagy

    jwp Guest

    On Friday, August 31, 2012 10:17:18 PM UTC-7, Laszlo Nagy wrote:
    > Is there any extension for Python that can do async I/O for PostgreSQL


    As others point out, the easiest route is using one of the blocking driverswith threads and "emulate" async operations.

    However, the low-level parts of py-postgresql (python.projects.postgresql.org) were designed with arbitrary modes in mind. That is, the protocol code is independent of the transport so that it could be used with frameworks like twisted given some effort. Much of the work that will go into py-postgresql over the next couple years will be to make it easier to integrate into arbitrary frameworks. Currently, I suspect it would require some "heavy lifting".. =\

    cheers,

    github.com/jwp
    jwp, Sep 2, 2012
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steven
    Replies:
    0
    Views:
    348
    Steven
    Nov 30, 2005
  2. Werner Thie

    Re: Async client for PostgreSQL?

    Werner Thie, Sep 1, 2012, in forum: Python
    Replies:
    0
    Views:
    225
    Werner Thie
    Sep 1, 2012
  3. Laszlo Nagy

    Re: Async client for PostgreSQL?

    Laszlo Nagy, Sep 1, 2012, in forum: Python
    Replies:
    2
    Views:
    291
  4. Werner Thie

    Re: Async client for PostgreSQL?

    Werner Thie, Sep 1, 2012, in forum: Python
    Replies:
    0
    Views:
    275
    Werner Thie
    Sep 1, 2012
  5. Laszlo Nagy

    Re: Async client for PostgreSQL?

    Laszlo Nagy, Sep 2, 2012, in forum: Python
    Replies:
    0
    Views:
    199
    Laszlo Nagy
    Sep 2, 2012
Loading...

Share This Page