Async IO Server with Blocking DB

Discussion in 'Python' started by looking for, Apr 3, 2012.

  1. looking for

    looking for Guest

    Hi

    We are thinking about building a webservice server and considering
    python event-driven servers i.e. Gevent/Tornado/ Twisted or some
    combination thereof etc.

    We are having doubts about the db io part. Even with connection
    pooling and cache, there is a strong chance that server will block on
    db. Blocking for even few ms is bad.

    can someone suggest some solutions or is async-io is not at the prime-
    time yet.

    Thanks
    looking for, Apr 3, 2012
    #1
    1. Advertising

  2. looking for

    Jon Clements Guest

    On Tuesday, 3 April 2012 23:13:24 UTC+1, looking for wrote:
    > Hi
    >
    > We are thinking about building a webservice server and considering
    > python event-driven servers i.e. Gevent/Tornado/ Twisted or some
    > combination thereof etc.
    >
    > We are having doubts about the db io part. Even with connection
    > pooling and cache, there is a strong chance that server will block on
    > db. Blocking for even few ms is bad.
    >
    > can someone suggest some solutions or is async-io is not at the prime-
    > time yet.
    >
    > Thanks


    Maybe look at Cyclone (a Tornado variation built on Twisted), and various modules that will offer synch and events - GIYF! It's doable!

    Jon.
    Jon Clements, Apr 4, 2012
    #2
    1. Advertising

  3. looking for

    Steve Howell Guest

    On Apr 3, 3:13 pm, looking for <> wrote:
    > Hi
    >
    > We are thinking about building a webservice server and considering
    > python event-driven servers i.e. Gevent/Tornado/ Twisted or some
    > combination thereof etc.
    >
    > We are having doubts about the db io part. Even with connection
    > pooling and cache, there is a strong chance that server will block on
    > db. Blocking for even few ms is bad.
    >
    > can someone suggest some solutions or is async-io is not at the prime-
    > time yet.
    >


    Can you farm off DB requests to another process, so that you're at
    least not blocking the main process?
    Steve Howell, Apr 4, 2012
    #3
  4. On Apr 3, 6:13 pm, looking for <> wrote:
    > Hi
    >
    > We are thinking about building a webservice server and considering
    > python event-driven servers i.e. Gevent/Tornado/ Twisted or some
    > combination thereof etc.
    >
    > We are having doubts about the db io part. Even with connection
    > pooling and cache, there is a strong chance that server will block on
    > db. Blocking for even few ms is bad.
    >
    > can someone suggest some solutions or is async-io is not at the prime-
    > time yet.
    >
    > Thanks


    Twisted provides support for any DB-API module via
    twisted.enteprise.adbapi,
    which wraps the module in an asynchronous API (implemented using a
    thread pool).

    Since the calls all happen in separate threads, it doesn't matter that
    they block.

    If you're not talking about a SQL database or a DB-API module, maybe
    be more
    specific about the kind of database I/O you have in mind.

    Jean-Paul
    Jean-Paul Calderone, Apr 4, 2012
    #4

  5. > We are thinking about building a webservice server and considering
    > python event-driven servers i.e. Gevent/Tornado/ Twisted or some
    > combination thereof etc.
    >
    > We are having doubts about the db io part. Even with connection
    > pooling and cache, there is a strong chance that server will block on
    > db. Blocking for even few ms is bad.


    Libraries that support the event loop of your framework, be it Gevent,
    Tornado or Twisted, should yield to other work when the database
    connection (its socket) blocks.

    Now, database drivers that use C libraries are ussually a problem,
    unless designed to cooperate with event loops. One of those is psycopg2
    which at least cooperates with Gevent. CouchDBkit (using restkit ie
    http) also cooperates with Gevent fine.

    MySQLdb does not cooperate with Gevent as far as I know, so one
    sollution would be to run queries through a Thread Pool (built-in in
    Gevent 1.0dev). Another is to use the pure python mysqlconn
    (mysql-connector-repackaged on pypi) with gevents monkey patching.

    Sqlite should be probably run in a ThreadPool too, since it depends on
    disk I/O which is not async in Gevent.


    You can readup on Gevent integration of Couchdbkit and Psycopg2 in their
    documentation.


    > can someone suggest some solutions or is async-io is not at the prime-
    > time yet.



    --
    damjan
    Damjan Georgievski, Apr 6, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    12,444
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    918
    Valentin Tihomirov
    Oct 12, 2003
  3. nukleus
    Replies:
    14
    Views:
    817
    Chris Uppal
    Jan 22, 2007
  4. Christian
    Replies:
    5
    Views:
    723
    Esmond Pitt
    Dec 2, 2007
  5. Steven
    Replies:
    0
    Views:
    348
    Steven
    Nov 30, 2005
Loading...

Share This Page