Async IO Server with Blocking DB

[looking for]

Hi

We are thinking about building a webservice server and considering
python event-driven servers i.e. Gevent/Tornado/ Twisted or some
combination thereof etc.

We are having doubts about the db io part. Even with connection
pooling and cache, there is a strong chance that server will block on
db. Blocking for even few ms is bad.

can someone suggest some solutions or is async-io is not at the prime-
time yet.

Thanks

 

[Jon Clements]

Hi

We are thinking about building a webservice server and considering
python event-driven servers i.e. Gevent/Tornado/ Twisted or some
combination thereof etc.

We are having doubts about the db io part. Even with connection
pooling and cache, there is a strong chance that server will block on
db. Blocking for even few ms is bad.

can someone suggest some solutions or is async-io is not at the prime-
time yet.

Thanks

Maybe look at Cyclone (a Tornado variation built on Twisted), and various modules that will offer synch and events - GIYF! It's doable!

Jon.

 

[Steve Howell]

Hi

We are thinking about building a webservice server and considering
python event-driven servers i.e. Gevent/Tornado/ Twisted or some
combination thereof etc.

We are having doubts about the db io part. Even with connection
pooling and cache, there is a strong chance that server will block on
db. Blocking for even few ms is bad.

can someone suggest some solutions or is async-io is not at the prime-
time yet.

Can you farm off DB requests to another process, so that you're at
least not blocking the main process?

 

[Jean-Paul Calderone]

Hi

We are thinking about building a webservice server and considering
python event-driven servers i.e. Gevent/Tornado/ Twisted or some
combination thereof etc.

We are having doubts about the db io part. Even with connection
pooling and cache, there is a strong chance that server will block on
db. Blocking for even few ms is bad.

can someone suggest some solutions or is async-io is not at the prime-
time yet.

Thanks

Twisted provides support for any DB-API module via
twisted.enteprise.adbapi,
which wraps the module in an asynchronous API (implemented using a
thread pool).

Since the calls all happen in separate threads, it doesn't matter that
they block.

If you're not talking about a SQL database or a DB-API module, maybe
be more
specific about the kind of database I/O you have in mind.

Jean-Paul

 

[Damjan Georgievski]

We are thinking about building a webservice server and considering
python event-driven servers i.e. Gevent/Tornado/ Twisted or some
combination thereof etc.

We are having doubts about the db io part. Even with connection
pooling and cache, there is a strong chance that server will block on
db. Blocking for even few ms is bad.

Libraries that support the event loop of your framework, be it Gevent,
Tornado or Twisted, should yield to other work when the database
connection (its socket) blocks.

Now, database drivers that use C libraries are ussually a problem,
unless designed to cooperate with event loops. One of those is psycopg2
which at least cooperates with Gevent. CouchDBkit (using restkit ie
http) also cooperates with Gevent fine.

MySQLdb does not cooperate with Gevent as far as I know, so one
sollution would be to run queries through a Thread Pool (built-in in
Gevent 1.0dev). Another is to use the pure python mysqlconn
(mysql-connector-repackaged on pypi) with gevents monkey patching.

Sqlite should be probably run in a ThreadPool too, since it depends on
disk I/O which is not async in Gevent.


You can readup on Gevent integration of Couchdbkit and Psycopg2 in their
documentation.