XMLRPC Server

[viscanti]

Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
It's not too difficult to configure everything, but I would like to
tune it in order to receive up to 2000 calls per minute without any
problems. Do Pthon CGIs use threading?
I need to make it very efficient, but I haven't found much information
about Python CGI optimization.
The called function will update a table in a mysql db. I will use
triggers to export data from the table updated by the xmlrpc server to
other tables used by the backend application.

any hint?

lv

 

[Diez B. Roggisch]

Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
It's not too difficult to configure everything, but I would like to
tune it in order to receive up to 2000 calls per minute without any
problems. Do Pthon CGIs use threading?
I need to make it very efficient, but I haven't found much information
about Python CGI optimization.
The called function will update a table in a mysql db. I will use
triggers to export data from the table updated by the xmlrpc server to
other tables used by the backend application.

You might consider using the twisted application server framework instead,
and totally ditch the CGI, and even the apache.

http://twistedmatrix.com/

http://twistedmatrix.com/projects/web/documentation/examples/xmlrpc.py

Diez

 

[Lorenzo]

Unfortunately I have to use Apache. The server implementation will we
very easy, so I'm also considering more efficient solutions than
python

lv

 

[Lorenzo]

Unfortunately I have to use Apache. The server implementation will be
very easy, so I'm also considering more efficient solutions than
python

lv

 

[Brian Quinlan]

Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
It's not too difficult to configure everything, but I would like to
tune it in order to receive up to 2000 calls per minute without any
problems.

That doesn't seem like excessive volume. Why not just try it? You could
replace your database logic with time.sleep(1) for now.

Do Pthon CGIs use threading?

To do what? CGI requires that a new interpreter instance be launched to
handle every request. The requests will be handled in parallel with the
number of requests handled simultaneously depending on your apache
configuration.

I need to make it very efficient,

Actually, you might not have to. 2000 calls/minute isn't that big,
assuming you have a decent server.

Cheers,
Brian

 

[Paul Boddie]

Unfortunately I have to use Apache. The server implementation will we
very easy, so I'm also considering more efficient solutions than
python

You could try mod_python if there isn't an absolute requirement for
CGI:

http://www.modpython.org/

Some people might recommend other frameworks which operate in separate
long-running processes, but if you don't have the freedom to have such
processes, you might wish to consider focusing on the start-up costs
of your CGI programs. Once upon a time, CGI was deemed very expensive
because process creation was itself expensive, and many database-
related CGI programs had to open connections to database systems whose
connection costs were very expensive (eg. Oracle). While not spawning
new processes and not opening and closing database connections avoids
such costs, it is worth reviewing just how expensive such things are
on modern operating systems (on modern hardware) and with other
database systems (such as MySQL, which you said you were using).
Ultimately, some benchmarking/profiling will indicate whether your
performance expectations are realistic.

Paul

 

[Fredrik Lundh]

Actually, you might not have to. 2000 calls/minute isn't that big,
assuming you have a decent server.

well, if you're talking pure CGI, you need to start the interpreter,
import the required modules, connect to the database, unmarshal the
xml-rpc request, talk to the database, marshal the response, and shut
down, in less than 30 milliseconds.

just importing the CGI module (or the database module) can take longer
than that...

</F>

 

[Brian Quinlan]

well, if you're talking pure CGI, you need to start the interpreter,
import the required modules, connect to the database, unmarshal the
xml-rpc request, talk to the database, marshal the response, and shut
down, in less than 30 milliseconds.

just importing the CGI module (or the database module) can take longer
than that...

The original performance specification was "...receive up to 2000 calls
per minute". I don't believe that means that a call has to be serviced
in under 30ms (wall-clock time) but total CPU time would have to be
<30ms in order to not fall behind under a constant 2000 requests/second
load. So we can probably remove database connection and communication
time (i.e. IO-bound components). Still, it's a lot tighter than I though
it would be:

% time python -c "import SimpleXMLRPCServer; import MySQLdb"

real 0m0.144s
user 0m0.046s
sys 0m0.064s

So it's already almost 4x too slow. But I'm running this on Ubuntu,
running on VMWare on my 1.6GHz Pentium-M laptop. I would assume that a
beefy server would do a lot better.

Cheers,
Brian