XMLRPC Server

Discussion in 'Python' started by viscanti@gmail.com, Feb 6, 2007.

  1. Guest

    Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
    It's not too difficult to configure everything, but I would like to
    tune it in order to receive up to 2000 calls per minute without any
    problems. Do Pthon CGIs use threading?
    I need to make it very efficient, but I haven't found much information
    about Python CGI optimization.
    The called function will update a table in a mysql db. I will use
    triggers to export data from the table updated by the xmlrpc server to
    other tables used by the backend application.

    any hint?

    lv
     
    , Feb 6, 2007
    #1
    1. Advertising

  2. wrote:

    > Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
    > It's not too difficult to configure everything, but I would like to
    > tune it in order to receive up to 2000 calls per minute without any
    > problems. Do Pthon CGIs use threading?
    > I need to make it very efficient, but I haven't found much information
    > about Python CGI optimization.
    > The called function will update a table in a mysql db. I will use
    > triggers to export data from the table updated by the xmlrpc server to
    > other tables used by the backend application.


    You might consider using the twisted application server framework instead,
    and totally ditch the CGI, and even the apache.

    http://twistedmatrix.com/

    http://twistedmatrix.com/projects/web/documentation/examples/xmlrpc.py

    Diez
     
    Diez B. Roggisch, Feb 6, 2007
    #2
    1. Advertising

  3. Lorenzo Guest

    Unfortunately I have to use Apache. The server implementation will we
    very easy, so I'm also considering more efficient solutions than
    python

    lv

    On Feb 6, 11:36 am, "Diez B. Roggisch" <> wrote:
    > wrote:
    > > Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
    > > It's not too difficult to configure everything, but I would like to
    > > tune it in order to receive up to 2000 calls per minute without any
    > > problems. Do Pthon CGIs use threading?
    > > I need to make it very efficient, but I haven't found much information
    > > about Python CGI optimization.
    > > The called function will update a table in a mysql db. I will use
    > > triggers to export data from the table updated by the xmlrpc server to
    > > other tables used by the backend application.

    >
    > You might consider using the twisted application server framework instead,
    > and totally ditch the CGI, and even the apache.
    >
    > http://twistedmatrix.com/
    >
    > http://twistedmatrix.com/projects/web/documentation/examples/xmlrpc.py
    >
    > Diez
     
    Lorenzo, Feb 6, 2007
    #3
  4. Lorenzo Guest

    Unfortunately I have to use Apache. The server implementation will be
    very easy, so I'm also considering more efficient solutions than
    python

    lv

    On Feb 6, 11:36 am, "Diez B. Roggisch" <> wrote:
    > wrote:
    > > Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
    > > It's not too difficult to configure everything, but I would like to
    > > tune it in order to receive up to 2000 calls per minute without any
    > > problems. Do Pthon CGIs use threading?
    > > I need to make it very efficient, but I haven't found much information
    > > about Python CGI optimization.
    > > The called function will update a table in a mysql db. I will use
    > > triggers to export data from the table updated by the xmlrpc server to
    > > other tables used by the backend application.

    >
    > You might consider using the twisted application server framework instead,
    > and totally ditch the CGI, and even the apache.
    >
    > http://twistedmatrix.com/
    >
    > http://twistedmatrix.com/projects/web/documentation/examples/xmlrpc.py
    >
    > Diez
     
    Lorenzo, Feb 6, 2007
    #4
  5. wrote:
    > Hi, I'm trying to create an XMLRPC server using apache + python (cgi).
    > It's not too difficult to configure everything, but I would like to
    > tune it in order to receive up to 2000 calls per minute without any
    > problems.


    That doesn't seem like excessive volume. Why not just try it? You could
    replace your database logic with time.sleep(1) for now.

    > Do Pthon CGIs use threading?


    To do what? CGI requires that a new interpreter instance be launched to
    handle every request. The requests will be handled in parallel with the
    number of requests handled simultaneously depending on your apache
    configuration.

    > I need to make it very efficient,


    Actually, you might not have to. 2000 calls/minute isn't that big,
    assuming you have a decent server.

    Cheers,
    Brian
     
    Brian Quinlan, Feb 6, 2007
    #5
  6. Paul Boddie Guest

    On 6 Feb, 12:30, "Lorenzo" <> wrote:
    > Unfortunately I have to use Apache. The server implementation will we
    > very easy, so I'm also considering more efficient solutions than
    > python


    You could try mod_python if there isn't an absolute requirement for
    CGI:

    http://www.modpython.org/

    Some people might recommend other frameworks which operate in separate
    long-running processes, but if you don't have the freedom to have such
    processes, you might wish to consider focusing on the start-up costs
    of your CGI programs. Once upon a time, CGI was deemed very expensive
    because process creation was itself expensive, and many database-
    related CGI programs had to open connections to database systems whose
    connection costs were very expensive (eg. Oracle). While not spawning
    new processes and not opening and closing database connections avoids
    such costs, it is worth reviewing just how expensive such things are
    on modern operating systems (on modern hardware) and with other
    database systems (such as MySQL, which you said you were using).
    Ultimately, some benchmarking/profiling will indicate whether your
    performance expectations are realistic.

    Paul
     
    Paul Boddie, Feb 6, 2007
    #6
  7. Brian Quinlan wrote:

    > Actually, you might not have to. 2000 calls/minute isn't that big,
    > assuming you have a decent server.


    well, if you're talking pure CGI, you need to start the interpreter,
    import the required modules, connect to the database, unmarshal the
    xml-rpc request, talk to the database, marshal the response, and shut
    down, in less than 30 milliseconds.

    just importing the CGI module (or the database module) can take longer
    than that...

    </F>
     
    Fredrik Lundh, Feb 6, 2007
    #7
  8. Fredrik Lundh wrote:
    > well, if you're talking pure CGI, you need to start the interpreter,
    > import the required modules, connect to the database, unmarshal the
    > xml-rpc request, talk to the database, marshal the response, and shut
    > down, in less than 30 milliseconds.
    >
    > just importing the CGI module (or the database module) can take longer
    > than that...


    The original performance specification was "...receive up to 2000 calls
    per minute". I don't believe that means that a call has to be serviced
    in under 30ms (wall-clock time) but total CPU time would have to be
    <30ms in order to not fall behind under a constant 2000 requests/second
    load. So we can probably remove database connection and communication
    time (i.e. IO-bound components). Still, it's a lot tighter than I though
    it would be:

    % time python -c "import SimpleXMLRPCServer; import MySQLdb"

    real 0m0.144s
    user 0m0.046s
    sys 0m0.064s

    So it's already almost 4x too slow. But I'm running this on Ubuntu,
    running on VMWare on my 1.6GHz Pentium-M laptop. I would assume that a
    beefy server would do a lot better.

    Cheers,
    Brian
     
    Brian Quinlan, Feb 6, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jason Kratz

    xmlrpc server on red hat 9 question

    Jason Kratz, Jun 29, 2003, in forum: Python
    Replies:
    0
    Views:
    786
    Jason Kratz
    Jun 29, 2003
  2. =?ISO-8859-1?Q?Juan_Carlos_CORU=D1A?=

    Automation wrapped xmlrpc Server

    =?ISO-8859-1?Q?Juan_Carlos_CORU=D1A?=, Jul 3, 2003, in forum: Python
    Replies:
    0
    Views:
    362
    =?ISO-8859-1?Q?Juan_Carlos_CORU=D1A?=
    Jul 3, 2003
  3. Mark McClain

    xmlrpc server

    Mark McClain, Oct 17, 2003, in forum: Python
    Replies:
    0
    Views:
    340
    Mark McClain
    Oct 17, 2003
  4. Brian Quinlan

    RE: xmlrpc server

    Brian Quinlan, Oct 17, 2003, in forum: Python
    Replies:
    1
    Views:
    350
    A.M. Kuchling
    Oct 19, 2003
  5. Etienne Posthumus
    Replies:
    1
    Views:
    1,159
    Roger Binns
    Apr 1, 2004
Loading...

Share This Page