xmlrpc with Python and large datases

Discussion in 'Python' started by writeson, Mar 21, 2005.

  1. writeson

    writeson Guest

    Hi all,

    I helped one of my co-workers put together an XMLRPC Python script that
    allowed him to get database data from remote machines. This was done
    because the source of the data could be Oracle on a Sun/Solaris
    machine, and MySQL on our linux machines. Doing the script in Python
    allowed him to gather the date with a general purpose API and just send
    over some SQL queries.

    Since then he's run into a problem. If he sends a query that gets a
    very large recordset from the database the script fails and it vaguely
    reports about a "broken pipe", which I'm guessing is a problem with the
    network connection. Has anyone else seen this use XMLRPC and Python and
    large amounts of data being returned via XMLRPC? And if so, is there a
    solution?

    Thanks,
    Doug
     
    writeson, Mar 21, 2005
    #1
    1. Advertising

  2. writeson

    Duncan Booth Guest

    writeson wrote:

    > Since then he's run into a problem. If he sends a query that gets a
    > very large recordset from the database the script fails and it vaguely
    > reports about a "broken pipe", which I'm guessing is a problem with the
    > network connection. Has anyone else seen this use XMLRPC and Python and
    > large amounts of data being returned via XMLRPC? And if so, is there a
    > solution?
    >

    Most web servers have a configurable limit on the size of data sent to the
    server for a single request. It sounds as though you may be exceeding this.
    Check what setting has been used on your web server.

    If it is Apache then look at the limitrequestbody parameter, it is
    unlimited by default, but advisable to set a limit to deter certain types
    of DoS attack.
     
    Duncan Booth, Mar 21, 2005
    #2
    1. Advertising

  3. writeson

    writeson Guest

    Duncan,

    Thanks for the reply. We are running this on an Apache server on the
    linux box, and an iPlanet4.1 server on the solaris machines. However,
    both these servers are strictly 'inside' the firewall. I checked the
    apache configuration and there is no limitrequestbody parameter in the
    file at all. So I'm assuming from what you've said that our
    configuration would be unlimited. However, I will test that by
    inserting limitrequestbody and setting it to 0 for unlimited to see if
    that changes things.

    Thanks,
    Doug
     
    writeson, Mar 21, 2005
    #3
  4. writeson

    Steve Holden Guest

    writeson wrote:
    > Duncan,
    >
    > Thanks for the reply. We are running this on an Apache server on the
    > linux box, and an iPlanet4.1 server on the solaris machines. However,
    > both these servers are strictly 'inside' the firewall. I checked the
    > apache configuration and there is no limitrequestbody parameter in the
    > file at all. So I'm assuming from what you've said that our
    > configuration would be unlimited. However, I will test that by
    > inserting limitrequestbody and setting it to 0 for unlimited to see if
    > that changes things.
    >
    > Thanks,
    > Doug
    >

    Usually a "broken pipe" message is to do with disruption of a Unix
    pipeline rather than a network connection. I've seen the message mostly,
    IIRC, in BSD-based environments.

    A pipeline connects the standard output of one process to the standard
    input of another. If this second process terminated while the first is
    still writing its output the "broken pipe" message will be raised. It's
    therefore probable that some consumer process is terminating before its
    producer.

    Whether this is due to timeouts in the server environment is open to
    question, but at least you'll be looking for the right problem :)

    regards
    Steve
    --
    Meet the Python developers and your c.l.py favorites March 23-25
    Come to PyCon DC 2005 http://www.pycon.org/
    Steve Holden http://www.holdenweb.com/
     
    Steve Holden, Mar 22, 2005
    #4
  5. Steve Holden wrote:

    > Usually a "broken pipe" message is to do with disruption of a Unix pipeline rather than a network
    > connection. I've seen the message mostly, IIRC, in BSD-based environments.


    the socket layer uses EPIPE to indicate that a socket has been shut down
    by the remote end.

    </F>
     
    Fredrik Lundh, Mar 22, 2005
    #5
  6. writeson

    writeson Guest

    Steve,

    Thanks for your reply, I'll look into things based on your comments.
    Also, I've read your book "Python Web Programming" and wanted you to
    know it has helped me a lot with various python projects, thanks!

    Doug
     
    writeson, Mar 23, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Etienne Posthumus
    Replies:
    1
    Views:
    1,162
    Roger Binns
    Apr 1, 2004
  2. ted holden

    Python xmlrpc servers?

    ted holden, Dec 1, 2004, in forum: Python
    Replies:
    9
    Views:
    823
    ted holden
    Dec 1, 2004
  3. Jp Calderone

    Re: [Twisted-Python] xmlrpc deferred

    Jp Calderone, May 27, 2005, in forum: Python
    Replies:
    0
    Views:
    499
    Jp Calderone
    May 27, 2005
  4. News123
    Replies:
    9
    Views:
    3,210
    vilas
    Feb 15, 2012
  5. Replies:
    2
    Views:
    164
    Fábio Santos
    Jun 20, 2013
Loading...

Share This Page