xmlrpc with Python and large datases

W

writeson

Hi all,

I helped one of my co-workers put together an XMLRPC Python script that
allowed him to get database data from remote machines. This was done
because the source of the data could be Oracle on a Sun/Solaris
machine, and MySQL on our linux machines. Doing the script in Python
allowed him to gather the date with a general purpose API and just send
over some SQL queries.

Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?

Thanks,
Doug
 
D

Duncan Booth

writeson said:
Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?
Most web servers have a configurable limit on the size of data sent to the
server for a single request. It sounds as though you may be exceeding this.
Check what setting has been used on your web server.

If it is Apache then look at the limitrequestbody parameter, it is
unlimited by default, but advisable to set a limit to deter certain types
of DoS attack.
 
W

writeson

Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug
 
S

Steve Holden

writeson said:
Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug
Usually a "broken pipe" message is to do with disruption of a Unix
pipeline rather than a network connection. I've seen the message mostly,
IIRC, in BSD-based environments.

A pipeline connects the standard output of one process to the standard
input of another. If this second process terminated while the first is
still writing its output the "broken pipe" message will be raised. It's
therefore probable that some consumer process is terminating before its
producer.

Whether this is due to timeouts in the server environment is open to
question, but at least you'll be looking for the right problem :)

regards
Steve
 
F

Fredrik Lundh

Steve said:
Usually a "broken pipe" message is to do with disruption of a Unix pipeline rather than a network
connection. I've seen the message mostly, IIRC, in BSD-based environments.

the socket layer uses EPIPE to indicate that a socket has been shut down
by the remote end.

</F>
 
W

writeson

Steve,

Thanks for your reply, I'll look into things based on your comments.
Also, I've read your book "Python Web Programming" and wanted you to
know it has helped me a lot with various python projects, thanks!

Doug
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top