Batching HTTP requests with httplib (Python 2.7)

C

Chicken McNuggets

I'm writing a simple library that communicates with a web service and am
wondering if there are any generally well regarded methods for batching
HTTP requests?

The problem with most web services is that they require a list of
sequential commands to be executed in a certain order to complete a
given task (or at least the one I am using does) so having to manually
call each command is a bit of a pain. How would you go about the design
of a library to interact with these services?

Any help is greatly appreciated :).
 
C

Cameron Simpson

| I'm writing a simple library that communicates with a web service and am
| wondering if there are any generally well regarded methods for batching
| HTTP requests?
|
| The problem with most web services is that they require a list of
| sequential commands to be executed in a certain order to complete a
| given task (or at least the one I am using does) so having to manually
| call each command is a bit of a pain. How would you go about the design
| of a library to interact with these services?
|
| Any help is greatly appreciated :).

Maybe I'm missing something. What's hard about:

- wrapping the web services calls in a simple wrapper which
composes the call, runs it, and returns the result parts
This lets you hide all the waffle about the base URL,
credentials etc in the wrapper and only supply the essentials
at call time.

- writing your workflow thing then as a simple function:

def doit(...):
web_service_call1(...)
web_service_call2(...)
web_service_call3(...)

with whatever internal control is required?

This has worked for me for simple things.

What am I missing about the larger context?
 
C

Chicken McNuggets

| I'm writing a simple library that communicates with a web service and am
| wondering if there are any generally well regarded methods for batching
| HTTP requests?
|
| The problem with most web services is that they require a list of
| sequential commands to be executed in a certain order to complete a
| given task (or at least the one I am using does) so having to manually
| call each command is a bit of a pain. How would you go about the design
| of a library to interact with these services?
|
| Any help is greatly appreciated :).

Maybe I'm missing something. What's hard about:

- wrapping the web services calls in a simple wrapper which
composes the call, runs it, and returns the result parts
This lets you hide all the waffle about the base URL,
credentials etc in the wrapper and only supply the essentials
at call time.

- writing your workflow thing then as a simple function:

def doit(...):
web_service_call1(...)
web_service_call2(...)
web_service_call3(...)

with whatever internal control is required?

This has worked for me for simple things.

What am I missing about the larger context?

That is what I have at the moment but it is ugly as hell. I was
wondering if there was a somewhat more elegant solution that I was missing.
 
D

Dwight Hutto

| The problem with most web services is that they require a list of
That is what I have at the moment but it is ugly as hell. I was wondering if
there was a somewhat more elegant solution that I was missing.
--

Well if the code works it works. There probably is a better way than
what I came up with:


def web_service_call0(y):
print "Service call = %i)" % y

def web_service_call1(y):
print "Service call = %i)" % y

def web_service_call2(y):
print "Service call = %i)" % y

def web_service_call3(y):
print "Service call = %i)" % y

def web_service_call4(y):
print "Service call = %i)" % y

service_num_list = [num for num in range(0,5)]
for service_num in service_num_list:
eval("web_service_call%i(%i)" % (service_num,service_num))



I just define the service calls, and iterate through them with an eval
that places in the service call num, but you would have to have a
matching list of the params that would go with each one.

I'm almost positive there will be some more well written ones on the way.
 
X

Xavier Combelle

instead of

Le 14/09/2012 12:56, Dwight Hutto a écrit :
service_num_list = [num for num in range(0,5)]
for service_num in service_num_list:
eval("web_service_call%i(%i)" % (service_num,service_num))
service_num_list = [num for num in range(0,5)]
for service_num in service_num_list:
locals()["web_service_call%i" % (service_num,)](service_num)

would be safer eval is really bad
 
C

Cameron Simpson

| On 14/09/2012 03:31, Cameron Simpson wrote:
| > | I'm writing a simple library that communicates with a web service and am
| > | wondering if there are any generally well regarded methods for batching
| > | HTTP requests?
| > |
| > | The problem with most web services is that they require a list of
| > | sequential commands to be executed in a certain order to complete a
| > | given task (or at least the one I am using does) so having to manually
| > | call each command is a bit of a pain. How would you go about the design
| > | of a library to interact with these services?
| >
| > Maybe I'm missing something. What's hard about:
| >
| > - wrapping the web services calls in a simple wrapper which
| > composes the call, runs it, and returns the result parts
| > This lets you hide all the waffle about the base URL,
| > credentials etc in the wrapper and only supply the essentials
| > at call time.
| >
| > - writing your workflow thing then as a simple function:
| >
| > def doit(...):
| > web_service_call1(...)
| > web_service_call2(...)
| > web_service_call3(...)
| >
| > with whatever internal control is required?
| >
| > This has worked for me for simple things.
| > What am I missing about the larger context?
|
| That is what I have at the moment but it is ugly as hell. I was
| wondering if there was a somewhat more elegant solution that I was missing.

Leading disclaimer: I'm no web service expert.

Well, I presume you're using a web service library like ZSI or suds?
I'm using suds; started with ZSI but it had apparent maintenance
stagnancy and also some other issues. I moved to suds and it's been
fine.

I make a class wrapping a suds Client object and some convenience
functions. A suds Client object is made from the WSDL file or URL
and fills out all the services with callable methods accepting the web
service parameters. So the class has a setup method like this:

def set_client(self, wsdlfile, baseurl):
''' Common code to set the .client attribute to a suds SOAP Client.
'''
from suds.client import Client
self.client = Client('file://'+wsdlfile)
self.client.set_options(location = baseurl)
self.client.set_options(proxy = {'https': 'proxy:3128'})

wsdlfile points at a local WSDL file, removing dependence on some server
to provide the WSDL. Makes testing easier too.

Then there's a general call wrapper. The wscall() method below is a
slightly scoured version of something I made for work. You hand it the
service name and any parameters needed for that service call and it looks
it up in the Client, calls it, sanity checks the result in a generic
sense. If things are good it returns the suds reply object. But if
anything goes wrong it logs it and returns None. "Goes wrong" includes
a successful call whose internal result contains an error response -
obviously that's application dependent.

The upshot is that the wrapper for a specific web service call then becomes:

def thing1(self, blah, blah, ...):
reply = self.wscall('thing1', blah, blah, ...)
if reply is None:
# badness happened; it has been logged, just return in whatever
# fashion you desire - return None, raise exception, etc
...
# otherwise we pick apart the reply object to get the bits that
# matter and return them
return stuff from the reply object

and your larger logic looks like:

def doit(self, ...):
self.thing1(1,2,3)
self.thing2(...)

and so forth.

The general web service wrapper wscall() below has some stuff ripped out,
but I've left in some application specific stuff for illustration:

- optional _client and _cred keyword parameters to override the
default Client and credentials from the main class instance

- the Pfx context manager stuff arranges to prefix log messages and
exception strings with context, which I find very handy in debugging and
logging

- likewise the LogTime context manager logs things that exceed a time
threshold

- the wsQueue.submit(...) stuff punts the actual web service call
through a capacity queue (a bit like the futures module) because
this is a capacity limited multithreaded app.

For your purposes you could write:

websvc = getattr(_client.service, servicename)
with LogTime(servicename, threshold=3):
try:
reply = websvc(userId=_cred.username, password=_cred.password,
*args, **kwargs)
except:
exc_info = sys.exc_info
reply = None
else:
exc_info = None

which is much easier to follow.

- the 'callStatus' thing is application specific, left as an example
of a post-call sanity check you would add for calls that complete
but return with an error indication

Anyway, all that said, the general wrapper looks like this:

def doWScall(self, servicename, _client=None, _cred=None, *args, **kwargs):
''' General wrapper for a web service call.
Returns the native SUDS reply object if OK, otherwise None.
`_client` can be used to override use of self.client.
`_cred` can be used to override use of self.cred.
'''
if _client is None:
_client = self.client
if _cred is None:
_cred = self.cred
with self.pfx:
with Pfx("doWScall.%s(...)", servicename):
websvc = getattr(_client.service, servicename)
with LogTime(servicename, threshold=3):
reply, exc_info = self.wsQueue.submit( partial(websvc,
userId=_cred.username,
password=_cred.password,
*args,
**kwargs),
name=servicename).wait()
if exc_info:
error("FAIL, exception in web service call",
exc_info=exc_info)
return None
if reply is None:
arglist = listargs(args, kwargs)
error("web service called but got None anyway:
arglist=(%s)",
", ".join(arglist))
return None
if hasattr(reply, 'callStatus'):
if reply.callStatus == "OK":
info("OK")
return reply
... log an error message including relevant stuff from the reply ...
return None

Anyway, I hope that shows a way to get the main logic clear and the
overhead for wrapping specific calls quite lightweight.

Cheers,
--
Cameron Simpson <[email protected]>

Well, it's one louder, isn't it? It's not ten. You see, most blokes are
gonna be playing at ten, you're on ten here, all the way up, all the way up,
all the way up, you're on ten on your guitar, where can you go from there?
Where? Nowhere, exactly. What we do is, if we need that extra push over the
cliff, you know what we do? Eleven. Exactly. One louder.
- Nigel Tufnel, _This Is Spinal Tap_
 
N

Neil Cerutti

Le 14/09/2012 12:56, Dwight Hutto a ?crit :
service_num_list = [num for num in range(0,5)]
for service_num in service_num_list:
eval("web_service_call%i(%i)" % (service_num,service_num))
service_num_list = [num for num in range(0,5)]

service_num_list = list(range(5))
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top