multipart/form-data in an HTTP client

[Nelson Minar]

I'm writing some code to upload photos to Flickr. The Photo Upload API
requires documents be POSTed via a multipart/form-data request. I was
surprised to learn that Python 2.3's HTTP clients don't support this
form of POSTs. There is support in cgi.py, for servers.

There are some implementations of multipart/form-data on ASPN:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306
urllib2_file seems to meet my needs, but I'm not wild about how it's
implemented. Is there some other recommended way to do
multipart/form-data uploads with HTTP in Python?


References:
http://www.flickr.com/services/api/
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306
http://www.faqs.org/rfcs/rfc1867.html

 

[Anthony Baxter]

There are some implementations of multipart/form-data on ASPN:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306
urllib2_file seems to meet my needs, but I'm not wild about how it's
implemented. Is there some other recommended way to do
multipart/form-data uploads with HTTP in Python?

I like ClientForm. I hope one day it or something like it will be in the stdlib.

 

[John J. Lee]

I'm writing some code to upload photos to Flickr. The Photo Upload API
requires documents be POSTed via a multipart/form-data request. I was
surprised to learn that Python 2.3's HTTP clients don't support this
form of POSTs. There is support in cgi.py, for servers.

"Not supported" is an exaggeration, perhaps: there isn't special
support to make it especially easy, true, but neither is there
anything about urllib2 that makes it harder than it should be given
the level of the interface exposed: you just have to add the right
HTTP headers and HTTP request body data.

There are some implementations of multipart/form-data on ASPN:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306
urllib2_file seems to meet my needs, but I'm not wild about how it's
implemented.

What's wrong with that implementation? Looks reasonable to me, though
it seems to have a hack to work around bad servers that is different to
the one I have in my own code. I can well believe that both hacks are
required to work with as many servers as possible :-(


John

 

[Jeff Shannon]

I'm writing some code to upload photos to Flickr. The Photo Upload API
requires documents be POSTed via a multipart/form-data request. I was
surprised to learn that Python 2.3's HTTP clients don't support this
form of POSTs. There is support in cgi.py, for servers.

There are some implementations of multipart/form-data on ASPN:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306
urllib2_file seems to meet my needs, but I'm not wild about how it's
implemented. Is there some other recommended way to do
multipart/form-data uploads with HTTP in Python?

I've been using something closely modelled on that Cookbook recipe,
without any real problems. (I've updated it to use HTTPConnection()
and return the response object, and in my case I'm connecting to an
HTTPS server, but these are trivial modifications.)

I, too, was surprised that the existing Python libraries don't directly
support multipart/form-data already, and I hope that this gets added in
soon.

Jeff Shannon
Technician/Programmer
Credit International

 

[John J. Lee]

I've been using something closely modelled on that Cookbook recipe,
without any real problems. (I've updated it to use
HTTPConnection() and return the response object, and in my case I'm
connecting to an HTTPS server, but these are trivial modifications.)

I, too, was surprised that the existing Python libraries don't
directly support multipart/form-data already, and I hope that this
gets added in soon.

[...]

Have you published your function? If not, please do.

Several people would like to see such a function in 2.5 (including
me), and your function sounds a good candidate. Maybe a little
polishing is required, but don't let that stop you: if you don't get
time to do that polishing, this is a rare occasion when somebody else
is likely to push it the small additional distance it would need to go
in order to get into the Python stdlib!


John

 

[Jeff Shannon]

[...]

I've been using something closely modelled on that Cookbook recipe,
without any real problems. (I've updated it to use
HTTPConnection() and return the response object, and in my case I'm
connecting to an HTTPS server, but these are trivial modifications.)

[...]

Have you published your function? If not, please do.

Here it is, for what it's worth. (Hopefully my mailer won't mangle the
indentation...) As I noted, the differences between this and the
Cookbook recipe are minimal -- I literally copied the recipe into my
editor and made a few changes.

If I were going to put further effort into polishing it for library use
(which I probably won't have much opportunity to do), I'd make at least
two changes. One would be to set it up to select between HTTP and
HTTPS. (It currently tries to determine the protocol from the hostname
string, but does nothing with that information.) The other would be to
enable the file-subpart to add a Content-type header. (This is
irrelevant for my particular application, as the form(s) I'm uploading
to don't care.)

In case it matters to anyone, inasmuch as this code is nearly identical
to the published recipe, my changes can be considered to be under the
same license as the original recipe. If someone feels like using this
in any way, feel free.

Jeff Shannon
Technician/Programmer
Credit International


#! /usr/bin/python

import httplib

def post_multipart(host, selector, fields, files):
"""
Post fields and files to an http host as multipart/form-data.
fields is a sequence of (name, value) elements for regular form fields.
files is a sequence of (name, filename, value) elements for data to be uploaded as files.
Return an appropriate httplib.HTTPResponse object.
"""
content_type, body = encode_multipart_formdata(fields, files)
protocol = host.split(':')[0]
h = httplib.HTTPSConnection(host)
h.putrequest('POST', selector)
h.putheader('content-type', content_type)
h.putheader('content-length', str(len(body)))
h.endheaders()
h.send(body)
response = h.getresponse()
return response

def encode_multipart_formdata(fields, files):
"""
fields is a sequence of (name, value) elements for regular form fields.
files is a sequence of (name, filename, value) elements for data to be uploaded as files.
Return (content_type, body) ready for httplib.HTTPConnection instance
"""
BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_---$---'
CRLF = '\r\n'
L = []
for (key, value) in fields:
L.append('--' + BOUNDARY)
L.append('Content-Disposition: form-data; name="%s"' % key)
L.append('')
L.append(value)
for (key, filename, value) in files:
L.append('--' + BOUNDARY)
L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
L.append('')
L.append(value)
L.append('--' + BOUNDARY + '--')
L.append('')
body = CRLF.join(L)
content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
return content_type, body