creating tar file and streaming it over HTTP?

[pbienst]

I would like to bundle up a number of files in a tar file and send it
over a HTTP connection, but I would like to do this without creating
the tar file on disk first.

I know I can get tarfile to output to a stream by doing something like

tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)

However, I can't figure out which file object to use to send this over
a HTTP connection.

I tried

conn = httplib.HTTPConnection(hostname, port)
conn.putrequest("PUT", "/client/files")
conn.endheaders()
tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
for filename in filenames:
tar_pipe.add(filename)
tar_pipe.close()
conn.getresponse()

but that does not send any data...

 

[Steve Holden]

I would like to bundle up a number of files in a tar file and send it
over a HTTP connection, but I would like to do this without creating
the tar file on disk first.

I know I can get tarfile to output to a stream by doing something like

tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)

However, I can't figure out which file object to use to send this over
a HTTP connection.

I tried

conn = httplib.HTTPConnection(hostname, port)
conn.putrequest("PUT", "/client/files")
conn.endheaders()
tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
for filename in filenames:
tar_pipe.add(filename)
tar_pipe.close()
conn.getresponse()

but that does not send any data...

I haven't used socket.makefile() in a blue age so this is untested, but
I'm guessing you need a *write* file (remember, sockets have two
directions). So try

.... fileobj=conn.sock.makefile("w")...

and let me know if that helps.

regards
Steve

 

[pbienst]

Thanks for the tip! It doesn't change anything, though, so I've
debugged this a little bit further. The problem seems to be that the
receiving end (wsgi server) does not see the end of the data:

socket = environ["wsgi.input"]
while True:
sys.stderr.write("before")
chunk = socket.read(4096)
sys.stderr.write("after")
if not chunk:
sys.stderr.write("done")
break
sys.stderr.write(chunk)

There is data from the tar file being printed, but in the end it hangs
in the 'before' statement, and never gets to 'done'. I tried flushing
the fileobj created by makefile(), but that doesn't seem to help.
There is also no environ["CONTENT_LENGTH"] that I can use to detect
the end of the stream.

I'm probably missing something basic here...

 

[r0g]

I would like to bundle up a number of files in a tar file and send it
over a HTTP connection, but I would like to do this without creating
the tar file on disk first.

Stringio lets you treat a strings as a files...

http://docs.python.org/library/stringio.html

Roger.

 

[Steve Holden]

Stringio lets you treat a strings as a files...

http://docs.python.org/library/stringio.html

Roger.

.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve

 

[Steve Holden]

Stringio lets you treat a strings as a files...

http://docs.python.org/library/stringio.html

Roger.

.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve

 

[Gabriel Genellina]

The problem seems to be that the
receiving end (wsgi server) does not see the end of the data:

socket = environ["wsgi.input"]
while True:
sys.stderr.write("before")
chunk = socket.read(4096)
sys.stderr.write("after")
if not chunk:
sys.stderr.write("done")
break
sys.stderr.write(chunk)

There is data from the tar file being printed, but in the end it hangs
in the 'before' statement, and never gets to 'done'. I tried flushing
the fileobj created by makefile(), but that doesn't seem to help.
There is also no environ["CONTENT_LENGTH"] that I can use to detect
the end of the stream.

I'm probably missing something basic here...

Either send a "Connection: Close" header (and close the connection at the
end), or a "Transfer-Encoding: chunked" header (see
http://en.wikipedia.org/wiki/Chunked_transfer_encoding )

 

[pbienst]

OK, thanks to the feedback from everyone I got the PUT from a client
to the WSGI server working.

I'm now trying to go the other way around: use a tar stream in one of
the functions in the WSGI server in order to send files to the client.
Problem is that the WSGI specs expects an iterator as return value for
streaming, whereas TarFile needs to write to a file obj.

Is there any way I can get these two to work together?

Peter

 

[Gabriel Genellina]

OK, thanks to the feedback from everyone I got the PUT from a client
to the WSGI server working.

I'm now trying to go the other way around: use a tar stream in one of
the functions in the WSGI server in order to send files to the client.
Problem is that the WSGI specs expects an iterator as return value for
streaming, whereas TarFile needs to write to a file obj.

Is there any way I can get these two to work together?

I haven't actually tried it, but aren't you looking for a pipe? (os.pipe()
+ os.fdopen())