creating tar file and streaming it over HTTP?

Discussion in 'Python' started by pbienst, Jan 6, 2010.

  1. pbienst

    pbienst Guest

    I would like to bundle up a number of files in a tar file and send it
    over a HTTP connection, but I would like to do this without creating
    the tar file on disk first.

    I know I can get tarfile to output to a stream by doing something like

    tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)

    However, I can't figure out which file object to use to send this over
    a HTTP connection.

    I tried

    conn = httplib.HTTPConnection(hostname, port)
    conn.putrequest("PUT", "/client/files")
    conn.endheaders()
    tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
    for filename in filenames:
    tar_pipe.add(filename)
    tar_pipe.close()
    conn.getresponse()

    but that does not send any data...
     
    pbienst, Jan 6, 2010
    #1
    1. Advertising

  2. pbienst

    Steve Holden Guest

    pbienst wrote:
    > I would like to bundle up a number of files in a tar file and send it
    > over a HTTP connection, but I would like to do this without creating
    > the tar file on disk first.
    >
    > I know I can get tarfile to output to a stream by doing something like
    >
    > tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)
    >
    > However, I can't figure out which file object to use to send this over
    > a HTTP connection.
    >
    > I tried
    >
    > conn = httplib.HTTPConnection(hostname, port)
    > conn.putrequest("PUT", "/client/files")
    > conn.endheaders()
    > tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
    > for filename in filenames:
    > tar_pipe.add(filename)
    > tar_pipe.close()
    > conn.getresponse()
    >
    > but that does not send any data...
    >

    I haven't used socket.makefile() in a blue age so this is untested, but
    I'm guessing you need a *write* file (remember, sockets have two
    directions). So try

    .... fileobj=conn.sock.makefile("w")...

    and let me know if that helps.

    regards
    Steve

    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
    Holden Web LLC http://www.holdenweb.com/
    UPCOMING EVENTS: http://holdenweb.eventbrite.com/
     
    Steve Holden, Jan 6, 2010
    #2
    1. Advertising

  3. pbienst

    pbienst Guest

    Thanks for the tip! It doesn't change anything, though, so I've
    debugged this a little bit further. The problem seems to be that the
    receiving end (wsgi server) does not see the end of the data:

    socket = environ["wsgi.input"]
    while True:
    sys.stderr.write("before")
    chunk = socket.read(4096)
    sys.stderr.write("after")
    if not chunk:
    sys.stderr.write("done")
    break
    sys.stderr.write(chunk)

    There is data from the tar file being printed, but in the end it hangs
    in the 'before' statement, and never gets to 'done'. I tried flushing
    the fileobj created by makefile(), but that doesn't seem to help.
    There is also no environ["CONTENT_LENGTH"] that I can use to detect
    the end of the stream.

    I'm probably missing something basic here...
     
    pbienst, Jan 6, 2010
    #3
  4. pbienst

    r0g Guest

    pbienst wrote:
    > I would like to bundle up a number of files in a tar file and send it
    > over a HTTP connection, but I would like to do this without creating
    > the tar file on disk first.
    >


    Stringio lets you treat a strings as a files...

    http://docs.python.org/library/stringio.html

    Roger.
     
    r0g, Jan 6, 2010
    #4
  5. pbienst

    Steve Holden Guest

    r0g wrote:
    > pbienst wrote:
    >> I would like to bundle up a number of files in a tar file and send it
    >> over a HTTP connection, but I would like to do this without creating
    >> the tar file on disk first.
    >>

    >
    > Stringio lets you treat a strings as a files...
    >
    > http://docs.python.org/library/stringio.html
    >
    > Roger.


    .... though that does mean that the whole tar file has to be created
    before it can be sent, I suspect. If it will comfortably fit into memory
    that shouldn't matter, of course.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
    Holden Web LLC http://www.holdenweb.com/
    UPCOMING EVENTS: http://holdenweb.eventbrite.com/
     
    Steve Holden, Jan 6, 2010
    #5
  6. pbienst

    Steve Holden Guest

    r0g wrote:
    > pbienst wrote:
    >> I would like to bundle up a number of files in a tar file and send it
    >> over a HTTP connection, but I would like to do this without creating
    >> the tar file on disk first.
    >>

    >
    > Stringio lets you treat a strings as a files...
    >
    > http://docs.python.org/library/stringio.html
    >
    > Roger.


    .... though that does mean that the whole tar file has to be created
    before it can be sent, I suspect. If it will comfortably fit into memory
    that shouldn't matter, of course.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
    Holden Web LLC http://www.holdenweb.com/
    UPCOMING EVENTS: http://holdenweb.eventbrite.com/
     
    Steve Holden, Jan 6, 2010
    #6
  7. En Wed, 06 Jan 2010 13:39:08 -0300, pbienst <>
    escribió:

    > The problem seems to be that the
    > receiving end (wsgi server) does not see the end of the data:
    >
    > socket = environ["wsgi.input"]
    > while True:
    > sys.stderr.write("before")
    > chunk = socket.read(4096)
    > sys.stderr.write("after")
    > if not chunk:
    > sys.stderr.write("done")
    > break
    > sys.stderr.write(chunk)
    >
    > There is data from the tar file being printed, but in the end it hangs
    > in the 'before' statement, and never gets to 'done'. I tried flushing
    > the fileobj created by makefile(), but that doesn't seem to help.
    > There is also no environ["CONTENT_LENGTH"] that I can use to detect
    > the end of the stream.
    >
    > I'm probably missing something basic here...


    Either send a "Connection: Close" header (and close the connection at the
    end), or a "Transfer-Encoding: chunked" header (see
    http://en.wikipedia.org/wiki/Chunked_transfer_encoding )

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jan 7, 2010
    #7
  8. pbienst

    pbienst Guest

    OK, thanks to the feedback from everyone I got the PUT from a client
    to the WSGI server working.

    I'm now trying to go the other way around: use a tar stream in one of
    the functions in the WSGI server in order to send files to the client.
    Problem is that the WSGI specs expects an iterator as return value for
    streaming, whereas TarFile needs to write to a file obj.

    Is there any way I can get these two to work together?

    Peter
     
    pbienst, Jan 9, 2010
    #8
  9. En Sat, 09 Jan 2010 10:19:00 -0300, pbienst <>
    escribió:

    > OK, thanks to the feedback from everyone I got the PUT from a client
    > to the WSGI server working.
    >
    > I'm now trying to go the other way around: use a tar stream in one of
    > the functions in the WSGI server in order to send files to the client.
    > Problem is that the WSGI specs expects an iterator as return value for
    > streaming, whereas TarFile needs to write to a file obj.
    >
    > Is there any way I can get these two to work together?


    I haven't actually tried it, but aren't you looking for a pipe? (os.pipe()
    + os.fdopen())

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jan 13, 2010
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Claudio Grondi
    Replies:
    4
    Views:
    589
    Claudio Grondi
    Aug 20, 2005
  2. Replies:
    2
    Views:
    453
    Michael Hoffman
    Apr 24, 2007
  3. Ray Van Dolson
    Replies:
    0
    Views:
    347
    Ray Van Dolson
    Sep 23, 2009
  4. Ray Van Dolson
    Replies:
    0
    Views:
    854
    Ray Van Dolson
    Sep 25, 2009
  5. benoit Guyon
    Replies:
    2
    Views:
    237
    benoit Guyon
    Jul 26, 2005
Loading...

Share This Page