sending a file chunk by chunk instead as a whole to a web server

Discussion in 'Python' started by Sanjeeb, Aug 2, 2010.

  1. Sanjeeb

    Sanjeeb Guest

    Hi,
    I have a web client which send a file to a server as multipart form
    data, the sending of data is from
    http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.

    I dont want to open the whole file to memory(at cliend end) and then
    send, i just want to send part by part, say chunk of 1024 bytes to the
    server and then assemble at the server end.

    Could some one suggest what would be the best way to do this?

    Regards
    -Sanjeeb
     
    Sanjeeb, Aug 2, 2010
    #1
    1. Advertising

  2. Sanjeeb

    Sanjeeb Guest

    On Aug 2, 11:52 am, Sanjeeb <> wrote:
    > Hi,
    > I have a web client which send a file to a server as multipart form
    > data, the sending of data is fromhttp://code.activestate.com/recipes/146306-http-client-to-post-using-....
    >
    > I dont want to open the whole file to memory(at cliend end) and then
    > send, i just want to send part by part, say chunk of 1024 bytes to the
    > server and then assemble at the server end.
    >
    > Could some one suggest what would be the best way to do this?
    >
    > Regards
    > -Sanjeeb


    Hey any taker for this question???
     
    Sanjeeb, Aug 3, 2010
    #2
    1. Advertising

  3. On Mon, Aug 2, 2010 at 12:22 PM, Sanjeeb <> wrote:
    > Hi,
    > I have a web client which send a file to a server as multipart form
    > data, the sending of data is from
    > http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.
    >
    > I dont want to open the whole file to memory(at cliend end) and then
    > send, i just want to send part by part, say chunk of 1024 bytes to the
    > server and then assemble at the server end.
    >
    > Could some one suggest what would be the best way to do this?
    >


    There's no reason why sending the whole file implies reading the whole
    file into memory at one time. You can just read your desired chunk
    size and send it, then read the next chunk, and so on. You might have
    to first find the total size to calculate what to set Content-Length
    to.

    ISTR questions of this nature have been asked in the past on this list.

    --
    regards,
    kushal
     
    Kushal Kumaran, Aug 3, 2010
    #3
  4. Sanjeeb

    Ryan Kelly Guest

    On Tue, 2010-08-03 at 10:45 +0530, Kushal Kumaran wrote:
    > On Mon, Aug 2, 2010 at 12:22 PM, Sanjeeb <> wrote:
    > > Hi,
    > > I have a web client which send a file to a server as multipart form
    > > data, the sending of data is from
    > > http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.
    > >
    > > I dont want to open the whole file to memory(at cliend end) and then
    > > send, i just want to send part by part, say chunk of 1024 bytes to the
    > > server and then assemble at the server end.
    > >
    > > Could some one suggest what would be the best way to do this?
    > >

    >
    > There's no reason why sending the whole file implies reading the whole
    > file into memory at one time. You can just read your desired chunk
    > size and send it, then read the next chunk, and so on. You might have
    > to first find the total size to calculate what to set Content-Length
    > to.


    More concretely, you would restructure the encode_multipart_formdata()
    function as a generator, yielding chunks of data to send one at a time.
    Something like this:

    def post_multipart(host,selector,fields,files):
    ...snip...
    h.endheaders()
    for chunk in encode_multipart_formdata_chunks(fields,files):
    h.send(chunk)
    errcode, errmsg, headers = h.getreply()
    return h.file.read()


    def encode_multipart_formdata(fields,files):
    BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
    CRLF = '\r\n'
    for (key, value) in fields:
    yield '--' + BOUNDARY
    yield 'Content-Disposition: form-data; name="%s"' % key
    yield ''
    yield value
    for (key, filename, value) in files:
    yield '--' + BOUNDARY
    yield 'Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)
    ...etc...
    ...etc...

    There are many improvements to make, but this should get you started.
    For example, you'll need to calculate the total content-length rather
    than just calling len(body) to obtain it. That's left as an exercise to
    the reader.


    Ryan


    --
    Ryan Kelly
    http://www.rfk.id.au | This message is digitally signed. Please visit
    | http://www.rfk.id.au/ramblings/gpg/ for details


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.10 (GNU/Linux)

    iEYEABECAAYFAkxXrpwACgkQfI5S64uP50q+fACgzcvVbFuG25WI1B/9Daqk+Zvp
    LVIAoMbwFNlvrPsz5vZqwjR8s8P5FHhY
    =rhPT
    -----END PGP SIGNATURE-----
     
    Ryan Kelly, Aug 3, 2010
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ray Thomas
    Replies:
    1
    Views:
    3,277
    Ray Thomas
    Jul 21, 2004
  2. Jane Austine
    Replies:
    14
    Views:
    791
    Dennis Lee Bieber
    Oct 9, 2004
  3. \A_Michigan_User\
    Replies:
    2
    Views:
    898
    \A_Michigan_User\
    Aug 21, 2006
  4. Replies:
    0
    Views:
    1,377
  5. Homer J. Simpson
    Replies:
    1
    Views:
    480
    =?Utf-8?B?TGFkaXNsYXYgTXJua2E=?=
    Aug 15, 2007
Loading...

Share This Page