help on file storage for split multi part download

[coolman.guron]

HI everyone on this group!

i am want to write a split part downloading software for use in some
projects. (something like a download accelerator)

what i think i need is ( my brain is totally exausted at this moment
so pls ignore any typos i make)

storage class which can write the file splits that are currently being
downloaded to the disk. this is exactly what other download
accelerators do, i guess.

can this be done using the python file class?? i am pretty good at
handling file uploads (at server end) this is the first time i have to
think the other way round.

any code samples would be greatly appreciated.

thanks
binary-j
django web developer

 

[Gabriel Genellina]

storage class which can write the file splits that are currently being
downloaded to the disk. this is exactly what other download
accelerators do, i guess.

can this be done using the python file class?? i am pretty good at
handling file uploads (at server end) this is the first time i have to
think the other way round.

Uh, unless I misundersand you, a standard file object is enough. First
create a file with the required size (open(...,'wb'), seek(n-1),
write(chr(0))). For each downloaded chunk you have to know its position in
the file; then just seek() and write() it.

 

[coolman.guron]

Uh, unless I misundersand you, a standard file object is enough. First
create a file with the required size (open(...,'wb'), seek(n-1),
write(chr(0))). For each downloaded chunk you have to know its position in
the file; then just seek() and write() it.

well yes its possible to use the file class that way.

BUT the thing thats going in my mind is thread safety. i plan to start
each part of the file download in a different thread. and then when
each thread had downloaded more than 100kb (or eof or boundary
reached) write the buffer to the disk. can this be achieved using
mutex ? i have never shared objects between threads.

is there a way to write this without using threads at all ???

 

[Gabriel Genellina]

BUT the thing thats going in my mind is thread safety. i plan to start
each part of the file download in a different thread. and then when
each thread had downloaded more than 100kb (or eof or boundary
reached) write the buffer to the disk. can this be achieved using
mutex ? i have never shared objects between threads.

Use a different (single) thread to write the file; the others put write
requests on a Queue.queue object, and the writer just gets the requests
and processes them.

is there a way to write this without using threads at all ???

Using asyncore, and perhaps the Twisted framework.

 

[coolman.guron]

En Fri, 07 Mar 2008 04:16:42 -0200, <[email protected]> escribi�:





Use a different (single) thread to write the file; the others put write
requests on a Queue.queue object, and the writer just gets the requests
and processes them.


Using asyncore, and perhaps the Twisted framework.

asyncore is basically a server thing right?

 

[Gabriel Genellina]

asyncore is basically a server thing right?

asyncore is usually used to build servers, because in a server you want to
handle many requests with few resources, but you can use it to write a
client too.
Here is an example: http://effbot.org/zone/asyncore-ftp-client.htm

How many files and how many simultaneous connections do you plan to
handle? Using multiple threads to download and a single thread to write,
connected thru a queue, looks like the "simplest thing that probably
works" to me unless you have other constraints.