Newbie prob: How to write a file with 3 threads?

E

est

I need to write a file using 3 threads simutaniously, e.g. Thread 1
write the first byte of test.bin with an "a", second thread write the
second byte "b", third thread write the third byte "c". Anyone could
give a little example on how to do that?

I have my code, but it makes python intepreter crash everytime on my
Vista.
 
M

Marc 'BlackJack' Rintsch

I need to write a file using 3 threads simutaniously, e.g. Thread 1
write the first byte of test.bin with an "a", second thread write the
second byte "b", third thread write the third byte "c". Anyone could
give a little example on how to do that?

Simplest solution is: don't do that. Write from one thread and send the
date from the other threads via a `Queue.Queue` to the writing thread.
Send the number of the thread with the data so the writer thread knows in
which order the data has to be written.
I have my code, but it makes python intepreter crash everytime on my
Vista.

Show minimal (non-)working code, tell us the exception plus traceback and
explain "crash".

Ciao,
Marc 'BlackJack' Rintsch
 
M

MRAB

Simplest solution is: don't do that. Write from one thread and send the
date from the other threads via a `Queue.Queue` to the writing thread.
Send the number of the thread with the data so the writer thread knows in
which order the data has to be written.
[snip]
Or have a `Queue.Queue` for each source thread and make the writer
thread read from each in turn.
 
E

est

Simplest solution is: don't do that. Write from one thread and send the
date from the other threads via a `Queue.Queue` to the writing thread.
Send the number of the thread with the data so the writer thread knows in
which order the data has to be written.

[snip]
Or have a `Queue.Queue` for each source thread and make the writer
thread read from each in turn.


I'll try Queue.Queue, thank you. I didn't expect that multithread
write a file is so troublesome
 
G

Gary Herron

est said:
I need to write a file using 3 threads simutaniously, e.g. Thread 1
write the first byte of test.bin with an "a", second thread write the
second byte "b", third thread write the third byte "c". Anyone could
give a little example on how to do that?

Simplest solution is: don't do that. Write from one thread and send the
date from the other threads via a `Queue.Queue` to the writing thread.
Send the number of the thread with the data so the writer thread knows in
which order the data has to be written.
[snip]
Or have a `Queue.Queue` for each source thread and make the writer
thread read from each in turn.


I'll try Queue.Queue, thank you. I didn't expect that multithread
write a file is so troublesome

As a general rule, *ALL* multithread operations are at least that
troublesome, and most are far more so.

Pessimistically-yours,
Gary Herron
 
D

Dennis Lee Bieber

I'll try Queue.Queue, thank you. I didn't expect that multithread
write a file is so troublesome

Multiple writers to a file, threaded or not (ie, multiple processes,
hosts on network, etc.), is always a pain unless one can ensure that all
writers use a common locking mode to prevent overlapping calls. If all
the writers are internal to one program, one can implement internal
locks -- if external one needs OS support for multi-process locking (VMS
common event flag clusters, for example). Easier to dedicate one
(internal) writer and use the naturally locked Queue to control access.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 
E

est

Multiple writers to a file, threaded or not (ie, multiple processes,
hosts on network, etc.), is always a pain unless one can ensure that all
writers use a common locking mode to prevent overlapping calls. If all
the writers are internal to one program, one can implement internal
locks -- if external one needs OS support for multi-process locking (VMS
common event flag clusters, for example). Easier to dedicate one
(internal) writer and use the naturally locked Queue to control access.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

I guess I will write multiple files, one thread one file, and after
all threads terminates, I combine the files. That's a cheaper
solution, I think.
 
D

Dennis Lee Bieber

I guess I will write multiple files, one thread one file, and after
all threads terminates, I combine the files. That's a cheaper
solution, I think.

Given your first description:
I need to write a file using 3 threads simutaniously, e.g. Thread 1
write the first byte of test.bin with an "a", second thread write the
second byte "b", third thread write the third byte "c". Anyone could
give a little example on how to do that?

.... any other solution may not have worked anyway. That is, if you
really expect the three threads to interleave their output as
abcabcabcabc. If threading, you have no assurance that any single thread
will follow any other during a task switch. It all depends upon where a
task switch takes place.

But then, your example isn't too clear of what you really are
producing for output. If, instead of "bytes", you meant that each thread
was writing logging information, the solution would be to use the
logging module -- so far as I know, the logging module /does/ perform
the needed access locking.

OTOH, if you really mean to have three separate byte producers,
interleaving output, the only safe way would be to have /each/ have an
output Queue, and a fourth thread doing the writing using a loop of the
form:

while True:
a = aQueue.get()
fout.write(a)
b = bQueue.get()
fout.write(b)
c = cQueue.get()
fout.write(c)

Using the separate queues means that the writer WILL wait until the
next producer in the interleave has produced its data. Of course, this
structure has the drawback that all producers must produce the same
amount of data -- otherwise it blocks forever on the queue from the
producer has stopped generating data.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 
E

est

Given your first description:


... any other solution may not have worked anyway. That is, if you
really expect the three threads to interleave their output as
abcabcabcabc. If threading, you have no assurance that any single thread
will follow any other during a task switch. It all depends upon where a
task switch takes place.

But then, your example isn't too clear of what you really are
producing for output. If, instead of "bytes", you meant that each thread
was writing logging information, the solution would be to use the
logging module -- so far as I know, the logging module /does/ perform
the needed access locking.

OTOH, if you really mean to have three separate byte producers,
interleaving output, the only safe way would be to have /each/ have an
output Queue, and a fourth thread doing the writing using a loop of the
form:

while True:
a = aQueue.get()
fout.write(a)
b = bQueue.get()
fout.write(b)
c = cQueue.get()
fout.write(c)

Using the separate queues means that the writer WILL wait until the
next producer in the interleave has produced its data. Of course, this
structure has the drawback that all producers must produce the same
amount of data -- otherwise it blocks forever on the queue from the
producer has stopped generating data.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

I'd like to say VERY VERY VERY thank you for your detailed
information, that's a lot encourage for a beginner. In fact I am
writing a multi thread download ultility, and I found that very hard
for me. Can you recommand any sample code where I can start with? I
hope I can catch up with everyone here, I'll try my best learning
python. Thank you again.
 
D

Dennis Lee Bieber

I'd like to say VERY VERY VERY thank you for your detailed
information, that's a lot encourage for a beginner. In fact I am
writing a multi thread download ultility, and I found that very hard
for me. Can you recommand any sample code where I can start with? I
hope I can catch up with everyone here, I'll try my best learning
python. Thank you again.

Starting with a multithreaded program and IPC* is not what I'd
consider the best way to learn Python... <G>

The core syntax (language reference manual) and some key library
sections (the parts on lists, strings, dictionaries... sys and os
modules); then expand outward (time module, maybe the assorted web
modules).

Granted, /I/ can think in threaded interactions much easier than the
dispatch/event model used by asyncore/Twisted (and GUI toolkits), but
I'd still not recommend threading to anyone until they're comfortable
with base Python. Maybe this comes out of having 30 years of programming
experience in various languages (okay, I'm counting my college years too
<G>) and only easing into parallelism over the last 20 (via I/O
interrupt handlers [VMS ASTs]). Ada and Python were the first languages
I ever encountered with (relatively) easy tasking ability -- and I've
only done toy programs in Ada.

No idea of sample code, especially as you don't have a firm
requirement statement describing, at the least, the inputs, outputs, and
protocols to be used.


* If the acronym is unfamiliar to you, I'd suggest spending time with
some foundation textbooks. Unfortunately, CS textbooks have changed a
lot since the 70s-80s. Back then you could find books devoted to the
theoretical concepts behind concurrent programming (beyond the dining
philosophers <G>); I'm not familiar with the current crop -- they seem
to be dedicated to specific OS platforms. If this were the Ada group,
I'd suggest "Concurrency in Ada"; a whole fat book just on Ada's tasking
model.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 
B

Bjoern Schliessmann

est said:
I'd like to say VERY VERY VERY thank you for your detailed
information, that's a lot encourage for a beginner. In fact I am
writing a multi thread download ultility, and I found that very
hard for me.

You don't need any system threads for multiple download threads.
Since network connections are buffered and much slower than CPU
cycles, you can also use event based mechanisms. (I've written 30+
connection servers like this already.)

The general pseudocode strategy is

while True:
waitForEvents()
event = getEvent()
processEvent(event)
# in here, you inspect the event,
# e. g. from which connection it is,
# and get or write data

The only drawback to event based programming is that processor
cycle "rationing" is decided in your code and not by the OS, which
can make difficulties with really long calculations. But if these
problems arise you usually spawn a seperate worker thread :)
Can you recommand any sample code where I can start
with?

If you choose event based programming, definitely have a look at
Twisted. It lets you create multiple connection clients and servers
in few code lines without synchronisation hassle.

http://twistedmatrix.com/projects/core/documentation/howto/
http://twistedmatrix.com/projects/core/documentation/howto/clients.html

(BTW, don't get scared by "Deferred"s, I never explicitly used them
in my code ... though they often work in the background)

Regards,


Björn
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top