Threading questions

[Antal Rutz]

Hi, All!

I'm new to threading. I have some design questions:
Task: I collect data and store them in an RDBMS (mysql or pgsql)

The question is how to do that with threading?
The data-collecting piece of the code runs in a thread.

1. Open the db, and each thread writes the result immediately.
(Sub-question: which is better: cursor object passed to the thread
or stored in a global var.)
2. Threads return data, which is written to DB after that.
3. Threads write to global variable, after 'join()' data is written.
(Can be used global (write-only) variables with threads at all?)
4. ?...

I think variable locking isn't an issue here because they are write-only.

Maybe I have fundamentaly misunderstood something...

Thanks

 

[F. GEIGER]

Just an idea: You could have n data collector threads, that all put their
results into a queue connected to 1 db thread, that stores the results into
the db.

Cheers
Franz GEIGER

 

[wittempj]

For threading I use usually this recipe as a base:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65448, I think
you can use it for your purpose as well.

 

[Magnus Lycka]

Hi, All!

I'm new to threading. I have some design questions:
Task: I collect data and store them in an RDBMS (mysql or pgsql)

The question is how to do that with threading?
The data-collecting piece of the code runs in a thread.

1. Open the db, and each thread writes the result immediately.
(Sub-question: which is better: cursor object passed to the thread
or stored in a global var.)
2. Threads return data, which is written to DB after that.
3. Threads write to global variable, after 'join()' data is written.
(Can be used global (write-only) variables with threads at all?)
4. ?...

I'm not sure why you need threading at all here, but I would avoid
both globals and passing cursors between threads. I don't think all
DB-API implementations can handle threading properly.

I suspect it might be useful to do data collection in one thread,
DB communication in another, and to use a Queue.Queue object to
pass data from data collection to DB insertion thread.

 

[Antal Rutz]

I'm not sure why you need threading at all here, but I would avoid
both globals and passing cursors between threads. I don't think all
DB-API implementations can handle threading properly.

I 'snmpwalk' routers and some are very slow, so it's better to spread
the routers among data-collecting threads.

I suspect it might be useful to do data collection in one thread,
DB communication in another, and to use a Queue.Queue object to
pass data from data collection to DB insertion thread.

Yes, I just figure out how the many-collecting-one-dbwriting-thread
design could be made. It doesn't seem to be too difficult.

Thanks for the hints!