D
David
Hi list.
I have 2 questions about the Pythonicity of some algorithms. One
question is about generator usage, the other is about producer and
consumer threads
GENERATOR QUESTION
Usually for generators, the usage is like this:
for var in generator():
do_something(var)
I sometimes have cases where I want to use a generator, but I don't
want to stall the main program if the generator function doesn't have
any new data for me (maybe it's retrieving data from a device, but I
want to do other things with the same device if there isn't any data
yet).
So currently, some of my code looks something like this:
def get_serial_data():
while True:
block = request_a_packet_from_serial_device() # Get some raw data
parsed_list = parse_block(block) # Extract a list of records
from the packet
if parsed_list:
# The record contains 1 or more records
for record in parse_list:
yield record
else:
# Sometimes there aren't any records in the packet
yield None
def thread_function_for_handling_serial_device():
record_iter = iter(get_serial_data())
while True:
record = record_iter.next()
if record:
handle_record() # eg: add to a queue for the main thread to process
other_serial_logic1()
other_serial_logic2()
other_serial_logic3()
This works for me. But I'd like to know if this is considered
Pythonic, and if there are other, better ways of doing the above in
Python.
PRODUCER AND CONSUMER THREAD QUESTION
In some of my Python apps I use the producer/consumer model, where the
producer and consumer threads communicate through Queues.
Usually code (in the consumer thread function) looks something like this:
def consumer_thread():
while True:
record = some_queue.get()
process_record(record)
In some cases I'd like to do something unusual to threads consuming a
queue - interrupt them, make them terminate, etc. The problem is that
a lot of the time it's locking, waiting for a record from
some_queue.get()
A few things I've considered:
1) Add a timeout to some_queue.get(), so the loop has a chance to do
other things while waiting for new records. This is problematic
because most of the time there isn't other things for the thread to
do. Also, it won't be able to respond immediately to those events.
2) Add special values to the queue, the purpose of which is to notify
the consumers. This is problematic when you have multiple consumers.
You have to add the exact correct number of 'interrupt' objects to the
queue.
3) Make a custom thread-safe queue class, with an 'interrupt_get'
method. When your app calls queue.interrupt_get(), all threads
currently locking on a queue.get() will continue, but with a
GetInterrupted exception thrown.
Perhaps there is some other Python threading idiom I should be using
in this case. Any suggestions?
David.
I have 2 questions about the Pythonicity of some algorithms. One
question is about generator usage, the other is about producer and
consumer threads
GENERATOR QUESTION
Usually for generators, the usage is like this:
for var in generator():
do_something(var)
I sometimes have cases where I want to use a generator, but I don't
want to stall the main program if the generator function doesn't have
any new data for me (maybe it's retrieving data from a device, but I
want to do other things with the same device if there isn't any data
yet).
So currently, some of my code looks something like this:
def get_serial_data():
while True:
block = request_a_packet_from_serial_device() # Get some raw data
parsed_list = parse_block(block) # Extract a list of records
from the packet
if parsed_list:
# The record contains 1 or more records
for record in parse_list:
yield record
else:
# Sometimes there aren't any records in the packet
yield None
def thread_function_for_handling_serial_device():
record_iter = iter(get_serial_data())
while True:
record = record_iter.next()
if record:
handle_record() # eg: add to a queue for the main thread to process
other_serial_logic1()
other_serial_logic2()
other_serial_logic3()
This works for me. But I'd like to know if this is considered
Pythonic, and if there are other, better ways of doing the above in
Python.
PRODUCER AND CONSUMER THREAD QUESTION
In some of my Python apps I use the producer/consumer model, where the
producer and consumer threads communicate through Queues.
Usually code (in the consumer thread function) looks something like this:
def consumer_thread():
while True:
record = some_queue.get()
process_record(record)
In some cases I'd like to do something unusual to threads consuming a
queue - interrupt them, make them terminate, etc. The problem is that
a lot of the time it's locking, waiting for a record from
some_queue.get()
A few things I've considered:
1) Add a timeout to some_queue.get(), so the loop has a chance to do
other things while waiting for new records. This is problematic
because most of the time there isn't other things for the thread to
do. Also, it won't be able to respond immediately to those events.
2) Add special values to the queue, the purpose of which is to notify
the consumers. This is problematic when you have multiple consumers.
You have to add the exact correct number of 'interrupt' objects to the
queue.
3) Make a custom thread-safe queue class, with an 'interrupt_get'
method. When your app calls queue.interrupt_get(), all threads
currently locking on a queue.get() will continue, but with a
GetInterrupted exception thrown.
Perhaps there is some other Python threading idiom I should be using
in this case. Any suggestions?
David.