J
Johan Llewellyn
hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop. the examples I have found seem to assume it is known ahead of time how many items need to be processed.
specifically, I am reading from an external queue. I currently process items one at a time until the queue is empty. I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done. ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously.
I've attached a script that captures what I am doing now. unfortunately, the external queue object is not publicly accessible andI'm not quite sure how to set up a local object that would support testing.. any suggestions would be most welcome.
thanks,
Johan
specifically, I am reading from an external queue. I currently process items one at a time until the queue is empty. I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done. ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously.
I've attached a script that captures what I am doing now. unfortunately, the external queue object is not publicly accessible andI'm not quite sure how to set up a local object that would support testing.. any suggestions would be most welcome.
thanks,
Johan