J
Joseph L. Casale
I have a portion of code I need to speed up, there are 3 api calls to an external system
where the first enumerates a large collection of objects I then loop through and perform
two additional api calls each. The first call is instant, the second and third per object are
very slow. Currently after accumulating all the data I write the relevant data into a database.
I have the ability to hold all this in memory and dump it once fully accumulated, so performing
the second and third call in parallel with fixed batches would be great,
I took a look at coroutines and some skeleton code worked fine, but I am not sure how to
perform the acquisition in fixed groups like I might for example with multiprocessing and
a pool of workers.
Anyone done something like this or have an opinion?
Thanks,
jlc
where the first enumerates a large collection of objects I then loop through and perform
two additional api calls each. The first call is instant, the second and third per object are
very slow. Currently after accumulating all the data I write the relevant data into a database.
I have the ability to hold all this in memory and dump it once fully accumulated, so performing
the second and third call in parallel with fixed batches would be great,
I took a look at coroutines and some skeleton code worked fine, but I am not sure how to
perform the acquisition in fixed groups like I might for example with multiprocessing and
a pool of workers.
Anyone done something like this or have an opinion?
Thanks,
jlc