How to get unordered results when using async with?

[karolzlot]

Description

I have a lot of async jobs (>10000).

I want to process results as soon as they are available, so I am using:

async for result in pool.map(run_one_job, jobs_data):

However, some of them take way more time than others. So it happens that many results are already ready, but the above for loop is blocked and is waiting because it can only give results in order.

I would like to have something like .map_unordered()

It seems to me that it is currently impossible to do it, and it would require implementing this functionality in aiomultiprocess.

Details

• OS: Ubuntu on Win10 WSL
• Python version: 3.9
• aiomultiprocess version: 0.9.0
• Can you repro on master? ---
• Can you repro in a clean virtualenv? ---

[amyreese]

I think this would be a relatively easy addition to pool.Pool. I'd ideally like to see it called imap() to match the standard multiprocessing module, but would need to implement an unordered version of PoolResult and/or Pool.results(). Happy to give guidance if someone wants to work on this before I have the time.

[karolzlot]

I think if we want to match multiprocessing module naming then it should be imap() and imap_unordered().

source: https://stackoverflow.com/a/26521507/8896457

[amyreese]

Hmm, you're right. I misremembered what imap actually meant in this context. map_unordered does seem to be a better fit in that case, given that aiomultiprocess doesn't really do anything "lazy" in processing tasks/results.

[rahulkumawat1]

were you able to get this behavior from other library or any other way? @karolzlot

[karolzlot]

I think I just added results to shared queue, a and processed from it in another process. You can find this solution in another issue in this repo.