Python's asyncio library provides infrastructure for writing asynchronous code using the async/await syntax. Meanwhile, the multiprocessing module allows spawning processes to leverage multiple CPUs for parallel execution. Can these tools be combined?
The short answer is yes, with some care around passing data between async code and multiprocessing.
Why Combine AsyncIO and Multiprocessing?
There are a few potential benefits to using AsyncIO and multiprocessing together in Python:
Passing Data Between Processes
The main catch with mixing AsyncIO and multiprocessing is that concurrent data structures like Python queues are not compatible across the boundary.
The safest approach is to use multiprocessing queues, pipes or shared memory to pass data between the async event loop and processes. For example:
import asyncio
from multiprocessing import Queue
queue = Queue()
async def async_worker(queue):
data = await get_data()
queue.put(data)
def mp_worker(queue):
data = queue.get()
process(data)So in summary - AsyncIO and multiprocessing absolutely can combine forces in Python for improved performance, resource utilization and cleaner code. Just be careful in how data flows between the two worlds.