Asyncio gathering task results

The asyncio.gather() function is very useful when you need to launch multiple coroutines concurrently and wait for all of their results at once. However, understanding exactly how it works can be tricky for asyncio beginners.

In this guide, we'll walk through real examples to demystify asyncio.gather() and help you use it effectively in your own asyncio code.

The Problem asyncio.gather Solves

Imagine we need to fetch two web pages concurrently using the aiohttp library:

import asyncio
import aiohttp

async def fetch_page(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

asyncio.run(fetch_page('https://example.com')) 
asyncio.run(fetch_page('https://python.org'))

This launches each coroutine serially, one waiting for the other to finish.

But we want them to run concurrently and process both results when available. That's where gather comes in!

Gather Basics

asyncio.gather() runs coroutines concurrently and blocks until they have all completed:

import asyncio

async def coro1():
    return 1

async def coro2():
    return 2 

async def main():
    results = await asyncio.gather(
        coro1(), 
        coro2()
    )
    print(results)

asyncio.run(main())

# Prints [1, 2]

The key things to know about gather:

This handles our use case of waiting for multiple fetches nicely!

import asyncio
import aiohttp

async def fetch(url):
    async with aiohttp.request('GET', url) as response:
        data = await response.text() 
        return data

async def main():
    urls = ['https://example.com', 'https://python.org']
    results = await asyncio.gather(*[fetch(url) for url in urls])
    print(results[0][:100]) # Prints first 100 chars of first page
    print(results[1][:100]) # Prints first 100 chars of second page
    
asyncio.run(main())

Now both fetches run concurrently, yet we can process their responses sequentially.

Gathering Tasks That Raise Exceptions

As mentioned earlier, gather() cancels all pending tasks as soon as one raises an exception.

Let's see this in action:

import asyncio

async def bad_coro():
    raise Exception("Oops!")

async def main():
    try:
        await asyncio.gather(
            bad_coro(),
            asyncio.sleep(1), 
        )
    except Exception:
        print("Hit exception!")

asyncio.run(main()) 
# Prints "Hit exception!"

Here, our second task gets cancelled even though it wasn't the one that failed.

So in a real app, we'd typically wrap each task in its own try/except block to isolate failures:

async def main():
    results = await asyncio.gather(
        try_task1(), 
        try_task2(),
        try_task3(),
    )

async def try_task1():
    try:
        return await real_task1() 
    except Exception:
        return None

This keeps failures isolated and our other tasks can complete.

Gather Use Cases

Some examples where gather() shines:

It's a versatile tool for coordinating independent async work!

Key Takeaways

I hope these examples help demystify gather() and illuminate how you can apply it. Asyncio has a learning curve, but practice makes perfect!