Asyncio gather usage

Mar 25, 2024 ยท 2 min read

The asyncio module in Python provides powerful tools for writing asynchronous and concurrent code. One very useful function is asyncio.gather(), which allows you to simplify running multiple coroutines concurrently.

The key thing gather() does is take a list of coroutines or futures, execute them in parallel, and then return the results in a list in the original order. This saves you from having to manually coordinate waiting on multiple concurrent tasks.

Here's a quick example:

import asyncio

async def fetch_data(url):
    # Pretend we fetch data from URL
    return f"Downloaded from {url}"

urls = ["https://www.python1.org", "https://www.python2.org"]

async def main():
    results = await asyncio.gather(*[fetch_data(url) for url in urls])
    print(results)

asyncio.run(main())

This will fetch data from the two URLs concurrently and print out the results list when both are completed.

Some key points on using gather():

  • It helps avoid nested callbacks when coordinating multiple tasks
  • The order of results matches the order you passed to gather()
  • Errors in any of the tasks will cause the whole gather() call to fail
  • You can use it directly with coroutine functions or with futures
  • One common use case is kicking off I/O-bound work like multiple network requests concurrently:

    user_coros = [fetch_user_data(uid) for uid in user_ids] 
    user_data = await asyncio.gather(*user_coros)

    In summary, asyncio.gather() is immensely helpful for coordinating waiting on multiple asynchronous tasks to complete. It handles all the complexity so your code stays simple and linear.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
    ...

    X

    Don't leave just yet!

    Enter your email below to claim your free API key: