Making Async HTTP Requests in Python

Feb 3, 2024 ยท 2 min read

Python's requests library makes it easy to make synchronous HTTP requests in your code. But in async environments, like asyncio, you'll want to use an async HTTP client instead. Here's how to make async HTTP requests in Python.

Why Async HTTP?

Synchronous requests block the execution of code until a response is received. In async code this wastes CPU cycles waiting idly. Async requests allow other code to run while a response is pending.

For I/O bound tasks like HTTP requests, this allows full utilization of your CPU cores. This means better scalability and performance.

Enter aiohttp

The aiohttp library is the async counterpart to requests. To install:

pip install aiohttp

The API is similar to requests. Here's a GET request:

import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://api.example.com') as response:
            print(response.status)
            print(await response.text())

asyncio.run(main())

Async contexts like ClientSession and response objects must be opened and awaited asynchronously.

Client Session

Reusing a ClientSession brings performance benefits like connection pooling and session reuse. Create one session and reuse it for multiple requests.

Handling Responses

The response object has async methods like json() and text() to get the response body. Always await these methods.

Exceptions

Any exceptions in async code bubble up to the top try/except block. So wrap your requests in try/except to catch errors.

Going Further

That covers the basics of async HTTP with aiohttp! Some next topics:

  • Making concurrent requests with asyncio.gather()
  • Response timeouts
  • JSON handling with response.json()
  • POST requests and request data
  • Authentication
  • The aiohttp docs have many more examples too. Give it a try on your next async Python project!

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
    ...

    X

    Don't leave just yet!

    Enter your email below to claim your free API key: