Making the Most of aiohttp's TCPConnector for Asynchronous HTTP Requests

Mar 3, 2024 ยท 3 min read

The aiohttp library is a powerful tool for making asynchronous HTTP requests in Python. Under the hood, it utilizes a TCPConnector to handle connecting to HTTP servers. Properly configuring the TCPConnector is key to getting the most out of aiohttp.

Why Asynchronous HTTP Requests?

Synchronous HTTP requests can block the execution of your Python code while waiting for a response. This wastes CPU cycles and hinders scalability.

Asynchronous requests allow your code to do other work while awaiting responses. This allows you to make requests concurrently and scale to handle more traffic.

TCPConnector Basics

The TCPConnector handles the nitty-gritty details of managing connections to HTTP servers:

import aiohttp

connector = aiohttp.TCPConnector()

The default connector is ready to go, but tweaking its parameters can improve performance:

  • limit - Max number of simultaneous connections (default 100)
  • limit_per_host - Max connections per host (default 0 = unlimited)
  • enable_cleanup_closed - Close expired connections in background (default True)
  • Managing Connection Limits

    The limit and limit_per_host parameters prevent you from overwhelming a server with connections:

    connector = aiohttp.TCPConnector(

    This allows 30 connections total, 10 per host. The default 100 total/0 per host is often too aggressive.

    Start conservative and increase limits if you need higher concurrency. Monitoring for throttling/errors can help tuning.

    Reusing Connections

    Opening new connections is slow. aiohttp will reuse connections by default:

    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get(''):
            # Reuses this connection
            async with session.get(''):  

    Reuse saves time but can limit concurrency due to waiting for the shared connection.

    Connection Pooling

    aiohttp maintains a pool of open connections to each host for reuse. Tuning these pool sizes is key for efficiency.

    The limit_per_host parameter controls the maximum pool size per host. Set it higher to allow more reuse. Set it lower to enable more parallelism across hosts.

    There is also a connection_timeout parameter (default 5 minutes). It determines when unused pooled connections will be closed. Keep it short to free up resources.

    Cleaning Up Closed Connections

    Sometimes the server or network will close a connection unexpectedly.

    By default aiohttp will detect these closed connections and remove them from the pool automatically.

    This prevents reuse of closed connections, but has a performance cost. If experiencing high load, consider disabling:

    connector = aiohttp.TCPConnector(

    Just be aware that reused closed connections will fail on next use.

    Tuning for Infrastructure

    Consider reducing limit and limit_per_host if targeting a small server or VM to avoid resource exhaustion.

    For robust infrastructure like load balanced services, increasing limits can improve throughput.

    Monitor load, errors, and latency when tuning pools. Find the sweet spot for your infrastructure.

    Troubleshooting Issues

    Enable aiohttp logging to help diagnose connection issues:

    import logging

    This will log debug info like connections opened/closed, retries, and errors.

    Common issues include hitting limits, reuse of closed connections, and resource exhaustion. Tune based on debug logging.

    Other Optimizations

    Other TCPConnector parameters like ssl, family, etc enable further optimizations.

    An aiohttp.UnixConnector can be used for higher performance when connecting to Unix domain sockets.

    There are also great extensions like aiohttp-retry for automatically retrying requests.

    Key Takeaways

  • Manage total and per-host connection limits carefully
  • Reuse connections to reduce overhead
  • Tune connection pools for infrastructure
  • Debug issues with logging
  • Further optimize with advanced features
  • Carefully configuring aiohttp's TCPConnector is key to get the most out of asynchronous HTTP in Python. Follow these tips and you'll be making speedy, efficient requests in no time!

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: