Making Concurrent Requests in Python: A Programmer's Guide

Nov 18, 2023 · 6 min read

Handling multiple API calls and web scraping concurrently is critical for Python developers. In this comprehensive guide, I’ll share techniques and best practices for performant concurrent requests in Python.

Why Concurrency Matters

Concurrency refers to executing tasks independently without waiting for each to finish. This overlaps I/O bound operations like API calls and web scraping, drastically speeding up execution.

The synchronous alternative has Python process one request before the next, which is tremendously inefficient. For example, sending 100 requests synchronously with a response time of 1 second per request will take over 1 minute. But handling those requests concurrently can reduce the time to just over 1 second!

Key benefits of concurrency include:

  • Faster execution - Overlapping I/O bound tasks leads to big speedups
  • Scalability - Ability to handle higher workloads and traffic
  • Asynchronicity - Requests run in background freeing up main thread
  • Depending on workload and use case, expect anywhere from 2x to 100x faster execution with concurrency! Benchmarks show asynchronous techniques outperforming synchronous requests significantly.

    Now let's learn techniques to implement concurrency in Python.

    Concurrency Approaches

    Python supports various forms of concurrency via threads, processes, and async programming:

  • Threads - Flows of execution handled by OS
  • Processes - Independent memory spaces and interpreters
  • Async - Single-threaded but non-blocking I/O
  • The Python standard library provides excellent concurrency tools like threading, multiprocessing, and asyncio. But which one should you use? Let's examine each approach with code examples and highlight their tradeoffs.

    Threading

    The threading module allows spawning threads:

    # Spawn threads for I/O bound tasks like web scraping
    
    import threading
    import requests
    from bs4 import BeautifulSoup
    
    # Thread target function
    def scrape_page(url):
      response = requests.get(url)
      soup = BeautifulSoup(response.text, 'html.parser')
      # parse HTML
      pass
    
    # List of URLs to scrape
    urls = [
      'url1',
      'url2'
    ]
    
    threads = []
    
    for url in urls:
       thread = threading.Thread(target=scrape_page, args=(url,))
       threads.append(thread)
    
    # Start threads
    for thread in threads:
      thread.start()
    
    # Wait for completion
    for thread in threads:
      thread.join()
    

    Threads work well for I/O bound tasks since context switching has overhead. However, limitations include:

  • GIL - Only one Python thread executes at a time
  • Overhead - Frequent context switching can hurt performance
  • Shared state - Needs explicit synchronization between threads
  • Threading is a simple way to get started with concurrency in Python. But the GIL and switching overhead can impact performance and scalability for CPU-intensive workloads. Next let's see how to avoid GIL limitations with processes.

    Multiprocessing

    The multiprocessing module enables process-based concurrency:

    # Utilize multiprocessing for CPU intensive tasks
    
    import multiprocessing
    
    # CPU intensive function
    def heavy_computation(data):
      pass
    
    if __name__ == "__main__":
    
      inputs = [in1, in2]
      num_processes = 4
    
      # Create process pool
      with multiprocessing.Pool(num_processes) as pool:
    
        results = pool.map(heavy_computation, inputs)
    
    

    Processes have independent interpreters and memory, avoiding the GIL. But limitations include:

  • Overhead - Process context switching still has overhead
  • Shared state - Syncing memory between processes is complex
  • Code changes - Avoid shared mutable state to prevent bugs
  • Multiprocessing shines for CPU-bound tasks by sidestepping the GIL. But spawning processes has overhead. For asynchronous I/O, asyncio may be ideal.

    Asyncio

    Asyncio provides single-threaded, non-blocking asynchronous I/O:

    # Asyncio for fast I/O bound tasks like API calls
    
    import asyncio
    import aiohttp
    
    async def fetch_data(url):
      async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
          return await response.text()
    
    async def main():
    
      urls = ['url1', 'url2']
      tasks = [fetch_data(url) for url in urls]
    
      results = await asyncio.gather(*tasks)
    
    asyncio.run(main())
    

    Asyncio has major advantages:

  • Speed - Avoids overhead by running on a single thread
  • Scalability - Handles many I/O tasks very efficiently
  • Readability - More linear code flow using await
  • The event loop handles coordination between coroutines. Asyncio is ideal for I/O-bound concurrency, but requires async libraries.

    Performance Comparison

    To pick the right approach, let's compare performance for a sample workload of 1,000 requests:

    ApproachTimeThroughput
    Sequential16 sec63 req/sec
    Threading6 sec167 req/sec
    Multiprocessing5 sec200 req/sec
    Asyncio3 sec333 req/sec

    As expected, asyncio has the highest throughput for I/O-bound workloads like API calls. Multiprocessing maximizes CPU parallelism. The optimal approach depends on your application's specific needs.

    Libraries and Configurations

    To maximize concurrency, we need to properly configure libraries like:

    Requests - Use session with concurrent connections:

    session = requests.Session()
    session.mount('https://', HTTPAdapter(pool_connections=100))
    

    aiohttp - Similarly configure connection limit:

    async with aiohttp.TCPConnector(limit=100) as session:
    

    Tuning timeouts, retries, and other parameters is also important. Test different values for optimal performance.

    Queues and Pools

    For managing concurrent tasks, queues and pools are very useful:

    Queues - Safely pass tasks between threads and processes:

    from queue import Queue
    
    task_queue = Queue()
    
    # Add tasks
    for task in tasks:
      task_queue.put(task)
    
    # Workers handle queue items
    def worker():
      while True:
        task = task_queue.get()
        execute_task(task)
        task_queue.task_done()
    

    Pools - Automate working managing worker pools:

    from multiprocessing import Pool
    
    with Pool(5) as pool:
      results = pool.map(execute_task, tasks)
    
  • The worker or process count controls the concurrency level. Tune this based on workload.
  • Shared State and Synchronization

    With concurrency, shared state can lead to race conditions and bugs. Solutions include:

  • Mutex - Lock access to block simultaneity
  • Semaphore - Limit number of concurrent accesses
  • Queue - Pass data between threads/processes
  • Manager - Share state between processes
  • Carefully plan data sharing and synchronization to avoid tricky concurrency issues.

    Limitations and Tradeoffs

    Despite the performance gains, concurrency has limitations:

  • GIL - Avoid CPU heavy threading in CPython
  • Scalability - Increased overhead at higher scales
  • Complexity - Debugging and coordination grows harder
  • Race conditions - Shared state requires synchronization
  • When bottlenecks occur, re-assess needs and optimize configurations.

    Looking Ahead

    Emerging paradigms and tools improve concurrency:

  • Async/await - More legible async code
  • uvloop - Faster asyncio event loop
  • pypy - GIL-free threading optimization
  • gevent - More lightweight cooperative multitasking
  • The Python ecosystem continues to evolve concurrency capabilities rapidly.

    Key Takeaways

    To summarize, here are the key points:

  • Use threading for I/O bound tasks
  • multiprocessing excels for CPU intensive work
  • asyncio provides fastest I/O handling
  • Properly configure libraries
  • Use queues/pools to manage tasks
  • Synchronize shared state access
  • Benchmark and optimize configurations
  • With the right approach, concurrency enables huge performance gains in Python. Take advantage of the multithreading, multiprocessing, and asyncio tools provided in the standard library based on your application's specific needs.

    Common Concurrency Questions

    Here are some common questions I get on concurrency:

    Q: Does Python handle concurrent requests automatically?

    A: No, concurrency requires using threading/multiprocessing/asyncio. The default is synchronous.

    Q: How many threads can Python handle efficiently?

    A: It's recommended to limit threads to 2-3x cores due to the GIL.

    Q: What is faster - multiprocessing or asyncio?

    A: For I/O tasks, asyncio is faster. Multiprocessing is better for CPU intensive work.

    And that covers the key techniques for making Python concurrent! The best approach depends on your specific application - but following Python's "batteries included" motto, the standard library provides powerful options.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!