Why Python's Multithreading Perfoms Poorly (And What To Do About It)

Mar 24, 2024 ยท 2 min read

Python's multithreading capabilities often surprise newcomers - threads seem to run slowly and parallelism is limited. This stems from Python's design, but there are solutions.

The GIL: Serializing Concurrency

Python uses a Global Interpreter Lock (GIL) that allows only one thread to execute Python bytecodes at a time. This effectively serializes execution, limiting multicore scalability.

The GIL exists because Python's memory management is not thread-safe. The lock allows threads to run concurrently, but only one executes Python code at a time. Others awaiting the GIL still run cPython code.

import threading

def countdown(n):
    while n > 0:
        n -= 1

t1 = threading.Thread(target=countdown, args=(1000000,))  
t2 = threading.Thread(target=countdown, args=(1000000,))


This will take ~2 seconds despite having two threads, due to the GIL!


For CPU-bound processing, multiprocessing can sidestep the GIL by spawning new Python processes:

import multiprocessing

def countdown(n):
    # As before

if __name__ == "__main__":
    p1 = multiprocessing.Process(target=countdown, args=(1000000,))  
    p2 = multiprocessing.Process(target=countdown, args=(1000000,))

Now the countdowns run in parallel across processes, completing in ~1 second.

For IO-bound tasks like web serving, asynchronous frameworks like asyncio avoid the GIL by suspending threads instead of blocking:

import asyncio

async def countdown(n):
    while n > 0:
        n -= 1
        await asyncio.sleep(0) # Suspend thread

# Restarts each time await releases the GIL

So while Python's multithreading seems slow, solutions exist - you just need the right tool for the job!

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: