Stories from the Web Crawling trenches in thread safety

Simplifying HTTP Requests with PoolManager in Python

Author: Mohan Ganesan

Date: Feb 20, 2024

Making HTTP requests in Python is simplified and optimized with PoolManager from the urllib3 library, which handles connection pooling, reducing latency and resource utilization, ensuring thread safety, and abstracting away connection management logic.

Concurrency and Thread Safety in Python's asyncio

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables concurrency within a single thread using an event loop. Sharing data between coroutines is thread-safe. Multithreading requires new event loops and explicit synchronization. Blocking code must execute in threads to avoid blocking the event loop. Following these best practices ensures efficient, thread-safe asyncio code.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!