Stories from the Web Crawling trenches in state

Persisting Sessions with Httpx in Python

Author: Mohan Ganesan

Date: Feb 5, 2024

Guide on utilizing Httpx's session support to maintain state and persist cookies across multiple requests in Python.

Async IO and Generators: Key Differences in Python

Author: Mohan Ganesan

Date: Mar 24, 2024

Async IO and generators are powerful asynchronous programming concepts in Python with key differences. Generators produce data on demand, while Async IO enables concurrent work. Both are useful for different scenarios and can be used together to write highly scalable programs.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!