Stories from the Web Crawling trenches in site requirements

Customizing the User Agent for urllib in Python

Author: Mohan Ganesan

Date: Feb 6, 2024

Customize the user agent string in Python's urllib library to mimic a web browser, identify your application, or adhere to site requirements.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!