Stories from the Web Crawling trenches in encoding

Decoding URL Responses with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

Convert between bytes and strings in Python's urllib module using encode() and decode(). Specify correct encoding to avoid errors.

Passing Parameters in URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

Construct URL requests in Python using urllib module to pass parameters and handle encoding. GET requests use parameters in the URL, while POST requests use the request body.

Fixing aiohttp UnicodeDecodeErrors

Author: Mohan Ganesan

Date: Mar 3, 2024

Fixing UnicodeDecodeErrors in aiohttp: specify encoding, check actual encoding, decode manually, re-encode text

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!