Stories from the Web Crawling trenches in customization

Using Proxies With C++ httplib in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Using a proxy with C++ httplib is easy. Set up authentication, chain multiple proxies, customize settings, and troubleshoot issues. Proxies API offers a better solution for unblockable scraping.

What is PoolManager in urllib3?

Author: Mohan Ganesan

Date: Feb 20, 2024

Simplifying HTTP requests with PoolManager in Python. PoolManager manages a pool of connections for reusing, improving performance. Customize pool behavior for better resource usage.

Customizing the User Agent for urllib in Python

Author: Mohan Ganesan

Date: Feb 6, 2024

Customize the user agent string in Python's urllib library to mimic a web browser, identify your application, or adhere to site requirements.

Customizing HTTPX User Agents for Effective API Requests

Author: Mohan Ganesan

Date: Feb 5, 2024

Customize the User Agent header in HTTPX Python library for API analytics, compatibility checks, and access control.

Octoparse Alternative - Simplify Web Scraping with ProxiesAPI

Author: Mohan Ganesan

Date: Sep 30, 2023

Octoparse is a visual web scraping tool, but for more customization and scale, an API-based solution like ProxiesAPI is better.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!