Stories from the Web Crawling trenches in Servers

Why You May Not Get All Cookie Data with the Python Requests Module

Author: Mohan Ganesan

Date: Feb 3, 2024

Use Sessions or custom jars to ensure you have full cookie details when using Requests.

What is Urlencode in Python?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python's urllib module provides simple ways to handle URL encoding. Encoding URLs ensures special characters transmit safely through networks and servers.

What are the risks of web scraping?

Author: Mohan Ganesan

Date: Feb 22, 2024

Web scraping can collect large amounts of data from websites, but it comes with risks. Respect terms of service, avoid overloading servers, prevent data corruption, and mask scraping activities.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!