Stories from the Web Crawling trenches in Python code

Web Scraping into Excel using ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Web scraping with ChatGPT allows easy extraction of data from websites and saving it in Excel using Python code. Use Pandas to format and output data. Get started now!

Handling Cross-Origin Requests in Python with CORS

Author: Mohan Ganesan

Date: Feb 3, 2024

Make HTTP requests from Python code to APIs on different domains using CORS. Understand the same-origin policy and handle CORS nuances with flask-cors.

Troubleshooting Python Requests Through a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Common problems and solutions when sending requests through a proxy server in Python code.

Inspecting Requests in Python with the Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library makes sending HTTP requests simple. Use request.headers to view the headers sent in a Requests request. Access the request body with request.body. Set json instead of data parameter to have readable body printed.

How does Amazon detect scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

Amazon strictly prohibits scraping their site. Use proxies, randomize delays, limit volume, and scrape selectively to avoid detection. Python code provided.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!