Stories from the Web Crawling trenches in data

Passing Data in URLs with urllib Query Parameters in Python

Author: Mohan Ganesan

Date: Feb 8, 2024

Pass data through URLs using query parameters in Python's urllib module for HTTP requests.

Sending Data in GET Requests with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library provides a simple way to send GET requests with data using the requests.get() method. It encodes the data into a query string that is appended to the URL, making it perfect for sending non-sensitive data like filters or pagination options.

Accessing Websites in Python with urllib.request.urlopen

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib.request module in Python 3 provides a simple way to access and download data from websites via HTTP and HTTPS.

Why My Python requests.post() is Sending a GET Instead of POST

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with Python's popular requests library, calling requests.post() may send a GET request instead of POST due to forgetting to pass data/json or server redirection.

How to query an API?

Author: Mohan Ganesan

Date: May 7, 2024

APIs allow software systems to communicate. Querying APIs involves finding documentation, setting up authentication, choosing an endpoint, sending a request, and handling the response. Tips include using Postman, inspecting responses, starting with simple queries, checking status codes, and using parameters. Learning how to query APIs properly enables the creation of powerful and integrated applications.

What is API in one sentence?

Author: Mohan Ganesan

Date: May 7, 2024

APIs are a standardized interface that allows software programs to communicate with each other, providing access to data and functionality. They save developers time, promote modularity, and encourage innovation.

Scraping Yelp Business Listings using CSharp

Author: Mohan Ganesan

Date: Dec 6, 2023

Yelp is a popular review site with over 200 million reviews. This article explains how to scrape Yelp using proxies and HTML parsing with XPath.

What are the basics of APIs?

Author: Mohan Ganesan

Date: May 7, 2024

APIs allow different software systems to communicate by defining a common interface with standard inputs and outputs.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!