Stories from the Web Crawling trenches in web APIs

How to fix MissingSchema error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

The MissingSchema error occurs when making a request to a URL without specifying the protocol. This article explains the causes of the error and provides various ways to fix and handle it properly.

Processing JSON Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Handle JSON data in Python's aiohttp library for web APIs and services. Use request.json() for parsing and validate with JSON schemas.

Fetching Data from APIs with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Accessing data from web APIs using Python's Requests library. Learn how to make GET requests, process responses, and handle errors.

Keeping Data Flowing with aiohttp Streaming Responses

Author: Mohan Ganesan

Date: Feb 22, 2024

Streaming responses in aiohttp allow for efficient data transfer, reduced memory usage, and improved client experience.

Is Requests a Built-In Python Library?

Author: Mohan Ganesan

Date: Oct 22, 2023

Requests is a popular Python library for making HTTP requests, providing an elegant API and handling details like encoding parameters, cookies, and authentication. It simplifies HTTP calls compared to the built-in urllib module, but needs to be installed separately.

Testing Asynchronous Code with Aiohttp Test Utilities

Author: Mohan Ganesan

Date: Mar 3, 2024

The aiohttp library in Python provides utilities for testing asynchronous code. Use aiohttp.test_utils module to test web APIs and apps.

Making HTTP Requests in Python: Requests and urllib3 Explained

Author: Mohan Ganesan

Date: Feb 3, 2024

Python code interacts with web APIs or crawls websites using HTTP requests. requests and urllib3 are popular libraries for this.

Handling HTTP Status Codes Gracefully with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies working with web APIs and handling HTTP status codes. Properly handling status codes is crucial for robust Python code.

Do I need to install Urllib in Python?

Author: Mohan Ganesan

Date: Feb 8, 2024

urllib is included automatically with Python and comes pre-installed with standard Python distributions. No separate installation required.

Why use urllib3?

Author: Mohan Ganesan

Date: Feb 20, 2024

urllib3 is a full-featured HTTP client for making requests in Python. It handles connection pooling, SSL/TLS verification, and more, making it a popular choice for web APIs.

Simplifying HTTP Requests in Python with urllib

Author: Mohan Ganesan

Date: Feb 3, 2024

The urllib module in Python provides tools for fetching data from the web. It allows making HTTP requests, handling responses, and constructing customized requests.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!