Stories from the Web Crawling trenches in error handling

Using Python Requests to Ping an IP Address

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library provides a simple way to ping an IP address and check if it is reachable. This guide covers how to ping an IP address with Requests and handle errors gracefully.

The Complete Guide to Retrying Failed Requests with Axios

Author: Mohan Ganesan

Date: Jan 9, 2024

Automated retries using Axios interceptors provide reliability, speed, scalability, and resilience. Configuring retries involves setting the number of retries, delay between retries, and conditional retries. The Axios-Retry plugin simplifies the process. Other libraries like retry-axios offer similar capabilities. Testing and debugging retry logic is important, and caution must be taken to avoid circular retries. Axios is widely used in React apps and can be used for any HTTP backend. Automated retries are essential for building robust apps that handle remote services.

Downloading Files in Python with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

Python's aiohttp library allows for asynchronous and non-blocking downloading of files. It provides a simple API, handles streams efficiently, and supports progress reporting and error handling.

A Beginner's Guide to Uploading Files with Python Requests

Author: Mohan Ganesan

Date: Oct 31, 2023

Requests is a Python library for making HTTP requests, including file uploads. It simplifies the process and provides features like automatic JSON encoding and decoding. This guide walks through the steps for uploading single and multiple files, as well as additional options and error handling.

How to fix MissingSchema error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

The MissingSchema error occurs when making a request to a URL without specifying the protocol. This article explains the causes of the error and provides various ways to fix and handle it properly.

Parsing JSON Responses from APIs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, use response.json() to parse JSON data. Handle invalid JSON gracefully and check status codes and Content-Type before parsing.

Handling URL Errors Gracefully in Python urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Handle errors when working with URLs in Python using the urllib module. Catch HTTPError and URLError exceptions, and apply targeted handling and retries where applicable.

Working with JSON Data in Python using urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python's urllib module provides tools for fetching and parsing JSON data from web APIs, allowing for error handling and traversal of nested data.

Retrieving and Parsing Text from URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides tools for retrieving and parsing content from URLs. It can fetch text content, parse HTML and JSON, and handle errors.

Handling Client Errors with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

When building applications with aiohttp, it is important to handle client errors properly. Use the ClientResponseError exception and status code to identify client errors and implement custom error handling logic for expected cases.

Why Your Python Requests Timeout May Not Be Timing Out As Expected

Author: Mohan Ganesan

Date: Feb 3, 2024

When using the requests library in Python, you can specify a timeout value to prevent your code from hanging indefinitely if a request gets stuck.

Async IO in Python: Trio vs. Asyncio

Author: Mohan Ganesan

Date: Mar 25, 2024

Python developers have two main options for asynchronous I/O concurrency - asyncio and Trio. Both allow you to write non-blocking, concurrent code in Python. But which one is better for your use case?

Scraping all the Images from a Website with Rust

Author: Mohan Ganesan

Date: Dec 13, 2023

Learn how to use Rust for web scraping, including data extraction, image scraping, and error handling. Overcome IP blocking with a rotating proxy service like Proxies API.

Fetching Images Asynchronously with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Building web applications in Python with aiohttp for efficient asynchronous requests, including image downloading, streaming responses, and error handling.

Handling Errors Gracefully When URLs Fail in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests module provides built-in error handling for HTTP requests. Common errors include ConnectionError, Timeout, HTTPError, and RequestException. Handling errors gracefully ensures resilient applications.

Handling Failed Requests in Python: Techniques for Resilience

Author: Mohan Ganesan

Date: Feb 3, 2024

Best practices for handling failed requests in Python: use try/except blocks, implement exponential backoff for retries, and use a circuit breaker pattern.

Handling HTTP Status Codes Gracefully with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies working with web APIs and handling HTTP status codes. Properly handling status codes is crucial for robust Python code.

Handling Errors Gracefully with Asyncio Exceptions

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio provides an asynchronous programming framework in Python for non-blocking I/O code. Exception handling in asyncio requires special care, including handling CancelledError and propagating exceptions from tasks.

Fetching Content with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

The aiohttp library is a powerful tool for making asynchronous HTTP requests in Python. This guide demonstrates practical examples of using aiohttp to fetch content, handle errors, set request headers, post form data, stream response content, configure timeouts, and provides practical tips for working with aiohttp.

Fetching Data from APIs with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Accessing data from web APIs using Python's Requests library. Learn how to make GET requests, process responses, and handle errors.

Hands-On Guide to Python Requests Status Codes

Author: Mohan Ganesan

Date: Nov 17, 2023

Status codes are a vital part of working with the Python Requests library. Learn how to access, interpret, and handle status codes in Python Requests for writing robust scripts and applications.

Difference between urllib and urllib2

Author: Mohan Ganesan

Date: Feb 6, 2024

urllib for simple HTTP requests, urllib2 for robust HTTPS, redirects, custom headers, and error handling.

Handling Errors Gracefully in aiohttp with errors=ignore

Author: Mohan Ganesan

Date: Mar 3, 2024

errors='ignore' prevents aiohttp client errors from crashing your application. Customize exactly which errors to ignore and handle them programmatically. Vital for robust and resilient asynchronous services.

urllib retrieve

Author: Mohan Ganesan

Date: Feb 8, 2024

urllib in Python makes it easy to fetch resources from the web. Handle errors and include data in requests with URL encoding.

urllib read

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides functionality for retrieving data from URLs. It allows you to fetch web pages, decode and parse HTML, and handle errors. Practical examples include web scraping and checking broken links.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!