Stories from the Web Crawling trenches in error handling

The Complete Guide to Retrying Failed Requests with Axios

Author: Mohan Ganesan

Date: Jan 9, 2024

Automated retries using Axios interceptors provide reliability, speed, scalability, and resilience. Configuring retries involves setting the number of retries, delay between retries, and conditional retries. The Axios-Retry plugin simplifies the process. Other libraries like retry-axios offer similar capabilities. Testing and debugging retry logic is important, and caution must be taken to avoid circular retries. Axios is widely used in React apps and can be used for any HTTP backend. Automated retries are essential for building robust apps that handle remote services.

Using Python Requests to Ping an IP Address

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library provides a simple way to ping an IP address and check if it is reachable. This guide covers how to ping an IP address with Requests and handle errors gracefully.

How to fix MissingSchema error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

The MissingSchema error occurs when making a request to a URL without specifying the protocol. This article explains the causes of the error and provides various ways to fix and handle it properly.

Downloading Files in Python with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

Python's aiohttp library allows for asynchronous and non-blocking downloading of files. It provides a simple API, handles streams efficiently, and supports progress reporting and error handling.

A Beginner's Guide to Uploading Files with Python Requests

Author: Mohan Ganesan

Date: Oct 31, 2023

Requests is a Python library for making HTTP requests, including file uploads. It simplifies the process and provides features like automatic JSON encoding and decoding. This guide walks through the steps for uploading single and multiple files, as well as additional options and error handling.

Scraping all the Images from a Website with Rust

Author: Mohan Ganesan

Date: Dec 13, 2023

Learn how to use Rust for web scraping, including data extraction, image scraping, and error handling. Overcome IP blocking with a rotating proxy service like Proxies API.

Working with JSON Data in Python using urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python's urllib module provides tools for fetching and parsing JSON data from web APIs, allowing for error handling and traversal of nested data.

Fetching Data from APIs with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Accessing data from web APIs using Python's Requests library. Learn how to make GET requests, process responses, and handle errors.

Handling URL Errors Gracefully in Python urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Handle errors when working with URLs in Python using the urllib module. Catch HTTPError and URLError exceptions, and apply targeted handling and retries where applicable.

Parsing JSON Responses from APIs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, use response.json() to parse JSON data. Handle invalid JSON gracefully and check status codes and Content-Type before parsing.

Hands-On Guide to Python Requests Status Codes

Author: Mohan Ganesan

Date: Nov 17, 2023

Status codes are a vital part of working with the Python Requests library. Learn how to access, interpret, and handle status codes in Python Requests for writing robust scripts and applications.

Handling Failed Requests in Python: Techniques for Resilience

Author: Mohan Ganesan

Date: Feb 3, 2024

Best practices for handling failed requests in Python: use try/except blocks, implement exponential backoff for retries, and use a circuit breaker pattern.

Fetching Images Asynchronously with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Building web applications in Python with aiohttp for efficient asynchronous requests, including image downloading, streaming responses, and error handling.

Why Your Python Requests Timeout May Not Be Timing Out As Expected

Author: Mohan Ganesan

Date: Feb 3, 2024

When using the requests library in Python, you can specify a timeout value to prevent your code from hanging indefinitely if a request gets stuck.

Handling Errors Gracefully When URLs Fail in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests module provides built-in error handling for HTTP requests. Common errors include ConnectionError, Timeout, HTTPError, and RequestException. Handling errors gracefully ensures resilient applications.

Retrieving and Parsing Text from URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides tools for retrieving and parsing content from URLs. It can fetch text content, parse HTML and JSON, and handle errors.

Fetching Content with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

The aiohttp library is a powerful tool for making asynchronous HTTP requests in Python. This guide demonstrates practical examples of using aiohttp to fetch content, handle errors, set request headers, post form data, stream response content, configure timeouts, and provides practical tips for working with aiohttp.

Handling Client Errors with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

When building applications with aiohttp, it is important to handle client errors properly. Use the ClientResponseError exception and status code to identify client errors and implement custom error handling logic for expected cases.

Handling HTTP Status Codes Gracefully with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies working with web APIs and handling HTTP status codes. Properly handling status codes is crucial for robust Python code.

Difference between urllib and urllib2

Author: Mohan Ganesan

Date: Feb 6, 2024

urllib for simple HTTP requests, urllib2 for robust HTTPS, redirects, custom headers, and error handling.

Async IO in Python: Trio vs. Asyncio

Author: Mohan Ganesan

Date: Mar 25, 2024

Python developers have two main options for asynchronous I/O concurrency - asyncio and Trio. Both allow you to write non-blocking, concurrent code in Python. But which one is better for your use case?

Handling Errors Gracefully in aiohttp with errors=ignore

Author: Mohan Ganesan

Date: Mar 3, 2024

errors='ignore' prevents aiohttp client errors from crashing your application. Customize exactly which errors to ignore and handle them programmatically. Vital for robust and resilient asynchronous services.

urllib read

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides functionality for retrieving data from URLs. It allows you to fetch web pages, decode and parse HTML, and handle errors. Practical examples include web scraping and checking broken links.

Handling Errors Gracefully with Asyncio Exceptions

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio provides an asynchronous programming framework in Python for non-blocking I/O code. Exception handling in asyncio requires special care, including handling CancelledError and propagating exceptions from tasks.

urllib retrieve

Author: Mohan Ganesan

Date: Feb 8, 2024

urllib in Python makes it easy to fetch resources from the web. Handle errors and include data in requests with URL encoding.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!