Stories from the Web Crawling trenches in APIs

How to Handle Timeout error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Timeouts are critical for making requests in Python. They prevent hanging requests and wastage of resources. The requests library provides flexible ways to set timeouts globally or per-request.

Authenticating Python Requests: A Practical Guide to Using Tokens for API Access

Author: Mohan Ganesan

Date: Dec 6, 2023

API tokens are critical for securing web APIs. Learn how to obtain and use tokens for authenticated API calls in Python, and troubleshoot common token-related issues.

Caching in Python

Author: Mohan Ganesan

Date: Dec 6, 2023

Learn how to cache API responses in Python to improve performance. Caching reduces API requests, improves speed, and lowers costs.

Making Partial Updates with PATCH Requests in Python

Author: Mohan Ganesan

Date: Nov 17, 2023

PATCH requests allow partial updates to resources via APIs. Python's requests module makes it easy to send PATCH requests and modify specific attributes using JSON patch docs.

Easy Guide: Installing the Requests Module for Python in VS Code

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests module simplifies web tasks, such as HTTP requests, web scraping, and interacting with APIs. It can be easily installed in Visual Studio Code.

Sending Text Data in a POST Request with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Guide on how to send plain text data in POST requests using Python requests module and setting Content-Type header.

A Beginner's Guide to Uploading Files with Python Requests

Author: Mohan Ganesan

Date: Oct 31, 2023

Requests is a Python library for making HTTP requests, including file uploads. It simplifies the process and provides features like automatic JSON encoding and decoding. This guide walks through the steps for uploading single and multiple files, as well as additional options and error handling.

Demystifying Authentication with Python Requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Authentication can be tricky when working with APIs and web scraping. Python Requests provides various authentication schemes like basic, token-based, and digest authentication to make it easier. Understand the available auth classes and implement them properly to seamlessly integrate authentication into your Python scripts and apps.

Sending Data in aiohttp Requests

Author: Mohan Ganesan

Date: Mar 3, 2024

Building web apps and APIs with aiohttp requires sending data. JSON, form data, file uploads, and custom headers are common methods.

Working with JSON Data in Python using urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python's urllib module provides tools for fetching and parsing JSON data from web APIs, allowing for error handling and traversal of nested data.

Why Python Requests Get() Doesn't Refresh The Web Page

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library does not automatically refresh web pages like a browser. It only downloads static content.

Keeping Sessions Alive with Persistent Connections in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Using persistent sessions in Python Requests library improves performance and allows reusing connections for multiple requests.

Making HTTP POST Requests with Httpx in Python

Author: Mohan Ganesan

Date: Feb 5, 2024

Httpx library in Python provides a modern and intuitive HTTP client for making POST requests to APIs and web services. It handles request headers, form data, timeouts, retries, and more.

How to write URL in Python?

Author: Mohan Ganesan

Date: Feb 8, 2024

Best practices for handling URLs in Python for web applications, APIs, and scraping websites.

Configuring Headers with aiohttp Clients for Effective API Calls

Author: Mohan Ganesan

Date: Feb 22, 2024

Properly configuring headers in aiohttp is crucial for smooth API requests. Headers serve purposes like authentication, context, security, and caching.

Sending POST Data with HTTPX in Python

Author: Mohan Ganesan

Date: Feb 5, 2024

HTTPX is a popular Python library for making HTTP requests. This guide explains how to properly structure and send POST data with HTTPX.

Simplifying HTTP Requests in Python: Requests vs urllib3

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python: choose between requests and urllib3. Requests is simple and beginner friendly, while urllib3 offers more control and customization.

Downloading ZIP Files with aiohttp in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

aiohttp is a Python library for asynchronous HTTP clients and servers. It allows for streaming ZIP file downloads in web applications and APIs.

Passing Parameters in aiohttp Requests

Author: Mohan Ganesan

Date: Feb 22, 2024

Pass parameters in Python aiohttp requests using query string, form parameters, or JSON data to modify the response.

Parsing JSON Responses from APIs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, use response.json() to parse JSON data. Handle invalid JSON gracefully and check status codes and Content-Type before parsing.

Handling Failed Requests in Python: Techniques for Resilience

Author: Mohan Ganesan

Date: Feb 3, 2024

Best practices for handling failed requests in Python: use try/except blocks, implement exponential backoff for retries, and use a circuit breaker pattern.

Simplifying HTTP Requests in Python: urllib2 vs urllib vs requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python: from urllib2 to requests. urllib2 - Python's Default HTTP Client. urllib - A Minor Improvement. requests - A Simple Yet Powerful Library.

Troubleshooting Python Requests Returning HTML Instead of JSON

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, it is important to handle authentication, set the Accept header, and monitor for HTML responses to ensure JSON data is returned.

Handling Timeouts Gracefully with aiohttp in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

When building asynchronous web applications and APIs in Python with aiohttp, properly handling timeouts is essential. Use ClientTimeout to configure request timeouts and wrap requests in try/except blocks to catch ClientTimeout. Configure global timeout on aiohttp servers with timeout parameter.

Rate Limiting Requests with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

Prevent abuse and reduce server load by rate limiting requests using aiohttp's ThrottleConcurrency middleware.

Building Asynchronous APIs with aiohttp and Queue

Author: Mohan Ganesan

Date: Mar 3, 2024

Asynchronous programming with aiohttp and queues in Python enables efficient web development and API creation.

Sending Data in Requests: Payloads, Headers, and Parameters

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python requests module allows you to easily send HTTP requests to APIs and websites. You can attach data as JSON payloads, form-encoded data, or query parameters.

Enable Detailed HTTP Debug Logging in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Enable debug logging in Python Requests library to get detailed insight into HTTP requests and save time debugging issues.

Async HTTP Clients: aiohttp vs httpx

Author: Mohan Ganesan

Date: Feb 22, 2024

Python developers often make HTTP requests to access APIs and web services. Two popular async HTTP client libraries for Python are aiohttp and httpx. This article compares the two libraries and discusses their key differences, features, and performance. The choice between aiohttp and httpx depends on specific needs, such as client/server use cases, HTTP/2 support, ease of use, and control over limits and configuration.

Working with Request Parameters in aiohttp

Author: Mohan Ganesan

Date: Mar 3, 2024

aiohttp makes it easy to get request parameters. Adding validation middleware helps create robust APIs and catch issues early.

How to use URL in Python?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python has great URL handling capabilities out of the box. Whether you need to parse URLs, download files, call web APIs, or interact with websites, Python has you covered!

Sending and Receiving JSON Data with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library makes it easy to send HTTP requests and receive responses in JSON format. It simplifies working with APIs and web services.

Easy Guide: Installing the Requests Library for Python on Windows

Author: Mohan Ganesan

Date: Feb 3, 2024

Learn how to install and use the Python requests library for making HTTP requests in your projects.

Python Requests Library: Making Authenticated POST Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library provides a simple way to make HTTP requests in Python, including POST requests with Basic HTTP Authentication for authenticated API requests.

Fixing aiohttp UnicodeDecodeErrors

Author: Mohan Ganesan

Date: Mar 3, 2024

Fixing UnicodeDecodeErrors in aiohttp: specify encoding, check actual encoding, decode manually, re-encode text

Streamlining HTTP Requests in Python with the Requests Module

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests module is an essential tool for interacting with APIs and websites in your Python code.

Is API better than web scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

APIs vs web scraping: pros and cons of structured data retrieval and HTML parsing for flexible data access.

Returning HTML Responses with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

aiohttp allows easy return of HTML content in Python web applications and APIs. Use template engines and response streaming for robust web apps.

urllib get

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides a simple interface for fetching data over HTTP. With just a few lines of code, you can easily make GET and POST requests to access web pages and APIs.

how long does it take to learn web scraping

Author: Mohan Ganesan

Date: Feb 20, 2024

Learn web scraping in 0-3 months with Python or JavaScript. Master advanced techniques in 4-12 months. Keep leveling up your skills!

Async IO for Python: aiohttp 3.7.4

Author: Mohan Ganesan

Date: Mar 3, 2024

The aiohttp library provides asynchronous HTTP client/server functionality for Python based on the asyncio event loop. Version 3.7.4 contains useful updates that make aiohttp even more powerful and developer-friendly.

Is web scraping for beginners?

Author: Mohan Ganesan

Date: Feb 20, 2024

Web scraping is the process of extracting data from websites. Beginners can learn it with programming knowledge in HTML/CSS, Python, and JavaScript.

urllib retrieve

Author: Mohan Ganesan

Date: Feb 8, 2024

urllib in Python makes it easy to fetch resources from the web. Handle errors and include data in requests with URL encoding.

What is the function of the Urllib library?

Author: Mohan Ganesan

Date: Feb 20, 2024

The urllib library in Python is a powerful tool for web scraping, interacting with APIs, and handling HTTP requests.

Will Google ban you for scraping?

Author: Mohan Ganesan

Date: Feb 22, 2024

Web scraping involves collecting data from websites. Google allows scraping within limits, but bans excessive scraping, complete site downloads, circumventing captchas/blocks, and compromising security. Best practices include using official APIs, rotating IP addresses, using random delays, and stopping if encountering captchas or blocks.

What is the difference between web scraping and data scraping?

Author: Mohan Ganesan

Date: Feb 5, 2024

Web scraping extracts data from web pages, while data scraping is a broader term for extracting data from any online source.

The Murky Legality of Scraping Public APIs

Author: Mohan Ganesan

Date: Feb 20, 2024

APIs provide easy access to public data, but scraping them may be illegal. Factors like rate limits and terms of service impact legality. Best practices include respecting restrictions, citing sources, and not selling or spamming with scraped data.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!