Stories from the Web Crawling trenches in JSON

Sending JSON vs Form Data in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests module, you can send request bodies in different formats like JSON or form-urlencoded data.

Parsing JSON Responses from APIs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, use response.json() to parse JSON data. Handle invalid JSON gracefully and check status codes and Content-Type before parsing.

Working with JSON Data in Python using urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python's urllib module provides tools for fetching and parsing JSON data from web APIs, allowing for error handling and traversal of nested data.

Retrieving and Parsing Text from URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides tools for retrieving and parsing content from URLs. It can fetch text content, parse HTML and JSON, and handle errors.

Fixing "Content-Type incorrect" Errors with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When using Python Requests library, invalid Content-Type errors can occur due to incorrect format or missing header. Take care to set Content-Type correctly.

Fetching Web Resources with urllib in MicroPython

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in MicroPython provides a simple interface for fetching resources from the web. It can handle HTTP requests and responses, making it easy to fetch JSON data, download images, and more.

Processing JSON Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Handle JSON data in Python's aiohttp library for web APIs and services. Use request.json() for parsing and validate with JSON schemas.

Troubleshooting Python Requests Returning HTML Instead of JSON

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, it is important to handle authentication, set the Accept header, and monitor for HTML responses to ensure JSON data is returned.

Accessing the YouTube API with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The YouTube API allows developers to integrate YouTube functionality into their own applications. This article explains how to query the YouTube API v3 using the Python Requests library.

Sending Numerical Data in a Python Requests POST

Author: Mohan Ganesan

Date: Feb 3, 2024

Requests library in Python handles POST requests seamlessly, allowing you to send numerical data like integers and floats as JSON without any special handling or conversions.

How to Add Comments in JSON

Author: Mohan Ganesan

Date: Oct 4, 2023

JSON is a lightweight data format without native comment support. Use YAML or XML for commenting. JSONC is an emerging standard for comments in JSON.

Web Scraping with Visual Basic & ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Visual Basic provides a straightforward way to build web scrapers. ChatGPT is an AI assistant that can explain concepts and generate VB code for scraping.

Convert Object to JSON String in JavaScript

Author: Mohan Ganesan

Date: Oct 4, 2023

Converting a JavaScript object to a JSON string requires handling types like objects, arrays, and primitives. Recursively stringify nested values. Use valid JSON syntax.

Why My Python requests.post() is Sending a GET Instead of POST

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with Python's popular requests library, calling requests.post() may send a GET request instead of POST due to forgetting to pass data/json or server redirection.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!