Stories from the Web Crawling trenches in JSON

Parsing JSON Responses from APIs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, use response.json() to parse JSON data. Handle invalid JSON gracefully and check status codes and Content-Type before parsing.

Retrieving and Parsing Text from URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides tools for retrieving and parsing content from URLs. It can fetch text content, parse HTML and JSON, and handle errors.

Sending JSON vs Form Data in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests module, you can send request bodies in different formats like JSON or form-urlencoded data.

Working with JSON Data in Python using urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python's urllib module provides tools for fetching and parsing JSON data from web APIs, allowing for error handling and traversal of nested data.

Processing JSON Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Handle JSON data in Python's aiohttp library for web APIs and services. Use request.json() for parsing and validate with JSON schemas.

Fixing "Content-Type incorrect" Errors with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When using Python Requests library, invalid Content-Type errors can occur due to incorrect format or missing header. Take care to set Content-Type correctly.

Troubleshooting Python Requests Returning HTML Instead of JSON

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with APIs in Python, it is important to handle authentication, set the Accept header, and monitor for HTML responses to ensure JSON data is returned.

Fetching Web Resources with urllib in MicroPython

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in MicroPython provides a simple interface for fetching resources from the web. It can handle HTTP requests and responses, making it easy to fetch JSON data, download images, and more.

Accessing the YouTube API with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The YouTube API allows developers to integrate YouTube functionality into their own applications. This article explains how to query the YouTube API v3 using the Python Requests library.

Sending Numerical Data in a Python Requests POST

Author: Mohan Ganesan

Date: Feb 3, 2024

Requests library in Python handles POST requests seamlessly, allowing you to send numerical data like integers and floats as JSON without any special handling or conversions.

Why My Python requests.post() is Sending a GET Instead of POST

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with Python's popular requests library, calling requests.post() may send a GET request instead of POST due to forgetting to pass data/json or server redirection.

How to Add Comments in JSON

Author: Mohan Ganesan

Date: Oct 4, 2023

JSON is a lightweight data format without native comment support. Use YAML or XML for commenting. JSONC is an emerging standard for comments in JSON.

Convert Object to JSON String in JavaScript

Author: Mohan Ganesan

Date: Oct 4, 2023

Converting a JavaScript object to a JSON string requires handling types like objects, arrays, and primitives. Recursively stringify nested values. Use valid JSON syntax.

Web Scraping with Visual Basic & ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Visual Basic provides a straightforward way to build web scrapers. ChatGPT is an AI assistant that can explain concepts and generate VB code for scraping.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!