Stories from the Web Crawling trenches in API Requests

Sending Parameters in URLs with the Python Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

Making API requests with Python Requests library, passing parameters as a dictionary, handling URL parameters and headers for complex requests.

Tips for Handling JavaScript Content with BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

Dealing with heavy JavaScript sites takes specialized tools like browser automation or APIs. BeautifulSoup can still effectively access and parse content.

Making API Requests Safely with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making API requests in Python, it's important to consider security. Use HTTPS, validate certificates, use tokens for authentication, and handle sensitive data safely.

Speed Up Your API Requests: 5 Simple Optimization Tips

Author: Mohan Ganesan

Date: Feb 3, 2024

Making API requests faster with async/await, setting timeout limits, caching options, using a CDN, and throttling concurrent requests.

What is the basic structure of an API?

Author: Mohan Ganesan

Date: May 7, 2024

APIs allow software systems to communicate. APIs consist of a server and a client. The server handles requests and responses, while the client interacts with the API. Both sides use a common language like JSON or XML to share data and instructions.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!