Stories from the Web Crawling trenches in Concurrent Requests

Making Concurrent Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

When building applications with aiohttp in Python, it's common to need to make multiple requests concurrently rather than sequentially. Use asyncio.gather, reuse session, and avoid limits with asyncio.Semaphore for better performance.

Effective Strategies for Rate Limiting Asynchronous Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Making asynchronous HTTP requests in Python applications and effectively rate limiting them using queues, retrying failed requests with backoff, and monitoring usage to stay under limits.

Making Async HTTP Requests in Python with requests and asyncio

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests library provides API for HTTP requests. asyncio and aiohttp enable non-blocking requests. grequests uses asyncio for concurrent requests. asyncio is efficient for I/O heavy work.

Bypassing Cloudflare Error 1015 in PHP

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP address.

Bypassing Cloudflare Error 1015 in C++

Author: Mohan Ganesan

Date: Apr 15, 2024

If you're into web scraping, you've probably encountered the dreaded Cloudflare Error 1015. It's like hitting a brick wall when you're just trying to gather some data.

Bypassing Cloudflare Error 1015 in Python

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP address.

Bypassing Cloudflare Error 1015 in Java

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP address.

Bypassing Cloudflare Error 1015 in Rust

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP addresses and user agents.

Speed Up Your API Requests: 5 Simple Optimization Tips

Author: Mohan Ganesan

Date: Feb 3, 2024

Making API requests faster with async/await, setting timeout limits, caching options, using a CDN, and throttling concurrent requests.

Bypassing Cloudflare Error 1015 in CSharp

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP address.

Bypassing Cloudflare Error 1015 in R

Author: Mohan Ganesan

Date: Apr 15, 2024

Cloudflare Error 1015 occurs when web scraping due to rate limiting. To avoid it, add delays, limit concurrent requests, and rotate IP address.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!