Stories from the Web Crawling trenches in dependencies

Getting HTTP Requests Working in AWS Lambda with the Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

When building AWS Lambda functions in Python, developers often run into issues with the Requests library. This guide covers common problems and solutions for using Requests in Lambda.

Running Asyncio Web Apps with aiohttp in Docker

Author: Mohan Ganesan

Date: Mar 3, 2024

Dockerizing aiohttp web apps requires the right base image, dependencies, and config. Limit workers, use dynamic ports, and handle graceful shutdowns.

Beautiful Soup Installation

Author: Mohan Ganesan

Date: Oct 6, 2023

Python library Beautiful Soup is a popular tool for web scraping. Install it using pip in a virtual environment and manage dependencies for proper setup.

Resolving aiohttp Version Conflicts in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

Version conflicts occur when dependencies require incompatible package versions. Check package documentation for shared dependency versions. Use virtual environments to isolate packages and dependency versions. Upgrading to the latest compatible package release can often resolve conflicts.

Troubleshooting Error Code 1 When Installing aiohttp Python Package

Author: Mohan Ganesan

Date: Mar 3, 2024

Error code 1 when installing aiohttp or other Python packages with native C code can be caused by missing dependencies, incorrect gcc version, permissions issue, or corrupted build.

Resolving Telepot's Incompatible Aiohttp Version Error

Author: Mohan Ganesan

Date: Mar 3, 2024

Error encountered when installing Telepot library due to incompatible aiohttp version. Upgrade aiohttp or install compatible Telepot version. Use virtual environments for projects with incompatible dependencies.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!