Stories from the Web Crawling trenches in reqwest

Using Proxies in reqwest with Rust in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Proxies are intermediaries that forward along your requests. Reqwest has first-class proxy support for routing requests through proxies. Proxy authentication, custom proxy rules, and bypassing proxies for certain domains are also covered. Advanced proxy usage techniques such as capturing traffic and using asynchronous proxies are discussed. Proxies API is recommended as a managed API service for proxy functionality.

Downloading Images from URLs in Rust

Author: Mohan Ganesan

Date: May 5, 2024

Learn how to download images efficiently using Rust with reqwest, hyper, surf, ureq, and attohttpc libraries.

Downloading Images from a Website with Rust and scraper

Author: Mohan Ganesan

Date: Oct 15, 2023

Learn how to use Rust and the reqwest and scraper crates to download all the images from a Wikipedia page.

Whats the equivalent of pythons request package for rust?

Author: Mohan Ganesan

Date: Feb 3, 2024

Rust is a systems programming language focused on performance, reliability, and efficiency. reqwest is a popular HTTP client library for Rust, providing a similar developer experience to Python's requests package.

Scraping Multiple Pages in Rust with reqwest and selectors

Author: Mohan Ganesan

Date: Oct 15, 2023

Web scraping in Rust using reqwest and selectors crates to extract data from multiple pages. Use proxies for scaling up scraping.

Scraping eBay Listings in Rust in 2023

Author: Mohan Ganesan

Date: Oct 5, 2023

Learn how to scrape and extract data from eBay listings using Rust, reqwest, and select crates.

Scraping Craigslist Listings with Rust

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Rust and the reqwest and selectors crates.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!