Stories from the Web Crawling trenches in proxy

Using Proxies in file_get_contents in PHP in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Proxying web requests in PHP using stream_context_create and file_get_contents. Adding authentication for secure proxies. Advanced HTTP options through stream contexts. Debugging common PHP proxy problems. Scraping via cURL. Leveraging Proxy-as-a-Service for robust web scraping with Proxies API.

How do I Make cURL Ignore the Proxy?

Author: Mohan Ganesan

Date: Jan 9, 2024

Unset HTTP_PROXY and HTTPS_PROXY environment variables. Set NO_PROXY to exclude specific hosts/domains from the proxy. Use --noproxy or related curl options to disable the proxy per request.

Making HTTP Requests Through a Proxy in Elixir with HTTPoison in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to install HTTPoison in Elixir, make requests, configure global and per-request proxies, use SOCKS proxies, handle authentication and TLS, and manage IP blocks and captchas with proxy rotation services.

Using Proxies With C++ httplib in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Using a proxy with C++ httplib is easy. Set up authentication, chain multiple proxies, customize settings, and troubleshoot issues. Proxies API offers a better solution for unblockable scraping.

HttpWebRequest Proxies in C# in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

The article explains how to direct HttpWebRequest traffic through a proxy using the WebProxy class. It covers creating a WebProxy, assigning it to HttpWebRequest, proxy authentication, default system proxy settings, and making requests via proxy.

Using AFNetworking Proxies for Web Scraping in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Setting up a basic AFNetworking proxy, working with different proxy protocols, advanced proxy functionality, troubleshooting common AFNetworking proxy problems.

Making HTTP Requests in Python Without a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Make HTTP requests in Python without a proxy using the requests library. Customize requests with headers, parameters, and handle timeouts.

Making Python Requests Appear Mobile

Author: Mohan Ganesan

Date: Feb 3, 2024

Make Python requests appear as mobile by setting User-Agent header, using mobile HTTP client library, or proxying through a mobile device.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!