Stories from the Web Crawling trenches in proxy

How do I Make cURL Ignore the Proxy?

Author: Mohan Ganesan

Date: Jan 9, 2024

Unset HTTP_PROXY and HTTPS_PROXY environment variables. Set NO_PROXY to exclude specific hosts/domains from the proxy. Use --noproxy or related curl options to disable the proxy per request.

Using Proxies in file_get_contents in PHP in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Proxying web requests in PHP using stream_context_create and file_get_contents. Adding authentication for secure proxies. Advanced HTTP options through stream contexts. Debugging common PHP proxy problems. Scraping via cURL. Leveraging Proxy-as-a-Service for robust web scraping with Proxies API.

Making HTTP Requests Through a Proxy in Elixir with HTTPoison in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to install HTTPoison in Elixir, make requests, configure global and per-request proxies, use SOCKS proxies, handle authentication and TLS, and manage IP blocks and captchas with proxy rotation services.

Using Proxies With C++ httplib in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Using a proxy with C++ httplib is easy. Set up authentication, chain multiple proxies, customize settings, and troubleshoot issues. Proxies API offers a better solution for unblockable scraping.

HttpWebRequest Proxies in C# in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

The article explains how to direct HttpWebRequest traffic through a proxy using the WebProxy class. It covers creating a WebProxy, assigning it to HttpWebRequest, proxy authentication, default system proxy settings, and making requests via proxy.

Making Python Requests Appear Mobile

Author: Mohan Ganesan

Date: Feb 3, 2024

Make Python requests appear as mobile by setting User-Agent header, using mobile HTTP client library, or proxying through a mobile device.

Making HTTP Requests in Python Without a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Make HTTP requests in Python without a proxy using the requests library. Customize requests with headers, parameters, and handle timeouts.

Using AFNetworking Proxies for Web Scraping in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Setting up a basic AFNetworking proxy, working with different proxy protocols, advanced proxy functionality, troubleshooting common AFNetworking proxy problems.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!