Stories from the Web Crawling trenches in proxy server

How to Build a Simple HTTP Proxy in CSharp in just 25 lines of code

Author: Mohan Ganesan

Date: Oct 1, 2023

Build a basic proxy server in C# using the .NET framework. Use HttpListener and WebClient classes. Avoid IP blocking with rotating proxy service.

How to Build a Super Simple HTTP Proxy in JavaScript in just 20 lines of code

Author: Mohan Ganesan

Date: Oct 1, 2023

Build a basic proxy server with JavaScript using Node.js http and request modules. Avoid IP blocking with a rotating proxy service.

Debugging HTTP Requests with httpx Debug

Author: Mohan Ganesan

Date: Feb 5, 2024

Making HTTP requests is core functionality for many Python applications. httpx debug is a debugging proxy server that captures HTTP traffic, logs request/response data, and allows for mocking and modifying traffic for testing scenarios.

Accessing Web Content Through a Proxy Server with Python's urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Fetch web content in Python through a proxy server using urllib. Proxies provide security, network access control, and anonymity.

Scraping Craigslist Listings with CSharp

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using C# and HtmlAgilityPack. Avoid IP blocking with a rotating proxy server.

The Complete Guide to Datacenter Proxies

Author: Mohan Ganesan

Date: Jan 9, 2024

Datacenter proxies allow anonymous internet access. They act as intermediaries between users and websites, providing privacy and security. Forward proxies fetch web content for users, while reverse proxies distribute client traffic and add a protective layer. Datacenter proxies are used for accessing geo-restricted content, competitive price monitoring, gathering social media data, and more. Popular datacenter proxy providers include Bright Data, Oxylabs, and Smartproxy. Configuring datacenter proxies involves integrating server access credentials into programming scripts or browser settings. Choosing the right proxies depends on factors like shared vs. dedicated proxies, HTTP vs. SOCKS proxies, and rotating vs. static proxies. Pro tips for maximizing proxy usage include chaining multiple providers, automating IP cycling, persisting sessions, and caching common responses. Datacenter proxies are legal but usage should respect website terms. Proxies API is a SaaS platform that simplifies large-scale scraping by handling proxy configuration and rotation automatically.

Authenticating Requests Through a Proxy with Digest Auth in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Configure Python Requests module to handle proxy and digest authentication for secure access through authenticated proxy.

Scraping Craigslist Listings with Go

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Go and goquery. Avoid IP blocking with a rotating proxy server.

Troubleshooting Python Requests Through a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Common problems and solutions when sending requests through a proxy server in Python code.

What is a Transparent Proxy & How is It Used?

Author: Mohan Ganesan

Date: May 5, 2024

A transparent proxy is a proxy server that intercepts and forwards requests without modifying them. It offers benefits such as content filtering, bandwidth optimization, and enhanced network security.

Scraping Craigslist Listings with Perl

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Perl and modules LWP::UserAgent and HTML::TreeBuilder. Avoid IP blocking with a rotating proxy server.

Scraping Craigslist Listings with Ruby

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Ruby and Nokogiri. Avoid IP blocking with a rotating proxy server.

Scraping Craigslist Listings with Scala

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Scala and the play-ws library. Use XML parsing and a rotating proxy server to avoid IP blocking.

Scraping Craigslist Listings with Visual Basic

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Visual Basic and HtmlAgilityPack library. Avoid IP blocking with a rotating proxy server.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!