Using AFNetworking Proxies for Web Scraping in 2024

Jan 9, 2024 ยท 6 min read

Setting up a Basic AFNetworking Proxy

The first step is importing AFNetworking and creating an AFHTTPSessionManager:

#import <AFNetworking.h>

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
AFHTTPSessionManager *manager = [[AFHTTPSessionManager alloc] initWithSessionConfiguration:configuration];

We can then make a request by specifying the proxy server's hostname and port:

[manager setSessionProxyConfiguration:@"HTTP" host:@"192.168.1.100" port:8080];

NSURL *url = [NSURL URLWithString:@"<https://api.example.com/data>"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];

[manager GET:url.absoluteString
   parameters:nil
     progress:nil
      success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {

			// Request succeeded

     } failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {

           // Request failed

}];

This proxies all requests through the server at 192.168.1.100 on port 8080.

Easy enough! Now let's look at configuring other proxy types...

Working with Different Proxy Protocols

AFNetworking supports the three major proxy protocols:

HTTP Proxy

These are the simplest and most common proxies on the internet. To use an HTTP proxy:

[manager setSessionProxyConfiguration:@"HTTP" host:@"proxy.example.com" port:3128];

HTTP proxies are easy to setup but lack encryption. For secure proxy connections, we need to use SOCKS instead.

SOCKS Proxy

SOCKS proxies route traffic at the TCP level for added privacy. Configuring them takes a bit more work:

NSDictionary *proxyConfig = @{
  (NSString *)kCFStreamPropertySOCKSProxyHost : @"proxy.example.com",
  (NSString *)kCFStreamPropertySOCKSProxyPort : @1080,
};

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
configuration.connectionProxyDictionary = proxyConfig;

AFHTTPSessionManager *manager = [[AFHTTPSessionManager alloc] initWithBaseURL:url sessionConfiguration:configuration];

We specify the SOCKS host and port in a proxy dictionary, and pass it to the session configuration.

One catch with SOCKS proxies is that domain name resolution happens on the client first. This leaks your actual IP address before traffic is routed through the proxy. To fix this, make sure your proxies support "remote DNS resolution".

Self-Signed SSL Certificate Proxy

If your proxy uses a custom SSL certificate, you need to implement the NSURLSessionDelegate method to allow the unrecognized cert initially:

- (void)URLSession: (NSURLSession *)session
              task: (NSURLSessionTask *)task
didReceiveChallenge: (NSURLAuthenticationChallenge *)challenge
 completionHandler: (void (^)(NSURLSessionAuthChallengeDisposition disposition, NSURLCredential *credential))completionHandler
{

    if ([challenge.protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust]) {
            NSURLCredential *credential = [NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust];
            completionHandler(NSURLSessionAuthChallengeUseCredential, credential);
        } else {
            completionHandler(NSURLSessionAuthChallengePerformDefaultHandling, nil);
        }

}

This allows what would normally be blocked as an invalid SSL certificate, by explicitly trusting it.

Advanced Proxy Functionality

Now that you know the basics, let's look at some advanced configuration and features available for AFNetworking proxies:

Authentication

If your proxy requires a username and password, you can specify them like so:

NSString *username = @"testuser";
NSString *password = @"testpass";

NSDictionary *proxyConfig = @{
   (NSString *)kCFStreamPropertyHTTPProxyHost : @"proxy.example.com",
   (NSString *)kCFStreamPropertyHTTPProxyPort : @8080,
   (NSString *)kCFStreamPropertyHTTPProxyUser : username,
   (NSString *)kCFStreamPropertyHTTPProxyPassword : password
};

Custom HTTP Headers

To add a custom header like User-Agent with your proxied requests:

[manager.requestSerializer setValue:@"My App Name" forHTTPHeaderField:@"User-Agent"];

Caching Support

To avoid repeat requests, you can check for cached responses first by specifying cachePolicy:

NSURLRequestCachePolicy policy = NSURLRequestUseProtocolCachePolicy; // Cache if directed by server headers

[manager GET:url cachePolicy:policy
   parameters:nil
     progress:nil
      success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {}

      failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {}
];

There are several other cache policies like forcing a reload that you can experiment with.

Debugging Tools

If you run into issues with your proxy implementation, enable logging to help diagnose:

manager.taskDidCompleteWithErrorBlock = ^(NSURLSession *session, NSURLSessionTask *task, NSError *error) {
 	// Log errors
  	if (error != nil) {
     NSLog(@"%@", error);
  	}
};

This lets you catch errors from individual requests.

Example: Building a Weather App with AFNetworking Proxies

To see proxies in action, let's walk through a simple weather app that fetches data from a third-party API:

// Import frameworks
@import AFNetworking;

// Create manager with proxy
AFHTTPSessionManager *manager = [AFHTTPSessionManager manager];
[manager setSessionProxyConfiguration:@"HTTP" host:@"proxy.example.com" port:8080];

// API and app keys
NSString *apiEndpoint = @"<https://api.weatherapi.com/v1/current.json>";
NSString *apiKey = @"123abc";

// city to fetch weather for
NSString *city = @"Boston";

// Parameter dict
NSDictionary *parameters = @{@"key": apiKey, @"q": city};

// Fetch weather data
[manager GET:apiEndpoint parameters:parameters progress:nil success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {

  // Parse JSON response
  NSDictionary *current = [responseObject valueForKey:@"current"];

  // Extract values
  NSString *conditionText = [current valueForKey:@"condition"]["text"];
  NSString *tempC = [current valueForKey:@"temp_c"];

  // Update UI
  self.conditionLabel.text = conditionText;
  self.temperatureLabel.text = tempC;

  NSLog(@"Success!");

} failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {

  NSLog(@"Error: %@", error);

}];

Here's what's happening step-by-step:

  1. Import AFNetworking
  2. Create an AFHTTPSessionManager instance and configure an HTTP proxy server
  3. Define the API endpoint, keys, and city search parameter
  4. Make a GET request with the search criteria
  5. On success, parse the JSON response
  6. Extract the current weather condition text and temperature
  7. Update the UI labels
  8. Handle any errors from the network request

And that's it! By routing requests through the proxy server, we can reliably access the remote weather API, even from areas that may be blocked or to bypass usage limits.

The proxy integration was only a few lines of additional code but unlocks lots of possibilities.

Let's wrap up with some best practices for diagnosing issues...

Troubleshooting Common AFNetworking Proxy Problems

Despite the simplicity above, I've faced my fair share of frustrations getting proxies in iOS apps to work properly.

Here are some common pitfalls and how I debug them:

SSL Errors

If you see SSL certificate errors like NSURLErrorDomain error -1012 connect failures when trying to use a proxy server, double check:

  • The set host and port match your proxy setup
  • You configured certificate/transport security exceptions correctly
  • The proxy server has SSL enabled if connecting via HTTPS
  • HTTP Status Codes

    If requests succeed but you receive unexpected HTTP codes like 403 or 500 errors, ensure:

  • The proxy IP isn't blocked from overuse
  • There are no connection limits or access rules on the proxy service blocking you
  • You have correctly allowed self-signed certs if relevant
  • Enabling debug logging is invaluable here to pinpoint if issues are from the proxy itself versus app code.

    JSON Parsing Failures

    If data loads fine over a proxy but JSON serialization is failing:

  • Verify the remote server content type headers match the serialized types
  • Log and inspect raw response data - JSON may be malformed
  • Handle edge cases like empty responses gracefully
  • Again, adding debug checks saves tons of time finding parsing issues.

    App Store Submissions

    When submitting your app using proxies to the iOS App Store:

  • Double check your proxy calls wrap in #if DEBUG blocks to disable for release builds
  • Omit proxy host details from any logs or analytics data
  • Consider allowing proxy configuration via app settings versus hardcoding server details
  • Following best practices here ensures a smooth review process.

    Try Proxies API for Simple, Scalable Web Scraping

    While AFNetworking proxies work well, managing scrapers involves dealing with proxies getting blocked, needing CAPTCHA solving, and handling IP rotations.

    If you just want to focus on writing your web scrapers, check out Proxies API:

    It provides a single API to fetch web pages that handles:

  • Automatic proxy rotation with millions of IPs to prevent blocks
  • Headless browsers rendering JavaScript
  • BUILT-IN CAPTCHA solving when needed
  • Simple usage in any language with just fetch API calls
  • There is even a generous free tier to get started.

    I recommend giving Proxies API a try if you want to skip proxy management headaches! It makes web scraping projects much easier to work on at scale.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!