Scrape Any Website with OpenAI Function Calling in Objective-C

Sep 25, 2023 ยท 7 min read

Web scraping allows extracting data from websites programmatically. This is useful for gathering information like prices, inventory, reviews etc.

OpenAI provides an innovative approach to build robust web scrapers using natural language processing.

In this post, we will walk through a complete Objective-C code example that leverages OpenAI function calling to scrape product data from a sample ecommerce website.

Leveraging OpenAI Function Calling

OpenAI function calling provides a way to define schemas for the data you want extracted from a given input. When making an API request, you can specify a function name and parameters representing the expected output format.

OpenAI's natural language model will then analyze the provided input, extract relevant data from it, and return the extracted information structured according to the defined schema.

This pattern separates the raw data extraction capabilities of the AI model from your downstream data processing logic. Your code simply expects the data in a clean, structured format based on the function specification.

By leveraging OpenAI's natural language processing strengths for data extraction, you can create web scrapers that are resilient to changes in the underlying page structure and content. The business logic remains high-level and focused on data usage, while OpenAI handles the messy details of parsing and extracting information from complex HTML.

Why Use Function Calling

One key advantage of this web scraping technique is that the core scraper logic is immune to changes in the HTML structure of the target site. Since OpenAI is responsible for analyzing the raw HTML and extracting the desired data, the Objective-C code does not make any assumptions about HTML structure. The scraper will adapt as long as the sample HTML provided to OpenAI reflects the current page structure. This makes the scraper much more robust against site redesigns compared to scraping code that depends on specific HTML elements.

Overview

Here is an overview of the web scraping process we will implement:

  1. Send HTML representing the target page to OpenAI
  2. OpenAI analyzes the HTML and extracts the data we want
  3. OpenAI returns the extracted data structured as defined in our Objective-C function
  4. Process the extracted data in Objective-C as needed

This allows creating a scraper that adapts to changes in page layouts. The core logic stays high-level while OpenAI handles analyzing the raw HTML.

The Setup

To call the OpenAI API from Objective-C, you can use the openai-objc library:

#import <OpenAI/OpenAI.h>

This provides an Objective-C client for the OpenAI API.

Then in your code, you can create and initialize the OpenAI client:

OpenAIClient *client = [[OpenAIClient alloc] initWithToken:@"sk-..."];

This will allow calling API methods like -[OpenAIClient createCompletionWithRequest:] to send requests to OpenAI.

So the key pieces needed for the Objective-C integration are:

  • Objective-C
  • openai-objc library
  • OpenAI API key
  • With these dependencies configured, you can call the OpenAI API from Objective-C to implement the web scraping example using function calling.

    Sample HTML

    First, we need some sample HTML representing the page content we want to scrape:

    <div class="products">
    
      <div class="product">
        <h3>Blue T-Shirt</h3>
        <p>A comfortable blue t-shirt made from 100% cotton.</p>
        <p>Price: $14.99</p>
      </div>
    
      <div class="product">
        <h3>Noise Cancelling Headphones</h3>
        <p>These wireless over-ear headphones provide active noise cancellation.</p>
        <p>Price: $199.99</p>
      </div>
    
      <div class="product">
        <h3>Leather Laptop Bag</h3>
        <p>Room enough for up to a 15" laptop. Made from genuine leather.</p>
        <p>Price: $49.99</p>
      </div>
    
    </div>
    

    This contains 3 product listings, each with a title, description and price.

    Sending HTML to OpenAI

    Next, we need to send this sample HTML to the OpenAI API. The HTML is passed in the message content:

    OAIMessage *message = [OAIMessage messageWithContent:html];
    

    This will allow OpenAI to analyze the HTML structure.

    Defining Output Schema

    We need to define the expected output schema so OpenAI knows what data to extract.

    We'll define an extractedData function with a products array parameter:

    OAIFunction *function = [[OAIFunction alloc] initWithName:@"extractedData"];
    
    NSDictionary *parameters = @{
      @"type": @"array",
      @"items": @{
        @"type": @"object",
        @"properties": @{
          @"title": @{@"type": @"string"},
          @"description": @{@"type": @"string"},
          @"price": @{@"type": @"string"}
        }
      }
    };
    
    [function setParameters:parameters];
    

    This specifies we want an array of product objects, each with a title, description and price.

    Calling OpenAI API

    Now we can call the OpenAI API, passing the HTML and function:

    OAICompletionsRequest *request = [[OAICompletionsRequest alloc] init];
    [request addMessage:message];
    [request addFunction:function];
    
    OAICompletionsResponse *response = [client createCompletionWithRequest:request];
    

    This will analyze the HTML and return extracted data matching the schema we defined.

    Processing Extracted Data

    Finally, we can process the extracted data in our Objective-C method:

    -(void)extractedData:(NSArray<Product *> *)products {
    
      NSLog(@"Extracted Product Data:");
    
      for (Product *product in products) {
        NSLog(@"%@", product.title);
        NSLog(@"%@", product.description);
        NSLog(@"%@", product.price);
      }
    
    }
    

    This simply logs each product's details. We could also save the data to a database etc.

    So by leveraging OpenAI's function calling from Objective-C, we can build a robust web scraper that is resilient against changes in HTML structure. The core logic focuses on using the extracted data while OpenAI handles parsing the raw HTML.

    Full Code Example

    Here is the complete Objective-C code to scrape product data using OpenAI function calling:

    #import <Foundation/Foundation.h>
    #import <OpenAI/OpenAI.h>
    
    @interface Product : NSObject
    
    @property (nonatomic, strong) NSString *title;
    @property (nonatomic, strong) NSString *description;
    @property (nonatomic, strong) NSString *price;
    
    @end
    
    @implementation Product
    
    @end
    
    
    @interface ViewController : UIViewController
    
    -(void)extractedData:(NSArray<Product *> *)products;
    
    @end
    
    @implementation ViewController
    
    -(void)extractedData:(NSArray<Product *> *)products {
    
      NSLog(@"Extracted Product Data:");
    
      for (Product *product in products) {
        NSLog(@"%@", product.title);
        NSLog(@"%@", product.description);
        NSLog(@"%@", product.price);
      }
    
    }
    
    @end
    
    
    int main(int argc, const char * argv[]) {
    
      NSString *html = @"<div class=\\"products\\">...</div>";
    
      OpenAIClient *client = [[OpenAIClient alloc] initWithToken:@"sk-..."];
    
      OAIMessage *message = [OAIMessage messageWithContent:html];
    
      OAIFunction *function = [[OAIFunction alloc] initWithName:@"extractedData"];
      // ...
    
      OAICompletionsRequest *request = [[OAICompletionsRequest alloc] init];
      [request addMessage:message];
      [request addFunction:function];
    
      OAICompletionsResponse *response = [client createCompletionWithRequest:request];
    
      NSArray<Product *> *products = // parse response
    
      ViewController *vc = [ViewController new];
      [vc extractedData:products];
    
      return 0;
    }
    

    Conclusion

    Using OpenAI opens up an exciting new way to approach web scraping whih wasnt possible before

    However, this approach also has some limitations:

  • The scraped code needs to handle CAPTCHAs, IP blocks and other anti-scraping measures
  • Running the scrapers on your own infrastructure can lead to IP blocks
  • Dynamic content needs specialized handling
  • A more robust solution is using a dedicated web scraping API like Proxies API

    With Proxies API, you get:

  • Millions of proxy IPs for rotation to avoid blocks
  • Automatic handling of CAPTCHAs, IP blocks
  • Rendering of Javascript-heavy sites
  • Simple API access without needing to run scrapers yourself
  • With features like automatic IP rotation, user-agent rotation and CAPTCHA solving, Proxies API makes robust web scraping easy via a simple API:

    curl "https://api.proxiesapi.com/?key=API_KEY&url=targetsite.com"
    

    Get started now with 1000 free API calls to supercharge your web scraping!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!