Scrape Like a Pro! Get Your Free API Key

The Complete Puppeteer Cheatsheet

Mohan Ganesan
Dec 6, 2023 ยท 8 min read

Overview

Puppeteer is a Node.js library developed by Google for controlling headless Chrome and Chromium over the DevTools Protocol. It allows you to automate UI testing, scraping, screenshot testing, and more.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('<https://example.com>');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

Launching Browser

Launch a headless browser instance:

const browser = await puppeteer.launch();

Launch a full version of Chrome:

const browser = await puppeteer.launch({
  headless: false
});

Launch browser with custom args:

const browser = await puppeteer.launch({
  args: ['--start-maximized']
});

Custom launch options:

codepuppeteer.launch({
  executablePath: '/path/to/Chrome',// Custom Chrome binary
  product: 'firefox'// Launch Firefox instead
});

Creating Pages

Create a new page:

const page = await browser.newPage();

Create incognito page:

const context = await browser.createIncogniteBrowserContext();
const page = await context.newPage();

Access an existing page:

const pages = await browser.pages();
const page = pages[0];

Tabs

Switch between tabs/bring them into focus:

await page1.bringToFront();
await page2.bringToFront();

Actions

Navigate to URL:

await page.goto('<https://example.com>');

Click on element:

await page.click('#element');

Type into input:

await page.type('#input', 'Text');

Press keyboard keys:

await page.keyboard.press('Shift');

Upload files:

await page.setInputFiles('#upload', ['/path/to/file1', '/path/to/file2']);

Execute Javascript code on page:

const result = await page.evaluate(() => {
  return document.querySelector('#result').textContent;
});

Hover over element:

await page.hover('#element');

Capture screenshot:

await page.screenshot({path: 'screenshot.png'});

Emulate mobile device:

await page.emulate(puppeteer.devices['iPhone 6']);

Scroll into view:

await page.evaluate(el => el.scrollIntoView(), await page.$('.item'));

Type in iframe:

const frame = page.frames().find(f => f.name() === 'frame');
await frame.$eval('#input', el => el.value = 'Text');

Tap element on mobile:

await page.touchscreen.tap(200, 75);

Trigger drag and drop:

await page.mouse.down();
await page.mouse.move(0, 100);
await page.mouse.up();

Selectors

Get element by CSS Selector:

const nav = await page.$('nav');

Get multiple elements:

const items = await page.$$('.item');

Use XPath selectors:

const button = await page.$x('//*[@id="button"]');

Get text content:

const text = await page.textContent('.results');

Advanced Selectors

Use text selector:


const link = await page.$('a:text("Next")');

Visibility selector:

const hidden = await page.$('.element:hidden');

Attribute selector:

const checkbox = await page.$('input[type="checkbox"]');

XPath selector:

const submit = await page.$x('//button[@type="submit"]');

Get by text content:

const p = await page.$eval('p', el => el.innerText === 'Hello');

Query shadow DOM:

const shadow = await page.$(.element/shadow-root');
const text = await shadow.$eval('.text', el => el.textContent);

Accessibility Testing

Audit for issues:

const issues = await page.accessibility.audit({
  runA11yChecks: true
});

expect(issues).toHaveLength(0);

Check colors contrast:

const contrastratio = await page.$eval('.button', button => {
  const bgColor = window.getComputedStyle(button).backgroundColor;
  // compute contrast ratio
});

expect(contrastratio).toBeGreaterThan(4.5);

Tab focus order:

await page.keyboard.press('Tab');

const active = await page.evaluate(() => document.activeElement.id);
expect(active).toBe('username');

Debugging & Reporting

Trace console errors:

page.on('console', msg => {
  if (msg.type() === 'error') {
    console.error(msg.text());
  }
});

HTML report generation:

const html = '<h1>Test Report</h1>';
fs.writeFileSync('report.html', html);

Track test coverage:

const coverage = await page.coverage.startJSCoverage();

// run tests

const results = await coverage.stopJSCoverage();

Waiting

Wait for navigation:

await page.waitForNavigation();

Wait for selector:

await page.waitForSelector('div.loaded');

Wait fixed time:

await page.waitFor(1000); // Wait for 1 second

For function result:

await page.waitForFunction(() => window.fetchDone);

For XHR request:

await page.waitForRequest(request => request.url() === 'data.json');

Navigation with timeout:

await page.waitForNavigation({timeout: 60000});

Element for 30 seconds:

await page.waitForSelector('.item', {timeout: 30000});

Frames

Get page frames:

const frames = page.mainFrame().childFrames();

Set current frame:

const frame = page.frames().find(f => f.name() === 'frame');
await frame.evaluate(() => {
  // Run code inside frame
});

Inputs

Get HTML/text/attributes of element:

const html = await page.$eval('.item', el => el.outerHTML);

const text = await page.$eval('.item', el => el.innerText);

const class = await page.$eval('.item', el => el.getAttribute('class'));

Fill out and submit form:

await page.type('#input', 'Text');
await page.click('#submitButton');

Samples

Take screenshot of element:

const el = await page.$('.element');
await el.screenshot({path: 'element.png'});

Emulate device and viewport:

const devices = puppeteer.devices;
const iPhone = devices['iPhone 6'];

await page.emulate(iPhone);
await page.setViewport(iPhone.viewport);

Fetch resource timing data:

const metrics = await page.metrics();
const requests = metrics.requestfinished;

PDFs

Generate PDF report:

await page.pdf({
  path: 'page.pdf',
  format: 'A4'
});

Landscape oriented PDF:

await page.pdf({
  path: 'page.pdf',
  landscape: true
});

Events

Page load event:

page.once('load', () => {
  // Page loaded completely
});

Network request failed event:

page.on('requestfailed', request => {
  console.log(request.url + ' ' + request.failure().errorText);
});

Console message event:

page.on('console', msg => {
  console.log(`${msg.type()} ${msg.text()}`);
});

Authentication

Set user agent:

await page.setUserAgent('CustomAgent');

Set custom headers:

await page.setExtraHTTPHeaders({
  'Accept-Language': 'en-US'
});

Set cookies:

await page.setCookie({name: 'session', value: '1234'});

Set credentials:

await page.authenticate({
  username: 'user',
  password: 'pass'
});

Set bypass CSP:

await browser.launch({ignoreHTTPSErrors: true});

Use proxy server:

await page.authenticate({username: 'user', password: 'pass'});

Network

Set cache disabled:

await page.setCacheEnabled(false);

Set throttling rate:

await page.setRequestInterception(true);
page.on('request', request => {
  request.continue({
    throttling: 0.5 // 50% slower
  });
});

Mock response:

page.on('request', interceptedRequest => {
  interceptedRequest.respond({
    contentType: 'text/html',
    body: '<html>Mock page</html>'
  });
});

Mock redirect response:

await page.route('**/*', route => {
  route.continue({ url: '/mock-page.html' });
});

Mock 404 status:

page.on('request', route => {
  route.abort('notfound');
});

Advanced Usage

Wait for more complex conditions:

// Wait for text content to change
await page.waitForFunction(selector => {
  return document.querySelector(selector).textContent === 'Updated';
}, {}, selector);

// Wait until no network requests for 500ms
await page.waitForTimeout(500);

Handle popups and new tabs:

page.on('dialog', dialog => {
  dialog.accept(); // or dismiss()
});

const [popup] = await Promise.all([
  new Promise(resolve => browser.once('targetcreated', target => resolve(target.page()))),
  page.click('#open-popup'), // clicks a button that opens a popup
]);

await popup.waitForSelector('h1'); // wait for popup content

Stabilize flaky tests using automatic retries:

// Automatically retry failed steps up to 4 times
const autoRetry = require('puppeteer-autoretry');
autoRetry.setDefaults({ retries: 4 });

await autoRetry(page).type('#input', 'Text');

Touch Interactions

Tap on element:

await page.tap('button');

Scroll on mobile:

await page.touchscreen.scroll(50, 100);

Drag and drop:

await page.touchscreen.down();
await page.touchscreen.move(50, 100);
await page.touchscreen.up();

Geolocation and Permissions

Set geolocation:

await page.setGeolocation({latitude: 0, longitude: 0});

Grant camera access:

await page.grantPermissions(['camera']);

Advanced Use Cases

Submit forms and upload files:

// Submit form
await page.type('#email', 'test@example.com');
await page.click('#submit');

// Upload file
const input = await page.$('input#file');
input.uploadFile('/path/to/file.txt');

Scrape content from site:

// Extract text from all p elements
const texts = await page.$$eval('p', elements => {
  return elements.map(el => el.textContent);
});

Cross-browser visual testing:

const devices = puppeteer.devices;

for (const browserType of ['chromium', 'firefox', 'webkit']) {
  const browser = await puppeteer.launch({browserType});

  // Emulate devices and test
}

Visual Regression Testing

Compare screenshots:

const screenshot = await page.screenshot();

const diff = await visualDiff.compare(screenshot, 'baseline.png');
expect(diff.misMatchPercentage).toBeLessThan(0.01);

Parallel Testing

Test in parallel:

const browser = await puppeteer.launch();
const pagePromises = [
  browser.newPage(),
  browser.newPage()
];

const pages = await Promise.all(pagePromises);

// Run tests in parallel
await Promise.all([
  pages[0].goto('url1'),
  pages[1].goto('url2')
])

Tips and Tricks

Speed up executing by persisting context:

// Persist browser context
const browserContext = await browser.createIncogniteBrowserContext();

await browserContext.close();
await browserContext.waitForTarget(page => page.url() === 'about:blank');

// Restore context
const page = await browserContext.newPage();

Profile CPU usage during test:

await page.profiling.start({path: 'trace.json'});

// Run CPU intensive tasks

await page.profiling.stop();

Use incognito context:

const context = await browser.createIncogniteBrowserContext();
const page = await context.newPage();

Transfer cookies between contexts:

const context1 = await browser.createIncogniteBrowserContext();
const context2 = await browser.createIncogniteBrowserContext();

await context1.addCookies([cookieObj1, cookieObj2]);

const transferred = await context2.transferCookies(context1);

Work with persistent context:

const context = await browser.createPersistentContext();
await context.close();

// Restore later
const page = await context.newPage();

Test Automation Strategies

Reusable Page Object:

class LoginPage {

  constructor(page) {
    this.page = page;
  }

  async login(username, password) {
    await this.page.type('#username', username);
    await this.page.type('#password', password);
    await this.page.click('#submit');
  }

}

// Usage:
const loginPage = new LoginPage(page);
await loginPage.login('user1', '123456');

Synchronizing test sequence:

const [response] = await Promise.all([
  page.click('#submit'), // returns after request sent
  page.waitForNavigation() // resolves after page load
]);

// Assert response after navigation
expect(response.status()).toBe(200);

Retry failed test cases:

for (let retry = 0; retry < 3; retry++) {
  try {
    await loginPage.login('invalid', 'password');
    break; // Test passed, so we break
  } catch (error) {
    if (retry === 2) {
      throw error; // Fail after 3 retries
    }
    // retry test otherwise
  }
}

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


Try ProxiesAPI for free

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

<!doctype html>
<html>
<head>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
...


Tired of getting blocked while scraping the web?

Get access to 1,000 free API credits, no credit card required!

Try for free


X

Don't leave just yet!

Enter your email below to claim your free API key: