Warning

Zyte API is replacing Smart Proxy Manager. It is no longer possible to sign up to Smart Proxy Manager. If you are an existing Smart Proxy Manager user, see Migrating from Smart Proxy Manager to Zyte API.

Using Smart Proxy Manager with Selenium

Warning

For the code below to work you must first install the Zyte CA certificate.

Note

Selenium requires language-specific libraries, WebDriver installed and an active Smart Proxy Manager account. Log in here.

Note

All the code in this documentation has been tested with Geckodriver 0.30.0, Python 3.9.5 (Selenium 4.0.0), NodeJS v14.15.1 (selenium-webdriver 4.1.0) and Java 17.

Option 1 - Zyte SmartProxy Selenium

1. Install ChromeDriver for Chrome.

2. Download and install Zyte SmartProxy Selenium:

   $ python3 -m pip install zyte-smartproxy-selenium
   

3. Save the following code as a sample.py:

from zyte_smartproxy_selenium import webdriver

browser = webdriver.Chrome(spm_options={'spm_apikey': '<Smart Proxy Manager API KEY>'})
browser.get('https://toscrape.com')
browser.save_screenshot('screenshot.png')
browser.close()

4. Run sample.py with:

   $ python3 sample.py
   

Option 2 - Zyte SmartProxy Headless Proxy

1. Setup the Zyte SmartProxy (formerly Crawlera) Headless Proxy as described in Using Headless Browsers with Zyte Smart Proxy Manager.

2. Install geckodriver for Firefox.

Python

Save the following code as a sample.py and run using python sample.py:

from selenium import webdriver

HEADLESS_PROXY = "localhost:3128"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
    "httpProxy": HEADLESS_PROXY,
    "sslProxy": HEADLESS_PROXY,
    "proxyType": "MANUAL",
}

with webdriver.Firefox() as driver:
    driver.get("https://toscrape.com")
    driver.save_screenshot("screenshot.png")

Javascript

Save the following code as a sample.js and run using node sample.js:

let webdriver = require('selenium-webdriver');
let firefox = require('selenium-webdriver/firefox');
let proxy = require('selenium-webdriver/proxy');
const fs = require('fs').promises
let opts = new firefox.Options();

(async function example() {
    opts.setProxy(proxy.manual({http: 'localhost:3128'}));
    let driver = new webdriver.Builder()
        .forBrowser('firefox')
        .setFirefoxOptions(opts)
        .build();
    try {
        await driver.get("https://toscrape.com");
        let image = await driver.takeScreenshot()
        await fs.writeFile('screenshot.png', image, 'base64');
    }
    finally {
        await driver.quit();
    }
}());

Java

A sample Java class:

import org.openqa.selenium.*;
import org.openqa.selenium.firefox.*;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;

public class SPMTest {

    public static void main(String[] args) throws InterruptedException, IOException {

        String headlessProxyHost = "localhost:3128";

        FirefoxOptions options = new FirefoxOptions();
        options.addArguments("--ignore-certificate-errors");

        Proxy proxy = new Proxy();
        proxy.setHttpProxy(headlessProxyHost);
        proxy.setSslProxy(headlessProxyHost);
        options.setCapability("proxy", proxy);

        WebDriver browser = new FirefoxDriver(options);

        String url = "https://toscrape.com";

        browser.get(url);
        System.out.println(browser.getPageSource());
        File image = ((TakesScreenshot) browser).getScreenshotAs(OutputType.FILE);
        FileUtils.copyFile(image, new File("screenshot.png"));
        browser.quit();
    }
}