Using Smart Proxy Manager with Selenium#

Warning

zyte-smartproxy-ca.crt should be installed in your OS for the below code to work. You can follow these instructions in order to install it.

Note

Selenium requires language-specific libraries, WebDriver installed and an active Smart Proxy Manager account. Log in or signup here.

Note

All the code in this documentation has been tested with Geckodriver 0.30.0, Python 3.9.5 (Selenium 4.0.0), NodeJS v14.15.1 (selenium-webdriver 4.1.0) and Java 17.

Option 1 - Zyte SmartProxy Selenium#

  1. Install ChromeDriver for Chrome.

  2. Download and install Zyte SmartProxy Selenium:

    $ python3 -m pip install zyte-smartproxy-selenium
    
  3. Save the following code as a sample.py:

from zyte_smartproxy_selenium import webdriver

browser = webdriver.Chrome(spm_options={'spm_apikey': '<Smart Proxy Manager API KEY>'})
browser.get('https://toscrape.com')
browser.save_screenshot('screenshot.png')
browser.close()
  1. Run sample.py with:

    $ python3 sample.py
    

Option 2 - Zyte SmartProxy Headless Proxy#

  1. Setup the Zyte SmartProxy (formerly Crawlera) Headless Proxy as described in Using Headless Browsers with Zyte Smart Proxy Manager.

  2. Install geckodriver for Firefox.

Python#

Save the following code as a sample.py and run using python sample.py:

from selenium import webdriver

HEADLESS_PROXY = "localhost:3128"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
    "httpProxy": HEADLESS_PROXY,
    "sslProxy": HEADLESS_PROXY,
    "proxyType": "MANUAL",
}

with webdriver.Firefox() as driver:
    driver.get("https://toscrape.com")
    driver.save_screenshot("screenshot.png")

Javascript#

Save the following code as a sample.js and run using node sample.js:

let webdriver = require('selenium-webdriver');
let firefox = require('selenium-webdriver/firefox');
let proxy = require('selenium-webdriver/proxy');
const fs = require('fs').promises
let opts = new firefox.Options();

(async function example() {
    opts.setProxy(proxy.manual({http: 'localhost:3128'}));
    let driver = new webdriver.Builder()
        .forBrowser('firefox')
        .setFirefoxOptions(opts)
        .build();
    try {
        await driver.get("https://toscrape.com");
        let image = await driver.takeScreenshot()
        await fs.writeFile('screenshot.png', image, 'base64');
    }
    finally {
        await driver.quit();
    }
}());

Java#

A sample Java class:

import org.openqa.selenium.*;
import org.openqa.selenium.firefox.*;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;

public class SPMTest {

    public static void main(String[] args) throws InterruptedException, IOException {

        String headlessProxyHost = "localhost:3128";

        FirefoxOptions options = new FirefoxOptions();
        options.addArguments("--ignore-certificate-errors");

        Proxy proxy = new Proxy();
        proxy.setHttpProxy(headlessProxyHost);
        proxy.setSslProxy(headlessProxyHost);
        options.setCapability("proxy", proxy);

        WebDriver browser = new FirefoxDriver(options);

        String url = "https://toscrape.com";

        browser.get(url);
        System.out.println(browser.getPageSource());
        File image = ((TakesScreenshot) browser).getScreenshotAs(OutputType.FILE);
        FileUtils.copyFile(image, new File("screenshot.png"));
        browser.quit();
    }
}