Warning

Zyte API is replacing Smart Proxy Manager. It is no longer possible to sign up to Smart Proxy Manager. If you are an existing Smart Proxy Manager user, see Migrating from Smart Proxy Manager to Zyte API.

Using Headless Browsers with Zyte Smart Proxy Manager#

Note

In this documentation you may see references to “Crawlera”, which was the previous name of “Smart Proxy Manager” service. They're effectively interchangeable.

What's different about headless browsers?#

Smart Proxy Manager does not provide raw IPs like most regular (data center) proxy providers. Instead, it delivers requests. It also introduces artificial delays to ensure websites are crawled politely while also increasing the success rates.

Because of the artificial delays, using the standard Smart Proxy Manager directly with a headless browser will cause the page to load very slowly and timeout frequently.

The Headless Proxy tool#

Smart Proxy Manager provides a tool that you can use to connect your headless browser with Smart Proxy Manager. This tool is called “Zyte SmartProxy (formerly Crawlera) Headless Proxy” and it’sbasically a proxy (yes, another proxy) that sits between Smart Proxy Manager and your headless browser.

The Headless Proxy takes care of the following tasks, which are typically not required for a non-headless environment (such as a Python script or Scrapy spider):

  • manages Smart Proxy Manager sessions automatically, as requests from a browser need to go out from the same IP and sessions is the way to achieve that with Smart Proxy Manager

  • blocks ads, to speed up page loads and save costs (remember Smart Proxy Manager is priced per requests)

  • directly downloads static assets (images, css, javascript, etc) to speed up page load and save costs in Smart Proxy Manager requests

For more details about how each of this work please refer to the Headless Proxy README in Github. Please note that Headless Proxy code is open source.

Installing the Headless Proxy tool#

The recommended way to install Headless Proxy is by downloading an executable, compiled for your platform. There are other ways to install the Headless Proxy, including via a Docker image. For more information check the project README.

Download the Headless Proxy from the Releases page. There are executables available for Windows, Linux, and Mac (darwin), Free BSD and Open BSD.

Download this config.toml file into the same folder of the Headless Proxy executable. This config comes with adblocking and direct access enabled. Some websites may not work with adblocking and direct access enabled. Try disabling it if you encounter any issues.

Install the Zyte CA certificate.

You are now ready to start the Headless Proxy by running:

crawlera-headless-proxy -c config.toml -a API_KEY

Note

If you are in recent Mac OS, you may need to give this executable permissions to run, as explained in this page. If you are in Windows, you may need to add the .exe extension to the file name before running. If you are in Linux, you may need to set executable permission with this command: chmod +x filename

Get your API_KEY from https://app.zyte.com/o/smart-proxy-manager/setup if you already have an account, and then go to the aforementioned URL.

Using the Headless Proxy tool#

Please note that while using the Headless Proxy, the first page may take more time to load while Smart Proxy Manager warms up a pool of clean IPs to the target website you’re trying to access. Typical times could be up to 30 seconds for the first page, and 2-5 seconds for subsequent page loads from the same site.

This warm up time should be seen as a small inconvenience. In exchange, you get a perpetually refeshing pool of healthy IPs, which translates to better performance in the long run.

Once you have setup the Headless Proxy, you can use Smart Proxy Manager with multiple headless browsers, please refer to the documentation of the tool you use for code samples and more specific instructions: