Migrating from Smart Proxy Manager to Zyte API

Learn how to migrate from Smart Proxy Manager to Zyte API.

Key differences

The following table summarizes the feature differences between both products:

Feature	Smart Proxy Manager	Zyte API
API	Proxy	HTTP or proxy
Ban avoidance	Good	Great
Residential proxies	Add-on	Automatic (configurable)
Session management	Client-managed	Server-managed or client-managed
Geolocation	Manual	Automatic (configurable)
Browser HTML	No	Yes (HTTP API only, proxy mode support planned)
Screenshots	No	Yes (HTTP API only)
Browser actions	No	Yes (HTTP API only)
Network capture	No	Yes (HTTP API only)
HTTP redirection	Not followed	Followed by default, can be disabled
User throttling	Concurrency-based	Request-based

See also Parameter mapping below for some additional, lower-level differences.

Ban avoidance

Smart Proxy Manager does a good job at avoiding bans through proxy rotation, ban detection, retrying algorithms, and browser mimicking through browser profiles.

Zyte API improves on it by using an actual browser, if that is required to prevent bans on a particular website.

Zyte API also supports webpage interaction.

Residential proxies

Zyte API supports both data center and residential IP addresses. It automatically chooses the right type of IP address as needed, but it also allows you to force a specific IP type.

Session management

While Smart Proxy Manager only supports client-managed sessions, Zyte API supports both client-managed sessions and server-managed sessions.

The main difference between both implementations of client-managed sessions is that, in Zyte API, it is the client (you), not the server, who generates the session ID. See session.id for details.

Additionally, for some scenarios, using browser actions can remove the need for multiple requests with a shared session.

Geolocation

Both products let you choose which country of origin to use for a request.

However, with Zyte API you usually do not need to manually choose which country of origin to use for each request, because Zyte API automatically chooses the best country of origin based on the target website.

Smart Proxy Manager does support a richer list of countries of origin that you can set manually. However, if you let Zyte API choose the right country of origin, it can use additional countries not available for manual override.

Smart Proxy Manager also allows defining an account with a set of geolocations, so that requests using that account pick a geolocation from that set. Zyte API does not support this, you can either omit the geolocation and let Zyte API choose the best geolocation, or set a specific geolocation for a given request.

For more information, see Geolocation.

Authentication

You cannot use your Smart Proxy Manager API key for Zyte API, you need to get a separate API key to use Zyte API.

Proxy mode

Zyte API offers a proxy mode. It is not as powerful as the HTTP API, but it makes it easier to migrate from Zyte Smart Proxy Manager.

To migrate, update your proxy endpoint and API key. You must also update your proxy headers as indicated below, unless you are using scrapy-zyte-smartproxy 2.3.1 or higher, which translates them automatically.

Warning

The proxy mode is not optimized for use in combination with browser automation tools. Consider using Zyte API’s browser automation features instead. See Migrating from browser automation to Zyte API.

The following example shows a basic request using Smart Proxy Manager:

C#

using System;
using System.IO;
using System.Net;
using System.Text;

var proxy = new WebProxy("http://proxy.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var request = (HttpWebRequest)WebRequest.Create("https://toscrape.com");
request.Proxy = proxy;
request.PreAuthenticate = true;
request.AllowAutoRedirect = false;

var response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var httpResponseBody = reader.ReadToEnd();
reader.Close();
response.Close();

Console.WriteLine(httpResponseBody);

curl

curl \
    --proxy proxy.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

JS

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'proxy.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

PHP

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@proxy.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Python

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@proxy.zyte.com:8011"
        for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

Scrapy

After you install and configure scrapy-zyte-smartproxy, you can use Scrapy as usual and all requests will be proxied through Smart Proxy Manager automatically.

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

And this is an identical request using the proxy mode of Zyte API:

C#

using System;
using System.IO;
using System.Net;
using System.Text;

var proxy = new WebProxy("http://api.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var request = (HttpWebRequest)WebRequest.Create("https://toscrape.com");
request.Proxy = proxy;
request.PreAuthenticate = true;
request.AllowAutoRedirect = false;

var response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var httpResponseBody = reader.ReadToEnd();
reader.Close();
response.Close();

Console.WriteLine(httpResponseBody);

curl

curl \
    --proxy api.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

JS

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'api.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

PHP

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@api.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Python

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@api.zyte.com:8011" for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

Scrapy

When using scrapy-zyte-smartproxy, set the ZYTE_SMARTPROXY_URL setting to "http://api.zyte.com:8011" and the ZYTE_SMARTPROXY_APIKEY setting to your API key for Zyte API.

Then you can continue using Scrapy as usual and all requests will be proxied through Zyte API automatically.

from scrapy import Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

HTTP API

To get access to all Zyte API features, the HTTP API is the way to go.

Tip

If you are using scrapy-zyte-smartproxy (previously scrapy-crawlera), see Migrating from scrapy-zyte-smartproxy to scrapy-zyte-api for detailed migration steps.

The main challenge then becomes switching from a proxy API to an HTTP API. Because Zyte API has a wider range of features and can hence provide a richer output, you need JSON parsing, and in some cases base64-decoding, to get your data.

The following example shows a basic request using Smart Proxy Manager:

C#

using System;
using System.IO;
using System.Net;
using System.Text;

var proxy = new WebProxy("http://proxy.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var request = (HttpWebRequest)WebRequest.Create("https://toscrape.com");
request.Proxy = proxy;
request.PreAuthenticate = true;
request.AllowAutoRedirect = false;

var response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var httpResponseBody = reader.ReadToEnd();
reader.Close();
response.Close();

Console.WriteLine(httpResponseBody);

curl

curl \
    --proxy proxy.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

JS

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'proxy.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

PHP

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@proxy.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Python

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@proxy.zyte.com:8011"
        for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

Scrapy

After you install and configure scrapy-zyte-smartproxy, you can use Scrapy as usual and all requests will be proxied through Smart Proxy Manager automatically.

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

And this is an identical request using the HTTP API of Zyte API:

Note

Install and configure code example requirements and the Zyte CA certificate to run the example below.

C#

using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.All
};
HttpClient client = new HttpClient(handler);

var apiKey = "YOUR_API_KEY";
var bytes = Encoding.GetEncoding("ISO-8859-1").GetBytes(apiKey + ":");
var auth = System.Convert.ToBase64String(bytes);
client.DefaultRequestHeaders.Add("Authorization", "Basic " + auth);

client.DefaultRequestHeaders.Add("Accept-Encoding", "br, gzip, deflate");

var input = new Dictionary<string, object>(){
    {"url", "https://toscrape.com"},
    {"httpResponseBody", true}
};
var inputJson = JsonSerializer.Serialize(input);
var content = new StringContent(inputJson, Encoding.UTF8, "application/json");

HttpResponseMessage response = await client.PostAsync("https://api.zyte.com/v1/extract", content);
var body = await response.Content.ReadAsByteArrayAsync();

var data = JsonDocument.Parse(body);
var base64HttpResponseBody = data.RootElement.GetProperty("httpResponseBody").ToString();
var httpResponseBody = System.Convert.FromBase64String(base64HttpResponseBody);

CLI client

input.jsonl

{"url": "https://toscrape.com", "httpResponseBody": true}

zyte-api input.jsonl \
    | jq --raw-output .httpResponseBody \
    | base64 --decode \
    > output.html

curl

input.json

{
    "url": "https://toscrape.com",
    "httpResponseBody": true
}

curl \
    --user YOUR_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .httpResponseBody \
    | base64 --decode \
    > output.html

Java

import com.google.common.collect.ImmutableMap;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Map;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.HttpHeaders;
import org.apache.hc.core5.http.ParseException;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.io.entity.StringEntity;

class Example {
  private static final String API_KEY = "YOUR_API_KEY";

  public static void main(final String[] args)
      throws InterruptedException, IOException, ParseException {
    Map<String, Object> parameters =
        ImmutableMap.of("url", "https://toscrape.com", "httpResponseBody", true);
    String requestBody = new Gson().toJson(parameters);

    HttpPost request = new HttpPost("https://api.zyte.com/v1/extract");
    request.setHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON);
    request.setHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate");
    request.setHeader(HttpHeaders.AUTHORIZATION, buildAuthHeader());
    request.setEntity(new StringEntity(requestBody));

    try (CloseableHttpClient client = HttpClients.createDefault()) {
      try (CloseableHttpResponse response = client.execute(request)) {
        HttpEntity entity = response.getEntity();
        String apiResponse = EntityUtils.toString(entity, StandardCharsets.UTF_8);
        JsonObject jsonObject = JsonParser.parseString(apiResponse).getAsJsonObject();
        String base64HttpResponseBody = jsonObject.get("httpResponseBody").getAsString();
        byte[] httpResponseBodyBytes = Base64.getDecoder().decode(base64HttpResponseBody);
        String httpResponseBody = new String(httpResponseBodyBytes, StandardCharsets.UTF_8);
      }
    }
  }

  private static String buildAuthHeader() {
    String auth = API_KEY + ":";
    String encodedAuth = Base64.getEncoder().encodeToString(auth.getBytes());
    return "Basic " + encodedAuth;
  }
}

JS

const axios = require('axios')

axios.post(
  'https://api.zyte.com/v1/extract',
  {
    url: 'https://toscrape.com',
    httpResponseBody: true
  },
  {
    auth: { username: 'YOUR_API_KEY' }
  }
).then((response) => {
  const httpResponseBody = Buffer.from(
    response.data.httpResponseBody,
    'base64'
  )
})

PHP

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('POST', 'https://api.zyte.com/v1/extract', [
    'auth' => ['YOUR_API_KEY', ''],
    'headers' => ['Accept-Encoding' => 'gzip'],
    'json' => [
        'url' => 'https://toscrape.com',
        'httpResponseBody' => true,
    ],
]);
$data = json_decode($response->getBody());
$http_response_body = base64_decode($data->httpResponseBody);

Proxy mode

With the proxy mode, you always get a response body.

curl \
    --proxy api.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com \
> output.html

Python

from base64 import b64decode

import requests

api_response = requests.post(
    "https://api.zyte.com/v1/extract",
    auth=("YOUR_API_KEY", ""),
    json={
        "url": "https://toscrape.com",
        "httpResponseBody": True,
    },
)
http_response_body: bytes = b64decode(api_response.json()["httpResponseBody"])

Python client

import asyncio
from base64 import b64decode

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": "https://toscrape.com",
            "httpResponseBody": True,
        }
    )
    http_response_body = b64decode(api_response["httpResponseBody"]).decode()
    print(http_response_body)


asyncio.run(main())

Scrapy

In transparent mode, when you target a text resource (e.g. HTML, JSON), regular Scrapy requests work out of the box:

from scrapy import Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        http_response_text: str = response.text

While regular Scrapy requests also work for binary responses at the moment, they may stop working in future versions of scrapy-zyte-api, so passing httpResponseBody is recommended when targeting binary resources:

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    def start_requests(self):
        yield Request(
            "https://toscrape.com",
            meta={
                "zyte_api_automap": {
                    "httpResponseBody": True,
                },
            },
        )

    def parse(self, response):
        http_response_body: bytes = response.body

Output (first 5 lines):

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Scraping Sandbox</title>

See Zyte API usage documentation for richer Zyte API examples, covering more scenarios and features.

If your code seems to run slower with Zyte API, see Optimizing Zyte API usage.

There is no easy way to use Zyte API to drive requests from browser automation tools. If you are using Smart Proxy Manager as a proxy for a browser automation tool, consider using Zyte API for your browser automation needs instead. See Migrating from browser automation to Zyte API.

Parameter mapping

The following table shows a mapping of Smart Proxy Manager request headers and their corresponding proxy mode headers and Zyte API parameters:

Smart Proxy Manager	Zyte API (proxy mode)	Zyte API (HTTP API)
X-Crawlera-Client	Zyte-Client	User-Agent header
X-Crawlera-Cookies bc	See below	See below
X-Crawlera-JobId bc	Zyte-JobId	jobId
X-Crawlera-Max-Retries	Not planned	Not planned
X-Crawlera-No-Bancheck	Planned	Not planned
X-Crawlera-Profile bc	Zyte-Device *	See below
X-Crawlera-Profile-Pass bc	Zyte-Override-Headers *	See below
X-Crawlera-Region	Zyte-Geolocation	geolocation
X-Crawlera-Session	Zyte-Session-ID	session.id
X-Crawlera-Timeout	Not planned	Not planned
[Deprecated] X-Crawlera-Use-Https	N/A	N/A

Headers tagged with bc can be used in Zyte API proxy mode. See Header backward compatibility.

Replacing X-Crawlera-Cookies

X-Crawlera-Cookies supports 3 values:

• enable causes automatic cookies to override request cookies.

  To achieve this in Zyte API, do not set requestCookies.

• disable causes request cookies to override automatic cookies.

  This is the default behavior of Zyte API, using requestCookies overrides automatic cookies.

• discard causes both request cookies and automatic cookies to be discarded.

  To achieve this in Zyte API, do not set requestCookies, and set cookieManagement (Zyte-Cookie-Management in proxy mode) to discard.

Replacing X-Crawlera-Profile and X-Crawlera-Profile-Pass

In general, you can replace X-Crawlera-Profile with the device Zyte API request parameter.

Mind, however, that the behavior of Zyte API is actually a middle ground between the desktop (or mobile) and pass values of X-Crawlera-Profile: browser-specific headers are always sent (unlike pass, which disables them altogether), but you can override them (unlike desktop or mobile, which force them unless you use X-Crawlera-Profile-Pass). See Request headers for more information.

Header backward compatibility

When using Zyte API proxy mode, migrating to Zyte API proxy mode headers is recommended.

However, the following Smart Proxy Manager request headers can be used in Zyte API proxy mode: X-Crawlera-Cookies, X-Crawlera-JobId, X-Crawlera-Profile, X-Crawlera-Profile-Pass.

If any of these Smart Proxy Manager headers is used, the response will include X-Crawlera-Error if needed for the following error codes: banned, invalid_request, bad_auth, bad_proxy_auth, max_header_size_exceeded, internal_server_error, timeout, domain_forbidden.

Tip

To force getting X-Crawlera-Error on a request without Smart Proxy Manager request headers, add a no-op Smart Proxy Manager request header, e.g. X-Crawlera-Profile-Pass: Foo.

Unsubscribe from Smart Proxy Manager

Once you have successfully migrated to Zyte API, remember to unsubscribe from Smart Proxy Manager. If in doubt, ask our support team.