Migrating from Smart Proxy Manager to Zyte API#

Learn how to migrate from Smart Proxy Manager to Zyte API.

Key differences#

The following table summarizes the feature differences between both products:

Feature

Smart Proxy Manager

Zyte API

API

Proxy

HTTP or proxy

Ban avoidance

Good

Great

Residential proxies

Add-on

Automatic (configurable)

Session management

Client-managed

Server-managed or client-managed

Geolocation

Manual

Automatic (configurable)

Browser HTML

No

Yes (HTTP API only, proxy mode support planned)

Screenshots

No

Yes (HTTP API only)

Browser actions

No

Yes (HTTP API only)

Network capture

No

Yes (HTTP API only)

HTTP redirection

Not followed

Followed by default, can be disabled

User throttling

Concurrency-based

Request-based

See also Parameter mapping below for some additional, lower-level differences.

Ban avoidance#

Smart Proxy Manager does a good job at avoiding bans through proxy rotation, ban detection, retrying algorithms, and browser mimicking through browser profiles.

Zyte API improves on it by using an actual browser, if that is required to prevent bans on a particular website.

Zyte API also supports webpage interaction.

Residential proxies#

Zyte API supports both data center and residential IP addresses. It automatically chooses the right type of IP address as needed, but it also allows you to force a specific IP type.

Session management#

While Smart Proxy Manager only supports client-managed sessions, Zyte API supports both client-managed sessions and server-managed sessions.

The main difference between both implementations of client-managed sessions is that, in Zyte API, it is the client (you), not the server, who generates the session ID. See session.id for details.

Additionally, for some scenarios, using browser actions can remove the need for multiple requests with a shared session.

Geolocation#

Both products let you choose which country of origin to use for a request.

However, with Zyte API you usually do not need to manually choose which country of origin to use for each request, because Zyte API automatically chooses the best country of origin based on the target website.

Smart Proxy Manager does support a richer list of countries of origin that you can set manually. However, if you let Zyte API choose the right country of origin, it can use additional countries not available for manual override.

Smart Proxy Manager also allows defining an account with a set of geolocations, so that requests using that account pick a geolocation from that set. Zyte API does not support this, you can either omit the geolocation and let Zyte API choose the best geolocation, or set a specific geolocation for a given request.

For more information, see Geolocation.

Authentication#

You cannot use your Smart Proxy Manager API key for Zyte API, you need to get a separate API key to use Zyte API.

Proxy mode#

Zyte API offers a proxy mode, which makes it easier to migrate from Smart Proxy Manager.

Note

Before you decide whether to use the proxy mode or the HTTP API, learn their differences.

To migrate, update your proxy endpoint and API key. You may also need to update some proxy headers as indicated below.

Warning

The proxy mode is not optimized for use in combination with browser automation tools. Consider using Zyte API’s browser automation features instead. See Migrating from browser automation to Zyte API.

The following example shows a basic request using Smart Proxy Manager:

using System;
using System.IO;
using System.Net;
using System.Text;

var proxy = new WebProxy("http://proxy.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var request = (HttpWebRequest)WebRequest.Create("https://toscrape.com");
request.Proxy = proxy;
request.PreAuthenticate = true;
request.AllowAutoRedirect = false;

var response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var httpResponseBody = reader.ReadToEnd();
reader.Close();
response.Close();

Console.WriteLine(httpResponseBody);

curl \
    --proxy proxy.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'proxy.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@proxy.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@proxy.zyte.com:8011"
        for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

After you install and configure scrapy-zyte-smartproxy, you can use Scrapy as usual and all requests will be proxied through Smart Proxy Manager automatically.

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

And this is an identical request using the proxy mode of Zyte API:

using System;
using System.Net;
using System.Net.Http;

var proxy = new WebProxy("http://api.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var httpClientHandler = new HttpClientHandler
{
    Proxy = proxy,
};

var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
var message = new HttpRequestMessage(HttpMethod.Get, "https://toscrape.com");
var response = client.Send(message);
var body = await response.Content.ReadAsStringAsync();

Console.WriteLine(body);

curl \
    --proxy api.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.apache.hc.client5.http.auth.AuthCache;
import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.CredentialsProvider;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.auth.BasicAuthCache;
import org.apache.hc.client5.http.impl.auth.BasicScheme;
import org.apache.hc.client5.http.impl.auth.CredentialsProviderBuilder;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.routing.DefaultProxyRoutePlanner;
import org.apache.hc.client5.http.protocol.HttpClientContext;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.ParseException;
import org.apache.hc.core5.http.io.entity.EntityUtils;

class Example {
  public static void main(final String[] args)
      throws InterruptedException, IOException, ParseException {

    HttpHost proxy = new HttpHost("api.zyte.com", 8011);
    DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);
    CredentialsProvider credentialsProvider =
        CredentialsProviderBuilder.create()
            .add(new AuthScope(proxy), "YOUR_API_KEY", "".toCharArray())
            .build();

    AuthCache authCache = new BasicAuthCache();
    BasicScheme basicAuth = new BasicScheme();
    authCache.put(proxy, basicAuth);
    HttpClientContext context = HttpClientContext.create();
    context.setCredentialsProvider(credentialsProvider);
    context.setAuthCache(authCache);

    CloseableHttpClient client =
        HttpClients.custom()
            .setRoutePlanner(routePlanner)
            .setDefaultCredentialsProvider(credentialsProvider)
            .build();

    HttpGet request = new HttpGet("https://toscrape.com");
    client.execute(
        request,
        response -> {
          HttpEntity entity = response.getEntity();
          String httpResponseBody = EntityUtils.toString(entity, StandardCharsets.UTF_8);
          System.out.println(httpResponseBody);
          return null;
        });
  }
}

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'api.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@api.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@api.zyte.com:8011" for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

# frozen_string_literal: true

require 'net/http'

url = URI('https://toscrape.com/')
proxy_host = 'api.zyte.com'
proxy_port = '8011'

http = Net::HTTP.new(url.host, url.port, proxy_host, proxy_port, 'YOUR_API_KEY', '')
http.use_ssl = true

r = http.start do |h|
  h.request(Net::HTTP::Get.new(url))
end

puts r.body

When using scrapy-zyte-smartproxy, set the ZYTE_SMARTPROXY_URL setting to "http://api.zyte.com:8011" and the ZYTE_SMARTPROXY_APIKEY setting to your API key for Zyte API.

Then you can continue using Scrapy as usual and all requests will be proxied through Zyte API automatically.

from scrapy import Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

HTTP API#

Tip

If you are using scrapy-zyte-smartproxy (previously scrapy-crawlera), see Migrating from scrapy-zyte-smartproxy to scrapy-zyte-api for detailed migration steps.

When migrating from Smart Proxy Manager to the HTTP API of Zyte API, the main challenge is switching from a proxy API to an HTTP API. To read its rich output data, you need JSON parsing and sometimes base64-decoding.

The following example shows a basic request using Smart Proxy Manager:

using System;
using System.IO;
using System.Net;
using System.Text;

var proxy = new WebProxy("http://proxy.zyte.com:8011", true);
proxy.Credentials = new NetworkCredential("YOUR_API_KEY", "");

var request = (HttpWebRequest)WebRequest.Create("https://toscrape.com");
request.Proxy = proxy;
request.PreAuthenticate = true;
request.AllowAutoRedirect = false;

var response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var httpResponseBody = reader.ReadToEnd();
reader.Close();
response.Close();

Console.WriteLine(httpResponseBody);

curl \
    --proxy proxy.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com

const axios = require('axios')

axios
  .get(
    'https://toscrape.com',
    {
      proxy: {
        protocol: 'http',
        host: 'proxy.zyte.com',
        port: 8011,
        auth: {
          username: 'YOUR_API_KEY',
          password: ''
        }
      }
    }
  )
  .then((response) => {
    const httpResponseBody = response.data
    console.log(httpResponseBody)
  })

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'https://toscrape.com', [
    'proxy' => 'http://YOUR_API_KEY:@proxy.zyte.com:8011',
]);
$http_response_body = (string) $response->getBody();
fwrite(STDOUT, $http_response_body);

Note

You need to install and configure our CA certificate for the requests library.

import requests

response = requests.get(
    "https://toscrape.com",
    proxies={
        scheme: "http://YOUR_API_KEY:@proxy.zyte.com:8011"
        for scheme in ("http", "https")
    },
)
http_response_body: bytes = response.content
print(http_response_body.decode())

After you install and configure scrapy-zyte-smartproxy, you can use Scrapy as usual and all requests will be proxied through Smart Proxy Manager automatically.

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        print(response.text)

And this is an identical request using the HTTP API of Zyte API:

Note

Install and configure code example requirements and the Zyte CA certificate to run the example below.

using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.All
};
HttpClient client = new HttpClient(handler);

var apiKey = "YOUR_API_KEY";
var bytes = Encoding.GetEncoding("ISO-8859-1").GetBytes(apiKey + ":");
var auth = System.Convert.ToBase64String(bytes);
client.DefaultRequestHeaders.Add("Authorization", "Basic " + auth);

client.DefaultRequestHeaders.Add("Accept-Encoding", "br, gzip, deflate");

var input = new Dictionary<string, object>(){
    {"url", "https://toscrape.com"},
    {"httpResponseBody", true}
};
var inputJson = JsonSerializer.Serialize(input);
var content = new StringContent(inputJson, Encoding.UTF8, "application/json");

HttpResponseMessage response = await client.PostAsync("https://api.zyte.com/v1/extract", content);
var body = await response.Content.ReadAsByteArrayAsync();

var data = JsonDocument.Parse(body);
var base64HttpResponseBody = data.RootElement.GetProperty("httpResponseBody").ToString();
var httpResponseBody = System.Convert.FromBase64String(base64HttpResponseBody);
input.jsonl#
{"url": "https://toscrape.com", "httpResponseBody": true}

zyte-api input.jsonl \
    | jq --raw-output .httpResponseBody \
    | base64 --decode \
    > output.html
input.json#
{
    "url": "https://toscrape.com",
    "httpResponseBody": true
}

curl \
    --user YOUR_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .httpResponseBody \
    | base64 --decode \
    > output.html

import com.google.common.collect.ImmutableMap;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Map;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.HttpHeaders;
import org.apache.hc.core5.http.ParseException;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.io.entity.StringEntity;

class Example {
  private static final String API_KEY = "YOUR_API_KEY";

  public static void main(final String[] args)
      throws InterruptedException, IOException, ParseException {
    Map<String, Object> parameters =
        ImmutableMap.of("url", "https://toscrape.com", "httpResponseBody", true);
    String requestBody = new Gson().toJson(parameters);

    HttpPost request = new HttpPost("https://api.zyte.com/v1/extract");
    request.setHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON);
    request.setHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate");
    request.setHeader(HttpHeaders.AUTHORIZATION, buildAuthHeader());
    request.setEntity(new StringEntity(requestBody));

    CloseableHttpClient client = HttpClients.createDefault();
    client.execute(
        request,
        response -> {
          HttpEntity entity = response.getEntity();
          String apiResponse = EntityUtils.toString(entity, StandardCharsets.UTF_8);
          JsonObject jsonObject = JsonParser.parseString(apiResponse).getAsJsonObject();
          String base64HttpResponseBody = jsonObject.get("httpResponseBody").getAsString();
          byte[] httpResponseBodyBytes = Base64.getDecoder().decode(base64HttpResponseBody);
          String httpResponseBody = new String(httpResponseBodyBytes, StandardCharsets.UTF_8);
          System.out.println(httpResponseBody);
          return null;
        });
  }

  private static String buildAuthHeader() {
    String auth = API_KEY + ":";
    String encodedAuth = Base64.getEncoder().encodeToString(auth.getBytes());
    return "Basic " + encodedAuth;
  }
}

const axios = require('axios')

axios.post(
  'https://api.zyte.com/v1/extract',
  {
    url: 'https://toscrape.com',
    httpResponseBody: true
  },
  {
    auth: { username: 'YOUR_API_KEY' }
  }
).then((response) => {
  const httpResponseBody = Buffer.from(
    response.data.httpResponseBody,
    'base64'
  )
})

<?php

$client = new GuzzleHttp\Client();
$response = $client->request('POST', 'https://api.zyte.com/v1/extract', [
    'auth' => ['YOUR_API_KEY', ''],
    'headers' => ['Accept-Encoding' => 'gzip'],
    'json' => [
        'url' => 'https://toscrape.com',
        'httpResponseBody' => true,
    ],
]);
$data = json_decode($response->getBody());
$http_response_body = base64_decode($data->httpResponseBody);

With the proxy mode, you always get a response body.

curl \
    --proxy api.zyte.com:8011 \
    --proxy-user YOUR_API_KEY: \
    --compressed \
    https://toscrape.com \
> output.html

from base64 import b64decode

import requests

api_response = requests.post(
    "https://api.zyte.com/v1/extract",
    auth=("YOUR_API_KEY", ""),
    json={
        "url": "https://toscrape.com",
        "httpResponseBody": True,
    },
)
http_response_body: bytes = b64decode(api_response.json()["httpResponseBody"])

import asyncio
from base64 import b64decode

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": "https://toscrape.com",
            "httpResponseBody": True,
        }
    )
    http_response_body = b64decode(api_response["httpResponseBody"]).decode()
    print(http_response_body)


asyncio.run(main())

In transparent mode, when you target a text resource (e.g. HTML, JSON), regular Scrapy requests work out of the box:

from scrapy import Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"
    start_urls = ["https://toscrape.com"]

    def parse(self, response):
        http_response_text: str = response.text

While regular Scrapy requests also work for binary responses at the moment, they may stop working in future versions of scrapy-zyte-api, so passing httpResponseBody is recommended when targeting binary resources:

from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    def start_requests(self):
        yield Request(
            "https://toscrape.com",
            meta={
                "zyte_api_automap": {
                    "httpResponseBody": True,
                },
            },
        )

    def parse(self, response):
        http_response_body: bytes = response.body

Output (first 5 lines):

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Scraping Sandbox</title>

See Zyte API usage documentation for richer Zyte API examples, covering more scenarios and features. See also Parameter mapping to migrate your request parameters.

If your code seems to run slower with Zyte API, see Optimizing Zyte API usage.

There is no easy way to use Zyte API to drive requests from browser automation tools. If you are using Smart Proxy Manager as a proxy for a browser automation tool, consider using Zyte API for your browser automation needs instead. See Migrating from browser automation to Zyte API.

Parameter mapping#

The following table shows a mapping of Smart Proxy Manager request headers and their corresponding proxy mode headers and Zyte API parameters:

Smart Proxy Manager

Zyte API (proxy mode)

Zyte API (HTTP API)

X-Crawlera-Client

Zyte-Client

User-Agent header

X-Crawlera-Cookies bc

See below

See below

X-Crawlera-JobId bc

Zyte-JobId

jobId

X-Crawlera-Max-Retries

Not planned

Not planned

X-Crawlera-No-Bancheck

Planned

Not planned

X-Crawlera-Profile bc

Zyte-Device *

See below

X-Crawlera-Profile-Pass bc

Zyte-Override-Headers *

See below

X-Crawlera-Region bc

Zyte-Geolocation

geolocation

X-Crawlera-Session bc

Zyte-Session-ID

session.id

X-Crawlera-Timeout

Not planned

Not planned

[Deprecated] X-Crawlera-Use-Https

N/A

N/A

Headers tagged with bc can be used in Zyte API proxy mode. See Header backward compatibility.

Replacing X-Crawlera-Cookies#

X-Crawlera-Cookies supports 3 values:

  • enable causes automatic cookies to override request cookies.

    To achieve this in Zyte API, do not set requestCookies.

  • disable causes request cookies to override automatic cookies.

    This is the default behavior of Zyte API, using requestCookies overrides automatic cookies.

  • discard causes both request cookies and automatic cookies to be discarded.

    To achieve this in Zyte API, do not set requestCookies, and set cookieManagement (Zyte-Cookie-Management in proxy mode) to discard.

Replacing X-Crawlera-Profile and X-Crawlera-Profile-Pass#

In general, you can replace X-Crawlera-Profile with the device Zyte API request parameter.

Mind, however, that the behavior of Zyte API is actually a middle ground between the desktop (or mobile) and pass values of X-Crawlera-Profile: browser-specific headers are always sent (unlike pass, which disables them altogether), but you can override them (unlike desktop or mobile, which force them unless you use X-Crawlera-Profile-Pass). See Request headers for more information.

Header backward compatibility#

When using Zyte API proxy mode, migrating to Zyte API proxy mode headers is recommended.

However, the following Smart Proxy Manager request headers can be used in Zyte API proxy mode: X-Crawlera-Cookies, X-Crawlera-JobId, X-Crawlera-Profile, X-Crawlera-Profile-Pass, X-Crawlera-Region, X-Crawlera-Session.

If any of these Smart Proxy Manager headers is used, the response will include X-Crawlera-Error if needed for the following error codes: banned, invalid_request, bad_auth, bad_proxy_auth, max_header_size_exceeded, internal_server_error, timeout, domain_forbidden.

Tip

To force getting X-Crawlera-Error on a request without Smart Proxy Manager request headers, add a no-op Smart Proxy Manager request header, e.g. X-Crawlera-Profile-Pass: Foo.

Unsubscribe from Smart Proxy Manager#

Once you have successfully migrated to Zyte API, remember to unsubscribe from Smart Proxy Manager. If in doubt, reach out to us.