CDP headless browser¶

Zyte API exposes a headless browser over the Chrome DevTools Protocol (CDP), letting you connect with Playwright, Puppeteer, or any other CDP-compatible library and drive that browser directly with your own scripts.

This is different from browser requests, where Zyte API drives the browser for you and returns HTML or a screenshot. With CDP you get a live browser connection and full control over what happens inside it.

Use CDP when you need to:

Run existing Playwright or Puppeteer scripts without rewriting them.
Handle complex, branching, or stateful workflows that cannot be expressed as a fixed sequence of actions, without deploying custom browser scripts to Zyte infrastructure.
Navigate multi-step user flows, fill forms, click, scroll, and interact with JavaScript-heavy websites.
Intercept and inspect network traffic during a session.
Debug sessions using browser developer tools.

Zyte manages the browser infrastructure, proxy routing, TLS fingerprinting, and CAPTCHA management, so you write scraping logic, not browser operations.

CDP vs browser requests¶

The following table maps browser request features to their CDP equivalent:

Browser request feature	CDP equivalent
Browser HTML	`page.content()`
Screenshot	`page.screenshot()`
Actions	Not needed: call your browser library directly.
Network capture	Your browser library’s network API, e.g. `page.on("response")` or `page.route()`.
Request headers	Your browser library’s request header API, e.g. `page.setExtraHTTPHeaders()`. Unlike browser requests, not limited to the `Referer` header.
JavaScript toggle	Your browser library’s API for it, e.g. Puppeteer’s `page.setJavaScriptEnabled()`.
Geolocation	The `proxy_region` query parameter.
IP type	The `proxy_type` query parameter.
Cookies	Your browser library’s cookie API. See Cookies.
Client-managed sessions	Not supported. See Sessions.
Server-managed sessions	Not supported. See Sessions.
Response headers	Your browser library’s network API, e.g. `page.on("response")`.
Metadata	Not needed: track state in your own script.

Sessions¶

CDP does not support client-managed or server-managed sessions: a CDP connection already gives you a single browser, with one IP address and one cookie jar, for the life of the connection, so there is nothing to opt into for that. What you cannot do is make a separate CDP connection, or an HTTP or browser request, reuse the IP address or cookie jar of an existing or past CDP session.

If you need that kind of continuity across multiple requests, use browser requests with sessions instead. Mind their own expiration limits: a client-managed session lasts at most 15 minutes, shorter than a single CDP session’s maximum ttl, while a server-managed session lasts up to 4 hours.

Browser requests also benefit from ban-avoidance work Zyte may do behind the scenes, at no extra cost to you, beyond what a single request needs. CDP gives you none of that: every session starts clean. What you get in exchange is full control over sessions, cookies, headers, navigation, and retries, so you can tune ban avoidance yourself, at the cost of a potentially lower success rate on sites where Zyte’s automatic measures would otherwise help.

Cookies¶

CDP does not expose the requestCookies and responseCookies request fields. Instead, you get full read and write access to cookies through your browser library’s own API, e.g. Playwright’s context.cookies() and context.addCookies(), or Puppeteer’s page.cookies() and page.setCookie().

Requirements¶

To use CDP with your Zyte API account:

Get a subscription or, if using a pay-as-you-go plan, set a spending limit. CDP sessions are not allowed on free trials. See Zyte API pricing.
Complete our Know Your Customer procedure.

Once both requirements are met, CDP access is enabled automatically. See Zyte API › Browser CDP in the Zyte dashboard for details.

Connecting¶

Connect to the https://browser.zyte.com/ endpoint with an Authorization header set to Basic <token>, where <token> is your Zyte API key and a colon (YOUR_ZYTE_API_KEY:) encoded in Base64.

Quick start

Python (Playwright)

import base64
from playwright.sync_api import sync_playwright

CDP_ENDPOINT = "https://browser.zyte.com/"

auth = base64.b64encode(b"YOUR_ZYTE_API_KEY:").decode()
headers = {"Authorization": f"Basic {auth}"}

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(CDP_ENDPOINT, headers=headers)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/scroll")
    page.evaluate(
        """async () => {
            let quoteCount = 0;
            while (true) {
                window.scrollTo(0, document.body.scrollHeight);
                await new Promise((resolve) => setTimeout(resolve, 500));
                const newCount = document.querySelectorAll(".quote").length;
                if (newCount === quoteCount) break;
                quoteCount = newCount;
            }
        }"""
    )
    print(page.locator(".quote").count())
    browser.close()

JS (Playwright)

const { chromium } = require("playwright");

const CDP_ENDPOINT = "https://browser.zyte.com/";

(async () => {
  const headers = {
    Authorization:
      "Basic " + Buffer.from("YOUR_ZYTE_API_KEY:").toString("base64"),
  };
  const browser = await chromium.connectOverCDP(CDP_ENDPOINT, { headers });
  const page = await browser.newPage();
  await page.goto("https://quotes.toscrape.com/scroll");
  await page.evaluate(async () => {
    let quoteCount = 0;
    while (true) {
      window.scrollTo(0, document.body.scrollHeight);
      await new Promise((resolve) => setTimeout(resolve, 500));
      const newCount = document.querySelectorAll(".quote").length;
      if (newCount === quoteCount) break;
      quoteCount = newCount;
    }
  });
  console.log(await page.locator(".quote").count());
  await browser.close();
})();

JS (Puppeteer)

const puppeteer = require("puppeteer-core");

const WS_ENDPOINT = "wss://pop-us.browser.zyte.com/wss";

(async () => {
  const headers = {
    Authorization:
      "Basic " + Buffer.from("YOUR_ZYTE_API_KEY:").toString("base64"),
  };
  const browser = await puppeteer.connect({
    browserWSEndpoint: WS_ENDPOINT,
    headers,
  });
  const page = await browser.newPage();
  await page.goto("https://quotes.toscrape.com/scroll");
  await page.evaluate(async () => {
    let quoteCount = 0;
    while (true) {
      window.scrollTo(0, document.body.scrollHeight);
      await new Promise((resolve) => setTimeout(resolve, 500));
      const newCount = document.querySelectorAll(".quote").length;
      if (newCount === quoteCount) break;
      quoteCount = newCount;
    }
  });
  console.log((await page.$$(".quote")).length);
  await browser.close();
})();

Python (scrapy-playwright)

settings.py¶

from w3lib.http import basic_auth_header

ZYTE_API_KEY = "YOUR_ZYTE_API_KEY"
PLAYWRIGHT_CDP_URL = "https://browser.zyte.com/"
PLAYWRIGHT_CDP_HEADERS = {"Authorization": basic_auth_header(ZYTE_API_KEY, "")}

spiders/quotes_toscrape_com.py¶

from scrapy import Request, Spider
from scrapy_playwright.page import PageMethod

SCROLL_TO_BOTTOM = """async () => {
    let quoteCount = 0;
    while (true) {
        window.scrollTo(0, document.body.scrollHeight);
        await new Promise((resolve) => setTimeout(resolve, 500));
        const newCount = document.querySelectorAll(".quote").length;
        if (newCount === quoteCount) break;
        quoteCount = newCount;
    }
}"""


class QuotesToScrapeComSpider(Spider):
    name = "quotes_toscrape_com"

    async def start(self):
        yield Request(
            "https://quotes.toscrape.com/scroll",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("evaluate", SCROLL_TO_BOTTOM),
                ],
            },
        )

    def parse(self, response):
        quote_count = len(response.css(".quote"))
        print(quote_count)

Output:

Query parameters¶

Configure a session with query parameters on the connect URL:

https://browser.zyte.com/?ttl=600&proxy_region=GB&proxy_type=residential

Parameter	Description
`proxy_region`	Geolocation of the session’s proxy, as the same 2-letter country code accepted by geolocation. Default: the most fitting geolocation based on the first target website.
`proxy_type`	IP type of the session’s proxy, `datacenter` or `residential`. Default: the type that best avoids bans on the first target website.
`ttl`	Session time-to-live, in seconds, from 15 to 3600. Default: 60. See Session duration.

Pricing¶

Sessions are billed in tier units, each costing the same as one browser request at the tier of the domain of the first page.goto() call. See Pricing @ zyte.com for tier prices.

The number of tier units that a session consumed is the higher of:

The number of page.goto() calls.
The session duration divided by 15 seconds and rounded up.

A single page load, done in under 15s, costs 1 unit, i.e. the same as a regular browser request. Longer or busier sessions cost more, whichever count is higher.

A session is billed once it has started, regardless of how it ends: whether you close the browser, delete it explicitly, let it run until its ttl expires, or the connection between Zyte and the browser is unexpectedly lost mid-session. A session is not billed if it never starts, or if Zyte ends it for its own reasons, e.g. a server restart.

Note

Sessions using a residential proxy, either Zyte’s automatic choice for the target website or forced with proxy_type=residential, have a bandwidth cap per unit. Sessions that exceed it are charged per-GB on the overage. Typical sessions are not affected.

Session duration¶

A session starts when you connect to the CDP endpoint and ends when Zyte receives the Browser.close CDP command.

Warning

Playwright’s browser.close() does not send Browser.close: it only disconnects your script from the session, without ending it. To end a session before its ttl, send Browser.close explicitly:

const cdp = await page.context().newCDPSession(page);
await cdp.send("Browser.close");

Puppeteer’s browser.close() does send Browser.close, ending the session.

Whether you end the session explicitly or disconnect without doing so, e.g. your script exiting or crashing, Zyte keeps the session, and its proxy allocation, reserved for you until it reaches its ttl, and you are billed for that time.

By default, a session’s ttl is 60 seconds. See Query parameters for how to set a different value. Zyte terminates the session once it reaches its ttl, and you are not billed for any time beyond it.