# Migrating from browser automation to Zyte API

Learn how to migrate from browser automation tools, like [Playwright](https://playwright.dev/),
[Puppeteer](https://pptr.dev/), [Selenium](https://www.selenium.dev/), or [Splash](https://splash.readthedocs.io/en/stable/), to [Zyte API](../../get-started.md#zyte-api).

## Feature comparison

The following table summarizes the feature differences between Zyte API and
browser automation tools:

| Feature           | Zyte API                                           | Browser automation   |
|-------------------|----------------------------------------------------|----------------------|
| API               | HTTP                                               | Varies               |
| Website-aware API | [Yes](../../usage/browser.md#zapi-special-actions) | No                   |
| Avoid bans        | Yes                                                | Hard                 |
| Scalable          | Yes                                                | Hard                 |

## Migration examples

The following examples show common browser automation functionality implemented
using many browser automation tools, followed by an example of the same
functionality implemented using Zyte API. Use these examples to get started
porting your own code.

To learn more about the browser automation features of Zyte API, see
[Zyte API browser automation](../../usage/browser.md#zapi-browser).

If your code requires a non-linear flow or something else that cannot be
translated into a JSON array with a static sequence of [actions](../../usage/browser.md#zapi-actions), you may [need Zyte API browser scripts](../../ide/index.md#zapi-scripts).

### Getting browser HTML

This is how you get a browser [DOM](https://en.wikipedia.org/wiki/Document_Object_Model) rendered as HTML using browser automation
tools:

### scrapy-playwright

> [!NOTE]
> This example uses [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright).

```python
from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield Request(
            "https://toscrape.com",
            meta={"playwright": True},
        )

    def parse(self, response):
        browser_html: str = response.text
```

### scrapy-splash

> [!NOTE]
> This example uses [scrapy-splash](https://github.com/scrapy-plugins/scrapy-splash).

```python
from scrapy import Spider
from scrapy_splash import SplashRequest


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield SplashRequest("https://toscrape.com")

    def parse(self, response):
        browser_html: str = response.text
```

### Selenium

> [!NOTE]
> This example uses [Selenium](https://www.selenium.dev/) with [Python bindings](https://pypi.org/project/selenium/) for browser
> automation and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://toscrape.com")
browser_html = driver.page_source
driver.close()
```

### Splash

> [!NOTE]
> This example uses Python with [Splash](https://splash.readthedocs.io/en/stable/) for browser automation,
> [requests](https://requests.readthedocs.io/en/latest/) to use the HTTP API of Splash, and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from urllib.parse import quote

import requests

splash_url = "YOUR_SPLASH_URL"
url = "https://toscrape.com"
response = requests.get(f"{splash_url}/render.html?url={quote(url)}")
browser_html: str = response.content.decode()
```

And this is how you do it using Zyte API:

> [!NOTE]
> Install and configure [code example requirements](https://docs.pytest.org/en/stable/example/index.html#examples) and
> the [Zyte CA certificate](../../../misc/ca.md#ca) to run the example below.

### CLI client

input.jsonl
```json
{"url": "https://toscrape.com", "browserHtml": true}
```

```shell
zyte-api input.jsonl \
    | jq --raw-output .browserHtml
```

### curl

input.json
```json
{
    "url": "https://toscrape.com",
    "browserHtml": true
}
```

```shell
curl \
    --user YOUR_ZYTE_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .browserHtml
```

### Proxy mode

```shell
curl \
    --proxy api.zyte.com:8011 \
    --proxy-user YOUR_ZYTE_API_KEY: \
    --compressed \
    -H "Zyte-Browser-Html: true" \
    https://toscrape.com
```

### Python client

```python
import asyncio

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": "https://toscrape.com",
            "browserHtml": True,
        }
    )
    print(api_response["browserHtml"])


asyncio.run(main())
```

### Scrapy

```python
from scrapy import Request, Spider


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield Request(
            "https://toscrape.com",
            meta={
                "zyte_api_automap": {
                    "browserHtml": True,
                },
            },
        )

    def parse(self, response):
        browser_html: str = response.text
```

Output (first 5 lines):

```html
<!DOCTYPE html><html lang="en"><head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Scraping Sandbox</title>
        <link href="./css/bootstrap.min.css" rel="stylesheet">
        <link href="./css/main.css" rel="stylesheet">
```

See [Browser HTML](../../usage/browser.md#zapi-browser-html).

### Taking a screenshot

This is how you take a screenshot using browser automation tools:

### scrapy-playwright

> [!NOTE]
> This example uses [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright).

```python
from scrapy import Request, Spider
from scrapy_playwright.page import PageMethod


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield Request(
            "https://toscrape.com",
            meta={
                "playwright": True,
                "playwright_context": "new",
                "playwright_context_kwargs": {
                    "viewport": {"width": 1920, "height": 1080},
                },
                "playwright_page_methods": [
                    PageMethod("screenshot", type="jpeg"),
                ],
            },
        )

    def parse(self, response):
        screenshot: bytes = response.meta["playwright_page_methods"][0].result
```

### scrapy-splash

> [!NOTE]
> This example uses [scrapy-splash](https://github.com/scrapy-plugins/scrapy-splash).

```python
from scrapy import Spider
from scrapy_splash import SplashRequest


class ToScrapeSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield SplashRequest(
            "https://toscrape.com",
            endpoint="render.jpeg",
            args={
                "viewport": "1920x1080",
            },
        )

    def parse(self, response):
        screenshot: bytes = response.body
```

### Selenium

> [!NOTE]
> This example uses [Selenium](https://www.selenium.dev/) with [Python bindings](https://pypi.org/project/selenium/) for browser
> automation and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from io import BytesIO
from tempfile import NamedTemporaryFile

from PIL import Image
from selenium import webdriver

# https://stackoverflow.com/a/37183295
def set_viewport_size(driver, width, height):
    window_size = driver.execute_script(
        """
        return [window.outerWidth - window.innerWidth + arguments[0],
          window.outerHeight - window.innerHeight + arguments[1]];
        """,
        width,
        height,
    )
    driver.set_window_size(*window_size)


def get_jpeg_screenshot(driver):
    f = NamedTemporaryFile(suffix=".png")
    driver.save_screenshot(f.name)
    f.seek(0)
    image = Image.open(f)
    rgb_image = image.convert("RGB")
    image_io = BytesIO()
    rgb_image.save(image_io, format="JPEG")
    return image_io.getvalue()


driver = webdriver.Firefox()
set_viewport_size(driver, 1920, 1080)
driver.get("https://toscrape.com")
screenshot = get_jpeg_screenshot(driver)
driver.close()
```

### Splash

> [!NOTE]
> This example uses Python with [Splash](https://splash.readthedocs.io/en/stable/) for browser automation,
> [requests](https://requests.readthedocs.io/en/latest/) to use the HTTP API of Splash, and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from urllib.parse import quote

import requests

splash_url = "YOUR_SPLASH_URL"
url = "https://toscrape.com"
response = requests.get(f"{splash_url}/render.jpeg?url={quote(url)}&viewport=1920x1080")
screenshot: bytes = response.content
```

And this is how you do it using Zyte API:

> [!NOTE]
> Install and configure [code example requirements](https://docs.pytest.org/en/stable/example/index.html#examples) and
> the [Zyte CA certificate](../../../misc/ca.md#ca) to run the example below.

### CLI client

input.jsonl
```json
{"url": "https://toscrape.com", "screenshot": true}
```

```shell
zyte-api input.jsonl \
    | jq --raw-output .screenshot \
    | base64 --decode \
    > screenshot.jpg
```

### curl

input.json
```json
{
    "url": "https://toscrape.com",
    "screenshot": true
}
```

```shell
curl \
    --user YOUR_ZYTE_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .screenshot \
    | base64 --decode \
    > screenshot.jpg
```

### Python client

```python
import asyncio
from base64 import b64decode

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": "https://toscrape.com",
            "screenshot": True,
        }
    )
    screenshot = b64decode(api_response["screenshot"])
    with open("screenshot.jpg", "wb") as f:
        f.write(screenshot)


asyncio.run(main())
```

### Scrapy

```python
from base64 import b64decode

from scrapy import Request, Spider


class ToScrapeComSpider(Spider):
    name = "toscrape_com"

    async def start(self):
        yield Request(
            "https://toscrape.com",
            meta={
                "zyte_api_automap": {
                    "screenshot": True,
                },
            },
        )

    def parse(self, response):
        screenshot: bytes = b64decode(response.raw_api_response["screenshot"])
```

Output:

![](zyte-api/usage/code-examples/output/screenshot.jpg)

See [Screenshot](../../usage/browser.md#zapi-screenshot).

### Consuming scroll-based pagination

This is how you use browser automation tools to load a webpage on a web
browser, scroll to the bottom in a loop until it stops loading more content,
and get the resulting [DOM](https://en.wikipedia.org/wiki/Document_Object_Model) rendered as HTML:

### scrapy-playwright

> [!NOTE]
> This example uses [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright).

```python
from asyncio import sleep

from scrapy import Request, Spider


class QuotesToScrapeComSpider(Spider):
    name = "quotes_toscrape_com"

    async def start(self):
        yield Request(
            "https://quotes.toscrape.com/scroll",
            meta={
                "playwright": True,
                "playwright_include_page": True,
            },
        )

    # Based on https://stackoverflow.com/a/69193325
    async def scroll_to_bottom(self, page):
        await page.evaluate(
            """
            var scrollInterval = setInterval(
                function () {
                    var scrollingElement = (document.scrollingElement || document.body);
                    scrollingElement.scrollTop = scrollingElement.scrollHeight;
                },
                100
            );
            """
        )
        previous_height = None
        while True:
            current_height = await page.evaluate(
                "(window.innerHeight + window.scrollY)"
            )
            if not previous_height:
                previous_height = current_height
                await sleep(0.5)
            elif previous_height == current_height:
                await page.evaluate("clearInterval(scrollInterval)")
                break
            else:
                previous_height = current_height
                await sleep(0.5)

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await self.scroll_to_bottom(page)
        body = await page.content()
        response = response.replace(body=body)
        quote_count = len(response.css(".quote"))
        await page.close()
```

### scrapy-splash

> [!NOTE]
> This example uses [scrapy-splash](https://github.com/scrapy-plugins/scrapy-splash).

```python
from scrapy import Spider
from scrapy_splash import SplashRequest

# Based on https://stackoverflow.com/a/40366442
SCROLL_TO_BOTTOM_LUA = """
function main(splash)
    local num_scrolls = 10
    local scroll_delay = 0.1

    local scroll_to = splash:jsfunc("window.scrollTo")
    local get_body_height = splash:jsfunc(
        "function() {return document.body.scrollHeight;}"
    )
    assert(splash:go(splash.args.url))

    for _ = 1, num_scrolls do
        scroll_to(0, get_body_height())
        splash:wait(scroll_delay)
    end
    return splash:html()
end
"""


class QuotesToScrapeComSpider(Spider):
    name = "quotes_toscrape_com"

    async def start(self):
        yield SplashRequest(
            "https://quotes.toscrape.com/scroll",
            endpoint="execute",
            args={"lua_source": SCROLL_TO_BOTTOM_LUA},
        )

    def parse(self, response):
        quote_count = len(response.css(".quote"))
```

### Selenium

> [!NOTE]
> This example uses [Selenium](https://www.selenium.dev/) with [Python bindings](https://pypi.org/project/selenium/) for browser
> automation and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from time import sleep

from parsel import Selector
from selenium import webdriver

# Based on https://stackoverflow.com/a/69193325
def scroll_to_bottom(driver):
    driver.execute_script(
        """
        var scrollInterval = setInterval(
            function () {
                var scrollingElement = (document.scrollingElement || document.body);
                scrollingElement.scrollTop = scrollingElement.scrollHeight;
            },
            100
        );
        """
    )
    previous_height = None
    while True:
        current_height = driver.execute_script(
            "return window.innerHeight + window.scrollY"
        )
        if not previous_height:
            previous_height = current_height
            sleep(0.5)
        elif previous_height == current_height:
            driver.execute_script("clearInterval(window.scrollInterval)")
            break
        else:
            previous_height = current_height
            sleep(0.5)


driver = webdriver.Firefox()
driver.get("https://quotes.toscrape.com/scroll")
scroll_to_bottom(driver)
selector = Selector(driver.page_source)
quote_count = len(selector.css(".quote"))
driver.close()
```

### Splash

> [!NOTE]
> This example uses Python with [Splash](https://splash.readthedocs.io/en/stable/) for browser automation,
> [requests](https://requests.readthedocs.io/en/latest/) to use the HTTP API of Splash, and [Parsel](https://parsel.readthedocs.io/en/latest/) for HTML parsing.

```python
from urllib.parse import quote

import requests
from parsel import Selector

# Based on https://stackoverflow.com/a/40366442
SCROLL_TO_BOTTOM_LUA = """
function main(splash)
    local num_scrolls = 10
    local scroll_delay = 0.1

    local scroll_to = splash:jsfunc("window.scrollTo")
    local get_body_height = splash:jsfunc(
        "function() {return document.body.scrollHeight;}"
    )
    assert(splash:go(splash.args.url))

    for _ = 1, num_scrolls do
        scroll_to(0, get_body_height())
        splash:wait(scroll_delay)
    end
    return splash:html()
end
"""

splash_url = "YOUR_SPLASH_URL"
url = "https://quotes.toscrape.com/scroll"
response = requests.get(
    f"{splash_url}/execute?url={quote(url)}&lua_source={quote(SCROLL_TO_BOTTOM_LUA)}"
)
selector = Selector(text=response.content.decode())
quote_count = len(selector.css(".quote"))
```

And this is how you do it using Zyte API:

> [!NOTE]
> Install and configure [code example requirements](https://docs.pytest.org/en/stable/example/index.html#examples) and
> the [Zyte CA certificate](../../../misc/ca.md#ca) to run the example below.

### CLI client

input.jsonl
```json
{"url": "https://quotes.toscrape.com/scroll", "browserHtml": true, "actions": [{"action": "scrollBottom"}]}
```

```shell
zyte-api input.jsonl \
    | jq --raw-output .browserHtml \
    | xmllint --html --xpath 'count(//*[@class="quote"])' - 2> /dev/null
```

### curl

input.json
```json
{
    "url": "https://quotes.toscrape.com/scroll",
    "browserHtml": true,
    "actions": [
        {
            "action": "scrollBottom"
        }
    ]
}
```

```shell

curl \
    --user YOUR_ZYTE_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .browserHtml \
    | xmllint --html --xpath 'count(//*[@class="quote"])' - 2> /dev/null
```

### Python client

```python
import asyncio

from parsel import Selector
from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": "https://quotes.toscrape.com/scroll",
            "browserHtml": True,
            "actions": [
                {
                    "action": "scrollBottom",
                },
            ],
        },
    )
    browser_html = api_response["browserHtml"]
    quote_count = len(Selector(browser_html).css(".quote"))
    print(quote_count)


asyncio.run(main())
```

### Scrapy

```python
from scrapy import Request, Spider


class QuotesToScrapeComSpider(Spider):
    name = "quotes_toscrape_com"

    async def start(self):
        yield Request(
            "https://quotes.toscrape.com/scroll",
            meta={
                "zyte_api_automap": {
                    "browserHtml": True,
                    "actions": [
                        {
                            "action": "scrollBottom",
                        },
                    ],
                },
            },
        )

    def parse(self, response):
        quote_count = len(response.css(".quote"))
```

Output:

```none
100
```

See [Actions](../../usage/browser.md#zapi-actions).