# Zyte API automatic extraction

**Automatic extraction** gets you structured data from web data.

Automatic extraction supports [AI-powered extraction](#ai-extraction)
of e-commerce, article and job posting data from any website, as well as
**non-AI extraction** of search engine results.

You can use [Zyte API requests](../reference.md#zapi-reference) to get structured data
from webpages.

## Structured data types

In a [Zyte API request](../reference.md#zapi-reference), enable any of the following
fields to get matching structured data:

> [!NOTE]
> You can only enable 1 of these fields per Zyte API request.

> #### E-commerce
> [product](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/product) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/product)) ai
> [productList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productList) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/productList)) ai
> [productNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productNavigation) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/productNavigation)) ai

> #### Articles
> [article](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/article) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/article)) ai
> [articleList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/articleList) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/articleList)) ai
> [articleNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/articleNavigation) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/articleNavigation)) ai
> [forumThread](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/forumThread) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/forumThread)) ai

> #### Job postings
> [jobPosting](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/jobPosting) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/jobPosting)) ai
> [jobPostingNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/jobPostingNavigation) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/jobPostingNavigation)) ai

> #### Generic
> [pageContent](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/pageContent) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/pageContent)) ai

> #### Search Engine Results
> 
> [serp](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/serp) ([output](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/serp)) non-ai

### Example

> [!NOTE]
> Install and configure [code example requirements](https://docs.pytest.org/en/stable/example/index.html#examples) and
> the [Zyte CA certificate](../../../misc/ca.md#ca) to run the example below.

### CLI client

input.jsonl
```json
{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "product": true}
```

```shell
zyte-api input.jsonl \
    | jq --raw-output .product
```

### curl

input.json
```json
{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "product": true
}
```

```shell
curl \
    --user YOUR_ZYTE_API_KEY: \
    --header 'Content-Type: application/json' \
    --data @input.json \
    --compressed \
    https://api.zyte.com/v1/extract \
    | jq --raw-output .product
```

### Python client

```python
import asyncio
import json

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    api_response = await client.get(
        {
            "url": (
                "https://books.toscrape.com/catalogue"
                "/a-light-in-the-attic_1000/index.html"
            ),
            "product": True,
        }
    )
    product = api_response["product"]
    print(json.dumps(product, indent=2, ensure_ascii=False))


asyncio.run(main())
```

### Scrapy

```python
from scrapy import Request, Spider


class BooksToScrapeComSpider(Spider):
    name = "books_toscrape_com"

    async def start(self):
        yield Request(
            (
                "https://books.toscrape.com/catalogue"
                "/a-light-in-the-attic_1000/index.html"
            ),
            meta={
                "zyte_api_automap": {
                    "product": True,
                },
            },
        )

    def parse(self, response):
        product = response.raw_api_response["product"]
        print(product)
```

Output (first 5 lines):

```json
{
  "name": "A Light in the Attic",
  "price": "51.77",
  "currency": "GBP",
  "currencyRaw": "£",
```

## AI-powered extraction

Automatic extraction uses AI-powered extraction for the following structured
data types: [product](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/product), [productList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productList),
[productNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productNavigation), [article](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/article),
[articleList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/articleList), [articleNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/articleNavigation),
[forumThread](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/forumThread), [jobPosting](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/jobPosting),
[jobPostingNavigation](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/jobPostingNavigation), [pageContent](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/pageContent).

AI-powered extraction also supports [LLM-based extraction of custom
attributes](custom-attributes.md#custom-attributes), as well as:
[geolocation](../features.md#zapi-geolocation),
[IP type](../features.md#zapi-ip-type),
[cookies](../features.md#zapi-cookies),
[sessions](../features.md#zapi-sessions),
[redirection](../http.md#zapi-redirection),
[response headers](../features.md#zapi-headers), and
[metadata](../features.md#zapi-metadata),
plus additional features depending on your
[extraction source](#zapi-extract-from).

### Extraction source

Use the corresponding `extractFrom` option, e.g.
[productOptions.extractFrom](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productOptions.extractFrom) when extracting a
[product](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/product), to indicate which sources to use for automatic
extraction:

- `httpResponseBody` extracts from [httpResponseBody](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/httpResponseBody). It is
  usually faster and cheaper.
- `browserHtmlOnly` extracts from [browserHtml](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/browserHtml). It
  typically improves quality over `httpResponseBody` on JavaScript-heavy
  web pages.
- `browserHtml` extracts from both [browserHtml](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/response/200/browserHtml) and visual
  features of the rendered web page. It typically improves quality over
  `browserHtmlOnly`, but is not as robust in case of rendering issues.
- `userHtml` extracts from HTML you provide directly. See
  [Bring your own HTML (userHtml)](#zapi-user-html).

If not specified, `browserHtml` is currently used by default for [AI
extraction](#ai-extraction), while `httpResponseBody` is used by default for
[non-AI extraction](#non-ai-extraction). In the future, the default value
may depend on the target website.

Automatic extraction using an HTTP request (`httpResponseBody`) supports HTTP
request attributes for [method](../http.md#zapi-set-method), [body](../http.md#zapi-set-body), and [headers](../http.md#zapi-body-request-headers).

Automatic extraction using a browser request (`browserHtmlOnly` or
`browserHtml`) supports [browser HTML](../browser.md#zapi-browser-html),
[screenshots](../browser.md#zapi-screenshot), [some request headers](../browser.md#zapi-set-browser-headers), [actions](../browser.md#zapi-actions), [network
capture](../browser.md#zapi-network-capture), and [toggling JavaScript](../browser.md#zapi-javascript). The [limitations of browser requests](../browser.md#zapi-browser-limitations) also apply in this case.

### Bring your own HTML (`userHtml`)

With `userHtml`, you send pre-fetched HTML directly for extraction instead of
having Zyte API download the page. Set `extractFrom` to `"userHtml"` in the
matching options object and provide the HTML in the `userHtml` field of the
same object:

```json
{
  "url": "https://www.example.com/product/123",
  "product": true,
  "productOptions": {
    "extractFrom": "userHtml",
    "userHtml": "<html>...</html>"
  }
}
```

This works for all AI extraction types: [product](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/product),
[productList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productList), [article](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/article),
[articleList](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/articleList), and others.

The same extraction model is used as for `extractFrom: httpResponseBody`.

**URL and link resolution**

The [url](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/url) field is not downloaded. Its value is used only to
resolve relative links inside the HTML you provide, so it should match the
actual page the HTML came from if you need links to resolve correctly.

**Billing**

You are charged the extraction cost only — the download portion of the request
cost is not charged.

**Payload size**

The `userHtml` string is limited to 2.5 MiB. If you receive an error
indicating the payload is too large, contact Zyte support to have the limit
adjusted for your use case.

### Model pinning

The AI models of AI-powered extraction are retrained regularly, usually a few
times per year. While new model versions aim to improve overall accuracy, they
may become less accurate for specific fields of specific websites.

For certain data types, we provide an option to pin a specific model version,
which allows you to postpone an update to the latest model.

To pin a model, use the corresponding `model` option, e.g.
[productOptions.model](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/productOptions.model) when extracting a [product](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/product).

Model versions remain available for at least 1 year after their release. For
example, a product model version `"2024-02-01"` would remain available at
least until the 1st of February 2025.

When we decide to remove a model version, we announce its end-of-life date by
email to its users at least 3 months in advance, and we list that date in the
table below.

| Data type   | Model name   | Description           |
|-------------|--------------|-----------------------|
| product     | 2024-02-01   |                       |
| product     | 2024-09-16   | Default product model |