Zyte API error handling#

While using Zyte API, you may get the following type of responses:

Successful responses#

Zyte API sends a successful response, i.e. a response with an HTTP status code of 200, when that response provides the requested data, ban-free.

A Zyte API response is considered successful even in the following scenarios:

  • The response from the target website is a bad response for a reason other than a ban.

  • Some browser actions have failed.

  • The webpage content does not match the specified automatic extraction property.

Bad website responses#

When a website sends a response with an HTTP status code other than 200, and that response is not the result of a ban, Zyte API sends that response to you.

For example, if you send a request to https://toscrape.com/not-found, you get a successful response from Zyte API, where the value of the statusCode response field is 404.

Browser action failures#

Browser action failures, such as timeouts, do not cause Zyte API to send an unsuccessful response.

Zyte API returns your requested output (e.g. browser HTML, screenshot) the way it was at the time of the action error, and the Zyte API response includes an actions field with details about the outcome of each action.

Automatic extraction mismatches#

Mismatches between the webpage content and the specified automatic extraction request field do not cause Zyte API to send an unsuccessful response.

Zyte API returns your requested output, including the metadata.probability field that indicates the probability that the specified automatic extraction property matches the webpage content.

Rate-limiting responses#

Note

You are not charged for rate-limiting responses.

Tip

scrapy-zyte-api and python-zyte-api handle rate limiting automatically.

Zyte API may send a response with an HTTP status code of 429 or 503 for rate-limiting purposes.

The right way to handle any rate-limiting response is to retry its request as many times as needed until you get a non-rate-limiting response.

Rate-limiting responses are sent in the following scenarios:

  • You have exceeded your API key rate limit.

    {"status": 429, "type": "/limits/over-user-limit"}

    After signup, Zyte API keys are allowed, by default, a maximum of 500 requests per minute (RPM). You may open a support ticket to request a higher rate limit for a given API key.

    There is no concurrency limit. In practice, concurrency is limited by your RPM limit and your response time. Given a 500 RPM limit, 5 s response times allow ~42 concurrent requests (500*5/60), while 200 ms response times allow ~2 concurrent requests (500*0.2/60).

    When making an efficient use of Zyte API, getting a small percentage of rate-limiting responses due to exceeding your API key rate limit is expected and normal.

  • The global rate limit for the target website has been exceeded.

    {"status": 429, "type": "/limits/over-domain-limit"}

    Each website has a different, website-specific rate limit. These limits exist to avoid causing issues on websites. You may open a support ticket to request a higher rate limit for a given website.

  • You have exceeded your account rate limit for the target website.

    {"status": 429, "type": "/limits/over-org-domain-limit"}

    Each Zyte API account also has its own rate limit for each website, so that no single account can hog traffic to any given website. You may open a support ticket to request a higher rate limit for a given website.

  • Zyte API is overloaded.

    {"status": 503, "type": "/limits/over-global-limit"}

See also

Rate limiting at Zyte API stats API.

Unsuccessful responses#

Note

You are not charged for unsuccessful responses.

Zyte API sends an unsuccessful response, i.e. a response with an HTTP status code of 400 or higher that is not a rate-limiting response, when Zyte API cannot provide the requested data.

Zyte API sends unsuccessful responses in the following scenarios:

Temporary download errors#

Tip

By default, scrapy-zyte-api and python-zyte-api automatically retry temporary download errors up to 3 times before giving up.

Zyte API sends an HTTP 520 response when a temporary error prevents downloading the requested URL.

{"status": 520, "type": "/download/temporary-error"}

You can retry your request until you get a different type of response.

If retries do not make a difference, please open a support ticket to report the issue.

Permanent download errors#

Zyte API sends an HTTP 521 response when a permanent error prevents downloading the requested URL.

{"status": 521, "type": "/download/internal-error"}

You can wait for us to address the issue, or open a support ticket and ask to be notified when the issue is resolved.

Tip

For some websites, Zyte API may sometimes accidentally flag some temporary download errors as permanent download errors. If sending the same Zyte API request multiple times returns an HTTP 521 error only sometimes, you might want to treat HTTP 521 errors as HTTP 520 errors for the target website, i.e. retry them automatically, until we resolve your issue report.

Service errors#

If Zyte API sends an HTTP 500 response, it means that the request took too long or that there was an unexpected issue in Zyte API.

{"status": 500, "type": "/server/timed-out"}

{"status": 500, "type": "/server/internal"}

If the issue persists, feel free to open a support ticket and ask to be notified when the issue is resolved.

Invalid requests#

Zyte API may send a response with an HTTP status code of 400, 401, or 422 if there is an error in your request, including:

  • You are using invalid parameters or parameter values.

    {"status": 400, "type": "/request/invalid"}

  • Your request body is invalid JSON.

    {"status": 400, "type": "/request/invalid-json"}

  • Your API key is not properly specified, e.g. missing or malformed.

    {"status": 401, "type": "/auth/not-valid"}

  • Your API key is unknown, e.g. it might be the wrong API key.

    {"status": 401, "type": "/auth/key-not-found"}

  • You are using incompatible parameters, such as mixing browserHtml and httpResponseBody.

    {"status": 422, "type": "/request/unprocessable"}

  • You are targeting a domain that Zyte API does not allow.

    {"status": 451, "type": "/download/domain-forbidden"}

Account suspension#

Zyte API sends an HTTP 403 response if you Zyte API account is suspended.

{"status": 403, "type": "/auth/account-suspended"}

Causes of account suspension include:

  • Reaching the end of your trial.

    Setting a spending limit lifts your account suspension immediately.

  • Reaching your spending limit.

    Increasing your spending limit lifts your account suspension immediately.

Retrying requests#

Tip

scrapy-zyte-api and python-zyte-api handle retries for rate limiting and temporary download errors automatically.

You should automatically retry requests that get a rate-limiting or a temporary download error response.

When retrying requests automatically, please use an exponential backoff algorithm: for any given request, increase the time between retries exponentially.

Ban handling#

A banned response is a response from a website that is different from the response that a person would have received.

Zyte API handles banned responses automatically and transparently, so that you never get a banned response.

For a given request, if Zyte API cannot avoid a banned response in a reasonable time, Zyte API sends you a temporary download error response, for which you are not charged. You can then retry your request as many times as needed until Zyte API succeeds.

We monitor and proactively work on improving Zyte API success rates and response times. However, if you wish, you may open a support ticket and ask to be notified when an issue that affects you is resolved.

If you ever get a successful Zyte API response that you believe is the result of a ban, please open a support ticket to report the issue.

Zyte API uses many different techniques to avoid bans. However, Zyte API does not log into websites automatically. Zyte API cannot automatically get you data that is always locked behind a user login.

By default, Zyte API may use residential IP addresses or CAPTCHA-management solutions. You may open a support ticket to request Zyte API not to use these features for your requests, e.g. for compliance reasons. You can also disable residential IP address usage per request, see IP type.