# Scrapy Cloud API Scrapy Cloud provides an HTTP API for interacting with your spiders, jobs and scraped data. ## Getting started ### Authentication You’ll need to authenticate using your [Scrapy Cloud API key](https://app.zyte.com/o/settings/apikey). > ##### IMPORTANT > > Scrapy Cloud uses a different API key than [Zyte API](../../../../zyte-api/get-started.md#zyte-api). There are two ways to authenticate: HTTP Basic: ``` $ curl -u YOUR_SCRAPY_CLOUD_API_KEY: https://storage.zyte.com/foo ``` URL Parameter: ``` $ curl https://storage.zyte.com/foo?apikey=YOUR_SCRAPY_CLOUD_API_KEY ``` ### Example Running a spider is simple: ``` $ curl -u YOUR_SCRAPY_CLOUD_API_KEY: https://app.zyte.com/api/run.json -d project=PROJECT -d spider=SPIDER ``` Where `YOUR_SCRAPY_CLOUD_API_KEY` is your Scrapy Cloud API key, `PROJECT` is the spider’s project ID, and `SPIDER` is the name of the spider you want to run. It’s possible to override Scrapy settings for a job: ``` $ curl \ -u YOUR_SCRAPY_CLOUD_API_KEY: \ https://app.zyte.com/api/run.json \ -d project=PROJECT \ -d spider=SPIDER \ -d job_settings='{"LOG_LEVEL": "DEBUG"}' ``` `job_settings` should be a valid JSON and will be merged with project and spider settings provided for given spider. ## API endpoints ### app.zyte.com * [Jobs API](jobs.md) * [run.json](jobs.md#run-json) * [jobs/list.{json,jl}](jobs.md#jobs-list-json-jl) * [jobs/update.json](jobs.md#jobs-update-json) * [jobs/delete.json](jobs.md#jobs-delete-json) * [jobs/stop.json](jobs.md#jobs-stop-json) * [Periodic Jobs API](periodicjobs.md) * [Periodic job object](periodicjobs.md#periodic-job-object) * [Cron format](periodicjobs.md#cron-format) * [projects/:project_id/periodicjobs](periodicjobs.md#projects-project-id-periodicjobs) * [projects/:project_id/periodicjobs/:periodic_job_id](periodicjobs.md#projects-project-id-periodicjobs-periodic-job-id) * [Comments API](comments.md) * [Comment object](comments.md#comment-object) * [comments/:comment_id](comments.md#comments-comment-id) * [comments/:project_id/:spider_id/:job_id](comments.md#comments-project-id-spider-id-job-id) * [comments/:project_id/stats](comments.md#comments-project-id-stats) * [comments/:project_id/:spider_id/:job_id/:item_no[/:field]](comments.md#comments-project-id-spider-id-job-id-item-no-field) ### storage.zyte.com * [JobQ API](jobq.md) * [jobq/:project_id/count](jobq.md#jobq-project-id-count) * [jobq/:project_id/list](jobq.md#jobq-project-id-list) * [Job metadata API](jobmeta.md) * [jobs/:project_id/:spider_id/:job_id[/:field_name]](jobmeta.md#jobs-project-id-spider-id-job-id-field-name) * [Items API](items.md) * [Item object](items.md#item-object) * [items/:project_id[/:spider_id][/:job_id][/:item_no][/:field_name]](items.md#items-project-id-spider-id-job-id-item-no-field-name) * [items/:project_id/:spider_id/:job_id/stats](items.md#items-project-id-spider-id-job-id-stats) * [Logs API](logs.md) * [Log object](logs.md#log-object) * [logs/:project_id/:spider_id/:job_id](logs.md#logs-project-id-spider-id-job-id) * [Requests API](requests.md) * [Request object](requests.md#request-object) * [requests/:project_id[/:spider_id][/:job_id][/:request_no]](requests.md#requests-project-id-spider-id-job-id-request-no) * [requests/:project_id/:spider_id/:job_id](requests.md#requests-project-id-spider-id-job-id) * [requests/:project_id/:spider_id/:job_id/stats](requests.md#requests-project-id-spider-id-job-id-stats) * [Activity API](activity.md) * [activity/:project_id](activity.md#activity-project-id) * [activity/projects](activity.md#activity-projects) * [Collections API](collections.md) * [Quickstart](collections.md#quickstart) * [Details](collections.md#details) * [API](collections.md#api) * [Frontier API](frontier.md) * [Batch object](frontier.md#batch-object) * [Request object](frontier.md#request-object) * [/hcf/:project_id/:frontier/s/:slot](frontier.md#hcf-project-id-frontier-s-slot) * [/hcf/:project_id/:frontier/s/:slot/q](frontier.md#hcf-project-id-frontier-s-slot-q) * [/hcf/:project_id/:frontier/s/:slot/q/deleted](frontier.md#hcf-project-id-frontier-s-slot-q-deleted) * [/hcf/:project_id/:frontier/s/:slot/f](frontier.md#hcf-project-id-frontier-s-slot-f) * [/hcf/:project_id/list](frontier.md#hcf-project-id-list) * [/hcf/:project_id/:frontier/list](frontier.md#hcf-project-id-frontier-list) ### Python client You can use the [python-scrapinghub](https://github.com/scrapinghub/python-scrapinghub) library to interact with Scrapy Cloud API. Check the [documentation](https://python-scrapinghub.readthedocs.io/) for installation instructions and usage examples. ## Pagination You can paginate the results for the majority of the APIs using a number of parameters. The pagination parameters differ depending on the target host for a given endpoint. ### app.zyte.com | Parameter | Description | |-------------|--------------------------------------| | count | Number of results per page. | | offset | Offset to retrieve specific records. | ### storage.zyte.com | Parameter | Description | |-------------|--------------------------------------------------------------------| | count | Number of results per page. | | index | Offset to retrieve specific records. Multiple values supported. | | start | Skip results before the given one. See a note about format below. | | startafter | Return results after the given one. See a note about format below. | > ##### NOTE > > The parameters naming inconsistency is caused by historical reasons and will be fixed in the coming platform updates. > ##### NOTE > > While `index` parameter is just a short `` (ex: `index=4`), `start` and `startafter` parameters should have the full form `///` (ex: `start=1/2/3/4`, `startafter=1/2/3/3`). ## Result formats There are two ways to specify the format of results: Using the `Accept` header, or using the `format` parameter. The `Accept` header supports the following values: * application/x-jsonlines * application/json * application/xml * text/plain * text/csv The `format` parameter supports the following values: * json * jl * xml * csv * text [XML-RPC data types](http://en.wikipedia.org/wiki/XML-RPC#Data_types) are used for XML output. ### CSV parameters | Parameter | Description | Required | |-----------------|-------------------------------------------------------------------------|------------| | fields | Comma delimited list of fields to include, in order from left to right. | Yes | | include_headers | When set to ‘1’ or ‘Y’, show header names in first row. | No | | sep | Separator character. | No | | quote | Quote character. | No | | escape | Escape character. | No | | lineend | Line end string. | No | When using CSV, you will need to specify the `fields` parameter to indiciate required fields and their order. Example: ``` $ curl -u YOUR_SCRAPY_CLOUD_API_KEY: "https://storage.zyte.com/items/53/34/7?format=csv&fields=id,name&include_headers=1" ``` ## Headers *gzip* compression is supported. A client can specify that *gzip* responses can be handled using the `accept-encoding: gzip` request header. `content-encoding: gzip` header must be present in the response to signal the *gzip* content encoding. You can use the `saveas` request parameter to specify a filename for browser downloads. For example, specifying `?saveas=foo.json` will cause a header of `Content-Disposition: Attachment; filename=foo.json` to be returned. ## Meta parameters You can use the `meta` parameter to return metadata for the record in addition to its core data. The following values are available: | Parameter | Description | |-------------|-----------------------------------------------------------------------| | \_key | The item key in the format `:project_id/:spider_id/:job_id/:item_no`. | | \_ts | Timestamp in milliseconds for when the item was added. | Example: ``` $ curl "https://storage.zyte.com/items/53/34/7?meta=_key&meta=_ts" {"_key":"1111111/1/1/0","_ts":1342078473363, ... } ``` > ##### NOTE > > If the data contains fields with the same name as the requested fields, they will both appear in the result.