Periodic Jobs API

The periodic jobs API lets you list, create, update, and delete scheduled spider or script runs for a project. Each periodic job defines one or more tasks to run on a cron schedule.

Note

Most of the features provided by the API are also available through the python-scrapinghub client library.

Periodic job object

Field	Description
id	Periodic job ID.
disabled	If true, the job is paused and will not run on schedule.
project	Project ID.
tasks	List of task objects to run. See Task object.
addtags	Tags to attach to every job run triggered by this schedule.
type	Job type. Either "spider" or "script".
description	Human-readable description of the periodic job.
cron	Cron expression that controls the run schedule.

Task object

Field	Description
name	Spider or script name.
priority	Job priority. Supported values: 0 (lowest) to 4 (highest). Default: 2.
spider_args	Spider arguments as a JSON object.
script_args	Script arguments string.
jobq_id	Spider or script ID in the project/spider/job identifier.

Cron format

Cron expressions follow the standard five-field format:

minute  hour  day-of-month  month  day-of-week

Note

Only simple cron expressions are supported. Ranges (1-5), steps (*/2), and lists (1,3,5) are not supported. Each field must be either a specific value or *.

Valid example: 0 9 * * 1 (every Monday at 09:00)

Invalid examples: */15 * * * *, 0 9,17 * * *, 0 9 1-5 * *

projects/:project_id/periodicjobs

Lists or creates periodic jobs for a project.

Parameter	Description	Required
project_id	Project ID.	Yes

Method	Description	Supported parameters
GET	List periodic jobs.	project_id
POST	Create a periodic job.	project_id

GET example:

$ curl -u YOUR_SCRAPY_CLOUD_API_KEY: \
    "https://app.zyte.com/api/projects/123/periodicjobs"
{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": 3,
      "disabled": true,
      "project": 123,
      "tasks": [
        {
          "name": "example",
          "priority": 2,
          "spider_args": {},
          "script_args": "",
          "jobq_id": 1
        }
      ],
      "addtags": [
        "periodic",
        "test"
      ],
      "type": "spider",
      "description": "",
      "cron": "0 9 * * 1"
    }
  ],
  "meta": {
    "suggested_hour": "0"
  }
}

The meta.suggested_hour field returns a recommended UTC hour to schedule jobs in order to distribute load across projects.

POST example:

$ curl -u YOUR_SCRAPY_CLOUD_API_KEY: -X POST \
    "https://app.zyte.com/api/projects/123/periodicjobs" \
    -H "Content-Type: application/json" \
    -d '{
      "cron": "0 14 * * 1",
      "tasks": [{"name": "example", "priority": 2, "spider_args": {}}],
      "addtags": ["periodic"],
      "disabled": false,
      "description": "My new job"
    }'
{
  "id": 6619,
  "disabled": false,
  "project": 123,
  "tasks": [
    {
      "name": "example",
      "priority": 2,
      "spider_args": {},
      "script_args": "",
      "jobq_id": 1
    }
  ],
  "addtags": ["periodic"],
  "type": "spider",
  "description": "My new job",
  "cron": "0 14 * * 1"
}

projects/:project_id/periodicjobs/:periodic_job_id

Retrieves, updates, or deletes a single periodic job.

Parameter	Description	Required
project_id	Project ID.	Yes
periodic_job_id	Periodic job ID.	Yes

For PATCH, all body fields are optional. Only the fields included in the request body are updated; omitted fields retain their current values.

Method	Description	Supported parameters
GET	Retrieve a periodic job.	project_id, periodic_job_id
PATCH	Update a periodic job.	project_id, periodic_job_id
DELETE	Delete a periodic job.	project_id, periodic_job_id

PATCH example that updates the cron schedule:

$ curl -u YOUR_SCRAPY_CLOUD_API_KEY: -X PATCH \
    "https://app.zyte.com/api/projects/123/periodicjobs/3" \
    -H "Content-Type: application/json" \
    -d '{"cron": "30 18 * * 2"}'
{
  "id": 3,
  "disabled": true,
  "project": 123,
  "tasks": [
    {
      "name": "example",
      "priority": 2,
      "spider_args": {},
      "script_args": "",
      "jobq_id": 1
    }
  ],
  "addtags": ["periodic", "test"],
  "type": "spider",
  "description": "",
  "cron": "30 18 * * 2"
}

For script periodic jobs the same structure applies, with type set to "script" and name containing the script filename prefixed with "py:":

$ curl -u YOUR_SCRAPY_CLOUD_API_KEY: \
    "https://app.zyte.com/api/projects/123/periodicjobs/7"
{
  "id": 7,
  "disabled": false,
  "project": 123,
  "tasks": [
    {
      "name": "py:environ.py",
      "priority": 2,
      "spider_args": {},
      "script_args": "",
      "jobq_id": 5
    }
  ],
  "addtags": [],
  "type": "script",
  "description": "nightly env check",
  "cron": "0 1 * * *"
}

DELETE example:

$ curl -u YOUR_SCRAPY_CLOUD_API_KEY: -X DELETE \
    "https://app.zyte.com/api/projects/123/periodicjobs/3"

A successful delete returns HTTP 204 with an empty response body.