Job metadata API#

The Job metadata API allows you to get metadata for the given jobs.


Most of the features provided by the API are also available through the python-scrapinghub client library.


Retrieve job data or specific meta field.


Get metadata for the job


$ curl -u APIKEY:

    "close_reason": "finished",
    "completed_by": "jobrunner",
    "deploy_id": 1,
    "finished_time": 1566311833872,
    "pending_time": 1566311800654,
    "priority": 2,
    "project": 1,
    "running_time": 1566311801163,
    "scheduled_by": "testuser",
    "scrapystats": {
        "downloader/request_bytes": 594,
        "downloader/request_count": 2,
        "downloader/request_method_count/GET": 2,
        "downloader/response_bytes": 1866,
        "downloader/response_count": 2,
        "downloader/response_status_count/200": 1,
        "downloader/response_status_count/404": 1,
        "elapsed_time_seconds": 3.211014,
        "finish_reason": "finished",
        "finish_time": 1566311822568.0,
        "item_scraped_count": 1,
        "log_count/DEBUG": 3,
        "log_count/INFO": 11,
        "log_count/WARNING": 1,
        "memusage/max": 72433664,
        "memusage/startup": 72433664,
        "response_received_count": 2,
        "robotstxt/request_count": 1,
        "robotstxt/response_count": 1,
        "robotstxt/response_status_count/404": 1,
        "scheduler/dequeued": 1,
        "scheduler/dequeued/disk": 1,
        "scheduler/enqueued": 1,
        "scheduler/enqueued/disk": 1,
        "start_time": 1566311819357.0
    "spider": "testspider",
    "spider_args": {"arg1": "val1", "arg2": "val2"},
    "spider_type": "manual",
    "started_by": "jobrunner",
    "state": "finished",
    "tags": [
    "units": 2,
    "version": "6d32f52-master"


Please consider the example response with caution. Some of the fields appear only on specific conditions: for example, after finishing/deleting or restoring a job. Some other fields highly depend on the given spider/job configuration. There also might be some additional fields for internal use only which can be changed at any given moment without prior notice.

Get specific metadata field for the job


$ curl -u APIKEY: