Logs API

The logs API lets you work with logs from your crawls.

Log object

Field

Description

Required

message

Log message.

Yes

level

Integer log level as defined in the table below.

Yes

time

UNIX timestamp of the message, in milliseconds.

No

Log levels

Value

Log level

10

DEBUG

20

INFO

30

WARNING

40

ERROR

50

CRITICAL

logs/:project_id/:spider_id/:job_id

Retrieve or upload logs for a given job.

Parameter

Description

Required

format

Results format. See Result formats.

No

Note

Pagination and meta parameters are supported, see Pagination and Meta parameters.

Method

Description

Supported parameters

GET

Retrieve logs.

format

POST

Upload logs.

Retrieving logs

HTTP:

$ curl -u APIKEY: https://storage.scrapinghub.com/logs/1111111/1/1/
{"time":1444822757227,"level":20,"message":"Log opened."}
{"time":1444822757229,"level":20,"message":"[scrapy.log] Scrapy 1.0.3.post6+g2d688cd started"}

Submitting logs

HTTP:

$ curl -u APIKEY: https://storage.scrapinghub.com/logs/53/34/7 -X POST -T log.jl