Stats API#
Use the stats HTTP API to access Smart Proxy Manager usage data.
Authentication#
This API uses HTTP Basic authentication. You’ll need to use your API key.
API endpoints#
Endpoint root: https://crawlera-stats.scrapinghub.com
/stats#
Smart Proxy Manager usage stats.
Stats object#
Field |
Description |
---|---|
time_gte |
Start of interval. ISO 8601 formatted date |
clean |
Number of successful responses |
failed |
Number of unsuccessful responses |
concurrency |
80 percentile of concurrent connections |
total_time |
80 percentile of response time (milliseconds) |
traffic |
Total traffic (bytes) |
domain |
Website name |
Parameters#
Field |
Description |
Required |
---|---|---|
start_date |
ISO 8601 formatted date. Defaults to 7 days ago from now |
No |
end_date |
ISO 8601 formatted date. Defaults to UTC now |
No |
groupby |
How to group results. Defaults to no grouping. Available values: max, hour, day, month, year “max” means group by the most granular datetime precision possible (5min) |
No |
users |
Only fetch data for this set of users (comma separated) |
No |
limit |
Number of desired items per page. Defaults to 500 |
No |
after |
Token for requesting next items on timeline |
No |
netlocs |
Only fetch stats for this set of domains (comma separated) |
No |
domain-wise |
Show stats grouped per domain (true/false) |
No |
Examples#
Last 7 days traffic:
$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/'
{
"limit":500,
"after":””,
"results": [
{
"time_gte":"2018-12-12T11:05:00+00:00",
"failed":112275,
"traffic":125085476006,
"concurrency":2,
"total_time":1930,
"clean":3758963
}]
}
Last 7 days traffic max resolution:
$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max'
{
"limit":500,
"after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
"results": [
{
"clean":175,
"total_time":2032,
"time_gte":"2018-12-17T15:30:00+00:00",
"failed":16,
"concurrency":2,
"traffic":3554065
}
....
{
"clean":166,
"total_time":2036,
"time_gte":"2018-12-17T16:15:00+00:00",
"failed":4,
"concurrency":1,
"traffic":11257159
}]
}
Consume next page:
$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max&after=MHgxLjcwNWYzMmMwMDAwMDBwKzMw'
{
"limit":500,
"after":"MHgxLjcwNWYzNzcwMDAwMDBwKzMw",
"results":[....]
}
One day traffic per hour:
$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?start_date=2019-01-01T00%3A00&end_date=2019-01-01T23%3A59&groupby=hour'
{
“limit":500,
“after”: “”,
"results":[....]
}
Last 7 days traffic for two particular domains grouped by domain:
$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?netlocs=test1.com,test2.com&domain-wise=true'
{
"limit":500,
"after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
"results": [
{
"clean":175,
"total_time":2032,
"time_gte":"2018-12-17T15:30:00+00:00",
"failed":16,
"concurrency":2,
"traffic":3554065,
"domain":"test1.com"
},
{
"clean":166,
"total_time":2036,
"time_gte":"2018-12-17T16:15:00+00:00",
"failed":4,
"concurrency":1,
"traffic":11257159,
"domain":"test2.com"
}]
}