Stats API

Use the stats HTTP API to access Smart Proxy Manager usage data.

Authentication

This API uses HTTP Basic authentication. You’ll need to use your API key.

API endpoints

Endpoint root: https://crawlera-stats.scrapinghub.com

/stats

Smart Proxy Manager usage stats.

Stats object

Field	Description
time_gte	Start of interval. ISO 8601 formatted date
clean	Number of successful responses
failed	Number of unsuccessful responses
concurrency	80 percentile of concurrent connections
total_time	80 percentile of response time (milliseconds)
traffic	Total traffic (bytes)
domain	Website name

Parameters

Field	Description	Required
start_date	ISO 8601 formatted date. Defaults to 7 days ago from now	No
end_date	ISO 8601 formatted date. Defaults to UTC now	No
groupby	How to group results. Defaults to no grouping. Available values: max, hour, day, month, year “max” means group by the most granular datetime precision possible (5min)	No
users	Only fetch data for this set of users (comma separated)	No
limit	Number of desired items per page. Defaults to 500	No
after	Token for requesting next items on timeline	No
netlocs	Only fetch stats for this set of domains (comma separated)	No
domain-wise	Show stats grouped per domain (true/false)	No

Examples

Last 7 days traffic:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/'
{
    "limit":500,
    "after":””,
    "results": [
    {
        "time_gte":"2018-12-12T11:05:00+00:00",
        "failed":112275,
        "traffic":125085476006,
        "concurrency":2,
        "total_time":1930,
        "clean":3758963
    }]
}

Last 7 days traffic max resolution:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max'
{
    "limit":500,
    "after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
    "results": [
    {
        "clean":175,
        "total_time":2032,
        "time_gte":"2018-12-17T15:30:00+00:00",
        "failed":16,
        "concurrency":2,
        "traffic":3554065
    }
    ....
    {
        "clean":166,
        "total_time":2036,
        "time_gte":"2018-12-17T16:15:00+00:00",
        "failed":4,
        "concurrency":1,
        "traffic":11257159
    }]
}

Consume next page:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max&after=MHgxLjcwNWYzMmMwMDAwMDBwKzMw'
{
    "limit":500,
    "after":"MHgxLjcwNWYzNzcwMDAwMDBwKzMw",
    "results":[....]
}

One day traffic per hour:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?start_date=2019-01-01T00%3A00&end_date=2019-01-01T23%3A59&groupby=hour'
{
    “limit":500,
    “after”: “”,
    "results":[....]
}

Last 7 days traffic for two particular domains grouped by domain:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?netlocs=test1.com,test2.com&domain-wise=true'
{
    "limit":500,
    "after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
    "results": [
    {
        "clean":175,
        "total_time":2032,
        "time_gte":"2018-12-17T15:30:00+00:00",
        "failed":16,
        "concurrency":2,
        "traffic":3554065,
        "domain":"test1.com"
    },
    {
        "clean":166,
        "total_time":2036,
        "time_gte":"2018-12-17T16:15:00+00:00",
        "failed":4,
        "concurrency":1,
        "traffic":11257159,
        "domain":"test2.com"
    }]
}