Stats API

Use the stats HTTP API to access Smart Proxy Manager usage data.

Authentication

This API uses HTTP Basic authentication. You’ll need to use your API key.

API endpoints

Endpoint root: https://crawlera-stats.scrapinghub.com

/stats

Smart Proxy Manager usage stats.

Stats object

Field

Description

time_gte

Start of interval. ISO 8601 formatted date

clean

Number of successful responses

failed

Number of unsuccessful responses

concurrency

80 percentile of concurrent connections

total_time

80 percentile of response time (milliseconds)

traffic

Total traffic (bytes)

domain

Website name

Parameters

Field

Description

Required

start_date

ISO 8601 formatted date. Defaults to 7 days ago from now

No

end_date

ISO 8601 formatted date. Defaults to UTC now

No

groupby

How to group results. Defaults to no grouping. Available values: max, hour, day, month, year “max” means group by the most granular datetime precision possible (5min)

No

users

Only fetch data for this set of users (comma separated)

No

limit

Number of desired items per page. Defaults to 500

No

after

Token for requesting next items on timeline

No

netlocs

Only fetch stats for this set of domains (comma separated)

No

domain-wise

Show stats grouped per domain (true/false)

No

Examples

Last 7 days traffic:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/'
{
    "limit":500,
    "after":””,
    "results": [
    {
        "time_gte":"2018-12-12T11:05:00+00:00",
        "failed":112275,
        "traffic":125085476006,
        "concurrency":2,
        "total_time":1930,
        "clean":3758963
    }]
}

Last 7 days traffic max resolution:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max'
{
    "limit":500,
    "after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
    "results": [
    {
        "clean":175,
        "total_time":2032,
        "time_gte":"2018-12-17T15:30:00+00:00",
        "failed":16,
        "concurrency":2,
        "traffic":3554065
    }
    ....
    {
        "clean":166,
        "total_time":2036,
        "time_gte":"2018-12-17T16:15:00+00:00",
        "failed":4,
        "concurrency":1,
        "traffic":11257159
    }]
}

Consume next page:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?groupby=max&after=MHgxLjcwNWYzMmMwMDAwMDBwKzMw'
{
    "limit":500,
    "after":"MHgxLjcwNWYzNzcwMDAwMDBwKzMw",
    "results":[....]
}

One day traffic per hour:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?start_date=2019-01-01T00%3A00&end_date=2019-01-01T23%3A59&groupby=hour'
{
    “limit":500,
    “after”: “”,
    "results":[....]
}

Last 7 days traffic for two particular domains grouped by domain:

$ curl -u APIKEY: 'https://crawlera-stats.scrapinghub.com/stats/?netlocs=test1.com,test2.com&domain-wise=true'
{
    "limit":500,
    "after":"MHgxLjcwNWYzMmMwMDAwMDBwKzMw",
    "results": [
    {
        "clean":175,
        "total_time":2032,
        "time_gte":"2018-12-17T15:30:00+00:00",
        "failed":16,
        "concurrency":2,
        "traffic":3554065,
        "domain":"test1.com"
    },
    {
        "clean":166,
        "total_time":2036,
        "time_gte":"2018-12-17T16:15:00+00:00",
        "failed":4,
        "concurrency":1,
        "traffic":11257159,
        "domain":"test2.com"
    }]
}