Warning

Zyte API is replacing Smart Proxy Manager. See Migrating from Smart Proxy Manager to Zyte API.

Using Desktop and Mobile profiles using Zyte Smart Proxy Manager

To use Desktop and Mobile profiles, Zyte Smart Proxy Manager (formerly Crawlera) allows the use of X-Crawlera-Profile header while making a request. It not only sets the User-Agent but also applies a set of headers that are used by the browser. For example, all modern browsers set Accept-Language and Accept-Encoding headers and some browsers set DNT and Upgrade-Insecure-Requests headers.

Note

All the code in this documentation has been tested with curl 7.64.1.

Using Desktop Profiles

To request Desktop Profiles X-Crawlera-Profile:desktop header should be passed. Smart Proxy Manager will set appropriate headers, please check X-Crawlera-Profile for details.

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "Accept-Language": "en-US,en;q=0.9",
        "Forwarded": "for=50.114.29.222;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Ch-Ua": "\"Chromium\";v=\"94\", \"Microsoft Edge\";v=\"94\", \";Not A Brand\";v=\"99\"",
        "Sec-Ch-Ua-Mobile": "?0",
        "Sec-Ch-Ua-Platform": "\"Windows\"",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36 Edg/94.0.992.31"
    }
}

Note

The header X-Crawlera-Profile:desktop is default even if you make a request with it. For example, curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers".

Using Mobile Profiles

To request Mobile Profiles X-Crawlera-Profile:mobile header should be passed. Smart Proxy Manager will set appropriate headers, please check X-Crawlera-Profile for details.

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "Accept-Language": "en-US,en;q=0.9",
        "Cache-Control": "max-age=259200",
        "Forwarded": "for=23.236.230.6;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Ch-Ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
        "Sec-Ch-Ua-Mobile": "?1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Linux; Android 10; moto g(9) play) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36"
    }
}

Override Headers

To override a header combined with X-Crawlera-Profile functionality, try X-Crawlera-Profile-Pass in the following way:

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile-Pass:Accept-Language" -H "Accept-Language:en-GB"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Language": "en-GB",
        "Forwarded": "for=191.101.154.16;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:90.0) Gecko/20100101 Firefox/90.0"
    }
}

To learn more, check out X-Crawlera-Profile-Pass.

Override X-Crawlera-Profile Header

To override X-Crawlera-Profile either X-Crawlera-Profile-Pass can be used to mention the headers Smart Proxy Manager should override or use X-Crawlera-Profile:pass. Smart Proxy Manager will pass the headers to the website as received in the request. For example,

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:pass"

Expected response:

{
    "headers": {
        "Accept": "*/*",
        "Cache-Control": "max-age=259200",
        "Forwarded": "for=23.236.230.6;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "User-Agent": "curl/7.64.1"
    }
}