Using Desktop and Mobile profiles using Zyte Smart Proxy Manager

To use Desktop and Mobile profiles, Zyte Smart Proxy Manager (formerly Crawlera) allows the use of X-Crawlera-Profile header while making a request. It not only sets the User-Agent but also applies a set of headers that are used by the browser. For example, all modern browsers set Accept-Language and Accept-Encoding headers and some browsers set DNT and Upgrade-Insecure-Requests headers.

Note

All the code in this documentation has been tested with curl 7.64.1.

Using Desktop Profiles

To request Desktop Profiles X-Crawlera-Profile:desktop header should be passed. Smart Proxy will set appropriate headers, please check X-Crawlera-Profile for details.

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "Accept-Language": "en-US,en;q=0.9",
        "Forwarded": "for=50.114.29.222;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Ch-Ua": "\"Chromium\";v=\"94\", \"Microsoft Edge\";v=\"94\", \";Not A Brand\";v=\"99\"",
        "Sec-Ch-Ua-Mobile": "?0",
        "Sec-Ch-Ua-Platform": "\"Windows\"",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36 Edg/94.0.992.31"
    }
}

Note

The header X-Crawlera-Profile:desktop is default even if you make a request with it. For example, curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers".

Using Mobile Profiles

To request Mobile Profiles X-Crawlera-Profile:mobile header should be passed. Smart Proxy will set appropriate headers, please check X-Crawlera-Profile for details.

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "Accept-Language": "en-US,en;q=0.9",
        "Cache-Control": "max-age=259200",
        "Forwarded": "for=23.236.230.6;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Ch-Ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
        "Sec-Ch-Ua-Mobile": "?1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Linux; Android 10; moto g(9) play) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36"
    }
}

Override Headers

To override a header combined with X-Crawlera-Profile functionality, try X-Crawlera-Profile-Pass in the following way:

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile-Pass:Accept-Language" -H "Accept-Language:en-GB"

Expected response:

{
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Language": "en-GB",
        "Forwarded": "for=191.101.154.16;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "same-origin",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:90.0) Gecko/20100101 Firefox/90.0"
    }
}

To learn more, check out X-Crawlera-Profile-Pass.

Override X-Crawlera-Profile Header

To override X-Crawlera-Profile either X-Crawlera-Profile-Pass can be used to mention the headers Smart Proxy should override or use X-Crawlera-Profile:pass. Smart Proxy Manager will pass the headers to the website as received in the request. For example,

curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:pass"

Expected response:

{
    "headers": {
        "Accept": "*/*",
        "Cache-Control": "max-age=259200",
        "Forwarded": "for=23.236.230.6;proto=http",
        "Host": "httpbin.scrapinghub.com",
        "User-Agent": "curl/7.64.1"
    }
}