Fetching HTTPS pages with Zyte Smart Proxy Manager

Zyte Smart Proxy Manager (formerly Crawlera) performs HTTPS requests using CONNECT method, transparently tunneling packets over default HTTPS port 443 to and from the destination server in a secure session.

To fetch HTTPS web pages through Smart Proxy Manager you need to download and install the certificate file for Smart Proxy Manager Certificate Authority or disable SSL certificate verification in your HTTP client.

The Smart Proxy Manager Certificate Authority can be downloaded here: zyte-smartproxy-ca.crt.

Installing the certificate file on Windows 10

  1. Press the Win key + R hotkey and input mmc in Run to open the Microsoft Management Console window.

  2. Click File and select Add/Remove Snap-ins.

  3. In the opened window select Certificates and press the Add > button.

  4. In the Certificates Snap-in window select Computer account > Local Account, and press the Finish button to close the window.

  5. Press the OK button in the Add or Remove Snap-in window.

  6. Back in the Microsoft Management Console window, select Certificates under Console Root and right-click Trusted Root Certification Authorities.

  7. From the context menu select All Tasks > Import to open the Certificate Import Wizard window from which you can add Smart Proxy Manager’s certificate.

More details can be found here.

Installing the certificate file on Mac

  1. Open Keychain Access window (Launchpad > Other > Keychain Access).

  2. Select System tab under Keychains, drag and drop the downloaded certificate file (or select File > Import Items... and navigate to the file).

  3. Enter the administrator password to modify the keychain.

  4. Double-click the Smart Proxy Manager CA certificate entry, expand Trust, next to When using this certificate: select Always Trust.

  5. Close the window and enter the administrator password again to update the settings.

Installing the certificate file on Linux/Ubuntu

  1. Copy the downloaded certificate file using the following command:

sudo cp zyte-smartproxy-ca.crt /usr/local/share/ca-certificates/zyte-smartproxy-ca.crt

  1. Update stored Certificate Authority files:

sudo update-ca-certificates

Installing the certificate file into the Mozilla Firefox browser

  1. Open Preferences, visit Privacy & Security tab, scroll down to the Certificates section, click View Certificates… button to open Certificate Manager.

  2. Under Authorities tab click Import… button, navigate to the certificate file.

  3. In the opened window (You have been asked to trust a new Certificate Authority (CA)) check the first option Trust this CA to identify websites and click the OK button to finish importing the certificate.

  4. Click the OK button to save settings and exit Certificate Manager.

Installing the certificate file into the Google Chrome browser

  1. Click the triple-dot icon in the top right corner and choose Settings.

  2. Scroll to the Privacy and security section and click on Security.

  3. Scroll down to find and click Manage Certificates.

  4. The next steps will depend on the operating system. In the case of Mac OS, the previous action will open the Keychain Access, see Installing the certificate file on OS X section above. In the case of Windows, the Certificates application should appear, select Trusted Root Certification Authorities tab, click Import... button, navigate to the certificate file, verify the import was successful and the installed certificate is displayed under Trusted Root Certification Authorities tab, close the Certificates window.

In case of requiring a certificate with PKCS#12 format, you can generate it with the following OpenSSL command:

openssl pkcs12 -export -nokeys -password pass: -in zyte-smartproxy-ca.crt -out zyte-smartproxy-ca.p12

The minimal example to check if the CA is working for you is to try:

curl -vx https://proxy.zyte.com:8014 -U <API key>: --cacert zyte-smartproxy-ca.crt https://httpbin.org/ip