Fetching HTTPS pages with Zyte Smart Proxy Manager#
Zyte Smart Proxy Manager (formerly Crawlera) performs HTTPS requests using CONNECT method, transparently tunneling packets over default HTTPS port 443 to and from the destination server in a secure session.
To fetch HTTPS web pages through Smart Proxy Manager you need to download and install the certificate file for Smart Proxy Manager Certificate Authority or disable SSL certificate verification in your HTTP client.
The Smart Proxy Manager Certificate Authority can be downloaded here: zyte-smartproxy-ca.crt.
Installing the certificate file on Windows 10#
Press the
Win key + R
hotkey and inputmmc
in Run to open the Microsoft Management Console window.Click
File
and selectAdd/Remove Snap-ins
.In the opened window select
Certificates
and press theAdd >
button.In the Certificates Snap-in window select
Computer account > Local Account
, and press theFinish
button to close the window.Press the
OK
button in the Add or Remove Snap-in window.Back in the Microsoft Management Console window, select
Certificates
under Console Root and right-clickTrusted Root Certification Authorities
.From the context menu select
All Tasks > Import
to open the Certificate Import Wizard window from which you can add Smart Proxy Manager’s certificate.
More details can be found here.
Installing the certificate file on Mac#
Open Keychain Access window (
Launchpad > Other > Keychain Access
).Select
System
tab under Keychains, drag and drop the downloaded certificate file (or select File >Import Items...
and navigate to the file).Enter the administrator password to modify the keychain.
Double-click the
Smart Proxy Manager CA
certificate entry, expand Trust, next to When using this certificate: selectAlways Trust
.Close the window and enter the administrator password again to update the settings.
Installing the certificate file on Linux/Ubuntu#
Copy the downloaded certificate file using the following command:
sudo cp zyte-smartproxy-ca.crt /usr/local/share/ca-certificates/zyte-smartproxy-ca.crt
Update stored Certificate Authority files:
sudo update-ca-certificates
Installing the certificate file into the Mozilla Firefox browser#
Open Preferences, visit Privacy & Security tab, scroll down to the Certificates section, click View Certificates… button to open Certificate Manager.
Under Authorities tab click Import… button, navigate to the certificate file.
In the opened window (You have been asked to trust a new Certificate Authority (CA)) check the first option Trust this CA to identify websites and click the OK button to finish importing the certificate.
Click the OK button to save settings and exit Certificate Manager.
Installing the certificate file into the Google Chrome browser#
Click the triple-dot icon in the top right corner and choose
Settings
.Scroll to the
Privacy and security
section and click onSecurity
.Scroll down to find and click
Manage Certificates
.The next steps will depend on the operating system. In the case of Mac OS, the previous action will open the
Keychain Access
, see Installing the certificate file on OS X section above. In the case of Windows, the Certificates application should appear, selectTrusted Root Certification Authorities
tab, clickImport...
button, navigate to the certificate file, verify the import was successful and the installed certificate is displayed under Trusted Root Certification Authorities tab, close the Certificates window.
In case of requiring a certificate with PKCS#12 format, you can generate it with the following OpenSSL command:
openssl pkcs12 -export -nokeys -password pass: -in zyte-smartproxy-ca.crt -out zyte-smartproxy-ca.p12
The minimal example to check if the CA is working for you is to try:
curl -vx https://proxy.zyte.com:8014 -U <API key>: --cacert zyte-smartproxy-ca.crt https://httpbin.org/ip