Using Smart Proxy Manager with C#

using System;
using System.IO;
using System.Net;

namespace ProxyRequest
{
    class MainClass
    {
        public static void Main(string[] args)
        {
            var myProxy = new WebProxy("http://proxy.zyte.com:8011", true);
            myProxy.Credentials = new NetworkCredential("<CRAWLERA_APIKEY>", "");

            var request = (HttpWebRequest)WebRequest.Create("https://httpbin.scrapinghub.com/headers");
            request.Proxy = myProxy;
            request.PreAuthenticate = true;
            request.AllowAutoRedirect = false;
            request.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;

            var response = request.GetResponse();
            Console.WriteLine("Response Status: " + ((HttpWebResponse)response).StatusDescription);
            Console.WriteLine("\nResponse Headers:\n" + ((HttpWebResponse)response).Headers);
            var dataStream = response.GetResponseStream();
            var reader = new StreamReader(dataStream);
            string responseFromServer = reader.ReadToEnd();
            Console.WriteLine("Response Body:\n" + responseFromServer);
            reader.Close();
            response.Close();
        }
    }
}

Note

You may need to update the following setting in the applications App.config so proxy autherization works:

<system.net>
  <settings>
    <httpWebRequest useUnsafeHeaderParsing="true" />
  </settings>
</system.net>

Warning

Some HTTP client libraries, including Apache HttpComponents Client and .NET, don’t send authentication headers by default. This can result in doubled requests, so preemptive authentication should be enabled where this is the case.

In the above example we are making an HTTPS request through Smart Proxy Manager. It is assumed that the Zyte proxy SSL Certificate has been installed, since the CONNECT method will be employed.

If you use WebClient and receive 407s from Smart Proxy Manager, try setting AllowAutoRedirect to false.