Using Smart Proxy Manager with Ruby

Warning

For the code below to work you must first install the Zyte CA certificate.

Note

All the code in this documentation has been tested with Ruby 2.6.3p62, Curb 0.9.11, Typhoeus 1.4.0, and Mechanize 2.8.2.

Using Curb

Making use of curb, a Ruby binding for libcurl:

require 'curb'

url = "https://toscrape.com"
proxy = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"

c = Curl::Easy.new(url) do |curl|
  curl.proxypwd = proxy_auth
  curl.proxy_url = proxy
  curl.verbose = true
end

c.perform
puts c.body_str

Using Typhoeus

Making use of typhoeus, another Ruby binding for libcurl:

require 'typhoeus'

url = "https://toscrape.com"
proxy_host = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"

request = Typhoeus::Request.new(
  url,
  method: :get,
  proxy: proxy_host,
  proxyuserpwd: proxy_auth,
  headers: {"X-Crawlera-Timeout" => "60000"}
)

request.run
print "Response Code: "
puts request.response.code
print "Response Time: "
puts request.response.total_time
print "Response Headers: "
puts request.response.headers
print "Response Body: "
puts request.response.body

Using Mechanize

Making use of mechanize, a Ruby library for automated web interaction:

require 'rubygems'
require 'mechanize'

url = "https://toscrape.com"
proxy_host = "proxy.zyte.com"
proxy_api_key = "<CRAWLERA_APIKEY>"

agent = Mechanize.new
agent.set_proxy proxy_host, 8011, proxy_api_key, ''

res = agent.get(url)
puts res.body

To allow Mechanize to use the Zyte CA certificate, point the SSL_CERT_FILE environment variable to its path:

export SSL_CERT_FILE=/path/to/zyte-ca.crt