Using Smart Proxy Manager with Ruby

Warning

zyte-smartproxy-ca.crt should be installed in your OS for the below code to work. You can follow these instructions in order to install it.

Note

All the code in this documentation has been tested with Ruby 2.6.3p62, Curb 0.9.11, Typhoeus 1.4.0, and Mechanize 2.8.2.

Using Curb

Making use of curb, a Ruby binding for libcurl:

require 'curb'

url = "https://toscrape.com"
proxy = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"

c = Curl::Easy.new(url) do |curl|
  curl.proxypwd = proxy_auth
  curl.proxy_url = proxy
  curl.verbose = true
end

c.perform
puts c.body_str

Using Typhoeus

Making use of typhoeus, another Ruby binding for libcurl:

require 'typhoeus'

url = "https://toscrape.com"
proxy_host = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"

request = Typhoeus::Request.new(
  url,
  method: :get,
  proxy: proxy_host,
  proxyuserpwd: proxy_auth,
  headers: {"X-Crawlera-Timeout" => "60000"}
)

request.run
print "Response Code: "
puts request.response.code
print "Response Time: "
puts request.response.total_time
print "Response Headers: "
puts request.response.headers
print "Response Body: "
puts request.response.body

Using Mechanize

Making use of mechanize, a Ruby library for automated web interaction: Don’t forget to load the Certificate file zyte-smartproxy-ca.crt, and set it using the env variable export SSL_CERT_FILE=/path/to/zyte-smartproxy-ca.crt

require 'rubygems'
require 'mechanize'

url = "https://toscrape.com"
proxy_host = "proxy.zyte.com"
proxy_api_key = "<CRAWLERA_APIKEY>"

agent = Mechanize.new
agent.set_proxy proxy_host, 8011, proxy_api_key, ''

res = agent.get(url)
puts res.body