Using Smart Proxy Manager with Ruby#
Warning
zyte-smartproxy-ca.crt
should be installed in your OS for the below code to work. You can follow these instructions in order to install it.
Note
All the code in this documentation has been tested with Ruby 2.6.3p62, Curb 0.9.11, Typhoeus 1.4.0, and Mechanize 2.8.2.
Using Curb#
Making use of curb, a Ruby binding for libcurl:
require 'curb'
url = "https://toscrape.com"
proxy = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"
c = Curl::Easy.new(url) do |curl|
curl.proxypwd = proxy_auth
curl.proxy_url = proxy
curl.verbose = true
end
c.perform
puts c.body_str
Using Typhoeus#
Making use of typhoeus, another Ruby binding for libcurl:
require 'typhoeus'
url = "https://toscrape.com"
proxy_host = "proxy.zyte.com:8011"
proxy_auth = "<CRAWLERA_APIKEY>:"
request = Typhoeus::Request.new(
url,
method: :get,
proxy: proxy_host,
proxyuserpwd: proxy_auth,
headers: {"X-Crawlera-Timeout" => "60000"}
)
request.run
print "Response Code: "
puts request.response.code
print "Response Time: "
puts request.response.total_time
print "Response Headers: "
puts request.response.headers
print "Response Body: "
puts request.response.body
Using Mechanize#
Making use of mechanize, a
Ruby library for automated web interaction: Don’t forget to load the
Certificate file zyte-smartproxy-ca.crt
,
and set it using the env variable export SSL_CERT_FILE=/path/to/zyte-smartproxy-ca.crt
require 'rubygems'
require 'mechanize'
url = "https://toscrape.com"
proxy_host = "proxy.zyte.com"
proxy_api_key = "<CRAWLERA_APIKEY>"
agent = Mechanize.new
agent.set_proxy proxy_host, 8011, proxy_api_key, ''
res = agent.get(url)
puts res.body