Using Smart Proxy Manager with Scrapy Cloud

To employ Smart Proxy Manager in Scrapy Cloud projects the Crawlera addon is used. Go to Settings > Addons > Crawlera to activate.

Settings

CRAWLERA_URL

proxy URL (default: http://proxy.zyte.com:8011)

CRAWLERA_ENABLED

tick the checkbox to enable Smart Proxy Manager

CRAWLERA_APIKEY

Smart Proxy Manager API key

CRAWLERA_MAXBANS

number of bans to ignore before closing the spider (default: 20)

CRAWLERA_DOWNLOAD_TIMEOUT

timeout for requests (default: 190)