Warning
Zyte API is replacing Smart Proxy Manager. See Migrating from Smart Proxy Manager to Zyte API.
Smart Proxy Manager Next Steps#
After subscribing to Smart Proxy Manager, here are some of the additional and advanced use cases you might want to explore.
Best Practices#
Setting Download Timeout#
While using Smart Proxy Manager, users might face an issue with short download timeout in their web crawlers or scraping applications. This might result in a failed request to Smart Proxy Manager. This can be resolved by increasing the download timeout value in the application you’re using like Scrapy, 600 seconds is the recommended value. This is because Smart Proxy Manager uses internal throttling to crawl websites politely and keep the proxy pool safe.
Keep in mind, Download Timeout is configured in the application used to connect with Smart Proxy Manager. This timeout value is the amount of time the application will wait while receiving a response from Smart Proxy Manager. Don’t confuse this with X-Crawlera-Timeout which is the amount of time used by Smart Proxy Manager internally for each request to the target website.
Adjust Concurrency#
In the context of web crawling, concurrency is the way of crawling multiple web pages at the same time. To speed up the crawl, concurrency can be adjusted according to the Smart Proxy Manager plan you’re on. Increasing concurrency in your crawling application will increase the number of requests to Smart Proxy Manager and will reflect on your usage and bills.
Retrying 503 responses#
Even though Smart Proxy Manager should protect you against bans, sometimes it runs out of capacity and will return a 503 response. Because of this, we recommend you retry 503 responses up to 5 times. Consider using the X-Crawlera-Next-Request-In to retry more efficiently.
Advance Use Cases#
Troubleshooting#
In case you’re having issues with Smart Proxy Manager checkout Smart Proxy Manager FAQ and Smart Proxy Manager Troubleshooting.