Warning

Zyte API is replacing Smart Proxy Manager. It is no longer possible to sign up to Smart Proxy Manager. If you are an existing Smart Proxy Manager user, see Migrating from Smart Proxy Manager to Zyte API.

Smart Proxy Manager Next Steps

After subscribing to Smart Proxy Manager, here are some of the additional and advanced use cases you might want to explore.

Best Practices

Setting Download Timeout

While using Smart Proxy Manager, users might face an issue with short download timeout in their web crawlers or scraping applications. This might result in a failed request to Smart Proxy Manager. This can be resolved by increasing the download timeout value in the application you’re using like Scrapy, 600 seconds is the recommended value. This is because Smart Proxy Manager uses internal throttling to crawl websites politely and keep the proxy pool safe.

Keep in mind, Download Timeout is configured in the application used to connect with Smart Proxy Manager. This timeout value is the amount of time the application will wait while receiving a response from Smart Proxy Manager. Don’t confuse this with X-Crawlera-Timeout which is the amount of time used by Smart Proxy Manager internally for each request to the target website.

Adjust Concurrency

In the context of web crawling, concurrency is the way of crawling multiple web pages at the same time. To speed up the crawl, concurrency can be adjusted according to the Smart Proxy Manager plan you’re on. Increasing concurrency in your crawling application will increase the number of requests to Smart Proxy Manager and will reflect on your usage and bills.

Retrying 503 responses

Even though Smart Proxy Manager should protect you against bans, sometimes it runs out of capacity and will return a 503 response. Because of this, we recommend you retry 503 responses up to 5 times. Consider using the X-Crawlera-Next-Request-In to retry more efficiently.

Advance Use Cases

• Restricting Zyte Smart Proxy Manager IPs to a specific region

• Sending POST request with Zyte Smart Proxy Manager

• Fetching HTTPS pages with Zyte Smart Proxy Manager

• Using Desktop and Mobile profiles using Zyte Smart Proxy Manager

• Using Smart Proxy Manager with Splash and Scrapy

Troubleshooting

In case you’re having issues with Smart Proxy Manager checkout Smart Proxy Manager FAQ and Smart Proxy Manager Troubleshooting.