Deploying code to Scrapy Cloud projects

For information about deploying your code to a project, see:

• Deploying your spiders to Scrapy Cloud

• Deploying a Project from a Github Repository

• Versioning your deploys to Zyte Developer Tool Scrapy Cloud

• Deploying non-code files

Tip

Web Scraping Copilot supports Scrapy Cloud deployment.

Stacks

For information about Scrapy Cloud stacks, see:

• Changing the Deploy Environment With Scrapy Cloud Stacks

• Deploying Python 3 spiders to Scrapy Cloud

Requirements

For information about installing additional Python packages into stacks, see:

• Deploying Python Dependencies for Your Projects in Scrapy Cloud

Addons

When using a stack, you can use Scrapy Cloud add-ons to extend your code, including:

• Autothrottle, to crawl gently.

• DeltaFetch, to crawl only new pages.

• DotScrapy Persistence, to persist data between jobs.

• Images, to download images into S3 storage.

• Magic Fields, to add item fields.

• Page Storage, to store visited pages.

• Query Cleaner, to clean request URL query parameters.

Using Docker images

For information about using Docker images to deploy your code, see:

• Deploying Custom Docker images on Scrapy Cloud

• Errors while deploying Custom Image to Scrapy Cloud