Run your code on Scrapy Cloud#

Running your code locally is fine during development, but for production you usually want something better.

You will now deploy and run your code on Scrapy Cloud.

Deploy to Scrapy Cloud#

  1. Create a Scrapy Cloud project on the Zyte dashboard.

  2. Install the latest version of shub, the Scrapy Cloud command-line application:

    pip install --upgrade shub
  3. Create a text file at web-scraping-tutorial/requirements.txt with the following content:

  4. Create a YAML file at web-scraping-tutorial/scrapinghub.yml with the following content:

      file: requirements.txt
      default: scrapy:2.8
  5. Copy your API key from the Zyte dashboard.

  6. Run the following command and, when prompted, paste your API key and press Enter:

    shub login
  7. On the Zyte dashboard, select your Scrapy Cloud project under Scrapy Cloud Projects, and copy your Scrapy Cloud project ID from the browser URL bar.

    For example, if the URL is, 000000 is your Scrapy Cloud project ID.

  8. Make sure web-scraping-tutorial is your current working folder.

  9. Run the following command, replacing 000000 with your actual project ID:

    shub deploy 000000

Your Scrapy project has now been deployed to your Scrapy Cloud project.

Run a Scrapy Cloud job#

Now that you have deployed your Scrapy project to your Scrapy Cloud project, it is time to run one of your spiders on Scrapy Cloud:

  1. On the Zyte dashboard, select your Scrapy Cloud project under Scrapy Cloud Projects.

  2. On the Dashboard page of your project, select Run on the top-right corner.

  3. On the Run dialog box:

    1. Select the Spiders field and, from the spider list that appears, select your spider name.

    2. Select Run.


    A new Scrapy Cloud job will appear in the Running job list:

    Once the job finishes, it will move to the Completed job list:

  4. Follow the link from the Job column, 1/1.

  5. On the job page, select the Items tab.

  6. On the Items page, select Export → CSV.

The downloaded file will have the same data as the books.csv file that you generated locally with your first spider.