Scrapy Cloud jobs

A job is the execution of a Scrapy spider or a Python script in Scrapy Cloud.

Running a job

For information about running a job, see:

• Running a Scrapy spider

  Tip

  Starting a job through the API allows defining Scrapy setting overrides for the job.

• Managing your jobs in the Jobs Dashboard

• Understanding Job Outcomes

• Deploy Project and run Spiders with settings of different environments

• Sharing data between spiders

• What are the differences between running a spider locally and on Scrapy Cloud?

• Can I run the same spider in parallel?

• Starting jobs programmatically (HTTP API or Python)

• Can I use an HTTP cache on Scrapy Cloud?

• Why do I get “Rejected message because it was too big” error?

• I clicked on STOP button but spider not stopped.

Monitoring jobs

For information about monitoring jobs, see:

• Getting Notifications on Certain Events

• Inspecting your spider’s runtime environment with the Job Console

Scheduling periodic jobs

For information about scheduling periodic jobs, see:

• Scheduling Periodic Jobs

• Is it possible to schedule jobs to run sequentially?

• How Can I Set a Number of Scrapy Cloud Units to Use for a Periodic Job?

• Can I Configure My Periodic Job To Run a Spider Every Minute?

See also

• Scrapy Cloud job items