# Scrapy Cloud spiders

A Scrapy Cloud spider is a [Scrapy spider](https://docs.scrapy.org/en/latest/topics/spiders.html#topics-spiders) that is part
of a [Scrapy project](https://docs.scrapy.org/en/latest/index.html#topics-index) that has been [deployed](deployment.md#sc-deployment) into a [Scrapy Cloud project](projects.md#sc-projects). You can start [jobs](jobs/index.md#jobs) to execute the code
of a spider.

Our [web scraping tutorial](../../web-scraping/tutorials/main/index.md#tutorial) covers creating, deploying, and
running spiders. For more information, see the [Scrapy documentation](https://docs.scrapy.org/en/latest/index.html#topics-index).

## Spider templates and virtual spiders

Scrapy Cloud supports defining [spider templates](#spider-templates), that
you can use from the Scrapy Cloud UI to create [virtual spiders](#virtual-spiders) that run the code of the corresponding spider template with
predefined parameters.

### Spider templates

To create a spider template:

1. Add [scrapy-spider-metadata](https://scrapy-spider-metadata.readthedocs.io/en/latest/) as a dependency to your [Scrapy Cloud
   project](projects.md#sc-projects).
2. On the spiders that you wish to use as templates, define [metadata](https://scrapy-spider-metadata.readthedocs.io/en/latest/metadata.html#defining-spider-metadata)
   including a `title` and `description` of your choice, and setting
   `template` to `True`:
   ```python
   from scrapy import Spider

   class MySpider(Spider):
       ...
       metadata = {
           "title": "My Template",
           "description": "Description of my template.",
           "template": True,
       }
   ```

When you [redeploy](deployment.md#sc-deployment) your code, you can start
creating [virtual spiders](#virtual-spiders) from your spider templates.

> [!NOTE]
> Spider templates are also regular spiders, and can be [executed](jobs/index.md#jobs) directly as well.

### Virtual spiders

To create a virtual spider from a spider template, go to your Scrapy Cloud
project page and, on the left-hand sidebar, under **Spiders**, select **Create
spider**.

On the **Create Spider** page, you can select a template, define the parameters
of your new virtual spider, and save your spider.

You can then use your virtual spider from Scrapy Cloud as if it were a regular
spider.

Virtual spiders exist only in Scrapy Cloud, not in your code. However, changes
to the code of their spider template will affect them.

### Spider parameters

The point of spider templates is to be able to create virtual spiders from them
that each works differently based on predefined parameters.

To expose parameters to the Scrapy Cloud UI so that they can be defined when
creating a virtual spider, add a [parameter specification](https://scrapy-spider-metadata.readthedocs.io/en/latest/params.html) to your template
spiders using [scrapy-spider-metadata](https://scrapy-spider-metadata.readthedocs.io/en/latest/):

```python
from pydantic import BaseModel
from scrapy import Spider
from scrapy_spider_metadata import Args

class MyParams(BaseModel):
    foo: str

class MySpider(Args[MyParams], Spider):
    ...
```

#### Parameter types

Scrapy Cloud supports the following parameter types:

- `bool`
- `int`, `float` (with `gt`, `lt`, `ge`, and `le` [numeric
  constraint](https://docs.pydantic.dev/latest/concepts/fields/#numeric-constraints) support)
- `str` (with [string constraint](https://docs.pydantic.dev/latest/concepts/fields/#string-constraints) support)

  Scrapy Cloud also supports defining a placeholder through
  [json_schema_extra](https://docs.pydantic.dev/latest/api/fields/#pydantic.fields.Field):
  ```python
  from pydantic import BaseModel, Field

  class MyParams(BaseModel):
      url: str = Field(
          json_schema_extra={
              "placeholder": "https://books.toscrape.com",
          },
      )
  ```
- `str` + `Enum`

  Define `enumMeta` in [json_schema_extra](https://docs.pydantic.dev/latest/api/fields/#pydantic.fields.Field) to give your enumeration
  choices an optional title and description:
  ```python
  from enum import Enum

  from pydantic import BaseModel, Field

  class Foo(str, Enum):
      bar: str = "bar"
      baz: str = "baz"

  class MyParams(BaseModel):
      foo: Foo = Field(
          json_schema_extra={
              "enumMeta": {
                  Foo.bar: {
                      "title": "Bar",
                      "description": "Bar description.",
                  },
                  Foo.baz: {
                      "title": "Baz",
                      "description": "Baz description.",
                  },
              },
          },
      )
  ```

#### Widgets

Scrapy Cloud also supports a few special UI widgets that you can enable through
the `widget` key of [json_schema_extra](https://docs.pydantic.dev/latest/api/fields/#pydantic.fields.Field), e.g.

```python
from pydantic import BaseModel, Field

class MyParams(BaseModel):
    foo: int = Field(
        json_schema_extra={
            "widget": "widget-id",
        },
    )
```

The following widgets are supported:

- `custom-attrs`, to specify a [custom attributes schema](https://docs.zyte.com/zyte-api/usage/reference.html#operation/extract/request/customAttributes).
- `request-limit`, to specify a maximum number of requests.
- `textarea`, for multi-line text input.

#### Parameter groups

Scrapy Cloud also supports defining 2 or more optional parameters so that
filling 1 of them (and only 1) is required:

```python
from pydantic import BaseModel, ConfigDict

    class MyParams(BaseModel):
        model_config = ConfigDict(
            json_schema_extra={
                "groups": [
                    {
                        "id": "a-or-b",
                        "title": "A or B",
                        "description": "Fill A or B.",
                        "widget": "exclusive",
                    },
                ],
            },
        )
        a: str = Field(
            "json_schema_extra": {
                "group": "a-or-b",
                "exclusiveRequired": True,
            },
        )
        b: str = Field(
            "json_schema_extra": {
                "group": "a-or-b",
                "exclusiveRequired": True,
            },
        )
```