# Start a Scrapy project

To build your web scraping project, you will use [Scrapy](https://scrapy.org/), a popular open source
web scraping framework written in [Python](https://www.python.org/) and maintained by Zyte.

## Set up your project

### Claude Code

> [!NOTE]
> Uses [Agentic Web Data](../../../zyte-web-data/index.md#agentic-web-data).

1. [Install Agentic Web Data](../../../zyte-web-data/install.md#agentic-web-data-install).
2. Create a `web-scraping-tutorial` folder and start a **Claude
   Code** session from it:
   ```shell
   mkdir web-scraping-tutorial
   cd web-scraping-tutorial
   claude
   ```
3. Prompt **Claude Code** to:
   > Create a Scrapy project named `web-scraping-tutorial` in the
   > current folder.

### Copilot

> [!NOTE]
> Uses [Web Scraping Copilot](../../../copilot/index.md#copilot).

1. [Install Web Scraping Copilot](../../../copilot/install.md#copilot-install).
2. On the Web Scraping Copilot sidebar view, select **Start building ›
   Create new project**.
3. On the **Create new Scrapy project** page, set the **Scrapy project
   name** to `web-scraping-tutorial`, select a projects folder, and
   click **Create**.

   Your new `web-scraping-tutorial` workspace will be created and
   set up.

### CLI

1. [Install Python](https://wiki.python.org/moin/BeginnersGuide/Download), version `3.10` or higher.
2. Create a `web-scraping-tutorial` folder and make it your working
   folder:
   ```bash
   mkdir web-scraping-tutorial
   cd web-scraping-tutorial
   ```
3. Create and activate a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments).

   ### Windows

   ```batch
   python3 -m venv venv
   venv\Scripts\activate.bat
   ```

   ### macOS, Linux

   ```bash
   python3 -m venv venv
   . venv/bin/activate
   ```
4. Install the latest version of Scrapy:
   ```bash
   pip install scrapy==2.14.2
   ```
5. Make `web-scraping-tutorial` a [Scrapy](https://scrapy.org/) project folder:
   ```bash
   scrapy startproject web_scraping_tutorial .
   ```

Your `web-scraping-tutorial` folder should now contain at least the following
folders and files:

```text
web-scraping-tutorial/
├── .venv/
│   └── …
├── web_scraping_tutorial/
│   ├── spiders/
│   │   └── __init__.py
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   └── settings.py
└── scrapy.cfg
```

## Create your first spider

Now that you are all set up, you will write code to extract data from all books
in the Mystery category of [books.toscrape.com](http://books.toscrape.com/).

Create a file at `web_scraping_tutorial/spiders/books_toscrape_com.py`
with the following code:

```
from scrapy import Spider


class BooksToScrapeComSpider(Spider):
    name = "books_toscrape_com"
    custom_settings = {
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY": 0.01,
    }
    start_urls = [
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
    ]

    def parse(self, response):
        next_page_links = response.css(".next a")
        yield from response.follow_all(next_page_links)
        book_links = response.css("article a")
        yield from response.follow_all(book_links, callback=self.parse_book)

    def parse_book(self, response):
        yield {
            "name": response.css("h1::text").get(),
            "price": response.css(".price_color::text").re_first("£(.*)"),
            "url": response.url,
        }
```

In the code above:

- You define a [Scrapy spider class](https://docs.scrapy.org/en/latest/topics/spiders.html) named `books_toscrape_com`.
- You set custom values for [`CONCURRENT_REQUESTS_PER_DOMAIN`](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-CONCURRENT_REQUESTS_PER_DOMAIN) and
  [`DOWNLOAD_DELAY`](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-DOWNLOAD_DELAY) to speed crawls during the tutorial.
  [https://toscrape.com](https://toscrape.com) is a test site, so it is safe to do so.
- Your spider starts by sending a request for the Mystery category URL,
  [http://books.toscrape.com/catalogue/category/books/mystery_3/index.html](http://books.toscrape.com/catalogue/category/books/mystery_3/index.html)
  (`start_urls`), and parses the response with the default callback method:
  `parse`.
- The `parse` callback method:
  - Finds the link to the next page and, if found, yields a request for it,
    whose response will also be parsed by the `parse` callback method.

    As a result, the `parse` callback method eventually parses all pages
    of the Mystery category.
  - Finds links to book detail pages, and yields requests for them, whose
    responses will be parsed by the `parse_book` callback method.

    As a result, the `parse_book` callback method eventually parses all
    book detail pages from the Mystery category.
- The `parse_book` callback method extracts a record of book information
  with the book name, price, and URL.

> [!TIP]
> What if, instead of writing parsing code manually, you could use AI to
> generate it? See the tutorials of [Coding Agent Add-Ons](../../../ai-code.md#ai-code).

Now run your spider:

### Claude Code

> [!NOTE]
> Uses [Agentic Web Data](../../../zyte-web-data/index.md#agentic-web-data).

In a separate terminal, *not* in your **Claude Code** session, run:

```bash
scrapy crawl books_toscrape_com -O books.csv
```

### Copilot

> [!NOTE]
> Uses [Web Scraping Copilot](../../../copilot/index.md#copilot).

1. Select **Web Scraping Copilot** on the sidebar.
2. Expand the **Spiders** view. Click the **Refresh** button if your
   spider is not listed.
3. Click the **Run Spider Locally** button of your spider.
4. Paste the following in the **Arguments** field:
   ```none
   -O books.csv
   ```
5. Click **Run Spider**.

### CLI

```bash
scrapy crawl books_toscrape_com -O books.csv
```

Once execution finishes, the generated `books.csv` file will contain records
for all books from the Mystery category of [books.toscrape.com](http://books.toscrape.com/) in [CSV](https://en.wikipedia.org/wiki/Comma-separated_values)
format. You can open `books.csv` with any spreadsheet app.

Continue to the [next chapter](cloud.md#tutorial-cloud) to learn how you can
easily deploy and run you web scraping project on the cloud.