# Generate parsing code with AI

Now that [your project is ready](setup.md#copilot-tutorial-setup), you will use AI
to generate code to parse book webpages from [https://books.toscrape.com](https://books.toscrape.com).

## 1. Generate an item class

First, you need to define the type of data that you want to parse from each
book page.

> [!TIP]
> Select a somewhat smart model in the chat view, i.e. **GPT-5** or
> similar. **GPT-5 mini** is OK if you prefer a non-premium model. GPT-4.1 is
> problematic for web scraping.

Ask the AI to:

> Define a dataclass item called Book with title, price and url fields. Make
> them optional and of type str | None.

The AI should edit `copilot-tutorial/copilot_tutorial/items.py` to add:

`copilot-tutorial/copilot_tutorial/items.py`
```python
from dataclasses import dataclass

@dataclass
class Book:
    url: str | None = None
    title: str | None = None
    price: str | None = None
```

> [!TIP]
> You can use any [item type supported by Scrapy](https://docs.scrapy.org/en/latest/topics/items.html#item-types),
> [`dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass) is one of many options.
> 
> You can also use a pre-made [item type](https://zyte-common-items.readthedocs.io/en/latest/reference/items.html#item-api) from
> [zyte-common-items](https://zyte-common-items.readthedocs.io/en/latest/index.html), like
> [`zyte_common_items.Product`](https://zyte-common-items.readthedocs.io/en/latest/reference/items.html#zyte_common_items.Product), instead of writing your own item type
> from scratch.

## 2. Generate parsing code

Select **Web Scraping Copilot › Page Objects › Generate Parsing Code with AI**:

![image](_static/copilot/generate-0.1.0.png)

The chat view will open with the **WebScraping** agent, a prompt will be sent,
and the AI will start assisting. It should:

1. Ask you for some **input**.

   It usually detects the right item type to use and the right path to save
   your page objects (more on them later), but it always needs you to specify
   **example target URLs**.

   You are generating a page object for book detail pages, so choose a few
   such URLs and share them in chat.
   For example:
   [https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)
   [https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html](https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html)
   [https://books.toscrape.com/catalogue/soumission_998/index.html](https://books.toscrape.com/catalogue/soumission_998/index.html)
2. Create
   `copilot-tutorial/copilot_tutorial/pages/books_toscrape_com.py` with
   something like:
   ```python
   from copilot_tutorial.items import Book
   from web_poet import Returns, WebPage, field, handle_urls

   @handle_urls("books.toscrape.com")
   class BooksToscrapeComBookPage(WebPage, Returns[Book]):
       pass
   ```

   > [!NOTE]
   > This is a [page object class](https://web-poet.readthedocs.io/en/stable/page-objects/index.html#page-objects). It
   > defines how to extract a given [type of data](https://web-poet.readthedocs.io/en/stable/page-objects/items.html#items)
   > (e.g. `Book`) from a given [URL pattern](https://web-poet.readthedocs.io/en/stable/page-objects/rules.html#rules) (e.g.
   > the `books.toscrape.com` domain).
3. Generate tests for the target example URLs.
   > [!NOTE]
   > [web-poet tests](https://web-poet.readthedocs.io/en/stable/page-objects/testing.html#web-poet-testing) are example inputs and
   > expected outputs for a page object class. They can also assert that a
   > given input should raise an expected exception. You can use them to
   > test your code, and the AI can use them to generate the right parsing
   > code.
4. Populate test expectations.
5. Generate parsing code for your new page object class.
6. Run the generated tests to check that the generated parsing code extracts
   the expected data.

By the end, you should have a working page object class that can extract book
data from any book URL from [https://books.toscrape.com](https://books.toscrape.com).

## 3. Create a spider

Now that you have a working page object, it is time to implement a Scrapy
spider that uses it.

Create the following file:

`copilot-tutorial/copilot_tutorial/spiders/books.py`
```python
from scrapy import Request, Spider

from copilot_tutorial.items import Book


class BookSpider(Spider):
    name = "book"
    url: str

    async def start(self):
        yield Request(self.url, callback=self.parse_book)

    async def parse_book(self, _, book: Book):
        yield book
```

The spider expects a `url` argument, which you can pass to a spider with the
`-a url=<url>` syntax.

When a request targets the `parse_book` callback, [scrapy-poet](https://scrapy-poet.readthedocs.io/en/stable/index.html) sees the `Book` type hint and injects a `book`
parameter built with your page object class.

Your spider can now extract book data from any book details page from
[https://books.toscrape.com](https://books.toscrape.com). For example, try [running](../main/setup.md#tutorial-run-spider) your `book` spider with the following arguments:

```bash
-a url=https://books.toscrape.com/catalogue/soumission_998/index.html
-o books.jsonl
```

It will generate a `books.jsonl` file with the following JSON object:

```json
{
    "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
    "title": "Soumission",
    "price": "50.10"
}
```

You can also repeat step 2 for other book stores, and this spider will also
work for them, no need to have separate spiders per website.

Continue to the [next chapter](crawl.md#copilot-tutorial-crawl) to use AI to
generate *crawling* code, to be able to write a spider that can crawl an entire
book store, and not just a single book URL.