Set up an AI web scraping project#

To set up a project for AI-assisted web scraping:

  1. Install Web Scraping Copilot and its requirements.

  2. Create a folder named ai-web-scraping-project and open it with Visual Studio Code.

  3. Select Web Scraping Copilot on the sidebar, and under Extension Status, complete the setup steps.

    On the Python step, select Switch Interpreter › Create Virtual Environment › Venv and then select an installed Python executable.

    ../../../_images/guide.png

Once the setup is done, your ai-web-scraping-project folder should contain the following folders and files:

ai-web-scraping-project/
├── .venv/
│   └── …
├── project/
│   ├── spiders/
│   │   └── __init__.py
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   └── settings.py
└── scrapy.cfg

You have now everything you need to start generating parsing code with AI.