Set up an AI web scraping project#

To set up a project for AI-assisted web scraping:

  1. Install Web Scraping Copilot and its requirements.

  2. Create a folder named ai-web-scraping-project and open it with Visual Studio Code.

  3. Open the chat (Ctrl + Shift + I) and ask the AI to:

    Create and enable a Python virtual environment with Scrapy installed, and turn my workspace into a Scrapy project named “project”.

Once the AI is done, your ai-web-scraping-project folder should now contain the following folders and files:

ai-web-scraping-project/
├── project/
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders/
│       └── __init__.py
└── scrapy.cfg

You have now everything you need to start generating parsing code with AI.