Set up an AI web scraping project#
To set up a project for AI-assisted web scraping:
Create a folder named
ai-web-scraping-project
and open it with Visual Studio Code.Open the chat (
Ctrl + Shift + I
) and ask the AI to:Create and enable a Python virtual environment with Scrapy installed, and turn my workspace into a Scrapy project named “project”.
Once the AI is done, your ai-web-scraping-project
folder should now
contain the following folders and files:
ai-web-scraping-project/
├── project/
│ ├── __init__.py
│ ├── items.py
│ ├── middlewares.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders/
│ └── __init__.py
└── scrapy.cfg
You have now everything you need to start generating parsing code with AI.