AI-assisted web scraping tutorial

With the latest Visual Studio Code, GitHub Copilot and a modern Scrapy project (see requirements for details), you can install Web Scraping Copilot and use it to write maintainable web scraping code using AI:

Warning

The GitHub Copilot Free plan is not recommended for AI-assisted web scraping. See Code generation requirements for details.

1. Set up a project

Prepare Visual Studio Code and a Scrapy project with all prerequisites.

2. Generate parsing code

Use AI to generate parsing code for a webpage.

3. Generate crawling code

Use AI to generate crawling code for a website.