AI-assisted web scraping tutorial

With GitHub Copilot and Web Scraping Copilot you can write maintainable web scraping code using AI:

Warning

The GitHub Copilot Free plan is not recommended for AI-assisted web scraping. See Code generation requirements for details.

1. Set up a project

Prepare Visual Studio Code and a Scrapy project with all prerequisites.

Set up an AI web scraping project

2. Generate parsing code

Use AI to generate parsing code for a webpage.

Generate parsing code with AI

3. Generate crawling code

Use AI to generate crawling code for a website.

Generate crawling code with AI