Web Scraping Copilot features

Web Scraping Copilot provides the following features:

Code generation

Generate maintainable web scraping code with GitHub Copilot.

../_images/ai-workflow-0.1.0.gif

See Tutorial for Web Scraping Copilot.

Test management

Compare extracted data to expectations, including expected exceptions, check target pages in the embedded browser, and more.

../_images/test-management-1.0.0.gif

Project setup

Start a new project in seconds.

../_images/new-project-1.0.0.gif

Scrapy Cloud integration

If you use Scrapy Cloud, you can deploy your spiders with a click, and monitor cloud jobs from the spiders view.

../_images/scrapy-cloud-integration-0.1.0.png