Web Scraping Copilot features

Web Scraping Copilot provides the following features:

Code generation

Generate maintainable web scraping code with GitHub Copilot.

../../_images/ai-workflow-0.1.0.gif

See AI-assisted web scraping tutorial.

Test management

Compare extracted data to expectations, check target pages in the embedded browser, and more.

../../_images/test-management-1.0.0.gif

Project setup

Start a new project in seconds.

../../_images/new-project-1.0.0.gif

Scrapy Cloud integration

If you use Scrapy Cloud, you can deploy your spiders with a click, and monitor cloud jobs from the spiders view.

../../_images/scrapy-cloud-integration-0.1.0.png