Web Scraping Copilot features#

Web Scraping Copilot provides the following features:

Code generation#

Generate maintainable web scraping code with GitHub Copilot.

../../_images/ai-workflow-0.1.0.gif

See AI-assisted web scraping tutorial.

Spider and page object views#

Browse your spiders and page objects with new interactive views. Run your spiders locally with a click, generate new test fixtures for your page objects, and more.

../../_images/spider-and-page-object-views-0.1.0.png

Scrapy Cloud integration#

If you use Scrapy Cloud, you can deploy your spiders with a click, and monitor cloud jobs from the spiders view.

../../_images/scrapy-cloud-integration-0.1.0.png