Exporting scraped data¶
How would you like to get your scraped data?
Download from Scrapy Cloud.
Use Scrapy to export to a file storage service, like Amazon S3 or Google Cloud Storage.
Use Scrapy to export to a database, message queue, indexer, or similar service.
File storage¶
Choose to which file storage service you wish to export using Scrapy:
File storage exporting with Scrapy also provides many options, including: batching, field customization, item filtering, compression.
You can also create your own Scrapy storage backend. Check the code of existing storage backends to learn more.
Item storage¶
Choose to which item storage service you wish to export using Scrapy:
You can also create a custom Scrapy item pipeline to implement item-based storage, for example using an existing Python asyncio client library for a database or a message queue service.