# Exporting to Azure Storage with Scrapy

To configure a [Scrapy](https://scrapy.org) project or spider to export scraped data to
[Azure Storage](https://learn.microsoft.com/en-us/azure/storage/):

1. You need Python 3.8 or higher.

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), make sure you are
   using [stack](https://support.zyte.com/support/solutions/articles/22000200402-changing-the-deploy-environment-with-scrapy-cloud-stacks) `scrapy:1.7-py38` or higher. Using the latest stack
   (`scrapy:2.14-20260217`) is generally recommended.
2. Install [scrapy-feedexporter-azure-storage](https://github.com/scrapy-plugins/scrapy-feedexporter-azure-storage):
   ```bash
   pip install git+https://github.com/scrapy-plugins/scrapy-feedexporter-azure-storage
   ```

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), remember to add the
   following line to your `requirements.txt` file:
   ```none
   scrapy-feedexporter-azure-storage @ git+https://github.com/scrapy-plugins/scrapy-feedexporter-azure-storage
   ```
3. In your `settings.py` file, define `FEED_STORAGES` as follows:
   settings.py
   ```python
   FEED_STORAGES = {
       "azure": "scrapy_azure_exporter.AzureFeedStorage",
   }
   ```

   If the setting already exists in your `settings.py` file, modify the existing
   setting to add the key-value pair above, instead of re-defining the setting.
4. Add a [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEEDS)
   setting to your project or spider, if not added yet.

   The value of `FEEDS` must be a JSON object (`{}`).

   If you have `FEEDS` already defined with key-value pairs, you can keep
   those if you want — `FEEDS` supports exporting data to multiple file
   storage service locations.

   To add `FEEDS` to a project, define it in your [Scrapy Cloud project
   settings](https://support.zyte.com/support/solutions/articles/22000200670-customizing-scrapy-settings-in-scrapy-cloud)
   or add it to your `settings.py` file:
   settings.py
   ```python
   FEEDS = {}
   ```

   To add `FEEDS` to a spider, define it in your Scrapy Cloud
   spider-specific settings (open a spider in Scrapy Cloud and select the
   **Settings** tab) or add it to your spider code with the [update_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.update_settings)
   method or the [custom_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.custom_settings) class variable:
   spiders/myspider.py
   ```python
   class MySpider:
       custom_settings = {
           "FEEDS": {},
       }
   ```
5. Add the following key-value pair to `FEEDS`:
   ```python
   {
       "azure://<ACCOUNT>.blob.core.windows.net/<CONTAINER>/<PATH>": {
           "format": "<FORMAT>"
       }
   }
   ```

   Where:
   - `<ACCOUNT>` is the name of your [storage account](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#storage-accounts), e.g.
     `myaccount`.
   - `<CONTAINER>` is the name of your [container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers), e.g. `mycontainer`.
   - `<PATH>` is the path where you want to store the scraped data file, e.g.
     `scraped/data.csv`.

     The path can include [placeholders](https://docs.scrapy.org/en/latest/topics/feed-exports.html#storage-uri-parameters) that are replaced at run time, such
     as `%(time)`, which is replaced by the current timestamp.
   - `<FORMAT>` is the desired [output file format](https://docs.scrapy.org/en/latest/topics/feed-exports.html#serialization-formats).

     Possible values include: `csv`, `json`, `jsonlines`, `xml`. You can
     also [implement support for more formats](https://docs.scrapy.org/en/latest/topics/exporters.html).
     > [!WARNING]
     > If you export in CSV format, and in your spider code you yield
     > items as Python dictionaries, only the fields present on the first yielded
     > item are exported for all items.
     > 
     > One solution is to [customize output fields](https://docs.scrapy.org/en/latest/topics/exporters.html#scrapy.exporters.BaseItemExporter.fields_to_export) through the `fields` [feed
     > option](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-options) of [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feeds) or
     > through the [FEED_EXPORT_FIELDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-fields) Scrapy setting to explicitly indicate all
     > fields to export.
     > 
     > You can alternatively yield something other than a Python dictionary that
     > supports declaring all possible fields, such as an [Item object](https://docs.scrapy.org/en/latest/topics/items.html#item-objects) or an
     > [attrs object](https://docs.scrapy.org/en/latest/topics/items.html#attr-s-objects).
6. Define the `AZURE_ACCOUNT_URL` and `AZURE_ACCOUNT_KEY` settings with
   your [credentials](https://learn.microsoft.com/en-us/python/api/overview/azure/storage-blob-readme?view=azure-python#types-of-credentials):
   settings.py
   ```python
   AZURE_ACCOUNT_URL = "https://<ACCOUNT>.blob.core.windows.net"
   AZURE_ACCOUNT_KEY = "<KEY>"
   ```

   You can alternatively define the `AZURE_CONNECTION_STRING` setting to a
   [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string):
   settings.py
   ```python
   AZURE_CONNECTION_STRING = "DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=xxxx;EndpointSuffix=core.windows.net"
   ```

   Or, if you have an [account URL that includes a SAS token](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens), use the
   `AZURE_ACCOUNT_URL_WITH_SAS_TOKEN` setting instead:
   settings.py
   ```python
   AZURE_ACCOUNT_URL_WITH_SAS_TOKEN = "https://my.blob.core.windows.net/source-en/source-english.docx?sv=2019-12-12&st=2021-01-26T18%3A30%3A20Z&se=2021-02-05T18%3A30%3A00Z&sr=c&sp=rl&sig=d7PZKyQsIeE6xb%2B1M4Yb56I%2FEEKoNIF65D%2Fs0IFsYcE%3D"
   ```

Running your spider now, locally or on Scrapy Cloud, will export your scraped
data to the configured Azure Storage location.