# Exporting to Amazon S3 with Scrapy

To configure a [Scrapy](https://scrapy.org) project or spider to export scraped data to
[Amazon S3](https://aws.amazon.com/pm/serv-s3/):

1. Install [boto3](https://github.com/boto/boto3):
   ```bash
   pip install boto3
   ```

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), remember to add the
   following line to your `requirements.txt` file:
   ```none
   boto3
   ```
2. Add a [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEEDS)
   setting to your project or spider, if not added yet.

   The value of `FEEDS` must be a JSON object (`{}`).

   If you have `FEEDS` already defined with key-value pairs, you can keep
   those if you want — `FEEDS` supports exporting data to multiple file
   storage service locations.

   To add `FEEDS` to a project, define it in your [Scrapy Cloud project
   settings](https://support.zyte.com/support/solutions/articles/22000200670-customizing-scrapy-settings-in-scrapy-cloud)
   or add it to your `settings.py` file:
   settings.py
   ```python
   FEEDS = {}
   ```

   To add `FEEDS` to a spider, define it in your Scrapy Cloud
   spider-specific settings (open a spider in Scrapy Cloud and select the
   **Settings** tab) or add it to your spider code with the [update_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.update_settings)
   method or the [custom_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.custom_settings) class variable:
   spiders/myspider.py
   ```python
   class MySpider:
       custom_settings = {
           "FEEDS": {},
       }
   ```
3. Add the following key-value pair to `FEEDS`:
   ```python
   {
       "s3://<BUCKET>/<PATH>": {
           "format": "<FORMAT>"
       }
   }
   ```

   Where:
   - `<BUCKET>` is your [bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) name, e.g. `mybucket`.
   - `<PATH>` is the path where you want to store the scraped data file, e.g.
     `scraped/data.csv`.

     The path can include [placeholders](https://docs.scrapy.org/en/latest/topics/feed-exports.html#storage-uri-parameters) that are replaced at run time, such
     as `%(time)`, which is replaced by the current timestamp.
     > [!WARNING]
     > Any pre-existing file in the specified path will be
     > overwritten. [Amazon S3 does not support appending to a file](https://stackoverflow.com/a/41783997).
   - `<FORMAT>` is the desired [output file format](https://docs.scrapy.org/en/latest/topics/feed-exports.html#serialization-formats).

     Possible values include: `csv`, `json`, `jsonlines`, `xml`. You can
     also [implement support for more formats](https://docs.scrapy.org/en/latest/topics/exporters.html).
     > [!WARNING]
     > If you export in CSV format, and in your spider code you yield
     > items as Python dictionaries, only the fields present on the first yielded
     > item are exported for all items.
     > 
     > One solution is to [customize output fields](https://docs.scrapy.org/en/latest/topics/exporters.html#scrapy.exporters.BaseItemExporter.fields_to_export) through the `fields` [feed
     > option](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-options) of [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feeds) or
     > through the [FEED_EXPORT_FIELDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-fields) Scrapy setting to explicitly indicate all
     > fields to export.
     > 
     > You can alternatively yield something other than a Python dictionary that
     > supports declaring all possible fields, such as an [Item object](https://docs.scrapy.org/en/latest/topics/items.html#item-objects) or an
     > [attrs object](https://docs.scrapy.org/en/latest/topics/items.html#attr-s-objects).
4. Define the [AWS_ACCESS_KEY_ID](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-AWS_ACCESS_KEY_ID) and [AWS_SECRET_ACCESS_KEY](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-AWS_SECRET_ACCESS_KEY) Scrapy settings
   with your [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html):
   settings.py
   ```python
   AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
   AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
   ```

   You can alternatively define the [AWS_SESSION_TOKEN](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-AWS_SESSION_TOKEN) setting to configure
   access with [temporary security credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys).

   [Additional settings](https://docs.scrapy.org/en/latest/topics/feed-exports.html#s3)
   exist to define a target region, a custom access-control list, or a custom
   endpoint.

Running your spider now, locally or on Scrapy Cloud, will export your scraped
data to the configured Amazon S3 location.