# Exporting to an SFTP server with Scrapy

> [!NOTE]
> Not to be confused with [FTP](https://en.wikipedia.org/wiki/File_Transfer_Protocol) (see [Exporting to an FTP server with Scrapy](ftp.md#ftp))
> or [FTPS](https://en.wikipedia.org/wiki/FTPS).

To configure a [Scrapy](https://scrapy.org) project or spider to export scraped data to an [SFTP
server](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol):

1. Install [scrapy-feedexporter-sftp](https://github.com/scrapy-plugins/scrapy-feedexporter-sftp):
   ```bash
   pip install git+https://github.com/scrapy-plugins/scrapy-feedexporter-sftp
   ```

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), remember to add the
   following line to your `requirements.txt` file:
   ```none
   scrapy-feedexporter-sftp @ git+https://github.com/scrapy-plugins/scrapy-feedexporter-sftp
   ```
2. In your `settings.py` file, define `FEED_STORAGES` as follows:
   settings.py
   ```python
   FEED_STORAGES = {
       "sftp": "scrapy_feedexporter_sftp.SFTPFeedStorage",
   }
   ```

   If the setting already exists in your `settings.py` file, modify the existing
   setting to add the key-value pair above, instead of re-defining the setting.
3. Add a [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEEDS)
   setting to your project or spider, if not added yet.

   The value of `FEEDS` must be a JSON object (`{}`).

   If you have `FEEDS` already defined with key-value pairs, you can keep
   those if you want — `FEEDS` supports exporting data to multiple file
   storage service locations.

   To add `FEEDS` to a project, define it in your [Scrapy Cloud project
   settings](https://support.zyte.com/support/solutions/articles/22000200670-customizing-scrapy-settings-in-scrapy-cloud)
   or add it to your `settings.py` file:
   settings.py
   ```python
   FEEDS = {}
   ```

   To add `FEEDS` to a spider, define it in your Scrapy Cloud
   spider-specific settings (open a spider in Scrapy Cloud and select the
   **Settings** tab) or add it to your spider code with the [update_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.update_settings)
   method or the [custom_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.custom_settings) class variable:
   spiders/myspider.py
   ```python
   class MySpider:
       custom_settings = {
           "FEEDS": {},
       }
   ```
4. Add the following key-value pair to `FEEDS`:
   ```python
   {
       "sftp://<USER>:<PASSWORD>@<HOST>/<PATH>": {
           "format": "<FORMAT>"
       }
   }
   ```

   Where:
   - `<USER>` and `<PASSWORD>` are your credentials for the SFTP server,
     [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding).
   - `<HOST>` is the SFTP server host, e.g. `sftp.example.com` or
     `203.0.113.123`.
   - `<PATH>` is the path where you want to store the scraped data file, e.g.
     `scraped/data.csv`.

     The path can include [placeholders](https://docs.scrapy.org/en/latest/topics/feed-exports.html#storage-uri-parameters) that are replaced at run time, such
     as `%(time)`, which is replaced by the current timestamp.
   - `<FORMAT>` is the desired [output file format](https://docs.scrapy.org/en/latest/topics/feed-exports.html#serialization-formats).

     Possible values include: `csv`, `json`, `jsonlines`, `xml`. You can
     also [implement support for more formats](https://docs.scrapy.org/en/latest/topics/exporters.html).
     > [!WARNING]
     > If you export in CSV format, and in your spider code you yield
     > items as Python dictionaries, only the fields present on the first yielded
     > item are exported for all items.
     > 
     > One solution is to [customize output fields](https://docs.scrapy.org/en/latest/topics/exporters.html#scrapy.exporters.BaseItemExporter.fields_to_export) through the `fields` [feed
     > option](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-options) of [FEEDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feeds) or
     > through the [FEED_EXPORT_FIELDS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-fields) Scrapy setting to explicitly indicate all
     > fields to export.
     > 
     > You can alternatively yield something other than a Python dictionary that
     > supports declaring all possible fields, such as an [Item object](https://docs.scrapy.org/en/latest/topics/items.html#item-objects) or an
     > [attrs object](https://docs.scrapy.org/en/latest/topics/items.html#attr-s-objects).

Running your spider now, locally or on Scrapy Cloud, will export your scraped
data to the configured SFTP server location.