Exporting to an SFTP server with Scrapy#

Note

Not to be confused with FTP (see Exporting to an FTP server with Scrapy) or FTPS.

To configure a Scrapy project or spider to export scraped data to an SFTP server:

pip install git+https://github.com/scrapy-plugins/scrapy-feedexporter-sftp

If you are using Scrapy Cloud, remember to add the following line to your requirements.txt file:

scrapy-feedexporter-sftp @ git+https://github.com/scrapy-plugins/scrapy-feedexporter-sftp

In your settings.py file, define FEED_STORAGES as follows:
settings.py#
```
FEED_STORAGES = {
    "sftp": "scrapy_feedexporter_sftp.SFTPFeedStorage",
}
```
If the setting already exists in your settings.py file, modify the existing setting to add the key-value pair above, instead of re-defining the setting.
Add a FEEDS setting to your project or spider, if not added yet.

The value of FEEDS must be a JSON object ({}).

If you have FEEDS already defined with key-value pairs, you can keep those if you want — FEEDS supports exporting data to multiple file storage service locations.

To add FEEDS to a project, define it in your Scrapy Cloud project settings or add it to your settings.py file:
settings.py#
```
FEEDS = {}
```
To add FEEDS to a spider, define it in your Scrapy Cloud spider-specific settings (open a spider in Scrapy Cloud and select the Settings tab) or add it to your spider code with the update_settings method or the custom_settings class variable:
spiders/myspider.py#
```
class MySpider:
    custom_settings = {
        "FEEDS": {},
    }
```
Add the following key-value pair to FEEDS:
```
{
    "sftp://<USER>:<PASSWORD>@<HOST>/<PATH>": {
        "format": "<FORMAT>"
    }
}
```
Where:
- <USER> and <PASSWORD> are your credentials for the SFTP server, percent-encoded.
- <HOST> is the SFTP server host, e.g. sftp.example.com or 203.0.113.123.
- <PATH> is the path where you want to store the scraped data file, e.g. scraped/data.csv.
  
  The path can include placeholders that are replaced at run time, such as %(time), which is replaced by the current timestamp.
- <FORMAT> is the desired output file format.
  
  Possible values include: csv, json, jsonlines, xml. You can also implement support for more formats.
  
  Warning
  
  If you export in CSV format, and in your spider code you yield items as Python dictionaries, only the fields present on the first yielded item are exported for all items.
  
  One solution is to customize output fields through the fields feed option of FEEDS or through the FEED_EXPORT_FIELDS Scrapy setting to explicitly indicate all fields to export.
  
  You can alternatively yield something other than a Python dictionary that supports declaring all possible fields, such as an Item object or an attrs object.

Running your spider now, locally or on Scrapy Cloud, will export your scraped data to the configured SFTP server location.