Exporting to Google Cloud Storage with Scrapy#

To configure a Scrapy project or spider to export scraped data to Google Cloud Storage:

Install google-cloud-storage:
```
pip install google-cloud-storage
```
If you are using Scrapy Cloud, remember to add the following line to your requirements.txt file:
```
google-cloud-storage
```
Add a FEEDS setting to your project or spider, if not added yet.

The value of FEEDS must be a JSON object ({}).

If you have FEEDS already defined with key-value pairs, you can keep those if you want — FEEDS supports exporting data to multiple file storage service locations.

To add FEEDS to a project, define it in your Scrapy Cloud project settings or add it to your settings.py file:
settings.py#
```
FEEDS = {}
```
To add FEEDS to a spider, define it in your Scrapy Cloud spider-specific settings (open a spider in Scrapy Cloud and select the Settings tab) or add it to your spider code with the update_settings method or the custom_settings class variable:
spiders/myspider.py#
```
class MySpider:
    custom_settings = {
        "FEEDS": {},
    }
```
Add the following key-value pair to FEEDS:
```
{
    "gs://<BUCKET>/<PATH>": {
        "format": "<FORMAT>"
    }
}
```
Where:
- <BUCKET> is your bucket name, e.g. mybucket.
- <PATH> is the path where you want to store the scraped data file, e.g. scraped/data.csv.
  
  The path can include placeholders that are replaced at run time, such as %(time), which is replaced by the current timestamp.
  
  Warning
  
  Any pre-existing file in the specified path will be overwritten. Google Cloud Storage does not support appending to a file.
- <FORMAT> is the desired output file format.
  
  Possible values include: csv, json, jsonlines, xml. You can also implement support for more formats.
  
  Warning
  
  If you export in CSV format, and in your spider code you yield items as Python dictionaries, only the fields present on the first yielded item are exported for all items.
  
  One solution is to customize output fields through the fields feed option of FEEDS or through the FEED_EXPORT_FIELDS Scrapy setting to explicitly indicate all fields to export.
  
  You can alternatively yield something other than a Python dictionary that supports declaring all possible fields, such as an Item object or an attrs object.
Configure credential provision to ADC.

Also define the GCS_PROJECT_ID Scrapy setting with your project ID:
settings.py#
```
GCS_PROJECT_ID = "myproject"
```
Additional settings exist to define, for example, a custom access-control list.

Running your spider now, locally or on Scrapy Cloud, will export your scraped data to the configured Google Cloud Storage location.