Exporting to Google Drive with Scrapy

To configure a Scrapy project or spider to export scraped data to Google Drive:

1. You need Python 3.8 or higher.

   If you are using Scrapy Cloud, make sure you are using stack scrapy:1.7-py38 or higher. Using the latest stack (scrapy:2.11) is generally recommended.

2. Install scrapy-feedexporter-google-drive:

   pip install git+https://github.com/scrapy-plugins/scrapy-feedexporter-google-drive
   

   If you are using Scrapy Cloud, remember to add the following line to your requirements.txt file:

   scrapy-feedexporter-google-drive @ git+https://github.com/scrapy-plugins/scrapy-feedexporter-google-drive
   

3. In your settings.py file, define FEED_STORAGES as follows:

   settings.py

   FEED_STORAGES = {
       "gdrive": "scrapy_gdrive_exporter.gdrive_exporter.GoogleDriveFeedStorage",
   }
   

   If the setting already exists in your settings.py file, modify the existing setting to add the key-value pair above, instead of re-defining the setting.

4. Add a FEEDS setting to your project or spider, if not added yet.

   The value of FEEDS must be a JSON object ({}).

   If you have FEEDS already defined with key-value pairs, you can keep those if you want — FEEDS supports exporting data to multiple file storage service locations.

   To add FEEDS to a project, define it in your Scrapy Cloud project settings or add it to your settings.py file:

   settings.py

   FEEDS = {}
   

   To add FEEDS to a spider, define it in your Scrapy Cloud spider-specific settings (open a spider in Scrapy Cloud and select the Settings tab) or add it to your spider code with the update_settings method or the custom_settings class variable:

   spiders/myspider.py

   class MySpider:
       custom_settings = {
           "FEEDS": {},
       }
   

5. Add the following key-value pair to FEEDS:

   {
       "gdrive://drive.google.com/<FOLDER ID>/<PATH>": {
           "format": "<FORMAT>"
       }
   }
   

   Where:

   • <FOLDER ID> is the ID of the target root folder, e.g. 1uWBpSBe3CvF8u21qTrzDqjZ6uexample.

     Tip

     When inside a folder, the URL ends with the folder ID, e.g: https://drive.google.com/drive/folders/1uWBpSBe3CvF8u21qTrzDqjZ6uexample.

   • <PATH> is the path where you want to store the scraped data file, e.g. scraped/data.csv.

     The path can include placeholders that are replaced at run time, such as %(time), which is replaced by the current timestamp.

     Note

     scrapy-feedexporter-google-drive does not support overwriting or appending to files, it can only create new files every time.

   • <FORMAT> is the desired output file format.

     Possible values include: csv, json, jsonlines, xml. You can also implement support for more formats.

     Warning

     If you export in CSV format, and in your spider code you yield items as Python dictionaries, only the fields present on the first yielded item are exported for all items.

     One solution is to customize output fields through the fields feed option of FEEDS or through the FEED_EXPORT_FIELDS Scrapy setting to explicitly indicate all fields to export.

     You can alternatively yield something other than a Python dictionary that supports declaring all possible fields, such as an Item object or an attrs object.

6. Define the GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON setting as a Python string containing your service account credentials in JSON format:

   settings.py

   GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON = '{ "type": "service_account", "project_id": "myproject", "private_key_id": "…", "private_key": "…", "client_email": "…@email.iam.gserviceaccount.com", "client_id": "…", "auth_uri": "…", "token_uri": "…", "auth_provider_x509_cert_url": "…", "client_x509_cert_url": "…" }'
   

   Make sure you give your service account write access on the target folder. You can do that by sharing the folder with the email of the service account (client_email in the JSON above).

Running your spider now, locally or on Scrapy Cloud, will export your scraped data to the configured Google Drive location.