# Exporting to Google BigQuery with Scrapy

To configure a [Scrapy](https://scrapy.org) project or spider to export scraped data to
[Google BigQuery](https://cloud.google.com/bigquery/):

1. You need Python 3.7 or higher and Scrapy 2.4 or higher.

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), make sure you are
   using [stack](https://support.zyte.com/support/solutions/articles/22000200402-changing-the-deploy-environment-with-scrapy-cloud-stacks) `scrapy:2.4` or higher. Using the latest stack
   (`scrapy:2.14-20260217`) is generally recommended.
2. Install [scrapy-bigquery](https://github.com/8W9aG/scrapy-bigquery):
   ```bash
   pip install scrapy-bigquery
   ```

   If you are using [Scrapy Cloud](../../../../scrapy-cloud/get-started.md#scrapy-cloud), remember to add the
   following line to your `requirements.txt` file:
   ```none
   scrapy-bigquery
   ```
3. Define the `BIGQUERY_DATASET` and `BIGQUERY_TABLE` Scrapy settings to
   point to the target table. For example:
   settings.py
   ```python
   BIGQUERY_DATASET = "my-dataset"
   BIGQUERY_TABLE = "my-table"
   ```

   [Additional settings](https://github.com/8W9aG/scrapy-bigquery#bigquery_add_scraped_time-optional)
   are available.
   > [!TIP]
   > To add Scrapy settings to a project, define them in your [Scrapy
   > Cloud project settings](https://support.zyte.com/support/solutions/articles/22000200670-customizing-scrapy-settings-in-scrapy-cloud)
   > or add them to your `settings.py` file.
   > settings.py
   > ```python
   > MY_SETTING = ...
   > ```
   > 
   > To add settings to a spider, define them in your Scrapy Cloud
   > spider-specific settings (open a spider in Scrapy Cloud and select the
   > **Settings** tab) or add it to your spider code with the [update_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.update_settings)
   > method or the [custom_settings](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.custom_settings) class variable:
   > spiders/myspider.py
   > ```python
   > class MySpider:
   >     custom_settings = {
   >         "MY_SETTING": ...,
   >     }
   > ```
4. Define the `BIGQUERY_SERVICE_ACCOUNT` setting as a string with your
   [service account credentials](https://developers.google.com/identity/protocols/oauth2/service-account) in base64-encoded JSON format:
   settings.py
   ```python
   BIGQUERY_SERVICE_ACCOUNT = "eyJ0eX=="
   ```

   You can use the following command to generate the required value from your
   service account JSON file:
   ```shell
   cat service-account.json | jq . -c | base64
   ```

   Make sure you give your service account write access on the target table.
   You can do that by sharing the table with the email of the service account
   (`client_email` in the service account JSON).

Running your spider now, locally or on Scrapy Cloud, will export your scraped
data to the configured Google BigQuery table.