3.6 Practical exercise

You're a Data Engineer at NYC's Performance Analytics team who has just securely stored your credentials in Mage's Secrets Manager. Now you need to establish the actual connection between your Mage pipeline environment and Google BigQuery so your data transformations can write to the warehouse.

Your challenge is to configure Mage's io_config.yml file with your GCP service account details.

Prerequisites:

Your service account credential file should have been stored in your project folder. If this has not yet been completed follow the steps in section 3.3 to upload your GCP credentials file.

Complete the steps below to configure a connection between Mage and Google Cloud Platform. You can substitute other data warehouses or databases if BigQuery is not a part of your data stack:

• Configure the file path in your io_config.yml file

• Add the following yaml configurations to your io_config.yml file:

  • project_id

  • private_key_id

  • private_key

  • client_email

  GOOGLE_SERVICE_ACC_KEY:
    type: service_account
    project_id: project-id
    private_key_id: key-id
    private_key: "-----BEGIN PRIVATE KEY-----\\your_private_key\\n-----END_PRIVATE_KEY"
    client_email: your_service_account_email
    auth_uri: "<https://accounts.google.com/o/oauth2/auth>"
    token_uri: "<https://accounts.google.com/o/oauth2/token>"
    auth_provider_x509_cert_url: "<https://www.googleapis.com/oauth2/v1/certs>"
    client_x509_cert_url: "<https://www.googleapis.com/robot/v1/metadata/x509/your_service_account_email>"
  GOOGLE_SERVICE_ACC_KEY_FILEPATH: "/path/to/your/service/account/key.json"
  GOOGLE_LOCATION: US # Optional