TLDR

Migrations are usually scary, but they don’t have to be. Moving your dbt Cloud project to Mage Pro is almost as simple as cloning your dbt project into Mage Pro and running a pipeline. You can be fully migrated in about 10 minutes and start connecting your dbt models to Mage Pro blocks. This is a smooth transition

Table of contents

• Introduction

• Push your base Mage Pro project to GitHub

• Clone your dbt Cloud repo into Mage Pro

• The migration process and configuration

• Testing your connection

• Conclusion

Introduction

Migrating complex data projects can often feel like a monumental task. They can be riddled with potential pitfalls and lengthy configurations. They don’t have to be this way. If you follow along with this blog article you’ll see how easy it is to migrate from dbt Cloud to Mage Pro. This article will walk you through the detailed process, mirroring the "under 10 minutes" migration, highlighting every crucial step to ensure a seamless shift of your dbt workflows to Mage Pro.

Push your base Mage Pro project to GitHub

First, you’ll want to create a new GitHub repo to store your new Mage Pro project. To do this you’ll want to do the following:

1. Create a new GitHub repo

2. Authenticate and connect your Mage Pro cluster to the GitHub repo (check out this article for more details)

3. Use the Git terminal in Mage Pro to push the base Mage Pro project to GitHub

You'll know it worked when you see the basic folder structures, and some example files like load_titanic.py in the repo.

Clone your dbt Cloud repo into Mage Pro

Once you have authenticated with GitHub and pushed the base Mage Pro project to your repo, it’s time to migrate your dbt Cloud project. This is as simple as cloning your project from your dbt Cloud GitHub repo. To do this you should:

1. Navigate to the Mage Pro terminal or Mage Pro Git Terminal

2. Run cd <your_mage_pro_project>/dbt to enter into your dbt folder

3. Once you have entered your dbt folder from the Mage terminal or the Git terminal you can run git clone <https://github.com/><username>/<github_repo>.git

4. Refresh your file editor page and then you should be able to see your dbt project in your the file editor.

Once you successfully cloned your dbt Cloud project into Mage Pro you should see a file structure similar to what is pictured below.

The migration process and configuration

After cloning your dbt Cloud repository from GitHub, the next step is configuring two key files: profiles.yml and dbt_project.yml. These configurations are what actually connect your dbt project to your data warehouse - whether that's BigQuery, Snowflake, or whatever platform you're running.

Next, you need to update the name and profile settings in your dbt_project.yml file. Here's how to do it:

1. Open the Mage Pro editor and navigate to your dbt_project.yml file

2. Update both the name and profile fields to match your dbt project name in Mage (reference the screenshot below)

3. Save the file

Next you’ll want to configure the profiles.yml file. This file will have the default Duckdb configuration provided by Mage. To configure this file for Google BigQuery you’ll want to take the following steps:

1. Delete the default Duckdb configuration

2. Add the configuration from the code block under step 3

3. Save the file

<dbt project name>:
    outputs:
        dev:
            dataset: <BigQuery dataset name>
            job_execution_timeout_seconds: 300
            job_retries: 1
            keyfile: </home/src/<path_to_your_credentials_file>
            location: US
            method: service-account
            priority: interactive
            project: <dbt_project_id>
            threads: 4
            type: bigquery
    target: dev

After configuring your profiles.yml file, you need to copy it to your dbt project directory. The profiles.yml file should exist in two locations: your main dbt folder and your specific dbt project folder. For this tutorial, we're working with the Chicago_Land_Crime_Data project, so you'll need the profiles.yml file in both the dbt directory and the Chicago_Land_Crime_Data subdirectory.

Testing your connection

Now it’s time to test your integration with your data warehouse. Let’s make sure the migration worked and was configured correctly. Once we know we’re connected we can load dbt dependencies. Here’s how you can test your connection:

1. Navigate to your Mage terminal

2. run cd <mage_project_name/dbt/<dbt_project_name>

3. Run dbt debug in your terminal

If you followed the configuration steps correctly, you should see all checks passing.

If your dbt debug command fails with a connection error, verify that you have the profiles.yml file in both your dbt folder and your dbt project folder. Double-check that the name and profile fields in your dbt_project.yml file exactly match your project name.

Finally, you’ll want to load all your dependencies to complete the migration. Go ahead and run dbt deps from your Mage terminal in your dbt project folder.

This loads all the dependencies from your packages.yaml file. You'll see packages like dbt_utilities and dbt_date getting installed. Check your package-lock.yaml file to confirm everything loaded correctly.

Conclusion

Migrations don't have to be terrifying. What we just walked through proves that moving from dbt Cloud to Mage Pro is more about understanding the process than wrestling with complex technical challenges.

The key takeaways:

• Preparation matters: Spend time setting up your Git connections properly

• Configuration is critical: Get those yaml files right, especially the naming

• Test your configurations: Always verify with dbt debug before declaring victory

• Integration is where the magic happens: Use Mage Pro's orchestration capabilities to build something bigger than just transformations

Want to see this migration in action? Get a free trial of Mage Pro today and migrate your dbt Cloud project over to Mage.