🦆 Making Magic: MotherDuck with Mage

First published on February 9, 2024

 

3 minute read

Matt Palmer

TLDR

Mage now supports a suite of DuckDB & MotherDuck features— from reading and writing DuckDB databases to executing dbt with dbt-duckdb!

🤝 Mage & MotherDuck

Here at Mage, we're always looking for the latest and greatest ways to make data engineering simple, accessible, and

fun

.

That's why we've parntered with

to introduce

complete

support for DuckDB and Motherduck! 🪄 🦆

🚀 New Features

In version

0.9.28

and above, you'll have access to a suite of DuckDB features like:

  • Reading from/writing to DuckDB databases:

    Mage can read data from DuckDB databases directly into your piplines

    and

    output to DuckDB. That includes our easy-to-use templates for loading/exporting!

  • Streaming data to DuckDB destinations:

    Since Mage supports to streaming to

    any

    of our supported destinations, DuckDB is a de-facto streaming sink!

  • Executing dbt models using

    : With an

    amazing

    contribution by our community, Mage supports executing dbt models in DuckDB SQL!

And new in version

0.9.63

is support for:

  • Reading/writing to MotherDuck datasources:

    With DuckDB support comes out-of-the-box support for MotherDuck's serverless analytics platform.

Exporting to a database is as simple as using a data exporter, which can select directly from an upstream block:

DuckDB Exporter

You can also

read

from a database in one-line with our DuckDB data loader:

Since DuckDB is implemented in Mage as it's own

IO class

, that also makes DuckDB available as a

streaming

destination. That's right, you can stream from any source

directly

do a DuckDB database.

Last, but certainly not least, you can take this data and execute

dbt models

! Talk about a cherry on top! 🍒

Leveraging the

library, Mage can seamlessly integrate your dbt project with DuckDB sources and destinations, allowing you to leverage our hybrid architecture to execute SQL, Python, and dbt models in the same pipeline. 🤯

You can read more about configuring DuckDB as a storage location

(and the MotherDuck companion

) get started building today! 🦆