Mage Pro doesn’t just process data—it revolutionizes scalability. By dynamically splitting and distributing workloads, Mage intelligently scales pipelines vertically and horizontally in real-time, maintaining peak performance while cutting infrastructure costs by up to 40%.

Dynamic blocks transform rigid data pipelines into living, adaptive systems. Instead of hardcoding static DAGs, workloads evolve based on runtime conditions, input data, and resource availability—building fractal-like processing trees that optimize for both speed and efficiency without duplicating code.

This new paradigm in data orchestration delivers faster processing, better resource utilization, and seamless elasticity—all without overpaying for unused compute.

Dynamic blocks

A dynamic block will create multiple downstream blocks at runtime.

The number of blocks it creates is equal to the number of items in the output data of the dynamic block multiplied by the number of its downstream blocks.

For example, the following data loader block will output a list of 2 lists. The 1st item in the list contains a list of 3 dictionaries containing user information.

from typing import Dict, List


@data_loader
def load_data(*args, **kwargs) -> List[List[Dict]]:
    users = []
    metadata = []

    for i in range(3):
        i += 1
        users.append(dict(id=i, name=f'user_{i}'))
        metadata.append(dict(block_uuid=f'for_user_{i}'))

    return [
        users,
        metadata, # optional
    ]

If the above data loader has 2 downstream blocks (e.g. 2 blocks that depend on this data loader), then this dynamic block will create 6 total downstream blocks at runtime (3 users multiplied by 2 downstream blocks).

Data output shape of a dynamic block

A dynamic block must return a list of 2 lists of dictionaries (e.g. List[List[Dict]]).

For example, a data loader block returns the following output:

[
    {
        "id": 1,
        "name": "user_1"
    },
    {
        "id": 2,
        "name": "user_2"
    },
    {
        "id": 3,
        "name": "user_3"
    }
]

The number of items in this list of dictionaries will correspond to how many downstream blocks get created at runtime.

For example, if this dynamic block has a downstream transformer block with the following code:

from typing import Dict, List


@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
    data['id'] = int(data['id']) * 100
    return [data]

The return output data of this downstream transformer block will be:

[
    {
        "id": 100,
        "name": "user_1"
    }
]

Dynamically created blocks

Every downstream block from a dynamic block is referred to as a dynamically created block.

The number of these blocks created are determined by the return output data of a dynamic block.

For example, if a dynamic block returns the following data:

[
    [
        {
            "id": 1,
            "name": "user_1"
        },
        {
            "id": 2,
            "name": "user_2"
        },
        {
            "id": 3,
            "name": "user_3"
        }
    ],
    [
        {
            "block_uuid": "for_user_1"
        },
        {
            "block_uuid": "for_user_2"
        },
        {
            "block_uuid": "for_user_3"
        }
    ]
]

Then there will be 3 dynamically created blocks for every downstream block of that dynamic block.

If the dynamic block had 1 downstream block, then 3 blocks will be dynamically created.

For example, if the downstream block was a transformer with UUID anonymize_user_data and the following code:

from typing import Dict, List


@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
    data['id'] = int(data['id']) * 100
    return [data]

There will be 3 dynamically created blocks with the following UUID and return output data:

​Downstream blocks of dynamically created blocks

If a dynamically created block has downstream blocks, those downstream blocks will be created multiple times. The number of times those are created correspond to how many blocks were created dynamically from the dynamic block.

In the current example, 3 blocks were created dynamically (3 users, 1 downstream block from the dynamic block). If the above transformer block has 2 downstream blocks, then 6 more blocks will be created.

For example, if the transformer block anonymize_user_data has 2 downstream blocks with UUID clean_column_names and compute_engagement_score, the following blocks will also be dynamically created:

1. anonymize_user_data:for_user_1

   → clean_column_names:for_user_1
   → compute_engagement_score:for_user_1

2. anonymize_user_data:for_user_2

   → clean_column_names:for_user_2
   → compute_engagement_score:for_user_2

3. anonymize_user_data:for_user_3

   → clean_column_names:for_user_3
   → compute_engagement_score:for_user_3

The data that is passed into each of these downstream blocks (e.g. clean_column_names, compute_engagement_score) will come from its upstream block (e.g. anonymize_user_data:for_user_1, anonymize_user_data:for_user_2, anonymize_user_data:for_user_3).

For example, the block clean_column_names:for_user_1 and compute_engagement_score:for_user_1 will have input data with the following shape and value:

{
    "id": 100,
    "name": "user_1"
}

Reduce output

By default, dynamically created blocks will create more dynamically created blocks from their own downstream blocks. A dynamically created block will pass its return output data to its downstream blocks and so on.

However, a dynamically created block can be configured to reduce all the outputs across each dynamically created block and combine them into a single list of items.

If we configure the above transformer block anonymize_user_data to reduce its output, then there will only be 2 downstream blocks instead of 6.

The 2 downstream blocks will be clean_column_names and compute_engagement_score. The input data to each of these 2 downstream blocks will be:

[
    {
        "id": 100,
        "name": "user_1"
    },
    {
        "id": 200,
        "name": "user_2"
    },
    {
        "id": 300,
        "name": "user_3"
    }
]

Dynamic SQL blocks

SQL blocks can be dynamic or be a child of a dynamic block.

Here is an example pipeline with code:

    [data_loader]
          ↓
 [dynamic_sql_loader]
          ↓
    [transformer]

data_loader.py: dynamic block

@data_loader
def load_data(*args, **kwargs):
    return [[dict(uuid=1), dict(uuid=2)]]

dynamic_sql_loader: dynamic block, dynamic child

SELECT * FROM {{ df_1 }}
UNION ALL
SELECT 2 AS uuid

transformer: dynamic child

@transformer
def transform(data, *args, **kwargs):
    return data

When this pipeline is triggered, the following block runs will be created:

                                [data_loader]
                                 ↓         ↓
              [dynamic_sql_loader:0]     [dynamic_sql_loader:1]
                ↓                ↓         ↓                ↓
 [transformer:0:0]  [transformer:0:1]   [transformer:1:0]  [transformer:1:1]

How it works

Automatically generate new downstream blocks based on input data. Each unit adapts to its context, maximizing concurrency while preserving clean data lineage.

Hyper-concurrency engine:

Splits workloads into independent, self-managing units that execute across available compute resources with no manual intervention.

Adaptive pipeline architecture:

Create hybrid pipelines that combine static and dynamic parents, auto-generate UUIDs to avoid collisions, and orchestrate complex multi-parent relationships effortlessly.

Asynchronous execution matrix:

Sibling blocks execute independently without bottlenecks. Failures are isolated to individual branches without affecting other workflows.

Stream mode execution:

Process data before the full dataset lands, achieving up to 60% faster data delivery SLAs and up to 90% lower memory usage compared to batch processing.

Recursive reduction engine:

Combine outputs from dynamically created blocks through flexible reduction strategies like concatenation, summation, or merging—while maintaining full traceability.

Dynamic SQL support:

Dynamically generate and execute SQL blocks alongside Python blocks in fluid, scalable pipeline trees.

Why it matters

Traditional static pipelines force you to overprovision compute—or worse, fail at scale. Mage’s dynamic scaling and dynamic blocks offer:

• Infrastructure savings without sacrificing speed

• Scalability that adapts in real-time to data complexity

• Faster SLA attainment with continuous data processing

• Greater resiliency by isolating failure domains

• Simpler architecture with less duplicated code

Mage Pro lets you treat infrastructure as a flexible, intelligent ally—not a fixed bottleneck.

Scale smarter, spend less, and deliver faster. All without lifting a finger.