Mage Pro’s Global Data Products feature allows you to generate a data output once and reuse it across any pipeline in your project. Whether the data is an in-memory DataFrame, a serialized JSON object, or a database table, it can be registered globally, referenced anywhere, and intelligently refreshed only when needed.

Instead of re-running expensive pipelines or duplicating data across workflows, Global Data Products act as shared, lazy-evaluated building blocks—ensuring consistent outputs, reducing compute costs, and minimizing redundant processing.

Global Data Products are registered once, versioned automatically, and lazily triggered only when their data is outdated or requested by another pipeline, ensuring maximum efficiency at scale.

How it works

Global registration:

Promote any pipeline output into a Global Data Product by assigning it a unique ID (UUID) through the global registry.

Universal referencing:

Use any Global Data Product as a block inside any pipeline. Other blocks can depend on it just like any other in-pipeline block.

Lazy triggering:

A Global Data Product only runs when its data is requested and the data is outdated—preventing unnecessary recomputation.

Outdated data management:

Configure how long a data product stays fresh (e.g., 12 hours, 1 week) and define when it should become eligible for reprocessing based on specific times or dates.

Partition control:

Customize how much historical data to retrieve by setting partition windows when consuming a Global Data Product.

Override settings per use:

Pipelines referencing a Global Data Product can override freshness thresholds, output block selections, and partition ranges for precise control.

Concurrency protection:

If multiple pipelines request a Global Data Product at the same time, Mage Pro ensures it runs once and shares the output, avoiding duplicate executions.

Why it matters

Global Data Products transform your pipeline architecture from siloed workflows into a modular, reusable system. They allow teams to:

• Build once, reuse everywhere across pipelines

• Reduce compute costs by preventing redundant executions

• Centralize key data assets with guaranteed freshness controls

• Accelerate downstream workflows with precomputed, ready-to-use datasets

• Simplify dependency management across complex multi-pipeline projects

• 
  

With Mage Pro, data pipelines aren’t just about transformation—they become living ecosystems of reusable, intelligent data products.