Pipeline Retries and Backfills: Reproduce History, Rewrite Futures

Mage Pro makes pipeline recovery, reprocessing, and historical backfills seamless. Instantly resume workflows from failure points, recreate datasets for any past period, or backfill years of data with precision and control.

Through built-in retry mechanics, partitioned backfilling, and powerful recovery options, Mage Pro ensures you can fix fast and recover faster—without re-running entire processes or losing progress.

Whether you’re debugging failures, reprocessing stale data, or migrating large datasets, Mage Pro empowers you to manage pipeline history and future execution effortlessly.

How it works

Define global, pipeline-level, or block-specific retry settings using retry_config in your metadata. Automatically retry block runs on failure with customizable delay, retries, and backoff strategies.

Instead of restarting entire pipeline runs, retry only the failed block runs or resume from a selected block within the individual pipeline run page.

Create backfills by:

• Setting date and time windows (e.g., backfill daily data over a year)
• Generating custom partitions through code (for dynamic backfill scenarios)

Access variables like interval_start_datetime, interval_end_datetime, and interval_seconds inside Python or SQL blocks to build smart, time-aware backfills.

Every retry is recorded in detailed logs, ensuring transparency and auditability.

Why it matters

Data workflows aren’t always linear—and when failures or gaps happen, restarting from scratch wastes time, compute, and budget. Mage Pro’s recovery and backfilling features offer:

• Fast recovery without re-running successful steps
• Flexible backfilling to restore missing or outdated data
• Higher developer efficiency through surgical retries and partial re-execution
• Lower operational risk by giving full control over history reproduction

Mage Pro helps you not just monitor history—but rebuild it better whenever you need.