3.8 Add on blocks

Add on blocks are specialized components that enhance regular pipeline blocks with additional functionality without disrupting the main data flow. Unlike standard blocks that process and pass data between each other, add-on blocks attach to existing blocks to provide supplementary capabilities like conditional execution, error handling, and notifications. They operate independent of the primary data pipeline while responding to the execution state of their parent blocks. There are two types of add on blocks, Conditional blocks and Callback blocks.

Conditionals

Conditional blocks control whether their parent blocks execute based on dynamic criteria evaluated at runtime. A conditional block is associated with another block and determines if that block should run by evaluating specific conditions before execution begins.

Understanding conditional execution

When you attach a conditional block to a parent block, the conditional logic runs first. If the conditional block returns True, the parent block executes normally. If it returns False, the parent block is skipped entirely and its status is set to condition_failed. This creates intelligent pipeline flows that adapt based on data characteristics, external factors, or business rules.

Callbacks

A callback block is associated to another block. When the parent block succeeds or fails, the callback block functions are executed. Callbacks provide a powerful mechanism for implementing side effects, notifications, and cleanup operations that should occur based on pipeline execution results.

Understanding callback execution

Callbacks respond to the success or failure of their parent blocks, enabling you to implement sophisticated error handling, monitoring, and notification strategies. This reactive pattern ensures your data engineering team stays informed about pipeline health and can respond quickly to issues.

You can have as many functions in a callback block as you want. Use @callback or @callback('success') to decorate a function that should only run when the parent block successfully completes.

Practical callback applications

Monitoring and alerting: Send notifications to Slack, email, or monitoring systems when pipelines succeed or fail.

Data quality reporting: Generate and distribute data quality reports after transformation blocks complete.

Cleanup operations: Remove temporary files, close connections, or release resources after processing.

Audit logging: Record pipeline execution details for compliance and debugging purposes.

External system coordination: Trigger downstream processes or update external dashboards based on pipeline results.

Conclusion

Add-on blocks extend Mage's core functionality by providing conditional execution and reactive capabilities that make pipelines more intelligent and resilient. Conditional blocks enable dynamic pipeline behavior that responds to real-world conditions, while callbacks ensure proper handling of execution outcomes. Together, these add-on blocks transform static data processing sequences into adaptive workflows that can handle complex business requirements and operational scenarios without manual intervention.