3.5 Sensor

A sensor is a block that continuously evaluates a condition until it's met or until a period of time has elapsed. Sensors act as gatekeepers in your pipeline, ensuring downstream blocks only execute when specific conditions are satisfied.

Understanding sensor functionality

If there is a block with a sensor as an upstream dependency, that block won't start running until the sensor has evaluated its condition successfully. This creates powerful orchestration capabilities where your pipelines can wait for external events, data availability, or other pipelines to complete.

Sensors can check for anything. Examples of common sensors check for: Does a table exist? Does a partition of a table exist? Does a file in a remote location exist? Has another pipeline finished running successfully?

Common sensor use cases

Pipeline dependencies: Wait for another pipeline to complete before starting dependent processing.

Data availability: Check if new files have arrived in S3 or if database tables have been updated.

External system readiness: Verify that APIs are responding or that downstream systems can accept new data.

Time-based conditions: Wait for specific times or business hours before processing sensitive data.

Resource availability: Check system resources or queue lengths before starting intensive processing.

Sensor timing and behavior

Currently, the sensor makes a check every 60 seconds. Sensors run continuously until their condition is met or a timeout period elapses. This prevents pipelines from running indefinitely while waiting for conditions that may never be satisfied.

When designing sensors, consider:

• Timeout strategies: Set reasonable time limits to prevent indefinite waiting

• Check frequency: Balance responsiveness with system resource usage

• Condition specificity: Make sensor conditions precise to avoid false positives

Conclusion

Sensors transform Mage pipelines from simple sequential processes into intelligent, event-driven workflows that respond to real-world conditions. By implementing sensors thoughtfully, with appropriate timeouts, reasonable check frequencies, and precise conditions, you create data pipelines that automatically coordinate with external systems and dependencies.