2.4 Creating pipelines

Pipeline creation in Mage Pro transforms data engineering concepts into executable workflows. Pipelines orchestrate the flow of data through various processing stages, from initial extraction to final delivery to target systems.

Understanding Mage Pro pipelines

A pipeline in Mage Pro represents a complete data workflow composed of interconnected blocks. Each pipeline is essentially a Directed Acyclic Graph (DAG) where blocks represent processing steps and dependencies determine execution order.

Pipelines solve real business problems by automating data flows. For example, you might create a pipeline that extracts customer data from your CRM, enriches it with demographic information, and delivers analytics-ready datasets to your data warehouse.

Pipeline types in Mage Pro

Batch: Traditional ETL workflows that process data in scheduled batches. Use these for daily sales reports, monthly customer analytics, or any scenario where you process data at regular intervals.

Streaming: Real-time data processing pipelines for continuous data flows. Choose streaming when you need immediate insights, like fraud detection or live recommendation engines.

Data Integration: Specialized pipelines for syncing data between systems. These excel at keeping multiple databases synchronized or migrating data between platforms.

Step-by-step pipeline creation

Step 1: Access pipeline creation From the Mage Pro home page, navigate to the pipelines page by hovering over the left popout navigation menu and click the pipelines button. Once you are in the pipelines page click the green “New pipeline” button. After creating the new pipeline click “Start from scratch.”

Step 2: Choose your pipeline type Select the appropriate type based on your use case:

• Choose "Standard (Batch)" for most ETL scenarios

• Select "Streaming" for real-time processing needs

• Pick "Data Integration" for system-to-system synchronization

Step 3: Configure basic settings Provide a descriptive pipeline name that clearly indicates its purpose (e.g., "customer churn analysis" or "daily sales etl"). Add a detailed description documenting the pipeline's data sources, transformations, and business purpose. Don’t worry about adding any blocks at the moment, we’ll take care of those steps in a later section.

Conclusion

Pipeline creation forms the foundation of data engineering work in Mage Pro. By understanding the different pipeline types and following the systematic creation process, you establish the framework for all your data processing activities. The pipeline you create serves as the blueprint that determines how data flows through your system, making this initial setup crucial for project success. Once you've configured your pipeline's basic settings and chosen the appropriate type for your use case, you're ready to begin adding the blocks that will perform the actual data processing work.