7.6 Practice exercise

Scenario

You work for NYC's Performance Analytics team. City managers need a daily summary of 311 service requests by 8:00 AM to plan daily operations and resource allocation. Your pipeline processes the latest 311 data to track response times by department, identify emerging neighborhood issues, monitor seasonal trends, and support resource planning for city services.

Exercise Requirements

Create a schedule trigger with the following specifications:

• Pipeline Name: "nyc_311_daily_analytics"

• Business Requirement: 311 analytics dashboard must be updated by 8:00 AM EST daily

• Data Source: NYC Open Data 311 Service Requests (updates continuously)

• Estimated Processing Time: 25 minutes (processing ~50MB of daily data)

• Audience: City managers who review metrics during 8:30 AM operational meetings

Step-by-Step Trigger Implementation

Step 1: Navigate to Trigger Configuration

1. Open your "nyc_311_daily_analytics" pipeline in Mage

2. Click the "Triggers" tab

3. Select "Add trigger" → "Schedule"

Step 2: Basic Trigger Setup

• Name: daily_311_analytics_trigger

• Description: Processes NYC 311 service requests for city manager dashboard - must complete by 8:00 AM EST for operational planning

Step 3: Configure Schedule Settings

• Frequency: Select "Daily"

• Timezone: Remember all times are in GMT you must convert this in the UI to Eastern time.

• Start Date: Set to tomorrow's date to begin the schedule

Step 4: Enable Landing Time (Mage Pro)

• Toggle "Enable landing time" to ON

• Pipeline complete by:

  • Hour: 07

  • Minute: 59

  • Second: 00

Step 5: Configure Run Settings

• Timeout: 2400 seconds (40 minutes) - provides buffer for data volume spikes

• Status for timed out runs: Failed (default) - ensures immediate alerts for missed deadlines

• Skip run if previous run still in progress: Enable - prevents overlapping executions during heavy processing days

Expected Trigger Behavior

• Initial runs: Pipeline starts around 7:35 AM EST (25-minute buffer before 8:00 AM deadline)

• After optimization: Mage Pro adjusts start time based on actual processing performance

• Weekend processing: Continues daily schedule to capture weekend service requests

• Holiday handling: Automatic execution ensures no gaps in operational data

Validation Checklist

• [ ] Trigger successfully created with daily frequency

• [ ] Landing time configured for 8:00 AM EST completion

• [ ] Timeout settings provide adequate processing buffer

• [ ] Manual test run completes within expected timeframe

This trigger configuration ensures NYC city managers have reliable, timely access to 311 analytics data for daily operational decision-making.