How to migrate an AWS Step Functions data workflow

To migrate an AWS Step Functions data workflow, pick one state machine with a known output, such as a checked file manifest. Export the deployed definition, map each transition and side effect, then rebuild the workflow with explicit state and idempotent writes. Use a fixture that interrupts the run after one write to prove the retry is safe.

Should a timed-out Step Functions task be retried?

A required file task times out after dispatch. Look up the destination operation key and use the executed state machine and history to distinguish a confirmed write, a retryable error and a terminal failure.

Recovery. Keep an existing matching receipt and retry an absent write only when the destination contract makes that safe. Hold an ambiguous result without read-back or dedupe. An unhandled permanent required-file failure blocks publication.

Cutover check. Compare the old and shadow run's state outputs, selected Catch or reject branches, file receipts and manifest. The worked case requires A and C once, a B schema reject, and no manifest when required C fails permanently.

Translate a file workflow's data shape and receipts into runnable tasks

This synthetic Standard workflow loads a file batch into a sandbox. Before implementing, replay an observed successful execution and an expected failure against the deployed ASL. In a JSONPath task, record what each projection does to the input and response. The replacement must pass equivalent data to the next task even when the storage and runtime change. Step Functions JSONPath processing · ResultPath

For example, a per-file task can use Parameters to take the file ID and immutable object version, ResultSelector to keep only the sink receipt, and ResultPath: "$.delivery" to attach the result to the task input. A connector response containing {"Payload":{"receiptId":"R-A","bytes":140}} then supplies $.delivery.receiptId downstream if the deployed selector maps that field. Test the actual response path for your service integration. A result filtered out by OutputPath will not be available to the next state.

Test the real branch behavior

Use the accepted ASL execution as a trace oracle. For this example pass orders generation g1 with A (valid), B (bad schema), C (valid). Simulate A's first call accepting the write then losing its response. Expect A and C receipts, a BAD_SCHEMA reject for B and one manifest with sorted accepted IDs A and C. Next fail required C with an unhandled permanent error. Its planned work item ends failed, A remains in the sandbox and no final manifest appears. Reset the test and put a connector error inside a matching Catch. Compare the captured error path and error/causal context against the actual legacy execution event. Include a default Choice branch and an input missing a field that an earlier Choice expects. Retry and Catch handling

Transfer trigger ownership with a per-item ledger

Deploy the new scheduler or durable workflow and a worker limited to the downstream write capacity. Persist the transition and side-effect receipts so a restart can finish a batch after a worker disappears. Use the stable key feed:generation:file_id:operation. Keep attempt count only in logs and retry budget. For task token callbacks, heartbeats, waits or compensation, select a runtime that explicitly supports those behaviors and test restoration with an in-flight execution. Service integration patterns

Route initial runs to a separate sink and keep the old trigger in charge. Compare the planned input, normalized state output, file side effects, reject/catch states and final manifest. Stop new old-side triggers at the cutover boundary. Let old callback or waiting work finish under its existing owner, or move it with a documented transfer of task state. A rollback stops new dispatch and reconciles the per-item receipts before letting the old trigger reclaim an incomplete batch.

Implementation runbook

Export the definition used by a run

For Standard workflows capture an observed execution and the matching historical state machine. For Express, use the deployed release artifact and CloudWatch history when enabled. The Standard execution-history API is unavailable. Capture traffic routing as well as latest definition when production uses an alias.

aws stepfunctions describe-state-machine --state-machine-arn "$STATE_MACHINE_ARN" \
  --region "$REGION" > current-state-machine.json
aws stepfunctions describe-state-machine-for-execution --execution-arn "$EXECUTION_ARN" \
  --region "$REGION" > executed-state-machine.json
aws stepfunctions get-execution-history --execution-arn "$EXECUTION_ARN" \
  --no-include-execution-data --region "$REGION" > execution-history.json
jq -r .definition executed-state-machine.json > workflow.asl.json

Collect restricted redacted example inputs separately. If alias routing splits versions, record both versions and match each sample to the right one. Capture upstream invokers (API, EventBridge, another state machine, explicit client), IAM/KMS/queue dependencies, downstream code or service endpoints, service integration invocation pattern and response mapping. Export Lambda code and any aliases/layers from the original release sources. Describe state machine · Definition for execution · Get execution history · Express execution details

Decode state-machine semantics before translating to tasks

Record effective QueryLanguage per state. For JSONPath reconstruct order of InputPath, Parameters, connector result, ResultSelector, ResultPath, OutputPath. A null ResultPath discards the task result and retains original input. For JSONata, record Arguments, Output, Assign and the state input/result variables. For Choice preserve rule order, type/presence checks and default. For Map record ItemsPath/item selection, inline vs Distributed Map, concurrency, item batching, tolerated failure, and result locations. Model an entire Parallel/Map retry separately from a single work-item retry. Record ordered Retry and Catch selectors, interval, backoff, maximum attempts, jitter, timeout/heartbeat, catcher payload and abort semantics. Avoid generalizing workflow-level exactly-once behavior to external HTTP, database or message writes. Any .sync, callback task token, multi-day Wait or compensation requires a durable state machine or workflow engine. JSONPath processing · ResultPath · JSONata · Service integration patterns · Retry/Catch

Implement one bounded file state machine

• Define ValidateManifest → PlanFiles → ProcessFile[bounded] → CheckControls → PublishManifest with a reject route for business validation and separate terminal route for permanent service failures. Feed the planner stable input generation/IDs and sort output manifest keys explicitly if consumers rely on order.
• Store run(feed,business_date,input_manifest_hash,status) and step(run_id,item_id,step,attempt,status,destination_receipt). Plan file work idempotently. Give each side effect a stable destination key that excludes attempt number. Use a sink uniqueness constraint/idempotency token or conditional write and reconcile ambiguous receipts. A local “started” row is insufficient to know whether a timed-out destination accepted a write. If no safe read-back/dedupe exists, stop ambiguous retries for an operator.
• Limit workers according to destination limits. Handle backoff/retry budget separately for throttles/timeouts and avoid retrying schema failures. Save error class plus mapped input context on the reject path. Permit final publication only when planned file count equals accepted + terminal business rejects and all required writes have receipts. File manifest with missing required files stays uncommitted.
• During shadow tests compare per-file transformed payload, branch choices, final result object/manifest, retry outcomes, and logged delivery timing. Preserve durable task identity for restored work. At the switch stop new invocations, account for all running/callback executions and reconcile receipts before handing off. Send identical trigger events only to the one designated production writer.

Worked trace and crash test

Manifest orders/2026-01-07#v1: A valid, B invalid schema, C valid. With two workers, A lands and its response times out, B emits one reject BAD_SCHEMA, C writes once. Reconcile A by the key orders/2026-01-07#v1:A:load and obtain its original receipt. Expect exactly 2 writes, 1 reject, 1 manifest with A and C, with no terminal unaccounted input. In a second run C fails permanently: no published manifest and A remains in the isolated test sink. Test the alternative branch and Catch payload against the Standard execution history, and test a single retry and a replay of the entire Map.

AI prompt to convert one state machine

Using this sanitized ASL, its execution events and documented service result shapes, walk each example through input mapping, branch order, service side effects, retry/catch and output mapping. Return a transition table with state names and evidence. Then generate a local fake adapter for success, schema rejection, throttling and timeout after acceptance. Assert one destination effect per stable file key and no final manifest after a permanent required-branch failure.

Try a restart fixture

This Python 3 example uses a set to model a destination that accepts a stable idempotency key. Run it and inspect the keys after a retry. Replace the set with the destination's actual operation for integration testing.

published = set()

def process(fail_after_first=False):
    for key in ("A", "B"):
        published.add(("2026-01-07", key))
        if fail_after_first:
            raise TimeoutError("Write may have succeeded")

try:
    process(fail_after_first=True)
except TimeoutError:
    pass
process()
assert published == {("2026-01-07", "A"), ("2026-01-07", "B")}

Add an invalid file fixture and verify it produces a reject and no success output. Test both the intended Catch path and an error that must stop publishing entirely.

Copy these prompts into your AI assistant

Read this sanitized ASL definition and sample inputs. List each state, its data mapping, possible next states, external side effects, and retry rules. Show which transitions depend on behavior defined in service code or runtime settings that I still need to collect.

Draft a Python file-processing flow with injectable adapters for the input and destination. Keep the agreed Choice and Catch behavior. Include tests for a timeout after a successful write, a failed Map item, and an invalid file. Use isolated staging and a stable key for each write.

Schedule and switch the trigger

Route each trigger to one active runner and keep a ledger keyed by event or batch ID. Cap retries and route exhausted attempts to an owner. A scheduled chatbot can summarize failed branches and identify batches missing manifests from the run log. Give it a read-only log view.

Replay representative histories in a sandbox, comparing branch paths, reject reasons, output keys, and timing. Stop new starts on the old trigger, account for in-flight executions, then enable the new trigger at the agreed boundary. To roll back, restore the old route and replay only events whose outputs are absent from the ledger.

References

• AWS CLI describe-state-machine
• Step Functions error handling
• Standard and Express workflow types

Need help with the migration?

Explore AWS Step Functions migration services or browse all DIY migration guides.