How to migrate an AWS Glue job and replay its input safely

To migrate an AWS Glue job, capture the script and job configuration, then trace what its bookmark treats as processed. Rebuild the transform and checkpoint together. Use staging and a repeatable target write so a replay can correct records without inserting them twice. A bookmark tracks input progress. Rewinding it leaves existing target files in place. AWS Glue job bookmarks

How do you retry an AWS Glue migration without reviving deleted rows?

A customer load restarts after losing its checkpoint response. Inspect the sink receipt for the immutable object manifest and check key, source version and payload hash against target state.

Recovery. Reuse a matching commit receipt and continue from it. If the manifest is uncommitted, merge the admitted higher versions and preserve tombstones. Hold conflicting payloads at the same key and version under the agreed policy.

Cutover check. Compare objects actually read by Glue, target key/version/delete state, and rejected rows against the shadow run. Require one commit receipt after replay and verify that a late update cannot resurrect a newer tombstone.

Run python3 glue-bookmark-replay.py after saving the Glue replay lab. It compares the replacement with an independent synthetic expected result. Version 3 deletes the customer, and a later replay of version 2 leaves the tombstone intact. A different payload at the same version is held. Reopening the local database and replaying the committed delete reuses its one receipt. The lab also rejects conflicting stale rows within the same manifest.

Rebuild a bookmarked customer feed with transactional version checks

This synthetic slice replaces an S3-to-database Glue job. It assumes the producer supplies a monotonic version per customer and explicit delete records. If the legacy producer overwrites S3 keys, collect object versions where enabled or stage immutable content before processing. Bookmark snapshots help explain which inputs the old job selected. Confirm actual consumed objects against its logs and sink records. Glue bookmark behavior

Freeze and validate the stage before merging

Build a stage snapshot for the exact selected object versions. Store (batch_id, source_object_version, source_row, customer_id, version, operation, payload_hash). Distinct payload hashes for the same customer and version are a conflict, whether the conflicting rows came from the same object or different objects. Detect the same conflict against the existing target. Quarantine that key or block the batch according to the agreed policy. Within the admitted set, collapse byte-equivalent normalized payloads and choose a single highest version per key with a stable source-row tie-breaker. Include delete state in the digest.

In a SQLite/PostgreSQL-style target, the accepted one-winner-per-key stage can use this upsert. Adapt parameter binding, transaction locking and dialect to the actual target.

INSERT INTO customer_state AS target
  (customer_id, version, customer_name, is_deleted, payload_hash)
SELECT customer_id, version, customer_name, is_deleted, payload_hash
FROM customer_winner
WHERE batch_id = :batch_id
ON CONFLICT (customer_id) DO UPDATE SET
  version = excluded.version,
  customer_name = excluded.customer_name,
  is_deleted = excluded.is_deleted,
  payload_hash = excluded.payload_hash
WHERE excluded.version > target.version;

Keep tombstones in customer_state and filter them out in the consumer view. Equal versions with equal digests have no write effect. A stale late file remains attributable to its object/version with a disposition, even when it cannot change current state. Store a batch receipt with the admitted source manifest, upsert count, stale count, rejects and the output checksum or destination transaction reference. PostgreSQL INSERT and conditional ON CONFLICT

Prove the sink result and the source boundary

Seed C1 v1. Supply C1 v2, C2 v1, one duplicate C2 v1, one malformed version and a conflicting C1 v2. First test without the conflict and expect two current keys plus one parse reject. Then test the conflicting input as its own versioned batch and expect a conflict with C1 v2 unchanged. Apply C1 DELETE v3 and replay the former C1 v2 object. Keep C1 v3 as a tombstone and C2 as the single live key. Inject a crash after sink commit and before the external checkpoint update. The same manifest must recover one receipt and leave exactly the committed rows.

In parallel operation read both pipelines from a known immutable batch and write to a shadow table. Compare source objects, key/version/state, rejects and receipt counts. Reconcile current C1 state and the source history separately. At cutover freeze the Glue trigger and any external workflow invocation, wait for active writers, then transfer ownership at the last verified source manifest. Keep a bounded overlap only when the target merge and receipt logic are proven safe under replay.

Implementation runbook

Export the real job run and its control plane

aws glue get-job --job-name "$JOB_NAME" --region "$REGION" > glue-job.json
aws glue get-job-run --job-name "$JOB_NAME" --run-id "$RUN_ID" \
  --region "$REGION" > glue-run.json
aws glue get-job-bookmark --job-name "$JOB_NAME" --region "$REGION" > glue-bookmark.json
aws glue get-triggers --dependent-job-name "$JOB_NAME" --region "$REGION" > glue-triggers.json
aws glue get-table --database-name "$DATABASE" --name "$TABLE" \
  --region "$REGION" > glue-source-table.json

Use the job/run metadata and deployment revision to obtain Job.Command.ScriptLocation, for example aws s3 cp "$SCRIPT_URI" ./glue-script.py for the current script. Get the pinned S3 object version or archived deployed build when comparing a historical run. Save catalog partition definitions, crawlers or schema deployment if relevant. Save Glue version, Spark/runtime type, workers, Connections, jar/wheel/archive versions, connector and IAM/VPC context as identifiers. Resolve effective arguments in order: NonOverridableArguments > run Arguments > DefaultArguments. Capture condition/schedule actions and verify their JobName. Also inspect Glue Workflows or external EventBridge/Scheduler callers, because the trigger list is not all invocation sources. Logs/arguments and bookmark state may contain restricted data. Get job · Get run · Get bookmark · Argument precedence

Replace bookmark state, not just the SQL

Map each source's transformation_ctx, job.init and job.commit, bookmark mode and actual target write mode. For S3 inputs bookmarks use object modification time. For JDBC check actual bookmark key columns, order and monotonicity, whether keys are append-only, and how corrections/deletes arrive. A JDBC business timestamp alone can miss records with equal or earlier timestamps. Choose source CDC sequence/offset for mutable tables where available. A compound (timestamp, primary_key) cursor covers equal timestamps in an ordered stable extract but still requires overlap/reconciliation or CDC for older late changes/deletes. Bookmark pause reprocesses an incremental range without advancing bookmark. Rewind/reset does not delete target outputs. A changed S3 path under the same context may retain old bookmark history. Do not use a bookmark run ID as a transferable offset. Recover the last approved committed source identity from old-run and sink evidence. AWS Glue bookmark semantics · Program bookmark keys

Implement a batch with versioned files and monotonic merges

1. Build source_manifest(object_key, version_id, observed_at, content_hash, rows) for one closed interval. With S3 Versioning, read the exact VersionId. On an unversioned source copy to a stable immutable staging location and verify the bytes/hash before using it. An ETag is an identifier/check rather than a universal SHA or MD5 content hash. Record the old consumed files in a comparable manifest and compare them to the new selection.
2. Stage rows with customer_id, version, source_file, source_row, payload_hash, is_deleted. Check for null keys and conflicting same-version payloads. Collapse only identical version records and pick highest version deterministically. Let invalid rows follow documented quarantine or batch-abort policy. If deleting, keep a tombstone version or history so an older late update cannot recreate a removed row.
3. Run a sink-specific transaction: write higher versions only, skip exact equal-version repeats, reject differing equal-version payloads, persist a commit receipt for the manifest-derived batch ID. In one database advance the input cursor and commit receipt with the merge. Across systems, confirm sink batch ID/receipt or replay an idempotent merge before updating external progress. Pin a high boundary at launch and keep a closed-interval policy across retries.
4. Schedule a bounded lookback for late updates. Retry same manifest and same batch ID on transient errors. Surface schema drift/conflicts with input version and row coordinates. Shadow on a separate table, validate raw and distinct input counts, winners and rejects by key/version, tombstones and downstream timing. Only then transfer the trigger to the new writer.

Test the crash points, late updates and deletes

Start target C1 v1 Ana. Stage C1 v2 Ana Lee, C2 v1 Dev, late C1 v1 Ana, and exact repeat C2 v1 Dev. Expect two committed keys with versions 2 and 1. Stop after sink commit, before external checkpoint. Retry same manifest and confirm same two records plus one sink receipt. Send C1 v2 with different text: conflict, preserve prior approved v2. Next send C1 tombstone v3 then late C1 v2: the live view contains only C2 and the tombstone still blocks resurrection. Test overwriting an S3 key under a new VersionId so replay reads intended bytes. Compare a full historical slice, a live delta with a late record, and an isolated rollback. Hand off at the last verified input manifest rather than merely the last Glue run timestamp.

AI prompt to generate adapters and parity checks

Given this sanitized job, job run, bookmark contexts, catalog schema and two object manifests, extract the effective arguments and source selection rule with artifact references. Implement stage-only loading and monotonic upsert with a durable commit receipt. Include an exact duplicate, same-version conflict, tombstone and timeout after commit. Print both expected target state and checkpoint state after every retry. Flag where dialect and connector guarantees need operator confirmation.

Lake Formation access in AWS Glue 4.0 and 5.0

Capture the deployed Glue version, read API and Lake Formation permissions together. AWS documents the GlueContext DynamicFrame Lake Formation table-level access pattern for Glue 4.0 and earlier. The Glue 5.0 migration guide retires that legacy Lake Formation integration. Full-table access has a separate support path. For row, column or cell access controls in 5.0, use Spark DataFrames with Spark-native fine-grained access control. Jobs that do not need fine-grained access control can have different choices. Check the job's actual access mode before changing the read API. AWS Glue 5.0 migration guide

Run the same allowed and disallowed row and column checks under the production-equivalent roles in each version. Compare the records the old job could actually read and the target role should be allowed to read. The SQLite replay lab covers target writes and receipts. Lake Formation authorization requires a separate sandbox test.

Try a duplicate-safe replay fixture

Use Python 3 to test the merge rule independently of the scheduler. Here the greater version wins, and an equal version is a no-op for identical input.

target = {"C1": (1, "Ana")}
batch = [("C1", 2, "Ana Lee"), ("C2", 1, "Dev"), ("C1", 1, "Ana")]

def apply(rows):
    for key, version, name in rows:
        previous = target.get(key)
        if previous and version == previous[0] and name != previous[1]:
            raise ValueError("Different content at the same source version")
        if previous is None or version > previous[0]:
            target[key] = (version, name)

apply(batch)
apply(batch)
assert target == {"C1": (2, "Ana Lee"), "C2": (1, "Dev")}
try:
    apply([("C1", 2, "conflicting name")])
except ValueError:
    pass
else:
    raise AssertionError("A same-version conflict must be rejected")

Run the same fixture against your staging table. Then interrupt a test after the target commit and before the checkpoint update. Replay the batch and confirm the target and checkpoint converge.

Copy these prompts into your AI assistant

Inspect this sanitized Glue script, job arguments, and trigger configuration. Mark each read that participates in bookmarks. List its ordering key, destination write mode, and a failure case that would create duplicates. Cite the code you used.

Draft an incremental loader for this input manifest and target schema. Include a higher-version-only merge, a separate rejects table, and fixtures for duplicate replay and late updates. Return a run summary with counts and a batch ID.

Schedule and cut over the replacement

Schedule one active writer per destination partition and bound retries for temporary connection failures. Stop on bad keys or schema drift and route those errors to the data owner. A scheduled chatbot can summarize new batches, rejects, and missing checkpoints from the run log for review.

Run a historical slice and at least one live delta into a separate sink. Compare keys, versions, counts, rejects, and delivery time. Freeze the old trigger at an agreed boundary and let the replacement begin from the last verified input boundary. Keep the old definition and manifest available for rollback, and reconcile which batches reached the target before restoring a prior schedule.

References

• AWS CLI get-job
• AWS Glue bookmark programming
• AWS Glue job parameters

Need help with the migration?

Explore AWS Glue migration services or browse all DIY migration guides.