How to onboard and validate a custodian data feed

Onboard a custodian by agreeing on account entitlements, mapping sample files into a canonical schema, and running the new feed in parallel against independent control totals. Release the connection only after a replay produces the same records and users can see missing or corrected data.

Write the feed contract

Start with the account population, as-of schedule, transport, file types, versions, time zones, and the recipient's expected output. Identify full snapshots and incremental files separately. Schwab's daily data files are one example of a feed providing positions, balances, transactions, and profile information, and its API integration offers a separate access path. Confirm the capabilities and access terms of the custodian you actually use. Schwab daily data files · Schwab API integration

Move a sample feed through acceptance

What to automate

Approve access

Write a feed contract with the custodian and downstream owner. Specify authorized accounts, expected files, transport, business date, time zone and missing-file contacts. Keep test and production connection details separate and store credentials with the approved secret mechanism. Require an entitlement check on every input and record who can approve a scope change.

Land and inventory

At each expected arrival, record the custodian, file type, as-of date, sequence, checksum and receipt time. Preserve the raw copy and compare the manifest against the calendar. An identical checksum is a replay. A changed file for the same date needs the agreed correction signal before replacement. Alert the operator on missing files or unapproved conflicting copies.

Map and normalize

Use the partner's dictionary and the approved account crosswalk to map signs, units, dates, security IDs and transaction types. Save mappings with versions and examples. Ask AI for a draft from a synthetic sample, then have the implementation analyst confirm ambiguous fields with the custodian. Retain unknown rows and their original positions in a quarantine file.

Prove parity

Run the same date through both feeds and compare account counts, record counts, native-currency cash and position quantities against independent custody totals. Separate rounding, expected timing and true data breaks with codes agreed by Operations. Compare an original delivery, replay and authorized correction. Hold acceptance if a required file is missing or an unexplained break exceeds tolerance.

Cut over and monitor

Write an acceptance checklist naming the feed owner, recipient and rollback conditions. After an approved parallel run, switch the recipient to the new output and watch freshness, account scope and schema changes. Keep the previous delivery available through the agreed fallback window. Record the exact cutover version and practice restarting from the last accepted source date.

Test corrections and duplicate files

Use a synthetic file containing two authorized accounts with balances of $500 and $700. Load it twice and expect two account balances totaling $1,200. Then load an approved correction that changes the second account to $730. The total should be $1,230 and the prior file should remain traceable. Add an unexpected account and a missing position file. The first belongs in quarantine and the second keeps the acceptance gate open. Measure first-valid-feed time, on-time arrival, exception age, and per-custodian reconciliation differences.

Implement a feed manifest and a shadow reconciliation

Create feed_scope(partner_id, tenant_id, permitted_account, data_type, environment, approval_ref) before provisioning the endpoint. Use a scoped non-production credential for testing. Track the issuer, account, as-of date, source revision, original checksum, file count and declared row count for each delivery.

1. Poll approved SFTP or API locations without enabling destination writes. Retrieve each complete file into immutable staging. Bind a received file to the expected manifest and effective feed version. Retry interrupted downloads before parsing and never log key material or data rows into an incident channel.
2. Distinguish retry, correction and conflict. An identical account/date/version/hash is a no-op. A new approved revision is a correction. The same logical revision with changed bytes is a hold. If the partner supplies no revision, confirm correction semantics and use a controlled intake approval before switching.
3. Normalize account and instrument keys, sign, unit and source currency. Apply separate checks for authorized account, validated schema, row count, source totals and destination totals. Reconcile cash and positions against the independent report before switching.
4. Record a receipt only after the destination confirms the published version. Shadow multiple business days with replay, wrong account, absent file, correction and late receipt. Assign post-cutover ownership, expected arrival and escalation targets.

def classify(current_revision, current_hash, new_revision, new_hash, allowed):
    if not allowed:
        return "hold unauthorized account"
    if new_revision < current_revision:
        return "preserve history"
    if new_revision == current_revision:
        return "duplicate" if new_hash == current_hash else "hold conflict"
    return "stage correction"

assert classify(1, "a", 1, "a", True) == "duplicate"
assert classify(1, "a", 1, "b", True) == "hold conflict"
assert classify(1, "a", 2, "b", True) == "stage correction"
assert classify(1, "a", 2, "b", False) == "hold unauthorized account"

Implement the work step by step

The client implementation lead decides whether the recipient can rely on a new connection with account scope, output parity, recoverability and independently controlled cutover. The workflow begins when an approved scope and partner contact exist. Ends after parallel acceptance, cutover and handoff to ongoing feed operations. Post-cutover routine feed processing belongs to its consuming operation.

Authorize and configure

• Confirm scope and entitlements. Generate checklists and run entitlement preflight in test environment. AI can extract candidate terms from redacted feed contract. Read Custody onboarding forms, account roster, delivery contract. Record Signed feed scope to integrations. Owner Client implementation lead. Obtain customer and custodian authorizations. Check No credential or scoped account confirmation means no production setup.
• Set up transfer and endpoint test. Provision scoped secrets in approved vault and run read-only handshake. Read API or SFTP contract, vault, test endpoint. Record Validated credentials and test connection to feed owner. Owner Integration engineer. Partner approves access and controls rotation contacts. Check Keep prod and test separate and log receipt without key material.

Map the canonical custodian schema

• Inventory sample files and revisions. Validate manifests and types, persist revisions and checksums, list missing files. Read Position, balance, transaction and profile manifests. Record Sample set with expected as-of dates to engineer. Owner Implementation analyst. Escalate missing approved fixture to custodian. Check Cannot treat correction as duplicate.
• Prepare canonical schema transform. Compile approved transform and fixture tests, isolate parse errors. AI can suggest mapping with data dictionary references. Read Sample records, field dictionary, target schema. Record Versioned parser and mapping to QA. Owner Integration engineer. Approve sign, scale and effective dates. Check Validate row grain and preserve source currency.

Validate and cut over

• Run daily parallel reconciliation. Calculate control comparisons by account/date/currency and route differences. Read Approved files, incumbent output, source controls. Record Parity report to client implementation lead. Owner Operations analyst. Inspect tolerance exceptions. Check Require sign-off across five feeds and keep permitted output scope.
• Resolve rejected or late partner file. Route by error type, retain replay lineage, preserve last good output. AI can draft a redacted explanation for internal review. Read Exception queue, protocol messages, amended file. Record Validated replay or blocked connection. Owner Implementation analyst. Resolve genuine scope and protocol changes. Check Do not publish a partial feed as accepted.
• Approve cutover and support handoff. Enforce exact approved config. Capture new delivery receipt and monitor first run. Read Parity report, recipients, rollback runbook. Record Active connection, agreed service owner and rollback path. Owner Client implementation lead. Customer or authorized owner signs off and accepts support scope. Check Test rollback and cutover rollback receipt.

Parallel run. Pilot one custodian and one controlled account group across at least five delivery days with duplicate, correction, unauthorized account and missing file tests.

Build it yourself with AI

Start with two synthetic accounts and a mock custodian directory. Save the approved columns, file cadence and control totals in feed_contract.md. Give AI the fictional sample files and a data dictionary, then let it draft a local intake command without connecting to the custodian.

1. Draft the field mapping and test plan

   Propose a typed mapping for positions, balances and transactions. Record the source field, target field, sign, scale, identity key, null policy and evidence for each choice. Return unresolved meanings as questions for the implementation analyst.

2. Generate a replayable intake prototype

   Create a Python command that writes a raw manifest, standardized rows and a quarantine file. Load synthetic balances of 500 and 700 twice and expect two accounts totaling 1200. Load an approved correction of 730 for the second account and expect 1230 with the earlier version retained. Add an unauthorized account and a missing position file. Show both issues in the acceptance report.

3. Prepare the cutover handoff

   Using the parity results, draft a runbook for the feed owner. Include the last accepted date, alert recipients, correction sequence, required approvals and the rollback procedure. Label each open break and cite its source file.

Have Operations reconcile the parallel run against a trusted reference and independently check entitlements. Use test endpoints to exercise the missing-file and correction paths. Switch the production recipient only once the feed owner and receiving team have signed off on the same accepted version.

Need help automating this?

See how Mage handles file arrivals, field mapping, validation and delivery for customer and vendor feeds. See Mage's financial file orchestration solution.