Onboarding calls a KYC vendor that is flaky (timeouts, duplicate callbacks, stale status). How do you design retries, state machine, and merchant UX so applications neither get stuck nor auto-approve wrongly?

KYCThird-PartyState MachineReliability

How I frame it

External KYC is unreliable and legally sensitive. I never treat "no response" as approve. Own an internal application state machine; the vendor is an input, not the system of record. UX must show progress without lying about clearance.

Internal state machine

Persist applications(id, merchant_id, status, vendor_ref, attempt, updated_at) before every vendor call.
Statuses: DRAFT → SUBMITTED → VENDOR_PENDING, then terminal APPROVED | REJECTED | NEEDS_INFO | MANUAL_REVIEW. Optional RETRY_WAIT for transient failures.
Never transition to APPROVED on timeout, empty body, or stale cached vendor status.
Terminal approve/reject only from: verified webhook signature + known vendor_ref, or successful status poll that matches a signed response, or human ops decision.

KYC application state machine (vendor ≠ source of truth)

DRAFT
  │ submit
  ▼
SUBMITTED ──► VENDOR_PENDING ──► VENDOR_APPROVED ──► MANUAL_REVIEW? ──► APPROVED
                  │                      │                    │
                  │ timeout/5xx          │ duplicate webhook  │ reject
                  ▼                      │                    ▼
             RETRY_WAIT ───────────────┘              REJECTED / NEEDS_INFO
                  │
                  │ max attempts / aged out
                  ▼
             NEEDS_MANUAL (never auto-APPROVED on silence)

Retries, duplicates, stale callbacks

Outbound call: Short timeout (e.g. 5–10s). On timeout → stay VENDOR_PENDING / move RETRY_WAIT; schedule poll. Do not invent failure.
Idempotent submit: Same application_id / vendor idempotency key on retry — vendor must not create two cases.
Webhooks: Verify HMAC + timestamp skew window. Dedupe on event_id. Apply only if vendor_status_version or event time ≥ last applied — drop stale "pending" after "approved".
Dual path: Webhook + periodic poll of OPEN applications (reconcile). Poll wins conflict via version / updated_at.
Backoff: Exponential + jitter for retries; cap attempts then NEEDS_MANUAL — not silent loop forever, not auto-approve.

Webhook order problems
  t1: status=APPROVED (event_id E2)
  t2: late retry status=PENDING (event_id E1)  → IGNORE (stale)
  Duplicate E2 → dedupe table hit → 200, no state change

Merchant UX

After submit: "Verification in progress — usually under 2 hours", not Success.
Polling/SSE on application status for the dashboard; email/push on terminal states.
If stuck in VENDOR_PENDING past SLA: show "Taking longer than usual — our team is on it" + support ticket link. Unlock limited product features only if product policy allows (read-only), never full money movement.
NEEDS_INFO: clear checklist (re-upload ID) — do not restart from blank without reason codes.

Walkthrough

Merchant submits KYC. Vendor API times out after you already POSTed — you store vendor_ref from a prior 202 or reconcile by external idempotency key. Status stays PENDING; UI shows in progress. Duplicate APPROVED webhooks arrive twice — first transitions to MANUAL_REVIEW (policy: high-value always human check); second is deduped. Late PENDING webhook ignored. Ops never sees false APPROVED from flakiness alone.

Weak answer

"If vendor times out after 30s, mark rejected and let them reapply." May reject an actually-approved case and create duplicate vendor records. "Trust the last webhook blindly." Stale reorder breaks state.

Follow-up questions

Vendor returns APPROVED then later REVOKED?

Support revoke transition → freeze payouts, flag compliance queue. Versioned status history table, never overwrite without audit row.

PII storage?

Minimize: store vendor reference + decision, not raw ID images if vendor hosts them. Encrypt at rest; retention policy aligned with regulation.

Multi-vendor failover?

Second vendor only after primary exhaustion; map statuses through your enum. Never APPROVED if vendors disagree — MANUAL_REVIEW.

How do you load-test this?

Stub vendor with chaos (timeout, duplicate, out-of-order). Assert no APPROVED without signed success path; measure stuck-application age SLI.