You need to deploy a new version of a critical payment service with zero downtime. If problems occur, how would you safely roll back while minimizing customer impact?

How I frame it

Payment deploys are not just "swap pods." I clarify: in-flight transactions, webhook ordering, DB schema changes, and client retries all continue during rollout. Zero downtime means no failed payments attributable to deploy — not merely HTTP 200 from load balancer.

Zero-downtime deploy strategy

• Rolling / blue-green: New pods pass health + readiness (DB, PSP sandbox ping) before receiving traffic. Old pods drain via preStop.
• Canary: 1% → 5% → 25% → 100% on new version. Watch payment success rate, 5xx, p99 authorize latency, duplicate charge alerts.
• Backward-compatible API: v2 reads v1 request schema; new fields optional. Feature flags gate new PSP routing logic.
• DB migrations — expand-contract: Add columns/tables in prior release; dual-write; switch reads; remove old column in later release — never destructive DDL in same deploy as code flip.

Safe rollback

• Keep previous version warm (blue-green) — flip traffic back in seconds, not rebuild.
• Automated rollback if canary burns SLO: payment failure rate > 2× baseline for 5 min.
• Idempotent payments: Same Idempotency-Key across v1/v2 — client retry during rollback does not double-charge.
• Graceful drain: preStop waits for in-flight authorize/capture to finish (bounded timeout); reject new work early via readiness probe.
• Webhook consumers: Both versions must handle same event schema during overlap; dedupe by event_id.

Deploy v2 canary 5% → payment failure 3× baseline
  Auto-rollback: 100% traffic → v1 within 60s
  In-flight v2 authorize: idempotency key → v1 returns same result on retry
  DB: only additive schema — rollback never needs column drop

Walkthrough

Friday 14:00: v2 canary at 5%. Metrics green. 14:30: spike in PSP timeout errors on v2 only. Pipeline auto-rolls back to v1. Customer double-clicked Pay during flip — second request hits v1 with same idempotency key → returns original payment_id, no duplicate charge. Ops postmortem: v2 had tighter HTTP timeout to PSP; fix in v2.1 before next canary.

Weak answer

"Deploy at 3 AM when traffic is low." That is maintenance window, not zero downtime. "Kubernetes rolling update" alone ignores payment idempotency and schema compatibility.

Follow-up questions

Can we deploy DB and code in one step?

No for payments. Expand-contract across at least two releases. Rollback must not require undoing destructive migration.

What if PSP webhook arrives at v2 but we rolled back to v1?

Both versions share processed_events dedupe table and payment state machine — webhook handler is version-agnostic.

Feature flag vs canary?

Canary controls traffic % to binary; flags control code paths within a version. Use both — flag new routing inside canary slice first.