Payment succeeds but inventory reserve fails (or payment succeeds and fulfillment service is down). How do you design compensating actions so money and stock never drift?

How I frame it

Cross-service checkout cannot be one ACID transaction. I use a saga: a sequence of local commits with compensating actions that undo prior steps if a later step fails. Goal: never charge without a valid business hold, never leave inventory reserved after a failed payment, never auto-approve a shipment you cannot fulfill without an explicit path (retry / refund / ops).

Clarify order of steps. Preferred: Create order → Reserve stock → Charge → Fulfill. Charging before reserve creates unnecessary refunds when stock is gone — avoid that when possible.

Preferred happy path (and why)

1. Order PENDING with saga_id / idempotency_key.
2. Inventory reserve — atomic hold with TTL (or row lock). Idempotent on order_id. Soft reservation, not ship-confirm.
3. Payment charge — same idempotency key across retries. Only after reserve succeeds.
4. Mark PAID → enqueue fulfillment via outbox. Fulfillment is async and retries; payment is already committed.

Client → Order service (orchestrator)
           |
           |-- 1. Create order PENDING -----------------> Order DB
           |-- 2. Reserve inventory (idempotent key) ---> Inventory
           |       FAIL? → mark order FAILED, stop
           |-- 3. Charge payment (idempotency key) -----> Payment / PSP
           |       SUCCESS + fulfillment down?
           |         → keep PAID, enqueue fulfill (don't refund yet)
           |       FAIL after reserve?
           |         → ReleaseInventory compensate → order FAILED
           |-- 4. Fulfill / ship ------------------------> Fulfillment
           |       FAIL forever?
           |         → RefundPayment + ReleaseInventory (or hold for ops)

Failure matrix — what to compensate

• Reserve fails (sold out): Order → FAILED. No payment call. No compensation needed.
• Reserve OK, payment fails: ReleaseInventory(order_id) (must be idempotent). Order → FAILED.
• Reserve OK, payment UNKNOWN (timeout): Do not release inventory or refund yet. Stay PAYMENT_UNKNOWN; reconcile with PSP. Only after confirmed FAILED → release stock.
• Payment SUCCESS, fulfillment down: Money is real — do not immediately refund. Keep order PAID / AWAITING_FULFILLMENT, retry fulfillment with backoff. Inventory stay reserved until ship or cancel policy fires.
• Fulfillment permanently impossible (warehouse fire, SKU discontinued): RefundPayment (idempotent refund key) + ReleaseInventory. Order → REFUNDED. Alert ops if refund itself is UNKNOWN.

Orchestration details

• Orchestrator pattern: Order service owns state machine and next-step worker. Simpler ops than choreography for money paths.
• Outbox: Every state transition and compensate command written in the same DB txn as the order row, then published.
• Idempotent compensates: ReleaseInventory / Refund keyed by order_id — safe under dual delivery.
• Deadlines: Reserve TTL (e.g. 15 min) auto-releases if saga stalls in PAYING — prevents permanent stock lock on abandoned checkout.

Payment SUCCESS, Inventory already released (race)?
  Bug or TTL fired early → cancel/refund path
  Monitoring: paid_without_reservation must stay 0
  Repair: query PSP + inventory ledger by order_id nightly

Walkthrough

Checkout for iPhone: reserve succeeds (hold #H991). PSP charge times out — order stays PAYMENT_UNKNOWN, hold kept. Webhook later: SUCCESS. Fulfillment API is 503 for 40 minutes — workers retry; customer sees "Order confirmed, preparing shipment." Fulfillment recovers → ship. Alternate ending: warehouse marks SKU dead → orchestrator issues Refund(order_id) + Release (no-op if already released) → customer notified.

Weak answer

"Use 2PC / distributed XA across Order, Inventory, Payment." Awkward with external PSPs; poor availability. "Refund immediately whenever fulfillment is slow." Creates chargeback noise and bad UX for transient outages.

Follow-up questions

Choreography instead of orchestrator?

Inventory emits Reserved, Payment listens, etc. Works at scale but debugging money bugs is harder — prefer orchestrator for checkout, choreography for non-money fan-out.

Payment succeeded and reserve never ran (wrong order)?

Treat as saga design bug. Auto-refund + alert. Process rule: never charge without prior reserve confirmation in this product.

How do you test compensations?

Chaos fixtures: kill inventory after payment mock SUCCESS; assert refund + release idempotency under replay. Contract tests on compensate APIs.

Multi-item cart — partial stock?

All-or-nothing reserve in one inventory txn, or per-line reserves with compensate-all on any miss — product policy. Don't leave half the cart reserved after pay failure.