You receive webhooks for transaction status updates. How do you handle duplicate webhooks?

How I frame it in the interview

I'd clarify: webhooks are at-least-once — the gateway retries on timeouts, 5xx, or slow handlers. Duplicates are normal, not edge cases. The goal is exactly-once effect on our side: one status transition, one fulfillment, one ledger entry — while always returning a fast 2xx once we've safely recorded the outcome (including duplicates).

Core approach — two layers

Layer 1 — event dedupe: Every webhook carries a stable event_id (or delivery_id). Persist processed IDs in a table with a unique constraint. First sighting processes; replays short-circuit after lookup. TTL the table if volume is high (e.g. 7–30 days — longer than gateway retry window).

Layer 2 — business idempotency: Dedupe alone isn't enough. The same business outcome might arrive under two different event_ids (gateway bug, provider migration) or we might crash after updating payment but before recording event_id. So the handler must be safe when payment is already in the target state — compare current status, use a state machine, and make downstream side effects (ship order, credit wallet) conditional on an actual transition.

Payment gateway                Your service
     |                              |
     |-- POST /webhooks/payment ---->| 1) Verify signature + timestamp
     |   event_id=E1                 | 2) Dedupe: processed_events(E1)?
     |   txn_id=T99 status=SUCCESS   |    hit  -> return 200 (no side effect)
     |                               |    miss -> BEGIN TX
     |                               |      lock/load payment T99
     |                               |      if status already SUCCESS -> skip
     |                               |      else apply transition PENDING->SUCCESS
     |                               |      insert processed_events(E1)
     |                               |    COMMIT
     |<----------- 200 OK ----------|
     |                               |
     |-- duplicate E1 (retry) ----->| dedupe hit -> 200, zero business effect

Implementation details that matter

• Verify before mutate — HMAC signature + reject stale timestamps to block replay attacks; don't dedupe unauthenticated payloads.
• Atomicity — status update + processed_events insert in one DB transaction; or outbox pattern if side effects go async.
• Row lock — SELECT … FOR UPDATE on the payment row so two concurrent deliveries of the same event don't double-apply.
• Out-of-order events — model allowed transitions (PENDING → SUCCESS → REFUNDED). Ignore or quarantine backwards moves; don't blindly overwrite.
• Respond 200 on duplicate — gateways retry non-2xx; returning 200 on a dedupe hit stops the retry storm. Log metric webhook_duplicate_total for monitoring.

// Idempotent webhook handler (sketch)
@PostMapping("/webhooks/payment")
ResponseEntity<Void> onPaymentWebhook(@RequestBody WebhookEvent event) {
  verifySignature(event);  // reject forgeries before any DB work

  if (processedEventRepo.exists(event.getEventId())) {
    return ResponseEntity.ok().build();  // safe ack — gateway stops retrying
  }

  transactionTemplate.executeWithoutResult(status -> {
    Payment p = paymentRepo.findByIdForUpdate(event.getTxnId());

    if (p.getStatus() == PaymentStatus.SUCCESS) {
      processedEventRepo.insertIgnore(event.getEventId());
      return;  // business already applied — late/duplicate delivery
    }

    if (!p.canTransitionTo(event.getStatus())) {
      throw new InvalidTransitionException(...);  // out-of-order / illegal
    }

    p.apply(event);
    paymentRepo.save(p);
    processedEventRepo.insert(event.getEventId());  // unique on event_id
  });

  return ResponseEntity.ok().build();
}

Walkthrough — duplicate SUCCESS webhook

T99 is PENDING. Gateway sends event_id=E1, status SUCCESS. We transition T99, insert E1, return 200. Gateway never got the response — retries E1. Dedupe table has E1 → we return 200 immediately; T99 stays SUCCESS, no second shipment. If we had crashed after committing T99 but before inserting E1, the retry re-enters the handler; payment is already SUCCESS → no-op transition, then we record E1.

Weak answer to avoid

"We check Redis if we saw this webhook before." Without a unique DB constraint, transaction boundaries, and state-machine guards, you still double-charge under races or lose events on Redis eviction. Dedupe cache is an optimization on top of durable idempotency, not a substitute.

Follow-up questions

What if the duplicate has a different event_id but same txn_id and status?

Rely on layer 2 — business idempotency on txn_id + target state. Record both event IDs if needed for audit. Alert if duplicate rate spikes; may indicate provider issue.

Webhook says FAILED after we already marked SUCCESS — which wins?

Define policy upfront. Usually SUCCESS is terminal; FAILED after SUCCESS is a dispute/chargeback flow, not a silent overwrite. Route to manual review or a separate reversal workflow with its own idempotent events.

Handler is slow — gateway times out and floods duplicates. What do you do?

Return 202 fast after durable enqueue (DB/Kafka), process async with the same dedupe keys. Rate-limit per txn_id. Scale consumers; fix p99 before tuning retry logic.

Do you need distributed locks if you have a unique constraint on event_id?

Unique on event_id prevents double-insert of the same event. Concurrent different events for the same payment still need row-level lock or optimistic versioning on the payment row.

How do you reconcile if webhooks were missed entirely?

Scheduled polling of provider API by txn_id / time window. Reconciliation job is idempotent too — same state machine, source tagged reconciliation vs webhook.