After an order is placed, the customer should receive exactly one confirmation email. The notification service may crash after sending the email but before updating its database. How would you solve this?

How I frame it

SMTP is at-least-once; crashes create send-or-record ambiguity. Business goal: customer sees one confirmation per order — not zero, not two.

Outbox + idempotent delivery

• Transactional outbox: Order service inserts order + outbox(order_id, EMAIL_CONFIRM, PENDING) in same DB txn.
• Worker claims job: UPDATE outbox SET status=PROCESSING WHERE id=? AND status=PENDING — one worker owns row.
• Send then mark SENT: Call email provider with idempotency key order_id+template. Then status=SENT.
• Crash after send, before SENT: Retry job — provider idempotency returns same Message-ID; no second email. Or query provider API by key.
• Dedupe constraint: UNIQUE(order_id, notification_type) where status=SENT — belt and suspenders.

Order TX: order + outbox PENDING
Worker: PROCESSING → send(SendGrid, key=order-99-confirm) → SENT
Crash after send: retry → SendGrid idempotent → SENT (no duplicate)

Walkthrough — crash after send

Order #99 placed. Outbox row PENDING. Worker claims PROCESSING, calls SendGrid with key order-99-confirm, email delivered. JVM OOM before status=SENT update. Another worker picks up stale PROCESSING row (lease timeout), retries SendGrid — provider returns same Message-ID (idempotent). Worker marks SENT. Customer received exactly one email.

Trade-off

Missing email worse than rare duplicate — reconciliation job resends if not SENT within 5 min. Monitor outbox_pending_age.

Weak answer

"Check if email already sent before sending." Race between two workers without DB claim or unique constraint still double-sends. "Fire-and-forget async email in order handler" couples order commit to SMTP latency and loses retry on crash.

Follow-up questions

Email provider has no idempotency API?

Store provider Message-ID after first send; on retry check logs or custom header X-Entity-Ref-ID: order-99 before resending.

vs full notification platform?

Same outbox pattern extended to SMS/push with channel column and per-channel dedupe — see notification platform question.