Kafka provides at-least-once delivery, so consumers may receive duplicate messages. How would you design a consumer that processes each business event exactly once?

KafkaIdempotencyEvent-DrivenDistributed Systems

How I frame it

Kafka consumers typically commit offset after processing → crash before commit = redelivery. True EOS on the wire is hard; interview answer: idempotent consumer — duplicate messages, one business effect.

Idempotent consumer pattern

Business key: event_id from producer (or topic-partition-offset if absent).
Processed table: processed_events(event_id PK, processed_at)
Atomic apply: Same DB txn: INSERT processed + business update (e.g. order PAID).
Duplicate delivery: INSERT fails unique → skip side effect, still commit Kafka offset (or commit after successful skip).

Event E1: payment captured
  BEGIN → INSERT processed(E1) → UPDATE order PAID → COMMIT → offset commit
Redelivery E1: INSERT fails → skip → offset commit

Kafka transactions (senior+)

read_process_write with transactional.id — consume, update DB, produce output in one Kafka transaction. Heavier ops complexity; idempotent consumer often enough.

Walkthrough

Consumer processes PaymentCaptured E1: txn inserts processed_events + marks order PAID, commits offset. Consumer crashes before offset commit — Kafka redelivers E1. Second attempt: INSERT processed_events fails unique → skip business update → commit offset. Order stays PAID once; no double shipment trigger.

Weak answer

"Commit offset before process." You lose messages on crash. "Manual commit once" without dedupe still double-applies on redelivery.

Follow-up questions

Consumer lag during replay?

Bulk replay still hits processed table — duplicates cheap. Scale consumers; partition count bounds parallelism.

Side effect calls external API (ship order)?

Outbox pattern: DB txn records intent; separate worker calls ship API with idempotency key = event_id.