A Kafka consumer keeps failing on one message (bad payload). It blocks the partition. How do you design retries, dead-letter queues, and recovery so poison messages do not stall the system?

How I frame it

Poison pill: one record throws forever. With a single-threaded partition consumer, offset cannot advance → lag grows → business stalls. Need bounded retries, isolation of bad records, and a human/automated repair path.

Handling strategy

• Classify errors: Transient (timeout, 503) → retry with backoff. Permanent (JSON schema fail, null FK) → do not infinite-loop.
• Bounded retries: e.g. 5 attempts with jitter, then publish to topic.DLQ (include original topic, partition, offset, payload, error, trace_id) and commit past the poison offset.
• Idempotent handler: Replays and DLQ redrives must not double-apply side effects.
• Separate DLQ consumer / ops tool: Fix data, redrive to main topic or repair DB, then mark resolved.
• Alert: DLQ depth and age; never silent drop.

Partition 3 stuck on offset 892341 (invalid schema)
  Retry ×5 → still fail
  Produce to orders.DLQ + commit offset 892342
  Lag resumes; bad msg awaits repair
  Redrive after producer fix → main topic

What not to do

• Swallow exception and commit without DLQ — silent data loss.
• Block forever waiting for a pageable human — product outage.
• Auto-skip without metrics — poison storms go unnoticed.

Weak answer

"Increase consumer concurrency." Same partition still processes ordered; poison still blocks that partition's order. "Delete the message from the topic." Not how Kafka works for consumers.

Follow-up questions

Ordering requirements after skip?

Skipping breaks total order for that key. If order matters for a key, park the key in a side state "blocked" until DLQ resolved, while allowing other keys — key-level isolation (more complex).

Schema evolution?

Schema registry + compat checks reduce poison rate; still keep DLQ for unexpected producer bugs.