Kafka Basics
Partitioned log-based messaging for high-throughput event pipelines.
📌 Key IdeaKafka stores ordered, durable event logs split into partitions. Producers append; consumer groups pull and track offsets. It excels at high throughput, replay, and multiple independent consumers — but ordering is per-partition only, and end-to-end exactly-once requires careful configuration plus idempotent sinks.
Problem it solves
- Services need to decouple without losing event history — traditional queues delete messages after consumption.
- Multiple downstream systems (analytics, search, billing) must process the same event stream independently.
- Systems need to replay events after bugs, new features, or disaster recovery.
- High-throughput ingestion (clickstreams, logs, order events) overwhelms single-queue brokers.
Definition
Apache Kafka is a distributed commit log. Producers write records to topics split into partitions (ordered, append-only sequences). Brokers store replicas; consumers in a consumer group each read assigned partitions and commit offsets (read positions). Retention is time/size-based — messages persist even after consumption.
Why It Matters
- Decouples producers and consumers with durable buffering — producers never block on slow consumers.
- Replayability: reset offset and reprocess history for new pipelines or bug fixes.
- Horizontal scale: add partitions and consumers to increase throughput.
- Foundation for event-driven architecture, CQRS read models, and stream processing (Kafka Streams, Flink).
Visual
📨 Kafka core model
Producer -> Topic (partitions) -> Consumer Group Broker: stores append-only log per partition Offset: position in partition log (consumer bookmark) Consumer group: each partition -> one consumer in group Replication: leader + follower replicas per partition Ordering: guaranteed WITHIN a partition (via message key) Scaling: more partitions -> more parallel consumers
When to Use
- Event-driven pipelines and async service decoupling at scale.
- High-throughput stream ingestion with replay needs (audit, reprocessing).
- Multiple consumer groups reading the same events independently (ops + analytics + search).
- Log compaction for changelog topics (Kafka as lightweight KV store for config/state).
When NOT to Use
- Simple task queues with low volume — SQS/RabbitMQ ops overhead is lower.
- Strict global ordering across all events (Kafka orders per partition only).
- Request-response patterns needing immediate reply.
- Tiny teams without capacity to operate a Kafka cluster (or managed Kafka cost).
Code Variations
1️⃣ Producer with key for partition orderingjava
ProducerRecord<String, String> rec =
new ProducerRecord<>("orders", orderId, payload);
producer.send(rec, (meta, ex) -> {
if (ex != null) log.error("send failed", ex);
});Same key → same partition → preserved relative order for that key.
2️⃣ Consumer with manual offset commitjava
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
process(rec.value());
}
consumer.commitSync(); // commit after batch processedCommit after successful processing. Commit before process = may lose messages on crash.
3️⃣ Idempotent producer configjava
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);Idempotent producer prevents duplicate writes on retry within a session.
4️⃣ Kafka production checklisttext
- Partition key = entity that needs ordering
- acks=all + min.insync.replicas >= 2
- Monitor consumer lag per partition
- DLQ/retry topic for poison messages
- Idempotent consumer logic (at-least-once default)
- Size partitions for target throughput + consumer countKafka core concepts
| Aspect | Concept | What it does | Interview gotcha |
|---|---|---|---|
| Topic | Named stream of records | Logical grouping — physical storage is in partitions | |
| Partition | Ordered append-only log shard | Parallelism unit — more partitions = more consumers | |
| Offset | Monotonic position in partition | Consumers commit offset after processing — enables replay | |
| Consumer group | Cooperative consumers sharing work | One consumer per partition per group; extra consumers idle | |
| Replication factor | Copies of partition on multiple brokers | RF=3 typical; ISR (in-sync replicas) affects durability | |
| Retention | Time (retention.ms) or size based | Messages kept after consumption — unlike traditional queues |
Key Kafka trade-offs
| Area | Strength | Caveat | Mitigation |
|---|---|---|---|
| High throughput | Batching + sequential disk I/O | Consumer lag builds under slow consumers | Scale consumers/partitions, lag alerts, backpressure |
| Replayability | Reset offset and reprocess | Duplicate processing on replay | Idempotent consumers, dedupe keys, transactional sinks |
| Partition ordering | Strong order within partition | No global order across partitions | Choose partition key by ordering domain (e.g. orderId) |
| Durability | Replication + acks=all | Producer acks=1 or 0 can lose data | Set acks=all, min.insync.replicas=2 |
| Ops complexity | Mature ecosystem, managed options (MSK, Confluent) | Broker tuning, rebalancing, partition planning | Start with managed Kafka; monitor lag and ISR |
Common Confusions Cleared Up
Log vs queue mindset
Traditional queues delete messages after ack. Kafka retains them — consumers track offset. Multiple consumer groups each maintain their own offset on the same topic.
Partition key drives ordering
Records with the same key go to the same partition → relative order preserved for that key. Wrong key choice (e.g. random UUID for order line items) breaks ordering guarantees.
Consumer group rebalancing
When consumers join/leave, partitions are reassigned (rebalance). During rebalance, processing pauses — frequent rebalances (slow consumers, short session timeout) cause instability.
Common Mistakes / Anti-Patterns
- More consumers than partitions — extra consumers sit idle.
- Random partition keys when per-entity ordering is required.
- Committing offset before processing completes.
- Single partition for high-throughput topic — bottleneck.
- Treating Kafka like a task queue (delete-on-consume mental model).
- Ignoring consumer lag until pipelines fall hours behind.
Interview Q&A
Likely Follow-up Questions
- How do you choose partition count for a new topic?
- What is log compaction and when would you use it?
- How do retry topics and DLQ patterns work in Kafka?
Quick Cheat Sheet
OrderingPer partition, not global — key matters.
Scale knobPartition count (+ matching consumers).
ReplayReset consumer group offset.
Durabilityacks=all, replication factor ≥ 3.
SafetyIdempotent producer + idempotent consumer.
vs queueKafka = log + replay; queue = task dispatch.