A messaging service experiences a network partition. Would you delay messages or allow temporary inconsistencies? Why?
How I frame it
CAP is not one checkbox for the whole product. I ask what "messaging" means here — WhatsApp-style chat, Slack channels, email, or payment notification bus? The right answer is almost always path-by-path: delay where causal order / money safety matters; allow temporary inconsistency where perceived availability matters more.
My default for consumer chat
I would prefer availability with controlled inconsistency on the send path during a partition — user can still compose and see local "sent (pending sync)" — and repair on heal. Delaying every message until the remote region is reachable feels like an outage to users, which chat products rarely accept.
- Local accept: Persist in the reachable region / device outbox; show optimistic UI.
- Sync on reconnect: Ship with Lamport / vector clocks or server sequence per conversation; merge duplicates by
message_id. - Visible honesty: Single ticks vs double ticks, "waiting for network" — users tolerate delay of delivery confirmation, not total freeze of compose.
Region A ──X── Region B (network split) CP / delay delivery: Hold outbound until quorum / replica ack Receiver sees gap or "sending…" — never out-of-order causal break Availability of SEND drops for some clients AP / allow temporary inconsistency: Accept & deliver inside each side of the split Heal later: reconcile logs, maybe reorder or show "edited/late" Availability stays high — ordering / exactly-once is softer
When I would delay instead (choose C)
- Financial or compliance alerts where showing message A before B is wrong, or delivering twice is worse than waiting.
- Strongly ordered ops channels (e.g. trading desk, medical) with explicit SLO: never deliver out of causal order across replicas.
- Exactly-once business side effects triggered by the message (not the chat bubble itself) — queue behind quorum write.
Mechanism: reject or buffer produce until majority ack (Kafka acks=all + min ISR; Redis Raft; consensus log). Clients see temporary send failure / retry — better than silent split-brain delivery.
Healing after the partition
- Dedupe on stable
message_id(client-generated UUID). - Per-conversation total order via single leader or CRDT/OT if concurrent edits.
- If both sides accepted conflicting admin actions (e.g. both deleted vs renamed channel) — deterministic conflict policy + audit, not silent drop.
Walkthrough
Alice in us-east, Bob in eu-west; link between regions dead. Alice sends "On my way." AP design: message lands on east replicas, Alice sees single tick; Bob still offline relative to Alice. Link heals → replicate → Bob receives, double tick. CP design: Alice's client blocks or errors until eu quorum reachable — safer ordering across regions, worse UX for a social chat.
Weak answer
"Always consistency" or "always availability" with no product context. Interviewers want you to name the user harm of each choice and pick deliberately.
Follow-up questions
Group chat of 50 members across 3 regions?
Prefer per-conversation primary region or causal broadcast with buffering. Full global linearizability on every keystroke is too expensive — session causal consistency is the usual compromise.
Is "delay" the same as sacrificing availability?
Yes for the send API that waits on cross-partition quorum. You can still serve reads of already-local history — partial availability.
Message bus inside your own microservices?
Skew toward delay / lose produce on partition (don't dual-write conflicting domain events). Consumer apps can retry; silent divergent ledgers are worse.
How do you test this?
Inject partition in staging (Toxiproxy / chaos); assert no duplicate visible messages after heal, and that money-path topics refuse dirty publishes.