One seller gets 80% of traffic during a livestream. Their product and inventory keys thrash Redis and the database. How do you detect and handle hot partitions without breaking fair checkout?
How I frame it
Hot key / hot partition: one seller_id or sku receives disproportionate load. Sharding by hash does not help if everyone hashes to the same key. Separate browse fan-out from checkout correctness.
Detection
- Redis
HOTKEYS/ proxy metrics: QPS and CPU per key. - App metrics: top-N
product_idby request rate; alert when one key > X% of shard traffic. - DB: wait events / lock waits concentrated on one inventory row.
Mitigations
- Local + edge cache for reads: Product page at CDN / POP with short TTL + SWR. Origin sees a fraction of livestream viewers.
- Key splitting for counters (approximate): View counts / "watching now" across
sku:views:{0..N}shards, sum on read. Not for stock. - Inventory: Do not put authoritative stock on a single Redis string without a plan. Use atomic DECR with backoff; for extreme fireworks sales, pre-shard inventory into N buckets (100 units × 10 buckets) and assign checkout to a random non-empty bucket — only if product policy allows.
- Request coalescing / single-flight on cache miss for that SKU.
- Dedicated isolation: Pin celebrity catalog to dedicated Redis node / DB partition / even a dedicated checkout pool so other sellers stay healthy.
- Fair checkout: Queue or lottery at "Buy" — protect reserve path with rate limits per user; never let browse stampede erase stock CAS fairness.
Livestream: 500k viewers → 1 SKU CDN serves PDP HTML/JSON Buy click → rate limit + reserve() on inventory Other sellers: untouched shard / pool
Weak answer
"Add more Redis replicas." Writes and hot GETs on one key still concentrate. Replicas help read scale only if you accept slightly stale reads and distribute them — still need coalescing and edge cache.
Follow-up questions
Can you shard a single stock counter correctly?
Only with careful merge (buckets or CRDT PN-counters) and a final authoritative reconcile. Oversell risk — prefer DB conditional update or reserved inventory rows for money path.
Seller dashboard also hot?
Async analytics via stream; dashboard reads aggregated views, not every heartbeat on OLTP key.