A very popular product page receives one million requests per minute. If the cache entry expires, all requests may hit the database simultaneously. How would you prevent this?

How I frame it

Classic cache stampede / thundering herd. One hot key expires → ~16k RPS on one DB row → connection pool exhaustion. Fix: ensure at most one rebuild per key expiry window.

Mitigations (layer them)

• Single-flight / request coalescing: First miss acquires lock SET lock:product:123 NX EX 5; only that request hits DB. Others wait 50ms or get stale value.
• Stale-while-revalidate: Return expired cache value immediately; one background thread refreshes. User sees 1s stale data, not 503.
• Probabilistic early expiration: Each key logically expires at T+jitter; spread rebuilds over time before hard TTL.
• Never hard-expire hot keys without prewarm: Cron refreshes top-100 products every 30s; TTL is backup not primary.
• CDN edge cache: Product JSON at edge — origin sees fraction of 1M RPM.

TTL expires on product:123 @ 12:00:00
  Req 1: miss → acquire lock → DB load → set cache
  Req 2..16k: lock held → wait OR serve stale-while-revalidate
  Result: 1 DB query, not 16,000

Walkthrough

Viral product at 1M RPM. Without protection: DB dies in seconds. With single-flight + CDN: edge serves 95%; origin sees occasional single rebuild per POP. DB stays flat.

Weak answer

"Increase cache TTL to 1 hour." Hot product stock/price goes stale; flash sale oversell risk. Use SWR + prewarm instead of naive long TTL.

Follow-up questions

Lock holder crashes mid-rebuild?

Lock TTL expires; next request rebuilds. Stale-while-revalidate prevents user errors during gap.

Duplicate of cache penetration attack?

Stampede is many requests for same valid hot key. Penetration is many invalid keys. Both need caching strategy; different fixes.