What happens if Redis goes down? How would you design the system to handle Redis failure?

How I frame it

Redis is rarely the financial ledger. Answer per use case — cache, lock, rate limit, session, hot counter each degrade differently. No single "Redis down = site down."

Degradation by Redis role

• Read cache: Fall through to DB. Add circuit breaker + request coalescing to protect DB from stampede. Higher latency, site stays up.
• Rate limiter: Fail open (allow traffic, abuse risk) vs fail closed (503) — product/legal call. Often fail open on browse, fail closed on login.
• Distributed lock: Pause job workers or fall back to DB SELECT FOR UPDATE SKIP LOCKED. Jobs must be idempotent.
• Hot inventory counter: Fall back to DB conditional UPDATE WHERE stock > 0. Lower TPS during outage; correctness preserved.
• Session store: Users re-login; sticky sessions on load balancer as bridge.

Redis cluster failover (Sentinel/Cluster)
  Brief blip: clients retry with backoff
  Extended outage:
    Cache  → DB + breaker
    Locks  → DB queue
    Limits → fail open OR edge WAF

Prevention

• Redis Sentinel or Cluster — automatic failover, replicas in another AZ.
• Timeouts on every Redis call (e.g. 50ms) — never block payment threads 30s.
• Health check removes Redis from critical path when cluster red.

Recovery

When Redis returns, warm hot keys from DB/async. Reconcile counters: compare Redis stock vs DB ledger — fix drift before next flash sale.

Walkthrough — 20 min Redis outage

Redis cluster fails at 10:00. Product pages fall through to DB with circuit breaker limiting to 2k QPS — slower but up. Checkout rate limit fails open; fraud team watches. Distributed locks on nightly batch jobs fall back to Postgres SKIP LOCKED queue — jobs continue one-at-a-time. At 10:20 Redis recovers; async warmer repopulates top 10k SKUs; counter reconciliation fixes 3 drift rows.

Weak answer

"We have Redis replica so it never goes down." Failover still causes blips; apps must handle connection errors gracefully.

Follow-up questions

Cache and DB both overloaded during Redis outage?

Pre-planned load shed: static product pages, queue checkout, disable non-essential APIs. Priority tier for paying users.

Split brain during Redis cluster partition?

Minority partition stops accepting writes; apps treat as unavailable. Prefer consistency over dual-writer split brain.