A downstream dependency becomes slow, causing clients to repeatedly retry requests. Traffic increases tenfold and the system begins to fail. How would you design the system to prevent cascading failures?

How I frame it

This is a retry storm feeding a cascade. Slow dependency + aggressive retries (clients, gateways, mesh sidecars) → amplified load → thread pools exhaust → upstream timeouts → more retries. I'd map the retry sources first — mobile app, API gateway, service mesh, cron reconcilers — each layer can multiply traffic.

What’s happening

Dependency latency rises from 50ms to 5s. Each caller with 3 retries and 10 parallel requests turns 1 logical call into up to 30. Ten services each doing this → 300× effective load on the sick dependency. The dependency never recovers.

Normal: 1k RPS → Payment svc
Slow + retries: 10k apparent RPS → Payment svc (still 1k real users)
Payment thread pool full → Checkout 503 → Checkout retries → more load

Defence layers

• Circuit breaker: After error/latency threshold, fail fast — stop calling sick dependency for cooldown (half-open probe after).
• Timeouts + bounded pools: Hard timeout (e.g. 2s) < client timeout. Never unbounded wait — frees threads.
• Retry policy: Exponential backoff + full jitter; cap at 2–3 retries; retry only idempotent ops (GET, idempotent POST).
• Bulkhead: Separate thread pools per dependency — one slow PSP cannot starve catalog service calls.
• Load shed / adaptive concurrency: Drop or queue requests when p99 breaches SLO — 503 with Retry-After beats hanging.
• Graceful degradation: Cached quote, "payment temporarily unavailable" — partial UX vs total outage.
• Rate limit at edge: Protect origin from retry amplification before it hits core.

Walkthrough

Payment PSP degrades. Checkout circuit opens after 50% errors in 30s window. New checkouts get fail-fast in 50ms with clear message — no thread blocked 30s. Mobile app backs off 1s, 2s, 4s with jitter instead of 5 parallel retries. Payment team fixes PSP; circuit half-open allows 10% probe traffic; success rate recovers → circuit closes. Total user impact: delayed checkout, not site-wide 503.

Weak answer

"Add more servers." Retries scale faster than capacity; you amplify the blast radius. "Disable retries entirely" without idempotency analysis loses legitimate recovery on transient blips.

Follow-up questions

Who should own retry policy — client or server?

Server enforces circuit breaker and timeout. Client retries only idempotent ops with backoff. Gateway may retry once on 502 to different pod — not on 503 from circuit open.

How do you test this before prod?

Chaos: inject latency/fault on dependency in staging; verify circuit opens, no thread pool exhaustion, recovery on half-open.

Circuit open — what do users see?

Explicit degraded mode with retry guidance — not infinite spinner. Queue order for async fulfillment if business allows.

Retry storm from one bad client?

Per-client rate limit at API key level; abusive client throttled without affecting others (bulkhead + fairness).