🧯Fault Tolerance (Recover from Failures Gracefully)

Design so partial failures degrade service, not crash the system.

📌 Key IdeaFault tolerance is the ability to continue delivering acceptable service when components fail. Combine timeouts, retries with jitter, circuit breakers, bulkheads, idempotency, and fallback paths to contain blast radius.

📖

Definition

In distributed systems, failures are normal (timeouts, dependency outages, node crashes, network partitions). A fault-tolerant design detects failure quickly, isolates impact, and recovers safely.

✨

Why It Matters

Prevents one dependency outage from cascading into full platform failure.
Protects latency SLOs by failing fast and shedding non-critical work.
Improves user trust: partial functionality beats total downtime.

💻

Code Variations

1️⃣ Timeout + bounded retry + jitter (pseudo)java

for (int attempt = 1; attempt <= 3; attempt++) {
  try {
    return client.callWithTimeout(req, Duration.ofMillis(300));
  } catch (TransientException e) {
    if (attempt == 3) throw e;
    long backoffMs = ThreadLocalRandom.current().nextLong(50, 150) * attempt;
    Thread.sleep(backoffMs); // jittered backoff
  }
}

Retry only idempotent operations or operations protected with idempotency key.

2️⃣ Circuit breaker + fallbackjava

Supplier<Response> primary = () -> paymentClient.charge(req);
Supplier<Response> fallback = () -> Response.accepted("queued_for_retry");

return breaker.execute(primary, fallback);

Fallback should be explicit product decision, not silent data corruption.

3️⃣ Failure budget aware policytext

if (errorRate(depA, 5m) > threshold) {
  openCircuit(depA);
  routeToFallback();
  alertOnCall();
}

Use SLO/error-budget signals to drive automatic protection.

📊

Failure Modes and Recovery Tactics

Aspect	Failure Symptom	Primary Control
Slow downstream dependency	Timeout + fallback	Bound wait time and preserve responsiveness
Transient errors (5xx/network blips)	Retry with backoff + jitter	Recover without causing retry storms
Dependency hard-down	Circuit breaker	Fail fast, avoid exhausting threads
Resource saturation in one path	Bulkhead isolation	Contain blast radius to one workload
Duplicate request execution	Idempotency keys	Safe retries without double effects

🧪

Variations / Implementations

Tiered degradation strategy

Classify features as critical/non-critical. Keep checkout and auth alive; degrade recommendations, notifications, and analytics first.

Synchronous API + asynchronous recovery

Return accepted/partial response quickly, then reconcile via outbox/event processing for eventually consistent completion.

Active health + passive protection

Combine readiness/liveness checks with runtime resilience controls (timeouts/retries/breakers) for faster detection and containment.

⚠️

Common Mistakes / Anti-Patterns

Unlimited retries without backoff/jitter (amplifies outage).
Global thread pool for all dependencies (one slow service blocks all).
No timeout defaults on outbound calls.
Fallback returns stale/incorrect data without marking degraded mode.
No runbook for breaker-open incidents.

🎤

Interview Q&A

🔮

Likely Follow-up Questions

How do you test fault tolerance (chaos experiments, failure injection)?
How do you avoid retry storms across service layers?
Which metrics indicate graceful degradation is working?
How do you design user-facing messaging for degraded mode?

🧾

Quick Cheat Sheet

Detect fastShort timeouts + health signals.

ContainBulkhead + circuit breaker.

Recover safelyRetry with jitter + idempotency.

CommunicateExplicit degraded-mode behavior and alerting.

🔗