Design a distributed rate limiter for an API running across hundreds of servers. How would you ensure the rate limit is enforced consistently?

Rate LimitingRedisSystem DesignAPI Gateway

How I frame it

Hundreds of stateless servers each see a fraction of traffic — per-process counters are useless. Need centralized counter or coordinated view per client/key.

Centralized Redis (standard answer)

Key: rate:{api_key}:{window} — atomic INCR + EXPIRE via Lua script.
Sliding window log or token bucket for smooth burst control.
Enforce at API gateway before app servers — one check per request.
Return 429 + Retry-After header when over limit.

100 app servers → 1 Redis cluster
  Each request: EVAL incr_limit key 100 60
  Global count accurate within 1 RTT (~1ms)

Scale & failure

Redis Cluster shards by api_key hash.
Redis down: fail open (availability) vs fail closed (protect abuse) — document per API tier.
Local token bucket as approximate pre-filter — reduces Redis load 90% on edge POPs (advanced).

Walkthrough

API key ABC limited to 100 req/min globally. Request hits any of 200 pods → gateway evaluates INCR rate:ABC:window in Redis → count 87 → allow. Burst of 50 parallel from client → count hits 101 → 429 with Retry-After: 12. All pods see same count within one RTT — no per-server 100×200 over-limit.

Weak answer

"Each server tracks 100/min locally." 200 servers → 20,000/min effective limit. In-memory rate limit without shared store is wrong for distributed APIs.

Follow-up questions

Rate limit 1000/min per user globally vs per region?

Global needs single Redis or CRDT-style merge; per-region counter if policy allows regional quotas.

Fairness between free and paid tiers?

Different limits in key prefix rate:tier:pro:{user}; config from feature flags.