Traffic suddenly increases from one thousand requests per second to one million. How would you design this system?

How I frame it

1k → 1M RPS is a 1000× jump. I clarify: is it mostly reads (product pages) or writes (checkout)? How long does the spike last (minutes vs hours)? Blindly "add servers" fails if the DB or a single hot key is the ceiling. Design for absorb at the edge, protect the core, scale the elastic tiers, and degrade gracefully when capacity is still short.

First cut — where the million requests go

• CDN / edge cache for anonymous catalog, images, static assets — often 70–95% of a retail spike never hits origin.
• API gateway auth + global and per-client rate limits — scrapers and misbehaving apps shed early.
• Stateless app pods with aggressive horizontal scale; keep pods CPU-bound not connection-bound.
• Separate read/write: Read replicas + caches for GET; writes go through tighter pools and often queues.

Edge / CDN  (cache static + public GETs)     ← kills most read amplification
    │
API Gateway (WAF, auth, per-key rate limit, load shed)
    │
Stateless app tier (HPA / pre-warmed ASG)   ← scale horizontally
    │
    ├── Read path: cache → read replicas / search
    └── Write path: queue (Kafka/SQS) for async work
              │
         Workers scale independently
              │
         DB: partition / shard hot tables; connection pool caps

Core capacity design

• Caching layers: Edge → Redis → DB. Stampede protection on hot keys (single-flight / SWR) — mandatory at this QPS.
• Async the slow path: Receipts, recommendations, analytics → Kafka. Synchronous path does auth, validate, persist minimal durable write, return 202/200.
• DB: Connection pooling (PgBouncer), prepared statements, partition hot tables, avoid cross-partition transactions on the spike path. Shard by user/tenant if one primary cannot take write QPS.
• Backpressure: Load shed when p99 or queue depth breaches — return 503 with Retry-After for non-critical APIs; protect checkout/pay with reserved capacity (bulkhead).
• Auto-scale with lag: HPA on CPU is too late alone — scale on RPS, concurrent requests, and consumer lag. Pre-warm before known events (sale start).

Sudden vs planned spike

• Planned (flash sale): Pre-scale, warm caches, read-only mode for non-essential features, inventory reserve path already load-tested.
• Unplanned (viral / attack): WAF + bot score, emergency rate limits, feature flags to disable heavy modules (search facets, personalization), static fallback pages.

Priority under 1M RPS
  P0: auth, pay, reserve inventory, health
  P1: browse cached product
  P2: search, recommendations, notifications  ← shed first

Walkthrough

Celebrity posts your product link. RPS 1k → 800k in 3 minutes, mostly GET product + images. CDN absorbs images and HTML shells. Origin Redis + single-flight rebuild product JSON. Checkout stays ~5k RPS with dedicated pool. Search start failing SLOs → flag disables autocomplete. Payment and reserve unaffected. Over 20 minutes HPA adds app pods; no DB meltdown because writes never saw the full million.

Weak answer

"Kubernetes will auto-scale." Without edge caching, DB pooling, and shed priorities, you scale app pods into a dead database. "One giant Vertically scaled server." Cannot buy 1000× with vertical scale alone.

Follow-up questions

What if the spike is 1M writes/sec?

Rare for user-facing apps — ingest via partitioned Kafka, batch DB writes, CQRS read models. Accept eventual visibility on non-critical data. Money writes still rate-limited and sharded carefully.

How do you know you are ready for 1M?

Load test the critical path to 2–3× expected peak; chaos on dependency latency; capacity model: RPS × cost per request vs pool sizes.

Multi-region?

Geo-DNS / Anycast to nearest edge; regional caches; pin writes to primary region or use careful multi-master only where conflict-free. Cross-region RTT kills naive global writes.

Cost control during spike?

Autoscale caps + load shed beats unlimited bill. Cache hit rate is the cheapest capacity.