How would you monitor this system in production?

How I frame it

I'd scope to the system we just designed — e.g. payments, orders, cache — and cover metrics, logs, traces plus business KPIs. Monitoring proves SLOs and catches incidents before users flood support.

Three pillars

• Metrics (RED + USE): Rate, Errors, Duration (p50/p95/p99) per API. For infra: Utilization, Saturation, Errors on DB, Redis, Kafka.
• Structured logs: JSON with trace_id, order_id, user_id, idempotency_key. Correlate across services. No PAN/passwords.
• Distributed tracing: OpenTelemetry from gateway → order → payment → notification. Find which hop added 2s latency.

Business metrics (what leadership cares about)

• Payment success rate, checkout conversion funnel drop-off.
• Inventory oversell count (must stay 0).
• Duplicate order / double-charge incidents.
• Outbox / Kafka consumer lag — emails and events delayed.

Alerting — signal, not noise

• SLO-based: Burn rate on 99.9% availability — page only when error budget at risk.
• Symptom-based: p99 checkout latency > 3× baseline for 5 min.
• Queue depth: Outbox backlog > 10k, payment UNKNOWN state count spike.
• Synthetic probes: Canary checkout every 60s from 2 regions.

Every alert links to a runbook: "if payment success drops, check gateway status, then UNKNOWN table, then recent deploy."

Dashboards

Per-service dashboard + one incident war-room board: traffic, errors, latency, DB connections, Redis hit rate, recent deploys overlaid.

Walkthrough — payment success drops

Dashboard shows payment success 98% → 91% over 10 min. Trace sample reveals PSP timeout spike on authorize hop. Alert fires on SLO burn. On-call checks gateway status page, queries payments WHERE status=UNKNOWN count, correlates with deploy at T-15min. Rollback + reconcile UNKNOWN rows via PSP query API. Synthetic canary would have caught authorize-only regression before full traffic.

Weak answer

"We log everything and grep when users complain." Too slow for payments. "Alert on any 5xx" — noise without SLO burn rate; on-call ignores pages.

Follow-up questions

How do you monitor idempotency health?

Metric idempotency_hit_rate — high is normal on retries. Spike in idempotency_conflict_409 signals client bugs (same key, different body).

Cardinality explosion from user_id labels?

Never high-cardinality labels on Prometheus. Aggregate by endpoint, status, region — drill into logs/traces for single user.

On-call gets alert fatigue?

Reduce pages to user-impacting SLOs; warnings to Slack. Post-incident review every false positive alert.