You have multiple application servers running scheduled jobs. How would you guarantee that a daily settlement job executes exactly once across the entire cluster?

How I frame it

N servers all fire cron at 00:00 — need at-most-one active runner per logical job per business day. Settlement must not double-run (double payouts) or never run.

Options

• DB claim row (simplest): INSERT INTO job_runs(job_name, run_date, status) VALUES ('settlement', '2026-07-11', 'RUNNING') with UNIQUE(job_name, run_date). Winner runs; losers get duplicate key and exit 0.
• Distributed lock: acquireLock('settlement-2026-07-11') TTL > job duration + heartbeat renew.
• Leader election: Only K8s leader pod runs cron — etcd lease.
• External scheduler: Cloud Scheduler triggers single HTTP endpoint / SQS once.

3 nodes @ midnight
  Node1: INSERT job_runs ✓ → runs settlement
  Node2/3: unique violation → skip (not error)

Failure handling

• Runner crashes mid-job: job must be idempotent or resumable from checkpoint.
• Stale RUNNING row: timeout detector resets or alerts; manual reconcile before retry same run_date.
• Use business date in key — not server local clock alone.

Weak answer

"Run cron only on server 1." Server 1 dies — job never runs. "Check last_run timestamp then run" — race without unique claim lets two nodes both pass check and double-settle.

Follow-up questions

Job takes 2 hours — lock TTL 30 min?

Heartbeat extends lock/lease every 5 min. TTL only for crash detection, not job length estimate.

Exactly-once vs at-least-once for settlement?

At-most-one runner + idempotent settlement steps = effectively-once. Steps keyed by (account, run_date).