Several instances of your service compete to process the same resource, but only one should process it at a time. How would you design a distributed lock? What failure scenarios would you consider?

Distributed LocksRedisConcurrencySystem Design

How I frame it

Multiple workers, one mutable resource (settlement batch, inventory file, cron slice). I ask: do workers share a database row already? If yes, prefer SELECT FOR UPDATE SKIP LOCKED or unique job claim — simpler than Redis lock. Distributed lock when coordination spans services without a natural DB row.

Redis lock pattern (common answer)

SET resource:lock <token> NX EX <ttl> — acquire only if not held.
Unique token per holder — release via Lua: delete only if value matches (never delete another owner's lock).
TTL mandatory — worker crash cannot deadlock cluster; TTL > p99 job time.
Heartbeat extend: Long jobs renew EXPIRE every N seconds; TTL is crash detector, not job length cap.

Worker A: SET lock:batch-99 token-A NX EX 30 → OK → process
Worker B: SET lock:batch-99 token-B NX EX 30 → FAIL → skip or poll
A crashes @ 10s → lock expires @ 30s → B acquires safely

Failure scenarios

Process pause > TTL (GC, STW): Lock expires; second worker enters. Mitigation: fencing token (monotonic) passed to storage — stale worker's write rejected.
Clock skew: Use Redis TTL, not app wall clock, for expiry.
Redis primary failover: Lock may be lost during failover — Redlock (multi-node quorum) or accept risk + idempotent job design.
Split brain: Two workers think they hold lock — business logic must be idempotent; lock is optimization, not sole correctness guarantee.
Forgot to release: TTL saves you; still release in finally when job completes cleanly.

Alternatives (often better)

PostgreSQL pg_advisory_lock or FOR UPDATE SKIP LOCKED on job queue table.
ZooKeeper / etcd ephemeral sequential nodes — lock tied to session.
DB unique constraint: INSERT INTO job_claims(resource_id, worker) — winner processes; same pattern as distributed cron.

Weak answer

"SETNX without TTL." Crashed worker deadlocks resource forever. "Check-then-set" without atomicity — race between GET and SET allows double acquire.

Follow-up questions

Redlock — is it worth the complexity?

Martin Kleppmann critique: failover can violate safety. For many apps, single Redis + fencing token + idempotent work is enough. Financial settlement may need DB-based claim instead.

Lock vs message queue consumer group?

Queue partition assigns one consumer per message — no explicit lock. Use lock when multiple workers poll same resource list (e.g. scan S3 prefix).

How long should TTL be?

Short enough to recover from crash quickly (30–60s); extend via heartbeat for long jobs. Not "job max duration + 1 day."