🔒Distributed Locks

Coordinate critical work across nodes when local locks are insufficient.

📌 Key IdeaDistributed locks prevent multiple nodes from running the same critical section simultaneously. They require unique lock tokens, TTL expiry, conditional unlock, and often fencing tokens to stay safe under pauses, crashes, and clock skew.

🧩

Problem it solves

Local JVM locks (synchronized, ReentrantLock) only coordinate threads inside one process — they cannot stop duplicate execution across multiple app instances.
Without cross-node coordination, scheduled jobs, inventory reservation, or payment finalization can run twice and corrupt shared state.
Distributed locks provide a single-owner guarantee for a critical section spanning multiple nodes — but only when implemented with expiry, ownership checks, and fencing.

📖

Definition

A distributed lock is a coordination primitive where exactly one process (across many JVMs/containers) holds exclusive access to a named resource for a bounded time. Common implementations use Redis (SET NX PX), ZooKeeper ephemeral nodes, or database rows with unique constraints.

✨

Why It Matters

Prevents duplicate side effects in horizontally scaled deployments.
Protects scarce shared resources (inventory slots, promo codes, file writes).
Enables single-execution cron/scheduler tasks across a cluster.
Must be paired with fencing or idempotency — a lock alone does not guarantee end-to-end correctness under all failure modes.

✅

When to Use

Single-execution cron/jobs across multiple app instances (only one node runs the batch).
Protecting scarce shared resources during allocation (inventory, seat booking).
Short, bounded critical sections where lock TTL can be estimated reliably.
Leader-election-lite when full consensus (Raft/ZK) is overkill.

🚫

When NOT to Use

When a DB unique constraint + transaction or idempotency key can enforce correctness more simply.
Long unpredictable critical sections without heartbeat/renewal — TTL will expire mid-work.
High-contention hot keys where lock thrashing degrades throughput (prefer partitioning or queue serialization).
As a substitute for proper transactional design in the primary database.

💻

Code Variations

1️⃣ Redis lock — acquire with tokenjava

String lockKey = "lock:order:" + orderId;
String token = UUID.randomUUID().toString();
boolean acquired = redis.set(lockKey, token,
    SetArgs.setArgs().nx().px(30_000)) != null;
if (!acquired) return; // another node owns it

NX = set only if not exists. PX = TTL in milliseconds.

2️⃣ Safe unlock — compare token before deletejava

String unlockLua =
    "if redis.call('get', KEYS[1]) == ARGV[1] then " +
    "  return redis.call('del', KEYS[1]) else return 0 end";

try {
  processOrder(orderId);
} finally {
  redis.eval(unlockLua, 1, lockKey, token);
}

Lua ensures atomic compare-and-delete. Without this, you can release someone else's lock.

3️⃣ Fencing token on downstream writejava

long fenceToken = lockService.acquire("resource-42");
try {
  storage.write("resource-42", data, fenceToken);
  // storage rejects if fenceToken <= lastAppliedToken
} finally {
  lockService.release("resource-42", fenceToken);
}

Fencing protects against stale owner writing after TTL expiry.

4️⃣ Distributed lock safety checklisttext

- Unique random token per acquisition
- TTL always set (never infinite lock)
- Conditional unlock (token match)
- Fencing token for downstream writes
- Heartbeat/renewal if work > TTL
- Monitor lock wait time + hold duration
- Prefer DB constraint + idempotency when possible

📊

Distributed lock implementations

Aspect	Redis (SET NX PX)	ZooKeeper / etcd	DB row lock (SELECT FOR UPDATE)
Latency	Very low	Medium	Higher (DB round-trip)
Correctness model	TTL-based; needs fencing for stale owners	Ephemeral nodes — strong session semantics	Transactional — tied to DB isolation level
Ops complexity	Low (if Redis already in stack)	Higher (ensemble to operate)	Low (reuse existing DB)
Best fit	Fast coordination, short locks	Strong coordination, leader election	Already in DB transaction boundary
Failure risk	Stale owner after GC pause	Session expiry on network partition	Deadlock / long-held row locks

🔀

Common Confusions Cleared Up

Local lock vs distributed lock

Local locks coordinate threads inside one JVM. Distributed locks coordinate multiple JVMs/nodes/processes. Never assume synchronized protects you in a multi-instance deployment.

TTL lock vs fencing token

TTL prevents a crashed owner from holding the lock forever. Fencing token (monotonic counter passed to downstream writes) prevents a stale owner — whose lock expired — from writing after a new owner acquired the lock. Martin Kleppmann's analysis shows TTL-only locks are unsafe without fencing.

Lock vs idempotency vs unique constraint

Often the simplest correct design is: DB unique constraint on (resource_id) + idempotent handler, not a distributed lock. Locks add latency and failure modes; use them when coordination cannot be expressed as a constraint.

🧪

Variations / Implementations

1) Redis lock with token + Lua unlock

Acquire with SET key token NX PX ttl. Release with Lua script: delete only if stored value equals token. Never DEL key blindly — you may delete another owner's lock.

2) Lock renewal (watchdog)

For work that may exceed TTL, a background thread renews the lock periodically while the owner is alive. Stop renewal on completion or crash — if renewal stops, lock expires safely.

3) Redlock (multi-master Redis)

Acquire majority of N independent Redis nodes. Controversial for correctness under clock skew and GC pauses — understand trade-offs before adopting in finance-grade flows.

4) Fencing token on downstream writes

Lock service returns monotonically increasing token. Every write to storage (DB, blob store) includes token; storage rejects writes with token ≤ last seen. Protects against stale lock holder.

⚠️

Common Mistakes / Anti-Patterns

Deleting lock without verifying ownership token.
Infinite TTL locks — crashed owner blocks everyone forever.
No fencing token when lock protects writes to shared storage.
Using distributed lock where a DB unique constraint suffices.
Long GC pauses causing TTL expiry while owner still believes it holds lock.
Single Redis instance treated as fault-tolerant lock store without understanding split-brain.

🎤

Interview Q&A

🔮

Likely Follow-up Questions

How do you implement lock renewal without extending a dead owner's lease?
What happens to your cron job if the lock holder dies mid-execution?
How would you design inventory reservation — lock, optimistic concurrency, or queue?

🧾

Quick Cheat Sheet

Must-haveUnique lock token + conditional unlock (Lua).

TTLAlways bounded; renew if work exceeds TTL.

SafetyFencing token on downstream writes.

AlternativeDB unique constraint + idempotency often simpler.

Redis cmdSET key token NX PX ttl

🔗