← Daily

Concept · No. 009 · October 24, 2026

A lease without a fencing token is only a polite suggestion

Distributed locks with expiry protect you until a process pauses longer than its lease and wakes up still believing it owns the resource. Fencing tokens move the safety check to the place where the damage would happen.

The idea

Many systems use a lease to make sure only one worker acts on a resource at a time. A worker acquires a lock with a time limit, does its work, and renews or releases it. If the worker dies, the lease expires and someone else can take over.

The gap is that a worker cannot reliably know it still holds the lease. A garbage collection pause, a suspended VM, a slow disk or a delayed network packet can stall a process past its expiry. When it resumes, its code continues from the line after the lock check. It writes as if nothing happened, while a second worker legitimately holds the same lease.

A fencing token closes that gap. Every time the lock service grants a lease, it also issues a number that only ever increases. The worker attaches that number to every write. The resource remembers the highest token it has accepted and rejects anything lower. Safety no longer depends on the worker's view of time. It depends on a comparison the resource makes at write time.

An example

Consider a scheduler that lets one worker at a time rebuild a customer's billing summary.

  • Worker A acquires the lease and receives token 33.
  • A begins a long computation and is paused by the runtime.
  • The lease expires. Worker B acquires it and receives token 34.
  • B writes the summary with token 34. The table stores 34 as the current fence.
  • A resumes, finishes its stale computation, and writes with token 33.
  • The database compares 33 to 34 and rejects the write.

Without the token, A's stale result silently overwrites B's correct one, and nothing in the logs looks wrong. With it, the stale write fails loudly and can be counted.

Where it breaks

Fencing only works if the resource enforces it. If the protected thing is a third-party API, an email provider or a payment processor that ignores your token, the check never happens. There you need idempotency keys or conditional operations that the provider supports.

The token issuer must be linearizable. A lock service that can lose state on failover and hand out a token it already issued breaks the guarantee. Consensus-backed stores such as etcd or ZooKeeper give you a usable monotonic value, for example a key's revision. A single cache node with asynchronous replication does not.

Tokens also check each resource independently. If a worker writes to two stores and is fenced on the second, the first write already landed. Fencing prevents stale writes, not partial ones.

And sometimes it is unnecessary. If the work is idempotent and duplicates are harmless, a plain lease for efficiency is enough. If the resource already supports compare-and-set on a version column, that version may be all the protection you need.

In practice

I treat the lease as an efficiency mechanism and the token as the correctness mechanism, and I design them that way from the start.

  • Store the fence next to the data, for example a fence column, and write with UPDATE ... SET ..., fence = :token WHERE id = :id AND fence < :token. Check the affected row count and treat zero as a rejection.
  • Pass the token through every layer that writes. A token that stops at the service boundary protects nothing.
  • Emit a metric for fenced writes. A steady trickle tells you pauses are longer than you assumed.
  • Test it by stopping a worker with SIGSTOP past its lease, then resuming it. If the stale write succeeds, the design is not done.
  • Keep leases comfortably longer than expected pauses, but never rely on that margin for correctness.
A lock tells a worker it may act, but only a check at the resource can stop a worker that has stopped being allowed to.