Synchronizing access to shared resources across independent server nodes is a classic challenge in distributed architectures. Relying on a naive SET resource_id client_id NX PX 30000 command on a single Redis instance introduces catastrophic race conditions under network partitions, GC pauses, and master-replica failovers.

1. Why Single-Node Redis Locks Fail Under Failure

Redis master-replica replication is fundamentally asynchronous. Consider this standard failure scenario:

  1. Client 1 acquires a lock on the Redis Master node.
  2. The Master node crashes before the write replication packet reaches the Replica node.
  3. Sentinel or Cluster promotes the Replica to the new Master.
  4. Client 2 requests the same lock. The new Master grants it because the key is missing.
  5. Catastrophe: Both Client 1 and Client 2 believe they hold the exclusive lock simultaneously, leading to silent database state corruption.

2. Martin Kleppmann's Critique & Fencing Tokens

Even with consensus-backed lock managers, a client can experience an unexpected pause (e.g. Stop-The-World GC or CPU throttling) immediately after acquiring a lock. By the time the client resumes, its lock lease has expired and been acquired by another node.

The Fencing Token Solution

Every granted lock must return a strictly monotonically increasing fencing number. When writing to the storage tier, the storage engine rejects any update possessing a token lower than the highest token observed so far.

etcd_distributed_lock.go 
package main

import (
	"context"
	"fmt"
	"log"
	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

func RunWithEtcdLock() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints: []string{"localhost:2379"},
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	session, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	mutex := concurrency.NewMutex(session, "/locks/order-settlement/1001")
	ctx := context.Background()

	if err := mutex.Lock(ctx); err != nil {
		log.Fatalf("Failed to acquire lock: %v", err)
	}
	fmt.Println("Consensus-backed lock acquired safely via Raft!")

	// Execute critical section...

	if err := mutex.Unlock(ctx); err != nil {
		log.Fatal(err)
	}
}
Advertisement / Sponsored