Distributed Lock Manager

This project designs a fault-tolerant and highly available append-only log that can be accessed by a distributed set of clients. This distributed lock manager ensures mutual exclusion, preventing concurrent access that could lead to inconsistencies or race conditions. It allows processes running on different nodes to synchronise their actions when accessing shared files, ensuring that only one process can hold a lock on a resource at any given time. It allows processes to safely acquire and release locks on shared resources. The lock manager is implemented using grpc and Python.

Structure

A client library is implemented with an attached interceptor. Every Remote Procedure Call (RPC) function call passes through the interceptor, whose purpose is to handle requests, retries and timeouts for all RPC calls. A bounded consistency model is implemented, which utilises the lock_release method as the trigger for duplication of server data from the leader server to the replica servers. The server library handles client requests. Message receipt is asynchronous and messages can take a long time to be delivered, can be duplicated and lost in the network. It is assumed that messages are never corrupted, and Byzantine faults are not considered.

Design flow

The below diagram shows the flow of RPC calls in the system in the basic case, i.e. assuming no faults in the network or system components. Whenever the client wants to append to a shared file, it must acquire the lock for that file first. The system ensures atomicity through the use of a critical section cache. Critical section tasks are appended to the critical section cache and only commited once the client calls lock_release.

1. Clients initialise with the server.
2. lock_acquire RPC call is sent from the client and passes through the interceptor/middleware to the server.
3. The server receives the request:
   1. If the lock is free, the server will send a SUCCESS message back to the client along with the id number for the lock.
   2. If the lock is currently held by someone else, the interceptor will retry the RPC request up to a maximum number of attempts. The client is blocked and will wait for the lock.
4. When a client acquires the lock, it can modify the shared file by sending a file_append RPC call , which again is passed through the interceptor on its way to the server.
5. The server will then add the append request to the critical section cache, and will reply to the client with a status message WAITING_FOR_LOCK_RELEASE
6. Client sends lock_release or times out:
   1. If client performs successful lock_release: the server will release the lock and perform all the operations in the critical section cache, send requests to duplicate that data on replica servers and returns SUCCESS message back to the client.
   2. If the lock times out: Server will wipe the critical section cache and send only the new lock counter to the replicas. The critical section cache will not be appended, as the critical section was not committed with an explicit lock_release RPC call.
7. The server can now grant the lock to the next waiting client.
8. The new client can modify the file.

Fault Tolerance

Single Server Failure

The system uses a persistent log. This log is updated with every committed critical section to ensure atomicity. The server can be fully rebuilt from the log file and will reallocate lock ownership to the appropriate client.

Network Failure

The client may retry requests up to a maximum number of attampts. Each request from the client has a request ID. The server can recognise which requests are duplicate requests, so will crucially not execute the same append requests twice, ensuring idempotency.

Distributed Server Failure

When a server in a distributed cluster fails, RAFT leader election takes place to elect a new leader. This ensures fault-tolerance, the system can remain available as long as n/2 servers are available, with n being the number of servers in the cluster. Servers that have failed are rebuilt by receiving the full persistent log from the current leader.