Paper: Chubby
Chubby / Distributed Locking Service Goal Design a highly available and consistent service that can store small objects and provide a locking mechanism on those objects. What is Chubby? Chubby is a service that provides a distributed locking mechanism and also stores small files. Internally, it is implemented as a key/value store that also provides a locking mechanism on each object stored in it. Extensively used in various systems inside Google to provide storage and coordination services for systems like GFS and BigTable. Apache ZooKeeper is the open-source alternative to Chubby. Chubby is a centralized service offering developer-friendly interfaces (to acquire/release locks and create/read/delete small files). It does all this with just a few extra lines of code to any existing application without a lot of modification to application logic. At a high level, Chubby provides a framework for distributed consensus. Chubby Use Cases Primarily Chubby was developed to provide a reliable locking service. Other use cases evolved like: Leader Election Any lock service can be seen as a consensus service, as it converts the problem of reaching consensus to handing out locks. A set of distributed applications compete to acquire a lock, and whoever gets the lock first gets the resource. Similarly, an application can have multiple replicas running and wants one of them to be chosen as the leader. Chubby can be used for leader election among a set of replicas. Naming Service(Like DNS) It is hard to make faster updates to DNS due to its time-based caching nature, which means there is generally a potential delay before the latest DNS mapping is effective. Storage(Small Objects that rarely change) Chubby provides a Unix-style interface to reliably store small files that do not change frequently (complementing the service offered by GFS). Applications can then use these files for any usage like DNS, configs, etc. Distributed Locking Mechanism Chubby provides a developer-friendly interface for coarse-grained distributed locks (as opposed to fine-grained locks) to synchronize distributed activities in a distributed environment. Application needs a few lines, and chubby can take care of all lock management so that devs can focus on business logic, and not solve distributed Locking problems in a Distributed system’s setting. We can say that Chubby provides mechanisms like semaphores and mutexes for a distributed environment. When Not to Use Chubby? Bulk Storage is needed Data update rate is high. Locks are acquired/released frequently. Usage is more like a publish/subscribe model. Background Chubby is neither really a research effort nor does it claim to introduce any new algorithms. Rather, Chubby describes a certain design and implementation done at Google in order to provide a way for its clients to synchronize their activities and agree(Consensus) on basic information about their environment Chubby and Paxos Chubby uses Paxos underneath to manage the state of the Chubby system at any point in time. Getting all nodes in a distributed system to agree on anything (e.g., election of primary among peers) is basically a kind of distributed consensus problem. Distributed consensus using Asynchronous Communication is already solved by Paxos protocol. Chubby Common Terms Chubby Cell Chubby cell is a Chubby Cluster. Most Chubby Cells are single Data Center(DC) but there can be some configuration where Chubby replicas exist Cross DC as well. Chubby cell has two main components, server and client, that communicate via remote procedure call (RPC). Chubby Servers A Chubby Cell consists of a small set of servers(typically 5) known as Replicas. Using Paxos, one of the servers is selected as Master which handles all client requests. Fails over to another replica if the master fails. Each replica maintains a small database to store files/directories/locks. The master writes directly to its own local database, which gets synced asynchronously to all the replicas(Reliability). For Fault Tolerance, replicas are placed on different racks. Chubby Client Library Client applications use a Chubby library to communicate with the replicas in the chubby cell using RPC. Chubby API Chubby exports a unix-like file system interface similar to POSIX but simpler. It consists of a strict tree of files and directories with name components separated by slashes. E.g. File format: /ls/chubby_cell/directory_name/…/file_name A special name, /ls/local, will be resolved to the most local cell relative to the calling application or service. What is the most local Cell? Chubby can be used for locking or storing a small amount of data or both, i.e., storing small files with locks. API Categories General Open() : Opens a given named file or directory and returns a handle. Close() : Closes an open handle. Poison() : Allows a client to cancel all Chubby calls made by other threads without fear of deallocating the memory being accessed by them. Delete() : Deletes the file or directory. File GetContentsAndStat() : Returns (atomically) the whole file contents and metadata associated with the file. This approach of reading the whole file is designed to discourage the creation of large files, as it is not the intended use of Chubby. GetStat() : Returns just the metadata. ReadDir() : Returns the contents of a directory – that is, names and metadata of all children. SetContents() : Writes the whole contents of a file (atomically). SetACL() : Writes new access control list information. Locking Acquire() : Acquires a lock on a file. TryAcquire() : Tries to acquire a lock on a file; it is a non-blocking variant of Acquire. Release() : Releases a lock. Sequencer GetSequencer() : Get the sequencer of a lock. A sequencer is a string representation of a lock. SetSequencer() : Associate a sequencer with a handle. CheckSequencer() : Check whether a sequencer is valid. Chubby does not support operations like append, seek, move files between directories, or making symbolic or hard links. Files can only be completely read or completely written/overwritten. This makes it practical only for storing very small files.
...