CMPS290S, Fall 2018

An overview of Erlang

2018-12-14T00:00:00+00:00

by Natasha Mittal ⋅ edited by Abhishek Singh and Lindsey Kuper

Introduction

In 1981, the Ericsson Computer Science Laboratory (CSLab) had been experimenting with ways to program telephony features in Prolog, a declarative language. Telecom applications in general are distributed systems with a large number of concurrent actions taking place. The downside to Prolog was that such declarative languages did not have error-handling facilities and also lacked the means for concurrency control across multiple systems. Thus began a series of collaborations which led to the development of Erlang.

Erlang is described in Joe Armstrong’s 2003 PhD thesis, “Making reliable distributed systems in the presence of software errors”. In it, he describes how Erlang supports building fault-tolerant systems. Since its inception in 1986, Erlang has grown popular for building reliable telecom applications. It has been used in Web Prioritizer and Mail Robustifier, two products developed by Bluetail, a company founded by Joe Armstrong. Ericsson’s AXD301, a scalable ATM switching system developed using Erlang middleware, was one of the company’s most successful new products from 1998 to the mid-2000s.

Armstrong describes Erlang as “a concurrent programming language designed for programming large-scale distributed soft real-time control applications”. It is used in conjunction with a collection of libraries and tools called OTP (Open Telecom Platform), which uses “supervision trees” to provide descriptions of error recovery actions to take for a given error. Erlang has been described as a “process-oriented” language; individual processes do not share memory and communicate via asynchronous message passing, hence maintaining strong isolation between concurrent processes. Since processes are isolated from each other, errors occuring in one process cannot propagate to other processes, so Erlang’s programming model is able to use fail-fast processes. Erlang also offers support for dynamic code replacement, which aids in code updating and maintenance without stopping the system. This is essential since telecom applications are long-lived, or, more often than not, aren’t shut down ever.

Concurrency-Oriented Programming

In his thesis, Armstrong coined the term COPL, which stands for “concurrency-oriented programming language”, and argued that Erlang falls into this category of languages. Armstrong wrote that the advantage of using COPLs is the way they can easily model real-world concurrent activities and map them onto concurrent processes in a one-to-one fashion, in contrast to non-COPLs, where one process or thread might control several independent activities.

As described in section 2.4.2 in Armstrong’s thesis, there are six essential characteristics of a COPL:

It supports lightweight processes, i.e., the computation required to generate and destroy processes is minimal.
It supports isolation of processes.
Every process is identified uniquely by a Pid.
There is no shared state between processes.
Message passing does not guarantee delivery, and is pure (no dangling pointers or data references).
Processes can detect the occurrence of, and the reason for, failures in other processes.

A critical requirement in COPLs is isolation. There must be strong isolation between the multiple processes running on a single machine. Unless programmed, no faults in any process should affect any of the other processes on the machine. To enable isolation, all processes have “share nothing” semantics and message passing between processes is asynchronous to prevent a sender of the message from getting indefinitely blocked in case software errors occur in the receiver. Data is also immutable within individual processes.

Another requirement is unforgeability of process names so that it is impossible to guess their names. Each process can only know its own name and the names of the child processes it has created. The act of revealing names to other processes is called the “name distribution problem”, mentioned in section 2.4.4 in Armstrong’s thesis. Such revelations have to be limited to trusted processes only to maintain system security.

Message passing has “send and pray” semantics in Erlang. Every message is assumed to be received in its entirety or not received at all. Messages can be sent to and received via mailboxes, which every process has. To aid isolation, a message cannot contain pointers or references to data structures residing on other machines. Additionally, messages are received in the exact order they were sent. A key advantage of message passing is scalability: message-passing systems are relatively easy to replicate over multiple isolated machines, thereby enabling fault tolerance as well. Even though individual components may fail, the probability of all of them failing at the same time is low.

Fault tolerance in Erlang

According to section 5.1 of Armstrong’s thesis, the main strategy for implementing fault tolerance is to try to perform a task, and if unsuccessful, try to perform a simpler task. In this manner, a hierarchy of tasks is established. This strategy helps avoid unnecessary complexity that might result in the system becoming less reliable.

Supervision hierarchies

Armstrong refers to these hierarchical organizations of tasks as “supervision hierarchies”. In this level-based organization, the highest-level task is to run an application according to specific parameters, and if not possible, to run simpler lower-level tasks. A system failure occurs if the lowest-level task cannot be performed successfully. As we go down to simpler tasks, failure to perform the task becomes more unlikely. It is also interesting to note that on encountering more and more failures at different levels, the emphasis becomes less towards providing complete service and more towards protecting the system. Hence it becomes important to have some mechanism to log all failures and their particular reasons.

The supervision hierarchy detects and attempts to stop errors from propagating upwards in the system. Every task is associated with a supervisor process which assigns it to a worker for achieving the goals necessary to complete the given task.

A supervisor needs to have the information about how to start, stop or restart every worker under it. This data is stored in an SSRS (Start Stop and Restart Specification). In a linear hierarchy, the rule is: stop all child processes if a parent asks to stop the supervisor; and to restart a child in case it dies.

Programming model

Process creation

Spawning is the way new processes are created in Erlang. THe spawn() function takes as arguments a module name and the name of a function within that module. For example, the call spawn(moduleA, response, [thank you]) creates a new process which executes a function named response defined within module moduleA with the argument thank you.

The newly spawned process executes the function, and spawn() returns the identifier for the spawned process, i.e., the Pid. Pids can then be used for message passing between individual processes.

Message Passing

All processes in Erlang use the same message-passing interface. Message passing is the only form of data exchange between two processes, and there is no data sharing. A message can be a list, a tuple, integers, and so on. Message passing takes place causally. The receive language construct directs a process to wait for a message to come from another process.

The example below is taken from the “Message Passing” subsection of the Erlang documnentation, which explains Erlang’s constructs for sending and receiving messages. In this example, two processes are created that send messages to each other a number of times.

-module(msg_passing).
-export([start/0, ping/2, pong/0]).

ping(0, Pong_PID) ->			
  Pong_PID ! finished,					%% syntax to send message, string ‘finished is being sent to process with Pid=Pong_PID
  io:format("ping finished~n", []);		

ping(N, Pong_PID) ->
  Pong_PID ! {ping, self()},		
  Receive						%% receive block does a pattern matching
    pong ->
      io:format("Ping received pong~n", [])
  end,
  ping(N - 1, Pong_PID).				%% recursive call to itself

pong() ->
  receive
    finished ->
      io:format("Pong finished~n", []);
    {ping, Ping_PID} ->
      io:format("Pong received ping~n", []),
      Ping_PID ! pong,
      pong()
  end.

start() ->
  Pong_PID = spawn(msg_passing, pong, []),		%% a process is spawned which executes pong function
  spawn(msg_passing, ping, [3, Pong_PID]).			%% another process is spawned which executes ping message

An example run of the code looks like this:

Eshell V5.9.3.1  (abort with ^G)
1> c(msg_passing).
{ok,msg_passing}
2> msg_passing:start().
Pong received ping
<0.39.0>
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
ping finished
Pong finished

Process Linking

Linking of Erlang processes is done via link() or spawn_link() functions. The example discussed below has been taken from the “Errors and Processes” chapter of Learn You Some Erlang for Great Good!, which clearly explains how linking works in Erlang.

The chain() function in the linking module below spawns N processes, which are linked to each other.

-module(linking).				%% a module named linking is created
-export([chain/1]).				%% export makes chain function public

chain(0) ->					%% base case when N=0 and process dies after 2000 milliseconds
  receive
    _ -> ok
  after 2000 ->
    exit("chain dies here")
  end;

chain(N) ->					%% recursive call to spawn N processes and linking them to each other
  Pid = spawn(fun() -> chain(N-1) end),	%% spawns a new process running chain(N-1) func
  link(Pid),
  receive
    _ -> ok
  end.

After three recursive calls to linking:chain(), the process running chain(0) dies and this error propagates to other processes running with N=1, 2, and 3 consecutively. Eventually, the error propagates to the top-level Erlang shell process, erl shell, which also dies, as seen in the stack trace below.

Execution:
Eshell V6.4.1.7  (abort with ^G)
1> link(spawn(linking, chain, [3])).		%% N = 3, so three processes are spawned
true
2> 2> 2> ** exception error: "chain dies here”
Stack Trace::
[erl shell] == [N=3] == [N=2] == [N=1] == [N=0]
[erl shell] == [N=3] == [N=2] == [N=1] == *dead*
[erl shell] == [N=3] == [N=2] == *dead*
[erl shell] == [N=3] == *dead*
[erl shell] == *dead*
*dead, error message shown*
[erl shell] <-- restarted       %% here erl also dies eventually.

The error propagation that linking enables is used by the aforementioned OTP supervision trees.

Conclusion

The past couple of decades have seen an increase in Erlang’s popularity. Databases like CouchDB, Scalaris, and Amazon SimpleDB have been implemented using Erlang. Erlang is also an increasingly popular choice for general-purpose programming, as in the Nitrogen framework for web development, or even Wings 3D, designed for graphics modeling. In conclusion, the following quote from Armstrong in a 2013 interview seems apt: “If Java is ‘write once, run anywhere’, then Erlang is ‘write once, run forever’.”

Conflict resolution in collaborative text editing with operational transformation (Part 2 of 2)

2018-12-13T00:00:00+00:00

by Abhishek Singh ⋅ edited by Devashish Purandare and Lindsey Kuper

Introduction

In this post we continue the discussion on collaborative text editing started in my previous post. The goal of part 1 of the post was to provide an overview of the dOPT operational transformation algorithm from Ellis and Gibbs’ “Concurrency control in groupware systems” paper and how it addresses the problem of conflict resolution in collaborative text editing. We looked at the problem through a toy example under a set of assumptions that limited the scope of the problem. In this post we remove some of those assumptions and discuss the details of the dOPT algorithm as discussed in the paper.

Here’s the list of the assumptions I made in my previous post:

Operation messages from either site are received exactly once.

There are exactly two editors in the system: one at Alice’s end and the other at Bob’s end.

My implementation does not use clocks to timestamp operations, so the happens before relationship is established based on message delivery. It is assumed that LOCAL and REMOTE operations happen concurrently.

Operations are processed in the order in which they are seen and executed at each particular site. In our implementation, the executed operations are stored in a list OTEditor.Ops.

Unlike the implementation in the paper, we do not assign priorities to an operation. Every operation has equal priority.

An operation is sent to others immediately after it was executed at one particular site. There is no out-of-order delivery of messages.

For reference, here’s the example collaborative editing session between Alice and Bob that we looked at last time:

Figure 1. Operations received by Alice and Bob are transformed before being applied to local data.

Let us consider the consequences of removing each of the above assumptions. Primarily, there is a problem of ascertaining causality of messages in the design. Consider the second and third operations exchanged between Alice and Bob in Figure 1. There is no way for Alice to know if the operation INSERT ("x", 2) sent by Bob happened before or after Bob received the INSERT("y", 0) operation from Alice (even though we can see the figure and know that the operations were concurrent). Figure 2 shows Alice’s view of the world: Alice cannot know if there is a causal relationship between the operations, because we did not address causality in part 1. The ordering of the messages was enforced only by the order in which the messages were delivered to each user.

Figure 2. Alice does not know how Bob executed the operations at his end and therefore cannot know how received operations should be applied at her end.

The possibility of receiving out-of-order messages is another issue that the design must address. The algorithm in part 1 executes messages on each replica in the order in which they are received. This aggravates the problem of data inconsistency among the replicas if messages were delayed in transit and were not received in the same order in which they were generated. Suppose the operation INSERT ("x", 2) from Bob in Figure 1 was delayed and received by Alice after the DELETE (3) operation from Bob. This would lead to the operations being executed in different orders on Alice’s and Bob’s ends leading to data inconsistency between the replicas. Since in-order message delivery is subsumbed by causal delivery, we can also solve this problem by addressing causality.

Finally, another assumption we made has to do with the number of users participating in the system. It is relatively easy to understand how operation messages are exchanged when only two users exist in the system. With more users, the number of messages needed to share changes with other users grows. If there are n users in the system, every state change at one user must be communicated with n-1 users. With each user making changes to their document replica, the version of dOPT developed for the two-user collaborative text editor developed in part 1 cannot be used, as there is no way of knowing the exact state of a replica when an operation was executed.

Precedence, convergence, and quiescence

The crux of the data consistency problem discussed until now is that a site does not know the historical trace of an operation when it receives the operation request. The approach proposed for dOPT in Ellis and Gibbs’s paper involves the use of a vector clock to order the operations generated on different sites (each site or user is associated with a single document replica). The paper defines a Convergence Property and a Precedence Property and a notion of quiescence to describe the correctness of the algorithm. The authors define the properties as follows:

The Precedence Property states that if one operation, o, precedes another, p, then at each site the execution of o happens before the execution of p.

A groupware session is quiescent iff all generated operations have been executed at all sites, that is, there are no requests in transit or waiting to be executed by a site process.

The Convergence Property states that site objects are identical at all sites at quiescence.

The Precedence Property makes use of the precedes relation, which is reminiscent of Lamport’s “happens-before” relation. If an operation o precedes p, then the Precedence Property ensures that o is executed before p everywhere.

The Convergence Property combined with the notion of quiescence is nearly identical to the Strong Convergence property defined by Shapiro et al.. In the Ellis and Gibbs paper, however, quiescence is enforced periodically. The detection of quiescence is done via a distributed consensus algorithm, which is not discussed in the paper.

The Distributed Operational Transformation (dOPT) algorithm

The dOPT algorithm in essence is an attempt to order operations for execution by maintaining their causal links to other operations and, based on that order, decide whether transformation of the operation indices are really needed when executed by individual sites. The dOPT algorithm described in the paper assumes a constant number of sites. For every change that a site makes to its replica, a request is generated and sent to other sites. To achieve the two properties the design uses a request queue Q_i and a request log L_i, where the subscript i is the site identifier. Throughout the rest of the discussion, site i will be the location where the request is being processed and site j will be the site which sent the request.

The request queue contains operation requests either sent by remote sites (j) or from the user of the current site (i). These are requests that are waiting to be processed, and the queue acts as a buffer where all incoming requests are stored for further processing. The request log, on the other hand, is a log of requests that have been executed by the site. The log is a list of requests ordered by the order in which the requests were executed.

Each request has the form <i, s, o, p>, where i represents the site identifier and s represents the state vector of the site i. The state vector, as discussed by Ellis and Gibbs, is essentially a vector clock that specify when an operation was executed on site j and its relation to the operations in the request queue in site i. o represents the operation to be performed (insert or delete). Finally, p specifies the priority of the operation. As discussed in my previous post, operations must commute with each other.

The dOPT algorithm builds a transformation matrix for the operations that are supported: insert and delete. A 2x2 transformation matrix is created which reflects the cases seen during operation execution: insert-insert, insert-delete, delete-insert and delete-delete. The priority of an operation is used to compute if an operation should be transformed and what the values of the recomputed indexes should be.

The algorithm uses state vectors to order events causally. Given two state vectors s_i and s_j, Ellis and Gibbs define the following relations:

s_i = s_j if each component of s_i is equal to the corresponding component of s_j.

s_i < s_j if each component of s_i is less than or equal to the corresponding component of s_j and at least one component of s_i is less than the corresponding component in s_j.

s_i > s_j if at least one component of s_i is greater than the corresponding component in s_j.

For example, if s_i = [ 1 2 3 3 ] and s_j = [ 1 2 3 4 ], we have s_i < s_j. However, if s_i = [ 4 3 2 1 ] and s_j = [ 1 2 3 4 ], we have s_i > s_j.

During initialization the following operations are done:

Set Q_i to empty
Set L_i to empty
Set s_i to [0 0 ... 0]

The algorithm defines three possible execution states:

Operation request generation,
Operation request reception, and
Operation execution.

During operation request generation the site i generates an insert or delete operation. The operation is not executed immediately; the local data is not modified during operation request generation. Once the request is generated, it is appended to the site’s request queue Q_i and broadcast to all other sites.

Generate operation <i ,s_i , o, p>

Q_i := Q_i + <i ,s_i , o, p>

A request generated on a site j is eventually received by site i which then moves to the “operation request reception” state. In this state, the received operations are appended to the site’s request queue.

Receive operation request from remote site j: <j ,s_j , o_j, p_j >

Q_i := Q_i + <j ,s_j , o_j, p_j >

During operation execution, requests from the request queue are processed. The order of execution of requests in the request queue is determined by the total order of events in the request queue as determined by the comparison of the state vectors. Briefly, in this step the operation from the request queue is chosen based on the executed operations in the request log. We locate the operation older than the current state vector at site i. Transformation is performed based on the operation logs in the request log. Comparison of the state vectors follows the conditions stated previously:

If s_j > s_j, this means that the site j has executed operations which site i has not seen yet. So this operation will have to stay in the queue till all operations between i and j have been executed.
If s_j = s_j, the two state vectors are identical and operation o_j can be executed without transformation.
If s_j < s_j, site i has executed operations not seen by site j. The operation can be applied immediately, but requires operations to be transformed because other changes not visible to site j have already been executed by site i.

The principles behind transformations were discussed in my previous post. The main idea is that the transformations must commute. This allows the operations to be executed in any order. The idea of commutative operations is vital to any operational transformation technique. In fact, the idea of commutative (and associative) operations comes up very frequently in discussions of synchronization-free convergence algorithms in distributed systems. A fairly recent example is in a paper by Attiya et al. when discussing convergence of replica state in their formalization of the RGA protocol for collaborative text editing. Another example is in conflict-free replicated data types, where operation commutativity is key to state convergence.

Final thoughts

The idea of operational transformation originally proposed by Ellis and Gibbs has morphed into a compendium of technologies used to build collaborative systems. A prominent example is the Jupiter collaboration system. Instead of a peer-to-peer system as we have been discussing until now, Jupiter used a centralised architecture where a server maintains a single copy and all operation requests are handled via the server. This system became the basis of the Google Wave and Google Docs projects, as mentioned in my previous post.

Over the course of these two blog posts, I have aimed to understand the key ideas behind the early specification of operational transformation and convey them with some clarity. Although collaborative text editing has been a topic of research since at least the 1980s, writing these posts allowed me to study the problem in some detail and has kindled my interest in building systems where multiple agents can work together towards a common goal. Thinking about building useful abstractions for collaborative computing agents is something that will keep me busy for some time.

State is Progressive, or: hey, what happens if we make literally _everything_ append-only?

2018-12-13T00:00:00+00:00

by Sohum Banerjea ⋅ edited by Natasha Mittal and Lindsey Kuper

State is weird.

State is weird because we want the freedom to mutate and mangle it however we like. That’s the default conceptualisation we have of state, since we all learnt to program – state is a bag of bits, or structs, or objects, that we can mangle however we like.

But we keep finding situations in which this view of state loses us verifiability, or reproducibility, or, god forbid, performance. Whether it’s multicore processors or distributed systems, storage systems or language models, we keep finding ourselves in cases where the bag-of-bits model lets us do less than we could.

In the effort to do more, there’s a common thread: designs that don’t mutate state, only add to it; that conceptualise knowledge as something that only monotonically grows, and never decreases.

In this post, we’ll look at a few examples of these “progressive systems”. For our purposes, it’s easiest to discuss them in terms of the axis along which they restrict state to be monotonic. Treating the data itself as always monotonic, independent of time, gets you lattice models, whereas treating time as the property that is always monotonic gives you stream models.

Lattice Models

Placing this monotonicity guarantee on your data means ensuring that “bigger” data is always closer to the final state, for some definition of “bigger”. Mathematically, enforcing this structure on data cashes out to structuring it as a join-semilattice, which we obviously have to shorten to “lattice”. A join-semilattice is a set with an associated partial order, and where every pair of elements have a least upper bound (“join”) that respects that partial order. The archetypical example here is sets under the subset partial order. Every pair of sets has a least upper bound — the union of those two sets — that respects subset inclusion as a partial order. Thus, sets, under the partial order of subset inclusion and the join operation of union, form a join-semilattice.

Okay. So we can think of the states our data can take on as members of a join-semilattice. What does this actually buy us?

It means that if we have conflicting copies of our data ({2,3} and {3,5}, let’s say) at any point, we know what to do to resolve the conflict: take the least upper bound ({2,3,5}). Since we know our data can never get “smaller”, this is a correct operation — the state of our data is at least what it is at each replica. This is the fundamental idea that links the following three distinct approaches to state.

CRDTs

CRDTs, or conflict-free replicated data types, try to present the concept of join-semilattices to the programmer as transparently as possible. They’re data structures designed for replication, and a lot of their complexity comes from providing expressive, efficient data structures while maintaining the property that the data is always monotonically growing. Research in the field tries hard to make the resulting data structures efficiently support the operations we want them to.

There ends up being a lot of subtlety in how more complex operations can be supported when we try to represent a data structure as a lattice — even the canonical CRDT, the set, gets complicated once we dare to do anything as gauche as deleting elements. (In short: deleting elements from a set is a non-growing operation, and so if we’re not careful, when we delete an element from one replica, the rest of our replicas will just assume it’s an addition we haven’t seen yet, and add it right back again.)

A major issue with using CRDTs in practice is how they need to be garbage collected — as you may imagine, having your state grow without bound is not a desirable property for a data structure. This is an instance of non-monotonicity in our programs: as much as we’d like our programs to be monotonic, we end up needing these synchronisation points for real-world practical usage.

LVars

LVars are another data structure that associates states with lattice elements. Any write to an LVar updates it to the least upper bound of the value being written and the state the LVar already had. Where they differ from CRDTs is in the additional restrictions they impose on how this data can be interacted with: LVars are not directly readable. The read operation on an LVar is a “threshold read” — an operation that blocks until one of a set of specified points in the lattice has been reached or surpassed, and thus can never return any intermediate state. To preserve determinism, this threshold read doesn’t even necessarily return the current state of the LVar when it unblocks. Rather, it deterministically returns a state that the current state is guaranteed to be at or above.

In exchange, LVars recover determinism. That is, all interleavings of a program that coordinates data with LVars will produce the same result. This design decision is a more natural set of tradeoffs to want in their domain: LVars were designed for shared memory, whereas CRDTs were designed for replicated data.

There is a way to read the exact state of the LVar, but only after it’s been “frozen”. This creates the opportunity for races between attempts to write to the LVar and attempts to freeze it, and thus this weakens the determinism guarantee to “quasi-determinism” — all interleavings of the program are guaranteed to produce the same result as long as that interleaving doesn’t error.

LVars don’t aim to adddress non-monotonic use cases; instead, the philosophy is that they can just give up control over the data once a non-monotonic operation is necessary. The assumption is that a non-monotonic operation will not be necessary until after a (often lengthy) phase of monotonic computation, and so LVars allow parallelisation of the monotonic phase.

Bloom and Bloom^L

Bloom and its successor Bloom^L build monotonicity right into a programming language – or, to be more precise, they remove operations until the only operations left are monotonic.

The idea is that non-monotonicity is what leads to the need for synchronisation in distributed programs. If your program is not monotone, then it will need to have all of its inputs to produce the correct result. Some approaches to distributed programming weaken the “correctness” of the result in response, but if you could program with purely monotonic operations most of the time, you could narrow down the places you need to synchronise to those places where you’re using non-monotonic operations. This is the CALM principle — “Consistency As Logical Monotonicity”.

Bloom^L has you write monotone functions on data structures, so it can ensure that the entire program’s use of state is monotone and thus doesn’t need synchronisation to ensure eventual consistency. One particularly useful class of monotone functions is homomorphic functions — functions that map from one lattice to another, ensuring the map preserves the structure of the lattice while doing so. For instance, under the standard lattice for sets, a set.is_empty() operation is homomorphic if and only if the resulting boolean is false-biasing. That is, false > true in its lattice, and thus ifnot is the only monotonic operation available on it, not if.

The constraints of Bloom^L allow it to guarantee monotonicity without static checking in most cases.

(Psst. Don’t tell anyone, but Bloom^L also belongs in the next section. See you there!)

Data as Streams

Another way to have grow-only state is to just totally order it in time. Despite all the problems in trying to totally order time, there are some cases in which we can structure the entirety of how we process state such that we can exploit this total ordering.

In practice, these designs look like stream processing languages. Streams are inherently ordered data, which means we always know what’s newer and what’s older, and stream processing is inherently an operation in time, such that any stream processor can only ever emit at time t data that is dependent on input data up to time t. This is, of course, monotonicity again, just pushed a layer outward, into metadata — timestamps — associated with our data cells.

In this framework, one key requirement keeps showing up: for the system or a stream to somehow guarantee that no further messages with timestamps ≤ t will appear. Most commonly referred to as “punctuation”, this shows us the key point at which we fundamentally need to guarantee the (local) monotonicity of time for our operations on these streams to continue being monotonic.

CQL

CQL, the “Continuous Query Language”, takes this framing and applies the well-known declarative language of relational logic (i.e., SQL) to streams.

The core of CQL is in its stream-to-relation and relation-to-stream operators, in order to let the standard SQL operators do work on the intermediate relations. It deliberately does not provide stream-to-stream operators, requiring all operations to be done on the relational layer. To do this, they augment their concept of relations to natively include time; all relations in CQL are time-varying (parameterised by timestamp).

Both of these classes of operators are simple to specify:

stream-to-relation operators capture a given time window over the stream and present the data within that window as a relation with the same schema
relation-to-stream operators emit elements on the stream at time t when
- an element is added to the relation at time t (Istream)
- an element is deleted from the relation at time t (Dstream)
- or, an element just is in the relation at time t (Rstream)

The language attempts to add a default Istream in cases where a conservative static check of the underlying relation can guarantee that the relational operation being described is monotonic.

Punctuation isn’t considered a language-level concern for CQL — the language assumes you have access to all data up to timestamp t when computing something at timestamp t, and it’s left to the runtime to validate this requirement. However, they do describe that their runtime does need to have ways of generating these “heartbeats”, and various policies for doing so.

Bloom^L

Bloom^L also uses streams as a core data type. More precisely, it parameterises its data cells by time, thus providing a similar set of properties to stream based languages. It does this in order to model the nondeterminism of asynchronous communication (a message from one node to another is modelled as data inserted into a shared object at a nondeterministic future timestamp).

Bloom^L is capable of reasoning about non-monotone parts of your program, and this integration of time into its modelling is a major reason why it can do so.

When you allow Bloom^L to reason about punctuation, you get Edelweiss, a generic solution to the garbage collection problem CRDTs have to face. Edelweiss has a far more general concept of punctuation than either other system — it’s a guarantee that no more messages will arrive matching any arbitrary predicate. Edelweiss can then track these predicates across channels compositionally (if the set of peers at a particular epoch is fixed, and all of those peers have responded to a given message, it’s guaranteed that one will not receive any further responses to that message).

Pervasive monotonicity, combined with the strong guarantees Edelweiss enables via its punctuation system, allow the user to reason directly about what state they no longer need to keep around and can thus garbage-collect.

Timely Dataflow

Timely dataflow is an approach to stream processing that has handling of iterative loops as an explicit design goal. Loops are constrained specifically to ensure they manage the timestamps of messages flowing through the system in a precise, predictable, monotonic way.

Since the system has more control over the timestamps than a general stream processing system, it can reason about how messages could result in later messages at given timestamps — essentially providing a flexible, productive method for generating punctuation across the whole system in the presence of loops. This allows for complex dataflows across the resulting stream processing nodes, including messages sent to nodes at specific future timestamps, and nodes that wait for punctuation before computing some non-incremental property of the data.

Again, we find monotonicity, as modified to address the problem, to provide strong properties we can exploit — even when computing non-monotonic properties.

What Does All This Mean?

State is progressive.

…Okay, that may be overselling it. While all of these models have important benefits when applied to the use cases they were designed for, none of them attempt to be a general-purpose model of state. We’re not yet trying to overthrow the von Neumann machine as the fundamental architecture on which computation is built.

That said, these designs do point to a “turning point” in how we think about state. If you can stomach the limitations of designing your state to be progressive, you get strong, relevant properties in return.

State may be progressive. Maybe it’s time to seriously consider those designs.

Diversifying Consistency in Ceph

2018-12-12T00:00:00+00:00

by Aldrin Montana · edited by Austen Barker and Lindsey Kuper

Introduction

In my previous post, we provided background and motivation for mixing consistency in a programmable storage system. In this post, we describe approaches to supporting a range of consistency options in the Ceph storage system to enable mixed-consistency, programmable storage on top of Ceph.

We will compare approaches to implementing a variety of consistency models by first looking at the implementation of PROAR, a recently proposed replication strategy for Ceph that trades strong consistency for lower write latency. We will also consider other approaches to implementing eventual consistency in Ceph. Finally, we’ll consider implementing [“bolt-on” causal consistency][bolt-on-paper] on top of Ceph, which requires that the underlying datastore support eventual consistency.

Replication and Data Placement in Ceph

To understand how Ceph might support weaker consistency models such as eventual consistency or causal consistency, we describe how the storage subsystem in Ceph, RADOS, manages persistent storage and replication. We also mention the role of Ceph’s CRUSH algorithm for determining data placement.

Storage in RADOS consists of four layers of abstraction. At the highest level are files or abstractions for applications to write and read data. The second layer maps files to data objects. Then, the third layer distributes (stripes) data objects across the storage cluster into placement groups (PGs) using a hash function. Each PG has a replication factor, which represents how many replicas a data object assigned to the PG will have. A PG is a logical entity, not a server, that can be likened to a horizontal shard in data management systems. At the lowest level of abstraction, object storage devices (OSDs) handle data persistence and replication.

The diagram below depicts an overview of the data placement in Ceph. OSDs are assigned to, or organized into, PGs. OSDs of the same PG cooperate to replicate data objects and exchange state, and do not communicate with OSDs associated with other PGs. The diagram below is based on descriptions in the RADOS and Ceph papers; for a slightly higher level of abstraction, see Figure 3 in the Ceph paper. The mapping of files to objects to PGs to OSDs is succinctly described in Section 5 of the Ceph paper. The organization of OSDs into a PG is calculated using the CRUSH (“controlled replication under scalable hashing”) algorithm, detailed in the CRUSH paper.

Overview of object assignment to placement groups and object storage devices in Ceph

To facilitate communication and failure recovery, monitor services maintain an up-to-date representation of cluster state called the cluster map, which includes all PGs, OSDs, and other monitors. During the first interaction clients and other services have with the storage cluster, a request is sent to a monitor for the current version of the cluster map. On subsequent communications, OSDs lazily provide incremental updates of the cluster map. The RADOS paper describes how Ceph clients, OSDs, and monitors interact.

Eventual Consistency in Ceph

Zhang et al. describe their approach to eventual consistency in Ceph in their PROAR paper. Their replication strategy differs from standard Ceph in how it determines which OSD (i.e., which replica) receives read and write operations from clients, called primary role hash ring (PROAR). The PROAR (“primary role hash ring”) algorithm cycles through the ordered list of OSDs calculated with CRUSH, using the ID of the data object being written. For instance, if the PG replication factor is 3, and we have the object 4, then 3 % 4 = 1, thus the write operation is sent to OSD 1. This is depicted in the diagram below, adapted from Figure 3 in the PROAR paper. When the write operation is received and applied, OSD 1 immediately acknowledges the write, without waiting for acknowledgements from other replicas. Presumably, OSDs are still expected to communicate and lazily replicate data among themselves. The name “hash ring” suggests a similarity to the consistent hashing ring described in the Dynamo paper. While PROAR iterates over the OSDs circularly, CRUSH provides consistent hashing.

By distributing writes across the OSDs of a PG and acknowledging the applied write from the same OSD, PROAR improves write latency at the expense of consistency. PROAR claims to provide eventual consistency, and the fact that operations on a given object are deterministically directed to the same replica can help to ameliorate typical replication anomalies, such as stale reads.

PROAR PG replication circle, based on Figure 3 in the PROAR paper

Other eventually consistent replication strategies

The PROAR paper proposes one approach to supporting eventual consistency in Ceph. We could also consider other ways to modify the various replication strategies currently supported by RADOS. The RADOS paper describes three replication strategies: primary-copy, chain, and splay. Figure 2 from the RADOS paper is included below as a reference point. We’ll look at each of these replication strategies and consider how we might tweak them to lower write latency.

Figure 2 from the RADOS paper: replication strategies implemented by RADOS

The default replication strategy for Ceph OSDs is primary-copy, in which clients are expected to send all write and read operations to the primary OSD for the PG. An acknowledgment of an applied write, or ack, is sent to the client from the primary OSD. A client only receives an ack when it is known that all replicas have received the write. This ensures strong consistency, therefore maximizing data safety, or the ability to recover data after failures. The different replication strategies have different policies for which OSD sends an ack to the client and to which OSD a client should send read operations, but in all three replication strategies, write operations are always sent to the primary OSD.

In each replication strategy that RADOS uses, write latency includes time for each replica to acknowledge a write operation. As a consequence of where read operations are directed and when the primary OSD acks to the client, read operations are always strongly consistent. In the below discussion, we consider alternative approaches that could lower write latency, including changing when a client receives an ack and which OSD to send read and write operations to. We want to continue to ensure that read operations will only observe the effects of writes that have already been applied at some OSD; we won’t allow reading of a write that has not yet been applied. However, reads may observe the effects of writes that have been applied at only one replica, or a subset of replicas, instead of all replicas.

For primary-copy replication, let’s compare the behavior of read operations between the primary OSD and secondary OSDs if we change when the primary OSD acks relative to a received read operation. Currently, the primary replica acks after a write has been applied at all replicas, thus reads return a strongly consistent value. Alternatively, we could ack once the write has been applied just at the primary. Other replicas could send an ack to the client asynchronously after applying the write. In this case, reads are strongly consistent as long as the primary is active, and weak reads occur if the primary goes down and other replicas have not received and applied all writes. This level of consistency is the same as PROAR, the difference being that all writes and reads are still funneled through the primary OSD which could have increased ack latency during times of high load.

In this weakening of the primary-copy replication strategy, an epoch for tracking which replicas have received which updates would be necessary, because we can no longer rely on the primary logically being the last to apply an update. Note that when the primary OSD fails in the standard primary-copy replication strategy, a newly elected primary OSD may not have received all writes from the former primary OSD. The same recovery process can be used for the weakened and standard primary-copy replication strategies, although recovery time for the weakened replication strategy would likely be longer. PROAR approaches failure recovery by maintaining a PG log at each OSD in the PG.

For chain replication, we consider how we can change which replica receives read operations and which replica acks to the client. Reads sent to the primary will always be strongly consistent, but will return values for writes that may not have been propagated to other replicas. Typically, reads sent to the last replica in the chain (the tail) are strongly consistent because acks are sent from the tail. If the primary replica were to ack instead of the tail, then reads sent to the tail can be, at most, n-1 versions old, assuming n updates in a row, where each OSD in the chain from the primary to the tail has a version of the data object that is at least as old as that of its predecessor. In general, if the replica that acks writes (say, OSD_ack) is at least as far down the chain than the replica that receives reads (OSD_read), then reads are strongly consistent. Furthermore, the number of replicas in the chain between OSD_read and OSD_ack is the “replication factor”, or the number of replicas on which each write will be applied.

Finally, we consider splay replication. The main difference between primary-copy and splay is which OSD acks to the client that a write has been applied. This allows better load balancing by splitting read and write workloads across the primary and tail replicas. This difference is considered in our above weakening of primary-copy replication. So, we observe that because the primary replica applies writes before propagation to other replicas, then to reduce latency the tail should ack concurrently while applying writes. It seems that changes to primary-copy and splay replication converge to similar protocols for which replica acks to the client, and which replica receives read operations. This is due to wanting to prevent dirty reads at the primary replica.

Bolt-on Causal Consistency

Bolt-on causal consistency, as proposed by Bailis et al., is a mechanism for upgrading the consistency guarantee provided by a datastore from eventual to causal consistency. This approach is useful for datastores such as Cassandra, which support weak and strong consistency via quorum consistency, but cannot natively support causal consistency.

Bolt-on causal consistency assumes that the underlying datastore is eventually consistent and implements a shim layer over it. This would seem to suggest that if we can support eventual consistency in Ceph, whether with the PROAR approach or with one of the other replication approaches I’ve sketched out, then we can also have causal consistency “for free” by means of the bolt-on approach. Another assumption is that the bolt-on shim layer has access to the merge function of the datastore. For instance, Bailis et al. note that the last-writer-wins merge function requires knowledge of which writes are concurrent, or do not have sequential dependence. This merge function is not implemented in Ceph, since it currently only supports strong consistency. Though it may be possible to support last-writer-wins semantics by adding a version to each object, it would result in an undesirable data overhead for applications with many objects. Another major, but straightforward, assumption is that the shim layer will be deployed with clients as well.

Finally, bolt-on assumes that reads will obey the happens-before relation, that the datastore provides single-value register semantics instead of multiple versions, and that there are no callbacks from the datastore. These assumptions all hold in Ceph: the happens-before relation is obeyed because clients deterministically send write operations to the same replica for a given object, and the weakened versions of each replication strategy do not allow dirty reads; Ceph updates objects in place, and so provides single-value register semantics; and callbacks are not present in the POSIX interface to Ceph’s subsystem.

Final Thoughts and Next Steps

It seems to me that implementing weaker consistency in Ceph can be consolidated in OSDs because of the decoupling between cluster state (maintained by the cluster map) and write replication. This means that Ceph’s monitors and use of CRUSH can provide the state of the cluster in a strongly consistent manner, without affecting the consistency level of read and write operations. The proposed weakenings above require changes to when a write is acknowledged, which replica acknowledges an applied write, or which replicas should receive read operations. These changes appear possible in specific places in the protocol, but they may affect many other aspects such as the correctness, complexity, or runtime of disaster recovery. Understanding the depths of these details is beyond the scope of this post.

Once the use of a variety of consistency models is possible in Ceph, providing tools to make development easier in a mixed-consistency environment would be nice. Language-level abstractions such as the ones I discussed in my previous post could play an important role here.

Simplifying Agreement: Language Support for Consensus

2018-12-10T00:00:00+00:00

by Devashish Purandare · edited by Aldrin Montana and Lindsey Kuper

In my last post, we explored consensus mechanisms and why it is so hard to get them right. Implementing consensus and other distributed protocols often requires implementing a lot of supporting functionality, resulting in long and complex code. This makes it hard to verify whether the implemented protocol is true to its formal definition. The “Paxos Made Moderately Complex” implementation may be only 451 lines of Python code, but implementations used in state-of-the-art systems are much more complex. Popular Paxos and Raft implementations run into the thousands of lines of code. As the authors of “Paxos Made Live” point out:

While Paxos can be described with a page of pseudo-code, our complete implementation contains several thousand lines of C++ code. The blow-up is not due simply to the fact that we used C++ instead of pseudo notation, nor because our code style may have been verbose. Converting the algorithm into a practical, production-ready system involved implementing many features and optimizations – some published in the literature and some not.

When using a general-purpose programming language to implement Paxos, then, we seem to have two options:

We can use a higher-level language like Python; such an implementation may be concise and relatively easy to understand, but not suitable for scalability and high performance.
Alternatively, we can use a lower-level language, such as C++, and write a scalable, high-performance implementation that is verbose and hard to understand.

Is there any way to have both high performance and ease of programmability? Domain-specific languages (DSLs) promise a way to have both programmability and performance by trading off generality – so let’s see how well that works!

In this blog post I plan to explore a special class of domain-specific languages (DSLs) that have been designed for implementing consensus protocols.

Domain-Specific Languages for Distributed Systems

Programming distributed systems requires complex reasoning about the timing and order of messages sent and received, synchronization conditions, and other distributed-systems-specific concepts. In general-purpose programming languages, it is hard to implement distributed algorithms while maintaining simplicity.

Various DSLs intended for implementing distributed algorithms – especially consensus algorithms – have been proposed, such as DistAlgo, Bloom, Overlog, and PSync. In this post we will do a deep dive into one of these: DistAlgo.

DistAlgo

You might remember the infamous abstract from “Paxos Made Simple”: “The Paxos algorithm, when presented in plain English, is very simple.” Yet, as we’ve seen, implementing Paxos in practice is not so simple. Some of the difficulty in trying to convert English pseudo-code into running code comes from the fact that the programming language we are using is not especially suited for expressing consensus protocols.

A New Hope

What if we could express distributed protocols in working code as easily as they are stated in pseudo-code? This is where DistAlgo comes in! DistAlgo is a programming language that emphasizes the ability to clearly describe distributed algorithms. Prototyped in Python, DistAlgo was built specifically to address the lack of support in general-purpose programming languages for expressing constructs required for distributed algorithms.

The DistAlgo team has implemented a variety of distributed algorithms to showcase their language’s ability to express these algorithms in running code, which can then be optimized, tested, and even formally verified.

Paxos in DistAlgo

In “Moderately Complex Paxos Made Simple,” the creators of DistAlgo discuss implementing a version of Paxos in DistAlgo. The paper is an excellent read, not only for the implementation of Paxos that they walk the reader through, but also the insight they offer on what makes Paxos so hard to understand and implement. For instance, Lamport’s prose description of Paxos in the “Paxos Made Simple” paper involves the notion of a “promise”, and the authors of “Moderately Complex Paxos Made Simple” devote some effort to making this notion precise, commenting:

Indeed, this is the hardest part for understanding Paxos, because understanding an earlier phase requires understanding a later phase, which requires understanding the earlier phases.

The cyclic nature of the definition of the Paxos protocol is the cause of a lot of the confusion surrounding it, and that’s where Raft diverges from Paxos in its effort to simplify consensus. (Although, as mentioned in my previous post, it is an open question whether Raft succeeds in doing that.)

DistAlgo’s Superpowers!

DistAlgo supports the notion of a process, which presents interfaces for initialization (setup), code execution (run), and handling of received messages (receive). yield makes the process allow handling of unhandled messages, and await allows timeouts. Queries on message histories are divided into comprehensions, aggregations, and quantifications over sets.

This allows an easy implementation of simple Paxos, using just 72 lines of code. The DistAlgo implementation can be compiled to Python or run directly. The authors also argue that the high-level nature of the DistAlgo code allows easy translation to TLA+ for subsequent verification.

Moderately Complex Paxos Made Simple

The authors implement the version of Paxos we saw in Paxos Made Moderately Complex, the version with slots and scouts and commanders! Unlike the original paper, though, they encapsulate the functionality of commanders and scouts back into the leader to allow simpler code. The split in the original paper seemed unnecessary from a correctness point of view; this assigns their actions to the leader.

The implementation ends up being about 100 lines of code in DistAlgo, and 300 in Python. While this is smaller than the 450-line Python implementation of the original, it also reduces or repackages some functionality.

In DistAlgo, it is trivial to implement an optimization which keeps just the maximum numbered ballot for each slot. The paper shows how this can be achieved by changing just one line of the code:

accepted := {(b,s,c): received (’2a’,b,s,c)}

accepted := {(b,s,c): received (’2a’,b,s,c), b = max {b: received (’2a’,b,=s,_)}}

This will pick the maximum ballot for each slot instead of keeping every ballot.

They also implement leader leases with timeout, similar optimization to the Google paper to prevent unnecessary contention because of multiple readers.

The authors also manually translate the DistAlgo code into TLA+ and mechanically verify the TLA+ specification using TLAPS, discovering and correcting a safety violation in the original “Paxos Made Moderately Complex” pseudo-code which may cause preempt messages to never be delivered to the leaders.

Although initially enticed by the idea of implementing Paxos in DistAlgo, I am not sure DistAlgo achieves what it sets out to do. It doesn’t save a lot on the size of the code, and while the code complexity is reduced, the original code for “Paxos Made Moderately Complex” was not very complex to begin with. What is somewhat disappointing is that both resort to Python in the end, in a prototype which is not practical or useful to the systems community and is limited by Python’s limited support for multi-core processing. For now, it offers us a relatively simple-to-write implementation of Multi-Paxos that is presumably easy to manually convert to a TLA+ specification, but at least for the current stage of the prototype, it fails to deliver on practical applicability.

Raft in DistAlgo

Although the DistAlgo developers have an implementation of Raft in the “examples” directory of their GitHub repo, Raft isn’t described in any of their papers. This makes it unclear if the committed example is indeed complete or correct. Nevertheless I dived into the specification code, so we can check for ourselves. Testing the implementation was harder than expected. The pip module fails installation, and the setup.py stops with an error. Whatever is working for the authors doesn’t seem to work for me. I checked with a fresh uncorrupted Python virtualenv just to make sure, but it fails with similar issues. Thankfully, the authors also provide binaries with the distribution, which seem to work.

Executing the Raft code spits out a wall of text that is the log messages produced by an example run of the protocol:

> ./dar ../da/examples/raft/orig.da
../da/examples/raft/orig.da compiled with 0 errors and 0 warnings.
[369] da.api<MainProcess>:INFO: <Node_:b2c01> initialized at 127.0.0.1:(UdpTransport=45231, TcpTransport=41250).
[369] da.api<MainProcess>:INFO: Starting program <module 'orig' from '../da/examples/raft/orig.da'>...
[370] da.api<MainProcess>:INFO: Running iteration 1 ...
[370] da.api<MainProcess>:INFO: Waiting for remaining child processes to terminate...(Press "Ctrl-C" to force kill)
[2542] orig.Server<Server:17c06>:OUTPUT: Heartbeat timeout, transitioning to Candidate state.
[2549] orig.Server<Server:17c06>:OUTPUT: Transitioning to Leader.
[3898] orig.Server<Server:17c06>:OUTPUT: LogEntry:1:<Client:17c09>:0  at index 1 applied to state machine.
[3899] orig.Client<Client:17c09>:OUTPUT: Request 1 complete.
[3899] orig.Server<Server:17c06>:OUTPUT: LogEntry:1:<Client:17c08>:0  at index 2 applied to state machine.
[3900] orig.Client<Client:17c08>:OUTPUT: Request 1 complete.
[3901] orig.Server<Server:17c05>:OUTPUT: LogEntry:1:<Client:17c09>:0  at index 1 applied to state machine.
[3901] orig.Server<Server:17c03>:OUTPUT: LogEntry:1:<Client:17c09>:0  at index 1 applied to state machine.
[3901] orig.Server<Server:17c06>:OUTPUT: LogEntry:1:<Client:17c07>:0  at index 3 applied to state machine.
[3902] orig.Client<Client:17c07>:OUTPUT: Request 1 complete.
.
.
.
.
[6923] orig.Client<Client:17c07>:OUTPUT: Request 3 complete.
[6926] orig.Server<Server:17c04>:OUTPUT: LogEntry:1:<Client:17c07>:2  at index 9 applied to state machine.
[6927] orig.Server<Server:17c02>:OUTPUT: LogEntry:1:<Client:17c07>:2  at index 9 applied to state machine.
[6926] orig.Node_<Node_:b2c01>:OUTPUT: All clients done.
[6929] orig.Server<Server:17c05>:OUTPUT: LogEntry:1:<Client:17c07>:2  at index 9 applied to state machine.
[6929] orig.Server<Server:17c03>:OUTPUT: LogEntry:1:<Client:17c07>:2  at index 9 applied to state machine.
[6929] da.api<MainProcess>:INFO: Main process terminated.

Curious as to what it was doing, I profiled the execution in the instruments provided by Xcode.

The process spawns three threads (processes in an instance of Raft) and tries the leader election protocol between them. In the execution, we can see the distribution of work between threads, with more weight associated with more work, the particular leader of that round. The Raft process updates state machines with the committed entries.

Sadly, the comments do not tell us anything about the code and the execution. At best it seems to be a proof of concept that Raft could be implemented in DistAlgo. One interesting thing is that the implementation ends up being ~240 lines of code, with the expanded Python being around 360 lines. This is in contrast to the much shorter ~100-line implementations of the Paxos protocols. It is not completely clear why the Raft implementation is so much longer, but based on my interpretation of the code, the leader election code and changes to allow support for “idle mode” increase the length of the code. The code also combines several checks into single conditions, resulting in dense code like:

  if await(currentRole is not Leader):
            return
        elif some(n in range(len(log) - 1),
                  has= (n > commitIndex and
                        len(setof(i, i in matchIndex, matchIndex[i] >= n)) >
                        len(peers) / 2 and
                        log[n].term == currentTerm)):
            debug("Updating commitIndex from %d to %d" % (commitIndex, n))
            commitIndex = n
        # Idle timeout is half of normal term timeout:
        elif timeout(termTimeout/2):
            debug("Idle timeout triggered.")
            has_idled = True

Closing Thoughts

When I started writing this post, the area of DSLs for distributed systems seemed very promising. However, the state of the art turns out to be a bit demoralizing. The DSLs mentioned in this post include fascinating ideas that could change how we program distributed systems, but all of them are buggy and incomplete prototypes. None of the languages mentioned at the start of this post have advanced beyond the proof-of-concept stage.

If you want to use consensus in your system today, it would be better to go with battle-tested implementations in general-purpose languages, as verbose and complex as they might be. We need more researchers to look into the space of performance- and scalability-oriented DSLs for expressing distributed algorithms, and I hope the projects discussed in this post mature beyond the prototype stage and become strong contenders.

Implementing a Garbage-Collected Graph CRDT (Part 2 of 2)

2018-12-08T00:00:00+00:00

by Austen Barker · edited by Sohum Banerjea and Lindsey Kuper

Background & Recap

In my previous blog post, we discussed Conflict-Free Replicated Types (CRDTs), a class of specialized data structures designed to be replicated across a distributed system. We implemented a few CRDTs as described in Shapiro et al.’s A Comprehensive Study of Convergent and Commutative Replicated Data Types.

CRDT implementations often run into the issue of distributed garbage collection, a perennial problem in distributed systems. My previous post discussed multiple challenges involved with implementing distributed garbage collection: high metadata storage costs, fault intolerance, and the need for synchronization, which runs counter to the asynchronous nature of a CRDT.

To tackle these problems, of research avenues, including Paxos Commit and Two-Phase Commit, delta state CRDTs, and various methods for reducing metadata overhead in Shapiro et al.’s garbage collection scheme. We explore these different garbage collection mechanisms in the context of our two-phase set (2P-Set) CRDT implementation.

Delta State CRDTs

Almeida, Shoker, and Baquero proposed delta state CRDTs, or delta-CRDTs for short, which help to avoid the issue of sending the entire state of a data structure over a network. The potentially large message sizes involved with classical state-based CRDTs result in them only being practical for small objects such as counters. A solution to this problem is to transmit a delta that encompasses only changes made to a replica. Not only do delta-CRDTs address the problem of large message sizes, they also give us an efficient way to enable asynchronous garbage collection.

In delta-CRDTs, deltas use the same join operations as traditional state-based CRDTs to apply updates to a local state, with the additional ability to join multiple deltas into a group. Temporally ordered deltas can also be joined into groups called delta-intervals which allow the programmer to transmit and join deltas in batches and assist in establishing a causal ordering of events. Almeida, Shoker, and Baquero present two anti-entropy algorithms that respectively ensure eventual consistency and per-object causal consistency of delta-CRDTs.

In the causal-consistency-ensuring anti-entropy algorithm, each node maintains two metadata maps (for keeping track of the sequence of deltas $D$ and the sequence of acknowledgments $A$), and a counter that is incremented each time a delta is joined with the local state. When a node sends a delta-interval to another, the receiving node replies with an acknowledgment after merging the interval into its local state. In practice, this delta-interval is every delta merged into the sending node’s local state after the last acknowledgment from the receiving node. A delta that has been acknowledged by all of a node’s neighbors is then garbage-collected and removed from the map $D$.

In my previous blog post, we saw that for traditional CRDTs, the space complexity for storing the metadata necessary for $N$ nodes is $O(N^2)$ ($N$ vector clocks, each of size $N$, keeping track of the latest received vector clock from each replica). This metadata is needed to show causal relationships between updates, so that we can know when an update is “stable” or received by all replicas.

For delta-CRDTs, Almeida, Shoker, and Baquero show that for delta-CRDTs, the metadata storage cost at each node with $\lvert A \rvert$ neighbors and $\lvert D \rvert$ stored deltas is $O(\lvert A \rvert + \lvert D \rvert)$. That is, instead of the state scaling quadratically with more replicas, it grows linearly with how many neighbors each node is keeping track of and how many deltas have been sent.

Even though the delta-CRDT metadata is only described as a way to enable garbage collection of deltas cached on each node, it can serve an additional purpose. The causal-consistency-enforcing anti-entropy algorithm does pretty much the same thing as the $O(N^2)$ scheme from the previous paper, but to garbage-collect deltas. So, why not garbage-collect deltas and tombstones together? When a delta that contains an operation that created a tombstone is garbage-collected, we can assume that the tombstone is also garbage-collected. We do need to associate tombstones with their deltas, but that can be done with a simple pointer.

To apply this mechanism to our 2P-Set implementation from my previous post, we need only add the acknowledgment map, the delta map, and a counter. We can also re-use the struct Twopset to represent deltas.

type IntMap map[int]interface{}

type Twopset struct{
        addGset         *Gset.Gset
        remGset         *Gset.Gset
        ACK             IntMap //map of acknowledgments from all neighbors
        Deltas          IntMap //map of deltas currently "in flight"
        DeltaCounter    int //incremented to the latest delta merged into the local state
        Interval        int //if this is an interval then this is the earliest counter value encompassed by the interval, -1 if not an interval
}

func Newtwopset() *Twopset{
        return &Twopset{ 
                addGset: Gset.NewGset(),
                remGset: Gset.NewGset(),
                DeltaCounter: 0, //overridden if the Twopset is a delta/interval
                Interval: -1,
        }
}

In practice, it is difficult to perform add/remove operations on the local state and then forward them to other replicas. Instead, we can treat all local operations as deltas that, though immediately merged into the local state, are persisted in the local delta map until they have been acknowledged by all neighbors. This creates a log similar to those in database systems.

Since the causally consistent delta-CRDT metadata allows us to garbage-collect tombstones, we can compare its space usage to the $O(N^2)$ overhead of Shapiro et al.’s approach. If we treat every node as neighboring every other node, the space complexity is $O(N+D)$. Therefore, if we keep the number of deltas in check and run local garbage collection regularly, the metadata overhead is considerably decreased. The $O(N)$ overhead may still prove costly at scale. Therefore, instead of keeping track of each replica, we only keep track of a replica’s immediate neighbors, which effectively caps the maximum metadata size on each node.

There are still a few drawbacks of delta-CRDTs. Firstly, they do not entirely eliminate the metadata overhead problem. Secondly, there still are cases where it is necessary to transmit the entire state, including when we have to send the empty map of deltas, and during crash recovery.

Synchronized Garbage Collection

One way of providing distributed garbage collection for CRDTs is with strong synchronization using a distributed commitment protocol such as Paxos Commit or two-phase commit. In this approach, the system will periodically run a garbage collection operation. The frequency of this operation should be tuned to the rate at which tombstones are created, or the operation can be triggered when the system approaches a maximum number of tombstones.

Under two-phase commit, each replica will vote on whether each tombstone is still necessary. If all the replicas agree that the tombstone is no longer needed, it is garbage-collected when the replicas are notified of the vote’s outcome. The replicas need not vote on each tombstone sequentially, and can submit a list of their tombstones to a coordinator, instead, receiving in return a list of tombstones to delete. The trade-off between this mechanism and the delta-CRDT approach is that while very little extra metadata is needed in order to perform garbage collection, it requires a lot of coordination between nodes, which runs counter to the asynchronous nature of CRDTs. In addition, the need for a coordinator creates a single point of failure and undermines the inherent fault tolerance of CRDTs.

Many implementations of two-phase commit will require additional entities (“transaction managers”) with which the replicas must register in order to perform garbage collection. These additional entities further complicate the programmer’s job, as they must not only account for the CRDT implementation (a non-trivial task, as we saw in my previous post), but also separate synchronization mechanisms.

In the end, whether or not to use synchronization for garbage collection depends on which resources are scarce in the production environment. If local storage space is scarce enough, it may be worth the additional network overhead and compromised availability to explicitly synchronize.

Space-Saving Optimizations

If a CRDT is not expected to have a long lifespan, instead of doing garbage collection it is likely sufficient to simply perform some common space-saving optimizations, in order to minimize the effect of state explosion.

As discussed in my previous post, a tombstone can actually be rather small. In the case of a 2P-Set or an ARPO, the tombstone set can be represented as a bitmap with each bit corresponding to an element in the set being deleted. Using this trick, the storage space needed for tombstones become negligible. This does not eliminate the problem of garbage collection in its entirety, though, as the set can still contain deleted elements after all replicas have marked them as deleted. Therefore this sort of optimization is best applied in conjunction with another garbage collection scheme.

As a programmer, a metadata overhead of $O(N^2)$ gives me pause, but as we saw with delta-CRDTs, it is possible to use less space and still perform distributed garbage collection. There are also other ways of achieving the same goal of $O(N)$ metadata overhead.

Recall that in Shapiro et al.’s approach, the key to garbage-collecting tombstones is to track causal relationships. In a blog post, Peter Bailis discusses work by Bernadette Charron-Bost that showed that $O(N)$ is the lowest timestamp overhead one can achieve while still providing the information necessary to show causal relationships. Bailis suggests various ways to reduce the space costs of vector clocks. The last method he describes is restricting the number of replicas participating, either by putting a total upper bound on the number of participants, or by only requiring a replica to store information about its immediate “neighbors”. This may be one of the optimizations omitted from the delta-CRDT paper that the authors nevertheless describe as being “important in practice”.

Conclusion

CRDTs often require maintaining tombstones in order to properly handle deleted elements. Without some kind of garbage collection mechanism, these tombstones cause a CRDT to require unbounded space.

Distributed garbage collection mechanisms are difficult to implement and are costly, whether that cost is in metadata overhead or synchronization. In the former case, we’ve discussed ways in which a programmer can significantly reduce the metadata cost, either through a few simple optimizations or through the use of delta-CRDTs. If a programmer wishes to use a state-based CRDT, it may be prudent for them to instead use a delta-CRDT as it provides a relatively easy solution to implementing garbage collection, among other benefits. In the latter case — synchronisation — there may be situations in which the additional overhead of performing a synchronized operation is considered acceptable.

There are a multitude of environmental factors that can influence the programmer’s choice of CRDT garbage collection mechanism. For instance, if it’s desirable to be able to adjust the frequency of the garbage collection operation based on load, it would be easier to avoid synchronization and choose a mechanism that runs individually at each replica.

The conclusion may be, of course, the same conclusion we drew in the last post: the easiest and best way to tackle this problem may just be to rely on CRDTs that do not require tombstones, or that keep the lifespan of a CRDT short.

Consistency in Cassandra

2018-11-29T00:00:00+00:00

by Natasha Mittal ⋅ edited by Devashish Purandare and Lindsey Kuper

Introduction

Today, popular NoSQL databases like Cassandra, MongoDB or HBase claim to provide eventual consistency and offer mechanisms to tune consistency.

A consistency model is a contract between the distributed data store and its clients, in which if the clients agree to obey rules for ordering the read/write operations, the underlying data store will precisely specify the result of those operations. In the context of Cassandra, consistency “refers to how up-to-date and synchronized a row of Cassandra data is on all of its replicas”.

Under strong consistency, all operations are seen in the same order by all the nodes in the cluster; that is, there must be a global total ordering of all read and write operations. This introduces high latency, as a lot of synchronization is required which hampers availability and scalability. On the other hand, eventual consistency merely guarantees that if no updates are made to a given data item, eventually all replicas will converge and return the last updated value of the data item. It provides lower latency, as there is less synchronization overhead.

Cassandra’s tunable consistency is intended to give clients the flexibility to adjust consistency levels to meet application requirements. Cassandra provides different read and write consistency levels, and users can fine-tune these levels by explicitly modifying Cassandra’s configuration file. The consistency level associated with an operation determines the number of replicas in the cluster that must respond with an acknowledgment for that operation to succeed.

In this blog post, we will go over Cassandra’s consistency levels, Light Weight Transactions (LWTs) which provide serial consistency, some background on vector clocks, and the 2013 Jepsen analysis of Cassandra that revealed a number of consistency-related bugs.

Cassandra’s Model of Consistency

Let’s establish a few definitions before getting started:

RF (Replication Factor): the number of copies of each data item
R: the number of replicas that are contacted when a data object is accessed through a read operation (the read set)
W: the number of replicas that need to acknowledge the receipt of the update before the update completes (the write set)
QUORUM: sum_of_replication_factors/2 + 1, where sum_of_replication_factors = sum of all the replication factor settings for each data center

If R + W > RF, the write set and the read set always overlap, resulting in what the Cassandra documentation describes as “strong” consistency.

The right choices for RF, R and W in this model depend on the application for which the storage system is being used. In a write-intensive application, setting W=1 and R=RF can affect durability, as failures can result in conflicting writes. In read-intensive applications, setting W=RF and R=1 can affect the probability of the write succeeding. To provide strong consistency and fault tolerance for a balanced mix of reads and writes, one should ensure that these two properties hold:

R + W > RF
R = W = QUORUM

For example, a system with configuration RF=3, W=2, and R=2 provides strong consistency.

R + W <= RF is a weaker consistency model, where there is a possibility that the read and write set will not overlap and the system is vulnerable to reading from nodes that have not yet received updates that have already completed on other nodse.

Read Requests in Cassandra

As the Cassandra documentation explains, Cassandra can send three types of read requests to a replica: direct read requests, digest requests, and background read repair requests. A digest request returns only a hash of the data being read instead of the actual data. The purpose of a digest request is to allow quick comparisons of the contents of replicas: if the hashes disagree, then the actual data will disagree as well.

When reading, the coordinator node sends a direct read request to one replica, and a digest request to a number of replicas determined by the read consistency level specified by the client. If all replicas are not in sync, the coordinator chooses the data with the latest timestamp and sends the result back to the client. Meanwhile, a background read repair request is sent to out-of-date replicas to ensure that the requested data is made consistent on all replicas.

The following table shows some of the read consistency levels that Cassandra provides:

Consistency Level	Usage
ALL	highest consistency and lowest availability
QUORUM	strong consistency with some level of failure
LOCAL_QUORUM	strong consistency which avoids inter-datacenter communication latency
ONE	lowest consistency and highest availability

Write Requests in Cassandra

For writes to a row, the coordinator node sends a write request to all the replicas that comprise the write set for that particular row. As long as all the replicas are available, they will get the write request regardless of the write consistency level specified by the client. For a write operation to succeed, the number of replicas required to respond with an acknowledgement is determined by the write consistency level. So, if W = QUORUM and RF = 3, then write request will be sent to all three replicas, but an acknowledgment is expected from any two.

Here are some of the write consistency levels Cassandra provides:

Consistency Level	Usage
ALL	highest consistency and lowest availability
EACH_QUORUM	strong consistency but write fails when a data center is down
QUORUM	strong consistency with some level of failure
LOCAL_QUORUM	strong consistency which avoids inter-datacenter communication latency
ONE	low consistency and high availability
ANY	lowest consistency and highest availability and guarantees that write will never fail

Lightweight Transactions (LWT)

Applications like banking or airline reservations require operations to be appear to be performed at a single, instantaneous, global time. This is linearizable consistency, which is one of the strongest single-object consistency models. Cassandra provides linearizability via lightweight transactions (LWTs).

In the SQL-like Cassandra Query Language, LWTs are used for INSERT, UPDATE, and DELETE operations that are conditioned on an IF or IF NOT EXISTS condition, such as:

INSERT INTO account (transaction_date, customer_id, amount) 
VALUES (2016-11-02, 356, 125.00) 
IF NOT EXISTS

UPDATE account SET amount = 230.00 
WHERE payment_date = 2016-11-02
AND customer_id = 356 
IF amount = 125.00

To synchronize replica nodes for LWTs, Cassandra uses an implementation of the Paxos consensus protocol. There are four phases in this implementation of Paxos: prepare/promise, read/results, propose/accept and commit/ack. Thus, Cassandra makes four network round trips between the coordinator node and other replicas in the cluster to ensure linearizable execution, so performance is affected. In fact, the Cassandra documentation points out that “IF conditions will incur a non-negligible performance cost (internally, Paxos will be used) so this should be used sparingly.”

Prepare/promise is the most time-consuming phase of the Paxos algorithm. The leader node proposes a ballot number and sends it to all the replicas in the cluster. The replicas accept the proposal if the ballot number is the highest it has seen so far and sends back a promise message which includes the most recent proposal it has already received in advance.

If a majority of the nodes promise to accept the ballot number, the leader can then move on to the next phase of the protocol. But if a majority of the nodes sent an earlier proposal with their promise message, the leader must use that value.

If a leader node interrupts a previous leader node, then it must finish the previous leader’s proposal first and then proceed with its own proposal, thereby assuring the desired linearizable behavior. After the commit phase, the value written by LWT is visible to non-LWTs.

The following is a (slightly anonymized) example of a Paxos trace in Cassandra (taken from one of my own Cassandra logs from a system I worked on):

Parsing insert into users (username, password, email ) values ( ‘mick’, ’mick’, ’mick@gmail.com' ) if
not exists; [SharedPool-Worker-1] | 2013-05-12 10:32:12.112000 | 127.0.0.1 | 1125
Sending PAXOS_PREPARE message to /127.0.0.3 [MessagingService-Outgoing-/127.0.0.3] | 2016-08-22 12:38:44.141000
| 127.0.0.1 | 10414
Sending PAXOS_PREPARE message to /127.0.0.2 [MessagingService-Outgoing-/127.0.0.2] | 2013-05-12 12:38:44.144200
| 127.0.0.1 | 10908
Promising ballot fb282190-685c-11e6-71a2-e0d2d098d5d6 [SharedPool-Worker-1] | 2013-05-12 12:38:44.149000 |
127.0.0.3 | 4325
Promising ballot fb282190-685c-11e6-71a2-e0d2d098d5d6 [SharedPool-Worker-1] | 2013-05-12 12:38:52.147000 |
127.0.0.3 | 4325
Promising ballot fb282190-685c-11e6-71a2-e0d2d098d5d6 [SharedPool-Worker-3] | 2013-05-12 12:38:52.166000 |
127.0.0.1 | 35282
Accepting proposal Commit(fb282190-685c-11e6-71a2-e0d2d098d5d6, [lwts.users] key=mick columns=[[] | [email
password]]\n Row: EMPTY | email=mick@gmail.com, password=mick) [SharedPool-Worker-2] |
2013-05-12 12:38:52.199000 | 127.0.0.1 | 67804

There are two consistency levels associated with LWTs:

SERIAL: where the Paxos consensus protocol will involve all the nodes across multiple data centers.
LOCAL_SERIAL: where the Paxos consensus protocol will run on the local datacenter.

Jepsen

Jepsen is an open-source Clojure library, written by Kyle Kingsbury, designed to test the partition tolerance of distributed systems by fuzzing them with random operations. The results of these tests are analyzed to expose failure modes and to verify if the system violates any of the consistency properties it claims to have. The Jepsen project did an analysis of Cassandra in 2013.

As Joel Knighton explains in his talk about how the DataStax team uses Jepsen, a Jepsen test has three key properties:

It is generative: relies on randomized testing to explore the state space of distributed systems
It’s “black box”: observes the system at client boundaries (does not need any tracing framework or apply some code patch in the distributed system to run the test)
It relies on invariants: it checks invariants from the recorded history of operations rather than runtime

Knighton’s talk also covers the Jepsen test data structure:

{:name                    ...| name of the results
 :os                      ...| prepares the operating system
 :db                      ...| configures/starts/stops the database being tested
 :client                  ...| client protocol to interact with database
 :generator               ...| instructs on how to interact
 :conductors{:nemesis  ...}  | interacts with the environment
 :checker              ...}  | looks at and assesses the test run

Finally, Knighton’s talk discusses how a Jepsen test runs. As he explains, in Jepsen an orchestration node has one thread representing each client of the system being tested, and a thread for the nemesis, which is the Jepsen process that injects failures into the system as it runs. The orchestration node connects to several nodes on which the Cassandra cluster is running. Jepsen generates a stream of read and write operations for client threads and crash/corrupt/partition operations for the nemesis thread. The result is a history that shows which operations happened during the test and when. Operations in the history are expressed as windows that show when they began and ended. Finally, Jepsen runs a checker that can determine whether the history is valid according to some metric of correctness. The checker can also produce an artifact to help explain performance characteristics, such as the latency of operations.

Jepsen Analysis of Cassandra

Background: Vector Clocks

Vector clocks are a technique for determining whether pairs of events are causally related in a distributed system. Logical timestamps are generated for each event in the system, and their potential causality (i.e., their happens-before relationship) is determined by comparing those logical timestamps.

 The timestamp for an event is a vector of numbers, with each number corresponding to a process. Each process knows its position in the vector. Each process assigns a timestamp to each event.

For a message send event, the entire vector associated with that event is sent along with the message payload. When the message is received by a process, the receiving process does the following:

Increments the counter for the process’s position in the vector.
Performs an element-by-element comparison of the received vector with the process’s timestamp vector, and sets the process’s timestamp vector to the higher of the values:

for (i=0; i < num_elements; i++) 
    if (received[i] > system[i])
        system[i] = received[i];

To determine if two events are concurrent, their vector timestamps are compared element-by-element.

if each element of V1 timestamp <= each element of V2 timestamp, then V1 causally precedes V2, or
if each element of V2 timestamp <= each element of V1 timestamp, then V2 causally precedes V1, or
if neither of these conditions applies and some elements in V1 timestamp is greater than while others are less than the corresponding elements in V2 timestamp, then the events are concurrent.

The following animation shows an example of vector clocks in a system with three interacting processes:

Conflicting writes in Cassandra

Cassandra uses a last-write-wins (LWW) policy to resolve write conflicts and does not implement vector clocks “for performance reasons”. In this case, if a client A writes x=1, and another client B writes x=2, it is possible that the final value of x can be 1 or 2 depending upon which write comes second (the one with the most recent timestamp).

In order to avoid this problem, Cassandra uses the concept of immutable data. For every update operation for a particular column, a <value,timestamp> pair is added. For example, for a particular column called name, one might have a series of updates, as follows:

name
Mick, 2017-11-23 12:11:23
Micky, 2017-11-24 17:13:45
Micky Lawson, 2017-12-02 09:34:09

When a client makes a read request, a client-specific merge function is applied to all the column values and the desired result is obtained. In the case of equal timestamps (i.e., a tie), the lexicographically greater value is chosen.

For this to happen, two timestamps need to collide, and it would seem to be a rare possibility that two writes would get exactly the same microsecond-resolution timestamp. However, the Jepsen analysis of Cassandra tested this by repeatedly changing a column value and found that 1 row is corrupted per 250 transactions.

total
acknowledged
survivors
acknowledged writes lost!         //writes lost means corrupt data

This is happening because in Cassandra, the time resolution is in milliseconds, not microseconds. The probability of writes conflicting is much higher for millisecond resolution, resulting in much corrupted data.

Session consistency

Since Cassandra uses a last-write-wins policy, the writes are ordered by wall-clock timestamps. One might expect that the “Read Your Writes” and “Monotonic Reads” session guarantees would hold. However, the Jepsen tests of Cassandra introduce clock drifts due to which system clocks are unsynchronized, and the session guarantees no longer hold.

Another issue arises in the event of a leap second, which is “a one-second adjustment that is occasionally applied to civil time Coordinated Universal Time (UTC) to keep it close to the mean solar time at Greenwich, in spite of the Earth’s rotation slowdown and irregularities”. Linux kernel systems handle leap seconds by taking a one-second backward jump. When that happens, the following situation can arise in Cassandra, as Kyle Kingsbury explains:

Say a client writes w1 just prior to a leap second, then writes w2 just after the leap second. Session consistency demands that any subsequent read will see w2–but since w2 has a lower timestamp than w1, Cassandra immediately ignores w2 on any nodes where w1 is visible.

Since system clocks are not monotonic, timestamps alone cannot be used for global total ordering of operations across all the data centers.

Having worked extensively with Cassandra as a backend developer, I can say that these issues are prominent, and occur frequently during the copious amounts of transaction processing. This has forced enterprises to introduce hacks at the application level, thereby increasing complexity and making the application code lengthy.

Bugs that Jepsen analysis found in Cassandra

The Jepsen analysis of Cassandra found numerous issues that challenged Cassandra’s claim to offer linearizability via LWTs. Here I’ll highlight two specific bugs.

WriteTimeoutException when LWT concurrency level = QUORUM

As explained in Cassandra bug 9328 during high contention, the coordinator node “loses track” of whether the value it submitted to Paxos has been applied or not. For instance, in a banking application, the following situation could occur:

Thread 1: Reads account_balance=$0.
Thread 1: Updates account_balance=$0+$100=$100 successfully, but still receives WriteTimeoutException.
Thread 2: Reads account_balance=$100.
Thread 2: Updates account_balance=$100+500=$600 successfully with no WriteTimeoutException.
Thread 1: Tries again and reads account_balance=$600, which is greater than its previous update.

In this case, thread 1 cannot clearly identify whether its update failed or succeeded. It might assume that it failed and try again and add another $100 to the balance.

Incorrect implementation of Paxos

In Paxos, the leader node proposes the highest-number ballot that has been accepted by the nodes. In case no node responds back with a value, the leader then proposes its own value.

But Cassandra’s implementation of Paxos had a bug in which a value already accepted by some nodes could be ignored. As discussed in Cassandra bug 6012, the result is a system that “can mistakenly accept two different values for the same round.”

Since the first analysis of Cassandra by Jepsen in 2013, the DataStax team has adapted Jepsen and further extended it by incorporating new tests to break the new versions of Cassandra, and this has helped to identify critical bugs in the implementation.

Mixing Consistency in a Programmable Storage System

2018-11-21T00:00:00+00:00

by Aldrin Montana · edited by Abhishek Singh and Lindsey Kuper

Introduction

Modern distributed applications call for ways to store and access data using a range of consistency guarantees. Consider a distributed shared log, like that described in the FuzzyLog paper. A log service is useful in many contexts for applications including data management systems and storage systems. The FuzzyLog paper describes the use of log services for funneling updates for data management systems and other services (e.g., metadata and coordination services, filesystem namespaces) within data centers. The authors motivate their work by discussing the benefit of distributing the log across several servers and relaxing constraints on data consistency.

Reasoning about mixed consistency becomes especially challenging where data accessed at one level of consistency is used in the computation for data at a different level of consistency. In FuzzyLog, for instance, updates to the log are marked with colors that enable a variety of consistency choices. FuzzyLog makes operations to a single color across regions causally consistent, while operations to a single color within a region are serializable. Hence, FuzzyLog uses at least two levels of consistency in its implementation.

The need to use a range of consistency levels in the same application motivates the need for language-level support for mixed consistency. Recent language-level abstractions such as QUELEA, IPA, and MixT aim to make programming in a mixed-consistency world safer and easier. QUELEA allows the developer to declaratively specify the consistency guarantees provided by a datastore and the consistency requirements for application-level operations. IPA provides types representing consistency levels that can be placed on abstract data type (ADT) instances, and policies for determining the consistency of a data type at runtime. MixT is a domain-specific language (DSL) that uses information flow analysis to prevent weakly consistent data from influencing strongly consistent data.

In this blog post, we are interested in exploring approaches for specifying a range of consistency guarantees in a programmable storage system. After giving an informal overview of data consistency in distributed systems, I’ll discuss what programmable storage is and how a programmable storage system might support a mixture of consistency levels.

Data consistency

In a distributed system where data is copied between multiple servers, keeping the copies consistent with each other may not always be possible. For instance, consider a small system where each data object has two copies (Copy₁ and Copy₂), distributed over two servers. What if you’re accessing Copy₁ and it is somehow different from Copy₂?

A consistency model defines the ways in which Copy₁ and Copy₂ may disagree. Informally, strong consistency means that all clients agree on the order that operations on a data object appear. Under eventual consistency, on the other hand, copies of a data object may appear different at any point, but given enough time without updates, all copies will converge to the same state. In a hybrid consistency model, such as RedBlue consistency, individual operations on data objects may be strongly consistent (red) or eventually consistent (blue). Red operations must be ordered with respect to each other, while blue operations may be in any order (and must commute).

In some contexts, such as in the MixT and IPA programming models, consistency is considered a property of the data being operated on, rather than a property of the operations themselves. Since ADTs are defined by the operations that can be invoke on them, though, these two points of view are necessarily difficult to disentangle.

Programmable storage

Across all fields of computing, data storage is extremely important. In fact, even abstract models of computation require the concept of tape, as an infinite, contiguous sequence of locations that can store symbols. It likely isn’t surprising that significant improvements in storage devices have huge impacts for many areas of computing. However, hardware improvements have not always come fast enough. As application requirements for storage have grown, storage systems have grown more complex. To accommodate web-scale and high-performance applications, storage systems have become distributed, and spanned many storage devices.

The performance of a storage system has significant impact on the design and implementation of applications that communicate with it – consider HDFS and cloud services such as Amazon’s S3. In the case of HDFS, knowledge of how it partitions files or caches data can help the application programmer make choices to reduce latency or increase throughput. And so, as applications increase in both complexity and concurrency, there is an increasing need to extract better performance from the storage system. In addition to growing application needs, there are also improvements in hardware that storage systems have yet to make use of. Due to reliability needs, storage systems must be well tested and extensively exercised. On the other hand, due to increasing application requirements and the rate at which underlying hardware is improving, it is necessary to iterate quickly, or to periodically re-design various subsystems. Finally, there are a variety of storage devices to tune for, and that can significantly affect software system design and implementation.

Programmable storage is an approach to developing storage interfaces, pioneered by storage systems researchers at UC Santa Cruz, that emphasizes programmability and reusability of storage subsystems to address these challenges. Due to the inherently high reliability expectations of storage systems, the programmable storage approach discourages rewriting storage subsystems or components, because this only invites younger, error-prone code. The intuition is that reusing subsystems of a storage system means that the community supporting these subsystems is larger, and these subsystems are exercised and improved more frequently. The Malacology programmable storage system is an interface built on top of the Ceph storage stack. Malacology abstracts storage subsystems into building blocks that can be combined to more easily build a service on top of the storage system.

Ceph is an open source, distributed, large-scale storage system that aims to be “completely distributed without a single point of failure, scalable to the exabyte level, and freely-available.” Ceph has been part of storage systems research at UC Santa Cruz for over a decade, from the original Ceph paper (2006), the CRUSH algorithm (2006) and the RADOS data store (2007), to Noah Watkins’s recent dissertation on programmable storage built on top of Ceph.

Although Ceph is a distributed, large-scale storage system, it was designed to fill the role of reliable, durable storage. This expectation is common (and preferred) for many applications, especially scientific applications, where the complexity of weaker consistency models is too difficult to work with. This makes Ceph’s support for only strong consistency, via primary-copy replication, reasonable. However, the trade-off between strong consistency and availability or performance is very important for some Dynamo-like applications. For Ceph to support these types of applications, it would need to offer weaker consistency as an option. Recent work on a weak consistency model for Ceph, PROAR, has been published by researchers at the Graduate School at Shenzhen, Tsinghua University.

Further building up our motivating example, we would like to consider extensions to ZLog, an distributed shared log developed on top of Malacology. ZLog is an implementation of the CORFU strongly-consistent shared log. If Ceph (and Malacology on top of it) supported multiple consistency levels, then ZLog could as well. It would be interesting to compare FuzzyLog to a mixed-consistency version of ZLog built on Malacology. Using an approach in the spirit of QUELEA, MixT, or IPA for mixing consistencies would align well with the programmability aspect of programmable storage.

Mixing consistency

For developers working in distributed systems, it can be cumbersome to think about whether the correct consistency guarantees are being satisfied, especially when building on storage systems that support a mixture of consistency levels. In the past few years, there have been a variety of language-level abstractions that support reasoning about mixtures of consistency guarantees.

Inconsistent, Performance-bound, Approximate (IPA) programming

IPA provides a consistency type system that makes consistency models explicit in types. Developers are able to verify that important data types are used at an appropriate consistency level. This type of support from the programming model is useful for developers to be both more efficient and more correct. IPA is motivated by taking a principled approach to the trade-off between consistency and performance.

There is a prototype implementation of IPA in Scala on top of Cassandra. Scala’s powerful type system allows for ergonomically deconstructing consistency types. This approach allows the developer to directly interact with the consistency type of their data, using features such as pattern matching. IPA allows consistency guarantees to be specified as a policy on an ADT in two ways:

Static consistency policies allow specification of a consistency model (e.g., strong, weak, causal) that can be enforced by the data store.
Dynamic consistency policies allow specification of performance or correctness bounds within which to achieve the strongest consistency possible, and which require additional runtime support to enforce.

Static consistency policies can be implemented relatively straightforwardly on top of the data store. For this particular implementation, consistency levels for data store operations are determined via Cassandra’s use of quorum reads and writes.

Dynamic consistency policies are specifications of performance or behavior properties, within which the strongest consistency constraints should be satisfied. More concretely, IPA provides two dynamic consistency types: rushed types and interval types. The dynamic consistency types are out of scope for this blog post, but they are a very interesting aspect of the IPA paper.

MixT: consistency enforced with information flow

MixT is a domain-specific language that aims to keep consistent computations “untainted” by inconsistent computations. Because the use of inconsistent data can weaken computations that are expected to be strongly consistent, MixT takes an information flow approach to preventing unsafe, or unexpected, interactions between computations running at different consistency levels. Furthermore, MixT offers support for mixed-consistency transactions that execute in separate phases depending on the consistency level of the operations therein.

QUELEA: declarative programming over eventually consistent data stores

QUELEA takes a declarative programming approach that allows developers to specify constraints on a data store and on the operations that interact with it. Programmers can annotate operations with contracts that enforce application-level invariants, such as preventing a negative bank account balance. An SMT-based contract classification system analyzes these contracts and automatically determines the minimum consistency level at which an operation can be run, simplifying reasoning about consistency from the developer’s perspective. Further, QUELEA supports transactional contracts even when the backend datastore does not.

Mixing consistency in programmable storage

The typical IO path to Ceph’s storage cluster does not support consistency models other than strong consistency. To support weaker consistency models in our programmable storage system, we would need to build support for a range of consistency models directly in Ceph. That, in turn, would motivate the need for a language-level abstraction to help programmers deal with the resulting “consistency zoo”.

To support an IPA-like system on top of Ceph with as little modification as possible, it would be useful to add a quorum interface to Ceph. Ceph’s RADOS datastore uses OSDs (object storage daemons) for data persistence. By communicating with these OSDs directly, rather than through a RADOS gateway (RGW), it may be possible to provide a quorum interface in Ceph. Understanding the details of OSD communication will be important for understanding whether Ceph can provide a similar interface to IPA as what Cassandra provides.

MixT, being built on top of Postgres, describes an alternate possible mechanism for supporting weak consistency in a programmable storage system. To enable causal consistency on top of Postgres, Milano et al. replicated data over several Postgres servers. Clients were then partitioned to separate servers and each server was configured to use snapshot isolation for transactions. Version numbers for each row allowed operation ordering across servers, and vector clocks used microsecond-resolution wall clock time. If we encode versions and vector clocks in data stored in Ceph OSDs, it may be possible to simply treat each OSD as a replica server as MixT does with Postgres, hence enabling causal consistency in Ceph.

The implementation of QUELEA, like IPA, uses Cassandra as the backend data store. It uses a “bolt-on”-style mechanism implemented in a shim layer on top of the data store to enable causal consistency on top of Cassandra. Because QUELEA requires programmers to specify contracts on operations that interact with the store and ensures that store operations happen at a consistency level that satisfies those contracts, it seems that implementing QUELEA on top of Ceph may benefit from the ability to communicate with Ceph OSDs individually, just like IPA.

Next steps

In a follow-up blog post, I will try to explore mechanisms by which Ceph could support a range of consistency levels. The approaches to investigate will include quorum consistency, bolt-on causal consistency, and MixT’s approach to causal consistency. Once we understand how these mechanisms might fit into Ceph, we can investigate various approaches to mixing consistency.

Conflict resolution in collaborative text editing with operational transformation (Part 1 of 2)

2018-11-20T00:00:00+00:00

by Abhishek Singh · edited by Austen Barker and Lindsey Kuper

Introduction

Anyone who has used a version control system or collaboratively edited a document knows all too well the problems that arise when versions of a document conflict. Suppose Alice and Bob decide to collaborate on a document. Alice creates a new document, shares it with Bob, and starts editing. Meanwhile, Bob edits his copy. If Alice and Bob’s respective copies of the document magically synchronized with each other with zero latency, collaborative text editing would be easy. But if there is any latency between Alice and Bob — perhaps due to a network partition — then their copies of the document may diverge, leading to a conflict that must eventually be resolved.

If Alice and Bob had been writing on paper, they would have to manually create a unified document incorporating both their changes. Collaborative software — encompassing both real-time collaborative editing tools such as Google Docs, and version control systems such as Git, Mercurial, and Subversion — aims to automate as much of this conflict resolution process as possible.

Operational Transformation

Operational transformation is an algorithm first discussed in a 1989 paper called “Concurrency control in groupware systems” by Ellis and Gibbs and intended to allow systems to collaboratively perform a common task. The technique allowed users keep track of operations performed on shared data as a means of keeping track of changes in the data. Additionally, it formalized certain properties functions must have to allow tasks to be performed collaboratively. A more thorough survey of operational transformation was published by Sun and Ellis in 1998. The technology grew beyond Ellis and Gibb’s original definition and now describes a host of architectures, data models and algorithms for building collaborative software systems. For the rest of this post, we will restrict ourselves to operational transformation as described in the original paper, and its use in collaborative text editing.

Operational transformation was popularized by Google in its Google Wave project. Since Google Wave was discontinued, some of the original papers and definitions can be hard to find, but some documents are available via the Wayback Machine. Operational transformation has also made it into Google’s other products, such as Google Drive and Google Docs.

The Google Wave project itself was based on the Jupiter collaboration system. Jupiter was aimed towards simplifying the algorithm created by Ellis and Gibbs by creating a centralized architecture, as opposed to the free-range collaborative system that Ellis and Gibbs drew out in their paper. In our discussion we deal with the decentralized idea as presented in Ellis and Gibbs’ original 1989 paper. To be specific, we look at the Distributed Operational Transformation (dOPT) algorithm and its use in Ellis and Gibbs’ GROVE editor.

To describe the problem in in its most basic form, let’s say we have a document which is being edited by two users, Alice and Bob. Alice creates a local copy of the document with the string abcd and shares it with Bob. She then starts editing the document, and these changes are shared with Bob as shown in the figure below.

Figure 1. Operations received by both Alice and Bob are applied to local data as they are received. This leads to data inconsistencies. (Data index starts from 0.)

In Figure 1, the red arrows denote operations performed by Alice on her local copy of the data and then their subsequent transmission to Bob for synchronizing their data. The purple arrows show operations executed by Bob on his copy of the data and then transmitted to Alice. For example, Alice inserts y at index 0 in the string with the operation INSERT ("y", 0), and Bob inserts x at index 2 with INSERT("x", 2), then deletes the character at index 1 with DELETE(1). Any changes that either Alice or Bob make to their copy of the document is sent over to the other as an operation message.

The problem is that neither user applies the operation to their local data with any consideration of how the other user applied the operation at their end. Clearly, the data on which Alice executes an operation is not the same as the data on which Bob applied the same operation. The problem is that indices used by Alice do not always correspond to the indices used by Bob. This leads to data inconsistencies at both ends when operations are executed by either of them.

A mechanism is clearly needed that will allow Alice to correctly apply the the operations that Bob did on his data, and vice-versa. The dOPT algorithm specifies the properties of a transformation function which could help us transform the operation and indices received from one user and apply it safely on the other user. Next, we will discuss a rudimentary implementation of such a transformation.

An example of collaborative editing using distributed operational transformation (dOPT)

For this blog post, I wrote a simple Go program and a set of test cases which showcase dOPT. The implementation follows the algorithm roughly as stated in the 1989 paper by Ellis and Gibbs.

To see what the code does, let’s begin by walking through the example shown in the figure below. Again, Alice and Bob have a shared document that both start editing. At the end of each edit, the edit operation performed is sent to the other. When a message is received, the indices are recomputed based on some criteria and the correct operation is executed.

Figure 2. Operations received by Alice and Bob are transformed before being applied to local data.

As seen from Figure 2, the following operations are performed by Alice and Bob:

Initially, the data is abcd, inserted by Alice and sent over to Bob. Alice and Bob have the same data and are in a consistent state.
Alice inserts a y at index 0 on her copy of the data that now becomes yabcd. This operation is then sent to Bob as well.
While Alice was performing the insert in step 2, Bob inserts x at index 2 to his copy, which is abcd as of this moment. This happens concurrently with the operation in step 2. The insert modifies Bob’s local copy to abxcd. Bob then sends this operation to Alice.
Bob then receives Alice’s operation from step 2, correctly computes the index of the operation on his copy of the data, and executes the operation on his copy, which is modified to yabxcd.
Bob’s insert operation from step 3 is received by Alice at some point and Alice correctly computes the index for the x to be inserted on her local data. Her local data is now yabxcd — the same as Bob’s.
If this were the end of the editing process, Alice and Bob would both find that their local copies are now consistent. But they don’t stop. They continue editing their copies, and every edit and subsequent transmission to the other keeps their local data consistent with the others.
Bob deletes the character at index 1 from his copy of the data, which was yabxcd, modifying it to ybxcd and sending this information to Alice.
When Alice receives the delete operation from Bob, she correctly computes the indices and modifies her copy to ybxcd.
While Alice is executing the operation in step 8, Bob deletes a character at index 3 of his local data, which is ybxcd, changing it to ybxd and sending this operation over to Alice.
At some point after Alice executes step 8 but before receiving the message from Bob from step 9, Alice inserts f at index 1 to her local copy, modifying it from ybxcd to yfbxcd.
The operation from step 9 is received by Alice, who deletes the character from her index 4, with the data finally becoming yfbxd.
Meanwhile, Bob receives Alice’s insert from step 10 and updates his copy to now read yfbxd.

There are some things that should be noted from both Figure 2 and its explanation above. First, steps 2 and 3 in the above discussion are concurrent. There is no way to know when exactly steps 2 or 3 were executed by either parties. We can however state unequivocally that step 2 happened before step 5 at Alice’s end (and for Bob, step 3 happens before step 4). Thus, at their own ends, operations are executed sequentially, whereas some operations are concurrent when we look at the system as a whole.

Based on Figure 2, we create a set of test cases that will be the basis of further discussion of the process. In my implementation, I make a few assumptions that differ from the dOPT algorithm described in Ellis and Gibbs’ paper:

Operation messages from either sites are received exactly once.
There are exactly two editors in the system: one at Alice’s end and the other at Bob’s end.
My implementation does not use clocks to timestamp operations, so the happens before relationship is established based on message delivery. It is assumed that LOCAL and REMOTE operations happen concurrently.
Operations are processed in the order in which they are seen and executed at a particular site. In our implementation the executed operations are stored in a list OTEditor.Ops.
Unlike the implementation in the paper, we do not assign priorities to an operation. Every operation has equal priority.
An operation is sent to others immediately after it was executed at one particular site. There is no out-of-order delivery of messages.

One of the main issues in this implementation is that we overlook establishing causality between operations. There is an implicit happens before relationship established by the order in which operations are stored in the OTEditor.Ops list. This can easily be broken by packets that arrive out of order, leading to data inconsistency. Suppose Bob performs two operations Op1 and Op2, and Op2 reaches Alice before Op1; this would cause data inconsistency in my implementation of dOPT, since the transformation depends on the previous operation to compute the new indices.

The code below shows test cases based on the operations performed in Figure 2. The comments in the program identify the operations and the generator of the operations.

func TestOTEditor_Transformation(t *testing.T) {
    ot := OTEditor { Data: "yabcd", Ops: []Op{

        // OPERATION #1 : ALICE
        {Data: "abcd", Index: 0, Location: LOCAL, Op: INSERT},

        // OPERATION #2 : ALICE
        {Data: "y", Index: 0, Location: LOCAL, Op: INSERT},
        },
    }

    // OPERATION #3 : BOB
    fmt.Println("Operation 3. BOB (remote) insert 'x' at index 2")
    ot.AppendOperation(INSERT, "x", 2, REMOTE)
    assertEquals(ot.Data, "yabxcd")

    // OPERATION #4 : BOB
    fmt.Println("Operation 4. BOB (remote) delete char at index 1")
    ot.AppendOperation(DELETE, "", 1, REMOTE)
    assertEquals(ot.Data, "ybxcd")

    // OPERATION #5 : ALICE
    fmt.Println("Operation 5. ALICE (local) insert 'f' at index 1")
    ot.AppendOperation(INSERT, "f", 1, LOCAL)
    assertEquals(ot.Data, "yfbxcd")

    // OPERATION #6 : BOB
    fmt.Println("Operation 6. BOB (remote) delete char at index 3")
    ot.AppendOperation(DELETE, "", 3, REMOTE)
    assertEquals(ot.Data, "yfbxd")
}

Ordering of the operations is implicit in the test cases. Operations are executed from the perspective of Alice in Figure 2, and are processed in the order in which they are seen by Alice.

In the code, an OTEditor is represented by a Data string and a list of operations, Ops, that have taken place on it. There are two supported operations: INSERT and DELETE. LOCAL and REMOTE are constants distinguishing two operation locations; LOCAL operations should be thought of as being performed on a local copy of the data, while REMOTE operations should be thought of as being performed by another user (Bob in this case) on their own copy of the data and sent over as part of the synchronization process. At the end of each operation execution, data at both Alice and Bob’s end is synchronized and is in a consistent state.

Running the tests produces the following output, including log messages produced by calls to AppendOperation:

Operation 3. BOB (remote) insert 'x' at index 2
2018/11/12 15:15:46 Existing Data: {yabcd [{1 abcd 0 0} {1 y 0 0}]}
2018/11/12 15:15:46 New Operation received: {1 x 2 1}
2018/11/12 15:15:46 Executing new operation: {1 x 2 1}, current data: yabcd
2018/11/12 15:15:46 Current value of data: {yabxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1}]}

Operation 4. BOB (remote) delete char at index 1
2018/11/12 15:15:46 Existing Data: {yabxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1}]}
2018/11/12 15:15:46 New Operation received: {2 '' 1 1}
2018/11/12 15:15:46 Executing new operation: {2 '' 1 1}, current data: yabxcd
2018/11/12 15:15:46 Current value of data: {ybxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1} {2 '' 1 1}]}

Operation 5. ALICE (local) insert 'f' at index 1
2018/11/12 15:15:46 Existing Data: {ybxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1} {2  1 1}]}
2018/11/12 15:15:46 New Operation received: {1 f 1 0}
2018/11/12 15:15:46 Executing new operation: {1 f 1 0}, current data: ybxcd
2018/11/12 15:15:46 Current value of data: {yfbxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1} {2  1 1} {1 f 1 0}]}

Operation 6. BOB (remote) delete char at index 3
2018/11/12 15:15:46 Existing Data: {yfbxcd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1} {2  1 1} {1 f 1 0}]}
2018/11/12 15:15:46 New Operation received: {2 '' 3 1}
2018/11/12 15:15:46 Executing new operation: {2 '' 3 1}, current data: yfbxcd
2018/11/12 15:15:46 Current value of data: {yfbxd [{1 abcd 0 0} {1 y 0 0} {1 x 3 1} {2 '' 1 1} {1 f 1 0} {2 '' 4 1}]}

In the paper, a necessary (but not sufficient) condition to ensure that the transformation function works is commutativity of the operations. When dOPT is applied commutativity of operations must be ensured. Additionally, we need to understand if transformation is required. Transformation in the paper is a technique to resolve conflicts. If there are no conflicts, we do not apply any transformations.

First, we need to understand when a transformation is not required. In the above test case, we have three cases where transformation is not required.

During the execution of operation #2 where LOCAL (or Alice) inserts y. These are LOCAL operations. Data consistency is ensured because the LOCAL editor is “reading your writes”.
Operation #4 does not require any transformation because the last operation (#3) was also a REMOTE edit. Since there were no other edits performed anywhere between operation #3 and #4, the data is consistent at both Alice and Bob’s end before the execution of operation 4 (check Figure 2).
Operation #5 does not required transformation because execution of #4 synchronized edits at both LOCAL and REMOTE ends.

In all of the three cases, there are no conflicts where a transformation function is needed. The only case where transformation is required is when the last operation was LOCAL and the current operation is REMOTE. This is because the remote user (Bob) may not be aware of the last operation performed by the local user (Alice) when he made the edit. This rule must be taken with a pinch of salt as it is specific only to this implementation as there exist only two editors in the system.

Next, we discuss what commutativity is and how it applies to our test cases. Given two operations A and B, the transformation T generates the following transformations:

B’ := T (B, A)
A’ := T (A, B)

The transformation T is implemented such that B x A’ = A x B’. Here the symbol x in the expression is used to refer to order of operation execution. For example, for operations A and B, A x B means apply operation A, then B. The symbol = in LHS = RHS refers to the fact that data after application of RHS operations equals LHS. This transformation works in the test case shown above. Let’s first consider operation #3:

A := INSERT ("y", 0 ) ; operation #2.
A’ := INSERT ("y", 0 ) ; The transformation is the same A, as operation #1 was LOCAL.
B := INSERT ("x", 2 ) ; operation #3.
B’ := INSERT ("x", 3 ) ; the operation after the transformation.
Data at both ‘remote’ and LOCAL systems before operations A and B is abcd.
B x A’ yields data abxcd and yabxcd.
A x B’ yields data yabcd and yabxcd.
Both sequences of operations lead to the same state after test case #1, so commutativity holds.

Updates by operations #4 and #5 are all conflict-free and do not require transformation. Operation #6 does. To prove commutativity for operation #5 and #6:

A := INSERT ("f", 1 ) ; operation #5.
A’ := INSERT ("f", 1 ) ; operation #4 did not violate consistency , no transformation required.
B := DELETE ( 3 ), operation #6.
B’ := DELETE ( 4 ), operation after the transformation.
Data at both ‘remote’ and LOCAL systems before operations A and B is ybxcd.
B x A’ yields data ybxd and yfbxd.
A x B’ yields data yfbxcd and yfbxd.
Again, both sequences of operations lead to the same state.

The case of concurrent local and remote updates is the only case where conflict resolution is needed, and therefore the cases discussed above ensure that the transformations produced are always commutative.

The index recomputation algorithm

As mentioned in the previous section and discussed in the test case above, indexes are recomputed when operation messages are received by either Alice or Bob. We now discuss how the index recomputation is performed. The index recomputation follows the following rules:

Deletion is performed one-character-at-a-time in this implementation.
We maintain a list containing the previously executed operations. This list maintains the order of the sequence of operations that have been executed.
As discussed previously, transformation is required only when the locations of the current and last operation differs: last operation was LOCAL and current REMOTE, or vice-versa.
When transformation is required, we check the index of current and last operations, and the operation types. Operation types in our implementation can be either INSERT or DELETE.
If currentOperation.Index > lastOperation.Index AND lastOperation.Type == DELETE, currentOperation.Index = currentOperation.Index - 1.
If currentOperation.Index > lastOperation.Index AND lastOperation.Type == INSERT, currentOperation.Index = currentOperation.Index + length (lastOperation.Data).

Indices are recomputed in steps 4 and 5. In both these steps, indices are recomputed only if the index of the current operation is greater than that of the last operation. This is done because only in this case can the current operation alter the consistency of the data.

Suppose the data before the “last operation” executed was abcd and the “last operation” was INSERT (“x”, 3). Data after the execution of the “last operation” is abcxd. Now suppose the “current operation” had an index of 2. Regardless of what exactly the operation is, it is clear that it can be applied without any need for index recomputation, as there is no offset in the index of the “current operation”, whatever that operation might be.

Let us now suppose that (beginning with the same initial data abcd) the “last operation” was DELETE(2). After the execution of the “last operation”, the data is modified to abd. If the “current operation” has an index of 3, it can be seen in this example that there is no index 3 in the data after the execution of the “last operation”. In this case, there is a need to offset the index of the “current operation” before it can be applied. To offset the index, we depend on the type of the current operation. If the “last operation” was an INSERT, the “current operation”’s index must be offset by the length of the data, which was inserted by adding the index of the “current operation” with the length of the data in the “last operation”. If the “last operation” was DELETE, the current index is offset by 1 (as we only allow deletion one character at a time).

Editor implementation

Now that expectations are set, we implement a simplified operational transformation editor. We continue borrowing language from our previous discussion based on Figure 2. The first part of the implementation deals with setting up data structures. We have two main operations: INSERT and DELETE. These operations can be done either locally (by the same process where the program is executing) or could be sent as a synchronization message from a another process ( or remote user). Operations are tagged LOCAL if they are performed by the process executing the program (Alice) and tagged REMOTE if they are sent from another process (Bob). The operations are described by the struct Op, which defines the structure of the operations which the editor (OTEditor) can process. The editor OTEditor itself has two fields: Data, which stores application data, and Ops, which keeps a record of operations that have been processed by the OTEditor.

type Operation string

const (
    INSERT Operation = "1"
    DELETE Operation = "2"
)

type OpLocation int

const (
    LOCAL  OpLocation = 0
    REMOTE OpLocation = 1
)

type Op struct {
    Op    Operation
    Data  string
    Index int
    Location  OpLocation
}

type OTEditor struct {
    Data string
    Ops  []Op
}

The OTEditor has several methods that implement Operational Transformation. The AppendOperation method is called whenever a new operation is performed by the user (either remote or local). All operations are added to the OTEditor.ops slice (think of it as an list). The data is stored in the OTEditor.Data field. When AppendOperation is called, the OTEditor performs a transformation as previously discussed to compute the new value of the index where the data has to be inserted or deleted. This transformation is performed in the performTransformation method.

The AppendOperation and exec methods are straightforward, and we will not spend too much time on them except to note that exec modifies the OTEditor.Data field according to the operation that needs to be performed. Before modifying the Data field, exec calls performTransformation.

func (c *OTEditor) AppendOperation(op Operation, data string, index int, opLocation OpLocation) {
    operation := Op{Op: op, Data: data, Index: index, Location: opLocation}
    log.Printf("Existing Data: %v\n", *c)
    log.Printf("New Operation received: %v\n", operation)
    c.exec(operation)
}

func (c *OTEditor) exec(operation Op) {
    log.Printf("Executing new operation: %v, current data: %s\n", operation, c.Data)

    // Recompute indices for the operation
    c.performTransformation(&operation)

    b := []byte(c.Data)
    status := false
    switch operation.Op {
    case INSERT:
        if len(b) <= operation.Index {
            log.Printf("Cannot perform operation %v, index out of bounds.", operation)
        } else {
            b1 := b[0:operation.Index]
            b2 := []byte(operation.Data)
            b3 := b[operation.Index:]
            var b0 []byte
            finalData := append(b0, b1...)
            finalData = append(finalData, b2...)
            finalData = append(finalData, b3...)
            c.Data = string(finalData)
            status = true
        }

    case DELETE:
        if len(b) <= operation.Index {
            log.Printf("Cannot perform operation %v, index out of bounds.", operation)
        } else {

            b1 := b[0:operation.Index]
            b3 := b[operation.Index:]
            var b0 []byte
            finalData := append(b0, b1...)
            finalData = append(finalData, b3[1:]...)
            c.Data = string(finalData)
            status = true
        }
    }
    if status {
        ops := append(c.Ops, operation)
        c.Ops = ops
        log.Printf("Current value of data: %v", *c)
    }

}

The performTransformation method implements the heart of the application. Combined with the exec method, performTransformation implements the algorithm for recomputing indices and ensuring that operations remain commutative.

func (c *OTEditor) performTransformation(op *Op) {
    l := len(c.Ops)
    lastOp := c.Ops[l-1]

    // Transformation required only when synchronizing user changes.
    if lastOp.Location == LOCAL && lastOp.Location != op.Location {
        if op.Index > lastOp.Index {
            if lastOp.Op == DELETE {
                op.Index -= 1
            } else if lastOp.Op == INSERT {
                op.Index += len(lastOp.Data)
            }
        }
    }
}

We have implemented a very simple collaborative text editor. As we discussed before, one factor that we overlook in this implementation is that of establishing causality between a set of operations. The implementation, therefore, relies on the order on which the operations are received to establish consistency in the system. We will explore how these issues affect systems and how these can be resolved in my next blog post.

What’s in Part 2?

In my next post, we will review the assumptions we made in this post and see how those assumptions can be removed. We will then develop a more generalized technique for distributed operational transformation and see how we can build a better collaborative editor.

Manufacturing Consensus: An Overview of Distributed Consensus Implementations

2018-11-19T00:00:00+00:00

by Devashish Purandare · edited by Sohum Banerjea and Lindsey Kuper

Introduction

Consensus protocols, as the name suggests, are a class of techniques by which some number of distributed participants can agree on a single value – for instance, to resolve conflicting values at different replicas of a data object in a replicated data store. Consensus protocols are some of the most complex algorithms in distributed systems. How do we get replicas to agree?

Don’t do it!

The first rule of consensus is “avoid using it wherever possible”. Consensus is expensive in terms of performance, and it is notoriously hard to reason about and implement. If you can avoid it, you should. There are several ways to avoid running a full-blown consensus algorithm to resolve conflicts between replicas:

Skip coordination!: The CALM principle shows us that if we can design our program so that “adding things to the input can only increase the output”, we can reliably guarantee eventual consistency as long as all the updates are sent to each replica at least once. The Bloom programming language makes use of this idea.
Exploit commutativity and idempotence!: CRDTs (Conflict-free Replicated Data Types) are built around the notion of monotonic growth with respect to a lattice. For instance, if you maintain a replicated set that can only grow by means of the set union operation, every replica of the set will eventually converge. There are several challenges with implementations of CRDTs, including “state explosion” (the state maintained for each replica can grow exponentially, to the detriment of storage and performance), complexity, and garbage collection. Austen’s previous blog post goes into more depth about the specifics.
Talk is cheap; when in doubt, ask around! Gossip or epidemic protocols are widely used to resolve conficts between replicas. Introduced in a 1987 Xerox PARC paper, gossip protocols make progress by sharing update information between replicas. Replicas contact other, randomly chosen replicas and compare contents. Any differences found are fixed by comparing timestamps, or using an application-specific conflict resolution approach. Rumor-mongering is used to maintain hot data changes as special values, until a certain number of replicas reflect them.

Reasons to avoid coordination and consensus when possible

It is slow: Consensus requires a significant overhead of message passing, locking, voting, and the like until all replicas agree. This is time- and resource-intensive, and can be complicated by failures and delays in the underlying network.
Partial failures: One of the biggest issues with distributed systems is that partial failures can happen, leading to arbitrary message drops, while the system continues to work. Partial failures are notoriously hard to detect, and can compromise the correctness property of any protocol not designed explicitly to deal with them.
It is impossible!!: In their Dijkstra-Award-winning paper “Impossibility of Distributed Consensus with One Faulty Process”, Fischer, Lynch, and Patterson proved that in an asynchronous network environment, it is impossible to ensure that distributed consensus protocols will succeed – the famous FLP result. A failure at a particular moment can void the correctness of any consensus algorithm. As disheartening as this seems, there’s a way around it. FLP assumes deterministic processes, and by making our algorithms non-deterministic (as in “Another Advantage of Free Choice: Completely Asynchronous Agreement Protocols”) can guarantee that a solution is generated “as long as a majority of the processes continue to operate.”

Then why do we need consensus?

If consensus is so terrible, why are we still using complex techniques to achieve it?

Sometimes, eventually consistent isn’t good enough! Strong consistency conditions such as linearizability and serializability require not only that replicas’ final states agree, but that clients cannot observe different intermediate states on the way there. Ensuring that these conditions hold sometimes leaves us with no choice but to use coordination and consensus protocols.

How do consensus protocols work?

It is really hard! Ask the experts! The abstract of Paxos Made Simple (rather infamously) states:

The Paxos algorithm, when presented in plain English, is very simple.

We will discuss why it can be hard to implement nevertheless.

Listen and Learn: Most consensus protocols involve similar steps:

In an initial phase, replicas pick a Leader that they are going to listen to. The Leader is picked by a majority of nodes using an election-like process.
Leaders propose values and changes.
These values are broadcast to Listeners, who select a correct value based on heuristics, such as a minimum quorum.
The rest of the replicas do not play part in picking a winner; they are told the winning value and update themselves to reflect it.

We’ll discuss two popular consensus protocols, Paxos and Raft.

Implementing Paxos

Leslie Lamport’s paper “The Part-Time Parliament” introduced the world to the Paxos algorithm – or it would have, had anyone understood it. Lamport followed up with “Paxos Made Simple” after realizing that the original paper was a little too Greek for the target audience. Since then, there have been numerous variations on, and optimizations of, the original Paxos algorithm.

Paxos Made Simple

The “Paxos Made Simple” paper states three invariants for correctness:

Only a value that has been proposed may be chosen,

Only a single value is chosen, and

A process never learns that a value has been chosen unless it actually has been.

The algorithm breaks down processes into Proposers (P), Acceptors (A), and Learners (L). The following rules govern how acceptors accept proposals:

Each acceptor must accept the first value v that it gets.
After that, the acceptor can only accept any proposal n as long as it has value v and n is greater than its current proposal number.
Acceptors in the majority with the highest proposal number are picked winners.

Proposers start with a prepare request with number n. An accepted prepare request means that the acceptor will never accept any other request with value less than n. The latest accepted value is communicated along with this response to the proposer. Once a majority has accepted a value, it can be communicated as an accept request to the acceptors as well as the Learners.

The paper’s described system has a distinguished proposer and Learner elected by the processes from among themselves. Lamport’s Paxos is simpler when there is a single P; issues arise when there are multiple proposers. But the paper is short on implementation details, and implementing Paxos in practice is nontrivial, as the next paper we’ll look at shows.

Paxos Made Live

“Paxos Made Live”, originally an invited talk at PODC ‘07 from an engineering team at Google, offers us insight into Google’s Chubby system, which implements distributed locking over the GFS filesystem using Paxos. Chubby uses Multi-Paxos, which repeatedly runs instances of the Paxos algorithm so that a sequence of values, such as a log, can be agreed upon. The implementation is similar to the one we just discussed: elect a coordinator, broadcast an accept message, and if accepted, broadcast a commit message. Coordinators have ordering, and restrictions on values, to ensure that everybody settles on a single value.

Because Multi-Paxos can leave behind some replicas (say, in a network partition), they design a catch up mechanism, using a technique not too different from write-ahead logging.

Optimizations

If you can ensure that the leader doesn’t crash and network doesn’t cause arbitrary delays, the authors suggest chaining updates across the Paxos instances, as there’s only a small chance of network changes between subsequent transfers. This can eliminate doing the propose (prepare) phase for each Paxos instance, saving a lot of time.

You can add bias to the coordinator-picking process to prefer a single coordinator, and skip the election process or avoid having competing coordinators in a majority of cases, saving time.

Engineering Challenges

My favorite part of “Paxos Made Live” is the engineering challenges the authors encountered while implementing Paxos on a real system for Chubby, including:

Hard disk failures: These can be addressed by converting the coordinator into a non-voter and using the logging mechanism to track everything until the next instance of Paxos picks up.
Who’s the boss?: The replicas can elect a new coordinator without notifying the present coordinator, causing issues. They solve the problem by adding leases on each coordinator’s duration, during which other coordinators cannot be elected.

“Paxos Made Live” glosses over their handling of the group membership problem:

While group membership with the core Paxos algorithm is straightforward, the exact details are non-trivial when we introduce Multi-Paxos, disk corruptions, etc. Unfortunately the literature does not spell this out, nor does it contain a proof of correctness for protocols related to group membership changes using Paxos. We had to fill in these gaps to make group membership work in our system. The details – though relatively minor – are subtle and beyond the scope of this paper.

How did they solve this? We may never know.

Paxos Made Moderately Complex

Much like “Paxos Made Simple”, “Paxos Made Moderately Complex” implements a much more complex system than what the name implies. The authors implement Multi-Paxos and go into great depth about their implementation and optimizations to it, both as pseudocode and as C++/Python code (which is available online). The authors discuss several Paxos variants, concluding that Paxos is more like a family of protocols rather than a single implementation.

The authors introduce a new term, slots, to reason about Paxos implementation. Slots are commands for each replica to drive its state towards the right direction. In case different operations end up in the same slot across different replicas, Paxos can be used to pick a winner. The slots are not dissimilar to a write-ahead log discussed in other approaches. Confirmed operations in slots can then be put in slot\_out to be communicated to other replicas. The system then describes several invariants over the slots and replicas to ensure the Multi-Paxos algorithm remains safe and live.

The authors also introduce the notions of commanders and scouts, commanders being analogous to coordinators. The idea of scouts is interesting: scouts just act during the prepare phase, ensuring enough responses before passing the data back to their leaders. Leaders spawn scouts and wait for an adopted message from them, entering a passive mode loop while scouts try to get a majority response. Once in the active mode, the leaders get consensus to decide which commands are agreed upon in which slot (only one per slot) and use the commanders to dispatch the commands in the second phase to all the replicas.

This paper goes in-depth discussing the original Synod protocol — i.e., the subprotocol of Paxos that implements consensus, as opposed to handling state machine replication — and how it is limited by practical considerations.

The authors of “Paxos made Moderately Complex” also have a great resource which goes in depth about Paxos: paxos.systems.

Paxos Made Pragmatic

The authors of “Paxos Made Moderately Complex” note that satisfying the aforementioned invariants leads to rapid growth in the state that has to be maintained, as we keep all incoming messages at each slot along with the winner. The authors provide a few ways to optimize the implementation and actually make it practical:

Reduce state: The system can reduce state maintained by keeping only the accepted value for each slot. This can cause issues if there are competing leaders or crashes, as agreed-upon values may be different for the same slot.
Garbage collection: Once values are agreed upon by a majority of replicas, it is not necessary to keep the old values of applied slots. However, just deleting them might cause an impression that they are empty and can be filled. The authors address this by adding a variable to the acceptor state to track which slots have been garbage-collected.
Flushing to disk: Periodically, the log can be truncated and flushed to disk, as keeping the entire state in memory is expensive and can be lost in a power failure.
Read-only operations do not change the state, yet need to get data from a particular state. Treating such operations differently can help us avoid expensive operations.

riak_ensemble: Implementing Vertical Paxos in Erlang

The riak_ensemble consensus library implements Vertical Paxos (i.e., Paxos that allows hardware reconfiguration even if it is in the middle of agreement process) in Erlang to create consensus ensembles on top of key-value stores. riak_ensemble borrows ideas from other implementations, such as the leader leases described in “Paxos Made Live”, and allows leadership changes, tracking metadata with the clever use of Merkle trees. This allows creation of ensembles: consensus groups that implement Multi-Paxos on a cluster. The authors discuss their implementation in this video.

Raft

Raft is often touted as a simpler alternative to Paxos. Winner of a Best Paper award at Usenix ATC 2014, Raft promises a way to navigate through the dangerous and scary waters of consensus protocols. The original paper, in fact includes an in-depth user study at Stanford to provide evidence that Raft is easier to understand and explain than Paxos. While it is an open question whether Raft achieved its simplicity goals, it is widely used in a lot of large-scale systems. In this section we will discuss the Raft protocol and some implementations and contrast it with Paxos.

Base Implementation

Raft performs leader elections to pick leaders for each round. The condition for leadership is stronger in Raft than in Paxos, as leaders are the only nodes that handle reads and writes to the log as well as log replication to all the replicas. Conflicting entries in followers’ logs can be overwritten to reflect the leader. The leader election protocol works with the help of a timeout mechanism for sending heartbeats to collect votes. The election phase itself comes with a timeout to reduce conflicts.

Formal verification and implementation of Raft with `vard` and `etcd`

etcd is a lightweight key-value store implemented using Go. It uses the Raft protocol for distributed consensus to manage its replicated log. etcd’s Raft library is also used by other big projects, like “Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, and more”, and is a feature-complete implementation of Raft in Go, including some optional enhancements.

The verification project Verdi base their own key-value store, vard, on the etcd implementation of Raft. They implement the Raft protocol in the Verdi framework and verify it using Coq, a popular formal verification tool; from there, it can be extracted to OCaml and is available to use.

Raft in CockroachDB

CockroachDB is an open source alternative to Google’s Spanner, a highly available distributed store which uses TrueTime, a globally synchronized clock system, to support ACID properties on top of a distributed data store. The big advantage of these stores is that despite being scalable across continents, they allow linearizability and serializability, along with referential integrity. Raft is used extensively in CockroachDB to ensure that replicas remain consistent.

CockroachDB implements Multi-Raft on top of the Raft protocol to allow better scalability. This involves certain changes to how Raft works. It divides replicas into ranges, which locally implement Raft. Each range performs leader elections and other Raft protocol operations. Ranges can have overlapping memberships. Multi-Raft converts each node’s associated ranges into a group for Raft, limiting the heartbeat exchange to once per tick.

Next steps

As we’ve seen, consensus protocols can be hard to understand and implement. Could programming language support for expressing these protocols help? In my next post, we’ll consider languages that are specifically designed for the task of implementing distributed protocols such as consensus protocols, and compare implementations in those languages to the general-purpose language implementations discussed here.