Verifying atomic data types

Atomic transactions are a widely-accepted technique for organizing computation in fault-tolerant distributed systems. In most languages and systems based on transactions, atomicity is implemented through atomic objects, typed data objects that provide their own synchronization and recovery. Hence, atomicity is the key correctness condition required of a data type implementation. This paper presents a technique for verifying the correctness of implementations of atomic data types. The significant aspect of this technique is the extension of Hoare's abstraction function to map to a set of sequences of abstract operations, not just to a single abstract value. We give an example of a proof for an atomic queue implemented in the programming language Avalon/C++.

focus on the behavior and correctness of objects in a system and not on the processes (transactions) that manipulate them. We base the proof of correctness of the entire system on a local property of the objects in the system; if the property holds for each object, the correctness of the entire system is guaranteed.
Thus, we transform the problem of proving an entire distributed system correct into the more manageable problem of proving each of the objects in the system correct.
This paper is organized as follows. In Section 2 we present our model and basic definitions, and illustrate most of them through simple examples. In Section 3 we describe three pieces in our verification technique, the most important of which is an extension of Hoare's abstraction function for data implementations. In Section 4, we introduce and motivate relevant Avalon/C++ programming language primitives. We give in Section 5 an extended example using these primitives and a correctness proof following the technique outlined in Section 3. Section 6 discusses related work, in particular contrasting the particular correctness condition we use with another more conventional one and contrasting our extended abstraction function with other kinds of mappings. Finally, we close in Section 7 with a summary of relevant current and future work.

Model for Transaction-Based Distributed Systems A distributed system is composed of a set of transactions and a set of objects. A transaction corresponds
to a sequential process. We disallow concurrency within a transaction, but allow for multiple transactions to execute concurrently. Objects contain the state of the system. Each object has a type, which defines a set of possible values and a set of operations that provide the only means to create and manipulate objects of that type. A transaction can either complete successfully, in which case it commits, or unsuccessfully, in which case it aborts. We use the term termination for the end of the execution of an operation and completion for the end of the execution of a transaction.
Typically, a transaction executes by invoking an operation on an object, receiving results when the operation terminates, then invoking another operation on a possibly different object, receiving results when it terminates, etc. It then commits or aborts.

Histories
We model a computation as a history, which is a finite sequence of events. There are four kinds of

events: invocations, responses, commits, and aborts. An invocation event is written as xop(args*)A,
where x is an object name, op an operation name, args* a sequence of arguments, and A a transaction name. A response event is written as x term(res') A, where term is a termination condition, and res 0 is a sequence of results. We use "Ok" for normal termination. A commit or abort event is written x Commit A or x Abort A, and it indicates that the object x has learned that transaction A has committed or aborted.
A response matches an earlier invocation if their object names agree and their transaction names agree.
An invocation is pending if it has no matching response. An operation in a history is a pair consisting of matching invocation and response events. An operation op 0 lies within op 1 in H if the invocation event for op 1 precedes that of op 0 in H, and the response event for op 1 follows that of op 0 . For histories, we use "•" to denote concatenation, and "A" the empty history.

committed(H) u aborted(H), and active(H) to be the set of transactions in H not in completed(H).
Note that we can model a failure event (e.g,. node crash) with abort events.

Example
The following history, H v involves two queue objects p and q, and four transactions A, S, C, and D:

Q.
Ok() C H 1 shows an example of an atomic (to be formally defined) or intuitively "correct" history. H 1 is correct because there is some ordering on nonaborted transactions that is "equivalent" to a "sequential" version of H 1 and because As effects are ignored. It would have been incorrect for C to dequeue 1 from p since A aborts. If A were to commit instead, then it would be correct either to have C dequeue 1, by ordering A before S, or to have C dequeue a 2, by ordering B before A. Notice that a transaction can perform more than one operation, possibly on different objects. A performs two Enq's on p and B performs one each on p and q. The intuition we would like to capture in our formal definitions is as follows: At the end of H 1 (1) p's first and only element is either 2 (C aborts) or 6 (C commits); and (2) cfs first and only element is 4 (q does not have 5 in it because D*s invocation is pending, yet it definitely has a 4 in it because £Ts commit precedes /7s invocation). 1. The first event of H \ A is an invocation.

2.
Each invocation in H | A except possibly the last, is immediately followed by a matching response or by an abort event.

A transaction can either commit or abort, but not both, i.e., committed(H) r\ aborted(H)
These constraints capture the requirement that each transaction performs a sequence of operations. It cannot invoke one operation on an object x and then another on x (or any other object) without first receiving a response from its first invocation. if isEmp(q) then e else first(q) rest(ins(q, e)) == if isEmp(q) then emp else ins(rest(q), e) isEmp(emp) == true isEmp(ins(q, e)) == false Definition 4: Given a sequential specification of an object, a sequential object history is legal if the state of the object before each invocation event satisfies the pre-condition of the object's invoked operation and the state of the object before each matching response satisfies the corresponding postcondition.
A sequential history H involving multiple objects is legal if it is legal at each object, i.e., each subhistory H | x is legal with respect to the sequential specification for x.

Atomicity = Serializability + Recoverability
We are interested in defining when a history is atomic, i.e., serializable and recoverable. We first define when a history is serializable and then when it is atomic, by adding the recoverability property.

For example, if A 1% A n are transactions in H in the order 7, then Seq(H, T) • H\ A 1 • ... • H\ A n .
Serializability picks off only those equivalent sequential histories that are legal.

Local Atomicity
The only practical way to ensure atomicity in a decentralized distributed system is to have each object perform its own synchronization and recovery. In other words, we want to be able to verify the atomicity of a system composed of multiple objects by verifying the atomicity of individual objects.
However, atomicity as defined so far is too weak a property to let us perform such local reasoning. To ensure that all objects choose compatible serialization orders, it is necessary to impose certain additional restrictions on the behavior of atomic objects. These restrictions let us reason about atomicity locally. Thus, if each object is guaranteed to satisfy a local atomicity property, the entire system will be globally atomic. Avalon/C++ uses a local atomicity properly that Weihl calls hybrid atomicity [36].

commit.
To capture formally the restriction that transactions must be serializable in commit-time order, we make the following adjustments to our model. When a transaction commits, it is assigned a logical timestamp [21], which appears as an argument to that transaction's commit events. These timestamps determine the transactions' serialization order. Commit timestamps are subject to the following well-formedness constraint, which reflects the behavior of logical clocks: if B executes a response event after A commits, then B must receive a later commit timestamp. For a given history H, let TS(H) be the partial order such

Definition 8: A history H is hybrid atomic if H \ committed(H) is serializable in the order TS(H).
Serializability requires only that there exists some total order on transactions in H] atomicity implies we need order only the committed transactions; finally, hybrid atomicity picks an order (commit-time order) for which there must be a legal sequential equivalent. Weihl shows that hybrid atomicity is an optimal local atomicity property: no strictly weaker local property suffices to ensure global atomicity [36]. is atomic, but not hybrid atomic. It is serializable in the order in which 8 precedes A, but not in which A precedes 8.

Objects
Since hybrid atomicity is local, we henceforth need only consider object subhistories.

On-line Atomicity
Since an object may hear about the commitment of transactions out-of-order, it may be difficult for it to choose an appropriate response to a pending invocation of an active transaction. Thus, we focus on "pessimistic" atomicity, where an active transaction with no pending invocation is always allowed to commit. Using this stronger property, called on-line hybrid atomicity, gives us the additional advantage that we can perform inductive reasoning over events in a history, which is not possible using simple hybrid atomicity.

Definition 9: H is on-line atomic if every well-formed history H' constructed by appending wellformed commit events to H is atomic.
We call any sequential history equivalent to H' | committed(H') a serialization of H.

This definition implies that H is on-line atomic if every one of its serializations is legal. We will typically
work with serializations of H, letting us tack on zero, one, or more commit events to H. On-line atomicity allows us to choose to complete any number of active transactions, and thereby introduces inherent nondeterminism into our correctness condition.

Examples
The following history,

is on-line (hybrid) atomic. It has two serializations: one in which B precedes A, and one in which B
precedes A and A precedes C, and it is easily verified that both are legal.
However, the following history,

End examples
In summary, we henceforth consider a history to be atomic if its transactions are serializable in committime order, and to be on-line atomic if the result of appending commit events with well-formed commit timestamps is atomic.

Verification Method
We first define our notion of correctness based on the atomicity property presented in the previous section. We then give a verification method for proving the correctness of implementations of atomic objects.

An implementation is a set of histories in which events of two objects, a representation object r of type
Rep and an abstract object a of type Abs, are interleaved in a constrained way: for each history H in the

implementation, (1) the subhistories H | rand H \ a satisfy the usual well-formedness conditions; and (2) for each transaction A, each representation operation in H | A lies within an abstract operation.
Informally, an abstract operation is implemented by the sequence of representation operations that occur within it.
Our correctness r-~-rion for the implementation of an atomic object is as follows: An object a is atomic if for every history implementation, H | a is atomic. We typically do not require H\ rXobe atomic.
To show the correctness of an atomic object implementation, we must generalize techniques from the sequential domain. We use three "tools" in our method: (1) a representation invariant, (2)

Ser(H)QA(r).
These two properties ensure that every serialization of H is a legal sequential history, and hence that H is on-line atomic. We use the object's sequential specification to help establish the first property. Note that if we were to replace the second property with the stronger requirement that Ser(H) » A(r), then we could not verify certain correct implementations that keep track of equivalence classes of serializations. In the inductive step of our proof technique, we show the invariance of these two properties across a history's events, e.g., as encoded as statements in program text.

Implementing Atomic Objects
Given that atomicity is the fundamental correctness condition for objects in a transaction-based distributed system, how does one actually implement atomic objects? In this section we discuss some of the programming language support needed for constructing atomic objects. We have built this support in a programming language called Avalon/C++ [8], which is a set of extensions to C++ [35].
Essentially, Avalon/C++ provides ways to enable programmers to define abstract atomic types. For example, if we want to define an atomic array type, we define a new class, atomic_array, which perhaps provides fetch and store operations. (Syntactically, a class is a collection of members, which are the components of the object's representation, and a collection of operation implementations.) The intuitive difference between a conventional array type and an atomic_array type is that objects of array type will not in general ensure serializability and recoverability of the transactions that access them whereas objects of atomic_array type will. However, the programmer who defines the abstract atomic type is still responsible for proving that the new type is correct, i.e., that all objects of the newly defined type are atomic. By providing language support for constructing atomic objects, we gain the advantage that this proof is done only once per class definition, not each time a new object is created.
The verification method used for proving that an atomic type definition is correct is the heart of this paper.  Objects defined in a class that inherits from subatomic can also provide commit and abort operations that are called by the system as transactions commit or abort. A user-defined commit typically discards recovery information for the committing transaction, and a user-defined abort typically discards the tentative changes made by the aborting transaction. Intuitively, commit and abort operations in Avalon/C++ are expected to affect liveness, but not safety. For example, delaying a commit or abort operation may delay other transactions (e.g., by failing to release locks) or reduce efficiency (e.g., by

Ensuring Serializability and Recoverability An atomic object in Avalon is defined by a C++ class that inherits from the Avalon built-in class
failing to discard unneeded recovery information), but it should never cause a transaction to observe an erroneous state. We do not address liveness properties in this paper, though certain ones are dearly of great interest. We would need to rely on the extensive work on temporal logic, e.g., [28], for reasoning about liveness.

An Example: A Highly Concurrent FIFO Queue
In this section, we Illustrate our verification technique by applying it to a highly concurrent atomic FIFO queue implementation. Our implementation is interesting for two reasons. First, it supports more concurrency than commutativity-based concurrency control schemes such as two-phase locking. For example, it permits concurrent enqueuing transactions, even though enqueuing operations do not commute. Second, it supports more concurrency than any locking-based protocol, because it takes advantage of state information. For example, it permits concurrent enqueuing and dequeuing transactions while the queue is non-empty.
We first give the Avaion/C++ implementation of the queue, then define the verification tools needed to prove its correctness, and then give a correctness proof.

The Implementation
As in the implementation of any abstract type, we present first the representation of the abstract type and then the implementations of each of the operations.

The Representation
We  Here is the code for deq:

Application of Verification Method
As outlined in Section 3, we need to provide a representation invariant, abstraction function, and sequential specification in order to apply our verification method.

Representation Invariant
The queue operations preserve the following representation invariant. For brevity, we assume items in the queue are distinct, an assumption that could easily be relaxed by tagging each item in the queue with a timestamp. For all representation values r: 1. No item is present in both the deqd and enqd components: where < d is the total ordering on deq_rec's imposed by the deqd stack.
Thus, given an arbitrary state of the queue representation as in Figure 5-

Each operation is tagged with a trans_id (e.enqr, d.enqr, or d.deqr). These trans_id's induce a partial order on the elements of OPS(r, P).
We define Present to take a representation value r and a prefix set P, from which we can define OPS(r t P). The value of Presenter, P) is a set of sequences of (queue) operations where each sequence has the same elements as in OPS(r, P) in a (total) order that extends the partial order of OPS(r, P). A(r)), we must consider all possible past histories that could have gotten the representation to its present state. So, we use Past to generate an infinite set of finite prefixes for a queue history:

Note that A(r) typically includes more histories than Ser(H), the set of serializations of H.
Defining abstraction functions for other implementations of atomic queues and for implementations of other data types follow the same pattern. Since operations are typically tagged by a trans_id, we gather up these trans_id's into a prefix set P, define a type-specific OPS(r, P) (the elements are just individual instances of the operations provided by the type), define a Past function that lets us generate all possible legal prefixes 2 , and finally a Presenter, P) function that lets us turn a set of partially ordered operations into a set of totally ordered ones, i.e., sequences of operations.

Type-Specific Correctness Condition
The sequential queue specification in Figures 2-1 and 2-2  Lemma 11: Let Q = Q 1 • Q 2 be a legal sequential queue history, and let p be a queue

This lemma captures the "prefix" property of a queues behavior. It indirectly characterizes the conditions under which queue operations may execute concurrently; an analogous lemma would be needed for any
other data type to be verified.
To illustrate why we need to possibly reorder operations in a history, consider the following history: q Enq(5) A q Ok() A q Enq(7) A q Ok() A q Deq() A q Ok(5) A q Commit(1:00) A q Enq(1) B q Ok() B q Deq() C q Ok(7) C q Enq(8) C q Ok() C q must permit for B and C to commit, and in either order. Though the sequence of operations that q sees is:

A(r) and that every sequential history in A(r) is legal. First we give an informal summary of our arguments
and then the more formal proofs for the Enq and Deq operations.

and the second conjunct then ensures that x is the first element of ENQ(Q) -DEQ(Q). Together, they imply that DEQ(Q') -DEQ(Q) • x is a prefix of ENQ(Q') -ENQ(Q), hence that Q is legal by Lemma 11.
If a Commit or Abort event carries the accepted history Hto H\ and the corresponding commit or abort

DEQ(H) prefix ENQ(H f ) O
At the point of return, let e * Enq(x). From POST we have that:

Dequeue
Here is the annotated deq operation:

Hybrid Atomicity Revisited
Atomicity has long been recognized as a basic correctness property within the database community [3]. Moreover, in the case when the queue is non-empty, we can permit a dequeuing transaction to proceed concurrently with an enqueuing one. Consider this example, which is a variation of the example drawn in The queue can permit C to perform a Deq operation and even return an element to C because it knows that A has committed and thus it knows what its first element is. Whether B commits or not, C still receives the correct element. Were two-phase read-write locking used, B and C would not be allowed to proceed concurrently because the Enq and Deq operations would both be classified as writers.

More recently, several research projects have chosen atomicity as a useful foundation for general
The use of commit-time serialization distinguishes Avalon from other transaction-based languages and systems, which are typically based on some form of strict two-phase locking. We chose to support commit-time serialization because it permits more concurrency than two-phase locking [36], as well as better availability for replicated data [17]. Because commit-time serialization is compatible with strict two-phase locking, applications that use locking can still be implemented in Avalon/C++. In fact, we optimize for this more traditional case: As an alternative way to build atomic data types, programmers can inherit from another built-in Avalon/C++ called atomic, which provides access to read and write locks.
To summarize the results of this discussion and that in Section 2, the Venn diagram in Figure 6-1 shows the relationship between atomic, hybrid atomic, and "two-phase-locking" atomic histories. Every "twophase locking" history is hybrid atomic, but not conversely; every hybrid atomic history is atomic, but not conversely. The key point of this section is that hybrid atomicity provides more concurrency than "twophase locking." The key point with respect to this paper, however, is that hybrid atomity is local, whereas atomicity is not. In our initial approach to verification we tried to stick to a purely axiomatic approach in our verification method where we relied on Hoare-like axioms to reason about program statements, invariant assertions to reason about local state changes and global invariants, and auxiliary variables to record the states (e.g., program counters) of concurrent processes. In the transactional domain, however, an atomic object's state must be given by a set of possible serializations, and each new operation (or sequence of operations) is inserted somewhere "in the middle" of certain serializations. This distinction between physical and logical ordering is easily expressed in terms of reordering histories, but seems awkward to express axiomatically, i.e., using assertions expressed in terms of program text alone. Though the proofs given in this paper fall short of a pure syntax-directed verification, they could be completely axiomatized by encoding the set of serializations as auxiliary data. Even so, we have found that the resulting invariant assertions are syntactically intimidating and the proofs unintuitive and unnatural.

Other Models for Transactions
Best and Randall [4], and Lynch and Merritt [26]  constructs that include support for user-defined abstract data types.

Current and Future Work
We have applied our verification method to a directory atomic data type, whose behavior is much more complex than a queue's. A directory stores key-item pairs and provides operations to insert, remove, alter, and lookup items given a key. Synchronization is done per key so transactions operating on different keys can execute concurrently. Moreover, we use an operation's arguments and results to permit, in some cases, operations on the same key to proceed concurrently. Our work on specifying and verifying atomic data types and more recently, our work on using machine aids, has led us to explore extensions to our specification language. Two kinds of extensions seem necessary. First, we need a way to specify precisely and formally the synchronization conditions placed on each operation of an atomic object. We propose using a when clause analogous to the when