Inheritance of synchronization and recovery properties in Avalon/C++

The inheritance mechanism of object-oriented languages is used in the new domain of fault-tolerant distributed systems. Inheritance in Avalon/C++ is used to transmit properties, such as serializability and crash resilience, that are of specific interest in distributed applications. Three base classes are presented: resilient, atomic, and dynamic. They are arranged in a linear hierarchy. Examples of derived classes with objects that guarantee desirable fault-tolerance properties are given.<<ETX>>


Inheritance of Synchronization and Recovery Properties
in Avalon/C++

Introduction
The inheritance mechanism provided by a number of object-oriented programming languages (cited in Section 7) is generally thought to be an effective program structuring technique, increasing the modularity and reusability of programs. In these languages, inheritance allows programmers to define types with behavioral properties similar to previously existing types.
In this paper, we show that inheritance can be used effectively in a new application domain, fault-tolerant distributed systems. In this domain, we are concerned not only with the usual desired behavioral properties of types as manifest in the sequential domain, but also with properties that are desired in the presence of concurrency and faults. Thus, we show that inheritance is not only useful for transmitting sequential properties from one type to another (e.g., the UFO property of a stack), but also for transmitting properties like serializability and crash resilience. In particular, we discuss how Avalon/C-H-[Herlihy&Wing 87] uses the C++ [Stroustrup 86] inheritance mechanism to provide support for highly concurrent fault-tolerant applications, via objects that provide their own synchronization and recovery.
1 Fault-tolerant distributed systems require properties that are typically not of interest in the sequential domain. In the presence of concurrency and failures, the data they manage must satisfy application-dependent consistency constraints, which may encompass objects stored at multiple sites in a distributed system. The data must be highly available, that is, highly likely to be accessible when needed. Data must be reliable, that is, unlikely to be lost or corrupted by system failures. Examples of applications that require such properties include airline reservations and electronic banking systems, where incorrect or unavailable data may be extremely expensive.
This paper is organized as follows: Section 2 describes the transaction model used to organize distributed computations, and some relevant features of C++; Sections 3,4, and 5 describe the base Avalon/C++ class hierarchy; Section 6 describes some restrictions on the use of these classes that must be obeyed to preserve their semantic intent; and Section 7 presents related work and a discussion.

Model of Computation
We assume that distributed computation is done at a set of nodes that communicate over a network. Such a system is subject to a number of failures: nodes may fail, perhaps destroying local disk storage, and communications may fail, via lost messages or network partitions.
A widely-accepted technique for preserving consistency in the presence of failures and concurrency is to organize computations as sequential processes called transactions. Transactions are atomic, that is, serializable, recoverable, and permanent. Serializability means that transactions appear to execute in a serial order. Recoverability means that a transaction either succeeds completely and commits, or aborts and has no effect Permanence means that the effects of a committed transaction survive failures. Transactions may be nested.
Avalon/C++ provides transaction semantics via atomic objects. Atomic objects ensure the serializability, recoverability, and permanence of the transactions that use their operations. All objects shared by transactions must be atomic. Avalon/C++ provides a collection of built-in atomic types, and users may define their own atomic types by inheriting from the built-in types.
Sometimes it may be too expensive to guarantee atomicity at all levels of a system. Instead it is often useful to implement atomic objects from non-atomic components [Weihl&Liskov 85]. In Avalon, such components are called resilient objects; they satisfy certain weak consistency properties in the presence of crashes.

Overview of C++
C++ is an object-oriented extension of C [Kernighan&Ritchie 78], designed to combine advantages of C, such as concise syntax, efficient object code, and portability, with important features of object-oriented programming, such as abstract data types, inheritance, and generic functions.
New types are defined in C++ using the class construct A class is declared to contain members. These may be objects, including functions, of any C++ type. Members may either be public or private. Private members may be manipulated only within the class that implements a type; code external to the class {clients of the class) may access only public members. The public/private distinction is often used to provide an abstraction boundary for a type by hiding its representation in private members, and making its operations public members.

C++ supports inheritance.
A class can be derived from another class, and becomes a subclass of that class. Each derived class has a single, explicit superclass. The public members of the superclass become public members of the subclass. The subclass does not inherit the superclass's private members; thus, even in the subclass implementation, the inherited superclass object must be manipulated using its public members. The latest version of C++ (Version 1.2) recognizes that finer control over the visibility of inherited members is sometimes necessary, and adds a new member classification called protected. Protected members are something of a compromise between public and private members: when a class with protected members is inherited, its protected members become private members of the derived class. The Avalon/C++ class hierarchy uses all three kinds of members. This hierarchy is made up of three classes, RESILIENT, ATOMIC, and DYNAMIC, arranged in a linear inheritance tree (DYNAMIC inherits from ATOMIC, which inherits from RESILIENT). The next three sections describe these classes in detail.

RESILIENT Objects
The most basic class in our hierarchy is RESILIENT, RESILIENT provides the means for its derived classes to ensure operation-consistency: after a crash, a resilient object will be recovered in a state that reflects only completed operations; the effects of operations in progress at the time of the crash are never observed. Resilient objects are also permanent: the recovered state of a resilient object is guaranteed to reflect all operations performed by transactions that committed before the crash. Before presenting the class definition for RESILIENT, we first describe our underlying model of storage.

Three-Level Model of Storage
Conceptually, there are three kinds of storage for objects: volatile, non-volatile, and stable. We assume that local storage of nodes of a distributed system is structured as a virtual memory system, where volatile semiconductor memory serves as a cache for memory pages from non-volatile backing store, such as magnetic disk. Resilient objects reside in this local storage of nodes. Since nodes are subject to crashes that destroy all their local storage, if resilient objects are to survive such crashes, they must be written to stable storage, a storage medium that survives crashes with high probability. (Stable storage may be implemented using redundant hardware [Lampson 81] or replication [Daniels 87].) If every resilient object is logged to stable storage after modifying operations are performed on it, then an operation-consistent state may be recovered after a crash by "replaying" the log.
Replaying the log is a sufficient method for restoring a system's state (e.g., it is used by the Argus system These steps assure that resilient objects always appear in an operation-consistent state in non-volatile storage. In the case of node failure, then, where the contents of non-volatile storage are preserved, the states of resilient objects may be recovered much more quickly. Of course, modifications must still be logged to stable storage against the occurrence of a media failure. Avalon uses the Camelot system [Spector et al. 86] to handle operating-system level details of transaction management, such as the write-ahead logging protocol described above. Other Camelot facilities we use include inter-node communication, commit protocols, and crash recovery.  Since PIN and UNPIN should always be called in pairs, Avalon/C++ provides a special control structure, the pinning block, that enforces this constraint The structure pinning (object) <stmnt>;

Class Definition
is equivalent to object .pin () ; <*tmnt>; object.unpin(); with the additional guarantee that the UNPIN will be executed even if <stmnt> causes control to pass outside the block prematurely, e.g., by executing a break or return. If object is omitted in a pinning statement, it defaults to the value this, which refers to the current object in C++.

Using the Class RESILIENT
Consider defining a resilient array of integers, RESJNT.ARRAY. An object of this type should provide normal array operations such as STORE and FETCH, but should do so in an operation-consistent manner. We could implement RES JNT.ARRAY as a subclass of RESILIENT as follows: Now, suppose we have a RES_INT_ARRAY of 100 integers, and we want to add 1 to each element. We can use a loop where each element is fetched, incremented, and stored back into the array. Given the above implementation of STORE we would make 100 calls to each of the PIN and UNPIN operations. Unfortunately, the log write done by UNPIN is expensive, both in terms of space taken on stable storage, which is a scarce resource, and in terms of time.
Clients can choose to avoid this expense by explicitly enclosing the loop in a pinning block. Clients are permitted to call PIN and UNPIN because they are public members of RESILIENT, and are therefore visible to classes derived from RESILIENT. Thus, this loop could be written as: // Pin and log once, rather than 100 times, pinning (a) for (i; i < 100; i++) a. store (i, a. fetch (i) + 1); Here, the PIN and UNPIN calls made by the STORE operation are not expensive because they recognize that their object is already pinned, and return immediately.
In summary, resilient types can be defined as subclasses of RESILIENT. If an operation that modifies a resilient object calls the inherited PIN and UNPIN operations properly, the object will be operation-consistent and permanent.
If a client calls an object's operations many times, as in a loop, performance can be enhanced by enclosing those operations in a pinning block. class atomic: public resilient { protected : void seize(); // Gains short-term lock, void release(); // Releases short-term lock, void pause (); // Releases short-term lock for some period //of time, then regains it. public:

Class Definition
// "Pin" and "unpin" are public, by inheritance from "resilient." virtual void commit (tid) ; // Called after a transaction that operated on // the object commits, virtual void abort (tid); // Called after a transaction that operated on // the object aborts. Objects of class ATOMIC are intended to be atomic, that is, serializable, recoverable, and permanent We will now discuss how the operations of ATOMIC may be used to satisfy these three aspects of atomicity. Permanence is "inherited" from RESILIENT; since ATOMIC is a subclass of RESILIENT, the PIN and UNPIN operations of RESILIENT are public operations of ATOMIC, and may be used in the same way to ensure permanence and operation-consistency.
Atomic objects must be recoverable, that is, the effects of aborted transactions, including those aborted by crashes, must be undone. Recoverability is a stronger property than resilience. Resilience permits the effects of aborted transactions to be observed, while recoverability does not. To implement recoverability, ATOMIC provides COMMIT and ABORT operations. Whenever a top-level transaction commits (aborts), the Avalon system executes the COMMIT (ABORT) operation on all the atomic objects on which the transaction or its descendents have operated.
ABORT operations are also called when nested transactions "voluntarily" abort, ABORT operations usually undo the effects of aborted transactions, while COMMIT operations discard recovery information that is no longer needed.
COMMIT and ABORT are C++ virtual operations: classes derived from ATOMIC may reimplement these operations, and when COMMIT or ABORT are called by the system, the most specific implementation for the object will be called.
(Many other languages would call these generic functions.) Thus, ATOMIC allows type-specific commit and abort processing, which is useful and often necessary in implementing user-defined atomic types efficiently, as described

Using the Class ATOMIC
Consider an example of a user-defined atomic type. We will create the class ATOMJNT_ARRAY, an atomic array of int, that ensures synchronization using two-phase or read/write locking [Eswaran et al. 76]. Below is the interface of a RW_LOCK class that will be used in the representation of ATOMJNT_ARRAY. The type of read/write locking and version stack recovery used in the above example is a very common method of ensuring atomicity. Avalon/C+f provides a special subclass of ATOMIC, called DYNAMIC, which is optimized for this case, and is described in the next section. While DYNAMIC is a convenient method for implementing many atomic types, for some types greater concurrency can be gained by implementing type-specific transaction synchronization and recovery [Weihl&Liskov 85, Herlihy&Wing 87]. These types need to be derived from class ATOMIC directly.

DYNAMIC Objects
DYNAMIC is a subclass of ATOMIC, specialized to provide two-phase read/write locking and automatic recovery.
Locking is used to ensure transaction serializability, and the automatic recovery mechanism "rolls back" the effects of aborted transactions on DYNAMIC objects. (The name dynamic refers to the particular local atomicity property that two-phase locking implements [Weihl 84].) // Note that "commit" and "abort" are inherited from "atomic", and that // "pin" and "unpin" are inherited from "resilient" via "atomic." Classes derived from DYNAMIC should divide their operations into writers and readers, that is, operations that do and do not modify the objects of the class. To ensure serializability, reader operations should call READ_LOCK on entry, and writer operations should call WRITE_LOCK. Note that no short-term mutual exclusion lock on the object is necessary: if any transaction holds a read lock on an object, then no transaction holds a write lock, so all are free to read the object without fear of its being modified as they read it; conversely, if one transaction holds a write lock on an object, no other transaction may hold any type of lock, so it need not fear interference.

Class Definition
These locking mechanisms can be used to ensure serializability; atomicity also requires that we ensure recoverability and permanence, DYNAMIC inherits the PIN and UNPIN operations from RESILIENT (via ATOMIC); these can be used to guarantee permanence and operation-consistency. If these are used properly, the Avalon run-time system guarantees recoverability by performing special abort processing that "undoes" the effects of aborted transactions. Thus, atomic types derived from DYNAMIC need not provide commit or abort operations.

Using the Class DYNAMIC.
We now present another implementation of the ATOMJNTJVRRAY class defined in Section 4. This implementation inherits from DYNAMIC; the contrast in code size and complexity should show the utility of DYNAMIC. transactions obtain locks on the objects they require in some system-wide canonical order. Therefore, clients might want to structure their code so that each transaction obtains all the locks it requires before executing any operations.
They would do this with explicit calls to READ_LOCK and WRTTE_LOCK.
DYNAMIC uses specially optimized facilities provided by the Camelot system, and is therefore quite efficient It is probably appropriate for deriving most atomic types. As noted previously, however, for certain types more concurrency can be allowed by a customized implementation inheriting from ATOMIC instead of DYNAMIC.

Restrictions on Containers
Some types are (conceptually) parameterized over the types of objects that they can contain. In order to preserve the intended meaning of the type, some restrictions are necessary on the types that can be used to instantiate these parameterized container types. 12

Restrictions for Resilience
Let us consider the class RES_ARRAY, a generalization of the RESJNT.ARRAY class that is parameterized over the element type of the array. It is necessary to ask what kinds of objects we can put in RES_ARRAYS, and still maintain resilience of the array object considered as a whole. First, any type that is stored in-line is permissible. An in-line type is any type that contains no pointers. The fundamental types of C++ (char, int, or float) are in-line. A struct whose members are all in-line is in-line. Similarly, a (C++) array whose elements are all in-line is in-line. Note that if a RES.ARRAY has an in-line element type, then logging the array to stable storage will log all the elements as well.
Problems arise when we start to consider pointer types. If we declare A to be a RES_ARRAY of pointers to ints, is A a resilient object? The answer is no, since A[l] points to an int, which is not a resilient object. We could change the value of this int during a transaction, thus conceptually modifying the state of the array, but no record of this modification would ever reach stable storage, allowing permanence to be violated.
Here, then, is a rule for ensuring that a type is resilient: if objects of a type may contain other objects, and the containing type is intended to be resilient, then the type of the contained objects must either be an in-line type, or it must be a pointer to a resilient object This rule ensures that the latest version of a resilient object will be written to stable storage every time an operation that modifies it completes.
The inverse problem occurs when we have a object that is not meant to be resilient, but conceptually contains some resilient object The Camelot system requires that resilient and non-resilient data be allocated in different sections of memory. If we allow a non-resilient object to contain an in-line resilient object, then we must allocate space for the aggregate object in one of these sections of memory. We cannot put it in the non-resilient section of memory since then the resilient object would not be resilient. We also cannot put the object in the resilient section of memory, for a more subtle reason. If we were to allocate memory there, and there were a node crash, the non-resilient part of the object would become meaningless after recovery: the storage allocator would think it had been allocated, although no variables reference it. This type of garbage would build up over time. Therefore, as a rule we forbid non-resilient objects to contain resilient objects *'in-line;'* they can point only to resilient objects.

Restrictions for Serializability
A rule very similar to the rule for resilience applies for serializability. If a container type is intended to ensure serializability of the transactions accessing it, then it should be instantiated either with an in-line type or with a pointer type to another type that ensures serializability. Care must be taken that nested atomic objects do not lead to deadlock. 13

Related Work and Discussion
The use of inheritance to provide resilience and atomicity in Avalon/C-H-is not closely tied to the details of the C++ inheritance mechanism. It could be adapted to inheritance mechanisms in languages such as Smalltalk (2) Avalon provides explicit linguistic support for both hybrid and dynamic atomicity, not just dynamic atomicity; and (3) Avalon uses the Camelot system for its low-level support, thereby gaining the ability to recover efficiently the state of the system's objects from non-volatile storage after node failures. Argus always reconstructs state from the log after crashes.
Dixon and Shrivastava [Dixon&Shrivastava 87] explore the use of C++ for just recoverability. They assume a simpler failure model, where transactions "voluntarily*' abort; crashes are not handled. Crash recovery places more stringent demands on our language. Synchronization primitives are not integrated into their class structure at all.
We are currently implementing Avalon/C++. The implementation will take the form of a preprocessor that produces C++ code. We make extensive use of the Camelot system for low-level transaction support; Camelot, in turn, relies on the Mach operating system [Accetta et al. 86] for memory management, inter-node communication, and lightweight processes.
We have shown how inheritance can be applied to a novel domain: fault-tolerant distributed computing. In doing so, we have used inheritance to transmit properties quite different from the normal behavioral properties transmitted in the serial domain. We believe that the inheritance of properties such as synchronization and recovery is an important paradigm for the extension of object-oriented languages to distributed environments, especially in environments with stringent reliability requirements.