Reliable servers: design and implementation in avalon/C++

Avalon/C++ is a programming language that supports the construction of reliable programs consisting of a set of servers communicating over a network. It provides high-level language support for user-defined data types with customized synchronization and fault-tolerance properties. These data types are encapsulated in servers, and accessed through exported server operations. Avalon/C++ greatly simplifies the programming of these servers by hiding the distributed nature of a server from both the implementor and callers of a server. Avalon/C++ exploits the similarity of servers and classes, making server definition and use look like that for C++ classes. A detailed description is given of a typical Avalon/C++ server, the catalog server, which is used by clients to locate servers.


Introduction
A distributed system consists of multiple computers (called nodes) that communicate through a network.
Distributed systems are typically subject to several kinds of failures: nodes may crash, perhaps destroying local disk storage, and communication may fail, via lost messages or network partitions.Writing reliable programs for distributed systems is difficult and has been the subject of many research projects.Avalon/C+ is the result of one such project.Avalon/C-H-is a programming language that supports the construction of reliable programs consisting of a set of servers communicating over a network.
It provides high-level language support for user-defined data types with customized synchronization and fault-tolerance properties.These data types are encapsulated in servers, and accessed through exported server operations.Avalon/C+ greatly simplifies the programming of these servers by hiding the distributed nature of a server from both the implementor and callers of a server.This paper explores the representation of servers in AvalonXtt.
In the first section, we give a brief description of Avalon/C+ and its run-time environment.A full description of Avalon's server type and its use is the topic of Section 2. The following sections discuss the various mechanisms used by the server representation.
Section 3 describes how clients locate running servers.
Section 4 describes a typical Avalon/C++ server, the catalog server, which is used in locating servers.
A complete description of a user-defined concurrent data type used by the catalog server is given in Section 5.The last section details the current status and future plans for Avalon/Ctt, and discusses related work.

Overview of AvalonEcc
A widely-accepted technique for preserving consistency in the presence of failures and concurrency is to organize computations as sequential processes called transactions.Transactions are atomic, that is, serializable, transactionconsistent, and permanent.Serializability means that transactions appear to execute in a serial order.Transaction-consisfency ("ail-or-nothing") means that a transaction either succeeds completely and commifs, or aborfs and has no effect.Permanence means that the effects of a committed transaction survive failures.
Avalon/C+t provides transaction semantics via atomic objects.Atomic objects ensure the serializability, transactionconsistency, and permanence of the transactions that use their operations.
All objects used by transactions must be atomic.Avalon/Ctt provides a collection of built-in atomic types; users may define their own atomic types by subtyping the built-in types.Avalon/C++ includes a variety of primitives (not discussed here) for creating transactions in sequence or in parallel, and for aborting and committing transactions.Included with these primitives is a mechanism to handle transaction aborts as exceptions.
The most basic class in our hierarchy is recoverable.
It ensures permanence: after a crash, a recoverable object will be restored to a state that reflects all operations performed by transactions that committed before the crash.
Atomic is a subclass of recoverable, specialized to provide two-phase read/write locking and automatic recovery.
Locking is used to ensure serializability, and the automatic recovery mechanism ensures transaction-consistency.
The third, and perhaps most interesting, base class in the hierarchy is subatomic.
While atomic provides a quick and convenient way to define new atomic objects, subatomic provides primitives to give programmers more detailed control over their objects' synchronization and recovery mechanisms.
This control can be used to exploit type-specific properties of objects to permit higher levels of concurrency and more efficient recovery.[7, 31.An AvalonKtt program consists of a set of servers, each of which encapsulates some data objects.
Each server provides concurrent access to a set of objects through exported operations.
A server resides at a single physical node, but each node may be home to multiple servers.Rather than sharing data directly, servers communicate by calling one another's operations.
An operation call is a remote procedure call (RPC) with call-by-value transmission of arguments and results.A server's objects are stable, that is, they survive crashes.
Avalon/Ccc uses the Camelot system [5] to handle operating-system level details of transaction management, inter-node communication, commit protocols, and automatic crash recovery.It also uses light-weight processes (threads), provided by the Mach/Unix [I] operating system, to support concurrency within a server.

Avalon's Server Type
The goal of Avalon's representation of servers is to hide the complexity associated with invoking and processing operation requests.
A server is an encapsulation of some data objects with operations to access the data much like an ordinary Ctt class.Avalon exploits this similarity, making servers look like ctasses, both from the point of view of the server implementor and the server client.Before describing how servers are defined and used in Avalon, it is Useful to explain how a server and client are implemented in the underlying machine.
A server in the underlying machine is a program, running on a single node, that accepts RPC messages containing operation invocation requests.The server's receiver, running as a light-weight process in the server (a thread), unpacks the operation's parameters from the message and calls the server function that implements the operation.
When a server function returns, the results are packaged into the return message sent to the client.To allow concurrency within the server, multiple threads are used to service the requests.Whenever a request arrives, a new thread is started to handle it.
A sewer may contain additional threads to perform background processing independent of operation requests.
A client invokes a server's operations by sending a request to the server and waiting for a reply.If it cannot contact the server or times out waiting for the reply, the innermost transaction will abort, allowing Avalon's transaction abort handler mechanism to be invoked.
The server construct in Avalon hides all of the communication processing inherent in servers.
A server is defined as a special C++ class.
Like any other class, it contains some data declarations and operations which manipulate the data.
The former are the data objects encapsulated by the server arid the latter are the operations that the server exports.
Implementations of the operations are provided in the usual manner, as if the sewer were simply a class rather than a separate process.In particular, Avalon takes care of generating the code that initializes the process as a server, receives and unpacks RPC messages, calls the proper operation (in a new thread), and packages the results into a reply sent back to the client.To use a server, a client program simply includes the server definition and invokes operations on instances of the server, as with any other C+t class.
From the client's perspective, the only difference between a sever and a normal class concerns how instances of the server are obtained.A simple server is shown in Figure 2-  Both the client and server programs include the same server definition.
2A server may not be derived from another class since all servers are, ifl effect, derived from the class rrrvrr-root.Also, no rlas?or server may be derived from a server.
Avalon does not define semantics for derived savers.
server process: server object: server reference: server type name: the process running on a single node that services requests from clients for server operation invocations.
A server process is described by a server definition and the accompanying operation implementations.
an instance of a server definition.
A server object is the data structure in a client that represents a particular running server process of the appropriate type.When a new server object is created, a new server process is also created.Likewise, when a server object is destroyed, the server process is killed.
a C+t reference to a server object.Clients generally use references to server objects, rather than the objects themselves.
This allows multiple references to the same server object.the name used in the server definition.The server type name is used to declare server object and server reference variables.It is also used when locating a server to restrict the search to servers of the appropriate type.server unique name: the name that a server process uses to identify itself for the purposes of RPC communication.
The underlying machine uses a global namespace for these names.

Servers from the client's perspective
From the client's perspective, an Avalon server is an instance of a server definition, a server object.
A client invokes an operation on a server by calling a member function of a server object.Creating a new server object causes a new server process to be started.When a server object is deleted, the server is killed.When there are multiple instances of a server, a client often does not know the unique name of the specific server to use.It may not care which instance to use, or it may wish to select the server based on some attributes particular to one instance of the server (e.g., the machine on which it is running, or the printer it is controlling).The Avalon library provides the locate-server function to allow selections of this sort.This function allows clients to locate a server object based on a list of attributes describing the server.The operation of this function is fully described in Section 3.
All servers inherit from the class server-root.This class provides the data and operations required by all servers.A constructor is provided to start a new server process when a client creates a new server object.
When a server is created, the server's constructor calls this constructor with appropriate parameters.
If a server does not provide its own destructor to gracefully kill a server process, server-root provides a destructor that sends a kill signal to the underlying machine to kill the server process whenever a client destroys a server object.The server-root also provides a place to store the information necessary for the client to communicate with the server process, including the unique name of the server process and its RPC address.Server-root actually provides a number of constructors, reflecting the varying amount of information which may be supplied in order to start any server.Common parameters to the constructors are: the node on which to start the server, an executable to use, and the amount of recoverable memory to allocate for the server.Except for these parameters, the programmer need not know about the server-root class.When looking at a server definition, a programmer sees what looks like a class definition.Invoking an operation on the server is identical to invoking an operation on a Ctt class (as shown in the third line of Figure 2-2).When a client invokes a server operation, however, Avalon takes over and a lot of behind-the-scenes action takes place.
Within a client, Avalon replaces the implementation of each server operation with code that packs the arguments into a message and initiates an RPC call to the server named by the unique name stored in the server-root component.When the RPC call returns, the results are unpacked and form the result of the client's call.Avalon makes similar replacements for the server's constructor and destructor to provide for starting and killing servers.
Since server objects are really just Ctt objects with special operations, they can be manipulated in the same manner as other C-H objects.
In particular, server objects and references to servers can be passed as parameters to and returned as values from functions.
2.2.Servers from the implementor's perspective From a server implementor's perspective, a server is completely specified by its definition and member function implementations.
A simple server definition is shown in Figure 2 These data objects are restored following a failure.It is important to note that, to be properly restored, the data objects must be derived from one of Avalon's three built-in classes recoverable, atomic or subatomic.
Furthermore, they must be implemented to control concurrent manipulation.
For example, the server in Figure 2-l declares an atomic integer.The operations on this type3, assignment and coercion to integer, are implemented to allow multiple readers or a single writer.
The exported operations list the parameters that Avalon must package up for the client and unpack for the server.The implementation of an operation provides the function that is called in the server process when a client invokes the operation.
The communication code is generated by the Avalon compiler.
Private operations are simply functions which can be called within the server from other member functions.
A server's constructor is very much like an operation.It defines the parameters that a client must use when creating a new sewer and provides code to execute when the server is started.The primary difference is that a constructor must also define the parameters that are sent to the underlying machine to start the server process.These parameters are specified as parameters to the parent's constructor (text to the right of the colon in Figure 2-l).
When a client calls a server's constructor, the specified parameters are passed to the constructor for the server-root class (described above) where they are given to the underlying machine.Once the server process is started, the client constructor code invokes the server's constructor operation in the same manner as any other operation.
The main member function provides a place for the implementor to put code which is executed as a background process when the server is started.This function can be used to provide code which needs to be run independently of operation invocations.
A printer server, for example, could 3atcmic-int is one of a number of basic atomic types provided by the Avalon Library.
use main for the code to run the printer.tdain must exist, even if empty, because Avalon uses the existence of a main implementation to determine that the current compilation is for a server, rather than just for a client.
The recover member function is optional, but provides a place for code that will be executed whenever the server is re-started after a failure.

The locate-server Function
In order for a client to invoke operations on a server, it needs to obtain a reference to its server object.Servers are identified in the underlying machine by a unique name.This name, in general, is not known by a client.Furthermore, if there are multiple instances of the server (e.g., printer servers for multiple printers), the client may wish to select a particular server based on some set of attributes.
To provide this service, the Avalon library contains the function locate-server.
This function takes the server's type name and an optional attribute list and returns a reference to a server object matching those attributes.Since locate-server is not specific to a particular server, it returns a reference to a server-root object.This should then be coerced to the appropriate setvefl.Since it is expected that the result will be coerced to the desired server type, it is crucial that locate-server looks only for servers of the appropriate type.Thus, locate-server explicitly requires the server's type name as its first parameter.If any instance of the named server will do, the attribute list may be empty.
Attribute lists are currently very simple structures.They consist of a list of name-value pairs.A future enhancement may allow attribute expressions rather than simply attribute lists.These would allow boolean expressions on attribute values.
Avalon presents the model that creating a server object starts a new sewer.When locating a server, however, it is necessary to physically create a server object in the client.Hiding this creation is a secondary purpose Figure 4-1: The Catalog server definition it uses an Avalon seNer, the catalog server, to find the unique name of the desired seNer and uses a special constructor (provided by server-root) to create and initialize a server object representing the server.This special constructor does not start a new seNer.
It simply allocates the structure.Locate-server then uses a reference to this object as its return value.
Although clients, in general, should not delete server objects obtained from locate-server (killing the server), the server objects created by locate-server have an internal flag set that inhibits the actual de-allocation of the object if deleted.Thus, a client is allowed to lookup a server, delete it, and look it up again.The final result would be a valid reference to a server object.The object, however, may represent a server that is not running.This would become apparent if a server operation were called (aborting the innermost transaction).

The Catalog Server
The catalog server is part of the Avalon runtime environment.
It provides a repository of information about running selvers.It is the job of the catalog seNer to maintain a mapping of seNer attributes to unique names, and to service lookup requests.
The catalog seNer is a good example of a typical Avalon server.It must reliably maintain the server-attribute mappings and provide concurrent access to this database.The seNer, as shown in Figure 4-1, provides operations to check in attributes for a new seNer, modify attributes, and to locate a server that matches a given attribute list.
There is exactly one catalog server.Since it is expected to be used relatively infrequently, we do not expect it to be a bottleneck.However, if experience shows otherwise, we may decide to run one per node in future versions of Avalon.
When a server starts, it must check in its attributes.The required attributes (i.e., type name, unique name and, node) are checked in by the initialization code for starting a server.
The implementor of a seNer may provide additional information in the constructor code for the server.
For eXampfe, the printer server should add the name of the printer it is servicing.
When a client wants to locate a server, the locate-server function adds the server's type name (the first parameter) to the given attribute list, and calls the Catalog operation name.To avoid boot-strapping problems, Avalon ensures that all clients have a reference to the catalog server, which has a fixed unique name.it is given a unique id.This id can later be used to look at and modify the attributes for the entry.In addition to check-in, the Server provides a number of operations to set and query the attributes of a seNer as well as two operations to find a server based on an attribute list.The first form, find, returns the unique id of the described entry.The second form, name, returns the value of the selected entry's unique-name attribute.
This form is equivalent to a find followed by a get-attribute, and is provided since this is a common operation.
Currently the catalog server is implemented using atomic hash tables.
An entry in the catalog is created for each server.
An entry is represented by a small hash table mapping the seryer's attribute names to values.The entries are kept in the hash table servers, keyed by id.The next section describes the atomic hash table implementation.

Atomic Hash Table
The main data type used by the catalog server is a hash table.Since the server's interface operations are expected to run concurrently, concurrent access to the hash tables must be allowed.This section describes an implementation derived primarily from the class atomic.
The hash table is implemented as a fixed length array containing pointers to a linked list (the buckets) of entries which hash to the bucket.Each bucket in the array is an atomic object consisting of three fields: key, value, and next.
The object representing the hash table itself is derived from recoverable.
The hash table provides the following operations: Lookup: Insert: Given a key return the value.
Given a key and a value, add a new entry, or return FALSE if the entry already exists. Alter: Given a key and a value, modify an existing entry, or return FALSE if the entry is not found.

Remove:
Given a key, remove the entry from the hash table, or return FALSE if the entry is not found.

Synchronization Conditions
Concurrency is controlled at the bucket level with the read/write locking provided by Avalon's atomic class.This is the standard multiple readers/single writer concurrency control.At the hash table level, there may be multiple writers as long as each writer is dealing with a unique bucket.The hash table proper does not need locking since, with this implementation, there are never any changes to the hash table structure.All changes occur within the buckets.The operations work as follows: Lookup: acquires a read lock on the appropriate bucket and searches the list of entries in the bucket for the desired key.
Insert,Alter,R-ve: acquires a read lock on the appropriate bucket and searches for the indicated entry.If the entry exists (does not exist), it returns FALSE.Otherwise, it acquires a write lock on the bucket and adds (modifies or removes) the entry.
A successful insert, alter, or remove blocks all readers and writers of that bucket.If unsuccessful, they block only other writers.A lookup blocks only writers.In all cases, once an operation has returned a result, no other transaction will be able to make a change that would contradict the result, until the first transaction commits or aborts.
The bucket remains locked until the end of the transaction.

Implementation
The hash table is built from three types of structures.The top-level structure, shown in Figure 5-1, is the atomic-hashtsble class.This structure contains an array of buckets and a size.The size of the array is set when the hash table is constructed.
This class is only recoverable since none of the operations change the contents except through operations on the buckets.
Since there are no concurrency problems, there is no need for concurrency control.The basic classes do not specify the types for the hash table keys and values.It is expected that a programmer will specialize these classes with the appropriate types.

Figure 2 - 1 :
Figure 2-1: A simple server definition For the remainder of the paper, the following definitions are used: server, server definition: the definition of the server.This looks like a C++ class definition with the keyword server replacing class2.Both the client and server programs include the same server definition.
Figure 2-2: Example of Using Servers For an Avalon program to make use of a server it must first obtain an instance of the appropriate server.As shown in the Figure 2-2, the client may either create a new server object, starting a new server process, or it may, with the locate-server function, obtain a reference to an existing server object representing a running server process.The underlying machine associates an RPC communication address with a server's unique name.A name, however, is not necessarily the most convenient means of identifying a server.When there are multiple

- 1 .
The definition contains the following parts: data objects used by the server are declared in the server definition.

Figure 4 -
Figure 4-l shows the definition of the catalog server.When a new entry is created (with check-in)it is given a unique id.This id can later be used to look at and modify the attributes for the entry.In addition to check-in, the Server provides a number of operations to set and query the attributes of a seNer as well as two operations to find a server based on an attribute list.The first form, find, returns the unique id of the described entry.The second form, name, returns the value of the selected entry's unique-name attribute.This form is equivalent to a find followed by a get-attribute, and is provided since this is a common operation.Currently the catalog server is implemented using atomic hash tables.An entry in the catalog is created for each server.An entry is represented by a small hash table mapping the seryer's attribute names to values.The entries are kept in the hash table servers, keyed by id.The next section describes the atomic hash table implementation.
n, path p, size s, int i):(n,p,s); int get(); // Exported Op. void set(int i); // Exported Op. void recover(); / / (recovervl main0 ; / / (background) 1; The functions hash-tag and newentry are virtual and are expected to be provided by the specialized hash table class.Bash-tag provides a key type specific routine to generate an integer from a key.The specialized version of this function allocates an instance of the appropriate specialization of hashentry rather than the generic one.Each bucket in the hash table is of class atomic-bucket.This class represents an atomic set implemented by a linked list of hashentry%The class inherits from atomic and each operation acquires either a read lock or a write lock on newentry function is called whenever a new hashentry is needed.