DISTRIBUTED SYSTEMS principles and paradigms Second Edition phần 5 pot

Consequently,when an update operation is performed on one copy, the update should be pro-pagated to all copies before a subsequent operation takes place, no matter atwhich copy that oper

266 SYNCHRONIZA nON CHAP 6

A Ring Algorithm

Another election algorithm is based on the use of a ring Unlike some ring gorithms, this one does not use a token We assume that the processes are physi-cally or logically ordered, so that each process knows who its successor is Whenany process notices that the coordinator is not functioning, it builds an ELEC- TION message containing its own process number and sends the message to' itssuccessor If the successor is down, the sender skips over the successor and goes

al-to the next member along the ring or the one after that, until a running process islocated At each step along the way, the sender adds its own process number tothe list in the message effectively making itself a candidate to be elected as coor-dinator

Eventually, the message gets back to the process that started it all That ess recognizes this event when it receives an incoming message containing itsown process number At that point, the message type is changed to COORDINA- TOR and circulated once again, this time to inform everyone else who the coordi-nator is (the list member with the highest number) and who the members of thenew ring are When this message has circulated once, it is removed and everyonegoes back to work

proc-Figure 6-21 Election algorithm using a ring.

In Fig 6-21 we see what happens if two processes, 2 and 5, discover taneously that the previous coordinator, process 7, has crashed Each of thesebuilds an ELECTION message and and each of them starts circulating its mes-sage, independent of the other one Eventually, both messages will go all the wayaround, and both 2 and 5 will convert them into COORDINATOR messages, withexactly the same members and in the same order When both have gone aroundagain, both will be removed It does no harm to have extra messages circulating;

simul-at worst it consumes a little bandwidth, but this not considered wasteful

6.5.2 Elections in Wireless Environments

Traditional election algorithms are generally based on assumptions that arenot realistic in wireless environments For example, they assume that messagepassing is reliable and that the topology of the network does not change Theseassumptions are false in most wireless environments, especially those for mobile

ad hoc networks

Only few protocols for elections have been developed that work in ad hoc works Vasudevan et al (2004) propose a solution that can handle failing nodes

net-and partitioning networks An important property of their solution is that the best

leader can be elected rather than just a random as was more or less the case in thepreviously discussed solutions Their protocol works as follows To· simplify ourdiscussion, we concentrate only on ad hoc networks and ignore that nodes canmove

Consider a wireless ad hoc network To elect a leader, any node in the

net-work, called the source, can initiate an election by sending an ELECTION

mes-sage to its immediate neighbors (i.e., the nodes in its range) When a node

receives an ELECTION for the first time, it designates the sender as its parent, and subsequently sends out an ELECTION message to all its immediate neigh- bors, except for the parent When a node receives an ELECTION message from a

node other than its parent, it merely acknowledges the receipt

When node R has designated node Q as its parent, it forwards the ELECTION

message to its immediate neighbors (excluding Q) and waits for acknowledgments

to come in before acknowledging the ELECTION message from Q This waiting

has an important consequence First, note that neighbors that have already

selected a parent will immediately respond to R More specifically, if all bors already have a parent, R is a leaf node and will be able to report back to Q

neigh-quickly In doing so, it will also report information such as its battery lifetime andother resource capacities

This information will later allow Q to compare R's capacities to that of otherdownstream nodes, and select the best eligible node for leadership Of course, Q

had sent an ELECTION message only because its own parent P had done so as well In tum, when Q eventually acknowledges the ELECTION message previ-

ously sent by P, it will pass the most eligible node to P as well In this way, thesource will eventually get to know which node is best to be selected as leader,after which it will broadcast this information to all other nodes

This process is illustrated in Fig 6-22 Nodes have been labeled a to j, along with their capacity Node a initiates an election by broadcasting an ELECTION message to nodes band j, as shown in Fig 6-22(b) After that step, ELECTION

messages are propagated to all nodes, ending with the situation shown in Fig 22(e), where we have omitted the last broadcast by nodes f and i: From there on,each node reports to its parent the node with the best capacity, as shown inFig.6-22(f) For example, when node g receives the acknowledgments from its

6-268 SYNCHRONIZATION CHAP 6

Figure 6-22 Election algorithm in a wireless network, with node a as the source.

(a) Initial network (b)-(e) The build-tree phase (last broadcast step by nodes f

and i not shown) (f) Reporting of best node to source.

children e and h, it will notice that h is the best node, propagating [h, 8] to its ownparent, node b. In the end, the source will note that h is the best leader and willbroadcast this information to all other nodes

When multiple elections are initiated, each node will decide to join only one

election To this end, each source tags its ELECTION message with a unique

i-dentifier Nodes will participate only in the election with the highest identifier,stopping any running participation in other elections

With some minor adjustments, the protocol can be shown to operate alsowhen the network partitions, and when nodes join and leave The details can befound in Vasudevan et al (2004)

6.5.3 Elections in Large-Scale Systems

The algorithms we have been discussing so far generally apply· to relativelysmall distributed systems Moreover, the algorithms concentrate on the selection

of only a single node There are situations when several nodes should actually beselected, such as in the case of superpeers in peer-to-peer networks, which wediscussed in Chap 2 In this section, we concentrate specifically on the problem

of selecting superpeers

Lo et al (2005) identified the following requirements that need to be met forsuperpeer selection:

1 Normal nodes should have low-latency access to superpeers

2 Superpeers should be evenly distributed across the overlay network

3 There should be a predefined portion of superpeers relative to the

total number of nodes in the overlay network

4 Each superpeer should not need to serve more than a fixed number ofnormal nodes

Fortunately, these requirements are relatively easy to meet in most peer-to-peersystems, given the fact that the overlay network is either structured (as in DHT-based systems), or randomly unstructured (as, for example, can be realized withgossip-based solutions) Let us take a look at solutions proposed by Lo et al.(2005)

In the case of DHT-based systems, the basic idea is to reserve a fraction of theidentifier space for superpeers Recall that in DHT-based systems each nodereceives a random and uniformly assigned m-bit identifier Now suppose wereserve the first (i.e., leftmost) k bits to identify superpeers For example, if weneed N superpeers, then the first rlog2(N)l bits of any key can be used to identify

these nodes

To explain, assume we have a (small) Chord system with m =8 and k =3.When looking up the node responsible for a specific keyp,we can first decide toroute the lookup request to the node responsible for the pattern

p AND 11100000

270 SYNCHRONIZATION CHAP 6

to see if this request is routed to itself Provided node identifiers are uniformlyassigned to nodes it can be seen that with a total of N nodes the number of

A completely different approach is based on positioning nodes in an

m-dimensional geometric space as we discussed above In this case, assume we need

to place N superpeers evenly throughout the overlay The basic idea is simple: atotal of N tokens are spread across N randomly-chosen nodes No node can holdmore than one token Each token represents a repelling force by which anothertoken is inclined to move away The net effect is that if all tokens exert the samerepulsion force, they will move away from each other and spread themselvesevenly in the geometric space

This approach requires that nodes holding a token learn about other tokens

To this end, La et al propose to use a gossiping protocol by which a token's force

is disseminated throughout the network If a node discovers that the total forcesthat are acting on it exceed a threshold, it will move the token in the direction ofthe combined forces, as shown in Fig 6-23

Figure 6-23 Moving tokens in a two-dimensional space using repulsion forces.

When a token is held by a node for a given amount of time, that node will mote itself to superpeer

pro-6.6 SUMMARY

Strongly related to communication between processes is the issue of howprocesses in distributed systems synchronize Synchronization is all about doingthe right thing at the right time A problem in distributed systems, and computernetworks in general, is that there is no notion of a globally shared clock In otherwords, processes on different machines have their own idea of what time it is.which is then treated as the superpeer Note that each node id can check whether

it is a suoemeer bv looking up

There are various way to synchronize clocks in a distributed system, but allmethods are essentially based on exchanging clock values, while taking intoaccount the time it takes to send and receive messages Variations in communica-tion delays and the way those variations are dealt with, largely determine theaccuracy of clock synchronization algorithms.

Related to these synchronization problems is positioning nodes in a geometricoverlay The basic idea is to assign each node coordinates from an rn-dimensionalspace such that the geometric distance can be used as an accurate measure for thelatency between two nodes The method of assigning coordinates strongly resem-bles the one applied in determining the location and time in GPS

In many cases, knowing the absolute time is not necessary What counts isthat related events at different processes happen in the correct order Lamportshowed that by introducing a notion of logical clocks, it is possible for a collec-tion of processes to reach global agreement on the correct ordering of events In

essence, each event e, such as sending or receiving a message, is assigned a

glo-bally unique logical timestamp C(e) such that when event a happened before b,

C(a) < C(b). Lamport timestamps can be extended to vector timestamps: if

C(a) < C(b), we even know that eventa causally preceded b.

An important class of synchronization algorithms is that of distributed mutualexclusion These algorithms ensure that in a distributed collection of processes, atmost one process at a time has access to a shared resource Distributed mutualexclusion can easily be achieved if we make use of a coordinator that keeps track

of whose turn it is Fully distributed algorithms also exist, but have the drawbackthat they are generally more susceptible to communication and process failures.Synchronization between processes often requires that one process acts as acoordinator In those cases where the coordinator is not fixed, it is necessary thatprocesses in a distributed computation decide on who is going to be that coordina-tor Such a decision is taken by means of election algorithms Election algorithmsare primarily used in cases where the coordinator can crash However, they canalso be applied for the selection of superpeers in peer-to-peer systems

3 One of the modem devices that have (silently) crept into distributed systems are GPS receivers Give examples of distributed applications that can use GPS information.

272 SYNCHRONIZATION CHAP 6

4 When a node synchronizes its clock to that of another node, it is generally a good idea

to take previous measurements into account as well Why? Also, give an example of how such past readings could be taken into account.

5 Add a new message to Fig 6-9 that is concurrent with message A, that is, it neither happens before A nor happens after A.

6 To achieve totally-ordered multicasting with Lamport timestamps, is it strictly

7 Consider a communication layer in which messages are delivered only in the order that they were sent Give an example in which even this ordering is unnecessarily re- strictive.

8 Many distributed algorithms require the use of a coordinating process To what extent can such algorithms actually be considered distributed? Discuss.

9 In the centralized approach to mutual exclusion (Fig 6-14), upon receiving a message from a process releasing its exclusive access to the resources it was using, the coordi- nator normally grants permission to the first process on the queue Give another pos- sible algorithm for the coordinator.

10 Consider Fig 6-14 again Suppose that the coordinator crashes Does this always bring the system down? If not, under what circumstances does this happen? Is there any way

to avoid the problem and make the system able to tolerate coordinator crashes?

11 Ricart and Agrawala's algorithm has the problem that if a process has crashed and does not reply to a request from another process to access a resources, the lack of response will be interpreted as denial of permission We suggested that all requests be answered immediately to make it easy to detect crashed processes Are there any cir- cumstances where even this method is insufficient? Discuss.

12 How do the entries in Fig 6-17 change if we assume that the algorithms can be mented on a LAN that supports hardware broadcasts?

imple-13 A distributed system may have multiple, independent resources Imagine that process

owants to access resource A and process 1 wants to access resource B Can Ricart and

Agrawala's algorithm lead to deadlocks? Explain your answer.

14 Suppose that two processes detect the demise of the coordinator simultaneously and both decide to hold an election using the bully algorithm What happens?

15 In Fig 6-21 we have two ELECTION messages circulating simultaneously While it

does no harm to have two of them, it would be more elezant if one could be killed off.Devise an algorithm for doing this without affecting the operation of the basic election algorithm.

"-16 (Lab assignment) UNIX systems provide many facilities to keep computers in synch,

notably the combination of the crontab tool (which allows to automatically schedule

operations) and various synchronization commands are powerful Configure a UNIX system that keeps the local time accurate with in the range of a single second Like- wise, configure an automatic backup facility by which a number of crucial files are automatically transferred to a remote machine once every 5 minutes Your solution should be efficient when it comes to bandwidth usage.

CONSISTENCY AND REPLICATION

:i.:Animportant issue in distributed systems is the replication of data Data aregenerally replicated to enhance reliability or improve performance One of themajor problems is keeping replicas consistent Informally, this means that whenone copy is updated we need to ensure that the other copies are updated as well;otherwise the replicas will no longer be the same In this chapter, we take a de-tailed look at what consistency of replicated data actually means and the variousways that consistency can be achieved

We start with a general introduction discussing why replication is useful andhow it relates to scalability We then continue by focusing on what consistencyactually means An important class of what are known as consistency models as-sumes that multiple processes simultaneously access shared data Consistency forthese situations can be formulated with respect to what processes can expect whenreading and updating the shared data, knowing that others are accessing that data

as well

Consistency models for shared data are often hard to implement efficiently inlarge-scale distributed systems Moreover, in many cases simpler models can beused, which are also often easier to implement One specific class is formed byclient-centric consistency models, which concentrate, on consistency from the per-spective of a single (possibly mobile) client Client-centric consistency models arediscussed in a separate section

Consistency is only half of the story We also need to consider how

consisten-cy is actually implemented There are essentially two, more or less independent,

273

274 CONSISTENCY AND REPLICATION CHAP 7issues we need to consider First of all, we start with concentrating on managingreplicas, which takes into account not only the placement of replica servers, butalso how content is distributed to these servers.

The second issue is how replicas are kept consistent In most cases, tions require a strong form of consistency Informally, this means that updates are

applica-to be propagated more or less immediately between replicas There are various ter/natives for implementing strong consistency, which are discussed in a separatesection Also, attention is paid to caching protocols, which form a special case ofconsistency protocols

al-7.1 INTRODUCTION

In this section, we start with discussing the important reasons for wanting toreplicate data in the first place We concentrate on replication as a technique forachieving scalability, and motivate why reasoning about consistency is so impor-tant

7.1.1 Reasons for Replication

There are two primary reasons for replicating data: reliability and mance First, data are replicated to increase the reliability of a system If a filesystem has been replicated it may be possible to continue working after one rep-lica crashes by simply switching to one of the other replicas Also, by maintainingmultiple copies, it becomes possible to provide better protection against corrupteddata For example, imagine there are three copies of a file and every read andwrite operation is performed on each copy We can safeguard ourselves against asingle, failing write operation, by considering the value that is returned by at leasttwo copies as being the correct one

The other reason for replicating data is performance Replication for mance is important when the distributed system needs to scale in numbers andgeographical area Scaling in numbers occurs, for example, when an increasingnumber of processes needs to access data that are managed by a single server Inthat case, performance can be improved by replicating the server and subse-quently dividing the work

perfor-Scaling with respect to the size of a geographical area may also require cation The basic idea is that by placing a copy of data in the proximity of theprocess using them, the time to access the data decreases As a consequence, theperformance as perceived by that process increases This example also illustratesthat the benefits of replication for performance may be hard to evaluate Although

repli-a client process mrepli-ay perceive better performrepli-ance, it mrepli-ay repli-also be the crepli-ase threpli-atmore network bandwidth is now consumed keeping all replicas up to date

If replication helps to improve reliability and performance, who could beagainst it? Unfortunately, there is a price to be paid when data are replicated The•

problem with replication is that having multiple copies may lead to consistencyproblems Whenever a copy is modified, that copy becomes different from therest Consequently, modifications have to be carried out on all copies to ensureconsistency Exactly when and how those modifications need to be carried outdetermines the price of replication

To understand the problem, consider improving access times to Web pages If

no special measures are taken, fetching a page from a remote Web server maysometimes even take seconds to complete To improve performance, Web brow-sers often locally store a copy of a previously fetched Web page (i.e., theycacheaWeb page) If a user requires that page again, the browser automatically returnsthe local copy The access time as perceived by the user is excellent However, ifthe user always wants to have the latest version of a page, he may be in for badluck The problem is that if the page has been modified in the meantime, modifi-cations will not have been propagated to cached copies, making those copies out-of-date

One solution to the problem of returning a stale copy to the user is to forbidthe browser to keep local copies in the first place, effectively letting the server befully in charge of replication However, this solution may still lead to poor accesstimes if no replica is placed near the user Another- solution is to let the Webserver invalidate or update each cached copy, but this requires that the serverkeeps track of all caches and sending them messages This, in turn, may degradethe overall performance of the server We return to performance versus scalabilityissues below

7.1.2 Replication as Scaling Technique

Replication and caching for performance are widely applied as scaling niques Scalability issues generally appear in the form of performance problems.Placing copies of data close to the processes using them can improve performancethrough reduction of access time and thus solve scalability problems

tech-A possible trade-off that needs to be made is that keeping copies up to datemay require more network bandwidth Consider a process P that accesses a localreplica N times per second, whereas the replica itself is updated M times per sec-ond Assume that an update completely refreshes the previous version of the localreplica If N «M, that is, the access-to-update ratio is very low, we have thesituation where many updated versions of the local replica will never be accessed

by P, rendering the network communication for those versions useless In thiscase, it may have been better not to install a local replica close toP, or to apply adifferent strategy for updating the replica We return to these issues below

A more serious problem, however, is that keeping multiple copies consistentmay itself be subject to serious scalability problems Intuitively, a collection of

276 CONSISTENCY AND REPLlCA TION CHAP 7copies is consistent when the copies are always the same This means that a readoperation performed at any copy will always return the same result Consequently,when an update operation is performed on one copy, the update should be pro-pagated to all copies before a subsequent operation takes place, no matter atwhich copy that operation is initiated or performed.

This type of consistency is sometimes informally (and imprecisely) referred, to

as tight consistency as provided by what is also called synchronous replication.(In the next section, we will provide precise definitions of consistency and intro-duce a range of consistency models.) The key idea is that an update is performed

at all copies as a single atomic operation, or transaction Unfortunately, menting atomicity involving a large number of replicas that may be widely dis-persed across a large-scale network is inherently difficult when operations arealso required to complete quickly

imple-Difficulties come from the fact that we need to synchronize all replicas Inessence, this means that all replicas first need to reach agreement on when exactly

an update is to be performed locally For example, replicas may need to decide on

a global ordering of operations using Lamport timestamps, or let a coordinatorassign such an order Global synchronization simply takes a lot of communicationtime, especially when replicas are spread across a wide-area network

We are now faced with a dilemma On the one hand, scalability problems can

be alleviated by applying replication and.caching, leading to improved mance On the other hand, to keep all copies consistent generally requires globalsynchronization, which is inherently costly in terms of performance The curemay be worse than the disease

perfor-In many cases, the only real solution is to loosen the consistency constraints

In other words, if we can relax the requirement that updates need to be executed

as atomic operations, we may be able to avoid (instantaneous) global tions, and may thus gain performance The price paid is that copies may not al-ways be the same everywhere As it turns out, to what extent consistency can beloosened depends highly on the access and update patterns of the replicated data,

synchroniza-as well synchroniza-as on the purpose for which those data are used

In the following sections, we first consider a range of consistency models byproviding precise definitions of what consistency actually means We then con-tinue with our discussion of the different ways to implement consistency modelsthrough what are called distribution and consistency protocols Different ap-proaches to classifying consistency and replication can be found in Gray et a1.(1996) and Wiesmann et a1 (2000)

7.2 DATA-CENTRIC CONSISTENCY MODELS

Traditionally, consistency has been discussed in the context of read and writeoperations on shared data, available by means of (distributed) shared memory a(distributed) shared database, or a (distributed) file system In this section, we use

the broader term data store A data store may be physically distributed acrossmultiple machines In particular, each process that can access data from the store

is assumed to have a local (or nearby) copy available of the entire store Write erations are propagated to the other copies, as shown in Fig 7-1 A data operation

op-is classified as a write operation when it changes the data, and op-is otherwop-ise tied as a read operation

classi-Figure 7·1 The general organization of a logical data store, physically

distrib-uted and replicated across multiple processes.

A consistency model is essentially a contract between processes and the datastore It says that if processes agree to obey certain.rules, the store promises towork correctly Normally, a process that performs a read operation on a data item,expects the operation to return a value that shows the results of the last write oper-ation on that data

In the absence of a global clock, it is difficult to define precisely which writeoperation is the last one As an alternative, we need to provide other definitions,leading to a range of consistency models Each model effectively restricts thevalues that a read operation on a data item can return As is to be expected, theones with major restrictions are easy to use, for example when developing appli-cations, whereas those with minor restrictions are sometimes difficult The trade-off is, of course, that the easy-to-use models do not perform nearly as well as thedifficult ones Such is life

7.2.1 Continuous Consistency

From what we have discussed so far, it should be clear that there is no suchthing as a best solution to replicating data Replicating data poses consistencyproblems that cannot be solved efficiently in a general way Only if we loosenconsistency can there be hope for attaining efficient solutions Unfortunately,there are also no general rules for loosening consistency: exactly what can betolerated is highly dependent on applications

There are different ways for applications to specify what inconsistencies theycan tolerate Yu and Vahdat (2002) take a general approach by distinguishing

278 CONSISTENCY AND REPLICA nON CHAP 7three independent axes for defining inconsistencies: deviation in numerical valuesbetween replicas, deviation in staleness between replicas, and deviation withrespect to the ordering of update operations They refer to these deviations asforming continuous consistency ranges.

Measuring inconsistency in terms of numerical deviations can be used by plications for which the data have numerical semantics One obvious example isthe replication of records containing stock market prices In this case, an applica-tion may specify that two copies should not deviate more than $0.02, which would

ap-be an absolute numerical deviation. Alternatively, a relative numerical deviation

could be specified, stating that two copies should differ by no more than, for ample, 0.5% In both cases, we would see that if a stock goes up (and one of thereplicas is immediately updated) without violating the specified numerical devia-tions, replicas would still be considered to be mutually consistent

ex-Numerical deviation can also be understood in terms of the number of updatesthat have been applied to a given replica, but have not yet been seen by others.For example, a Web cache may not have seen a batch of operations carried out by

a Web server In this case, the associated deviation in the value is also referred to

to a local copy, awaiting global agreement from all replicas As a consequence,some updates may need to be rolled back and applied in a different order beforebecoming permanent Intuitively, ordering deviations are much harder to graspthan the other two consistency metrics We will provide examples below to clarifymatters

The Notion of a Conit

To define inconsistencies, Yu and Vahdat introduce a consistency unit, viated to conit A conit specifies the unit over which consistency is to be meas-ured For example, in our stock-exchange example, a conit could be defined as arecord representing a single stock Another example is an individual weather re-port

abbre-To give an example of a conit, and at the same time illustrate numerical andordering deviations, consider the two replicas as shown in Fig 7-2 Each replica imaintains a two-dimensional vector clock vq,just like the ones we described in

Figure 7-2 An example of keeping track of consistency deviations [adapted

from (Yu and Vahdat, 2002)].

In this example we see two replicas that operate on a conit containing the dataitems x andy. Both variables are assumed to have been initialized to O.Replica A

received the operation

5,B : x ~x +2

from replica B and has made it permanent (i.e., the operation has been committed

at A and cannot be rolled back) Replica A has three tentative update operations:

8,A, 12,A, and 14,A, which brings its ordering deviation to 3 Also note thatdue to the last operation 14,A, A's vector clock becomes (15,5)

The only operation from B that A has not yet seen is IO,B, bringing itsnumerical deviation with respect to operations to 1 In this example, the weight ofthis deviation can be expressed as the maximum difference between the (commit-ted) values ofx andy atA, and the result from operations at B not seen byA. Thecommitted value atA is(x,y) =(2,0), whereas the-for A unseen-operation at B

yields a difference ofy =5

A similar reasoning shows that B has two tentative update operations: 5,B

and 10,B , which means it has an ordering deviation of 2 Because B has not yetseen a single operation from A, its vector clock becomes (0, 11) The numericaldeviation is 3 with a total weight of 6 This last value comes from the fact B'scommitted value is (x,y) =(0,0), whereas the tentative operations at A willalready bring x to 6

Note that there is a trade-off between maintaining fine-grained and grained conits If a conit represents a lot of data, such as a complete database,then updates are aggregated for all the data in the conit As a consequence, this

may bring replicas sooner in an inconsistent state For example, assume that inFig 7-3 two replicas may differ in no more than one outstanding update In thatcase, when the data items in Fig 7-3(a) have each been updated once at the firstreplica, the second one will need to be updated as well This is not the case whenchoosing a smaller conit, as shown in Fig 7-3(b) There, the replicas are still con-sidered to be up to date This problem is particularly important when the dataitems contained in a conit are used completely independently, in which case theyare said to falsely share the conit

Figure 7-3 Choosing the appropriate granularity for a conit (a) Two updates

lead to update propagation (b) No update propagation is needed (yet).

Unfortunately, making conits very small is not a good idea, for the simple son that the total number of conits that need to be managed grows as well In otherwords, there is an overhead related to managing the conits that needs to be takeninto account This overhead, in tum, may adversely affect overall performance,which has to be taken into account

rea-Although from a conceptual point of view conits form an attractive means forcapturing consistency requirements, there are two important issues that need to bedealt with before they can be put to practical use First, in order to enforce consis-tency we need to have protocols Protocols for continuous consistency are dis-cussed later in this chapter

A second issue is that program developers must specify the consistency quirements for their applications Practice indicates that obtaining such require ments may be extremely difficult Programmers are generally not used to handlingreplication, let alone understanding what it means to provide detailed information

re-on cre-onsistency Therefore, it is mandatory that there are simple and stand programming interfaces

easy-to-under-Continuous consistency can be implemented as a toolkit which appears to grammers as just another library that they link with their applications A conit issimply declared alongside an update of a data item For example, the fragment ofpseudocode

pro-AffectsConit(ConitQ, 1, 1);

append message m to queue Q;

CHAP 7 CONSISTENCY AND REPLICA nON

states that appending a message to queue Q belongs to a conit named ""ConitQ."Likewise, operations may now also be declared as being dependent on conits:DependsOnConit(ConitQ, 4, 0, 60);

read message m from head of queue Q;

In this case, the call to DependsOnConitO specifies that the numerical deviation,ordering deviation, and staleness should be limited to the values 4, 0, and 60 (sec-onds), respectively This can be interpreted as that there should be at most 4unseen update operations at other replicas, there should be no tentative localupdates, and the local copy of Q should have been checked for staleness no morethan 60 seconds ago If these requirements are not fulfilled, the underlyingmiddle ware will attempt to bring the local copy of Q to a state such that the readoperation can be carried out

7.2.2 Consistent Ordering of Operations

Besides continuous consistency, there is a huge body of work on data-centricconsistency models from the past decades An important class of models comesfrom the field of concurrent programming Confronted with the fact that in paral-lel and distributed computing multiple processes will need to share resources andaccess these resources simultaneously, researchers have sought to express thesemantics of concurrent accesses when shared resources are replicated This hasled to at least one important consistency model that is widely used In the follow-ing, we concentrate on what is known as sequential consistency, and we will alsodiscuss a weaker variant, namely causal consistency

The models that we discuss in this section all deal with consistently orderingoperations on shared, replicated data In principle, the models augment those ofcontinuous consistency in the sense that when tentative updates at replicas need to

be committed, replicas will need to reach agreement on a global ordering of thoseupdates In other words, they need to agree on a consistent ordering of thoseupdates The consistency models we discuss next are all about reaching such con-sistent orderings

Sequential Consistency

In the following, we will use a special notation in which we draw the tions of a process along a time axis The time axis is always drawn horizontally,with time increasing from left to right The symbols

opera-mean that a write by process P; to data item x with the value a and a read from

that item by Pi returning b have been done, respectively We assume that each data item is initially NIL When there is no confusion concerning which process is

accessing data, we omit the index from the symbols Wand R.

282 CONSISTENCY AND REPLICATION CHAP 7

As an example, in Fig 7-4 PI does a write to a data item x, modifying its

val-ue toa. Note that, in principle, this operation WI (x)a is first performed on a copy

of the data store that is local to PI, and is then subsequently propagated to theother local copies In our example, P 2 later reads the value NIL, and some time

after that a (from its local copy of the store) What we are seeing here is that it

took some time to propagate the update ofx to P 2, which is perfectly acceptable.Sequential consistency is an important data-centric consistency model,which was first defined by Lamport (1979) in the context of shared memory formultiprocessor systems In general, a data store is said to be sequentially con-sistent when it satisfies the following condition:

The result of any execution is the same as if the (read and write) tions by all processes on the data store were executed in some sequential order and the operations of-each individual process appear in this se- quence in the order specified by its program.

opera-What this definition means is that when processes run concurrently on bly) different machines, any valid interleaving of read and write operations isacceptable behavior, but all processes see the same interleaving of operations.

(possi-Note that nothing is said about time; that is, there is no reference to the "mostrecent" write operation on a data item Note that in this context, a process "sees"writes from all processes but only its own reads

That time does not playa role can be seen from Fig 7-5 Consider four

proc-esses operating on the same data item x In Fig 7-5(a) process PI first performs

W(x)a to x. Later (in absolute time), process P 2 also performs a write operation,

by setting the value ofx to b. However, both processes P 3 and P 4 first read value

b, and later value a In other words, the write operation of process P 2 appears tohave taken place before that ofPI·

In contrast, Fig.7-5(b) violates sequential consistency because not all esses see the same interleaving of write operations In particular, to process P 3, itappears as if the data item has first been changed to b, and later to a On the other

proc-hand,P4 will conclude that the final value is b.

To make the notion of sequential consistency more concrete, consider threeconcurrently-executing processes PI, P 2, and P 3, shown in Fig 7-6 (Dubois et aI.,1988) The data items in this example are formed by the three integer variables x,

y, and z, which are stored in a (possibly distributed) shared sequentially consistent

Figure 7-4 Behavior of two processes operating on the same data item The

horizontal axis is time.

Figure 7-5 (a) A sequentially consistent data store (b) A data store that is not

sequentially consistent.

Figure 7-6 Three concurrently-executing processes.

data store We assume that each variable is initialized to O In this example, anassignment corresponds to a write operation, whereas a print statement corres-ponds to a simultaneous read operation of its two arguments All statements areassumed to be indivisible

Various interleaved execution sequences are possible With six independentstatements, there are potentially 720 (6!) possible execution sequences, althoughsome of these violate program order Consider the 120 (5!) sequences that beginwithx ~ 1 Half of these haveprint (r.z) before y ~ 1 and thus violate programorder Half also have print (x,y) before z ~ 1 and also violate program order.Only 1/4 of the 120 sequences, or 30, are valid Another 30 valid sequences are

possible starting with y ~ 1 and another 30 can begin with z ~ 1, for a total of 90

valid execution sequences Four of these are shown in Fig 7-7

In Fig 7-7(a), the three processes are run in order, first Ph then P 2, then P 3.

The other three examples demonstrate different, but equally valid, interleavings ofthe statements in time Each of the three processes prints two variables Since theonly values each variable can take on are the initial value (0), or the assignedvalue (1), each process produces a 2-bit string The numbers after Prints are theactual outputs that appear on the output device

If weconcatenate the output of PI, P 2, and P 3 in that order, we get a 6-bitstring that characterizes a particular interleaving of statements This is the stringlisted as the Signature in Fig 7-7 Below we will characterize each ordering byits signature rather than by its printout

Not all 64 signature patterns are allowed As a trivial example, 000000 is notpermitted, because that would imply that the print statements ran before theassignment statements, violating the requirement that statements are executed in

284 CONSISTENCY AND REPLICATION CHAP 7

Figure 7-7 Four valid execution sequences for the processes of Fig 7-6 The

vertical axis is time.

program order A more subtle example is 001001 The first two bits, 00, mean that

y and z were both 0 when PI did its printing This situation occurs only when PI

executes both statements before P 2 or P 3 starts The next two bits, 10, mean that

P 2 must run after P, has started but before P 3 has started The last two bits, 01,mean that P 3 must complete before P, starts, but we have already seen that PI

must go first Therefore, 001001 is not allowed

In short, the 90 different valid statement orderings produce a variety of ferent program results (less than 64, though) that are allowed under the assump-tion of sequential consistency The contract between the processes and the distrib-uted shared data store is that the processes must accept all of these as valid re-sults In other words, the processes must accept the four results shown in Fig 7-7and all the other valid results as proper answers, and must work correctly if any ofthem occurs A program that works for some of these results and not for othersviolates the contract with the data store and is incorrect

dif-Causal Consistency

The causal consistency model (Hutto and Ahamad, 1990) represents a ening of sequential consistency in that it makes a distinction between events thatare potentially causally related and those that are not We already came acrosscausality when discussing vector timestamps in the previous chapter If event b is

weak-caused or influenced by an earlier event a, causality requires that everyone else first see a, then see b.

Consider a simple interaction by means of a distributed shared database pose that process P, writes a data item x Then P 2 reads x and writes y. Here thereading of x and the writing of y are potentially causally related because the

Sup-computation of y may have depended on the value of x as read by Pz (i.e., thevalue written by PI)'

On the other hand, if two processes spontaneously and simultaneously writetwo different data items, these are not causally related Operations that are notcausally related are said to be concurrent

For a data store to be considered causally consistent, it is necessary that thestore obeys the following condition:

Writes that are potentially causally related must be seen by all processes

in the same order Concurrent writes may be seen in a different order on different machines.

As an example of causal consistency, consider Fig 7-8 Here we have an eventsequence that is allowed with a causally-consistent store, but which is forbiddenwith a sequentially-consistent store or a strictly consistent store The thing to note

is that the writes Wz(x)b and WI(x)c are concurrent, so it is not required that allprocesses see them in the same order

Figure 7-8 This sequence is allowed with a causally-consistent store, but not

with a sequentially consistent store.

Now consider a second example In Fig 7-9(a) we have Wz(x)b potentiallydepending on WI(x)a because the b may be a result of a computation involvingthe value read by Rz(x)a. The two writes are causally related, so all processesmust see them in the same order Therefore, Fig 7-9(a) is incorrect On the otherhand, in Fig 7-9(b) the read has been removed, so WI(x)a and Wz(x)b are nowconcurrent writes A causally-consistent store does not require concurrent writes

to be globally ordered, so Fig.7-9(b) is correct Note that Fig.7-9(b) reflects asituation that would not be acceptable for a sequentially consistent store

Figure 7-9 (a) A violation of a causally-consistent store (b) A correct

se-quence of events in a causally-consistent store.

Implementing causal consistency requires keeping track of which processeshave seen which writes It effectively means that a dependency graph of which

286 CONSISTENCY AND REPLICA DON CHAP 7operation is dependent on which other operations must be constructed and main-tained One way of doing this is by means of vector timestamps, as we discussed

in the previous chapter We return to the use of vector timestamps to capturecausality later in this chapter

Grouping Operations

Sequential and causal consistency are defined at the level read and write ations This level of granularity is for historical reasons: these models have ini-tially been developed for shared-memory multiprocessor systems and were actual-

oper-ly implemented at the hardware level

The fine granularity of these consistency models in many cases did not matchthe granularity as provided by applications What we see there is that concurrencybetween programs sharing data is generally kept under control through synchroni-zation mechanisms for mutual exclusion and transactions Effectively, what hap-pens is that at the program level read and write operations are bracketed by thepair of operations ENTER_CS and LEAVE_CS where "CS" stands for criticalsection As we explained in Chap 6, the synchronization between processes takesplace by means of these two operations In terms of our distributed data store, thismeans that a process that has successfully executed ENTER_CS will be ensuredthat the data in its local store is up to date At that point, it can safely execute aseries of read and write operations on that store, and subsequently wrap things up

by callingLEAVE_CS.

In essence, what happens is that within a program the data that are operated

on by a series of read and write operations are protected against concurrent cesses that would lead to seeing something else than the result of executing theseries as a whole Put differently, the bracketing turns the series of read and writeoperations into an atomically executed unit, thus raising the level of granularity

ac-In order to reach this point, we do need to have precise semantics concerningthe operations ENTER_CS and LEAVE_CS. These semantics can be formulated

in terms of shared synchronization variables There are different ways to usethese variables We take the general approach in which each variable has someassociated data, which could amount to the complete set of shared data We adoptthe convention that when a process enters its critical section it should acquire therelevant synchronization variables, and likewise when it leaves the critical sec-tion, it releases these variables Note that the data in a process' critical sectionmay be associated to different synchronization variables

Each synchronization variable has a current owner, namely, the process thatlast acquired it The owner may enter and exit critical sections repeatedly withouthaving to send any messages on the network A process not currently owning asynchronization variable but wanting to acquire it has to send a message to thecurrent owner asking for ownership and the current values of the data associatedwith that synchronization variable It is also possible for several processes to

Figure 7·10 A valid event sequence for entry consistency.

One of the programming problems with entry consistency is properly ing data with synchronization variables One straightforward approach is to expli-citly tell the middleware which data are going to be accessed, as is generally done

associat-simultaneously own a synchronization variable in nonexclusive mode, meaningthat they can read, but not write, the associated data

We now demand that the following criteria are met (Bershad et al., 1993):

1 An acquire access of a synchronization variable is not allowed to perform with respect to a process until all updates to the guarded shared data have been performed with respect to that process.

2 Before an exclusive mode access to a synchronization variable by a process is allowed to perform with respect to that process, no other process may hold the synchronization variable, not even in nonex- clusive mode.

3 After an exclusive mode access to a synchronization variable has been performed, any other process' next nonexclusive mode access

to that synchronization variable may not be performed until it has performed with respect to that variable's owner.

The first condition says that when a process does an acquire, the acquire may notcomplete (i.e., return control to the next statement) until all the guarded shareddata have been brought up to date In other words, at an acquire, all remotechanges to the guarded data must be made visible

The second condition says that before updating a shared data item, a processmust enter a critical section in exclusive mode to make sure that no other process

is trying to update the shared data at the same time

The third condition says that if a process wants to enter a critical region innonexclusive mode, it must first check with the owner of the synchronization vari-able guarding the critical region to fetch the most recent copies of the guardedshared data

Fig 7-10 shows an example of what is known as entry consistency Instead

of operating on the entire shared data, in this example we associate locks witheach data item In this case, PIdoes an acquire for x, changes x once, after which

it also does an acquire for y. Process P 2 does an acquire for x but not for Y'.so that

it will read value a for x, but may read NIL for y. Because process P 3 first does an

acquire for y, it will read the value b when y is released by Pl'

288 CONSISTENCY AND REPLICATION CHAP 7

by declaring which database tables will be affected by a transaction In an based approach, we could implicitly associate a unique synchronization variablewith each declared object, effectively serializing all invocations to such objects.Consistency versus Coherence

object-At this point, it is useful to clarify the difference between two closely relatedconcepts The models we have discussed so far all deal with the fact that a number

of processes execute read and write operations on a set of data items A tency model describes what can be expected with respect to that set when multi-ple processes concurrently operate on that data The set is then said to be con-sistent if it adheres to the rules described by the model

consis-Where data consistency is concerned with a set of data items, coherencemodels describe what can be expected to only a single data item (Cantin et aI.,2005) In this case, we assume that a data item is replicated at several places; it issaid to be coherent when the various copies abide to the rules as defined by its as-sociated coherence model A popular model is that of sequential consistency, butnow applied to only a single data item In effect, it means that in the case ofconcurrent writes, all processes will eventually see the same order of updates tak-ing place

7.3 CLIENT-CENTRIC CONSISTENCY MODELS

The consistency models described in the previous section aim at providing asystemwide consistent view on a data store An important assumption is thatconcurrent processes may be simultaneously updating the data store, and that it isnecessary to provide consistency in the face of such concurrency For example, inthe case of object-based entry consistency, the data store guarantees that when anobject is called, the calling process is provided with a copy of the object that re-flects all changes to the object that have been made so far, possibly by other proc-esses During the call, it is also guaranteed that no other process can interfere-that is, mutual exclusive access is provided to the calling process

Being able to handle-concurrent operations on shared data while maintainingsequential consistency is fundamental to distributed systems For performancereasons, sequential consistency may possibly be guaranteed only when processesuse synchronization mechanisms such as transactions or locks

In this section, we take a look at a special class of distributed data stores Thedata stores we consider are characterized by the lack of simultaneous updates, orwhen such updates happen, they can easily be resolved Most operations involvereading data These data stores offer a very weak consistency model, called even-tual consistency By introducing special client-centric consistency models, it turnsout that many inconsistencies can be hidden in a relatively cheap way

7.3.1 Eventual Consistency

To what extent processes actually operate in a concurrent fashion, and to whatextent consistency needs to be guaranteed, may vary There are many examples inwhich concurrency appears only in a restricted form For example, in many data-base systems, most processes hardly ever perform update operations; they mostlyread data from the database Only one, or very few processes perform update op-erations The question then is how fast updates should be made available to only-reading processes

As another example, consider a worldwide naming system such as DNS TheDNS name space is partitioned into domains, where each domain is assigned to anaming authority, which acts as owner of that domain Only that authority is al-lowed to update its part of the name space Consequently, conflicts resulting from

two operations that both want to perform an update on the same data (i.e., write conflicts), never occur The only situation that needs to be handled are read-write conflicts, in which one process wants to update a data item while an-

write-other is concurrently attempting to read that item As it turns out, it is oftenacceptable to propagate an update in a lazy fashion, meaning that a reading proc-ess will see an update only after some time has passed since the update took place.Yet another example is the World Wide Web In virtually all cases, Webpages are updated by a single authority, such as a webmaster or the actual owner

of the page There are normally no write-write conflicts to resolve On the otherhand, to improve efficiency, browsers and Web proxies are often configured tokeep a fetched page in a local cache and to return that page upon the next request

An important aspect of both types of Web caches is that they may return of-date Web pages In other words, the cached page that is returned to the re-questing client is an older version compared to the one available at the actual Webserver As it turns out, many users find this inconsistency acceptable (to a certaindegree)

out-These examples can be viewed as cases of (large-scale) distributed and cated databases that tolerate a relatively high degree of inconsistency They have

repli-in common that if no updates take place for a long time, all replicas will gradually

become consistent This form of consistency is called eventual consistency.

Data stores that are eventually consistent thus have the property that in theabsence of updates, all replicas converge toward identical copies of each other.Eventual consistency essentially requires only that updates are guaranteed to pro-pagate to all replicas Write-write conflicts are often relatively easy to solve whenassuming that only a small group of processes can perform updates Eventual con-sistency is therefore often cheap to implement

Eventual consistent data stores work tine as long as clients always access thesame replica However, problems arise when different replicas are accessed over ashort period of time This is best illustrated by considering a mobile user ac-cessing a distributed database, as shown in Fig 7-11

290 CONSISTENCY AND REPLICATION CHAP 7

Figure '-11 The principle of a mobile user accessing different replicas of a

distributed database.

The mobile user accesses the database by connecting to one of the replicas in

a transparent way In other words, the application running on the user's portablecomputer is unaware on which replica it is actually operating Assume the userperforms several update operations and then disconnects again Later, he accessesthe database again, possibly after moving to a different location or by using a dif-ferent access device At that point, the user may be connected to a different rep-lica than before, as shown in Fig 7-11 However, if the updates performed prev-iously have not yet been propagated, the user will notice inconsistent behavior Inparticular, he would expect to see all previously made changes, but instead, itappears as if nothing at all has happened

This example is typical for eventually-consistent data stores and is caused bythe fact that users may sometimes operate on different replicas The problem can

be alleviated by introducing client-centric consistency In essence, client-centric

consistency provides guarantees for a single client concerning the consistency of

accesses to a data store by that client No guarantees are given concerning rent accesses by different clients

concur-Client-centric consistency models originate from the work on Bayou [see, forexample Terry et al (1994) and Terry et aI., 1998)] Bayou is a database systemdeveloped for mobile computing, where it is assumed that network connectivity isunreliable and subject to various performance problems Wireless networks andnetworks that span large areas, such as the Internet, fall into this category

Bayou essentially distinguishes four different consistency models To explainthese models, we again consider a data store that is physically distributed acrossmultiple machines When a process accesses the data store, it generally connects

to the locally (or nearest) available copy, although, in principle, any copy will dojust fine All read and write operations are performed on that local copy Updatesare eventually propagated to the other copies To simplify matters, we assume thatdata items have an associated owner, which is the only process that is permitted tomodify that item In this way, we avoid write-write conflicts

Client-centric consistency models are described using the following notations.LetXi[t] denote the version of data item x at local copyL, at time t.VersionXi(t]

is the result of a series of write operations atL i that took place since initialization

\Ve denote this set as WS(xi[tD. If operations in WS(xJtIJ) have also been formed at local copyL j at a later time t2, we write WS(xi(td~[t2]). If the order-ing of operations or the timing is clear from the context, the time index will beomitted

per-7.3.2 Monotonic Reads

The first client-centric consistency model is that of monotonic reads A datastore is said to provide monotonic-read consistency if the following conditionholds:

If a process reads the value of a data item x, any successive read tion on x by that process will always return that same value or a more recent value.

opera-In other words, monotonic-read consistency guarantees that if a process has seen avalue ofx at time t,it will never see an older version ofx at a later time

As an example where monotonic reads are useful, consider a distributed mail database In such a database, each user's mailbox may be distributed andreplicated across multiple machines Mail can be inserted in a mailbox at any lo-cation However, updates are propagated in a lazy (i.e., on demand) fashion Onlywhen a copy needs certain data for consistency are those data propagated to thatcopy Suppose a user reads his mail in San Francisco Assume that only readingmail does not affect the mailbox, that is, messages are not removed, stored insubdirectories, or even tagged as having already been read, and so on When theuser later flies to New York and opens his mailbox again, monotonic-read consis-tency guarantees that the messages that were in the mailbox in San Francisco willalso be in the mailbox when it is opened in New York

e-Using a notation similar to that for data-centric consistency models, tonic-read consistency can be graphically represented as shown in Fig 7-12

mono-Along the vertical axis, two different local copies of the data store are shown, L I

and L2. Time is shown along the horizontal axis as before In all cases, we are

292 CONSISTENCY AND REPLICATION CHAP 7interested in the operations carried out by a single process P.These specific oper-ations are shown in boldface are connected by a dashed line representing the order

in which they are carried out by P.

Figure 7-12 The read operations performed by a single process P at two

dif-ferent local copies of the same data store (a) A monotonic-read consistent datu

store (b) A data store that does not provide monotonic reads.

In Fig 7-l2(a), process P first performs a read operation on x at L I,returningthe value of Xl (at that time) This value results from the write operations in

WS(xI) performed at LI Later, P performs a read operation on x at L2, shown as

R (X2)' To guarantee monotonic-read consistency, all operations in WS(x 1) shouldhave been propagated to L 2 before the second read operation takes place In otherwords, we need to know for sure that WS(x I) is part of WS(x2)' which isexpressed as WS(xI;X2)'

In contrast, Fig 7-l2(b) shows a situation in which monotonic-read

consisten-cy is not guaranteed After process P has read x I atL I ,it later performs the ation R(X2) atL2. However, only the write operations in WS(X2) have been per-formed at L 2• No guarantees are given that this set also contains all operationscontained in WS (x I)'

oper-7.3.3 Monotonic Writes

In many situations, it is important that write operations are propagated in thecorrect order to all copies of the data store This property is expressed in mon-otonic-write consistency In a monotonic-write consistent store, the followingcondition holds:

A write operation by a process on a data item x is completed before any successive write operation on X by the same process.

Thus completing a write operation means that the copy on which a successive eration is performed reflects the effect of a previous write operation by the sameprocess, no matter where that operation was initiated In other words, a write op-eration on a copy of item x is performed only if that copy has been brought up todate by means of any preceding write operation, which may have taken place onother copies ofx. If need be, the new write must wait for old ones to finish

op-Note that monotonic-write consistency resembles data-centric FIFO tency The essence of FIFO consistency is that write operations by the same proc-ess are performed in the correct order everywhere This ordering constraint alsoapplies to monotonic writes, except that we are now considering consistency onlyfor a single process instead of for a collection of concurrent processes.

consis-Bringing a copy ofx up to date need not be necessary when each write tion completely overwrites the present value of x. However, write operations areoften performed on only part of the state of a data item Consider, for example, asoftware library In many cases, updating such a library is done by replacing one

opera-or mopera-ore functions, leading to a next version With monotonic-write consistency,guarantees are given that if an update is performed on a copy of the library, allpreceding updates will be performed first The resulting library will then indeedbecome the most recent version and will include all updates that have led to previ-ous versions of the library

Monotonic-write consistency is shown in Fig 7-13 In Fig 7-13(a), process P

performs a write operation on x at local copy L 1, presented as the operation

W(XI). Later, P performs another write operation on x, but this time at L 2, shown

as W(X2). To ensure monotonic-write consistency, it is necessary that the previouswrite operation at L 1 has already been propagated to L 2• This explains operation

W(Xl) atL2, and why it takes place before W(X2)·

Figure 7-13 The write operations performed by a single process P at two

dif-ferent local copies of the same data store (a) A monotonic-write consistent data

store (b) A data store that does not provide monotonic-write consistency.

In contrast, Fig 7-13(b) shows a situation in which monotonic-write tency is not guaranteed Compared to Fig 7-13(a), what is missing is the propaga-tion of W (x 1) to copy L2. In other words, no guarantees can be given that thecopy of x on which the second write is being performed has the same or morerecent value at the time W (xI)completed atL I.

consis-Note that, by the definition of monotonic-write consistency, write operations

by the same process are performed in the same order as they are initiated Asomewhat weaker form of monotonic writes is one in which the effects of a writeoperation are seen only if all preceding writes have been carried out as well, butperhaps not in the order in which they have been originally initiated This consis-tency is applicable in those cases in which write operations are commutative, sothat ordering is really not necessary Details are found in Terry et al (1994)

294 CONSISTENCY AND REPLICATION CHAP 7

7.3.4 Read Your Writes

A client-centric consistency model that is closely related to monotonic reads

is as follows A data store is said to provide read-your-writes consistency, if thefollowing condition holds:

The effect of a write operation by a process on data item x will always be seen by a successive read operation on x by the same process.

In other words, a write operation is always completed before a successive read eration by the same process, no matter where that read operation takes place.The absence of read-your-writes consistency is sometimes experienced whenupdating Web documents and subsequently viewing the effects Update operationsfrequently take place by means of a standard editor or word processor, whichsaves the new version on a file system that is shared by the Web server Theuser's Web browser accesses that same file, possibly after requesting it from thelocal Web server However, once the file has been fetched, either the server or thebrowser often caches a local copy for subsequent accesses Consequently, whenthe Web page is updated, the user will not see the effects if the browser or theserver returns the cached copy instead of the original file Read-your-writes con-sistency can guarantee that if the editor and browser are integrated into a singleprogram, the cache is invalidated when the page is updated, so that the updatedfile is fetched and displayed

op-Similar effects occur when updating passwords For example, to enter a tal library on the Web, it is often necessary to have an account with an accom-panying password However, changing a password make take some time to comeinto effect, with the result that the library may be inaccessible to the user for a fewminutes The delay can be caused because a separate server is used to manage pass-words and it may take some time to subsequently propagate (encrypted) passwords

digi-to the various servers that constitute the library

Fig.7-14(a) shows a data store that provides read-your-writes consistency.Note that Fig 7-14(a) is very similar to Fig 7-12(a), except that consistency isnow determined by the last write operation by process P, instead of its last read

Figure 7-14 (a) A data store that provides read-your-writes consistency (b) A

data store that does not.

In Fig 7-14(a), process P performed a write operation W(XI) and later a readoperation at a different local copy Read-your-writes consistency guarantees that

the effects of the write operation can be seen by the succeeding read operation.This is expressed by WS(XI ;X2), which states that W(Xl) is part of WS(X2)' Incontrast, in Fig 7-14(b), W(Xl) has been left out of WS(X2), meaning that the ef-

fects of the previous write operation by process P have not been propagated to L 2·

7.3.5 Writes Follow Reads

The last client-centric consistency model is one in which updates are pagated as the result of previous read operations A data store is said to providewrites-follow-reads consistency, if the following holds

pro-A write operation by a process on a data item x following a previous read operation on x by the same process is guaranteed to take place on the same or a more recent value of x that was read.

In other words, any successive write operation by a process on a data item x will

be performed on a copy ofx that is up to date with the value most recently read bythat process

Writes-follow-reads consistency can be used to guarantee that users of a work newsgroup see a posting of a reaction to an article only after they have seenthe original article (Terry et aI., 1994) To understand the problem, assume that auser first reads an articleA. Then, he reacts by posting a response B. By requiringwrites-follow-reads consistency, B will be written to any copy of the newsgrouponly afterA has been written as well Note that users who only read articles neednot require any specific client-centric consistency model The writes-follows-reads consistency assures that reactions to articles are stored at a local copy only

net-if the original is stored there as well

Figure 7-15 (a) A writes-follow-reads consistent data store (b) A data store

that does not provide writes-follow-reads consistency.

This consistency model is shown in Fig 7-15 In Fig 7-15(a), a process reads

x at local copy L1. The write operations that led to the value just read, also appear

in the write set at L2. where the same process later performs a write operation

(Note that other processes at L 2 see those write operations as well.) In contrast, no

guarantees are given that the operation performed at L 2, as shown in Fig 7-15(b),

are performed on a copy that is consistent with the one just read at L1 •

We will return to client-centric consistency models when we discuss mentations later on in this chapter

imple-296 CONSISTENCY AND REPLICA nON CHAP 7

7.4 REPLICA MANAGEMENT

A key issue for any distributed system that supports replication is to decidewhere, when, and by whom replicas should be placed, and subsequently whichmechanisms to use for keeping the replicas consistent The placement problem it-

self should be split into two subproblems: that of placing replica servers, and that

of placing content The difference is a subtle but important one and the two issues

are often not clearly separated Replica-server placement is concerned with ing the best locations to place a server that can host (part of) a data store Contentplacement deals with finding the best servers for placing content Note that thisoften means that we are looking for the optimal placement of only a single dataitem Obviously, before content placement can take place, replica servers willhave to be placed first In the following, take a look at these two different place-ment problems, followed by a discussion on the basic mechanisms for managingthe replicated content

find-7.4.1 Replica-Server Placement

The placement of replica servers is not an intensively studied problem for thesimple reason that it is often more of a management and commercial issue than anoptimization problem Nonetheless, analysis of client and network properties areuseful to come to informed decisions

There are various ways to compute the.best placement of replica servers, butall boil down to an optimization problem in which the best K out of N locationsneed to be selected (K <N). These problems are known to be computationallycomplex and can be solved only through heuristics Qiu et al (2001) take the dis-tance between clients and locations as their starting point Distance can be meas-ured in terms of latency or bandwidth Their solution selects one server at a timesuch that the average distance between that server and its clients is minimal given

that already k servers have been placed (meaning that there are N - k locations

left)

As an alternative, Radoslavov et aI (2001) propose to ignore the position ofclients and only take the topology of the Internet as formed by the autonomoussystems An autonomous system (AS) can best be.viewed as a network in whichthe nodes all run the same routing protocol and which is managed by a singleorganization As of January 2006, there were just over 20,000 ASes Radoslavov

et aI first consider the largest AS and place a server on the router with the largestnumber of network interfaces (i.e., links) This algorithm is then repeated with thesecond largest AS, and so on

As it turns out, client-unaware server placement achieves similar results asclient-aware placement, under the assumption that clients are uniformly distrib-uted across the Internet (relative to the existing topology) To what extent this as-sumption is true is unclear It has not been well studied

One problem with these algorithms is that they are computationally sive For example, both the previous algorithms have a complexity that is higherthanO(N'2), where N is the number of locations to inspect In practice, this meansthat for even a few thousand locations, a computation may need to run for tens ofminutes This may be unacceptable, notably when there are flash crowds (a sud-den burst of requests for one specific site, which occur regularly on the Internet).

expen-In that case, quickly determining where replica servers are needed is essential,after which a specific one can be selected for content placement

Szymaniak et al (2006) have developed a method by which a region for ing replicas can be quickly identified A region is identified to be a collection ofnodes accessing the same content, but for which the internode latency is low Thegoal of the algorithm is first to select the most demanding regions-that is, theone with the most nodes-and then to let one of the nodes in such a region act asreplica server

plac-To this end, nodes are assumed to be positioned in an m-dimensional metric space, as we discussed in the previous chapter The basic idea is to identifytheK largest clusters and assign a node from each cluster to host replicated con-tent To identify these clusters, the entire space is partitioned into cells The K

geo-most dense cells are then chosen for placing a replica server A cell is nothing but

an m-dimensional hypercube For a two-dimensional space, this corresponds to arectangle

Obviously, the cell size is important, as shown in Fig 7-16 If cells arechosen too large, then multiple clusters of nodes may be contained in the samecell In that case, too few replica servers for those clusters would be chosen Onthe other hand, choosing small cells may lead to the situation that a single cluster

is spread across a number of cells, leading to choosing too many replica servers

Figure 7-16 Choosing a proper cell size for server placement.

As it turns out, an appropriate cell size can be computed as a simple function

of the average distance between two nodes and the number of required replicas.With this cell size, it can be shown that the algorithm performs as well as theclose-to-optimal one described in Qiu et al (2001), but having a much lower com-plexity: O(Nxmax {log (N), K}). To give an impression what this result means:

298 CONSISTENCY AND REPLICATION CHAP 7experiments show that computing the 20 best replica locations for a collection of64,000 nodes is approximately 50.000 times faster As a consequence, replica-server placement can now be done in real time.

7.4.2 Content Replication and Placement

Let us now move away from server placement and concentrate on contentplacement When it comes to content replication and placement, three differenttypes of replicas can be distinguished logically organized as shown in Fig 7- 17

Figure 7-17 The logical organization of different kinds of copies of a data store

into three concentric rings.

Permanent Replicas

Permanent replicas can be considered as the initial set of replicas that tute a distributed data store In many cases, the number of permanent replicas issmall Consider, for example, a Web site Distribution of a Web site generallycomes in one of two forms The first kind of distribution is one in which the filesthat constitute a site are replicated across a limited number of servers at a singlelocation Whenever a request comes in, it is forwarded to one of the servers, forinstance, using a round-robin strategy

consti-The second form of distributed Web sites is what is called mirroring In thiscase, a Web site is copied to a limited number of servers, called mirror sites.which are geographically spread across the Internet In most cases, clients simplychoose one of the various mirror sites from a list offered to them Mirrored v 'ebsites have in common with cluster-based Web sites that there are only a few num-ber of replicas, which are more or less statically configured

Similar static organizations also appear with distributed databases (OSZu andValduriez, 1999) Again, the database can be distributed and replicated acrOSS ~\number of servers that together form a cluster of servers, often referred to as ~\shared-nothing architecture, emphasizing that neither disks nor main memory

are shared by processors Alternatively, a database is distributed and possibly licated across a number of geographically dispersed sites This architecture is gen-erally deployed in federated databases (Sheth and Larson, 1990).

rep-Server-Initiated Replicas

In contrast to permanent replicas, server-initiated replicas are copies of a datastore that exist to enhance performance and which are created at the initiative ofthe (owner of the) data store Consider, for example, a Web server placedin NewYork Normally, this server can handle incoming requests quite easily, but it mayhappen that over a couple of days a sudden burst of requests come in from anunexpected location far from the server In that case, it may be worthwhile toinstall a number of temporary replicas in regions where requests are coming from.The problem of dynamically placing replicas is also being addressed in Webhosting services These services offer a (relatively static) collection of serversspread across the Internet that can maintain and provide access to Web filesbelonging to third parties To provide optimal facilities such hosting services candynamically replicate files to servers where those files are needed to enhance per-formance, that is, close to demanding (groups of) clients Sivasubramanian et

al (2004b) provide an in-depth overview of replication in Web hosting services towhich we will return in Chap 12

Given that the replica servers are already in place, deciding where to placecontent is easier than in the case of server placement An approach to dynamicreplication of files in the case of a Web hosting service is described in Rabinovich

et al (1999) The algorithm is designed to support Web pages for which reason itassumes that updates are relatively rare compared to read requests Using tiles asthe unit of data, the algorithm works as follows

The algorithm for dynamic replication takes two issues into account First,replication can take place to reduce the load on a server Second, specific files on

a server can be migrated or replicated to servers placed in the proximity of clientsthat issue many requests for those files In the following pages, we concentrateonly on this second issue We also leave out a number of details, which can befound in Rabinovich et al (1999)

Each server keeps track of access counts per file, and where access requestscome from In particular, it is assumed that, given a client C, each server candetermine which of the servers in the Web hosting service is closest to C (Suchinformation can be obtained, for example, from routing databases.) If client C1

and client C2 share the same "closest" server P, all access requests for file Fat

server Q from eland C2 are jointly registered at Q as a single access count

cntQ(P,F). This situation is shown in Fig 7-18

When the number of requests for a specific file F at server S drops below adeletion threshold del (S,F), that file can be removed from S As a consequence,the number of replicas of that file is reduced, possibly leading to higher work

300 CONSISTENCY AND REPLICATION CHAP 7

Figure 7-18 Counting access requests from different clients.

loads at other servers Special measures are taken to ensure that at least one copy

of each file continues to exist

A replication threshold rep (5, F), which is always chosen higher than the

deletion threshold, indicates that the number of requests for a specific file is sohigh that it may be worthwhile replicating it on another server If the number ofrequests lie somewhere between the deletion and replication threshold, the file isallowed only to be migrated In other words, in that case it is important to at leastkeep the number of replicas for that file the same

When a server Q decides to reevaluate the placement of the files it stores, it

checks the access count for each file If the total number of access requests for F

at Q drops below the deletion threshold del (Q,F), it will delete F unless it is the

last copy Furthermore, if for some server P, cntQ(p,F) exceeds more than half ofthe total requests for F at Q, server P is requested to take over the copy of F. In

other words, server Q will attempt to migrate F to P.

Migration of file F to server P may not always succeed, for example, because

P is already heavily loaded or is out of disk space In that case, Q will attempt to

replicate F on other servers Of course, replication can take place only if the total number of access requests for F at Q exceeds the replication threshold rep (Q,F).

Server Q checks all other servers in the Web hosting service, starting with the onefarthest away If, for some server R, cntQ(R,F) ex-ceeds a certain fraction of all re-quests for F at Q, an attempt is made to replicate F to R.

Server-initiated replication continues to increase in popularity in time, cially in the context of Web hosting services such as the one just described Notethat as long as guarantees can be given that each data item is hosted by at leastone server, it may suffice to use only server-initiated replication and not have anypermanent replicas Nevertheless, permanent replicas are still often useful as aback-up facility, or to be used as the only replicas that are allowed to be changed

espe-to guarantee consistency Server-initiated replicas are then used for placing only copies close to clients