Guide to reliable distributed systems building high assurance applications and cloud hosted services

Texts in Computer Science Editors David Gries Fred B Schneider For further volumes: www.springer.com/series/3191 Kenneth P Birman Guide to Reliable Distributed Systems Building High-Assurance Applications and Cloud-Hosted Services Kenneth P Birman Department of Computer Science Cornell University Ithaca, NY, USA Series Editors David Gries Department of Computer Science Cornell University Ithaca, NY, USA Fred B Schneider Department of Computer Science Cornell University Ithaca, NY, USA ISSN 1868-0941 e-ISSN 1868-095X Texts in Computer Science ISBN 978-1-4471-2415-3 e-ISBN 978-1-4471-2416-0 DOI 10.1007/978-1-4471-2416-0 Springer London Dordrecht Heidelberg New York British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: 2012930225 © Springer-Verlag London Limited 2012 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licenses issued by the Copyright Licensing Agency Enquiries concerning reproduction outside those terms should be sent to the publishers The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface Setting the Stage The Guide to Reliable Distributed Systems: Building High-Assurance Applications and Cloud-Hosted Services is a heavily edited new edition of a prior edition that went under the name Reliable Distributed Computing; the new name reflects a new focus on Cloud Computing The term refers to the technological infrastructure supporting today’s web systems, social networking, e-commerce and a vast array of other applications The emergence of the cloud has been a transformational development, for a number of reasons: cost, flexibility, new ways of managing and leveraging large data sets There are other benefits that we will touch on later The cloud is such a focus of product development and so associated with overnight business success stories today that one could easily write a text focused on the cloud “as is” and achieve considerable success with the resulting text After all, the cloud has enabled companies like Netflix, with a few hundred employees, to create a movie-on-demand capability that may someday scale to reach every potential consumer in the world Facebook, with a few thousand employees, emerged overnight to create a completely new form of social network, having the importance and many of the roles that in the past one associated with vast infrastructures like email, or the telephone network The core Google search infrastructure was created by just a few dozen employees (by now, of course, Google has tens of thousands, and does far more than just search) And the cloud is an accelerator for such events: companies with a good idea can launch a new product one day, and see it attract a million users a week later without breaking a sweat This capability is disruptive and profoundly impactful and is reshaping the technology sector at an accelerating pace Of course there is a second side to the cloud, and one that worries many corporate executives both at the winners and at other companies: the companies named above were picked by the author in the hope that they would be success stories for as long as the text is in active use After all this text will quickly seem dated if it seems to point to yesterday’s failures as if they were today’s successes Yet we all know that companies that sprang up overnight have a disconcerting way of vanishing just as quickly: the cloud has been a double-edged sword A single misstep can spell doom A single new development can send the fickle consumer community rushing to some new and even more exciting alternative The cloud, then, is quite a stormy v vi Preface place! And this book, sadly, may well be doomed to seem dated from the very day it goes to press But even if the technical landscape changes at a dizzying pace, the cloud already is jam-packed with technologies that are fascinating to learn about and use, and that will certainly live on in some form far into the future: BitTorrent, for example (a swarm-style download system) plays key roles in the backbone of Twitter’s data center, Memcached (a new kind of key-value store) has displaced standard file system storage for a tremendous range of cloud computing goals MapReduce and its cousin Hadoop enable a new kind of massively parallel data reduction Chubby supports scalable locking and synchronization, and is a critical component at the core of Google’s cloud services platform ZooKeeper plays similar roles in Yahoo!’s consistency-based services Dynamo, Amazon’s massively replicated keyvalue store, is the basis for its shopping cart service BigTable, Google’s giant tablestructured storage system, manages sparse but enormous tabular data sets JGroups and Spread, two commercially popular replication technologies, allow cloud services to maintain large numbers of copies of heavily accessed data The list goes on, including global file systems, replication tools, load balancing subsystems, you name it Indeed, the list is so long that even today, we will only touch on a few representative examples; it would take many volumes to cover everything the cloud can do, and to understand all the different ways it does those things We will try and work our way in from the outside, identifying deep problems along the way, and then we will tackle those fundamental questions Accordingly, Part I of the book gives a technical overview of the whole picture, covering the basics but without delving deeply on the more subtle technology questions that arise, such as data replication We will look at those harder questions in Parts II and III of the text; Part IV covers some additional technologies that merit inclusion for reasons of completeness, but for which considerations of length limit us to shallow reviews Above, we hinted at one of the deeper questions that sit at the core of Parts II and III If the cloud has a dark side, it is this: there are a great many applications that need forms of high assurance, but the cloud, as currently architected, only offers very limited support for scalable high assurance computing Indeed, if we look at high assurance computing in a broad way, and then look at how much of high assurance computing maps easily to the cloud, the only possible conclusion is that the cloud really does not support high assurance applications at all Yes, the cloud supports a set of transactional-security features that can be twisted this way and that to cover a certain class of uses (as mentioned earlier, those concerned with credit card purchases and with streaming copyright-protected content like movies and music from the cloud to your playback device), but beyond those limited use cases, high assurance technologies have been perceived as not scaling adequately for use in the cloud, at least in the scalable first tier that interacts with external clients The story is actually pretty grim First, we will encounter two theorems about things we cannot in cloud settings: one proves that fault-tolerant distributed computing is impossible in standard networks, and the second that data consistency cannot be achieved under the performance and availability requirements of the cloud Next, we will find that the existing cloud platforms are designed to violate consistency as a short-cut towards higher performance and better scalability Thus: “High Preface vii assurance in the cloud? It cannot be done, it cannot scale to large systems, and even if it could be done and it could be made to scale, it is not the way we it.” The assertion that high-assurance is not needed in most elements of most modern cloud computing applications may sound astonishing, yet if one looks closely, it turns out that the majority of web and cloud applications are cleverly designed to either completely avoid the need for high-assurance capabilities, or find ways to minimize the roles of any high assurance components, thereby squeezing the highassurance side of the cloud into smaller subsystems that not see remotely as much load as the main systems might encounter (if you like, visualize a huge cache-based front end that receives most of the workload, and then a second smaller core system that only sees update transactions which it applies to some sort of files or databases, and then as updates commit, pushes new data out to the cache, or invalidates cached records as needed) For example, just to pick an example from the air, think about a massive government program like the Veteran’s Administration Benefits program here in the United States This clearly needs strong assurance (all sorts of sensitive data moves back and forth), money changes hands (the VA system is, in part, a big insurance system), sensitive records are stored within the VA databases Yet if you study such a system carefully, as was done in a series of White House reviews during 2010 and 2011, the match with today’s cloud is really very good Secure web pages can carry that sensitive data with reasonable protection The relatively rare transactions against the system have much the same character as credit card transactions And if we compare the cost of operating a system such as this using the cloud model, as opposed to having the Veteran’s Administration run its own systems, we can predict annual savings in the tens of millions hundreds! Yet not a single element of the picture seems to be deeply at odds with today’s most successful cloud computing models Thus, our statements about high-assurance are not necessarily statements about limitations that every single high assurance computing use would encounter Ecommerce transactions on the web work perfectly well as long as the transactional system is not down, and when we use a secured web page to purchase a book or provide a credit card number, that action is about as secure as one can make it given some of the properties of the PCs we use as endpoints (as we will see, many home computers are infected with malware that does not anything visibly horrible, yet can still be actively snooping on the actions you as the user take, and could easily capture all sorts of passwords and other security-related data, or even initiate transactions on its own while you are fast asleep!) Notice that we have made a statement that does not demand continuous fault-tolerance (we all realize that these systems will sometimes be temporarily unavailable), and does not expose the transactional system to huge load (we all browse extensively and make actual purchases rarely: browsing is a high-load activity; purchasing, much less so) The industry has honed this particular high-assurance data path to the point that most of us, for most purposes, incur only limited risks in trusting these kinds of solutions Moreover, one cannot completely eliminate risk When you hand your credit card to a waiter, you also run some risks, and we accept those all the time viii Preface Some authors, knowing about the limitations of the cloud, would surely proclaim the web to be “unsafe at any speed;” Indeed, I once wrote an article that had this title (but with a question mark at the end, which does change the meaning) The bottom line is that even with its limitations today, such a claim would be pure hyperbole But it would be quite accurate to point out that the vast majority of the web makes with very weak assurance properties Moreover, although the web provides great support for secure transactions, the model it uses works for secure transmission of a credit card to a cloud provider and for secure delivery of the video you just licensed back to your laptop or Internet-connected TV, not for other styles of high-assurance computing Given the dismal security properties of the laptops, the computing industry views Web security as a pretty good story But could we extend this model to tackle a broader range of security challenges? We can then ask another question Is it possible that scalable high-assurance computing, outside what the cloud offers today, just is not needed? We emphasized the term “scalable” for a reason: the cloud is needed for large-scale computing; the methods of the past few decades were often successful in solving high-assurance computing challenges, but also limited to problems that ran on more modest scales The cloud is the place to turn when an application might involve tens of thousands of simultaneous users With six users, the cloud could be convenient and cheap, but is certainly not the only option Thus unless we can identify plenty of important examples of large-scale uses that will need high assurance, it might make sense to conclude that the cloud can deal with high-assurance in much the same way that it deals with credit card purchases: using smaller systems that are shielded from heavy loads and keep up with the demand because they aren’t really forced to work very hard There is no doubt that the weak-assurances of the cloud suffice for many purposes; a great many applications can be twisted to fit them The proof is right on our iPads and Android telephones: they work remarkably well and all sorts of amazing tricks and they this within the cloud model as currently deployed, and they even manage to twist the basic form of web security into so many forms that one could easily believe that the underlying mechanism is far more general than it really is Yet the situation would change if we tried to move more of today’s computing infrastructure as a whole to a cloud model Think about what high assurance really means Perhaps your first reaction is that the term mostly relates to a class of very esoteric and specialized systems that provide services for tasks such as air traffic control, banking, or perhaps management of electronic medical records and medical telemetry in critical care units The list goes on: one could add many kinds of military applications (those might need strong security, quick response, or other kinds of properties) There is a lot of talk about new ways of managing the electric power grid to achieve greater efficiency and to share power in a more nimble way over large regions, so that we can make more use of renewable electric generation capacity Many government services need to be highly assured And perhaps even non-politicians would prefer that it was a bit harder to hack their twitter, email and Facebook accounts Preface ix So here we have quite a few examples of high assurance applications: systems that genuinely need to the right thing, and to it at the right time, where we’re defining “right” in different ways depending on the case Yet the list did not include very many of what you might call bread-and-butter computing cases, which might lead you to conclude that high assurance is a niche area After all, not many of us work on air traffic control systems, and it is easy to make that case against migrating things like air traffic control to cloud models (even privately operated cloud models) Thus, it is not surprising that many developers assume that they not really work on systems of this kind We’re going to question that quick conclusion One reason is that the average enterprise has many high assurance subsystems playing surprisingly mundane roles; they operate the factory floor equipment, run the corporate payroll, and basically keep the lights on These are high assurance roles simply because if they are not performed correctly, the enterprise is harmed Of course not many run on the cloud today, but perhaps if cloud computing continues to gain in popularity and continues to drop in cost (and if the reliability of the cloud were just a touch higher), operators may start to make a case for migrating them to cloud settings This is just the area where scalability and high assurance seem to collide: if we imagine using the cloud to control vast numbers of physical things that can break or cause harm if controlled incorrectly, then we definitely encounter limitations that today’s cloud cannot easily surmount The cloud is wonderful for scalable delivery of insecure data, and adequate for scalable delivery of certain kinds of sensitive data, and for conducting relatively infrequent purchase-style transactions All of this works wonderfully well But the model does not fit nearly so well if we want to use it in high-assurance control applications This is a bit worrying, because the need for high assurance cloud-hosted control systems could easily become a large one if cloud computing starts to displace other styles of computing to any substantial degree, a trend the author believes to increasingly probable The root cause here is the tendency of the computing industry to standardize around majority platforms that then kill off competitors simply for economic reasons: lacking adequate investment, they wither and die As cloud computing has flourished, it has also become the primary platform for most kinds of application development, displacing many other options for reasons of cost, ease of development, and simply because the majority platform tends to attract the majority of developers Some of the most exciting future uses of computing presume that computers will penetrate into the home and car and office to such a degree that we will be able to start to intelligent, environmentally aware, dynamic control of those kinds of systems Traffic lights and water heaters will begin to be cloud-controlled systems Fragile, elderly patients will manage to live at home for many years, rather than in assisted living settings, because computing systems will play vital monitoring and assistance roles Cars will literally drive themselves on densely packed highways, at far higher speeds and with tighter spacings than today’s human drivers can manage Those kinds of visions of the future appear, at least superficially, to presume a new kind of high assurance cloud computing that appears, at least superficially, to x Preface be at odds with what today’s cloud platforms are able to Indeed, they appear, again superficially, to be at odds with those theorems we mentioned earlier If faulttolerant computing is impossible, how can we possibly trust computing systems in roles like these? If the cloud cannot offer high assurance properties, how can the US government possibly bet so heavily on the cloud in sensitive government and military applications? Accordingly, we must pose a follow-on question What are the consequences of putting a critical application on a technology base not conceived to support high assurance computing? The danger is that we could wander into a future in which computing applications, playing critical roles, simply cannot be trusted to so in a correct, secure, consistent manner This leads to the second and perhaps more controversial agenda of the present text: to educate the developer (be that a student or a professional in the field) about the architectures of these important new cloud computing platforms and about their limitations: not just what they can do, but also what they cannot Some of these limitations are relatively easily worked around; others, much less so We will not accept that even the latter kind of limitations are show-stoppers Instead, the book looks to a future well beyond what current cloud platforms can support We will ask where cloud computing might go next, how it can get there, and will seek to give the reader hands-on experience with the technologies that would enable that future cloud Some of these enablers exist in today’s commercial market place, but others are lacking Consequently, rather than teaching the reader about options that would be very hard to put into practice, we have taken the step of creating a new kind of cloud computing software library (all open source), intended to make the techniques we discuss here practical, so that readers can easily experiment with the ideas the book will cover, using them to build applications that target real cloud settings, and could be deployed and used even in large-scale, performance-intensive situations A consequence is that this text will view some technical options as being practical (and might even include exercises urging the reader to try them out him or herself using our library, or using one of those high-assurance technologies), and if you were to follow that advice, with a few hundred lines of code and a bit of debugging you would be able to run your highly assured solution on a real cloud platform, such as Amazon’s EC2 or Microsoft Azure Doing so could leave you with the impression would be that the technique is perfectly practical Yet if you were to ask one of those vendors, or some other major cloud vendor, what they think about this style of high-assured cloud computing, you might well be told that such services not belong in the cloud! Is it appropriate to include ideas that the industry has yet to adopt into a textbook intended for real developers who want to learn to build reliable cloud computing solutions? Many authors would decide not to so, and that decision point differentiates this text from others in the same general area We will not include concepts that we have not implemented in our Isis2 software library (you will hear more and more about Isis2 as we get to Parts II and III of the book, and are welcome to download it, free of any charges, and to use it as you like) or that someone we trust has not worked with in some hands-on sense—anything you read in this book is real enough 716 References Moser, L.E., Melliar-Smith, P.M., Agarwal, U.: Processor membership in asynchronous distributed systems IEEE Trans Parallel Distrib Syst 5(5), 459–473 (1994b) Moser, L.E., Melliar-Smith, P.M., Agarwal, D.A., Budhia, R.K., Lingley-Papadopoulos, C.A.: Totem: A fault-tolerant multicast group communication system Commun ACM 39(4), 54–63 (1996) Moss, J.E.: Nested transactions and reliable distributed computing In: Proceedings of the Second Symposium on Reliability in Distributed Software and Database Systems, pp 33–39 (1982) Mullender, S.J., et al.: Amoeba—A distributed operating system for the 1990s IEEE Comput 23(5), 44–53 (1990) Mummert, L.B., Ebling, M.R., Satyanarayanan, M.: Exploiting weak connectivity for mobile file access In: Proceedings of the Fifteenth Symposium on Operating Systems Principles, Copper Mountain Resort, CO, December 1995, pp 143–155 ACM Press, New York (1995) Also ACM Trans Comput Syst 13(1) (1996) Muthitacharoen, A., Chen, B., Mazieres, D.: A low-bandwidth network file system In: 18th ACM Symposium on Operating Systems Principles (SOSP ’01), Chateau Lake Louise, Banff, Canada, October 2001 Muthitacharoen, A., Morris, R., Gil, T., Ivy, B Chen: A read/write peer-to-peer file system In: Proceedings of the 5th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’02), Boston, Massachusetts, December 2002 National Bureau of Standards: Data Encryption Standard Federal Information Processing Standards Publication, vol 46 Government Printing Office, Washington (1977) Needham, R.M., Schroeder, M.D.: Using encryption for authentication in large networks of computers Commun ACM 21(12), 993–999 (1988) Neiger, G.: A new look at membership services In: Proceedings of the Fifteenth ACM Symposium on Principles of Distributed Computing, Vancouver (1996) In press Nelson, M., Welsh, B., Ousterhout, J.: Caching in the Sprite network file system In: Proceedings of the Eleventh ACM Symposium on Operating Systems Principles, Austin, November 1987 ACM Press, New York (1987) Also ACM Trans Comput Syst 6(1) (1988) Object Management Group and X/Open: Common Object Request Broker: Architecture and Specification Reference OMG 91.12.1 (1991) Oki, B., Liskov, B.: Viewstamped replication: A new primary copy method to support highlyavailable distributed systems In: Proceedings of the Seventh Annual ACM Symposium on Principles of Distributed Computing (PODC ’88), pp 8–17 ACM, New York (1988) Oki, B., Pfluegl, M., Siegel, A., Skeen, D.: The information bus-an architecture for extensible distributed systems In: Proceedings of the Thirteenth ACM Symposium on Operating Systems Principles, Asheville, NC, December 1993, pp 58–68 ACM Press, New York (1993) Ostrowski, K., Birman, K.: Storing and accessing live mashup content in the cloud SIGOPS Oper Syst Rev 44(2) (2009) Ostrowski, K., Birman, K., Dolev, D.: QuickSilver scalable multicast (QSM) In: 7th IEEE International Symposium on Network Computing and Applications (IEEE NCA 2008), Cambridge, MA, July 2008a Ostrowski, K., Birman, K., Dolev, D., Ahnn, J.H.: Programming with live distributed objects In: Vitek, J (ed.) 22nd European Conference on Object-Oriented Programming (ECOOP 2008), Paphos, Cyprus, July 7–11, 2008 Lecture Notes in Computer Science, vol 5142, pp 463–489 Springer, Berlin (2008b) Ousterhout, J.: Why aren’t operating systems getting faster as fast as hardware In: USENIX Summer Conference Proceedings, Anaheim, CA, pp 247–256 (1990) Ousterhout, J.: TCL and the TK Toolkit Addison-Wesley, Reading (1994) Ousterhout, J., Da Costa, H., Harrison, D., Kunze, J.A., Kupfer, M., Thompson, J.G.: A tracedriven analysis of the UNIX 4.2 BSD file system In: Proceedings of the Tenth ACM Symposium on Operating Systems Principles, Orcas Island, WA, December 1985, pp 15–24 ACM Press, New York (1985) Ousterhout, J., et al.: The sprite network operating system Computer 21(2), 23–36 (1988) References 717 Pai, V.S., Druschel, P., Zwaenepoel, W.: IO-Lite: A unified I/O buffering and caching system In: 3rd Symposium on Operating Systems Design and Implementation (OSDI), New Orleans, LA, February 1999 Patino-Martinez, M., Jimenez-Peris, R., Kemme, B., Alonso, G.: MIDDLE-R: Consistent database replication at the middleware level Trans Comput Syst 23(4) (2005) Patterson, D., Gibson, G., Katz, R.: A case for redundant arrays of inexpensive disks (RAID) In: Proceedings of the 1988 ACM Conference on Management of Data (SIGMOD), Chicago, June 1988, pp 109–116 (1988) Pedone, F., Guerraoui, R., Schiper, A.: Exploiting atomic broadcast in replicated databases In: Proceedings EuroPar 98 (1998) Petersen, K., Spreitzer, M., Terry, D., Theimer, M., Demers, A.: Flexible update propagation for weakly consistent replication In: Proceedings of the 16th ACM Symposium on Operating Systems Principles (SOSP-16), Saint Malo, France, October 5–8, pp 288–301 (1997) Peterson, L.: Preserving context information in an IPC abstraction In: Proceedings of the Sixth Symposium on Reliability in Distributed Software and Database Systems, March 1987, pp 22–31 IEEE Computer Society Press, New York (1987) Peterson, I.: Fatal Defect: Chasing Killer Computer Bugs Time Books/Random House, New York (1995) Peterson, L., Buchholz, N.C., Schlicting, R.D.: Preserving and using context information in interprocess communication ACM Trans Comput Syst 7(3), 217–246 (1989a) Peterson, L., Hutchinson, N., O’Malley, S., Abbott, M.: RPC in the x-kernel: Evaluating new design techniques In: Proceedings of the Twelfth Symposium on Operating Systems Principles, Litchfield Park, AZ, November 1989, pp 91–101 ACM Press, New York (1989b) Powell, D (ed.): Delta-4: A Generic Architecture for Dependable Distributed Computing Springer-Verlag ESPRIT Research Reports, vol I, Project 818/2252 (1991) Powell, D.: Lessons learned from Delta-4 IEEE MICRO 14(4), 36–47 (1994) Powell, D.: Introduction to special section on group communication Commun ACM 39(4), 50–53 (1996) Pritchett, D.: BASE: An acid alternative Queue 6(3), 48–55 (2008) Pu, D.: Relaxing the limitations of serializable transactions in distributed systems Oper Syst Rev 27(2), 66–71 (1993) (Special issue on the Workshop on Operating Systems Principles at Le Mont St Michel, France) Rabin, M.: Randomized Byzantine generals In: Proceedings of the Twenty-Fourth Annual Symposium on Foundations of Computer Science, pp 403–409 IEEE Computer Society Press, New York (1983) Rashid, R.F.: Threads of a new system UNIX Rev 4, 37–49 (1986) Reed, D.P., Kanodia, R.K.: Synchronization with eventcounts and sequencers Commun ACM 22(2), 115–123 (1979) Reiher, P., et al.: Resolving file conflicts in the ficus file system In: Proceedings of the Summer USENIX Conference, June 1994, pp 183–195 (1994) Reiter, M.K.: A security architecture for fault-tolerant systems Ph.D diss., Cornell University, August (1993) Also Technical Report, Department of Computer Science, Cornell University Reiter, M.K.: Secure agreement protocols: Reliable and atomic group multicast in rampart In: Proceedings of the Second ACM Conference on Computer and Communications Security, Oakland, November 1994, pp 68–80 (1994a) Reiter, M.K., A secure group membership protocol In: Proceedings of the 1994 Symposium on Research in Security and Privacy, Oakland, May 1994, pp 89–99 IEEE Computer Society Press, New York (1994b) Reiter, M.K.: Distributing trust with the Rampart toolkit Commun ACM 39(4), 71–75 (1996) Reiter, M.K., Birman, K.P.: How to securely replicate services ACM Trans Program Lang Syst 16(3), 986–1009 (1994) Reiter, M.K., Birman, K.P., Gong, L.: Integrating security in a group-oriented distributed system In: Proceedings of the IEEE Symposium on Research in Security and Privacy, Oakland, May 1992, pp 18–32 IEEE Computer Society Press, New York (1992) 718 References Reiter, M., Birman, K.P., van Renesse, R.: A security architecture for fault-tolerant systems Trans Comput Syst 12(4), 340–371 (1994) Ricciardi, A.M.: The group membership problem in asynchronous systems Ph.D diss., Cornell University, January (1993) Ricciardi, A.: The impossibility of (repeated) reliable broadcast Technical Report TR-PDS-1996003, Department of Electrical and Computer Engineering, University of Texas, Austin, April (1996) Ricciardi, A., Birman, K.P.: Using process groups to implement failure detection in asynchronous environments In: Proceedings of the Eleventh ACM Symposium on Principles of Distributed Computing, Quebec, August 1991, pp 341–351 ACM Press, New York (1991) Ricciardi, A., Birman, K.P., Stephenson, P.: The cost of order in asynchronous systems In: WDAG 1992 Lecture Notes in Computer Science, pp 329–345 Springer, Berlin (1992) Ritchie, D.M.: A stream input-output system AT&T Bell Lab Tech J 63(8), 1897–1910 (1984) Rivest, R.L., Shamir, A., Adleman, L.: A method for obtaining digital signatures and public key cryptosystems Commun ACM 22(4), 120–126 (1978) Rodeh, O., Birman, K., Dolev, D.: Using AVL trees for fault-tolerant group key management Int J Inf Secur 1(2), 84–99 (2002) Rodrigues, L., Verissimo, P.: xAMP: A MultiPrimitive group communications service In: Proceedings of the Eleventh Symposium on Reliable Distributed Systems, Houston, October 1989 IEEE Computer Society Press, New York (1989) Rodrigues, L., Verissimo, P.: Causal separators for large-scale multicast communication In: Proceedings of the Fifteenth International Conference on Distributed Computing Systems, May 1995, pp 83–91 (1995) Rodrigues, L., Verissimo, P., Rufino, J.: A low-level processor group membership protocol for LANs In: Proceedings of the Thirteenth International Conference on Distributed Computing Systems, May 1993, pp 541–550 (1993) Rodrigues, L., Guo, K., Verissimo, P., Birman, K.P.: A dynamic light-weight group service J Parallel Distrib Comput 60, 1449–1479 (2000) Rosenblum, M., Ousterhout, J.K.: The design and implementation of a log-structured file system In: Proceedings of the Twelfth ACM Symposium on Operating Systems Principles, Asilomar, CA, October 1991, pp 1–15 ACM Press, New York (1991) Also ACM Trans Comput Syst 10(1), 26–52 (1992) Rowe, L.A., Smith, B.C.: A continuous media player In: Proceedings of the Third International Workshop on Network and Operating Systems Support for Digital Audio and Video, San Diego, CA, November 1992 Rozier, M., et al.: Chorus distributed operating system Comput Syst J 1(4), 305–370 (1988a) Rozier, M., et al.: The Chorus distributed system Comput Syst 299–328 (1988b) Sabel, L., Marzullo, K.: Simulating fail-stop in asynchronous distributed systems In: Proceedings of the Thirteenth Symposium on Reliable Distributed Systems, Dana Point, CA, October 1994, pp 138–147 IEEE Computer Society Press, New York (1994) Saltzer, J.H., Reed, D.P., Clark, D.D.: End-to-end arguments in system design ACM Trans Comput Syst 39(4) (1990) Santry, D.S., Feeley, M.J., Hutchinson, N.C., Veitch, A.C., Carton, R.W., Ofir, J.: Deciding when to forget in the Elephant file system In: Proceedings of the Seventeenth ACM Symposium on Operating Systems Principles, SOSP ’99, Charleston, South Carolina, United States, pp 110–123 ACM, New York (1999) Satyanarayanan, M., et al.: The ITC distributed file system: Principles and design In: Proceedings of the Tenth ACM Symposium on Operating Systems Principles, Orcas Island, WA, December 1985, pp 35–50 ACM Press, New York (1985) Satyanarayanan, M., et al.: Integrating security in a large distributed system ACM Trans Comput Syst 7(3), 247–280 (1989) Schantz, R.E., Thomas, R.H., Bono, G.: The architecture of the Chronus distributed operating system In: Proceedings of the Sixth International Conference on Distributed Computing Systems, New York, June 1986, pp 250–259 IEEE Computer Society Press, New York (1986) References 719 Schiller, J.I.: Secure distributed computing Sci Am 72–76 (1994) Schiper, A., Raynal, M.: From group communication to transactions in distributed systems Commun ACM 39(4), 84–87 (1996) Schiper, A., Eggli, J., Sandoz, A.: A new algorithm to implement causal ordering In: Proceedings of the Third International Workshop on Distributed Algorithms Lecture Notes in Computer Science, vol 392, pp 219–232 Springer, Berlin (1989) Schiper, A., Shvartsman, A.A., Weatherspoon, H., Zhao, B.: Future Directions in Distributed Computing, Research and Position Papers Springer, Berlin (2003) Schmuck, F.: The use of efficient broadcast primitives in asynchronous distributed systems Ph.D diss., Cornell University, August (1988) Also Technical Report, Department of Computer Science, Cornell University Schmuck, F., Wyllie, J.: Experience with transactions in quicksilver In: Proceedings of the Twelfth ACM Symposium on Operating Systems Principles, Asilomar, CA, October 1991, pp 239–252 ACM Press, New York (1991) Schneider, F.B.: Byzantine generals in action: Implementing fail-stop processors ACM Trans Comput Syst 2(2), 145–154 (1984) Schneider, F.B.: The StateMachine approach: A tutorial In: Proceedings of the Workshop on FaultTolerant Distributed Computing, Asilomar, CA Lecture Notes on Computer Science, vol 448, pp 18–41 Springer, Berlin (1988) Schneider, F.B.: Implementing fault-tolerant services using the StateMachine approach ACM Comput Surv 22(4), 299–319 (1990) Schneider, F.B., Walsh, K., Sirer, E.G.: Nexus authorization logic (NAL): Design rationale and applications ACM Trans Inf Syst Secur 14(1), (2011) 28 pages Seltzer, M.: Transaction support in a log-structured file system In: Proceedings of the Ninth International Conference on Data Engineering, April 1993 Shieh, A., Sirer, E.G., Schneider, F.B.: NetQuery: A knowledge plane for reasoning about network properties In: Proceedings of the ACM SIGCOMM 2011 Conference on SIGCOMM (SIGCOMM ’11), pp 278–289 ACM, New York (2011) Shraer, A., Martin, J.-P., Malkhi, D., Keidar, I.: Data-centric reconfiguration with network-attached disks In: Large-Scale Distributed Systems and Middleware (LADIS 2010), July 2010 Shroeder, M., Burrows, M.: Performance of firefly RPC In: Proceedings of the Eleventh ACM Symposium on Operating Systems Principles, Litchfield Springs, AZ, December 1989, pp 83– 90 (1989) Also ACM Trans Comput Syst 8(1), 1–17 (1990) Siegal, A.: Performance in flexible distributed file systems Ph.D diss., Cornell University, February (1992) Also Technical Report TR-92-1266, Department of Computer Science, Cornell University Siegel, A., Birman, K.P., Marzullo, K.: Deceit: A flexible distributed file system Technical Report 89-1042, Department of Computer Science, Cornell University (1989) Simons, B., Welch, J.N., Lynch, N.: An overview of clock synchronization In: Simons, B., Spector, A (eds.) Fault-Tolerant Distributed Computing Lecture Notes in Computer Science, vol 448, pp 84–96 Springer, Berlin (1990) Skeen, D.: Crash recovery in a distributed database system Ph.D diss., Department of EECS, University of California, Berkeley, June (1982a) Skeen, D.: A quorum-based commit protocol In: Proceedings of the Berkeley Workshop on Distributed Data Management and Computer Networks, Berkeley, CA, February 1982, pp 69–80 (1982b) Skeen, D.: Determining the last process to fail ACM Trans Comput Syst 3(1), 15–30 (1985) Spasojevic, M., Satyanarayanan, M.: An empirical study of a wide area distributed file system ACM Trans Comput Syst 14(2) (1996) Spector, A.: Distributed transactions for reliable systems In: Proceedings of the Tenth ACM Symposium on Operating Systems Principles, Orcas Island, WA, December 1985, pp 12–146 (1985) Srikanth, T.K., Toueg, S.: Optimal clock synchronization J ACM 34(3), 626–645 (1987) 720 References Srinivasan, V., Mogul, J.: Spritely NFS: Experiments with cache consistency protocols In: Proceedings of the Eleventh ACM Symposium on Operating Systems Principles, Litchfield Springs, AZ, December 1989, pp 45–57 (1989) Steiner, J.G., Neuman, B.C., Schiller, J.I.: Kerberos: An authentication service for open network systems In: Proceedings of the 1988 USENIX Winter Conference, Dallas, February 1988, pp 191–202 (1988) Stephenson, P.: Fast causal multicast Ph.D diss., Cornell University, February (1991) Also Technical Report, Department of Computer Science, Cornell University Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup service for Internet applications In: ACM SIGCOMM 2001, San Diego, CA, August 2001, pp 149–160 (2001) Strom, R.E., Banavar, G., Chandra, T.D., Kaplan, M., Miller, K., Mukherjee, B., Sturman, D.C., Ward, M.: Gryphon: An information flow based approach to message brokering CoRR (1998) Tanenbaum, A.: Computer Networks, 2nd edn Prentice Hall, Englewood Cliffs (1988) Tanenbaum, A., van Renesse, R.: A critique of the remote procedure call paradigm In: Proceedings of the EUTECO ’88 Conference, Vienna, April 1988, pp 775–783 (1988) Terry, D.B., et al.: Managing update conflicts in a weakly connected replicated storage system In: Proceedings of the Fifteenth Symposium on Operating Systems Principles, Copper Mountain Resort, CO, December 1995, pp 172–183 ACM Press, New York (1995) Thekkath, C.A., Levy, H.M.: Limits to low-latency communication on high-speed networks ACM Trans Comput Syst 11(2), 179–203 (1993) Thekkath, C., Mann, T., Lee, E.: Frangipani: A scalable distributed file system In: 16th ACM Symposium on Operating Systems Principles (SOSP), Saint-Malo, France, October 1997 Thomas, T.: A majority consensus approach to concurrency control for multiple copy databases ACM Trans Database Syst 4(2), 180–209 (1979) Tock, Y., Naaman, N., Harpaz, A., Gershinsky, G.: Hierarchical clustering of message flows in a multicast data dissemination system In: IASTED PDCS (2005) van Renesse, R.: Causal controversy at Le Mont St.-Michel Oper Syst Rev 27(2), 44–53 (1993) van Renesse, R.: Why bother with CATOCS? Oper Syst Rev 28(1), 22–27 (1994) van Renesse, R.: Paxos made moderately simple Technical report, Cornell University, March (2011) van Renesse, R., Schneider, F.B.: Chain replication for supporting high throughput and availability In: Sixth Symposium on Operating Systems Design and Implementation (OSDI ’04), San Francisco, CA, December 2004 van Renesse, R., van Staveren, H., Tanenbaum, A.: Performance of the world’s fastest operating system Oper Syst Rev 22(4), 25–34 (1988) van Renesse, R., van Staveren, H., Tanenbaum, A.: The performance of the Amoeba distributed operating system Softw Pract Exp 19(3), 223–234 (1989) van Renesse, R., Birman, K.P., Friedman, R., Hayden, M., Karr, D.: A framework for protocol composition in Horus In: Proceedings of the Fourteenth Symposium on the Principles of Distributed Computing, Ottawa, August 1995, pp 80–89 ACM Press, New York (1995) van Renesse, R., Birman, K.P., Maffeis, S.: Horus: A flexible group communication system Commun ACM 39(4), 76–83 (1996) van Renesse, R., Birman, K.P., Hayden, M., Vaysburd, A., Karr, D.: Building adaptive systems using Ensemble In: Software—Practice and Experience, August 1998 van Renesse, R., Birman, K.P., Vogels, W.: Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining ACM Trans Comput Syst 21(2), 164–206 (2003) Verissimo, P.: Real-time communication In: Mullender, S.J (ed.) Distributed Systems, 2nd edn., pp 447–490 Addison-Wesley/ACM Press, Reading (1993) Verissimo, P.: Ordering and timeliness requirements of dependable real-time programs J RealTime Syst 7(2), 105–128 (1994) Verissimo, P.: Uncertainty and predictability: Can they be reconciled In: Future Directions in Distributed Computing, pp 108–113 Springer, Berlin (2003) References 721 Verissimo, P., Rodrigues, L.: A-posteriori agreement for fault-tolerant clock synchronization on broadcast networks In: Proceedings of the Twenty-Second International Symposium on FaultTolerant Computing, Boston, July 1992 Vigfusson, Y.: Affinity in distributed systems PhD dissertation Cornell University, Sept (2009) (Degree conferred Feb 2010) Vigfusson, Y., Birman, K., Huang, Q., Nataraj, D.P.: GO: Platform support for Gossip applications In: IEEE P2P 2009, Seattle, WA, September 9–11, pp 222–231 (2009) Vigfusson, Y., Abu-Libdeh, H., Balakrishnan, M., Birman, K., Burgess, R., Li, H., Chockler, G., Tock, Y.: Dr Multicast: Rx for data center communication scalability In: Eurosys, ACM SIGOPS, Paris, France, April 2010, pp 349–362 (2010) Vitenberg, R., Chockler, G.V., Keidar, I.: Group communication specifications: A comprehensive study ACM Comput Surv 33(4) (2001) Vogels, W.: The private investigator Technical Report, Department of Computer Science, Cornell University, April (1996) Vogels, W.: File system usage in Windows NT 4.0 In: Proceedings of the Seventeenth ACM Symposium on Operating Systems Principles, SOSP’99, Charleston, South Carolina, United States, pp 93–109 ACM, New York (1999) Vogels, W.: Eventually consistent—Revisited http://www.allthingsdistributed.com/2008/12/ eventually_consistent.html Dec (2008) Vogels, W., Re, C.: WS-membership—failure management in a Web-Services World In: 12th International World Wide Web Conference, Budapest, Hungary, May 2003 von Eicken, T., Culler, D.E., Goldstein, S.C., Schauser, K.E.: Active messages: A mechanism for integrated communication and computation In: Proceedings of the Nineteenth International Symposium on Computer Architecture, May 1992, pp 256–266 (1992) von Eicken, T., Basu, A., Buch, V., Vogels, W.: U-Net: A user-level network interface for parallel and distributed computing In: Proceedings of the Fifteenth Symposium on Operating Systems Principles, Copper Mountain Resort, CO, December 1995, pp 40–53 ACM Press, New York (1995) Voulgaris, S., van Steen, M.: Epidemic-style management of semantic overlays for content-based searching In: Euro-Par 2005, pp 1143–1152 (2005) Voulgaris, S., Kermarrec, A.M., Massoulié, L., van Steen, M.: Exploiting semantic proximity in peer-to-peer content searching In: FTDCS 2004, pp 238–243 (2004) Voulgaris, S., van Steen, M., Iwanicki, K.: Proactive gossip-based management of semantic overlay networks Concurr Comput 19(17), 2299–2311 (2007) Wahbe, R., Lucco, S., Anderson, T., Graham, S.: Efficient software-based fault isolation In: Proceedings of the Thirteenth ACM Symposium on Operating Systems Principles, Asheville, NC, December 1993, pp 203–216 ACM Press, New York (1993) Waldman, M., Mazières, D.: Tangler: A censorship-resistant publishing system based on document entanglements In: Proceedings of the 8th ACM Conference on Computer and Communications Security, November, pp 126–135 (2001) Walter, B., et al.: The locus distributed operating system In: Proceedings of the Ninth ACM Symposium on Operating Systems Principles, Bretton Woods, NH, October 1993, pp 49–70 (1993) Wang, Y., Keller, E., Biskeborn, B., van der Merwe, J., Rexford, J.: Virtual routers on the move: Live router migration as a network-management primitive In: Proc ACM SIGCOMM, August 2008 Weatherspoon, H., Kubiatowicz, J.: Erasure coding vs replication: A quantitative comparison In: IPTPS 2002, pp 328–338 (2002) Weatherspoon, H., Ganesh, L., Marian, T., Balakrishnan, M., Birman, K.: Smoke and mirrors: Reflecting files at a geographically remote location without loss of performance In: Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST), February 2009 Welsh, M., Culler, D., Brewer, E.: SEDA: An architecture for well-conditioned, scalable Internet services In: 18th Symposium on Operating Systems Principles (SOSP), Banff, Canada, October 2001 722 References Wilkes, J., et al.: The HP AutoRAID hierarchical storage system In: Proceedings of the Fifteenth Symposium on Operating Systems Principles, Copper Mountain Resort, CO, December 1995, pp 96–108 ACM Press, New York (1995) Also ACM Trans Comput Syst 13(1) (1996) Wong, B., Slivkins, A., Sirer, E.G.: Meridian: A lightweight network location service without virtual coordinates In: Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM ’05), pp 85–96 ACM, New York (2005) Wood, M.D.: Fault-tolerant management of distributed applications using a reactive system architecture Ph.D diss., Cornell University, December (1991) Also Technical Report TR 91-1252, Department of Computer Science, Cornell University Wood, M.D.: Replicated RPC using Amoeba closed-group communication In: Proceedings of the Twelfth International Conference on Distributed Computing Systems, Pittsburgh (1993) Yeger-Lotem, E., Keidar, I., Dolev, D.: Dynamic voting for consistent primary components In: 16th ACM Symposium on Principles of Distributed Computing (PODC’97), pp 63–71 (1997) Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, U., Gunda, P., Currey, J.: DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language In: ACM Symposium on Operating System Design and Implementation (OSDI), San Diego, CA, December 8–10, 2008 Zagorodnov, D., Marzullo, K., Alvisi, L., Bressoud, T.C.: Practical and low-overhead masking of failures of TCP-based servers ACM Trans Comput Syst 27(2) (2009) Zhao, B., Duan, Y., Huang, L., Joseph, A.D., Kubiatowicz, J.: Brocade: Landmark routing on overlay networks In: IPTPS 2002, pp 34–44 (2002a) Zhao, Duan, B.Y., Huang, L., Joseph, A.D., Kubiatowicz, J.D.: Brocade: Landmark routing on overlay networks In: First International Workshop on Peer-to-Peer Systems (IPTPS), Cambridge, MA, March 2002b Index A A-posteriori clock synchronization, 575 Acceptor processes (Paxos), 463 Access control technologies, 551 Accidental failures compared to attack, 295 ACE (Adaptive Computing Environment), 250 ACID database consistency model, 448 ACID properties in database systems, 150 ACID properties of a database system, 241 Active messages, 276 Active Registry (Windows), 201 AES (Advanced Encryption System), as used web security, 83 AFS, 227–230, 232, 235, 552 Agarwal’s replication algorithm for transactional databases, 605 Aggregation, 625 Amnesia-freedom in a first-tier cloud service, 448 Amnesia-freedom in cloud services, 374 ANSA project, 250 Architectural structures for reliable systems, 510 AS/400 database server product, 597 ASP (Application Service Page), 72 Astrolabe, 623 Asynchronous consensus problem, 362 Asynchronous model, 317, 360 Asynchronous system (as used in FLP result), 21 At least once semantics, 206 At most once semantics, 206 ATM, 282 ATOM web page encoding, 74 Atomic rename operation, 245 Atomicity, 241, 383 Attack spectrum seen in the Internet, 140 Authentication, 202, 227, 228, 475, 545, 547, 554, 559, 567, 568 in Kerberos, 547 Authentication (with wrappers), 484 Authority chains in secure systems, 564 Authorization, 298 Availability in secure systems, 559, 567 Availability (as used in CAP), 23 Azure, 473 Azure, used for cloud development, 473 B BASE methodology, 28, 448 Beehive, 452 Beehive one-hop DHT, 128 BGP (Border Gateway Protocol), 109 BiFocals network instrumentation, 167 BigTable, vi, 443, 452 BigTable (Google’s scalable row-column store), 159 Bimodal multicast, 609 Binding, 187, 195, 259, 265 BitTorrent, vi, 12, 15, 136 Bloom filters (used to implement IP multicast routing), 14 Bohrbugs, 289 Brewer’s CAP conjection, 22 Broadcast storms, 11 Byzantine Agreement, 355 Byzantine Agreement used in highly robust servers, 359 Byzantine failure model, 63, 354–361, 365, 387, 420, 547, 568 lower bounds, 356 Byzantine failure model, used for cloud computing, 357 Byzantine Quorum Systems, 359 C Cache coherency in Sprite, 229 Caching, 214, 224 Caching web proxies consistency options, 493 K.P Birman, Guide to Reliable Distributed Systems, Texts in Computer Science, DOI 10.1007/978-1-4471-2416-0, © Springer-Verlag London Limited 2012 723 724 CAP Conjecture and Theorem, 22 CAP Theorem, 148 CAP theorem, 448 CASD protocols, 577 Cassandra key-value store, 164 Causal gap freedom, 412 Causal order, 317 in overlapping process groups, 411 with multiple, overlapping groups, 397 CausalSend, 370, 388, 403, 434 compared to synchronous multicast protocol, 583 in overlapping process groups, 411 Chains of authority, 564 Chord DHT, 123 Chubby, vi, 26 Chubby (Google’s locking service), 165 Client mobility, 95 Client/server computing, 185, 245 Clients of a process group, 399 Clock synchronization, 571 Closely synchronous execution, 421 Cloud computing: replicating soft-state, 497 Cloud security limitations, 543 Cloud Storage, 180 Cluster computing used in telecommunications coprocessor, 586 CODA, 229 Coherent cache, 73 Coherent caching Web proxy, 492 Commercial off the shelf (COTS), 474 Communication from a group to a non-member, 402 Communication from non-members to a group, 399 Communications segment, 279 Complexity as a threat to reliability, 292 Components of a reliable distributed computing system, 63 Computer network, 50 Concurrency control, 240 Concurrency control tools, 488 Concurrency in cloud applications, 73 Conservative scheme, 412 Consistency, 61, 301, 458, 459, 599 Consistency, need for in soft-state services, 443 Consistency (as used in CAP), 23 Consistency in caching web proxies, 510 Consistency (in cloud settings), 34 Consistency in network routing, 115 Consistent and inconsistent cuts, 318, 394 Consistent caching, 484 Continuous availability, 61 Index Continuous Media Toolkit (CMT), 520 Controlling and monitoring an insulin pump from a cloud service, 29 Convoy phenomenon in reliable systems, 534 Coordinator-cohort algorithm, 440 Corba, xiii, 249, 252–256, 258–260, 262–265, 293, 481, 522, 523 event notification service, 262 hardening applications with Horus/Electra, 522 inter-object broker protocol (IOP), 264 interface definition language (IDL), 255, 260 introducing reliability technologies with, 477 life cycle service, 264 naming service, 262 object request broker (ORB), 254, 261 Orbix example, 257 persistent object service, 264 rebinding to a different server after failure, 259 reference model, 254 reliability issues, 259 reliability properties of Corba solutions, 265 transactional services, 259, 264 viewed as a wrapper technology, 477 CORBA Fault Tolerance Standard, 477 Correct specification, 61 COTS (Commercial off the shelf), 474 Cyclic UDP, 520 D Data access patterns in distributed file systems, 229 Data aggregation, 625 Data mining, 623 Data privacy (in cloud settings), 34 Data replication, 35 Data replication in cloud systems, 12 DCE, xiii, 202 Defense against intrusion and attack, 295 Denial of access in secure systems, 559 DES, 556 Detecting failures, 294 DHTs (Distributed Hash Tables), 122 Dialback security mechanisms, 549 Dining Philosophers problem, 414 DiskLogger, 447, 463 Distances within the Internet, 55 Distributed commit problem, 323 Distributed computing system, 50 Index Distributed database systems, 236, 237 abort viewed as a “tool”, 242 ACID properties, 241 concurrency control, 239, 592 nested transactions, 594 serializability, 238 state shared by clients and server, 241 transactional model, 238, 587 write-ahead log, 589 Distributed programming languages, 488 Distributed shared memory, 499, 505 Distributed system control, 623 Distributed transaction, 592 DNS, 200 Dr Multicast, 11, 17 Dryad/Linq, 488 DSM, 505 Durability (as implied by CAP definition of consistency), 23 Dynamic membership model, 313, 314, 339, 458 Dynamic uniformity, 601 Dynamo, 443, 452 Dynamo (Amazon’s scalable key-value store), 162 E Electra, 522 Embarrassing parallelism, 73 Encryption used in virtual private networks, 551 ENS, 262 Ensemble system relationship with Horus, 509 Enterprise web servers, 510 Eternal System (UCSD implementation of CORBA Fault-Tolerance Standard), 477 Event notification service, 262 Exactly once semantics, 206 Expander graphs (in gossip peer selection), 133 Expansion property of a gossip overlay graph, 616 Extended virtual synchrony, 424 Extensible routers, 116 External Data Representation, 197 F Fail-safe behavior, 34 Fail-stop failures, 62 Failover in Corba, 477 Failure detectors, 294 Failures, 62 725 False sharing, 504, 505 Fault-tolerance, 61, 484, 506 Fault-Tolerance (FLP result, CAP Theorem), 21 Fault-tolerance (in cloud settings), 34 Fault-tolerance tools, 488 Fault-tolerant real-time control, 573 Fbufs, 274 Ficus, 232 Filtering actions as a way to enforce security, 566 Firefly RPC costs, 271 Firewall protection (with wrappers), 484 Firewalls, 475, 548 First tier cloud services, 155 Flash, 74 Flush, 416 Flush primitive, 374 G Gap freedom guarantee, 424 GFS, 233, 247 Global file systems, vi Globally total order, 414 GMS property summary, 385 Google AppEngine, 473 Google File System (GFS), 26, 233, 247 Gossip Objects (GO), 633 GPS receivers, 571, 576, 694 Group address, 516 Group communication and Java applets, 510 Group communication in Web applications, 492 Group membership protocol, 210, 343, 468 Group membership service, 385 extensions to allow partition and merge, 352 primary partition properties, 351 summary of properties, 385 Group object, 516 Guaranteed execution tools, 488 H Hadoop, vi, 27, 174, 443 Halting failures, 62 Hard state in the cloud, 157 Hard-state replication, 453 Hard-state replication using virtual synchrony model, 442 Hardware cryptographic protection, 551 Harp, 225 HCPI, 518 Health care records in cloud settings, 28 Heisenbugs, 289, 422 726 High assurance soft-state services, 443 High availability, 61 High availability TCP sessions, 113 High-assurance for health-care systems, 497 Highly available routers, 113 Horus system basic performance, 523, 535 Horus Common Protocol Interface (HCPI), 518 protocol accelerator, 526 real-time protocols, 584 replication in the Web, 513 robust groupware application, 519 Hostile environments, 295 HPC (High Performance Computing), on Cloud Platforms, 177 HRT (Hardware Root of Trust), 85 HTML (Hypertext Markup language, 71 HTTP (Hypertext Transfer Protocol, 71 HTTPS (Secure version of HTTP protocol), 84 I ID-90, 278 IDL, 255 Impossibility results for the asynchronous model, 364 IN coprocessor fault-tolerance, 586 Information warfare, 295 Insulin pump (controlled by a cloud application), 28 Intentional threats, 295 Internet Indirection Infrastructure (I3), 135 Internet of things, 76 IP multicast, 12 IP multicast (management of the IPMC address space), 17 Isis2 System, 509 Iterated multicast, 400 J J2EE, 203, 204, 249, 252, 253, 625 Java groupware opportunities, 492 integrated with group communication tools, 510 Java applets structured as object groups, 492 Java Enterprise Edition, 249 JavaGroups (JGroups), 510 JavaGroups (JGroups), in JBOSS, 489 Javascript, 74 JBOSS, 489 JDBC, 625 JGroups, vi JVM (Java Virtual Machine), 74 Index K Kelips one-hop DHT, 129 Kerberos, 202, 203, 227, 545, 548, 554, 557, 559, 567 Kerberos authentication, 228 Key-value storage services, 158 Key-value stores, 443 L Learner processes (Paxos), 463 LFS, 235 Lightweight Probabilistic Broadcast Protocol, 616 Lightweight process groups, 417 Lightweight remote procedure call, 271–274 Live Distributed Objects, 686 Load balancing, 437 Local procedure call, 206 Locally total order, 414 Log-structured file system, 235 Logical clock, 317 used to implement CausalSend, 389 Long-haul connection (security exposures), 551 Lotus Notes, 236 LPC, 206 M Maelstrom network encoding scheme, 167 Malkhi, 447, 463 Management information base (MIB), 625 MapReduce, vi, 27, 174, 443 MARS, 573 Marshalling, 197 Master-slave parallel programming style, 278 Measurements of distributed file systems, 229 Memcached, vi, 443, 452 Memcached (scalable key-value store), 158 Meridian network coordinates system, 55 Message, 50 Message Queuing Middleware, 169 MIB used for failure detection, 294 Microsoft NET, 189, 191, 192, 195, 252, 253, 291, 371, 625 Microsoft Azure, 473 Microsoft Visual Studio, used for cloud development, 473 Mobile computing, 95 MOM (Message Oriented Middleware), 169 Monitoring, 623 Monitoring and controlling an insulin pump from a cloud service, 29 Index Monitoring and logging (with wrappers), 485 Multi-level architecture for group computing tools, 510 Multi-phase commit, 592 Multicast consistency issues, 468 ordering domains, 415 ordering protocols (causal and total), 386 stability, 412 totally ordered, 397 Multicast delivery flush used to force strong durability, 374 Multicast delivery property, 374 Multicore client systems, 97 N Network Coordinates, 55 Network database server, 186, 236, 237 Network file server, 186, 219, 237 Network file servers replication for high availability, 225 Network File System, 197 Network Information Service, 200 Network partitioning, 62, 424 Network Perspective, 101 Network Reliability, 101 Nexus: A secure microkernel, 546 NFS, xiii, 187, 197, 209, 210, 223–227, 230–232, 235, 552, 553, 560 prefetching, 224 reliability of, 225 security of, 560 security problems with, 553 NIS, 200 NET, 189, 191, 195, 204, 249, 252, 253, 489, 625 O Object groups, 477, 522 Object orientation groupware solutions, 478 technologies supporting, 478 viewed as a form of wrapper, 478 Object Request Broker, 254 ODBC, 625 Off the shelf components (COTS), 474 OLE-2 introducing reliability technologies with, 477 viewed as a wrapper technology, 477 Omission failures, 62 On-the-fly security enforcement, 566 ONC, xiii, 488, 560 security of, 560 727 One-hop DHTs, 128, 129 OpenFlow standard, 105 Orbix, 257 Orbix+Isis, 522 OrderedCausalSend, 371 OrderedSend, 371, 403, 434 locally total and globally total ordering, 414 Orleans distributed programming language, 488 Overhead of layered protocols, 529 Overlay Networks, 118 P P2P protocols, 609 Packet filter, 475 Packet sniffers, 548 Parallel computing (communication support), 278 Partition failures, 424 Partition Tolerance (as used in CAP), 23 Passwords, 548 Pastry DHT, 126 Paxos, 380, 387, 427, 447, 462, 463 Paxos (history of the area), 303 Paxos protocol, 463 Pbcast, 610 Peer-to-peer aggregation, 625 Peer-to-peer communication, 609 Performance, 62 Performance issues in file systems, 229 Persistent data, 588 PNUTS, 452 PNUTS key-value store, 164 Potential causality, 317 Potential groupware uses of the Web, 492 Power law distributions, 153 Prefetching in NFS, 224 Prefetching versus whole-file transfer, 224 Primary component of a partitioned network, 424, 466 Primary partition model, 343 Primary-backup fault-tolerance, 438 Privacy, 61 Private investigator (failure detection scheme), 294 ProbabilisticSend, 610 Protocol, 60 Protocol stack, 514 Publish subscribe process group implementation, 519 Publish-subscribe implemented over scalable multicast, 620 Publish-subscribe model, 137 728 Q Quality of service negotiation (with wrappers), 485 QuickSilver, 244, 620 Quorum replication, 433 R RAID, 226 Real-time, 573 CASD protocol, 577 fault-tolerant, in MARS, 573 Horus protocols, 584 Real-time virtual synchrony, 586 Recoverability, 61, 588 Release consistency, 502 Reliability in distributed computing systems, 61 Reliability of NFS, 224 Reliability standards (for web services), 81 Reliable multicast in synchronous systems, 577 Remote procedure call, 185, 186, 189 authentication service, 202 binding, 195 error handling, 194 lightweight, 272 marshalling, 197 naming service, 200 over TCP, 211 performance issues, 271 protocol, 204 replay problem, 206 use in reliable distributed systems, 208 Remote procedure call stub generation, 191 Rename (atomic operation), 245 Replicated data, 225, 421, 430 best solution, 433 high performance transactional scheme, 603 in transactional systems, 593 with Byzantine failures, 354 Replicated data tools, 488 Replication algorithm for transactional databases, 605 Replication and load-balancing in Web servers, 492 Replication of web servers, 510 RON (Resilient Overlay Network), 119 RPC, 186 RSA, 555 RSS Feed, 74 Runtime security enforcement, 566 Index S Safe execution of downloaded code, 87 SafeSend, 372, 387 SafeSend, with a DiskLogger durability method, 447, 463 SafeSend protocol, 463 Scalability, 61 Scalability analysis of Spread Toolkit, 535 Scalable data mining, 623 Scalable virtual synchrony, 617 Security, 61, 202, 207, 265, 506, 543 and Availability, 567 by filtering operations on-the-fly, 566 enclave, 484 in cloud systems, 34 in Isis2 , 543 policy languages and inference, 564 standards (for web services), 83 with wrappers, 484 SEDA, 284 Selecting peers in a gossip protocol, 617 Self-optimizing mesh construction, 133 Send, 370 Send, 388 Send, 403 Sharded data, 443 Sharded data (updated using ordered multicast), 414 Sharding, 452 Shared memory used for ATM communication in U-Net, 280 Shared memory tools, 488 Sienna content-based publish-subscribe system, 137 Simple Object Access Protocol, 187 Singularity distributed programming environment, 488 Sliver, 633 Small World Graphs (importance in gossip peer selection), 133 Smart power grid, 497 SMFS: the Smoke and Mirrors File System, 167 Snapshot isolation, 27, 164, 443 Snapshot Isolation, 606 Snapshot of a distributed system, 318 SOAP (Simple Object Access Protocol), 76, 187 Soft state, 24 Soft state data replication, 157, 448, 497 Soft-state replication using virtual synchrony model, 442 SPKI/SDSI, 564 Index Split C, 278 Split secret schemes, 568 Spread, vi, 510, 535 Spread Toolkit, 509 Sprite, 230 SS7 telecommunications protocol, 586 SSL (Secure Socket Layer), 84 SSL security protocol, 561 Stability of a multicast, 412 State machine approach, 421 State transfer, 402, 435 Stateful client/server systems, 216, 238 Stateful File Servers, 227 Stateless client/server systems, 216 Static membership model, 309, 339, 458 Streams unbreakable (wrapped for reliability), 496 Strong and weak virtual synchrony in Horus, 586 Strong durability, 376, 387, 399, 423, 430, 433, 434 performance implications, 462 Strong durability multicast delivery property, 374 Synchronous model, 354 System management, 623 T TAO (The ACE Orb), 250, 489 Tcl/Tk, 519 TCP used to support remote procedure call, 211 TCP protocol (over U-Net), 282 TCP (Trusted Computing Base), 85 Telecommunications coprocessor, 586 Testing for liveness of an application, 294 The Layers of a Cloud, 146 The Sinfonia replication service, 166 The Web, 186, 216, 489 architectural structures and reliability tools, 510 fault-tolerance and load-balancing, 492 groupware tools and solutions, 492 Java applets structured as object groups, 492 replication and reliability, 513 security with digital signatures, 492 Three-phase commit, 592 Tiers of cloud computing services, 22 Time in distributed systems, 316 Timed asynchronous model, 576 Timeliness, 62 Timing failures, 62 Token passing, 371, 397 729 Toolkits, 474, 486 Tools for consistent caching, 510 Top-level transactions, 598 Topological knowledge used in CausalSend, 396 Transactional commit protocols, 592 Transactional model compared to virtual synchrony, 588 problems encountered in distributed uses, 597 weak consistency, 599 Transactional system architectures, 587 Transactions, 237 replicated data used for scalability, 605 Transportation control, 497 Trusted cloud computing, 28 Two-phase commit, 326, 592 Two-phase locking, 239 U U-Net, 278 UDDI, 187, 200 UDP protocol over U-Net, 282 Unauthorized use of resources, 298 Unintentional threats, 295 Universal Description, Discovery and Integration language (UDDI), 200 Universe of things, 76 V Vector clock, 321 causal communication with non-members of a group, 409 timestamp compression, 393 used to implement CausalSend, 390 View (of a process group), 381, 516 View synchronous multicast delivery, 424 Viewstamped Replication, 303 Virtual memory used in communication architecture, 275 Virtual private networks, 297, 551 Virtual synchrony applied in cloud settings, 442 data replication in cloud caches, 451 data replication in first-tier (soft-state) cloud services, 448 execution model, 419 hard-state cloud services, 453 key-value storage and caches, 452 parallel search, 454 used to develop new routing services, 455 Virtualization in cloud systems, 175 730 Virtually synchronous process groups, 513 algorithms and tools, 430 compared to transactional model, 588 execution model, 419 extended virtual synchrony, 424 flush protocol, 384 guarantees compared to synchronous model, 583 in Horus, 516 replicated data, 430 reporting membership through “views”, 381 security, 455, 568 summary of membership properties, 385 Virus, 296 Vivaldi network coordinates system, 55 W Weakly durable multicast delivery, 374 Web proxy, 513 Web server wrapped for fault-tolerance, 489 Web servers (replication and load-balancing), 492 Web Services, 70, 187, 197, 200 Suggested experimental project, 685 White-Pages, 201 Index Whole file transfer compared with prefetching, 224 Wide-area group communication for the Web, 492 Windows operating system, 201, 216, 229, 481, 499, 597, 625 Worm, 296 Wrappers, 474, 481 Write-ahead log, 589 WS_RELIABILITY standard, 81 WS_SECURITY standard, 83 X x-Kernel, 274, 275, 514, 518, 529 X.500, 201 XDR, 197 XML (Extensible Markup Language, 71 Y Yellow Pages, 200 Z Zipf-like distribution, 153 Zookeeper (Yahoo’s scalable and reliable storage system), vi, 26, 165 ... The Guide to Reliable Distributed Systems: Building High-Assurance Applications and Cloud-Hosted Services is a heavily edited new edition of a prior edition that went under the name Reliable Distributed. .. Science Editors David Gries Fred B Schneider For further volumes: www.springer.com/series/3191 Kenneth P Birman Guide to Reliable Distributed Systems Building High-Assurance Applications and Cloud-Hosted. .. considerable investment, and keeping them all functioning properly, and configured to talk to one-another and to the Internet, can be a real chore K.P Birman, Guide to Reliable Distributed Systems,