Chapter 18 - Recovery and fault tolerance. This chapter discusses recovery and fault tolerance techniques used in a distributed operating system. Resiliency, which is a technique for minimizing the impact of a fault, is also discussed.
PROPRIETARY MATERIAL. © 2007 The McGrawHill Companies, Inc. All rights reserved. No part of this PowerPoint slide may be displayed, reproduced or distributed in any form or by any means, without the prior written permission of the publisher, or used beyond the limited distribution to teachers and educators permitted by McGrawHill for their individual course preparation. If you are a student using this PowerPoint slide, you are using it without permission. Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 1 Copyright © 2008 Introduction • A fault may have several consequences – It may damage state of some data or processes, leading to * Malfunctioning of a server * Non-availability of resources and services * Disruption of system operation – Its consequences can be avoided using three approaches * Recovery Some data or processes are rolled back to previous states * Fault tolerance Provides un-interrupted operation of a system despite faults * Resiliency Reduces cost of re-execution following a fault – Recovery is the generic term for all three approaches Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 2 Copyright © 2008 Faults and failures • A fault damages the state of a system – We say it causes an error in state of a system – It leads to unexpected behavior, which we call a failure – Recovery restores the system to an error-free state Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 3 Copyright © 2008 Recovery after a fault • System operation is initiated in state S0 at time 0 ’ ; a failure is detected at time t when state is S • A fault occurs at time t i i RecoveryputsthesystemintoanewstateSnew,whichiserrorưfree Chapter18:Recoveryand FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:4 Copyrightâ2008 Classes of faults • A fault model describes properties of a fault – System fault is a system crash caused by a power outage or component fault * Amnesia fault The system loses its state completely * Fail-stop fault The system stops operating when a fault occurs This property permits an error in system state to be corrected * Byzantine fault A process suffering this fault behaves maliciously * Storage fault Bad block on a store medium Chapter18:Recoveryand FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:5 Copyrightâ2008 Recovery techniques Four techniques of recovery – Data recovery * State of data or file is restored to that in its latest back-up – Process recovery * A checkpoint is a recording of the state of a system * At a fault, a process is rolled back to the state in some checkpoint – Fault tolerance * The error in state is corrected without interruption of system operation – Resiliency * Some results produced by a computation before a fault occurred are used during recovery Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 6 Copyright © 2008 Recovery techniques • Two classes of recovery techniques – Backward recovery * The state of an entity or application is reset to a prior state Simple to implement However, state recording should be both feasible and practical – Forward recovery * The erroneous state of an entity or application is repaired Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 7 Copyright © 2008 Byzantine faults and agreement protocols • A faulty entity may perform wrong actions; a fault tolerance technique is used to avoid their consequences – The Byzantine agreement problem: A group of processes must agree on a value * Initiator broadcasts the value to all other processes * On receiving the value, each process broadcasts it to all others A non-faulty process broadcasts the received value A faulty process may broadcast a different value * All non-faulty processes must conclude the correct value * Lamport’s protocol involves m+1 rounds of messages, where number of faulty processes does not exceed m * Impossibility result: a group of processes involving one faulty process cannot reach agreement Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— AConceptưBasedApproach,2ed SlideNo:8 Copyrightâ2008 Recovery schemes A recovery scheme consists of two algorithms: – Checkpointing algorithm * Decides when and how to create checkpoints of processes * Cij denotes the j’th checkpoint of process Pi * State (Cij) denotes the state of process Pi in checkpoint Cij – Recovery algorithm * When a failure is noticed, it decides which checkpoint a process should be rolled back to Chapter 18: Recovery and Fault Tolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:9 Copyrightâ2008 Checkpoints and recovery Attimetf,failureoccursinnodeN3,whereprocessP3isinoperation RollingbackP3toC32wouldbeinconsistentbecausesendingof messagem3byP3wouldbenullified,whereasP2hasreceivedm3 Chapter18:Recoveryand FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:10 Copyrightâ2008 Orphan messages Definition – A message mk sent by process Pi to Pj is an orphan in the new state of a system during recovery if the new state * Records mk as received by Pj * Does not record mk as sent by Pi • If rolling back of Pi makes mk an orphan, then Pj must be rolled back to a state before it had received mk Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— AConceptưBasedApproach,2ed SlideNo:11 Copyrightâ2008 Fault tolerance techniques A fault tolerance technique ensures that – Either a fault cannot cause an error – Or the error can be removed easily * It is typically achieved through data recovery • We discuss tolerance of storage faults – Its two aspects are * Fault tolerance for replicated data Data should remain available; processes should see correct values * Fault tolerance for distributed data Different parts of data should not become mutually inconsistent Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 12 Copyright © 2008 Logs, forward recovery and backward recovery • A log is a record of activities in a process Its contents are used for fault tolerance and recovery – Do log * Records those actions that should be performed to ensure correctness of state of an entity or system – Undo log * Records actions that should be undone to remove an error in state if a fault occurs – Write ahead logging * A log record is written before an action is taken * Ensures that the log contains correct entries when faults occur – Operation log contains a list of actions – Value log contains a list of values of data Chapter 18: Recovery and FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:13 Copyrightâ2008 Reliable data replication A quorum is a specified number of copies of data D that should be accessed by a process – Qr: quorum for reading, Qw: quorum for writing – How to ensure that D can be updated by only one process but can be read by many, and that a process would always read a latest value of D? * x Qw > n * * Qr + Qw > n Use time-stamps to identify the latest value – How to tolerate up to k failures? * Qr = k + * * Qw = n – k n>2xk Chapter18:Recoveryand FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:14 Copyrightâ2008 Reliable distributed data Inability to update some parts of distributed data due to link or node faults would cause data inconsistency – A distributed transaction is used to avoid it – The two phase commit protocol (also called PC protocol) is used to tolerate link or node faults * It ensures one of the following conditions All sites containing parts of distributed data can perform updates reliably None of the parts of the distributed data is updated Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 15 Copyright © 2008 Two phase commit protocol – Phase I Actions of transaction coordinator: a Write a ‘Prepare Ti’ record in the log b Set a time-out and send a ‘Prepare Ti’ message to all nodes participating in the transaction Actions of a participating node: a If it is ready to commit, write updates in stable storage and a ‘Prepared Ti’ record in the log, and send a ‘Prepared Ti’ message to coordinator b Otherwise, write an ‘Abandoned Ti’ record in the log and send an ‘Abandoned Ti’ message to coordinator Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 16 Copyright © 2008 Two phase commit protocol – Phase II Actions of transaction coordinator: a If it receives a ‘Prepared’ reply from all nodes before time-out occurs, write a ‘Commit Ti’ record in the log and send ‘Commit Ti’ messages to all nodes b Otherwise, write an ‘Abort Ti’ record in the log and send ‘Abort Ti’ messages to all nodes c Wait for an acknowledgment from each node and write a ‘Complete Ti’ message in the log Actions of a participating node: Depending on the coordinator’s message, a Write a ‘Commit Ti’ record in the log and perform commit processing b Write an ‘Abort Ti’ record in the log and abandon updates of Ti Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed SlideNo:17 Copyrightâ2008 Resiliency Consider a distributed computation Parts of the computation are executed in different nodes * These parts communicate through interprocess messages – Faults can arise during its operation * A node containing a part of the computation or its data fails * A link fails – However, failures are partial * A node or link failure may affect only some part of the computation * Resiliency techniques exploit this fact to reduce the recovery cost Some of the results computed before the failure are re-used Only some parts of the computation are re-executed Chapter18:Recoveryand FaultTolerance Dhamdhere:OperatingSystems AConceptưBasedApproach,2ed SlideNo:18 Copyrightâ2008 Nested transaction A nested transaction Tik is a part of an atomic transaction Ti – It commits only if its parent transaction Ti commits – It is implemented as follows * When Tik completes, it is said to have reached a tentative commit * When Ti wishes to commit, it checks whether all nested transactions have reached a tentative commit and can participate in commit processing It is implemented using a 2PC protocol Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 19 Copyright © 2008 Nested transaction • Resiliency is implemented as follows: – If a nested transaction Tik does not respond to a ‘Prepare’ message, the coordinator can retry Tik in the same node or in some other node * If Tik had reached tentative commit and its node had failed when ‘Prepare’ message was sent * If the failed node recovers and the coordinator retries Tik in it * The results of Tik, computed before the failure, can be used Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 20 Copyright © 2008 ... parts of the computation are re-executed Chapter1 8:Recoveryand FaultTolerance Dhamdhere: OperatingSystems AConceptưBasedApproach,2ed SlideNo :18 Copyrightâ2008 Nested transaction A nested transaction... back of Pi makes mk an orphan, then Pj must be rolled back to a state before it had received mk Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— AConceptưBasedApproach,2ed... state of an entity or application is repaired Chapter 18: Recovery and Fault Tolerance Dhamdhere: Operating Systems— A ConceptBased Approach, 2 ed Slide No: 7 Copyright © 2008 Byzantine faults