Transactions and concurrency

10/21/2010 1 Transactions and ConcurrencyTransactions and Concurrency Topics for this lectureTopics for this lecture  Transactions and Concurrency control ◦ Transactions ◦ Concurrency control ◦ Nested Transactions ◦ Locks  Distributed Transactions ◦ Flat and nested distributed transactions ◦ Atomic commit protocols ◦ Concurrency control ◦ Distributed Deadlocks ◦ Transaction recovery TransactionsTransactions  The goal of transactions ◦ the objects managed by a server must remain in a consistent state  when they are accessed by multiple transactions and  in the presence of server crashes  Recoverable objects ◦ can be recovered after their server crashes ◦ objects are stored in permanent storage Transaction AtomicityTransaction Atomicity 1. All or nothing: ◦ Either every operation in the transaction completes OR ◦ No operations complete at all ◦ Failure atomicity  effects are atomic even when the server crashes ◦ Durability  after a transaction has completed successfully  all its effects are saved in permanent storage Transaction AtomicityTransaction Atomicity 2. Isolation ◦ Each transaction must be performed without interference from other transactions ◦ Other transactions can see a transaction's intermediate effects ◦ Only once ALL of the operations are complete can other transactions see the results ACID propertiesACID properties  Atomicity - All of nothing  Consistent - A transaction moves from one consistent state to another consistent state  Isolated - Intermediate effects are isolated until the transaction has completed  Durable - Once completed a transaction’s effects are permanent 10/21/2010 2 ConcurrencyConcurrency  Concurrency occurs when two or more execution flows are able to run simultaneously  Concurrency can cause problems if it is not managed  We shall examine two problems ◦ Lost update ◦ Inconsistent retrieval Serial execution of T and USerial execution of T and U Opening balances A=100, B = 200 and C = 300 Transaction T balance = b.getBalance(); // balance = 200 b.setBalance(balance*1.1); // balance*1.1 = 220 a.withdraw(balance/10); // balance/10 = 20 Transaction U balance = b.getBalance(); // balance = 220 b.setBalance(balance*1.1); // balance = 242 c.withdraw(balance/10); // balance/10 = 22 Resulting balances A=80, B = 242 and C = 278 Interleaved execution and a lost updateInterleaved execution and a lost update Transaction T balance = b.getBalance(); b.setBalance(balance*1.1); a.withdraw(balance/10) Transaction U balance = b.getBalance(); b.setBalance(balance*1.1); c.withdraw(balance/10) balance = b.getBalance(); $200 balance = b.getBalance(); $200 b.setBalance(balance*1.1); $220 b.setBalance(balance*1.1); $220 a.withdraw(balance/10) $80 c.withdraw(balance/10) $280 Resulting balances from this interleaving A=80, B = 220 and C = 280 Different to serial execution!! Serial execution of V and WSerial execution of V and W Opening balances A=100, B = 200 Transaction V a.withdraw(100); // a.balance = 0 b.deposit(100); // b.balance = 300 Transaction W total=a.getBalance(); // balance = 0 total=total+b.getBalance(); // balance = 300 // total = 300 Resulting balances A=0, B = 300 and total = 300 Interleaved execution and an Interleaved execution and an inconsistent retrievalinconsistent retrieval Transaction V a.withdraw(100); b.deposit(100); Transaction W aBranch.branchTotal(); a.withdraw(100); total=a.getBalance(); total=total+b.getBalance(); b.deposit(100); Resulting balances from this interleaving A=0, B = 300 and total=200 <<< WRONG!! Interleaved execution that is serially Interleaved execution that is serially equivalent to T and Uequivalent to T and U Transaction T balance = b.getBalance(); b.setBalance(balance*1.1); a.withdraw(balance/10) Transaction U balance = b.getBalance(); b.setBalance(balance*1.1); c.withdraw(balance/10) balance = b.getBalance(); $200 balance = b.getBalance(); $220 b.setBalance(balance*1.1); $242 b.setBalance(balance*1.1); $220 a.withdraw(balance/10) $80 c.withdraw(balance/10) $278 Resulting balances from this interleaving A=80, B = 242 and C = 278 10/21/2010 3 Conflicts in transactionsConflicts in transactions Operations within different transactions Conflict? Reason read read No No dependency between read operations read write Yes Effects from Read and Write operations depends on their order write write Yes Effects from write and write operations depends on their ordering Commit / AbortCommit / Abort  Commit ◦ Makes permanent (durable) the isolated effects of the transaction  Abort ◦ If any part fails (atomicity) ◦ Does not make permanent the isolated effects of the transaction ◦ Intermediate effects are undone Nested transactionsNested transactions  Transactions that are themselves composed of sub-transactions  Form a hierarchy with parents and sub- transactions  Allow for concurrency with the transaction  Failure with a flat transaction implies that the whole transaction should be repeated  With a nested transaction only the sub- transaction that aborted needs to be repeated Nested transactionsNested transactions T 1 Top level transaction T 1 OpenSubTransaction T 11 T 12 provisional commit abort Commit rules for nested transactionsCommit rules for nested transactions  The transaction may commit once children have completed  Sub-transactions make independent and final commit / abort decisions  Parent abort implies sub-transaction aborts  Parents can still decide to commit in the presence of sub-transaction aborts.  Top-level transaction abort implies all sub- transactions abort LocksLocks  Locks provides a means for ensuring serial equivalence  Exclusive locking is where a transaction locks objects exclusively until it commits 10/21/2010 4 Serial equivalence with exclusive Serial equivalence with exclusive lockinglocking Transaction T Transaction U Open transaction balance = b.getBalance(); Lock B b.setBalance(balance*1.1); a.withdraw(balance/10); Lock A Closetransaction Unlock A B balance = b.getBalance(); b.setBalance(balance*1.1); a.withdraw(balance/10); Open transaction balance = b.getBalance(); Waits for T to release Lock B Waitis balance = b.getBalance(); Lock B b.setBalance(balance*1.1); c.withdraw(balance/10); Lock C closeTransaction Unlock B C balance = b.getBalance(); b.setBalance(balance*1.1); c.withdraw(balance/10); Strict two phase lockingStrict two phase locking  Strict two phase locking ◦ Transactions are not allowed to apply more locks once they have released locks ◦ Growing phase = applying locks ◦ Shrinking phase = releasing locks ◦ Locks held until commit LocksLocks  What do we lock? ◦ Lockable unit should be as small as possible  Lock types ◦ Many transactions can read without conflict ◦ One writer implies possible conflict ◦ We use two different locks read and write locks Lock requested Read Write Lock present None OK OK Read OK Wait Write Wait Wait Lock compatibility Nested transaction lockingNested transaction locking  Parents acquire locks from sub-transactions once they commit ◦ The top level eventually holds all the locks ◦ Aim is to prevent sub-transaction inconsistencies  Parents do not run concurrent to sub- transactions ◦ Sub-transactions acquire locks during their execution from parents if they need them ◦ Thus, parent and sub-transaction can lock same data Nested transaction lockingNested transaction locking  Sub transactions can acquire read locks providing that no other sub-transactions hold current write locks ◦ We could have many read locks ◦ Once completed the parent acquires the read locks ◦ Requests for write locks in the presence of read locks would require the sub-transactions to wait for the read locks to released  Once committed locks are passed to parents  Parents may hold write locks and may give them to sub- transactions as required and in accordance with lock compatibility rules Dead locksDead locks  Two transactions waiting for each other to release locks T U T V U W T U V 10/21/2010 5 Dead lock detectionDead lock detection Transactions Wait for transaction T U, W V W W T, U, V • A deadlock manager has responsibility for detecting deadlocks • It stores details of who transactions are waiting for • Uses the details to detect deadlocks • Can abort transactions if deadlock is detected Deadlock edge graph T V U W Wait for graph Dead lock timeoutsDead lock timeouts  Invulnerable locks ◦ Locks have an initial period of invulnerability ◦ During this time they have exclusive access to the object ◦ Lock requests from other transactions are declined  Venerable locks ◦ Once a time limit has exceeded locks become venerable ◦ Subsequent requests from other transactions cuase the venerable lock to be broken Other locking schemesOther locking schemes  two-version locking ◦ allows writing of tentative versions with reading of committed versions  hierarchic locks ◦ Uses different lockable units ◦ i.e. the branchTotal operation locks all the accounts with one lock whereas the other operations lock individual accounts (reduces the number of locks needed) ConcurrencyConcurrency  Disadvantages of locks ◦ Lock maintenance is an overhead ◦ Locks can result in deadlock ◦ Locks do not fully use concurrency  Optimistic concurrency control ◦ Presume that transactions to not interfere with each other ◦ If a conflict does occur then abort and restart ConcurrencyConcurrency  Working phase ◦ Transactions work on tentative objects (most recently committed versions) ◦ Stores a list of which objects are used and how (read and write lists)  Validation phase ◦ Occurs once the working phase is complete ◦ The read / write lists are used to determine if conflicts exist  Update phase ◦ tentative objects committed Transaction validationTransaction validation  Transactions are assigned an incremental transaction number when they start validation  The following rules then ensure serialisability of transactions when they overlap  Where: T v is the transaction being validated T v and T i are overlapping transactions T v T i Rule Write Read 1 T i must not read objects written by T v Read Write 2 T v must not read objects read by T i write Write 3 T i must not write objects written by T v T v must not write objects written by T i 10/21/2010 6 Transaction validationTransaction validation T 1 T 2 T v Working phase validation phase commit phase T v work phase overlaps withT 2 work phase Backward validation: Rule 2: compares write set of T 2 against read set of T v T v aborts if a conflict is found T active1 T active2 Transaction validationTransaction validation T 1 T 2 T v Working phase validation phase commit phase T v work phase overlaps withT active1 and T active2 work phases Forward validation: Rule 1: compares write set of T v against read set of T i (T active1 and T active2 ) choice of actions if a conflict is found (defer, abort T v abort T i ) T active1 T active2 Distributed TransactionsDistributed Transactions Distributed TransactionsDistributed Transactions  Distributed Transactions ◦ Flat and nested distributed transactions ◦ Atomic commit protocols ◦ Concurrency control ◦ Distributed Deadlocks ◦ Transaction recovery Distributed transactionsDistributed transactions  A distributed transaction is a transaction that invokes operations in several different servers  Can be either: ◦ Flat or ◦ Nested Flat distributed transactionsFlat distributed transactions ◦ Makes invocations to various remote servers ◦ The flat transaction waits for each request to be serviced before continuing ◦ The flat transaction is sequential T T Client X Y Z Flat transaction 10/21/2010 7 Nested distributed transactionsNested distributed transactions ◦ Makes invocations to various remote servers ◦ Sub-transactions at the same level can process independently and concurrently T T Client T 1 T 2 T 11 T 12 T 21 T 22 X Y M N P Nested transaction CoordinationCoordination  Coordination is required when committing ◦ A coordinator could exist in any server ◦ The coordinator is responsible for aborting or committing the distributed transaction ◦ It requires a list of participants in the transaction  The client then begins its transaction with a request to the coordinator ◦ The coordinator passes back a DIS unique transaction ID (TID) ◦ When the client makes requests of other servers it passes the TID with the request in the transaction  Servers that are accessed become participants ◦ They register their participation using the TID when they receive invocations Atomic commit protocolsAtomic commit protocols  All nodes in a DIS agree to commit or all nodes abort  One phase commit protocol ◦ Coordinator keeps sending messages to participants until they acknowledge ◦ Concurrency management can cause problems ◦ Coordinator cannot abort once a client has requested commit Atomic commit protocolsAtomic commit protocols  Two phase commit protocol ◦ Allows any participant to abort its part ◦ Any abort implies all abort (atomicity)  Two phases ◦ Phase 1: Participant votes (commit or abort)  Once voted commit they must be able to meet that obligation  To ensure this participants store their intermediate objects ◦ Phase 2: vote execution  Coordinator collects decisions  If no failures and all votes were yes then the coordinator tells participants to commit  Otherwise it tells them all to abort Atomic commit protocolsAtomic commit protocols  Distributed transactions are message based and messages can get lost Coordinator Participent Yes canCommit? doCommit haveCommitted Atomic commit protocolsAtomic commit protocols  Time-outs to detect failures ◦ No coordinator doCommit  Participants can request decision from coordinator if they have been waiting for a long time (getDecision) ◦ No coordinator canCommit  Participants can abort if excessive time has elapsed since it completed its portion of the transaction and the server has not requested voting ◦ Coordinator waiting on a client vote  Decides to abort  Coordinator failure ◦ A big problem  Only the coordinator knew the participants  Retry with new coordinator or use a cooperative protocol 10/21/2010 8 Atomic commit protocolsAtomic commit protocols  Two phase commit protocols and nested transactions T T Client T 1 T 2 T 11 T 12 T 21 T 22 X Y M N P Nested transaction T T 1 T 2 T 11 T 12 T 21 T 22 Abort Provisional commit Abort Provisional commit Provisional commit Provisional commit Atomic commit protocolsAtomic commit protocols  Top level transaction is the coordinator  Sub-transactions become coordinators of other sub- transactions  Results propagate up the hierarchy  When a sub-transaction aborts it merely passes to its parent the abort decision  When they provisionally commit it passes their details to the parent Atomic commit protocolsAtomic commit protocols Coordinator Child transactions Participant Provisional commit list Abort list T T 1 , T 2 Yes T 1 , T 12 T 11 , T 2 T 1 T 11 , T 12 Yes T 1 , T 12 T 11 T 2 T 21 , T 22 No (aborted) T 2 T 11 No (aborted) T 11 T 12 ,T 21 T12, not T21 T 21 , T 12 T 22 No (parent aborted) T 22 Coordinator information Transaction status Concurrency Control for Distributed Concurrency Control for Distributed TransactionsTransactions • “Each server manages a set of objects and is responsible for ensuring that they remain consistent when accessed by concurrent transactions – therefore, each server is responsible for applying concurrency control to its own objects. – the members of a collection of servers of distributed transactions are jointly responsible for ensuring that they are performed in a serially equivalent manner – therefore if transaction T is before transaction U in their conflicting access to objects at one of the servers then they must be in that order at all of the servers whose objects are accessed in a conflicting manner by both T and U” Coulouris et al 2005 LocksLocks • Locks are held locally by a local lock manager • This can grant locks and tell other transactions to wait • Releasing locks is less simple • Must wait for all other servers to commit their portions of a distributed transaction before releasing its locks • Lock remain through the A2PC (Atomic Two Phase Commit) Optimistic concurrency ControlOptimistic concurrency Control • Recall that with optimistic concurrency control transactions are validated before commit • With distributed transactions – transaction numbers assigned at start of validation – transactions serialized according to transaction numbers – validation takes place in phase 1 of 2PC protocol • Transaction is validated at many servers • Each validates its objects against other ongoing transactions 10/21/2010 9 Commitment deadlockCommitment deadlock  T is validated at X, U is validated at U  U cannot validate as T has not committed  T cannot validate as U has not committed  Thus, we have commitment deadlock T U Read(A) at X Read(B) at Y Write(A) Write(B) Read(B) at Y Read(A) at X Write(B) Write(A) Optimistic concurrency ControlOptimistic concurrency Control • Distributed validation can be slow as the A2PC protocol waits for validation • Parallel validation can help • This involves forward and backward validation – Rule 2 + Rule 3 is used for backward validation • Parallel validation can lead to different servers using different serialised orders (U->T and T->U) • This can be avoided by checking and the use of globally unique transaction IDs Distributed deadlockDistributed deadlock • Single server transactions can suffer deadlocks – We can prevent, detect and resolve – Timeouts can be used for this but are not ideal – Detection is preferable possibly using wait-for graphs • Distributed transactions lead to distributed deadlocks – We can build global wait for graphs from local graphs – However, with distributed transactions we can obtain deadlocks that do not show in local wait for graph – This are what are referred to as distributed deadlocks. Distributed deadlockDistributed deadlock U V W d.Deposit(10) Lock D b.Deposit(10) Lock B at Y a.Deposit(20) Lock A at X c.Deposit(30) Lock C at Z b.Withdraw(30) Wait at Y c.Withdraw(20) Wait at Z a.Withdraw(20) Wait at X Distributed deadlockDistributed deadlock W U V Waits forHeld by Held by Held by Waits for Held by Waits for C D A B Z X Y W U V Waits forHeld by Held by Waits for Held by Waits for C A B Z Distributed deadlockDistributed deadlock • Centralised deadlock detector – Servers periodically send their local wait for graphs to the server – The server joins them together to check for distributed deadlock – It then decides how best to resolve the problem • Which transaction to abort • Phantom deadlock – Deadlocks that do not really exist – Occur due to time delays in determining deadlocks 10/21/2010 10 Distributed deadlock detectionDistributed deadlock detection • Edge chasing – Does not construct a wait for graph – Instead uses probes to detect deadlocks • Algorithm consists of three steps – Initiation – Detection – Resolution • Resolution – This simply involves aborting one of the transactions and thus breaking the deadlock Distributed deadlock detectionDistributed deadlock detection • Initiation – If a server notices a transaction (T A ) is waiting for another transaction T B • It sends a probe to the server that is blocked (server S x ) • The probe contains the edge T A -> T B (A waiting for B) • Detection – Receivers of probes (S x ) check whether the T B is waiting for another transaction • If it is (lets say T B is waiting on T C ) then it sends a probe to its blocked server (S y ) • The new probe contains the edge T A ->T B ->T C – Eventually a probe might contain a cycle • T A ->T B ->T C- >T A is a cycle as it loops back to itself Distributed transaction recoveryDistributed transaction recovery • Atomicity requirements – All of the effects of completed transactions are permanent – None of the effects of partially or aborted transactions are permanent • Durability – Objects are saved in permanent storage • Failure atomicity – Effects of transactions are permanent even when servers crash • Both aspects are governed by the recovery manager Distributed transaction recoveryDistributed transaction recovery • Recovery manager responsibilities – Saving objects in permanent storage – Restoring server objects subsequent to crashes – Managing the performance through reorganisation of recovery files – Reclaiming storage space from the recover file • For security we need to consider back-up recovery facilities – Preferably off-site and isolated Distributed transaction recoveryDistributed transaction recovery • Intention lists – Held at each server for each transaction – Is a list of references to objects that were used within a transaction – Contains the values of the objects too – The commit process uses the intention list to determine what needs to be committed – Changed objects are then written to a recovery file – Intentions are also written to the recovery file – Abort uses intention list to delete tentative objects Distributed transaction recoveryDistributed transaction recovery  Logging technique ◦ A recovery file contains history of all the transactions ◦ It contains  Intention lists, values of objects and status entries ◦ Status  Prepared: ready to commit changes  Committed: previously commit changes  Abort: aborted transaction [...]... at P1 and intention list contains reference to tentative object B at P0 • Recovery could work forwards from the beginning of recovery file • Or backwards from the end (this can be more efficient) Rollback position prior to T is at P0 Summary END Transactions and Concurrency control ◦ ◦ ◦ ◦ Transactions Concurrency control Nested Transactions Locks Distributed Transactions ◦ ◦ ◦ ◦ ◦ Flat and nested... the recovery manager • Objects are created and then their details filled by the recovery manager Trans:T Prepared 220 Trans:T Distributed transaction recovery committed • It applies atomicity P0 p1 p0 p2 p3 – Transactions fully or not at – All in the same order p4 Position 0 Position 1 Position 2 Position 3 Position 4 The state of objects A, B and C prior to transition T The tentative... P0 Summary END Transactions and Concurrency control ◦ ◦ ◦ ◦ Transactions Concurrency control Nested Transactions Locks Distributed Transactions ◦ ◦ ◦ ◦ ◦ Flat and nested distributed transactions Atomic commit protocols Concurrency control Distributed Deadlocks Transaction recovery 11 . 10/21/2010 1 Transactions and ConcurrencyTransactions and Concurrency Topics for this lectureTopics for this lecture  Transactions and Concurrency control ◦ Transactions ◦ Concurrency control ◦ Nested Transactions ◦. efficient) SummarySummary  Transactions and Concurrency control ◦ Transactions ◦ Concurrency control ◦ Nested Transactions ◦ Locks  Distributed Transactions ◦ Flat and nested distributed transactions ◦. T i ) T active1 T active2 Distributed TransactionsDistributed Transactions Distributed TransactionsDistributed Transactions  Distributed Transactions ◦ Flat and nested distributed transactions ◦ Atomic commit protocols ◦ Concurrency