probability and computing randomized algorithms and probabilistic analysis

Tiếng Anh và mức độ quan trọng đối với cuộc sống của học sinh, sinh viên Việt Nam.Khi nhắc tới tiếng Anh, người ta nghĩ ngay đó là ngôn ngữ toàn cầu: là ngôn ngữ chính thức của hơn 53 quốc gia và vùng lãnh thổ, là ngôn ngữ chính thức của EU và là ngôn ngữ thứ 3 được nhiều người sử dụng nhất chỉ sau tiếng Trung Quốc và Tây Ban Nha (các bạn cần chú ý là Trung quốc có số dân hơn 1 tỷ người). Các sự kiện quốc tế , các tổ chức toàn cầu,… cũng mặc định coi tiếng Anh là ngôn ngữ giao tiếp.

Probability and Computing Randomized Algorithms and Probabilistic Analysis Michael Mitzenmacher Eli Upfal Probability and Computing Randomization and probabilistic techniques play an important role in modern com­ puter science, with appl ications ranging from combinatorial optimization and machine learning to communication networks and secure protocols This textbook is designed to accompany a one- or two - semester course for advanced undergraduates or beginning graduate students in computer science and applied mathe­ matics It gives an excellent introduction to the probabil istic techniques and paradigms used in the development of probabilistic algorithms and analyses It assumes only an elementary background in discrete mathematics and gives a rigorous yet accessible treatment of the material, with numerous examples and applications The first half of the book covers core material , including random sampling, expec­ tations, Markov's inequality, Chebyshev 's inequality, ChernotT bounds, bal ls-and-bins models, the probabilistic method, and Markov chains I n the second half, the authors delve into more advanced topics such as continuous probability, applications of limited i ndependence, entropy, Markov chain Monte Carlo methods coupling, martingales, and balanced allocations With its comprehensive selection of topics, along with many examples and exercises, this book is an indispensable teaching tool Michael Mitzenmacher is John L Loeb Associate Professor in Computer Science at Harvard University He received his Ph D from the University of California Berke­ ley, in 996 Prior to joining Harvard in 999, he was a research staff member at Digital Systems Research Laboratory in Palo Alto He has received an NSF CAREER Award and an Alfred P S loan Research Fel lowship In 2002, he shared the IEEE Information Theory Society " B est Paper" Award for his work on error- correcting codes Eli Upfal is Professor and Chair of Computer Science at Brown University He received his Ph D from the Hebrew University, Jerusale m , Israe l Prior to joining Brown in 1997 , he was a research staff member at the IBM research division and a professor at the Weizm ann Institute of Science in I srael His main research interests are randomized computation and probabilistic analysis of algorithms, with applications to optimization al gorithms, communication networks, parallel and distributed computing and compu­ tational biology Probability and Computing Randomized Algorithms and Probabilistic Analysis Michael Mitzenmacher Harrarr.l Uni1·ersitY CAMBRIDGE C:\Jl VE}{SJTY Pl{ESS Eli Upfal Brmm Unirersity PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE The Pitt B u i l di ng , Trumpington Street Cambridge U n i ted Ki ngdom CAMBRIDGE UNIVERSITY PRESS The Edinburgh B u i ldi ng Cambridge C B 2RU U K 40 West 20th Street New York N Y IOCIII-421 USA 7 Wi l l i amstown Road Port M e l bourne VIC 3207 Austra l i a R u i z de Alarcon 2801 \1adrid S p a i n Dock House, The Waterfront C a p e Town 8001 South Africa http://www.cambridge org © M i chael M i tzenmacher and El i l'pfal 2005 This book is in copyright S ubject to statutory exception and to the provi sions of relevant c o l l e ctive l i ce n s i n g agreements no reproduction of any part may take p l ace \\ ithout the wri tten perm i s s i o n of Cambridge U n i versity Press First pub l i shed 2005 Printed i n the U n ited States of America Type{ace Times 10 /1 pt System A M S -TEX [FH] A catalog record for this book is available from the British Library LibrarY of Congress Cataloging in Publication data M i tzenmacher, M ichael, 1969Probab i l ity and computi ng : rando m i zed al gorithms and probab i l i stic analysis I M i chael M itze n m acher E l i Upfa l p em Incl udes i ndex I S B N -521-8354 - ( a l k paper) I Al gorithms Probabi l i ties Stochastic anal y s i s I U p fa l Eli 1954- J l Title QA2 74 M 74 2005 518'.1 - dc22 ISBN 21 540 hardback 2004054540 Contents page Preface xiii Events and Probability 1.1 Application: Verifying Polynomial Identities 1.2 Axioms of Probability 1.3 Application: Verifying Matrix Multiplication 1.4 Application: A Randomized Min-Cut Algorithm 1.5 Exercises Discrete Random Variables and Expectation 20 2.1 20 22 23 25 26 30 32 34 38 Random Variables and Expectation 2.1.1 Linearity of Expectations 2.1.2 Jensen's Inequality 2.2 The Bernoulli and Binomial Random Variables 2.3 Conditional Expectation 2.4 The Geometric Distribution 2.4.1 3 12 14 Example: Coupon Collector's Problem 2.5 Application: The Expected Run-Time of Quicksort 2.6 Exercises Moments and Deviations 44 3.1 Markov's Inequality 3.2 Variance and Moments of a Random Variable 3.3 Chebyshev's Inequality 44 45 48 48 50 52 53 54 57 3.2.1 3.3.1 3.4 3.5 Example: Variance of a Binomial Random Variable Example: Coupon Collector's Problem Application: A Randomized Algorithm for Computing the Median 3.4.1 The Algorithm 3.4.2 Analysis of the Algorithm Exercises vii C ONTENTS Chernoff Bounds 61 4.1 Moment Generating Functions 4.2 Deriving and Applying Chernoff Bounds 61 63 63 67 67 69 71 72 73 78 83 4.2.1 Chernoff Bounds for the Sum of Poisson Trials 4.2.2 Example: Coin Flips 4.2.3 Application: Estimating a Parameter 4.3 Better Bounds for Some Special Cases 4.4 Application: Set Balancing 4.5* Application: Packet Routing in Sparse Networks 4.6 4.5.1 Permutation Routing on the Hypercube 4.5.2 Permutation Routing on the Butterfly Exere ises Balls, Bins, and Random Graphs 5.1 Example: The Birthday Paradox 5.2 Balls into Bins 5.3 5.2.1 The Balls-and-Bins Model 5.2.2 Application: Bucket Sort The Poisson Distribution 5.3.1 5.4 90 Limit of the Binomial Distribution The Poisson Approximation 5.4.1 * Example: Coupon Collector's Problem, Revisited 5.5 5.6 Application: Hashing 5.5.1 Chain Hashing 5.5.2 Hashing: Bit Strings 5.5.3 Bloom Filters 5.5.4 Breaking Symmetry Random Graphs 5.6.1 Random Graph Models 5.6.2 Application: Hamiltonian Cycles in Random Graphs 5.7 Exercises 5.8 An Exploratory Assignment 90 92 92 93 94 98 99 04 06 06 08 09 12 112 12 13 18 24 The Probabilistic Method 26 6.1 The Basic Counting Argument 6.2 The Expectation Argument 26 28 29 30 131 133 33 34 34 35 6.3 6.4 6.5 6.2.1 Application: Finding a Large Cut 6.2.2 Application: Maximum Satisfiability Derandomization Using Conditional Expectations Sample and Modify 6.4.1 Application: Independent Sets 6.4.2 Application: Graphs with Large Girth The Second Moment Method 6.5.1 Application: Threshold Behavior in Random Graphs viii CONTENTS 6.6 The Conditional Expectation Inequality 6.7 The Lovasz Local Lemma 6.8* 6.7.1 Application: Edge-Disjoint Paths 6.7.2 Application: Satistiability Explicit Constructions 6.8.1 Application: A Satistiability Algorithm 6.9 Lovasz Local Lemma: The General Case 6.10 Exercises :\Iarkov Chains and Random Walks 53 15 156 159 63 166 167 173 74 176 177 182 7.2 Markov Chains: Definitions and Representations 7.1.1 Application: A Randomized Algorithm for 2-Satisfiability 7.1.2 Application: A Randomized Algorithm for 3-Satisfiability Classification of States 7.2.1 7.3 7.4 Example: The Gambler's Ruin Stationary Distributions 7.3.1 Example: A Simple Queue Random Walks on Undirected Graphs 7.4.1 the Local Lemma Application: An 7.5 Parrondo's Paradox 7.6 Exercises Connectivity Algorithm Continuous Distributions and the Poisson Process 88 8.1 88 188 191 193 94 196 97 99 20 204 205 207 210 212 213 216 216 219 Continuous Random Variables 1 8.1.2 8.2 8.3 Probability Distributions in R Joint Distributions and Conditional Probability The Uniform Distribution 8.2.1 Additional Properties of the Uniform Distribution The Exponential Distribution 8.3.1 Additional Properties of the Exponential Distribution 8.3.2* Example: Balls and Bins with Feedback 8.4 8.5 8.6 The Poisson Process 8.4.1 Interarrival Distribution 8.4.2 Combining and Splitting Poisson Processes 8.4.3 Conditional Arrival Time Distribution Continuous Time Markov Processes Example: Markovian Queues 8.6.1 8.6.2 8.6.3 8.7 tJ 36 138 141 42 42 43 46 48 MIMI I Queue in Equilibrium MIMI IlK Queue in Equilibrium The Number of Customers in an lvfI MIx Queue Exercises Entropy, Randomness, and Information 225 9.1 The Entropy Function 9.2 Entropy and Binomial Coefficients 225 228 ix CONT E NTS 9.3 Entropy: A Measure of Randomness 9.4 Compression 9.5* Coding: Shannon's Theorem 9.6 10 Exercises The Monte Carlo Method 25 10.1 The Monte Carlo Method 10.2 Application: The DNF Counting Problem 25 255 255 257 259 263 265 267 270 10.2.1 The Na"ive Approach 10.2.2 A Fully Polynomial Randomized Scheme for DNF Counting 10.3 From Approximate Sampling to Approximate Counting 10.4 The Markov Chain Monte Carlo Method 10.4.1 11* 12 The Metropolis Algorithm 10.5 Exercises 10.6 An Exploratory Assignment on Minimum Spanning Trees Coupling of Markov Chains 27 11.1 Variation Distance and Mixing Time 11.2 Coupling 27 274 275 276 277 278 28 282 286 289 11.2.1 Example: Shuffling Cards 11.2.2 Example: Random Walks on the Hypercube 11.2.3 Example: Independent Sets of Fixed Size 11.3 Application: Variation Distance Is Nonincreasing 11.4 Geometric Convergence 11.5 Application: Approximately Sampling Proper Colorings 11.6 Path Coupling 11.7 Exercises Martingales 295 12.1 Martingales 12.2 Stopping Times 295 297 299 300 303 305 305 307 308 308 309 12.2.1 12.3 Example: A Ballot Theorem Wald "s Equation 12.4 Tail Inequalities for Martingales 12.5 Applications of the Azuma-Hoeffding Inequality 12.6 13 230 234 237 245 12.5.1 General Formalization 12.5.2 Application: Pattern Matching 12.5.3 Application: Balls and Bins 12.5.4 Application: Chromatic Number Exercises Pairwise Independence and Universal Hash Functions 14 13.1 14 315 16 Pairwise Independence 13.1.1 Example: A Construction of Pairwise Independent Bits 13.1.2 Application: Derandomizing an Algorithm for Large Cuts X C ONTENTS 13.1.3 Example: Constructing Pairwise Independent Values Modulo 13.4 Application: Finding Heavy Hitters in Data Streams 13.5 Exercises 317 318 19 32 323 24 326 328 333 Balanced Allocations 336 14.1 336 336 34 344 344 345 345 a Prime 13.2 Chebyshev's Inequality for Pairwise Independent Variables 13.2.1 13.3 13.3.1 14 * Application: Sampling Using Fewer Random Bits Families of Universal Hash Functions Example: A 2-Universal Family of Hash Functions 13.3.2 Example: A Strongly 2-Universal Family of Hash Functions 13.3.3 Application: Perfect Hashing The Power of Two Choices 14.1.1 The Upper Bound 14.2 Two Choices: The Lower Bound 14.3 Applications of the Power of Two Choices 14.4 14.3.1 Hashing 14.3.2 Dynamic Resource Allocation Exercises Further Reading 349 Index 350 \·ore: Asterisks indicate advanced material xi Preface \\.hy Randomness? \\.hy should computer scientists study and use randomness? Computers appear to �have far too unpredictably as it is ! Adding randomness would seemingly be a dis­ JJ\ antage, adding further complications to the already c hallenging task of efficiently uti l i zi ng computers Science has learned in the last century to accept randomness as an essential com­ r,)ncnt i n modeling and analyzing nature In physics, for example, Newton ' s laws led �l)ple to believe that the universe was a deterministic place ; given a big enough calcu­ ; 1tnr and the appropriate i nitial conditions, one could determine the location of planets �cars from now The development of quantum theory suggests a rather different view ; the universe stil l behaves according t o laws, but the bac kbone o f these laws i s proba­ �IIi stic " God does not play dice with the universe" was Einstein ' s anecdotal obj ection :,, modern quantum mechanics Nevertheless, the prevail i ng theory today for subpar­ :I,:k physics is based on random behavior and statistical laws, and randomness plays a ,I�n i ti cant role in almost every other field of science ranging from genetics and evolu­ :11 111 i n biology to modeling price fluctuations in a free-market economy Computer science is no exception From the highly theoretical notion of proba­ �lli tic theorem proving to the very practical design of PC Ethernet cards, randomness 1nJ probabilistic methods play a key role i n modern computer science The last two Jcl 'a des have witnessed a tremendous growth in the use of probabil ity theory in com­ puting Increasingly more advanced and sophisticated probabilistic techniques have Xcn developed for use within broader and more chal lenging computer science appli­ : 1ti ons In this book , we study the fundamental ways in which randomness comes h' hear on computer science: random ized algorithms and the probabilistic analysis of 1l�orithms Randomized algorithms: Randomized algorithms are algorithms that make random :hnices during their execution In practice, a randomized program would use values �cncrated by a random number generator to decide the next step at several branches xiii THE POWER OF T WO C H O I C E S This holds independently of any of the events £i , owing to our careful defi nition of }� ( Had we not included the condition that Y1 = I only if v i ( t - I ) � f3i · the inequality would not necessarily hold.) Conditioned on £i , we have L �1= Yt = fl i - · Since v i+ I � fl i + l ' we have = Pr (t I= I Y, > k I E, Pr ( L ;1= Yt > k ) < -­ Pr ( [i ) ) Pr ( B ( 1 , p i ) > k ) < -­ Pr ( £ ) We bound the tail of the binomial distribution by using the Chernoff bound of Lemma Letting k = {3 + = np i in the preYious equations yields which gives whenever P i n 2:: ln n We now remove the conditioning by using the fact that Pr ( -,[i+ ) = Pr ( -,[i+ I £i ) Pr (£i ) + Pr ( -, [i + I -, £i ) Pr ( -, [i ) � Pr ( , £ i + I £ i ) Pr ( £ i ) + Pr ( , Ei ) : ( 14.3) then, by Eqns ( 4.2) and ( 14.3), ( 14.4) as long as P i ll 2:: ln n Hence, whenever P i 1 2:: ln n and £i holds with high probability, then so does £i + • To conclude we need two more steps First , we need to show that p i n < ln n when i is approximately In In 1 / I n d, since this is our desired bound on the maximum load Second, we must carefully handle the case where Pi n < ln 11 separately, since the Chernoff bound is no longer strong enough to give appropriate bounds once P i is this small Let i * be the smallest value of i such that P i = f311/n d < ln 11 j n We show that i * i s ln ln n j l n d + ( ) To this, w e prove inductively the bound This holds true when i = 0, and the induction argument follow s: 339 BALANCED A L L OC AT I ON S f3t�4 11 £1- ( , '"' - ;::, d i n d- � r n The first line i s the definition of {31 ; the second follows from the induction hypothesis It follows that f3t+-+ nl and hence that i* is l n ln n l ln d + ( ) B y i nductively applying Eqn ( 4.4) we find that i* l n n We have We now handle the case where p1 n 18 ln n Pr ( Vt* + 1 I n n I £i * ) Pr ( B ( n In n In ) 2: In n ) Pr ( 11 i I * ,._ > Pr ( £i " ) Pr ( E t • ) · where the last inequality again fol lows from the Chernoff bound Removing the con­ ditioning as before then yields Pr ( v i * + > i* + I 18 ln n) ( 4.5 ) To wrap up we note that 2) and bound the latter quantity as follows: Pr( 11 � -+- 2: I Vt* + l � ln n ) _ P r ( B ( n, ( I n n In ) d ) Pr ( v + < In n ) -­ -* Here the last inequality comes from applying the crude union bound; there arc G) way:- of choosing two balls, and for each pair the probability that both ball s have height at least i* + i s ( In n ln ) d Removing the conditioning as before and then using Eqn ( 4.5) yields Pr ( v i * Pr ( B (n , p ) > k i= l j , k) be the event that v i ( n ( l - l / i ) ) n: Yo Yi , where Yi is ( : )" " Yi + l _ ' Clearly holds with probability I We now show inductively that successive :F;· hold with sufficiently high probability to obtain the desired lower bound We want to compute I With thi s in mind, for t in the range R random variable by (n( I I :Fi ) l/ 2i ), n ( l - ) ] , define the binary if and only if h ( t ) Hence i s always I i f l' i + l ( t - I ) > Yi + l · The probability that the t th bal l has he ight exactly i + is c ,(t - I ) /1 r The first term is the probability that al l the d bins chosen by the t th ball have load at least i This is necessary for the height of the t th ball to have height exactly i + However, we must subtract out the probability that all d choices have at least i + I balls, because in this case the height of the ball will be larger than i + I Again letting w.i represent the bins selected by the j th balL we conclude that This is because Zr is automatically I if v i + ( t I ) Yi + l : hence we can consider the probability in the case where v i + l ( t - ) :::; Yi + l · Also, conditioned on :Fi , we have Vi ( t - ) ::::: Yi · 342 1-'.2 TWO CHOIC E S : THE L OW E R B O l :\ D From the definition o f the Yi w e can further conclude that '/1 d � '1 ( Let Pi = ( yz /11 ) d Applying Lemma 14.6 yields )II ( II ) � ( II ), _ 2: � : Now our choice of Yi nicely satisfies Yi + By the Chernoff bound, (( p· Pr B _!!_ i+1 ' I ) = < n Pi · 2i+ ) y·t + -< e-np, /( 8·2'+ ) ' which is o ( l ln ) provided that Pi n l i + 2: ln n Let i * be a lower bound on the largest integer for which this holds We subsequently show that i* can be chosen to be In !n n l !n d - ( ) ; for now let us assume that thi s is the case Then, for i :=:; i *, we have shown that Further, by defi nition we have that L t E R zl < Yi + implies -,:f'i+ · Hence for i :=::: i *, Therefore, for sufficiently large 11 , Pr (.J; * ) 2: 2: = · Pr ( Fi� I Fi* - ) Pr(Fi*- I Fi* - ) · · · Pr(F1 I Fo ) · Pr (Fo ) ( I - I l11 ) i ' I - o ( ! ll l ) All that remains is to demonstrate that In In n I In d - ( ) i s indeed an appropriate choice for i * It suffices to show that Yi 2: ln n when i is In In n I In d - ( ) From the recursions Yi + = y/1( i +JI1 d - ) , we fi nd by a simple induction that n A very rough bound gives 11 Y1· >- 10 i-I d We therefore look for the maximum i such that n - >- ln n -10 '-1 d 343 BALANC E D ALL OCATIONS For n sufficiently large, we fi nd that we can take i as large as In I n n / In d - ( ) by using the fol lowing chain of inequalities : 11 - > l n n , -1 ()(/1-l 2 I Od ' - -< !Od i - l ::S n , 17 ln n log2 n - log2 ( ln 11 ) , d - I < - In n ' - 20 In in n ln d i < - 0( ) • Applications of the Power of Two Choices The balanced allocation paradigm has a number of interesting applications to comput­ ing problems We elaborate here on two simple applications When considering these applications, keep in mind that the In In n / In d + ( ) bound we obtain for the balanced allocation paradigm generally corresponds to a maximum load of at most in practice 14 Hashing When we considered hashing in Chapter 5, we related it to the balls-and-bins parad igm by assuming that the hash function maps the items being hashed to random entries in the hash table Subject to this assumption, we proved that (a) when ( 11 ) items are hashed to a table with 11 entries the expected number of items hashed to each individ­ ual entry in the table is ( ), and ( b J with high probability, the maximum number of items hashed to any entry in the table is (;;) ( in n j ln ln n ) These results are satisfactory for most appl ications, but for some they are not, since the expected value of the worst-case lookup time over all items is 0) ( In n / l n I n n ) For example, when storing a routing table in a router, the worst-case time for a lookup in a hash table can be an important performance criterion, and the 0) ( In n / In In 11 ) result is too large Another potential problem is wasted memory For example, suppose that we design a hash table where each bin should fit in a si ngle fixed-size cache line of memory Because the maximum load is so much larger than the average, we will have to use a large number of cache lines and many of them will be completely empty For some appl ications, such as routers, this waste of memory is undes irable Applying the balanced allocation paradigm, we obtain a hashing scheme with 0( I ) expected and O( l n In n ) maximum access time The 2-vvay chainint.: technique uses two random hash functions The two hash functions define two possible entries in the table for each item The item is inserted to the location that is least ful l at the time of insertion Items in each entry of the table are stored in a linked list If n items are se­ quentially inserted into a table of size 11 , the expected insertion and lookup time is stil l ( ) (See Exercise ) Theorem 14 implies that with high probability the maxi­ mum time to fi nd an item is O( l n In n ) , versus the (�) ( In n / In In n ) time when a single 344 14A EXERCISES random hash function is used This i mprovement does not come \vithout cost Since a search for an item now involves a search i n two bins i nstead of one the improvement in the expected maximum search time comes at the cost of rough ly doubl the average search time This cost can be mitigated i f the two bins can be searched in parallel 14.3.2 Dynamic Resource Allocation Suppose a user or a process has to choose on-line between a number of i dentical re­ sources (choosing a server to use among servers i n a network; choosing a disk to store a directory; choosing a printer; etc ) To find the least loaded resource, users may check the load on all resources before placing their requests Thi s process is expensive, since it requires sendi ng an interrupt to each of the resources A second approach is to send the task to a random resource This approach has minimal overhead, but if all users fol low it then loads will vary significantly among servers The balanced allocation a more efficient solution If each user samples the load of two re­ paradigm sources and sends his request to the least loaded one, then the total overhead remains small while the load on the n resources varies much less Exercises Exercise 14 : (a) For Theorems 14 and 14.4 the statement of the proof is for the case that ties are broken randomly Argue informally that if the bins are numbered from to n and if ties are broken in favor of the lovver -numbered bin then the theorems still hold ( b) Argue i nformal ly that the theorems apply to any tie-breaking mechanism that has no knowledge of the bin choices made by balls that have not yet been placed Exercise 14.2: Consider the following variant of the balanced allocation paradigm: 11 balls are placed sequentially in n bins with the bins labeled from to 11 - Each ball chooses a bin i uniformly at random and the bal l i s placed in the least loaded of bins i i I mod n i mod n i + d - I mod n Argue that, when d is a constant, the maximu m load grows as (�) ( In n j !n In n ) That i s the balanced allocation paradigm does not yield an 0( In In n ) result in this case Exercise 14.3: Explain why with 2-way chaining the expected time to i nsert an item and to search for an item in a hash table of size 11 w ith n items is O ( l ) Consider two cases: the search is for an item that is in the table; and the search is for an item that i s not in the table Exercise 14.4: Consider the following variant of the balanced al location paradigm: balls are placed sequenti ally in n bins Each ball comes vvith d choices chosen i ndependently and uniformly at random from the n bins When a ball is placed v.·e are also allowed to move balls among these d bins to equal ize their load a� much a s possible Show that the maximum load is still at least In In n / In d Of I ) with proba­ bility I - o ( ! /n ) in this case 11 345 BALANC E D ALL OCATIONS Exercise 14 5: Suppose that in the balanced allocation setup there are n bins, but the bins are not chosen uniformly at random Instead, the bins have two types : 1 of the bins are type A and I of the bins are type B When a bin is chosen at random , each of the type-A bins is chosen with probability 1n and each of the type-B bins is chosen with probability l l2 n Prove that the maximum load of any bin when each ball has d bin choices is sti ll ln ln n l ln d + 0( ) Exercise 14.6: Consider a parallel version of the balanced allocation paradigm in which we have n I k rounds, where k new balls arrive in each round Each ball is placed in the least loaded of its d choices, where in this setting the load of each bin is the load at the end of the previous round Ties are broken randomly Note that the k new balls cannot affect each other's placement Give an upper bound on the maximum load as a function of n , d, and k Exercise 14.7 : We have shown that sequentially throwing n balls into n bins randomly, using two bin choices for each ball, yields a maximum load of ln ln n I In + 0( ) with high probability Suppose that, instead of placing the balls sequentially, we had ac­ cess to all of the 2n choices for the n balls, and suppose we wanted to place each ball into one of its choices while minimizing the maximum load In this setting, with high probability, we can obtain a maximum load that is constant Write a program to explore this scenario Your program should take as input a param­ eter k and implement the following greedy algorithm At each step, some subset of the balls are active; initially, all balls are active Repeatedly find a bin that has at least one but no more than k active balls that have chosen it, assign these active balls to that bin, and then remove these balls from the set of active balls The process stops either when there are no active balls remaining or when there is no su itable bin If the algorithm stops with no active balls remaining, then every bin is assigned no more than k balls Try running your program with \0,000 balls and \ 0,000 bins What is the smallest value of k for which the program terminates with no active balls remaining at least four out of five times? If your program is fast enough , try experimenting with more trials Also, if your program is fast enough , try answering the same question for \00,000 balls and 100,000 bins Exercise 14.8: The followi ng problem models a simple distributed system where agents contend for resources and back off in the face of contention As i n Exercise I I balls represent agents and bins represent resources The system evolves over rounds In the first part of every round, balls are thrown independently and uniformly at random into n bins In the second part of each round, each bin in which at least one ball has landed in that round serves exactly one ball from that round The remaining balls are thrown again in the next round We begin with n balls in the first round, and we finish when every ball is served (a) Show that, with probability I o ( I In ) , this approach takes at most log log n + ( ) rounds ( Hint: Let b k be the number of balls left after k rounds; show that b k + l ::::; c(b k ) 2ln , for a su itable constant c with high probability, as long as b k + l is sufficiently large ) - 346 UA E X E R C I S E S (b) Suppose that we modify the system so that a bin accepts a ball in a round if a nd on ly if that ball was the on ly ball to req uest that bin in that round Show that a g a i n with probability I - o ( I In ) , this approach takes at most Iog Iog2 n + 0( I J rounds Exercise 14.9: The natural way to simu late experiments with balls and bins is to cre­ ate an array that stores the load at each bin To simulate I ,000,000 bal l s being placed into 000.000 bins wou ld require an array of ,000,000 counters An alternative ap­ proach is to keep an array that records in the jth cell the number of bins with load j Explain how this could be used to simulate placing I ,000,000 balls into I ,000,000 bins using the standard balls-and-bins parad igm and the balanced al location paradigm with much less space Exercise 14.10: Write a program to compare the performance of the standard balls­ and-bins paradigm and the balanced allocation paradig m Run simulations placing 11 balls into 11 bins, with each ball having = I , 2, , and random choices You should try 11 = I 0.000 1 = I 00.000, and 11 = 000.000 Repeat each experiment at least I 00 times and compute the expectation and variance of the max imum load for each value of based on your trials You may wish to use the idea of Exercise 14.9 d d Exercise 14.11 : Write a simulation showing how the balanced allocation paradigm can improve performance for distributed queueing systems Consider a bank of 1 FIFO queues with a Poisson arrival stream of customers to the entire bank of rate A ll per second, where A < I Upon entry a customer chooses a queue for service and the service time for each customer is exponentially distributed with mean I second You shou ld compare two settings : ( i ) where each customer chooses a queue independently and uniformly at random from the n queues for service: and ( ii J where each customer chooses two queues independently and uniform ly at random from the n queues and waits at the queue with fewer customers breaking ties randomly Notice that the first setting is equivalent to having a bank of n MIMI I FIFO q ueues, each w ith Poisson ar­ rivals of rate ) < I per second You may find the discussion in Exercise 26 helpfu l i n constructing your simulation Your simulation should run for t seconds and it should return the average (over all customers that have completed service ) of the time spent in the system as well as the av­ erage (over all customers that have arrived ) of the number of customers found waiting in the queue they selected for service You should present results for your simulations for 1 = 00 and for t = 0.000 seconds with A = 0.5, 0.8, 0.9, and 0.99 Exercise 14.12: Write a program to compare the performance of the following vari­ ation of the standard balls-and-bins paradigm and the balanced al location parad i g m Initially 1 points are placed uniformly at random o n the boundary of a circle o f circum­ ference I These 1 points divide the circle into 1 arcs, which correspond t o b i n s \Ye now place 11 balls into the bins as follows: each ball chooses d points on t h e b o u n d a r: of the circle, uniformly at random These points correspond to the a rc s ( o r e q u i \ a ­ lently, bins ) that they lie on The bal l is placed in the least loaded of the d b i n s b re a k i n g ties in favor of the smallest arc d 347 BALANC E D ALL OCATIONS Run simulations placing n balls into n bins for the cases d = and d = You should try n = ,000, n = 10,000, and n = 100,000 Repeat each experiment at least 100 times; for each run , the n initial points should be re- chosen Give a chart showing the number of times the maximum load was k, based on your trials for each value of d You may note that some arcs are much larger than others, and therefore when d = the maximum load can be rather high Also, to fi nd which bin each ball is placed i n may require implementing a binary search o r some other additional data structure to quickly map points on the circle boundary to the appropriate bin Exercise 14.13: There is a small but interesting improvement that can be made to the balanced allocation scheme we have described Again we will place n balls into n bins We assume here than n is even Suppose that we divide the n bins into two groups of size n / We call the two groups the left group and the right group For each ball, we independently choose one bin uniformly at random from the left and one bin uniformly at random from the right We put the ball in the least loaded bin, but if there is a tie we always put the ball in the bin from the left group With this scheme, the max imum load is reduced to ln In n / ln ¢ + 0( 1) , where ¢ = ( + J5 )/ is the golden ratio This improves the result of Theorem 14 by a constant factor ( Note the two changes to our original scheme: the bins are split into two groups, and ties are broken in a consistent way; both changes are necessary to obtain the improvement we describe ) (a) Write a program t o compare the performance o f the original balanced allocation paradigm with this variation Run simulations placing n balls into 1 bins, with each ball having d = choices You should try n = 0,000, 1 = 00,000, and n = ,000,000 Repeat each experiment at least 100 times and compute the expec­ tation and variance of the maximum load based on your trials Describe the extent of the improvement of the new variation (b) Adapt Theorem 14 to prove this result The key idea in how the theorem 's proof must change is that we now require two sequences, f3i and Yi Similar to Theo­ rem 14 , f3i represents a desired upper bound on the number of bins on the left with load at least i , and Yi is a desired upper bound on the number of bins on the right with load at least i Argue that choosing and for some constants c and c2 is suitable (as long as f3 i and Yi are large enough that Chernoff bounds may apply) Now let Fk be the k th Fibonacci number Apply induction to show that, for suf­ · ficiently large i , f3i ::S n c c :2; and Yi ::S n c c: i + l for some constants c and c4 Following Theorem 14 , use this to prove the ln ln n / ln ¢ + ( ) upper bound (c) This variation can easily be extended to the case of d > choices by splitting the n bins into d ordered groups, choosing one bin uniformly at random from each group, and breaking ties in favor of the group that comes first in the ordering Sug­ gest what would be the appropriate upper bound on the maximum load for this case, and give an argument backing your suggestion ( You need not give a com­ plete formal proof ) 348 Further Reading N Alon and J Spencer, The Probabilistic Method, 2nd ed Wiley, New York 2000 B Bollobas, Random Graphs, 2nd ed Academic Orlando, FL , 999 T H Corman C E Leiserson, R L Rivest and C S tein, Introduction to A lgorithms, 2nd ed MIT Press I McGraw-Hill, Cambridge I New York, 200 T M Cover and J A Thomas, Elements qf' ln.formation Theory, W iley, New York, 99 '\;hl /, 3rd ed Wiley, New York, 968 W Feller, A n Introduction to the Prohahilit.v W Feller, A n Introduction t o the Probability Theory Vol Wiley, New York 966 S Karlin and H M Taylor A First Course in Stochastic Processes, 2nd ed Academic Press, New York 97 S Karlin and H Yl Taylor A Second Course i n Stochastic Processes Academic Press, New York F Leighton, Parallel Algorithms and A rchitectures Morgan Kauffmann San Mateo C A , 992 Cambridge University Press 99 R Motwani and P Raghavan Randomi::,ed J H Spencer Ten Lectures 011 the Probabilistic Method 2nd ed S IA M , Philadelphia , 994 S Ross Stochastic Processes Wiley, New York 996 S Ross A First Course in Probability, 6th ed , Prentice-Hall, Englewood Cliffs, NJ, 2002 S Ross Prohahi!ity Models for Computer Scien ce Academic Press, Orlando, FL , 2002 R W Wolff Stochastic Modeling and the Theor.v of Queues Prentice-Hall NJ, 989 349 Cliffs Index ::P 255 Poisson random variable IJ7 �rithmetic coding, Poisson tri a l s 63-7 250 sum of { - I , + I } random variables 69-7 Azum�-Hoeffding i neq u a l i ty, 303 weig hted sum of Poisson triab 86 compression 234-7 compression fu nction 235 balanced � l l ocation 336-45 bal lot theorem 291.) cond i ti onal density fu nctio n 193 ba l l s and bins model 92-1 04 hal b and bins with feedbac k 91.) - W I condi tional entropy 246 Bernou l l i random variable 25 cond i tional expectation i nequ a l i ty 36-8 condi tional expectation 26 26-30 I J I -3 random variable Bayes· law expectat i on 25 27 I J.O 319 condi tional probabi l i ty confidence i nt en·a l 68 variance 48 Bernou l l i trials 63 connectiv ity a lgori t h m , 76 b i n ary erasure chan n e l 25 convex function 24 hinary sym metric channel 238 count-min fi lter 329-32 binomial coefficients hounds 228-9 b i nomial distri bution 25 binomial identi ty 26 binomial random \ ariab l e 25 expectation 26 \·ariance -18 binomial theorem -IK conservative update 3 c o u p l i n g 274 27 -89 2lJ coupon col l ector's problem, 32- -1- 50 0-1-6 I expectat ion 3 variance covarianc� 46 35 2� birthday paradox lJ0-2 Cllh - ( st ream bit-fi x ing routing algori t h m 7-1 71.) go_ �8 data B l oom fi l ter 109-1 29 decoding fu nct i o n 238 branching process 30 delay sequence B u bblesort , 58 94 density fu nction 89 B ucket sort 1.)3 model 329 condi ti onal 93 B utlon's needle 267 dependency graph 39 burst errors derandomization 26 cond i t ional e x pectat ions -3 butterfly wrapped 78 routing a lgorit h m 80 chain ha'> hi n g 06-7 sy mmetric pairwise independence - di screte probabi l i ty space set> probabi l i ty -;pace d i screte channel 2.'8 binary 59 92 238 d i screte random \'ari able set' randum \ ariable d i -;netl' d i stribution function 89 joint d i stri but i o n, Chebyshev's i ney u a l i ty -l�-1.), 56 I J J pairw i se independent variables �- I I.) D N F for m u l a 255 Chernoff bound 63 -83 appro x i mat i ng the nu mber of satisfying assignme nt geometric random variables, 85 255-9 350 I l'i D E X Doob mart i ng a l e 296 dynamic resource a l l l)L·at i on �-+5 i n c l u -.,i on- exclu sion pri n c i p l e -+ l h i ndependen c e I k - w i s e edge - d i sjoint path � -+ m utual edge exposure marti ngal e 296 pairwise embedded M arkm chai n independent set 3 - encoding fun c t i on 238 appro x i mate c o u n t i n g entropy 225 225 -45 entropy fu nction bin ary 226 indicator random vari abl e 25 26 c' - u n i form sam ple 259 l ensen ·.s i nequal i ty 23 + ( E r5 ) -appr0\ i m at i o n 253 25-+ eve n t set notati on simple e x pectat i o n 20-6 28-3 Bernou l l i random variab l e 25 binomial random variable conditional s e e conditi onal ex pectation conti nuous random variab le 90 coupon col lector " s problem 3 e x ponenti a l random variab le geometric random variable l i nearity of see l i nearity o f expectations sam p l i ng 7-8 2.86-9 Kraft i n e q u a l i ty 249 k -S AT 42-6 large c u t s 29-30 Las Vegas al gorithm 57 28 30 l aw of total probab i l i ty leader e l ectio n 1 l i near i n sertion sort 94 l i n earity of l'xpectat i ons 2 Litt l e ' s resu l t Lovasz local l e m m a 8-48 ex p l i c i t constructi ons 42 Poi sson random variable 96 general case 4f1 48 product of random vari abl es 46 symmetriL· versi o n 39-4 u n i form random vari ab le 94 e x ponential d i s tri bution 96 205 den�ity 97 marg i n a l d i stribution fu ncti on Markov chai n 53 3-2 di stribution function 96 aperi odic 65 memor y l e s s property commu n i cating states 64 mini mu m coupl i ng 274 2.7 -89 e xponenti al random variable 96 directed graph representat i o n 5 ex pectat i o n embedded equ i l ibri um d i stri bution 67 variance extracting random bits 30-4 ergod ic 66 extraction function 230 248 i rreduc i b l e 64 factori al bou nds 93 02 memory less property Markov property 54 fu l ly polynomial a l most u n i form sampler ( F PAU S ) m i x i ng time 274 259 59-()3 n u l l recurre n t 65 fu l l y po lynomial random i zed appro x i mation scheme I pat h coup l i ng 286-9 F PR A S 25 7-63 267 periodic 65 positive recurrent 65 gambl e r \ ru i n l f1f1-7 2Y8 pidly mi x i ng 274 geometric di stribu t i on 30 62 recurre n t 64 geometric random \ ari able 30 -.,kel eton e x pectat i o n -,tate 53 variance t i m e - reversi ble 83 264 266 memory less propert ) 30 statio nary d i stribution 67 67-72 g i rth 34 transi ent 64 Ham i lto n i an cycle 1 tran -.,ition probabi l ity 54: n - '>t e p 54 transition matri x 54 al gori t h m 1 harmonic number 33 has h i n g 06- 344-5 fi ngerpri n t 08 perfect hashing f1-8 u n iwr.sal hash fun cti ons -S heavy hi tters 3 al gori t h m hypercube 73 routing algori t h m 75 t\1arkm chain Monte Carlo m ethl1d 2h.� - :=; !\ larkm proce-.,�e� 10 \1 arkm · , i nelj ua l i t ) -+-+-5 mart i ng a l e 295 295- �0S -\z u m a-H odfd i n g i n c q u Jh � t u pp i n g t h enre m l) :-; \ erte.\ ex pu�urc 2lJ- m a x i m u m L· ut - ' m a x i m u m �a t i �fiabi l n : [ _, 1 - l 35 -� � � � INDEX median a lgori t h m 52-7 random walk 74, 17 4-7 cover t i me 76 memoryless property, 30 stationary distri but ion 75 Metropo l i s scheme, 265 m i n-cut a lgori t h m 2- randomized routing 72-83 m i n i mu m cut-set hypercube, moment generating function , L -3 wrapped butterfly 80 Poi sson di stributi o n , 97 re.�ervoir sampl ing, 40 moments 45 Monte Carlo a l gori t h m 57, 335 sample space Monte Carlo method 252-66 sam p l i ng, pairw i se i ndependence -2 negative b i n o m i al random variab l e 40 order stati stics, 207 u n i form random variables 208 packet routing 72-83 hypercube, 75 wrapped butterfly 80 pai rwi se i ndependence, constructions 5- sam p l i n g -2 Parrondo ' s paradox, 77 with rep l acement wi thout rep l acemen t sati sfiabi l i ty 30 42-6 second moment method 34-6 set balancing set members h i p prob l e m , 08 S hannon's theore m - 45 skeleton Markov c hai n , standard deviati on 45 stat i onary di stribut i o n 79 continuous ti me Markov processes, 0- perfect hashing, 326-8 s-t connect ivity al gori t h m , 76 permutat i o n stoc hastic cou nti ng process, 20 PASTA pri n c i p l e Poisson appro x i mation 99- 05, I l l Poisson distribut i o n , 95 202 moment generat i n g funct i o n 97 Poi sson process 202 20 -9 comb i n i n g 2.05-7 i nterarrival distributi o n s p l i t t i n g , 205-7 Poisson random variable 63 95 Stirl i n g ' s form u l a 62 246 stochastic process 53 stoppi ng t i m e 297, 297-300 symmetry break i n g 1 - 2-SAT algori t h m 6-9 - S AT 59 algori t h m Chernoff bound, 97 expectation, 96 sums, 96 u n i form di stribution 93 208 conditional probabi l i ty, 94 Poi sson tri al , 63 density fu nct i o n 94 pre fi x code 34 249 pri n c i p l e of deferred dec i s i o n s 1 pri ority queue 224 d i stributi o n function 94 u n i form random vari able 93 expectat i o n 94 probabi l i stic method 26-48 242 counting arg u ment 26-8 expectation argu ment 28-3 Lovasz local lemma 8-48 sample and modi fy 3-4 second moment method 34-6 variance 94 u n i on bound 4, u niversal hash functions, 32 -8 variance 45 Bernoul l i random vari ab le 48 probabi l ity fu nction b i n o m i al random vari abl e 48 probabi l ity space continuous random vari abl e 90 discrete e x ponent i a l random vari ab le, 97 geometric random vari ab le, q ueues 3-4 2 - M/ Mj I , 3- M/M/ /K MjMjXJ 6- notat i o n , 2 Qui cksort 34-8 s u m 46 sum of pairw i se i ndependent vari ab les u n i form random vari ab le 94 variati on di stance 272 veri fy i ng m atri x m u l t i p l ication, 8- pivot 34 random graphs 1 2-1 296 threshold functi o n , 35-8 polynomi al i dentities 1-8 vertex c o l oring sam p l i n g , 82-6 vertex exposure mart i ngal e 297 random vari able, 20, 20-6 conti nuous 89, 88-93 d i screte 20 Wald ' s equation 300-2 weak l aw of large numbers 60 352 " Probability is part of the conceptual core of modern computer science Probabilistic analysis of algorithms, randomized algorithms, and probabilistic combinatorial constructions have become fundamental tools for computer science and applied mathematics This book provides a thorough grounding in discrete probability and its applications in computing, at a level accessible to advanced undergraduates in the computational, mathematical, and engineering sciences " - Richard M Karp, University Professor, University of California at Berkeley "An exciting new book on randomized algorithms It nicely covers all the basics, and also has some interesting modern applications for the more advanced student " - Alan Frieze, Professor of Mathematics, Carnegie-Mellon University Randomization and probabilistic techniques play an important role in modern computer science, with applications ranging from combinatorial opti­ mization and machine learning to communication networks and secure protocols This textbook is designed to accompany a one- or two-semester course for advanced undergraduates or beginning graduate students in computer science and applied mathematics It gives an excellent introduction to the probabilistic tech­ niques and paradigms used in the development of probabilistic algorithms and analyses It assumes only an elementary background in discrete mathematics and gives a rigorous yet accessible treatment of the material, with numerous examples and applications The first half of the book covers core material, including random sampling, expectations, Markov's inequality, Chebyshev's inequality, Chernoffbounds, balls­ and-bins models, the probabilistic method, and Markov chains In the second half, the authors delve into more advanced topics such as continuous probability, appli­ cations of limited independence, entropy, Markov chain Monte Carlo methods, coupling, martingales, and balanced allocations With its comprehensive selection of topics, along with many examples and exercises, this book is an indispensable teaching tool Michael Mitzenmacher is John L Loeb Associate Professor in Computer Science at Harvard University Eli Upfal is Professor and Chair of Computer Science at Brown University CAMBRIDGE UNIVERS ITY PRE S S www cambridge.org I S B N 0-5 -83540-2 Cover illustration by Sir John Tennielftom The Nursery "Alice" by Lewis Carroll, published by Macmillan and Company in 1899 Cover desi_gn by james F Brisson 1111 1111 11 111 111 1111 7805 835404 > [...]... analysis of randomized algorithms, where the probability space is defined by the random choices made by the algorithm and the probabilistic analysis of deterministic algorithms, where the probabil ity space is defined by some probability di stribution on the inputs Along the way we define the Bernoulli binomiaL and geometric random variables, study the expected size of a simple branching process and analyze... excellent book Randomized Algorithms We are grateful to Lauren Cowles of Cambridge University Press for her editorial help and advice in preparing and organizing the manuscript Writing of this book was supported i n part by NSF ITR Grant no CCR-0 1 2 1 1 54 xvi C HAPTE R ONE Events and Probability Thi s chapter introduces the notion of randomized algorithms and reviews some basic L"oncepts of probability. .. 5 ) and Harvard (CS 223) in recent years The e mphasis xiv PREFA C E 1n these courses and in thi s textbook i s on the probabilistic techniques and paradigms, not on particular applications Each chapter of the book is devoted to one such method )r technique Techniques are clarified though examples based on analyzing random­ lied algorithms or developing probabilistic analysis of algorithms on random... Bob chooses die A and Alice chooses die B Write out all of the pos­ sible events and their probabilities, and show that the probability that Alice wins is greater than 1 / 2 t b ) Suppose that Bob chooses die B and Alice chooses die C Write out all of the pos­ sible events and their probabilities, and show that the probability that Alice wins is greater than 1 / 2 t c ) Since die A and die B lead to... die C and Alice chooses die A Write out all o f the possible events and their probabi lities, and show that the prob­ ability that Alice wins is still greater than 1 / 2 15 E VENTS AND PROBABILITY Exercise 1.6: Consider the fol l owing balls -and- bin game We start with one black ball and one white ball in a bin We repeatedly do the following: choose one ball from the bin uniformly at random , and then... shared Ethernet communication medium The randomness i s useful for breaking symmetry, preventing different cards from repeatedly accessing the medium at the same time Other commonly used ap­ plications of randomized algorithms include Monte Carlo simulations and primality testing in cryptography In these and many other i mportant applications, randomized algorithms are significantly more efficient... events extends t o random variables Definition 2.2: Two random variables X and Y are independent (f and only �f Pr( ( X = x) n (Y y)) Pr( X x ) · Pr ( Y = y ) t(;r all values x and y Similarly, random variables X1 • X 2 X �.: a re mutually independent if and onl_v u; for any subset I [ I k ] and any \ 't.tlues Xi · i E / • Pr (n • • • • n Pr ( Xi = ri ) I I -\ basic characteristic of a random variable... XIOMS OF PROBAB IL ITY 1.2 Axioms of Probability We turn now to a formal mathematical setting for analyzing the randomized algorithm Any probabilistic statement must refer to the underlying probability space Definition 1.1: A probability space has three components: 1 2 3 a sample space Q , which is the set of all possible outcomes of the random process modeled by the probability space; a family of sets... land­ ing heads with probability 2 / 3 We are not told which of the three coins is biased We 10 1.3 APPL ICATION: VERIFYING MAT R I X MULT IPL I C ATION permute the coins randomly, and then flip each of the coins The first and second coins come up heads, and the third comes up tails What is the probability that the first coin is the biased one? The coins are i n a random order and so, before our observing... elementary probability theory while introduci ng some i nter­ c ting applications Topics covered include random sampling, expectation, Markov ' s mequality, variance, and Chebyshev's inequality If the class has sufficient background m probability, then these chapters can be taught q uickly We do not suggest skipping them, however, because they i ntroduce the concepts of random ized algorithms and rrobabilistic