1032 PROBABILITY THEORY Events H 1 , , H n form a complete group of pairwise incompatible events (are hy- potheses) if exactly one of the events necessarily occurs for each realization of the condition set Σ, i.e., if H 1 ∪···∪H n = Ω and H i ∩ H j = ∅ (i ≠ j). Main properties of random events: 1. A ∪ B = B ∪ A and A ∩ B = B ∩ A (commutativity). 2. (A ∪B) ∩C =(A ∩C) ∪(B ∩C)and(A ∩B) ∪C =(A ∪C)∩(B ∪C) (distributivity). 3. (A ∪ B) ∪C = A ∪ (B ∪ C)and(A ∩ B) ∩C = A ∩ (B ∩ C) (associativity). 4. A ∪ A = A and A ∩ A = A . 5. A ∪ Ω = Ω and A ∩ Ω = A. 6. A ∪ A = Ω and A ∩A = ∅. 7. ∅ = Ω, Ω = ∅,andA = A; 8. A\B = A ∩ B. 9. A ∪ B = A ∩ B and A ∩ B = A ∪ B (de Morgan’s laws). 20.1.1-2. Axiomatic definition of probability. In the case of an uncountable sample space Ω, not all subsets of Ω but only subsets belonging to certain classes, called algebras of sets and σ-algebras, are viewed as events. AclassF of subsets of the space Ω is called an algebra of sets (events) if ∅ F, Ω F, and the following conditions are satisfied: 1. If A F,thenA F. 2. If A F and B F,thenA ∪ B F and A ∩ B F. The simplest example of an algebra of events is the system F ={∅, Ω}. Indeed, applying any of the operations listed above to any elements of the class F, we obtain an element of this class: ∅ ∪ Ω = Ω, ∅ ∩ Ω = ∅, ∅ = Ω,and∅ = Ω. An algebra of sets F of subsets of the space Ω is called a σ-algebra if the following condition is satisfied: if A n F, n = 1, 2, ,then ∪ n A n F and ∩ n A n F. The elements of F are called random events. The probability of an event is defined to be a single-valued real function P (A)defined on the σ-algebra of events F and satisfying the following three axioms: 1. Nonnegativity: P (A) ≥ 0 for any A F. 2. Normalization: P(Ω)=1. 3. Additivity: P ∪ n A n = n P (A n ), provided that A i ∩ A j = ∅ whenever i ≠ j. A probability space isatriple(Ω, F,P ), where Ω = Ω(ω) is a space of elementary events, F is a σ-algebra of subsets of Ω, called random events, and P (A) is a probability definedontheσ-algebra F. Properties of probability: 1. The probability of an impossible event is zero; i.e., P (∅)=0. 2. The probability of the event A opposite to an event A is equal to P (A)=1 – P (A). 3. Probability is a bounded function; i.e., 0 ≤ P (A) ≤ 1. 4. If an event A implies an event B (A ⊆ B), then P (A) ≤ P (B). 5. If events H 1 , , H n form a complete group of pairwise incompatible events, then n i=1 P (H i )=1. 20.1. SIMPLEST PROBABILISTIC MODELS 1033 20.1.1-3. Discrete probability space. Classical definition of probability. Suppose that Ω = {ω 1 , , ω n } is a finite space. The σ-algebra F of events includes all 2 n subsets of Ω. To each elementary event ω i Ω (i = 1, 2, , n) there corresponds a number p(ω i ), called the probability of the elementary event ω i . Thus a real function satisfying the following two conditions is definedonthesetΩ: 1. Nonnegativity condition: p(ω i ) ≥ 0 for any ω i Ω. 2. Normalization condition: n i=1 p(ω i )=1. The probability P (A)ofaneventA for any subset A ⊂ Ω is definedtobethesumof probabilities of the elementary events that form A; i.e., P (A)= ω i A p(ω i ). (20.1.1.1) The triple (Ω, F,P ) thus defined is a finite discrete probability space. A special case of the definition of probability (20.1.1.1) is the classical definition of probability, in which all elementary events are equiprobable: p(ω 1 )=··· = p(ω n )=1/n. Then the probability P (A)ofaneventA = {ω i 1 , , ω i m } is equal to the ratio of the number of elementary events ω i contained in A (the number of outcomes that favor A) to the total number of elementary events in Ω (the number of all possible outcomes), P (A)= |A| |Ω| = m n .(20.1.1.2) Example 1. Let two dice be thrown. Under the assumption that the elementary events are equiprobable, find the probability of the event A that the sum of numbers shown is greater than 10. Obviously, the sample space can be represented as Ω = {(i, j):i, j = 1, 2, 3, 4, 5, 6},wherei is the number shown by the first die and j is the number shown by the second die. The total number of elementary events is |Ω| = 36.The event A corresponds to the subset A = {(5, 6), (6, 5), (6, 6)} of Ω.Since|A| = 3, formula (20.1.1.2) gives P (A)=|A|/|Ω| = 3/36 = 1/12. 20.1.1-4. Sampling without replacement. Let Δ = {1, , n} be the set of n numbers, and let ω =(i 1 , , i m ) be an ordered sequence of m elements of Δ. Random sampling without replacement is the sampling scheme in which Ω = {ω =(i 1 , , i m ):i k Δ, k = 1, , m,andalli 1 , , i m are distinct} (20.1.1.3) and all elementary events ω are equiprobable. When calculating probabilities by formula (20.1.1.2), the following combinatorial for- mulas are often useful. Suppose that a set Δ = {a 1 , , a n } of n elements is given. Subsets of Δ are called combinations. The number of distinct combinations of n elements of Δ taken m at a time is denoted by C m n or n m . The following formula holds: C m n ≡ n m = n! (n – m)! m! .(20.1.1.4) Ordered sequences a i 1 , , a i m of distinct elements of Δ are called arrangements,or permutations of n elements taken m at a time. The number of arrangements of m out of n 1034 PROBABILITY THEORY elements, i.e., the number of ordered sequences of m distinct elements selected from n elements, is denoted by A m n . One has A m n = n(n – 1) (n – m + 1)= n! (n – m)! .(20.1.1.5) Arrangements with m = n are called permutations. The number of distinct permutations P n on n elements is given by the formula P n = n!. (20.1.1.6) 20.1.1-5. Sampling with replacement. The sampling scheme in which Ω = {ω =(i 1 , , i m ):i k Δ, k = 1, 2, , n} (20.1.1.7) and all elementary events ω are equiprobable is called random sampling with replacement. If, for random sampling of m out of n elements with replacement, no subsequent ordering is performed (i.e., each of the n elements can occur 0, 1, ,orm times in any combination), then one speaks of combinations with repetitions. The number C m n of all distinct combinations with repetitions of n elements taken m at a time is given by the formula C m n = C m n+m–1 .(20.1.1.8) If, for random sampling of m out of n elements with replacement, the chosen elements are ordered in some way, then one speaks of arrangements with repetitions. The number A m n of distinct arrangements with repetitions of n elements taken m at a time is given by the formula A m n = n m .(20.1.1.9) Suppose that a set of n elements contains k distinct elements, of which the first occurs n 1 times, the second occurs n 2 times, ,andthekth occurs n k times, n 1 + ···+ n k = n. Permutations of n elements of this set are called permutations with repetitions on n elements. The number P n (n 1 , , n k ) of permutations with repetitions on n elements is given by the formula P n (n 1 , , n k )= n! n 1 ! n k ! .(20.1.1.10) Example 2. Consider the set {1, 2, 3} of n = 3 elements. The elements of this set give P 3 = 3!=6 permutations: (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1). For m = 2,thereare 1. C 2 3 ≡ 3 2 = 3 combinations without repetitions [(1, 2), (1, 3), (2, 3)]. 2. C 2 3 = C 2 3+2–1 = C 2 4 = 6 combinations with repetitions [(1, 2), (1, 3), (2, 3), (1, 1), (2, 2), (3, 3)]. 3. A 2 3 = 3 arrangements without repetitions [(1, 2), (1, 3), (2, 3), (2, 1), (3, 2), (3, 1)]. 4. A 2 3 = 3 2 = 9 arrangements with repetitions [(1, 2), (1, 3), (2, 3), (2, 1), (3,2), (3, 1), (1, 1), (2, 2), (3, 3)]. 20.1.1-6. Geometric definition of probability. Let Ω be a set of positive finite measure μ(Ω)inthen-dimensional Euclidean space, and let the σ-algebra F consist of all measurable (i.e., having a measure) subsets A ⊆ Ω.The geometric probability of an event A is defined to be the ratio of the measure of A to that of Ω, P (A)= μ(A) μ(Ω) (A F). (20.1.1.11) The notion of geometric probability is not invariant under transformations of the domain Ω and depends on how the measure μ(A) is introduced. 20.1. SIMPLEST PROBABILISTIC MODELS 1035 Example 3. A point is randomly thrown into a disk of radius R = 1. Find the probability of the event that the point lands in the disk of radius r = 1 2 centered at the same point. First method.LetA be the event that the point lands in the smaller disk. We find the probability P (A)as the ratio of the area of the smaller disk to that of the larger disk: P (A)= πr 2 πR 2 = 1 4 . Second method. Consider the polar coordinate system in which the position of a point is determined by the angle ϕ between the position vector of the point and the axis OX and by the distance ρ from the point to the origin. Since all points equidistant from the center either belong or do not belong to the smaller disk simultaneously, it follows that the probability of landing in this disk is equal to the ratio of the radii: P (A)= r R = 1 2 . Thus we have obtained two different answers in the same problem. The cause is that the notion of geometric probability is not invariant under transformations of the domain Ω and depends on how the measure μ(A)is introduced. 20.1.2. Conditional Probability and Simplest Formulas 20.1.2-1. Probability of the union of events. The probability of realization of at least one of two events H 1 and H 2 is given by the formula P (H 1 ∪ H 2 )=P (H 1 )+P (H 2 )–P (H 1 ∩ H 2 ). (20.1.2.1) In particular, for H 1 ∩ H 2 = ∅,wehave P (H 1 ∪ H 2 )=P (H 1 )+P (H 2 ). (20.1.2.2) The probability of realization of at least one of n events is given by the formula P (H 1 ∪···∪H n )= n k=1 P (H k )– 1≤k 1 <k 2 ≤n P (H k 1 ∩ H k 2 ) + 1≤k 1 <k 2 <k 3 ≤n P (H k 1 ∩ H k 2 ∩ H k 3 )–···+(–1) n–1 P (H 1 ∩···∩H n ). (20.1.2.3) 20.1.2-2. Conditional probability. The conditional probability P (A|H), or P H (A), of an event A given the occurrence of some other event H is defined by the formula P (A|H)= P (A ∩ H) P (H) , P (H)>0.(20.1.2.4) The conditional probability P (A|H) can be treated as the probability of the event A under the condition that the event H occurs. Relation (20.1.2.4) can be written as the “probability multiplication theorem” P (A ∩ H)=P (H) P (A|H). (20.1.2.5) The formula P (A 1 ∩···∩A n )=P (A 1 ) P (A 2 |A 1 ) P (A 3 |A 1 ∩ A 2 ) P(A n |A 1 ∩···∩A n–1 ) is a generalization of (20.1.2.5). 1036 PROBABILITY THEORY 20.1.2-3. Independence of events. Two random events A and B are said to be statistically independent if the conditional probability of A,givenB, coincides with the unconditional probability of A, P (A|B)=P (A). (20.1.2.6) Random events A 1 , , A n are jointly statistically independent if the relation P m k=1 A i k = m k=1 P (A i k )(20.1.2.7) holds whenever 1 ≤ i 1 < ··· < i m ≤ n and m ≤ n. The pairwise independence of the events A i and A j for all i ≠ j (i, j = 1, 2, , n) does not imply that the events A 1 , , A n are jointly independent. Example 1. Suppose that the experiment is to draw one of four balls. Let three of them be labeled by the numbers 1, 2,and3, and let the fourth ball bear all these numbers. By A i (i = 1, 2, 3) we denote the event that the chosen ball bears the number i.AretheeventsA 1 , A 2 ,andA 3 dependent? Since each number is encountered twice, P (A 1 )=P (A 2 )=P (A 3 )=1/2. Since any two distinct numbers are present only on one of the balls, we have P (A 1 A 2 )=P (A 2 A 3 )=P (A 1 A 3 )=1/4, and hence the events A 1 , A 2 ,andA 3 are pairwise independent. All three distinct numbers are present only on one of the balls, and P (A 1 A 2 A 3 )=1/4 ≠ P (A 1 ) P (A 2 ) P (A 3 )=1/8. Thus we see that the events A 1 , A 2 ,andA 3 are jointly dependent, even though theyare pairwise independent. 20.1.2-4. Total probability formula. Bayes’ formula. Suppose that a complete group of pairwise incompatible events H 1 , , H n is given and the unconditional probabilities P (H 1 ), , P (H n ), as well as the conditional probabilities P (A|H 1 ), , P (A|H n )ofaneventA, are known. Then the probability of A can be determined by the total probability formula P (A)= n k=1 P (H k ) P (A|H k ). (20.1.2.8) Example 2. There are two urns; the first urn contains a white and b black balls, and the second urn contains c white and d black balls. We take one ball from the first urn and put it into the second urn. After this, we draw one ball from the second urn. Find the probability of the event that this ball is white. Let A be the event of drawing a white ball. Consider the following complete group of events: H 1 , a white ball is taken from the first urn and put into the second urn. H 2 , a black ball is taken from the first urn and put into the second urn. Obviously, P (H 1 )= a a + b , P (H 2 )= b a + b ; P (A|H 1 )= c + 1 c + d + 1 , P (A|H 2 )= c c + d + 1 . Now by the total probability formula (20.1.2.8) we obtain P (A)=P (H 1 ) P (A|H 1 )+P (H 2 ) P (A|H 2 )= a a + b c + 1 c + d + 1 + b a + b c c + d + 1 . If it is known that the event A has occurred but it is unknown which of the events H 1 , , H n has occurred, then Bayes’ formula is used: P (H k |A)= P (H k ) P (A|H k ) P (A) (k = 1, 2, , n). (20.1.2.9) 20.1. SIMPLEST PROBABILISTIC MODELS 1037 Example 3. The urn contains one ball. It is known that this ball is either white or black. We put a black ball into the urn, then thoroughly shuffle the balls and draw one of the balls, which turns out to be black. Find the probability of the event that the ball remaining in the urn is black. We consider the following random events that form a complete group of pairwise incompatible events: H 1 , the urn initially contained a white ball. H 2 , the urn initially contained a black ball. We have P (H 1 )=P (H 2 )=1/2. Suppose that the event A is that a black ball is drawn. Then P (A|H 1 )=1/2 and P (A|H 2 )=1. The desired probability of the event H 2 |A is determined by Bayes’ formula: P (H 2 |A)= P (H 2 ) P (A|H 2 ) P (A) = 1× 1 2 1× 1 2 + 1 2 × 1 2 = 2 3 . 20.1.3. Sequences of Trials 20.1.3-1. Mathematical model of a sequence of n independent trials. Trials in which events occurring in distinct trials are independent are said to be independent. Each trial S k can be viewed as a probability space (Ω k , F k , P k ). Independent trials are described by the probability space (Ω, F, P) that is the direct product of the probability spaces (Ω k , F k , P k ). Here the probability of each event A of the form A = A 1 × ···× A n is defined as P (A)=P (A 1 ) P(A n ). A sequence of n independent trials is also called a Bernoulli scheme.Letp k = P (A k ). Then the probability of the event that n 1 events A 1 , n 2 events A 2 , ,andn k events A k occur in n independent trials is equal to P n (n 1 , , n k )= n! n 1 ! n k ! p n 1 1 p n k k .(20.1.3.1) The probability (20.1.3.1) is the coefficient of x n 1 1 x n k k in the expansion of the poly- nomial (p 1 x 1 + ···+ p k x k ) n in powers of x 1 , , x k . The probability P n (n 1 , , n k ) can be found by the technique of generating functions, since this probability is the coefficient of z n 1 1 z n k k in the generating function ϕ X (z 1 , , z k )=(p 1 z 1 + ···+ p k z k ) n . 20.1.3-2. Bernoulli process. The special case of the Bernoulli scheme with N = 2 is called the Bernoulli process.In this case, some event A occurs with probability p = P (A) (the probability of “success”) and does not occur with probability q = P( A)=1 – P (A)=1 – p (the probability of “failure”) in each trial. If μ n is the number of occurrences of the event A (the number of “successes”) in n independent Bernoulli trials, then the probability that A occurs exactly k times is given by the formula P (μ n = k)=p n (k)=C k n p k (1 – p) n–k (k = 0, 1, , n). (20.1.3.2) Relation (20.1.3.2) is called the Bernoulli formula (binomial distribution). The probability that the event occurs at least m times in n independent trials is calculated by the formula p n (k ≥ m)= n k=m p n (k)=1 – m–1 k=0 p n (k). 1038 PROBABILITY THEORY The probability that the event occurs at least once in n independent trials is calculated by the formula p n (k ≥ 1)=1 –(1 – p) n . The number n of independent trials necessary for the event to occur at least once with probability at least P is given by the formula n ≥ ln(1 – P ) ln(1 – p) . Example (Banach’s problem). A smoker mathematician has two matchboxes on him, each of which initially contains exactly n matches. Each time he needs to light a cigarette, he selects a matchbox at random. Find the probability of the event that as the mathematician takes out an empty box for the first time, precisely k matches will be left in the other box (k ≤ n). The mathematician has taken matches 2n – k times, n out of them from the box that is eventually empty. This scheme corresponds to the scheme of 2n – k independent Bernoulli trials with n “successes.” The probability of a “success” in a single trial is equal to 0.5. The desired probability can be found by the formula P (μ 2n–k = n)=p 2n–k (n)=C 2n–k n p n (1 – p) n–k = C 2n–k n 1 2 2n–k . 20.1.3-3. Limit theorems for Bernoulli process. It is very difficult to use Bernoulli’s formula (20.1.3.2) for large n and m. In this case, one has to use approximate formulas for calculating p n (k) with desired accuracy. Poisson formula. If the number of independent trials increases unboundedly (n →∞) and the probability p simultaneously decays (p → 0) so that their product np is a constant (np = λ = const), then the probability P (μ n = k)=p n (k) satisfies the limit relation lim n→∞ P (μ n = k)= λ k k! e –λ .(20.1.3.3) Local theorem of de Moivre–Laplace. Suppose that n →∞, p = const, 0 < p < 1,and 0 < c 1 ≤ x n,k =(k – np)[np(1 – p)] –1/2 ≤ c 2 < ∞ ;then P (μ n = k)= 1 √ 2πnp(1 – p) e –x 2 n,k /2 [1 + O(1/ √ n)] (20.1.3.4) uniformly with respect to x n,k [c 1 , c 2 ]. Integral theorem of de Moivre–Laplace. Suppose that n →∞and p = const, 0 < p < 1; then P x 1 < μ n – np √ np(1 – p) < x 2 → 1 √ 2π x 2 x 1 e –t 2 /2 dt (20.1.3.5) uniformly with respect to x 1 and x 2 . The approximate formula (20.1.3.3) is normally used for n ≥ 50 and np ≤ 10.The approximate formulas (20.1.3.4) and (20.1.3.5) are used for np (1 – p)>9. The error in formulas (20.1.3.4) and (20.1.3.5) can increase owing to the fact that the prelimit distribution is discrete. This error is of the order of O([np(1 – p)] –1/2 ). The limit expression in (20.1.3.4) can readily be expressed via the probability density ϕ(x)= 1 √ 2π e –x 2 /2 of the standard normal distribution, and the limit expression in (20.1.3.5) can be expressed in terms of the cumulative distribution function Φ(x)= 1 √ 2π x –∞ e –t 2 /2 dt of the standard normal distribution. . (20.1.3.5) uniformly with respect to x 1 and x 2 . The approximate formula (20.1.3.3) is normally used for n ≥ 50 and np ≤ 10.The approximate formulas (20.1.3.4) and (20.1.3.5) are used for np (1. (20.1.1.7) and all elementary events ω are equiprobable is called random sampling with replacement. If, for random sampling of m out of n elements with replacement, no subsequent ordering is performed. probability of an event A is defined to be the ratio of the measure of A to that of Ω, P (A)= μ(A) μ(Ω) (A F). (20.1.1.11) The notion of geometric probability is not invariant under transformations of