Bài Giải Giáo trình này giành cho sinh viên ngành kỹ thuật viễn thông,Giáo trình được sử dụng cho bậc đại học để có thể hiểu được lý thuyết thông tin và cách truyền tin.Cảm ơn bạn đã xem tải tài liệu của mình .Theo dõi trang của mình để nhận thêm nhiều tài liệu nhé.
Elements of Information Theory Second Edition Solutions to Problems Thomas M Cover Joy A Thomas October 17, 2006 COPYRIGHT 2006 Thomas Cover Joy Thomas All rights reserved Contents Introduction Entropy, Relative Entropy and Mutual Information The Asymptotic Equipartition Property 49 Entropy Rates of a Stochastic Process 61 Data Compression 97 Gambling and Data Compression 139 Channel Capacity 163 Differential Entropy 203 Gaussian channel 217 10 Rate Distortion Theory 241 11 Information Theory and Statistics 273 12 Maximum Entropy 301 13 Universal Source Coding 309 14 Kolmogorov Complexity 321 15 Network Information Theory 331 16 Information Theory and Portfolio Theory 377 17 Inequalities in Information Theory 391 CONTENTS Preface Here we have the solutions to all the problems in the second edition of Elements of Information Theory First a word about how the problems and solutions were generated The problems arose over the many years the authors taught this course At first the homework problems and exam problems were generated each week After a few years of this double duty, the homework problems were rolled forward from previous years and only the exam problems were fresh So each year, the midterm and final exam problems became candidates for addition to the body of homework problems that you see in the text The exam problems are necessarily brief, with a point, and reasonable free from time consuming calculation, so the problems in the text for the most part share these properties The solutions to the problems were generated by the teaching assistants and graders for the weekly homework assignments and handed back with the graded homeworks in the class immediately following the date the assignment was due Homeworks were optional and did not enter into the course grade Nonetheless most students did the homework A list of the many students who contributed to the solutions is given in the book acknowledgment In particular, we would like to thank Laura Ekroot, Will Equitz, Don Kimber, Mitchell Trott, Andrew Nobel, Jim Roche, Vittorio Castelli, Mitchell Oslick, Chien-Wen Tseng, Michael Morrell, Marc Goldberg, George Gemelos, Navid Hassanpour, Young-Han Kim, Charles Mathis, Styrmir Sigurjonsson, Jon Yard, Michael Baer, Mung Chiang, Suhas Diggavi, Elza Erkip, Paul Fahn, Garud Iyengar, David Julian, Yiannis Kontoyiannis, Amos Lapidoth, Erik Ordentlich, Sandeep Pombra, Arak Sutivong, Josh Sweetkind-Singer and Assaf Zeevi We would like to thank Prof John Gill and Prof Abbas El Gamal for many interesting problems and solutions The solutions therefore show a wide range of personalities and styles, although some of them have been smoothed out over the years by the authors The best way to look at the solutions is that they offer more than you need to solve the problems And the solutions in some cases may be awkward or inefficient We view that as a plus An instructor can see the extent of the problem by examining the solution but can still improve his or her own version The solution manual comes to some 400 pages We are making electronic copies available to course instructors in PDF We hope that all the solutions are not put up on an insecure website—it will not be useful to use the problems in the book for homeworks and exams if the solutions can be obtained immediately with a quick Google search Instead, we will put up a small selected subset of problem solutions on our website, http://www.elementsofinformationtheory.com, available to all These will be problems that have particularly elegant or long solutions that would not be suitable homework or exam problems CONTENTS We have also seen some people trying to sell the solutions manual on Amazon or Ebay Please note that the Solutions Manual for Elements of Information Theory is copyrighted and any sale or distribution without the permission of the authors is not permitted We would appreciate any comments, suggestions and corrections to this solutions manual Tom Cover Durand 121, Information Systems Lab Stanford University Stanford, CA 94305 Ph 650-723-4505 FAX: 650-723-8473 Email: cover@stanford.edu Joy Thomas Stratify 701 N Shoreline Avenue Mountain View, CA 94043 Ph 650-210-2722 FAX: 650-988-2159 Email: joythomas@stanfordalumni.org Chapter Introduction Introduction Chapter Entropy, Relative Entropy and Mutual Information Coin flips A fair coin is flipped until the first head occurs Let X denote the number of flips required (a) Find the entropy H(X) in bits The following expressions may be useful: ∞ ∞ , r = 1−r n=0 n nr n = n=0 r (1 − r)2 (b) A random variable X is drawn according to this distribution Find an “efficient” sequence of yes-no questions of the form, “Is X contained in the set S ?” Compare H(X) to the expected number of questions required to determine X Solution: (a) The number X of tosses till the first head appears has the geometric distribution with parameter p = 1/2 , where P (X = n) = pq n−1 , n ∈ {1, 2, } Hence the entropy of X is H(X) = − = − ∞ pq n−1 log(pq n−1 ) n=1 ∞ n=0 ∞ pq n log p + n=0 −p log p pq log q = − 1−q p2 −p log p − q log q = p = H(p)/p bits If p = 1/2 , then H(X) = bits npq n log q Information Theory and Portfolio Theory 382 so b∗ satisfies the Kuhn-Tucker conditions (b)+(c) Putting an equal amount in each stock we get E log bT∗ X = E log m m Xi i=1 = E log Thus the growth rate is Convexity We are interested in the set of stock market densities that yield the same optimal porfolio Let Pb0 be the set of all probability densities on R m + for which b0 t is optimal Thus Pb0 = {p(x) : ln(b x)p(x)dx is maximized by b = b0 } Show that Pb0 is a convex set It may be helpful to use Theorem 16.2.2 Solution: Convexity Let f1 and f2 be two stock-market densities in the set P b0 Since both f1 and f2 are in this set, then, by definition, b is the optimal constant-rebalance portfolio when the stock market vector is drawn according to f , and it is also the optimal constantrebalance portfolio when when stock market vector is drawn according to f In order to show that the set Pb0 is convex, we need to show that any arbitrary mixture ¯ , is also in the set; that is, we must show that b is also the distribution, f = λf1 + λf optimal portfolio for f We know that W (b, f ) is linear in f So ¯ 2) W (b, f ) = W (b, λf1 + λf ¯ (b, f2 ) = λW (b, f1 ) + λW But by assumption each of the summands in the last expression is maximized when b = b0 , so the entire expression is also maximized when b = b Hence, f is in Pb0 and the set is convex Short selling Let X= (1, 2), p (1, 21 ), − p Let B = {(b1 , b2 ) : b1 + b2 = 1} Thus this set of portfolios B does not include the constraint b i ≥ (This allows short selling.) (a) Find the log optimal portfolio b ∗ (p) Information Theory and Portfolio Theory 383 (b) Relate the growth rate W ∗ (p) to the entropy rate H(p) Solution: Short selling First, some philosophy What does it mean to allow negative components in our portfolio vector? Suppose at the beginning of a trading day our current wealth is S We want to invest our wealth S according to the portfolio b If b i is positive, then we want to own bi S dollars worth of stock i But if bi is negative, then we want to owe bi S dollars worth of stock i This is what selling-short means It means we sell a stock we don’t own in exchange for cash, but then we end up owing our broker so many shares of the stock we sold Instead of owing money, we owe stock The difference is that if the stock goes down in price by the end of the trading day, then we owe less money! So selling short is equivalent to betting that the stock will go down So, this is all well and good, but it seems to me that there may be some problems First of all, why we still insist that the components sum to one? It made a lot of sense when we interpreted the components, all positive, as fractions of our wealth, but it makes less sense if we are allowed to borrow money by selling short Why not have the components sum to zero instead? Secondly, if you owe money, then it’s possible for your wealth to be negative This is bad for our model because the log of a negative value is undefined The reason we take logs in the first place is to turn a product into a sum that converges almost surely But we are only justified in taking the logs in the first place if the product is positive, which it may not be if we allow short-selling Now, having gotten all these annoying philosophical worries out of the way, we can solve the problem quite simply by viewing it just as an unconstrained calculus problem and not worrying about what it all means (a) We’ll represent an arbitrary portfolio as b = (b, − b) The quantity we’re trying to maximize is W (b) = E[log(bT X)] = E[log(bX1 + (1 − b)X2 )] = p log(b + 2(1 − b)) + (1 − p) log(b + (1 − b)) 1 = p log(b + − 2b) + (1 − p) log(b + − b) 2 1 = p log(2 − b) + (1 − p) log( + b) 2 = p log(2 − b) + (1 − p) log + (1 − p) log(1 + b) We solve for the maximum of W (b) by taking the derivative and solving for zero: Information Theory and Portfolio Theory 384 dW db −p 1−p + =0 2−b 1+b ⇒ b = − 3p = ⇒ b = (2 − 3p, 3p − 1) (b) This questions asks us to relate the growth rate W ∗ to the entropy rate H(p) of the market Evidently there is some equality or inequality we should discover, as is the case with the horse race Our intuition should tell us that low entropy rates correspond to high doubling rates and that high entropy rates correspond to low doubling rates Quite simply, the more certain we are about what the market is going to next (low entropy rate), the more money we should be able to make in it W∗ = = = = = = = ∗ W + H(p) ⇒ = ≤ W (2 − 3p) p log((2 − 3p) + 2(3p − 1)) + (1 − p) log((2 − 3p) + (3p − 1)) p log(2 − 3p + 6p − 2) + (1 − p) log(2 − 3p + p − ) 2 3 p log 3p + (1 − p) log( − p) 2 p log p + p log + (1 − p) log + (1 − p) log(1 − p) −H(p) + p log + (1 − p) log − (1 − p) log −H(p) + log − (1 − p) log log − (1 − p) log Hence, we can conclude that W ∗ + H(p) ≤ log Normalizing x Suppose we define the log optimal portfolio b ∗ to be the portfolio maximizing the relative growth rate ln bt x m dF (x1 , , xm ) m i=1 xi The virtue of the normalization m Xi , which can be viewed as the wealth associated with a uniform portfolio, is that the relative growth rate is finite, even when the growth rate ln bt xdF (x) is not This matters, for example, if X has a St Petersburg-like distribution Thus the log optimal portfolio b ∗ is defined for all distributions F , even those with infinite growth rates W ∗ (F ) Information Theory and Portfolio Theory 385 (a) Show that if b maximizes ln(bt x)dF (x) , it also maximizes 1 , m, , m ) u = (m (b) Find the log optimal portfolio b ∗ for k t ln but xx dF (x) , where k (22 +1 , 22 ), 2−(k+1) k k (22 , 22 +1 ), 2−(k+1) X= where k = 1, 2, (c) Find EX and W ∗ (d) Argue that b∗ is competitively better than any portfolio b in the sense that Pr{bt X > cb∗t X} ≤ 1c Solution: Normalizing x T (a) E[ loguTbXX ] = E[log bT X − log uT X] = E[log bT X] − E[log uT X] where the second quantity in this last expression is just a number that does not change as the portfolio b changes So any portfolio that maximizes the first quantity in the last expression maximizes the entire expression (b) Well, you can grunge out all the math here, which is messy but not difficult But you can also notice that the symmetry of the values that X can take on demands that, if there is any optimum solution, it must be at b = ( 12 , 12 ) For every value of the form (a, b) that X can take on, there is a value of the form (b, a) that X takes on with equal probability, so there is absolutely no bias in the market between allocating funds to stock vs stock k k k Normalizing X by ut x = 21 (22 +1 + 22 ) = 23 22 , we obtain ˆ= X ( 34 , 32 ), ( 32 , 34 ), with probability 2−(k+1) with probability 2−(k+1) (16.21) ˆ only takes on two values, we can sum over k and obtain Since X ˆ = X ( 34 , 23 ), ( 23 , 43 ), with probability with probability 2 (16.22) The doubling rate for a portfolio on this distbribution is W (b) = 2 log b1 + (1 − b1 ) + log b1 + (1 − b1 ) 3 3 (16.23) Differentiating and setting to zero and solving gives b = ( 12 , 21 ) (c) It is easy to calculate that E[X] = = ∞ k=1 ∞ k=1 = ∞ 22 k +1 · 22 + 22 k −k−1 k 2−(k+1) (16.24) (16.25) (16.26) Information Theory and Portfolio Theory 386 and similarly that W∗ = ∞ log k=1 2k +1 2k + 2 + log 2k 2k +1 + 2 2−(k+1) (16.27) = = ∞ 2−k log 22 k=1 ∞ (16.30) for the standard definition of W ∗ If we use the new definition, then obviously W ∗ = , since the maximizing distribution b ∗ is the uniform distribution, which is the distribution by which we are normalizing (d) The inequality can be shown by Markov’s inequality and Theorem 16.2.2 as follows bt X >c b∗t X = Pr (16.31) t ≤ ≤ E bb∗tXX c c (16.32) (16.33) and therefore no portfolio exists that almost surely beats b ∗ Also the probability that any other portfolio is more than twice the return of b ∗ is less than 12 , etc Universal portfolio We examine the first n = steps of the implementation of the universal portfolio for m = stocks Let the stock vectors for days and be x1 = (1, 12 ) , and x2 = (1, 2) Let b = (b, − b) denote a portfolio (a) Graph S2 (b) = i=1 bt xi , ≤ b ≤ (b) Calculate S2∗ = maxb S2 (b) (c) Argue that log S2 (b) is concave in b (d) Calculate the (universal) wealth Sˆ2 = S2 (b)db (e) Calculate the universal portfolio at times n = and n = : ˆ1 = b ˆ (x1 ) = b bdb 1 bS1 (b)db/ (16.28) 2−k 2k log + (16.29) log k=1 = ∞, Pr bt X > cb∗t X k S1 (b)db ˆ are unchanged if we permute the order of appearance (f) Which of S2 (b), S2∗ , Sˆ2 , b of the stock vector outcomes, i.e., if the sequence is now (1, 2), (1, 21 ) ? Information Theory and Portfolio Theory 387 Solution: Universal portfolio All integrals, unless otherwise stated are over [0, 1] (a) S2 (b) = (b/2 + 1/2)(2 − b) = + b/2 − b2 /2 (b) Maximizing over S2 (b) we have S2∗ = S2 (1/2) = 9/8 (c) S2 (b) is concave and log(·) is a monotonic increasing concave function so log S (b) is concave as well (check!) (d) Using (a) we have Sˆ2 = (1 + b/2 + b2 /2)db = 13/12 (e) Clearly ˆb1 = 1/2 , and ˆb2 (x1 ) = = bS1 (b)db/ S1 (b)db 0.5b(b + 1)db/ 0.5(b + 1)db = 5/9 (f) Only ˆb2 (x1 ) changes 10 Growth optimal Let X1 , X2 ≥ , be price relatives of two independent stocks Suppose EX1 > EX2 Do you always want some of X1 in a growth rate optimal portfolio S(b) = bX1 + ¯bX2 ? Prove or provide a counterexample Solution: Growth optimal Yes, we always want some of X1 The following is a proof by contradiction Assume that b∗ = (0, 1)t so that X1 is not active Then the KKT conditions for this choice X1 ≤ and E X of b∗ imply that E X X2 = , because by assumption stock is inactive and stock is active The second condition is obviously satisfied, so only the first condition needs to be checked Since X and X2 are independent the expectation can be rewritten as EX1 E X12 Since X2 is nonnegative, X12 is convex over the region of EX1 1 interest, so by Jensen’s inequality E X12 ≥ EX This gives that E X X2 ≥ EX2 > since EX1 > EX2 But this contradicts the KKT condition, therefore the assumption that b∗ = (0, 1)t must be wrong, and so we must want some of X Note that we never want to short sell X For any b < , we have X1 + (1 − b)) X2 X1 + (1 − b)) ≤ ln (bE X2 < ln = E ln(bX1 + (1 − b)X2 ) − E log X2 ≤ E ln (b Hence, the short selling on X1 is always worse than b = (0, 1) Alternatively, we can prove the same result directly as follows Let −∞ < b < ∞ Consider the growth rate W (b) = E ln(bX − (1 − b)X2 ) Differentiating w.r.t b , we get X1 − X W (b) = E bX1 + (1 − b)X2 Information Theory and Portfolio Theory 388 Note that W (b) is concave in b Thus W (b) is monotonically nonincreasing Since X1 − > , it is immediate that b∗ > W (b∗ ) = and W (0) = E X 11 Cost of universality In the discussion of finite horizon universal portfolios, it was shown that the loss factor due to universality is n n = Vn k=0 k k n k n−k n n−k (16.34) Evaluate Vn for n = 1, 2, Solution: Cost of universality Simple computation of the equation allows us to calculate n 10 Vn 0.125 0.197530864197531 0.251953125 0.29696 0.336076817558299 0.371099019723317 0.403074979782104 0.43267543343584 0.460358496 Vn ∞ 5.0625 3.96899224806202 3.36745689655172 2.97551020408163 2.69469857599079 2.48092799146348 2.31120124398809 2.17222014731754 12 Convex families This problem generalizes Theorem 16.2.2 We say that S is a convex family of random variables if S , S2 ∈ S implies λS1 + (1 − λ)S2 ∈ S Let S be a closed convex family of random variables Show that there is a random variable S ∗ ∈ S such that S ≤0 (16.35) E ln S∗ for all S ∈ S if and only if E S S∗ ≤1 (16.36) for all S ∈ S Solution: Convex families Define S ∗ as the random variable that maximizes E ln S over all S ∈ S Since this is a maximization of a concave function over a convex set, there is a global maximum For this value of S ∗ , we have E ln S ≤ E ln S ∗ (16.37) for all S ∈ S , and therefore for all S ∈ S E ln S ≤0 S∗ (16.38) Information Theory and Portfolio Theory 389 We need to show that for this value of S ∗ , that E S ≤1 S∗ (16.39) for all S ∈ S Let T ∈ S be defined as T = λS + (1 − λ)S ∗ = S ∗ + λ(S − S ∗ ) Then as λ → , expanding the logarithm in a Taylor series and taking only the first term, we have E ln T − E ln S ∗ = E ln S ∗ + λ(S − S ∗ ) S∗ S = λ E ∗ −1 S ≤ λ(S − S ∗ ) S∗ = E − E ln S ∗ (16.40) (16.41) (16.42) (16.43) where the last inequality follows from the fact that S ∗ maximizes the expected logarithm Therefore if S ∗ maximizes the expected logarithm over the convex set, then for every S in the set, S E ∗ ≤1 (16.44) S The other direction follows from Jensen’s inequality, since if ES/S ∗ ≤ for all S , then E ln S S ≤ ln E ∗ ≤ ln = ∗ S S (16.45) 390 Information Theory and Portfolio Theory Chapter 17 Inequalities in Information Theory Sum of positive definite matrices For any two positive definite matrices, K and K2 , show that |K1 + K2 | ≥ |K1 | Solution: Sum of positive definite matrices Let X , Y be independent random vectors with X ∼ φ K1 and Y ∼ φK2 Then X+Y ∼ φK1 +K2 and hence 21 ln(2πe)n |K1 +K2 | = h(X+Y) ≥ h(X) = 12 ln(2πe)n |K1 | , by Lemma 17.2.1 Fan’s inequality[5] for ratios of determinants For all ≤ p ≤ n , for a positive definite K = K(1, 2, , n) , show that p |K(i, p + 1, p + 2, , n)| |K| ≤ |K(p + 1, p + 2, , n)| i=1 |K(p + 1, p + 2, , n)| (17.1) Solution: Ky Fan’s inequality for the ratio of determinants We use the same idea as in Theorem 17.9.2, except that we use the conditional form of Theorem 17.1.5 |K| ln(2πe)p |K(p + 1, p + 2, , n)| = h(X1 , X2 , , Xp |Xp+1 , Xp+2 , , Xn ) ≤ p = i=1 h(Xi |Xp+1 , Xp+2 , , Xn ) |K(i, p + 1, p + 2, , n)| ln 2πe (17.2) |K(p + 1, p + 2, , n)| Convexity of determinant ratios For positive definite matrices K , K , show that 0| ln |K+K is convex in K |K| Solution: Convexity of determinant ratios The form of the expression is related to the capacity of the Gaussian channel, and hence we can use results from the concavity of mutual information to prove this result 391 Inequalities in Information Theory 392 Consider a colored noise Gaussian channel Yi = X i + Z i , (17.3) where X1 , X2 , , Xn ∼ N (0, K0 ) and Z1 , Z2 , , Zn ∼ N (0, K) , and X and Z are independent Then I(X1 , X2 , , Xn ; Y1 , Y2 , , Yn ) = h(Y1 , Y2 , , Yn ) − h(Y1 , Y2 , , Yn |X1 , X2 , (17.4) , Xn ) = h(Y1 , Y2 , , Yn ) − h(Z1 , Z2 , , Zn ) 1 = log(2πe)n |K + K0 | − log(2πe)n |K| 2 |K0 + K| log = |K| (17.5) (17.6) (17.7) Now from Theorem 2.7.2, relative entropy is a convex function of the the distributions (The theorem should be extended to the continuous case by replacing probability mass functions by densities and summations by integrations.) Thus if f λ (x, y) = λf1 (x, y) + (1 − λ)f2 (x, y) , gλ (x, y) = λg1 (x, y) + (1 − λ)g2 (x, y) , we have D(fλ (x, y)||gλ (x, y)) ≤ λD(f1 (x, y)||g1 (x, y)) + (1 − λ)D(f2 (x, y)||g2 (x, y)) (17.8) Let Z n ∼ N (0, K1 ) with probability λ and Z n ∼ N (0, K2 ) with probability − λ Let f1 (xn , y n ) be the joint distribution corresponding to Y n = X n + Z n when Z n ∼ N (0, K1 ) , and g1 (x, y) = f1 (x)f1 (y) be the corresponding product distribution Then I(X1n ; Y1n ) = D(f1 (xn , y n )||f1 (xn )f1 (y n )) = D(f1 (xn , y n )||g1 (xn , y n )) = |K0 + K1 | log |K1 | (17.9) Similarly |K0 + K2 | log |K2 | (17.10) However, the mixture distribution is not Guassian, and cannot write the same expression in terms of determinants Instead, using the fact that the Gaussian is the worst noise given the moment constraints, we have by convexity of relative entropy I(X2n ; Y2n ) = D(f1 (xn , y n )||f1 (xn )f1 (y n )) = D(f1 (xn , y n )||g1 (xn , y n )) = |K0 + Kλ | log |Kλ | ≤ I(Xλn ; Yλn ) (17.11) = D(fλ (xn , y n )||fλ (xn )fλ (y n )) (17.12) ≤ λD(f1 (x, y)||g1 (x, y)) + (1 − λ)D(f2 (x, y)||g2 (x, y))(17.13) = λI(X1n ; Y1n ) + (1 − λ)I(X2n ; Y2n ) |K0 + K1 | |K0 + K2 | + (1 − λ) log = λ log |K1 | |K2 | proving the convexity of the determinant ratio (17.14) (17.15) Inequalities in Information Theory 393 Data Processing Inequality: Let random variable X , X2 , X3 and X4 form a Markov chain X1 → X2 → X3 → X4 Show that I(X1 ; X3 ) + I(X2 ; X4 ) ≤ I(X1 ; X4 ) + I(X2 ; X3 ) (17.16) Solution: Data Processing Inequality: (repeat of Problem 4.33) X1 → X → X → X +I(X2 ; X3 ) − I(X1 ; X3 ) − I(X2 ; X4 ) I(X1 ; X4 ) (17.17) = H(X1 ) − H(X1 |X4 ) + H(X2 ) − H(X2 |X3 ) − (H(X1 ) − H(X1 |X3 )) −(H(X2 ) − H(X2 |X4 )) = H(X1 |X3 ) − H(X1 |X4 ) + H(X2 |X4 ) − H(X2 |X3 ) (17.18) (17.19) = H(X1 , X2 |X3 ) − H(X2 |X1 , X3 ) − H(X1 , X2 |X4 ) + H(X2 |X1 , X4(17.20) ) +H(X1 , X2 |X4 ) − H(X1 |X2 , X4 ) − H(X1 , X2 |X3 ) + H(X1 |X(17.21) , X3 )) = −H(X2 |X1 , X3 ) + H(X2 |X1 , X4 ) (17.22) = I(X2 ; X3 |X1 , X4 ) (17.24) − H(X2 |X1 , X4 ) − H(X2 |X1 , X3 , X4 ) (17.23) ≥ (17.25) where H(X1 |X2 , X3 ) = H(X1 |X2 , X4 ) by the Markovity of the random variables Markov chains: Let random variables X, Y, Z and W form a Markov chain so that X → Y → (Z, W ) , i.e., p(x, y, z, w) = p(x)p(y|x)p(z, w|y) Show that I(X; Z) + I(X; W ) ≤ I(X; Y ) + I(Z; W ) (17.26) Solution: Markov chains: (repeat of Problem 4.34) X → Y → (Z, W ) , hence by the data processing inequality, I(X; Y ) ≥ I(X; (Z, W )) , and hence I(X : Y ) +I(Z; W ) − I(X; Z) − I(X; W ) ≥ I(X : Z, W ) + I(Z; W ) − I(X; Z) − I(X; W ) (17.27) (17.28) = H(Z, W ) + H(X) − H(X, W, Z) + H(W ) + H(Z) − H(W, Z) −H(Z) − H(X) + H(X, Z)) − H(W ) − H(X) + H(W, X)(17.29) = −H(X, W, Z) + H(X, Z) + H(X, W ) − H(X) (17.30) = H(W |X) − H(W |X, Z) (17.31) = I(W ; Z|X) (17.32) ≥ (17.33) 394 Inequalities in Information Theory Bibliography [1] T Berger Multiterminal source coding In G Longo, editor, The Information Theory Approach to Communications Springer-Verlag, New York, 1977 [2] M Bierbaum and H.M Wallmeier A note on the capacity region of the multiple access channel IEEE Trans Inform Theory, IT-25:484, 1979 [3] I Csisz´ar and J Kăorner Information Theory: Coding Theorems for Discrete Memoryless Systems Academic Press, 1981 [4] Ky Fan On a theorem of Weyl concerning the eigenvalues of linear transformations II Proc National Acad Sci U.S., 36:31–35, 1950 [5] Ky Fan Some inequalities concerning positive-definite matrices Proc Cambridge Phil Soc., 51:414–421, 1955 [6] R.G Gallager Information Theory and Reliable Communication Wiley, New York, 1968 [7] R.G Gallager Variations on a theme by Huffman IEEE Trans Inform Theory, IT24:668–674, 1978 [8] L Lovasz On the Shannon capacity of a graph IEEE Trans Inform Theory, IT-25:1–7, 1979 [9] J.T Pinkston An application of rate-distortion theory to a converse to the coding theorem IEEE Trans Inform Theory, IT-15:66–71, 1969 [10] A R´enyi Wahrscheinlichkeitsrechnung, mit einem Anhang u ăber Informationstheorie Veb Deutscher Verlag der Wissenschaften, Berlin, 1962 [11] A.A Sardinas and G.W Patterson A necessary and sufficient condition for the unique decomposition of coded messages In IRE Convention Record, Part 8, pages 104–108, 1953 [12] C.E Shannon Communication theory of secrecy systems Bell Sys Tech Journal, 28:656–715, 1949 [13] C.E Shannon Coding theorems for a discrete source with a fidelity criterion IRE National Convention Record, Part 4, pages 142–163, 1959 395 396 BIBLIOGRAPHY [14] C.E Shannon Two-way communication channels In Proc 4th Berkeley Symp Math Stat Prob., volume 1, pages 611–644 Univ California Press, 1961 [15] J.A Storer and T.G Szymanski Data compression via textual substitution J ACM, 29(4):928–951, 1982