matrix algebra pdf

This page intentionally left blank www.TechnicalBooksPDF.com Econometric Exercises, Volume Matrix Algebra Matrix Algebra is the first volume of the Econometric Exercises Series It contains exercises relating to course material in matrix algebra that students are expected to know while enrolled in an (advanced) undergraduate or a postgraduate course in econometrics or statistics The book contains a comprehensive collection of exercises, all with full answers But the book is not just a collection of exercises; in fact, it is a textbook, though one that is organized in a completely different manner than the usual textbook The volume can be used either as a self-contained course in matrix algebra or as a supplementary text Karim Abadir has held a joint Chair since 1996 in the Departments of Mathematics and Economics at the University of York, where he has been the founder and director of various degree programs He has also taught at the American University in Cairo, the University of Oxford, and the University of Exeter He became an Extramural Fellow at CentER (Tilburg University) in 2003 Professor Abadir is a holder of two Econometric Theory Awards, and has authored many articles in top journals, including the Annals of Statistics, Econometric Theory, Econometrica, and the Journal of Physics A He is Coordinating Editor (and one of the founding editors) of the Econometrics Journal, and Associate Editor of Econometric Reviews, Econometric Theory, Journal of Financial Econometrics, and Portuguese Economic Journal Jan Magnus is Professor of Econometrics, CentER and Department of Econometrics and Operations Research, Tilburg University, the Netherlands He has also taught at the University of Amsterdam, The University of British Columbia, The London School of Economics, The University of Montreal, and The European University Institute among other places His books include Matrix Differential Calculus (with H Neudecker), Linear Structures, Methodology and Tacit Knowledge (with M S Morgan), and Econometrics: A First Course (in Russian with P K Katyshev and A A Peresetsky) Professor Magnus has written numerous articles in the leading journals, including Econometrica, The Annals of Statistics, The Journal of the American Statistical Association, Journal of Econometrics, Linear Algebra and Its Applications, and The Review of Income and Wealth He is a Fellow of the Journal of Econometrics, holder of the Econometric Theory Award, and associate editor of The Journal of Economic Methodology, Computational Statistics and Data Analysis, and the Journal of Multivariate Analysis www.TechnicalBooksPDF.com Econometric Exercises Editors: Karim M Abadir, Departments of Mathematics and Economics, University of York, UK Jan R Magnus, CentER and Department of Econometrics and Operations Research, Tilburg University, The Netherlands Peter C.B Phillips, Cowles Foundation for Research in Economics, Yale University, USA Titles in the Series (* = planned): * * * * * * * * * * * * * * Matrix Algebra (K M Abadir and J R Magnus) Statistics (K M Abadir, R D H Heijmans and J R Magnus) Econometric Models, I: Theory (P Paruolo) Econometric Models, I: Empirical Applications (A van Soest and M Verbeek) Econometric Models, II: Theory Econometric Models, II: Empirical Applications Time Series Econometrics, I Time Series Econometrics, II Microeconometrics Panel Data Bayesian Econometrics Nonlinear Models Nonparametrics and Semiparametrics Simulation-Based Econometric Methods Computational Methods Financial Econometrics Robustness Econometric Methodology www.TechnicalBooksPDF.com Matrix Algebra Karim M Abadir Departments of Mathematics and Economics, University of York, UK Jan R Magnus CentER and Department of Econometrics and Operations Research, Tilburg University, The Netherlands www.TechnicalBooksPDF.com CAMBRIDGE UNIVERSITY PRESS Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521822893 © Karim M Abadir and Jan R Magnus 2005 This publication is in copyright Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press First published in print format 2005 eBook (EBL) ISBN-13 978-0-511-34440-4 ISBN-10 0-511-34440-6 eBook (EBL) ISBN-13 ISBN-10 hardback 978-0-521-82289-3 hardback 0-521-82289-0 ISBN-13 ISBN-10 paperback 978-0-521-53746-9 paperback 0-521-53746-0 Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate www.TechnicalBooksPDF.com To my parents, and to Kouka, Ramez, Naguib, N´evine To Gideon and Hedda www.TechnicalBooksPDF.com www.TechnicalBooksPDF.com Contents List of exercises Preface to the Series Preface Vectors 1.1 1.2 Real vectors Complex vectors 11 Matrices 15 2.1 2.2 19 39 Real matrices Complex matrices Vector spaces 43 3.1 3.2 3.3 47 61 67 Complex and real vector spaces Inner-product space Hilbert space Rank, inverse, and determinant 73 4.1 4.2 4.3 75 83 87 Rank Inverse Determinant Partitioned matrices 5.1 5.2 5.3 5.4 5.5 xi xxv xxix 97 Basic results and multiplication relations Inverses Determinants Rank (in)equalities The sweep operator 98 103 109 119 126 Systems of equations 131 6.1 6.2 132 137 Elementary matrices Echelon matrices www.TechnicalBooksPDF.com viii Contents 6.3 6.4 6.5 Gaussian elimination Homogeneous equations Nonhomogeneous equations Eigenvalues, eigenvectors, and factorizations 155 7.1 7.2 7.3 7.4 7.5 7.6 158 175 182 187 192 201 Eigenvalues and eigenvectors Symmetric matrices Some results for triangular matrices Schur’s decomposition theorem and its consequences Jordan’s decomposition theorem Jordan chains and generalized eigenvectors Positive (semi)definite and idempotent matrices 209 8.1 8.2 8.3 211 228 231 Positive (semi)definite matrices Partitioning and positive (semi)definite matrices Idempotent matrices Matrix functions 243 9.1 9.2 9.3 246 255 265 Simple functions Jordan representation Matrix-polynomial representation 10 Kronecker product, vec-operator, and Moore-Penrose inverse 10.1 10.2 10.3 10.4 10.5 11 12 13 143 148 151 The Kronecker product The vec-operator The Moore-Penrose inverse Linear vector and matrix equations The generalized inverse 273 274 281 284 292 295 Patterned matrices: commutation- and duplication matrix 299 11.1 11.2 11.3 11.4 300 307 311 318 The commutation matrix The symmetrizer matrix The vech-operator and the duplication matrix Linear structures Matrix inequalities 321 12.1 12.2 12.3 12.4 322 325 341 343 Cauchy-Schwarz type inequalities Positive (semi)definite matrix inequalities Inequalities derived from the Schur complement Inequalities concerning eigenvalues Matrix calculus 351 13.1 13.2 13.3 13.4 13.5 13.6 13.7 355 356 360 361 364 368 369 Basic properties of differentials Scalar functions Vector functions Matrix functions The inverse Exponential and logarithm The determinant www.TechnicalBooksPDF.com 420 Appendix B: Notation Functions We denote functions by: f :S→T f , g, ϕ, ψ, ϑ f, g F, G g ◦ f, G ◦ F g∗f function defined on S with values in T scalar-valued function vector-valued function matrix-valued function composite function ∞ convolution (g ∗ f )(x) = −∞ g(y)f (x − y) dy Two special functions are the gamma (generalized factorial) function, Γ (x), satisfying Γ (x + 1) = xΓ (x), and the beta function B(x, y) := Γ (x)Γ (y)/Γ (x + y) Derivatives and differentials The treatment of lowercase single-letter constants is somewhat controversial For example, the base of natural logarithms e and the imaginary unit i are often written as e and i The same applies to operators (such as the derivative operator d — often written as d) We recommend the use of i, e, and d, in order to avoid potential confusion with variables (such as the index i in i = 1, , n or the distance d(·, ·)) Thus, for differentials and derivatives, we write: d dn Dj ϕ(x) Dj fi (x) D2kj ϕ(x) D2kj fi (x) ϕ(n) (x) Dϕ(x), ∂ϕ(x)/∂x Df (x), ∂f (x)/∂x DF (X) ∂ vec F (X)/∂(vec X) ∇ϕ, ∇f , ∇F Hϕ(x), ∂ ϕ(x)/∂x∂x [f (x)]ba , f (x)|ba differential n-th order differential partial derivative, ∂ϕ(x)/∂xj partial derivative, ∂fi (x)/∂xj second-order partial derivative, ∂Dj ϕ(x)/∂xk second-order partial derivative, ∂Dj fi (x)/∂xk n-th order derivative of ϕ(x) derivative of ϕ(x) derivative (Jacobian matrix) of f (x) derivative (Jacobian matrix) of F (X) derivative of F (X), alternative notation gradient (transpose of derivative) second derivative (Hessian matrix) of ϕ(x) f (b) − f (a) Instead of ϕ(1) (x) and ϕ(2) (x), we may write the more common ϕ (x) and ϕ (x), but otherwise we prefer to reserve the prime for matrix transposes rather than derivatives To emphasize the difference between transpose and derivative, we write f (x) for the derivative of f and f (x) for the transpose Other mathematical symbols Various other symbols in common use are: i e, exp log imaginary unit exponential natural logarithm B.2 Mathematical symbols, functions, and operators loga ! ν j δij sgn(x) x , int(x) 1K |x| x∗ Re(x) Im(x) arg(x) 421 logarithm to the base a factorial binomial coefficient Kronecker delta: equals if i = j, otherwise sign of x integer part of x, that is, largest integer ≤ x indicator function (1, not I): equals if condition K is satisfied, otherwise absolute value (modulus) of scalar x ∈ C complex conjugate of scalar x ∈ C real part of x imaginary part of x argument of x Statistical symbols We not use many statistical symbols in this volume The ones we use are: ∼ Pr E(·) var(·) (θ) H(θ) I(θ) N(µ, Ω), Nm (µ, Ω) Wm (n, V , M M ) Wm (n, V ) is distributed as probability expectation variance log-likelihood function Hessian matrix (Fisher) Information matrix m-dimensional normal distribution Wishart distribution central Wishart distribution (M = O) www.TechnicalBooksPDF.com Bibliography Abadir, K M (1999) An introduction to hypergeometric functions for economists, Econometric Reviews, 18, 287–330 Abadir, K M., R D H Heijmans, and J R Magnus (2006) Statistics, Econometric Exercises Series, Volume 2, Cambridge University Press, New York Abadir, K M and J R Magnus (2002) Notation in econometrics: a proposal for a standard, Econometrics Journal, 5, 76–90 Aigner, M and G M Ziegler (1999) Proofs from the Book, 2nd corrected printing, Springer-Verlag, Berlin Ayres, F Jr (1962) Matrices, Schaum’s Outline Series, McGraw-Hill, New York Beaton, A E (1964) The Use of Special Matrix Operators in Statistical Calculus, Ed.D thesis, Harvard University Reprinted as Educational Testing Service Research Bulletin, 64–51, Princeton Beckenbach, E F and R Bellman (1961) Inequalities, Springer-Verlag, Berlin Bellman, R (1970) Introduction to Matrix Analysis, 2nd edition, McGraw-Hill, New York Binmore, K G (1980) Logic, Sets and Numbers, Cambridge University Press, Cambridge Binmore, K G (1981) Topological Ideas, Cambridge University Press, Cambridge Bretscher, O (1997) Linear Algebra with Applications, Prentice-Hall, Upper Saddle River, New Jersey Bronson, R (1989) Theory and Problems of Matrix Operations, Schaum’s Outline Series, McGraw-Hill, New York Browne, M W (1974) Generalized least squares estimators in the analysis of covariance structures, South African Statistical Journal, 8, 1–24 Reprinted in: Latent Variables in Socioeconomic Models (eds D J Aigner and A S Goldberger), North-Holland, Amsterdam, 205–226 423 424 Bibliography Dempster, A P (1969) Elements of Continuous Multivariate Analysis, Addison-Wesley, Reading Driscoll, M F and W R Gundberg (1986) A history of the development of Craig’s theorem, The American Statistician, 40, 65–70 Gantmacher, F R (1959) The Theory of Matrices, volumes, Chelsea, New York Graybill, F A and G Marsaglia (1957) Idempotent matrices and quadratic forms in the general linear hypothesis, Annals of Mathematical Statistics, 28, 678–686 Hadley, G (1961) Linear Algebra, Addison-Wesley, Reading, Mass Halmos, P R (1974) Finite-Dimensional Vector Spaces, Springer-Verlag, New York Hardy, G H., J E Littlewood and G P´olya (1952) Inequalities, 2nd edition, Cambridge University Press, Cambridge Harville, D A (2001) Matrix Algebra: Exercises and Solutions, Springer-Verlag, New York Hedayat, A and W D Wallis (1978) Hadamard matrices and their applications, Annals of Statistics, 6, 1184–1238 Horn, R A and C R Johnson (1985) Matrix Analysis, Cambridge University Press, New York Horn, R A and C R Johnson (1991) Topics in Matrix Analysis, Cambridge University Press, Cambridge Kay, D C (1988) Tensor Calculus, Schaum’s Outline Series, McGraw-Hill, New York Koopmans, T C., H Rubin and R B Leipnik (1950) Measuring the equation systems of dynamic economics, in: Statistical Inference in Dynamic Economic Models (ed T C Koopmans), Cowles Foundation for Research in Economics, Monograph 10, John Wiley, New York, Chapter Kostrikin, A I and Yu I Manin (1981) Linear Algebra and Geometry, Gordon and Breach Science Publishers, New York Kreyszig, E (1978) Introductory Functional Analysis with Applications, John Wiley, New York Lang, S (1995) Differential and Riemannian Manifolds, 3rd edition, Springer-Verlag, Berlin Lipschutz, S (1989) 3000 Solved Problems in Linear Algebra, Schaum’s Solved Problems Series, McGraw-Hill, New York Lipschutz, S and M Lipson (2001) Theory and Problems of Linear Algebra, 3rd edition, Schaum’s Outline Series, McGraw-Hill, New York Liu, S and W Polasek (1995) An equivalence relation for two symmetric idempotent matrices, Econometric Theory, 11, 638 Solution (by H Neudecker) in: Econometric Theory, 12, 590 MacDuffee, C C (1946) The Theory of Matrices, Chelsea, New York Magnus, J R (1988) Linear Structures, Charles Griffin & Company, London and Oxford University Press, New York Magnus, J R (1990) On the fundamental bordered matrix of linear estimation, in: Advanced Lectures In Quantitative Economics (ed F van der Ploeg), Academic Press, London, 583–604 Bibliography 425 Magnus, J R and H Neudecker (1979) The commutation matrix: some properties and applications, The Annals of Statistics, 7, 381–394 Magnus, J R and H Neudecker (1980) The elimination matrix: some lemmas and applications, SIAM Journal on Algebraic and Discrete Methods, 1, 422–449 Magnus, J R and H Neudecker (1999) Matrix Differential Calculus with Applications in Statistics and Econometrics, revised edition, Wiley, Chichester/New York McCullagh, P (1987) Tensor Methods in Statistics, Chapman and Hall, London Mirsky, L (1955) An Introduction to Linear Algebra, Oxford University Press, London Moore, E H (1920) On the reciprocal of the general algebraic matrix (Abstract), Bulletin of the American Mathematical Society, 26, 394–395 Moore, E H (1935) General Analysis, Memoirs of the American Philosophical Society, Volume I, American Philosophical Society, Philadelphia Muirhead, R J (1982) Aspects of Multivariate Statistical Theory, Wiley, New York Ortega, J M (1987) Matrix Theory: A Second Course, Plenum Press, New York Penrose, R (1955) A generalized inverse for matrices, Proceedings of the Cambridge Philosophical Society, 51, 406–413 Penrose, R (1956) On best approximate solutions of linear matrix equations, Proceedings of the Cambridge Philosophical Society, 52, 17–19 Pollock, D S G (1979) The Algebra of Econometrics, Wiley, New York Prasolov, V V (1994) Problems and Theorems in Linear Algebra, Translations of Mathematical Monographs, Vol 134, American Mathematical Society, Providence, Rhode Island Proskuryakov, I V (1978) Problems in Linear Algebra, Mir Publishers, Moscow Rao, C R and S K Mitra (1971) Generalized Inverse of Matrices and Its Applications, Wiley, New York Roth, W E (1934) On direct product matrices, Bulletin of the American Mathematical Society, 40, 461–468 Rudin, W (1976) Principles of Mathematical Analysis, 3rd edition, McGraw-Hill, New York Shilov, G E (1974) Elementary Functional Analysis, MIT Press, Cambridge, Mass Spiegel, M R (1971) Calculus of Finite Differences and Difference Equations, Schaum’s Outline Series, McGraw-Hill, New York Whittaker, E T and G N Watson (1996) A Course of Modern Analysis, 4th edition, Cambridge University Press, Cambridge Zhan, X (2002) Matrix Inequalities, Springer-Verlag, Berlin Zhang, F (1996) Linear Algebra: Challenging Problems for Students, The Johns Hopkins University Press, Baltimore Zhang, F (1999) Matrix Theory: Basic Results and Techniques, Springer-Verlag, New York Index Addition matrix, 16, 20 vector, 2, Adjoint, 75, 95, 96 Aitken’s theorem, 384 Almost sure, 50 Analytic continuation, 270 Angle, 3, 10, 23 Asymptotic equivalence, 407 Basis, 45, 56, 57 dimension, 59, 60 existence, 58 extension to, 58 reduction to, 58 Bayesian sensitivity, 377 Bilinear form, 356, 357 Binomial coefficient, 405 Block block-diagonal, 97 column, 97 row, 97 Bolzano-Weierstrass theorem, 229, 402 Cauchy, 13, 96 Cauchy-Schwarz, see Inequality, Cauchy-Schwarz criterion, 403 inequality, 324 rule of invariance, 353 not valid for second differential, 353 sequence, 47, 67, 402 Cayley’s transform, 264 Cayley-Hamilton theorem, 190, 201, 271 Characteristic 426 equation, 155, 159, 174, 190 polynomial, 155, 156, 160, 161, 163, 170, 181 root, see Eigenvalue Cofactor, 75, 90, 91, 370 Column block, 97 rank, 74, 77, 80 space, 73, 75–77 Commutation matrix, 300–307 as derivative of X , 363 commutation property, 301, 302 definition, 299 determinant, 305 eigenvalues, 305 explicit expression, 302, 303 is orthogonal, 300 trace, 304, 305 Completeness, 46, 47, 67, 68, 403 Completion of square, 216 Complex numbers, 4, 11–13, 18–19, 39–42, 155, 398–400 argument, 398 complex conjugate, 4, 11, 399 modulus, 4, 12, 398 polar form, 270, 398 Concavity of λn , 344 of log |A|, 334, 391 of |A|1/n , 391 Conformable, 17 Continuity argument, 96, 116, 165, 223, 229, 322, 333 Contraction, 229, 242 Index Convexity, 410 of λ1 , 344 of Lagrangian, 354, 413 sufficient condition, 354 Craig-Sakamoto lemma, 181, 208 Cramer’s rule, 146, 147, 154 Derivative, 351 identification, 352 notation, 351 with respect to symmetric matrix, 366, 367, 373 Determinant, 74, 87–96, 173 axiomatic definition, 92 definition, 74 differential, 369–373 elementary operation, 89 equality, 116, 117, 167 expansion by row (column), 90 Hessian matrix, 380, 381 inequality, 225, 325, 326, 333, 339, 349 of × matrix, 75 of × matrix, 87 of commutation matrix, 305 of conjugate transpose, 88 of diagonal matrix, 90 of elementary matrix, 136 of inverse, 95 of orthogonal matrix, 95 of partitioned matrix, 109–118 of positive (semi)definite matrix, 215, 216 of product, 94, 112 of skew-Hermitian matrix, 255 of transpose, 88 of triangular matrix, 92 of tridiagonal matrix, 92 product of eigenvalues, 167, 189 Vandermonde, 93, 148 zero, 89, 94 Deviations from the mean, 239, 242 dg-function, 17 diag-function, 17 Differential, 351–395 first, 352, 355–373 second, 353 with respect to symmetric matrix, 366, 367, 373 Dimension, 45 finite, 45, 55, 56, 59, 60 infinite, 45, 56, 147 of Cn , 60 of column space, 76, 77 of kernel, 73, 82, 149 of orthogonal complement, 70, 73, 76, 77 427 Direct sum, 70, 97 Distance, 46, 64 Duplication matrix, 311–317 and commutation matrix, 313 and Kronecker product, 315 definition, 299 properties of D n (A ⊗ A)Dn , 317 properties of Dn Dn , 314 properties of Dn+ (A ⊗ A)Dn , 315–317 Eigenvalue algebraic multiplicity, 163 complex, 160 concavity of λn , 344 convexity of λ1 , 344 definition, 155 distinct, 157, 170, 171, 175 geometric multiplicity, 163 inequality, 343–350 monotonicity, 346 multiple, 155, 178 multiplicity, 155 of × matrix, 158 of × matrix, 159 of AB and BA, 167 of commutation matrix, 305 of diagonal matrix, 164 of Hermitian matrix, 175 of idempotent matrix, 232 of inverse, 163 of orthogonal matrix, 165, 175 of positive (semi)definite matrix, 215 of power, 163 of rank-one matrix, 172 of skew-Hermitian matrix, 255 of skew-symmetric matrix, 164, 255 of symmetric matrix, 175, 181 of transpose, 163 of triangular matrix, 164 of unitary matrix, 165 ordering, 322 product of, 167, 189 quasilinear representation, 343, 346 real, 175, 225 simple, 155, 174, 190 sum of, 168, 189 variational description, 345 zero, 164, 190 Eigenvector definition, 157 example, 162 existence, 161 428 Eigenvector (Cont.) generalized, 157, 201–208 degree, 157 left, 173 linear combination, 161 linearly independent, 170, 171, 176 normalized, 157 of symmetric matrix, 175, 176, 178, 179 orthogonal, 170, 171, 173, 175, 176 orthonormal, 187, 192 right, 157 uniqueness, 157, 161, 176 Elementary matrix, 132–137 determinant, 136 does not commute, 136 explicit expression, 133 inverse, 134 product, 135, 140 transpose, 134 operation, 89, 100, 101, 131, 133, 136 symmetric function, 156, 169 Elimination backward, 132, 145 forward, 132, 145 Gaussian, 132, 143–148 Equation, linear, 131–153 characterization, 152 consistency, 151, 152, 293, 294, 296 general solution, 294, 296 homogeneous, 79, 132, 148–150, 292 matrix, 294–295 nonhomogeneous, 132, 151–153 nontrivial solution, 148, 149 number of solutions, 148, 151 trivial solution, 148, 149 unique solution, 294, 295 vector, 79, 292–294 Equivalence, 141, 157 class, 50 Euclid, 328 Euclidean space, m-dimensional, Euler, 143, 144, 154, 398 Exponential of a matrix differential, 368 series expansion, 244, 249, 252, 256, 257, 260, 262, 265–269 Factorization as product of elementary matrices, 140 Cholesky, 210, 220, 242 Index diagonalization conditions, 171, 187, 200 of idempotent matrix, 234, 236 of normal matrix, 158, 191 of positive (semi)definite matrix, 210, 215, 219 of symmetric matrix, 158, 177, 189 of triangular matrix, 158, 186 simultaneous, 158, 174, 180, 211, 225 with distinct eigenvalues, 158, 171 Jordan’s theorem, 157, 158, 199, 245 polar decomposition, 211, 226, 398 QR, 158, 172 rank, 80, 158 Schur’s triangularization theorem, 157, 158, 187 singular-value decomposition, 211, 225, 226 using echelon matrices, 140, 158 Fibonacci sequence, 35, 42 Fischer inequality, 228, 341 min-max theorem, 346 Frobenius’s inequality, 122, 129 Fundamental theorem of algebra, 155, 401 Gauss, 154 Gauss-Markov theorem, 384 Generalized inverse, 295 and the solution of linear equations, 296 definition, 274 existence, 295 explicit expression, 295 rank, 296 Gram-Schmidt orthogonalization, 67, 98, 172 Hadamard product, 321, 340 definition, 340 positive definite, 340 Heine-Borel theorem, 402 Hermitian form, 19, 39, 210 Idempotent matrix, 18, 231–242 checks, 210 definition, 18, 210 differential, 365 eigenvalues, 232, 233 idempotent operation, 37, 210 in econometrics, 238 necessary and sufficient condition, 235 nonsymmetric, 37, 210, 232 of order two, 37 rank equals trace, 235 sum of, 236, 240–242 Index Inequality arithmetic-geometric mean, 328, 392, 393 Bergstrom, 323 Bunyakovskii, 13 Cauchy, 324 Cauchy-Schwarz, 7, 8, 13, 62, 322, 325, 330 concerning eigenvalues, 343–350 derived from Schur complement, 341–343 determinantal, 225, 325, 326, 333, 339, 349 Fischer, 228, 341 Frobenius, 122, 129 Hadamard, 337, 349 Kantorovich, 331 Minkowski, 329 Olkin, 340 rank, 78, 81, 120–122, 124, 179, 324 Schur, 305, 325, 331 Sylvester, 122, 129, 179, 180 trace, 329, 338, 348 triangle, 7, 12, 13, 38, 63 geometric interpretation, Inner product continuity, 65 in vector space, 45 induced by norm, 63, 64 of two complex vectors, 4, 12 of two real matrices, 18, 38 of two real vectors, 2, 6, 20 Inverse, 74, 83–87, 95 and rank, 83 differential, 364–367 existence, 83 Jacobian of transformation, 373, 374 of A + ab , 87, 173, 248 of A − BD −1 C, 107 of partitioned matrix, 103–109 of product, 84 of triangular matrix, 186 series expansion, 249 uniqueness, 83 Jacobian matrix, 351 of transformation, 354, 373–375 Jordan, 192–200 block, 192–194 power, 258 symmetrized, 194 chain, 157, 201–208 lemma, 195, 196 matrix, 157, 158, 197, 198, 200 number of blocks, 200 representation, 260 theorem, 199 429 Kato’s lemma, 218, 331 Kernel, 73, 149 Kronecker delta, 66 Kronecker product, 274–281 and vec-operator, 282 definition, 273 determinant, 279 differential, 355 eigenvalues, 278 eigenvectors, 279 inverse, 278 multiplication rules, 275–277 noncommutativity, 275 rank, 279 trace, 277 tranpose, 277 Lagrange function, 411 multiplier method, 411–413 multipliers, 411 matrix of, 354, 413 symmetric matrix of, 354 Laplace expansion, 74 Leading element, 132, 138 Least squares and best linear unbiased estimation, 382–387 constrained (CLS), 383 estimation of σ , 386 generalized (GLS), 339, 342, 383 multicollinearity, 385 ordinary (OLS), 339, 342, 382 residuals, 376 sensitivity analysis, 375–377 Leibniz, 154 Length, 46, 62, 212 l’Hˆopital’s rule, 408 Line, in the plane, 327 line segment, Linear combination, 44, 52 dependence, 44, 53, 54 conditions for, 53 in triangular matrix, 53 difference equation, 36, 409–410 equation, see Equation, linear form, 18, 19 independence, 44 and span, 59 of powers, 147 space, see Vector space structure, 300, 318–320, 374 430 Logarithm of a matrix differential, 368–369 series expansion, 244, 250, 253 Matrix augmented, 145, 151 block-diagonal, 97 cofactor, 75, 91, 187 commutation, see Commutation matrix commuting, 17, 23, 24, 29, 34, 101, 174 companion, 173 complex, 16, 18–19, 39–42 conformable, 17 conjugate transpose, 18, 39 definition, 15 diagonal, 17, 28, 29, 337 duplication, see Duplication matrix echelon, 132, 137–143 definition, 132 factorization, 140 finding inverse, 142 rank, 132, 137, 141 reduction of, 139 reduction to, 137 element, 15 elementary, see Elementary, matrix equality, 16, 19 equicorrelation, 241 function, 243–271 determinant, 262 of diagonal matrix, 246, 247 of idempotent matrix, 248 of nilpotent matrix, 247 of symmetric matrix, 255 trace, 262 generalized inverse, see Generalized inverse Gramian, bordered, 230–231, 242 Hermitian, 18, 39, 175 diagonal elements, 39 Hessian, 378–382 identification, 353, 378, 379 in maximum likelihood estimation, 390 of composite function, 353 symmetry, 353 idempotent, see Idempotent matrix identity, 17, 21, 28 indefinite, 209, 219 inverse, see Inverse invertible, see Matrix, nonsingular Jacobian, 351 see also Derivative Jordan, 157, 158, 197, 198, 200 Moore-Penrose inverse, see Moore-Penrose inverse Index negative (semi)definite, 209 see also Positive (semi)definite matrix nilpotent, 23, 183, 192 index, 23 nonsingular, 74, 83, 140 normal, 18, 19, 33, 41, 171, 182, 191, 192 notation, 15 null, 16, 20 order, 15, 17, 19 orthogonal × example, 31, 95, 254 definition, 18 determinant, 95 eigenvalues, 175 preserves length, 212 properties, 86 real versus complex, 19, 85, 166 representation, 31, 254, 263 partitioned, see Partitioned matrix permutation, 18, 32 is orthogonal, 33 positive, 243 positive (semi)definite, see Positive (semi)definite matrix power, 18, 34, 35, 217 and difference equations, 36 Fibonacci sequence, 35 noninteger, 261 product, 16, 21, 22, 25 different from scalar multiplication, 22 real, 16 scalar, 17 scalar multiplication, 16 semi-orthogonal, 84 singular, 74, 149, 227 skew-Hermitian, 18, 40 determinant, 255 diagonal elements, 40 eigenvalues, 255 skew-symmetric, 30 definition, 18 diagonal elements, 30 eigenvalues, 164, 255 representation, 255 skew-symmetrizer, see Skew-symmetrizer matrix square, 17 square root, see Positive (semi)definite matrix, square root submatrix definition, 15 in partitioned matrix, 26 leading principal, 156, 168, 337 Index Matrix (Cont.) principal, 156, 222 rank, 82 sum of matrices, 16, 20 symmetric, 17, 29, 51, 157, 171, 175–182 definition, 17 power, 35 real eigenvalues, 175 real eigenvectors, 175 real versus complex, 19, 175, 191 special properties, 157 symmetrizer, see Symmetrizer matrix transpose, 15, 20, 26 triangular, 17, 51, 157, 182–187 inverse, 186 linearly independent columns, 53 lower, 17 power, 35 product, 29, 184 strictly, 17, 183, 195 unit, 17, 186 upper, 17 tridiagonal, 92 unipotent, 101 unitary, 18, 40 representation, 264 Vandermonde, 93, 96, 147, 148 Maximum likelihood estimation, 387–391 Hessian matrix, 390 information matrix, 390 log-likelihood, 387 treatment of positive definiteness, 389 treatment of symmetry, 388, 389 Minimum global, 354 local, 354 under constraints, 354, 410–413 Minor, 370 leading principal, 156, 223 principal, 156, 223 sum of, 156 Modulus, see Complex numbers, modulus Moore-Penrose inverse, 284–292 and least squares, 384 and the solution of linear equations, 292–295 definition, 274 existence, 284 of positive semidefinite matrix, 290 of symmetric matrix, 289 rank, 286 uniqueness, 285 Multiplication scalar, 2, 5, 16 in vector space, 43 431 Norm general definition, 63 in vector space, 46, 62 induced by inner product, 63 of complex vector, 4, 12 of real matrix, 18, 38 of real vector, 2, 7, Normal distribution, 377, 386–391 and symmetrizer matrix, 309–311 Notation, 415–421 Null space, see Kernel O, o, oh notation, 407, 419 Orthogonal complement, 46, 66, 70, 73, 76, 237 matrix, see Matrix, orthogonal projection, 68 set, 46 subspace, 46, 66, 73 vectors, 3, 8–10, 46 Orthonormal basis, 67 set, 46, 66 vectors, 3, 66 Parallelogram equality, 62 Partitioned matrix, 26, 97–129 3-by-3 block matrix, 107, 118, 125 bordered, 108, 118, 125 commuting, 101 determinant, 109–118 diagonal, 100 inverse, 103–109 positive (semi)definite, 228–231 determinant, 114, 335–336 inverse, 107 necessary and sufficient conditions, 228, 229 power, 108 product, 98 rank, 119–125 sum, 98 symmetric, 100 trace, 100 transpose, 99 triangular, 100 Permutation matrix, see Matrix, permutation of integers, 74, 89 Pivot, 98, 132, 145 Poincar´e’s separation theorem, 347–348 Polynomial, 56, 93, 96, 147, 168, 169, 400–401 characteristic, 155, 156, 160, 161, 163, 170, 181 432 Polynomial (Cont.) matrix, 243–271 order of, 243 representation, 265–270 monic, 401 Positive (semi)definite matrix, 211–231 checks, 210 definite versus semidefinite, 210 definition, 209 determinant, 215, 216 diagonal elements, 213 eigenvalues, 215, 222 inequality, 325–340 inverse, 216 matrix quadratic form, 221–222 of rank two, 218 partitioned, 228–231 power, 217, 332 principal minors criterion, 223, 224 principal submatrices, 222 quadratic form, 209, 211 square root, 220, 221, 227 uniqueness, 220 trace, 215, 216 transforming symmetric matrix into, 218 upper bound for elements, 213, 323 variance matrix, 56 versus negative (semi)definite, 209, 213 Positivity, treatment of (in optimization problems), 386, 389 Postmultiplication, 17 Power sum, 156, 169 Premultiplication, 17 Projection, 239 oblique, 210 orthogonal, 68, 210 theorem, 68, 69 Proof by contradiction, 398 by contrapositive, 397 by deduction, 398 by induction, 398 Pythagoras, 9, 65, 69, 70 Quadratic form, 18, 211, 212 differential, 356, 357, 359, 363 Hessian matrix, 378, 381 Quasilinearization, 322, 324, 328, 329, 343, 344 Rank, 74, 75 and zero determinant, 94 equality, 81, 82, 85, 123, 124 full column rank, 74 Index full row rank, 74 inequality, 78, 81, 120–122, 124, 179, 324 matrix of rank one, 80, 172 matrix of rank two, 218 number of nonzero eigenvalues, 165, 179, 190 of diagonal matrix, 79 of partitioned matrix, 119–125 of product, 82 of submatrix, 82 of triangular matrix, 79 theorem, 77 Rayleigh quotient, 181, 343 Reflection, 32, 95 Rotation, 23, 32, 95, 254 Row block, 97 rank, 74, 77, 80 Scalar, Scalar product, see Inner product Scaling, 24 Schur complement, 102, 106, 228, 322 inequality derived from, 341–343 notation for, 102 decomposition theorem, 157, 158, 187 inequality, 305, 325, 331 Schwarz, 13 inequality, see Inequality, Cauchy-Schwarz Sensitivity analysis, 375–378 Bayesian, 377 least squares, 375–377 Series expansion, 401–409 absolutely convergent, 243, 260, 403 binomial alternative expansion, 246 definition, 244, 405 radius of convergence, 244, 260, 405 with two matrices, 251 conditionally convergent, 245, 403 definition, 243 exponential function as limit of binomial, 249 definition, 244, 398 Jordan representation, 257 multiplicative property, 252, 256 nonsingularity, 262 polynomial representation, 265–269 radius of convergence, 244, 260, 404 inverse, 249 Jordan representation, 255–264 logarithmic function additive property, 253 Index Series expansion (Cont.) explicit expression, 250 explicit representation, 244, 403, 405 implicit definition, 244, 405 radius of convergence, 244, 403 matrix-polynomial representation, 265–270 nonconvergent, 403 not unique, 243, 246, 261 power, 261 radius of convergence, 244, 403 summable, 248, 403–405, 414 Series representation, see Series expansion Set compact, 402 of real numbers, Shift backward, 193 forward, 193 Similarity, 157, 185 and eigenvalues, 166 Singular value, 226 Skew-symmetrizer matrix, 305 definition, 305 idempotent, 305 orthogonal to symmetrizer, 307 Space complex space, Euclidean space, vector space, see Vector space Span, 44, 54, 55, 81 and linear independence, 59 Spectral radius, 243, 369 Spectral theorem, see Factorization, diagonalization Stirling’s series, 407 Subspace, 44, 50, 51 closed, 68 dimension, 60, 61 intersection, 44, 52 of R2 , 50 of R3 , 51 sum, 44, 52, 61 union, 44, 52 Sweep operator, 98, 126–129 calculates inverse, 127 pivot, 98 solves linear equation, 128 Sylvester, 327, 350 law of nullity, 122, 129, 179, 180 Symmetrizer matrix, 307–311 and Kronecker product, 307 and normal distribution, 309–311 and Wishart distribution, 310 definition, 299 433 idempotent, 307 orthogonal to skew-symmetrizer, 307 rank, 307 Trace, 18, 30 and vec-operator, 283 differential, 357, 358 Hessian matrix, 380 inequality, 329, 338, 348 linear operator, 18, 30 of A A, 31, 38, 214, 322 of commutation matrix, 304, 305 of conjugate transpose, 39 of matrix product, 18, 31 of positive (semi)definite matrix, 215, 216 of power, 168, 169, 189 of transpose, 18, 30 sum of eigenvalues, 168, 189 Transposition, 74, 89, 96 Unit circle, 244, 400 Vec-operator, 281–284 and Kronecker product, 282 and trace, 283 definition, 273 differential, 355 linearity, 281 notation, 273 Vech-operator, 311, 312 definition, 299 Vector, 1–13 as a point in Rm , as an arrow, collinear, 2, 7, 13, 20 norm of, column vector, component of, elementary, see Vector, unit equality, 1, inequality, length, normal to plane, 65 normalized, 3, 8, 32 notation, null vector, 1, 5, 47 uniqueness, 5, 47 order, 1, orthogonal, 3, 8–10, 46, 65 orthonormal, 3, 9, 32 row vector, 16 scalar multiplication, 2, sum of vectors, 2, 434 Vector (Cont.) sum vector, 1, 10, 31 unit, Vector analysis algebraic, geometric, 1, Vector space, 43–71 addition, 43, 47 axioms, 43–44 complex, 43 Hilbert space, 44, 46, 47, 67–71 completeness, 47 inner-product space, 44, 45, 61–67 l2 -space, 49, 56, 61, 71 L2 -space, 50, 56 Index L†2 -space, 49, 56, 68, 71 real, 43, 48 representation in terms of basis, 58 scalar multiplication, 43, 47 vector in, 43 Weierstrass, 92, 96 Wishart distribution central, 310 definition, 310 noncentrality matrix, 310 variance, 310, 317 Zorn’s lemma, 71 ... www.TechnicalBooksPDF.com Econometric Exercises, Volume Matrix Algebra Matrix Algebra is the first volume of the Econometric Exercises Series It contains exercises relating to course material in matrix algebra. .. of a Hermitian matrix, a unitary matrix, a skew-Hermitian matrix, and a Hermitian form specialize in the real case to a symmetric matrix, an orthogonal matrix, a skew-symmetric matrix, and a quadratic... × n matrix A, then the resulting m1 × n1 matrix is called a submatrix of A For example, if A= , then both and are submatrices of A The transpose of an m × n matrix A = (aij ) is the n × m matrix