Linear algebra and its applications

Linear Algebra and Its Applications Fourth Edition Gilbert Strang x  y z Ax  b y Ay  b b 0 z Az  Contents Preface iv 1 13 21 36 50 66 72 77 77 86 103 115 129 140 154 159 159 171 180 195 211 221 Matrices and Gaussian Elimination 1.1 Introduction 1.2 The Geometry of Linear Equations 1.3 An Example of Gaussian Elimination 1.4 Matrix Notation and Matrix Multiplication 1.5 Triangular Factors and Row Exchanges 1.6 Inverses and Transposes 1.7 Special Matrices and Applications Review Exercises Vector Spaces 2.1 Vector Spaces and Subspaces 2.2 Solving Ax = and Ax = b 2.3 Linear Independence, Basis, and Dimension 2.4 The Four Fundamental Subspaces 2.5 Graphs and Networks 2.6 Linear Transformations Review Exercises Orthogonality 3.1 Orthogonal Vectors and Subspaces 3.2 Cosines and Projections onto Lines 3.3 Projections and Least Squares 3.4 Orthogonal Bases and Gram-Schmidt 3.5 The Fast Fourier Transform Review Exercises i CONTENTS ii Determinants 4.1 Introduction 4.2 Properties of the Determinant 4.3 Formulas for the Determinant 4.4 Applications of Determinants Review Exercises Eigenvalues and Eigenvectors 5.1 Introduction 5.2 Diagonalization of a Matrix 5.3 Difference Equations and Powers Ak 5.4 Differential Equations and eAt 5.5 Complex Matrices 5.6 Similarity Transformations Review Exercises Positive Definite Matrices 6.1 Minima, Maxima, and Saddle Points 6.2 Tests for Positive Definiteness 6.3 Singular Value Decomposition 6.4 Minimum Principles 6.5 The Finite Element Method Computations with Matrices 7.1 Introduction 7.2 Matrix Norm and Condition Number 7.3 Computation of Eigenvalues 7.4 Iterative Methods for Ax = b Linear Programming and Game Theory 8.1 Linear Inequalities 8.2 The Simplex Method 8.3 The Dual Problem 8.4 Network Models 8.5 Game Theory A Intersection, Sum, and Product of Spaces A.1 The Intersection of Two Vector Spaces A.2 The Sum of Two Vector Spaces A.3 The Cartesian Product of Two Vector Spaces A.4 The Tensor Product of Two Vector Spaces A.5 The Kronecker Product A ⊗ B of Two Matrices 225 225 227 236 247 258 260 260 273 283 296 312 325 341 345 345 352 367 376 384 390 390 391 399 407 417 417 422 434 444 451 459 459 460 461 461 462 CONTENTS iii B The Jordan Form 466 C Matrix Factorizations 473 D Glossary: A Dictionary for Linear Algebra 475 E MATLAB Teaching Codes 484 F Linear Algebra in a Nutshell 486 Ax = b C(AT ) C(A) AT y = c dim r dim r Row Space Column Space all AT y Rn all Ax Rm AT y = 0 Ax = N (A) Null Space Left Null Space dim n − r     N (AT ) dim m − r Preface Revising this textbook has been a special challenge, for a very nice reason So many people have read this book, and taught from it, and even loved it The spirit of the book could never change This text was written to help our teaching of linear algebra keep up with the enormous importance of this subject—which just continues to grow One step was certainly possible and desirable—to add new problems Teaching for all these years required hundreds of new exam questions (especially with quizzes going onto the web) I think you will approve of the extended choice of problems The questions are still a mixture of explain and compute—the two complementary approaches to learning this beautiful subject I personally believe that many more people need linear algebra than calculus Isaac Newton might not agree! But he isn’t teaching mathematics in the 21st century (and maybe he wasn’t a great teacher, but we will give him the benefit of the doubt) Certainly the laws of physics are well expressed by differential equations Newton needed calculus—quite right But the scope of science and engineering and management (and life) is now so much wider, and linear algebra has moved into a central place May I say a little more, because many universities have not yet adjusted the balance toward linear algebra Working with curved lines and curved surfaces, the first step is always to linearize Replace the curve by its tangent line, fit the surface by a plane, and the problem becomes linear The power of this subject comes when you have ten variables, or 1000 variables, instead of two You might think I am exaggerating to use the word “beautiful” for a basic course in mathematics Not at all This subject begins with two vectors v and w, pointing in different directions The key step is to take their linear combinations We multiply to get 3v and 4w, and we add to get the particular combination 3v + 4w That new vector is in the same plane as v and w When we take all combinations, we are filling in the whole plane If I draw v and w on this page, their combinations cv + dw fill the page (and beyond), but they don’t go up from the page In the language of linear equations, I can solve cv + dw = b exactly when the vector b lies in the same plane as v and w iv S o l utio n s to S e l e cted Exe rcises The cost to be minimized is 1000x + 2000y + 3000z + 1500u + 3000v + 37 00w The amounts x , y, Z to Chicago and u , v, w to New England satisfy x + u ,000,000; y + v == ,000,000; z + w == ,000,000; x + y + z == 800,000; u + v + w == 2,200,000 Problem S et 8e26 page 391 At present X4 11 and Xs are in the basis, and the cost is zero The entering variable should be X3 , to reduce the cost The leaving variable should be xs, since 2/ is less than 4/ With X3 and X4 in the basis, the constraints give X3 == 2, X4 = 2, and the cost is now Xl + X2 - X3 == -2 The "reduced costs" are r == [ 1], so change is not good and the corner is optimal At P , r = [-5 3]; then at Q , r == [� � J ; R is optimal because r > 00 For a maximum problem the stopping test becomes r < O If this fails, and the ith component is the largest, then that column of N enters the basis; the rule 8C for the vector leaving the basis is the same B E == B [· v 0] = [ u ], since B v = u So E is the correct matrix If Ax � 0, then Px = x - A T (AA T )- l Ax == x == == - Problem S et 8�ju page 399 Maximize '4 Yl + l Y2 , with Yl > 0, Y2 > 0, 2Yl + Y2 < , x i = , x i == �, the dual has yi == yi == , cost = The dual maximizes yb, with �, � y > c Therefore x 11 Y2 b and y ::S ; the primal has c are feasible , and give the same value c b for the cost in the primal and dual; by SF they must be optimal b ) and y * == (0, C2 , " ' " cn ) lf b l < O, then the optimal x * is changed to (O, b , b == [0 l]T and c == [- 0] Since cx = = yb, x and y are optimal by SF x* == [1 O] T , y * [1 0], with y * b == cx* The second inequalities in both Ax* > b and y* A < c are strict, so the second components of y * and x* are zero (a) x i = 0, xi = , �: == 0, cT X == (b) It is the first quadrant with the tetrahedron in the comer cut off (c) Maximize Yl , subject to Yl > 0, Yl < 5, ' < < Y l , Yl 4; Y l* = = , 471 := ' 13 Here c = [1 1 ] with A == n := = [� � � ] No constraint x > so the dual will have equality yA == c (or A T y == cT) That gives Yl = and Yl == and Y2 == and no feasible solution So the primal must have 00 as maximum: Xl = -N and X2 = 2N and X3 = give Cost = Xl + X2 + X == N (arbitrarily large) 15 17 0 -1 0 -1 0 The columns of 0 or - -1 o 0 -1 0 -1 Take y == [1 - 1] ; then yA > 0, yb < O ns to S e l e cted Exe r c i s e s Problem Set 8�4, page 406 The maximal flow is , with the minimal cut separating node from the other nodes Increasing the capacity of pipes from node to node or node to node will produce the largest increase in the maximal flow The maximal flow increases from to Assign capacities = to all edges The maximum number of disjoint paths from s to t then equals the maximum flow The minimum number of edges whose removal disconnects s from t is the minimum cut Then max flow = cut Rows , 4, and violate Hall's condition; the by submatrix coming from rows , , , and columns 1, 2, has + > (a) The matrix has 2n I s which cannot be covered by less than n lines because each line covers exactly two Is It takes n lines; there must be a complete matching 1 1 1 0 (b) 0 The s can be covered with four lines; five marriages are 0 1 1 1 not possible 11 If each m + marries the only acceptable man m , there is no one for #1 to marry (even though all are acceptable to #1) Algorithm gives 1-3, 3-2, 2-5, 2-4, 4-6, and algorithm gives 2-5, 4-6, 2-4, 3-2, 1-3 These are equal-length shortest spanning trees 15 (a) Rows , 3, only have I s in columns and (b) Columns , , (in rows 2, 4) (c) Zero submatrix from rows , 3, and columns , 3, (d) Rows 2, and columns 2, cover all I s 13 Problem Set B.5, page 41 - 10X I + 70(I - X I ) = 10X I - I 0( I - x I ) , or xl = � , X2 = � ; - 10 YI + 10( I - Y I ) = 70YI - 10( - YI ) , or Y I = � , Y2 = � ; average payoff y Ax = If X chooses column j, Y will choose its smallest entry aij (in row i) X will not move , since this is the largest entry in that row In Problem 2, al = was an equilibrium of this kind If we exchange the and below it, no entry has this property, and mixed strategies are required The best strategy for X combines the two lines to produce a horizontal 1ine, guaran­ teeing this height of /3 The combination is � (3y + 2(1 - y ) ) + � ( y + ( - y)) = /3, so X chooses the columns with frequencies �, 0, � For columns, we want Xi a + (1 - x I ) b = Xl e + ( - x I ) d = u , so Xl (a - b - e + d ) = d - b For rows, Y i a + (1 - YI ) e = yl b + ( - YI )d = v exchanges b and e Compare u with v : ad - be (a - b) (d - b) +b u -· x I (a - b) + b a -b - e+d a -b-e+d _ - same after b � e = v 473 S o l uti o n s to S e l e cted Exercises 11 The inner maximum is the larger of Y l and Y2 ; x concentrates on that one Subj ect to Y I + Y2 == , the minimtim of the �arger Y is � Notice A == J Ax* == [� �J T and yAx* == [� � - -lJ and y* Ax == between is y* Ax* == � 13 Value (fair game) X � YI � Xl + + � Y2 == � for all strategies of Y ; y * A == � X2 - X3 - X4 , which cannot exceed �; in chooses or , y chooses odd or even: x* Problem S et A, page 420 = y* == (b) Smallest dim (S n T) Yea) Largest dim (S n T) == when S e T (e) Smallest dim (S + T) == when S e T (d) Largest dim (S + T) - (all of R1 ) (� , �) == al l a 0 a u a 12 a a 14 a2 a22 a23 a 24 and a 22 a23 V + W and V n W contain a 23 a 0 a3 a33 a34 0 a44 a43 a44 0 dim (V + W) == and dim (V n W) == ; add to get 20 == dim V + dim W The lines through ( , , 1) and (1 , , 2) have V n W == {O} One basis for V + W is VI , V , W I ; dim (V n W) == with basis (0, , , 0) %t� The intersection of column spaces is the line through y y == [ 11] 0 == [ ] matches [A 2 B]x - == (6 , , 6) : == 0 The column spaces have dimension Their sum and intersection give + 1 1 -1 13 -2 -3 == == O 2+ -1 1- -1-1 -11 A 3D == (A I D J ® J) + ( A ID ® ) + ( ® ® A I D ) ' Problem S et 8, page 421 1 J = [� �] (A is diagonalizable) ; (2 , - , 0) ) B e t == J = � o t 0 � � 2t 0 == I+ J o = 0 0 B t since B == O Also e l t (distinct eigenvalues); J = (eigenvectors (1 , 0, 0) and == I + J t [� �] (B has A = 0, but rank ) Appendix C Matrix Factorizations lower triangular L 1s on the diagonal A = LU = upper triangular U pivots on the diagonal Requirements: No row exchanges as Gaussian elimination reduces A to U A = LDU = lower triangular L 1s on the diagonal pivot matrix D is diagonal upper triangular U 1s on the diagonal Requirements: No row exchanges The pivots in D are divided out to leave 1s in U If A is symmetric, then U is LT and A = LDLT PA = LU (permutation matrix P to avoid zeros in the pivot positions) Requirements: A is invertible Then P, L, U are invertible P does the row exchanges in advance Alternative: A = L1 P1U1 EA = R (m × m invertible E) (any A) = rref(A) Requirements: None! The reduced row echelon form R has r pivot rows and pivot columns The only nonzero in a pivot column is the unit pivot The Last m − r rows of E are a basis for the left nullspace of A and the first r columns of E −1 are a basis for the column space of A A = CCT = lower triangular matrix C transpose is upper triangular Requirements: A is symmetric and positive definite (all n pivots in D are positive) √ This Cholesky factorization has C = L D A = QR = orthonormal columns in Q upper triangular R Requirements: A has independent columns Those are orthogonalized in Q by the Gram-Schmidt process If A is square, then Q−1 = QT A = SΛS−1 = eigenvectors in S eigenvalues in Λ left eigenvectors in S−1 Requirements: A must have n linearly independent eigenvectors 474 Appendix C Matrix Factorizations A = QΛQT = orthogonal matrix Q QT is Q−1 real eigenvalue matrix Λ Requirements: A is symmetric This is the Spectral Theorem A = MJM −1 = generalized eigenvectors in M M −1 Jordan blocks in J Requirements: A is any square matrix Jordan form J has a block for each independent eigenvector of A Each block has one eigenvalue 10 A = UΣV T = orthogonal U is m × m m × n matrix Σ σ1 , , σr on diagonal orthogonal V is n × n Requirements: None This singular value decomposition (SVD) has the eigenvectors of AAT in U and of AT A in V ; σi = λi (AT A) = λi (AAT ) 11 A+ = V Σ+U T = orthogonal n×n diagonal n × m 1/σ1 , , 1/σr orthogonal m×m Requirements: None The pseudoinverse has A+ A = projection onto row space of A and AA+ = projection onto column space The shortest least-squares solution to Ax = b is x = A+ b This solves AT Ax = AT b 12 A = QH = orthogonal matrix Q symmetric positive definite matrix H Requirements: A is invertible This polar decomposition has H = AT A The factor H is semidefinite if A is singular The reverse polar decomposition A = KQ has K = AAT Both have Q = UV T from the SVD 13 A = UΛU −1 = unitary U eigenvalue matrix Λ U −1 = U H = U T Requirements: A is normal: AH A = AAH Its orthonormal (and possibly complex) eigenvectors are the columns of U Complex λ ’s unless A = AH 14 A = UTU −1 = unitary U triangular T with λ ’s on diagonal U −1 = U H Requirements: Schur triangularization of any square A There is a matrix U with orthonormal columns that makes U −1 AU triangular 15 Fn = I D I −D Fn/2 Fn/2 even-odd = one step of the FFT permutation Requirements: Fn = Fourier matrix with entries w jk where wn = 1, w = e2π i/n Then Fn F n = nI D has 1, w, w2 , on its diagonal For n = the Fast Fourier Transform has 21 n multiplications from stages of D’s Appendix D Glossary: A Dictionary for Linear Algebra Adjacency matrix of a graph Square matrix with j = when there is an edge from node i to node j; otherwise j = A = AT for an undirected graph Affine transformation T (v) = Av + v0 = linear transformation plus shift Associative Law (AB)C = A(BC) Parentheses can be removed to leave ABC Augmented matrix [A b] Ax = b is solvable when b is in the column space of A; then [A b] has the same rank as A Elimination on [A b] keeps equations correct Back substitution Upper triangular systems are solved in reverse order xn to x1 Basis for V Independent vectors v1 , , vd whose linear combinations give every v in V A vector space has many bases! Big formula for n by n determinants det(A) is a sum of n! terms, one term for each permutation P of the columns That term is the product a1α · · · anω down the diagonal of the reordered matrix, times det(P) = ±1 Block matrix A matrix can be partitioned into matrix blocks, by cuts between rows and/or between columns Block multiplication of AB is allowed if the block shapes permit (the columns of A and rows of B must be in matching blocks) Cayley-Hamilton Theorem p(λ ) = det(A − λ I) has p(A) = zero matrix Change of basis matrix M The old basis vectors v j are combinations ∑ mi j wi of the new basis vectors The coordinates of c1 v1 + · · · + cn = d1 w1 + · · · + dn wn are related by d = Mc (For n = 2, set v1 = m11 w1 + m21 w2 , v2 = m12 w1 + m22 w2 ) Characteristic equation det(A − λ I) = The n roots are the eigenvalues of A √ √ Cholesky factorization A = CCT = (L D)(L D)T for positive definite A 476 Appendix D Glossary: A Dictionary for Linear Algebra Circulant matrix C Constant diagonals wrap around as in cyclic shift S Every circulant is c0 I + c1 S + · · · + cn−1 Sn−1 Cx = convolution c ∗ x Eigenvectors in F Cofactor Ci j Remove row i and column j; multiply the determinant by (−1)i+ j Column picture of Ax = b The vector b becomes a combination of the columns of A The system is solvable only when b is in the column space C (A) Column space C (A) Space of all combinations of the columns of A Commuting matrices AB = BA If diagonalizable, they share n eigenvectors Companion matrix Put c1 , , cn in row n and put n − 1s along diagonal Then det(A − λ I) = ±(c1 + c2 λ + c3 λ + · · · ) Complete solution x = x p + xn to Ax = b Complex conjugate (Particular x p ) + (xn in nullspace) z = a − ib for any complex number z = a + ib.Then zz = |z|2 Condition number cond(A) = κ (A) = A A−1 = σmax /σmin In Ax = b, the relative change δ x / x is less than cond(A) times the relative change δ b / b Condition numbers measure the sensitivity of the output to change in the input Conjugate Gradient Method A sequence of steps to solve positive definite Ax = b by minimizing 21 xT Ax − xT b over growing Krylov subspaces Covariance matrix Σ When random variables xi have mean = average value = 0, their covariances Σi j are the averages of xi x j With means xi , the matrix Σ = mean of (x − x)(x − x)T is positive (semi)definite; it is diagonal if the xi are independent Cramer’s Rule for Ax = b B j has b replacing column j of A, and x j = |B j |/|A| Cross product u × v in R3 Vector perpendicular to u and v, length u v | sin θ | = parallelogram area, computed as the “determinant” of [i j k; u1 u2 u3 ; v1 v2 v3 ] Cyclic shift S Permutation with s21 = 1, s32 = 1, , finally s1n = Its eigenvalues are nth roots e2π ik/n of 1; eigenvectors are columns of the Fourier matrix F Determinant |A| = det(A) Defined by det I = 1, sign reversal for row exchange, and linearity in each row Then |A| = when A is singular Also |AB| = |A||B|, |A−1 | = 1/|A|, and |AT | = |A| The big formula for det(A) has a sum of n! terms, the cofactor formula uses determinants of size n − 1, volume of box = | det(A)| Diagonal matrix D di j = if i = j Block-diagonal: zero outside square blocks Dii Diagonalizable matrix A Must have n independent eigenvectors (in the columns of S; automatic with n different eigenvalues) Then S−1 AS = Λ = eigenvalue matrix Appendix D Glossary: A Dictionary for Linear Algebra 477 Diagonalization Λ = S−1 AS Λ = eigenvalue matrix and S = eigenvector matrix A must have n independent eigenvectors to make S invertible All Ak = SΛk S−1 Dimension of vector space dim(V) = number of vectors in any basis for V Distributive Law A(B +C) = AB + AC Add then multiply, or multiply then add Dot product xT y = x1 y1 + · · · + xn yn Complex dot product is xT y Perpendicular vectors have zero dot product (AB)i j = (row i of A) · (column j of B) Echelon matrix U The first nonzero entry (the pivot) in each row comes after the pivot in the previous row All zero rows come last Eigenvalue λ and eigenvector x Ax = λ x with x = 0, so det(A − λ I) = Eigshow Graphical by eigenvalues and singular values (MATLAB or Java) Elimination A sequence of row operations that reduces A to an upper triangular U or to the reduced form R = rref(A) Then A = LU with multipliers i j in L, or PA = LU with row exchanges in P, or EA = R with an invertible E Elimination matrix = Elementary matrix Ei j The identity matrix with an extra − i j in the i, j entry (i = j) Then Ei j A subtracts i j times row j of A from row i Ellipse (or ellipsoid) xT Ax = A √must be positive definite; the axes of the ellipse are eigenvectors of A, with lengths 1/ λ (For x = the vectors y = Ax lie on the ellipse A−1 y = yT (AAT )−1 y = displayed by eigshow; axis lengths σi ) Exponential eAt = I + At + (At)2 /2! + · · · has derivative AeAt ; eAt u(0) solves u = Au Factorization A = LU If elimination takes A to U without row exchanges, then the lower triangular L with multipliers i j (and ii = 1) brings U back to A Fast Fourier Transform (FFT) A factorization of the Fourier matrix Fn into = log2 n matrices Si times a permutation Each Si needs only n/2 multiplications, so Fn x and Fn−1 c can be computed with n /2 multiplications Revolutionary Fibonacci numbers 0, 1, 1, 2, 3, 5, satisfy √ Fn = Fn−1 + Fn−2 = (λ1n − λ2n )/(λ1 − λ2 ) Growth rate λ1 = (1 + 5)/2 the largest eigenvalue of the Fibonacci matrix 11 10 Four fundamental subspaces of A C (A), N (A), C (AT ), N (AT ) T Fourier matrix F Entries Fjk = e2π i jk/n give orthogonal columns F F = nI Then y = Fc is the (inverse) Discrete Fourier Transform y j = ∑ ck e2π i jk/n Free columns of A Columns without pivots; combinations of earlier columns 478 Appendix D Glossary: A Dictionary for Linear Algebra Free variable xi Column i has no pivot in elimination We can give the n − r free variables any values, then Ax = b determines the r pivot variables (if solvable!) Full column rank r = n Independent columns, N (A) = {0}, no free variables Full row rank r = m Independent rows, at least one solution to Ax = b, column space is all of Rm Full rank means full column rank or full row rank Fundamental Theorem The nullspace N (A) and row space C (AT ) are orthogonal complements (perpendicular subspaces of Rn with dimensions r and n − r) from Ax = Applied to AT , the column space C (A) is the orthogonal complement of N (AT ) Gauss-Jordan method Invert A by row operations on [A I] to reach [I A−1 ] Gram-Schmidt orthogonalization A = QR Independent columns in A, orthonormal columns in Q Each column q j of Q is a combination of the first j columns of A (and conversely, so R is upper triangular) Convention: diag(R) > Graph G Set of n nodes connected pairwise by m edges A complete graph has all n(n − 1)/2 edges between nodes A tree has only n − edges and no closed loops A directed graph has a direction arrow specified on each edge Hankel matrix H Constant along each antidiagonal; hi j depends on i + j T Hermitian matrix AH = A = A Hessenberg matrix H Complex analog of a symmetric matrix: a ji = j Triangular matrix with one extra nonzero adjacent diagonal Hilbert matrix hilb(n) Entries Hi j = 1/(i + j − 1) = but extremely small λmin and large condition number Hypercube matrix PL2 Identity matrix I (or In ) i−1 j−1 x dx x Positive definite Row n + counts corners, edges, faces, , of a cube in Rn Diagonal entries = 1, off-diagonal entries = Incidence matrix of a directed graph The m by n edge-node incidence matrix has a row for each edge (node i to node j), with entries −1 and in columns i and j Indefinite matrix A symmetric matrix with eigenvalues of both signs (+ and −) Independent vectors v1 , , vk No combination c1 v1 + · · · + ck vk = zero vector unless all ci = If the v’s are the columns of A, the only solution to Ax = is x = Inverse matrix A−1 Square matrix with A−1 A = I and AA−1 = I No inverse if det A = and rank(A) < n, and Ax = for a nonzero vector x The inverses of AB and AT are B−1 A−1 and (A−1 )T Cofactor formula (A−1 )i j = C ji / det A Appendix D Glossary: A Dictionary for Linear Algebra Iterative method 479 A sequence of steps intended to approach the desired solution Jordan form J = M −1 AM If A has s independent eigenvectors, its “generalized” eigenvector matrix M gives J = diag(J1 , , Js ) The block Jk is λk Ik + Nk where Nk has 1s on diagonal Each block has one eigenvalue λk and one eigenvector (1, 0, , 0) Kirchhoff’s Laws Current law: net current (in minus out) is zero at each node Voltage law: Potential differences (voltage drops) add to zero around any closed loop Kronecker product (tensor product) A ⊗ B Blocks j B, eigenvalues λ p (A)λq (B) Krylov subspace K j (A, b) The subspace spanned by b, Ab, , A j−1 b Numerical methods approximate A−1 b by x j with residual b − Ax j in this subspace A good basis for K j requires only multiplication by A at each step Least-squares solution x The vector x that minimizes the error e AT Ax = AT b Then e = b − Ax is orthogonal to all columns of A Left inverse A+ solves If A has full column rank n, then A+ = (AT A)−1 AT has A+ A = In Left nullspace N (AT ) Nullspace of AT = “left nullspace” of A because yT A = 0T Length x Square root of xT x (Pythagoras in n dimensions) Linear combination cv + dw or ∑ c j v j Vector addition and scalar multiplication Linear transformation T Each vector v in the input space transforms to T (v) in the output space, and linearity requires T (cv + dw) = cT (v) + dT (w) Examples: Matrix multiplication Av, differentiation in function space Linearly dependent v1 , , A combination other than all ci = gives ∑ ci vi = n Lucas numbers L = 2, 1, √3, 4, , satisfy Ln = Ln−1 + Ln−2 = λ1 + λn , with eigenvalues λ1 , λ2 = (1 ± 5)/2 of the Fibonacci matrix 11 10 Compare L0 = with Fibonacci Markov matrix M All mi j ≥ and each column sum is Largest eigenvalue λ = If mi j > 0, the columns of M k approach the steady-state eigenvector Ms = s > Matrix multiplication AB The i, j entry of AB is (row i of A) · (column j of B) = ∑ aik bk j By columns: column j of AB = A times column j of B By rows: row i of A multiplies B Columns times rows: AB = sum of (column k)(row k) All these equivalent definitions come from the rule that AB times x equals A times Bx Minimal polynomial of A The lowest-degree polynomial with m(A) = zero matrix The roots of m are eigenvalues, and m(λ ) divides det(A − λ I) Multiplication Ax = x1 (column 1) + · · · + xn (column n) = combination of columns 480 Appendix D Glossary: A Dictionary for Linear Algebra Multiplicities AM and GM The algebraic multiplicity AM of an eigenvalue λ is the number of times λ appears as a root of det(A − λ I) = The geometric multiplicity GM is the number of independent eigenvectors (= dimension of the eigenspace for λ ) Multiplier i j The pivot row j is multiplied by i j and subtracted from row i to eliminate the i, j entry: i j = (entry to eliminate)/( jth pivot) Network A directed graph that has constants c1 , , cm associated with the edges Nilpotent matrix N Some power of N is the zero matrix, N k = The only eigenvalue is λ = (repeated n times) Examples: triangular matrices with zero diagonal Norm A of a matrix The “ norm” is the maximum ratio Ax / x = σmax Then Ax ≤ A x , AB ≤ A B , and A + B ≤ A + B Frobenius norm A 2F = ∑ ∑ a2i j ; and ∞ norms are largest column and row sums of |ai j | Normal equation AT Ax = AT b Gives the least-squares solution to Ax = b if A has full rank n The equation says that (columns of A) · (b − Ax) = Normal matrix N NN T = N T N, leads to orthonormal (complex) eigenvectors Nullspace matrix N The columns of N are the n − r special solutions to As = Nullspace N (A) Solutions to Ax = Dimension n − r = (# columns) − rank Orthogonal matrix Q Square matrix with orthonormal columns, so QT Q = I implies QT = Q−1 Preserves length and angles, Qx = x and (Qx)T (Qy) = xT y All |λ | = 1, with orthogonal eigenvectors Examples: Rotation, reflection, permutation Orthogonal subspaces Every v in V is orthogonal to every w in W Orthonormal vectors q1 , , qn Dot products are qTi q j = 0, if i = j and qTi q j = The matrix Q with these orthonormal columns has QT Q = I If m = n, then QT = Q−1 and q1 , , qn is an orthonormal basis for Rn : every v = ∑(vT q j )q j Outer product is uvT column times row = rank-1 matrix Partial pivoting In elimination, the jth pivot is chosen as the largest available entry (in absolute value) in column j Then all multipliers have | i j | ≤ Roundoff error is controlled (depending on the condition number of A) Particular solution x p Any solution to Ax = b; often x p has free variables = j−2 Pascal matrix PS = pascal(n) The symmetric matrix with binomial entries i+i−1 PS = PL PU all contain Pascal’s triangle with det = (see index for more properties) Appendix D Glossary: A Dictionary for Linear Algebra 481 Permutation matrix P There are n! orders of 1, , n; the n! P’s have the rows of I in those orders PA puts the rows of A in the same order P is a product of row exchanges Pi j ; P is even or odd (det P = or −1) based on the number of exchanges Pivot columns of A Columns that contain pivots after row reduction; not combinations of earlier columns The pivot columns are a basis for the column space Pivot d The first nonzero entry when a row is used in elimination Plane (or hyperplane) in Rn perpendicular to a = Polar decomposition A = QH Solutions to aT x = give the plane (dimension n − 1) Orthogonal Q, positive (semi)definite H Positive definite matrix A Symmetric matrix with positive eigenvalues and positive pivots Definition: xT Ax > unless x = Projection matrix P onto subspace S Projection p = Pb is the closest point to b in S, error e = b − Pb is perpendicular to S P2 = P = PT , eigenvalues are or 0, eigenvectors are in S or S⊥ If columns of A = basis for S, then P = A(AT A)−1 AT Projection p = a(aT b/aT a) onto the line through a P = aaT /aT a has rank Pseudoinverse A+ (Moore-Penrose inverse) The n by m matrix that “inverts” A from column space back to row space, with N (A+ ) = N (AT ) A+ A and AA+ are the projection matrices onto the row space and column space rank(A+ ) = rank(A) Random matrix rand(n) or randn(n) MATLAB creates a matrix with random entries, uniformly distributed on [0 1] for rand, and standard normal distribution for randn Rank matrix A = uvT = Column and row spaces = lines cu and cv Rank r(A) Equals number of pivots = dimension of column space = dimension of row space Rayleigh quotient q(x) = xT Ax/xT x For A = AT , λmin ≤ q(x) ≤ λmax Those extremes are reached at the eigenvectors x for λmin (A) and λmax (A) Reduced row echelon form R = rref(A) Pivots= 1; zeros above and below pivots; r nonzero rows of R give a basis for the row space of A Reflection matrix Q = I − 2uuT The unit vector u is reflected to Qu = −u All vectors x in the plane uT x = are unchanged because Qx = x The “Householder matrix” has QT = Q−1 = Q Right inverse A+ If A has full row rank m, then A+ = AT (AAT )−1 has AA+ = Im 482 Appendix D Glossary: A Dictionary for Linear Algebra θ − sin θ rotates the plane by θ , and R−1 = RT rotates back Rotation matrix R = cos sin θ cos θ by −θ Orthogonal matrix, eigenvalues eiθ and e−iθ , eigenvectors (1, ±i) Row picture of Ax = b Each equation gives a plane in Rn planes intersect at x Row space C (AT ) All combinations of rows of A Column vectors by convention Saddle point of f (x1 , , xn ) A point where the first derivatives of f are zero and the second derivative matrix (∂ f /∂ xi ∂ x j = Hessian matrix) is indefinite Schur complement S = D −CA−1 B Schwarz inequality |v · w| ≤ v w Appears in block elimination on A B CD Then |vT Aw|2 ≤ (vT Av)(wT Aw) if A = CTC Semidefinite matrix A (Positive) semidefinite means symmetric with xT Ax ≥ for all vectors x Then all eigenvalues λ ≥ 0; no negative pivots B = M −1 AM has the same eigenvalues as A Similar matrices A and B Simplex method for linear programming The minimum cost vector x∗ is found by moving from corner to lower-cost corner along the edges of the feasible set (where the constraints Ax = b and x ≥ are satisfied) Minimum cost at a corner! Singular matrix A A square matrix that has no inverse: det(A) = Singular Value Decomposition (SVD) A = UΣV T = (orthogonal U) times (diagonal Σ) times (orthogonal V T ) First r columns of U and V are orthonormal bases of C (A) and C (AT ), with Avi = σi ui and singular value σi > Last columns of U and V are orthonormal bases of the nullspaces of AT and A Skew-symmetric matrix K The transpose is −K, since Ki j = −K ji Eigenvalues are pure imaginary, eigenvectors are orthogonal, eKt is an orthogonal matrix Solvable system Ax = b The right side b is in the column space of A Spanning set v1 , , vm , for V Special solutions to As = Every vector in V is a combination of v1 , , vm One free variable is si = 1, other free variables = Spectral theorem A = QΛQT Real symmetric A has real λi and orthonormal qi , with Aqi = λi qi In mechanics, the qi give the principal axes Spectrum of A The set of eigenvalues {λ1 , , λm } Spectral radius = |λmax | Standard basis for Rn Columns of n by n identity matrix (written i, j, k in R3 ) Stiffness matrix K When x gives the movements of the nodes in a discrete structure, Kx gives the internal forces Often K = ATCA, where C contains spring constants from Hooke’s Law and Ax = stretching (strains) from the movements x Appendix D Glossary: A Dictionary for Linear Algebra Subspace S of V 483 Any vector space inside V, including V and Z = {zero vector} Sum V + W of subspaces Space of all (v in V )+(w in W) Direct sum: dim(V + W) = dim V + dim W, when V and W share only the zero vector Symmetric factorizations A = LDLT and A = QΛQT in D and positive eigenvalues in Λ is the same The number of positive pivots Symmetric matrix A The transpose is AT = A, and j = a ji A−1 is also symmetric All matrices of the form RT R and LDLT and QΛQT are symmetric Symmetric matrices have real eigenvalues in Λ and orthonormal eigenvectors in Q Toeplitz matrix T Constant-diagonal matrix, so ti j depends only on j − i Toeplitz matrices represent linear time-invariant filters in signal processing Trace of A Sum of diagonal entries = sum of eigenvalues of A TrAB = TrBA Transpose matrix AT Entries ATi j = A ji AT is n by m, AT A is square, symmetric, positive semidefinite The transposes of AB and A−1 are BT AT and (AT )−1 Triangle inequality u + v ≤ u + v Tridiagonal matrix T For matrix norms, A + B ≤ A + B ti j = if |i − j| > T −1 has rank above and below diagonal T Unitary matrix U H = U = U −1 Orthonormal columns (complex analog of Q) Vandermonde matrix V V c = b gives the polynomial p(x) = c0 + · · · + cn−1 xn−1 with p(xi ) = bi at n points Vi j = (xi ) j−1 , and detV = product of (xk − xi ) for k > i Vector addition v + w = (v1 + w1 , , + wn ) = diagonal of parallelogram Vector space V Set of vectors such that all combinations cv + dw remain in V Eight required rules are given in Section 2.1 for cv + dw Vector v in Rn Sequence of n real numbers v = (v1 , , ) = point in Rn Volume of box The rows (or columns) of A generate a box with volume | det(A)| Wavelets w jk (t) or vectors w jk Rescale and shift the time axis to create w jk (t) = w00 (2 j t − k) Vectors from w00 = (1, 1, −1, −1) would be (1, −1, 0, 0) and (0, 0, 1, −1) Appendix E MATLAB Teaching Codes cofactor Compute the n by n matrix of cofactors cramer Solve the system Ax = b by Cramer’s Rule deter Matrix determinant computed from the pivots in PA = LU eigen2 Eigenvalues, eigenvectors, and det(A − λ I) for by matrices eigshow Graphical demonstration of eigenvalues and singular values eigval Eigenvalues and their multiplicity as roots of det(A − λ I) = eigvec Compute as many linearly independent eigenvectors as possible elim Reduction of A to row echelon form R by an invertible E findpiv Find a pivot for Gaussian elimination (used by plu) fourbase Construct bases for all four fundamental subspaces grams Gram-Schmidt orthogonalization of the columns of A house by 12 matrix giving corner coordinates of a house inverse Matrix inverse (if it exists) by Gauss-Jordan elimination leftnull Compute a basis for the left nullspace linefit Plot the least squares fit to m given points by a line lsq Least-squares solution to Ax = b from AT A = AT b normal Eigenvalues and orthonormal eigenvectors when AT A = AAT nulbasis Matrix of special solutions to Ax = (basis for null space) orthcomp Find a basis for the orthogonal complement of a subspace partic Particular solution of Ax = b, with all free variables zero ... Newton needed calculus—quite right But the scope of science and engineering and management (and life) is now so much wider, and linear algebra has moved into a central place May I say a little... balance toward linear algebra Working with curved lines and curved surfaces, the first step is always to linearize Replace the curve by its tangent line, fit the surface by a plane, and the problem... and current homeworks and exams with solutions The goals of the course, and conceptual questions Interactive Java demos (audio is now included for eigenvalues) Linear Algebra Teaching Codes and