Linear algebra (2)

Review of Linear Algebra Review of Linear Algebra CS224W: Social and Information Network Analysis Fall 2015 Nihit Desai, Sameep Bagadia Based on Fall 2014 slides by David Hallac Review of Linear Algebra Matrices and Vectors Matrix: A rectangular array of numbers, e.g., A ∈ Rm×n :   a11 a12 a1n  a21 a22 a2n    A=    am1 am2 amn Vector: A matrix consisting of only one column (default) or one row, e.g., x ∈ Rn   x1  x2    x =    xn Review of Linear Algebra Matrix Multiplication If A ∈ Rm×n , B ∈ Rn×p , then their product AB ∈ Rm×p Number of columns of A must equal number of rows of B We can compute the product C = AB using this formula: Cij = n X k=1 Aik Bkj Review of Linear Algebra Matrix Multiplication (AB)(x) = A(B(x)) Review of Linear Algebra Properties of Matrix Multiplication Associative: (AB)C = A(BC) Distributive: A(B + C) = AB + AC Non-commutative: AB 6= BA They don’t even have to be the same size! Review of Linear Algebra Linear Transformation and Matrices A linear transformation T is a function from Rn to Rm that satisfies two properties: For all x, y ∈ Rn , T (x + y) = T (x) + T (y) For all x ∈ Rn and all a ∈ R (scalar) T (ax) = aT (x) Every linear transformation can be represented by a matrix Every matrix is a linear transformation Review of Linear Algebra Transform Example Let  M=  0.3 If we apply M to every point on the Mona Lisa, we get the following: Review of Linear Algebra Matrix Transpose Transpose: A ∈ Rm×n , then AT ∈ Rn×m : (AT )ij = Aji For example, if   A= then   AT = 2 5 Properties: (AT )T = A (AB)T = B T AT (A + B)T = AT + B T Review of Linear Algebra Matrix Inverse If A ∈ Rn×n and invertible, then the inverse of A, denoted A−1 is the matrix that: AA−1 = A−1 A = I Properties: (A−1 )−1 = A (AB)−1 = B −1 A−1 (A−1 )T = (AT )−1 Review of Linear Algebra Identity Matrix Identity matrix: I = In ∈ Rn×n : ( i=j, Iij = otherwise ∀A ∈ Rm×n : AIn = Im A = A Review of Linear Algebra Diagonal Matrix Diagonal matrix: D = diag(d1 , d2 , , dn ): ( di j=i, Dij = otherwise Review of Linear Algebra Other Special Matrices Symmetric matrices: A ∈ Rn×n is symmetric if A = AT Orthogonal matrices: U ∈ Rn×n is orthogonal if UU T = I = U T U Every column is orthogonal to every other column (dot product = 0) The inverse of an orthogonal matrix is its transpose Review of Linear Algebra Linear combinations and Span Given a set of vectors S = {x1 , , xn } where xi ∈ Rn , a linear combination of this set of vectors is an expression of P the form: ni=1 αi xi where αi ∈ R Span(S) is the set of all linear combinations of the elements of S Review of Linear Algebra Linear Independence and Rank A set of vectors S =P {x1 , , xn } is linearly independent if the following holds: ni=1 αi xi = only if α1 = α2 = αn = Rank: A ∈ Rm×n , then rank(A) is the maximum number of linearly independent columns (or equivalently, rows) Properties: rank (A) ≤ min{m, n} rank (A) = rank (AT ) rank (AB) ≤ min{rank (A), rank(B)} rank (A + B) ≤ rank (A) + rank (B) Review of Linear Algebra Example of Linear Dependence These three vectors are linearly dependent because they all lie in the same plane The matrix with these three vectors as rows has rank Review of Linear Algebra Eigenvalues and Eigenvectors Given A ∈ Rn×n , λ ∈ C is an eigenvalue of A with the corresponding eigenvector x ∈ Cn (x 6= 0) if: Ax = λx For example, if  A=    then the vector is an eigenvector with eigenvalue 1, −3 because Review of Linear Algebra Solving for Eigenvalues/Eigenvectors Characteristic Polynomial: If Ax = λx then (A − λI)x = so (A − λI) is singular (not full rank), so det(A − λI) = Thus the eigenvalues are exactly the n possibly complex roots of the degree n polynomial equation det(A − λI) = This is known as the characteristic polynomial Once we solve for all λ’s, we can plug in to find each corresponding eigenvector Review of Linear Algebra Eigenvalue/Eigenvector Properties Usually eigenvectors are normalized to unit length If A is symmetric, then all the eigenvalues are real P tr (A) = ni=1 λi Q det(A) = ni=1 λi Review of Linear Algebra Matrix Eigendecomposition A ∈ Rn×n , λ1 , , λn the eigenvalues, and x1 , , xn the eigenvectors P = [x1 |x2 | |xn ], D = diag(λ1 , , λn ), then: Review of Linear Algebra Matrix Eigendecomposition Therefore, A = PDP −1 In addition: A2 = (PDP −1 )(PDP −1 ) = PD(P −1 P)DP −1 = PD P −1 By induction, An = PD n P −1