40 Murio 2.2.4 Equivalence Relations Now we concentrate our attention on the properties of a binary relation ` de®ned in a set X 1 2 3 ` is called re¯exive in X, if and only if, for all x P X, x`x ` is called symmetrical in X, if and only if, for all x; y P X, x`y implies y`x ` is called transitive in X, if and only if, for all x; y; z P X, x`y and y`z implies x`z A binary relation ` is called an equivalence relation on X if it is re¯exive, symmetrical and transitive As an example, consider the set Z of integer numbers and let n be an arbitrary positive integer The congruence relation modulo n on the set Z is de®ned by x y (modulo n) if and only if x À y kn for some k P Z The congruence relation is an equivalence relation on Z Proof 1 2 3 For each x P Z, x À x 0n This means that x x (modulo n) which implies that the congruence relation is re¯exive If x y (modulo n), x À y kn for some k P Z Multiplying both sides of the last equality by À1, we get y À x Àkn which implies that y x (modulo n) Thus, the congruence relation is symmetrical If x y (modulo n) and y z (modulo n), we have x À y k1 n and y À z k2 n for some k1 and k2 in Z Writing x À z x À y y À z, we get x À z k1 k2 n Since k1 k2 P Z, we conclude that x z (modulo n) This shows that the congruence relation is transitive From 1±3 it follows that the congruence relation (modulo n) is an equivalence relation on the set Z of integer numbers In particular, we observe that if we choose n 2, then x y (modulo 2) means that x À y 2k for some integer k This is equivalent to saying that either x and y are both even or both x and y are odd In other words, any two even integers are equivalent, any two odd integers are equivalent but an even integer can not be equivalent to an odd one The set Z has been divided into two disjoint subsets whose union gives Z One such proper subset is the set of even integers and the other one is the set of odd integers Copyright © 2000 Marcel Dekker, Inc 2.2.4.1 Partitions and Equivalence Relations The situation described in the last example is quite general To study equivalence relations in more detail we need to introduce the concepts of partition and equivalence class Given a nonempty set X, a partition S of X is a collection of nonempty subsets of X such that 1 2 If A; B P S; A T B, then A B Y APS A X If ` is an equivalence relation on a nonempty set X, for each member x P X the equivalence class associated with x, denoted x=`, is given by x=` fz P X X x`xg The set x=` is a subset of X and, consequently, an element of the power set P X Thus, the set X=` fy P P x X y x=` for some y P Xg is also a well-de®ned subset of P x called the quotient set of X by ` The correspondence between the partition of a nonempty set and the equivalence relation determined by it is established in the following propositions The quotient set x=` of a set X by an equivalence relation ` is a partition of the set X The converse of this statement also holds; that is, each partition of X generates an equivalence relation on X In fact, if S is a partition of a nonempty set X, we can de®ne the relation X=S f x; y P X  X X x P s and y P s for some s P Sg This is an equivalence relation on X, and the equivalent classes induced by it are precisely the elements of the partition S, i.e., X= X=S S Intuitively, equivalence relations and partitions are two different ways to describe the same collection of subsets 2.2.5 Order Relations Order relations constitute another common type of relations Once again, we begin by introducing several de®nitions A binary relation ` in X is said to be antisymmetrical if for all x; y P X; x`y and y`x imply x y Introduction to Sets and Relations 41 A binary relation ` in X is asymmetrical if for any x; y P X; x`y implies that y`x does not hold In other words, we can not have x`y an y`x both true A binary relation ` in X is a partial ordering of X if and only if it is re¯exive, antisymmetrical, and transitive The pair X; ` is called and ordered set A binary relation in X is a strict (or total) ordering of X if and only if it is asymmetrical and transitive For example, consider the set of integers X f1; 3; 2g and the binary relation in X given by `1 f x; y X x; y P X and x yg `1 f x; y X x; y P X and x < yg f 1; 2; 1; 3; 2; 3g is an example of a strict ordering of X It is also possible to establish a correspondence between partial orderings and strict orderings of a set: If `1 is a partial ordering of X, then the binary relation `2 de®ned in X by x`2 y if and only if x`1 y and x T y is a strict ordering of X Finally, if `2 is a strict ordering of X, then the relation `1 de®ned in X by x`1 y if and only if x`2 y or x y is a partial ordering of X This gives explicitly `1 f 1; 1; 2; 2; 3; 3; 1; 2; 1; 3; 2; 3g It is a simple task to check that `1 is a partial ordering of the set X It requires a little extra thinking to realize that now the least and the greatest elements of X have been identi®ed On the same set X, the binary relation de®ned by Copyright © 2000 Marcel Dekker, Inc GENERAL REFERENCES 1 PR Halmos Naive Set Theory New York: Van  Nostrand Reinhold, 1960 2 K Hrbacek, T Jech Introduction to Set Theory New York: Marcel Dekker, 1978 Chapter 1.3 Linear Algebra William C Brown Michigan State University, East Lansing, Michigan 3.1 3.1.1 MATRICES De®nition 2 The set of all m  n matrices with entries from F will be denoted by MmÂn F Shapes and Sizes Matrices of various shapes are given special names in linear algebra Here is a brief list of some of the more famous shapes and some pictures to illustrate the de®nitions Throughout this chapter, F will denote a ®eld The four most commonly used ®elds in linear algebra are Q rationals, R reals, C complex numbers and Zp the integers modulo a prime p We will also let N f1; 2; F F Fg, the set of natural numbers 1 A matrix is square if m n H a a11 a12 d 11 ; a21 a; a21 a22 a31 De®nition 1 Let m; n P N An m  n matrix A with entries from F is a rectangular array of m rows and n columns of numbers from F size 1  1; 2  2; The most common notation used to represent an m  n (read ``m by n'') matrix A is displayed in Eq (1): H a11 ; a12 f F Ad F F am1 ; am2 ;FFF; 1 If A is the m  n matrix displayed in Eq (1), then the ®eld elements aij i 1; F F F ; mY j 1; F F F ; n are called the entries of A We will also use Aij to denote the i; jth entry of A Thus, aij Aij is the element of F which lies in the ith row and jth column of A By the size of A, we will mean the expression m  n Thus, size A m  n if A has m rows and n columns Notice that the size of a matrix is a pair of positive integers with a ``Â'' put between them Negative numbers and zero are not allowed to appear in the size of a matrix 3  3; F F F size 1  1; 2  1; F F F ; n  1 2a 2b 3 An m  n matrix is called a row vector if m 1 a; a; b; F F F ; a1 ; F F F ; an size 1  1; 1  2; F F F ; 1Ân 2c 4 An m  n matrix A is upper triangular if Aij 0 whenever i > j 43 Copyright © 2000 Marcel Dekker, Inc I a13 a23 e; F F F a33 2 An m  n matrix is called a column vctor if n 1 H I a1 a f F g a; ;FFFd F e F b an I a1n F g F e F ; F F F ; amn a12 a22 a32 44 Brown H a11 f f 0 f f Af 0 f F f F d F 0 if m a12 a22 a13 a23 FFF FFF a1m a2m FFF FFF 0 F F F 0 a33 F F F 0 FFF a3m F F F FFF amm FFF FFF I a1n a2n g g g a3n g g F g F g F e amn n 2d 5 6 7 An m  n matrix A is lower triangular if Aij 0 whenever i < j H I a11 0 0 FFF 0 0 FFF 0 fa 0 FFF 0 0 FFF 0g f 21 a22 g f g f a31 a32 a33 F F F 0 0 FFF 0g Af g f F Fg F F F F f F Fg F F F F Fe F F F F d F am1 am2 am3 F F F amm 0 F F F 0 if m n 2e An m  n matrix A is diagonal if Aij 0 whenever i T j I H a11 0 F F F 0 f 0 a22 F F F 0 g g f if m n 2f f F F F g F F e d F F F F 0 0 F F F ann A square matrix is symmetric (skew-symmetric) if Aij Aji ÀAji for all i; j 1; F F F ; n H I a b c a b d a; ; b d e e; F F F symmetric b c c e f x y x x; y; z; w; ; ; x; y; z; w; z w z 0; 0 Àb H 0 f ; d Àb 0 Àc b b 0 Àe c I g e e; F F F 0 2h skew-symmetric De®nition 3 A submatrix of A is a matrix obtained from A by deleting certain rows and/or columns of A A partition of A is a series of horizontal and vertical lines drawn in A which divide A into various submatrices x y Example 1 Suppose A Then z w Copyright © 2000 Marcel Dekker, Inc 3 is a complete list of the submatrices of A x y x y x z x y ; ; ; z w z w y w z w 4 are all partitions of A The most important partitions of a matrix A are its column and row partitions De®nition 4 Let H a11 ; F F F ; f F Ad F F am1 ; F F F ; 1 2 3 4 2g size 1  1; 2  2; 3  3 y w I a1n F g F e P MmÂn F F amn For each j 1; F F F ; n, the m  1 submatrix I H a1j f F g Colj A d F e F amj of A is called the jth column of A A Col1 A j Col2 A j F F F j Coln A is called the column partition of A For each i 1; F F F ; m, the 1  n submatrix Rowi A ai1 ; F F F ; ain of A is called the ith row of A H I Row1 A f g F Ad F e F Rowm A is called the row partition of A We will cut down on the amount of space required to show a column or row partition by employing the following notation In De®nition 4, let j Colj A for j 1; F F F ; n and let i Rowi A for i 1; F F F ; m Then the column partition of A will be written A 1 j 2 j F F F j n and the row partition of A will be written A 1 Y 2 Y F F F Y m 3.1.2 Matrix Arithmetic De®nition 5 Two matrices A and B with entries from F are said to be equal if size A size B and Aij Bij for all i 1; F F F ; m; j 1; F F F ; n Here m  n size A Linear Algebra 45 If A and B are equal, we will write A B Notice that two matrices which are equal have the same size Thus, the 1  1 matrix (0) is not equal to the 1  2 matrix (0,0) Matrix addition, scalar multiplication, and multiplication of matrices are de®ned as follows De®nition 6 Let A, B P MmÂn F Then A B is the m  n matrix whose i,jth entry is given by A Bij Aij Bij for all i 1; F F F ; m and j 1; F F F ; n If A P MmÂn F and x P F, then xA is the m  n matrix whose i,jth entry is given by xAij xAij for all i 1; F F F ; m and j 1; F F F ; n Let A P MmÂn F and C P MnÂp F Then AC is the m  p matrix whose i,jth entry is given by 1 2 3 ACij n Aik Ckj k1 for i 1; F F F ; mY j 1; F F F ; p Example 2 Let 1 0 3 A 1 1 2 and let H 0 C d À1 4 Then AB AC B 1 0 1 0 1 2 P M2Â3 Q A 0 4 2 1 4 12 2 7 3 1 2 3 4 5 2 2 B 2A B 2C 6A 6 0 18 1 0 0 2 C 0 1 1 1 6 6 12 A B B A: A B C A B C: A 0 A: A À1A 0: xyA x yA: 0 1 0 2 6 The rules for matrix multiplication are as follows: 5 Let A, B, C P MmÂn F Let x, y P F Copyright © 2000 Marcel Dekker, Inc 1 1 Let We view A, B, C P M2Â2 Z3 Then Notice that addition is de®ned only for matrices of the same size Multiplication is de®ned only when the number of columns of the ®rst matrix is equal to the number of rows of the second matrix The rules for matrix addition and scalar multiplication are summarized in the following theorem: Theorem 1 Then When no explicit reference is given, a proof of the quoted theorem can be found in Brown [1] The number zero appearing in 3 and 4 above denotes the m  n matrix all of whose entries are zero The eight statements given in Theorem 1 imply MmÂn F is a vector space over F (see de®nition 16 in Sec 3.2) when vector addition and scalar multiplication are given as in 1 and 2 of De®nition 6 Theorem 1(2) implies matrix addition is associative It follows from this statement that expressions of the form x1 A1 Á Á Á xr Ar Ai P MmÂn F and xi P F can be used unambiguously Any placement of parentheses in this expression will result in the same answer The sum x1 A1 Á Á Á xr Ar is called a linear combination of A1 ; F F F ; Ar The numbers x1 ; F F F ; xr are called the scalars of the linear combination Example 3 I 2 1 e P M3Â2 Q 0 2 6 x A B xA xB: 7 x yA xA yA: 8 1A A: Theorem 2 Let A, D P MmÂn F, B P MnÂp F, C P MpÂq F and E P MrÂm F Let x P F Then 1 2 3 4 5 6 ABC A BC: A DB AB DB: E A D EA ED: 0A 0 and A0 0: Im A A and AIn A: x AB xAB A xB: In Theorem 2(4), the zero denotes the zero matrix of various sizes In Theorem 2(5), In denotes the n  n identity matrix This is the diagonal matrix given by In jj 1 for all j 1; F F F ; n Theorem 2 implies MnÂn F is an associative algebra with identity [2, p 36] over the ®eld F Consider the following system of m equations in unknowns x1 ; F F F ; xnX 46 Brown a11 x1 a12 x2 Á Á Á a1n xn b1 F F F F F F 7 am1 x1 am2 x2 Á Á Á amn xn bm In Eq (7), the aij 's and bi 's are constants in F Set H I H I H I a11 ; F F F ; a1n b1 x1 f F g f F g f F g F g F e Bf F gX f F g Af F F d F d F e d F e am1 ; F F F ; amn bm xn 8 Using matrix multiplication, the system of linear equations in Eq (7) can be written succinctly as AX B 9 We will let F n denote the set of all column vectors of size n Thus, F n MnÂ1 F A column vector P F n is called a solution to Eq (9) if A B The m  n matrix A aij P MmÂn F is called the coef®cient matrix of Eq (7) The partitioned matrix A j B P Mm n1 F is called the augmented matrix of Eq (7) Matrix multiplication was invented to handle linear substitutions of variables in Eq (7) Suppose y1 ; F F F ; yp are new variables which are related to x1 ; F F F ; xn by the following set of linear equations: x1 c11 y1 Á Á Á c1p yp F F F xn cn1 y1 Á Á Á cnp yp (here cuv P F for all u; v A A c11 ; f F Cd F F cn1 ; FFF; FFF; Substituting the expressions in Eq (10) into Eq (7) produces m equations in y1 ; F F F ; yp The coef®cient matrix of the new system is AC, the matrix product of A and C De®nition 7 A square matrix A P MnÂn F is said to be invertible (or nonsingular) if there exists a square matrix B P MnÂn F such that AB BA In If A P MnÂn F is invertible and AB BA In for some B P MnÂn F, then B is unique and will be denoted by AÀ1 AÀ1 is called the inverse of A Example 4 Let Copyright © 2000 Marcel Dekker, Inc P M2Â2 F À1 w=Á Ày=Á Àz=Á x=Á 11 De®nition 8 Let A P MmÂn F The transpose of A is denoted by At At is the n  m matrix whose entries are given by At ij Aji for all i 1; F F F ; n and j 1; F F F ; m A square matrix is symmetric (skew-symmetric) if and only if A At ÀAt When the ®eld F C, the complex numbers, the Hermitian conjugate (or conjugate transpose) of A is more useful than the transpose De®nition 9 Let A P MmÂn C The Hermitian conjugate of A is denoted by Aà Aà is the n  m matrix whose " entries are given by Aà ij Aji for all i 1; F F F ; n and j 1; F F F ; m In De®nition 9, the bar over Aji denotes the conjugate of the complex number Aji For example, I c1p F g P M F F e nÂp F cnp If m n in Eq (7) and the coef®cient matrix A is invertible, then AX B has the unique solution AÀ1 b 1i 2 2Ài i 1i 2 2Ài H y w and assume Á xw À yz T 0 Then A is invertible with inverse given by 10 Set x z i à t 1Ài 2i 2 Ài 1i 2Ài 2 i and 12 The following facts about transposes and Hermitian conjugates are easy to prove Theorem 3 Then 1 2 3 4 Let A, C P MmÂn F and B P MnÂp F A Ct At Ct : ABt Bt At : At t A: If m n and A is invertible, then At is also invertible In this case, At À1 AÀ1 t If F C, then we also have 5 6 7 8 A Cà Aà C à : ABà Bà Aà : Aà à A: If A is invertible, so is Aà and Aà À1 AÀ1 à Linear Algebra 3.1.3 47 Block Multiplication of Matrices Theorem 4 Let A P MmÂn F and B P MnÂp F Suppose I H A11 F F F A1k f F F g F g and Af F F e d F Ar1 F F F Ark H I B11 F F F B1t f F F g F g Bf F F e d F Bk1 F F F Bkt are partitions of A and B such that size Aij mi  nj and size Bjl nj  pl Thus, m1 Á Á Á mr m, n1 Á Á Á nk n, and p1 Á Á Á pt p For each i 1; F F F ; r and j 1; F F F ; t, set Cij Then k q1 Aiq Bqj (multiplication of blocks) C11 f F AB d F F Cr1 FFF 2 3 4 F F F Crk 1 2 Let A P MmÂn F t n n If x1 ; F F F ; xn P F , then A i1 xi Coli A If B 1 j F F F j p P MnÂp F, then AB A1 j F F F j Ap : If y1 ; F F F ; ym P M1Âm F, then A m i1 yi Rowi A If C 1 Y F F F Y r P MrÂm F, CA 1 AY F F F Y r ; A: De®nition 10 13 The column space of A is particularly important for the theory of linear equations Suppose A P MmÂn F and B P F m Theorem 5 implies that AX B has a solution if and only if B P CS A: 3.1.4 Gaussian Elimination The three elementary row operations that can be performed on a given matrix A are as follows: Interchange two rows of A Add a scalar times one row of A to another row of A Multiply a row of A by a nonzero scalar There are three corresponding elementary column operations which can be preformed on A as well I C1t F g F e F Notice that the only hypothesis in Theorem 4 is that every vertical line drawn in A must be matched with the corresponding horizontal line in B There are four special cases of Theorem 4 which are very useful We collect these in the next theorem 1 CS AB CS A RS AB RS B 14 H Theorem 5 RS A is all linear combinations of the rows of A Using all four parts of Theorem 5, we have Let A P MmÂn F CS A fA j P F n g is called the column space of A RS A fA j P M1Âm Fg is called the row space of A Theorem 5 implies that the column space of A consists of all linear combinations of the columns of A Copyright © 2000 Marcel Dekker, Inc De®nition 11 Let A1 ; A2 P MmÂn F A1 and A2 are said to be row (column) equivalent if A2 can be obtained from A1 by applying ®nitely many elementary row (column) operations to A1 : If A1 and A2 are row (column) equivalent, we will ~ ~ write A1 r A2 A1 c A2 Either one of these relations is an equivalence relation on MmÂn F By this, we mean ~ A1 r A1 ~ ~ A1 r A2 D A2 r A1 ~ ~ ~ A1 r A2 ; A2 r A3 A A1 r A3 (~ is re¯exive) r (~ is symmetric) r (~ is transitive) r (15) Theorem 6 Let A, C P MmÂn F and B, D P F m ~ Suppose A j B r C j D Then the two linear systems of equations AX B and CX D have precisely the same solutions Gaussian elimination is a strategy for solving a system of linear equations To ®nd all solutions to the linear system of equations a11 x1 a12 x2 Á Á Á a1n xn b1 F F F am1 x1 am2 x2 Á Á Á amn xn bm carry out the following three steps 16 48 Brown 1 2 3 Set up the augmented matrix of Eq (16): I H a11 ; F F F ; a1n b1 f F F F g F F e A j B d F F F F am1 ; F F F ; amn bm Thus, x 13=4, y 3=2 and z À1=4 is the (unique) solution to à Apply elementary row operations to A j B to obtain a matrix C j D in upper triangular form Solve CX D by back substitution By Theorem 6, this algorithm yields a complete set of solutions to AX B Example 5 Theorem 7 F Q à xz3 H 2 f d1 1 1 2 Following steps 1±3, we have I 3 4 10 g À1 À1 2 e 0 1 3 3.1.5 is the augmented matrix of à 2 Using Gaussian elimination, we can prove the following theorem Solve 2x 3y 4z 10 xÀyÀz2 1 De®nition 12 Let A P MmÂn F A system of equations of the form AX 0 is called a homogeneous system of equations A nonzero, column vector P F n is called a nontrivial solution of AX 0 if A 0 I I H 3 4 10 1 0 1 3 À 3 g g f À1 À1 2 e d 1 À1 À1 2 e 1 0 1 3 2 3 4 10 H I 1 0 1 3 À 3 f g À3 d 0 À1 À2 À1 e 0 3 2 4 I H I H 1 0 13 1 0 13 À 3 g f g f 21e d 0 1 2 1 e for all ; P V and x P F V7 x y x y for all P V and x; y P F V8 1 for all P V Suppose V; ; 3 ; x; 3 x is a vector space over F The elements in V are called vectors and will usually be denoted by Greek letters The elements in F are called scalars and will be represented by lowercase English letters The function ; 3 is called vector addition The function x; 3 x is called scalar multiplication Notice that a vector space is actually an ordered triple consisting of a set of vectors, the function vector addition and the function scalar multiplication It is possible that a given set V can be made into a vector space over F in many different ways by specifying different vector additions or scalar multiplications on V Thus, when de®ning a vector space, all three pieces of information (the vec- Copyright © 2000 Marcel Dekker, Inc 1 2 3 4 5 Let V be a vector space over F Then Any parentheses placed in 1 Á Á Á n result in the same vector x0 0 for all x P F 0 0 for all P V À1 À for all P V If x 0 then, x 0 or 0 The reader will notice that we use 0 to represent the zero vector in V as well as the zero scalar in F This will cause no real confusion in what follows Theorem 14(1) implies linear combinations of vectors in V, i.e., sums of the form x1 1 Á Á Á xn n , can be written unambiguously with no parentheses The notation for various vector spaces students encounter when studying linear algebra is becoming standard throughout most modern textbooks Here is a short list of some of the more important vector spaces If the reader is in doubt as to what addition or scalar multiplication is in the given example, consult Brown [1, 2] 1 F S all functions from a set S to the ®eld F 2 MmÂn F the set of all m  n matrices with entries from F 3 FX the set of all polynomials in X with coef®cients from F 4 CS A; RS A, and NS A f P F n j A 0g for any A P MmÂn F (NS A is called the null space of A.) 5 C k I f f P RI j f is k times differentiable on Ig (I here is usually some open or closed set contained in R) 6 a; b f f P Ra;b j f is Riemann integrable on a; bg (24) 52 Brown De®nition 17 Let W be a nonempty subset of a vector space V W is a subspace of V if P W and x P W for all ; P W and x P F Thus, a subset is a subspace if it is closed under vector addition and scalar multiplication RX; C k a; b and a; b are all subspaces of Ra;b If A P MmÂn F, then NS A is a subspace of F n , CS A is a subspace of F m and RS A is a subspace of M1Ân F One of the most important sources of subspaces are linear spans De®nition 18 Let S be a subset of a vector space V The set of all linear combinations of vectors from S is called the linear span of S We will let L S denote the linear span of S If S Y, i.e., S is empty, then we set L S 0 Notice, P L S if x1 1 Á Á Á xn n for some 1 ; F F F ; n P S and x1 ; F F F ; xn P F If S is ®nite, say S f 1 ; F F F ; r g, then we often write L 1 ; F F F ; r for L S For example, if A 1 j F F F jn P MmÂn F, then L 1 ; F F F ; n CS A Theorem 15 1 2 3 4 5 Let V be a vector space over F For any subset S V, L S is a subspace of V If S1 S2 V, then L S1 L S2 V If P L S, then P L S1 for some ®nite subset S1 S L L S L S Exchange principle: If P L S f g and P L S, then P L S f ... relation `2 de®ned in X by x `2 y if and only if x`1 y and x T y is a strict ordering of X Finally, if `2 is a strict ordering of X, then the relation `1 de®ned in X by x`1 y if and only if x `2 y or... g e e; F F F 2h skew-symmetric De®nition A submatrix of A is a matrix obtained from A by deleting certain rows and/ or columns of A A partition of A is a series of horizontal and vertical lines... x `2 y or x y is a partial ordering of X This gives explicitly `1 f 1; 1; ? ?2; 2? ?; 3; 3; 1; 2? ?; 1; 3; ? ?2; 3g It is a simple task to check that `1 is a partial ordering of the set X It requires