www.TechnicalBooksPDF.com www.TechnicalBooksPDF.com 7814tp.indd 2/8/11 8:31 AM This page intentionally left blank www.TechnicalBooksPDF.com Hrishikesh D Vinod Fordham University, USA World Scientific NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TA I P E I • CHENNAI www.TechnicalBooksPDF.com 7814tp.indd 2/8/11 8:31 AM Published by World Scientific Publishing Co Pte Ltd Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library HANDS-ON MATRIX ALGEBRA USING R Active and Motivated Learning with Applications Copyright © 2011 by World Scientific Publishing Co Pte Ltd All rights reserved This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA In this case permission to photocopy is not required from the publisher ISBN-13 ISBN-10 ISBN-13 ISBN-10 978-981-4313-68-1 981-4313-68-8 978-981-4313-69-8 (pbk) 981-4313-69-6 (pbk) Printed in Singapore www.TechnicalBooksPDF.com RokTing - Hands-On Matrix Algebra.pmd 2/14/2011, 4:16 PM February 21, 2011 9:40 World Scientific Book - 9in x 6in To my wife Arundhati, daughter Rita and her children Devin and Troy www.TechnicalBooksPDF.com v AllChapters February 21, 2011 9:40 World Scientific Book - 9in x 6in This page intentionally left blank www.TechnicalBooksPDF.com AllChapters February 21, 2011 9:40 World Scientific Book - 9in x 6in Preface In high school, I used to like geometry better than algebra or arithmetic I became excited about matrix algebra after my teacher at Harvard, Professor Wassily Leontief, Nobel laureate in Economics showed me how his input-output analysis depends on matrix inversion Of course, inverting a 25×25 matrix was a huge deal at that time It got me interested in computer software for matrix algebra tasks This book brings together my two fascinations, matrix algebra and computer software to make the algebraic results fun to use, without the drudgery of patient arithmetic manipulations I was able to find a flaw in Nobel Laureate Paul Samuelson’s published work by pointing out that one of his claims for matrices does not hold for scalars Further excitement came when I realized that Italian economist Sraffa’s work, extolled in Professor Samuelson’s lectures can be understood better in terms of eigenvectors My interest about matrix algebra further increased when I started working at Bell Labs and talking to many engineers and scientists My enthusiasm for matrix algebra increased when I worked with my friend Sharad Sathe on our joint paper Sathe and Vinod (1974) My early publication in Econometrica on joint production, Vinod (1968), heavily used matrix theory My generalization of the Durbin-Watson test in Vinod (1973) exploited the Kronecker product of matrices In other words, a study of matrix algebra has strongly helped my research agenda over the years Research oriented readers will find that matrix theory is full of useful results, ripe for applications in various fields The hands-on approach here using the R software and graphics hopes to facilitate the understanding of results, making such applications easy to accomplish An aim of this book is to facilitate and encourage such applications vii www.TechnicalBooksPDF.com AllChapters February 21, 2011 viii 9:40 World Scientific Book - 9in x 6in AllChapters Hands-on Matrix Algebra Using R The primary motivation for writing this book has been to make learning of matrix algebra fun by using modern computing tools in R I am assuming that the reader has very little knowledge of R and am providing some help with learning R However, teaching R is not the main purpose, since on-line free manuals are available I am providing some tips and hints which may be missed by some users of R For something to be fun, there needs to be a reward at the end of an effort There are many matrix algebra books for those purists who think learning matrix algebra is a reward in itself We take a broader view of a researcher who wants to learn matrix algebra as a tool for various applications in sciences and engineering Matrices are important in statistical data analysis An important reference for Statistics using matrix algebra is Rao (1973) This book should appeal to the new generation of students, “wired differently” with digitally nimble hands, willing to try difficult concepts, but less skilled with arithmetic manipulations I believe this generation may not have a great deal of patience with long tedious manipulations This book shows how they can readily create matrices of any size and satisfying any properties in R with random entries and then check if any alleged matrix theory result is plausible A fun example of Fibonacci numbers is used in Sec 17.1.3 to illustrate inaccuracies in floating point arithmetic of computers It should be appealing to the new generation, since many natural (biological) phenomena follow the pattern of these numbers, as they can readily check on Google This book caters to students and researchers who not wish to emphasize proofs of algebraic theorems Applied people often want to ‘see’ what a theorem does and what it might mean in the context of several examples, with a view to applying the theorem as a practical tool for simplifying or deeply understanding some data, or for solving some optimization or estimation problem For example, consider the familiar regression model y = Xβ + , (0.1) in matrix notation, where y is a T × vector, X is T × p matrix, β is a p × vector and ε is T × vector In statistics it is well known that b = (X X)−1 X y is the ordinary least squares (OLS) estimator minimizing error sum of squares ε ε It can be shown using some mathematical theorems that a deeper understanding of the X matrix of regressors in (0.1) is available provided one computes a ‘singular value decomposition’ (SVD) of the X matrix The www.TechnicalBooksPDF.com February 21, 2011 9:40 World Scientific Book - 9in x 6in Preface AllChapters ix theorems show that when a ‘singular value’ is close to zero, the matrix of regressors is ‘ill-conditioned’ and regression computations and statistical inference based on computed estimates are often unreliable See Vinod (2008a, Sec 1.9) for econometric examples and details The book does not shy away from mentioning applications making purely matrix algebraic concepts like the SVD alive I hope to provide a motivation for learning them as in Chapter 16 Section 16.8 in the same Chapter uses matrix algebra and R software to expose flaws in the popular Hodrick-Prescott filter, commonly used for smoothing macroeconomic time series to focus on underlying business cycles Since the flaw cannot be ‘seen’ without the matrix algebra used by Phillips (2010) and implemented in R, it should provide further motivation for learning both matrix algebra and R Even pure mathematicians are thrilled when their results come alive in R implementations and find interesting applications in different applied scientific fields Now I include some comments on the link between matrix algebra and computer software We want to use matrix algebra as a tool for a study of some information and data The available information can be seen in any number of forms These days a familiar form in which the information might appear is as a part of an ‘EXCEL’ workbook popular with practitioners who generally need to deal with mixtures of numerical and character values including names, dates, classification categories, alphanumeric codes, etc Unfortunately EXCEL is good as a starting point, but lacks the power of R Matrix algebra is a branch of mathematics and cannot allow fuzzy thinking involving mixed content Its theorems cannot apply to mixed objects without important qualifications Traditional matrices usually deal with purely numerical content In R traditional algebraic matrices are objects called ‘matrix,’ which are clearly distinguished from similar mixed objects needed by data analysts called ‘data frames.’ Certain algebraic operations on rows and columns can also make sense for data frames, while not others For example, the ‘summary’ function summarizes the nature of information in a column of data and is a very fundamental tool in R EXCEL workbooks can be directly read into R as data frame objects after some adjustments For example one needs to disallow spaces and certain symbols in column headings if a workbook is to become a data frame object Once in R as a data frame object, the entire power of R is at our disposal including superior plotting and deep numerical analysis with fast, reliable and powerful algorithms For a simple example, the reader www.TechnicalBooksPDF.com February 21, 2011 9:40 World Scientific Book - 9in x 6in Numerical Accuracy and QR Decomposition AllChapters 315 The QR decomposition plays an important role in many statistical techniques In particular it can be used to solve the equation Ax = b for given matrix A, and vector b It is useful for computing regression coefficients and in applying the Newton-Raphson algorithm # R program snippet 17.4.1 is next set.seed(921); X=matrix(sample(1:40,30),10,3) y=sample(3:13, 10) XtX=t(X) %*% X; #define XtX= X transpose X qrx=qr(XtX)#apply qr dcomposition to XtX Q=qr.Q(qrx);Q #Display Q matrix #verify that it is orthogonal inverse=transpose solve(Q)#this is inverse of Q t(Q) #this is transpose of Q R=qr.R(qrx); R #Display R matrix #Note that R is upper triangular Q %*% R #multiplication Q R #verify that QR equals the XtX matrix XtX #this matrix got qr decomposition above #apply qr to regression problem b=solve(t(X) %*% X) %*% (t(X) %*% y);b qr.solve(X, y, tol = 1e-10) Many regression programs use QR algorithm as default In snippet 17.4.1 we apply the QR decomposition to A = X X, where X contains regressor data in standard regression problem: y = Xβ + We need to use the function ‘qr.R’ to get the R matrix and ‘qr.Q’ to get the orthogonal matrix The snippet verifies that Q is indeed orthogonal by checking whether its transpose equals its inverse, Q = Q−1 , numerically The snippet also checks that the R matrix is upper triangular by visual inspection of the following abridged output > Q=qr.Q(qrx);Q #Display Q matrix [,1] [,2] [,3] [1,] -0.7201992 -0.02614646 -0.6932745 [2,] -0.4810996 -0.70116139 0.5262280 [3,] -0.4998563 0.71252303 0.4923968 > #verify that it is orthogonal inverse=transpose > solve(Q)#this is inverse of Q [,1] [,2] [,3] www.TechnicalBooksPDF.com February 21, 2011 316 9:40 World Scientific Book - 9in x 6in Hands-on Matrix Algebra Using R [1,] -0.72019916 -0.4810996 -0.4998563 [2,] -0.02614646 -0.7011614 0.7125230 [3,] -0.69327450 0.5262280 0.4923968 > t(Q) #this is transpose of Q [,1] [,2] [,3] [1,] -0.72019916 -0.4810996 -0.4998563 [2,] -0.02614646 -0.7011614 0.7125230 [3,] -0.69327450 0.5262280 0.4923968 > R=qr.R(qrx); R #Display R matrix [,1] [,2] [,3] [1,] -10342.97 -6873.9420 -7772.7825 [2,] 0.00 -971.1732 1831.2117 [3,] 0.00 0.0000 548.2268 > #Note that R is upper triangular > Q %*% R #multiplication Q R [,1] [,2] [,3] [1,] 7449 4976 5170 [2,] 4976 3988 2744 [3,] 5170 2744 5460 > #verify that QR equals the XtX matrix > XtX #this matrix got qr decomposition above [,1] [,2] [,3] [1,] 7449 4976 5170 [2,] 4976 3988 2744 [3,] 5170 2744 5460 > #apply qr to regression problem > b=solve(t(X) %*% X) %*% (t(X) %*% y);b [,1] [1,] -0.3529716 [2,] 0.5260656 [3,] 0.3405383 > qr.solve(X, y, tol = 1e-10) [1] -0.3529716 0.5260656 0.3405383 QR decomposition or factorization of A can be computed by using a sequence of ‘Householder reflections’ that successively reduce to zero all ‘below diagonal’ elements starting with the first column of A, proceeding sequentially to the remaining columns A Householder reflection (or Householder transformation) starts with an m×1 vector v and reflects it about www.TechnicalBooksPDF.com AllChapters ... available from the British Library HANDS- ON MATRIX ALGEBRA USING R Active and Motivated Learning with Applications Copyright © 2011 by World Scientific Publishing Co Pte Ltd All rights reserved This... into R and wait for the R prompt Only if the function is logically consistent, R will return the prompt If some errors are present, R will try to report to the screen those errors The first line... ‘summary’ on the data frame This book will also review results related to matrix algebra which are relevant for numerical analysts For example, inverting ill-conditioned sparse matrices and error