matrix algorithms (vol 1 basic decompositions) stewart 1998 12 Cấu trúc dữ liệu và giải thuật

com Matrix Algorithms CuuDuongThanCong.com This page intentionally left blank CuuDuongThanCong.com Matrix Algorithms Volume I:Basic Decompositions G W Stewart University of Maryland College Park, Maryland siam Society for Industrial and Applied Mathematics Philadelphia CuuDuongThanCong.com Copyright ©1998 by the Society for Industrial and Applied Mathematics 1098765432 All rights reserved Printed in the United States of America No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher For information, write to the Society for Industrial and Applied Mathematics, 3600 University City Science Center, Philadelphia, PA 19104-2688 Library of Congress Cataloging-in-Publication Data Stewart, G W (Gilbert W.) Matrix algorithms / G W Stewart p cm Includes bibliographical references and index Contents: v Basic decompositions ISBN 0-89871-414-1 (v : pbk.) Matrices I Title QA188.S714 1998 512.9'434-dc21 98-22445 0-89871-414-1 (Volume I) 0-89871-418-4 (set) siam is a registered trademark CuuDuongThanCong.com CONTENTS Algorithms xiii Notation xv Preface xvii Matrices, Algebra, and Analysis Vectors 1.1 Scalars Real and complex numbers Sets and Minkowski sums 1.2 Vectors 1.3 Operations with vectors and scalars 1.4 Notes and references Representing vectors and scalars The scalar product Function spaces Matrices 2.1 Matrices 2.2 Some special matrices Familiar characters Patterned matrices 2.3 Operations with matrices The scalar-matrix product and the matrix sum The matrix product The transpose and symmetry The trace and the determinant 2.4 Submatrices and partitioning Submatrices Partitions Northwest indexing Partitioning and matrix operations Block forms 2.5 Some elementary constructions Inner products Outer products Linear combinations Column and row scaling Permuting rows and columns Undoing a permutation Crossing a matrix Extracting and inserting submatrices 2.6 LU decompositions 2.7 Homogeneous equations v CuuDuongThanCong.com 2 7 13 17 21 23 25 vi CONTENTS 2.8 Notes and references Indexing conventions Hyphens and other considerations Nomenclature for triangular matrices Complex symmetric matrices Determinants Partitioned matrices The LU decomposition Linear Algebra 3.1 Subspaces, linear independence, and bases Subspaces Linear independence Bases Dimension 3.2 Rank and nullity A full-rank factorization Rank and nullity 3.3 Nonsingularity and inverses Linear systems and nonsingularity Nonsingularity and inverses 3.4 Change of bases and linear transformations Change of basis Linear transformations and matrices 3.5 Notes and references Linear algebra Full-rank factorizations Analysis 4.1 Norms Componentwise inequalities and absolute values Vector norms Norms and convergence Matrix norms and consistency Operator norms Absolute norms Perturbations of the identity The Neumann series 4.2 Orthogonality and projections Orthogonality The QR factorization and orthonormal bases Orthogonal projections 4.3 The singular value decomposition Existence Uniqueness Unitary equivalence Weyl's theorem and the min-max characterization The perturbation of singular values Low-rank approximations 4.4 The spectral decomposition 4.5 Canonical angles and the CS decomposition Canonical angles between subspaces The CS decomposition 4.6 Notes and references Vector and matrix norms Inverses and the Neumann series The QR factorization Projections The singular value decomposition The spectral decomposition Canonical angles and the CS decomposition Addenda 5.1 Historical On the word matrix History 5.2 General references Linear algebra and matrix theory Classics of matrix CuuDuongThanCong.com 26 28 28 33 36 39 42 42 42 55 61 70 73 75 77 77 78 CONTENTS vii computations Textbooks Special topics Software Historical sources Matrices and Machines Pseudocode 1.1 Generalities 1.2 Control statements The if statement The for statement The while statement Leaving and iterating control statements The goto statement 1.3 Functions 1.4 Notes and references Programming languages Pseudocode Triangular Systems 2.1 The solution of a lower triangular system Existence of solutions The forward substitution algorithm Overwriting the right-hand side 2.2 Recursive derivation 2.3 A "new" algorithm 2.4 The transposed system 2.5 Bidiagonal matrices 2.6 Inversion of triangular matrices 2.7 Operation counts Bidiagonal systems Full triangular systems General observations on operations counts Inversion of a triangular matrix More observations on operation counts 2.8 BLAS for triangular systems 2.9 Notes and references Historical Recursion Operation counts Basic linear algebra subprograms (BLAS) Matrices in Memory 3.1 Memory, arrays, and matrices Memory Storage of arrays Strides 3.2 Matrices in memory Array references in matrix computations Optimization and the BLAS Economizing memory—Packed storage 3.3 Hierarchical memories Virtual memory and locality of reference Cache memory A model algorithm Row and column orientation Level-two BLAS Keeping data in registers Blocking and the level-three BLAS 3.4 Notes and references The storage of arrays Strides and interleaved memory The BLAS Virtual memory Cache memory Large memories and matrix problems Blocking CuuDuongThanCong.com 81 82 82 83 85 86 87 87 89 90 92 92 93 94 99 100 101 102 104 109 119 CONTENTS viii Rounding Error 4.1 Absolute and relative error Absolute error Relative error 4.2 Floating-point numbers and arithmetic Floating-point numbers The IEEE standard Rounding error Floating-point arithmetic 4.3 Computing a sum: Stability and condition A backward error analysis Backward stability Weak stability Condition numbers Reenter rounding error 4.4 Cancellation 4.5 Exponent exceptions Overflow Avoiding overflows Exceptions in the IEEE standard 4.6 Notes and references General references Relative error and precision Nomenclature for floating-point numbers The rounding unit Nonstandard floating-point arithmetic Backward rounding-error analysis Stability Condition numbers Cancellation Exponent exceptions Gaussian Elimination Gaussian Elimination 1.1 Four faces of Gaussian elimination Gauss's elimination Gaussian elimination and elementary row operations Gaussian elimination as a transformation to triangular form Gaussian elimination and the LU decomposition 1.2 Classical Gaussian elimination The algorithm Analysis of classical Gaussian elimination LU decompositions Block elimination Schur complements 1.3 Pivoting Gaussian elimination with pivoting Generalities on pivoting Gaussian elimination with partial pivoting 1.4 Variations on Gaussian elimination Sherman's march Pickett's charge Grout's method Advantages over classical Gaussian elimination 1.5 Linear systems, determinants, and inverses Solution of linear systems Determinants Matrix inversion 1.6 Notes and references Decompositions and matrix computations Classical Gaussian elimination Elementary matrix The LU decomposition Block LU decompositions and Schur complements Block algorithms and blocked algorithms Pivoting Exotic orders of elimination Gaussian elimination and its variants Matrix inversion Augmented matrices Gauss-Jordan elimination A Most Versatile Algorithm CuuDuongThanCong.com 121 121 124 129 136 138 141 147 148 148 153 165 169 174 180 185 CONTENTS 2.1 Positive definite matrices Positive definite matrices The Cholesky decomposition The Cholesky algorithm 2.2 Symmetric indefinite matrices 2.3 Hessenberg and tridiagonal matrices Structure and elimination Hessenberg matrices Tridiagonal matrices 2.4 Band matrices 2.5 Notes and references Positive definite matrices Symmetric indefinite systems Band matrices The Sensitivity of Linear Systems 3.1 Normwise bounds The basic perturbation theorem Normwise relative error and the condition number Perturbations of the right-hand side Artificial ill-conditioning 3.2 Componentwise bounds 3.3 Backward perturbation theory Normwise backward error bounds Componentwise backward error bounds 3.4 Iterative refinement 3.5 Notes and references General references Normwise perturbation bounds Artificial ill-conditioning Componentwise bounds Backward perturbation theory Iterative refinement The Effects of Rounding Error 4.1 Error analysis of triangular systems The results of the error analysis 4.2 The accuracy of the computed solutions The residual vector 4.3 Error analysis of Gaussian elimination The error analysis The condition of the triangular factors The solution of linear systems Matrix inversion 4.4 Pivoting and scaling On scaling and growth factors Partial and complete pivoting Matrices that not require pivoting Scaling 4.5 Iterative refinement A general analysis Double-precision computation of the residual Single-precision computation of the residual Assessment of iterative refinement 4.6 Notes and references General references Historical The error analyses Condition of CuuDuongThanCong.com ix 185 190 194 202 207 208 209 217 219 221 224 225 226 227 229 235 242 245 INDEX 444 of positive definite matrix, 187 properties, 16-17 devil's stairs, 380 diagonally dominant matrix, 157, 240 growth factor, 239-241,247 diagonal matrix, 10 row and column scaling, 22 diagonal permutation, 186 Dijkstra, E W., 86 dimension of a subspace, 32 distance to a subspace, see orthogonal projection Dixon, J D., 399 Dongarra, J J., 101 dot product, see inner product downdating, 327 2-norm of a vector, 347-348a, 356, 369 see also Cholesky downdating under QR updating Duff, I S., 147, 400 Duncan, W J., 354 Dwyer,P S., 27, 80,180 Eckart, C, 76 Edwards, D B G., 120 eigenpair, 71 eigenvalue, 71, 101 and 2-norm, 52 and determinant, 16 and nonsingularity, 38 and singular values, 71 Cauchy interlacing theorem, 72 iterative refinement, 225 min-max characterization, 72, 76 of 2x2 matrix, 396 of positive definite matrix, 186 perturbation theory, 72 similarity transformation, 41 eigenvector, 7J_ and singular vector, 71 EISPACK, 80 element, see matrix elementary Hermitian matrix, see Householder transformation elementary matrix, 181 lower triangular 152, 181 CuuDuongThanCong.com elementary reflector, see Householder transformation error absolute, 122, me relative, 122-124, me rounding, 127-128, me Euclidean norm, 15, 45, 85, 140a see also 2-norm, Frobenius norm exchange matrix, exponent exception, 138-140 coding around, 139-140 in IEEE standard, 140 overflow, 138 Clenshaw-Olver proposal, 144 underflow, 138 flush to zero, 139 gradual underflow, 139 exponential windowing, 355, 401, 412 forgetting factor, 401 factor analysis, 365 Faddeev, V N., 79 Faddeeva, D K., 79 fast Fourier transform, 101 Fierro, R D., 366 Fischer, E., 72, 76 floating-point arithmetic, 81,128-129 cancellation, 129, me elementary functions, 129 exponent exception, 138-140, me fl notation, 128, 143 general references, 141 guard digit, 143 IEEE standard, 129, me nonstandard, 142 square root, 129 standard bound, 128 floating-point number, 124-126 base, 125 characteristic, 142 exponent, 125 biased, 126 fraction, 141 general references, 141 IEEE standard, 126, me mantissa, 125, 142 normalization, 125 precision, 125 INDEX flop, 96,100 forgetting factor, see exponential windowing FORTRAN, 85, 86 storage of arrays, 103,115 forward rounding-error analysis, 130 Gaussian elimination, 245 forward substitution, 88a Fourier transform, 101 Fox, L., 79 Francis, J G F., 76 Frobenius, F G., 75 Frobenius norm, 75 absoluteness, 53 and 2-norm, 65 and singular values, 64 and trace, 49 consistency, 49 consistency with 2-norm, 52 of identity matrix, 52 of orthonormal matrix, 57 unitary invariance, 57 full-rank factorization, 21,34,42, 357 from singular value decomposition, 64 full-rank matrix, 37 and linear systems, 37 fundamental subspaces, see singular value decomposition Furnival, G., 354 Caches, J., 225 GAMS (Guide to Available Mathematical Software), 80 Gantmacher, F R., 79 gap in singular values, 359 362 366,401 revealed by pivoted QLP decomposition, 380-381, 387 revealed by pivoted QR decomposition, 373-375, 386 revealed by ULV decomposition, 413 revealed by URV decomposition, 403 Gauss, C F, 27, 78,100,181, 207, 289, 353 and least squares, 323-324 Gauss-Jordan elimination, 184, 332, 354 Gauss-Markov theorem, 324 CuuDuongThanCong.com 445 Gaussian elimination, 25, 27-28, 78, 100, 113, 147,148^453, 181-184 after kih step, 158 and elementary lower triangular matrices, 152 and elementary row operations, 150 and least squares, 324 and Schur complements, 150 as computing LU decomposition, 152-153 as elimination in linear system, 148-150 as transformation to triangular form, 151-152,249 augmented matrices, 184 backward rounding-error analysis, 229-231,245 block elimination, 160-163a, 191 after k steps, 162 operation count, 161 successful completion, 162 classical, 153-165a, 181, 183-184, 376 compared with Householder triangularization, 259 comparison of variants, 173 condition of triangular factors, 231-232,246 conditions for completion, 156 Grout's method, 117,172-173 a, 183-184 exotic elimination orders, 183, 208 growth factor, 230, me Hessenberg matrix, 195-197a, 237, 247 in constrained least squares, 315 partial pivoting, 168a patterned matrices, 195 Pickett's charge, 111-172 a, 183-184 pivoting, 172 pivoted elimination, 166a after k steps, 166 and LU decomposition, 166 rank determination, 167 rectangular matrices, 157 Sherman's march, 169-17la, I83-J84 tridiagonal matrix, 237, 247 INDEX 446 see also Cholesky decomposition, growth factor, LU decomposition, pivoting, Schur complement generalized eigenvalue problem, 367 generalized inverse, 289 generalized singular value decomposition, 367 Gentleman, W M., 291 Geuder, J C., 246 Gill, P E., 325 Givens, W., 143, 245, 271, 290 Givens rotation, see plane rotation Goldberg, D., 141, 144 Goldstine, H H., 143, 245 Golub, G H., 76, 77, 79,100, 292, 296, 324, 325, 326, 354, 355, 385, 386 graded matrix, 266 and plane rotation, 290 Householder triangularization, 264-267,290, 320 in rank determination, 366 plane rotation, 276 Gragg, W B., 292, 354, 400 Gram, J P., 76, 289 Gram-Schmidt algorithm, 252, 292, 339 and Krylov sequences, 292 and least squares, 324 classical 59 277-278 a loss of orthogonality, 281-284 modified, 278-280 a, 292 and Householder triangularization, 280, 292, 354 rounding-error analysis, 280-281, 292, 324 weak stability, 144 reorthogonalization, 284-288,292 twice-is-enough algorithm, 288, 292 Greub, W H., 79 Grimes, R G., 400 growth factor, 230, 236, 246-247 complete pivoting, 238, 247 diagonally dominant matrix, 239-241,247 Hessenberg matrix, 237, 247 partial pivoting, 236-238,246 CuuDuongThanCong.com positive definite matrix, 239,247 totally positive matrix, 241,247 tridiagonal matrix, 237, 247 see also pivoting Guttman, L., 354 Hager, W W., 400 Halmos, P R., 79 Hammarling, S., 291 Hanson, R J., 288, 326, 386 Haynsworth, E V., 182 Hermitian matrix, 15 Hessenberg matrix, JU, 150 growth factor, 237,247 Hessenberg system, 198a reduction by Gaussian elimination, 195-197a triangularization by plane rotations, 274-275a Hessian matrix, 207 hierarchical memory, 81,109-110 cache, 112.120 blocks, 112 miss, 112,115 write back, 112 write through, 112, 116 locality of reference in space, 110, 113 in time, 112 orientation of algorithms, 113-115 virtual memory, 110, 120 page, 110 page fault, 110, 115 Higham, D J., 246, 247 Higham, N J., 141, 182,184,225,246, 247, 290, 399, 400 Hoffman, W., 292 homogeneous system, 25 Hopcroft, J E., 100 Horn, R A., 79 Hosoda, Y., 387 Hotelling, H., 77, 245, 365 Householder, A S., 7, 27, 79,180, 181, 289, 296, 324 Householder transformation, 59,152, 254-258 a 289-290 elementary Hermitian matrix, 289 elementary reflector, 255, 289 INDEX hyperbolic, 355 introducing zeros, 256 multiplication by, 255 storage of, 255 Householder triangularization, 101, 258-260a, 290, 324 and the modified Gram-Schmidt algorithm, 280, 292, 354 backward rounding-error analysis, 261-264,290 blocked algorithm, 267-270a, 290 compared with Gaussian elimination, 259 computing projections, 260a computing Q-factors, 260 graded matrices, 264-267,290, 320 in null-space method, 313 orthogonality of computed Q-factor, 264 pivoting, 260 identity matrix, 9, 14 Frobenius norm, 52 operator norm, 50 perturbation theory, 53-54 componentwise bound, 55 IEEE floating-point standard, 126,141 exponent exceptions, 140 NaN (not a number), 141 rounding, 129 rounding unit, 128 ill-posed problem, 357, 365, 387 inferior subspace, see singular value decomposition oo-norm matrix oo-norm, 51_ absoluteness, 53 vector oo-norm, 44 inner product, 7, 2J_ interleaved memory, 119 inverse matrix, 38, 38 avoiding computation of, 180, 184 determinant of, 16 from LU decomposition, 177-178a, 184 left inverse, 40, me norm estimator, 389-390,391-393a, 394-396a CuuDuongThanCong.com 447 of orthogonal matrix, 57 perturbation theory, 54, 76 rounding-error analysis, 234-235, 246 transpose, 38 triangular matrix, 93,94a, 100 updating, 328-330 numerical difficulties, 329-330 via sweep operator, 331 invert-and-multiply algorithm, 39, 176, 234-235,246, 332 compared with LU solution of linear systems, 178 iterative refinement, 221-223,225, 242-244,247 convergence, 222 double-precision residual, 244 eigenvalue problems, 225 general algorithm, 222 least squares, 320-323, 326 nonlinear functions, 223, 225 seminormal equations, 304 single-precision residual, 244 Jacobi, C G J., 28, 78, 181, 290 Johnson, C R., 79 Jordan, C., 76, 77, 78 Kahan, W., 77, 292, 386 K effect, 325 Kato, T., 79 Kaufman, L., 208, 292, 354 Kayyam, O., 326 Kilburn, T., 120 Kline, M., 78, 80 Knuth, D E., 86, 141 Kowalewski, G., 292 Kronecker, L., 78 L-values, see pivoted QLP decomposition Lagrange, J.-L., 27, 78, 207 Lanigan, M J., 120 LAPACK, 80 1-norm estimator, 387 2x2 eigenvalue problem, 396 choice of block size, 119 column orientation, 115 pivoted orthogonal triangularization, 385 INDEX 448 storage of band matrices, 208 symmetric indefinite system, 208 Laplace, P S., 76, 323 Lawson, C L., 288, 326, 386 leading submatrix, j/7 least squares, 100, 144, 181, 292 assessment of computational methods, 311 augmented cross-product matrix, 301 augmented least squares matrix, 297, 324 QR decomposition, 297 augmented matrix, 326 constrained, see constrained least squares cross-product matrix, 299, me Golub-Householder algorithm, 296 Hessenberg matrix, 296a historical, 323-324 iterative refinement, 320-323, 326 K2 effect, 309 least squares approximation, 294 normal equations, 298-299 accuracy, 310-311 backward rounding-error analysis, 305 formation, 299-300a improving condition, 324 in weighting method, 317 perturbation theory, 306-307 perturbation theory, 308-309,325 QR equation, 299 residual sum of squares, 294 residual system, 321-322a, 326 rounding-error analysis, 322 residual vector, 294 effect on perturbation bound, 309 seminormal equations, 304, 325 sparse, 300 statisticians' notation, 249 subset of columns, 337, 354 sweep operator, 332 updating, 353-354 via Householder triangularization, 324 accuracy, 310-311 backward stability, 305, 325 via modified Gram-Schmidt CuuDuongThanCong.com algorithm, 297-298a accuracy, 310-311 backward stability, 305 via QR decomposition, 293-296a via QR factorization, 296-297 left inverse, 40 left rotation, see plane rotation Legendre, A M., 323, 353 Lewis, J G., 400 linear algebra, 1, 28 general references, 79 linear combination, matrix representation, 21-22 of independent vectors, 30 linear independence, 29 and nonsingularity, 38 matrix characterization, 30 uniqueness of linear combinations, 30 linear space, linear system and nonsingularity, 38 artificial ill-conditioning, see condition backward componentwise stability, 244, 247 backward error analysis, 232-233 backward perturbation theory, 219-221,225 componentwise, 220 normwise, 219 band matrix, 202-207a, 208 bidiagonal, 93a existence of solution, 36 full rank, 37 Hessenberg, 198 invert-and-multiply algorithm, 39, me iterative refinement, 242-244 matrix representation, more than one right-hand side, 176 nonuniqueness of solution, 35 perturbation theory, 224 componentwise bounds, 217-219, 224, 390-391 individual components, 218, 225 normwise bounds, 209-210,224 right-hand side, 213 positive definite tridiagonal, INDEX 200-202a residual system, 321-322a, 326 residual vector, 233 solution via augmented matrix, 184 solution via LU decomposition, 147, 174-176a transposed system, 175a symmetric indefinite, see symmetric indefinite matrix transposed system, 147 triangular, 88a, 90, 100 accuracy of solutions, 227-228 backward error analysis, 226-227 transposed, 92 tridiagonal, 198-200a underdetermined, 25 homogeneous, 25 least norm solution, 314 uniqueness of solution, 25 updating, 328-329a see also bidiagonal matrix, positive definite matrix, triangular matrix, etc linear transformation between subspaces, 41-42 change of basis, 41 matrix representation, 41 representation with respect to a basis, 41 UNPACK, 80, 355 BLAS, 120 column orientation, 115 iterative refinement, 247 norm and condition estimation, 391-397, 400 pivoted Cholesky decomposition, 386 pivoted orthogonal triangularization, 385 QR updating, 354 storage of band matrices, 208 symmetric indefinite system, 183, 208 low-rank approximation, 357 from pivoted QLP decomposition, 384-385 from pivoted QR decomposition, 372 from the singular value decomposition, 69-70, 76, 357 CuuDuongThanCong.com 449 in URV updating, 411^12 LU decomposition, 23-25, 27-28,34, 63, 90, 181 and determinants, 176-177 and pivoted Gaussian elimination, 166 and Schur complement, 153,181 block, 156,161,163,181,191 symmetric indefinite matrix, 190-194 computation of inverse matrix, 177-178a computed by Gaussian elimination, 152 condition of triangular factors, 231-232,246 existence, 157,159 normalization, 159 perturbation theory, 246 pivoted, 24,42 solution of linear systems, 174-176a transposed system, 175a uniqueness, 159 see also Gaussian elimination, Schur complement LU factorization, see LU decomposition M-matrix, 157 Mac Duffee, C C., 80 Marcus, M., 79 Markov, A A., 324 Martin, R S., 290 Mathias, R., 366 MATLAB, 80 matrix, absolute value, 43 and bilinear form, 78 basis for M m x n , 31 column index, column space, 34, me componentwise inequalities, 43 conjugate transpose, 15 contrasted with array, element, entomology, 77 general references, 79 history, 77-78 indexing conventions, 26 INDEX 450 inverse matrix, 38, me matrix-vector product, 14 nonnegative, 43 nonsingularity, 37, me notational conventions, 8-9 nullity, 35, me null space, 35, me order of a square matrix, positive, 43 product, 14 determinant of, 16 noncommutativity, 14 transpose, 15 rank, 34, me representation of linear systems, row index, row space, 34, me scalar-matrix product, 13_ sum, J3 transpose, 15 transpose, 14 of sum and product, 15 see also square matrix, identity matrix, triangular matrix, partitioned matrix, submatrix, etc matrix decomposition, 78, 147 and matrix computations, 147, 180 matrix norm, 48, 75 1-norm, 51, me 2-norm, 52, me absolute norm, 52-53, me consistency, 48-49 of Frobenius norm, 49 of operator norms, 50 consistent vector norm, 50 family of norms, 50 Frobenius norm, 49, me oo-norm, 51, me operator norm, 50 spectral norm, see 2-norm McWhirter, J G., 326 measurement error models, 365 memory, see computer memory, hierarchical memory method of elimination, see constrained least squares method of mixed rotation, see QR CuuDuongThanCong.com updating, Cholesky downdating min-max characterization, see eigenvalue, singular value Mine, H., 79 Minkowski, H., 75 Minkowski operations, Mirsky, L., 76 modulus, see absolute value Moler, C B., 80, 120 Moonen, M., 355 Moore, E H., 289 Morrison, W J., 354 Muir, T, 78 Murray, W., 325 NaN (not a number), 141 Nash, S G., 325 natural basis, 367 371, 384 negative definite matrix, 186 NETLIB, 80 Neumann series, 54-55, 76 nonsingularity, 37,43 characterizations, 38 existence of inverse, 38 of positive definite matrix, 187 of Schur complements, 155 of triangular matrix, 87 norm, 1, 44, 75 1-norm, 44, 51, me 2-norm, 44, 52, me absolute norm, 52-53, me dual norms, 45 equivalence of norms, 46-47 family of norms, 50 Frobenius norm, 49, me Holder norms, 45 oo-norm,44, 51, me Manhattan norm (1-norm), 45 matrix norm, 48, me normalization, 44 on £[0,1], unitary invariance, 57 vector norm, 44 see also convergence, norm and condition estimation normal equations, see least squares, 100, 207 INDEX norm and condition estimation, 212,219, 399 1-norm estimator, 388-389a, 400 2-norm estimator, 391-393a, 397-399a, 400 approximate null vector, 390 componentwise bounds, 390-391, 399 condition estimation, 389-390 in UNPACK, 396-397 incremental condition estimation, 396,400 in UTV updating, 406,408,415 inverse matrix, 389-390 inverse triangular matrix 2-norm estimator, 394-396a LINPACK-style estimators, 391-397, 400 normwise relative error, see relative error northwest indexing, see partitioned matrix null-space method, see constrained least squares nullity, 35 and nonsingularity, 38 null space, 35, 365 null vector, 35 Oettli, W., 220, 225 Oettli-Prager theorem, 220 Olver, F W J., 145 one-norm, see 1-norm operation count, 81.94-98, 100-101 1-norm estimator, 389 2-norm estimator, 393, 396, 398 banded system, 206 bidiagonal system, 95 big O notation, 95,101 blocked Householder triangularization, 269 by integration, 97 Cholesky algorithm, 190 Cholesky downdating, 346 comparing algorithms, 98 complex arithmetic, 96 complexity of matrix algorithms, JOO cross-product matrix, 300 dominant term, 95 flop, 96,100 CuuDuongThanCong.com 451 Gaussian elimination, 154 Gram-Schmidt algorithm, 278 Hessenberg reduction, 196, 275 Hessenberg system, 198 Householder triangularization, 259 inverse of triangular matrix, 97 limitations, 98, 105 linear systems via LU decomposition, 176 lower bounds on execution time, 98 matrix inversion, 178 method of areas, 336 method of elimination, 317 method of integrals, 100 multiplication by Householder transformation, 255 nomenclature, 96, 100 null-space method, 313 order, 95 order constant, 95, 107 order n, n , n3 compared, 97 orthogonal projection, 260 pivoted QLP decomposition, 380 positive definite tridiagonal system, 201 Q-factors, 261 QR updating appending columns, 339 appending rows, 340 interchange of columns, 336 rank-one update, 349 removing columns, 338 removing rows from a decomposition, 342 removing rows from a factorization, 344 reduction of a symmetric indefinite matrix, 194 sweep operator, 331 triangular system, 95 tridiagonal system, 200 ULV updating, 416 URV refinement, 411 URV updating, 408 UTU form, 268 weighting method, 320 operator norm, see matrix norm order constant, see operation count INDEX 452 order of a square matrix, orientation of algorithms, 183 and hierarchical memories, 113-115 axpy algorithm, 91 forward substitution, 91 Hessenberg reduction, 197, 275 orthogonal complement, 36, 59 orthogonal matrix, 56, 238 inverse, 57 role in matrix computations, 57 with specified initial columns, 59 orthogonal projection, 60 and pseudoinverse, 252 as Hermitian idempotent matrix, 61 distance to a subspace, 61 from Householder triangularization, 260a from QR decomposition, 251, 253 from QR factorization, 61 orthogonal triangularization, see Householder triangularization, pivoted orthogonal triangularization, plane rotation orthogonal vectors, 56 orthonormal matrix, 56 norm of, 57 orthonormal vectors, 56 Ouellette, D V., 182 outer product, 2J_ overflow, see exponent exception overwriting in Gaussian elimination, 155 in linear systems, 176 matrix by its inverse, 94 of matrix by inverse, 179a right-hand side of a linear system, 88 packed storage, 108-109 and BLAS, 108 band matrix, 205 Cholesky algorithm, 189 Paige, C C., 77, 292, 324, 367 Park, H., 290, 291, 354 Parlett, B N., 101, 208,290, 292 partitioned matrix, 9,18, 27 block, 18 by columns, 18 by rows, 18 CuuDuongThanCong.com in pseudocode, 83 northwest indexing, 19, 27 Peano, G., 75 Penrose, R., 289 permutation matrix, 9, 22 permuting rows and columns, 22 perturbation theory, 81, 134 backward perturbation theory, see linear system componentwise perturbation theory, see linear system eigenvalue, 72 first order, 306 fundamental subspaces, 359-363, 365-366 identity matrix, 53-54 componentwise bound, 55 inverse matrix, 54, 76 least squares, 308-309, 325 linear system, see linear system, condition, condition number LU decomposition, 246 normal equations, 306-307 positive definite matrix, 302 pseudoinverse, 307-308 QR decomposition, 289 singular value decomposition, 69, 359-363, 365-366 sum of scalars, 134 Peters, G., 324 Pickett's charge, see Gaussian elimination Pinkus, A., 247 pivotal condensation, 182 pivoted Cholesky decomposition, 190, 375-378a, 386 see also pivoted QR decomposition pivoted orthogonal triangularization, 368-370a, 385 computation of column norms, 368 early termination, 370, 373 incompatibility with blocking, 385 pivoted QLP decomposition, 378-385a, 387 2-norm estimation, 397, 400 and ULV decomposition, 387 bases for fundamental subspaces, 381-384 accuracy, 382 INDEX natural basis, 384 early termination, 384-385,387 L-values, 379 low-rank approximation, 384-385 revealing gap in singular values, 380-381,387 pivoted QR decomposition, 42, 319, 368-375 assessment, 375 bases for fundamental subspaces, 370-373 accuracy, 373 natural basis, 371 properties of the R-factor, 370 R-values, 373 revealing gap in singular values, 373-375, 386 see also pivoted Cholesky decomposition pivoting, 160.165-166 182 and scaling, 235 Cholesky algorithm, 190 complete, 167, 182, 238, 247, 316 diagonal, 167 for size, 167 forsparsity, 167 for stability, 167 Gaussian elimination, 166a in Pickett's charge, 172 nomenclature, 168 order forced by scaling, 241, 247 partial, 167, 168a, 182,230, 236-238,246 failure, 247 pivot element, 155, 165 positive definite matrix, 239 symmetric indefinite system, 192-193,208 see also growth factor, Householder triangularization Plackett, R L., 324 plane rotation, 271, 290 application, 273a backward rounding-error analysis, 275, 290, 354 fast, 291 generation, 272a Givens rotation, 271, 290 CuuDuongThanCong.com 453 graded matrices, 276, 290 in (i, j)-plane, 271 introducing zeros, 271,273 left rotation, 402 preserving and destroying small element, 401^02 right rotation, 402 storage, 291 triangularization of Hessenberg matrix, 274-275a Plemmons, R J., 79 polar representation, see complex number positive definite matrix, 43,157,181.186 207 cross-product matrix, 186 determinant of, 187 diagonal permutation, 186 nonsingularity of, 187 nonsymmetric, 186 perturbations of, 302 pivoting, 239 positivity of eigenvalues, 186 principle submatrix, 186 Schur complements in, 187 square root of, 187 sweep operator, 332 tridiagonal system, 200-202a see also Cholesky algorithm, Cholesky decomposition Powell, M J D., 290 Prager, W., 220, 225 Pratt, T W., 86 principal component analysis, 365 principal submatrix, j_7 of positive definite matrix, 186 product, see vector, matrix projection, see orthogonal projection nonorthogonal, 76 pseudocode, 81, 82-85, 86 comments, 85 for statement, 84 functions and subprograms, 85 goto statement, 85, 86 if statement, 83 inconsistent dimensions, 83 inconsistent loops, 84, 88 leave statement, 85 INDEX 454 parameters passed by reference, 85, 106 statement label, 85 while statement, 84 pseudoinverse, 252,289 and orthogonal projections, 252 and singular value decomposition, 252 perturbation theory, 307-308 Pythagorean equality, 55, 56, 61, 295 Q-factor, see QR decomposition QLP decomposition, see pivoted QLP decomposition QR algorithm, 76 QR decomposition, 62, 250 and Cholesky decomposition, 251 and Schur complement, 253 and singular value decomposition, 253 of augmented least squares matrix, 297 orthogonal projections, 251,253 orthonormal bases, 251 partitioned, 253 perturbation theory, 289 Q-factor, 251,264 R-factor, 251 via cross-product matrix, 251 see also Gram-Schmidt algorithm, Householder triangularization, least squares, pivoted QR decomposition, QR updating QR equation, 294 QR factorization, 57, 76 and Gram-Schmidt algorithm, 59 and orthonormal bases, 59 see also QR decomposition QR updating, 327, 354 a general approach, 333-334 appending columns, 338-339a appending rows, 339-340a backward stability, 351-352, 354 block, 340 Cholesky downdating, 345-346a, 355 hyperbolic rotations, 355 method of mixed rotations, 346, 355 CuuDuongThanCong.com numerical properties, 352-353 relational stability, 352 Saunder's method, 355 Cholesky updating, 327, 340 downdating the vector 2-norm, 347-348a, 356, 369 exponential windowing, 355 interchanging columns, 334-337a contiguous, 336 least squares, 337 rank-one update, 348-349 decomposition, 349 factorization, 348-349a removing a column, 337-338a removing rows from a decomposition, 342-343a from a factorization, 343-344a quaternion, 77 R-factor, see QR decomposition R-values, see pivoted QR decomposition Rabideau, D J., 416 rank, 34 and nonsingularity, 38 and singular values, 64 degeneracy, 37 rank determination, 42, 357, 365, 366 by Gaussian elimination, 167 drawbacks of the singular value decomposition, 367 error bounds for fundamental subspaces, 364-365 from the singular value decomposition, 363-364 scaling, 366-367 whitening noise, 366 see also gap in singular values real n-space (R"), unit vector basis, 31 recursion, 100 in deriving algorithms, 89-90 LU decomposition, 90 seldom used in matrix algorithms, 90 register management, 116 Reid, J K., 208, 247, 290 Reinsch, C, 290 relational stability, 352 relative error, 122-124 INDEX and rounding error, 127 and significant digits, 124, 141,210 norm wise, 210 reciprocity, 123, 209 residual system, see least squares Rew, R K., 400 Rice, J R., 292 Rigal, J L., 225 right rotation, see plane rotation rounding-error analysis, 81,128 cross-product matrix, 301 forward error analysis, 130 general references, 141 inverse matrix, 234-235,246 modified Gram-Schmidt algorithm, 280-281,292,324 residual system, 322 see also backward rounding-error analysis rounding-error bounds assessment, 136 pessimism of bounds, 136 slow growth, 135 rounding error, 127-128 adjusted rounding unit, 131, 143 chopping, 128 effects on a sum, 136 equivalence of forward substitution and the axpy algorithm, 91 first-order bounds, 131 fl notation, 127, 143 general references, 141 inevitability, 121 in hexadecimal arithmetic, 144 rigorous bounds, 131 rounding unit, 128, 142 approximation of, 142 slow accumulation, 135 truncation, 128 varieties of rounding, 127 rounding unit, see rounding error row-sum norm, see oo-norm row index, see matrix row orientation, see orientation of algorithms row space, 34, 365 row vector, CuuDuongThanCong.com 455 Saunders, M A., 77, 325, 355, 367 scalar, as a 1-vector or a x matrix, notational conventions, 2, scalar-matrix product, see matrix scalar-vector product, see vector scalar product, see inner product scaling, 247 and condition, 216-217,224 and Gaussian elimination, 241-242 and pivoting, 235 approximate balancing, 247 equal error, 216-217,242, 247 for minimum condition, 247 in rank determination, 366-367 whitening noise, 366 scaling a matrix, 22 Schmidt, E., 76, 289 Schmidt-Mirsky theorem, 69 Schreiber, R S., 247,290 Schur,J., 27, 181,289 Schur complement, 150, 155 and block LU decomposition, 163 and Gaussian elimination, 150 and LU decomposition, 153, 181 and QR decomposition, 253 generated by k steps of Gaussian elimination, 158 in positive definite matrix, 187 nested, 164 nonsingularity, 155 via sweep operator, 331 see also Gaussian elimination, LU decomposition semidefinite matrix, 186 seminormal equations, see least squares corrected, 305 a Sheffield, C., 292 Shepherd, T J., 326 Sherman's march, see Gaussian elimination Sherman, J., 354 signal processing, 357, 365, 400, 4J6 signal and noise subspaces, 365 significant digits and relative error, 124 similarity transformation, 41 singular matrix, 37 INDEX 456 singular value, 63 2-norm, 64 and eigenvalue, 71 and Frobenius norm, 64 and rank, 64 gap in singular values, 362, me invariance under unitary equivalence, 66 min-max characterization, 68 perturbation theory, 69, 359-363, 365-366 uniqueness, 65-66 Weyl's theorem, 67 singular value decomposition, 42,62, 76 and 2-norm, 52 and determinant, 16 and nonsingularity, 38 and pseudoinverse, 252 and QR decomposition, 253 and the spectral decomposition, 71 and unitary equivalence, 66 approximate updating, 416 block triangular matrix, 362 drawbacks, 367,401 full-rank factorization, 64 fundamental subspaces, 358-359, 365 error bounds, 364-365 from pivoted QLP decomposition, 381-384 from pivoted QR decomposition, 370-373 from URV decomposition, 403 perturbation theory, 359-363, 365-366 signal and noise subspaces, 365 gap in singular values, 362, me inferior subspace, 358-359, 365 low-rank approximation, 69-70, me perturbation theory, 359-363, 365-366 Schmidt-Mirsky theorem, 69 singular subspace, 358, 365 singular value, 63, me singular value factorization, 64 singular vector, 63 and eigenvectors, 71 superior subspace, 358-359, 365 CuuDuongThanCong.com uniqueness, 65-66 singular value factorization, see singular value decomposition Skeel, R D., 217, 224 skew Hermitian matrix, 15 skew symmetric matrix, 15 Sofer, 325 software, see LAPACK, UNPACK, MATLAB span of a set of vectors, 29 sparse matrix, 167 least squares, 300 spectral decomposition, 70-72, 76, 77 and the singular value decomposition, 71 in solution of symmetric indefinite systems, 187,208 uniqueness, 71 spectral norm, see 2-norm square matrix, square root of positive definite matrix, 187 stability, see backward stability, relational stability, weak stability Sterbenz, P H., 141 Stewart, G W., 77, 79, 224, 291, 292, 354, 355, 366, 386, 400, 416 Stigler, S M., 324 storage of arrays, 102-104 by blocks, 119 by columns, 103 by rows, 102 column major order, 103 in C, 103 in FORTRAN, 103, 115 lexicographical order, 103 packed storage, 108-109 and BLAS, 108 row major order, 103 stride, 103,107 interleaved memory, 119 stride, see storage of arrays submatrix, j/7 extracting and inserting, 23 subspace, 28 canonical angles between subspaces, 73, me canonical orthonormal bases for two subspaces, 73_ INDEX dimension, 32, me direct sum, 29 disjoint subspace, 29 intersection and sum, 28-29 orthonormal basis, 59, 60 sum, see vector, matrix Sumner, F H., 120 Sun, J.-G., 79, 224 superior subspace, see singular value decomposition sweep operator, 330-333 a, 354 inverse matrix, 331 least squares, 332 positive definite matrix, 332 Schur complement, 331 stability, 332 Sylvester, J J., 77 symmetric indefinite matrix, 208 block LU decomposition, 190-194 complete diagonal pivoting, 192, 208 partial diagonal pivoting, 192, 208 reduction to tridiagonal form, 208 symmetric matrix, 15 packed storage, 109 total least squares, 365 totally positive matrix, 157, 241 growth factor, 241, 247 trace, 16 and Frobenius norm, 49 trailing submatrix, 17 transpose, see matrix trapezoidal matrix, H independence of columns, 30 Trefethen, L N., 79, 247 triangle inequality, 43, 44 triangular matrix accuracy of solutions, 227-228 axpy algorithm, 90-91 back substitution algorithm, 92 backward rounding-error analysis, 226-227 BLAS, 99 forward substitution algorithm, 88a independence of columns, 30 inverse, 93, 94a, 100 lower, H nomenclature, 26 CuuDuongThanCong.com 457 nonsingularity, 87 orientation of algorithms, 91 packed storage, 108 strict, 11 transposed system, 92 triangular system, 100 unit, 11 upper, 10 tridiagonal matrix, 12, 150 growth factor, 237,247 linear system, 198-200a packed storage, 109 positive definite system, 200-202a storage of, 200 Turing, A M., 143, 144, 224,245 Turnbull, H W., 80, 289 two-norm, see 2-norm two-sided orthogonal decomposition, 416 Ullman, J D., 100 ULV decomposition, see UTV decomposition underdetermined system, see linear system minimum norm solution, 325 underflow, see exponent exception unitary equivalence, 66 unitary matrix, see orthogonal matrix unit vector, basis for M n , 31,39 updating, 326 inverse matrix, 328-330 numerical difficulties, 329-330 least squares, 353-354 linear programming, 354 linear system, 328-329a optimization, 354 sweep operator, 330-333, me URV decomposition, see UTV decomposition UTU form, 267-268a UTV decomposition, 401, 416 downdating, 416 gap in singular values, 401 ULV decomposition, 401,413 and pivoted QLP decomposition, 387 gap revealing, 413 updating, 413-416 INDEX 458 URV decomposition, 401,403 deflation, 406-408 effects of rounding error, 408 gap adjustment, 406 gap revealing, 403 incorporation, 403-406 low-rank splitting, 411-412,416 refinement, 409-411 a updating, 402-409 updating algorithm, 408^409a Van Dooren, P., 355 Van Loan, C R, 76, 79, 100, 290, 326, 367, 400 Varga, R., 79 vector, angle between vectors, 56, me as an n x matrix, component, dimension, inner product, 21, me matrix-vector product, 14 normalization, 44 notational conventions, 4, orthogonal vectors, 56, me orthonormal vectors, 56, me outer product, 21, me scalar-vector product, 5, sum, see also unit vector, zero vector, etc vector analysis, 78 vector space, 1,6,78 £[0,1], 6, complexro-space(Cn), 4, me function space, infinite dimensional, 6, 46 real, real n-space (M"), 4, me subspace, 28, me the space of m x n matrices, 13 Veselic, K., 290 virtual memory, see hierarchical memory void matrix, in pseudocode, 83 von Neumann, J., 143,245 Watkins, D S., 79 weak stability, 133, 144,245, 324, 325 corrected seminormal equations, 304 CuuDuongThanCong.com Wedderburn, J H M., 80 Wedin, R-A., 325, 366 Wei, M., 77 Weierstrass, K., 78 Weierstrass approximation theorem, weighting method, see constrained least squares Weyl's theorem, 67 Weyl, H., 72, 76 whitening noise, 366 Wiener, N., 75 Wilkes, M., 120 Wilkinson,! H., 79, 141,143,180,182, 219, 224, 225, 225,237, 245, 246, 247, 262,290, 292, 324, 325, 326 Wilkinson diagram, K) Wilson, R., 354 Woodbury's formula, 328, 354 numerical difficulties, 329-330, 351, 354 Woodbury, M A., 328, 354 Wright, M H., 325 Yoo, K., 354 Young, G., 76 Zelkowitz, M V, 86 zero matrix, zero vector, Zielke, G., 354 ... Contents: v Basic decompositions ISBN 0-8 987 1-4 1 4-1 (v : pbk.) Matrices I Title QA188.S714 1998 512.9'434-dc21 9 8-2 2445 0-8 987 1-4 1 4-1 (Volume I) 0-8 987 1-4 1 8-4 (set) siam is a registered trademark... 35 35 38 38 38 39 43 43 44 NOTATION xvi The vector 1-, 2-, and oo-norms 44 A matrix norm 48 The Frobenius norm 49 The matrix 1-, 2-, and oo-norms 51 The angle between x and y 56 The vector x... for Xj in the form Xj = a-l Y^i^j