Numerical Linear Approximation in C © 2008 by Taylor & Francis Group, LLC CHAPMAN & HALL/CRC Numerical Analysis and Scientific Computing Aims and scope: Scientific computing and numerical analysis provide invaluable tools for the sciences and engineering This series aims to capture new developments and summarize state-of-the-art methods over the whole spectrum of these fields It will include a broad range of textbooks, monographs and handbooks Volumes in theory, including discretisation techniques, numerical algorithms, multiscale techniques, parallel and distributed algorithms, as well as applications of these methods in multidisciplinary fields, are welcome The inclusion of concrete real-world examples is highly encouraged This series is meant to appeal to students and researchers in mathematics, engineering and computational science Editors Choi-Hong Lai School of Computing and Mathematical Sciences University of Greenwich Frédéric Magoulès Applied Mathematics and Systems Laboratory Ecole Centrale Paris Editorial Advisory Board Mark Ainsworth Mathematics Department Strathclyde University Peter Jimack School of Computing University of Leeds Todd Arbogast Institute for Computational Engineering and Sciences The University of Texas at Austin Takashi Kako Department of Computer Science The University of Electro-Communications Craig C Douglas Computer Science Department University of Kentucky Ivan Graham Department of Mathematical Sciences University of Bath Peter Monk Department of Mathematical Sciences University of Delaware Francois-Xavier Roux ONERA Arthur E.P Veldman Institute of Mathematics and Computing Science University of Groningen Proposals for the series should be submitted to one of the series editors above or directly to: CRC Press, Taylor & Francis Group 24-25 Blades Court Deodar Road London SW15 2NU UK © 2008 by Taylor & Francis Group, LLC Numerical Linear Approximation in C Nabih N Abdelmalek, Ph.D William A Malek, M.Eng © 2008 by Taylor & Francis Group, LLC Chapman & Hall/CRC Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2008 by Taylor & Francis Group, LLC Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S Government works Printed in the United States of America on acid-free paper 10 International Standard Book Number-13: 978-1-58488-978-6 (Hardcover) This book contains information obtained from authentic and highly regarded sources Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use The Authors and Publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint Except as permitted under U.S Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers For permission to photocopy or use material electronically from this work, please access www copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400 CCC is a not-for-profit organization that provides licenses and registration for a variety of users For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe Library of Congress Cataloging-in-Publication Data Abdelmalek, Nabih N Numerical linear approximation in C / Nabih Abdelmalek and William A Malek p cm (CRC numerical analysis and scientific computing) Includes bibliographical references and index ISBN 978-1-58488-978-6 (alk paper) Chebyshev approximation Numerical analysis Approximation theory I Malek, William A II Title III Series QA297.A23 2008 518 dc22 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com © 2008 by Taylor & Francis Group, LLC 2008002447 v Contents List of figures xvii Preface xix Acknowledgements xxiii Warranties xxiii About the authors xxv PART Preliminaries and Tutorials Chapter 1.1 1.2 1.3 1.4 Applications of Linear Approximation Introduction Applications to social sciences and economics 1.2.1 Systolic blood pressure and age 1.2.2 Annual teacher salaries 1.2.3 Factors affecting survival of island species 1.2.4 Factors affecting fuel consumption 1.2.5 Examining factors affecting the mortality rate 1.2.6 Effects of forecasting 1.2.7 Factors affecting gross national products Applications to industry 10 1.3.1 Windmill generating electricity 10 1.3.2 A chemical process 11 Applications to digital images 12 1.4.1 Smoothing of random noise in digital images 12 1.4.2 Filtering of impulse noise in digital images 14 1.4.3 Applications to pattern classification 15 © 2008 by Taylor & Francis Group, LLC vi Numerical Linear Approximation in C 1.4.4 1.4.5 1.4.6 1.4.7 Chapter 2.1 2.2 2.3 2.4 2.5 2.6 3.2 3.3 3.4 3.5 3.6 3.7 16 18 19 20 Preliminaries Introduction Discrete linear approximation and solution of overdetermined linear equations Comparison between the L1, the L2 and the L∞ norms by a practical example 2.3.1 Some characteristics of the L1 and the Chebyshev approximations Error tolerances in the calculation Representation of vectors and matrices in C Outliers and dealing with them 2.6.1 Data editing and residual analysis Chapter 3.1 Restoring images with missing high-frequency components De-blurring digital images using the Ridge equation De-blurring images using truncated eigensystem De-blurring images using quadratic programming 25 26 30 32 32 33 34 36 Linear Programming and the Simplex Algorithm Introduction 3.1.1 Exceptional linear programming problems Notations and definitions The simplex algorithm 3.3.1 Initial basic feasible solution 3.3.2 Improving the initial basic feasible solution The simplex tableau The two-phase method 3.5.1 Phase 3.5.2 Phase 3.5.3 Detection of the exceptional cases Duality theory in linear programming 3.6.1 Fundamental properties of the dual problems 3.6.2 Dual problems with mixed constraints 3.6.3 The dual simplex algorithm Degeneracy in linear programming and its resolution 3.7.1 Degeneracy in the simplex method 3.7.2 Avoiding initial degeneracy in the simplex algorithm © 2008 by Taylor & Francis Group, LLC 39 41 44 45 47 48 51 55 55 56 56 57 59 60 61 62 62 63 Contents 3.8 3.9 3.7.3 Resolving degeneracy resulting from equal θmin 3.7.4 Resolving degeneracy in the dual simplex method Linear programming and linear approximation 3.8.1 Linear programming and the L1 approximation 3.8.2 Linear programming and Chebyshev approximation Stability of the solution in linear programming Chapter 4.1 4.2 4.3 4.4 4.5 4.6 4.7 vii 63 64 64 64 66 67 Efficient Solutions of Linear Equations Introduction 71 Vector and matrix norms and relevant theorems 72 4.2.1 Vector norms 72 4.2.2 Matrix norms 73 4.2.3 Hermitian matrices and vectors 73 4.2.4 Other matrix norms 76 4.2.5 Euclidean and the spectral matrix norms 77 4.2.6 Euclidean norm and the singular values 78 4.2.7 Eigenvalues and the singular values of the sum and the product of two matrices 79 4.2.8 Accuracy of the solution of linear equations 80 Elementary matrices 83 Gauss LU factorization with complete pivoting 84 4.4.1 Importance of pivoting 85 4.4.2 Using complete pivoting 86 4.4.3 Pivoting and the rank of matrix A 88 Orthogonal factorization methods 90 4.5.1 The elementary orthogonal matrix H 90 4.5.2 Householder’s QR factorization with pivoting 90 4.5.3 Pivoting in Householder’s method 92 4.5.4 Calculation of the matrix inverse A–1 93 Gauss-Jordan method 94 Rounding errors in arithmetic operations 95 4.7.1 Normalized floating-point representation 95 4.7.2 Overflow and underflow in arithmetic operations 96 4.7.3 Arithmetic operations in a d.p accumulator 97 4.7.4 Computation of the square root of a s.p number 100 4.7.5 Arithmetic operations in a s.p accumulator 100 4.7.6 Arithmetic operations with two d.p numbers 101 4.7.7 Extended simple s.p operations in a d.p accumulator 101 4.7.8 Alternative expressions for summations and innerproduct operations 104 © 2008 by Taylor & Francis Group, LLC viii Numerical Linear Approximation in C 4.7.9 4.7.10 4.7.11 4.7.12 4.7.13 More conservative error bounds D.p summations and inner-product operations Rounding error in matrix computation Forward and backward round-off error analysis Statistical error bounds and concluding remarks 104 105 106 107 109 PART The L1 Approximation Chapter 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 Introduction 5.1.1 Characterization of the L1 solution Linear programming formulation of the problem Description of the algorithm The dual simplex method Modification to the algorithm Occurrence of degeneracy A significant property of the L1 approximation Triangular decomposition of the basis matrix Arithmetic operations count Numerical results and comments DR_L1 LA_L1 DR_Lone LA_Lone Chapter 6.1 6.2 6.3 6.4 Linear L1 Approximation 113 116 116 118 120 124 126 129 130 132 134 140 145 165 170 One-Sided L1 Approximation Introduction 6.1.1 Applications of the algorithm 6.1.2 Characterization and uniqueness A special problem of a general constrained one Linear programming formulation of the problem Description of the algorithm 6.4.1 Obtaining an initial basic feasible solution 6.4.2 One-sided L1 solution from above © 2008 by Taylor & Francis Group, LLC 183 186 186 186 187 188 188 191 Contents 6.5 6.6 6.7 6.4.3 The interpolation property Numerical results and comments DR_Loneside LA_Loneside Chapter 7.1 7.2 7.3 7.4 7.5 7.6 7.7 8.2 8.3 8.4 8.5 8.6 8.7 191 191 195 203 L1 Approximation with Bounded Variables Introduction 7.1.1 Linear L1 approximation with non-negative parameters (NNL1) A special problem of a general constrained one Linear programming formulation of the problem 7.3.1 Properties of the matrix of constraints Description of the algorithm Numerical results and comments DR_Lonebv LA_Lonebv Chapter 8.1 ix 213 216 216 217 220 221 222 225 233 L1 Polygonal Approximation of Plane Curves Introduction 8.1.1 Two basic issues 8.1.2 Approaches for polygonal approximation 8.1.3 Other unique approaches 8.1.4 Criteria by which error norm is chosen 8.1.5 Direction of error measure 8.1.6 Comparison and stability of polygonal approximations 8.1.7 Applications of the algorithm The L1 approximation problem Description of the algorithm Linear programming technique 8.4.1 The algorithm using linear programming Numerical results and comments DR_L1pol LA_L1pol © 2008 by Taylor & Francis Group, LLC 245 245 246 248 249 249 250 251 251 252 253 255 255 261 267 914 Numerical Linear Approximation in C eLaRc LA_tmfuel_part_1 (int iout, int *pJin, int m, tMatrix_R c, tVector_R f, tVector_I icbs, tVector_I irbs, int *pIrank, int *pIter); void LA_tmfuel_marg_costs (int m, tMatrix_R c, tVector_R f, tVector_I icbs, tVector_I ibnd, tVector_I kbnd, tVector_R zc, int *pIrank); void LA_tmfuel_vleav (int *pIrank, int *pIout, int *pIvo, tVector_R f, tNumber_R *pXb); void LA_tmfuel_th_tu (int itf, int m, int ivo, int iout, int *pJout, tMatrix_R c, tVector_I icbs, tVector_I ibnd, tVector_I kbnd, tVector_R th, tVector_R tu, tVector_R zc); void LA_tmfuel_vent (int m, int *pJin, int ivo, int *pItest, tVector_R th, tVector_R tu); void LA_tmfuel_swap (int itf, int *pIrank, int *pJin, tMatrix_R c, tVector_I ibnd, tVector_I kbnd, tVector_I ib, tVector_R th, tVector_R tu, tVector_R zc); void LA_tmfuel_cascade (int iout, int jin, int *pJout, tNumber_R xb, tNumber_R *pPivot, tNumber_R pivoto, tMatrix_R c, tVector_R f, tVector_I ibnd, int *pIrank); void LA_tmfuel_test (int iout, int jin, int *pItest, tNumber_R *pXb, tNumber_R pivot, tVector_R f, tVector_I icbs, tVector_R th); void LA_tmfuel_gauss_jordn (int iout, int jin, int *pIrank, int m, tMatrix_R c, tVector_R f); eLaRc LA_tmfuel_res (int itf, int m, tVector_R f, tVector_I icbs, tVector_I ibnd, tVector_I kbnd, tVector_I ib, tVector_R zc, int *pIrank, tVector_R a, tNumber_R *pZnorm); eLaRc LA_Effort (int m, int n, tMatrix_R ct, tVector_R f, int *pIrank, int *pIter, tVector_R a, tNumber_R *pZ); void LA_effort_gauss_jordn (int iout, int jin, int m, int n, tNumber_R pivot, tMatrix_R ct, tVector_I ic, tVector_I nb); © 2008 by Taylor & Francis Group, LLC Appendix C: Constants, Types and Function Prototypes 915 eLaRc LA_effort_res (int m, int n, tMatrix_R ct, tVector_I ib, tVector_I ic, tVector_I ip, tVector_I nb, tVector_R zc, int *pIrank, tVector_R a, tNumber_R *pZ); void LA_effort_init (int m, int n, tMatrix_R ct, tVector_I ib, tVector_I ic, tVector_I ip, tVector_I nb); void LA_effort_marg_costs (int m, int n, tMatrix_R ct, tVector_R f, tVector_I ib, tVector_R zc); void LA_effort_vent (int *pIvo, int *pJin, int n, tVector_I nb, tVector_R zc, tNumber_R *pZ); void LA_effort_vleav (int *pItest, int jin, int *pIout, int m, tMatrix_R ct); eLaRc LA_Energy (int m, int n, tMatrix_R ct, tVector_R f, int *pIrank, int *pIter, tVector_R a, tNumber_R *pZ); void LA_energy_init (int m, int n, tMatrix_R ct, tVector_I ic, tVector_I ir, tVector_I ik, tVector_R a); eLaRc LA_energy_phase_1 (int m, int n, tMatrix_R ct, tVector_R f, tVector_I ic, tVector_R a, int *pIrank, int *pIter); void LA_energy_phase_2 (int m, int n, tMatrix_R ct, tVector_I ir, tVector_I ik, int *pIrank, tVector_R a); void LA_energy_norm (int m, tVector_R a, tNumber_R *pZ); void LA_energy_gauss_jordn_e0 (int iout, int m, int n, tMatrix_R ct, tVector_R f, tVector_R a); void LA_energy_gauss_jordn_e (int m, int n, int iout, int nj0, tMatrix_R ct, tVector_I ir, tVector_I ik, tVector_R a); void LA_energy_vent (int *pIvo, int *pJin, int m, int n, tVector_I ir, tVector_R a); void LA_energy_vleav (int *pItest, int jin, int m, int n, int *pIout, int j0, tMatrix_R ct, tVector_I ir, tVector_R a); © 2008 by Taylor & Francis Group, LLC 916 Numerical Linear Approximation in C void LA_energy_res (int m, int n, tMatrix_R ct, tVector_I ir, tVector_R a); #endif © 2008 by Taylor & Francis Group, LLC 917 Appendix D Utilities and Common Functions /* LA_Utils.h Linear Approximation Utility Prototypes -*/ #ifndef _LA_UTILS_H_ #define _LA_UTILS_H_ #include #include #include #include "LA_Defs.h" /* Function prototypes */ void LA_check_rank_def (int n, int irank); void void void void prn_la_rc prn_dr_bnr prn_algo_bnr prn_example_delim (eLaRc rc); (char *pszMsg); (char *pszMsg); (void); tVector_I void void void alloc_Vector_I swap_elems_Vector_I prn_Vector_I free_Vector_I (int nElems); (tVector_I v, int elemA, int elemB); (int *v, int nElems); (int *v); tVector_R void void void void void alloc_Vector_R swap_elems_Vector_R prn_Vector_R prn_Vector_R_nDec prn_Vector_R_exp free_Vector_R (int nElems); (tVector_R v, int elemA, int elemB); (tNumber_R *v, int nElems); (tNumber_R *v, int nElems, int nDec); (tNumber_R *v, int nElems); (tNumber_R *v); tMatrix_R alloc_Matrix_R (int nRows, int mCols); tMatrix_R init_Matrix_R (tNumber_R *c, int nRows, int mCols); void swap_rows_Matrix_R (tMatrix_R m, int elemA, int elemB); © 2008 by Taylor & Francis Group, LLC 918 Numerical Linear Approximation in C void void void prn_Matrix_R free_Matrix_R uninit_Matrix_R (tMatrix_R, int, int); (tMatrix_R, int nRows); (tNumber_R **m); #endif /* LA_Utils.c Linear Approximation Utility Functions -*/ #include /* malloc () */ #include /* memcpy () */ #include #include "LA_Utils.h" /* Interpretation of results functions -*/ void LA_check_rank_def(int n, int irank) { if (irank < n) { PRN ("System is rank deficient\n"); } } /* Print functions -*/ void prn_la_rc (eLaRc rc) { PRN(" RC = "); switch (rc) { case LaRcSolutionFound: PRN ("Solution Found"); case LaRcSolutionUnique: PRN ("Solution is Unique"); case LaRcSolutionProbNotUnique: © 2008 by Taylor & Francis Group, LLC break; break; Appendix D: Utilities and Common Functions PRN ("Solution is probably not Unique"); case LaRcSolutionDefNotUniqueRD: PRN ("Solution is definitely not Unique" " due to Rank deficiency"); case LaRcNoFeasibleSolution: PRN ("No Feasible Solution"); case LaRcInconsistentSystem: PRN ("Inconsistent System"); case LaRcInconsistentConstraints: PRN ("Inconsistent Constraints"); case LaRcErrBounds: PRN ("ERROR: Argument out of bounds"); case LaRcErrNullPtr: PRN ("ERROR: NULL pointer argument"); case LaRcErrAlloc: PRN ("ERROR: Memory allocation failure"); case LaRcOk: /* Should never return this */ default: PRN ("ERROR: Unknown"); 919 break; break; break; break; break; break; break; break; break; } PRN(" \n"); } void prn_dr_bnr (char *pszMsg) { PRN("\n=========================================================\n"); PRN("===== Executing Driver: %s", pszMsg); PRN("\n=========================================================\n"); } void prn_algo_bnr (char *pszMsg) { PRN("\n -\n"); PRN(" - Executing Algorithm: %s", pszMsg); PRN("\n -\n"); } void prn_example_delim (void) { PRN(" \n"); } © 2008 by Taylor & Francis Group, LLC 920 Numerical Linear Approximation in C /* Vector Functions -*/ /* Allocate an integer vector */ tVector_I alloc_Vector_I (int nElem) { int *v; if (nElem