Bonnans j f et al numerical optimization theoretical and practical aspects (2ed universitext 2006)(ISBN 354035445x)(488s)

J Fr´ed´eric Bonnans · J Charles Gilbert Claude Lemar´echal · Claudia A Sagastiz´abal Numerical Optimization Theoretical and Practical Aspects Second Edition With 52 Figures J Fr´ed´eric Bonnans J Charles Gilbert Centre de Math´ematiques Appliqu´ees Ecole Polytechnique 91128 Palaiseau France e-mail: Frederic.Bonnans@inria.fr INRIA Rocquencourt BP 105 78153 Le Chesnay France e-mail: Jean-Charles.Gilbert@inria.fr Claude Lemar´echal Claudia A Sagastiz´abal INRIA Rhˆone-Alpes 655, avenue de I’Europe Montbonnot 38334 Saint Ismier France e-mail: Claude.Lemarechal@inria.fr On leave from INRIA Rocquencourt Correspondence to: IMPA 110, Estrada dona Castorina 22460-320 Jardim Botˆanico Rio de Janeiro–RJ Brazil e-mail: sagastiz@impa.br Original French edition “Optimisation Num´erique” was published by Springer-Verlag Berlin Heidelberg, 1997 Mathematics Subject Classification (2000): 65K10, 90-08, 90-01, 90CXX Library of Congress Control Number: 2006930998 ISBN: 3-540-35445-X Springer-Verlag Berlin Heidelberg New York This work is subject to copyright All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer Violations are liable for prosecution under the German Copyright Law Springer-Verlag Berlin Heidelberg New York a member of Bertelsmann Springer Science+Bussiness Media GmbH springer.com c Springer-Verlag Berlin Heidelberg 2006 The use of general descriptive names, registered names, trademarks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use Cover design: Erich Kirchner, Heidelberg Typesetting by the authors using a LATEX macro package Printed on acid-free paper: SPIN: 11777410 41/2141/SPi - Preface This book is entirely devoted to numerical algorithms for optimization, their theoretical foundations and convergence properties, as well as their implementation, their use, and other practical aspects The aim is to familiarize the reader with these numerical algorithms: understanding their behaviour in practice, properly using existing software libraries, adequately designing and implementing “home-made” methods, correctly diagnosing the causes of possible difficulties Expected readers are engineers, Master or Ph.D students, confirmed researchers, in applied mathematics or from various other disciplines where optimization is a need Our aim is therefore not to give most accurate results in optimization, nor to detail the latest refinements of such and such method First of all, little is said concerning optimization theory itself (optimality conditions, constraint qualification, stability theory) As for algorithms, we limit ourselves most of the time to stable and well-established material Throughout we keep as a leading thread the actual practical value of optimization methods, in terms of their efficiency to solve real-world problems Nevertheless, serious attention is paid to the theoretical properties of optimization methods: this book is mainly based upon theorems Besides, some new and promising results or approaches could not be completely discarded; they are also presented, generally in the form of special sections, mainly aimed at orienting the reader to the relevant bibliography An introductory chapter gives some generalities on optimization and iterative algorithms It contains in particular motivating examples, ranking from meteorological forecast to power production management; they illustrate the large field of branches where optimization finds its applications Then come four parts, rather independent of each other The first one is devoted to algorithms for unconstrained optimization which, in addition to their direct usefulness, are a basis for more complex problems The second part concerns rather special methods, applicable when the usual differentiability assumptions are not satisfied Such methods appear in the decomposition of large-scale problems and the relaxation of combinatorial problems Nonlinearly constrained optimization forms the third part, substantially more technical, as the subject is still in evolution Finally, the fourth part gives a deep account of the more recent interior point methods, originally designed VI Preface for the simpler problems of linear and quadratic programming, and whose application to more general situations is the subject of active research This book is a translated and improved version of the monograph [43], written in French The French monograph was used as the textbook of an intensive two week course given several times by the authors, both in France and abroad Each topic was presented from a theoretical point of view in morning lectures The afternoons were devoted to implementation issues and related computational work The conception of such a course is due to J.-B Hiriart-Urruty, to whom the authors are deeply indebted Finally, three of the authors express their warm gratitude to Claude Lemar´echal for having given the impetus to this new work by providing a first English version Notes on this revised edition Besides minor corrections, the present version contains substantial changes with respect to the first edition First of all, (simplified but) nontrivial application problems have been inserted They involve the typical operations to be performed when one is faced with a real-life application: modelling, choice of methodology and some theoretical work to motivate it, computer implementation Such computational exercises help getting a better understanding of optimization methods beyond their theoretical description, by addressing important features to be taken into account when passing to implementation of any numerical algorithm In addition, the theoretical background in Part I now includes a discussion on global convergence, and a section on the classical pivotal approach to quadratic programming Part II has been completely reorganized and expanded The introductory chapter, on basic subdifferential calculus and duality theory, has two examples of nonsmooth functions that appear often in practice and serve as motivation (pointwise maximum and dual functions) A new section on convergence results for bundle methods has been added The chapter on applications of nonsmooth optimization, previously focusing on decomposition of complex problems via Lagrangian duality, describes also extensions of bundle methods for handling varying dimensions, for solving constrained problems, and for solving generalized equations Also, a brief commented review of existing software for nonlinear optimization has been added in Part III Finally, the reader will find additional information at http://www-rocq inria.fr/~gilbert/bgls The page gathers the data for running the test problems, various optimization codes, including an SQP solver (in Matlab), and pieces of software that solve the computational exercises Paris, Grenoble, Rio de Janeiro, May 2006 J Fr´ed´eric Bonnans J Charles Gilbert Claude Lemar´echal Claudia A Sagastiz´ abal Table of Contents Preliminaries General Introduction 1.1 Generalities on Optimization 1.1.1 The Problem 1.1.2 Classification 1.2 Motivation and Examples 1.2.1 Molecular Biology 1.2.2 Meteorology 1.2.3 Trajectory of a Deepwater Vehicle 1.2.4 Optimization of Power Management 1.3 General Principles of Resolution 1.4 Convergence: Global Aspects 1.5 Convergence: Local Aspects 1.6 Computing the Gradient Bibliographical Comments 3 5 10 12 14 16 19 Part I Unconstrained Problems Basic Methods 2.1 Existence Questions 2.2 Optimality Conditions 2.3 First-Order Methods 2.3.1 Gauss-Seidel 2.3.2 Method of Successive Approximations, or Gradient Method 2.4 Link with the General Descent Scheme 2.4.1 Choosing the -Norm 2.4.2 Choosing the -Norm 2.5 Steepest-Descent Method 2.6 Implementation Bibliographical Comments 25 25 26 27 27 28 28 29 30 30 34 35 VIII Table of Contents Line-Searches 3.1 General Scheme 3.2 Computing the New t 3.3 Optimal Stepsize (for the record only) 3.4 Modern Line-Search: Wolfe’s Rule 3.5 Other Line-Searches: Goldstein and Price, Armijo 3.5.1 Goldstein and Price 3.5.2 Armijo 3.5.3 Remark on the Choice of Constants 3.6 Implementation Considerations Bibliographical Comments 37 37 40 42 43 47 47 47 48 49 50 Newtonian Methods 4.1 Preliminaries 4.2 Forcing Global Convergence 4.3 Alleviating the Method 4.4 Quasi-Newton Methods 4.5 Global Convergence 4.6 Local Convergence: Generalities 4.7 Local Convergence: BFGS Bibliographical Comments 51 51 52 53 54 57 59 61 65 Conjugate Gradient 5.1 Outline of Conjugate Gradient 5.2 Developing the Method 5.3 Computing the Direction 5.4 The Algorithm Seen as an Orthogonalization Process 5.5 Application to Non-Quadratic Functions 5.6 Relation with Quasi-Newton Bibliographical Comments 67 67 69 70 70 72 74 75 Special Methods 6.1 Trust-Regions 6.1.1 The Elementary Problem 6.1.2 The Elementary Mechanism: Curvilinear Search 6.1.3 Incidence on the Sequence xk 6.2 Least-Squares Problems: Gauss-Newton 6.3 Large-Scale Problems: Limited-Memory Quasi-Newton 6.4 Truncated Newton 6.5 Quadratic Programming 6.5.1 The basic mechanism 6.5.2 The solution algorithm 6.5.3 Convergence Bibliographical Comments 77 77 78 79 81 82 84 86 88 89 90 92 95 Table of Contents A Case Study: Seismic Reflection Tomography 7.1 Modelling 7.2 Computation of the Reflection Points 7.3 Gradient of the Traveltime 7.4 The Least-Squares Problem to Solve 7.5 Solving the Seismic Reflection Tomography Problem General Conclusion IX 97 97 99 100 101 102 103 Part II Nonsmooth Optimization Introduction to Nonsmooth Optimization 8.1 First Elements of Convex Analysis 8.2 Lagrangian Relaxation and Duality 8.2.1 Primal-Dual Relations 8.2.2 Back to the Primal Recovering Primal Solutions 8.3 Two Convex Nondifferentiable Functions 8.3.1 Finite Minimax Problems 8.3.2 Dual Functions in Lagrangian Duality 109 109 111 111 113 116 116 117 Some Methods in Nonsmooth Optimization 9.1 Why Special Methods? 9.2 Descent Methods 9.2.1 Steepest-Descent Method 9.2.2 Stabilization A Dual Approach The ε-subdifferential 9.3 Two Black-Box Methods 9.3.1 Subgradient Methods 9.3.2 Cutting-Planes Method 119 119 120 121 124 126 127 130 10 Bundle Methods The Quest for Descent 10.1 Stabilization A Primal Approach 10.2 Some Examples of Stabilized Problems 10.3 Penalized Bundle Methods 10.3.1 A Trip to the Dual Space 10.3.2 Managing the Bundle Aggregation 10.3.3 Updating the Penalization Parameter Reversal Forms 10.3.4 Convergence Analysis 137 137 140 141 144 147 11 Applications of Nonsmooth Optimization 11.1 Divide to conquer Decomposition methods 11.1.1 Price Decomposition 11.1.2 Resource Decomposition 11.1.3 Variable Partitioning or Benders Decomposition 11.1.4 Other Decomposition Methods 161 161 163 167 169 171 150 154 X Table of Contents 11.2 Transpassing Frontiers 11.2.1 Dynamic Bundle Methods 11.2.2 Constrained Bundle Methods 11.2.3 Bundle Methods for Generalized Equations 172 173 177 180 12 Computational Exercises 12.1 Building Prototypical NSO Black Boxes 12.1.1 The Function maxquad 12.1.2 The Function maxanal 12.2 Implementation of Some NSO Methods 12.3 Running the Codes 12.4 Improving the Bundle Implementation 12.5 Decomposition Application 183 183 183 184 185 186 187 187 Part III Newton’s Methods in Constrained Optimization 13 Background 13.1 Differential Calculus 13.2 Existence and Uniqueness of Solutions 13.3 First-Order Optimality Conditions 13.4 Second-Order Optimality Conditions 13.5 Speed of Convergence 13.6 Projection onto a Closed Convex Set 13.7 The Newton Method 13.8 The Hanging Chain Project I Notes Exercises 197 197 199 200 202 203 205 205 208 213 214 14 Local Methods for Problems with Equality Constraints 14.1 Newton’s Method 14.2 Adapted Decompositions of Rn 14.3 Local Analysis of Newton’s Method 14.4 Computation of the Newton Step 14.5 Reduced Hessian Algorithm 14.6 A Comparison of the Algorithms 14.7 The Hanging Chain Project II Notes Exercises 215 216 222 227 230 235 243 245 250 251 15 Local Methods for Problems with Equality and Inequality Constraints 15.1 The SQP Algorithm 15.2 Primal-Dual Quadratic Convergence 15.3 Primal Superlinear Convergence 255 256 259 264 Table of Contents XI 15.4 The Hanging Chain Project III 267 Notes 270 Exercise 270 16 Exact Penalization 16.1 Overview 16.2 The Lagrangian 16.3 The Augmented Lagrangian 16.4 Nondifferentiable Augmented Function Notes Exercises 271 271 274 275 279 284 285 17 Globalization by Line-Search 17.1 Line-Search SQP Algorithms 17.2 Truncated SQP 17.3 From Global to Local 17.4 The Hanging Chain Project IV Notes Exercises 289 291 298 307 316 320 321 18 Quasi-Newton Versions 18.1 Principles 18.2 Quasi-Newton SQP 18.3 Reduced Quasi-Newton Algorithm 18.4 The Hanging Chain Project V 323 323 327 331 340 Part IV Interior-Point Algorithms for Linear and Quadratic Optimization 19 Linearly Constrained Optimization and Simplex Algorithm 19.1 Existence of Solutions 19.1.1 Existence Result 19.1.2 Basic Points and Extensions 19.2 Duality 19.2.1 Introducing the Dual Problem 19.2.2 Concept of Saddle-Point 19.2.3 Other Formulations 19.2.4 Strict Complementarity 19.3 The Simplex Algorithm 19.3.1 Computing the Descent Direction 19.3.2 Stating the algorithm 19.3.3 Dual simplex 19.4 Comments 353 353 353 355 356 357 358 362 363 364 364 365 367 368 480 References 285 E Polak Optimization – Algorithms and Consistent Approximations Number 124 in Applied Mathematical Sciences Springer, 1997 286 B.T Polyak Introduction to Optimization Optimization Software, New York, 1987 287 F.A Potra An O(nL) infeasible-interior-point algorithm for LCP with quadratic convergence Annals of Operations Research, 62:81–102, 1996 288 M.J.D Powell A method for nonlinear constraints in minimization problems In R Fletcher, editor, Optimization, pages 283–298 Academic Press, London, New York, 1969 289 M.J.D Powell On the convergence of the variable metric algorithm Journal of the Institute of Mathematics and its Applications, 7:21–36, 1971 290 M.J.D Powell Some global convergence properties of a variable metric algorithm for minimization without exact line searches In R.W Cottle and C.E Lemke, editors, Nonlinear Programming, number in SIAM-AMS Proceedings American Mathematical Society, Providence, RI, 1976 291 M.J.D Powell Algorithms for nonlinear constraints that use Lagrangian functions Mathematical Programming, 14:224–248, 1978 292 M.J.D Powell The convergence of variable metric methods for nonlinearly constrained optimization calculations In O.L Mangasarian, R.R Meyer, and S.M Robinson, editors, Nonlinear Programming 3, pages 27–63, 1978 293 M.J.D Powell A fast algorithm for nonlinearly constrained optimization calculations In G.A Watson, editor, Numerical Analysis Dundee 1977, number 630 in Lecture Notes in Mathematics, pages 144–157 Springer-Verlag, Berlin, 1978 294 M.J.D Powell Nonconvex minimization calculations and the conjugate gradient method In Lecture Notes in Mathematics 1066, pages 122–141 SpringerVerlag, Berlin, 1984 295 M.J.D Powell Convergence properties of algorithms for nonlinear optimization SIAM Review, 28:487–500, 1985 296 M.J.D Powell On the quadratic programming algorithm of Goldfarb and Idnani Mathematical Programming Study, 25:46–61, 1985 297 M.J.D Powell The performance of two subroutines for constrained optimization on some difficult test problems In P.T Boggs, R.H Byrd, and R.B Schnabel, editors, Numerical Optimization 1984, pages 160–177 SIAM Publication, Philadelphia, 1985 298 M.J.D Powell A view of nonlinear optimization In J.K Lenstra, A.H.G Rinnooy Kan, and A Schrijver, editors, History of Mathematical Programming, A Collection of Personal Reminiscences, pages 119–125 CWI North-Holland, Amsterdam, 1991 299 M.J.D Powell and Y Yuan A recursive quadratic programming algorithm that uses differentiable exact penalty functions Mathematical Programming, 35:265–278, 1986 300 B.N Pshenichnyj Algorithm for a general mathematical programming problem Kibernetika, 5:120–125, 1970 301 B.N Pshenichnyj The Linearization Method for Constrained Optimization Number 22 in Computational Mathematics Springer-Verlag, 1994 302 B.N Pshenichnyj and Yu.M Danilin Numerical Methods for Extremal Problems MIR, Moscow, 1978 303 W Queck The convergence factor of preconditioned algorithms of the ArrowHurwicz type SIAM Journal on Numerical Analysis, 26:1016–1030, 1989 References 481 304 J.K Reid A sparsity-exploiting variant of the Bartels-Golub decomposition for linear programming bases Mathematical Programming, 24:55–69, 1982 305 P.A Rey and C.A Sagastiz´ abal Dynamical adjustment of the prox-parameter in variable metric bundle methods Optimization, 51(2):423–447, 2002 306 S.M Robinson A quadratically convergent algorithm for general nonlinear programming problems Mathematical Programming, 3:145–156, 1972 307 S.M Robinson Perturbed Kuhn-Tucker points and rates of convergence for a class of nonlinear-programming algorithms Mathematical Programming, 7:1–16, 1974 308 S.M Robinson Generalized equations and their solutions, part II: applications to nonlinear programming Mathematical Programming Study, 19:200–221, 1982 309 R.T Rockafellar Convex Analysis Number 28 in Princeton Mathematics Ser Princeton University Press, Princeton, New Jersey, 1970 310 R.T Rockafellar New applications of duality in convex programming In Proceedings of the 4th Conference of Probability, Brasov, Romania, pages 73– 81, 1971 311 R.T Rockafellar Augmented Lagrange multiplier functions and duality in nonconvex programming SIAM Journal on Control, 12:268–285, 1974 312 R.T Rockafellar Augmented Lagrangians and applications of the proximal point algorithm in convex programming Mathematics of Operations Research, 1:97–116, 1976 313 R.T Rockafellar Monotone operators and the proximal point algorithm SIAM Journal on Control and Optimization, 14:877–898, 1976 314 R.T Rockafellar and R.J.-B Wets Variational Analysis Springer Verlag, Heidelberg, 1998 315 T Rusten and R Winthier A preconditioned iterative method for saddle point problems SIAM Journal on Matrix Analysis and Applications, 13:887– 904, 1992 316 A Ruszczy´ nski Decomposition methods in stochastic programming In Mathematical Programming, volume 79, 1997 317 C Sagastiz´ abal and M.V Solodov On the relation between bundle methods for maximal monotone inclusions and hybrid proximal point algorithms In Inherently parallel algorithms in feasibility and optimization and their applications (Haifa, 2000), volume of Stud Comput Math., pages 441–455 North-Holland, Amsterdam, 2001 318 C Sagastiz´ abal and M.V Solodov An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter SIAM Journal on Optimization, 16(1):146–169, 2005 319 R Saigal Linear Programming: A Modern Integrated Analysis Kluwer Academic Publishers, Boston, 1995 320 R.W.H Sargent The development of SQP algorithm for nonlinear programming In L.T Biegler, T.F Coleman, A.R Conn, and F.N Santosa, editors, Large-Scale Optimization with Applications, part II: Optimal design and Control, pages 1–19 IMA Vol Math Appl 93, 1997 321 R.W.H Sargent and M Ding A new SQP algorithm for large-scale nonlinear programming SIAM Journal on Optimization, 11:716–747, 2000 322 K Schittkowski The nonlinear programming method of Wilson, Han and Powell with an augmented Lagrangian type line search function, Part 1: convergence analysis Numerische Mathematik, 38:83–114, 1981 482 References 323 K Schittkowski NLPQL: a FORTRAN subroutine solving constrained nonlinear programming problems Annals of Operations Research, 5:485–500, 1985 324 K Schittkowski Solving nonlinear programming problems with very many constraints Optimization, 25:179–196, 1992 325 K Schittkowski Numerical Data Fitting in Dynamical Systems Kluwer Academic Press, Dordrecht, 2002 326 H Schramm and J Zowe A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results SIAM Journal on Optimization, 2(1):121–152, 1992 327 Scilab A free scientific software package http://www.scilab.org/ 328 D.F Shanno Conjugate gradient methods with inexact searches Mathematics of Operations Research, 3:244–256, 1978 329 D.F Shanno and K.H Phua Algorithm 500, minimization of unconstrained multivariate functions ACM Transactions on Mathematical Software, 2:87– 94, 1976 330 A Shapiro and J Sun Some properties of the augmented Lagrangian in cone constrained optimization Mathematics of Operations Research, 29:479–491, 2004 331 N Shor Utilization of the operation of space dilatation in the minimization of convex functions Kibernetica, 1:6–12, 1970 (English translation: Cybernetics, 6, 7-15) 332 N.Z Shor Minimization methods for non-differentiable functions Springer Verlag, Berlin, 1985 333 D Silvester and A Wathen Fast iterative solution of stabilized Stokes systems part II: using general block preconditioners SIAM Journal on Numerical Analysis, 31:1352–1367, 1994 334 M Slater Lagrange multipliers revisited: a contribution to non-linear programming Cowles Commission Discussion Paper, Math 403, 1950 335 M V Solodov A class of decomposition methods for convex optimization and monotone variational inclusions via the hybrid inexact proximal point framework Optim Methods Softw., 19(5):557–575, 2004 336 M V Solodov and B F Svaiter A hybrid approximate extragradientproximal point algorithm using the enlargement of a maximal monotone operator Set-Valued Anal., 7(4):323–345, 1999 337 M V Solodov and B F Svaiter A hybrid projection-proximal point algorithm J Convex Anal., 6(1):59–70, 1999 338 M V Solodov and B F Svaiter A unified framework for some inexact proximal point algorithms Numerical Functional Analysis and Optimization, 22:1013–1035, 2001 339 B Speelpenning Compiling fast partial derivatives of functions given by algorithms PhD thesis, Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, 1980 340 P Spellucci Numerische Verfahren der nichtlinearen Optimierung Birkhă auser, 1993 341 P Spellucci A new technique for inconsistent problems in the SQP method Mathematical Methods of Operations Research, 47:355–400, 1998 342 P Spellucci An SQP method for general nonlinear programs using only equality constrained subproblems Mathematical Programming, 82:413–448, 1998 343 J E Spingarn Partial inverse of a monotone operator Appl Math Optim., 10(3):247–265, 1983 References 483 344 M Spivak A Comprehensive Introduction to Differential Geometry Publish or Perish, 1979 345 T Steihaug The conjugate gradient method and trust regions in large scale optimization SIAM Journal on Numerical Analysis, 20:626–637, 1983 346 R.A Tapia Diagonalized multiplier methods and quasi-Newton methods for constrained optimization Journal of Optimization Theory and Applications, 22:135–194, 1977 347 R.A Tapia On secant updates for use in general constrained optimization Mathematics of Computation, 51:181–202, 1988 348 T Terlaky, editor Interior Point Methods of Mathematical Programming Kluwer Academic Publishers, Boston, 1996 349 T Terlaky, J P Vial, and K Roos Theory and algorithms for linear optimization: an interior point approach Wiley intersciences, New York, 1997 350 M.J Todd On convergence properties of algorithms for unconstrained minimization IMA Journal of Numerical Analysis, 9(3):435–441, 1989 351 K Tone Revisions of constraint approximations in the successive QP method for nonlinear programming problems Mathematical Programming, 26:144– 152, 1983 352 P Tseng Alternating projection-proximal methods for convex programming and variational inequalities SIAM J Optim., 7(4):951–965, 1997 353 R.J Vanderbei Linear Programming: Foundations and extensions Kluwer Academic Publishers, Boston, 1997 354 S.A Vavasis Nonlinear Optimization – Complexity Issues Oxford University Press, New York, 1991 355 R Verfă urth A combined conjugate gradient-multigrid algorithm for the numerical solution of the Stokes problem IMA Journal of Numerical Analysis, 4:441–455, 1984 356 K Veseli´c Finite catenary and the method of Lagrange SIAM Review, 37:224–229, 1995 357 M Wagner and M.J Todd Least-change quasi-Newton updates for equalityconstrained optimization Mathematical Programming, 87:317–350, 2000 358 A Wathen and D Silvester Fast iterative solution of stabilized Stokes systems part I: using simple diagonal preconditioners SIAM Journal on Numerical Analysis, 30:630–649, 1993 359 R.B Wilson A simplicial algorithm for concave programming PhD thesis, Graduate School of Business Administration, Harvard University, Cambridge, MA, USA, 1963 360 P Wolfe A duality theorem for nonlinear programming Quarterly Applied Mathematics, 19:239–244, 1961 361 P Wolfe Convergence conditions for ascent methods SIAM Review, 11:226– 235, 1969 362 P Wolfe A method of conjugate subgradients for minimizing nondifferentiable functions Mathematical Programming Study, 3:145–173, 1975 363 H Wolkowicz, R Saigal, and L Vandenberghe, editors Handbook of Semidefinite Programming – Theory, Algorithms, and Applications Kluwer Academic Publishers, 2000 364 S.J Wright A path-following infeasible-interior-point algorithm for linear complementarity problems Optimization Methods and Software, 2:79–106, 1993 484 References 365 S.J Wright An infeasible interior point algorithm for linear complementarity problems Mathematical Programming, 67:29–52, 1994 366 S.J Wright Primal-dual interior-point methods SIAM, Philadelphia, 1996 367 S.J Wright Superlinear convergence of a stabilized SQP method to a degenerate solution Computational Optimization and Applications, 11:253–275, 1998 368 Y Xie and R.H Byrd Practical update criteria for reduced Hessian SQP: global analysis SIAM Journal on Optimization, 9:578–604, 1999 369 X Xu, P.F Hung, and Y Ye A simplification of the homogeneous and self-dual linear programming algorithm and its implementation Annals of Operations Research, 62:151–172, 1996 370 Y Ye Interior point algorithms Wiley-Interscience Series in Discrete Mathematics and Optimization John Wiley & Sons Inc., New York, 1997 371 Y Ye On homogeneous and self-dual algorithm for LCP Mathematical Programming, 76:211–222, 1997 √ 372 Y Ye, M.J Todd, and S Mizuno An O( nL)-iteration homogeneous and self-dual linear programming algorithm Mathematics of Operations Research, 19:53–67, 1994 373 Y Yuan An only 2-step Q-superlinearly convergence example for some algorithms that use reduced Hessian informations Mathematical Programming, 32:224–231, 1985 374 W.I Zangwill Non-linear programming via penalty functions Management Science, 13:344–358, 1967 375 Y Zhang On the convergence of a class of infeasible interior-point methods for the horizontal linear complementarity problem SIAM J Optimization, 4:208–227, 1994 376 G Zoutendijk Nonlinear programming, computational methods In J Abadie, editor, Integer and Nonlinear Programming, pages 37–86 North-Holland, Amsterdam, 1970 Index active set method, 88 adjoint, see state admissibility of the unit stepsize, 48, 63, 307, 322, 337 admissible, see point, set algorithm, see also method – BFGS, see BFGS – Bunch & Kaufman, 231 – bundle, see bundle method, constrained bundle – conjugate gradient, 84, 86 – – preconditioned, 76 – – truncated, 300, 303 – cutting-planes, see cutting-planes method – descent, 120 – ellipsoid, 129 – Karmarkar, 457 – largest step, 435 – – with safeguard, 435 – Newton, see Newton’s algorithm – predictor-corrector, 397 – quasi-Newton, see quasi-Newton or secant (algorithm) – simplex, 364 – SQP, see sequential quadratic programming algorithm – subgradient, see subgradient method – Uzawa, 232 analytic center, 391, 437 Armijo, 47, 80, 84, 138, 295, 304, 305 automatic differentiation, see computational differentiation auxiliary problem principle, 171 basis matrix, 365 BFGS, 55, 58, 75, 85, 86, 325 – limited memory, 195 bisection, 40 black box, 11, 126 – constrained, 178 – dynamic, 173 bracket, 39 Bunch & Kaufman, 231 Bunch & Parlett, 53 bundle, 137, 144 – aggregation, 139, 143 – compression, 147, 149 – disaggregate, 166 – selection, 149 bundle method – constrained, see constrained bundle method – dual, 147 – dynamic, 175 – – finite termination, 176 – for generalized equations, 182 – general, 138 – level, 141 – penalized, 141 – – convergence, 155, 156 – – implementation, 186 – – parameter update, 150 – trust region, 140 Cauchy-Schwarz inequality, 205 – generalized, 279 central path, 373, 375 – perturbed, 412 chain, see hanging chain project Cholesky, 53, 269, 317, 320 code n1cv2, 153 coercive, 117, 171 combinatorial optimization, 3, 10, 96, 163 complementarity condition, 115, 200 486 Index – strict, 201, 380 complementarity problem (linear), 89, 371, 374 – canonical form, 379, 380 – monotone, 374 – standard form, 378 complexity, 451 computational differentiation, 19, 20, 95 cone, 202 – critical, 202 conjugate, see algorithm, direction constrained bundle method – feasible, 179 – filter, 180 – infeasible, 178 constraint, – active, 4, 89, 192, 194 – equality, – inequality, – strongly active, 201 – weakly active, 201 constraint qualification, 113, 116 – (A-CQ), 201 – (S-CQ), of Slater, 201 – (LI-CQ), 201 – (MF-CQ), of MangasarianFromovitz, 201 control, see also state – problem, 4, 7, 8, 16, 224 – variable, 7, 16 convergence – global, 12, 26, 45, 52, 53, 74, 296, 306 – in p steps, 204 – linear, 14, 204 – local, 14, 206, 227, 229, 241, 259, 262 – of bundle method, 155, 156 – of cutting-planes method, 133 – of subgradient method, 128 – quadratic, 14, 204, 253 – speed of, 14, 33, 51, 86, 88, 203–204 – superlinear, 14, 204, 265 convex, see function, problem, set convex hull, 110 correction – Powell, 328, 330, 332, 340 – second-order, 310 critical, see cone, direction, point cubic fit, 40 curvature condition, 325 curvilinear search, see also line-search, 85, 141, 153, 333 cuts – feasibility, 170 – optimality, 170 cutting-planes method, 131 – convergence, 133 – implementation, 186 Davidon, Fletcher & Powell, 55 decomposition – Benders, 169 – Dantzig-Wolfe, 166 – energy application, 187 – price, 162, 165 – – algorithm, 164 – proximal, 172 – resource, 162, 167 decomposition of Rn – by partitioning, 224 – oblique, 226 – orthogonal, 225, 253 Dennis & Mor´e, 60, 63, 82 dilation, 129 direct communication, 211 direction, 12, 85 – affine, 383 – centralization, 383 – conjugate, 69, 74 – critical, 202 – of ε-descent, 125 – of descent, 29, 37, 75, 111, 127, 289, 321 – of steepest descent, 30, 121, 123 – quasi-negative curvature, 300 directional derivative, 109, 198 divergent series, 129 duality, 356 – gap, 113, 114, 116 – weak, 114, 168 elliptic, see function equivalent sequences, 204 Everett, 113 existence of solution, 25, 199 extrapolation, 40, 47 Index feasible, see point, set Fermat, 99, 111 filter strategy, 180 finite difference, 101, 119 Finsler, 285 Fletcher (initialization of), 39, 48 Fletcher-Reeves, 73 Fromovitz, see constraint qualification function – maxanal, 184 – maxquad, 153, 183 – affine, 4, 26, 83 – convex, 25, 26, 57 – – strongly, 27, 61, 63, 65, 67 – convex-concave, 359 – dual, 112 – elliptic, 27, 48, 53, 74 – improvement, 177 – inf-compact, 25 – lower semi-continuous, 25 – merit, 13, 29, 37, 79, 271 – penalty, 271 – – exact, 272 – value, 167, 169 Gauss, 101 Gauss-Newton, 83, 86 Gauss-Seidel, 29 generalized equation, 180 globalization of an algorithm, 271 – by line-search, 52, 289 – by trust regions, 77 gradient, 3, 13, 23, 26 – projected, 90 – reduced, 233 group, 378 growth condition (quadratic), 27, 33, 46 hanging chain project, 208–213, 245–250, 267–270, 316–320, 340–344 Hessian, see also reduced Hessian, 3, 26, 27, 51–53, 57, 63–65, 73, 82, 83, 95, 102, 103 – of the Lagrangian, 227 I (x), 194 I∗0 , I∗0+ , I∗00 , 201 487 identification, see parameter identification inf-compact, see function instability – of cutting-planes method, 134 – of steepest-descent, 31, 122 interpolation, 40, 296, 305, 317 invariant, 378 inverse problem, see also parameter identification, 7, 101 Karush, Kuhn, and Tucker (KKT), see multiplier, optimality conditions Lagrange multiplier, see multiplier Lagrangian, 11, 78, 112, 200, 272, 274, 357 – augmented, 118, 163, 272, 276, 285, 330 – relaxation, 10, 112, 163, 173 – – dynamic, 173 length of a linear problem, 452 Levenberg-Marquardt, 84 line-search, see also curvilinear search, 12, 72, 77, 78, 91 – Armijo, 295, 305 – backtracking, 296, 304, 305, 329 – nonmonotone, 321 – piecewise (PLS), 336 – watchdog, 321 – Wolfe, see also Wolfe conditions, 58, 63, 65, 75, 83, 85, 87, 326 linear complementarity problem, see complementarity problem linearization error, 144 local minimum, see solution Mangasarian, see constraint qualification Maratos effect, 308, 329 master program, 161, 165, 167 matrix – basis, 222 – inertia, 252 – positive definite, 26, 27, 67, 82, 83, 86, 88 – right inverse, 222, 252, 253 method, see also algorithm – local, 216 – multiplier, 163 488 Index – primal-dual, 217 minimax – finite, 116 – infinite, 117 minimizing sequence, 13, 140, 147, 154, 155 minimum, see solution model, 12, 52, 77, 85 – cutting-planes, 130, 137, 144 – – aggregate, 150 – – disaggregate, 166 – – improving, 139 – piecewise affine, 130 modified field, 387, 388, 418 monotone, 374 Moreau-Yosida regularization, 150 multifunction, 124 – closed, 124 – continuous, 124 multiplier, 112, 116, 166, 200 – first-order, 235 – Lagrange, 12, 103, 200, 360 – least-squares, 228, 235, 253 – second-order, 235 neighborhood, 397 – large, 375, 406 – small, 375, 398 Newton’s algorithm, 39, 79 – for equality constrained problems – – primal version, 229 – – primal-dual version, 221 – – reduced Hessian, 239 – – simplified Newton, 240 – for inequality constrained problems, see sequential quadratic programming – for nonlinear equations, 51, 205 – for unconstrained optimization, 207 Newton’s step – longitudinal component, 223 – transversal component, 223 nominal decrease, 49, 80, 131, 147 norm, 321 – associated with a scalar product, 205 – dual, 279, 286 O(·), big O, 203 o(·), little o, 14, 203 objective function, 3, 16 optimal control, see control optimal partition, 364, 379 optimal stepsize, 30 optimality conditions, 13 – necessary – – 1st order (NC1), 26 – – 2nd order (NC2), 26, 202 – – Karush, Kuhn, and Tucker (KKT), 200 – – reduced, 236 – sufficient – – 2nd order (SC2), 26, 203 – – semi-strong 2nd order, 203, 286 – – strong 2nd order, 203 – – weak 2nd order, 203, 286 optimality system, 360 oscillation, 31, 122 osculating quadratic problem, 219, 232, 256, 259 – equality constrained, 218 – inequality constrained, 257 – unconstrained, 208 parameter – augmentation, 276 – penalty, 279 parameter identification, 6, 82, 100 partition of variables, 379 penalization, see also function (penalty) – exact, 272 – – augmented Lagrangian, 277, 287 – – Fletcher, 285, 320 – – , 287 – – Lagrangian, 274 – – of the objective, 272 – logarithmic, 371 – quadratic, 101 performance profile, 341 piecewise line-search (PLS), see line-search pivoting, 92, 368 point – basic, 355 – – regular, 366 – critical or stationary, 26, 27, 52, 82, 201, 359 – feasible or admissible, 89, 193 – interior, 374 Index – optimal, 111 Polak-Ribi`ere, 73 potential – Karmarkar, 457 – logarithmic, 371, 390 Powell, see correction preconditioning, 34 problem – constrained convex, 26, 194 – convex, 88, 113, 116 – dual, 112, 357, 358 – least-squares, 82, 253 – linear, 354 – osculating quadratic, see osculating quadratic problem – (PE ), 215 – (PEI ), 193 – primal, 111, 358 – quadratic, 354 – saddle-point, 358 project, see seismic reflection tomography project, hanging chain project projection onto a convex set, 205 proximal point, 151 – implementable form, 151, 180 proximity measure, 375, 437 quasi-Newton or secant – algorithm – – quasi-Newton SQP, 328 – – reduced, for equality constrained problems, 338 – equation, 54, 325 – matrix, 54 – method, 56, 78, 86 – – poor man, 75, 84, 151 R∗+ , 276 reduced cost, 365 reduced Hessian of the Lagrangian, 221, 233, 237, 252 reflection tomography, see seismic reflection tomography project regular stationary point, 221, 252 relative distance, 444 relative interior, 113 row/column generation, 166 489 saddle-point, 274 safeguard, 41, 436 scaling, 35 search, see curvilinear search, line-search secant, see quasi-Newton seismic reflection tomography project, 97–103 self-duality, 425 separating hyperplane, 111, 121 sequential quadratic programming (SQP) algorithm, 191 – line-search SQP, 292 – local, 257 – truncated (TSQP), 305 set – convex, 26 – feasible or admissible, 193, 353, 374 – – perturbed, 412 set-valued map, see multifunction simulator, 11, 16, 37, 67, 100, 211 Slater, see also constraint qualification, 113, 116, 201 slice, 25, 46, 58 solution, see also existence, uniqueness, 199, 353 – global, 12, 199 – local, 26, 82, 199 – primal-dual, 201 – strict local, 199 – strong, 203 speed of convergence, see convergence spline (cubic), 98 SQP, see sequential quadratic programming algorithm stabilization principle, 12, 30, 37, 137 standard form – of linear constraints, 353 state – adjoint, 18 – – equation, 17, 18 – constraint on the -, – equation, – variable, stationary, see point step – null, 139 – serious, 138 490 Index stopping test, 34, 41 – formal, 120 – implementable, 131, 138, 147 subdifferential, 110 – approximate, 124, 125 subgradient, 110 – inequality, 110 – smeared, 147 subgradient method, 127 – convergence, 128 – implementation, 186 submersion, 193 test problem, see seismic reflection tomography project, hanging chain project tomography, see seismic reflection tomography project trap of nonsmooth optimization, 119 trust region, 53, 84, 87, 138, 140 uniqueness of solution, 199 update criterion, 339 value, 353 variable – control, decision, 3, 224 – dual, 112 – state, 224 weak duality, see duality Wolfe – duality, 363 Wolfe conditions, 43 – generalized, 80, 334 zigzag, 31, 94, 122, 123, 129 Universitext Aguilar, M.; Gitler, S.; Prieto, C.: Algebraic Topology from a Homotopical Viewpoint Băorger, E.; Grăadel, E.; Gurevich, Y.: The Classical Decision Problem Aksoy, A.; Khamsi, M A.: Methods in Fixed Point Theory Băottcher, A; Silbermann, B.: Introduction to Large Truncated Toeplitz Matrices Alevras, D.; Padberg M W.: Linear Optimization and Extensions Boltyanski, V.; Martini, H.; Soltan, P S.: Excursions into Combinatorial Geometry Andersson, M.: Topics in Complex Analysis Boltyanskii, V G.; Efremovich, V A.: Intuitive Combinatorial Topology Aoki, M.: State Space Modeling of Time Series Arnold, V I.: Lectures on Partial Differential Equations Bonnans, J F.; Gilbert, J C.; Lemar´echal, C.; Sagastiz´abal, C A.: Numerical Optimization Arnold, V I.; Cooke, R.: Ordinary Differential Equations Booss, B.; Bleecker, D D.: Topology and Analysis Audin, M.: Geometry Borkar, V S.: Probability Theory Aupetit, B.: A Primer on Spectral Theory Brunt B van: The Calculus of Variations Bachem, A.; Kern, W.: Linear Programming Duality Carleson, L.; Gamelin, T W.: Dynamics Bachmann, G.; Narici, L.; Beckenstein, E.: Fourier and Wavelet Analysis Cecil, T E.: Lie Sphere Geometry: With Applications of Submanifolds Badescu, L.: Algebraic Surfaces Chae, S B.: Lebesgue Integration Balakrishnan, R.; Ranganathan, K.: A Textbook of Graph Theory Chandrasekharan, K.: Transform Balser, W.: Formal Power Series and Linear Systems of Meromorphic Ordinary Differential Equations Charlap, L S.: Bieberbach Groups and Flat Manifolds Bapat, R.B.: Linear Algebra and Linear Models Benedetti, R.; Petronio, C.: Lectures on Hyperbolic Geometry Benth, F E.: Option Theory with Stochastic Analysis Berberian, S K.: Analysis Fundamentals of Real Berger, M.: Geometry I, and II Bliedtner, J.; Hansen, W.: Potential Theory Complex Classical Fourier Chern, S.: Complex Manifolds without Potential Theory Chorin, A J.; Marsden, J E.: Mathematical Introduction to Fluid Mechanics Cohn, H.: A Classical Invitation to Algebraic Numbers and Class Fields Curtis, M L.: Abstract Linear Algebra Curtis, M L.: Matrix Groups Cyganowski, S.; Kloeden, P.; Ombach, J.: From Elementary Probability to Stochastic Differential Equations with MAPLE Blowey, J F.; Coleman, J P.; Craig, A W (Eds.): Theory and Numerics of Differential Equations Da Prato, G.: An Introduction to Infinite Dimensional Analysis Blyth, T S.: Lattices and Ordered Algebraic Structures Das, A.: The Special Theory of Relativity: A Mathematical Exposition Dalen, D van: Logic and Structure Debarre, O.: Higher-Dimensional Algebraic Geometry Deitmar, A.: A First Course in Harmonic Analysis Demazure, M.: strophes Bifurcations and Cata- Devlin, K J.: Fundamentals of Contemporary Set Theory DiBenedetto, E.: Equations Degenerate Parabolic Diener, F.; Diener, M.(Eds.): Nonstandard Analysis in Practice Dimca, A.: Sheaves in Topology Dimca, A.: Singularities and Topology of Hypersurfaces DoCarmo, M P.: Differential Forms and Applications Duistermaat, J J.; Kolk, J A C.: Lie Groups Dumortier.: Qualitative Theory of Planar Differential Systems Edwards, R E.: A Formal Background to Higher Mathematics Ia, and Ib Edwards, R E.: A Formal Background to Higher Mathematics IIa, and IIb Emery, M.: Stochastic Calculus in Manifolds Endler, O.: Valuation Theory Engel, K.-J.; Nagel, R.: A Short Course on Operator Semigroups Erez, B.: Galois Modules in Arithmetic Everest, G.; Ward, T.: Heights of Polynomials and Entropy in Algebraic Dynamics Farenick, D R.: Algebras of Linear Transformations Foulds, L R.: Graph Theory Applications Franke, J.; Hrdle, W.; Hafner, C M.: Statistics of Financial Markets: An Introduction Frauenthal, J C.: Mathematical Modeling in Epidemiology Friedman, R.: Algebraic Surfaces and Holomorphic Vector Bundles Fuks, D B.; Rokhlin, V A.: Course in Topology Beginner’s Fuhrmann, P A.: A Polynomial Approach to Linear Algebra Gallot, S.; Hulin, D.; Lafontaine, J.: Riemannian Geometry Gardiner, C F.: A First Course in Group Theory G˚arding, L.; Tambour, T.: Algebra for Computer Science Godbillon, C.: Dynamical Systems on Surfaces Godement, R.: Analysis I, and II Goldblatt, R.: Orthogonality and Spacetime Geometry Gouvˆea, F Q.: p-Adic Numbers Gross, M et al.: Calabi-Yau Manifolds and Related Geometries Gustafson, K E.; Rao, D K M.: Numerical Range The Field of Values of Linear Operators and Matrices Gustafson, S J.; Sigal, I M.: Mathematical Concepts of Quantum Mechanics Hahn, A J.: Quadratic Algebras, Clifford Algebras, and Arithmetic Witt Groups H´ajek, P.; Havr´anek, T.: Mechanizing Hypothesis Formation Heinonen, J.: Lectures on Analysis on Metric Spaces Hlawka, E.; Schoißengeier, J.; Taschner, R.: Geometric and Analytic Number Theory Holmgren, R A.: A First Course in Discrete Dynamical Systems Howe, R., Tan, E Ch.: Non-Abelian Harmonic Analysis Howes, N R.: Modern Analysis and Topology Hsieh, P.-F.; Sibuya, Y (Eds.): Basic Theory of Ordinary Differential Equations Humi, M., Miller, W.: Second Course in Ordinary Differential Equations for Scientists and Engineers Hurwitz, A.; Kritikos, N.: Lectures on Number Theory Huybrechts, D.: Complex Geometry: An Introduction Isaev, A.: Introduction to Mathematical Methods in Bioinformatics Istas, J.: Mathematical Modeling for the Life Sciences Iversen, B.: Cohomology of Sheaves Jacod, J.; Protter, P.: Probability Essentials Jennings, G A.: Modern Geometry with Applications Jones, A.; Morris, S A.; Pearson, K R.: Abstract Algebra and Famous Impossibilities Jost, J.: Compact Riemann Surfaces Jost, J.: Dynamical Systems Examples of Complex Behaviour Jost, J.: Postmodern Analysis Jost, J.: Riemannian Geometry and Geometric Analysis Kac, V.; Cheung, P.: Quantum Calculus Kannan, R.; Krueger, C K.: Analysis on the Real Line Advanced Kelly, P.; Matthews, G.: The Non-Euclidean Hyperbolic Plane Kempf, G.: Complex Abelian Varieties and Theta Functions Kitchens, B P.: Symbolic Dynamics Kloeden, P.; Ombach, J.; Cyganowski, S.: From Elementary Probability to Stochastic Differential Equations with MAPLE Kloeden, P E.; Platen; E.; Schurz, H.: Numerical Solution of SDE Through Computer Experiments Kostrikin, A I.: Introduction to Algebra Krasnoselskii, M A.; Pokrovskii, A V.: Systems with Hysteresis Kurzweil, H.; Stellmacher, B.: The Theory of Finite Groups An Introduction Lang, S.: Introduction to Differentiable Manifolds Luecking, D H., Rubel, L A.: Complex Analysis A Functional Analysis Approach Ma, Zhi-Ming; Roeckner, M.: Introduction to the Theory of (non-symmetric) Dirichlet Forms Mac Lane, S.; Moerdijk, I.: Sheaves in Geometry and Logic Marcus, D A.: Number Fields Martinez, A.: An Introduction to Semiclassical and Microlocal Analysis Matouˇsek, J.: Using the Borsuk-Ulam Theorem Matsuki, K.: Introduction to the Mori Program Mazzola, G.; Milmeister G.; Weissman J.: Comprehensive Mathematics for Computer Scientists Mazzola, G.; Milmeister G.; Weissman J.: Comprehensive Mathematics for Computer Scientists Mc Carthy, P J.: Introduction to Arithmetical Functions McCrimmon, K.: A Taste of Jordan Algebras Meyer, R M.: Essential Mathematics for Applied Field Meyer-Nieberg, P.: Banach Lattices Mikosch, T.: Non-Life Insurance Mathematics Mines, R.; Richman, F.; Ruitenburg, W.: A Course in Constructive Algebra Moise, E E.: Introductory Problem Courses in Analysis and Topology Montesinos-Amilibia, J M.: Classical Tessellations and Three Manifolds Morris, P.: Introduction to Game Theory Nikulin, V V.; Shafarevich, I R.: Geometries and Groups Oden, J J.; Reddy, J N.: Variational Methods in Theoretical Mechanics Øksendal, B.: Stochastic Differential Equations Øksendal, B.; Sulem, A.: Applied Stochastic Control of Jump Diffusions Poizat, B.: A Course in Model Theory Polster, B.: A Geometrical Picture Book Porter, J R.; Woods, R G.: Extensions and Absolutes of Hausdorff Spaces Radjavi, H.; Rosenthal, P.: Simultaneous Triangularization Ramsay, A.; Richtmeyer, R D.: Introduction to Hyperbolic Geometry Rautenberg, W.: A Concise Introduction to Mathematical Logic Reisel, R B.: Elementary Theory of Metric Spaces Smith, K E.; Kahanpăaaă, L.; Kekăalăainen, P.; Traves, W.: An Invitation to Algebraic Geometry Rey, W J J.: Introduction to Robust and Quasi-Robust Statistical Methods Smith, K T.: Power Series from a Computational Point of View Ribenboim, P.: Classical Theory of Algebraic Numbers Smory´nski, C.: Logical Number Theory I An Introduction Rickart, C E.: Natural Function Algebras Stichtenoth, H.: Algebraic Function Fields and Codes Rees, E G.: Notes on Geometry Rotman, J J.: Galois Theory Rubel, L A.: Functions Entire and Meromorphic Ruiz-Tolosa, J R.; Castillo E.: From Vectors to Tensors Runde, V.: A Taste of Topology Rybakowski, K P.: The Homotopy Index and Partial Differential Equations Sagan, H.: Space-Filling Curves Samelson, H.: Notes on Lie Algebras Stillwell, J.: Geometry of Surfaces Stroock, D W.: An Introduction to the Theory of Large Deviations Sunder, V S.: An Invitation to von Neumann Algebras ´ Tamme, G.: Introduction to Etale Cohomology Tondeur, P.: Manifolds Foliations on Riemannian Sauvigny, F.: tions I Partial Differential Equa- Toth, G.: Finite Măobius Groups, Minimal Immersions of Spheres, and Moduli Sauvigny, F.: tions II Partial Differential Equa- Verhulst, F.: Nonlinear Differential Equations and Dynamical Systems Schiff, J L.: Normal Families Sengupta, J K.: Optimal Decisions under Uncertainty Wong, M W.: Weyl Transforms Xamb´o-Descamps, S.: recting Codes Block Error-Cor- S´eroul, R.: Programming for Mathematicians Zaanen, A.C.: Continuity, Integration and Fourier Theory Seydel, R.: Tools for Computational Finance Zhang, F.: Matrix Theory Shafarevich, I R.: Discourses on Algebra Zong, C.: Strange Phenomena in Convex and Discrete Geometry Shapiro, J H.: Composition Operators and Classical Function Theory Simonnet, M.: Measures and Probabilities Zong, C.: Sphere Packings Zorich, V A.: Mathematical Analysis I Zorich, V A.: Mathematical Analysis II .. .J Fr´ed´eric Bonnans · J Charles Gilbert Claude Lemar´echal · Claudia A Sagastiz´abal Numerical Optimization Theoretical and Practical Aspects Second Edition With 52 Figures J Fr´ed´eric Bonnans. .. type Lij (xi , xj ) = λij (|xi − xj | − dij )2 – There is also a Van der Waals energy, say Vij (xi , xj ) = vij δij |xi − xj | − wij δij |xi − xj | 12 Here, the λij , vij , wij , dij , δij ’s... analytical formula, but only pointwise: the only available information is the numerical value of q(t) for each numerical value of t; most often, we will also assume that the numerical value of