Optimal control theory applications to management science and economics, 3rd edition

SethiOptimal Control Theory Applications to Management Science and Economics Third Edition 123... Preface to Second EditionThe first edition of this book, which provided an introduction t

Optimal Control Theory

Suresh P Sethi

Optimal Control Theory

Applications to Management Science and Economics

Third Edition

123

Suresh P Sethi

Jindal School of Management, SM30

University of Texas at Dallas

© Springer Nature Switzerland AG 2019

This work is subject to copyright All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

This book is dedicated to the memory of

my parents Manak Bai and Gulab Chand Sethi

Preface to Third Edition

The third edition of this book will not see my co-author Gerald L.Thompson, who very sadly passed away on November 9, 2009 Gerryand I wrote the first edition of the 1981 book sitting practically side byside, and I learned a great deal about book writing in the process Hewas also my PhD supervisor and mentor and he is greatly missed.After having used the second edition of the book in the classroomfor many years, the third edition arrives with new material and manyimprovements Examples and exercises related to the interpretation ofthe adjoint variables and Lagrange multipliers are inserted in Chaps.2

4 Direct maximum principle is now discussed in detail in Chap.4alongwith the existing indirect maximum principle from the second edition.Chattering or relaxed controls leading to pulsing advertising policies areintroduced in Chap.7 An application to information systems involvingchattering controls is added as an exercise

The objective function in Sect.11.1.3is changed to the more popularobjective of maximizing the total discounted society’s utility of consump-tion Further discussion leading to obtaining a saddle-point path on thephase diagram leading to the long-run stationary equilibrium is provided

in Sect.11.2 For this purpose, a global saddle-point theorem is stated

in Appendix D.7 Also inserted in Appendix D.8 is a discussion of theSethi-Skiba points which lead to nonunique stable equilibria Finally,

a new Sect.11.4 contains an adverse selection model with continuum ofthe agent types in a principal-agent framework, which requires an appli-cation of the maximum principle

Chapter 12 of the second edition is removed except for the material

on differential games and the distributed parameter maximum principle.The differential game material joins new topics of stochastic Nash differ-ential games and Stackelberg differential games via their applications tomarketing to form a new Chap.13titled Differential Games As a result,Chap 13 of the second edition becomes Chap.12 The material on thedistributed parameter maximum principle is now Appendix D.9

The exposition is revised in some places for better reading Newexercises are added and the list of references is updated Needless to say,the errors in the second edition are corrected, and the notation is madeconsistent

vii

viii Preface to Third Edition

Thanks are due to Huseyin Cavusoglu, Andrei Dmitruk, Gustav ichtinger, Richard Hartl, Yonghua Ji, Subodha Kumar, Sirong Lao, Hel-mut Maurer, Ernst Presman, Anyan Qi, Andrea Seidl, Atle Seierstad,

Fe-Xi Shan, Lingling Shi, Fe-Xiahong Yue, and the students in my OptimalControl Theory and Applications course over the years for their sug-gestions for improvement Special thanks go to Qi (Annabelle) Fengfor her dedication in updating and correcting the forthcoming solutionmanual that went with the first edition I cannot thank Barbara Gordonand Lindsay Wilson enough for their assistance in the preparation ofthe text, solution manual, and presentation materials In addition, themeticulous copy editing of the entire book by Lindsay Wilson is muchappreciated Anshuman Chutani, Pooja Kamble, and Shivani Thakkarare also thanked for their assistance in drawing some of the figures inthe book

June 2018

Preface to Second Edition

The first edition of this book, which provided an introduction to timal control theory and its applications to management science to manystudents in management, industrial engineering, operations research andeconomics, went out of print a number of years ago Over the years wehave received feedback concerning its contents from a number of instruc-tors who taught it, and students who studied from it We have also kept

op-up with new results in the area as they were published in the literature.For this reason we felt that now was a good time to come out with anew edition While some of the basic material remains, we have madeseveral big changes and many small changes which we feel will make theuse of the book easier

The most visible change is that the book is written in Latex and thefigures are drawn in CorelDRAW, in contrast to the typewritten textand hand-drawn figures of the first edition We have also included someproblems along with their numerical solutions obtained using Excel.The most important change is the division of the material in theold Chap 3, into Chaps 3 and 4 in the new edition Chapter 3 nowcontains models having mixed (control and state) constraints, currentvalue formulations, terminal conditions and model types, while Chap 4covers the more difficult topic of pure state constraints, together withmixed constraints Each of these chapters contain new results that werenot available when the first edition was published

The second most important change is the expansion of the material inthe old Sect 12.4 on stochastic optimal control theory and its becomingthe new Chap 13 The new Chap 12 now contains the following ad-vanced topics on optimal control theory: differential games, distributedparameter systems, and impulse control The new Chap 13 provides abrief introduction to stochastic optimal control problems It containsformulations of simple stochastic models in production, marketing andfinance, and their solutions We deleted the old Chap 11 of the firstedition on computational methods, since there are a number of excellentreferences now available on this topic Some of these references are listed

in Sect 4.2 of Chap 4 and Sect 8.3 of Chap 8

ix

x Preface to Second Edition

The emphasis of this book is not on mathematical rigor, but rather

on developing models of realistic situations faced in business and agement For that reason we have given, in Chaps 2 and 8, proofs of thecontinuous and discrete maximum principles by using dynamic program-ming and Kuhn-Tucker theory, respectively More general maximumprinciples are stated without proofs in Chaps 3, 4 and 12

man-One of the fascinating features of optimal control theory is its traordinarily wide range of possible applications We have covered some

ex-of these as follows: Chap 5 covers finance; Chap 6 considers productionand inventory problems; Chap 7 covers marketing problems; Chap 9treats machine maintenance and replacement; Chap 10 deals with prob-lems of optimal consumption of natural resources (renewable or ex-haustible); and Chap 11 discusses a number of applications of controltheory to economics The contents of Chaps 12 and 13 have been de-scribed earlier

Finally, four appendices cover either elementary material, such asthe theory of differential equations, or very advanced material, whoseinclusion in the main text would interrupt its continuity At the end

of the book is an extensive but not exhaustive bibliography of relevantmaterial on optimal control theory including surveys of material devoted

to specific applications

We are deeply indebted to many people for their part in making thisedition possible Onur Arugaslan, Gustav Feichtinger, Neil Geismar,Richard Hartl, Steffen Jørgensen, Subodha Kumar, Helmut Maurer, Ger-hard Sorger, and Denny Yeh made helpful comments and suggestionsabout the first edition or preliminary chapters of this revision Manystudents who used the first edition, or preliminary chapters of this revi-sion, also made suggestions for improvements We would like to expressour gratitude to all of them for their help In addition we express ourappreciation to Eleanor Balocik, Frank (Youhua) Chen, Feng Cheng,Howard Chow, Barbara Gordon, Jiong Jiang, Kuntal Kotecha, MingTam, and Srinivasa Yarrakonda for their typing of the various drafts ofthe manuscript They were advised by Dirk Beyer, Feng Cheng, Sub-odha Kumar, Young Ryu, Chelliah Sriskandarajah, Wulin Suo, HouminYan, Hanqin Zhang, and Qing Zhang on the technical problems of usingLATEX

We also thank our wives and children—Andrea, Chantal, Anjuli,Dorothea, Allison, Emily, and Abigail—for their encouragement and un-derstanding during the time-consuming task of preparing this revision

Preface to Second Edition xi

Finally, while we regret that lack of time and pressure of other ties prevented us from bringing out a second edition soon after the firstedition went out of print, we sincerely hope that the wait has been worth-while In spite of the numerous applications of optimal control theorywhich already have been made to areas of management science and eco-nomics, we continue to believe there is much more that remains to bedone We hope the present revision will rekindle interest in furtheringsuch applications, and will enhance the continued development in thefield

January 2000

Preface to First Edition

The purpose of this book is to exposit, as simply as possible, somerecent results obtained by a number of researchers in the application ofoptimal control theory to management science We believe that these re-sults are very important and deserve to be widely known by managementscientists, mathematicians, engineers, economists, and others Becausethe mathematical background required to use this book is two or threesemesters of calculus plus some differential equations and linear algebra,the book can easily be used to teach a course in the junior or seniorundergraduate years or in the early years of graduate work For thispurpose, we have included numerous worked-out examples in the text,

as well as a fairly large number of exercises at the end of each chapter.Answers to selected exercises are included in the back of the book Asolutions manual containing completely worked-out solutions to all ofthe 205 exercises is also available to instructors

The emphasis of the book is not on mathematical rigor, but on eling realistic situations faced in business and management For thatreason, we have given in Chaps 2 and 7 only heuristic proofs of the con-tinuous and discrete maximum principles, respectively In Chap 3 wehave summarized, as succinctly as we can, the most important modeltypes and terminal conditions that have been used to model manage-ment problems We found it convenient to put a summary of almost allthe important management science models on two pages: see Tables 3.1and 3.3

mod-One of the fascinating features of optimal control theory is the traordinarily wide range of its possible applications We have tried tocover a wide variety of applications as follows: Chap 4 covers finance;Chap 5 considers production and inventory; Chap 6 covers marketing;Chap 8 treats machine maintenance and replacement; Chap 9 deals withproblems of optimal consumption of natural resources (renewable or ex-haustible); and Chap 10 discusses several economic applications

ex-In Chap 11 we treat some computational algorithms for solving timal control problems This is a very large and important area thatneeds more development

op-xiii

xiv Preface to First Edition

Chapter 12 treats several more advanced topics of optimal trol: differential games, distributed parameter systems, optimal filtering,stochastic optimal control, and impulsive control We believe that some

con-of these models are capable con-of wider applications and further theoreticaldevelopment

Finally, four appendixes cover either elementary material, such asdifferential equations, or advanced material, whose inclusion in the maintext would spoil its continuity Also at the end of the book is a bibliogra-phy of works actually cited in the text While it is extensive, it is by nomeans an exhaustive bibliography of management science applications

of optimal control theory Several surveys of such applications, whichcontain many other important references, are cited

We have benefited greatly during the writing of this book by ing discussions with and obtaining suggestions from various colleaguesand students Our special thanks go to Gustav Feichtinger for his care-ful reading and suggestions for improvement of the entire book CarlNorstr¨om contributed two examples to Chaps.4 and 5 and made manysuggestions for improvement Jim Bookbinder used the manuscript for

hav-a course hav-at the University of Toronto, hav-and Tom Morton suggested someimprovements for Chap 5 The book has also benefited greatly from var-ious coauthors with whom we have done research over the years Both of

us also have received numerous suggestions for improvements from thestudents in our applied control theory courses taught during the pastseveral years We would like to express our gratitude to all these peoplefor their help

The book has gone through several drafts, and we are greatly debted to Eleanor Balocik and Rosilita Jones for their patience andcareful typing

in-Although the applications of optimal control theory to managementscience are recent and many fascinating applications have already beenmade, we believe that much remains to be done We hope that this bookwill contribute to the popularity of the area and will enhance futuredevelopments

August 1981

1 What Is Optimal Control Theory? 1

1.1 Basic Concepts and Definitions 2

1.2 Formulation of Simple Control Models 4

1.3 History of Optimal Control Theory 9

1.4 Notation and Concepts Used 11

1.4.1 Differentiating Vectors and Matrices with Respect To Scalars 12

1.4.2 Differentiating Scalars with Respect to Vectors 13 1.4.3 Differentiating Vectors with Respect to Vectors 14 1.4.4 Product Rule for Differentiation 16

1.4.5 Miscellany 16

1.4.6 Convex Set and Convex Hull 20

1.4.7 Concave and Convex Functions 20

1.4.8 Affine Function and Homogeneous Function of Degree k 22

1.4.9 Saddle Point 22

1.4.10 Linear Independence and Rank of a Matrix 23

1.5 Plan of the Book 23

2 The Maximum Principle: Continuous Time 27 2.1 Statement of the Problem 27

2.1.1 The Mathematical Model 28

2.1.2 Constraints 28

2.1.3 The Objective Function 29

2.1.4 The Optimal Control Problem 29

2.2 Dynamic Programming and the Maximum Principle 32

2.2.1 The Hamilton-Jacobi-Bellman Equation 32

2.2.2 Derivation of the Adjoint Equation 36

xv

xvi CONTENTS

2.2.3 The Maximum Principle 39

2.2.4 Economic Interpretations of the Maximum Principle 40

2.3 Simple Examples 42

2.4 Sufficiency Conditions 53

2.5 Solving a TPBVP by Using Excel 57

3 The Maximum Principle: Mixed Inequality Constraints 69 3.1 A Maximum Principle for Problems with Mixed Inequality Constraints 70

3.2 Sufficiency Conditions 79

3.3 Current-Value Formulation 80

3.4 Transversality Conditions: Special Cases 86

3.5 Free Terminal Time Problems 93

3.6 Infinite Horizon and Stationarity 103

3.7 Model Types 109

4 The Maximum Principle: Pure State and Mixed Inequality Constraints 125 4.1 Jumps in Marginal Valuations 127

4.2 The Optimal Control Problem with Pure and Mixed Constraints 129

4.3 The Maximum Principle: Direct Method 132

4.4 Sufficiency Conditions: Direct Method 136

4.5 The Maximum Principle: Indirect Method 137

4.6 Current-Value Maximum Principle: Indirect Method 147

5 Applications to Finance 159 5.1 The Simple Cash Balance Problem 160

5.1.1 The Model 160

5.1.2 Solution by the Maximum Principle 161

5.2 Optimal Financing Model 164

5.2.1 The Model 165

5.2.2 Application of the Maximum Principle 167

5.2.3 Synthesis of Optimal Control Paths 170

5.2.4 Solution for the Infinite Horizon Problem 180

CONTENTS xvii

6 Applications to Production and Inventory 191

6.1 Production-Inventory Systems 192

6.1.1 The Production-Inventory Model 192

6.1.2 Solution by the Maximum Principle 193

6.1.3 The Infinite Horizon Solution 196

6.1.4 Special Cases of Time Varying Demands 197

6.1.5 Optimality of a Linear Decision Rule 200

6.1.6 Analysis with a Nonnegative Production Constraint 202

6.2 The Wheat Trading Model 204

6.2.1 The Model 205

6.2.2 Solution by the Maximum Principle 206

6.2.3 Solution of a Special Case 206

6.2.4 The Wheat Trading Model with No Short-Selling 208 6.3 Decision Horizons and Forecast Horizons 213

6.3.1 Horizons for the Wheat Trading Model with No Short-Selling 214

6.3.2 Horizons for the Wheat Trading Model with No Short-Selling and a Warehousing Constraint 214

7 Applications to Marketing 225 7.1 The Nerlove-Arrow Advertising Model 226

7.1.1 The Model 226

7.1.2 Solution by the Maximum Principle 228

7.1.3 Convex Advertising Cost and Relaxed Controls 232 7.2 The Vidale-Wolfe Advertising Model 235

7.2.1 Optimal Control Formulation for the Vidale-Wolfe Model 236

7.2.2 Solution Using Green’s Theorem When Q Is Large 237

7.2.3 Solution When Q Is Small 245

7.2.4 Solution When T Is Infinite 247

8 The Maximum Principle: Discrete Time 259 8.1 Nonlinear Programming Problems 259

8.1.1 Lagrange Multipliers 260

8.1.2 Equality and Inequality Constraints 262

8.1.3 Constraint Qualification 267

8.1.4 Theorems from Nonlinear Programming 268

xviii CONTENTS

8.2 A Discrete Maximum Principle 269

8.2.1 A Discrete-Time Optimal Control Problem 269

8.2.2 A Discrete Maximum Principle 270

8.2.3 Examples 272

8.3 A General Discrete Maximum Principle 276

9 Maintenance and Replacement 283 9.1 A Simple Maintenance and Replacement Model 284

9.1.1 The Model 284

9.1.2 Solution by the Maximum Principle 285

9.1.3 A Numerical Example 287

9.1.4 An Extension 289

9.2 Maintenance and Replacement for a Machine Subject to Failure 290

9.2.1 The Model 291

9.2.2 Optimal Policy 293

9.2.3 Determination of the Sale Date 296

9.3 Chain of Machines 297

9.3.1 The Model 297

9.3.2 Solution by the Discrete Maximum Principle 299

9.3.3 Special Case of Bang-Bang Control 301

9.3.4 Incorporation into the Wagner-Whitin Framework for a Complete Solution 301

9.3.5 A Numerical Example 302

10 Applications to Natural Resources 311 10.1 The Sole-Owner Fishery Resource Model 312

10.1.1 The Dynamics of Fishery Models 312

10.1.2 The Sole Owner Model 313

10.1.3 Solution by Green’s Theorem 314

10.2 An Optimal Forest Thinning Model 317

10.2.1 The Forestry Model 317

10.2.2 Determination of Optimal Thinning 318

10.2.3 A Chain of Forests Model 321

10.3 An Exhaustible Resource Model 324

10.3.1 Formulation of the Model 324

10.3.2 Solution by the Maximum Principle 327

CONTENTS xix

11 Applications to Economics 335

11.1 Models of Optimal Economic Growth 335

11.1.1 An Optimal Capital Accumulation Model 336

11.1.2 Solution by the Maximum Principle 336

11.1.3 Introduction of a Growing Labor Force 338

11.1.4 Solution by the Maximum Principle 339

11.2 A Model of Optimal Epidemic Control 343

11.2.1 Formulation of the Model 343

11.2.2 Solution by Green’s Theorem 344

11.3 A Pollution Control Model 346

11.3.1 Model Formulation 347

11.3.2 Solution by the Maximum Principle 348

11.3.3 Phase Diagram Analysis 349

11.4 An Adverse Selection Model 352

11.4.1 Model Formulation 352

11.4.2 The Implementation Problem 353

11.4.3 The Optimization Problem 354

11.5 Miscellaneous Applications 360

12 Stochastic Optimal Control 365 12.1 Stochastic Optimal Control 366

12.2 A Stochastic Production Inventory Model 370

12.2.1 Solution for the Production Planning Problem 372 12.3 The Sethi Advertising Model 375

12.4 An Optimal Consumption-Investment Problem 377

12.5 Concluding Remarks 383

13 Differential Games 385 13.1 Two-Person Zero-Sum Differential Games 386

13.2 Nash Differential Games 387

13.2.1 Open-Loop Nash Solution 388

13.2.2 Feedback Nash Solution 388

13.2.3 An Application to Common-Property Fishery Resources 389

13.3 A Feedback Nash Stochastic Differential Game in Advertising 392

13.4 A Feedback Stackelberg Stochastic Differential Game of Cooperative Advertising 395

xx CONTENTS

A Solutions of Linear Differential Equations 409

A.1 First-Order Linear Equations 409

A.2 Second-Order Linear Equations with Constant Coefficients 410

A.3 System of First-Order Linear Equations 410

A.4 Solution of Linear Two-Point Boundary Value Problems 413 A.5 Solutions of Finite Difference Equations 414

A.5.1 Changing Polynomials in Powers of k into Factorial Powers of k 415

A.5.2 Changing Factorial Powers of k into Ordinary Powers of k 416

B Calculus of Variations and Optimal Control Theory 419 B.1 The Simplest Variational Problem 420

B.2 The Euler-Lagrange Equation 421

B.3 The Shortest Distance Between Two Points on the Plane 424 B.4 The Brachistochrone Problem 424

B.5 The Weierstrass-Erdmann Corner Conditions 427

B.6 Legendre’s Conditions: The Second Variation 428

B.7 Necessary Condition for a Strong Maximum 429

B.8 Relation to Optimal Control Theory 430

C An Alternative Derivation of the Maximum Principle 433 C.1 Needle-Shaped Variation 434

C.2 Derivation of the Adjoint Equation and the Maximum Principle 436

D Special Topics in Optimal Control 441 D.1 The Kalman Filter 441

D.2 Wiener Process and Stochastic Calculus 444

D.3 The Kalman-Bucy Filter 447

D.4 Linear-Quadratic Problems 448

D.4.1 Certainty Equivalence or Separation Principle 451

D.5 Second-Order Variations 452

D.6 Singular Control 454

CONTENTS xxi

D.7 Global Saddle Point Theorem 456

D.8 The Sethi-Skiba Points 458

D.9 Distributed Parameter Systems 460

E Answers to Selected Exercises 465

List of Figures

1.1 The Brachistochrone problem 9

1.2 Illustration of left and right limits 18

1.3 A concave function 21

1.4 An illustration of a saddle point 23

2.1 An optimal path in the state-time space 34

2.2 Optimal state and adjoint trajectories for Example 2.2 44 2.3 Optimal state and adjoint trajectories for Example 2.3 46 2.4 Optimal trajectories for Examples 2.4 and 2.5 48

2.5 Optimal control for Example 2.6 53

2.6 The flowchart for Example 2.8 58

2.7 Solution of TPBVP by excel 60

2.8 Water reservoir of Exercise 2.18 63

3.1 State and adjoint trajectories in Example 3.4 93

3.2 Minimum time optimal response for Example 3.6 101

4.1 Feasible state space and optimal state trajectory for Examples 4.1 and 4.4 128

4.2 State and adjoint trajectories in Example 4.3 143

4.3 Adjoint trajectory for Example 4.4 147

4.4 Two-reservoir system of Exercise 4.8 151

4.5 Feasible space for Exercise 4.28 157

5.1 Optimal policy shown in (λ1, λ2) space . 163

5.2 Optimal policy shown in (t, λ2/λ1) space . 164

5.3 Case A: g ≤ r 169

5.4 Case B: g > r 170

5.5 Optimal path for case A: g ≤ r 174

5.6 Optimal path for case B: g > r 179

xxiii

xxiv LIST OF FIGURES

5.7 Solution for Exercise 5.4 186

5.8 Adjoint trajectories for Exercise 5.5 187

6.1 Solution of Example 6.1 with I0 = 10 . 199

6.2 Solution of Example 6.1 with I0 = 50 . 199

6.3 Solution of Example 6.1 with I0 = 30 . 200

6.4 Optimal production rate and inventory level with different initial inventories 204

6.5 The price trajectory (6.56) 207

6.6 Adjoint variable, optimal policy and inventory in the wheat trading model 209

6.7 Adjoint trajectory and optimal policy for the wheat trad-ing model 212

6.8 Decision horizon and optimal policy for the wheat trading model 215

6.9 Optimal policy and horizons for the wheat trading model with no short-selling and a warehouse constraint 216

6.10 Optimal policy and horizons for Example 6.3 218

6.11 Optimal policy and horizons for Example 6.4 219

7.1 Optimal policies in the Nerlove-Arrow model 230

7.2 A case of a time-dependent turnpike and the nature of optimal control 231

7.3 A near-optimal control of problem (7.15) 233

7.4 Feasible arcs in (t, x)-space 238

7.5 Optimal trajectory for Case 1: x0 ≤ x s and x T ≤ x s 240

7.6 Optimal trajectory for Case 2: x0 < x s and x T > x s 241

7.7 Optimal trajectory for Case 3: x0 > x s and x T < x s 241

7.8 Optimal trajectory for Case 4: x0 > x s and x T > x s 242

7.9 Optimal trajectory (solid lines) 243

7.10 Optimal trajectory when T is small in Case 1: x0 < x s and x T > x s 243

7.11 Optimal trajectory when T is small in Case 2: x0 > x s and x T > x s 244

7.12 Optimal trajectory for Case 2 of Theorem 7.1 for Q = ∞ 244 7.13 Optimal trajectories for x(0) < ˆ x 249

7.14 Optimal trajectory for x(0) > ˆ x 250

8.1 Shortest distance from point (2,2) to the semicircle 266

8.2 Graph of Example 8.5 267

LIST OF FIGURES xxv

8.3 Discrete-time conventions 270

8.4 Optimal state x k ∗ and adjoint λ k 275

9.1 Optimal maintenance and machine resale value 289

9.2 Sat function optimal control 291

10.1 Optimal policy for the sole owner fishery model 316

10.2 Singular usable timber volume ¯x(t) 320

10.3 Optimal thinning u ∗ (t) and timber volume x ∗ (t) for the

forest thinning model when x0< ¯ x(t0) . 320

10.4 Optimal thinning u ∗ (t) and timber volume x ∗ (t) for the

chain of forests model when T > ˆ t 322

10.5 Optimal thinning and timber volume x ∗ (t) for the chain

of forests model when T ≤ ˆt 323

10.6 The demand function 324

10.7 The profit function 326

10.8 Optimal price trajectory for T ≥ ¯ T 329

10.9 Optimal price trajectory for T < ¯ T 330

11.1 Phase diagram for the optimal growth model 340

11.2 Optimal trajectory when x T > x s 346

11.3 Optimal trajectory when x T < x s 347

11.4 Food output function 348

11.5 Phase diagram for the pollution control model 351

11.6 Violation of the monotonicity constraint 358

11.7 Bunching and ironing 359

12.1 A sample path of optimal production rate I ∗

t with

I0= x0 > 0 and B > 0 . 374

13.1 A sample path of optimal market share trajectories 396

13.2 Optimal subsidy rate vs (a) Retailer’s margin and (b)

Manufacturer’s margin 404

B.1 Examples of admissible functions for the problem 420

B.2 Variation about the solution function 421

B.3 A broken extremal with corner at τ 428

xxvi LIST OF FIGURES

C.1 Needle-shaped variation 434

C.2 Trajectories x ∗ (t) and x(t) in a one-dimensional case . 434

D.1 Phase diagram for system (D.73) 457

D.2 Region D with boundaries Γ1 and Γ2 . 461

List of Tables

1.1 The production-inventory model of Example 1.1 4

1.2 The advertising model of Example 1.2 6

1.3 The consumption model of Example 1.3 8

3.1 Summary of the transversality conditions 89

3.2 State trajectories and switching curves 100

3.3 Objective, state, and adjoint equations for various modeltypes 111

5.1 Characterization of optimal controls with c < 1 168

13.1 Optimal feedback Stackelberg solution 403

A.1 Homogeneous solution forms for Eq (A.5) 411

A.2 Particular solutions for Eq (A.5) 411

xxvii

Chapter 1

What Is Optimal Control Theory?

Many management science applications involve the control of dynamic

systems, i.e., systems that evolve over time They are called time systems or discrete-time systems depending on whether time varies

continuous-continuously or discretely We will deal with both kinds of systems in thisbook, although the main emphasis will be on continuous-time systems

Optimal control theory is a branch of mathematics developed to find

optimal ways to control a dynamic system The purpose of this book is

to give an elementary introduction to the mathematical theory, and thenapply it to a wide variety of different situations arising in managementscience We have deliberately kept the level of mathematics as simple aspossible in order to make the book accessible to a large audience Theonly mathematical requirements for this book are elementary calculus,including partial differentiation, some knowledge of vectors and matri-ces, and elementary ordinary and partial differential equations The lasttopic is briefly covered in Appendix A Chapter 12 on stochastic opti-mal control also requires some concepts in stochastic calculus, which areintroduced at the beginning of that chapter

The principle management science applications discussed in this bookcome from the following areas: finance, economics, production and in-ventory, marketing, maintenance and replacement, and the consumption

of natural resources In each major area we have formulated one or moresimple models followed by a more complicated model The reader may

© Springer Nature Switzerland AG 2019

S P Sethi, Optimal Control Theory,

https://doi.org/10.1007/978-3-319-98237-3 1

1

2 1 What Is Optimal Control Theory?

wish at first to cover only the simpler models in each area to get an idea

of what could be accomplished with optimal control theory Later, thereader may wish to go into more depth in one or more of the appliedareas

Examples are worked out in most of the chapters to facilitate theexposition At the end of each chapter, we have listed exercises that thereader should solve for deeper understanding of the material presented

in the chapter Hints are supplied with some of the exercises Answers

to selected exercises are given in AppendixE

1.1 Basic Concepts and Definitions

We will use the word system as a primitive term in this book The only

property that we require of a system is that it is capable of existing in

various states Let the (real) variable x(t) be the state variable of the system at time t ∈ [0, T ], where T > 0 is a specified time horizon for the system under consideration For example, x(t) could measure the inventory level at time t, the amount of advertising goodwill at time t,

or the amount of unconsumed wealth or natural resources at time t.

We assume that there is a way of controlling the state of the system

Let the (real) variable u(t) be the control variable of the system at time t For example, u(t) could be the production rate at time t, the advertising rate at time t, etc.

Given the values of the state variable x(t) and the control variable u(t) at time t, the state equation, a differential equation,

˙x(t) = f (x(t), u(t), t), x(0) = x0, (1.1)specifies the instantaneous rate of change in the state variable, where

˙x(t) is a commonly used notation for dx(t)/dt, f is a given function of

x, u, and t, and x0 is the initial value of the state variable If we know

the initial value x0 and the control trajectory, i.e., the values of u(t) over

the whole time interval 0 ≤ t ≤ T, then we can integrate (1.1) to get

the state trajectory, i.e., the values of x(t) over the same time interval.

We want to choose the control trajectory so that the state and control

trajectories maximize the objective functional, or simply the objective function,

J =

 T

0 F (x(t), u(t), t)dt + S[x(T ), T ]. (1.2)

1.1 Basic Concepts and Definitions 3

In (1.2), F is a given function of x, u, and t, which could measure

the benefit minus the cost of advertising, the utility of consumption, thenegative of the cost of inventory and production, etc Also in (1.2), the

function S gives the salvage value of the ending state x(T ) at time T.

The salvage value is needed so that the solution will make “good sense”

at the end of the horizon

Usually the control variable u(t) will be constrained We indicate

this as

where Ω(t) is the set of feasible values for the control variable at time t.

Optimal control problems involving (1.1), (1.2), and (1.3) will betreated in Chap.2

In Chap.3, we will replace (1.3) by inequality constraints involvingcontrol variables In addition, we will allow these constraints to depend

on state variables These are called mixed inequality constraints andwritten as

g(x(t), u(t), t) ≥ 0, t ∈ [0, T ] , (1.4)

where g is a given function of u, t, and possibly x.

In addition, there may be constraints involving only state variables,but not control variables These are written as

where h is a given function of x and t Such constraints are the most

difficult to deal with, and are known as pure state inequality constraints.Problems involving (1.1), (1.2), (1.4), and (1.5) will be treated in Chap.4.Finally, we note that all of the imposed constraints limit the values

that the terminal state x(T ) may take We denote this by saying

ab-4 1 What Is Optimal Control Theory?

1.2 Formulation of Simple Control Models

We now formulate three simple models chosen from the areas of tion, advertising, and economics Our only objective here is to identifyand interpret in these models each of the variables and functions de-scribed in the previous section The solutions for each of these modelswill be given in detail in later chapters

produc-Example 1.1 A Production-Inventory Model The various quantities

that define this model are summarized in Table1.1for easy comparisonwith the other models that follow

Table 1.1: The production-inventory model of Example1.1

State variable I(t) = Inventory level

Control variable P (t) = Production rate

State equation I(t) = P (t)˙ − S(t), I(0) = I0

Objective function Maximize

State constraint I(t) ≥ 0

Control constraints 0≤ Pmin≤ P (t) ≤ Pmax

Terminal condition I(T ) ≥ Imin

Exogenous functions S(t) = Demand rate

h(I) = Inventory holding cost c(P ) = Production cost

Parameters T = Terminal time

Imin = Minimum ending inventory

Pmin= Minimum possible production rate

Pmax= Maximum possible production rate

I0 = Initial inventory level

1.2 Formulation of Simple Control Models 5

We consider the production and inventory storage of a given good,such as steel, in order to meet an exogenous demand The state variable

I(t) measures the number of tons of steel that we have on hand at time

t ∈ [0, T ] There is an exogenous demand rate S(t) tons of steel per day

at time t ∈ [0, T ], and we must choose the production rate P (t) tons of steel per day at time t ∈ [0, T ] Given the initial inventory of I0 tons of

steel on hand at t = 0, the state equation

˙

I(t) = P (t) − S(t) describes how the steel inventory changes over time Since h(I) is the cost of holding inventory I in dollars per day, and c(P ) is the cost of producing steel at rate P, also in dollars per day, the objective function

is to maximize the negative of the sum of the total holding and

produc-tion costs over the period of T days Of course, maximizing the negative

sum is the same as minimizing the sum of holding and production costs

The state variable constraint, I(t) ≥ 0, is imposed so that the demand

is satisfied for all t In other words, backlogging of demand is not mitted (An alternative formulation is to make h(I) become very large when I becomes negative, i.e., to impose a stockout penalty cost.) The control constraints keep the production rate P (t) between a specified lower bound Pmin and a specified upper bound Pmax Finally, the termi-

per-nal constraint I(T ) ≥ Imin is imposed so that the terminal inventory is

at least Imin.

The statement of the problem is lengthy because of the number ofvariables, functions, and parameters which are involved However, withthe production and inventory interpretations as given, it is not difficult

to see the reasons for each condition In Chap.6, various versions of thismodel will be solved in detail In Sect.12.2, we will deal with a stochasticversion of this model

Example 1.2 An Advertising Model The various quantities that define

this model are summarized in Table1.2

We consider a special case of the Nerlove-Arrow advertising modelwhich will be discussed in detail in Chap.7 The problem is to determine

the rate at which to advertise a product at each time t Here the state variable is advertising goodwill, G(t), which measures how well the prod- uct is known at time t We assume that there is a forgetting coefficient δ,

which measures the rate at which customers tend to forget the product

6 1 What Is Optimal Control Theory?

To counteract forgetting, advertising is carried out at a rate measured

by the control variable u(t) Hence, the state equation is

˙

G(t) = u(t) − δG(t), with G(0) = G0 > 0 specifying the initial goodwill for the product.

Table 1.2: The advertising model of Example 1.2

State variable G(t) = Advertising goodwill

Control variable u(t) = Advertising rate

State equation G(t) = u(t)˙ − δG(t), G(0) = G0

Objective function Maximize

Exogenous function π(G) = Gross profit rate

Parameters δ = Goodwill decay constant

ρ = Discount rate

Q = Upper bound on advertising rate

G0= Initial goodwill level

The objective function J requires special discussion Note that the integral defining J is from time t = 0 to time t = ∞; we will later

call a problem having an upper time limit of ∞, an infinite horizon problem Because of this upper limit, the integrand of the objective function includes the discount factor e −ρt , where ρ > 0 is the (constant)

discount rate Without this discount factor, the integral would (in mostcases) diverge to infinity Hence, we will see that such a discount factor

is an essential part of infinite horizon models The rest of the integrand

in the objective function consists of the gross profit rate π(G(t)), which

1.2 Formulation of Simple Control Models 7

results from the goodwill level G(t) at time t less the cost of advertising assumed to be proportional to u(t) (proportionality factor = 1); thus π(G(t)) − u(t) is the net profit rate at time t Also [π(G(t)) − u(t)]e −ρtis

the net profit rate at time t discounted to time 0, i.e., the present value

of the time t profit rate Hence, J can be interpreted as the total value of

discounted future profits, and is the quantity we are trying to maximize.There are control constraints 0 ≤ u(t) ≤ Q, where Q is the upper

bound on the advertising rate However, there is no state constraint Itcan be seen from the state equation and the control constraints that the

goodwill G(t) in fact never becomes negative.

You will find it instructive to compare this model with the previousone and note the similarities and differences between the two

Example 1.3 A Consumption Model Rich Rentier plans to retire at

age 65 with a lump sum pension of W0 dollars Rich estimates his

re-maining life span to be T years He wants to consume his wealth during these T retirement years, beginning at the age of 65, and leave a bequest

to his heirs in a way that will maximize his total utility of consumptionand bequest

Since he does not want to take investment risks, Rich plans to puthis money into a savings account that pays interest at a continuously

compounded rate of r In order to formulate Rich’s optimization problem, let t = 0 denote the time when he turns 65 so that his retirement period can be denoted by the interval [0, T ] If we let the state variable W (t) denote Rich’s wealth and the control variable C(t) ≥ 0 denote his rate of consumption at time t ∈ [0, T ], it is easy to see that the state equation is

˙

W (t) = rW (t) − C(t), with the initial condition W (0) = W0 > 0 It is reasonable to require that

W (t) ≥ 0 and C(t) ≥ 0, t ∈ [0, T ] Letting U(C) be the utility function

of consumption C and B(W ) be the bequest function of leaving a bequest

of amount W at time T, we see that the problem can be stated as an

optimal control problem with the variables, equations, and constraintsshown in Table 1.3

Note that the objective function has two parts: first the integral of

the discounted utility of consumption from time 0 to time T with ρ as the discount rate; and second the bequest function e −ρT B(W ), which

measures Rich’s discounted utility of leaving an estate W to his heirs

8 1 What Is Optimal Control Theory?

at time T If he has no heirs and does not care about charity, then B(W ) = 0 However, if he has heirs or a favorite charity to whom he wishes to leave money, then B(W ) measures the strength of his desire

to leave an estate of amount W The nonnegativity constraints on state

and control variables are obviously natural requirements that must beimposed

You will be asked to solve this problem in Exercise 2.1 after youhave learned the maximum principle in the next chapter Moreover, astochastic extension of the consumption problem, known as a consump-tion/investment problem, will be discussed in Sect.12.4

Table 1.3: The consumption model of Example1.3

State variable W (t) = Wealth

Control variable C(t) = Consumption rate

Control constraint C(t) ≥ 0

Exogenous U (C) = Utility of consumption

Functions B(W ) = Bequest function

Parameters T = Terminal time

W0= Initial wealth

ρ = Discount rate

r = Interest rate

1.3 History of Optimal Control Theory 9

1.3 History of Optimal Control Theory

Optimal control theory is an extension of the calculus of variations (seeAppendixB), so we discuss the history of the latter first

The creation of the calculus of variations occurred almost ately after the formalization of calculus by Newton and Leibniz in theseventeenth century An important problem in calculus is to find anargument of a function at which the function takes on its maximum orminimum The extension of this problem posed in the calculus of vari-ations is to find a function which maximizes or minimizes the value of

immedi-an integral or functional of that function As might be expected, theextremum problem in the calculus of variations is much harder than theextremum problem in differential calculus Euler and Lagrange are gen-erally considered to be the founders of the calculus of variations Newton,Legendre, and the Bernoulli brothers also contributed much to the earlydevelopment of the field

Figure 1.1: The Brachistochrone problem

A celebrated problem first solved using the calculus of variations was

the path of least time or the Brachistochrone problem The problem is

illustrated in Fig.1.1 It involves finding the shape of a curve Γ necting the two points A and B in the vertical plane with the propertythat a bead sliding along the curve under the influence of gravity willmove from A to B in the shortest possible time The problem was posed

con-10 1 What Is Optimal Control Theory?

by Johann Bernoulli in 1696, and it played an important part in thedevelopment of calculus of variations It was solved by Johann Bernoulli,Jakob Bernoulli, Newton, Leibnitz, and L’Hˆopital In Sect.B.4, we pro-vide a solution to the Brachistochrone problem by using what is known

as the Euler-Lagrange equation, stated in Sect.B.2, and show that the

shape of the solution curve is represented by a cycloid.

In the nineteenth and early twentieth centuries, many cians contributed to the calculus of variations; these include Hamilton,Jacobi, Bolza, Weierstrass, Carath´eodory, and Bliss

mathemati-Converting calculus of variations problems into control theory lems requires one more conceptual step—the addition of control variables

prob-to the state equations Isaacs (1965) made such an extension in person pursuit-evasion games in the period 1948–1955 Bellman (1957)made a similar extension with the idea of dynamic programming.Modern control theory began with the publication (in Russian in

two-1961 and English in 1962) of the book, The Mathematical Theory of Optimal Processes, by Pontryagin et al (1962) Well-known Americanmathematicians associated with the maximum principle include Valen-tine, McShane, Hestenes, Berkovitz, and Neustadt The importance ofthe book by Pontryagin et al lies not only in a rigorous formulation of

a calculus of variations problem with constrained control variables, butalso in the proof of the maximum principle for optimal control problems.See Pesch and Bulirsch (1994) and Pesch and Plail (2009) for historicalperspectives on the topics of the calculus of variations, dynamic pro-gramming, and optimal control

The maximum principle permits the decoupling of the dynamic lem over time, using what are known as adjoint variables or shadow prices, into a series of problems, each of which holds at a single instant

prob-of time The optimal solution prob-of the instantaneous problems can beshown to give the optimal solution to the overall problem

In this book we will be concerned principally with the application ofthe maximum principle in its various forms to find the solutions of a widevariety of applied problems in management science and economics It ishoped that the reader, after reading some of these problems and theirsolutions, will appreciate, as we do, the importance of the maximumprinciple

Some important books and surveys of the applications of themaximum principle to management science and economics are Con-

1.4 Notation and Concepts Used 11

nors and Teichroew (1967), Arrow and Kurz (1970), Hadley andKemp (1971), Bensoussan et al (1974), St¨oppler (1975), Clark (1976),Sethi (1977a, 1978a), Tapiero (1977, 1988), Wickwire (1977), Book-binder and Sethi (1980), Lesourne and Leban (1982), Tu (1984), Fe-ichtinger and Hartl (1986), Carlson and Haurie (1987b), Seierstadand Sydsæter (1987), Erickson (2003), L´eonard and Long (1992),Kamien and Schwartz (1992), Van Hilten et al (1993), Feichtinger

et al (1994a), Maimon et al (1998), Dockner et al (2000), puto (2005), Grass et al (2008), and Bensoussan (2011) Nev-ertheless, we have included in our bibliography many works ofinterest

Ca-1.4 Notation and Concepts Used

In order to make the book readable, we will adopt the following notationwhich will hold throughout the book In addition, we will define someimportant concepts that are required, including those of concave, convexand affine functions, and saddle points

We use the symbol “=” to mean “is equal to” or “is defined to beequal to” or “is identically equal to” depending on the context Thesymbol “:=” means “is defined to be equal to,” the symbol “≡” means

“is identically equal to,” and the symbol “≈” means “is approximately

equal to.” The double arrow “⇒” means “implies,” “∀” means “for all,”

and “∈” means “is a member of.” The symbol 2 indicates the end of a

proof

Let y be an n-component column vector and z be an m-component

row vector, i.e.,

where the superscriptT on a vector (or, a matrix) denotes the transpose

of the vector (or, the matrix) At times, when convenient and not fusing, we will use the superscript for the transpose operation If y and

con-12 1 What Is Optimal Control Theory?

z are functions of time t, a scalar, then the time derivatives ˙ y := dy/dt and ˙z := dz/dt are defined as

˙y = dy

dt = ( ˙y1, · · · , ˙y n)T and ˙z = dz

dt = ( ˙z1, , ˙z m ), where ˙y i and ˙z j denote the time derivatives dy i /dt and dz j /dt, respec-

be appropriately differentiable for their derivatives being defined

To Scalars

Let f : E1 → E k be a k-dimensional function of a scalar variable t If f

is a row vector, then we define

df

dt = f t = (f 1t , f 2t , · · · , f kt ), a row vector.

1.4 Notation and Concepts Used 13

We will also use the notation f  = (f1 , f2 , · · · , f k  ) and f  (t) in place of f t

If f is a column vector, then

Once again, f (t) may also be written as f  or f  (t).

A similar rule applies if a matrix function is differentiated with spect to a scalar

If F (y, z) is a scalar function defined on E n ×E m with y an n-dimensional column vector and z an m-dimensional row vector, then the gradients

F y and F z are defined, respectively, as

F y = (F y1, · · · , F y n ), a row vector, (1.9)and

F z = (F z1, · · · , F z m ), a row vector, (1.10)

where F y i and F z j denote the partial derivatives with respect to thesubscripted variables

Thus, we always define the gradient with respect to a row or column

vector as a row vector Alternatively, F y and F z are also denoted as∇ y F

and ∇ z F, respectively In this notation, if F is a function of y only or

z only, then the subscript can be dropped and the gradient of F can be

written simply as ∇F.

14 1 What Is Optimal Control Theory?

Example 1.5 Let F (y, z) = y1 y3z2 + 3y2ln z1 + y1y2, where y =

(y1, y2, y3)T and z = (z1, z2) Obtain F y and F z

Solution F y = (F y1, F y2, F y3) = (2y1y3z2+ y2, 3 ln z1 + y1, y1 z2) and

F z = (F z1, F z2) = (3y2/z1, y1 y3).

If f : E n × E m → E k is a k-dimensional vector function, f either row or

column, i.e.,

f = (f1, · · · , f k ) or f = (f1, · · · , f k)T , where each component f i = f i (y, z) depends on the column vector y ∈ E n

and the row vector z ∈ E m , then f z will denote the k × m matrix

∂f1/∂y1 ∂f1/∂y2 · · · ∂f1/∂y n

∂f2/∂y1 ∂f2/∂y2 · · · ∂f2/∂y n

Matrices f z and f y are known as Jacobian matrices It should be

emphasized that the rule of defining a Jacobian does not depend on therow or column nature of the function or its arguments Thus,

f z = (f T)z = f z T = (f T)z T

Example 1.6 Let f : E3× E2→ E3 be defined by f (y, z) = (y

1 y3z2+

3y2ln z1, z1z2 y3, z1y1 + z2y2)T with y = (y1, y2, y3)T and z = (z1, z2) Obtain f z and f y

1.4 Notation and Concepts Used 15

Example 1.7 Obtain F yz and F zy for F (y, z) specified in Example1.5

Since the given F (y, z) is twice continuously differentiable, check also that F zy = (F yz)T

16 1 What Is Optimal Control Theory?

Solution Applying rule (1.11) to F y obtained in Example1.5and rule(1.12) to F z obtained in Example1.5, we have, respectively,

Also, it is easily seen from these matrices that F zy = (F yz)T

Let g be an n-component row vector function and f be an n-component column vector function of an n-component vector x Then in Exercise1.9,you are asked to show that

(gf ) x = gf x + f T g x = gf x + f T (g T)x (1.15)

In Exercise 1.10, you are asked to show further that with g = F x , where

x ∈ E n and the function F : E n → E1is twice continuously differentiable

so that F xx = (F xx)T , called the Hessian, then

(gf ) x =(F x f ) x =F x f x + f T F xx = F x f x + (F xx f ) T (1.16)The latter result will be used in Chap.2 for the derivation of (2.25).Many mathematical expressions in this book will be vector equations

or inequalities involving vectors and vector functions Since scalars are

a special case of vectors, these expressions hold just as well for scalarequations or inequalities involving scalars and scalar functions In fact,

it may be a good idea to read them as scalar expressions on the firstreading Then in the second and further readings, the extension to vectorform will be easier