Optimization: Insights and Applications Jan Brinkhuis Vladimir Tikhomirov PRINCETON UNIVERSITY PRESS PRINCETON AND OXFORD
Contents Preface 0.1 Optimization: insights and applications xiii 0.2 Lunch, dinner, and dessert xiv 0.3 For whom is this book meant? xvi 0.4 What is in this book? xviii 0.5 Special features xix Necessary Conditions: What Is the Point? 1 xi Chapter 1. Fermat: One Variable without Constraints 3 1.0 Summary 3 1.1 Introduction 5 1.2 The derivative for one variable 6 1.3 Main result: Fermat theorem for one variable 14 1.4 Applications to concrete problems 30 1.5 Discussion and comments - 43 1.6 Exercises 59 Chapter 2. Fermat: Two or More Variables without Constraints 85 2.0 Summary \ 85 2.1 Introduction 87 2.2 The derivative for two or more variables 87 2.3 Main result: Fermat theorem for two or more variables 96 2.4 Applications to concrete problems 101 2.5 Discussion and comments 127 2.6 Exercises 128 Chapter 3. Lagrange: Equality Constraints 135 3.0 Summary 135 3.1 Introduction 138 3.2 Main result: Lagrange multiplier rule 140 3.3 Applications to concrete problems 152 3.4 Proof of the Lagrange multiplier rule 167 3.5 Discussion and comments 181 3.6 Exercises 190
viii Chapter 4. Inequality Constraints and Convexity 199 4.0 Summary 199 4.1 Introduction 202 4.2 Main result: Karush-Kuhn-Tucker theorem. 204 4.3 Applications to concrete problems 217 4.4 Proof of the Karush-Kuhn-Tucker theorem 229 4.5 Discussion and comments 235 4.6 Exercises 250 Chapter 5. Second Order Conditions 261 5.0 Summary 261 5.1 Introduction 262 5.2 Main result: second order conditions 262 5.3 Applications to concrete problems 267 5.4 Discussion and comments 271 5.5 Exercises 272 Chapter 6. Basic Algorithms 273 6.0 Summary 273 6.1 Introduction 275 6.2 Nonlinear optimization is difficult 278 6.3 Main methods of linear optimization 283 6.4 Line search 286 6.5 Direction of descent 299 6.6 Quality of approximation 301 6.7 Center of gravity method 304 6.8 Ellipsoid method 307 6.9 Interior point methods... 316 Chapter 7. "Advanced Algorithms 325 7.1 Introduction 325 7.2 Conjugate gradient method 325 7.3 Self-concordant barrier methods 335 Chapter 8. Economic Applications 363 8.1 Why you should not sell your house to the highest bidder 363 8.2 Optimal speed of ships and the cube law 366 8.3 Optimal discounts on airline tickets with a Saturday stayover 368 8.4 Prediction of flows of cargo 370 8.5 Nash bargaining 373 8.6 Arbitrage-free bounds for prices 378 8.7 Fair price for options: formula of Black and Scholes 380 8.8 Absence of arbitrage and existence of a martingale 381 8.9 How to take a penalty kick, and the minimax theorem 382 8.10 The best lunch and the second welfare theorem 386 Chapter 9. Mathematical Applications 391 9.1 Fun and the quest for the essence 391
ix 9.2 Optimization approach to matrices 392 9.3 How to prove results on linear inequalities 395 9.4 The problem of Apollonius 397 9.5 Minimization of a quadratic function: Sylvester's criterion and Gram's formula 409 9.6 Polynomials of least deviation 411 9.7 Bernstein inequality 414 Chapter 10. Mixed Smooth-Convex Problems 417 10.1 Introduction 417 10.2 Constraints given by inclusion in a cone 419 10.3 Main result: necessary conditions for mixed smooth-convex problems 422 10.4 Proof of the necessary conditions 430 10.5 Discussion and comments 432 Chapter 11. Dynamic Programming in Discrete Time 441 11.0 Summary 441 11.1 Introduction 443 11.2 Main result: Hamilton-Jacobi-Bellman equation 444 11.3 Applications to concrete problems 446 11.4 Exercises 471 Chapter 12. Dynamic Optimization in Continuous Time 475 12.1 Introduction 475 12.2 Main results: necessary conditions of Euler, Lagrange, Pontryagin, and Bellman 478 12.3 Applications to concrete problems - 492 12.4 Discussion and comments 498 Appendix A. On Linear Algebra: Vector and Matrix Calculus 503 A.I Introduction \ 503 A.2 Zero-sweeping or Gaussian elimination, and a formula for the dimension of the solution set 503 A.3 Cramer's rule 507 A.4 Solution using the inverse matrix 508 A.5 Symmetric matrices 510 A.6 Matrices of maximal rank 512 A.7 Vector notation 512 A.8 Coordinate free approach to vectors and matrices 513 Appendix B. On Real Analysis 519 B.I Completeness of the real numbers 519 B.2 Calculus of differentiation 523 B.3 Convexity 528 B.4 Differentiation and integration 535 Appendix C. The Weierstrass Theorem on Existence of Global Solutions 537
x C.I On the use of the Weierstrass theorem 537 C.2 Derivation of the Weierstrass theorem 544 Appendix D. Crash Course on Problem Solving 547 D.I One variable without constraints 547 D.2 Several variables without constraints 548 D.3 Several variables under equality constraints 549 D.4 Inequality constraints and convexity 550 Appendix E. Crash Course on Optimization Theory: Geometrical Style 553 E.I The main points 553 E.2 Unconstrained problems 554 E.3 Convex problems 554 E.4 Equality constraints 555 E.5 Inequality constraints 556 E.6 Transition to infinitely many variables 557 Appendix F. Crash Course on Optimization Theory: Analytical Style 561 F.I Problem types 561 F.2 Definitions of differentiability 563 F.3 Main theorems of differential and convex calculus 565 F.4 Conditions that are necessary and/or sufficient 567 F.5 Proofs 571 Appendix G. Conditions of Extremum from Fermat to Pontryagin 583 G.I Necessary first order conditions from Fermat to Pontryagin 583 G.2 Conditions of extremum of the second order - 593 Appendix H. Solutions of Exercises of Chapters 1-4 601 Bibliography 645 Index 651