LARGE SCALE LINEAR AND INTEGER OPTIMIZATION: A UNIFIED APPROACH

LARGE SCALE LINEAR AND INTEGER OPTIMIZATION: A UNIFIED APPROACH

LARGE SCALE LINEAR AND INTEGER OPTIMIZATION: A UNIFIED APPROACH Richard Kipp Martin Graduate School of Business Universify of Chicago ~. " Springer Science+Business Media, LLC

Library of Congress Cataloging-in-Publication Data Martin, Richard Kipp. Large scale linear and integer optimization : a united approach / Richard Kipp Martin. p. cm. Includes bibliographical references and index. ISBN 978-1-46l3-7258-5 ISBN 978-1-4615-4975-8 (ebook) DOI 10.1007/978-1-4615-4975-8 1. Linear programming. 2. Mathematical optimization. 1. Title. T57.75.M375 1999 519.7'2--dc2 1 98-46062 CIP Copyright 1999 Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 1999 Softcover reprint of the hardcover 1 st edition 1999 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, record ing, or otherwise, without the prior written permis sion of the publisher, Springer Science+Business Media, LLC. Printed an acid-free paper.

This book is dedicated to my parents, Bruce and Phyllis Martin.

CONTENTS Preface xv Part I MOTIVATION 1 1 LINEAR AND INTEGER LINEAR OPTIMIZATION 3 1.1 Introduction 3 1.2 Linear and Integer Linear Optimization 5 1.3 A Guided Tour of Applications 7 1.4 Special Structure 21 1.5 Linear and Integer Linear Programming Codes 25 1.6 Other Directions 28 1.7 Exercises 29 Part II THEORY 33 2 LINEAR SYSTEMS AND PROJECTION 35 2.1 Introduction 35 2.2 Projection for Equality Systems: Gaussian Elimination 36 2.3 Projection for Inequality Systems: Fourier-Motzkin Elimination 39 2.4 Applications of Projection 46 2.5 Theorems of the Alternative 49 2.6 Duality Theory 57 2.7 Complementary Slackness 61 2.8 Sensitivity Analysis 65 2.9 Conclusion 75 2.10 Exercises 75

Vlll LARGE SCALE LINEAR AND INTEGER OPTIMIZATION 3 LINEAR SYSTEMS AND INVERSE PROJECTION 81 3.1 Introduction 81 3.2 Deleting Constraints by Adding Variables 81 3.3 Dual Relationships 91 3.4 Sensitivity Analysis 93 3.5 Conclusion 99 3.6 Homework Exercises 100 4 INTEGER LINEAR SYSTEMS: PROJECTION AND INVERSE PROJECTION ]03 4.1 Introduction 103 4.2 Background Material 105 4.3 Solving A System of Congruence Equations 114 4.4 Integer Linear Equalities 122 4.5 Integer Linear Inequalities: Projection 124 4.6 Integer Linear Inequalities: Inverse Projection 127 4.7 Conclusion 136 4.8 Exercises 137 Part III ALGORITHMS 141 5 THE SIMPLEX ALGORITHM 143 5.1 Introduction 143 5.2 Motivation 143 5.3 Pivoting 147 5.4 Revised Simplex 154 5.5 Product Form of the Inverse 162 5.6 Degeneracy and Cycling 168 5.7 Complexity of the Simplex Algorithm 177 5.8 Conclusion 178 5.9 Exercises 178 6 MORE ON SIMPLEX 183 6.1 Introduction 183 6.2 Sensitivity Analysis 184 6.3 The Dual Simplex Algorithm 191

Contents ix 6.4 Simple Upper Bounds and Special Structure 198 6.5 Finding a Starting Basis 201 6.6 Pivot Column Selection 205 6.7 Other Computational Issues 209 6.8 Conclusion 215 6.9 Exercises 216 7 INTERIOR POINT ALGORITHMS: POLYHEDRAL TRANSFORMATIONS 219 7.1 Introduction 219 7.2 Projective Transformations 225 7.3 Karmarkar's Algorithm 231 7.4 Polynomial Termination 234 7.5 Purification, Standard Form and Sliding Objective 239 7.6 Affine Polyhedral Transformations 243 7.7 Geometry of the Least Squares Problem 254 7.8 Conclusion 258 7.9 Exercises 258 8 INTERIOR POINT ALGORITHMS: BARRIER METHODS 261 8.1 Introduction 261 8.2 Primal Path Following 266 8.3 Dual Path Following 272 8.4 Primal-Dual Path Following 277 8.5 Polynomial Termination of Path Following Algorithms 283 8.6 Relation to Polyhedral Transformation Algorithms 292 8.7 Predictor-Corrector Algorithms 297 8.8 Other Issues 300 8.9 Conclusion 306 8.10 Exercises 310 9 INTEGER PROGRAMMING 313 9.1 Introduction 313 9.2 Modeling with Integer Variables 314 9.3 Branch-and-Bound 319 9.4 Node and Variable Selection 324

x LARGE SCALE LINEAR AND INTEGER OPTIMIZATION 9.5 More General Branching 328 9.6 Conclusion 341 9.7 Exercises 341 Part IV SOLVING LARGE SCALE PROBLEMS: DECOMPOSITION METHODS 347 10 PROJECTION: BENDERS' DECOMPOSITION 349 10.1 Introduction 349 10.2 The Benders' Algorithm 350 10.3 A Location Application 354 10.4 Dual Variable Selection 360 10.5 Conclusion 364 10.6 Exercises 365 11 INVERSE PROJECTION: DANTZIG-WOLFE DECOMPOSITION 369 11.1 Introduction 369 11.2 Dantzig-Wolfe Decomposition 370 11.3 A Location Application 375 11.4 Taking Advantage of Block Angular Structure 384 11.5 Computational Issues 386 11.6 Conclusion 390 11.7 Exercises 391 12 LAGRANGIAN METHODS 393 12.1 Introduction 393 12.2 The Lagrangian Dual 394 12.3 Extension to Integer Programming 398 12.4 Properties of the Lagrangian Dual 402 12.5 Optimizing the Lagrangian Dual 408 12.6 Computational Issues 426 12.7 A Decomposition Algorithm for Integer Programming 429 12.8 Conclusion 434 12.9 Exercises 435

Contents xi Part V SOLVING LARGE SCALE PROBLEMS: USING SPECIAL STRUCTURE 437 13 SPARSE METHODS 439 13.1 Introduction 439 13.2 LU Decomposition 439 13.3 Sparse LU Update 446 13.4 Numeric Cholesky Factorization 462 13.5 Symbolic Cholesky Factorization 466 13.6 Storing Sparse Matrices 471 13.7 Programming Issues 472 13.8 Computational Results: Barrier versus Simplex 478 13.9 Conclusion 479 13.10 Exercises 480 14 NETWORK FLOW LINEAR PROGRAMS 481 14.1 Introduction 481 14.2 Totally Unimodular Linear Programs 482 14.3 Network Simplex Algorithm 493 14.4 Important Network Flow Problems 505 14.5 Almost Network Problems 513 14.6 Integer Polyhedra 515 14.7 Conclusion 523 14.8 Exercises 524 15 LARGE INTEGER PROGRAMS: PREPROCESSING AND CUTTING PLANES 527 15.1 Formulation Principles and Techniques 527 15.2 Preprocessing 533 15.3 Cutting Planes 542 15.4 Branch-and-Cut 555 15.5 Lifting 557 15.6 Lagrangian Cuts 560 15.7 Integer Programming Test Problems 561 15.8 Conclusion 562 15.9 Exercises 563

xii LARGE SCALE LINEAR AND INTEGER OPTIMIZATION 16 LARGE INTEGER PROGRAMS: PROJECTION AND INVERSE PROJECTION 565 16.1 Introduction 565 16.2 Auxiliary Variable Methods 573 16.3 A Projection Theorem 601 16.4 Branch-and-Price 601 16.5 Projection of Extended Formulations: Benders' Decomposition Revisited 613 16.6 Conclusion 630 16.7 Exercises 630 Part VI APPENDIX 633 A POLYHEDRAL THEORY 635 A.1 Introduction 635 A.2 Concepts and Definitions 635 A.3 Faces of Polyhedra 640 A.4 Finite Basis Theorems 645 A.5 Inner Products, Subspaces and Orthogonal Subspaces 651 A.6 Exercises 653 B COMPLEXITY THEORY 657 B.1 Introduction 657 B.2 Solution Sizes 660 B.3 The Turing Machine 661 B.4 Complexity Classes 663 B.5 Satisfiability 667 B.6 NP-Completeness 669 B.7 Complexity of Gaussian Elimination 670 B.8 Exercises 674 C BASIC GRAPH THEORY 677 D SOFTWARE AND TEST PROBLEMS 681 E NOTATION 683

Contents XUI References AUTHOR INDEX TOPIC INDEX 687 723 731

PREFACE This is a textbook about linear and integer linear optimization. There is a growing need in industries such as airline, trucking, and financial engineering to solve very large linear and integer linear optimization problems. Building these models requires uniquely trained individuals. Not only must they have a thorough understanding of the theory behind mathematical programming, they must have substantial knowledge of how to solve very large models in today's computing environment. The major goal of the book is to develop the theory of linear and integer linear optimization in a unified manner and then demonstrate how to use this theory in a modern computing environment to solve very large real world problems. After presenting introductory material in Part I, Part II of this book is devoted to the theory of linear and integer linear optimization. This theory is developed using two simple, but unifying ideas: projection and inverse projection. Through projection we take a system of linear inequalities and replace some of the variables with additional linear inequalities. Inverse projection, the dual of this process, involves replacing linear inequalities with additional variables. Fundamental results such as weak and strong duality, theorems of the alternative, complementary slackness, sensitivity analysis, finite basis theorems, etc. are all explained using projection or inverse projection. Indeed, a unique feature of this book is that these fundamental results are developed and explained before the simplex and interior point algorithms are presented. We feel that projection and inverse projection are the very essence of these fundamental results and proofs based upon simplex or interior point algorithms are not insightful. The ideas of projection and inverse projection are also extended to integer linear optimization. With the projection-inverse projection approach theoretical results in integer linear optimization become much more analogous to their linear optimization counterparts. Thus, once armed with these two concepts the reader is equipped to understand fundamental theorems in an intuitive way. In Part III of the book we present the most important algorithms that are used in commercial software for solving real world problems. Even though the algorithms developed in Part III are appropriate for solving large realistic

xvi LARGE SCALE LINEAR AND INTEGER OPTIMIZATION problems, they will often fail for very large scale applications unless advantage is taken of the special structure present in the problem. In Part IV we show how to take advantage of special structure through decomposition. We explain these decomposition algorithms as extensions of the projection and inverse projection concepts developed in Part II. In Part V we show how to take advantage of special structure by modifying and enhancing the algorithms developed in Part III. This section contains a discussion of some of the most current research in linear and integer programming. We also show in Part V how to take different problem formulations and appropriately "modify" them so that the algorithms from Part III are much more efficient. Once again, the projection and inverse projection concepts are used in Part V to present the current research in linear and integer linear optimization in a very unified way. An ambitious one quarter or semester course in linear optimization would include Chapters 1-3 and Chapters 5-8. This set of chapters gives the student exposure to the most important aspects of linear optimization theory as well as the simplex and barrier algorithms. If two quarters are available, then the instructor has several options. The second quarter can be devoted to a brief introduction to integer linear optimization and then move on to large scale linear optimization. This material is in Chapters 9-14. Or, if the instructor wishes to cover integer linear optimization in more depth, then Chapters 4, 9, 12, and 14-16 provide a solid background. Chapters 4, 9, 12, 14-16 provide a nice two quarter, or intense semester course in discrete optimization. No prior knowledge of linear or integer linear optimization is assumed, although this text is written for a mathematically mature audience. Our target audience is upper level undergraduate students and graduate students in computer science, applied mathematics, industrial engineering and operations research/management science. Coursework in linear algebra and analysis is sufficient background. Researchers wishing to brush up on recent developments in large scale linear and integer linear programming will also find this text useful. Readers not familiar with any polyhedral theory should read the first two sections of Appendix A. The necessary background on complexity theory is provided in Appendix B. A brief tutorial on graph theory is provided in Appendix C. This book has benefited from the help of numerous people. Former students Mohan Bala, John Borse, Peter Cacioppi, Kevin Cunningham, Zeger Degraeve, Syam Menon, Santosh Nabar, Fritz Raffensperger, Larry Robinson, Jayaram Sankaran, Jack Shi, Sri Sinnathamby, Kamaryn Tanner, Jurgen Tistaert, Mala Viswanath, and Hongsuk Yang suffered through constant revisions, found numerous errors, and provided useful suggestions for improvement. I thank James

Preface xvii Evans, William Lippert, Richard Murphy, James Powell, and the late Richard DeMar for contributing greatly to my mathematical education. I have also benefited from my colleagues Gary Eppen, Ronald Rardin, and the late Robert Jeroslow. Luis Gouveia has been particularly helpful in reading much of the manuscript and making suggestions for improvement. Most of all, I thank Dennis Sweeney for teaching me so much mathematical optimization over the years and Linus Schrage for so graciously enduring a constant barrage of questions related to topics in this book. I thank Gary Folven at Kluwer for being a patient editor and enthusiastically supporting this book. Finally, I thank my parents, Bruce and Phyllis Martin, and Gail Honda for their constant encouragement and support throughout this long project.