Improving Performance of The Interior Point Method by Preconditioning Mid-Point Status Report Project by: Ken Ryals For: AMSC 663-664 Fall 27-Spring 28 6 December 27
Background / Refresher The IPM method solves a sequence of optimization problems using penalty functions such that the sequence of solutions approaches the true solution from within the valid region. As the penalty functions are relaxed and the problem is re-solved, the numerical properties of the problem become more interesting as the system approaches the true constrained optimization problem. μ 12/12/27 2
Application Why is the IPM method of interest? It applies to a wide range of problem types: Linear Constrained Optimization Semidefinite Problems Second Order Cone Problems Once in the good region of a solution to the set of problems in the solution path: Convergence properties are great ( quadratic ). It keeps the iterates in the valid region. Specific Research Problem: Optimization of Distributed Command and Control 12/12/27 3
Optimization Problem The linear optimization problem can be formulated follows: inf{ c T x Ax = b}. The search direction is implicitly defined by the system: Δx + πδz = r A Δx = A T Δy + Δz =. For this, the Reduced Equation is: A π A T Δy = Ar (= b) From Δy we can get Δx = r π ( A T Δy ). x is the unknown y is the dual of x z is the slack Def: π = D D, where: π z = x, so D is the metric geometric mean of X and Z 1 From these three equations, the Reduced Equations for Δy are: A π A T Δy = Ar (= b) 12/12/27 4
Reminder The Math Behind It All We are solving: A π A T Δy = Ar (= b) A is not square, so it isn t invertible; but AA T is What if we pre-multiplied by (AA T ) -1? (AA T ) -1 A π A T Δy = (AA T ) -1 Ar Conceptually, we have: (AT) -1 π A T Δy = (AAT) -1 b Since, this looks like a similarity transform, it might have nice properties 12/12/27 5
The Project Goal: Develop a more stable LP IPM solver. Develop a Matlab system to apply the IPM method using the preconditioned conjugate gradient solver for the linear system of equations using (AA T ) -1 as the preconditionner. Also, incorporate the stability benefits of factorization in the system. Time permitting, apply one speed improvement to Matlab solver. Application: Solve the OSD A&T distributed command and control problem. 12/12/27 6
Development Test Problems A simple canonical problem is available for use during development. min (-x 1-2x 2 ) subject to the following constraints: -2x 1 + x 2 + x 3 = 2 -x 1 + 2x 2 + x 4 = 7 x 1 + 2x 2 + x 5 = 3 x 1 ; x 2 ; x 3 ; x 4 ; x 5 for which the closed form solution is: x 2 1½ x 1 = 3-2x 2 x 5 = x 3 = 8-5x 2 x 4 = 1-4x 2 The AFIRO problem has been identified and a version obtained for use during development as a suitable test case until data for the OSD A&T application is available. Published solutions exist from several standard solvers. 12/12/27 7
The AFIRO Test Problem X: 51 parameters C: depends on 5 of 51 parameters A: 27 constraint equations (12 non-zeros) 1 Sparsity Pattern for Matrix A 1 19 27 1 14 27 4 51 12/12/27 8
Intermediate Results - AFIRO Started with all 51 values of x and z Ended with 31 x s and 22 z s = 2 parameters (15 & 17) had x=z= Condition Numbers Initial Iteration 15 Iteration 3 D 2 8523 1.36e+22 6.9e+35 AD 2 A T 1211 1.73e+2 1.44e+34 QR(AD 2 A T ) 11 1.3e+1 1.99e+17 Parameter Values 5 4 3 2 1 Solution Parameter Paths 4 8 12 16 2 24 28 32 Iteration Number 12/12/27 9
Development Process Flow Start Develop Basic IPM System in Matlab Obtain Test Data Done Planned No Yes Yes Add Preconditionner to Basic Matlab IPM Solution Match? Time for Development? Add PCG Solver No Yes Yes No Solution Match? Add Factorization Solver Time for Development? Solution Match? No Yes No Conduct Final V&V Obtain OSD A&T Data Test on OSD A&T Data Identify Areas for Speed Increase 12/12/27 1
Schedule / Progress Fall 27 Task Name Obtain AFIRO Data Develop Basic IPM System in Matlab Test Code Add Preconditioner to Basic Matlab IPM Test Code Brief Fall 27 Progress Duration (days) 5 15 2 15 2 1 Plan Dates Start End 1-Oct-27 8-Oct-27 9-Oct-27 2-Nov-27 5-Nov-27 7-Nov-27 8-Nov-27 6-Dec-27 7-Dec-27 11-Dec-27 12-Dec-27 13-Dec-27 Add PCG Solver 15 14-Dec-27 24-Jan-28 Spring 28 Test Code Add Factorization Solver Test Code Conduct V&V Test on OSD/A&T Data Identify Areas for Speed Improvements Incorporate One Speed improvement 2 15 2 15 1 5 15 25-Jan-28 3-Jan-28 21-Feb-28 26-Feb-28 19-Mar-28 3-Apr-28 11-Apr-28 29-Jan-28 2-Feb-28 25-Feb-28 18-Mar-28 2-Apr-28 1-Apr-28 2-May-28 Conduct Incremental V&V 3 5-May-28 8-May-28 Update OSD/A&T Testing 2 9-May-28 13-May-28 Brief Spring 28 Progress 2 14-May-28 15-May-28 12/12/27 11
Status - Summary In summary, the development is slightly ahead of schedule Caveat: When things appear to be going well, it proves that you don t know how things are really going. Risk area: Obtaining the OSD A&T data, even in a sanitized form, may be difficult due to delays in the parent OSD A&T project resulting from the delay in Congress passing a DoD appropriations bill. Mitigation Strategy: The following NETLIB LP test problems are of appropriate dimension to use as testing surrogates: KB2, SC5A, SC5B, ADLITTLE (in increasing dimension) 12/12/27 12
Backup Material 12/12/27 13
Developing The System Of Equations (1/3) (Primal) Problem: Min c T x Dual Problem Max b T y subject to Ax=b with x subject to A T y c Alternately, subject to A T y+z=c with z Penalty function augmented version: Min B(x; µ) = c T x - µ ln x i Optimality Conditions: c - µx -1 e - A T y = Ax - b = 12/12/27 14
Developing The System Of Equations (2/3) Collecting all these conditions: Ax - b = A T y+z=c z x c - µx -1 e - A T y = Xz = µe This produces the system of equations: Xz - µe = Ax - b = A T y+z - c = 12/12/27 15
12/12/27 16 Developing The System Of Equations (3/3) Solve this system using Newton s method: Newton s method increments x by: J(x)Δx = -gradient(x) So, the Newton step is: If we multiply the first equation by X -1 we get: Δx + (X -1 Z) Δz = µ X -1 e Z Δx + π Δz = r Similarly, the next two lines produce: A Δx = A T Δy + Δz = = I A A X Z J T : is Jacobian The = Δ Δ Δ z y A c - Ax b - Xz µe - T z y x I A A X Z T