Solving the normal equations system arising from interior po. linear programming by iterative methods

Size: px

Start display at page:

Download "Solving the normal equations system arising from interior po. linear programming by iterative methods"

Eleanore Pitts
5 years ago
Views:

1 Solving the normal equations system arising from interior point methods for linear programming by iterative methods Aurelio Oliveira - aurelio@ime.unicamp.br IMECC - UNICAMP April

2 Partners Interior Point Methods Alessandro Coelho Carla Ghidini Frederico Campos Daniele Silva Danny Sorensen Jair Silva Jordi Castro Lino Silva Luciana Casacio Marilene Silva Marta Velazco Porfirio Suñagua Silvana Bocanegra

3 Summary Interior Point Methods Introduction Interior Point Methods Iterative Methods Preconditioning Numerical Experiments Conclusions Future Works

4 Introduction Sophisticated codes for interior point methods Few iterations Expensive iterations Linear System Solution at each iteration Choice between Augmented System and Normal Equations System Choice between direct methods and iterative methods

5 Introduction Improve iteration time performance Reduce the number of iterations Specialized methods for a given class of problems

6 Main Goal Interior Point Methods Improve iteration time performance using iterative methods on the normal equations system. Conjugate Gradient Method Preconditioners

7 Interior Point Methods Interior Point Methods Standard LP form Min s.t. c T x Ax = b x 0, A m n - rank(a)= m c, b, x column vectors

8 Interior Point Methods Interior Point Methods Dual Problem Max s.t. b T y A T y + z = c z 0, free variable y and slack dual variable z

9 Interior Point Methods Interior Point Methods Optimality conditions Ax b = 0, A T y + z c = 0, XZe = 0, x, z 0. X = diag(x), Z = diag(z) and e is the vector of all ones Primal-dual like Newton s method

10 Interior Point Methods Interior Point Methods Search Directions 0 I A T Z X 0 A 0 0 x y z = r d r a r p r d = c A T y z, r a = XZe, r p = b Ax

11 Interior Point Methods Interior Point Methods Given y 0 and (x 0, z 0 ) > 0. For k = 0, 1, 2,..., do Primal-dual Interior Point Methods (1) Choose σ k [0, 1) and set µ k = σ k ( (x k ) t z k n (2) Solve the given linear system (3) Choose a step length α k = min(1,τ k ρ k p,τ k ρ k d ) for τ k (0, 1) where ρ k p = 1 ( x k min i i xi k ) and ρ k d = 1 min i ( z k i z k i ). ).

12 Interior Point Methods Interior Point Methods (4) Form the new iterate (x k+1, y k+1, z k+1 ) = (x k, y k, z k )+α k ( x k, y k, z k ).

13 Interior Point Methods Interior Point Methods Predictor-Corrector Variant Affine Direction ra k = X k Z k e Final Direction rm k = µk e X k Z k e X k Z k e

14 Interior Point Methods Interior Point Methods Eliminating z [ D A T A 0 ][ x y ] = [ r1 r p ], D = X 1 Z. Augmented System Eliminating x Normal Equations System AD 1 A T y = r

15 Linear Systems Augmented system Symmetric Indefinite Matrix D Sparse Very Ill-conditioned Normal Equations System Symmetric Definite Positive Matrix D Usually Sparse Very Very Ill-conditioned

16 Linear Systems Augmented system LTL T Factorization Bunch-Parlett Factorization Cholesky type Factorization MINRES SYMMLQ QMR Conjugate Gradient Method Normal Equations System Cholesky Factorization Conjugate Gradient Method COELHO, OLIVEIRA AND VELAZCO, TEMA, Accepted.

17 Preconditioning Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Preconditioning Given Bx = b solve (MBN t ) x = b where x = N t x and b = Mb.

18 Preconditioning Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Lemma Given a Preconditioner for the Schur Complement G(AD 1 A t + E)H t = GSH t There Exists an Equivalent Preconditioner for Augmented System ( ) ( ) ( ) D D A t D 1 2 D 1 A t H t GAD 1 G A E 0 H t = ( I 0 0 GSH t )

19 Preconditioning Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Lemma Consider the augmented system given by ( )( ) ( D A t x ry X = r z + V r t A E y r x + Y r w ) and its Schur complement S = AD 1 A t + E where D is nonsingular and A is full row ( rank. ) Then any symmetric block H 0 triangular preconditioner leads to a preconditioned F G system independent of both: H and F.

20 Preconditioning Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Proof Modified Augmented System ( HDH t B t B C )( x ỹ ) ( = H(r p X 1 r c ) Fr p + G(r d X 1 r c ) ) where, B = FDH t + GAH t and C = FDF t + FA t G t + GAF t + GEG t. Modified Schur Complement GSG t ỹ = G(r p + AD 1 r d AZ 1 r c ). OLIVEIRA AND SORENSEN, Linear Algebra and Its Applications, 394 (2005).

21 Hybrid Preconditioner Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Matrix A full row rank Preconditioner based upon a basic solution First Iterations: Generic Preconditioner Final Iterations: Specially tailored preconditioner Middle Iterations BOCANEGRA, CAMPOS, AND OLIVEIRA, Computational Optimization and Applications, 36 (2007), pp BOCANEGRA, CASTRO, AND OLIVEIRA, European Journal of Operational Research, 231 (2013), pp

22 Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Controlled Cholesky Factorization A kind of incomplete Cholesky factorization S = ADA T = LL T = L L T + R where L complete factorization L incomplete R remainder matrix E = L L= L = E + L L 1 (ADA T ) L T = ( L 1 L)( L 1 L) T = (I + L 1 E)(I + L 1 E) T L L = E 0 = L 1 (ADA T ) L T I

23 Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Controlled Cholesky Factorization m minimize E 2 F = c j with c j = j=1 m l ij l ij 2 i=1 c j = S j +η k=1 l ik j l ik j 2 + η extra entries allowed per column lij l ij as S j +η m k= S j +η+1 l ik j 2 E F minimized increasing η (c j decreases) and choosing the largest entries of L j

24 Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Controlled Cholesky Factorization Generic Developed for Symmetric Positive Definite Systems Conjugate Gradient Method No Need to Compute S = ADA t Allows Drop-out (η < 0) Versatile Diagonal Preconditioner Complete Factorization Predictable Storage Loss of Positive Definiteness: Exponential Shift F. CAMPOS Analysis of Conjugate Gradients - type methods for solving linear equations PHD THESIS, Campos and Birkett, SIAM J. Sci. Comput., 19 (1998.), pp

25 Splitting Preconditioner Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach A = [B N]P, where P permutation matrix and B nonsingular ADA t = BD B B t + ND N N t D 1 2 B B 1 ADA t B t D 1 2 B = I m + D 1 2 B B 1 ND N N t B t D 1 2 B Finding a Permutation P Rules for Reordering the Columns of A Diagonal value of D ADe j VELAZCO, OLIVEIRA AND CAMPOS, Pesquisa Operacional, 34 (2011), pp Compute LU Factorization to Determine the Columns of B Careful Implementation GHIDINI, OLIVEIRA AND SORENSEN, Annals of Management Science, 3 (2014), pp

26 Splitting Preconditioner Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Normal Equations System Conjugate Gradient Method No Need to Compute S = ADA T Works Fine near a Solution Designed for Last IP Iterations It is not Efficient far from a Solution First IP Iterations Hybrid Approach Diagonal Preconditioner Controlled Cholesky Preconditioner Change of Phases VELAZCO, OLIVEIRA AND CAMPOS, Optimization Methods and Software, 25 (2010), pp

27 Hydrid Approach Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach No Need to Compute ADA t Early Iterations Transition Preconditioned Conjugate Gradient Method Diagonal Preconditioner Controlled Cholesky Decomposition Fast MINRES Normal Equations System Indefinite System Robust Final Iterations Splitting Preconditioner MINRES Indefinite System Normal Equations System Preconditioned Conjugate Gradient Method Fast GHIDINI, OLIVEIRA AND SILVA, TEMA, 15 (2014), pp

28 New Approach to Compute B Preconditioning Hybrid Preconditioner Controlled Cholesky Factorization Splitting Preconditioner Hybrid Approach New Approach Regularization (penalization) ADA t +δi Compute LU Factorization of D 1 2 A t Reorder columns for sparsity Perform row permutation for stability Threshold parameter σ UMFPACK More stable that rectangular LU May need much more storage P. SUÑAGUA, Uma Nova Abordagem para Encontrar uma Base do Precondicionador Separador para Sistemas Lineares no Método de Pontos Interiores PhD Thesis, 2014.

29 for Linear programming Consider A R m n and A j = 1 Ax = 0, e T x = 1, x 0, OBS: Any LP can be reduced to this form

30 for Linear programming

31 Von Neumann Algorithm Simple but with slow convergence Find A column A s with largest angle with the residual b k 1. The next residual is given by the line connecting b k 1 to A s

32 Von Neumann Algorithm

33 Adjustment Algorithm for p coordinates Given: x 0 0, with e T x 0 = 1. Compute b 0 = Ax 0. For k = 1, 2, 3,... do: 1. Compute: {A η +,..., A 1 η + s1 } with largest angle with b k 1.,..., A η } with smallest angle with b k 1 s2 {A η 1 such that x k 1 i > 0, i = η 1,...,η s 2 s 1 + s 2 = p. v k 1 = min i=1,...,s1 A T η + i b k If v k 1 > 0, then Stop. The problem is infeasible

34 Adjustment Algorithm for p coordinates 3. Solve the subproblem Min b 2 s 1 s.t. λ 0 1 λ η + i λ η i i=1 x k 1 η + i s 2 x k 1 η j j=1 0, for i = 1,...,s 1, 0, for j = 1,...,s 2. s 1 + i=1 λ η + i s 2 + j=1 λ η j = 1, b = λ 0 b k 1 s 1 i=1 x k 1 η + i A η + i s 2 j=1 x k 1 η j A η j + s 1 i=1 λ η + i A η + i + s 2 j=1 λ η j A η. j

35 Adjustment Algorithm for p coordinates 4. Update b k = λ 0 b k 1 u k = b k, x k j = s 1 i=1 x k 1 η + i A η + i s 2 j=1 x k 1 η j A η j λ 0 x k 1 j, j / {η + 1,...,η+ s 1,η 1,...,η s 2 }, λ η +, j = η + i, i = 1,...,s 1, i, j = η j, j = 1,...,s 2. λ η j k = k s 1 i=1 λ η + i A η + i + s 2 j=1 λ η j A η, j

36 Subproblem Solution For p = 2: small number of cases in KKT conditions. For any p: 1+2C p 1 + 2Cp 2 + 2Cp Cp p = 2 (p+1) 1. Solution: Apply interior point methods Small dimension problem if p is small. GHIDINI, OLIVEIRA, SILVA AND VELAZCO, Linear Algebra and its Applications, 218 (2012), pp

37 Subproblem - Interior Point Method Min 1 2 Wλ 2 s.t. c T λ = 1, λ 0, [ ] W = w A η +...A 1 η + s1 A η...a 1 η, s2 w = b k 1 s 1 A η + s 2 ( λ = λ 0,λ η + 1 i=1 xk 1 η + i,...,λ η + s1,...,λ η 1 i j=1 xk 1 A η η j j ),...,λ η, s2 c = (c 1, 1,...,1),c 1 = 1 s 1 i=1 xk 1 η + i, s 2 j=1 xk 1 η j.

38 Subproblem - Interior Point Method Solve the Linear System at each subproblem iteration W T W c I U 0 Λ c T 0 0 λ l µ = r 1 r 2 r 3 where U = diag(µ), Λ = diag(λ), r 1 = µ cl W T Wλ, r 2 = l T λ, r 3 = 1 c T λ.

39 Subproblem - Interior Point Method Search Directions: µ = Λ 1 r 2 Λ 1 Udλ, λ = (W T W +Λ 1 U) 1 r 4 (W T W +Λ 1 U) 1 c l, c T (W T W +Λ 1 U) 1 c l = c T (W T W +Λ 1 U) 1 r 4 r 3 Linear Systems (W T W +Λ 1 U)v1 = c (W T W +Λ 1 U)v2 = r 4 r 4 = r 1 +Λ 1 r 2

40 Numerical Experiments Conclusions Problems NETLIB QAP Kennington Total: 189 Medium Problems Large Problems BL fit1p pds-30 ken18 BL2 fit2p pilot87 kra30a 1 chr22b GE qap12 kra30b 1 chr25a greenbeb qap15 nug07-3rd CO5 ken13 rou20 nug20 1 CO9 NL scr20 pds-70 CQ9 nug06-3rd stocfor3 pds-80 cre-b nug12 pds-90 cre-d nug15 pds-100 dfl001 osa-60 ste36a 1 els19 pds-10 ste36b 1 ex09 pds-20 ste36c 1 1 Need large storage

41 Numerical Experiments Conclusions Running Time (sec.) Problem LUrec LUrec+Reg LUstd LUstd+Reg pds ken pds pds pds kra30a 6693, ste36a ste36b kra30b ste36c nug Remaining Total Common

42 Numerical Experiments Conclusions Processing Time LUrec LUrec+Reg LUstd LUstd+Reg Solved Problems % 62.96% 78.84% 86.24% not solved % 37.04% 21.16% 13.76% Total (sec) Prob Prob Difference: h. LUstd more efficient then LUrec

43 Numerical Experiments Conclusions Conclusions Iterative Methods Very large problems Preconditioners Normal Equations Augmented System Sophisticated implementations Fast and Robust

44 Numerical Experiments Conclusions Future Work Heuristics for changing Preconditioners More robust iterative methods in the change of phases New simple methods Augmented system New Preconditioners

Preconditioning and iterative solution of symmetric indefinite linear systems arising from interior point methods for linear programming

Preconditioning and iterative solution of symmetric indefinite linear systems arising from interior point methods for linear programming Joo-Siong Chai and Kim-Chuan Toh Abstract We propose to compute