McMaster University Advanced Optimization Laboratory Title: Computational Experience with Self-Regular Based Interior Point Methods Authors: Guoqing Zhang, Jiming Peng, Tamás Terlaky, Lois Zhu AdvOl-Report No. 2002/1 June 2002, Hamilton, Ontario, Canada
CORS 2002 Toronto Computational Experience with Self-Regular Based Interior Point Methods Guoqing Zhang Coauthors: J. Peng, T. Terlaky, Lois Zhu Department of Computing and Software McMaster University Toronto, Ontario June 3, 2002
Outline Introduction Interior Point Methods for LO The Infeasible SR IPM Algorithms Implementation Issues Numerical Results Comparison Conclusions, Ongoing Works 1
Introduction Linear Optimization (LO) problems: min{c T x : Ax = b, x 0}, x R n, A R m n 1947: (Dantzig) Simplex method: the first practical method for LO. 1956: (Tucker) Homogeneous Self-dual model. 1972: (Klee, Minty) Exponential example for the Simplex method: O(2 n ) pivots in the worst case. 1978: (Khachiyan) Ellipsoid method: the first polynomial algorithm for LO. Complexity: O(n 2 L) iterations, O(n 4 L) bits operations, where L = the input size of the problem. 1984: (Karmarkar) Interior Point Methods (IPMs): O(nL) iterations, O(n 3.5 L) bit operations. 2
Interior Point Methods for LO 1950s-60s: (Frisch) Logarithmic barrier method, (Huard ) Method of centers, (Dikin) Affine scaling methods: first IPMs. 1984: (Kamarkar) Projective methods. 1989: (Kojima-Mizuno-Yoshise) Primal-dual path following methods. 1989, 1992: (Mehrotra) Predictor-corrector. 2000: (Peng, Roos, Terlaky) Self-Regular IPMs: Best known worst-case complexity for large-update IPMs: O ( n log n log n ε ). 3
Self-Regular Function ψ(t) is Self-Regular (SR) if 3 SR1: ψ(t) is strongly convex, global minimum: ψ(1) = 0,, ν 1, ν 2 > 0 and p, q 1, such that for t (0, + ) 2.5 2 (p=1,q=3) (p=1, q=1) (p=1, q=3) ν 1 (t p 1 + t 1 q ) ψ (t) ν 2 (t p 1 + t 1 q ), 1.5 SR2: For any t 1, t 2 > 0, r [0, 1]. 1 ψ(t r 1 t1 r 2 ) rψ(t 1 ) + (1 r)ψ(t 2 ), 0.5 (p=1,q=1) q : barrier degree; p : growth degree. 0 0 0.5 1 1.5 2 2.5 3 4
Examples of Self-Regular Functions We have two sets of SR functions in R n ++ R +: and Υ p,q (t) = t p+1 1 p(p + 1) + p 1 (t 1) log t, q = 1 p t p+1 1 p(p + 1) + t1 q 1 q(q 1) + p q (t 1), q > 1. pq (1) Γ p,q (t) = tp+1 1 p + 1 + t1 q 1, p 1, q > 1. (2) q 1 Υ 1,1 (t) is a well-known logarithmic barrier function. Ψ is a proximity measure, distance of v to e = (1, 1,..., 1) T. Let Ψ(v) = n i=1 ψ(v i ) where ψ(t) is a SR function. 5
The Linear Optimization Problem (I) Consider the following LO problem: min c T x s.t Ax = b, 0 x i u i, i I, 0 x j, j J, where A R m n, rank(a) = m, I J = {1, 2,..., n} and I J =. The last two constraints can be written as F x + s = u, x 0, s 0, F R m f n, the rows of F are unit vectors, s, u R m f, m f = I. 6
The Linear Optimization Problem (II) The primal LO problem can be rewritten in the form: (LP) min c T x s.t Ax = b, F x +s = u, x, s 0, where c, x R n, b R m, A R m n, F R m f n. The dual problem is: (LD) max b T y u T w s.t A T y F T w +z = c, w, z 0, where y R m, w R m f and z R n. 7
The Linear Optimization Problem (III) Optimality Conditions Ax = b, F x + s = u, A T y F T w + z = c, (3) W s = 0, Xz = 0, where X = diag(x), W = diag(w). 8
The Linear Optimization Problem (IV) The primal-dual central path is defined as the set of solutions (x(µ), s(µ)) and (y(µ), w(µ), z(µ)) for µ > 0 of the system where e = [1, 1,..., 1] T. Ax = b, F x + s = u, A T y F T w + z = c, (4) W s = µe, Xz = µe, SR proximity: Ψ( x z, µ) = n i=1 ψ ( ) xi z i µ + m f i=1 ψ ( ) si w i, x z = µ ( ) xz. sw 9
SR-Infeasible IPM: Newton equation Self-regular proximity based Newton equation is given as: where A 0 0 0 0 F I 0 0 0 0 0 A T I F T Z 0 0 X 0 0 W 0 0 S x s y z w = r b r u r c µv 1 Ψ(v 1 ) µv 2 Ψ(v 2 ) r b = Ax b, v 1 = xz µ, r u = F x + s u, v 2 = sw µ, r c = A T y F T w + z c, v = [v 1 v 2 ]., (5) 10
Predictor-Corrector Search Directions The affine-scaling direction: A 0 0 0 0 F I 0 0 0 0 0 A T I F T Z 0 0 X 0 0 W 0 0 S x a s a y a z a w a The Self-Regular based corrector direction: A 0 0 0 0 F I 0 0 0 0 0 A T I F T Z 0 0 X 0 0 W 0 0 S For µ + see the next page. x s y z w = = r b r u r c XZ W S. (6) r b r u r c µ + v 1 + Ψ(v+ 1 ) xa z a e µ + v 2 + Ψ(v+ 2 ) wa s a e (7). 11
Normal Equation for SR-Infeasible IPM After simplifying, normal equation is given as: AD 2 A T y = AD 2 r h r b. (8) Where D 2 = (X 1 Z + S 1 W ) 1 r h = r c + S 1 W r u + S 1 µv 1 Ψ(v 1 ) + X 1 µv 2 Ψ(v 2 ) (9) Then, compute z = X 1 ( µv 1 Ψ(v 1 ) Z x), s = r u x, w = S 1 (w(r u + x) µv 2 Ψ(v 2 )), x = D 2 (A T y r h ). (10) 12
SR-Infeasible IPMs Input: Given initial point (x 0, s 0, y 0, z 0, w 0 ), such that (x 0, s 0, z 0, w 0 ) > 0 an accuracy parameter ɛ > 0 begin while x T z + s T w ɛ do predictor solve system (6) and compute the max. feas. step size let µ a = (x+αa p x a ) T (z+α a d za )+(s+α a p s a ) T (w+α a d wa ) n+m f corrector end end compute target µ + = (µ a /µ) 3 µ solve (7) for x, s, y, z, w Compute the max. feas. step size α p, α d Update the iterate (x, s) = (x, s) + α p ( x, s) Update the iterate (y, z, w) = (y, z, w) + α d ( y, z, w) 13
Sparse Cholesky Solver If the matrix AD 2 A T is sparse, we can use WSMP directly to solve the system AD 2 A T y = ξ by using the following steps: 0. Ordering and Symbolic factorization for the structure of AD 2 A T, just once in the whole algorithm 1. Numerical factorization of AD 2 A T 2. Back solve to get the solution 3. Iterative refinement if necessary If matrix A has one or more dense columns, then AD 2 A T will be dense. 14
Numerical Problems with Cholesky Handle Bad and Small Pivots by WSMP: 0 10 200 10-16 Bad pivot threshold 10-10 Small pivot threshold 10-7 Small pivot subst. Bad pivot subst. If the pivot value Bad pivot threshold, it will be substituted by Bad pivot subst. If the pivot value Small pivot threshold, it will be substituted by Small pivot subst. Singularity: [ART] check and handle by K. Anderson s algorithm 15
Choice of Self-Regular Function We choose SR functions as Υ 1,q (t) or Γ 1,q (t) Fixed settings: p = 1, and q = 1, 2 or 3 Dynamic update of q: Set q = 1 as default, steptol = 10 2 calculate search direction while α steptol, then let q = q + 2 change µ to x T z x T z 1 calculate search direction if q 5, then break end make step reset q = 1 16
Numerical Results: compare with OSL,LIPSOL System environment: IBM RS/6000 44P Model 270 Workstation with AIX 4.3 Coding with C, WSMP, OSL, and ESSL. Benchmark problems in Netlib. 17
Testing Results From SR-Infeasible IPM and Comparisons (1) SR-IIPM Problem Iter Flag-q Digits 25fv47 25 0 11 80bau3b 40 0 10 adlittle 11 0 10 afiro 8 0 11 agg 17 0 11 agg2 17 0 11 agg3 16 0 12 bandm 18 0 11 beaconfd 9 0 10 blend 15 0 11 bnl1 31 0 7 bnl2 31 0 11 boeing1 24 0 11 boeing2 15 0 10 bore3d 19 0 11 Iter OSL Digits 27 9 44 12 13 10 9 10 22 11 18 11 17 11 17 10 9 11 12 11 26 7 34 11 24 9 17 10 19 12 Iter LIPSOL Digits 25 11 40 9 13 11 8 11 21 11 18 12 17 12 18 10 13 11 12 11 26 7 31 9 21 9 19 9 18 11 19
Testing Results From SR-Infeasible IPM and Comparisons (2) SR-IIPM Problem Iter Flag-q Digits brandy 15 0 11 capri 19 0 11 cycle 27 0 12 czprob 34 0 11 d2q06c 35 0 7 d6cube 26 0 11 degen2 15 1 12 degen3 20 1 11 dfl001 43 * 1 10 e226 22 0 10 etamacro 30 0 7 fffff800 29 0 7 finnis 24 0 10 fit1d 20 0 12 fit1p 16 0 9 Iter OSL Digits 17 12 19 11 25 10 31 11 32 7 21 8 14 9 19 11 56 10 21 1 34 7 32 7 26 6 21 10 16 10 Iter LIPSOL Digits 17 11 19 11 25 8 36 11 32 7 23 9 14 11 25 10 79 7 21 11 25 7 26 7 30 11 19 11 16 10 20
Testing Results From SR-Infeasible IPM and Comparisons (3) SR-IIPM Problem Iter Flag-q Digits fit2d 22 0 11 forplan 25 1 6 ganges 17 0 6 gfrd-pnc 16 0 10 greenbea 38* 1 7 greenbeb 38 0 4 grow15 16 0 11 grow22 16 0 12 grow7 16 0 11 israel 21 0 10 kb2 15 0 12 lotfi 15 0 11 maros 28 0 11 maros-r7 18 0 11 Iter OSL Digits 26 11 25 6 15 6 17 10 48 3 59 5 17 11 20 11 17 11 21 9 17 11 15 10 24 8 14 11 Iter LIPSOL Digits 7 6 22 6 18 6 21 11 43 4 38 4 17 11 19 11 16 11 23 11 15 12 18 11 33 11 15 11 21
Testing Results From SR-Infeasible IPM and Comparisons (4) SR-IIPM Problem Iter Flag-q Digits modszk1 22 0 11 nesm 31 0 6 pilot 41 0 4 pilot87 41 0 6 pilotnov 22 0 11 recipe 9 0 11 sc105 10 0 10 sc205 11 0 11 sc50a 10 0 11 sc50b 8 0 12 scagr25 16 0 11 scagr7 12 0 7 scfxm1 18 0 11 scfxm2 23 * 0 10 scfxm3 23 * 0 8 Iter OSL Digits 24 7 38 6 34 4 44 6 23 11 11 11 10 10 11 10 10 9 8 10 18 11 14 7 16 2 28 7 22 9 Iter LIPSOL Digits 24 10 33 6 31 4 37 6 38 6 9 11 10 11 10 9 10 11 7 8 17 11 14 7 19 11 21 12 21 11 22
Testing Results From SR-Infeasible IPM and Comparisons (5) SR-IIPM Problem Iter Flag-q Digits scorpion 16 0 11 scrs8 22 0 5 scsd1 9 0 10 scsd6 12 0 11 scsd8 11 0 11 sctap1 18 0 12 sctap2 19 0 12 sctap3 20 0 10 seba 19 0 11 share1b 21 0 12 share2b 14 0 12 shell 17 0 10 ship04l 13 0 11 ship04s 13 0 10 ship08l 17 0 12 Iter OSL Digits 14 11 22 5 10 8 12 12 11 11 16 10 17 10 18 12 20 12 22 9 13 10 19 10 17 11 15 11 16 10 Iter LIPSOL Digits 15 11 24 5 9 6 11 7 11 10 17 12 19 11 18 12 22 11 22 11 13 11 21 12 14 11 14 11 16 11 23
Testing Results From SR-Infeasible IPM and Comparisons(7) SR-IIPM Problem Iter Flag-q Digits ship08s 14 0 11 ship12l 16 0 11 ship12s 16 0 12 sierra 16 0 11 stair 17 0 3 standata 18 0 11 standgub 18 0 10 standmps 27 0 11 stocfor1 12 0 9 stocfor2 24 0 12 stocfor3 34 0 5 truss 18 0 11 tuff 20 0 10 Iter OSL Digits 16 11 18 11 17 11 19 11 18 3 15 10 15 9 21 12 16 10 22 11 33 5 18 10 19 12 Iter LIPSOL Digits 15 11 18 11 18 12 17 11 14 10 17 12 17 12 24 12 16 11 21 11 32 5 19 11 20 6 24
Testing Results From SR-Infeasible IPM and Comparisons(8) SR-IIPM Problem Iter Flag-q Digits vtpbase 10 0 11 wood1p 26 1 11 woodw 36 0 11 Total 1832 902 Iter OSL Digits 11 10 19 4 25 10 1882 830 Iter LIPSOL Digits 23 11 19 7 28 8 1877 869 25
Comparisons of SR with Different q Values q=1 q=2 Dynamic Problem Iter Digits Iter Digits Iter Digits degen2 17 11 25 11 15 12 degen3 35 8 100 2 20 11 dfl001 65 7 100 1 43 9 forplan 25 6 50? 25 6 greenbea 38 9 100 8 38 7 wood1p 26 11 48 10 26 11 26
Conclusions: Conclusions, Ongoing Work Our infeasible algorithms promise high precision with reasonable iteration number, though several problems still are in testing. The average iteration number is less than that of LIPSOL, OSL while the solution has a little higher precision. Encouraging computational results, still space to improve. Ongoing work: Embedding IPM based on MATLAB, LIPSOL and WSMP numerical stability of search direction handle numerical singularity of the matrix AD 2 A T adaptive choice of Self-Regular proximity preprocessing and postprocessing Embedding IPM based on OSL, ESSL and WSMP 27
Reference Peng J.Peng, C. Roos, and T. Terlaky. Self-Regular proximities and new search directions for linear and semidefinite optimization, Mathematical Programming (2002). ART: E.D.Anderson, C.Roos, T.Terlaky, T.Traflis and J.P.Warners. The use of low-rank updates in interior-point methods. Technical Report, Delft University of Technology, The Netherlands, 1996. RTV: C.Roos, T.Terlaky and J.-Ph. Vial. Theory and Algorithms for Linear Optimization. An Interior Approach. John Wiley and Sons, Chichester, UK, 1997. W: S. J. Wright. Primal-Dual Interior-Point Methods. SIAM, Philadelphia, 1996. Ye: Y. Ye. Interior-Point Algorithms, Theory and Analysis. John Wiley & Sons, Chichester, UK, 1997. 28
Appendix Computational Results From SR-Infeasible IPM (1) Problems iter Flag-q Residual Object-value Digits 25fv47 25 0 3.72E-13 5.501845888287E+03 11 80bau3b 40 0 3.24E-10 9.872241924578E+05 10 adlittle 11 0 2.66E-10 2.254949631862E+05 10 afiro 8 0 1.84E-13-4.647531428571E+02 11 agg 17 0 9.13E-11-3.599176728622E+07 11 agg2 17 0 1.38E-10-2.023925235532E+07 11 agg3 16 0 2.23E-12 1.031211593510E+07 12 bandm 18 0 5.34E-10-1.586280184408E+02 11 beaconfd 9 0 1.76E-10 3.359248580905E+04 10 blend 15 0 5.43E-12-3.081214984574E+01 11 bnl1 31 0 1.44E-11 1.977629561551E+03 7 bnl2 31 0 2.24E-13 1.811236540359E+03 11 boeing1 24 0 7.69E-12-3.352135675062E+02 11 boeing2 15 0 5.89E-10-3.150187279583E+02 10
Computational Results From SR-Infeasible IPM (2) Problems iter Flag-q Residual Object-value Digits bore3d 19 0 1.72E-11 1.373080394211E+03 11 brandy 15 0 3.51E-10 1.518509896428E+03 11 capri 19 0 1.02E-10 2.690012913860E+03 11 cycle 27 0 1.50E-12-5.226393024891E+00 12 czprob 34 0 1.92E-12 2.185196698858E+06 11 d2q06c 35 0 5.38E-11 1.227842108141E+05 7 d6cube 26 0 1.69E-11 3.154916666676E+02 11 degen2 15 1 1.63E-11-1.435177999993E+03 12 degen3 20 1 8.96E-12-9.872939999946E+02 11 dfl001 43 1 7.90E-08 1.126639611975E+07 10 e226 22 0 8.00E-10-1.875192905869E+01 10 etamacro 30 0 5.73E-10-7.557152332621E+02 7 fffff800 29 0 1.42E-13 5.556795648176E+05 7
Computational Results From SR-Infeasible IPM (3) Problems iter Flag-q Residual Object-value Digits finnis 24 0 8.92E-10 1.727910655446E+05 10 fit1d 20 0 4.75E-12-9.146378092396E+03 12 fit1p 16 0 5.08E-10 9.146378097048E+03 9 fit2d 22 0 1.11E-11-6.846429329382E+04 11 forplan 25 1 7.94E-11-6.642189613204E+02 6 ganges 17 0 4.00E-12-1.095857361295E+05 6 gfrd-pnc 16 0 2.90E-11 6.902235999632E+06 10 greenbea 38 1 5.03E-08-7.255525217295E+07 7 greenbeb 38 0 6.47E-10-4.302260259447E+06 4 grow15 16 0 8.51E-13-1.068709412936E+08 11 grow22 16 0 1.37E-10-1.608343364806E+08 12 grow7 16 0 6.17E-14-4.778781181471E+07 11 israel 21 0 1.25E-10-8.966448218228E+05 10 kb2 15 0 3.09E-12-1.749900129905E+03 12 lotfi 15 0 1.76E-13-2.526470606187E+01 11 maros 28 0 8.89E-12-5.806374370058E+04 11
Computational Results From SR-Infeasible IPM (4) Problems iter Flag-q Residual Object-value Digits maros-r7 18 0 8.72E-14 1.497185166480E+06 11 modszk1 22 0 9.11E-12 3.206197290615E+02 11 nesm 31 0 3.43E-10 1.407603649085E+07 6 pilot 41 0 6.28E-10-5.574897292382E+02 4 pilot87 41 0 4.99E-12 3.017103473346E+02 6 pilotnov 22 0 1.64E-10-4.497276188212E+03 11 recipe 9 0 2.26E-11-2.666159999973E+02 11 sc105 10 0 7.25E-10-5.220206120329E+01 10 sc205 11 0 1.51E-12-5.220206121168E+01 11 sc50a 10 0 7.43E-13-6.457507705853E+01 11 sc50b 8 0 1.74E-13-6.999999999999E+01 13 scagr25 16 0 8.55E-13-1.475343306076E+07 11 scagr7 12 0 8.03E-10-2.331389823703E+06 7 scfxm1 18 0 3.05E-11 1.841675902835E+04 11
Computational Results From SR-Infeasible IPM (5) Problems iter Flag-q Residual Object-value Digits scfxm2 21 0 2.06E-09 3.666026156703E+04 10 scfxm3 21 0 3.32E-09 5.490125465379E+04 8 scorpion 16 0 8.94E-12 1.878124822745E+03 11 scrs8 22 0 2.27E-11 9.042969538083E+02 5 scsd1 9 0 1.64E-11 8.666666674475E+00 10 scsd6 12 0 1.59E-10 5.050000007814E+01 11 scsd8 11 0 2.58E-11 9.049999999345E+02 11 sctap1 18 0 2.96E-12 1.412250000001E+03 13 sctap2 19 0 4.20E-11 1.724807142907E+03 12 sctap3 20 0 9.94E-10 1.424000000409E+03 10 seba 19 0 9.93E-11 1.571160000054E+04 11 share1b 21 0 4.14E-12-7.658931857897E+04 12 share2b 14 0 1.79E-11-4.157322407390E+02 12 shell 17 0 6.32E-10 1.208825346113E+09 10 ship04l 13 0 1.54E-13 1.793324537971E+06 11 ship04s 13 0 5.10E-11 1.798714700521E+06 10
Computational Results From SR-Infeasible IPM (6) Problems iter Flag-q Residual Object-value Digits ship08l 17 0 8.56E-12 1.909055211405E+06 12 ship08s 14 0 2.14E-11 1.920098210574E+06 11 ship12l 16 0 3.81E-13 1.470187919330E+06 11 ship12s 16 0 5.22E-13 1.489236134407E+06 12 sierra 16 0 2.53E-12 1.539436218364E+07 11 stair 17 0 5.72E-10-2.505464059458E+02 3 standata 18 0 3.74E-10 1.257699500077E+03 11 stangub 18 0 3.74E-10 1.257699500077E+03 10 standmps 27 0 1.34E-10 1.406017500067E+03 11 stocfor1 12 0 7.76E-10-4.113197620875E+04 9 stocfor2 24 0 4.53E-12-3.902440853793E+04 12 stocfor3 34 0 5.06E-11-3.997678394285E+04 5 truss 18 0 3.37E-13 4.588158471857E+05 11 tuff 20 0 4.24E-10 2.921477655496E-01 10 vtpbase 10 0 9.64E-11 1.298314624687E+05 11 wood1p 26 1 3.08E-11 1.442902411588E+00 11 woodw 36 0 8.37E-10 1.304476333083E+00 11