Lecture 10. Primal-Dual Interior Point Method for LP

IE 8534 1 Lecture 10. Primal-Dual Interior Point Method for LP

IE 8534 2 Consider a linear program (P ) minimize c T x subject to Ax = b x 0 and its dual (D) maximize b T y subject to A T y + s = c s 0. Let F P be the feasible set for (P ). Let F D be the feasible set for (D).

IE 8534 3 Suppose that both (P ) and (D) satisfy the Slater Condition (strong feasibility). Then the central path {(x(µ), y(µ), s(µ)) µ (0, ), x(µ) F P, (y(µ), s(µ)) F D, x(µ) s(µ) = µe} exists, where stands for the Hadamard product. The idea now is: Try to approach the optimal solutions by tracing the central path as µ tends to zero. x i (µ)s i (µ) = µ for 1 i n Suppose that our current interior primal-dual solution is (x, s). Let our next target solution be (x, s ). Assume that (x ) T s < x T s Moreover, let w := Xs and w := X s. Denote x := x x and s := s s.

IE 8534 4 We have A x = 0 A T y + s = 0 S x + X s = w w + X s This is a nonlinear equation and is difficult to solve. Linearization (Newton s method) is a good approximation: Let D := (XS 1 ) 1/2 and A x = 0 A T y + s = 0 S x + X s = w w p x := D 1 x and p s := D s

IE 8534 5 What does this mean? Well, this suggests that we scale the problems (P ) minimize c T x subject to Ax = b x 0 into and (D) maximize b T y subject to A T y + s = c s 0 (P ) minimize (Dc) T x subject to ADx = b x 0 and (D) maximize b T y subject to (AD) T y + s = Dc s 0.

IE 8534 6 The scaling transformation is given as x = D 1 x and y = y and s = Ds. Facts: (P ) and (D) remain a pair of primal-dual linear programs. (P ) and (D) are identical to the original problems (P ) and (D). The feasible solutions x and s for the original problems (P ) and (D) become w and w.

IE 8534 7 Let v := w (and v := w ). It follows that ADp x = 0 (AD) T y + p s = 0 p x + p s = p where p := V 1 (w w). The solution of the above equation is apparent: p x = P AD p = [I DA T (AD 2 A T ) 1 AD]p and p s = p p x = DA T (AD 2 A T ) 1 ADp. Because the Newton method is only an approximation, we may not exactly have x + x = x and s + s = s as we expected.

IE 8534 8 We introduce the step length t (> 0) and consider the iterates x(t) := x + t x and s(t) := s + t s Our idea is to make X(t)s(t) stay close to the central line and converge to zero. Introduce the neighborhoods of the central line { } N 2 (β) := v 2 1 µ v2 e β 2 and N (β) := { } v 2 1 µ v2 e β where 0 < β < 1 is a constant and µ = et v 2 n.

IE 8534 9 Primal-dual path following method The essence of this method is to select the next target point w as w := ρµe where 0 < ρ < 1. In this case we have p := v + ρµv 1 Assume that Xs = v 2 N (β). Therefore, v 2 i (1 β)µ for all i Clearly, iff x + t x > 0 and s + t s > 0 v + tp x > 0 and v + tp s > 0

IE 8534 10 Lemma 1 It holds that 1). (p x ) (p s ) 2 1 2 p 2 2 ; 2). (p x ) (p s ) 1 4 p 2 2.

IE 8534 11 Proof. 1). We have (p x ) (p s ) 2 = n [(p x ) i (p s ) i ] 2 i=1 n (p x ) 2 n i (p s ) 2 i i=1 i=1 1 2 [ p x 2 2 + p s 2 2 ] = 1 2 p 2 2

IE 8534 12 To prove 2). we observe for every i (p x ) i (p s ) i 1 2 (p x) (p s ) 1 1 4 n [ (px ) 2 i + (p s ) 2 ] i i=1 = 1 4 p 2 2 Lemma 2 It holds that ( 1 1 p 2 1 β µ v2 e + (1 ρ) µ n) 2

IE 8534 13 Proof. Using v 2 i (1 β)µ we have p 2 = V 1 ( µe v 2 + (ρ 1)µe ) 2 ( 1 1 1 β µ v2 e + (1 ρ) µ n) 2

IE 8534 14 The Algorithm: (Assume x (0) s (0) N (β)). Step 0 Let k := 0; Step 1 Compute x (k) and s (k). Step 2 Compute maximum step length t > 0 such that (x (k) + t x (k) ) (s (k) + t s (k) ) N (β) Step 3 Let and x (k+1) := x (k) + t x (k) s (k+1) := s (k) + t s (k) Let k := k + 1 and go to Step 1.

IE 8534 15 We are interested in how fast the algorithm converges. Clearly, v(t) 2 := (x + t x) (s + t s) = (1 t)v 2 + tρµe + t 2 p x p s and so µ(t) := et v(t) 2 n = (1 t + tρ)µ

IE 8534 16 Therefore = 1 µ(t) v(t)2 e ( ) ( ) tρ 1 1 1 t + tρ µ v2 e + t2 p x p s (1 t + tρ) µ ( ) tρ t 2 p 2 2 1 β + 1 t + tρ 2(1 t + tρ)µ ( ( ) tρ t 2 v 2 µ e 2 + (1 ρ) n 1 β + 1 t + tρ 2(1 t + tρ)(1 β) ( v 2 t µ e 2 + (1 ρ) ) 2 n = β ρβ t. 1 t + tρ 2(1 β) ) 2 Moreover, x(t) T s(t) = (1 (1 ρ)t) x T s

IE 8534 17 Now it follows: Theorem 1 Let β = Ω(1) and 1 ρ = Ω(1). Either the N 2 (β) or the N (β) neighborhood is used. There exists a constant c 1 = Ω(1) such that the sequence produced by the primal-dual algorithm satisfies ( (x (k+1) ) T s (k+1) 1 c ) 1 (x (k) ) T s (k) n Theorem 2 Let β = Ω(1) and 1 ρ = Ω( 1 n ). Assume that the N 2 (β) neighborhood is used. Then, there exists a constant c 2 = Ω(1) such that the sequence produced by the primal-dual algorithm satisfies ( (x (k+1) ) T s (k+1) 1 c ) 2 (x (k) ) T s (k) n

IE 8534 18 In general, if 1 ρ = Ω(1) then the algorithm is called a long step path following algorithm, and if 1 ρ = Ω( 1 n ) it is called a short step path following algorithm. In particular, there are two choices of ρ s that are interesting, i.e. ρ := 0 (primal-dual affine scaling) and ρ := 1 (centering). An important variant of primal-dual interior point algorithm is called the predictor-corrector algorithm. Lemma 3 If x s N 2 (1/2), then by letting ρ := 1 and the step length t := 1 we have x(1) s(1) N 2 (1/4)

IE 8534 19 The Predictor-Corrector Algorithm: (Assume x (0) s (0) N 2 (1/2)). Step 1 If k even, let ρ := 1; otherwise go to Step 2. Compute x (k) and s (k). Let x (k+1) := x (k) + x (k), s (k+1) := s (k) + s (k) Let k := k + 1 and go to Step 1. Step 2 Let ρ := 0. Compute maximum step length t > 0 such that (x (k) + t x (k) ) (s (k) + t s (k) ) N 2 (1/2) Let x (k+1) := x (k) + t x (k) Let k := k + 1 and go to Step 1. s (k+1) := s (k) + t s (k)

IE 8534 20 Theorem 3 In the predictor-corrector algorithm, there exists a constant c 3 = Ω(1) such that the sequence of iterates satisfies ( (x (k+1) ) T s (k+1) 1 c ) 3 (x (k) ) T s (k) n for all even k. The general convergence results are included in the following three theorems: Theorem 4 Assume that x (0) s (0) N (β) and (x (0) ) T s (0) 1/ϵ then in at most k = O(n log 1 ϵ ) iterations the long step primal-dual interior point algorithm will find iterates satisfying (x (k) ) T s (k) < ϵ.

IE 8534 21 Theorem 5 Assume that x (0) s (0) N 2 (β), (x (0) ) T s (0) 1/ϵ and the N 2 (β) neighborhood is used, then in at most k = O( n log 1 ϵ ) iterations the short step primal-dual interior point algorithm will find iterates satisfying (x (k) ) T s (k) < ϵ. Theorem 6 Assume that x (0) s (0) N 2 ( 1 2 ) and (x(0) ) T s (0) 1/ϵ then in at most k = O( n log 1 ϵ ) iterations the predictor-corrector algorithm will find iterates satisfying (x (k) ) T s (k) < ϵ.

IE 8534 22 Key References: M. Kojima, S. Mizuno, and A. Yoshise, A Primal-Dual Interior Point Algorithm for Linear Programming, ed. N. Megiddo, Progress in Mathematical Programming: Interior point and related methods, 29 37, 1989. S.J. Wright, Primal-Dual Interior-Point Methods, SIAM, 1997. S. Mizuno, M.J. Todd, and Y. Ye, On Adaptive-Step Primal-Dual Interior-Point Algorithms for Linear Programming, Mathematics of Operations Research, 18, 964 981, 1993.