October 13, 1995. Revised November 1996. AN INTERIOR POINT METHOD, BASED ON RANK-ONE UPDATES, FOR LINEAR PROGRAMMING Jos F. Sturm 1 Shuzhong Zhang Report 9546/A, Econometric Institute Erasmus University Rotterdam P.O. Box 1738, 3000 DR Rotterdam, The Netherls ABSTRACT We propose a polynomial time primal-dual potential reduction algorithm for linear programming. The algorithm generates sequences d k v q q k rather than a primal-dual interior point (x k s k ), where d k i = x k i =sk i vi k = x k i sk i for i =1 ::: n. Only one element of d k is changed in each iteration, so that the work per iteration is bounded by O(mn) using rank-one updating techniques. The usual primal-dual iterates x k s k are not needed explicitly in the algorithm, whereas d k v k are iterated so that the interior primal-dual solutions can always be recovered by afore mentioned relations between (x k s k ) (d k v k ) with improving primal-dual potential function values. Moreover, no approximation of d k is needed in the computation of projection directions. Key words: linear programming, interior point method, potential function. 1 Econometric Institute, Erasmus University Rotterdam, The Netherls, sturm@few.eur.nl. Econometric Institute, Erasmus University Rotterdam, The Netherls, zhang@few.eur.nl
Sturm Zhang: IPM based on rank-one updates 1 1. Introduction In his seminal paper [8], Karmarkar proposed a projective potential reduction method that solves linear programs in O(nL) main iterations, where n sts for the number of inequality constraints, L is the input length of the problem. A single iteration of Karmarkar's algorithm requires solving a system of linear equations, which takes O(n 3 ) operations in the worst case. By using incomplete updates of this equation system, the average work per step can be reduced to O(n :5 ). Hence, Karmarkar's algorithm achieves an overall complexity of O(n 3:5 L) for linear programming. The O(n 3:5 L) complexity was further reduced to O(n 3 L)byVaidya [17] Gonzaga [5]. Kojima, Mizuno Yoshise [9] Monteiro Adler [13] showed that this complexity can also be achieved by the primal-dual path-following method, if partial updates are used. However, the algorithms proposed by [8, 17, 5, 9, 13] use a xed step length per iteration, which makes the algorithms inecient for practical purposes. Adaptive step algorithms that are based on a partial updating scheme were proposed by Anstreicher [1], Anstreicher Bosch [], Den Hertog [6], Den Hertog, Roos Vial [7], among others. The afore mentioned partial updating methods require O(n :5 )operationsonaverage per iteration. Algorithms that require on average only O(n ) operations per iteration were proposed by Ye[19] Mizuno [11]. Interestingly, Mizuno [1] also obtained O(n 3 L) complexity with a potential reduction algorithm that requires at most O(n ) operations per iteration, as opposed to [19, 11] where each iteration requires O(n ) operations on average. At each iteration of Mizuno's method [1], the amount of required operations is thus comparable to that of one simplex pivot step. A variant of Mizuno's algorithm that allows for up to any xed number of updates is analysed by Bosch[3]. The algorithm to be proposed in this paper is similar to Mizuno's algorithm [1] in the sense that it requires only O(n ) operations per main iteration. However, the way in which the new algorithm computes the iterative primal dual solutions is dierent. In general, the major computational eorts needed in interior point methods for linear programming are related to the computation of projections. Denote x k s k to be the primaldual iterates at iteration k. We scale the linear program by a positive vector d k, then determine a vector v k as the unique intersection point of the scaled primal dual feasible sets. For primal methods, one needs to project onto the null space of AX k, where X k = diag(x k 1 xk ::: xk n). For dual methods, the projection onto the range space of (S k ) ;1 A T is needed with S k = diag(s k 1 sk ::: sk n). Finally, for primal-dual methods, one has to project onto the null space of AD k with D k = diag(d k 1 dk ::: dk n). Previous partial updating methods work with approximations of either X k, S k or D k, which are updated in such away that the computational costs are reduced. In our new method, the scaling vector d k is computed
Sturm Zhang: IPM based on rank-one updates exactly as (X k ) 1= (S k ) ;1= e.however, we change only one component ofd k at a time. As a result, only O(n ) arithmetic operations is needed using a rank-one updating scheme for the projection matrices. In this procedure, no approximation is involved. Another feature of our new algorithm is that the traditional primal dual iterates x k s k are invisible in the solution procedure. Their functions are completely replaced by the new vectors d k v k. Unfortunately, our algorithm seems to be less eective in reducing the potential function value than Mizuno's algorithm. To be more precise, we arrive atano(n 1:5 L)worst case iteration bound. Hence, the new algorithm needs O(n 3:5 L) operations in total, just like Karmarkar's algorithm. The paper is organized as follows. In Section, we briey discuss the primal-dual geometry of linear programming. The eect of a rank-1 update on the primal-dual solution pair is derived in Section 3. Subsequently, the new potential reduction algorithm is stated in Section 4, we proceed in Section 5 by deriving the complexity result. We conclude the paper in Section 6.. Problem discussions In this paper we consider the stard linear program: (P ) min x fc T x j Ax = b x 0g its dual (D) max y s fb T y j A T y + s = c s 0g where A is an mn matrix, n, b c are vectors of dimension m n respectively. We make the stard assumptions that A has full row rank that (P) (D) have interior feasible solutions. For given positive vector d < n ++, let F P (d) :=fx j ADx = bg where D := diag(d 1 d ::: d n ) let F D (d) :=fs j DA T y + s = Dc for some y < m g: Denoting the all-one vector by e, it is easily seen that primal feasibility amounts to x F P (e) x 0
Sturm Zhang: IPM based on rank-one updates 3 dual feasibility is equivalent to s F D (e) s 0: As F P (d) F D (d) are orthogonal complementary ane spaces, they share a single intersection point, say v d : fv d g = F P (d) \F D (d): Let x d := Dv d s d := D ;1 v d : Obviously, x d F P (e) s d F D (e). Because of the positivity ofd, we also have x d 0s d 0 if only if v d 0: A natural question arises: Is v d always nonnegative for positive d? Or equivalently: Are x d s d always feasible? As expected, the answer is negative. A simple example is: Example.1. Let A = h 1 h 1 1 i v d = yields 4 ; 3 3 5 which has a negative component. i b = h 4 i c T = h 0 7 i. Then, choosing d T = If, on the other h, interior feasible solutions x F P (e), x>0s F D (e), s>0 are known, then it is easy to construct d such that v d > 0. Namely, letting q d i = x i =s i for i =1 ::: n yields (v d ) i = p x i s i for i =1 ::: n: This triggers the next question: Assuming that we have a pair of interior feasible solutions, is it possible to update the related vector d such that the feasibility is retained, at the same time, the solutions are improved?
Sturm Zhang: IPM based on rank-one updates 4 The goal of this paper is to give an armative answer to the above question. Namely, we propose an algorithm that generates a sequence fd k j k =0 1 g for which the associated sequence of primal-dual feasible pairs (x d k s d k)converges to a pair of optimal solutions for (P) (D). First we derive formulas to express v d, x d s d for a given vector d < n ++.LetQ d P d denote the projection matrices onto the image of DA T the kernel of AD respectively, i.e. Q d := DA T (AD A T ) ;1 AD P d := I ; Q d : By denition, v d F P (d) \F D (d), i.e. ADv d = b (1) DA T y d + v d = Dc () for some y d < m. Pre-multiplying (1) with DA T (AD A T ) ;1 yields Q d v d = DA T (AD A T ) ;1 b pre-multiplying () with P d yields P d v d = P d Dc: Hence, v d = Q d v d + P d v d = DA T (AD A T ) ;1 b +(I ; DA T (AD A T ) ;1 AD)Dc which satises (1) () with y d =(AD A T ) ;1 (AD c ; b): (3) 3. Single component update In this section, we analyze the eect of a change in a single component ofd on the associated pair (x d s d ). Suppose that we update component d j, for some j f1 ng. For a step size t>;1, let the new vector of scalings d(t) be dened as follows: d j (t) = p 1+td j
Sturm Zhang: IPM based on rank-one updates 5 d i (t) =d i for i f1 ngnfjg i.e. the components other than j remain unchanged. Let We dene D(t) := diag(d 1 (t) d (t) ::: d n (t)): u d := (AD A T ) ;1 ADe j where e j is the j-th unit vector. Since AD(t) A T = AD A T + t(ade j )(ADe j ) T (ADe j ) T (AD A T ) ;1 (ADe j )=e T j Q de j = e T j QT d Q de j = kq d e j k using the rank-1 updating formula of Sherman, Morrison Woodbury, we obtain (AD(t) A T ) ;1 =(AD A T ) ;1 ; Furthermore, AD(t) c = AD c + t(d j c j )ADe j : It follows using (3) that y d(t) = (AD(t) A T ) ;1 (AD(t) c ; b) = y d + t(d j c j )u d From (3) () we have t 1+t kq d e j k u du T d : ;t t(d jc j ) kq d e j k + u T d (AD c ; b) 1+t kq d e j k u d : u T d (AD c ; b)=e T j DAT y d = d j c j ; (v d ) j therefore y d(t) = y d + Now it follows that t(v d ) j 1+t kq d e j k u d: Ds d(t) = Dc ; DA T y d(t) t = v d ; 1+tkQ d e j k (v d) j Q d e j : (4)
Sturm Zhang: IPM based on rank-one updates 6 As x d(t) = D(t)v(t)=D(t) s d(t),wehave D ;1 x d(t) = (D ; D(t) )Ds d(t) We introduce (t), (t) := = (I + te j e T j )Ds d(t) t = v d ; 1+tkQ d e j k (v d) j Q d e j +t(v d ) j e j ; t kq d e j k 1+tkQ d e j k (v d) j e j = t v d + 1+tkQ d e j k (v d) j P d e j : (5) t 1+t kq d e j k which is a strictly increasing function for t>;1. This function maps the interval (;1 1) into the interval f jkp d e j k >;1 kq d e j k <1g: The inverse transformation of (t) is t() = 1 ; kq d e j k : (6) From now on, we consider as the step size parameter, we compute t() from(6).asa matter of notation, we write x() :=x d(t()) s():=s d(t()) v() :=v d(t()) : If no confusion is possible, we will dispense with the argument d simply write P, Q, v, x, s y to denote P d, Q d, v d, x d, s d y d.nowwe rewrite (5) (4) as follows: D ;1 x() =v + v j Pe j (7) Ds() =v ; v j Qe j : (8) These relations imply that kv()k = x() T s() = (v + v j Pe j ) T (v ; v j Qe j ) = kvk + v j e T j (P ; Q)v: (9)
Sturm Zhang: IPM based on rank-one updates 7 Hence, the duality gap between x() s() is linear in. As is common in the interior point methodology, we restrict ourselves to positive, i.e. interior, solutions x>0 s>0. Introducing the quantity v min := minfv 1 v v n g we obtain from (7) (8) that x() > 0 s() > 0 as long as j j< v min =v j. 4. A potential reduction algorithm Consider for x>0 s>0 the Tanabe-Todd-Ye primal-dual potential function [16, 18] (x s) =(n + l)log(x T s) ; nx i=1 log(x i s i ) ; n log n where we letl := p n in this paper. From the arithmetic-geometric mean inequality, it is easily seen that (x s) p n log(x T s) equality holds only if v = X 1= S 1= e is a positive multiple of the all-one vector e. This observation is the essence of the proof of the following lemma, which is due to Todd Ye [16] Ye [18]. Lemma 4.1. If an algorithm reduces (x s) by at least a xed quantity >0 in each iteration, then the algorithm can be used to solve the pair (P), (D) in O( p nl=) iterations. Let p ij := e T i Pe j for i =1 ::: n q ij := e T i Qe j for i =1 ::: n: For j j< v min =v j,wehave from (7), (8) (9) that (x() s()) = (x s)+(n + p n) log(1 + v je T j ; nx i=1 log(1 + v j v i p ij ) ; nx i=1 (P ; Q)v kvk ) log(1 ; v j v i q ij ): (10)
Sturm Zhang: IPM based on rank-one updates 8 Let where r := (P ; Q)( (n + p n) kvk v ; V ;1 e) (11) V := diag(v 1 v ::: v n ): It is easily seen from (10) that (x() s()) = (x s)+v j r j + o(): We propose to select the index j such that j r j j= krk 1. This choice results in the following algorithm. Algorithm 1. (A,b,c,d 0 ) The initial scaling d 0 > 0 is chosen such that v d 0 > 0. Step 0 Initialization. Set k =0. Step 1 Optimality test.let d = d k.stopifbased on(x d s d ) an optimal pair (x s ) can be found. Step Choose the updating index. Compute r according to (11). Set j = arg max 1in j r i j Step 3 Step size choice. Compute such that (x d () s d ()) <(x d s d ) ; 1 6(n +) Step 4 Take step. Set d k+1 = d +( p 1+t() ; 1)d j e j. Step 5 Set k = k +1 return to Step 1. Notice that in order to perform the algorithm it is necessary to iteratively compute v r. To do this, it is sucient to update the matrices (AD A T ) ;1 (AD A T ) ;1 AD. Using the rank-one updating formula, this will take O(mn) operations, whereas a main iteration of other interior point algorithms typically requires O(m n) operations in the worst case. Also notice that the index selection rule, i.e. Step, is similar to the most negative pivot rule of the simplex method.
Sturm Zhang: IPM based on rank-one updates 9 5. Convergence analysis In estimating the reduction (x s) ; (x() s()), we shall make use of the inequality log(1 + ) ; (1; j j) which is proven by Karmarkar [8]. Applying (1) to (10) yields for j j< v min =v j for any (;1 1) (1) (x s) ; (x() s()) ;v j r j ; v j kpe j k + kqe j k vmin (1; j j v j =v min ) = (v j =v min ) ;v j r j ; (1; j j v j =v min ) : (13) Due to the index selection rule in Algorithm 1, there holds j r j j= krk 1. The following lemma provides a lower bound on krk 1. Lemma 5.1. It holds that Proof: krk 1 > 1 p nvmin : Obviously, wehave forany w < n that k(p ; Q)wk = k(p + Q)wk = kwk : Therefore, we have krk = (n + p n) kvk v ; V ;1 e = p n kvk v + n kvk v ; V ;1 e : Using v? n kvk v ; V ;1 e, it follows that krk = n kvk + n = > kvk v ; V ;1 e n kvk +( n kvk v min ; 1 ) v min 1 v min 1 v min : + n v min kvk 4 : Since krk 1 krk = p n, (14) implies the desired result. (14)
Sturm Zhang: IPM based on rank-one updates 10 Combining Lemma 5.1 (13) we obtain a lower bound on the reduction (x s) ; (x() s()): Lemma 5.. For = ; jr jj r j Proof: (x s) ; (x() s()) > v p min =v j, we have 3 n+ 1 3(n +) : From (13) the fact that j r j j= krk 1 it follows that (x s) ; (x() s()) ; r j j r j j v (v j =v min ) j krk 1 ; (1; j j v j =v min ) : Therefore, by choosing = ; jr jj v r min =v j j 3 n+ using Lemma 5.1, we obtain (x s) ; (x() s()) > p 3 n(n +) ; 9(n + )(1 ; =(3 p n +)) > 3(n +) (1 ; 1 )= 1 3(n +) : Lemma 5. provides a possible step size rule for Step 3 in Algorithm 1. A better step size may be obtained by means of some line search procedure. Combining Lemma 4.1 Lemma 5., we obtain the main result of this paper: Theorem 5.1. Algorithm 1 solves (P) (D) in O(mn :5 L) operations. Proof: From Lemma 5. we know that in every iteration, the potential function is reduced by 1 at least a quantity of. Using Lemma 4.1, this implies that Algorithm 1 can be used 3(n+) to solve (P) (D) in O(n 1:5 L) iterations. A single iteration involves O(mn) operations. Thus, in total the algorithm requires O(mn :5 L) operations in the worst case. 6. Concluding Remarks It is known that interior point algorithms usually require signicantly fewer iterations for solving linear programs than simplex algorithms. Yet, the simplex method can outperform the interior point method in running time on a wide range of problems because of the low computational complexity per step. Similar to the algorithm of Mizuno [1], the interior point algorithm that we proposed in this paper has the property that the amount ofwork per step is comparable to the work in a simplex iteration. Using a new approach for computing the primal dual iterates, we do not need to distinguish \real" \approximate" scaling vectors as in [1, 3].
Sturm Zhang: IPM based on rank-one updates 11 References [1] K.M. Anstreicher, \A stard form variant, safeguarded linesearch, for the modied Karmarkar algorithm," Mathematical Programming 47 (1990) 337-351. [] K.M. Anstreicher R.A. Bosch, \Long steps in a O(n 3 L) algorithm for linear programming," Mathematical Programming 54 (199) 51-65. [3] R.A. Bosch, \On Mizuno's rank one updating algorithm for linear programming," SIAM Journal on Optimization 3 (1993) 861-867. [4] R.A. Bosch K.M. Anstreicher, \On partial updating in a potential reduction linear programming algorithm of Kojima, Mizuno, Yoshise," Algorithmica 9 (1993) 184-197. [5] C.C. Gonzaga, \An algorithm for solving linear programming problems in O(n 3 L) operations," in: N. Megiddo, ed. Progress in Mathematical Programming, Springer{Verlag (Berlin, 1989) 1-8. [6] D. Den Hertog, Interior point approach to linear, quadratic convex programming, algorithms complexity. Kluwer Publishers, Dordrecht, The Netherls, 1994. [7] D. Den Hertog, C. Roos J.-Ph. Vial, \A complexity reduction for the long{step path{ following algorithm for linear programming," SIAM Journal on Optimization (199) 71-87. [8] N.K. Karmarkar, \A new polynomial-time algorithm for linear programming," Combinatorica 4 (1984) 373-395. [9] M. Kojima, S. Mizuno A. Yoshise, \A polynomial{time algorithm for a class of linear complementarity problems," Mathematical Programming 44 (1989) 1-6. [10] S. Mehrotra, \Deferred rank one updates in O(n 3 L)interior point algorithms," Technical Report 90-11, Dept. of Industrial Engineering Management Sciences, Northwestern University (Evanston, IL, 1990). [11] S. Mizuno, \O(n L) iteration O(n 3 L) potential reduction algorithms for linear programming," Linear Algebra its Applications 15 (1991) 155-168. [1] S. Mizuno, \A rank one updating algorithm for linear programming," The Arabian Journal for Science Engineering 15 (1990) 671-677. [13] R.D.C. Monteiro I. Adler, \Interior path following primal{dual algorithms. Part II: convex quadratic programming," Mathematical Programming 44 (1989) 43-66. [14] J. Renegar, \A polynomial time algorithm, based on Newton's method, for linear programming," Mathematical Programming 40 (1988) 59-93. [15] D.F. Shanno, \Computing Karmarkar projections quickly," Mathematical Programming 41 (1988) 61-71. [16] M.J. Todd Y. Ye, \A centered projective algorithm for linear programming," Mathematics of Operations Research 15 (1990) 508-59. [17] P.M. Vaidya, \An algorithm for linear programming which requires O(((m + n)n +
Sturm Zhang: IPM based on rank-one updates 1 (m + n) 1:5 n)l) arithmetic operations," Proceedings of the 19th ACM Symposium on the Theory of Computing (1987) 9-38. [18] Y. Ye, \An O(n 3 L) potential reduction algorithm for linear programming," Mathematical Programming 50 (1991) 39-58. [19] Y. Ye, \Further developments in potential reduction algorithm," Technical Report, Department of Management Sciences, University ofiowa, 1989.