Operations Research Lecture 4: Linear Programming Interior Point Method

Size: px

Start display at page:

Download "Operations Research Lecture 4: Linear Programming Interior Point Method"

Dana Sharp
6 years ago
Views:

1 Operations Research Lecture 4: Linear Programg Interior Point Method Notes taen by Kaiquan School, Nanjing University April 14th The affine scaling algorithm one of the most efficient and yet simple interior point algorithms for solving the LP problem. The standard form of LP and its dual c T x x 0 max p T b p T A c T Let P = {x AX = b, x 0} be the feasible set. {x P x > 0} is called the interior of P, and its elements are called interior points. It is difficult to imize c T x over P, but it is rather easy to optimize the same cost function over an ellipsoid, and the solution can be obtained in close form. So we can solve a series of optimization problems over ellipsoids: Starting with x 0 > 0, an interior point of P, we form an ellipsoid S 0 centered at x 0, which is contained in the interior of P. Then we optimize c T x over all x S, and find a new interior point x 1. We draw another ellipsoid S 1 centered at x 1 and proceed in a similar way. As illustrated in Figure ( 1). The first step: create an ellipsoid centered at a given y, such that all elements of the ellipsoid are positive Lemma 1. Let β (0, 1),y R n and y > 0, let Y = diag(y 1,..., y n ), n S = {x R n (x i y i ) 2 i=1 y 2 i β 2 } = {x R n Y 1 (x y) β} Then x > 0 for every x S Proof: (sip) The set S 0 = S {x } is a section of the ellipsoid S, and is itself an ellipsoid contained in the feasible set. 1

2 Figure 1: Affine Scaling Algorithm The second step: imize over the ellipsoid S 0 c T x Y 1 (x y) β let d = x y, then Ad = 0, since of Ay = b and. The above problem becomes The closed form solution of this problem is: c T d Ad = 0 Y 1 d β Lemma 2. Assume that the rows of A are linearly independent and that c is not a linear combination of the rows of A. Let y be a positive vector. Then, an optimal solution d to problem is given by d = β Y2 (c A T p) Y(c A T p) where p = (AY 2 A T ) 1 AY 2 c. Furthermore, the vector x = y + d belongs to P and c T x = c T y β Y(c A T p) < c T y Proof: General idea: first show p is well defined; then d is feasible solution to the problem by validating d satisfies constrains; to prove the optimality of d, compare c T d and c T d (d is a feasible solution), here the Schwartz inequality is used. An interpretation of the formula for p: if y is a nondegenerate basic feasible solution, by some calculation, p = (B T ) 1 c B, which is the corresponding dual basic solution. So we call p the dual estimates even if y is not a basic feasible solution. r = c A T p becomes r = c A T (B T ) 1 c B, the reduced cost in the simplex method. r T y = (c A T p) T y = c T y p T Ay = c T y p T b, the difference in objective values between the primal solution y and the dual solution p, which is called duality gap 2

3 Lemma 3. Let y and p be a primal and a dual feasible solution, respectively, such that c T y b T p < ɛ. Let y and p be optimal primal and dual solutions, respectively. Then c T y c T y < c T y + ɛ This is called ɛ-optimization. b T p ɛ < b T p b T p Proof: Since y is a primal feasible solution and y is an optimal primal solution, c T y c T y. By wea duality, b T p c T y. We have c T y < b T p + ɛ c T y + ɛ Similarly, we obtain b T p = c T y c T y < b T p + ɛ The affine scaling algorithm 1. (Initialization) Start with some feasible x 0 > 0, let = 0 2. (Computation of dual estimates and reduced costs) Given some feasible x > 0, let X = diag(x 1,..., x n) p = (AX 2 A T ) 1 AX 2 c r = c A T p 3. (Optimality chec) Let e = (1, 1,..., 1). If r 0 and e T X r < ɛ, then stop; the current solution x is primal ɛ-optimal and p is dual ɛ-optimal. 4.(Unboundedness chec) If X 2 r 0 then stop; the optimal cost is. 5.(Update of primal solution) Let, here 0 < β < 1 x +1 = x β X2 r X r This version is called short-step method. There are some long-step methods: or here u = max i u i and γ(u) = max{u i u i > 0} X2 r x +1 = x β X r X2 r x +1 = x β γ(x r ) Convergence: Under certain assumption, the affine scaling algorithm converge to the optimal primal and dual solutions respectively. Initialization: We need an interior feasible solution, by solving the augmented problem c T x + Mx n+1 Ax + (b Ae)x n+1 = b (x, x n+1 ) 0 3

4 where M is a large positive scalar. Notice that (x, x n+1 ) = (e, 1) is a positive feasible solution, and the affine scaling algorithm can be applied. If M is very large, the augmented problem will provide an optimal solution to the original problem. Computational performance: If the current point is near the boundary of the feasible set, the algorithm taes small steps. But if the current point lie deep inside the feasible set, the algorithm maes rapid progress. 2 The primal path following algorithm The primal path following algorithm offers interesting geometric insights, and has been incorporated in large-scale implementation. The standard form of LP and its dual c T x x 0 max p T b p T A c T The core difficulty for solving LP problem is the presence of the inequality constrains x 0. So we can convert the LP problem to a problem with only equality constrains using barrier function, which prevents any variable from reaching the boundary: n B µ (x) = c T x µ log x j where µ > 0 For the barrier problems j=1 B µ x (1) Given a value of µ, the barrier problem has an optimal solution x(µ). As µ varies, x(µ) form the central path. When µ 0, x(µ) x, the optimal solution of the initial LP problem. The reason is: when µ is very small, the logarithmic term is negligible. A barrier problem from the dual problem is max p T b + µ n log s j j=1 p T A + s T = c T When µ =, the optimal solution of the problem ( 1) is called the analytic center of the feasible set. As illustrated in Figure ( 2) 4

x 2 µ log x 3 x 1 + x 2 + x 3 = 1 x 2 µ log x 1 µ log x 2 µ log (1 x 1 x 2 ) By taing derivatives, and setting them equal to zero, we find the central path is given by x 1 (µ)

5 Figure 2: The central path and the analytic center Example 4. (Computation of the central path) Consider the problem The barrier problem is Substituting x 3 = 1 x 1 x 2, we have x 2 x 1 + x 2 + x 3 = 1 x 1, x 2, x 3 0 x 2 µ log x 1 µ log x 2 µ log x 3 x 1 + x 2 + x 3 = 1 x 2 µ log x 1 µ log x 2 µ log (1 x 1 x 2 ) By taing derivatives, and setting them equal to zero, we find the central path is given by x 1 (µ) = 1 x2(µ) 2 x 2 (µ) = 1+3µ 1+9µ 2 +2µ 2 x 3 (µ) = 1 x2(µ) 2 The analytic center can be found by letting µ, which is (1/3, 1/3, 1/3). And the central path can be got when µ taes value from 0 to. As illustrated in Figure ( 3). Figure 3: The central path and the analytic center in the example 5

6 The barrier problem is still difficult to solve, since the objective function is neither linear nor quadratic. Here, we approximate the barrier function around some given x by its Taylor series expansion. B µ (x + d) B µ (x) + n i=1 B µ (x) x i d i Then the approximating problem becomes: n i,j=1 2 B µ (x) d i d j = B µ (x) + (c T µe T X 1 )d + 1 x i x j 2 µdt X 2 d (c µe T X 1 )d µdt X 2 d Ad = 0 This problem can be solved in closed form using the Lagrange multipliers method: associate a vector p of Lagrange multipliers to the constrains Ad = 0 and form the Lagrangean function L(d, p) = (c T µe T X 1 )d µdt X 2 d p T Ad By taing derivatives, and setting them equal to zero, we get Solving this system of equations, c µx 1 e + µx 2 d A T p = 0 Ad = 0 (2) d(µ) = (I X 2 A T (AX 2 A T ) 1 A)(Xe 1 µ X2 c) p(µ) = (AX 2 A T ) 1 A(X 2 c µxe) (3) d(µ) is called the Newton direction. Now, x moves to x + d(µ), and the dual solution becomes (p(µ), c A T p(µ)) Geometric interpretation: With fixing µ, we carry out several Newton steps, x would converge to x(µ). We then reduce µ to µ = αµ, here 0 < α < 1. While α is close to 1, then x( µ) is close to x(µ). So x is close to x( µ). By carrying out a Newton step, x moves even closer to x( µ). That is the feasible solution x approximately follows the central path, so this method is called path following. As illustrated in Figure ( 4). Figure 4: Geometric interpretation The primal path following algorithm 1.(Initialization) Start with some primal and dual feasible x 0 > 0, s 0 > 0, p 0, and set = 0 2.(Optimality test) If (s ) T x < ɛ stop; else go to Step 3. 6

7 3. Let 4. (Computation of directions) Solve the linear system X = diag(x 1,.., x n), µ +1 = αµ. µ +1 X 2 d AT p = µ +1 X 1 e c Ad = 0 (4) for p and d 5. (Update of solutions) Let 6. Let := + 1 go to step 2 x +1 = x + d p +1 = p s +1 = c A T p 3 The primal-dual path following algorithm This method approximates the central path by taing Newton steps, and finds Newton directions both in the primal and the dual space. It has excellent performance in large scale applications. The barrier optimization problems and c T x µ n log x j j=1 max p T b + µ n log s j j=1 (6) p T A + s T = c T Similar to the complementary slacness for a LP problem, the following conditions are sufficient and necessary for an optimal solution to the above problems (called KKT optimization conditions, to be covered in the following lectures). Ax(µ) = b x(µ) 0 A T p(µ) + s(µ) = c s(µ) 0 X(µ)S(µ)e = eµ This system of equation is nonlinear, and difficult to solve directly. Here the Newton s method is used to solve it interactively. Newton s method for finding roots of nonlinear equations For finding a z such that F(Z ) = 0, here F is a mapping from R r into R r. Assug z is an approximation for z. The following method is used to improve the approximation. Expand F(z) around z using a first order multivariable Taylor series F(z + d) F(z ) + J(z )d (5) (7) 7

8 Here J(z ) is the Jacobian matrix. We loo for some d that satisfies F(z ) + J(z )d = 0 (8) Then set z +1 = z + d. d is called a Newton direction. The method converges quicly to z under certain conditions. Application of Newton s method to LP Use Newton s method to solve the system of nonlinear equations ( 7), here z = (x, p, s) and Ax b F(z) = A T p + s c XSe µe Let (x, p, s ) be the current primal and dual feasible solution, with x > 0 and s > 0. By applying Eq. ( 8), the Newton direction is detered by: A 0 0 d 0 A T x Ax b I d p = A T p + s c S 0 X d s X S e µe Since Ax = b and A T p + s = c, this is equivalent to Ad x = 0 (9) A T d p + d s = 0 (10) S d x + X d s = µ e X S e (11) The solution to this linear system is given by where Step lengths d x = D (I P )v (µ ) d p = (AD 2 A T ) 1 AD v (µ ) d s = D 1 P v (µ ) D 2 = X S 1 P = D A T (AD 2 A T ) 1 AD v (µ ) = X 1 D (µ e X S e) The new solution is updated by: x +1 = x + β P d x p +1 = p + β D d p s +1 = s + β D d s where βp and β D are step lengths for the primal and dual variables. For preserving nonnegativity of x+1 and s +1, we tae βp = {1, α ( x i )} {i (d x )i<0} (d x )i βd = {1, α ( s i {i (d s )i<0} (d s )i )} 8

9 where 0 < α < 1 to avoid approaching the boundary of the primal and dual feasible set. The equality constrains of primal and dual problems are also satisfied, because of Eqs.( 9) and ( 10). Updating the barrier parameter µ One way to update µ that has had excellent computational performance is to set µ = ρ (x ) T s n where ρ is typically set to 1, except if the algorithm has not progressed, ρ is set to a value smaller than 1. The primal-dual path following algorithm 1. (Initialization) Start with some feasible x 0 > 0,s 0 > 0, p 0, and set = (Optimality test) If (s ) T x < ɛ stop; else go to Step (Computation of Newton directions) Let ρ (0, 1] and µ = ρ (x ) T s n X = diag(x 1,..., x n), S = diag(s 1,..., s n), Solve the linear system ( 9)-( 11) for d x,d p and d s. 4. (Find step lengths) Let βp = {1, α ( x {i (d x )i<0} βd = {1, α ( s i {i (d s )i<0} (d s 5. (Solution update) Update the solution vectors according to Let := + 1 and go to Step 2. x +1 = x + β P d x p +1 = p + β D d p s +1 = s + β D d s i )} (d x )i Infeasible primal-dual path following methods Without requiring x, s and p necessarily feasible, i.e. Ax b and A T p + s c. The Newton direction is derived in exactly the same manner and requires to solve the following system of equations: A A T I S 0 X d x d p d s = )i )} Ax b A T p + s c X S e µe 4 References 1. Chapter X, Operations Research (Third Version), Tsinghua Unversity Press, Chapter 9, Dimitris Bertsimas, John N. Tsitsilis, Introduction to Linear Programg, Athena Scientific,

10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside