Primal-dual IPM with Asymmetric Barrier

Size: px

Start display at page:

Download "Primal-dual IPM with Asymmetric Barrier"

Coleen Cummings
5 years ago
Views:

1 Primal-dual IPM with Asymmetric Barrier Yurii Nesterov, CORE/INMA (UCL) September 29, 2008 (IFOR, ETHZ) Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 1/28

2 Outline 1 Symmetric and asymmetric barriers 2 Feasible-start potential reduction IPM 3 Infeasible start potential-reduction IPM 4 Infeasible-start long-step path-following IPM 5 Computational aspects Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 2/28

3 Nonlinear Conic Optimization Problem Primal-Dual Problem: min x { c, x : Ax = b, x K}? = max s,y { b, y : s+a y = c, s K }, where K is a normal cone (convex, pointed, solid), and K = {s : s, x 0, x K}. Main Assumptions: 1. x 0 int K, s 0 int K, y 0 H : Ax = b, s + A y 0 = c. 2. F P (x) is a ν FP -normal barrier for K (self-concordant, log-homogenerous: F P (τx) = F (x) ν FP ln τ, x int K) Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 3/28

4 Nonlinear Conic Optimization Problem Primal-Dual Problem: min x { c, x : Ax = b, x K}? = max s,y { b, y : s+a y = c, s K }, where K is a normal cone (convex, pointed, solid), and K = {s : s, x 0, x K}. Main Assumptions: 1. x 0 int K, s 0 int K, y 0 H : Ax = b, s + A y 0 = c. 2. F P (x) is a ν FP -normal barrier for K (self-concordant, log-homogenerous: F P (τx) = F (x) ν FP ln τ, x int K) 3. Important: we can compute the dual barrier F D (s) = FP def (s) = max[ s, x F P (x)] x Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 3/28

5 Examples Positive orthant K = {x R n : x 0} (= K ), F P (x) = n ln x (i), ν P = n. i=1 Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 4/28

6 Examples Positive orthant Lorentz cone K = {x R n : x 0} (= K ), F P (x) = n ln x (i), ν P = n. i=1 K = {(τ, x) R n+1 : τ x 2 } (= K ), F P (τ, x) = ln(τ 2 x 2 2 ), ν P = 2. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 4/28

7 Examples Positive orthant Lorentz cone K = {x R n : x 0} (= K ), F P (x) = n ln x (i), ν P = n. i=1 K = {(τ, x) R n+1 : τ x 2 } (= K ), F P (τ, x) = ln(τ 2 x 2 2 ), ν P = 2. Cone of positive-semidefinite matrices K = {X S n : X 0} (= K ), F P (X ) = ln det X, ν P = n. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 4/28

8 Symmetric primal-dual IPM (F D = F P ) Primal-dual central path: (x(t), s(t), y(t)) K K H: x(t) = arg min [t c, x + F P(x)], t > 0, Ax=b y(t) = arg max[t b, y F D (c A T y)]. y Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 5/28

9 Symmetric primal-dual IPM (F D = F P ) Primal-dual central path: (x(t), s(t), y(t)) K K H: x(t) = arg min [t c, x + F P(x)], t > 0, Ax=b y(t) = arg max[t b, y F D (c A T y)]. y Many important identities: Ax(t) = b, s(t) + A y(t) = c, s(t) = 1 t F P(x(t)), unbounded {}}{ x(t) = 1 t F D(s(t)), c, x(t) b, y(t) = 1 t ν F P, F P (x(t)) + F D (s(t)) = ν ν ln t. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 5/28

10 Symmetric primal-dual IPM (F D = F P ) Primal-dual central path: (x(t), s(t), y(t)) K K H: x(t) = arg min [t c, x + F P(x)], t > 0, Ax=b y(t) = arg max[t b, y F D (c A T y)]. y Many important identities: Ax(t) = b, s(t) + A y(t) = c, s(t) = 1 t F P(x(t)), unbounded {}}{ x(t) = 1 t F D(s(t)), c, x(t) b, y(t) = 1 t ν F P, F P (x(t)) + F D (s(t)) = ν ν ln t. Consequences: Good search directions (primal-dual affine-scaling, centering) Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 5/28

11 Symmetric primal-dual IPM (F D = F P ) Primal-dual central path: (x(t), s(t), y(t)) K K H: x(t) = arg min [t c, x + F P(x)], t > 0, Ax=b y(t) = arg max[t b, y F D (c A T y)]. y Many important identities: Ax(t) = b, s(t) + A y(t) = c, s(t) = 1 t F P(x(t)), unbounded {}}{ x(t) = 1 t F D(s(t)), c, x(t) b, y(t) = 1 t ν F P, F P (x(t)) + F D (s(t)) = ν ν ln t. Consequences: Good search directions (primal-dual affine-scaling, centering) Long-step IPM, infeasible-start IPM. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 5/28

12 Symmetric primal-dual IPM (F D = F P ) Primal-dual central path: (x(t), s(t), y(t)) K K H: x(t) = arg min [t c, x + F P(x)], t > 0, Ax=b y(t) = arg max[t b, y F D (c A T y)]. y Many important identities: Ax(t) = b, s(t) + A y(t) = c, s(t) = 1 t F P(x(t)), unbounded {}}{ x(t) = 1 t F D(s(t)), c, x(t) b, y(t) = 1 t ν F P, F P (x(t)) + F D (s(t)) = ν ν ln t. Consequences: Good search directions (primal-dual affine-scaling, centering) Long-step IPM, infeasible-start IPM. Powerful software (SeDuMi, T3, Mosek, etc.) Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 5/28

13 Another applications 1. Flows in fluid networks min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

14 Another applications 1. Flows in fluid networks Electricity: γ = 2. min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

15 Another applications 1. Flows in fluid networks min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Electricity: γ = 2. Gas networks: γ = 3. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

16 Another applications 1. Flows in fluid networks min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Electricity: γ = 2. Gas networks: γ = 3. Water: γ = Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

17 Another applications 1. Flows in fluid networks min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Electricity: γ = 2. Gas networks: γ = 3. Water: γ = Usually, the dimension is not too big! Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

18 Another applications 1. Flows in fluid networks min c α f α γ (= Energy) f α A s.t. E f = d, (node balance) f i 0, i I. Electricity: γ = 2. Gas networks: γ = 3. Water: γ = Usually, the dimension is not too big! 2. Random scheduling: minimize in x the objective T, x + n n i=1 j=i+1 ( d (i) d (j) ln e x(i) /d (i) + e x(j) /d (j)) D ē, x. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 6/28

19 Asymmetric barriers: 1. Power cone { K α = x R+ 2 R : ( x (1)) α ( x (2) ) 1 α x (3) }, α (0, 1), F P,α (x) = ln [ (x (1) ) 2α ( x (2) ) 2(1 α) ( x (3) ) 2 ] ln x (1) ln x (2), K α = { s R 2 + R : ( ) s (1) α ( ) s (2) 1 α α 1 α s } (3), F D,α (s) = ln with ν FP,α = ν FD,α = 4. K α = [ ( ) s (1) 2α ( ) s (2) 2(1 α) ( α 1 α s (3) ) 2] ln s (1) ln s (2), ( α α ) K α, but F D,α F P,α. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 7/28

20 Asymmetric barriers: 2. Conic hull of the epigraph of exponent { } K P,e = x R R+ 2 : x (1) x (2) ln x(2), x (3) ( ) F P,e (x) = ln x (1) x (2) ln x(2) ln x (2) ln x (3), x (3) K D,e = { } s R + R R + : s (1) + s (2) s (1) ln s(1), s (3) ( ) F D,e (x) = ln s (1) + s (2) s (1) ln s(1) ln s (1) ln s (3), s (3) with ν FP,e = ν FD,e = 3. K D,e = F D,e F P,e. ( ) K P,e, and Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 8/28

21 Asymmetric barrier for primal-dual cone Denote Define Ψ(z) Ψ(x, s) = F P (x) + F D (s), ν Ψ = ν FP + ν FD, z = (x, s) int K int K def = int K. κ(z 0 ) = Ψ(z 0 ) min x,s {Ψ(z) : s 0, x + s, x 0 = 2 s 0, x 0 }, where z 0 = (x 0, s 0 ) is the point from A1. Note that κ(z 0 ) 0. Lemma: For any z K we have Ψ(z) Ψ(z 0 ) κ(z 0 ) ν Ψ ln s 0,x + s,x 0 2 s 0,x 0. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 9/28

22 Feasible-start potential reduction IPM Primal-dual problem: w = (x, s, y, τ) K K H R + : Ax = τb, s + A y = τc, Normalization constraint: τ = 1. Optimal value of the objective: c, x b, y = 0. Karmarkar setting: For cone C with ν-normal barrier F (x), consider min{ d, w : Bw = 0, e, w = 1} = 0. w C Can be solved by minimizing homogeneous potential ν ln d, w + F (w). Condition: the set {w C : Bw = 0, e, w = 1} is bounded. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 10/28

23 Feasible-start potential reduction IPM Primal-dual problem: w = (x, s, y, τ) K K H R + : Ax = τb, s + A y = τc, Normalization constraint: τ = 1. Optimal value of the objective: c, x b, y = 0. Karmarkar setting: For cone C with ν-normal barrier F (x), consider min{ d, w : Bw = 0, e, w = 1} = 0. w C Can be solved by minimizing homogeneous potential ν ln d, w + F (w). Condition: the set {w C : Bw = 0, e, w = 1} is bounded. NOTE: the set {(x, s, y) K K H : Ax = b, s + A y = c} is never bounded Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 10/28

24 Feasible-start potential reduction IPM Denote F = {w = (x, s, y, τ) K K H R + : Ax = τb, s + A y = τc}, F (w) = F P (x) + F D (s) ln τ, ν F = ν P + ν D + 1, φ(w) = ν F ln[ c, x b, y ] + F (w), w int F. Main identity: for w F we have s 0, x + s, x 0 = c, x b, y + τ s 0, x 0. Theorem: Let w 0 (x 0, s 0, y 0, 1). If we form w k = (x k, s k, y k, τ k ) by minimizing φ(w) by the Newton method, then for the point we have x k = x k τ k, s k = s k τ k, ỹ k = y k τ k { } s 0,x 0 c, x k b,ỹ k 2 exp ω k κ(z0 ) ln 2 ν F 1. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 11/28

25 Feasible-start potential reduction IPM Advantages: Polynomial-time O(ν F ln 1 ɛ ) complexity bound. Asymmetric barrier F (w). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 12/28

26 Feasible-start potential reduction IPM Advantages: Polynomial-time O(ν F ln 1 ɛ ) complexity bound. Asymmetric barrier F (w). Disadvantages: The worst-case complexity bound is not the best one. Feasible point is needed. Unavoidable expensive exact matrix operations. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 12/28

27 Full-dimensional potential reduction IPM Denote C {w = (x, s, y, τ) K K H R + }. By two positive-definite operators B H : H H, B E : E E, define the convex quadratic function Qw, w = Ax τb 2 B 1 H +[ c, x b, y ] 2 Optimal set: W = {w C : Qw = 0}. Main inequality: For any w C we have + s + A y τc 2 B 1 def = w 2 Q, E s 0, x + s, x 0 def = Ω {}}{ [1 + B H y 0, y 0 + B E x 0, x 0 ] 1/2 w Q +τ s 0, x 0, Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 13/28

28 Quadratic potential function Denote F (w) = Ψ(z) ln τ = F P (x) + F D (s) ln τ, Main properties Φ(w) = 1 2 w 2 Q + F (w), w int C. Φ(w) is a s.-c. function with positive definite Hessians. It is unbounded from below. Hence, the local norms of the gradients are all 1. For w int C, define homogeneous quadratic potential: Φ(w) = min Φ(λw) Φ(w). λ>0 The minimum is attained at λ = λ(w) def = ν1/2 F w Q. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 14/28

29 Properties of homogeneous potential It admits a closed-form representation: Φ(w) = ν F ln w Q + F (w) + ν F 2 [1 ln ν F ]. This is a quasi-convex function with zero degree of homogeneity. For any point w int C, we have Φ(w) Ψ(z 0 ) κ(z 0 ) + ν Ψ ln 2 + ν F 2 [1 ln ν F ] ν F ln ( Ω s 0,x 0 + τ w Q ). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 15/28

30 Possible strategies A. Minimization of quadratic potential 1 Compute point w k by a Newton step from w k. 2 Define w k+1 = λ(w k ) w k. B. Minimization of homogeneous potential At each iteration, apply Newton step to an upper convex approximation of Φ. In both cases, Φ is decreased by an absolute constant (at least). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 16/28

31 Possible strategies A. Minimization of quadratic potential 1 Compute point w k by a Newton step from w k. 2 Define w k+1 = λ(w k ) w k. B. Minimization of homogeneous potential At each iteration, apply Newton step to an upper convex approximation of Φ. In both cases, Φ is decreased by an absolute constant (at least). New even for LP! Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 16/28

32 Full-dimensional potential reduction IPM Advantages: Polynomial-time O(ν F ln 1 ɛ ) complexity bound. Asymmetric barrier F (w). Can start from infeasible points. Search directions may be computed by gradient schemes with reasonable accuracy. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 17/28

33 Full-dimensional potential reduction IPM Advantages: Polynomial-time O(ν F ln 1 ɛ ) complexity bound. Asymmetric barrier F (w). Can start from infeasible points. Search directions may be computed by gradient schemes with reasonable accuracy. Disadvantages: The worst-case complexity bound is not the best one. However, the long steps can accelerate the convergence. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 17/28

34 Central path for infeasible-start IPM Denote v = (x, s, τ) Ĉ def = K K R +, and ˆQv, v y(v) def = min{ Qw, w : w = (x, s, y, τ)}, def y = arg min{ Qw, w : w = (x, s, y, τ)}, y ˆF (v) = F P (x) + F D (s) ln τ, v Ĉ, νˆf = ν F, Φ(v) = 1 2 v 2ˆQ + ˆF (v), v int Ĉ. Ĉ is normal 2 ˆF ( ) 0. Central path: fix v 1 int Ĉ with λ(v 1) = 1. Define the central path v(µ) as µ 0: Φ(v(µ)) ˆQv(µ) + ˆF (v(µ)) = µ[ ˆQv 1 + ˆF (v 1 )] def = µg 1. Clearly, v(1) = v 1. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 18/28

35 Properties of the central path v(µ) ˆQ ν 1/2 F, µ (0, 1]. Denote V = {v Ĉ : ˆQv = 0}. For any v V we have µ ˆF (v 1 ), v = ˆF (v(µ)), v 2 ˆF (v(µ))v, v 1/2. If v = (x, s, 1) V, then s 0,x 0 µ ˆF (v 1 ),v s 0, x(µ) + s(µ), x 0. Since s 0, x + s, x 0 Ω v ˆQ + τ s 0, x 0, we get Lemma: For any µ (0, 1] τ(µ) v(µ) ˆQ 1 Ω ν 1/2 F ˆF (v 1 ),v µ s 0,x 0. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 19/28

36 Augmented self-concordant barriers (N. & Vial 2004) Denote ψ µ (v) = Φ(v) µ g 1, w. For v int Ĉ, consider two local metrics: σ v (g) = 2 ˆF (v)[ 2 Φ(v)] 1 g, [ 2 Φ(v)] 1 g, θ v (g) = g, [ 2 ˆF (v)] 1 g 1/2. (Simple to compute!) Properties: σ v (g) θ v (g). For all v int Ĉ, σ v ( Φ(v)) ν 1/2 F. For the Newton iterate v + = v [ 2 ψ µ (v)] 1 ψ µ (v), we have ( θv σ + ( ψ µ (v + ))) v ( ψ 2 µ(v)) 1 σv ( ψ µ(v))). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 20/28

37 Predictor-corrector technique Neighborhood of the Central Path: N β (µ) = {v : θ v ( ψ µ (v)) β}, [ µ (0, 1], β 0, ). Predictor step T v (α) = v [ 2 Φ(v)] 1 ψ µ (v) }{{} v(µ) α [ 2 Φ(v)] 1 g }{{} 1, α 0. v (µ) Goal: for v N β0 (µ) with β 0 β 1, choose the maximal α: T v (α) N β1 (µ α). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 21/28

38 Predictor-corrector technique Neighborhood of the Central Path: N β (µ) = {v : θ v ( ψ µ (v)) β}, [ µ (0, 1], β 0, ). Predictor step T v (α) = v [ 2 Φ(v)] 1 ψ µ (v) }{{} v(µ) α [ 2 Φ(v)] 1 g }{{} 1, α 0. v (µ) Goal: for v N β0 (µ) with β 0 β 1, choose the maximal α: T v (α) N β1 (µ α). Theorem: for v N β0 (µ), and α 0 satisfying β 0 + α µ (β 0 + σv ( Φ(v))) we have T v (α) N β1 (µ α). β1 1+ β 1, Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 21/28

39 Possible path-following strategies 1 Lazy scheme. β 1 = β 0. Efficiency estimate : α k 5 µ k ν 1/2 F (β 0 = 1 9 ). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 22/28

40 Possible path-following strategies 1 Lazy scheme. β 1 = β 0. Efficiency estimate : α k 5 µ k ν 1/2 F (β 0 = 1 9 ). 2 Real predictor-corrector. Note that θ T vk (α k ) ( ψ µ k α k (T vk (α k ))) [ ] β0 2 (1 β 0 + 2β ) 2 0 (1 β 0 β ) σv k ( Φ(v k )) αk µ k + O ( α 2 k µ 2 k ). Thus, it is better to keep β 0 small with respect to β 1. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 22/28

41 Computational aspects 1. Path-following scheme. The measure θ v ( ψ µ (v)) = ψ µ (v), [ 2 ˆF (v)] 1 ψ µ (v) 1/2 is easy to compute. Examples. a) LP. For positive orthant θ v (g) = [ n i=1 ] ( 1/2 g (i) v (i)) 2. b) Separable cones. Direct product of many cones of small dimension. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 23/28

42 Computational aspects 2. Potential-reduction scheme. For homogeneous quadratic potential Φ(w) = ν F ln w Q + F (w) + ν F 2 [1 ln ν F ], we have an upper bound Φ(w + h) Φ(w) ν F 2 Qw,h + Qh,h 2 Qw,w Hence, for the Newton step we need to minimize ν F 2 Qw,h + Qh,h 2 Qw,w + F (w + h) F (w). + F (w), h + 2 F (w)h, h. Note: We need to achieve a constant drop (by CG?). Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 24/28

43 Computational aspects Main competitors: Fast gradient schemes for solving the problem Note: min{ Qw, w : τ = 1, w F}. w The memory requirements are the same. One iteration of CG for potential-reduction methods has the same complexity as one iteration of GM. Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 25/28

44 n m LP Lorentz K 1 2 K 1 3 K exp Table: Performance of long-step PF-method Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 26/28

45 Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 27/28

46 Potential-reduction scheme for LP n m Iterations Average Φ Yu. Nesterov Primal-dual IPM with Asymmetric Barrier 28/28

Nonsymmetric potential-reduction methods for general cones

CORE DISCUSSION PAPER 2006/34 Nonsymmetric potential-reduction methods for general cones Yu. Nesterov March 28, 2006 Abstract In this paper we propose two new nonsymmetric primal-dual potential-reduction