Accelerated Proximal Gradient Methods for Convex Optimization

Transcription

1 Accelerated Proximal Gradient Methods for Convex Optimization Paul Tseng Mathematics, University of Washington Seattle MOPTA, University of Guelph August 18, 2008

2 ACCELERATED PROXIMAL GRADIENT METHODS 1 Talk Outline A Convex Opimization Problem

3 ACCELERATED PROXIMAL GRADIENT METHODS 1 Talk Outline A Convex Opimization Problem Proximal Gradient Method

4 ACCELERATED PROXIMAL GRADIENT METHODS 1 Talk Outline A Convex Opimization Problem Proximal Gradient Method Accelerated Proximal Gradient Method I Accelerated Proximal Gradient Method II

5 ACCELERATED PROXIMAL GRADIENT METHODS 1 Talk Outline A Convex Opimization Problem Proximal Gradient Method Accelerated Proximal Gradient Method I Accelerated Proximal Gradient Method II Example: Matrix Game

6 ACCELERATED PROXIMAL GRADIENT METHODS 1 Talk Outline A Convex Opimization Problem Proximal Gradient Method Accelerated Proximal Gradient Method I Accelerated Proximal Gradient Method II Example: Matrix Game Conclusions & Extensions

7 ACCELERATED PROXIMAL GRADIENT METHODS 2 A Convex Optimization Problem min x E f P (x) := f(x) + P (x) E is a real linear space with norm. E is the dual space of cont. linear functionals on E, with dual norm x = sup x 1 x, x. P : E (, ] is proper, convex, lsc (and simple ). f : E R is convex diff. f(x) f(y) L x y x, y domp (L 0).

8 ACCELERATED PROXIMAL GRADIENT METHODS 2 A Convex Optimization Problem min x E f P (x) := f(x) + P (x) E is a real linear space with norm. E is the dual space of cont. linear functionals on E, with dual norm x = sup x 1 x, x. P : E (, ] is proper, convex, lsc (and simple ). f : E R is convex diff. f(x) f(y) L x y x, y domp (L 0). Constrained case: P δ X with X E nonempty, closed, convex. δ X (x) = { 0 if x X else

9 ACCELERATED PROXIMAL GRADIENT METHODS 3 Examples: E = R n, P (x) = x 1, f(x) = Ax b 2 2 Basis Pursuit/Lasso E = R n 1 R n N, P (x) = w 1 x w N x N 2 (w j > 0), f(x) = g(ax) with g(y) = m i=1 ln(1 + ey i) b i y i group Lasso E = R n, P { δ X with X = {x x 0, x x n = 1}, f(x) = g (Ax) m with g(y) = i=1 y i ln y i if y 0, y y m = 1 else E = S n, { P δ X with X = {x x ij ρ i, j}, f(x) = g (x + s) with g(y) = ln dety if αi y βi (ρ, α, β > 0) else matrix game covariance selection

10 ACCELERATED PROXIMAL GRADIENT METHODS 4 How to solve this (nonsmooth) convex optimization problem? In applications, m and n are large (m, n 1000), A may be dense. 2nd-order methods (Newton, interior-point)? Few iterations, but each iteration can be too expensive (e.g., O(n 3 ) ops). 1st-order methods (gradient)? Each iteration is cheap (by using suitable prox function ), but often too many iterations. Accelerate convergence by interpolation Nesterov.

11 ACCELERATED PROXIMAL GRADIENT METHODS 5 Proximal Gradient Method Let l(x; y) := f(y) + f(y), x y + P (x) D(x, y) := h(x) h(y) h(y), x y, Bregman,... with h : E (, ] strictly convex, differentiable on X h int(domp ), and D(x, y) 1 2 x y 2 x domp, y X h.

12 ACCELERATED PROXIMAL GRADIENT METHODS 5 Proximal Gradient Method Let l(x; y) := f(y) + f(y), x y + P (x) D(x, y) := h(x) h(y) h(y), x y, Bregman,... with h : E (, ] strictly convex, differentiable on X h int(domp ), and D(x, y) 1 2 x y 2 x domp, y X h. For k = 0, 1,..., x k+1 = arg min x with x 0 domp. Assume x k X h k. {l(x; x k ) + LD(x, x k )} Special cases: steepest descent, gradient-projection Goldstein, Levitin, Polyak,..., mirror-descent Yudin, Nemirovski, iterative thresholding Daubechies et al.,...

13 ACCELERATED PROXIMAL GRADIENT METHODS 6 For the earlier examples, x k+1 has closed form when h is chosen suitably: E = R n, P (x) = x 1, h(x) = x 2 2/2. E = R n 1 R n N, P (x) = w 1 x w N x N 2 (w j > 0), h(x) = x 2 2/2. E = R n, P δ X with X = {x x 0, x x n = 1}, h(x) = n j=1 x j ln x j. E = S n, P δ X with X = {x x ij ρ i, j}, h(x) = x 2 F /2.

14 ACCELERATED PROXIMAL GRADIENT METHODS 7 Fact 1: f P (x) l(x; y) f P (x) L 2 x y 2 x, y domp. Fact 2: For any proper convex lsc ψ : E (, ] and z X h, let z + = arg min x {ψ(x) + D(x, z)}. If z + X h, then ψ(z + ) + D(z +, z) ψ(x) + D(x, z) D(x, z + ) x domp.

15 ACCELERATED PROXIMAL GRADIENT METHODS 8 Prop. 1: For any x domp, with e k := f P (x k ) f P (x). min{e 1,..., e k } LD(x, x 0), k = 1, 2,... k

16 ACCELERATED PROXIMAL GRADIENT METHODS 8 Prop. 1: For any x domp, with e k := f P (x k ) f P (x). Proof: min{e 1,..., e k } LD(x, x 0), k = 1, 2,... k f P (x k+1 ) l(x k+1 ; x k ) + L 2 x k+1 x k 2 Fact 1 l(x k+1 ; x k ) + LD(x k+1, x k ) l(x; x k ) + LD(x, x k ) LD(x, x k+1 ) Fact 2 f P (x) + LD(x, x k ) LD(x, x k+1 ), Fact 1 so 0 LD(x, x k+1 ) LD(x, x k ) e k+1 LD(x, x 0 ) (e e k+1 ) LD(x, x 0 ) (k + 1) min{e 1,..., e k+1 }

17 ACCELERATED PROXIMAL GRADIENT METHODS 9 We will improve the global convergence rate by interpolation. Idea: At iteration k, use a stepsize of O(k/L) instead of 1/L and backtrack towards x k.

18 ACCELERATED PROXIMAL GRADIENT METHODS 10 Accelerated Proximal Gradient Method I For k = 0, 1,..., y k = (1 θ k )x k + θ k z k z k+1 = arg min {l(x; y k ) + θ k LD(x, z k )} x k+1 = x (1 θ k )x k + θ k z k+1 1 θ k+1 θk θk 2 (0 < θ k+1 1) with θ 0 = 1, x 0, z 0 domp Nesterov, Auslender, Teboulle, Lan, Lu, Monteiro,... Assume z k X h k. For example, θ k = 2 k + 2 or θ k+1 = θ 4 k + 4θk 2 θ2 k. 2

19 ACCELERATED PROXIMAL GRADIENT METHODS 11 Prop. 2: For any x domp, min{e 1,..., e k } LD(x, z 0 )θ 2 k, k = 1, 2,... with e k := f P (x k ) f P (x).

20 ACCELERATED PROXIMAL GRADIENT METHODS 11 Prop. 2: For any x domp, min{e 1,..., e k } LD(x, z 0 )θ 2 k, k = 1, 2,... with e k := f P (x k ) f P (x). Proof: f P (x k+1 ) l(x k+1 ; y k ) + L 2 x k+1 y k 2 Fact 1 = l((1 θ k )x k + θ k z k+1 ; y k ) + L 2 (1 θ k)x k + θ k z k+1 y k 2 (1 θ k )l(x k ; y k ) + θ k l(z k+1 ; y k ) + L 2 θ2 k z k+1 z k 2 (1 θ k )l(x k ; y k ) + θ k (l(z k+1 ; y k ) + θ k LD(z k+1, z k )) (1 θ k )l(x k ; y k ) + θ k (l(x; y k ) + θ k LD(x, z k ) θ k LD(x, z k+1 )) Fact 2 (1 θ k )f P ( (x k ) + θ k f P (x) + θ k LD(x, z k ) θ k LD(x, z k+1 ) ) Fact 1

21 ACCELERATED PROXIMAL GRADIENT METHODS 12 so, subtracting by f P (x) and then dividing by θk 2, we have etc. 1 θ 2 k e k+1 1 θ k θk 2 e k + LD(x; z k ) LD(x; z k+1 ) Thus, global convergence rate improves from O(1/k) to O(1/k 2 ) with little extra work per iteration!

22 ACCELERATED PROXIMAL GRADIENT METHODS 13 Can also replace l(x; y k ) by a certain weighted sum of l(x; y 0 ), l(x; y 1 ),..., l(x; y k ).

23 ACCELERATED PROXIMAL GRADIENT METHODS 14 Accelerated Proximal Gradient Method II For k = 0, 1,..., y k = (1 θ k )x { k + θ k z k k l(x; y i ) z k+1 = arg min x ϑ i=0 i x k+1 = (1 θ k )x k + θ k z k+1 1 θ k+1 θ k+1 ϑ k+1 = + Lh(x) 1 θ k ϑ k (ϑ k+1 θ k+1 > 0) } with ϑ 0 θ 0 = 1, x 0 domp, and z 0 = arg min x domp Assume z k X h k. h(x) Nesterov, d Aspremont et al., Lu,... For example, ϑ k = 2 k + 1, θ k = 2 k + 2 or ϑ k+1 = θ k+1 = θ 4 k + 4θk 2 θ2 k. 2

24 ACCELERATED PROXIMAL GRADIENT METHODS 15 Prop. 3: For any x domp, min{e 1,..., e k } L(h(x) h(z 0 ))θ k 1 ϑ k 1, k = 1, 2,... with e k := f P (x k ) f P (x).

25 ACCELERATED PROXIMAL GRADIENT METHODS 15 Prop. 3: For any x domp, min{e 1,..., e k } L(h(x) h(z 0 ))θ k 1 ϑ k 1, k = 1, 2,... with e k := f P (x k ) f P (x). Proof replaces Fact 2 with: Fact 3: For any proper convex lsc ψ : E (, ], let z = arg min x {ψ(x) + h(x)}. If z X h, then ψ(z) + h(z) ψ(x) + h(x) D(x, z) x domp. Advantage?

26 ACCELERATED PROXIMAL GRADIENT METHODS 16 Example: Matrix Game min max x X v V v, Ax with X and V unit simplices in R n and R m, and A R m n. Generate A ij U[ 1, 1] with probab. p; otherwise A ij = 0. Nesterov, Nemirovski Set P { δ X and f(x) = g (Ax/µ), with µ = m g(v) = i=1 v i ln v i if v V else ɛ 2 ln m (ɛ > 0) and (L = 1 µ, = 1-norm)

27 ACCELERATED PROXIMAL GRADIENT METHODS 16 Example: Matrix Game min max x X v V v, Ax with X and V unit simplices in R n and R m, and A R m n. Generate A ij U[ 1, 1] with probab. p; otherwise A ij = 0. Nesterov, Nemirovski Set P { δ X and f(x) = g (Ax/µ), with µ = m g(v) = i=1 v i ln v i if v V else ɛ 2 ln m (ɛ > 0) and (L = 1 µ, = 1-norm) Implement PGM, APGM I & II in Matlab, with h(x) = n j=1 x j ln x j. Dynamically adjust L. Initialize x 0 = z 0 = ( 1 n,..., 1 n ). Terminate when max i (Ax k ) i min(a v k ) j ɛ with v k V a weighted sum of dual vectors associated with x 0, x 1,..., x k. j

28 ACCELERATED PROXIMAL GRADIENT METHODS 17 PGM APGM I APGM II n/m/p ɛ k/cpu (sec) k/cpu (sec) k/cpu (sec) 1000/100/ / / / / / /100/ / / /100/ / / /1000/ / / /1000/ / /695 Table 1: Performance of PGM, APGM I & II for different n, m, sparsity p, and soln accuracy ɛ.

29 ACCELERATED PROXIMAL GRADIENT METHODS 18 Conclusions & Extensions 1. Accelerated prox gradient method is promising in theory and practice. Applicable to convex-concave optimization by using smoothing Nesterov. Further extension to add cutting planes.

30 ACCELERATED PROXIMAL GRADIENT METHODS 18 Conclusions & Extensions 1. Accelerated prox gradient method is promising in theory and practice. Applicable to convex-concave optimization by using smoothing Nesterov. Further extension to add cutting planes. 2. Application to matrix completion, where E = R m n and P (x) = σ(x) 1?

31 ACCELERATED PROXIMAL GRADIENT METHODS 18 Conclusions & Extensions 1. Accelerated prox gradient method is promising in theory and practice. Applicable to convex-concave optimization by using smoothing Nesterov. Further extension to add cutting planes. 2. Application to matrix completion, where E = R m n and P (x) = σ(x) 1? 3. Extending the interpolation technique to incremental gradient methods and coordinate-wise gradient methods?

32 ACCELERATED PROXIMAL GRADIENT METHODS 18 Conclusions & Extensions 1. Accelerated prox gradient method is promising in theory and practice. Applicable to convex-concave optimization by using smoothing Nesterov. Further extension to add cutting planes. 2. Application to matrix completion, where E = R m n and P (x) = σ(x) 1? 3. Extending the interpolation technique to incremental gradient methods and coordinate-wise gradient methods? Thanks for coming!..