A semi-algebraic look at first-order methods

Transcription

1 splitting A semi-algebraic look at first-order Université de Toulouse / TSE Nesterov s 60th birthday, Les Houches, 2016

2 in large-scale first-order optimization splitting Start with a reasonable FOM (some splitting method, gradient projection method, alternating projection method, Lagrangian method...): Criticality / necessary optimality conditions (discrete Lasalle s invariance principle called Zangwill s theorem )? But what about convergence guarantees for the iterates?? Decrease rates for the iterates/value? Answer: well-designed notions of piecewise smoothness? Does not work!

3 Smooth counter-examples: Palis-De Melo, an extension Absil et al. splitting There exist f : R 2 R C coercive nonnegative with f 1 ({0}) = argmin f =unit disk a bounded gradient sequence with constant step s > 0 (as small as we want): such that but with this awful property x k+1 = x k s f (x k ) f (x k ) min f = 0, f (x k ) 0 The set of limit-points of x k is the unit circle

4 Smooth counter-examples: Palis-De Melo, an extension Absil et al. Easier to underst with ẋ(t) = f (x(t)) Idea: Spiraling bump yields spiraling curves splitting A function that yields similar : ( f (r, θ) = exp(1 r 2 r 4 ) ( ) 1 r 4 + (1 r 2 ) 4 sin θ 1 ) 1 r 2

5 Source of counter-examples: oscillations splitting Oscillations in Optimization: Oscillations = gradient direction=very bad predictors arbitrarily bad rates/complexity for any fixed dimension... Even worst behaviors for nonsmooth Same awful behaviors for more complex : e.g. Forward-Backward

6 Solutions to the oscillation issue? Alternative framework? Solutions to these bad behaviors? splitting Topological approach: avoid wild oscillations. Use semi-algebraic (or definable) Too vague!! Ad hoc approach: study deformations of a class of nonsmooth sharp function Too specific??

7 Semi-algebraic objects splitting Defined by finitely many polynomials Easy to recognize (Tarski-Seidenberg, quantifier elimination) Very stable (image/pre-image, derivation, composition, subdifferentiation...) Oscillations are controlled: monotonicity lemma/ finiteness of the number of connected components. Take A R p R n A x = {y R p : (x, y) A}. There exists N N such that cc(a x ) N, x R p

8 Concrete examples splitting Polynomials 1 2 Ax b 2, (A, B) 1 2 AB M 2 max or min of polynomials rank, l p norms (p rational or p = /infinity norm, p = 0/zero norm) stard cones: R n +, Lorenz cone, SDP... What about non semi-algebraic problems? analytic, e.g., log det l p norm with p arbitrary... a similar theory holds: van den Dries / Shiota: global subanalyticity, Dirichlet series, log-exp structure...

9 Fréchet subdifferential f : R n R {+ } lsc proper. First-order formulation: Let x in dom f (Fréchet) subdifferential : p ˆ f (x) iff splitting f (u) f (x) + p u x + o( u x ), u R n. u f (u) u f (x) + p u x

10 Subdifferential splitting Definition (Mordhukovich 76) It is denoted by f defined through: x f (x) iff (x k, x k ) (x, x ) such that f (x k ) f (x) f (u) f (x k ) + x k u x k + o( u x k ). Example: f (x) = x 1, f (0) = B = [ 1, 1] n Set f (x) = min{ x : x f (x)} Properties (Critical point) Fermat s rule if f has a minimizer at x then f (x) 0. Conversely when 0 f (x), the point x is called critical.

11 An elementary remedy to gradient oscillation : ness A function f : R n R {+ } is called sharp on the slice [r 0 < f < r 1 ] := {x R n : r 0 < f (x) < r 1 }, if there exists c > 0 f (x) c > 0, x [r 0 < f < r 1 ] Basic example f (x) = x splitting Many works since 78, Rockafellar, Polyak, Ferris, Burke many many others...

12 ness: example Nonconvex illustration with a continuum of minimizers 5 splitting

13 Finite convergence splitting Why is sharpness a remedy? Slopes towards the minimizers overcome frankly other parasite slopes Gradient curves reach the valley within a finite time. Easy to see the phenomenon on proximal descent Prox descent = formal setting for implicit gradient x + = x step. f (x + ) Prox operator: prox s f x = argmin{f (u) + 1 2s u x 2 : u R n } (Moreau) x + = prox step f (x) When f is sharp, convergence occurs in finite time!!

14 Proof splitting Assume f is sharp, then if x k+1 is non critical f (x k+1 ) f (x k ) δ. Write (assume step is constant) f (x k+1 ) f (x k ) x k+1 x k 2 x k+1 x k step. f (x k+1 ) thus x k+1 x k c.step f (x k+1 ) f (x k ) (c.step) 2 Set δ = (c.step) 2.

15 Measuring the default of sharpness splitting 0 is a critical value of f (true up to a translation). Set [0 < f < r 0 ] := {x R n : 0 < f (x) < r 0 } assume that there are no critical points in [0 < f < r 0 ]. f has the KL property on [0 < f < r 0 ] if there exists a function g : [0 < f < r 0 ] R {+ } whose sublevel sets are those of f, ie [f r] r (0,r0 ), such that g(x) 1 for all x in [0 < f < r 0 ]. EX (Concentric ellipsoids ) f (x) = 1 2 Ax, x g(x) = f

16 Formal KL property splitting Formally : Desingularizing on (0, r 0 ) : ϕ C([0, r 0 ), R + ), concave, ϕ C 1 (0, r 0 ), ϕ > 0 ϕ(0) = 0. Definition f has the KL property on [0 < f < r 0 ] if there exists a desingularizing function ϕ such that (ϕ f )(x) 1, x [0 < f < r 0 ]. Local version : replace [0 < f < r 0 ] by the intersection of [0 < f < r 0 ] with a closed ball.

17 Some in the smooth world splitting Theorem [ Lojasiewicz 1963/Kurdyka 98] If f is analytic/ o-minimal, it is KL with ϕ(s) = Ks 1 θ. Gradient dominated of Polyak 63, for f convex f (x) 2 K(f (x) min f ) KL with 1 K s. Many ideas were present... Y. Ol Khoovsky (1972) analyzed the gradient method (Absil, Mahony, Andrews 05 similar independently) In 2006 Polyak & Nesterov 06, introduced f (x) p K(f (x) min f ) for complexity purposes of second/third order. This is exactly Lojasiewicz inequality when p > 1.

18 Going beyond smoothness/analyticity? splitting Theorem (B-Daniilidis-Lewis 2006) Nonsmooth semialgebraic/subanalytic case 2006: Take f : R n R {+ } lower semicontinuous semialgebraic then f has KL property around each point. Many many satisfy KL inequality see: B-Daniilidis-Lewis-Shiota 2007 B-Daniilidis-Ley-Mazet 2010 In Optimization see also but also Attouch, B., Redont 2010, B-Sabach-Teboulle

19 Brief technical comments on KL splitting The ingredients A nonsmooth Sard s theorem: finiteness of critical values Amenability to sharpness is equivalent to the fact that there exists a talweg ( path in the valley ) of finite length a result from B., Daniilidis, Ley, Mazet. In the semi-algebraic world, desingularization are of the form ϕ(s) = cs 1 θ this is Puiseux Lemma. How all this

20 Descent at large (?P. Tseng?) splitting Let f : R n R {+ } be a proper lower semicontinuous function; a, b > 0. Let x k be a sequence in dom f such that Sufficient decrease condition f (x k+1 ) + a x k+1 x k 2 f (x k ); k 0 Relative error condition For each k N, there exists w k+1 f (x k+1 ) such that w k+1 b x k+1 x k ;

21 theorem splitting Theorem (Attouch-B-Svaiter 2012 / B. Sabach-Teboulle 13) Let f be a KL function x k a descent subgradient sequence for f. If x k is bounded then it converges to a critical point of f. Corollary Let f be a coercive semi-algebraic function x k a descent sequence for f. Then x k converges to a critical point of f.

22 Rate of convergence splitting Assume that ϕ(s) = cs 1 θ with c > 0, θ [0, 1). Theorem (i) If θ (0, 1 2 ] then there exist d > 0 Q [0, 1) such that x k x d Q k, (ii) If θ ( 1 2, 1) then there exists c 1, c 2 > 0 such that x k x c 1 k 1 θ 2θ 1, f (xk ) f (x ) c 2 k 1 2θ 1 Conv. rate θ 1 θ 2θ θ flatness degree = o ( 1 k )

23 Forward-backward splitting algorithm splitting From now on all objects are assumed to be SA Minimizing nonsmooth+smooth structure: f = g + h with { h C 1 h L Lipschitz continuous g lsc bounded from below + prox is easily computable Forward-backward splitting (Lions-Mercier 79): Let γ k be such that 0 < γ < γ k < γ < 1 L x k+1 prox γk g (x k γ k h(x k )). Theorem If the problem is coercive x k is a converging sequence

24 Gradient projection splitting C a semi-algebraic set, f a semi-algebraic function. ) x k+1 P C (x k γ k f (x k ) Theorem If the sequence x k is bounded it converges to a critical point of the problem min C f Example: Inverse problems with sparsity constraints: { } 1 min 2 Ax b 2 : x C C=sparsity constraint/rank constraint/simple constraints

25 von Neumann alternating projections? splitting Assumptions: C semi-algebraic closed D semi-algebraic convex. Example: D=Hankel matrices C =matrices of rank lower than r. x k+1 P C (y k ), y k+1 = P D (x k ). Careful oscillations are possible... A circle for C its center D = {center} A solution under-relax: Theorem Bounded sequences of the form ) x k+1 P C (ɛx k + (1 ɛ)p D (x k ) are converging. More subtle by Noll-Rondepierre in the same category based on nonsmooth KL

26 Averaged projections with underrelaxation splitting x k+1 (1 θ) x k + θ ( 1 p ) p P Fi (x k ), θ ]0, 1[ i=1 Theorem (Averaged projection method) F 1,..., F p be closed semi-algebraic which satisfy p i=1 F i. If x 0 is sufficiently close to p i=1 F i, then x k converges to a feasible point x, i.e. such that x p F i. i=1

27 Alternating versions of prox algorithm splitting F : R m 1... R mp R {+ } lsc semi-algebraic Structure of F : F (x 1,..., x p ) = p f i (x i ) + Q(x 1,..., x p ) i=1 f i are proper lsc Q is C 1. Gauss-Seidel method/ Prox (Auslender 1993) x k+1 1 argmin u R m 1 x k+1 p F (u, x k 2,..., x k p ) + 1 2µ 1 u x 1 k 2... argmin u R mp F (x k+1 1,..., x k+1 p 1, u) + 1 2µ p u x p k 2

28 Alternating versions of prox algorithm splitting F : R m 1... R mp R {+ } lsc semi-algebraic Structure of F : F (x 1,..., x p ) = p f i (x i ) + Q(x 1,..., x p ) i=1 f i are proper lsc Q is C 1. Gauss-Seidel method/ Prox (Auslender 1993) x k+1 1 argmin u R m 1 x k+1 p F (u, x k 2,..., x k p ) + 1 2µ 1 u x 1 k 2... argmin u R mp F (x k+1 1,..., x k+1 p 1, u) + 1 2µ p u x p k 2

29 Proximal Gauss-Seidel splitting F (x 1,..., x p ) = p f i (x i ) + Q(x 1,..., x p ) i=1 f i are proper lsc Q is C 1. x k+1 1 argmin u R m 1 x k+1 p Theorem f 1 (u) + Q(u,..., x p ) + 1 2µ 1 u x 1 k 2 argmin u R mp f p (u) + Q(x 1,..., u) + 1 2µ p u x p k 2 Bounded sequences of the prox Gauss-Seidel method converge

30 Proximal alternating linearized method: PALM (B. Sabach, Teboulle) F (x) = Q(x 1, x 2 ) + f 1 (x 1 ) + f 2 (x 2 ) (B., Sabach, Teboulle, 14) linearization idea: splitting x k+1 1 argmin u R m 1 x k+1 2 argmin u R m 2 f 1 (u) + Q(u, x 2 ) u x µ 1 u x 1 k 2 f 2 (u) + Q(x 1, u) u x µ 2 u x 2 k 2 ( x1 k+1 prox 1 µ k f 1 x1 k 1 ) µ k x1 Q(x1 k, x2 k ), 1 1 ( x2 k+1 prox 1 µ k f 1 x1 k 1 ) µ k x2 Q(x1 k+1, x2 k ). 2 2 Good choice of steps?

31 Proximal alternating linearized method splitting x 1, Q(x 1, ) is C 1 with L 2 (x 1 ) > 0 Lipschitz continuous gradient (same for x 2 with L 1 (x 2 )) ( x1 k+1 prox 1 L 1 (x 2 k f x k ) x k+1 2 prox 1 L 2 (x k 1 ) f 1 ) L 1 (x2 k) x 1 Q(x1 k, x2 k ) ( x k 1 1 L 2 (x k 1 ) x 2 Q(x k+1 1, x k 2 ). ). Example: Several applications in sparse NMF, blind deconvolution (Pesquet, Repetti...), dictionary learning (Gribonval, Malgouyres..). General structure: M a fixed matrix, r, s integers, min { 1 2 AB M 2 : A 0 r, B 0 s } Theorem Bounded sequences of PALM converge

32 Going further in this line? splitting Complex problems à la SQP (with Pauwels) Lagrangian (with Sabach-Teboulle) Alternating have a different nature (with Pauwels-Ngambou) but there are works by Li, Noll-Rondepierre, Druviatskii-Ioffe-Lewis... Fast???

33 A flavour of this complications: A simple SQP/SCQP method splitting We wish to solve min{f (x) : x R n, f i (x) 0} which can be written min f + i C with C = [f i 0, i]. We assume: f is L f Lipschitz i, f i is L fi Lipschitz continuous. Prox operator here are out of reach: too complex!!

34 Moving balls splitting Classical Sequential Quadratic Programming (SQP) idea, replace by some simple approximations. Moving balls method (Auslender-Shefi-Teboulle, Math. Prog., 2010) min x R n f (x k) + f (x k )(x x k )+ L f 2 x x k 2 f 1 (x k ) + f 1(x k )(x x k )+ L f 1 2 x x k f m (x k ) + f m(x k )(x x k )+ L f m 2 x x k 2 0 Bad surprise: x k is not a descent sequence for f + i C.

35 Moving balls Introduce splitting Theorem val(x) = min y R n f (x) + f (x)(y x)+ L f y x 2 2 f 1 (x) + f 1(x)(y x)+ L f 1 2 y x f m (x) + f m(x)(y x)+ L f m 2 y x 2 0 Good surprise: x k is a descent sequence for val. Assume Mangasarian-Fromovitz qualification condition. If x k is bounded it converges to a KKT point of the original problem.