A proximal-newton method for monotone inclusions in Hilbert spaces with complexity O(1/k 2 ).

Transcription

1 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 A proximal-newton method for monotone inclusions in Hilbert spaces with complexity O(1/k 2 ). Hedy ATTOUCH Université Montpellier 2 ACSIOM, I3M UMR CNRS 5149 joint work with M. Marques Alves, and B.F. Svaiter Convegno Italo-Francese, Atelier Italo-Français Optimisation et processus dynamiques en apprentissage et problèmes inverses Fondazione Mediaterraneo, Sestri Levante (GE), Italia Sept. 8-12, 2014

2 1A. General presentation: dynamical approach H real Hilbert space; x 2 = x, x ; A : H H maximal monotone operator. Fast methods for solving: find x H such that 0 Ax. (1) (I + λa) 1 : H H resolvent of index λ > 0 of A. Fixed point formulation of (1): x (I + λa) 1 x = 0. Dynamical system: control variable: t λ(t). ẋ(t) + x(t) (I + λ(t)a) 1 x(t) = 0. (2) H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

3 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 1B. General presentation: dynamical approach Differential-algebraic system: ẋ(t) + x(t) (I + λ(t)a) 1 x(t) = 0, (LSP) λ(t) (I + λ(t)a) 1 x(t) x(t) = θ. (3) (LSP): Large Step Proximal method. (x( ), λ( )) variables, θ positive parameter. Closed-loop control: λ is taken inversely proportional to the speed. Asymptotic equilibration (t + ): ẋ(t) 0 (I + λ(t)a) 1 x(t) x(t) 0 λ(t) +.

4 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 1C. General presentation: dynamical approach Cauchy problem, main results: ẋ(t) + x(t) (I + λ(t)a) 1 x(t) = 0, (LSP) λ(t) (I + λ(t)a) 1 x(t) x(t) = θ, x(0) = x 0 H. 1 Existence and uniqueness of (x, λ) global solution of (LSP). 2 A 1 (0) : λ(t) +, w lim t + x(t) = x A 1 (0). 3 A = f, f : H R {+ } convex, lower semicontinuous, proper f (x(t)) inf H f = O( 1 t 2 ).

5 1D. General presentation: Large step proximal method A large-step proximal method for convex optimization: (0) x 0 dom(f ), σ [0, 1[, θ > 0 given, set k = 1; (1) choose λ k > 0, and find x k, v k H, ε k 0 such that v k εk f (x k ), (4) λ k v k + x k x k λ k ε k σ 2 x k x k 1 2, (5) λ k x k x k 1 θ or v k = 0; (6) (2) if v k = 0 then STOP and output x k ; otherwise let k k + 1 and go to step 1. end. Main result: f (x k ) inf H f C k 2. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

6 Contents 1 General presentation. 2 Algebraic relationship linking λ and x. 3 Existence and uniqueness for the Cauchy problem. 4 Asymptotic behavior. 5 Link with the regularized Newton system. 6 The convex subdifferential case. 7 A large-step proximal method for convex optimization. 8 O( 1 ɛ ) proximal Newton method for convex optimization. 9 Perspective, open questions. 10 Appendix. Some examples H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

7 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 2A. Algebraic relationship linking λ and x λ (I + λa) 1 x x = θ. (7) Set ϕ : [0, [ H R +, ϕ(λ, x) := λ x (I + λa) 1 x. (8) Some classical results on resolvents, λ > 0, µ > 0, x H (1) Jλ A = (I + λa) 1 : H H nonexpansive, (2) Jλ Ax = ( µ JA µ λ x + ( 1 µ ) λ J A λ x ) ; (3) Jλ Ax JA µ x λ µ A λ x ; (4) lim λ 0 Jλ Ax = proj x; dom(a) (5) lim λ + Jλ Ax = proj A 1 (0)x, if A 1 (0).

8 2B. Algebraic relationship linking λ and x Properties of ϕ(λ, x) := λ x (I + λa) 1 x For any x 1, x 2 H and λ > 0, ϕ(λ, x 1 ) ϕ(λ, x 2 ) λ x 2 x 1. For any x H and 0 < λ 1 λ 2, λ 2 λ 1 ϕ(λ 1, x) ϕ(λ 2, x) and ϕ(λ, x) = 0 if and only if 0 A(x). ( λ2 λ 1 ) 2 ϕ(λ 1, x) (9) For any x / A 1 (0), λ [0, [ ϕ(λ, x) R + is continuous, strictly increasing, ϕ(0, x) = 0, and lim λ + ϕ(λ, x) = +. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

9 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 2C. Algebraic relationship linking λ and x λ [0, [ ϕ(λ, x) R + continuous, strict. increasing, 0 +. Definition Λ θ : H \ A 1 (0) ]0, [, Λ θ (x) := ϕ(, x) 1 (θ). (10) λ ϕ(λ, x) θ 0 Λ θ (x) λ Example: A = rot(0; π 2 )

10 3A. Existence result for the Cauchy problem ẋ(t) + x(t) (I + λ(t)a) 1 x(t) = 0, λ(t) (I + λ(t)a) 1 x(t) x(t) = θ, x(0) = x 0 H \ A 1 (0). (11) ẋ(t) + x(t) (I + Λ θ (x(t))a) 1 x(t) = 0; x(0) = x 0. ẋ(t) = F (x(t)); x(0) = x 0. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

11 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 3B. Existence result for the Cauchy problem F : Ω = H \ A 1 (0) H F (x) := J A Λ θ (x) x x. Continuity properties of F F : locally Lipschitz continuous. x H \ A 1 (0) 1 Λ θ (x): Lipschitz continuous with constant θ. Cauchy problem: ẋ(t) = F (x(t)); x(0) = x 0. Cauchy-Lipschitz theorem: local existence, and uniqueness. Global existence: estimate 0 λ( ) λ( ).

12 3C. Existence result for the Cauchy problem (LSP) ẋ(t) + x(t) (I + λ(t)a) 1 x(t) = 0, λ(t) (I + λ(t)a) 1 x(t) x(t) = θ, x(0) = x 0 H \ A 1 (0). Theorem 1 (existence and uniqueness) There exists a unique solution (x, λ) : [0, + [ H R ++ of (LSP); x( ) is C 1, and λ( ) is locally Lipschitz continuous. Moreover, (i) λ( ) is non-decreasing; 0 λ( ) λ( ); (ii) t Jλ(t) A x(t) x(t) is non-increasing. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

13 4. Asymptotic behavior Theorem 2 (convergence) Suppose that A 1 (0). Let (x, λ) : [0, + [ H R ++ be the unique solution of the Cauchy problem (LSP) with x 0 H \ A 1 (0). Then, (i) ẋ(t) = x(t) J A λ(t) x(t) d 0/ 2t; hence lim t + ẋ(t) = 0; (ii) λ(t) θ d 0 2t; hence lim t + λ(t) = + ; (iii) w lim t + x(t) = x exists, for some x A 1 (0), where d 0 is the distance from x 0 to A 1 (0). Weak convergence: Opial s lemma, Fejer monotone property. Strong convergence: A strongly monotone; A = f, f inf-compact. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

14 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 5A. Link with the regularized Newton system x 0 / A 1 (0), (x, λ) : [0, + [ H R ++ solution of (LSP). y(t) := (I + λ(t)a) 1 x(t), v(t) := 1 (x(t) y(t)). (12) λ(t) Claim: y( ) is solution of a regularized Newton system. Time derivation of λ(t)v(t) + y(t) x(t) = 0, and (LSP) gives v(t) Ay(t); ẏ(t) + λ(t) v(t) + (λ(t) + λ(t))v(t) = 0. (13)

15 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 5B. Link with the regularized Newton system Time rescaling Since 1 τ(t) = t 0 λ(u) + λ(u) λ(u) λ(u) + λ(u) du = t + ln(λ(t)/λ(0)). (14) λ(u) 2, t τ(t) 2t. Set y(t) = ỹ(τ(t)), v(t) = ṽ(τ(t)). { ṽ Aỹ; 1 λ τ 1 d dτ ỹ + d dτ ṽ + ṽ = 0. (15) Regularized Newton system [AS, SICON 2011]. Levenberg-Marquardt regularization parameter 1 λ τ 1 0 as τ +.

16 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 6A. The subdifferential case Suppose arg min f ; d 0 := inf{ x 0 z : z arg min f } = x 0 z. Theorem 3 (rate of convergence) Suppose that f (x(0)) < +. Then, (i) t f (x(t)) and t f (y(t)) are non-increasing; (ii) Set κ = θ/d0 3. For any t 0 f (x(t)) f ( z) [ f (x 0 ) f ( z) ] 1 + tκ 2 = O( 1 f (x 0 ) f ( z) t 2 ) κ f (x 0 ) f ( z) (16)

17 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 6B. The subdifferential case Differential inequality: d dt β cβ3/2, β(t) := f (x(t)) f ( z). Step 1: d f (x(t)) f (y(t)) f (x(t)). dt Integrate ẋ + x = y and apply Jensen s inequality (convexity of f ) f (x(t + h)) t+h f (y( )) non-increasing t [ e h f (x(t)) + (1 e h )f (y(u)) ] e u f (x(t + h)) e h f (x(t)) + (1 e h )f (y(t)) f (x(t + h)) f (x(t)) h 1 e h (f (y(t)) f (x(t)). h e t+h e t du.

18 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 6C. The subdifferential case Step 2: κ(f (x) f ( z)) 3/2 f (y) f (x) 1 + (3κ/2)(f (x 0 ) f ( z)) 1/2. Since v(t) f (y(t)), λ(t)v(t) = x(t) y(t), and λ(t) 2 v(t) = θ f (x(t)) f (y(t)) + x(t) y(t), v(t) f (y(t)) + λ(t) v(t) 2 Since v(t) f (y(t)), for any t 0 = f (y(t)) + θ v(t) 3/2. f (y(t)) f ( z) y(t) z, v(t) y(t) z v(t) x(t) z v(t) d 0 v(t) where we have used y(t) = J A λ(t) (x(t)), z = JA λ(t) ( z), JA λ(t) nonexpansive, and t x(t) z non-increasing. Combining the above inequalities f (x(t)) f (y(t)) + (f (y(t) f ( z)) 3/2 θ/d 3 0.

19 6D. The subdifferential case f (x(t)) f ( z) f (y(t)) f ( z) + (f (y(t) f ( z)) 3/2 θ/d 3 0. (17) Convexity of r r 3/2 : If a, b, c 0 and a b + cb 3/2 then b a ca 3/2 1 + (3c/2)a 1/2 Hence f (y) f (x) κ(f (x) f ( z)) 3/2, (18) 1 + (3κ/2)(f (x) f ( z)) 1/2 Since f (x( )) is non-increasing κ(f (x) f ( z)) 3/2 f (y) f (x). (19) 1 + (3κ/2)(f (x 0 ) f ( z)) 1/2 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

20 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 6E. The subdifferential case: y( ) versus x( ) y(t) = (I + λ(t) f ) 1 x(t) = J f λ(t) x(t) = prox λ(t)f x(t). (1) y( ): solution of a regularized Newton system. (2) x(t) J f λ(t) x(t) = ẋ(t) d 0/ 2t 0. w lim y(t) = w t + lim t + (3) f (x(t)) f λ(t) (x(t)) f (Jλ(t) f x(t)). Hence Hence x(t) arg min f. 0 f (y(t)) inf f f (x(t)) inf f = O( 1 H H t 2 ). (4) y( ) dom f : more (space) regularity than x(t). (5) f (y(t)) inf H f even if arg min f =.

21 7A. A large-step prox. method for convex optimization Algorithm 1: (0) x 0 dom(f ), σ [0, 1[, θ > 0 given, set k = 1; (1) choose λ k > 0, and find x k, v k H, ε k 0 such that v k εk f (x k ), (20) λ k v k + x k x k λ k ε k σ 2 x k x k 1 2, (21) λ k x k x k 1 θ or v k = 0; (22) (2) if v k = 0 STOP, output x k ; otherwise k k + 1, go to step 1. end Relative error for HPE, Solodov and Svaiter, SVVA, JCA (1999). Large-step condition, Montero and Svaiter, SIOPT (2010, 2012). H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

22 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 7B. A large-step prox. method for convex optimization Theorem 4 (complexity, ε k = 0) D 0 = sup{ x y max{f (x), f (y)} f (x 0 )} < +, κ 0 := θ(1 σ) f (x 0 ) f ( x) (i) f (x k ) f ( x) [ ] 2 = O(1/k 2 ). κ 0 f (x0 ) f ( x) 1 + k 2 + 3κ 0 f (x0 ) f ( x) (ii) for each k 2 even, there exists j {k/2 + 1,..., k} such that 4 v j 3 θ(1 σ) k 2+k f (x 0 ) f ( x) κ 0 f (x0 ) f ( x) 2 + 3κ 0 f (x0 ) f ( x) 2 2/3 D 3 0. = O(1/k 2 ).

23 8A. O( 1 ɛ ) prox. Newton method for convex optim. (P) minimize f (x) s.t. x H. AS1) f : H R convex, twice continuously differentiable; AS2) 2 f (x) 2 f (y) L x y x, y H; AS3) D 0 = sup{ y x max{f (x), f (y)} f (x 0 )} <. Algorithm 2 (0) x 0 H, 0 < σ l < σ u < 1 b given, set k = 1; (1) if f (x k 1 ) = 0 then stop. Otherwise, compute λ k > 0 s.t. 2σ l L λ k (I + λ k 2 f (x k 1 )) 1 λ k f (x k 1 ) 2σ u L ; (23) (2) set x k = x k 1 (I + λ k 2 f (x k 1 )) 1 λ k f (x k 1 ); (3) set k k + 1 and go to step 1. end. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

24 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 8B. O( 1 ɛ ) prox. Newton method for convex optim. Proximal method y = (I + λ f ) 1 x λ f (y) + y x = 0. Perform a single Newton iteration from the current iterate x λ f (x) + (I + λ 2 f (x))(y x) = 0. i.e., y = x (I + λ 2 f (x)) 1 λ f (x). Hence the Proximal-Newton method x k = x k 1 (I + λ k 2 f (x k 1 )) 1 λ k f (x k 1 ).

25 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 8C. O( 1 ɛ ) prox. Newton method for convex optim. Theorem 5 (complexity) Suppose AS1), AS2), AS3) hold, {x k } generated by Algorithm 2. Let x be a solution of (P). For any given tolerance ε > 0 define κ 0 = 2σ l (1 σ u ) LD0 3, K = 2 + 3κ 0 f (x0 ) f ( x), κ 0 ε ) 2/3 2L (2 1/6 + 3κ 0 f (x0 ) f ( x) J = [2σ l (1 σ u )] 1/6 κ 1/3. 0 ε Then, the following statements hold: (a) for any k K, f (x k ) f ( x) ε; (b) there exists j 2 J such that f (x j ) ε.

26 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 8D. O( 1 ɛ ) prox. Newton method for convex optim. Solving w.r. λ > 0 2σ l L λ (I + λ 2 f (x k 1 )) 1 λ f (x k 1 ) 2σ u L ; The solution set is a closed interval [λ l, λ u ] s.t. λu λ l Binary search in ln λ may be used to find λ k. σu σ l. Φ(λ) := λ (I + λ 2 f (x)) 1 λ f (x), Ψ(µ) := Φ( 1 λ ). 2σ l L Ψ(µ) 2σ u L ; µ Ψ 2 (µ), and µ ln(ψ(µ)) are convex.

27 9A. Perspective, large step proximal methods Main result (LSP) is a large step proximal method with O( 1 k 2 ) convergence. Related results Compare with Nesterov steepest descent. Rockafellar, SICON 1976: A is α-strongly monotone, A 1 (0) = x, x k+1 x αλ k x k x. Superlinear convergence when λ k +. Att-Redont-Svaiter, JOTA 2013, regularized Newton method, Levenberg-Marquardt parameter µ k = 1 λ k = f (x k ) 1 3, yield convergence of (x k ) at the order 4 3. Other algebraic relation: λ (I + λa) 1 x x γ = θ. (LSP): γ = 1. H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

28 9B. Perspective, open questions Possible extensions Extension of f (x(t)) inf f = O( 1 t 2 ) and f (x k ) inf f = O( 1 k 2 ) to the non potential case: convex-concave saddle value problems? general maximal monotone operators (via Fitzpatrick function)? Combine (LSP) with splitting methods: forward-backward method, alternating proximal minimization? Implementation of the method and applications. Fast converging methods and second-order methods: in time (Nesterov, FISTA...), or space (Newton-like methods: LSP). Compare, combine them? (LSP) is linked to a regularized Newton method which still works in the nonconvex nonsmooth setting. Convergence in KL setting? H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

29 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Isotropic linear monotone operator α > 0 positive constant, A = αi, i.e., for every x H, Ax = αx. (λa + I ) 1 1 x = 1 + λα x (24) x (λa + I ) 1 x = λα x. 1 + λα (25) Given x 0 0, (LSP) can be written ẋ(t) + αλ(t) 1+αλ(t) x(t) = 0, λ(t) > 0, αλ(t) 2 1+αλ(t) x(t) = θ, x(0) = x 0 H \ A 1 (0). (26) Let us first integrate the above linear differential equation.

30 Appendix: Isotropic linear monotone operator We have x(t) = e (t) x 0 with (t) := t 0 αλ(τ) 1+αλ(τ) dτ. Hence αλ(t) αλ(t) e (t) = θ x 0. (27) First, check this equation at time t = 0. Equivalently αλ(0) αλ(0) = θ x 0. (28) This equation defines uniquely λ(0) > 0, because ξ αξ2 1+αξ is strictly increasing from [0, + [ onto [0, + [. Thus, the only thing we have to prove is the existence of a positive function t λ(t) such that h(t) := αλ(t)2 1 + αλ(t) e (t) is constant on [0, + [. (29) Writing that the derivative h is identically zero on [0, + [, we obtain H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40

31 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Isotropic linear monotone operator After integration, we obtain λ (t)(αλ(t) + 2) αλ(t) 2 = 0. (30) α ln λ(t) 2 2 = αt + α ln λ(0) λ(t) λ(0). (31) Let us introduce the function g : ]0, + [ R g(ξ) = α ln ξ 2 ξ. (32) As t increases from 0 to +, g(t) is strictly increasing from to +. Thus, for each t > 0, (31) has a unique solution λ(t) > 0. Moreover, t λ(t) is increasing, continuously differentiable, and lim t λ(t) = +. Returning to (31), we obtain that λ(t) e t.

32 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Antisymmetric linear monotone operator H = R 2, A = rot(0, π 2, A = A, A(ξ, η) = ( η, ξ). (λa + I ) 1 x = 1 ( ) 1 + λ 2 ξ + λη, η λξ (33) x (λa + I ) 1 x = λ ( ) 1 + λ 2 λξ η, λη + ξ. (34) ξ(t) + η(t) + λ(t) λ(t) 2 λ(t) ( ) 1 + λ(t) 2 λ(t)ξ(t) η(t) = 0, λ(t) > 0, (35) ) λ(t) ( 1 + λ(t) 2 λ(t)η(t) + ξ(t) = 0, λ(t) > 0, (36) ξ(t) 2 + η(t) 2 = θ, (37)

33 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Antisymmetric linear monotone operator Set u(t) = ξ(t) 2 + η(t) 2. After multiplying (35) by ξ(t), and multiplying (36) by η(t), then adding the results, we obtain Set We have Equation (37) becomes u (t) + 2λ(t)2 u(t) = λ(t) (t) := t 0 2λ(τ) 2 dτ. (38) λ(τ) u(t) = e (t) u(0). (39) λ(t) 2 e 1 (t) 2 = θ + λ(t) 2 x 0. (40)

34 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Antisymmetric linear monotone operator First, check this equation at time t = 0. Equivalently λ(0) 2 = θ 1 + λ(0) 2 x 0. (41) This equation defines uniquely λ(0) > 0, because the function ρ ρ2 is strictly increasing from [0, + [ onto [0, + [. Thus, we 1+ρ 2 just need to prove the existence of a positive function t λ(t) s.t. h(t) := λ(t) 2 e 1 (t) 2 is constant on [0, + [. (42) + λ(t) 2 Writing that the derivative h is identically zero on [0, + [, we obtain that λ( ) must satisfy λ (t)(2λ(t) + λ(t) 3 ) λ(t) 3 = 0. (43)

35 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 Appendix: Antisymmetric linear monotone operator After integration of this first-order differential equation, with Cauchy data λ(0), we obtain λ(t) 2 λ(t) = t + λ(0) 2 λ(0). (44) Let us introduce the function g : ]0, + [ R g(ρ) = ρ 2 ρ. (45) As t increases from 0 to +, g(t) is strictly increasing from to +. Thus, for each t > 0, (44) has a unique solution λ(t) > 0. Moreover, the mapping t λ(t) is increasing, continuously differentiable, and lim t λ(t) = +. Returning to (44), we obtain that λ(t) t as t +.

36 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 References B. Abbas, H. Attouch, and B. F. Svaiter, Newton-like dynamics and forward-backward methods for structured monotone inclusions in Hilbert spaces, JOTA, DOI /s , (2013). H. Attouch, Viscosity solutions of minimization problems, SIAM J. Optim., 6 (1996), No. 3, pp H. Attouch, J. Bolte, and B. F. Svaiter, Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods, Math. Program., 137 (2013), No. 1, pp H. Attouch, P. Redont, and B. F. Svaiter, Global convergence of a closed-loop regularized Newton method for solving monotone inclusions in Hilbert spaces, JOTA, 157 (2013), pp

37 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 References H. Attouch and B. F. Svaiter, A continuous dynamical Newton-like approach to solving monotone inclusions, SIAM J. Control Optim., 49 (2011), pp H. Bauschke and P. Combettes, Convex analysis and monotone operator theory, CMS books in Mathematics, Springer, H. Brézis, Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert, North-Holland/Elsevier, New-York, R.E. Bruck, Asymptotic convergence of nonlinear contraction semigroups in Hilbert spaces, J. Funct. Anal., 18 (1975), pp

38 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 References C. Gonzaga, E. W. Karas, Fine tuning Nesterov s steepest descent algorithm for differentiable convex programming, Math. Program., 138 (2013), pp A. Griewank, The modification of Newton s method for unconstrained optimization by bounding cubic terms, Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge, B. Martinet, Régularisation d inéquations variationnelles par approximations successives, Rev. Française Informat. Recherche Opérationnelle, 4 (1970), (Ser. R-3), pp R. D. C. Monteiro and B. F. Svaiter, On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean, SIAM J. Optim., 20 (2010), No. 6, pp

39 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 References R. D. C. Monteiro and B. F. Svaiter, Iteration-complexity of a Newton proximal extragradient method for monotone variational inequalities and inclusion problems, SIAM J. Optim., 22 (2012), No. 3, pp Y. Nesterov, Introductory lectures on convex optimization, A basic course, Kluwer, Boston (2004). Y. Nesterov and B. T. Polyak, Cubic regularization of Newton method and its global performance, Math. Program., 108 (2006), (1, Ser. A), pp Z. Opial, Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Amer. Math. Soc. 73 (1967), pp

40 H. ATTOUCH (Univ. Montpellier 2) Fast proximal-newton method Sept. 8-12, / 40 References R. T. Rockafellar, Augmented Lagrangians and applications of the proximal point algorithm in convex programming, Math. Oper. Res., 1 (1976), No. 2, pp R. T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM J. Control Optim., 14 (1976), No. 5, pp M. V. Solodov and B. F. Svaiter, A hybrid approximate extragradient-proximal point algorithm using the enlargement of a maximal monotone operator. Set-Valued Anal., 7 (1999), No. 4, pp M. V. Solodov and B. F. Svaiter, A hybrid projection-proximal point algorithm. J. Convex Anal., 6 (1999), No. 1, pp M. Weiser, P. Deuflhard, and B. Erdmann, Affine conjugate adaptive Newton methods for nonlinear elastomechanics, Optim. Methods Softw., 22 (2007), No. 3, pp