Coordinate Update Algorithm Short Course Operator Splitting

Transcription

1 Coordinate Update Algorithm Short Course Operator Splitting Instructor: Wotao Yin (UCLA Math) Summer / 25

2 Operator splitting pipeline 1. Formulate a problem as 0 A(x) + B(x) with monotone operators A and B. (Of course, some problems require more than two operators.) 2. Apply an operator-splitting scheme to obtain an nonexpansive operator T, ensuring: from z = T z, a solution x can be recovered computing T computing A and B separately 3. Run the iteration z k+1 T z k 4. Upon stopping, recover x k from z k 2 / 25

3 Background of splitting

4 What is splitting? Sun-Tzu: (400 BC) Caesar: divide-n-conquer ( BC) Universal principal in computational mathematics: breaks a problem into simple subproblems combine the subproblem solutions in a controlled fashion 3 / 25

5 You might have heard of... von Neumann s alternating projection Crank-Nicolson alternating implicit method (ADI) Douglas-Rachford, Peaceman-Rachford forward-backward (semi-implicit Euler, prox-gradient) alternating direction method of multipliers (ADM or ADMM) split Bregman primal-dual splitting (Zhu-Chan PDHG, Chambolle-Pock) many more 4 / 25

6 Some basic principles of splitting split x/y directions split linear from nonlinear split smooth from nonsmooth split convection from diffusion in differential equations domain decomposition split objective functions and constraints in optimization split composite operators split resolvent from (I λ(a + B)) 1 to (I λa) 1 and (I λb) 1 5 / 25

7 Why operator splitting? A simple, powerful approach to derive many algorithms Give rise to algorithms that are easy-to-code yet also scalable (nearly) state-of-the-art performance Rigorous convergence and speed analysis 6 / 25

8 Split Monotone Inclusion

9 Monotone inclusion A are monotone operators (possibly set-valued) if Ax Ay, x y 0, x, y H examples: self-adjoint positive semi-definite linear operators skew-symmetric linear operators f: subdifferential of a convex function f, possible nonsmooth monotone inclusion: find x such that 0 Ax, where A is monotone split monotone inclusion: 0 Ax + Bx 7 / 25

10 The big-three splitting schemes

11 The big three splitting schemes find an averaged operator T such that 0 A(x) + B(x) z = T A,B(z) Douglas-Rachford (Lion-Mercier 79) (maximally monotone) + (maximally monotone) forward-backward (Mercier 79) (maximally monotone) + (cocoercive) forward-backward-forward (Tseng 00) (maximally monotone) + (Lipschitz monotone) re-invented many times, though reduction not obvious and gone unnoticed T A,B is built from forward operator and backward operator (next slides) 8 / 25

12 Forward operator require: A is monotone and single-valued convergence theory requires: A is either Lipschitz or cocoercive A is cocoercive if β > 0 such that β Ax Ay 2 Ax Ay, x y, x, y H forward operator: F γa := (I γa) examples: forward Euler gradient descent: (I γ f) 9 / 25

13 Backward operator require: A is maximally monotone, possibly set-valued backward operator: (I + γa) 1 special cases: regularized matrix inversion backward Euler projection proximal map: for a closed proper convex function r y = (I + γ r) 1 (y) y = prox γr (y) 10 / 25

14 Operator splitting on diagrams (in supplementary slides)

15 Operator splitting by algebra (in supplementary slides)

16 Some applications

17 Feasibility problems C 1 and C 2 are two convex sets feasibility problem: find x C 1 C 2 monotone inclusion formulation: 0 N C1 (x) + N C2 (x) applicable: double backward: recovers von Neuman alternating projection Douglas-Rachford: reflect, reflect, average 11 / 25

18 split feasibility problem: x C 1 and Lx C 2 applications: conic self-dual embedding, computerized tomography,... equivalent optimization: minimize 1 2 Lx proj C 2 (Lx) 2 subject to x C 1 monotone inclusion formulation: 0 N C1 (x) + L T (Lx proj C2 (Lx)) applicable: forward-backward 12 / 25

19 multiple sets: x C 1 C N the product-space trick: introduce copies x (i) H of x equivalent problem in H N : find x = (x (1),..., x (N) ) such that 0 N C1 (x) + N C2 (x) where C 1 = { x : x (i) C i, i = 1,..., N } and C 2 = { x : x (1) = = x (N)} 13 / 25

20 Problems with two blocks Evolution of PDE with two spatial dimensions u = (u xx + u yy) = u Crank-Nicolson: u n+1 ij u n ij t = 1 2 (δ2 x + δ 2 y)(u n+1 ij + u n ij) involves solving equations (Caley s transform) with a large band. ADI: update x-direction and then y-direction, each involving a tridiagonal matrix They are special cases of Douglas-Rachford iterations 14 / 25

21 Linear control Continuous Lyapunov equations: find X such that AX + XA T + Q = 0 X is symmetric positive semidefinite Alternating direction implicit (ADI) method: X ( k+ 1 2 = (A + µi) 1 X k (A µi) T + Q ) ) X k+1 = ((A µi)x k Q (A + µi) T It is a special case of Douglas-Rachford iteration 15 / 25

22 Regularization least squares minimize x r(x) + 1 F (x) b 2 2 F : the differentiable operator describing the forward process b: observation r: enforces a structure on x. examples: l 2 2, l 1, sorted l 1, l 2, TV, nuclear norm, r(x) = r(lx) monotone inclusion: 0 r(x) + (J F (x)) T (F (x) b) applicable: forward-backward 16 / 25

23 Constrained optimization minimize x f(x) subject to x X monotone inclusion: 0 f(x) + N C(x) methods: if f is smooth, apply projected-gradient iteration (special case of forward-backward iteration) if f is nonsmooth directly apply Douglas-Rachford introduce a variable, then apply ADMM (a special case of Douglas-Rachford) 17 / 25

24 Monotropic program minimize x,y f(x) + h(y) subject to Ax + By = b x X, y Y. examples: minimize x r(x) Ax b 2, introduce dummy variable: y = Ax signal processing, imaging, network information processing dual formulation has the form: minimize w d 1(w) + d 2(w) monotone inclusion: 0 d 1(w) + d 2(w) applicable: Douglas-Rachford 18 / 25

25 infimal postcomposition: ( A (f + ιx) ) (p) := min{f(x) : x X, Ax = p} original problem is also equivalent to or where minimize p f(p) + h(b p) 0 f(p) h(b p) f := ( A (f + ι X) ) h := ( B (h + ι Y ) ) applicable: Douglas-Rachford all the three algorithms are in fact equivalent! 19 / 25

26 Triple-set intersection find a point x x C 1 C 2 C 3 triple-set feasibility problem: x C 1 C 2 and Lx C 3 applicable: Davis-Yin splitting (will project to each set and applies Lx and L x) applications: conic programming self-dual embedding, copositive programming 20 / 25

27 Triple-function minimization Assume h is Lipschitz differentiable problems: minimize x f(x) + g(x) + h(lx) minimize x f(x) + h(lx), subject to x C minimize x h(lx), subject to x C 1 C 2 examples: double regularization double constraints (e.g. portfolio optimization) dual SVM (quadratic objective, linear constraint, box constraints) applicable: Davis-Yin 21 / 25

28 3-block ADMM problem: minimize f 1(x 1) + f 2(x 2) + f 3(x 3) x 1, x 2, x 3 subject to L 1x 1 + L 2x 3 + L 3x 3 = b, Directly extended ADMM may fail to converge (Chen-He-Ye-Yuan 12) Dual problem minimize w d 1(w) + d 2(w) + d 3(w) If f 1 is strongly convex, then apply Davis-Yin gives: 1. x k+1 1 = arg min x1 L(x 1, x k 2, x k 3; w k ), Lagrangian 2. x k+1 2 arg min x2 L γ(x k+1 1, x 2, x k 3; w k ), augmented Lagrangian 3. x k+1 3 arg min x3 L γ(x k+1 1, x k+1 2, x 3; w k ), augmented Lagrangian 4. w k+1 = w k + γ(l 1x k L 2x k L 3x k+1 3 b) 22 / 25

29 Color texture image inpainting treat a color image as a 3-way tensor x reconstruction model by J.Liu et al: minimize x ω x (1) + ω x (2) P Ωx P Ω b 2 2 where x (i) is the ith matrix unfolding and P Ω b is the trusted pixels. Matlab results (150 iterations, 5 10 seconds) text 23 / 25

30 Summary Operator splitting is simple yet powerful Many two block applications are solved by forward-backward and Douglas-Rachford Three block applications exist and solvable by Davis-Yin 24 / 25

31 Not covered Barely mentioned: dual + splitting Completely missed: Tseng s forward-backward-forward Eckstein-Svaitor s parallel splitting Very recently, works by Q.Li N.Zhang and P.Chen J.Huang X.Zhang Convergence speed 25 / 25