Primal-Dual Splitting: Recent Improvements and Variants

Transcription

1 Primal-Dual Spliing: Recen Improvemens and Varians 1 Thomas Pock and 2 Anonin Chambolle 1 Insiue for Compuer Graphics and Vision, TU Graz, Ausria 2 CMAP & CNRS École Polyechnique, France

2 The proximal poin algorihm Consider he problem of finding a x in a Hilber space such ha T (x) 0, where T is a maximal monoone operaor. The proximal poin algorihm: [Marine 70], [Rockafellar 76] Choose a x 0 and generae he sequence {x n } for each n 0 via 0 T (x n+1 ) + λ 1 n (x n+1 x n ) 2 / 27

3 The proximal poin algorihm Consider he problem of finding a x in a Hilber space such ha T (x) 0, where T is a maximal monoone operaor. The proximal poin algorihm: [Marine 70], [Rockafellar 76] Choose a x 0 and generae he sequence {x n } for each n 0 via 0 T (x n+1 ) + λ 1 n (x n+1 x n ) The proximal poin algorihm can be wrien as [Miny 62] x n+1 = (I + λ nt ) 1 (x n ), where (I + λ nt ) 1 is he resolven operaor. 2 / 27

4 The proximal poin algorihm Consider he problem of finding a x in a Hilber space such ha T (x) 0, where T is a maximal monoone operaor. The proximal poin algorihm: [Marine 70], [Rockafellar 76] Choose a x 0 and generae he sequence {x n } for each n 0 via 0 T (x n+1 ) + λ 1 n (x n+1 x n ) The proximal poin algorihm can be wrien as [Miny 62] x n+1 = (I + λ nt ) 1 (x n ), where (I + λ nt ) 1 is he resolven operaor. Le F (x) be a proper, l.s.c. convex funcion and T (x) = F (x), hen he algorihm reads x n+1 = arg min x F (x) + 1 2λ n x x n / 27

5 The proximal poin is O(1/n) Convergence of he proximal poin algorihm in case of summable errors have been proven [Rockafellar 76] in and in case of non-summable errors recenly in [Zaslavski 11] 3 / 27

6 The proximal poin is O(1/n) Convergence of he proximal poin algorihm in case of summable errors have been proven [Rockafellar 76] in and in case of non-summable errors recenly in [Zaslavski 11] Moreover, if λ n+1 λ n, one has [Dong 12] x n+1 (I + λ n+1t ) 1 (x n+1 ) 2 2 x n (I + λ nt ) 1 (x n ) 2 2 from which follows ha x n (I + λ nt ) 1 (x n ) 2 2 O(1/n) 3 / 27

7 The proximal poin is O(1/n) Convergence of he proximal poin algorihm in case of summable errors have been proven [Rockafellar 76] in and in case of non-summable errors recenly in [Zaslavski 11] Moreover, if λ n+1 λ n, one has [Dong 12] x n+1 (I + λ n+1t ) 1 (x n+1 ) 2 2 x n (I + λ nt ) 1 (x n ) 2 2 from which follows ha x n (I + λ nt ) 1 (x n ) 2 2 O(1/n) Unforunaely, compuing he resolven is ofen as difficul as solving he problem However, in case T (x) can be wrien as sum of wo operaors each wih simple o compue resolvens, hings will change... 3 / 27

8 A class of problems Le us consider he following class of srucured convex opimizaion problems min F (Kx) + G(x), x X K : X Y is a linear and coninuous operaor from a Hilber space X o a Hilber space Y and F, G are convex, proper, l.s.c. funcions. Noe ha he subdifferenial of his problem is he sum of wo operaors T (x) = K F (Kx) + G(x) 4 / 27

9 A class of problems Le us consider he following class of srucured convex opimizaion problems min F (Kx) + G(x), x X K : X Y is a linear and coninuous operaor from a Hilber space X o a Hilber space Y and F, G are convex, proper, l.s.c. funcions. Noe ha he subdifferenial of his problem is he sum of wo operaors T (x) = K F (Kx) + G(x) Main assumpion: F, G are simple in he sense ha hey have easy o compue resolven operaors: (I + F ) 1 p ˆp 2 (ˆp) = arg min + F (p) p 2λ (I + G) 1 x ˆx 2 (ˆx) = arg min + G(x) x 2λ 4 / 27

10 A class of problems Le us consider he following class of srucured convex opimizaion problems min F (Kx) + G(x), x X K : X Y is a linear and coninuous operaor from a Hilber space X o a Hilber space Y and F, G are convex, proper, l.s.c. funcions. Noe ha he subdifferenial of his problem is he sum of wo operaors T (x) = K F (Kx) + G(x) Main assumpion: F, G are simple in he sense ha hey have easy o compue resolven operaors: (I + F ) 1 p ˆp 2 (ˆp) = arg min + F (p) p 2λ (I + G) 1 x ˆx 2 (ˆx) = arg min + G(x) x 2λ I urns ou ha many sandard problems can be cas in his framework. 4 / 27

11 Some examples The ROF model min u u 2,1 + λ 2 u f 2 2, 5 / 27

12 Some examples The ROF model min u u 2,1 + λ 2 u f 2 2, Basis pursui problem (LASSO) min x x 1 + λ 2 Ax b / 27

13 Some examples The ROF model min u u 2,1 + λ 2 u f 2 2, Basis pursui problem (LASSO) min x x 1 + λ 2 Ax b 2 2 Linear suppor vecor machine λ n min w,b 2 w max (0, 1 y i ( w, x i + b)) i=1 5 / 27

14 Some examples The ROF model min u u 2,1 + λ 2 u f 2 2, Basis pursui problem (LASSO) min x x 1 + λ 2 Ax b 2 2 Linear suppor vecor machine λ n min w,b 2 w max (0, 1 y i ( w, x i + b)) i=1 General linear programming problems min c, x, s.. x { Ax = b x 0 5 / 27

15 Primal, dual, primal-dual The real power of convex opimizaion comes hrough dualiy Recall he convex conjugae: we can ransform our iniial problem F (y) = max x, y F (x), x X min F (Kx) + G(x) (Primal) x X 6 / 27

16 Primal, dual, primal-dual The real power of convex opimizaion comes hrough dualiy Recall he convex conjugae: we can ransform our iniial problem F (y) = max x, y F (x), x X min F (Kx) + G(x) (Primal) x X min max Kx, y + G(x) F (y) x X y Y (Primal-Dual) 6 / 27

17 Primal, dual, primal-dual The real power of convex opimizaion comes hrough dualiy Recall he convex conjugae: we can ransform our iniial problem F (y) = max x, y F (x), x X min F (Kx) + G(x) (Primal) x X There is a primal-dual gap min max Kx, y + G(x) F (y) x X y Y max (F (y) + G ( K y)) y Y (Primal-Dual) (Dual) G(x, y) = F (Kx) + G(x) + (F (y) + G ( K y)) ha vanishes if and only if (x, y) is opimal 6 / 27

18 Opimaliy condiions We focus on he primal-dual formulaion: min max Kx, y + G(x) F (y) x X y Y We assume, here exiss a saddle-poin (ˆx, ŷ) X Y which saisfies he Euler-Lagrange equaions { K ˆx F (ŷ) 0 K ŷ + G(ˆx) 0 Example for a saddle-poin of a convex-concave funcion 7 / 27

19 Known algorihms Many sandard algorihms o solve he considered class of problems: Classical Arrow-Hurwicz mehod [Arrow-Hurwicz, 58] Exragradien-mehods [Korpelevich 76, Popov 80] Douglas-Rachford Spliing [Mercier-Lions 79] Alernaing direcion mehod of mulipliers Globinski, Marroco 75] [Gabay, Mercier 76] [Ecksein, Bersekas 89], [Goldsein, Osher 09] Many more algorihms for special cases: [Neserov 03], [Daubechies, Defrise, De Mol 04], [Combees, Pesque, 08], [Beck, Teboulle 09], [Rague, Fadili, Peyré, 11] 8 / 27

20 A firs-order primal-dual algorihm Proposed in a series of papers: [P., Cremers, Bischof, Chambolle, 09], [Chambolle, P., 10], [P., Chambolle, 11] Iniializaion: Choose T, Σ S ++, θ [0, 1], (x 0, y 0 ) X Y. Ieraions (n 0): Updae x n, y n as follows: { x n+1 = (I + T G) 1 (x n TK y n ) y n+1 = (I + Σ F ) 1 (y n + ΣK(x n+1 + θ(x n+1 x n ))) Alernaes gradien descend in x and gradien ascend in y Linear exrapolaion of ieraes of x in he y sep T, Σ can be seen as precondiioning marices 9 / 27

21 A firs-order primal-dual algorihm Proposed in a series of papers: [P., Cremers, Bischof, Chambolle, 09], [Chambolle, P., 10], [P., Chambolle, 11] Iniializaion: Choose T, Σ S ++, θ [0, 1], (x 0, y 0 ) X Y. Ieraions (n 0): Updae x n, y n as follows: { x n+1 = (I + T G) 1 (x n TK y n ) y n+1 = (I + Σ F ) 1 (y n + ΣK(x n+1 + θ(x n+1 x n ))) Alernaes gradien descend in x and gradien ascend in y Linear exrapolaion of ieraes of x in he y sep T, Σ can be seen as precondiioning marices Can be derived from a pre-condiioned ADMM algorihm Can be seen as a relaxed Arrow-Hurwicz scheme Can be seen as an approximae exragradien scheme 9 / 27

22 Relaions o he proximal poin algorihm Recall he proximal poin algorihm 0 T (x n+1 ) + λ 1 n (x n+1 x n ), n 0 10 / 27

23 Relaions o he proximal poin algorihm Recall he proximal poin algorihm 0 T (x n+1 ) + λ 1 n (x n+1 x n ), n 0 The ieraions of he primal-dual algorihm can be rewrien as ( ) ( ) G(x 0 n+1 ) + K y n+1 x F (y n+1 ) Kx n+1 + M n+1 x n y n+1 y n, n 0 [ T 1 M = θk K Σ 1 ] 10 / 27

24 Relaions o he proximal poin algorihm Recall he proximal poin algorihm 0 T (x n+1 ) + λ 1 n (x n+1 x n ), n 0 The ieraions of he primal-dual algorihm can be rewrien as ( ) ( ) G(x 0 n+1 ) + K y n+1 x F (y n+1 ) Kx n+1 + M n+1 x n y n+1 y n, n 0 [ T 1 M = θk K Σ 1 This is exacly he proximal poin algorihm, bu wih a norm in M. ] 10 / 27

25 Relaions o he proximal poin algorihm Recall he proximal poin algorihm 0 T (x n+1 ) + λ 1 n (x n+1 x n ), n 0 The ieraions of he primal-dual algorihm can be rewrien as ( ) ( ) G(x 0 n+1 ) + K y n+1 x F (y n+1 ) Kx n+1 + M n+1 x n y n+1 y n, n 0 [ T 1 M = θk K Σ 1 This is exacly he proximal poin algorihm, bu wih a norm in M. Convergence of he proximal poin algorihm is ensured if M is symmeric and posiive definie ] 10 / 27

26 Convergence Theorem Le θ = 1, T and Σ symmeric posiive definie maps saisfying Σ KT 2 2 < 1, hen he primal-dual algorihm converges o a saddle-poin. 11 / 27

27 Convergence Theorem Le θ = 1, T and Σ symmeric posiive definie maps saisfying Σ 1 2 KT < 1, hen he primal-dual algorihm converges o a saddle-poin. The algorihm gives differen convergence raes on differen problem classes [Chambolle, P., 10] F and G nonsmooh: O(1/n) 11 / 27

28 Convergence Theorem Le θ = 1, T and Σ symmeric posiive definie maps saisfying Σ 1 2 KT < 1, hen he primal-dual algorihm converges o a saddle-poin. The algorihm gives differen convergence raes on differen problem classes [Chambolle, P., 10] F and G nonsmooh: O(1/n) F or G uniformly convex: O(1/n 2 ) 11 / 27

29 Convergence Theorem Le θ = 1, T and Σ symmeric posiive definie maps saisfying Σ 1 2 KT < 1, hen he primal-dual algorihm converges o a saddle-poin. The algorihm gives differen convergence raes on differen problem classes [Chambolle, P., 10] F and G nonsmooh: O(1/n) F or G uniformly convex: O(1/n 2 ) F and G uniformly convex: O(ω n ), ω < 1 11 / 27

30 Convergence Theorem Le θ = 1, T and Σ symmeric posiive definie maps saisfying Σ 1 2 KT < 1, hen he primal-dual algorihm converges o a saddle-poin. The algorihm gives differen convergence raes on differen problem classes [Chambolle, P., 10] F and G nonsmooh: O(1/n) F or G uniformly convex: O(1/n 2 ) F and G uniformly convex: O(ω n ), ω < 1 Coincides wih so far bes known raes of firs-order mehods 11 / 27

31 Exensions Since he primal-dual algorihm is in principle a proximal poin algorihm, we can perform an addiional overrelaxaion [Gol shein, Tre yakov 79] x ( n+ 1 2 = (I + T G) 1 x n T K T y n) ) y n+ 2 1 = (I + Σ F ) (y 1 n + Σ K(2x n+ 1 2 x n ) (x n+1, y n+1 ) = (x n+ 2 1, y n+ 1 2 ) + γ(x n+ 1 2 x n, y n+ 2 1 y n ). where γ [0, 1[ Speeds up he convergence in many cases. 12 / 27

32 Exensions Since he primal-dual algorihm is in principle a proximal poin algorihm, we can perform an addiional overrelaxaion [Gol shein, Tre yakov 79] x ( n+ 1 2 = (I + T G) 1 x n T K T y n) ) y n+ 2 1 = (I + Σ F ) (y 1 n + Σ K(2x n+ 1 2 x n ) (x n+1, y n+1 ) = (x n+ 2 1, y n+ 1 2 ) + γ(x n+ 1 2 x n, y n+ 2 1 y n ). where γ [0, 1[ Speeds up he convergence in many cases. The algorihm has recenly been generalized by Lauren Conda in case he funcion G(x) can be wrien as G(x) = G 1(x) + G 2(x) where G 1(x) is convex, proper, l.s.c. bu G 2(x) is differeniable wih Lipschiz coninuous gradien Has advanages in case here is some smoohness in he problem. 12 / 27

33 Effec of he overrelaxaion Applicaion o he ROF model using he acceleraed O(1/n 2 ) algorihm 10 0 γ = 0 γ = No overrelaxaion: γ = 0, 1050 ieraions Overrelaxaion: γ = 0.95, 700 ieraions 13 / 27

34 α-precondiioning I is imporan o choose he precondiioner such ha he prox-operaors are sill easy o compue Resric he precondiioning marices o diagonal marices 14 / 27

35 Lemma α-precondiioning I is imporan o choose he precondiioner such ha he prox-operaors are sill easy o compue Resric he precondiioning marices o diagonal marices Le T = diag(τ 1,...τ n) and Σ = diag(σ 1,..., σ m). τ j = 1 m i=1 K i,j, σ 1 2 α i = n j=1 K i,j α hen for any α [0, 2] Σ 1 2 KT = Σ KT 2 x 2 sup 1. x X, x 0 x 2 14 / 27

36 Lemma α-precondiioning I is imporan o choose he precondiioner such ha he prox-operaors are sill easy o compue Resric he precondiioning marices o diagonal marices Le T = diag(τ 1,...τ n) and Σ = diag(σ 1,..., σ m). τ j = 1 m i=1 K i,j, σ 1 2 α i = n j=1 K i,j α hen for any α [0, 2] Σ 1 2 KT = Σ KT 2 x 2 sup 1. x X, x 0 x 2 The parameer α can be used o vary beween pure primal (α = 0) and pure dual (α = 2) precondiioning 14 / 27

37 Lemma α-precondiioning I is imporan o choose he precondiioner such ha he prox-operaors are sill easy o compue Resric he precondiioning marices o diagonal marices Le T = diag(τ 1,...τ n) and Σ = diag(σ 1,..., σ m). τ j = 1 m i=1 K i,j, σ 1 2 α i = n j=1 K i,j α hen for any α [0, 2] Σ 1 2 KT = Σ KT 2 x 2 sup 1. x X, x 0 x 2 The parameer α can be used o vary beween pure primal (α = 0) and pure dual (α = 2) precondiioning I urns ou ha for α = 0, he primal-dual algorihm is equivalen o he alernaing sep mehod [Ecksein 89] 14 / 27

38 Relaions o marix scaling The α-precondiioner ries o normalize he row- and column norms of he marix K. 15 / 27

39 Relaions o marix scaling The α-precondiioner ries o normalize he row- and column norms of he marix K. This echnique is known as marix scaling or marix binormalizaion [Livne, Golub, 04], [Bradeley 10] Given a symmeric n n marix A, find a diagonal scaling marix D = diag(d 1,..., d n) such ha D AD 2 has row and column 2-norms equal o one 15 / 27

40 Relaions o marix scaling The α-precondiioner ries o normalize he row- and column norms of he marix K. This echnique is known as marix scaling or marix binormalizaion [Livne, Golub, 04], [Bradeley 10] Given a symmeric n n marix A, find a diagonal scaling marix D = diag(d 1,..., d n) such ha D AD 2 has row and column 2-norms equal o one This is equivalen finding a posiive soluion o he equaions n d i A 2 i,jd j = 1, i = 1...n j=1 15 / 27

41 Relaions o marix scaling The α-precondiioner ries o normalize he row- and column norms of he marix K. This echnique is known as marix scaling or marix binormalizaion [Livne, Golub, 04], [Bradeley 10] Given a symmeric n n marix A, find a diagonal scaling marix D = diag(d 1,..., d n) such ha D AD 2 has row and column 2-norms equal o one This is equivalen finding a posiive soluion o he equaions n d i A 2 i,jd j = 1, i = 1...n j=1 Exensions o general marices are discussed bu mos heoreical resuls hold only in he symmeric case 15 / 27

42 Lef-righ-precondiioning For he recangular case we require ha Σ KT 2 column 2-norms as close as possible o one should have row- and 16 / 27

43 Lef-righ-precondiioning For he recangular case we require ha Σ KT 2 should have row- and column 2-norms as close as possible o one Opimized diagonal lef-righ precondiioners can be compued via ( m n 2 ( n m σ i (K i,j ) 2 τ j 1) + σ i (K i,j ) 2 τ j 1 min σ i >0,τ j >0 i=1 j=1 j=1 i=1 ) 2 Can be solved via alernaing minimizaion (slow) Finally, we have o rescale T and Σ such ha Σ 1 2 KT / 27

44 Linear programming Many applicaions of LP relaxaions in compuer vision and machine learning LP program in inequaliy form min c T x s.. Ax b, x 0, Precondiioned primal-dual algorihm {x k+1 = proj [0, ) ( x k T(A T y k + c) ) y k+1 = proj [0, ) ( y k + Σ(A(2x k+1 x k ) b) ), 17 / 27

45 A (simple) 2D example Consider he following 2D linear program ( ) c = ( 1, 1) T 20/3 1, A =, b = (20/3, 20) T primal / 27

46 No precondiioning (14030 ieraions) primal au dual sigma / 27

47 α-precondiioning (α = 0) (910 ieraions) 20 primal au dual sigma / 27

48 α-precondiioning (α = 1) (880 ieraions) 20 primal au dual sigma / 27

49 α-precondiioning (α = 2) (4350 ieraions) 20 primal au dual sigma / 27

50 Lef-righ-precondiioning (190 ieraions) 20 primal au dual sigma / 27

51 Linear programming, furher examples IP PD P-PD sc50b 0.01s 1.75s 0.49s densecolumn 0.17s s 0.61s Table: Comparison of of IP, PD and P-PD on wo sandard LP es problems. 24 / 27

52 Graph cus Graph cus are widely used in compuer vision Can be wrien as a weighed oal variaion energy [Chambolle, 05] min kdw uk`1 + hu, w u i, s.. 0 u 1, u Precondiioned primal-dual algorihm ( u k+1 = proj[0,1] u k + T(DwT y k w u ) y k+1 = proj[ 1,1] y k + Σ(Dw (2u k+1 u k )), Comparison MAXFLOW 0.160s PD 15.75s P-PD 8.56s P-PD-GPU 0.045s 25 / 27

53 Coninuous Pos model The Coninuous Pos model [Chambolle, Cremers, P. 05] wih k phases can be wrien as a convex problem min max u S q B k Du l, q l + u l, f l, l=1 where S is he simplex consrain and B is an inerecion of non-local l 2 balls. Comparison PD P-PD Speedup Synheic (3 phases) s 75.65s 2.9 Synheic (4 phases) s s 2.6 Naural (8 phases) s s / 27

54 Conclusion Precondiioned primal-dual algorihm for convex saddle poin problems wih known srucure Equivalen o he proximal poin algorihm in a paricular norm Differen choices for diagonal precondiioning Applicaions o non-smooh problems (LP, graph cus, Pos model,...) 27 / 27

55 Conclusion Precondiioned primal-dual algorihm for convex saddle poin problems wih known srucure Equivalen o he proximal poin algorihm in a paricular norm Differen choices for diagonal precondiioning Applicaions o non-smooh problems (LP, graph cus, Pos model,...) Ieraive updae of he precondiioners involving also informaion from F and G Acceleraed algorihm performs some kind of scalar precondiioning - relaions need o be undersood beer Can we improve he convergence rae facor? 27 / 27

56 Conclusion Precondiioned primal-dual algorihm for convex saddle poin problems wih known srucure Equivalen o he proximal poin algorihm in a paricular norm Differen choices for diagonal precondiioning Applicaions o non-smooh problems (LP, graph cus, Pos model,...) Ieraive updae of he precondiioners involving also informaion from F and G Acceleraed algorihm performs some kind of scalar precondiioning - relaions need o be undersood beer Can we improve he convergence rae facor? Thank you for your aenion! 27 / 27