The Method of Alternating Projections

Transcription

1 Matthew Tam Variational Analysis Session 56th Annual AustMS Meeting 24th 27th September 2012

2 My Year So Far... Closed Subspaces Honours student supervised by Jon Borwein. Thesis topic: alternating projections. Over the past year I ve learnt about: Classical alternating projection results. Difficulties of nonconvex alternating projections (AMSI Vacation) including development of an interactive Cinderella interface. Alternating Bregman projection in Banach Spaces. Computational experiments including nonconvex instances.

3 My Year So Far... Closed Subspaces Honours student supervised by Jon Borwein. Thesis topic: alternating projections. Over the past year I ve learnt about: Classical alternating projection results. Difficulties of nonconvex alternating projections (AMSI Vacation) including development of an interactive Cinderella interface. Alternating Bregman projection in Banach Spaces. Computational experiments including nonconvex instances. Now routinely applied to nonconvex problems (convergence not guaranteed) including hard combinatorial problems with good results. Eg. Diophantine equations, protein folding, sphere-packing, 3SAT, Sudoku, image reconstruction

4 My Year So Far... Closed Subspaces Honours student supervised by Jon Borwein. Thesis topic: alternating projections. Over the past year I ve learnt about: Classical alternating projection results. Difficulties of nonconvex alternating projections (AMSI Vacation) including development of an interactive Cinderella interface. Alternating Bregman projection in Banach Spaces. Computational experiments including nonconvex instances. Now routinely applied to nonconvex problems (convergence not guaranteed) including hard combinatorial problems with good results. Eg. Diophantine equations, protein folding, sphere-packing, 3SAT, Sudoku, image reconstruction We are designing large scale experiments to understand this better.

5 Introduction Closed Subspaces Let H be a Hilbert space. The (metric) projection of x H onto the set M is a point p M such that p x m x for all m M We write P M (x) = p when p exists uniquely.

6 Introduction Closed Subspaces Let H be a Hilbert space. The (metric) projection of x H onto the set M is a point p M such that p x m x for all m M We write P M (x) = p when p exists uniquely. Given sets M, N such that M N can we: Compute P M N (x) given x H? (Best approximation) Find a point x M N? (Feasibility)

7 Introduction Closed Subspaces Let H be a Hilbert space. The (metric) projection of x H onto the set M is a point p M such that p x m x for all m M We write P M (x) = p when p exists uniquely. Given sets M, N such that M N can we: Compute P M N (x) given x H? (Best approximation) Find a point x M N? (Feasibility) We address the question: Can these problems be solved knowing only P M and P N?

8 Two Closed Subspaces Let M, N be closed subspaces. Then: Fact P M, P N commute if and only if their composition is equal to P M N. If the projections commute then their composition gives P M N.

9 Two Closed Subspaces Let M, N be closed subspaces. Then: Fact P M, P N commute if and only if their composition is equal to P M N. If the projections commute then their composition gives P M N. Otherwise, try projecting alternatively: What happens in the limit? P x M P 0 x N P 1 M P x2 x N P 3 M P x4 x N 5...

10 Why It Works? Closed Subspaces N x 0 P M N (x 0 ) M If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

11 Why It Works? Closed Subspaces N x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

12 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

13 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

14 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

15 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

16 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

17 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

18 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

19 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

20 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

21 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

22 Why It Works? Closed Subspaces N P N P M (x 0 ) x 0 P M N (x 0 ) M P M (x 0 ) If H is the hyperplane given by a, x = b then. P H (x) = x a, x a 2 a

23 von Neumann s Alternating Projections Theorem (von Neumann, 1933) Let M, N H be closed subspaces then x H: (P M P N ) n (x) P M N (x) Proof. To show that (x n ) is Cauchy: P N (... P M P N P M P N ) = (P N P N... P N P M P N ) }{{}}{{} k terms (k+1) terms or (P N P M... P N P M P N ) }{{} (k+1) terms

24 von Neumann s Alternating Projections Theorem (von Neumann, 1933) Let M, N H be closed subspaces then x H: (P M P N ) n (x) P M N (x) Proof. To show that (x n ) is Cauchy: P N (... P M P N P M P N ) }{{} k terms = (P N... P N P M P N ) }{{} k terms or (P N P M... P N P M P N ) }{{} (k+1) terms

25 Halperin s Extension Theorem (Halperin, 1962) Let S 1, S 2,..., S r H be closed subspaces then x H: (P Sr... P S2 P S1 ) n (x) (P r i=1 S i )(x) Proof. If T linear, nonexpansive then H = ker(i T ) cl(range(i T )).

26 Halperin s Extension Theorem (Halperin, 1962) Let S 1, S 2,..., S r H be closed subspaces then x H: (P Sr... P S2 P S1 ) n (x) (P r i=1 S i )(x) Proof. If T linear, nonexpansive then H = ker(i T ) cl(range(i T )). If T linear, idempotent, nonexpansive then T = P ker(i T ).

27 Halperin s Extension Theorem (Halperin, 1962) Let S 1, S 2,..., S r H be closed subspaces then x H: T n (x) = (P Sr... P S2 P S1 ) n (x) (P r i=1 S i )(x) Proof. If T linear, nonexpansive then H = ker(i T ) cl(range(i T )). If T linear, idempotent, nonexpansive then T = P ker(i T ). T n x T n+1 x 0 hence T n (I T )x 0,

28 Halperin s Extension Theorem (Halperin, 1962) Let S 1, S 2,..., S r H be closed subspaces then x H: T n (x) = (P Sr... P S2 P S1 ) n (x) (P r i=1 S i )(x) Proof. If T linear, nonexpansive then H = ker(i T ) cl(range(i T )). If T linear, idempotent, nonexpansive then T = P ker(i T ). T n x T n+1 x 0 hence T n y 0, y range(i T )

29 Halperin s Extension Theorem (Halperin, 1962) Let S 1, S 2,..., S r H be closed subspaces then x H: T n (x) = (P Sr... P S2 P S1 ) n (x) (P r i=1 S i )(x) Proof. If T linear, nonexpansive then H = ker(i T ) cl(range(i T )). If T linear, idempotent, nonexpansive then T = P ker(i T ). T n x T n+1 x 0 hence T n y 0, y cl range(i T )

30 Best Approximation for Half-Spaces? x 0 N M Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

31 Best Approximation for Half-Spaces? x 0 N P N (x 0 ) M Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

32 Best Approximation for Half-Spaces? x 0 N P N (x 0 ) M P M P N (x 0 ) Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

33 Best Approximation for Half-Spaces? x 0 N P N (x 0 ) M P N P M P N (x 0 ) Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

34 Best Approximation for Half-Spaces? x 0 N P N (x 0 ) M P N P M P M P N (x 0 ) Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

35 Best Approximation for Half-Spaces? x 0 N P N (x 0 ) M P M N (x 0 ) P N P M P M P N (x 0 ) Beyond hyperplanes, half-spaces, spheres, balls,... projections are difficult to compute. Even for the ellipse in R 2, given by x 2 /a 2 + y 2 /b 2 = 1, we have: ( a 2 u P E (u, v) = a 2 t, b 2 ) v a 2 u 2 b 2 where t a 2 t + b2 v 2 b 2 t = 1

36 Bregman s Generalisation Theorem (Bregman, 1965) Let C 1, C 2,..., C r H be closed convex sets then x H: (P Cr... P C2 P C1 ) n (x) w. x r i=1 C i Proof. Use weak compactness to extract a weakly convergence subsequence.

37 Bregman s Generalisation Theorem (Bregman, 1965) Let C 1, C 2,..., C r H be closed convex sets then x H: (P Cr... P C2 P C1 ) n (x) w. x r i=1 C i Proof. Use weak compactness to extract a weakly convergence subsequence. Can MAP fail to converge in norm?

38 The Hundal Example (Revisited) Failure of Norm Convergence (Hundal, 2004) Let H = l 2 and {e i } an orthonormal basis. Take x 0 = e 3 and C 1 = ker(e 1 ) and C 2 = cl conv k=2 epi C 0,k then MAP fails to converge is norm. Note: C 1 is a hyperplane and C 2 a closed convex cone. h k (t) = exp( (t + kπ/2) 3 ) C 1 C 2 = {0}

39 The Hundal Example (Revisited) Failure of Norm Convergence (Hundal, 2004)(Matoušková & Reich, 2003) Let H = l 2 and {e i } an orthonormal basis. Take x 0 = e k0 and C 1 = ker(e 1 ) and C 2 = cl conv k=k 0 epi C 0,k then MAP fails to converge is norm. Note: C 1 is a hyperplane and C 2 a closed convex cone. h k (t) = exp( (t + kπ/2) 3 ) C 1 C 2 = {0}

40 The Hundal Example (Revisited) Failure of Norm Convergence (Hundal, 2004)(Matoušková & Reich, 2003) Let H = l 2 and {e i } an orthonormal basis. Take x 0 = e k0 and C 1 = ker(e 1 ) and C 2 = cl conv k=k 0 epi C 0,k then MAP fails to converge is norm. Note: C 1 is a hyperplane and C 2 a closed convex cone. h k (t) = exp( (t + kπ/2) 3 ) C 1 C 2 = {0}

41 The Hundal Example (Revisited) Failure of Norm Convergence (Hundal, 2004)(Matoušková & Reich, 2003) Let H = l 2 and {e i } an orthonormal basis. Take x 0 = e k0 and C 1 = ker(e 1 ) and C 2 = cl conv k=k 0 epi C 0,k then MAP fails to converge is norm. Note: C 1 is a hyperplane and C 2 a closed convex cone. h k (t) = exp( (t + kπ/2) 3 ) Final step: C 1 C 2 = {0} (P C2 P C1 ) Nm e k0 e m < 1/7 = (P C2 P C1 ) Nm e k0 > 6/7

42 The Hundal Example (Revisited) Can it fail to converge in norm on a real problem? Conjecture (Borwein & Bauschke, 1993) If C 1 is closed and affine with finite codimension, C 2 is the nonnegative cone in l 2 (N) then MAP is norm convergent. True when C 1 is a hyperplane (unlike Hundal). This captures most concrete applications.

43 Averaged Projections (The Crucial Product Space Trick) Given C 1, C 2,..., C r H consider H r = H H. Define: C = {(x 1, x 2,..., x r ) : x i C i }, D = {(x 1, x 2,..., x r ) : x 1 = x i }

44 Averaged Projections (The Crucial Product Space Trick) Given C 1, C 2,..., C r H consider H r = H H. Define: C = {(x 1, x 2,..., x r ) : x i C i }, D = {(x 1, x 2,..., x r ) : x 1 = x i } It is easily verified that: (P C x) i = P Ci x i, (P D x) i = 1 r r j=1 x j

45 Averaged Projections (The Crucial Product Space Trick) Given C 1, C 2,..., C r H consider H r = H H. Define: C = {(x 1, x 2,..., x r ) : x i C i }, D = {(x 1, x 2,..., x r ) : x 1 = x i } It is easily verified that: (P C x) i = P Ci x i, (P D x) i = 1 r r j=1 x j Each iteration, P D P C : H r H r, can be described by: (P D P C x) i = 1 r r P Cj x j j=1

46 Averaged Projections (The Crucial Product Space Trick) Given C 1, C 2,..., C r H consider H r = H H. Define: C = {(x 1, x 2,..., x r ) : x i C i }, D = {(x 1, x 2,..., x r ) : x 1 = x i } It is easily verified that: (P C x) i = P Ci x i, (P D x) i = 1 r r j=1 x j Each iteration, P D P C : H r H r, can be described by: (P D P C x) i = 1 r r P Cj x j=1

47 Averaged Projections (The Crucial Product Space Trick) Given C 1, C 2,..., C r H consider H r = H H. Define: C = {(x 1, x 2,..., x r ) : x i C i }, D = {(x 1, x 2,..., x r ) : x 1 = x i } It is easily verified that: (P C x) i = P Ci x i, (P D x) i = 1 r r j=1 x j Each iteration, T : H H, can be described by: T x = 1 r r P Cj x j=1

48 Douglas-Rachford and Dykstra Methods Theorem (Lions & Mercier, 1979) Let C 1, C 2 H be closed convex sets, x H iterate: x n+1 := x n + R C2 R C1 (x n ) 2 where R Ci (x) := 2P Ci (x) x then x n w. x, a fixed point, with P C1 (x) C 1 C 2. Theorem (Boyle & Dykstra, 1980) Let C 1,..., C r H be closed convex sets, x H iterate: x i n := P Ci (x i 1 n I i n 1), I i n := x i n (x i 1 n I i n 1), x 0 n := x r n 1 with initial values x 0 1 := x, I i 0 := 0 then x n (P r i=1 C i )(x).

49 Closed Subspaces Projections onto non-convex sets are no longer guaranteed to be: In R n : Nonexpansive/Firmly nonexpansive Unique (i.e. P C is set-valued). The method becomes: x 2n+1 P C1 (x 2n ), x 2n P C2 (x 2n 1 ) Local linear convergence for alternating and averaged nonconvex projections, Lewis, Luke & Malick (2009). Restricted normal cones and the method of alternating projections, Bauschke, Luke, Phan & Wang (2012). The Douglas-Rachford algorithm in the absence of convexity, Borwein & Sims (2011).

50 Sudoku: Modelling an NP-Complete Non-Convex Problem

51 Sudoku: Modelling an NP-Complete Non-Convex Problem

52 Sudoku: Modelling an NP-Complete Non-Convex Problem

53 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4.

54 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4.

55 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4.

56 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4.

57 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4.

58 Sudoku: Modelling an NP-Complete Non-Convex Problem Let A (R 9 ) 3 indexed by (i, j, k). Constraint types are: C 1 = {A ij is a standard unit vector} C 2 = {A ik is a standard unit vector} C 3 = {A jk is a standard unit vector} C 4 = {3 3 submatrix = standard unit vector} A solution is x C 1 C 2 C 3 C 4. Similar modelling can be done for: N-queens 3-SAT (NP-Complete) TetraVex (NP-Complete) Solves large instances! (Sudoku = R 2916 )

59 J. von Neumann. Functional operators Vol. II. The geometry of orthogonal spaces. Annal. of Math. Studies, 22, I. Halperin. The product of projection operators. Acta. Sci. Math., 23:96 99, L.M. Bregman. The method of successive projection for finding a common point of convex sets. Soviet Mathematics, 6: , H.S. Hundal. An alternating projection that does not converge in norm. Nonlinear analysis, 57(1):35 61, E. Matoušková & S. Reich. The Hundal example revisited. J. Nonlinear Convex Anal, 4: , P.L. Lions & B. Mercier. Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis, , J.P. Boyle & R.L. Dykstra. A method for finding projections onto the intersection of convex sets in Hilbert spaces. Advances in Order Restricted Statistical Inference, J. Schaad. Modelling the 8-queens problem and Sudoku using an algorithm based on projections onto nonconvex sets, Master s Thesis, 2010.