Randomized projection algorithms for overdetermined linear systems

Transcription

1 Randomized projection algorithms for overdetermined linear systems Deanna Needell Claremont McKenna College ISMP, Berlin 2012

2 Setup Setup Let Ax = b be an overdetermined, standardized, full rank system of equations.

3 Setup Setup Let Ax = b be an overdetermined, standardized, full rank system of equations. Goal From A and b we wish to recover unknown x. Assume m n.

4 Method Kaczmarz The Kaczmarz method is an iterative method used to solve Ax = b. Due to its speed and simplicity, it s used in a variety of applications.

5 Method Kaczmarz The Kaczmarz method is an iterative method used to solve Ax = b. Due to its speed and simplicity, it s used in a variety of applications.

6 Method Kaczmarz The Kaczmarz method is an iterative method used to solve Ax = b. Due to its speed and simplicity, it s used in a variety of applications.

7 Method Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i = (k mod m)+1 3 Repeat (2)

8 Method Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i = (k mod m)+1 3 Repeat (2)

9 Method Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i = (k mod m)+1 3 Repeat (2)

10 Method Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i = (k mod m)+1 3 Repeat (2)

11 Geometrically Denote H i = {w : a i,w = b[i]}.

12 Geometrically Denote H i = {w : a i,w = b[i]}.

13 Geometrically Denote H i = {w : a i,w = b[i]}.

14 Geometrically Denote H i = {w : a i,w = b[i]}.

15 But what if... Denote H i = {w : a i,w = b[i]}.

16 But what if... Denote H i = {w : a i,w = b[i]}.

17 But what if... Denote H i = {w : a i,w = b[i]}.

18 But what if... Denote H i = {w : a i,w = b[i]}.

19 But what if... Denote H i = {w : a i,w = b[i]}.

20 But what if... Denote H i = {w : a i,w = b[i]}.

21 But what if... Denote H i = {w : a i,w = b[i]}.

22 But what if... Denote H i = {w : a i,w = b[i]}.

23 But what if... Denote H i = {w : a i,w = b[i]}.

24 Randomized Kaczmarz Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i is chosen randomly 3 Repeat (2)

25 Randomized Kaczmarz Kaczmarz 1 Start with initial guess x 0 2 x k+1 = x k +(b[i] a i,x k )a i where i is chosen randomly 3 Repeat (2)

26 Randomized Kaczmarz Theorem [Strohmer-Vershynin]: Consistent case Ax = b 1 Start with initial guess x 0 2 x k+1 = x k +(b p a p,x k )a p where P(p = i) = a i 2 2 A 2 F 3 Repeat (2) = 1/m

27 Randomized Kaczmarz Theorem [Strohmer-Vershynin]: Consistent case Ax = b 1 Start with initial guess x 0 2 x k+1 = x k +(b p a p,x k )a p where P(p = i) = a i 2 2 A 2 F 3 Repeat (2) = 1/m

28 Randomized Kaczmarz (RK) Theorem [Strohmer-Vershynin] Let R = m A 1 2 ( A 1 def = inf{m : M Ax 2 x 2 for all x}) Then E x k x 2 2 (1 1 R) k x0 x 2 2 Well conditioned A Convergence in O(n) iterations O(n 2 ) total runtime. Better than O(mn 2 ) runtime for Gaussian elimination and empirically often faster than Conjugate Gradient.

29 Randomized Kaczmarz (RK) Theorem [Strohmer-Vershynin] Let R = m A 1 2 ( A 1 def = inf{m : M Ax 2 x 2 for all x}) Then E x k x 2 2 (1 1 R) k x0 x 2 2 Well conditioned A Convergence in O(n) iterations O(n 2 ) total runtime. Better than O(mn 2 ) runtime for Gaussian elimination and empirically often faster than Conjugate Gradient.

30 Randomized Kaczmarz (RK) Theorem [Strohmer-Vershynin] Let R = m A 1 2 ( A 1 def = inf{m : M Ax 2 x 2 for all x}) Then E x k x 2 2 (1 1 R) k x0 x 2 2 Well conditioned A Convergence in O(n) iterations O(n 2 ) total runtime. Better than O(mn 2 ) runtime for Gaussian elimination and empirically often faster than Conjugate Gradient.

31 Randomized Kaczmarz (RK) Theorem [Strohmer-Vershynin] Let R = m A 1 2 ( A 1 def = inf{m : M Ax 2 x 2 for all x}) Then E x k x 2 2 (1 1 R) k x0 x 2 2 Well conditioned A Convergence in O(n) iterations O(n 2 ) total runtime. Better than O(mn 2 ) runtime for Gaussian elimination and empirically often faster than Conjugate Gradient.

32 Randomized Kaczmarz (RK) with noise System with noise We now consider the system Ax = b +e.

33 Randomized Kaczmarz (RK) with noise Theorem [N] Let Ax = b +e. Then. E x k x 2 ( 1 1 R) k/2 x0 x 2 + R e This bound is sharp and attained in simple examples.

34 Randomized Kaczmarz (RK) with noise Theorem [N] Let Ax = b +e. Then. E x k x 2 ( 1 1 R) k/2 x0 x 2 + R e This bound is sharp and attained in simple examples.

35 Randomized Kaczmarz (RK) with noise Error in estimation: Gaussian 2000 by 100 after 800 iterations Error 0.05 Threshold Error in estimation: Partial Fourier 700 by 101 after 1000 iterations. Error Threshold Error Trials Error Trials Figure: Comparison between actual error (blue) and predicted threshold (pink). Scatter plot shows exponential convergence over several trials.

36 Even better convergence? Recall x k+1 = x k +(b[i] a i,x k )a i Since these projections are orthogonal, the optimal projection is one that maximizes x k+1 x k 2. What if we relax: x k+1 = x k +γ(b[i] a i,x k )a i Can we choose γ optimally? Idea: In each iteration, project once with relaxation optimally and then project normally.

37 Even better convergence? Recall x k+1 = x k +(b[i] a i,x k )a i Since these projections are orthogonal, the optimal projection is one that maximizes x k+1 x k 2. What if we relax: x k+1 = x k +γ(b[i] a i,x k )a i Can we choose γ optimally? Idea: In each iteration, project once with relaxation optimally and then project normally.

38 Even better convergence? Recall x k+1 = x k +(b[i] a i,x k )a i Since these projections are orthogonal, the optimal projection is one that maximizes x k+1 x k 2. What if we relax: x k+1 = x k +γ(b[i] a i,x k )a i Can we choose γ optimally? Idea: In each iteration, project once with relaxation optimally and then project normally.

39 Even better convergence? Recall x k+1 = x k +(b[i] a i,x k )a i Since these projections are orthogonal, the optimal projection is one that maximizes x k+1 x k 2. What if we relax: x k+1 = x k +γ(b[i] a i,x k )a i Can we choose γ optimally? Idea: In each iteration, project once with relaxation optimally and then project normally.

40 Even better convergence? Recall x k+1 = x k +(b[i] a i,x k )a i Since these projections are orthogonal, the optimal projection is one that maximizes x k+1 x k 2. What if we relax: x k+1 = x k +γ(b[i] a i,x k )a i Can we choose γ optimally? Idea: In each iteration, project once with relaxation optimally and then project normally.

41 Even better convergence? Two-subspace Kaczmarz Randomly select two rows, a s and a r Perform initial projection: y = x k +γ(b[i] a i,x k )a i with γ optimal Peform second projection: x k+1 = y +(b[i] a i,y )a i Repeat

42 Even better convergence? Two-subspace Kaczmarz Randomly select two rows, a s and a r Perform initial projection: y = x k +γ(b[i] a i,x k )a i with γ optimal Peform second projection: x k+1 = y +(b[i] a i,y )a i Repeat

43 Even better convergence? Two-subspace Kaczmarz Randomly select two rows, a s and a r Perform initial projection: y = x k +γ(b[i] a i,x k )a i with γ optimal Peform second projection: x k+1 = y +(b[i] a i,y )a i Repeat

44 Even better convergence? Two-subspace Kaczmarz Randomly select two rows, a s and a r Perform initial projection: y = x k +γ(b[i] a i,x k )a i with γ optimal Peform second projection: x k+1 = y +(b[i] a i,y )a i Repeat

45 Two-subspace Kaczmarz Geometrically, we choose γ in such a way:

46 Two-subspace Kaczmarz The optimal choice of γ in a single iteration is γ = a r a s,a r a s,x k x +(b s x k,a s )a s (b r x k,a r ) a r a s,a r a s 2. 2 Two-Subspace Kaczmarz method Select two distinct rows of A uniformly at random µ k a r,a s y k x k 1 +(b s x k 1,a s )a s v k ar µ ka s 1 µk 2 β k br bsµ k 1 µk 2 x k y k +(β k y k,v k )v k

47 Two-subspace Kaczmarz The optimal choice of γ in a single iteration is γ = a r a s,a r a s,x k x +(b s x k,a s )a s (b r x k,a r ) a r a s,a r a s 2. 2 Two-Subspace Kaczmarz method Select two distinct rows of A uniformly at random µ k a r,a s y k x k 1 +(b s x k 1,a s )a s v k ar µ ka s 1 µk 2 β k br bsµ k 1 µk 2 x k y k +(β k y k,v k )v k

48 Two-Subspace Kaczmarz (a) (b) Figure: For coherent systems, the one-subspace randomized Kaczmarz algorithm (a) converges more slowly than the two-subspace Kaczmarz algorithm (b).

49 Two-Subspace Kaczmarz Define the coherence parameters: = (A) = max j k a j,a k and δ = δ(a) = min j k a j,a k. (1) Error RK 2SRK Iterations Figure: Randomized Kaczmarz (RK) versus two-subspace RK (2SRK). A has highly coherent rows with δ = and =

50 Two-Subspace Kaczmarz RK 2SRK RK 2SRK Error Error Iterations Iterations RK 2SRK RK 2SRK Error Error Iterations Iterations Figure: Randomized Kaczmarz (RK) versus two-subspace RK (2SRK). A has highly coherent rows with coherence parameters (a) δ = and = 0.967, (b) δ = and = 0.904, (c) δ = and = 0.819, and (d) δ = 0 and =

51 Results Recall the coherence parameters: = (A) = max j k a j,a k and δ = δ(a) = min j k a j,a k. (2) Theorem [N-Ward] Let b = Ax +e, then the two-subspace Kaczmarz method yields E x x k 2 η k/2 x x η e, 1 2 { } where D = min δ 2 (1 δ) 1+δ, 2 (1 ) 1+, R = m A 1 2 denotes the scaled condition number, and η = ( 1 R) 1 2 D R.

52 Results Recall the coherence parameters: = (A) = max j k a j,a k and δ = δ(a) = min j k a j,a k. (2) Theorem [N-Ward] Let b = Ax +e, then the two-subspace Kaczmarz method yields E x x k 2 η k/2 x x η e, 1 2 { } where D = min δ 2 (1 δ) 1+δ, 2 (1 ) 1+, R = m A 1 2 denotes the scaled condition number, and η = ( 1 R) 1 2 D R.

53 Results Remarks 1. When = 1 or δ = 0 we recover the same convergence rate as provided for the standard Kaczmarz method since the two-subspace method utilizes two projections per iteration. 2. The bound presented in the theorem is a pessimistic bound. Even when = 1 or δ = 0, the two-subspace method improves on the standard method if any rows of A are highly correlated (but not equal).

54 Results Remarks 1. When = 1 or δ = 0 we recover the same convergence rate as provided for the standard Kaczmarz method since the two-subspace method utilizes two projections per iteration. 2. The bound presented in the theorem is a pessimistic bound. Even when = 1 or δ = 0, the two-subspace method improves on the standard method if any rows of A are highly correlated (but not equal).

55 The parameter D Figure: A plot of the improved convergence factor D as a function of the coherence parameters δ and δ.

56 Generalization to more than two rows?

57 Block Kaczmarz [N-Tropp] Randomized Block Kaczmarz method Given a partition of the rows, T: Select a block τ of the partition at random x k x k 1 +A τ (b τ A τ x k 1 ) The convergence rate heavily depends on the conditioning of the blocks A τ need to control geometric properties of the partition.

58 Block Kaczmarz [N-Tropp] Randomized Block Kaczmarz method Given a partition of the rows, T: Select a block τ of the partition at random x k x k 1 +A τ (b τ A τ x k 1 ) The convergence rate heavily depends on the conditioning of the blocks A τ need to control geometric properties of the partition.

59 Block Kaczmarz [N-Tropp] Randomized Block Kaczmarz method Given a partition of the rows, T: Select a block τ of the partition at random x k x k 1 +A τ (b τ A τ x k 1 ) The convergence rate heavily depends on the conditioning of the blocks A τ need to control geometric properties of the partition.

60 Block Kaczmarz [N-Tropp] Randomized Block Kaczmarz method Given a partition of the rows, T: Select a block τ of the partition at random x k x k 1 +A τ (b τ A τ x k 1 ) The convergence rate heavily depends on the conditioning of the blocks A τ need to control geometric properties of the partition.

61 Block Kaczmarz Row paving An (d,α,β) row paving of a matrix A is a partition T = {τ 1,...,τ d } of the row indices that verifies α λ min (A τ A τ) and λ max (A τ A τ) β for each τ T. Theorem [N-Tropp] Suppose A admits an (d,α,β) row paving T and that b = Ax +e. The convergence of the block Kaczmarz method satisfies E x k x 2 2 [ 1 σ2 min (A) ] k x 0 x β βd α e 2 2 (3) (A). σ 2 min

62 Block Kaczmarz Row paving An (d,α,β) row paving of a matrix A is a partition T = {τ 1,...,τ d } of the row indices that verifies α λ min (A τ A τ) and λ max (A τ A τ) β for each τ T. Theorem [N-Tropp] Suppose A admits an (d,α,β) row paving T and that b = Ax +e. The convergence of the block Kaczmarz method satisfies E x k x 2 2 [ 1 σ2 min (A) ] k x 0 x β βd α e 2 2 (3) (A). σ 2 min

63 Block Kaczmarz Good row pavings [Bougain-Tzafriri, Tropp] For any δ (0,1), A admits a row paving with d C δ 2 A 2 log(1+n) and 1 δ α β 1+δ. Theorem [N-Tropp] Let A have row paving above with δ = 1/2. The block Kaczmarz method yields E x k x 2 2 [ ] 1 k 1 Cκ 2 x 0 x e 2 2 (A)log(1+n) σmin 2 (A).

64 Block Kaczmarz Good row pavings [Bougain-Tzafriri, Tropp] For any δ (0,1), A admits a row paving with d C δ 2 A 2 log(1+n) and 1 δ α β 1+δ. Theorem [N-Tropp] Let A have row paving above with δ = 1/2. The block Kaczmarz method yields E x k x 2 2 [ ] 1 k 1 Cκ 2 x 0 x e 2 2 (A)log(1+n) σmin 2 (A).

65 Block Kaczmarz Theorem [Bougain-Tzafriri, Tropp] A random partition of the row indices with m A 2 blocks is a row paving with upper bound β 6log(1+n), with probability at least 1 n 1. Theorem [Bourgain-Tzafriri, Tropp] Suppose that A is incoherent. A random partition of the row indices into m blocks where m C δ 2 A 2 log(1+n) is a row paving of A whose paving bounds satisfy 1 δ α β 1+δ, with probability at least 1 n 1.

66 Block Kaczmarz Figure: The matrix A is a fixed matrix consisting of 15 partial circulant blocks. Error x k x 2 per flop count.

67 Block Kaczmarz Figure: The matrix A is a fixed matrix with rows drawn randomly from the unit sphere, with d = 10 blocks. Error x k x 2 over various computational resources.

68 For more information Web: References: Strohmer, Vershynin, A randomized Kaczmarz algorithm with exponential convergence, J. Four. Ana. and App Tropp, Column subset selection, matrix factorization, and eigenvalue optimization, Proc. ACM-SIAM Symposium on Discrete Algorithms, Needell, Randomized Kaczmarz solver for noisy linear systems, BIT Num. Math., 50(2) Needell, Ward, Two-subspace Projection Method for Coherent Overdetermined Systems, submitted. Needell, Tropp, Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method, Submitted.