Solving linear equations with Gaussian Elimination (I)

Transcription

1 Term Projects Solving linear equations with Gaussian Elimination The QR Algorithm for Symmetric Eigenvalue Problem The QR Algorithm for The SVD Quasi-Newton Methods

2 Solving linear equations with Gaussian Elimination I Gaussian Elimination with Partial Pivoting GEPP Gaussian Elimination with Complete Pivoting GECP Iterative Refinement Comparison with matlab \ function

3 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn

4 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs

5 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs row interchange: permutation

6 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs row interchange: permutation eliminating xs from E s+1 through E n : l js = a js a ss, s + 1 j n, a jk = a jk l js a sk, s + 1 j, k n

7 GEPP as LU factorization Theorem: Let A = a ij R n n be non-singular Then GEPP computes an LU factorization with permutation matrix P such that P A = L U = P is the accumulation of row interchanges, L is unit lower triangular, and U upper triangular

8 Solving general linear equations with GEPP A x = b, P A = L U interchanging components in b P A x = P b, L U x = P b solving for b with forward and backward substitution x = L U 1 P b = U 1 L 1 P b

9 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn

10 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: ip, jp def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp

11 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: ip, jp def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp eliminating x s from E s+1 through E n : a jk = a jk a js a ss a sk overwrite ==== a jk, s + 1 j n, s + 1 k n, More numerically stable, too expensive GECP computes LU factorization with permutations P row and P column, P row A P column = L U =

12 Error Bounds and Iterative Refinement Assume that x is an approximation to the solution x of A x = b Residual r def = b A x = A x x Thus small x x implies small r T hm: Assume x is an approximate solution with r = b A x Then for any natural norm, x x A 1 r, x x x κ A r b, where κ A def = A A 1 is the condition number of A

13 Iterative Refinement I Let A x = b with non-singular A and non-zero b Let F be an in-exact equation solver, so F b is approximate solution Assume F is accurate enough that there exists a ρ < 1 so b A F b b ρ for any b 0

14 Iterative Refinement I Let A x = b with non-singular A and non-zero b Let F be an in-exact equation solver, so F b is approximate solution Assume F is accurate enough that there exists a ρ < 1 so b A F b b ρ for any b 0 For this project, F will be: LU factorization without pivoting, LU factorization with partial pivoting, and LU factorization with complete pivoting F b = U 1 L 1 b

15 Iterative Refinement II Given a tolerance τ > 0 and x 0 Initialize r 0 = b A x 0 for k = 0, 1, Compute x k = F r k, x k+1 = x k + x k, r k+1 = r k A x k If r k+1 τ b stop

16 Project Requirements Implement GE, GEPP, GECP, with and without iterative refinement Compare with matlab \ function in terms of accuracy residual norm and execution time

17 The QR Algorithm for Symmetric Eigenvalue Problem Householder reduction to tridiagonal Implicit bulge Chasing scheme for tridiagonal QR Algorithm Recover eigenvectors

18 Householder Reduction for symmetric A R n n Let v j R n j be the unit vector in the Householder Reflection matrix Ĥj = I 2 v j vj T so that def = R n n, j = 1,, n 1 H j Ij Ĥj Perform n 1 Householder reflections so that α 1 β 1 β 1 α 2 β 2 H n 1 H 1 A H1 T Hn 1 T = β 2 α 3 β 3 def = T β n 1 α n Householder s Method, where H def = H T 1 HT n 1 H T A H = T, or A = H T H T,

19 Computing H For j = n 2, n 3,, 2, 1, let I j def H j+1 H n 1 = 1 H j H j+1 H n 1 = where H j = = Ij Ĥj I 2 v j v T j 1 H j+1 1 I j 1 H j+1 H j+1 2 v j vj T H j+1 1 Then = Ij H j+1 H j,

20

21 Project Requirements Householder reduction to tridiagonal form Implicit Bulge Chasing tridiagonal QR Algorithm for all eigenvalues Accumulate all Householder reflections and Givens rotations to recover all eigenvectors Total cost On 3 flops Compare with matlab eig function in terms of accuracy residual and orthogonality and execution time Algorithm Details

22 The QR Algorithm for SVD Householder reduction to bidiagonal form Implicit bulge chasing scheme for bidiagonal SVD Recover singular vectors

23 Householder Bidiagonal Reduction for A R n n I Partition matrix A = a 1 Â 1 R n n, Â 1 R n n 1 Let u 1 R n be the unit vector in the Householder Reflection def matrix G 1 = Ĝ1 = I 2 u 1 u T 1 so that ± a1 Ĝ 1 a 1 = 2 def α1 = and 0 0 def α1 b G 1 A = Ĝ1 a 1 Ĝ 1 Â 1 = T 1 0 A 1 Let v 1 R n 1 be the unit vector in the Householder Reflection matrix Ĥ1 = I 2 v 1 v1 T so that, ± b1 Ĥ 1 b 1 = 2 def β1 1 = With H = Ĥ1 α G 1 A H 1 = 1 b T 1 Ĥ1 def α1 β 1 0 T = 0 A 1 Ĥ 1 0 a 2 Â 2,

24 α1 β 1 0 T G 1 A H 1 = 0 a 2 Â 2, Â 2 R n 1 n 2 Let u 2 R n 1 be the unit vector in Ĝ2 = I 2 u 2 u T 2 so that ± a2 Ĝ 2 a 2 = 2 def α2 1 = With G = Ĝ2 α1 β 1 0 T α 1 β 1 0 T def G 2 G 1 A H 1 = = 0 α 2 b T 2 0 Ĝ 2 a 2 Ĝ 2 Â A 2, Let v 2 R n 2 be the unit vector in Ĥ2 = I 2 v 2 v2 T so that, ± b2 Ĥ 2 b 2 = 2 def β2 I2 = With H = Ĥ2 α 1 β 1 0 T α 1 β T G 2 G 1 A H 1 H 2 = 0 α 2 b T 2 Ĥ2 def = 0 α 2 β 2 0 T 0 0 A 2 Ĥ a 3 Â 3,

25 Householder Bidiagonal Reduction for A R n n III After n 1 pairs of Householder Reflections, α 1 β 1 α 2 β 2 G n 1 G 2 G 1 A H 1 H 2 H n 1 = α 3 βn 1 B is an upper bidiagonal matrix Next step is QR algorithm for bidiagonal SVD α n def = B

26 Input matrix B = α 1 β 1 α 2 β 2 α 3 βn 1, tolerance τ α n β n Algorithm 1 BidiSVD B, τ while not yet converged do if n = 1 then return SVD of B for j = 1,, n 1 do if β j τ then return BidiSVD B 1 : j, 1 : j, τ and BidiSVD B j + 1 : n, j + 1 : n + 1, τ end if if α j τ then return BidiSVD B 1 : j 1, 1 : j, τ and BidiSVD B j : n, j + 1 : n + 1, τ end if end for Compute shift σ Bulge-chasing QR with σ end while

27 All forms Bidiagonal B = α 1 β 1 α 2 β 2 α 3 βn 1, B = β 1 α 2 β 2 α 3 βn 1 B = α 1 β 1 α 2 β 2 α n α 3 βn 1 β n, B = β 1 α 2 β 2 α 3 βn 1 α n α n α n β n

28 All forms Bidiagonal B = α 1 β 1 α 2 β 2 α 3 βn 1, B = β 1 α 2 β 2 α 3 βn 1 B = α 1 β 1 α 2 β 2 α n α 3 βn 1 β n, B = β 1 α 2 β 2 α 3 βn 1 α n α n Next: QR for SVD of B = β 1 α 2 β 2 α 3 βn 1 α n β n α n β n

29 Bidiagonal SVD B B T is symmetric tridiagonal β1 2 α 2 β 1 α 2 β 1 α2 2 + β2 2 α 3 β 2 B B T = α 3 β 2 α3 2 + β2 3 αn β n 1 Bottom 2 2 matrix α 2 n 1 + βn 1 2 α n β n 1 α n β n 1 αn 2 + βn 2 = S S T, with S def = α n β n 1 α 2 n + β 2 n αn 1 β n 1 α n β n Shift σ is the singular value of S closer to αn 2 + βn 2 Bulge-chasing QR based on QR factorization of β1 2 σ2 α 2 β 1 B B T σ 2 α I = 2 β 1 α2 2 + β2 2 σ2 αn β n 1 α n β n 1 αn 2 + βn 2 σ 2

30 Bulge-chasing QR for Bidiagonal SVD I Givens rotation on first column of B B T σ 2 I : G 1 B B T σ 2 def I = I n 2 s 1 c 1 α 2 β 1 c1 s 1 β 2 1 σ 2 0 = 0 0 G 1 is the first Givens rotation towards QR factorization of B B T σ 2 I d1 f Creating bulge: 1 def c1 s = 1 β1 0 d 2 f 2 s 1 c 1 α 2 β 2 G 1 B = d 1 f 1 d 2 f 2 α 3 βn 1 = + α n β n Next: Chase Bulge + down along the diagonals

31 Bulge-chasing QR for Bidiagonal SVD II G 1 B = d 1 f 1 d 2 f 2 α 3 βn 1 = + α n β n Givens rotation on first row, H 1 def = c1 s 1 s 1 c 1 I n 2 d1 f 1 0 T H1 T = d1 2 + f T def = β1 0 T Compute d2 f 2 α 3 c1 s 1 s 1 c 1 T def = d 2 f 2 d 3 f 3

32 Bulge-chasing QR for Bidiagonal SVD III 1 B H 1 = β 1 d 2 f 2 d 3 f 3 β 3 α 4 βn 1 = + α n β n Givens rotation on first column, G 2 def = 1 c 2 s 2 s 2 c 2 I n 3 c2 s 2 d 2 f 2 0 s 2 c 2 d 3 f 3 β 3 def = α2 d 2 f 2 0 d 3 f 3

33 Bulge-chasing QR for Bidiagonal SVD IV G 2 G 1 B H 1 = β 1 α 2 d 2 f 2 d 3 f 3 α 4 βn 1 = + α n β n New Givens rotation: d2 f 2 T β c2 s 2 0 d 3 f 2 3 = d s 0 α 2 c 3 f d 4 f 4 Repeat procedure until bulge disappears at the lower right corner

34 Project Requirements Householder reduction to bidiagonal form Implicit Bulge Chasing bidiagonal QR Algorithm for all singular values Accumulate all Householder reflections and Givens rotations to recover all singular vectors Total cost On 3 flops Compare with matlab svd function in terms of accuracy residual and orthogonality and execution time

35 Quasi-Newton Methods BFGS for minx g x I BFGS Method: xk+1 = xk α Bk 1 g xk, I I I I where Bk Rn n Desired Property I: Mimics B xk in some sense 1 Desired Property II: Easy to compute Bk+1 from Bk 1 Ensure symmetry in Bk Step size selection as in Steepest Descent

36 BFGS Method: Derivation Assume for some step k 0, we have available x k R n and B k R n n By BFGS Method: x k+1 = x k B 1 k g x k Secant equation g x k+1 g x k H x k x k+1 x k BFGS ideas: Approximate secant equation: Bk+1 s k+1 = y k+1, where def y k+1 = g x k+1 g x k def, s k+1 = x k+1 x k Choose a special, symmetric Bk+1 that does not differ much from B k B k+1 = B k + y k+1yk+1 T yk+1 T s B k s k+1 s T k+1 B k k+1 s T k+1 B k s k+1

37 BFGS Method Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, x k+1 = x k α B 1 k g B k+1 = B k + y k+1yk+1 T yk+1 T s k+1 x k+1 g y k+1 = g x k, B k s k+1 s T k+1 B k s T k+1 B k s k+1, with x k, s k+1 = x k+1 x k

38 BFGS Method Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, x k+1 = x k α B 1 k g B k+1 = B k + y k+1yk+1 T yk+1 T s k+1 x k+1 g y k+1 = g Practical details: x k, B k s k+1 s T k+1 B k s T k+1 B k s k+1, with x k, s k+1 = x k+1 x k B k remains SPD for convex problems Cholesky factorization of B k can help to compute Cholesky factorization of B k+1? updating Cholesky factorization BFGS inverse update: T + s k+1 s T k+1 B 1 k+1 = I s k+1 y T k+1 s T k+1 y k+1 B 1 k I s k+1 y T k+1 s T k+1 y k+1 s T k+1 y k+1

39 Limited Memory BFGS L-BFGS Method I B 1 k+1 maybe too large to fit in memory L-BFGS: Mimics BFGS, but with linear memory BFGS inverse update, from B 1 k B 1 k+1 = I s k+1 y T k+1 s T k+1 y k+1 B 1 k to B 1 k+1 : I s k+1 yk+1 T T s T k+1 y + s k+1 s T k+1 k+1 s T k+1 y k+1 Big Idea: Go back only m + 1 steps in inverse update

40 Limited Memory BFGS L-BFGS Method II B 1 k+1 = s k+1 s T k+1 s T k+1 y k+1 B 1 k m m 1 + I s k+1 y T k+1 s T k+1 y k+1 I y k+1 m s T k+1 m s T k+1 m y k+1 m + I s k+1 yk+1 T s T j=0 k+1 y k+1 sk j s T k j s T k j y k j I y k+1 j s T k+1 j s T k+1 j y k+1 j I s k+1 m yk+1 m T s T k+1 m y k+1 m I y k+1 s T k+1 s T k+1 y k+1 I s k+1 j yk+1 j T s T k+1 j y k+1 j I y k+1 s T k+1 s T k+1 y k+1 Big Idea: Go back only m + 1 steps in inverse update: Set B 1 k m = B 1 0

41 Computing B 1 k+1 h for any vector h I B 1 k+1 h = α k+1 s k+1 + m 1 I s k+1 yk+1 T s T k+1 y k+1 + I s k+1 yk+1 T s T j=0 k+1 y k+1 sk j s T k j s T k j y q k+1 j, k j I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j y T k+1 j s T k+1 j y k+1 j B 1 0 q k+1 m where α k+1 = q t def = s T k+1 h s T k+1 y, q k+1 = h α k+1 y k+1, k+1 I y t s T t s T I y k+1 s T k+1 t y t s T k+1 y h k+1 = q t+1 α t y t, α t = st t q t+1 s T t y t, t = k,, k + 1 m

42 Computing B 1 h for any vector h II k+1 Let z k+1 m = B0 1 q k+1 m Then B 1 k+1 h = α k+1 s k m 1 j=0 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 y T k+1 s T k+1 y k+1 I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j y T k+1 j s T k+1 j y k+1 j z k+1 m α k j s k j

43 Computing B 1 h for any vector h II k+1 Let z k+1 m = B0 1 q k+1 m Then B 1 k+1 h = α k+1 s k m 1 j=0 = α k+1 s k m 2 j=0 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 y T k+1 s T k+1 y k+1 I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j yk+1 j T s T k+1 j y k+1 j I s k+2 m yk+2 m T s T k+2 m y k+2 m I s k+1 j y T k+1 j s T k+1 j y k+1 j z k+1 m α k j s k j z k+2 m α k j s k j, where z k+2 m = z k+1 m + α k+1 m β k+1 m s k+1 m, β k+1 m = yt k+1 m z k+1 m s T k+1 m y k+1 m

44 Algorithm 2 Computing B 1 k+1 h for any vector h Set q = h for t = k + 1, k,, k + 1 m do Compute α t = st t q, q = q α s T t y t y t t end for Compute z = B0 1 q for t = k + 1 m,, k + 1 do Compute β = yt t z, z = z + α s T t y t β s t t end for Return z

45 Step-size Selection Ensure reduction in g x: Algorithm 3 will halt with an α 3 > 0 so that g x k + α 3 d k < g x k Algorithm 3 g x Reduction Set α 3 = 1 while g x k + α 3 d k g x k do Set α 3 = α 3 2 end while

46 Step-size Selection Ensure reduction in g x: Algorithm 3 will halt with an α 3 > 0 so that g x k + α 3 d k < g x k Algorithm 3 g x Reduction Set α 3 = 1 while g x k + α 3 d k g x k do Set α 3 = α 3 2 end while Ensure sufficient reduction in g x with α k : Algorithm 4 g x Sufficient Reduction Set α 2 = α 3 /2 Compute coefficients h 0, h 1, h 2 so P α = h 0 + h 1 α + h 2 α α α 2 interpolates g x k + α d k at α = 0, α 2, α 3 Set α 0 = 1 2 α 2 h 1 /h 2 α k = argmin α [α0,α 2,α 3 ]g x k + α d k

47 BFGS Method for min x g x Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, Compute x d = B 1 k g x k Compute step size α x k+1 = x k α B 1 k g x k, B k+1 = B k + y k+1yk+1 T yk+1 T s B k s k+1 s T k+1 B k k+1 s T k+1 B, with k s k+1 y k+1 = g x k+1 g x k, s k+1 = x k+1 x k

48 Project Requirements Implement BFGS and L-BFGS Run BFGS and L-BFGS with logistic objective function