Low Rank Approximation Lecture 3. Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL

Transcription

1 Low Rank Approximation Lecture 3 Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL daniel.kressner@epfl.ch 1

2 Sampling based approximation Aim: Obtain rank-r approximation of m n matrix A from selected entries of A. Two different situations: Unstructured sampling: Let Ω {1,..., m} {1,..., n}. Solve min A BC T Ω, M 2 Ω = mij 2. (i,j) Ω Matrix completion problem solved by general optimization techniques (ALS, Riemannian optimization). Will discuss later. Column/row sampling: Focus of this lecture. 2

3 Row selection from orthonormal basis Task. Given orthonormal basis U R n r find a good r r submatrix of U. Classical problem already considered by Knuth. 1 Quantification of good : Smallest singular value not too small. Some notation: Given an m n matrix A and index sets I = {i 1,..., i k }, 1 i 1 < i 2 < i k m, J = {j 1,..., j l }, 1 j 1 < j 2 < j l n, we let A(I, J) = a i1,j 1 a i1,j n.. R k l. a im,j 1 a im,j n The full index set is denoted by :, e.g., A(I, :). det A denotes the volume of a square matrix A. 1 Knuth, Donald E. Semioptimal bases for linear dependencies. Linear and Multilinear Algebra 17 (1985), no. 1,

4 Row selection from orthonormal basis Lemma (Maximal volume yields good submatrix) Let index set I, #I = r, be chosen such that det(u(i, :)) is maximized among all r r submatrices. Then 1 σ min (U(I, :)) r(n r) + 1 Proof. 2 W.l.o.g. I = {1,..., r}. Consider Ũ = UU(I, :) 1 = ( Ir B). Leading submatrix is still of maximal volume. This implies that every entry of B satisfies b ij 1. (EFY: Why? Hint: By contradiction + clever choice of submatrix.) Hence, U(I, :) 1 2 = UU(I, :) B (n r)r. 2 Following Lemma 2.1 in [Goreinov, S. A.; Tyrtyshnikov, E. E.; Zamarashkin, N. L. A theory of pseudoskeleton approximations. Linear Algebra Appl. 261 (1997), 1â21]. 4

5 Greedy row selection from orthonormal basis EFY. Develop an extension of the lemma: Prove that for an arbitrary n r matrix U of rank r, there is an index set I with #I = r such that 1 σ min (U(I,:)) r(n r) σ min (U). Finding submatrix of maximal volume is NP hard. 3 Greedy algorithm (column-by-column): 4 First step is easy: Choose i such that u i1 is maximal. Now, assume that k < r steps have been performed and the first k columns have been processed. Task: Choose optimal index in column k Civril, A., Magdon-Ismail, M.: On selecting a maximum volume sub-matrix of a matrix and related problems. Theoret. Comput. Sci. 410(47-49), (2009) 4 Reinvented multiple times in the literature. 5

6 Greedy row selection from orthonormal basis By a suitable permutation, suppose that the first k indices are given by I k = {1,..., k} and partition ( ) U11 U U = 12, U U 21 U 11 R k k. 22 We have ( ) ( ) Ik 0 U11 U 12 U = U 21 U 1, Ũ 11 I n k 0 Ũ 22 = U 22 U 21 U 1 11 U This implies det(u(i k {k + i}, I k {k + 1})) = det(u 11 ) Ũ22(i, 1). Greedily maximizing determinant: Choose i such that Ũ22(i, 1)) is maximal. 6

7 Greedy row selection from orthonormal basis Choose i = argmax i=1,...,n u i1. Set I = {i }. for k = 2,..., r do res = U(:, k) U(:, 1 : k)u(i, 1 : k) 1 U(I, k) Choose i = argmax i=k,...,n res(i) Set I I i. end for For I = {i 1,..., i k } define selection operator: S I = ( e i1 e i2 e ik ). Then residual can be expressed as res = U(:, k) U(:, 1 : k)(s T I U(:, 1 : k)) 1 S T I U(:, k) Literally implemented, requires O(nr 2 + r 4 ) ops. EFY. k steps of Gausian elimination (with or without pivoting) yield the Schur complement needed to form r. Use this observation to reduce the complexity to O(nr 2 ) ops 7

8 Greedy row selection from orthonormal basis Performance of greedy algorithm in practice often quite good, but there are counter examples (see later). Theorem For the index set returned by the greedy algorithm, it holds that (S T I U) 1 2 = U(I, :) 1 2 (1 + 2n) r 1 U(:, 1) 1. Proof. 5 W.l.o.g., may assume I = {1,..., r}. Proof by induction wrt k. For k = 1, u 11 is maximal element in first column U(:, 1), which yields result. Now, assume it is satisfied for k 1 and partition U 11 u 12 U = u21 T u 22, U 11 R (k 1) (k 1). ( ) Aim is to estimate Ũ 1 2 for Ũ := U11 u 12 u21 T. u 22 5 See Lemma 3.2 in [Chaturantabut, S. and Sorensen, D. C. Nonlinear model reduction via discrete empirical interpolation. SIAM Journal on Scientific Computing, 32(5), , 2010]. 8

9 Greedy row selection from orthonormal basis From the construction, we have the block LU decomposition ( ) ( ) ( ) ( ) ( I 0 U11 u Ũ = 12 U11 0 I 0 I U 1 u21 T = U ρ 0 1 u21 T 11 u ) ρ with ρ = u 22 u21 T U 1 11 u 11. We have ( res = u(:, r) U(:, 1 : r 1)U 1 U 11 u 1 12 = U(:, 1 : r) 11 u ) By construction, ρ is the element of maximal absolute value of res. Hence, ( U 1 11 u ) 12 = res 1 2 n res = nρ. 2 9

10 Greedy row selection from orthonormal basis Therefore, Ũ 1 = Ũ 1 2 U ( I ρ 1 U 1 11 u ) ( 12 I 0 0 ρ 1 u21 T 1 ) ( U 1 ) ( ( 1 + ρ 1 U 1 11 u ) 12 ( u T ) ) 2 U (1 + 2n) Together with induction assumption, completes proof. 10

11 Vector approximation Goal: Want to approximate vector f in subspace range(u). Minimal error attained by orthogonal projection UU T. When replaced by oblique projection U(S T I U) 1 S T I f increase of error bounded by result of lemma. Lemma f U(S T I U) 1 S T I f 2 (S T I U) 1 2 f UU T f 2. Proof. Let Π = U(S T I U) 1 S T I. Then (I Π)f 2 = (I Π)(f UU T f ) 2 I Π 2 f UU T f 2. EFY. Prove that any projector Π {0, In} satisfies I Π 2 = Π 2. Hint: Choose orthonormal bases U, U of range(π) and its complement, respectively, and express Π and I Π in [U, U ]. The proof is completed by noting I Π 2 = Π 2 (S T I U) 1 S T I 2 = (S T I U)

12 Connection to interpolation We have S T I (f U(S T I U) 1 S T I f ) 2 = 0 f is interpolated exactly at selected indices. Let f contain discretization of exp(x) on [ 1, 1] let U contain orthonormal basis of discretized monomials {1, x, x 2,...}

13 Connection to interpolation Iteration 1, Err 14.8 Iteration 2, Err Iteration 3, Err 0.7 Iteration 4, Err

14 Connection to interpolation Comparison between best approximation, greedy approximation, approximation obtained by selecting first r indices Names: Continuous setting: EIM (Empirical Interpolation method), [M. Barrault, Y. Maday, N. C. Nguyen, and A. T. Patera, An empirical interpolation method: Application to efficient reduced-basis discretization of partial differential equations, C. R. Math. Acad. Sci. Paris, 339 (2004), pp ]. Discrete setting: DEIM (Discrete EIM). 14

15 POD+DEIM Consider LARGE ODE of the form A is n n matrix. Idea of POD 6 : ẋ(t) = Ax(t) + F (x(t)). 1. Simulate ODE for one or more initial conditions and collect trajectories at discrete time points into snapshot matrix: X = ( x(t 1 ) x(t m ) ). 2. Compute ONB V R n r, r n, of dominant left subspace of X (e.g., by SVD). 3. Assume approximation x UU T x = Uy and project dynamical system onto range(u): ẏ(t) = U T AUy(t) + U T F (Uy(t)). 6 See [S. Volkwein. Proper Orthogonal Decomposition: Theory and Reduced-Order Modelling. Lecture Notes, 2013] for a comprehensive introduction. 15

16 POD+DEIM Problem: U T F(Uy(t)) still involves (large) dimension of original system. Using DEIM: U T F(Uy(t)) (S T I U) 1 S T I F(Uy(t)). ẏ(t) = U T AUy(t) + (S T I U) 1 S T I F (Uy(t)). Only need to evaluate #I = r instead of n entries of function F. Particularly efficient for f 1 (x 1 ) f i1 (x i1 ) F(x) = S T I F (Uy(t)) =. f n (x n ). f ir (x ir ) Example from [Chaturantabut/Sorensen 2010]: Discretized FitzHugh-Nagumo equations involve F(x) = x (x 0.1) (1 x). 16

17 Greedy row selection from orthonormal basis QR-based variant of index selection: Inspired from Orthogonal Matching Pursuit.. In step k + 1: 1. Select row i k+1 of maximal 2-norm from U. 2. Set x = U(i k+1, :) T / U(i k+1, :) Update U U(I r xx T ). This is QR with column pivoting applied to U T. Theorem For the index set I Q returned by QR with pivoting, it holds that n r + 1 (S T I Q U) 1 2 = U(I Q, :) 1 2 4r + 6r 1. 3 Proof. See Theorem 2.1 in [Drmač, Z. and Gugercin, S. A New Selection Operator for the Discrete Empirical Interpolation Method Improved A Priori Error Bound and Extensions. SIAM Journal on Scientific Computing, :2, A631 A648]. EFY. Using MATLAB s command qr, implement the method above. Apply it to the exponential function, as above, and compare with the standard greedy method. 17

18 Counter example for greedy Let U be Q-factor of economy sized QR factorization of n r matrix A = Variation of famous example by Wilkinson. (QR-based) greedy do no pivoting, at least in exact arithmetic U(I, :) 1 2 vs. r for n = 100 and U returned by greedy. 18

19 Row selection beyond greedy Improve upon maxvol-based greedy (in a deterministic framework) via Knuth s iterative exchange of rows. Given index set I, #I = r, and µ 1, µ 1, form Ũ = UU(I, :) 1. Search for largest element If (i, j ) = argmax ũ ij. ũ i j µ, (1) terminate algorithm. Otherwise, set I I\{j } {i } and repeat. EFY. Show that the this row exchange increases the volume of Ũ(I, :) and U(I, :) by at least µ. EFY. Implement this strategy and apply it to Wilkinson s example from the previous slide. How many iterates are needed for r = 30 to achieve µ = 1.01? QR-based greedy can be improved by existing methods for rank-revealing QR [Golub/Van Loan 2013]. 19

20 Theorem Let U(I K, :) be the submatrix obtained upon termination of Knuth s procedure. Then 1 det U(I K, :) (µ r) max det U(I, :) r I Proof. By construction, all entries of UU(I K, :) 1 are bounded by µ in absolute value. In particular this holds for X = U(I, :)U(I K, :) 1 for any index set I {1,..., n} with #I = r. Hence, by Hadamard s inequality, det U(I, :) r det U(I K, :) = det(x) X(:, j) 2 r r j=1 r X(:, j) (µ r) r. j=1 20

21 The CUR decomposition: Existence results A = CUR, where C contains columns of A, R contains rows of A, U is chosen wisely. Theorem (Goreinov/Tyrtyshnikov/Zamarshkin 1997). Let ε := σ k+1 (A). Then there exist row indices I {1,..., m} and column indices J {1,..., n} and a matrix S R k k such that A A(:, J)SA(I, :) 2 ε(1 + 2 k( m + n)). Proof. Let U k, V k contain k dominant left/right singular vectors of A. Choose I, J by selecting rows from U k, V k. According to max volume lemma, the square matrices Û = U k(i, :), ˆV = V k (J, :) satisfy Û 1 2 k(m k) + 1, ˆV 1 2 k(n k) + 1. Remains to choose S. 21

22 The CUR decomposition: Existence results Form Φ = ÛΣ ˆV k T and choose k such that Φ T k(φ) ε 2 Û 1 2 ˆV. 1 2 We set S = T k(φ) +. We now estimate the four different components of U k Σ k Vk T A(:, J)SA(I, :) wrt U k, V k and their complements: 1. U k Uk T (...)V kvk T : Σ k Uk T A(:, J)SA(I, :)V k 2 = Σ k Σ k ˆV S ÛΣ k 2 = Σ k Û 1ΦT k(φ) + Φ ˆV 1 2 = Û 1 (Φ T k(φ)) ˆV 1 2 ε Û 1 2 ˆV (I m U k Uk T )(...)V kvk T (I U k U k ) T A(:, J)SA(I, :)V k 2 ε SA(I, :)V k 2 = ε T k(φ) + Φ ˆV 1 ε ˆV 1 2 Û 1 3. U k Uk T (...)(I V kvk T ): As in 2, bounded by ε 2. 22

23 The CUR decomposition: Existence results 4. (I U k U T k )(...)(I V kv T k ): (I U k U k ) T A(:, J)SA(I, :)(I V k Vk T ) 2 ε 2 S 2 ε Û 1 2 ˆV 1 2 In summary, we obtain ( ( ) ) 2 A A(:, J)SA(I, :) 2 ε 1 + Û ˆV 1 2, which implies the result. 23

24 The CUR decomposition: Existence results Choice of S = (A(I, J)) 1 in CUR Remainder term R := A A(:, J)(A(I, J)) 1 A(I, :) has zero rows at I and zero columns at J. Cross approximation:

25 Adaptive Cross Approximation (ACA) A more direct attempt to find a good cross.. Theorem (Goreinov/Tyrtyshnikov 2001). Suppose that [ ] A11 A A = 12 A 21 A 22 where A 11 R r r has maximal volume among all r r submatrices of A. Then where M C := max ij m ij A 22 A 21 A 1 11 A 12 C (r + 1)σ r+1 (A), As we already know, finding A 11 is NP hard [Çivril/Magdon-Ismail 2013]. 25

26 Adaptive Cross Approximation (ACA) Proof of theorem for (r + 1) (r + 1) matrices. Consider ( ) A11 a A = 12 a21 T, A a 11 R r r, a 12, a 21 R r 1, a 22 R, 22 with invertible A 11. We recall the definition of the adjunct of a matrix: adj(a) = C T, c ij = ( 1) i+j m ij, where m ij is the determinant of the matrix obtained by deleting row i and j of A. By the max-volume assumption, m r+1,r+1 is maximal among all m ij. On the other hand, A 1 = 1 det A adj(a). This implies that the element of A 1 of maximum absolute value is at position (r + 1, r + 1): (A 1 ) r+1,r+1 = A 1 C := max (A 1 ) ij. i,j 26

27 Adaptive Cross Approximation (ACA) Proof of theorem for continued. On the other hand, we have for any k l matrix B that Thus, B 2 B F kl B C. σ r+1 (A) 1 = A 1 2 (r + 1) A 1 C = (r + 1) (A 1 ) r+1,r+1. This completes the proof, using (e.g., via Schur complement) (A 1 ) r+1,r+1 = 1 a 22 a21 T A 1 11 a

28 Adaptive Cross Approximation (ACA) ACA with full pivoting [Bebendorf/Tyrtyshnikov 2000] 1: Set R 0 := A, I := {}, J := {}, k := 0 2: repeat 3: k := k + 1 4: (i k, j k ) := arg max i,j R k 1 (i, j) 5: I I {i k }, J J {j k } 6: δ k := R k 1 (i k, j k ) 7: u k := R k 1 (:, j k ), v k := R k 1 (i k, :) T /δ k 8: R k := R k 1 u k v T k 9: until R k F ε A F This is greedy for maxvol. (Proof on next slide.) Still too expensive for general matrices. 28

29 Adaptive Cross Approximation (ACA) Lemma (Bebendorf 2000). Let I k = {i 1,..., i k } and J k = {j 1,..., j k } be the row/column index sets constructed in step k of the algorithm. Then det(a(i k, J k )) = R 0 (i 1, j 1 ) R k 1 (i k, j k ). Proof. From lines 7 and 8 of the algorithm, R k 1 (I k, j k ) is obtained from A(I k, j k ) by subtracting scalar multiples of columns j 1,..., j k 1 of A. Hence, there is a vector y such that A(I k, J k ) = [ A(I k, J k 1 ) R k 1 (I k, j k ) ] [ ] I k 1 y. 0 1 This implies det(ãk) = det(a(i k, J k )). However, R k 1 (i, j k ) = 0 for all i = i 1,..., i k 1 and hence det A(I k, J k ) = R k 1 (i k, j k ) det(a(i k 1, J k 1 )). Since det A(I 1, J 1 ) = A(i 1, j 1 ) = R 0 (i 1, j 1 ), the result follows by induction. 29

30 Adaptive Cross Approximation (ACA) ACA with partial pivoting 1: Set R 0 := A, I := {}, J := {}, k := 1, i := 1 2: repeat 3: j := arg max j R k 1 (i, j) 4: δ k := R k 1 (i, j ) 5: if δ k = 0 then 6: if #I = min{m, n} 1 then 7: Stop 8: end if 9: else 10: u k := R k 1 (:, j ), v k := R k 1 (i, :) T /δ k 11: R k := R k 1 u k vk T 12: k := k : end if 14: I I {i }, J J {j } 15: i := arg max i I u k (i) 16: until stopping criterion is satisfied 30

31 Adaptive Cross Approximation (ACA) ACA with partial pivoting. Remarks: R k is never formed explicitly. Entries of R k are computed from R k (i, j) = A(i, j) k u l (i)v l (j). l=1 Ideal stopping criterion u k 2 v k 2 ε A F elusive. Replace A F by A k F, recursively computed via k 1 A k 2 F = A k 1 2 F + 2 uk T u j vj T v k + u k 2 2 v k 2 2. j=1 31

32 Adaptive Cross Approximation (ACA) Two matrices: (a) The Hilbert matrix A defined by A(i, j) = 1/(i + j 1). (b) The matrix A defined by A(i, j) = exp( γ i j /n) with γ = Hilbert matrix Full pivoting Partial pivoting SVD 10 0 Exponential matrix Full pivoting Partial pivoting SVD A Ã 2/ A A Ã 2/ A k k 1. Excellent convergence for Hilbert matrix. 2. Slow singular value decay impedes partial pivoting. 32

33 ACA is Gaussian elimination We have R k = R k 1 δ k R k 1 e k e T k R k 1 = (I δ k R k 1 e k e T k )R k 1 = L k R k 1, where L k R m m is given by L k = 0, l i,k = et i R k 1 e k l k+1,k 1 ek T R. k 1e k.... l m,k 1 for i = k + 1,..., m. Matrix L k differs only in position (k, k) from usual lower triangular factor in Gaussian elimination. 33

34 ACA for SPSD matrices For symmetric positive semi-definite matrix A R n n : SVD becomes spectral decomposition. Can use trace instead of Frobenius norm to control error. R k stays SPSD. Choice of rows/columns, e.g., by largest diagonal element of R k. ACA becomes = Cholesky (with diagonal pivoting). Analysis in [Higham 1990]. = Nyström method [Williams/Seeger 2001]. See [Harbrecht/Peters/Schneider 2012] for more detais. 34

35 Outlook on randomized algorithms Coming back to DEIM for exponential function. Comparison between best approximation, greedy approximation, approximation obtained by randomly selecting r indices

36 Outlook on randomized algorithms A simple way to fool uniformly random selection: ( ) 0(n r) r U = I r for n very large and r n. 36