Spectrum-Revealing Matrix Factorizations Theory and Algorithms

Transcription

1 Spectrum-Revealing Matrix Factorizations Theory and Algorithms Ming Gu Department of Mathematics University of California, Berkeley April 5, 2016 Joint work with D. Anderson, J. Deursch, C. Melgaard, J. Xiao

2 Content Low-rank matrix approximations Rank-revealing matrix factorizations vs. Spectrum-revealing matrix factorizations. Related new algorithms Numerical Experiments Future work

3 Background Applications: Bing Search Scientific Computing/Data Analysis Computer Vision Machine learning; kernel learning.

4 Google Search on Low-rank Approximation

5 Goals in Low-rank Matrix Approximations Highly efficient Minimum communication As accurate/reliable as truncated SVD Beyond SVD: preserve sparsity, structure, interpretability

6 Background: Low-Rank Approximation Methods Deterministic low-rank approximations Interpolative decomposition (ID) (Cheng et. al. 2005, Liberty et. al 2007) Deterministic CUR with various column selection algorithms Randomized low-rank approximations Using the subsampled fft technique, for l k: optimal complexity is O ( mn log(k) + l 2 (m + n) ) Several approximation algorithms (Halko et. al. 2010) Some of the first randomized algorithms. (Rokhlin et. al. 2008) Randomized LU decomposition for low-rank approximation (Shabat et. al. 2015) Spectrum-revealing factorizations stronger in theory and efficiency

7 Background: Other Data Analysis Methods Data Analysis: Low-rank matrix approximations that respect data sparsity, structure, and interpretability; more informative than truncated SVD. CX decomposition: for medical data, text data, etc A CX, where C consists of well-chosen columns of A. CUR decomposition: for recommendation systems, text data analysis A CUR, where C and R consist of well-chosen columns and rows of A. LHL T decomposition: Approximate kernel methods, independent component analysis A LHL T, where L consists of well-chosen columns of SPD matrix A.

8 Google Search on CUR Decomposition

9 Golden Standard: Truncated SVD Given matrix M R m n with m n, its SVD is M = UΣV T = ( u 1 ) u n σ 1... σ n ( v 1 v n ) T, where σ j is the j th largest singular value of M. The rank-k truncated SVD is M k = U k Σ k Vk T = ( u 1 u k ) σ 1... ( v 1 v k ) T. σ k Theorem: (Eckart & Young, 1936) Min rank(h) k M H 2 = M M k 2 = σ k+1 (M).

10 Review: Rank-revealing Matrix factorizations: QR Given: Matrix M C m n and 1 l min(m, n) = n, Existence: QR factorization with column permutation Π ( ) Al B MΠ = Q l with 0 C l σ j (M) 1 + l(n l) σ j (A l ) σ j (M), j = 1,, l, σ l+1 (M) C l 2 σ l+1 (M) 1 + l(n l), where σ j ( ) is the j th largest singular value. Rank-revealing: A l reveals numerical rank of M.

11 Example: Kahan Matrix For s, c > 0 and s 2 + c 2 1, Kahan Matrix K n = S n C n : 1 c c c 1 c c S n = diag(1, s,, s n 1 ), C n = 1 c Kahan Matrix not in rank revealed form: (( ) n ) s σ min (K n ) = O s n c Little consensus on definition of reveal.

12 New: Spectrum-revealing factorizations (I): QR/CX Given: Matrix M C m n and 1 k l min(m, n) = n, Existence: QR factorization with column permutation Π ( Al B MΠ = Q l 0 C l ) with τ j = σ l+1 (M) σ j (M) j = 1,, k + 1, σ j (M) ( ) σ (( j Al )) B l σj (M) 1 + O τj 2 σ k+1 (M) MΠ QM k σ k+1 (M) 1 + O ( ) τk+1 2 ( cf. σ j (M) σ j (A l ), C l 2 σ l+1 (M) ) 1 + l(n l). 1 + l(n l) ( ) Al B M k = truncated SVD of l ; σ 0 j ( ) = j th largest singular value. QM k well approximates MΠ and reveals its leading singular values.

13

14 New: Spectrum-revealing factorizations (II) Cholesky Given: SPD matrix M R n n and 1 k l n, Existence: Cholesky factorization with diagonal permutation ( ΠMΠ T = LL T Al 0, L = B l C l ) with τ j = σ l+1 (M) σ j (M) σ j (M) 1 + O (τ j ) σ2 j ( Al B l ) σ j (M), j = 1,, k, σ k+1 (M) ΠMΠ T L k L T k σ k+1 (M) (1 + O (τ k+1 )) L k = truncated SVD of ( Al B l ).

15 New: Spectrum-revealing factorizations (II) LHL T Given: SPD matrix M R n n and 1 k l n, Existence: LHL T factorization with diagonal permutation ( ) ΠMΠ T = LL T Al 0, L =, M = Π T ( Al B l σ j (M) ( 1 + O τ 2 j ) ( Al B l ( ) ) σ j M B l C l ) ( ΠMΠ T ) ( A l B l ) ( Al B l σ j (M), j = 1,, k σ k+1 (M) M M k σ k+1 (M) ) Π 1 + O ( ) τk+1 2 M k = truncated SVD of M; τj = σ l+1(m) σ j (M)

16 New: Spectrum-revealing factorizations (III) LU Given: Matrix M C m n and 1 k l min(m, n) = n, Existence: LU with row, column permutations Π l, Π r ( L11 0 Π l MΠ r = LU, L = L 21 L 22 ) ( U11 U, U = 12 0 U 22 ) with τ j = σ l+1 (M) σ j (M), j = 1,, l σ j (M) 1 + O (τ j ) σ j (( L11 L 21 ) ( U 11 U 12 ) ) σ j (M), σ k+1 (M) M M k σ k+1 (M) (1 + O (τ k+1 )) M k = truncated SVD of ( L11 L 21 ) ( U 11 U 12 ).

17 New: Spectrum-revealing factorizations (III) CUR Given: Matrix M C m n and 1 k l min(m, n) = n, Existence: CUR with row, column permutations Π l, Π r ( L11 0 Π l MΠ r = LU, L = L 21 L 22 ) ( U11 U, U = 12 0 U 22 ) Π l MΠr = ( L11 L 21 ) ( L11 L 21 ) (Π l MΠ r ) ( U 11 U 12 ) ( U11 U 12 ). σ j (M) ( 1 + O τ 2 j ( ) ) σ j M σ j (M), j = 1,, k σ k+1 (M) M M k σ k+1 (M) 1 + O ( ) τk+1 2 M k = truncated SVD of M; τj = σ l+1(m) σ j (M)

18 Talks: Spectrum-Revealing Matrix Factorizations and Randomized Algorithms

19 New Algorithms (Ia) Solving linear systems of equations Ax = b: Standard method: Gaussian Elimination with Partial Pivoting (GEPP): Π A = LU: most efficient, but can be unstable. New method: randomized Gaussian Elimination with Complete Pivoting (RGECP): Π l AΠ r = LU: almost as efficient, but is always stable. (joint work with C. Melgaard.) Communication-avoiding version would be numerically stable, in contrast to CA-LU with partial pivoting.

20 New Algorithms (Ib) Column pivots random: Successive Schur Sampling. RGECP is O(rn 2 ) more flops than GEPP: Theorem: Given δ (0, 1). If ( ) n(n + 1) r > 32 ln, 2δ then the pivot growth factor of RGECP satisfies ρ(a) 3 e(n + 1)n ln(n) with probability greater than 1 δ.

21 New Algorithms (IIa) Low-rank Matrix Approximations Randomized algorithm for computing a Spectrum-revealing LU Decomposition (joint work with D. Anderson): First of its kind. Computes ( ) L11 Π l AΠ r LU, L =, U = ( ) U L 11 U without ever computing the Schur complement. Likely the most efficient for low-rank matrix approximation. Non-classical communication patterns: Computing/Updating random projection. Both row and column swaps. No Schur updating.

22 New Algorithms (IIb)

23 New Algorithms (III) Low-rank Matrix Approximations Randomized algorithm for computing a Spectrum-revealing QR factorization (J. Duersch, Nov. 2015) ( ) R11 R AΠ Q Never need to update trailing submatrix in R. Up to twice as fast as truncated QR.

24 New Algorithms (IV) Low-rank Matrix Approximations Randomized algorithm for computing a Spectrum-revealing Cholesky factorization Randomized block left looking Cholesky. Much promise in quality and efficiency.

25 Numerical Experiments for RGECP (I) Fortran implementation, optimized BLAS. Randomized GECP: Execution Time Relative to GEPP

26 Numerical Experiments for RGECP (II) GEPP and Randomized GECP Element Growth GERCP Wilkinson GEPP Wilkinson GERCP Gen Wilkinson GEPP Gen Wilkinson GERCP Volterra GEPP Volterra Element growth Matrix size n

27 Numerical Experiments for RGECP (III) GEPP and Randomized GECP Residuals for random linear systems.

28 Numerical Experiments for Spectrum-revealing LU (I) SRLU vs. truncated LU, execution times.

29 Numerical Experiments for Spectrum-revealing LU (II) SRLU vs. truncated LU, Complexity comparison.

30 Numerical Experiments for Spectrum-revealing LU (III) SRLU vs. truncated SVD.

31 Numerical Experiments for Spectrum-revealing QR (I) Edison (12 core, shared memory machine). Matrix A is dgeqp2 (full decomposition), 75.84s dgeqrf (full decomposition), 5.91s truncated k=1200 randomized qrcp, 1.59s truncated k=600 randomized qrcp, 0.76s

32 Numerical Experiments for Spectrum-revealing QR (II)

33 Numerical Experiments for Spectrum-revealing QR (III)

34 Numerical Experiments for Spectrum-revealing QR (IV)

35 Conclusions A new suite of randomized algorithms for numerical linear algebra and low-rank matrix approximations, along with theoretical analysis. Codes in different stages of development. Communication-avoiding variants needed. Thank you