Rank revealing factorizations, and low rank approximations
|
|
- Dinah Adams
- 5 years ago
- Views:
Transcription
1 Rank revealing factorizations, and low rank approximations L. Grigori Inria Paris, UPMC January 2018
2 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization with column and row tournament pivoting Experimental results, LU CRTP Randomized algorithms for low rank approximation 2 of 70
3 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization with column and row tournament pivoting Experimental results, LU CRTP Randomized algorithms for low rank approximation 3 of 70
4 Low rank matrix approximation Problem: given m n matrix A, compute rank-k approximation ZW T, where Z is m k and W T is k n. Problem with diverse applications from scientific computing: fast solvers for integral equations, H-matrices to data analytics: principal component analysis, image processing,... Flops Ax ZW T x 2mn 2(m + n)k 4 of 70
5 Singular value decomposition Given A R m n, m n its singular value decomposition is A = UΣV T = ( ) Σ 1 0 U 1 U 2 U 3 0 Σ 2 (V ) T 1 V where U is m m orthogonal matrix, the left singular vectors of A, U 1 is m k, U 2 is m n k, U 3 is m m n Σ is m n, its diagonal is formed by σ 1 (A)... σ n (A) 0 Σ 1 is k k, Σ 2 is n k n k V is n n orthogonal matrix, the right singular vectors of A, V 1 is n k, V 2 is n n k 5 of 70
6 Norms A F = m i=1 j=1 n a ij 2 = A 2 = σ max (A) = σ 1 (A) σ 2 1 (A) +... σ2 n(a) Some properties: A 2 A F min(m, n) A 2 Orthogonal Invariance: If Q R m m and Z R n n are orthogonal, then QAZ F = A F QAZ 2 = A 2 6 of 70
7 Low rank matrix approximation Best rank-k approximation A k = U k Σ k V k is rank-k truncated SVD of A [Eckart and Young, 1936] Original image of size min A Ãk 2 = A A k 2 = σ k+1 (A) (1) rank(ãk ) k n min A Ã k F = A A k F = σj 2 (A) (2) rank(ã k ) k j=k+1 Rank-38 approximation, SVD Rank-75 approximation, SVD Image source: https: //upload.wikimedia.org/wikipedia/commons/a/a1/alan_turing_aged_16.jpg 7 of 70
8 Large data sets Matrix A might not exist entirely at a given time, rows or columns are added progressively. Streaming algorithm: can solve an arbitrarily large problem with one pass over the data (a row or a column at a time). Weakly streaming algorithm: can solve a problem with O(1) passes over the data. Matrix A might exist only implicitly, and it is never formed explicitly. 8 of 70
9 Low rank matrix approximation: trade-offs 9 of 70
10 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization with column and row tournament pivoting Experimental results, LU CRTP Randomized algorithms for low rank approximation 10 of 70
11 Rank revealing QR factorization Given A of size m n, consider the decomposition [ R11 R AP c = QR = Q 12 R 22 ], (3) where R 11 is k k, P c and k are chosen such that R 22 2 is small and R 11 is well-conditioned. By the interlacing property of singular values [Golub, Van Loan, 4th edition, page 487], σ i (R 11 ) σ i (A) and σ j (R 22 ) σ k+j (A) for 1 i k and 1 j n k. σ k+1 (A) σ max (R 22 ) = R of 70
12 Rank revealing QR factorization Given A of size m n, consider the decomposition [ R11 R AP c = QR = Q 12 R 22 If R 22 2 is small, ]. (4) Q(:, 1 : k) forms an approximate orthogonal basis for the range of A, P c [ R 1 11 R 12 I a(:, j) = min(j,k) r ij Q(:, i) i=1 ran(a) span{q(:, 1),... Q(:, k)} span{q(:, 1),... Q(:, k)} ] is an approximate right null space of A. 12 of 70
13 Rank revealing QR factorization The factorization from equation (4) is rank revealing if 1 σ i(a) σ i (R 11 ), σ j(r 22 ) σ k+j (A) q 1(n, k), for 1 i k and 1 j min(m, n) k, where σ max (A) = σ 1 (A)... σ min (A) = σ n (A) It is strong rank revealing [Gu and Eisenstat, 1996] if in addition R 1 11 R 12 max q 2 (n, k) Gu and Eisenstat show that given k and f, there exists a P c such that q 1 (n, k) = 1 + f 2 k(n k) and q 2 (n, k) = f. Factorization computed in 4mnk (QRCP) plus O(mnk) flops. 13 of 70
14 QR with column pivoting [Businger and Golub, 1965] Idea: At first iteration, trailing columns decomposed into parallel part to first column (or e 1 ) and orthogonal part (in rows 2 : m). The column of maximum norm is the column with largest component orthogonal to the first column. Implementation: Find at each step of the QR factorization the column of maximum norm. Permute it into leading position. If rank(a) = k, at step k + 1 the maximum norm is 0. No need to compute the column norms at each step, but just update them since Q T v = w = [ w 1 w(2 : n) ], w(2 : n) 2 2 = v 2 2 w of 70
15 QR with column pivoting [Businger and Golub, 1965] Sketch of the algorithm column norm vector: colnrm(j) = A(:, j) 2, j = 1 : n. for j = 1 : n do Find column p of largest norm if colnrm[p] > ɛ then 1. Pivot: swap columns j and p in A and modify colnrm. 2. Compute Householder matrix H j s.t. H j A(j : m, j) = ± A(j : m, j) 2 e Update A(j : m, j + 1 : n) = H j A(j : m, j + 1 : n). 4. Norm downdate colnrm(j + 1 : n) 2 = A(j, j + 1 : n) 2. else Break end if end for If algorithm stops after k steps σ max (R 22 ) n k max R 22(:, j) 2 n kɛ 1 j n k 15 of 70
16 Strong RRQR [Gu and Eisenstat, 1996] Since k n k det(r 11 ) = σ i (R 11 ) = det(a T A)/ σ i (R 22 ) i=1 i=1 a stron RRQR is related to a large det(r 11 ). The following algorithm interchanges columns that increase det(r 11 ), given f and k. Compute a strong RRQR factorization, given k: Compute AΠ = QR by using QRCP while there exist i and j such that det( R 11 )/det(r 11 ) > f, where R 11 = R(1 : k, 1 : k), Π i,j+k permutes columns i and j + k, RΠ i,j+k = Q R, R 11 = R(1 : k, 1 : k) do Find i and j Compute RΠ i,j+k = Q R and Π = ΠΠ i,j+k end while 16 of 70
17 Strong RRQR (contd) It can be shown that det( R 11 ) det(r 11 ) = (R 1 11 R 12) 2 i,j + ω2 i (R 11 ) χ 2 j (R 22) (5) for any 1 i k and 1 j n k (the 2-norm of the j-th column of A is χ j (A), and the 2-norm of the j-th row of A 1 is ω j (A) ). Compute a strong RRQR factorization, given k: Compute AΠ = QR by using QRCP (R 1 while max 1 i k,1 j n k 11 R 2 12) i,j + ω2 i (R 11 ) χ 2 j (R 22) > f do (R 1 Find i and j such that 11 R 2 12) + i,j ω2 i (R 11 ) χ 2 j (R 22) > f Compute RΠ i,j+k = Q R and Π = ΠΠ i,j+k end while 17 of 70
18 Strong RRQR (contd) det(r 11 ) strictly increases with every permutation, no permutation repeats, hence there is a finite number of permutations to be performed. 18 of 70
19 Strong RRQR (contd) Theorem [Gu and Eisenstat, 1996] If the QR factorization with column pivoting as in equation (4) satisfies inequality (R 1 11 R 12) 2 i,j + ω2 i (R 11 ) χ 2 j (R 22) < f for any 1 i k and 1 j n k, then 1 σ i(a) σ i (R 11 ), σ j(r 22 ) σ k+j (A) 1 + f 2 k(n k), for any 1 i k and 1 j min(m, n) k. 19 of 70
20 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
21 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
22 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
23 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
24 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
25 Tournament pivoting [Demmel et al., 2015] One step of CA RRQR, tournament pivoting used to select k columns Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Permute A ji in leading positions, compute QR with no pivoting ( ) R11 AP c1 = Q 1 20 of 70
26 Select k columns from a tall and skinny matrix Given W of size m 2k, m >> k, k columns are selected as: W = QR 02 using TSQR R 02 P c = Q 2 R 2 using QRCP Return WP c (:, 1 : k) 21 of 70
27 Reduction trees Any shape of reduction tree can be used during CA RRQR, depending on the underlying architecture. Binary tree: A 00 A 10 A 20 A 30 f (A 00 ) f (A 10 ) f (A 20 ) f (A 30 ) f (A 01 ) f (A 11 ) f (A 02 ) Flat tree: A 00 A 10 A 20 A 30 f (A 00 ) f (A 01 ) f (A 02 ) f (A 03 ) Notation: at each node of the reduction tree, f (A ij ) returns the first b columns obtained after performing (strong) RRQR of A ij. 22 of 70
28 Rank revealing properties of CA-RRQR It is shown in [Demmel et al., 2015] that the column permutation computed by CA-RRQR satisfies χ 2 ( j R 1 11 R 12) + (χj (R 22 ) /σ min (R 11 )) 2 FTP, 2 for j = 1,..., n k. (6) where F TP depends on k, f, n, the shape of reduction tree used during tournament pivoting, and the number of iterations of CARRQR. 23 of 70
29 CA-RRQR - bounds for one tournament Selecting k columns by using tournament pivoting reveals the rank of A with the following bounds: 1 σ i(a) σ i (R 11 ), σ j(r 22 ) σ k+j (A) 1 + FTP 2 (n k), R 1 11 R 12 max F TP Binary tree of depth log 2 (n/k), F TP 1 2k (n/k) log 2( 2fk). (7) The upper bound is a decreasing function of k when k > Flat tree of depth n/k, n/( 2f ). F TP 1 2k ( 2fk ) n/k. (8) 24 of 70
30 Cost of CA-RRQR Cost of CA-RRQR vs QR with column pivoting n n matrix on P P processor grid, block size k Flops : 4n 3 /P + O(n 2 klogp/ P) vs (4/3)n 3 /P Bandwidth : O(n 2 log P/ P) vs same Latency : O(n log P/k) vs O(n log P) Communication optimal, modulo polylogarithmic factors, by choosing k = 1 n 2log 2 P P 25 of 70
31 Numerical results Stability close to QRCP for many tested matrices. Absolute value of diagonals of R, L referred to as R-values, L-values. Methods compared RRQR: QR with column pivoting CA-RRQR-B with tournament pivoting based on binary tree CA-RRQR-F with tournament pivoting based on flat tree SVD 26 of 70
32 Numerical results - devil s stairs Devil s stairs (Stewart), a matrix with multiple gaps in the singular values. Matlab code: Length = 20; s = zeros(n,1); Nst = floor(n/length); for i = 1 : Nst do s(1+length*(i-1):length*i) = -0.6*(i-1); end for s(length Nst : end) = 0.6 (Nst 1); s = 10. s; A = orth(rand(n)) * diag(s) * orth(randn(n)); QLP decomposition (Stewart) AP c1 = Q 1 R 1 using ca rrqr R T 1 = Q 2 R x QRCP CARRQR B 9 CARRQR F SVD 8 R values & singular values Trustworthiness check 2 1 i Column No. 27 of 70
33 Numerical results - devil s stairs Devil s stairs (Stewart), a matrix with multiple gaps in the singular values. Matlab code: Length = 20; s = zeros(n,1); Nst = floor(n/length); for i = 1 : Nst do s(1+length*(i-1):length*i) = -0.6*(i-1); end for s(length Nst : end) = 0.6 (Nst 1); s = 10. s; A = orth(rand(n)) * diag(s) * orth(randn(n)); QLP decomposition (Stewart) AP c1 = Q 1 R 1 using ca rrqr R T 1 = Q 2 R x QRCP CARRQR B 9 CARRQR F SVD QRCP + QLP CARRQR B + QLP CARRQR F + QLP SVD 7 R values & singular values Trustworthiness check L values & singular values i Column No Column No. i 27 of 70
34 Numerical results (contd) R values, singular values & trustworthiness check QRCP CARRQR B CARRQR F SVD R values, singular values & trustworthiness check QRCP CARRQR B CARRQR F SVD Column No. i Column No. i Left: exponent - exponential Distribution, σ 1 = 1, σ i = α i 1 (i = 2,..., n), α = 10 1/11 [Bischof, 1991] Right: shaw - 1D image restoration model [Hansen, 2007] ɛ min{ (AΠ 0 )(:, i) 2, (AΠ 1 )(:, i) 2, (AΠ 2 )(:, i) 2 } (9) ɛ max{ (AΠ 0 )(:, i) 2, (AΠ 1 )(:, i) 2, (AΠ 2 )(:, i) 2 } (10) where Π j (j = 0, 1, 2) are the permutation matrices obtained by QRCP, CARRQR-B, and CARRQR-F, and ɛ is the machine precision. 28 of 70
35 Numerical results - a set of 18 matrices Ratios R(i, i) /σ i (R), for QRCP (top plot), CARRQR-B (second plot), and CARRQR-F (third plot). The number along x-axis represents the index of test matrices. 29 of 70
36 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization with column and row tournament pivoting Experimental results, LU CRTP Randomized algorithms for low rank approximation 30 of 70
37 LU versus QR - filled graph G + (A) Consider A is SPD and A = LL T Given G(A) = (V, E), G + (A) = (V, E + ) is defined as: there is an edge (i, j) G + (A) iff there is a path from i to j in G(A) going through lower numbered vertices. G(L + L T ) = G + (A), ignoring cancellations. Definition holds also for directed graphs (LU factorization). A = x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x L + L T = x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 31 of
38 LU versus QR Filled column intersection graph G + (A) Graph of the Cholesky factor of A T A G(R) G + (A) A T A can have many more nonzeros than A 32 of 70
39 LU versus QR Numerical stability Let ˆL and Û be the computed factors of the block LU factorization. Then ( ) ˆLÛ = A + E, E max c(n)ɛ A max + ˆL max Û max. (11) For partial pivoting, L max 1, U max 2 n A max In practice, U max n A max 33 of 70
40 Low rank approximation based on LU factorization Given desired rank k, the factorization has the form ) ( ) ) (Ā11 Ā 12 I (Ā11 Ā 12 P r AP c = = Ā 21 Ā 22 Ā 21 Ā 1 11 I S(Ā11), (12) where A R m n, Ā 11 R k,k, S(Ā11) = Ā22 Ā21Ā 1 11 Ā12. The rank-k approximation matrix à k is à k = ( I Ā 21 Ā 1 11 ) (Ā11 Ā 12 ) = (Ā11 Ā 21 ) Ā 1 ) 11 (Ā11 Ā 12. (13) Ā 1 11 is never formed, its factorization is used when Ãk is applied to a vector. In randomized algorithms, U = C + AR +, where C +, R + are Moore-Penrose generalized inverses. 34 of 70
41 Design space Non-exhaustive list for selecting k columns and rows: 1. Select k linearly independent columns of A (call result B), by using 1.1 (strong) QRCP/tournament pivoting using QR, 1.2 LU / tournament pivoting based on LU, with some form of pivoting (column, complete, rook), 1.3 randomization: premultiply X = ZA where random matrix Z is short and fat, then pick k rows from X T, by some method from 2) below, 1.4 tournament pivoting based on randomized algorithms to select columns at each step. 2. Select k linearly independent rows of B, by using 2.1 (strong) QRCP / tournament pivoting based on QR on B T, or on Q T, the rows of the thin Q factor of B, 2.2 LU / tournament pivoting based on LU, with pivoting (row, complete, rook) on B, 2.3 tournament pivoting based on randomized algorithms to select rows. 35 of 70
42 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
43 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
44 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
45 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
46 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
47 Select k cols using tournament pivoting Partition A = (A 1, A 2, A 3, A 4 ). Select k cols from each column block, by using QR with column pivoting At each level i of the tree At each node j do in parallel Let A v,i 1, A w,i 1 be the cols selected by the children of node j Select k cols from (A v,i 1, A w,i 1 ), by using QR with column pivoting Return columns in A ji 36 of 70
48 LU CRTP factorization - one block step One step of truncated block LU based on column/row tournament pivoting on matrix A of size m n: 1. Select k columns by using tournament pivoting, permute them in front, bounds for s.v. governed by q 1 (n, k) ( ) ( ) ( ) R11 R AP c = Q 12 Q11 Q = 12 R11 R 12 R 22 Q 21 Q 22 R Select k rows from (Q 11 ; Q 21 ) T of size m k by using tournament pivoting, ( ) Q11 Q12 P r Q = Q 21 Q22 such that Q 21 Q 1 11 max F TP and bounds for s.v. governed by q 2 (m, k). 37 of 70
49 Orthogonal matrices Given orthogonal matrix Q R m m and its partitioning ( ) Q11 Q 12 Q =, (14) Q 21 Q 22 the selection of k cols by tournament pivoting from (Q 11 ; Q 21 ) T leads to the factorization ( ) ( ) ( ) Q11 Q12 I Q11 Q12 P r Q = = Q 21 Q 22 Q Q I S( Q (15) 11) where S( Q 11 ) = Q 22 Q 21 Q 1 11 Q 12 = Q T of 70
50 Orthogonal matrices (contd) The factorization satisfies: P r Q = ( Q11 Q12 Q 21 Q22 ) = ( ) ( ) I Q11 Q12 Q Q I S( Q 11) (16) ρ j ( Q Q ) F TP, (17) 1 q 2 (m, k) σ i ( Q 11 ) 1, (18) σ min ( Q 11 ) = σ min ( Q 22 ) (19) for all 1 i k, 1 j m k, where ρ j (A) is the 2-norm of the j-th row of A, q 2 (m, k) = 1 + FTP 2 (m k). 39 of 70
51 Sketch of the proof P r AP c = = ) ( (Ā11 Ā 12 I = Ā 21 Ā 22 Ā 21 Ā 1 ( I Q Q I 11 I ) ( Q11 Q12 S( Q 11 ) ) ) (Ā11 Ā 12 S(Ā11) ) ( R11 R 12 R 22 ) (20) where Q Q = Ā21Ā 1 11, S(Ā11) = S( Q 11 )R 22 = Q T 22 R of 70
52 Sketch of the proof (contd) We obtain Ā 11 = Q 11 R 11, (21) S(Ā11) = S( Q T 11 )R 22 = Q 22 R 22. (22) σ i (A) σ i (Ā 11 ) σ min ( Q 11 )σ i (R 11 ) We also have that 1 q 1 (n, k)q 2 (m, k) σ i(a), σ k+j (A) σ j (S(Ā11)) = σ j (S( Q 11 )R 22 ) S( Q 11 ) 2 σ j (R 22 ) q 1 (n, k)q 2 (m, k)σ k+j (A), where q 1 (n, k) = 1 + FTP 2 (n k), q 2(m, k) = 1 + FTP 2 (m k). 41 of 70
53 LU CRTP factorization - bounds if rank = k Given A of size m n, one step of LU CRTP computes the decomposition ) ( ) ) (Ā11 Ā 12 I (Ā11 Ā 12 Ā = P r AP c = = Ā 21 Ā 22 Q 21 Q 1 (23) 11 I S(Ā 11) where Ā11 is of size k k and S(Ā11) = Ā22 Ā21Ā 1 11 Ā12 = Ā22 Q Q Ā12. (24) It satisfies the following properties: ρ l (Ā21Ā 1 11 ) = ρ l( Q 21 Q 1 11 ) F TP, (25) S(Ā11) max min((1 + F TP k) A max, F TP 1 + F 2 TP (m k)σ k(a)) 1 σ i(a) σ i (Ā11), σ j(s(ā11)) q(m, n, k), (26) σ k+j (A) for any 1 l m k, 1 i k, and 1 j min(m, n) k, q(m, n, k) = (1 + FTP 2 (n k)) (1 + F TP 2 (m k)). 42 of 70
54 Arithmetic complexity - arbitrary sparse matrices Let d i be the number of nonzeros in column i of A, nnz(a) = n i=1 d i. A is permuted such that d 1... d n. A = [A 00,..., A n/k,0 ] is partitioned into n/k blocks of columns. At first step of TP: Pick k cols from A 1 = [A 00, A 10 ] nnz(a 1 ) 2k 2k i=1 d i, flops QR (A 1 ) 8k 2 2k i=1 d i. At the second step of TP: Pick k cols from A 2 nnz(a 2 ) 2k 3k i=k+1 d i flops QR (A 2 ) 8k 2 3k i=k+1 d i Bounds attained when: A = of 70
55 Arithmetic complexity - arbitrary sparse matrices (2) nnz max (TP FT ) 4d n k 2 ( 2k nnz total (TP FT ) 2k d i + 4k i=1 3k ) n d i d i i=k+1 i=n 2k+1 n d i = 4nnz(A)k, i=1 flops(tp FT ) 16nnz(A)k 2, 44 of 70
56 Tournament pivoting for sparse matrices Arithmetic complexity A has arbitrary sparsity structure G(A T A) is an n 1/2 - separable graph flops(tp FT ) 16nnz(A)k 2 flops(tp FT ) O(nnz(A)k 3/2 ) flops(tp BT ) 8 nnz(a) k 2 log n flops(tp BT ) O( nnz(a) k 3/2 log n P k P k ) Randomized algorithm by Clarkson and Woodruff, STOC 13 Given n n matrix A, it computes LDW T, where D is k k such that with failure probability 1/10 A LDW T F (1 + ɛ) A A k F, A k is best rank-k approximation. The cost of this algorithm is flops O(nnz(A)) + nk 2 ɛ 4 log O(1) (nk 2 ɛ 4 ) Tournament pivoting is faster if ɛ 1 (nnz(a)/n) 1/4 or if ɛ = 0.1 and nnz(a)/n of 70
57 Tournament pivoting for sparse matrices Arithmetic complexity A has arbitrary sparsity structure G(A T A) is an n 1/2 - separable graph flops(tp FT ) 16nnz(A)k 2 flops(tp FT ) O(nnz(A)k 3/2 ) flops(tp BT ) 8 nnz(a) k 2 log n flops(tp BT ) O( nnz(a) k 3/2 log n P k P k ) Randomized algorithm by Clarkson and Woodruff, STOC 13 Given n n matrix A, it computes LDW T, where D is k k such that with failure probability 1/10 A LDW T F (1 + ɛ) A A k F, A k is best rank-k approximation. The cost of this algorithm is flops O(nnz(A)) + nk 2 ɛ 4 log O(1) (nk 2 ɛ 4 ) Tournament pivoting is faster if ɛ 1 (nnz(a)/n) 1/4 or if ɛ = 0.1 and nnz(a)/n of 70
58 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization with column and row tournament pivoting Experimental results, LU CRTP Randomized algorithms for low rank approximation 46 of 70
59 Numerical results Singular value 10 0 Evolution of singular values for exponential QRCP LU-CRQRCP LU-CRTP SVD Singular value 10 0 Evolution of singular values for foxgood QRCP LU-CRQRCP LU-CRTP SVD Index of singular values Index of singular values Left: exponent - exponential Distribution, σ 1 = 1, σ i = α i 1 (i = 2,..., n), α = 10 1/11 [Bischof, 1991] Right: foxgood - Severely ill-posed test problem of the 1st kind Fredholm integral equation used by Fox and Goodwin 47 of 70
60 Numerical results Here k = 16 and the factorization is truncated at K = 128 (bars) or K = 240 (red lines). LU CTP: Column tournament pivoting + partial pivoting All singular values smaller than machine precision, ɛ, are replaced by ɛ. The number along x-axis represents the index of test matrices. 48 of 70
61 Results for image of size Original image Rank-38 approx, SVD Singular value distribution Rank-38 approx, LUPP Rank-38 approx, LU CRTP Rank-75 approx, LU CRTP 49 of 70
62 Results for image of size Original image Rank-105 approx, SVD Singular value distribution Rank-105 approx, LUPP Rank-105 approx, LU CRTP Rank-209 approx, LU CRTP 50 of 70
63 Comparing nnz in the factors L, U versus Q, R Name/size Nnz Rank K Nnz QRCP/ Nnz LU CRTP/ A(:, 1 : K) Nnz LU CRTP Nnz LUPP gemat wang Rfdevice Parab fem Mac econ of 70
64 Performance results Selection of 256 columns by tournament pivoting Edison, Cray XC30 (NERSC): 2x12-core Intel Ivy Bridge (2.4 GHz) Tournament pivoting uses SPQR (T. Davis) + dgeqp3 (Lapack), time in secs Matrices: dimension at leaves on 32 procs Parab fem: Mac econ: Time Time leaves Number of MPI processes 2k cols 32procs SPQR + dgeqp3 Parab fem Mac econ of 70
65 More details on CA deterministic algorithms [Demmel et al., 2015] Communication avoiding rank revealing QR factorization with column pivoting Demmel, Grigori, Gu, Xiang, SIAM J. Matrix Analysis and Applications, Low rank approximation of a sparse matrix based on LU factorization with column and row tournament pivoting, with S. Cayrols and J. Demmel, Inria TR of 70
66 References (1) Bischof, C. H. (1991). A parallel QR factorization algorithm with controlled local pivoting. SIAM J. Sci. Stat. Comput., 12: Businger, P. A. and Golub, G. H. (1965). Linear least squares solutions by Householder transformations. Numer. Math., 7: Demmel, J., Grigori, L., Gu, M., and Xiang, H. (2015). Communication-avoiding rank-revealing qr decomposition. SIAM Journal on Matrix Analysis and its Applications, 36(1): Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1: Eisenstat, S. C. and Ipsen, I. C. F. (1995). Relative perturbation techniques for singular value problems. SIAM J. Numer. Anal., 32(6): Gu, M. and Eisenstat, S. C. (1996). Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM J. Sci. Comput., 17(4): Hansen, P. C. (2007). Regularization tools: A matlab package for analysis and solution of discrete ill-posed problems. Numerical Algorithms, (46): of 70
67 Results used in the proofs Interlacing property of singular values [Golub, Van Loan, 4th edition, page 487] Let A = [a 1... a n ] be a column partitioning of an m n matrix with m n. If A r = [a 1... a r ], then for r = 1 : n 1 σ 1 (A r+1 ) σ 1 (A r ) σ 2 (A r+1 )... σ r (A r+1 ) σ r (A r ) σ r+1 (A r+1 ). Given n n matrix B and n k matrix C, then ([Eisenstat and Ipsen, 1995], p. 1977) σ min (B)σ j (C) σ j (BC) σ max (B)σ j (C), j = 1,..., k. 55 of 70
Rank revealing factorizations, and low rank approximations
Rank revealing factorizations, and low rank approximations L. Grigori Inria Paris, UPMC February 2017 Plan Low rank matrix approximation Rank revealing QR factorization LU CRTP: Truncated LU factorization
More informationLow Rank Approximation of a Sparse Matrix Based on LU Factorization with Column and Row Tournament Pivoting
Low Rank Approximation of a Sparse Matrix Based on LU Factorization with Column and Row Tournament Pivoting James Demmel Laura Grigori Sebastien Cayrols Electrical Engineering and Computer Sciences University
More informationLow rank approximation of a sparse matrix based on LU factorization with column and row tournament pivoting
Low rank approximation of a sparse matrix based on LU factorization with column and row tournament pivoting Laura Grigori, Sebastien Cayrols, James W. Demmel To cite this version: Laura Grigori, Sebastien
More informationDense LU factorization and its error analysis
Dense LU factorization and its error analysis Laura Grigori INRIA and LJLL, UPMC February 2016 Plan Basis of floating point arithmetic and stability analysis Notation, results, proofs taken from [N.J.Higham,
More informationCommunication avoiding parallel algorithms for dense matrix factorizations
Communication avoiding parallel dense matrix factorizations 1/ 44 Communication avoiding parallel algorithms for dense matrix factorizations Edgar Solomonik Department of EECS, UC Berkeley October 2013
More information14.2 QR Factorization with Column Pivoting
page 531 Chapter 14 Special Topics Background Material Needed Vector and Matrix Norms (Section 25) Rounding Errors in Basic Floating Point Operations (Section 33 37) Forward Elimination and Back Substitution
More informationIntroduction to communication avoiding algorithms for direct methods of factorization in Linear Algebra
Introduction to communication avoiding algorithms for direct methods of factorization in Linear Algebra Laura Grigori Abstract Modern, massively parallel computers play a fundamental role in a large and
More informationLU factorization with Panel Rank Revealing Pivoting and its Communication Avoiding version
1 LU factorization with Panel Rank Revealing Pivoting and its Communication Avoiding version Amal Khabou Advisor: Laura Grigori Université Paris Sud 11, INRIA Saclay France SIAMPP12 February 17, 2012 2
More informationParallel Numerical Algorithms
Parallel Numerical Algorithms Chapter 6 Matrix Models Section 6.2 Low Rank Approximation Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign CS 554 / CSE 512 Edgar
More informationLU Factorization with Panel Rank Revealing Pivoting and Its Communication Avoiding Version
LU Factorization with Panel Rank Revealing Pivoting and Its Communication Avoiding Version Amal Khabou James Demmel Laura Grigori Ming Gu Electrical Engineering and Computer Sciences University of California
More informationAlgorithm 853: an Efficient Algorithm for Solving Rank-Deficient Least Squares Problems
Algorithm 853: an Efficient Algorithm for Solving Rank-Deficient Least Squares Problems LESLIE FOSTER and RAJESH KOMMU San Jose State University Existing routines, such as xgelsy or xgelsd in LAPACK, for
More informationSpectrum-Revealing Matrix Factorizations Theory and Algorithms
Spectrum-Revealing Matrix Factorizations Theory and Algorithms Ming Gu Department of Mathematics University of California, Berkeley April 5, 2016 Joint work with D. Anderson, J. Deursch, C. Melgaard, J.
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationSubset Selection. Deterministic vs. Randomized. Ilse Ipsen. North Carolina State University. Joint work with: Stan Eisenstat, Yale
Subset Selection Deterministic vs. Randomized Ilse Ipsen North Carolina State University Joint work with: Stan Eisenstat, Yale Mary Beth Broadbent, Martin Brown, Kevin Penner Subset Selection Given: real
More informationc 2013 Society for Industrial and Applied Mathematics
SIAM J. MATRIX ANAL. APPL. Vol. 34, No. 3, pp. 1401 1429 c 2013 Society for Industrial and Applied Mathematics LU FACTORIZATION WITH PANEL RANK REVEALING PIVOTING AND ITS COMMUNICATION AVOIDING VERSION
More informationSparse linear solvers
Sparse linear solvers Laura Grigori ALPINES INRIA and LJLL, UPMC On sabbatical at UC Berkeley March 2015 Plan Sparse linear solvers Sparse matrices and graphs Classes of linear solvers Sparse Cholesky
More informationRank Revealing QR factorization. F. Guyomarc h, D. Mezher and B. Philippe
Rank Revealing QR factorization F. Guyomarc h, D. Mezher and B. Philippe 1 Outline Introduction Classical Algorithms Full matrices Sparse matrices Rank-Revealing QR Conclusion CSDA 2005, Cyprus 2 Situation
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationETNA Kent State University
C 8 Electronic Transactions on Numerical Analysis. Volume 17, pp. 76-2, 2004. Copyright 2004,. ISSN 1068-613. etnamcs.kent.edu STRONG RANK REVEALING CHOLESKY FACTORIZATION M. GU AND L. MIRANIAN Abstract.
More informationSUBSET SELECTION ALGORITHMS: RANDOMIZED VS. DETERMINISTIC
SUBSET SELECTION ALGORITHMS: RANDOMIZED VS. DETERMINISTIC MARY E. BROADBENT, MARTIN BROWN, AND KEVIN PENNER Advisors: Ilse Ipsen 1 and Rizwana Rehman 2 Abstract. Subset selection is a method for selecting
More informationOrthogonalization and least squares methods
Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Logistics Notes for 2016-09-14 1. There was a goof in HW 2, problem 1 (now fixed) please re-download if you have already started looking at it. 2. CS colloquium (4:15 in Gates G01) this Thurs is Margaret
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More informationSubset Selection. Ilse Ipsen. North Carolina State University, USA
Subset Selection Ilse Ipsen North Carolina State University, USA Subset Selection Given: real or complex matrix A integer k Determine permutation matrix P so that AP = ( A 1 }{{} k A 2 ) Important columns
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More information1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )
Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical
More informationc 2015 Society for Industrial and Applied Mathematics
SIAM J MATRIX ANAL APPL Vol 36, No, pp 55 89 c 05 Society for Industrial and Applied Mathematics COMMUNICATION AVOIDING RANK REVEALING QR FACTORIZATION WITH COLUMN PIVOTING JAMES W DEMMEL, LAURA GRIGORI,
More informationIntroduction to Numerical Linear Algebra II
Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49 Overview We will cover this material in
More informationLecture 12 (Tue, Mar 5) Gaussian elimination and LU factorization (II)
Math 59 Lecture 2 (Tue Mar 5) Gaussian elimination and LU factorization (II) 2 Gaussian elimination - LU factorization For a general n n matrix A the Gaussian elimination produces an LU factorization if
More informationApplied Numerical Linear Algebra. Lecture 8
Applied Numerical Linear Algebra. Lecture 8 1/ 45 Perturbation Theory for the Least Squares Problem When A is not square, we define its condition number with respect to the 2-norm to be k 2 (A) σ max (A)/σ
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 13
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 208 LECTURE 3 need for pivoting we saw that under proper circumstances, we can write A LU where 0 0 0 u u 2 u n l 2 0 0 0 u 22 u 2n L l 3 l 32, U 0 0 0 l n l
More informationCommunication-avoiding LU and QR factorizations for multicore architectures
Communication-avoiding LU and QR factorizations for multicore architectures DONFACK Simplice INRIA Saclay Joint work with Laura Grigori INRIA Saclay Alok Kumar Gupta BCCS,Norway-5075 16th April 2010 Communication-avoiding
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationLow-Rank Approximations, Random Sampling and Subspace Iteration
Modified from Talk in 2012 For RNLA study group, March 2015 Ming Gu, UC Berkeley mgu@math.berkeley.edu Low-Rank Approximations, Random Sampling and Subspace Iteration Content! Approaches for low-rank matrix
More informationSparse BLAS-3 Reduction
Sparse BLAS-3 Reduction to Banded Upper Triangular (Spar3Bnd) Gary Howell, HPC/OIT NC State University gary howell@ncsu.edu Sparse BLAS-3 Reduction p.1/27 Acknowledgements James Demmel, Gene Golub, Franc
More informationHybrid static/dynamic scheduling for already optimized dense matrix factorization. Joint Laboratory for Petascale Computing, INRIA-UIUC
Hybrid static/dynamic scheduling for already optimized dense matrix factorization Simplice Donfack, Laura Grigori, INRIA, France Bill Gropp, Vivek Kale UIUC, USA Joint Laboratory for Petascale Computing,
More informationInstitute for Advanced Computer Studies. Department of Computer Science. Two Algorithms for the The Ecient Computation of
University of Maryland Institute for Advanced Computer Studies Department of Computer Science College Park TR{98{12 TR{3875 Two Algorithms for the The Ecient Computation of Truncated Pivoted QR Approximations
More informationThis can be accomplished by left matrix multiplication as follows: I
1 Numerical Linear Algebra 11 The LU Factorization Recall from linear algebra that Gaussian elimination is a method for solving linear systems of the form Ax = b, where A R m n and bran(a) In this method
More informationReview Questions REVIEW QUESTIONS 71
REVIEW QUESTIONS 71 MATLAB, is [42]. For a comprehensive treatment of error analysis and perturbation theory for linear systems and many other problems in linear algebra, see [126, 241]. An overview of
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More information5.6. PSEUDOINVERSES 101. A H w.
5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and
More informationLecture Note 2: The Gaussian Elimination and LU Decomposition
MATH 5330: Computational Methods of Linear Algebra Lecture Note 2: The Gaussian Elimination and LU Decomposition The Gaussian elimination Xianyi Zeng Department of Mathematical Sciences, UTEP The method
More informationDirect Methods for Solving Linear Systems. Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le
Direct Methods for Solving Linear Systems Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le 1 Overview General Linear Systems Gaussian Elimination Triangular Systems The LU Factorization
More informationOrthonormal Transformations and Least Squares
Orthonormal Transformations and Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 30, 2009 Applications of Qx with Q T Q = I 1. solving
More informationDirect solution methods for sparse matrices. p. 1/49
Direct solution methods for sparse matrices p. 1/49 p. 2/49 Direct solution methods for sparse matrices Solve Ax = b, where A(n n). (1) Factorize A = LU, L lower-triangular, U upper-triangular. (2) Solve
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More informationCALU: A Communication Optimal LU Factorization Algorithm
CALU: A Communication Optimal LU Factorization Algorithm James Demmel Laura Grigori Hua Xiang Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-010-9
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationNumerical Linear Algebra
Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix
More informationETNA Kent State University
Electronic Transactions on Numerical Analysis. Volume 17, pp. 76-92, 2004. Copyright 2004,. ISSN 1068-9613. ETNA STRONG RANK REVEALING CHOLESKY FACTORIZATION M. GU AND L. MIRANIAN Ý Abstract. For any symmetric
More informationLAPACK-Style Codes for Pivoted Cholesky and QR Updating. Hammarling, Sven and Higham, Nicholas J. and Lucas, Craig. MIMS EPrint: 2006.
LAPACK-Style Codes for Pivoted Cholesky and QR Updating Hammarling, Sven and Higham, Nicholas J. and Lucas, Craig 2007 MIMS EPrint: 2006.385 Manchester Institute for Mathematical Sciences School of Mathematics
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More informationLAPACK-Style Codes for Pivoted Cholesky and QR Updating
LAPACK-Style Codes for Pivoted Cholesky and QR Updating Sven Hammarling 1, Nicholas J. Higham 2, and Craig Lucas 3 1 NAG Ltd.,Wilkinson House, Jordan Hill Road, Oxford, OX2 8DR, England, sven@nag.co.uk,
More informationPreconditioned Parallel Block Jacobi SVD Algorithm
Parallel Numerics 5, 15-24 M. Vajteršic, R. Trobec, P. Zinterhof, A. Uhl (Eds.) Chapter 2: Matrix Algebra ISBN 961-633-67-8 Preconditioned Parallel Block Jacobi SVD Algorithm Gabriel Okša 1, Marián Vajteršic
More informationSUBSPACE ITERATION RANDOMIZATION AND SINGULAR VALUE PROBLEMS
SUBSPACE ITERATION RANDOMIZATION AND SINGULAR VALUE PROBLEMS M. GU Abstract. A classical problem in matrix computations is the efficient and reliable approximation of a given matrix by a matrix of lower
More informationLU Factorization. LU Decomposition. LU Decomposition. LU Decomposition: Motivation A = LU
LU Factorization To further improve the efficiency of solving linear systems Factorizations of matrix A : LU and QR LU Factorization Methods: Using basic Gaussian Elimination (GE) Factorization of Tridiagonal
More informationMulti-Linear Mappings, SVD, HOSVD, and the Numerical Solution of Ill-Conditioned Tensor Least Squares Problems
Multi-Linear Mappings, SVD, HOSVD, and the Numerical Solution of Ill-Conditioned Tensor Least Squares Problems Lars Eldén Department of Mathematics, Linköping University 1 April 2005 ERCIM April 2005 Multi-Linear
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationRandomized Numerical Linear Algebra: Review and Progresses
ized ized SVD ized : Review and Progresses Zhihua Department of Computer Science and Engineering Shanghai Jiao Tong University The 12th China Workshop on Machine Learning and Applications Xi an, November
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationGaussian Elimination for Linear Systems
Gaussian Elimination for Linear Systems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 3, 2011 1/56 Outline 1 Elementary matrices 2 LR-factorization 3 Gaussian elimination
More informationMatrix Factorization and Analysis
Chapter 7 Matrix Factorization and Analysis Matrix factorizations are an important part of the practice and analysis of signal processing. They are at the heart of many signal-processing algorithms. Their
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 2 Systems of Linear Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationWhy the QR Factorization can be more Accurate than the SVD
Why the QR Factorization can be more Accurate than the SVD Leslie V. Foster Department of Mathematics San Jose State University San Jose, CA 95192 foster@math.sjsu.edu May 10, 2004 Problem: or Ax = b for
More informationLecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 2. Systems of Linear Equations
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T. Heath Chapter 2 Systems of Linear Equations Copyright c 2001. Reproduction permitted only for noncommercial,
More informationV C V L T I 0 C V B 1 V T 0 I. l nk
Multifrontal Method Kailai Xu September 16, 2017 Main observation. Consider the LDL T decomposition of a SPD matrix [ ] [ ] [ ] [ ] B V T L 0 I 0 L T L A = = 1 V T V C V L T I 0 C V B 1 V T, 0 I where
More informationarxiv: v1 [math.na] 23 Nov 2018
Randomized QLP algorithm and error analysis Nianci Wu a, Hua Xiang a, a School of Mathematics and Statistics, Wuhan University, Wuhan 43007, PR China. arxiv:1811.09334v1 [math.na] 3 Nov 018 Abstract In
More informationSYMBOLIC AND EXACT STRUCTURE PREDICTION FOR SPARSE GAUSSIAN ELIMINATION WITH PARTIAL PIVOTING
SYMBOLIC AND EXACT STRUCTURE PREDICTION FOR SPARSE GAUSSIAN ELIMINATION WITH PARTIAL PIVOTING LAURA GRIGORI, JOHN R. GILBERT, AND MICHEL COSNARD Abstract. In this paper we consider two structure prediction
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationNumerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??
Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationPerformance of Random Sampling for Computing Low-rank Approximations of a Dense Matrix on GPUs
Performance of Random Sampling for Computing Low-rank Approximations of a Dense Matrix on GPUs Théo Mary, Ichitaro Yamazaki, Jakub Kurzak, Piotr Luszczek, Stanimire Tomov, Jack Dongarra presenter 1 Low-Rank
More informationIMPROVING THE PERFORMANCE OF SPARSE LU MATRIX FACTORIZATION USING A SUPERNODAL ALGORITHM
IMPROVING THE PERFORMANCE OF SPARSE LU MATRIX FACTORIZATION USING A SUPERNODAL ALGORITHM Bogdan OANCEA PhD, Associate Professor, Artife University, Bucharest, Romania E-mail: oanceab@ie.ase.ro Abstract:
More informationDominant feature extraction
Dominant feature extraction Francqui Lecture 7-5-200 Paul Van Dooren Université catholique de Louvain CESAME, Louvain-la-Neuve, Belgium Goal of this lecture Develop basic ideas for large scale dense matrices
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Logistics Notes for 2016-08-26 1. Our enrollment is at 50, and there are still a few students who want to get in. We only have 50 seats in the room, and I cannot increase the cap further. So if you are
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationIndex. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)
page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation
More informationPRECONDITIONING IN THE PARALLEL BLOCK-JACOBI SVD ALGORITHM
Proceedings of ALGORITMY 25 pp. 22 211 PRECONDITIONING IN THE PARALLEL BLOCK-JACOBI SVD ALGORITHM GABRIEL OKŠA AND MARIÁN VAJTERŠIC Abstract. One way, how to speed up the computation of the singular value
More informationLeast Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo
Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 26, 2010 Linear system Linear system Ax = b, A C m,n, b C m, x C n. under-determined
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 5: Numerical Linear Algebra Cho-Jui Hsieh UC Davis April 20, 2017 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationMath Camp II. Basic Linear Algebra. Yiqing Xu. Aug 26, 2014 MIT
Math Camp II Basic Linear Algebra Yiqing Xu MIT Aug 26, 2014 1 Solving Systems of Linear Equations 2 Vectors and Vector Spaces 3 Matrices 4 Least Squares Systems of Linear Equations Definition A linear
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More informationSome Notes on Least Squares, QR-factorization, SVD and Fitting
Department of Engineering Sciences and Mathematics January 3, 013 Ove Edlund C000M - Numerical Analysis Some Notes on Least Squares, QR-factorization, SVD and Fitting Contents 1 Introduction 1 The Least
More informationBLAS: Basic Linear Algebra Subroutines Analysis of the Matrix-Vector-Product Analysis of Matrix-Matrix Product
Level-1 BLAS: SAXPY BLAS-Notation: S single precision (D for double, C for complex) A α scalar X vector P plus operation Y vector SAXPY: y = αx + y Vectorization of SAXPY (αx + y) by pipelining: page 8
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 1. qr and complete orthogonal factorization poor man s svd can solve many problems on the svd list using either of these factorizations but they
More informationEfficient Preprocessing in the Parallel Block-Jacobi SVD Algorithm. Technical Report June Department of Scientific Computing
Efficient Preprocessing in the Parallel Block-Jacobi SVD Algorithm Gabriel Okša a Marián Vajteršic a Mathematical Institute, Department of Informatics, Slovak Academy of Sciences, Bratislava, Slovak Republic
More informationAx = b. Systems of Linear Equations. Lecture Notes to Accompany. Given m n matrix A and m-vector b, find unknown n-vector x satisfying
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T Heath Chapter Systems of Linear Equations Systems of Linear Equations Given m n matrix A and m-vector
More informationLU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b
AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization
More informationSince the determinant of a diagonal matrix is the product of its diagonal elements it is trivial to see that det(a) = α 2. = max. A 1 x.
APPM 4720/5720 Problem Set 2 Solutions This assignment is due at the start of class on Wednesday, February 9th. Minimal credit will be given for incomplete solutions or solutions that do not provide details
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationNumerical Linear Algebra And Its Applications
Numerical Linear Algebra And Its Applications Xiao-Qing JIN 1 Yi-Min WEI 2 August 29, 2008 1 Department of Mathematics, University of Macau, Macau, P. R. China. 2 Department of Mathematics, Fudan University,
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME MS&E 38 (CME 338 Large-Scale Numerical Optimization Course description Instructor: Michael Saunders Spring 28 Notes : Review The course teaches
More informationRandomized algorithms for the approximation of matrices
Randomized algorithms for the approximation of matrices Luis Rademacher The Ohio State University Computer Science and Engineering (joint work with Amit Deshpande, Santosh Vempala, Grant Wang) Two topics
More informationBindel, Fall 2009 Matrix Computations (CS 6210) Week 8: Friday, Oct 17
Logistics Week 8: Friday, Oct 17 1. HW 3 errata: in Problem 1, I meant to say p i < i, not that p i is strictly ascending my apologies. You would want p i > i if you were simply forming the matrices and
More informationarxiv: v2 [math.na] 14 Mar 2018
Low-Rank Matrix Approximations with Flip-Flop Spectrum-Revealing QR Factorization arxiv:1803.01982v2 [math.na] 14 Mar 2018 Yuehua Feng Jianwei Xiao Ming Gu March 16, 2018 Abstract We present Flip-Flop
More information