From Matrix to Tensor. Charles F. Van Loan
|
|
- Gilbert Garrett
- 5 years ago
- Views:
Transcription
1 From Matrix to Tensor Charles F. Van Loan Department of Computer Science January 28, 2016 From Matrix to Tensor From Tensor To Matrix 1 / 68
2 What is a Tensor? Instead of just A(i, j) it s A(i, j, k) or A(i 1, i 2,..., i d ) From Matrix to Tensor From Tensor To Matrix 2 / 68
3 Where Might They Come From? Discretization A(i, j, k, l) might house the value of f (w, x, y, z) at (w, x, y, z) = (w i, x j, y k, z l ). High-Dimension Evaluations Given a basis {φ i (r)} n i=1 A(p, q, r, s) = Multiway Analysis R 3 φ p (r 1 )φ q (r 1 )φ r (r 2 )φ s (r 2 ) R 3 r 1 r 2 dr 1 dr 2. A(i, j, k, l) is a value that captures an interaction between four variables/factors. From Matrix to Tensor From Tensor To Matrix 3 / 68
4 You May Have Seen them Before... Here is a 3x3 block matrix with 2x2 blocks: A = a 11 a 12 a 13 a 14 a 15 a 16 a 21 a 22 a 23 a 24 a 25 a 26 a 31 a 32 a 33 a 34 a 35 a 36 a 41 a 42 a 43 a 44 a 45 a 46 a 51 a 52 a 53 a 54 a 55 a 56 a 61 a 62 a 63 a 64 a 65 a 66 This is a reshaping of a tensor: Matrix entry a 45 is the (2,1) entry of the (2,3) block. Matrix entry a 45 is A(2, 3, 2, 1). From Matrix to Tensor From Tensor To Matrix 4 / 68
5 A Tensor Has Parts A matrix has columns and rows. A tensor has fibers. A fiber of a tensor A is a vector obtained by fixing all but one A s indices. Given A = A(1:3, 1:5, 1:4, 1:7), here is a mode-2 fiber: A(2, 1:5, 4, 6) = This is the (2,4,6) mode-2 fiber. A(2, 1, 4, 6) A(2, 2, 4, 6) A(2, 3, 4, 6) A(2, 4, 4, 6) A(2, 5, 4, 6) From Matrix to Tensor From Tensor To Matrix 5 / 68
6 Fibers Can Be Assembled Into a Matrix The mode-1, mode-2, and mode-3 unfoldings of A IR : A (1) = a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a 432 (1,1) (2,1) (3,1) (1,2) (2,2) (3,2) A (2) = a 111 a 211 a 311 a 411 a 112 a 212 a 312 a 412 a 121 a 221 a 321 a 421 a 122 a 222 a 322 a 422 a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432 A (3) = (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) [ a111 a 211 a 311 a 411 a 121 a 221 a 321 a 421 a 131 a 231 a 331 a 431 a 112 a 212 a 312 a 412 a 122 a 222 a 322 a 422 a 132 a 232 a 332 a 432 (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) (1,3) (2,3) (3,3) (4,3) ] From Matrix to Tensor From Tensor To Matrix 6 / 68
7 There are Many Ways to Unfold a Given Tensor Here is one way to unfold A(1:2, 1:3, 1:2, 1:2, 1:3): B = (1,1) (2,1) (1,2) (2,2) (1,3) (2,3) a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a (1,1,1) (2,1,1) (1,2,1) (2,2,1) (1,3,1) (2,3,1) (1,1,2) (2,1,2) (1,2,2) (2,2,2) (1,3,2) (2,3,2) With the Matlab Tensor Toolbox: B = tenmat(a,[1 2 3],[4 5]) From Matrix to Tensor From Tensor To Matrix 7 / 68
8 There are Many Ways to Unfold a Given Tensor tenmat(a,[1 2 3],[4 5]) tenmat(a,[1 2 4],[3 5]) tenmat(a,[1 2 5],[4 5]) tenmat(a,[1 3 4],[2 5]) tenmat(a,[1 3 5],[2 5]) tenmat(a,[1 4 5],[2 3]) tenmat(a,[2 3 4],[1 5]) tenmat(a,[2 3 5],[1 4]) tenmat(a,[2 4 5],[1 3]) tenmat(a,[3 4 5],[1 2]) tenmat(a,[4 5],[1 2 3]) tenmat(a,[3,5],[1 2 4]) tenmat(a,[4 5],[1 2 5]) tenmat(a,[2 5],[1 3 4]) tenmat(a,[2 5],[1 3 5]) tenmat(a,[2 3],[1 4 5]) tenmat(a,[1 5],[2 3 4]) tenmat(a,[1 4],[2 3 5]) tenmat(a,[1 3],[2 4 5]) tenmat(a,[1 2],[3 4 5]) tenmat(a,[1],[ ]) tenmat(a,[2],[ ]) tenmat(a,[3],[ ]) tenmat(a,[4],[ ]) tenmat(a,[5],[ ]) tenmat(a,[ ],[1]) tenmat(a,[ ],[2]) tenmat(a,[ ],[3]) tenmat(a,[ ],[4]) tenmat(a,[ ],[5]) Choice makes life complicated... From Matrix to Tensor From Tensor To Matrix 8 / 68
9 Paradigm for Much of Tensor Computations To say something about a tensor A: 1. Thoughtfully unfold tensor A into a matrix A. 2. Use classical matrix computations to discover something interesting/useful about matrix A. 3. Map your insights back to tensor A. Computing (parts of) decompositions is how we do this in classical matrix computations. From Matrix to Tensor From Tensor To Matrix 9 / 68
10 Matrix Factorizations and Decompositions A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T A = ULV It s T PAQ T = LUa A = Language UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR From Matrix to Tensor From Tensor To Matrix 10 / 68
11 Matrix Factorizations and Decompositions A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T A = ULV It s T PAQ T = LUa A = Language UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR From Matrix to Tensor From Tensor To Matrix 11 / 68
12 The Singular Value Decomposition Perhaps the most versatile and important of all the different matrix decompositions is the SVD: [ a11 a 12 a 21 a 22 ] = [ = σ 1 [ c 1 s 1 s 1 c 1 c 1 s 1 ] [ ] [ σ1 0 0 σ 2 c 2 s 2 ] [ c 2 s 2 s 2 c 2 ] T + σ 2 [ s1 c 1 ] T ] [ s2 c 2 ] T = σ 1 [ c 1 s 1 ] [c2 s 2 ] + σ 2 [ s1 c 1 ] [s2 c 2 ] where c s2 1 = 1 and c2 2 + s2 2 = 1. This is a very special sum of rank-1 matrices. From Matrix to Tensor From Tensor To Matrix 12 / 68
13 Rank-1 Matrices: You have Seen Them Before T = From Matrix to Tensor From Tensor To Matrix 13 / 68
14 Rank-1 Matrices: They Are Data Sparse T = = vv T v = From Matrix to Tensor From Tensor To Matrix 14 / 68
15 The Matrix SVD Expresses the matrix as a special sum of rank-1 matrices. If A IR n n then A = n σ k u k vk T k=1 Here σ 1 σ 2 σ r > σ r+1 = = σ n = 0 and U = [u 1 u 2 u n ] V = [v 1 v 2 v n ] have columns that are mutually orthogonal. From Matrix to Tensor From Tensor To Matrix 15 / 68
16 The Matrix SVD: Nearness Problems Expresses the matrix as a special sum of rank-1 matrices. If A IR n n then n A = σ k u k vk T k=1 Here σ 1 σ 2 σ r > σ r+1 = = σ n = 0 and U = [u 1 u 2 u n ] V = [v 1 v 2 v n ] have columns that are mutually orthogonal. That s how far A is from being rank deficient. From Matrix to Tensor From Tensor To Matrix 16 / 68
17 The Matrix SVD: Data Sparse Approximation Expresses the matrix as a special sum of rank-1 matrices. If A IR n n then A r k=1 σ k u k v T k = A r Here σ 1 σ 2 σ r > σ r+1 = = σ n = 0 and U = [u 1 u 2 u n ] V = [v 1 v 2 v n ] have columns that are mutually orthogonal. That s the closest matrix to A that has rank r. If r << n, then that is a data sparse approximation of A because O(n r) << O(n 2 ). From Matrix to Tensor From Tensor To Matrix 17 / 68
18 There is a New Definition of Big In Matrix Computations, to say that A IR n 1 n 2 that both n 1 and n 2 are big. E.g., is big is to say n 1 = n 2 = In Tensor Computations, to say that A IR n 1 n d is big is to say that n 1 n 2 n d is big and this need not require big n k. E.g. n 1 = n 2 = = n 1000 = 2. From Matrix to Tensor From Tensor To Matrix 18 / 68
19 Why Data Sparse Tensor Approximation is Important 1. If you want to see this Matrix-Based Scientific Computation Tensor-Based Scientific Computation you will need tensor algorithms that scale with d. 2. This requires a framework for low-rank tensor approximation. 3. This requires some kind of tensor-level SVD. From Matrix to Tensor From Tensor To Matrix 19 / 68
20 What is a Rank-1 Tensor? Think Matrix First This: [ ] r11 r 12 R = = fg T = r 21 r 22 [ f1 f 2 ] [ f1 g 1 f 1 g 2 [g 1 g 2 ] = f 2 g 1 f 2 g 2 ] Is the same as this: Is the same as this: vec(r) = vec(r) = r 11 r 21 r 12 r 22 r 11 r 21 r 12 r 22 = = [ g1 g 2 g 1 f 1 g 1 f 2 g 2 f 1 g 2 f 2 ] [ f1 f 2 ] From Matrix to Tensor From Tensor To Matrix 20 / 68
21 The Kronecker Product of Vectors x y = x 1 x 2 x 3 [ y1 y 2 ] = x 1 y 1 x 1 y 2 x 2 y 1 x 2 y 2 x 3 y 1 x 3 y 2 = x 1 y x 2 y x 3 y From Matrix to Tensor From Tensor To Matrix 21 / 68
22 So What is a Rank-1 Tensor? R IR is rank-1 if there exist f, g, h IR 2 such that vec(r) = r 111 r 211 r 121 r 221 r 112 r 212 r 122 r 222 = h 1 g 1 f 1 h 1 g 1 f 2 h 1 g 2 f 1 h 1 g 2 f 2 h 2 g 1 f 1 h 2 g 1 f 2 h 2 g 2 f 1 h 2 g 2 f 2 = [ h1 h 2 ] [ g1 g 2 ] [ f1 f 2 ] r ijk = h k g j f i From Matrix to Tensor From Tensor To Matrix 22 / 68
23 What Might a Tensor SVD Look Like? vec(r) = r 111 r 211 r 121 r 221 r 112 r 212 r 122 r 222 A special sum of rank-1 tensors. = h (1) g (1) f (1) +h (2) g (2) f (2) +h (3) g (3) f (3) From Matrix to Tensor From Tensor To Matrix 23 / 68
24 What Does the Matrix SVD Look Like? This: [ a11 a 12 a 21 a 22 ] = [ u11 u 12 u 21 u 22 ] [ σ1 0 0 σ 2 ] [ v11 v 12 v 21 v 22 ] T = σ 1 [ u11 u 21 ] [ v11 v 21 ] T + σ 2 [ u12 u 22 ] [ v12 v 22 ] T Is the same as this: a 11 a 21 a 12 a 22 = σ 1 = σ 1 [ v11 v 21 v 11 u 11 v 11 u 21 v 21 u 11 v 21 u 21 ] + σ 2 [ u11 u 21 ] v 12 u 12 v 12 u 22 v 22 u 12 v 22 u 22 + σ 2 [ v12 v 22 ] [ u12 u 22 ] From Matrix to Tensor From Tensor To Matrix 24 / 68
25 What Might a Tensor SVD Look Like? vec(r) = r 111 r 211 r 121 r 221 r 112 r 212 r 122 r 222 = h (1) g (1) f (1) + h (2) g (2) f (2) + h (3) g (3) f (3). A special sum of rank-1 tensors. Getting that special sum often requires multilinear optimiziation. We better understand that before we proceed. From Matrix to Tensor From Tensor To Matrix 25 / 68
26 A Nearest Rank-1 Tensor Problem Find σ 0 and [ ] [ ] [ c1 cos(θ1 ) c2 = sin(θ 1 ) s 1 s 2 ] = [ cos(θ2 ) sin(θ 2 ) ] [ c3 s 3 ] = [ cos(θ3 ) sin(θ 3 ) ] so that φ(σ, θ 1, θ 2, θ 3 ) = a 111 a 211 a 121 a 221 a 112 a 212 a 122 a 222 [ c3 σ s 3 ] [ c2 s 2 ] [ c1 s 1 ] 2 is minimized. From Matrix to Tensor From Tensor To Matrix 26 / 68
27 A Nearest Rank-1 Tensor Problem Find σ 0 and [ ] [ ] [ c1 cos(θ1 ) c2 = sin(θ 1 ) s 1 s 2 ] = [ cos(θ2 ) sin(θ 2 ) ] [ c3 s 3 ] = [ cos(θ3 ) sin(θ 3 ) ] so that is minimized. φ(σ, θ 1, θ 2, θ 3 ) = a 111 a 211 a 121 a 221 a 112 a 212 a 122 a 222 σ c 3 c 2 c 1 c 3 c 2 s 1 c 3 s 2 c 1 c 3 s 2 s 1 s 3 c 2 c 1 s 3 c 2 s 1 s 3 s 2 c 1 s 3 s 2 s 1 2 From Matrix to Tensor From Tensor To Matrix 27 / 68
28 Alternating Least Squares Freeze c 2, s 2, c 3 and s 3 and minimize a 111 a 211 a 121 φ = a 221 a 112 σ 6 a a a 222 with respect to c 3c 2c 1 c 3c 2s 1 c 3s 2c 1 c 3s 2s 1 s 3c 2c 1 s 3c 2s 1 s 3s 2c 1 s 3s 2s a 111 a 211 a 121 = a 221 a a a a 222 x 1 = σc 1 y 1 = σs c 3c c 3c 2 c 3s c 3s 2 s 3c s 3c 2 s 3s s 3s » x1 y 1 2 This is an ordinary linear least squares problem. We then get improved σ, c 1, and s 1 via σ = [ ] [ ] x y 2 c1 x1 1 = /σ s 1 y 1 From Matrix to Tensor From Tensor To Matrix 28 / 68
29 Alternating Least Squares Freeze c 1, s 1, c 3 and s 3 and minimize a 111 c 3c 2c 1 a 211 c 3c 2s 1 a 121 c 3s 2c 1 φ = a 221 a 112 σ c 3s 2s 1 s 3c 2c 1 = 6 a s 3c 2s a s 3s 2c a 222 s 3s 2s 1 with respect to 2 a 111 a 211 a 121 a 221 a 112 a 212 a 122 a x 2 = σc 2 y 2 = σs c 3c 1 0 c 3s c 3c 1 0 c 3s 1 s 3c 1 0 s 3s s 3c 1 0 s 3s » x2 y 2 2 This is an ordinary linear least squares problem. We then get improved σ, c 2, and s 2 via σ = [ ] [ ] x y 2 c2 x2 2 = /σ s 2 y 2 From Matrix to Tensor From Tensor To Matrix 29 / 68
30 Alternating Least Squares Freeze c 1, s 1, c 2 and s 2 and minimize a 111 a 211 a 121 φ = a 221 a 112 σ 6 a a a 222 with respect to c 3c 2c 1 c 3c 2s 1 c 3s 2c 1 c 3s 2s 1 s 3c 2c 1 s 3c 2s 1 s 3s 2c 1 s 3s 2s a 111 a 211 a 121 = a 221 a a a a 222 x 3 = σc 3 y 3 = σs c 2c 1 0 c 2s 1 0 s 2c 1 0 s 2s c 2s 1 0 c 2s 1 0 s 2c 1 0 s 2s » x3 y 3 2 This is an ordinary linear least squares problem. We then get improved σ, c 3, and s 3 via σ = [ ] [ ] x y 2 c3 x3 3 = /σ s 3 y 3 From Matrix to Tensor From Tensor To Matrix 30 / 68
31 Componentwise Optimization A Common Framework for Tensor-Related Optimization: Choose a subset of the unknowns such that if they are (temporarily) fixed, then we are presented with some standard matrix problem in the remaining unknowns. By choosing different subsets, cycle through all the unknowns. Repeat until converged. The standard matrix problem that we end up solving is usually some kind of linear least squares problem. From Matrix to Tensor From Tensor To Matrix 31 / 68
32 We Are Now Ready For This! U T V = That is, we are ready to look at SVD ideas at the tensor level. From Matrix to Tensor From Tensor To Matrix 32 / 68
33 The Higher-Order SVD Motivation: In the matrix case, if A IR n 1 n 2 and A = U 1 SU T 2, then vec(a) = n n S(j 1, j 2 ) U 2 (:, j 2 ) U 1 (:, j 1 ) j 1 =1 j 2 =1 We are able to choose orthogonal U 1 and U 2 so that S = U T 1 AU 2 is diagonal. From Matrix to Tensor From Tensor To Matrix 33 / 68
34 The Higher-Order SVD Definition: Given A IR n 1 n 2 n 3, compute the SVDs of the modal unfoldings A (1) = U 1 Σ 1 V T 1 A (2) = U 2 Σ 2 V T 2 A (3) = U 3 Σ 3 V T 3 and then compute S IR n 1 n 2 n 3 so that vec(a) = n 1 j 1 =1 n 2 j 2 =1 n 3 j 3 =1 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) From Matrix to Tensor From Tensor To Matrix 34 / 68
35 Recall... The mode-1, mode-2, and mode-3 unfoldings of A IR : A (1) = a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a 432 (1,1) (2,1) (3,1) (1,2) (2,2) (3,2) A (2) = a 111 a 211 a 311 a 411 a 112 a 212 a 312 a 412 a 121 a 221 a 321 a 421 a 122 a 222 a 322 a 422 a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432 A (3) = (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) [ a111 a 211 a 311 a 411 a 121 a 221 a 321 a 421 a 131 a 231 a 331 a 431 a 112 a 212 a 312 a 412 a 122 a 222 a 322 a 422 a 132 a 232 a 332 a 432 (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) (1,3) (2,3) (3,3) (4,3) ] From Matrix to Tensor From Tensor To Matrix 35 / 68
36 The Truncated Higher-Order SVD The HO-SVD: vec(a) = n 1 j 1 =1 n 2 j 2 =1 n 3 j 3 =1 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) The core tensor S is not diagonal, but its entries get smaller as you move away from the (1,1,1) entry. The Truncated HO-SVD: vec(a) = r 1 j 1 =1 r 2 j 2 =1 r 3 j 3 =1 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) From Matrix to Tensor From Tensor To Matrix 36 / 68
37 The Tucker Nearness Problem Assume that A IR n1 n2 n3. Given integers r 1, r 2 and r 3 compute U 1 : n 1 r 1, orthonormal columns U 2 : n 2 r 2, orthonormal columns U 3 : n 3 r 3, orthonormal columns and tensor S IR r1 r2 r3 so that r 1 vec(a) r 2 r 3 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) is minimized. j 1=1 j 2=1 j 3=1 2 From Matrix to Tensor From Tensor To Matrix 37 / 68
38 Componentwise Optimization 1. Fix U 2 and U 3 and minimize with respect to S and U 1 : r 1 vec(a) r 2 r 3 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) j 1=1 j 2=1 j 3=1 2. Fix U 1 and U 3 and minimize with respect to S and U 2 : r 1 vec(a) r 2 r 3 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) j 1=1 j 2=1 j 3= Fix U 1 and U 2 and minimize with respect to S and U 3 : r 1 vec(a) r 2 r 3 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) j 1=1 j 2=1 j 3=1 2 From Matrix to Tensor From Tensor To Matrix 38 / 68
39 The CP-Decomposition It also goes by the name of the CANDECOMP/PARAFAC Decomposition. CANDECOMP = Canonical Decomposition PARAFAC = Parallel Factors Decomposition From Matrix to Tensor From Tensor To Matrix 39 / 68
40 A Different Kind of Rank-1 Summation The Tucker representation vec(a) = r 1 j 1=1 r 2 j 2=1 r 3 j 3=1 S(j 1, j 2, j 3 ) U 3 (:, j 3 ) U 2 (:, j 2 ) U 1 (:, j 1 ) uses orthogonal U 1, U 2, and U 3. The CP representation r vec(a) = λ j U 3 (:, j) U 2 (:, j) U 1 (:, j) j=1 uses nonorthogonal U 1, U 2, and U 3. The smallest possible r is called the rank of A. From Matrix to Tensor From Tensor To Matrix 40 / 68
41 Tensor Rank is Trickier than Matrix Rank If a 111 a 211 a 121 a 221 a 112 a 212 a 122 a 222 rank = 2 with prob 79% = randn(8,1), then rank = 3 with prob 21% This is Different from the Matrix Case If A = randn(n,n), then rank(a) = n with probability 1. From Matrix to Tensor From Tensor To Matrix 41 / 68
42 Componentwise Optimization Fix r rank(a) and minimize: r vec(a) λ j U 3 (:, j) U 2 (:, j) U 1 (:, j) j=1 2 Improve U 1 and the λ j by fixing U 2 and U 3 and minimizing r vec(a) λ j U 3 (:, j) U 2 (:, j) U 1 (:, j) j=1 2 Etc. The component optimizations are highly structured least squares problems. From Matrix to Tensor From Tensor To Matrix 42 / 68
43 The Tensor Train Decomposition Idea: Approximate a high-order tensor with a collection of order-3 tensors. Each order-3 tensor is connected to its left and right neighbor through a simple summation. An example of a tensor network. From Matrix to Tensor From Tensor To Matrix 43 / 68
44 Tensor Train: An Example Given the carriages... G 1 : n 1 r 1 G 2 : r 1 n 2 r 2 G 3 : r 2 n 3 r 3 G 4 : r 3 n 4 r 4 G 5 : r 4 n 5 We define the train A(1:n 1, 1:n 2, 1:n 3, 1:n 4, 1:n 5 )... r 1 r 2 r 3 r 4 k 1=1 k 2=1 k 3=1 k 4=1 A(i 1, i 2, i 3, i 4, i 5 ) = G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) From Matrix to Tensor From Tensor To Matrix 44 / 68
45 Tensor Train: An Example Given the carriages... G 1 : n 1 r 1 G 2 : r 1 n 2 r 2 G 3 : r 2 n 3 r 3 G 4 : r 3 n 4 r 4 G 5 : r 4 n 5 We define the train A(1:n 1, 1:n 2, 1:n 3, 1:n 4, 1:n 5 )... r 1 r 2 r 3 r 4 k 1=1 k 2=1 k 3=1 k 4=1 A(i 1, i 2, i 3, i 4, i 5 ) = G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) From Matrix to Tensor From Tensor To Matrix 45 / 68
46 Tensor Train: An Example Given the carriages... G 1 : n 1 r 1 G 2 : r 1 n 2 r 2 G 3 : r 2 n 3 r 3 G 4 : r 3 n 4 r 4 G 5 : r 4 n 5 We define the train A(1:n 1, 1:n 2, 1:n 3, 1:n 4, 1:n 5 )... r 1 r 2 r 3 r 4 k 1=1 k 2=1 k 3=1 k 4=1 A(i 1, i 2, i 3, i 4, i 5 ) = G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) From Matrix to Tensor From Tensor To Matrix 46 / 68
47 Tensor Train: An Example Given the carriages... G 1 : n 1 r 1 G 2 : r 1 n 2 r 2 G 3 : r 2 n 3 r 3 G 4 : r 3 n 4 r 4 G 5 : r 4 n 5 We define the train A(1:n 1, 1:n 2, 1:n 3, 1:n 4, 1:n 5 )... r 1 r 2 r 3 r 4 k 1=1 k 2=1 k 3=1 k 4=1 A(i 1, i 2, i 3, i 4, i 5 ) = G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) From Matrix to Tensor From Tensor To Matrix 47 / 68
48 Tensor Train: An Example Given the carriages... G 1 : n 1 r 1 G 2 : r 1 n 2 r 2 G 3 : r 2 n 3 r 3 G 4 : r 3 n 4 r 4 G 5 : r 4 n 5 We define the train A(1:n 1, 1:n 2, 1:n 3, 1:n 4, 1:n 5 )... r 1 r 2 r 3 r 4 k 1=1 k 2=1 k 3=1 k 4=1 A(i 1, i 2, i 3, i 4, i 5 ) = G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) From Matrix to Tensor From Tensor To Matrix 48 / 68
49 Tensor Train: An Example Given the carriages... G 1 : G 2 : G 3 : G 4 : n 1 r r n 2 r r n 3 r r n 4 r G 5 : r n 5 A(i 1, i 2, i 3, i 4, i 5 ) r r r r G 1 (i 1, k 1 ) G 2 (k 1, i 2, k 2 ) G 3 (k 2, i 3, k 3 ) G 4 (k 3, i 4, k 4 ) G 5 (k 4, i 5 ) k 1=1 k 2=1 k 3=1 k 4=1 Data Sparse: O(nr 2 ) instead of O(n 5 ). From Matrix to Tensor From Tensor To Matrix 49 / 68
50 The Kronecker Product SVD A way to obtain a data sparse representation of an order-4 tensor. It is based on the Kronecker product of matrices, e.g., u 11 u 12 u 11 V u 12 V A = u 21 u 22 V = u 21 V u 22 V u 31 u 32 u 31 V u 32 V and the fact that an order-4 tensor is a reshaped block matrix, e.g., A(i 1, i 2, i 3, i 4 ) = U(i 1, i 2 )V (i 3, i 4 ) From Matrix to Tensor From Tensor To Matrix 50 / 68
51 Kronecker Products are Data Sparse If B and C are n-by-n, then B C is n 2 -by-n 2. = Thus, we need O(n 2 ) numbers to describe an O(n 4 ) object. From Matrix to Tensor From Tensor To Matrix 51 / 68
52 The Nearest Kronecker Product Problem Find B and C so that A B C F = min: a 11 a 12 a 13 a 14 a 21 a 22 a 23 a 24 b a 31 a 32 a 33 a b 12 [ ] c11 c a 41 a 42 a 43 a 44 b 21 b c 21 c 22 a 51 a 52 a 53 a 54 b 31 b 32 a 61 a 62 a 63 a 64 a 11 a 21 a 12 a 22 a 31 a 41 a 32 a 42 a 51 a 61 a 52 a 62 a 13 a 23 a 14 a 24 a 33 a 43 a 34 a 44 a 53 a 63 a 54 a 64 = b 11 b 21 b 31 b 12 b 22 b 32 F [ c11 c 21 c 12 c 22 ] F From Matrix to Tensor From Tensor To Matrix 52 / 68
53 The Kronecker Product SVD If A 11 A 1n A =..... A n1 A nn A ij IR n n then there exist U 1,..., U r IR n n, V 1,..., V r IR n n, and scalars σ 1 σ r > 0 such that A = r σ k U k V k. k=1 From Matrix to Tensor From Tensor To Matrix 53 / 68
54 A Tensor Approximation Idea Unfold A IR n n n n into an n 2 -by-n 2 matrix A. Express A as a sum of Kronecker products: A = r σ k B k C k k=1 B k, C k IR n n Back to tensor: A(i 1, i 2, j 1, j 2 ) = r σ k C k (i 1, i 2 )B k (j 1, j 2 ) k=1 Sums of tensor products of matrices instead of vectors. O(n 2 r) From Matrix to Tensor From Tensor To Matrix 54 / 68
55 The Higher-Order Generalized Singular Value Decomposition We are given a collection of m-by-n data matrices {A 1,..., A N } each of which has full column rank. Do an SVD thing on each of them simultaneously: A 1 = U 1 Σ 1 V T. A N = U N Σ N V T that exposes common features. From Matrix to Tensor From Tensor To Matrix 55 / 68
56 The 2-Matrix GSVD If A 1 = A 2 = then there exist orthogonal U 1, orthogonal U 2 and nonsingular X so that c s U1 T 0 c 2 0 A 1 X = Σ 1 = 0 0 c 3 U T 0 s A 2 X = Σ 2 = 0 0 s From Matrix to Tensor From Tensor To Matrix 56 / 68
57 The Higher-Order GSVD Framework 1. Compute V 1 S N V = diag(λ i ) where S N = 1 N(N 1) N N i=1 j=i+1 ( (A T i A i )(A T j A j ) 1 + (A T j A j )(A T i A i ) 1). 2. For k = 1:N compute A k V T = U k Σ k where the U k have unit 2-norm columns and the Σ k are diagonal. The eigenvalues of S are never smaller than 1. From Matrix to Tensor From Tensor To Matrix 57 / 68
58 The Common HO-GSVD Subspace: Definition The eigenvectors associated with the unit eigenvalues of S N common HO-GSVD subspace: define the HO-GSVD(A 1,..., A N ) = { v : S N v = v } We are able to stably compute this without ever forming S explicitly. A sequence of 2-matrix GSVDs. From Matrix to Tensor From Tensor To Matrix 58 / 68
59 The Common HO-GSVD Subspace: Relevance In general, we have these rank-1 expansions A k = U k Σ k V T = where V = [v 1,..., v n ]. n i=1 σ (k) i u (k) i v T i k = 1:N But if (say) the HO-GSVD(A 1,..., A N ) = span{v 1, v 2 }, then A k and {u (k) = σ 1 u (k) 1 v T 1 + σ 2 u (k) 2 v T 2 + 1, u(k) 2 n i=3 σ (k) i u (k) i v T i k = 1:N } is an orthonormal basis for span{u(k) 3,..., u(k) n }. Moreover, u (k) 1 and u (k) 2 are left singular vectors for A k. This expansion identifies features that are common across the datasets A 1,..., A N. From Matrix to Tensor From Tensor To Matrix 59 / 68
60 The Pivoted Cholesky Decomposition PAP T = d x x x x x x x x d x x x x x x x x d x x x x x x x x x x x x x x x x x x x x x x x x We will use this on a problem where the tensor has multiple symmetries and unfolds to a highly structured positive semidefinite matrix with multiple symmetries. From Matrix to Tensor From Tensor To Matrix 60 / 68
61 The Two-Electron Integral Tensor (TEI) Given a basis {φ i (r)} n i=1 of atomic orbital functions, we consider the following order-4 tensor: φ p (r 1 )φ q (r 1 )φ r (r 2 )φ s (r 2 ) A(p, q, r, s) = dr 1 dr 2. R 3 R 3 r 1 r 2 The TEI tensor plays an important role in electronic structure theory and ab initio quantum chemistry. The TEI tensor has these symmetries: A(q, p, r, s) A(p, q, r, s) = A(p, q, s, r) A(r, s, p, q) (i) (ii) (iii) We say that A is ((12)(34))-symmetric. From Matrix to Tensor From Tensor To Matrix 61 / 68
62 The [1, 2] [3, 4] Unfolding of a ((12)(34)) Symmetric A If A = A [1,2] [3,4], then A is symmetric and (among other things) is perfect shuffle symmetric. A = Each column reshapes into a 3x3 symmetric matrix, e.g., A(:, ) reshapes to What is perfect shuffle symmetry? From Matrix to Tensor From Tensor To Matrix 62 / 68
63 Perfect Shuffle Symmetry An n 2 -by-n 2 matrix A has perfect shuffle symmetry if where A = Π n,n AΠ n,n Π n,n = I n 2(:, v), v = [ 1:n:n 2 2:n:n 2 n:n:n 2 ]. e.g., Π 3,3 = From Matrix to Tensor From Tensor To Matrix 63 / 68
64 Structured Low-Rank Approximation We have an n 2 -by-n 2 matrix A that is symmetric and perfect shuffle symmetric and it basically has rank n. Using PAP T = LDL T we are able to write A = n d k u k uk T k=1 where each rank-1 is symmetric and perfect shuffle symmetric. This structured data-sparse representation reduces work by an order of magnitude in the application we are considering. From Matrix to Tensor From Tensor To Matrix 64 / 68
65 Notation: The Challenge Scientific computing is increasingly tensor-based. It is hard to spread the word about tensor computations because summations, transpositions, and symmetries are typically described through multiple indices. And different camps have very different notations, e.g. t i 1i 2 i 3 i 4 i 5 = a i 1 j1 b i 2 j1 j 2 c i 2 j2 j 3 d i 2 j 3 j 4 e i 2 j4 From Matrix to Tensor From Tensor To Matrix 65 / 68
66 Brevity is the Soul of Wit Multiple Summations n j=1 n 1 j 1 =1 n d j d =1 Transposition If T = [ ] then B = A T means B(i 1, i 2, i 3, i 4 ) = A(i 2, i 1, i 4, i 3 ) Contractions For all 1 i m and 1 j n: A(i, j) = p B(i, k)c(k, j) k=1 From Matrix to Tensor From Tensor To Matrix 66 / 68
67 From Jacobi s 1846 Eigenvalue Paper A system of linear equations: (a, a)α + (a, b)β + (a, c)γ + + (a, p) ω = α x (b, a)α + (b, b)β + (b, c)γ + + (b, p) ω = β x (p, a)α + (p, b)β + (p, c)γ + + (p, p) ω = ω x Somewhere between 1846 and the present we picked up conventional matrix-vector notation: Ax = b How did the transition from scalar notation to matrix-vector notation happen? From Matrix to Tensor From Tensor To Matrix 67 / 68
68 The Next Big Thing... Scalar-Level Thinking 1960 s Matrix-Level Thinking The factorization paradigm: LU, LDL T, QR, UΣV T, etc s Block Matrix-Level Thinking 2000 s Tensor-Level Thinking Cache utilization, parallel computing, LAPACK, etc. High-dimensional modeling, cheap storage, good notation etc. From Matrix to Tensor From Tensor To Matrix 68 / 68
1. Connecting to Matrix Computations. Charles F. Van Loan
Four Talks on Tensor Computations 1. Connecting to Matrix Computations Charles F. Van Loan Cornell University SCAN Seminar October 27, 2014 Four Talks on Tensor Computations 1. Connecting to Matrix Computations
More informationLecture 4. Tensor-Related Singular Value Decompositions. Charles F. Van Loan
From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 4. Tensor-Related Singular Value Decompositions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010
More informationLecture 4. CP and KSVD Representations. Charles F. Van Loan
Structured Matrix Computations from Structured Tensors Lecture 4. CP and KSVD Representations Charles F. Van Loan Cornell University CIME-EMS Summer School June 22-26, 2015 Cetraro, Italy Structured Matrix
More informationLecture 2. Tensor Iterations, Symmetries, and Rank. Charles F. Van Loan
Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Iterations, Symmetries, and Rank Charles F. Van Loan Cornell University CIME-EMS Summer School June 22-26, 2015 Cetraro, Italy Structured
More informationBlockMatrixComputations and the Singular Value Decomposition. ATaleofTwoIdeas
BlockMatrixComputations and the Singular Value Decomposition ATaleofTwoIdeas Charles F. Van Loan Department of Computer Science Cornell University Supported in part by the NSF contract CCR-9901988. Block
More informationTensor Network Computations in Quantum Chemistry. Charles F. Van Loan Department of Computer Science Cornell University
Tensor Network Computations in Quantum Chemistry Charles F. Van Loan Department of Computer Science Cornell University Joint work with Garnet Chan, Department of Chemistry and Chemical Biology, Cornell
More informationforms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms
Christopher Engström November 14, 2014 Hermitian LU QR echelon Contents of todays lecture Some interesting / useful / important of matrices Hermitian LU QR echelon Rewriting a as a product of several matrices.
More informationLecture 2. Tensor Unfoldings. Charles F. Van Loan
From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 2. Tensor Unfoldings Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010 Selva di Fasano, Brindisi,
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationApplied Numerical Linear Algebra. Lecture 8
Applied Numerical Linear Algebra. Lecture 8 1/ 45 Perturbation Theory for the Least Squares Problem When A is not square, we define its condition number with respect to the 2-norm to be k 2 (A) σ max (A)/σ
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationNumerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??
Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement
More informationFundamentals of Multilinear Subspace Learning
Chapter 3 Fundamentals of Multilinear Subspace Learning The previous chapter covered background materials on linear subspace learning. From this chapter on, we shall proceed to multiple dimensions with
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More informationENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition
ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Lecture 8: QR Decomposition
More informationMath 671: Tensor Train decomposition methods
Math 671: Eduardo Corona 1 1 University of Michigan at Ann Arbor December 8, 2016 Table of Contents 1 Preliminaries and goal 2 Unfolding matrices for tensorized arrays The Tensor Train decomposition 3
More information(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =
. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)
More informationFall TMA4145 Linear Methods. Exercise set Given the matrix 1 2
Norwegian University of Science and Technology Department of Mathematical Sciences TMA445 Linear Methods Fall 07 Exercise set Please justify your answers! The most important part is how you arrive at an
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationHomework 2 Foundations of Computational Math 2 Spring 2019
Homework 2 Foundations of Computational Math 2 Spring 2019 Problem 2.1 (2.1.a) Suppose (v 1,λ 1 )and(v 2,λ 2 ) are eigenpairs for a matrix A C n n. Show that if λ 1 λ 2 then v 1 and v 2 are linearly independent.
More informationThe Singular Value Decomposition and Least Squares Problems
The Singular Value Decomposition and Least Squares Problems Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 27, 2009 Applications of SVD solving
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More informationTBP MATH33A Review Sheet. November 24, 2018
TBP MATH33A Review Sheet November 24, 2018 General Transformation Matrices: Function Scaling by k Orthogonal projection onto line L Implementation If we want to scale I 2 by k, we use the following: [
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationNotes on singular value decomposition for Math 54. Recall that if A is a symmetric n n matrix, then A has real eigenvalues A = P DP 1 A = P DP T.
Notes on singular value decomposition for Math 54 Recall that if A is a symmetric n n matrix, then A has real eigenvalues λ 1,, λ n (possibly repeated), and R n has an orthonormal basis v 1,, v n, where
More informationMath 671: Tensor Train decomposition methods II
Math 671: Tensor Train decomposition methods II Eduardo Corona 1 1 University of Michigan at Ann Arbor December 13, 2016 Table of Contents 1 What we ve talked about so far: 2 The Tensor Train decomposition
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More information13-2 Text: 28-30; AB: 1.3.3, 3.2.3, 3.4.2, 3.5, 3.6.2; GvL Eigen2
The QR algorithm The most common method for solving small (dense) eigenvalue problems. The basic algorithm: QR without shifts 1. Until Convergence Do: 2. Compute the QR factorization A = QR 3. Set A :=
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationSingular Value Decomposition
Chapter 5 Singular Value Decomposition We now reach an important Chapter in this course concerned with the Singular Value Decomposition of a matrix A. SVD, as it is commonly referred to, is one of the
More informationThird-Order Tensor Decompositions and Their Application in Quantum Chemistry
Third-Order Tensor Decompositions and Their Application in Quantum Chemistry Tyler Ueltschi University of Puget SoundTacoma, Washington, USA tueltschi@pugetsound.edu April 14, 2014 1 Introduction A tensor
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationMatrix Multiplication Chapter IV Special Linear Systems
Matrix Multiplication Chapter IV Special Linear Systems By Gokturk Poyrazoglu The State University of New York at Buffalo BEST Group Winter Lecture Series Outline 1. Diagonal Dominance and Symmetry a.
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationProblem set 5: SVD, Orthogonal projections, etc.
Problem set 5: SVD, Orthogonal projections, etc. February 21, 2017 1 SVD 1. Work out again the SVD theorem done in the class: If A is a real m n matrix then here exist orthogonal matrices such that where
More informationReview of Some Concepts from Linear Algebra: Part 2
Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationLinear Algebra and Matrices
Linear Algebra and Matrices 4 Overview In this chapter we studying true matrix operations, not element operations as was done in earlier chapters. Working with MAT- LAB functions should now be fairly routine.
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationLecture: Face Recognition and Feature Reduction
Lecture: Face Recognition and Feature Reduction Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab Lecture 11-1 Recap - Curse of dimensionality Assume 5000 points uniformly distributed
More informationStructure in Data. A major objective in data analysis is to identify interesting features or structure in the data.
Structure in Data A major objective in data analysis is to identify interesting features or structure in the data. The graphical methods are very useful in discovering structure. There are basically two
More informationLeast Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo
Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 26, 2010 Linear system Linear system Ax = b, A C m,n, b C m, x C n. under-determined
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationA Review of Linear Algebra
A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose
More informationCheat Sheet for MATH461
Cheat Sheet for MATH46 Here is the stuff you really need to remember for the exams Linear systems Ax = b Problem: We consider a linear system of m equations for n unknowns x,,x n : For a given matrix A
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationECS130 Scientific Computing Handout E February 13, 2017
ECS130 Scientific Computing Handout E February 13, 2017 1. The Power Method (a) Pseudocode: Power Iteration Given an initial vector u 0, t i+1 = Au i u i+1 = t i+1 / t i+1 2 (approximate eigenvector) θ
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationTensor Decompositions and Applications
Tamara G. Kolda and Brett W. Bader Part I September 22, 2015 What is tensor? A N-th order tensor is an element of the tensor product of N vector spaces, each of which has its own coordinate system. a =
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More information33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM
33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the
More information. = V c = V [x]v (5.1) c 1. c k
Chapter 5 Linear Algebra It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix Because they form the foundation on which we later work,
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLecture: Face Recognition and Feature Reduction
Lecture: Face Recognition and Feature Reduction Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 1 Recap - Curse of dimensionality Assume 5000 points uniformly distributed in the
More informationCVPR A New Tensor Algebra - Tutorial. July 26, 2017
CVPR 2017 A New Tensor Algebra - Tutorial Lior Horesh lhoresh@us.ibm.com Misha Kilmer misha.kilmer@tufts.edu July 26, 2017 Outline Motivation Background and notation New t-product and associated algebraic
More informationEXAM. Exam 1. Math 5316, Fall December 2, 2012
EXAM Exam Math 536, Fall 22 December 2, 22 Write all of your answers on separate sheets of paper. You can keep the exam questions. This is a takehome exam, to be worked individually. You can use your notes.
More informationLecture 6, Sci. Comp. for DPhil Students
Lecture 6, Sci. Comp. for DPhil Students Nick Trefethen, Thursday 1.11.18 Today II.3 QR factorization II.4 Computation of the QR factorization II.5 Linear least-squares Handouts Quiz 4 Householder s 4-page
More informationLinear Algebra- Final Exam Review
Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationWe will discuss matrix diagonalization algorithms in Numerical Recipes in the context of the eigenvalue problem in quantum mechanics, m A n = λ m
Eigensystems We will discuss matrix diagonalization algorithms in umerical Recipes in the context of the eigenvalue problem in quantum mechanics, A n = λ n n, (1) where A is a real, symmetric Hamiltonian
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationAssignment 2 (Sol.) Introduction to Machine Learning Prof. B. Ravindran
Assignment 2 (Sol.) Introduction to Machine Learning Prof. B. Ravindran 1. Let A m n be a matrix of real numbers. The matrix AA T has an eigenvector x with eigenvalue b. Then the eigenvector y of A T A
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition
More informationChapter 1. Matrix Algebra
ST4233, Linear Models, Semester 1 2008-2009 Chapter 1. Matrix Algebra 1 Matrix and vector notation Definition 1.1 A matrix is a rectangular or square array of numbers of variables. We use uppercase boldface
More informationLecture 8. Principal Component Analysis. Luigi Freda. ALCOR Lab DIAG University of Rome La Sapienza. December 13, 2016
Lecture 8 Principal Component Analysis Luigi Freda ALCOR Lab DIAG University of Rome La Sapienza December 13, 2016 Luigi Freda ( La Sapienza University) Lecture 8 December 13, 2016 1 / 31 Outline 1 Eigen
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 13
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 208 LECTURE 3 need for pivoting we saw that under proper circumstances, we can write A LU where 0 0 0 u u 2 u n l 2 0 0 0 u 22 u 2n L l 3 l 32, U 0 0 0 l n l
More informationLecture 10: Eigenvectors and eigenvalues (Numerical Recipes, Chapter 11)
Lecture 1: Eigenvectors and eigenvalues (Numerical Recipes, Chapter 11) The eigenvalue problem, Ax= λ x, occurs in many, many contexts: classical mechanics, quantum mechanics, optics 22 Eigenvectors and
More informationLecture 2 INF-MAT : , LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky
Lecture 2 INF-MAT 4350 2009: 7.1-7.6, LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky Tom Lyche and Michael Floater Centre of Mathematics for Applications, Department of Informatics,
More information3D Computer Vision - WT 2004
3D Computer Vision - WT 2004 Singular Value Decomposition Darko Zikic CAMP - Chair for Computer Aided Medical Procedures November 4, 2004 1 2 3 4 5 Properties For any given matrix A R m n there exists
More informationCSL361 Problem set 4: Basic linear algebra
CSL361 Problem set 4: Basic linear algebra February 21, 2017 [Note:] If the numerical matrix computations turn out to be tedious, you may use the function rref in Matlab. 1 Row-reduced echelon matrices
More informationSolution of Linear Equations
Solution of Linear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 7, 07 We have discussed general methods for solving arbitrary equations, and looked at the special class of polynomial equations A subclass
More informationProblem # Max points possible Actual score Total 120
FINAL EXAMINATION - MATH 2121, FALL 2017. Name: ID#: Email: Lecture & Tutorial: Problem # Max points possible Actual score 1 15 2 15 3 10 4 15 5 15 6 15 7 10 8 10 9 15 Total 120 You have 180 minutes to
More informationHouseholder reflectors are matrices of the form. P = I 2ww T, where w is a unit vector (a vector of 2-norm unity)
Householder QR Householder reflectors are matrices of the form P = I 2ww T, where w is a unit vector (a vector of 2-norm unity) w Px x Geometrically, P x represents a mirror image of x with respect to
More informationMATH 581D FINAL EXAM Autumn December 12, 2016
MATH 58D FINAL EXAM Autumn 206 December 2, 206 NAME: SIGNATURE: Instructions: there are 6 problems on the final. Aim for solving 4 problems, but do as much as you can. Partial credit will be given on all
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 12 1 / 18 Overview
More informationPositive Definite Matrix
1/29 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Positive Definite, Negative Definite, Indefinite 2/29 Pure Quadratic Function
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationOrthonormal Transformations and Least Squares
Orthonormal Transformations and Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 30, 2009 Applications of Qx with Q T Q = I 1. solving
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More information5.3 The Power Method Approximation of the Eigenvalue of Largest Module
192 5 Approximation of Eigenvalues and Eigenvectors 5.3 The Power Method The power method is very good at approximating the extremal eigenvalues of the matrix, that is, the eigenvalues having largest and
More informationEcon Slides from Lecture 7
Econ 205 Sobel Econ 205 - Slides from Lecture 7 Joel Sobel August 31, 2010 Linear Algebra: Main Theory A linear combination of a collection of vectors {x 1,..., x k } is a vector of the form k λ ix i for
More informationThroughout these notes we assume V, W are finite dimensional inner product spaces over C.
Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal
More informationSingular Value Decomposition
Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =
More information