Computational Multilinear Algebra

Transcription

1 Computational Multilinear Algebra Charles F. Van Loan Supported in part by the NSF contract CCR

2 Involves Working With Kronecker Products b 11 b 12 b 21 b 22 c 11 c 12 c 13 c 21 c 22 c 23 c 31 c 32 c 33 = b 11 c 11 b 11 c 12 b 11 c 13 b 12 c 11 b 12 c 12 b 12 c 13 b 11 c 21 b 11 c 22 b 11 c 23 b 12 c 21 b 12 c 22 b 12 c 23 b 11 c 31 b 11 c 32 b 11 c 33 b 12 c 31 b 12 c 32 b 12 c 33 b 21 c 11 b 21 c 12 b 21 c 13 b 22 c 11 b 22 c 12 b 22 c 13 b 21 c 21 b 21 c 22 b 21 c 23 b 22 c 21 b 22 c 22 b 22 c 23 b 21 c 31 b 21 c 32 b 21 c 33 b 22 c 31 b 22 c 32 b 22 c 33 Replicated Block Structure

3 Properties Quite predictable: (B C) T = B T C T (B C) 1 = B 1 C 1 (B C)(D F ) = BD CF B (C D) = (B C) D Think twice: B C = C B B C = (Perfect Shuffle)(C B)(Perfect Shuffle) T

4 The Perfect Shuffle S p,q S 3, = S 3, Takes the length-pq card deck x, splits it into p piles of length-q each, and then takes one card from each pile in turn until the deck is reassembled.

5 C B = S m1,m 2 (B C)S T n 1,n 2 B IR m 1 n 1,C IR m 2 n 2 E.g., A = B C = b 11 c 11 b 11 c 12 b 11 c 13 b 12 c 11 b 12 c 12 b 12 c 13 b 11 c 21 b 11 c 22 b 11 c 23 b 12 c 21 b 12 c 22 b 12 c 23 b 21 c 11 b 21 c 12 b 21 c 13 b 22 c 11 b 22 c 12 b 22 c 13 b 21 c 21 b 21 c 22 b 21 c 23 b 22 c 21 b 22 c 22 b 22 c 23 S 2,2 AS T 2,3 = = C B = c 11 b 11 c 11 b 12 c 12 b 11 c 12 b 12 c 13 b 11 c 13 b 12 c 11 b 21 c 11 b 22 c 12 b 21 c 12 b 22 c 13 b 21 c 13 b 22 c 21 b 11 c 21 b 12 c 22 b 11 c 22 b 12 c 23 b 11 c 23 b 12 c 21 b 21 c 21 b 22 c 22 b 21 c 22 b 22 c 23 b 21 c 23 b 22

6 Inheriting Structure If B and C are nonsingular lower(upper) triangular banded symmetric positive definite stochastic Toeplitz permutations orthogonal then B C is nonsingular lower(upper)triangular block banded symmetric positive definite stochastic block Toeplitz a permutation orthogonal

7 Factorizations of B C Obvious... B C =(P T B L B U B ) (P T C L C U C )=(P B P C ) T (L B L C )(U B U C ) B C =(G B G T B) (G C G T C) = (G B G C )(G B G C ) T B C =(Q B R B ) (Q C R C ) = (Q B Q C )(R B R C ) Sort of... B C =(Q B Λ B Q T B) (Q C Λ C Q T C)=(Q B Q C )(Λ B Λ C )(Q B Q C ) T B C =(U B Σ B V T B ) (U C Σ C V T C ) = (U B U C )(Σ B Σ C )(V B V C ) T Problematic... B C =(Q B R B P T B ) (Q C R C P T C )=(Q B Q C )(R B R C )(P B P C ) T

8 Sort of =

9 P =

10 Fast Factoring Suppose B IR m m and C IR n n are positive definite. A = B C is an mn-by-mn symmetric positive definite natrix. Ordinarily, a Cholesky factorization GG T this size would cost O(m 3 n 3 ) flops. But G A = G B G C and G B and G C require O(m 3 + n 3 ) flops.

11 Fast Solving If B,C IR m m, then the m 2 -by-m 2 system (B C)x = f CXB T = F x = vec(x), f= vec(f ) canbesolvedino(m 3 ) flops: CY = F XB T = Y via factorizations of B and C.

12 The vec Operation X IR m n vec(x) = X(:, 1) X(:, 2). X(:,n) Y = CXB T vec(y )=(B C)vec(X)

13 Matrix Equations Sylvester: FX + XG T = C (I n F + G I m ) vec(x) = vec(c) Lyapunov: FX + XF T = C (I n F + F I n )vec(x) = vec(c) Generalized Sylvester: F 1 XG T 1 + F 2 XG T 2 = C (G 1 F 1 + G 2 F 2 ) vec(x) = vec(c) These can be solved in O(n 3 ) time via the (generalized) Schur decomposition. See Bartels and Stewart, Golub, Nash, and Van Loan, and J. Gardiner, M.R. Wette, A.J. Laub, J.J. Amato, and C.B. Moler.

14 Talk Outline How do we solve (1) min W ((B C)x d) (weighted least quares) (2) U T [B C d] V = Σ (total least squares) (3) (A p A 1 λi)x = b (shifted linear systems) given that these problems (1 ) min (B C)x d (2 ) U T (B C)V = Σ are easy (3 ) (A p A 1 )x = b

15 The Rise of the Kronecker Product Thriving Application Areas Sparse Factorizations for fast linear transforms Tensoring Low-Dimension Techniques

16 Thriving application areas such as Semidefinite Programming... Some sample problems... X X + A T DA u = f. 0 A T I A 0 0 Z I 0 X I x y z = r d r p r c. See Alizadeh, Haeberly, and Overton (1998).

17 Symmetric Kronecker Products For symmetric X IR n n and arbitrary B,C IR n n this operation is defined by 1 (B C)svec(X) =svec CXB T + BXC T 2 where the svec operation is a normalized stacking of X s subdiagonal columns, e.g., X = x 11 x 12 x 13 x 21 x 22 x 23 x 31 x 32 x 33 svec(x) = x 11 2x21 2x31 x 22 2x32 x 33 T. svec stacks the subdiagonal portion of X s columns.

18 The Rise of the Kronecker Product Cont d Sparse Factorizations Kronecker products are proving to be a very effective way to look at fast linear transforms.

19 A Sparse Factorization of the DFT Matrix n =2 t F n = A t A 1 P n P n = S 2,n/2 (I 2 S 2,n/4 ) (I n/4 S 2,2 ) A q = I r I L/2 Ω L/2 I L/2 Ω L/2 L =2 q,r= n/l Ω L/2 = diag(1, ω L,...,ω L/2 1 L ) ω L =exp( 2πi/L)

20 Different FFTs Correspond to Different Factorizations of F n TheCooley-TukeyFFTisbasedony = F n x = A t A 1 P n x x P n x for k =1:t x A q x end y x The Gentleman-Sande FFT is based on y = F n x = F T n x = P T n A T 1 A T t x for k = t: 1:1 x A T q x end y P T n x

21 Matrix Transpose A IR m n and B = A T, then vec(b) =S n,m vec(a). a 11 a 12 a 13 a 21 a 22 a 23 = a 11 a 21 a 12 a 22 a 13 a 23 a 11 a 21 a 12 a 22 a 13 a 23 = a 11 a 12 a 13 a 21 a 22 a 23 T

22 Multiple Pass Transpose To compute B = A T where A IR m n factor and then execute a =vec(a) for k =1:t a Γ k a end Define B IR n m by vec(b) =a. S n,m = Γ t Γ 1 Different transpose algorithms correspond to different factorizations of S n,m.

23 An Example If m = pn, then S n,m = Γ 2 Γ 1 = (I p S n,n )(S n,p I n ) The first pass b (1) = Γ 1 vec(a) corresponds to a block transposition: A = A 1 A 2 A 3 A 4 B (1) = A 1 A 2 A 3 A 4. The second pass b (2) = Γ 2 b (1) carries out the transposition of the blocks. B (1) = A 1 A 2 A 3 A 4 B (2) = A T 1 A T 2 A T 3 A T 4. Note that B (2) = A T.

24 The Rise of the Kronecker Product Cont d Tensoring Low-Dimension Techniques Greater willingness to entertain problems of high dimension.

25 Tensoring Low Dimensional Ideas Quadrature in one and three dimensions: b a f(x) dx n i=1 w i f(x i ) = w T f(x) b 1 b 2 b 3 a 1 a 2 a 3 f(x, y, z) dx dy dz n x n y n z i=1 j=1 k=1 w (x) i w (y) j w (z) k f(x i,y j,z k ) = (w (x) w (y) w (z) ) T f(x y z)

26 Higher Dimensional KP Problems 1. Andre, Nowak, and Van Veen (1997). In higher-order statistics the calculations involve the cumulants x x x. (Note that vec(xx T )=x x.) 2. de Lathauwer, de Moor, and Vandewalle (2000) multilinear SVD, low rank approximation of tensors

27 Least Squares Problems that Involve Kronecker Products

28 Some Least Squares Problems Least squares problems of the form min (B C)x b can be efficiently solved by computing the QR factorizations (or SVDs) of B and C. See Fausett and Fulton (1994) and Fausett, Fulton, and Hashish (1997). Barrlund (1998) on surface fitting with splines... (A 1 A 2 )x f such that (B 1 B 2 )x = g

29 Weighted Least Squares min W 1/2 ((B C)x b) 2 W B C B T C T 0 r x = b 0 Compute the QR factorizations B = Q B R B 0 C = Q C R C 0

30 The augmented system transforms to E 11 E 12 R B R C E 21 E 22 0 R T B R T C 0 0 r 1 r 2 x = b1 b2 0 where E 11 E 12 E 21 E 22 is a simple permutation of (Q B Q C ) T W (Q B Q C ). Solve the E 22 system via congugate gradients exploiting structure.

31 When the blocks are Kronecker products... min B 1 C 1 B 2 C 2 x b 2 Compute a pair of GSVD s: B 1 = U 1B D 1B X T B B 2 = U 2B D 2B X T B C 1 = U 1C D 1C X T C C 2 = U 2C D 2C X T C. B 1 C 1 B 2 C 2 = U 1B U 2B 0 0 U 1C U 2C D 1B D 2B D 1C D 2C X T B X T C.

32 Total Least Squares LS: min b + r ran(a) r 2 2 Ax LS = b + r opt TLS: min b + r ran(a + E) E 2 F + r 2 2 (A + E opt )x TLS = b + r opt Errors in Variables

33 Total Least Squares Solution To solve min b + r ran(a + E) E 2 F + r 2 2 (A + E opt )x TLS = b + r opt compute the SVD of [A b] IR m n+1 : U T A b V = Σ and set x TLS = V (1:n, n +1)/V (n +1,n+1)

34 If A is a Kronecker Product B C... We need the last column of V in U T FV = Σ where First compute the SVDs of B and C: F = B C b U T B BV B = Σ B U T C CV C = Σ C If then Ũ = U B U C Ṽ = V B V C Ũ T F Ṽ = Σ B Σ C g F where g = Ũ T b We need the smallest right singular vector of F.

35 Think Eigenvector The right singular vectors of F are the eigenvectors of where C = F T F = ΣB Σ C g T Σ B Σ C g = D = Σ T BΣ B Σ T CΣ C (square and diagonal) D z z T α z =(Σ B Σ C ) T g α = g T g

36 Eigenvectors/Eigenvalues of Bordered Matrices C = d z 1 0 d z d 3 0 z d 4 z 4 z 1 z 2 z 3 z 4 α The eigenvalues of C are the zeros of f(µ) =µ α + z T (D µi) 1 z = µ α + N i=1 z 2 i d i µ Can assume that the d i are distinct and that the z i are nonzero. The zeros of f are separated by the d i. Can show that the smallest root of f (a.k.a. the smallest eigenvalue of F ) is between 0 and the smallest d i.

37 From the eigenvalue equation it follows that D z z T α x 1 = µ min x =(D µ min I) 1 z Thus the sought-after singular vector of F is just a unit vector in the direction x 1 (D µ min I) 1 z 1

38 Overall Solution Procedure 1. First compute the SVDs of B and C: U T B BV B = Σ B U T C CV C = Σ C 2. Find smallest root of f(µ) =µ α + z T (D µi) 1 z. 3. Get smallest singular vector w of ΣB Σ C g 4. Get the TLS solution from v = V B V C w

39 A Shifted Kronecker Product System

40 Frequency Response Suppose we wish to evaluate for many different values of µ. φ(µ) = c T (A µi) 1 d It pays to recompute the real Schur decomposition.. Q T AQ = T = φ(µ) = c T Q(T µi) 1 Q T d (O(n 2 ) per evaluation

41 Solve A p A 1 λi x = b. First compute the real Schur decompositions A k = Q k T k Q T k,i=1:p Set Q = Q (p) Q (1). It follows that Q T A (p) A (1) Q = T (p) T (1) T and we get T (p) T (1) λi n y = c where y IR n and c IR n are defined by x = Q (p) Q (1) y c = Q (p) Q (1) T b.

42 Unshifted Case The standard approach for solving is to recognize that T (p) T (1) y = c T = p i=1 Iρi T (i) I µi T (i) IR n i n i where ρ i = n 1 n i 1 and µ i = n i+1 n p for i =1:p. We then obtain... y c for i =1:p y I ρi T (i) I µi 1 y end

43 The p =2case: (F G λi)x = b f 11 G λi m f 12 G f 13 G f 14 G 0 f 22 G λi m f 23 G f 24 G 0 0 f 33 G λi m f 34 G f 44 G λi m y 1 y 2 y 3 y 4 = c 1 c 2 c 3 c 4 Solve the triangular system (f 44 G λi m ) y 4 = c 4 for y 4. Substituting this into the above system reduces it to f 11 G λi m f 12 G f 13 G 0 f 22 G λi m f 23 G 0 0 f 33 G λi m y 1 y 2 y 3 = c 1 c 2 c 3 where c i = c i f i4 Gy 4, i =1:3.

44 It is recursive... The General Triangular Case function y = KPShiftSolve(T,c,λ,n,α) n is a p-by-1 integer vector T is a p-by-1 cell array where T {i} is an n i -by-n i upper triangular matrix for i =1:p. Assume that c IR N where N = n 1 n p and λ and α are scalars so A =(αt {p} T {1} λi N ) A is nonsingular. y IR N solves Ay = c.

45 p length(n) if p == 1 y (αt {1} λi)\c else m n 1 n p 1 for i = n p : 1:1 idx 1+(i 1)m:im α T {p}(i, i) y(idx) KPShiftSolve(T,b(idx), λ,n(1:p 1), α) z (T {p 1} T {1})y(idx) for j =1:i 1 jdx 1+(j 1)m:jm c(jdx) c(jdx) T {p}(j, i)z end end end

46 Two-by-Two Bumps f 11 G λi m f 12 G f 13 G f 14 G 0 f 22 G λi m f 23 G f 24 G 0 0 f 33 G λi m f 34 G 0 0 f 43 G f 44 G λi m y 1 y 2 y 3 y 4 = c 1 c 2 c 3 c 4 Solve for y 3 and y 4 at the same time. Revert to localized complex arithmetic. Z H f 33 f 34 f 43 f 44 Z = t 33 t 34 0 t 44

47 High Order Kronecker Products y =(F p F 1 ) x F i IR n i n i It is convenient to make use of the factorization F p F 1 = M p M 1 where M i = Π T n i,n/n i IN/ni F i N = n 1 n p Matlab: Z x for i =1:p Z (F i reshape(z, n i,n/n i )) T end y reshape(z, N, 1)

48 Conclusion Our goal is to heighten the profile of the Kronecker product and develop an infrastructure of methods thereby making it easier for the numerical linear algebra community to spot Kronecker opportunities. With precedent... The development of effective algorithms for the QR and SVD factorizations turned many A T A problems into least square/singular value calculations. The development of the QZ algorithm (Moler and Stewart (1973)) for the problem Ax = λbx prompted a rethink of standard eigenproblems that have the form B 1 Ax = λx.