Hyperdeterminants, secant varieties, and tensor approximations

Transcription

1 Hyperdeterminants, secant varieties, and tensor approximations Lek-Heng Lim University of California, Berkeley April 23, 2008 Joint work with Vin de Silva L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

2 Hypermatrices Totally ordered finite sets: [n] = {1 < 2 < < n}, n N. Vector or n-tuple f : [n] R. If f (i) = a i, then f is represented by a = [a 1,..., a n ] R n. Matrix f : [m] [n] R. If f (i, j) = a ij, then f is represented by A = [a ij ] m,n i,j=1 Rm n. Hypermatrix (order 3) f : [l] [m] [n] R. If f (i, j, k) = a ijk, then f is represented by A = a ijk l,m,n i,j,k=1 Rl m n. Normally R X = {f : X R}. Ought to be R [n], R [m] [n], R [l] [m] [n]. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

3 Hypermatrices and tensors Up to choice of bases a R n can represent a vector in V (contravariant) or a linear functional in V (covariant). A R m n can represent a bilinear form V W R (contravariant), a bilinear form V W R (covariant), or a linear operator V W (mixed). A R l m n can represent trilinear form U V W R (covariant), bilinear operators V W U (mixed), etc. A hypermatrix is the same as a tensor if 1 we give it coordinates (represent with respect to some bases); 2 we ignore covariance and contravariance. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

4 Basic operation on a hypermatrix A matrix can be multiplied on the left and right: A R m n, X R p m, Y R q n, where (X, Y ) A = XAY = [c αβ ] R p q c αβ = m,n i,j=1 x αiy βj a ij. A hypermatrix can be multiplied on three sides: A = a ijk R l m n, X R p l, Y R q m, Z R r n, (X, Y, Z) A = c αβγ R p q r where c αβγ = l,m,n i,j,k=1 x αiy βj z γk a ijk. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

5 Basic operation on a hypermatrix Covariant version: A (X, Y, Z ) := (X, Y, Z) A. Gives convenient notations for multilinear functionals and multilinear operators. For x R l, y R m, z R n, A(x, y, z) := A (x, y, z) = l,m,n a ijkx i y j z k, i,j,k=1 A(I, y, z) := A (I, y, z) = m,n a ijky j z k. j,k=1 L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

6 Symmetric hypermatrices Example Cubical hypermatrix a ijk R n n n is symmetric if a ijk = a ikj = a jik = a jki = a kij = a kji. Invariant under all permutations σ S k on indices. S k (R n ) denotes set of all order-k symmetric hypermatrices. Higher order derivatives of multivariate functions. Example Moments of a random vector x = (X 1,..., X n ): m k (x) = [ E(x i1 x i2 x ik ) ] [ n = x i 1,...,i k =1 i1 x i2 x ik dµ(x i1 ) dµ(x ik ) ] n. i 1,...,i k =1 L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

7 Symmetric hypermatrices Example Cumulants of a random vector x = (X 1,..., X n ): ( ) ( ) κ k (x) = ( 1) p 1 (p 1)!E x i E x i n i A 1 i A p A 1 A p={i 1,...,i k } i 1,...,i k =1 For n = 1, κ k (x) for k = 1, 2, 3, 4 are the expectation, variance, skewness, and kurtosis. Important in Independent Component Analysis (ICA).. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

8 Inner products and norms l 2 ([n]): a, b R n, a, b = a b = n i=1 a ib i. l 2 ([m] [n]): A, B R m n, A, B = tr(a B) = m,n i,j=1 a ijb ij. l 2 ([l] [m] [n]): A, B R l m n, A, B = l,m,n i,j,k=1 a ijkb ijk. In general, l 2 ([m] [n]) = l 2 ([m]) l 2 ([n]), l 2 ([l] [m] [n]) = l 2 ([l]) l 2 ([m]) l 2 ([n]). Frobenius norm A 2 F = l,m,n i,j,k=1 a2 ijk. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

9 DARPA mathematical challenge eight One of the twenty three mathematical challenges announced at DARPA Tech Problem Beyond convex optimization: can linear algebra be replaced by algebraic geometry in a systematic way? Algebraic geometry in a slogan: polynomials are to algebraic geometry what matrices are to linear algebra. Polynomial f R[x 1,..., x n ] of degree d can be expressed as f (x) = a 0 + a 1 x + x A 2 x + A 3 (x, x, x) + + A d (x,..., x). a 0 R, a 1 R n, A 2 R n n, A 3 R n n n,..., A d R n n. Numerical linear algebra: d = 2. Numerical multilinear algebra: d > 2. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

10 Multilinear spectral theory Let A R n n n (easier if A symmetric). 1 How should one define its eigenvalues and eigenvectors? 2 What is a decomposition that generalizes the eigenvalue decomposition of a matrix? Let A R l m n 1 How should one define its singular values and singular vectors? 2 What is a decomposition that generalizes the singular value decomposition of a matrix? Somewhat surprising: (1) and (2) have different answers. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

11 Multilinear spectral theory May define eigenvalues/vectors of A S k (R n ) as critical values/points of the multilinear Raleigh quotient A(x,..., x)/ x k k. Lagrangian L(x, λ) := A(x,..., x) λ( x k k 1). At a critical point A(I n, x,..., x) = λx k 1. Ditto for singular values/vectors of A R d 1 d k. Perron-Frobenius theorem for irreducible non-negative hypermatrices, spectral hypergraph theory: L, Singular values and eigenvalues of tensors: a variational approach, Proc. IEEE Int. Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 1 (2005). L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

12 Tensor ranks (Hitchcock, 1927) Matrix rank. A R m n. rank(a) = dim(span R {A 1,..., A n }) (column rank) = dim(span R {A 1,..., A m }) (row rank) = min{r A = r i=1 u iv T i } (outer product rank). Multilinear rank. A R l m n. rank (A) = (r 1 (A), r 2 (A), r 3 (A)), r 1 (A) = dim(span R {A 1,..., A l }) r 2 (A) = dim(span R {A 1,..., A m }) r 3 (A) = dim(span R {A 1,..., A n }) Outer product rank. A R l m n. rank (A) = min{r A = r i=1 u i v i w i } where u v w : = u i v j w k l,m,n i,j,k=1. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

13 Eigenvalue and singular value decompositions Rank revealing decompositions associated with outer product rank. Symmetric eigenvalue decomposition of A S 3 (R n ), A = r i=1 λ iv i v i v i (1) where rank S (A) = min { r A = r i=1 λ } iv i v i v i = r. P. Comon, G. Golub, L, B. Mourrain, Symmetric tensor and symmetric tensor rank, SIAM J. Matrix Anal. Appl. Singular value decomposition of A R l m n, where rank (A) = r. A = r i=1 σ iu i v i w i (2) V. de Silva, L, Tensor rank and the ill-posedness of the best low-rank approximation problem, SIAM J. Matrix Anal. Appl. (1) used in applications of ICA to signal processing; (2) used in applications of the parafac model to analytical chemistry. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

14 Eigenvalue and singular value decompositions Rank revealing decompositions associated with the multilinear rank. Symmetric eigenvalue decomposition of A S 3 (R n ), A = (U, U, U) C (3) where rank (A) = (r, r, r), U R n r has orthonormal columns and C S 3 (R r ). Singular value decomposition of A R l m n, A = (U, V, W ) C (4) where rank (A) = (r 1, r 2, r 3 ), U R l r 1, V R m r 2, W R n r 3 have orthonormal columns and C R r 1 r 2 r 3. L. De Lathauwer, B. De Moor, J. Vandewalle A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 (2000), no. 4. B. Savas, L, Best multilinear rank approximation with quasi-newton method on Grassmannians, preprint. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

15 Segre variety and its secant varieties The set of all rank-1 hypermatrices is known as the Segre variety in algebraic geometry. It is a closed set (in both the Euclidean and Zariski sense) as it can be described algebraically: Seg(R l, R m, R n ) = {A R l m n A = u v w} = {A R l m n a i1 i 2 i 3 a j1 j 2 j 3 = a k1 k 2 k 3 a l1 l 2 l 3, {i α, j α } = {k α, l α }} Hypermatrices that have rank > 1 are elements on the higher secant varieties of S = Seg(R l, R m, R n ). E.g. a hypermatrix has rank 2 if it sits on a secant line through two points in S but not on S, rank 3 if it sits on a secant plane through three points in S but not on any secant lines, etc. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

16 Decomposition approach to data analysis More generally, F = C, R, R +, R max (max-plus algebra), R[x 1,..., x n ] (polynomial rings), etc. Dictionary, D F N, not contained in any hyperplane. Let D 2 = union of bisecants to D, D 3 = union of trisecants to D,..., D r = union of r-secants to D. Define D-rank of A F N to be min{r A D r }. If ϕ : F N F N R is some measure of nearness between pairs of points (e.g. norms, Bregman divergences, etc), we want to find a best low-rank approximation to A: argmin{ϕ(a, B) D-rank(B) r}. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

17 Decomposition approach to data analysis In the presence of noise, approximation instead of decomposition A α 1 B α r B r D r. B i D reveal features of the dataset A. Note that another way to say best low-rank is sparsest possible. Examples 1 candecomp/parafac: D = {A rank (A) 1}, ϕ(a, B) = A B F. 2 De Lathauwer model: D = {A rank (A) (r 1, r 2, r 3 )}, ϕ(a, B) = A B F. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

18 Fundamental problem of multiway data analysis A hypermatrix, symmetric hypermatrix, or nonnegative hypermatrix. Solve argmin rank(b) r A B. rank may be outer product rank, multilinear rank, symmetric rank (for symmetric hypermatrix), or nonnegative rank (nonnegative hypermatrix). Example Given A R d 1 d 2 d 3, find u i, v i, w i, i = 1,..., r, that minimizes A u 1 v 1 w 1 u 2 v 2 w 2 u r v r z r or C R r 1 r 2 r 3 and U R d 1 r 1, V R d 2 r 2, W R d 3 r 3, that minimizes A (U, V, W ) C. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

19 Fundamental problem of multiway data analysis Example Given A S k (C n ), find u i, i = 1,..., r, that minimizes A u k 1 u k 2 u k r or C R r 1 r 2 r 3 and U R n r i that minimizes A (U, U, U) C. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

20 Separation of variables Approximation by sum or integral of separable functions Continuous f (x, y, z) = θ(x, t)ϕ(y, t)ψ(z, t) dt. Semi-discrete f (x, y, z) = r p=1 θ p(x)ϕ p (y)ψ p (z) θ p (x) = θ(x, t p ), ϕ p (y) = ϕ(y, t p ), ψ p (z) = ψ(z, t p ), r possibly. Discrete a ijk = r u ipv jp w kp p=1 a ijk = f (x i, y j, z k ), u ip = θ p (x i ), v jp = ϕ p (y j ), w kp = ψ p (z k ). L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

21 Separation of variables Useful for data analysis, machine learning, pattern recognition. Gaussians are separable exp(x 2 + y 2 + z 2 ) = exp(x 2 ) exp(y 2 ) exp(z 2 ). More generally for symmetric positive-definite A R n n, exp(x Ax) = exp(z Λz) = n exp(λ iz 2 i=1 i ). Gaussian mixture models f (x) = m α j exp[(x µ j=1 j ) A j (x µ j )], f is a sum of separable functions. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

22 Integral kernels Approximation by sum or integral kernels Continuous f (x, y, z) = K(x, y, z )θ(x, x )ϕ(y, y )ψ(z, z ) dx dy dz. Semi-discrete f (x, y, z) = p,q,r c i,j,k i =1 j k θ i (x)ϕ j (y)ψ k (z) c i j k = K(x i, y j, z k ), θ i (x) = θ(x, x i ), ϕ j (y) = ϕ(y, y j ), ψ k (z) = ψ(z, z k ), p, q, r possibly. Discrete a ijk = p,q,r c i,j,k i =1 j k u ii v jj w kk a ijk = f (x i, y j, z k ), u ii = θ i (x i ), v jj = ϕ j (y j ), w kk = ψ k (z k ). L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

23 Lemma (de Silva, L) Let r 2 and k 3. Given the norm-topology on R d 1 d k, the following statements are equivalent: 1 The set S r (d 1,..., d k ) := {A rank (A) r} is not closed. 2 There exists a sequence A n, rank (A n ) r, n N, converging to B with rank (B) > r. 3 There exists B, rank (B) > r, that may be approximated arbitrarily closely by hypermatrices of strictly lower rank, i.e. inf{ B A rank (A) r} = 0. 4 There exists C, rank (C) > r, that does not have a best rank-r approximation, i.e. inf{ C A rank (A) r} is not attained (by any A with rank (A) r). L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

24 Non-existence of best low-rank approximation For x i, y i R d i, i = 1, 2, 3, A := x 1 x 2 y 3 + x 1 y 2 x 3 + y 1 x 2 x 3. For n N, ( A n := n x ) n y 1 ( x ) ( n y 2 x ) n y 3 nx 1 x 2 x 3. Lemma (de Silva, L) rank (A) = 3 iff x i, y i linearly independent, i = 1, 2, 3. Furthermore, it is clear that rank (A n ) 2 and lim n A n = A. Original result, in a different form, due to: D. Bini, G. Lotti, F. Romani, Approximate solutions for the bilinear form computational problem, SIAM J. Comput., 9 (1980), no. 4. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

25 Outer product approximations are ill-behaved Such phenomenon can and will happen for all orders > 2, all norms, and many ranks: Theorem (de Silva, L) Let k 3 and d 1,..., d k 2. For any s such that 2 s min{d 1,..., d k }, there exists A R d 1 d k with rank (A) = s such that A has no best rank-r approximation for some r < s. The result is independent of the choice of norms. For matrices, the quantity min{d 1, d 2 } will be the maximal possible rank in R d 1 d 2. In general, a hypermatrix in R d 1 d k can have rank exceeding min{d 1,..., d k }. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

26 Outer product approximations are ill-behaved Tensor rank can jump over an arbitrarily large gap: Theorem (de Silva, L) Let k 3. Given any s N, there exists a sequence of order-k hypermatrix A n such that rank (A n ) r and lim n A n = A with rank (A) = r + s. Hypermatrices that fail to have best low-rank approximations are not rare. May occur with non-zero probability; sometimes with certainty. Theorem (de Silva, L) Let µ be a measure that is positive or infinite on Euclidean open sets in R l m n. There exists some r N such that µ({a A does not have a best rank-r approximation}) > 0. In R 2 2 2, all rank-3 hypermatrices fail to have best rank-2 approximation. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

27 Message That the best rank-r approximation problem for hypermatrices has no solution poses serious difficulties. It is incorrect to think that if we just want an approximate solution, then this doesn t matter. If there is no solution in the first place, then what is it that are we trying to approximate? i.e. what is the approximate solution an approximate of? L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

28 Weak solutions For a hypermatrix A that has no best rank-r approximation, we will call a C {A rank (A) r} attaining inf{ C A rank (A) r} a weak solution. In particular, we must have rank (C) > r. It is perhaps surprising that one may completely parameterize all limit points of order-3 rank-2 hypermatrices. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

29 Weak solutions Theorem (de Silva, L) Let d 1, d 2, d 3 2. Let A n R d 1 d 2 d 3 with rank (A n ) 2 and be a sequence of hypermatrices lim n A n = A, where the limit is taken in any norm topology. If the limiting hypermatrix A has rank higher than 2, then rank (A) must be exactly 3 and there exist pairs of linearly independent vectors x 1, y 1 R d 1, x 2, y 2 R d 2, x 3, y 3 R d 3 such that A = x 1 x 2 y 3 + x 1 y 2 x 3 + y 1 x 2 x 3. In particular, a sequence of order-3 rank-2 hypermatrices cannot jump rank by more than 1. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

30 Hyperdeterminant Work in C (d 1+1) (d k +1) for the time being (d i 1). Consider M := {A C (d 1+1) (d k +1) A(x 1,..., x k ) = 0 Theorem (Gelfand, Kapranov, Zelevinsky) M is a hypersurface iff for all j = 1,..., k, d j i j d i. for non-zero x 1,..., x k }. The hyperdeterminant Det(A) is the equation of the hypersurface, i.e. a multivariate polynomial in the entries of A such that M = {A C (d 1+1) (d k +1) Det(A) = 0}. Det(A) may be chosen to have integer coefficients. For C m n, condition becomes m n and n m, i.e. square matrices. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

31 2 2 2 hyperdeterminant Hyperdeterminant of A = a ijk R [Cayley; 1845] is Det(A) = 1 [ ([ ] [ ]) a000 a 010 a100 a 110 det + 4 a 001 a 011 a 101 a 111 ([ ] [ ])] 2 a000 a 010 a100 a 110 det a 001 a 011 a 101 a 111 [ ] a000 a det det a 001 a 011 [ a100 a 110 a 101 a 111 A result that parallels the matrix case is the following: the system of bilinear equations a 000x 0y 0 + a 010x 0y 1 + a 100x 1y 0 + a 110x 1y 1 = 0, a 001x 0y 0 + a 011x 0y 1 + a 101x 1y 0 + a 111x 1y 1 = 0, a 000x 0z 0 + a 001x 0z 1 + a 100x 1z 0 + a 101x 1z 1 = 0, a 010x 0z 0 + a 011x 0z 1 + a 110x 1z 0 + a 111x 1z 1 = 0, a 000y 0z 0 + a 001y 0z 1 + a 010y 1z 0 + a 011y 1z 1 = 0, a 100y 0z 0 + a 101y 0z 1 + a 110y 1z 0 + a 111y 1z 1 = 0, has a non-trivial solution iff Det(A) = 0. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37 ].

32 2 2 3 hyperdeterminant Hyperdeterminant of A = a ijk R is a 000 a 001 a 002 a 100 a 101 a 102 Det(A) = det a 100 a 101 a 102 det a 010 a 011 a 012 a 010 a 011 a 012 a 110 a 111 a 112 a 000 a 001 a 002 a 000 a 001 a 002 det a 100 a 101 a 102 det a 010 a 011 a 012 a 110 a 111 a 112 a 110 a 111 a 112 Again, the following is true: a 000x 0y 0 + a 010x 0y 1 + a 100x 1y 0 + a 110x 1y 1 = 0, a 001x 0y 0 + a 011x 0y 1 + a 101x 1y 0 + a 111x 1y 1 = 0, a 002x 0y 0 + a 012x 0y 1 + a 102x 1y 0 + a 112x 1y 1 = 0, a 000x 0z 0 + a 001x 0z 1 + a 002x 0z 2 + a 100x 1z 0 + a 101x 1z 1 + a 102x 1z 2 = 0, a 010x 0z 0 + a 011x 0z 1 + a 012x 0z 2 + a 110x 1z 0 + a 111x 1z 1 + a 112x 1z 2 = 0, a 000y 0z 0 + a 001y 0z 1 + a 002y 0z 2 + a 010y 1z 0 + a 011y 1z 1 + a 012y 1z 2 = 0, a 100y 0z 0 + a 101y 0z 1 + a 102y 0z 2 + a 110y 1z 0 + a 111y 1z 1 + a 112y 1z 2 = 0, has a non-trivial solution iff Det(A) = 0. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

33 Cayley hyperdeterminant and tensor rank The Cayley hyperdeterminant Det 2,2,2 may be extended to any A R d 1 d 2 d 3 with rank (A) 2. Theorem (de Silva, L) Let d 1, d 2, d 3 2. A R d 1 d 2 d 3 is a weak solution, i.e. iff Det 2,2,2 (A) = 0. Theorem (Kruskal) A = x 1 x 2 y 3 + x 1 y 2 x 3 + y 1 x 2 x 3, Let A R Then rank (A) = 2 if Det 2,2,2 (A) > 0 and rank (A) = 3 if Det 2,2,2 (A) < 0. See de Silva-L for a proof via the Cayley hyperdeterminant. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

34 Symmetric hypermatrices for blind source separation Problem Given y = Mx + n. Unknown: source vector x C n, mixing matrix M C m n, noise n C m. Known: observation vector y C m. Goal: recover x from y. Assumptions: 1 components of x statistically independent, 2 M full column-rank, 3 n Gaussian. Method: use cumulants κ k (y) = (M, M,..., M) κ k (x) + κ k (n). By assumptions, κ k (n) = 0 and κ k (x) is diagonal. So need to diagonalize the symmetric hypermatrix κ k (y). L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

35 Diagonalizing a symmetric hypermatrix A best symmetric rank approximation may not exist either: Example (Comon, Golub, L, Mourrain) Let x, y R n be linearly independent. Define for n N, and ( A n := n x + 1 ) k n y nx k A := x y y + y x y + + y y x. Then rank S (A n ) 2, rank S (A) = k, and lim n A n = A. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

36 Nonnegative hypermatrices and nonnegative tensor rank Let 0 A R d 1 d k. The nonnegative rank of A is rank + (A) := min { r r i=1 u i v i z i, u i,..., z i 0 } Clearly nonnegative decomposition exists for any A 0. Arises in the Naïve Bayes model. Theorem (L) Let A = a j1 j k R d 1 d k be nonnegative. Then inf { A r u i v i z i u i,..., z i 0 } i=1 is always attained. L.-H. Lim (Berkeley) Tensor approximations April 23, / 37

37 Advertisement Geometry and representation theory of tensors for computer science, statistics, and other areas 1 MSRI Summer Graduate Workshop July 7 to July 18, 2008 Organized by J.M. Landsberg, L.-H. Lim, J. Morton Mathematical Sciences Research Institute, Berkeley, CA sgw 2 AIM Workshop July 21 to July 25, 2008 Organized by J.M. Landsberg, L.-H. Lim, J. Morton, J. Weyman American Institute of Mathematics, Palo Alto, CA L.-H. Lim (Berkeley) Tensor approximations April 23, / 37