Tensors. Lek-Heng Lim. Statistics Department Retreat. October 27, Thanks: NSF DMS and DMS

Transcription

1 Tensors Lek-Heng Lim Statistics Department Retreat October 27, 2012 Thanks: NSF DMS and DMS L.-H. Lim (Stat Retreat) Tensors October 27, / 20

2 tensors on one foot a tensor is a multilinear functional f : V 1 V d C if we give f coordinates, get hypermatrix A = (a j1 j d ) C n 1 n d where n 1 = dim V 1,..., n d = dim V d d-dimensional hypermatrix represents d-tensor the same way matrix represents 2-tensor (i.e. linear operators, bilinear forms, bivectors) for more info: P. McCullagh, Tensor Methods in Statistics, Chapman and Hall, London, plug: L.-H. Lim, Tensors, in L. Hogben (Ed.), Handbook of Linear Algebra, 2nd Ed., CRC Press, Boca Raton, FL, L.-H. Lim (Stat Retreat) Tensors October 27, / 20

3 where do we find tensors? higher-order derivatives f (x) R, f (x) R n, 2 f (x) R n n, 3 f (x) R n n n, 4 f (x) R n n n n,... multivariate moments and cumulants [Fisher-Wishart, 1932]: log E(exp(i t, x )) coefficients are symmetric tensors: m α =1 (κ α (x)) α =1 C p, (κ α (x)) α =2 C p p, i α κ α (x) tα α!. (κ α (x)) α =3 C p p p, (κ α (x)) α =4 C p p p p,... L.-H. Lim (Stat Retreat) Tensors October 27, / 20

4 where do we find tensors? quantum mechanics H1,..., H k state spaces, state space of unified system is H H 1 H k H contains factorizable states ψ 1 ψ k but also mixed states αψ 1 ψ k + + βϕ 1 ϕ k Hj : H j H j Hamiltonian of jth system and I identity operator H 1 I I + I H 2 I + + I I H k Hamiltonian of unified system provided systems do not interact self-concordance in convex optimization 3 f (x) 3 f (x) 4 2 f (x) 2 f (x) 2 f (x) L.-H. Lim (Stat Retreat) Tensors October 27, / 20

5 what can we do with a single tensor? rank hyperdeterminant various decompositions system of multilinear equations multilinear programming multilinear least squares eigenvalues and eigenvectors singular values and singular vectors Gaussian elimination and QR factorization nonnegative tensors and Perron-Frobenius theory spectral, operator, Hölder, Schatten, Ky Fan norms symmetric positive definite tensors and Cholesky decomposition linear preservers of rank, hyperdeterminant, singular, and eigenvalues L.-H. Lim (Stat Retreat) Tensors October 27, / 20

6 why study tensors? a rich source of new problems hypermatrix analogues of matrix notions problems trivial for matrices become non-trivial a rich source of tools for known applications quantum systems holographic algorithms algebraic complexity of matrix multiplication and inversion a rich source of tools for new applications causal inference phylogenetics inference higher order optimization theory principal components of higher order moments and cumulants spectral hypergraph theory encoding NP-hard and #P-hard problems multiarray signal processing diffusion MRI imaging caveat: there will be obstacles L.-H. Lim (Stat Retreat) Tensors October 27, / 20

7 tensor rank rank of A C l m n [Hitchcock, 1927] is rank(a) := min { r A = r i=1 σ iu i v i w i } computational complexity: Strassen matrix multiplication/inversion inf { ω rank ( n i,j,k=1 ϕ ik ϕ kj E ij ) = O(n ω ) } = 2? quantum computing: algebraic measure of entanglement GHZ = C machine learning: naïve Bayes model Pr(x, y, z) = Pr(h) Pr(x h) Pr(y h) Pr(z h) h H X Y Z L.-H. Lim (Stat Retreat) Tensors October 27, / 20

8 example: phylogenetic invariants Markov model for evolution of 3-taxon tree [Allman-Rhodes, 2006] probability distribution given by table with model P = π A ρ A σ A θ A + π C ρ C σ C θ C for i, j, k {A, C, G, T }, + π G ρ G σ G θ G + π T ρ T σ T θ T p ijk = π A ρ Ai σ Aj θ Ak + π C ρ Ci σ Cj θ Ck + π G ρ Gi σ Gj θ Gk + π T ρ Ti σ Tj θ Tk L.-H. Lim (Stat Retreat) Tensors October 27, / 20

9 multilinear systems and hyperdeterminants hyperdeterminant of A = (a ijk ) R [Cayley, 1845] is Det 2,2,2(A) = 1 [ ([ ] [ ]) a000 a 010 a100 a 110 det + 4 a 001 a 011 a 101 a 111 ([ ] [ ])] 2 a000 a 010 a100 a 110 det a 001 a 011 a 101 a 111 [ ] [ ] a000 a 010 a100 a det det a 001 a 011 a 101 a 111 a result that parallels the matrix case: system of bilinear equations a 000x 0y 0 + a 010x 0y 1 + a 100x 1y 0 + a 110x 1y 1 = 0, a 001x 0y 0 + a 011x 0y 1 + a 101x 1y 0 + a 111x 1y 1 = 0, a 000x 0z 0 + a 001x 0z 1 + a 100x 1z 0 + a 101x 1z 1 = 0, a 010x 0z 0 + a 011x 0z 1 + a 110x 1z 0 + a 111x 1z 1 = 0, a 000y 0z 0 + a 001y 0z 1 + a 010y 1z 0 + a 011y 1z 1 = 0, a 100y 0z 0 + a 101y 0z 1 + a 110y 1z 0 + a 111y 1z 1 = 0, has non-trivial solution iff Det 2,2,2 (A) = 0 L.-H. Lim (Stat Retreat) Tensors October 27, / 20

10 eigenvalues and singular values of tensors eigenvalues and singular values are Lagrange multipliers eigenvalues/vectors of S = (s ijk ) S 3 (C n ): cubic Rayleigh quotient [LHL, 2005; Qi, 2005] S(x, x, x) = n constrained to unit l 3 -sphere x 3 = 1 i,j,k=1 s ijkx i x j x k singular values/vectors of A = (a ijk ) C l m n : trilinear Rayleigh quotient [LHL, 2005] A(x, y, z) = l,m,n i,j,k=1 a ijkx i y j z k constrained to product of unit l 3 -spheres x 3 = y 3 = z 3 = 1 Perron-Frobenius theorem for nonnegative tensors [LHL, 2005], [Chang-Pearson-Zhang, 2010], [Friedland-Gaubert-Han, 2012] L.-H. Lim (Stat Retreat) Tensors October 27, / 20

11 tensor norms operator norm of A C l m n A 2,2,2 = A(x, y, z) max x 0,y 0,z 0 x y z = σ max(a) i.e. equals largest singular value of A Schatten and Ky Fan norms [LHL-Comon, 2012] { [ r A,p := inf λ i p] 1/p i=1 A = r λ iu i v i w i, i=1 } u i = v i = w i = 1, r N one interesting property [LHL-Comon, 2012] A,1 rank(a) A, analogue of v 1 v 0 v and M rank(m) M 2 for v C n and M C m n L.-H. Lim (Stat Retreat) Tensors October 27, / 20

12 most tensor problems are NP-hard NP-Hard Tensor Problems NP-Complete NP P some have no FPTAS some are NP-hard even to approximate some are #P-hard some are undecidable C.J. Hillar and L.-H. Lim, Most tensor problems are NP hard, J. Assoc. Comput. Mach., to appear. Matrix Problems L.-H. Lim (Stat Retreat) Tensors October 27, / 20

13 3-coloring encoded as tensor problem colorings of left graph can be encoded as nonzero real solutions to following square set of n = 35 quadratic polynomials in 35 real unknowns a i, b i, c i, d i (i = 1,..., 4), u, w i (i = 1,..., 18): a 1 c 1 b 1 d 1 u 2, b 1 c 1 + a 1 d 1, c 1 u a b2 1, d 1u 2a 1 b 1, a 1 u c d2 1, b 1u 2d 1 c 1, a 2 c 2 b 2 d 2 u 2, b 2 c 2 + a 2 d 2, c 2 u a b2 2, d 2u 2a 2 b 2, a 2 u c d2 2, b 2u 2d 2 c 2, a 3 c 3 b 3 d 3 u 2, b 3 c 3 + a 3 d 3, c 3 u a b2 3, d 3u 2a 3 b 3, a 3 u c d2 3, b 3u 2d 3 c 3, a 4 c 4 b 4 d 4 u 2, b 4 c 4 + a 4 d 4, c 4 u a b2 4, d 4u 2a 4 b 4, a 4 u c d2 4, b 4u 2d 4 c 4, a 2 1 b2 1 + a 1a 3 b 1 b 3 + a 2 3 b2 3, a2 1 b2 1 + a 1a 4 b 1 b 4 + a 2 4 b2 4, a2 1 b2 1 + a 1a 2 b 1 b 2 + a 2 2 b2 2, a 2 2 b2 2 + a 2a 3 b 2 b 3 + a 2 3 b2 3, a2 3 b2 3 + a 3a 4 b 3 b 4 + a 2 4 b2 4, 2a 1b 1 + a 1 b 2 + a 2 b 1 + 2a 2 b 2, 2a 2 b 2 + a 2 b 3 + a 3 b 2 + 2a 3 b 3, 2a 1 b 1 + a 1 b 3 + a 2 b 1 + 2a 3 b 3, 2a 1 b 1 + a 1 b 4 + a 4 b 1 + 2a 4 b 4, 2a 3 b 3 + a 3 b 4 + a 4 b 3 + 2a 4 b 4, w w w w 2 18 equivalent to checking if bilinear system has non-trivial solution: y A k z = 0, x B k z = 0, x C k y = 0, k = 1,..., n L.-H. Lim (Stat Retreat) Tensors October 27, / 20

14 spectral hypergraph theory G = (V, E) is 3-hypergraph, V vertices, E hyperedges { adjacency hypermatrix A C n n n 1 [i, j, k] E, a ijk = 0 otherwise Lemma (L, 2007) G m-regular 3-hypergraph and A adjacency hypermatrix. Then 1 m is an eigenvalue of A 2 if λ is an eigenvalue of A, then λ m 3 λ has multiplicity 1 if and only if G is connected Lemma (L, 2007) G connected m-regular k-partite k-hypergraph on n vertices. Then 1 k 1 mod 4, eigenvalue of A occurs with multiplicity a multiple of k 2 k 3 mod 4, spectrum of A symmetric, ie. λ is eigenvalue iff λ is L.-H. Lim (Stat Retreat) Tensors October 27, / 20

15 higher order optimization first and second order conditions for local minimum necessary: f (x) = 0, 2 f (x) 0 sufficient: f (x) = 0, 2 f (x) 0 for local minimum at x, wlog a 11 [ ]... 2 A 0 a f (x) = = pp 0 0 0, a 11,..., a pp > 0... A R p p : (1, 1)-block of 2 f (x) B R (n p) (n p) (n p) : (2, 2, 2)-block of 3 f (x) B R p (n p) (n p) : (1, 2, 2)-block of 3 f (x) C R (n p) (n p) (n p) (n p) : (2, 2, 2, 2)-block of 4 f (x) third and fourth order conditions for local minimum necessary: B = 0, 4C A 1, B B 0 sufficient: B = 0, 4C A 1, B B 0 L.-H. Lim (Stat Retreat) Tensors October 27, / 20 0

16 mapping the connectome identify neural fibers as accurately as possible from diffusion MRI data fodf Maxima Schultz-Seidel L.-H. Lim (Stat Retreat) Tensors October 27, / 20

17 mapping the connectome after preprocessing, may regard signal as function f : S 2 R f is homogeneous polynomial of even degree, f (x) = n j 1,...,j p=1 a j 1 j p x j1 x jp R[x 1,..., x n ] p coefficients are hypermatrices A = (a j1 j p ) R n n n model mandates that f must be sum of powers of linear forms: f (x) = r i=1 (vt i x) p equivalently, A has Cholesky decomposition : A = r i=1 v p i v i gives direction of ith fiber in a voxel [Schultz-Seidel, 2008], [Schultz-Fuster-Ghosh-Florack-Deriche-LHL, 2012], [LHL-Schultz, 2012] L.-H. Lim (Stat Retreat) Tensors October 27, / 20

18 principal components for higher-order cumulants 0.4 PCA right singular vectors 0.4 Principal kurtosis components Comp Comp 1 Comp Comp 1 Figure: 17 and 39 non-gaussian; all others Gaussian; left: 1st vs 2nd principal components; right: 1st vs 2nd principal kurtosis components; [LHL-Morton, 2012] L.-H. Lim (Stat Retreat) Tensors October 27, / 20

19 multiarray signal processing ith sensor, i = 1,..., l, impinged by r narrowband waves transmitted by independent radiating sources through linear stationary medium assumption: arrays may overlap but differ only by translations (a) (b) (c) signal received by ith sensor in jth array, j = 1,..., m, s i,j (k) = r σ p(t k )ε i,j (θ p ) p=1 assumption implies i and j decouple [LHL-Comon, 2010, 2012], ε i,j (θ p ) = ε i,1 (θ p )ϕ(j, p) may identify individual signals using low-rank tensor approximation L.-H. Lim (Stat Retreat) Tensors October 27, / 20

20 more plugs C.J. Hillar and L.-H. Lim, Most tensor problems are NP hard, J. Assoc. Comput. Mach., to appear. L.-H. Lim, Tensors, in L. Hogben (Ed.), Handbook of Linear Algebra, 2nd Ed., CRC Press, Boca Raton, FL, L.-H. Lim and J. Morton, Principal components of cumulants, preprint, (2012). L.-H. Lim and P. Comon, Multisensor signal processing: tensor decomposition meets compressed sensing, C. R. Acad. Sci. Paris, 338 (2010), no. 6, pp L.-H. Lim and P. Comon, Separable identification, preprint, (2012). T. Schultz, A. Fuster, A. Ghosh, L. Florack, R. Deriche, and L.-H. Lim, Higher-order tensors in diffusion imaging, in B. Burgeth, A.V. Bartroli, and C.-F. Westin (Eds.), Visualization and Processing of Tensors and Higher Order Descriptors for Multi-Valued Data, Springer Verlag, Berlin, L.-H. Lim (Stat Retreat) Tensors October 27, / 20