Most tensor problems are NP hard

Transcription

1 Most tensor problems are NP hard (preliminary report) Lek-Heng Lim University of California, Berkeley August 25, 2009 (Joint work with: Chris Hillar, MSRI) L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

2 Innocent looking problem Problem (Minimal rank-1 matrix subspace) Let A 1,..., A l R m n. Find smallest r such that there exist rank-1 matrices u 1 v1,..., u r vr with A 1,..., A l span{u 1 v 1,..., u r v r }. NP-complete over F q, NP-hard over Q [Håstad; 90]. Slight extension: NP-hard over R and C [L & Hillar; 09]. Convex relaxation along the lines of compressive sensing? 1 0, rank. Ky Fan/nuclear/Schatten/trace norm, A = rank(a) σ i (A). i=1 [Fazel, Hindi, Boyd; 01], [Recht, Fazel, Parrilo; 09], [Candès, Recht; 09], [Zhu, So, Ye; 09], [Toh, Yun; 09]. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

3 Tensors as hypermatrices Up to choice of bases on U, V, W, a 3-tensor A U V W may be represented as a hypermatrix A = a ijk l,m,n i,j,k=1 Rl m n where dim(u) = l, dim(v ) = m, dim(w ) = n if 1 we give it coordinates; 2 we ignore covariance and contravariance. Henceforth, tensor = hypermatrix. Scalar = 0-tensor, vector = 1-tensor, matrix = 2-tensor. Cubical 3-tensor a ijk R n n n is symmetric if a ijk = a ikj = a jik = a jki = a kij = a kji. Set of symmetric 3-tensors denoted S 3 (R). L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

4 Examples Higher order derivatives of real-valued multivariate functions: D (p) p f n f (x) = x j1 x j2 x jp. j 1,...,j p=1 grad f (x) R n, Hess f (x) R n n,..., D (p) f (x) R n n. Moments of a vector-valued random variable x = (x 1,..., x n ): A 1 A q={j 1,...,j p} S p (x) = E(x j1 x j2 x jp ) n j 1,...,j p=1. Cumulants of a vector-valued random variable x = (x 1,..., x n ): X ««Q Q K p(x) = ( 1) q 1 (q 1)!E x j E x j n j A 1 j A q j 1,...,j p=1. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

5 Blaming the math Wired: Gaussian copulas for CDOs. NYT: normal market in VaR. January 4, 2009 Risk Mismanagement By JOE NOCERA THERE AREN T MANY widely told anecdotes about the current financial crisis, at least not yet, but there s one that made the rounds in 2007, back when the big investment banks were first starting to write down L.H. billions Lim (Berkeley) of dollars in mortgage-backed Most derivatives tensor problems and other are so-called NP hard toxic securities. This was August well before 25, / 21

6 Why not Gaussian Log characteristic function log E(exp(i t, x )) = α =1 i α κ α (x) tα α!. Gaussian assumption equivalent to quadratic approximation: If x is multivariate Gaussian, then = 2. log E(exp(i t, x )) = i E(x), t t Cov(x)t. K 1 (x) mean (vector), K 2 (x) variance (matrix), K 3 (x) skewness (3-tensor), K 4 (x) kurtosis (4-tensor),.... Non-Gaussian data: Not enough to look at just mean and covariance. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

7 Why not copulas Nassim Taleb: Anything that relies on correlation is charlatanism. Even if marginals normal, dependence might not be. X2 ~ N(0,1) 1000 Simulated Clayton(3) Dependent N(0,1) Values X1 ~ N(0,1) For heavy tail phenomena, important to examine tensor-valued quantities like kurtosis. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

8 Why not VaR Paul Wilmott: The relationship between two assets can never be captured by a single scalar quantity. Multivariate f : R n R f (x) = a 0 + a 1 x + x A 2 x + A 3 (x, x, x) + + A k (x,..., x) +, Hooke s law in 1D: x extension, F force, k spring constant, F = kx. Hooke s law in 3D: x = (x 1, x 2, x 3 ), elasticity tensor C R , stress Σ R 3 3, strain Γ R 3 3 σ ij = 3 k,l=1 c ijklγ kl. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

9 Numerical linear algebra F = R or C. A F m n, b F m, and r min{m, n}. Rank and numerical rank Determine rank(a). Linear system of equations Determine if Ax = b has a solution and if so determine a solution x F n. Linear least squares problem Determine an x F n that minimizes Ax b 2. Spectral norm Determine the value of A 2,2 := max x 2 =1 Ax 2 = σ max (A). Eigenvalue problem If m = n, determine λ F and non-zero x F n with Ax = λx. Singular value problem Determine σ F and non-zero x F n, y F m with Ax = σy, A y = σx. Low rank approximation Determine A r F m n with rank(a r ) r and A A r F = min rank(b) r A B F. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

10 Numerical multilinear algebra? Computing the spectral norm of a 3-tensor: sup x,y,z 0 A(x, y, z) x 2 y 2 z 2. Computing the spectral norm of a symmeric 3-tensor: sup x 0 S(x, x, x) x 3. 2 Computing a best rank-1 approximation to a tensor: min A x y z F. x,y,z Computing a best rank-1 approximation to a symmetric tensor: min x S x x x F. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

11 Numerical multilinear algebra? Determine if a given value is a singular value of a 3-tensor crit x,y,z 0 A(x, y, z) x 3 y 3 z 3. Determine if a given value is an eigenvalue of a symmetric 3-tensor crit x 0 S(x, x, x) x 3. 3 Solving a system of bilinear equations in the exact sense A(x, y, I ) = b or approximating it in the least-squares sense min x,y A(x, y, I ) b 2. Shorthand: A(x, y, z) = l,m,n i,j,k=1 a ijkx i y j z jk, A(x, y, I ) = l,m i,j=1 a ijkx i y j. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

12 Tensor rank = Innocent looking roblem Segre outer product is u v w := u i v j w k l,m,n i,j,k=1. A decomposable tensor is one that can be expressed as u v w. Definition (Hitchcock, 1927) Let A R l m n. Tensor rank is defined as rank (A) := min { r A = r U V W Hom(U, V W ). i=1 σ iu i v i w i }. Write A = [A 1,..., A l ] where A 1,..., A l R m n. Then rank (A) = min { r A 1,..., A l span{u 1 v 1,..., u r v r } } L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

13 Best low rank approximation of a matrix Given A R m n. Want argmin rank(b) r A B. More precisely, find σ i, u i, v i, i = 1,..., r, that minimizes Theorem (Eckart Young) A σ 1 u 1 v 1 σ 2 u 2 v 2 σ r u r v r. Let A = UΣV = rank(a) i=1 σ i u i v i be singular value decomposition. For r rank(a), let A r := r i=1 σ iu i v i. Then A A r F = min rank(b) r A B F. No such thing for tensors of order 3 or higher. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

14 Best low-rank approximation of a tensor Given A R l m n, find σ i, u i, v i, w i, minimizing A σ 1 u 1 v 1 w 1 σ 2 u 2 v 2 w 2 σ r u r v r w r. Surprise: In general, existence of solution guaranteed only if r = 1. Theorem (de Silva-L) Let k 3 and d 1,..., d k 2. For any s such that 2 s min{d 1,..., d k }, there exists A R d 1 d k with rank (A) = s such that A has no best rank-r approximation for some r < s. The result is independent of the choice of norms. Symmetric variant: A S(R n ), find λ i, v i minimizing A λ 1 v 1 v 1 v 1 λ 2 v 2 v 2 v 2 λ r v r v r v r. Also ill-behaved [Comon, Golub, L, Mourrain; 08]. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

15 Best low-rank approximation of a tensor Explanation: Set of tensors (resp. symmetric tensors) of rank (resp. symmetric rank) r is not closed unless r = 1. Another well behaved case: If σ i, u i, v i, w i constrained to nonnegative orthant, then a solution always exist [L & Comon; 09]. Nonnegative tensor approximations: General non-linear programming algorithms, cf. [Friedlander & Hatz; 08] and many others. On-going work with Kojima and Toh: SDP algorithm for convex relaxation of well-behaved cases. Best rank-1 approximation of a symmetric tensor also NP-hard [L & Hillar; 09] min x S x x x F. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

16 Variational approach to eigenvalues/vectors S R m n symmetric. Eigenvalues and eigenvectors are critical values and critical points of x Sx/ x 2 2. Equivalently, critical values/points of x Sx constrained to unit sphere. Lagrangian: L(x, λ) = x Sx λ( x 2 2 1). Vanishing of L at critical (x c, λ c ) R n R yields familiar Sx c = λ c x c. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

17 Eigenvalues/vectors of a tensor Extends to tensors [L; 05]. For x = [x 1,..., x n ] R n, write x p := [x p 1,..., x p n ]. Define the l p -norm x p = (x p x p n ) 1/p. Define eigenvalues/vectors of S S p (R n ) as critical values/points of the multilinear Rayleigh quotient S(x,..., x)/ x p p. Lagrangian L(x, λ) := S(x,..., x) λ( x p p 1). At a critical point S(I n, x,..., x) = λx p 1. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

18 Some observations If S is symmetric, S(I n, x, x,..., x) = S(x, I n, x,..., x) = = S(x, x,..., x, I n ). Defined in [Qi; 05] and [L; 05] independently. Related to rank-1 approximation: attained when min S λx x x F x λ = max S, x x x. x =1 For p = 3, λ is an eigenvalue of S iff x (S i λe i )x = 0, i = 1,..., n, for some non-zero x. Here S i = [s ijk ] n j,k=1 and E i = e i e i. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

19 Graph coloring as polynomial system Found in various forms [Bayer; 82], [Lovász; 94], [de Loera; 95]. Form below from [L & Hillar; 09]. Lemma Let G be a graph. The 3n + E polynomials in 2n + 1 indeterminates { x i y i z 2, y i z xi 2, x iz yi 2, i = 1,..., n, xi 2 + x i x j + xi 2, {i, j} E, has a common nontrivial complex solution if and only if the graph G is 3-colorable. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

20 Tensor eigenvalue is NP-hard Problem (SymmQuadFeas) Let S = s ijk S 3 (R n ), i.e. s ijk = s ikj = s jik = s jki = s kij = s kji. Let G i (x) = x S i x for i = 1,..., n be the associated n homogeneous, real quadratic forms. Determine if the system of equations {G i (x) = c i } n i=1 has a nontrivial real solution 0 x R n. Theorem (L & Hillar; 09) Graph coloring is polynomial reducible to SymmQuadFeas. Corollary Given symmetric tensor S S 3 (R n ). 1 Given λ R. NP hard to check if λ is eigenvalue of S. 2 NP hard to find a best rank-1 approximation λx x x to S. L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21

21 Caveat on complexity theory Different notions of NP-hardness: Cook-Karp-Levin (traditional); Blum-Shub-Smale; Valiant. Real versus bit complexity Real complexity: number of field operations required to determine feasibility as a function of n. Bit complexity: if matrices A has rational entries, number of bit operations necessary as a function of the number of bits required to specify all the a ijk. E.g. multiply two integers of size N, i.e. log N bits: real complexity is O(1), bit complexity is O(log N log log N). L.H. Lim (Berkeley) Most tensor problems are NP hard August 25, / 21