Tensors and graphical models

Transcription

1 Tensors and graphical models Mariya Ishteva with Haesun Park, Le Song Dept. ELEC, VUB Georgia Tech, USA INMA Seminar, May 7, 2013, LLN

2 Outline Tensors Random variables and graphical models Tractable representations Structure learning

3 3 Tensors R M N P

4 Ranks Multilinear rank (R 1, R 2, R 3 ) Rank-R Rank-1 tensor: R = min(r), s.t. A = r {rank-1 tensor} i i=1

5 5 Matrix representations of tensors Mode-1 A = A (1) = Mode-2 Mode-3 Multilinear rank: (rank(a (1) ), rank(a (2) ), rank(a (3) ))

6 6 Tensor-matrix multiplication Tensor-matrix product Contraction A R I J M B R K L M M C = A, B 3 C(i, j, k, l) = a ijm b klm m=1 4 th order tensor C R I J K L

7 Basic decompositions Singular value decomposition (SVD) MLSVD / HOSVD CP / CANDECOMP / PARAFAC

8 8 Outline Tensors Random variables and graphical models Tractable representations Structure learning

9 9 Discrete random variables Random variable X; 1,..., n P x (1),..., P x (n) P x R n, R n +, [0, 1] X 1, X 2 ; P(X 1, X 2 ) 1 n 1 P 12 (1, 1) P 12 (1, n). n P 12 (n, 1) P 12 (n, n) P 12 R n n P(x 1, x 2 ) := P(X 1 = x 1, X 2 = x 2 )

10 10 2 random variables X 1, X 2 ; P(X 1, X 2 ) P 12 R n n X 1 X 2 P(x 1, x 2 ) = P(x 1 )P(x 2 ) rank-1 matrix = H X 1 X 2 P(x 1, x 2 ) = h P(x 1 h)p(x 2 h)p(h) low-rank matrix rank-k matrix, k < n = Conditional probability tables (CPTs) P(X 1 H), P(X 2 H)

11 3 random variables X 1, X 2, X 3 ; P(X 1, X 2, X 3 ) P 123 R n n n X 1, X 2, X 3 independent P(x 1, x 2, x 3 ) = P(x 1 )P(x 2 )P(x 3 ) rank-1 tensor = rank-k tensor, k < n H X 1 X 2 X 3 = = P(x 1, x 2, x 3 ) = h P(x 1 h)p(x 2 h)p(x 3 h)p(h)

12 4 random variables X 1, X 2, X 3, X 4 ; P(X 1, X 2, X 3, X 4 ) P 1234 R n n n n X 1, X 2, X 3, X 4 independent H X 1 X 2 X 3 X 4 P(x 1, x 2, x 3, x 4 ) = h P(x 1 h)p(x 2 h)p(x 3 h)p(x 4 h)p(h) more variables more hidden variables

13 13 Challenges 10 variables, 10 states each entries We need tractable representations Latent variable models / low-rank factors # parameters: exponential polynomial H X 1 X 1 X X 1 X 1 X 1 Challenges: Choose a good representation Learn the correct structure Estimate the parameters

14 Outline Tensors Random variables and graphical models Tractable representations Structure learning

15 15 Tensors and graphical models CP / CANDECOMP / PARAFAC H X 1 X 2 Xn Tensor train H 1 H 2 H 3 Hn X 1 X 2 X 3 Xn HMM Hierarchical Tucker H X 1 X 1 X X 1 X 1 X 1 Latent tree model Tucker / MLSVD Block term decomposition

16 16 Tensor train (TT) decomposition A(i 1,...,i d )= α 0,...,α d G 1 (α 0, i 1,α 1 )G 2 (α 1, i 2,α 2 )...G d (α d 1, i d,α d ) [I. V. Oseledets, SIAM J. Scientific Computing, 2011] Avoids curse of dimensionality Small number of parameters, compared to Tucker model Slightly more parameters than CP but more stable G k (α k 1, n k,α k ) has dimensions r k 1 n k r k, r 0 = r d = 1 r k are called compression ranks: A k = A k (i 1,...,i k ; i k+1,...,i d ), rank(a k ) = r k Computation based on SVD Computation: top bottom H 1 H 2 H 3 Hn X 1 X 2 X 3 Xn

17 Hierarchical Tucker decomposition [L. Grasedyck, SIMAX, 2010] Similar properties as TT decomposition Computation: bottom top H X 1 X 1 X X 1 X 1 X 1

18 18 Potential advantages of tensor approach Real data are often multi-way Provides higher-level view Flexibility: different ranks in each mode: Tucker Uniqueness: CP, Block term decomposition No curse of dimensionality: Tensor train, hierarch. Tucker

19 19 Outline Tensors Random variables and graphical models Tractable representations Structure learning

20 20 Structure learning Given: (samples of) observed variables Assumption: the variables can be connected via hidden variables in a tree structure in a meaningful way Find: the tree / the relationships between the variables Additional difficulty: unknown number of hidden states? H H H X X X X X 3 X 5 X 2 X 1 X X 1 X 1 X 1 X 1 X 1

21 Quartet relationships: topologies X 1 X 3 X 1 X 2 X 1 X 2 H G H G H G X 2 X 4 X 3 X 4 X 4 X 3 P(x 1, x 2, x 3, x 4 ) = h,g P(x 1 h)p(x 2 h)p(h, g)p(x 3 g)p(x 4 g)

22 Building trees based on quartet relationships Choose 3 variables and form a tree Add all other variables, one by one Split the current tree into 3 subtrees Choose 3 variables from different subtrees Resolve the quartet relation with current and chosen variables Insert the current variable in a subtree or connect to the tree [For simplicity, assume each latent variable has 3 neighbors]

23 23 Tensor view of quartets X 1 X 3 H G X 2 X 4 P 1 H P 4 G P(X 1,X 2,X 3,X 4 ) = P 2 H IH PHG IG P 3 G A = reshape(p, n 2, n 2 ); B = reshape(permute(p,[1, 3, 2, 4]), n 2, n 2 ); C = reshape(permute(p,[1, 4, 2, 3]), n 2, n 2 ). Notation: P 1 H, P 2 H, etc. stand for P(X 1 H), P(X 2 H), etc.

24 Rank properties of matrix representations A = B = P 2 H P 1 H P HG P 4 G P 3 G ( ( ( P 3 G P 1 H diag(p HG (:)) P 4 G P 2 H ( ( ( rank(a) = rank(p HG ) = k rank(b) = rank(c) = nnz(p HG ) rank(a) rank(b) = rank(c) Sampling noise ( ( Nuclear norm relaxation A = n 2 i=1 σ i(a)

25 25 Resolving quartet relations Algorithm 1 i = Quartet(X 1, X 2, X 3, X 4 ) 1: Estimate P(X 1, X 2, X 3, X 4 ) from a set of m i.i.d. samples. 2: Unfold P into matrices Â, B and Ĉ, and compute a 1 = Â, a 2 = B and a 3 = Ĉ. 3: Return i = arg min i {1,2,3} a i. Easy to compute Recovery conditions Finite sample guarantees Agnostic to the number of hidden states Compares favorably to alternatives

26 26 Example: stock data Given: stock prices (25 years, discretized into 10 values) Find: relations between stocks Finance: C (Citigroup) JPM (JPMorgan Chase) AXP (American Express) F (Ford Motor: Automotive and Financial Services) Retailers: TGT (Target) WMT (WalMart) RSH (RadioShack)

27 Conclusions Tensor decompositions are related to graphical models A common goal: tractable representations Tensors can be used for structure learning

28 28 Thank you!