Tensor Analysis. Topics in Data Mining Fall Bruno Ribeiro

Transcription

1 Tensor Analysis Topics in Data Mining Fall 2015 Bruno Ribeiro

2 Tensor Basics But First 2

3 Mining Matrices 3

4 Singular Value Decomposition (SVD) } X(i,j) = value of user i for property j i 2 j 5 X(Alice, cholesterol) = 10 X(i,j) = number of times i buys j X(i,j) = how much i pays j 2 X(i,j) = 1 if i and j are friends, 0 otherwise X(i,j) = temperature of sensor j at time i 4

5 SVD Dimensions x 1 X U x 2 x m u 1 u 2 u k Σ V T v 1 v 2 σ 1 σ2 = σ k singular values v k Data Left singular vectors Right singular vectors 5

6 SVD Definition } SVD gives best rank-k approximation of X in L 2 and Frobenius norm σ 1 v 1 σ 2 v 2 u X = 1 + u 2 + Outer product 6

7 SVD Proper ties (I) } Almost unique decomposition σ 1 v 1 σ 2 v 2 u 1 u 2 X = + + } There are two sources of ambiguity Orientation of singular vectors Permute rows of left singular vector and corresponding rows of left singular vector If I is identity matrix: I = UIU T, for all orthonormal U Hypersphere ambiguity Related to rotational ambiguity of PCA 7

8 SVD Proper ties (II) } Theorem (Eckart-Young, 1936) UΣ 1 V T is best rank 1 approximation of X, that is X UΣ 1 V T 2 X Y 2 for every rank 1 matrix Y UΣ 1 V T + UΣ 2 V T is the best rank 2 approximation of X, that is X UΣ 1 V T UΣ 2 V T 2 X Y 2 for every rank 2 matrix Y also for 3, 4,, r 8

9 SVD Properties (III) U and V are orthogonal 9

10 Understanding SVD singular vectors } As U and V have orthogonal rows Now you explain: What do V and U represent? 10

11 My Help } } If X(i,j) = user i buys product j What is XTX? Product-to-product similarity matrix What does V represent? i 2 j 5 2 } What is XXT? User-to-user similarity matrix What does U represent? 11

12 Suggestion for SVD > 2 dimensions? } Simple extension: } Representation of X? 12

13 Tensor Motivation 13

14 App: Social Network Analysis } Traditional focus on static networks and find community structures } Tensors can monitor change of community structure over time Ack: Faloutsos,

15 App: Find Functional Relationships in Brain } See word ( apple ) } Answer fundamental human question ( Is it edible? ) } How do different parts of your brain communicate in the meanwhile? } Functional Connectivity c 1 c F nouns X b b F fmri voxels a 1 a F Ack: Papalexakis et al. SDM 2014 edible things tools 15

16 Tensor Decomposition (KruskalTensor) c 1 c 2 σ 1 b 1 σ 2 b 2 users X a = 1 + a 2 + products 16

17 Tensor Definitions } Krooneberg 1983 slices fibers 17

18 PARAFAC Decomposition (Harshman 1970) } Parallel Factors Decomposition σ 1 c 1 c 2 b 1 σ 2 b 2 } Columns of A, B, C not orthogonal I } If has r factors & r is minimal, then rank(x)= r } Important: possible that rank(x) > min(i,j,k) X J a = 1 + a 2 + } Decomposition often unique* *(Kruskal 1977): A, B, C are unique up to rescaling / counterscaling and joint permutations if where k A is k-th rank of A: max number k A such that every set of k columns of A are linearly independent 18

19 Tucker Tensor Decomposition (Tucker 1966) I x J x K I x r K x s X = A G r x s x t B 19

20 Tucker Decomposition Interpretation } user x product x time I x J x K I x r K x s } A = user x user factors X = A G r x s x t B } B = products x products factors } C = time x time factors } G = how these groups are mixed } Common Properties: A, B, C orthonormal G is not diagonal Decomposition not unique 20

21 Useful Rearrangements : Matricize tensor Tube fiber X (3) Row fiber X X (2) X Column fiber X (1) Vectorization 21

22 Products } 3-way outer product X Rank 1 tensor = a c b } Kronecker product (generalization of outer product) wikipedia } Khatri-Raoproduct (same elements but as a thinner matrix) M x r N x r MN x r 22

23 Solving Tucker: } Tucker Decomposition in Matrix form: } Error is then Tensor L 2 norm is square root of square sum of all elements Minimize = Maximize s.t.a, B, C or thonormal 23

24 Solving Tucker: } A, B, C have to be orthonormal } Ideas? Minimize X (1) A G (1) (C B) 2 This minimization can be reduced by solving first for G as G*=A T X (1) (C T B T ), and substituting G* into the loss function to obtain X (1) - A T A X (1) (C T B T )(C B) 2 24

25 Solving Tucker (II) } A simpler way to write it 25

26 Solving Tucker (III) } If B and C were fixed we could solve for A: } A has to be orthonormal } Algorithm? Optimal A from leading r left singular vectors of Same for B and C (Alternating Least Squares) } Algorithm (at step k): (Kroonenberg & De Leeuw 1980) A k = r leading left singular vectors of B k = s leading left singular vectors of C k = t leading left singular vectors of } Final step (K): get Not unique as EE T = I can be inserted before X (.) 26

27 Solving PARAFAC: } Same idea: Alternating Least Squares } With Fixed B and C Solve } Noting that Hadamard Product wikipedia 27

28 Next Class } Probabilistic interpretation } Non-negative tensor factorization } How tensors can be used for mixture model inference, topic modeling, etc. 28

29 } Acknowledgments: Faloutsos, Kolda, Sun ICML 07 29