Must-read Material : Multimedia Databases and Data Mining. Indexing - Detailed outline. Outline. Faloutsos

Transcription

1 Must-read Material : Multimedia Databases and Data Mining Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. Technical Report SAND , Sandia National Laboratories, Albuquerque, NM and Livermore, CA, November 2007 Lecture #21: Tensor decompositions C. Faloutsos 2 Outline Indexing - Detailed outline Goal: Find similar / interesting things Intro to DB Indexing - similarity search Data Mining 3 primary key indexing secondary key / multi-key indexing spatial access methods fractals text Singular Value Decomposition (SVD) - - Tensors multimedia 4 1

2 Most of foils by Dr. Tamara Kolda (Sandia N.L.) csmr.ca.sandia.gov/~tgkolda Dr. Jimeng Sun (CMU -> IBM) Outline Motivation - Definitions Tensor tools Case studies 3h tutorial: Motivation 0: Why matrix? Why matrices are important? John Peter Mary Nick Examples of Matrices: Graph - social network John Peter Mary Nick

3 Examples of Matrices: cloud of n-d points Examples of Matrices: Market basket market basket as in Association Rules John Peter Mary Nick chol# blood# age.. John Peter Mary Nick milk bread choc. wine 9 10 Examples of Matrices: Documents and terms Examples of Matrices: Authors and terms Paper#1 Paper#2 Paper#3 Paper#4 data mining classif. tree John Peter Mary Nick data mining classif. tree

4 Examples of Matrices: sensor-ids and time-ticks Motivation: Why tensors? Q: what is a tensor? t1 t2 t3 t4 temp1 temp2 humid. pressure Motivation 2: Why tensor? A: N-D generalization of matrix: Motivation 2: Why tensor? A: N-D generalization of matrix: KDD 05 KDD 06 KDD 07 data mining classif. tree KDD 07 data mining classif. tree John Peter Mary Nick John Peter Mary Nick

5 Tensors are useful for 3 or more modes Terminology: mode (or aspect ): Mode#3 Motivating Applications Why matrices are important? Why tensors are useful? P1: social networks P2: web mining data mining classif. tree Mode#2 Mode (== aspect) # P1: Social network analysis Traditionally, people focus on static networks and find community structures We plan to monitor the change of the community structure over time P2: Web graph mining How to order the importance of web pages? Kleinberg s algorithm HITS PageRank Tensor extension on HITS (TOPHITS) context-sensitive hypergraph analysis

6 Motivation Definitions Tensor tools Case studies Outline Tensor Basics Tucker PARAFAC Tensor Basics 21 Reminder: SVD Reminder: SVD n n n σ 1 u 1 v 1 σ 2 u 2 v 2 m A m Σ V T m A + U Best rank-k approximation in L2 Best rank-k approximation in L

7 Goal: extension to >=3 modes C K x R I x R J x R ~ B = + + A R x R x R Main points: 2 major types of tensor decompositions: PARAFAC and Tucker both can be solved with ``alternating least squares (ALS) Details follow Specially Structured Tensors Tucker Tensor Kruskal Tensor Specially Structured Tensors Our Notation Our Notation core W K x T W K x R I x R J x S w 1 I x R w R J x R = U V R x S x T = = u 1 v 1 U + + V R x R x R u R v R 28 7

8 details Specially Structured Tensors Tucker Tensor Kruskal Tensor Tensor Decompositions In matrix form: In matrix form: 29 Tucker Decomposition - intuition C K x T I x R J x S ~ B A R x S x T Intuition behind core tensor 2-d case: co-clustering [Dhillon et al. Information-Theoretic Coclustering, KDD 03] author x keyword x conference A: author x author-group B: keyword x keyword-group C: conf. x conf-group : how groups relate to each other Needs elaboration!

9 med. doc cs doc m k k l l m n n eg, terms x documents term group x doc. group doc x doc group med. terms cs terms common terms 33 term x term-group 34 Tucker Decomposition C K x T I x R J x S ~ B A R x S x T Given A, B, C, the optimal core is: Proposed by Tucker (1966) AKA: Three-mode factor analysis, three-mode PCA, orthogonal array decomposition A, B, and C generally assumed to be orthonormal (generally assume they have full column rank) is not diagonal Not unique Recall the equations for converting a tensor to a matrix

10 Motivation Definitions Tensor tools Case studies Outline Tensor Basics Tucker PARAFAC CANDECOMP/PARAFAC Decomposition ¼ I x R A C K x R R x R x R J x R B = CANDECOMP = Canonical Decomposition (Carroll & Chang, 1970) PARAFAC = Parallel Factors (Harshman, 1970) Core is diagonal (specified by the vector λ) Columns of A, B, and C are not orthonormal If R is minimal, then R is called the rank of the tensor (Kruskal 1977) Can have rank( ) > min{i,j,k} 38 Tucker vs. PARAFAC Decompositions Tucker Variable transformation in each mode Core G may be dense A, B, C generally orthonormal Not unique C K x T IMPORTANT PARAFAC Sum of rank-1 components No core, i.e., superdiagonal core A, B, C may have linearly dependent columns Generally unique Tensor tools - summary Two main tools PARAFAC Tucker Both find row-, column-, tube-groups but in PARAFAC the three groups are identical To solve: Alternating Least Squares I x R J x S ~ B A R x S x T ~ c 1 b c R b R a 1 a R 39 Toolbox: from Tamara Kolda:

11 Outline Motivation - Definitions Tensor tools Case studies P1: Web graph mining How to order the importance of web pages? Kleinberg s algorithm HITS PageRank Tensor extension on HITS (TOPHITS) P1: Web graph mining Kleinberg s Hubs and Authorities (the HITS method) T. G. Kolda, B. W. Bader and J. P. Kenny, Higher-Order Web Link Analysis Using Multilinear Algebra, ICDM 2005: ICDM, pp , November 2005, doi: /icdm [PDF] Sparse adjacency matrix and its SVD: to from 43 Kleinberg, JACM,

12 HITS Authorities on Sample Data Three-Dimensional View of the Web We started our crawl from and crawled 4700 pages, resulting in 560 cross-linked hosts. Observe that this tensor is very sparse! to from 45 Kolda, Bader, Kenny, ICDM05 46 Topical HITS (TOPHITS) Main Idea: Extend the idea behind the HITS model to incorporate term (i.e., topical) information. Topical HITS (TOPHITS) Main Idea: Extend the idea behind the HITS model to incorporate term (i.e., topical) information. term term scores term scores term term scores term scores to to from from

13 TOPHITS Terms & Authorities on Sample Data TOPHITS uses 3D analysis to find the dominant groupings of web pages and terms. w k = # unique links using term k GigaTensor: Scaling Tensor Analysis Up By 100 Times Algorithms and Discoveries U Kang Evangelos Papalexakis Abhay Harpale Christos Faloutsos Tensor PARAFAC term term scores term scores KDD 2012 to from 49 P2: N.E.L.L. analysis NELL: Never Ending Language Learner Q1: dominant concepts / topics? Q2: synonyms for a given new phrase? A1: Concept Discovery Concept Discovery in Knowledge Base Eric Clapton plays guitar Barrack Obama is the president of U.S. (48M) (26M) (26M) NELL (Never Ending Language Learner) Nonzeros =144M 51 13

14 A1: Concept Discovery A2: Synonym Discovery Synonym Discovery in Knowledge Base a 1 a 2 a R (Given) subject (Discovered) synonym 1 (Discovered) synonym 2 A2: Synonym Discovery Conclusions Real data are often in high dimensions with multiple aspects (modes) Matrices and tensors provide elegant theory and algorithms c 1 c R ~ b b R a 1 a R 56 14

15 References Inderjit S. Dhillon, Subramanyam Mallela, Dharmendra S. Modha: Information-theoretic co-clustering. KDD 2003: T. G. Kolda, B. W. Bader and J. P. Kenny. Higher- Order Web Link Analysis Using Multilinear Algebra. In: ICDM 2005, Pages , November Jimeng Sun, Spiros Papadimitriou, Philip Yu. Windowbased Tensor Analysis on High-dimensional and Multiaspect Streams, Proc. of the Int. Conf. on Data Mining (ICDM), Hong Kong, China, Dec