Three right directions and three wrong directions for tensor research

Three right directions and three wrong directions for tensor research Michael W. Mahoney Stanford University ( For more info, see: http:// cs.stanford.edu/people/mmahoney/ or Google on Michael Mahoney )

Lots and lots of large data! High energy physics experimental data Hyperspectral medical and astronomical image data DNA microarray data and DNA SNP data Medical literature analysis data Collaboration and citation networks Internet networks and web graph data Advertiser-bidded phrase data Static and dynamic social network data

Scientific and Internet data individuals SNPs AG AG AG AG AA CC GG AG CG AC CC AA CC AA GG TT AG CT CG CG CG AT CT CT AG CT AA AG AG AG AA CC AG GG CC AC CC AA CG AA GG TT AG CT CG CG CG AT CT CT AG CT AA GG GG GG AA CT GG AA CC AC CG AA CC AA GG TT GG CC CG CG CG AT CT CT AG CT AG AG AG AG AA CT GG AG CC CC CG AA CC AA GT TT AG CT CG CG CG AT CT CT AG CT AA AG AG AG AA CC AG AG CG AA CC AA CG AA GG TT AA TT GG GG GG TT TT CC GG TT

Algorithmic vs. Statistical Perspectives Lambert (2000) Computer Scientists Data: are a record of everything that happened. Goal: process the data to find interesting patterns and associations. Methodology: Develop approximation algorithms under different models of data access since the goal is typically computationally hard. Statisticians Data: are a particular random instantiation of an underlying process describing unobserved patterns in the world. Goal: is to extract information about the world from noisy data. Methodology: Make inferences (perhaps about unseen events) by positing a model that describes the random variability of the data around the deterministic model.

Matrices and Data individuals SNPs AG AG AG AG AA CC GG AG CG AC CC AA CC AA GG TT AG CT CG CG CG AT CT CT AG CT AA AG AG AG AA CC AG GG CC AC CC AA CG AA GG TT AG CT CG CG CG AT CT CT AG CT AA GG GG GG AA CT GG AA CC AC CG AA CC AA GG TT GG CC CG CG CG AT CT CT AG CT AG AG AG AG AA CT GG AG CC CC CG AA CC AA GT TT AG CT CG CG CG AT CT CT AG CT AA AG AG AG AA CC AG AG CG AA CC AA CG AA GG TT AA TT GG GG GG TT TT CC GG TT Matrices provide simple representations of data: A ij = 0 or 1 (perhaps then weighted), depending on whether word i appears in document j A ij = -1,0,+1, if homozygous for the major allele, heterozygous, or homozygous for the minor allele Can take advantage of nice properties of vector spaces: structural properties: SVD, Euclidean geometry algorithmic properties: everything is O(n 3 ) statistical properties: PCA, regularization, etc.

Graphs and Data Interaction graph model of networks: Nodes represent entities Edges represent interaction between pairs of entities Common variations include: Directed graphs Weighted graphs Bipartite graphs

Why model data as graphs and matrices? Graphs and matrices - provide natural mathematical structures that provide algorithmic, statistical, and geometric benefits provide nice tradeoff between rich descriptive framework and sufficient algorithmic structure provide regularization due to geometry, either explicitly due to R n or implicitly due to approximation algorithms

What if graphs/matrices don t work? Employ more general mathematical structures: Hypergraphs Attributes associated with nodes Kernelize the data using, e.g., a similarity notion Generalized linear or hierarchical models Tensors!! These structures provide greater descriptive flexibility, that typically comes at a (moderate or severe) computational cost.

What is a tensor? (1 of 3) See L.H.Lim s tutorial on tensors at MMDS 2006.

What is a tensor? (2 of 3)

What is a tensor? (3 of 3) IMPORTANT: This is similar to NLA --- but, there is no reason to expect the subscript manipulation methods, so useful in NLA, to yield anything meaningful for more general algebraic structures.

Tensor ranks and data analysis (1 of 3)

Tensor ranks and data analysis (2 of 3) IMPORTANT: These ill-posedness results are NOT pathological--- they are ubiquitous and essential properties of tensors.

Tensor ranks and data analysis (3 of 3) THAT IS: To get a simple or low-rank tensor approximation, we focus on exceptions to fundamental ill-posedness properties of tensors (i.e., rank-1 tensors and 2-mode tensors).

Historical Perspective on NLA NLA grew out of statistics (among other areas) (40s and 50s) NLA focuses on numerical issues (60s, 70s, and 80s) Large-scale data generation increasingly common (90s and 00s) NLA has suffered due to the success of PageRank and HITS. Large-scale scientific and Internet data problems invite us to take a broader perspective on traditional NLA: revisit algorithmic basis of common NLA matrix algorithms revisit statistical underpinnings of NLA expand traditional NLA view of tensors

The gap between NLA and TCS Matrix factorizations: in NLA and scientific computing - used to express a problem s.t. it can be solved more easily. in TCS and statistical data analysis - used to represent structure that may be present in a matrix obtained from object-feature observations. NLA: emphasis on optimal conditioning, backward error analysis issues, is running time a large or small constant multiplied by n 2 or n 3. TCS: motivated by large data applications space-constrained or pass-efficient models over-sampling and randomness as computational resources. MMDS06, MMDS08, were designed to bridge the gap between NLA, TCS, and data applications.

How to bridge the gap (Lessons from MMDS) In a vector space, everything is easy, multi-linear captures the inherent intractability of NP-hardness. Convexity is an appropriate generalization of linear nice algorithmic framework, as with kernels in Machine Learning Randomness, over-sampling, approximation... are powerful algorithmic resources but you need to have a clear objective you are solving Geometry of combinatorial objects (e.g., graphs, tensors, etc.) has positive algorithmic, statistical, and conceptual benefits Approximate computation induces implicit statistical regularization

Examples of tensor data (Acar and Yener 2008) Chemistry: model fluorescence excitation-emission data in food science: A ijk is samples x emission x excitation. Neuroscience: EEG data as patients, doses, conditions, etc. varied: A ijk is time samples x frequency x electrodes. Social network and Web analysis: to discover hidden structures: A ijk is webpages x webpages x anchor text. A ijk is users x queries x webpages. A ijk is advertisers x bidded-phrases x time. Computer Vision: image compression and face recognition: A ijk is pixel x illumination x expression x viewpoint x person. Quantum mechanics, large-scale computation, hyperspectral data, climate data, ICA, nonnegative data, blind source separation, NP-hard problems, Tensor-based data are particularly challenging due to their size and since many data analysis tools based on graph theory and linear algebra do not easily generalize. -- MMD06

Three Right Directions

Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.)

Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.)

Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.) 3. Understand WHY tensors work in physical applications and what this says about less structured data applications (and vice-versa, which has been very fruitful for matrices*.) *(E.g., low-rank off-diagonal blocks are common in matrices -- since the world is 3D, which is not true in less structured applications -- this has significant algorithmic implications.)

Four! Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.) 3. Understand WHY tensors work in physical applications and what this says about less structured data applications (and vice-versa, which has been very fruitful for matrices*.) 4. Understand unfolding as a process of defining features. (Since this puts you in a nice algorithmic place.) *(E.g., low-rank off-diagonal blocks are common in matrices -- since the world is 3D, which is not true in less structured applications -- this has significant algorithmic implications.)

Three Wrong directions

Three Wrong directions 1. Viewing tensors as matrices with additional subscripts. (That may be true, but it hampers you, since R n is so nice.)

Three Wrong directions 1. Viewing tensors as matrices with additional subscripts. (That may be true, but it hampers you, since R n is so nice.) 2. Using methods that damage geometry and enhance sparsity. (BTW, you will do this if you don t understand the underlying geometric and sparsity structure.)

Three Wrong directions 1. Viewing tensors as matrices with additional subscripts. (That may be true, but it hampers you, since R n is so nice.) 2. Using methods that damage geometry and enhance sparsity. (BTW, you will do this if you don t understand the underlying geometric and sparsity structure.) 3. Doing Applied Ramsey Theory : Theorem: Given a large enough universe of data, then for any algorithm there exists a data set s.t. it performs well. (Show me where your method fails AND where it succeeds! Otherwise, is your result about your data or your method? Of course, this applies more generally in data analysis.)

Conclusions Large-scale data applications have been the main driver for a lot of the interest in tensors. Tensors are tricky to deal with, both algorithmically and statistically. Let s use this meeting to refine my directions in light of motivating data applications.