ENGG5781 Matrix Analysis and Computations Lecture 10: Non-Negative Matrix Factorization and Tensor Decomposition
|
|
- Lenard Calvin McKenzie
- 5 years ago
- Views:
Transcription
1 ENGG5781 Matrix Analysis and Computations Lecture 10: Non-Negative Matrix Factorization and Tensor Decomposition Wing-Kin (Ken) Ma Term 2 Department of Electronic Engineering The Chinese University of Hong Kong
2 Topic 1: Nonnegative Matrix Factorization 1
3 Nonnegative Matrix Factorization Consider again the low-rank factorization problem min Y AB 2 A R m r,b R r n F. The solution is not unique: if (A, B ) is a solution to the above problem, then (A Q T, QB ) for any orthogonal Q is also a solution. Nonnegative Matrix Factorization (NMF): min Y AB 2 A R m r,b R r n F s.t. A 0, B 0 where X 0 means that X is elementwise non-negative. found to be able to extract meaningful features (by empirical studies) under some conditions, the NMF solution is provably unique numerous applications, e.g., in machine learning, signal processing, remote sensing W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 2
4 As can be seen from Fig. 1, the NMF basis and encodings contain a large fraction of vanishing coefficients, so both the basis images and image encodings are sparse. The basis images are sparse because they are non-global and NMF contain Examples several versions of mouths, noses and other facial parts, where the various versions are in different Image Processing: locations or forms. The variability of a whole face is generated by combining these different parts. Although all parts are used by at A 0 constraints the basis elements to be nonnegative. B 0 imposes an additive reconstruction. update rules yield the technical disadv controlling the lea through trial and the matrix V is ver Fig. 2 may be adv bases. NMF Original = Figure 1 Non-negative m faces, whereas vector q holistic representations. m ¼ 2;429 facial imag n m matrix V. All thre three different types of c and methods. As shown r ¼ 49 basis images. P with red pixels. A partic represented by a linear superposition are shown superpositions are show learns to represent face the basis elements extract facial features such as eyes, nose and lips. Source: VQ [Lee-Seung1999] W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 3
5 Text Mining basis elements allow us to recover different topics; weights allow us to assign each text to its corresponding topics. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 4
6 NMF NP-hard in general min Y AB 2 A R m r,b R r n F s.t A 0, B 0. a practical way to go: alternating optimization w.r.t. A, B given B, minimizing Y AB 2 F over A 0 is convex; given A, minimizing Y AB 2 F over B 0 is also convex despite that, there is no closed-form solution with min A 0 Y AB 2 F or min B 0 Y AB 2 F ; it takes time to solve them when Y is big state of the art, such as the Lee-Seung multiplicative update, applies inexact alternating optimization W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 5
7 Toy Demonstration of NMF A face image dataset. Image size = , number of face images = Each x n is the vectorization of one face image, leading to m = = 10201, N = W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 6
8 Toy Demonstration of NMF: NMF-Extracted Features NMF settings: r = 49, Lee-Seung multiplicative update with 5000 iterations. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 7
9 Toy Demonstration of NMF: Comparison with PCA Mean face 1st principal left singular vector 2nd principal left singular vector 3th principal left singular vector last principal left singular vector Energy Rank Energy Concentration W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 8
10 Suggested Further Readings for NMF [Gillis2014] for a review of some classical NMF algorithms and separable NMF separable NMF is an NMF subclass that features provably polynomial-time or very simple algorithms for solving NMF under certain assumptions [Fu-Huang-Sidiropoulos-Ma2018] for an overview of classic and most recent developments of NMF identifiability note volume minimization NMF, which has provably better identifiability than classic NMF W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 9
11 Topic 2: Tensor Decomposition W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 10
12 Tensor A tensor is a multi-way numerical array. R I 1 I 2... I N and its entries by x i1,i 2,...,i N. An N-way tensor is denoted by X natural extension of matrices (which are two-way tensors) example: a color picture is a 3-way tensor, video 4-way Color = applications: blind signal separation, chemometrics, data mining,... focus: decomposition for 3-way tensors X R I J K (sufficiently complicated) W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 11
13 Outer Product The outer product of a R I and b R J is an I J matrix a b = ab T = [ b 1 a, b 2 a,..., b J a ]. Here, is used to denote the outer product operator.... W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 12
14 Outer Product The outer product of a matrix A R I J and a vector c R K, denoted by A c, is a three-way I J K tensor that takes the following form:... Specifically, if we let X = A c, then X (:, :, k) = c k A, k = 1,..., K. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 13
15 Outer Product Tensors in the form of a b c are called rank-1 tensors. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 14
16 Tensor Decomposition Problem: decompose X R I J K as x i,j,k = R a il b jl c kl, r=1 for some a ir, b jr, and c kr, i = 1,..., I, j = 1,..., J, k = 1,..., K, r = 1,..., R, or equivalently, R X = a r b r c r, r=1 where A = [a 1,..., a R ] R I R, B = [b 1,..., b R ] R J R, C = [c 1,..., c R ] R K R. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 15
17 Tensor Decomposition X = R a r b r c r, r=1 ( )... a sum of rank-1 tensors the smallest R that satisfies ( ) is called the rank of the tensor many names: tensor rank decomposition, canonical polyadic decomposition (CPD), parallel factor analysis (PARARFAC), CANDECOMP W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 16
18 Application Example: Enron Data About Enron once ranked the 6th largest energy company in the world shares were worth $90.75 at their peak in Aug 2000 and dropped to $0.67 in Jan 2002 most top executives were tried for fraud after it was revealed in Nov 2001 that Enron s earning has been overstated by several hundred million dollars Enron database a large database of over 600,000 s generated by 158 employees of the Enron Corporation and acquired by the Federal Energy Regulatory Commission during its investigation after the company s collapse, according to wiki W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 17
19 Application Example: Enron Data Source: W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 18
20 Uniqueness of Tensor Decomposition The low-rank matrix factorization problem X = AB is not unique. We also have many matrix decompositions: SVD, QR,... Theorem Let (A, B, C) be a factor for X = R r=1 a r b r c r. If krank(a) + krank(b) + krank(c) 2R + 2, then (A, B, C) is the unique tensor decomposition factor for X up to a common column permutation and scaling. Implication: under some mild conditions with A, B, C, low-rank tensor decomposition is essentially unique the above theorem is just among one of the known sufficient conditions for unique tensor decomposition; some other results suggest much more relaxed conditions for unique tensor decomposition W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 19
21 Slabs A slab of a tensor is a matrix obtained by fixing one index of the tensor. Horizontal slabs: {X (1) i = X (i, :, :)} I i=1 Lateral slabs: {X (2) j = X (:, j, :)} J j=1 Frontal slabs: {X (3) k = X (:, :, k)} K k=1 Horizontal slabs Lateral slabs Frontal slabs W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 20
22 PARAFAC Formulation Consider the horizontal slabs as an example. X (1) i = R a i,r (b r c r ) = r=1 R a i,r b r c T r r=1 We can write X (1) i = BD A(i,:) C T, D A(i,:) = Diag(A(i, :)). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 21
23 PARAFAC Formulation Khatri-Rao (KR) product: the KR product of A R I R and B R J R is A B = [a 1 b 1,..., a R a R ]. A key KR property: let D = diag(d). We have vec(adb T ) = (B A)d. Idea: recall the horizontal slabs expression It can be reexpressed as X (1) i = BD A(i,:) C T, D A(i,:) = Diag(A(i, :)). vec(x (1) i ) = (C B)A T (i, :). roughly speaking, we can do A T (i, :) = (C B) vec(x (1) i ). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 22
24 PARAFAC Formulation By the trick above, we can write Horizontal slabs: X (1) i Lateral slabs: X (2) j Frontal slabs: X (3) k = X (i, :, :) = BD A(i,:) C T vec(x (1) i ) = (C B)A T (i, :) = X (:, j, :) = CD B(j,:) A T vec(x (2) j ) = (A C)B T (j, :) = X (:, :, k) = AD C(k,:) B T vec(x (3) k ) = (B A)CT (k, :) Observation: fixing B, C, solving for A is a linear system problem fixing A, C, solving for B is a linear system problem fixing A, B, solving for C is a linear system problem W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 23
25 PARAFAC Formulation A tensor decomposition formulation: given X R I J K, R > 0, solve min A,B,C R X 2 a r b r c r r=1 F, NP-hard in general can be conveniently handled by alternating optimization w.r.t. A, B, C e.g., optimization w.r.t. A and fixing B, C is min A I i=1 X (1) i BD A(i,:) C T 2 F = min A I i=1 vec(x (1) i ) (C B)A T (i, :) 2 2 and a solution is (A (i, :)) T = (C B) vec(x (1) i ), i = 1,..., I. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 24
26 Suggested Further Readings for Tensor Decomposition [Sidiropoulos-De Lathauwer-Fu-Huang-Papalexakis-Faloutsos2017] W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 25
27 References [Lee-Seung1999] D.D. Lee and H.S. Seung. Learning the parts of objects by non-negative matrix factorization, Nature, [Gillis2014] N. Gillis, The Why and how of nonnegative matrix factorization, in Regularization, Optimization, Kernels, and Support Vector Machines, J.A.K. Suykens, M. Signoretto and A. Argyriou (eds), Chapman & Hall/CRC, Machine Learning and Pattern Recognition Series, [Fu-Huang-Sidiropoulos-Ma2018] X. Fu, K. Huang, N. D. Sidiropoulos, and W.-K. Ma, Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications, arxiv preprint, Online avaiable: https: //arxiv.org/abs/ [Sidiropoulos-De Lathauwer-Fu-Huang-Papalexakis-Faloutsos2017] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos, Tensor decomposition for signal processing and machine learning, IEEE Trans. Signal Process., W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, Term 2. 26
ENGG5781 Matrix Analysis and Computations Lecture 0: Overview
ENGG5781 Matrix Analysis and Computations Lecture 0: Overview Wing-Kin (Ken) Ma 2018 2019 Term 1 Department of Electronic Engineering The Chinese University of Hong Kong Course Information W.-K. Ma, ENGG5781
More informationA randomized block sampling approach to the canonical polyadic decomposition of large-scale tensors
A randomized block sampling approach to the canonical polyadic decomposition of large-scale tensors Nico Vervliet Joint work with Lieven De Lathauwer SIAM AN17, July 13, 2017 2 Classification of hazardous
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf-Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2018/19 Part 6: Some Other Stuff PD Dr.
More informationENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition
ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Lecture 8: QR Decomposition
More informationIntroduction to Tensors. 8 May 2014
Introduction to Tensors 8 May 2014 Introduction to Tensors What is a tensor? Basic Operations CP Decompositions and Tensor Rank Matricization and Computing the CP Dear Tullio,! I admire the elegance of
More informationCoupled Matrix/Tensor Decompositions:
Coupled Matrix/Tensor Decompositions: An Introduction Laurent Sorber Mikael Sørensen Marc Van Barel Lieven De Lathauwer KU Leuven Belgium Lieven.DeLathauwer@kuleuven-kulak.be 1 Canonical Polyadic Decomposition
More informationCVPR A New Tensor Algebra - Tutorial. July 26, 2017
CVPR 2017 A New Tensor Algebra - Tutorial Lior Horesh lhoresh@us.ibm.com Misha Kilmer misha.kilmer@tufts.edu July 26, 2017 Outline Motivation Background and notation New t-product and associated algebraic
More informationRank Selection in Low-rank Matrix Approximations: A Study of Cross-Validation for NMFs
Rank Selection in Low-rank Matrix Approximations: A Study of Cross-Validation for NMFs Bhargav Kanagal Department of Computer Science University of Maryland College Park, MD 277 bhargav@cs.umd.edu Vikas
More information矩阵论 马锦华 数据科学与计算机学院 中山大学
矩阵论 马锦华 数据科学与计算机学院 中山大学 About The Instructor Education - Bachelor and master in mathematics, SYSU - PhD in computer science, HKBU Research Experience - Postdoc at Rutgers - Postdoc at Johns Hopkins - Postdoc
More informationSemidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization
Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization Nicolas Gillis nicolas.gillis@umons.ac.be https://sites.google.com/site/nicolasgillis/ Department
More informationDealing with curse and blessing of dimensionality through tensor decompositions
Dealing with curse and blessing of dimensionality through tensor decompositions Lieven De Lathauwer Joint work with Nico Vervliet, Martijn Boussé and Otto Debals June 26, 2017 2 Overview Curse of dimensionality
More information1 Non-negative Matrix Factorization (NMF)
2018-06-21 1 Non-negative Matrix Factorization NMF) In the last lecture, we considered low rank approximations to data matrices. We started with the optimal rank k approximation to A R m n via the SVD,
More informationLecture Notes 10: Matrix Factorization
Optimization-based data analysis Fall 207 Lecture Notes 0: Matrix Factorization Low-rank models. Rank- model Consider the problem of modeling a quantity y[i, j] that depends on two indices i and j. To
More informationCS264: Beyond Worst-Case Analysis Lecture #15: Topic Modeling and Nonnegative Matrix Factorization
CS264: Beyond Worst-Case Analysis Lecture #15: Topic Modeling and Nonnegative Matrix Factorization Tim Roughgarden February 28, 2017 1 Preamble This lecture fulfills a promise made back in Lecture #1,
More informationMulti-Way Compressed Sensing for Big Tensor Data
Multi-Way Compressed Sensing for Big Tensor Data Nikos Sidiropoulos Dept. ECE University of Minnesota MIIS, July 1, 2013 Nikos Sidiropoulos Dept. ECE University of Minnesota ()Multi-Way Compressed Sensing
More informationLinear dimensionality reduction for data analysis
Linear dimensionality reduction for data analysis Nicolas Gillis Joint work with Robert Luce, François Glineur, Stephen Vavasis, Robert Plemmons, Gabriella Casalino The setup Dimensionality reduction for
More informationSparseness Constraints on Nonnegative Tensor Decomposition
Sparseness Constraints on Nonnegative Tensor Decomposition Na Li nali@clarksonedu Carmeliza Navasca cnavasca@clarksonedu Department of Mathematics Clarkson University Potsdam, New York 3699, USA Department
More informationQuick Introduction to Nonnegative Matrix Factorization
Quick Introduction to Nonnegative Matrix Factorization Norm Matloff University of California at Davis 1 The Goal Given an u v matrix A with nonnegative elements, we wish to find nonnegative, rank-k matrices
More informationOrthogonal tensor decomposition
Orthogonal tensor decomposition Daniel Hsu Columbia University Largely based on 2012 arxiv report Tensor decompositions for learning latent variable models, with Anandkumar, Ge, Kakade, and Telgarsky.
More informationTensor Decompositions for Machine Learning. G. Roeder 1. UBC Machine Learning Reading Group, June University of British Columbia
Network Feature s Decompositions for Machine Learning 1 1 Department of Computer Science University of British Columbia UBC Machine Learning Group, June 15 2016 1/30 Contact information Network Feature
More informationBOOLEAN TENSOR FACTORIZATIONS. Pauli Miettinen 14 December 2011
BOOLEAN TENSOR FACTORIZATIONS Pauli Miettinen 14 December 2011 BACKGROUND: TENSORS AND TENSOR FACTORIZATIONS X BACKGROUND: TENSORS AND TENSOR FACTORIZATIONS C X A B BACKGROUND: BOOLEAN MATRIX FACTORIZATIONS
More informationENGG5781 Matrix Analysis and Computations Lecture 9: Kronecker Product
ENGG5781 Matrix Analysis and Computations Lecture 9: Kronecker Product Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Kronecker product and
More informationMajorization Minimization (MM) and Block Coordinate Descent (BCD)
Majorization Minimization (MM) and Block Coordinate Descent (BCD) Wing-Kin (Ken) Ma Department of Electronic Engineering, The Chinese University Hong Kong, Hong Kong ELEG5481, Lecture 15 Acknowledgment:
More informationAutomatic Rank Determination in Projective Nonnegative Matrix Factorization
Automatic Rank Determination in Projective Nonnegative Matrix Factorization Zhirong Yang, Zhanxing Zhu, and Erkki Oja Department of Information and Computer Science Aalto University School of Science and
More informationData Mining and Matrices
Data Mining and Matrices 6 Non-Negative Matrix Factorization Rainer Gemulla, Pauli Miettinen May 23, 23 Non-Negative Datasets Some datasets are intrinsically non-negative: Counters (e.g., no. occurrences
More informationc Springer, Reprinted with permission.
Zhijian Yuan and Erkki Oja. A FastICA Algorithm for Non-negative Independent Component Analysis. In Puntonet, Carlos G.; Prieto, Alberto (Eds.), Proceedings of the Fifth International Symposium on Independent
More informationCOMS 4721: Machine Learning for Data Science Lecture 18, 4/4/2017
COMS 4721: Machine Learning for Data Science Lecture 18, 4/4/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University TOPIC MODELING MODELS FOR TEXT DATA
More informationDecompositions of Higher-Order Tensors: Concepts and Computation
L. De Lathauwer Decompositions of Higher-Order Tensors: Concepts and Computation Lieven De Lathauwer KU Leuven Belgium Lieven.DeLathauwer@kuleuven-kulak.be 1 L. De Lathauwer Canonical Polyadic Decomposition
More informationA Modular NMF Matching Algorithm for Radiation Spectra
A Modular NMF Matching Algorithm for Radiation Spectra Melissa L. Koudelka Sensor Exploitation Applications Sandia National Laboratories mlkoude@sandia.gov Daniel J. Dorsey Systems Technologies Sandia
More informationPerformance Evaluation of the Matlab PCT for Parallel Implementations of Nonnegative Tensor Factorization
Performance Evaluation of the Matlab PCT for Parallel Implementations of Nonnegative Tensor Factorization Tabitha Samuel, Master s Candidate Dr. Michael W. Berry, Major Professor Abstract: Increasingly
More informationBig Tensor Data Reduction
Big Tensor Data Reduction Nikos Sidiropoulos Dept. ECE University of Minnesota NSF/ECCS Big Data, 3/21/2013 Nikos Sidiropoulos Dept. ECE University of Minnesota () Big Tensor Data Reduction NSF/ECCS Big
More informationNON-NEGATIVE SPARSE CODING
NON-NEGATIVE SPARSE CODING Patrik O. Hoyer Neural Networks Research Centre Helsinki University of Technology P.O. Box 9800, FIN-02015 HUT, Finland patrik.hoyer@hut.fi To appear in: Neural Networks for
More informationPrincipal Component Analysis
Machine Learning Michaelmas 2017 James Worrell Principal Component Analysis 1 Introduction 1.1 Goals of PCA Principal components analysis (PCA) is a dimensionality reduction technique that can be used
More informationDecomposing a three-way dataset of TV-ratings when this is impossible. Alwin Stegeman
Decomposing a three-way dataset of TV-ratings when this is impossible Alwin Stegeman a.w.stegeman@rug.nl www.alwinstegeman.nl 1 Summarizing Data in Simple Patterns Information Technology collection of
More informationCPSC 340: Machine Learning and Data Mining. More PCA Fall 2016
CPSC 340: Machine Learning and Data Mining More PCA Fall 2016 A2/Midterm: Admin Grades/solutions posted. Midterms can be viewed during office hours. Assignment 4: Due Monday. Extra office hours: Thursdays
More informationSPARSE signal representations have gained popularity in recent
6958 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 57, NO. 10, OCTOBER 2011 Blind Compressed Sensing Sivan Gleichman and Yonina C. Eldar, Senior Member, IEEE Abstract The fundamental principle underlying
More information/16/$ IEEE 1728
Extension of the Semi-Algebraic Framework for Approximate CP Decompositions via Simultaneous Matrix Diagonalization to the Efficient Calculation of Coupled CP Decompositions Kristina Naskovska and Martin
More informationCP DECOMPOSITION AND ITS APPLICATION IN NOISE REDUCTION AND MULTIPLE SOURCES IDENTIFICATION
International Conference on Computer Science and Intelligent Communication (CSIC ) CP DECOMPOSITION AND ITS APPLICATION IN NOISE REDUCTION AND MULTIPLE SOURCES IDENTIFICATION Xuefeng LIU, Yuping FENG,
More informationMatrix Factorization with Applications to Clustering Problems: Formulation, Algorithms and Performance
Date: Mar. 3rd, 2017 Matrix Factorization with Applications to Clustering Problems: Formulation, Algorithms and Performance Presenter: Songtao Lu Department of Electrical and Computer Engineering Iowa
More informationFaloutsos, Tong ICDE, 2009
Large Graph Mining: Patterns, Tools and Case Studies Christos Faloutsos Hanghang Tong CMU Copyright: Faloutsos, Tong (29) 2-1 Outline Part 1: Patterns Part 2: Matrix and Tensor Tools Part 3: Proximity
More informationA BLIND SPARSE APPROACH FOR ESTIMATING CONSTRAINT MATRICES IN PARALIND DATA MODELS
2th European Signal Processing Conference (EUSIPCO 22) Bucharest, Romania, August 27-3, 22 A BLIND SPARSE APPROACH FOR ESTIMATING CONSTRAINT MATRICES IN PARALIND DATA MODELS F. Caland,2, S. Miron 2 LIMOS,
More informationCPSC 340: Machine Learning and Data Mining. Sparse Matrix Factorization Fall 2018
CPSC 340: Machine Learning and Data Mining Sparse Matrix Factorization Fall 2018 Last Time: PCA with Orthogonal/Sequential Basis When k = 1, PCA has a scaling problem. When k > 1, have scaling, rotation,
More informationMatrix Factorization & Latent Semantic Analysis Review. Yize Li, Lanbo Zhang
Matrix Factorization & Latent Semantic Analysis Review Yize Li, Lanbo Zhang Overview SVD in Latent Semantic Indexing Non-negative Matrix Factorization Probabilistic Latent Semantic Indexing Vector Space
More informationThe multiple-vector tensor-vector product
I TD MTVP C KU Leuven August 29, 2013 In collaboration with: N Vanbaelen, K Meerbergen, and R Vandebril Overview I TD MTVP C 1 Introduction Inspiring example Notation 2 Tensor decompositions The CP decomposition
More informationScalable Tensor Factorizations with Incomplete Data
Scalable Tensor Factorizations with Incomplete Data Tamara G. Kolda & Daniel M. Dunlavy Sandia National Labs Evrim Acar Information Technologies Institute, TUBITAK-UEKAE, Turkey Morten Mørup Technical
More informationIntroduction to Machine Learning. PCA and Spectral Clustering. Introduction to Machine Learning, Slides: Eran Halperin
1 Introduction to Machine Learning PCA and Spectral Clustering Introduction to Machine Learning, 2013-14 Slides: Eran Halperin Singular Value Decomposition (SVD) The singular value decomposition (SVD)
More informationMust-read Material : Multimedia Databases and Data Mining. Indexing - Detailed outline. Outline. Faloutsos
Must-read Material 15-826: Multimedia Databases and Data Mining Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. Technical Report SAND2007-6702, Sandia National Laboratories,
More informationTensor Analysis. Topics in Data Mining Fall Bruno Ribeiro
Tensor Analysis Topics in Data Mining Fall 2015 Bruno Ribeiro Tensor Basics But First 2 Mining Matrices 3 Singular Value Decomposition (SVD) } X(i,j) = value of user i for property j i 2 j 5 X(Alice, cholesterol)
More informationNon-negative matrix factorization with fixed row and column sums
Available online at www.sciencedirect.com Linear Algebra and its Applications 9 (8) 5 www.elsevier.com/locate/laa Non-negative matrix factorization with fixed row and column sums Ngoc-Diep Ho, Paul Van
More informationHigh Performance Parallel Tucker Decomposition of Sparse Tensors
High Performance Parallel Tucker Decomposition of Sparse Tensors Oguz Kaya INRIA and LIP, ENS Lyon, France SIAM PP 16, April 14, 2016, Paris, France Joint work with: Bora Uçar, CNRS and LIP, ENS Lyon,
More informationNon-Negative Matrix Factorization
Chapter 3 Non-Negative Matrix Factorization Part : Introduction & computation Motivating NMF Skillicorn chapter 8; Berry et al. (27) DMM, summer 27 2 Reminder A T U Σ V T T Σ, V U 2 Σ 2,2 V 2.8.6.6.3.6.5.3.6.3.6.4.3.6.4.3.3.4.5.3.5.8.3.8.3.3.5
More informationInformation Retrieval
Introduction to Information CS276: Information and Web Search Christopher Manning and Pandu Nayak Lecture 13: Latent Semantic Indexing Ch. 18 Today s topic Latent Semantic Indexing Term-document matrices
More informationLearning From Hidden Traits: Joint Factor Analysis and Latent Clustering
1 Learning From Hidden Traits: Joint Factor Analysis and Latent Clustering Bo Yang, Student Member, IEEE, Xiao Fu, Member, IEEE, Nicholas D. Sidiropoulos, Fellow, IEEE arxiv:165.6711v1 [cs.lg] 1 May 16
More informationOrthogonal Nonnegative Matrix Factorization: Multiplicative Updates on Stiefel Manifolds
Orthogonal Nonnegative Matrix Factorization: Multiplicative Updates on Stiefel Manifolds Jiho Yoo and Seungjin Choi Department of Computer Science Pohang University of Science and Technology San 31 Hyoja-dong,
More informationCSC 576: Variants of Sparse Learning
CSC 576: Variants of Sparse Learning Ji Liu Department of Computer Science, University of Rochester October 27, 205 Introduction Our previous note basically suggests using l norm to enforce sparsity in
More informationMain matrix factorizations
Main matrix factorizations A P L U P permutation matrix, L lower triangular, U upper triangular Key use: Solve square linear system Ax b. A Q R Q unitary, R upper triangular Key use: Solve square or overdetrmined
More informationA Characterization of Sampling Patterns for Union of Low-Rank Subspaces Retrieval Problem
A Characterization of Sampling Patterns for Union of Low-Rank Subspaces Retrieval Problem Morteza Ashraphijuo Columbia University ashraphijuo@ee.columbia.edu Xiaodong Wang Columbia University wangx@ee.columbia.edu
More informationAnomaly Detection in Temporal Graph Data: An Iterative Tensor Decomposition and Masking Approach
Proceedings 1st International Workshop on Advanced Analytics and Learning on Temporal Data AALTD 2015 Anomaly Detection in Temporal Graph Data: An Iterative Tensor Decomposition and Masking Approach Anna
More informationMachine learning for pervasive systems Classification in high-dimensional spaces
Machine learning for pervasive systems Classification in high-dimensional spaces Department of Communications and Networking Aalto University, School of Electrical Engineering stephan.sigg@aalto.fi Version
More informationUsing Matrix Decompositions in Formal Concept Analysis
Using Matrix Decompositions in Formal Concept Analysis Vaclav Snasel 1, Petr Gajdos 1, Hussam M. Dahwa Abdulla 1, Martin Polovincak 1 1 Dept. of Computer Science, Faculty of Electrical Engineering and
More informationHigher-Order Singular Value Decomposition (HOSVD) for structured tensors
Higher-Order Singular Value Decomposition (HOSVD) for structured tensors Definition and applications Rémy Boyer Laboratoire des Signaux et Système (L2S) Université Paris-Sud XI GDR ISIS, January 16, 2012
More informationCS168: The Modern Algorithmic Toolbox Lecture #10: Tensors, and Low-Rank Tensor Recovery
CS168: The Modern Algorithmic Toolbox Lecture #10: Tensors, and Low-Rank Tensor Recovery Tim Roughgarden & Gregory Valiant May 3, 2017 Last lecture discussed singular value decomposition (SVD), and we
More informationMatrix Decomposition in Privacy-Preserving Data Mining JUN ZHANG DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF KENTUCKY
Matrix Decomposition in Privacy-Preserving Data Mining JUN ZHANG DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF KENTUCKY OUTLINE Why We Need Matrix Decomposition SVD (Singular Value Decomposition) NMF (Nonnegative
More informationTensor Decompositions for Signal Processing Applications
Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis A. Cichocki, D. Mandic, A-H. Phan, C. Caiafa, G. Zhou, Q. Zhao, and L. De Lathauwer Summary The widespread
More informationKronecker Decomposition for Image Classification
university of innsbruck institute of computer science intelligent and interactive systems Kronecker Decomposition for Image Classification Sabrina Fontanella 1,2, Antonio Rodríguez-Sánchez 1, Justus Piater
More informationExample: Face Detection
Announcements HW1 returned New attendance policy Face Recognition: Dimensionality Reduction On time: 1 point Five minutes or more late: 0.5 points Absent: 0 points Biometrics CSE 190 Lecture 14 CSE190,
More informationNonnegative Tensor Factorization using a proximal algorithm: application to 3D fluorescence spectroscopy
Nonnegative Tensor Factorization using a proximal algorithm: application to 3D fluorescence spectroscopy Caroline Chaux Joint work with X. Vu, N. Thirion-Moreau and S. Maire (LSIS, Toulon) Aix-Marseille
More informationNon-Negative Tensor Factorization with Applications to Statistics and Computer Vision
Non-Negative Tensor Factorization with Applications to Statistics and Computer Vision Amnon Shashua shashua@cs.huji.ac.il School of Engineering and Computer Science, The Hebrew University of Jerusalem,
More informationJOS M.F. TEN BERGE SIMPLICITY AND TYPICAL RANK RESULTS FOR THREE-WAY ARRAYS
PSYCHOMETRIKA VOL. 76, NO. 1, 3 12 JANUARY 2011 DOI: 10.1007/S11336-010-9193-1 SIMPLICITY AND TYPICAL RANK RESULTS FOR THREE-WAY ARRAYS JOS M.F. TEN BERGE UNIVERSITY OF GRONINGEN Matrices can be diagonalized
More informationLinear Algebra Methods for Data Mining
Linear Algebra Methods for Data Mining Saara Hyvönen, Saara.Hyvonen@cs.helsinki.fi Spring 2007 PCA, NMF Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki Summary: PCA PCA is SVD
More informationNon-Negative Matrix Factorization with Quasi-Newton Optimization
Non-Negative Matrix Factorization with Quasi-Newton Optimization Rafal ZDUNEK, Andrzej CICHOCKI Laboratory for Advanced Brain Signal Processing BSI, RIKEN, Wako-shi, JAPAN Abstract. Non-negative matrix
More informationData Mining and Matrices
Data Mining and Matrices 11 Tensor Applications Rainer Gemulla, Pauli Miettinen July 11, 2013 Outline 1 Some Tensor Decompositions 2 Applications of Tensor Decompositions 3 Wrap-Up 2 / 30 Tucker s many
More informationNesterov-based Alternating Optimization for Nonnegative Tensor Completion: Algorithm and Parallel Implementation
Nesterov-based Alternating Optimization for Nonnegative Tensor Completion: Algorithm and Parallel Implementation Georgios Lourakis and Athanasios P. Liavas School of Electrical and Computer Engineering,
More informationFitting a Tensor Decomposition is a Nonlinear Optimization Problem
Fitting a Tensor Decomposition is a Nonlinear Optimization Problem Evrim Acar, Daniel M. Dunlavy, and Tamara G. Kolda* Sandia National Laboratories Sandia is a multiprogram laboratory operated by Sandia
More informationMachine Learning. Principal Components Analysis. Le Song. CSE6740/CS7641/ISYE6740, Fall 2012
Machine Learning CSE6740/CS7641/ISYE6740, Fall 2012 Principal Components Analysis Le Song Lecture 22, Nov 13, 2012 Based on slides from Eric Xing, CMU Reading: Chap 12.1, CB book 1 2 Factor or Component
More informationOn Identifiability of Nonnegative Matrix Factorization
On Identifiability of Nonnegative Matrix Factorization Xiao Fu, Kejun Huang, and Nicholas D. Sidiropoulos Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, 55455,
More informationKronecker Product Approximation with Multiple Factor Matrices via the Tensor Product Algorithm
Kronecker Product Approximation with Multiple actor Matrices via the Tensor Product Algorithm King Keung Wu, Yeung Yam, Helen Meng and Mehran Mesbahi Department of Mechanical and Automation Engineering,
More informationPROJECTIVE NON-NEGATIVE MATRIX FACTORIZATION WITH APPLICATIONS TO FACIAL IMAGE PROCESSING
st Reading October 0, 200 8:4 WSPC/-IJPRAI SPI-J068 008 International Journal of Pattern Recognition and Artificial Intelligence Vol. 2, No. 8 (200) 0 c World Scientific Publishing Company PROJECTIVE NON-NEGATIVE
More informationPostgraduate Course Signal Processing for Big Data (MSc)
Postgraduate Course Signal Processing for Big Data (MSc) Jesús Gustavo Cuevas del Río E-mail: gustavo.cuevas@upm.es Work Phone: +34 91 549 57 00 Ext: 4039 Course Description Instructor Information Course
More informationNon-convex Robust PCA: Provable Bounds
Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and Sujay Sanghavi. Learning with Big Data High Dimensional Regime Missing
More informationFast Coordinate Descent methods for Non-Negative Matrix Factorization
Fast Coordinate Descent methods for Non-Negative Matrix Factorization Inderjit S. Dhillon University of Texas at Austin SIAM Conference on Applied Linear Algebra Valencia, Spain June 19, 2012 Joint work
More information1 Feature Vectors and Time Series
PCA, SVD, LSI, and Kernel PCA 1 Feature Vectors and Time Series We now consider a sample x 1,..., x of objects (not necessarily vectors) and a feature map Φ such that for any object x we have that Φ(x)
More informationTo be published in Optics Letters: Blind Multi-spectral Image Decomposition by 3D Nonnegative Tensor Title: Factorization Authors: Ivica Kopriva and A
o be published in Optics Letters: Blind Multi-spectral Image Decomposition by 3D Nonnegative ensor itle: Factorization Authors: Ivica Kopriva and Andrzej Cichocki Accepted: 21 June 2009 Posted: 25 June
More informationDictionary Learning Using Tensor Methods
Dictionary Learning Using Tensor Methods Anima Anandkumar U.C. Irvine Joint work with Rong Ge, Majid Janzamin and Furong Huang. Feature learning as cornerstone of ML ML Practice Feature learning as cornerstone
More informationA concise proof of Kruskal s theorem on tensor decomposition
A concise proof of Kruskal s theorem on tensor decomposition John A. Rhodes 1 Department of Mathematics and Statistics University of Alaska Fairbanks PO Box 756660 Fairbanks, AK 99775 Abstract A theorem
More informationINTERNATIONAL JOURNAL OF ENGINEERING SCIENCES & RESEARCH TECHNOLOGY
[Gaurav, 2(1): Jan., 2013] ISSN: 2277-9655 IJESRT INTERNATIONAL JOURNAL OF ENGINEERING SCIENCES & RESEARCH TECHNOLOGY Face Identification & Detection Using Eigenfaces Sachin.S.Gurav *1, K.R.Desai 2 *1
More informationMultiscale Tensor Decomposition
Multiscale Tensor Decomposition Alp Ozdemir 1, Mark A. Iwen 1,2 and Selin Aviyente 1 1 Department of Electrical and Computer Engineering, Michigan State University 2 Deparment of the Mathematics, Michigan
More informationLinear Classification: Perceptron
Linear Classification: Perceptron Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong 1 / 18 Y Tao Linear Classification: Perceptron In this lecture, we will consider
More informationSimplicial Nonnegative Matrix Tri-Factorization: Fast Guaranteed Parallel Algorithm
Simplicial Nonnegative Matrix Tri-Factorization: Fast Guaranteed Parallel Algorithm Duy-Khuong Nguyen 13, Quoc Tran-Dinh 2, and Tu-Bao Ho 14 1 Japan Advanced Institute of Science and Technology, Japan
More informationDonald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016
Optimization for Tensor Models Donald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016 1 Tensors Matrix Tensor: higher-order matrix
More informationOptimal solutions to non-negative PARAFAC/multilinear NMF always exist
Optimal solutions to non-negative PARAFAC/multilinear NMF always exist Lek-Heng Lim Stanford University Workshop on Tensor Decompositions and Applications CIRM, Luminy, France August 29 September 2, 2005
More informationNonnegative Matrix Factorization
Nonnegative Matrix Factorization Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationUnsupervised Machine Learning and Data Mining. DS 5230 / DS Fall Lecture 7. Jan-Willem van de Meent
Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 7 Jan-Willem van de Meent DIMENSIONALITY REDUCTION Borrowing from: Percy Liang (Stanford) Dimensionality Reduction Goal:
More informationRobust Asymmetric Nonnegative Matrix Factorization
Robust Asymmetric Nonnegative Matrix Factorization Hyenkyun Woo Korea Institue for Advanced Study Math @UCLA Feb. 11. 2015 Joint work with Haesun Park Georgia Institute of Technology, H. Woo (KIAS) RANMF
More informationONP-MF: An Orthogonal Nonnegative Matrix Factorization Algorithm with Application to Clustering
ONP-MF: An Orthogonal Nonnegative Matrix Factorization Algorithm with Application to Clustering Filippo Pompili 1, Nicolas Gillis 2, P.-A. Absil 2,andFrançois Glineur 2,3 1- University of Perugia, Department
More informationDimitri Nion & Lieven De Lathauwer
he decomposition of a third-order tensor in block-terms of rank-l,l, Model, lgorithms, Uniqueness, Estimation of and L Dimitri Nion & Lieven De Lathauwer.U. Leuven, ortrijk campus, Belgium E-mails: Dimitri.Nion@kuleuven-kortrijk.be
More informationThe Canonical Tensor Decomposition and Its Applications to Social Network Analysis
The Canonical Tensor Decomposition and Its Applications to Social Network Analysis Evrim Acar, Tamara G. Kolda and Daniel M. Dunlavy Sandia National Labs Sandia is a multiprogram laboratory operated by
More informationMatrix Algebra. Learning Objectives. Size of Matrix
Matrix Algebra 1 Learning Objectives 1. Find the sum and difference of two matrices 2. Find scalar multiples of a matrix 3. Find the product of two matrices 4. Find the inverse of a matrix 5. Solve a system
More informationCS 231A Section 1: Linear Algebra & Probability Review
CS 231A Section 1: Linear Algebra & Probability Review 1 Topics Support Vector Machines Boosting Viola-Jones face detector Linear Algebra Review Notation Operations & Properties Matrix Calculus Probability
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More information