Lecture 8. Principal Component Analysis. Luigi Freda. ALCOR Lab DIAG University of Rome La Sapienza. December 13, 2016
|
|
- Leslie McDowell
- 5 years ago
- Views:
Transcription
1 Lecture 8 Principal Component Analysis Luigi Freda ALCOR Lab DIAG University of Rome La Sapienza December 13, 2016 Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
2 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
3 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
4 Eigenvalues given a matrix A C n n a nonzero vector v C n is said to be its (right) eigenvector if Av = λv for some scalar λ C; λ is called an eigenvalue of A the set of all eigenvalues of A is called its spectrum and it s denoted by σ(a) the Matlab command [ V,D]= e i g (A) produces a diagonal matrix D of eigenvalues and a matrix V such that AV = VD with D = diag(λ i ) if A is diagonalizable then V is full-rank (i.e. rank(v ) = n) and the columns of V correspond to n linearly independent eigenvectors, i.e. V = {v 1,..., v n}; in this case one has n A = VDV 1 = λ i v i v T i if à = TAT 1, where T is a nonsingular matrix, then à and A are called similar matrices; at the previous point, A and D are similar matrices Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31 i=1
5 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
6 Eigenvalues Real Matrix given a real matrix A R n n all its eigenvalues σ(a) are the roots of the characteristic polynomial equation det(λi A) = 0 given that A is a real matrix, if λ C is an eigenvalue then its conjugate λ C is also an eigenvalue, i.e., σ(a) = σ(a) det(λ I A) = 0 = det(λ I A) = det(λ I A) = det(λ I A) = 0 σ(a) = σ(a T ) since det(λi A) = det((λi A) T ) = det(λi A T ) if à = TAT 1, where T is a nonsingular matrix, then σ(ã) = σ(a) since det(λi Ã) = det(λtt 1 TAT 1 ) = det(t(λi A)T 1 ) = = det(t) det(λi A) det(t 1 ( ) = det(λi A) det I = det(t 1 ) det T = 1 ) Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
7 Eigenvalues Real Matrix given a real matrix A R n n eigenvectors of A associated to different eigenvalues are linearly independent Av i = λ i v i, λ 1 λ 2, v 1 v 2 α 1v 1 + α 2v 2 = 0 = A(α 1v 1 + α 2v 2) = 0 = α 1λ 1v 1 + α 2λ 2v 2 = 0 since λ 1(α 1v 1 + α 2v 2) = 0 = α 2(λ 2 λ 1)v 2 = 0 = α 2 = 0 = α 1 = 0 if σ(a) = n, i.e. σ(a) = {λ 1, λ 2,..., λ n} with λ i λ j for i j, then A is diagonalizable Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
8 Eigenvalues Real Matrix given a real matrix A R n n in general σ(a) = {λ 1,..., λ m} with m n and λ i λ j for i j, and one has det(λi A) = (λ λ 1) µa(λ 1) (λ λ 2) µa(λ 2)...(λ λ m) µa(λm) m i=1 µ a(λ i ) = n where µ a(λ i ) is the algebraic multiplicity of λ i in the general case one can always find the Jordan canonical form J λ i 1 1 A = VJV 1 J 2. J =. J.. i = λ.. i... 1 J p λ i Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
9 Eigenvalues Real Matrix given a real matrix A R n n let E(λ i ) ker(a λ i I) = {v R n : (A λ i I)v = 0} be the eigenspace of λ i let µ g (λ i ) dim(e(λ i )) be the geometric multiplicity of λ i (µ g (λ i ) µ a(λ i )) if E(λ i ) = span{v i1,..., v iµi }, with µ i µ g (λ i ), then AV i = V i D i with V i = [v i1,..., v iµi ] R n µ i and D i = λ i I µi if µ g (λ i ) = µ a(λ i ) for each i {1,..., m} then and A is diagonalizable R n = E(λ 1)... E(λ m) n µ g (λ i ) = n i=1 Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
10 Eigenvalues Real Symmetric Matrix given a real symmetric matrix S R nxn (i.e. S = S T ) all eigenvalues of S are real, i.e. σ(s) R eigenvectors v 1, v 2 of S associated to different eigenvalues are linearly independent and orthogonal Sv i = λ i v i, λ 1 λ 2, v 1 v 2, v T 2 (Sv 1 λ 1v 1) = 0 = (Sv 2) T v 1 λ 1v T 2 v 1 = 0 = v T 2 v 1(λ 2 λ 1) = 0 = v T 2 v 1 = 0 S is diagonalizable and one can find an orthonormal basis V such that V T = V 1 S = VDV T = n λ i v i v T i i=1 if S > 0 (S 0) then λ i > 0 (λ i 0) for i {1,..., m} N.B.: the above results can be applied for instance to covariance matrices Σ which are symmetric and positive definite Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
11 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
12 Singular Value Decomposition idea: the singular values generalize the notion of eigenvalues to any kind of matrix given a matrix X R N D this can be decomposed as follows X N D = U S = V T N N N D D D r σ i u i v T i i=1 U R N N is orthonormal, U T U = UU T = I N V R D D is orthonormal, V T V = VV T = I D σ 1 σ 2... σ r 0 are the singular values, r = min(n, D) and S R N D σ 1... σ 1 S = if N > D or S =... if N D 0D N 0 N D σ D σ N Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
13 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
14 Singular Value Decomposition Connection with Eigenvalues given a matrix X R N D one has X N D V T = U S N N N D D D U R N N is orthonormal, U T U = UU T = I N V R D D is orthonormal, V T V = VV T = I D = r i=1 σ iu i v T i where r = min(n, D) S R N D contains the singular values σ i on the main diagonal and 0s elsewhere we have X T X = (VS T U T )(USV T ) = VS T SV T = VDV T where D S T S is a diagonal matrix containing the squared singular values σ 2 i (and possibly some additional zeros) on the main diagonal and 0s elsewhere since (X T X)V = VD, then V contains the eigenvectors of X T X the eigenvalues of X T X are the squared singular values σ 2 i, contained in D the columns of V are called the right singular vectors of X Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
15 Singular Value Decomposition Connection with Eigenvalues given a matrix X R N D one has X N D V T = U S N N N D D D U R N N is orthonormal, U T U = UU T = I N V R D D is orthonormal, V T V = VV T = I D = r i=1 σ iu i v T i where r = min(n, D) S R N D contains the singular values σ i on the main diagonal and 0s elsewhere we have XX T = (USV T )(VS T U T ) = USS T U T = U ˆDU T where ˆD SS T is a diagonal matrix containing the squared singular values σ 2 i (and possibly some additional zeros) on the main diagonal and 0s elsewhere since (XX T )U = U ˆD, then U contains the eigenvectors of XX T the eigenvalues of XX T are the squared singular values σ 2 i, contained in ˆD the columns of U are called the left singular vectors of X Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
16 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
17 Singular Value Decomposition given a matrix X R N D with N > D this can be decomposed as follows X N D = U S = V T N N N D D D σ 1... S = 0 N D D σ i u i v T i the last N D columns of U are irrelevant, since they will be multiplied by zero i=1 σ D Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
18 Singular Value Decomposition given a matrix X R N D with N > D this can be decomposed as X N D = U S V T N N N D D D = D i=1 σ iu i v T i since the last N D columns of U are irrelevant, we can directly neglect them and consider the economy sized SVD X N D = Û Ŝ = V T N D D D D D D σ i u i v T i where Û T Û = I D, V T V = VV T = I D and Ŝ = diag(σ 1,..., σ D ) i=1 Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
19 Singular Value Decomposition given a matrix X R N D with N > D this can be decomposed as X = U S V T = D i=1 σ iu i v T i N D N N N D D D a rank L approximation of X considers only the first L < D columns of U and the first L singular values we can then consider the truncated SVD X L = U L N D N L S L L L V T L D L = L σ i u i v T i where U T L U L = I L, V T L V L = I D and S L = diag(σ 1,..., σ L ) i=1 Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
20 Singular Value Decomposition given a matrix X R N D with N > D it is possible to show that X L = U L S L V T L which solve is the best rank L approximation matrix X L = argmin X ˆX F ˆX with the constraint rank(ˆx) = L where A F denotes the Frobenious norm ( m n A F = i=1 j=1 a 2 ij ) 1/2 = ( trace(a T A) ) 1/2 by replacing ˆX = X L one obtains [ SL X X L F = U(S 0 ] [ )V T SL F = S 0 ] F = r i=l+1 σ i Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
21 Outline 1 Eigen Decomposition Eigenvalues and Eigenvectors Some Properties 2 Singular Value Decomposition Singular Values Connection with Eigenvalues Singular Value Decomposition 3 Principal Component Analysis Discovering Latent Factors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
22 Discovering Latent Factors An example dimensionality reduction: it is often useful to reduce the dimensionality by projecting the data to a lower dimensional subspace which captures the essence of the data for example in the above plot: the 2d approximation is quite good, most points lie close to this subspace projecting points onto the red line 1d approx is a rather poor approximation Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
23 Discovering Latent Factors Motivations although data may appear high dimensional, there may only be a small number of degrees of variability, corresponding to latent factors low dimensional representations, when used as input to statistical models, often result in better predictive accuracy (focusing on the essence ) low dimensional representations are useful for enabling fast nearest neighbor searches two dimensional projections are very useful for visualizing high dimensional data Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
24 Principal Component Analysis PCA the most common approach to dimensionality reduction is called Principal Components Analysis or PCA given x i R D, we compute an approximation ˆx i = Wz i with z i R L where L < D (dimensionality reduction) W R D L and W T W = I (orthogonal W) so as to minimize the reconstruction error J(W, Z) = 1 N x i ˆx i 2 = 1 N x i Wz i 2 N N i=1 i=1 since W T W = I one has x 2 = (x T x) 1/2 = (z T W T Wz) 1/2 = (z T z) 1/2 = z 2 in 3D or 2D W can represent part of a rotation matrix this can be thought of as an unsupervised version of (multi-output) linear regression, where we observe the high-dimensional response y = x, but not the low-dimensional cause z Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
25 Principal Component Analysis PCA the objective function (reconstruction error) J(W, Z) = 1 N x i ˆx i 2 = 1 N x i Wz i 2 N N is equivalent to J(W, Z) = X T WZ T 2 F where A F denotes the Frobenious norm i=1 ( m n A F = i=1 j=1 a 2 ij i=1 ) 1/2 = ( trace(a T A) ) 1/2 Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
26 Principal Component Analysis PCA we have to minimize the objective function J(W, Z) = 1 N x i ˆx i 2 = 1 N x i Wz i 2 N N i=1 or equivalently J(W, Z) = X T WZ T 2 F subject to W R D L and W T W = I it is possible to prove that the optimal solution is obtained by setting where X L = U L S L V T L Ŵ = V L i=1 Ẑ = XW (z i = ŴT x i = V T L x i ) is the rank L truncated SVD of X we have X T X = VS 2 V T hence (X T X)V L = V L S 2 L since V = [V L ] V L contains the L eigenvectors (principal directions) with the largest eigenvalues of the empirical covariance matrix Σ = 1 N XT X = 1 N N i=1 x ix T i z i = V T L x i is the orthogonal projection of x i on the eigenvectors in V L Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
27 Principal Component Analysis PCA by replacing Z = XW in the objective function and recalling that W T W = I ( ) J(Z) = X T WZ T 2 F = trace (X T WZ T ) T (X T WZ T ) = = trace(xx T XWZ T ZW T X T + ZZ T ) = trace(x T X) trace(w T X T XW) = = trace(x T X) trace(z T Z) = trace(nσ X ) trace(nσ Z ) where we used the following facts: trace(ab) = trace(ba), trace(a T ) = trace(a) minimizing the reconstruction error is equivalent to maximizing the variance of the projected data z i = Wx i Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
28 Principal Component Analysis PCA the principal directions are the ones along which the data shows maximal variance this means that PCA can be misled by directions in which the variance is high merely because of the measurement scale in the figure below, the vertical axis (weight) uses a large range than the horizontal axis (height), resulting in a line that looks somewhat unnatural it is therefore standard practice to standardize the data first where µ j = 1 N x ij (x ij µ j ) N i=1 x ij and σj 2 = 1 N N 1 i=1 (x ij µ j ) 2 σ j Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
29 Principal Component Analysis PCA left: the mean and the first three PC basis vectors (eigendigits) based on 25 images of the digit 3 (from the MNIST dataset) right: reconstruction of an image based on 2, 10, 100 and all the basis vectors Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
30 Principal Component Analysis PCA low rank approximations to an image top left: the original image is of size , so has rank 200 subsequent images have ranks 2, 5, and 20. Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
31 Credits Kevin Murphy s book Luigi Freda ( La Sapienza University) Lecture 8 December 13, / 31
14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationPrincipal Component Analysis
Principal Component Analysis Yuanzhen Shao MA 26500 Yuanzhen Shao PCA 1 / 13 Data as points in R n Assume that we have a collection of data in R n. x 11 x 21 x 12 S = {X 1 =., X x 22 2 =.,, X x m2 m =.
More informationEigenvalues, Eigenvectors. Eigenvalues and eigenvector will be fundamentally related to the nature of the solutions of state space systems.
Chapter 3 Linear Algebra In this Chapter we provide a review of some basic concepts from Linear Algebra which will be required in order to compute solutions of LTI systems in state space form, discuss
More informationMachine Learning. B. Unsupervised Learning B.2 Dimensionality Reduction. Lars Schmidt-Thieme, Nicolas Schilling
Machine Learning B. Unsupervised Learning B.2 Dimensionality Reduction Lars Schmidt-Thieme, Nicolas Schilling Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University
More informationMATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors.
MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v
More informationMatrix Vector Products
We covered these notes in the tutorial sessions I strongly recommend that you further read the presented materials in classical books on linear algebra Please make sure that you understand the proofs and
More informationLEC 2: Principal Component Analysis (PCA) A First Dimensionality Reduction Approach
LEC 2: Principal Component Analysis (PCA) A First Dimensionality Reduction Approach Dr. Guangliang Chen February 9, 2016 Outline Introduction Review of linear algebra Matrix SVD PCA Motivation The digits
More information15 Singular Value Decomposition
15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationDIAGONALIZATION. In order to see the implications of this definition, let us consider the following example Example 1. Consider the matrix
DIAGONALIZATION Definition We say that a matrix A of size n n is diagonalizable if there is a basis of R n consisting of eigenvectors of A ie if there are n linearly independent vectors v v n such that
More informationChapter 7. Canonical Forms. 7.1 Eigenvalues and Eigenvectors
Chapter 7 Canonical Forms 7.1 Eigenvalues and Eigenvectors Definition 7.1.1. Let V be a vector space over the field F and let T be a linear operator on V. An eigenvalue of T is a scalar λ F such that there
More informationDesigning Information Devices and Systems II
EECS 16B Fall 2016 Designing Information Devices and Systems II Linear Algebra Notes Introduction In this set of notes, we will derive the linear least squares equation, study the properties symmetric
More information1 Principal Components Analysis
Lecture 3 and 4 Sept. 18 and Sept.20-2006 Data Visualization STAT 442 / 890, CM 462 Lecture: Ali Ghodsi 1 Principal Components Analysis Principal components analysis (PCA) is a very popular technique for
More informationPCA and admixture models
PCA and admixture models CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar, Alkes Price PCA and admixture models 1 / 57 Announcements HW1
More informationTutorial on Principal Component Analysis
Tutorial on Principal Component Analysis Copyright c 1997, 2003 Javier R. Movellan. This is an open source document. Permission is granted to copy, distribute and/or modify this document under the terms
More informationChapter 6: Orthogonality
Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products
More informationMachine Learning. Principal Components Analysis. Le Song. CSE6740/CS7641/ISYE6740, Fall 2012
Machine Learning CSE6740/CS7641/ISYE6740, Fall 2012 Principal Components Analysis Le Song Lecture 22, Nov 13, 2012 Based on slides from Eric Xing, CMU Reading: Chap 12.1, CB book 1 2 Factor or Component
More information235 Final exam review questions
5 Final exam review questions Paul Hacking December 4, 0 () Let A be an n n matrix and T : R n R n, T (x) = Ax the linear transformation with matrix A. What does it mean to say that a vector v R n is an
More informationLecture: Face Recognition and Feature Reduction
Lecture: Face Recognition and Feature Reduction Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 1 Recap - Curse of dimensionality Assume 5000 points uniformly distributed in the
More informationUnsupervised Machine Learning and Data Mining. DS 5230 / DS Fall Lecture 7. Jan-Willem van de Meent
Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 7 Jan-Willem van de Meent DIMENSIONALITY REDUCTION Borrowing from: Percy Liang (Stanford) Dimensionality Reduction Goal:
More informationLecture: Face Recognition and Feature Reduction
Lecture: Face Recognition and Feature Reduction Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab Lecture 11-1 Recap - Curse of dimensionality Assume 5000 points uniformly distributed
More information7 Principal Component Analysis
7 Principal Component Analysis This topic will build a series of techniques to deal with high-dimensional data. Unlike regression problems, our goal is not to predict a value (the y-coordinate), it is
More informationContents. Preface for the Instructor. Preface for the Student. xvii. Acknowledgments. 1 Vector Spaces 1 1.A R n and C n 2
Contents Preface for the Instructor xi Preface for the Student xv Acknowledgments xvii 1 Vector Spaces 1 1.A R n and C n 2 Complex Numbers 2 Lists 5 F n 6 Digression on Fields 10 Exercises 1.A 11 1.B Definition
More informationProbabilistic Latent Semantic Analysis
Probabilistic Latent Semantic Analysis Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationLecture 5 Supspace Tranformations Eigendecompositions, kernel PCA and CCA
Lecture 5 Supspace Tranformations Eigendecompositions, kernel PCA and CCA Pavel Laskov 1 Blaine Nelson 1 1 Cognitive Systems Group Wilhelm Schickard Institute for Computer Science Universität Tübingen,
More informationMA 265 FINAL EXAM Fall 2012
MA 265 FINAL EXAM Fall 22 NAME: INSTRUCTOR S NAME:. There are a total of 25 problems. You should show work on the exam sheet, and pencil in the correct answer on the scantron. 2. No books, notes, or calculators
More informationPrincipal Component Analysis and Singular Value Decomposition. Volker Tresp, Clemens Otte Summer 2014
Principal Component Analysis and Singular Value Decomposition Volker Tresp, Clemens Otte Summer 2014 1 Motivation So far we always argued for a high-dimensional feature space Still, in some cases it makes
More informationLinear Algebra 1. M.T.Nair Department of Mathematics, IIT Madras. and in that case x is called an eigenvector of T corresponding to the eigenvalue λ.
Linear Algebra 1 M.T.Nair Department of Mathematics, IIT Madras 1 Eigenvalues and Eigenvectors 1.1 Definition and Examples Definition 1.1. Let V be a vector space (over a field F) and T : V V be a linear
More informationPrincipal Component Analysis
Principal Component Analysis Yingyu Liang yliang@cs.wisc.edu Computer Sciences Department University of Wisconsin, Madison [based on slides from Nina Balcan] slide 1 Goals for the lecture you should understand
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationDimensionality Reduction
Dimensionality Reduction Le Song Machine Learning I CSE 674, Fall 23 Unsupervised learning Learning from raw (unlabeled, unannotated, etc) data, as opposed to supervised data where a classification of
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationDefinition (T -invariant subspace) Example. Example
Eigenvalues, Eigenvectors, Similarity, and Diagonalization We now turn our attention to linear transformations of the form T : V V. To better understand the effect of T on the vector space V, we begin
More informationEigenvalues and diagonalization
Eigenvalues and diagonalization Patrick Breheny November 15 Patrick Breheny BST 764: Applied Statistical Modeling 1/20 Introduction The next topic in our course, principal components analysis, revolves
More informationChapter 7: Symmetric Matrices and Quadratic Forms
Chapter 7: Symmetric Matrices and Quadratic Forms (Last Updated: December, 06) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationApplied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.2: Fundamentals 2 / 31 Eigenvalues and Eigenvectors Eigenvalues and eigenvectors of
More informationSingular Value Decomposition
Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =
More informationPrincipal Component Analysis
Machine Learning Michaelmas 2017 James Worrell Principal Component Analysis 1 Introduction 1.1 Goals of PCA Principal components analysis (PCA) is a dimensionality reduction technique that can be used
More informationPRINCIPAL COMPONENTS ANALYSIS
121 CHAPTER 11 PRINCIPAL COMPONENTS ANALYSIS We now have the tools necessary to discuss one of the most important concepts in mathematical statistics: Principal Components Analysis (PCA). PCA involves
More informationE2 212: Matrix Theory (Fall 2010) Solutions to Test - 1
E2 212: Matrix Theory (Fall 2010) s to Test - 1 1. Let X = [x 1, x 2,..., x n ] R m n be a tall matrix. Let S R(X), and let P be an orthogonal projector onto S. (a) If X is full rank, show that P can be
More information1 Singular Value Decomposition and Principal Component
Singular Value Decomposition and Principal Component Analysis In these lectures we discuss the SVD and the PCA, two of the most widely used tools in machine learning. Principal Component Analysis (PCA)
More informationPrincipal Component Analysis
Principal Component Analysis CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Principal
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationLinear System Theory
Linear System Theory Wonhee Kim Lecture 4 Apr. 4, 2018 1 / 40 Recap Vector space, linear space, linear vector space Subspace Linearly independence and dependence Dimension, Basis, Change of Basis 2 / 40
More informationPCA, Kernel PCA, ICA
PCA, Kernel PCA, ICA Learning Representations. Dimensionality Reduction. Maria-Florina Balcan 04/08/2015 Big & High-Dimensional Data High-Dimensions = Lot of Features Document classification Features per
More informationBare minimum on matrix algebra. Psychology 588: Covariance structure and factor models
Bare minimum on matrix algebra Psychology 588: Covariance structure and factor models Matrix multiplication 2 Consider three notations for linear combinations y11 y1 m x11 x 1p b11 b 1m y y x x b b n1
More informationNumerical Methods I Singular Value Decomposition
Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)
More informationComputational Methods CMSC/AMSC/MAPL 460. Eigenvalues and Eigenvectors. Ramani Duraiswami, Dept. of Computer Science
Computational Methods CMSC/AMSC/MAPL 460 Eigenvalues and Eigenvectors Ramani Duraiswami, Dept. of Computer Science Eigen Values of a Matrix Recap: A N N matrix A has an eigenvector x (non-zero) with corresponding
More informationLecture 5. Gaussian Models - Part 1. Luigi Freda. ALCOR Lab DIAG University of Rome La Sapienza. November 29, 2016
Lecture 5 Gaussian Models - Part 1 Luigi Freda ALCOR Lab DIAG University of Rome La Sapienza November 29, 2016 Luigi Freda ( La Sapienza University) Lecture 5 November 29, 2016 1 / 42 Outline 1 Basics
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More informationLec 2: Mathematical Economics
Lec 2: Mathematical Economics to Spectral Theory Sugata Bag Delhi School of Economics 24th August 2012 [SB] (Delhi School of Economics) Introductory Math Econ 24th August 2012 1 / 17 Definition: Eigen
More informationALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA
ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA Kent State University Department of Mathematical Sciences Compiled and Maintained by Donald L. White Version: August 29, 2017 CONTENTS LINEAR ALGEBRA AND
More informationLecture 10 - Eigenvalues problem
Lecture 10 - Eigenvalues problem Department of Computer Science University of Houston February 28, 2008 1 Lecture 10 - Eigenvalues problem Introduction Eigenvalue problems form an important class of problems
More informationLECTURE 16: PCA AND SVD
Instructor: Sael Lee CS549 Computational Biology LECTURE 16: PCA AND SVD Resource: PCA Slide by Iyad Batal Chapter 12 of PRML Shlens, J. (2003). A tutorial on principal component analysis. CONTENT Principal
More information33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM
33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the
More informationNo books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question.
Math 304 Final Exam (May 8) Spring 206 No books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question. Name: Section: Question Points
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationRemark 1 By definition, an eigenvector must be a nonzero vector, but eigenvalue could be zero.
Sec 5 Eigenvectors and Eigenvalues In this chapter, vector means column vector Definition An eigenvector of an n n matrix A is a nonzero vector x such that A x λ x for some scalar λ A scalar λ is called
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationMLCC 2015 Dimensionality Reduction and PCA
MLCC 2015 Dimensionality Reduction and PCA Lorenzo Rosasco UNIGE-MIT-IIT June 25, 2015 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationDiagonalization. Hung-yi Lee
Diagonalization Hung-yi Lee Review If Av = λv (v is a vector, λ is a scalar) v is an eigenvector of A excluding zero vector λ is an eigenvalue of A that corresponds to v Eigenvectors corresponding to λ
More informationRemark By definition, an eigenvector must be a nonzero vector, but eigenvalue could be zero.
Sec 6 Eigenvalues and Eigenvectors Definition An eigenvector of an n n matrix A is a nonzero vector x such that A x λ x for some scalar λ A scalar λ is called an eigenvalue of A if there is a nontrivial
More informationGI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis. Massimiliano Pontil
GI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis Massimiliano Pontil 1 Today s plan SVD and principal component analysis (PCA) Connection
More informationLecture Notes 2: Matrices
Optimization-based data analysis Fall 2017 Lecture Notes 2: Matrices Matrices are rectangular arrays of numbers, which are extremely useful for data analysis. They can be interpreted as vectors in a vector
More informationSingular Value Decomposition
Chapter 6 Singular Value Decomposition In Chapter 5, we derived a number of algorithms for computing the eigenvalues and eigenvectors of matrices A R n n. Having developed this machinery, we complete our
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationMATH 583A REVIEW SESSION #1
MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationLecture 13. Principal Component Analysis. Brett Bernstein. April 25, CDS at NYU. Brett Bernstein (CDS at NYU) Lecture 13 April 25, / 26
Principal Component Analysis Brett Bernstein CDS at NYU April 25, 2017 Brett Bernstein (CDS at NYU) Lecture 13 April 25, 2017 1 / 26 Initial Question Intro Question Question Let S R n n be symmetric. 1
More informationFall TMA4145 Linear Methods. Exercise set Given the matrix 1 2
Norwegian University of Science and Technology Department of Mathematical Sciences TMA445 Linear Methods Fall 07 Exercise set Please justify your answers! The most important part is how you arrive at an
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis (Numerical Linear Algebra for Computational and Data Sciences) Lecture 14: Eigenvalue Problems; Eigenvalue Revealing Factorizations Xiangmin Jiao Stony Brook University Xiangmin
More informationControl Systems. Linear Algebra topics. L. Lanari
Control Systems Linear Algebra topics L Lanari outline basic facts about matrices eigenvalues - eigenvectors - characteristic polynomial - algebraic multiplicity eigenvalues invariance under similarity
More informationLinear Algebra - Part II
Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T
More informationLinear Algebra- Final Exam Review
Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.
More informationELE/MCE 503 Linear Algebra Facts Fall 2018
ELE/MCE 503 Linear Algebra Facts Fall 2018 Fact N.1 A set of vectors is linearly independent if and only if none of the vectors in the set can be written as a linear combination of the others. Fact N.2
More informationLinear Subspace Models
Linear Subspace Models Goal: Explore linear models of a data set. Motivation: A central question in vision concerns how we represent a collection of data vectors. The data vectors may be rasterized images,
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationThroughout these notes we assume V, W are finite dimensional inner product spaces over C.
Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal
More informationUniversity of Colorado at Denver Mathematics Department Applied Linear Algebra Preliminary Exam With Solutions 16 January 2009, 10:00 am 2:00 pm
University of Colorado at Denver Mathematics Department Applied Linear Algebra Preliminary Exam With Solutions 16 January 2009, 10:00 am 2:00 pm Name: The proctor will let you read the following conditions
More information. = V c = V [x]v (5.1) c 1. c k
Chapter 5 Linear Algebra It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix Because they form the foundation on which we later work,
More informationBackground Mathematics (2/2) 1. David Barber
Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and
More informationCS 340 Lec. 6: Linear Dimensionality Reduction
CS 340 Lec. 6: Linear Dimensionality Reduction AD January 2011 AD () January 2011 1 / 46 Linear Dimensionality Reduction Introduction & Motivation Brief Review of Linear Algebra Principal Component Analysis
More informationDATA MINING LECTURE 8. Dimensionality Reduction PCA -- SVD
DATA MINING LECTURE 8 Dimensionality Reduction PCA -- SVD The curse of dimensionality Real data usually have thousands, or millions of dimensions E.g., web documents, where the dimensionality is the vocabulary
More informationPseudoinverse & Orthogonal Projection Operators
Pseudoinverse & Orthogonal Projection Operators ECE 174 Linear & Nonlinear Optimization Ken Kreutz-Delgado ECE Department, UC San Diego Ken Kreutz-Delgado (UC San Diego) ECE 174 Fall 2016 1 / 48 The Four
More informationIntroduction to Machine Learning. PCA and Spectral Clustering. Introduction to Machine Learning, Slides: Eran Halperin
1 Introduction to Machine Learning PCA and Spectral Clustering Introduction to Machine Learning, 2013-14 Slides: Eran Halperin Singular Value Decomposition (SVD) The singular value decomposition (SVD)
More informationAssignment #10: Diagonalization of Symmetric Matrices, Quadratic Forms, Optimization, Singular Value Decomposition. Name:
Assignment #10: Diagonalization of Symmetric Matrices, Quadratic Forms, Optimization, Singular Value Decomposition Due date: Friday, May 4, 2018 (1:35pm) Name: Section Number Assignment #10: Diagonalization
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationLeast Squares Optimization
Least Squares Optimization The following is a brief review of least squares optimization and constrained optimization techniques. Broadly, these techniques can be used in data analysis and visualization
More informationUniversity of Colorado Denver Department of Mathematical and Statistical Sciences Applied Linear Algebra Ph.D. Preliminary Exam June 8, 2012
University of Colorado Denver Department of Mathematical and Statistical Sciences Applied Linear Algebra Ph.D. Preliminary Exam June 8, 2012 Name: Exam Rules: This is a closed book exam. Once the exam
More informationIV. Matrix Approximation using Least-Squares
IV. Matrix Approximation using Least-Squares The SVD and Matrix Approximation We begin with the following fundamental question. Let A be an M N matrix with rank R. What is the closest matrix to A that
More informationKnowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationDS-GA 1002 Lecture notes 10 November 23, Linear models
DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More information