STA141C: Big Data & High Performance Statistical Computing
|
|
- Coral McCoy
- 5 years ago
- Views:
Transcription
1 STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018
2 Linear Algebra Background
3 Vectors A vector has a direction and a magnitude (norm) Example (2-norm): x = [1, 2] T, x 2 = = 5 Properties satisfied by a vector norm ( ) x 0 and x = 0 if and only if x = 0 αx = α x (homogeneity) x + y x + y (triangle inequality)
4 Examples of vector norms x = [x 1, x 2,, x n ] T x 2 = x x x n 2 (2-norm) x 1 = x 1 + x x n (1-norm) x p = p x 1 p + x 2 p + + x n p (p-norm) x = max 1 i n x i ( -norm)
5 Distances x = [1, 2], y = [2, 1] x y = [1 2, 2 1] = [ 1, 1] Distance: x y 2 = [ 1, 1] 2 = 2 x y 1 = 2 x y = 1
6 Metrics Metrics: d(x, y) will satisfy the following properties: d(x, y) 0, and d(x, y) = 0 if and only if x = y d(x, y) = d(y, x) d(x, z) d(x, y) + d(y, z) (triangle inequality) Is d(x, y) = x y a valid metric if is a norm?
7 Metrics Metrics: d(x, y) will satisfy the following properties: d(x, y) 0, and d(x, y) = 0 if and only if x = y d(x, y) = d(y, x) d(x, z) d(x, y) + d(y, z) (triangle inequality) Is d(x, y) = x y a valid metric if is a norm? Yes. d(x, z) = x z = (x y) + (y z) x y + y z = d(x, y) + d(y, z)
8 Inner Product between Vectors x = [x 1,, x n ] T, y = [y 1,,, y n ] T Inner product: x T y = x 1 y 1 + x 2 y x n y n = n x i y i i=1 x T x = x 2 2, x y 2 2 = (x y)t (x y) Orthogonal: x y x T y = 0 (x and y are orthogonal to each other)
9 Projection onto a vector x T y = x 2 y 2 cos(θ) cos θ = x T y x 2 cos θ = x T y ( ) = x T ŷ x 2 y 2 y 2 Projection of x onto y: x 2 cos θ ŷ = ŷŷ T x }{{} projection matrix
10 Linearly independent Suppose we have 3 vectors x 1, x 2, x 3 x 1 = α 2 x 2 + α 3 x 3 x 1 is linearly dependent on x 2 and x 3 When are x 1,, x n linearly independent? α 1 x 1 + α 2 x α n x n = 0 if and only if α 1 = α 2 = = α n = 0
11 Linearly independent Suppose we have 3 vectors x 1, x 2, x 3 x 1 = α 2 x 2 + α 3 x 3 x 1 is linearly dependent on x 2 and x 3 When are x 1,, x n linearly independent? α 1 x 1 + α 2 x α n x n = 0 if and only if α 1 = α 2 = = α n = 0 A vector space is a set of vectors that is closed under vector addition & scalar multiplications. If x 1, x 2 V, then α 1 x 1 + α 2 x 2 V
12 Linearly independent Suppose we have 3 vectors x 1, x 2, x 3 x 1 = α 2 x 2 + α 3 x 3 x 1 is linearly dependent on x 2 and x 3 When are x 1,, x n linearly independent? α 1 x 1 + α 2 x α n x n = 0 if and only if α 1 = α 2 = = α n = 0 A vector space is a set of vectors that is closed under vector addition & scalar multiplications. If x 1, x 2 V, then α 1 x 1 + α 2 x 2 V A basis of the vector space is the maximal set of vectors in the subspace that are linearly independent of each other.
13 Linearly independent Suppose we have 3 vectors x 1, x 2, x 3 x 1 = α 2 x 2 + α 3 x 3 x 1 is linearly dependent on x 2 and x 3 When are x 1,, x n linearly independent? α 1 x 1 + α 2 x α n x n = 0 if and only if α 1 = α 2 = = α n = 0 A vector space is a set of vectors that is closed under vector addition & scalar multiplications. If x 1, x 2 V, then α 1 x 1 + α 2 x 2 V A basis of the vector space is the maximal set of vectors in the subspace that are linearly independent of each other. An orthogonal basis is a basis where all basis vectors are orthogonal to each other.
14 Linearly independent Suppose we have 3 vectors x 1, x 2, x 3 x 1 = α 2 x 2 + α 3 x 3 x 1 is linearly dependent on x 2 and x 3 When are x 1,, x n linearly independent? α 1 x 1 + α 2 x α n x n = 0 if and only if α 1 = α 2 = = α n = 0 A vector space is a set of vectors that is closed under vector addition & scalar multiplications. If x 1, x 2 V, then α 1 x 1 + α 2 x 2 V A basis of the vector space is the maximal set of vectors in the subspace that are linearly independent of each other. An orthogonal basis is a basis where all basis vectors are orthogonal to each other. Dimension of the vector space: number of vectors in the basis.
15 Matrices An m by n matrix: A R m n : a 11 a 12 a 1n a m1 a m2 a mn A matrix is also a linear transform: A : R m R n x Ax αx + βy A(αx + βy) = αax + βay (Linear Transform)
16 Matrix Norms Popular matrix norm: Frobenius norm m A F = i=1 j=1 n a ij 2 Matrix norms satisfy following properties: A 0 and A = 0 if and only if A = 0 (positivity) αa = α A (homogeneity) A + B A + B
17 Induced Norm Given a vector norm, we can define the corresponding induced norm or operator norm by Induced p-norm: Examples: A = sup{ Ax : x = 1} x = sup{ Ax x x : x 0} Ax p A p = sup x 0 x p Ax A 2 = sup 2 x 0 x 2 = σ max (A) (induced 2-norm is the maximum singular value) m A 1 = max j i=1 a ij n A = max i j=1 a ij
18 Rank of a matrix Column rank of A: the dimension of column space (vector space formed by column vectors) Row rank of A: the dimension of row space Column rank = row rank := rank (always true) Examples: Rank 2 matrix: [ ] = Rank 1 matrix: [ ] [ ] [1 ] =
19 Singular Value Decomposition
20 Low-rank Approximation Big Matrix A: Want to approximate by simple matrix  Goodness of approximation of  can be measured by (e.g., using F ) Low-rank approximation: A BC A Â
21 Important Question Given A and k, what is the best rank-k approximation? min B,C are rank k A BC F Optimal B and C are obtained by SVD (Singular Value Decomposition) of A
22 Singular Value Decomposition Any real matrix A R m n has the following Singular Value Decomposition (SVD): A = U Σ V T U is a m m unitary matrix (U T U = I ), columns of U are orthogonal V is a n n unitary matrix (V T V = I ), columns of V are orthogonal Σ is a diagonal m n matrix with non-negative real numbers on the diagonal. Usually, we assume the diagonal numbers are organized in descending order: Σ = diag(σ 1, σ 2,, σ n ), σ 1 σ 2 σ n
23 Singular Value Decomposition u 1, u 2,, u m : left singular vectors, basis of column space ui T u j = 0 i j, ui T u i = 1 i = 1,, m v 1, v 2,, v n : right singular vectors, basis of row space v T i v j = 0 i j, vi T v i = 1 i = 1,, n SVD is unique (up to permutations, rotations)
24 Linear Transformations Matrix A is a linear transformation For right singular vectors v 1,, v n, what are the vectors after transformation?
25 Linear Transformations Matrix A is a linear transformation For right singular vectors v 1,, v n, what are the vectors after transformation? A = UΣV T AV = UΣV T V = UΣ Therefore, Av i = σ i u i, i = 1,, n
26 Geometry of SVD Given an input vector x, how do we define the linear transform Ax? Assume A = UΣV T is the SVD, then Ax is: Step 1: Represent x as x = a 1 v a n v n, (a i = v T i a) Step 2: Transform v i σ i u i for all i: Ax = a 1 σ 1 u a n σ n v n
27 Reduced SVD (Thin SVD) If m > n, then only first n left singular vectors u 1,, u n are useful: Therefore, when m > n, we often just use the following thin SVD : A = UΣV T where U R m n, Σ, V R n n : U, V are unitary, and Σ is diagonal
28 Best Rank-k approximation Given a matrix A, how to form the best rank k approximation? arg min X :rank(x )=k Solution: truncated SVD: X A or arg min W R n k,h R m k X WHT A U r Σ r V T r, where U r, V r, Σ r are the first r columns of SVD matrices Why? Keep the k-most important components in SVD
29 Application: Data Approximation/Compression Given a data matrix A R m n Approximate the data matrix by the best rank-k approximation: A WH T Compress the data: O(mn) memory O(mk + nk) memory De-noising (sometimes) Example: if each column of A is a data, then w i (i-th column of W ) is dictionary, and H is the coefficient: a i k H ij w j j=1
30 Eigenvalue Decomposition
31 Eigen Decomposition For a n by n matrix H, if Hy = λy, then we say λ is an eigenvalue of H y is the corresponding eigenvector
32 Eigen Decomposition Consider A R n n to be a square, symmetric matrix. The eigenvalue decomposition of A is: A = V ΛV T, V T V = I (V is unitary), Λ is diagonal A = V ΛV T AV = V Λ Av i = λ i v i, i = 1,, n Each v i is an eigenvector, and each λ i is an eigenvalue
33 Eigen Decomposition Consider A R n n to be a square, symmetric matrix. The eigenvalue decomposition of A is: A = V ΛV T, V T V = I (V is unitary), Λ is diagonal A = V ΛV T AV = V Λ Av i = λ i v i, i = 1,, n Each v i is an eigenvector, and each λ i is an eigenvalue Usually, we assume the diagonal numbers are organized in descending order: Λ = diag(λ 1, λ 2,, λ n ), λ 1 λ 2 λ n Eigenvalue decomposition is unique when there are n unique eigenvalues.
34 Eigen Decomposition Each eigenvector v i will be mapped to Av i = λv i after the linear transform: Scaling without changing the direction of eigenvectors Ax = m i=1 λ iv i (vi T x) Project x to eigenvectors, and then scaling each vector
35 Eigen Decomposition
36 How to compute SVD/eigen decomposition?
37 Eigen Decomposition & SVD SVD can be transformed to eigenvalue decomposition!
38 Eigen Decomposition & SVD SVD can be transformed to eigenvalue decomposition! Given A R m n, and A = UΣV T (SVD) We have AA T = UΣV T V ΣU T = UΣ 2 U T (eigen decomposition of AA T ) After getting U, Σ, we can compute V by V T = Σ 1 U T A
39 Eigen Decomposition & SVD SVD can be transformed to eigenvalue decomposition! Given A R m n, and A = UΣV T (SVD) We have AA T = UΣV T V ΣU T = UΣ 2 U T (eigen decomposition of AA T ) After getting U, Σ, we can compute V by V T = Σ 1 U T A A T A = V Σ 2 V T : Eigen decomposition of AA T After getting V, Σ, we can compute U = AV Σ 1
40 Eigen Decomposition & SVD Which one should we use? eigenvalue decomposition of AA T or A T A? Depends on dimensionality m vs n
41 Eigen Decomposition & SVD Which one should we use? eigenvalue decomposition of AA T or A T A? Depends on dimensionality m vs n If A R m n and m > n: Step 1: Compute the eigenvalue decomposition of A T A = V Σ 2 V T Step 2: Compute U = AV Σ 1 If A R m n and m < n: Step 1: Compute the eigenvalue decomposition of AA T = UΣ 2 U T Step 2: Compute V = A T UΣ 1
42 Computing Eigenvalue Decomposition If A is a dense matrix: Call scipy.linalg.eig If A is a (large) sparse matrix: usually we only compute compute top-k eigenvectors with a small k Power method Lanczos algorithm (implemented in build-in packages) Call scipy.sparse.linalg.eigs
43 Dense Eigenvalue Decomposition Only for dense matrix Will compute all the eigenvalues and eigenvectors >>> import numpy as np >>> from scipy import linalg >>> A = np.array([[1, 2], [3, 4]]) >>> S, V = linalg.eig(a) >>> S array([ j, j]) >>> V array([[ , ], [ , ]])
44 Dense SVD Only for dense matrix Specify full matrices=false for the thin SVD >>> A = np.array([[1,2],[3,4], [5,6]]) >>> U, S, V = linalg.svd(a) >>> U array([[ , , ], [ , , ], [ , , ]]) >>> s array([ , ]) >>> U, S, V = linalg.svd(a, full_matrices=false) >>> U array([[ , ], [ , ], [ , ]])
45 Sparse Eigenvalue Decomposition Usually only compute the top-k eigenvalues and eigenvectors (since the U, V will be dense) Use iterative methods to compute U k (top-k eigenvectors): Initialize U k by a n-by-k random matrix U (0) k Iteratively update U k : Converge to the SVD solution: U (0) k U (1) k U (2) k lim t U(t) k = U k
46 Power Iteration for top eigenvector Main idea: compute (A T v)/ A T v for large T. A T v/ A T v top eigen vector when T
47 Power Iteration for top eigenvector Main idea: compute (A T v)/ A T v for large T. A T v/ A T v top eigen vector when T Power Iteration for top eigenvector Input: A R d d, number of iterations T Initialize a random vector v R d For t = 1, 2,, T v Av v v/ v Output v (top eigenvector)
48 Power Iteration for top eigenvector Power Iteration for top eigenvector Input: A R d d, number of iterations T Initialize a random vector v R d For t = 1, 2,, T v Av v v/ v Output v (top eigenvector) Top eigenvalue is v T Av (= v T λv = λ v 2 = λ) Only need one matrix-vector product at each iteartion Time complexity: O(nnz(A)) per iteration If A = XX T, then Av = X (X T v) No need for explicitly forming A
49 Power Iteration If A = UΛU T is the eigenvalue decomposition of A What is the eigenvalue decomposition of A t? λ t A t = (UΛU T )(UΛU T ) (UΛU T 0 λ t 2 0 ) = U λ t d UT A t v = d i=1 λ t i (u T i v)u i u 1 + ( λ 2 λ 1 ) t ( ut 2 v u T 1 v )u ( λ d λ 1 ) t ( ut d v u T 1 v )u d If v is a random vector, then u1 T v 0 with probability 1, so A t v A t v u 1 as t
50 Power iteration for top k eigenvectors Power Iteration for top eigenvector Input: A R d d, number of iterations T, rank k Initialize a random matrix V R d k For t = 1, 2,, T V AV [Q, R] qr(v ) V Q S V T AV [Ū, S] eig( S) Output eigenvectors U = VU, and eigenvalues S Sometimes faster than default functions (linalg.eigs and linalg.svds), since they aim to compute very accurate solutions.
51 Coming up Numerical Linear Algebra for Machine Learning Questions?
STA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 5: Numerical Linear Algebra Cho-Jui Hsieh UC Davis April 20, 2017 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationReview of Some Concepts from Linear Algebra: Part 2
Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationBasic Calculus Review
Basic Calculus Review Lorenzo Rosasco ISML Mod. 2 - Machine Learning Vector Spaces Functionals and Operators (Matrices) Vector Space A vector space is a set V with binary operations +: V V V and : R V
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More information7 Principal Component Analysis
7 Principal Component Analysis This topic will build a series of techniques to deal with high-dimensional data. Unlike regression problems, our goal is not to predict a value (the y-coordinate), it is
More information4 Linear Algebra Review
4 Linear Algebra Review For this topic we quickly review many key aspects of linear algebra that will be necessary for the remainder of the course 41 Vectors and Matrices For the context of data analysis,
More information2. Review of Linear Algebra
2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationLinear Algebra Review
January 29, 2013 Table of contents Metrics Metric Given a space X, then d : X X R + 0 and z in X if: d(x, y) = 0 is equivalent to x = y d(x, y) = d(y, x) d(x, y) d(x, z) + d(z, y) is a metric is for all
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationSingular Value Decomposition
Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =
More informationData Mining Lecture 4: Covariance, EVD, PCA & SVD
Data Mining Lecture 4: Covariance, EVD, PCA & SVD Jo Houghton ECS Southampton February 25, 2019 1 / 28 Variance and Covariance - Expectation A random variable takes on different values due to chance The
More informationFall TMA4145 Linear Methods. Exercise set Given the matrix 1 2
Norwegian University of Science and Technology Department of Mathematical Sciences TMA445 Linear Methods Fall 07 Exercise set Please justify your answers! The most important part is how you arrive at an
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationNORMS ON SPACE OF MATRICES
NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationInner products and Norms. Inner product of 2 vectors. Inner product of 2 vectors x and y in R n : x 1 y 1 + x 2 y x n y n in R n
Inner products and Norms Inner product of 2 vectors Inner product of 2 vectors x and y in R n : x 1 y 1 + x 2 y 2 + + x n y n in R n Notation: (x, y) or y T x For complex vectors (x, y) = x 1 ȳ 1 + x 2
More information4 Linear Algebra Review
Linear Algebra Review For this topic we quickly review many key aspects of linear algebra that will be necessary for the remainder of the text 1 Vectors and Matrices For the context of data analysis, the
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationLinear Algebra - Part II
Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationLinear Algebra Methods for Data Mining
Linear Algebra Methods for Data Mining Saara Hyvönen, Saara.Hyvonen@cs.helsinki.fi Spring 2007 1. Basic Linear Algebra Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki Example
More informationThe Singular Value Decomposition
The Singular Value Decomposition An Important topic in NLA Radu Tiberiu Trîmbiţaş Babeş-Bolyai University February 23, 2009 Radu Tiberiu Trîmbiţaş ( Babeş-Bolyai University)The Singular Value Decomposition
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 9: Dimension Reduction/Word2vec Cho-Jui Hsieh UC Davis May 15, 2018 Principal Component Analysis Principal Component Analysis (PCA) Data
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More informationMachine Learning. B. Unsupervised Learning B.2 Dimensionality Reduction. Lars Schmidt-Thieme, Nicolas Schilling
Machine Learning B. Unsupervised Learning B.2 Dimensionality Reduction Lars Schmidt-Thieme, Nicolas Schilling Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 6: Numerical Linear Algebra: Applications in Machine Learning Cho-Jui Hsieh UC Davis April 27, 2017 Principal Component Analysis Principal
More informationExercise Sheet 1. 1 Probability revision 1: Student-t as an infinite mixture of Gaussians
Exercise Sheet 1 1 Probability revision 1: Student-t as an infinite mixture of Gaussians Show that an infinite mixture of Gaussian distributions, with Gamma distributions as mixing weights in the following
More informationNormed & Inner Product Vector Spaces
Normed & Inner Product Vector Spaces ECE 174 Introduction to Linear & Nonlinear Optimization Ken Kreutz-Delgado ECE Department, UC San Diego Ken Kreutz-Delgado (UC San Diego) ECE 174 Fall 2016 1 / 27 Normed
More informationChapter 7: Symmetric Matrices and Quadratic Forms
Chapter 7: Symmetric Matrices and Quadratic Forms (Last Updated: December, 06) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationGI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis. Massimiliano Pontil
GI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis Massimiliano Pontil 1 Today s plan SVD and principal component analysis (PCA) Connection
More informationIV. Matrix Approximation using Least-Squares
IV. Matrix Approximation using Least-Squares The SVD and Matrix Approximation We begin with the following fundamental question. Let A be an M N matrix with rank R. What is the closest matrix to A that
More informationLinear Least Squares. Using SVD Decomposition.
Linear Least Squares. Using SVD Decomposition. Dmitriy Leykekhman Spring 2011 Goals SVD-decomposition. Solving LLS with SVD-decomposition. D. Leykekhman Linear Least Squares 1 SVD Decomposition. For any
More informationKnowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationLatent Semantic Analysis. Hongning Wang
Latent Semantic Analysis Hongning Wang CS@UVa VS model in practice Document and query are represented by term vectors Terms are not necessarily orthogonal to each other Synonymy: car v.s. automobile Polysemy:
More informationLecture 8: Linear Algebra Background
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 8: Linear Algebra Background Lecturer: Shayan Oveis Gharan 2/1/2017 Scribe: Swati Padmanabhan Disclaimer: These notes have not been subjected
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationThe following definition is fundamental.
1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic
More informationSingular Value Decomposition
Chapter 5 Singular Value Decomposition We now reach an important Chapter in this course concerned with the Singular Value Decomposition of a matrix A. SVD, as it is commonly referred to, is one of the
More informationProbabilistic Latent Semantic Analysis
Probabilistic Latent Semantic Analysis Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition
More informationMATH 612 Computational methods for equation solving and function minimization Week # 2
MATH 612 Computational methods for equation solving and function minimization Week # 2 Instructor: Francisco-Javier Pancho Sayas Spring 2014 University of Delaware FJS MATH 612 1 / 38 Plan for this week
More informationSingular Value Decomposition (SVD)
School of Computing National University of Singapore CS CS524 Theoretical Foundations of Multimedia More Linear Algebra Singular Value Decomposition (SVD) The highpoint of linear algebra Gilbert Strang
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationNumerical Methods I Singular Value Decomposition
Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)
More informationIntroduction to Matrix Algebra
Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p
More informationThroughout these notes we assume V, W are finite dimensional inner product spaces over C.
Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal
More informationPrincipal Component Analysis
Machine Learning Michaelmas 2017 James Worrell Principal Component Analysis 1 Introduction 1.1 Goals of PCA Principal components analysis (PCA) is a dimensionality reduction technique that can be used
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More information1 Inner Product and Orthogonality
CSCI 4/Fall 6/Vora/GWU/Orthogonality and Norms Inner Product and Orthogonality Definition : The inner product of two vectors x and y, x x x =.., y =. x n y y... y n is denoted x, y : Note that n x, y =
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationComputational math: Assignment 1
Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationLecture 7. Econ August 18
Lecture 7 Econ 2001 2015 August 18 Lecture 7 Outline First, the theorem of the maximum, an amazing result about continuity in optimization problems. Then, we start linear algebra, mostly looking at familiar
More informationLinear Algebra Methods for Data Mining
Linear Algebra Methods for Data Mining Saara Hyvönen, Saara.Hyvonen@cs.helsinki.fi Spring 2007 The Singular Value Decomposition (SVD) continued Linear Algebra Methods for Data Mining, Spring 2007, University
More informationLecture notes: Applied linear algebra Part 1. Version 2
Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and
More information15 Singular Value Decomposition
15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationLinear Algebra and Eigenproblems
Appendix A A Linear Algebra and Eigenproblems A working knowledge of linear algebra is key to understanding many of the issues raised in this work. In particular, many of the discussions of the details
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More information2. LINEAR ALGEBRA. 1. Definitions. 2. Linear least squares problem. 3. QR factorization. 4. Singular value decomposition (SVD) 5.
2. LINEAR ALGEBRA Outline 1. Definitions 2. Linear least squares problem 3. QR factorization 4. Singular value decomposition (SVD) 5. Pseudo-inverse 6. Eigenvalue decomposition (EVD) 1 Definitions Vector
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationPseudoinverse & Moore-Penrose Conditions
ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego p. 1/1 Lecture 7 ECE 275A Pseudoinverse & Moore-Penrose Conditions ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego
More informationMain matrix factorizations
Main matrix factorizations A P L U P permutation matrix, L lower triangular, U upper triangular Key use: Solve square linear system Ax b. A Q R Q unitary, R upper triangular Key use: Solve square or overdetrmined
More informationHomework 1. Yuan Yao. September 18, 2011
Homework 1 Yuan Yao September 18, 2011 1. Singular Value Decomposition: The goal of this exercise is to refresh your memory about the singular value decomposition and matrix norms. A good reference to
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationVector Spaces and Linear Transformations
Vector Spaces and Linear Transformations Wei Shi, Jinan University 2017.11.1 1 / 18 Definition (Field) A field F = {F, +, } is an algebraic structure formed by a set F, and closed under binary operations
More informationA matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and
Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.
More informationQuantum Computing Lecture 2. Review of Linear Algebra
Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces
More informationReview of Linear Algebra
Review of Linear Algebra Dr Gerhard Roth COMP 40A Winter 05 Version Linear algebra Is an important area of mathematics It is the basis of computer vision Is very widely taught, and there are many resources
More informationDATA MINING LECTURE 8. Dimensionality Reduction PCA -- SVD
DATA MINING LECTURE 8 Dimensionality Reduction PCA -- SVD The curse of dimensionality Real data usually have thousands, or millions of dimensions E.g., web documents, where the dimensionality is the vocabulary
More informationA matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and
Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.
More informationFundamentals of Matrices
Maschinelles Lernen II Fundamentals of Matrices Christoph Sawade/Niels Landwehr/Blaine Nelson Tobias Scheffer Matrix Examples Recap: Data Linear Model: f i x = w i T x Let X = x x n be the data matrix
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 26, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 55 High dimensional
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationSingular Value Decomposition
Chapter 6 Singular Value Decomposition In Chapter 5, we derived a number of algorithms for computing the eigenvalues and eigenvectors of matrices A R n n. Having developed this machinery, we complete our
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More informationLEC 2: Principal Component Analysis (PCA) A First Dimensionality Reduction Approach
LEC 2: Principal Component Analysis (PCA) A First Dimensionality Reduction Approach Dr. Guangliang Chen February 9, 2016 Outline Introduction Review of linear algebra Matrix SVD PCA Motivation The digits
More information