# Linear Algebra for Machine Learning. Sargur N. Srihari

Size: px
Start display at page:

## Transcription

1 Linear Algebra for Machine Learning Sargur N. 1

2 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it Essential for understanding ML algorithms Here we discuss: Discuss scalars, vectors, matrices, tensors Multiplying matrices/vectors Inverse, Span, Linear Independence SVD, PCA 2

3 Scalar Single number Represented in lower-case italic x E.g., let x! be the slope of the line Defining a real-valued scalar E.g., let n! be the number of units Defining a natural number scalar 3

4 Vector An array of numbers Arranged in order Each no. identified by an index Vectors are shown in lower-case bold x= x 1 x 2 x n x T = x 1,x 2,..x n If each element is in R then x is in R n We think of vectors as points in space Each element gives coordinate along an axis 4

5 Matrix 2-D array of numbers Each element identified by two indices Denoted by bold typeface A Elements indicated as A m,n E.g., A 1,1 A 1,2 A = A 2,1 A 2,2 A i: is i th row of A, A :j is j th column of A If A has shape of height m and width n with 5 real-values then A! m n

6 Tensor Sometimes need an array with more than two axes An array arranged on a regular grid with variable number of axes is referred to as a tensor Denote a tensor with bold typeface: A Element (i,j,k) of tensor denoted by A i,j,k 6

7 Transpose of a Matrix Mirror image across principal diagonal A = A 1,1 A 1,2 A 1,3 A 2,1 A 2,2 A 2,3 A 3,1 A 3,2 A 3,3 A T = A 1,1 A 2,1 A 3,1 A 1,2 A 2,2 A 3,2 A 1,3 A 2,3 A 3,3 Vectors are matrices with a single column Often written in-line using transpose x = [x 1,..,x n ] T Since a scalar is a matrix with one element a=a T 7

8 Matrix Addition If A and B have same shape (height m, width n) C = A+ B C i,j = A i,j + B i,j A scalar can be added to a matrix or multiplied by a scalar D = ab + c D i,j = ab i,j + c Vector added to matrix (non-standard matrix algebra) C = A+ b C i,j = A i,j + b j Called broadcasting since vector b is added to each 8 row of A

9 Multiplying Matrices For product C=AB to be defined, A has to have the same no. of columns as the no. of rows of B If A is of shape mxn and B is of shape nxp then matrix product C is of shape mxp C = AB C i,j = A i,k B k,j k Note that the standard product of two matrices is not just the product of two individual elements 9

10 Multiplying Vectors Dot product of two vectors x and y of same dimensionality is the matrix product x T y Conversely, matrix product C=AB can be viewed as computing C ij the dot product of row i of A and column j of B 10

11 Matrix Product Properties Distributivity over addition: A(B+C)=AB+AC Associativity: A(BC)=(AB)C Not commutative: AB=BA is not always true Dot product between vectors is commutative: x T y=y T x Transpose of a matrix product has a simple form: (AB) T =B T A T 11

12 Linear Transformation Ax=b where A! n n and More explicitly b! n A 11 x 1 + A 12 x A 1n x n = b 1 A 21 x 1 + A 22 x A 2n x n = b 2 n equations in n unknowns A n1 x 1 + A m2 x A n,n x n = b n A = A 1,1! A 1,n " " " A n,1! A nn x= x 1 " x n b= b 1 " b n n x n n x 1 n x 1 Can view A as a linear transformation of vector x to vector b Sometimes we wish to solve for the unknowns x ={x 1,..,x n } when A and b provide constraints 12

13 Identity and Inverse Matrices Matrix inversion is a powerful tool to analytically solve Ax=b Needs concept of Identity matrix Identity matrix does not change value of vector when we multiply the vector by identity matrix Denote identity matrix that preserves n-dimensional vectors as I n Formally and Example of I 3 I n! n n x! n,i n x = x 13

14 Matrix Inverse Inverse of square matrix A defined as We can now solve Ax=b as follows: Ax = b A 1 Ax = A 1 b I n x = A 1 b A 1 A = I n x = A 1 b This depends on being able to find A -1 If A -1 exists there are several methods for finding it 14

15 Solving Simultaneous equations Ax = b where A is (M+1) x (M+1) x is (M+1) x 1: set of weights to be determined b is N x 1 Two closed-form solutions 1. Matrix inversion x=a -1 b 2. Gaussian elimination 15

16 Linear Equations: Closed-Form Solutions 1. Matrix Formulation: Ax=b Solution: x=a -1 b 2. Gaussian Elimination followed by back-substitution L 2-3L 1 àl 2 L 3-2L 1 àl 3 -L 2 /4àL 2

17 Example: System of Linear Equations in Linear Regression Instead of Ax=b We have Φw = t where Φ is design matrix of m features (basis functions ϕ i (x j ) ) for samples x j, t is targets of sample We need weight w to be used with m basis functions to determine output y(x, w)= m i=1 w i φ i ( x) 17

18 Disadvantage of closed-form solutions If A -1 exists, the same A -1 can be used for any given b But A -1 cannot be represented with sufficient precision It is not used in practice Gaussian elimination also has disadvantages numerical instability (division by small no.) O(n 3 ) for n x n matrix Software solutions use value of b in finding x E.g., difference (derivative) between b and output is 18 used iteratively

19 How many solutions for Ax=b exist? System of equations with n variables and m equations is Solution is x=a -1 b In order for A -1 to exist Ax=b must have exactly one solution for every value of b A 11 x 1 + A 12 x A 1n x n = b 1 A 21 x 1 + A 22 x A 2n x n = b 2 A m1 x 1 + A m2 x A mn x n = b m It is also possible for the system of equations to have no solutions or an infinite no. of solutions for some values of b It is not possible to have more than one but fewer than infinitely many solutions If x and y are solutions then z=α x + (1-α) y is a 19 solution for any real α

20 Span of a set of vectors Span of a set of vectors: set of points obtained by a linear combination of those vectors A linear combination of vectors {v (1),.., v (n) } with coefficients c i is c i v (i) i System of equations is Ax=b A column of A, i.e., A :i specifies travel in direction i How much we need to travel is given by x i This is a linear combination of vectors Ax= x i A :, i i Thus determining whether Ax=b has a solution is equivalent to determining whether b is in the span of columns of A This span is referred to as column space or range of A

21 Conditions for a solution to Ax=b Matrix must be square, i.e., m=n and all columns must be linearly independent Necessary condition is For a solution to exist when column space be all of! m Sufficient Condition n m b! m we require the If columns are linear combinations of other columns, column space is less than! m Columns are linearly dependent or matrix is singular For column space to encompass of m linearly independent columns! m For non-square and singular matrices at least one set Methods other than matrix inversion are used

22 Norms Used for measuring the size of a vector Norms map vectors to non-negative values Norm of vector x is distance from origin to x It is any function f that satisfies: ( ) = 0 x= 0 f x f (x+ y ) f ( x)+ f y α! f ( α x)= α f x ( ) Triangle Inequality ( ) 22

23 Definition L 2 Norm L P Norm Called Euclidean norm, written simply as x Squared Euclidean norm is same as x T x L 1 Norm Useful when 0 and non-zero have to be distinguished (since L 2 increases slowly near origin, e.g., =0.01) L Norm x = p x = max i i x i x i p 1 p Called max norm 23

24 Size of a Matrix Frobenius norm 2 A = A i,j F i,j 1 2 It is analogous to L 2 norm of a vector 24

25 Angle between Vectors Dot product of two vectors can be written in terms of their L 2 norms and angle θ between them x T y= x 2 y 2 cosθ 25

26 Special kinds of Matrices Diagonal Matrix Mostly zeros, with non-zero entries in diagonal diag (v) is a square diagonal matrix with diagonal elements given by entries of vector v Multiplying diag(v) by vector x only needs to scale each element x i by v i diag(v)x= v x Symmetric Matrix Is equal to its transpose: A=A T E.g., a distance matrix is symmetric with A ij =A ji

27 Special Kinds of Vectors Unit Vector A vector with unit norm Orthogonal Vectors A vector x and a vector y are orthogonal to each other if x T y=0 Vectors are at 90 degrees to each other Orthogonal Matrix x 2 =1 A square matrix whose rows are mutually orthonormal A -1 =A T 27

28 Matrix decomposition Matrices can be decomposed into factors to learn universal properties about them not discernible from their representation E.g., from decomposition of integer into prime factors 12=2x2x3 we can discern that 12 is not divisible by 5 or any multiple of 12 is divisible by 3 But representations of 12 in binary or decimal are different Analogously, a matrix is decomposed into Eigenvalues and Eigenvectors to discern universal properties 28

29 Eigenvector An eigenvector of a square matrix A is a non-zero vector v such that multiplication by A only changes the scale of v Av=λv The scalar λ is known as eigenvalue If v is an eigenvector of A, so is any rescaled vector sv. Moreover sv still has the same eigen value. Thus look for a unit eigenvector Wikipedia 29

30 Eigenvalue and Characteristic Polynomial Consider Av=w A = A 1,1! A 1,n " " " A n,1! A nn If v and w are scalar multiples, i.e., if Av=λv then v is an eigenvector of the linear transformation A and the scale factor λ is the eigenvalue corresponding to the eigen vector This is the eigenvalue equation of matrix A Stated equivalently as (A-λI)v=0 This has a non-zero solution if A-λI =0 as The polynomial of degree n can be factored as A-λI = (λ 1 -λ)(λ 2 -λ) (λ n -λ) The λ 1, λ 2 λ n are roots of the polynomial and are eigenvalues of A v = v 1 " v n w= w 1 " w n

31 Example of Eigenvalue/Eigenvector Consider the matrix A = Taking determinant of (A-λI), the char poly is A λi = 2 λ λ = 3 4λ + λ2 It has roots λ=1 and λ=3 which are the two eigenvalues of A The eigenvectors are found by solving for v in Av=λv, which are v λ=1 = 1 1,v = 1 λ=3 1 31

32 Example of Eigen Vector Vectors are grid points Wikipedia 32

33 Eigendecomposition Suppose that matrix A has n linearly independent eigenvectors {v (1),..,v (n) } with eigenvalues {λ 1,..,λ n } Concatenate eigenvectors to form matrix V Concatenate eigenvalues to form vector λ=[λ 1,..,λ n ] Eigendecomposition of A is given by A=Vdiag(λ)V -1 33

34 Decomposition of Symmetric Matrix Every real symmetric matrix A can be decomposed into real-valued eigenvectors and eigenvalues A=QΛQ T where Q is an orthogonal matrix composed of eigenvectors of A: {v (1),..,v (n) } orthogonal matrix: components are orthogonal or v (i)t v (j) =0 Λ is a diagonal matrix of eigenvalues {λ 1,..,λ n } We can think of A as scaling space by λ i in direction v (i) See figure on next slide 34

35 Effect of Eigenvectors and Eigenvalues Example of 2x2 matrix Matrix A with two orthonormal eigenvectors v (1) with eigenvalue λ 1, v (2) with eigenvalue λ 2 Plot of unit vectors (circle) u! 2 Plot of vectors Au (ellipse) with two variables x 1 and x 2 35

36 Eigendecomposition is not unique Eigendecomposition is A=QΛQ T where Q is an orthogonal matrix composed of eigenvectors of A Decomposition is not unique when two eigenvalues are the same By convention order entries of Λ in descending order: Under this convention, eigendecomposition is unique if all eigenvalues are unique 36

37 What does eigendecomposition tell us? Many useful facts about the matrix obtained 1. Matrix is singular iff any of the eigenvalues are zero 2. Can be used to optimize quadratic expressions of the form f(x)=x T Ax subject to x 2 =1 Whenever x is equal to an eigenvector, f is equal to its eigenvalue Max value of f is max eigen value, min value is min eigen value 37

38 Positive Definite Matrix A matrix whose eigenvalues are all positive is called positive definite Positive or zero is called positive semidefinite If eigen values are all negative it is negative definite Positive definite matrices guarantee that x T Ax 0 38

39 Singular Value Decomposition (SVD) Eigendecomposition has form: A=Vdiag(λ)V -1 If A is not square eigendecomposition undefined SVD is a decomposition of the form A=UDV T SVD is more general than eigendecomposition Used with any matrix rather than symmetric ones Every real matrix has a SVD (not so of eigen)

40 SVD Definition We write A as the product of three matrices A=UDV T If A is mxn, then U is defined to be mxm, D is mxn and V is nxn D is a diagonal matrix not necessarily square Elements of Diagonal of D are called singular values U and V are orthogonal matrices Component vectors of U are called left singular vectors Left singular vectors of A are eigenvectors of AA T Right singular vectors of A are eigenvectors of A T A

41 Moore-Penrose Pseudoinverse Most useful feature of SVD is that it can be used to generalize matrix inversion to nonsquare matrices Practical algorithms for computing the pseudoinverse of A are based on SVD A + =VD + U T where U,D,V are the SVD of A Psedudoinverse D + of D is obtained by taking the reciprocal of its nonzero elements when taking transpose of resulting matrix 41

42 Trace of a Matrix Trace operator gives the sum of the elements along the diagonal Tr(A )= A i,i i,i Frobenius norm of a matrix can be represented as A = Tr(A) F ( )

43 Determinant of a Matrix Determinant of a square matrix det(a) is a mapping to a scalar It is equal to the product of all eigenvalues of the matrix Measures how much multiplication by the matrix expands or contracts space 43

44 Example: PCA A simple ML algorithm is Principal Components Analysis It can be derived using only knowledge of basic linear algebra 44

45 PCA Problem Statement Given a collection of m points {x (1),..,x (m) } in R n represent them in a lower dimension. For each point x (i) find a code vector c (i) in R l If l is smaller than n it will take less memory to store the points This is lossy compression Find encoding function f (x) = c and a decoding function x g ( f (x) ) 45

46 PCA using Matrix multiplication One choice of decoding function is to use matrix multiplication: g(c) =Dc where D is a matrix with l columns To keep encoding eacy, we PCA require columns of D to be orthogonal to each other To constrain solutions we require columns of D to have unit norm We need to find optimal code c* given D Then we need optimal D D! n l 46

47 Finding optimal code given D To generate optimal code point c* given input x, minimize the distance between input point x and its reconstruction g(c*) c* = argmin c x g(c) 2 Using squared L 2 instead of L 2, function being minimized is equivalent to (x g(c)) T (x g(c)) Using g(c)=dc optimal code can be shown to be equivalent to c* = argmin 2x T Dc+c T c c 47

48 Optimal Encoding for PCA Using vector calculus Thus we can encode x using a matrix-vector operation To encode we use f(x)=d T x c ( 2x T Dc+c T c)= 0 2D T x+2c = 0 c = D T x For PCA reconstruction, since g(c)=dc we use r(x)=g(f(x))=dd T x Next we need to choose the encoding matrix D 48

49 Method for finding optimal D Revisit idea of minimizing L 2 distance between inputs and reconstructions But cannot consider points in isolation So minimize error over all points: Frobenius norm subject to D T D=I l Use design matrix X, D*= argmin D ( x (i) j r( x ) (i) j Given by stacking all vectors describing the points To derive algorithm for finding D* start by considering the case l=1 In this case D is just a single vector d i,j X! m n )

50 Final Solution to PCA For l=1, the optimization problem is solved using eigndecomposition Specifically the optimal d is given by the eigenvector of X T X corresponding to the largest eigenvalue More generally, matrix D is given by the l eigenvectors of X corresponding to the largest eigenvalues (Proof by induction) 50

### Linear Algebra - Part II

Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T

### The study of linear algebra involves several types of mathematical objects:

Chapter 2 Linear Algebra Linear algebra is a branch of mathematics that is widely used throughout science and engineering. However, because linear algebra is a form of continuous rather than discrete mathematics,

### Deep Learning Book Notes Chapter 2: Linear Algebra

Deep Learning Book Notes Chapter 2: Linear Algebra Compiled By: Abhinaba Bala, Dakshit Agrawal, Mohit Jain Section 2.1: Scalars, Vectors, Matrices and Tensors Scalar Single Number Lowercase names in italic

### Large Scale Data Analysis Using Deep Learning

Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset

### Linear Algebra. Shan-Hung Wu. Department of Computer Science, National Tsing Hua University, Taiwan. Large-Scale ML, Fall 2016

Linear Algebra Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Large-Scale ML, Fall 2016 Shan-Hung Wu (CS, NTHU) Linear Algebra Large-Scale ML, Fall

### Linear Algebra Review. Vectors

Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

### [POLS 8500] Review of Linear Algebra, Probability and Information Theory

[POLS 8500] Review of Linear Algebra, Probability and Information Theory Professor Jason Anastasopoulos ljanastas@uga.edu January 12, 2017 For today... Basic linear algebra. Basic probability. Programming

### Linear Algebra (Review) Volker Tresp 2018

Linear Algebra (Review) Volker Tresp 2018 1 Vectors k, M, N are scalars A one-dimensional array c is a column vector. Thus in two dimensions, ( ) c1 c = c 2 c i is the i-th component of c c T = (c 1, c

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### Linear Algebra and Eigenproblems

Appendix A A Linear Algebra and Eigenproblems A working knowledge of linear algebra is key to understanding many of the issues raised in this work. In particular, many of the discussions of the details

### Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### CS 246 Review of Linear Algebra 01/17/19

1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### Mathematical foundations - linear algebra

Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Linear Algebra (Review) Volker Tresp 2017

Linear Algebra (Review) Volker Tresp 2017 1 Vectors k is a scalar (a number) c is a column vector. Thus in two dimensions, c = ( c1 c 2 ) (Advanced: More precisely, a vector is defined in a vector space.

### UNIT 6: The singular value decomposition.

UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T

### Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB

Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the

### 3 (Maths) Linear Algebra

3 (Maths) Linear Algebra References: Simon and Blume, chapters 6 to 11, 16 and 23; Pemberton and Rau, chapters 11 to 13 and 25; Sundaram, sections 1.3 and 1.5. The methods and concepts of linear algebra

### Linear Algebra & Geometry why is linear algebra useful in computer vision?

Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Review of Linear Algebra

Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=

### Image Registration Lecture 2: Vectors and Matrices

Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this

### Linear Algebra Review. Fei-Fei Li

Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector

### Introduction to Matrix Algebra

Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### Elementary maths for GMT

Elementary maths for GMT Linear Algebra Part 2: Matrices, Elimination and Determinant m n matrices The system of m linear equations in n variables x 1, x 2,, x n a 11 x 1 + a 12 x 2 + + a 1n x n = b 1

### Foundations of Computer Vision

Foundations of Computer Vision Wesley. E. Snyder North Carolina State University Hairong Qi University of Tennessee, Knoxville Last Edited February 8, 2017 1 3.2. A BRIEF REVIEW OF LINEAR ALGEBRA Apply

### Quantum Computing Lecture 2. Review of Linear Algebra

Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces

### Chapter 3 Transformations

Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

### Knowledge Discovery and Data Mining 1 (VO) ( )

Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory

### CS 143 Linear Algebra Review

CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see

### TBP MATH33A Review Sheet. November 24, 2018

TBP MATH33A Review Sheet November 24, 2018 General Transformation Matrices: Function Scaling by k Orthogonal projection onto line L Implementation If we want to scale I 2 by k, we use the following: [

### COMP6237 Data Mining Covariance, EVD, PCA & SVD. Jonathon Hare

COMP6237 Data Mining Covariance, EVD, PCA & SVD Jonathon Hare jsh2@ecs.soton.ac.uk Variance and Covariance Random Variables and Expected Values Mathematicians talk variance (and covariance) in terms of

### Computational Methods CMSC/AMSC/MAPL 460. Eigenvalues and Eigenvectors. Ramani Duraiswami, Dept. of Computer Science

Computational Methods CMSC/AMSC/MAPL 460 Eigenvalues and Eigenvectors Ramani Duraiswami, Dept. of Computer Science Eigen Values of a Matrix Recap: A N N matrix A has an eigenvector x (non-zero) with corresponding

### Eigenvalues and diagonalization

Eigenvalues and diagonalization Patrick Breheny November 15 Patrick Breheny BST 764: Applied Statistical Modeling 1/20 Introduction The next topic in our course, principal components analysis, revolves

### Singular Value Decomposition

Chapter 6 Singular Value Decomposition In Chapter 5, we derived a number of algorithms for computing the eigenvalues and eigenvectors of matrices A R n n. Having developed this machinery, we complete our

### Review of Linear Algebra

Review of Linear Algebra Dr Gerhard Roth COMP 40A Winter 05 Version Linear algebra Is an important area of mathematics It is the basis of computer vision Is very widely taught, and there are many resources

### Lecture II: Linear Algebra Revisited

Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function

### Review of Basic Concepts in Linear Algebra

Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra

### Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014

Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document

### Singular Value Decompsition

Singular Value Decompsition Massoud Malek One of the most useful results from linear algebra, is a matrix decomposition known as the singular value decomposition It has many useful applications in almost

### Econ Slides from Lecture 7

Econ 205 Sobel Econ 205 - Slides from Lecture 7 Joel Sobel August 31, 2010 Linear Algebra: Main Theory A linear combination of a collection of vectors {x 1,..., x k } is a vector of the form k λ ix i for

### Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p

Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the

### 14 Singular Value Decomposition

14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

### Singular Value Decomposition and Principal Component Analysis (PCA) I

Singular Value Decomposition and Principal Component Analysis (PCA) I Prof Ned Wingreen MOL 40/50 Microarray review Data per array: 0000 genes, I (green) i,i (red) i 000 000+ data points! The expression

### Linear Algebra: Matrix Eigenvalue Problems

CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

### Linear Algebra Review. Fei-Fei Li

Linear Algebra Review Fei-Fei Li 1 / 37 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector

### Computational Methods. Eigenvalues and Singular Values

Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

### Basic Concepts in Linear Algebra

Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear

### The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)

Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will

### A Quick Tour of Linear Algebra and Optimization for Machine Learning

A Quick Tour of Linear Algebra and Optimization for Machine Learning Masoud Farivar January 8, 2015 1 / 28 Outline of Part I: Review of Basic Linear Algebra Matrices and Vectors Matrix Multiplication Operators

### Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic

Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.2: Fundamentals 2 / 31 Eigenvalues and Eigenvectors Eigenvalues and eigenvectors of

### 10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang)

10-701/15-781 Recitation : Linear Algebra Review (based on notes written by Jing Xiang) Manojit Nandi February 1, 2014 Outline Linear Algebra General Properties Matrix Operations Inner Products and Orthogonal

### Basic Calculus Review

Basic Calculus Review Lorenzo Rosasco ISML Mod. 2 - Machine Learning Vector Spaces Functionals and Operators (Matrices) Vector Space A vector space is a set V with binary operations +: V V V and : R V

### Math Matrix Algebra

Math 44 - Matrix Algebra Review notes - (Alberto Bressan, Spring 7) sec: Orthogonal diagonalization of symmetric matrices When we seek to diagonalize a general n n matrix A, two difficulties may arise:

### Mathematical foundations - linear algebra

Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar

### MATH 583A REVIEW SESSION #1

MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),

### Linear algebra II Homework #1 solutions A = This means that every eigenvector with eigenvalue λ = 1 must have the form

Linear algebra II Homework # solutions. Find the eigenvalues and the eigenvectors of the matrix 4 6 A =. 5 Since tra = 9 and deta = = 8, the characteristic polynomial is f(λ) = λ (tra)λ+deta = λ 9λ+8 =

### Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

### EIGENVALUES AND SINGULAR VALUE DECOMPOSITION

APPENDIX B EIGENVALUES AND SINGULAR VALUE DECOMPOSITION B.1 LINEAR EQUATIONS AND INVERSES Problems of linear estimation can be written in terms of a linear matrix equation whose solution provides the required

### Background Mathematics (2/2) 1. David Barber

Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and

### MAC Module 12 Eigenvalues and Eigenvectors. Learning Objectives. Upon completing this module, you should be able to:

MAC Module Eigenvalues and Eigenvectors Learning Objectives Upon completing this module, you should be able to: Solve the eigenvalue problem by finding the eigenvalues and the corresponding eigenvectors

### Math Linear Algebra Final Exam Review Sheet

Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

### 15 Singular Value Decomposition

15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

### LECTURE 16: PCA AND SVD

Instructor: Sael Lee CS549 Computational Biology LECTURE 16: PCA AND SVD Resource: PCA Slide by Iyad Batal Chapter 12 of PRML Shlens, J. (2003). A tutorial on principal component analysis. CONTENT Principal

### Lecture 7. Econ August 18

Lecture 7 Econ 2001 2015 August 18 Lecture 7 Outline First, the theorem of the maximum, an amazing result about continuity in optimization problems. Then, we start linear algebra, mostly looking at familiar

### Multivariate Statistical Analysis

Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions

### Foundations of Matrix Analysis

1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the

### Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008

Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008 Exam 2 will be held on Tuesday, April 8, 7-8pm in 117 MacMillan What will be covered The exam will cover material from the lectures

### Stat 206: Linear algebra

Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two

### Linear Algebra. Lecture slides for Chapter 2 of Deep Learning Ian Goodfellow

Linear lgebra Lecture slides for Chapter 2 of Deep Learning Ian Goodfellow 206-06-24 bout this chapter Not a comprehensive survey of all of linear algebra Focused on the subset most relevant to deep learning

### Chap 3. Linear Algebra

Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions

### 7 Principal Component Analysis

7 Principal Component Analysis This topic will build a series of techniques to deal with high-dimensional data. Unlike regression problems, our goal is not to predict a value (the y-coordinate), it is

### 1 9/5 Matrices, vectors, and their applications

1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric

### Multistat2. The course is focused on the parts in bold; a short description will be presented for the other parts

1 2 The course is focused on the parts in bold; a short description will be presented for the other parts 3 4 5 What I cannot teach you: how to look for relevant literature, how to use statistical software

### 8. Diagonalization.

8. Diagonalization 8.1. Matrix Representations of Linear Transformations Matrix of A Linear Operator with Respect to A Basis We know that every linear transformation T: R n R m has an associated standard

### Vectors and Matrices Statistics with Vectors and Matrices

Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc

### Linear Algebra and Matrices

Linear Algebra and Matrices 4 Overview In this chapter we studying true matrix operations, not element operations as was done in earlier chapters. Working with MAT- LAB functions should now be fairly routine.

### 2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian

FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian

### Dimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas

Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx

### Conceptual Questions for Review

Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.

### Quiz ) Locate your 1 st order neighbors. 1) Simplify. Name Hometown. Name Hometown. Name Hometown.

Quiz 1) Simplify 9999 999 9999 998 9999 998 2) Locate your 1 st order neighbors Name Hometown Me Name Hometown Name Hometown Name Hometown Solving Linear Algebraic Equa3ons Basic Concepts Here only real

### Linear Algebra & Geometry why is linear algebra useful in computer vision?

Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia

### Matrix & Linear Algebra

Matrix & Linear Algebra Jamie Monogan University of Georgia For more information: http://monogan.myweb.uga.edu/teaching/mm/ Jamie Monogan (UGA) Matrix & Linear Algebra 1 / 84 Vectors Vectors Vector: A

### EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten

### MTH 2032 SemesterII

MTH 202 SemesterII 2010-11 Linear Algebra Worked Examples Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2011 ii Contents Table of Contents

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### Linear Algebra March 16, 2019

Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented

### MATH 240 Spring, Chapter 1: Linear Equations and Matrices

MATH 240 Spring, 2006 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 8th Ed. Sections 1.1 1.6, 2.1 2.2, 3.2 3.8, 4.3 4.5, 5.1 5.3, 5.5, 6.1 6.5, 7.1 7.2, 7.4 DEFINITIONS Chapter 1: Linear

### 10-725/36-725: Convex Optimization Prerequisite Topics

10-725/36-725: Convex Optimization Prerequisite Topics February 3, 2015 This is meant to be a brief, informal refresher of some topics that will form building blocks in this course. The content of the

### MAC Module 12 Eigenvalues and Eigenvectors

MAC 23 Module 2 Eigenvalues and Eigenvectors Learning Objectives Upon completing this module, you should be able to:. Solve the eigenvalue problem by finding the eigenvalues and the corresponding eigenvectors

### POLI270 - Linear Algebra

POLI7 - Linear Algebra Septemer 8th Basics a x + a x +... + a n x n b () is the linear form where a, b are parameters and x n are variables. For a given equation such as x +x you only need a variable and

### Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination

Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

### STA141C: Big Data & High Performance Statistical Computing

STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018 Linear Algebra Background Vectors A vector has a direction and a magnitude