Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function Spaces Note: We will need many of these concepts as basic tools to quantify and evaluate the performance of machine learning algorithms and also to come up with more efficient and effective solutions Lecture II: LSC - Dr. Sethu Vijayakumar 1
Vectors Usually denoted by lower case, bold letters, e.g. x, y Operations : ultiplication by scalar (αx). Addition of vectors (x+y) x and y have to be of same dimensions. Linear combination. u = αx+βy (x and y have to be of same dimensions). Angle between vectors. < v, w > cosθ = v w Linear independence. When one vector cannot be written as a linear combination of other, then the vectors are said to be linearly independent. Lecture II: LSC - Dr. Sethu Vijayakumar 2
etric Definition 1 (etric/ Distance) Example 1 (Trivial etric) Example 2 (anhattan Distance) Lecture II: LSC - Dr. Sethu Vijayakumar 3
Vector Spaces Definition 2 (Vector Spaces) Definition 3 (Cauchy Series) Definition 4 (Completeness) Lecture II: LSC - Dr. Sethu Vijayakumar 4
Examples of Vector Spaces Rational Numbers, Real Numbers, Polynomials are all Vector Spaces Lecture II: LSC - Dr. Sethu Vijayakumar 5
Norm & Banach Spaces Definition 5 (Norm / Length) Definition 6 (Banach Space) Lecture II: LSC - Dr. Sethu Vijayakumar 6
Examples of Banach Spaces Lecture II: LSC - Dr. Sethu Vijayakumar 7
Dot Products & Hilbert Spaces Definition 7 (Dot Product/ Inner Product) Example : v 3 = ; v = v, v 4 1 2 = 3 2 + 4 2 = 5 Definition 8 (Hilbert Space) Lecture II: LSC - Dr. Sethu Vijayakumar 8
Examples of Hilbert Spaces Lecture II: LSC - Dr. Sethu Vijayakumar 9
atrices v1 v = v m Review: Domain m R u =v Range n R u = u 1 u n Addition of atrices ultiplication of matrices by scalars, vectors and matrices. Domain and Range of a atrix Lecture II: LSC - Dr. Sethu Vijayakumar 10
Special matrices Square and Diagonal atrix A square matrix has equal number of rows and columns. A diagonal matrix has all off-diagonal elements zero. Symmetric atrix Anti-symmetric atrix Orthogonal atrix (Often denoted as O(m) ) Lecture II: LSC - Dr. Sethu Vijayakumar 11
atrix Concepts Rank of a atrix Range, Domain and Null Space Range of, denoted by R(), is the space of all vectors that can be obtained by the operation of on the vectors in the domain of denoted by D(). v R( ) iff u D( ) s. t. u = v Null Space of, denoted by N(), is a subspace of all the vectors in the domain of D() that map to the zero (null) vector in R() when operated upon by the matrix. v N( ) iff v = 0 Lecture II: LSC - Dr. Sethu Vijayakumar 12
atrix Invariants Trace Properties of Trace : tr( aa) = a tr( A) tr( A + B) = tr( A) + tr( B) tr( AB) = tr( BA) Determinant Determinant can be written as the product of the eigenvalues : Note: Trace and Determinant are invariant under orthogonal transformation Lecture II: LSC - Dr. Sethu Vijayakumar 13
atrix Norms Operator Norm Frobeius Norm Lecture II: LSC - Dr. Sethu Vijayakumar 14
Eigensystems Definition 9 (Eigenvalues/ Eigenvectors) IP: Defined only for square atrices Intuitive Explanation. A square matrix is a mapping from n-dim to n-dim space. ost vectors change both direction and length when undergoing this mapping transformation. Those vectors which only change length (i.e., multiplying them by a matrix is similar to multiplying by a scalar) are called eigenvectors and the eigenvalue indicates how much they are shortened or lengthened. Lecture II: LSC - Dr. Sethu Vijayakumar 15
Eigensystems II Eigenvectors/Eigenvalues of Symmetric atrices o o o All eigenvalues of symmetric matrices are real Symmetric matrices are fully diagonalizable, i.e. we can find m eigenvectors All eigenvectors of symmetric matrices with different eigenvalues are mutually orthogonal (Prove!!) Decomposition of Symmetric atrices Example Lecture II: LSC - Dr. Sethu Vijayakumar 16
Positive atrices Definition 10 (Positive Definite atrices) Induced Norm and etrics Lecture II: LSC - Dr. Sethu Vijayakumar 17
Lecture II: LSC - Dr. Sethu Vijayakumar 18 ahalanobis Distance y x x,y y x, T d = = ) ( = = 1 0 0 1 I = Euclidean Distance = 1 0 0 2 I = 1 1.4 1.4 2
Singular Value Decomposition Note: Eigenvalue/Eigenvector decompositions are valid only for square matrices Do we have some decomposition for rectangular matrices?? Singular value Decomposition (SVD) Lecture II: LSC - Dr. Sethu Vijayakumar 19
atrix Inverse v = v 1 v m Domain m R 1 Range n R u = u 1 u n 1 = I; 1 = Note: A regular inverse exists only for square matrices with linearly independent column vectors Interpretation: We need a one-to-one mapping to uniquely go from one element of a space to another and back. Square matrices and linearly independent columns ensure this!! I Lecture II: LSC - Dr. Sethu Vijayakumar 20
Pseudoinverse For rectangular matrices and square matrices with linearly dependent columns, there exists the pseudoinverse or generalized inverse which performs the inverse mapping. In general, these inverses are not unique. The oore-penrose Pseudoinverse ( + ) + = T ( ) 1 T or T ( The above generalized inverse is called the oore-penrose Psuedoinverse and is unique. Among the multiple inverse solutions, it chooses the one with the minimum norm. T ) 1 Lecture II: LSC - Dr. Sethu Vijayakumar 21
Operators Linear Operators Generalization of matrix a mapping from one Banach space to another. Norms and eigenvalues/eigenvectors are defined as for matrices; so are Range & Null Spaces. Notation A : F G denotes a linear operator A mapping from space F to space G. A atrix-operator Correspondence atrix Transpose T A Symmetric atrix A = A Orthogonal atrix T 1 A = A T Adjoint Operator Af, g Self Adjoint Operator Af, g Isometry = f, A = A * * g f, g for all f F, g G. for all f F, g G. f, g = Af, Ag for all f F, g G. Lecture II: LSC - Dr. Sethu Vijayakumar 22
Linear Operators - Examples Input transformation Sampling Sampling from a function to yield scalar outputs. ( We will see later why this is so!!! ) f Fourier Transform x1 x2 x3 Lecture II: LSC - Dr. Sethu Vijayakumar 23
Range and Null Space of Operators Recollect: Definition of Range, Domain and Null space of a matrix N(A) A N(A * ) R(A * ) R(A) A * Lecture II: LSC - Dr. Sethu Vijayakumar 24