10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang)

Transcription

1 10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang) Manojit Nandi February 1, 2014

2 Outline Linear Algebra General Properties Matrix Operations Inner Products and Orthogonal Matrices Eigenvalue and Eigenvectors Matrix Calculus Gradients Hessians Derivatives References

3 Linear Algebra Manojit Nandi Carnegie Mellon University Machine Learning Department 3/28

4 General Properties A Vector Space V is a space X defined over some field F that satisfies the following properties. For all x, y V and for all α R 1. x + y V (Closure under addition) 2. αx V (Closure under scalar multiplication) Manojit Nandi Carnegie Mellon University Machine Learning Department 4/28

5 A vector space V defined over the field R mxn is an mxn matrix with m rows and n columns. Examples: 3 x 3 matrix: 2 x 3 matrix: ( ) Manojit Nandi Carnegie Mellon University Machine Learning Department 5/28

6 Matrix Addition: If A is a mxn matrix, and B is a mxn matrix, then C = A + B is a mxn matrix s.t., C ij = A ij + B ij Matrix Multiplication: If A is a mxn matrix, and B is a nxp matrix, then C = AB is a mxp matrix s.t., C ij = n A ik B kp k=1 Manojit Nandi Carnegie Mellon University Machine Learning Department 6/28

7 Matrix Multiplication has the following properties: Associativity: (AB)C = A(BC) Distributive: A(B + C) = AB + AC Matrix multiplication is not commutative. AB BA, except in certain cases. Manojit Nandi Carnegie Mellon University Machine Learning Department 7/28

8 Matrix Operations The transpose of a matrix A R mxn, is A T R nxm where the entries of A T are given by: (A T ) ij = A ji Properties of the transpose operator: (A + B) T = B T + A T = A T + B T (AB) T = B T A T (A T ) T = A Manojit Nandi Carnegie Mellon University Machine Learning Department 8/28

9 A set of vectors x 1, x 2,...x n R m are linearly independent if no vector can be represented as a linear combination of the remaining vectors. The rank of a matrix is the cardinality of the largest subset of the columns of some matrix A that is a linearly independent set. A matrix A R mxn is full rank, if rank(a) = min(m, n). Manojit Nandi Carnegie Mellon University Machine Learning Department 9/28

10 A norm of a vector x reflects the magnitude of the vector. Commonly used norms fall into a family of norms called the L p norm: x p = ( n i=1 x i p ) 1 p Examples: Eucldiean norm (L 2 ): x 2 = ( n i=1 x i 2 ) 1 2 L 1 norm: x 1 = ( n i=1 x i 1 ) 1 1 = n i=1 x i L 0 norm: x 0 = n i=1 1(x i 0) L norm: x = max i x i Manojit Nandi Carnegie Mellon University Machine Learning Department 10/28

11 Suppose we have a vector x = (x 1, x 2,..., x n ), and we want to represent it using a single point s = (s, s,..., s) such that s R. Therefore we want to choose the s that minimizes x s. However, which we chooses effects the value of s. Eucldiean norm (L 2 ): x s 2 = ( n i=1 x i s 2 ) 1 2 L 1 norm: x s 1 = ( n i=1 x i 1 ) 1 1 = n i=1 x i s L 0 norm: x s 0 = n i=1 1(x i s 0) Manojit Nandi Carnegie Mellon University Machine Learning Department 11/28

12 Suppose we have a vector x = (x 1, x 2,..., x n ), and we want to represent it using a single point s = (s, s,..., s) such that s R. Therefore we want to choose the s that minimizes x s. However, which we chooses effects the value of s. Eucldiean norm (L 2 ): x s 2 = ( n i=1 x i s 2 ) 1 2 = Mean L 1 norm: x s 1 = n i=1 x i s = Median L 0 norm: x s 0 = n i=1 1(x i s 0) = Mode A vector is said to be normalized if x = 1. Manojit Nandi Carnegie Mellon University Machine Learning Department 12/28

13 The trace of a square matrix, Tr(A) A R nxn is the sum of the diagonal elements. n T r(a) = i=1 A ii Properties of the Trace: A R nxn, T r(a) = T r(a T ) A, B R nxn, T r(a + B) = T r(a) + T r(b) A R nxn, c R, T r(ca) = ct r(a) A, B R nxn, T r(a T B) = T r(ab T ) A, B, C R nxn, T r(abc) = T r(bca) = T r(cba) Manojit Nandi Carnegie Mellon University Machine Learning Department 13/28

14 The inverse of a square matrix A R nxn, written A 1 is defined such that: AA 1 = A 1 A = I If A 1 exists,t he matrix is said to be nonsingular, otherwise it is singular. Properties of the Inverse: (A 1 ) 1 = A (AB) 1 = B 1 A 1 (A 1 ) T = (A T ) 1 Manojit Nandi Carnegie Mellon University Machine Learning Department 14/28

15 Inner Products and Orthogonal Matrices An Inner Product Space is a vector space V equipped with a operation called the inner product, denoted <, >, that maps <, >: V xv F. For x, y, z V (when V is defined over the real numbers) and a constant a, b R, the inner product satisfies the following properties: < x, y >=< y, x > [Symmetry] < ax + bz, y >= a < x, y > +b < z, y > [Linearity] < x, x > 0 [Positive Semi-Definite] Manojit Nandi Carnegie Mellon University Machine Learning Department 15/28

16 For the vector space V defined over R n, given two vectors x, y R n, the inner product is defined as: < x, y >= n i=1 x iy i = x T y. As a result, < x, x >= n x i x i = i=1 n x 2 i = x 2 i=1 Manojit Nandi Carnegie Mellon University Machine Learning Department 16/28

17 Two vectors x, y are said to orthogonal if x T y = 0. A square matrix A R nxn is orthogonal if all its normalized colums are orthogonal to one another. If U is an orthogonal matrix, U T = U 1, then U T U = I. If U is a square matrix, then U T U = UU T = I Manojit Nandi Carnegie Mellon University Machine Learning Department 17/28

18 Eigenvalue and Eigenvectors Given a square matrix A R nxn, λ C, x C, λ and x are the eigenvalue and eigenvector of A respectively iff Ax = λx, x 0 (X λi)x = 0 We often use the determinant to expand this expression into the characteristic polynomial in terms of λ and then find the roots of the characteristic polynomial to find the eigenvalues. Manojit Nandi Carnegie Mellon University Machine Learning Department 18/28

19 Now that we have eigenvalues and eigenvectors, some important properties of matrices relating to eigenvalues are: T r(a) = n i=1 λ i det(a) = n i=1 λ i Rank(A) = number of non-zero eigenvalues of A Manojit Nandi Carnegie Mellon University Machine Learning Department 19/28

20 A square matrix A is diagonalizable if there exists an invertible matrix X and a diagonal matrix Λ such that A = XΛX 1. When the columns of X are the eigenvectors of A, and the eigenvalues of A form the diagonal entries of Λ, this corresponds to the eigendecomposition. One nice property of this is: A = XΛX 1 A 2 = (XΛX 1 )(XΛX 1 ) = XΛIΛX 1 = XΛ 2 X 1 Furthermore, the eigenvectors of A are orthonormal, so X is orthonormal, meaning X 1 = X T. Therefore: A = XΛX T Manojit Nandi Carnegie Mellon University Machine Learning Department 20/28

21 Matrix Calculus Manojit Nandi Carnegie Mellon University Machine Learning Department 21/28

22 Gradients Let f : R mxn R, the gradient of f with respect to the matrix A R mxn is an mxn matrix of the partial derivatives of f with respect to A. f(a) A 11 f(a) ( A f(a)) = A 21 f(a) A 31 f(a) A 12 f(a) A 22 f(a) A 32 f(a) A 13 f(a) A 23 f(a) A 33 Manojit Nandi Carnegie Mellon University Machine Learning Department 22/28

23 As a collorary, when we have a vector x rather than a matrix A, the gradient of f with respect to the vector is: ( x f(x)) = ( f x 1,..., f x n ) Example, if f(x, z) = x sin(z)e x, then the gradient of f with respect to the standard basis vectors x = (x, z) is ( x f(x)) = ( f x, f z ) = ((1 x) sin(z)e x, xe x cos(z)) For vectors, the gradient represents the direction of steepest ascent. Manojit Nandi Carnegie Mellon University Machine Learning Department 23/28

24 Hessians Let f : R n R (Note: This function takes is vector inputs, not matrix inputs). The Hessian of f with respect to a vector x R n is a nxn matrix of partial derivatives. ( 2 xf(x)) ij = 2 f x i x j ( x f(x)) = f(x) 2 x 1 x 1 2 f(x) x 2 x 1 2 f(x) x 3 x 1 2 f(x) x 1 x 2 2 f(x) x 2 x 2 2 f(x) x 3 x 2 2 f(x) x 1 x 3 2 f(x) x 2 x 3 2 f(x) x 3 x 3 The geometric interpretation of the Hessian is the curvature of a surface. Manojit Nandi Carnegie Mellon University Machine Learning Department 24/28

25 Derivatives Let x R m and f : R m R n, and y = f(x). The derivative of y with respect to the vector x is an mxn matrix: dy dx = dy 1 dy 2 dy n... dx 1 dx 1 dx 1 dy 1 dy 2 dy n... dx 2 dx 2 dx dy 1 dy 2 dy n... dx m dx m dx m Manojit Nandi Carnegie Mellon University Machine Learning Department 25/28

26 Let a be some arbitrary vector. Some common matrix derivatives are: (at x) x (xt Ax) x (at Xb) X = a = Ax xt x + xt A x x = (A + AT )x = ab T Manojit Nandi Carnegie Mellon University Machine Learning Department 26/28

27 References Zico Kolter s Linear Algebra Reiw and Reference: htpp://cs229.stanford.edu/section/cs229-linalg.pdf The Matrix Cookbook: hwolkowi/matrixcookbook.pdf Randall J. Barne s Matrix Differentiation Notes: dennis/matrixcalculus.pdf Manojit Nandi Carnegie Mellon University Machine Learning Department 27/28

28 Questions?