11 a 12 a 21 a 11 a 22 a 12 a 21. (C.11) A = The determinant of a product of two matrices is given by AB = A B 1 1 = (C.13) and similarly.

Transcription

1 C PROPERTIES OF MATRICES 697 to whether the permutation i 1 i 2 i N is even or odd, respectively Note that I =1 Thus, for a 2 2 matrix, the determinant takes the form A = a 11 a 12 = a a 21 a 11 a 22 a 12 a 21 (C11) 22 The determinant of a product of two matrices is given by AB = A B (C12) as can be shown from (C10) Also, the determinant of an inverse matrix is given by A 1 1 = (C13) A which can be shown by taking the determinant of (C2) and applying (C12) If A and B are matrices of size N M, then IN + AB T = IM + A T B (C14) A useful special case is IN + ab T =1+a T b (C15) where a and b are N-dimensional column vectors Matrix Derivatives Sometimes we need to consider derivatives of vectors and matrices with respect to scalars The derivative of a vector a with respect to a scalar x is itself a vector whose components are given by ( ) a = a i (C16) x x i with an analogous definition for the derivative of a matrix Derivatives with respect to vectors and matrices can also be defined, for instance ( ) x = x (C17) a a i and similarly ( ) a = a i b b ij j The following is easily proven by writing out the components i (C18) ( x T a ) = ( a T x ) = a (C19) x x

2 698 C PROPERTIES OF MATRICES Similarly (AB) =A x x B + AB x The derivative of the inverse of a matrix can be expressed as ( ) A 1 1 A = A x x A 1 (C20) (C21) as can be shown by differentiating the equation A 1 A = I using (C20) and then right multiplying by A 1 Also ( ) 1 A ln A = Tr A (C22) x x which we shall prove later If we choose x to be one of the elements of A,wehave A ij Tr (AB) =B ji (C23) as can be seen by writing out the matrices using index notation We can write this result more compactly in the form A Tr (AB) =BT With this notation, we have the following properties (C24) A Tr ( A T B ) = B (C25) Tr(A) = I (C26) A A Tr(ABAT ) = A(B + B T ) (C27) which can again be proven by writing out the matrix indices We also have A ln A = ( A 1) T which follows from (C22) and (C26) (C28) Eigenvector Equation For a square matrix A of size M M, the eigenvector equation is defined by Au i = λ i u i (C29)

3 8 APPENDIX A MATHEMATICAL FOUNDATIONS M = xy t = x 1 x 2 x d (y 1 y 2 y n ) = x 1 y 1 x 1 y 2 x 1 y n x 2 y 1 x 2 y 2 x 2 y n x d y 1 x d y 2 x d y n, (11) that is, the components of M are m ij = x i y j Of course, if the dimensions of x and y are not the same, then M is not square A24 Derivatives of matrices Suppose f(x) is a scalar-valued function of d variables x i, i =1, 2, d, which we represent as the vector x Then the derivative or gradient of f with respect to this vector is computed component by component, ie, f(x) = gradf(x) = f(x) x = f(x) x 1 f(x) x 2 f(x) x d (12) Jacobian matrix If we have an n-dimensional vector-valued function f (note the use of boldface), of a d-dimensional vector x, we calculate the derivatives and represent them as the Jacobian matrix J(x) = f(x) x = f 1(x) f 1(x) x d x 1 f n(x) x 1 f n(x) x d (13) If this matrix is square, its determinant (Sect A25) is called simply the Jacobian or occassionally the Jacobian determinant If the entries of M depend upon a scalar parameter θ, we can take the derivative of M component by component, to get another matrix, as M θ = m 11 θ m 21 θ m n1 θ m 12 m 1d θ m 2d θ θ m 22 θ m n2 θ m nd θ (14) In Sect A26 we shall discuss matrix inversion, but for convenience we give here the derivative of the inverse of a matrix, M 1 : θ M 1 = M 1 M θ M 1 (15)

4 A2 LINEAR ALGEBRA 9 Consider a matrix M that is independent of x The following vector derivative identities can be verified by writing out the components: [Mx] = M (16) x x [yt x] = x [xt y]=y (17) x [xt Mx] = [M + M t ]x (18) In the case where M is symmetric (as for instance a covariance matrix, cf Sect A410), then Eq 18 simplifies to x [xt Mx] =2Mx (19) We first recall the use of second derivatives of a scalar function of a scalar x in writing a Taylor series (or Taylor expansion) about a point: f(x) =f(x 0 )+ df (x) dx (x x 0 )+ 1 2! x=x0 d 2 f(x) dx 2 (x x 0 ) 2 + O((x x 0 ) 3 ) (20) x=x0 Analogously, if our scalar-valued f is a instead function of a vector x, we can expand f(x) in a Taylor series around a point x 0 : [ ] t [ f f(x) =f(x 0 )+ (x x 0 )+ 1 }{{} x 2! (x x 0) t x=x 0 J 2 f x 2 }{{} H ] t (x x 0 )+O( x x 0 3 ), (21) x=x 0 where H is the Hessian matrix, the matrix of second-order derivatives of f( ), here evaluated at x 0 (We shall return in Sect A8 to consider the O( ) notation and the order of a function used in Eq 21 and below) A25 Determinant and trace The determinant of a d d (square) matrix is a scalar, denoted M, and reveals properties of the matrix For instance, if we consider the columns of M as vectors, if these vectors are not linearly independent, then the determinant vanishes In pattern recognition, we have particular interest in the covariance matrix Σ, which contains the second moments of a sample of data In this case the absolute value of the determinant of a covariance matrix is a measure of the d-dimensional hypervolume of the data that yielded Σ (It can be shown that the determinant is equal to the product of the eigenvalues of a matrix, as mentioned in Sec A27) If the data lies in a subspace of the full d-dimensional space, then the columns of Σ are not linearly independent, and the determinant vanishes Further, the determinant must be non-zero for the inverse of a matrix to exist (Sec A26) The calculation of the determinant is simple in low dimensions, and a bit more involved in high dimensions If M is itself a scalar (ie, a 1 1 matrix M), then M = M IfM is 2 2, then M = m 11 m 22 m 21 m 12 The determinant of a general square matrix can be computed by a method called expansion by minors, and this Hessian matrix expansion by minors

5 440 Linear algebra The singular values of A are the square roots of the non-zero eigenvalues of AA T or A T AThepseudo-inverse or generalised inverse is the n ð m matrix A A D V 1 U T D rx id1 and the solution for x that minimises the squared error jjax bjj 2 1 i v i u T i (C1) is given by x D A b If the rank of A is less than n, then there is not a unique solution for x and singular value decomposition delivers the solution with minimum norm The pseudo-inverse has the following properties: AA A D A A AA D A AA / T D AA A A/ T D A A Finally, in this section, we introduce some results about derivatives We shall denote the partial derivative @x T p Thus, the derivative of the scalar function f of the vector x is the D f p Similarly, the derivative of a scalar function of a matrix is denoted by the f=@a, where f ij In particular, D adjoint of A/T DjAjA 1 / T if A 1 exists and for a xt Ax D 2Ax Also, an important derivative involving traces of TrfAT MAg/ D MA C M T A