Topics in Eigen-analysis

Transcription

1 Topics i Eige-aalysis Li Zajiag 28 July 2014 Cotets 1 Termiology Some Basic Properties ad Results Eige-properties of Hermitia Matrices Basic Theorems Quadratic Forms & Noegative Defiite Matrices Defiitios Eigevalue Properties of Noegative Defiite Matrices Iequalities ad Extremal Properties of Eigevalues The Rayleigh Quotiet ad the Courat-Fischer Mi-max Theorem Some Eigevalue Iequalities Applicatio to Pricipal Compoet Aalysis (PCA)

2 1 Termiology (1) c A (x) = det(xi A) the characteristic polyomial of A (2) The eigevalues λ of A = the roots of c A (x) (3) The λ eigevectors x = the ozero solutios to (λi A)x = 0 (4) The eigevalue-eigevector equatio: Ax = λx (5) S A (λ) = The eigespace of a matrix A correspodig to the eigevalue λ (6) The characteristic equatio: xi A = 0 (7) Stadard ier product: z, w = z 1 w z w z, w C 2 Some Basic Properties ad Results Theorem 2.1 (a) The eigevalues of A are the same as the eigevalues of A T (b) A is sigular if ad oly if at least oe eigevalue of A is equal to zero (c) The eigevalues ad correspodig geometric multiplicities of BAB 1 are the same as those of A, if B is a osigular matrix (d) The modulus of each eigevalue of A is equal to 1 if A is a orthogoal matrix Theorem 2.2(revisio) Suppose that λ is a eigevalue, with multiplicity r 1, of the matrix A. The 1 dim{s A (λ)} r If λ is a eigevalue of A, the by defiitio a x 0 satisfyig the equatio Ax = λx exists, ad so clearly dim {S A (λ)} 1. Now let k = dim {S A (λ)}, ad let x 1,, x k be liearly idepedet eigevectors correspodig to λ. Form a osigular matrix X that has these k vectors as its first k colums; that is, X has the form X = [X 1 X 2 ], where X 1 = (x 1,, x k ) ad X 2 is ( k). Sice each colum of X 1 is a eigevector of A correspodig to the eigevalue λ, it follows that AX 1 = λx 1, ad 2

3 X 1 X 1 = [ I k (0) ], which follows from the fact that X 1 X = I. As a result, we fid that X 1 AX = X 1 [AX 1 AX 2 ] = X 1 [λx 1 AX 2 ] = [ λi k B 1 (0) B 2 ], where B 1 ad B 2 are a partitio of the matrix X 1 AX 2. If u is a eigevalue of X 1 AX, the 0 = X 1 AX μi = (λ μ)i k B 1 (0) B 2 μi k = (λ μ) k B 2 μi k, Thus, λ must be a eigevalue of X 1 AX with multiplicity of at least k. The result follows because from Theorem 2.1(c), the eigevalues ad correspodig geometric multiplicities of X 1 AX are the same as those of A. Theorem 2.3 Let λ be a eigevalue of the matrix A, ad let x be a correspodig eigevector. The, (a) If k 1 is a iteger, λ k is a eigevalue of A k correspodig to the eigevector x. (b) If A is osigular, λ 1 is a eigevalue of A 1 correspodig to the eigevector x. (a) Let us prove by iductio. Clearly, (a) holds whe k = 1 because it follows from the defiitio of eigevalue ad eigevector. Next, if (a) holds for k 1, that is, A k 1 x = λ k 1 x, the A k x = A(A k 1 x) = A(λ k 1 x) = λ k 1 (Ax) = λ k 1 (λx) = λ k x (b) Let us premultiply the equatio Ax = λx by A 1, which gives the equatio x = λa 1 x Sice A is osigular, we kow from Theorem 2.1(b) that λ 0, ad so dividig 3

4 both sides of the above equatio by λ yields A 1 x = λ 1 x, which implies that A 1 has a eigevalue λ 1 ad correspodig eigevector x. Theorem 2.4 Let A be a matrix with eigevalues λ 1,, λ. The (a) tr(a) = (b) A = λ i λ i Express the characteristic equatio, xi A = 0 ito the polyomial form x + α 1 x α 1 x + α 0 = 0 To determie a 0, we ca substitute x = 0 ito the equatio, thus, α 0 = (0)I A = A. To determie α 1, we are actually goig to fid the coefficiet of x 1. Recall that the determiat is actually a sum of terms that are products of oe etry i each colum(row) with row(colum) positios spaig over all permutatios of the iteger (1, 2,, m) with proper +/ sigs, it ca be easily see that the oly term that ivolves at least 1 of the diagoal elemets of (xi A) is the term that ivolves the product of all the diagoal elemets. Now that this term ivolves a eve permutatio as there exists a trivial compositio of zero traspositios, the sig term should be +1, therefore, α 1 will be the coefficiet of x 1 i (x a 11 )(x a 22 ) (x a ), which is obviously tr(a). Now fially, ote that λ 1,, λ are the roots to the characteristic equatio, it follows that With coefficiet matchig, we fid that (x λ 1 )(x λ 2 ) (x λ ) = 0 which completes the proof. A = λ i, tr(a) = λ i 4

5 3 Eige-properties of Hermitia Matrices 3.1 Basic Theorems Theorem Let A M be a matrix, the A is hermitia if ad oly if For all vectors x, y C. Ax, y = x, Ay (1) If A is hermitia, the Ax, y = y T Ax = y T A T x = (Ay) T x = x, Ay, which proves the " " part. (2) For the coverse directio, let x, y take the form of e 1,, e respectively, which are the stadard basis of R, the it is immediately clear that i, j = 1,,, a ij = a ji A is hermitia. Theorem (The Schur s Theorem Revisited) Let A M be a matrix, there exists a uitary matrix U such that U H AU = T is upper triagular, which is called the Schur s Theorem. Actually, it is just the complex couterpart of the Triagulatio Theorem we leart i class. With a mior modificatio, we may write A = UTU H Ad this is called the Schur Decompositio, with the diagoal etries of T beig the eigevalues of A Theorem (The Spectral Theorem Revisited) If the A i the last theorem turs out to be hermitia, the the correspodig T will become diagoal. This is called the Spectral Theorem. Similarly, it is the complex extesio of the Pricipal Axis Theorem we leart i class. Agai, with the same modificatio, we may rewrite it as A = UTU H This is called the Spectral Decompositio. These two eigevalue decompositios are just special cases of the Sigular Value Decompositio which applies to osquare 5

6 matrices Theorem Let A M be a hermitia matrix, the (a) x H Ax is real x C. (b) All eigevalues of A are real (c) S H AS is hermitia for all S M. (d) Eigevectors of A correspodig to distict eigevalues are orthogoal. (e) It is possible to costruct a set of orthoormal eigevectors of A. (a) (x ) H Ax = (x H Ax) H = x H A H x = x H Ax, that is, x H Ax equals its complex cojugate ad hece is real. (b) (1) If Ax = λx ad x H x = k R +, the λ = λ k xh x = 1 k xh λx = 1 k (xh Ax ), which is real by (a). (2) Alterative proof for (b): let λ, μ be 2 eigevalues of A, havig eigevectors x, y correspodigly, it follows that Ax = λx ad Ay = μy, accordig to Theorem 3.1.1, we have λ x, y = λx, y = Ax, y = x, Ay = x, μy = μ x, y. I the case where μ = λ & y = x, it becomes λ x, x = λ x, x, which i tur implies that λ = λ sice we kow x, x = x 2 > 0 for a ozero eigevector x. Therefore, λ must be real. Ad similarly, μ is real. (c) (S H AS) H = S H A H S = S H AS, so S H AS is always hermitia. (d) Followig the discussio i (b)(2), fially we ca get the equatio λ x, y = μ x, y, so if λ μ, the it follows immediately that x, y = 0, thus implyig that x ad y are orthogoal. (e) Followig from the Spectral Decompositio, we rewrite it as AU = UT λ 1 0 A[x 1 x ] = [x 1 x ] [ 0 ] λ [Ax 1 Ax ] = [λx 1 λx ] Therefore, it ca be easily see that (x 1 x ) are the correspodig eigevectors to (λ 1 λ ). As U is uitary, it follows that these eigevectors are orthoormal. 3.2 Quadratic Forms & Noegative Defiite Matrices 6

7 3.2.1 Defiitios (a) Let A M be a symmetric matrix (hermitia matrix with real etries) ad x deote a 1 colum vector, the Q = x T Ax is said to be a quadratic form. Observe that a 11 a 1 x 1 Q = x T Ax = (x 1 x ) ( ) ( ) a 1 a x = (x 1 x ) a 1i x i a i x i) ( = a ij x i x j i,j (b) Let A M be a symmetric matrix, the A is (1) Positive defiite if x 0 & x R, Q = x T Ax > 0 (2) Noegative defiite (Positive semidefiite) if x R, Q = x T Ax 0 (3) Negative defiite if x 0 & x R, Q = x T Ax < 0 (4) Nopositive defiite (Negative semidefiite) if x R, Q = x T Ax 0 (5) Idefiite if Q > 0 for some x while Q < 0 for some other x We are oly iterested i positive or oegative cases because all theorems will be similar for egative or opositive cases Eigevalue Properties of Noegative Defiite Matrices Theorem Let λ 1,, λ be the eigevalues of the symmetric matrix A, the (a) A is positive defiite if ad oly if λ i > 0 for all i, (b) A is oegative defiite if ad oly if λ i 0 for all i. (a) Let the colums of U = (x 1 x ) be a set of orthoormal eigevectors of A correspodig to the eigevalues λ 1,, λ, so that A = UTU T, where T = diag ( λ 1,, λ ). If A is positive defiite, the x T Ax > 0 for all x 0, so i particular, choosig x = x i, we have x i T Ax i = x i T (λ i x i ) = λ i x i T x i = λ i > 0 7

8 Coversely, if λ i > 0 for all i, the for ay x 0 defie y = U T x, ad ote that x T Ax = x T UTU T x = y T Ty = y i 2 λ i has to be positive sice the λ i s are all positive ad at least oe of the y i 2 s is positive because y 0. (b) By similar argumet as i (a), it is easy to show that A is oegative defiite if ad oly if λ i 0 for all i Theorem Let T be a m real matrix with rak(t) = r. The (a) T T T has r positive eigevalues. It is always oegative defiite ad positive defiite if r =. (b) The positive eigevalues of T T T are equal to the positive eigevalues of TT T. (a) For ay ozero 1 vector x, let y = Tx, the x T T T Tx = y T y = y i 2 is oegative, so T T T is oegative defiite, ad thus by Theorem (b), all of its eigevalues are oegative. Further, observe that x 0 is a eigevector of T T T correspodig to a zero eigevalue if ad oly if y = Tx = 0 ad thus the above equatio equals zero. Therefore, the umber of zero eigevalues equals the dimesio of ull(t), which is r, so (a) is proved. (b) Let λ > 0 be a eigevalue of T T T with multiplicity h. Sice the matrix T T T is symmetric, we ca fid a h matrix X, whose colums are orthoormal, satisfyig T T TX = λx. Let Y = TX ad observe that TT T Y = TT T TX = T(λX) = λtx = λy, so that λ is also a eigevalue of TT T with multiplicity also beig h because rak(y) = rak(tx) = rak((tx) T TX) = rak(x T T T TX) = rak(λx T X) 8

9 = rak(λi h ) = h So the proof is doe. 4 Iequalities ad Extremal Properties of Eigevalues 4.1 The Rayleigh Quotiet ad the Courat-Fischer Mi-max Theorem I this sectio, we are goig to ivestigate some extremal properties of the eigevalues of a hermitia matrix, ad see how to tur the problem of fidig the eigevalues ito a costraied optimizatio problem Defiitio Let A M be a hermitia matrix, the the Rayleigh quotiet of A, deoted as R A (x), is a fuctio from C \{0} to R, defied as follows: R A (x) = xh Ax x H x It is ot difficult to see that whe the orm of x, x = 1, the Rayleigh quotiet of A actually equals its quadratic form. I the ext part, we are ready to relate the Rayleigh quotiet of a hermitia matrix to it eigevalues Theorem Let A be a hermitia matrix with ordered eigevalues λ 1 λ 2 λ. For ay x C \{0} Ad, i particular λ xh Ax x H x λ 1 x H Ax λ = mi x 0 x H x λ 1 = max x 0 x H Ax x H x 9

10 Let A = UTU H be the spectral decompositio of A, where the colums of U = (x 1 x ) are the orthoormal set of eigevectors correspodig to λ 1,, λ, which make up the diagoal etries of the diagoal matrix T. As i the proof of Theorem , defie y = U H x, the we have x H Ax x H x = xh UTU H x x H UU H x = yh Ty y H y = λ 2 iy i y 2 i Together with the fact that λ y i λ i y i λ 1 y i The proof is complete. I fact, we ca see that the implicatio of this theorem is that we may regard the problem of fidig the largest ad smallest eigevalues of a hermitia matrix as a costraied optimizatio problem: maximize: x H Ax subject to: x H x = 1 Below is a theorem that geeralizes the above theorem to all eigevalues of A Theorem (the Courat-Fischer mi-max theorem) Let A be a hermitia matrix, the (a) λ i = max dim(v)=i mi x H Ax (b) λ i = mi dim(v)= i+1 max x H Ax (a) Recall that x H Ax = y H 2 Ty = λ i y i, where y = U H x, or equivaletly, x = Uy, ad also ote that the liear trasformatio from x to y is a isomorphism ad there is o scalig as U is uitary, so we may chage the costrait to dim(w) = i, y W, y = 1. Ad i order to get the maximum uder the costrait dim(w) = i, we just choose W = spa{e 1,, e i }. Therefore, it is easily verified that 10

11 λ i mi λ 2 jy j y spa{e 1,,e i } i j=1 = max dim(v)=i = max dim(w)=i mi x H Ax mi y W, y =1 y H Ty Now it remais to prove that the left-had side of the above equatio is greater tha or equal to the right-had side to fially get the equality. To prove it, we must show that every i dimesioal subspace V of C cotais a uit vector x such that λ i x H Ax Ad from previous discussio, we kow that it is equivalet to say that every i dimesioal subspace W of C cotais a uit vector y such that λ i y H Ay Now let Ω = spa{e i,, e } with dimesio i + 1, so it must have oempty itersectio with every W. The let w be a uit vector i Ω W, we may write with w H Tw = λ j w j 2 j=i w j = 1 j=i Thus it is immediately clear that the reverse iequality is proved. So fially we achieve the equality. (b) This ca be proved simply by replacig A by A ad usig the fact that λ i ( A) = λ i+1 (A). 4.2 Some Eigevalue Iequalities I this sectio we are goig to itroduce a few iequalities cocerig eigevalues, which may be applied i eigevalue estimatio ad ifereces ad also eigevalue perturbatio theories. It turs out that may of these iequalities ca be proved usig the mi-max theorem derived i the last sectio. 11

12 4.2.1 Theorem (Weyl s iequality) Let A, B M be hermitia matrices, the for 1 i, we have First, we have λ i (A) + λ (B) λ i (A + B) λ i (A) + λ 1 (B) λ i (A + B) = max dim(v)=i mi x H (A + B)x = max dim(v)=i mi ( x H Ax + x H Bx) max dim(v)=i mi x H Ax + mi x H Bx = max dim(v)=i mi x H Ax + λ (B) = λ i (A) + λ (B) Which proves the left iequality. The right iequality ca be proved i exactly the same maer Corollary Let A M be a hermitia matrix, B M be a positive semidefiite matrix, the for 1 i, we have λ i (A) λ i (A + B) Theorem Let A, B M be hermitia matrices, if 1 j 1 j k, the k λ jl (A + B) λ jl (A) + λ jl (B) l=1 k l= Theorem (Cauchy Iterlacig Theorem) Let A M be a hermitia matrix with eigevalues λ 1 λ 2 λ, ad partitioed as follows: A = [ H B B R ] k l=1 where H M m m with eigevalues θ 1 θ 2 θ m, the λ k θ k λ k+ m 12

13 4.3 Applicatio to Pricipal Compoet Aalysis (PCA) Pricipal compoet aalysis (PCA) is a techique that is useful for the compressio ad classificatio of data. The purpose is to reduce the dimesioality of a data set (sample) by fidig a ew set of variables, smaller tha the origial set of variables that oetheless retais most of the sample's iformatio Defiitio Let X = (x 1, x 2,, x p ) be a p data matrix with p beig the umber of variables ad beig the umber of observatios for each variable. Defie the first pricipal compoet of the sample by the liear trasformatio p z 1 = Xa 1 = a i1 x i where the vector a 1 = (a 11, a 21,, a p1 ) T is chose such that var[z 1 ] is maximal. Likewise, the k th pricipal compoet is defied as the liear trasformatio p z k = Xa k = a i1 x i, k = 1,, p where the vector a k = (a 1k, a 2k,, a pk ) T is chose such that var[z k ] is maximal subject to cov[z k, z l ] = 0 a k T a l = 0, k > l 1 ad to a k T a k = 1. After some computatio, we get var[z k ] = a k T Sa k, where S = 1 1 XT X is the covariace matrix of the data matrix X. It is clear that S is oegative defiite, thus havig oegative eigevalues. If we wat to fid k(k < p) pricipal compoets, the we get Z = [z 1 z k ] = X[a 1 a k ] = XA The matrix Z is called the score matrix while A is called the loadig matrix Methods for implemetig PCA 13

14 (a) Costraied optimizatio To maximize a 1 T Sa 1 subject to a 1 T a 1 = 1, we use the techique of Lagrage multipliers. We wat to maximize the fuctio by differetiatig w. r. t. a 1 : a 1 T Sa 1 μ(a 1 T a 1 1) w. r. t. a 1 d da 1 (a 1 T Sa 1 μ(a 1 T a 1 1)) = 0 Sa 1 μa 1 = 0 Sa 1 = μa 1 From this step, it is obvious that μ is a eigevalue of S(ad of course, we ca proceed to this step without usig the Lagrage multiplier, but istead usig the mi-max theorem), so to maximize a 1 T Sa 1, certaily we are goig to choose the largest eigevalue λ 1 of S. Ad the set a 1 to be its correspodig eigevector, the we get our first pricipal compoet z 1 = a 1 T X, which fiishes our first step. To fid the pricipal compoet, ote that a 1,, a p costitute a orthoormal basis, so to satisfy the zero-covariace costrait, we have to do the optimizatio i the orthogoal complemet of spa{a 1,, a k 1 }, thus accordig to the mi-max theorem, max a k T Sa k = λ k, z k = a k T X, where a k is the uit eigevector correspodig to λ k. Fially, we coclude that Z = [z 1 z k ] = X[a 1 a k ]. (2) Spectral decompositio ad sigular value decompositio (SVD) First, we draw the coclusio from (1) that if we write the spectral decompositio of S as S = ATA T where T = diag( λ 1, λ 2,, λ ) with λ 1 λ 2 λ, the the first k colums of A just makes up the loadig matrix that we wat. Next, let X = UΣV T be the sigular value decompositio of the data matrix X, recall that X T X = S = VΣ 2 V T, so the colums of V are the uit eigevectors of S, the let A = V, we fially get Z = XA = XV = UΣV T V = UΣ. I practice, the sigular value decompositio is the stadard way to do PCA sice it avoids the trouble of computig X T X. 14