More Linear Algebra. Edps/Soc 584, Psych 594. Carolyn J. Anderson

Transcription

1 More Linear Algebra Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University of Illinois Spring 2017

2 Overview Eigensystems: decomposition of square matrix Singular Value Decompositions: decomposition of rectangular matrix Maximization: Reading: Johnson & Wichern pages 60 66, 73 75, C.J. Anderson (Illinois) More Linear Algebra Spring / 40

3 Eigensystems Let A be a p p square matrix, then the scalars λ 1,λ 2,...,λ p that satisfy the polynomial equation A λi = 0 are called eigenvalues (or characteristic roots ) of matrix A. The equation A λi = 0 is called the characteristic equation. Example: A = A λi = ( ) (1 λ) 5 5 (1 λ) = 0 (1 λ) 2 ( 5)( 5)) = 0 λ 2 2λ 24 = 0 (λ 6)(λ+4) = 0 λ 1 = 6 and λ 2 = 4 Quadratic Formula: ax 2 +bx +c = 0 ( b ± b 2 4ac)/(2a) C.J. Anderson (Illinois) More Linear Algebra Spring / 40

4 Eigenvectors A square matrix A is said to have eigenvalues λ with a corresponding eigenvector x 0 if Ax = λx or (A λi)x = 0 We usually normalize x so that it has length = 1. e = x Lx = x x x e is also an eigenvector of A because Ae = λe A(Lxe) = λ(lxe) Ax = λx so e e = 1 Any multiple of x is an eigenvector associated with λ. All that matters is the direction and not the length of x. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

5 Example: Eigenvectors continued ( ) 1 5 A = 5 1 ( )( ) 1 5 x1 = λ 5 1 x 2 ( x1 x 2 ) x 1 5x 2 = λx 1 5x 1 +x 2 = λx 2 So we have 2 equations and 3 unknowns (x 1, x 2 and λ). Set λ = 6, now there are 2 equations with 2 unknowns: x 1 5x 2 = 6x 1 5x 1 +x 2 = 6x 2 x = e = ( ) 1/ 2 1/ 2 C.J. Anderson (Illinois) More Linear Algebra Spring / 40

6 Symmetric Matrix Now A is (p p) symmetric Let A (p p) be a symmetric matrix. Then A has p pairs of eigenvalues and eigenvectors λ 1,e 1 ; λ 2,e 2 ; ; λ p,e p. The eigenvectors are chosen to have length= 1: e 1 e 1 = e 2 e 2 = = e p e p = 1. The eigenvectors are also chosen to be mutually orthogonal (perpendicular): e i e k that is e ie k = 0 for all i k The eigenvectors are all unique if no 2 eigenvalues are equal. Typically the eigenvalues are ordered from largest to smallest C.J. Anderson (Illinois) More Linear Algebra Spring / 40

7 Little Example continued A = ( ) and λ 1 = 6 λ 2 = 4 e 1 = ( 1/ 2 1/ 2 ) e 2 = ( 1/ 2 1/ 2 ) Note that e 1 e 2 = 0 and Le 1 = Le 2 = 1. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

8 Spectral Decomposition of A The Spectral Decomposition of A where A (p p) symmetric. A = λ 1 e 1 e 1+λ }{{} 2 e 2 e 2+ +λ }{{} p e k e k }{{} p p p p p p If A is also positive definite, then k = p. Matrix A is decomposed into p (p p) component matrices. where e i e i = 1 for all i, and e i e j = 0 for all i j. ( ) ( ) 1 5 λ A = 1 = 6 1/ 2 e 5 1 λ 2 = 4 1 = 1/ e 2 = ( ) ( 2 ) λ 1 e 1 e 1 +λ 2e 2 e 1/2 1/2 1/2 1/2 2 = 6 4 1/2 1/2 1/2 1/2 ( ) 1 5 = = A 5 1 ( 1/ 2 1/ 2 ) C.J. Anderson (Illinois) More Linear Algebra Spring / 40

9 A Bigger Example A = e 1 = e 2 = λ 1 = λ 2 = 9,λ 3 = e 3 = Note that since λ 1 = λ 2 the labeling of e 1 and e 2 is arbitrary. The lengths: e 1 e 1 = e 2 e 2 = e 3 e 3 = 1. Orthogonality: e 1 e 2 = e 1 e 3 = e 2 e 3 = 0. Decomposition: A = 9e 1 e 1 +9e 2 e 2 +18e 3 e 3 C.J. Anderson (Illinois) More Linear Algebra Spring / 40

10 Decomposition of (3 3) A = ) 1 ( , 1 2, ( = = ) = ( ) C.J. Anderson (Illinois) More Linear Algebra Spring / 40

11 Recall: Quadratic Form is defined as x Ax for x p and A p p symmetric The terms of x Ax are squares of x i (i.e., xi 2 ) and cross-products of x i and x k (i.e., x i x k ): p p x Ax = a ik x i x k e.g., ( a11 a (x 1,x 2 ) 12 a 21 a 22 i=1 k=1 )( x1 x 2 = ((a 11 x 1 +a 21 x 2 ),(a 12 x 1 +a 22 x 2 )) ) ( x1 = a 11 x 2 1 +a 21x 1 x 2 +a 12 x 1 x 2 +a 22 x 2 2 = 2 C.J. Anderson (Illinois) More Linear Algebra i=1 k=1 Spring / 40 x 2 ) 2 a ik x i x k

12 Eigenvalues and Definiteness IF x Ax > 0 for all x, matrix A is positive definite. IF x A x 0 for all x, matrix A is non-negative definite. Important: All eigenvalues of A > 0 A is positive definite. All eigenvalues of A 0 A is non-negative definite Implication: If A is positive definite, then the diagonal elements of A must be positive. If x = (0,..., }{{} 1,...0) then x Ax = a ii xi 2 > 0 i th position C.J. Anderson (Illinois) More Linear Algebra Spring / 40

13 More on Spectral Decomposition When A p p symmetric and positive definite, (i.e., diagonals of A are all > 0, and λ i > 0 for all i). We can write the spectral decomposition of A as the sum of the weighted vector products, p A p p = λ i e i e i i=1 In matrix form this is A = PΛP where λ λ 2 0 Λ p p = diag(λ i ) = λ p and P p p = (e 1,e 2,,e p ). C.J. Anderson (Illinois) More Linear Algebra Spring / 40

14 Showing that A = PΛP A p p = P p p Λ p p P p p λ λ 2 0 = (e 1,e 2,,e p ) λ p e 1 e 2 = (λ 1 e 1,λ 2 e 2,,λ p e p ). e p p = λ i e i e i i=1 e 1 e 2. e p C.J. Anderson (Illinois) More Linear Algebra Spring / 40

15 More about P Since The lengths of e i equal 1 (i.e., e i e i = 1), and e i and e k are orthogonal for all i k (i.e., e i e k = 0). e P e P = 2. (e ,e 2,,e p ) = e p = I = PP P is an orthogonal matrix. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

16 Rank r decompositions If A is non-negative definite (semi-definite): So λ i > 0 for i = 1,...,r < p λ i = 0 for i = r +1,...,p A p p = P p r Λ r r P r p. If A is positive or positive semi-definite, we sometimes want to approximate A by a rank r decomposition, where r < Rank of A, B = λ 1 e 1 e λ r e r e r This decomposition minimized the loss function p p (a ik b ik ) 2 = λ 2 r+1 +λ 2 r+2 + λ 2 p i=1 k=1 C.J. Anderson (Illinois) More Linear Algebra Spring / 40

17 Inverse of A If A is positive definite, the inverse of A equals where Why: A 1 = PΛ 1 P 1/λ ( ) 1 0 1/λ 2 0 diag = λ i /λ p AA 1 = (PΛ P ) (P Λ 1 P ) = PΛΛ }{{}}{{ 1 } P = PP = I I I What does A 1 A equal? C.J. Anderson (Illinois) More Linear Algebra Spring / 40

18 Square Root Matrix If A is symmetric, the Square Root Matrix of A is A 1/2 = p λi e i e i = PΛ 1/2 P i=1 Common mistake: A 1/2 = { a ij }. Properties of A 1/2 : (A 1/2 ) = A 1/2...since A 1/2 is symmetric. A 1/2 A 1/2 = A (A 1/2 ) 1 = p i=1 (1/ λ i )e i e i = PΛ 1/2 P = A 1/2 A 1/2 A 1/2 = A 1/2 A 1/2 = I A 1/2 A 1/2 = A 1 C.J. Anderson (Illinois) More Linear Algebra Spring / 40

19 Determinant, Trace and Eigenvalues A = p λ i = λ 1 λ 2 λ p. i=1 Implication: A positive definite matrix has A > 0, because λ 1 > λ 2 > > λ p > 0 p a ii = trace(a) = i=1 p i=1 Now let s consider what s true for Σ and S. λ i C.J. Anderson (Illinois) More Linear Algebra Spring / 40

20 Numerical Example We ll use the psychological test data from Rencher (2002) who got it from Beall (1945) to illustrate these properties 32 males and 32 females had measures on four psychological tests. The tests were x 1 = pictorial inconsistencies x 2 = paper form board x 3 = tool recognition x 4 = vocabulary S = Note that the total sample variance = trace(s) = and that the generalize sample variance = det(s) = C.J. Anderson (Illinois) More Linear Algebra Spring / 40

21 Numerical Example continued Eigenvalue of S are Λ = and the eigenvectors are P = Note that (for example) = (e 1, e 2, e 3, e 4 ) e 1 e 1 = ( ) = 1 = L 2 e 1 = Le 1. e 1 e 2 = (.274(.002)+.284(.185)+.856(.409)+.333(.894)) = 0. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

22 Example: eigenvalues of S Sum of eigenvalues: λ 1 +λ 2 +λ 3 +λ 4 = = = trace(s) = Total sample variance Product of the eigenvalues: 4 λ i = i=1 = = det(s) = GSV C.J. Anderson (Illinois) More Linear Algebra Spring / 40

23 Properties of Covariance Matrices Σ p p & S p p symmetric population and sample covariance matrices, respectively. Most of following holds true for both. Eigenvalues and eigenvectors: S has p pairs of eigenvalues and eigenvectors λ 1,e 1 ; λ 2,e 2 ; ; λ p,e p The λ i s are the roots of the characteristic equation S λi = 0 Eigenvectors are the solutions of the equation Se i = λ i e i C.J. Anderson (Illinois) More Linear Algebra Spring / 40

24 Properties of Covariance Matrices (continued) Since any multiple of e i will solve the above equation, we (usually) set the length of e i = 1 (i.e., L 2 e i = Le i = e i e i = 1). Eigenvectors are orthogonal: e i e k = 0 for all i k. Convention to order eigenvalues: λ 1 λ 2 λ p. Since S (& Σ) are symmetric, eigenvalues are Real numbers. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

25 More about Covariance Matrices Spectral Decomposition: S = λ 1 e 1 e 1 +λ 2 e 2 e 2 + +λ p e p e p = PΛP Pp p = (e 1,e 2,...,e p ) Λp p = diag(λ i ). P P = {e i e k} = PP = I, which implies that P = P 1. Implications for quadratic forms: If x Sx > 0 for all x 0, then S is positive definite and λ i > 0 for all i. If x Sx 0 for all x 0, then S is non-negative or positive semi-definite and λ i 0 for all i. The inverse of S (if S is non-singular, i.e., λ i > 0 for all i) is S 1 = PΛ 1 P = P{diag{1/λ i }P C.J. Anderson (Illinois) More Linear Algebra Spring / 40

26 Numerical Example & Spectral Decomposition S = PΛP = Do SAS/IML Demonstration of this and S 1 = PΛ 1 P. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

27 and Even More about Covariance Matrices If {λ i,e i ;i = 1,...,p} for Σ and Σ is non-singular, then {1/λ i,e i ;i = 1,...p} for Σ 1 That is, Σ and Σ 1 have the same eigenvectors and their eigenvalues are the inverses of each other. S = λ 1 λ 2 λ p = p i=1 λ i. This is the generalized sample variance (GSV). p i=1 s ii = trace(s) = tr(s) = p i=1 λ i. This is the Total Sample Variance. If λ p, the smallest eigenvalue, is greater than 0, then S > 0. If S is singular, then there is at least 1 or more eigenvalues equal to 0. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

28 The Rank of S (and Σ) Definition of rank: The Rank of S = the number of linearly independent rows (columns) = the number of non-zero eigenvalues If S p p is of Full Rank (i.e., rank = p), then λ p > 0 S is positive definite S > 0 S 1 exists S is non-singular definition: p linearly independent rows/columns C.J. Anderson (Illinois) More Linear Algebra Spring / 40

29 Singular Value Decomposition Given matrix A n p, the Singular Value Decomposition (SVD) of A is where A n p = P n r r r Q r p The r columns of P = (p 1,p 2,...p r ) are orthogonal: p i p i = 1 and p i p k = 0 for i k; that is, P P = I r. The r columns of Q = (q 1,q 2,...,q r ) are orthogonal: q i q i = 1 and q i q k = 0 for i k; that is, Q Q = I r. is a diagonal matrix with ordered positive values δ 1 δ 2 δ r r is the rank of A, which must be r min(n,p). C.J. Anderson (Illinois) More Linear Algebra Spring / 40

30 Singular Value Decomposition (continued) A n p = P n r r r Q r p Terminology: P are the left singular vectors Q are the right singular vectors The elements of are the singular values C.J. Anderson (Illinois) More Linear Algebra Spring / 40

31 Relationship between Eigensystems and SVD To show this let X n p which has rank p, and X n p = P n p p p Q p p. The product X p nx n p is a square and symmetric matrix. X p n X n p = (P n p p p Q p p ) (P n p p p Q p p ) = ( Q p p p p P p n )( P n p p p Q p p ) }{{} I = Q p p p p p p Q p p = Q p p 2 p p Q p p }{{}}{{}}{{} vectors valuesvectors If A (e.g., X p nx n p ) is square and symmetric, then SVD gives the same as eigenvector/value decomposition. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

32 Lower Rank SVD Sometimes we want to summarize or approximate the basic structure of a matrix. In particular, let A n p = P n r r r Q r p, then B n p = P n r r r Q r p where r < r (note: r =rank of matrix A). This Lower Rank Decomposition minimizes the loss function n p (a ji b ji ) 2 = δr δr 2 j=1 i=1 This result of the least squared approximation of one matrix by another of lower rank is known as the Eckart-Young theorem. See Eckart, C. & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1, C.J. Anderson (Illinois) More Linear Algebra Spring / 40

33 So What can I do with SVD? Biplot: Lower rank representation of a data matrix. Correspondence Analysis: Lower rank representation of the relationship between two categorical variables. Multiple Correspondence Analysis: Lower rank representations of the relationship between multiple categorical variables. Multidimensional Scaling Reduce the number of parameters in a complex model. and Many other scaling and data analytic methods. We ll examine what a Biplot can give us... Consider the psychological test data: The rank of the data matrix is 4, so X c = (X x) = P Q 4 4 = (P ) Q }{{} cases }{{} variables C.J. Anderson (Illinois) More Linear Algebra Spring / 40

34 Biplot Example: Singular Values Cumulative i δ i δi 2 percent sum percent where percent = (δi 2/ ) 100%, sum = ik=1 δ2 i, and cumulative percent = ( ik=1 δ2 k / ) 100%. If we take a rank 2 decomposition, B = 2 l=1 δ lp l q l = {δ 1 p j1 q i1 +δ 2 p j2 q i2 } = {b ji } and the value of the loss function is loss = n j=1 i=1 4 (x c,ji b ji ) 2 = = Only losing (1096/6692) 100% = 16.39% of the information in the data matrix (loosely speaking). C.J. Anderson (Illinois) More Linear Algebra Spring / 40

35 Biplot Example: Singular Vectors Left Singular Vectors: P 64 4 Right Singular Vectors: Q 4 4 p 1 p 2 p 3 p 4 q 1 q 2 q 3 q etc. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

36 Biplot: Representing Cases First let s look at the rank 2 solution/approximation X c }{{} (64 4) = P }{{} }{{} Q }{{} (64 2) (2 2)(2 4) For our rank 2 solution, to represent subjects or cases, we ll plot the rows of the product P as points in a 2-dimensional space. Let q il = the value in the i th row of q l, so post-multiplying both side by Q gives P = X c,(64 4) Q (4 4) = 4i=1 q i1 x 4i=1 c,1i q i2 x 4i=1 c,1i q i3 x 4i=1 c,1i q i4 x c,1i 4i=1 q i1 x 4i=1 c,2i q i2 x 4i=1 c,2i q i3 x 4i=1 c,2i q i4 x c,2i.... 4i=1 q i1 x 4i=1 c,64i q i2 x 4i=1 c,64i q i3 x 4i=1 c,64i q i4 x c,64i C.J. Anderson (Illinois) More Linear Algebra Spring / 40

37 Biplot: Representing Cases & Variables For cases, what we are plotting are linear combination of the data (mean centered) matrix. For example, for subject one, we plot the point (p j1 δ 1,p j2 δ 2 ) = (( 0.002)(67.685),( 0.248)(31.859))= ( 0.135, 7.901). To represent variables, we ll plot the rows of Q 4 2 as vectors in the 2-dimensional space. For example, for variable one, we ll plot (0.274, 0.001). For the plot, I actually plotted variable vectors multiplied by 30 for cosmetic purposes it doesn t effect the interpretation. C.J. Anderson (Illinois) More Linear Algebra Spring / 40

38 The Graph & Foreshadowing of Things to Come C.J. Anderson (Illinois) More Linear Algebra Spring / 40

39 Maximization of Quadratic Forms for Points on the Unit Sphere In multivariate analyses, we have different goals and purposes different criteria to maximize (or minimize). Let B p p be a positive definite matrix with eigenvalues λ 1 λ 2 λ p and eigenvectors e 1, e 2,..., e p. x Bx Maximization: max x 0 x x = λ 1 is obtained when x = e 1 x Bx Minimization: min x 0 x x = λ p is obtained when x = e p Maximization under an orthogonality constraint: x Bx max x e 1,...,e k x x = λ k+1 is obtained when x = e k+1 C.J. Anderson (Illinois) More Linear Algebra Spring / 40

40 Overview of the Rest of the Semester See pages on web-site... C.J. Anderson (Illinois) More Linear Algebra Spring / 40