Closed Leotief Model Chapter 6 Eigevalues I a closed Leotief iput-output-model cosumptio ad productio coicide, i.e. V x = x = x Is this possible for the give techology matrix V? This is a special case of a so called eigevalue problem. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Eigevalue ad Eigevector Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Example Eigevalue ad Eigevector A vector x R, x = 0, is called eigevector of a matrix A correspodig to eigevalue λ R, if A x = λ x The eigevalues of matrix A are all umbers λ for which a eigevector does exist. For a 3 3 diagoal matrix we fid a 0 0 a A e = 0 a 0 0 = 0 = a e 0 0 a 33 0 0 Thus e is a eigevector correspodig to eigevalue λ = a. Aalogously we fid for a diagoal matrix A e i = a ii e i So the eigevalue of a diagoal matrix are its diagoal elemets with uit vectors e i as the correspodig eigevectors. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 3 / 45 Computatio of Eigevalues I order to fid eigevectors of matrix A we have to solve equatio A x = λx = λix (A λi)x = 0. If (A λi) is ivertible the we get x = (A λi) 0 = 0. However, x = 0 caot be a eigevector (by defiitio) ad thus λ caot be a eigevalue. Thus λ is a eigevalue of A if ad oly if (A λi) is ot ivertible, i.e., if ad oly if det(a λi) = 0 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 4 / 45 Example Eigevalues Compute the eigevalues of matrix A =. 4 We have to fid all λ R where A λi vaishes. λ det(a λi) = 4 λ = λ 5λ + 6 = 0 The roots of this quadratic equatio are λ = ad λ = 3. Thus matrix A has eigevalues ad 3. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 5 / 45 Characteristic Polyomial Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 6 / 45 Computatio of Eigevectors For a matrix A det(a λi) is a polyomial of degree i λ. It is called the characteristic polyomial of matrix A. The eigevalues are the the roots of the characteristic polyomial. For that reaso eigevalues ad eigevectors are sometimes called the characteristic roots ad characteristic vectors, resp., of A. The set of all eigevalues of A is called the spectrum of A. Spectral methods make use of eigevalues. Remark: It may happe that characteristic roots are complex (λ C). These are the called complex eigevalues. Eigevectors x correspodig to a kow eigevalue λ 0 ca be computed by solvig equatio (A λ 0 I)x = 0. Eigevectors of A = correspodig to λ = : 4 x 0 (A λ I)x = = 0 Gaussia elimiatio yields: = α ad x = α v = α for a α R \ {0} Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 7 / 45 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 8 / 45
Eigespace If x is a eigevector correspodig to eigevalue λ, the each multiple αx is a eigevector, too: A (αx) = α(a x) = αλ x = λ (αx) If x ad y are eigevectors correspodig to the same eigevalue λ, the x + y is a eigevector, too: A (x + y) = A x + A y = λ x + λ y = λ (x + y) The set of all eigevectors correspodig to eigevalue λ (icludig zero vector 0) is thus a subspace of R ad is called the eigespace correspodig to λ. Computer programs always retur bases of eigespaces. (Beware: Bases are ot uiquely determied!) Example Eigespace Let A =. 4 Eigevector correspodig to eigevalue λ = : v = Eigevector correspodig to eigevalue λ = 3: v = Eigevectors correspodig to eigevalue λ i are all o-vaishig multiples of v i (i.e., = 0). Computer programs retur ormalized eigevectors: v = 5 5 ad v = Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 9 / 45 Example 0 Eigevalues ad Eigevectors of A = 0 3. 0 6 Create the characteristic polyomial ad compute its roots: λ 0 det(a λi) = 0 3 λ = ( λ) λ (λ 5) = 0 0 6 λ Eigevalues: λ =, λ = 0, ad λ 3 = 5. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 0 / 45 Example Eigevector(s) correspodig to eigevalue λ 3 = 5 by solvig equatio ( 5) 0 (A λ 3 I)x = 0 (3 5) 0 6 ( 5) Gaussia elimiatio yields 3 0 0 0 0 0 6 3 0 x = 0 3 0 0 0 0 0 0 0 0 Thus = α, = α, ad x = 3α for arbitrary α R \ {0}. Eigevector v 3 = (, 3, 6) t Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Example Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Properties of Eigevalues Eigevector correspodig to λ = : v = 0 0 3 λ = 0: v = 6 λ 3 = 5: v 3 = 3 6. A ad A t have the same eigevalues.. Let A ad B be -Matrices. The A B ad B A have the same eigevalues. 3. If x is a eigevector of A correspodig to λ, the x is a eigevector of A k correspodig to eigevalue λ k. 4. If x is a eigevector of regular matrix A correspodig to λ, the x is a eigevector of A correspodig to eigevalue λ. Eigevectors correspodig to eigevalue λ i are all o-vaishig multiples of v i. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 3 / 45 Properties of Eigevalues Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 4 / 45 Eigevalues of Similar Matrices 5. The product of all eigevalues λ i of a matrix A is equal to the determiat of A: det(a) = λ i i= This implies: A is regular if ad oly if all its eigevalues are o-zero. 6. The sum of all eigevalues λ i of a matrix A is equal to the sum of the diagoal elemets of A (called the trace of A). tr(a) = i= a ii = λ i i= Let U be the trasformatio matrix ad C = U A U. If x is a eigevector of A correspodig to eigevalue λ, the U x is a eigevector of C correspodig to λ: C (U x) = (U AU)U x = U Ax = U λx = λ (U x) Similar matrices A ad C have the same eigevalues ad (if we cosider chage of basis) the same eigevectors. We wat to choose a basis such that the matrix that represets the give liear map becomes as simple as possible. The simplest matrices are diagoal matrices. Ca we (always) fid a represetatio by meas of a diagoal matrix? Ufortuately ot i the geeral case. But... Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 5 / 45 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 6 / 45
Symmetric Matrix A matrix A is called symmetric, if A t = A. For a symmetric matrix A we fid: All eigevalues are real. Eigevectors u i correspodig to distict eigevalues λ i are orthogoal (i.e., u t i u j = 0 if i = j). There exists a orthoormal basis {u,..., u } (i.e. the vectors u i are ormalized ad mutually orthogoal) that cosists of eigevectors of A, Trasformatio matrix U = (u,..., u ) is the a orthogoal matrix: U t U = I U = U t Diagoalizatio For the i-th uit vector e i we fid Ad thus U t A U e i = U t A u i = U t λ i u i = λ i U t u i = λ i e i λ 0... 0 U t 0 λ A U = D =... 0...... 0 0... λ Every symmetric matrix A becomes a diagoal matrix with the eigevalues of A as its etries if we use the orthoormal basis of eigevectors. This procedure is called diagoalizatio of matrix A. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 7 / 45 Example Diagoalizatio We wat to diagoalize A =. Eigevalues λ = ad λ = 3 with respective ormalized eigevectors u = ad u = ( With respect to basis {u, u } matrix A becomes diagoal matrix 0 ) Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 8 / 45 A Geometric Iterpretatio I Fuctio x Ax = x maps the uit circle i R ito a ellipsis. The two semi-axes of the ellipsis are give by λ v ad λ v, resp. v v A v 3v 0 3 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 9 / 45 Quadratic Form Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 0 / 45 Example Quadratic Form Let A be a symmetric matrix. The fuctio q A : R R, x q A (x) = x t A x is called a quadratic form. 0 0 Let A = 0 0. The 0 0 3 t x 0 0 q A (x) = 0 0 0 0 3 x = + x + 3 I geeral we fid for matrix A = (a ij ): q A (x) = = q A (x) = x x i= j= a ij x i x j t 3 3 x t x + x + + 3 x + 3 + = + x 4x + + 6 + 3 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Defiiteess Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues / 45 Defiiteess A quadratic form q A is called positive defiite, if for all x = 0, q A (x) > 0. positive semidefiite, if for all x, q A (x) 0. egative defiite, if for all x = 0, q A (x) < 0. egative semidefiite, if for all x, q A (x) 0. idefiite i all other cases. A matrix A is called positive (egative) defiite (semidefiite), if the correspodig quadratic form is positive (egative) defiite (semidefiite). Every symmetric matrix is diagoalizable. Let {u,..., u } be the orthoormal basis of eigevectors of A. The for every x: x = i= c i (x)u i = Uc(x) U = (u,..., u ) is the trasformatio matrix for the orthoormal basis, c the correspodig coefficiet vector. So if D is the diagoal matrix of eigevalues λ i of A we fid q A (x) = x t A x = (Uc) t A Uc = c t U t AU c = c t D c ad thus q A (x) = i= c i λ i Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 3 / 45 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 4 / 45
Defiiteess ad Eigevalues Equatio q A (x) = i= c i λ i immediately implies: Let λ i be the eigevalues of symmetric matrix A. The A (the quadratic form q A ) is positive defiite, if all λ i > 0. positive semidefiite, if all λ i 0. egative defiite, if all λ i < 0. egative semidefiite, if all λ i 0. idefiite, if there exist λ i > 0 ad λ j < 0. Example Defiiteess ad Eigevalues The eigevalues of are λ = 6 ad λ =. 5 Thus the matrix is positive defiite. 5 4 The eigevalues of are 4 5 λ = 0, λ = 3, ad λ 3 = 9. The matrix is positive semidefiite. 7 5 4 The eigevalues of 5 7 4 are 4 4 λ = 6, λ = 6 ad λ 3 =. Thus the matrix is idefiite. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 5 / 45 Leadig Priciple Miors Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 6 / 45 Leadig Priciple Miors ad Defiiteess The defiiteess of a matrix ca also be determied by meas of miors. Let A = (a ij ) be a symmetric matrix. The the determiat of submatrix a... a k A k =..... a k... a kk is called the k-th leadig priciple mior of A. A symmetric Matrix A is positive defiite, if ad oly if all A k > 0. egative defiite, if ad oly if ( ) k A k > 0 for all k. idefiite, if A = 0 ad oe of the two cases is holds. ( ) k A k > 0 meas that A, A 3, A 5,... < 0, ad A, A 4, A 6,... > 0. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 7 / 45 Example Leadig Priciple Miors Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 8 / 45 Example Leadig Priciple Miors Defiiteess of matrix 0 A = 3 0 A = det(a ) = a = > 0 a a A = a a = 3 = 5 > 0 Defiiteess of matrix A = 3 3 A = det(a ) = a = > 0 a a A = a a = = > 0 A ad q A are positive defiite. 0 A 3 = A = 3 = 8 > 0 0 A ad q A are idefiite. A 3 = A = 3 = 8 < 0 3 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 9 / 45 Priciple Miors Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 30 / 45 Priciple Miors ad Semidefiiteess Ufortuately the coditio for semidefiiteess is more tedious. Let A = (a ij ) be a symmetric matrix. The the determiat of submatrix a i,i... a i,i k A i,...,i k =..... a ik,i... a ik,i k is called a priciple mior of order k of A. i <... < i k. A symmetric matrix A is positive semidefiite, if ad oly if all A i,...,i k 0. egative semidefiite, if ad oly if ( ) k A i,...,i k 0 for all k. idefiite i all other cases. ( ) k A i,...,i k 0 meas that A i,...,i k 0, if k is eve, ad A i,...,i k 0, if k is odd. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 3 / 45 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 3 / 45
Example Priciple Miors Example Priciple Miors Defiiteess of matrix 5 4 A = 4 5 The matrix is positive semidefiite. (But ot positive defiite!) priciple miors of order : A = 5 0 A = 0 A 3 = 5 0 priciple miors of order : 5 A, = = 9 0 5 4 A,3 = 4 5 = 9 0 A,3 = 5 = 9 0 Defiiteess of matrix 5 4 A = 4 5 The matrix is egative semidefiite. (But ot egative defiite!) priciple miors of order : A = 5 0 A = 0 A 3 = 5 0 priciple miors of order : 5 A, = = 9 0 5 4 A,3 = 4 5 = 9 0 A,3 = 5 = 9 0 priciple miors of order 3: A,,3 = A = 0 0 priciple miors of order 3: A,,3 = A = 0 0 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 33 / 45 Leadig Priciple Miors ad Semidefiiteess Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 34 / 45 Recipe for Semidefiiteess Obviously every positive defiite matrix is also positive semidefiite (but ot ecessarily the other way roud). Matrix 0 A = 3 0 is positive defiite as all leadig priciple miors are positive (see above). Therefore A is also positive semidefiite. I this case there is o eed to compute the o-leadig priciple miors. Recipe for fidig semidefiiteess of matrix A:. Compute all leadig priciple miors: If the coditio for positive defiiteess holds, the A is positive defiite ad thus positive semidefiite. Else if the coditio for egative defiiteess holds, the A is egative defiite ad thus egative semidefiite. Else if det(a) = 0, the A is idefiite.. Else also compute all o-leadig priciple miors: If the coditio for positive semidefiiteess holds, the A is positive semidefiite. Else if the coditio for egative semidefiiteess holds, the A is egative semidefiite. Else A is idefiite. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 35 / 45 Ellipse Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 36 / 45 A Geometric Iterpretatio II Equatio a + by =, a, b > 0 describes a ellipse i caoical form. Term a + by is a quadratic form with matrix a 0 A = 0 b / b / a It has eigevalues ad ormalized eigevectors λ = a with v = e ad λ = b with v = e. The semi-axes have legth a ad b, resp. λ v λ v These eigevectors coicide with the semi-axes of the ellipse. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 37 / 45 A Geometric Iterpretatio II Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 38 / 45 A Geometric Iterpretatio II Now let A be a symmetric matrix with positive eigevalues. Equatio x t Ax = describes a ellipse where the semi-axes (priciple axes) coicide coicides with its ormalized eigevectors as see below. λ v λ v By a chage of basis from {e, e } to {v, v } by meas of trasformatio U = (v, v ) this ellipse is rotated ito caoical form. λ v λ v U t λ e λ e (That is, we have diagoalize matrix A.) Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 39 / 45 Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 40 / 45
A Applicatio i Statistics Suppose we have observatios of k metric attributes X,..., X k which we combie ito a vector: x i = (x i,..., x ik ) R k A Applicatio i Statistics A chage of basis by meas of a orthogoal matrix does ot chage TSS. However, it chages the cotributios of each of the compoets. The arithmetic mea the is (as for uivariate data) x = i= x i = (x,..., x k ) The total sum of squares is a measure for the statistical dispersio TSS = i= x i x = ad ca be computed compoet-wise. ( k ) x ij x j = j= i= k j= TSS j Ca we fid a basis such that a few compoets cotribute much more to the TSS tha the remaiig oes? Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 4 / 45 Priciple Compoet Aalysis (PCA) Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 4 / 45 Priciple Compoet Aalysis (PCA) Assumptios: The data are approximately multiormal distributed. Procedure:. Compute the covariace matrix Σ.. Compute all eigevalues ad ormalized eigevectors of Σ. 3. Sort eigevalues such that λ λ... λ k. 4. Use correspodig eigevectors v,..., v k as ew basis. 5. The cotributio to TSS of the first m compoets i this basis is m j= λ j m j= TSSj k j= TSSj k j= λ. j By meas of PCA it is possible to reduce the umber of dimesios without reducig the TSS substatially. Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 43 / 45 Summary Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 44 / 45 Eigevalues ad eigevectors Characteristic polyomial Eigespace Properties of eigevalues Symmetric Matrices ad Diagoalizatio Quadratic forms Defiitess Priciple miors Priciple compoet aalysis Josef Leydold Mathematical Methods WS 08/9 6 Eigevalues 45 / 45