Unit 5. Matrix diagonaliza1on

Transcription

1 Unit 5. Matrix diagonaliza1on Linear Algebra and Op1miza1on Msc Bioinforma1cs for Health Sciences Eduardo Eyras Pompeu Fabra University hlp://comprna.upf.edu/courses/master_mat/

2 We have seen before that a linear map can have different matrix representa1ons depending on the basis used. We will see that there is a special basis for which this matrix representa1on is diagonal. Consider a linear map from R n to itself represented by a square matrix: B M nxn (R) Vectors are transformed as w = Bu We have seen before how to change basis w' = Pw u' = Pu The linear map can now be represented with the vectors in the new basis: P 1 w' = BP 1 u' w' = PBP 1 u' And now the linear map is represented in terms of this new matrix: PBP 1 M nxn (R)

3 Defini&on: Two matrices are called similar if they are related through a third matrix in the following way: A, B M nxn (R) similar if P M nxn (R) invertible / A = P 1 BP Recall that P invertible det(p) Note that two similar matrices have the same determinant. Proof: A, B similar: A = P 1 BP Given We rewrite det(a) det(a) = det(p 1 BP) = det(p 1 )det(b)det(p) = 1 det(b)det(p) = det(b) det(p)

4 Some matrices can be transformed to a diagonal form Defini&on: A matrix is diagonalizable if it is similar to a diagonal matrix, i.e.: A M nxn (R) is diagonalizable if P M nxn (R) invertible / P 1 AP is diagonal P is a matrix of change to a basis where A has a diagonal form. We will see next how to calculate P and the values in the diagonal form

5 Eigenvectors and Eigenvalues Defini&on: Let A be a square matrix A M mxn (R) λ R Is an eigenvalue of A if For some vector u Au = λu We can rewrite this as a system of equa1ons: Au = λu ( λi n A)u = For example, in R 2: a b c d u 1 u 2 = λ u 1 u 2 λ λ a b c d a λ b c d λ u 1 u 2 u 1 u 2 = =

6 Eigenvectors and Eigenvalues The system of equa1ons: Au = λu ( λi n A)u = Is homogeneous. It always has the trivial solu1on (u = null vector). In fact, it has only the trivial solu1on if the determinant of the matrix is non-zero (it has an inverse): Bu = B 1 Bu = u = Understood as a vector of zeroes. det(b) And it has infinity many solu1ons if the determinant of the matrix is zero det(b) = Matrix column (or row) vectors are linearly dependent

7 Eigenvectors and Eigenvalues How to calculate eigenvalues and eigenvectors? Defini&on: Let A be a square matrix A M mxn (R) λ R Is an eigenvalue of A det λi n A ( ) = A vector u is an eigenvector of λ ( λi n A)u = To compute eigenvalues we solve the equa1on ( ) = (the characteris1c equa1on) det λi n A det( λi n A) = λ n +α n 1 λ n α 2 λ 2 +α 1 λ +α (characteris1c polynomial) Each eigenvalue λ i is a solu1on of the characteris1c polynomial. To compute the eigenvectors, we solve the linear equa1on for reach eigenvalue: ( λ i I n A)u =

8 Eigenvectors and Eigenvalues The set of eigenvectors for a given eigenvalue form a vector subspace: The set of solu1ons for a given eigenvalue is called the Eigenspace of A corresponding to an eigenvalue λ: E(λ) = { u / ( λi n A)u = } Note that E(λ) is a vector subspace because it is the Kernel of a linear map: E(λ) = Ker ( λi n A) Recall the proof that Ker(f) is a subspace. Leb as an exercise: show that λi n A is a linear map Given a square matrix represen1ng a linear map on a vector space, the eigenvectors describe the subspaces in which the matrix works as a mul1plica1on by a number (the eigenvalues)

9 Eigenvectors and Eigenvalues Example Consider a diagonal matrix in 2 dimensions: We write the characteris1c equa1on ( ) = det det λi 2 A λ λ 2 A = = (λ + 3)(λ 2)+ 4 = eigenvalues λ 2 + λ 2 = (λ + 2)(λ 1) = λ = 2,1 Eigenvector for the eigenvalue λ = 2 ( 2I 2 A)u = u 1 u 2 Hence the eigenvectors have the form: E( 2) = u = a a / 4, a R (vector subspace for the eigenvalue -2) = u 4u 1 2 u 1 4u 2 In par1cular, u = = 1 1/ 4 u 1 = 4u 2 is an eigenvector with eigenvalue -2

10 Eigenvectors and Eigenvalues Eigenvector for the eigenvalue λ = 1 ( I 2 A)u = u 1 u 2 Hence the eigenvectors have the form: E(1) = u = a a, a R = 4u 1 4u 2 u 1 u 2 = u 1 = u 2 In par1cular, u = 1 1 is an eigenvector with eigenvalue 1 (vector subspace for the eigenvalue 1)

11 Eigenvectors and Eigenvalues Example: det λi 3 A A = ( ) = det solu1ons: λ = 5, 7,3 λ λ λ 3 Eigenvector for -5: x ( 5I 3 A)u = 3 12 y = z The eigenvector subspace is: x E(λ = 5) = u = y, x = 16 z 9 z, 4 9 z, z R = ( λ + 5) ( λ 7) ( λ 3) = u = 16 9 z 4 9 z z

12 Eigenvectors and Eigenvalues Eigenvector for 7: 7I 3 A ( )u = x y z = u = 2z z E(λ = 7) = u = 2z z, z R The eigenvector subspace is: Eigenvector for -5: 3I 3 A ( )u = x y z = u = z E(λ = 3) = u = z, z R The eigenvector subspace is:

13 Calcula1ng the eigenvalues and eigenvectors allows use to obtain a diagonal form of the matrix, and a set of subspaces on which the matrix acts by a simply mul1plying by a number (as a diagonal). This is what is known as matrix diagonaliza1on. A matrix is diagonalizable if it is similar to a diagonal matrix: A M nxn (R) is diagonlizable if P M nxn (R) invertible / P 1 AP is diagonal Now let s see the meaning of this diagonal matrix.

14 Theorem: A, B M nxn (R) similar A, B have the same eigenvalues Proof: Given we two square matrices that are similar: A, B M nxn (R), A = P 1 BP The eigenvalues are calculated with the characteris1c polynomial, that is: ( ) = det λp 1 P P 1 BP det λi n A ( ) = det( P 1 (λi n B)P) = det(p 1 )det( λi n B)det(P) = det λi n B ( ) Hence, two similar matrices have the same characteris1c polynomial and therefore will have the same eigenvalues. Thus, to diagonalize a matrix is to establish its similarity to a diagonal matrix containing its eigenvalues

15 There is a rela1on between the rank of a matrix and its eigenvalues: Recall that if A is diagonalizable, it is similar to a diagonal matrix: D = P 1 AP is diagonal We have seen before that they have the same determinant: D = P 1 AP det(d) = det(a) We conclude that a matrix is singular (det(a) = ) if at least one of its eigenvalues is zero. (Recall that the determinant of a diagonal matrix is simply the product of the diagonal elements). Recalling the rela1on of determinant with rank(a) we can then say: rank(a) = number of different non-zero eigenvalues of A

16 Proof: Eigenvectors and Eigenvalues Theorem: The eigenvectors of a matrix are linearly independent Consider the case of two non-zero eigenvectors for a 2x2 matrix A: u 1,u 2, Au 1 = λ 1 u 1, Au 2 = λ 1 u 2 We first assume they are linearly dependent: u 1 = cu 2 Apply the matrix A on both sides and use the fact that they are eigenvectors: λ 1 u 1 = cλ 2 u 2 = λ 2 u 1 (λ 1 λ 2 )u 1 = u 1 = cu 2 The eigenvalues are generally different, so it must be that u 1 = We arrive at a contradic1on, since we assumed that the eigenvectors are non-zero So the eigenvectors cannot be linearly dependent.

17 Eigenvectors and Eigenvalues The proof is similar for n eigenvectors: Assume linear dependence: u 1 = n j=2 α j u j we assume that any one vector can be wrilen as a linear combina1on of the rest Apply the matrix A: Can be re-wrilen as: n λ 1 u 1 = α j λ j u j = λ 1 α j u j n j=2 j=2 j=2 (λ 1 λ j )α j u j n = α j =, j Eigenvalues are generally different λ i λ i, i j, so for this to be true, necessarily all coefficients α j must be zero. This means that the vectors are linearly independent. So we arrive at a contradic1on and the vectors must be linearly independent. As a result, the eigenvectors of a matrix with maximal rank (no zero eigenvalues) form a basis of the vector space.

18 Theorem: A M nxn (R) is diagonalizable A has n linearly independent eigenvectors Proof: Assume A is diagonalizable. Then, we know it must be similar to a diagonal matrix, i.e: We can write: P 1 AP = D = P M nxn (R) / P 1 AP is diagonal λ 1! and P = p 1! p n λ n We define P in terms of column vectors p i We mul1ply both sides of the equa1on by P from the leb: P 1 AP = D AP = PD

19 AP = PD A Which we can rewrite as: p 1! p n Ap 1! Ap n = λ 1 p 1! λ n p n = p 1! p n λ 1! λ n Ap i = λ i p i So the column vectors of P, p i, are actually eigenvectors of A Since A is diagonalizable, P is inver1ble, so the column eigenvectors p i cannot be linearly dependent of each other, since otherwise det(p) = Thus, A M nxn (R) is diagonalizable A has n linearly independent eigenvectors And P is built from these eigenvectors

20 We now prove the converse. Assume that A has n linearly independent eigenvectors. That means (1) p i,i =1,..., n / Ap i = λ i p i We define a matrix P by using p i as the column vectors P = p 1! p n We define a diagonal matrix D, where the diagonal values are these eigenvalues: D = λ 1! λ n We can rewrite the equa1on (1) above in terms of the matrices P and D: Ap i = λ i p i,i =1...n AP = PD D = P 1 AP Since A is similar to a Diagonal matrix, then A is diagonalizable, thus. A M nxn (R) is diagonalizable A has n linearly independent eigenvectors

21 Conclusion: A matrix is diagonalizable if we can write (1) A = PDP 1 Where P is the matrix containing the vector columns of eigenvectors P = p 1! p n And D is the diagonal matrix containing the eigenvalues: D = λ 1! λ n

22 Example: Consider the following matrix A = det(a λi 2 ) = det 1 λ λ det(a λi 2 ) = λ = 2 ± Calculate its eigenvalues and eigenvectors and build the matrix P to transform it into a diagonal matrix through P -1 AP We write down the characteris1c polynomial Eigenvectors: x y x y = 3 = 1 x y x y = (1 λ) 2 4 = λ 2 2λ 3 λ = 3, 1 x + 2y = 3x 2x + y = 3y x + 2y = x 2x + y = y It has two solu1ons x = y x x x = y x x Eigenvectors of 3 Eigenvectors of -1

23 We have the following subspaces (set of solu1ons of Au=λu) a E(λ = 3) =, a R a R2 E(λ = 1) = b b, b R R2 We choose two par1cular eigenvectors, one from each space: 1 1 E(λ = 3), 1 1 E(λ = 1) We build P from these vectors: P = Now we need to calculate P -1 and check that P -1 AP is a diagonal matrix with the eigenvalues in the diagonal

24 We calculate the inverse of the matrix Recall the defini1on of inverse: P 1 = 1 det P Adj(P) = 1 det P C T P = 1 2 We confirm the that it is the inverse P 1 P = P 1 AP = P = = Now we confirm that A is similar to a diagonal matrix through P, and that this diagonal matrix contains the eigenvalues: = 3 1 Important: any other matrix P with 2 eigenvectors will produce the same result.

25 Exercise Consider the matrix A, and the matrix of eigenvectors P: A = , P = Verify that A is diagonalizable by compu1ng P -1 AP

26 In general, the eigenvectors of a matrix give rise to eigenspaces, whose eigenvectors are linearly independent between each other (see previous result). But in general they are NOT orthogonal to each other. However, for a symmetric matrix, the corresponding eigenvectors are always orthogonal Theorem. If v 1,..., v r are eigenvectors for a real symmetric matrix A and if the corresponding eigenvalues are all different, then the eigenvectors corresponding to different eigenvalues are orthogonal to each other.

27 Proof: First we show that for a given n n real matrix A, if u is an eigenvector for A T and if v is an eigenvector for A, and if the corresponding eigenvalues are different, then u and v must be orthogonal: A T u = λ u u, Av = λ v v, ( ) u, v = u T v = u 1 u n ( ) u, Bv = u 1 u n A T u, v = λ u u, v = λ u v 1 v n b 11! b nn u, v A T u, v = ( A T u) T v = u T Av = λ v u T v = λ v u, v if λ u λ v u, v = v 1 v n = u T (Bv) = (B T u) T v = B T u, v ( λ u λ v ) u, v = Where we have used the following property of the scalar product: In the case of a symmetric matrix, A T =A, so the result is true for any pair of eigenvectors for different eigenvalues of A Note: the eigenvalues of A and A T are generally different

28 Consider the example from before: A = , P = In this case, any two vectors from either eigenspace are orthogonal to each other: a E(λ = 3) =, a R a R2 E(λ = 1) = b b, P 1 AP =, b R R2 3 1 Consider two vectors: u E(λ = 3) u = ( ) u, v = a a b b a a, a R; v E(λ = 1) v = = ab ab = u v b b, b R

29 Exercise: Consider the following matrix A = 1 1/ Show that A is not diagonalizable Hint: use the previous theorem: A M nxn (R) is diagonalizable A has n linearly independent eigenvectors

30 Exercise: Consider the following linear map: f : P 1 P 1 ax + b! f (ax + b) = (a + b)x + (a + b) where P 1 = { ax + b, a, b R} is the vector space of polynomials of degree 1 a) Calculate the associated matrix A b) Calculate the eigenvalues and eigenvectors associated to this linear map. c) What is the matrix P such that P -1 AP is diagonal