Eigenvalues and Eigenvectors

Chapter 6 Eigenvalues and Eigenvectors 6. Introduction to Eigenvalues Eigenvalues are the key to a system of n differential equations : dy=dt ay becomes dy=dt Ay. Now A is a matrix and y is a vector.y.t/; : : : ; y n.t//. The vector y changes with time. Here is a system of two equations with its by matrix A : y 4y C y y y C y is y y 4 y : () y How to solve this coupled system, y Ay with y and y in both equations? The good way is to find solutions that uncouple the problem. We want y and y to grow or decay in exactly the same way (with the same e t ) : Look for y.t/ e t a y.t/ e t b In vector notation this is y.t/ e t x () That vector x.a; b/ is called an eigenvector. The growth rate is an eigenvalue. This section will show how to find x and. Here I will jump to x and for the matrix in (). a First eigenvector x = and first eigenvalue 5 in y e b 5t x y e 5t y 5 e 5t 4y C y has y e 5t y 5 e 5t y C y a Second eigenvector x = and second eigenvalue in y e b t x This y e t x is a second solution y e t y e t has y e t 4y C y y e t y C y

6 Chapter 6. Eigenvalues and Eigenvectors Those two x s and s combine with any c, c to give the complete solution to y Ay : e 5t e Complete solution y.t/ c e 5t C c t e t c e 5t C c e t : () This is exactly what we hope to achieve for other equations y Ay with constant A. The solutions we want have the special form y.t/ e t x. Substitute that solution into y Ay, to see the equation Ax x for an eigenvalue and its eigenvector x : d dt.et x/ A.e t x/ is e t x Ae t x: ivide both sides by e t : Eigenvalue and eigenvector of A Ax x (4) Those eigenvalues (5 and for this A) are a new way to see into the heart of a matrix. This chapter enters a different part of linear algebra, based on Ax x. The last page of Chapter 6 has eigenvalue-eigenvector information about many different matrices. Finding Eigenvalues from det.a I/ Almost all vectors change direction, when they are multiplied by A. Certain very exceptional vectors x are in the same direction as Ax. Those are the eigenvectors. The vector Ax (in the same direction as x) is a number times the original x. The eigenvalue tells whether the eigenvector x is stretched or shrunk or reversed or left unchanged when it is multiplied by A. We may find or or or. The eigenvalue could be zero! Ax x puts this eigenvector x in the nullspace of A. If A is the identity matrix, every vector has Ax x. All vectors are eigenvectors of I. Most by matrices have two eigenvector directions and two eigenvalues and. To find the eigenvalues, write the equation Ax x in the good form.a I/x. If.A I/x, then A I is a singular matrix. Its determinant must be zero. The determinant of A I a b c d is.a /.d / bc : Our goal is to shift A by the right amount I, so that.a I/x has a solution. Then x is the eigenvector, is the eigenvalue, and A I is not invertible. So we look for numbers that make det.a I/. I will start with the matrix A in equation (). 4 Example For A, subtract from the diagonal and find the determinant : det.a 4 I/ det 6 C 5. 5/. /: (5) I factored the quadratic, to see the two eigenvalues 5 and. The matrices A 5I and A I are singular. We have found the s from det.a I/.

6.. Introduction to Eigenvalues 7 For each of the eigenvalues 5 and, we now find an eigenvector x :.A 5I/ x is.a I/ x is x x and and x x Those were the vectors.a; b/ in our special solutions y e t x. Both components of y have the growth rate, so the differential equation was easily solved : y e t x. Two eigenvectors gave two solutions. Combinations c y C c y give all solutions. Example Find the eigenvalues and eigenvectors of the Markov matrix A :8 : : :7. det.a :8 : I/ det : :7 C. / : I factored the quadratic into times, to see the two eigenvalues and. The eigenvectors x and x are in the nullspaces of A I and A I..A I/ x is Ax x The first eigenvector is x.:6; :4/.A I/ x is Ax x The second eigenvector is x.; / :6 x :4 x and and :8 : :6 Ax x (Ax x means that ) : :7 :4 :8 : :5 Ax (this is : :7 :5 x so ). If x is multiplied again by A, we still get x. Every power of A will give A n x x. Multiplying x by A gave x, and if we multiply again we get. / times x. When A is squared, the eigenvectors x stay the same. A x A.x/.Ax/ x. Notice. This pattern keeps going, because the eigenvectors stay in their own directions. They never get mixed. The eigenvectors of A are the same x and x. The eigenvalues of A are and. / very small number. We mention that this particular A is a Markov matrix. Its entries are positive and every column adds to. Those facts guarantee that the largest eigenvalue must be.

8 Chapter 6. Eigenvalues and Eigenvectors :6 Ax x :4 A x./ x :5 :5 Ax x :5 x :5 A A x.:5/ x :8 : : :7 Ax x A n x n x :5 :5 Figure 6.: The eigenvectors keep their directions. A has eigenvalues and.:5/. The eigenvector Ax x is the steady state which all columns of A k will approach. Giant Markov matrices are the key to Google s search algorithm. It ranks web pages. Linear algebra has made Google one of the most valuable companies in the world. Powers of a Matrix When the eigenvalues of A are known, we immediately know the eigenvalues of all powers A k and shifts A C ci and all functions of A. Each eigenvector of A is also an eigenvector of A k and A and A C ci : If Ax x then A k x k x and A x x and.a C ci/x. C c/x: (6) Start again with A x, which is A times Ax x. Then Ax is the same as Ax for any number, and Ax is x. We have proved that A x x. For higher powers A k x, continue multiplying Ax x by A. Step by step you reach A k x k x. For the eigenvalues of A, first multiply by A and then divide by : Eigenvalues of A are Ax x x A x A x x (7) We are assuming that A exists! If A is invertible then will never be zero. Invertible matrices have all. Singular matrices have the eigenvalue. The shift from A to A C ci just adds c to every eigenvalue (don t change x) : Shift of A If Ax x then.a C ci/x Ax C cx. C c/x: (8) As long as we keep the same eigenvector x, we can allow any function of A : Functions of A.A C A C 5I/x. C C 5/x e A x e x: (9)

6.. Introduction to Eigenvalues 9 I slipped in e A I C A C A C to show that infinite series produce matrices too. Let me show you the powers of the Markov matrix A in Example. That starting matrix is unrecognizable after a few steps. :8 : :7 :45 :65 :55 :6 :6 : :7 : :55 :5 :475 :4 :4 () A A A A A was found by using and its eigenvector Œ:6; :4, not by multiplying matrices. The eigenvalues of A are and, so the eigenvalues of A are and. /. That last number is extremely small, and we can t see it in the first digits of A. How could you multiply A 99 times another vector like v.:8; :/? This is not an eigenvector, but v is a combination of eigenvectors. This is a key idea, to express any vector v by using the eigenvectors. Separate into eigenvectors v x C.:/x :8 :6 : v C : () : :4 : Each eigenvector is multiplied by its eigenvalue, when we multiply the vector by A. After 99 steps, x is unchanged and x is multiplied by. /99 : :8 A 99 is A 99.x C :x / x C.:/. :6 : /99 x C 4 very small 5 : :4 vector This is the first column of A, because v.:8; :/ is the first column of A. The number we originally wrote as :6 was not exact. We left out.:/. /99 which wouldn t show up for decimal places. The eigenvector x.:6; :4/ is a steady state that doesn t change (because /. The eigenvector x is a decaying mode that virtually disappears (because =/. The higher the power of A, the more closely its columns approach the steady state. Bad News About AB and A C B Normally the eigenvalues of A and B (separately) do not tell us the eigenvalues of AB. We also don t know about A C B. When A and B have different eigenvectors, our reasoning fails. The good results for A are wrong for AB and A C B, when AB is different from BA. The eigenvalues won t come from A and B separately : A B AB BA A C B All the eigenvalues of A and B are zero. But AB has an eigenvalue, and A C B has eigenvalues and. But one rule holds : AB and BA have the same eigenvalues.

Chapter 6. Eigenvalues and Eigenvectors eterminants The determinant is a single number with amazing properties. It is zero when the matrix has no inverse. That leads to the eigenvalue equation det.a I/. When A is invertible, the determinant of A is =.det A/. Every entry in A is a ratio of two determinants. I want to summarize the algebra, leaving the details for my companion textbook Introduction to Linear Algebra. The difficulty with det.a I/ is that an n by n determinant involves n Š terms. For n 5 this is terms generally impossible to use. For n there are six terms, three with plus signs and three with minus. Each of those six terms includes one number from every row and every column : 6 7 4 4 5 65 4 5 7 8 9 7 8 C C C eterminant from nš 6 terms Three plus signs, three minus signs C./.5/.9/ C./.6/.7/ C./.4/.8/./.5/.7/./.6/.8/./.4/.9/ That shows how to find the six terms. For this particular matrix the total must be det A, because the matrix happens to be singular : row C row equals.row /. Let me start with five useful properties of determinants, for all square matrices.. Subtracting a multiple of one row from another row leaves det A unchanged.. The determinant reverses sign when two rows are exchanged.. If A is triangular then det A product of diagonal entries. 4. The determinant of AB equals.det A/ times.det B/. 5. The determinant of A T equals the determinant of A. By combining ; ; you will see how the determinant comes from elimination : The determinant equals (product of the pivots): () Property says that A and U have the same determinant, unless rows are exchanged. Property says that an odd number of exchanges would leave det A det U. Property says that det U is the product of the pivots on its main diagonal. When elimination takes A to U, we find det A (product of the pivots). This is how all numerical software (like MATLAB or Python or Julia ) would compute det A. Plus and minus signs play a big part in determinants. Half of the n Š terms have plus signs, and half come with minus signs. For n, one row exchange puts 5 7 or 6 8 or 4 9 on the main diagonal. A minus sign from one row exchange.

6.. Introduction to Eigenvalues Two row exchanges (an even number) take you back to () (6) (7) and () (4) (8). This indicates how the 4 terms would go for n 4, twelve terms with plus and twelve with minus. Even permutation matrices have det P and odd permutations have det P. Inverse of A If det A, you can solve Av b and find A using determinants : Cramer s Rule v det B det A v det B det A v n det B n det A () The matrix B j replaces the j th column of A by the vector b. Cramer s Rule is expensive! To find the columns of A, we solve AA I. That is the Gauss-Jordan idea : For each column b in I, solve Av b to find a column v of A. In this special case, when b is a column of I, the numbers det B j in Cramer s Rule are called cofactors. They reduce to determinants of size n, because b has so many zeros. Every entry of A is a cofactor of A divided by the determinant of A. I will close with three examples, to introduce the trace of a matrix and to show that real matrices can have imaginary (or complex) eigenvalues and eigenvectors. Example Find the eigenvalues and eigenvectors of S. Solution You can see that x.; / will be in the same direction as Sx.; /. Then x is an eigenvector of S with. We want the matrix S I to be singular. S det.s I/ det 4 C : Notice that is the determinant of S (without ). And 4 is the sum C down the central diagonal of S. The diagonal sum 4 is the trace of A. It equals C C. Now factor 4 C into. /. /. The matrix S I is singular (zero determinant) for and. Each eigenvalue has an eigenvector:.s I/ x.s I/ x The eigenvalues and are real. The eigenvectors.; / and.; / are orthogonal. Those properties always come together for symmetric matrices (Section 6.5). Here is an antisymmetric matrix with A T A. It rotates all real vectors by 9 ı. Real vectors can t be eigenvectors of a rotation matrix because it changes their direction.

Chapter 6. Eigenvalues and Eigenvectors Example 4 This real matrix has imaginary eigenvalues i, i and complex eigenvectors : A A T det.a I/ det C : That determinant C is zero for i and i. The eigenvectors are.; i/ and.; i/ : i i i i i i i i Somehow those complex vectors x and x don t get rotated (I don t really know how). Multiplying the eigenvalues.i/. i/ gives det A. Adding the eigenvalues gives.i/ C. i/. This equals the sum C down the diagonal of A. Product of eigenvalues = determinant Sum of eigenvalues = trace (4) Those are true statements for all square matrices. The trace is the sum a C C a nn down the main diagonal of A. This sum and product are is especially valuable for by matrices, when the determinant ad bc and the trace C a C d completely determine and. Look now at rotation of a plane through any angle. Example 5 Rotation comes from an orthogonal matrix Q. Then e i and e i : cos Q sin sin cos cos C i sin cos i sin C cos trace determinant I multiplied. /. / to get cos C sin. In polar form e i times e i is. The eigenvectors of Q are.; i/ and.; i/ for all rotation angles. Before ending this section, I need to tell you the truth. It is not easy to find eigenvalues and eigenvectors of large matrices. The equation det.a I/ is more or less limited to by and by. For larger matrices, we can gradually make them triangular without changing the eigenvalues. For triangular matrices the eigenvalues are on the diagonal. A good code to compute and x is free in LAPACK. The MATLAB command is eig.a/. REVIEW OF THE KEY IEAS. Ax x says that eigenvectors x keep the same direction when multiplied by A.. Ax x also says that det.a I/. This equation determines n eigenvalues.. The eigenvalues of A and A are and, with the same eigenvectors as A. 4. Singular matrices have. Triangular matrices have s on their diagonal.

6.. Introduction to Eigenvalues 5. The sum down the main diagonal of A (the trace) is the sum of the eigenvalues. 6. The determinant is the product of the s. It is also (product of the pivots). Problem Set 6. Example has powers of this Markov matrix A : :8 : :7 :45 A and A : :7 : :55 and A :6 :6 : :4 :4 (a) A has eigenvalues and. Find the eigenvalues of A and A. (b) What are the eigenvectors of A? One eigenvector is in the nullspace. (c) Check the determinant of A and A. Compare with.det A/ and.det A/. Find the eigenvalues and the eigenvectors of these two matrices : 4 4 A and A C I : 4 A C I has the eigenvectors as A. Its eigenvalues are by. Compute the eigenvalues and eigenvectors of A and also A : A and A = : = A has the eigenvectors as A. When A has eigenvalues and, its inverse has eigenvalues. Check that C trace of A C. 4 Compute the eigenvalues and eigenvectors of A and A : 7 A and A : 6 A has the same as A. When A has eigenvalues and, the eigenvalues of A are. In this example, why is C? 5 Find the eigenvalues of A and B (easy for triangular matrices) and A C B : 4 A and B and A C B : 4 Eigenvalues of A C B (are equal to) (might not be equal to) eigenvalues of A plus eigenvalues of B.

4 Chapter 6. Eigenvalues and Eigenvectors 6 Find the eigenvalues of A and B and AB and BA : A and B and AB and BA : (a) Are the eigenvalues of AB equal to eigenvalues of A times eigenvalues of B? (b) Are the eigenvalues of AB equal to the eigenvalues of BA? Yes! 7 Elimination produces a triangular matrix U. The eigenvalues of U are on its diagonal (why?). They are not the eigenvalues of A. Give a by example of A and U. 8 (a) If you know that x is an eigenvector, the way to find is to. (b) If you know that is an eigenvalue, the way to find x is to. 9 What do you do to the equation Ax x, in order to prove (a), (b), and (c)? (a) is an eigenvalue of A, as in Problem 4. (b) is an eigenvalue of A, as in Problem. (c) C is an eigenvalue of A C I, as in Problem. Find the eigenvalues and eigenvectors for both of these Markov matrices A and A. Explain from those answers why A is close to A : :6 : A and A :4 :8 = = : = = A by matrix B has eigenvalues ; ;. This information allows you to find : (a) the rank of B (b) the eigenvalues of B (c) the eigenvalues of.b CI/. Find three eigenvectors for this matrix P. Projection matrices only have and. Eigenvectors are in or orthogonal to the subspace that P projects onto. : :4 Projection matrix P P P T P 4 :4 :8 5 : If two eigenvectors x and y share the same repeated eigenvalue, so do all their combinations cx C d y. Find an eigenvector of P with no zero components. From the unit vector u 6 ; 6 ; 6 ; 5 6 construct the rank one projection matrix P uu T. This matrix has P P because u T u. (a) Explain why P u.uu T /u equals u. Then u is an eigenvector with. (b) If v is perpendicular to u show that Pv. Then. (c) Find three independent eigenvectors of P all with eigenvalue.

6.. Introduction to Eigenvalues 5 4 Solve det.q I/ by the quadratic formula to reach cos i sin : cos sin Q rotates the xy plane by the angle. No real s. sin cos Find the eigenvectors of Q by solving.q I/x. Use i. 5 Find three by matrices that have. The trace is zero and the determinant is zero. A might not be the zero matrix but check that A is all zeros. 6 This matrix is singular with rank one. Find three s and three eigenvectors : Rank one A 4 5 4 4 4 5 : 7 When a C b c C d show that.; / is an eigenvector and find both eigenvalues : 5 a b Use the trace to find A A : 4 c d 8 If A has 4 and 5 then det.a I/. 4/. 5/ 9 C. Find three matrices that have trace a C d 9 and determinant, so 4 and 5. 9 Suppose Au u and Av v and Aw 5w. The eigenvalues are ; ; 5. (a) Give a basis for the nullspace of A and a basis for the column space. (b) Find a particular solution to Ax v C w. Find all solutions. (c) Ax u has no solution. If it did then would be in the column space. Choose the last row of A to produce.a/ eigenvalues 4 and 7.b/ any and. Companion matrix A : The eigenvalues of A equal the eigenvalues of A T. This is because det.a I/ equals det.a T I/. That is true because. Show by an example that the eigenvectors of A and A T are not the same. Construct any by Markov matrix M : positive entries down each column add to. Show that M T.; ; /.; ; /. By Problem, is also an eigenvalue of M. Challenge : A by singular Markov matrix with trace has what s? Suppose A and B have the same eigenvalues ; : : :; n with the same independent eigenvectors x ; : : :; x n. Then A B. Reason : Any vector v is a combination c x C C c n x n. What is Av? What is Bv?

6 Chapter 6. Eigenvalues and Eigenvectors 4 The block B has eigenvalues ; and C has eigenvalues ; 4 and has eigenvalues 5; 7. Find the eigenvalues of the 4 by 4 matrix A : B C A 6 4 7 4 6 5 : 6 5 Find the rank and the four eigenvalues of A and C : A 6 7 4 5 and C 6 7 4 5 : 6 Subtract I from the previous A. Find the eigenvalues of B and B : B A I 6 7 4 5 and B 6 7 4 5 : 7 (Review) Find the eigenvalues of A, B, and C : A 4 4 5 5 and B 4 5 and C 4 5 : 6 8 Every permutation matrix leaves x.; ; : : :; / unchanged. Then. Find two more s (possibly complex) for these permutations, from det.p I/ : P 4 5 and P 4 5 : 9 The determinant of A equals the product n. Start with the polynomial det.a I/ separated into its n factors (always possible). Then set : det.a I/. /. /. n / so det A : The sum of the diagonal entries (the trace) equals the sum of the eigenvalues : a b A has det.a I/.a C d/ C ad bc : c d The quadratic formula gives the eigenvalues.acd C p /= and. Their sum is. If A has and 4 then det.a I/.