Linear Algebra: Characteristic Value Problem

Linear Algebra: Characteristic Value Problem

. The Characteristic Value Problem Let < be the set of real numbers and { be the set of complex numbers. Given an n n real matrix A; does there exist a number 2 { and a non-zero vector x 2 { n such that Ax = x? () This is known as the characteristic value problem or the eigenvalue problem. If x 6= 0 and satisfy the equation Ax = x; then is called a characteristic value or eigenvalue of A; and x is called a characteristic vector or eigenvector of A: Clearly, () holds if and only if (A I) x = 0: (2) - But (2) holds for non-zero x if and only if the column vectors of (A I) are linearly dependent, that is, ja Ij = 0: (3)

2 This equation is called the characteristic equation of A: Consider the expression f () = ja Ij : (4) - f is a polynomial of degree n in : It is called the characteristic polynomial of A. Example: Consider the 2 2 matrix A = 2 2 : The characteristic equation is: 2 2 = 0; ) (2 )2 = 0; ) ( ) ( 3) = 0: Thus, the characteristic roots are = and = 3:

3 Putting = in (2), we get which yields x x 2 = x + x 2 = 0: - Thus the general solution of the characteristic vector corresponding to the characteristic root = is given by 0 0 (x ; x 2 ) = (; ) ; for 6= 0: - Similarly, corresponding to the characteristic root = 3; we have the characteristic vector given by (x ; x 2 ) = (; ) ; for 6= 0:

In general, the characteristic equation will have n roots in the complex plane (by the Fundamental Theorem of Algebra ), since it is a polynomial equation (in ) of degree n: (Of course some of these roots might be repeated.) In general, the corresponding eigenvectors will also have their components in the complex plane. 4 Example: Consider the 2 2 matrix A = 2 2 : Check that the eigenvalues are complex numbers and the eigenvectors have complex components.

2. Characteristic Values, Trace & Determinant of a Matrix If A is an n n matrix, the trace of A; denoted by tr(a), is the number dened by nx tr(a) = a ii : a a #. Consider the 2 2 matrix A = 2 : a 2 a 22 Write down the characteristic equation. Suppose and 2 are two characteristic values of A: Prove that (a) + 2 = tr(a); and (b) 2 = jaj : In general, for an n n matrix A; with eigenvalues ; 2 ; :::; n ; we have nx ny i = tr(a); and i = jaj : i= i= i= 5

3. Characteristic Values & Vectors of Symmetric Matrices There is considerable simplication in the theory of characteristic values if A is a symmetric matrix. Theorem : If A is an n n symmetric matrix, then all the eigenvalues of A are real numbers and its eigenvectors are real vectors. We will develop the theory of eigenvalues and eigenvectors only for symmetric matrices. Normalized Eigenvectors: Note that if x is an eigenvector corresponding to an eigenvalue ; then so is tx; where t is any non-zero scalar. So we normalize the eigenvectors. A normalized eigenvector is an eigenvector with (Euclidean) norm equal to. 6

7 Let x = (x ; x 2 ; :::; x n ) be an eigenvector with norm kxk = x 2 + x 2 2 + ::: + x 2 n - Since x is a non-zero vector, kxk > 0: Dene x y = kxk ; x 2 kxk ; :::; x n : kxk Now we have kyk = ; that is, y is a normalized eigenvector. 2 :

8 4. Spectral Decomposition of Symmetric Matrices (To proceed, we specialize further we consider symmetric matrices with distinct eigenvalues, that is, i 6= j :) Orthogonal Matrix: An nn matrix C is called an orthogonal matrix if C is invertible and its inverse equals its transpose, that is, C T = C : Theorem 2: Let A be an n n symmetric matrix with n distinct eigenvalues, ; 2 ; :::; n : If y ; y 2 ; :::; y n are (normalized) eigenvectors corresponding to the eigenvalues ; 2 ; :::; n ; then the matrix B such that B i ; the i-th column vector of B; is the vector y i (i = ; 2; :::; n), is an orthogonal matrix. Proof: To be discussed in class.

9 Hints: 3 steps: - Step. If i 6= j; then y i T y j = 0: Use the denition (equation ()) to prove that same time, y j T Ay i = j y j T y i : y j T Ay i = i y j T y i and, at the - Step 2. B is invertible. Show that the vectors y ; y 2 ; :::; y n are linearly independent. - Step 3. B = B T. Show directly that B T B = I (the identity matrix).

Theorem 3 (Spectral Decomposition): Let A be an n n symmetric matrix with n distinct eigenvalues, ; 2 ; :::; n : Let y ; y 2 ; :::; y n are (normalized) eigenvectors corresponding to the eigenvalues ; 2 ; :::; n : Let B be the n n matrix such that B i ; the i-th column vector of B; is the vector y i (i = ; 2; :::; n). Then A = BLB T where L is the diagonal matrix with the eigenvalues of A ( ; 2 ; :::; n ) on its diagonal. Remark: The above expression shows that matrix A can be decomposed into a matrix L consisting of its eigenvalues on the diagonal and the matrices B and B T which consists of its eigenvectors. Proof: To be discussed in class. 0

Hints: Consider any two n n matrices, C and D: - Let (C ; C 2 ; :::; C n ) be the set of row vectors of C and D ; D 2 ; :::; D n be the set of column vectors of D: - To prove Theorem 3 use the following observation on matrix multiplication: 0 C CD = B C 2 C @. A D D 2 D n C 4 = 0 C D C D 2 C D n B C 2 D C 2 D 2 C 2 D n C @...... A C n D C n D 2 C n D n = CD CD 2 CD n :

5. Quadratic Forms 2 A quadratic form on < n is a real-valued function of the form nx nx Q (x ; x 2 ; :::; x n ) = a ij x i x j i= j= = a x x + a 2 x x 2 + ::: + a n x x n +a 2 x 2 x + a 22 x 2 x 2 + ::: + a 2n x 2 x n +::: + a n x n x + a n2 x n x 2 + ::: + a nn x n x n : A quadratic form Q can be represented by a matrix A so that Q (x) = x T Ax where A nn = 0 B @ a a 2 a n a 2. a 22. a 2n.... a n a n2 a nn C A and x n = 0 B @ x x 2. x n C A :

3 Note that a ij and a ji are both coefcients of x i x j when i 6= j: The coefcient of x i x j is (a ij + a ji ) when i 6= j: If a ij 6= a ji ; we can uniquely dene new coefcients b ij = b ji = a ij + a ji ; for all i; j 2 so that b ij + b ji = a ij + a ji ; and B = (b ij ) = B T ; that is B is a symmetric matrix. This redenition of the coefcients does not change the value of Q for any x: Thus we can always assume that the matrix A associated with the quadratic form x T Ax is symmetric.

4 Examples: The general quadratic form in two variables is a x 2 + 2a 2 x x 2 + a 22 x 2 2: In matrix form this can be written as a a (x x 2 ) 2 a 2 a 22 x x 2 : The general quadratic form in three variables a x 2 + a 22 x 2 2 + a 33 x 2 3 + a 2 x x 2 + a 3 x x 3 + a 23 x 2 x 3 can be written as 0 (x x 2 x 3 ) B @ a 2 a 2 2 a 3 2 a 2 a 22 2 a 23 2 a 3 2 a 23 a 33 0 C B A @ x x 2 x 3 C A :

5 Deniteness of Quadratic Forms: Let A be a symmetric n n matrix. Then A is positive denite if x T Ax > 0 for all x in < n ; x 6= 0: A is negative denite if x T Ax < 0 for all x in < n ; x 6= 0: A is positive semi-denite if x T Ax 0 for all x in < n : A is negative semi-denite if x T Ax 0 for all x in < n : A is indenite if x T Ax > 0 for some x in < n and x T Ax < 0 for some other x in < n : Remarks: Consider, for example, the denition of positive deniteness. Note that the relevant inequality must hold for every vector x 6= 0 in < n : - Observation : If we know that A is positive denite, then we should be able to infer some useful properties of A quite easily.

#2. Prove that if a symmetric n n matrix A is positive denite then all its diagonal elements must be positive. - Observation 2: On the other hand, if we do not know that A is positive denite, then the above denition by itself will not be very easy to check to determine whether A is positive denite or not.! This observation leads one to explore convenient characterizations of quadratic forms. 6

6. Characterization of Quadratic Forms 7 Theorem 4: Let A be a symmetric n n matrix with distinct eigenvalues, ; 2 ; :::; n : Then (a) A is positive (negative) denite if and only if every eigenvalue of A is positive (negative); (b) A is positive (negative) semi-denite if and only if every eigenvalue of A is nonnegative (non-positive); (c) A is indenite if and only if A has a positive eigenvalue and a negative eigenvalue. Proof: To be discussed in class. Hints: Use the spectral decomposition of A given in Theorem 3. #3. Examples: Consider the following matrices: 0 A = ; B = 0 0 3 ; C = 0 0 0 Find out their eigenvalues, and characterize them (in terms of deniteness).

7. Alternative Characterization of Quadratic Forms 8 An alternative way to characterize quadratic forms is in terms of the signs of the principal minors of the corresponding matrix. If A is an n n matrix, a principal minor of order r is the determinant of the r r submatrix that remains when (n r) rows and (n r) columns with the same indices are deleted from A: Example: Consider the 3 3 matrix 0 A = @ a a 2 a 3 a 2 a 22 a 23 a 3 a 32 a 33 A : The principal minors of order 2 are: a a 2 a 2 a 22 ; a a 3 a 3 a 33 ; a 22 a 23 a 32 a 33 :

9 The principal minors of order are: The principal minors of order 3 is: jaj : a ; a 22 ; a 33 : If A is an n n matrix, the leading principal minor of order r is dened as a a r A r =..... a r a rr : Example: For the 3 3 matrix A, the three leading principal minors are A = a ; A 2 = a a 2 a 2 a 22 ; A 3 = jaj :

20 Theorem 5: Let A be a symmetric n n matrix. Then (a) A is positive denite if and only if all its n leading principal minors are positive. (b) A is negative denite if and only if all its n leading principal minors alternate in sign, starting with negative. (That is, the r-th leading principal minor, A r ; r = ; 2; :::; n; has the same sign as ( ) r :) (c) If some r-th leading principal minor of A (or some pair of them) is non-zero but does not t into either of the two sign patterns in (a) and (b), then A is indenite. (d) A is positive semi-denite if and only if every principal minor of A of every order is non-negative. (e) A is negative semi-denite if and only if every principal minor of A of odd order is non-positive and every principal minor of even order is non-negative. Proof: See section 6.4 (pages 393 395) of the textbook.

2 Examples: Consider the matrices studied above: 0 A = ; B = ; C = 0 0 3 0 0 0 #4. Characterize the matrices A; B and C (in terms of deniteness) using the principal minors criteria.

22 References Must read the following sections from the textbook: Section 23. (pages 579 585): Denitions and Examples of Eigenvalues and Eigenvectors; Section 23.3 (pages 597 60): Properties of Eigenvalues; Section 23.7 (pages 620 626): Symmetric Matrices; Sections 6. and 6.2 (pages 375 386): Quadratic Forms and Deniteness of Quadratic Forms; Section 23.8 (pages 626 627): Deniteness of Quadratic Forms. Most of the materials covered can be found in. Hohn, Franz E., Elementary Matrix Algebra, New Delhi: Amerind, 97 (chapters 9, 0), 2. Hadley, G., Linear Algebra, Massachusetts: Addison-Wesley, 964 (chapter 7).