Math 44 - Matrix Algebra Review notes - (Alberto Bressan, Spring 7) sec: Orthogonal diagonalization of symmetric matrices When we seek to diagonalize a general n n matrix A, two difficulties may arise: (i) Even if all the entries of A are real numbers, the eigenvalues and eigenvectors of A may be complex valued, (ii) If A has multiple eigenvalues, we are not guaranteed to find a basis of eigenvectors When A is symmetric, these two bad situations (i)-(ii) do not occur: Theorem Let A be an n n symmetric matrix (with real coefficients) Then All the eigenvalues are real Eigenvectors corresponding to distinct eigenvalues are perpendicular to each other There exists an orthonormal basis of R n consisting of eigenvectors of A v vn If A is symmetric, we can thus find an orthogonal matrix Q = v, whose columns are eigenvectors of A, and such that λ λ Q AQ = D = λ n is diagonal Its diagonal elements are precisely the eigenvalues of A Example Consider the symmetric matrix A = 3 Computing the roots of the polynomial p(λ) = det(a λi) = ( λ(3 λ) 4 ) (4 λ), one finds the 4 eigenvalues λ =, λ = λ 3 = 4
In connection with the eigenvalue λ =, we find the eigenvector v = For the eigenvalue λ = 4, we find the two eigenvectors v =, v 3 = By normalizing these three eigenvectors, we obtain a matrix Q whose columns form an orthonormal basis of R 3, namely / / Q = / / A direct computation yields Q T Q = I, while / / Q T AQ = / / / / 3 / / = 4 = D 4 4 sec: Applications to quadratic forms Consider a quadratic form, that is: a homogeneous polynomial of degree in the variables x,, x n q(x) = a ij x i x j = x T Ax () QF i,i= here A = (a ij ) i,j=,,n We can assume that a ij = a ij, so that the matrix A is symmetric Otherwise, we replace both a ij and a ji with the average value a ij+a ji By the previous theorem, there exists an orthogonal matrix Q such that D = Q AQ is diagonal Its diagonal elements are precisely the eigenvalues of A Making the change of variables x = Qy, y = Q T x, () Q one obtains q(x) = x T Ax = (Qy) T A(Qy) = y T Q T AQy = y T Dy Theorem Given the quadratic form ( QF ), one can always make a change of variables x = Qy so that, in terms of the new variables y = (y,, y n ), the quadratic form is diagonal, namely q(x) = y T Dy = λ k yk (3) Q3 k= Example Consider the quadratic form q(x) = 3x + 4x x + 6x = x x 3 6 x x The matrix A is symmetric Its eigenvalues and (normalized) eigenvectors are / / λ = 7, v = /, λ =, v = /
Introducing the orthogonal matrix Q = v v = D = 7 / / / / = Q AQ, we obtain We can now perform the change of variable x = Qy, so that x = y = y y x y y + y (4) R In terms of the new variables, the quadratic form becomes q(x) = 3x + 4x x + 6x = = ( 3y + y ) = 7y + y (3(y y ) + 4(y y )(y + y ) + 6(y + y ) ) Notice that the change of variables ( 4) R represents a rotation of the coordinate axis See Fig f:la x y y θ x Figure : The change of coordinates in Example corresponds to a rotation of the axes by an angle θ, with cos θ = / f:la Positive definite matrices A quadratic form q(x) = x T Ax is positive semidefinite if x T Ax for every x = (x, x,, x n ) In this case we also say that the symmetric matrix A is positive semidefinite A quadratic form q(x) = x T Ax is positive definite if x T Ax > for every x = (x, x,, x n ) (,,, ) In this case we also say that the symmetric matrix A is positive definite A quadratic form q(x) = x T Ax is indefinite if it takes both positive and negative values 3
Example 3 The quadratic form q(x) = x x x + x is positive definite Indeed, we can write q(x) = ( x + x + (x x ) ) > whenever (x, x ) (, ) On the other hand, the quadratic form q(x) = x + 3x x + x is indefinite Indeed taking x = (x, x ) = (, ) we obtain q(x) =, while taking x = (x, x ) = (, ) we obtain q(x) = q(x) q(x) x x x x q(x) x x Figure : Left: a positive definite quadratic form Center: a positive semidefinite quadratic form Right: a quadratic form which is indefinite f:la3 To decide whether the quadratic form q(x) = x T Ax is positive definite, it suffices to find the eigenvalues of the matrix A Theorem Consider the quadratic form q(x) = x T Ax (i) If all eigenvalues of A are, then the quadratic form q(x) is positive semidefinite (ii) If all eigenvalues of A are >, then the quadratic form q(x) is positive definite (ii) If some eigenvalues of A are > and other eigenvalues are <, then the quadratic form q(x) is indefinite Indeed, let λ, λ,, λ n be the eigenvalues of A Performing the change of variables x = Qy, as in ( )-( Q 3) Q3 we obtain q(x) = y T Dy = λ k yk () Q4 If λ k for every k, then the right hand side of ( ) Q4 is for every y = (y,, y n ) If λ k > for every k, then the right hand side of ( ) Q4 is strictly positive for every y = (y,, y n ) (,, ) If some eigenvalues are positive and other are negative, then the right hand side of ( Q4 ) can be sometimes positive, sometimes negative, depending on the choice of (y, y,, y n ) k= Example 3 (continued) The quadratic form q(x) = x x x + x = x T Ax where A = / / is positive definite Indeed, the matrix A has eigenvalues λ =, λ = 3, which are both strictly positive 4
On the other hand, the quadratic form q(x) = x + 3x x + x = x T Bx where B = 3/ 3/ is indefinite Indeed, the matrix B has eigenvalues λ =, λ = One of these is positive, the other is negative Here is another way to check whether an n n symmetric matrix A is positive definite As in Fig 3, f:la4 for every k =,,, n, let A k be the k k submatrix containing the elements in the first k rows and k columns of A These are called the principal submatrices of A A = a a a 3 a a a 3 a a a 3 3 33 a n a n a n a n3 a nn Figure 3: The principal submatrices of the n n matrix A = (a ij ) f:la4 Theorem (i) A symmetric matrix A is positive semidefinite if and only if all the principal submatrices A k have determinant (ii) A symmetric matrix A is positive definite if and only if all the principal submatrices A k have determinant > Example Consider the matrix A = submatrices: det =, det 3 3 4 4 6 We compute the determinants of the principal =, det 3 4 4 6 We conclude that the matrix A is positive semidefinite, but not positive definite =