Linear Algebra, part 2 Eigenvalues, eigenvectors and least squares solutions Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2013 Main problem of linear algebra 2: Given an n n matrix A, find eigenvector(s) and eigenvalue(s) x and such that Ax = x. Rewrite the equation as (A I)x = 0. This equation can only have a non-trivial solution if the matrix (A I) is singular. The number is an eigenvalue of A if and only if det(a I) = 0. det(a I) is a polynomial in of degree n. Will have n roots, 1,..., n. Can have multiplicity larger than one, and can have complex eigenvalues, even if A is real. Can compute eigenvalues by hand using this approach for small systems. Other approaches needed for computer algorithms for large systems. Not the focus here.
Facts about eigenvalues The product of the n eigenvalues equals the determinant of A: det(a) =( 1 )...( n ). The sum of the eigenvalues equals the trace (sum of diagonal entries) of A: 1 + 2 + + n = a 11 + a 22 + + a nn. If A is triangular, then its eigenvalues lie along the main diagonal. The eigenvalues of A 2 are 2 1,..., 2 n. The eigenvalues of A 1 are 1/ 1,...,1/ n. Eigenvectors of A are also eigenvectors of A 2 and A 1 (and any function of A). Eigenvalues of A + B and AB are in general not known from eigenvalues of A and B, exceptforthespecialcasewhena and B commute, i.e. when AB = BA. Linear di erential equations with constant coe cients The simple equation dy dt = ay has the general solution y(t) =Ce at. (Initial condition det. C). Consider a system (u is n 1 vector, A is n n): du dt = Au. Let x i be an eigenvector of A with corresponding eigenvalue define u i = e i t x i.wethenhave i, and Au i = A(e i t x i )=e i t Ax i = e i t ix i = i u i which is equal to du i dt = d dt (e i t x i )= i e i t x i = i u i. The general solution is u(t) =C 1 e 1t x 1 + + C n e nt x n
Growth and decay For the scalar equation, solution u(0)e at decays if a < 0, grows if a > 0. If a is complex, the real part of a determines the growth or decay. For the system, the that will decay. is determine which modes that will grow and which Example 1 - A 2 2 system with real eigenvalues. Example 2 - Rigid body rotation: complex eigenvalues. Diagonalization of a matrix Suppose that the n n matrix A has n linearly independent eigenvectors x 1,...,x n.formamatrixs, whose columns are the eigenvectors of A. Then, S 1 AS = is diagonal. The diagonal entries of are the eigenvalues 1,..., n. If A has no repeated eigenvalues, i.e. all is aredistinct,thena has n linearly independent eigenvectors, and A is diagonalizable. If one or more eigenvalues of A have multiplicity larger than one A might have or might not have a full set of linearly independent eigenvectors. If A is symmetric, all eigenvalues are real and it has a full set of orthonormal eigenvectors. Denote by Q the orthogonal matrix whose columns are eigenvectors of A. (Defined on p 54 in Strang). We have that Q 1 = Q T and Q T AQ =. [WORKSHEET]
Vector and Matrix norms, Quadratic forms Euclidean norm of x: kxk 2 = P n k=1 x k 2 1/2 = p x T x. 2-norm of matrix defined as : kak 2 =max kaxk 2 kxk 2 =max Consider the Rayleigh quotient: R K (x) =(x T Kx)/(x T x). q x T Ax x T x. Di erentiating R K (x) withrespecttox k (DO IT!), can show that = 0, k =1,...,n if and only if Kx = R K (x)x. @R @x k Theorem: If Kx = x,thenx is a stationary point of R K (x) and = R K (x ). Hence, the Rayleigh quotient is maximized by the largest eigenvalue of K. Denote the largest eigenvalue of A T A by M. We get kak 2 = p M. This is well defined, since A T A is SPD and all eigenvalues are positive. Positive definite matrices and minimum principles A symmetric matrix A is positive definite (SPD, introduced last time) if x T Ax > 0 for all non-zeros vectors x. We know that a matrix is SPD if it is symmetric and all pivots are positive, or equivalently, that all eigenvalues are positive. The solution x to Ax = b can also be viewed as a solution to the following minimization problem: If A is positive definite, then the quadratic P(x) = 1 2 xt Ax is minimized at the point where Ax = b. The minimum value is P(A 1 b)= 1 2 bt A 1 b. x T b Proof: Example by calculus
Least squares solution Consider again Ax = b, wherea is m n with m > n. (x is n 1, b is m 1. Example: Find the line y = c + dt that passes through four given points (t 1, y 1 ),...(t 4, y 4 ). If the four points all fall on a straight line, this overdetermined system has a solution. Otherwise, we want to find the straight line that best fit the data points. Normal equations The vector x that minimizes kax bk 2 is the solution to the normal equations A T Ax = A T b. This vector x =(A T A) 1 A T b is the least squares solution to Ax = b. Proof: kax bk 2 =(Ax b) T (Ax b) =((Ax) T b T )(Ax b) = = x T A T Ax x T A T b b T Ax + b T b. b T Ax is a scalar, and can be transposed: b T Ax =(b T Ax) T = x T A T b. The term b T b is constant, and does not a ect the minimization. So, the form to minimize (scaled by a factor of 2, which is arbitrary) is P(x) = 1 2 xt A T Ax x T A T b. According to earlier thm, this quadratic form is minimized for x which is a solution to A T Ax = A T b. [NOTES CONTD.]