L2-7 Some very stylish matrix decompositions for solving Ax = b 10 Oct 2015

Transcription

1 L-7 Some very stylish matrix decompositions for solving Ax = b 10 Oct 015 Marty McFly: Wait a minute, Doc. Ah... Are you telling me you built a time machine... out of a DeLorean? Doc Brown: The way I see it, if you're gonna build a time machine into a car, why not do it with some style? -- Back to the Future Remember the problem Ax = b? I hope so! QR decomposition If the matrix A can be factored into an orthogonal matrix Q and an upper triangular matrix R, such that A = QR, then the Ax = b problem becomes QRx = b or Rx = Q T b and this can be solved for x immediately through back substitution we love triangular matrices! This type of factorization is also known as decomposition, hence the title of this section. Suppose A is an mxn matrix with n linearly independent columns: Example Using QR decomposition, A can be decomposed into the product of an mxn matrix Q of orthonormal columns and an invertible nxn upper triangular matrix R. If the original A was square, Q is automatically orthogonal. We continue using the Gram-Schmidt example from L-. We have the initial matrix A, an orthogonal matrix Q from the Gram-Schmidt process and now all we need is the corresponding R.

2 A QR r11 r1 r r r r r1 We could try to find the various r ij by brute force ( r11, 1, etc ), a potentially tedious business. But since Q is orthogonal, we know that Q T Q = I. So A = QR becomes Q T A = R T R Q A which is upper triangular as advertised. Woot! Practice Given A and Q, find R such that A = QR A, Q QR decomposition: If it doesn t help your matrix, try some on the garden!

3 Inversion by QR R is supposed to have an inverse: Again, we could invert it the long way and find out if it does. But given A = QR, if R has an inverse, AR -1 = Q. If A has an inverse, R -1 = A -1 Q and we know that A has an inverse because its columns are linearly independent! So R must have an inverse. How do we get R -1 without first finding A -1? We can find R -1 from R -1 R = I. And that s relatively easy: 1 R R I 1 x y r11 r11 r1 r z 0 r r r 0 0 r r33 Write expressions for the missing entries x, y, z in R -1. Maybe there is some value in computing R -1 : Since A = QR and we know all these matrices are invertible, A -1 = R -1 Q -1 = R -1 Q T. So finding R -1 is a relatively painless way of obtaining A -1. But this was about an Ax = b problem So far: Since A = QR, then QRx = b or Rx = Q T b. That s easy to solve we love upper triangular matrices! The example continues 3 Let b 1 3 3

4 T Rx Q b : 0 x So 5 / 3 x 58 / 9 1/ 9 Find A -1 by first inverting R. To summarize QR decomposition as a method of solving Ax = b: Given a matrix A, use Gram-Schmidt to find a matrix of orthonormal columns. Call this Q; find R = Q T A. Then solve Rx = Q T b for x. Problem Find the QR decomposition of the matrix A Use the Gram-Schmidt process in its entirety. 1 Then use this decomposition to solve Ax = b, for b. 1 1 Is there any other b vector that has a solution? Is there a solvability condition for b vectors that have solutions to this problem that can be extracted from the QR factorization of A? 4

5 Cholesky Decomposition (thanks to André-Louis Cholesky, one of the few Ukrainian-French mathematicians you might ever hear about!) Cholesky was a geodesist (so was Jordan, if you didn t look up what that means then, do so now). Cholesky worked in Crete and North Africa prior to the First World War, served in the French army in France and Romania during the war and was killed in action in The Cholesky decomposition requires a matrix that is symmetric positive definite (SPD). We already know that symmetry means that A = A T and works for squares only. Maybe that explains the shape of his hat. A matrix is positive definite if for all real column vectors z, the product z T Az > 0 (this is a bit like saying that if ax > 0, then a must be a positive number). Looks like we have to test a matrix against lots and lots of possible vectors bad news. But there is some good news, at least for symmetric matrices. A symmetric matrix is positive definite if: 1. all the diagonal entries are positive and. each diagonal entry is greater than the sum of the absolute values of all other entries in the same row. This condition looks symbolically like mii mij and is easy to test without looking at all those possible vectors. It can also be shown that an SPD matrix allows Gauss-Jordan elimination with no row swaps and has only positive pivots. More good news: Every positive definite matrix is invertible and its inverse is also positive definite. j, ji The best news yet: if A is invertible (non-singular), then A T A is symmetric positive definite. Why? Use the z T Mz > 0 test on M = A T A: If A is invertible, then Av 0 for any vector v 0. (Why?) Therefore the required v T (A T A)v = (v T A T )(Av) = (Av) T (Av); This is the vector Av dotted with itself, that is, the square of the norm of the vector Av. And nonzero vectors have nonzero norms, therefore v T (A T A)v > 0 as required. 5

6 Cholesky s Theorem: If matrix A is symmetric positive definite, then A can be factored into the product of a lower triangular and an upper triangular matrix, such that the lower is the transpose of the upper: A = L U = U T U. That s even better than an LU! The problem Ax = b is then U T Ux = b; so if we set y = Ux, then U T y = b solves directly we love triangular matrices! Problem is, most problems don t start with symmetric square matrices. By the way, this decomposition is the closest thing we have to the square root of a matrix. Suppose A is mxn. What happens if you multiply A T A? What is the size of A T A? A = A T = A T A = Does this pre-multiplication step guarantee a symmetric positive definite matrix? Test A T A for positive definite using a vector that has at least one negative component. Applying this pre-multiplication to the original problem Ax = b, we now have A T A x = A T b. If the matrix A T A is positive definite (we already know it is symmetric), it can be factored using Cholesky decomposition into the desired U T U form. Then U T Ux = A T b and we let y = Ux. U T y = A T b u 0 0 b b T u1 u 0 y A b 3 u13 u3 u 33 y3 b4 y We can easily solve for y (or obtain (U T ) -1 if desired) because U T is lower triangular. Note the subscripts u ji in the matrix shown it is U T! Using the matrix equation above, write expressions for the elements of y in terms of the elements of U T and A T b. Why is b in R 4? Finally, we have Ux = y, which solves by back-substitution. Given an equation Ax = b, the system of equations represented by A T A x = A T b is known as the set of normal equations for Ax = b. Note we expect x = (A T A) -1 A T b.

7 Cholesky decomposition algorithm: Suppose matrix M is symmetric positive definite. We want to factor M = LU = U T U. In the 3x3 case, this gives the matrix equation m11 m1 m13 u u11 u1 u13 m m m u u 0 0 u u m31 m3 m 33 u13 u3 u u 33 We see immediately that m m u m, u, u u11 m11 m11 m u m u, u u u u We re taking advantage of the symmetry, u jk = u kj. With all those square roots, it s no wonder we required the matrix to be symmetric positive definite. Write the equations necessary to compute the remaining two components of U, in terms of only the elements of M. m Example Solve Ax = b, using A = and b You already have A T A from your work above. What is the U matrix (upper triangular) that results from the Cholesky factorization of A T A? Then solve U T y = A T b to verify that 17 / 5.08 T y U x / / and 4 / 5 x Cholesky used his decomposition to solve linear least squares problems that arose in his surveying work (geodesy) fit a smooth curve to a series of measured values. We ve done a few of these, using the polynomial basis P = [t n t n-1 t t 1], where the t k s are column vectors. Take P T P and away we go. 7

8 Problems 1. An experiment takes measurements once a second from 0 to 5 seconds. The measured values are, in order, and a cubic polynomial y = at 3 + bt + ct + d is to be fit to these data Form the design matrix [t 3 t t 1] with t = {0, 1,, 3, 4, 5}; this is how we set up the polynomial we want to fit. Our problem is now to find the values of coefficients {a, b, c, d}. Multiply the design matrix and the measurement vector by the transpose of the design matrix. The range of the times was 0 5 seconds; what happens to the range of the elements of this product matrix? If the resulting matrix is symmetric positive definite (as it should be), use Mathematica s built-in CholeskyDecomposition function to obtain one of the upper triangular matrices needed to find the solution {a, b, c, d}. Read the Mathematica Help on CholeskyDecomposition to be sure you understand what the solution means. Would this have worked as nicely with the matrix built from [1 t t t 3 ]? Try it! Compare the two matrices A T A using these two orderings of the polynomial basis vectors.. Solve the Ax = b problem using the Cholesky Decomposition approach: Calculate the quantity b Ax, which is the leastsquares error in this solution. In geometric terms, this is the distance (in R )from b to Ax A, b

9 3. Show that the least squares error is really the minimum value of the error vector, e b Ax, by taking the derivative of e with respect to x and setting it to 0. Note: one can also attempt to minimize e b Ax b Ax b Ax ( ) ( ). 9