Gaussian Elimination without/with Pivoting and Cholesky Decomposition

Gaussian Elimination without/with Pivoting and Cholesky Decomposition Gaussian Elimination WITHOUT pivoting Notation: For a matrix A R n n we define for k {,,n} the leading principal submatrix a a k A (k) := a k a kk We found out that Gaussian elimination without pivoting can fail even if the matrix A is nonsingular Example: For 4 2 2 2 3 2 2 2 we use l 2 = 2 4,l 3 = 2 4 and obtain after step of the elimination U = Now we have u 22 = and the algorithm fails in column 2 Note that a linear system with the matrix A (2) = [ 4 2 equivalent to a linear system with the matrix U (2) = 4 2 2 4 2 [ 4 2 2 This matrix is singular, hence the matrix A (2) is singular This explains why Gaussian elimination fails in column 2: The matrix A () = [4 was nonsingular, so elimination worked in column But the matrix A (2) is singular, so the elimination fails in column 2 For a matrix A R n n we consider the submatrices A (),,A (n) If all of these matrices are nonsingular, then Gaussian elimination WITHOUT pivoting succeeds, and we obtain an upper triangular matrix U with nonzero elements on the diagonal If one of these submatrices is singular: let A (k) be the first submatrix which is singular Then Gaussian elimination WITHOUT pivoting works in columns,,k, but fails in column k Theorem: Proof: For a matrix A R n n the following three statements are equivalent: All the submatrices A (),,A (n) are nonsingular 2 Gaussian elimination WITHOUT pivoting succeeds and yields u j j for j =,,n 3 The matrix A has a decomposition LU where L is lower triangular with s on the diagonal and U is upper triangular with nonzero diagonal elements () = (2): Assume Gaussian elimination fails in column k, yielding a matrix U with u kk = Then a linear system with the matrix A (k) is equivalent to a linear system with the matrix This matrix is singular, hence the matrix A (k) is singular U (k) = (2) = (3): The row operations given by the multipliers l i j turn the matrix A into the matrix U Hence reversing these row operations turns the matrix U back into the matrix A, yielding the equation LU (3) = () Let j {,,n} LU implies A ( j) = L ( j) U ( j) Hence solving a linear system with the matrix A ( j) is equivalent to solving a linear system with the matrix U ( j) Since U ( j) is nonsingular the matrix A ( j) is nonsingular is

Gaussian Elimination WITH Pivoting We now have for each column several pivot candidates: the diagonal element and all elements below it If one of the pivot candidates is nonzero we use a row interchange to move it to the diagonal position, and we can perform elimination in this column If all pivot candidates are zero the algorithm breaks down If we can perform elimination for columns,,n and then obtain u nn = we say that the algorithm breaks down in column n The algorithm succeeds if we can perform elimination in columns,,n and we obtain u nn In this case we obtain an upper triangular matrix U with nonzero diagonal elements If we start with a nonsingular matrix A, then Gaussian elimination with pivoting succeeds Solving a linear system Ax = b is then equivalent to a linear system Ux = y which we can solve by back substitution Theorem: Proof: For a matrix A R n n the following three statements are equivalent: The matrix A R n n is nonsingular 2 Gaussian elimination WITH pivoting succeeds and yields u j j for j =,,n 3 There exists a decomposition row p of A row p n of A = LU where L is lower triangular with s on the diagonal, U is upper triangular with nonzero diagonal elements, and p,, p n is a permutation of the numbers,,n () = (2): Assume Gaussian elimination fails in column k, yielding a matrix U where all pivot candidates u kk,,u nk are zero Since elimination succeeded in columns,,k the diagonal elements u,,u k,k are all nonzero: U = We want to show that the linear system Ax = has a solution x : The linear system Ax = is equivalent x k to the linear system Ux = Consider a vector x with x k =, x k+ = = x n =, ie, x = Then equations k,k +,,n are satisfied We can find x,,x k such that equations,,k are satisfied by using back substitution: x

equation k yields a unique x k (since u k,k ),, equation yields a unique x (since u ) We found a nonzero vector x such that Ax =, hence A is singular row p of A (2) = (3): Let à := row p n of A exactly the same L,U Hence we have à = LU Then applying Gaussian elimination WITHOUT pivoting to the matrix à yields (3) = () The linear system with the matrix Ax = b is equivalent to solving a linear system Ux = y Since U is nonsingular the matrix A is nonsingular Quadratic forms A symmetric matrix A R n n defines a quadratic function q(x,,x n ): q(x,x n ) = Such a function is called a quadratic form For x = n n i= j= a i j x i x j = x Ax we obviously have q(x) = What happens for x Examples: [ [ [ 2 2 2 2 3 2 2 2 3 2 q(x,x 2 ) = 2x 2 + 2x x 2 + 2x2 2 q(x,x 2 ) = 2x 2 + 4x x 2 + 2x2 2 q(x,x 2 ) = 2x 2 + 6x x 2 + 2x2 2? [ for x [ : q(x) is positive for x [ : q(x) is positive or zero for x : q(x) is positive or zero or negative eigenvalues of A are 3, eigenvalues of A are 4, eigenvalues of A are 5, If q(x) = x Ax > for all x we call the matrix A positive definite A symmetric matrix has real eigenvalues λ,,λ n and an orthonormal basis of eigenvectors v (),,v (n) If we write the vector x in terms of eigenvectors we obtain λ c x = c v () + + c n v (n), q(x) = [c,,c n = λ c 2 + + λ n c 2 n λ n c n We see: a symmetric matrix is positive definite all eigenvalues are positive

Cholesky decomposition Many application problems give a linear system with a symmetric matrix A R n n, ie, a i j = a ji for all i, j Example : Consider the matrix obtain LU: 6 5 6 5 6 5 6 5 = We perform Gaussian elimination WITHOUT pivoting and 2 3 2 3 3 3 2 We can then use this to solve a linear system Ax = b: First solve Ly = b using forward substitution, then solve Ux = b using back substitution We would like to exploit the fact that A is symmetric and use a decomposition which reflects this: we want to find an upper triangular matrix C R n n with C C 6 5 6 5 = = c c 2 c 22 c 3 c 23 c 33 3 2 2 3 2 c c 2 c 3 c 22 c 23 c 33 3 2 2 3 2 Note: this decomposition C C is called Cholesky decomposition C C implies A = C C = A, ie, such a decomposition can only be obtained for symmetric A We can then use this decomposition to solve a linear system Ax = b: First solve C y = b using forward substitution, then solve Cx = y using back substitution How can we find the matrix C? row of A: diagonal element: a = c c = c = a for k = 2,n: a k = c k c = c k = a k /c row 2 of A: diagonal element: a 22 = c 2 2 + c2 22 = c 22 = a 22 c 2 2 for k = 3,,n: a 2k = c k c 2 + c 2k c 22 = c 2k = (a 2k c k c 2 )/c 22 Algorithm: For j =,,n: s := a j j j l= c2 l j If s : stop with error matrix not positive definite c j j = s For k = j +,,n: ( c jk = a jk c l j c lk )/c j j j l=

How to compute row j = 3 of C: C = v M v v ( v M)/c j j Here v = C(:j-,j) and M = C(:j-,j+:n) We obtain the following Matlab code: function C = cholesky(a) n = size(a,); C = zeros(n,n); for j=:n v = C(:j-,j); s = A(j,j) - v *v; if s<= error( matrix is not positive definite ) end C(j,j) = sqrt(s); C(j,j+:n) = (A(j,j+:n)-v *C(:j-,j+:n))/C(j,j); end Work of Cholesky decomposition: For row j: compute c j j : ( j ) multiplications and square root compute c jk for k = j +,,n: (n j) times (( j ) multiplications and division ) If we count multiplications, divisions and square roots we have j + (n j) j = (n j + ) j operations for row j The total is therefore n j= (n j + ) j = = n3 6 + O(n2 ) operations This is half the work of the LU decomposition which was n3 3 operations Example 2: Consider the matrix n(n + )(n + 2) 6 6 5 6 2 The algorithm breaks down when we try to compute c 33 : s = 2 (2 2 + 3 2 ) = : C C = 3 2 2 3 3 2 2 3 = 6 5 6 3 Therefore we have s = 2 3 =, and the Cholesky decomposition does not exist The matrix A is symmetric and nonsingular Why is there no Cholesky decomposition? Assme that a matrix A has a Cholesky decomposition C C where C has nonzero elements For a vector x R n we consider the scalar x Ax = n i, j= a i jx i x j (this is called a quadratic form ) For any vector x we have y := Cx since C is nonsingular (upper triangular with nonzero diagonal entries) Hence the quadratic form is positive: x Ax = x C Cx = y y > ()

Definition: A symmetric matrix A R n n is called positive definite if for all x [ [ 2 Eg, the matrix is positive definite since x 3 Ax = 2x 2 + 3x2 2 > for x [ [ 2 On the other hand the matrix is not positive definite since [, A we have x Ax > = Note: A symmetric matrix A has real eigenvalues The matrix is positive definite if and only if all eigenvalues are positive We can check whether a matrix is positive definite by trying to find the Cholesky decomposition If C with nonzero diagonal elements exists, the matrix is positive definite because of () If the algorithm breaks down we claim that the matrix is not positive definite Eg, the matrix A from Example 2 is not positive definite: Pick x 3 = Solve Cx = [,, using back substitution: x = 8/3 3, x Ax = x C Cx + x x = Theorem: For a symmetric matrix A R n n the following three statements are equivalent: The matrix is positive definite 2 The Cholesky algorithm succeeds and gives C R n n with nonzero diagonal elements 3 There exists a decomposition C C where C R n n is upper triangular with nonzero diagonal elements Proof: () = (2): Assume the algorithm breaks down in row j with s We want to show that A is not positive definite by finding a nonzero vector x with x Ax : Let, eg, j = 3 We have already computed the first two rows of C with c, c 22 and have s = a 33 (c 2 3 + c2 23 ) C(3) C (3) = Now we construct a vector x = c c 2 c 22 c 3 c 23 A (3) = C(3) C (3) + x x 2 with s c c 2 c 3 c 22 c 23 c c 2 c 3 c 22 c 23 } {{ } C (3) x x 2 = = a a 2 a 3 a 2 a 22 a 23 a 3 a 3 c 2 3 + c2 23 then x This works since c,c 22 Hence x A (3) x = x C(3) C (3) x + x x = + s }{{} s : We use back substitution to find x 2 and

x x 2 Now we define x = and obtain x Ax = x A (3) x = s (2) = (3): The algorithm gives C with C C = A (3) = (): as shown above in ()