Solving Linear Systems of Equations

Transcription

1 Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department These slides are a supplement to the book Numerical Methods with Matlab: Implementations and Applications, by Gerald W. Recktenwald, c, Prentice-Hall, Upper Saddle River, NJ. These slides are copyright c Gerald W. Recktenwald. The PDF version of these slides may be downloaded or stored or printed only for noncommercial, educational use. The repackaging or sale of these slides in any form, without written consent of the author, is prohibited. The latest version of this PDF file, along with other supplemental material for the book, can be found at or web.cecs.pdx.edu/~gerry/nmm/. Version.88 August, page 1 Primary Topics Basic Concepts Basic Concepts Gaussian Elimination Limitations on Numerical Solutions to Ax = b Factorization Methods Nonlinear Systems of Equations Matrix Formulation Requirements for a Solution Consistency The Role of rank(a) Formal Solution when A is n n NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

2 Pump Curve Model (1) Pump Curve Model () Objective: Find the coefficients of the quadratic equation that approximates the pump curve data. Head (m) Flow rate (m /s) x 1 Model equation: h = c 1 q + c q + c Write the model equation for three points on the curve. This gives three equations for the three unknowns c 1, c, and c. Points from the pump curve: q (m /s) h (m) Substitute each pair of data points into the model equation 11 = c c + c, 11 = 1 8 c c + c, 9. = c c + c, Rewrite in matrix form as c c = c 9. NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Pump Curve Model () Pump Curve Model () Using more compact symbolic notation where Ax = b A = , c 1 11 x = c, b = 11. c 9. In general, for any three (q, h) pairs the system is still Ax = b with q 1 q 1 1 c 1 h 1 A = q q 1, x = c, b = h. q q 1 c h NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

3 Matrix Formulation Thermal Model of an IC Package (1) Recommended Procedure 1. Write the equations in natural form.. Identify unknowns, and order them.. Isolate the unknowns.. Write equations in matrix form. Objective: Find the temperature of an integrated circuit (IC) package mounted on a heat spreader. The system of equations is obtained from a thermal resistive network model. Physical Model: air flow Mathematical Model: Q c Τ c Q 1 Τ p Q R 1 R Τ w T p Tw Q R Q R Q R ambient air at T a Τ a Τ a Τ a NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9 Thermal Model of an IC Package () Thermal Model of an IC Package () 1. Write the equations in natural form: Use resistive model of heat flow between nodes to get Q 1 = 1 (T c T p ) R 1 Q = 1 (T p T w ) R Q = 1 (T c T a ) R Q = 1 (T p T a ) R Q = 1 (T w T a ) R Q c = Q 1 + Q Q 1 = Q + Q. Identify unknowns, and order them: Unknowns are Q 1, Q, Q, Q, T c, T p, and T w. Thus, Q 1 Q Q x = Q. T c T p T w. Isolate the Unknowns R 1 Q 1 T c + T p =, R Q T p + T w =, R Q T c = T a, R Q T p = T a, R Q T w = T a, Q 1 + Q = Q c, Q 1 Q Q =. NMM: Solving Systems of Equations page 1 NMM: Solving Systems of Equations page 11

4 Thermal Model of an IC Package () Requirements for a Solution. Write in Matrix Form R 1 1 R 1 R R R Q 1 Q Q Q T c T p T w T a = T a T a Q c 1. Consistency. The Role of rank(a). Formal Solution when A is n n. Summary Note: The coefficient matrix has many more zeros than non-zeros. This is an example of a sparse matrix. NMM: Solving Systems of Equations page 1 NMM: Solving Systems of Equations page 1 Consistency (1) Consistency () If an exact solution to Ax = b exists, b must lie in the column space of A. Ifitdoes, then the system is said to be consistent. If the system is consistent, an exact solution exists. rank(a) gives the number of linearly independent columns in A [A b] is the augmented matrix formed by combining the b vector with the columns of A. a 11 a 1 a 1,n b 1 [A b]= a 1 a a,n.. b.... a m,1 a m, a n,n b n If rank([a b]) > rank(a) then b does not lie in the column space of A. Inother words, since [A b] has a larger set of basis vectors than A, and since the difference in the size of the basis set is solely due to the b vector, the b vector cannot be constructed from the column vectors of A. NMM: Solving Systems of Equations page 1 NMM: Solving Systems of Equations page 1

5 Role of rank(a) Summary of Solution to Ax = b where A is m n If A is m n, and z is an n-element column vector, then Az =has a nontrivial solution only if the columns of A are linearly dependent 1. In other words, the only solution to Az =is z =when A is full rank. Given the m n matrix A, the system Ax = b has a unique solution if the system is consistent and if rank(a) =n. For the general case where A is m n and m n, If rank(a) =n and the system is consistent, the solution exists and it is unique. If rank(a) =n and the system is inconsistent, no solution exists. If rank(a) <nand the system is consistent, an infinite number of solutions exist. If A is n n and rank(a) =n, then the system is consistent and the solution is unique. 1 isthem-element column vector filled with zeros NMM: Solving Systems of Equations page 1 NMM: Solving Systems of Equations page 1 Formal Solution when A is n n Formal Solution when A is n n The formal solution to Ax = b is x = A b where A is n n. If A exists then A is said to be nonsingular. If A does not exist then A issaidtobesingular. If A exists then but Ax = b = x = A b Do not compute the solution to Ax = b by finding A, and then multiplying b by A! We see: We do: x = A b Solve Ax = b by Gaussian elimination or an equivalent algorithm NMM: Solving Systems of Equations page 18 NMM: Solving Systems of Equations page 19

6 Singularity of A Summary of Requirements for Solution of Ax = b If an n n matrix, A, issingular then the columns of A are linearly dependent therowsofa are linearly dependent rank(a) <n det(a) = A does not exist a solution to Ax = b may not exist If a solution to Ax = b exists, it is not unique Given the n n matrix A and the n 1 vector, b the solution to Ax = b exists and is unique for any b if and only if rank(a) =n. rank(a) =n automatically guarantees that the system is consistent. NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1 Gaussian Elimination Solving Diagonal Systems (1) Solving Diagonal Systems Solving Triangular Systems Gaussian Elimination Without Pivoting Hand Calculations Cartoon Version The Algorithm Gaussian Elimination with Pivoting Row or Column Interchanges, or Both Implementation Solving Systems with the Backslash Operator The system defined by 1 A = b = NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

7 Solving Diagonal Systems (1) Solving Diagonal Systems (1) The system defined by 1 A = b = The system defined by 1 A = b = is equivalent to x 1 = x = x = is equivalent to The solution is x 1 = x = x = x 1 = x = = x = = NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Solving Diagonal Systems () Triangular Systems (1) Algorithm 8.1 given A, b for i =1...n x i = b i /a i,i end In Matlab: >> A =... % A is a diagonal matrix >> b =... >> x = b./diag(a) This is the only place where element-by-element division (.*) has anything to do with solving linear systems of equations. The generic lower and upper triangular matrices are and The triangular systems l 11 L = l 1 l..... l n1 l nn u 11 u 1 u 1n U = u u n..... u nn Ly = b Ux = c are easily solved by forward substitution and backward substitution, respectively NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

8 Solving Triangular Systems () Solving Triangular Systems () 1 A = b = 9 8 is equivalent to 1 A = b = 9 8 x 1 + x + x = 9 x + x = x = 8 NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9 Solving Triangular Systems () Solving Triangular Systems () is equivalent to 1 A = b = 9 8 x 1 + x + x = 9 x + x = x = 8 is equivalent to 1 A = b = 9 8 x 1 + x + x = 9 x + x = x = 8 Solve in backward order (last equation is solved first) x = 8 = Solve in backward order (last equation is solved first) x = 8 = x = 1 ( +x )= =1 NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1

9 Solving Triangular Systems () Solving Triangular Systems () is equivalent to 1 A = b = 9 8 x 1 + x + x = 9 x + x = x = 8 Solve in backward order (last equation is solved first) x = 8 = x = 1 ( +x )= =1 x 1 = 1 (9 x x )= = Solving for x n,x n,...,x 1 for an upper triangular system is called backward substitution. Algorithm 8. given U, b x n = b n /u nn for i = n s = b i for j = i +1...n s = s u i,j x j end x i = s/u i,i end NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Solving Triangular Systems (8) Gaussian Elimination Solving for x 1,x,...,x n for a lower triangular system is called forward substitution. Algorithm 8. given L, b x 1 = b 1 /l 11 for i =...n s = b i for j =1...i 1 s = s l i,j x j end x i = s/l i,i end Goal is to transform an arbitrary, square system into the equivalent upper triangular system so that it may be easily solved with backward substitution. The formal solution to Ax = b, wherea is an n n matrix is In Matlab: >> A =... >> b =... >> x = A\b x = A b Using forward or backward substitution is sometimes referred to as performing a triangular solve. NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

10 Gaussian Elimination Hand Calculations (1) Gaussian Elimination Hand Calculations () Solve The elimination phase transforms the matrix and right hand side to an equivalent system x 1 +x = x 1 +x = x 1 +x = x 1 +x = x 1 +x = x = Subtract times the first equation from the second equation x 1 +x = x = This equation is now in triangular form, and can be solved by backward substitution. The two systems have the same solution. The right hand system is upper triangular. Solve the second equation for x x = = Substitute the newly found value of x into the first equation and solve for x 1. x 1 = ()() = NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Gaussian Elimination Hand Calculations () Gaussian Elimination Hand Calculations () When performing Gaussian Elimination by hand, we can avoid copying the x i by using a shorthand notation. For example, to solve: A = b = Form the augmented system à =[A b]= Add times row 1 to row, and add (1 times) row 1 to row à (1) = Subtract (1 times) row from row à () = 9 9 The vertical bar inside the augmented matrix is just a reminder that the last column is the b vector. NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9

11 Gaussian Elimination Hand Calculations () The transformed system is now in upper triangular form à () = 9 Solve by back substitution to get x = = x = 1 ( 9 x )= x 1 = 1 ( x + x )= Gaussian Elimination Cartoon Version (1) Start with the augmented system The xs represent numbers, they are not necessarily the same values. Begin elimination using the first row as the pivot row and the first element of the first row as the pivot element NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1 Gaussian Elimination Cartoon Version () Eliminate elements under the pivot element in the first column. x indicates a value that has been changed once. x x x x x x x x x x x x x x x x x x x x x x x x Gaussian Elimination Cartoon Version () The pivot element is now the diagonal element in the second row. Eliminate elements under the pivot element in the second column. x indicates a value that has been changed twice. x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

12 Gaussian Elimination Cartoon Version () Gaussian Elimination Cartoon Version () The pivot element is now the diagonal element in the third row. Eliminate elements under the pivot element in the third column. x indicates a value that has been changed three times. x x x x x x x x x x x x x x x x x x x Summary Gaussian Elimination is an orderly process of transforming an augmented matrix into an equivalent upper triangular form. The elimination operation is Elimination requires three nested loops. ã kj =ã kj (ã ki /ã ii )ã ij The result of the elimination phase is represented by the image below. x x x x x x x x x NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Gaussian Elimination Algorithm The Need for Pivoting (1) Algorithm 8. form à =[A b] for i =1...n 1 for k = i +1...n for j = i...n+1 ã kj =ã kj (ã ki /ã ii )ã ij end end end GEshow: The GEshow function in the NMM toolbox uses Gaussian elimination to solve a system of equations. GEshow is intended for demonstration purposes only. Solve: A = 1 8 b = 1 Note that there is nothing wrong with this system. A is full rank. The solution exists and is unique. Form the augmented system NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

13 The Need for Pivoting () The Need for Pivoting () Subtract 1/ times the first row from the second row, add / times the first row to the third row, add 1/ times the first row to the fourth row. The result of these operations is: 1 The next stage of Gaussian elimination will not work because there is a zero in the pivot location, ã. Swap second and fourth rows of the augmented matrix. 1 Continue with elimination: subtract (1 times) row from row. NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9 The Need for Pivoting () Pivoting Strategies Another zero has appear in the pivot position. Swap row and row. The augmented system is now ready for backward substitution. Partial Pivoting: Exchange only rows Exchanging rows does not affect the order of the x i For increased numerical stability, make sure the largest possible pivot element is used. This requires searching in the partial column below the pivot element. Partial pivoting is usually sufficient. NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1

14 Partial Pivoting Pivoting Strategies () To avoid division by zero, swap the row having the zero pivot with one of the rows below it. Full (or Complete) Pivoting: Exchange both rows and columns * Rows completed in forward elimination. Row with zero pivot element Rows to search for a more favorable pivot element. Column exchange requires changing the order of the x i For increased numerical stability, make sure the largest possible pivot element is used. This requires searching in the pivot row, and in all rows below the pivot row, starting the pivot column. Full pivoting is less susceptible to roundoff, but the increase in stability comes at a cost of more complex programming (not a problem if you use a library routine) and an increase in work associated with searching and data movement. To minimize the effect of roundoff, always choose the row that puts the largest pivot element on the diagonal, i.e., find i p such that a ip,i = max( a k,i ) for k = i,...,n NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Full Pivoting Gauss Elimination with Partial Pivoting * Columns to search for a more favorable pivot element. * Rows completed in forward elimination. Row with zero pivot element Rows to search for a more favorable pivot element. Algorithm 8. form à =[A b] for i =1...n 1 find i p such that max( ã ipi ) max( ã ki ) for k = i...n exchange row i p with row i for k = i +1...n for j = i...n+1 ã k,j =ã k,j (ã k,i /ã i,i )ã i,j end end end NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

15 Gauss Elimination with Partial Pivoting The Backslash Operator (1) GEshow: The GEshow function in the NMM toolbox uses naive Gaussian elimination without pivoting to solve a system of equations. GEpivshow: The GEpivshow function in the NMM toolbox uses Gaussian elimination with partial pivoting to solve a system of equations. GEpivshow is intended for demonstration purposes only. GEshow and GEpivshow are for demonstration purposes only. They are also a convenient way to check your hand calculations. Consider the scalar equation x = = x =() The extension to a system of equations is, of course Ax = b = x = A b where A b is the formal solution to Ax = b In Matlab notation the system is solved with x = A\b NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page The Backslash Operator () Limits on Numerical Solution to Ax = b Given an n n matrix A, and an n 1 vector b the \ operator performs a sequence of tests on the A matrix. Matlab attempts to solve the system with the method that gives the least roundoff and the fewest operations. When A is an n n matrix: 1. Matlab examines A to see if it is a permutation of a triangular system If so, the appropriate triangular solve is used. Machine Limitations RAM requirements grow as O(n ) flop count grows as O(n ) The time for data movement is an important speed bottleneck on modern systems. Matlab examines A to see if it appears to be symmetric and positive definite. If so, Matlab attempts a Cholesky factorization and two triangular solves.. If the Cholesky factorization fails, or if A does not appear to be symmetric, Matlab attempts an LU factorization and two triangular solves. NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9

16 cache FPU internal bus CPU system bus RAM Data Access Time: shortest Limits on Numerical Solution to Ax = b Limits of Floating Point Arithmetic Exact singularity Effect of perturbations to b Effect of perturbations to A The condition number Stand alone computer Disk network Computer Computer Computer longest NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1 Geometric Interpretation of Singularity (1) Geometric Interpretation of Singularity () Consider a system describing two lines that intersect The matrix form of this equation is» 1 / 1 y = x + y = 1 x +1» x1 x =» 1 The equations for two parallel but not intersecting lines are»»» 1 x1 = 1 x The equations for two parallel and coincident lines are» 1 1» x1 x The equations for two nearly parallel lines are» 1 +δ 1» x1 x =»» = +δ Here the coefficient matrix is singular (rank(a) =1), and the system is inconsistent NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

17 Geometric Interpretation of Singularity () Effect of Perturbations to b 8 A and b are consistent A is nonsingular 1 8 A and b are consistent A is singular 1 8 A and b are inconsistent A is singular 1 8 A and b are consistent A is ill conditioned 1 Consider the solution of a system where One expects that the exact solutions to» 1 Ax = /» 1 b = /» 1 and Ax =. will be different. Should these solutions be a lot different or a little different? NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Perturb b with δb such that The perturbed system is The perturbations satisfy Effect of Perturbations to b δb b 1, A(x + δx b )=b + δb Aδx b = δb Analysis shows (see next two slides for proof) that δx b x A A δb b Thus, the effect of the perturbation is small only if A A is small. Effect of Perturbations to b (Proof) Let x + δx b be the exact solution to the perturbed system A(x + δx b )=b + δb (1) Expand Ax + Aδx b = b + δb Subtract Ax from left side and b from right side since Ax = b Aδx b = δb Left multiply by A δx b = A δb () δx b x 1 only if A A 1 NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

18 Effect of Perturbations to b (Proof, p. ) Effect of Perturbations to b (Proof) Take norm of equation () δx b = A δb Applying consistency requirement of matrix norms δx A δb () Similarly, Ax = b gives b = Ax, and b A x () Multiply Equation () by Equation () to get δx b x A A δb b Summary: If x + δx b is the exact solution to the perturbed system A(x + δx b )=b + δb () Rearrangement of equation () yields 1 x A b () then δx b x A A δb b NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9 Effect of Perturbations to A Effect of Perturbations to both A and b Perturb both A with δa and b with δb such that Perturb A with δa such that The perturbed system is Analysis shows that δa A 1, (A + δa)(x + δx A )=b δx A x + δx A A A δa A Thus, the effect of the perturbation is small only if A A is small. δx A x + δx A 1 only if A A 1 The perturbation satisfies Analysis shows that δa δb A 1 and b 1 (A + δa)(x + δx) =b + δb δx x + δx A A 1 A A δa A» δa A + δb b Thus, the effect of the perturbation is small only if A A is small. δx x + δx 1 only if A A 1 NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page 1

19 Condition number of A Computational Stability The condition number κ(a) A A indicates the sensitivity of the solution to perturbations in A and b. The condition number can be measured with any p-norm. The condition number is always in the range 1 κ(a) κ(a) is a mathematical property of A Any algorithm will produce a solution that is sensitive to perturbations in A and b if κ(a) is large. In exact math a matrix is either singular or non-singular. κ(a) = for a singular matrix κ(a) indicates how close A is to being numerically singular. Amatrixwithlargeκ issaidtobeill-conditioned In Practice, applying Gaussian elimination with partial pivoting and back substitution to Ax = b gives the exact solution, ˆx, to the nearby problem (A + E)ˆx = b where E ε m A Gaussian elimination with partial pivoting and back substitution gives exactly the right answer to nearly the right question. Trefethen and Bau NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Computational Stability The Residual An algorithm that gives the exact answer to a problem that is near to the original problem issaidtobebackward stable. Algorithms that are not backward stable will tend to amplify roundoff errors present in the original data. As a result, the solution produced by an algorithm that is not backward stable will not necessarily be the solution to a problem that is close to the original problem. Gaussian elimination without partial pivoting is not backward stable for arbitrary A. IfA is symmetric and positive definite, then Gaussian elimination without pivoting in backward stable. Let ˆx be the numerical solution to Ax = b. ˆx x (x is the exact solution) because of roundoff. The residual measures how close ˆx is to satisfying the original equation It is not hard to show that ˆx x ˆx r = b Aˆx κ(a) r b Small r does not guarantee a small ˆx x. If κ(a) is large the ˆx returned by Gaussian elimination and back substitution (or any other solution method) is not guaranteed to be anywhere near the true solution to Ax = b. NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page

20 Rules of Thumb (1) Rules of Thumb () Applying Gaussian elimination with partial pivoting and back substitution to Ax = b yields a numerical solution ˆx such that the residual vector r = b Aˆx is small even if the κ(a) is large. If A and b are stored to machine precision ε m, the numerical solution to Ax = b by any variant of Gaussian elimination is correct to d digits where d = log 1 (ε m ) log 1 (κ(a)) d = log 1 (ε m ) log 1 (κ(a)) Example: Matlab computations have ε m. 1. For a system with κ(a) 1 1 the elements of the solution vector will have d = log 1 (. 1 ) log =11 = correct digits NMM: Solving Systems of Equations page NMM: Solving Systems of Equations page Summary of Limits to Numerical Solution of Ax = b Factorization Methods 1. κ(a) indicates how close A is to being numerically singular. If κ(a) is large, A is ill-conditioned and even the best numerical algorithms will produce a solution, ˆx that cannot be guaranteed to be close to the true solution, x. In practice, Gaussian elimination with partial pivoting and back substitution produces a solution with a small residual r = b Aˆx even if κ(a) is large. LU factorization Cholesky factorization Use of the backslash operator NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 9

21 LU Factorization (1) Find L and U such that A = LU and L is lower triangular, and U is upper triangular. 1 l,1 1 L = l,1 l, l n,1 l n, ln 1,n 1 u 1,1 u 1, u 1, u 1,n u, u, u,n U = u n,n u n,n Since L and U are triangular, it is easy to apply their inverses. LU Factorization () Since L and U are triangular, it is easy to apply their inverses. Consider the solution to Ax = b. Regroup, matrix multiplication is associative A = LU = (LU)x = b L(Ux)=b Let Ux = y, then Ly = b Since L is triangular it is easy (without Gaussian elimination) to compute y = L b This expression should be interpreted as Solve Ly = b with a forward substitution. NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 81 LU Factorization () LU Factorization () Now, since y is known, solve for x x = U y which is interpreted as Solve Ux = y with a backward substitution. Algorithm 8. Solve Ax = b with LU factorization Factor A into L and U Solve Ly = b for y use forward substitution Solve Ux = y for x use backward substitution NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 8

22 The Built-in lu Function Cholesky Factorization (1) Refer to lunopiv and lupiv functions in the NMM toolbox for expository implementations of LU factorization Use the built-in lu function for routine work A must be symmetric and positive definite (SPD) For SPD matrices, pivoting is not required Cholesky factorization requires one half as many flops as LU factorization. Since pivoting is not required, Cholesky factorization will be more than twice as fast as LU factorization since data movement is avoided. Refer to the Cholesky function in NMM Toolbox for a view of the algorithm Use built-in chol function for routine work NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 8 Backslash Redux Nonlinear Systems of Equations The \ operator examines the coefficient matrix before attempting to solve the system. \ uses: A triangular solve if A is triangular, or a permutation of a triangular matrix Cholesky factorization and triangular solves if A is symmetric and the diagonal elements of A are positive (and if the subsequent Cholesky factorization does not fail.) LU factorization if A is square and the preceding conditions are not met. QR factorization to obtain the least squares solution if A is not square. The system of equations Ax = b is nonlinear if A = A(x) or b = b(x) Characteristics of nonlinear systems Solution requires iteration Each iteration involves the work of solving a related linearized system of equations For strongly nonlinear systems it may be difficult to get the iterations to converge Multiple solutions might exist NMM: Solving Systems of Equations page 8 NMM: Solving Systems of Equations page 8

23 Nonlinear Systems of Equations Nonlinear Systems of Equations Example: Intersection of a parabola and a line or, using x 1 = x, x = y y = αx + β y = x + σx + τ αx 1 x = β (x 1 + σ)x 1 x = τ which can be expressed in matrix notation as» α x 1 + σ The coefficient matrix, A depends on x» x1 x =» β τ Graphical Interpretation of solutions to:» α x 1 + σ» x1 x =» β τ Change values of α and β to get Two distinct solutions - - One solution sensitive to inputs One solution - - No solution - - NMM: Solving Systems of Equations page 88 NMM: Solving Systems of Equations page 89 Newton s Method for Nonlinear Systems (1) Newton s Method for Nonlinear Systems () Given where A is n n, write Note: f = r Ax = b f = Ax b The solution is obtained when f 1 (x 1,x,...,x n )) f(x) = f (x 1,x,...,x n )). =. f n (x 1,x,...,x n )) Let x (k) be the guess at the solution for iteration k Look for Δx (k) so that x (k+1) = x (k) +Δx (k) f(x (k+1) )= Expand f with the multidimensional Taylor Theorem f(x (k+1) )=f(x (k) )+f (x (k) )Δx (k) + O Δx (k) NMM: Solving Systems of Equations page 9 NMM: Solving Systems of Equations page 91

24 Newton s Method for Nonlinear Systems () Newton s Method for Nonlinear Systems () f (x (k) ) is the Jacobian of the system of equations f 1 f 1 x 1 x f f f (x) J(x) = x 1 x.... f n f n x 1 x Neglect the higher order terms in the Taylor expansion f(x (k+1) )=f(x (k) )+J(x (k) )Δx (k) f 1 x n f x n f n x n Now, assume that we can find the Δx (k) that gives f(x (k+1) )= =f(x (k) )+J(x (k) )Δx (k) = J(x (k) )Δx (k) = f(x (k) ) The essence of Newton s method for systems of equations is 1. Make a guess at x. Evaluate f. If f is small enough, stop. Evaluate J. solve J Δx = f for Δx. update: x x +Δx. Go back to step NMM: Solving Systems of Equations page 9 NMM: Solving Systems of Equations page 9 Newton s Method for Nonlinear Systems () Newton s Method for Nonlinear Systems () Example: Intersection of a line and parabola y = αx + β y = x + σx + τ Recast as αx 1 x + β = x 1 + σx 1 x + τ = Evaluate each term in the Jacobian f 1 x 1 = α therefore f x 1 =x 1 + σ f 1 x = f x =» α J = (x 1 + σ) or» αx1 x f = + β x 1 + σx = 1 x + τ» NMM: Solving Systems of Equations page 9 NMM: Solving Systems of Equations page 9