Introduction to Mathematical Programming

Transcription

1 Introduction to Mathematical Programming Ming Zhong Lecture 6 September 12, 2018 Ming Zhong (JHU) AMS Fall / 20

2 Table of Contents 1 Ming Zhong (JHU) AMS Fall / 20

3 Solving Linear Systems A fast, efficient and effective computational method for solving a large linear system of equations, A x = y is a central concern in many applications. Tractable computation, minimize the operations it take to solve a linear system. direct methods (within a finite number of operations in exact arithmetic): Gaussian elimination (with or without pivoting), banded Gaussian elimination, factorization (LU, QR), and inverting the matrix. iterative methods (provide an approximated solution within certain tolerance) are more efficient for large systems. Gaussian elimination: reduce the matrix, [A y] to row echelon form. then use back substitution to get x. Ming Zhong (JHU) AMS Fall / 20

4 Elementary Row Operations We use the following elementary row operations to conduct the Gaussian elimination, Multiply the i th row by a number a: a R i. Swap the i th and j th rows: Ri R j. Add a multiple of the i th row to the j th row: a R i + R j R j. These row operations can be expressed as a row operation matrix multiplied to the right of the target matrix. ( ) ( ) 0 1 a11 a Let E =, and EA for any A = 12, it swaps the first and 1 0 a 21 a 22 second rows of A. The Gaussian elimination is: E N 1 E N 2 E 1 A = U. Ming Zhong (JHU) AMS Fall / 20

5 : Gaussian Elimination Consider a 3 3 system, x 1 + x 2 + x 3 = 1 x 1 + 2x 2 + 4x 3 = 1 x 1 + 3x 2 + 9x 3 = 1. Re-written in the matrix-vector form, i.e., A x = y, we define, x 1 1 A = x = x 2 y = x 3 1 We then construct the augmented matrix, [A y]. Ming Zhong (JHU) AMS Fall / 20

6 , cont. Continue on, [A y] = = = we put it back into the equation form. Ming Zhong (JHU) AMS Fall / 20

7 , cont. Then we will solve it using back substitution, x 3 = 2 x 3 = 2 x 2 + 3x 3 = 2 x 2 = 8 x 1 + x 2 + x 3 = 1 x 1 = 7. This procedure works, When A is non-singular, i.e., det(a) 0. We might need to shift the rows to avoid a zero pivot. Operation counts for the Gaussian Elimination (without pivoting) on a N N matrix, Movement down the N pivots. For each pivot, perform N additions/subtractions across the columns, For each pivot, perform N addition/subtractions across the rows. Ming Zhong (JHU) AMS Fall / 20

8 Another Let us consider another example The augmented matrix [A y] is, [A y] = 2x 2 + x 3 = 8 x 1 2x 2 3x 3 = 0 x 1 + x 2 + 2x 3 = 3. = Ming Zhong (JHU) AMS Fall / 20

9 Another, cont. Continue on, [A y] = = = Ming Zhong (JHU) AMS Fall / 20

10 Pivoting Strategies Problem with Gaussian Elimination: consider [ ] [A y] = ɛ 1 1 [ ] = 0 1 ɛ 1 2ɛ we obtain a solution x 1 = 1 2ɛ 1 ɛ 1 x 2 = 2 x 1 1 We shall consider pivoting strategies: partial pivoting, scaled partial pivoting, and total pivoting. Ming Zhong (JHU) AMS Fall / 20

11 Factorization: LU Decomposition So it does O(N 3 ) operations, and the back-substitution needs O(N 2 ) operations. It becomes computationally prohibitive for large matrices. We can consider factorization scheme, for example LU factorization, A = LU, where L is a low triangular matrix with 1 s on the diagonal and U is any upper triangular matrix (Doolittle algorithm); and once factorized, they can be re-used over and over again. Let us consider a 3 3 example, a 11 a 12 a u 11 u 12 u 13 A = LU a 21 a 22 a 23 = l u 22 u 23 a 31 a 32 a 33 l 31 l u 33 Ming Zhong (JHU) AMS Fall / 20

12 LU Factorization We have a total of 9 equations to solve, a 11 = u 11 a 12 = u 12 a 13 = u 13 a 21 = l 21 u 11 a 22 = l 21 u 12 + u 22 a 23 = l 21 u 13 + u 23 a 31 = l 31 u 11 a 32 = l 31 u 12 + l 32 u 22 a 33 = l 31 u 13 + l 32 u 23 + u 33 Ming Zhong (JHU) AMS Fall / 20

13 Solving the Linear System with LU Once we obtain the factorization of A, A x = y LU x = y Let z = U x, then we solve the following, L z = y and U x = y. Solving the system, L z = y, z 1 = y 1 l 21 z 1 + z 2 = y 2 l 31 z 1 + l 32 z 2 + z 3 = y 3 requires only O(N 2 ); then for U x = z, u 11 x 1 + u 12 x 2 + u 13 x 3 = z 1 u 22 x 2 + u 33 x 3 = z 2 u 33 x 3 = z 3 Ming Zhong (JHU) AMS Fall / 20

14 Let us consider another 3 3 example A = we will use Gaussian Elimination to reduce A to row echelon form, We use 1/2 of the first row added to the second row, so l 21 = 1/2, and 1/4 of the first row added to the third row, so l 31 = 1/4. Ming Zhong (JHU) AMS Fall / 20

15 , cont. Continue on, We use 1/2 of the second row added to the third row, so l 32 = 1/2. Therefore, L = 1/2 1 0 and U = /4 1/ still requires O(N 3 ) operations. Ming Zhong (JHU) AMS Fall / 20

16 The Permutation Matrix The original Gaussian elimination algorithm often encounters the need to shift a row of zero pivot; it can be handled by a permutation matrix P, for example, P = then, we can A x = y PA x = P y LU x = P y. If permutation of rows is necessary, MATLAB can supply the permutation matrix associated with the LU decomposition. Ming Zhong (JHU) AMS Fall / 20

17 The Cholesky Factorization In the special case of A being a symmetric positive definite matrix (A = A and x A x > 0 for all x 0), A = LU = LDŨ with D = diag(u 11,, u NN ). Then A = (LDŨ) = Ũ DL = LDŨ = A. Hence L = Ũ, A = LDL = L L, absorbing u ii. This method is called the Cholesky factorization, where A = LL. Ming Zhong (JHU) AMS Fall / 20

18 The mldivide in MATLAB One can solve the linear system A x = y very easily in MATLAB by either using mldivide or A\ y, First, it checks to see if A is triangular, or some permutation. If it is, it does a O(N 2 ) substitution routine. Second, it checks to see if A is symmetric, i.e., Hermitian of self-adjoint. If so, a Cholesky factorization is attempted. If A is positive definite, the Cholesky algorithm is always successful and takes half the run time of LU. Third, it checks to see if A is Hessenberg. If so, it can be written as an upper triangular matrix and solved by a substitution routine. Fourth, if the above all fail, then LU factorization is used and the forward- and backward- substitution routines are used. Ming Zhong (JHU) AMS Fall / 20

19 More on mldivide Continue on, If A is not square, a QR (Householder) routine is used to solve the system. If A is not square and sparse, a least-square solution using QR factorization is performed. Note that, Solving the linear system by x = A 1 y is the slowest of all methods. LU factorization is always recommended, once LU are obtained, they can be reused over and over again. When det(a) 0, the matrix is ill-conditioned, we might have to use some pivoting strategies. Finding A 1 is not advised. Ming Zhong (JHU) AMS Fall / 20

20 MATLAB Commands Other linear solvers, A\ y: solve the linear system. x = mldivide(a, y) solve the linear system similar to \. [L, U] = lu(a) generate the L and U matrices. [L, U, P] = lu(a) generate the L and U matrices along with the permutation matrix P. x = linsolve(a, y) solve the linear system. x = inv(a) y also solve the linear system with an exact inverse of A (not recommended). x = pinv(a) y solve the linear system with persudo-inverse of A. Ming Zhong (JHU) AMS Fall / 20