Numerical Linear Algebra

Transcription

1 Numerical Linear Algebra Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign Sep 19th, 2017 C. Hurtado (UIUC - Economics) Numerical Methods

2 On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

3 Numerical Python On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

4 Numerical Python Numerical Python The NumPy package (read as NUMerical PYthon) provides access to a new data structure called arrays which allow us to perform efficient vector and matrix operations. NumPy is the updated version of two previous modules: Numeric and Numarray. In 2006 it was decided to merge the best aspects of Numeric and Numarray into the Scientific Python (scipy) package and to provide an array data type under the module name NumPy. NumPy contains some linear algebra functions. C. Hurtado (UIUC - Economics) Numerical Methods 1 / 39

5 Numerical Python Numerical Python NumPy introduces a new data type which is called array An array appears to be very similar to a list but an array can keep only elements of the same type (arrays are more efficient to store) Vectors and matrices are all called arrays in NumPy. To create a Vector (one dimensional array) we do: 1 >>> import numpy as np 2 >>> x = np. array ([0, 0.5, 1, 1.5]) 3 [0, 0.5, 1, 1.5] We can also creation a vector using ArrayRange 1 x=np. arange (0,2,0.5) 2 [0, 0.5, 1, 1.5] C. Hurtado (UIUC - Economics) Numerical Methods 2 / 39

6 Numerical Python Numerical Python There are some useful functions for arrays: 1 >>> y=np. zeros (4) 2 [ ] Remember, we need to be aware of the reference 1 >>> z=y 2 >>> y [0]=99 3 >>> y [2]= >>> print z 5 [ ] Sometimes it is better to work with a copy of the object: 1 >>> z=y. copy () 2 >>> z [0]=0 3 >>> z [2]=0 4 >>> print z 5 [ ] 6 >>> print y 7 [ ] C. Hurtado (UIUC - Economics) Numerical Methods 3 / 39

7 Numerical Python Numerical Python we can perform calculations on every element in the vector with a single statement: 1 >>> x array ([ 10., 10.5, 11., 11.5]) 3 >>> x **2 4 array ([ 0., 0.25, 1., 2.25]) 5 >>> 2* x 6 array ([ 0., 1., 2., 3.]) To create a matrix we use a list of lists: 1 >>> X=np. array ([[1,2],[3,4]]) 2 [[1 2] 3 [3 4]] C. Hurtado (UIUC - Economics) Numerical Methods 4 / 39

8 Numerical Python Numerical Python There are several useful functions: 1 >>> Y=np. zeros ((3,3) ) 2 [[ ] 3 [ ] 4 [ ]] 5 >>> Z=np. ones ((2,2) ) 6 [[ 1. 1.] 7 [ 1. 1.]] 8 >>> I=np. identity (3) 9 [[ ] 10 [ ] 11 [ ]] We can get the dimension of the matrix: 1 >>> A=np. array ([[1, 2, 3], [4, 5, 6]]) 2 [[1 2 3] 3 [4 5 6]] 4 >>> A. shape 5 (2, 3) C. Hurtado (UIUC - Economics) Numerical Methods 5 / 39

9 Numerical Python Numerical Python Individual elements can be accessed using the standard syntaxes. It is also possible to recover all the elements form a row or a column 1 >>> A [:,1] 2 [2 5] 3 >>> A [0,:] 4 [1 2 3] We can also transform arrays to list 1 >>> list (A [1,:]) 2 [4, 5, 6] C. Hurtado (UIUC - Economics) Numerical Methods 6 / 39

10 Solving Systems of Linear Equations On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

11 Solving Systems of Linear Equations Solving Systems of Linear Equations We can multiply matrices and vectors using the dot product function 1 >>> A=np. random. rand (5,5) 2 >>> x=np. random. rand (5) 3 >>> b=np.dot (A,x) To solve a system of equations A x = b (given in matrix form) we can use the linear algebra package 1 >>> x2=np. linalg. solve (A,b) How does the computer solve a system of equations? C. Hurtado (UIUC - Economics) Numerical Methods 7 / 39

12 LU Decomposition On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

13 LU Decomposition LU Decomposition Consider a system of equations Ax = b. Suppose that we can factorize A into two matrices, L and U, where L is a lower triangular matrix and U is an upper triangular matrix. The system is then Ax = b LUx = b L }{{} Ux = b y Ly = b We could perform a 2-step solution for the system: 1. Solve the lower triangular system Ly = b, by forward substitution. 2. Solve the upper triangular system Ux = y, by back substitution. C. Hurtado (UIUC - Economics) Numerical Methods 8 / 39

14 LU Decomposition LU Decomposition If we have a system with lower triangular matrix l l 21 l l n1 l n2 l nn The solution can be computed as: x 1 = b 1 l 11 x 1 x 2. x n = b 1 b 2. b n for k = 2, 3,, n x k = b k k 1 j=1 l kjx j l kk C. Hurtado (UIUC - Economics) Numerical Methods 9 / 39

15 LU Decomposition LU Decomposition A similar method can be used to solve an upper triangular system (but starting form the last equation x n = bn u nn ). The total number of divisions is n, the number of multiplications and additions is n(n 1)/2 (why?). For a big n, the total number of operations is close to n 2 /2 That means that the time it takes to solve a system of equations increases in a quadratic proportion to the number of variables in the system This is know as quadratic time computation. C. Hurtado (UIUC - Economics) Numerical Methods 10 / 39

16 LU Decomposition LU Decomposition Any nonsingular matrix A can be decomposed into two matrices L and U. Let us consider A = We would like to transform the matrix to get an upper triangular matrix. - Leave the first row unchanged. - replace the second row: multiply the first row by -2 and add to the second row - replace the third row: multiply the first row by -4 and add to the third row C. Hurtado (UIUC - Economics) Numerical Methods 11 / 39

17 LU Decomposition LU Decomposition The previous can be computed as follows: L 1 A = = Now we would like to remove the 2 located at the third row and second column. - Leave the first and second rows unchanged. - replace the third row: multiply the second row by -2 and add to the third row L 2 (L 1 A) = = = U C. Hurtado (UIUC - Economics) Numerical Methods 12 / 39

18 LU Decomposition LU Decomposition So we can write where and L 1 = A = L 2 L 1 A = U, L 2 =, U = Then we can write A = L 1 2 L 1 1 U C. Hurtado (UIUC - Economics) Numerical Methods 13 / 39

19 LU Decomposition LU Decomposition Notice that: and L 1 1 = L 1 2 = The multiplication of lower triangular matrices is also a lower triangular matrix. In particular: L = L 1 2 L 1 1 = C. Hurtado (UIUC - Economics) Numerical Methods 14 / 39

20 LU Decomposition LU Decomposition In summary: A = = = LU Notice that L is a unit lower triangular matrix, i.e., it has ones on the diagonal. What should we do for a general matrix A? C. Hurtado (UIUC - Economics) Numerical Methods 15 / 39

21 LU Decomposition LU Decomposition Let us start with A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn Suppose that a Define li1 1 = a i1/a 11, for i = 2,, n. Then: a L 1 A = I l a12 1 a 1 1n A = 0 a 22 2 a 2 2n A(2) ln an2 n ann n C. Hurtado (UIUC - Economics) Numerical Methods 16 / 39

22 LU Decomposition LU Decomposition Proceeding column by column in similar fashion, we can construct a series of lower triangular matrices that replaces the elements below the diagonal with zeros. Let us denote by a k ij the elements of the k th matrix A (k). If akk k 0, we define lij k = and a k ij a k kk 0 for j = k, i = k + 1,, n otherwise a k+1 ij = { a k ij l k ik ak kj a k ij for i = k + 1,, n and j = k,, n otherwise C. Hurtado (UIUC - Economics) Numerical Methods 17 / 39

23 LU Decomposition LU Decomposition We have just defined a sequence of matrices such that A (k+1) = I 0 0 l k k+1,k 0 A (k) lnk k 0 In this way we can rewrite the matrix A as the multiplication LU. C. Hurtado (UIUC - Economics) Numerical Methods 18 / 39

24 LU Decomposition LU Decomposition How does it work in practice? Let us consider the system Ax = b: x 1 x 2 x 3 We can rewrite the system as: = x 1 x 2 x = Let us define y 1 y 2 y 3 = x 1 x 2 x 3 C. Hurtado (UIUC - Economics) Numerical Methods 19 / 39

25 LU Decomposition LU Decomposition We can first solve y 1 y 2 y 3 = The solution is y 1 = 3, y 2 = 4 and y 3 = 3. Then we solve: x 1 x 2 x 3 = The solution of the system is then x 1 = 10, x 2 = 17 2 and x 3 = 3 2. C. Hurtado (UIUC - Economics) Numerical Methods 20 / 39

26 LU Decomposition LU Decomposition The proposed method for LU decomposition assumes that a kk 0. What can we do to decompose the following matrix? We can change (pivot) first and second row pre-multiplying by Then proceed as before C. Hurtado (UIUC - Economics) Numerical Methods 21 / 39

27 LU Decomposition LU Decomposition Another point to consider has to do with the rounding error. If the elements of the diagonal are small, the fraction a ij a kk the significant digits are removed. is big, and Consider for example the matrix [ ] If we compute 3/10E 8 we get , chopping away the significant digits. To avoid this we can pivot the rows. C. Hurtado (UIUC - Economics) Numerical Methods 22 / 39

28 LU Decomposition LU Decomposition In practice, the LU factorization in python uses pivoting. To perform the decomposition we use the lu function from the module linalg in scipy. 1 >>> from numpy import array 2 >>> from scipy. linalg import lu 3 >>> A = array ([[0,1,2],[2,0,3],[1,1,1]]) 4 >>> P, L, U = lu(a) The previous code will generate the matrix L and U, but also a pivoting matrix P. The matrix A can be recover as P LU 1 >>> from numpy import matmul 2 >>> print matmul (P, matmul (L,U)) C. Hurtado (UIUC - Economics) Numerical Methods 23 / 39

29 LU Decomposition LU Decomposition Let us consider another example 1 >>> B= array ([[10E -8,2],[3,4]]) 2 >>> P, L, U = lu(b) 3 >>> print P 4 >>> print matmul (P, matmul (L,U)) Notice the permutation matrix and the product P LU 1 [[ 0. 1.] 2 [ 1. 0.]] and 1 [[ e e +00] 2 [ e e +00]] C. Hurtado (UIUC - Economics) Numerical Methods 24 / 39

30 Cholesky Factorization On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

31 Cholesky Factorization Cholesky Factorization There is a special case of the LU decomposition for a subset of square matrices A matrix is symmetric if A = A A matrix A is positive semidefinite if: - A is symmetric and x Ax 0 for all x 0 If the previous inequality is strict, we call A a positive definite matrix. Examples: A 1 = [ ], A 2 = [ ], A 3 = [ ] C. Hurtado (UIUC - Economics) Numerical Methods 25 / 39

32 Cholesky Factorization Cholesky Factorization A 1 is positive definite: [ ] [ x 9 6 x1 A 1 x = [x 1 x 2 ] 6 5 x 2 ] = 9x x 1 x 2 +5x 2 2 = (3x 1 +2x 2 ) 2 +x 2 2 A 2 is positive semidefinite (but not positive definite): [ ] [ ] x 9 6 x1 A 2 x = [x 1 x 2 ] = 9x x x 2 + 4x2 2 = (3x 1 + 2x 2 ) 2 x 2 A 3 is not positive semidefinite: [ ] [ ] x 9 6 x1 A 3 x = [x 1 x 2 ] 6 3 x 2 = 9x x 1 x 2 +3x 2 2 = (3x 1 +2x 2 ) 2 x 2 2 C. Hurtado (UIUC - Economics) Numerical Methods 26 / 39

33 Cholesky Factorization Cholesky Factorization More Examples: for a given matrix X, A = X X is positive semidefinite v Av = v X Xv = (Xv) (Xv) = Xv 2 Some properties of a positive definite matrix A: - The diagonal elements of A are positive [ ] a11 A - If we rewrite A = 21, the matrix A A 21 A 22 (1/a 11 )A 21 A 21 is 22 also positive definite. [ ] [ ] [ Hint: 1 a 11 A 21 v v a 11 A 21 1 a 11 A 21 v ] > 0 A 21 A 22 v C. Hurtado (UIUC - Economics) Numerical Methods 27 / 39

34 Cholesky Factorization Cholesky Factorization Every positive definite matrix A can be factored as A = LL where L is lower triangular with positive diagonal elements. L is called the Cholesky factor of A L can be interpreted as the square root of a positive definite matrix C. Hurtado (UIUC - Economics) Numerical Methods 28 / 39

35 Cholesky Factorization Cholesky Factorization We want to partition the matrix A = LL as [ ] [ ] [ ] a11 A 21 l11 0 l11 L = 21 A 21 A 22 L 21 L 22 0 L 22 [ ] l = 11 2 l 11 L 21 l 11 L 21 L 21 L 21 + L 22L 22 The elements of the diagonal of a positive definite matrix are positive, so l 11 = a 11 is well defined and positive. We can define L 21 = (1/l 11 )A 21 Finally, we can compute the Cholesky factorization defined by L 22 L 22 = A 22 L 21 L 21 = A 22 1 a 11 A 21 A 21 C. Hurtado (UIUC - Economics) Numerical Methods 29 / 39

36 Cholesky Factorization Cholesky Factorization Example = l l 21 l 22 0 l 31 l 32 l 33 l 11 l 21 l 31 0 l 22 l l 33 First column of L second column of L [ ] [ = l l 32 l 33 ] [ ] 18 0 [3 1] = 0 11 [ ] 9 3 = l 22 l l 33 [ C. Hurtado (UIUC - Economics) Numerical Methods 30 / 39 ]

37 Cholesky Factorization Cholesky Factorization second column of L [ ] = [ l 33 ] [ l 33 ] third column of L: = l 2 33 = l 33 = 3 In conclusion: = C. Hurtado (UIUC - Economics) Numerical Methods 31 / 39

38 Cholesky Factorization Cholesky Factorization How can we use the Cholesky Factorization? Let us assume that there is a stock market with returns that are normally distributed with µ = 5% and σ 2 = 10% If we want to simulate the returns of the stock we can try the following: - Define the returns as r = µ + σz, where Z N(0, 1) is a Standard Normal random variable. The linear combination that defines r simulates the stock: E[r] = E[µ + σz] = E[µ] + E[σZ] = µ and Var(r) = E[(r µ) 2 ] = E[σ 2 Z 2 ] = σ 2 C. Hurtado (UIUC - Economics) Numerical Methods 32 / 39

39 Cholesky Factorization Cholesky Factorization In a more realistic situation we would like to simulate the returns of a portfolio with more than one asset. Let us start with a portfolio of two assets (you can generalize to any number of assets). If the proportion invested on the first asset is λ, the return of the portfolio is r = λr 1 + (1 λ)r 2 where r 1 is the return of the first asset and r 2 is the return of the second asset. It may be the case that the returns of the assets are correlated. For example, if the first asset is stock, and the second asset is U.S. Treasuries, the returns will exhibit strong negative correlation (Why?) We need to account for the covariance of r 1 and r 2 C. Hurtado (UIUC - Economics) Numerical Methods 33 / 39

40 Cholesky Factorization Cholesky Factorization Let us assume that E[r 1 ] = µ 1 and E[r 2 ] = µ 2. The expected return of the portfolio is E[r] = E[λr 1 + (1 λ)r 2 ] = λe[r 1 ] + (1 λ)e[r 2 ] = λµ 1 + (1 λ)µ 2 The variance of the portfolio is: Var(r) = E[(r λµ 1 (1 λ)µ 2 ) 2 ] = E[(λ(r 1 µ 1 ) + (1 λ)(r 2 µ 2 )) 2 ] = E[λ 2 (r 1 µ 1 ) 2 + (1 λ) 2 (r 2 µ 2 ) 2 + 2λ(1 λ)(r 1 µ 1 )(r 2 µ 2 )] = λ 2 E[(r 1 µ 1 ) 2 ] + (1 λ) 2 E[(r 2 µ 2 ) 2 ] + 2λ(1 λ)e[(r 1 µ 1 )(r 2 µ 2 )] = λ 2 Var(r 1 ) + (1 λ) 2 Var(r 2 ) + 2λ(1 λ)cov(r 1, r 2 ) Also computable as [ Var(r) = λ (1 λ) ] [ Var(r 1 ) Cov(r 1, r 2 ) Cov(r 1, r 2 ) Var(r 2 ) ] [ λ 1 λ ] C. Hurtado (UIUC - Economics) Numerical Methods 34 / 39

41 Cholesky Factorization Cholesky Factorization This matrix is known as the variance-covariance matrix: [ ] Var(r Σ = 1 ) Cov(r 1, r 2 ) Cov(r 1, r 2 ) Var(r 2 ) The variance-covariance matrix is symmetric, with all the elements on the diagonal positive and positive definite. Notice that the we could generate a vector of random variables (a.k.a. multivariate normal distribution) x = (r 1, r 2 ), with µ = (µ 1, µ 2 ) and variance-covariance matrix Σ using the square root of Σ. That is, the Cholesky factorization s.t. Σ = LL That is: ] ] ] [ r1 r 2 = [ µ1 µ 2 + L [ z1 z 2, where z 1 N(0, 1) and z 2 N(0, 1) C. Hurtado (UIUC - Economics) Numerical Methods 35 / 39

42 Accuracy of Operations On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods

43 Accuracy of Operations Accuracy of Operations We want to find the solution of Ax = b. Suppose that, using some algorithm, we have computed a numerical solution ˆx We would like to be able to evaluate the absolute error x ˆx, or the relative error x ˆx x We don t know the error, but we would like to find an upper bound We begin analyzing the residual r = b Aˆx C. Hurtado (UIUC - Economics) Numerical Methods 36 / 39

44 Accuracy of Operations Accuracy of Operations How does the residual r relate to the error in ˆx? r = b Aˆx = Ax Aˆx = A(x ˆx) We have then x ˆx = A 1 r We can define the norm of a matrix as follows A = max Ax s.t. x = 1 x Using that definition of norm of matrix it es easy to show that x ˆx = A 1 r A 1 r This gives a bound on the absolute error in ˆx in terms of A 1 C. Hurtado (UIUC - Economics) Numerical Methods 37 / 39

45 Accuracy of Operations Accuracy of Operations Usually the relative error is more meaningful. Using the definition of norm of a matrix we know that b A x The previous implies that 1 x A b Hence, we have an upper bound for the relative error. x ˆx x A 1 r A b We are going to call the condition number cond(a) = A A 1 C. Hurtado (UIUC - Economics) Numerical Methods 38 / 39

46 Accuracy of Operations Accuracy of Operations How big can the relative error be? For a matrix A we have 1 r cond(a) b x ˆx x cond(a) r b If the condition number is close to 1, then the relative error and relative residual will be close The accuracy of the solution depends on the conditioning number of the matrix. If a matrix is ill-conditioned, then a small roundoff error can have a drastic effect on the output If the matrix is well-conditioned, then the computerized solution is quite accurate. The condition number is a property of the matrix A. C. Hurtado (UIUC - Economics) Numerical Methods 39 / 39