Computational Economics and Finance

Transcription

1 Computational Economics and Finance Part II: Linear Equations Spring 2016 Outline Back Substitution, LU and other decomposi- Direct methods: tions Error analysis and condition numbers Iterative methods: (Gauss-) Jacobi and Gauss-Seidel Acceleration and stabilization methods Application: Google PageRank 2

2 Linear Equations System of linear equations Ax = b where A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn Rn n is an n n matrix and b = b 1 b 2. b n Rn is a column vector of dimension n 3 Importance of Solution Methods for Linear Systems Some important problems are linear. Many nonlinear solution methods are sequences of linear problems. Solution methods for linear systems illustrate general ideas and concepts for solving systems of equations. 4

3 Triangular Systems The matrix A is lower triangular if all nonzero elements lie on or below the diagonal: a a A = 21 a a n1 a n2 a nn Upper triangular: all nonzero entries on or above the diagonal Triangular: either lower or upper triangular Diagonal: all nonzero entries are on the diagonal A triangular matrix is nonsingular if and only if all diagonal elements are nonzero. Lower (upper) triangular matrices are closed under addition, multiplication and inversion. 5 Direct Method: Back Substitution Suppose A is lower triangular and nonsingular Back substitution (forward substitution?): x 1 = b 1 a 11, x k = b k k 1 j=1 a kjx j a kk, k =2, 3,...,n. Suppose A is upper triangular and nonsingular Back substitution: x n = b n, a nn x k = b k n j=k+1 a kj x j a kk, k = n 1,n 2,...,2, 1. 6

4 Direct Method: LU Decomposition LU decomposition: If A is nonsingular, then A = LU, wherel is lower triangular and U is upper triangular. L and U computed by Gaussian elimination Ax = b LUx = b Lz = b with Ux = z Step 1: Solve Lz = b for z by back substitution Step 2: Solve Ux = z for x by back substitution 7 LU Decomposition with Pivoting Gaussian elimination may require pivoting LU = PA for permutation matrix P Matlab: [L, U] =lu(a), [L, U, P ]=lu(a) Solving linear equations in Matlab with Gaussian elimination x = A\b (backslash or left division) mldivide(a, b) 8

5 Direct Method: Cholesky Decomposition A is symmetric and positive definite Example: covariance matrix A = LL,whereL (L ) is lower (upper) triangular. L called Cholesky factor Special case of LU factorization: L is upper triangular and takes role of U Matlab: C = chol(a) (careful, C is upper triangular) 9 Error Analysis Direct methods can be exact but are susceptible to round-off errors Iterative methods (Jacobi, Gauss-Seidel) are approximate by nature and typically suffer from truncation error Question: How large is the error? Exact solution x such that Ax = b (exactly) Numerical solution ˆx such that Aˆx b Unknown true error e =ˆx x Known residual r = Aˆx b = Aˆx Ax = Ae Residual r does not indicate magnitude of the error in the solution ˆx for x 10

6 A = [ ; ]; b = [2;2]; A*[1.02;1.02]-b ans = A*[2;0]-b ans = A \ b Example in Matlab ans = Vector Norms Norms : R n R + for vectors x R n l 1 norm: x 1 = n i=1 x i (1-norm) l 2 norm: x 2 = ( ni=1 x 2 i ) 1/2 = ( x x ) 1/2 (Euclidean norm) l norm: x =max i=1,...,n x i (infinity norm) Measures for the length or size of a vector 12

7 Norms in Mathematica x = {3, 4}; Norm[x, 1] Norm[x, 2] Norm[x, ] Norm[x] Matrix Norms For a vector norm on R n the induced matrix norm of an n n matrix A is defined by A =max x 0 Ax x = max x =1 Ax A 1 =max j=1,...,n ni=1 a ij A 2 = ρ ( A A ) (matrix 1-norm) (matrix 2-norm) A =max i=1,...,n nj=1 a ij (matrix infinity norm) Spectral radius ρ (B) =max{ λ : λ is an eigenvalue of the matrix B} Spectral radius ρ (B): largest (in absolute value) eigenvalue For any induced norm: ρ (A) A 14

8 Frobenius Norm of a Matrix n n A F = a 2 ij i=1 j=1 1/2 Not induced by a vector norm Induced vector norms have the property Ax A x 15 Matrix Norms in Matlab A = [1 2; 3 4]; [norm(a,1),norm(a,2),norm(a,inf),norm(a, fro ),norm(a)] ans = sqrt(eig(a *A)) ans =

9 Matrix Norms in Mathematica A = {{1, 2}, {3, 4}} ; N[{Norm[A,1], Norm[A,2], Norm[A, ], Norm[A,Frobenius], Norm[A]}] {6., , 7., , } N[Sqrt[Eigenvalues[Transpose[A].A]]] { , } 17 Error Bounds Recall error e =ˆx x and residual r = Aˆx b = Aˆx Ax = Ae Ae = r A e r x = A 1 b A 1 b x A e x r A 1 b e = A 1 r A 1 r e Ax = b A x b A 1 r b e A x Lower and upper bound on the relative error e x 1 r A A 1 b e x A A 1 r b 18

10 Condition Number Condition number: cond(a) = A A 1 Lower and upper bound on the relative error e x : 1 r cond(a) b e x cond(a) r b Condition number of matrix A bounds the relationship between relative error in b, r b, and relative error in the solution, e x. 1 cond(a) e / x r / b cond(a) is the percentage error in x relative to the percentage error in b, i.e., the elasticity of the output error with respect to the input error. 19 Condition Number Rule of thumb: As the condition number increases by an order of magnitude, one decimal digit of accuracy is lost. Drawback of condition number: norm dependent Spectral condition number: Ratio of largest to smallest (in absolute value) eigenvalue cond (A) =ρ(a)ρ(a 1 ) Independent of norm, lower bound on condition number A ρ(a) cond(a) cond (A) cond (A) often used as estimate for condition number If A is singular, then the condition number is infinite 20

11 Condition Number in Matlab A = [1 2; 3 4]; [cond(a,1),cond(a,2),cond(a,inf),cond(a, fro ),cond(a)] ans = max(abs(eig(a)))/min(abs(eig(a))) ans = A = [ ; ]; cond(a) ans = b = [2;2]; r = A*[2;0]-b r= cond(a)*norm(r)/norm(b) Error Analysis: Example ans =

12 Error Analysis: Example Exercise 2.3: Consider the system of linear equations x y = 1, x y = 0. The exact solution is x = and y = However, double-precision arithmetic yields x = e and y = e Note that cond(a) = e Rescaling and Pre-conditioning Condition numbers are sensitive to scaling Linear system x = a, my = b for x, y R trivial to solve Matrix ( m has spectral condition number m ) Define z = my then problem becomes x = a, z = b Condition number for new system is 1 Change in units ( rescaling ) or a linear transformation ( preconditioning ) can improve conditioning 24

13 Iterative Solution Methods Iterative solution approach for solving linear system: Rewrite Ax = b as a linear fixed-point iteration with n n identity matrix I x =(I A)x + b Richardson iteration x (k+1) =(I A)x (k) + b Advantage: can handle very large problems 25 Stationary Iterative Methods More generally: solve system of linear equations x = Mx + c by generating a sequence {x (k) } via the linear fixed-point iteration x (k+1) = Mx (k) + c Initial iterate x (0), n n iteration matrix M Error after k + 1 iterations e (k+1) = x (k+1) x = Mx (k) + c ( Mx + c ) = M ( x (k) x ) = Me (k) =...= M k e (0) e (k+1) = M k e (0) 0 for arbitrary e (0) if and only if ρ(m) < 1 Theorem: The linear fixed-point iteration converges to x = (I M) 1 c for all c R n and all initial iterates x (0) R n if and only if ρ(m) < 1. 26

14 Linear Equations Ax = b a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn x 1 x 2. x n b 1 b 2. = b n The ith equation in a system of linear equations is n j=1 a ij x j = b i If a ii 0 we can isolate the variable x i : x i = 1 a ii b i a ij x j j i 27 Jacobi Iterative Method Idea: replace system of linear equations with sequence of singlevariable linear equations Given kth iterate x (k) =(x (k) 1,x(k) 2,...,x(k) n Jacobi iteration: ) x (k+1) i = 1 a ii b i j i a ij x (k) j, i =1,...,n Delayed replacement: no x (k+1) i is used until the entire (k +1)th iterate x (k+1) has been computed Iterations continue until some stopping rule is satisfied 28

15 Gauss-Seidel Iterative Method Idea: replace system of linear equations with sequence of singlevariable linear equations and use new information immediately Given kth iterate x (k) =(x (k) 1,x(k) 2,...,x(k) n Compute new iterate for x 1 from first equation x (k+1) 1 = 1 a 11 ) ( b 1 a 12 x (k) 2 a 13x (k) 3... a 1nx n (k) Use x (k+1) 1 immediately to compute x (k+1) 2 x (k+1) 2 = 1 a 22 ( b 2 a 21 x (k+1) 1 a 23 x (k) 3... a 2nx n (k) ) ) Gauss-Seidel iteration x (k+1) i = 1 a ii b i i 1 j=1 a ij x (k+1) j n j=i+1 a ij x (k) j, i =1,...,n. 29 Iterative Methods: Jacobi and Gauss-Seidel Gauss-Seidel method is sensitive to matching between variables and equations ordering of equations Gauss-Seidel method requires less computer memory than the Jacobi method Block Jacobi/Gauss-Seidel iteration: Combine a direct method to solve a subset of equations with an iterative method for the entire system 30

16 Operator Splitting Approach Transform problem into another problem with identical solutions where fixed-point iteration is easy Split A into two operators A = N P, then Ax = b (N P )x = b Nx = b + Px and if N is easily invertible the iteration becomes x (k+1) = N 1 b + ( N 1 P ) x (k) Compute N 1 b and N 1 P only once Convergence for ρ(n 1 P ) < 1 31 Operator Splitting and Jacobi Iteration N = a a a nn, P = 0 a 12 a 1n a 21 0 a 2n a n1 a n2 0 N is diagonal matrix and thus easy to invert Convergence criterion ρ(n 1 P ) N 1 P = max 1 i n j i a ij a ii < 1 Equivalent to (row) diagonal dominance of the matrix A, a ij < a ii i =1, 2,...,n j i 32

17 Operator Splitting and Gauss-Seidel Iteration N = a a 21 a a n1 a n2 a nn, P = 0 a 12 a 1n 0 0 a 2n Same sufficient condition for convergence Theorem: Let A be an n n matrix that is diagonally dominant. Then A is nonsingular and both the Jacobi iteration and the Gauss-Seidel iteration converge to x = A 1 b for all b and all initial iterates. Jacobi and Gauss-Seidel are (at best) linearly convergent at rate ρ(n 1 P ) 33 Equilibrium Problem Table 3.2: supply and demand problem Supply curve q = p 2 +1 Inverse demand curve p =10 q Equilibrium requires supply = demand, quantity q and price p such that p = 10 q q = p

18 Jacobi Iteration Jacobi iteration q (k+1) = p(k) 2 +1 p (k+1) = 10 q (k) Initial iterate (q (0),p (0) )=(1, 4) First iterate (q (1),p (1) )=(3, 9) Second iterate (q (2),p (2) )=(5.5, 7) andsoon Iterative Methods: Gauss-Jacobi and Gauss-Seidel Gauss-Jacobi and Gauss-Seidel iteration. Source: Judd (1998), Figure

19 Gauss-Seidel iteration Gauss-Seidel iteration q (k+1) = p(k) 2 +1 p (k+1) = 10 q (k+1) Initial iterate (q (0),p (0) )=(1, 4) First iterate (q (1),p (1) )=(3, 7) Second iterate (q (2),p (2) )=(4.5, 5.5) andsoon Iterative Methods: Acceleration and Stabilization Methods Recall general fixed-point iteration approach x (k+1) = Gx (k) + b At best linearly convergent at rate ρ(g) But we may be able to increase the linear rate of convergence Extrapolation and dampening: x (k+1) = ω ( Gx (k) + b ) +(1 ω)x (k) = G [ω] x (k) + ωb with G [ω] = ωg +(1 ω)i 38

20 Extrapolation and Dampening ω>1: extrapolation Convergence of initial approach indicates that ( Gx (k) + b ) x (k) is a good direction to move towards solution Try to accelerate convergence by going further in that direction ω<1: dampening ( Gx (k) + b ) x (k) may indicate good direction but full step overshoots Try to avoid overshooting by taking smaller steps 39 Extrapolation and Dampening Choose ω to minimize ρ(g [ω] ): ω = 2 M m 2,whereM =maxσ(g) andm =minσ(g) M<1 implies ρ(g [ω ] )= M m 2 M m < 1 independent of m, i.e., ω either ensures or accelerates convergence 40

21 Extrapolation Procedure for Gauss-Seidel Successive overrelaxation (SOR) x (k+1) i = ω 1 a ii b i i 1 j=1 a ij x (k+1) j n j=i+1 a ij x (k) j +(1 ω)x(k) i Write A as A = L + D + U, wherel, D, andu consists of the elements of A below, on, and above the diagonal, respectively. Then, x (k+1) = ωd 1 b ωd 1 Lx (k+1) ωd 1 Ux (k) +(1 ω)x (k) which is equivalent to x (k+1) = M 1 ω ( Nω x (k) + ωb ) = Mω 1 N ωx (k) + ωmω 1 b, where M ω = D + ωl and N ω =(1 ω)d ωu. 41 Gauss-Seidel and SOR Example Figure 3.4: Inverse demand function is p =21 q and supply function is q = p 2 3 Gauss-Seidel iteration: SOR: where ω =0.75 p (k+1) = 21 3q (k) q (k+1) = p(k+1) 2 3 p (k+1) = ω ( 21 3q (k)) +(1 ω)p (k) q (k+1) = ω 3 +(1 ω)q (k) p(k+1) 2 42

22 SOR Example Dampening. Source: Judd (1998), Figure Gauss-Seidel and SOR Example Gauss-Seidel diverges A is not diagonally dominant Absolute value of largest eigenvalue of N 1 P exceeds 1 SOR converges Altered iteration matrix has small eigenvalues 44

23 Duopoly Game Figure 3.5: Nash equilibrium computation Firm 1 s reaction function: p 1 = BR 1 (p 2 )=1+0.75p 2 Firm 2 s reaction function: p 2 = BR 2 (p 1 )=2+0.8p 1 Gauss-Seidel iteration p (k+1) 1 = p (k) 2 p (k+1) 2 = 2+0.8p (k+1) 1 SOR where ω =1.5 p (k+1) ( 1 = ω p (k) 2 ( p (k+1) 2 = ω 2+0.8p (k+1) 1 ) +(1 ω)p (k) 1 ) +(1 ω)p (k) 2 45 Nash Equilibrium of Duopoly Game Acceleration. Source: Judd (1998), Figure

24 Nonlinear Equations Many concepts and methods carry over from linear to nonlinear equations Idea: f(x) = 0 is approximated by f(x 0 )+f x (x 0 )(x x 0 )=0 Error analysis, condition number Iterative methods: Jacobi, Gauss-Seidel Acceleration and stabilization methods Convergence (local instead of global) 47 Further Topics Sparse matrix techniques Krylov space methods conjugate-gradient method GMRES-method... and other algorithms Matlab; help sparfun 48

25 Application: Google PageRank Google s original algorithm for establishing the importance of web pages: PageRank L. Page, S. Brin, R. Motwani and T. Winograd The PageRank Citation Ranking: Bringing Order to the Web, Stanford Digital Library working paper SIDL-WP Intuitive basis for the algorithm: the random surfer model Mathematically: a random walk on a graph (network) The critical step in the algorithm: solving a large system of linear equations in order to determine the steady-state distribution of amarkovchain 49 World Wide Web Set W of web pages that can be reached from some root page (a portion of the web) n pages in W = {1, 2,...,n} Connectivity matrix G of W : n n matrix with elements g ij { 1 link from page i to page j g ij = 0 link from page i to page j Matrix G can be huge but is very sparse, the number of ones in G is the total number of hyperlinks in W Row and column sums of G n n r i = g ij i W, c j = g ij j=1 i=1 j W 50

26 Random Surfer Model Sum r i is the out-degree of page i: the number of links on page i pointing to other pages Sum c j is the in-degree of page j: the number of links on other pages pointing to page j Random Surfer Model: random walk on the graph of the web Model part I: Surfer on page i randomly clicks on a link Probability to click on the link to page j: g ij r i Model part II: If the surfer enters a small loop or gets bored, then (s)he randomly jumps to another web page in W Probability p of clicking a link, probability 1 p of jumping to another web page; typical value: p = Markov Chain n n matrix A of transition probabilities a ij a ij = p g ij + 1 p r i n assuming r i 0foralli W a ij is the probability of jumping from page i to page j A is the transition probability matrix of a Markov chain; all elements are between 0 and 1 and its row sums are all equal to 1 Markov chain is irreducible and aperiodic; it has a unique stationary probability distribution (invariant measure) x Row vector x R n with x i [0, 1] and n i x i =1 52

27 Stationary Distribution Perron-Frobenius Theorem implies that the stationary distribution x satisfies x lim k Ak = x. x Stationary distribution also satisfies the equation x = xa This system has a one-dimensional solution set. Additional restriction n i x i = 1 yields unique solution; this solution is Google s PageRank; the larger the element x i,themore important the web page i 53 Two Helpful Matrices Let D be a diagonal matrix with d ii = 1 r i,so D = 1 r r r n Let e be the column n-vector of all ones, so ee T isthematrixof all ones, that is, ee T =

28 Computing the PageRank Using these two matrices, the system x = xa canbewrittenas ( x = x pdg + 1 p ) n eet For x to be a probability distribution we need and so we need to solve Perhaps more familiar xe = n i=1 x i =1 x (I pdg) = 1 p n et (I pdg) T x T = 1 p n e 55 Example Matlab: PageRankExample.m First approach: Solve linear system of equation x (I pdg) = 1 p n et Second approach: Take a large power K of the transition matrix ( pdg + 1 p ) K n eet Third approach: Power iteration From random start vector calculate ya, (ya)a, (ya 2 )A,... 56