lecture 2 and 3: algorithms for linear algebra

lecture 2 and 3: algorithms for linear algebra STAT 545: Introduction to computational statistics Vinayak Rao Department of Statistics, Purdue University August 27, 2018

Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations 1/23

Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations Consider AX = b, where A is N N, and X and b are N k. Solve for X: X = A 1 b 1/23

Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations Consider AX = b, where A is N N, and X and b are N k. Solve for X: X = A 1 b Calculate the inverse of A and multiply? No! 1/23

Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] 2/23

Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 2 0 1 x 2 v 21 v 22 v 23 = 10 0 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 2/23

Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 2 0 1 x 2 v 21 v 22 v 23 = 10 0 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 Manipulate to get: I [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] 2/23

Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 2 0 1 x 2 v 21 v 22 v 23 = 10 0 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 Manipulate to get: I [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] At step i: Make element a ii = 1 (by scaling or pivoting) Set other elements in column i to 0 by multiplying and subtracting that row 2/23

Gauss-Jordan elimination 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 2 0 1 x 2 v 21 v 22 v 23 = 10 0 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 3/23

Gauss-Jordan elimination 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 2 0 1 x 2 v 21 v 22 v 23 = 10 0 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 Multiply row 1 by 2 and subtract 3/23

Gauss-Jordan elimination 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 0 0 1 x 2 v 21 v 22 v 23 = 2 2 1 0 1 2 1 x 3 v 31 v 32 v 33 3 0 0 1 3/23

Gauss-Jordan elimination 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 0 0 1 x 2 v 21 v 22 v 23 = 2 2 1 0 0 2 2 x 3 v 31 v 32 v 33 1 1 0 1 Subtract row 1 3/23

Gauss-Jordan elimination 1 0 1 x 1 v 11 v 12 v 13 4 1 0 0 0 2 2 x 2 v 21 v 22 v 23 = 1 1 0 1 0 0 1 x 3 v 31 v 32 v 33 2 2 1 0 Pivot 3/23

Gauss-Jordan elimination 1 0 0 x 1 v 11 v 12 v 13 6 1 1 0 0 1 0 x 2 v 21 v 22 v 23 = 2.5 1.5 1 0.5 0 0 1 x 3 v 31 v 32 v 33 2 2 1 0 Continue till we get an identity matrix 3/23

Gauss-Jordan elimination 1 0 0 x 1 v 11 v 12 v 13 6 1 1 0 0 1 0 x 2 v 21 v 22 v 23 = 2.5 1.5 1 0.5 0 0 1 x 3 v 31 v 32 v 33 2 2 1 0 What is the cost of this algorithm? 3/23

Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] 4/23

Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] O(N 3 ) manipulation to get: U [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] Here, U is an upper-triangular matrix. 4/23

Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] O(N 3 ) manipulation to get: U [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] Here, U is an upper-triangular matrix. Cannot just read off solution. Need to backsolve. 4/23

LU decomposition What are we actually doing? A = LU Here L and U are lower and upper triangular matrices. 1 0 0 u 11 u 12 u 13 L = l 21 1 0, U = 0 u 22 u 23 l 31 l 32 1 0 0 u 33 5/23

LU decomposition What are we actually doing? A = LU Here L and U are lower and upper triangular matrices. Is this always possible? 1 0 0 u 11 u 12 u 13 L = l 21 1 0, U = 0 u 22 u 23 l 31 l 32 1 0 0 u 33 [ ] 0 1 A = 1 0 PA = LU, P is a permutation matrix Crout s algorithm, O(N 3 ), stable, L, U can be computed in place. 5/23

Backsubstitution AX = b 6/23

Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb 6/23

Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb 1 0 0 l 21 1 0 l 31 l 32 1 y 1 y 2 y 3 ˆb 1 = ˆb 2 ˆb 3 6/23

Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb 1 0 0 l 21 1 0 l 31 l 32 1 y 1 y 2 y 3 ˆb 1 = ˆb 2 ˆb 3 Then solve X by back substitution UX = Y 6/23

Comments LU-decomposition can be reused for different b s. Calculating LU decomposition: O(N 3 ). Given LU decomposition, solving for X: O(N 2 ). A = P 1 LU = ( 1) S N i=1 u ii (S: num. of exchanges) LUA 1 = PI, can solve for A 1. (back to Gauss-Jordan) 7/23

Cholesky decomposition If A is symmetric positive-definite: A = LL T (but now L need not have a diagonal of ones) Square-root of A More stable. Twice as efficient. Related: A = LDL T (but now L has a unit diagonal). 8/23

Eigenvalue decomposition An N N matrix A: a map from R N R N. An eigenvector v undergoes no rotation: Av = λv λ is the corresponding eigenvalue, and gives the rescaling. 9/23

Eigenvalue decomposition An N N matrix A: a map from R N R N. An eigenvector v undergoes no rotation: Av = λv λ is the corresponding eigenvalue, and gives the rescaling. Let Λ = diag(λ 1,, λ N ) with λ 1 λ 2,..., λ N, and V = [v 1,, v N ] be the matrix of corresponding eigenvectors AV = VΛ 9/23

Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? 10/23

Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) This is the power method (Google s PageRank). 10/23

Gauss-Jordan elimination 10 7 8 7 7 5 6 5 8 6 10 9 7 5 9 10 x 1 x 2 x 3 x 4 32 = 23 33 31 What is the solution? How about for b = [32.1, 22.9, 33.1, 30.9] T? 11/23

Gauss-Jordan elimination 10 7 8 7 7 5 6 5 8 6 10 9 7 5 9 10 x 1 x 2 x 3 x 4 32 = 23 33 31 What is the solution? How about for b = [32.1, 22.9, 33.1, 30.9] T? Why the difference? the determinant? the inverse? the condition number? An ill-conditioned problem can strongly amplify errors. 11/23

Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v 12/23

Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v For a symmetric, real matrix, A = λ max (A) (why?) For a general, real matrix, A = λ max (A T A) (why?) 12/23

Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v For a symmetric, real matrix, A = λ max (A) (why?) For a general, real matrix, A = λ max (A T A) (why?) For A in R N N and any v R N, Av A v (why?) 12/23

Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b 13/23

Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b From b = Ax and δx = A 1 δb, we have b A x, and δx A 1 δb : δx x A A 1 δb b 13/23

Stability analysis κ(a) 1 (why?) 14/23

Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) 14/23

Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) For a real matrix, κ(a) = λmax (A T A) λ min (A T A) (why?) 14/23

Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) For a real matrix, κ(a) = λmax (A T A) λ min (A T A) (why?) Condition number is a property of a problem Stability is a property of an algorithm A bad algorithm can mess up a simple problem 14/23

Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 15/23

Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 [ ] 1e 4 1 1 1 15/23

Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 [ ] 1e 4 1 1 1 Partial pivoting: Pivot rows to bring max r v r1 to top Can dramatically improve performance. E.g. 15/23

Gaussian elimination with partial pivoting Recall Gaussian elimination decomposes A = LU and solves two intermediate problems. What are the condition numbers of L and U? Try [ ] 1e 4 1 1 1 Note: does pivoting for you automatically! (see the function in package ) 16/23

QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) 17/23

QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I 17/23

QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I κ(a) = κ(q)κ(r) (why?) Can use to solve Ax = b (How?) Most stable decomposition 17/23

Gram-Schmidt orthonormalization Given N vectors x 1,..., x N construct an orthonormal basis: 18/23

Gram-Schmidt orthonormalization Given N vectors x 1,..., x N construct an orthonormal basis: u 1 = x 1 / x 1 ũ i = x i i 1 j=1 (xt i u j)u i, u i = ũ i / ũ i i = 2..., N 18/23

QR decomposition A = Q R a 11 a 12 a 13 q 11 q 12 q 13 r 11 r 12 r 13 a 21 a 22 a 23 = q 21 q 22 q 23 0 r 22 r 23 a 31 a 32 a 33 q 31 q 32 q 33 0 0 r 33 19/23

QR decomposition A = Q R a 11 a 12 a 13 q 11 q 12 q 13 r 11 r 12 r 13 a 21 a 22 a 23 = q 21 q 22 q 23 0 r 22 r 23 a 31 a 32 a 33 q 31 q 32 q 33 0 0 r 33 QR decomposition: Gram-Schmidt on columns of A (can you see why?) Of course, there are more stable/efficient ways of doing this (Householder rotation/givens rotation) O(N 3 ) algorithms (though about twice as slow as LU) 19/23

QR algorithm Algorithm to calculate all eigenvalues/eigenvectors of a (not too-large) matrix Start with A 0 = A. At iteration i: A i = Q i R i A i+1 = R i Q i Then A i and A i+1 have the same eigenvalues (why?), and the diagonal contains the eigenvalues. Can be made more stable/efficient. One of Dongarra & Sullivan (2000) s list of top 10 algoirithms. See also number 4, decompositional approach to matrix computations 20/23

Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 21/23

Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) 21/23

Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) log(p(x µ, Σ)) = 1 2 YT Y N 2 log 2π log Σ 21/23

Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) log(p(x µ, Σ)) = 1 2 YT Y N 2 log 2π log Σ Can also just forward solve for L 1 : LL 1 = I (Inverted triangular matrix isn t too bad) 21/23

Sampling a multivariate normal Sampling a univariate normal: Inversion method (default for?). Box-Muller transform: (Z 1, Z 2 ) : independent standard normals. 22/23

Sampling a multivariate normal Sampling a univariate normal: Inversion method (default for?). Box-Muller transform: (Z 1, Z 2 ) : independent standard normals. Let Z N (0, I) X = µ + LZ Z = N (µ, L T L) 22/23

x 10 10 5 0 5 10 5 0 5 10 y [ ] 1 0 Σ = 0 1 23/23

x 10 10 5 0 5 10 5 0 5 10 y [ ] 4 0 Σ = 0 1 23/23

x 10 10 5 0 5 10 5 0 5 10 y [ ] 4 2 Σ = 2 4 23/23

x 10 10 5 0 5 10 5 0 5 10 y [ ] 4 3.8 Σ = 3.8 4 23/23