12. Cholesky factorization

Size: px

Start display at page:

Download "12. Cholesky factorization"

Carmel Farmer
5 years ago
Views:

1 L. Vandenberghe ECE133A (Winter 2018) 12. Cholesky factorization positive definite matrices examples Cholesky factorization complex positive definite matrices kernel methods 12-1

2 Definitions a symmetric matrix A R n n is positive semidefinite if x T Ax 0 for all x a symmetric matrix A R n n is positive definite if x T Ax > 0 for all x 0 this is a subset of the positive semidefinite matrices note: if A is symmetric and n n, then x T Ax is the function x T Ax = n n A ij x i x j = n A ij x i x j i=1 j=1 i=1 A ii x 2 i + 2 i>j this is called a quadratic form Cholesky factorization 12-2

3 Example A = [ a ] x T Ax = 9x x 1 x 2 + ax 2 2 = (3x 1 + 2x 2 ) 2 + (a 4)x 2 2 A is positive definite for a > 4 x T Ax > 0 for all nonzero x A is positive semidefinite but not positive definite for a = 4 x T Ax 0 for all x, x T Ax = 0 for x = (2, 3) A is not positive semidefinite for a < 4 x T Ax < 0 for x = (2, 3) Cholesky factorization 12-3

4 Simple properties every positive definite matrix A is nonsingular Ax = 0 = x T Ax = 0 = x = 0 (last step follows from positive definiteness) every positive definite matrix A has positive diagonal elements A ii = e T i Ae i > 0 every positive semidefinite matrix A has nonnegative diagonal elements A ii = e T i Ae i 0 Cholesky factorization 12-4

5 Schur complement partition n n symmetric matrix A as A = [ A11 A T 2:n,1 A 2:n,1 A 2:n,2:n ] the Schur complement of A 11 is defined as the (n 1) (n 1) matrix S = A 2:n,2:n 1 A 11 A 2:n,1 A T 2:n,1 if A is positive definite, then S is positive definite to see this, take any x 0 and define y = (A T 2:n,1x)/A 11 ; then x T Sx = [ y x ] T [ A11 A T 2:n,1 A 2:n,1 A 2:n,2:n ] [ y x ] > 0 because A is positive definite Cholesky factorization 12-5

6 Singular positive semidefinite matrices we have seen that positive definite matrices are nonsingular (page 12-4) if A is positive semidefinite, but not positive definite, then it is singular to see this, suppose A is positive semidefinite but not positive definite there exists a nonzero x with x T Ax = 0 since A is positive semidefinite the following function is nonnegative: f(t) = (x tax) T A(x tax) = x T Ax 2tx T A 2 x + t 2 x T A 3 x = 2t Ax 2 + t 2 x T A 3 x f(t) 0 for all t is only possible if Ax = 0, i.e., Ax = 0 hence there exists a nonzero x with Ax = 0 Cholesky factorization 12-6

7 Exercises show that if A R n n is positive semidefinite, then B T AB is positive semidefinite for any B R n m show that if A R n n is positive definite, then B T AB is positive definite for any B R n m with linearly independent columns Cholesky factorization 12-7

8 Outline positive definite matrices examples Cholesky factorization complex positive definite matrices kernel methods

9 Exercise: resistor circuit R 1 R 2 x 1 x 2 y 1 + R 3 + y 2 [ y1 y 2 ] = [ ] [ ] R1 + R 3 R 3 x1 R 2 + R 3 x 2 R 3 show that A = [ ] R1 + R 3 R 3 R 3 R 2 + R 3 is positive definite if R 1, R 2, R 3 are positive Cholesky factorization 12-8

10 Solution Solution from physics x T Ax = y T x is the power delivered by sources, dissipated by resistors power dissipated by the resistors is positive unless both currents are zero Algebraic solution x T Ax = (R 1 + R 3 )x R 3 x 1 x 2 + (R 2 + R 3 )x 2 2 = R 1 x R 2 x R 3 (x 1 + x 2 ) 2 0 and x T Ax = 0 only if x 1 = x 2 = 0 Cholesky factorization 12-9

11 Gram matrix recall the definition of Gram matrix of a matrix B (page 4-21): A = B T B every Gram matrix is positive semidefinite x T Ax = x T B T Bx = Bx 2 0 x a Gram matrix is positive definite if x T Ax = x T B T Bx = Bx 2 > 0 x 0 in other words, B has linearly independent columns Cholesky factorization 12-10

12 Graph Laplacian recall definition of node-arc incidence matrix of a directed graph (page 3-29) B ij = 1 if arc j points to node i 1 if arc j points from node i 0 otherwise assume there are no self-loops and at most one arc between any two nodes B = Cholesky factorization 12-11

13 Graph Laplacian the positive semidefinite matrix A = BB T is called the Laplacian of the graph A ij = degree of node i 1 0 otherwise if i = j if i j and there is an arc i j or j i the degree of a node is the number of arcs incident to it A = BB T = Cholesky factorization 12-12

14 Laplacian quadratic form recall the interpretation of matrix-vector multiplication with B T (page 3-31) if y is vector of node potentials, then B T y contains potential differences: (B T y) j = y k y l if arc j goes from node l to k y T Ay = y T BB T y is the sum of squared potential differences y T Ay = B T y 2 = (y j y i ) 2 arcs i j (this is also known as the Dirichlet energy function) Example: for the graph on the previous page y T Ay = (y 2 y 1 ) 2 + (y 4 y 1 ) 2 + (y 3 y 2 ) 2 + (y 1 y 3 ) 2 + (y 4 y 3 ) 2 Cholesky factorization 12-13

15 Outline positive definite matrices examples Cholesky factorization complex positive definite matrices kernel methods

16 Cholesky factorization every positive definite matrix A R n n can be factored as A = R T R where R is upper triangular with positive diagonal elements complexity of computing R is (1/3)n 3 flops R is called the Cholesky factor of A can be interpreted as square root of a positive definite matrix gives a practical method for testing positive definiteness Cholesky factorization 12-14

17 Cholesky factorization algorithm [ A11 A 1,2:n A 2:n,1 A 2:n,2:n ] = [ R11 0 R1,2:n T R2:n,2:n T ] [ R11 R 1,2:n 0 R 2:n,2:n ] = [ R 2 11 R 11 R T 1,2:n R 11 R 1,2:n R1,2:nR T 1,2:n + R2:n,2:nR T 2:n,2:n ] 1. compute first row of R: 2. compute 2, 2 block R 2:n,2:n from R 11 = A 11, R 1,2:n = 1 R 11 A 1,2:n A 2:n,2:n R T 1,2:nR 1,2:n = R T 2:n,2:nR 2:n,2:n this is a Cholesky factorization of order n 1 Cholesky factorization 12-15

18 Discussion the algorithm works for positive definite A of size n n step 1: if A is positive definite then A 11 > 0 step 2: if A is positive definite, then A 2:n,2:n R T 1,2:nR 1,2:n = A 2:n,2:n 1 A 11 A 2:n,1 A T 2:n,1 is positive definite (see page 12-5) hence the algorithm works for n = m if it works for n = m 1 it obviously works for n = 1; therefore it works for all n Cholesky factorization 12-16

19 Example = R R 12 R 22 0 R 13 R 23 R 33 R 11 R 12 R 13 0 R 22 R R 33 = Cholesky factorization 12-17

20 Example = R R 12 R 22 0 R 13 R 23 R 33 R 11 R 12 R 13 0 R 22 R R 33 first row of R = 3 R R 23 R R 22 R R 33 second row of R [ ] [ 3 1 [ ] [ ] [ ] [ ] R22 0 R22 R 3 1 = 23 R 23 R 33 0 R 33 ] = [ ] [ ] R 33 0 R 33 third column of R: 10 1 = R 2 33, i.e., R 33 = 3 Cholesky factorization 12-18

21 Solving equations with positive definite A solve Ax = b with A a positive definite n n matrix Algorithm factor A as A = R T R solve R T Rx = b solve R T y = b by forward substitution solve Rx = y by back substitution Complexity: (1/3)n 3 + 2n 2 (1/3)n 3 flops factorization: (1/3)n 3 forward and backward substitution: 2n 2 Cholesky factorization 12-19

22 Cholesky factorization of Gram matrix suppose B is an m n matrix with linearly independent columns the Gram matrix A = B T B is positive definite (page 4-21) two methods for computing the Cholesky factor of A, given B 1. compute A = B T B, then Cholesky factorization of A A = R T R 2. compute QR factorization B = QR; since A = B T B = R T Q T QR = R T R the matrix R is the Cholesky factor of A Cholesky factorization 12-20

23 Example B = , A = B T B = [ ] 1. Cholesky factorization: A = [ ] [ ] 2. QR factorization B = = 3/5 0 4/ [ ] Cholesky factorization 12-21

24 Comparison of the two methods Numerical stability: QR factorization method is more stable see the example on page 8-16 QR method computes R without squaring B (i.e., forming B T B) this is important when the columns of B are almost linearly dependent Complexity method 1: cost of symmetric product B T B plus Cholesky factorization mn 2 + (1/3)n 3 flops method 2: 2mn 2 flops for QR factorization method 1 is faster but only by a factor of at most two (if m n) Cholesky factorization 12-22

25 Sparse positive definite matrices Cholesky factorization of dense matrices (1/3)n 3 flops on a standard computer: a few seconds or less, for n up to several 1000 Cholesky factorization of sparse matrices if A is very sparse, R is often (but not always) sparse if R is sparse, the cost of the factorization is much less than (1/3)n 3 exact cost depends on n, number of nonzero elements, sparsity pattern very large sets of equations can be solved by exploiting sparsity Cholesky factorization 12-23

26 Sparse Cholesky factorization if A is sparse and positive definite, it is usually factored as A = P R T RP T P a permutation matrix; R upper triangular with positive diagonal elements Interpretation: we permute the rows and columns of A and factor P T AP = R T R choice of permutation greatly affects the sparsity R there exist several heuristic methods for choosing a good permutation Cholesky factorization 12-24

27 Example sparsity pattern of A Cholesky factor of A pattern of P T AP Cholesky factor of P T AP Cholesky factorization 12-25

28 Solving sparse positive definite equations solve Ax = b with A a sparse positive definite matrix Algorithm 1. compute sparse Cholesky factorization A = P R T RP T 2. permute right-hand side: c := P T b 3. solve R T y = c by forward substitution 4. solve Rz = y by back substitution 5. permute solution: x := P z Cholesky factorization 12-26

29 Outline positive definite matrices examples Cholesky factorization complex positive definite matrices kernel methods

30 Quadratic form suppose A is n n and Hermitian (A ij = Āji) x H Ax = = = n n A ij x i x j i=1 n j=1 (A ij x i x j + Āijx i x j ) A ii x i 2 + i=1 i>j n A ii x i Re A ij x i x j i=1 i>j note that x H Ax is real for all x C n Cholesky factorization 12-27

31 Complex positive definite matrices a Hermitian n n matrix A is positive semidefinite if x H Ax 0 for all x C n a Hermitian n n matrix A is positive definite if x H Ax > 0 for all nonzero x C n Cholesky factorization every positive definite matrix A C n n can be factored as A = R H R where R is upper triangular with positive real diagonal elements Cholesky factorization 12-28

32 Outline positive definite matrices examples Cholesky factorization complex positive definite matrices kernel methods

33 Regularized least squares model fitting we revisit the data fitting problem with linear-in-parameters model (page 9-9) ˆf(x) = θ 1 f 1 (x) + θ 2 f 2 (x) + + θ p f p (x) = θ T F (x) F (x) = (f 1 (x),..., f p (x)) is a p-vector of basis functions f 1 (x),..., f p (x) Regularized least squares model fitting (page 10-7) minimize N (θ T F (x (k) ) y (k)) 2 p + λ k=1 j=1 θ 2 j (x (1), y (1) ),..., (x (N), y (N) ) are N examples to simplify notation, we add regularization for all coefficients θ 1,..., θ p next discussion can be modified to handle f 1 (x) = 1, regularization p j=2 θ2 j Cholesky factorization 12-29

34 Regularized least squares problem in matrix notation minimize Aθ y d 2 + λ θ 2 A has size N p (# examples # basis functions) A = F (x (1) ) T F (x (2) ) T. F (x (N) ) T = f 1 (x (1) ) f 2 (x (1) ) f p (x (1) ) f 1 (x (2) ) f 2 (x (2) ) f p (x (2) )... f 1 (x (N) ) f 2 (x (N) ) f p (x (N) ) y d is the N-vector y d = (y (1),..., y (N) ) we discuss methods for problems with N p (A is very wide) equivalent stacked least squares problem (page 10-3) has size (p + N) p QR factorization method may be too expensive when N p Cholesky factorization 12-30

35 Solution of regularized LS problem from the normal equations: ˆθ = (A T A + λi) 1 A T y d = A T (AA T + λi) 1 y d second expression follows from the push-through identity (A T A + λi) 1 A T = A T (AA T + λi) 1 this is easily proved, by writing it as A T (AA T + λi) = (A T A + λi)a T from the second expression for ˆθ and the definition of A, ˆf(x) = ˆθ T F (x) = w T AF (x) = N w i F (x (i) ) T F (x) i=1 where w = (AA T + λi) 1 y d Cholesky factorization 12-31

36 Algorithm 1. compute the N N matrix Q = AA T, which has elements Q ij = F (x (i) ) T F (x (j) ), i, j = 1,..., N 2. use a Cholesky factorization to solve the equation (Q + λi)w = y d Remarks ˆθ = A T w is not needed; w is sufficient to evaluate the function ˆf(y): ˆf(y) = N w i F (x (i) ) T F (y) i=1 complexity: (1/3)N 3 flops for factorization plus cost of computing Q Cholesky factorization 12-32

37 Example: multivariate polynomials ˆf(x) is a polynomial of degree d (or less) in n variables x = (x 1,..., x n ) ˆf(x) is a linear combination of all possible monomials x k 1 1 xk 2 2 xk n n where k 1,..., k n are nonnegative integers with k 1 + k k n d number of different monomials is ( n + d n ) = (n + d)! n! d! Example: for n = 2, d = 3 there are ten monomials 1, x 1, x 2, x 2 1, x 1 x 2, x 2 2, x 3 1, x 2 1x 2, x 1 x 2 2, x 3 2 Cholesky factorization 12-33

38 Multinomial formula (x 0 + x x n ) d = k 0 + +k n =d (d + 1)! k 0! k 1! k n! xk 0 0 xk 1 1 xk n n sum is over all nonnegative integers k 0, k 1,..., k n with sum d setting x 0 = 1 gives (1 + x 1 + x x n ) d = k 1 + +k n d c k1 k 2 k n x k 1 1 xk 2 2 xk n n the sum includes all monomials of degree d or less with variables x 1,..., x n coefficient c k1 k 2 k n is defined as c k1 k 2 k n = (d + 1)! k 0! k 1! k 2! k n! with k 0 = d k 1 k n Cholesky factorization 12-34

39 Vector of monomials write polynomial of degree d or less, with variables z R n, as ˆf(x) = θ T F (x) F (x) is vector of basis functions ck1 k n x k 1 1 xk 2 2 xk n n for all k 1 + k k n d length of F (x) is p = (n + d)!/(n! d!) multinomial formula gives simple formula for inner products F (u) T F (v): F (u) T F (v) = k 1 + +k n d = (1 + u 1 v u n v n ) d c k1 k 2 k n (u k 1 1 uk n n )(v k 1 1 vk n n ) only 2n + 1 flops needed for inner product of length p = (n + d)!/(n! d!) Cholesky factorization 12-35

40 Example vector of monomials of degree d = 3 or less in n = 2 variables F (u) T F (v) = 1 3u1 3u2 3u 2 1 6u1 u 2 3u 2 2 u 3 1 3u 2 1 u 2 3u1 u 2 2 T 1 3v1 3v2 3v 2 1 6v1 v 2 3v 2 2 v 3 1 3v 2 1 v 2 3v1 v 2 2 u 3 2 v 3 2 = (1 + u 1 v 1 + u 2 v 2 ) 3 Cholesky factorization 12-36

41 Least squares fitting of multivariate polynomials fit polynomial of n variables, degree d, to points (x (1), y (1) ),..., (x (N), y (N) ) Algorithm (see page 12-32) 1. compute the N N matrix Q with elements Q ij = K(x (i), x (j) ) where K(u, v) = (1 + u T v) d 2. use a Cholesky factorization to solve the equation (Q + λi)w = y d the fitted polynomial is ˆf(x) = N w i K(x (i), x) = i=1 N w i (1 + (x (i) ) T x) d i=1 complexity: nn 2 flops for computing Q, plus (1/3)N 3 for the factorization, i.e., nn 2 + (1/3)N 3 flops Cholesky factorization 12-37

42 Kernel methods Kernel function: a generalized inner product K(u, v) K(u, v) is inner product of vectors of basis functions F (u) and F (v) F (u) may be infinite-dimensional kernel methods work with K(u, v) directly, do not require F (u) Examples the polynomial kernel function K(u, v) = (1 + u T v) d the Gaussian radial basis function kernel u v 2 K(u, v) = exp ( 2σ 2 ) kernels exist for computing with graphs, texts, strings of symbols,... Cholesky factorization 12-38

43 Example: handwritten digit classification we apply the method of page to least squares classification training set is images from MNIST data set ( 1000 examples per digit) vector x is vector of pixel intensities (size n = 28 2 = 784) we use the polynomial kernel with degree d = 3: K(u, v) = (1 + u T v) 3 hence F (z) has length p = (n + d)!/(n! d!) = we calculate ten Boolean classifiers ˆf k (x) = sign( f k (x)), k = 1, ˆf k (x) distinguishes digit k 1 (outcome +1) form other digits (outcome 1) the Boolean classifiers are combined in the multi-class classifier ˆf(x) = argmax k=1,...,10 f k (x) Cholesky factorization 12-39

44 Least squares Boolean classifier Algorithm: compute Boolean classifier for digit k 1 versus the rest 1. compute N N matrix Q with elements Q ij = (1 + (x (i) ) T x (j) ) d, i, j = 1,..., N 2. define N-vector y d with elements y d i = { +1 x (i) is an example of digit k 1 1 otherwise 3. solve the equation (Q + λi)w = y d the solution w gives the Boolean classifier for digit k 1 versus rest f k (x) = N w i (1 + (x (i) ) T x) d i=1 Cholesky factorization 12-40

45 Complexity the matrix Q is the same for each of the ten Boolean classifiers hence, only the right-hand side of the equation (Q + λi)w = y d is different for each Boolean classifier Complexity constructing Q requires N 2 /2 inner products of length n: nn 2 flops Cholesky factorization of Q + λi: (1/3)N 3 flops solve the equation (Q + λi)w = y d for the 10 right-hand sides: 20N 2 flops total is (1/3)N 3 + nn 2 Cholesky factorization 12-41

46 Classification error 6 Training set Test set Classification error (%) λ percentage of misclassified digits versus λ Cholesky factorization 12-42

47 Confusion matrix Predicted digit Digit Total All multiclass classifier (λ = 10 4 ) on test examples 292 digits are misclassified (2.9% error) Cholesky factorization 12-43

Examples of misclassified digits Predicted digit Digit 0

48 Examples of misclassified digits Predicted digit Digit Cholesky factorization 12-44

49 Examples of misclassified digits Predicted digit Digit Cholesky factorization 12-45

ECE133A Applied Numerical Computing Additional Lecture Notes

Winter Quarter 2018 ECE133A Applied Numerical Computing Additional Lecture Notes L. Vandenberghe ii Contents 1 LU factorization 1 1.1 Definition................................. 1 1.2 Nonsingular sets