ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong
Lecture 8: QR Decomposition QR decomposition Gram-Schmidt orthonormalization Householder transformations Givens rotations W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 1
Main Result QR decomposition: any A R m n can be decomposed as A = QR, where Q R m m is orthogonal, R R m n takes an upper triangular form. efficient to compute, poor man s SVD can be used to compute LS solutions; a basis for R(A) ; other decompositions such as SVD. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 2
QR Decomposition QR decomposition: any A R m n can be decomposed as A = QR, where Q R m m is orthogonal; R R m n takes an upper triangular form. Questions: how do we know that QR decomposition exists? answer: by the algorithm, as we will see how to compute the factors Q and R algorithmically? answer: more than one way: Gram-Schmidt, Householder, and Givens. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 3
Application: LS Consider the LS problem for a full column-rank A R m n : Let min x R n Ax y 2 2. A = QR = [ ] [ ] R Q 1 Q 1 2 0 = Q 1 R 1, where Q 1 R m n, Q 2 R m (m n), R 1 R n n (upper triangular). Since Ax y 2 2 = Q T (Ax y) 2 2 = = R 1 x Q T 1 y 2 2 + Q T 2 y 2 2, [ ] Q T 1 (Ax y) 2 Q T 2 (Ax y) the LS problem is equivalent to solving the upper triangular system R 1 x = Q T 1 y. 2 W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 4
QR for Full Column-Rank Matrices Theorem 8.1. Let A R m n, and suppose that A has full column rank. Then, A admits a decomposition A = Q 1 R 1, ( ) where Q 1 R m n is semi-orthogonal; R 1 R n n is upper triangular and nonsingular. we call ( ) a thin QR of A, and (Q 1, R 1 ) a thin QR pair of A it follows directly that Property 8.1. For the thin QR in Theorem 8.1, we have R(A) = R(Q 1 ). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 5
QR for Full Column-Rank Matrices Proof of Theorem 8.1: 1. A T A is PD 2. let R 1 be the (upper triangular and nonsingular) Cholesky factor of A T A; i.e., Also, let Q 1 = AR 1 1. A T A = R T 1 R 1. 3. it can be verified that Q T 1 Q 1 = I, Q 1 R 1 = A. alternatively, Theorem 8.1 can be proven by using the Gram-Schmidt procedure W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 6
Classical Gram-Schmidt for Thin QR Recall the Gram-Schmidt procedure in Lecture 1: Algorithm: Gram-Schmidt input: a collection of linearly independent vectors a 1,..., a n q 1 = a 1, q 1 = q 1 / q 1 2 for i = 2,..., n why q i = a i i 1 j=1 (qt j a i)q j why q i = q i / q i 2 end output: q 1,..., q n let r ii = q i 2, r ij = q T j a i for j = 1,..., i 1 from the above procedure we see that a i = i j=1 r ijq i ; thus A = Q 1 R 1 where Q 1 = [ q 1,..., q m ]; R 1 is upper triangular with [R 1 ] ij = r ij for j i W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 7
Classical Gram-Schmidt for Thin QR the Gram-Schmidt procedure gives an alternative proof of the existence of thin QR decomposition for full-column rank A it also tells us how to compute the thin QR factors there are many variants with the Gram-Schmidt procedure e.g., one may want to improve the numerical stability of the procedure W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 8
Modified Gram-Schmidt the following modified Gram-Schmidt procedure is known to be numerically more stable. let r i denote the ith row of R 1, and note A = Q 1 R 1 = n q i r T i i=1 suppose q 1,..., q k 1, r 1,..., r k 1 are known. Then, A k 1 i=1 we can obtain q k, r k via q i r T i = n i=k q i r T i = [ 0,..., 0, }{{} k 1 zeros r kk q k,... ], q k = q k / q k 2, r T i = q T k (A k 1 i=1 q ir T i ). q k = (A k 1 i=1 q ir T i )e k, W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 9
Householder Transformations Consider an alternative QR via the Householder transformations (to be introduced). Reflection matrix: a matrix of the form H = I 2P, where P R m m is an orthogonal projector, is called a reflection matrix. by letting x = Px + P x, P = I P, we see that where Px is reflected. a reflection matrix is orthogonal: Hx = Px + P x, H T H = I 2P 2 P T }{{} =P +4 PP }{{} =P = I. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 10
Householder Transformations Problem: given x R m, find a reflection matrix H R m m such that Hx = βe 1, for some β R. Householder transformations: let v R m, v 0, and let H = I 2 v 2 vv T. 2 Suppose x x 1 e 1 (if x = x 1 e 1 we simply choose H = I). By choosing v = x + x 2 e 1, we get (verify this as a mini-exercise). Hx = x 2 e 1 ; W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 11
Householder QR Stage 1: let H 1 R m m be the Householder transformation w.r.t. a 1. Then,... A (1) = H 1 A = 0...... 0... Stage 2: let H 2 R (m 1) (m 1) be the Householder transformation w.r.t. the 1st column of A (1) 2:m,2:n. Then,... [ ] [ ] 1 0... 0... A (2) = H 0 1 A = H2 0 H2 A (1) =. 0... }{{} 2:m,2:n.... =H 2 0 0... The same idea applies to stage k, 1 k n 1. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 12
Householder QR Householder QR: for k = 1,..., n 1, compute where A (k) = H k A (k 1), H k = [ ] Ik 1 0, 0 Hk in which I k is the k k identity matrix, and H k R (m k+1) (n k+1) is the Householder transformation w.r.t. A (k 1) k:m,k. The above procedure results in A (n 1) = H n 1... H 2 H 1 A, A (n 1) being upper triangular. A = QR is obtained by letting Q = H T 1 H T 2... H T n 1 (orthogonal), R = A (n 1). the above QR procedure already implies the existence of full QR; also, A can be any matrix (rather than being of full column rank as in Gram-Schmidt). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 13
Consider yet another way for QR. Givens Rotations Example: given x R 2, consider [ y1 y 2 ] = [ ] c s s c }{{} =J [ ] x1 x 2 = [ ] cx1 + sx 2, sx 1 + cx 2 where c = cos(θ), s = sin(θ), for some θ. It can be verified that J is orthogonal y 2 = 0 if θ = tan 1 (x 2 /x 1 ), or if c = x 1 x 2 1 + x 2 2, s = x 2 x 2 1 + x 2 2. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 14
Givens Rotations Givens rotations: I c s J(i, k, θ) = I s c I where c = cos(θ), s = sin(θ); c lies in the (k, k)th and (i, i) entries; s the (i, k)th entry; s the (k, i)th entry. J(i, k, θ) is orthogonal let y = J(i, k, θ)x. We have y j = cx i + sx k, sx i + cx k, x j, j = i j = k j i, k Also, y k can be forced to zero by choosing θ = tan 1 (x k /x i ). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 15
Givens QR Example: A = 0 0 0 0 consider a 4 3 matrix. J 2,1 J 4,2 0 0 0 0 0 0 J 3,1 J 4,3 0 J 4,1 0 0 0 0 0 0 0 = R 0 0 0 J 3,2 where B J C means B = JC; J i,k = J(i, k, θ), with θ that is chosen to zero out the (i, k)th entry of the matrix transformed by J i,k. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 16
Givens QR Givens QR: perform a sequence of Givens rotations to annihilate the lower triangular parts of A to obtain J n,n 1... (J n,2... J 32 )(J n1... J 21 ) A = R }{{} =Q T where R is upper triangular, and Q is orthogonal. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 17