Notes on Householder QR Factorization

Transcription

1 Notes on Householder QR Factorization Robert A van de Geijn Department of Computer Science he University of exas at Austin Austin, X 7872 rvdg@csutexasedu September 2, 24 Motivation A fundamental problem to avoid in numerical codes is the situation where one starts with large values and one ends up with small values with large relative errors in them his is known as catastrophic cancellation he Gram-Schmidt algorithms can inherently fall victim to this: column a j is successively reduced in length as components in the directions of {q,, q j } are subtracted, leaving a small vector if a j was almost in the span of the first j columns of A Application of a unitary transformation to a matrix or vector inherently preserves length hus, it would be beneficial if the QR factorization can be implementated as the successive application of unitary transformations he Householder QR factorization accomplishes this he first fundamental insight is that the product of unitary matrices is itself unitary If, given A C m n with m n, one could find a sequence of unitary matrices, {H,, H n }, such that R H n H A, where R C m n is upper triangular, then R H n H A H H }{{ n } Q Q R Q L Q R R Q L R, where Q L equals the first n columns of A hen A Q L R is the QR factorization of A he second fundamental insight will be that the desired unitary transformations {H,, H n } can be computed and applied cheaply 2 Householder ransformations Reflectors 2 he general case In this section we discuss Householder transformations, also referred to as reflectors

2 H x z + u x u H u x u x v x y u z H u x u u H I 2 u u x H I 2 u u x y u Figure : Left: Illustration that shows how, given vectors x and unit length vector u, the subspace orthogonal to u becomes a mirror for reflecting x represented by the transformation I 2uu H Right: Illustration that shows how to compute u given vectors x and y with x 2 y 2 Definition Let u C n be a vector of unit length u 2 hen H I 2uu H is said to be a reflector or Householder transformation We observe: Any vector z that is perpendicular to u is left unchanged: I 2uu H z z 2uu H z z Any vector x can be written as x z + u H xu where z is perpendicular to u and u H xu is the component of x in the direction of u hen I 2uu H x I 2uu H z + u H xu z + u H xu 2u }{{} u H z 2uu H u H xu z + u H xu 2u H x u H u }{{} u z u H xu his can be interpreted as follows: he space perpendicular to u acts as a mirror : any vector in that space along the mirror is not reflected, while any other vector has the component that is orthogonal to the space the component outside, orthogonal to, the mirror reversed in direction, as illustrated in Figure Notice that a reflection preserves the length of the vector Exercise 2 Show that if H is a reflector, then HH I reflecting the reflection of a vector results in the original vector, 2

3 H H H, and H H H I a reflection is a unitary matrix and thus preserves the norm Next, let us ask the question of how to reflect a given x C n into another vector y C n with x 2 y 2 In other words, how do we compute vector u so that I 2uu H x y From our discussion above, we need to find a vector u that is perpendicular to the space with respect to which we will reflect From Figure right we notice that the vector from y to x, v x y, is perpendicular to the desired space hus, u must equal a unit vector in the direction v: u v/ v 2 Remark 3 In subsequent discussion we will prefer to give Householder transformations as I uu H /τ, where τ u H u/2 so that u needs no longer be a unit vector, just a direction he reason for this will become obvious later In the next subsection, we will need to find a Householder transformation H that maps a vector x to a multiple of the first unit basis vector e Let us first discuss how to find H in the case where x R n We seek v so that I 2 v v vv x ± x 2 e Since the resulting vector that we want is y ± x 2 e, we must choose v x y x x 2 e Exercise 4 Show that if x R n, v x x 2 e, and τ v v/2 then I τ vv x ± x 2 e In practice, we choose v x + signχ x 2 e where χ denotes the first element of x he reason is as follows: the first element of v, ν, will be ν χ x 2 If χ is positive and x 2 is almost equal to χ, then χ x 2 is a small number and if there is error in χ and/or x 2, this error becomes large relative to the result χ x 2, due to catastrophic cancellation Regardless of whether χ is positive or negative, we can avoid this by choosing x χ + signχ x 2 e 22 As implemented for the Householder QR factorization real case Next, we discuss a slight variant on the above discussion that is used in practice o do so, we view x as a vector that consists of its first element, χ, and the rest of the vector, : More precisely, partition x χ, 3

4 where χ equals the first element of x and is the rest of x hen we will wish to find a Householder vector u so that I τ χ Notice that y in the previous discussion equals the vector by v χ x 2 ± x 2 ± x 2 We now wish to normalize this vector so its first entry equals : u v χ x 2 ν χ x 2 /ν, so the direction of u is given where ν χ x 2 equals the first element of v Note that if ν then can be set to 23 he complex case optional Next, let us work out the complex case, dealing explicitly with x as a vector that consists of its first element, χ, and the rest of the vector, : More precisely, partition x χ where χ equals the first element of x and is the rest of x hen we will wish to find a Householder vector u so that I τ H, χ ± x 2 Here ± denotes a complex scalar on the complex unit circle By the same argument as before χ ± x 2 v We now wish to normalize this vector so its first entry equals : u v χ ± x 2 v 2 χ ± x 2 /ν where ν χ ± x 2 If ν then we set to 4

5 Algorithm: [ ρ ], τ Housev χ χ 2 : x 2 2 χ α : x 2 2 χ 2 ρ signχ x 2 ρ : signχ α ν χ + signχ x 2 ν : χ ρ /ν : /ν χ 2 χ 2/ ν 2 τ + u H 2 /2 τ + χ 2 2/2 Figure 2: Computing the Householder transformation Left: simple formulation Right: efficient computation Note: I have not completely double-checked these formulas for the complex case hey work for the real case Exercise 5 Verify that I τ H χ ρ where τ u H u/2 + u H 2 /2 and ρ ± x 2 Hint: ρ ρ ρ 2 x 2 2 since H preserves the norm Also, x 2 2 χ and z z z z Again, the choice ± is important For the complex case we choose ± signχ χ χ 24 A routine for computing the Householder vector We will refer to the vector as the Householder vector that reflects x into ± x 2 e and introduce the notation [ ] ρ χ, τ : Housev as the computation of the above mentioned vector, and scalars ρ and τ, from vector x We will use the notation Hx for the transformation I τ uuh where u and τ are computed by Housevx 3 Householder QR Factorization Let A be an m n with m n We will now show how to compute A QR, the QR factorization, as a sequence of Householder transformations applied to A, which eventually zeroes out all elements of that matrix below the diagonal he process is illustrated in Figure 3 5

6 [ ρ α a 2 Original matrix Housev, τ ] α a 2 : a 2 A 22 ρ a 2 w2 A 22 w2 Move forward Figure 3: Illustration of Householder QR factorization 6

7 In the first iteration, we partition A α a 2 a 2 A 22 Let [ ρ, τ ] Housev α be the Householder transform computed from the first column of A hen applying this Householder transform to A yields H α a 2 : I α a 2 a 2 A 22 τ a 2 A 22 ρ a 2 w 2 A 22 w2, where w 2 a 2 + uh 2 A 22/τ Computation of a full QR factorization of A will now proceed with the updated matrix A 22 Now let us assume that after k iterations of the algorithm matrix A contains A R L R R a 2 R r R 2 α a 2 a 2 A 22, where R L and R are k k upper triangular matrices Let [ ρ, τ ] Housev α a 2 and update A : I R r R 2 H I α a 2 τ a 2 A 22 H I R r R 2 τ α a 2 a 2 A 22 R r R 2 ρ a 2 w 2, A 22 w 2 7

8 where again w2 a 2 + uh 2 A 22/τ Let H k I τ k k H be the Householder transform so computed during the k + st iteration hen upon completion matrix A contains R R L H n H H  where  denotes the original contents of A and R L is an upper triangular matrix Rearranging this we find that  H H H n R which shows that if Q H H H n then  QR Exercise 6 Show that I H I τ I τ H ypically, the algorithm overwrites the original matrix A with the upper triangular matrix, and at each step is stored over the elements that become zero, thus overwriting a 2 It is for this reason that the first element of u was normalized to equal In this case Q is usually not explicitly formed as it can be stored as the separate Householder vectors below the diagonal of the overwritten matrix he algorithm that overwrites A in this manner is given in Fig 4 We will let [{U\R}, t] HouseholderQRA denote the operation that computes the QR factorization of m n matrix A, with m n, via Householder transformations It returns the Householder vectors and matrix R in the first argument and the vector of scalars τ i that are computed as part of the Householder transformations in t heorem 7 Given A C m n the cost of the algorithm in Figure 4 is given by C HQR m, n 2mn n3 flops Proof: he bulk of the computation is in w2 a 2 + uh 2 A 22/τ and A 22 w2 During the kth iteration when R L is k k, this means a matrix-vector multiplication u H 2 A 22 and rank- 8

9 Algorithm: [A, t] HouseholderQRA A L A R Partition A and t where A L is and t while n do Repartition A L A R t has elements A a A 2 a α a 2 A 2 a 2 A 22 where α and τ are scalars [ α a 2 Update, τ ] a 2 A 22 : via the steps [ ρ : I τ ], τ Housev and α u H 2 t a 2 a 2 A 22 t τ t 2 w2 : a 2 + a H 2A 22/τ a 2 a 2 w2 : A 22 A 22 a 2w2 Continue with A L A R endwhile A a A 2 a α a 2 A 2 a 2 A 22 and t t τ t 2 Figure 4: Unblocked Householder transformation based QR factorization update with matrix A 22 which is of size approximately m k n k for a cost of 4m kn k flops hus the total cost is approximately n n 4m kn k 4 m n + jj 4m n k j n 2m nnn + 4 2m nn n j j 2 n j n j + 4 j dx 2mn 2 2n n3 2mn n3 j 2 4 Forming Q Given A C m n, let [A, t] HouseholderQRA yield the matrix A with the Householder vectors stored below the diagonal, R stored on and above the diagonal, and the τ i stored in vector t We 9

10 Original matrix α a 2 : a 2 A 22 /τ u H 2A 22/τ Move forward /τ A 22 + a 2 Figure 5: Illustration of the computation of Q

11 now discuss how to form the first n columns of Q H H H n he computation is illustrated in Figure?? Notice that to pick out the first n columns we must form Q I n n H H n where B k is defined as indicated I n n In n H H k H k H n }{{} B k Lemma 8 B k has the form B k H k H n In n Ik k Bk Proof: he proof of this is by induction on k: Base case: k n hen B n In n, which has the desired form Inductive step: Assume the result is true for B k We show it is true for B k : In n Ik k B k H k H k H n H k B k H k Bk I k k I k k I τ k u H k u k Bk I k k I τ k u H k u k Bk I k k /τ k yk Bk u k I k k /τk yk u k /τ k B k u k yk I k k /τ k yk Ik k u k /τ k B k u k y k where y k uh k B k /τ k Bk

12 Algorithm: [A] FormQA, t A L A R Partition A and t where A L is na na and t while na R do Repartition A L A R t A a A 2 a α a 2 A 2 a 2 A 22 where α and τ are scalars α a 2 Update a 2 A 22 via the steps : α : /τ a 2 : a H 2A 22/τ A 22 : A 22 + a 2a 2 a 2 : a 2/τ I τ has na elements and t u H 2 t τ t 2 A 22 Continue with A L A R endwhile A a A 2 a α a 2 A 2 a 2 A 22 and t t τ t 2 Figure 6: Algorithm for overwriting A with Q from the Householder transformations stored as Householder vectors below the diagonal of A as produced by [A, t] HouseholderQRA By the Principle of Mathematical Induction the result holds for B,, B n heorem 9 Given [A, t] HouseholderQRA from Figure 4, the algorithm in Figure 6 overwrites A with the first n na columns of Q as defined by the Householder transformations stored below the diagonal of A and in the vector t Proof: he algorithm is justified by the proof of Lemma 8 heorem Given A C m n the cost of the algorithm in Figure 6 is given by C FormQ m, n 2mn n3 flops 2

13 Algorithm: [y] ApplyQtA, t, y A L A R Partition A, t where A L is and t, y while n do Repartition A L A R t has elements A a A 2 a α a 2 A 2 a 2 A 22 where α, τ, and ψ are scalars Update ψ via the steps : I τ, and y, u H 2 t y ψ y B t τ t 2, and y y B y ψ ω : ψ + a H 2/τ ψ ψ ω : ω Continue with A L A R endwhile A a A 2 a α a 2 A 2 a 2 A 22, t t τ t 2, and y y B y ψ Figure 7: Algorithm for computing y : H n H y given the output from routine HouseholderQR Proof: Hence the proof for heorem 7 can be easily modified to establish this result Exercise If m n then Q could be accumulated by the sequence Q IH H H n Give a high-level reason why this would be much more expensive than the algorithm in Figure 6 5 Applying Q H In a future Note, we will see that the QR factorization is used to solve the linear least-squares problem o do so, we need to be able to compute ŷ Q H y where Q H H n H 3

14 Let us start by computing H y: I τ H ψ ψ H ψ /τ }{{} ψ ω u2 ω ψ ω ω More generally, let us compute H k y: I τ H y ψ y ψ ω ω, where ω ψ + u H 2 /τ his motivates the algorithm in Figure 7 for computing y : H n H y given the output matrix A and vector t from routine HouseholderQR he cost of this algorithm can be analyzed as follows: When y is of length k, the bulk of the computation is in an inner product with vectors of length m k to compute ω and an axpy operation with vectors of length m k to subsequently update ψ and hus, the cost is approximately given by n 4m k 4mn 2n 2 k Notice that this is much cheaper than forming Q and then multiplying 4