QR decomposition: History and its Applications

Transcription

1 Mathematics & Statistics Auburn University, Alabama, USA Dec 17, 2010 decomposition: and its Applications Tin-Yau Tam èuî Æâ w f ŒÆêÆ ÆÆ Page 1 of 37 tamtiny@auburn.edu Website: tamtiny

2 1. decomposition Recall the decomposition of A GL n (C): A = where Q GL n (C) is unitary and R GL n (C) is upper with positive diagonal entries. Such decomposition is unique. Set a(a) := diag (r 11,..., r nn ) where A is written in column form A = (a 1 a n ) Page 2 of 37 Geometric interpretation of a(a): r ii is the distance (w.r.t. 2-norm) between a i and span {a 1,..., a i 1 }, i = 2,..., n.

3 Example: /7 69/175 58/ = 3/7 158/175 6/ /7 6/35 33/ decomposition is the matrix version of the Gram-Schmidt orthonormalization process. decomposition can be extended to rectangular matrices, i.e., if A C m n with m n (tall matrix) and full rank, then A = where Q C m n has orthonormal columns and R C n n is upper with positive diagonal entries. Page 3 of 37

4 2. history When Erhard Schmidt presented the formulae on p. 442 of his E. Schmidt, Zur Theorie der linearen und nichtlinearen Integralgleichungen. I. Teil: Entwicklung willkülicher Funktionen nach Systemen vorgeschriebener, Math. Ann., 63 (1907) he said that essentially the same formulae were in J. P. Gram, Ueber die Entwickelung reeler Funtionen in Reihen mittelst der Methode der kleinsten Quadrate, Jrnl. für die reine und angewandte Math. 94 (1883) Page 4 of 37 Modern writers, however, distinguish the two procedures, sometimes using the term Gram-Schmidt for the Schmidt form and modified Gram- Schmidt for the Gram version.

5 But Gram-Schmidt orthonormalization appeared earlier in the work of Laplace and Cauchy. In the theory of semisimple Lie grougs, Gram-Schmidt process is extended to the Iwasawa decomposition G = KAN. A JSTOR search: the term Gram-Schmidt orthogonalization process first appears on p.57 of Y. K. Wong, An Application of Orthogonalization Process to the Theory of Least Squares, Annals of Mathematical Statistics, 6 (1935), Page 5 of 37

6 In 1801 Gauss predicted the orbit of the steroid Ceres using the method of least squares. Since then, the principle of least squares has been the standard procedure for the analysis of scientific data. Least squares problem i.e., finding ˆx that would yield Ax b min x Ax b 2. The solution is characterized by r R(A), where r = b Ax is the residual vector, or equivalently, given by the normal equation A Ax = A b. Page 6 of 37 With A = and A C m n tall with full rank R Rx = () x = R Q b Rx = Q b.

7 There are several methods for computing the decomposition: GS or modified GS, Givens rotations (real A), or Householder reflections. Disadvantage of GS: sensitive to rounding error (orthogonality of the computed vectors can be lost quickly or may even be completely lost) modified Gram-Schmidt. Page 7 of 37 Idea of modified GS: do the projection step with a number of projections which will be against the errors introduced in computation.

8 Example: A = 1 + ɛ ɛ ɛ with very small ɛ such that 3+2ɛ will be computed accurately but 3+2ɛ+ɛ 2 will be computed as 3 + 2ɛ. For example, ɛ < Then Q = 1+ɛ ɛ ɛ ɛ 0 2 and cos θ 12 and cos θ 13 π/2 but and cos θ 23 π/3. Page 8 of 37 See for a heuristic analysis of why Gram-Schmidt not stable.

9 Computing by Householder reflections A Householder reflection is a reflection about some hyperplane. Consider Q v = I 2vv, v 2 = 1. Q v sends v to v and fix pointwise the hyperplane to v. Householder reflections are Hermitian and unitary. Let e 1 = (1, 0,..., 0) T. Recall If a 1 2 = α 1, set A = (a 1 a n ) GL n (C) u = a 1 α 1 e 1, v = u u 2, Q 1 = I 2vv Page 9 of 37 so that Qa 1 = (α 1, 0,, 0) T = α 1 e 1.

10 Then Q 1 A = A 1 0 α 1 After t iterations of this process, t n 1, R = Q t Q 2 Q 1 A is upper. So, with Q = Q 1 Q 2 Q t A = is the decomposition of A. Page 10 of 37 This method has greater numerical stability than GS. On the other hand, GS produces the q j vector after the jth iteration, while Householder reflections produces all the vectors only at the end.

11 3. An Theorem 3.1. (Huang and Tam 2007) Given A GL n (C). Let A = Y 1 JY be the Jordan decomposition of A, where J is the Jordan form of A, diag J = diag (λ 1,..., λ n ) satisfying λ 1 λ n. Then lim m a(am ) 1/m = diag ( λ ω(1),..., λ ω(n) ), where the permutation ω is uniquely determined by the Gelfand-Naimark decomposition of Y = LωU: Page 11 of 37 rank ω(i j) = rank (Y )(i j), 1 i, j n. Here ω(i j) denotes the submatrix formed by the first i rows and the first j columns of ω, 1 i, j n.

12 Gelfand-Naimark decomposition of Y = LωU is different from the Guassian decomposition Y = P T LU obtained by Gaussian elimination with row exchanges. None of P, U and L in the Gaussian decomposition is unique. But ω and diag U are unique in the Gelfand-Naimark decomposition. H. Huang and T.Y. Tam, An asymptotic behavior of decomposition, Linear Algebra and Its Applications, 424 (2007) H. Huang and T.Y. Tam, An asymptotic result on the a-component in Iwasawa decomposition, Journal of Lie Theory, 17 (2007) Page 12 of 37

13 Numerical experiments: Computing the discrepancy between [a(a m )] 1/m and λ(a) of randomly generated A GL n (C). The graph of 100 [a(a m )] 1/m diag ( λ 1,..., λ n ) 2 80 versus m (m = 1,..., 100) 60 Page 13 of

14 If we consider a 1 (A m ) 1/m λ 1 (A) instead of [a(a m )] 1/m diag ( λ 1,..., λ n ) 2 for the above example, convergence occurs. The graph of a 1 (A m ) 1/m λ 1 (A) versus m (m = 1,..., 100) 30 Page 14 of

15 4. Because of Abel s theorem (1824), the roots of a general fifth order polynomial cannot be solved by radicals. Thus the computation of eigenvalues of A C n n has to be approximative. Given A GL n (C), define a sequence {A k } k N of matrices with A 1 := A = Q 1 R 1 and if A k = Q k R k,, k = 1, 2,... then A k+1 := R k Q k = Q kq k R k Q k = Q ka k Q k Page 15 of 37 So the eigenvalues are fixed in the process. One hopes to have some sort of convergence of the sequence {A k } k N so that the limit would provide the eigenvalue approximation of A.

16 Theorem 4.1. (Francis (1961/62), Kublanovskaja (1961), Huang and Tam (2005)) Suppose that the moduli of the eigenvalues λ 1,..., λ n of A GL n (C) are distinct: Let λ 1 > λ 2 > > λ n (> 0). A = Y 1 diag (λ 1,..., λ n )Y. Assume Y = LωU, where ω is a permutation, L is lower and U is unit upper. Then 1. the strictly lower part of A k converge to zero. 2. diag A k diag (λ ω(1),..., λ ω(n) ). H. Huang and T.Y. Tam, On the s of real matrices, Linear Algebra and Its Applications, 408 (2005) Page 16 of 37

17 It rates as one of the most important algorithmic developments of the past century Parlett (2000): The algorithm solves the eigenvalue problem in a very satisfactory way... What makes experts in matrix computations happy is that this algorithm is a genuinely new contribution to the field of numerical analysis and not just a refinement of ideas given by Newton, Gauss, Hadamard, or Schur. Higham (2003): The algorithm for solving the nonsymmetric eigenvalue problem is one of the jewels in the crown of matrix computations. Page 17 of 37

18 A common misconception Horn and Johnson s Matrix Analysis (p.114): Under some circumstances (for example, if all the eigenvalues of A 0 has distinct absolute values), the iterates A k will converge to an upper triangular matrix as k... Quarteroni, Sacco, and Saleri s Numerical Mathematics (p ): Let A R n n be a matrix with real eigenvalues such that λ 1 > λ 2 > > λ n. Then lim T (k) = k λ 1 t t 1n 0 λ 2 t λ n Wikipedia: Under certain conditions, the matrices A k converge to a triangular matrix, the Schur form of A. Page 18 of 37

19 D. Serre s Theory of Matrices (p ): Let us recall that the sequence A k is not always convergent. For example, if A is already triangular, its factorization is Q = D, R = D 1 A, with d j = a jj / a jj. Hence, A 1 = D 1 AD is triangular, with the same diagonal as that of A. By induction, A k is triangular, with the same diagonal as that of A so that A k = D k AD k... Hence, the part above the diagonal of A k does not converge. Summing up, a convergence theorem may concern only the diagonal of A k and what is below it. Page 19 of 37

20 More on algorithm The algorithm is numerically stable because it proceeds by orthogonal/unitary similarity transforms. A Hessenberg form algorithm It would be cost effective if we convert A to an upper Hessenberg form, with a finite sequence of orthogonal similarities. Then determine the decomposition of an upper Hessenberg matrix costs 6n 2 +O(n) arithmetic operations. A practical algorithm will use shifts to increase separation and accelerate convergence. The is further extended to semisimple Lie groups: H. Huang. R.R. Holmes and T.Y.Tam, Asymptotic behavior of Iwasawa and Cholesky iterations, manuscript. Page 20 of 37

21 5. Application in MIMO In radio, multiple-input and multiple-output, or MIMO, is the use of multiple antennas at both the transmitter and receiver to improve communication performance. Page 21 of 37

22 MIMO is one of several forms of smart antenna technology. MIMO offers significant increases in data throughput and link range without additional bandwidth or transmit power. MIMO is an important part of modern wireless communication standards such as IE n (Wifi) and 4G. Page 22 of 37

23 In MIMO systems, a transmitter sends multiple streams by multiple transmit antennas. The transmit streams go through a matrix channel which consists of all N t N r paths between the N t transmit antennas at the transmitter and N r receive antennas at the receiver. Page 23 of 37 Then, the receiver gets the received signal vectors by the multiple receive antennas and decodes the received signal vectors into the original information.

24 Mathematical description Let x = (x 1,..., x n ) T be a vector to be transmitted over a noisy channel. Each x i is chosen from a finite-size alphabet X. A general MIMO system is modeled as r = Hx + ξ H is the m n full column rank channel (tall, i.e., m n) matrix (known to the receiver) ξ = (ξ 1,..., ξ m ) T is a white Gaussian noise vector where E(ξξ ) = σ 2 I r = (r 1,..., r m ) T is the observed received vector. Our task is to detect/estimate the vector ˆx = (ˆx 1,..., ˆx n ) T X n given the noisy observation r. Page 24 of 37 decomposition of the channel matrix H can be used to form the back-cancellation detector.

25 A. Successive Cancellation Detection Using Decomposition (1) decomposition Let H = be the decomposition of the m n channel matrix H, i.e., Q = m n matrix with orthonormal columns R = n n upper matrix. So r = Hx + ξ Q r = Rx + Q ξ Set r := Q r, ξ := Q ξ So r 1 r 2. = r 11 r 12 r 1n x 1 ξ 1 0 r 22 r 2n x ξ 2. Page 25 of 37 r n 0 0 r nn x n ξ n

26 (2) Hard decision Estimate x n by making the hard decision: ˆx n := Quant [ r n r nn ] where Quant (t) is the element X that is closest w.r.t. 2-norm to t. (3) Cancellation Backward substitutions: ˆx n := Quant [ r n r nn ] ˆx k := Quant [ r k n i=k+1 r kmˆx i r kk ], k = n 1, n 2,..., 1 The algorithm is essentially least squares solution via decomposition. Page 26 of 37

27 B. Optimally Ordered Detection Golden et al proposed a vertical Bell Laboratories layered space-time (V-BLAST) system with an optimal ordered detection algorithm that maximizes the signal-to-noise ratio (SNR). G.D. Golden, G.J. Foschini, R.A. Valenzuela, and P.W. Wolniansky, Detection algorithm and initial laboratory results using V-BLAST spacetime communication architecture, Electron. Lett., vol. 35, pp. 14õ15, Jan The idea is (equivalently) to find an n n permutation matrix P such that if H := HP = then the distance between h n and the other columns h 1,..., h n 1 is maximal. Then repeat the process by stripping off the column vectors one. Page 27 of 37

28 Doing so amounts to finding a subchannel whose SNR is the highest among all n possible subchannels. Geometrically it is to find the column of H whose distance to the span of the other columns of H is maximum. Then do it recursively. Equivalently r = H x + ξ where H := HP, x = Px i.e., if we precode a vector with the permutation matrix P, and apply Algorithm A to detect x, we get the optimally ordered successive-cancellation detector of Golden et al. Page 28 of 37

29 A recent optimal decomposition, called equal-diagonal decomposition, or briefly the S decomposition in introduced in J.K. Zhang, A. Kavčić and K. M. Wong, Equal-Diagonal Decomposition and its Application to Precoder Design for Successive-Cancellation Detection, IE Transactions on Information Theory, 51 (2005) The S decomposition is applied to precode successive-cancellation detection, where we assume that both the transmitter and the receiver have perfect channel knowledge. Page 29 of 37

30 Theorem 5.1. (Zhang, Kavčić and Wong 2005) For any channel matrix H, there exists a unitary precoder S, such that the nonzero diagonal entries of the upper matrix R are all equal where HS =. Nice properties The precoder S and the resulting successive-cancellation detector have many nice properties. The minimum Euclidean distance between two signal points at the channel output is equal to the minimum Euclidean distance between two constellation points at the precoder input up to a multiplicative factor that equals the diagonal entry in the R-factor. Page 30 of 37

31 The superchannel HS naturally exhibits an optimally ordered column permutation, i.e., the optimal detection order for the vertical Bell Labs layered spaceõtime (V-BLAST) detector is the natural order. The precoder S minimizes the block error probability of the successive cancellation detector. A lower and an upper bound for the free distance at the channel output is expressible in terms of the diagonal entries of the R-factor in the decomposition of a channel matrix. The precoder S maximizes the lower bound of the channels free distance subject to a power constraint. For the optimal precoder S, the performance of the detector is asymptotically (at large signal-to-noise ratios (SNRs)) equivalent to that of the maximum-likelihood detector (MLD) that uses the same precoder. Page 31 of 37

32 Recall the S decomposition HS = Let H = UΣV s 1 be the SVD of H where Σ =... s n. Then set S = V Ŝ O ( ) HS = HV diag (s 1,..., s n )Ŝ Ŝ = UΣŜ = U 0 where the unitary Ŝ is to be determined so that R has equal diagonal entries. Clearly the R factors of ΣŜ and UΣŜ are the same. WLOG, assume that m = n, i.e., Σ = diag (s 1,..., s n ). The problem is reduced to finding a unitary S such that Page 32 of 37 and R has identical diagonal entries. ΣS =

33 Theorem 5.2. (Kostant 1973) Let Σ = s 1... s n C n n s 1 s n > 0. For any unitary S C n n, if ΣS =, where Q C m n has orthonormal columns, R C n n is upper, then k i=1 r ii n r ii = i=1 k s i, k = 1,..., n 1 (1) i=1 n s i, (2) i=1 where r 11 r nn are the rearrangement of r 11,..., r nn. Conversely if (1) and (2) are satisfied, then there is an n n unitary S such that the diagonal entries of R in ΣS = are those r ii, i = 1,..., n. Page 33 of 37

34 B. Kostant, On convexity, the Weyl group and Iwasawa decomposition. Ann. Sci. Ecole Norm. Sup. (4), 6 (1973) Clearly Kostant (1973) Zhang, Kavčić and Wong (2005). The proof of Kostant is not constructive. Indeed his result is true for semisimple Lie groups. The construction of the S precoder in [ZKW] is not very cost effective and involves a number of steps. Determinant is also involved. Shuangchi He and Tin-Yau Tam, On equal-diagonal decomposition, manuscript. Page 34 of 37

35 Use induction. WLOG assume that m = n. 2 2 case: ΣS = Then for any s 1 µ s 2, ( ) ( ) s 1 0 cos θ sin θ = 0 s 2 sin θ cos θ ( ) s 1 cos θ s 1 sin θ s 2 sin θ s 2 cos θ (s 1 cos θ, s 2 sin θ) T 2 = s 1 cos 2 θ + s 2 sin 2 θ = µ for some θ R. In particular µ = s 1 s 2 Suppose that the statement holds true for n 1. For r = n > 2, suppose that k is the smallest index such that s k 1 λ s k, where n λ := ( s i ) 1/n. i=1 Page 35 of 37

36 There is a unitary S 1 C 2 2 such that Set S 2 = S 1 I n 2. Then Note that diag (s 1, s k )S 1 = A 1 := ( λ 0 s 1s k λ A 2 := diag (s 1, s k, s 2,, s k 1, s k+1,, s n ) S 2 = A 1 diag (s 2,, s k 1, s k+1,, s n ). λ n 1 = s 1s k λ n i=2,i k By the inductive hypothesis, there exists a unitary S 3 C (n 1) (n 1) such that ( s1 s ) k A 3 = diag λ, s 2,, s k 1, s k+1,, s n S 3 is upper with equal-diagonal entries λ. Then S 4 = 1 S 3. s i. ). Page 36 of 37

37 T HAN K YOU FOR YOUR AT T EN T ION Page 37 of 37