NUMERICS OF THE GRAM-SCHMIDT ORTHOGONALIZATION PROCESS Miro Rozložník Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic email: miro@cs.cas.cz joint results with Luc Giraud, Julien Langou and Jasper van den Eshof but also Alicja Smoktunowicz and Jesse Barlow! Seminarium Pierścienie, macierze i algorytmy numeryczne, Politechnika Warszawska, Warszawa, November 13, 2009 1
OUTLINE 1. HISTORICAL REMARKS 2. CLASSICAL AND MODIFIED GRAM-SCHMIDT ORTHOG- ONALIZATION 3. GRAM-SCHMIDT IN FINITE PRECISION ARITHMETIC: LOSS OF ORTHOGONALITY IN THE CLASSICAL GRAM- SCHMIDT PROCESS 4. NUMERICAL NONSINGULARITY AND GRAM-SCHMIDT ALGORITHM WITH REORTHOGONALIZATION 2
GRAM-SCHMIDT PROCESS AS QR ORTHOGONALIZATION A = (a 1,..., a n ) R m,n m rank(a) = n orthogonal basis Q of span(a) Q = (q 1,..., q n ) R m,n, Q T Q = I n A = QR, R upper triangular (A T A = R T R) 3
HISTORICAL REMARKS I origination of the QR factorization used for orthogonalization of functions: J. P. Gram: Über die Entwicklung reeller Funktionen in Reihen mittelst der Methode der Kleinsten Quadrate. Journal f. r. a. Math., 94: 41-73, 1883. algorithm of the QR decomposition but still in terms of functions: E. Schmidt: Zur Theorie der linearen und nichlinearen Integralgleichungen. I Teil. Entwicklung willkürlichen Funktionen nach system vorgeschriebener. Mathematische Annalen, 63: 433-476, 1907. name of the QR decomposition in the paper on nonsymmetric eigenvalue problem, rumor: the Q in QR was originally an O standing for orthogonal: J.G.F. Francis: The QR transformation, parts I and II. Computer Journal 4:265-271, 332-345, 1961, 1962. 4
HISTORICAL REMARKS II modified Gram-Schmidt (MGS) interpreted as an elimination method using weighted row sums not as an orthogonalization technique: P.S. Laplace: Theorie Analytique des Probabilités. Courcier, Paris, third edition, 1820. Reprinted in P.S. Laplace. (Evres Compeétes. Gauthier-Vilars, Paris, 1878-1912). classical Gram-Schmidt (CGS) algorithm to solve linear systems of infinitely many solutions: E. Schmidt: Über die Auflösung linearen Gleichungen mit unendlich vielen Unbekanten, Rend. Circ. Mat. Palermo. Ser. 1, 25 (1908), pp. 53-77. first application to finite-dimensional set of vectors: G. Kowalewski: Einfuehrung in die Determinantentheorie. Verlag von Veit & Comp., Leipzig, 1909. 5
CLASSICAL AND MODIFIED GRAM-SCHMIDT ALGORITHMS classical (CGS) modified (MGS) for j = 1,..., n for j = 1,..., n u j = a j u j = a j for k = 1,..., j 1 for k = 1,..., j 1 u j = u j (a j, q k )q k end q j = u j / u j end u j = u j (u j, q k )q k end q j = u j / u j end 6
CLASSICAL AND MODIFIED GRAM-SCHMIDT ALGORITHMS finite precision arithmetic: Q = ( q 1,..., q n ), Q T Q I n, I Q T Q? A Q R, A Q R? R?, cond( R)? classical and modified Gram-Schmidt are mathematically equivalent, but they have different numerical properties classical Gram-Schmidt can be quite unstable, can quickly lose all semblance of orthogonality 7
ILLUSTRATION, EXAMPLE Läuchli, 1961, Björck, 1967: A = 1 1 1 σ 0 0 0 σ 0 0 0 σ κ(a) = σ 1 (n + σ 2 ) 1/2 σ 1 n, σ 1 σ min (A) = σ, A = n + σ 2 assume first that σ 2 u, so fl(1 + σ 2 ) = 1 if no other rounding errors are made, the matrices computed in CGS and MGS have the following form: 8
ILLUSTRATION, EXAMPLE 1 0 0 σ 1 0 0 0 1 2 2 1 0 2 1 2, 1 0 0 σ 1 0 0 0 1 2 6 1 1 2 6 2 3 CGS: ( q 3, q 1 ) = σ/ 2, ( q 3, q 2 ) = 1/2, MGS: ( q 3, q 1 ) = σ/ 6, ( q 3, q 2 ) = 0 complete loss of orthogonality ( loss of lin. independence, loss of (numerical) rank ): σ 2 u (CGS), σ u (MGS) 9
GRAM-SCHMIDT PROCESS VERSUS ROUNDING ERRORS modified Gram-Schmidt (MGS): assuming ĉ 1 uκ(a) < 1 I Q T Q ĉ2uκ(a) 1 ĉ 1 uκ(a) classical Gram-Schmidt (CGS)? Björck, 1967, Björck, Paige, 1992 I Q T Q c 2uκ n 1 (A) 1 c 1 uκ n 1 (A)? Kielbasinski, Schwettlik, 1994 Polish version of the book, 2nd edition 10
TRIANGULAR FACTOR FROM CLASSICAL GRAM-SCHMIDT VS. CHOLESKY FACTOR OF THE CROSS-PRODUCT MATRIX exact arithmetic: r i,j = ( ) a j, q i = a j, a i i 1 k=1 r k,iq k r i,i = (a j, a i ) i 1 k=1 r k,ir k,j r i,i The computation of R in the classical Gram-Schmidt is closely related to the left-looking Cholesky factorization of the crossproduct matrix A T A = R T R 11
r i,j = fl(a j, q i ) = (a j, q i ) + e (1) = r i,i r i,j = a j, fl(a i i 1 k=1 q k r k,i ) + e (2) i + e (1) i,j r i,i a j, a i i 1 k=1 i,j q k r k,i + e (3) [ + r i,i (a j, e (2) i ) + e (1) i,j = (a i, a j ) i 1 k=1 i ] r k,i [ r k,j e (1) k,j ] + (a j, e (3) [ i ) + r i,i (a j, e (2) i ) + e (1) i,j ] 12
CLASSICAL GRAM-SCHMIDT PROCESS: COMPUTED TRIANGULAR FACTOR i k=1 r k,i r k,j = (a i, a j ) + e i,j, i < j [A T A + E 1 ] i,j = [ R T R] i,j! E 1 c 1 u A 2 The CGS process is another way how to compute a backward stable Cholesky factor of the cross-product matrix A T A! 13
CLASSICAL GRAM-SCHMIDT PROCESS: DIAGONAL ELEMENTS u j = (I Q j 1 Q T j 1 )a j u j = a j Q j 1 (Q T j 1 a j) = ( a j 2 Q T j 1 a j 2 ) 1/2 computing q j = u j ( a j Q T j 1 a j ) 1/2 ( a j + Q T j 1 a j ) 1/2, a j 2 = j k=1 r2 k,j + e j,j, e j,j c 1 u a j 2 J. Barlow, A. Smoktunowicz, Langou, 2006 14
CLASSICAL GRAM-SCHMIDT PROCESS: THE LOSS OF ORTHOGONALITY A T A + E 1 = R T R, A + E 2 = Q R R T (I Q T Q) R = ( E 2 ) T A A T E 2 ( E 2 ) T E 2 + E 1 assuming c 2 uκ(a) < 1 I Q T Q c 3uκ 2 (A) 1 c 2 uκ(a) 15
GRAM-SCHMIDT PROCESS VERSUS ROUNDING ERRORS modified Gram-Schmidt (MGS): assuming ĉ 1 uκ(a) < 1 I Q T Q ĉ2uκ(a) 1 ĉ 1 uκ(a) Björck, 1967, Björck, Paige, 1992 classical Gram-Schmidt (CGS): assuming c 2 uκ(a) < 1 I Q T Q c 3uκ 2 (A) 1 c 2 uκ(a)! Giraud, Van den Eshof, Langou, R, 2006 J. Barlow, A. Smoktunowicz, Langou, 2006 16
10 0 CGS x CHOL QR x MGS x CGS2 10 2 10 4 loss of orthgonality I Q k T Qk 10 6 10 8 10 10 10 12 10 14 10 16 1 2 3 4 5 6 7 8 logarithm of the condition number κ(a k ) Stewart, Matrix algorithms book, p. 284, 1998 17
GRAM-SCHMIDT ALGORITHMS WITH COMPLETE REORTHOGONALIZATION classical (CGS2) modified (MGS2) for j = 1,..., n for j = 1,..., n u j = a j u j = a j for i = 1,2 for i = 1,2 for k = 1,..., j 1 for k = 1,..., j 1 u j = u j (a j, q k )q k end end q j = u j / u j end u j = u j (u j, q k )q k end end q j = u j / u j end 18
GRAM-SCHMIDT PROCESS VERSUS ROUNDING ERRORS Gram-Schmidt (MGS, CGS): assuming c 1 uκ(a) < 1 I Q T Q c 2uκ 1,2 (A) 1 c 1 uκ(a) Björck, 1967, Björck, Paige, 1992 Giraud, van den Eshof, Langou, R, 2005 Gram-Schmidt with reorthogonalization (CGS2, MGS2): assuming c 3 uκ(a) < 1 I Q T Q c 4 u Hoffmann, 1988 Giraud, van den Eshof, Langou, R, 2005 19
ROUNDING ERROR ANALYSIS OF REORTHOGONALIZATION STEP u j = (I Q j 1 Q T j 1 )a j, v j = (I Q j 1 Q T j 1 )2 a j u j = r j,j σ min (R j ) = σ min (A j ) σ min (A) a j u j κ(a), u j v j = 1 A + E 2 = Q R, E 2 c 2 u A a j ū j κ(a) 1 c 1 uκ(a), ū j v j [1 c 2uκ(A)] 1 20
FLOPS VERSUS PARALLELISM classical Gram-Schmidt (CGS): mn 2 saxpys classical Gram-Schmidt with reorthogonalization (CGS2): 2mn 2 saxpys Householder orthogonalization: 2(mn 2 n 3 /3) saxpys in parallel environment and using BLAS2, CGS2 may be faster than (plain) MGS! Frank, Vuik, 1999, Lehoucq, Salinger, 2001 21
THANK YOU FOR YOUR ATTENTION! 22
THE ARNOLDI PROCESS AND THE GMRES METHOD WITH THE CLASSICAL GRAM-SCHMIDT PROCESS V n = [v 1, v 2,..., v n ] [r 0, AV n ] = V n+1 [ r 0 e 1, H n+1,n ] H n+1,n is an upper Hessenberg matrix Arnoldi process is a (recursive) column-oriented QR decomposition of the (special) matrix [r 0, AV n ]! x n = x 0 + V n y n, r 0 e 1 H n+1,n y n = min y r 0 e 1 H n+1,n y 23
THE GRAM-SCHMIDT PROCESS IN THE ARNOLDI CONTEXT: LOSS OF ORTHOGONALITY modified Gram-Schmidt (MGS): I V T n+1 V n+1 c 1 uκ([ v 1, A V n ]) classical Gram-Schmidt (CGS): I V T n+1 V n+1 c 2 uκ 2 ([ v 1, A V n ]) Björck, Paige 1967, 1992 Giraud, Langou, R, Van den Eshof 2004 24
CONDITION NUMBER IN ARNOLDI VERSUS RESIDUAL NORM IN GMRES The loss of orthogonality in Arnoldi is controlled by the convergence of the residual norm in GMRES: I V T n+1 V n+1 c α uκ α ([ v 1, A V n ]), α = 1, 2 Björck 1967, Björck and Paige, 1992 Giraud, Langou, R, Van den Eshof 2003 κ([ v 1, A V n ]) [ v 1,A V n ] ˆrn r 0 [1+ ŷ n 2 1 δn 2 ] 1/2 ˆr n r 0 = v 1 A V n ŷ n = min y v 1 A V n y, δ n = σ n+1([ v 1,A V n ]) σ n (A V n ) < 1 Paige, Strakoš, 2000-2002 Greenbaum, R, Strakoš, 1997 25
THE GMRES METHOD WITH THE GRAM-SCHMIDT PROCESS The total loss of orthogonality (rank-deficiency) in the Arnoldi process with Gram-Schmidt can occur only after GMRES reaches its final accuracy level: modified Gram-Schmidt (MGS): ˆr n r 0 [1+ ŷ n 2 c 1[ v 1, A V n ] u 1 δn 2 ] 1/2 classical Gram-Schmidt (CGS): ˆr n r 0 [1+ ŷ n 2 [ c 2 [ v 1, A V n ] u ] 1/2 1 δn 2 ] 1/2 26
10 2 implementations of the GMRES method 10 0 10 2 ADD1(4960) 10 4 loss of orthogonality 10 6 10 8 10 10 10 12 10 14 10 16 10 18 0 50 100 150 iteration number 27
10 2 implementations of the GMRES method 10 0 10 2 ADD1(4960) relative true residual norms 10 4 10 6 10 8 10 10 10 12 10 14 10 16 10 18 0 50 100 150 iteration number 28
LEAST SQUARES PROBLEM WITH CLASSICAL GRAM-SCHMIDT b Ax = min u b Au, r = b Ax A T Ax = A T b r = (I Q Q T )b + e 1 ( R + E 3 ) x = Q T b + e 2 e 1, e 2 c 0 u b, E 3 c 0 u R 29
LEAST SQUARES PROBLEM WITH CLASSICAL GRAM-SCHMIDT R T ( R + E 3 ) x = ( Q R) T b + R T e 2 (A T A + E 1 + R T E 3 ) x = (A + E 2 ) T b + R T e 2 (A T A + E) x = A T b + e E c 4 u A 2, e c 4 u A b 30
LEAST SQUARES PROBLEM WITH CLASSICAL GRAM-SCHMIDT r r b κ(a)(2κ(a) + 1) c 5 u [1 c 1 )uκ 2 (A)] 1/2 x x x κ 2 (A) ( 2 + r ) A x c 5 u 1 c 1 uκ 2 (A) The least square solution with classical Gram-Schmidt has the same forward error bound as the normal equation method: R Q T A = R R T (A + E 2 ) T A = R T [ E 1 + ( E 2 ) T A] Björck, 1967 31
LEAST SQUARES PROBLEM WITH BACKWARD STABLE QR FACTORIZATION r r b (2κ(A) + 1)c 6 u x x x κ(a) [ 2 + (κ(a) + 1) r ] A x c 6 u 1 c 6 uκ(a) Householder QR factorization, modified Gram-Schmidt Wilkinson, Golub, 1966, Björck, 1967 32