Linear Algebra. Brigitte Bidégaray-Fesquet. MSIAM, September Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble.

Transcription

1 Brigitte Bidégaray-Fesquet Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble MSIAM, September 215

2 Overview 1 Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations 2 Power iteration algorithm Deflation Galerkin Jacobi QR 3 Direct methods Iterative methods Preconditioning 4 Band storage Sparse storage 5

3 Elementary operations on vectors Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations C n resp. R n : linear space of vectors with n entries in C resp. R. Generically: F n, where F is a field. Linear combination of vectors w = α u + β v, α, β C or R. f o r i = 1 to n w( i ) = a l p h a u ( i ) + b e t a v ( i ) Scalar product of 2 vectors P u v = n u i v i i=1 uv = f o r i = 1 to n uv = uv + u ( i ) v ( i ) l 2 norm of a vector s np u 2 = ui 2 i=1 uu = f o r i = 1 to n uu = uu + u ( i ) u ( i ) norm = s q r t ( uu )

4 Elementary operations on matrices Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations M np(f ): linear space of matrices with n p entries in F. Linear combination of matrices C = αa + βb, α, β F. f o r i = 1 to n f o r j = 1 to p C( i, j ) = a l p h a A( i, j ) + b e t a B( i, j ) Matrix vector product P w = A u, w i = p A ij u j j=1 f o r i = 1 to n wi = f o r j = 1 to p wi = wi + A( i, j ) u ( j ) w( i ) = wi Matrix matrix product P C = AB, C ij = p A ik B kj k=1 f o r i = 1 to n f o r j = 1 to q c i j = f o r k = 1 to p c i j = c i j + A( i, k ) B( k, j ) C( i, j ) = c i j

5 Gram Schmidt orthonormalization Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Let { v 1,..., v p} be a free family of vectors. It generates the vector space E p with dimension p. We want to construct { e 1,..., e p}, an orthonormal basis of E p. Gram Schmidt algorithm u 1 = v 1 e 1 = u 2 = v 2 ( v 2 e 1) e 1 e 2 = u p = v p p 1 P ( v p e k ) e k e p = k=1 u 1 u 1 2 u 2 u 2 2 u p u p 2 u 2 e 2 v 2 e 1 ( v 2 e 1) e 1 u 1 = v 1

6 Matrix norms Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Definition A, A M nn(f ), F = C or R. A = A =. λa = λ A, A M nn(f ), λ F. A + B A + B, AB A B, Subordinate matrix norms A, B M nn(f ) (triangle inequality). A, B M nn(f ) (specific for matrix norms). s Ax p A p = max = max Ax p, x F n, where x p = x p x p p x p=1 P P in particular: A 1 = max j i A ij and A = max i j A ij. np i=1 x p i. Matrix-vector product estimate A p Ax p x p and hence Ax p A p x p for all x F n.

7 Matrix conditioning Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Definition Cond(A) = A 1 A. Properties Cond(A) 1, Cond(A 1 ) = Cond(A), Cond(αA) = Cond(A). For the Euclidian norm Cond 2(A) = λmax λ min.

8 Conditioning and Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Problem (S ) A x = b, (S per) (A + δa)( x + δ x) = ( b + δ b). (S per) (S ): Aδ x + δa( x + δ x) = δ b, δ x = A δ 1 b δa( x + δ x), δ x A 1 δ b δa( x + δ x) (for a subordinate matrix norm), δ x A 1 δ b + δa x + δ x, δ x x + δ x δ A 1! b x + δ x + δa. Result δ x x + δ x Cond(A) δ! b A x + δ x + δa. A relative error on x = Cond(A) (relative error on b + relative error on A).

9 Hermitian, orthogonal... Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Transposed matrix: ( t A) ij = A ji. Adjoint matrix: (A ) ij = A ji. Symmetric matrix t A = A. Hermitian matrix A = A and hence t A = Ā. Orthogonal matrix (in M nn(r)) t AA = I. Unitary matrix (in M nn(c)) A A = I. Similar matrices ( semblables in French) A and B are similar if P/B = P 1 AP.

10 Profiles Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Lower triangular Upper triangular Tridiagonal Lower Hessenberg Upper Hessenberg

11 Householder matrices Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Definition H v = I 2 v t v v 2 2 Properties 1 H v is orthogonal. 2 If v = a b and a 2 = b 2, then H v a = b. t v v = a 2 2 t a b + b 2 = 2 a 2 2 t a b = 2 t a v = 2 t v a H v a = a 2 v t v a v 2 = a v = b. Application Let a K n, we look for H v such that H v a = t (α,,..., ). Solution: take b = t (α,,..., ) with α = a 2, and v = a b. Then H v a = b.

12 Householder tridiagonalisation Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Aim A: symmetric matrix. Construct a sequence A (1) = A,..., A (n) tridiagonal and A (n) n = HA t H. A (2) = A (3) = A (4) = A (5) =... A (n) = First step A (1) A(1) 11 a (1) 21 t a (1) 12 à (1)! «H (1) 1 t A (2) H(1) A (1) 11 H (1) a (1) 21! t ( H (1) a (1) 21 ) H (1). (1)t à H(1) Choose H (1) such that H (1) a (1) 21 = t (α,,..., ) n 1 = α( e 1) n 1. α = a (1) 21 2, u1 = a(1) 21 α( e1)n 1, H (1) = H u1. Complexity Order 2 3 n3 products.

13 Givens tridiagonalization Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations c s Let G pq(c, s) = 1 with c 2 + s 2 = 1. B s c 1 A 1 t GAG = If A is symmetric: Complexity... else: leads to Hessenberg matrix Order 4 3 n3 products.

14 Principles of LU factorization Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations = 1 A L U Some regular matrix (with non-zero determinant) are not LU-transformable, e.g. ([ 1; 1 1]) is not. If it exists, the LU decomposition of A is not unique. It is unique if A is non-singular. A is non-singular and LU-transformable all the determinants of the fundamental principal minors are non zero (and in this case the decomposition is unique).

15 Doolittle LU factorization principle Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations It proceeds line by line. 8 A 11 = L 11U 11 >< A 12 = L 11U 12 L 11 = 1... >: A 1n = L 11U 1n {U 1j } j=1,...,n 8 >< >: 8 >< >: A 21 = L 21U 11 L 21 A 22 = L 21U 12 + U {U 2j } j=2,...,n A 2n = L 21U 1n + U 2n A 31 = L 31U 11 L 31 A 32 = L 31U 12 + L 32U 22 L 32 A 33 = L 31U 13 + L 32U 23 + U {U 3j } j=3,...,n A 3n = L 31U 1n + L 32U 2n + U 3n...

16 Doolittle LU factorization algorithm Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Doolittle algorithm L ij = A ij j 1 P k=1 U jj L ik U kj f o r i = 1 to n f o r j = 1 to i 1 sum = f o r k=1 to j 1 sum = sum + L ( i, k ) U( k, j ) L ( i, j ) = (A( i, j ) sum )/U( j, j ) L ( i, i ) = 1 f o r j = i to n sum = f o r k = 1 to i 1 sum = sum + L ( i, k ) U( k, j ) U( i, j ) = A( i, j ) sum U ij = A ij i 1 P k=1 L ik U kj Complexity Order n 3 products

17 Cholesky factorization for an Hermitian matrix Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Principle A = C t C Cholesky algorithm s C ii = A ii i 1 P C ik C ik C ij = k=1 C( 1, 1 ) = s q r t (A( 1, 1 ) ) f o r i = 2 to n f o r j = 1 to i 1 sum = f o r k = 1 to j 1 sum = sum + C( i, k ) C( j, k ) C( i, j ) = (A( i, j ) sum )/C( j, j ) sum= f o r k = 1 to i 1 sum = sum + C( i, k ) C( i, k ) C( i, i ) = s q r t (A( i, i ) sum ) j 1 P A ij k=1 C ik C jk, j i C jj Complexity Order n 3 products

18 LU factorization profiles Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Non-zero elements In blue: A In red: superposition of L and U The interior of the profile is filled!

19 QR factorization principle Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Principle A = QR, where Q orthogonal and R right (upper) triangular. G m... G 2G 1A = R, A = t G t 1 G 2... t G {z m R } Q c s Let G ij (c, s) = 1 with c 2 + s 2 = 1. B s c 1 A 1 (GA) ij = sa jj + ca ij A jj c = s = = (i, j) q A 2 ij + A2 jj A ij q A 2 ij + A2 jj

20 QR factorization algorithm Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations Algorithm R = A Q = I d // s i z e o f A f o r i = 2 to n f o r j = 1 to i 1 r o o t = s q r t (R( i, j ) R( i, j )+R( j, j ) R( j, j ) ) i f r o o t!= c = R( j, j )/ r o o t s = R( i, j )/ r o o t e l s e c = 1 s = end i f C o n s t r u c t G j i R = G j i R // m a t r i x p r o d u c t Q = Q t r a n s p o s e ( G j i ) // m a t r i x p r o d u c t Complexity Order n 3 products

21 QR factorization Python example Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning Specific matrices Tridiagonalisation LU and QR factorizations A = B A R = B A Q = B

22 Power iteration algorithm Python experiment Power iteration algorithm Deflation Galerkin Jacobi QR : Construct the series x = A = λ 1 = 1, λ 2 = 1, v 1 = 1 «9 1 x k = A x k 1 ««1, v 1 2 =. 1 ««««2, x 1 2 =, x 2 2 =, x 3 2 = x tends to the direction of the eigenvector associated to the higher modulus eigenvalue. x k / x k 1 tends to the higher modulus eigenvalue.

23 Power iteration algorithm Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Computation of the eigenvalue with higher modulus. A may be diagonalizable or not, the dominant eigenvalue can be unique or not. Algorithm choose q ( ) f o r k = 1 to c o n v e r g e n c e x ( k ) = A q ( k 1) q ( k ) = x ( k ) / norm ( x ( k ) ) lambdamax = x ( k ) ( j )/ q ( k 1)( j ) Attention: good choice of component j.

24 Power iteration algorithm Python example Power iteration algorithm Deflation Galerkin Jacobi QR 1 1 A 5 A 2 Rotations: cos(1) 1 sin(1) 1 1 R 1 1 A, R 2 cos(2) sin(2) A sin(1) cos(1) sin(2) cos(2) B = R 2R 1A t R t 1 R A : λ 1 = 2, λ 2 = 5, λ 3 = 1, v 1 v 2 A, v 3 A..352

25 Power iteration algorithm Remarks Power iteration algorithm Deflation Galerkin Jacobi QR 1 Convergence results depend on the fact that the matrix is diagonalizable or not the dominant eigenvalue is multiple or not 2 The choice of the norm is not explicit: usually max norm or euclidian norm 3 q should not be orthogonal to the eigen-subspace associated to the dominant eigenvalue.

26 Inverse iteration algorithm Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Computation of the eigenvalue with smallest modulus. A may be diagonalizable or not, the dominant eigenvalue can be unique or not. Based on the fact that 1 λ min(a) = λ max(a ) 1. Algorithm choose q ( ) f o r k = 1 to c o n v e r g e n c e s o l v e A x ( k ) = q ( k 1) q ( k ) = x ( k ) / norm ( x ( k ) ) lambdamin = q ( k 1)( j ) / x ( k ) ( j )

27 Inverse iteration algorithm Python example Power iteration algorithm Deflation Galerkin Jacobi QR 1 1 A 5 A 2 Rotations: cos(1) 1 sin(1) 1 1 R 1 1 A, R 2 cos(2) sin(2) A sin(1) cos(1) sin(2) cos(2) B = R 2R 1A t R t 1 R A : λ 1 = 2, λ 2 = 5, λ 3 = 1, v 1 v 2 A, v 3 A..352

28 Generalized inverse iteration algorithm Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Computation of the closest eigenvalue to a given µ. The eigenvalues of A µi are the λ i µ, where λ i are the eigenvalues of A. apply the inverse iteration algorithm to A µi. Algorithm choose q ( ) f o r k = 1 to c o n v e r g e n c e s o l v e (A mu I ) x ( k ) = q ( k 1) q ( k ) = x ( k ) / norm ( x ( k ) ) lambda = q ( k 1)( j ) / x ( k ) ( j ) + mu

29 Generalized inverse iteration algorithm Python example Power iteration algorithm Deflation Galerkin Jacobi QR 1 1 A 5 A µ = 4. 2 Rotations: cos(1) 1 sin(1) 1 1 R 1 1 A, R 2 cos(2) sin(2) A sin(1) cos(1) sin(2) cos(2) B = R 2R 1A t R t 1 R A : λ 1 = 2, λ 2 = 5, λ 3 = 1, v 1 v 2 A, v 3 A..352

30 Deflation Algorithm and Python example Power iteration algorithm Deflation Galerkin Jacobi QR Computation of all the eigenvalues in modulus decreasing order. When an eigenelement (λ, q) is found, it is removed from further computation by replacing A A λ q t q. Algorithm f o r i = 1 to n choose q ( ) f o r k = 1 to c o n v e r g e n c e x ( k ) = A q ( k 1) q ( k ) = x ( k ) / norm ( x ( k ) ) lambda = x ( k ) ( j ) / q ( k 1)( j ) A = A lambda q t r a n s p o s e ( q ) // e l i m i n a t e s d i r e c t i o n q

31 Galerkin method Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Let H be a subspace of dimension m, generated by the orthonormal basis ( q 1,..., q m). Construct the rectangular matrix Q = ( q 1,..., q m). Remark: Q Q = Id m Goal Look for in H. P If u H, u = m α i q i (unique). i=1 u = Q U, where U = t (α 1,..., α m). A u = λ u AQ U = λq U. Project on H: Q AQ U = λq Q U = λ U. We look for eigenelements of B = Q AQ. Vocabulary: {λ i, u i } are the Ritz elements, B is the Rayleigh matrix.

32 Jacobi method Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Goal Diagonalize the (real symmetric) matrix. Until a reasonably diagonal matrix is obtained: Choose the largest off-diagonal element (largest modulus) Construct a rotation matrix that annihilates this term In the end, the eigenvalues are the diagonal elements.

33 QR method Algorithm Power iteration algorithm Deflation Galerkin Jacobi QR Algorithm A( 1 ) = A f o r k = 1 to c o n v e r g e n c e [Q( k ),R( k ) ] = Q R f a c t o r (A( k ) ) A( k+1) = R( k ) Q( k ) The eigenvalues are the diagonal elements of the last matrix A k+1. Properties A k+1 = R k Q k = Q k Q k R k Q k = Q k A k Q k A k+1 and A k are similar. If A k is tridiagonal or Hessenberg, A k+1 also is First restrict to this case keeping similar matrices.

34 QR method Convergence and Python example Power iteration algorithm Deflation Galerkin Jacobi QR Theorem Let V be the matrix of left of A (A u = λ u ). If the principal minors of V are non-zero. the eigen-values of A are such that λ 1 > > λ n. Then the QR method converges A k+1 tends to an upper triangular form and (A k ) ii tends to λ i.

35 Eigenvalues Summary Power iteration algorithm Deflation Galerkin Jacobi QR We want to know all the eigenvalues QR method better than Jacobi Preprocessing: find a similar tridiagonal or Heisenberg matrix (Householder or Givens algorithm). We only want one eigenvector whose eigenvalue is known (or an approximation) Power iteration algorithm and variants... We only want a sub-set of eigenelements We know the eigenvalues and look for : deflation and variants We know the subspace for : Galerkin and variants

36 Principles Direct methods Iterative methods Preconditioning A x = b Elimination methods The solution to the system remains unchanged if lines are permuted, line i is replaced by a linear combination P l i n µ k l k, with µ i. k=1 Factorisation methods A = LU LU x = b We solve two triangular systems L y = b U x = y.

37 Lower triangular matrix Direct methods Iterative methods Preconditioning Algorithm x i = b i i 1 P i f A(1,1)== then s t o p x ( 1 ) = b ( 1 ) /A( 1, 1 ) f o r i = 2 to n i f A( i, i )== then s t o p ax = f o r k = 1 to i 1 ax = ax + A( i, k ) x ( k ) x ( i ) = ( b ( i ) ax )/A( i, i ) k=1 A ii A ik x k Complexity Order n 2 /2 products.

38 Upper triangular matrix Direct methods Iterative methods Preconditioning Algorithm x i = b i i f A( n, n)== then s t o p x ( n ) = b ( n )/A( n, n ) f o r i = n 1 to 1 i f A( i, i )== then s t o p ax = f o r k = i +1 to n ax = ax + A( i, k ) x ( k ) x ( i ) = ( b ( i ) ax )/A( i, i ) n P k=i+1 A ii A ik x k Complexity Order n 2 /2 products.

39 Gauss elimination Principle Direct methods Iterative methods Preconditioning Aim Transform A to upper triangular matrix. At rank p 1: l i l i A ip l p A pp A ij = if i > j, j < p. f o r p = 1 to n p i v o t = A( p, p ) i f p i v o t == s t o p l i n e ( p ) = l i n e ( p )/ p i v o t f o r i = p+1 to n Aip = A( i, p ) l i n e ( i ) = l i n e ( i ) Aip l i n e ( p ) x = s o l v e (A, b ) // upper t r i a n g u l a r Complexity Still order n 3 products.

40 Gauss Jordan elimination Principle Direct methods Iterative methods Preconditioning Aim Transform A to identity. At rank p 1: l i l i A ip l p A pp A ii = 1 if i < p, A ij = if i j, j < p. f o r p = 1 to n p i v o t = A( p, p ) i f p i v o t == s t o p l i n e ( p ) = l i n e ( p )/ p i v o t f o r i = 1 to n, i!=p Aip = A( i, p ) l i n e ( i ) = l i n e ( i ) Aip l i n e ( p ) x = b Attention take into account le right-hand side in the line. what if A pp =?

41 Gauss Jordan elimination Algorithm Direct methods Iterative methods Preconditioning // unknown e n t r i e s numbering f o r i = 1 to n num( i ) = i f o r p=1 to n // maximal p i v o t pmax = abs (A( p, p ) ) imax = p jmax = p f o r i = p to n f o r j = p to n i f abs (A( i, j ) ) > pmax then pmax = abs (A( i, j ) ) imax = i jmax = j end i f // l i n e p e r m u t a t i o n f o r j = p to n permute (A( p, j ),A( imax, j ) permute ( b ( p ), b ( imax ) ) // column p e r m u t a t i o n f o r i = p to n permute (A( i, p ),A( i, jmax ) permute (num( p ), num( jmax ) ) p i v o t = A( p, p ) i f pivot == stop, rank (A) = p 1 f o r j = p to n A( p, j ) = A( p, j )/ p i v o t b ( p ) = b ( p )/ p i v o t f o r i = 1 to n, i!=p Aip = A( i, p ) f o r j = p to n A( i, j ) = A( i, j ) Aip A( p, j ) b ( i ) = b ( i ) Aip b ( p ) // l o o p on p f o r i = 1 to n x (num( i ) ) = b ( i ) Complexity Order n 3 products. Remark Also computes the rank of the matrix.

42 Factorization methods Thomas algorithm principle Direct methods Iterative methods Preconditioning LU decomposition for tridiagonal matrices = A L U We suppose that L ij and U ij are known for i < p. Then A p,p 1 = L p,p 1U p 1,p 1, A p,p = L p,p 1U p 1,p + U p,p, A p,p+1 = U p,p+1. L p,p 1 = A p,p 1/U p 1,p 1, U p,p = A p,p L p,p 1U p 1,p = A p,p A p,p 1U p 1,p/U p 1,p 1, U p,p+1 = A p,p+1.

43 Factorization methods Thomas algorithm algorithm Direct methods Iterative methods Preconditioning Algorithmm // f a c t o r i z a t i o n U( 1, 1 ) = A( 1, 1 ) U( 1, 2 ) = A( 1, 2 ) f o r i = 2 to n i f U( i 1, i 1) = then s t o p L ( i, i 1) = A( i, i 1)/U( i 1, i 1) U( i, i ) = A( i, i ) L ( i, i 1) U( i 1, i ) U( i, i +1) = A( i, i +1) // c o n s t r u c t i o n o f t h e s o l u t i o n y = s o l v e ( L, b ) // l o w e r t r i a n g u l a r x = s o l v e (U, y ) // upper t r i a n g u l a r Complexity Order 5n products.

44 Factorization methods general matrices Direct methods Iterative methods Preconditioning For general matrices: Factorize the matrix LU algorithm Choleski algorithm Solve upper triangular system Solve lower triangular system.

45 Iterative methods Principle Direct methods Iterative methods Preconditioning A= E + D + F To solve A x = b, write A = M N and iterate M x k+1 N x k = b, i.e. x k+1 = M 1 N x k + M 1 b. Attention M should be easy to invert. M 1 N should lead to a stable algorithm. Jacobi M = D, N = (E + F ), Gauss Seidel M = D + E, N = F, Successive Over Relaxation M = D «1 ω + E, N = ω 1 D F.

46 Jacobi method Direct methods Iterative methods Preconditioning Algorithm choose x ( k=) f o r k = to c o n v e r g e n c e f o r i = 1 to n r h s = b ( i ) f o r j = 1 to n, j!= i r h s = r h s A( i, j ) x ( j, k ) x ( i, k+1) = r h s / A( i, i ) t e s t = norm ( x ( k+1) x ( k))< e p s i l o n ( w h i l e not t e s t ) x k+1 i = 1 A i nx j=1,j i A ij xj k A x k+1 = D 1 ( b (E + F ) x k ) = D 1 ( b + (D A) x k ) = D 1 b + (I D 1 A) x k. 1 Remarks simple, two copies of the variable x k+1 and x k, insensible to permutations, converges if the diagonal is strictly dominant.

47 Gauss Seidel method Direct methods Iterative methods Preconditioning Algorithm x k+1 i = 1 A ii Xi 1 b i A ij x k+1 j j=1 choose x ( k=) f o r k = to c o n v e r g e n c e f o r i = 1 to n r h s = b ( i ) f o r j = 1 to i 1 r h s = r h s A( i, j ) x ( j, k+1) f o r j = i +1 to n r h s = r h s A( i, j ) x ( j, k ) x ( i, k+1) = r h s / A( i, i ) t e s t = norm ( x ( k+1) x ( k))< e p s i l o n ( w h i l e not t e s t )! nx A ij xj k j=i+1 Remarks still simple, one copy of the variable x, sensible to permutations, often converges better than Jacobi.

48 SOR method Direct methods Iterative methods Preconditioning i = ω Xi 1 b i A ij x k+1 j A ii x k+1 x k+1 = j=1! nx A ij xj k j=i+1 + (1 ω)x k i «1 «1» «D ω + E b D 1 + ω + E ω 1 D F x k Remarks still simple, one copy of the variable x, Necessary condition for convergence: < ω < 2.

49 Descent method general principle Direct methods Iterative methods Preconditioning For A symmetric definite positive!! Principle Construct a series of approximations of the solution to the system x k+1 = x k + α k p k, where p k descent direction and α k to be determined. The solution x minimizes the functional J( x) = t xa x 2 t b x. J x i ( x) X x j A jk x k 2 X x i j,k j 1 b j x j A = X A ik x k + X k j = 2 A x b, i J ( x) =. x i x j A ji 2b i

50 Descent method determining α k Direct methods Iterative methods Preconditioning x also minimizes the functional E( x) = t ( x x)a ( x x), and E( x) =. For a given p k, which α minimizes E( x k+1 )? E( x k + α p k ) = t ( x k + α p k x)a( x k + α p k x), α E( x k + α p k ) = t p k A( x k + α p k x) + t ( x k + α p k x)a p k = 2 t ( x k + α p k x)a p k. t ( x k + α k p k x)a p k = t x k A p k + α k t p k A p k t xa p k = t p k A x k + α k t p k A p k t p k A x =. α k t p k A x k t p k A x = t p k A p k

51 Descent method functional profiles (good cases) Direct methods Iterative methods Preconditioning ««2 A =, 2 2 b = A 1 Cond(A) = A = Cond(A) = 7 «, b = A «

52 Descent method functional profiles (bad cases) Direct methods Iterative methods Preconditioning Nonpositive «case «2 8 A =, b = A Nonsymmetric case 2 3 A = 3 2 «, b = A «

53 Descent method optimal parameter (principle) Direct methods Iterative methods Preconditioning Principle Choose p k = r k b A x k. Choose α k is such that r k+1 is orthogonal to p k. r k+1 = b A x k+1 = b A( x k + α p k ) = r k α k A p k, = t p k r k+1 = t p k r k α k t p k A p k. α k = t p k r k t p k A p k. E( x k+1 ) = (1 γ k )E( x k ) with γ k ( t p k r k ) 2 = ( t p k A p k )( t r k A 1 r k ) 1 t p k r k Cond(A) p k r k.

54 Descent method optimal parameter (algorithm) Direct methods Iterative methods Preconditioning Algorithm choose x ( k=1) f o r k = 1 to c o n v e r g e n c e r ( k ) = b A x ( k ) p ( k ) = r ( k ) a l p h a ( k ) = r ( k ). p ( k ) / p ( k ). A p ( k ) x ( k+1) = x ( k ) + a l p h a ( k ) p ( k ) // r ( k ) s m a l l

55 Descent method conjugate gradient (principle) Direct methods Iterative methods Preconditioning Principle Choose p k = r k + β k p k 1. Choose β k to minimize the error, i.e. maximize the factor γ k Properties t r k p j = j < k, Span( r 1, r 2,..., r k ) = Span( r 1, A r 1,..., A k 1 r 1 ) Span( p 1, p 2,..., p k ) = Span( r 1, A r 1,..., A k 1 r 1 ) t p k A p j = j < k t r k A p j = j < k The algorithm converges in at most n iterations.

56 Descent method conjugate gradient (algorithm) Direct methods Iterative methods Preconditioning Algorithm choose x ( k=1) p ( 1 ) = r ( 1 ) = b A x ( 1 ) f o r k = 1 to c o n v e r g e n c e a l p h a ( k ) = r ( k ). p ( k ) / p ( k ). A p ( k ) x ( k+1) = x ( k ) + a l p h a ( k ) p ( k ) r ( k+1) = r ( k ) a l p h a ( k ) A p ( k ) b e t a ( k+1) = r ( k+1). r ( k+1) / r ( k ). r ( k ) p ( k+1) = r ( k+1) + b e t a ( k+1) p ( k ) // r ( k ) s m a l l

57 Descent method GMRES Direct methods Iterative methods Preconditioning For generic matrices A GMRES: General Minimal RESidual method Take a fair approximation x k of the solution Construct the m-dimensional set of free vectors This spans the Krylov space H k m. { r k, A r k,..., A m 1 r k } Construct an orthonormal basis for H k m e.g. via Gram-Schmidt { v 1,..., v m} Look for a new approximation x k+1 H k m: x k+1 = mx X j v j = [V ] X j=1 We obtain a system of n equations with m unknowns A x k+1 = A[V ] X = b.

58 Descent method GMRES (cont d) Direct methods Iterative methods Preconditioning Project on Hm k [ t V ]A[V ] X = [ t V ] b. Solve this system of m equations with m unknowns x k+1 = [V ] X. and so on... To work well GMRES should be preconditioned!

59 Preconditioning principle Direct methods Iterative methods Preconditioning Principle Replace system A x = b by C 1 A x = C 1 b where Cond(C 1 A) Cond(A). Which matrix C? C should be easily invertible, typically the product of two triangular matrices. C = diag(a), simplest but well... incomplete Cholesky or LU factorization...

60 Preconditioning symmetry issues Direct methods Iterative methods Preconditioning Symmetry Even if A and C are symmetric, C 1 A may not be symmetric. What if symmetry is needed? Let C 1/2 such that C 1/2 C 1/2 = C 1. Then C 1/2 AC 1/2 is similar to C 1 A. We consider the system C +1/2 (C 1 A)C 1/2 C +1/2 x = C +1/2 C 1 b (C 1/2 AC 1/2 )C +1/2 x = C 1/2 b Solve and then (C 1/2 AC 1/2 ) y = C 1/2 b y = C +1/2 x.

61 Preconditioning preconditioned conjugate gradient Direct methods Iterative methods Preconditioning Algorithm choose x ( k=1) r ( 1 ) = b A x ( 1 ) s o l v e Cz ( 1 ) = r ( 1 ) p ( 1 ) = r ( 1 ) f o r k = 1 to c o n v e r g e n c e a l p h a ( k ) = r ( k ). z ( k ) / p ( k ). A p ( k ) x ( k+1) = x ( k ) + a l p h a ( k ) p ( k ) r ( k+1) = r ( k ) a l p h a ( k ) A p ( k ) s o l v e C z ( k+1) = r ( k+1) b e t a ( k+1) = r ( k+1). z ( k+1) / r ( k ). z ( k ) p ( k+1) = z ( k+1) + b e t a ( k+1) p ( k ) At each iteration a system C z = r is solved.

62 main issues Band storage Sparse storage Problems involve often a large number of variables, of degrees of freedom, say 1 6. To store a full matrix for a 1 6 -order system, 1 12 real numbers (if real) are needed... In simple precision this necessitates 4 To of memory. But high order problems are often very sparse. We therefore use a storage structure which consists in only storing relevant, non-zero, data. Access to one element A ij should be very efficient.

63 CDS: Compressed Diagonal Band storage Sparse storage where L(A) = max L i (A) i L i (A) = max i j j/a ij

64 CRS: Compressed Row Band storage Sparse storage tabi tabj tabv All the non-zero values of the matrix are stored in a table tab; they are stored line by line in the increasing order of columns. A table tabj, with same size than tab stores the column number of the values in tabv. A table tabi with size n + 1 stores the indices in tabj of the first element of each line. The last entry is the size of tabv. CCS: Compressed Column = Harwell Boeing Generalization to symmetric matrices

65 CRS: Compressed Row exercise Band storage Sparse storage Question CRS storage? A = B A 3 8 Solution tabi = {1, 4, 6, 8, 1} tabj = {2, 3, 4, 1, 3, 2, 3, 3, 4} tabv = {4, 1, 6, 2, 5, 9, 7, 3, 8}

66 Cuthill McKee algorithm construction of a graph Cuthill McKee algorithm Goal Reduce the bandwidth of a large sparse matrix by renumbering the unknowns. Construction The nodes of the graph are the unknowns of the system. They are labelled with a number from 1 to n. The edges are the relations between the unknowns. Two unknowns i and j are linked if A ij. The distance d(i, j) between two nodes is the minimal number of edges to follow to join both nodes. The excentricity E(i) = max j d(i, j) Far neighbors are λ(i) = {j/d(i, j) = E(i)} Graph diameter D = max i E(i) Peripheral nodes P = {j/e(j) = D}.

67 Cuthill McKee algorithm bandwidth reduction Cuthill McKee algorithm This graph is used to renumber the unknonws. Choose a first node and label it with 1. Attribute the new numbers (2,3,... ) to the neighbors of node 1 with have the less non-labelled neighbors. Label the neighbors of node 2 and so on... until all nodes are labelled. once this is done the numbering is reversed: the first become the last.

68 Cuthill McKee algorithm example 1 Cuthill McKee algorithm

69 Cuthill McKee algorithm example 2 Cuthill McKee algorithm

70 Bibliography P. Lascaux, R. Théodor, Analyse numérique matricielle appliquée à l art de l ingénieur Volumes 1 and 2, 2ème édition, Masson (1997). Gene H. Golub, Charles F. van Loan, Matrix Computations, 3rd edition, Johns Hopkins University Press (1996).

71 TheEnd TheEnd