Scientific Computing WS 2018/2019. Lecture 10. Jürgen Fuhrmann Lecture 10 Slide 1

Transcription

1 Scientific Computing WS 2018/2019 Lecture 10 Jürgen Fuhrmann Lecture 10 Slide 1

2 Homework assessment Lecture 10 Slide 2

3 General Please apologize terse answers - on the bright side of this I found time to reply to all individually who handed things in by yesterday noon please stick to the filename scheme, this makes it easier for me to give feedback to all of you Good style with zip files is that they unpack into subdir with the same name. E.g. abc.zip unpacks into directory abc. Mac users: try to pack your stuff without the.ds Store subdirectories No need to include binaries MACOSX and Always try to calculate errors if exact data is available (I should have been more specific in assignment text) Lecture 10 Slide 3

4 Code style Try to specify datatypes in constants: 0.1f for float, 0.1l for long double and avoid mixing of datatypes in expressions. In particular write x/2.0 instead of x/2 if you do division of a double number. (There are reasonable automatic conversion rules, but things are clearer if they are explicit). Cast ints to double explicitely in floating point expressions. This ensures that you don t accidentally create an integer intermediate result. ( 1/i*i was the reason of many overflow errors in your codes) Math headers: use <cmath> instead of <math.h>. In particular, this gives you long double version of functions if needed, in particular for abs. When using printf, use the right format specifiers for output of floating point numbers: %e for float and double, and %Le for long double. %e,%le give the exponential notation, and %f, %Lf give a fixed point notation without exponential which is not very helpful for accuracy assessment. Lecture 10 Slide 4

5 Lecture 10 Slide 5 Representation of real numbers Any real number x R can be expressed via representation formula: x = ± d iβ i β e i=0 β N, β 2: base d i N, 0 d i < β: mantissa digits e Z : exponent Scientific notation of floating point numbers: e.g. x = β = 10 d = (6, 0, 2, 2, 0... ) e = 23 Non-unique: x = β = 10 d = (0, 6, 0, 2, 2, 0... ) e = 24 Infinite for periodic decimal numbers, irrational numbers Lecture 4 Slide 27

6 Lecture 10 Slide 6 Floating point numbers Computer representation uses β = 2, therefore d i {0, 1} Truncation to fixed finite size t 1 x = ± d iβ i β e i=0 t: mantissa length Normalization: assume d 0 = 1 save one bit for mantissa k: exponent size β k + 1 = L e U = β k 1 Extra bit for sign storage size: (t 1) + k + 1 IEEE 754 single precision (C++ float ): k = 8, t = bit IEEE 754 double precision (C++ double ): k = 11, t = bit Lecture 4 Slide 28

7 Lecture 10 Slide 7 Floating point limits Finite size of reprensentation there are minimal and maximal possible numbers which can be represented symmetry wrt. 0 because of sign bit smallest positive normalized number: d 0 = 1, d i = 0, i = 1... t 1 x min = β L float: e 38 double: e 308 smallest positive denormalized number: d i = 0, i = 0... t 2, d t 1 = 1 x min = β 1 t β L largest positive normalized number: d i = β 1, 0... t 1 x max = β(1 β 1 t )β U float: e+38 double: e+308 Lecture 4 Slide 29

8 Lecture 10 Slide 8 Machine precision There cannot be more than 2 t+k floating point numbers almost all real numbers have to be approximated Let x be an exact value and x be its approximation Then: x x < ɛ is the x best accuracy estimate we can get, where ɛ = β 1 t (truncation) ɛ = 1 2 β1 t (rounding) Also: ɛ is the smallest representable number such that 1 + ɛ > 1. Relative errors show up in partiular when subtracting two close numbers adding smaller numbers to larger ones Lecture 4 Slide 30

9 Machine epsilon Smallest floating point number ɛ such that 1 + ɛ > 1 in floating point arithmetic In exact math it is true that from 1 + ε = 1 it follows that 0 + ε = 0 and vice versa. In floating point computations this is not true Many of you used the right algorithm and used the first value or which 1 + ε = 1 as the result. This is half the desired quantity. Some did not divide start with 1.0 but by other numbers. E.g. 0.1 is not represented exactly in floating point arithmetic Recipe for calculation: Set ɛ = 1.0; while ɛ/2.0 > 1.0 do ɛ = ɛ/2.0 end But... this may be optimized away... Lecture 10 Slide 9

10 Normalized floating point number IEEE bit floating point number normally the same as C++ float ± e 0 e 1 e 1 e 2 e 4 e 5 e 6 e 7 d 1 d 2 d 3 d 4 d 5 d 6 d 7 d 8 d 9 d 10 d 11 d 12 d 13 d 14 d 15 d 16 d 17 d 18 d 19 d 20 d 21 d 22 d 23 Storage layout for a normalized number (d 0 = 1) bit 0: sign, 0 +, 1 bit : r = 8 exponent bits, value e + 2 r 1 1 = 127 is stored no need for sign bit in exponent bit : t = 23 mantissa bits d1... d 23 d0 = 1 not stored hidden bit Examples 1 0_ _ e = 0, stored _ _ e = 1, stored _ _ e = 1, stored _ _ infinite periodic 0 0_ _ Numbers which are exactly represented in decimal system may not be exactly represented in binary system. Lecture 10 Slide 10

11 How Additionworks? General: 1. Adjust exponent of number to be added: Until both exponents are equal, add one to exponent, shift mantissa to right by one bit 2. Add both numbers 3. Normalize result For 1+ɛ, We have at maximum t bit shifts of normalized mantissa until mantissa becomes 0, so ɛ = 2 t. Lecture 10 Slide 11

12 Data of IEEE 754 floating point representations size t r ɛ float e-07 double e-16 long double e-19 Floating point format not standardized by language but by IEEE comitee Implementation of long double varies, may even be the same as double, or may be significantly slower, so it is mostly no good option There are high accuracy floating point number packages available, which however perform calculations without support of the CPU floating point arithmetic Lecture 10 Slide 12

13 Summation Basel sum: K n=1 1 n 2 = π2 6 Intended answer for accuracy: sum in reverse order. Start with adding up many small values which would be cancelled out if added to an already large sum value. Results for float: n forward sum forward sum error reverse sum reverse sum error e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e-07 No gain in accuracy for forward sum for n > Lecture 10 Slide 13

14 Kahan summation Some of you hinted at the Kahan compensated summation algorithm (thanks!): T sum_kah=0.0; T error_compensation=0.0; for (int i=1; i<=n;i++) { T x=i; T increment=1.0/(x*x); T corrected_increment=increment-error_compensation; T good_sum=sum_kah+corrected_increment; error_compensation= (good_sum-sum_kah)-corrected_increment; sum_kah =good_sum; } When implementing, be careful that expressions are not optimized away... William Kahan (1933-) is the principle architect of the IEEE 754 floating point standard... Lecture 10 Slide 14

15 Recap on nonnegative matrices Lecture 10 Slide 15

16 Lecture 10 Slide 16 The Gershgorin Circle Theorem (Semyon Gershgorin,1931) (everywhere, we assume n 2) Theorem (Varga, Th. 1.11) Let A be an n n (real or complex) matrix. Let Λ i = a ij j=1...n j i If λ is an eigenvalue of A then there exists r, 1 r n such that λ a rr Λ r Proof Assume λ is eigenvalue, x a corresponding eigenvector, normalized such that max i=1...n x i = x r = 1. From Ax = λx it follows that (λ a ii )x i = a ij x j λ a rr = j=1...n j i j=1...n j r a rj x j j=1...n j r a rj x j j=1...n j r a rj = Λ r Lecture 9 Slide 11

17 Lecture 10 Slide 17 Gershgorin Circle Corollaries Corollary: Any eigenvalue of A lies in the union of the disks defined by the Gershgorin circles λ {µ V : µ a ii Λ i } Corollary: i=1...n ρ(a) max i=1...n ρ(a) max j=1...n n a ij = A j=1 n a ij = A 1 i=1 Proof n µ a ii Λ i µ Λ i + a ii = a ij j=1 Furthermore, σ(a) = σ(a T ). Lecture 9 Slide 12

18 Lecture 10 Slide 18 Gershgorin circles: example A = , λ 1 = 1, λ 2 = 2, λ 3 = 3, Λ 1 = 5.2, Λ 2 = 0.8, λ 3 = Im Re Lecture 9 Slide 13

19 Lecture 10 Slide 19 Gershgorin circles: heat example I A = B = (I D 1 A) = 2 h 1 h 1 h 2 h 1 h h 2 h 1 h h h 1 h h 2 h 1 h h 2 h We have b ii = 0, Λ i = { 1 2, i = 1, n 1 i = 2... n 1 estimate λ i 1 Lecture 9 Slide 14

20 Lecture 10 Slide 20 Gershgorin circles: heat example II Let n=11, h=0.1: ( ) ihπ λ i = cos 1 + 2h (i = 1... n) Im Re the Gershgorin circle theorem is too pessimistic... Lecture 9 Slide 15

21 Lecture 10 Slide 21 Weighted directed graph representation of matrices Define a directed graph from the nonzero entries of a matrix A = (a ik ): 4 Nodes: N = {N i } i=1...n Directed edges: E = { N k N l a kl 0} Matrix entries weights of directed edges A = N N 1 9 N N N :1 equivalence between matrices and weighted directed graphs Convenient e.g. for sparse matrices Lecture 9 Slide 16

22 Lecture 10 Slide 22 Reducible and irreducible matrices Definition A is reducible if there exists a permutation matrix P such that PAP T = ( ) A11 A 12 0 A 22 A is irreducible if it is not reducible. Theorem (Varga, Th. 1.17): A is irreducible the matrix graph is connected, i.e. for each ordered pair (N i, N j ) there is a path consisting of directed edges, connecting them. Equivalently, for each i, j there is a sequence of consecutive nonzero matrix entries a ik1, a k1k 2, a k2k 3..., a kr 1k r a kr j. Lecture 9 Slide 17

23 Lecture 10 Slide 23 Taussky theorem (Olga Taussky, 1948) Theorem (Varga, Th. 1.18) Let A be irreducible. Assume that the eigenvalue λ is a boundary point of the union of all the disks λ {µ C : µ a ii Λ i } i=1...n Then, all n Gershgorin circles pass through λ, i.e. for i = 1... n, λ a ii = Λ i Lecture 9 Slide 18

24 Lecture 10 Slide 24 Consequences for heat example from Taussky theorem B = I D 1 A We had b ii = 0, Λ i = { 1 2, i = 1, n 1 i = 2... n 1 estimate λ i 1 Assume λ i = 1. Then λ i lies on the boundary of the union of the Gershgorin circles. But then it must lie on the boundary of both circles with radius 1 2 and 1 around 0. Contradiction λ i < 1, ρ(b) < 1! Lecture 9 Slide 20

25 Lecture 10 Slide 25 Diagonally dominant matrices Definition Let A = (a ij ) be an n n matrix. A is diagonally dominant if (i) for i = 1... n, a ii j=1...n j i a ij A is strictly diagonally dominant (sdd) if (i) for i = 1... n, a ii > a ij j=1...n j i A is irreducibly diagonally dominant (idd) if (i) A is irreducible (ii) A is diagonally dominant for i = 1... n, a ii j=1...n j i a ij (iii) for at least one r, 1 r n, a rr > j=1...n j r a rj Lecture 9 Slide 21

26 Lecture 10 Slide 26 A very practical nonsingularity criterion Theorem (Varga, Th. 1.21): Let A be strictly diagonally dominant or irreducibly diagonally dominant. Then A is nonsingular. If in addition, a ii > 0 is real for i = 1... n, then all real parts of the eigenvalues of A are positive: Reλ i > 0, i = 1... n Lecture 9 Slide 22

27 Lecture 10 Slide 27 Corollary Theorem: If A is complex hermitian or real symmetric, sdd or idd, with positive diagonal entries, it is positive definite. Proof: All eigenvalues of A are real, and due to the nonsingularity criterion, they must be positive, so A is positive definite. Lecture 9 Slide 25

28 Lecture 10 Slide 28 Perron-Frobenius Theorem (1912/1907) Definition: A real n-vector x is positive (x > 0) if all entries of x are positive nonnegative (x 0) if all entries of x are nonnegative Definition: A real n n matrix A is positive (A > 0) if all entries of A are positive nonnegative (A 0) if all entries of A are nonnegative Theorem(Varga, Th. 2.7) Let A 0 be an irreducible n n matrix. Then (i) A has a positive real eigenvalue equal to its spectral radius ρ(a). (ii) To ρ(a) there corresponds a positive eigenvector x > 0. (iii) ρ(a) increases when any entry of A increases. (iv) ρ(a) is a simple eigenvalue of A. Proof: See Varga. Lecture 9 Slide 27

29 Lecture 10 Slide 29 Perron-Frobenius for general nonnegative matrices Each n n matrix can be brought to the normal form R 11 R R 1m PAP T 0 R R 2m = R mm where for j = 1... m, either R jj irreducible or R jj = (0). Theorem(Varga, Th. 2.20) Let A 0 be an n n matrix. Then (i) A has a nonnegative eigenvalue equal to its spectral radius ρ(a). This eigenvalue is positive unless A is reducible and its normal form is strictly upper triangular (ii) To ρ(a) there corresponds a nonzero eigenvector x 0. (iii) ρ(a) does not decrease when any entry of A increases. m Proof: See Varga; σ(a) = σ(r jj ), apply irreducible Perron-Frobenius to R jj. j=1 Lecture 9 Slide 28

30 Lecture 10 Slide 30 Jacobi method convergence Corollary: Let A be sdd or idd, and D its diagonal. Assume that a ii > 0 and a ij 0 for i j. Then ρ(i D 1 A) < 1, i.e. the Jacobi method converges. Proof In this case, B = B. Lecture 9 Slide 31

31 Lecture 10 Slide 31 Regular splittings A = M N is a regular splitting if M is nonsingular M 1, N are nonnegative, i.e. have nonnegative entries Regard the iteration u k+1 = M 1 Nu k + M 1 b. We have I M 1 A = M 1 N. Lecture 9 Slide 32

32 Lecture 10 Slide 32 Convergence theorem for regular splitting Theorem: Assume A is nonsingular, A 1 0, and A = M N is a regular splitting. Then ρ(m 1 N) < 1. Proof: Let G = M 1 N. Then A = M(I G), therefore I G is nonsingular. In addition A 1 N = (M(I M 1 N)) 1 N = (I M 1 N) 1 M 1 N = (I G) 1 G By Perron-Frobenius (for general matrices), ρ(g) is an eigenvalue with a nonnegative eigenvector x. Thus, 0 A 1 Nx = ρ(g) 1 ρ(g) x Therefore 0 ρ(g) 1. As I G is nonsingular, ρ(g) < 1. Lecture 9 Slide 33

33 Lecture 10 Slide 33 Convergence rate comparison Corollary: ρ(m 1 N) = Proof: Rearrange τ = τ 1+τ where τ = ρ(a 1 N). ρ(g) 1 ρ(g) Corollary: Let A 0, A = M 1 N 1 and A = M 2 N 2 be regular splittings. If N 2 N 1 0, then 1 > ρ(m 1 2 N 2 ) ρ(m 1 1 N 1 ). Proof: τ 2 = ρ(a 1 N 2 ) ρ(a 1 N 1 ) = τ 1 But τ 1+τ is strictly increasing. Lecture 9 Slide 34

34 Lecture 10 Slide 34 M-Matrix definition Definition Let A be an n n real matrix. A is called M-Matrix if (i) a ij 0 for i j (ii) A is nonsingular (iii) A 1 0 Corollary: If A is an M-Matrix, then A 1 > 0 A is irreducible. Proof: See Varga. Lecture 9 Slide 35

35 Lecture 10 Slide 35 Main practical M-Matrix criterion Corollary: Let A be sdd or idd. Assume that a ii > 0 and a ij 0 for i j. Then A is an M-Matrix. Proof: We know that A is nonsingular, but we have to show A 1 0. Let B = I D 1 A. Then ρ(b) < 1, therefore I B is nonsingular. We have for k > 0: I B k+1 = (I B)(I + B + B B k ) (I B) 1 (I B k+1 ) = (I + B + B B k ) The left hand side for k converges to (I B) 1, therefore (I B) 1 = k=0 B k As B 0, we have (I B) 1 = A 1 D 0. As D > 0 we must have A 1 0. Lecture 9 Slide 36

36 Lecture 10 Slide 36 Application Let A be an M-Matrix. Assume A = D E F. Jacobi method: M = D is nonsingular, M 1 0. N = E + F nonnegative convergence Gauss-Seidel: M = D E is an M-Matrix as A M and M has non-positive off-digonal entries. N = F 0. convergence Comparison: N J N GS Gauss-Seidel converges faster. More general: Block Jacobi, Block Gauss Seidel etc. Lecture 9 Slide 37

37 Lecture 10 Slide 37 Intermediate Summary Given some matrix, we now have some nice recipies to establish nonsingularity and iterative method convergence: Check if the matrix is irreducible. This is mostly the case for elliptic and parabolic PDEs. Check if the matrix is strictly or irreducibly diagonally dominant. If yes, it is in addition nonsingular. Check if main diagonal entries are positive and off-diagonal entries are nonpositive. If yes, in addition, the matrix is an M-Matrix, its inverse is nonnegative, and elementary iterative methods converge. These critera do not depend on the symmetry of the matrix! Lecture 9 Slide 38

38 Incomplete LU factorizations (ILU) Idea (Varga, Buleev, 1960): fix a predefined zero pattern apply the standard LU factorization method, but calculate only those elements, which do not correspond to the given zero pattern Result: incomplete LU factors L, U, remainder R: A = LU R Problem: with complete LU factorization procedure, for any nonsingular matrix, the method is stable, i.e. zero pivots never occur. Is this true for the incomplete LU Factorization as well? Lecture 10 Slide 38

39 Comparison of M-Matrices Theorem(Saad, Th. 1.33): Let A, B n n matrices such that (i) A B (ii) b ij 0 for i j. Then, if A is an M-Matrix, so is B. Proof: For the diagonal parts, one has D B D A > 0, D A A D B B 0 Therefore I D 1 A A D 1(D B B) D 1 (D B B) = I D 1 B =: G 0. A B B Perron-Frobenius ρ(g) = ρ(i D 1 B B) ρ(i D 1 A A) < 1 I G is nonsingular. From the proof of the M-matrix criterion, B = (I G) 1 = k=0 G k 0. As D B > 0, we get B 0. D 1 B Lecture 10 Slide 39

40 M-Property propagation in Gaussian Elimination Theorem:(Ky Fan; Saad Th 1.10) Let A be an M-matrix. Then the matrix A 1 obtained from the first step of Gaussian elimination is an M-matrix. Proof: One has aij 1 = a ij ai1a1j a 11, a ij, a i1, a 1j 0, a 11 > 0 aij 1 0 for i j a 12 a A = L 1 A 1 with L 1 = a 1n a A 1 nonsingular nonsingular, nonnegative Let e 1... e n be the unit vectors. Then A 1 1 e 1 = 1 a e For j > 1, A 1 1 e j = A 1 L 1 e j = A 1 e j 0. A Lecture 10 Slide 40

41 Stability of ILU Theorem (Saad, Th. 10.2): If A is an M-Matrix, then the algorithm to compute the incomplete LU factorization with a given nonzero pattern A = LU R is stable. Moreover, A = LU R is a regular splitting. Lecture 10 Slide 41

42 Stability of ILU decomposition II Proof Let à 1 = A 1 + R 1 = L 1 A + R 1 where R 1 is a nonnegative matrix which occurs from dropping some off diagonal entries from A 1. Thus, à 1 A 1 and Ã1 is an M-matrix. We can repeat this recursively à k = A k + R k = L k A k 1 + R k = L k L k 1 A k 2 + L k R k 1 + R k = L k L k 1... L 1 A + L k L k 1... L 2 R R k Let L = (L n 1... L 1 ) 1, U = à n 1. Then U = L 1 A + S with S = L n 1 L n 2... L 2 R 1 + +R n 1 = L n 1 L n 2... L 2 (R 1 +R R n 1 ) Let R = R 1 + R R n 1, then A = LU R where U 1 L 1, R are nonnegative. Lecture 10 Slide 42

43 ILU(0) Special case of ILU: ignore any fill-in. Representation: M = ( D E) D 1 ( D F ) D is a diagonal matrix (wich can be stored in one vector) which is calculated by the incomplete factorization algorithm. Setup: for(int i=0;i<n;i++) d(i)=a(i,i) for(int i=0;i<n;i++) { d(i)=1.0/d(i) for (int j=i+1;j<n;j++) d(j)=d(j)-a(i,j)*d(i)*a(j,i) } Lecture 10 Slide 43

44 ILU(0) Solve Mu = v for(int i=0;i<n;i++) { double x=0.0; for (int j=0;j<i;i++) x=x+a(i,j)*u(j) u(i)=d(i)*(v(i)-x) } for(int i=n-1;i>=0;i--) { double x=0.0 for(int j=i+1;j<n;j++) x=x+a(i,j)*u(j) u(i)=u(i)-d(i)*x } Lecture 10 Slide 44

45 ILU(0) Generally better convergence properties than Jacobi, Gauss-Seidel One can develop block variants Alternatives: ILUM: ( modified ): add ignored off-diagonal entries to D ILUT: zero pattern calculated dynamically based on drop tolerance Dependence on ordering Can be parallelized using graph coloring Not much theory: experiment for particular systems I recommend it as the default initial guess for a sensible preconditioner Incomplete Cholesky: symmetric variant of ILU Lecture 10 Slide 45