Vector and Matrix Norms I
|
|
- Ada Phoebe French
- 5 years ago
- Views:
Transcription
1 Vector and Matrix Norms I Scalar, vector, matrix How to calculate errors? Scalar: absolute error: ˆα α relative error: Vectors: vector norm Norm is the distance!! ˆα α / α Chih-Jen Lin (National Taiwan Univ.) 1 / 91
2 Vector and Matrix Norms II l-norm: x l = l x 1 l + + x n l 1-norm: -norm: x 1 = n x i i=1 l l x = lim x l = lim l = max x i x 1 l + + x n l l (max x i ) l x l 1 l + + x n l l n(max x i ) l Chih-Jen Lin (National Taiwan Univ.) 2 / 91
3 Vector and Matrix Norms III Example: x = ˆx x = 2, ˆx x x and ˆx = = 0.02, ˆx x ˆx = ˆx x 2 = 2.238, ˆx x 2 x 2 = , ˆx x 2 ˆx 2 = All norms are equivalent Chih-Jen Lin (National Taiwan Univ.) 3 / 91
4 Vector and Matrix Norms IV For l 1 and l 2 norms, there exist c 1 and c 2 such that for all x R n Example: c 1 x l1 x l2 c 2 x l1 x 2 x 1 n x 2 (1) x x 2 n x (2) x x 1 n x (3) Chih-Jen Lin (National Taiwan Univ.) 4 / 91
5 Vector and Matrix Norms V Therefore, you can just choose a norm for your convenience HW5-1: prove (1)-(3) Matrix norm: How to define the distance between two matrices? Usually a norm should satisfy where α is a scalar A 0 A + B A + B (4) αa = α A, Chih-Jen Lin (National Taiwan Univ.) 5 / 91
6 Vector and Matrix Norms VI Definition: A l max x 0 Ax l x l = max x =1 Ax l Proof of (4) A + B = max (A + B)x x =1 max ( Ax + Bx ) max Ax + max Bx x =1 x =1 x =1 Chih-Jen Lin (National Taiwan Univ.) 6 / 91
7 Relative error I Usually calculating ˆα α α is not practical The reason is that α is unknown ˆα α ˆα is a more reasonable estimate Chih-Jen Lin (National Taiwan Univ.) 7 / 91
8 Relative error II If then 1 ˆα α 1.1 ˆα ˆα α ˆα ˆα α α 0.1, 1 ˆα α 0.9 ˆα Proof: α ˆα ˆα α 0.1 ˆα α 1.1 ˆα Chih-Jen Lin (National Taiwan Univ.) 8 / 91
9 Relative error III Similarly, ˆα α 0.1 ˆα α 0.9 ˆα Chih-Jen Lin (National Taiwan Univ.) 9 / 91
10 Condition of a Linear System I Solving a linear system x x x 3 = 23 33, solution = x 4 Right-hand side slightly modified x x x 3 = , solution = x 4 Chih-Jen Lin (National Taiwan Univ.) 10 / 91
11 Condition of a Linear System II A small modification causes a huge error Matrix slightly modified x x x 3 = x 4 81 solution = Chih-Jen Lin (National Taiwan Univ.) 11 / 91
12 Condition of a Linear System III Right-hand side slightly modified Ax = b A(x + δx) = b + δb δx = A 1 δb δx A 1 δb b A x δx x A A 1 δb b Chih-Jen Lin (National Taiwan Univ.) 12 / 91
13 Condition of a Linear System IV Matrix modified Ax = b (A + δa)(x + δx) = b Ax + Aδx + δa(x + δx) = b δx = A 1 δa(x + δx) δx A 1 δa x + δx δx x + δx A A 1 δa A A A 1 is defined as the condition of A A smaller condition number is better Chih-Jen Lin (National Taiwan Univ.) 13 / 91
14 Sparse matrices: Storage Schemes I Most elements are zero Very common in engineering applications Without saving zeros, can handle very large matrices An example A = Storage schemes: There are different ways to store sparse matrices Chih-Jen Lin (National Taiwan Univ.) 14 / 91
15 Sparse matrices: Storage Schemes II Coordinate format a arow_ind acol_ind Indices may not be well ordered Is it easy to do operations? A + B, Ax A + B: if (i, j) are not ordered, difficult y = Ax: for l = 1:nnz i = arow_ind(l) j = acol_ind(l) y(i) = y(i) + a(l)*x(j) end Chih-Jen Lin (National Taiwan Univ.) 15 / 91
16 Sparse matrices: Storage Schemes III nnz: usually used to represent the number of nonzeros x: vector in dense format In general we directly store a vector without using sparse format Access one column for l = 1:nnz if acol_ind(l) == i x(arow_ind(l)) = a(l) end end Cost: O(nnz) Chih-Jen Lin (National Taiwan Univ.) 16 / 91
17 Sparse matrices: Storage Schemes IV When is this used: Solving Lx = b l 11 x 1 l 21 l 22 x = b 2. l n1 l n2 l nn x n b n b 2. b n x 1 Compressed column format l 21. l n1 b 1 Chih-Jen Lin (National Taiwan Univ.) 17 / 91
18 Sparse matrices: Storage Schemes V a arow_ind acol_ptr jth column: from a(acol ptr(j)) to a(acol ptr(j+1)-1) Example: 3rd column acol_ptr(3) = 5 acol_prr(4) = 7 a(5) = 7 a(6) = 10 nnz = acol ptr(n+1) - 1 acol ptr contains n + 1 elements Chih-Jen Lin (National Taiwan Univ.) 18 / 91
19 Sparse matrices: Storage Schemes VI C = A + B for j = 1:n get A s jth column get B s jth column do a vector addition end C is still with column format y = Ax = A :,1 x A :,n x n for j = 1:n for l = acol_ptr(j):acol_ptr(j+1)-1 y(arow_ind(l)) = y(arow_ind(l)) + a(l)*x(j) end end Chih-Jen Lin (National Taiwan Univ.) 19 / 91
20 Sparse matrices: Storage Schemes VII Row indices of the same column may not be sorted a arow_ind acol_ptr C = AB is similar Access one column is easy Access one row is very difficult Compressed row format A = Chih-Jen Lin (National Taiwan Univ.) 20 / 91
21 Sparse matrices: Storage Schemes VIII a acol_ind arow_ptr C a arow_ind acol_ptr There are so many variations of sparse structures. Very difficult to have standard sparse libraries Different formats are suitable for different matrices A C implementation: row format Chih-Jen Lin (National Taiwan Univ.) 21 / 91
22 Sparse matrices: Storage Schemes IX typedef struct row_elt { int col, nxt_row, nxt_idx; Real val; } row_elt; typedef struct SPROW { int len, maxlen, diag; row_elt *elt; } SPROW; typedef struct SPMAT { int m, n, max_m, max_n; char flag_col, flag_diag; SPROW *row; int *start_row; int *start_idx; Chih-Jen Lin (National Taiwan Univ.) 22 / 91
23 Sparse matrices: Storage Schemes X } SPMAT; To scan a row len = A->row[i].len ; for (j_idx = 0; j_idx < len; j_idx++) printf( %d %d %g, i, A->row[i].elt[j_idx].col, A->row[i].elt[j_idx].val) ; Objected oriented design is a big challenge for sparse matrices Homework 5-2: Chih-Jen Lin (National Taiwan Univ.) 23 / 91
24 Sparse matrices: Storage Schemes XI Write your own sparse matrix-matrix code and compare with Intel MKL (which has now supported sparse opertions) You generate random matrices by yourself. Any reasonable size is ok. Chih-Jen Lin (National Taiwan Univ.) 24 / 91
25 Sparse Matrix and Factorization I A more advanced topic Factorization generates fill-ins fill-ins: new nonzero positions A matlab program A = sprandsym(200, 0.05, 0.01, 1) ; L = chol(a) ; spy(a) ; print -deps A spy(l) ; print -deps L 0.05: density 0.01: 1/(condition number) Chih-Jen Lin (National Taiwan Univ.) 25 / 91
26 Sparse Matrix and Factorization II 1: type of matrix, 1 gives a matrix with 1/(condition number) exactly 0.01 spy: draw the sparsity pattern nz = 1984 (a) A nz = 2919 (b) L Chih-Jen Lin (National Taiwan Univ.) 26 / 91
27 Sparse Matrix and Factorization III Clearly L is denser Chih-Jen Lin (National Taiwan Univ.) 27 / 91
28 Permutation and Reordering I A = , chol(a) = Chih-Jen Lin (National Taiwan Univ.) 28 / 91
29 Permutation and Reordering II P = , AP = Chih-Jen Lin (National Taiwan Univ.) 29 / 91
30 Permutation and Reordering III = = PAP Chih-Jen Lin (National Taiwan Univ.) 30 / 91
31 Permutation and Reordering IV chol(pap T ) = chol(pap T ) is sparser Ax = b (PAP T )Px = Pb, solve Px and get x There are different ways of permutations Chih-Jen Lin (National Taiwan Univ.) 31 / 91
32 Permutation and Reordering V Reordering algorithms. colmmd - Column minimum degree permutat symmmd - Symmetric minimum degree permu symrcm - Symmetric reverse Cuthill-McKe permutation. colperm - Column permutation. randperm - Random permutation. dmperm - Dulmage-Mendelsohn permutation Finding the ordering with the least entries in the factorization minimum fill-in problem Minimum fill-in may not be the best: need to consider the numerical stability, implementation efforts, etc Chih-Jen Lin (National Taiwan Univ.) 32 / 91
33 Iterative Methods I Chapter 10 of the book matrix computation An iterative process: x 1, x 2,..., x We hope Ax = b Gaussian elimination is O(n 3 ) If x k x k+1 takes O(n r ), l iterations, and n r l < n 3, iterative methods can be faster Accuracy and sparsity are other considerations Chih-Jen Lin (National Taiwan Univ.) 33 / 91
34 Jacobi and Gauss-Seidel Method I A three by three system Ax = b x 1 = (b 1 a 12 x 2 a 13 x 3 )/a 11 x 2 = (b 2 a 21 x 1 a 23 x 3 )/a 22 x 3 = (b 3 a 31 x 1 a 32 x 2 )/a 33 x k : an approximation to x = A 1 b (x k+1 ) 1 = (b 1 a 12 (x k ) 2 a 13 (x k ) 3 )/a 11 (x k+1 ) 2 = (b 2 a 21 (x k ) 1 a 23 (x k ) 3 )/a 22 (x k+1 ) 3 = (b 3 a 31 (x k ) 1 a 32 (x k ) 2 )/a 33 Chih-Jen Lin (National Taiwan Univ.) 34 / 91
35 Jacobi and Gauss-Seidel Method II The general case for i = 1:n (x k+1 ) i = (b i i 1 j=1 a ij(x k ) j n j=i+1 a ij(x k ) j )/a ii end Gauss-Seidel iteration (x k+1 ) 1 = (b 1 a 12 (x k ) 2 a 13 (x k ) 3 )/a 11 (x k+1 ) 2 = (b 2 a 21 (x k+1 ) 1 a 23 (x k ) 3 )/a 22 (x k+1 ) 3 = (b 3 a 31 (x k+1 ) 1 a 32 (x k+1 ) 2 )/a 33 The general case Chih-Jen Lin (National Taiwan Univ.) 35 / 91
36 Jacobi and Gauss-Seidel Method III for i = 1:n (x k+1 ) i = (b i i 1 j=1 a ij(x k+1 ) j n j=i+1 a ij(x k ) j )/a ii end The iterates may diverge A = , b = 5 5, sol = Chih-Jen Lin (National Taiwan Univ.) 36 / 91
37 Jacobi and Gauss-Seidel Method IV det = x 0 = 0 0, x 1 = , x 2 = = x 3 = = Chih-Jen Lin (National Taiwan Univ.) 37 / 91
38 Jacobi and Gauss-Seidel Method V Convergence Does the method eventually go to a solution? This is like a 11 a 22 a 11 x 1 = b 1 a 12 x 2 a 13 x 3 a 22 x 2 = b 2 a 21 x 1 a 23 x 3 a 33 x 3 = b 3 a 31 x 1 a 32 x 2 x 1 x 2 = 0 a 12 a 13 a 21 0 a 23 x 1 x 2 +b a 33 a 31 a 32 0 x 3 Chih-Jen Lin (National Taiwan Univ.) 38 / 91 x 3
39 Jacobi and Gauss-Seidel Method VI If M = a 11 then a 22 and N = 0 a 12 a 13 a 21 0 a 23 a 33 a 31 a 32 0 A = M N and Mx k+1 = Nx k + b Chih-Jen Lin (National Taiwan Univ.) 39 / 91
40 Jacobi and Gauss-Seidel Method VII Spectral radius: ρ(a) = max{ λ λ λ(a)} λ(a) contains all eigenvalues of A Theorem 1 A = M N, A, M non-singular, ρ(m 1 N) < 1 Mx k+1 = Nx k + b leads to the convergence of {x k } to A 1 b for any starting vector x 0 Chih-Jen Lin (National Taiwan Univ.) 40 / 91
41 Jacobi and Gauss-Seidel Method VIII Proof: Ax = b Mx = Nx + b M(x k+1 x) = N(x k x) x k+1 x = M 1 N(x k x) x k+1 x = (M 1 N) k (x 1 x) ρ(m 1 N) < 1 (M 1 N) k 0 x k+1 x 0 Chih-Jen Lin (National Taiwan Univ.) 41 / 91
42 Jacobi and Gauss-Seidel Method IX Reasons why ρ(m 1 N) < 1 (M 1 N) k 0? This is quite complicated, so we omit the derivation here. Chih-Jen Lin (National Taiwan Univ.) 42 / 91
43 Reasons of Gauss-Seidel Methods I An optimization problem is the same as solving 1 min x 2 x T Ax b T x Ax b = 0 if A is symmetric positive definite Chih-Jen Lin (National Taiwan Univ.) 43 / 91
44 Reasons of Gauss-Seidel Methods II If then f (x) = 1 2 x T Ax b T x f (x) = Ax b f (x) f (x) x 1. f (x) x n Chih-Jen Lin (National Taiwan Univ.) 44 / 91
45 Reasons of Gauss-Seidel Methods III Remember x T Ax = = n x i (Ax) i i=1 n i=1 n x i A ij x j j=1 = x 1 A 11 x x 1 A 1n x n +x 2 A 21 x x n A n1 x 1 + Chih-Jen Lin (National Taiwan Univ.) 45 / 91
46 Reasons of Gauss-Seidel Methods IV Therefore x T Ax x 1 = 2A 11 x A 1n x n +x 2 A x n A n1 = 2(A 11 x A 1n x n ) Chih-Jen Lin (National Taiwan Univ.) 46 / 91
47 Reasons of Gauss-Seidel Methods V Sequentially update one variable min f (x 1, x2 k,..., xn k ) x 1 min f (x x 1 k+1, x 2,..., xn k ) 2 min f (x x 1 k+1, x2 k+1, x 3,..., xn k ) 3. Chih-Jen Lin (National Taiwan Univ.) 47 / 91
48 Reasons of Gauss-Seidel Methods VI 1 min d 2 (x + de i) T A(x + de i ) b T (x + de i ) 1 min d 2 d 2 A ii + d(ax) i b i d A ii d + (Ax) i b i = 0 d = b i (Ax) i A ii x i + d b i j:j i A ijx j A ii Chih-Jen Lin (National Taiwan Univ.) 48 / 91
49 Reasons of Gauss-Seidel Methods VII Note that e i = [0,..., 0, 1, 0,..., 0] }{{} T i 1 Chih-Jen Lin (National Taiwan Univ.) 49 / 91
50 Conjugate Gradient Method I For symmetric positive definite matrices only One of the most frequently used iterative methods Before introducing CG, we discuss a related method called steepest descent method We still consider solving 1 min x 2 x T Ax b T x Chih-Jen Lin (National Taiwan Univ.) 50 / 91
51 Steepest Descent Method I Gradient direction f (x) =f (x k ) + f (x k )(x x k ) f (x k )(x x k ) 2 + f (x) =f (x k ) + f (x k ) T (x x k )+ 1 2 (x x k) T 2 f (x k )(x x k ) + Omit 1 2 (x x k) T 2 f (x k )(x x k ) + f (x) f (x k ) + f (x k ) T (x x k ) Chih-Jen Lin (National Taiwan Univ.) 51 / 91
52 Steepest Descent Method II minimize If then is the direction. f (x k ) T (x x k ) min f (x k) T (x x k ), x x k =1 x x k = f (x k) f (x k ) Chih-Jen Lin (National Taiwan Univ.) 52 / 91
53 Steepest Descent Method III Note a b = a b cos θ cos π = 1, minimum OK for 2D. But how do you prove that for general cases? Now f (x k ) = Ax k b. x x k α f (x k ) Chih-Jen Lin (National Taiwan Univ.) 53 / 91
54 Steepest Descent Method IV Let r = f (x k ) = b Ax k, x = x k Minimize along the direction 1 min α 2 (x + αr)t A(x + αr) (x + αr) T b 1 min α 2 α2 r T Ar + αr T Ax αr T b 1 min α 2 α2 r T Ar + αr T (Ax b) Chih-Jen Lin (National Taiwan Univ.) 54 / 91
55 Steepest Descent Method V A problem of one variable: Note αr T Ar + r T (Ax b) = 0 α = r T (b Ax) r T Ar r T Ar 0 if A is positive definite Now r = b Ax k α = (b Ax k) T (b Ax k ) (b Ax k ) T A(b Ax k ) Chih-Jen Lin (National Taiwan Univ.) 55 / 91
56 Steepest Descent Method VI The algorithm: k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 α k = r T k 1 r k 1/r T k 1 Ar k 1 x k = x k 1 + α k r k 1 r k = b Ax k end It converges but may be very slow Chih-Jen Lin (National Taiwan Univ.) 56 / 91
57 General Search Directions I Suppose we have x k obtained by f (x k 1 + αp k ) min α f (x k 1 + αp k ) = 1 2 (x k 1 + αp k ) T A(x k 1 + αp k ) b T (x k 1 + αp k ) = + αp T k (Ax k 1 b) α2 p T k Ap k α = pt k (r k 1) p T k Ap k Chih-Jen Lin (National Taiwan Univ.) 57 / 91
58 General Search Directions II A more general algorithm: k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 Choose a direction p k such that p T k r k 1 0 α k = p T k r k 1/p T k Ap k x k = x k 1 + α k p k r k = b Ax k end By this setting x k x 0 + span{p 1,..., p k } Chih-Jen Lin (National Taiwan Univ.) 58 / 91
59 General Search Directions III The question is then how to choose suitable directions? Chih-Jen Lin (National Taiwan Univ.) 59 / 91
60 Conjugate Gradient Method I We hope that p 1,..., p n are linearly independent and x k = arg min f (x) (5) x x 0 +span{p 1,...,p k } With (5), so Then span{p 1,..., p n } = R n, x n = arg min x R n f (x) Ax n = b Chih-Jen Lin (National Taiwan Univ.) 60 / 91
61 Conjugate Gradient Method II and the procedure stops at most n iterations But how to maintain (5)? Let x k = x 0 + P k 1 y + αp k, where P k 1 = [ p 1,..., p k 1 ], y R k 1, α R Chih-Jen Lin (National Taiwan Univ.) 61 / 91
62 Conjugate Gradient Method III f (x k ) = 1 2 (x 0 + P k 1 y + αp k ) T A(x 0 + P k 1 y + αp k ) b T (x 0 + P k 1 y + αp k ) = 1 2 (x 0 + P k 1 y) T A(x 0 + P k 1 y) b T (x 0 + P k 1 y) + αp T k A(x 0 + P k 1 y) b T (αp k ) + α2 2 pt k Ap k =f (x 0 + P k 1 y) + αp T k AP k 1 y αp T k r 0 + α2 2 pt k Ap k Chih-Jen Lin (National Taiwan Univ.) 62 / 91
63 Conjugate Gradient Method IV min f (x) x x 0 +span{p 1,...,p k } = min y,α f (x 0 + P k 1 y + αp k ) is difficult because the term αp T k AP k 1 y involves both α and y Chih-Jen Lin (National Taiwan Univ.) 63 / 91
64 Conjugate Gradient Method V If then and p k span{ap 1,..., Ap k 1 }, min f (x) x x 0 +span{p 1,...,p k } = min y αp T k AP k 1 y = 0 f (x 0 + P k 1 y) + min( αpk T r 0 + α2 α 2 pt k Ap k ), two independent optimization prolems Chih-Jen Lin (National Taiwan Univ.) 64 / 91
65 Conjugate Gradient Method VI Therefore, we require p k is A-conjugate to p 1,..., p k 1. That is By induction, p T i Ap k = 0, i = 1,..., k 1 x k 1 = arg min y f (x 0 + P k 1 y) The solution of the second problem is α k = pt k r 0 p T k Ap k Chih-Jen Lin (National Taiwan Univ.) 65 / 91
66 Conjugate Gradient Method VII Because of A-conjugacy, pk T r k 1 = pk T (b Ax k 1 ) = pk T (b A(x 0 + P k 1 y k 1 )) = pk T r 0 We have x k = x k 1 + α k p k New algorithm Chih-Jen Lin (National Taiwan Univ.) 66 / 91
67 Conjugate Gradient Method VIII k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 Choose any p k span{ap 1,..., Ap k 1 } such that p T k r k 1 0 α k = p T k r k 1/p T k Ap k x k = x k 1 + α k p k r k = b Ax k end Next how to choose p k? One way is to minimize the distance to r k 1 : Chih-Jen Lin (National Taiwan Univ.) 67 / 91
68 Conjugate Gradient Method IX Reason: r k 1 now negative gradient direction The algorithm becomes k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 if k = 1 p 1 = r 0 else Let p k minimize p r k 1 2 over all vectors p span{ap 1,..., Ap k 1 } end Chih-Jen Lin (National Taiwan Univ.) 68 / 91
69 Conjugate Gradient Method X end α k = p T k r k 1/p T k Ap k x k = x k 1 + α k p k r k = b Ax k Chih-Jen Lin (National Taiwan Univ.) 69 / 91
70 Conjugate Gradient Method XI Lemma 2 If p k minimizes p r k 1 2 over all vectors p span{ap 1,..., Ap k 1 }, then where z k 1 solves p k = r k 1 AP k 1 z k 1 min z r k 1 AP k 1 z 2, P k = [p 1,..., p k ]: an n k matrix Chih-Jen Lin (National Taiwan Univ.) 70 / 91
71 Conjugate Gradient Method XII Proof: AP k 1 z: space spanned by Ap 1,..., Ap k 1 r k 1 is not in the above space z: Coefficients of the linear combination of Ap 1,..., Ap k 1 p k span{ap 1,..., Ap k 1 } p k = r k 1 AP k 1 z k 1 Chih-Jen Lin (National Taiwan Univ.) 71 / 91
72 Conjugate Gradient Method XIII Theorem 3 After j iterations, we have Proof: r j = r j 1 α j Ap j Pj T r j = 0 (6) span{p 1,..., p j } = span{r 0,..., r j 1 } = span{b, Ab,..., A j 1 b} ri T r j = 0 for all i j Chih-Jen Lin (National Taiwan Univ.) 72 / 91
73 Conjugate Gradient Method XIV r j = b Ax j = b Ax j 1 + A(x j 1 x j ) = r j 1 α j Ap j r i, r j mutually orthogonal Proofs of others omitted Now we want to find z k 1 Chih-Jen Lin (National Taiwan Univ.) 73 / 91
74 Conjugate Gradient Method XV z k 1 a vector with length k 1 [ w z k 1 =, w : (k 2) 1, µ : 1 1 µ] p k = r k 1 AP k 1 z k 1 (7) = r k 1 AP k 2 w µap k 1 = (1 + µ )r k 1 + s k 1 α k 1 (8) and r k 1 = r k 2 α k 1 Ap k 1 Chih-Jen Lin (National Taiwan Univ.) 74 / 91
75 Conjugate Gradient Method XVI where We have and s k 1 µ α k 1 r k 2 AP k 2 w (9) ri T r j = 0, i, j AP k 2 w span{ap 1,..., Ap k 2 } = span{ab,..., A k 2 b} span{r 0,..., r k 2 } Chih-Jen Lin (National Taiwan Univ.) 75 / 91
76 Conjugate Gradient Method XVII Hence r T k 1 (AP k 2w) = 0 s T k 1r k 1 = 0 (10) Select µ, s k 1 such that r k 1 AP k 1 z k 1 is minimized (using Lemma 2) The reason of minimizing instead of r k 1 AP k 1 z k 1 p r k 1 2, p span{ap 1,..., Ap k 1 } (11) is that (11) is constrained. Chih-Jen Lin (National Taiwan Univ.) 76 / 91
77 Conjugate Gradient Method XVIII From (8) and (10), select µ and w such that (1 + µ α k 1 )r k 1 + s k 1 2 = (1 + µ α k 1 )r k µ α k 1 r k 2 AP k 2 w 2 is minimized From Lemma 2, the solution of min r k 2 AP k 2 z is p k 1 = r k 2 AP k 2 z Chih-Jen Lin (National Taiwan Univ.) 77 / 91
78 Conjugate Gradient Method XIX Therefore, s k 1 is a multiple of p k 1 From (8), p k span{r k 1, p k 1 } Assume p k = r k 1 + β k p k 1 (12) This assumption is fine as we will adjust α later. That is, a direction parallel to the real solution of min p r k 1 is enough From (6), p T k 1Ap k = 0, p T k 1r k 1 = 0 Chih-Jen Lin (National Taiwan Univ.) 78 / 91
79 Conjugate Gradient Method XX With (12), Ap k = Ar k 1 + β k Ap k 1 p T k 1Ap k = p T k 1Ar k 1 + β k p T k 1Ap k 1 = 0 β k = pt k 1 Ar k 1 p T k 1 Ap k 1 α k = (r k 1 + β k p k 1 ) T r k 1 p T k Ap k = r T k 1 r k 1 p T k Ap k The conjugate gradient method Chih-Jen Lin (National Taiwan Univ.) 79 / 91
80 Conjugate Gradient Method XXI k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 if k = 1 p 1 = r 0 else β k = p T k 1 Ar k 1/p T k 1 Ap k 1 p k = r k 1 + β k p k 1 end α k = r T k 1 r k 1/p T k Ap k x k = x k 1 + α k p k r k = b Ax k Chih-Jen Lin (National Taiwan Univ.) 80 / 91
81 Conjugate Gradient Method XXII end The computational efforts 3 matrix-vector products each iteration: Ar k 1, Ap k 1, Ax k Chih-Jen Lin (National Taiwan Univ.) 81 / 91
82 Conjugate Gradient Method XXIII Further simplification r k r k 1 r T k 1r k 1 = r k 1 α k Ap k = r k 2 α k 1 Ap k 1 = rk 1r T k 2 α k 1 rk 1Ap T k 1 = 0 α k 1 r T k 1Ap k 1 r T k 2r k 1 = r T k 2r k 2 α k 1 r T k 2Ap k 1 = r T k 2r k 2 α k 1 (p k 1 β k 1 p k 2 ) T Ap k 1 = r T k 2r k 2 α k 1 p T k 1Ap k 1 = 0 r T k 2r k 2 = α k 1 p T k 1Ap k 1 Chih-Jen Lin (National Taiwan Univ.) 82 / 91
83 Conjugate Gradient Method XXIV β k = pt k 1 Ar k 1 p T k 1 Ap k 1 A simplified version: k = 0; x 0 = 0; r 0 = b while r k 0 k = k + 1 if k = 1 p 1 = r 0 else β k = r T k 1 r k 1/r T k 2 r k 2 p k = r k 1 + β k p k 1 = r T k 1 r k 1/α k 1 r T k 2 r k 2/α k 1 = r T k 1 r k 1 r T k 2 r k 2 Chih-Jen Lin (National Taiwan Univ.) 83 / 91
84 Conjugate Gradient Method XXV end α k = rk 1 T r k 1/pk T Ap k x k = x k 1 + α k p k r k = r k 1 α k Ap k end One matrix vector product r k 0 is not a practical termination criterion Too many inner products The final version Chih-Jen Lin (National Taiwan Univ.) 84 / 91
85 Conjugate Gradient Method XXVI k = 0; x = 0; r = b; ρ 0 = r 2 2 while ρ k > ɛ b 2 and k < k max k = k + 1 if k = 1 p = r else β = ρ k 1 /ρ k 2 p = r + βp end w = Ap α = ρ k 1 /p T w x = x + αp Chih-Jen Lin (National Taiwan Univ.) 85 / 91
86 Conjugate Gradient Method XXVII r = r αw ρ k = r 2 2 end Numerical error will cause the number of iterations > n Slow convergence Convergence properties Theorem 4 If A = I + B is an n n symmetric positive definite matrix and rank(b) = r, then the conjugate gradient method converges in at most r + 1 steps Chih-Jen Lin (National Taiwan Univ.) 86 / 91
87 Conjugate Gradient Method XXVIII The case of A = I : x 0 = 0, r 0 = b p 1 = r 0 = b α = r T 0 r 0 /p T 1 Ap 1 = b T b/b T Ab = 1 x 1 = p 1 = b r 1 = r 0 α 1 Ap 1 = b b = 0 The conjugate gradient method stops in one iteration Chih-Jen Lin (National Taiwan Univ.) 87 / 91
88 Conjugate Gradient Method XXIX An error bound in terms of the norm x T Ax: Theorem 5 If Ax = b, κ 1 x x k A 2 x x 0 A ( ) k, κ + 1 where x A = x T Ax κ: the condition number of A κ 1, ( κ 1 κ+1 ) k smaller Chih-Jen Lin (National Taiwan Univ.) 88 / 91
89 Conjugate Gradient Method XXX In general, if the condition of A is better fewer CG iterations Where is PD used? For α, 1 2 α2 p T Ap + We need p T Ap > 0 for the minimization. That is, we need PD to ensur that CG solves 1 min x 2 x T Ax b T x Chih-Jen Lin (National Taiwan Univ.) 89 / 91
90 Homework 6 I Solving a linear system with the largest symmetric positive definite matrix in Matrix Market Implement three methods on C or C++: Jacobi, Gauss-Seidel, and CG Solve Ax = e, e: vector of all ones You may need to set the maximal number iterations if a method takes too many iterations If none converges, try to do diagonal scaling first: Let C = diag(a) 1/2 Chih-Jen Lin (National Taiwan Univ.) 90 / 91
91 Homework 6 II Solve CACy = Ce where x = Cy. After finding solutions, analyse the error by Ax b Chih-Jen Lin (National Taiwan Univ.) 91 / 91
Vector and Matrix Norms I
Vector and Matrix Norms I Scalar, vector, matrix How to calculate errors? Scalar: absolute error: ˆα α relative error: Vectors: vector norm Norm is the distance!! ˆα α / α Chih-Jen Lin (National Taiwan
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationIterative techniques in matrix algebra
Iterative techniques in matrix algebra Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan September 12, 2015 Outline 1 Norms of vectors and matrices 2 Eigenvalues and
More informationSolving Dense Linear Systems I
Solving Dense Linear Systems I Solving Ax = b is an important numerical method Triangular system: [ l11 l 21 if l 11, l 22 0, ] [ ] [ ] x1 b1 = l 22 x 2 b 2 x 1 = b 1 /l 11 x 2 = (b 2 l 21 x 1 )/l 22 Chih-Jen
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationLecture Note 7: Iterative methods for solving linear systems. Xiaoqun Zhang Shanghai Jiao Tong University
Lecture Note 7: Iterative methods for solving linear systems Xiaoqun Zhang Shanghai Jiao Tong University Last updated: December 24, 2014 1.1 Review on linear algebra Norms of vectors and matrices vector
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationCME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.
CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationCOURSE Numerical methods for solving linear systems. Practical solving of many problems eventually leads to solving linear systems.
COURSE 9 4 Numerical methods for solving linear systems Practical solving of many problems eventually leads to solving linear systems Classification of the methods: - direct methods - with low number of
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationInterpolation and Polynomial Approximation I
Interpolation and Polynomial Approximation I If f (n) (x), n are available, Taylor polynomial is an approximation: f (x) = f (x 0 )+f (x 0 )(x x 0 )+ 1 2! f (x 0 )(x x 0 ) 2 + Example: e x = 1 + x 1! +
More informationScientific Computing: Solving Linear Systems
Scientific Computing: Solving Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 September 17th and 24th, 2015 A. Donev (Courant
More informationLINEAR SYSTEMS (11) Intensive Computation
LINEAR SYSTEMS () Intensive Computation 27-8 prof. Annalisa Massini Viviana Arrigoni EXACT METHODS:. GAUSSIAN ELIMINATION. 2. CHOLESKY DECOMPOSITION. ITERATIVE METHODS:. JACOBI. 2. GAUSS-SEIDEL 2 CHOLESKY
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More informationJACOBI S ITERATION METHOD
ITERATION METHODS These are methods which compute a sequence of progressively accurate iterates to approximate the solution of Ax = b. We need such methods for solving many large linear systems. Sometimes
More informationCOURSE Iterative methods for solving linear systems
COURSE 0 4.3. Iterative methods for solving linear systems Because of round-off errors, direct methods become less efficient than iterative methods for large systems (>00 000 variables). An iterative scheme
More informationScientific Computing with Case Studies SIAM Press, Lecture Notes for Unit VII Sparse Matrix
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 1: Direct Methods Dianne P. O Leary c 2008
More informationComputational Methods. Systems of Linear Equations
Computational Methods Systems of Linear Equations Manfred Huber 2010 1 Systems of Equations Often a system model contains multiple variables (parameters) and contains multiple equations Multiple equations
More information6. Iterative Methods for Linear Systems. The stepwise approach to the solution...
6 Iterative Methods for Linear Systems The stepwise approach to the solution Miriam Mehl: 6 Iterative Methods for Linear Systems The stepwise approach to the solution, January 18, 2013 1 61 Large Sparse
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationLab 1: Iterative Methods for Solving Linear Systems
Lab 1: Iterative Methods for Solving Linear Systems January 22, 2017 Introduction Many real world applications require the solution to very large and sparse linear systems where direct methods such as
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationConjugate Gradient Method
Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest
More informationG1110 & 852G1 Numerical Linear Algebra
The University of Sussex Department of Mathematics G & 85G Numerical Linear Algebra Lecture Notes Autumn Term Kerstin Hesse (w aw S w a w w (w aw H(wa = (w aw + w Figure : Geometric explanation of the
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More information7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.
7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 7.3 The Jacobi and Gauss-Siedel Iterative Techniques
More informationThe conjugate gradient method
The conjugate gradient method Michael S. Floater November 1, 2011 These notes try to provide motivation and an explanation of the CG method. 1 The method of conjugate directions We want to solve the linear
More informationComputational Economics and Finance
Computational Economics and Finance Part II: Linear Equations Spring 2016 Outline Back Substitution, LU and other decomposi- Direct methods: tions Error analysis and condition numbers Iterative methods:
More informationTopics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52 Conjugate
More informationConjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More informationJordan Journal of Mathematics and Statistics (JJMS) 5(3), 2012, pp A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS
Jordan Journal of Mathematics and Statistics JJMS) 53), 2012, pp.169-184 A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS ADEL H. AL-RABTAH Abstract. The Jacobi and Gauss-Seidel iterative
More informationChapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices
Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay
More informationApplied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give
More information9. Iterative Methods for Large Linear Systems
EE507 - Computational Techniques for EE Jitkomut Songsiri 9. Iterative Methods for Large Linear Systems introduction splitting method Jacobi method Gauss-Seidel method successive overrelaxation (SOR) 9-1
More informationChapter 3. Linear and Nonlinear Systems
59 An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them Werner Heisenberg (1901-1976) Chapter 3 Linear and Nonlinear Systems In this chapter
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationGaussian Elimination for Linear Systems
Gaussian Elimination for Linear Systems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 3, 2011 1/56 Outline 1 Elementary matrices 2 LR-factorization 3 Gaussian elimination
More informationCHAPTER 5. Basic Iterative Methods
Basic Iterative Methods CHAPTER 5 Solve Ax = f where A is large and sparse (and nonsingular. Let A be split as A = M N in which M is nonsingular, and solving systems of the form Mz = r is much easier than
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationthe method of steepest descent
MATH 3511 Spring 2018 the method of steepest descent http://www.phys.uconn.edu/ rozman/courses/m3511_18s/ Last modified: February 6, 2018 Abstract The Steepest Descent is an iterative method for solving
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 3: Iterative Methods PD
More informationThe amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.
AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative
More informationNumerical Linear Algebra And Its Applications
Numerical Linear Algebra And Its Applications Xiao-Qing JIN 1 Yi-Min WEI 2 August 29, 2008 1 Department of Mathematics, University of Macau, Macau, P. R. China. 2 Department of Mathematics, Fudan University,
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationITERATIVE METHODS BASED ON KRYLOV SUBSPACES
ITERATIVE METHODS BASED ON KRYLOV SUBSPACES LONG CHEN We shall present iterative methods for solving linear algebraic equation Au = b based on Krylov subspaces We derive conjugate gradient (CG) method
More informationYORK UNIVERSITY. Faculty of Science Department of Mathematics and Statistics MATH M Test #1. July 11, 2013 Solutions
YORK UNIVERSITY Faculty of Science Department of Mathematics and Statistics MATH 222 3. M Test # July, 23 Solutions. For each statement indicate whether it is always TRUE or sometimes FALSE. Note: For
More informationAnalytical formulas for calculating the extremal ranks and inertias of A + BXB when X is a fixed-rank Hermitian matrix
Analytical formulas for calculating the extremal ranks and inertias of A + BXB when X is a fixed-rank Hermitian matrix Yongge Tian CEMA, Central University of Finance and Economics, Beijing 100081, China
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationMath 471 (Numerical methods) Chapter 3 (second half). System of equations
Math 47 (Numerical methods) Chapter 3 (second half). System of equations Overlap 3.5 3.8 of Bradie 3.5 LU factorization w/o pivoting. Motivation: ( ) A I Gaussian Elimination (U L ) where U is upper triangular
More informationSolutions and Notes to Selected Problems In: Numerical Optimzation by Jorge Nocedal and Stephen J. Wright.
Solutions and Notes to Selected Problems In: Numerical Optimzation by Jorge Nocedal and Stephen J. Wright. John L. Weatherwax July 7, 2010 wax@alum.mit.edu 1 Chapter 5 (Conjugate Gradient Methods) Notes
More informationNo books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question.
Math 304 Final Exam (May 8) Spring 206 No books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question. Name: Section: Question Points
More informationNotes on PCG for Sparse Linear Systems
Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationGATE Engineering Mathematics SAMPLE STUDY MATERIAL. Postal Correspondence Course GATE. Engineering. Mathematics GATE ENGINEERING MATHEMATICS
SAMPLE STUDY MATERIAL Postal Correspondence Course GATE Engineering Mathematics GATE ENGINEERING MATHEMATICS ENGINEERING MATHEMATICS GATE Syllabus CIVIL ENGINEERING CE CHEMICAL ENGINEERING CH MECHANICAL
More informationLecture 18 Classical Iterative Methods
Lecture 18 Classical Iterative Methods MIT 18.335J / 6.337J Introduction to Numerical Methods Per-Olof Persson November 14, 2006 1 Iterative Methods for Linear Systems Direct methods for solving Ax = b,
More informationThe Solution of Linear Systems AX = B
Chapter 2 The Solution of Linear Systems AX = B 21 Upper-triangular Linear Systems We will now develop the back-substitution algorithm, which is useful for solving a linear system of equations that has
More informationSOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA
1 SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 2 OUTLINE Sparse matrix storage format Basic factorization
More informationClassical iterative methods for linear systems
Classical iterative methods for linear systems Ed Bueler MATH 615 Numerical Analysis of Differential Equations 27 February 1 March, 2017 Ed Bueler (MATH 615 NADEs) Classical iterative methods for linear
More informationScientific Computing: Dense Linear Systems
Scientific Computing: Dense Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 February 9th, 2012 A. Donev (Courant Institute)
More informationLecture 11. Fast Linear Solvers: Iterative Methods. J. Chaudhry. Department of Mathematics and Statistics University of New Mexico
Lecture 11 Fast Linear Solvers: Iterative Methods J. Chaudhry Department of Mathematics and Statistics University of New Mexico J. Chaudhry (UNM) Math/CS 375 1 / 23 Summary: Complexity of Linear Solves
More informationIntroduction. Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods. Example: First Order Richardson. Strategy
Introduction Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 Solve system Ax = b by repeatedly computing
More informationIntroduction to Iterative Solvers of Linear Systems
Introduction to Iterative Solvers of Linear Systems SFB Training Event January 2012 Prof. Dr. Andreas Frommer Typeset by Lukas Krämer, Simon-Wolfgang Mages and Rudolf Rödl 1 Classes of Matrices and their
More informationLecture # 20 The Preconditioned Conjugate Gradient Method
Lecture # 20 The Preconditioned Conjugate Gradient Method We wish to solve Ax = b (1) A R n n is symmetric and positive definite (SPD). We then of n are being VERY LARGE, say, n = 10 6 or n = 10 7. Usually,
More informationMath Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012.
Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems
More information10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method
54 CHAPTER 10 NUMERICAL METHODS 10. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after
More informationLinear Algebra. Brigitte Bidégaray-Fesquet. MSIAM, September Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble.
Brigitte Bidégaray-Fesquet Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble MSIAM, 23 24 September 215 Overview 1 Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationMath 1080: Numerical Linear Algebra Chapter 4, Iterative Methods
Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 March 2015 1 / 70 Topics Introduction to Iterative Methods
More informationA Review of Linear Algebra
A Review of Linear Algebra Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab: Implementations
More informationMA 265 FINAL EXAM Fall 2012
MA 265 FINAL EXAM Fall 22 NAME: INSTRUCTOR S NAME:. There are a total of 25 problems. You should show work on the exam sheet, and pencil in the correct answer on the scantron. 2. No books, notes, or calculators
More informationSolving Linear Systems of Equations
November 6, 2013 Introduction The type of problems that we have to solve are: Solve the system: A x = B, where a 11 a 1N a 12 a 2N A =.. a 1N a NN x = x 1 x 2. x N B = b 1 b 2. b N To find A 1 (inverse
More informationIterative solvers for linear equations
Spectral Graph Theory Lecture 15 Iterative solvers for linear equations Daniel A. Spielman October 1, 009 15.1 Overview In this and the next lecture, I will discuss iterative algorithms for solving linear
More informationScientific Computing WS 2018/2019. Lecture 9. Jürgen Fuhrmann Lecture 9 Slide 1
Scientific Computing WS 2018/2019 Lecture 9 Jürgen Fuhrmann juergen.fuhrmann@wias-berlin.de Lecture 9 Slide 1 Lecture 9 Slide 2 Simple iteration with preconditioning Idea: Aû = b iterative scheme û = û
More informationMath 577 Assignment 7
Math 577 Assignment 7 Thanks for Yu Cao 1. Solution. The linear system being solved is Ax = 0, where A is a (n 1 (n 1 matrix such that 2 1 1 2 1 A =......... 1 2 1 1 2 and x = (U 1, U 2,, U n 1. By the
More informationJae Heon Yun and Yu Du Han
Bull. Korean Math. Soc. 39 (2002), No. 3, pp. 495 509 MODIFIED INCOMPLETE CHOLESKY FACTORIZATION PRECONDITIONERS FOR A SYMMETRIC POSITIVE DEFINITE MATRIX Jae Heon Yun and Yu Du Han Abstract. We propose
More informationIntroduction to Numerical Analysis
Université de Liège Faculté des Sciences Appliquées Introduction to Numerical Analysis Edition 2015 Professor Q. Louveaux Department of Electrical Engineering and Computer Science Montefiore Institute
More informationSparse Linear Systems. Iterative Methods for Sparse Linear Systems. Motivation for Studying Sparse Linear Systems. Partial Differential Equations
Sparse Linear Systems Iterative Methods for Sparse Linear Systems Matrix Computations and Applications, Lecture C11 Fredrik Bengzon, Robert Söderlund We consider the problem of solving the linear system
More informationEigenvalues and Eigenvectors
Eigenvalues and Eigenvectors Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) Eigenvalues and Eigenvectors Fall 2015 1 / 14 Introduction We define eigenvalues and eigenvectors. We discuss how to
More informationLinear Algebraic Equations
Linear Algebraic Equations 1 Fundamentals Consider the set of linear algebraic equations n a ij x i b i represented by Ax b j with [A b ] [A b] and (1a) r(a) rank of A (1b) Then Axb has a solution iff
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationNumerical Analysis: Solving Systems of Linear Equations
Numerical Analysis: Solving Systems of Linear Equations Mirko Navara http://cmpfelkcvutcz/ navara/ Center for Machine Perception, Department of Cybernetics, FEE, CTU Karlovo náměstí, building G, office
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that
More informationTsung-Ming Huang. Matrix Computation, 2016, NTNU
Tsung-Ming Huang Matrix Computation, 2016, NTNU 1 Plan Gradient method Conjugate gradient method Preconditioner 2 Gradient method 3 Theorem Ax = b, A : s.p.d Definition A : symmetric positive definite
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More informationSummer Review Packet AP Calculus
Summer Review Packet AP Calculus ************************************************************************ Directions for this packet: On a separate sheet of paper, show your work for each problem in this
More informationSparse Matrices and Iterative Methods
Sparse Matrices and Iterative Methods K. 1 1 Department of Mathematics 2018 Iterative Methods Consider the problem of solving Ax = b, where A is n n. Why would we use an iterative method? Avoid direct
More informationyou expect to encounter difficulties when trying to solve A x = b? 4. A composite quadrature rule has error associated with it in the following form
Qualifying exam for numerical analysis (Spring 2017) Show your work for full credit. If you are unable to solve some part, attempt the subsequent parts. 1. Consider the following finite difference: f (0)
More information