Solving linear equations with Gaussian Elimination (I)
|
|
- Gabriella Pearson
- 5 years ago
- Views:
Transcription
1 Term Projects Solving linear equations with Gaussian Elimination The QR Algorithm for Symmetric Eigenvalue Problem The QR Algorithm for The SVD Quasi-Newton Methods
2 Solving linear equations with Gaussian Elimination I Gaussian Elimination with Partial Pivoting GEPP Gaussian Elimination with Complete Pivoting GECP Iterative Refinement Comparison with matlab \ function
3 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn
4 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs
5 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs row interchange: permutation
6 Gaussian Elimination with partial pivoting a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn Let E j be the j-th row of A for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv s def = argmax s j n a js, E s E pivs row interchange: permutation eliminating xs from E s+1 through E n : l js = a js a ss, s + 1 j n, a jk = a jk l js a sk, s + 1 j, k n
7 GEPP as LU factorization Theorem: Let A = a ij R n n be non-singular Then GEPP computes an LU factorization with permutation matrix P such that P A = L U = P is the accumulation of row interchanges, L is unit lower triangular, and U upper triangular
8 Solving general linear equations with GEPP A x = b, P A = L U interchanging components in b P A x = P b, L U x = P b solving for b with forward and backward substitution x = L U 1 P b = U 1 L 1 P b
9 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn
10 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: ip, jp def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp
11 Gaussian Elimination with complete pivoting A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: ip, jp def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp eliminating x s from E s+1 through E n : a jk = a jk a js a ss a sk overwrite ==== a jk, s + 1 j n, s + 1 k n, More numerically stable, too expensive GECP computes LU factorization with permutations P row and P column, P row A P column = L U =
12 Error Bounds and Iterative Refinement Assume that x is an approximation to the solution x of A x = b Residual r def = b A x = A x x Thus small x x implies small r T hm: Assume x is an approximate solution with r = b A x Then for any natural norm, x x A 1 r, x x x κ A r b, where κ A def = A A 1 is the condition number of A
13 Iterative Refinement I Let A x = b with non-singular A and non-zero b Let F be an in-exact equation solver, so F b is approximate solution Assume F is accurate enough that there exists a ρ < 1 so b A F b b ρ for any b 0
14 Iterative Refinement I Let A x = b with non-singular A and non-zero b Let F be an in-exact equation solver, so F b is approximate solution Assume F is accurate enough that there exists a ρ < 1 so b A F b b ρ for any b 0 For this project, F will be: LU factorization without pivoting, LU factorization with partial pivoting, and LU factorization with complete pivoting F b = U 1 L 1 b
15 Iterative Refinement II Given a tolerance τ > 0 and x 0 Initialize r 0 = b A x 0 for k = 0, 1, Compute x k = F r k, x k+1 = x k + x k, r k+1 = r k A x k If r k+1 τ b stop
16 Project Requirements Implement GE, GEPP, GECP, with and without iterative refinement Compare with matlab \ function in terms of accuracy residual norm and execution time
17 The QR Algorithm for Symmetric Eigenvalue Problem Householder reduction to tridiagonal Implicit bulge Chasing scheme for tridiagonal QR Algorithm Recover eigenvectors
18 Householder Reduction for symmetric A R n n Let v j R n j be the unit vector in the Householder Reflection matrix Ĥj = I 2 v j vj T so that def = R n n, j = 1,, n 1 H j Ij Ĥj Perform n 1 Householder reflections so that α 1 β 1 β 1 α 2 β 2 H n 1 H 1 A H1 T Hn 1 T = β 2 α 3 β 3 def = T β n 1 α n Householder s Method, where H def = H T 1 HT n 1 H T A H = T, or A = H T H T,
19 Computing H For j = n 2, n 3,, 2, 1, let I j def H j+1 H n 1 = 1 H j H j+1 H n 1 = where H j = = Ij Ĥj I 2 v j v T j 1 H j+1 1 I j 1 H j+1 H j+1 2 v j vj T H j+1 1 Then = Ij H j+1 H j,
20
21 Project Requirements Householder reduction to tridiagonal form Implicit Bulge Chasing tridiagonal QR Algorithm for all eigenvalues Accumulate all Householder reflections and Givens rotations to recover all eigenvectors Total cost On 3 flops Compare with matlab eig function in terms of accuracy residual and orthogonality and execution time Algorithm Details
22 The QR Algorithm for SVD Householder reduction to bidiagonal form Implicit bulge chasing scheme for bidiagonal SVD Recover singular vectors
23 Householder Bidiagonal Reduction for A R n n I Partition matrix A = a 1 Â 1 R n n, Â 1 R n n 1 Let u 1 R n be the unit vector in the Householder Reflection def matrix G 1 = Ĝ1 = I 2 u 1 u T 1 so that ± a1 Ĝ 1 a 1 = 2 def α1 = and 0 0 def α1 b G 1 A = Ĝ1 a 1 Ĝ 1 Â 1 = T 1 0 A 1 Let v 1 R n 1 be the unit vector in the Householder Reflection matrix Ĥ1 = I 2 v 1 v1 T so that, ± b1 Ĥ 1 b 1 = 2 def β1 1 = With H = Ĥ1 α G 1 A H 1 = 1 b T 1 Ĥ1 def α1 β 1 0 T = 0 A 1 Ĥ 1 0 a 2 Â 2,
24 α1 β 1 0 T G 1 A H 1 = 0 a 2 Â 2, Â 2 R n 1 n 2 Let u 2 R n 1 be the unit vector in Ĝ2 = I 2 u 2 u T 2 so that ± a2 Ĝ 2 a 2 = 2 def α2 1 = With G = Ĝ2 α1 β 1 0 T α 1 β 1 0 T def G 2 G 1 A H 1 = = 0 α 2 b T 2 0 Ĝ 2 a 2 Ĝ 2 Â A 2, Let v 2 R n 2 be the unit vector in Ĥ2 = I 2 v 2 v2 T so that, ± b2 Ĥ 2 b 2 = 2 def β2 I2 = With H = Ĥ2 α 1 β 1 0 T α 1 β T G 2 G 1 A H 1 H 2 = 0 α 2 b T 2 Ĥ2 def = 0 α 2 β 2 0 T 0 0 A 2 Ĥ a 3 Â 3,
25 Householder Bidiagonal Reduction for A R n n III After n 1 pairs of Householder Reflections, α 1 β 1 α 2 β 2 G n 1 G 2 G 1 A H 1 H 2 H n 1 = α 3 βn 1 B is an upper bidiagonal matrix Next step is QR algorithm for bidiagonal SVD α n def = B
26 Input matrix B = α 1 β 1 α 2 β 2 α 3 βn 1, tolerance τ α n β n Algorithm 1 BidiSVD B, τ while not yet converged do if n = 1 then return SVD of B for j = 1,, n 1 do if β j τ then return BidiSVD B 1 : j, 1 : j, τ and BidiSVD B j + 1 : n, j + 1 : n + 1, τ end if if α j τ then return BidiSVD B 1 : j 1, 1 : j, τ and BidiSVD B j : n, j + 1 : n + 1, τ end if end for Compute shift σ Bulge-chasing QR with σ end while
27 All forms Bidiagonal B = α 1 β 1 α 2 β 2 α 3 βn 1, B = β 1 α 2 β 2 α 3 βn 1 B = α 1 β 1 α 2 β 2 α n α 3 βn 1 β n, B = β 1 α 2 β 2 α 3 βn 1 α n α n α n β n
28 All forms Bidiagonal B = α 1 β 1 α 2 β 2 α 3 βn 1, B = β 1 α 2 β 2 α 3 βn 1 B = α 1 β 1 α 2 β 2 α n α 3 βn 1 β n, B = β 1 α 2 β 2 α 3 βn 1 α n α n Next: QR for SVD of B = β 1 α 2 β 2 α 3 βn 1 α n β n α n β n
29 Bidiagonal SVD B B T is symmetric tridiagonal β1 2 α 2 β 1 α 2 β 1 α2 2 + β2 2 α 3 β 2 B B T = α 3 β 2 α3 2 + β2 3 αn β n 1 Bottom 2 2 matrix α 2 n 1 + βn 1 2 α n β n 1 α n β n 1 αn 2 + βn 2 = S S T, with S def = α n β n 1 α 2 n + β 2 n αn 1 β n 1 α n β n Shift σ is the singular value of S closer to αn 2 + βn 2 Bulge-chasing QR based on QR factorization of β1 2 σ2 α 2 β 1 B B T σ 2 α I = 2 β 1 α2 2 + β2 2 σ2 αn β n 1 α n β n 1 αn 2 + βn 2 σ 2
30 Bulge-chasing QR for Bidiagonal SVD I Givens rotation on first column of B B T σ 2 I : G 1 B B T σ 2 def I = I n 2 s 1 c 1 α 2 β 1 c1 s 1 β 2 1 σ 2 0 = 0 0 G 1 is the first Givens rotation towards QR factorization of B B T σ 2 I d1 f Creating bulge: 1 def c1 s = 1 β1 0 d 2 f 2 s 1 c 1 α 2 β 2 G 1 B = d 1 f 1 d 2 f 2 α 3 βn 1 = + α n β n Next: Chase Bulge + down along the diagonals
31 Bulge-chasing QR for Bidiagonal SVD II G 1 B = d 1 f 1 d 2 f 2 α 3 βn 1 = + α n β n Givens rotation on first row, H 1 def = c1 s 1 s 1 c 1 I n 2 d1 f 1 0 T H1 T = d1 2 + f T def = β1 0 T Compute d2 f 2 α 3 c1 s 1 s 1 c 1 T def = d 2 f 2 d 3 f 3
32 Bulge-chasing QR for Bidiagonal SVD III 1 B H 1 = β 1 d 2 f 2 d 3 f 3 β 3 α 4 βn 1 = + α n β n Givens rotation on first column, G 2 def = 1 c 2 s 2 s 2 c 2 I n 3 c2 s 2 d 2 f 2 0 s 2 c 2 d 3 f 3 β 3 def = α2 d 2 f 2 0 d 3 f 3
33 Bulge-chasing QR for Bidiagonal SVD IV G 2 G 1 B H 1 = β 1 α 2 d 2 f 2 d 3 f 3 α 4 βn 1 = + α n β n New Givens rotation: d2 f 2 T β c2 s 2 0 d 3 f 2 3 = d s 0 α 2 c 3 f d 4 f 4 Repeat procedure until bulge disappears at the lower right corner
34 Project Requirements Householder reduction to bidiagonal form Implicit Bulge Chasing bidiagonal QR Algorithm for all singular values Accumulate all Householder reflections and Givens rotations to recover all singular vectors Total cost On 3 flops Compare with matlab svd function in terms of accuracy residual and orthogonality and execution time
35 Quasi-Newton Methods BFGS for minx g x I BFGS Method: xk+1 = xk α Bk 1 g xk, I I I I where Bk Rn n Desired Property I: Mimics B xk in some sense 1 Desired Property II: Easy to compute Bk+1 from Bk 1 Ensure symmetry in Bk Step size selection as in Steepest Descent
36 BFGS Method: Derivation Assume for some step k 0, we have available x k R n and B k R n n By BFGS Method: x k+1 = x k B 1 k g x k Secant equation g x k+1 g x k H x k x k+1 x k BFGS ideas: Approximate secant equation: Bk+1 s k+1 = y k+1, where def y k+1 = g x k+1 g x k def, s k+1 = x k+1 x k Choose a special, symmetric Bk+1 that does not differ much from B k B k+1 = B k + y k+1yk+1 T yk+1 T s B k s k+1 s T k+1 B k k+1 s T k+1 B k s k+1
37 BFGS Method Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, x k+1 = x k α B 1 k g B k+1 = B k + y k+1yk+1 T yk+1 T s k+1 x k+1 g y k+1 = g x k, B k s k+1 s T k+1 B k s T k+1 B k s k+1, with x k, s k+1 = x k+1 x k
38 BFGS Method Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, x k+1 = x k α B 1 k g B k+1 = B k + y k+1yk+1 T yk+1 T s k+1 x k+1 g y k+1 = g Practical details: x k, B k s k+1 s T k+1 B k s T k+1 B k s k+1, with x k, s k+1 = x k+1 x k B k remains SPD for convex problems Cholesky factorization of B k can help to compute Cholesky factorization of B k+1? updating Cholesky factorization BFGS inverse update: T + s k+1 s T k+1 B 1 k+1 = I s k+1 y T k+1 s T k+1 y k+1 B 1 k I s k+1 y T k+1 s T k+1 y k+1 s T k+1 y k+1
39 Limited Memory BFGS L-BFGS Method I B 1 k+1 maybe too large to fit in memory L-BFGS: Mimics BFGS, but with linear memory BFGS inverse update, from B 1 k B 1 k+1 = I s k+1 y T k+1 s T k+1 y k+1 B 1 k to B 1 k+1 : I s k+1 yk+1 T T s T k+1 y + s k+1 s T k+1 k+1 s T k+1 y k+1 Big Idea: Go back only m + 1 steps in inverse update
40 Limited Memory BFGS L-BFGS Method II B 1 k+1 = s k+1 s T k+1 s T k+1 y k+1 B 1 k m m 1 + I s k+1 y T k+1 s T k+1 y k+1 I y k+1 m s T k+1 m s T k+1 m y k+1 m + I s k+1 yk+1 T s T j=0 k+1 y k+1 sk j s T k j s T k j y k j I y k+1 j s T k+1 j s T k+1 j y k+1 j I s k+1 m yk+1 m T s T k+1 m y k+1 m I y k+1 s T k+1 s T k+1 y k+1 I s k+1 j yk+1 j T s T k+1 j y k+1 j I y k+1 s T k+1 s T k+1 y k+1 Big Idea: Go back only m + 1 steps in inverse update: Set B 1 k m = B 1 0
41 Computing B 1 k+1 h for any vector h I B 1 k+1 h = α k+1 s k+1 + m 1 I s k+1 yk+1 T s T k+1 y k+1 + I s k+1 yk+1 T s T j=0 k+1 y k+1 sk j s T k j s T k j y q k+1 j, k j I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j y T k+1 j s T k+1 j y k+1 j B 1 0 q k+1 m where α k+1 = q t def = s T k+1 h s T k+1 y, q k+1 = h α k+1 y k+1, k+1 I y t s T t s T I y k+1 s T k+1 t y t s T k+1 y h k+1 = q t+1 α t y t, α t = st t q t+1 s T t y t, t = k,, k + 1 m
42 Computing B 1 h for any vector h II k+1 Let z k+1 m = B0 1 q k+1 m Then B 1 k+1 h = α k+1 s k m 1 j=0 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 y T k+1 s T k+1 y k+1 I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j y T k+1 j s T k+1 j y k+1 j z k+1 m α k j s k j
43 Computing B 1 h for any vector h II k+1 Let z k+1 m = B0 1 q k+1 m Then B 1 k+1 h = α k+1 s k m 1 j=0 = α k+1 s k m 2 j=0 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 yk+1 T s T k+1 y k+1 I s k+1 y T k+1 s T k+1 y k+1 I s k+1 m yk+1 m T s T k+1 m y k+1 m I s k+1 j yk+1 j T s T k+1 j y k+1 j I s k+2 m yk+2 m T s T k+2 m y k+2 m I s k+1 j y T k+1 j s T k+1 j y k+1 j z k+1 m α k j s k j z k+2 m α k j s k j, where z k+2 m = z k+1 m + α k+1 m β k+1 m s k+1 m, β k+1 m = yt k+1 m z k+1 m s T k+1 m y k+1 m
44 Algorithm 2 Computing B 1 k+1 h for any vector h Set q = h for t = k + 1, k,, k + 1 m do Compute α t = st t q, q = q α s T t y t y t t end for Compute z = B0 1 q for t = k + 1 m,, k + 1 do Compute β = yt t z, z = z + α s T t y t β s t t end for Return z
45 Step-size Selection Ensure reduction in g x: Algorithm 3 will halt with an α 3 > 0 so that g x k + α 3 d k < g x k Algorithm 3 g x Reduction Set α 3 = 1 while g x k + α 3 d k g x k do Set α 3 = α 3 2 end while
46 Step-size Selection Ensure reduction in g x: Algorithm 3 will halt with an α 3 > 0 so that g x k + α 3 d k < g x k Algorithm 3 g x Reduction Set α 3 = 1 while g x k + α 3 d k g x k do Set α 3 = α 3 2 end while Ensure sufficient reduction in g x with α k : Algorithm 4 g x Sufficient Reduction Set α 2 = α 3 /2 Compute coefficients h 0, h 1, h 2 so P α = h 0 + h 1 α + h 2 α α α 2 interpolates g x k + α d k at α = 0, α 2, α 3 Set α 0 = 1 2 α 2 h 1 /h 2 α k = argmin α [α0,α 2,α 3 ]g x k + α d k
47 BFGS Method for min x g x Initialization: Given x 0 R n Choose symmetric B 0 R n n, typically SPD For k = 0, 1,, Compute x d = B 1 k g x k Compute step size α x k+1 = x k α B 1 k g x k, B k+1 = B k + y k+1yk+1 T yk+1 T s B k s k+1 s T k+1 B k k+1 s T k+1 B, with k s k+1 y k+1 = g x k+1 g x k, s k+1 = x k+1 x k
48 Project Requirements Implement BFGS and L-BFGS Run BFGS and L-BFGS with logistic objective function
Math128B: Numerical Analysis Programming Project, Due April 30, 2018
Math128B: Numerical Analysis Programming Project, Due April 30, 2018 Choose any one of the following 4 projects, details of which are in Week 11 Lecture Notes. Turn in a.m file Projectxyyyy.m, where x
More information7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.
7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 7.3 The Jacobi and Gauss-Siedel Iterative Techniques
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationCS412: Lecture #17. Mridul Aanjaneya. March 19, 2015
CS: Lecture #7 Mridul Aanjaneya March 9, 5 Solving linear systems of equations Consider a lower triangular matrix L: l l l L = l 3 l 3 l 33 l n l nn A procedure similar to that for upper triangular systems
More informationToday s class. Linear Algebraic Equations LU Decomposition. Numerical Methods, Fall 2011 Lecture 8. Prof. Jinbo Bi CSE, UConn
Today s class Linear Algebraic Equations LU Decomposition 1 Linear Algebraic Equations Gaussian Elimination works well for solving linear systems of the form: AX = B What if you have to solve the linear
More informationMATH 3511 Lecture 1. Solving Linear Systems 1
MATH 3511 Lecture 1 Solving Linear Systems 1 Dmitriy Leykekhman Spring 2012 Goals Review of basic linear algebra Solution of simple linear systems Gaussian elimination D Leykekhman - MATH 3511 Introduction
More informationLecture 5 Singular value decomposition
Lecture 5 Singular value decomposition Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 12: Gaussian Elimination and LU Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Gaussian Elimination
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationThis can be accomplished by left matrix multiplication as follows: I
1 Numerical Linear Algebra 11 The LU Factorization Recall from linear algebra that Gaussian elimination is a method for solving linear systems of the form Ax = b, where A R m n and bran(a) In this method
More informationSolving Dense Linear Systems I
Solving Dense Linear Systems I Solving Ax = b is an important numerical method Triangular system: [ l11 l 21 if l 11, l 22 0, ] [ ] [ ] x1 b1 = l 22 x 2 b 2 x 1 = b 1 /l 11 x 2 = (b 2 l 21 x 1 )/l 22 Chih-Jen
More information1. Search Directions In this chapter we again focus on the unconstrained optimization problem. lim sup ν
1 Search Directions In this chapter we again focus on the unconstrained optimization problem P min f(x), x R n where f : R n R is assumed to be twice continuously differentiable, and consider the selection
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationSpectrum-Revealing Matrix Factorizations Theory and Algorithms
Spectrum-Revealing Matrix Factorizations Theory and Algorithms Ming Gu Department of Mathematics University of California, Berkeley April 5, 2016 Joint work with D. Anderson, J. Deursch, C. Melgaard, J.
More informationIntroduction to Mathematical Programming
Introduction to Mathematical Programming Ming Zhong Lecture 6 September 12, 2018 Ming Zhong (JHU) AMS Fall 2018 1 / 20 Table of Contents 1 Ming Zhong (JHU) AMS Fall 2018 2 / 20 Solving Linear Systems A
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationSection 4.5 Eigenvalues of Symmetric Tridiagonal Matrices
Section 4.5 Eigenvalues of Symmetric Tridiagonal Matrices Key Terms Symmetric matrix Tridiagonal matrix Orthogonal matrix QR-factorization Rotation matrices (plane rotations) Eigenvalues We will now complete
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Algorithms Notes for 2016-10-31 There are several flavors of symmetric eigenvalue solvers for which there is no equivalent (stable) nonsymmetric solver. We discuss four algorithmic ideas: the workhorse
More informationLU Factorization. LU Decomposition. LU Decomposition. LU Decomposition: Motivation A = LU
LU Factorization To further improve the efficiency of solving linear systems Factorizations of matrix A : LU and QR LU Factorization Methods: Using basic Gaussian Elimination (GE) Factorization of Tridiagonal
More informationOrthonormal Transformations and Least Squares
Orthonormal Transformations and Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 30, 2009 Applications of Qx with Q T Q = I 1. solving
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationOrthogonal Transformations
Orthogonal Transformations Tom Lyche University of Oslo Norway Orthogonal Transformations p. 1/3 Applications of Qx with Q T Q = I 1. solving least squares problems (today) 2. solving linear equations
More informationGaussian Elimination without/with Pivoting and Cholesky Decomposition
Gaussian Elimination without/with Pivoting and Cholesky Decomposition Gaussian Elimination WITHOUT pivoting Notation: For a matrix A R n n we define for k {,,n} the leading principal submatrix a a k A
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationSolving Linear Systems Using Gaussian Elimination. How can we solve
Solving Linear Systems Using Gaussian Elimination How can we solve? 1 Gaussian elimination Consider the general augmented system: Gaussian elimination Step 1: Eliminate first column below the main diagonal.
More informationMATCOM and Selected MATCOM Functions
book-onlin page 567 Appendix C MATCOM and Selected MATCOM Functions C.1 MATCOM and Selected MATCOM Functions C.1.1 What is MATCOM? MATCOM is a MATLAB-based interactive software package containing the implementation
More information7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix
EE507 - Computational Techniques for EE 7. LU factorization Jitkomut Songsiri factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization
More informationLU Factorization. Marco Chiarandini. DM559 Linear and Integer Programming. Department of Mathematics & Computer Science University of Southern Denmark
DM559 Linear and Integer Programming LU Factorization Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark [Based on slides by Lieven Vandenberghe, UCLA] Outline
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 3: Positive-Definite Systems; Cholesky Factorization Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 11 Symmetric
More informationNumerical Analysis Fall. Gauss Elimination
Numerical Analysis 2015 Fall Gauss Elimination Solving systems m g g m m g x x x k k k k k k k k k 3 2 1 3 2 1 3 3 3 2 3 2 2 2 1 0 0 Graphical Method For small sets of simultaneous equations, graphing
More informationEIGENVALUE PROBLEMS. EIGENVALUE PROBLEMS p. 1/4
EIGENVALUE PROBLEMS EIGENVALUE PROBLEMS p. 1/4 EIGENVALUE PROBLEMS p. 2/4 Eigenvalues and eigenvectors Let A C n n. Suppose Ax = λx, x 0, then x is a (right) eigenvector of A, corresponding to the eigenvalue
More informationFundamentals of Numerical Linear Algebra
Fundamentals of Numerical Linear Algebra Seongjai Kim Department of Mathematics and Statistics Mississippi State University Mississippi State, MS 39762 USA Email: skim@math.msstate.edu Updated: November
More information632 CHAP. 11 EIGENVALUES AND EIGENVECTORS. QR Method
632 CHAP 11 EIGENVALUES AND EIGENVECTORS QR Method Suppose that A is a real symmetric matrix In the preceding section we saw how Householder s method is used to construct a similar tridiagonal matrix The
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationCS 323: Numerical Analysis and Computing
CS 323: Numerical Analysis and Computing MIDTERM #1 Instructions: This is an open notes exam, i.e., you are allowed to consult any textbook, your class notes, homeworks, or any of the handouts from us.
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationDirect Methods for Solving Linear Systems. Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le
Direct Methods for Solving Linear Systems Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le 1 Overview General Linear Systems Gaussian Elimination Triangular Systems The LU Factorization
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME MS&E 38 (CME 338 Large-Scale Numerical Optimization Course description Instructor: Michael Saunders Spring 28 Notes : Review The course teaches
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationFEM and sparse linear system solving
FEM & sparse linear system solving, Lecture 9, Nov 19, 2017 1/36 Lecture 9, Nov 17, 2017: Krylov space methods http://people.inf.ethz.ch/arbenz/fem17 Peter Arbenz Computer Science Department, ETH Zürich
More information1 Number Systems and Errors 1
Contents 1 Number Systems and Errors 1 1.1 Introduction................................ 1 1.2 Number Representation and Base of Numbers............. 1 1.2.1 Normalized Floating-point Representation...........
More informationBlockMatrixComputations and the Singular Value Decomposition. ATaleofTwoIdeas
BlockMatrixComputations and the Singular Value Decomposition ATaleofTwoIdeas Charles F. Van Loan Department of Computer Science Cornell University Supported in part by the NSF contract CCR-9901988. Block
More informationNumerical Linear Algebra
Chapter 3 Numerical Linear Algebra We review some techniques used to solve Ax = b where A is an n n matrix, and x and b are n 1 vectors (column vectors). We then review eigenvalues and eigenvectors and
More informationDense LU factorization and its error analysis
Dense LU factorization and its error analysis Laura Grigori INRIA and LJLL, UPMC February 2016 Plan Basis of floating point arithmetic and stability analysis Notation, results, proofs taken from [N.J.Higham,
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 13
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 208 LECTURE 3 need for pivoting we saw that under proper circumstances, we can write A LU where 0 0 0 u u 2 u n l 2 0 0 0 u 22 u 2n L l 3 l 32, U 0 0 0 l n l
More information1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )
Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical
More informationNumerical Methods I: Numerical linear algebra
1/3 Numerical Methods I: Numerical linear algebra Georg Stadler Courant Institute, NYU stadler@cimsnyuedu September 1, 017 /3 We study the solution of linear systems of the form Ax = b with A R n n, x,
More informationLinear System of Equations
Linear System of Equations Linear systems are perhaps the most widely applied numerical procedures when real-world situation are to be simulated. Example: computing the forces in a TRUSS. F F 5. 77F F.
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationMatrix Multiplication Chapter IV Special Linear Systems
Matrix Multiplication Chapter IV Special Linear Systems By Gokturk Poyrazoglu The State University of New York at Buffalo BEST Group Winter Lecture Series Outline 1. Diagonal Dominance and Symmetry a.
More informationACM106a - Homework 2 Solutions
ACM06a - Homework 2 Solutions prepared by Svitlana Vyetrenko October 7, 2006. Chapter 2, problem 2.2 (solution adapted from Golub, Van Loan, pp.52-54): For the proof we will use the fact that if A C m
More informationLecture 12 (Tue, Mar 5) Gaussian elimination and LU factorization (II)
Math 59 Lecture 2 (Tue Mar 5) Gaussian elimination and LU factorization (II) 2 Gaussian elimination - LU factorization For a general n n matrix A the Gaussian elimination produces an LU factorization if
More informationOrthogonalization and least squares methods
Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular
More informationApril 26, Applied mathematics PhD candidate, physics MA UC Berkeley. Lecture 4/26/2013. Jed Duersch. Spd matrices. Cholesky decomposition
Applied mathematics PhD candidate, physics MA UC Berkeley April 26, 2013 UCB 1/19 Symmetric positive-definite I Definition A symmetric matrix A R n n is positive definite iff x T Ax > 0 holds x 0 R n.
More informationLinear Systems of n equations for n unknowns
Linear Systems of n equations for n unknowns In many application problems we want to find n unknowns, and we have n linear equations Example: Find x,x,x such that the following three equations hold: x
More informationGaussian Elimination for Linear Systems
Gaussian Elimination for Linear Systems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 3, 2011 1/56 Outline 1 Elementary matrices 2 LR-factorization 3 Gaussian elimination
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Taylor s Theorem Can often approximate a function by a polynomial The error in the approximation
More informationDraft. Lecture 12 Gaussian Elimination and LU Factorization. MATH 562 Numerical Analysis II. Songting Luo
Lecture 12 Gaussian Elimination and LU Factorization Songting Luo Department of Mathematics Iowa State University MATH 562 Numerical Analysis II ongting Luo ( Department of Mathematics Iowa State University[0.5in]
More informationBindel, Spring 2016 Numerical Analysis (CS 4220) Notes for
Cholesky Notes for 2016-02-17 2016-02-19 So far, we have focused on the LU factorization for general nonsymmetric matrices. There is an alternate factorization for the case where A is symmetric positive
More informationC&O367: Nonlinear Optimization (Winter 2013) Assignment 4 H. Wolkowicz
C&O367: Nonlinear Optimization (Winter 013) Assignment 4 H. Wolkowicz Posted Mon, Feb. 8 Due: Thursday, Feb. 8 10:00AM (before class), 1 Matrices 1.1 Positive Definite Matrices 1. Let A S n, i.e., let
More informationImportant Matrix Factorizations
LU Factorization Choleski Factorization The QR Factorization LU Factorization: Gaussian Elimination Matrices Gaussian elimination transforms vectors of the form a α, b where a R k, 0 α R, and b R n k 1,
More informationANONSINGULAR tridiagonal linear system of the form
Generalized Diagonal Pivoting Methods for Tridiagonal Systems without Interchanges Jennifer B. Erway, Roummel F. Marcia, and Joseph A. Tyson Abstract It has been shown that a nonsingular symmetric tridiagonal
More informationScientific Computing with Case Studies SIAM Press, Lecture Notes for Unit VII Sparse Matrix
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 1: Direct Methods Dianne P. O Leary c 2008
More informationOrthonormal Transformations
Orthonormal Transformations Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 25, 2010 Applications of transformation Q : R m R m, with Q T Q = I 1.
More informationMath 471 (Numerical methods) Chapter 3 (second half). System of equations
Math 47 (Numerical methods) Chapter 3 (second half). System of equations Overlap 3.5 3.8 of Bradie 3.5 LU factorization w/o pivoting. Motivation: ( ) A I Gaussian Elimination (U L ) where U is upper triangular
More information13-2 Text: 28-30; AB: 1.3.3, 3.2.3, 3.4.2, 3.5, 3.6.2; GvL Eigen2
The QR algorithm The most common method for solving small (dense) eigenvalue problems. The basic algorithm: QR without shifts 1. Until Convergence Do: 2. Compute the QR factorization A = QR 3. Set A :=
More informationDirect Methods for Solving Linear Systems. Matrix Factorization
Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011
More informationLinear Algebraic Equations
Linear Algebraic Equations 1 Fundamentals Consider the set of linear algebraic equations n a ij x i b i represented by Ax b j with [A b ] [A b] and (1a) r(a) rank of A (1b) Then Axb has a solution iff
More informationIndex. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)
page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation
More informationComputational Methods. Least Squares Approximation/Optimization
Computational Methods Least Squares Approximation/Optimization Manfred Huber 2011 1 Least Squares Least squares methods are aimed at finding approximate solutions when no precise solution exists Find the
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Logistics Notes for 2016-09-14 1. There was a goof in HW 2, problem 1 (now fixed) please re-download if you have already started looking at it. 2. CS colloquium (4:15 in Gates G01) this Thurs is Margaret
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More information(Linear equations) Applied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB (Linear equations) Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots
More informationLU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b
AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationExample: Current in an Electrical Circuit. Solving Linear Systems:Direct Methods. Linear Systems of Equations. Solving Linear Systems: Direct Methods
Example: Current in an Electrical Circuit Solving Linear Systems:Direct Methods A number of engineering problems or models can be formulated in terms of systems of equations Examples: Electrical Circuit
More informationLecture # 11 The Power Method for Eigenvalues Part II. The power method find the largest (in magnitude) eigenvalue of. A R n n.
Lecture # 11 The Power Method for Eigenvalues Part II The power method find the largest (in magnitude) eigenvalue of It makes two assumptions. 1. A is diagonalizable. That is, A R n n. A = XΛX 1 for some
More informationLecture 2 Decompositions, perturbations
March 26, 2018 Lecture 2 Decompositions, perturbations A Triangular systems Exercise 2.1. Let L = (L ij ) be an n n lower triangular matrix (L ij = 0 if i > j). (a) Prove that L is non-singular if and
More informationLinear Algebra Linear Algebra : Matrix decompositions Monday, February 11th Math 365 Week #4
Linear Algebra Linear Algebra : Matrix decompositions Monday, February 11th Math Week # 1 Saturday, February 1, 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x
More informationMatrix Algorithms. Volume II: Eigensystems. G. W. Stewart H1HJ1L. University of Maryland College Park, Maryland
Matrix Algorithms Volume II: Eigensystems G. W. Stewart University of Maryland College Park, Maryland H1HJ1L Society for Industrial and Applied Mathematics Philadelphia CONTENTS Algorithms Preface xv xvii
More informationLINEAR SYSTEMS (11) Intensive Computation
LINEAR SYSTEMS () Intensive Computation 27-8 prof. Annalisa Massini Viviana Arrigoni EXACT METHODS:. GAUSSIAN ELIMINATION. 2. CHOLESKY DECOMPOSITION. ITERATIVE METHODS:. JACOBI. 2. GAUSS-SEIDEL 2 CHOLESKY
More informationNotes on Householder QR Factorization
Notes on Householder QR Factorization Robert A van de Geijn Department of Computer Science he University of exas at Austin Austin, X 7872 rvdg@csutexasedu September 2, 24 Motivation A fundamental problem
More informationMatrix Computations: Direct Methods II. May 5, 2014 Lecture 11
Matrix Computations: Direct Methods II May 5, 2014 ecture Summary You have seen an example of how a typical matrix operation (an important one) can be reduced to using lower level BS routines that would
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationIntroduction to Applied Linear Algebra with MATLAB
Sigam Series in Applied Mathematics Volume 7 Rizwan Butt Introduction to Applied Linear Algebra with MATLAB Heldermann Verlag Contents Number Systems and Errors 1 1.1 Introduction 1 1.2 Number Representation
More informationIndex. for generalized eigenvalue problem, butterfly form, 211
Index ad hoc shifts, 165 aggressive early deflation, 205 207 algebraic multiplicity, 35 algebraic Riccati equation, 100 Arnoldi process, 372 block, 418 Hamiltonian skew symmetric, 420 implicitly restarted,
More informationLinear Algebra Section 2.6 : LU Decomposition Section 2.7 : Permutations and transposes Wednesday, February 13th Math 301 Week #4
Linear Algebra Section. : LU Decomposition Section. : Permutations and transposes Wednesday, February 1th Math 01 Week # 1 The LU Decomposition We learned last time that we can factor a invertible matrix
More information4.8 Arnoldi Iteration, Krylov Subspaces and GMRES
48 Arnoldi Iteration, Krylov Subspaces and GMRES We start with the problem of using a similarity transformation to convert an n n matrix A to upper Hessenberg form H, ie, A = QHQ, (30) with an appropriate
More informationMath 411 Preliminaries
Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector
More informationSolving large scale eigenvalue problems
arge scale eigenvalue problems, Lecture 4, March 14, 2018 1/41 Lecture 4, March 14, 2018: The QR algorithm http://people.inf.ethz.ch/arbenz/ewp/ Peter Arbenz Computer Science Department, ETH Zürich E-mail:
More informationPowerPoints organized by Dr. Michael R. Gustafson II, Duke University
Part 3 Chapter 10 LU Factorization PowerPoints organized by Dr. Michael R. Gustafson II, Duke University All images copyright The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
More information