lecture 2 and 3: algorithms for linear algebra
|
|
- Ezra Gibson
- 5 years ago
- Views:
Transcription
1 lecture 2 and 3: algorithms for linear algebra STAT 545: Introduction to computational statistics Vinayak Rao Department of Statistics, Purdue University August 27, 2018
2 Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations 1/23
3 Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations Consider AX = b, where A is N N, and X and b are N k. Solve for X: X = A 1 b 1/23
4 Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations Consider AX = b, where A is N N, and X and b are N k. Solve for X: X = A 1 b Calculate the inverse of A and multiply? No! 1/23
5 Solving a system of linear equations Preliminaries: cost of standard matrix algebra operations Consider AX = b, where A is N N, and X and b are N k. Solve for X: X = A 1 b Calculate the inverse of A and multiply? No! Directly solving for X is faster, and more stable numerically A 1 need not even exist 1/23
6 Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] 2/23
7 Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v /23
8 Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Manipulate to get: I [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] 2/23
9 Gauss-Jordan elimination A X = b A [ X, A 1 ] = [ b, I ] x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Manipulate to get: I [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] At step i: Make element a ii = 1 (by scaling or pivoting) Set other elements in column i to 0 by multiplying and subtracting that row 2/23
10 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v /23
11 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Multiply row 1 by 2 and subtract 3/23
12 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v /23
13 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Subtract row 1 3/23
14 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Pivot 3/23
15 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v Continue till we get an identity matrix 3/23
16 Gauss-Jordan elimination x 1 v 11 v 12 v x 2 v 21 v 22 v 23 = x 3 v 31 v 32 v What is the cost of this algorithm? 3/23
17 Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] 4/23
18 Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] O(N 3 ) manipulation to get: U [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] Here, U is an upper-triangular matrix. 4/23
19 Gauss elimination with back-substitution A [ X, A 1 ] = [ b, I ] O(N 3 ) manipulation to get: U [ X, A 1 ] = [ ĉ 1, Ĉ 2 ] Here, U is an upper-triangular matrix. Cannot just read off solution. Need to backsolve. 4/23
20 LU decomposition What are we actually doing? A = LU Here L and U are lower and upper triangular matrices u 11 u 12 u 13 L = l , U = 0 u 22 u 23 l 31 l u 33 5/23
21 LU decomposition What are we actually doing? A = LU Here L and U are lower and upper triangular matrices. Is this always possible? u 11 u 12 u 13 L = l , U = 0 u 22 u 23 l 31 l u 33 [ ] 0 1 A = 1 0 5/23
22 LU decomposition What are we actually doing? A = LU Here L and U are lower and upper triangular matrices. Is this always possible? u 11 u 12 u 13 L = l , U = 0 u 22 u 23 l 31 l u 33 [ ] 0 1 A = 1 0 PA = LU, P is a permutation matrix Crout s algorithm, O(N 3 ), stable, L, U can be computed in place. 5/23
23 Backsubstitution AX = b 6/23
24 Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb 6/23
25 Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb l l 31 l 32 1 y 1 y 2 y 3 ˆb 1 = ˆb 2 ˆb 3 6/23
26 Backsubstitution AX = b LUX = Pb First solve Y by forward substitution LY = Pb l l 31 l 32 1 y 1 y 2 y 3 ˆb 1 = ˆb 2 ˆb 3 Then solve X by back substitution UX = Y 6/23
27 Comments LU-decomposition can be reused for different b s. Calculating LU decomposition: O(N 3 ). Given LU decomposition, solving for X: O(N 2 ). A = P 1 LU = ( 1) S N i=1 u ii (S: num. of exchanges) LUA 1 = PI, can solve for A 1. (back to Gauss-Jordan) 7/23
28 Cholesky decomposition If A is symmetric positive-definite: A = LL T (but now L need not have a diagonal of ones) Square-root of A More stable. Twice as efficient. Related: A = LDL T (but now L has a unit diagonal). 8/23
29 Eigenvalue decomposition An N N matrix A: a map from R N R N. An eigenvector v undergoes no rotation: Av = λv λ is the corresponding eigenvalue, and gives the rescaling. 9/23
30 Eigenvalue decomposition An N N matrix A: a map from R N R N. An eigenvector v undergoes no rotation: Av = λv λ is the corresponding eigenvalue, and gives the rescaling. Let Λ = diag(λ 1,, λ N ) with λ 1 λ 2,..., λ N, and V = [v 1,, v N ] be the matrix of corresponding eigenvectors AV = VΛ 9/23
31 Eigenvalue decomposition An N N matrix A: a map from R N R N. An eigenvector v undergoes no rotation: Av = λv λ is the corresponding eigenvalue, and gives the rescaling. Let Λ = diag(λ 1,, λ N ) with λ 1 λ 2,..., λ N, and V = [v 1,, v N ] be the matrix of corresponding eigenvectors Real Symmetric matrices have real eigenvalues AV = VΛ different eigenvalues have orthogonal eigenvectors 9/23
32 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? 10/23
33 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i 10/23
34 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) 10/23
35 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) This is the power method (Google s PageRank). 10/23
36 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) This is the power method (Google s PageRank). How would we calculate λ 1? 10/23
37 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) This is the power method (Google s PageRank). How would we calculate λ 1? What if we wanted the second eigenvector? 10/23
38 Calculating eigenvalues and -vectors Let A be any real symmetric matrix. How to calculate the (absolute) largest eigenvalue and vector? Start with a random vector u 0 Define u 1 = Au 0, and normalize length. Repeat: u i = Au i 1, u i = u i / u i u i v 1 (Why?) This is the power method (Google s PageRank). How would we calculate λ 1? What if we wanted the second eigenvector? Adjust A so eigenvalue corresponding to v 1 equals 0 10/23
39 Gauss-Jordan elimination x 1 x 2 x 3 x 4 32 = What is the solution? How about for b = [32.1, 22.9, 33.1, 30.9] T? 11/23
40 Gauss-Jordan elimination x 1 x 2 x 3 x 4 32 = What is the solution? How about for b = [32.1, 22.9, 33.1, 30.9] T? Why the difference? the determinant? the inverse? the condition number? An ill-conditioned problem can strongly amplify errors. 11/23
41 Gauss-Jordan elimination x 1 x 2 x 3 x 4 32 = What is the solution? How about for b = [32.1, 22.9, 33.1, 30.9] T? Why the difference? the determinant? the inverse? the condition number? An ill-conditioned problem can strongly amplify errors. Even without any rounding error 11/23
42 Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v 12/23
43 Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v For a symmetric, real matrix, A = λ max (A) (why?) For a general, real matrix, A = λ max (A T A) (why?) 12/23
44 Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v For a symmetric, real matrix, A = λ max (A) (why?) For a general, real matrix, A = λ max (A T A) (why?) For A in R N N and any v R N, Av A v (why?) 12/23
45 Stability analysis The operator norm of a matrix A is Av A 2 = A = max Av = max v =1 v v For a symmetric, real matrix, A = λ max (A) (why?) For a general, real matrix, A = λ max (A T A) (why?) For A in R N N and any v R N, Av A v (why?) If A = BC, and all matrices are in R N N, A B C (why?) 12/23
46 Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b 13/23
47 Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b From b = Ax and δx = A 1 δb, we have b A x, and δx A 1 δb : 13/23
48 Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b From b = Ax and δx = A 1 δb, we have b A x, and δx A 1 δb : δx x A A 1 δb b 13/23
49 Stability analysis For a perturbation, δb let δx be the change in solution to Ax = b δx x A(x + δx) = b + δb δb is the relative change in the solution from the change b From b = Ax and δx = A 1 δb, we have b A x, and δx A 1 δb : δx x A A 1 δb b Condition number of a matrix A is given by κ(a) = A A 1 Large condition number implies unstable solution 13/23
50 Stability analysis κ(a) 1 (why?) 14/23
51 Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) 14/23
52 Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) For a real matrix, κ(a) = λmax (A T A) λ min (A T A) (why?) 14/23
53 Stability analysis κ(a) 1 (why?) For a real symmetric matrix, κ(a) = λ max(a) λ min (A) (why?) For a real matrix, κ(a) = λmax (A T A) λ min (A T A) (why?) Condition number is a property of a problem Stability is a property of an algorithm A bad algorithm can mess up a simple problem 14/23
54 Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 15/23
55 Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 [ ] 1e /23
56 Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 [ ] 1e Partial pivoting: Pivot rows to bring max r v r1 to top Can dramatically improve performance. E.g. 15/23
57 Gaussian elimination with partial pivoting Consider reducing to upper triangular v 11 v 12 v 13 v 21 v 22 v 23 v 31 v 32 v 33 Gaussian elimination: divide row 1 by v 11 [ ] 1e Partial pivoting: Pivot rows to bring max r v r1 to top Can dramatically improve performance. E.g. Why does it work? 15/23
58 Gaussian elimination with partial pivoting Recall Gaussian elimination decomposes A = LU and solves two intermediate problems. What are the condition numbers of L and U? Try [ ] 1e Note: does pivoting for you automatically! (see the function in package ) 16/23
59 QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) 17/23
60 QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I 17/23
61 QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I κ(a) = κ(q)κ(r) (why?) 17/23
62 QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I κ(a) = κ(q)κ(r) (why?) Can use to solve Ax = b (How?) Most stable decomposition 17/23
63 QR decomposition In general, for A = BC, κ(a) κ(b)κ(c) (why?) QR decomposition: A = QR Here, R is an upper (right) triangular matrix. Q is an orthonormal matrix: Q T Q = I κ(a) = κ(q)κ(r) (why?) Can use to solve Ax = b (How?) Most stable decomposition Does this mean we should use QR decomposition? 17/23
64 Gram-Schmidt orthonormalization Given N vectors x 1,..., x N construct an orthonormal basis: 18/23
65 Gram-Schmidt orthonormalization Given N vectors x 1,..., x N construct an orthonormal basis: u 1 = x 1 / x 1 ũ i = x i i 1 j=1 (xt i u j)u i, u i = ũ i / ũ i i = 2..., N 18/23
66 QR decomposition A = Q R a 11 a 12 a 13 q 11 q 12 q 13 r 11 r 12 r 13 a 21 a 22 a 23 = q 21 q 22 q 23 0 r 22 r 23 a 31 a 32 a 33 q 31 q 32 q r 33 19/23
67 QR decomposition A = Q R a 11 a 12 a 13 q 11 q 12 q 13 r 11 r 12 r 13 a 21 a 22 a 23 = q 21 q 22 q 23 0 r 22 r 23 a 31 a 32 a 33 q 31 q 32 q r 33 QR decomposition: Gram-Schmidt on columns of A (can you see why?) 19/23
68 QR decomposition A = Q R a 11 a 12 a 13 q 11 q 12 q 13 r 11 r 12 r 13 a 21 a 22 a 23 = q 21 q 22 q 23 0 r 22 r 23 a 31 a 32 a 33 q 31 q 32 q r 33 QR decomposition: Gram-Schmidt on columns of A (can you see why?) Of course, there are more stable/efficient ways of doing this (Householder rotation/givens rotation) O(N 3 ) algorithms (though about twice as slow as LU) 19/23
69 QR algorithm Algorithm to calculate all eigenvalues/eigenvectors of a (not too-large) matrix Start with A 0 = A. At iteration i: A i = Q i R i A i+1 = R i Q i Then A i and A i+1 have the same eigenvalues (why?), and the diagonal contains the eigenvalues. 20/23
70 QR algorithm Algorithm to calculate all eigenvalues/eigenvectors of a (not too-large) matrix Start with A 0 = A. At iteration i: A i = Q i R i A i+1 = R i Q i Then A i and A i+1 have the same eigenvalues (why?), and the diagonal contains the eigenvalues. Can be made more stable/efficient. One of Dongarra & Sullivan (2000) s list of top 10 algoirithms. See also number 4, decompositional approach to matrix computations 20/23
71 Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 21/23
72 Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) 21/23
73 Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) log(p(x µ, Σ)) = 1 2 YT Y N 2 log 2π log Σ 21/23
74 Multivariate normal log(p(x µ, Σ)) = 1 2 (X µ)t Σ 1 (X µ) N 2 log 2π 1 log Σ 2 Σ = LL T Y = L 1 (X µ) (Forward solve) log(p(x µ, Σ)) = 1 2 YT Y N 2 log 2π log Σ Can also just forward solve for L 1 : LL 1 = I (Inverted triangular matrix isn t too bad) 21/23
75 Sampling a multivariate normal Sampling a univariate normal: Inversion method (default for?). Box-Muller transform: (Z 1, Z 2 ) : independent standard normals. 22/23
76 Sampling a multivariate normal Sampling a univariate normal: Inversion method (default for?). Box-Muller transform: (Z 1, Z 2 ) : independent standard normals. Let Z N (0, I) X = µ + LZ Z = N (µ, L T L) 22/23
77 x y [ ] 1 0 Σ = /23
78 x y [ ] 4 0 Σ = /23
79 x y [ ] 4 2 Σ = /23
80 x y [ ] Σ = /23
lecture 3 and 4: algorithms for linear algebra
lecture 3 and 4: algorithms for linear algebra STAT 545: Introduction to computational statistics Vinayak Rao Department of Statistics, Purdue University August 30, 2016 Solving a system of linear equations
More informationLinear Algebra Section 2.6 : LU Decomposition Section 2.7 : Permutations and transposes Wednesday, February 13th Math 301 Week #4
Linear Algebra Section. : LU Decomposition Section. : Permutations and transposes Wednesday, February 1th Math 01 Week # 1 The LU Decomposition We learned last time that we can factor a invertible matrix
More informationSolution of Linear Equations
Solution of Linear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 7, 07 We have discussed general methods for solving arbitrary equations, and looked at the special class of polynomial equations A subclass
More informationThis can be accomplished by left matrix multiplication as follows: I
1 Numerical Linear Algebra 11 The LU Factorization Recall from linear algebra that Gaussian elimination is a method for solving linear systems of the form Ax = b, where A R m n and bran(a) In this method
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationLINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12,
LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, 2000 74 6 Summary Here we summarize the most important information about theoretical and numerical linear algebra. MORALS OF THE STORY: I. Theoretically
More informationLinear Algebraic Equations
Linear Algebraic Equations 1 Fundamentals Consider the set of linear algebraic equations n a ij x i b i represented by Ax b j with [A b ] [A b] and (1a) r(a) rank of A (1b) Then Axb has a solution iff
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationScientific Computing: Dense Linear Systems
Scientific Computing: Dense Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 February 9th, 2012 A. Donev (Courant Institute)
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationCheat Sheet for MATH461
Cheat Sheet for MATH46 Here is the stuff you really need to remember for the exams Linear systems Ax = b Problem: We consider a linear system of m equations for n unknowns x,,x n : For a given matrix A
More informationToday s class. Linear Algebraic Equations LU Decomposition. Numerical Methods, Fall 2011 Lecture 8. Prof. Jinbo Bi CSE, UConn
Today s class Linear Algebraic Equations LU Decomposition 1 Linear Algebraic Equations Gaussian Elimination works well for solving linear systems of the form: AX = B What if you have to solve the linear
More information3 QR factorization revisited
LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 2, 2000 30 3 QR factorization revisited Now we can explain why A = QR factorization is much better when using it to solve Ax = b than the A = LU factorization
More informationLinear Analysis Lecture 16
Linear Analysis Lecture 16 The QR Factorization Recall the Gram-Schmidt orthogonalization process. Let V be an inner product space, and suppose a 1,..., a n V are linearly independent. Define q 1,...,
More informationLU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b
AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More informationLinear Equations and Matrix
1/60 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Gaussian Elimination 2/60 Alpha Go Linear algebra begins with a system of linear
More informationLinear Algebra Linear Algebra : Matrix decompositions Monday, February 11th Math 365 Week #4
Linear Algebra Linear Algebra : Matrix decompositions Monday, February 11th Math Week # 1 Saturday, February 1, 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x
More informationIntroduction to Mathematical Programming
Introduction to Mathematical Programming Ming Zhong Lecture 6 September 12, 2018 Ming Zhong (JHU) AMS Fall 2018 1 / 20 Table of Contents 1 Ming Zhong (JHU) AMS Fall 2018 2 / 20 Solving Linear Systems A
More information1 GSW Sets of Systems
1 Often, we have to solve a whole series of sets of simultaneous equations of the form y Ax, all of which have the same matrix A, but each of which has a different known vector y, and a different unknown
More informationCOURSE Numerical methods for solving linear systems. Practical solving of many problems eventually leads to solving linear systems.
COURSE 9 4 Numerical methods for solving linear systems Practical solving of many problems eventually leads to solving linear systems Classification of the methods: - direct methods - with low number of
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More informationNumerical Linear Algebra
Chapter 3 Numerical Linear Algebra We review some techniques used to solve Ax = b where A is an n n matrix, and x and b are n 1 vectors (column vectors). We then review eigenvalues and eigenvectors and
More informationY = ax + b. Numerical Applications Least-squares. Start with Self-test 10-1/459. Linear equation. Error function: E = D 2 = (Y - (ax+b)) 2
Ch.10 Numerical Applications 10-1 Least-squares Start with Self-test 10-1/459. Linear equation Y = ax + b Error function: E = D 2 = (Y - (ax+b)) 2 Regression Formula: Slope a = (N ΣXY - (ΣX)(ΣY)) / (N
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 12: Gaussian Elimination and LU Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Gaussian Elimination
More informationforms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms
Christopher Engström November 14, 2014 Hermitian LU QR echelon Contents of todays lecture Some interesting / useful / important of matrices Hermitian LU QR echelon Rewriting a as a product of several matrices.
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationImportant Matrix Factorizations
LU Factorization Choleski Factorization The QR Factorization LU Factorization: Gaussian Elimination Matrices Gaussian elimination transforms vectors of the form a α, b where a R k, 0 α R, and b R n k 1,
More informationComputational Methods. Systems of Linear Equations
Computational Methods Systems of Linear Equations Manfred Huber 2010 1 Systems of Equations Often a system model contains multiple variables (parameters) and contains multiple equations Multiple equations
More informationDirect Methods for Solving Linear Systems. Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le
Direct Methods for Solving Linear Systems Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le 1 Overview General Linear Systems Gaussian Elimination Triangular Systems The LU Factorization
More informationLinear Algebra. Solving Linear Systems. Copyright 2005, W.R. Winfrey
Copyright 2005, W.R. Winfrey Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization Topics Preliminaries Echelon Form of a Matrix Elementary
More informationScientific Computing: Solving Linear Systems
Scientific Computing: Solving Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 September 17th and 24th, 2015 A. Donev (Courant
More informationSolving Ax = b w/ different b s: LU-Factorization
Solving Ax = b w/ different b s: LU-Factorization Linear Algebra Josh Engwer TTU 14 September 2015 Josh Engwer (TTU) Solving Ax = b w/ different b s: LU-Factorization 14 September 2015 1 / 21 Elementary
More informationOrthogonal Transformations
Orthogonal Transformations Tom Lyche University of Oslo Norway Orthogonal Transformations p. 1/3 Applications of Qx with Q T Q = I 1. solving least squares problems (today) 2. solving linear equations
More informationLecture 9. Errors in solving Linear Systems. J. Chaudhry (Zeb) Department of Mathematics and Statistics University of New Mexico
Lecture 9 Errors in solving Linear Systems J. Chaudhry (Zeb) Department of Mathematics and Statistics University of New Mexico J. Chaudhry (Zeb) (UNM) Math/CS 375 1 / 23 What we ll do: Norms and condition
More informationLU Factorization. Marco Chiarandini. DM559 Linear and Integer Programming. Department of Mathematics & Computer Science University of Southern Denmark
DM559 Linear and Integer Programming LU Factorization Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark [Based on slides by Lieven Vandenberghe, UCLA] Outline
More information1.5 Gaussian Elimination With Partial Pivoting.
Gaussian Elimination With Partial Pivoting In the previous section we discussed Gaussian elimination In that discussion we used equation to eliminate x from equations through n Then we used equation to
More informationElementary Linear Algebra
Matrices J MUSCAT Elementary Linear Algebra Matrices Definition Dr J Muscat 2002 A matrix is a rectangular array of numbers, arranged in rows and columns a a 2 a 3 a n a 2 a 22 a 23 a 2n A = a m a mn We
More informationMatrix Algebra for Engineers Jeffrey R. Chasnov
Matrix Algebra for Engineers Jeffrey R. Chasnov The Hong Kong University of Science and Technology The Hong Kong University of Science and Technology Department of Mathematics Clear Water Bay, Kowloon
More informationLinear Algebra Review
Linear Algebra Review CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Linear Algebra Review 1 / 16 Midterm Exam Tuesday Feb
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 13
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 208 LECTURE 3 need for pivoting we saw that under proper circumstances, we can write A LU where 0 0 0 u u 2 u n l 2 0 0 0 u 22 u 2n L l 3 l 32, U 0 0 0 l n l
More informationMAA507, Power method, QR-method and sparse matrix representation.
,, and representation. February 11, 2014 Lecture 7: Overview, Today we will look at:.. If time: A look at representation and fill in. Why do we need numerical s? I think everyone have seen how time consuming
More informationMatrix notation. A nm : n m : size of the matrix. m : no of columns, n: no of rows. Row matrix n=1 [b 1, b 2, b 3,. b m ] Column matrix m=1
Matrix notation A nm : n m : size of the matrix m : no of columns, n: no of rows Row matrix n=1 [b 1, b 2, b 3,. b m ] Column matrix m=1 n = m square matrix Symmetric matrix Upper triangular matrix: matrix
More informationMIDTERM. b) [2 points] Compute the LU Decomposition A = LU or explain why one does not exist.
MAE 9A / FALL 3 Maurício de Oliveira MIDTERM Instructions: You have 75 minutes This exam is open notes, books No computers, calculators, phones, etc There are 3 questions for a total of 45 points and bonus
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 1. qr and complete orthogonal factorization poor man s svd can solve many problems on the svd list using either of these factorizations but they
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More information1.Chapter Objectives
LU Factorization INDEX 1.Chapter objectives 2.Overview of LU factorization 2.1GAUSS ELIMINATION AS LU FACTORIZATION 2.2LU Factorization with Pivoting 2.3 MATLAB Function: lu 3. CHOLESKY FACTORIZATION 3.1
More information1 Last time: least-squares problems
MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that
More informationL2-7 Some very stylish matrix decompositions for solving Ax = b 10 Oct 2015
L-7 Some very stylish matrix decompositions for solving Ax = b 10 Oct 015 Marty McFly: Wait a minute, Doc. Ah... Are you telling me you built a time machine... out of a DeLorean? Doc Brown: The way I see
More informationAMS 147 Computational Methods and Applications Lecture 17 Copyright by Hongyun Wang, UCSC
Lecture 17 Copyright by Hongyun Wang, UCSC Recap: Solving linear system A x = b Suppose we are given the decomposition, A = L U. We solve (LU) x = b in 2 steps: *) Solve L y = b using the forward substitution
More informationMODEL ANSWERS TO THE THIRD HOMEWORK
MODEL ANSWERS TO THE THIRD HOMEWORK 1 (i) We apply Gaussian elimination to A First note that the second row is a multiple of the first row So we need to swap the second and third rows 1 3 2 1 2 6 5 7 3
More informationMain matrix factorizations
Main matrix factorizations A P L U P permutation matrix, L lower triangular, U upper triangular Key use: Solve square linear system Ax b. A Q R Q unitary, R upper triangular Key use: Solve square or overdetrmined
More informationReview of matrices. Let m, n IN. A rectangle of numbers written like A =
Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an
More informationExample Linear Algebra Competency Test
Example Linear Algebra Competency Test The 4 questions below are a combination of True or False, multiple choice, fill in the blank, and computations involving matrices and vectors. In the latter case,
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationA Review of Linear Algebra
A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose
More informationNext topics: Solving systems of linear equations
Next topics: Solving systems of linear equations 1 Gaussian elimination (today) 2 Gaussian elimination with partial pivoting (Week 9) 3 The method of LU-decomposition (Week 10) 4 Iterative techniques:
More informationChapter 2. Solving Systems of Equations. 2.1 Gaussian elimination
Chapter 2 Solving Systems of Equations A large number of real life applications which are resolved through mathematical modeling will end up taking the form of the following very simple looking matrix
More informationLecture 11. Linear systems: Cholesky method. Eigensystems: Terminology. Jacobi transformations QR transformation
Lecture Cholesky method QR decomposition Terminology Linear systems: Eigensystems: Jacobi transformations QR transformation Cholesky method: For a symmetric positive definite matrix, one can do an LU decomposition
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More informationMAT 343 Laboratory 3 The LU factorization
In this laboratory session we will learn how to MAT 343 Laboratory 3 The LU factorization 1. Find the LU factorization of a matrix using elementary matrices 2. Use the MATLAB command lu to find the LU
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More information30.3. LU Decomposition. Introduction. Prerequisites. Learning Outcomes
LU Decomposition 30.3 Introduction In this Section we consider another direct method for obtaining the solution of systems of equations in the form AX B. Prerequisites Before starting this Section you
More informationOutline. Math Numerical Analysis. Errors. Lecture Notes Linear Algebra: Part B. Joseph M. Mahaffy,
Math 54 - Numerical Analysis Lecture Notes Linear Algebra: Part B Outline Joseph M. Mahaffy, jmahaffy@mail.sdsu.edu Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationMATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
1 MATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 2 Linear Systems and solutions Systems of linear
More informationMath Computation Test 1 September 26 th, 2016 Debate: Computation vs. Theory Whatever wins, it ll be Huuuge!
Math 5- Computation Test September 6 th, 6 Debate: Computation vs. Theory Whatever wins, it ll be Huuuge! Name: Answer Key: Making Math Great Again Be sure to show your work!. (8 points) Consider the following
More informationMATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors.
MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v
More information(Mathematical Operations with Arrays) Applied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB (Mathematical Operations with Arrays) Contents Getting Started Matrices Creating Arrays Linear equations Mathematical Operations with Arrays Using Script
More informationMIT Final Exam Solutions, Spring 2017
MIT 8.6 Final Exam Solutions, Spring 7 Problem : For some real matrix A, the following vectors form a basis for its column space and null space: C(A) = span,, N(A) = span,,. (a) What is the size m n of
More informationEngineering Computation
Engineering Computation Systems of Linear Equations_1 1 Learning Objectives for Lecture 1. Motivate Study of Systems of Equations and particularly Systems of Linear Equations. Review steps of Gaussian
More informationNumerical methods, midterm test I (2018/19 autumn, group A) Solutions
Numerical methods, midterm test I (2018/19 autumn, group A Solutions x Problem 1 (6p We are going to approximate the limit 3/2 x lim x 1 x 1 by substituting x = 099 into the fraction in the present form
More informationCS227-Scientific Computing. Lecture 4: A Crash Course in Linear Algebra
CS227-Scientific Computing Lecture 4: A Crash Course in Linear Algebra Linear Transformation of Variables A common phenomenon: Two sets of quantities linearly related: y = 3x + x 2 4x 3 y 2 = 2.7x 2 x
More informationLinear algebra for MATH2601 Numerical methods
Linear algebra for MATH2601 Numerical methods László Erdős August 12, 2000 Contents 1 Introduction 3 1.1 Typesoferrors... 4 1.1.1 Rounding errors... 5 1.1.2 Truncationerrors... 6 1.1.3 Conditioningerrors...
More informationSystems of Linear Equations
Systems of Linear Equations Last time, we found that solving equations such as Poisson s equation or Laplace s equation on a grid is equivalent to solving a system of linear equations. There are many other
More informationThere are six more problems on the next two pages
Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with
More informationSUMMARY OF MATH 1600
SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You
More informationMATLAB Project: LU Factorization
Name Purpose: To practice Lay's LU Factorization Algorithm and see how it is related to MATLAB's lu function. Prerequisite: Section 2.5 MATLAB functions used: *, lu; and ludat and gauss from Laydata4 Toolbox
More informationGAUSSIAN ELIMINATION AND LU DECOMPOSITION (SUPPLEMENT FOR MA511)
GAUSSIAN ELIMINATION AND LU DECOMPOSITION (SUPPLEMENT FOR MA511) D. ARAPURA Gaussian elimination is the go to method for all basic linear classes including this one. We go summarize the main ideas. 1.
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationTotal 170. Name. Final Examination M340L-CS
1 10 2 10 3 15 4 5 5 10 6 10 7 20 8 10 9 20 10 25 11 10 12 10 13 15 Total 170 Final Examination Name M340L-CS 1. Use extra paper to determine your solutions then neatly transcribe them onto these sheets.
More information1 Number Systems and Errors 1
Contents 1 Number Systems and Errors 1 1.1 Introduction................................ 1 1.2 Number Representation and Base of Numbers............. 1 1.2.1 Normalized Floating-point Representation...........
More informationLecture 4: Applications of Orthogonality: QR Decompositions
Math 08B Professor: Padraic Bartlett Lecture 4: Applications of Orthogonality: QR Decompositions Week 4 UCSB 204 In our last class, we described the following method for creating orthonormal bases, known
More informationLINEAR SYSTEMS (11) Intensive Computation
LINEAR SYSTEMS () Intensive Computation 27-8 prof. Annalisa Massini Viviana Arrigoni EXACT METHODS:. GAUSSIAN ELIMINATION. 2. CHOLESKY DECOMPOSITION. ITERATIVE METHODS:. JACOBI. 2. GAUSS-SEIDEL 2 CHOLESKY
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More informationLINEAR ALGEBRA 1, 2012-I PARTIAL EXAM 3 SOLUTIONS TO PRACTICE PROBLEMS
LINEAR ALGEBRA, -I PARTIAL EXAM SOLUTIONS TO PRACTICE PROBLEMS Problem (a) For each of the two matrices below, (i) determine whether it is diagonalizable, (ii) determine whether it is orthogonally diagonalizable,
More informationEXAM. Exam 1. Math 5316, Fall December 2, 2012
EXAM Exam Math 536, Fall 22 December 2, 22 Write all of your answers on separate sheets of paper. You can keep the exam questions. This is a takehome exam, to be worked individually. You can use your notes.
More information22A-2 SUMMER 2014 LECTURE 5
A- SUMMER 0 LECTURE 5 NATHANIEL GALLUP Agenda Elimination to the identity matrix Inverse matrices LU factorization Elimination to the identity matrix Previously, we have used elimination to get a system
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationSection 6.4. The Gram Schmidt Process
Section 6.4 The Gram Schmidt Process Motivation The procedures in 6 start with an orthogonal basis {u, u,..., u m}. Find the B-coordinates of a vector x using dot products: x = m i= x u i u i u i u i Find
More informationRoundoff Analysis of Gaussian Elimination
Jim Lambers MAT 60 Summer Session 2009-0 Lecture 5 Notes These notes correspond to Sections 33 and 34 in the text Roundoff Analysis of Gaussian Elimination In this section, we will perform a detailed error
More informationSince the determinant of a diagonal matrix is the product of its diagonal elements it is trivial to see that det(a) = α 2. = max. A 1 x.
APPM 4720/5720 Problem Set 2 Solutions This assignment is due at the start of class on Wednesday, February 9th. Minimal credit will be given for incomplete solutions or solutions that do not provide details
More informationy b where U. matrix inverse A 1 ( L. 1 U 1. L 1 U 13 U 23 U 33 U 13 2 U 12 1
LU decomposition -- manual demonstration Instructor: Nam Sun Wang lu-manualmcd LU decomposition, where L is a lower-triangular matrix with as the diagonal elements and U is an upper-triangular matrix Just
More information