BlockMatrixComputations and the Singular Value Decomposition. ATaleofTwoIdeas
|
|
- Eunice Leonard
- 5 years ago
- Views:
Transcription
1 BlockMatrixComputations and the Singular Value Decomposition ATaleofTwoIdeas Charles F. Van Loan Department of Computer Science Cornell University Supported in part by the NSF contract CCR
2 Block Matrices A block matrix is a matrix with matrix entries, e.g., A = = A 11 A 12 A 21 A 22 A ij IR 3 2 Operations are pretty much business as usual, e.g. A 11 A 12 A 21 A 22 T A 11 A 12 A 21 A 22 = A T 11 A T 21 A T 12 AT 22 A 11 A 12 A 21 A 22 = A T 11A 11 + A T 21A 21 etc etc etc
3 E.g., Strassen Multiplication C 11 C 12 C 21 C 22 = A 11 A 12 A 21 A 22 B 11 B 12 B 21 B 22 P 1 =(A 11 + A 22 )(B 11 + B 22 ) P 2 =(A 21 + A 22 )B 11 P 3 = A 11 (B 12 B 22 ) P 4 = A 22 (B 21 B 11 ) P 5 =(A 11 + A 12 )B 22 P 6 =(A 21 A 11 )(B 11 + B 12 ) P 7 =(A 12 A 22 )(B 21 + B 22 ) C 11 = P 1 + P 4 P 5 + P 7 C 12 = P 3 + P 5 C 21 = P 2 + P 4 C 22 = P 1 + P 3 P 2 + P 6
4 Singular Value Decomposition (SVD) If A IR m n then there exist orthogonal U IR m m and V IR n n so U T AV = Σ = diag(σ 1,...,σ n ) = where σ 1 σ 2 σ n are the singular values. σ σ 2 0 0
5 Singular Value Decomposition (SVD) If A IR m n then there exist orthogonal U IR m m and V IR n n so U T AV = Σ = diag(σ 1,...,σ n ) = where σ 1 σ 2 σ n are the singular values. σ σ Fact 1. The columns of V = v 1 v n and U = u 1 u n are the right and left singular vectors and they are related: Av j = σ j u j j =1:n A T u j = σ j v j
6 Singular Value Decomposition (SVD) If A IR m n then there exist orthogonal U IR m m and V IR n n so U T AV = Σ = diag(σ 1,...,σ n ) = where σ 1 σ 2 σ n are the singular values. σ σ Fact 2. The SVD of A is related to the eigen-decompositions of A T A and AA T : V T (A T A)V =diag(σ 2 1,...,σ 2 n) U T (AA T )U =diag(σ1,...,σ 2 n, 2 0,...,0) m n
7 Singular Value Decomposition (SVD) If A IR m n then there exist orthogonal U IR m m and V IR n n so U T AV = Σ = diag(σ 1,...,σ n ) = where σ 1 σ 2 σ n are the singular values. σ σ Fact 3. The smallest singular value is the distance from A to the set of rank deficient matrices: σ min = min rank(b) <n A B F
8 Singular Value Decomposition (SVD) If A IR m n then there exist orthogonal U IR m m and V IR n n so U T AV = Σ = diag(σ 1,...,σ n ) = where σ 1 σ 2 σ n are the singular values. σ σ Fact 4. The matrix σ 1 u 1 v T 1 is the closest rank-1 matrix to A, i.e., it solves the problem: σ min = min rank(b) =1 A B F
9 The High-Level Message.. It is important to be able to think at the block level because of problem structure. It is important to be able to develop block matrix algorithms There is a progression... Simple Linear Algebra Block Linear Algebra Multilinear Algebra
10 Reasoning at the Block Level
11 Uncontrollability The system ẋ = Ax + Bu A IR n n,b IR n p,n>p is uncontrollable if G = t 0 ea(t τ) BB T e Aτ dτ is singular.
12 Uncontrollability The system ẋ = Ax + Bu A IR n n,b IR n p,n>p is uncontrollable if G = t 0 ea(t τ) BB T e Aτ dτ is singular. Ã = A BB T 0 A T eãt = F 11 F 12 G = F 11F T 12 0 F 22
13 Nearness to Uncontrollability The system ẋ = Ax + Bu A IR n n,b IR n p,n>p is nearly uncontrollable if the minimum singular value of is small. G = t 0 ea(t τ) BB T e Aτ dτ Ã = A BB T 0 A eãt = F 11 F 12 G = F 11F T 12 0 F 22
14 Developing Algorithms at the Block Level
15 Block Matrix Factorizations: A Challenge By a block algorithm we mean an algorithm that is rich in matrix-matrix multiplication. Is there a block algorithm for Gaussian elimination? I.e., is there a way to rearrange the O(n 3 ) operations so that the implementation spends at most O(n 2 )timenot doing matrix-matrix multiplication?
16 Why? Re-use ideology: when you touch data, you want to use it a lot. Not all linear algebra operations are equal in this regard. Level Example Data Work 1 α = y T z O(n) O(n) y = y + αz O(n) O(n) 2 y = y + As O(n 2 ) O(n 2 ) A = A + yz T O(n 2 ) O(n 2 ) 3 A = A + BC O(n 2 ) O(n 3 ) Here, α is a scalar, y, z vectors, and A, B, C matrices
17 Scalar LU a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 = u 11 u 12 u 13 0 u 22 u u 33 a 11 = u 11 a 12 = u 12 a 13 = u 13 a 21 = 21 u 11 u 11 = a 11 u 12 = a 12 u 13 = a = a 21 /u 11 a 31 = 31 u = a 31 /u 11 a 22 = 21 u 12 + u 22 u 22 = a u 12 a 23 = 21 u 13 + u 23 u 23 = a u 13 a 32 = 31 u u =(a u 12 )/u 22 a 33 = 31 u u 23 + u 33 u 33 = a u u 23
18 Recursive Description If α IR, v, w IR n 1,andB IR (n 1) (n 1) then A = α v w T B = 1 0 v/α L α w T 0 Ũ is the LU factorization of A if LŨ = à is the LU factorization of à = B vw T /α. Rich in level-2 operations.
19 Block LU: Recursive Description L 11 0 L 21 L 22 U 11 U 12 0 U 22 = A 11 A 12 A 21 A 22 p n p p n p A 11 = L 11 U 11 Get L 11, U 11 via LU of A 11 A 21 = L 21 U 11 Solve triangular systems for L 21 A 12 = L 11 U 12 Solve triangular systems for U 12 A 22 = L 21 U 12 + L 22 U 22 Form à = A 22 L 21 U 12 Get L 22, U 22 via LU of Ã
20 Block LU: Recursive Description L 11 0 L 21 L 22 U 11 U 12 0 U 22 = A 11 A 12 A 21 A 22 p n p p n p A 11 = L 11 U 11 Get L 11, U 11 via LU of A 11 O(p 3 ) A 21 = L 21 U 11 Solve triangular systems for L 21 O(np 2 ) A 12 = L 11 U 12 Solve triangular systems for U 12 O(np 2 ) A 22 = L 21 U 12 + L 22 U 22 Form à = A 22 L 21 U 12 O(n 2 p) Recur Get L 22, U 22 via LU of à Rich in level-3 operations!!!
21 Consider: Ã = A 22 L 21 U 12 p =3: p must be big enough so that the advantage of data re-use is realized.
22 Block Algorithms for (Some) Matrix Factorizations LU with pivoting: PA = LU P permutation, L lower triangular, U upper triangular Cholesky factorization for symmetric positive definite A: A = GG T G lower triangular The QR factorization for rectangular matrices: A = QR Q IR m m orthogonal R IR m n upper triangular
23 Block Matrix Factorization Algorithms via Aggregation
24 Developing a Block QR Factorization The standard algorithm computes Q as a product of Householder reflections, After 2 steps... Q T A = H n H 1 A = R H 2 H 1 A = The H matrices look like this: H = I 2vv T v a unit 2-norm column vector
25 Developing a Block QR Factorization The standard algorithm computes Q as a product of Householder reflections, After 2 steps... Q T A = H n H 1 A = R H 2 H 1 A = The H matrices look like this: H = I 2vv T v a unit 2-norm column vector
26 Developing a Block QR Factorization The standard algorithm computes Q as a product of Householder reflections, After 3 steps... Q T A = H n H 1 A = R H 3 H 2 H 1 A = The H matrices look like this: H = I 2vv T v a unit 2-norm column vector
27 Aggregation A = A 1 A 2 p n p p<<n Generate the first p Householders based on A 1 : H p H 1 A 1 = R 11 (upper triangular) Aggregate H 1,...,H p : H p H 1 = I 2WY T W, Y IR m p Apply to rest of matrix: (H p H 1 )A = (H p H 1 )A 1 (I 2WY T )A 2
28 The WY Representation Aggregation: (I 2WY T )(I 2vv T ) = I 2W + Y T + where W + = W Y + = Y (I 2WY T )v v Application A (I 2WY T )A = A (2W )(Y T A)
29 The Curse of Similarity Transforms
30 A Block Householder Tridiagonalization? A symmetric. Compute orthogonal Q such that Q T AQ = The standard algorithm computes Q as a product of Householder reflections, Q = H 1 H n 2
31 A Block Householder Tridiagonalization? H 1 introduces zeros in first column H T 1 A = Scrambles rows 2 through 6.
32 A Block Householder Tridiagonalization? Must also post-multiply... H T 1 AH 1 = H 1 scrambles columns 2 through 6.
33 A Block Householder Tridiagonalization? H 2 introduces zeros into second column H T 1 AH 1 = Note that because of H 1 s impact, H 2 depends on all of A s entries. The H i can be aggregated, but A must be completely updated along the way destroys the advantage.
34 Jacobi Methods for Symmetric Eigenproblem These methods do not initially reduce A to condensed form. They repeatedly reduce the sum-of-squares of the off-diagonal elements. A = c 0 s s 0 c T A c 0 s s 0 c The (2,4) subproblem: d d 2 = c s s c T a 22 a 24 a 42 a 44 c s s c c 2 + s 2 = 1 and the off-diagonal sum of squares is reduced by 2a 2 24.
35 Jacobi Methods for Symmetric Eigenproblem Cycle through subproblems: (1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4) (1,2): A = 0 0 c s 0 0 s c T A c s 0 0 s c (1,3): A = 0 0 c 0 s s 0 c T A c 0 s s 0 c etc.
36 Parallel Jacobi Parallel Ordering: {(1, 2), (3, 4)}, {(1, 4), (2, 3)}, { (1, 3), (2, 4)} (1,2): A = 0 0 c s 0 0 s c T A c s 0 0 s c (3,4): A = c s 0 0 s c T A c s 0 0 s c
37 Block Jacobi A = 0 0 I Q 11 0 Q I 0 0 Q 21 0 Q 22 T A I Q 11 0 Q I 0 0 Q 21 0 Q 22 The (2,4) subproblem: D D 2 = Q 11 Q 12 Q 21 Q 22 T A 22 A 24 A 42 A 44 Q 11 Q 12 Q 21 Q 22 Convergence analysis requires an understanding of 2-by-2 block matrices that are orthogonal...
38 The CS Decomposition The blocks of an orthogonal matrix have related SVDs: Q 11 Q 12 Q 21 Q 22 = U U 2 C S S C V V 2 T C =diag(cos(θ 1 ),...,cos(θ m )) S =diag(sin(θ 1 ),...,sin(θ m )) U1 T Q 11V 1 = C U1 T Q 12 V 2 = S U2 T Q 21 V 1 = S U2 T Q 22 V 2 = C
39 The Curse of Sparsity
40 HowtoExtractLevel-3Performance? Suppose we want to solve a large sparse Ax = b problem. Iterative methods for this problem can often be dramatically accelerated if you can find a matrix M with two properties: It is easy/efficient to solve linear systems of the form Mz = r. M approximates the essence of A. M is called a preconditioner and the original iteration is modified to effectively solve the equivalent linear system (M 1 A)x =(M 1 b).
41 Idea: Kronecker Product Preconditioners b 11 b 12 b 21 b 22 c 11 c 12 c 13 c 21 c 22 c 23 c 31 c 32 c 33 = b 11 c 11 b 11 c 12 b 11 c 13 b 12 c 11 b 12 c 12 b 12 c 13 b 11 c 21 b 11 c 22 b 11 c 23 b 12 c 21 b 12 c 22 b 12 c 23 b 11 c 31 b 11 c 32 b 11 c 33 b 12 c 31 b 12 c 32 b 12 c 33 b 21 c 11 b 21 c 12 b 21 c 13 b 22 c 11 b 22 c 12 b 22 c 13 b 21 c 21 b 21 c 22 b 21 c 23 b 22 c 21 b 22 c 22 b 22 c 23 b 21 c 31 b 21 c 32 b 21 c 33 b 22 c 31 b 22 c 32 b 22 c 33 Replicated Block Structure
42 Some Properties and a Big Fact Properties: (B C) T = B T C T (B C) 1 = B 1 C 1 (B C)(D F ) = BD CF B (C D) = (B C) D Big Fact: If B and C are m-by-m, then the m 2 -by-m 2 linear system (B C)z = r can be solved in O(m 3 ) time rather than O(m 6 )time.
43 Capturing Essence with a Kronecker Product Given A = A 11 A 12 A 13 A 21 A 22 A 23 A 31 A 32 A 33 A ij IR m m choose B and C to minimize A B C F = A 11 A 12 A 13 A 21 A 22 A 23 A 31 A 32 A 33 b 11 C b 12 C b 13 C b 21 C b 22 C b 23 C b 31 C b 32 C b 33 C F TheexactsolutioncanbeobtainedviatheSVD..
44 Solution of the min A B C Problem Makes the blocks of A into vectors and arrange block-column major order: Ã = col(a 11 ) col(a 21 ) col(a 31 ) col(a 12 ) col(a 33 ) Compute the largest singular value σ max and the corresponding singular vectors u max and v max. B opt = σ max reshape(v max, 3, 3)). C opt = σ max reshape(u max,m,m)).
45 Conclusions It is important to be able to think at the block level because of problem structure. It is important to be able to develop block matrix algorithms There is a progression... Simple Linear Algebra Block Linear Algebra Multilinear Algebra (Thursday)
Today s class. Linear Algebraic Equations LU Decomposition. Numerical Methods, Fall 2011 Lecture 8. Prof. Jinbo Bi CSE, UConn
Today s class Linear Algebraic Equations LU Decomposition 1 Linear Algebraic Equations Gaussian Elimination works well for solving linear systems of the form: AX = B What if you have to solve the linear
More informationLecture 4. Tensor-Related Singular Value Decompositions. Charles F. Van Loan
From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 4. Tensor-Related Singular Value Decompositions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More informationFrom Matrix to Tensor. Charles F. Van Loan
From Matrix to Tensor Charles F. Van Loan Department of Computer Science January 28, 2016 From Matrix to Tensor From Tensor To Matrix 1 / 68 What is a Tensor? Instead of just A(i, j) it s A(i, j, k) or
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationTensor Network Computations in Quantum Chemistry. Charles F. Van Loan Department of Computer Science Cornell University
Tensor Network Computations in Quantum Chemistry Charles F. Van Loan Department of Computer Science Cornell University Joint work with Garnet Chan, Department of Chemistry and Chemical Biology, Cornell
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationSVD, PCA & Preprocessing
Chapter 1 SVD, PCA & Preprocessing Part 2: Pre-processing and selecting the rank Pre-processing Skillicorn chapter 3.1 2 Why pre-process? Consider matrix of weather data Monthly temperatures in degrees
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More informationCS227-Scientific Computing. Lecture 4: A Crash Course in Linear Algebra
CS227-Scientific Computing Lecture 4: A Crash Course in Linear Algebra Linear Transformation of Variables A common phenomenon: Two sets of quantities linearly related: y = 3x + x 2 4x 3 y 2 = 2.7x 2 x
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationCS 323: Numerical Analysis and Computing
CS 323: Numerical Analysis and Computing MIDTERM #1 Instructions: This is an open notes exam, i.e., you are allowed to consult any textbook, your class notes, homeworks, or any of the handouts from us.
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationMain matrix factorizations
Main matrix factorizations A P L U P permutation matrix, L lower triangular, U upper triangular Key use: Solve square linear system Ax b. A Q R Q unitary, R upper triangular Key use: Solve square or overdetrmined
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 5: Numerical Linear Algebra Cho-Jui Hsieh UC Davis April 20, 2017 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationAPPLIED NUMERICAL LINEAR ALGEBRA
APPLIED NUMERICAL LINEAR ALGEBRA James W. Demmel University of California Berkeley, California Society for Industrial and Applied Mathematics Philadelphia Contents Preface 1 Introduction 1 1.1 Basic Notation
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationANALYTICAL MATHEMATICS FOR APPLICATIONS 2018 LECTURE NOTES 3
ANALYTICAL MATHEMATICS FOR APPLICATIONS 2018 LECTURE NOTES 3 ISSUED 24 FEBRUARY 2018 1 Gaussian elimination Let A be an (m n)-matrix Consider the following row operations on A (1) Swap the positions any
More information9. Numerical linear algebra background
Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationLinear Equations and Matrix
1/60 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Gaussian Elimination 2/60 Alpha Go Linear algebra begins with a system of linear
More informationReview of matrices. Let m, n IN. A rectangle of numbers written like A =
Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an
More informationLinear Algebra Section 2.6 : LU Decomposition Section 2.7 : Permutations and transposes Wednesday, February 13th Math 301 Week #4
Linear Algebra Section. : LU Decomposition Section. : Permutations and transposes Wednesday, February 1th Math 01 Week # 1 The LU Decomposition We learned last time that we can factor a invertible matrix
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationSolving linear equations with Gaussian Elimination (I)
Term Projects Solving linear equations with Gaussian Elimination The QR Algorithm for Symmetric Eigenvalue Problem The QR Algorithm for The SVD Quasi-Newton Methods Solving linear equations with Gaussian
More informationPractical Linear Algebra: A Geometry Toolbox
Practical Linear Algebra: A Geometry Toolbox Third edition Chapter 12: Gauss for Linear Systems Gerald Farin & Dianne Hansford CRC Press, Taylor & Francis Group, An A K Peters Book www.farinhansford.com/books/pla
More informationApplied Numerical Linear Algebra. Lecture 8
Applied Numerical Linear Algebra. Lecture 8 1/ 45 Perturbation Theory for the Least Squares Problem When A is not square, we define its condition number with respect to the 2-norm to be k 2 (A) σ max (A)/σ
More informationSingular Value Decomposition
Singular Value Decomposition (Com S 477/577 Notes Yan-Bin Jia Sep, 7 Introduction Now comes a highlight of linear algebra. Any real m n matrix can be factored as A = UΣV T where U is an m m orthogonal
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME MS&E 38 (CME 338 Large-Scale Numerical Optimization Course description Instructor: Michael Saunders Spring 28 Notes : Review The course teaches
More informationLecture 4. CP and KSVD Representations. Charles F. Van Loan
Structured Matrix Computations from Structured Tensors Lecture 4. CP and KSVD Representations Charles F. Van Loan Cornell University CIME-EMS Summer School June 22-26, 2015 Cetraro, Italy Structured Matrix
More informationLecture # 11 The Power Method for Eigenvalues Part II. The power method find the largest (in magnitude) eigenvalue of. A R n n.
Lecture # 11 The Power Method for Eigenvalues Part II The power method find the largest (in magnitude) eigenvalue of It makes two assumptions. 1. A is diagonalizable. That is, A R n n. A = XΛX 1 for some
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationNumerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??
Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More information5.6. PSEUDOINVERSES 101. A H w.
5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and
More informationThere are six more problems on the next two pages
Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with
More informationLinear Systems of n equations for n unknowns
Linear Systems of n equations for n unknowns In many application problems we want to find n unknowns, and we have n linear equations Example: Find x,x,x such that the following three equations hold: x
More informationBlock Bidiagonal Decomposition and Least Squares Problems
Block Bidiagonal Decomposition and Least Squares Problems Åke Björck Department of Mathematics Linköping University Perspectives in Numerical Analysis, Helsinki, May 27 29, 2008 Outline Bidiagonal Decomposition
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationLinear Algebra Methods for Data Mining
Linear Algebra Methods for Data Mining Saara Hyvönen, Saara.Hyvonen@cs.helsinki.fi Spring 2007 The Singular Value Decomposition (SVD) continued Linear Algebra Methods for Data Mining, Spring 2007, University
More informationNext topics: Solving systems of linear equations
Next topics: Solving systems of linear equations 1 Gaussian elimination (today) 2 Gaussian elimination with partial pivoting (Week 9) 3 The method of LU-decomposition (Week 10) 4 Iterative techniques:
More informationDense LU factorization and its error analysis
Dense LU factorization and its error analysis Laura Grigori INRIA and LJLL, UPMC February 2016 Plan Basis of floating point arithmetic and stability analysis Notation, results, proofs taken from [N.J.Higham,
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 7: More on Householder Reflectors; Least Squares Problems Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 15 Outline
More informationCME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 6
CME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 6 GENE H GOLUB Issues with Floating-point Arithmetic We conclude our discussion of floating-point arithmetic by highlighting two issues that frequently
More informationMATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
1 MATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 2 Linear Systems and solutions Systems of linear
More informationProblem Set (T) If A is an m n matrix, B is an n p matrix and D is a p s matrix, then show
MTH 0: Linear Algebra Department of Mathematics and Statistics Indian Institute of Technology - Kanpur Problem Set Problems marked (T) are for discussions in Tutorial sessions (T) If A is an m n matrix,
More information3 QR factorization revisited
LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 2, 2000 30 3 QR factorization revisited Now we can explain why A = QR factorization is much better when using it to solve Ax = b than the A = LU factorization
More informationMatrices and Matrix Algebra.
Matrices and Matrix Algebra 3.1. Operations on Matrices Matrix Notation and Terminology Matrix: a rectangular array of numbers, called entries. A matrix with m rows and n columns m n A n n matrix : a square
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten
More informationLinear Algebra Review
Linear Algebra Review CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Linear Algebra Review 1 / 16 Midterm Exam Tuesday Feb
More informationlecture 2 and 3: algorithms for linear algebra
lecture 2 and 3: algorithms for linear algebra STAT 545: Introduction to computational statistics Vinayak Rao Department of Statistics, Purdue University August 27, 2018 Solving a system of linear equations
More information1 Eigenvalues and eigenvectors
1 Eigenvalues and eigenvectors 1.1 Introduction A non-zero column-vector v is called the eigenvector of a matrix A with the eigenvalue λ, if Av = λv. (1) If an n n matrix A is real and symmetric, A T =
More information1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )
Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical
More informationThis can be accomplished by left matrix multiplication as follows: I
1 Numerical Linear Algebra 11 The LU Factorization Recall from linear algebra that Gaussian elimination is a method for solving linear systems of the form Ax = b, where A R m n and bran(a) In this method
More informationComputational Multilinear Algebra
Computational Multilinear Algebra Charles F. Van Loan Supported in part by the NSF contract CCR-9901988. Involves Working With Kronecker Products b 11 b 12 b 21 b 22 c 11 c 12 c 13 c 21 c 22 c 23 c 31
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 13
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 208 LECTURE 3 need for pivoting we saw that under proper circumstances, we can write A LU where 0 0 0 u u 2 u n l 2 0 0 0 u 22 u 2n L l 3 l 32, U 0 0 0 l n l
More informationA Method for Constructing Diagonally Dominant Preconditioners based on Jacobi Rotations
A Method for Constructing Diagonally Dominant Preconditioners based on Jacobi Rotations Jin Yun Yuan Plamen Y. Yalamov Abstract A method is presented to make a given matrix strictly diagonally dominant
More informationHOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)
HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More information. =. a i1 x 1 + a i2 x 2 + a in x n = b i. a 11 a 12 a 1n a 21 a 22 a 1n. i1 a i2 a in
Vectors and Matrices Continued Remember that our goal is to write a system of algebraic equations as a matrix equation. Suppose we have the n linear algebraic equations a x + a 2 x 2 + a n x n = b a 2
More informationOrthogonalization and least squares methods
Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular
More informationLinear Algebra and Matrices
Linear Algebra and Matrices 4 Overview In this chapter we studying true matrix operations, not element operations as was done in earlier chapters. Working with MAT- LAB functions should now be fairly routine.
More informationSolution of Linear Equations
Solution of Linear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 7, 07 We have discussed general methods for solving arbitrary equations, and looked at the special class of polynomial equations A subclass
More informationAM205: Assignment 2. i=1
AM05: Assignment Question 1 [10 points] (a) [4 points] For p 1, the p-norm for a vector x R n is defined as: ( n ) 1/p x p x i p ( ) i=1 This definition is in fact meaningful for p < 1 as well, although
More informationMatrix & Linear Algebra
Matrix & Linear Algebra Jamie Monogan University of Georgia For more information: http://monogan.myweb.uga.edu/teaching/mm/ Jamie Monogan (UGA) Matrix & Linear Algebra 1 / 84 Vectors Vectors Vector: A
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 12 1 / 18 Overview
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationforms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms
Christopher Engström November 14, 2014 Hermitian LU QR echelon Contents of todays lecture Some interesting / useful / important of matrices Hermitian LU QR echelon Rewriting a as a product of several matrices.
More informationLU Factorization. LU Decomposition. LU Decomposition. LU Decomposition: Motivation A = LU
LU Factorization To further improve the efficiency of solving linear systems Factorizations of matrix A : LU and QR LU Factorization Methods: Using basic Gaussian Elimination (GE) Factorization of Tridiagonal
More informationPOLI270 - Linear Algebra
POLI7 - Linear Algebra Septemer 8th Basics a x + a x +... + a n x n b () is the linear form where a, b are parameters and x n are variables. For a given equation such as x +x you only need a variable and
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationMatrix Factorization and Analysis
Chapter 7 Matrix Factorization and Analysis Matrix factorizations are an important part of the practice and analysis of signal processing. They are at the heart of many signal-processing algorithms. Their
More information18.06SC Final Exam Solutions
18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon
More informationlecture 3 and 4: algorithms for linear algebra
lecture 3 and 4: algorithms for linear algebra STAT 545: Introduction to computational statistics Vinayak Rao Department of Statistics, Purdue University August 30, 2016 Solving a system of linear equations
More informationIndex. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)
page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More information13-2 Text: 28-30; AB: 1.3.3, 3.2.3, 3.4.2, 3.5, 3.6.2; GvL Eigen2
The QR algorithm The most common method for solving small (dense) eigenvalue problems. The basic algorithm: QR without shifts 1. Until Convergence Do: 2. Compute the QR factorization A = QR 3. Set A :=
More informationENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition
ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Lecture 8: QR Decomposition
More informationLecture 5 Singular value decomposition
Lecture 5 Singular value decomposition Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn
More information7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix
EE507 - Computational Techniques for EE 7. LU factorization Jitkomut Songsiri factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 12: Gaussian Elimination and LU Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Gaussian Elimination
More informationLecture 6. Numerical methods. Approximation of functions
Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Least-square method basis functions design matrix residual weighted least squares normal equation
More informationLinear Analysis Lecture 16
Linear Analysis Lecture 16 The QR Factorization Recall the Gram-Schmidt orthogonalization process. Let V be an inner product space, and suppose a 1,..., a n V are linearly independent. Define q 1,...,
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationMaths for Signals and Systems Linear Algebra for Engineering Applications
Maths for Signals and Systems Linear Algebra for Engineering Applications Lectures 1-2, Tuesday 11 th October 2016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE LONDON
More information9. Numerical linear algebra background
Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization
More informationMAT 610: Numerical Linear Algebra. James V. Lambers
MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More information