# Numerical Methods I Eigenvalue Problems

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 1 MATH-GA / CSCI-GA , Fall 2014 October 2nd, 2014 A. Donev (Courant Institute) Lecture IV 10/ / 23

2 Outline 1 Review of Linear Algebra: Eigenvalues 2 Conditioning of Eigenvalue Problems 3 Computing Eigenvalues and Eigenvectors 4 Methods based on QR factorizations 5 Conclusions A. Donev (Courant Institute) Lecture IV 10/ / 23

3 Review of Linear Algebra: Eigenvalues Eigenvalue Decomposition For a square matrix A C n n, there exists at least one λ such that Ax = λx (A λi) y = 0 Putting the eigenvectors x j as columns in a matrix X, and the eigenvalues λ j on the diagonal of a diagonal matrix Λ, we get AX = XΛ. A matrix is non-defective or diagonalizable if there exist n linearly independent eigenvectors, i.e., if the matrix X is invertible: X 1 AX = Λ A = XΛX 1. The transformation from A to Λ = X 1 AX is called a similarity transformation and it preserves the eigenspace. A. Donev (Courant Institute) Lecture IV 10/ / 23

4 Review of Linear Algebra: Eigenvalues Unitarily Diagonalizable Matrices A matrix is unitarily diagonalizable if there exist n linearly independent orthogonal eigenvectors, i.e., if the matrix X can be chosen to be unitary (orthogonal), X U, where U 1 = U : A = UΛU. Note that unitary matrices generalize orthogonal matrices to the complex domain, so we use adjoints (conjugate transposes) instead of transposes throughout. Theorem: A matrix is unitarily diagonalizable iff it is normal, i.e., it commutes with its adjoint: A A = AA. Theorem: Hermitian (symmetric) matrices, A = A, are unitarily diagonalizable and have real eigenvalues. A. Donev (Courant Institute) Lecture IV 10/ / 23

5 Review of Linear Algebra: Eigenvalues Left Eigenvectors The usual eigenvectors are more precisely called right eigenvectors. There is also left eigenvector corresponding to a given eigenvalue λ y A = λy A y = λy. Y A = ΛY For a matrix that is diagonalizable, observe that Y = X 1 and so the left eigenvectors provide no new information. For unitarily diagonalizable matrices, Y = ( X 1) = U, so that the left and right eigenvectors coincide. A. Donev (Courant Institute) Lecture IV 10/ / 23

6 Review of Linear Algebra: Eigenvalues Non-diagonalizable Matrices For matrices that are not diagonalizable, one can use Jordan form factorizations, or, more relevant to numerical mathematics, the Schur factorization (decomposition): where T is upper-triangular. A = UTU, The eigenvalues are on the diagonal of T. Note: Observe that A = (UTU ) = UT U so for Hermitian matrices T = T is real diagonal. An important property / use of eigenvalues: A n = (UTU ) (UTU ) (UTU ) = UT (U U) T (U U) TU A n = UT n U A. Donev (Courant Institute) Lecture IV 10/ / 23

7 Conditioning of Eigenvalue Problems Sensitivity of Eigenvalues Now consider a perturbation of a diagonalizable matrix δa and see how perturbed the similar matrix becomes: X 1 (A + δa) X = Λ + δλ δλ = X 1 (δa) X δλ X 1 δa X = κ (X) δa Conclusion: The conditioning of the eigenvalue problem is related to the conditioning of the matrix of eigenvectors. If X is unitary then X 2 = 1 (from now on we exclusively work with the 2-norm): Unitarily diagonalizable matrices are always perfectly conditioned! Warning: The absolute error in all eigenvalues is of the same order, meaning that the relative error will be very large for the smallest eigenvalues. A. Donev (Courant Institute) Lecture IV 10/ / 23

8 Conditioning of Eigenvalue Problems Sensitivity of Individual Eigenvalues A. Donev (Courant Institute) Lecture IV 10/ / 23

9 Conditioning of Eigenvalue Problems Sensitivity of Individual Eigenvalues δλ y (δa) x y x Recalling the Cauchy-Schwartz inequality: y x = x y cos θ xy x y δλ x δa y = δa x y cos θ xy cos θ xy Defining a conditioning number for a given eigenvalue κ (λ, A) = sup δa δλ δa = 1 cos θ xy For unitarily diagonalizable matrices y = x and thus κ (λ, A) = 1: perfectly conditioned! A. Donev (Courant Institute) Lecture IV 10/ / 23

10 Conditioning of Eigenvalue Problems Sensitivity of Eigenvectors A priori estimate: The conditioning number for the eigenvector itself depends on the separation between the eigenvalues κ (x, A) = ( ) 1 min λ λ j j This indicates that multiple eigenvalues require care. Even for Hermitian matrices eigenvectors are hard to compute. If there is a defective (non-diagonalizable) matrix with eigenvalue for which the difference between the algebraic and geometric multiplicities is d > 0, then δλ δa 1/(1+d), which means the conditioning number is infinite: Defective eigenvalues are very badly conditioned. A. Donev (Courant Institute) Lecture IV 10/ / 23

11 Computing Eigenvalues and Eigenvectors The need for iterative algorithms The eigenvalues are roots of the characteristic polynomial of A, which is generally of order n. According to Abel s theorem, there is no closed-form (rational) solution for n 5. All eigenvalue algorithms must be iterative! This is a fundamental difference from, example, linear solvers. There is an important distinction between iterative methods to: Compute all eigenvalues (similarity transformations). Compute only one or a few eigenvalues, typically the smallest or the largest one (power-like methods). Bounds on eigenvalues are important, e.g., Courant-Fisher theorem for the Rayleigh quotient: min λ r A (x) = x Ax x x max λ A. Donev (Courant Institute) Lecture IV 10/ / 23

12 Computing Eigenvalues and Eigenvectors The Power Method Recall that for a diagonalizable matrix A n = XΛ n X 1 and assume λ 1 > λ 2 λ 3 λ n and that the columns of X are normalized, x j = 1. Any initial guess vector q 0 can be represented in the linear basis formed by the eigenvectors q 0 = Xa Recall iterative methods for linear systems: Multiply a vector with the matrix A many times: q k+1 = Aq k q n = A n q 0 = ( XΛ n X 1) Xa = X (Λ n a) A. Donev (Courant Institute) Lecture IV 10/ / 23

13 Computing Eigenvalues and Eigenvectors Power Method As n, the eigenvalue of largest modulus λ 0 will dominate, { ( ) n } Λ n = λ n λ2 1Diag 1,,... λ n 1Diag {1, 0,..., 0} λ 1 q n = X (Λ n a) λ n 1X a = λn 1x 1 Therefore the normalized iterates converge to the eigenvector: q n = q n q n x 1 The Rayleigh quotient converges to the eigenvalue: r A (q n ) = q naq n q n q n = q na q n λ 1 A. Donev (Courant Institute) Lecture IV 10/ / 23

14 Computing Eigenvalues and Eigenvectors An alternative derivation A. Donev (Courant Institute) Lecture IV 10/ / 23

15 Computing Eigenvalues and Eigenvectors Power Method Implementation Start with an initial guess q 0, and then iterate: 1 Compute matrix-vector product and normalize it: q k = Aq k 1 Aq k 1 2 Use Raleigh quotient to obtain eigenvalue estimate: ˆλ k = q kaq k 3 Test for convergence: Evaluate the residual r k = Aq k ˆλ k q k and terminate if the error estimate is small enough: λ 1 ˆλ k r k cos θ xy < ε A. Donev (Courant Institute) Lecture IV 10/ / 23

16 Computing Eigenvalues and Eigenvectors Convergence Estimates The normalized iterates converge to the eigenvector linearly: ( λ 2 ) k q k (±x 1 ) = O If A is normal the eigenvalue estimate converges a bit faster but still linearly ( λ 2 ) 2k ˆλ k λ 1 O The power method is fast when the dominant eigenvalue is well-separated from the rest (even if it is degenerate). This conclusion is rather general for all iterative methods: Convergence is good for well-separated eigenvalues, bad otherwise. The power method is typically too slow to be used in practice and there are more sophisticated alternatives (Lanczos/Arnoldi iteration). A. Donev (Courant Institute) Lecture IV 10/ / 23 λ 1 λ 1

17 Computing Eigenvalues and Eigenvectors Inverse Power Iteration Observe that applying the power method to A 1 will find the largest of λ 1 j, i.e., the smallest eigenvalue (by modulus). If we have an eigenvalue estimate µ λ, then doing the power method for the matrix (A µi) 1 will give the eigenvalue closest to µ. Convergence will be faster if µ is much closer to λ then to other eigenvalues. Recall that in practice (A µi) 1 q is computed by solving a linear system, not matrix inversion (one can reuse an LU factorization)! Finally, if we have an estimate of both the eigenvalue and the eigenvector, we can use Rayleigh Quotient Iteration (see homework). A. Donev (Courant Institute) Lecture IV 10/ / 23

18 Methods based on QR factorizations Estimating all eigenvalues / eigenvectors Iterative methods akin the power method are not suitable for estimating all eigenvalues. Basic idea: Build a sequence of matrices A k that all share eigenvalues with A via similarity transformations: A k+1 = P 1 A k P, starting from A 1 = A. A numerically stable and good way to do this is to use the QR factorization: A k = Q k+1 R k+1 A k+1 = Q 1 k+1 A kq k+1 = ( Q 1 k+1 Q k+1) Rk+1 Q k+1 = R k+1 Q k+1. Note that the fact the Q s are orthogonal is crucial to keep the conditioning from getting worse. A. Donev (Courant Institute) Lecture IV 10/ / 23

19 Methods based on QR factorizations The basic QR method The behavior of the QR iteration can be understood most transparently as follows [following Trefethen and Bau]: Observation: The range of the matrix A k converges to the space spanned by the eigenvectors of A, with the eigenvectors corresponding to the largest eigenvalues dominating as k (so this is ill-conditioned). Recall: The columns of Q in A = QR form an orthonormal basis for the range of A. Idea: Form a well-conditioned basis for the eigenspace of A by factorizing: A k = Q k R k and then calculate A k = Q 1 k A Q k = Q ka Q k. It is not too hard to show that this produces the same sequence of matrices A k as the QR algorithm. A. Donev (Courant Institute) Lecture IV 10/ / 23

20 Methods based on QR factorizations Why the QR algorithm works Summary: The columns of Q k converge to the eigenvectors, and A k = Q ka Q k. We can recognize the above as a matrix of Rayleigh quotients, which for diagonalizable matrices { (A k ) ij = q λ i if i = j i A q j λ i δ ij = 0 if i j showing that (under suitable assumptions): It can also be shown that A k Λ Q k = Q 1 Q 2 Q k X A. Donev (Courant Institute) Lecture IV 10/ / 23

21 Methods based on QR factorizations More on QR algorithm The convergence of the basic QR algorithm is closely related to that of the power method: It is only fast if all eigenvalues are well-separated. For more general (non-diagonalizable) matrices in complex arithmetic, the algorithm converges to the Schur decomposition A = UTU, A k T and Q k U. It is possible to implement the algorithm entirely using real arithmetic (no complex numbers). There are several key improvements to the basic method that make this work in practice: Hessenberg matrices for faster QR factorization, shifts and deflation for acceleration. There are other sophisticated algorithms as well, such as the divide-and-conquer algorithm, and the best are implemented in the library LAPACK (MATLAB). A. Donev (Courant Institute) Lecture IV 10/ / 23

22 Conclusions Eigenvalues in MATLAB In MATLAB, sophisticated variants of the QR algorithm (LAPACK library) are implemented in the function eig: Λ = eig(a) [X, Λ] = eig(a) For large or sparse matrices, iterative methods based on the Arnoldi iteration (ARPACK library), can be used to obtain a few of the largest/smallest/closest-to-µ eigenvalues: Λ = eigs(a, n eigs ) [X, Λ] = eigs(a, n eigs ) The Schur decomposition is provided by [U, T ] = schur(a). A. Donev (Courant Institute) Lecture IV 10/ / 23

23 Conclusions Conclusions/Summary Eigenvalues are well-conditioned for unitarily diagonalizable matrices (includes Hermitian matrices), but ill-conditioned for nearly non-diagonalizable matrices. Eigenvectors are well-conditioned only when eigenvalues are well-separated. Eigenvalue algorithms are always iterative. The power method and its variants can be used to find the largest or smallest eigenvalue, and they converge fast if there is a large separation between the target eigenvalue and nearby ones. Estimating all eigenvalues and/or eigenvectors can be done by combining the power method with QR factorizations. MATLAB has high-quality implementations of sophisticated variants of these algorithms. A. Donev (Courant Institute) Lecture IV 10/ / 23

### Numerical Methods I Singular Value Decomposition

Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)

### Numerical Methods I Solving Square Linear Systems: GEM and LU factorization

Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,

### The Eigenvalue Problem: Perturbation Theory

Jim Lambers MAT 610 Summer Session 2009-10 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just

### Scientific Computing: Solving Linear Systems

Scientific Computing: Solving Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 September 17th and 24th, 2015 A. Donev (Courant

### Numerical Methods I: Eigenvalues and eigenvectors

1/25 Numerical Methods I: Eigenvalues and eigenvectors Georg Stadler Courant Institute, NYU stadler@cims.nyu.edu November 2, 2017 Overview 2/25 Conditioning Eigenvalues and eigenvectors How hard are they

### Numerical Methods I Solving Nonlinear Equations

Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)

### Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic

Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give

### Eigenvalues and Eigenvectors

Chapter 1 Eigenvalues and Eigenvectors Among problems in numerical linear algebra, the determination of the eigenvalues and eigenvectors of matrices is second in importance only to the solution of linear

### Numerical Solution of Linear Eigenvalue Problems

Numerical Solution of Linear Eigenvalue Problems Jessica Bosch and Chen Greif Abstract We review numerical methods for computing eigenvalues of matrices We start by considering the computation of the dominant

### Linear algebra and applications to graphs Part 1

Linear algebra and applications to graphs Part 1 Written up by Mikhail Belkin and Moon Duchin Instructor: Laszlo Babai June 17, 2001 1 Basic Linear Algebra Exercise 1.1 Let V and W be linear subspaces

### Computational Methods. Eigenvalues and Singular Values

Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Definition (T -invariant subspace) Example. Example

Eigenvalues, Eigenvectors, Similarity, and Diagonalization We now turn our attention to linear transformations of the form T : V V. To better understand the effect of T on the vector space V, we begin

### Review of some mathematical tools

MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

### Eigenvalues, Eigenvectors, and Diagonalization

Week12 Eigenvalues, Eigenvectors, and Diagonalization 12.1 Opening Remarks 12.1.1 Predicting the Weather, Again Let us revisit the example from Week 4, in which we had a simple model for predicting the

### HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)

HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

### Math Linear Algebra Final Exam Review Sheet

Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

### I. Multiple Choice Questions (Answer any eight)

Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

### Eigenvalue Problems and Singular Value Decomposition

Eigenvalue Problems and Singular Value Decomposition Sanzheng Qiao Department of Computing and Software McMaster University August, 2012 Outline 1 Eigenvalue Problems 2 Singular Value Decomposition 3 Software

### The QR Decomposition

The QR Decomposition We have seen one major decomposition of a matrix which is A = LU (and its variants) or more generally PA = LU for a permutation matrix P. This was valid for a square matrix and aided

### Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

### Jordan Canonical Form Homework Solutions

Jordan Canonical Form Homework Solutions For each of the following, put the matrix in Jordan canonical form and find the matrix S such that S AS = J. [ ]. A = A λi = λ λ = ( λ) = λ λ = λ =, Since we have

### Linear Algebra. Workbook

Linear Algebra Workbook Paul Yiu Department of Mathematics Florida Atlantic University Last Update: November 21 Student: Fall 2011 Checklist Name: A B C D E F F G H I J 1 2 3 4 5 6 7 8 9 10 xxx xxx xxx

### 13-2 Text: 28-30; AB: 1.3.3, 3.2.3, 3.4.2, 3.5, 3.6.2; GvL Eigen2

The QR algorithm The most common method for solving small (dense) eigenvalue problems. The basic algorithm: QR without shifts 1. Until Convergence Do: 2. Compute the QR factorization A = QR 3. Set A :=

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### MATH 304 Linear Algebra Lecture 34: Review for Test 2.

MATH 304 Linear Algebra Lecture 34: Review for Test 2. Topics for Test 2 Linear transformations (Leon 4.1 4.3) Matrix transformations Matrix of a linear mapping Similar matrices Orthogonality (Leon 5.1

### 18.06SC Final Exam Solutions

18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon

### AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Rayleigh Quotient Iteration Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Solving Eigenvalue Problems All

### Lecture 3: QR-Factorization

Lecture 3: QR-Factorization This lecture introduces the Gram Schmidt orthonormalization process and the associated QR-factorization of matrices It also outlines some applications of this factorization

### Linear Algebra in Actuarial Science: Slides to the lecture

Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations

### Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

### 22m:033 Notes: 7.1 Diagonalization of Symmetric Matrices

m:33 Notes: 7. Diagonalization of Symmetric Matrices Dennis Roseman University of Iowa Iowa City, IA http://www.math.uiowa.edu/ roseman May 3, Symmetric matrices Definition. A symmetric matrix is a matrix

### Math 411 Preliminaries

Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector

### Homework 1 Elena Davidson (B) (C) (D) (E) (F) (G) (H) (I)

CS 106 Spring 2004 Homework 1 Elena Davidson 8 April 2004 Problem 1.1 Let B be a 4 4 matrix to which we apply the following operations: 1. double column 1, 2. halve row 3, 3. add row 3 to row 1, 4. interchange

### BASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x

BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2

EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko

### The Lanczos and conjugate gradient algorithms

The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization

### 5. Orthogonal matrices

L Vandenberghe EE133A (Spring 2017) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal

### AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline

### 18.06 Problem Set 8 - Solutions Due Wednesday, 14 November 2007 at 4 pm in

806 Problem Set 8 - Solutions Due Wednesday, 4 November 2007 at 4 pm in 2-06 08 03 Problem : 205+5+5+5 Consider the matrix A 02 07 a Check that A is a positive Markov matrix, and find its steady state

### Eigenvalues and Singular Values

Chapter 10 Eigenvalues and Singular Values This chapter is about eigenvalues and singular values of matrices. Computational algorithms and sensitivity to perburbations are both discussed. 10.1 Eigenvalue

### ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA

ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA Kent State University Department of Mathematical Sciences Compiled and Maintained by Donald L. White Version: August 29, 2017 CONTENTS LINEAR ALGEBRA AND

### and let s calculate the image of some vectors under the transformation T.

Chapter 5 Eigenvalues and Eigenvectors 5. Eigenvalues and Eigenvectors Let T : R n R n be a linear transformation. Then T can be represented by a matrix (the standard matrix), and we can write T ( v) =

### Diagonalization by a unitary similarity transformation

Physics 116A Winter 2011 Diagonalization by a unitary similarity transformation In these notes, we will always assume that the vector space V is a complex n-dimensional space 1 Introduction A semi-simple

### Today: eigenvalue sensitivity, eigenvalue algorithms Reminder: midterm starts today

AM 205: lecture 22 Today: eigenvalue sensitivity, eigenvalue algorithms Reminder: midterm starts today Posted online at 5 PM on Thursday 13th Deadline at 5 PM on Friday 14th Covers material up to and including

### Orthogonal iteration revisited

Week 10 11: Friday, Oct 30 and Monday, Nov 2 Orthogonal iteration revisited Last time, we described a generalization of the power methods to compute invariant subspaces. That is, starting from some initial

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Jordan normal form notes (version date: 11/21/07)

Jordan normal form notes (version date: /2/7) If A has an eigenbasis {u,, u n }, ie a basis made up of eigenvectors, so that Au j = λ j u j, then A is diagonal with respect to that basis To see this, let

### Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) Eigenvalues and Eigenvectors Fall 2015 1 / 14 Introduction We define eigenvalues and eigenvectors. We discuss how to

### MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization.

MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization. Eigenvalues and eigenvectors of an operator Definition. Let V be a vector space and L : V V be a linear operator. A number λ

### ICS 6N Computational Linear Algebra Symmetric Matrices and Orthogonal Diagonalization

ICS 6N Computational Linear Algebra Symmetric Matrices and Orthogonal Diagonalization Xiaohui Xie University of California, Irvine xhx@uci.edu Xiaohui Xie (UCI) ICS 6N 1 / 21 Symmetric matrices An n n

### Matrix Theory, Math6304 Lecture Notes from October 25, 2012

Matrix Theory, Math6304 Lecture Notes from October 25, 2012 taken by John Haas Last Time (10/23/12) Example of Low Rank Perturbation Relationship Between Eigenvalues and Principal Submatrices: We started

### Orthogonal iteration to QR

Week 10: Wednesday and Friday, Oct 24 and 26 Orthogonal iteration to QR On Monday, we went through a somewhat roundabout algbraic path from orthogonal subspace iteration to the QR iteration. Let me start

### 5.3.5 The eigenvalues are 3, 2, 3 (i.e., the diagonal entries of D) with corresponding eigenvalues. Null(A 3I) = Null( ), 0 0

535 The eigenvalues are 3,, 3 (ie, the diagonal entries of D) with corresponding eigenvalues,, 538 The matrix is upper triangular so the eigenvalues are simply the diagonal entries, namely 3, 3 The corresponding

### Jordan Canonical Form of a Nilpotent Matrix. Math 422

Jordan Canonical Form of a Nilpotent Matrix Math Schur s Triangularization Theorem tells us that every matrix A is unitarily similar to an upper triangular matrix T However, the only thing certain at this

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### COMP6237 Data Mining Covariance, EVD, PCA & SVD. Jonathon Hare

COMP6237 Data Mining Covariance, EVD, PCA & SVD Jonathon Hare jsh2@ecs.soton.ac.uk Variance and Covariance Random Variables and Expected Values Mathematicians talk variance (and covariance) in terms of

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### 1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)

Notes for 2017-01-30 Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty

### Eigenvalue problems III: Advanced Numerical Methods

Eigenvalue problems III: Advanced Numerical Methods Sam Sinayoko Computational Methods 10 Contents 1 Learning Outcomes 2 2 Introduction 2 3 Inverse Power method: finding the smallest eigenvalue of a symmetric

### 7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.

7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 7.3 The Jacobi and Gauss-Siedel Iterative Techniques

### Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### Math 471 (Numerical methods) Chapter 4. Eigenvalues and Eigenvectors

Math 471 (Numerical methods) Chapter 4. Eigenvalues and Eigenvectors Overlap 4.1 4.3 of Bradie 4.0 Review on eigenvalues and eigenvectors Definition. A nonzero vector v is called an eigenvector of A if

### Homework 2. Solutions T =

Homework. s Let {e x, e y, e z } be an orthonormal basis in E. Consider the following ordered triples: a) {e x, e x + e y, 5e z }, b) {e y, e x, 5e z }, c) {e y, e x, e z }, d) {e y, e x, 5e z }, e) {

### The Spectral Theorem for normal linear maps

MAT067 University of California, Davis Winter 2007 The Spectral Theorem for normal linear maps Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (March 14, 2007) In this section we come back to the question

### 4. Linear transformations as a vector space 17

4 Linear transformations as a vector space 17 d) 1 2 0 0 1 2 0 0 1 0 0 0 1 2 3 4 32 Let a linear transformation in R 2 be the reflection in the line = x 2 Find its matrix 33 For each linear transformation

### 4. Determinants.

4. Determinants 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 2 2 determinant 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 3 3 determinant 4.1.

### AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 9 Minimizing Residual CG

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors week -2 Fall 26 Eigenvalues and eigenvectors The most simple linear transformation from R n to R n may be the transformation of the form: T (x,,, x n ) (λ x, λ 2,, λ n x n

### LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT FALL 2006 PRINCETON UNIVERSITY

LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT 204 - FALL 2006 PRINCETON UNIVERSITY ALFONSO SORRENTINO 1 Adjoint of a linear operator Note: In these notes, V will denote a n-dimensional euclidean vector

### B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### MATH2071: LAB #5: Norms, Errors and Condition Numbers

MATH2071: LAB #5: Norms, Errors and Condition Numbers 1 Introduction Introduction Exercise 1 Vector Norms Exercise 2 Matrix Norms Exercise 3 Compatible Matrix Norms Exercise 4 More on the Spectral Radius

### Linear Algebra using Dirac Notation: Pt. 2

Linear Algebra using Dirac Notation: Pt. 2 PHYS 476Q - Southern Illinois University February 6, 2018 PHYS 476Q - Southern Illinois University Linear Algebra using Dirac Notation: Pt. 2 February 6, 2018

### Numerical Methods I Orthogonal Polynomials

Numerical Methods I Orthogonal Polynomials Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001, Fall 2010 Nov. 4th and 11th, 2010 A. Donev (Courant Institute)

### Contents. Bibliography 125. Index 129

Contents 9 Matrix Eigenvalue Problems 1 9.1 Basic Properties........................... 1 9.1.1 Introduction...................... 1 9.1.2 Complex Matrices................... 2 9.1.3 Theoretical Background................

### 0.1 Eigenvalues and Eigenvectors

0.. EIGENVALUES AND EIGENVECTORS MATH 22AL Computer LAB for Linear Algebra Eigenvalues and Eigenvectors Dr. Daddel Please save your MATLAB Session (diary)as LAB9.text and submit. 0. Eigenvalues and Eigenvectors

### Mathematical Foundations

Chapter 1 Mathematical Foundations 1.1 Big-O Notations In the description of algorithmic complexity, we often have to use the order notations, often in terms of big O and small o. Loosely speaking, for

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### Linear Algebra Practice Problems

Linear Algebra Practice Problems Math 24 Calculus III Summer 25, Session II. Determine whether the given set is a vector space. If not, give at least one axiom that is not satisfied. Unless otherwise stated,

### Computational Methods CMSC/AMSC/MAPL 460. EigenValue decomposition Singular Value Decomposition. Ramani Duraiswami, Dept. of Computer Science

Computational Methods CMSC/AMSC/MAPL 460 EigenValue decomposition Singular Value Decomposition Ramani Duraiswami, Dept. of Computer Science Hermitian Matrices A square matrix for which A = A H is said

### c c c c c c c c c c a 3x3 matrix C= has a determinant determined by

Linear Algebra Determinants and Eigenvalues Introduction: Many important geometric and algebraic properties of square matrices are associated with a single real number revealed by what s known as the determinant.

### Computational math: Assignment 1

Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange

### Lecture Notes: Eigenvalues and Eigenvectors. 1 Definitions. 2 Finding All Eigenvalues

Lecture Notes: Eigenvalues and Eigenvectors Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk 1 Definitions Let A be an n n matrix. If there

### Math 2331 Linear Algebra

5. Eigenvectors & Eigenvalues Math 233 Linear Algebra 5. Eigenvectors & Eigenvalues Shang-Huan Chiu Department of Mathematics, University of Houston schiu@math.uh.edu math.uh.edu/ schiu/ Shang-Huan Chiu,

### Linear vector spaces and subspaces.

Math 2051 W2008 Margo Kondratieva Week 1 Linear vector spaces and subspaces. Section 1.1 The notion of a linear vector space. For the purpose of these notes we regard (m 1)-matrices as m-dimensional vectors,

### Honors Linear Algebra, Spring Homework 8 solutions by Yifei Chen

.. Honors Linear Algebra, Spring 7. Homework 8 solutions by Yifei Chen 8... { { W {v R 4 v α v β } v x, x, x, x 4 x x + x 4 x + x x + x 4 } Solve the linear system above, we get a basis of W is {v,,,,

### Numerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??

Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement

### Off-diagonal perturbation, first-order approximation and quadratic residual bounds for matrix eigenvalue problems

Off-diagonal perturbation, first-order approximation and quadratic residual bounds for matrix eigenvalue problems Yuji Nakatsukasa Abstract When a symmetric block diagonal matrix [ A 1 A2 ] undergoes an

### Section 7.3: SYMMETRIC MATRICES AND ORTHOGONAL DIAGONALIZATION

Section 7.3: SYMMETRIC MATRICES AND ORTHOGONAL DIAGONALIZATION When you are done with your homework you should be able to Recognize, and apply properties of, symmetric matrices Recognize, and apply properties

### Ph.D. Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2) EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified.

PhD Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2 EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified Problem 1 [ points]: For which parameters λ R does the following system

### MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2.

MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. Diagonalization Let L be a linear operator on a finite-dimensional vector space V. Then the following conditions are equivalent:

### Math 110 Linear Algebra Midterm 2 Review October 28, 2017

Math 11 Linear Algebra Midterm Review October 8, 17 Material Material covered on the midterm includes: All lectures from Thursday, Sept. 1st to Tuesday, Oct. 4th Homeworks 9 to 17 Quizzes 5 to 9 Sections

### Extra Problems for Math 2050 Linear Algebra I

Extra Problems for Math 5 Linear Algebra I Find the vector AB and illustrate with a picture if A = (,) and B = (,4) Find B, given A = (,4) and [ AB = A = (,4) and [ AB = 8 If possible, express x = 7 as

### Notes on Linear Algebra and Matrix Analysis

Notes on Linear Algebra and Matrix Analysis Maxim Neumann May 2006, Version 0.1.1 1 Matrix Basics Literature to this topic: [1 4]. x y < y,x >: standard inner product. x x = 1 : x is normalized x y = 0