# Today: eigenvalue sensitivity, eigenvalue algorithms Reminder: midterm starts today

Save this PDF as:

Size: px
Start display at page:

Download "Today: eigenvalue sensitivity, eigenvalue algorithms Reminder: midterm starts today" ## Transcription

1 AM 205: lecture 22 Today: eigenvalue sensitivity, eigenvalue algorithms Reminder: midterm starts today Posted online at 5 PM on Thursday 13th Deadline at 5 PM on Friday 14th Covers material up to and including lecture 16 Open book. No collaboration allowed. Send questions as private messages on Piazza

2 Gershgorin s Theorem (from last time) Theorem: All eigenvalues of A C n n are contained within the union of the n Gershgorin disks of A

3 Sensitivity of Eigenvalue Problems We shall now consider the sensitivity of the eigenvalues to perturbations in the matrix A Suppose A is nondefective, and hence A = VDV 1 Let δa denote a perturbation of A, and let E V 1 δav, then V 1 (A + δa)v = V 1 AV + V 1 δav = D + E

4 Sensitivity of Eigenvalue Problems For a nonsingular matrix X, the map A X 1 AX is called a similarity transformation of A Theorem: A similarity transformation preserves eigenvalues Proof: We can equate the characteristic polynomials of A and X 1 AX (denoted p A (z) and p X 1 AX (z), respectively) as follows: p X 1 AX (z) = det(zi X 1 AX ) = det(x 1 (zi A)X ) = det(x 1 ) det(zi A) det(x ) = det(zi A) = p A (z), where we have used the identities det(ab) = det(a) det(b), and det(x 1 ) = 1/ det(x )

5 Sensitivity of Eigenvalue Problems The identity V 1 (A + δa)v = D + E is a similarity transformation Therefore A + δa and D + E have the same eigenvalues Let λ k, k = 1, 2,..., n denote the eigenvalues of A, and λ denote an eigenvalue of A + δa Then for some w C n, ( λ, w) is an eigenpair of (D + E), i.e. (D + E)w = λw

6 Sensitivity of Eigenvalue Problems This can be rewritten as w = ( λi D) 1 Ew This is a promising start because: we want to bound λ λ k for some k ( λi D) 1 is a diagonal matrix with entries 1/( λ λ k ) on the diagonal

7 Sensitivity of Eigenvalue Problems Taking norms yields w 2 ( λi D) 1 2 E 2 w 2, or ( λi D) E 2 Note that the norm of a diagonal matrix is given by its largest entry (in abs. val.) 1 max v 0 Dv v (D 11 v 1, D 22 v 2,..., D nn v n ) = max v 0 v { } max D v ii max i=1,2,...,n v 0 v = max D ii i=1,2,...,n 1 This holds for any induced matrix norm, not just the 2-norm

8 Sensitivity of Eigenvalue Problems Hence ( λi D) 1 2 = 1/ λ λ k, where λ k of A closest to λ is the eigenvalue Therefore it follows from ( λi D) E 2 that λ λ k = ( λi D) E 2 = V 1 δav 2 V 1 2 δa 2 V 2 = cond(v ) δa 2 This result is known as the Bauer Fike Theorem

9 Sensitivity of Eigenvalue Problems Hence suppose we compute the eigenvalues, λ i, of the perturbed matrix A + δa Then Bauer Fike tells us that each λ i must reside in a disk of radius cond(v ) δa 2 centered on some eigenvalue of A If V is poorly conditioned, then even for small perturbations δa, the disks can be large: sensitivity to perturbations If A is normal then cond(v ) = 1, in which case the Bauer Fike disk radius is just δa 2

10 Sensitivity of Eigenvalue Problems Note that a limitation of Bauer Fike is that it does not tell us which disk λ i will reside in Therefore, this doesn t rule out the possibility of, say, all λ i clustering in just one Bauer Fike disk In the case that A and A + δa are hermitian, we have a stronger result

11 Sensitivity of Eigenvalue Problems Weyl s Theorem: Let λ 1 λ 2 λ n and λ 1 λ 2 λ n be the eigenvalues of hermitian matrices A and A + δa, respectively. Then max λ i λ i δa 2. i=1,...,n Hence in the hermitian case, each perturbed eigenvalue must be in the disk 2 of its corresponding unperturbed eigenvalue! 2 In fact, eigenvalues of a hermitian matrix are real, so disk here is actually an interval in R

12 Sensitivity of Eigenvalue Problems The Bauer Fike Theorem relates to perturbations of the whole spectrum We can also consider perturbations of individual eigenvalues Suppose, for simplicity, that A C n n is symmetric, and consider the perturbed eigenvalue problem (A + E)(v + v) = (λ + λ)(v + v) Expanding this equation, dropping second order terms, and using Av = λv gives A v + Ev λv + λ v

13 Sensitivity of Eigenvalue Problems Premultiply A v + Ev λv + λ v by v to obtain v A v + v Ev λv v + λv v Noting that v A v = (v A v) = v Av = λ v v = λv v leads to v Ev λv v, or λ = v Ev v v

14 Sensitivity of Eigenvalue Problems Finally, we obtain λ v Ev v 2 2 v 2 Ev 2 v 2 2 = E 2, so that λ E 2 We observe that perturbation bound does not depend on cond(v ) when we consider only an individual eigenvalue this individual eigenvalue perturbation bound is asymptotic; it is rigorous only in the limit that the perturbations 0

15 Algorithms for Eigenvalue Problems

16 Power Method

17 Power Method The power method is perhaps the simplest eigenvalue algorithm It finds the eigenvalue of A C n n with largest modulus 1: choose x 0 C n arbitrarily 2: for k = 1, 2,... do 3: x k = Ax k 1 4: end for Question: How does this algorithm work?

18 Power Method Assuming A is nondefective, then the eigenvectors v 1, v 2,..., v n provide a basis for C n Therefore there exist coefficients α i such that x 0 = n j=1 α jv j Then, we have x k = Ax k 1 = A 2 x k 2 = = A k x 0 = n n A k α j v j = α j A k v j = j=1 n α j λ k j v j j=1 j=1 j=1 n 1 [ ] k = λ k λj n α n v n + α j v j λ n

19 Power Method Then if λ n > λ j, 1 j < n, we see that x k λ k nα n v n as k This algorithm converges linearly: the error terms are scaled by a factor at most λ n 1 / λ n at each iteration Also, we see that the method converges faster if λ n is well-separated from the rest of the spectrum

20 Power Method However, in practice the exponential factor λ k n could cause overflow or underflow after relatively few iterations Therefore the standard form of the power method is actually the normalized power method 1: choose x 0 C n arbitrarily 2: for k = 1, 2,... do 3: y k = Ax k 1 4: x k = y k / y k 5: end for

21 Power Method Convergence analysis of the normalized power method is essentially the same as the un-normalized case Only difference is we now get an extra scaling factor, c k R, due to the normalization at each step n 1 [ ] k x k = c k λ k λj n α n v n + α j v j λ n j=1

22 Power Method This algorithm directly produces the eigenvector v n One way to recover λ n is to note that y k = Ax k 1 λ n x k 1 Hence we can compare an entry of y k and x k 1 to approximate λ n We also note two potential issues: 1. We require x 0 to have a nonzero component of v n 2. There may be more than one eigenvalue with maximum modulus

23 Power Method Issue 1: In practice, very unlikely that x 0 will be orthogonal to v n Even if x 0 v n = 0, rounding error will introduce a component of v n during the power iterations Issue 2: We cannot ignore the possibility that there is more than one max. eigenvalue In this case x k would converge to a member of the corresponding eigenspace

24 Power Method An important idea in eigenvalue computations is to consider the shifted matrix A σi, for σ R We see that (A σi)v i = (λ i σ)v i and hence the spectrum of A σi is shifted by σ, and the eigenvectors are the same For example, if all the eigenvalues are real, a shift can be used with the power method to converge to λ 1 instead of λ n

25 Power Method Python example: Consider power method and shifted power method for [ ] 4 1 A =, 1 2 which has eigenvalues λ 1 = , λ 2 =

26 Inverse Iteration

27 Inverse Iteration The eigenvalues of A 1 are the reciprocals of the eigenvalues of A, since Av = λv A 1 v = 1 λ v Question: What happens if we apply the power method to A 1?

28 Inverse Iteration Answer: We converge to the largest (in modulus) eigenvalue of A 1, which is 1/λ 1 (recall that λ 1 is the smallest eigenvalue of A) This is called inverse iteration 1: choose x 0 C n arbitrarily 2: for k = 1, 2,... do 3: solve Ay k = x k 1 for y k 4: x k = y k / y k 5: end for

29 Inverse Iteration Hence inverse iteration gives λ 1 without requiring a shift This is helpful since it may be difficult to determine what shift is required to get λ 1 in the power method Question: What happens if we apply inverse iteration to the shifted matrix A σi?

30 Inverse Iteration The smallest eigenvalue of A σi is (λ i σ), where i = arg min λ i σ, i=1,2,...,n and hence... Answer: We converge to λ = 1/(λ i σ), then recover λ i via λ i = 1 λ + σ Inverse iteration with shift allows us to find the eigenvalue closest to σ

31 Rayleigh Quotient Iteration

32 Rayleigh Quotient For the remainder of this section (Rayleigh Quotient Iteration, QR Algorithm) we will assume that A R n n is real and symmetric 3 The Rayleigh quotient is defined as r(x) x T Ax x T x If (λ, v) R R n is an eigenpair, then r(v) = v T Av v T v = λv T v v T v = λ 3 Much of the material generalizes to complex non-hermitian matrices, but symmetric case is simpler

33 Rayleigh Quotient Theorem: Suppose A R n n is a symmetric matrix, then for any x R n we have λ 1 r(x) λ n Proof: We write x as a linear combination of (orthogonal) eigenvectors x = n j=1 α jv j, and the lower bound follows from r(x) = x T n Ax x T x = j=1 λ jαj 2 n j=1 α2 j n j=1 λ α2 j 1 n j=1 α2 j = λ 1 The proof of the upper bound r(x) λ n is analogous

34 Rayleigh Quotient Corollary: A symmetric matrix A R n n is positive definite if and only if all of its eigenvalues are positive Proof: ( ) Suppose A is symmetric positive definite (SPD), then for any nonzero x R n, we have x T Ax > 0 and hence λ 1 = r(v 1 ) = v T 1 Av 1 v T 1 v 1 > 0 ( ) Suppose A has positive eigenvalues, then for any nonzero x R n x T Ax = r(x)(x T x) λ 1 x 2 2 > 0

35 Rayleigh Quotient But also, if x is an approximate eigenvector, then r(x) gives us a good approximation to the eigenvalue This is because estimation of an eigenvalue from an approximate eigenvector is an n 1 linear least squares problem: xλ Ax x R n is our tall thin matrix and Ax R n is our right-hand side Hence the normal equation for xλ Ax yields the Rayleigh quotient, i.e. x T xλ = x T Ax

36 Rayleigh Quotient Question: How accurate is the Rayleigh quotient approximation to an eigenvalue? Let s consider r as a function of x, so r : R n R r(x) x j = x j (x T Ax) x T x (x T Ax) x j (x T x) (x T x) 2 = 2(Ax) j x T (x T Ax)2x j x (x T x) 2 2 = x T x (Ax r(x)x) j (Note that the second equation relies on the symmetry of A)

37 Rayleigh Quotient Therefore r(x) = 2 x T (Ax r(x)x) x For an eigenpair (λ, v) we have r(v) = λ and hence r(v) = 2 v T (Av λv) = 0 v This shows that eigenvectors of A are stationary points of r

38 Rayleigh Quotient Suppose (λ, v) is an eigenpair of A, and let us consider a Taylor expansion of r(x) about v: r(x) = r(v) + r(v) T (x v) (x v)t H r (v)(x v) + H.O.T. = r(v) (x v)t H r (v)(x v) + H.O.T. Hence as x v the error in a Rayleigh quotient approximation is r(x) λ = O( x v 2 2) That is, the Rayleigh quotient approx. to an eigenvalue squares the error in a corresponding eigenvector approx.

39 Rayleigh Quotient Iteration The Rayleigh quotient gives us an eigenvalue estimate from an eigenvector estimate Inverse iteration gives us an eigenvector estimate from an eigenvalue estimate It is natural to combine the two, this yields the Rayleigh quotient iteration 1: choose x 0 R n arbitrarily 2: for k = 1, 2,... do 3: σ k = x T k 1 Ax k 1/x T k 1 x k 1 4: solve (A σ k I)y k = x k 1 for y k 5: x k = y k / y k 6: end for

40 Rayleigh Quotient Iteration Suppose, at step k, we have x k 1 v ɛ Then, from the Rayleigh quotient in line 3 of the algorithm, we have σ k λ = O(ɛ 2 ) In lines 4 and 5 of the algorithm, we then perform an inverse iteration with shift σ k to get x k Recall the eigenvector error in one inverse iteration step is scaled by ratio of second largest to largest eigenvalues of (A σ k I) 1

41 Rayleigh Quotient Iteration Let λ be the closest eigenvalue of A to σ k, then the magnitude of largest eigenvalue of (A σ k I) 1 is 1/ σ k λ The second largest eigenvalue magnitude is 1/ σ k ˆλ, where ˆλ is the eigenvalue of A second closest to σ k Hence at each inverse iteration step, the error is reduced by a factor σ k λ σ k ˆλ = σ k λ (σ k λ) + (λ ˆλ) const. σ k λ as σ k λ Therefore, we obtain cubic convergence as k : x k v (const. σ k λ ) x k 1 v = O(ɛ 3 )

42 Rayleigh Quotient Iteration A drawback of Rayleigh iteration: we can t just LU factorize A σ k I once since the shift changes each step Also, it s harder to pick out specific parts of the spectrum with Rayleigh quotient iteration since σ k can change unpredictably Python demo: Rayleigh iteration to compute an eigenpair of A =

43 QR Algorithm

44 The QR Algorithm The QR algorithm for computing eigenvalues is one of the best known algorithms in Numerical Analysis 4 It was developed independently in the late 1950s by John G.F. Francis (England) and Vera N. Kublanovskaya (USSR) The QR algorithm efficiently provides approximations for all eigenvalues/eigenvectors of a matrix We will consider what happens when we apply the power method to a set of vectors this will then motivate the QR algorithm 4 Recall that here we focus on the case in which A R n n is symmetric

45 The QR Algorithm Let x (0) 1,..., x p (0) denote p linearly independent starting vectors, and suppose we store these vectors in the columns of X 0 We can apply the power method to these vectors to obtain the following algorithm: 1: choose an n p matrix X 0 arbitrarily 2: for k = 1, 2,... do 3: X k = AX k 1 4: end for

46 The QR Algorithm From our analysis of the power method, we see that for each i = 1, 2,..., p: ) x (k) i = (λ k nα i,n v n + λ k n 1α i,n 1 v n λ k 1α i,1 v 1 n ( ) k n p = λ k λj ( ) k λj n p α i,jv j + α i,jv j j=n p+1 λ n p j=1 λ n p Then, if λ n p+1 > λ n p, the sum in green will decay compared to the sum in blue as k Hence the columns of X k will converge to a basis for span{v n p+1,..., v n }

47 The QR Algorithm However, this method doesn t provide a good basis: each column of X k will be very close to v n Therefore the columns of X k become very close to being linearly dependent We can resolve this issue by enforcing linear independence at each step

48 The QR Algorithm We orthonormalize the vectors after each iteration via a (reduced) QR factorization, to obtain the simultaneous iteration: 1: choose n p matrix Q 0 with orthonormal columns 2: for k = 1, 2,... do 3: X k = A ˆQ k 1 4: ˆQ k ˆR k = X k 5: end for The column spaces of ˆQ k and X k in line 4 are the same Hence columns of ˆQ k converge to orthonormal basis for span{v n p+1,..., v n }

49 The QR Algorithm In fact, we don t just get a basis for span{v n p+1,..., v n }, we get the eigenvectors themselves! Theorem: The columns of ˆQ k converge to the p dominant eigenvectors of A We will not discuss the full proof, but we note that this result is not surprising since: the eigenvectors of a symmetric matrix are orthogonal columns of ˆQ k converge to an orthogonal basis for span{v n p+1,..., v n } Simultaneous iteration approximates eigenvectors, we obtain eigenvalues from the Rayleigh quotient ˆQ T A ˆQ diag(λ 1,..., λ n )

50 The QR Algorithm With p = n, the simultaneous iteration will approximate all eigenpairs of A We now show a more convenient reorganization of the simultaneous iteration algorithm We shall require some extra notation: the Q and R matrices arising in the simultaneous iteration will be underlined Q k, R k (As we will see shortly, this is to distinguish between the matrices arising in the two different formulations...)

### Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.2: Fundamentals 2 / 31 Eigenvalues and Eigenvectors Eigenvalues and eigenvectors of

### EIGENVALUE PROBLEMS. Background on eigenvalues/ eigenvectors / decompositions. Perturbation analysis, condition numbers.. EIGENVALUE PROBLEMS Background on eigenvalues/ eigenvectors / decompositions Perturbation analysis, condition numbers.. Power method The QR algorithm Practical QR algorithms: use of Hessenberg form and

### Eigenvalue and Eigenvector Problems Eigenvalue and Eigenvector Problems An attempt to introduce eigenproblems Radu Trîmbiţaş Babeş-Bolyai University April 8, 2009 Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems

### AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Eigenvalue Problems; Similarity Transformations Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 18 Eigenvalue

### Last Time. Social Network Graphs Betweenness. Graph Laplacian. Girvan-Newman Algorithm. Spectral Bisection Eigenvalue Problems Last Time Social Network Graphs Betweenness Girvan-Newman Algorithm Graph Laplacian Spectral Bisection λ 2, w 2 Today Small deviation into eigenvalue problems Formulation Standard eigenvalue

### Bindel, Fall 2016 Matrix Computations (CS 6210) Notes for 1 Power iteration Notes for 2016-10-17 In most introductory linear algebra classes, one computes eigenvalues as roots of a characteristic polynomial. For most problems, this is a bad idea: the roots of

### Math 405: Numerical Methods for Differential Equations 2016 W1 Topics 10: Matrix Eigenvalues and the Symmetric QR Algorithm Math 405: Numerical Methods for Differential Equations 2016 W1 Topics 10: Matrix Eigenvalues and the Symmetric QR Algorithm References: Trefethen & Bau textbook Eigenvalue problem: given a matrix A, find

### Eigenvalues and Eigenvectors Chapter 1 Eigenvalues and Eigenvectors Among problems in numerical linear algebra, the determination of the eigenvalues and eigenvectors of matrices is second in importance only to the solution of linear

### Eigenvalue Problems. Eigenvalue problems occur in many areas of science and engineering, such as structural analysis Eigenvalue Problems Eigenvalue problems occur in many areas of science and engineering, such as structural analysis Eigenvalues also important in analyzing numerical methods Theory and algorithms apply

### Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 4 Eigenvalue Problems Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give

### AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Rayleigh Quotient Iteration Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Solving Eigenvalue Problems All

### The Eigenvalue Problem: Perturbation Theory Jim Lambers MAT 610 Summer Session 2009-10 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just

### Notes on Eigenvalues, Singular Values and QR Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square

### 33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM 33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the

### Computational Methods. Eigenvalues and Singular Values Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

### AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis

### MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. Diagonalization Let L be a linear operator on a finite-dimensional vector space V. Then the following conditions are equivalent:

### Numerical Analysis Lecture Notes Numerical Analysis Lecture Notes Peter J Olver 8 Numerical Computation of Eigenvalues In this part, we discuss some practical methods for computing eigenvalues and eigenvectors of matrices Needless to

### Announcements Monday, November 06 Announcements Monday, November 06 This week s quiz: covers Sections 5 and 52 Midterm 3, on November 7th (next Friday) Exam covers: Sections 3,32,5,52,53 and 55 Section 53 Diagonalization Motivation: Difference

### MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization. MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization. Eigenvalues and eigenvectors of an operator Definition. Let V be a vector space and L : V V be a linear operator. A number λ

### arxiv: v1 [math.na] 5 May 2011 ITERATIVE METHODS FOR COMPUTING EIGENVALUES AND EIGENVECTORS MAYSUM PANJU arxiv:1105.1185v1 [math.na] 5 May 2011 Abstract. We examine some numerical iterative methods for computing the eigenvalues and

### Lecture 4 Eigenvalue problems Lecture 4 Eigenvalue problems Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn

### DIAGONALIZATION. In order to see the implications of this definition, let us consider the following example Example 1. Consider the matrix DIAGONALIZATION Definition We say that a matrix A of size n n is diagonalizable if there is a basis of R n consisting of eigenvectors of A ie if there are n linearly independent vectors v v n such that

### Lecture 15, 16: Diagonalization Lecture 15, 16: Diagonalization Motivation: Eigenvalues and Eigenvectors are easy to compute for diagonal matrices. Hence, we would like (if possible) to convert matrix A into a diagonal matrix. Suppose

### EIGENVALUE PROBLEMS. EIGENVALUE PROBLEMS p. 1/4 EIGENVALUE PROBLEMS EIGENVALUE PROBLEMS p. 1/4 EIGENVALUE PROBLEMS p. 2/4 Eigenvalues and eigenvectors Let A C n n. Suppose Ax = λx, x 0, then x is a (right) eigenvector of A, corresponding to the eigenvalue

### 1. In this problem, if the statement is always true, circle T; otherwise, circle F. Math 1553, Extra Practice for Midterm 3 (sections 45-65) Solutions 1 In this problem, if the statement is always true, circle T; otherwise, circle F a) T F If A is a square matrix and the homogeneous equation

### Applied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization

### Throughout these notes we assume V, W are finite dimensional inner product spaces over C. Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal

### Eigenproblems II: Computation Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Eigenproblems II: Computation 1 / 31 Setup A

### Numerical Methods I Eigenvalue Problems Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 2nd, 2014 A. Donev (Courant Institute) Lecture

### Computational Methods CMSC/AMSC/MAPL 460. Eigenvalues and Eigenvectors. Ramani Duraiswami, Dept. of Computer Science Computational Methods CMSC/AMSC/MAPL 460 Eigenvalues and Eigenvectors Ramani Duraiswami, Dept. of Computer Science Eigen Values of a Matrix Recap: A N N matrix A has an eigenvector x (non-zero) with corresponding

### Computation of eigenvalues and singular values Recall that your solutions to these questions will not be collected or evaluated. Math 504, Homework 5 Computation of eigenvalues and singular values Recall that your solutions to these questions will not be collected or evaluated 1 Find the eigenvalues and the associated eigenspaces

### Eigenvalues and eigenvectors Chapter 6 Eigenvalues and eigenvectors An eigenvalue of a square matrix represents the linear operator as a scaling of the associated eigenvector, and the action of certain matrices on general vectors

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Reduction to Hessenberg and Tridiagonal Forms; Rayleigh Quotient Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical

### MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v

### 22.4. Numerical Determination of Eigenvalues and Eigenvectors. Introduction. Prerequisites. Learning Outcomes Numerical Determination of Eigenvalues and Eigenvectors 22.4 Introduction In Section 22. it was shown how to obtain eigenvalues and eigenvectors for low order matrices, 2 2 and. This involved firstly solving

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### 18.06 Problem Set 8 Solution Due Wednesday, 22 April 2009 at 4 pm in Total: 160 points. 86 Problem Set 8 Solution Due Wednesday, April 9 at 4 pm in -6 Total: 6 points Problem : If A is real-symmetric, it has real eigenvalues What can you say about the eigenvalues if A is real and anti-symmetric

### Announcements Wednesday, November 7 Announcements Wednesday, November 7 The third midterm is on Friday, November 6 That is one week from this Friday The exam covers 45, 5, 52 53, 6, 62, 64, 65 (through today s material) WeBWorK 6, 62 are

### Eigenvalues, Eigenvectors, and Diagonalization Week12 Eigenvalues, Eigenvectors, and Diagonalization 12.1 Opening Remarks 12.1.1 Predicting the Weather, Again Let us revisit the example from Week 4, in which we had a simple model for predicting the

### Math 471 (Numerical methods) Chapter 4. Eigenvalues and Eigenvectors Math 471 (Numerical methods) Chapter 4. Eigenvalues and Eigenvectors Overlap 4.1 4.3 of Bradie 4.0 Review on eigenvalues and eigenvectors Definition. A nonzero vector v is called an eigenvector of A if

### Linear algebra & Numerical Analysis Linear algebra & Numerical Analysis Eigenvalues and Eigenvectors Marta Jarošová http://homel.vsb.cz/~dom033/ Outline Methods computing all eigenvalues Characteristic polynomial Jacobi method for symmetric

### Math 240 Calculus III Generalized Calculus III Summer 2015, Session II Thursday, July 23, 2015 Agenda 1. 2. 3. 4. Motivation Defective matrices cannot be diagonalized because they do not possess enough eigenvectors to make

### Lecture 10 - Eigenvalues problem Lecture 10 - Eigenvalues problem Department of Computer Science University of Houston February 28, 2008 1 Lecture 10 - Eigenvalues problem Introduction Eigenvalue problems form an important class of problems

### Lecture 8 : Eigenvalues and Eigenvectors CPS290: Algorithmic Foundations of Data Science February 24, 2017 Lecture 8 : Eigenvalues and Eigenvectors Lecturer: Kamesh Munagala Scribe: Kamesh Munagala Hermitian Matrices It is simpler to begin with

### Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

### 1 Last time: least-squares problems MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that

### Eigenvalues, Eigenvectors, and Diagonalization Week12 Eigenvalues, Eigenvectors, and Diagonalization 12.1 Opening Remarks 12.1.1 Predicting the Weather, Again View at edx Let us revisit the example from Week 4, in which we had a simple model for predicting

### MATH 583A REVIEW SESSION #1 MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),

### 4.2. ORTHOGONALITY 161 4.2. ORTHOGONALITY 161 Definition 4.2.9 An affine space (E, E ) is a Euclidean affine space iff its underlying vector space E is a Euclidean vector space. Given any two points a, b E, we define the distance

### AM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized

### Announcements Wednesday, November 7 Announcements Wednesday, November 7 The third midterm is on Friday, November 16 That is one week from this Friday The exam covers 45, 51, 52 53, 61, 62, 64, 65 (through today s material) WeBWorK 61, 62

### MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. Adjoint operator and adjoint matrix Given a linear operator L on an inner product space V, the adjoint of L is a transformation

### Announcements Monday, November 13 Announcements Monday, November 13 The third midterm is on this Friday, November 17 The exam covers 31, 32, 51, 52, 53, and 55 About half the problems will be conceptual, and the other half computational

### ft-uiowa-math2550 Assignment NOTRequiredJustHWformatOfQuizReviewForExam3part2 due 12/31/2014 at 07:10pm CST me me ft-uiowa-math2550 Assignment NOTRequiredJustHWformatOfQuizReviewForExam3part2 due 12/31/2014 at 07:10pm CST 1. (1 pt) local/library/ui/eigentf.pg A is n n an matrices.. There are an infinite number

### Linear Algebra Exercises 9. 8.03 Linear Algebra Exercises 9A. Matrix Multiplication, Rank, Echelon Form 9A-. Which of the following matrices is in row-echelon form? 2 0 0 5 0 (i) (ii) (iii) (iv) 0 0 0 (v) [ 0 ] 0 0 0 0 0 0 0 9A-2.

### LU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization

### 235 Final exam review questions 5 Final exam review questions Paul Hacking December 4, 0 () Let A be an n n matrix and T : R n R n, T (x) = Ax the linear transformation with matrix A. What does it mean to say that a vector v R n is an

### Linear Algebra in Actuarial Science: Slides to the lecture Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations

### 2 Eigenvectors and Eigenvalues in abstract spaces. MA322 Sathaye Notes on Eigenvalues Spring 27 Introduction In these notes, we start with the definition of eigenvectors in abstract vector spaces and follow with the more common definition of eigenvectors

### Chapter 5. Eigenvalues and Eigenvectors Chapter 5 Eigenvalues and Eigenvectors Section 5. Eigenvectors and Eigenvalues Motivation: Difference equations A Biology Question How to predict a population of rabbits with given dynamics:. half of the

### Quadratic forms. Here. Thus symmetric matrices are diagonalizable, and the diagonalization can be performed by means of an orthogonal matrix. Quadratic forms 1. Symmetric matrices An n n matrix (a ij ) n ij=1 with entries on R is called symmetric if A T, that is, if a ij = a ji for all 1 i, j n. We denote by S n (R) the set of all n n symmetric

### AM 205: lecture 6. Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization AM 205: lecture 6 Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization Unit II: Numerical Linear Algebra Motivation Almost everything in Scientific Computing

### EECS 275 Matrix Computation EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 17 1 / 26 Overview

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Chapters 5 & 6: Theory Review: Solutions Math 308 F Spring 2015 Chapters 5 & 6: Theory Review: Solutions Math 308 F Spring 205. If A is a 3 3 triangular matrix, explain why det(a) is equal to the product of entries on the diagonal. If A is a lower triangular or diagonal

### MATH 304 Linear Algebra Lecture 34: Review for Test 2. MATH 304 Linear Algebra Lecture 34: Review for Test 2. Topics for Test 2 Linear transformations (Leon 4.1 4.3) Matrix transformations Matrix of a linear mapping Similar matrices Orthogonality (Leon 5.1

### MA 265 FINAL EXAM Fall 2012 MA 265 FINAL EXAM Fall 22 NAME: INSTRUCTOR S NAME:. There are a total of 25 problems. You should show work on the exam sheet, and pencil in the correct answer on the scantron. 2. No books, notes, or calculators

### AM 205: lecture 6. Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization AM 205: lecture 6 Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization Unit II: Numerical Linear Algebra Motivation Almost everything in Scientific Computing

### Math 110 Linear Algebra Midterm 2 Review October 28, 2017 Math 11 Linear Algebra Midterm Review October 8, 17 Material Material covered on the midterm includes: All lectures from Thursday, Sept. 1st to Tuesday, Oct. 4th Homeworks 9 to 17 Quizzes 5 to 9 Sections

### The QR Decomposition The QR Decomposition We have seen one major decomposition of a matrix which is A = LU (and its variants) or more generally PA = LU for a permutation matrix P. This was valid for a square matrix and aided

### Math 3191 Applied Linear Algebra Math 9 Applied Linear Algebra Lecture 9: Diagonalization Stephen Billups University of Colorado at Denver Math 9Applied Linear Algebra p./9 Section. Diagonalization The goal here is to develop a useful

### Cheat Sheet for MATH461 Cheat Sheet for MATH46 Here is the stuff you really need to remember for the exams Linear systems Ax = b Problem: We consider a linear system of m equations for n unknowns x,,x n : For a given matrix A

### Jim Lambers MAT 610 Summer Session Lecture 2 Notes Jim Lambers MAT 610 Summer Session 2009-10 Lecture 2 Notes These notes correspond to Sections 2.2-2.4 in the text. Vector Norms Given vectors x and y of length one, which are simply scalars x and y, the

### Chapter 7. Canonical Forms. 7.1 Eigenvalues and Eigenvectors Chapter 7 Canonical Forms 7.1 Eigenvalues and Eigenvectors Definition 7.1.1. Let V be a vector space over the field F and let T be a linear operator on V. An eigenvalue of T is a scalar λ F such that there

### THE MATRIX EIGENVALUE PROBLEM THE MATRIX EIGENVALUE PROBLEM Find scalars λ and vectors x 0forwhich Ax = λx The form of the matrix affects the way in which we solve this problem, and we also have variety as to what is to be found. A

### Jordan Canonical Form Homework Solutions Jordan Canonical Form Homework Solutions For each of the following, put the matrix in Jordan canonical form and find the matrix S such that S AS = J. [ ]. A = A λi = λ λ = ( λ) = λ λ = λ =, Since we have

### Numerical Solution of Linear Eigenvalue Problems Numerical Solution of Linear Eigenvalue Problems Jessica Bosch and Chen Greif Abstract We review numerical methods for computing eigenvalues of matrices We start by considering the computation of the dominant

### Computing Eigenvalues and/or Eigenvectors;Part 1, Generalities and symmetric matrices Computing Eigenvalues and/or Eigenvectors;Part 1, Generalities and symmetric matrices Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo November 9, 2008 Today

### EE731 Lecture Notes: Matrix Computations for Signal Processing EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition

### Review problems for MA 54, Fall 2004. Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on

### Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017 Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

### In this section again we shall assume that the matrix A is m m, real and symmetric. 84 3. The QR algorithm without shifts See Chapter 28 of the textbook In this section again we shall assume that the matrix A is m m, real and symmetric. 3.1. Simultaneous Iterations algorithm Suppose we

### Linear Algebra Review Chapter 1 Linear Algebra Review It is assumed that you have had a course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc. I will review some of these terms here, but quite

### MIDTERM. b) [2 points] Compute the LU Decomposition A = LU or explain why one does not exist. MAE 9A / FALL 3 Maurício de Oliveira MIDTERM Instructions: You have 75 minutes This exam is open notes, books No computers, calculators, phones, etc There are 3 questions for a total of 45 points and bonus

### Computing Eigenvalues and/or Eigenvectors;Part 1, Generalities and symmetric matrices Computing Eigenvalues and/or Eigenvectors;Part 1, Generalities and symmetric matrices Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo November 8, 2009 Today

### j=1 u 1jv 1j. 1/ 2 Lemma 1. An orthogonal set of vectors must be linearly independent. Lecture Notes: Orthogonal and Symmetric Matrices Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Orthogonal Matrix Definition. Let u = [u

### Linear Algebra: Matrix Eigenvalue Problems CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

### Linear Algebra. Carleton DeTar February 27, 2017 Linear Algebra Carleton DeTar detar@physics.utah.edu February 27, 2017 This document provides some background for various course topics in linear algebra: solving linear systems, determinants, and finding

### On the Modification of an Eigenvalue Problem that Preserves an Eigenspace Purdue University Purdue e-pubs Department of Computer Science Technical Reports Department of Computer Science 2009 On the Modification of an Eigenvalue Problem that Preserves an Eigenspace Maxim Maumov

### Math 205, Summer I, Week 4b: Math 205, Summer I, 2016 Week 4b: Chapter 5, Sections 6, 7 and 8 (5.5 is NOT on the syllabus) 5.6 Eigenvalues and Eigenvectors 5.7 Eigenspaces, nondefective matrices 5.8 Diagonalization [*** See next slide

### (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax = . (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)

### ECS231 Handout Subspace projection methods for Solving Large-Scale Eigenvalue Problems. Part I: Review of basic theory of eigenvalue problems ECS231 Handout Subspace projection methods for Solving Large-Scale Eigenvalue Problems Part I: Review of basic theory of eigenvalue problems 1. Let A C n n. (a) A scalar λ is an eigenvalue of an n n A

### Eigenvalues, Eigenvectors, and Diagonalization Math 240 TA: Shuyi Weng Winter 207 February 23, 207 Eigenvalues, Eigenvectors, and Diagonalization The concepts of eigenvalues, eigenvectors, and diagonalization are best studied with examples. We will Review of Linear Algebra Definitions, Change of Basis, Trace, Spectral Theorem Steven J. Miller June 19, 2004 Abstract Matrices can be thought of as rectangular (often square) arrays of numbers, or as BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,