The Singular Value Decomposition and Least Squares Problems
|
|
- Karin Norris
- 5 years ago
- Views:
Transcription
1 The Singular Value Decomposition and Least Squares Problems Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 27, 2009
2 Applications of SVD solving over-determined equations statistics, principal component analysis numerical determination of the rank of a matrix search engines (Google,...) theory of matrices and lots of other applications...
3 Diagonalization A square matrix A can be diagonalized by a unitary similarity transformation if and only if it is normal. U H AU = D := diag(λ 1,..., λ n ) or A = UDU H. If U = [u 1,..., u n ] then Au j = λ j u j, u H j u k = δ jk. If A is real and symmetric then U and D are real. Today: Any matrix, even a rectangular one, can be diagonalized provided we allow two different unitary matrices. A = UΣV H is called a Singular Value Decomposition (SVD) if Σ is a diagonal matrix of the same dimension as A, and U and V are square and unitary. The diagonal entries of Σ are called the singular values of the matrix.
4 Hermitian matrix Recall Theorem Suppose A C n,n is Hermitian. Then A has real eigenvalues λ 1,..., λ n. Moreover, there is a unitary matrix U C n,n such that U H AU = diag(λ 1,..., λ n ). For the columns {u 1,..., u n } of U we have Au j = λ j u j for j = 1,..., n. Thus {u 1,..., u n } are orthonormal eigenvectors of A.
5 Eigenvalues of A H A Lemma Suppose m, n N and A C m,n. The matrix A H A has eigenpairs (λ j, v j ) for j = 1,..., n, where v H j v k = δ jk and λ 1 λ 2 λ n 0. Moreover σ j := λ j = Av j 2, for j = 1,..., n. (1) Proof: A H A C n,n is Hermitian. It has real eigenvalues λ j and orthonormal eigenvectors v j. Av j 2 2 = (Av j ) H Av j = v H j A H Av j = v H j λ j v j = λ j 0.
6 Definition (Singular Values) The positive square roots σ j := λ j, j = 1,..., n of the eigenvalues of A H A are called the singular values of A C m,n. σ 1 σ 2 σ r > 0 = σ r+1 = = σ n. [ ] Σ1 0 Σ := r,n r R 0 m r,r 0 m,n, m r,n r Σ 1 := diag(σ 1,..., σ r ), 0 k,l = [ ], the empty matrix, if k = 0 or l = 0. Σ = [σ 1 e 1,..., σ r e r, 0,..., 0]. We will show that r is the rank of A.
7 Examples [ ] [ ] A := 1, Σ = = Σ [ ] [ ] A := , Σ = 0 1 Σ1 2 0 =, Σ = ,2 0 1 [ ] [ ] [ ] A 1 := 1, Σ =, Σ = [ ] A = 1 1 Σ1 0 Σ =, Σ = [2]. 0 0
8 The Singular Value Decomposition Theorem (Existence of SVD) Let m, n N and suppose A C m,n has r nonzero singular values σ 1 σ r > 0 = σ r+1 = = σ n. Then A has the singular value decomposition A = UΣV H, where U C m,m and V C n,n are unitary, and [ Σ1 0 Σ = r,n r 0 m r,r 0 m r,n r ], Σ 1 = diag(σ 1,..., σ r ). (2) If A is real then A = UΣV T, where U R m,m and V R n,n are orthonormal, and Σ is given by (2).
9 A useful result From the eigenvectors of A H A we can derive orthonormal bases for the column space span(a) and null space ker(a) of A. Theorem Suppose A C m,n and let (σ 2 j, v j) for j = 1,..., n be orthonormal eigenpairs for A H A. Suppose r of the singular values are nonzero so that σ 1 σ r > 0 = σ r+1 = = σ n. (3) Then {Av 1,..., Av r } is an orthogonal basis for the column space of A and {v r+1,..., v n } is an orthonormal basis for the nullspace of A.
10 Part of Proof For j k (Av j ) H Av k = v H j A H Av k = v H j λ k v k = 0, and {Av 1,..., Av n } is an orthogonal set. σ j = Av j 2 (Av j 0 j {1,..., r}) But then span([av 1,..., Av r ]) span(a) and span([v r+1,..., v n ]) ker(a). For equality we need to show the opposite inclusions.
11 Outline of existence proof V := [v 1,..., v n ] C n,n where A H Av j = λ j v j and {v 1,..., v n } is orthonormal. σ j := λ j for j = 1,..., n. u j := Av j Av j 2 = 1 σ j Av j, for j = 1,..., r. U := [u 1,..., u m ] C m,m, where {u r+1,..., u m } is defined by extending {u 1,..., u r } to an orthonormal basis for C m. UΣ = U[σ 1 e 1,..., σ r e r, 0,..., 0] = [σ 1 u 1,..., σ r u r, 0,..., 0] = [Av 1,..., Av n ] = AV. Since V is unitary we find UΣV H = AVV H = A.
12 Uniqueness The singular values are unique The matrices U and V are in general not unique.
13 Examples [ ] [ ] [ ] [ ] = [ ] [ ] [ ] = [ ] = / 2 1/ = 1/ 2 1/ 2 0 [ ]
14 r < n < m Find the singular value decomposition of 1 1 A = [ ] 2 2 B := A T A = 2 2 [ ] [ ] 1 1 B = 4, B 1 1 [ ] 1 = 0 1 [ ] 1 1 u 1 = Av 1 /σ 1 = s 1 / 2, where s 1 = [1, 1, 0] T. Extend s 1 to a basis {s 1, s 2, s 3 } for R 3 Apply Gram-Schmidt to {s 1, s 2, s 3 }
15 Comments on computing the SVD The method we used to find the singular value decomposition in the previous example can be suitable for hand calculation with small matrices, but it is not appropriate as a general purpose numerical method. In particular, the Gram-Schmidt orthogonalization process which can be used to extend u 1,... u r to an orthonormal basis is not numerically stable. Forming A H A can lead to extra errors in the computation. State of the art computer implementations of the singular value decomposition uses an adapted version of the QR-algorithm where the matrix A H A is not formed. The QR-algorithm is discussed in Chapter 20.
16 SVD using Matlab [U,S,V]=svd(A), the singular value decomposition s=svd(a) the singular values [U,S,V]=svd(A,0), economy size, If if m > n then U C m,n, S R n,n, and V C n,n as before.
17 The Singular Value Factorization (SVF) SVD: A = UΣV H, U, V square. U = [u 1,..., u m ] = [U 1, U 2 ], U 1 R m,r, U 2 R m,m r, V = [v 1,..., v n ] = [V 1, V 2 ], V 1 R n,r, V 2 R n,n r, A = [ ] [ ] [ ] Σ U 1, U 1 0 V H V H = U 1 Σ 1 V H 1 2 singular value factorization.
18 Three forms of SVD m n m n n r r n V H Σ H 1 V 1 A = m U Σ = U 1 = σ 1 u 1 v 1 H +...+σr u r v r H A = UΣV H, SVD A = U 1 Σ 1 V H 1, SVF r = σ i u i v H i, outer product form. i=1
19 Normal and positive semidefinite matrices σ j = λ j if A H A = AA H. σ j = λ j if A is symmetric positive semidefinite. Spectral decomposition A = UDU H, D = diag(λ 1,..., λ n ) and U H U = I. A H A = UD H DU H, D H D = diag( λ 1 2,..., λ n 2 ) For a positive semi-definite matrix the factorization A = UDU H above is both an SVD and an SVF provided we have sorted the eigenvalues in nondecreasing order.
20 Geometric Interpretation y2 2 1 x2 2 u y1 1 2 u 2 Σ 1 Σ x1 1 2 Left: Unit circle S, right: the image AS. A := 1 [ ] R2,2., U = 1 [ ], AS = {x : Σ 1 U T x 2 2 = 1} = {(x 1, x 2 ) : ( 3 5 x x 2) 2 + ( 4 5 x x 2) 2 = 1} 9 1 AS is an ellipsoid The singular values give the length of the semi-axes. The semi-axes are along the left singular vectors.
21 Singular vectors The columns u 1,..., u m of U are called left singular vectors and the columns v 1,..., v n of V are called right singular vectors 1. AV 1 = U 1 Σ 1, or Av i = σ i u i for i = 1,..., r, 2. AV 2 = 0, or Av i = 0 for i = r + 1,..., n, 3. A H U 1 = V 1 Σ 1, or A H u i = σ i v i for i = 1,..., r, 4. A H U 2 = 0, or A H u i = 0 for i = r + 1,..., m. 1. U 1 is an orthonormal basis for span(a), 2. V 2 is an orthonormal basis for ker(a), 3. V 1 is an orthonormal basis for span(a H ), 4. U 2 is an orthonormal basis for ker(a H ). (4) (5)
22 SVD of A H A and AA H A = UΣV H = U 1 Σ 1 V H 1, (SVD and SVF) A H A = VΣ H ΣV H = V 1 Σ 2 1V H 1, (SVD and SVF) A H AV 1 = V 1 Σ 2 1, A H AV 2 = V 1 Σ 2 1V H 1 V 2 = 0, AA H = UΣΣ H U H = U 1 Σ 2 1U H 1, (SVD and SVF) AA H U 1 = U 1 Σ 2 1, AA H U 2 = 0,
23 Rank and nullity relations Corollary Suppose A C m,n. Then rank(a) + null(a) = n, rank(a) + null(a H ) = m, rank(a) = rank(a H ). Theorem For any A C m,n we have rank A = rank(a H A) = rank(aa H ), null(a H A) = null A,and null(aa H ) = null(a H ), span(a H A) = span A H and ker(a H A) = ker A.
24 The Pseudoinverse The pseudo-inverse of A C m,n is the matrix A C n,m given by A := V 1 Σ 1 1 UH 1, where A = U 1 Σ 1 V H 1 A. is the singular value factorization of A is independent of the factorization chosen to represent it. If A is square and nonsingular then A A = AA = I and A is the usual inverse of A. Any matrix has a pseudo-inverse, and so A is a generalization of the usual inverse.
25 How to find the pseudoinverse 1. Find the SVF of A 2. If A C m,n has rank n then A = (A H A) 1 A H. 3. If B C n,m satisfies ABA = A, BAB = B, (BA) H = BA, (AB) H = AB, then B = A 4. Use MATLAB: B = pinv(a)
26 Example [ ] A = = [ ] [ ] /2 0 0 A = [ ] A = B = ABA = A, BAB = B, (BA) H = BA, (AB) H = AB, [ ] A T A = 1,, ] A = (A T A) 1 A T = [ [ 14 4 ]
27 Theory; Direct sum and Orthogonal Sum Suppose S and T are subspaces of a vector space (V, F). We define Sum: X := S + T := {s + t : s S and t T }; Direct Sum: If S T = {0}, then S T := S + T. Orthogonal Sum: Suppose (V, F,, ) is an inner product space. Then S T is an orthogonal sum if s, t = 0 for all s S and all t T. span(a) ker(a H ) is an orthogonal sum with respect to the usual inner product s, t := s H t. For if y = Ax span(a) and z ker(a H ) then y H z = (Ax) H z = x H (A H z) = 0. orthogonal complement: T = S := {x X : s, x = 0 for all s S}.
28 Basic facts Suppose S and T are subspaces of a vector space (V, F). Then S + T = T + S and S + T is a subspace of V. dim(s + T ) = dim S + dim T dim(s T ) dim(s T ) = dim S + dim T. C m = span(a) ker(a H ) Every v S T can be decomposed uniquely as v = s + t, where s S and t T. If S T is an orthogonal sum then s is called the orthogonal projection of v into S. v t S s
29 Orthogonal Projections The singular value decomposition and the pseudo-inverse can be used to compute orthogonal projections into the subspaces span(a) and ker(a H ). Recall that if A = UΣV H is the SVD of A and U = [U 1, U 2 ] as before then U 1 is an orthonormal basis for span(a) and U 2 is an orthonormal basis for ker(a H ). Let b C m. Then b = UU H b = [U 1 U 2 ] [ ] U H 1 U H b = U 1 U H 1 b+u 2 U H 2 b =: b 1 +b 2. 2 b 1 := U 1 (U H 1 b) span(a) is the orthogonal projection into span(a). b 1 = AA b b 2 = U 2 (U H 2 b) ker(a H ) is the orthogonal projection into ker(a H ). b 2 = (I AA )b
30 Example 1 0 The singular value decomposition of A = 0 1 is 0 0 A = I 3 AI 2. [ ] [ ] A = I 2 I = [ ] AA = = I 3 AA = If b = [b 1, b 2, b 3 ] T, then b 1 = AA b = [b 1, b 2, 0] and b 2 = (I 3 AA )b = [0, 0, b 3 ] T.
31 Minmax and Maxmin Theorems R(x) = R A (x) := xh Ax x H x Theorem (The Courant-Fischer Theorem) Suppose A C n,n is Hermitian with eigenvalues λ 1, λ 2,..., λ n ordered so that λ 1 λ n. Then for k = 1,..., n λ k = min max R(x) = dim(s)=n k+1 x S x 0 max min R(x). (6) dim(s)=k x S x 0
32 Minmax and Maxmin Theorems for singular values Theorem (The Courant-Fischer Theorem for Singular Values) Suppose A C m,n has singular values σ 1, σ 2,..., σ n ordered so that σ 1 σ n. Then for k = 1,..., n Proof σ k = min max Ax 2 = max dim(s)=n k+1 x S x 2 dim(s)=k x 0 min x S x 0 Ax 2 x 2. Ax 2 2 x 2 2 = Ax, Ax x, x = x, AH Ax x, x = R A H A (x).
33 The largest and smallest singular value Ax 2 σ 1 = max x C n x 2 x 0 Ax 2 σ n = min x C n x 2 x 0 = max Ax 2, x C n x 2 =1 = min Ax 2. x C n x 2 =1
34 Hoffman-Wielandt Theorem Theorem (Eigenvalues) Suppose A, B C n,n are both Hermitian matrices with eigenvalues λ 1 λ n and µ 1 µ n, respectively. Then n µ j λ j 2 A B 2 F := j=1 n i=1 n a ij b ij 2. j=1
35 Hoffman-Wielandt Theorem Theorem (Singular values) For any m, n N and A, B C m,n we have n β j α j 2 A B 2 F, j=1 where α 1 α n and β 1 β n are the singular values of A and B, respectively.
36 Proof1 C := [ 0 ] A A H 0 and D := C H = C and D H = D. [ ] 0 B B H C 0 m+n,m+n. If C and D has eigenvalues λ 1 λ m+n and µ 1 µ m+n, respectively then m+n λ j µ j 2 C D 2 F. j=1 Suppose A has rank r and SVD UΣV H. Av i = α i u i, A H u i = α i v i for i = 1,..., r A H u i = 0 for i = r + 1,..., m, Av i = 0 for i = r + 1,..., n
37 Proof2 [ ] [ ] [ ] [ ] [ ] 0 A ui Avi αi u A H = 0 v i A H = i ui = α u i α i v i, i = 1,..., r, i v i [ ] [ ] [ ] [ ] [ ] 0 A ui Avi αi u A H = 0 v i A H = i ui = α u i α i v i, i = 1,..., r, i v i [ ] [ ] [ ] [ ] [ ] 0 A ui 0 0 ui A H = 0 0 A H = = 0, i = r + 1,..., m, u i 0 0 [ ] [ ] [ ] [ ] [ ] 0 A 0 Avi 0 0 A H = = = 0, i = r + 1,..., n v i v i
38 Proof3 Thus C has the 2r eigenvalues α 1, α 1,..., α r, α r and m + n 2r additional zero eigenvalues. Similarly, if B has rank s then D has the 2s eigenvalues β 1, β 1,..., β s, β s and m + n 2s additional zero eigenvalues. t := max(r, s). λ 1 λ m+n = α 1 α t 0 = = 0 α t α 1, µ 1 µ m+n = β 1 β t 0 = = 0 β t β 1.
39 Proof4 m+n λ j µ j 2 = j=1 t t t α i β i 2 + α i +β i 2 = 2 α i β i 2 i=1 i=1 i=1 [ ] C D 2 0 A B F = A H B H 0 = B A 2 F + (B A) H 2 F = 2 B A 2 F. 2 F 1 2 m+n j=1 λ j µ j 2 = t i=1 α i β i C D 2 F = B A 2 F. Since t n and α i = β i = 0 for i = t + 1,..., n we obtain the result.
Least Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo
Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 26, 2010 Linear system Linear system Ax = b, A C m,n, b C m, x C n. under-determined
More informationThe Singular Value Decomposition
CHAPTER 6 The Singular Value Decomposition Exercise 67: SVD examples (a) For A =[, 4] T we find a matrixa T A =5,whichhastheeigenvalue =5 Thisprovidesuswiththesingularvalue =+ p =5forA Hence the matrix
More informationSingular Value Decomposition
Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =
More informationOrthonormal Transformations and Least Squares
Orthonormal Transformations and Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 30, 2009 Applications of Qx with Q T Q = I 1. solving
More information18.06 Problem Set 10 - Solutions Due Thursday, 29 November 2007 at 4 pm in
86 Problem Set - Solutions Due Thursday, 29 November 27 at 4 pm in 2-6 Problem : (5=5+5+5) Take any matrix A of the form A = B H CB, where B has full column rank and C is Hermitian and positive-definite
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More informationThroughout these notes we assume V, W are finite dimensional inner product spaces over C.
Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationMaths for Signals and Systems Linear Algebra in Engineering
Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More information(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =
. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationSingular Value Decomposition (SVD)
School of Computing National University of Singapore CS CS524 Theoretical Foundations of Multimedia More Linear Algebra Singular Value Decomposition (SVD) The highpoint of linear algebra Gilbert Strang
More informationMATH36001 Generalized Inverses and the SVD 2015
MATH36001 Generalized Inverses and the SVD 201 1 Generalized Inverses of Matrices A matrix has an inverse only if it is square and nonsingular. However there are theoretical and practical applications
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T
More informationlinearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice
3. Eigenvalues and Eigenvectors, Spectral Representation 3.. Eigenvalues and Eigenvectors A vector ' is eigenvector of a matrix K, if K' is parallel to ' and ' 6, i.e., K' k' k is the eigenvalue. If is
More informationLinear Least Squares. Using SVD Decomposition.
Linear Least Squares. Using SVD Decomposition. Dmitriy Leykekhman Spring 2011 Goals SVD-decomposition. Solving LLS with SVD-decomposition. D. Leykekhman Linear Least Squares 1 SVD Decomposition. For any
More informationMath 408 Advanced Linear Algebra
Math 408 Advanced Linear Algebra Chi-Kwong Li Chapter 4 Hermitian and symmetric matrices Basic properties Theorem Let A M n. The following are equivalent. Remark (a) A is Hermitian, i.e., A = A. (b) x
More informationNotes on singular value decomposition for Math 54. Recall that if A is a symmetric n n matrix, then A has real eigenvalues A = P DP 1 A = P DP T.
Notes on singular value decomposition for Math 54 Recall that if A is a symmetric n n matrix, then A has real eigenvalues λ 1,, λ n (possibly repeated), and R n has an orthonormal basis v 1,, v n, where
More informationLecture notes: Applied linear algebra Part 1. Version 2
Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and
More informationChapter 7: Symmetric Matrices and Quadratic Forms
Chapter 7: Symmetric Matrices and Quadratic Forms (Last Updated: December, 06) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved
More informationSingular Value Decomposition
Singular Value Decomposition CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Singular Value Decomposition 1 / 35 Understanding
More informationSINGULAR VALUE DECOMPOSITION
DEPARMEN OF MAHEMAICS ECHNICAL REPOR SINGULAR VALUE DECOMPOSIION Andrew Lounsbury September 8 No 8- ENNESSEE ECHNOLOGICAL UNIVERSIY Cookeville, N 38 Singular Value Decomposition Andrew Lounsbury Department
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2017 LECTURE 5
STAT 39: MATHEMATICAL COMPUTATIONS I FALL 17 LECTURE 5 1 existence of svd Theorem 1 (Existence of SVD) Every matrix has a singular value decomposition (condensed version) Proof Let A C m n and for simplicity
More informationNumerical Methods I Singular Value Decomposition
Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More information2. LINEAR ALGEBRA. 1. Definitions. 2. Linear least squares problem. 3. QR factorization. 4. Singular value decomposition (SVD) 5.
2. LINEAR ALGEBRA Outline 1. Definitions 2. Linear least squares problem 3. QR factorization 4. Singular value decomposition (SVD) 5. Pseudo-inverse 6. Eigenvalue decomposition (EVD) 1 Definitions Vector
More informationOrthonormal Transformations
Orthonormal Transformations Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 25, 2010 Applications of transformation Q : R m R m, with Q T Q = I 1.
More informationMATH 612 Computational methods for equation solving and function minimization Week # 2
MATH 612 Computational methods for equation solving and function minimization Week # 2 Instructor: Francisco-Javier Pancho Sayas Spring 2014 University of Delaware FJS MATH 612 1 / 38 Plan for this week
More informationVector and Matrix Norms. Vector and Matrix Norms
Vector and Matrix Norms Vector Space Algebra Matrix Algebra: We let x x and A A, where, if x is an element of an abstract vector space n, and A = A: n m, then x is a complex column vector of length n whose
More informationFall TMA4145 Linear Methods. Exercise set Given the matrix 1 2
Norwegian University of Science and Technology Department of Mathematical Sciences TMA445 Linear Methods Fall 07 Exercise set Please justify your answers! The most important part is how you arrive at an
More informationThe Singular Value Decomposition
The Singular Value Decomposition An Important topic in NLA Radu Tiberiu Trîmbiţaş Babeş-Bolyai University February 23, 2009 Radu Tiberiu Trîmbiţaş ( Babeş-Bolyai University)The Singular Value Decomposition
More informationσ 11 σ 22 σ pp 0 with p = min(n, m) The σ ii s are the singular values. Notation change σ ii A 1 σ 2
HE SINGULAR VALUE DECOMPOSIION he SVD existence - properties. Pseudo-inverses and the SVD Use of SVD for least-squares problems Applications of the SVD he Singular Value Decomposition (SVD) heorem For
More informationOrthogonal Transformations
Orthogonal Transformations Tom Lyche University of Oslo Norway Orthogonal Transformations p. 1/3 Applications of Qx with Q T Q = I 1. solving least squares problems (today) 2. solving linear equations
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More information1. The Polar Decomposition
A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION MATAN GAVISH Part. Theory. The Polar Decomposition In what follows, F denotes either R or C. The vector space F n is an inner product space with
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationSummary of Week 9 B = then A A =
Summary of Week 9 Finding the square root of a positive operator Last time we saw that positive operators have a unique positive square root We now briefly look at how one would go about calculating the
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationChapter 0 Miscellaneous Preliminaries
EE 520: Topics Compressed Sensing Linear Algebra Review Notes scribed by Kevin Palmowski, Spring 2013, for Namrata Vaswani s course Notes on matrix spark courtesy of Brian Lois More notes added by Namrata
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More informationSingular Value Decomposition (SVD) and Polar Form
Chapter 2 Singular Value Decomposition (SVD) and Polar Form 2.1 Polar Form In this chapter, we assume that we are dealing with a real Euclidean space E. Let f: E E be any linear map. In general, it may
More informationPseudoinverse & Orthogonal Projection Operators
Pseudoinverse & Orthogonal Projection Operators ECE 174 Linear & Nonlinear Optimization Ken Kreutz-Delgado ECE Department, UC San Diego Ken Kreutz-Delgado (UC San Diego) ECE 174 Fall 2016 1 / 48 The Four
More informationMath 407: Linear Optimization
Math 407: Linear Optimization Lecture 16: The Linear Least Squares Problem II Math Dept, University of Washington February 28, 2018 Lecture 16: The Linear Least Squares Problem II (Math Dept, University
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationLecture 2 INF-MAT : , LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky
Lecture 2 INF-MAT 4350 2009: 7.1-7.6, LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky Tom Lyche and Michael Floater Centre of Mathematics for Applications, Department of Informatics,
More information. = V c = V [x]v (5.1) c 1. c k
Chapter 5 Linear Algebra It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix Because they form the foundation on which we later work,
More informationLinear Algebra Formulas. Ben Lee
Linear Algebra Formulas Ben Lee January 27, 2016 Definitions and Terms Diagonal: Diagonal of matrix A is a collection of entries A ij where i = j. Diagonal Matrix: A matrix (usually square), where entries
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More informationProblem Set (T) If A is an m n matrix, B is an n p matrix and D is a p s matrix, then show
MTH 0: Linear Algebra Department of Mathematics and Statistics Indian Institute of Technology - Kanpur Problem Set Problems marked (T) are for discussions in Tutorial sessions (T) If A is an m n matrix,
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationLecture 7: Positive Semidefinite Matrices
Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.
More informationLecture notes on Quantum Computing. Chapter 1 Mathematical Background
Lecture notes on Quantum Computing Chapter 1 Mathematical Background Vector states of a quantum system with n physical states are represented by unique vectors in C n, the set of n 1 column vectors 1 For
More informationNORMS ON SPACE OF MATRICES
NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system
More information2. Every linear system with the same number of equations as unknowns has a unique solution.
1. For matrices A, B, C, A + B = A + C if and only if A = B. 2. Every linear system with the same number of equations as unknowns has a unique solution. 3. Every linear system with the same number of equations
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More informationMATH 31 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL
MATH 3 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL MAIN TOPICS FOR THE FINAL EXAM:. Vectors. Dot product. Cross product. Geometric applications. 2. Row reduction. Null space, column space, row space, left
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationSingular Value Decomposition
Chapter 5 Singular Value Decomposition We now reach an important Chapter in this course concerned with the Singular Value Decomposition of a matrix A. SVD, as it is commonly referred to, is one of the
More informationEE263: Introduction to Linear Dynamical Systems Review Session 9
EE63: Introduction to Linear Dynamical Systems Review Session 9 SVD continued EE63 RS9 1 Singular Value Decomposition recall any nonzero matrix A R m n, with Rank(A) = r, has an SVD given by A = UΣV T,
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationIntroduction to Numerical Linear Algebra II
Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49 Overview We will cover this material in
More informationThe QR Decomposition
The QR Decomposition We have seen one major decomposition of a matrix which is A = LU (and its variants) or more generally PA = LU for a permutation matrix P. This was valid for a square matrix and aided
More informationLecture 4 Orthonormal vectors and QR factorization
Orthonormal vectors and QR factorization 4 1 Lecture 4 Orthonormal vectors and QR factorization EE263 Autumn 2004 orthonormal vectors Gram-Schmidt procedure, QR factorization orthogonal decomposition induced
More informationComputational math: Assignment 1
Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More information1 Last time: least-squares problems
MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that
More informationChapter 6: Orthogonality
Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products
More informationELE/MCE 503 Linear Algebra Facts Fall 2018
ELE/MCE 503 Linear Algebra Facts Fall 2018 Fact N.1 A set of vectors is linearly independent if and only if none of the vectors in the set can be written as a linear combination of the others. Fact N.2
More informationDesigning Information Devices and Systems II
EECS 16B Fall 2016 Designing Information Devices and Systems II Linear Algebra Notes Introduction In this set of notes, we will derive the linear least squares equation, study the properties symmetric
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationRecall the convention that, for us, all vectors are column vectors.
Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists
More informationPseudoinverse & Moore-Penrose Conditions
ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego p. 1/1 Lecture 7 ECE 275A Pseudoinverse & Moore-Penrose Conditions ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego
More informationComputational Methods CMSC/AMSC/MAPL 460. EigenValue decomposition Singular Value Decomposition. Ramani Duraiswami, Dept. of Computer Science
Computational Methods CMSC/AMSC/MAPL 460 EigenValue decomposition Singular Value Decomposition Ramani Duraiswami, Dept. of Computer Science Hermitian Matrices A square matrix for which A = A H is said
More informationI. Multiple Choice Questions (Answer any eight)
Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY
More informationMATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators.
MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. Adjoint operator and adjoint matrix Given a linear operator L on an inner product space V, the adjoint of L is a transformation
More informationA Review of Linear Algebra
A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose
More informationControl Systems. Linear Algebra topics. L. Lanari
Control Systems Linear Algebra topics L Lanari outline basic facts about matrices eigenvalues - eigenvectors - characteristic polynomial - algebraic multiplicity eigenvalues invariance under similarity
More informationCheat Sheet for MATH461
Cheat Sheet for MATH46 Here is the stuff you really need to remember for the exams Linear systems Ax = b Problem: We consider a linear system of m equations for n unknowns x,,x n : For a given matrix A
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationLinear Algebra Fundamentals
Linear Algebra Fundamentals It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix. Because they form the foundation on which we later
More informationLecture: Face Recognition and Feature Reduction
Lecture: Face Recognition and Feature Reduction Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 1 Recap - Curse of dimensionality Assume 5000 points uniformly distributed in the
More informationNotes on Linear Algebra
1 Notes on Linear Algebra Jean Walrand August 2005 I INTRODUCTION Linear Algebra is the theory of linear transformations Applications abound in estimation control and Markov chains You should be familiar
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationFoundations of Matrix Analysis
1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationMATH 350: Introduction to Computational Mathematics
MATH 350: Introduction to Computational Mathematics Chapter V: Least Squares Problems Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2011 fasshauer@iit.edu MATH
More informationYORK UNIVERSITY. Faculty of Science Department of Mathematics and Statistics MATH M Test #2 Solutions
YORK UNIVERSITY Faculty of Science Department of Mathematics and Statistics MATH 3. M Test # Solutions. (8 pts) For each statement indicate whether it is always TRUE or sometimes FALSE. Note: For this
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More information