HE SINGULAR VALUE DECOMPOSIION he SVD existence - properties. Pseudo-inverses and the SVD Use of SVD for least-squares problems Applications of the SVD he Singular Value Decomposition (SVD) heorem For any matrix A R m n there exist unitary matrices U R m m and V R n n such that A = UV where is a diagonal matrix with entries i 0. σ σ σ pp 0 with p = min(n, m) he i s are the singular values. Notation change i Proof: Let σ = A = max x, x = Ax. here exists a pair of unit vectors v, u such that Av = σ u 0- B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-0- Complete v into an orthonormal basis of R n V [v, V ] = n n unitary Complete u into an orthonormal basis of R m U [u, U ] = m m unitary Observe that ( σ ) ( A σ w + w = σ + σ ) w w his shows that w must be zero [why?] Complete the proof by an induction argument. Define U, V as single Householder reflectors. hen, it is easy to show that σ w AV = U U σ w AV = A 0 B 0 B 0-3 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-4 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-3 0-4
Case : he thin SVD A = U V Consider the Case-. It can be rewritten as A = [U U ] 0 V Case : Which gives: A = U V A = U V where U is m n (same shape as A), and and V are n n Referred to as the thin SVD. Important in practice. How can you obtain the thin SVD from the QR factorization of A and the SVD of an n n matrix? 0-5 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-6 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-5 0-6 A few properties. Assume that Properties of the SVD (continued) σ σ σ r > 0 and σ r+ = = σ p = 0 hen: rank(a) = r = number of nonzero singular values. Ran(A) = span{u, u,..., u r } Null(A ) = span{u r+, u r+,..., u m } Ran(A ) = span{v, v,..., v r } Null(A) = span{v r+, v r+,..., v n } he matrix A admits the SVD expansion: A = r u i v i i= A = σ = largest singular value A F = ( r i= ) / When A is an n n nonsingular matrix then A = /σ n 0-7 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-7 0-8
heorem then Let k < r and A k = k u i v i i= min A B = A A k = σ k+ rank(b)=k Proof: First: A B σ k+, for any rank-k matrix B. Consider X = span{v, v,, v k+ }. Note: dim(null(b)) = n k Null(B) X {0} [Why?] Let x 0 Null(B) X, x 0 0. Write x 0 = V y. hen (A B)x 0 = Ax 0 = UV V y = y But y σ k+ x 0 (Show this). A B σ k+ Second: take B = A k. Achieves the min. 0-9 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-0 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-9 0-0 Right and Left Singular vectors: A = u i A u j = σ j v j Consequence A A = σ i and AA u i = σ i u i Right singular vectors ( s) are eigenvectors of A A Left singular vectors (u i s) are eigenvectors of AA Possible to get the SVD from eigenvectors of AA and A A but: difficulties due to non-uniqueness of the SVD Define the r r matrix = diag(σ,..., σ r ) Let A R m n and consider A A ( R n n ): A A = V V A A = V 0 }{{} n n his gives the spectral decomposition of A A. V 0- B: 4-5; AB:.,.; GvL.4,5.5 SVD 0- B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-0-
Similarly, U gives the eigenvectors of AA. Pseudo-inverse of an arbitrary matrix AA = U Important: A A = V D V and AA U, V up to signs! 0 }{{} m m U = UD U give the SVD factors Let A = UV which we rewrite as A = U U 0 V hen the pseudo inverse of A is V = U V A = V U = r j= σ j v j u j he pseudo-inverse of A is the mapping from a vector b to the solution min x Ax b that has minimal norm (to be shown) In the full-rank overdetermined case, the normal equations yield x = (A } A) {{ A } b A 0-3 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-4 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-3 0-4 Least-squares problem via the SVD Pb: min b Ax in general case. Consider SVD of A: A = U U 0 V r = σ i u i V i= hen left multiply by U to get Ax b = 0 y U y U b y V with = y V x What are all least-squares solutions to the system? Among these which one has minimum norm? 0-5 B: 4-5; AB:.,.; GvL.4,5.5 SVD Answer: From above, must have y = U b and y = anything (free). Recall that x = V y and write y x = [V, V ] = V y y + V y = V U b + V y = A b + V y Note: A b Ran(A) and V y Null(A). herefore: least-squares solutions are of the form A b + w where w Null(A). Smallest norm when y = 0. 0-6 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-5 0-6
Minimum norm solution to min x Ax b satisfies y = U b, y = 0. It is: x LS = V U b = A b If A R m n what are the dimensions of A?, A A?, AA? Show that A A is an orthogonal projector. What are its range and null-space? Same questions for AA. Moore-Penrose Inverse he pseudo-inverse of A is given by A = V 0 U = Moore-Penrose conditions: r u i he pseudo inverse of a matrix is uniquely determined by these four conditions: i= () AXA = A () XAX = X (3) (AX) H = AX (4) (XA) H = XA In the full-rank overdetermined case, A = (A A) A 0-7 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-8 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-7 0-8 Least-squares problems and the SVD SVD can give much information about solving overdetermined and underdetermined linear systems. Let A be an m n matrix and A = UV its SVD with r = rank(a), V = [v,..., v n ] U = [u,..., u m ]. hen r u i x LS = b i= minimizes b Ax and has the smallest -norm among all possible minimizers. In addition, ρ LS b Ax LS = z with z = [u r+,..., u m ] b Least-squares problems and pseudo-inverses A restatement of the first part of the previous result: Consider the general linear least-squares problem min x, x S S = {x R n b Ax min}. his problem always has a unique solution given by x = A b 0-9 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-0 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-9 0-0
Consider the matrix: A = 0 0 Compute the singular value decomposition of A Find the matrix B of rank which is the closest to the above matrix in the -norm sense. What is the pseudo-inverse of A? What is the pseudo-inverse of B? Find the vector x of smallest norm which minimizes b Ax with b = (, ) Find the vector x of smallest norm which minimizes b Bx with b = (, ) 0- B: 4-5; AB:.,.; GvL.4,5.5 SVD Ill-conditioned systems and the SVD Let A be m m and A = UV its SVD Solution of Ax = b is x = A b = m u i b i= When A is very ill-conditioned, it has many small singular values. he division by these small s will amplify any noise in the data. If b = b + ɛ then A b = m i= u i b + m u i ɛ i= }{{} Error Result: solution could be completely meaningless. 0- B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-0- Remedy: SVD regularization Numerical rank and the SVD runcate the SVD by only keeping the σ i s that are τ, where τ is a threshold Gives the runcated SVD solution (SVD solution:) x SV D = τ u i b Many applications [e.g., Image and signal processing,..] 0-3 B: 4-5; AB:.,.; GvL.4,5.5 SVD Assuming the original matrix A is exactly of rank k the computed SVD of A will be the SVD of a nearby matrix A + E Can show: ˆ α σ u Result: zero singular values will yield small computed singular values and r larger sing. values. Reverse problem: numerical rank he ɛ-rank of A : r ɛ = min{rank(b) : B R m n, A B ɛ}, Show that r ɛ equals the number sing. values that are >ɛ Show: r ɛ equals the number of columns of A that are linearly independent for any perturbation of A with norm ɛ. Practical problem : How to set ɛ? 0-4 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-3 0-4
Pseudo inverses of full-rank matrices Case : m < n hen A = A (AA ) Case : m > n hen A = (A A) A hin SVD is A = U V. Now U, are m m and: hin SVD is A = U V and V, are n n. hen: Example: (A A) A = (V V ) V U = V V V U = V U = A 0 Pseudo-inverse of is? 0 0-5 B: 4-5; AB:.,.; GvL.4,5.5 SVD Example: A (AA ) = V U [U U ] = V U U U = V U = V U = A Pseudo-inverse of 0 0 is? Mnemonic: he pseudo inverse of A is A completed by the inverse of the smallest of (A A) or (AA ) where it fits (i.e., left or right) 0-6 B: 4-5; AB:.,.; GvL.4,5.5 SVD 0-5 0-6