Lecture 8: October 20, Applications of SVD: least squares approximation

Transcription

1 Mathematical Toolkit Autum 2016 Lecturer: Madhur Tulsiai Lecture 8: October 20, Applicatios of SVD: least squares approximatio We discuss aother applicatio of sigular value decompositio (SVD) of matrices. Let a 1,..., a R d be poits which we wat to fit to a low-dimesioal subspace. The goal is to fid a subspace S of R d of dimesio at most k to miimize i=1 (dist(a i, S)) 2, where dist(a i, S) deotes the distace of a i from the closest poit i S. We first prove the followig. Claim 1.1 Let u 1,..., u k be a orthoormal basis for S. The (dist(a i, S)) 2 = a i 2 k 2 2 ai, u j. Thus, the goal is to fid a set of k orthoormal vectors u 1,..., u k to mazimixe i=1 k ai, u j 2. Let A R d be a matrix with the i th row equal to ai T. The, we eed to fid orthoormal vectors u 1,..., u k to maximize Au Au k 2 2. We will prove the followig. Propositio 1.2 Let v 1,..., v r be the right sigular vectors of A correspodig to sigular values σ 1 σ r > 0. The, for all k r ad all orthoormal sets of vectors u 1,..., u k Au Au k 2 2 Av Av k 2 2 Thus, the optimal solutio is to take S = Spa (v 1,..., v k ). We prove the above by iductio o k. For k = 1, we ote that Au = A T Au 1, u 1 max R A v R d \{0} T A(v) = σ1 2 = Av To prove the iductio step for a give k r, defie V k 1 = { v R d v, v i = 0 i [k 1] }. First prove the followig claim. Claim 1.3 Give a orthoormal set u 1,..., u k, there exist orthoormal vectors u 1,..., u k such that 1

2 - u k V k 1. - Spa (u 1,..., u k ) = Spa ( u 1,..., u k). - Au Au k 2 2 = Au Au k 2 2. Thus, we ca assume without loss of geerality that the give vectors u 1,..., u k are such that u k V k 1. Hece, Au k 2 2 max Av 2 v V k 1 2 = σk 2 = Av k 2 2. v =1 Also, by the iductive hypothesis, we have that Au Au k Av Av k 1 2 2, which completes the proof. The above proof ca also be used to prove that SVD gives the best rak k approximatio to the matrix A i Frobeius orm. We will see this i the ext homework. 2 Boudig the eigevalues: Gershgori Disc Theorem We will ow see a simple but extremely useful boud o the eigevalues of a matrix, give by the Gershgori disc theorem. May useful variats of this boud ca also be derived from the observatio that for ay ivertible matrix S, the matrices S 1 MS ad M have the same eigevalues (prove it!). Theorem 2.1 (Gershgori Disc Theorem) Let M C. Let R i = j =i M ij. Defie the set Disc(M ii, R i ) := {x x C, x M ii R i }. If λ is a eigevalue of M, the λ Disc(M ii, R i ). i=1 Proof: Let z C be a eigevector correspodig to the eigevalue λ. Let i 0 = argmax i [] { z i }. Sice z is a eigevector, we have Mz = λz i [] M ij = λz i. 2

3 I particular, we have that for i = i 0, M i0 j = λ M i0 j = λ M i0 j = λ M i0 i 0. Thus, we have λ M i0 i 0 M i0 j M i0 j = R i A applicatio to compressed sesig The Gershgori disc theorem is quite useful i compressed sesig, to esure what is kow as the Restricted Isometry Property for the measuremet matrices. Defiitio 2.2 A matrix A R k is said to have the restricted isometry property with parameters (s, δ s ) if (1 δ s ) x 2 Ax 2 (1 + δ s ) x 2 for all x R which satisfy {i x i = 0} s. Thus, we wat the trasformatio A to be approximately orm preservig for all sparse vectors x. This ca of course be esured for all x by takig A = id, but we require k for the applicatios i compressed sesig. I geeral, the restricted isometry property is NP-hard to verify ad ca thus also be hard to reaso about for a give matrix. The Gershgori Disc Theorem lets us derive a much easier coditio which is sufficiet to esure the restricted isometry property. Defiitio 2.3 Let A R k be such that coherece of A as We will prove the followig µ(a) = max i =j Propositio 2.4 Let A R k be such that A (i) = 1 for each colum A (i) of A. Defie the A (i), A (j). A (i) = 1 for each colum A (i) of A. The, for ay s, the matrix A has the restricted isometry property with parameters (s, (s 1)µ(A)). Note that the boud becomes meaigless if s However, the above propositio µ(a) shows that it may be sufficiet to boud µ(a) (which is also easier to check i practice). 3

4 Proof: Cosider ay x such that {i x i = 0} s. Let S deote the support of x i.e., S = {i x i = 0}. Let A S deote the k S submatrix where we oly keep the colums correspodig to idices i S. Let x S deote x restricted to the o-zero etries. The Ax 2 = A S x S 2 = AS T A Sx S, x S. Thus, it suffices to boud the eigevalues of the matrix A T S A S. Note that (A S ) ij = A (i), A (j). Thus the diagoal etries are 1 ad the off-diagoal etries are bouded by µ(a) i absolute value. By the Gershgori Disc Theorem, for ay eigevalue λ of A, we have Thus, we have as desired. λ 1 (s 1) µ(a). (1 (s 1) µ(a)) x 2 Ax 2 (1 + (s 1) µ(a)) x 2, The theorem is also very useful for boudig how much the eigevalues of matrix chage due to a perturbatio. We will see a example of this i the homework. 3 Solvig systems of liear equatios: Gaussia elimiatio Give a system of liear equatios Ax = b for A F m, b F m, recall that we ca solve the system or determie that there is o solutio by covertig the matrix [A b] to a row-reduced form usig elemetary row operatios. Defiitio 3.1 A matrix M F m is said to be i row-reduced form if - The first o-zero etry i each row (kow as the leadig etry) is 1. - If the leadig etry i row i 0 is i colum j 0, the M ij = 0 for all i > i 0 ad j j 0. - All o-zero rows occur above the zero rows. Notice that a matrix i the row-reduced form is always upper triagular. Also, the system has o solutio if ad oly if there is a o-zero row with a leadig etry i the last colum (correspodig to the etries of b). Also, if the system has a solutio, the it ca easily be foud usig back-substitutio, startig from the last o-zero row. Also, recall that a elemetary row operatios cosist of the followig (usig M i to deote the i th row of M): 4

5 - Swappig the rows M i ad M j, for some i, j, [m]. - M i c M i for some i [m], c F \ {0}. - M i M i + c M j for some i, j [m], c F. A matrix M ca always be coverted to a row-reduced form usig elemetary row operatios, which gives a geeral algorithm for solvig a system of liear equatios over ay field. However, the time take by this algorithm ca be as large as Ω( 3 ), which is prohibitive for large matrices. I the ext lecture, we will discuss methods which ca take advatage of sparsity to sigificatly speed up the solutio of liear systems. Exercise 3.2 Prove that performig elemetary row operatios o a give matrix M chages either the row rak, or the colum rak of M. Use this to prove that for ay matrix M, the row-rak ad colum-rak are equal. 5