arxiv: v1 [cs.lg] 22 Mar 2014

Transcription

1 CUR lgorithm with Incomplete Matrix Observation Rong Jin an Shenghuo Zhu Dept. of Computer Science an Engineering, Michigan State University, NEC Laboratories merica, Inc., arxiv: v1 [cs.lg] 22 Mar Introuction CUR matrix ecomposition is a ranomize algorithm that can efficiently compute the low rank approximation for a given rectangle matrix Drineas et al., 2006, Mahoney an Drineas, 2008, Let R n m be the given matrix an k be the target rank for approximation. CUR ranomly samples c = Ok log k/ɛ 2 columns an r = Ok log k/ɛ 2 rows from, accoring to their leverage scores, to form matrices C an R, respectively. The approximate matrix  is then compute as CUR, where U = C R. It can be shown, that with a high probability,  F 2 + ɛ k F 1 where k is the best k-rank approximation of. In case when the maximum of statistical leverage scores, which is also referre to as incoherence measure in matrix completion Canès an Tao, 2010, Recht, 2011, Canès an Recht, 2012, are small, CUR matrix ecomposition can be simplifie by uniformly sampling rows an columns from. The simplifie algorithm will have a relative error boun similar to that in 1 except that the sample sizes c an r shoul be increase by the incoherence measure. In this raft, we will focus on the situation with boune incoherence measure where uniform sampling of columns an rows is in general sufficient. One limitation with the existing CUR algorithms is that they require an access to the full matrix for computing U. In this work, we aim to alleviate this limitation. In particular, we assume that besies having an access to ranomly sample rows an columns from, we only observe a subset of ranomly sample entries Ω from. Our goal is to evelop a low rank approximation algorithm, similar to CUR, base on i ranomly sample rows an columns from, an ii ranomly sample entries from. Compare to the stanar matrix completion theory Canès an Tao, 2010, Recht, 2011, Canès an Recht, 2012, the key avantage of the propose algorithm is its low sample complexity an high computational efficiency. In particular, unlike matrix completion that requires Orn log 2 n number of observe entries, the propose algorithm is able to perfectly recover the target matrix with only Orn log n number of observe entries incluing the ranomly sample entries an entries in ranomly sample rows an columns. In aition, instea of having to solve an optimization problem involve trace norm regularization, the propose algorithm only nees to solve a stanar regression problem. Finally, unlike most matrix completion theories that hol only when the target matrix is of low rank, we show a strong guarantee for the propose algorithm even when the target matrix is not low rank. We finally note that a closely relate algorithm, title Low-rank Matrix an Tensor Completion via aptive Sampling, was publishe recently Krishnamurthy an Singh, It is esigne to recover a low rank matrix with ranomly sample rows an entries, which is ifferent from the goal of this work i.e. computing a low rank approximation for a target matrix. 2 lgorithm an Notation Let M R n m be the target matrix, where n m. To approximate M, we first sample uniformly at ranom columns an rows from M, enote by = a 1,..., a R n an B = b 1,..., b R m, respectively, where each a i R n an b j R m is a row an column of M, respectively. Let r be the target rank for approximation, with r. Let Û = û 1,..., û r R n r an V = v 1,..., v r R m r be the first r eigenvectors of an BB, respectively. Besies an B,

2 we furthermore sample, uniformly at ranom, entries from matrix M. Let Ω inclue the inices of ranomly sample entries. Our goal is to approximately recover the matrix M using, B, an ranomly sample entries in Ω. To this en, we will solve the following optimization problem where R Ω : R n m R n m is efine as min Z R r r R Ω M R Ω ÛZ V 2 F 2 [R Ω M] i,j = Mi,j i, j Ω 0 o. w. Let Z be an optimal solution to 2. The recovere matrix is given by M = ÛZ V. The following notation will be use throughout the raft. We enote by σ i, i = 1,..., m the singular values of M in ranke in the escening orer, an by u i an v i be the corresponing left an right singular vectors. Define U = u 1,..., u m an V = v 1,..., v m. Given r [m], partition the SVD ecomposition of M as r n r M = UΣV = U 1 U 2 Let ũ i, i [n] be the ith column of U1 incoherence measure for U 1 an V 1 as µr = max Σ1 Σ2 V 1 V2 3 an ṽ i, i [m] be the ith column of V 1. Define the n max i [n] r ũ i 2 m, max i [m] r ṽ i 2 Similarly, we efine the incoherence measure for matrices Û an V. Let û i, i [n] be the ith column of Û an v i, i [m] be the ith column of V. Define the incoherence measure for Û an V as n µ = max max i [n] r û i 2 m, max i [m] r v i 2 Define projection operators P U = UU, P V = V V, PÛ = ÛÛ, an P V = V V. We will use 2 for spectral norm of matrix, an F for the Frobenius norm of matrix. 3 Supporting Theorems In this section, we present several theorems that are important to our analysis. Theorem 1 Halko et al., 2011 Let M be an n m matrix with singular value ecomposition M = UΣV, an a fixe r > 0. Choose a test matrix Ω R m an construct sample matrix Y = MΩ. Partition M as in 3 an efine Ω 1 = V1 Ω an Ω 2 = V2 Ω. ssuming Ω 1 has full row rank, the approximation error satisfies M P Y M 2 2 Σ Σ 2 Ω 2 Ω where P Y M project column vectors in M in the subspace spanne by the column vectors in Y. Theorem 2 Tropp, 2011 Let X be a finite set of PSD matrices with imension k, an suppose that max X X λ 1X B. Sample X 1,..., X l } uniformly at ranom from X without replacement. Compute Then µ max = lλ max E[X 1 ], µ min = lλ min E[X 1 ] λ max l i=1 λ min l i=1 X i X i } [ e δ 1 + δµ max k 1 + δ 1+δ } [ e δ 1 δµ min k 1 δ 1 δ ] µmax/b ] µmin/b 2

3 Theorem 3 Let = S HS an à = S HS be two symmetric matrices of size n n. Let λ i, i [n] an λ i, i [n] be the eigenvalues of an Ã, respectively, ranke in escening orer. Let U 1, Ũ1 R n r inclue the first r eigenvectors of an Ã, respectively. Let be any invariant norm. Define 2 λ = min 1 λ r+1 1, λ r 2 H = H 1 2 H H 1 H 1 2 H H 2 If λ H /2, we have sin ΘU 1, Ũ1 H 1 + H λ λ H /2 16 Since the above theorem follows irectly from Theorem 4.4 an iscussion in Section 5 from Li, 1999, we skip its proof. 4 Recovering a Low Rank Matrix In this section, we iscuss the recovery result when the rank of M is no more than r. We will first provie the key results for our analysis, an then present etaile proof for the key theorems. 4.1 Main Result Our analysis is ivie into two steps. We will first show that M PÛ MP V 2 2 is small, an then boun the strongly convexity of the objective function in 2. The following theorem shows that the ifference between M an M, measure in spectral norm, is well boune if M PÛ MP V 2 2 is small an the objective function in 2 is strongly convex. Theorem 4 ssume i M PÛ MP V 2 2, an ii the strongly convexity of the objective function is no less than Ω γ. Then M M γ To utilize Theorem 4, we nee to boun an γ, respectively, which are given in the following two theorems. Theorem 5 With a probability 1 2e t, we have, if 7µrrt + log r. oof: := M PÛ MP V 2 2 4σ 2 r+1 Our analysis is base on the following theorem. Theorem 6 With a probability 1 2e t, we have, M MP V 2 2 σr m provie that 7µrrt + log r. 1 + m + n, M PÛ M 2 σ 2 r n Using Theorem 6, we have, if 7µrrt + log r, with a probability 1 2e t M PÛ MP V M MP V M PÛ MP V 2 2 4σr n + m Theorem 7 With a probability 1 e t, we have that the strongly convexity for the objective function in 2 is boune from below by Ω /2, provie that Ω 7 µ 2 r 2 t + 2 log r 3

4 The following lemma allows us to replace µ in Theorem 7 with µ. Theorem 8 ssume rankm r. Then, with a probability 1 2e t, we have µ = µr, provie 7µrrt + log r. oof: When rankm r, accoring to Theorem 5, with a probability 1 2e t, we have M = PÛ MP V, provie that 7µrrt + log r. Hence P U1 = PÛ an P V1 = P V, which irectly implies that µ = µ. The following theorem follows irectly follows from Theorem 5, 7, 8, an 4. Theorem 9 ssume rankm r, 7µrrt + log r, an Ω 7µ 2 rr 2 t + 2 log r. Then, with a probability 1 3e t, we have M = M. Remark The result from Theorem 9 shows that, with a probability 1 δ, a low rank matrix M can be perfectly recovere from Orn + r 2 logr/δ number of observations from matrix M. This result significantly improves the result from Krishnamurthy an Singh, 2013, where Or 2 n log1/δ number of observations are neee for perfect recovery. We shoul note that unlike Krishnamurthy an Singh, 2013 where a small incoherence measure is assume only for column vectors in matrix M, we assume a small incoherence measure for both row an column vectors in M. It is this assumption that allows us to sample both rows an columns of M, leaing to the improvement in the sample complexity. 4.2 Detaile oofs oof of Theorem 4 Set Z = Û M V. Since M PÛ MP V 2 2, we have implying that Hence, we have M ÛZ V 2 2, M i,j [ÛZ V ] i,j 2, i [n], j [m] R Ω M R Ω ÛZ V 2 F Ω Let Z be the optimal solution to 2. Using the strongly convexity of 2, we have i.e. Z Z 2 F /γ. We thus have γ 2 Ω Z Z 2 F 1 2 Ω, M M M PÛ MP V PÛ MP V ÛZ V M PÛ MP V Z Z γ oof of Theorem 6 Let i 1,..., i are the selecte columns. Define Ω = e i1,..., e i R m, where e i is the ith canonical basis. To utilize Theorem 1, we nee to boun the minimum eigenvalue of Ω 1 Ω 1. We have Ω 1 Ω 1 = V1 ΩΩ V 1 Let ṽ i, i [] be the ith row vector of V 1. We have Ω 1 Ω 1 = ṽ ij ṽi j It is straightforwar to show that E [ Ω 1 Ω ] 1 = m I r To boun the minimum eigenvalue of Ω 1 Ω 1, we will use Theorem 2. To this en, we have j=1 B = max 1 i m ṽ i 2 µr r m 4

5 Thus, we have λ min Ω 1 Ω 1 1 δ } r exp m µrr By setting δ = 1/2, we have, with a probability 1 re /[7µrr] Uner the assumption that λ min Ω 1 Ω 1 λ min Ω 1 Ω 1 2m 2m, using Theorem 1, we have P V 2 2 σr Σ 2 Ω 2 Ω 1 2 σr m 2 Σ 2Ω σ2 r+1 We complete the proof using the fact that Ω oof of Theorem 7 We rewrite the objective function as R Ω M R Ω ÛZ V 2 F = R ΩM r i=1 j=1 [δ + 1 δ ln1 δ] r Z s,t R Ω û i v j 1 + 2m Ω Define matrix K = k 1,..., k r 2 R nm r2, where k i,j = vec R Ω û i v j. Our goal is to boun the minimum eigenvalue of K K. To Theorem 2, we boun B = max k i,j 2 µ2 r 2 i,j mn an λ min E[K K] = Ω mn λ min Û V = Ω mn where is Kronecker prouct. Thus, accoring to Theorem 2, with a probability 1 e t, we have λ min K K Ω 2mn provie that Ω 7 µ 2 r 2 t + 2 log r 5 Recovering the Low Rank pproximation of a Full Rank Matrix In this section, we consier a general case when M is of full rank but with skewe eigenvalue istribution. To capture the skewe eigenvalue istribution, we use the concept of numerical rank rm, λ with respect to non-negative constant λ > 0, which is efine as follows Hansen, 1987 rm, λ = m i=1 σ 2 i σ 2 i + mnλ Define H = λi + 1 mn MM, Ĥ = λi + 1 n an H B = λi + 1 mn M M, Ĥ B = λi + 1 m BB Next, we generalize the efinition of incoherence measure to numerical low rank. Define S = Σ 2 + mnλi, where Σ = iagσ 1,..., σ m, an incoherence measure µ with respect to a non-negative constant λ > 0 as n µλ = max max 1 i n rm, λ V i, ΣS 1/2 2 m, max 1 i n rm, λ U i, ΣS 1/2 2 It is easy to verify that µλ 1. Note that when the rank of matrix M is r, we have rm, 0 = r an µ0 = µr. In orer to utilize the theorems presente in Section 4 to boun M M 2, the key is to boun µr an µ by µλ. The following theorem allows us to boun µr by µλ. 2 F 5

6 Lemma 1 ssume σ 2 r σ 2 r + mnλrm, λ a r for some positive a > 0. We have µλ aµr. More specifically, if we choose λ = σr/mn, 2 we have 2rM, λ µr µλ r Using the above lemma, we have a moifie version for Theorem 5 Theorem 10 Set λ = σr/mn 2 for a fixe r. With a probability 1 2e t, we have, := M PÛ MP V 2 2 4σr m + n if 14µλrM, λt + log r. We note that Theorem 10 is almost ientical to Theorem 5 except that µrr is replace with µλrm, λ. Next we will boun µr by µλ. To this en, we nee the following theorem. Theorem 11 With a probability 1 4e t, for any k [n], we have if 1 δ λ k H 1/2 Ĥ H 1/2 1 + δ 1 δ λ k H 1/2 B Ĥ B H 1/2 B 1 + δ 4 µλrm, λ + 1t + log n δ2 Theorem 12 ssume that 16µλrM, λ + 1t + log n, an σ r 2σ r+1. Set λ = σ 2 r/mn. With a probability 1 4e t, we have where µr 2rM, λ µλ + 18nδ2 r r δ 2 = 4 µλrm, λ + 1t + log n Using Theorem 12, we have the following version of Theorem 7 Theorem 13 ssume 16µλrM, λ + 1t + log n an σ r 2σ r+1. Set λ = σr/mn. 2 With a probability 1 5e t, we have that the strongly convexity for the objective function in 2 is boune from below by Ω /2, provie that Ω 7 2µλrM, λ + 72 n 2 µλrm, λ + 1t + log n t + 2 log r Combining the above results, we have the final theorem for the recovering of M when its numerical rank rm, λ is small. Theorem 14 ssume 16µλrM, λ + 1t + log n an σ r 2σ r+1. Set λ = σr/mn. 2 We have, with a probability 1 7e t M M σr+1 2 m + n 1 + if Ω 7 2µλrM, λ + 72 n 2 µλrm, λ + 1t + log n t + 2 log r Remark The total number of observe entries are Õn + n2 / 2. It is minimize when = n 1/3, leaing to Õn4/3 for the number of observe entries an Õσ r+1n 1/3 for recovery error. 6

7 5.1 Detaile oof oof of Theorem 11 It is sufficient to show the result for H 1/2 Ĥ H 1/2. Define } X = M i = H 1/2 1 n a ia i + λi H 1/2, i = 1,..., m We have M i = US 1/2 U muσvi, V i, ΣU + mnλius 1/2 U = U ms 1/2 ΣVi, V i, ΣS 1/2 + mnλs 1 U Using the efinition of µλ, we have λ max M i µrm, λ + 1. Since we have λ max H 1/2 Using the fact that we have Ĥ H 1/2 λ max H 1/2 B = λ max E[M i ] = } 1 + δ n exp µλrm, λ δ log1 + δ δ δ2, δ [0, 1], Ĥ H 1/2 } 1 + δ n exp [1 + δ log1 + δ δ] δ 2 4µrM, λ + 1 The upper boun is obtaine by setting = 4µλrM, λ + 1log n + t/δ 2. Similarly, for the lower boun, we have } λ min H 1/2 Ĥ H 1/2 n 1 δ n exp [1 δ log1 δ + δ] µλrm, λ + 1 Using the fact that 1 δ log1 δ δ + δ2 2 We have the lower boun by setting m = 2µλrM, λ + 1log n + t/δ oof of Theorem 12 To utilize Theorem 3, we rewrite H an Ĥ, efine in Theorem 11, as H = H 1/2 IH1/2, Ĥ = H 1/2 DH1/2 where D = H 1/2 Ĥ H 1/2. ccoring to Theorem 11, with a probability 1 2e t, we have D I 2 δ, provie that 4 µλrm, λ + 1t + log n δ2 We then compute λ an H efine in Theorem 3. Using the fact 16µλrM, λ+1t+log n an Theorem 11, we have, with a probability 1 e t, δ 1/2. Hence H D I 2 δ = 2δ 1 D I 2 1 δ Using the assumption that σ r /σ r+1 2, we have δ 1/2 1 σ 2 r+1/σ 2 r an therefore λ = 1/ 2. s a result, accoring to Theorem 3, we have sin ΘU 1, Û 2 3 2δ Similarly, we have, sin ΘV 1, V 2 3 2δ Thus, with a probability 1 4e t, we have µr µλ + n r sin ΘV 1, V 2 2 µλ + 18nδ2 r 7

8 References E. Canès an B. Recht. Exact matrix completion via convex optimization. Commun. CM, 556: , E. Canès an T. Tao. The power of convex relaxation: near-optimal matrix completion. IEEE Transactions on Information Theory, 565: , P. Drineas, R. Kannan, an M.W. Mahoney. Fast Monte Carlo algorithms for matrices III: Computing a compresse approximate matrix ecomposition. SIM J Comput, 36: , N. Halko, P. G. Martinsson, an J.. Tropp. Fining structure with ranomness: obabilistic algorithms for constructing approximate matrix ecompositions. SIM Rev., 532: , P. C. Hansen. Rank-efficient an iscrete ill-pose problems: Numerical aspects of linear inversion. Society for Inustrial an pplie Mathematics, kshay Krishnamurthy an arti Singh. Low-rank matrix an tensor completion via aaptive sampling. In avance in Neural Information ocessing NIPS, R.-C. Li. Relative perturbation theory: II eigenspace an singular subspace variations. SIM J. Matrix nal. ppl., 20: , M. W. Mahoney an P. Drineas. Relative-error CUR matrix ecompositions. SIM J Matrix nal ppl, 30: , M. W. Mahoney an P. Drineas. CUR matrix ecompositions for improve ata analysis. oc. Natl. ca. Sci. US, 106: , B. Recht. simpler approach to matrix completion. JMLR, 12: , J. Tropp. Improve analysis of the subsample ranomize haamar transform. v. apt. Data nal, 3: ,