A Converse to Low-Rank Matrix Completion

Transcription

1 A Convese to Low-Rank Matix Completion Daniel L. Pimentel-Alacón, Robet D. Nowak Univesity of Wisconsin-Madison Abstact In many pactical applications, one is given a subset Ω of the enties in a d N data matix X, and aims to infe all the missing enties. Existing theoy in low-ank matix completion (LRMC) povides conditions on X (e.g., bounded coheence o geneicity) and Ω (e.g., unifom andom sampling o deteministic combinatoial conditions) to guaantee that if X is ank-, then X is the only ank- matix that agees with the obseved enties, and hence X can be uniquely ecoveed by some method (e.g., nuclea nom o altenating minimization). In many situations, though, one does not know befoehand the ank of X, and depending on X and Ω, thee may be ank- matices that agee with the obseved enties, even if X is not ank-. Hence one can be deceived into thinking that X is ank when it eally is not. In this pape we give conditions on X (geneicity) and a deteministic condition on Ω to guaantee that if thee is a ank- matix that agees with the obseved enties, then X is indeed ank-. While ou condition on Ω is combinatoial, we povide a deteministic efficient algoithm to veify whethe the condition is satisfied. Futhemoe, this condition is satisfied with high pobability unde unifom andom sampling schemes with only O(max{, log d}) samples pe column. This stengthens existing esults in LRMC, allowing to dop the assumption that X is known a pioi to be low-ank. I. INTRODUCTION In many moden applications, one has a lage multivaiate dataset that has been seveely coupted by missing values. Fotunately, in many situations, the undelying dataset is of intinsic low dimension, so the missing values may be infeed fom the obseved ones. Hence the gowing inteest on lowank matix completion (LRMC), which, as the name suggests, aims to infe the missing enties in a patially obseved lowank data matix [1]. Applications of this poblem aise in a wide vaiety of pactical scenaios, such as face ecognition [2], ecommende systems and collaboative filteing [3] and netwok topology estimation [4]. Given a d N data matix X, and a matix Ω indicating the locations of its obseved enties, existing theoy in LRMC has mainly focused on the following poblem: Poblem 1. ( ) Detemine conditions on X and Ω to guaantee that if X is a ank- matix, then X is the only ank- matix that agees with X on Ω. Examples of the conditions on X include bounded coheence [1, 5 12] and geneicity [13 15]. Coheence is a paamete indicating how aligned ae the columns in a matix with espect to the canonical axes; typically the lowe coheence the bette. Geneicity essentially asks that the columns of X ae dawn independently accoding to an absolutely continuous distibution with espect to the Lebesgue measue on an - dimensional subspace in geneal position (see Figue 1 fo some intuition). Examples of the conditions on Ω include unifom andom sampling [1, 5 11], biased andom sampling accoding to the coheence of X [12], and deteministic combinatoial conditions [15]. Thee even exist a wide vaiety of pactical methods that will povably complete subsampled low-ank matices with high pobability. Examples include nuclea nom minimization [1, 5 9, 12], altenating minimization [11], and methods based on singula value decomposition [9, 10, 16 18]. In many situations, though, one does not know a pioi whethe the given dataset is low-dimensional. To build some intuition, conside the full-data case. Imagine we ae given a data matix X, and we want to detemine whethe it is lowank. One thing we can do is compute its singula values. If only a few of them ae nonzeo, then we can be sue that X is indeed low-ank. But if data is missing, we can no longe compute singula values. One thing we can do is suppose that X is ank-1, and ty to find a ank-1 matix that agees with the obseved data. Of couse, if thee exists no such matix, then X cannot be ank-1, and we know that ank(x) 2. We can iteatively epeat this pocess until we find a ank- matix that agees with the obseved enties. At this point we know ank(x). Nonetheless, depending on X and Ω, it is possible to find a ank- matix that agees with X on Ω even if X is not ank. In othe wods, we could be deceived into thinking that X is ank-, when it tuly is of highe ank. This aises the following question: can we detemine whethe X is tuly ank, based on a pope subset of its enties? In geneal, the answe to this question is no. Fo instance, suppose we obseve all but the top-left enty of X, which will be denoted by x 11. Futhe suppose that fo evey j = 1,..., N, all the obseved enties of the j th column of X ae equal to some constant c j 0. This suggests that X is ank-1. Without any assumption on X, x 11 could take any value. The ank of X will be 1 if and only if x 11 = c 1, and 2 othewise. But since x 11 is unknown, we cannot know which is the case. On the othe hand, suppose in addition that the columns of X ae dawn independently accoding to an absolutely continuous distibution with espect to the Lebesgue measue on some undelying subspace (maybe 1-dimensional, but we do not know). This condition essentially asks that the columns of X ae dawn geneically fom some subspace (see Figue 1 fo some intuition). If the undelying subspace is of dimension > 1, then the pobability that any two columns of X ae linealy dependent is zeo. Since all the obseved enties of the j th column ae equal to c j, it follows that columns 2,..., N ae

2 linealy dependent. Then with pobability 1, the undelying subspace is 1-dimensional, and X is ank-1. Of couse, establishing conditions on X is not enough. Fo instance, conside the same scenaio as befoe, but suppose instead that we obseve none of the enties in the fist ow of X. Then thee exist infinitely many ank-1 matices that agee with the obseved enties. It is possible that X is one of these matices, but it is also possible that X is eally ank 2. In this case, because of the obseved locations, we cannot know whethe X is ank-1, even if we assume that its columns ae geneic o ideally coheent. We thus have the following convese of Poblem 1: Poblem 2. ( ) Detemine conditions on X and Ω to guaantee that if thee is a ank- matix that agees with X on Ω, then X is indeed ank-. In this pape we study Poblem 2. Ou main esult shows that if X is a geneic matix obseved on Ω satisfying a deteministic combinatoial condition, and thee is a ank- matix that agees with X on Ω, then X is indeed ank- with pobability 1. While ou condition on Ω is combinatoial, we povide a deteministic efficient algoithm to veify whethe this condition is satisfied. Futhemoe, we show that this condition is satisfied with high pobability if X is obseved on as little as O(max{, log d}) enties pe column, selected unifomly at andom. This stengthens existing esults in LRMC, allowing to dop the assumption that X is known a pioi to be low-ank. Oganization of the pape In Section II we fomally state the poblem and ou main esults. In Section III we discuss the impotance of Poblem 2, and why we should cae. In Section IV we pesent ou efficient algoithm to veify whethe ou combinatoial conditions on Ω ae satisfied, and we pove ou statements in Section V. II. MODEL AND MAIN RESULTS Let X be a d N data matix, and Ω be the d N matix with binay enties indicating the obseved locations of X: the (i, j) th enty of Ω will be 1 if the (i, j) th enty of X is obseved, and zeo othewise. We say that a matix agees with X on Ω if it is equal to X on all the nonzeo locations of Ω. As mentioned in Section I, since data is missing, we can no longe compute the singula values of X to detemine its ank. Instead, we can ty to find a ank-1 matix that agees with X on Ω. If thee exists no such matix, then X cannot be ank-1, and we know that ank(x) 2. We can iteatively epeat this pocess until we find a ank- matix that agees with X on Ω. At this point we know ank(x), and we want to detemine whethe ank(x) =. Depending on X and Ω, it is possible to find a ank- matix that agees with X on Ω even if ank(x) >. We thus want to establish conditions on X and Ω to guaantee that if ank(x), and thee is a ank- matix that agees with X on Ω, then ank(x) =. Fig. 1. Each column in a ank- matix X coesponds to a point in an - dimensional subspace S. In these figues, S is a 2-dimensional subspace (plane) in geneal position. In the left, the columns of X ae dawn geneically fom S, that is, independently accoding to an absolutely continuous distibution with espect to the Lebesgue measue on S, fo example, accoding to a gaussian distibution on S. In this case, the pobability of obseving a sample as in the ight, whee all columns lie in a line inside S, is zeo. We will show that this will be the case if Ω satisfies condition C1 below, and X satisfies the following assumption: (A1) The columns of X ae dawn independently accoding to an absolutely continuous distibution with espect to the Lebesgue measue on an -dimensional subspace in geneal position. A1 essentially asks that X is a geneic ank- matix. To bette undestand this, let G(, R d ) denote the Gassmannian manifold of -dimensional subspaces in R d. Obseve that each d N ank- matix X can be uniquely epesented in tems of a subspace S G(, R d ) (spanning the columns of X) and an N coefficient matix Θ. Let ν G denote the unifom measue on G(, R d ), and let ν Θ denote the Lebesgue measue on R N. Equivalent to A1, ou statements hold fo almost evey (a.e.) ank- matix X, with espect to the poduct measue ν G ν Θ. Ou condition on Ω builds on the esults in [15], which show that a set of enties in X obseved in the ight locations, will detemine, up to finite choice, the -dimensional subspaces that may explain the columns in X. The key insight of ou pape is that any additional column obseved on + 1 enties can be used to veify consistency: if ank(x) >, such additional column will agee with none of the candidate - dimensional subspaces, and equivalently, no ank- matix can agee with X on Ω. This is pecisely the contapositive of the statement we ae looking fo. Let us now intoduce the constaint matix Ω, as defined in [15], that will allow us to easily expess ou condition on Ω. Definition 1 (Constaint Matix). Let k 1,j, k 2,j,..., k lj,j denote the indices of the l j obseved enties in the j th column of X. If l j, define Ω j as the empty matix. Othewise, define Ω j as the d (l j ) matix, whose i th column has the value 1 in ows k 1,j, k 2,j,..., k,j and k +i,j, and zeos elsewhee. Define the constaint matix Ω as Ω = [Ω 1 Ω N ].

3 Fo example, if k 1 = 1, k 2 = 2,..., k lj = l j, then Ω j = 1 I 0 l j } l j } d l j, whee 1 denotes a block of all 1 s and I the identity matix. The key insight behind this constuction is that obseving moe than enties in a column of X places constaints on the -dimensional subspaces that can explain it. Fo example, if we obseve + 1 enties of a paticula column, then not all -dimensional subspaces will be consistent with the enties. If we obseve moe enties, then even fewe subspaces will be consistent with them. In effect, each obseved enty, in addition to the fist obsevations, places one constaint that an -dimensional subspace must satisfy in ode to be consistent with the obsevations. The matix Ω encodes all these constaints. C1 below is a simple, combinatoial condition on Ω that guaantees that if ank(x) >, then the obseved enties will poduce inconsistent constaints, implying that no -dimensional subspace can explain X, o equivalently, that no ank- matix can agee with X on Ω. Given a matix, let n( ) denote its numbe of columns and m( ) the numbe of its nonzeo ows. With this, we ae eady to pesent ou condition on Ω: (C1) The constaint matix Ω contains a column ω, in addition to disjoint matices { Ω τ } τ=1, each of size d (d ), such that fo evey τ: (C2) Evey matix Ω fomed with a subset of the columns in Ω τ satisfies m(ω ) n(ω ) +. (1) In wods, C2 asks that evey subset of n columns of Ω τ has at least n + nonzeo ows. Example 1. The following sampling satisfies C2. 1 } Ω τ = I d. While condition C2 is combinatoial, we show in Section IV that one can easily veify whethe this condition is satisfied by checking the dimension of the null-space of a spase matix. This is summaized in Algoithm 1, which in tun povides a pactical citeia to veify whethe X is indeed ank-. The pape s main esult is the following theoem, which gives an answe to Poblem 2. It states that fo almost evey matix X with ank(x) (thus establishing conditions on X, namely geneicity), if Ω satisfies condition C1 and thee is a ank- matix that agees with X on Ω, then X is indeed ank-. The poof is given in Section V. Theoem 1. Let A1 hold. Suppose ank(x) and Ω satisfies C1. If thee exists a ank- matix that agees with X on Ω, then ank(x) = with pobability 1. In a nutshell, Theoem 1 states that if X is geneic, and thee is a ank- matix that agees with X in the ight places, then X must be ank- with pobability 1. Futhemoe, the next theoem states that sampling pattens satisfying C1 appea with high pobability unde unifom andom sampling schemes with only O(max{, log d}) samples pe column. The poof is given in Section V. Theoem 2. Let 0 < 1 be given. Suppose d 6, N > (d ), and that each column of Ω has at least l nonzeo enties, distibuted unifomly at andom and independently acoss columns, with l max {12 (log( d ) + 1), 2}. (2) Then with pobability at least 1, Ω will satisfy C1. In a nutshell, Theoem 2 implies that if X is geneic, and thee is a ank- matix that agees with X on enough enties selected unifomly at andom, then X must be ank-. III. WHY SHOULD WE CARE? In many moden applications one is given an incomplete data matix, and aims to infe its missing enties. If the intinsic dimension of the complete (yet unknown) dataset is too lage fo the numbe of obseved enties, nothing can be done to infe the missing enties. Fotunately, in many situations the ank of the complete data matix is vey low, whence the whole matix can be infeed fom a small faction of its enties. The impotance of Poblem 2 is that in many situations we do not know a pioi the ank of the complete data matix X. In such case, all we can do is suppose that the matix is ank-, and ty to find a ank- matix that agees with the obseved data. If we find such a matix, ou hope is that it is X. But it is possible that it is not, even if it is the only ank- matix that agees with the obseved data. Theoem 1 states that if X is geneic, and Ω satisfies C1, then this will not be the case. Futhemoe, Theoem 2 implies that if X is geneic, and thee is a ank- matix that agees with X on O(max{, log d}) enties pe column, selected unifomly at andom, then X must be ank-. These ae incedibly good news! This implies that if ou data matix is known to be geneic (as is often the case), and Ω satisfies C1 (which will happen with high pobability accoding to Theoem 2), then we do not have to woy about being deceived into thinking that ou data is ank-, when it tuly is not.

4 Theoem 2 is paticulaly elevant because a lage potion of the theoy and methods fo LRMC opeate unde unifom andom sampling schemes with O( log d) obsevations pe column. We often used these methods without knowing whethe X is tuly low-ank, hoping (but not knowing) that it tuly is. Theoem 2 stengthens these esults fo data matices that ae both geneic and of bounded coheence (as is often the case). In such case, now we know that if we un any of these methods, and find a ank- matix that agees with the obseved data, then the undelying matix is tuly ank-. IV. SOLVING PROBLEM 2 IN PRACTICE Theoem 2 states that samplings with O(max{, log d}) obsevations pe column dawn unifomly at andom will satisfy C1 with high pobability. In many situations, though, sampling is not unifom. Fo instance, in vision, occlusion of objects can poduce missing data in vey non-unifom andom pattens. In cases like this, one would still like to veify whethe a given matix is ank. We can do this using Theoem 1 diectly. Fo example, we can split Ω (e.g., andomly) into disjoint matices { Ω τ } τ=1, and veify whethe each Ω τ satisfies C2. Of couse, C2 is a combinatoial condition, hence veifying it diectly may be computationally pohibitive, especially fo lage d. Fotunately, we can easily veify whethe a matix Ω τ satisfies C2 by checking the dimension of the null-space of a spase matix. This is summaized in Algoithm 1, which in tun povides a pactical citeia to veify whethe X is indeed ank-. To pesent this algoithm, let us intoduce the matix A that will allow us to detemine efficiently whethe a sampling Ω τ satisfies C2. To this end, let ω j denote the j th column of Ω τ, and let U be a d matix dawn accoding to ν U, an absolutely continuous distibution with espect to the Lebesgue measue on R d. Let U ωj denote the estiction of U to the nonzeo ows in ω j. Let a ωj R +1 be a nonzeo vecto in ke U T ω j, and a j be the vecto in R d with the enties of a ωj in the nonzeo locations of ω j and zeos elsewhee. Finally, let A denote the d (d ) matix with {a j } d j=1 as columns. Algoithm 1 will veify whethe dim ke A T =, and this will detemine whethe Ω τ satisfies C2. The key insight behind Algoithm 1 is that A encodes the infomation of the pojections of S = span{u} onto the canonical coodinates indicated by Ω τ. We know fom Theoem 1 in [19] that ν U - a.s., these pojections will uniquely detemine S if and only if dim ke A T =, which will be the case if and only if Ω τ has d columns and Ω τ satisfies C2. We have thus shown the following lemma, which states that with pobability 1, Algoithm 1 will detemine whethe Ω τ satisfies C2. Lemma 1. Let Ω τ be a matix fomed with d columns of Ω. Then ν U -a.s., Ω τ satisfies C2 if and only if dim ke A T =. V. PROOFS The poof of Theoem 1 is lagely based on Theoem 1 and Lemma 8 in [15], which togethe give a combinatoial Algoithm 1: Detemine whethe Ω τ satisfies C2. Input: Matix Ω τ with d columns of Ω. - Daw U R d dawn accoding to ν U. - fo j = 1 to d do - a ωj = nonzeo vecto in ke U T ω j. - a j = vecto in R d with enties of a ωj in the nonzeo locations of ω j and zeos elsewhee. - A = matix fomed with {a j } d j=1 as columns. - if dim ke A T = then - Output: Ω τ satisfies C2. - else - Output: Ω τ does not satisfy C2. condition of sampling pattens that can only be completed in finitely many ways. We combine these esults in the following lemma. Recall that Ω denotes the matix encoding all the constaints imposed by Ω, as defined in Section II. Lemma 2. Let A1 hold, and suppose ank(x) =. If Ω satisfies C3 below, then with pobability 1 thee exist at most finitely many ank- matices that agee with X on Ω. (C3) The constaint matix Ω contains disjoint matices { Ω τ } τ=1, each of size d (d ), such that each Ω τ satisfies C2. Lemma 2 implies that if ank(x) = and Ω satisfies C3, then thee exist at most finitely many -dimensional subspaces that may explain the columns of X. We now use Lemma 2 to show that this will also be the case if ank(x) >. Coollay 1. Let A1 hold, and suppose ank(x). If Ω satisfies C3, then with pobability 1 thee exist at most finitely many ank- matices that agee with X on Ω. Poof. If ank(x) =, the coollay follows diectly fom Lemma 2. Now suppose ank(x) >. If thee is no ank- matix that agees with X on Ω, then the coollay is tivially tue. Now suppose that thee is at least one ank- matix Y that agees with X on Ω. By Lemma 2, with pobability 1 thee exist at most finitely many ank- matices that agee with Y on Ω. It follows that thee exist at most finitely many ank- matices that agee with X on Ω. Coollay 1 shows that if ank(x) and Ω satisfies C3, then thee exist at most finitely many -dimensional subspaces that may explain the columns of X. Now we will show that any additional column obseved on + 1 enties can be used to veify whethe ank(x) = o ank(x) >. The main intuition is that if ank(x) >, the additional column will agee with none of the candidate -dimensional subspaces. Equivalently, no ank- matix can agee with X on Ω. This will be the contapositive of the statement in Theoem 1.

5 Poof. (Theoem 1) Suppose ank(x) and that Ω satisfies C1. Then Ω also satisfies C3. By Coollay 1, thee ae at most finitely many -dimensional subspaces that may explain the columns in X. Let S be one of these subspaces. In addition, let ω denote a column in Ω that is not in { Ω τ } τ=1. Recall that each column in Ω coesponds to a column in Ω, which in tun coesponds to a column in X. Let x denote the column in X coesponding to ω. Next suppose fo contapositive that ank(x) = >. This means that the columns of X lie in an -dimensional subspace S. Obseve that fo ν G -a.e. -dimensional subspace S, the estiction of S to l coodinates is R l. Let Sω, S ω and x ω denote the estictions of S, S and x to the nonzeo ows in ω. Since ω has exactly +1 nonzeo enties by constuction, it follows that Sω = R +1. In contast, S ω is an -dimensional subspace of R +1. Recall that x is dawn accoding to an absolutely continuous with espect to the Lebesgue measue on S. Equivalently, x ω is dawn accoding to an absolutely continuous distibution with espect to the Lebesgue measue on Sω = R +1. Intuitively, this means that that x ω could take any value in R +1. Since S ω is an -dimensional subspace of R +1, it has measue zeo. It follows that almost suely, x ω S ω. This is tue fo all of the finitely many -dimensional subspaces that could explain the columns in X. It follows that no -dimensional subspace can explain the obseved enties of X. Equivalently, thee exists no ank- matix that agees with X on Ω. This is the contapositive of the statement in Theoem 1. The poof of Theoem 2 is based on Lemma 9 in [15], which shows that sampling pattens satisfying C2 appea with high pobability unde unifom andom sampling schemes with only O(max{, log d}) samples pe column. We estate this esult as the following lemma. Lemma 3. Let the assumptions of Theoem 2 hold, and let Ω τ be a matix fomed with d columns of Ω. With pobability at least 1 d, Ω τ will satisfy C2. Theoem 2 follows diectly fom Lemma 3 by applying a union bound. Poof. (Theoem 2) If N > (d ), andomly select disjoint matices {Ω τ } τ=1, each fomed with d columns of Ω. Let E τ denote the even that Ω τ fails to satisfy C2. Union bounding ove τ, we may uppe bound the pobability that Ω fails to satisfy C1 by P(E τ ) < τ=1 τ=1 d < τ=1 =, whee the fist inequality follows by Lemma 3. ank- with pobability 1. Ou condition on Ω is combinatoial, yet we povide a deteministic efficient citeia to veify whethe this condition is satisfied. Futhemoe, we show that this condition is satisfied with high pobability if X is obseved on as little as O(max{, log d}) enties pe column, selected unifomly at andom. This stengthens existing esults in LRMC, allowing to dop the assumption that X is known a pioi to be low-ank. REFERENCES [1] E. Candès and B. Recht, Exact matix completion via convex optimization, Foundations of Computational Mathematics, [2] R. Basi and D. Jacobs, Lambetian eflectance and linea subspaces, IEEE Tansactions on Patten Analysis and Machine Intelligence, [3] J. Rennie and N. Sebo, Fast maximum magin matix factoization fo collaboative pediction, Intenational Confeence on Machine Leaning, [4] B. Eiksson, P. Bafod, J. Sommes and R. Nowak, DomainImpute: Infeing Unseen Components in the Intenet, IEEE INFOCOM Mini- Confeence, [5] E. Candès and T. Tao, The powe of convex elaxation: nea-optimal matix completion, IEEE Tansactions on Infomation Theoy, [6] D. Goss, Recoveing low-ank matices fom few coefficients in any basis, IEEE Tansactions on Infomation Theoy, [7] B. Recht, A simple appoach to matix completion, Jounal of Machine Leaning Reseach, [8] S. Bhojanapalli and P. Jain, Univesal matix completion, Intenational Confeence on Machine Leaning, [9] Y. Chen, Incoheence-optimal matix completion, IEEE Tansactions on Infomation Theoy, [10] R. Keshavan, A. Montanai and S. Oh, Matix completion fom a few enties, IEEE Tansactions on Infomation Theoy, [11] P. Jain, P. Netapalli and S. Sanghavi, Low-ank matix completion using altenating minimization, ACM Symposium on Theoy Of Computing, [12] Y. Chen, S. Bhojanapalli, S. Sanghavi and R. Wad, Coheent matix completion, Intenational Confeence on Machine Leaning, [13] A. Singe and M. Cucuingu, Uniqueness of low-ank matix completion by igidity theoy, SIAM Jounal on Matix Analysis and Applications, [14] F. Kiály and R. Tomioka, A combinatoial algebaic appoach fo the identifiability of low-ank matix completion, Intenational Confeence on Machine Leaning, [15] D. Pimentel-Alacón, N. Boston and R. Nowak, A chaacteization of deteministic sampling pattens fo low-ank matix completion, Alleton, [16] J. Cai, E. Candès and and Z. Shen, A singula value thesholding algoithm fo matix completion, SIAM Jounal on Optimization, [17] S. Ma, D. Goldfab, L. Chen, Fixed point and Begman iteative methods fo matix ank minimization, Mathematical Pogamming, [18] E. Chunikhina, R. Raich and T. Nguyen, Pefomance analysis fo matix completion via iteative had-thesholded SVD, IEEE Statistical Signal Pocessing Wokshop, [19] D. Pimentel-Alacón, N. Boston and R. Nowak, Deteministic conditions fo subspace identifiability fom incomplete sampling, IEEE Intenational Symposium on Infomation Theoy, VI. CONCLUSIONS In this pape we show that if a geneic data matix X is obseved on the locations indicated by Ω satisfying a deteministic combinatoial condition, and thee is a ank- matix that agees with the obseved data, then X is indeed