arxiv: v1 [stat.ml] 28 Mar 2017

Transcription

1 Algebraic Variety Moels for High-Rank Matrix Completion Greg Ongie Rebecca Willett Robert D. Nowak Laura Balzano March 29, 27 arxiv:73.963v [stat.ml] 28 Mar 27 Abstract We consier a generalization of low-rank matrix completion to the case where the ata belongs to an algebraic variety, i.e., each ata point is a solution to a system of polynomial equations. In this case the original matrix is possibly high-rank, but it becomes low-rank after mapping each column to a higher imensional space of monomial features. Many well-stuie extensions of linear moels, incluing a ne subspaces an their union, can be escribe by a variety moel. In aition, varieties can be use to moel a richer class of nonlinear quaratic an higher egree curves an surfaces. We stuy the sampling requirements for matrix completion uner a variety moel with a focus on a union of a ne subspaces. We also propose an e cient matrix completion algorithm that minimizes a convex or non-convex surrogate of the rank of the matrix of monomial features. Our algorithm uses the wellknown kernel trick to avoi working irectly with the high-imensional monomial matrix. We show the propose algorithm is able to recover synthetically generate ata up to the preicte sampling complexity bouns. The propose algorithm also outperforms stanar low rank matrix completion an subspace clustering techniques in experiments with real ata. Introuction Work in the last ecae on matrix completion has shown that it is possible to leverage linear structure in orer to interpolate missing values in a low-rank matrix []. The high-level iea of this work is that if the ata efining the matrix belongs to a structure having fewer egrees of freeom than the entire ataset, that structure provies reunancy that can be leverage to complete the matrix. The assumption that the matrix is low-rank is equivalent to assuming the ata lies on (or near) a low-imensional linear subspace. It is of great interest to generalize matrix completion to exploit low-complexity nonlinear structures in the ata. Several avenues have been explore in the literature, from generic manifol learning [2], to unions of subspaces [3, 4], to low-rank matrices perturbe by a nonlinear monotonic function [5, 6]. In each case missing ata has been consiere, but there lacks a clear, unifying framework for these ieas. In this work we stuy the problem of completing a matrix whose columns belong to an algebraic variety, i.e., the set of solutions to a system of polynomial equations [7]. This is a strict generalization of the linear (or a ne) subspace moel, which can be written as the set of points satisfying a system of linear equations. Unions of subspaces an unions of a ne spaces also are algebraic varieties. In aition, a much richer class of non-linear curves, surfaces, an their unions, are capture by a variety moel. The matrix completion problem using a variety moel can be formalize as follows. Let X = x,...,x s 2 R n s be a matrix of s ata points where each column x i 2 R n. Define : R n! R N as Department of EECS, University of Michigan, Ann Arbor, MI, USA. (gongie@umich.eu). Department of ECE, University of Wisconsin-Maison, Maison, WI, USA. (willett@iscovery.wisc.eu). Department of ECE, University of Wisconsin-Maison, Maison, WI, USA. (rnowak@wisc.eu). Department of EECS, University of Michigan, Ann Arbor, MI USA. (girasole@umich.eu).

2 the mapping that sens the vector x =(x,...,x n ) to the vector of all monomials in x,...,x n of egree at most, an let (X) enote the matrix that results after applying to each column of X, which we call the lifte matrix. We will show the lifte matrix is rank eficient if an only if the columns of X belong to an algebraic variety. This motivates the following matrix completion approach: min ˆX rank ( ˆX) such that P ( ˆX) =P (X) () where P ( ) represents a projection that restricts to some observation set {,...,n} {,...,s}. The rank of ( ˆX) epens on the choice of the polynomial egree an the unerlying complexity of the variety, in a sense we will make precise. Figure shows two examples of atasets that have low-rank in the lifte space for i erent polynomial egree. In this work we investigate the factors that influence the sampling complexity of varieties as well as algorithms for completion. The challenges are (a) to characterize varieties having low-rank (an therefore few egrees of freeom) in the lifte space, i.e., etermine when (X) is low-rank, (b) evise e cient algorithms for solving () that can exploit these few egrees of freeom in a matrix completion setting, an (c) etermine the trae-o s relative to existing matrix completion approaches. This work contributes consierable progress towars these goals. =2,R=5 =2,R=7 =3,R= Figure : Data belonging to algebraic varieties in R 3. The original ata is full rank, but a nonlinear embeing of the matrix to a feature space consisting of monomials of egree at most is low-rank with rank R, inicating the ata has few egrees of freeom. For a given variety moel, we seek to escribe the egrees of freeom that etermine the sampling complexity of the moel. For example, it is well-known that n n rank r matrix can be complete from O(rn polylog n) sample entries uner stanar incoherence assumptions [8]. This is very close to the O(rn) egrees of freeom for such a matrix. Similarly, the egrees of freeom in the lifte space is O(RN), where R is the rank of the lifte matrix an N is the number of higher-egree monomials. This is suggestive of the number of samples require for completion in the lifte space an in turn the number of samples require in the original observation space. We note that although N>nan R>r, for many varieties r/n R/N, implying potential for completion in the lifte space. Our contributions are as follows. We ientify bouns on the rank of a matrix (X) whenthe columns of the ata matrix X belong to an algebraic variety. We stuy how many entries of such a matrix shoul be observe in orer to recover the full matrix from an incomplete sample. We show as a case stuy that monomial representations prouce low-rank representations of unions of subspaces, an we characterize the rank. The stanar union of subspace representation as a iscrete collection of iniviual subspaces is inherently non-smooth in nature, whereas the algebraic variety allows for a purely continuous parameterization. This leas to general algorithms for completion of a ata matrix whose columns belong to a variety. The algorithms performance are showcase on ata simulate as a union of subspaces, a union of low-imensional parametric surfaces, an real ata from a motion segmentation ataset an 2

3 a motion capture ataset. The simulations show that the performance of our algorithm matches our preictions an outperforms other methos. In aition, the analysis of the egrees of freeom associate with the propose representations introuces several new research avenues at the intersection of nonlinear algebraic geometry an ranom matrix theory.. Relate Work There has been a great eal of research activity on matrix completion problems since [], where the authors showe that one can recover an incomplete matrix from few entries using a convex relaxation of the rank minimization optimization problem. At this point it is even well-known that O(rn) entries are necessary an su cient [9] for almost every matrix as long as the measurement pattern satisfies certain eterministic conitions. However, these methos an theory are restricte to low-rank linear moels. A great eal of real ata exhibit nonlinear structure, an so it is of interest to generalize this approach. Work in that irection has ealt with union of subspaces moels [3,,, 2, 3], locally linear approximations [2], as well as low-rank moels perturbe by an arbitrary nonlinear link function [5, 6, 4]. In this paper we instea seek a more general moel that captures both linear an nonlinear structure. The variety moel has as instances low-rank subspaces an their union as well as quaratic an higher egree curves an surfaces. Work on kernel PCA (cf., [5, 6]) leverage similar geometry to ours. In Kernel Spectral Curvature Clustering [7], the authors similarly consier clustering of ata points via subspace clustering in a lifte space using kernels. These works are algorithmic in nature, with promising numerical experiments, but o not systematically consier missing ata or analyze relative egrees of freeom. This paper also has close ties to algebraic subspace clustering (ASC) [8, 9, 2, 2], also known as generalize PCA. Similar to our approach, the ASC framework moels unions of subspaces as an algebraic variety, an makes use of monomial liftings of the ata to ientify the subspaces. Characterizations of the rank of ata belonging to union of subspaces uner the monomial lifting are use in the ASC framework [2] base on results in [22]. The i erence of the results in [22] an those in Prop. is that ours hol for monomial liftings of all egrees, not just k, wherek is the number of subspaces. Also, the main focus of ASC is to recover unions of subspaces or unions of a ne spaces, whereas we consier ata belonging to a more general class of algebraic varieties. Finally, the ASC framework has not been aapte to the case of missing ata, which is the main focus of this work. 2 Variety Moels 2. Toy example As a simple example to illustrate our approach, consier a matrix x, x X =,2 x,6 2 R 2 6 x 2, x 2,2 x 2,6 whose six columns satisfy the quaratic equation c + c x,i + c 2 x 2,i + c 3 x 2,i + c 4 x,i x 2,i + c 5 x 2 2,i = (2) 3

4 for i =,...,6 an some unknown constants c,...,c 5 that are not all zero. Generically, X will be full rank. However, suppose we vertically expan each column of the matrix to make a 6 6 matrix Y = x, x,2 x,6 x 2, x 2,2 x 2,6 x 2, x 2,2 x 2,6 x, x 2, x,2 x 2,2 x,6 x 2,6 x 2 2, x 2 2,2 x 2 2,6 i.e., we augment each column of X with a an with the quaratic monomials x 2,i, x,ix 2,i, x 2 2,i. This allows us to re-express the polynomial equation (2) as the matrix-vector prouct Y T c = where c =(c,c,..,c 5 ) T. In other wors, Y is rank eficient. Suppose, for example, that we are missing entry x, of X. SinceX is full rank, there is no way to uniquely complete the missing entry by leveraging linear structure alone. Instea, we ask: Can we complete x, using the linear structure present in Y? Due to the missing entry x,, the first column of Y will having the following pattern of missing entries: (,,x 2,,,,x 2 2, )T. However, assuming the five complete columns in Y are linearly inepenent, we can uniquely etermine the nullspace vector c up to a scalar multiple. Then from (2) we have, C A c 3 x 2, +(c + c 4 x 2, )x, = c c 2 x 2, c 5 x 2 2, In general, this equation will yiel at most two possibilities for x,. Moreover, there are conitions where we can uniquely recover x,, namely when c 3 = an c + c 4 x 2, 6=. This example shows that even without a priori knowlege of the particular polynomial equation satisfie by the ata, it is possible to uniquely recover missing entries in the original matrix by leveraging inuce linear structure in the matrix of expane monomials. We now show how to consierably generalize this example to the case of ata belonging to an arbitrary algebraic variety. 2.2 Formulation Let X = x,...,x s 2 R n s be a matrix of s ata points where each column x i 2 R n.define : R n! R N as the mapping that sens the vector x =(x,...,x n ) to the vector of all monomials in x,...,x n of egree at most : (x) =(x ) apple 2 R N (3) where =(,..., n ) is a multi-inex of non-negative integers, with x := x x n n, an := + + n. In the context of kernel methos in machine learning, the map is often calle a polynomial feature map [23]. Borrowing this terminology, we call (x) afeature vector, theentries of (x) features, an the range of feature space. Note that the number of features is given by N = N(n, ) = n+ n = n+, the number of unique monomials in n variables of egree at most. When X =[x,...,x s ] is an n s matrix, we use (X) to enote the N s matrix [ (x ),..., (x s )]. The problem we consier is this: can we complete a partially observe matrix X uner the assumption that (X) is low-rank? This can be pose as the optimization problem given above in Equation (). We give a practical algorithm for solving a relaxation of () in Section 4. Similar to previous work cite above on using polynomial feature maps, our metho leverages the kernel trick for e cient computations. However, it woul be naïve to think of the associate analysis as applying known results on matrix completion sample complexities to our high-imensional feature space. In particular, if we observe m 4

5 entries per column in a rank-r matrix of size n s an apply the polynomial feature map, then in the feature space we have M = m+ entries per column in a rank-r matrix of size N s. Generally, the number of samples, rank, an imensional all grow in the mapping to feature space, but they grow at i erent rates epening on the unerlying geometry; it is not immeiately obvious what conitions on the geometry an sampling rates impact our ability to etermine the missing entries. In the remainer of this section, we show how to relate the rank of (X) to the unerlying variety, an we stuy the sampling requirements necessary for the completion of the matrix in feature space. 2.3 Rank properties To better unerstan the rank of the matrix (X), we introuce some aitional notation an concepts from algebraic geometry. Let R[x] enote the space of all polynomials with real coe cients in n variables x =(x,...,x n ). We moel a collection of ata as belonging to a real (a ne) algebraic variety [7], which is efine as the common zero set of a system of polynomials P R[x]: V (P )={x 2 R n : f(x) = for all f 2 P }. Suppose the variety V (P ) is efine by the finite set of polynomials P = {f,...,f q }, where each f i has egree at most. Let C 2 R N q be the matrix whose columns are given by the vectorize coe cients (c,i ) apple of the polynomials f i (x), i=,...,q in P. Then the columns of X belong to the variety V (P ) if an only if (X) T C =. In particular, assuming the columns of C are linearly inepenent, this shows that (X) has rank apple min(n q, s). In particular, when the number of ata points s>n q, then (X) is rank eficient. However, the exact rank of (X) coul be much smaller than min(n q, s), especially when the egree is large. This is because the coe cients c of any polynomial that vanishes at every column of X satisfies (X) T c =. We will fin it useful to ientify this space of coe cients with a finite imensional vector space of polynomials. Let R [x] be the space of all polynomials in n real variables of egree at most. We efine vanishing ieal of egree corresponing to a set X R n, enote by I (X ), to be subspace of polynomials belonging to R [x] that vanish at all points in X : I (X ):={f 2 R [x] :f(x) = for all x 2X}. (4) We also efine the non-vanishing ieal of egree corresponing to X, enote by S (X ), to be the orthogonal complement of I (X )inr [x]: S (X ):={g 2 R [x] :hf,gi = for all f 2I (X )}, (5) where the inner prouct hf,gi of polynomials f,g 2 R [x] is efine as the inner prouct of their coe cient vectors. Hence, the rank of a ata matrix in feature space can be expresse in terms of the imension of non-vanishing ieal of egree corresponing to X = {x,...,x s }, the set of all columns of X. Specifically, we have rank (X) =min(r, s) where R =ims (X )=N im I (X ). (6) This follows from the rank-nullity theorem, since R [x] has imension N. In general the imension of the space I (X ) or S (X )isi culttoeterminewhenx is an arbitrary set of points. However, if we assume X is a subset of a variety V,sinceI (V ) I (X ) we immeiately have the boun rank (X) apple im S (V ). (7) In certain cases im S (V ) can be compute exactly or boune using properties of the polynomials efining V. For example, it is possible to compute the imension of S (V ) irectly from a Gröbner basis for the vanishing ieal associate with V [7]. In Section 3 we show how to boun the imension of S (V ) in the case where V is a union of subspaces. 5

6 2.4 Sampling rate Informally, the egrees of freeom of a class of objects is the minimum number of free variables neee to escribe an element in that class uniquely. For example, a n s rank r matrix has r(n + s r) egrees of freeom: nr parameters to escribe r linearly inepenent columns making up a basis of the column space, an r(s r) parameters to escribe the remaining s r columns in terms of this basis. It is impossible to uniquely complete a matrix in this class if we sample fewer than this many entries. We can make a similar argument to specify the minimum number of samples neee to uniquely complete a matrix that is low-rank when mappe to feature space. First, we characterize how missing entries of the ata matrix translate to missing entries in feature space. For simplicity, we will assume a sampling moel where we sample a fixe number of entries m from each column of the original ata matrix. Let x 2 R n represent a single column of the ata matrix, an {,...,n} with m = enote the inices of the sample entries of x. The pattern of reveale entries in (x) correspons to the set of multi-inices: { =(,..., n ): apple, i = for all i 2 c }, which has the same carinality as the set of all monomials of egree at most in m variables, i.e., If we call this quantity M, then the ratio of reveale entries in (x) to the feature space imension is m+. M N = m+ n+ = (m + )(m + ) (m + ) (n + )(n + ) (n + ), which is on the orer of ( m n ) for small. More precisely, we have the bouns an consequently m n apple M N apple m n apple M N m +, (8) n apple m n + n. (9) In total, observing m entries per column of the ata matrix translates to M entries per column in feature space. Suppose the N s lifte matrix (X) is rank R. By the preceing iscussion, we nee least R(N + s R) entries of the feature space matrix (X) to complete it uniquely among the class of all N s matrices of rank R. Hence, at minimum we nee to satisfy Ms R(N + s R). () Let m enote the minimal value of m such that M = m+ achieves the boun (), an set M = m +. Diviing () through by the feature space imension N an s gives M R N + s R R = N N s s + R R, () N s an so from (9) we see we can guarantee this boun by having := m n R s + R N R, (2) s an this in fact will result in tight satisfaction of () because (M /N ) m /n for small an large n. 6

7 At one extreme where the matrix (X) is full rank, then R/s = or R/N = an accoring to (2) we nee, i.e., full sampling of every ata column. At the other extreme where instea we have many more ata points than the feature space rank, R/s, then (2) gives the asymptotic boun (R/N). The above iscussion bouns the egrees of freeom of a matrix that is rank-r in feature space. Of course, the propose variety moel has potentially fewer egrees of freeom than this, because aitionally the columns of the lifte matrix are constraine to lie in the image of the feature map. We use the above boun only as a rule of thumb for sampling requirements on our matrix. Furthermore, we note that sample complexities for stanar matrix completion often require that locations are observe uniformly at ranom, whereas in our problem the locations of observations in the lifte space will necessarily be structure. However, there is recent work that shows matrix completion can sucee without these assumptions [9, 24] that gives reason to believe ranom samples in the original space may allow completion in the lifte space, an our empirical results support this rationale. 3 Case Stuy: Union of a ne subspaces A union of a ne subspaces can also be moele as an algebraic variety. For example, with (x, y, z) 2 R 3, the union of the plane z = an the line x = y is the zero-set of the quaratic polynomial q(x, y, z) = (z )(x y). In general, if A, A 2 R n are a ne spaces of imension r an r 2,respectively,thenwe can write A = {x 2 R n : f i (x) = for i =,...,n r } an A 2 = {x : g i (x) = for i =,...,n r 2 } where the f i an g i are linear, an their union A[B can be expresse as the common zero set of all possible proucts of the f i an g i : A [A 2 = {x 2 R n : f i (x)g j (x) = for all apple i apple n r, apple j apple n r 2 }. (3) i.e., A [A 2 is the common zero set of a system of (n r )(n r 2 ) quaratic equations. This argument can be extene to show a union of k a ne subspaces of imensions r,...,r k is a variety escribe by a system of Q k i= (n r i) polynomial equations of egree k. In this section we establish bouns on the feature space rank for ata belonging to a union of subspaces. We will make use of the following lemma that shows the imension of a vanishing ieal is fixe uner an a ne change of variables: Lemma. Let T : R n! R n be an a ne change of variables, i.e., T (x) =Ax + b, where b 2 R n an A 2 R n n is invertible. Then for any S R n, im I (S) =imi (T (S)). (4) We omit the proof for brevity, but the result is elementary an relies on the fact the egree of a polynomial is unchange uner an a ne change of variables. Our next result establishes a boun on the feature space rank for a single a ne subspace: Proposition. Let the columns of a matrix X n s belong to an a ne subspace A R n of imension r. Then r + rank (X) apple, for all. (5) Proof. By Lemma, im I (A) is preserve uner an a ne transformation of A. Note that we can always fin an a ne change of variables y = Ax + c with invertible A 2 R n n an c 2 R n such that in the coorinates y =(y,...,y n ) the variety A becomes A = {(y,...,y r,,...,) : y,...,y r 2 R}. (6) 7

8 For any polynomial f(y) = P apple c y, the only monomial terms in f(y) that o not vanish on A are those having the form y y r r. Furthermore, any polynomial in just these monomials that vanishes on all of A must be the zero polynomial, since the y,...,y r are free variables. Therefore, S (A) = span{y y r r : + + r apple } (7) i.e., the non-vanishing ieal coincies with the space of polynomials in r variables of egree at most, which is, proving the claim. r+ We note that for s su ciently large, the boun in (5) becomes an equality, provie the ata points are istribute generically within the a ne subspace, meaning they are not the solution to aitional non-trivial polynomial equations of egree at most. We now erive bouns on the imension of the non-vanishing/vanishing ieals for a union of a ne varieties. Below we give a more general argument for a union of arbitrary varieties, then specialize to a ne spaces. Let A, B R n be any two varieties. It follows irectly from efinitions that Applying orthogonal complements to both sies above gives Therefore, we have the boun I (A[B)=I (A) \I (B). (8) S (A[B)=(I (A[B))? = S (A)+S (B). (9) im S (A[B) apple im S (A)+imS (B). (2) In the case of an arbitrary union of varieties [ k i= A i, repeate application of (2) gives k! [ im S A i apple i= kx im S (A i ). (2) i= Specializing to the case where each A i is a in Prop. gives the following result: ne subspace of imension at most r an applying the result Proposition 2. Let the columns of a matrix X n s belong to a union of k a ne subspaces A,..., A k each having imension at most r. Then we have the boun: r + rank (X) apple k, for all. (22) We remark that in some cases the boun (22) is (nearly) tight. For example, if the ata lies on the union of two r-imensional a ne subspaces A an B that are mutually orthogonal, one can show rank (X) = 2 r+. The rank is one less than the boun in (22) because S (A) \S (B) has imension one, coinciing with the space of constant polynomials. Determining the exact rank for ata belonging to an arbitrary finite union of subspaces appears to be a intricate problem; see [22] which stuies the relate problem of etermining the Hilbert series of a union of subspaces. Empirically, we observe that the boun in (22) is orer-optimal with respect to k, r, an. This is a consequence of the Hilbert basis theorem [7], which shows that every vanishing ieal has a finite generating set. For relate iscussion see Appenix C of [2]. 8

9 The feature space rank to imension ratio R/N in this case is given by r+ R N k n+ r k (23) n Recall that the minimum sampling rate is approximately (R/N) for s R. Hence we woul nee m k r. (24) This rate is favorable to low-rank matrix completion approaches, which nee m kr measurements per column for a union of k subspaces having imension r. While this boun suggests it is always better to take the egree as large as possible, this is only true for su ciently large s. To take avantage of the improve sampling rate implie by (24), accoring to (2) we nee the number of ata vectors per subspace to be O(r ). In other wors, our moel is able to accommoate more subspaces with larger but at the expense of requiring exponentially more ata points per subspace. We note that if the number of ata points is not an issue, we coul take = log k an require only m O(r) observe entries per column. In this case, for moerately size k (e.g., k apple 2) we shoul choose we have = 2 or 3. In fact, we fin that for these values of we get excellent empirical results, as we show in Section 5. 4 Algorithms There are several existing matrix completion algorithms that coul potentially be aapte to solve a relaxation of the rank minimization problem (), such as singular value thresholing [25], or alternating minimization [26]. However, these approaches o not easily len themselves to kernelize implementations, i.e., ones that o not require forming the high-imensional lifte matrix (X) explicitly,but instea make use of the e ciently computable kernel function for polynomial feature maps 2 k (x, y) :=h (x), (y)i =(x T y + ). (25) For matrices X =[x,...,x s ], Y =[y,...,y s ] 2 R n s,weusek (X, Y ) to enote the matrix whose (i, j)-th entry is k (x i, y j ), equivalently, k (X, Y )=(X T Y + ). where 2 R s s is the matrix of all ones, an ( ) enotes the entrywise -th power of a matrix. A kernelize implementation of the matrix completion algorithm is critical for large, since the rows of the lifte matrix N scales exponentially with. One class of algorithm that kernelizes very naturally is the iterative reweighte least squares (IRLS) approach of [27, 28] for low-rank matrix completion. The algorithm also has the avantage of being able to accommoate the non-convex Schatten-p relaxation of the rank penalty, in aition to the convex nuclear norm relaxation. Specifically, we use an IRLS approach to solve where ky k Sp min X k (X)k p S p s.t. P (X) =P (X ), (VMC) is the Schatten-p quasi-norm efine as ky k Sp := X i i(y ) p! p, <papple (26) 2 Strictly speaking, k is not kernel associate with the polynomial feature map as efine in (3). Instea, it is the kernel of the relate map (x) :={ p c x : apple} where c are appropriately chosen multinomial coe cients. 9

10 with i (Y ) enoting the i th singular value of Y. Note that for p = we recover the nuclear norm. We call this optimization formulation variety-base matrix completion (VMC). The basic iea behin the IRLS approach can be illustrate in the case of the nuclear norm. First, we can re-express the nuclear norm as a weighte Frobenius norm: ky k = tr[(y T Y ) 2 ]=tr[(y T Y )(Y T Y ) 2 ] {z } W an then attempt to minimize the nuclear norm of a matrix Y belonging to a constraint set C by performing the iterations W n =(Y T n Y n ) 2 Y n+ = arg min Y 2C tr[(y T Y )W n ]. Note the Y -upate can be recast as a weighte least-squares problem subject to the iteratively upate weight matrix W, lening the algorithm its name. To ensure the matrix efining W is invertible, an to improve numerical stability, we can also introuce a smoothing parameter n > tothew-upate as W n =(Y T Y + n ) 2, satisfying n! as n!. Making the substitution Y = (X), an replicating the steps above gives the following IRLS approach for solving (VMC) with p = : W n =(k(x n, X n )+ n I) 2 X n+ = arg min X tr[k(x, X)W n ] s.t. P (X) =P (X ) Rather than fining the exact minimum in the X upate, which coul be costly, following the approach in [28], we instea take a single projecte graient escent step to upate X. A straightforwar calculation shows that the graient of the objective F (X) = tr[k(x, X)W n ] is given by rf (X) = X(W k (X, X)), where enotes an entry-wise prouct. Hence a projecte graient step is given by X n = X n X n (W k (X n, X n )) X n = P (X )+P c( X) where is a step-size parameter. The above erivation can easily be extene to the Schatten-p minimization problem (VMC) by simply p changing the W -upate to be the negative qth power of the kernel matrix, where q = 2. See Algorithm for pseuo-coe of the propose IRLS algorithm for solving (VMC). Heuristics are given in [28] for setting the optimization parameters an base on q, which we aopt as well. Specifically, we set n = / n,where an are user-efine parameters, an upate n = n q where q = p/2. The appropriate choice of an will epen on the scaling an spectral properties of the ata. Empirically, we fin that setting =(.) max, where max is the largest eigenvalue of the kernel matrix obtaine from the initialization, an =. work well in a variety of settings. For all our experiments in Section 5wefixp =/2, which was foun to give the best matrix recovery results for synthetic ata. 5 Numerical Experiments 5. Empirical valiation of sampling bouns In Figure 2 we report the results of two experiments to valiate the preicte minimum sampling rate in () on synthetic variety ata. In the first experiment we generate n s ata matrices whose columns

11 Algorithm Kernelize IRLS to solve (VMC). Require: Samples P (X ), initialization of X, polynomial kernel egree, value p of Schatten-p penalty, <papple, IRLS parameters,. p Set q = 2. while not converge o Step : Inverse power of kernel matrix K k (X, X) (V, S) =eig(k). W V (S + I) q V T Step 2: Projecte graient escent step q A W k (X, X) X X(I A) X P (X )+P c(x) / en while belong to a union of k subspaces each of imension r. We unersample each column of the matrix taking m entries uniformly at ranom, an attempte to recover the missing entries using our propose IRLS algorithm for VMC (Algorithm with p =/2) for egrees = 2 an = 3. As a baseline, we also compare with low-rank matrix completion (LRMC) via nuclear norm minimization in the original matrix omain. We also compare with matrix completion via non-convex Schatten-/2 minimization (LRMC-NCVX) in the original matrix omain, which is implemente using Algorithm, with a linear kernel, i.e., we set = in (25). We sai a column of the matrix was successfully complete when the relative recovery error kx x k/kx k was less than 5,wherexistherecovere column an x is the original column. We use the settings n = 5, s = k, r = 3, for varying measurements m an number of subspaces k, an measure the fraction of successful completions over ranom trials for each pair (m, k). In the secon experiment we attempte to recover synthetic variety ata with pre-etermine feature space rank. To construct this ata, we sample s = 3 ata points from a union of ranomly generate parametric curves surfaces of imension, 2 or 3 belonging to R n,n= 2. We generate several examples an sorte each ataset by its empirically etermine feature space rank R. We follow the same experimental proceure as before. Because the ata was high-rank, recovery via LRMC an LRMC-NCVX faile in all instances, an we omit these results from Figure 2(b). Consistent with our theory, we fin that VMC is successful at recovering most of the ata columns above the preicte minimum sampling rate an typically fails below it. While VMC often fails to recover % of the columns near the preicte rate, in fact a large proportion of the columns (%99 %9) are complete successfully right up to the preicte rate. Sometimes the recovery ips below the preicte rate (e.g., VMC, = 2 in Fig. 2(a) an VMC, = 3 in Fig. 2(b)). However, since the preicte rate relies on what is likely an over-estimate of the true egrees of freeom, it is not surprising that the VMC algorithm occasionally succees below this rate, too.

12 (a) Union of Subspaces (b) Parametric Data Figure 2: Phase transitions for matrix completion of synthetic variety ata. In (a) we simulate ata belonging to a union of k for varying k (ambient imension n = 5, subspace imension r = 3, an ata points sample from each subspace giving s = k). In (b) we simulate ata belonging union of few parametric curves an surfaces having known feature space rank R (ambient imension n = 2 an s = 3 ata points). In all cases we unersample each column of the ata matrix at a rate m/n, an perform matrix completion using the following algorithms: convex an non-convex low-rank matrix completion (LRMC,LRMC-NCVX) an our propose variety-base matrix completion (VMC) approach for egree =2, 3. The grayscale values inicate the fraction of ranom trials where the columns of the ata matrix were successfully recovere up to the specifie percentage. In all figures the re ashe line inicates the preicte minimal sampling rate = m /n etermine by (). 5.2 Motion segmentation of real ata We also apply the VMC approach to the motion segmentation problem in computer vision [29] using real ata belonging to the Hopkins 55 ataset [3]. This ata consists of several feature points tracke across F frames of the vieo, where the trajectories associate with one moving object can be moele as lying in an low-imensional a ne subspace of R 2F. We reprouce the experimental setting in [], an simulate the high-rank setting by unersampling frames of the ata. We also simulate missing trajectories by sampling uniformly at ranom from the feature points across all frames. To obtain a clustering from missing ata we first complete the missing entries using VMC an then ran the sparse subspace clustering (SSC) algorithm [3] on the result; we call the combination of these methos VMC+SSC. A similar approach of stanar LRMC followe by SSC (LRMC+SSC) has been shown to provie a consistent baseline for subspace clustering with missing ata [, ], an we use this a basis for comparison. We use publicly available coe to implement SSC 3. The SSC algorithm requires a regularization parameter. We use the ata aaptive choice = / min j max i6=j X T X i,j propose in [4], with = for all experiments. We also compare against SSC with entry-wise zerofill (SSC-EWZF), which was the most competitive metho among those propose in [] for the Hopkins 55 ataset with missing ata. In Figure 3 we show the results of our subspace clustering experiment on the Hopkins 55 ataset. We report the clustering error of each algorithm, i.e., the proportion of ata points clustere incorrectly, 3 2

13 clustering error all frames SSC-EWZF LRMC+SSC VMC+SSC,=2 VMC+SSC,=3 clustering error frames clustering error frames missing rate missing rate missing rate Figure 3: Subspace clustering error on Hopkins 55 ataset for varying rates of missing ata an unersampling of frames. The ata is high-rank when fewer frames are sample, in which case the propose VMC+SSC approaches gives substantially lower clustering error over other state-of-the-art algorithms for subspace clustering with missing ata. in the case where we retain all frames, 6 frames, an 3 frames of the ataset, an over a range of missing ata rates. We fin the VMC+SSC approach gives similar or lower clustering error than LRMC+SCC for low missing rates. Likewise, VMC+SSC also substantially outperforms SSC-EWZF for high missing rates. Note that unlike SSC-EWZF an the other algorithms introuce in [], VMC+SSC also succees in setting where the ata is low-rank (i.e., when all frames are retaine). This is because the performance of VMC is similar to stanar LRMC in the low-rank setting. 5.3 Completion of motion capture ata Finally, we also consier the problem of completing time-series trajectories from motion capture sensors. We experiment on a ataset rawn from the CMU Mocap atabase 4. Empirically, this ataset has been shown to be locally low-rank over the time frames corresponing to each separate activity, an can be moele as a union of subspaces []. We chose a ataset (subject 56, trial 6) consisting of nine istinct motions in succession: punch, grab, skip, etc.. The ata ha measurements from n = 62 sensors at s = 6784 time instants. We ranomly unersample the columns of this matrix an attempt to complete the ata using VMC, LRMC, an LRMC-NCVX an measure the resulting completion error: kx X k F /kx k F,whereX is the recovere matrix an X is the original matrix. We show the results of this experiment in Figure 4. Similar to results on synthetic ata, we fin the VMC approach outperforms LRMC an LRMC-NCVX for appropriately chosen egree. In particular, VMC with = 2, 3 perform similar for small missing rates, but VMC = 2 gives lower completion error over = 3 for large missing rates, consistent with the results in Figure 2. 6 Discussion An interesting feature of the IRLS algorithm presente in this work is that it can easily accommoate other smooth kernels, incluing the popular Gaussian raial basis function (RBF) [23]. A similar optimization formulation to ours was presente in the recent pre-print [32] using Gaussian RBF kernels in place of polynomial kernels, showing goo empirical results in a matrix completion context. However, the analysis of the sampling complexity in this case is complicate by the fact that a feature space representation 4 3

14 completion error LRMC LRMC-NCVX VMC, =2 VMC, = missing rate Figure 4: Completion error on CMU Mocap ataset using the propose VMC approach compare with convex an non-convex LRMC algorithms. for Gaussian RBF kernel is necessarily infinite-imensional. Unerstaning the sampling requirements in this case woul be an interesting avenue for future work. 7 Conclusion We introuce a matrix completion approach base on moeling ata as an algebraic variety that generalizes low-rank matrix completion to a much wier class of ata moels, incluing ata belonging to a union of subspaces. We present a hypothesize sampling complexity boun for the completion of a matrix whose columns belong to an algebraic variety. An surprising result of our analysis that that a union of k a ne subspaces of imension r shoul be recoverable from O(rk / ) measurements per column, provie we have O(r ) ata points (columns) per subspace, where is the egree of the feature space map. In particular, if we choose = log k, then we nee only O(r) measurements per column as long as we have O(r log k ) columns per subspace. We aitionally introuce an e cient algorithm base on an iterative reweighte least squares approach that realizes these hypothesize bouns on synthetic ata, an reaches state-of-the-art performance on for matrix completion on several real high-rank atasets. References [] E. Canes an B. Recht, Exact matrix completion via convex optimization, Communications of the ACM, vol. 55, no. 6, pp. 9, 22. [2] J. Lee, S. Kim, G. Lebanon, an Y. Singer, Local low-rank matrix approximation., ICML (2), vol. 28, pp. 82 9, 23. [3] B. Eriksson, L. Balzano, an R. D. Nowak, High-rank matrix completion., in AISTATS, pp , 22. [4] E. Elhamifar an R. Vial, Sparse subspace clustering: Algorithm, theory, an applications, IEEE transactions on pattern analysis an machine intelligence, vol. 35, no., pp , 23. [5] R. S. Ganti, L. Balzano, an R. Willett, Matrix completion uner monotonic single inex moels, in Avances in Neural Information Processing Systems, pp , 25. 4

15 [6] D. Song, C. E. Lee, Y. Li, an D. Shah, Blin regression: Nonparametric regression for latent variable moels via collaborative filtering, in Avances in Neural Information Processing Systems, pp , 26. [7] D. A. Cox, J. Little, an D. O Shea, Ieals, Varieties, an Algorithms. Springer International Publishing, 25. [8] B. Recht, A simpler approach to matrix completion, Journal of Machine Learning Research, vol. 2, no. Dec, pp , 2. [9] D. L. Pimentel-Alarcón, N. Boston, an R. D. Nowak, A characterization of eterministic sampling patterns for low-rank matrix completion, IEEE Journal of Selecte Topics in Signal Processing, vol., no. 4, pp , 26. [] C. Yang, D. Robinson, an R. Vial, Sparse subspace clustering with missing entries, in Proceeings of The 32n International Conference on Machine Learning, pp , 25. [] E. Elhamifar, High-rank matrix completion an clustering uner self-expressive moels, in Avances in Neural Information Processing Systems, pp. 73 8, 26. [2] D. Pimentel-Alarcón, L. Balzano, R. Marcia, R. Nowak, an R. Willett, Group-sparse subspace clustering with missing ata, in Statistical Signal Processing Workshop (SSP), 26 IEEE, pp. 5, IEEE, 26. [3] D. Pimentel-Alarcon an R. Nowak, The information-theoretic requirements of subspace clustering with missing ata, in Proceeings of The 33r International Conference on Machine Learning, pp. 82 8, 26. [4] N. Rao, R. Ganti, L. Balzano, R. Willett, an R. Nowak, On learning high imensional structure single inex moels, in Proceeings of the 3st AAAI conference on artificial intelligence, 27. [5] G. Sanguinetti an N. D. Lawrence, Missing ata in kernel PCA, in European Conference on Machine Learning, pp , Springer, 26. [6] M. H. Nguyen an F. Torre, Robust kernel principal component analysis, in Avances in Neural Information Processing Systems, pp , 29. [7] G. Chen, S. Atev, an G. Lerman, Kernel spectral curvature clustering (kscc), in Computer Vision Workshops (ICCV Workshops), 29 IEEE 2th International Conference on, pp , IEEE, 29. [8] R. Vial, S. Soatto, Y. Ma, an S. Sastry, An algebraic geometric approach to the ientification of a class of linear hybri systems, in Decision an Control, 23. Proceeings. 42n IEEE Conference on, vol., pp , IEEE, 23. [9] R. Vial, Y. Ma, an S. Sastry, Generalize principal component analysis (GPCA), IEEE Transactions on Pattern Analysis an Machine Intelligence, vol. 27, no. 2, pp , 25. [2] R. Vial, Y. Ma, an S. Sastry, Generalize Principal Component Analysis. Springer New York, 26. [2] M. C. Tsakiris an R. Vial, Algebraic clustering of a ne subspaces, arxiv preprint arxiv: , 25. 5

16 [22] H. Derksen, Hilbert series of subspace arrangements, Journal of pure an applie algebra, vol. 29, no., pp. 9 98, 27. [23] K.-R. Muller, S. Mika, G. Ratsch, K. Tsua, an B. Scholkopf, An introuction to kernel-base learning algorithms, IEEE Transactions on Neural Networks, vol. 2, no. 2, pp. 8 2, 2. [24] Y. Chen, S. Bhojanapalli, S. Sanghavi, an R. War, Coherent matrix completion, in Proceeings of The 3st International Conference on Machine Learning, pp , 24. [25] J.-F. Cai, E. J. Canès, an Z. Shen, A singular value thresholing algorithm for matrix completion, SIAM Journal on Optimization, vol. 2, no. 4, pp , 2. [26] P. Jain, P. Netrapalli, an S. Sanghavi, Low-rank matrix completion using alternating minimization, in Proceeings of the forty-fifth annual ACM symposium on Theory of computing, pp , ACM, 23. [27] M. Fornasier, H. Rauhut, an R. War, Low-rank matrix recovery via iteratively reweighte least squares minimization, SIAM Journal on Optimization, vol. 2, pp , oct 2. [28] K. Mohan an M. Fazel, Iterative reweighte algorithms for matrix rank minimization, The Journal of Machine Learning Research, vol. 3, no., pp , 22. [29] K.-i. Kanatani, Motion segmentation by subspace separation an moel selection, in Computer Vision, 2. ICCV 2. Proceeings. Eighth IEEE International Conference on, vol. 2, pp , IEEE, 2. [3] R. Tron an R. Vial, A benchmark for the comparison of 3- motion segmentation algorithms, in Computer Vision an Pattern Recognition, 27. CVPR 7. IEEE Conference on, pp. 8, IEEE, 27. [3] E. Elhamifar an R. Vial, Sparse subspace clustering, in Computer Vision an Pattern Recognition, 29. CVPR 29. IEEE Conference on, pp , IEEE, 29. [32] R. Garg, A. Eriksson, an I. Rei, Non-linear imensionality regularizer for solving inverse problems, arxiv preprint arxiv:63.55, 26. 6