Information-Theoretic Limits of Matrix Completion

Transcription

1 Information-Theoretic Limits of Matrix Completion Erwin Riegler, David Stotz, and Helmut Bölcskei Dept. IT & EE, ETH Zurich, Switzerland {eriegler, dstotz, Abstract We propose an information-theoretic framework for matrix completion. The theory goes beyond the low-rank structure and applies to general matrices of low description complexity. Specifically, we consider random matrices X R m n of arbitrary distribution continuous, discrete, discretecontinuous mixture, or even singular). With S R m n an ε-support set of X, i.e., P[X S] 1 ε, and dim B S) denoting the lower Minkowski dimension of S, we show that k > dim B S) measurements of the form A i, X, with A i denoting the measurement matrices, suffice to recover X with probability of error at most ε. The result holds for Lebesgue a.a. A i and does not need incoherence between the A i and the unknown matrix X. We furthermore show that k > dim B S) measurements also suffice to recover the unknown matrix X from measurements taken with rank-one A i, again this applies to a.a. rank-one A i. Rank-one measurement matrices are attractive as they require less storage space than general measurement matrices and can be applied faster. Particularizing our results to the recovery of low-rank matrices, we find that k > m+n r)r measurements are sufficient to recover matrices of rank at most r. Finally, we construct a class of rank-r matrices that can be recovered with arbitrarily small probability of error from k < m + n r)r measurements. I. INTRODUCTION Matrix completion refers to the recovery of a low-rank matrix from a small) subset of its entries or a small) number of linear combinations of its entries. This problem arises in a wide range of applications, including quantum state tomography, face recognition, recommender systems, and sensor localization see, e.g., [1] and references therein). The formal problem statement is as follows. Suppose we have k linear measurements of the m n matrix X with rankx) r in the form of y = A 1, X,..., A k, X ) T R k 1) where A i R m n denotes the measurement matrices and, stands for the standard trace inner product between matrices in R m n. The number of measurements k is typically much smaller than the total number of entries, mn, of X. Depending on the A i, the measurements can simply be individual entries of X or general linear combinations thereof. The vast literature on matrix completion, for a highly incomplete list see [1] [8], provides guarantees for the recovery of the unknown low-rank matrix X from the measurements y, under various assumptions on the measurement matrices A i and the low-rank models generating X. For example, in [2] the A i are assumed to be chosen randomly from an orthonormal basis for R n n and it is shown that an unknown n n matrix X of rank at most r can be recovered with high probability if k = Onrν ln 2 n). Here, ν quantifies the incoherence between the unknown matrix X and the orthonormal basis for R n n the A i are drawn from. The setting in [3] assumes random measurement matrices A i with the position of the only nonzero entry chosen uniformly at random. It is shown that almost all a.a.) matrices with respect to the random orthogonal model [3, Def. 2.1]) of rank at most r can be recovered with high probability with respect to the measurement matrices) provided that the number of measurements satisfies k Cn 1.25 r ln n, where C is a numerical constant. In [1] it is shown that for measurement matrices A i containing i.i.d. entries that are, e.g., Gaussian), a matrix X of rank at most r can be recovered with high probability from k Cm + n)r measurements, where C is a constant. The recovery guarantees in [1] [8] all pertain to recovery through nuclear norm minimization. In [9] measurement matrices A i containing i.i.d. entries drawn from an absolutely continuous with respect to Lebesgue measure) distribution are considered. It is shown that rank minimization which is NP-hard, in general) recovers an n n matrix X of rank at most r with probability one if k > 2n r)r. It is furthermore shown in [9] that all matrices X of rank at most n/2 can be recovered, again with probability one, provided that k 4nr 4r 2. The recovery thresholds in [1], [9] do not exhibit a log n term, but assume significant richness in the random measurement matrices A i. Storing and applying such measurement matrices is costly in terms of memory and computation time. To overcome this problem [8] considers rank-one measurement matrices of the form A i = a i b T i, where a i R m and b i R n are independent with i.i.d. Gaussian or sub-gaussian entries, and shows that nuclear norm minimization succeeds under the same recovery threshold as in [1], namely k Cm + n)r. Contributions: Inspired by the work of Wu and Verdú on analog signal compression [10], we formulate an informationtheoretic framework for almost lossless matrix completion. The theory is general in the sense of going beyond the low-rank structure and applying to general matrices of low description complexity. Specifically, we consider random matrices X R m n of arbitrary distribution continuous, discrete, discrete-continuous mixture, or even singular). With S R m n an ε-support set of X, i.e., P[X S] 1 ε, and dim B S) denoting the lower Minkowski dimension see Definition 3) of S, we show that k > dim B S) measurements suffice to recover X with probability of error no more than ε. The result holds for Lebesgue a.a. measurement matrices A i and does not need any incoherence between the A i and the unknown matrix X. What is more, we show that k > dim B S) measurements also suffice for recovery from measurements

2 taken with rank-one A i, again this applies to a.a. rank-one A i. Particularizing our results to low-rank matrices X, we show that X of rank at most r can be recovered from k > m+n r)r measurements taken with either general A i or with rankone A i. Perhaps surprisingly, it turns out that, depending on the specific distribution of the low-rank matrix X, even fewer than m + n r)r measurements can suffice. We construct a class of examples that illuminates this phenomenon. Notation: Roman letters A, B,... designate deterministic matrices and a, b,... stands for deterministic vectors. Boldface letters A, B,... and a, b,... denote random matrices and vectors, respectively. For the distribution of a random matrix A we write µ A and we use µ a to designate the distribution of a random vector a. The superscript T stands for transposition. For A = a 1,..., a n ) R m n we let veca) = a T 1,..., a T n) T. For a rank-r matrix A R m n with ordered singular values σ 1 A)... σ r A), we set A) = r σ ia). For a matrix A, tra) denotes its trace. For matrices A, B of the same dimensions, A, B = tra T B) is the trace inner product between A and B. We write A 2 = A, A for the Euclidean norm of the matrix A. For the Euclidean space R k, 2 ), we denote the open ball of radius s centered at u R k by B k u, s), V k, s) and Ak 1, s) stand for its volume and the area of its closure, respectively. Similarly, for the Euclidean space R m n, 2 ), we denote the open ball of radius s centered at A R m n by B m n A, s). We write M m n r and Nr m n for the set of matrices A R m n with ranka) r and ranka) = r, respectively. II. ALMOST LOSSLESS MATRIX COMPLETION We start by formulating the almost lossless matrix completion framework. Definition 1. For a random matrix X R m n of arbitrary distribution µ X with Lebesgue decompositon µ X = µ c X + µ d X + µs X continuous, discrete, and singular components, respectively), an m n, k) code consists of i) linear measurements A 1,,..., A k, ) T : R m n R k ; ii) a measurable decoder g : R k R m n. For given measurement matrices A i, we say that a decoder g achieves error probability ε if P [ g A 1, X,..., A k X ) T) X ] ε. Definition 2. For ε 0, we call a nonempty bounded set S R m n an ε-support set of the random matrix X R m n if P[X S] 1 ε. Definition 3. Minkowski dimension 1 ) Let S be a nonempty bounded set in R m n. The lower Minkowski dimension of S is defined as dim B S) = lim inf log N S ) 1 This quantity is sometimes also referred to as box-counting dimension, which is the origin for the subscript B in the notation dim B ) used below. and the upper Minkowski dimension is log N S ) dim B S) = lim sup where N S denotes the covering number of S given by N S ) = min {k N S B m n M i, ), M i R m n}. i {1,...,k} If dim B S) = dim B S) =: dim B S), we simply say that dim B S) is the Minkowski dimension of S. III. MAIN RESULTS The following result formalizes the statement on the operationally relevant description complexity being given by the lower Minkowski dimensions of ε-support sets of X. Theorem 1. Let S R m n be an ε-support set of X R m n. Then, for Lebesgue a.a. measurement matrices A i, i = 1,..., k, there exists a decoder achieving error probability ε, provided that k > dim B S). Proof. See Section V. Remark 1. The central conceptual element in the proof of Theorem 1 is the following probabilistic null space property, first reported in [11] in the context of almost lossless analog signal separation. For a.a. measurement matrices A i, i = 1,..., k, the dimension of the kernel of the mapping X A 1, X,..., A k, X ) T is mn k. If the lower Minkowski dimension of a set S is smaller than k, the set S will intersect the kernel of this mapping at most trivially. What is remarkable here is that the notions of Euclidean dimension for the kernel of the mapping) and of lower Minkowski dimension for S) are compatible. We next particularize Theorem 1 for low-rank matrices. To this end, we first establish an upper bound on dim B S) for nonempty and bounded subsets of M m n r. Lemma 1. Let S M m n r be a nonempty bounded set. Then dim B S) m + n r)r. Proof. We can decompose Mr m n according to r M m n r = N m n i. i=0 By [12, Ex. 5.30], N m n i is an embedded submanifold of R m n of dimension m + n i)i, i = 1,..., r. Let I = { i {1,..., r} S N m n i }. Then, for each i I, S N m n i is a nonempty bounded set and, therefore, dim B S N m n i ) is well-defined. By [13, Sec. 3.2, Properties i) and ii)], dim B S N m n i ) m+n i)i, i I. Since the upper Minkowski dimension is finitely stable [13, Sec. 3.2, Property iii)], we get dim B S) = max dim BS N m n i ) i I m + n r)r where in the last step we used the monotonicity of fs) = m+n s)s in the range s [0, m+n)/2] together with r m + n)/2, which in turn follows from r minm, n).

3 We can now put the pieces together to get the desired statement on low-rank matrices. Remark 2. Lemma 1 together with dim B ) dim B ), when used in Theorem 1, implies that for X M m n r and every ε > 0, there exists a decoder that achieves error probability ε for Lebesgue a.a. measurement matrices A i, i = 1,..., k, provided that k > m + n r)r. While the sufficient condition k > m + n r)r in Remark 2 is intuitively appealing as m + n r)r is the dimension of the manifold Nr m n, it is actually the lower Minkowski dimensions of ε-support sets of X M m n r that are of operational significance. Specifically, depending on the distribution of X, a smaller than m + n r)r) number of measurements may suffice for recovery of X with probability of error at most ε. The following example illuminates this phenomenon. Example 1. Let X = X T 1 X 2 M m n r, where X 1 R r m and X 2 R r n are independent. Suppose that X 1 has l 1 columns at positions drawn uniformly at random and containing i.i.d. Gaussian entries with all other columns equal to zero and X 2 has l 2 columns at positions drawn uniformly at random and containing i.i.d. Gaussian entries with all other columns equal to zero. Suppose further that r l 1 < m/2 and r l 2 n/2 1/r. The assumptions l i r, i = 1, 2, guarantee that P[rankX) = r] = 1. Next, we construct an ε-support set T for X with dim B T ) l 1 + l 2 )r, which by Theorem 1 together with l 1 + l 2 )r < m + n)r/2 1 m+n r)r 1 proves that we can recover the rank-r matrix X with probability of error at most ε from strictly less than m + n r)r measurements. Let A r m l Mr r m be the set of r m matrices with no more than l nonzero columns. Choose L N sufficiently large for i) S 1 = A r m l 1 B r m 0, L) to be an ε/2-support set of X 1 and ii) S 2 = A r n l 2 B r n 0, L) to be an ε/2-support set of X 2. By [13, Sec. 3.2, Properties i) and iii)], we have dim B S i ) = l i r 2) which is simply the maximum number of nonzero entries of X i S i, i = 1, 2. Set T = {X T 1 X 2 X i S i, i = 1, 2}. Then, P[X T ] = P[X 1 S 1, X 2 S 2 ] = P[X 1 S 1 ] P[X 2 S 2 ] 1 ε. The triangle inequality implies that for all X i, X i S i, i = 1, 2, we have X T 1 X 2 X T 1 X 2 2 X T 1 X 2 X T 1 X X T 1 X 2 X T 1 X 2 2 L X 1 X X 2 X 2 2 ) 3) where we used S 1 B r m 0, L) and S 2 B r n 0, L). Let N Si ) be the covering number of S i, i = 1, 2. We can cover S i by N Si ) balls of radius with centers X ji, j i = 1,..., N Si ), i = 1, 2. Therefore, 3) implies that T can be covered by N S1 )N S2 ) balls of radius centered at X T Xj2 j 1, j i = 1,..., N Si ), i = 1, 2. This yields N T ) N S1 )N S2 ) and we finally get log N T ) dim B T ) = lim log 1 log NS1 )N S2 ) ) lim log 1 log N S1 ) log N S2 ) = lim log 1 + lim log 1 = l 1 r + l 2 r where we used 2) in the last step. Remark 3. The derivation of the recovery thresholds in [9] is also based on a null space property similar to the one discussed in Remark 1. The relevant dimension in [9] is the dimension m + n r)r of the manifold Nr m n. Example 1 above, however, shows that k < m + n r)r measurements can suffice for recovery of rank-r matrices, thereby corroborating the operational significance of the lower Minkowski dimensions of ε-support sets of X. IV. RANK-ONE MEASUREMENT MATRICES Rank-one measurement matrices, i.e., matrices A i = a i b T i with a i R m and b i R n, i = 1,..., k, are attractive as they require less storage space than general measurement matrices and can also be applied faster. Interestingly, Theorem 1 continues to hold for rank-one measurement matrices although they exhibit much less richness than general measurement matrices. The technical challenges in establishing this result are quite different from those encountered in the case of general measurement matrices. In particular, we will need a stronger concentration of measure inequality cf. Lemma 4). Theorem 2. Let S R m n be an ε-support set of X R m n. Then, for Lebesgue a.a. a i R m and b i R n and corresponding measurement matrices A i = a i b T i, i = 1,..., k, there exists a decoder achieving error probability ε, provided that k > dim B S). Proof. See Section V. Remark 4. Example 1 can be shown to carry over to rank-one measurement matrices A i. Remark 5. Theorem 2, when used in combination with Lemma 1, implies that for X M m n r and every ε > 0, there exists a decoder achieving error probability ε for Lebesgue a.a. a i R m and b i R n, provided that k > m + n r)r. In contrast, the threshold k Cm + n)r in [8] for rank-one measurements requires a i and b i to be independent random vectors containing i.i.d. Gaussian or sub-gaussian entries. In addition, the constant C in [8] remains unspecified. V. PROOFS OF THEOREMS 1 AND 2 For both proofs, we first construct a measurable map g : R k R m n such that P [ [g A 1, X,..., A k, X ) T) X ] P [ Z S X \{0} A 1, Z,..., A k, Z ) T = 0, X S ] + ε 4)

4 with S X = {W X W S} for X S. The proofs are then concluded by showing that P [ Z S X \{0} A1, Z,..., A k, Z ) T = 0, X S ] = 0 5) for Lebesgue a.a. matrices A i R m n, i = 1,..., k, in the case of Theorem 1 and for Lebesgue a.a. vectors a i R m and b i R n with A i = a i b T i, i = 1,..., k, in the case of Theorem 2. Proof of 4): Let S R m n be an ε-support set of X with dim B S) < k. We define a measurable map g as follows: gy) = { Z, if { W S A 1, W,..., A k, W ) T = y } = {Z} E, else where E is an arbitrary, but fixed, matrix in R m n \ S used to declare a decoding error). Then, we have P [ g A 1, X,..., A k, X ) T) X ] P [ g A 1, X,..., A k, X ) T) X, X S ] + P[X / S] P [ g A 1, X,..., A k, X ) T) X, X S ] + ε 6) = P [ g A 1, X,..., A k, X ) T) = E, X S ] + ε 7) = P [ Z S X \{0} A1, Z,..., A k, Z ) T = 0, X S ] + ε where 6) is a consequence of S being an ε-support set and in 7) we used that the decoder declares an error if and only if {W S A 1, W,..., A k, W ) T = y} > 1 for y = A 1, X,..., A k, X ) T with X S. Finishing the proof of Theorem 1: Let s > 0 and suppose that A 1,..., A k, i = 1,..., k, are independent and uniformly distributed on B m n 0, s). Then, we have P [ Z S X \{0} A 1, Z,..., A k, Z ) T = 0, X S ] ) k B m n0,s) dµ A1... dµ Ak = P [ Z S X \{0} A 1, Z,..., A k, Z ) T = 0 ] dµ X S = 0 9) where 8) is a consequence of Fubini s theorem for nonnegative measurable functions and 9) follows from Lemma 2 below. With R m n = s N B m n0, s) and since s is arbitrary, 5) holds for Lebesgue a.a. measurement matrices A i, which concludes the proof of Theorem 1. Finishing the proof of Theorem 2: Let s > 0 and suppose that A = [a 1,..., a k ] R m k and B = [b 1,..., b k ] R n k are independent random matrices with columns a i independent and uniformly distributed on B m 0, s) and columns b i independent and uniformly distributed on B n 0, s). Then, we have P [ Z S X \{0} a T 1 Zb 1,..., a T k Zb k ) T = 0, X S ] ) k B m0,s) B n0,s) dµ a1 dµ b1... dµ ak dµ bk 8) = P [ Z S X \{0} a T 1 Zb 1,..., a T k Zb k ) T = 0 ] dµ X S 10) = 0 11) where 10) is a consequence of Fubini s theorem for nonnegative measurable functions and 11) follows from Lemma 3 below. Again, with R l = s N B l0, s) and since s is arbitrary, 5) holds for Lebesgue a.a. vectors a i R m and b i R n, thereby finishing the proof of Theorem 2. Lemma 2. Let s > 0 and A 1,..., A k, i = 1,..., k, be independent and uniformly distributed on B m n 0, s). Suppose that U R m n is a nonempty bounded set with dim B U) < k. Then, we have P [ X U \{0} A 1, X,..., A k, X ) T = 0 ] = 0. 12) Proof. Follows from rewriting the trace inner products A i, X, i = 1,..., k as inner products between vectors in R mn and subsequent application of [11, Prop. 1]. Lemma 3. Let s > 0 and take A = [a 1,..., a k ] R m k and B = [b 1,..., b k ] R n k to be independent random matrices with columns a i, i = 1,..., k, independent and uniformly distributed on B m 0, s) and columns b i, i = 1,..., k, independent and uniformly distributed on B n 0, s). Suppose that U R m n is a nonempty bounded set with dim B U) < k. Then, we have P := P [ X U \{0} a T 1 Xb 1,..., a T k Xb k ) T = 0 ] = 0. Proof. Let R = max X U rankx) and set { U L,r = X U X) > 1 } L, σ 1X) < L, rankx) = r for L N and r = 1,..., R. By the union bound, we have P R P L,r 13) L N r=1 where P L,r = P [ X U L,r a T 1 Xb 1,..., a T k Xb k ) T = 0 ]. We now prove by contradiction that P L,r = 0 for all L N and all r {1,..., R}. Suppose that there exists an L N and an r {1,..., R} such that P L,r > 0 by definition, P L,r 0). For this pair {L, r}, we would then have lim inf log P L,r = 0. 14) For > 0, let be the covering number of the set U L,r and denote corresponding covering balls centered at M i ) R m n as B m n M i ), ), i = 1,...,. We now fix matrices Since X i ) B m n M i ), ) U L,r, B m n M i ), ) B m n X i ), 2), i = 1,...,. 15) i = 1,...,

5 by the triangle inequality, we get P L,r P [ X B m n M i ), ) a T 1 Xb 1,..., a T k Xb k ) T = 0 ] P [ X B m n X i ), 2) a T 1 Xb 1,..., a T k Xb k ) T = 0 ] P [ X B m n X i ), 2) a T 1 Xb 1,..., a T k Xb k ) T 2 ], > 0. 16) Now, for a i B m 0, s), b i B n 0, s), and X B m n X i ), 2), we have a T 1 X i )b 1,..., a T k X i )b k ) T 2 a T 1 X X i ))b 1,..., a T k X X i ))b k ) T 2 + a T 1 Xb 1,..., a T k Xb k ) T 2 k a j 2 2 X X i) 2 2 b j 2 2 j=1 + a T 1 Xb 1,..., a T k Xb k ) T 2 2s 2 k + a T 1 Xb 1,..., a T k Xb k ) T 2, > 0. 17) Inserting 17) into 16) allows us to further upper-bound P L,r according to where P L,r P [ a T 1 X i )b 1,..., a T k X i )b k ) T s 2 k) ] k gl, r, k, m, n, s, ) k, > 0 18) gl, r, k, m, n, s, ) = 22s) m+n r s 2 k)l V m, s)v n, s) s log max 2 L ),, 1 if r = 1 2 r 1+2s 2 k) ) r 1 s + Ar 1,1)sL) r 1 r 1, if r > 1. Here, we applied the concentration of measure inequality in Lemma 4 below with δ = 1 + 2s 2 k) and used the fact that 1/ X i )) < L, σ 1 X i )) < L, and rankx i )) = r recall that by 15) all matrices X i ) are in the set U L,r ). With the upper bound on P L,r in 18) we now get lim inf lim inf log P L,r log) + k log + k log gl, r, k, m, n, s, ) = lim inf = lim inf lim inf log) k log gl, r, k, m, n, s, ) log 1 k + lim log) log 1 k logn U )) = dim B U) k k 19) < 0 20) where 19) follows from U L,r U and in the last step we used that dim B U) < k, by assumption. Since 20) contradicts 14), P L,r = 0 for all L N and all r {1,..., R}. By 13), this establishes that P = 0. Lemma 4. Let A = [a 1,..., a k ] R m k and B = [b 1,..., b k ] R n k be independent random matrices, with columns a i, i = 1,..., k, independent and uniformly distributed on B m 0, s) and columns b i, i = 1,..., k, independent and uniformly distributed on B n 0, s). Suppose that X R m n with r = rankx) > 0. Then, we have with P [ a T 1 Xb 1,..., a T k Xb k ) T 2 δ ] δ k fx, s, δ) k 22s)m+n r 1 fx, s, δ) = X)V m, s)v n, s) ) log max s 2 σ 1X) δ, 1, if r = 1 2 r ) δ r 1 s + Ar 1,1)sσ 1X)) r 1 r 1, if r > 1. 21) Proof. Omitted due to space limitations. REFERENCES [1] E. J. Candès and Y. Plan, Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements, IEEE Trans. Inf. Theory, vol. 4, no. 57, pp , Apr [2] D. Gross, Recovering low-rank matrices from few coefficients in any basis, IEEE Trans. Inf. Theory, vol. 57, no. 3, pp , Mar [3] E. J. Candès and B. Recht, Exact matrix completion via convex optimization, Found. Comput. Math., vol. 9, no. 6, pp , [4] E. J. Candès and T. Tao, The power of convex relaxation: Near-optimal matrix completion, IEEE Trans. Inf. Theory, vol. 56, no. 5, pp , May [5] B. Recht, M. Fazel, and P. A. Parrilo, Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization, SIAM Review, vol. 52, no. 3, pp , Dec [6] B. Recht, A simpler approach to matrix completion, J. Mach. Learn. Res., vol. 12, pp , [7] T. T. Cai and A. Zhang, Sparse representation of a polytope and recovery of sparse signals and low-rank matrices, IEEE Trans. Inf. Theory, vol. 60, no. 1, pp , Jan [8], ROP: Matrix recovery via rank-one projections, Ann. Stat., vol. 43, no. 1, pp , [9] Y. C. Eldar, D. Needell, and Y. Plan, Uniqueness conditions for lowrank matrix recovery, Appl. Comp. Harm. Anal., vol. 33, no. 2, pp , [10] Y. Wu and S. Verdú, Rényi information dimension: Fundamental limits of almost lossless analog compression, IEEE Trans. Inf. Theory, vol. 56, no. 8, pp , Aug [11] D. Stotz, E. Riegler, and H. Bölcskei, Almost lossless analog signal separation, in Proc. IEEE Int. Symp. Inf. Theory, Istanbul, Turkey, Jul. 2013, pp [12] J. M. Lee, Introduction to Smooth Manifolds, 1st ed. New York, NY: Springer, [13] K. Falconer, Fractal Geometry, 1st ed. New York, NY: Wiley, 1990.