On Some Extensions of Bernstein s Inequality for Self-Adjoint Operators

Transcription

1 On Some Extensions of Bernstein s Inequality for Self-Adjoint Operators Stanislav Minsker sminsker@math.duke.edu Abstract: We present some extensions of Bernstein s inequality for random self-adjoint operators. The main feature of these results is that they can be applied in the infinitedimensional setting. In particular, our inequalities refine the previous results of D. Hsu, S. M. Kakade and T. Zhang [4]. 1. Introduction Theoretical analysis of many problems, such as low-rank matrix recovery, covariance estimation and approximate matrix multiplication, is built upon exponential bounds for P X i > t where {X i } is a finite sequence of self-adjoint random matrices and is the operator norm. Starting with the pioneering work of R. Ahlswede and A. Winter [1], the moment-generating function technique was used to produce generalizations of Chernoff, Bernstein and Friedman inequalities to the non-commutative case; see [14],[13],[8] and references therein for thorough treatment and applications. Besides the definition of a correct variance parameter, the main difference in the tail bounds between the scalar version and the d d matrix version of the inequalities is the multiplicative factor d. While being sufficient for most problems, explicit dependence on the dimension of the matrix does not allow straightforward application of these results in the high-dimensional or infinite-dimensional setting. The main purpose of this note is to provide a dimension-free version of Bernstein s inequality for a sequence of independent random matrices as well as for the case of martingale differences. Some interesting results in this direction were previously obtained in [4], but with a slightly suboptimal tail. We show that in many cases the tail behavior can be improved through the modification of original Tropp s method [14] based on Lieb s concavity theorem. The trace quantity appearing in our bounds never exceeds the dimension of a matrix, therefore presented results can be seen as a generalization of previously existing bounds. We proceed by stating the main theorems followed by applications to vector-valued Bernstein inequalities and estimation of integral operators. i. Bernstein s inequality for independent random matrices.1. Definitions and notations Everywhere below, stands for the operator norm A := max λi A A, where λ i A A are the eigenvalues of a self-adjoint operator A A, and Y is the usual Euclidean norm or a vector Department of Mathematics, Duke University i 1

2 S. Minsker/Extensions of Bernstein s inequality Y C d. Let ξ be a real-valued random variable and ψ be a nondecreasing convex function such that ψ0 = 0. The Orlicz norm of ξ is defined as { } ξ ξ ψ = inf C > 0 : Eψ 1. C We will be mostly interested in the case ψ 1 x := e x 1 which corresponds to sub exponential random variables. Given a random matrix X, EX will stand for its expectation taken elementwise and E i [ ] denotes the conditional expectation E[ X 1,..., X i ]. For two self-adjoint matrices A and B, we will write A B iff A B is nonnegative definite, A B 0. We will often use the following simple fact: let A be a self-adjoint matrix with eigenvalue decomposition A = T ΛT, let f 1, f : R R be such that f 1 λ f λ for any eigenvalue λ = λa. Then f 1 A f A, where f i A = T f i ΛT and f i Λ is formed by applying f i to every element on the diagonal. Finally, we will define Ψ σ t := t /. This function often appears as an exponent in Bernsteintype concentration σ +t/3 results... Main results We are ready to state our first result, a version of Bernstein s inequality for the sequence of independent self-adjoint random matrices. Theorem.1. Let X 1,..., X n C d d be a sequence of independent self-adjoint random matrices such that EX i = 0 and σ EXi. Assume that X i U a.s., 1 i n. Then, for any t > 0 tr EXi P X i > t σ exp Ψ σ t r σ t.1 where r σ t = t log 1+t/σ. Remark 1. Note that 1 1. tr EX σ i d in fact, if EXi is approximately low rank, i.e. has many small 1 eigenvalues, tr can be much smaller than d. σ EXi. r σ t is decreasing, so in the range of t when the inequality becomes nontrivial, r σ t can be replaced by a constant. For example, when t σ we have r σ t σ log 1+1/σ 1.5 for σ Presented bounds easily extend to the case of rectangular matrices via the application of Paulsen dilation [8], see section.6 of [14] and Corollary 5.1 below for details. Next, we present a version of inequality that allows X i to be unbounded but requires exponential integrability instead. Such modifications of Bernstein s inequality are well known in scalar case; the noncommutative version appears in [5] see Theorem.7.

3 S. Minsker/Extensions of Bernstein s inequality 3 Theorem.. Let X 1,..., X n C d d be a sequence of independent self-adjoint random matrices such that EX i = 0 and σ EXi. Assume that U max Xi ψ1. Then, for any t > 0 4 P X i > t tr 4 tr EXi σ EXi σ where Kσ, d = log 16 d + log nu. σ 1 i n t exp r σ 1+1/d σ t, exp t 4UKσ,d r σ t, t σ 1+1/d U Kσ,d, t > σ 1+1/d U Kσ,d, Remark. The bound above is not dimension-free since Kσ, d increases with d, however it still gives certain improvement over existing versions. In particular, integrating inequality. tr EXi and setting Intdim := d, we get σ E X i C 1 σ log 4 Intdim U Kσ, d log 4 Intdim. This should be compared to an estimate implied by Theorem.7 of [5]: E X i C σ logd U log nu logd. Improvement can be significant whenever Intdim d. Proof of Theorem.1. Proof of.1 follows the lines of [14], where the key role is played by Lieb s concavity theorem [7]: Theorem.3 Lieb. Given a fixed self-adjoint matrix H, the function is concave on a positive definite cone. A tr exp H + log A In [14], section 4.8, advantages of this approach over the classic method of Ahlswede and Winter based on the Golden-Thompson inequality [1], [3] are discussed. Let φθ = e θ θ 1. Note that φ is nonnegative and increasing on 0,. Denote S n := n σ. X i and note that σ = ES n. First, we reduce the bounds on probability to the bounds on moment generating functions through a chain of simple inequalities. Let θ > 0; we have P λ max S n > t = P λ max θs n > θt = P λ max φθs n > φθt The following semidefinite relation is straightforward: Etr φθs n..3 φθt log Ee θx i φθex i..4

4 S. Minsker/Extensions of Bernstein s inequality 4 Indeed, writing the series expansion for e θx i and using that EX i = 0, we obtain [ Ee θx i 1 = I d + E θ Xi! θx i k 1 ] +... k + 1! 1 I d + θ EXi! θk X i k k + 1! +... = = I d + θ EXi e θ Xi θ X i 1 θ X i I d + EX i φθ, where in the last line we used the assumption that X i 1 and the fact that u eu u 1 is u increasing. It remains to apply the inequality I + A e A which holds for self-adjoint A := EXi. Next, since ES n = 0 d, Lieb s concavity theorem and Jensen s inequality for conditional expectation imply Etr φθs n = Etr expθs n 1 + log e θxn I d = = EE n 1 tr expθs n 1 + log e θxn I d E tr expθs n 1 + log Ee θxn I d. Iterating this argument, we get Etr φθs n tr exp log Ee θx i I d, which together with.4 gives Note that, since ES n is nonnegative definite, we can write Etr φθs n tr expφθes n I d..5 exp φθesn I d = = φθ ESn 1/ 1 + 1! φθes n φθes n 1 ES n! n / n φθesn 1 + 1! φθ ES n φθ ES n! n n = = ES nφθ expφθσ 1 σ φθ ES n σ expφθσ..6 Combining.6 with.3, we get P λ max S n > t tr ES n σ exp φθσ exp θt expθt φθt.

5 S. Minsker/Extensions of Bernstein s inequality 5 Note that for y > 0, e y φy = y e y y y y / + y 3 / y..7 Choose θ := log 1 + t σ to minimize expφθσ θt. Together with the well-known estimate 1 + y log1 + y y y / 1 + y/3, y 0.8 and.7, this concludes the proof of one-sided inequality. It remains to repeat the argument with X i s replaced by X i s to obtain a bound for the operator norm. Proof of Theorem.. We will now outline the main changes needed to obtain the second bound.. We use a truncation argument similar to the aforementioned result in [5]. To this end, we need to bound log Ee θx i in a different way. For any γ > 0, [ ] Ee θx i = I d + θ e θ Xi θ X i 1 E I d + θ EX i X i e θγ θγ 1 θ γ θ X i + θ I d E [ X i ] e θ Xi θ X i 1 θ X i I { X i > γ}. For θ 1 U, by monotonicity of u eu u 1 and Hölder s inequality, u ] E [ X i e θ Xi θ X i 1 θ X i I { X i > γ} U Xi Eφ I{ X i > γ} U U P 1/ X i γ E 1/ e X U. By definition of U, E 1/ e X U, while P X i γ exp γ U by the properties of Orlicz norms see section. of [15]. This gives Ee θx i I d + θ EX i e θγ θγ 1 θ γ + 8 θ U exp γ I d..9 U Now choose γ := U log 16 d nu and assume that θγ σ 1. For such θ, we have Ee θx i I d + EX θ i + σ nd I d, log Ee θx i EX θ i + σ nd I d..10 Repeating the steps of the previous proof with.10 instead of.4, we get θ σ Etr φθs n tr exp d I d + ESn I d.

6 = θ Q exp S. Minsker/Extensions of Bernstein s inequality 6 Following.6 and denoting Q := ESn + σ d I d, we further obtain θ exp Q I d = = θ Q1/ θ! Q θ n 1 n! Q +... Q 1/ θ Q θ! Q θ n 1 n! Q +... = θ Q 1 θ θ Q Q Q exp Q..11 Note that Q = σ 1 + d 1 and trq = σ + tr ESn tr ES n. Combine.11 with.3 and.7 to get P λ max S n > t trq θ expθt Q exp Q θt tr ES n θ φθt σ exp Q θt r σ t. Choose θ = Whenever t If t > t σ 1+1/d σ 1+ d 1 U log 16 d nu σ σ 1+ d 1 U log 16 d nu θ to minimize θ Q θt; recall that by our assumptions Uθ log 16 d nu 1..1 σ,.1 holds with θ = θ and gives subgaussian-type tail behavior., set θ := 1 σ U log 16 d nu σ to get subexponential part of the bound. 3. Concentration inequality for the sums of martingale differences Our next goal is to obtain a version of Friedman s inequality for the sums of matrix-valued martingale differences. Although we get a slightly weaker bound compared to the previous inequality, it still improves the multiplicative dimension factor. For t R, define pt := min t, 1. Note that 1. pt is concave;. gt := e t 1 + pt is non-negative for all t and increasing for t > 0. Recall the following useful result: Proposition 3.1 Peierls inequality, []. Let f : R R be a convex function and {u 1,..., u n } - any orthonormal basis of C n. For any self-adjoint A C n n f u i, Au i tr fa

7 S. Minsker/Extensions of Bernstein s inequality 7 An immediate corollary of this fact is that A tr fa is convex for a convex real-valued f and self-adjoint A: to show that A + B tr f 1 tr fa + tr fb, it is enough to apply Peierls inequality to the orthonormal system given by the eigenvectors of A + B. In particular, since pt is concave, it follows from Jensen s inequality that for any random selfadjoint matrix Y such that EY is well-defined We are ready to state and prove the main result of this section: E tr py tr pey. 3.1 Theorem 3.1. Let X 1,..., X n be a sequence of martingale differences with values in the set of d d self-adjoint matrices and such that X i 1 a.s. Denote W n := n E i 1 Xi. Then, for any t > 0 [ P X i > t, λ max W n σ tr p t ] σ EW n exp Ψ σ t v σ t, where v σ t = Ψ σ t. Remark 3. Note that 1. tr p t σ EW n d for all t > 0;. v σ t is decreasing, so whenever Ψ σ t 1, v σ t can be replaced by a constant. For example, for t σ, v σ t σ 44 whenever σ 1. Proof. Recall that φθ = e θ θ 1. Denote S n := n X i. Let θ be such that θt φθσ > 0 and define an event E by Note that triangle inequality implies We proceed by bounding P E: E := { λ max θs n φθw n θt φθσ }. E { λ max S n t, λ max W n σ }. P E = P λ max gθs n φθw n gθt φθσ tr E gθs n φθw n exp φθσ θt expθt φθσ gθt φθσ 3. The second term in the product, expφθσ θt, is minimized for θ := log1 + t σ and expφθ σ θ t exp Ψ σ t 3.3

8 S. Minsker/Extensions of Bernstein s inequality 8 by.8. To bound the first term in the product 3., note that by Lieb s theorem Y k := tr expθs k φθw k is a supermartingale with initial value d which can be shown similar by repeating conditioning argument of Theorem.1, or see [13] for details, so that Together with 3.1, this gives E tr exp θs n φθw n d. tr E gθs n φθw n = tr E expθs n φθw n I d + pθs n φθw n E tr p θs n φθw n tr p θes n φθw n = tr p φθew n. 3.4 Since EW n is nonnegative definite and due to the obvious estimate φθ e θ 1, θ 0 applied for θ = θ, bound 3.4 becomes tr EgθS n φθ W n tr p t σ EW n. 3.5 Finally, by.8 θ t φθ σ = 1 σ 1 + t/σ log1 + t/σ t/σ t / σ + t/3 > 0. Since φy = gy for y 0, the third term in the product 3. can be estimated by.7: expθ t φθ σ gθ t φθ σ Ψ σ t, 3.6 where, as before, Ψ σ t = t /. Combination of bounds 3.3,3.5,3.6 concludes the proof. σ +t/3 Remark 4. Expression tr p t EW σ n which replaces the dimension factor in our bound has a very simple meaning: acting on the cone of nonnegative definite operators, the function A p A just truncates the eigenvalues of A on the unit level. It is easy to see that if the eigenvalues of EW n decay polynomially, i.e., λ i EW n σ i, p > 1, then p tr p t σ EW n mind, ct 1/p. In particular, this gives an improvement over the multiplicative factor t appearing in the bounds of [4].

9 S. Minsker/Extensions of Bernstein s inequality 9 4. Extensions to the case of self-adjoint operators Clearly, both Theorem.1 and Theorem 3.1 apply to the case when {X i } i 1 is a sequence of selfadjoint Hilbert-Schmidt operators X i : H H acting on a separable Hilbert space H,, H, such that kerex i = H here, EX is defined as an operator such that EXz 1, z H = E Xz 1, z H for any z 1, z H. We will formally show how to extend results of Theorem.1; similar argument applies to Theorem 3.1. Let L 1 L... be a nested sequence of finite dimensional subspaces of H such that L j = j H, and let P Lj be an orthogonal projector on L j. For any fixed j, we will apply Theorem.1 to a sequence of finite dimensional operators { P Lj X i P Lj }i 1 mapping L j to itself. Note that Xi P Lj X i P Lj j 0 almost surely, hence P X i > t lim inf j P P Lj X i P Lj > t. 4.1 Note that, since A B implies SAS SBS, taking A = PL j = P Lj, B = I and S = P Lj X gives P Lj XP Lj P Lj X P Lj, thus tr E P Lj X i P Lj tr E P Lj Xi P L j lim inf j lim inf λ max EP Lj X i P Lj j tr EXi = lim sup λ max EP Lj X i P Lj j 4. λ max EP Lj X i P Lj tr EXi λ max EX i where in the last step we used a simple bound X i P Lj X i P Lj = X i X P Lj X i P Lj + X i P Lj X i P Lj P Lj X i P Lj X i X i P Lj X i P Lj 0 almost surely. 4.3 Since X i 1 a.s., 4.3 implies by dominated convergence that EP Lj X i P Lj E EXi E P Lj X i P Lj, EX i Theorem.1 applied to the right-hand side of 4.1, combined with 4. and 4.4, yields the result. 5. Applications Many relevant applications of dimension-free bounds, such as covariance estimation and approximate matrix multiplication, were demonstrated in [4]. Our bounds apply in those examples as well and yield slightly sharper tails. We demonstrate other immediate corollaries of our results.

10 5.1. Vector-valued Bernstein s inequality S. Minsker/Extensions of Bernstein s inequality 10 Corollary 5.1. Let Y 1,..., Y n C d be a sequence of independent random vectors such that EY i = 0, Y i 1 almost surely, and let Q i = EY i Yi be the associated covariance matrices. Denote σ := n E Y i = n trq i. Then for all t > 0 P Y i > t 4 exp Ψ σ t r σ t. Remark 5. Note that the inequality is dimension-free. It should be compared to result in [6] see formula 6.13 obtained by a combination of classical martingale methods and a trick of V. Yurinskii. Proof. The proof is based on a combination of Theorem.1 with the Paulsen dilation trick [9]: 0 Y define a linear mapping L : C d C d+1 d+1 by LY =. Set X Y 0 i := LY i and note that X i are self-adjoint random matrices with Xi = Yi 0 0 Y i Yi. Note that LY = Y, which gives X i = Y i 1 almost surely. We also have EXi = E Y i 0 = E Y i = trq i 0 Q i and, similarly, tr X i s. EXi = n trq i. Result now follows from Theorem.1 applied to It is straightforward to extend this result to separable Hilbert space - valued random variables by applying Corollary 5.1 to finite-dimensional projections and taking the limit, as was done in the previous section for operators. It is also easy to derive vector-valued Bernstein s inequality for the - norm rather than the norm by mapping a d-dimensional vector to a diagonal d d matrix; this version might sometimes be more useful than the scalar Bernstein s inequality coupled with the union bound: Corollary 5.. Let Y 1,..., Y n C d be a sequence of independent random vectors such that for all i, EY i = 0, Y i 1 almost surely, and let Q i = EY i Yi be the associated covariance matrices. Denote σ := max E 1 j d Y i, e j, where {e j, 1 j d} is the canonical basis. Then tr Q i P Y i > t σ exp Ψ σ t r σ t.

11 5.. Estimation of the integral operators S. Minsker/Extensions of Bernstein s inequality 11 Let S, Π be a measurable space, with Π being a probability measure. Let K, be a symmetric continuous positive definite kernel with κ := sup Kx, x < and let H K,, HK be the x S corresponding reproducing kernel Hilbert space. For x S, let K x := K, x. Define the integral operator L K : H k H K by L K fx := Kx, yfydπy = Kx, y K y, f HK dπy S where the second equality follows from the reproducing property. Note that L K is self-adjoint with tr L K = EKX, X. In many problems, Π is unknown and L K is approximated by its empirical version L K,n : L K,n fx := 1 f, K Xi n HK K Xi x, where X 1,..., X n is an iid sample from Π. The natural question to ask is: what is the degree of approximation provided by L K,n, measured in the operator norm? Note that the spectrum of L K,n can be easily evaluated in practice since the sets of non-zero eigenvalues of L K,n and a matrix K = 1 n KX i, X j n are equal, see Proposition 9 in [10]. Theorem.1 gives an answer to this i,j=1 question. To apply the theorem, define operator-valued random variables ξ i :=, K Xi HK K Xi L K and note that ξ i s are iid with mean zero. Setting u i = S K X i K Xi HK, we have, K Xi HK K Xi = K Xi H K, u i HK u i K Xi H K = KX i, X i κ, hence ξ i κ. At the same time, since U i :=, u i HK u i is a projector, it satisfies Ui Eξi, E K Xi HK K Xi = E K Xi H K, K Xi HK K Xi κ E, K Xi HK K Xi = κ L K. = U i and Note that L K EKX, X in particular, it can be much smaller than κ. Applying Theorem.1 and Remark 1 which is possible due to observations made in Section 4, we get κ L Corollary 5.3. Under our assumptions on the kernel, for t K n 1 n P L K,n L K > t 5 tr Eξ1 Eξ1 exp nt. κ L K + t/3 This can be used together with the fact that L K,n L K sup j 1 λ j L K λ j L K,n where the eigenvalues of λ j L K and λ j L K,n are ordered increasingly. In particular, in some cases the bound of Corollary 5.3 improves upon the estimate of Proposition 1 in [11].

12 S. Minsker/Extensions of Bernstein s inequality 1 References [1] R. Ahlswede and A. Winter, Strong converse for identification via quantum channels, IEEE Trans. Inform. Theory 48 00, no. 3, MR d:94069 [] E. Carlen, Trace inequalities and quantum entropy: an introductory course., Lecture notes 009, Available at: [3] S. Golden, Lower bounds for the Helmholtz function, Phys. Rev , B117 B118. MR #7113 [4] D. Hsu, S. M. Kakade, and T. Zhang, Tail inequalities for sums of random matrices that depend on the intrinsic dimension, Electron. Commun. Probab , no. 14, 13. MR [5] V. Koltchinskii, Oracle inequalities in empirical risk minimization and sparse recovery problems, Lecture Notes in Mathematics, vol. 033, Springer, 011, Lectures from the 38th Probability Summer School held in Saint-Flour, 008, École d Été de Probabilités de Saint-Flour. MR i:91165 [6] M. Ledoux and M. Talagrand, Probability in Banach spaces, vol. 3, Springer-Verlag, Berlin, 1991, Isoperimetry and processes. MR c:60001 [7] E. H. Lieb, Convex trace functions and the Wigner-Yanase-Dyson conjecture, Advances in Math , MR #10407 [8] R. I. Oliveira, Concentration of the adjacency matrix and of the laplacian in random graphs with independent edges, Preprint 010, [9] V. Paulsen, Completely bounded maps and operator algebras, Cambridge Studies in Advanced Mathematics, vol. 78, Cambridge University Press, Cambridge, 00. MR c:46118 [10] L. Rosasco, M. Belkin, and E. De Vito, On learning with integral operators, J. Mach. Learn. Res , MR h:687 [11] S. Smale and D.-X. Zhou, Geometry on probability spaces, Constr. Approx , no. 3, MR c:60006 [1] C. J. Thompson, Inequality with applications in statistical mechanics, J. Mathematical Phys , MR #7110 [13] J. A. Tropp, Freedman s inequality for matrix martingales, Electron. Commun. Probab , MR k:6004 [14], User-friendly tail bounds for sums of random matrices, Found. Comput. Math. 1 01, no. 4, MR [15] A. W. van der Vaart and J. A. Wellner, Weak convergence and empirical processes, Springer Series in Statistics, Springer-Verlag, New York, MR g:60035