arxiv: v1 [math.pr] 13 Oct 2011

Transcription

1 A tail iequality for quadratic forms of subgaussia radom vectors Daiel Hsu, Sham M. Kakade,, ad Tog Zhag 3 arxiv:0.84v math.pr] 3 Oct 0 Microsoft Research New Eglad Departmet of Statistics, Wharto School, Uiversity of Pesylvaia 3 Departmet of Statistics, Rutgers Uiversity October 4, 0 Abstract We prove a expoetial probability tail iequality for positive semidefiite quadratic forms i a subgaussia radom vector. The boud is aalogous to oe that holds whe the vector has idepedet Gaussia etries. Itroductio Suppose that x = x,...,x ) is a radom vector. Let A R m be a fixed matrix. A atural quatity that arises i may settigs is the quadratic form Ax = x A A)x. Throughout v deotes the Euclidea orm of a vector v, ad M deotes the spectral operator) orm of a matrix M. We are iterested i how close Ax is to its expectatio. Cosiderthespecial casewherex,...,x areidepedetstadardgaussiaradomvariables. The followig propositio provides a upper) tail boud for Ax. Propositio. Let A R m be a matrix, ad let Σ := A A. Let x = x,...,x ) be a isotropic multivariate Gaussia radom vector with mea zero. For all t > 0, Pr Ax > trσ)+ ] trσ )t+ Σ t e t. The proof, give i Appedix A., is straightforward give the rotatioal ivariace of the multivariate Gaussia distributio, together with a tail boudfor liear combiatios of χ radom variables due to Lauret ad Massart 000). We ote that a slightly weaker form of Propositio ca be proved directly usig Gaussia cocetratio Pisier, 989). I this ote, we cosider the case where x = x,...,x ) is a subgaussia radom vector. By this, we mea that there exists a σ 0, such that for all α R, E exp α x )] exp α σ / ). We provide a sharp upper tail boud for this case aalogous to oe that holds i the Gaussia case ideed, the same as Propositio whe σ = ). dahsu@microsoft.com, skakade@wharto.upe.edu, tzhag@stat.rutgers.edu

2 Tail iequalities for sums of radom vectors Oe motivatio for our mai result comes from the followig observatios about sums of radom vectors. Let a,...,a be vectors i a Euclidea space, ad let A = a a ] be the matrix with a i as its ith colum. Cosider the squared orm of the radom sum Ax = a i x i where x := x,...,x ) is a martigale differece sequece with Ex i x,...,x i ] = 0 ad Ex i x,...,x i ] = σ. Uder mild boudedess assumptios o the x i, the probability that the squared orm i ) is much larger tha its expectatio E Ax ] = σ a i = σ tra A) falls off expoetially fast. This ca be show, for istace, usig the followig lemma by takig u i = a i x i the proof is stadard, but we give it for completeess i Appedix A.). Propositio. Let u,...,u be a martigale differece vector sequece i.e., Eu i u,...,u i ] = 0 for all i =,...,) such that E u i ] u,...,u i v ad ui b for all i =,...,, almost surely. For all t > 0, Pr u i > v + ] 8vt+4/3)bt e t. After squarig the quatities i the stated probabilistic evet, Propositio gives the boud Ax σ tra A)+σ O tra A) t+t)+ tra A)max i ) ) a i t+t 3/ )+max a i t i with probability at least e t whe the x i are almost surely bouded by or ay costat). Ufortuately, this boud obtaied from Propositio ca be suboptimal whe the x i are subgaussia. For istace, if the x i are Rademacher radom variables, so Prx i = +] = Prx i = ] = /, the it is kow that ) Ax tra A)+O tra A) )t+ A t ) with probability at least e t. A similar result holds for ay subgaussia distributio o the x i Haso ad Wright, 97). This is a improvemet over the previous boud because the deviatio terms i.e., those ivolvig t) ca be sigificatly smaller, especially for large t. I this work, we give a simple proof of ) with explicit costats that match the aalogous boud whe the x i are idepedet stadard Gaussia radom variables.

3 Positive semidefiite quadratic forms Our mai theorem, give below, is a geeralizatio of ). Theorem. Let A R m be a matrix, ad let Σ := A A. Suppose that x = x,...,x ) is a radom vector such that, for some µ R ad σ 0, )] E exp α x µ) exp α σ / ) 3) for all α R. For all t > 0, Pr Ax > σ trσ)+ ) trσ )t+ Σ t + Aµ Σ ) / ) ] / +4 trσ ) t + 4 Σ trσ ) t e t. Remark. Note that whe µ = 0 ad σ = we have: Pr Ax > trσ)+ ] trσ )t+ Σ t e t which is the same as Propositio. Remark. Our proof actually establishes the followig upper bouds o the momet geeratig fuctio of Ax for 0 η < /σ Σ ): E exp η Ax )] E exp σ A z η +µ A z )] η exp σ trσ)η + σ4 trσ )η + Aµ ) η σ Σ η where z is a vector of m idepedet stadard Gaussia radom variables. Proof of Theorem. Let z be a vector of m idepedet stadard Gaussia radom variables sampled idepedetly of x). For ay α R m, )] E exp z α = exp α / ). Thus, for ay λ R ad ε 0, )] E exp λz Ax E exp λz Ax) ] Ax > ε Pr Ax > ε ] λ ) ε exp Pr Ax > ε ]. 4) Moreover, )] E exp λz Ax = E E exp λz Ax µ)) ] ) ] z exp λz Aµ λ σ )] E exp A z +λµ A z 5) 3

4 Let USV be a sigular value decompositio of A; where U ad V are, respectively, matrices of orthoormal left ad right sigular vectors; ad S = diag ρ,..., ρ m ) is the diagoal matrix of correspodig sigular values. Note that ρ = ρ i = trσ), ρ = ρ i = trσ ), ad ρ = maxρ i = Σ. i By rotatioal ivariace, y := U z is a isotropic multivariate Gaussia radom vector with mea zero. Therefore A z = z US U z = ρ y + +ρ mym ad µ A z = ν y = ν y + +ν m y m, where ν := SV µ ote that ν = SV µ = Aµ ). Let γ := λ σ /. By Lemma, )] γ E exp γ ρ i yi + ν i y i exp ρ γ + ρ γ + ν γ/σ ) 6) σ ρ γ for 0 γ < / ρ ). Combiig 4), 5), ad 6) gives Pr Ax > ε ] exp εγ/σ + ρ γ + ρ γ + ν γ/σ ) ρ γ for 0 γ < / ρ ) ad ε 0. Choosig ε := σ ρ +τ)+ ν + ρ τ ρ ad γ := ) ρ ρ ρ + ρ, τ we have Pr Ax > σ ρ +τ)+ ν + ρ τ ρ ] exp ρ ρ + ρ τ ρ )) = exp ρ ρ h ρ τ ρ + ρ τ ρ )) where h a) := +a +a, which has the iverse fuctio h b) = b+b. The result follows by settig τ := ρ t+ ρ t = trσ )t+ Σ t. The followig lemma is a stadard estimate of the logarithmic momet geeratig fuctio of a quadratic form i stadard Gaussia radom variables, proved much alog the lies of the estimate due to Lauret ad Massart 000). Lemma. Let z be a vector of m idepedet stadard Gaussia radom variables. Fix ay o-egative vector α R m + ad ay vector β Rm. If 0 λ < / α ), the loge exp λ α i zi + )] β i z i α λ+ α λ + β /. α λ 4

5 Proof. Fix λ R such that 0 λ < / α ), ad let η i := / α i λ > 0 for i =,...,m. We have E exp λα i z i +β i z i )] = π exp z i/ ) exp λα i z i +β i z i ) dzi = η i exp β i η i ) πηi exp ηi ) zi β i ηi ) dz i so )] loge exp λ α i zi + β i z i = βiη i + logηi. The right-had side ca be bouded usig the iequalities logηi = log α i λ) = α i λ) j j= j α λ+ α λ α λ ad β iη i β / α λ. Example: fixed-desig regressio with subgaussia oise We give a simple applicatio of Theorem to fixed-desig liear regressio with the ordiary least squares estimator. Let x,...,x be fixed desig vectors i R d. Let the resposes y,...,y be radom variables for which there exists σ > 0 such that )] ) E exp α i y i Ey i ]) exp σ for ay α,...,α R. This coditio is satisfied, for istace, if y i = Ey i ]+ε i α i for idepedet subgaussia zero-mea oise variables ε,...,ε. Let Σ := x ix i we assume is ivertible without loss of geerality. Let ) β := Σ x i Ey i ] /, which be the coefficiet vector of miimum expected squared error. The ordiary least squares estimator is give by ) ˆβ := Σ x i y i. 5

6 The excess loss Rˆβ) of ˆβ is the differece betwee the expected squared error of ˆβ ad that of β: ] Rˆβ) := E x i ˆβ y i ) ] E x i β y i ). It is easy to see that By Theorem, Rˆβ) = Σ / ˆβ β) = Pr Σ / ) x i yi Ey i ]). Rˆβ) > σ d+ dt+t ) ] Note that i the case that Ey i Ey i ]) ] = σ for each i, the ERˆβ)] = σ d ; e t. so the tail iequality above is essetially tight whe the y i are idepedet Gaussia radom variables. Refereces D. L. Haso ad F. T. Wright. A boud o tail probabilities for quadratic forms i idepedet radom variables. The Aals of Mathematical Statistics, 43): , 97. B. Lauret ad P. Massart. Adaptive estimatio of a quadratic fuctioal by model selectio. The Aals of Statistics, 85):30 338, 000. G. Pisier. The volume of covex bodies ad Baach space geometry. Cambridge Uiversity Press, 989. A Stadard tail iequalities A. Martigale tail iequalities The followig is a stadard form of Berstei s iequality stated for martigale differece sequeces. Lemma Berstei s iequality for martigales). Let d,...,d be a martigale differece sequece with respect to radom variables x,...,x i.e., Ed i x,...,x i ] = 0 for all i =,...,) such that d i b ad Ed i x,...,x i ] v. For all t > 0, Pr d i > ] vt+/3)bt e t. The proof of Propositio, which is etirely stadard, is a immediate cosequece of the followig two lemmas together with Jese s iequality. 6

7 Lemma 3. Let u,...,u be radom vectors such that E u i ] u,...,u i v ad ui b. for all i =,...,, almost surely. For all t > 0, ] Pr u i E u i > ] 8vt+4/3)bt e t. Proof. Let s := u + +u. Defie the Doob martigale d i := E s u,...,u i ] E s u,...,u i ] for i =,...,, so d + +d = s E s ]. First, clearly, Ed i u,...,u i ] = 0. Next, the triagle iequality implies d i = E s u i )+u i u,...,u i ] E s u i )+u i u,...,u i ] E s u i + u i u,...,u i ] E s u i u i u,...,u i ] = u i +E u i u,...,u i ], ad similarly, d i u i E u i u,...,u i ]. Therefore, Moreover, so d i u i +E u i u,...,u i ] b almost surely. E d i u,...,u i ] E u i + u i E u i u,...,u i ] +E u i u,...,u i ] u,...,u i ] = E u i u,...,u i ] +3 E ui u,...,u i ] 4 E u i ] u,...,u i, E d i u ],...,u i 4v almost surely. The claim ow follows from Berstei s iequality Lemma ). Lemma 4. If u,...,u is a martigale differece vector sequece i.e., Eu i u,...,u i ] = 0 for all i =,...,), the ] E u i = E u i ]. Proof. Let s i := u + +u i for i =,...,; we have E s ] = E E u +s ]] u,...,u ]] = E E u +u s + s u,...,u so the claim follows by iductio. = E u ] +E s ] 7

8 A. Gaussia quadratic forms ad χ tail iequalities It is well-kow that if z N0,) is a stadard Gaussia radom variable, the z follows a χ distributio with oe degree of freedom. The followig iequality due to Lauret ad Massart 000) gives a boud o liear combiatios of χ radom variables. Lemma 5 χ tailiequality; Lauret ad Massart,000). Let q,...,q be idepedet χ radom variables, each with oe degree of freedom. For ay vector γ = γ,...,γ ) R + with o-egative etries, ad ay t > 0, ] Pr γ i q i > γ + γ t+ γ t e t. Proof of Propositio. Let VΛV be a eige-decompositio of A A, where V is a matrix of orthoormal eigevectors, ad Λ := diagρ,...,ρ ) is the diagoal matrix of correspodig eigevalues ρ,...,ρ. By the rotatioal ivariace of the distributio, z := V x is a isotropic multivariate Gaussia radom vector with mea zero. Thus, Ax = z Λz = ρ z + +ρ z, ad the zi are idepedet χ radom variables, each with oe degree of freedom. The claim ow follows from a tail boud for χ radom variables Lemma 5, due to Lauret ad Massart, 000). 8