ECE534, Spring 2018: Solutions for Problem Set #2

Transcription

1 ECE534, Srig 08: s for roblem Set #. Rademacher Radom Variables ad Symmetrizatio a) Let X be a Rademacher radom variable, i.e., X = ±) = /. Show that E e λx e λ /. E e λx = e λ + e λ = + k= k=0 λ k k k! = + k= λ k k! + k=0 λ /) k k! λ) k = k! = e λ. k=0 λ k k)! b) Let X be a zero mea radom variable suorted o the iterval a, b. i. Assume that X is a ideedet coy of X, i.e., X, X are i.i.d. radom variables. Usig Jese s iequality, show that E e λx E X,X e λx X ), where E X,X deotes exectatio with resect to the joit distributio of X, X. ii. Assume that ε is a Rademacher radom variable, ideedet of X, X. Observe that the radom variables X X ad εx X ) have the same distributio, which is symmetric. Use the tower roerty of coditioal exectatio ad the iequality i a to show that E e λx e λ b a) /. i) E X e λx = E X e λx E X X ) E X,X e λx X ), where EX = 0 ad Jese s iequality have bee used. ii) E X,X e λx X ) = E X,X,ε e λεx X ) = E X,X E ε e λεx X ) E X,X e λ X X ).

2 Sice X a, b, we have that X X b a) ad therefore, E X,X e λ X X ) E X,X e λ b a) = e λ b a). Hece, E e λx e λ b a).. Cocetratio of χ Radom Variables ad Radom rojectios A chi-squared radom variable S with degrees of freedom is a radom variable of the form: S = Zi, Z i N 0, ), i =,,..., ideedet i= ad is deoted by S χ. For this radom variable, ES =, i.e., the mea value coicides with the degrees of freedom. Moreover, the followig iequality holds: ) Zi > t e t /8, t 0, ). ) i= This is a cocetratio iequality for chi-squared radom variables, because it demostrates that as the umber of degrees of freedom icreases, the radom variable S/ caot be very far away from its mea value. This iequality lays a imortat role i aalyzig radom rojectios. Suose that we are give m data oits {x, x,..., x m } i R d. If d is large, it may be too exesive to store these vectors. This motivates the itroductio of a maig F : R d R, which reserves the essetial iformatio i the data oits ad allows for the storage of the rojected vectors {F x ),..., F x m )} istead of the iitial set of vectors. reservig the essetial iformatio of the data oits corresods, e.g., to the requiremet that F satisfies δ) x i x j F x i ) F x j ) + δ) x i x j, i, j {,,..., m} ) for some δ 0, ). Here, corresods to the Euclidea orm either i R d or i R. Such a requiremet ca always be achieved if is large eough, but the goal is to guaratee these iequalities for a small relative to d. Defie F : x Zx/, where Z R d is a matrix cotaiig i.i.d. N 0, ) radom variables. Verify that such a F satisfies ) with high robability by rovig the followig stes: a) Let Z i R d deote the ith row of Z. Argue that Z i = Z i x/ x is a N 0, ) radom variable.

3 For a fixed x 0, we have that Z i = d j= Z ijx j / x Z = Z ij for the matrix Z). Therefore, Zi is a liear combiatio of i.i.d. N 0, ) radom variables ad thus, Zi is also ormal. Moreover, deotig the coefficiets of this liear trasfomratio as a i = x i x, we observe that Var Z i ) = j= E Z i = d x j EZ ij = 0, x j= d x j Var Z ij ) x = d j= x j x =, where i Var Z i ) the ideedece of Z ij has bee used. Thus, Z i N 0, ). b) Coclude that S = i= Z i is a chi-squared radom variable with degrees of freedom. S = Z i= i is clearly a chi-squared radom variable by the relimiary defiitios i this roblem. We oly eed to verify that Z,..., Z are ideedet. This is straightforward due to the fact that i each Z i, a differet set of Z ij articiates. c) Use ) to show that F x) x ) / δ), + δ) e δ /8, for ay 0 x R d. By alyig ), we have ) Z i δ e δ 8, δ 0, ) i= Zx ) or x δ e δ 8, F x) ) or x δ e δ 8, ) F x) or x / δ), + δ) e δ 8, δ 0, ). 3

4 d) Use the uio boud to coclude that all the iequalities i ) are satisfied with robability at least ɛ for ay ɛ 0, ) if > 6 logm) + 8 log ) δ δ ɛ. Note that there are at most m ) differet airs of data oits xi, x j ). Thus, by the uio boud F xi x j ) x i x j ) m e δ 8. Requirig / δ), + δ)) for at least oe air x i, x j ) such that x i x j ) ) m e δ 8 m e δ 8 ɛ yields that > 6 logm)+ 8 log ) δ δ ɛ. I.e., > 6 logm)+ 8 log ) δ δ ɛ is sufficiet for all the iequalities give by ) to hold with robability at least ɛ. 3. Covergece i robability a) Let X, X,... be a sequece of radom variables such that EX µ ad VarX ) 0 as. Show that X µ as i robability. Cosider the determiistic sequece {a = EX }. The, for ay ɛ > 0, there exists N ɛ > 0 such that a µ = EX µ < ɛ, > N ɛ. Therefore, > N ɛ : X µ > ɛ) = X EX + EX µ > ɛ) X EX + EX µ > ɛ) = X EX > ɛ ) VarX ) ɛ 0 as. /4) Here, we use Chebyshev s iequality i the last ste. Hece, ɛ > 0, we have lim X µ > ɛ) = 0, i.e., X µ. b) Suose that we distribute balls ito boxes ideedetly at radom. Let E = I + I + + I be the umber of emty boxes, where I j is the emtiess idicator of the jth box. Usig art 3a show that E /e as. Note that I j Ber ) ) radom variables, because the robability of ot occuyig the jth box i every dro is ). This leads to EIj = ) 4

5 ad EE = ). Moreover, E E = E I j + E I i I j = j= i j EI i I j E I j + j= i j = ) + EI i I j. We ow observe that EI i I j = boxes i ad j are both emty) = ). Combiig, we see that VarE ) = E E E E ) = ) + ) i j ). ) Defie the radom variable S = E. It is easy to see that ES = EE as, while VarS ) = e ) + ) ) ) 0 + e e = 0. Therefore, by the revious art, S = E e. 4. All Tyes of Covergece of Radom Sequeces Cosider a sequece of radom variables X, X,... with its geeric term defied as follows: X Ber/) ad X = X + ) mod. Does this sequece coverge almost surely, i robability, i mea square ad i distributio? Justify your aswer for each tye of covergece searately. The sequece has oly two outcomes or samle aths) deedig o X : X = : The X X... = X = 0 : The X X... = a) {X } does ot coverge almost surely because the robability of every jum is. b) {X } does ot coverge i robability because the frequecy of jums is equal to. 5

6 c) {X } does ot coverge to mea value of X for all ) i mea square sese sice lim E X ) = E X X + 4 = EX EX + 4 = d) As, the value of the X is determied by X ad therefore, X Ber d ), due to X ), hece X Ber ). 5. Etroy Bouds For a ositive radom variable X, the followig defiitio of etroy is itroduced: HX) = EX log X EX log EX. Cosider the radom variable e λx ad the momet geeratig fuctio m X λ) = E e λx. It is the easy to see that H e λx) = λm Xλ) m X λ) log m X λ). Assume that there is a costat σ < such that H e λx) λ σ m Xλ), λ R +. Show that E e λx EX) e λ σ, λ R +. We eed to show that log m X λ) λ σ. Note that H e λx) λ σ m Xλ) is equivalet to λm X λ) m Xλ) log m X λ) λ σ m Xλ) or m X λ) λm X λ) log m λ X λ) σ, where we have divided by λ m X λ). We ow see that Also, 0 d log m X λ) = m X λ) dλ λ λm X λ) log m Xλ) λ. log m X λ) lim = m X 0) λ 0 λ m X 0) = EX. Therefore, for a arbitrary λ > 0, λ d log m X λ) d λ = log m Xλ) d λ λ λ EX λ 0 σ d λ = σ λ. 6

7 Multilyig both sides by λ, we obtai log E e λx EX) = log E e λx e λex = log E e λx λex σ λ. 6. Rademacher Comlexity Let X, X,..., X be ideedet radom variables such that X i a, b for all i {,,..., }. Cosider a fuctio f : R R, which is covex i each of its argumets ad L-Lischitz with resect to the Euclidea orm. The the followig iequality holds: E e λfx,x,...,x ) EfX,X,...,X )) e λ L b a), λ 0. 3) Cosider a set of vectors C R. The, RC) = E su c C ε i c i corresods to the Rademacher comlexity of C. Here, {ε i } are ideedet Rademacher radom variables as these were defied i roblem ad c = c, c,..., c is a vector i C. Let ˆRC) = su c C i= ε ic i deote the emirical versio of RC). Usig 3), argue that where diamc) = su c C c. i= ˆRC) RC) + t) e t 6diamC)), Note: A fuctio f : R R is said to be L Lischitz for some L 0 with resect to the Euclidea orm if fx) fy) L x y, x, y R. Also, the oitwise suremum fx) = su i I f i x) of a arbitrary family of covex fuctios {f i x) i I} is covex. Hit: First, show that L = diamc) is a valid uer boud to the Lischitz costat of fε,..., ε ) = su c C i= ε ic i for ε i {, +}. Combie the 3) with the Cheroff boud to get the desired iequality by choosig λ = t 8diamC)). We ote that the fuctio fε,..., ε ) = su c C i= ε ic i is covex, sice it is the suremum of a family of liear i.e., covex) fuctios. Moreover, fε,..., ε ) is 7

8 Lischitz with L = su c C c beig a valid uer boud to the Lischitz costat. To see this, cosider first the liear form gε; c) = i= c iε i for c C. The, gε; c) gε ; c) = i= c i ε i ε i) }{{} a) c ε ε su c ε ε. c C }{{} L I a), the Cauchy-Schwarz iequality has bee used. Therefore, gε; c) is Lischitz ad L is a valid uer boud to the corresodig Lischitz costat. For arbitrary ε, ε, we ca ow see that gε; c) gε ; c) + L ε ε. Takig the suremum with resect to c C, first to the right had side ad the to the left, we obtai: Moreover, iterchagig ε, ε, we obtai: fε,..., ε ) fε,..., ε ) L ε ε. fε,..., ε ) fε,..., ε ) L ε ε. Combiig, we have: fε,..., ε ) fε,..., ε ) L ε ε. Therefore, L is a uer boud to the Lischitz costat of f. Emloyig ow the Cheroff boud ad 3), we obtai: ˆRC) E e λ ˆRC) E e λ ˆRC) RC)) RC) + t) = e λ ˆRC)+t) e λt e 4λ L λt. Miimizig 4λ L λt with resect to λ yields that λ = t 8 L = t 8diamC)). luggig this value i the last iequality yields the desired result. 7. Covergece i robability Agai rove that X X if ad oly if lim E X X = 0. + X X 8

9 Without loss of geerality, take X = 0. We wat to show that X if lim E = 0. i) X X + X 0 = lim E X + X = 0. 0 if ad oly By X 0, we have that ɛ > 0 : lim X > ɛ) = 0. Note that Therefore, X + X X + X I X > ɛ + ɛi X ɛ I X > ɛ + ɛ. X E EI X > ɛ + ɛ = X > ɛ) + ɛ. + X Takig the limit, we obtai lim E X have that lim E X + X = 0. ii) lim E X + X = 0 = X 0. Observe that the fuctio fx) = x x+ + X ɛ, ad sice ɛ > 0 is arbitrary, we is icreasig. Therefore, ɛ + ɛ I X > ɛ X + X I X > ɛ X + X. Takig exectatios ad the limits to both sides, we obtai: ɛ + ɛ lim X > ɛ) lim E X = 0 X + Sice this holds for ay ɛ > 0, we have that lim X > ɛ) = 0, ɛ > 0. therefore, X Almost Sure Covergece Agai Bous roblem Let X, X,... be a sequece of i.i.d. radom variables with X > x) = e x, x 0. Show that max{x, X,..., X } a.s. as. log Note: Use the Borel-Catelli Lemma. 9

10 Let Y = max{x,..., X }. For ɛ > 0, we have ) ) maxx,..., X ) Y < ɛ = log log < ɛ Usig art a) i the Borel-Catelli lemma, = = Y < ɛ) log ) = i={x i < ɛ) log }) = e ɛ) log ) = ) ɛ = ) ) ɛ ɛ ɛ ) Y log < ɛ = = ) ɛ ɛ ɛ e ) ɛ <. ) Therefore, Y log < ɛ i.o. = 0. Usig similar stes ad focusig o the radom variable X, we have ) X log > + ɛ = e log +ɛ) = +ɛ. Therefore, = ) X log > + ɛ = = = <. +ɛ ) By art a) i the Borel-Catelli lemma, X log > + ɛ i.o. = 0. Also, ) = X log > ɛ = ) = =. By art b) i the Borel-Catelli lemma, X ɛ log > ɛ i.o. =. Combiig the above results, ad therefore, lim su X log = a.s. X log > + ɛ i.o. ) = 0. Hece, Y log = max i X i log a.s.. 0