15.1 Upper bound via Sudakov minorization

Transcription

1 ECE598: Information-theoretic methos in high-imensional statistics Spring 206 Lecture 5: Suakov, Maurey, an uality of metric entropy Lecturer: Yihong Wu Scribe: Aolin Xu, Mar 7, 206 [E. Mar 24] In this lecture we stuy the upper an lower bouns on NB, 2, ɛ). From the last lecture, we know that for any Θ R an any ɛ > 0, volθ) volɛb) MΘ,, 2ɛ) NΘ,, ɛ) MΘ,, ɛ) volθ + ɛ 2 B) vol ɛ 2 B). where B is the ball of raius measure by. Therefore, MB, 2, ɛ) volb + ɛ 2 B 2) vol ɛ 2 B 2) vol + ɛ 2 )B ) vol ɛ 2 B = 2) + ɛ ) ) 2 c ɛ + c ) 2 2 ɛ, where we have use the fact that B 2 B by Cauchy-Schwarz inequality, volb ) /, an volb 2 ) /. On the other han, MB, 2, ɛ) volb ) volɛb 2 ) = ) ) volb ) c ɛ volb 2 ) = ɛ. Note that the lower boun erive above is useful only when ɛ. 5. Upper boun via Suakov minorization Recall that the Gaussian with of Θ R is efine as wθ) = E sup θ, Z, where Z N0, I ). θ Θ Theorem 5. Suakov minorization). For any Θ R an any ɛ > 0, wθ) ɛ log MΘ, 2, ɛ). The proof of Theorem 5. relies on Slepian s Gaussian comparison lemma: Lemma 5. Slepian s lemma). Let X = X,..., X n ) an Y = Y,..., Y n ) be Gaussian ranom vectors. If EY i Y j ) 2 EX i X j ) 2 for all i, j, then E max Y i E max X i. For a self-containe proof see [Cha05]. 2 See also [Pis99, Lemma 5.7, p. 70] for a simpler proof of a weaker version E max X i 2E max Y i, which suffices for our purposes though. To avoi measurability ifficulty, wθ) shoul be unerstoo as sup T Θ, T < E max θ T θ, Z. 2 If you took ECE 534 last fall, you shoul revisit Problem 5 of fall5a/homework/hw4.pf which follows [Cha05].

2 Proof of Theorem 5. assuming Slepian. Let {θ,..., θ M } be an he optimal ɛ-packing of Θ. Let i.i.. X i = θ i, Z for i [M], where Z N 0, I ). Let Y i N 0, ɛ 2 /2). Then for any pair i, j, X i an X j are jointly Gaussian, an EX i X j ) 2 = θ i θ j ) E[ZZ ]θ i θ j ) = θ i θ j 2 2 ɛ 2 = EY i Y j ) 2. It follows from Lemma 5. that E max i M X i E max i M Y i ɛ log M. This completes the proof because E sup θ Θ θ, Z E max i M X i. We can apply this theorem to Θ = B. In this case, by the efinition of the ual norm, wb ) = E The theorem then implies that sup x R : x x, Z = E Z log. log MB, 2, ɛ) log ɛ 2. 5.) This boun is almost optimal: When ɛ /, this upper boun is in fact optimal an) much better than what we get from the volume argument, which is log MB, 2, ɛ) log + c ) ɛ. However, 5.) is not always sharp. For example, when ɛ /, it gives log an we know even from volume boun) that the correct behavior is. This suggests we nee a more refine boun that interpolates between volume an Suakov. 5.2 Upper boun via Maurey s empirical metho We can construct a covering of B using the probabilistic metho. Let {e i, i =,..., } be the stanar basis of R. For an arbitrary x B, efine a imensional ranom vector Z as { sgnx i )e i w.p. x i Z = 0 w.p. x Z has the property that EZ i = x i for i =,...,, hence EZ = x, an Var[Z i ] = EZ i x i ) 2 for i =,...,. Let Z ),..., Z k) be i.i.. copies of Z, an let Z = k k j= Z j). Then E Z x 2 2 = E Z i x i ) 2 = Var[ Z i ] = k Var[Z i ] = k E Z x 2 2 k E Z x 2 k, where we have use the facts that Var[ Z i ] = k Var[Z i] an Z x 2 Z x. If we choose k = /ɛ 2, then E Z x 2 E Z x 2 2 ɛ. So there is a realization z of Z such that z x 2 ɛ. 2

3 Now we examine how many ifferent values Z can take regarless of x. Note that where Z = k k Z j) = k K,..., K ), j= K i k, with K i Z, an 0 K i k for i =,...,. 5.2) For any K,..., K ) satisfying inequality 5.2), we get a solution for the following inequality K i + + Ki k, with K Z, an 0 K k for i =,...,, 5.3) by setting K i + = K i an Ki = 0 if K i 0, an setting K i + = 0 an Ki = K i if K i < 0. Therefore, the number of values Z can take is upper boune by the number of solutions of inequality 5.3). Note that there are ) k+2 2 solutions for K i + + Ki = k, with K Z, an 0 K k for i =,...,, because the solutions are all possible types of the sequences of length k with alphabet size 2. It follows that the number of solutions of inequality 5.3) is ) ) ) ) ) k + 2 k + 2 k = =, k which is an upper boun on the number of Z s regarless of x. We thus have shown the existence of an ɛ-covering of B in 2 with carinality upper boune by ) + 2 ) + 2 ɛ 2 ɛ = 2. 2 Therefore, log NB, 2, ɛ) 2 log + ɛ 2 ) ɛ 2 log + 2ɛ 2). 2ɛ 2 We can see that the first upper boun recovers the result from the volume argument, while the secon upper boun is even stronger than the result obtaine from Suakov s minorization. 5.3 Lower boun via packing Hamming spheres Let S k = {x {0, } : w H x) = k} be the Hamming sphere of raius k. For the 2ρ-packing of S k in Hamming istance H, we have log MS k, H, 2ρ) S k k) B H ρ) = ρ i=0 i). This leas to the following lemma. 3

4 Lemma 5.2 Gilbert-Varshamov). There exist constants c an c 2 such that for all N an any k [], log MS k, H, c k) c 2 k log e k. Now we construct a packing of B base on a packing of S k. Let {x,..., x M } be a c k-packing of S k. Let θ i = x i /k. Then θ i B for i =,..., M, an θ i θ j 2 2 = k 2 x i x j H c k. Therefore, {θ,..., θ M } is a c /k-packing of B in 2. Choosing k = /ɛ 2, it follows from Lemma 5.2 that log MB, 2, c ɛ) c 2 ɛ 2 log eɛ 2) for some constants c an c 2. To summarize, combining the upper an lower bouns, we have log ɛ 2 ) ɛ ɛ 2 log NB, 2, ɛ) ɛ. 5.4) log ɛ ɛ Duality First we efine a more general notoin of covering number. For K, T R, efine the covering number of K using translates of T as NK, T ) = min{n : x,..., X N R such that K = N T + x i }. An amazing theorem of Artstein-Milman-Szarek [AMS04] establishes the following uality result for metric entropy: There exist constants α an β such that for any symmetric convex boy K, β log N B 2, ɛ α K ) log NK, ɛb 2 ) log NB 2, αɛk ), where B 2 is the usual unit l 2 -ball, { } K = y : sup x, y x K is the polar boy of K. For example, Bp = B q whenever p + q =. Therefore by uality, 5.4) also applies to log NB 2,, ɛ), which is what is neee for application to minimax risk. 5.5 Example Finally, we use the results in this lecture to erive the minimax lower boun for the p-imension, n-sample Gaussian location moel with respect to the istortion function θ ˆθ 2. 4

5 We can construct an ɛ-packing of B 2 δ) in. From the Fano s metho, R ɛ 2 iam KL{Nθ, n I ) p), θ B 2 δ)}) log MB 2 δ),, ɛ) = ɛ 2 nδ 2 ) log MB 2 δ),, ɛ) = ɛ 2 nδ 2 ) log MB, 2, ɛ/δ) ɛ 2 nδ2 ) δ 2 log + pɛ2 ɛ 2 δ 2 where we have use that fact that iam KL {Nθ, n I p), θ B 2 δ)}) = niam 2 2 B 2 δ)), the uality theorem, an the upper boun on log MB, 2, ɛ/δ). Choosing ɛ = c n an δ = c 2 ɛ with appropriate c an c 2 such that the parenthesis in the lower boun is a positive constant, we obtain R n. An alternative proof of this result is by choosing the packing set as τ{e,..., e p } for some τ > 0 to be etermine later. This set is a τ-packing of R in, because τe i e j ) = τ for all pairs {i, j}. We also have τe i e j ) 2 2 = 2τ 2. Then by Fano s metho, R τ 2 2nτ 2 ). Choosing τ = c n with some appropriate constant c such that the parenthesis in the above boun is a positive constant, we obtain References R n. [AMS04] Shiri Artstein, Vitali Milman, an Stanis law J Szarek. Duality of metric entropy. Annals of mathematics, pages , [Cha05] Sourav Chatterjee. An error boun in the Suakov-Fernique inequality. arxiv preprint arxiv:050424, [Pis99] G. Pisier. The volume of convex boies an Banach space geometry. Cambrige University Press,