Homework 1 Due: Wednesday, September 28, 2016

Transcription

1 0-704 Information Processing and Learning Fall 06 Homework Due: Wednesday, September 8, 06 Notes: For positive integers k, [k] := {,..., k} denotes te set of te first k positive integers. Wen p and Y q are random variables over te same sample space, D Y ), D q), and Dp Y ) sould all be read as Dp q). Te omework is out of 60 points.. Warm-up Problems a) 5 points) Two teams A and B play a best-of-five series tat terminates as soon as one of te teams wins tree games. Let be te random variable representing te outcome of te series, written as a string of wo won te individual games e.g., possible values of are AAA, BAAA, ABABB, etc.) Let Y be te number of games played before te series ends. Assuming tat A and B are equally matced and te outcomes of different games in te series are independent, calculate H), HY ), HY ), H Y ), and I; Y ) in bits). Let p A and q A be te distributions of and Y, respectively, given tat A wins te series. Calculate Dp A ) and Dq A Y ). b) 5 points) Suppose, Y, and Z are eac Bernoulli/) and are pairwise independent i.e., I; Y ) = IY ; Z) = I; Z) = 0). Wat is te minimum possible value of H, Y, Z)? Solution: a) Tere are ways te series can ave lengt 3, eac wit probability 3. 3 ) = 6 ways te series can ave lengt 4, eac wit probability 4. Tere are 4 ) = ways te series can ave lengt 5, eac wit probability 5. Hence, H) = 3 log log log 5 = 33 8, and HY ) = 4 log log 8 3 = log Since Y is a deterministic) function of, HY ) = 0, and I; Y ) = HY ) = log , H Y ) = H) HY ) = log Since p A x) is precisely twice P = x) werever p A is supported, we ave Dp A ) = E p A [log )] =. Finally, since Y is independent of weter A wins te series, q A is identical to te distribution of Y, and so Dq A Y ) = 0.

2 b) Applying te Cain Rule twice), te fact tat Sannon entropy is nonnegative, and te fact tat HY Z) = HY ) since Y and Z are independent) H, Y, Z) = H Y, Z) + HY Z) + HZ) HY Z) + HZ) = HY ) + HZ) =. Tis is acieved, for example, if Z = Y were denotes te exclusive or operation), since, in tis case, any of {, Y, Z} is a function of te remaining two.. General Data Processing a) 0 points) Suppose we ave two distributions p and p on [k], and, for eac i [k], a conditional distribution q i over [l]. Let q j) = k i= q ij)p i) and q j) = k i= q ij)p i) denote te marginal distributions over [l] induced by p and p, respectively. Prove te General Data Processing Inequality Dq q ) Dp p ). ) Hint: Use te log-sum inequality, wic states tat, for all non-negative sequences a,..., a n and b,..., b n, letting a = n i= a i and b = n i= b i, b) As special cases of ), sow: Solution: n i= a i log a i b i a log a b. i. 5 points) For random variables and Y taking values in [k] and function f wit domain [k], Df) fy )) D Y ) and Hf)) H). ii. 5 points) Te Data Processing Inequality from class: for a Markov cain Y Z, I; Z) I; Y ). a) For eac i [k] and j [l], define a j,i := q i j)p i) and b j,i := q i j)p i). Note tat q j) = k i= a j,i, and q j) = k i= b j,i. Hence, applying te log-sum inequality to eac term of te summation), D q q ) = l j= q j) log q j) q j) since, for eac i [k], l j= q ij) =. = = l k j= i= k l i= j= k i= a j,i log a j,i b j,i q i j)p i) log p i) p i) p i) log p i) p i) = D p p ),

3 b) i. Te inequality Df) fy )) D Y ) ) is precisely ) in te case tat q i is a delta function q i j) = {j=fi)} at fi). Since Hf) ) = 0, Hf)) = I; f)) = DP,f)) P P f) ). Applying ) wit te bivariate function x, y) x, fy)) gives Hf)) = DP,f)) P P f) ) DP,) P P ) = I; ) = H). ii. Recall tat, in general, I; Y ) is te expected divergence in te distribution of Y wen given. Applying te law of total probability, te fact tat P Z,Y = P Z Y, and inequality ), )] [ )] I; Z) = E D PZ PZ = E [D P Z,Y P Y P Z Y P Y Y Y )] = E [D P Z Y P Y P Z Y P Y 3. Plug-in estimator for differential entropy Y E [ D PY PY )] = I; Y ). Tis problem derives convergence rates for an estimator of te differential entropy Hp) = px) log px) dx of a probability density p, given n IID samples,..., n p. To simplify matters, we will make te following assumptions: i) Te sample space = [0, ] D is te D-dimensional unit cube. ii) We know positive lower and upper bounds on te true density p. 0 < κ inf x px) sup px) κ < x Te estimator in question is a plug-in estimator based on a truncated kernel density estimate KDE). Specifically, te estimate Ĥ is given by given by Ĥ = H p ) = p x) log p x) dx, 3) were, for some bandwidt > 0 and kernel K : R D R wit R D Ku) du =, is a truncated KDE of p. p x) = min { κ, max { κ, n d Y n ) }} x i K, 4) i= 3

4 You may take for granted te following facts about te integrated squared bias and variance of te truncated KDE: tere exist constants C 0, C > 0 suc tat, for all > 0, E [ p x)] px)) dx C 0 β 5) and V [ p x)] dx C n D. 6) Here, te Hölder parameter β > 0 is a measure of smootness of te probability density p. Larger β indicates smooter p, and ence less smooting bias. Te standard decomposition of mean-squared error into bias and variance gives [ E p x) px)) ] = E [ p x)] px)) + V [ p x)] dx C 0 β + C n D. Optimizing over gives te rate n β+d and plugging tis back in gives te integrated MSE rate E [ p x) px)) ] n β β+d. In tis problem, we will derive similar bounds for te plug-in entropy estimator, and study its optimal bandwidt and MSE. a) 5 points) Prove te bias bound ] E [Ĥ H C B β + β + n D for some C B depending only on C 0, C, κ, κ, and D. Hint: Along wit inequalities 5) and 6), a second-order Taylor expansion and Jensen s inequality may be useful.) b) 5 points) Tis part will use McDiarmid s inequality: Teorem. McDiarmid s Inequality): Suppose we ave n independent random variables,..., n taking values in a set Ω and a function f : Ω R suc tat, for some constants c,..., c n, ), sup fx,..., x n ) fx,..., x i, y, x i+,..., x n ) c i, x,...,x n,y Ω for eac i [n]. Ten, McDiarmid s inequality states tat, for any ε > 0, P [ f) E [f)] > ε] exp ε n i= c i ). Tese results can be found in any text on nonparametric estimation, suc as Tsybakov [008], Section.. 4

5 Essentially, if a function depends on many independent random variables, but not too muc on any one of tem, McDiarmid s inequality tells us tat te function s distribution is is tigtly concentrated around its expectation. Use McDiarmid s inequality to derive te exponential concentration bound [ ] ] P Ĥ E [Ĥ > ε exp C E ε n ), 7) for te plug-in estimator Ĥ, for some C E depending only on D, K, κ, and κ. Hint: Te mean value teorem will be useful ere.) ] c) 5 points) Use 7) to prove te variance bound V [Ĥ C V n, wit C V depending only on D, K, κ, and κ. Hint: Recall tat, for a non-negative random variable, E [] = 0 P [ > x] dx.) d) 5 points) Combine te [ bias and variance bounds to derive a bound on te mean ) ] squared error MSE) E Ĥ H of Ĥ. Optimize tis over. Wat are te optimal bandwidt and MSE rates asymptotically, as n )? How do tese compare to te optimal bandwidt and MSE rates for kernel density estimation smaller, same, or larger)? Note: Wen initially writing tis problem, I ad omitted te explicit assumption tat tere exists a known upper bound sup x px) κ <. Te existence of suc a κ uniformly over te class of distributions under consideration actually follows from assuming Hölder continuity. You were not required to sow tis, but, for completeness, we sow tis ere in te case tat p is β-hölder continuous wit constant L, for some β 0, ]. Proof: Since p is continuous and is compact, x := argmax x px) exists. Let u := ) /β, so tat, by te Hölder condition, for all x Bx, u), px) px )/. Since px ) L p is a probability density, Bx,u) px) dx Bx,u) px ) µb0, u)) D px ), te D term is due te fact tat at least one ortant of Bx, u) must lie entirely witin ). Since u increases wit px ), te rigt side clearly increases unboundedly wit px ), and so te latter must be bounded. Since we need Hölder continuity to bound te bias of p anyway, te existence of κ is a very mild assumption. Te existence of κ is a muc stronger assumption, but can only be weakened sligtly. Solution: Based on Liu et al. [0]) a) Define g : κ, κ ) R by gz) = z log z. Define constants G := sup z [κ,κ ] g z) = max { + logκ ), + logκ )} Here, Bx, u) denotes te ball of radius u centered at x, in te same metric as te Hölder condition, and µ denotes its Lebesgue measure. 5

6 and G := sup g z) =. z [κ,κ ] κ Note tat, for any x, y [κ, κ ], tere exists z [x, y] [y, x] [κ, κ ] suc tat ] E [Ĥ [ H E [ E gx) gy) = g y)y x) + g z) y x). g px)) gpx)) dx] g px))px) px)) + g ζx)) px) px)) dx] G px) E [ px)] + G E [ px) px)) ] dx Applying Jensen s inequality since as Lebesgue measure ), ] E [Ĥ H G px) E [ px)]) dx + G G C0 β + G C 0 β + C ) n D C B β + β + n D ), [ E px) px)) ] dx were te second inequality is by te given bounds on te integrated squared bias and integrated MSE of te kernel density estimator, and { C B := max G C0, G } max {C 0, C }. b) For sake of applying McDiarmid s inequality, let p denote te KDE p wen te i t sample i is replaced by an independent sample i, and let Ĥ denote te corresponding plug-in estimate. By te mean value teorem, for any x, y > 0, x log x y log y + max{ log x, log y }) x y. Hence, letting κ := + max{ log κ, log κ }, since bot p and p i) lie in [κ, κ ], Ĥ Ĥ = px) log px) p x) log p x) dx px) log px) p x) log p x) dx κ px) p x) dx. Note tat, since p and p differ in only one sample, almost all terms in p p cancel out: px) p x) = ) ) x n D K i x K i 6

7 Now, applying te cange of variables u = x i, since te Jacobian of tis transformation as determinant J u x = I D = D, ) ) Ĥ κ Ĥ x n D K i x K i dx κ ) x n D K i dx κ K u) du = κ n n K. Hence, by McDiarmid s inequality, for C E =, κ K [ ]] P Ĥ E [Ĥ exp c) By te previous part, for C V := C E, ] [ Ĥ ]) ] V [Ĥ = E E [Ĥ = = ) ε n i= κ n K ) = exp C E nε ). [ Ĥ ]) ] P E [Ĥ > ε dε [ ] ] P Ĥ E [Ĥ > ε dε exp C E εn) dε = C E n = C V n. d) Combining parts a) and c) via te usual bias-variance decomposition of MSE gives [ Ĥ ) ] [ ] ] E H = E Ĥ H + V [Ĥ CB β + β + ) n D + C V n. 8) Note tat te variance does not depend on, and so we can just optimize te bias bound over. Also, since 0 as n, te β term is negligible; we replace it wit a constant factor of. Hence, since te bias bound is convex in, at te optimal bandwidt, we ave 0 = d d β + n D = β β = Dn D+) and so n β+d. Plugging tis into 8) gives [ Ĥ ) ] ) { E H n β β+d + n = n min = β+d D β n, β, β+d }. For any values of β and D, tis rate is faster tan te n estimation, and uses a smaller bandwidt tan n β+d. β β+d optimal rate for density 7

8 References Han Liu, Larry Wasserman, and Jon D Lafferty. Exponential concentration for mutual information estimation wit application to forests. In Advances in Neural Information Processing Systems, pages , 0. A.B. Tsybakov. Introduction to Nonparametric Estimation. Springer Publising Company, Incorporated, st edition, 008. ISBN ,