Exponential Concentration for Mutual Information Estimation with Application to Forests

Transcription

1 Exponential Concentration for Mutual Information Estimation wit Application to Forests Han Liu Department of Operations Researc and Financial Engineering Princeton University, NJ 8544 Jon Lafferty Department of Computer Science Department of Statistics University of Cicago, IL 6637 Larry Wasserman Department of Statistics Macine Learning Department Carnegie Mellon University, PA 53 Abstract We prove a new exponential concentration inequality for a plug-in estimator of te Sannon mutual information. Previous results on mutual information estimation only bounded expected error. Te advantage of aving te exponential inequality is tat, combined wit te union bound, we can guarantee accurate estimators of te mutual information for many pairs of random variables simultaneously. As an application, we sow ow to use suc a result to optimally estimate te density function and grap of a distribution wic is Markov to a forest grap. Introduction We consider te problem of nonparametrically estimating te Sannon mutual information between two random variables. Let and be two random variables wit domains and and joint density px, x. Te mutual information between and is px, x I ; := px, x log dx dx = H H H,, px px were H, = px, x log px, x dx dx and similarly for H and H are te corresponding Sannon entropies 4]. Te mutual information is a measure of dependence between and. To estimate I ; well, it suffices to estimate H, := Hp. A simple way to estimate te Sannon entropy is to use a kernel density estimator DE,, 9, 5,, 7], i.e., te densities px, y, px, and py are separately estimated from samples and te estimated densities are used to calculate te entropy. Alternative metods involve estimation of te entropies using spacings 5, 6, 3], k-nearest neigbors, ], te Edgewort expansion 4], and convex optimization 7]. More discussions can be found in te survey articles, 9]. Tere ave been many recent developments in te problem of estimating Sannon entropy and related quantities as well as application of tese results to macine learning problems 8,, 8, 6]. Under weak conditions, it as been sown tat tere are estimators tat acieve te parametric n-rate of convergence in mean squared error MSE, were n is te sample size. In tis paper, we construct an estimator wit tis rate, but we also prove an exponential concentration inequality for te estimator. More specifically, we sow tat our estimator Ĥ of Hp satisfies sup P Ĥ Hp > ɛ exp nɛ p Σ 36κ.

2 were Σ is a nonparametric class of distributions defined in Section and κ is a constant. To te best of our knowledge, tis is te first suc exponential inequality for nonparametric Sannon entropy and mutual information estimation. Te advantage of tis result, over te usual results wic state tat E Ĥ Hp = On, is tat we can apply te union bound and tus guarantee accurate mutual information estimation for many pairs of random variables simultaneously. As an application, we consider forest density estimation 5], wic, in a d-dimenionsal problem, requires estimating dd mutual informations in order to apply te Cow-Liu algoritm. As long as log d n as n, we can estimate te forest grap well, even if d = dn increases wit n exponentially fast. Te rest of tis paper is organized as follows. Te assumptions and estimator are given in Section. Te main teoretical analysis is in Section 3. In Section 4 we sow ow to apply te result to forest density estimation. Some discussion and possible extensions are provided in te last section. Estimator and Main Result Let =, R be a random vector wit density px := px, x and let x,..., x n R be a random sample from p. In tis paper, we only consider te case of bounded domain =, ]. We want to estimate te Sannon entropy Hp = px log pxdx.. We start wit some assumptions on te density function px, x. Assumption. Density assumption. We assume te density px, x belongs to a nd-order Hölder class Σ κ, L and is bounded away from zero and infinity. In particular, tere exist constants κ, κ < κ min px max px κ <,. x x and for any x, x T, tere exists a constant L suc tat, for any u, v T px u, x v px, x px, x u px, x v Lu v..3 x x Assumption. Boundary assumption. If {x n } is any sequence converging to a boundary point x, we require te density px as vanising first order partial derivatives: px n px n lim = lim =..4 n x n x To efficiently estimate te entropy in., we use a DE based plug-in estimator. Bias at te boundaries turns out to be very important in tis problem; see ] for a discussion of boundary bias. To correct te boundary effects, we use te following mirror image kernel density estimator: { n x x i x x i x x i p x, x := n x x i i= x x i x x i x x i x x i x x i x x i x x i x x i x x i x x i x x i x x i x x i x x i }..5 Here is te bandwidt and is a univariate kernel function. We denote by u, v := uv te bivariate product kernel. Tis estimator as nine terms; one corresponds to te original data in te unit square, ], and eac of te remaining terms corresponds to reflecting te data across one of te four sides or four corners of te square.

3 Assumption.3 ernel assumption. Te kernel is nonnegative and as a bounded support, ] wit udu = and uudu =. By Assumption., te values of te true density lie in te interval κ, κ ]. We propose a clipped DE estimator p x = T κ,κ p x,.6 were T κ,κ a = κ Ia < κ a Iκ a κ κ Ia > κ, so tat te estimated density also as tis property. Letting gu = u log u, we propose te following plug-in entropy estimator: H p := g p x dx = p x log p xdx..7 Remark.. Te clipped estimator p requires te knowledge of κ and κ. In applications, we do not need to know te exact values of κ and κ ; lower and upper bounds are sufficient. Our main tecnical result is te following exponential concentration inequality on H p around te population quantity Hp. Our proof is given in Section 3. Teorem.. Under Assumptions.,., and.3, if we coose te bandwidt according to n /4, ten tere exists a constant N suc tat for all n > N, sup P p Σ κ,l were κ = max { log κ, log κ }. H p H p > ɛ exp nɛ 36κ,.8 To te best of our knowledge, tis is te first time an exponential inequality like.8 as been establised for Sannon entropy estimation over te Hölder class. It is easy to see tat.8 implies te parametric n-rate of convergence in mean squared error, E Ĥ Hp = On /. Te bandwidt n /4 in te above teorem is different from te usual coice for optimal bivariate density estimation, wic is P n /6 for te nd-order Hölder class. By using n /4, we undersmoot te density estimate. As we sow in te next section, suc a bandwidt coice is important for acieving te optimal rate for entropy estimation. Let Ip := I ; be te Sannon mutual information, and define p x, x I p := p x, x log dx dx..9 p x p x Te next corollary provides an exponential inequality for Sannon mutual information estimation. Corollary.. Under te same conditions as in Teorem., if we coose n /4, ten tere exists a constant N, suc tat for all n > N, sup P I p I p > ɛ 6 exp nɛ p Σ κ,l 34κ,. were κ = max { log κ, log κ }. Proof. Using te same proof for Teorem., we can sow tat.8 also olds for estimating univariate entropies H and H. Te desired result ten follows from te union bound since Ip := I ; = H H H,. Remark.. We use te same bandwidt n /4 to estimate te bivariate density px, x and univariate densities px, px. A related result is presented in 5]. Tey consider te same problem setting as ours and also use a DE based plug-in estimator to estimate te mutual information. However, unlike our proposal, tey advocate te use of different bandwidts for bivariate and univariate entropy estimations. For bivariate case tey use n /6 ; for univariate case tey use n /5. Suc bandwidts and are useful for optimally estimating te density functions. However, suc a coice acieves a suboptimal rate in terms of mutual information estimation: sup p Σκ,L P I p I p > ɛ c exp c n /3 ɛ, were c and c are two constants. Our metod acieves te faster parametric rate. 3

4 3 Teoretical Analysis Here we present te detailed proof of Teorem.. To analyze te error H p H p, we first decompose it into a bias or approximation error term, and a variance or estimation error term: H p H p H p EH p EH p H p. 3. }{{}}{{} Variance Bias We are going to sow tat sup P H p EH p > ɛ p Σ κ,l }{{} Variance sup p Σ κ,l EH p H p }{{} Bias exp nɛ 3κ, 3. c c 3 n, 3.3 were c and c 3 are two constants. Since te bound on te variance in 3. does not depend on, to optimize te rate, we only need to coose to minimize te rigtand side of 3.3. Terefore n /4 acieves te optimal rate. In te rest of tis section, we bound te bias and variance terms separately. 3. Analyzing te Bias Term Here we prove 3.3. Let u be a vector. We denote te sup norm by u. Te next lemma bounds te integrated squared bias of te kernel density estimator over te support :=, ]. Lemma 3.. Under Assumptions.,., and.3, tere exists a constant c > suc tat E p x px dx c sup p Σ κ,l Proof. We partition te support :=, ] into tree regions = B C I, te boundary area B, te corner area C, and te interior area I: C = {x : x u for u =, T, or, T, or, T, or, T }, 3.5 B = {x : x is witin distance to an edge of, but does not belong to C}, 3.6 I = \ C B. 3.7 We ave te following decomposition: E p x px dx = I C E p x px dx = T I T C T B. B From standard results on kernel density estimation, we know tat sup p Σ,L T I c 4. In te next two subsections, we bound T B := E p x px dx and T C := E p x px dx. 3.. Analyzing T B B Let A := {x : x and x }. We ave T B = E p x px dx c E p x px dx. 3.8 For x A, we ave p x = n n i= Terefore, for x A we ave B x x i E p x = A x x i x t 4 C x x i x t pt, t dt dt x x i ]. 3.9

5 = x x x t x t pt, t dt dt u u px u, x u du du Since p Σ κ, L and < x, we ave u u px u, x u du du. 3. px u, x u px, x px, u L u, px u, x u px, x px x u px u L u u x x ]. Since u, u, we ave px u, x u px, x px x px x L u. Similarly, px u, x u px, x 9 px x px x L. For any x A, we can bound te bias term E p x px 3. = E p x u u pt, t du du 3. x x u u px u, x u px, x du du 3.3 px x L L = 4L, were te last inequality follows from te fact tat u u p u x, x u px, x du du 3.4 px x L px x, px L, by te Hölder condition and te assumption tat te density px as vanising partial derivatives on te boundary points. Terefore, we ave T B c Analyzing T C Let A := {x : x, x }. We now analyze te term T C : T C = E p x px dx c E p x px dx. 3.5 C A For notational simplicity, we write x a U x, a, b = x x b. 3.6 For x A, we ave p x = n Ux, n x i, x i U x, x i, x i U x, x i, x i U x, x i, x i ]. 3.7 i= Terefore, for x A we ave E p x = U x, t, t U x, t, t U x, t, t U x, t, t ] pt, t dt dt 5

6 = x x x x x x x u u px u, x u du du 3.8 u u pu x, x u du du 3.9 u u pu x, x u du du 3. u u p x u, x u du du. 3. x Since is a symmetric kernel on, ], we ave x x x u u du du = x u u du du = Terefore, for x = x, x T A, px, x = x x x x x x x x x x u u du du, 3. x u u du du. 3.3 px, x u u du du. x Using te fact tat p Σ κ, L, x, x, and u, u, we ave px u, x u px, x 4L, 3.4 pu x, x u px, x L, 3.5 pu x, u x px, x L, 3.6 pu x, u x px, x 36L. 3.7 For x A, we can ten bound te bias term as E p x px 3.8 = E p x u u pt, t du du 3.9 x x x x x x u u px u, x u px, x du du 3.3 u u pu x, x u px, x du du 3.3 u u pu x, u x px, x du du 3.3 x x u u pu x, u x px, x du du L Terefore, we ave T C c 6. Combining te analysis of T B, T C, and T I, we sow tat te mirror image kernel density estimator is free of boundary bias. Tus te desired result of Lemma 3. is proved Analyzing te Bias of te Entropy Estimator Lemma 3.. Under Assumptions.,., and.3, tere exists a universal constant C tat does not depend on te true density p, suc tat sup EH p Hp C n p Σ κ,l 6

7 Proof. Recalling tat gu = u log u, by Taylor s teorem we ave g p x g px = logpx ] p x px, ξx p x px] 3.36 were ξx lies in between p x and px. It is obvious tat κ ξx κ. Let κ be as defined in te statement of te teorem. Using Fubini s teorem, Hölder s inequality and te fact tat te Lebesgue measure of is, we ave EH p Hp 3.37 = E g p x g px ] dx 3.38 = E g p x g px ] dx 3.39 ] dx logpx E p x px dx ξx E p x px] ] dx dx κ E p x px E p x px] 3.4 κ ] dx dx. κ E p x px E p x px] 3.4 κ c c 4 c 3 n. 3.4 Te last inequality follows from standard results of kernel density estimation and Lemma 3., were c, c, c 3 are tree constants. We get te desired result by setting n /4. 3. Analyzing te Variance Term Lemma 3.3. Under Assumptions.,., and.3, we ave, sup P H p EH p > ɛ exp nɛ p Σ κ,l 3κ Proof. Let p x be te kernel density estimator defined as in.6 but wit te jt data point x j replaced by an arbitrary value x j. Since g u = log u, by Assumption., we ave max g p x, g p x ] κ. For notational simplicity, we write te product kernel as =. Using te mean-value teorem and te fact tat T κ,κ is a contraction, we ave sup H p H p 3.44 x,...,x n,x j = sup g p x g p x ] dx 3.45 x,...,x n,x j κ sup p x p x dx 3.46 x,...,x n,x j = κ sup T κ,κ p x] T κ,κ p x] dx 3.47 x,...,x n,x j 4κ sup x j x,...,x n,x j n x x j n x dx 3.48 y x 8κ sup y n dx κ udu = 8κ n n. 3.5 Terefore, using McDiarmaid s inequality 6], we get te desired inequality Te uniformity result olds since te constant does not depend on te true density p. 7

8 4 Application to Forest Density estimation We apply te concentration inequality. to analyze an algoritm for learning ig dimensional forest grap models 5]. In a forest density estimation problem, we observe n data points x,..., x n R d from a d-dimensional random vector. We ave two learning tasks: i we want to estimate an acyclic undirected grap F = V, E, were V is te vertex set containing all te random variables and E is te edge set suc tat an edge j, k E if and only if te corresponding random variables j and k are conditionally independent given te oter variables \{j,k} ; ii once we ave an estimated grap F, we want to estimate te density function px. Using te negative log-likeliood loss, Liu et al. 5] sow tat te grap estimation problem can be recast as te problem of finding te maximum weigt spanning forest for a weigted grap, were te weigt of te edge connecting nodes j and k is I j ; k, te mutual information between tese two variables. Empirically, we replace I j ; k by its estimate Î j; k from.9. Te forest grap can be obtained by te Cow-Liu algoritm 3, 3], wic is an iterative algoritm. At eac iteration te algoritm adds an edge connecting tat pair of variables wit maximum mutual information among all pairs not yet visited by te algoritm, if doing so does not form a cycle. Wen stopped early, after s < d edges ave been added, it yields te best s-edge weigted forest. Once a forest grap F = V, Ê is estimated, we propose to estimate te forest density as p F x = p x j, x k p x j p x k p x u p x l, 4. j,k Ê u Û l V \Û were Û is te set of isolated vertices in te estimated forest F. Our estimator is different from te estimator proposed by 5] once te grap F is given, we treat te isolated variables differently tan te connected variables. As will be sown in Teorem 4., suc a coice leads to minimax optimal forest density estimation, wile te obtained rate from 5] is suboptimal. Let Fd s denote te set of forest graps wit d nodes and no more tan s edges. Let D be te ullback-leibler divergence. We define te s-oracle forest Fs := V, E and its corresponding oracle density estimator p F to be Fs = arg min Dp p F and p F := px j, x k px l. 4. F Fd s px j, x k j,k E l V Let Σ κ, L be defined as in Assumption.. We define a density class P κ as P κ := { p : p is a d-dimensional density wit px j, x k Σ κ, L for any j k }. 4.3 Te next two teorems sow tat te above forest density estimation procedure is minimax optimal for bot grap recovery and density estimation. Teir proofs are provided in a tecnical report 4]. Teorem 4. Grap Recovery. Let F be te estimated s-edge forest grap using te Cow-Liu algoritm. Under te same condition as Teorem in 5], If we coose n /4 for te mutual information estimator in.9, ten sup p P κ P F F s = O s n wenever log d n. 4.4 Teorem 4. Density Estimation. Once te s-edge forest grap F as in Teorem 4. as been obtained, we calculate te density estimator B. by coosing n /5 and n /6. Ten, sup E p F x p F x s dx C p P κ n d s. 4.5 /3 n4/5 5 Discussions and Conclusions Teorem 4. allows d to increase exponentially fast as n increases and still guarantees grap recovery consistency. Teorem 4. provides te rate of convergence for te L -risk. Te obtained rate is minimax optimal over te class P κ. Te term sn /3 corresponds to te price paid to estimate bivariate densities; wile te term d sn 4/5 corresponds to te price paid to estimate univariate densities. In tis way, we see tat te exponential concentration inequality for Sannon mutual information leads to significantly improved teoretical analysis of te forest density estimation, in terms of bot grap estimation and density estimation. Tis researc was supported by NSF grant IIS-673 and AFOSR contract FA

9 References ] Ibraim A. Amad and Pi-Er Lin. A nonparametric estimation of te entropy for absolutely continuous distributions corresp.. IEEE Transactions on Information Teory, 3:37 375, 976. ] J Beirlant, E J Dudewicz, L Györfi, and E C Van Der Meulen. Nonparametric entropy estimation: An overview. International Journal of Matematical and Statistical Sciences, 6:7 39, ] C. Cow and C. Liu. Approximating discrete probability distributions wit dependence trees. Information Teory, IEEE Transactions on, 43:46 467, ] Tomas M. Cover and Joy A. Tomas. Elements of Information Teory. Wiley, 99. 5] Paul P. B. Eggermont and Vincent N. LaRiccia. Best asymptotic normality of te kernel density entropy estimator for smoot densities. IEEE Transactions on Information Teory, 454:3 36, ] A. Gretton, R. Herbric, and A. J. Smola. Te kernel mutual information. In Acoustics, Speec, and Signal Processing, 3. Proceedings.ICASSP 3. 3 IEEE International Conference on, volume 4, pages IV 88. IEEE, 3. 7] Peter Hall and Sally Morton. On te estimation of entropy. Annals of te Institute of Statistical Matematics, 45:69 88, ] A. O. Hero III, B. Ma, O. J. J. Micel, and J. Gorman. Applications of entropic spanning graps. Signal Processing Magazine, IEEE, 95:85 95,. 9] Harry Joe. Estimation of entropy and oter functionals of a multivariate density. Annals of te Institute of Statistical Matematics, 44: , December 989. ] M. C. Jones, M. C. Linton, and J. P. Nielsen. A simple bias reduction metod for density estimation. Biometrika, 8:37 338, 995. ] Siraj an, Sarba Bandyopadyay, Auroop R. Ganguly, Sunil Saigal, David J. Erickson, Vladimir Protopopescu, and George Ostroucov. Relative performance of mutual information estimation metods for quantifying te dependence among sort and noisy data. Pys. Rev. E, 76:69, Aug 7. ] Alexander raskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Pysical review. E, Statistical, nonlinear, and soft matter pysics, 696 Pt, June 4. 3] Josep B. ruskal. On te sortest spanning subtree of a grap and te traveling salesman problem. Proceedings of te American Matematical Society, 7:48 5, ] Han Liu, Jon Lafferty, and Larry Wasserman. Optimal forest density estimation. Tecnical Report,. 5] Han Liu, Min u, Haijie Gu, Anupam Gupta, Jon D. Lafferty, and Larry A. Wasserman. Forest density estimation. Journal of Macine Learning Researc, :97 95,. 6] C. McDiarmid. On te metod of bounded differences. In Surveys in Combinatorics, number 4 in London Matematical Society Lecture Note Series, pages Cambridge University Press, August ] uanlong Nguyen, Martin J. Wainwrigt, and Micael I. Jordan. Estimating divergence functionals and te likeliood ratio by convex risk minimization. IEEE Transactions on Information Teory, 56: ,. 8] D. Pál, B. Póczos, and C. Szepesvári. Estimation of Rényi entropy and mutual information based on generalized nearest-neigbor graps. Arxiv preprint ariv:3.954,. 9] L. Paninski. Estimation of entropy and mutual information. Neural Computation, 56:9 53, 3. ] Liam Paninski and Masanao Yajima. Undersmooted kernel entropy estimators. IEEE Transactions on Information Teory, 549: , 8. ] Barnabás Póczos and Jeff G. Scneider. Nonparametric estimation of conditional information and divergences. Journal of Macine Learning Researc - Proceedings Track, :94 93,. ] B. W. Silverman. Density Estimation for Statistics and Data Analysis. Capman and Hall. New York, NY, ] A.B. Tsybakov and van den Meulen. Root-n Consistent Estimators of Entropy for Densities wit Unbounded Support, volume 3. Universite catolique de Louvain,Institut de statistique, ] Marc M. Van Hulle. Edgewort approximation of multivariate differential entropy. Neural Comput., 79:93 9, September 5. 5] O Vasicek. A test for normality based on sample entropy. Journal of te Royal Statistical Society Series B, 38:54 59, ] Ven Es Bert. Estimating functionals related to a density by a class of statistics based on spacings. Scandinavian Journal of Statistics, 9:6 7, 99. 9