A Note on Bound for Jensen-Shannon Divergence by Jeffreys

Transcription

1 OPEN ACCESS Conference Proceedngs Paper Entropy A Note on Bound for Jensen-Shannon Dvergence by Jeffreys Takuya Yamano, * Department of Mathematcs and Physcs, Faculty of Scence, Kanagawa Unversty, 946, 6-33 Tsuchya, Hratsuka, Kanagawa 59-93, Japan; E-Mals: yamano@amy.h-ho.ne.jp * Author to whom correspondence should be addressed; E-Mal: yamano@amy.h-ho.ne.jp; Tel.: ; Fax: Receved: 4 September 4 / Accepted: 6 October 4 / Publshed: 3 November 4 Abstract: We present a lower bound on the Jensen-Shannon dvergence by the Jeffrers dvergence when p q s satsfed. In the orgnal Ln's paper [IEEE Trans. Info. Theory, 37, 45 (99)], where the dvergence was ntroduced, the upper bound n terms of the Jeffreys was the quarter of t. In vew of a recent shaper one reported by Crooks, we present a dscusson on upper bounds by transcendental functons of Jeffreys by comparng those values for a bnary dstrbuton. Keywords: Jensen-Shannon dvergence; varatonal dstance; Kullback-Lebler dvergence; Jeffreys dvergence PACS Codes: a, 89.7.Cf. Introducton The Jensen-Shannon dvergence JS(p; q) s a smlarty measure between two probablty dstrbutons p and q. It s presently used n varous dscplnes rangng from nformaton theory to electron orbtal patterns. It was ntroduced n the descrete case as [, ] JS(p; q) p ln p p + q + q ln q p + q In terms of the Kullback-Lebler dvergence D(p; q) = p ln p, t can also be expressed as q JS(p; q) = p + q D (p; ) + p + q D (q; ) ()

2 () In addton, the square root of the JS dvergence becomes a metrc n that t satsfes the trangle nequalty [3]. The JS dvergence can be bounded by other dvergence. For example, the varatonal dstance V(p; q) = p q can be used both for upper and lower bounds: JS(p; q) V(p; q), ([]), JS(p; q) 8 V (p; q) + 5 V4 (p; q) + (3) The second nequalty (lower bound) s explaned n Appendx. The Jeffreys dvergence J(p; q), on the other hand, s an old measure [4] and s defned as J(p; q) D(p; q) + D(q; p). However bounds of JS dvergence n terms of the Jeffreys dvergence s not so much examned snce the ntroducton by Ln []. It was shown that the JS dvergence s upper-bounded by the quarter of the Jeffreys,.e., J(p; q)/4 wthout mentonng whether there exsts the best possble (We present an alternatve proof of the Ln s upper bound n Appendx). Later, Crooks [5] presented a sharper one than the Ln s bound. In addton, t s stated that there s no correspondng lower bound n terms of J(p; q). In ths artcle, however, we show that there exsts n a lmted case.. Results and Dscusson.. A lower bound We here are nterested n obtanng a lower bound n terms of the Jeffreys dvergence. It should practcally be better to have both upper and lower bounds n terms of the same dvergence. However, t was reported that there s no lower bound by Jeffreys [5]. We dsprove ths statement below. By a heurstc method (though t s easly verfed), we fnd the followng nequalty: ln x + x 4 ln( + ln x), (x ) (4) The equalty holds when x =. Substtutng x = p /q and takng an average wth the probablty p, we have p ln p p + q 4 p ln( + ln ( p ) ) q Smlarly for the probablty q, an nequalty q ln p p + q 4 q ln( + ln ( q ) ) p (6) holds. Snce the functon ln( + x) s convex, the Jensen s nequalty tells ln( + x) ln( + x ). Therefore, the rght-hand sdes of the above two nequaltes are bounded from below respectvely by 4 ln( + D(p; q)) and ln( + D(q; p)) 4 Summng the both sdes and applyng a Jensen s nequalty wth the equal weght, we reach a lower bound: JS(p; q) ln( + J(p; q)) 4 (5)

3 3 (8).. Comparson of upper bounds The Ln s upper bound JS(p; q) 4 J(p; q) (9) s based on the nequalty of the arthmetc and geometrc means,.e., (p + q )/ p q. On the other hand, a sharper one derved by Crooks uses the Jensen s nequalty and t s expressed as [5], JS(p; q) ln [ ] + e J(p;q) () It s expressed by a transcendental functon of the Jeffreys dvergence. Here, we pursut yet another smpler one n the followng. Frst, we recognze that the Kullback-Lebler dvergence satsfes D(p; q) p () Ths nequalty s easly confrmed by a functon xln x, ( x ), and thus the nequalty p [ p ln ( p )] holds. Therefore, as n the same lne n [5], we can evaluate the followng way: q q p + q D (p; ) p p + q = p + e lnq p + e D(p;q) () where we have used the Jensen s nequalty for a concave functon at the last nequalty,.e. + e x + e x In a smlar way, we have p + q D (q; ) + e D(q;p) (3) Puttng together the nequaltes () and (3), we have JS(p; q) ( + e D(p;q) ) + ( + e D(q;p) ) (4) Lastly, usng the Jensen s nequalty once more, we have an upper bound JS(p; q) + e J(p;q) (5) Ths bound s smlar to () and s expressed by a transcendental functon of the Jeffreys, however t s smpler. For practcal purposes, t s mportant to examne how the upper bound (5) relates to the prevous ones. In Fgure, we show three curves of the upper bounds n the JS-Jeffreys plane. These q

4 4 curves correspond to nequaltes (9), () and (5). We fnd that the bound (5) s sharper than Ln s, however not than Crook s. In addton, the comparson of these bounds for a bnary dstrbuton s shown n Fgure, n whch p = (t, t) and q = ( t, t), where t. We observe that the bounds gve nearly same value around t=/, whle the dfferences become promnent as t departs towards ether or. Among these bounds we fnd that the Crooks bound s sharper, however t ndcates that there stll seems to exst a sharper one for entre range. Fgure. Comparson of the bound functons n the JS-Jeffreys plane: Ln (red), Crooks (green) and the present (blue). Fgure. Comparson of the bounds for a bnary dstrbuton: Ln (red), Crooks (green) and the present (blue). 4. Conclusons We have derved a lower bound of the Jensen-Shannon dvergence n terms of the Jeffrey s dvergence under a condton of p q : JS(p; q) 4 ln( + J(p; q)). Regardng the upper bound, we found a smpler one that s another transcendental functon of the Jeffrey s dvergence along the lne of the

5 5 prevous study. It s sharper than Ln s, but not than Crooks. Comparsons of alternatve upper bounds may practcally be useful and t s worth seekng for a sharper one. Appendx On lower bound by varatonal dstance The Kullback-Lebler dvergence s lower bounded by the varatonal dstance V, whch s known as the Pnsker nequalty [6, 7]: D(p; q) V + 36 V4 + (Varous bounds on a generalzed Kullback-Lebler dvergence was gven n [8]). Therefore, an equalty D (p; p + q p + q ) = D (q; ) 8 V + 5 V4 + holds, where V (p; p+q p+q ) = V (q; ) = V(p; q)/ s used. Thus, for the JS dvergence, the second nequalty of Eq. (3) follows. An alternatve proof of the Ln s bound []: The Ln s bound JS(p; q) J(p; q) can be derved as follows. Let us put the rato of the two 4 probablty functons as θ = q. Then, we have p Notng ( + θ ) = we have Thus, we can evaluate as Smlarly, we have p ln e (+θ )x e (+θ)x dx p p + q = {ln + p ( + θ ) } dx and applyng the Schwartz nequalty to t, e x dx e θx dx = θ ln( + θ ) ln lnθ = ln + D(p; q) p ln p p + q D(p; q) q ln q p + q D(q; p) Therefore, puttng together the above two nequaltes, we have the Ln s bound: JS(p; q) 4 (D(p; q) + D(q; p)) = J(p; q) 4

6 6 Acknowledgments The author thanks the organzers for arrangng the e-conference. Author Contrbutons The author conducted ths research and wrote the whole part of the paper. Conflcts of Interest The author declares no conflct of nterest. References and Notes. Ln, J. Dvergence measures based on the Shannon entropy. IEEE Trans. Info. Theory 99, 37, Rao, C. R. Dfferental Geometry n Statstcal Inference. IMS-Lecture Notes 987,, Endres D. and Schndeln, J. A new metrc for probablty dstrbutons. IEEE Trans. Inf. Theory, 3 49, Jeffreys, H. An nvarant form for the pror probablty n estmaton problems. Proc. Roy. Soc. Lon., Ser. A 946, 86, Crooks, G.E. Inequaltes between the Jensen Shannon and Jeffreys dvergences. Tech. Note 4 (8) 6. Kraft, O. A note on exponental bounds for bnomal probabltes. Ann. Inst. Statst. Math. 969,, Fedotov, A., Harremoes, P., Topsoe, F. Refnements of Pnsker nequalty. IEEE Trans. Inf. Theory, 3, Yamano, T. Bounds on dvergence n nonextensve statstcal mechancs. J. Indones. Math. Soc. 3, 9, by the authors; lcensee MDPI, Basel, Swtzerland. Ths artcle s an open access artcle dstrbuted under the terms and condtons of the Creatve Commons Attrbuton lcense (