Joural of Iequalities ad Special Fuctios ISSN: 17-4303, URL: http://www.ilirias.com Volume 6 Issue 015, Pages 5-14. SOME INEQUALITIES FOR f-divergences VIA SLATER S INEQUALITY FOR CONVEX FUNCTIONS SILVESTRU SEVER DRAGOMIR Abstract. Some iequalities for f-divergece measures by the use of Slater s iequality for covex fuctios of a real varaible are established. 1. Itroductio Give a covex fuctio f : R + R +, the f-divergece fuctioal D f p, q := f 1.1 was itroduced i Csiszár [3], [4] as a geeralized measure of iformatio, a distace fuctio o the set of probability distributios P. The restrictio here to discrete distributio is oly for coveiece, similar results hold for geeral distributios. As i Csiszár [4], we iterpret udefied expressios by 0f f 0 = lim f t, 0f t 0+ a 0 = lim ε 0+ f a ε 0 0 = 0 f t = a lim, a > 0. t t The followig results were essetially give by Csiszár ad Körer [5]. Theorem 1.1. If f : R + R is covex, the D f p, q is joitly covex i p ad q. The followig lower boud for the f-divergece fuctioal also holds. Theorem 1.. Let f : R + R + be covex. The for every p, q R +, we have the iequality: D f p, q f. 1. 000 Mathematics Subject Classificatio. Primary 94Xxx; Secodary 6D15. Key words ad phrases. Divergece measures, Covex fuctios, Slater s iequality. c 015 Ilirias Publicatios, Prishtië, Kosovë. Submitted April 1, 015. Published Ju 1, 015. 5
6 S. S. DRAGOMIR If f is strictly covex, equality holds i 1. iff p 1 = p =... = p. 1.3 q 1 q q Corollary 1.3. Let f : R + R be covex ad ormalized, i.e., The for ay p, q R + with f 1 = 0. 1.4 = 1.5 we have the iequality D f p, q 0. 1.6 If f is strictly covex, the equality holds i 1.6 iff = for all i {1,..., }. I particular, if p, q are probability vectors, the 1.5 is assured. Corollary 1.3 the shows, for strictly covex ad ormalized f : R + R, D f p, q 0 for all p, q P. 1.7 The equality holds i 1.7 iff p = q. These are distace properties. However, D f is ot a metric: It violates the triagle iequality, ad is asymmetric, i.e, for geeral p, q R +, D f p, q D f q, p. I the examples below we obtai, for suitable choices of the kerel f, some of the best kow distace fuctios D f used i mathematical statistics [15], iformatio theory []-[4] ad sigal processig [13], [19]. Example 1.4. Kullback-Leibler For the f-divergece is f t := t log t, t > 0 1.8 D f p, q = the Kullback-Leibler distace [17]-[18]. log, 1.9 Example 1.5. Helliger Let f t = 1 1 t, t > 0. 1.10 The D f gives the Helliger distace [1] D f p, q = 1, 1.11 which is symmetric. Example 1.6. Reyi For α > 1, let The which is the α-order etropy [3]. f t = t α, t > 0. 1.1 D f p, q = p α i q 1 α i, 1.13
SOME INEQUALITIES FOR f-divergences 7 Example 1.7. χ -distace Let The f t = t 1, t > 0. 1.14 D f p, q = is the χ -distace betwee p ad q. Fially, we have: 1.15 Example 1.8. Variatioal distace. Let f t = t 1, t > 0. The correspodig divergece, called the variatioal distace, is symmetric, D f p, q =. For other examples of divergece measures, see the paper [16] by J. N. Kapur, where further refereces are give.. Slater Type Iequalities Suppose that I is a iterval of real umbers with iterior I ad f : I R is a covex fuctio o I. The f is cotiuous o I ad has fiite left ad right derivatives at each poit of I. Moreover, if x, y I ad x < y, the D f x D + f x D f y D + y, which shows that both D f ad D + f are odecreasig fuctios o I. It is also kow that a covex fuctio must be differetiable except for at most coutably may poits. For a covex fuctio f : I R, the subdifferetial of f deoted by f is the set of all fuctios ϕ : I [, ] such that ϕ I R ad f x f a + x a ϕ a for ay x, a I..1 It is also well kow that if f is covex o I, the f is oempty, D + f, D f f ad if ϕ f, the D f x ϕ x D + f x. for every x I. I particular, ϕ is a odecreasig fuctio. If f is differetiable covex o I, the f = {f }. The followig result is well kow i literature as Slater s iequality. For the origial proof due to Slater, see [5]. For related results, see Chapter I of the book [1] or Chapter of the book []. We shall here follow the presetatio i [6, pp. 19-130] where a slightly more geeral result for Slater s iequality is provided: Lemma.1. Let f : I R be a odecreasig oicreasig covex fuctio o I, x i I, 0 with P = > 0 ad for a give ϕ f assume that ϕ x i 0. The oe has the iequality 1 f x i f p ix i ϕ x i p..3 iϕ x i P
8 S. S. DRAGOMIR Proof. Let us give the proof for the case of odecreasig fuctios oly. I this case ϕ x 0 for ay x I ad x i ϕ x i p I iϕ x i beig a covex combiatio of x i I with the oegative weights ϕ x i p, i {1,..., }. iϕ x i Now, if we use the iequality. we deduce f x f x i x x i ϕ x i for ay x, x i I, i {1,..., }..4 Multiplyig.4 by /P 0 ad summig over i {1,..., }, we deduce f x 1 1 f x i x ϕ x i 1 x i ϕ x i.5 P P P for ay x I. If i.5 we choose x = x i ϕ x i ϕ x i, the we deduce the desired iequality.3. If we would like to drop the assumptio of mootoicity for the fuctio f, the we ca state ad prove i a similar way the followig result see also [6]: Lemma.. Let f : I R be a covex fuctio, x i I, 0 with P > 0 ad ϕ x i 0 for a give ϕ f. If x i ϕ x i p I, iϕ x i the the iequality.3 holds true. Proof. Sice x i ϕ x i ϕ x i I, hece we ca use the iequality.4 ad proceed as i the above Lemma.1. The details are omitted. The followig iequality is well kow i literature as Karamata s iequality, see [1, pp. 98] or [, p. 1]: Lemma.3. Assume that 0 < a a i A <, 0 < b b i B < for each i {1,..., }. The for > 0, = 1, oe has the iequalities K a i b i p ia i p K.6 ib i with K = ab+ AB ab+ ba > 1. Usig Karamata s result, we may poit out the followig reverse of Jese s iequality that may be useful i applicatios.
SOME INEQUALITIES FOR f-divergences 9 Lemma.4.. Let f : [0, R be a mootoic odecreasig covex fuctio. Assume that 0 < r x i R < for each i {1,...},,..., is a probability distributio ad for a give ϕ f cosider rϕr + RϕR Kr, R =. rϕr + Rϕr The we have the iequality fx i f K r, R x i..7 Proof. From Lemma.3 we kow that fx i f x i ϕx i ϕx i..8 If we apply Karamata s iequality for a i = x i, b i = ϕx i, we get successively f p ix i ϕx i p = f x i ϕx i iϕx i p ix i ϕx i x i f K r, R x i, sice, obviously, ϕx i [ϕr, ϕr] beig mootoic odecreasig o [r, R]. The iequality.7 is thus proved. 3. Some Iequalities for f-divergeces The followig result may be stated: Theorem 3.1. Let f : [0, ] R be a differetiable, covex ad ormalized fuctio, i.e. f 1 = 0 ad 0 r 1 R. If there exists a real umber m so that < m f x for ay x r, R, 3.1 the for ay probability distributio p, q P with r R for ay i {1,..., } 3. if r = 0 ad R =, the assumptio 3. is always satisfied, oe has the iequality DΦ p, q m D Φ p, q 0 D f p, q f m D f p, q m D f p, q m, 3.3 where Φ x := xf x, Φ x := x 1 f x for x [0, ] ad D f p, q m. Proof. Cosider the auxiliary fuctio f m x = f x m x 1, x [0, ]. Sice f m x = f x m, x r, R, it follows that f m is differetiable, covex ad mootoic odecreasig o r, R, ad we may apply Lemma.1 to get f m f m q i f m q. 3.4 i f m
10 S. S. DRAGOMIR It is easy to see that f m = ad f m [f = = ] m 1 = D f p, q [ ] f m f m = D Φ p, q m where Φ x has bee defied above. Also, oe may observe that f m [ ] = f m = D f p, q m ad f m DΦ p, q m D f p, q m DΦ p, q m = f D f p, q m DΦ p, q m = f D f p, q m DΦ p, q m = f D f p, q m which gives, by 3.4, the desired iequality 3.3. DΦ p, q m m D f p, q m 1 m DΦ p, q D f p, q D f p, q m m D Φ p, q D f p, q m, If oe would like to drop the assumptio of lower boudedess for the derivative f see 3.1, oe may eed to impose aother coditio as described i the followig theorem: Theorem 3.. Let f : [0, R be a differetiable, covex ad ormalized fuctio ad 0 r 1 R. As above, cosider Φ x = xf x ad assume that for two probabilities p ad q satisfyig 3. oe has D f p, q 0 ad D Φ p, q 0. 3.5 D f p, q The oe has the iequality DΦ p, q 0 D f p, q f. 3.6 D f p, q The proof follows i a similar way as the oe i Theorem 3.1 by utilizig Lemma.. We omit the details. Now we ca poit out aother result for f-divergeces whe bouds for the likelihood ratio p q are available: Theorem 3.3. Let f : [0, ] R be a differetiable covex ad ormalized fuctio ad 0 r 1 R ad let Kr, R be as stated i Lemma.4. If there exists a real umber m so that < m f x for ay x r, R
SOME INEQUALITIES FOR f-divergences 11 the for all probability distributios p, q P oe has the iequality satisfyig r R for each i {1,..., }, D f p, q fk r, R mk r, R 1. Proof. As i Theorem 3.1, the fuctio f m x = fx mx 1 is differetiable, covex ad mootoic odecreasig o r, R. If we apply Lemma.4 we get f m f m K r, R which completes the proof. = f K r, R m K r, R 1, 4. Applicatios for Particular Divergeces We cosider the Kullback-Leibler distace KL p, q = log that is the f-divergece for the covex fuctio f : 0, R, f t = t log t. If we take the covex fuctio f t = log t, the the correspodig f-divergece is ] qi D f p, q := f = [ log = log = KL q, p for all probability distributios p, q P. For the fuctio f t = log t we have Φ t := tf t = 1 ad Φ t := t 1 f t = 1 t, t > 0. t Now for 0 r 1 R < ad m = 1 R we have m f t = 1 t for ay t r, R ad the coditio 3.1 is satisfied. We also have where D Φ p, q = 1 ad D f p, q = D χ p, q = = q i + 1 1 = 1 D χ q, p p i 1 is the χ -distace betwee p ad q. We also have 1 q D Φ p, q = q i qi i = 1 = D χ q, p.
1 S. S. DRAGOMIR Therefore, for ay probability distributio p, q P with r R for ay i {1,..., } we have by 3.3 the iequality 1 + 1 R 0 KL q, p l 1 D χ q, p + 1 R + 1 R D χ q, p 1 D χ q, p + 1, R which is equivalet to R Dχ q, p + 1 1 D χ q, p 0 KL q, p l R 1 R D χ q, p + 1 1. 4.1 Observe that D Φ p, q D f p, q = 1 1 D χ q, p = 1 D χ q, p + 1 > 0, the by the iequality 3.6 we have 0 KL q, p l D χ q, p + 1 4. for ay p, q P. We otice that the iequality 4. ca be obtaied from 4.1 by lettig R. Now, for the fuctio f t = t log t, we have The ad D Φ p, q = Φ t := tf t = t log t + t. log + p i = = KL p, q + 1 D f p, q = log + log p i + 1 = 1 KL q, p. The, if we take p, q P with 1 > KL q, p, by utilizig the iequality 3.6 we get 1 + KL p, q 1 + KL p, q 0 KL p, q 1 KL q, p l. 4.3 1 KL q, p For α > 1 cosider α-order etropy D α p, q := p α i q 1 α i, which is a f-divergece for the covex fuctio f t = t α. We have rf r + Rf Kr, R = R rf R + Rf r = r α + R α. r 1 R α 1 + R 1 r α 1 We have f t = αt α 1 αr α 1.
SOME INEQUALITIES FOR f-divergences 13 If we apply Theorem 3.3, the for all probability distributios p, q P satisfyig 0 < r R < for each i {1,..., }, we have the iequality r α + R α α D α p, q 4.4 r 1 R α 1 + R 1 r α 1 [ αr α 1 r α + R α 1]. r 1 R α 1 + R 1 r α 1 Refereces [1] R. Bera, Miimum Helliger distace estimates for parametric models, A. Statist., 5 1977, 445-463. [] I. Burbea ad C. R. Rao, O the covexity of some divergece measures based o etropy fuctios, IEEE Trasactios o Iformatio Theory, 8 198, 489-495. [3] I. Csiszár, Iformatio measures: A critical survey, Tras. 7th Prague Cof. o Ifo. Th., Statist. Decis. Fuct., Radom Processes ad 8th Europea Meetig of Statist., Volume B, Academia Prague, 1978, 73-86. [4] I. Csiszár, Iformatio-type measures of differece of probability fuctios ad idirect observatios, Studia Sci. Math. Hugar, 1967, 99-318. [5] I. Csiszár ad J. Körer, Iformatio Theory: Codig Theorem for Discrete Memory-less Systems, Academic Press, New York, 1981. Aual Coferece of the Idia Society of Agricultural Statistics, 1984, 1-44. [6] S. S. Dragomir, A survey o Cauchy-Buiakowski-Schwarz s type discrete iequality, J. Ieq. Pure & Appl. Math., 4003, Issue 3, Article 63. [7] S. S. Dragomir ad N. M. Ioescu, Some coverse of Jese s iequality ad applicatios, Aal. Num. Theor. Approx., 3 1994, 71-78. [8] S. S. Dragomir ad C. J. Goh, A couterpart of Jese s discrete iequality for differetiable covex mapgs ad applicatios i iformatio theory, Math. Comput. Modellig, 4 1996, 1-11. [9] S. S. Dragomir ad C. J. Goh, Some couterpart iequalities i for a fuctioal associated with Jese s iequality, J. Ieq. & Appl., 1 1997, 311-35. [10] S. S. Dragomir ad C. J. Goh, Some bouds o etropy measures i iformatio theory, Appl. Math. Lett., 10 1997, 3-8. [11] S. S. Dragomir ad C. J. Goh, A couterpart of Jese s cotiuous iequality ad applicatios i iformatio theory, A. Ştiiţ. Uiv. Al. I. Cuza Iaşi. Mat. N.S. 47 001, o., 39 6 00. [1] S. S. Dragomir, J. Šude ad M. Scholz, Some upper bouds for relative etropy ad applicatios, Comput. Math. Appl. 39 000, o. 9-10, 91 100. [13] B. R. Friede, Image ehacemet ad restoratio, Picture Processig ad Digital Filterig T.S. Huag, Editor, Spriger-Verlag, Berli, 1975. [14] R. G. Gallager, Iformatio Theory ad Reliable Commuicatios, J. Wiley, New York, 1968. [15] J. H. Justice editor, Maximum Etropy ad Bayssia Methods i Applied Statistics, Cambridge Uiversity Press, Cambridge, 1986. [16] J. N. Kapur, A comparative assessmet of various measures of directed divergece, Advaces i Maagemet Studies, 3 1984, No. 1, 1-16. [17] S. KULLBACK, Iformatio Theory ad Statistics, J. Wiley, New York, 1959. [18] S. Kullback ad R. A. Leibler, O iformatio ad sufficiecy, Aals Math. Statist., 1951, 79-86. [19] R. M. Leahy ad C. E. Goutis, A optimal techique for costrait-based image restoratio ad mesuratio, IEEE Tras. o Acoustics, Speech ad Sigal Processig, 34 1986, 169-164. [0] M. Matić, Jese s Iequality ad Applicatios i Iformatio Theory i Croatia, Ph.D. Thesis, Uiv. of Zagreb, 1999.
14 S. S. DRAGOMIR [1] D. S. Mitriović, J. E. Pečarić ad A. M. Fik, Classical ad New Iequalities i Aalysis, Kluwer Academic, Dordrecht, 1993. [] J. E. Pečarić, F. Proscha ad Y. L. Tog, Covex Fuctios, Partial Orderigs ad Statistical Applicatios, Academic Press Ic., 1991. [3] A. Reyi, O measures of etropy ad iformatio, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, Uiversity of Califoria Press, Berkeley, 1961. [4] C. E. Shao, A mathematical theory of commuicatio, Bull. Sept. Tech. J., 7 1948, 370-43 ad 63-656. [5] M. S. Slater, A compaio iequality to Jese s iequality, J. Approx. Theory, 3 1984, 160-166. Mathematics, College of Egieerig & Sciece, Victoria Uiversity, PO Box 1448, Melboure City, MC 8001, Australia. E-mail address: sever.dragomir@vu.edu.au