q i i=1 p i ln p i Another measure, which proves a useful benchmark in our analysis, is the chi squared divergence of p, q, which is defined by

Transcription

1 CSISZÁR f DIVERGENCE, OSTROWSKI S INEQUALITY AND MUTUAL INFORMATION S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE Abstact. The Ostowski integal inequality fo an absolutely continuous function is used to povide a simple appoximation to Csiszá s f divegence measue fo the diffeence between two pobability distibutions defined on a finite set. Concete examples ae given and some applications made to mutual infomation.. Intoduction The diffeence between two pobability measues p, q on a set A = α i i n} is commonly measued in a vaiety of ways. Denote by p i, q i the associated point pobabilities fo the event α i A. To avoid tiviality we assume that p i + q i > 0 fo each i. The vaiational distance (l distance) and infomation divegence (Kullback Leible divegence) between the distibutions p and q ae defined espectively by V (p, q) := p i q i, D(p, q) := p i ln p i q i. Anothe measue, which poves a useful benchmak in ou analysis, is the chi squaed divegence of p, q, which is defined by p i (p i q i ) D χ (p, q) := =. q i q i The last two measues ae unfotunately infinite if p i > 0 but q i = 0 fo some i. This complication is obviated in the tiangula discimination between p and q, which is defined as in 0] by (p, q) := p i q i p i + q i. A genealization of this measue, paameteized by a natual numbe υ, is p i q i υ υ (p, q) := (p i + q i ) υ, 99 Mathematics Subject Classification. 9A7, 6D5. Key wods and phases. Csiszá f divegence, Ostowski s inequality, divegence measue, mutual infomation.

2 S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE which we efe to as tiangula discimination of ode υ (see 0]). Anothe common choice is the Hellinge discimination h (p, q) := ( p i q i ). Fo applications it is impotant to know how these divegences compae with one anothe. The basic elations between V, and h ae and V (p, q) (p, q) V (p, q) h (p, q) (p, q) h (p, q) (see LeCam 8] and Dacunha Castelle 5]). Fom these we may deduce that 8 V (p, q) h (p, q) V (p, q). The coefficients in these inequalities ae best possible (cf. 0]). The fist half of this esult has been impoved by Kaft 7], who showed that 8 V (p, q) h (p, q) ( h (p, q) We note also the impotant inequality D(p, q) ln ( h (p, q) ) (see Dacunha Castelle 5]). It follows fom this that D(p, q) h (p, q). Again the coefficient is best possible (see 0]). The key to unity in this divesity is that all the discepancy measues consideed above ae paticula instances of Csiszá f divegences. If f : 0, ) R is convex, the Csiszá f-divegence between p and q is defined by (.) I f (p, q) := q i f (p i /q i ) ). (see Csiszá ] ]). Thus the family (f s ) s of functions with f s (u) = u s (u + ) s gives ise to vaiational distance when s =, tiangula discimination when s = and tiangula discimination of ode υ when s = υ (see 0]). The choice f(u) = ( u ) gives ise to Hellinge discimination and f(u) = u ln u to Kullback- Leible divegence. The chi squaed divegence is given by f(u) = (u ). Fo all of the above choices f() = 0, so that I f (p, p) = 0. The convexity of f then ensues that I f (p, q) is nonnegative. In Section we deive, by the use of Ostowski s integal inequality fo absolutely continuous mappings with essentially bounded fist deivative, an appoximation fo the Csiszá f divegence in tems of an integal mean. With many concete examples this povides vey simple appoximations. Section 3 consides some of the examples noted above and Section the case when each pai p i, q i ae vey close. Finally, in Section 5, we look at applications to mutual infomation.

3 3 It needs to be stessed that as these estimates lose most of the detailed infomation involved in the values p i, q i, the appoximations, while vey simple, can also be vey cude.. An inequality fo Csiszá f divegence In some applications it is convenient to make use of definition (.) fo functions f : 0, ) R which ae continuous but not necessaily convex. An illustative example is given in Section 3. Accodingly ou main esult, Theoem below, does not assume convexity. We assume in what follows that thee exist eal numbes, R with 0 < p i /q i R < fo all i,..., n}. Note that if >, then p i > q i fo each i, which gives = i p i > i q i =, a contadiction. Hence. A simila agument gives R. Futhe suppose that the estiction of f to the compact inteval, R] is absolutely continuous. We deive an appoximation to the Csiszá f divegence in tems of the integal mean of f ove, R]. We shall show in Theoem below that if p and q ae close in the sense that is small, then the integal mean f(t)dt appoximates the Csiszá f divegence to fist ode. We make use of Ostowski s integal inequality, which states the following. See 6] fo a shot poof and some applications to numeical integation and special means. Theoem A. Assume that g : a, b] R is absolutely continuous with g L a, b], that is, that g := ess sup g (t) <. Then fo all x a, b]. g(x) b a b a t a,b] ( ) g(t)dt x a+b + g (b a) b a A futhe key esult is due to Diaz and Metcalf (see 9, p. 6]). Theoem B. Suppose a k ( 0) and b k (k =,..., n) ae eal numbes satisfying m b k /a k M. Then b k + mm a k (M + m) a k b k. k= k= k= Equality holds if and only if fo each k eithe b k = ma k o b k = Ma k. We shall make use of a slight extension of this.

4 S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE Poposition. Suppose the conditions of the Diaz Metcalf esult hold and t k > 0 fo k =,..., n. Then t k bk + mm t k a k (M + m) t k a k b k. k= k= Equality holds if and only if fo each k eithe b k = ma k o b k = Ma k. Poof. We have fo k =,,..., n that k= (b k /a k m)(m b k /a k )t k a k 0. The desied esult follows on summation ove k. Theoem. Assume that f :, R] R is absolutely continuous on, R] and f L, R]. Then I f (p, q) (.) f(t)dt ( ) }] + R + f () D χ (p, q) + () () f. Poof. By Ostowski s integal inequality, we have ( ) f pi ( f(t)dt q i pi + q i R+ ) () f fo each i,..., n}. We may multiply by q i, sum the esultant inequalities and use the genealized tiangle inequality to obtain Since I f (p, q) ( q i f pi q i + () f(t)dt ) ( pi q i R + ) = q i f(t)dt ) ] ( pi q i R + q i p i q i (R + ) + () f. ( R + p ( ) i R + + q i ( R + = D χ (p, q) + ), )

5 5 this yields the fist inequality in (.). Fo the second, set b k = p k /q k and a k = q k /p k (k =,..., n). Then a k /b k = p k /q k, R] (k,..., n}). On applying Poposition fo t k = p k (k =,..., n), we get k= ( ) pk p k + R q k k= ( ) qk p k ( + R) p k k= pk qk p k, q k p k o equivalently Thus k= p k q k + R R +. D χ (p, q) + R = ( )(R ) and so and the theoem is poved. + D () χ (p, q) + ] (R + ) Coollay. Let f satisfy the conditions of Theoem. If ε > 0 and then 0 ɛ/ f, I f (p, q) f(t)dt ɛ. Theoem can be efomulated to emphasize the appoximation aspect. Coollay. Let f : 0, ] R be absolutely continuous with f L 0, ]. If η (0, ) and p(η), q(η) ae such that p i (η) q i (η) η fo all i,..., n}, then I f (p(η), q(η)) = η +η η f(t)dt + R f (p, q, η) and the emainde R f (p, q, η) satisfies R f (p, q, η) η + ] f η D χ (p(η), q(η)) f η. This follows by Theoem with the choices R = + η and = η (η (0, )).

6 6 S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE 3. Paticula cases Fo Kullback Leible distance, we take f(u) = u ln u. With this choice we have f = ln(er) and f(t)dt = R ln R ln ( R )] = ( ln ) /(R (R ) ) (R ) ( ) ( ) e = I ( R, )], whee the identic mean I(a, b) fo positive aguments is given by a if b = a I(a, b) := ( ) /(b a) b b e a if b a. a The conclusion of Theoem eads R + D(p, q) ln I ( R, )] (3.) ( + R + () D χ (p, q) + ) ]] () ln(er) () ln(er). If we take the concave map f : (0, ) R given by f(u) = ln u, then we have I f (p, q) = q i ln p i = D(q, p). q i k= With this choice f = / and the identic mean eappeas though Theoem povides (3.) D(q, p) ln I(, R)] + () D χ (p, q) + ( ) R. f(t)dt = ln I(, R)]. ( R + Fo Hellinge discimination f(u) = ( u ) /, so f u (u) = u, f (u) = u ) ]] (R )

7 7 fo u (0, ) and f = Also sup f (u) = f (R) = u,r] f(t)dt = R + R R. 3 R + R + + R +, and inequality (.) becomes R + h (p, q) 3 R + R R ] ( ) }] ] + R + R () D χ (p, q) + () R R ()( R ). Fo vaiational distance, f(u) = u, which is absolutely continuous on, R]. We have f if u (, ) (u) := if u (, R), so that f = sup f (t) =. t,r] Futhe f(t)dt = Theoem povides V (p, q) + () = = () ( u)du + ] ( ) (R ) + () ( + R + ( + R + (u )du ) ]. ] ) ( ) }] R + D χ (p, q) + () (). Ou final example elates to tiangula discimination, which aises with f(u) = (u ) /(u + ). We have f(u) = u + u u +, f (u) = (u + )

8 8 S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE fo u 0, ), so that f = sup f (u) = f (R )(R + ) (R) = u,r] (R + ). Also and Theoem povides (p, q) R + + ln + () f(u)du = R + + ln ( ) ] /(R ) R ( R + D χ (p, q) + ()(R )(R + 3) (R + ). ( ) /(R ) R ) }]. Some numeical examples ()(R )(R + 3) (R + ) One situation of pactical inteest is whee p i and q i ae close, so that we have p i = p i (ε), q i = q i (ε) and p i (ε) (.) q i (ε) ε ε (0, ) fo all i,..., n}. With R = ε + and = ε, we obtain fom (3.) that D(p(ε), q(ε)) ( I ln ( + ε), ( ε) )] ε + ε ] D χ (p(ε), q(ε)) ln e( + ε)] ε ln e( + ε)]. Consequently if p(ε), q(ε) ae in the sense of (.), we can appoximate )] the Kullback-Leible distance D(p(ε), q(ε)) by (/) ln I (( + ε), ( ε) and the eo of the appoximation is less than E(ε) := ε lne( + ε)]. Fom (3.), we deive ] D(q(ε), p(ε)) ln ε + ε ] I( ε, + ε) ε D χ (p(ε), q(ε)) ε ε fo ε (0, ). Consequently fo p(ε), q(ε) satisfying (.), we can appoximate the Kullback Leible distance D(p(ε), q(ε)) by ln I ( ε, + ε) ] and the eo of the appoximation is less than ε/( ε) fo ε (0, ).

9 9 5. Application to mutual infomation We conside mutual infomation, which is a measue of the amount of infomation that one andom vaiable povides about anothe. It is the eduction of uncetainty about one vaiable due to knowledge of the othe (see, fo example, ]). Definition. Conside two discete valued andom vaiables X and Y with a joint pobability mass function t(x, y) and maginal pobability mass functions p(x) (x X ) and q(y) (y Y). The mutual infomation is the elative entopy between the joint distibution and the poduct distibution, that is, I(X; Y ) = ] t(x, y) t(x, y) ln = D (t(x, y), p(x)q(y)), p(x)q(y) x X y Y whee as befoe D(, ) denotes Kullback Leible distance. We assume that (5.) s t(x, y) S fo all (x, y) X Y. p(x)q(y) Much as with, R we have s S. We also may conside mutual infomation in a chi squaed sense, that is, t (x, y) I χ (X; Y ) := p(x)q(y). (x,y) X Y Inequality (3.) yields the following poposition. Poposition. If t, p and q satisfy (5.), then I(X; Y ) s + S ln I ( s, S )] ( + s + S (S s) I χ (X; Y ) + ) ]] (S s) lnes] (S s) ln(es). Remak. The condition t(x, y) = p(x)p(y) fo all (x, y) X Y means that the andom vaiables X and Y ae independent. We may efe to them as quasi independent if t(x, y) p(x)q(y) δ (δ (0, )) fo all (x, y) X Y. When this occus, we can appoximate the mutual infomation I(X; Y ) by ( + δ) ln( + δ) ( δ) ln( δ) δ with an eo less than E(δ) fo t (0, ). Refeences ] (δ (0, )) ] T. M. Cove and J. A. Thomas, Elements of infomation theoy, Wiley, 99. ] I. Csiszá, A note on Jensen s inequality, Studia Sci. Math. Hung., (966) ] I. Csiszá, Infomation type measues of diffeences of pobability distibutions and indiect obsevation, Studia Sci. Math. Hung., (967)

10 0 S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE ] I. Csiszá, On topological popeties of f divegence, Studia Sci. Math. Hung., (967) ] D. Dacunha Castelle, Ecole d été de pobabilités de Saint Fleu, VII-977, Belin, Heidelbeg, New Yok: Spinge, ] S. S. Dagomi and S. Wang, Application of Ostowski s inequality to the estimation of eo bounds fo some special means and fo some numeical quadatue ules, Appl. Math. Lett., (998), ] C. Kaft, Some conditions of consistency and unifom consistency of statistical pocedue, Univ. of Califonia Publ. in Statistics, Ph.D. Thesis, (955), 5. 8] L. LeCam, Asymptotic methods in statistical decision theoy, New Yok: Spinge, ] D. S. Mitinović, Analytic inequalities, Spinge Velag, ] F. Topsoe, Some inequalities fo infomation divegence and elated measues of discimination, RGMIA Res. Rep. Coll. (999) No., Depatment of Compute and Mathematical Sciences, Victoia Univesity of Technology, PO Box 8, MCMC, Melboune, Victoia 800, Austalia addess: seve@matilda.vu.edu.au Royal Austalian Ai Foce, ARDU, P.O. Box 500, Salisbuy SA 508, Austalia addess: vgluscev@maths.adelaide.edu.au Applied Mathematics Depatment, The Univesity of Adelaide, Adelaide, SA 5005, Austalia addess: cpeace@maths.adelaide.edu.au