Journal of Inequalities in Pure and Applied Mathematics

Transcription

1 Journal of Inequalities in Pure and Applied Mathematics Volume 7, Issue 2, Article 59, 2006 ENTROPY LOWER BOUNDS RELATED TO A PROBLEM OF UNIVERSAL CODING AND PREDICTION FLEMMING TOPSØE UNIVERSITY OF COPENHAGEN DEPARTMENT OF MATHEMATICS DENMARK topsoe@mathud URL: topsoe Received 8 August, 2004; accepted 20 March, 2006 Communicated by F Hansen ABSTRACT Second order lower bounds for the entropy function expressed in terms of the index of coincidence are derived Equivalently, these bounds involve entropy and Rényi entropy of order 2 The constants found either explicitly or implicitly are best possible in a natural sense The inequalities developed originated with certain problems in universal prediction and coding which are briefly discussed Key words and phrases: Entropy, Index of coincidence, Rényi entropy, Measure of roughness, Universal coding, Universal prediction 2000 Mathematics Subject Classification 94A5, 94A7 BACKGROUND, INTRODUCTION We study probability distributions over the natural numbers The set of all such distributions is denoted M +N) and the set of P M +N) which are supported by {, 2,, n} is denoted M +n) We use U to denote a generic uniform distribution over a -element set, and if also U +, U +2, are considered, it is assumed that the supports are increasing By H and by IC we denote, respectively entropy and index of coincidence, ie HP ) = ICP ) = p ln p, = p 2 = ISSN electronic): c 2006 Victoria University All rights reserved 53-04

2 2 FLEMMING TOPSØE Results involving index of coincidence may be reformulated in terms of Rényi entropy of order 2 H 2 ) as H 2 P ) = ln ICP ) In Harremoës and Topsøe [5] the exact range of the map P ICP ), HP )) with P varying over either M +n) or M +N) was determined Earlier related wor includes Kovalevsij [7], Tebbe and Dwyer [9], Ben-Bassat [], Vajda and Vaše [3], Golic [4] and Feder and Merhav [2] The ranges in question, termed IC/H-diagrams, were denoted, respectively n : = {ICP ), HP )) P M +N)}, n = {ICP ), HP )) P M +n)} By Jensen s inequality we find that HP ) ln ICP ), thus the logarithmic curve t t, ln t); 0 < t is a lower bounding curve for the IC/H-diagrams The points Q =, ln ) ; all lie on this curve They correspond to the uniform distributions: ICU ), HU )) =, ln ) No other points in the diagram lie on the logarithmic curve, in fact, Q ; are extremal points of in the sense that the convex hull they determine contains No smaller set has this property H ln n Q n ln +) Q + ln Q n 0 IC 0 n + Q Figure : The restricted IC/H-diagram n, n = 5) Figure, adapted from [5], illustrates the situation for the restricted diagrams n The ey result of [5] states that n is the bounded region determinated by a certain Jordan curve determined by n smooth arcs, viz the upper arc connecting Q and Q n and then n lower arcs connecting Q n with Q n, Q n with Q n 2 etc until Q 2 which is connected with Q In [5], see also [], the main result was used to develop concrete upper bounds for the entropy function Our concern here will be lower bounds The study depends crucially on the nature of the lower arcs In [5] these arcs were identified Indeed, the arc connecting Q + with Q is the curve which may be parametrized as follows: s ϕ s)u + + s U ) J Inequal Pure and Appl Math, 72) Art 59,

3 ENTROPY LOWER BOUNDS 3 with s running through the unit interval and with ϕ denoting the IC/H-map given by ϕp ) = ICP ), HP )); P M+N) The distributions in M+N) fall in IC-complexity classes The th class consists of all P M+N) for which ICU + ) < ICP ) ICU ) or, equivalently, for which < + ICP ) In order to determine good lower bounds for the entropy of a distribution P, one first determines the IC-complexity class of P One then determines that value of s ]0, ] for which ICP s ) = ICP ) with P s = s)u + + s U Then HP ) HP s ) is the theoretically best lower bound of HP ) in terms of ICP ) In order to write the sought lower bounds for HP ) in a convenient form, we introduce the th relative measure of roughness by ) MR P ) = ICP ) ICU +) = + ) ICP ) ) ICU ) ICU + ) + This definition applies to any P M +N) but really, only distributions of IC-complexity class will be of relevance to us Clearly, MR U + ) = 0, MR U ) = and for any distribution of IC-complexity class, 0 MR P ) For a distribution on the lower arc connecting Q + with Q one finds that 2) MR s)u + + s U ) = s 2 In view of the above, it follows that for any distribution P of IC-complexity class, the theoretically best lower bound for HP ) in terms of ICP ) is given by the inequality 3) HP ) H x)u + + x U ), where x is determined so that P and x)u + + x U have the same index of coincidence, ie 4) x 2 = MR P ) By writing out the right-hand-side of 3) we then obtain the best lower bound of the type discussed Doing so one obtains a quantity of mixed type, involving logarithmic and rational functions It is desirable to search for structurally simpler bounds, getting rid of logarithmic terms The simplest and possibly most useful bound of this type is the linear bound 5) HP ) HU )MR P ) + HU + ) MR P )), which expresses the fact mentioned above regarding the extremal position of the points Q in relation to the set Note that 5) is the best linear lower bound as equality holds for P = U + as well as for P = U Another comment is that though 5) was developed with a view to distributions of IC-complexity class, the inequality holds for all P M +N) but is weaer than the trivial bound H ln IC for distributions of other IC-complexity classes) Writing 5) directly in terms of ICP ) we obtain the inequalities 6) HP ) α β ICP ); with α and β given via the constants 7) u = ln + ) = ln + ) by α = ln + ) + u, β = + )u J Inequal Pure and Appl Math, 72) Art 59,

4 4 FLEMMING TOPSØE Note that the u In the present paper we shall develop sharper inequalities than those above by adding a second order term More precisely, for, we denote by γ the largest constant such that the inequality 8) H ln MR + ln + ) MR ) + γ 2 MR MR ) holds for all P M+N) Here, H = HP ) and MR = MR P ) Expressed directly in terms of IC = ICP ), 8) states that 9) H α β IC + γ + )2 IC ) ) 2 + IC for P M +N) The basic results of our paper may be summarized as follows: The constants γ ) increase with γ = ln and with limit value γ More substance will be given to this result by developing rather narrow bounds for the γ s in terms of γ and by other means The refined second order inequalities are here presented in their own right However, we shall indicate in the next section how the author was led to consider inequalities of this type This is related to problems of universal coding and prediction The reader who is not interested in these problems can pass directly to Section 3 2 A PROBLEM OF UNIVERSAL CODING AND PREDICTION Let A = {a,, a n } be a finite alphabet The models we shall consider are defined in terms of a subset P of M+A) and a decomposition θ = {A,, A } of A representing partial information A predictor θ-predictor) is a map P : A [0, ] such that, for each i, the restriction P A i is a distribution in M+A i ) The predictor P is induced by P 0 M+A), and we write P 0 P, if, for all x A, P A i = P 0 ) Ai, the conditional probability of P 0 given A i When we thin of a predictor P in relation to the model P, we say that P is a universal predictor since the model may contain many distributions) and we measure its performance by the guaranteed expected redundancy given θ: 2) RP ) = sup D θ P P ) P P Here, expected redundancy or divergence) given θ is defined by 22) D θ P P ) = i P A i )DP Ai P A i ) with D ) denoting standard Kullbac-Leibler divergence By R min we denote the quantity 23) R min = inf P RP ) and we say that P is the optimal universal predictor for P given θ or just the optimal predictor) if RP ) = R min and P is the only predictor with this property Concrete algebraic bounds for the u, which, via 6), may be used to obtain concrete lower bounds for HP ), are given by 2 2+ u This follows directly from 6) of [2] as u = λ ) in the notation of that manuscript) J Inequal Pure and Appl Math, 72) Art 59,

5 ENTROPY LOWER BOUNDS 5 In parallel to predictors we consider quantities related to coding A θ-coding strategy is a map κ : A [0, ] such that, for each i, Kraft s equality 24) exp κ x)) = x A i holds Note that there is a natural one-to-one correspondence, notationally written P κ, between predictors and coding strategies which is given by the relations 25) κ = ln P and P = exp κ ) When P κ, we may apply the lining identity 26) D θ P P ) = κ, P H θ P ) which is often useful for practical calculations Here, H θ P ) = i P A i)hp Ai ) is standard conditional entropy and, P denotes expectation wrt P From Harremoës and Topsøe [6] we borrow the following result: Theorem 2 Kuhn-Tucer criterion) Assume that A,, A m are distributions in P, that P 0 = ν m α νa ν is a convex combination of the A ν s with positive weights which induces the predictor P, that, for some finite constant R, D θ A ν P ) = R for all ν m and, finally, that RP ) R Then P is the optimal predictor and R min = R Furthermore, the convex set P given by 27) P = {P M +A) D θ P P ) R} can be characterized as the largest model with P as optimal predictor and R min = R This result is applicable in a great variety of cases For indications of the proof, see [6] and Section 43 of [0] The distributions A ν of the result are referred to as anchors and the model P as the maximal model The concrete instances of Theorem 2 which we shall now discuss have a certain philosophical flavour which is related to the following general and loosely formulated question: If we thin of Nature or God as deciding which distribution P P to choose as the true distribution, and if we assume that the model we consider is really basic and does not lend itself to further fragmentation, one may as if any other choice than a uniform distribution is really feasible In other words, one may maintain the view that God only nows the uniform distribution Whether or not the above view can be formulated more precisely and meaningfully, say within physics, is not that clear Anyhow, motivated by this ind of thining, we shall loo at some models involving only uniform distributions For models based on large alphabets, the technicalities become quite involved and highly combinatorial Here we present models with A 0 = {0, } consisting of the two binary digits as the source alphabet The three uniform distributions pertaining to A 0 are denoted U 0 and U for the two deterministic distributions and U 0 for the uniform distribution over A 0 For an integer t 2 consider the model P = {U0, t U, t U0} t with exponentiation indicating product measures We are interested in universal coding or, equivalently, universal prediction of Bernoulli trials x t = x x 2 x t A t 0 from this model, assuming that partial information corresponding to observation of x s = x x s for a fixed s is available This model is of interest for any integers s and t with 0 s < t However, in order to further simplify, we assume that s = t The model we arrive at is then closely related to the classical problem of succession going bac to Laplace, cf Feller [3] For a modern treatment, see Krichevsy [8] The former source is just a short proceedings contribution For various reasons, documentation in the form of comprehensive publications is not yet available However, the second source which reveals the character of the simple proof, may be helpful J Inequal Pure and Appl Math, 72) Art 59,

6 6 FLEMMING TOPSØE Common sense has it that the optimal coding strategy and the optimal predictor, respectively κ and P, are given by expressions of the form κ if x t = 0 00 or 28) κ x t ) = κ 2 if x t = 0 0 or 0 ln 2 otherwise and p if x t = 0 00 or 29) P x t ) = p 2 if x t = 0 0 or 0 otherwise 2 with p = exp κ ) and p 2 = exp κ 2 ) Note that p i is the weight P assigns to the occurrence of i binary digits in x t in case only one binary digit occurs in x s Clearly, if both binary digits occur in x s, it is sensible to predict the following binary digit to be a 0 or a with equal weights as also shown in 29) With t s as superscript to indicate partial information we find from 26) that D t s U t 0 P ) = D t s U t P ) = κ, D t s U t 0 P ) = 2 s κ + κ 2 ln 4) With an eye to Theorem 2 we equate these numbers and find that 20) 2 s )κ = κ 2 ln 4 Expressed in terms of p and p 2, we have p = p 2 and 2) 4p 2 = p 2 ) 2s Note that 2) determines p 2 [0, ] uniquely for any s It is a simple matter to chec that the conditions of Theorem 2 are fulfilled with U t 0, U t and U t 0 as anchors) With reference to the discussion above, we have then obtained the following result: Theorem 22 The optimal predictor for prediction of x t with t = s+, given x s = x x s for the Bernoulli model P = {U t 0, U t, U t 0} is given by 29) with p = p 2 and p 2 determined by 2) Furthermore, for this model, R min = ln p = κ and the maximal model, P, consists of all Q M + A t 0) for which D t s Q P ) κ It is natural to as about the type of distributions included in the maximal model P of Theorem 22 In particular, we as, sticing to the framewor of a Bernoulli model, which product distributions are included? Applying 26), this is in principle easy to answer We shall only comment on the three cases s =, 2, 3 For s = or s = 2 one finds that the inequality D t s P t P ) R min is equivalent to the inequality H ln 4 IC) which, by 6) for =, is nown to hold for any distribution Accordingly, in these cases, P contains every product distribution For the case s = 3 the situation is different Then, as the reader can easily chec, the crucial inequality D t s P t P ) R min is equivalent to the following inequality with H = HP ), IC = ICP )): 22) H IC) ln 4 + ln 2 + 3κ ) IC )) 2 This is a second order lower bound of the entropy function of the type discussed in Section In fact this is the way we were led to consider such inequalities As stated in Section J Inequal Pure and Appl Math, 72) Art 59,

7 ENTROPY LOWER BOUNDS 7 0,025 0,02 0,05 0,0 0, ,2 0,4 0,6 0,8-0,005 u Figure 2: A plot of R min D 4 3 P t P ) as function of p with P = p, q) and proved rigorously in Lemma 35 of Section 3, the largest constant which can be inserted in place of the constant ln 2 + κ 0438 in 22), if we want the inequality to hold for all P M +A 0 ), is 2γ = 2ln 4 ) Thus 22) does not hold for all P M +A 0 ) In fact, considering the difference between the left hand and the right hand side of 22), shown in Figure 2, we realize that when s = 3, P 4 with P = p, q) belongs to the maximal model if and only if either P = U 0 or else one of the probabilities p or q is smaller than or equal to some constant 0734) 3 BASIC RESULTS The ey to our results is the inequality 3) with x determined by 4) This leads to the following analytical expression for γ : Lemma 3 For define f : [0, ] [0, ] by [ 2 f x) = + x x 2 x 2 ) + ln + x ) Then γ = inf{f x) 0 < x < } x + ln x) + x2 ln + )] For the benefit of the reader we point out that this inequality can be derived rather directly from the lemma of replacement developed in [5] The relevant part of that lemma is the following result: If f : [0, ] R is concave/convex ie concave on [0, ξ], convex on [ξ, ] for some ξ [0, ]), then, for any P M +N), there exists and a convex combination P 0 of U + and U such that F P 0 ) F P ) with F defined by F Q) = fq n); Q M +N) J Inequal Pure and Appl Math, 72) Art 59,

8 8 FLEMMING TOPSØE Proof By the defining relation 8) and by 3) with x given by 4), recalling also the relation 2), we realize that γ is the infimum over x ]0, [ of 2 [ H x)u+ + xu x 2 x 2 ) ln x 2 ln + ) x 2 ) ] ) Writing out the entropy of x)u + +xu we find that the function defined by this expression is, indeed, the function f It is understood that f x) is defined by continuity for x = 0 and x = An application of l Hôspitals rule shows that 3) f 0) = 2u, f ) = Then we investigate the limiting behaviour of f ) for Lemma 32 The pointwise limit f = lim f exists in [0, ] and is given by 32) fx) = 2 x ln x)) x 2 + x) ; 0 < x < with f0) = and f) = Alternatively, 33) fx) = 2 x n + x n + 2 ; 0 x The simple proof, based directly on Lemma 3, is left to the reader We then investigate some of the properties of f: Lemma 33 The function f is convex, f0) =, f) = and f 0) = The real number 3 x 0 = argminf is uniquely determined by one of the following equivalent conditions: i) f x 0 ) = 0, ii) ln x 0 ) = 2x 0+x 0 x 2 0 ) n+ n+3 + n n+2) x n 0 = 6 3x 0 +2) x 0 ), iii) n= One has x and γ with γ = fx 0 ) = min f Proof By standard differentiation, say based on 32), one can evaluate f and f One also finds that i) and ii) are equivalent The equivalence with iii) is based on the expansion f 2 n + x) = + x) 2 n n ) x n n + 2 which follows readily from 4) The convexity, even strict, of f follows as f can be written in the form 2 fx) = 3 + ) x n + 2 x n + x, easily recognizable as a sum of two convex functions The approximate values of x 0 and γ were obtained numerically, based on the expression in ii) The convergence of f to f is in fact increasing: Lemma 34 For every, f f + or, as a power series in x, fx) = 2 0 )n l n+2 )x n with l n = n ) n=2 J Inequal Pure and Appl Math, 72) Art 59,

9 ENTROPY LOWER BOUNDS 9 Proof As a more general result will be proved as part i) of Theorem 4, we only indicate that a direct proof involving three times differentiation of the function x) = 2 x2 x 2 )f + x) f x)) is rather straightforward Lemma 35 γ = ln Proof We wish to find the best largest) constant c such that 34) HP ) ln 4 ICP )) + 2c ICP ) ) ICP )) 2 holds for all P M+N), cf 9), and now that we only need to worry about distributions P M+2) Let P = p, q) be such a distribution, ie 0 p, q = p Tae p as an independent variable and define the auxiliary function h = hp) by h = H ln 4 IC) 2c Here, H = p ln p q ln q and IC = p 2 + q 2 Then: IC 2 ) IC) h = ln q + 2p q) ln 4 2cp q)3 4IC), p h = pq + 4 ln 4 2c pq) Thus h0) = h ) = h) = 0, 2 h 0) =, h ) = 0 and 2 h ) = Further, h ) = ln 4 4c, hence h assumes negative values if c > ln 4 Assume now that c < ln 4 Then h ) > 0 As h has at most) two inflection points follows from the formula for 2 h ) we must conclude that h 0 otherwise h would have at least six inflection points!) Thus h 0 if c < ln 4 Then h 0 also holds if c = ln 4 The lemma is an improvement over an inequality established in [] as we shall comment more on in Section 4 With relatively little extra effort we can find reasonable bounds for each of the γ s in terms of γ What we need is the following lemma: Lemma 36 For and 0 x <, 35) f x) = and 2 + ) x 2 ) 2n + 2 [ x 2n+ 2n ) fx) = 2 x 2 Proof Based on the expansions ) + x2n + )] 2n+2 2n + 2n+ x 2n+ 2n + 2 2n ) x2n 2n + x ln x) = x 2 x n n + 2 J Inequal Pure and Appl Math, 72) Art 59,

10 0 FLEMMING TOPSØE and + x) ln + x ) = x + x 2 ) n x n n + 2)n + ) n+ which is also used for = with x replaced by x), one readily finds that + x) ln + x ) Upon writing in the form x) ln x) + + )x 2 ln = x 2 [ + = + ) n n + 2)n + ) n+ 2n + 2 2n + + ) 2n + 3 ) x n n + 2)n + ) ) ] ) n + n+ and collecting terms two-by-two, and subsequent division by x 2 and multiplication by 2, 35) emerges Clearly, 36) follows from 35) by taing the limit as converges to infinity Putting things together, we can now prove the following result: Theorem 37 We have γ γ 2, γ = ln and γ γ where γ can be defined as { )} 2 γ = min ln 0<x< x 2 + x) x x Furthermore, for each, 37) ) γ γ + ) γ 2 Proof The first parts follow directly from Lemmas 3 35 To prove the last statement, note that, for n 0, 2n+2 2 It then follows from Lemma 36 that + )f ) f, hence 2 f γ )γ follows )f and Similarly, note that + 2n+) + 3 for n and that, for n = 0, the second term in the summation in 35) vanishes) Then use Lemma 36 to conclude that + )f + )f 3 The inequality γ + )γ follows 2 The discussion contains more results, especially, the bounds in 37) are sharpened Justification: 4 DISCUSSION AND FURTHER RESULTS The justification for the study undertaen here is two-fold: As a study of certain aspects of the relationship between entropy and index of coincidence which is part of the wider theme of comparing one Rényi entropy with another, cf [5] and [4] and as a preparation for certain results of exact prediction in Bernoulli trials Both types of justification were carefully dealt with in Sections and 2 J Inequal Pure and Appl Math, 72) Art 59,

11 ENTROPY LOWER BOUNDS Lower bounds for distributions over two elements: Regarding Lemma 35, the ey result proved is really the following inequality for a twoelement probability distribution P = p, q): 4) 4pq ln 2 + ln 2 ) ) 4pq) Hp, q) 2 Let us compare this with the lower bounds contained in the following inequalities proved in []: ln p ln q 42) ln p ln q Hp, q), ln 2 43) ln 2 4pq Hp, q) ln 24pq) / ln 4 Clearly, 4) is sharper than the lower bound in 43) Numerical evidence shows that normally 4) is also sharper than the lower bound in 42) but, for distributions close to a deterministic distribution, 42) is in fact the sharper of the two More on the convergence of f to f: Although Theorem 37 ought to satisfy most readers, we shall continue and derive sharper bounds than those in 37) This will be achieved by a closer study of the functions f and their convergence to f as By looing at previous results, notably perhaps Lemma 3 and the proof of Theorem 37, one gets the suspicion that it is the sequence of functions + )f rather than the sequence of f s that are well behaved This is supported by the results assembled in the theorem below, which, at least for parts ii) and iii), are the most cumbersome ones to derive of the present research: Theorem 4 i) + )f f, ie 2f 3 2 f f 3 f ii) For each, the function f + )f is decreasing in [0, ] iii) For each, the function + )f /f is increasing in [0, ] The technique of proof will be elementary, mainly via torturous differentiations which may be replaced by MAPLE loo-ups, though) and will rely also on certain inequalities for the logarithmic function in terms of rational functions A setch of the proof is relegated to the appendix An analogous result appears to hold for convergence from above to f Indeed, experiments on MAPLE indicate that + + )f 2 f and that natural analogs of ii) and iii) of Theorem 4 hold However, this will not lead to improved bounds over those derived below in Theorem 42 Refined bounds for γ in terms of γ: Such bounds follow easily from ii) and iii) of Theorem 4: Theorem 42 For each, the following inequalities hold: 44) 2u )γ γ + γ u Proof Define constants a and b by a = inf fx) + ) ) f x), 0 x + b = inf ) f x) 0 x fx) J Inequal Pure and Appl Math, 72) Art 59,

12 2 FLEMMING TOPSØE Then b γ + ) γ γ a Now, by ii) and iii) of Theorem 4 and by an application of l Hôpitals rule, we find that a = 2 + ) u 2, b = + ) 2u ) The inequalities of 44) follow Note that another set of inequalities can be obtained by woring with sup instead of inf in the definitions of a and b However, inspection shows that the inequalities obtained that way are weaer than those given by 44) The inequalities 44) are sharper than 37) of Theorem 37 but less transparent Simpler bounds can be obtained by exploiting lower bounds for u obtained from lower bounds for ln + x), cf [2]) One such lower bound is given in footnote [] and leads to the inequalities 2 45) 2 + γ γ + γ Of course, the upper bound here is also a consequence of the relatively simple property i) of Theorem 4 Applying sharper bounds of the logarithmic function leads to the bounds 2 46) 2 + γ γ ) γ APPENDIX We shall here give an outline of the proof of Theorem 4 We need some auxiliary bounds for the logarithmic function which are available from [2] In particular, for the function λ defined by ln + x) λx) =, x one has 47) 2 x)λy) x + y valid for 0 x and 0 y <, cf 6) of [2] λxy) xλy) + x), Proof of i) of Theorem 4 Fix 0 x and introduce the parameter y = Put ψy) = + ) x 2 x 2 ) f x) + x) ln x) 2 with = ) Then, simple differentiation and an application of the right hand inequality of y 47) shows that ψ is a decreasing function of y in ]0, ] This implies the desired result Proof of ii) of Theorem 4 Fix and put ϕ = f + ) f Then ϕ can be written in the form ϕ 2x x) = x 4 x 2 ) ψx) 2 We have to prove that ψ 0 in [0, ] After differentiations, one finds that ψ0) = ψ) = ψ 0) = ψ ) = ψ 0) = 0 J Inequal Pure and Appl Math, 72) Art 59,

13 ENTROPY LOWER BOUNDS 3 Furthermore, we claim that ψ ) < 0 This amounts to the inequality 48) ln + y) > y8 + 7y) + y)8 + 3y) with y = This is valid for y > 0, as may be proved directly or deduced from a nown stronger inequality related to the function φ 2 listed in Table of [2]) Further differentiation shows that ψ 0) = y 3 < 0 With two more differentiations we find that ψ 5) x) = 8y3 + xy) 20y3 2 + xy) 6y3 y 2 ) + 24y3 y 2 ) 3 + xy) 4 + xy) 5 Now, if ψ assumes positive values in [0, ], ψ x) = 0 would have at least 4 solutions in ]0, [, hence ψ 5) would have at least one solution in ]0, [ In order to arrive at a contradiction, we put X = + xy and note that ψ 5) x) = 0 is equivalent to the equality 9X 3 0X 2 3 y 2 )X + 2 y 2 ) = 0 However, it is easy to show that the left hand side here is upper bounded by a negative number Hence we have arrived at the desired contradiction, and conclude that ψ 0 in [0, ] Proof of iii) of Theorem 4 Again, fix and put Then, once more with y =, ψx) = + )f x) fx) ψx) = + xy) ln + xy) x2 + y) ln + y) xy x) y x) x ln x)) We will show that ψ 0 Write ψ in the form ψ = y denominator 2 ξ, where denominator refers to the denominator in the expression for ψ Then ξ0) = ξ) = 0 Regarding the continuity of ξ at with ξ) = 0, the ey fact needed is the limit relation + xy lim ln x) ln x + y = 0 Differentiation shows that ξ 0) = 2y < 0 and that ξ ) = Further differentiation and exploitation of the left hand inequality of 47) gives: ξ x) y 0x 2xy + xy ) x y 2x + x + ) x + 6, and this quantity is 0 in [0, [ We conclude that ξ 0 in [0, [ The desired result follows All parts of Theorem 4 are hereby proved J Inequal Pure and Appl Math, 72) Art 59,

14 4 FLEMMING TOPSØE REFERENCES [] M BEN-BASSAT, f-entropies, probability of error, and feature selection, Information and Control, ), [2] M FEDER AND N MERHAV, Relations between entropy and error probability, IEEE Trans Inform Theory, ), [3] W FELLER, An Introduction to Probability Theory and its Applications, Volume I Wiley, New Yor, 950 [4] J Dj GOLIĆ, On the relationship between the information measures and the Bayes probability of error IEEE Trans Inform Theory, IT-335) 987), [5] P HARREMOËS AND F TOPSØE, Inequalities between entropy and index of coincidence derived from information diagrams IEEE Trans Inform Theory, 477) 200), [6] P HARREMOËS AND F TOPSØE, Unified approach to optimization techniques in Shannon theory, in Proceedings, 2002 IEEE International Symposium on Information Theory, page 238 IEEE, 2002 [7] VA KOVALEVSKIJ, The Problem of Character Recognition from the Point of View of Mathematical Statistics, pages 3 30 Spartan, New Yor, 967 [8] R KRICHEVSKII, Laplace s law of succession and universal encoding, IEEE Trans Inform Theory, ), [9] DL TEBBE AND SJ DWYER, Uncertainty and the probability of error, IEEE Trans Inform Theory, 4 968), [0] F TOPSØE, Information theory at the service of science, in Bolyai Society Mathematical Studies, Gyula OH Katona Ed), Springer Publishers, Berlin, Heidelberg, New Yor, 2006 to appear) [] F TOPSØE, Bounds for entropy and divergence of distributions over a two-element set, J Ineq Pure Appl Math, 22) 200), Art 25 [ONLINE: php?sid=4] [2] F TOPSØE, Some bounds for the logarithmic function, in Inequality Theory and Applications, Vol 4, Yeol Je Cho and Jung Kyo Kim and Sever S Dragomir Eds), Nova Science Publishers, New Yor, 2006 to appear) RGMIA Res Rep Coll, 72) 2004), [ONLINE: eduau/v7n2html] [3] I VAJDA AND K VAŠEK, Majorization, concave entropies, and comparison of experiments Problems Control Inform Theory, 4 985), 05 5 [4] K ZYCZKOWSKI, Rényi extrapolation of Shannon entropy, Open Systems and Information Dynamics, ), J Inequal Pure and Appl Math, 72) Art 59,