MAJORIZATION IN INFORMATION THEORY

Transcription

1 Journl of Inequlities nd Specil Functions ISSN: , URL: Volume 8 Issue 4(2017), Pges MAJORIZATION IN INFORMATION THEORY NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Abstrct. In this pper, we give the generlized results for mjoriztion inequlity in integrl form by using Csiszár f-divergence. We show how the Shnnon entropy is connected to the theory of mjoriztion. We lso give Shnnon entropy nd the Kullbck-Leibler divergence for obtined results. As pplictions, we present the mjoriztion inequlity for vrious distnces like vritionl distnce, Hellinger distnce, χ 2 -divergence, Bhttchryy distnce, Hrmonic distnce, Jeffreys distnce nd tringulr discrimintion which obtin by pplying some specil type of convex functions. 1. Introduction nd Preliminries Distnce or divergence mesures re of key importnce in different fields like theoreticl nd pplied sttisticl inference nd dt processing problems, such s estimtion, detection, clssifiction, compression, recognition, indextion, dignosis nd model selection etc. The literture on such types of issues is wide nd hs considerbly expnded in the recent yers. In prticulr, following the set of some books published during the second hlf of the eighties [1, 8, 14, 17] nd the number of some books hve been published during the lst decde or so [2, 3, 7, 11]. In report on divergence mesures nd their tight connections with the notion of entropy, informtion nd men vlues, n ttempt hs been mde to describe vrious procedures for building divergences mesures from entropy functions or from generlized men vlues nd conversely for defining entropies from divergence mesures [4]. Another gol ws to clrify the properties of nd reltionships between two min clsses of divergence mesures, nmely the f-divergence nd the Bregmn divergences [5]. One of the most importnt issues in mny pplictions of Probbility Theory is finding n pproprite mesure of distnce (or difference or discrimintion) between two probbility distributions. A number of divergence mesures for this purpose hve been proposed nd extensively studied by Csiszár [12], Kullbck nd Leibler [21], Rényi [34], Ro [35] nd Lin [22] nd others. These mesures hve been pplied in vriety of fields such 2010 Mthemtics Subject Clssifiction. 94A15, 94A17, 26A51, 26D15. Key words nd phrses. Mjoriztion inequilty, Csiszár f-divergence, Shnnon entropy, Kullbck- Leibler divergence, vrition distnce, Hellinger distnce nd χ 2 -divergence. Submitted My 14, Published July 28, Communicted by Guest Editor Kenjiro Yngi. c 2017 Iliris Reserch Institute, Prishtinë, Kosovë. 42

2 MAJORIZATION IN INFORMATION THEORY 43 s: nthropology [35], genetics [30], finnce, economics, nd politicl science [36, 37, 38], biology [33], the nlysis of contingency tbles [16], pproximtions of probbility distributions [10, 20], signl processing [18, 19] nd pttern recognition [6, 9]. A number of these mesures of distnce re specific cses of Csiszár f-divergence nd so further explortion of this concept will hve flow on effect to other mesures of distnce nd to res in which they re pplied. Mtić et l. [23], (1999)[24], (2000)[25] nd (2002)[26] continuously worked on Shnnon s inequlity nd relted inequlities in the probbility distribution nd informtion science. They studied nd discussed in [25, 26] severl spects of Shnnon s inequlity in discrete s well s in integrl forms, by presenting upper estimtes of the difference between its two sides. Applictions to the bounds in informtion theory re lso given. It is generlly common to tke log with bse of 2 in the introduced notions, but in our investigtions this is not essentil. For exmple, the functions x x log b (x) (x > 0, b > 1) nd x x rctn (x) (x R) re convex. They proved counterprt of the integrl Shnnon inequlity (see [25, p ]): Let I be mesurble subset of the rel line nd p(x) nd q(x) positive integrble functions on I such tht I p(x)dx = 1 nd α := I q(x)dx <. Suppose tht for b > 1 t lest one of the integrls J p := p(x) log 1 I p(x) dx nd J q := p(x) log 1 I q(x) dx is finite. If ( I p 2 (x)/q(x) ) dx <, then both J p nd J q re finite nd 0 J q J p + log b α log [ p 2 ] (x) α I q(x) dx 1 [ lnb α p 2 ] (x) I q(x) dx 1, with equlity throughout if nd only if q(x) = αp(x).e. on I. The notion of entropy H b (X) := i=1 p i log (1/p i ) cn be extended to the cse of generl rndom vrible X, by pproximting X by discrete rndom vribles. In the cse when X is non-discrete, H b (X) is usully infinite. For exmple, this lwys hppens when X is continuous (see [27, p.38]). In the cse when X is continuous rndom vrible with density p(x) ( nonnegtive mesurble function on R such tht R p(x)dx = 1), we my define the so-clled differentil entropy of X by h b (X) := p(x) log 1 dx (b > 1), p(x) R whenever the integrl exists. They showed tht h b (X) log ( s 2πe ) when the distribution of X is close to the Gussin distribution with vrince s 2. Also, h b (x) log (µe) if the distribution of X is close to the exponentil distribution with men µ. Finlly, h b (x) log (l) (1) whenever the distribution of X is close to the uniform distribution over n intervl of length l.

3 44 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Drgomir gve in his monogrph [15] bout these divergences like the Kullbck-Leibler divergence, vritionl distnce, Hellinger distnce, χ 2 -divergence, Bhttchryy distnce, Hrmonic distnce, Jeffreys distnce nd tringulr discrimintion. Now we introduce the min mthemticl theory explored in the present work, the theory of mjoriztion. It is powerful nd elegnt mthemticl tool which cn be pplied to wide vriety of problems s in quntum mechnics. The theory of mjoriztion is closely relted to the notions of rndomness nd disorder. It indeed llows us to compre two probbility distributions, in order for us to know which one of the two is more rndom. Let us now give the most generl definition of mjoriztion. Let x, y be rel vlued functions defined on n intervl [, b] such tht s x(τ)dτ, s y(τ)dτ both exist for ll s [, b]. [31, p.324] x(τ) is sid to mjorize y(τ), in symbol, x(τ) y(τ), for τ [, b] if they re decresing in τ [, b] nd s y(τ) dτ nd equlity in (2) holds for s = b. s x(τ) dτ for s [, b], (2) The following theorem cn be regrded s mjoriztion theorem in integrl cse [31, p.325]: Theorem 1. Let x nd y be rel vlued functions defined on n intervl [, b]. x y for τ [, b] iff they re decresing in [, b] nd f (y(τ)) dτ f (x(τ)) dτ holds for every f tht is continuous nd convex in [, b] such tht the integrls exist. The following theorem is simple consequence of Theorem in [32] (see lso [31, p.328]): Theorem 2. Let x, y : [, b] R, x nd y re continuous nd incresing nd let µ : [, b] R be function of bounded vrition. (i) If nd ν y(τ) dµ(τ) ν y(τ) dµ(τ) = x(τ) dµ(τ) ν [, b], (3) hold, then for every continuous convex function f, we hve f (y(τ)) dµ(τ) x(τ) dµ(τ) (4) f (x(τ)) dµ(τ). (5) (ii) If (3) holds, then (5) holds for every continuous incresing convex function f.

4 MAJORIZATION IN INFORMATION THEORY 45 The following theorem is slight extension of Lemm 2 in [29] which is proved by Mligrnd et l. (lso see [28]): Theorem 3. Let w, x nd y be positive functions on [, b]. Suppose tht f : [0, ) R is convex function nd tht ν y(t) w(t) dt ν y(t) w(t) dt = x(t) w(t) dt, ν [, b] nd (i) If y is decresing function on [, b], then f (y(t)) w(t) dt (ii) If x is n incresing function on [, b], then f (x(t)) w(t) dt x(t) w(t) dt. f (x(t)) w(t) dt. (6) f (y(t)) w(t) dt. (7) If f is strictly convex function nd x y (. e.), then (6) nd (7) re strict. 2. Min Results Csiszár [12, 13] introduced the notion of f-divergence s follows: Definition 1. Let f : (0, ) (0, ) be convex function. Let p, q : [, b] (0, ) be positive probbility densities. The f-divergence functionl is ( ) D f (p, q) := f dt. Bsed on the previous definition, we introduce new functionl: Definition 2. Let J := [0, ) R be n intervl, nd let f : J R be function. Let p, q : [, b] (0, ) such tht We define p(x) q(x) J, ˆD f (p, q) := x [, b]. f ( ) dt. The specil cse of the bove functionl, we define ( ) r(t) ˆD idj f (r, q) := r(t)f dt.

5 46 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Motivted by the ides in [25] (2000) nd [26] (2002), we discuss the behviour of the results in the form of divergences nd entropies. The next theorem is the generliztion of the result (1) in integrl cse given in [25]. We present the following theorem is the connection between Csiszár f-divergence nd weighted mjoriztion inequlity in integrl form s one function is monotonic: Theorem 4. Let J := [0, ) R be n intervl nd f : J R be convex function. Let lso p, q, r : [, b] (0, ) such tht nd with υ r(t)dt υ r(t)dt = dt, υ [, b] (8) dt, (9), r(t) J, t [, b]. (10) is decresing function on [, b], then ˆD f (r, q) ˆD f (p, q). (11) is n incresing function on [, b], then the inequlity is reversed, i.e. ˆD f (r, q) ˆD f (p, q). (12) If f is strictly convex function nd r(t) (. e.), then strict inequlity holds in (11) nd (12). If f is concve function then the reverse inequlities hold in (11) nd (12). If f is strictly concve nd r(t) (. e.), then the strict reverse inequlities hold in (11) nd (12). Proof. (i): We use Theorem 3 (i) with substitutions x(t) := > 0 t [, b] nd f := f nd lso using the fct tht r(t) then we get (11). r(t), y(t) :=, w(t) := is decresing function (ii) We cn prove with the similr substitutions s in the first prt by using Theorem 3 (ii) tht is the fct tht is n incresing function. Theorem 5. Let J := [0, ) R be n intervl nd f : J R be function such tht x xf(x), x J is convex function. Let lso p, q, r : [, b] (0, ) such tht stisfying (8) nd (9) with, r(t) J, t [, b]. (13) is decresing function on [, b], then ˆD idj f (r, q) ˆD idj f (p, q). (14)

6 MAJORIZATION IN INFORMATION THEORY 47 is n incresing function on [, b], then the inequlity is reversed, i.e. ˆD idj f (r, q) ˆD idj f (p, q). (15) If xf(x) is strictly convex function nd r(t) (. e.), then (14) nd (15) re strict. If xf(x) is concve function then the reverse inequlities hold in (14) nd (15). If xf(x) is strictly concve nd r(t) (. e.) then the strict reverse inequlities hold in (14) nd (15). Proof. (i): We use Theorem 3 (i) with substitutions x(t) := r(t), y(t) :=, w(t) = > 0, t [, b] nd f(x) := xf(x) nd lso using the fct tht r(t) is decresing function then we get (14). (ii) We cn prove with the similr substitutions s Prt (i) in Theorem 3 (ii) for f(x) := xf(x) nd is n incresing function. The theory of mjoriztion nd the notion of entropic mesure of disorder re closely relted. Bsed on this fct, the im of this pper is to look for mjoriztion reltions with the connection of entropic inequlities. This ws interesting to do for two min resons. The first one is the fct tht the mjoriztion reltions re usully stronger thn the entropic inequlities, in the sense tht they imply these entropic inequlities, but tht the converse is not true. The second reson is the fct tht when we dispose of mjoriztion reltions between two different quntum sttes, we know tht we cn trnsform one of the stte into the other using some unitry trnsformtion. The concept of entropy lone would not llow us to prove such property. The Shnnon entropy ws introduced by Shnnon himself in the field of clssicl informtion. There re two wys of viewing the Shnnon entropy. Suppose we hve rndom vrible X, nd we lern its vlue. In one point of view, the Shnnon entropy quntifies the mount of informtion we gin we lern the vlue of X (fter mesurement). In nother point of view, the Shnnon entropy tells us the mount of uncertinty bout the vrible of X before we lern its vlue (before mesurement). 3. Applictions We mention severl specil cses of the previous results. The first cse corresponds to the entropy of continuous probbility density (see [25, p.506]): Definition 3. (Shnnon Entropy) Let p : [, b] (0, ) be positive probbility density. The Shnnon entropy of p(x) is defined by H(p(x)) := whenever the integrl exists. p(x) log p(x)dx (b > 1), (16)

7 48 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Note tht there is no problem with the definition in the cse of zero probbility, since lim x log x = 0. (17) x 0 Corollry 1. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), t [, b]. is decresing function nd the bse of log is greter thn 1, then we hve estimtes for the Shnnon entropy of ( ) r(t) log H(). (18) If the bse of log is in between 0 nd 1, then the reverse inequlity holds in (18). is n incresing function nd the bse of log is greter thn 1, then we hve estimtes for the Shnnon entropy of ( ) H() log. (19) If the bse of log is in between 0 nd 1 then the reverse inequlity holds in (19). Proof. (i): Substitute f(x) := log x nd := 1, t [, b] in Theorem 4 (i) then we get (18). (ii) We cn prove by switching the role of with r(t) i.e., r(t) := 1 t [, b] nd f(x) := log x in Theorem 4 (ii) then we get (19). Corollry 2. Let p, r : [, b] (0, ) be functions such tht stisfying (8). is decresing function nd the bse of log is greter thn 1, then the following comprison inequlity between Shnnon entropies of nd r(t) H(r(t)) H(). (20) If the bse of log is in between 0 nd 1, then the reverse inequlity holds in (20). is n incresing function nd the bse of log is greter thn 1, then the following comprison inequlity between Shnnon entropies of nd r(t) H(r(t)) H(). (21) If the bse of log is in between 0 nd 1 then the reverse inequlity holds in (21). Proof. (i): Consider the function f(x) := log x. Then the function xf(x) := x log x is convex function. Substitute f(x) := log x nd := 1, t [, b] in Theorem 5 (i) then we get (20). (ii) We cn prove with the similr substitutions s Prt (i) in Theorem 5 (ii) then we get (21). The second cse corresponds to the reltive entropy or Kullbck-Leibler divergence between two probbility densities:

8 MAJORIZATION IN INFORMATION THEORY 49 Definition 4. (Kullbck Leibler Divergence) Let p, q : [, b] (0, ) be positive probbility densities. The Kullbck-Leibler (K-L) divergence between nd is defined by ( ) L (, ) := log dt. Corollry 3. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), t [, b]. is decresing function nd the bse of log is greter thn 1, then ˆD log x (r, q) ˆD log x (p, q). (22) If the bse of log is in between 0 nd 1, then the reverse inequlity holds in (22). is n incresing function nd the bse of log is greter thn 1, then ˆD log x (r, q) ˆD log x (p, q). (23) If the bse of log is in between 0 nd 1 then the reverse inequlity holds in (23). Proof. (i): Substitute f(x) := log x in Theorem 4 (i) then we get (22). (ii) We cn prove with substitution f(x) := log x in Theorem 4 (ii). Corollry 4. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), t [, b]. is decresing function nd the bse of log is greter thn 1, then the connection between K-L divergence of (r(t), ) nd (, ) L (r(t), ) L (, ). (24) If the bse of log is in between 0 nd 1, then the reverse inequlity holds in (24). is n incresing function nd the bse of log is greter thn 1, then the connection between K-L divergence of (r(t), ) nd (, ) L (r(t), ) L (, ). (25) If the bse of log is in between 0 nd 1 then the reverse inequlity holds in (25). Proof. (i): Substitute f(x) := log x in Theorem 5 (i) then we get (24). (ii) We cn prove with substitution f(x) := log x in Theorem 5 (ii) we get (25). In Informtion Theory nd Sttistics, vrious divergences re pplied in ddition to the Kullbck-Leibler divergence.

9 50 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Definition 5. (Vritionl Distnce) Let p, q : [, b] (0, ) be positive probbility densities. The vrition distnce between nd is defined by ˆD v (, ) := dt. Corollry 5. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then t [, b]. ˆD v (r(t), ) ˆD v (, ). (26) is n incresing function, then the inequlity is reversed, i.e. ˆD v (r(t), ) ˆD v (, ). (27) Proof. (i): Since f(x) := x 1 be convex function for x R +, therefore substitute f(x) := x 1 in Theorem 4 (i) then r(t) 1 b dt 1 dt, r(t) dt dt, since > 0 then we get (26). (ii) We cn prove with substitution f(x) := x 1 in Theorem 4 (ii). Definition 6. (Hellinger Distnce) Let p, q : [, b] (0, ) be positive probbility densities. The Hellinger distnce between nd is defined by [ ] 2 ˆD H (, ) := dt. Corollry 6. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then t [, b]. ˆD H (r(t), ) ˆD H (r(t), ). (28) is n incresing function, then the inequlity is reversed, i.e. ˆD H (r(t), ) ˆD H (r(t), ). (29)

10 MAJORIZATION IN INFORMATION THEORY 51 Proof. (i): Since f(x) := ( x 1) 2 be convex function for x R +, therefore substitute f(x) := ( x 1) 2 in Theorem 4 (i) then [ ] 2 b [ ] 2 r(t) b 1 dt 1 dt, since > 0 then we get (28). (ii) We cn prove with substitution f(x) := ( x 1) 2 in Theorem 4 (ii). Definition 7. (χ 2 Divergence) Let p, q : [, b] (0, ) be positive probbility densities. The χ 2 -divergence between nd is defined by [ b ( ) 2 ˆD idj χ 2 (, ) := 1] dt. Corollry 7. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), t [, b]. is decresing function, then ˆD idj χ 2 (r(t), ) ˆD idj χ2 (, ). (30) is n incresing function, then the inequlity is reversed, i.e. ˆD idj χ 2 (r(t), ) ˆD idj χ2 (, ). (31) Proof. (i): Since f(x) := x ( 1 1 ) be convex function for x R +, therefore substitute x 2 f(x) := x ( 1 1 ) in Theorem 4 (i) then x 2 [ b ( r(t) ) 2 1] dt r(t) [ b ( ) 2 1] dt, we get (30). We cn lso prove by using Theorem 5 (i) for function f(x) := 1 1 such x 2 tht x f(x) := x ( 1 1 ) be convex function for x R +, we get (30). x 2 (ii) We cn prove with substitution f(x) := x ( 1 1 ) in Theorem 4 (ii). x 2 Definition 8. (Bhttchryy Distnce) Let p, q : [, b] (0, ) be positive probbility densities. The Bhttchryy distnce between nd is defined by ˆD B (, ) := dt.

11 52 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Corollry 8. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then t [, b]. ˆD B (, ) ˆD B (r(t), ). (32) is n incresing function, then the inequlity is reversed, i.e. ˆD B (, ) ˆD B (r(t), ). (33) Proof. (i): Since f(x) := x be convex function for x R +, therefore substitute f(x) := x in Theorem 4 (i) then ( ) b ( ) r(t) b dt dt, we get (32). (ii) We cn prove with substitution f(x) := x in Theorem 4 (ii). Definition 9. (Hrmonic Distnce) Let p, q : [, b] (0, ) be positive probbility densities. between nd is defined by ˆD idj H (, ) := 2 + dt. The Hrmonic distnce Corollry 9. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then t [, b]. ˆD idj H (, ) ˆD idj H (r(t), ). (34) is n incresing function, then the inequlity is reversed, i.e. ˆD idj H (, ) ˆD idj H (r(t), ). (35) Proof. (i): Since f(x) := 2 2x x+1, then xf(x) := x+1 be concve function for x 0, therefore substitute f(x) := 2 in Theorem 5 (i) then x+1 2 / + 1 dt we get (34). (ii) We cn prove with substitution f(x) := 2 x+1 r(t) 2 r(t)/ + 1 dt, in Theorem 5 (ii).

12 MAJORIZATION IN INFORMATION THEORY 53 Definition 10. (Jeffreys Distnce) Let p, q : [, b] (0, ) be positive probbility densities. The Jeffreys distnce between nd is defined by [ ] ˆD J (, ) := [ ] ln dt. Corollry 10. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then t [, b]. ˆD J (r(t), ) ˆD J (, ). (36) is n incresing function, then the inequlity is reversed, i.e. ˆD J (r(t), ) ˆD J (, ). (37) Proof. (i): Since f(x) := (x 1) ln x be convex function for x R +, therefore substitute f(x) := (x 1) ln x in Theorem 4 (i) then ( ) ( ) r(t) r(t) 1 ln dt ( ) ( ) 1 ln dt, we get (36). (ii) We cn prove with substitution f(x) := (x 1) ln x in Theorem 4 (ii). Definition 11. (Tringulr Discrimintion) Let p, q : [, b] (0, ) be positive probbility densities. The tringulr discrimintion between nd is defined by ˆD (, ) := [ ] 2 + Corollry 11. Let p, q, r : [, b] (0, ) be functions such tht stisfying (8) with, r(t) J := (0, ), is decresing function, then dt. t [, b]. ˆD (r(t), ) ˆD (, ). (38) is n incresing function, then the inequlity is reversed, i.e. ˆD (r(t), ) ˆD (, ). (39)

13 54 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 Proof. (i): Since f(x) := (x 1)2 x+1 be convex function for x 0, therefore substitute f(x) := (x 1)2 x+1 in Theorem 4 (i) then (r(t)/ 1)2 r(t)/ + 1 ((r(t) )/)2 (r(t) + )/ dt dt (/ 1)2 / + 1 dt, (( )/)2 ( + )/ we get (38). (ii) We cn prove with substitution f(x) := (x 1)2 x+1 in Theorem 4 (ii). dt, AUTHOR S CONTRIBUTION All uthors contributed eqully. All uthors red nd pproved the finl mnuscript. COMPETING INTERESTS The uthors declre tht they hve no competing interests. References [1] S. I. Amri, Differentil-Geometricl Methods in Sttistics, 28 of Lecture Notes in Sttistics, Springer- Verlg, New York, USA, [2] S. I. Amri nd H. Ngok, Methods of Informtion Geometry, 191 of Trnsltions of Mthemticl Monogrphs, Americn Mthemticl Society nd Oxford University Press, Oxford, UK, [3] K. A. Arwini nd C. T. J. Dodson, Informtion Geometry-Ner Rndomness nd Ner Independence, 1953 of Lecture Notes in Mthemtics, Springer, [4] M. Bsseville, Informtion: entropies, divergences et moyennes, Reserch Report 1020, IRISA, [5] M. Bsseville nd J. F. Crdoso, On entropies, divergences nd men vlues, In Proceedings of the IEEE Interntionl Symposium on Informtion Theory (ISIT 95), Whistler, British Columbi, Cnd, [6] M. B. Bsst, f-entropies, probbility of error nd feture selection, Inform. Control, 39, , [7] A. Bsu, H. Shioy nd C. Prk, Sttisticl Inference: The Minimum Distnce Approch, Chpmn nd Hll/CRC Monogrphs on Sttistics nd Applied Probbility, CRC Press, Boc Rton, FL, [8] R. E. Blhut, Principles nd Prctice of Informtion Theory, Series in Electricl nd Computer Engineering, Addison Wesley Publishing Co., [9] C. H. Chen, Sttisticl Pttern Recognition, Rocelle Prk, New York, Hoyderc Book Co., [10] C. K. Chow nd C. N. Lin, Approximtimg discrete probbility distributions with dependence trees, IEEE Trn. Inf. Th., 14 (3), , [11] A. Cichocki, R. Zdunek, A. Phn nd S. I. Amri, Non-negtive Mtrix nd Tensor Fctoriztions: Applictions to Explortory Multi-Wy Dt Anlysis nd Blind Source Seprtion, John Wiley nd Sons Ltd, [12] I. Csiszár, Informtion-type mesures of difference of probbility distributions nd indirect observtions. Studi Sci. Mth. Hungr., 2: , [13] I. Csiszár, Informtion mesure: A criticl survey, Trns. 7th Prgue Conf. on Info. Th., Sttist. Decis. Funct., Rndom Processes nd 8th Europen Meeting of Sttist., B, Acdemi Prgue, 73-86, 1978.

14 MAJORIZATION IN INFORMATION THEORY 55 [14] T. M. Cover nd J. A. Thoms, Elements of Informtion Theory, Wiley Series in Telecommunictions, John Wiley nd Sons Ltd, [15] S. S. Drgomir, Inequlities for Csiszr f-divergence in Informtion Theory, RGMIA Monogrph, 2001, URL: [16] D. V. Gokhle nd S. Kullbck, Informtion in Contingency Tbles, New York, Mrcel Dekker, [17] R. M. Gry, Entropy nd Informtion Theory, Springer-Verlg, New York, USA, [18] T. T. Kdok nd L. A. Shepp, On the best finite set of liner observbles for desciminting two Gussin signls, IEEE Trns. Inf. Th., 13, , [19] T. Kilth, The divergence nd Bhttchryy distnce mesures in signl selection, IEEE Trns. Comm. Technology, COM-15, 52-60, [20] D. Kzkos nd T. Cotsids, A decision theory pproch to the pproximtion of discrete probbility densities, IEEE Trns. Perform. Anl. Mchine Intell., 1, 61-67, [21] S. Kullbck nd R. A. Leibler, On informtion nd sufficiency, Ann. Mth. Stt., 22, 79-86, [22] J. Lin, Divergence mesures bsed on the Shnnon entropy, IEEE Trns. Inf. Th., 37 (1) (1991), Sttistics, New York, [23] M. Mtić, C. E. M. Perce, J. Pečrić, Improvements of some bounds on entrop mesures in informtion theory, Mth. Inequl. Appl., 1, , [24] M. Mtić, C. E. M. Perce, J. Pečrić, On n inequlity for the entropy of probbility distribution, Act Mth. Hungr., 85, , [25] M. Mtić, C. E. M. Perce, J. Pečrić, Shnnon s nd relted inequlities in informtion theory. Survey on clssicl inequlities, editor Themistocles M. Rssis, Kluwer Acdemic Publ., , [26] M. Mtić, C. E. M. Perce, J. Pečrić, Some refinements of Shnnon s inequlities, ANZIAM J. (formerly J. Austrl. Mth. Soc. Ser. B) 43, , [27] R. J. McEliece, The theory of informtion nd coding, Addison-Wesley, Reding, Mss, [28] N. Ltif, J. Pečrić nd I. Perić, On Mjoriztion, Fvrd nd Berwld s Inequlities, Annls of Functionl Anlysis, 2, no. 1, 31-50, ISSN: , [29] L. Mligrnd, J. Pečrić nd L.E. Persson, Weighted Fvrd s nd Berwld s inequlities, J. Mth. Anl. Appl. 190, , [30] M. Mei, The theory of genetic distnce nd evlution of humn rces, Jpn J. Humn Genetics, 23, , [31] J. Pečrić, F. Proschn nd Y. L. Tong, Convex functions, Prtil Orderings nd Sttisticl Applictions, Acdemic Press, New York, [32] J. E. Pečrić, On some inequlities for functions with nondecresing increments, J. Mth. Anl. Appl., 98 (1), , [33] E. C. Pielou, Ecologicl Diversity, Wiley, New York, [34] A. Rényi, On mesures of entropy nd informtion, Proc. Fourth Berkeley Symp. Mth. Stt. nd Prob., University of Cliforni Press, 1, , [35] C. R. Ro, Diversity nd dissimilrity coefficients: unified pproch, Theor. Popul. Biol., 21, 24-43, [36] A. Sen, On Economic Inequlity, Oxford University Press, London, [37] H. Theil, Economics nd Informtion Theory, North-Hollnd, Amsterdm, [38] H. Theil, Sttisticl Decomposition Anlysis, North-Hollnd, Amsterdm, 1972.

15 56 NAVEED LATIF 1, DILDA PEČARIĆ2, AND JOSIP PEČARIĆ3 1 -Deprtment of Generl Studies, Jubil Industril College, Jubil Industril City 31961, Kingdom of Sudi Arbi E-mil ddress: nveed707@gmil.com 2 -Ctholic University of Croti, Ilic 242, Zgreb, Croti E-mil ddress: gildpec@gmil.com 3 -Fculty of Textile Technology Zgreb, University of Zgreb, Prilz Brun Filipović 30, Zgreb, Croti E-mil ddress: pecric@element.hr