arxiv: v2 [cs.it] 26 Sep 2011

Size: px

Start display at page:

Download "arxiv: v2 [cs.it] 26 Sep 2011"

Amanda Page
6 years ago
Views:

1 Sequences o Inequalities Among New Divergence Measures arxiv: v [cs.it] 6 Sep 011 Inder Jeet Taneja Departamento de Matemática Universidade Federal de Santa Catarina Florianópolis SC Brazil. ijtaneja@gmail.com taneja Abstract There are three classical divergence measures eist in the literature on inormation theory statistics. These are namely Jeryes-Kullback-Leiber [5 6] J-divergence. Sibson-Burbea- Rao[1] Jensen-Shannon divegernce Taneja[9] arithemtic-geometric mean divergence. These three measures bear an interesting relationship among each other are based on logarithmic epressions. The divergence measures like Hellinger discrimination symmetric χ divergence triangular discrimination are also known in the literature are not based on logarithmic epressions. Past years Dragomir et al. [3] Kumar Johnson [7] Jain Srivastava [4] studied dierent kind o divergence measures. In this paper we have presented some more new divergence measures obtained inequalities relating these new measures made connections with previous ones. The idea o eponential divergence is also introduced. Key words: J-divergence; Jensen-Shannon divergence; Arithmetic-Geometric divergence; Csiszár s -divergence; Inormation inequalities; Eponential divergence. AMS Classiication: 94A17; 6B10. 1 Introduction Let Γ n = P = p 1 p...p n p i > 0 } p i = 1 n be the set o all complete inite discrete probability distributions. For all PQ Γ n below we shall consider two groups o divergence measures. First Group P Q = p i p i 1 hp Q = 1 p i p i p i ΨP Q = p i 3 K 0 P Q = p i 4 1

2 FP Q = 1 p i q i 3 5 Second Group B 1 P Q = B P Q = B 3 P Q = B 4 P Q = B 5 P Q = p i p i p i p i 9 p i p i 10 p i B 6 P Q = p i 4 p i p i. 11 We observe that the measures appearing in irst group are already known in the literature. The irst three measures P Q hp Q ΨP Q are respectively known as triangular discrimination Hellingar s divergence symmetric chi-square divergence. The measures K 0 P Q FP Q are due to Jain Srivastava [4] Kumar Johnson [7] respectively. The measure B 1 P Q appearing in the second group is due to Dragomir et al. [3]. Other ive measures appearing in the second group are new. The measures are very much similar to each other the other eight measures are also similar to each other. The measures 6-11 can be written in terms o the measures 1-5. See the epression Classical Divergence Measures All the above eleven measures are without logarithmic epressions. There are three classical divergence measures known in the literature on inormation theory statistics are J-divergence Jensen-Shannon divergence Arithmetic-Geometric mean divergence given respectively as JP Q = p i ln p i 1 q i [ IP Q = 1 ] qi p i ln ln 13 p i p i

3 TP Q = ln p i. 14 We have the ollowing inequalities [10 11] [4] [7] among the measures P Q IP Q hp Q 1 JP Q TP Q K 0P Q 1 1 ΨP Q FP Q Some recent applications o Jensen s dierencereeq13 can be seen in Sachlas Papaioannou [8]. 1. Eponential Divergence For all P Q Γ n let consider the ollowing general measure K t P Q = p i t1 t = p i t1 When t = 0 we have the same measure as given in 4. When t = 1 we have K 1 P Q = B 1 P Q. When t 1 = k it reduces to one studied by Jain Srivastava [4]. We can easily check that the measures K t P Q are conve in the pair o probability distributions PQ Γ n t = Let us write E K P Q = 1 0! K 0P Q 1 1! K 1P Q 1! K P Q 1 3! K 3P Q The epression 17 leads us to ollowing eponential divergence E K P Q = p i ep p i PQ Γ n Γ n 18 The eight measures appearing in the inequalities 15 admits many nonnegative dierences. Here our aim to obtain inequalities relating these measures arising due to nonnegative dierences rom 15. Also our aim is to bring inequalities among the si measures B 1 P Q to B 6 P Q then again study their nonnegative dierences. Aim is also to connect the irst our terms o the series 17 with the known measures. Frequently we shall use the ollowing two lemmas. Lemma 1.1. I the unction : [0 R is conve normalized i.e. 1 = 0 then the -divergence C P Q given by C P Q = 19 is nonnegative conve in the pair o probability distribution PQ Γ n Γ n. 3

4 Lemma 1.. Let 1 : I R R two generating mapngs are normalized i.e. 1 1 = 1 = 0 satisy the assumptions: i 1 are twice dierentiable on ab; ii there eists the real constants αβsuch that α < β then we have the inequalities: α 1 β > 0 ab α C P Q C 1 P Q β C P Q. 0 The irst Lemma is due to Csiszár [] the second is due to author [10]. Some interesting properties o 19 see Taneja Kumar [1]. Conveity o Dierence o Divergences The inequalities given in 15 admit 8 nonnegative dierences. Conveity o some o these dierences is already studied in Taneja [11]. Here we shall study conveity o the dierences connected with new measures K 0 P Q FP Q. We can easily check that in all the cases 1 = 0. According to Lemma 1.1 it is suicient to show the conveity o the unctions i.e. to show that the second order derivative o i.e 0 or all > 0. We shall do each part separately. Throughout it is understood that > 0. i For D K0 TP Q: We can write D K0 TP Q = K0 T This gives K0 T = 1 8 K 0 T = ln 8. K 0 T = / 1 0. ii For D K0 JP Q: We can write D K0 JP Q = K0 J This gives K0 J = 1 8 K J = ln. K 0 J = / 0. 4

5 iii For D K0 hp Q: We can write D K0 hp Q = K0 h This gives K0 h = 1 8 K 0 h = iv For D K0 IP Q: We can write K h = 3 5/ 0. D K0 IP Q = K0 I = 8. This gives K0 I = 1 8 K 0 I = ln ln 8 K 0 I = 3 6 3/ / v For D K0 P Q: We can write D K0 P Q = K0 This gives K 0 = K0 = 1 8 K = = / / / / vi For D ΨK0 P Q: We can write D ΨK0 P Q = ΨK0 ΨK0 = 1 16 Ψ 1 8 K 0 5

6 This gives vii For D FΨ P Q: We can write This gives = = ΨK 0 = / D FΨ P Q = FΨ = 1 16 F 1 16 Ψ = 1 3 FΨ 1 3/ = / / 13 FΨ 1 3/ = 18 7/ 0. viii For D FK0 P Q: We can write This gives i For D FT P Q: We can write D FK0 P Q = FK0 FK0 = 1 16 F 1 8 K 0 = / 1 1 = /. FK 0 = / 0. D FT P Q = FT FT = 1 16 F T = / 1 1 ln. 6

7 This gives / / FT / = 18 7/ 1 For D FJ P Q: We can write This gives i For D Fh P Q: We can write This gives ii For D FI P Q: We can write This gives D FJ P Q = FJ FJ = 1 16 F 1 8 J = / 1 8 1ln / 45 FJ 44 3/ = 18 7/ 0. D Fh P Q = Fh Fh = 1 16 F h = / / = 3 3/. Fh = / 0. D FI P Q = 3/ FI FI = 1 16 F I = ln ln / / 1 FI = / /

8 iii For D F P Q: We can write D F P Q = F F = 1 16 F 1 4 = / = 1 1 3/ / 1. This gives F = / / / / / / Remark.1. In view o above epressions we can relate the measures o group in terms o measures o group 1 as B 1 P Q = 3D FK0 P Q B 3 P Q = 8D K0 hp Q B 4 P Q = 8D K0 P Q B 5 P Q = 16D ΨK0 P Q. The measures B P Q B 6 P Q can also be written as B P Q = 4D h P Q B 6 P Q = 16D Ψ P Q. 3 Sequences o Inequalities The epression15 admits 8 nonnegative dierences. Some o these dierences are already studied in Taneja [11]. Here we shall consider only those connected with the measures K 0 P Q ΨP Q FP Q. Based on these dierences the ollowing theorem hold: Theorem 3.1. The ollowing sequences o inequalities hold: 1 3 D 1 T 3 D K 0 1 D hi D Jh D K 0 I 3 D DK0 J K 0 h 1 1 DΨT 3 D ΨK0 1 9 D 1 F 8 D FI 15 D Fh 1 7 D FJ 1 6 or eample D T = T 1 4 D K 0 = 1 8 K etc. 6 D Ψ DFT 1 5 D ΨI 1 4 D ΨJ 1 3 D FΨ 1 D FK0 Proo. All the measures appearing in the inequalities 1 can be written as 19 we can easily check that all these dierences are conve unctions in the pair o probability distribution PQ Γ n Γ n. We shall make use o the Lemma 1. shall do each part separately. i For D T P Q D K0 P Q: For all > 0 1 let us consider the unction g T K0 = T K 0 = / / /

9 Calculating the irst order derivative o the unction g T K0 with respect to > 0 one gets g T K 0 = / / / / / / / In view o 3 we conclude that the unction g T K0 is increasing in 01 decreasing in 1. Also we have β T K0 = sup g T K0 = lim g T K0 0 1 = 1. 4 By the application o 0 with 4 we get the required result. From the above proo we observe that it suicient to get the epressions similar to 3 calculate the value o β as given in 4. For the other parts below we shall avoid all these details. We shall just write the epressions similar to 3 4. Then applying the Lemma 1. we get the required result. Throughout it is understood that > 0 1. ii For D hi P Q 1 6 D K 0 P Q: We have g hi K0 = / / / 06 3 g hi K 0 = / / / / / 3/ 53 3 β hi K0 = sup g hi K0 = lim g hi K0 0 1 = 1 6. iii For D K0 P Q 3 D K 0 IP Q: We have g K0 K 0 I = K 0 K 0 I = / / / /

10 g K 0 K 0 I = / / β K0 K 0 I = sup g K0 K 0 I = lim g K0 K I = 3. iv For D Jh P Q 1 4 D K 0 IP Q: We have g Jh K0 I = Jh K 0 I = / g Jh K 0 I = 1[ 1 1 ] 3 6 3/ β Jh K0 I = sup g Jh K0 I = lim g Jh K0 0 1 I = 1 4. v For D K0 IP Q 4 3 D K 0 hp Q: We have g K0 I K 0 h = K 0 I K 0 h = 3 6 3/ g K 0 I K 0 h = β K0 I K 0 h = sup g K0 I K 0 h = lim g K0 I K h = 4 3. vi For D K0 hp Q 3 D K 0 JP Q: We have g K0 h K 0 J = K 0 h K 0 J g K 0 h K 0 J = = 3 3 β K0 h K 0 J = sup g K0 h K 0 J = lim g K0 h K J = 3. vii For D K0 hp Q 1 4 D Ψ P Q: We have g K0 h Ψ = K 0 h Ψ =

11 g K 0 h Ψ = β K0 h Ψ = sup g K0 h Ψ = lim g K0 h 0 1 Ψ = 1 4. viii For D K0 JP Q 1 5 D ΨIP Q: We have g K0 J ΨI = K 0 J ΨI = g K 0 J ΨI = / β K0 J ΨI = sup g K0 J ΨI = lim g K0 J 0 1 ΨI = 1 5. i For D ΨJ P Q 4 3 D ΨK 0 P Q: We have g ΨJ ΨK0 = ΨJ ΨK 0 = / 65 4 g ΨJ ΨK 0 = / / 65 4 β ΨJ ΨK0 = sup g ΨJ ΨK0 = lim g ΨJ ΨK0 = For D ΨK0 P Q 1 3 D F P Q: We have g ΨK0 F = ΨK 0 F = / / 364 5/ 364 7/ 150 9/ 30 11/ g ΨK 0 F = / 364 5/ 364 7/ 150 9/ 30 11/ / 586 5/ 586 7/ 05 9/ 5 11/ 11

12 β ΨK0 F = sup g ΨK0 F = lim g ΨK0 0 1 F = 1 3. i For D ΨT P Q 3 8 D FIP Q: We have g ΨT FI = ΨT FI = / / / g ΨT 8 1 FI = / / / / / / β ΨT FI = sup g ΨT FI = lim g ΨT FI = ii For D F P Q 9 8 D FIP Q: We have g F FI = F FI = / 364 5/ 364 7/ 150 9/ 30 11/ / / / g F FI = 3 3/ / / / / / / / β F FI = sup g F FI = lim g F FI = iii For D FI P Q D FhP Q: We have g FI Fh = FI Fh = / / /

13 g FI Fh = β FI Fh = sup g FI Fh = lim g FI Fh = iv For D Fh P Q D FJP Q: We have g Fh FJ = Fh FJ = / / g Fh FJ = / / β Fh FJ = sup g Fh FJ = lim g Fh FJ = v For D FJ P Q 7 6 D FTP Q: We have 15 g FJ FT = FJ / 45 FT = 44 3/ / / / g FJ 8 1 FT = / / / / / / β FJ FT = sup g FJ FT = lim g FJ FT = vi For D FJ P Q 7 6 D FK 0 P Q: We have / 45 g FJ FK0 = FJ FK 0 = 44 3/ / 65 g FJ 68 3/ FK 0 =

14 β FJ FK0 = sup g FJ FK0 = lim g FJ FK0 = vii For D FT P Q D FΨ P Q: We have / / g FT FΨ = FT FΨ = / / / g FT 8 1 FΨ = / / / / / / / β FT FΨ = sup g FT FΨ = lim g FT FΨ =. 0 1 viii For D FK0 P Q D FΨ P Q: We have g FK0 FΨ = FK 0 FΨ = / / g FK 0 FΨ = / / / / / β FK0 FΨ = sup g FK0 FΨ = lim g FK0 0 1 FΨ =. Combining the parts i-i we complete the proo o the theorem. 3.1 Uniied Inequalities In [11] the author studied the ollowing inequalities based on the irst part o epression 15: D I 3 D h 1 D J 1 3 D T D TJ 3 D Th D Jh 1 6 D Ψ 1 5 D ΨI 9 D Ψh 1 4 D ΨJ 1 3 D ΨT. 5 14

15 3 D h D hi D TJ. 6 Combining the above inequalities given in we get ollowing uniied result: D I 3 D h 1 1 D J 3 D 1 T 3 D K 0 D hi D TJ 3 D Th D Jh 1 D K 0 I 3 D DK0 J K 0 h 1 6 D 1 Ψ 5 D ΨI 9 D Ψh 1 4 D ΨJ 1 DΨT 3 D ΨK0 1 9 D 1 F 8 D FI 15 D Fh 1 7 D FJ 1 DFT D FΨ. 7 4 Inequalities among New Divergence Measures D FK0 From the inequalities appearing in 4 we observe that the measures 6-11 bear the ollowing relation i.e. B P Q 1 4 B 4P Q 1 B 3P Q 1 16 B 6P Q 1 B 5P Q 1 4 B 1P Q D h P Q 1 D K 0 P Q D K0 hp Q 1 4 D Ψ P Q 1 D ΨK 0 P Q 1 4 D FK 0 P Q. 8 The epression 8 admits 15 nonnegative dierences. These are as ollows: L 1 P Q = L P Q = 1 L 3P Q = 1 D K 0 P Q D h P Q 6 p i = 1 16 L 4 P Q = 1 4 D Ψ P Q D K0 hp Q = 1 64 L 5 P Q = 1 4 D Ψ P Q 1 D K 0 P Q = p i p i p i 4 p i p i L 6 P Q = 1 4 D Ψ P Q D h P Q = 1 6 p i 64 p i p i L 7 P Q = 1 D ΨK 0 P Q 1 4 D Ψ P Q = 1 64 [p i ] p i p i p i 15

16 L 8 P Q = 1 D ΨK 0 P Q D K0 hp Q = 1 16 p i 4 p i L 9 P Q = 1 D ΨK 0 P Q 1 D K 0 P Q = 1 16 p i [ ] p i p i L 10 P Q = 1 D ΨK 0 P Q D h P Q [ = 1 p i ] 4 16 p i p i L 11 P Q = 1 4 D FK 0 P Q 1 D ΨK 0 P Q = 1 3 p i p i p 3/ i q 3/ i L 1 P Q = 1 4 D FK 0 P Q 1 4 D Ψ P Q = 1 64 [p i p i ] p i 4 p 3/ i q 3/ i p i L 13 P Q = 1 4 D FK 0 P Q D K0 hp Q = 1 64 p i p i 4 p i 4 p 3/ i q 3/ i p i L 14 P Q = 1 4 D FK 0 P Q 1 D K 0 P Q = 1 [ p i qi p i p i ] 4 3 p 3/ i q 3/ i p i L 15 P Q = 1 4 D FK 0 P Q D h P Q = p 3/ i q 3/ i p i [ p 3 i q3 i 4 p i p i qi 7 p i ]. Theorem 4.1. The ollowing inequalities hold: L 1 P Q 1 L 6P Q D K0 TP Q L 5 P Q. 9 16

17 L 8 P Q 1 3 L 7 P Q L 8 P Q L 9 P Q L 1 P Q 1 L 13 P Q L 11P Q L 14 P Q 1 L 11P Q 1 3 L 15 P Q.. 30 Proo. We shall prove the above theorem in parts. Following the similar lines o part i o Theorem 3.1 it is suicient to write in each case the epressions similar to -4. The rest part o the proo ollows by the application o Lemma 1.. i For L 1 P Q 1 6 L 6P Q: We have / 5 g L1 L 6 = L 1 6 = 40 3/ / / / g L L 6 = / / / / / / 78 1 β L1 L 6 = sup g L1 L 6 = lim g L1 L 6 = ii For L 1 P Q 3 D K 0 TP Q: We have g L1 D K0 T = L 1 D K0 T = / / / 35 g L 1 D = 40 3/ K0 T iii For K 0 T: We have β L1 D = sup K0 g T L1 D K0 = lim g T L 1 D = K0 T. g DK0 T L 5 = K 0 T 5 = / / 3 3 3/

18 g D K0 T L 5 = 4 5 7/ / 3 3 3/ / β DK0 T L 5 = sup g DK0 T L 5 = lim g DK0 0 1 T L 5 = 3. Combining the parts i-iii we get the proo o 8. iv For L 7 P Q L 8 P Q: We have g L7 L 8 = L 7 8 = / / / / / g L L 8 = / 5 6 9/ / / / β L7 L 8 = sup g L7 L 8 = lim g L7 L 8 = v For L 7 P Q L 9 P Q: We have g L7 L 9 = L 7 9 = / / / / / / / g L L 9 = / / / / / / / / β L7 L 9 = sup g L7 L 9 = lim g L7 L 9 =

19 vi For L 8 P Q 1 3 L 1P Q: We have g L8 L 1 = L 8 1 = 8 3/ g L L 1 = / / / / / / / 15 5 β L8 L 1 = sup g L8 L 1 = lim g L8 L 1 = vii For L 8 P Q 1 3 L 13P Q: We have g L8 L 13 = L 8 13 = 8 3/ g L L 13 = / / 0 7 3/ 35 5 β L8 L 13 = sup g L8 L 13 = lim g L8 L 13 = viii For L 1 P Q 3 L 11P Q: We have g L1 L 11 = L 1 11 = / / g L L 11 = / / / / / / / / /

20 β L1 L 11 = sup g L1 L 11 = lim g L1 L 11 = i For L 13 P Q 3 L 11P Q: We have g L13 L 11 = L = / / / / g L 0 7 3/ L 11 = / / For L 8 P Q 1 3 L 14P Q: We have β L13 L 11 = sup g L13 L 11 = lim g L13 L 11 = g L8 L 14 = L 8 14 = 8 3/ / / / / / g L 8 L 14 = / / / / / / / / / / / / 15 5 β L8 L 14 = sup g L8 L 14 = lim g L8 L 14 = i For L 8 P Q 1 3 L 15P Q: We have g L8 L 15 = L 8 15 = 8 3/ / / / / /

21 g L 8 L 15 = / / / / / / / / / / / / β L8 L 15 = sup g L8 L 15 = lim g L8 L 15 = ii For L 9 P Q 1 3 L 14P Q: We hav / / g L9 L 14 = L 9 14 = / / / / / / g L 9 L 14 = / / / / / / / / / / / β L9 L 14 = sup g L9 L 14 = lim g L9 L 14 = iii For L 9 P Q 1 3 L 15P Q: We have / / g L9 L 15 = L 9 15 = / / / / / /

22 g L 9 L 15 = / / / / / / / / / / / / β L9 L 15 = sup g L9 L 15 = lim g L9 L 15 = iv For L 14 P Q 3 L 11P Q: We have g L14 L 11 = L = / / / / / / / g L L 11 = / / / / / / / / β L14 L 11 = sup g L14 L 11 = lim g L14 L 11 = Combining the parts iii-iv we get the proo o b. Finally the parts i-iv completes the proo o the theorem. 4.1 Relationships with the Terms o Eponential Divergence Series In this section we shall relate the irst our terms o the series 17. Theorem 4.. The ollowing inequalities hold L 5 P Q K P Q 31 L 4 P Q K 3P Q 3

23 Proo. We shall use the same arguments as o Theorem 3.1 to prove this theorem. i For L 5 P Q K P Q: We have 3/ 4 5 7/ / g L5 D K = L 5 D K = 3 3 3/ g L D K = / / / / / / β L5 D K = sup g L5 D K = lim g L5 D 0 1 K = ii For L 4 P Q K 3P Q: We have g L4 D K3 = L 4 D K3 = 5/ 3/ g L 3 3/ 1 4 D K3 = / / / / β L4 D K3 = sup g L4 D K3 = lim g L4 D 0 1 K3 = Remark 4.1. Ater simpliications we have ollowing relations with the irst our terms o the eponential divergence series: 1 FP Q = K 0P Q 1 4 K 1P Q ΨP Q P Q K 0P Q 1 18 K P Q 8TP Q 3 56 K P Q 34 3

24 1 ΨP Q3hP Q P Q4K 0P Q K 3P Q 5K 0 P Q K 3P Q. 35 The epression 33 relates the measures K 0 P Q K 1 P Q. The epression 34 relates K 0 P Q K P Q the epression 35 relates K 0 P Q K 3 P Q with the other known measures. Reerences [1] J. BURBEA C.R. RAO On the conveity o some divergence measures based on entropy unctions IEEE Trans. on Inorm. Theory IT [] I. CSISZÁR Inormation type measures o dierences o probability distribution indirect observations Studia Math. Hungarica [3] S. S. DRAGOMIR J. SUNDE C. BUŞE New inequalities or jereys divergence measure Tamsui Oord Journal o Mathematical Sciences [4] K.C. JAIN AND A. SRIVASTAVA On Symmetric Inormation Divergence Measures o Csiszars - Divergence Class Journal o Applied Mathematics Statistics Inormatics JAMSI [5] H. JEFFREYS An invariant orm or the prior probability in estimation problems Proc. Roy. Soc. Lon. Ser. A [6] S. KULLBACK R.A. LEIBLER On inormation suiciency Ann. Math. Statist [7] P. KUMAR A. JOHNSON On A Symmetric Divergence Measure Inormation Inequalities Journal o Inequalities in Pure Appl. Math [8] A. SACHLAS T. PAPAIOANNOU Jensen s Dierence without probability vectors actuarial applications Applied Mathematics & Inormation Sciences [9] I.J. TANEJA New developments in generalized inormation measures Chapter in: Advances in Imaging Electron Physics Ed. P.W. Hawkes [10] I.J. TANEJA On symmetric non-symmetric divergence measures their generalizations Chapter in: Advances in Imaging Electron Physics Ed. P.W. Hawkes [11] I.J. TANEJA Reinement Inequalities Among Symmetric Divergence Measures The Australian Journal o Mathematical Analysis Applications 1005 Art. 8 pp [1] I.J. TANEJA P. KUMAR Relative Inormation o Type s Csiszar s -Divergence Inormation Inequalities Inormation Sciences

Nested Inequalities Among Divergence Measures

Appl Math Inf Sci 7, No, 49-7 0 49 Applied Mathematics & Information Sciences An International Journal c 0 NSP Natural Sciences Publishing Cor Nested Inequalities Among Divergence Measures Inder J Taneja