A Coding Theorem Connected on R-Norm Entropy

Transcription

1 Int. J. Contemp. Math. Sciences, Vol. 6, 2011, no. 17, A Coding Theorem Connected on -Norm Entropy Satish Kumar and Arun Choudhary Department of Mathematics Geeta Institute of Management & Technology Kanipla , Kurushetra, Haryana, India arunchoudhary07@gmail.com Abstract A relation between Shannon entropy and Kerridge inaccuracy, which is nown as Shannon inequality, is well nown in information theory. In this communication, first we generalized Shannon inequality and then given its application in coding theory. Mathematics Subject Classification: 94A15, 94A17, 94A24, 26D15 Keywords: Shannon inequality, Codeword length, Holder s inequality, Kraft inequality and Optimal code length 1 Introduction We consider the following set of positive real numbers : = { : >0, 1}. Let Δ n = {P =p 1,p 2,..., p n ); p 0 and p =1}. Boeee and Lubbe [4] studied -Norm entropy of the distribution P is given by H P )= n 1 p. 1.1) Actually, the -norm entropy 1.1) is a real function from Δ n, where n 2. This measure is different from the entropies of Shannon s [14], enyi s [13], Havrda and Charvat [8] and Daroczy [6]. The most interesting property of this measure is that when 1, it approaches to Shannon s [14] entropy and in case, H P ) 1 max p ), =1, 2,..., n. in 1.1), we get Setting r = 1 H r P )= 1 [ n ) r ] 1 p 1 r 1 r, r > 0 1), 1.2)

2 826 S. Kumar and A. Choudhary which is a measure mentioned by Arimoto [1] as an example of a generalized class of information measure. It may be mared that 1.2) also approaches to Shannon s [14] entropy as r 1. For P Δ n, Shannon s measure of information [14] is defined as H P )= p log D p. 1.3) The measure 1.3) has been generalized by various authors and has found applications in various disciplines such as economics, accounting, crime and physics etc. For P,Q Δ n, Kerridge [10] introduced a quantity nown as inaccuracy defined as: H P, Q) = p log D q. 1.4) There is well nown relation between HP) and HP, Q) which is given by HP) HP, Q). 1.5) The relation 1.5) is nown as Shannon inequality and its importance is well nown in coding theory. In the literature of information theory, there are many approaches to extend the relation 1.5) for other measures. Nath and Mittal [12] extended the relation 1.5) in the case of entropy of type β. Using the method of Nath and Mittal [12], Lubbe [18] generalized 1.5) in the case of enyi s entropy. On the other hand, using the method of Campbell [5], Lubbe [18] generalized 1.5) for the case of entropy of type β. Using these generalizations, coding theorems are proved by these authors for these measures. The objective of this communication is to generalize 1.5) for 1.1) and give its application in coding theory. 2 Generalization of Shannon Inequality For P, Q Δ n, we define a measure of inaccuracy, denoted by HP,Q;) as H P, Q; ) = n 1 p q 1),>0, ) Since H P, Q; ) H P ; ), we will not interpret 2.1) as a measure of inaccuracy. But H P, Q; ) is a generalization of the measure of inaccuracy

3 A coding theorem connected on -norm entropy 827 defined in 1.1). In spite of the fact that H P, Q; ) is not a measure of inaccuracy in its usual sense, its study is justified because it leads to meaningful new measures of length. In the following theorem, we will determine a relation between 1.1) and 2.1) of the type 1.5). Since 2.1) is not a measure of inaccuracy in its usual sense, we will call the generalized relation as pseudo-generalization of the Shannon inequality. Theorem 1. IfP,Q Δ n, then it holds that H P ; ) H P, Q; ) 2.2) under the condition n p 2.3) q and equality holds if q = p ;,2,...,n. Proof : a) If 0 <<1. By Holder s inequality [15] x p p n y q q x y 2.4) for all x,y > 0, i=1, 2,..., n and =1,p<1 0),q<0or p q q<1 0),p<0. We see that equality holds if and only if there exists a positive constant c such that x p = cyq. Maing the substitutions p =, q = in 2.4), we get x = p 1 q, y = p ) p q 1 1 Using the condition 2.3), we get p q ; >0, 1. p q 1 ) 1 p p ; >0, )

4 828 S. Kumar and A. Choudhary Since 0 <<1, 2.5) becomes p q 1 p 2.6) using 2.6) and the fact that 0 <<1, we get 2.2). b) If >1, the proof follows on the similar lines. 3 Application in Coding Theory We will now give an application of theorem 1 in coding theory. Let a finite set of n-input symbols with probabilities p 1,p 2,..., p n be encoded in terms of symbols taen from the alphabet {a 1,a 2,..., a n }. Then it is nown Feinstein [7] that there always exist a uniquely decipherable code with lengths N 1,N 2,..., N n iff D N ) If L = p N is the average codeword length, then for a code which satisfies 2.7), it has been shown that Feinstein [7], L H P ) 2.8) with equality iff N = log D p ; =1, 2,..., n and that by suitable encoded into words of long sequences, the average length can be made arbitrary close to H P ). This is Shannon s noiseless coding theorem. By considering enyi s [13] entropy, a coding theorem and analogous to the above noiseless coding theorem has been established by Campbell [5] and the log 1 α D p α authors obtained bounds for it in terms of H α P )= 1 ; α 1,α>0. Kieffer [11] defined a class rules and showed H α P ) is the best decision rule for deciding which of the two sources can be coded with expected cost of sequences of length n when n, where the cost of encoding a sequence is assumed to be a function of length only. Further Jeline [9] showed that coding with respect to Campbell [5] mean length is useful in minimizing the problem of buffer overflow which occurs when the source symbol are being produced at a fixed rate and the code words are stored temporarily in a finite buffer. Further, Boeee and Lubbe [4] and Lubbe [17] defined mean codeword length L P )= [ 1 p D N 1 ) ], > 0, 1 2.9)

5 A coding theorem connected on -norm entropy 829 and L P )= p 1 n, > 0, ) p DN ) and proved H P ) L P ) H P )+1 under condition 2.7). It may be seen that the mean codeword length L = p N had been generalized parametrically and their bounds had been studied in terms of generalized measures of entropies. We define the measure of length L ) by L ) = n 1 p D N ), > 0, ) Also, we have used the condition D N p 2.12) to find the bounds. when = 1, then 2.12) reduces to Kraft Inequality 2.7). Theorem ), then If N, =1, 2,..., n are the lengths of codewords satisfying H P ; ) L ) <D H P ; )+ Proof : In 2.2) choose Q =q 1,q 2,..., q n ) where ) 1 D. 2.13) q = D N 2.14) with choice of Q, 2.2) becomes H P ; α) n 1 p D N ) i.e. H P ; ) L ) which proves the first part of 2.13). The equality holds iff D N = p,,2,...,n which is equivalent to N = log D p ; =1, 2,..., n. 2.15) Choose all N such that log D p N < log D p +1.

6 830 S. Kumar and A. Choudhary Using the above relation, it follows that D N >p D ) We now have two possibilities: 1) If >1; 2.16) gives us ) p D N ) > p D 2.17) using 2.17) and the fact >1, we get right hand side in 2.13). 2) If 0 <<1, the proof follows on the same lines. Particular s Case If 1, then 2.13) becomes HP ) log D L log D <HP )+1. Which is the Shannon [14] classical noiseless coding theorem. 4 Conclusion We now that optimal code is that code for which the value L ) is equal to its lower bound. From the result of the theorem 2, it can be seen that the mean codeword length of the optimal code is dependent on parameter, while in the case of Shannon s theorem it does not depend on any parameter. So it can be reduced significantly by taing suitable values of parameter. eferences [1] S. Arimoto, Information Theoretical Considerations on Estimation Problems, Information and Control, ), [2] C. Arndt, Information Measure-Information and its description in Science and Engineering, Springer, Berlin, 2001). [3] M.A.K. Baig and ayeees Ahmad Dar, Coding theorems on a generalized information measures, J. KSIAM, 11 2) 2007), 3-8. [4] E. Boeee and J.C.A. Van Der Lubbe, The -norm Information Measure, Information and Control, ),

7 A coding theorem connected on -norm entropy 831 [5] L.L. Campbell, A coding theorem and enyi s entropy, Information and Control, ), [6] Z. Daroczy, Generalized Information Functions, Information and Control, ), [7] A. Feinstein, Foundations of Information Theory, McGraw-Hill, New Yor, [8] J.F. Havrda and F. Charvat, Quantification Methods of Classification Process, The Concept of structural α-entropy, Kybernetia, ), [9] F. Jeline, Buffer overflow in variable lengths coding of fixed rate sources, IEEE, ), [10] D. F. Kerridge, Inaccuracy and inference, J. oy. Statist Soc. Sec. B ), [11] J.C. Kieffer, Variable lengths source coding with a cost depending only on the codeword length, Information and Control, ), [12] P. Nath and D. P. Mittal, A generalization of Shannon s inequality and its application in coding theory, Inform. and Control, ), [13] A. enyi, On Measure of entropy and information, Proc. 4th Bereley Symp. Maths. Stat. Prob., ), [14] C. E. Shannon, A mathematical theory of information, Bell System Techn. J., ), , [15] O. Shisha, Inequalities, Academic Press, New Yor, [16].P. Singh,. Kumar and.k. Tuteja, Application of Holder s In equality in Information Theory, Information Sciences, ), [17] J. C. A. Van Der Lubbe, A generalized probabilistic theory of the measurement of certainty and information, Delft university Press, [18] J. C. A. Van Der Lubbe, On certain coding theorems for the information of order α and of type β, In: Trans. Eighth Prague Conf. Inform. Theory, Statist. Dec. Functions, andom Processes, Vol. C. Academia, Prague, 1978), eceived: October, 2010