54 D. S. HOODA AND U. S. BHAKER Belis and Guiasu [2] observed that a source is not completely specied by the probability distribution P over the sourc

Size: px

Start display at page:

Download "54 D. S. HOODA AND U. S. BHAKER Belis and Guiasu [2] observed that a source is not completely specied by the probability distribution P over the sourc"

Victoria Dickerson
6 years ago
Views:

1 SOOCHOW JOURNAL OF MATHEMATICS Volume 23, No. 1, pp , January 1997 A GENERALIZED `USEFUL' INFORMATION MEASURE AND CODING THEOREMS BY D. S. HOODA AND U. S. BHAKER Abstract. In the present communication a generalized cost function of utilities and lengths of output code words by a memoryless source is dened and its lower and upper bounds in terms of a generalized `useful' information measure of order and type are obtained. Its asymptotic behaviour with reference to the problem of encoding source blocks of increasing lengths is also studied. 1. Introduction Let a nite set of N source symbols X =(x 1 x 2 ::: x N ) be encoded using alphabet of D symbols then it has been shown (Feinstein [4]) that there is a uniquely decipherable/instantaneous code with lengths n 1 n 2 ::: n N if and only if the following Kraft inequality [8] is satised: D ;n i 1: (1.1) If L = P N p i n i be the average codeword length, then for a code which satises (1.1), it has been shown ([4]) that L H(P ) (1.2) with equality if and only if n i = ; log p i, for i =1 2 ::: N: This is Shannon's coding theorem for a noiseless channel. Received March 21, AMS Subject Classication. 94A15, 94A24. Key words. Kraft inequality, instantaneous code, mean codeword length, Holder's inequality and code sequence. 53

2 54 D. S. HOODA AND U. S. BHAKER Belis and Guiasu [2] observed that a source is not completely specied by the probability distribution P over the source alphabet X, in the absence of qualitative character. So it can also be assumed that the source alphabet letters are weighted according to their importance or utilities. Let U =(u 1 u 2 ::: u N ) be the set of positive real numbers, where u i is the utility of the outcome x i. The utility u i, in general, is independent of p i, the probability of encoding of source symbol x i. The information source is thus given by S = x 1 x 2 ::: x N p 1 p 2 ::: p N u 1 u 2 ::: u N u i > 0 0 <p i 1 p i =1: (1.3) Belis and Guiasu [2] introduced the \quantitative-qualitative" measure of information H(P U) =; u i p i log p i (1.4) which can be taken as a measure for the average quantity of `valuable' or `useful' information provided by the information source (1.3). Guiasu and Picard [5] considered the problem of encoding the letter output by the source (1.3) by means of a single letter prex code whose codewords w 1 w 2 ::: w N, are of lengths n 1 n 2 ::: n N respectively and satisfy the Kraft's inequality (1.1). They introduced the following `useful' mean length of the code : L u = P N u i p i n i PN u j p j : (1.5) Further they derived a lower bound for (1.5). However, Longo [9] interpreted (1.5) as the average transmission cost of the letters x i and derived the bounds for this cost function. Taneja, Hooda and Tuteja [12] dened the `useful' average code lengths of order t as given below: L u(t) = 1 t log D h P N u i p i D tn i u j p j i (0 <t<1): (1.6)

3 `USEFUL' INFORMATION MEASURE AND CODING THEOREMS 55 Evidently, when t! 0, (1.6) reduces to (1.5). They derived the bounds for the cost function (1.6) in terms of a generalized ` useful' information measure of order, given by under the condition H (P U) = 1 P N 1 ; log u i p i D 6= 1 >0 (1.7) u j p j u i D ;n i u i p i : (1.8) Actually, inequality (1.8) is a generalization of the Kraft's inequality (1.1) as when u i =1for each i, (1.8) reduces to (1.1). A code satisfying the generalized Kraft's inequality (1.8) is termed as `useful code'. It may be seen that (1.6) satises the additive property as follows: L(P Q U V N + M t) =L(P U N t)+l(q V M t): (1.9) They also obtained the following result regarding upper bound of the `useful' average code length of order t H (P U) L u(t) <H (P U)+1 where = 1 0 <t<1: (1.10) t A Measure of the Generalized Cost It is assumed that the cost is a linear function of code length. However, there are some instances when the cost does not vary linearly with code lengths but it is more nearly an exponential function of n i 's. Such types of functions occur frequently in market equilibrium and growth models in economics. Since linear dependence is the limiting case of exponential function. Therefore it is interesting to minimize the more generalized quantity. C = (u i p i ) D tn i (2.1) where t and are some parameters related to the cost. In order to make the result of the present paper more comparable with the usual noiseless coding theorem, instead of minimizing (2.1), we minimize

4 56 D. S. HOODA AND U. S. BHAKER L u(t) = 1 t log D P N (u i p i ) D tn i 0 <t<1 (2.2) which is a monotonic function of C. We dene (2.2) as the `useful' average code length of order t and type. Clearly, if = 1,(2.2) reduces to (1.6) which further reduces to (1.5) when t! 0. We may also note that (2.2) is a monotonic non-decreasing function of t and if all the n i 's are the same, say, n i = n for each i, then L u(t) = n. This is an important property for any measure of length to possesss. It is additive in analogous to (1.9). Now we derivethelower and upper bounds of the cost function (2.2) in terms of the following `useful' information measure of order and type, which was dened and studied by Hooda and Singh [6] H(P U) = 1 P N 1 ; log u i p+;1 i D 6= 1 > 0 (2.3) under the condition u i p;1 i D ;n i (u i p i ) : (2.4) It may be seen that in case = 1, (2.4) reduces to (1.8) and further when u i =1 for each i, it reduces to Kraft's inequality (1.1). It may also be noted that (2.3) reduces to (1.7) if =1and further reduces to Renyi's entropy [10] of order when utilities are ignored. If we consider u i =1foreach i, then (2.3) reduces H (P U) = 1 1 ; log D P N p +;1 i p 6= 1 > 0 j which is the entropy of order and type characterized by Aczel and Daroczy [1] and Kapur [7]. 3. Coding Theorem for Useful Codes We rst prove the following lemma: Lemma 1. Let fu i g N, fp ig N and fn ig N satisfy the inequality (2:4), then L u(t) H (P U) where = 1 t+1 : (3.1)

5 `USEFUL' INFORMATION MEASURE AND CODING THEOREMS 57 Proof. Holder's inequality, we have ( x p i )1=p ( where 1 p + 1 q =1,p<1andx i, y i > 0. Let p = ;t, x i = y q i )1=q h i (u i p i ) P ; 1 t N D ;n (u jp j ) i where 0 <t<1 h u q =1; and y i = Putting these values in (3:2), we have h P N (u i p i ) D tn i i i p+;1 i P 1; 1 N : i ; 1 t h P N u i i p+;1 i P 1; 1 N x i y i (3.2) P N u i p;1 i D ;n i : (3.3) Using (2.4) and taking logrithm of both sides of (3.3), we get (3.1). It may be shown that there is equality in (3.1) if D ;n i = p i P N u i p+;1 i or n i = ; log D p i +log D [ By putting = 1 in (3.1), we have 1 t log D. PN u i p+;1 i = P N u i p i D tn i u j p j 1 1 ; log D ]: P N u i p i u j p j (3.4) which is a result obtained by Taneja, Hooda and Tuteja [12]. Further, if we ignore the utilities i.e. u i =1foreach i, then (3.4) reduces to 1 t log D( a result obtained by Campbell [3]. p i D nit ) 1 1 ; log D( p i ) Next, we prove a theorem giving the upper bound to the `useful' average code length of order t.

6 58 D. S. HOODA AND U. S. BHAKER Theorem 1. By properly choosing the lengths n 1 n 2 ::: n N in the code of Lemma 1, L u(t) where = 1 t+1 : can be madeto satisfy the following inequality: H (P U) L u(t) H (P U)+1 (3.5) Proof. Let us suppose the codewords lengths n i as the integers satisfying P N u i ; log D p i + log p+;1 i D (u i p i ) From the left inequality of (3.6), we have P N u i n i <; log D p i + log p+;1 i D +1: (3.6) D ;n i p i P N u i p+;1 i. PN i =1 2 N which gives (1.1) when = 1. It proves that there exists a `useful' code with lengths n i for each i. From (3.6), we have p ;t i [ u i p+;1 i = ] t D n it <D t p ;t i [ u i p+;1 i = ] t : (3.7) Multiplying (3.7) by (u i p i ) P N =, summing over i, raising to the power 1=t, taking logrithm and using the relation = 1 1+t, we get (3.5). Hence Theorem 1 is proved. to Remark. When t! 0 (or! 1) and =1,the inequality (3.5) reduces H(P U) u log D L H(P U) u < +1 (3.8) u log D where L u is the average cost function (1.5) dened by Guiasu and Picard [5] and u = P N u j p j. Longo [9] obtained the lower and upper bounds on L u as given below: H(P U) ; u log u + u log u u log D L u < H(P U) ; u log u + u log u u log D +1 (3.9) where the bar means the mean value with respect to the probability distribution P = f(p 1 p 2 p N ) p i 0 and P N p i =1g:

7 `USEFUL' INFORMATION MEASURE AND CODING THEOREMS 59 Since x log x is a convex U function and u log u u log u holds, therefore H(P U) does not seem to be less basic in (3.9) in comparison to it is in (3.8). 4. Encoding for Sequences Now we consider a typical sequence of length M, each symbol x i is generated by the probability distribution P =(p 1 p 2 p N ) the utility distribution U = (u i u 2 u N ), u i > 0. Let s =[x i1 x i2 x im ] denotes the block sequence of length M. Next, we dene the probability p(s) and the utility u(s) for the block sequence s. Since x ij,j = 1 2 M are the outputs by the memoryless source (1.3), therefore these are independent for unique decipherable codes. Thus we dene p(s), the probability of s, as p(s) = MX p ij : (4.1) In general, we dene the utility of a source sequence either as the sum of utilities of its letters refer to Longo [9] or as the sum of utilities of its letters divided by its block length, Sgarro [11]. However, in each case the utility will be a monotonic funcion of the number of joint experiments. We dene u(s), the utility of the source sequence s, as where u ij is the utility ofletterx ij. u(s) = MX u ij (4.2) Let n(s) be the length of the code sequence for s in `useful' code and let the code length of order t and type for the M sequence be L u(t) (s) =1 t log D Ps[u(s)p(s)] D tn(s) Ps[u(s)p(s)] 0 <t<1 (4.3) where P s extends over N M M-sequences s. The generalized measure of useful information of order and type for this product sequence is H[P (s) U(s)] = 1 Ps 1 ; log u (s)p(s) +;1 D Ps[u(s)p(s)] 6= 1: (4.4)

8 60 D. S. HOODA AND U. S. BHAKER Using (4.1) and (4.2), we have Ps u (s)p(s) +;1 Ps[u(s)p(s)] = which implies that h u i p+;1 i. ihp N p +;1 i p j i M;1 H [P (s) U(s)] = H (P U)+(M ; 1)H (p) (4.5) where H (P )= 1 1 ; log D P N p +;1 i p 6= 1 >0 j is the generalized measure of entropy studied by Kapur [7]. Let n(s) betheinteger satisfying the inequality P s ; log D p(s)+log u (s)p +;1 (s) D P s u (s)p (s) P s n(s) < ; log D p(s) + log u (s)p +;1 (s) D P s u (s)p +1: (4.6) (s) From the left inequality of (4.6) we have D ;n(s) p (s) Ps u (s)p +;1 (s)= P s u (s)p (s) : It implies that there exists a ` useful' code with length n(s) which satises (4.6). We can re-write (4.6) as follows p ;t (s) h P s u (s)p +;1 (s) P s u (s)p (s) i t D tn(s) D t p (s)h P ;t s u (s)p +;1 (s) i t: P s u (s)p (s) (4.7) Multiplying (4.7) by [u(s)p(s)] = P s[u(s)p(s)], summing over all s, raising to the power 1=t, taking logrithm and using the relation = 1 t+1 weget H [P (s) U(s)] L u(t) (s) <H [P (s) U(s)] + 1: (4.8) Now (4.8) together with (4.5) gives H (P )+ 1 M [H (P U);H (P )] L u(t) (s) M <H (P )+ 1 M [H (P U);H (P )+1]

9 `USEFUL' INFORMATION MEASURE AND CODING THEOREMS 61 which can be written as H (P )+ 1 M 1 L u(t) (s) M <H (p)+ 1 M 2 (4.9) where 1 = H (P U) ; H (P ), 2 = H (P U) ; H (P )+1: The quantity L u(t) (s)=m may be called the `useful' mean length of order t and type per source letter. If the given system 1 is nite then, 2 is also nite and the average code length per source letter tends to H (P )whenm!1. In case =1,(4.9) reduces to a result obtained by Campbell [3]. In this way we have proved the following theorem: Theorem 2. Given a discrete memoryless source with additional parameter u i, there exists a sequence of `useful' codes for the M -length source sequences whose weighted mean length of order t and type per source letter tends to H (P ) where = 1 t +1 : Note 1. It may be seen that if M is very large, the exibility provided by the utilities disappears, since their inuence dies o. On the other hand if M is very large, one can suspect that there are rather more losses due to ineciency of single letter coding. Thus there must be an immediate optimum length for the blocks at which the function L u(t) (s)=m has a minimum value. Note 2. Since each sequence has M-symbols for each i, it follows that a better measure is L = L u(t) (s)=m. Thus for a suciently large M-sequence of code, we can bring the average codeword length L as close as we please to the generalized weighted entropy of order and type. References [1] J. Aczel and Z. Daroczy, Uber Verallegemeinerte Mittelveste, die mit grewinebtsfunktionen jebildet Sind, Pub. Math. Debreun, 10 (1963), [2] M. Belis and S. Guiasu, A quantitative-qualitative measure of information in cybernetic systems, IEEE Trans. Information Theory, 14 (1968), [3] L. L. Campbell, A coding theorem and Renyi's Entropy, Information and Control, 8 (1965), [4] A. Feinstein, Foundations of Information Theory, McGraw-Hill, New York.

10 62 D. S. HOODA AND U. S. BHAKER [5] S. Guiasu and C. F. Picard, Borne Inferieure de la Longueur de Certain Codes, C. R. Acad. Sci. Paris, 273 (1971), [6] D. S. Hooda and U. Singh, On ` useful' information generating functions, Statistica, XL VI :4 (1986), [7] J. N. Kapur, Generalized entropy of order and type, Maths. Seminar, Delhi, 4 (1967). [8] J. G. Kraft, A device for quantizing, grouping and coding amplitude modulated pulses, M. S. Thesis Electrical Engineering Department, MIT, [9] G. Longo, A noiseless coding theorem for sources having utilities, SIAM J. Appl. Math., 30 (1976), [10] A. Renyi, On measure of entropy and information, Proceeding 4th Berkeley Symp. on Math. Stat. and Probability, University of California Press, 1996, [11] A. Sgarro, Noiseless block-coding of `useful' information, Elektronische Information -Sverabeitung and Kybernetik EIK, 15 (1979), [12] H. C. Taneja, D. S. Hooda, and R. K. Tuteja, Coding theorem on a generalized `useful' information, Soochow Journal of Mathematics, 11 (1985), Department of Mathematics and Statistics, CCS Haryana Agricultural, University Hisar , India. Department of Mathematics, Govt. (P.G.) College, Bhiwani , India.

New Generalized Entropy Measure and its Corresponding Code-word Length and Their Characterizations

New Generalized Entropy Measure and its Corresponding Code-word Length and Their Characterizations Ashiq Hussain Bhat 1, Dr. M. A. K. Baig 2, Dr. Muzafar Hussain Dar 3 1,2 Post Graduate Department of Statistics