SOME PROPERTIES OF ENTROPY OF ORDER a AND TYPE P

Size: px

Start display at page:

Download "SOME PROPERTIES OF ENTROPY OF ORDER a AND TYPE P"

Christopher Dawson
5 years ago
Views:

1 SOME PROPERTIES OF ETROPY OF ORDER a AD TYPE P BY J.. KAPUR, F.A.Sc. (Indian Institute of Technology, Kanpur) Received March 10, 1967 ABSTRACT In a recent paper, 3 we defined entropy of order a and type P. For a = 1, it reduces to Renyi's 2 entropy of order a, while for a = 1, $= 1 and for a complete probability distribution, it reduces to Shannon's definition of entropy. In this paper we have studied some properties of this generalised entropy. We have also discussed the variability of Renyi's entropy in the continuous case with co-ordinate systems and invariance of transinformation of order a and type 8 for linear transformations. LET P = (Pr, P2'..., Px) 1. DEFIITIOS be a generalised probability distribution, so that then p 0, (i=1,2,...); 0< Ep ^1, (1) (i) Shannon's' entropy is defined as {6j H (P) _ -- E Pa log2pi ; 7 Pt = 1 (2) (ii) Renyi's entropy of order a is defined as 1 E Pi ' Ha (P) = logz ; 0< L' Pi < 1; a >0 a+1 (3) 1 a 2JPt ia2 A3 201

2 202 J.. KAPUR (iii) Renyi's 2 entropy of order 1 is defined as the limit of H. (P) as a 1 so that Pi loge Pi Hl (P)= ''1 0< ^7Pi< 1 (4) 2p 4-1 (iv) Ours entropy of order a and type 13 is given by 'i Pia +t-1 HP (P) = 1 1 a log e f --, 0< E Pi < 1, a ^ 1, (=1 c=i j3>0, a+9>1, (5) (v) Ours entropy of order '1 and type 14 is obtained as a limiting case of HI (P) as a * 1 so that H,fl (P) _ ` 1 E Pi ogs Pi E Pifl r ^ (6) The above entropies are defined for the discrete case. For the continuous case: (vi) Shannon'sl entropy is given by H (f) _ f.f (x) logf (x) dx;.f.f (x) dx = 1, (7) a a where _, '(x) is the density function of the random variate x which can lie between a and b. (vii) Renyi's4 entropy of order a is given by b x)adx f(f() Ha (.f) = 1 log b ---- ' f.f (x) dx < 1; a>0, a ^ l. 1 a f.f(x)dx a a (8)

3 Some Properties of Entropy of Order a and Type /3 203 (viii)renyi's4 entropy of order 1 is given by b f f(x) log f (x) dx b Hl (f) = b ; 0 < f f (x) dx < 1 (9) 51(x) dx (ix) Our4 entropy of order a and type /3 is given by f (f (x))a +P-1 dx Has (f) = 11 log" a ; a7 1; i9>0 a+ #> 1 f (.f (x)) gx a (10) (x) Our4 entropy of order 1 and type /3 is given by f (f (x)) glog f (x) dx Hip(f)_ b 0< f.f(x)dx<1. (11) f (1(x)) Pdx a The properties of Shannon's entropy are well known. It is proposed to give below similar properties of Renyi's and our entropies. 2. PROPERTIES OF REYI's ETROPY (i) Ha (pl, P2, ' ' ' I P) is a symmetric function of its arguments. (ii) H. (p,, P2,, P) is a continuous function of its arguments. (iii) Ha (P*Q) = Ha (P) + Ha (Q), (12) where P and Q are independent distributions. W (P) g (Ha (P)) + W (Q) g (Ha (Q)) (iv) g (H" (PUQ)) W (P) -F W (Q) (13) where PUQ = (Pi, P2,., P; q1, 92,..., 4M) (14) W (P) = 2 pi; W (Q) _ ' 43 (15 a)

4 204 J.. KAPUR 0<W(P)<1; 0<W(Q)<1; 0 <W(P)±W(Q)<1 (15b) 2(1...)x g (x))= (16) (v) Hs {(I)} = 1 (17) where {1} refers to the case when = 1 and the corresponding probability is 1. These five were used by Renyi as his postulates. We give below some other properties: (vi) Hs (P, p,.., P) = log2 p, (18) so that in the case of equal probabilities, H o, is independent of a. (vii) For a generalised probability distribution with weight K, and K K K so that all the probabilities are equal, H. (' '. ) = log K- = log (19) In this case Ha (P) is a monotonic increasing function of. (viii) If we denote this function by F 4 (), then Lt Fs = 0, Lt (Fa () Fa ( 1)) = 0. (20) x-^oe AI -"M E Pk (ix) H Q (P) = 0 =) 1 a loge R = 0 2,` Pk 7Pk6 = EPk Tint x=x _) E Pk (l Pk'-1) = 0. (21) keg If a> 1, all terms are > 0, If a < 1, all terms are < 0. In both cases Ha (P) = 0 implies that there is exactly one pt equal to unity and the rest are all zero.

5 Some Properties of Entropy of Order a and Type i3 205 Thus entropy of order a cannot vanish for an incomplete probability distribution. For a complete probability distribution, it Can - vanish if and only if one of the probabilities is unity and the rest are zero. (x) Ha (pl, Ps, ' P 0) = H e (Pie P2..., Px) (22) so that entropy of order a is not changed by adding an impossible event. (xi) He (P) is a monotonic decreasing function3 of a. Its maximum value is log (/W (P)) and its minimum value is log PM where pu is the maximum probability. This shows that He (P) > 0. (xii) Let pi < p j and let pf change to (pi + x) and pi change to (p3 -- x), [0<pt+x<1, 0 ^pj x^1, x>01 and let then if if.f (x) _ (Pi + X)e + (Pj X)a Pi g Pi, f(0) = 0,.f (Pj -- pt) = 0. f' (x) = a (pi + x) Q.-i a (pi x)a'' 1' (0) = a (p Pj "), 1' (Pj _` Pi) = a (Pã'' Pl') a> 1, f' (0) < 0,.f' (Pj P1) > 0. a < 1, f' (0) > 0, f' (Pj P6) <0. In the first case f (x) is negative for 0 < x < (p3 A. In the second case f (x) is positive for 0 < x < (p pi), ow Pi. + x)a + (P1 x)a + E Pk' 1 logy (x) = a ( Pi+x -f Pj x+ E Pk uw, s f (x) + E Pk 1 1 loss --- ^p (23)

6 206 J.. KAPUR If.f (x) + 2: Pka 0 (x) 0 (0) = 1 loge k1 (24) 2: P ka k=1 a > 1, f (x) < 0, 1 a < 0, 0 (x) 0 (0) > 0. If a< 1, f(x)>0, 1 a>0, c&(x) ^(0)>0. Thus in both cases, we find that replacing of pi, pj by pi + x, pj x increases entropy. Thus nearer probabilities come to one another, the greater the entropy becomes; of course, for a fixed W (P). (xiii)for a given W (P), the entropy is maximum when the probabilities are equal. (xiv)the maximum entropy is a non-increasing function of W (P). 3. PROPERTIES OF REYI's ETROPY OF ORDER UITY Most of the above properties are true for all a and they remain true in the limit when a 1, Thus (i), (iii), (v), (vi), (vii), (viii), (ix), (x), (xiii) and (xiv) are true. (ii) is true when the arguments are > 0. (iv) is true when g (x) = ax -j- b, a> 0. To see the truth of (xii), we consider.f (x) = (Pi + x) log (Pi ±x) +(p x) log (Ps x) and proceed as before. In addition the following properties are also true. (xv) H i (p, P2'..., Px-i, q1, q2,..., qm) = H 1 (P1, Pz,... Px) -f- 1 ç!! H, \px, PK (25) where del

7 Some Properties of Entropy of Order a and Type 207 The proof is straightforward Hi (pi, P2,..., P- q1, q2, '.., qm) -1 M _ f-1 _ iol Pi loge Pi + E q9 log2 qj r-1-1 2Jpi+ E qq i 1 11 Y M Pi loge Pi ` ^' qi log2 Px -I- 4 q; loge qi 1=1 }1-1 E Pi +Px i-1 P141-1 Px loge PH Hi (Pi, P2.... Px) + x q-i 2'p j PH = Hi (Pi, P2, -.. P) +W (P) Hl (Px ' P '..., P141 (xvi) H1 (q11, q12, ', glm 1 ; q21, q22,, q2m,, ', q11, q,,, qxm ) r... 9tml = H1(Pi, Ps'..' P) + E W ^P) (el Hl \ P1 ' Pi ' Pi J where gij+q{2+ - +qim,=pi This is a straight generalisation of the last result. (26) (xvii) H1(Pi, P2' P) = Hi (pi -1- P p) + Hi (W(P), W(P) (xviii) H 1 (pt, P2' ' ' ' I Pr; Pr+i' ' ' ', PT) = Hi (Pi, P2' '. ' 2 Pr, PT-i + PT Px) Pr+1 + PT+2 + * ± Px } W (P) x..., W(P)) - (27) H 1 ( Pr+i.,, Px (28) ^T+^ + "+P+q' ' P + " +PIr

8 208 J,. KAFUR 4. PROPERTIES OF OUR ETROPY OR ORDER a AD TYPE Most of the properties for H. (P) = H al (P) continue to hold for Hag (P). Thus (i), (ii) and (iii) are easily seen to be true. In (iv) W (P) and W (Q) have to be replaced respectively by Wf (p) = P1P Pxs; Wp (Q) = 41P qxp (29) (v) is true. (vi) is true and Has (p, p,, p) is independent of both a and /3. (vii) For a generalised probability distribution with given weight K. Ha (p, p,..., P) = log p = log K. (30) This is independent of a and /3 and is a monotonic increasing function of. (viii), (x), (xiii) and (xv) are true. (ix) HaP (P) = 0 =) F PiP (pi" 1) = 0. This implies that exactly one probability is unity and the rest are all zero. (xi) Ha, 9 (P) is a monotonic decreasing functions of a for a fixed P. 5. LACK OF IVARIACE OF REYI's ETROPY I THE COTIUOUS CASE WITH CO-ORDIATE SYSTEMS Let X be transformed into a new variable Y by a continuous one-to-one transformation. If the density functions of the two variates are f (x) and p (y), then p(y) _.f (x)j dy (31) I1a (P) =1 i a log f (p (Y))" dy f.p(y)dy

9 Some Properties of Entropy of Order a and Type P f (f (X))_ dx dy ^ dx dy dx, f i dx j_yjd f(x)^dy11 dxj x 00 1 r1 1x = 1 _a log 3 (32) f f (x) dx The entropy therefore changes. If however we consider a linear transformation. we get Y = AX + B, (33) Ha (p) = Ha (.f) -I- log I A I, (34) so that the change is independent of a. Also the difference in two entropies does not change so that Ha (Pi) Ha (P2) = Ha (f1) -' Ha (f2) (35) For the case of multivariate distributions (XI, xa1 (^... f(f(xi,x2,, xx.. dx1dx2,... a J (Yi, Yz> > Yx)... Xp ) s i Ha (p) = 1 a log V... f.f (x1, x2,.., X) dx,dx,... dx where -00 P (Yi, Y2,..., Y) and f (x^, x2,..., xx) are the density functions. For an orthogonal transformation, the absolute value of the Jacobian has the value unity and as such the entropy is invariant for translation and orthogonal transformations. The transinformation I (7^, Y) _ I (X) -f- H (Y) -- H (XY) (37) (36)

10 210 J.. KAPUR is also invariant for linear transformations for if then Z = AX + B, W = CY -I- D (38) I(Z, W) = H(Z)+H(W) H(ZW) = H (X) + log I A I -I- H (Y) + log I C H (XY) log I AC = H (X) + H (Y) H (XY) = I (X, Y) (39) 6. IVARIACE OF OUR TRASIFORMATIO FOR LIEAR TRASFORMATIO As above Ce 1 _f (P (y))a+p -1 dy Has (P) = 1 a log 00 f (p (y))s dy -00 = I a log 00 J (J (x s+s 1 dx (x))a+'-1 Iy f (f(x))s dx '-1 dy dx = Has (1) + log I A I, ( 40) so that the change is independent of both a and P. Also Has (P2) Hap (Pr) = Hap (f2) Has (fi) (41) For multivariate distributions, the entropy is invariant for translations and linear orthogonal transformations. For the same reason transinformation remains invariant for linear transformations. 7. ACKOWLEDGEMET The author is grateful to the referee for his useful suggestions,

11 Some Properties of Entropy of Order a and Type fi REFERECES 1. Shannon, E. C... Bell. System Tech. Journal, 1948, 27, 379, Renyi, A... Proc. 4th Berkeley Symp. Math. Statistics and Prob., , Kapur,1... The Maths. Seminar, 1967, 4, LLT./K. Maths. Research Report, 1967, o. 2.

A Gentle Introduction to Gradient Boosting. Cheng Li College of Computer and Information Science Northeastern University

A Gentle Introduction to Gradient Boosting Cheng Li chengli@ccs.neu.edu College of Computer and Information Science Northeastern University Gradient Boosting a powerful machine learning algorithm it can