New Information Measures for the Generalized Normal Distribution

Transcription

1 Information,, 3-7; doi:.339/info3 OPEN ACCESS information ISSN Article New Information Measures for the Generalized Normal Distribution Christos P. Kitsos * and Thomas L. Toulias Deartment of Mathematics, Agiou Syridonos & Palikaridi, Technological Educational Institute of Athens, Egaleo, Athens, Greece; t.toulias@teiath.gr * Author to whom corresondence should be addressed; xkitsos@teiath.gr. Received: 4 June ; in revised form: 4 August / Acceted: 5 August / Published: August Abstract: We introduce a three-arameter generalized normal distribution, which belongs to the Kotz tye distribution family, to study the generalized entroy tye measures of information. For this generalized normal, the Kullback-Leibler information is evaluated, which extends the well known result for the normal distribution, and lays an imortant role for the introduced generalized information measure. These generalized entroy tye measures of information are also evaluated and resented. Keywords: entroy ower; information measures; Kotz tye distribution; Kullback-Leibler information. Introduction The aim of this aer is to study the new entroy tye information measures introduced by Kitsos and Tavoularis [] and the multivariate hyer normal distribution defined by them. These information measures are defined, adoting a arameter, as the -moment of the score function (see Section ), where is an integer, while in rincile. One of the merits of this generalized normal distribution is that it belongs to the Kotz-tye distribution family [], i.e., it is an ellitically contoured distribution (see Section 3). Therefore it has all the characteristics and alications discussed in Baringhaus and Henze [3], Liang et al. [4] and Nadarajah [5]. The arameter information measures related to the entroy, are often crucial to the otimal design theory alications, see [6]. Moreover, it is roved that the defined generalized normal distribution rovides equality to a new generalized

2 Information, 4 information inequality (Kitsos and Tavoularis [,7]) regarding the generalized information measure as well as the generalized Shannon entroy ower (see Section 3). In rincile, the information measures are divided into three main categories: arametric (a tyical examle is Fisher s information [8]), non arametric (with Shannon s information measure being the most well known) and entroy tye [9]. The new generalized entroy tye measure of information J ( X), defined by Kitsos and Tavoularis [], is a function of density, as: From (), we obtain that J ( X) equals: J ( X ) f ( x) ln f ( x) () J f ( ) ( ) ( x ) ( ) X f x f x f ( x) f( x) () For, the measure of information J ( X) is the Fisher s information measure: f( x) J( X ) f ( x) 4 f ( x) f( x) i.e., J( X) J ( X). That is, J ( X) is a generalized Fisher s entroy tye information measure, and as the entroy, it is a function of density. Proosition.. When is a location arameter, then J ( ) J ( ). (3) Proof. Considering the arameter as a location arameter and transforming the family of densities to f ( x; ) f ( x ), the differentiation with resect to is equivalent to the differentiation with resect to x. Therefore we can rove that J( X) J ( ). Indeed, from () we have: J ( X ) f ( x ) ln f ( x ) f ( x) ln f ( x) J ( ) and the roosition has been roved. Recall that the score function is defined as: f( X; ) U ln f ( X ; ) (4) f( X; ) with EU ( ) and EU ( ) J ( ) under some regularity conditions, see Schervish [] for details. It can be easily shown that when, a, then J a( X) E( U ). Therefore J ( X) behaves as the -moment of the score function of f( X; ). The generalized ower is still the ower of the white Gaussian noise with the same entroy, see [], considering the entroy ower of a random variable. Recall that the Shannon entroy H( X) is defined as H ( X ) f ( x )ln f ( x ), see [9]. The entroy ower N ( X) is defined through H ( X) as:

3 Information, 5 H( X ) N ( X) e (5) e The definition of the entroy ower of a random variable X was introduced by Shannon in 948 [] as the indeendent and identically distributed comonents of a -dimensional white Gaussian random variable with entroy H ( X ). The generalized entroy ower N ( X) is of the form; H( ) X N ( X ) M e (6) with the normalizing factor being the aroriate generalization of ( e), i.e., M M a e π ( ) a (7) is still the ower of the white Gaussian noise with the same entroy. Trivially, with, the definition in (6) is reduced to the entroy ower, i.e., N( X) N ( X). In turn, the quantity: aears very often when we define various normalizing factors, under this line of thought. Theorem.. Generalizing the Information Inequality (Kitsos-Tavoularis []). For the variance of X, Var( X ) and the generalizing Fisher s entroy tye information measure J ( X), it holds: with M as in (7). πe X M X J Var( ) ( ) Corollary.. When then Var( X) J ( X), and the Gramer-Rao inequality ([9], Th...) holds. Proof. Indeed, as M (π e). For the above introduced generalized entroy measures of information we need a distribution to lay the "role of normal", as in the Fisher's information measure and Shannon entroy. In Kitsos and Tavoularis [] extend the normal distribution in the light of the introduced generalized information measures and the otimal function satisfying the extension of the LSI. We form the following general definition for an extension of the multivariate normal distribution, the -order generalized normal, as follows:

4 Information, 6 Definition.. The -dimensional random variable X follows the -order generalized Normal, with mean and covariance matrix, when the density function is of the form: ( ) (, ) (, ) det ex{ ( ) } KT C Q X (8) t with Q( X) ( X ), ( X ), where t t u, v is the inner roduct of uv, ) and u is the transose of u. We shall write X KT (, ). The normality factor C (, ) is defined as: C(, ) Notice that for, KT (, ) is the well known multivariate distribution. Recall that the symmetric Kotz tye multivariate distribution [] has density: m s Kotzm, r,( s, ) K( m, r, s) det Q ex{ rq } (9) where r, s, m n and the normalizing constant K( m, r, s ) is given by: K( m, r, s) s( ) r ( ) (m) s m s see also [] and []. Therefore, it can be shown that the distribution multivariate distribution for m, r and s ( ) KT (,Σ) follows from the symmetric Kotz tye, i.e., KT (, ) Kotz,( ), ( ) (, ). Also note that for the normal distribution it holds N(, ) KT (, ) Kotz,,(, ), while the normalizing factor is C(, ) K(,, ). ( ). The Kullback-Leibler Information for Order Generalized Normal Distribution Recall that the Kullback-Leibler (K-L) Information for two -variate density functions f, defined as [3]: f( x) KLI( f, g) f ( x)log gx ( ) g is The following Lemma rovides a generalization of the Kullback-Leibler information measure for the introduced generalized Normal distribution. Lemma.. The K-L information KLI of the generalized normals KT (, I ) is equal to: KT ( I, ) and C(, ) e e q x e q x q ( x) q ( x) q ( x) KLI log ( ) ( ) where qi( x) i x i, x, i,.

5 Information, 7 Proof. We have consecutively: KT (, )( x) KLI KLI( KT (, ), KT (, )) KT (, )( x)log (, )( x) KT C(, ) ex{ ( ) ( ) Q x } (, ) det ( ) ex{ Q ( x ) } log det C(, ) ( ) ex{ Q ( x) } C C(, ) det det det ( ) ( ) ( ) ex{ Q ( x) } log Q ( x) Q ( x) det and thus C(, ) det ( ) ( ) ( ) KLI log ex{ Q ( x) } ex{ Q ( x) } Q ( x) det det ( ) ( ) ex{ Q( x) } Q( x) For Σ I and I, we finally obtain: C(, ) e e q x e q x q ( x) q ( x) q ( x) KLI log ( ) ( ) ( ) where q ( x) Q ( x), x, i,. i i t Notice that the quadratic forms Q( x) ( x ), ( x ), i, can be written in the form of i( ) i,, i i Q x x i resectively, and thus the lemma has been roved. Recall now the well known multivariate K-L information measure between two multivariate normal distributions with is: KLI I log which, for the univariate case, is: ( ) KLI I log In what follows, an investigation of the K-L information measure is resented and discussed, concerning the introduced generalized normal. New results are rovided that generalize the notion of K-L information. In fact, the following Theorem. generalizes Ι ( KLI ) for the -order generalized Normal, assuming that. Various sequences of the K-L information measures

6 Information, 8 KLI are discussed in Corollary., while the generalized Fisher s information of the generalized Normal KT (, I ) is rovided in Theorem.. Theorem.. For the K-L information KLI is equal to: KLI log () Proof. We write the result from the revious Lemma. in the form of: KLI C(, ) log I I I3 () where: x ex{ } I x x, I ex{ } x x I3 ex{ } We can calculate the above integrals by writing them as: I ex{ ( ) x }, x I ex{ ( ) x } ( ) x I3 ex{ ( ) x } ( ) and then we substitute x z ( ) ( ). Thus, we get resectively: ( ) ex{ } I z dz, ( ) ex{ } I z z dz and: ( ) z I3 ( ) ex{ z } dz ( ) ( ) ex{ } z z dz ( ) () Using the known integrals:

7 Information, 9 e z dz π ( ) ( ) and (3) I and I can be calculated as: resectively. Thus, () can be written as: KLI z z π ( ) ( ) e z dz e dz π ( ) ( ) ( ) I π ( ) ( ) ( ) ( ) I I C(, ) C(, ) I log I 3 and by substitution of I from (5) and C (, ) from definition., we get: (4) and (5) ( ) π KLI log ( ) I 3 ( ) Assuming, from (), I 3 is equal to: (6) and using the known integral (4), we have: Thus, (6) finally takes the form: 3 ( ) ex{ } I z z dz I 3 ( ) π ( ) ( ) ( ) ( ) ( ) ( ) KLI log ( ) ( ) ( ) log ( ) ( ) However, ( ) ( ) and ( ) ( ), and so () has been roved. For and for from (6) we obtain: KLI log (π) I 3 (7) where, from ():

8 Information, I ex{ z } z dz ex{ z } z dz (8) 3 For we have the only case in which, and therefore, the integral of I 3 in () can be * simlified. Secifically, by setting z ( zi) i, ( i ) i and ( i ) i, from (8) we have consecutively: I z z z dz 3 ex{ } ( i i ) i i ex{ z } z dz ex{ z } dz ex{ z... z } ( ) t dz... dz i i i i Due to the known integrals (3) and (4), which for are reduced resectively to: we obtain: ex{ z } dz π and ex{ z } z dz π (π) ex{ } ( ) i i zi d zi i i (π) I3 (π) (π) ex{ i i z } zdz... ex{ z } z dz i lim ex{ } i i zi zi i i i.e., and hence: I (π)... 3 i i i times I 3 (π) Finally, using the above relationshi for I 3, (7) imlies: KLI log log and the theorem has been roved. Corollary.. For the Kullback-Leibler information KLI it holds: (i) Provided that, KLI * has a strict ascending order as dimension rises, i.e.,

9 Information, KLI KLI... KLI... (ii) Provided that and, and for a given, we have: (iii) For KLI in KLI (iv) Given, KLI KLI... KLI... KLI, we obtain KLI KLI... KLI... and, the K-L information KLI has a strict descending order as {} rises, i.e., KLI KLI 3... KLI... (v) KLI is a lower bound of all KLI for,3,... Proof. From Theorem. KLI * is a linear exression of, i.e., with a non-negative sloe: KLI log * This is because, from alying the known logarithmic inequality log x x, x (where equality holds only for x ) to, we obtain: log log and hence, where the equality holds resectively only for, i.e., only for. * Thus, as the dimension rises: KLI also rises. In the general -variate case, KLI is a linear exression of that, with a non-negative sloe: *, i.e., KLI, rovided Indeed, as: log log log

10 Information, and since, we get: log log ( ) which imlies that. The equality holds resectively only for, i.e., only for. * Thus, as the dimension rises, KLI also rises, i.e., KLI KLI..., and so, rovided that (see Figure ). Now, for given, and, if we choose, we have or. KLI KLI... Thus, i.e., KLI KLI. Consequently, KLI KLI 3... and so KLI KLI,,3,... (see Figure ). Figure. The grahs of the KLI, for dimensions,,...,, as functions of the quotient (rovided that ), where we can see that KLI KLI... Figure. The grahs of the trivariate 3 3 KLI, for,3,..., and KLI, as functions of the quotient (rovided that ), where we can see that KLI KLI 3... and KLI KLI,,3,....

11 Information, 3 The Shannon entroy of a random variable X which follows the generalized normal KT (, ) is: det H ( KT (, )) log (9) C (, ) This is due to the entroy of the symmetric Kotz tye distribution (9) and it has been calculated (see [7]) for m, s ( ), r. Theorem.. The generalized Fisher s information of the generalized Normal KT (, I ) is: ( ) ( ) ( KT (, )) ( ) J I () Proof. From: ( ) f f f ( ) f f imlies: f ( ) f f Therefore, from (), J ( X) equals eventually: J( KT (, I )) C(, ) x ex{ x } Switching to hyersherical coordinates and taking into account the value of C (, ) we have the result (see [7] for details). Corollary.. Due to Theorem., it holds that J ( KT (, I )) and N ( KT (, I )). Proof. Indeed, from (), J ( KT (, I )) and the fact that J ( X) N ( X) (which has been extensively studied under the Logarithmic Sobolev Inequalities, see [] for details and [4]), Corollary. holds. Notice that, also from (): and therefore, N ( KT (, I )). 3. Discussion and Further Analysis N ( KT (, I )) J We examine the behavior of the multivariate -order generalized Normal distribution. Using Mathematica, we roceed to the following helful calculations to analyze further the above theoretical results, see also []. a

12 Information, 4 Figure 3 reresents the univariate -order generalized Normal distribution for various values of : (normal distribution), 5,,, while Figure 4, reresents the bivariate -order generalized Normal KT (, I ) with mean and covariance matrix I. Figure 3. The univariate -order generalized Normals KT (,) for,5,,. Figure 4. The bivariate -order generalized normal KT (,). Figure 5 rovides the behavior of the generalized information measure for the bivariate generalized normal distribution, i.e., J( KT(, I )), where (, ) [,] [,]. 5 Figure 5. J( KT (, I 5)), a.

13 Information, 5 For the introduced generalized information measure of the generalized Normal distribution it holds that:. for J( KT (, I )), for, for For and, it is the tyical entroy ower for the normal distribution. The greater the number of variables involved at the multivariate -order generalized normal, the larger the generalized information ( KT J (, I )), i.e., J( KT (, I)) J( KT (, I )) for. In other words, the generalized information measure of the multivariate generalized Normal distribution is an increasing function of the number of the involved variables. Let us consider and to be constants. If we let vary, then the generalized information of the multivariate -order generalized Normal is a decreasing function of, i.e., for we have J ( KT (, I )) J ( KT (, I )) excet for. Proosition 3.. The lower bound of J ( KT (, I )) is the Fisher s entroy tye information. Proof. Letting vary and excluding the case, the generalized information of the multivariate generalized normal distribution is an increasing function of, i.e., J ( KT (, )) ( (, )) I J KT I for a a. That is the Fisher s information J( KT (, I )) is smaller than the generalized information measure J ( KT (, I )) for all, rovided that. When, the more variables involved at the multivariate generalized Normal distribution the larger the generalized entroy ower Ν ( KT (, I )), i.e., Ν ( KT (, I )) Ν ( KT (, I )) for. The dual occurs when. That is, when the number of involved variables defines an increasing generalized entroy ower, while for the generalized entroy ower is a decreasing function of the number of involved variables. Let us consider and to be constants and let vary. When the generalized entroy ower of the multivariate generalized Normal distribution Ν ( KT (, I )) is increasing in. When, the generalized entroy ower of the multivariate generalized Normal distribution Ν ( KT (, I )) is decreasing in. If we let vary and let then, for the generalized entroy ower of the multivariate generalized Normal distribution, we have Ν ( KT (, I )) Ν ( KT (, I )) for certain and, i.e., the generalized entroy ower of the multivariate generalized Normal distribution is an increasing function of. Now, letting vary, Ν ( KT (, I )) is a decreasing function of, i.e., Ν ( KT (, I )) Ν( KT (, I )) rovided that. The well known entroy ower Ν( KT (, I )) rovides an uer bound for the generalized entroy ower, i.e., Ν( KT (, I)) Ν( KT (, I )), for given and.

14 Information, 6 4. Conclusions In this aer new information measures were defined, discussed and analyzed. The introduced -order generalized Normal acts on these measures as the well known normal distribution does on the Fisher s information measure for the well known cases. The rovided comutations offer an insight analysis of these new measures, which can also be considered as the -moments of the score function. For the -order generalized Normal distribution, the Kullback-Leibler information measure was evaluated, roviding evidence that the generalization behaves well. Acknowledgements The authors would like to thank I. Vlachaki for helful discussions during the rearation of this aer. The referees comments are very much areciated. References. Kitsos, C.P.; Tavoularis, N.K. Logarithmic Sobolev inequalities for information measures. IEEE Trans. Inform. Theory 9, 55, Kotz, S. Multivariate distribution at a cross-road. In Statistical Distributions in Scientific Work; Patil, G.P., Kotz, S., Ord, J.F., Eds.; D. Reidel Publishing Comany: Dordrecht, The Netherlands, 975; Volume, Baringhaus, L.; Henze, N. Limit distributions for Mardia's measure of multivariate skewness. Ann. Statist. 99,, Liang, J.; Li, R.; Fang, H.; Fang, K.T. Testing multinormality based on low-dimensional rojection. J. Stat. Plan. Infer., 86, Nadarajah, S. The Kotz-tye distribution with alications. Statistics 3, 37, Kitsos, C.P. Otimal designs for estimating the ercentiles of the risk in multistage models of carcinogenesis. Biometr. J. 999, 4, Kitsos, C.P.; Tavoularis, N.K. New entroy tye information measures. In Proceedings of Information Technology Interfaces; Luzar-Stiffer, V., Jarec, I., Bekic, Z., Eds.; Dubrovnik, Croatia, 4 June 9; Fisher, R.A. Theory of statistical estimation. Proc. Cambridge Phil. Soc. 95,, Cover, T.M.; Thomas, J.A. Elements of Information Theory, nd ed; Wiley: New York, NY, USA, 6.. Scherrvish, J.M. Theory of Statistics; Sringer Verlag: New York, NY, USA, Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 948, 7, , Kitsos, C.P.; Tavoularis, N.K. On the hyer-multivariate normal distribution. In Proceedings of International Worksho on Alied Probabilities, Paris, France, 7 July Kullback, S.; Leibler, A. On information and sufficiency. Ann. Math. Statist. 95,,

15 Information, 7 4. Darbellay, G.A.; Vajda, I. Entroy exressions for multivariate continuous distributions. IEEE Trans. Inform. Theory, 46, by the authors; licensee MDPI, Basel, Switzerland. This article is an Oen Access article distributed under the terms and conditions of the Creative Commons Attribution license (htt://creativecommons.org/licenses/by/3./).