On the equivalence between Stein and De Bruijn identities

Transcription

1 On the equivlence between Stein n De Bruijn ientities Sngwoo Prk, Erchin Serpein, n Khli Qrqe rxiv:.5v4 cs.it Jul Abstrct This pper focuses on illustrting the equivlence between Stein s ientity n De Bruijn s ientity, n two extensions of De Bruijn s ientity. First, it is shown tht Stein s ientity is equivlent to De Bruijn s ientity uner itive noise chnnels with specific conitions. Secon, for rbitrry but fixe input n noise istributions uner itive noise chnnels, the first erivtive of the ifferentil entropy is expresse by function of the posterior men, n the secon erivtive of the ifferentil entropy is expresse in terms of function of Fisher informtion. Severl pplictions over number of fiels such s signl processing n informtion theory, re presente to support the usefulness of the evelope results in this pper. Inex Terms Stein s ientity, De Bruijn s ientity, entropy power inequlity EPI, Cost s EPI, Fisher informtion inequlity FII, Crmér-Ro lower boun CRLB, Byesin Crmér- Ro lower boun BCRLB I. INTRODUCTION STEIN S ientity or lemm ws first estblishe in 956, n since then it hs been wiely use by mny reserchers e.g.,, 3, 4. Due to its pplictions in the Jmes-Stein estimtion technique, empiricl Byes methos, n numerous other fiels, Stein s ientity hs ttrcte lot of interest see e.g., 5, 6, 7. Recently, nother ientity, De Bruijn s ientity, hs ttrcte increse interest ue to its pplictions in estimtion n turbo itertive ecoing schemes. De Bruijn s ientity shows link between two funmentl concepts in informtion theory: entropy n Fisher informtion 8, 9,. Verú n his collbortors conucte series of stuies,, 3 to nlyze the reltionship between the input-output mutul informtion n the minimum men-squre error MMSE, result referre to s the I-MMSE ientity for itive Gussin noise chnnels, stuies which were lter extene to non- Gussin chnnels in 4, 5. Also, the equivlence between De Bruijn s ientity n I-MMSE ientity ws shown in. The min theme of this pper is to stuy how Stein s ientity Theorem is relte to De Bruijn s ientity Theorem. To compre Stein s ientity with De Bruijn s ientity, itive noise chnnels of the following form re consiere in this pper: Y X + W, where input signl X n itive noise W re rbitrry rnom vribles, X n W re inepenent of ech other, n prmeter is ssume nonnegtive. First, when itive Deprtment of Electricl n Computer Engineering, Texs A&M University, College Sttion, TX USA, e-mil: serpein@ece.tmu.eu. This work ws supporte by QNRF-NPRP grnt noise W is Gussin with zero men n unit vrince, the equivlence between the generlize Stein s ientity Theorem n De Bruijn s ientity Theorem is prove. Since the stnr-form Stein s ientity in 3 requires both rnom vribles X n W to be Gussin, inste of the stnrform Stein s ientity, the generlize version of Stein s ientity in is use. If we further ssume tht input signl X is lso Gussin, then both rnom vribles X n W re Gussin, n the output signl Y is Gussin. In this cse, not only Stein s n De Bruijn s ientities re equivlent, but lso they re equivlent to the het eqution ientity, propose in. The secon mjor question tht we will ress in this pper is how De Bruijn s ientity coul be extene. De Bruijn s ientity shows the reltionship between the ifferentil entropy n the Fisher informtion of the output signl Y uner itive Gussin noise chnnels. Therefore, uner itive non-gussin noise chnnels, we cnnot use De Bruijn s ientity. However, we will erive similr form of De Bruijn s ientity for itive non-gussin noise chnnels. Consiering itive rbitrry noise chnnels, the first erivtive of the ifferentil entropy of output signl Y will be expresse by the posterior men, while the secon erivtive of the ifferentil entropy of output signl Y will be represente by function of Fisher informtion. Even though some of these reltionships o not inclue the Fisher informtion, they still show reltionships mong bsic concepts in informtion theory n estimtion theory, n these reltionships hol for rbitrry noise chnnels. Bse on the results mentione bove, we introuce severl pplictions eling with both estimtion theoretic n informtion theoretic spects. In the estimtion theory fiel, the Fisher informtion inequlity, the Byesin Crmér-Ro lower boun BCRLB, n new lower boun for the men squre error MSE in Byesin estimtion re erive. The surprising result is tht the newly erive lower boun for MSE is tighter thn the BCRLB. The propose new boun overcomes the min rwbck of BCRLB, i.e., its looseness in the low Signlto-Noise Rtio SNR regime, since it provies tighter boun thn BCRLB especilly t low SNRs. Even though some of the propose pplictions hve lrey been prove before, in this pper we show not only lterntive wys to prove them, but lso new reltionships mong them. In the informtion theory relm, Cost s entropy power inequlity- previously prove in 6, 7- is erive in two ifferent wys bse on our results. Both propose methos show novel, simple, n lterntive wys to prove Cost s entropy power inequlity. Finlly, pplictions in other res re briefly mentione. The rest of this pper is orgnize s follows. Vrious

2 reltionships between Stein s ientity n De Bruijn s ientity re estblishe in Section III. Some extensions of De Bruijn s ientity re provie in Section IV. In Section V, severl pplictions bse on the propose novel results re supplie. Finlly, conclusions re mentione in Section VI. All the etile mthemticl erivtions for the propose results re given in ppenices. II. PRELIMINARY RESULTS In this section, severl efinitions n preliminry theorems re provie. First, the concept of Fisher informtion is efine s follows. Fisher informtion of eterministic prmeter θ is efine s J θ Y f Y y;θ θ logf Yy;θ y E Y SYθ Y, where S Yθ Y enotes score function n is efine s /θlogf Y y;θ. Uner regulrity conition, E Y S Yθ Y, θ f Yy;θy the Fisher informtion in is equivlently expresse s J θ Y f Y y;θ θ logf Yy;θy E Y θ S Y θ Y. 3 This is generl efinition of Fisher informtion in signl processing, n Fisher informtion provies lower boun, clle the Crmér-Ro lower boun, for men squre error of ny unbise estimtor. Like other concepts, such s entropy n mutul informtion, in informtion theory, Fisher informtion lso shows informtion bout uncertinty. However, it is ifficult to irectly opt the efinition of Fisher informtion in informtion theory espite the fct tht it hs been commonly use in sttistics. Inste, more specific efinition of Fisher informtion is propose s follows. If θ is ssume to be loction prmeter, then θ f Yy;θ y f Yy θ;θ. 4 Therefore, the efinition of Fisher informtion in is chnge s follows: J θ Y f Y y;θ θ logf Yy;θ y f Y y θ;θ y logf Yy θ;θ y EỸ fỹỹ;θ ỹ logf Ỹ ỹ;θ ỹ SỸ, 5 where SỸ enotes score function, n it is efine s /ỹlogfỹỹ;θ. In eqution 5, since we only consier loction prmeter, we refer to Fisher informtion in 5 s Fisher informtion with respect to loction or trnsltion prmeter, n it is enote s JỸ even though the efinition of Fisher informtion with respect to loction prmeter in 5 is erive from the efinition of Fisher informtion in, the efinition in 5 is more commonly use in informtion theory, n we o not istinguish rnom vrible Ỹ Y θ from rnom vrible Y. Given the chnnel moel in, by substituting the prmeter for the unknown prmeter θ, the expressions of Fisher informtion in n 5 re respectively given by JY f Y y; y logf Yy; y n J Y E Y SY Y, 6 f Y y; logf Yy; y E Y SY Y. 7 Secon, two funmentl concepts, ifferentil entropy n entropy power, re efine s follows. Differentil entropy of rnom vrible Y, hy, is efine s hy f Y y;logf Y y;y, 8 where f Y y; enotes the probbility ensity function pf of rnom vrible Y, log enotes the nturl logrithm, n is eterministic prmeter in the pf. Similrly, the conitionl entropy of rnom vrible Y given rnom vrible X, hy X is efine s hy X f X,Y x,y;logf Y X y x;xy, 9 where f X,Y x,y; enotes the joint pf of rnom vribles X n Y, f Y X y x; is the conitionl pf of rnom vrible Y given rnom vrible X. Entropy power of rnom vrible Y, NY, n conitionl entropy power of rnom vrible Y given rnom vrible X, NY X re respectively efine s NY NY X πe exphy, exphy X. πe Bse on the efinitions mentione bove, three preliminry theorems- De Bruijn s, Stein s, n het eqution ientitiesre introuce next. Theorem De Bruijn s Ientity, 8: Given the itive noise chnnel Y X + W, let X be n rbitrry rnom vrible with finite secon-orer moment, n W be inepenent normlly istribute with zero men n unit vrince. Then, Proof: See. hy JY.

3 3 Theorem Generlize Stein s Ientity 3: Let Y be n bsolutely continuous rnom vrible. If the probbility ensity function f Y y stisfies the following equtions, n y f Yy f Y y lim kyf Yy, y ± for some function ky, then y ky ky + ν ty ky E Y ryty ν E Y Y ryky, for ny function ry which stisfies E Y ryty <, E Y ry <, n E Y ky Y ry <. E Y enotes the expecttion with respect to the pf of rnom vrible Y. In prticulr, when rnom vrible Y is normlly istribute with men µ y n vrince σy, eqution simplifies to E Y ryy µ y σy E Y Y ry. 3 Eqution 3 is the well-known clssic Stein s ientity. Proof: See 3. Theorem 3 Het Eqution Ientity : Let Y be normlly istribute with men µ n vrince +. Assume gy is twice continuously ifferentible function, n both gy n y gy re Oe c y for some c <. Then, E Y gy E Y Y gy. 4 Proof: See. III. RELATIONSHIPS BETWEEN STEIN S IDENTITY AND DE BRUIJN S IDENTITY In Section II, Theorems,, n 3 shre n nlogy: n ientity between expecttions of functions, which inclue erivtives. Especilly, the het eqution ientity mits the sme form s De Bruijn s ientity by choosing function gy s logf Y y;. If De Bruijn s ientity is equivlent to the het eqution ientity, it is lso equivlent to Stein s ientity, since the equivlence between the het eqution ientity n Stein s ientity ws prove in. However, there re two criticl issues tht stn in the wy of the equivlence between Stein s n De Bruijn s ientities: first, the function gy in Theorem 3 must be inepenent of the prmeter, which is not true when gy logf Y y;. Secon, in the het eqution ientity, rnom vrible Y must be Gussin, which my not be true in De Bruijn s ientity. Due to the ifficulties mentione bove, we will irectly compre De Bruijn s ientity Theorem with the generlize Stein s ientity Theorem. Theorem 4: Given the chnnel moel, let X be n rbitrry rnom vrible with finite secon-orer moment, n let W be normlly istribute with zero men n unit vrince. Inepenence between rnom vribles X n W O enotes the limiting behvior of the function, i.e., gy Oqy if n only if there exist positive rel numbers K n y such tht gy K qy for ny y which is greter thn y. is lso ssume. Then, De Bruijn s ientity is equivlent to the generlize Stein s ientity in uner specific conitions, i.e., hy JY E Y ry;ty; ν E Y Y ry;ky;, with ry; y logf Yy;, ky, ty; y f Yy; f Y y;, n ν, 5 where enotes the equivlence between before n fter the nottion. Proof: See Appenix A. Now, when rnom vrible Y is Gussin, i.e., both rnom vribles X n W re Gussin, we cn erive reltionships mong three ientities, De Bruijn, Stein, n het eqution, s specil cse of Theorem 4. Theorem 5: Given the chnnel moel, let rnom vrible X be normlly istribute with men µ n unit vrince. Assume W is inepenent normlly istribute with zero men n unit vrince. If we efine the functions in s follows: ry; y logf Yy;, ky;, ty; y, n ν µ, then Stein s ientity is equivlent to De Bruijn s ientity. Moreover, if we efine gy; s gy; logf Y y; in 4, then De Bruijn s ientity is lso equivlent to the het eqution ientity. Proof: In Theorem 4, given the chnnel moel with n rbitrry but fixe rnom vrible X n Gussin rnom vrible W, the equivlence between De Bruijn s ientity n the generlize Stein s ientity ws prove cf. Appenix A. Here, by choosing rnom vrible X s Gussin, this is specil cse of Theorem 4. Therefore, the equivlence between the two ientities is trivil, n the etils of the proof is omitte in this pper. The only thing to prove is the secon prt of this theorem, nmely, the equivlence between De Bruijn s ientity n the het eqution ientity. Since the equivlence between Stein s ientity n the het eqution ientity is prove in, this lso proves the secon prt of the theorem, n the proof is complete. The functions ky;, ry;, ty;, n gy; re the sme s ky, ry, n ty in Theorem n gy in Theorem 3, respectively. To show the epenence on prmeter, the functions ky;, ry;, ty;, n gy; re use inste of ky, ry, ty, n gy, respectively. IV. EXTENSION OF DE BRUIJN S IDENTITY De Bruijn s ientity is erive from the ttribute of Gussin ensity functions, which stisfy the het eqution. However,

4 4 in generl, probbility ensity functions o not stisfy the het eqution. Therefore, to exten De Bruijn s ientity to itive non-gussin noise chnnels, generl reltionship between ifferentils of probbility ensity function with respect to y n of the form: f Y Xy x; y xfy X y x;, 6 y is require, result tht it is obtine in Appenix H by exploiting the ssumptions 7. The reltionship 6 represents the key ingreient in estblishing the link between the erivtive of ifferentil entropy n posterior men, s escribe by the following theorem. Theorem 6: Consier the chnnel moel, where X n W re rbitrry rnom vribles inepenent of ech other. Given the following ssumptions: y E X fy X y X; E X y f Y Xy X;, E X fy X y X; E X f Y Xy X;, 7 lim y ± lim y ± f Y y;logf Y y;y f Y y;logf Y y; y, 7b E X XfY X y X; E X lim Xf Y X y X;, y ± E X fy X y X; E X lim f Y X y X;, lim y f Y y;, y ± E X XfY X y X; fy y; <, y ± 7c 7 where E X Y enotes the posterior men, the first erivtive of the ifferentil entropy is expresse s hy { } E Y Y E X Y X Y. 8 Proof: See Appenix B. Remrk : This is equivlent to the results in 4. It cn be observe tht the conitions 7 re require in the ominte convergence theorem n Fubini s theorem to ensure the interchngebility between limit n n integrl, n re not tht restrictive. Also, the conition lim y ± y f Y y; is not restrictive t ll, n it is stisfie by ll noise istributions of interest in prctice. Corollry De Bruijn s ientity: Given the chnnel moel in with n rbitrry but fixe rnom vrible X with finite secon moment n Gussin rnom vrible W with zero men n unit vrince, hy JY. Remrk : This is the well-known De Bruijn s ientity 8. Therefore, De Bruijn s ientity is specil cse of Theorem 6 when rnom vrible W is normlly istribute. When rnom vrible W is Gussin, ssumptions in 7 re simplifie to the existence of finite secon-orer moment. Corollry : Given the chnnel moel in with n rbitrry but fixe non-negtive rnom vrible X whose moment generting function exists n its pf is boune, n n exponentil rnom vrible W with unit vlue of the prmeter i.e., f W w exp wuw, where U enotes the unit step function, hy { EX X+E X EX Y X Y Y X }. When the rnom vrible W is exponentilly istribute, ssumptions in 7 re reuce to the existence of the moment generting function of X, s expline in Appenix I. Therefore, the ssumptions in 7 for n exponentil rnom vrible re s simple s the ssumptions 7 for Gussin rnom vrible. Corollry 3: Given the chnnel moel in with n rbitrry but fixe non-negtive rnom vrible X whose moment generting function exists n gmm rnom vrible W with shpe prmeter α α n n inverse scle prmeter β β, hy { EX X +E Yα EX Y X Y Y Y α }, where Y k X + W k, n W k enotes gmm rnom vrible with shpe prmeter k. Nottion Y α stns for Y. As expline in Appenix I, the ssumptions 7 re quite simplifie in the presence of the moment generting function of rnom vrible X. For itive non-gussin noise chnnels, the ifferentil entropy cnnot be expresse in terms of the Fisher informtion. Inste, the ifferentil entropy is expresse by the posterior men s shown in Theorem 6. Fortuntely, severl noise istributions of interest in communiction problems stisfy the require ssumptions 7 in Theorem 6 e.g., Gussin, gmm, exponentil, chi-squre with restrictions on prmeters, Ryleigh, etc.. Therefore, Theorem 6 is quite powerful. If the posterior men E X Y X Y is expresse by polynomil function of Y, e.g., X n W re inepenent Gussin rnom vribles in eqution or rnom vribles belonging to the nturl exponentil fmily of istributions 9, then eqution 8 cn be expresse in simpler forms. Exmple : Consier n itive white Gussin noise AWGN chnnel. Given the chnnel moel, let X n W be normlly istribute with zero men n unit vrince. Assume X n W re inepenent of ech other. Then, the posterior men is expresse s E X Y X Y y + y, which is liner to y. Therefore, eqution 8 is expresse s hy { } E Y Y E X Y X Y +.

5 5 Now, we consier the secon erivtive of the ifferentil entropy. One interesting property of the secon erivtive of the ifferentil entropy is tht it cn lwys be expresse s function of the Fisher informtion 7. Theorem 7: Given the chnnel moel, let X n W be rbitrry rnom vribles, inepenent of ech other. Given the following ssumptions: y E X fy X y X; E X y f Y Xy X;, E X fy X y X; E X f Y Xy X;, 9 f Y y;logf Y y;y f Y y;logf Y y; y, 9b lim E X X y f Y Xy X; y ± fy y; E X lim X y ± lim E X XfY X y X; y ± E X lim y ± y f Y Xy X;, fy y; lim Xf Y X y X; y ± E X fy X y X; E X lim y 8 f Y y;, y ± E X X f Y X y X; f Y y; 3/4 <,, 9c lim f Y X y X;, y ± 9 9e where E X Y enotes the posterior men, the following ientity hols: hy J Y or equivlently, 4 E Y hy J Y 4 E Y hy Y S YYE X Y Y X Y, 4 E Y Y E X Y Y X Y Y SYE X Y Y X Y. Proof: See Appenix C. Similr to the corollries of Theorem 6, by specifying noise istribution n mnipulting eqution in Theorem 7, we erive the following corollries. Corollry 4: Given the chnnel, let X be n rbitrry but fixe rnom vrible with finite secon-orer moment, n let W be inepenent normlly istribute with zero men n unit vrince. Then, hy J Y 4 JY 4 E Y E Y Y S YYE X Y Y X Y Y S YY Remrk 3: This result is sclr version of the result reporte in 3. At the sme time, this result is specil cse, when X is Gussin rnom vrible, of the generl result in Theorem 7. Corollry 5: Uner the chnnel, let X be n rbitrry but fixe non-negtive rnom vrible with finite moment generting function, n its pf is boune. Let W be inepenent exponentilly istribute with unit vlue s the prmeter λ of the istribution. Nmely, f W w exp wuw, where U enotes the unit step function. Then, hy J Y+ 3 4 E X EX Y Y X Y YX E X EX Y Y X Y YX. Corollry 6: Uner the chnnel, let X be n rbitrry but fixe non-negtive rnom vrible with finite moment generting function, n W be n inepenent gmm rnom vrible with prmeters α α 3 n β β, i.e., f W w β α w α exp βwuw/γα, where U enotes the unit step function n Γ stns for the gmm function. Then, hy 4 3E Y EX Y α Y X Y Y Y α 4 E Y α E X Y X Y Y Y α + α 4 E E X Y Y X Y Y α E X Yα Y α X Y α Y Y α J Y 4 EX X, where Y α X + W α, n W α enotes gmm rnom vrible with shpe prmeter α. Like Corollries,, n 3, the ssumptions 9 reuce to simplifie forms in Corollries 4, 5, n 6. Even though we hve not enumerte ll possible probbility ensity functions for Theorem 6 n Theorem 7, mny of the probbility ensity functions tht present n exponentil term stisfy the ssumptions 7 n 9, since such conition proves to be sufficient for the require interchnge between limit n integrl. V. APPLICATIONS As mentione in n, De Bruijn s ientity hs been wiely use in vriety of res such s informtion theory, estimtion theory, n so on. Similrly, De Bruijn-type ientities mentione in this pper cn be opte in mny pplictions. Here, we introuce severl pplictions from the.

6 6 estimtion theory relm s well s from the informtion theory fiel. A. Applictions in Estimtion Theory In estimtion theory, there exist two funmentl lower bouns: Crmér-Ro lower boun CRLB n Byesin Crmér-Ro lower boun BCRLB. CRLB is lower boun for the estimtion error of ny unbise estimtor, n it is erive from frequentist perspective. This lower boun is tight when the output istribution of the chnnel is Gussin. CRLB n its tightness cn be justifie using Cuchy- Schwrz inequlity. On the other hn, BCRLB is lower boun for the estimtion error of ny estimtor, n it is clculte from Byesin perspective. BCRLB oes not require unbiseness of estimtors unlike CRLB; however, BCRLB requires prior knowlege i.e., istribution of rnom prmeters. BCRLB is lso tight when ll rnom vribles re Gussin. Surprisingly, ssuming Gussin itive noise chnnel, both of these lower bouns cn be erive using De Bruijntype ientities, n there exist counterprts both in informtion theory n estimtion theory. Since CRLB n its counterprt, the worst itive noise lemm, re erive in, we will only show the erivtion of BCRLB n its counterprt in this pper. Lemm Byesin Crmér-Ro Lower Boun: Given the chnnel, let ˆX be n rbitrry estimtor of X in Byesin estimtion frmework. Then, the men squre error MSE of ˆX is lower boune s follows: MSE ˆX E X JY X+JX, where X is n rbitrry but fixe rnom vrible with finite secon-orer moment, W is Gussin rnom vrible with zero men n unit vrince, n JY X x logf Y Xy x f Y Xy xy. Proof: See Appenix D. Interestingly, there exists counterprt, bse on ifferentil entropies, of BCRLB in informtion theory, n this counterprt is tighter lower boun thn BCRLB. Lemm : Uner the sme conitions s in Lemm, MSE ˆX NX Y, where NX Y /πeexphx Y, Y X + W,, n X n W re inepenent of ech other. Proof: See Appenix E. Remrk 4: Lemm seems to be similr to the estimtion counterprt of Fno s inequlity, p. 55, Theorem However, the current result is completely ifferent thn, p. 55, Theorem In, to stisfy the inequlity, the hien ssumption is VrX Y VrX G Y G, 3 where VrX Y n VrX G Y G enote posterior vrinces for rnom vribles X n Y, n Gussin rnom vribles X G n Y G, respectively. With the ssumption 3, the following reltions hol: E X,Y X EX Y X Y VrX Y VrX G Y G πe exphx G Y G πe exphx Y NX Y. This is nothing but the entropy mximizing theorem, i.e., the Gussin rnom vrible being the one tht mximizes the entropy mong ll rel-vlue istributions with fixe men n vrince. However, uner the ssumptions VrX VrX G n VrY VrY G, which re common ssumptions in signl processing problems, 3 my not be lwys true ue to the following fct. Given the itive Gussin noise chnnel, Y X + W G, where X is n rbitrry non-gussin rnom vrible whose vrince is ienticl to tht of Gussin rnom vrible X G, n W G is Gussin rnom vrible with zero men n unit vrince, VrX Y < VrX G Y G, 4 where Y G is Gussin rnom vrible whose vrince is ienticl to tht of Y. Eqution 4 violtes the ssumption 3. Therefore, the result in, p. 55, Theorem cnnot be opte uner the ssumptions, VrX VrX G n VrY VrY G, which re common in signl processing problems. On the other hn, the inequlity in Lemm is obtine not by imposing ienticl posterior vrinces but by ssuming ienticl secon-orer moments. Thus, represents lower boun on the men squre error similr to BCRLB. Therefore, Lemm illustrtes novel lower boun on the men squre error from n informtion theoretic perspective. Surprisingly, this lower boun is tighter thn BCRLB s the following lemm inictes. Lemm 3: Uner the sme conitions s in Lemm, NX Y E X JY X+JX, 5 where Y X+ W, is nonnegtive,x is n rbitrry but fixe rnom vrible with finite secon-orer moment, W is Gussin rnom vrible with zero men n unit vrince, n JY X is efine s eqution. The equlity hols if the rnom vrible X is Gussin. Proof: See Appenix F. Figure illustrtes how tighter the new lower boun is compre to BCRLB when X is stuent-t rnom vrible, n W is Gussin rnom vrible. The egrees of freeom of X is 3, n the vrince of W is. As shown in Figure, the new lower boun is much tighter thn BCRLB especilly in low SNRs where the BCRLB is generlly loose. Also, Figure shows how tight the new lower boun is with respect to the minimum men squre error.

7 7 B. Applictions in Informtion Theory In informtion theory, the entropy power inequlity EPI is one of the most importnt inequlities since it is helps to prove the chnnel cpcity uner severl ifferent circumstnces, e.g., the cpcity of sclr Gussin brocst chnnel 3, the cpcity of Gussin MIMO brocst chnnel 4, 5, the secrecy cpcity of Gussin wire-tp chnnel 6, 7 n so on. The chnnel cpcity cn be prove not by EPI lone but by EPI in conjunction with Fno s inequlity. Depening on the chnnel moel, n itionl technique, chnnel enhncement technique 4, is require. Therefore, vrious versions of the EPI such s clssicl EPI 8, 8, 9, Cost s EPI 6, n n extreml inequlity 5 were propose by severl ifferent uthors. In this section, we will prove Cost s entropy power inequlity, stronger version of clssicl EPI using Theorem 7. Lemm 4 Cost s EPI: For Gussin rnom vrible W with zero men n unit vrince, NX + W NX+NX +W, 6 where, X n W re inepenent of ech other, n the entropy power NX is efine s NX /πe exphx. Alterntively, the inequlity 6 is expresse s NX + W, 7 i.e., NX + W is concve function of 6. Proof: See Appenix G. C. Applictions in Other Ares There re mny other pplictions of the propose results. First, since Theorem 6 is equivlent to Theorem in 4, Theorem 6 cn be use for pplictions such s generlize EXIT chrts n power lloction in systems with prllel non-gussin noise chnnels s mentione in 4. Secon, MSE MMSE BCRLB NewLB SNRB Fig.. Comprison of MMSE, BCRLB, n new lower boun New LB in with respect to SNR. by Theorem 4, we showe the equivlence mong Stein, De Bruijn, n het eqution ientities. Therefore, bro rnge of problems in probbility, ecision theory, Byesin sttistics n grph theory s escribe in coul be consiere s itionl potentil pplictions of Theorems 4 n 6. VI. CONCLUSIONS This pper minly isclose three informtion-estimtion reltionships. First, the equivlence between Stein ientity n De Bruijn ientity ws prove. Secon, it ws prove tht the first erivtive of the ifferentil entropy with respect to the prmeter cn be expresse in terms of the posterior men. Secon, this pper showe tht the secon erivtive of the ifferentil entropy with respect to the prmeter cn be expresse in terms of the Fisher informtion. Finlly, severl pplictions bse on the three min results liste bove were provie. The suggeste pplictions illustrte tht the propose results re useful not only in informtion theory but lso in the estimtion theory fiel n other fiels. APPENDIX A A PROOF OF THEOREM 4 Since Theorem 5 is consiere s specil cse of Theorem 4, we only show the proof of Theorem 4 in this pper. Proof: Theorem 4 Prior to proving Theorem 4, we first introuce the following reltionships in Lemm 5, which re require for the proof. Lemm 5: For rnom vribles W, X n Y efine in eqution when Gussin rnom vrible W hs zero men n unit vrince n rnom vrible X hs finite seconorer moment, the following ientities re stisfie: i logf Yy; yu+ w EX y X f Y Xy X; f Y y; ii logf Yu+ w; EX u Xy Xf Y X y X; f Y y; iii y logf Yy; yu+ w E X y Xf Y X y X;, f Y y; yu+ w w iv y logf Yy; yu+ w logf Yu+ w; logf Yy; yu+ w, yu+ w, yu+ w where fy y enotes lim y fy. In some cses, to voi confusion, fy y is use inste of fy y.,

8 8 Proof: Since f Y X y x; is normlly istribute with men x n vrince, the following reltionships hol: f Y X y x; y x exp, 8 π y f Y Xy x; y xf Y Xy x;, 9 f Y Xy x; + x y f Y X y x;, 3 f Y Xu+ w x; f Y X u+ w x; + u+ w xu x. 3 Eqution 3 is true since f Y Xu+ w x; exp π u+ w x exp π u+ w x + exp π u+ w x u+ w x w u+ w x f Y Xu+ w x; +f Y X u+ w x; u+ w xu x. Bse on eqution 3, i is prove by following these clcultions: logf Yy; yu+ w E X f Y Xy X; f Y y; yu+ w EX y X f Y X y X;.3 f Y y; yu+ w Secon, eqution ii is prove by the following clcultions: logf Yu+ w; E X f Y Xu+ w X; f Y u+ w; E X f Y Xu+ w X; f Y u+ w; + E X u+ w Xu Xf Y X u+ w X; f Y u+ w; 33 f Yu+ w; f Y u+ w; + E X u+ w Xu Xf Y X u+ w X; f Y u+ w; EX u+ w Xu Xf Y X u+ w X; f Y u+ w; EX y Xu Xf Y X y X; f Y y;. 34 yu+ w The equlity in 33 is ue to eqution 3. Thir, eqution iii is prove bse on eqution 9 s follows: y logf Yy; yu+ w E X y f Y Xy X; f Y y; yu+ w E X y XfY X y X;. 35 f Y y; yu+ w The equlity in 35 is ue to eqution 9. Eqution iv is trivil since eqution 35 multiplie by w/ is equl to eqution 34 minus eqution 3, n the proof is complete. Like the proof of Theorem 3 in, the equivlence is prove by showing tht ech ientity is erive from the other one, using Lemm 5. First, in the generlize Stein s ientity, ll necessry functions re efine s follows: ry; y logf Yy;, ky, ty; y f Yy; f Y y;, n ν. 36 Then, De Bruijn s ientity is erive from the generlize Stein s ientity s follows. E Y Y ry; E Y ry;ty; generlize Stein s ientity 37 y E X fy X y X; ry;y y X E X f Y X y X; y logf Yy;y y u f X u f Y Xy u; y logf Yy;yu. A 38 The interchngebility mong integrls n erivtives re ue to the ominte convergence theorem n Fubini s theorem. Chnging the vrible s y u + w, eqution A is

9 9 expresse s y u f Y X y u; y logf Yy;y w f Y Xu+ w u; y logf Yy; f Y X u+ w u; yu+ w logf Yu+ w; logf Yy; yu+ w w w 39 exp π w logf Yu+ w;w exp π w logf Yy; w yu+ w exp w logf Y u+ w;w π exp w π logf Yy; w. yu+ w 4 The equlity in eqution 39 is ue to Lemm 5, iv. Re-efining the vrible w y u/, eqution 38 is expresse s y u f X u f Y Xy u; f X u f Y X y u; logf Yy;y f Y y; logf Yy;y f Yy;y f Y y;y y logf Yy;y f Y X y u;logf Y y;y f Y y;logf Y y;y u u 4 f Y y;logf Y y;y 4 f Y y;logf Y y;y f Y y;logf Y y;y hy. The equlity in 4 is ue to the chnge of vrible, n the equlity in 4 is becuse of the inepenence of f X u with respect to. Since the left-hn sie of eqution 37 is equl tojy/, we obtin De Bruijn s ientity: JY hy, from the generlize Stein s ientity. Secon, the generlize Stein s ientity is erive from De Bruijn s ientity. We efine the function gy; y ru;u+q, 43 where q logf Y y; y. Here, q is lwys relvlue ue to the following: f Y y; lim E X f Y X y X; y y E X lim y E X exp π π exp X y X π. 44 The lst inequlity is ue to exp X. In ition, eqution 44 is lwys greter thn zero unless f X x is ienticl to zero or is infinite. However, neither cse hols. Therefore, q is lwys mpping to rel-vlue number. Then, the expecttion of gy; is expresse s E Y gy; y f Y y; y y f Y y;ru;uy + ru;u+q y y f Y y;ru;uy +q f Y y;ru;uy f Y y;ru;uy +q y f Y y;y ru; u u u f Y y;y ru;u+q E X f Y X y X;y ru; u u u E X f Y X y X;y ru; u +q u X E X Φ ru;u E X u X Φ ru;u +q, 45 where Φ enotes the stnr norml cumultive ensity function. We ifferentite both sies of eqution 45 with respect to

10 prmeter s follows. E Y gy; E X Φ u X ru;u u X E X Φ ru;u + q u X E X Φ ru;u u X +E X Φ ru;u u X E X Φ ru;u u X E X Φ ru;u + q u X E X Φ ru;u u X +E X Φ ru;u B u X E X Φ ru;u + q. 46 C Equtions B n C re further processe s u X E X Φ ru;u u X E X Φ ru;u E X f Y X y X;y ru;u u u E X f Y X y X;y ru;u y E X ru;uf Y Xy X;y E X y ru;uf Y Xy X;y y E X ru;uf Y Xy X;y y +E X ru;uf Y Xy X;y y E X ru;uf Y Xy X;y. 47 The interchngebility mong integrls is ue to Fubini s theorem n ominte convergence theorem. Due to eqution 43, gy; y ru;u+ q, eqution 47 is further simplifie s follows: y E X ru;uf Y Xy X;y y ru; u f Y y;y f Y y; gy;y q f Y y; logf Yy;y q 48 q. The equlity in 48 hols becuse gy; logf Y y;. Therefore, the lst three terms in eqution 46 vnish, n eqution 46 is expresse s u X E X Φ ru;u u X E X y Φy ru;u y u X u X u X E X φ ru;u u X E X exp u X π E X y f Y Xy X; ru; u u f Yu; f Y u; ru;f Yu;u E Y ty;ry;, ru; u where φ enotes the stnr norml probbility ensity function, n ty; y f Yy;/f Y y;. Since hy E Y gy; n E Y ty;ry;, JY E Y Y ry;, from De Bruijn s ientity, we erive the generlize Stein s ientity: hy JY E Y ty;ry; E Y Y ry;, where enotes equivlence between before n fter the nottion. APPENDIX B A PROOF OF THEOREM 6 Bse on eqution 6, Theorem 6 is prove next using integrtion by prts n the ominte convergence theorem.

11 Proof: Theorem 6 hy +logf Y y; f Yy;y f Yy;y logf Y y; f Yy;y 49 logf Y y; E X f Y X y X;y logf Y y;e X f Y Xy X; y. 5 The interchngebility between integrl n erivtive is ue to ssumptions 7 n 7b. Using eqution 6, eqution 5 is expresse s f Y Xy X; y logf Y y;e X logf Y y;e X y y Xf Y Xy X; y logf Y y; y E X y Xf Y X y X;y 5 logf Yy;E X y Xf Y X y X; y y logf Yy;E X y Xf Y X y X;y 5 y logf Yy;E X y Xf Y X y X;y 53 y f Yy;E X y X f Y Xy X; y, 54 f Y y; where fy y enotes lim fy lim fy. y y The first term in eqution 5 vnishes ue to the following reltionship: logf Y y;e X y XfY X y X; y yf Y y;logf Y y; y E X XfY X y X; logf Y y;. 55 y The first term in 55 is expresse s yf Y y;logf Y y; y y f Y y; f Y y;log f Y y;. 56 y Due to ssumptions 7, y f Y y; converges to zero s y goes to ±. Since x log x becomes zero s x goes to zero n f Y y; converges to zero s y goes to ±, fy y;log f Y y; in 56 lso becomes zero s y pproches ±. Similrly, the secon term in 55 is re-written s E X Xf Y X y X;logf Y y; y E X Xf Y X y X; f Y y;log f Y y; fy y; y.57 Since fctor tens to zero sy pproches±, n fctor is boune ue to ssumption 7, the right-hn sie of eqution 57 pproches zero s y goes to ±. Therefore, the first term in eqution 5 is zero, n the equlity in 53 is verifie. Agin, using integrtion by prts, eqution 54 is expresse s f Yy;E X + y f Yy;E X y X f Y Xy X; f Y y; y X f Y Xy X; f Y y; y y f Y y; y E X y X f Y Xy X; y58 f Y y; f Y y; y E X y X f Y Xy X; y 59 f Y y; f Y y; y { E Y Y E X Y X Y y E X X f Y Xy X; f Y y; } y. 6 The equlity in 59 is verifie by the following proceure: the first prt of eqution 58 is re-written s f Yy;E X y X f Y Xy X; f Y y; y yfy y; E X XfY X y X; y. 6 Due to ssumptions 7c n 7, both terms yf Y y; n E X Xf Y X y X; become zero s y goes to ±, n eqution 6 is zero. Therefore, hy n the proof is complete. { } E Y Y E X Y X Y, APPENDIX C A PROOF OF THEOREM 7 Proof: Theorem 7 From eqution 49, we know hy f Yy;y logf Y y; f Yy;y. logf Y y; f Yy;y

12 Therefore, the secon erivtive of ifferentil entropy is expresse s hy logf Yy; f Yy;y J Y logf Y y; f Yy;y, logf Y y; f Yy;y. 6 The lst equlity is ue to the efinition of Fisher informtion with respect to prmeter in 7. From eqution 6, we erive n itionl reltionship between the secon orer ifferentils with respect to y n : Since f Y Xy x; y xfy X y x; y y xfy X y x; y + 4 y y x y f Y X y x; y x y xfy X y x;. y y y xy xf Y Xy x; y y xf Y Xy x;+ y x y y y xf Y Xy x;, we obtin the following reltionship: f Y Xy x; 4 y y x f Y X y x; + 4 y y xfy X y x;.63 Tking the expecte vlue of both sies of 63, { f Yy; 4 y E X y X f Y X y X; + y E X y XfY X y X; }. 64 After substituting f Y y;/, from eqution 64, into eqution 6, the secon term of 6 tkes the expression: logf Y y; f Yy;y 4 logf Y y; y E X y X f Y X y X; y D 4 logf Y y; y E Xy Xf Y X y X;y. E Term E is exctly of the sme form s 5, n therefore, 4 logf Y y; y E X y Xf Y X y X;y Y E X Y Y X Y hy E Y Term D is further simplifie by the following proceures: 4 4 logf Yy; + 4 logf Y y; y E Xy X f Y X y X;y y E Xy X f Y X y X; y y logf Yy; y E Xy X f Y X y X;y. The first prt of 66 is expresse s 4 logf Yy; y E X y X f Y X y X; y 4 logf Yy; E X y Xf Y X y X; +E X y Xy+X y f Y Xy X; 4 logf Yy; yf Y y; E X Xf Y X y X; +y y f Yy; ye X +E X X y f Y Xy X; fy y;log f Y y; b X y f Y Xy X; y y f Y y; +E X X y f Y Xy X; fy y; b b 3 4 f Y y;log 4 f Y y; b y 4 y f Y y; E f Y Xy X; X fy y; b b 3 y 4 y f Y y; E X X f Y Xy X; fy y; b b 3 + fy y;log E X Xf Y X y X; f Y y; fy y; b b 4 y y 66.

13 3 Since xlogx becomes zero s x pproches zero n f Y y; converges to zero s y goes to ±, fctor b is zero s y ±. Due to ssumptions 9c n 9, term b becomes zero s y ± n term b 3 is boune. Also, fctor b 4 must be boune ue to ssumption 9e. Therefore, s y ±, the first prt of eqution 66 vnishes. Then, eqution 66 is further processe using integrtion by prts s follows: 4 y logf Yy; y E Xy X f Y X y X;y 4 y logf Yy;E X y X f Y X y X; y 4 y logf Yy;E X y X f Y X y X;y. Agin, the first prt of eqution 67 is re-written s 4 y logf Yy;E X y X f Y X y X; 4 E X E X y f Y Xy X; fy y; y y X f Y Xy X; fy y; 4 E y f Y Xy X; X y f Y y; fy y; c c 4 E y f Y Xy X; X y 4 f Y y; fy y; c c E X X f Y Xy X; f Y y; }{{ 3/4 } c 3 y E y f Y Xy X; 4 X fy y; fy y; c c E X X f Y Xy X; f Y y; }{{ 3/4. 68 } y c 3 Fctors c n c 3 re boune ue to ssumptions 9c n 9e, n, by ssumption 9, fctor c pproches zero s y ±. Then, eqution 67 is expresse s 4 y logf Yy; y E X y X f Y X y X; y 4 y logf Yy;E X y X f Y X y X; y. 69 Using equtions 65 n 69, eqution 6 is expresse s hy J Y logf Y y; f Yy;y J Y hy 4 E Y Y S YYE X Y Y X Y J Y 4 E Y Y E X Y Y X Y 4 E Y Y S YYE X Y Y X Y, n the proof is complete. APPENDIX D A PROOF OF LEMMA Proof: Lemm Before we prove this lemm, we first introuce two lemms which re necessry to prove Lemm. Lemm 6: Given the chnnel Y X + W in, the following ientity hols: JY E Y Y S YY, 7 where X is n rbitrry but fixe rnom vrible with finite secon-orer moment, n W is Gussin rnom vrible with zero men n unit vrince. Proof: In Theorems 4, 5, we showe the equivlence mong De Bruijn, generlize Stein, n het eqution ientities for specific conitions. Therefore, using one of the ientities, this lemm cn be prove. In this proof, Theorem 3 the het eqution ientity will be use with gy S Y y. Unlike the efinition of gy in Theorem 3, gy is epenent on the prmeter. Therefore, we use the nottion gy; inste of gy. Since JY ES Y Y, the right-hn sie of 7 is expresse s JY E Y SY Y f Yy;gy;y +E Y gy;.7 By the het eqution ientity, the first term in eqution 7 is expresse s f Yy;gy;y E Y Y gy;. Using integrtion by prts, the secon term in eqution 7 is expresse s E Y gy; E Y Y gy; E Y Y S YY +E Y S Y Y Y S YY.

14 4 Therefore, eqution 7 tkes the form: f Yy;gy;y +E Y gy; E Y Y S YY +E Y +E Y gy; Y S Y Y Y S YY. F Performing n integrtion by prts, the term F is shown to be equl to zero, n the proof is complete. Remrk 5: A vector version of this lemm ws reporte in 3. The resons why we introuce both this lemm n its proof re not only to present lterntive proofs, but lso to explin the usefulness of our novel results. For exmple, Lemm 6 ws prove bse on the het eqution ientity, which is novel pproch to prove this lemm. At the sme time, this lemm cn lso be lterntively prove using Theorem 7 or Corollry 4. Lemm 7 Fisher Informtion Inequlity: Consier the chnnel Y X + W in, where the rnom vrible X is ssume to hve n rbitrry istribution but fixe secon-orer moment n W is normlly istribute with zero men n unit vrince. Then, the following inequlity is lwys stisfie: JY JX + J W, where the equlity hols if n only if X is normlly istribute. Proof: Using Lemm 6 equivlently, Theorem 7 or Corollry 4 cn be use, JY E Y Eqution 7 is expresse s n it is equivlent to Y S YY E Y Y S YY JY. 7 JY JY, JY JY. 73 JY Since inequlity 73 is stisfie for ny, t t JY JY JX, t, JY JX + J W. 74 Since W is normlly istribute with unit vrince, /J W, n the lst equivlence hols. The lst eqution in 74 enotes the Fisher informtion inequlity, n the proof is complete. Remrk 6: This proof uses neither the convolutionl inequlity, the t processing inequlity, nor the EPI, unlike previous proofs. The proof only relies on De Bruijn s ientity, Stein s ientity, or the het eqution ientity. Nmely, Theorem,, 3, or 7 is the only opte result, n Theorems 4, 5 ensure Theorem,, 3, or 7 cn be equivlently opte to the proof. Even though Lemm 6 ws use in this proof, Lemm 6 itself ws lso prove using one of the bove ientities. Therefore, this proof only uses our results. Now, bse on Lemm 7, the proof of Lemm is strightforwr. From Lemm 7, JY JX + J W, JY JXJ W JX+J W. 75 Since X n W re inepenent, n W is normlly istribute, E X JY X f X x x logf Y Xy x; f Y Xy x;yx f X x J W. y x f Y X y x;yx 76 The equlity in 76 is ue to E Y X Y X X x. For Gussin rnom vrible W, JY VrX Y, 77 where VrX Y stns for E X,Y X E X Y X Y,. Substituting VrX Y n E X JY X for JY n J W, respectively, eqution 75 is expresse s JY JXJ W JX+J W, VrX Y JXJ W JX+J W, VrX Y VrX Y JX+J W, JX+E X JY X. Since VrX Y is equl to the minimum men squre error, MSE ˆX MMSE ˆX VrX Y JX+E X JY X,

15 5 where ˆX enotes Byesin estimtor, n the obtine inequlity is the Byesin Crmér-Ro lower boun BCRLB. APPENDIX E A PROOF OF LEMMA Proof: Lemm When is zero, the right-hn sie of is zero ue to the following reltions: NX Y πe exphx Y πe exphx+hy X hy πe exphx+h W hy NXN W NY NXNW NX + W. Therefore, when goes to zero, NXNW lim NX Y lim NX + W. 78 The equlity is ue to the fct tht lim NX + W NX. Since the left-hn sie of is lwys greter thn or equl to zero, the inequlity in is stisfie when is zero. Without loss of generlity, from now on, we ssume tht >. Since hx Y hx+hy X hy, by Theorem De Bruijn s ientity, NX Y πe exphx Y { NX Y hx+ hy X } hy { NX Y } JY 79 NX Y VrX Y. 8 Since hx is inepenent of n hy X h W, /hx is zero, n /hy X /. Therefore, the equlity in 79 is stisfie. The equlity in 8 is ue to eqution 77. Bse on eqution 77, VrX Y JY The equlity in 8 is ue to Theorem. hy. 8 Using Corollry 4 n eqution 77, eqution 8 is further processe s hy + hy hy JY+ E Y Y S YY 8 JY+ JY 83 JY VrX Y. The equlity in 8 is ue to Theorem n Corollry 4, n the inequlity in 83 hols becuse E Y Y S YY E Y Y S YY Therefore, JY. VrX Y VrX Y. 84 Using equtions 8 n 84, we obtin the following inequlity: lognx Y logvrx Y. Since NX G Y G VrX G Y G, where X G n Y G enote Gussin rnom vribles whose vrinces re equl to X n Y, respectively, the following inequlity lso hols: lognx G Y G lognx Y logvrx G Y G logvrx Y. 85 By performing n integrtion, from to, of both sies in 85, eqution 85 is expresse s t logn tx G Y G logn t X Yt t logvr tx G Y G logvr t X Yt logn t X G Y G logn t X Y t logvr t X G Y G logvr t X Y logn X G Y G logn X Y t lim t logn t X G Y G logn t X Y logvr X G Y G logvr X Y lim t logvr t X Y logvr t X G Y G 86 logn X Y logvr X Y, 87 where stns for equivlence between before n fter the nottion, subscript t or enotes tht function epens on

16 6 prmeter t or, respectively the subscript is only use when there my be confusion between n ctul prmeter vrible n ummy vrible. The equivlence in 87 is ue to the following: N X G Y G Vr X G Y G, n lim logn tx G Y G logn t X Y t limlog N tx G Y G t N t X Y / NX G N t Y G X G NXN t Y X limlog t N t Y G N t Y NX G N / tw limlog t NX G + NXN tw tw NX + tw NXG NX + tw limlog t NXNX G + tw NXG NX log NXNX G, 88 limlogvr t X G Y G logvr t X Y t lim log t t JX G + tw log t t JX+ tw t 89 lim log tjx G + tw log tjx+ tw t log log, where W is Gussin rnom vrible. The equlity in 89 is ue to eqution 77. Since log x is n incresing function with respect to x, eqution 87 is equivlent to n the proof is complete. NX Y VrX Y, APPENDIX F A PROOF OF LEMMA 3 Proof: Lemm 3 When, both sies of the inequlity in 5 re zero, n the inequlity in 5 is stisfie. Therefore, without loss of generlity, we ssume tht >. lognx Y NX Y NX Y VrX Y 9 9 log JX+J W JX+J W, where W is Gussin rnom vrible with zero men n unit vrince. The equlity in 9 is ue to eqution 8, the inequlity in 9 is becuse of BCRLB. SinceNX G Y G is equl to /JX G +J W, where X G n Y G re Gussin rnom vribles whose vrinces re equl to X n Y, respectively, the following inequlity is stisfie: lognx G Y G lognx Y log JX G +J W log JX+J.9 W By integrting both sies in 9, eqution 9 is equivlent to the following: t logn tx G Y G logn t X Yt log t JX G +J tw log JX+J t tw logn X G Y G logn X Y limlogn t X G Y G logn t X Y t log JX G +J W log JX+J W lim log t JX G +J tw log JX+J tw lognx Y log JX+J W, 93 where enotes the equivlence between before n fter the nottion, n subscript or t of function mens epenency of the function with respect to or t, respectively. The equivlence in 93 is ue to the following: NX G Y G is equl to /JX G +J W, n n lim log t log lim t JX G +J tw log t tjx G +JW log JX+J tw t tjx+jw lim t log tjx+jw tjx G +JW log JW JW, 94 lim logn tx G Y G logn t X Y t ue to eqution 88. Since logx is incresing function with respect to x, the inequlity in 93 is equivlent to NX Y JX+J W. 95 Since we hve lrey prove tht NX Y is lower boun for ny Byesin estimtor in Lemm, the inequlity in 95 mens tht the lower boun NX Y, the left-hn sie of 95, is tighter thn BCRLB, the right-hn sie of 95.

17 7 APPENDIX G A PROOF OF LEMMA 4 COSTA S EPI Proof: Lemm 4 The proof will be conucte in two ifferent wys. Inste of proving eqution 6, we re going to prove the inequlity in 7. Using De Bruijn s ientity, NY NY hy+ny hy, NY JY + hy, where Y X + W. Since NY, proving the inequlity in 7 is equivlent to proving the following inequlity: JY + hy. 96 Using Theorem 7, the inequlity in 96 is expresse s JY J Y E Y Y E X Y Y X Y E Y Y S YYE X Y Y X Y.97 By Corollry 4, eqution 97 is equivlent to JY J Y E Y Y E X YY X Y E Y Y S YYE X Y Y X Y JY E Y Y S YY E Y JY+ Y S YY 98. Since JY E/YS Y Y n ES Y Y, the equlity hols in 98. Therefore, NY E Y JY+ Y S YY,, n the proof is complete. Remrk 7: This proof mostly follows the proof in 3. However, by using Theorem 7 to prove Cost s EPI, we show tht Cost s EPI cn be prove by De Bruijn-like ientity without using the Fisher informtion inequlity. In the secon proof, the inequlity 7 is prove by slightly ifferent metho. First, efine function l s follows: l JX +JY, 99 +JX where Y X + W, X is n rbitrry but fixe rnom vrible, W is Gussin rnom vrible, n X n W re inepenent of ech other. For rbitrry non-negtive rel-vlue, l, n it is prove by the following proceure; using Lemm 6 Theorem 7 or Corollry 4 cn be use inste of Lemm 6, JY E Y Y S YY E Y Y S YY JY. Eqution is equivlent to the following inequlities: JY JY. JY Since inequlity is stisfie for rbitrry nonnegtive rel-vlue, t t JY t JY JX JY JX +JX, n therefore, eqution 99 is lwys non-positive. Since JY converges to JX s pproches zero, l, n the following inequlity hols for n rbitrry but fixe rnom vrible X n rbitrry smll non-negtive rel-vlue ǫ: lǫ l JX +ǫjx +JX + ǫw 3. 4 Therefore, ǫ lǫ ǫ, 5 for n rbitrry but fixe rnom vrible X. Since the inequlity in 5 hols for n rbitrry rnom vrible X, we efine X s X + W, where X is n rbitrry but fixe rnom vrible, W is Gussin rnom vrible whose vrince is ienticl to the vrince ofw, n X, W, nw re inepenent of one nother. Then, the inequlity in 5 is equivlent to the following inequlities: J X + W +ǫj X + W ǫ + ǫ J X + W + ǫw J X + W ǫ ǫ +ǫj X + W + ǫ J X + +ǫ W 6 ǫ

18 8 J X + W ǫ +ǫj X + W + J X + +ǫ W 7 ǫ J X + W + J X + W, 8 where enotes the equivlence between before n fter the nottion. The equivlence in 6 is ue to the fct tht J X + W + ǫw J X + +ǫ W for inepenent Gussin rnom vribles W n W whose vrinces re ienticl to ech other. The inequlity in 7 hols ue to the following proceure: first, the Fisher informtion J X + +ǫ W is expresse s J X + +ǫ W y f Yy;,ǫ y logf Yy;,ǫy y E f Y Xy X;,ǫ loge X X f Y Xy y X;,ǫ y y E exp y X X π+ǫ +ǫ loge X exp y X y, y π+ǫ +ǫ 9 where Y X + +ǫ W. Since f Y Xy x;,ǫ is Gussin ensity function with men x n vrince +ǫ, the equlity in 9 hols. In eqution 9, n ǫ re symmetriclly inclue in the eqution, n therefore, ǫ J X + +ǫ W J X + +ǫ W. Since rnom vrible X is rbitrry n is n rbitrry non-negtive rel-vlue number in eqution 8, the proof is complete. APPENDIX H DERIVATION OF EQUATION 6 Given the chnnel moel, rnom vribles X n W re inepenent of ech other, is eterministic prmeter, n rnom vrible Y is the summtion of X n W. Therefore, between the two probbility ensity functions f Y X y x; n f W w, there exists reltionship tht cn be estblishe s follows. f Y X y x; f W w w y x y x f W. Therefore, n y f Y Xy x; f Y Xy x; y x f W f W f W y x y x y f W w f W w + y x + y x f W w f W w, w y x y x w y x. Eqution is further processe s y x f W + y x w f Ww w y x y x f W + y x w f Ww w y x y y x f Y X y x;+y x y f Y Xy x; y y xf Y Xy x;, n therefore, f Y Xy x; y xfy X y x;. y APPENDIX I EXPLANATION OF ASSUMPTIONS 7 IN COROLLARIES, 3 Corollry Given the chnnel Y X+ W in, W is ssume to be exponentilly istribute with unit prmeter, i.e., its pf f W w is efine s exp wuw, where U enotes unit step function. Since rnom vribles X n W re inepenent of ech other, conitionl ensity function f Y X y x; is expresse s f Y X y x; exp y x Uy x, n its erivtives with respect to y n re respectively enote s y f Y Xy x; f Y X y x;+ exp y x δy x,

19 9 f Y Xy x; f y x Y Xy x;+ f Y Xy x;, 3 where δ is Dirc elt function. The bsolute vlues of equtions, 3 re boune s n y f Y Xy x; f Y X y x;+ y x exp δy x f Y X y x; + y x exp δy x + y x exp δy x, 4 f Y Xy x; f y x Y Xy x;+ f Y Xy x; f Y Xy x; + y x f Y Xy x; 5 +E, 6 where E mx y y xf Y X y x;. Since f Y X y x; is exponentilly ecresing s y pproches, the rel vlue E lwys exists. Also, mx y fy Xy x; /, n therefore, the inequlities in 4 n 6 re stisfie. The right-hn sie of 4 n 6 re now integrble s follows: E X + y X exp δy X +f Xy, E X +E +E. 7 If function f X x is boune, by ominte convergence theorem, ssumption 7 is verifie. Secon, ssumption 7b is verifie s follows. f Yy;logf Y y; 8 logf Yy; f Yy; + f Yy; logf Yy;E X f Y Xy X; y X + f Y Xy X; + f Yy; fy y;logf Y y; fy y; + y fy y; E X Xf Y X y X; f Y y; + f Yy; f Y y;log f Y y; fy y;+ y fy y; E XXf Y X y X; f Y y; + f Yy; 3 K f Y y;log f Y y; + f Yy;. 9 The term 3 is boune by n integrble function ue to eqution 5, fctor is boune by constnt K ue to ssumptions 7c n 7, which will be prove lter, n fctor is boune, n it is integrble: f Y y;log f Y y; y f Y y;logf Y y; y E X exp loge X exp 4 exp E X exp log exp X y X Uy X y X y Uy X y Uy X E X exp X Uy X y 4 exp E y X exp X log exp y E X exp X y 4 exp y M X log exp y y M X y,

20 where M X enotes the moment generting function of X. If the moment generting function of X exists, then eqution is boune n integrble, n so oes the term. Therefore, term is integrble with respect to y, n ssumption 7b is verifie by ominte convergence theorem. Similrly, ssumption 7c is verifie s follows. fy X y x; y x exp Uy x, xfy X y x; x y x exp Uy x x, n the right hn-sie terms of n re integrble s E X E X X, E X X, 3 n if E X X exists, ssumption 7c is stisfie. Since f Y X y x; is exponentilly ecresing, lim y y f Y y; is zero. In ition, lim y y f Y y; lim E X y f Y X y X; y lim E X y y x exp Uy x y E X x exp U x. 4 Assumption 7 is expresse s E X Xf Y X y X; fy y; E X Xf Y X y X; fy y; f Y y; xf Xx exp y x Uy xx fy y;5 f Xx exp y x Uy xx y y f Xx exp y x x fy y y; 6 f Xx exp y x x y f Y y;. The inequlity in 6 is ue to the fct tht, in 5, the term insie integrl is non-negtive, x is incresing, n integrtion is performe from to y. Therefore, the ssumptions in 7 require the following conitions: existence of E X X, existence of M X, 3 boune pf f X x, n these re further simplifie into the existence of the moment generting function of X n boune pf f X x. Corollry 3 Given the chnnel Y X+ W in, W is ssume to be gmm rnom vrible, n its pf is expresse s f W w Γα wα exp wuw, where Γ is gmm function, U enotes unit step function, n α. Since rnom vribles X n W re inepenent of ech other, the conitionl ensity function f Y X y x; is expresse s f Y X y x; α y x exp y x Uy x,7 Γα n its erivtives re enote s y f Y Xy x; f Y X y x; α y x + exp y x Uy x,8 Γα f Y Xy x; α f Y Xy x; + α α y x exp y x Uy x. Γα+ 9 The bsolute vlues of equtions 8, 9 re boune s y f Y Xy x; f Y X y x; α y x + exp y x Uy x Γα f Y X y x; α y x + exp y x Uy x Γα f Y X y x; + f Yα Xy x; f Y X y x;+ f Yα Xy x;, 3 where f Yα Xy x; α y x exp y x Uy x, 3 Γα