INTRODUCTION ABSTRACT

Transcription

1

2 ABSTRACT mm INTRODUCTION Let us cosider the decisio theory problem of classifyig observatio X as comig from oe of the m possible classes (hypothesis) 0 = {01,02,..., 0 }. Let P; = Pr {0 = 0;}, i = 1,2,..., deote the prior probability of the classes ad let fi(x), f2( x ),... f (x) deote the coditioal desity fuctios give the true class i.e. L(x) = Pr { X = x / 0 = 0]}, i = 1,2,.... We assume that f,(x) ad pl, i = 1,2,... are completely kow. Give that the observatio X = x,-we ca coclude that the coditioal probability of 0 by the Bayes rule: P (0i / x) = Pr { 0 = 0j / X = x } IP jfjw It is well kow that the decisio rule, which miimizes the probability of error, is the Baye s decisio rule, which chooses the hypothesis with the largest posterior probability. Usig the rule, the probability of error for give X = x is expressed by P (e / x) = 1 - max [ P(0j / x), P (02 / x ),...P (0 / x) ],

3 FANG S INEQUALITY Prior to observig X, the probability of error P (e) associated with X is defied as the expected probability after observig it. i.e., P (e) = Ex[l-max{P(G1 /x),p(e2 /x)...p(0/x)}] = 1 - Ex [max { P(0! / x), P (02 / x ),...P (0 / x)} ] Give a arbitrary code (s, ) cosistig of words x(1), x<2),... x(s). Let X = (Xj, X2, X3,... X ) be a radom vector that equals x(1) with probability p(xw), i = 1,2,3,...s, where X p(x(i)) = 1. [I other words, we are choosig a code word at radom accordig to the distributio p(x(l)) ]. Let Y = (Yls Y2,... Y) be the correspodig output sequece. If P(e) is the probability of error of the code, computed for the give iput distributio, the H ( X / Y) < H { p(e), 1 -p (e ) } + p(e) log (s - 1). (1.2) I the developmet of the above-metioed boud, we utilize several theoretic quatities as defied by Shao. These are the joit etropy, Coditioal etropy, ad mutual iformatio. For a discrete radom variable X, Shao s etropy [6] is give by H (X) = - p ( Xi)lo g p (x i). (1.3) Based o this defiitio, the joit etropy, mutual iformatio ad coditioal etropy are defied as h (x, y> I (X, Y) = -, z, p(xi>y )i gp(xi>yj) j=l X X p(xi>yj)iogp(xi,yj) j= l p(xi) p(yj) where H ( X / Y) = X H ( X /y j)p (y j) f j=l H (X / yj) = - X P(Xi / yj) log p(xj / yj) i=4 (1.4 ) (1.5) ad p (Xi, yj) ad p(xs / yj) are respectively the joit ad the coditioal probabilities of X ad Y. Reyi s etropy [5] for X is give by H <;(X) = log ^ p (x i), (1.6) 1 - a i=1 W h e re a is a rea l p o sitiv e c o sta t d iffe re t fro m 1. T h e (av e ra g e) m u tu a l i fo rm a tio a d (av erag e) coditioal etropy are cosequetly H ( X,Y ) = - L log Y. Z P (Xi,yj) 1 - i= l j= l I«(X,Y) = - L 1 - a where H a(y / X) lo Z Z 1=1 j=1 = ZpCxOH^Y/Xi), { Pa (xi, yj)} / { pa'1 (xo p^ 1(yj) } (1.7) (1-8) (1.9) H a(y / x 0 = L log X pa (yj/xi) l - a H 60 D IA S TECHNOLOGY REVIEW VOL. 2 No. 1 APRIL - SEPTEMBER 2005

4 FANG S INEQUALITY A large amout of work o probability of error has bee doe by M.E. Heilma ad J. Raviv [3], D.G. Laiiotis [4]. I this paper, we exted our idea of Fao s boud o the probability of error to a family of lower bouds based o Reyi s defiitio of etropy ad mutual iformatio. We relate the probability of error of a code to Reyi s etropy, a geeralizatio of Shao s etropy. I sectio I, A systematic method of computig Fao s boud for probability based o Reyi s iformatio is preseted ad i sectio II, the lower boud for the average probability of error is calculated i terms of chael capacity by usig Reyi s etropy. Shao measure does ot deped upo extraeous factors. But i practical situatios extraeous factors plays a importat role. I this paper, Bouds derived for probability of error depeds upo parameter a, which represets these extraeous factors such as evirometal factors, cost factors etc. As a particular case whe a -> 1, our result reduces to that oe correspodig to Shao s etropy [6]. FANO S INEQUALITY USING RENYI S ENTROPY I order to fid the Fao s boud for probability based o Reyi s iformatio we use Jese s iequality, which is as follows: Assume g (x) is covex (if cocave reverse iequality), x E [a, b] the for Wi = 1, w ; > 0, w e have g t Z Wi Xi ] < I W i g ( X i ). We also write the coditioal probability of error give a specific class as p (e / xi) = X p (yj! xi) 1 - P (e / X i ) = p (ys / X i ) (2.1) (2.2) (2.3) Theorem : 2.1 G ive a arbitrary code (s, ) cosistig of w ords x(1), x(2),...x(s). L et X = (Xj, X2, X3,... X) be a radom vector that equals x(l) with probability p(x(l)), i = 1,2,3,...s, where p(x0)) = 1. [I other words, we are choosig a code word at radom accordig to the distributio p(x(1)) ]. Let Y = (Yi, Y2,...Y) be the correspodig output sequece. If P(e) is the probability of error of the code, computed for the give iput distributio, the H «(X /Y ) < H a { p(e), 1 - p(e) } + p(e) log (s - 1). (2.4) Proof: Cosider Reyi s coditioal etropy [5] of Y give H a(y / Xj) = - i - log p (yj /xi) l - a i=1 l 1 -a log [ z: Pa(yj/Xi)+ P (yi/xi)] i*j (2.5) l-a log [ pa (e / X;) X { p (yj / xo / p (e / xo } + { 1 - p (e / x0 }a ]. Usig Jese s iequality, (2.2) ad (2.3), we obtai two iequality for a > 1 ad a < 1 cases a > 1 V 1 H a(y / x i) < p(e / Xi) log p ' 1 ( e / Xj) 1 P (y j1 x0 / P (e / x0 }a l - a yj + { 1 - p (e / X i ) ) 1 log { 1 - p (e / X i ) l-a (2.6) DIAS TECHNOLOGY REVIEW VOL. 2 No. 1 APRIL - SEPTEMBER

5 FANO S INEQUALITY a < 1 or H a(y / x j) > p(e / Xj) log p 1 ( e / xo { P (yj / *0 / p (e / xs) }a l-a ^ Recall that for (s - 1) poit etropy, we have + { 1 p (e / Xi) } log {1 - p (e / Xi) J0'1 (2.7) 1 - a = H (e / Xi) + p (e / xj) log X { P (Yj / xo / p (e / Xj) }a. l - a log X { p (yj/xi)/p (e/xo } < lo g (s-l). (2.8) equality beig achieved for a uiform distributio. Hece, for a > 1 from (2.6) ad (2.8) we obtai H a(y / x ;) < H a (e/xo + p (e/xo log ( s - 1). Fially, usig Baye s rule o the coditioal distributios ad etropies we get the lower boud for P(e). or H a(y / X) < H a (e) + p(e) log ( s - 1) H a ( X / Y) < H a { p(e), 1 - p(e) } + p(e) log (s - 1). T h e o re m : 2.2 The average probability o f error p(e)of ay code (s, ) satisfy p(e) > 1 - ( C + log 2) / ( log s) w here C a is the chael capacity. C osequetly if s > 2 (C + 6) w here 5 > 0, the (2.9) (Ca + 5) < Ca+ l or p(e)> 1 - (C + 1/) / (C a + 5) ^ 1 - [ C a / (C a + 5) ] Thus if R > Ca, o sequece o f codes ([2R], ) ca have a average probability o f error w hich -> 0 as > oo, hece o sequece of codes ([2R],, A*,) ca exist w ith lim X = 0 oo P ro o f: C h o o se a co d e w o rd a t ra dom w ith all w o rd s e q u a lly lik e ly, that is let X ad Y b e as i the Fao s iequality with p (x(0) = 1/s, i = 1,2,..., S. The H (X) = log S SO that I«(X/Y) = logs-h a(x/y) (2.10) L e t X i, X 2,... X be a seq u e ce o f iputs to a d iscrete m e m o ryless cha el, ad Y j, Y 2,...Y the correspodig outputs. The I a (Xi, X2,..., XD/ Yi, Y2,...,Y) < ^ I a (Xj / Yj) with equality if ad oly if Yi, Y2,...Y are i= l idepedet. Usig above, we have I«(X/Y) < X I a (Xj / Yi) (2.11) 62 DIAS TECHNOLOGY REVIEW VOL.Y No. 1 April - September 2005

6 FANG S INEQUALITY Sice I a (Xj / YO < C a (by defiitio of capacity), (2.10) ad (2.11) yield logs H a (X /Y ) < Ca (2.12) By Theorem (2.1), Hece H a( x / Y) < H a { p(e), l- P ( e ) }+ p (e) log(s-i) H a (X / Y) < log 2 + p(e) log (s) (2.13) The result ow follows from equatio (2.12) ad (2.13). i.e lo g s < ( C a + log 2) / ( 1 - p(e) ) or p(e) > 1 - ( C a + log 2) / ( log s) PARTICULAR CASES (i) Whe a -> 1 equatio (2.4) reduces to (1.1) refer Ash R. [1] (ii) WTie a -> 1 equatio (2.11) ad (2.12) reduces to (2.8) refer Ash R. [1]. CONCLUSIONS Fao's iequality is a importat outcome i Shao's iformatio theory. This boud is widely appreciated ad has acquired wide applicatio i the differet fields of commuicatio theory. Fao s lower boud has cosiderably sigificat effect as it provides th e a a ly s t to f i d lim it o f a tta i a b le p e r f o r m a c e i commuicatio chael, whereas, the upper boud, o the other had, assures that the worst-case performace of the fial product is improved with i the kow bouds. However, Fao's boud for probability based o Reyi's etropy ad the expressio for average probability of error is discussed i the preset paper. It has amply bee demostrated uder umerical dimesio the applicatio of proposed bouds to realistic situatios (problems) i I REFERENCES commuicatio theory. However, either oe of these bouds ca be utilized i existig practice iterchageably. Fially a cadid view has bee derived from the study is that these kids of iformatio which is geerally theoretic bouds always require a i f o r m a tio w h ic h is g e e ra lly s u ffic ie t to g e t a e s tim a te o f the probability of error itself. As such these bouds could be favourably helpful i determiig the cofidece iterval for this probability. Cocludigly, it ca favourably asserted herewith that with the help of Fao's iequality we ca also propose to derive the relatioship amog etropies. 1. Ash R., Iformatio Theory, Itersciece publishers, New York Erdogmus D. ad Pricipe J.C., Iformatio Trasfer Through Classifiers ad its Relatio to Probability o f Error, Iti. Joit Cof. O Neural Networks, pp , July HellmaM.E. adravivj., Probability o f Error,Equivocatio ad the Cheroff Boud, IEEE Tras. Iform. Theory, vol. IT 16, pp , Laiiotis D.G, A Class o f Upper Bouds o Probability o f Error for Multihypothesis Patter Recogitio, IEEE Tras. Iform. Theory, vol. IT 15,pp ,1969. J. Reyi A., OMeasures o f Etropy ad Iformatio, iproc. 4* Berkeley Symp. Math. A d Probability, vo l l,p p , Shao C.E., A Mathematical Theory o f Commuicatio, Bell system Tech. Joural, vol.27, , *» ' >- mv*<. -»» ^ * DIAS TECHNOLOGY REVIEW V0L.2 No. 1 APRIL - SEPTEMBER