CHERNOFF DISTANCE AND AFFINITY FOR TRUNCATED DISTRIBUTIONS *

Transcription

1 haper 5 HERNOFF DISTANE AND AFFINITY FOR TRUNATED DISTRIBUTIONS * 5. Inroducion In he case of disribuions ha saisfy he regulariy condiions, he ramer- Rao inequaliy holds and he maximum likelihood esimaor converges o normal disribuion, hose variance is he inverse of he Fisher informaion. Therefore, he Fisher informaion consolidaes he amoun of accessible informaion for a regular family of disribuions. Hoever, in a non-regular locaion shif family ha is generaed by a disribuion in R, hose suppor is no R, he Fisher informaion diverges and some imes canno be defined. Therefore, in order o characerize he bound of asympoic performance in esimaion, e need an informaion quaniy generalizing Fisher informaion. Akahira (996) proposed he limi of he hernoff disance (relaive Renyi enropy) as a subsiue informaion quaniy for a non-regular locaion shif family. Hayashi (007) has examined he relaionship of his measure ih Kullback-Leibler divergence measure. In he presen chaper, e exend he definiion of hernoff disance, for runcaed disribuions and examine is properies. Le X and Y be o non-negaive random variables ih absoluely coninuous disribuion funcions F( x ) and Gx and ih same suppor. Denoe by f ( x ) and gx he corresponding probabiliy densiy funcions. Then he hernoff disance beeen F( x ) and G( x ) is defined as = ; 0< <. (5.) 0 FG (, ) log f ( xg ) ( xdx ) * Some of he resuls in his chaper have been published in (a) Smiha S, Nair KRM and Sankaran PG (007) (b) Nair KRM, Sankaran PG and Smiha S. 7

2 This measure is an example of he Ali-Silvey class of informaion heoreic disance measures. The hernoff disance is alays non-negaive ih zero disance occurring eiher hen = 0, or hen he probabiliy disribuions are same. In equaion (5.), he parameer can be inerpreed as he eigh assigned o he disribuions hile compuing he disance beeen hem. Saisical uiliy of hernoff disance is ha if one uses a Baye s procedure for esing f ( x ) agains gx, hen FG (, ) is asympoically n imes he negaive logarihm of he Bayes risk for disinguishing he o. Asadi e al. (005) have exensively sudied he applicaion of his measure in he conex of reliabiliy sudies. 5. Definiion and properies Le X and Y be o non-negaive random variables ih absoluely coninuous disribuion funcions F( x ) and Gx respecively and ih densiy funcions f ( x ) and gx. onsider he random variables X = X X > and Y = Y Y >, > 0. (5.) Then he hernoff disance beeen FG (, ; ) = XY (, ; ) = X and Y akes he form f( x) g( x) =log dx, (5.3) F G here F and G are he survival funcions of X and Y. In he righ runcaed siuaion, he random variables under consideraion are X = X X < * and Y = Y Y <, > 0, * 7

3 and equaion (5.) becomes ( F, G;) = ( X, Y;) = * * * Noice ha hen 0 f( x) g( x) =log dx. (5.4) F G 0 *, ( F, G) and hen ( F, G) ( F, G ) is defined by equaion (5.). In erms of he hazard raes, equaions (5.3) and (5.4) can be rien as ', here = ( ) h h + h h e (5.5) and ' * = λ + λ λ λ e, (5.6) * here h and h are he hazard raes, λ and λ are he reversed hazard raes of F and G( ) respecively and ' and * ' represens he derivaives of and * ih respec o. We no discuss some properies of he runcaed versions of he hernoff disance defined in equaions (5.3) and (5.4). Theorem: 5. If φ (.) is an increasing funcion in he argumen and > 0, hen (, ; φ ) = ( φ, φ; ) X Y X Y and (, ; φ ) ( φ, φ; ) * * X Y X Y =. Proof The proof follos direcly from he definiions (5.3) and (5.4). 73

4 In he nex heorem, e obain an inequaliy concerning he measures defined in equaions (5.) and (5.3) under some mild condiions. Theorem: 5. Le X and Y are o non-negaive random variables ih disribuion funcions F( x ) and G( x ) and ih probabiliy densiy funcions f ( x ) and gx respecively. Denoe by Suppose ha h x and h x he failure raes of F and G. (i) h h and ( x) ( x) is increasing in x (ii) boh F and G are NBU. Then Proof Le ( F, G). X and Y denoe he runcaed random variables. Denoe he disribuion funcions of X and Y by F ( x ) and G probabiliy densiy funcions f ( x ) and x and he corresponding g x. Denoe by FG, (, ) he hernoff disance beeen F ( x ) and G( x ) defined by equaion (5.) and runcaed hernoff disance, defined in equaion (5.3). By aking, x = F ( y), y ( 0,),equaion (5.3) becomes he lef g( F ( y)) =log dy ; 0< <. (5.7) f ( ) 0 F y Using he definiion of hazard rae, e have 74

5 ( ) ( ) f F y = h F y F F y. (5.8) F Furher since x F ( y) = e have ( ) = F F y F x In vie of he fac ha ( ) f F y ( + ) F F F y =. (5.9) ( ) F f F y =, (5.0) + using equaions (5.9) and (5.0) in (5.8) e ge F ( ) h F y = ( + ) + f F y ( ) F F y = h F ( y) +. (5.) Equaion (5.8) no becomes ( ) ( ) f F y = h + F y F F y. (5.) From equaion (5.9), e have + F y = F y F (5.3) and ( ) F F y = y. Then, equaion (5.) can be rien as ( ) ( ( ( ) ))( ) f F y = h F y F y.. (5.4) Similarly, e ge 75

6 ( ) ( ) g F y = h F y G F y. (5.5) G Since be rien as ( ) h F y = h + F y, using equaion (5.3), equaion (5.5) can G ( ) ( ) ( ) ( ) g F y = h F y F G F y. (5.6) Dividing equaion (5.4) by equaion (5.6), e obain ( ) ( ( )) ( ) ( ) f h F y F G y y = g G y G F y h F y F Since ( ), from he condiion (i), e ge F y F F y ( ( ( ) )) ( ) ( ) ( ) h F y F h F y h F y F h F y Since F and G are NBU, e have ( ). (5.7). (5.8) G F y G F y. (5.9) Subsiuing equaions (5.8) and (5.9) in equaion (5.7), e ge Tha is, ( ) ( ) ( ) ( ) f F y h F y y. g F y h F y G F y ( ) ( ) ( ) f F y f F y g F y g F y. (5.0) Using equaion (5.0), equaion (5.7) becomes 76

7 gf ( ( y)) log f( F ( y)) 0 = FG (, ). dy The implicaion of his propery is ha he disance beeen o sysems of age is never smaller han he disance hen he sysems ere ne. The folloing heorem provides a sufficien condiion for he monooniciy of Theorem: 5.3. For he random variables X and Y considered in Theorem 5., assume ha (i) h h ( x) ( x) and is increasing in x, (ii) Then boh F and G are IFR. FG (, ; ) is increasing in. Proof When (ii) holds, e have, for all 0, 0< y <. G F y G F y ( ) ( ) Assumpion (i) implies F G ( { ( ). }) ( { ( ). }) h F y F h F y F is increasing in 0, 0 < y <. hich implies ( ) ( ) f F y is increasing. g F y 77

8 Tha is, ( ) ( ) g F y is decreasing. f F y Hence g[ F ( y)] FG (, ; ) =log dy is increasing in 0. f [ ] 0 F y In he nex heorem, e provide a bound for he runcaed hernoff disance in erms of he hazard raes. Theorem: 5.4 For he random variables X and Y considered in Theorem 5., if X is larger (smaller) han Y in he likelihood raio order ( X lr ( lr ) Y ) hen Proof h log, > 0, 0< <. (5.) h If ( X lr ( lr ) Y ) There fore, e have f x f ( ) ; x, > 0. g x g f g is increasing (decreasing) in. This gives f x g x h dx ( ) F G h. This implies f x g x h log dx log F G h. (5.) 78

9 Since f x g x = log dx, F G using equaion (5.), e ge h. log h This complees he proof. An analogous bound can be obained in he righ runcaed siuaion. This is given as Theorem 5.5 belo. Theorem: 5.5 If he random variables X and Y as defined as in Theorem 5., and if X ( lr lr Y ) is larger (smaller) han Y in he likelihood raio order X, hen λ ( ) log, > 0, 0< <. λ * The proof of he resul is similar o ha of Theorem 5.4 and hence omied. Remark: 5. X is smaller han Y in he likelihood raio order ( X Y ) implies ha h h for all > 0. Thus he righ side of he expression (5.) is non- negaive for all > 0. If is increasing, lr 0, here ' ' is he derivaive of From equaion (5.5), i follos ha. log + h h h h, > 0. 79

10 Remark: 5. Denoe by DFG (, ; ) is he modified Kullback-Leibler divergence measure defined in equaion (.48). I is immediae ha here exis he folloing relaionship beeen and Kullback-Leibler divergence measures, namely lim = DGF (, ; ) 0 and lim = DFG (, ; ). I may be observed ha he folloing funcional relaionship exiss beeen HE and, namely ( E ) =log H, f x g x here HE = dx is he Hellinger s disance for runcaed F G random variable. 5.3 haracerizaion heorems In his Secion, e look in o he siuaion here he runcaed hernoff disance is independen of. Theorem: 5.6 Le X and Y be o non negaive random variables admiing absoluely coninuous disribuion funcions and le defined as in equaion (5.3). is independen of if and only if ( YG, ) is he proporional hazards model of ( X, F ). 80

11 Proof The runcaed hernoff disance order defined by Renyi (96) hrough he relaionship = pk. f, g;, is relaed o he Renyi divergence of here, p =. Asadi e al. (005) has proved ha K ( f, g; ) is independen of if and only if F and G saisfy he condiion for being a proporional hazards model. The proof of he heorem is immediae from he above observaions. The folloing example describes an applicaion of he above heorem in he conex of series sysems. Example: 5. Le X, i =,,... n denoe he life imes of he componens in series sysem. i Assume ha he probabiliy densiy funcion of life imes is f ( x ) and ha survival funcion is F( x ).The lifeime of he sysem is hen n = { } ih probabiliy densiy funcion g( x) n F( x) f ( x) Y Min X, X,... Xn and ih survival funcion G( x) F( x) n = =. Observe ha X i, i =,,... n and Y saisfies he condiion for being a proporional hazards model. Furher in vie of equaion (5.3), e have e n f ( x) n F( x) f ( x) = n F ( F ) dx n = F + On simplifying equaion (5.3), e ge n ( n )( ) f x F x dx. (5.3) 8

12 = n n, hich is independen of as claimed in Theorem: ( ) The implicaion of he above resul is ha hen a sysem of componens ih life disribuion F ( x ) are in series he hernoff disance beeen he disribuion of minimum and he original disribuion F( x ) depends only on he number of componens and he parameer. Theorem: 5.7 Le X and Y be o non-negaive random variables ih disribuion funcions F ( x ) and G( x ) and ih probabiliy densiy funcions f ( x ) and gx respecively. Then only if he relaion holds. Proof F( x) = G( x) θ ; θ > 0, * defined in equaion (5.4) is independen of if and This resul follos from Theorem: of Asadi e al. (005), ho has considered he Renyi divergence of order beeen o disribuions for pas life ime namely * f x g x K ( f, g; ) = log dx F G 0 * I is esablished ha K ( f, g; ). is independen of if and only if F and G have proporional reversed hazards raes. In vie of he fac ha ( f, g; ) * K is funcionally relaed hrough he relaionship * and * * (, ; ) = pk f g, here p =, he heorem follos. 8

13 The folloing example shos an insance involving applicaion of he above heorem in he case of a parallel sysem. Example: 5. Le { } X, X... X n be independen and idenically disribued random variables represening he lifeime of he componens in a parallel sysem ih probabiliy densiy funcion f ( x ) and disribuion funcion F ( x ). The lifeime of he sysem is hen Y Max{ X X X } =,,... n ih probabiliy densiy funcion n = ( ) and disribuion funcion G( x) ( F( x) ) n g x n F x f x =. Here X i and Y saisfy he condiion for he proporional reversed hazards model. By direc calculaion using equaion (5.4), one can conclude ha he runcaed hernoff disance *, is independen of. In he nex secion, e invesigae he behavior of he hernoff disance beeen he original and eighed disribuions. 5.4 hernoff disance beeen original and eighed disribuions hernoff disance beeen he original random variable X and eighed random variable X, revieed in Secion.4, akes he form ( x) =log f ( x) f ( x) dx ; 0< <, (5.4) 0 ( x) f ( x) here f ( x) =, ( ) E( ( X) ) is he eighed disribuion. E X <, (5.5) In he lef runcaed siuaion, equaion (5.4) becomes f x f x =log dx ; 0< <. (5.6) F F 83

14 When e consider he lengh-biased model, given in equaion (.4), he above equaion becomes ( x) f, (5.7) µ FL L = ( ) log log x dx F F here, FL is he survival funcion of he lengh biased random variable Furher e have X L. and µ FL = v (5.8) F ( ) = E X X, is he vialiy funcion ( x) x dx E X X f F ( ) = > = v. (5.9) Using (5.8) and (5.9) in (5.7), e ge L ( ) log log ( ) = v v (5.30) The ideniy (5.30) enables one o find he non parameric esimaor of L from he non parameric esimae of v. In he case of equilibrium model namely F( x) fe x =, X E > 0, µ = E( ( X) ) <. µ Equaion (5.6) becomes ( ) E m ' ( ) hm e m + ' E =, 84

15 here, h and m are he hazard rae and mean residual life funcion of original random variable and beeen original and equilibrium random variable. Theorem: 5.8 E is he lef runcaed hernoff disance Le X and Y are o non-negaive random variables ih disribuion funcions F ( x ) and G( x ). Then eigh funcion has he form Proof F θ is independen of if and only if he =, θ > 0, θ. (5.3) Suppose ha = k, a consan. Then equaion (5.6) becomes f ( x) f ( x) log dx = k. (5.3) F F Differeniaing (5.3) ih respec o and assuming ha e ge (( ) ( ) f x f x ) lim = 0, x h k h e h = h. (5.33) Subsiuing h h = c in (5.33) and differeniaing ih respec o, e ge k ( ) ' c c e = 0. 85

16 ' The above equaion gives is consan, say θ. k e c = 0 or c =. In eiher case c( ) This gives θ h = h, θ > 0, θ. (5.34) Bu e have he relaion here h = h m, (5.35) ( ) m = E X X >. (5.36) From (5.34) and (5.35), e have m =. (5.37) θ Using (5.36) and (5.37), e ge F ( x) f ( x) dx=. (5.38) θ Differeniaing (5.38) ih respec o and assuming ( ( x) f ( x) ) lim = 0, x e ge (5.3). onversely assume ha F θ =. 86

17 We have he relaion m =, (5.39) F F µ here µ = E( ) =. θ Using equaion (5.39) in equaion (5.6), e ge Theorem: 5.9 θ =, hich is independen of. θ + For he random variables X and Y considered in Theorem 5.8, he eigh funcion of model (5.5) is increasing (decreasing) in, hen h log h, 0< <, (5.40) here h and h are he hazard raes of X and X respecively. Proof Suppose ha is increasing (decreasing) in. From (5.5), e ge f f Then is increasing (decreasing) in.this gives f f x ( ), x. f f x f x f x h dx ( ) h F F. Using he above expression, (5.6) becomes h log h. 87

18 orollary: 5. When = (lengh biased model), e have L log + m, > 0. h The resul follos from Theorem 5.9 and he relaionship + m =, h here m is he mean residual life funcion. orollary: 5. When = (equilibrium disribuion) is increasing (decreasing), h E log ( ) h m. L 5.5 Affiniy for runcaed disribuions As poined ou in haper he concep of affiniy defined by equaion (.50), is exensively used as a useful ool for discriminaion among disribuions. The measure of affiniy given in equaion (.50) is a special case of he hernoff disance defined in equaion (5.). In fac, hen = equaion (5.) reduces o log ρ, here ρ is he measure of affiniy given in equaion (.50). Affiniy finds applicaion in several pracical siuaions. In he reliabiliy conex, he concep of affiniy helps an experimener o decide heher he disribuion of life imes for o componens differ or are closer. hiy Babu (973) has used he concep for he exracion of effecive feaures from imperfecly labeled paerns. omaniciu e al. (000) used his measure o examine he similariy in images or secion of images in communicaion neorking. There are several pracical insances here complee daa are no available o he experimener. For insance, in life esing experimens he daa on failure imes are usually runcaed. Moivaed by his, in he presen Secion e exend he definiion of 88

19 affiniy o he runcaed siuaion. I may be noed ha he proposed measure is an exension of he Bhaacharyya measure given in Thacker e al. (997). In reliabiliy sudies, if X and Y represens he lifeime of o sysems, hen X and Y, defined in equaion (5.) represen he remaining life of he sysem. The affiniy beeen X and Y is a measure of similariy beeen he disribuion of he residual lifeime of he sysems. For insance, if X and Y represens he amoun of profi of o firms and is he ax exemp level, hen he affiniy beeen represens he similariy beeen he axable incomes of he o firms. beeen here X and Y Using he definiion for affiniy, given in equaion (.50), he affiniy X and Y akes he form (, ; ) A F G = f y g y dy, (5.4) f funcions of l 0 f ( + y) ( y) = and g ( y) F X and survival funcions of X and Y. Equaion (5.4) can also be rien as ( + y) G g = are he probabiliy densiy Y and F = P( X > ) and G P( Y ) = > are he Al( F, G; ) = Al = f ( x) g( x) dx. (5.4) F G In he case of righ runcaed random variables, he variables under consideraion are X = X X < and * affiniy urns ou o be Y = Y Y < are he measure of * Ar( F, G; ) = Ar = f ( x) g( x) dx. (5.43) F G 0 89

20 In vie of he fac ha he measure of affiniy, defined in equaion (.50) is a special case of general hernoff disance, hen =, e ge = log Al, he properies and characerizaions based on he hernoff disance can be suiably reformulaed in he conex of affiniy. Since affiniy is more ofen used in lieraure as a poenial measure of discriminaion, he formulaion of characerizaion resuls in his frame ork seems o be in order. In he sequel, e sae some imporan characerizaion resuls using he concep of affiniy. The proof of he resuls are similar o ha of hernoff disance discussed above. Theorem: 5.0 Le X and Y be o non-negaive random variables admiing absoluely coninuous disribuion funcions F ( x ) and G( x ) and probabiliy densiy funcions f ( x ) and g( x ) respecively. A l, defined by equaion (5.4) is independen of, if and only if ( YG, ) is he proporional hazards model of (, ) Proof X F. When A l is independen of, e have from equaion (5.4) = f x g x dx c F G, here 0< c < is a consan, no depending on. Differeniaing he above equaion ih respec o and using he condiion ( f ( x) g( x) ) lim = 0, x e obain or c ( ) f g = g F f G F G g F f G = +. c f G g F 90

21 The above equaion can be rien as h h c h h = +. (5.44) here h and h are he hazard raes of F and G respecively. Denoing by k h =, h equaion (5.44) akes he form 4 + =. c ( k ) k This gives k( ) is a consan (sayθ ), independen of Thus θ h = h, or equivalenly G x = F x θ θ > 0, (5.45) as claimed in heorem. onversely, hen equaion (5.45) holds, e have θ θ g = f F. (5.46) Using equaion (5.46) in equaion (5.4), e ge θ ( ) θ Al = F x f x dx. (5.47) F G 9

22 A l θ =, θ + hich is independen of, and he sufficiency par follos. In he righ runcaed siuaion, he propery ha A r is consan is characerisic o he proporional reversed hazards model. This resul is saed as Theorem 5. belo. Theorem: 5. Under he condiions of he above heorem, independen of if and only if he relaionship ( F ) G holds for all 0 model of ( X, F ). Proof φ A r defined in equaion (5.43), is = ; φ > 0, (5.48) >. Tha is hen (, ) When equaion (5.48) holds, e have φ φ ( ) YG is he proporional reversed hazards g = f F. (5.49) Using equaion (5.49) in equaion (5.43), e ge φ + φ F G Ar = φ ( F ) F G Ar. φ The soluion of he above equaion is A r φ =, a consan. φ + The proof of he converse par is similar o ha of Theorem: 5. 0 and hence omied. 9

23 Noe: ha Insead of assuming he condiion ha A l is independen of, if e assume A l is linear in, say Al = a+ b, here a and b are consans, he folloing relaionship beeen he hazard raes of F and G is immediae. + h h ( a + b) h h = a, here h and h are he hazard raes. Similarly, in he case of A r, e ge λ + λ ( a + b) λ λ =a, here λ and λ are he reversed hazard raes of F and G respecively. In cerain cases, he dependence srucure may be such ha G( x ) a eighed disribuion obained from F( x ). Denoe by survival and probabiliy densiy funcions of F and f, he X, he eighed random variable. The affiniy beeen he original and eighed random variables, namely X and X, akes he form f x f x =, (5.50) A dx F F ( x) f ( x) here f ( x) =, E ( X ) E ( X) The relaionship connecing A equaion (5.50), and is given by <. (5.5) and hazard raes are immediae from 93

24 + ' h h A = A h h d d variables X and ' here A = A, h and h X respecively., (5.5) are he hazard raes of he random 5.6 Relaionship ih oher discriminaion measures (i) Bhaacharyya disance Firs, e discuss he relaionship beeen he Bhaacharyya disance [Kailah (967)] and he affiniy in he runcaed siuaion. Equaion (5.4) can be rien as F G A = f x g x dx. This is equivalen o l = F G A f x g x dx f x g x dx l 0 0. (5.53) Using he equaions (.50) and (5.43), equaion (5.53) can be rien as ρ F G A Bu e have he relaionship B ρ = e, l = F G A. (5.54) r here B is he Bhaacharyya disance [Kailah (967)]. Then equaion (5.54) no becomes { } B = F G A + F G A. ln l r 94

25 (ii) Modified Kullback- Leibler divergence measure Here e consider he modified Kullback-Leibler divergence measure defined in equaion (.48) namely / / f x f x F D( F, G, ) = log dx F g x G. D( F, G, ) can also be rien as f ( x) g( x) / G D( F, G, ) = log dx. (5.55) F f ( x) / F Using Jensen s inequaliy, equaion (5.55) becomes Tha is, f x g x F D( F, G, ) log dx. F f x G ( x) g( x) f D( F, G, ) log dx. F G From equaion (5.4), he above expression becomes D F, G, loga l. (5.56) Hoever, modified Kullback-Leibler divergence measure is he difference beeen he residual inaccuracy measure and residual enropy funcion, so from equaion (.53), he expression (5.56) can be read as I FG,, H F, loga l, here I ( FG,, ) is he inaccuracy measure in runcaed seup and (, ) residual enropy. H F is he 95

26 (iii) Hellinger s disance Hellinger s disance for runcaed random variable is f x g x HE = dx. (5.57) F G On simplifying, e ge equaion (5.57) as ( x) g( x) f HE = dx F G ( A ) = l. 96