Probability Estimation with Maximum Entropy Principle

Transcription

1 Pape 0,CCG Annua Repot, 00 ( c 00) Pobabiity Estimation with Maximum Entopy Pincipe Yupeng Li and Cayton V. Deutsch The pincipe of Maximum Entopy is a powefu and vesatie too fo infeing a pobabiity distibution fom constaints that do not competey chaacteize the distibution. The pincipe of Minimum Reative Entopy which is a moe genea fom of Maximum Entopy method has a the impotant attibutes of the Maximum Entopy method with the advantage of moe easiy integation of the pio pobabiity distibution. The maximum entopy methods have been successfuy expoed in many discipines. Whie used in discete mutivaiate pobabiity distibution estimation, thee ae some chaenges with the taditiona Lagange mutipie appoach to Maximum Entopy and Minimum Reative Entopy. In this pape, the Iteative Scaing based on Minimum Reative Entopy is used in discete mutivaiate pobabiity estimation which makes the Minimum Reative Entopy pincipe appoach successfuy impementation in discete mutivaiate pobabiity estimation. The pincipe of Maximum Entopy and Minimum Reative Entopy ae intoduced. Then, the chaenges in discete Mutivaiate Pobabiity estimation ae iustated with a sma discete pobabiity estimation exampe. A soution based on iteative scaing is intoduced and expained with two numeica appication exampes. Maximum Entopy Pincipe The pincipe of maximum entopy (ME) is based on the pemise that when estimating the pobabiity distibution, the best estimation esuts wi keep the agest emaining uncetainty (the maximum entopy) consistent with a the known constaints. In that way, no moe additiona assumptions o biases wee intoduced in the estimation [, ]. Geneay, assuming any desied mutivaiate pobabiity is p : {p, =,, N}, the objective function wi be satisfied fom some constaints: Maximum: N H(p) = p Log(p ) () subject to: p = () am p = b m ; m =,,, M () Whee b m ae any ode of magina fo the desied discete mutivaiate pobabiity. Theoem Ony unde the noma constaints, the unifom distibution wi be the maximum entopy soution. Poof The we-known soution to the pobem of optimizing a function subject to constaints is the method of Lagange mutipies[]. The fist step is to fom a new entopy function L(p, λ) as defined beow: L = p og(p ) + λ ( p ) () 0

2 Pape 0,CCG Annua Repot, 00 ( c 00) The second step is equating the deivate of () to zeo with espect to each vaiabes p, =,, N and λ. The esuts wi be an equation set: p = og(p ) + λ = 0 (5) λ = (p ) = 0 (6) Fom equation (5), it is obtained that p = exp(λ ). This is independent of, Thus, a the pobabiities p shoud be equa and sum to. Then the unifom distibution p = /N is the ME estimation. When the fu constaints ae enfoced to the objective function. Using the method of Lagange mutipies, the new objective function L(p, λ, λ m ) wi be defined as: L(p, λ, λ m ) = p og(p ) + λ ( p ) + λm ( bm a m p ) (7) Then equate the deivative of equation (7) to zeo with espect to each of the vaiabes p, =,, N and λ, λ m ; m =,, M that is: p = Logp λ Then fom equation (8), it is obtained that M λ m a m = 0 (8) λ = p = 0 (9) λ m = b m a m p = 0 (0) ( p = exp λ 0 M ) λ m a m In infomation theoy and statistica mechanics, the pefeed fom is witten as: (λ 0 = λ + ) () p = M Λ exp( λ m a m ) () Whee Λ is caed patition function which is a function between λ 0 and a the othe λ m and witten as: N M Λ = exp(λ 0 ) = π exp( λ m a m ) () The pobabiity aw in () is aso named as the Gibbs distibution. The Lagange mutipie λ 0 is caed the potentia equation which has the popety that: λ 0 λ m = b m, m =,, M () Theoeticay, fom equation (), a the needed λ m ; m =,, M can be soved fom the m equations. Howeve, soving the m set of couped impicit noninea equation is a difficut task in pactice. 0

3 Pape 0,CCG Annua Repot, 00 ( c 00) Minimum eative entopy pincipe In pobabiity and infomation theoy, the eative entopy which is aso known as Kuback-Leibe divegence, infomation divegence, infomation gain is an asymmetic measue of the diffeence between two pobabiity distibutions [, 5, 6, 7]. Conside two sets of discete pobabiity (p,, p N ) and (π,, π N ). The eative entopy between them which is aso a measue of the diffeence of the infomation contained in them is defined as: J[p π] = p og p (5) π It is we known that: J[p π] 0 (6) J[p π] = 0 if ony if p = π (7) The Pincipe of Minimum Reative Entopy(MRE) is that given new facts, a new distibution p shoud be chosen which is as had to disciminate fom the oigina distibution π as possibe so that the new data poduce as sma an infomation gain J(p π) as possibe, thus no moe bias except satisfying the constaints ae intoduced. [8] It is witten as: Minimize: J(p π) = p og p π (8) subject to: p = (9) am p = b m ; m =,,, M (0) whee a m is the magina constuction matix. The constaint b m wi be the magina pobabiity and it wi satisfy M b m =. Theoem Maximum Entopy is equivaent to Minimum Reative Entopy pincipe between P and the unifom distibution U. Poof Given the unifom pobabiity distibution U = /N, then Minimum KL divegence esuts wi be : Minimize: J(P U) = = N p og(p N) () n p og p + N og N = H(P ) + (constant) As shown in Equation (), the esuts fom minimizing J(p π) wi bing maximum entopy esuts to H(p). Thus, the ME appoach is a specia case of the MRE. Howeve, the MRE fomuation is moe genea and offes geate fexibiity fo the nu-hypothesis function π can epesent any pobabiity function. 0

4 Pape 0,CCG Annua Repot, 00 ( c 00). Soving MRE by Lagange mutipie The same Lagange mutipies appoach is used in soving the soution. The fist step is to fom a new entopy function L(p, λ, λ m ) as defined beow: L(p, λ, λ m ) = N p og p + (λ 0 ) ( N p ) ( N ) + λ m a m p b m π () Then equate the deivative of equation () to zeo with espect to each desied pobabiities p, =,, N and a the Lagange mutipies, that is: p = og p π λ 0 + M λ m a m = 0 () λ 0 = p = 0 () λ m = b m a m p = 0 (5) Fom equation (),() and (5), the fina estimated pobabiity coud be expessed as: ( p = π exp λ 0 O M ) λ m a m (6) p = M Λ exp( λ m a m ) (7) Whee equation (7) is the expession that used most often in infomation theoy. Same as the ME soution, the expession Λ is caed patition function which is a function of a the Lagange mutipies as: N M Λ = exp(λ 0 ) = π exp( λ m a m ) (8) Whee λ 0 is caed the potentia equation. Same as the ME, fom the potentia equation, taking the deivative accoding to a the othe Lagange mutipies, the esut wi aso be a couped noninea equation set. It coud be soved with some numeica appoach fo vey simpe pobems.. Soving MRE using iteative scaing Fom pevious sections, it is shown that the maximum entopy soution wi be a couped noninea equation set. It coud be possibe soved with some numeica appoach. But unti now, no cea and tanspaent soution have been given. In this section, one kind of iteative scaing soution to the MRE is adopted[, 9]. The MRE fom iteative scaing is said that: Given the constaints in equation (0), thee exists an unique pobabiity distibution ˆP which satisfies them and is the imit of the iteative sequence {p (δ) ; δ = 0,,, } defined by ˆp (0) ˆp (δ+) = π = ˆp (δ) µ M [ bm ˆb(δ) m ] am δ = 0,,, (9) 0

5 Pape 0,CCG Annua Repot, 00 ( c 00) whee ˆb (δ) m = a m ˆp (δ), and µ is used to do nomaization. The poof of unique and convegence of this iteative pocess in Equation (9) is given by Kuback and Khaiat in 966[7]. This method is named as Iteative Scaing(IS) which is continued studied extensivey in mathematics and statistic eseaches[0,,,, ]. Moe geneay, conside R sets of constaints each of them is in the fom of (0), Let the th set constaint be witten as: a mp = b m, =,,, R; m =,, M (0) whee M b m =. In the mutivaiate pobabiity estimation, this th set constaints coud be any ode owe magina pobabiity. Povided that the constaints in (0) ae consistent to each othe, thee exists a unique positive pobabiity distibution p which satisfies them and is of the fom: p = π µ R = M (µ m) a m () which means that p can be obtained as the imit of a cycica iteative scaing pocess[9, 0]. The stating pobabiity of the iteation π can take the unifom distibution which was simpest and most natua choice (Daoch and Ratciff,97)[]. A moe easonabe and pactica choice of π is assuming that a vaiabes ae independent. that is the initia estimation wi be π = P (u α ). Thus, the estimated mutivaiate epesent a geneaized independent distibution subject to the inea constaints(owe-magina distibutions). Hee a simpe discete pobabiity P = {p, p, p, p, p 5 } is used to show one iteative scaing pocess. Let the inea constaints ae 5 a m p i = b m, m =,,, ; =,,,, 5 () Given an magina constuction matix a m, the inea constaints in Equation () witten in taditiona matix fom wi be: p p h p p 0 0 = h h () h p 5 Assuming the initia pobabiity distibution is {p (0), p(0), p(0), p(0), p(0) 5 }, the fist iteative scaing pocess 0 5

6 Pape 0,CCG Annua Repot, 00 ( c 00) wi be poceed as: p () = p (0) ( h p () = p (0) ( h p () = p (0) ( h p () = p (0) ( h p () 5 = p (0) 5 ( h ) ( h ) ( h ) 0 ( h ) 0 ( h ) ( h so afte the fist iteation the pobabiity wi be: p () i = p (0) i µ ) 0 ( h ) ( h ) ( h ) 0 ( h ) ( h = ) 0 ( h ) 0 ( h ) 0 ( h ) ( h ) ( h ) = p (0) ) = p (0) ) 0 = p (0) ) = p (0) ) 0 = p (0) 5 ( h = ( h = ( h = ( h = ( h = ) a ) a ) a ) a ) a5 [ h ] a i i =,, 5 () whee µ is used to nomaization to make it a pobabiity. Using the same iteative scaing pocess the iteated sequence {p (n) ; n = 0,,, } can be obtained, the imit of this sequence woud be the soution satisfying the inea constaints. Numeica exampe of iteative scaing The die pobem The die pobem seves as an iustation and of diffeent soution effots fom iteative scaing and Lagange mutipie. This pobem was oiginay poposed by Jaynes as an exampe to show the ME pincipe fo undetemined pobem[]. It wi show thee is no diffeence in the fina esuts fom using Lagange mutipie and iteative scaing appoach. Howeve, the pocedue of IS is moe staightfowad than the Lagange appoach. Conside a die of six faces is tossed fo T (T + ) times. One is tod that the aveage numbe of spots up was not.5 as we might expected fom an honest die but.5. Ony given this infomation, and nothing ese, what pobabiity shoud one assign to i spots on the next toss? Fom maximum entopy appoach, the soution coud be poceed as foowing pocedues. constaints to the entopy equation woud be: the new objective function with Lagange mutipies woud be: i= The i p i =.5 (5) i= p i = (6) i= L = p i og p i + λ 0 ( p i ) + λ ( i p i.5) (7) i= i= 0 6

7 Pape 0,CCG Annua Repot, 00 ( c 00) Afte doing the cassica optimization pocedue, the desied pobabiity woud be: p i = exp(λ 0 + iλ ) (8) Λ = exp(λ 0 ) = exp( iλ ) (9) i= Λ = x( x) ( x 6 ) whee: x = exp( λ ) (0) Λ λ =.5 popety of patition function () x 7 5x 6 + 9x 7 = 0 () Afte obtaining the desied oot fo the above equation, the maximum entopy pobabiities wi be: {0.055, , 0.6, 0.655, 0.977, 0.79} () Whie fo iteative scaing pocess, the magina constuction function woud be: a i = {,,,, 5, 6}. Accoding the iteative scaing pocess, the fist estimation to the desied pobabiity woud be: p 0 i : {/6, /6, /6, /6, /6}. The fist time of scaing woud be: p = µ p 0 (.5.5 ) p = µ p 0 (.5.5 ) p = µ p 0 (.5.5 ) p = µ p 0 (.5.5 ) p 5 = µ p 0 5 (.5.5 )5 p 6 = µ p 0 6 (.5.5 )6 Afte doing nomaization, the estimation esuts fom IS appoach woud be: {0.08, 0.0, 0.8, 0.765, 0.98, 0.850} A the ten times iteation esuts ae isted in tabe compaing the Lagange esut and the IS esut, they ae amost exacty the same. But the IS pocess is moe staightfowad and tanspaent. Expet intepetation pobem It can sti obtain the anaytica equation in the dice pobem; howeve, in genea it is not possibe. Suppose that an expet needs to estimate the ock type popotion fo one outcop. At fist situation, he ony knows thee ae five ock types {mud, sand, imestone, doomite, anhydite} in this outcop and without futhe sedimentay envionment anaysis infomed. In this case, without moe infomation in hand, the most intuitivey appeaing estimated popotion woud be: p(mud) = p = /5 p(sand) = p = /5 p(imestone) = p = /5 p(doomite) = p = /5 p(anhydite) = p 5 = /5 0 7

8 Pape 0,CCG Annua Repot, 00 ( c 00) iteation time p p p p p 5 p Tabe : Ten times iteation esuts fo the di pobem Suppose afte knowing the sedimentay backgound, it is known that eithe mud and sand woud have a 0% popotion to exist. And aso, in haf the cases, the expet expects mud and imestone woud be exist fom the out cop. These two piece of infomation ae incopoated into the estimation as two constaints: p [ ] p [ ] p / p = () / p 5 Fom the ME appoach, the objective function woud be: L = 5 5 p og p + λ 0 ( p ) + 5 λ m ( a m p ) (5) Using the cassica deivative appoach, the fina expession fo the desied pobabiity woud be: p = exp(λ 0 + Then the patition function woud be witten as: Λ = exp(λ 0 ) = λ m a m ) (6) 5 exp( λ m a m ) (7) Theoeticay, one set of noninea equation set can be obtained as: λ 0 = og( 5 exp( λ ma m )) = / (8) λ λ 5 exp( λ 0 = og( λ Obtaining the soution fo them is not easy. λ ma m )) λ = / (9) Whie using IS, the pobabiity estimation pocess woud be moe simpe. The initia estimation 0 8

9 Pape 0,CCG Annua Repot, 00 ( c 00) iteation time p p p p p Tabe : Ten times iteation esuts fo the expet tansato pobem woud be {/5, /5, /5, /5, /5}, the fist iteation pocess is: p = µ p 0 ( / /5 ) ( / /5 ) p = µ p 0 ( / /5 ) ( / /5 )0 p = µ p 0 ( / /5 )0 ( / /5 ) p = µ p 0 ( / /5 )0 ( / /5 )0 p 5 = µ p 0 5 ( / /5 )0 ( / /5 )0 Afte ten times iteation, the constaints ae vey cosey satisfied as shown in tabe. Concusion The pincipe of Minimum Reative Entopy (MRE) is a moe genea fom of the ME method. MRE has a the impotant attibutes of the ME method with the advantage of moe easy integation of the pio pobabiity distibution. Whie used in discete pobabiity distibution estimation, thee ae some chaenges. The iteative scaing soution to the MRE pincipe in discete mutivaiate pobabiity estimation is moe staightfowad and easy to impement. Using the iteative scaing makes its industia impementation possibe. Refeences [] E. T. Jaynes. Infomation theoy and statistica mechanics. Physica Review, 06(6):60 60, 957. [] E. T. Jaynes. Infomation theoy and statistica mechanics. ii. Physica Review, 08():7 90, 957. [] Edwin T. Jaynes. Whee do we stand on maximum entopy? In Raphae D. Levine and Myon Tibus, editos, The maximum entopy fomaism. The MIT Pess, May 978. [] H. Ku and S. Kuback. Appoximating discete pobabiity distibutions. Infomation Theoy, IEEE Tansactions on, 5(): 7, Ju

10 Pape 0,CCG Annua Repot, 00 ( c 00) [5] S. Kuback. Lette to the edito: The kuback-eibe distance. The Ameican Statistician, ():0, 987. [6] S. Kuback and R.A. Leibe. On infomation and sufficiency. Annas of Mathematica Statistics, :79 86, 95. [7] S. Kuback and M.A. Khaiat. A note on minimum discimination infomation. The Annas of Mathematica Statistics, 7:79 80, 966. [8] Jone E. Shoe and Rodney W. Johnson. Axiomatic deivation of the pincipe of maximum entopy and pincipe of minimum coss-entopy. IEEE Tansactions on Infomation Theoy, 6:6 7, 980. [9] Hay H. Ku and Soomon Kuback. Loginea modes in contingency tabe anaysis. The Ameican Statistician, 8():5, 97. [0] Yvonne M. M. Bishop. Fu contingency tabes, ogits, and spit contingency tabes. Biometics, 5():8 99, 969. [] J. N. Daoch and D. Ratciff. Geneaized iteative scaing fo og-inea modes. The Annas of Mathematica Statistics, (5):70 80, 97. [] D. V. Gokhae. Anaysis of og-inea modes. Jouna of the Roya Statistica Society. Seies B (Methodoogica), ():7 76, 97. [] I. J. Good. Maximum entopy fo hypothesis fomuation, especiay fo mutidimensiona contingency tabes. The Annas of Mathematica Statistics, ():9 9, 96. [] P.M. Lewis. Appoximating pobabiity distibutions to educe stoage equiement. Infomation and conto, : to 5,