Prior distributios Aledre Tchourbov PKI 357, UNOmh, South 67th St. Omh, NE 688-694, USA Phoe: 4554-64 E-mil: tchourb@cse.ul.edu July 9, Abstrct This documet itroduces prior distributios for the purposes of Byesi sttistics. Usig prior beliefs we c sigifictly improve our sttisticl ifereces bsed o observtios. Most of the books o sttistics do ot cover the mteril preseted. Here we try to collect the iformtio vilble o cojugte priors to certi distributios. Itroductio Accordig to the Byesi rule [], we c epress posterior probbility of certi evet H give some dt with the formul P dt HP H P H dt = P dt The probbility of H give the dt is clled the posterior probbility of H. The posterior equls to the likelihood time the prior divided by mrgil probbility of dt. The pper shows wht priors we c hve d how they ffect posterior distributios give likelihood. Prior d posterior distributios Sometimes prior distributio c be pproimted by oe tht is i coveiet fmily of distributios, which combies with the likelihood to produce posterior tht is mgeble. We see tht objective wy of buildig priors for the biomil prmeter ws to use the cojugte fmily distributio tht hs the property tht the updted distributio is i the sme fmily. I geerl, if the prior distributio belogs to fmily G, the dt hve distributio belogig to fmily H, d the posterior distributio lso belogs to G, the we sy tht G is fmily of cojugte priors to H. Thus, the bet distributio is cojugte prior to the biomil, d the orml is self cojugte. Cojugte priors my ot eist; whe they do, selectig member of the cojugte fmily s prior is doe mostly for mthemticl coveiece, sice the posterior c be evluted very simply. More geerlly, umericl methods of itegrtio would hve to be used to evlute the posterior. I would like to thk professors Heshm Ali d Jiteder Deogu for the opportuity to work o this project
Observtios Prior Posterior Beroulli Bet Bet Poisso Gmm Gmm Biomil Bet Bet Norml Norml Norml Norml Gmm Gmm Tble : Cojugte priors 3 Bet priors From Byes 763: A white billird bll W is rolled log lie d we look t where it stops, scle the tble from to. We suppose tht it hs uiform probbility of fllig ywhere o the lie. It stops t poit p. A red billird bll R is the rolled times uder the sme uiform ssumptio. X the deotes the umber of times R goes o further th W wet. Give X, wht iferece c we mke bout p? Here we re lookig for the posterior distributio of p give X. The prior distributio of p is uiform gp = Uiform, = Bet, =. Give p, X hs biomil distributio P X = p = p p The overll distributio of the umber of successes is the sum of probbilities for ll possible p s P < p < b, X = = p p dp P X = = p p dp Suppose we throw ll + blls o the tble, d choose the red oe. The the probbility tht the red oe hs whites to the left of it is. So we hve + P X = = p p dp = p p dp = + p p dp =!! +! ccordig to defiitio formul for bet fuctio Br, s = we hve X B, p p r p s dp = r!s! r + s! = ΓrΓs Γr + s
P X = = p p dp = B +, + P < p < b X = = p p dp B +, + P < p < b X = = p p dp B +, + which is bet distributio of p with prmeters + d +. The desity fuctio f of the bet distributio is fp = Γ + b ΓΓb p p b, p Emple Suppose tht the prior distributio of p is Bet, b, i.e. gp = p p b B, b Likelihood hs biomil distributio f p = p p The posterior distributio of p give is hp = f pgp f = B, b p p p p b p + p +b dp B, b = p+ p +b B +, + b = Bet +, + b This distributio is thus bet s well with prmeters = + d b = b +. 4 Norml prior Here we follow emple o pge 589 [], which proves the Norml cojugte prior for Norml distributio. The cojugte for Norml likelihood is the Norml distributio. Emple We cosider iferece cocerig ukow me with kow vrice. 3
First, suppose tht the prior distributio of µ is Nµ, σ. Nµ, σ is tke. The posterior distributio of µ is hµ = f µgµ = f µgµ f µgµ f µgµdµ A sigle observtio X µ µ σ π ep µ ep σ σ π σ µ ep µ µ σ σ ep µ σ + µ σ σ + µ + σ σ + µ σ Let, b d c be the coefficiets i the qudrtic polyomil i µ tht is the lst epressio. The my the be writte hµ = ep µ b µ + c To simplify this further, we use the techique of completig the squre d rewrite the epressio s hµ = ep ep µ b ep µ b c We see tht posterior distributio of µ is orml with me µ = + µ σ σ + σ σ b For prcticl resos, we defie the precisio s the iverse of the vrice: we deote by ξ = d d ξ σ = σ Theorem Suppose tht µ Nµ, σ. The the posterior distributio of µ is orml with me µ = σ µ + ξ ξ + ξ d precisio ξ = ξ + ξ The posterior me is weighted verge of the prior me d the dt, weights beig proportiol to the respective precisios. With very getle prior we would hve very low precisio ξ, very t prior d mostly the posterior is Norml with s its me. Of course wht we re usully iterested i is the posterior give iid smple of size, wht you could epect hppes it is equivlet to ddig oe observtio from distributio tht hs vrice σ. 4
5 Multiomil Dirichlet priors Dirichlet prior Dirichlet prior is cojugte to multiomil distributio. This is probbility distributio o o the simple. = { p = p, p,..., p, p +... + p =, p i } The Dirichlet distributio c be writte s DΘ α = Γ K i= α i K i= Γα i K i= Θ α i i where α = α,..., α K, with α i > re costts specifyig the Dirichlet distributio Θ i stisfy Θ i d K i= Θ i = The multiomil distributio correspodig to k blls dropped ito boes with fied probbility p,..., p, with ith bo cotiig k i blls is k k,..., k p k p k For two vribles K = the Dirichlet distributio reduces to Bet distributio, d ormlizig costt becomes Bet fuctio. The Dirichlet is coveiet prior becuse the posterior p hvig observed k,..., k is Dirichlet with probbility α +k,..., α +k. A importt chrcteriztio of the Dirichlet: it is the oly prior tht predicts outcomes lierly i the pst. Oe frequetly used specil cse is the symmetric Dirichlet whe ll α i = c >. We deote this prior s D c. Dirichlet priors re importt becuse They re turl cojugte priors for multiomil distributios, i.e. posterior prmeter distributio, fter hvig observed some dt from multiomil distributio with Dirichlet prior, lso hve form of Dirichlet distributio The Dirichlet distributio c be see s multivrite geerliztio of the bet distributio, over the spce of distributios P, with costt o the verge distce reltive etropy to referece distributio determied by Θ d α. Refereces [] Rev. Thoms Byes, A essy towrds solvig problem i the doctrie of chces, Philosophicl Trsctios of the Royl Society of Lodo 763. [] Joh. A. Rice, Mthemticl sttistics d dt lysis, Dubury Press, 995. 5