CHAPTER 3 POSTERIOR DISTRIBUTIONS

CHAPTER 3 POSTERIOR DISTRIBUTIONS If scece caot measure the degree of probablt volved, so much the worse for scece. The practcal ma wll stck to hs apprecatve methods utl t does, or wll accept the results of verse probablt of the Baes/Laplace brad tll better are forthcomg. Karl Pearso,90. 3.. Notato 3.. Cumulatve dstrbuto 3.3. Dest dstrbuto 3.3.. Defto 3.3.. Trasformed destes 3.4. Features of a dest dstrbuto 3.4.. Mea 3.4.. Meda 3.4.3. Mode 3.4.4. Credblt tervals 3.5. Codtoal dstrbuto 3.5.. Defto 3.5.. Baes Theorem 3.5.3. Codtoal dstrbuto of the sample of a Normal dstrbuto 3.5.4. Codtoal dstrbuto of the varace of a Normal dstrbuto 3.5.5. Codtoal dstrbuto of the mea of a Normal dstrbuto 3.6. Margal dstrbuto 3.6.. Defto 3.6.. Margal dstrbuto of the varace of a ormal dstrbuto 3.6.3. Margal dstrbuto of the mea of a ormal dstrbuto

Appedx 3. Appedx 3. Appedx 3.3 Appedx 3.4 3.. Notato We used a formal otato chapter because t was coveet for a coceptual troducto to the Baesa choce. From ow o, we wll use a more formal otato. Bold tpe wll be used for matrxes ad vectors, ad captal letters for matrxes. Thus represets a vector, A s a matrx ad s a scalar. Ukow parameters wll be represeted b Greek letters, lke ad. The letter f wll alwas be used for probablt dest fuctos. Thus f(x) ad f() are dfferet fuctos (for example, a ormal fucto ad a gamma fucto) although the both are probablt dest fuctos. We use sometmes dest as short ame of probablt dest dstrbuto. The letter P wll be reserved for probablt. For example, P() s a umber betwee 0 ad represetg the probablt of the sample. The varables wll be red colour. For example, f( ) exp s a fucto of, but t s ot a fucto of or a fucto of, whch are cosdered costats. Thus, ths example we represet a faml of dest fuctos for dfferet values of. A example of ths s the lkelhood, as we have see chapter.

3 The sg meas proportoal to. We wll ofte work wth proportoal fuctos, sce t s easer ad, as we wll see later, the results from proportoal fuctos ca be reduced to (almost) exact results easl wth MCMC. For example, f c ad k are costats, the dstrbuto N(0,) s f exp exp k expc exp k expc Thus we ca add or subtract costats from a expoet, or multpl or dvde the fucto f the costats are ot the expoet. Notce that we caot add costats or multpl the expoet b a costat;.e., f f exp f ot proportoal to k exp ot proportoal to p f exp c ex c 3.. Dest dstrbuto 3... Defto We wll use a auxlar fucto to descrbe our ucertat about the ukow parameters we wat to estmate. Ths s the probablt dest dstrbuto. I ths fucto, the area uder the curve for a terval [a, b] (fgure 3.) s the probablt that the true value of the parameter we estmate falls betwee the lmts defed b ths terval. b Pa x b f xdx a

4 Fgure 3.. Probablt dest fucto.the area blue betwee a ad b s the probablt that the true value of the parameter estmated falls betwee a ad b. Whe ths terval covers all possble values,, ths probablt s P x f x dx For a value of the parameter x0, otce that f(x0) s ot a probablt. Areas defed b f(x) are probabltes, ad the area f(x0) Δx s approxmatel a probablt whe Δx s small (fgure 3.). These small probabltes are usuall expressed as f(x)dx. Fgure 3.. A example of dest dstrbuto for the trat test of lver flavour of meat. Probabltes are areas of f(x), for example the pk area shows P(x<0), ad the ellow oe P(x>0). The small rectagles f(x) Δx are approxmate probabltes for a Δx aroud a value of x

5 3.3.. Trasformed destes We kow a dest f(x) ad we wat to fd the dest f() of a fucto = g(x). Itutvel, the probablt area f(x0) Δx should be the same as f(0) Δ whe Δx ad Δ are small eough. Thus we ca wrte f() d = f(x) dx but t ma happe that whe x creases, decreases, the f() d = f(x) dx takg both together dx f f f d x x d dx I Appedx 3. we show a more formal demostrato of ths. For example, we have a Normal dstrbuto f(x) ad we wat to kow the dstrbuto of = exp(x). We kow that x f( x) exp d x f( ) f( x) exp dx exp x

6 Sce x = l(), we fall get l l f( ) exp exp expl I the multvarate case, we kow f(x, ) ad we wat to kow the dest f(u, w), where u ad w are fuctos of (x,) u = g(x,) ; w = h(x,) For example, f(x, ) s a kow bvarate dstrbuto ad we would lke to kow the dest of x. Ths happes, for example, whe geetcs we have the x destes of the addtve ad evrometal varace of a trat ad we wat to kow the dest of the hertablt, whch s the rato of the addtve ad the sum of both varace compoets. I ths case u = x x w = x The correspodg formula for the bvarate case s u w x, f, f J Where J s the absolute value of what mathematcs s called the Jacoba, x f u J determat f u f x w f w whch ca be easl geeralzed for the multvarate case.

7 3.4. Features of a dest dstrbuto 3.4.. Mea The expectato or mea of a dest fucto s E x x f x dx If we have a fucto of the radom varable =g(x) We have see 3.3. that f fx dx d thus, ts expectato s dx E f d gx f x d gx f xdx d For example, f = x E f d x f x dx

8 3.4.. Meda The meda s the value dvdg the area defed b the dest dstrbuto two parts, each oe wth a 50% of probablt,.e.: the meda s the value mx that m x f(x)dx 0.50 3.4.3. Mode The mode s the maxmum of the dest fucto Mode = arg max f(x) The mode s the value aroud whch the probablt s at ts maxmum (.e.: the value of x for whch f(x) x s maxmum). 3.4.4. Credblt tervals A credblt terval for a gve probablt, for example 90%, s the terval [a,b] cotag 90% of the probablt defed b the dest fucto;.e., a values a ad b for whch b a f x dx 0.90 costtutes a credblt terval [a, b] at 90%. For example, whe performg fereces about some ukow parameter or about a dfferece betwee treatmets θ, a terval [a,b] wth 90% probablt would mea that the true value of θ les betwee the lmts a,b of the terval wth a probablt of 90%. Notce

9 that there are fte credblt tervals at 90% of probablt, but the have dfferet legth, as we have see chapter (fgure.5), provdg dfferet accuraces about θ. Oe of these tervals s the shortest oe (gvg the best accurac), ad Baesa ferece, whe the dest fucto used s the posteror dest, t s called the Hghest Posteror Dest terval at 90% (HPD90%). 3.5. Codtoal dstrbuto 3.5.. Defto We sa that the codtoal dstrbuto of x gve =0 s f x 0 x 0 f f, 0 Followg our otato, whch the varables are red ad the costats are black, we ca express the same formula as f x, f f x If we cosder that ca take several values, the formula ca be expressed as f x, f f x ad ths case t represets a faml of dest fuctos, wth has dfferet dest fucto for each value of = 0,,, For two gve values x ad, the formula s

0 f x f f x, 3.5.. Baes theorem Although f(x) s ot a probablt, we foud that f(x) x s deed a probablt (see fg. 3.4), thus applg Baes theorem, we have P A B PA PB P B A f x f x x f x f x f x x f x f f thus we ow have a verso of Baes theorem for dest fuctos. Cosderg x as the varable ad as a gve (costat) value, f x f x f f x whch ca be expressed proportoall, sce f() s a costat, x x x f f f For example, f we have a ormal dstrbuto ~ N(µ,σ ) whch we do ot kow the parameters µ, σ, the ucertat for both parameters ca be expressed as f, f, f f, f,, f 3.5.3. Codtoal dstrbuto of the sample of a Normal dstrbuto

Let us ow cosder a radom sample from a ormal dstrbuto, f, f, f, f,, f f, exp exp ths s the codtoal dstrbuto of the data because t s codtoed to the gve values of μ ad σ ;.e., for each gve value of the mea ad the varace we have a dfferet dstrbuto. For example; for a gve mea μ=5 ad varace σ =, ad for a sample of three elemets = [,, 3 ] we have 3 5 5 5 3 f, exp 3 Ths s a trvarate multormal dstrbuto, wth three varables, ad 3. 3.5.4. Codtoal dstrbuto of the varace of a Normal dstrbuto We ca also wrte the codtoal dstrbuto of the varace for a gve sample ad a gve mea μ. We do ot kow f(σ μ, ), but we ca appl Baes theorem ad we obta f, f, f applg the prcple of dfferece (see..3), we wll cosder ths example that a pror all values of the varace have the same probablt dest;.e., f(σ )=costat. Ths leads to

f, f, but we kow the dstrbuto of the data, whch we have assumed to be Normal, thus we ca wrte the codtoal dstrbuto of the varace, f, f, exp Notce that the varable s coloured red, thus here the varace s the varable ad the sample ad the mea are gve costats. For example, f the mea ad the sample are µ = ' = [, 3, 4] for ths gve mea ad ths gve sample, the codtoal dstrbuto of the varace s 3 4 7 f, exp exp 3 3 Notce that ths s ot a Normal dstrbuto. A Normal dstrbuto does ot have the varable ( red) there; a Normal dstrbuto looks lke f x x 5 exp 0 0 where 5 ad 0 are the mea ad the varace of the varable x.

3 What s, the, the codtoal dstrbuto of the varace? There s a tpe of dstrbuto called verted gamma that looks lke so: f x, exp x x where α ad β are parameters that determe the shape of the fucto. I fgure. we fd three dfferet shapes that we ca obta b gvg dfferet values to α ad β, a flat le ad two curves, oe of them sharper tha the other oe. We ca see that the codtoal dstrbuto of the varace s of the tpe verted gamma. If we take 7 3 we obta a verted gamma, ad geeral the codtoal dstrbuto of the varace of a Normal dstrbuto s a verted gamma ( ) wth parameters 3.5.5. Codtoal dstrbuto of the mea of a Normal dstrbuto We wll ow wrte the codtoal dstrbuto of the mea for a gve sample ad a gve varace σ. We do ot kow f(μ σ, ), but we ca appl the Baes theorem ad we obta f, f, f Ths tpe of uvarate verted gamma s usuall amed verted ch-square

4 Applg the prcple of dfferece (see..3), we wll cosder ths example that a pror all values of the mea have the same probablt dest;.e., f(μ)=costat. Ths leads to f, f, but we kow the dstrbuto of the data, that we have assumed to be Normal, thus we ca wrte the codtoal dstrbuto of the mea, f, exp Notce that the varable s coloured red, thus here the mea s the varable ad the sample ad the varace are gve costats. For example, f the varace ad the sample are σ = 9 ' = [, 3, 4] for ths gve varace ad ths gve sample, the codtoal dstrbuto of the mea s 9 3 4 3 89 f, exp exp 3 9 8 Ths ca be trasformed to a Normal dstrbuto easl

5 f, exp exp 9 9 6 3 9 9 3 3 8 8 3 3 9 9 3 3 3 exp exp exp 8 8 9 3 3 3 exp 9 3 3 9 3 whch s a Normal dstrbuto wth mea 3 (the sample mea) ad varace 9 (the gve varace dvded b the umber of data). I a geeral form, we have (Appedx 3.) f, exp exp whch s a Normal dstrbuto whch the mea s the sample mea ad the varace s the gve varace dvded b the sample sze. 3.6. Margal dstrbuto 3.6.. Defto We saw..3 the advatages of margalsato. Whe we have a bvarate dest f(x, ), a margal dest f(x) s represeted b:

6 f x f x, d f x f d The margal dest f(x) takes the average of all values of for each x;.e., adds all values of multpled b ther probablt. A dest ca be margal for oe varable ad codtoal for aother oe. For example, a margal dest of x wth respect to varable, codtoed z s show below: f x z f x, z d f x, z f z d where has bee tegrated out ad z s codtog the values of x. Notce that the varable does ot appear f x z because, for each value of x, all the possble values of have bee cosdered, multpled b ther respectve probablt ad summed up, thus we do ot eed to gve a value to order to obta f x z. However, the codtoal varable appears f x z because for each gve value of z we obta a dfferet value of f x z. We wll see a example the ext paragraph. 3.6.. Margal dstrbuto of the varace of a ormal dstrbuto The margal dest of the varace codtoed to the data s f f, d Here the mea s tegrated out ad the data are codtog the values of the varace, whch meas that we wll obta a dfferet dstrbuto for each sample. We wll call ths dstrbuto the margal dstrbuto of the varace as a short ame, because Baesa ferece t s mplct that we are alwas codtog

7 the data. Baesa ferece s alwas based the sample, ot coceptual repettos of the expermet; the sample s alwas gve that. We do ot kow f,, but we ca fd t out applg the Baes theorem because we kow the dstrbuto of the data f,. If the pror formato f, s costat because we appl the prcple of dfferece as before, we wll have: f, f, f, f, f, f, f ad the margal dest s f f, d f, d exp d We solved ths tegral Appedx 3.3, ad we kow that the soluto s f exp whch s a Iverted Gamma dstrbuto, as 3.4.4, wth parameters α, β 3 For example, f we take the same sample as 3.4.4

8 ' = [, 3, 4] for ths gve sample, the margal dstrbuto of the varace s 3 4 3 4 3 4 3 4 3 3 3 f exp 3 exp Notce that the mea does ot appear the formula. I 3.5.4, whe we calculated the dest of the varace codtoed to the mea ad to the data we had to gve a value for the mea ad we had to provde the data. Here we should ol provde the data because the mea has bee tegrated out. 3.6.3. Margal dstrbuto of the mea of a Normal dstrbuto The margal dest of the mea codtoed to the data s 0 f f, d Here the varace s tegrated out ad the data s codtog the values of the mea, whch meas that we wll obta a dfferet dstrbuto for each sample. We wll call ths dstrbuto the margal dstrbuto of the mea as a short ame, because, as we sad before, Baesa ferece t s mplct that we are alwas codtog the data. We do ot kow the fucto f,, but applg Baes theorem we ca fd t out, because we kow the dstrbuto of the data, f,.

9 f, f, f, f, f, f f we admt the dfferece prcple to show vague pror formato, the f(µ,σ ) s a costat, thus f, f, ad the margal dest of the mea s 0 f f, d exp d Ths tegral s solved Appedx 3.4, ad the result s where f s ; s Ths s a Studet t-dstrbuto wth degrees of freedom, havg a mea whch s the sample mea ad a varace that s the sample quas-varace, thus f t,s For example, f = [, 3, 4] 3 4 3 3

0 s 3 3 3 4 3 3 3 3 3 f Notce that the varace does ot appear the formula. I 3.5.5, whe we calculated the dest of the mea codtoed to the varace ad to the data, we had to gve a value for the varace ad we had to show the data. Here we should ol gve the data because the varace has bee tegrated out. Appedx 3. We defe the cumulatve dstrbuto fucto at the pot 0 as 0 F 0 P 0 f d, for all 0 We kow a dest f(x) ad what we wat s to fd the dest f() of a fucto = g(x) We wll frst assume that g(x) s a strctl creasg fucto, as fgure 3.5.a. F(0) = P( 0) = P[g(x) 0] but sce g(x) s a creasg fucto, we kow that g(x) 0 x x0 the, we would have

F(0) = P(x x0) = F(x0) for all 0 ad x0, F() = F(x) a b Fgure 3.3. a. Whe g(x) s a mootoous creasg fucto. b whe g(x) s a mootoous decreasg fucto. Now, from the defto of the dstrbuto fucto, we have df( ) df( ) df( ) d d f( ) x x x f( x) x d d dx d d Now suppose that g(x) s a strctl decreasg fucto, as fgure 3.5.b. B the defto of dstrbuto fucto, we would have F(0) = P( 0) = P[g(x) 0] but as g(x) s a decreasg fucto, we kow that g(x) 0 x x0 the, we have F(0) = P(x x0) = P(x x0)= F(x0) F() = F(x)

because ths apples for ever x0. Now, b defto of dest fucto, we have df( ) d F( x) dx dx f( ) f( x) d dx d d Fall, puttg together both cases, we have dx f( ) f( x) f( x) d d dx - Appedx 3. We cosder frst that the umerator the exp s because the double product s ull 0 The, substtutg the formula, we have

3 f, exp exp exp exp exp Appedx 3.3 f exp d We ca place out of the tegral everthg but μ, thus cosderg the factorzato of the umerator wth the exp that we have made Appedx 3., we ca wrte f exp exp d We ca clude wth the tegral a costat or varable wth the excepto of µ, thus we multpl out of the tegral ad dvde sde the tegral b the same expresso, ad wrte

4 f exp exp d The expresso sde the tegral s a dest fucto (a ormal fucto) of µ, ad the tegral s, thus f exp Appedx 3.4 0 f exp d We put the π the costat of proportoalt. We ca clude wth the tegral a costat or varable wth the excepto of σ, thus we dvde out of the tegral ad multpl sde the tegral b the same expresso, ad wrte 0 f exp d Now, f we call x = σ, the sde part of the tegral looks lke

5 f x, x exp x where ; Ths s, puttg the tegrato costat, results a verted gamma dstrbuto, ad thus the value of the tegral s. The, we have f Ths ca be trasformed accordg to the factorzato of Appedx 3. f s s s s where ; s