Mathematical Statistics Anna Janicka

Transcription

1 Mathematical Statistics Aa Jaicka Lecture XIV, BAYESIAN STATISTICS

2 Pla for Today. BayesiaStatistics a priori ad a posteriori distributios Bayesia estimatio: Maximum a posteriori probability(map) Bayes Estimator Credible itervals

3 Bayesia Statistics vs. traditioal statistics Frequetist: ukow parameters are give (fixed), observed data are radom Bayesia: observed data are give (fixed), parameters are radom

4 Bayesia Statistics Our kowledge about the ukow parameters is described by meas of probability distributios, ad additioal kowledge may affect our descriptio. Kowledge: geeral specific Example: coi toss

5 Bayesia Model X,..., X come from distributio P θ, with desity f θ (x) coditioal desity give a specific value of θ(likelihood fuctio). P family of probability distributios P θ, idexed by the parameter θ Θ Geeral kowledge: distributio Πover the parameter space Θ, give byπ(θ) the socalled a priori/prior distributio of θ, θ~ Π

6 Bayesia Model cot. Additioal kowledge (specific, cotextual): based o observatio. We have a joit distributio of observatios ad θ: f x, x,..., x, θ) = f( x, x,..., x θ) π( ) ( θ o this basis we ca derive the coditioal distributio of θ(give the observed data) f( x,..., x θ) π( θ) π ( θ x,..., x) =, m( x,..., x) where m x,..., x ) = f( x,..., x θ) π( θ dθ ( ) Θ is a margial distributio for the obs.

7 Bayesia Model a posteriori distributio π( θ x,..., x ) is called the a posteriori/ posterior distributio, deoted Π x The posterior distributio reflects all kowledge: geeral (iitial) ad specific (based o the observed data). Grouds for Bayesia iferece ad modelig

8 A priori ad a posteriori distributios: examples.let X,..., X be IID r.v. from a 0- distr. with α β prob of success θ; let θ ( θ) π( θ) = for θ (0,) B( α, β) α β Γ( α) Γ( β) where B( α, β) = u ( u) du = ad Γ( ) = α α u 0 0 exp( u) du Γ( α β) = ( α ) Γ( α ) the the a posteriori distributio: Beta( i= i x = x α, β) i i Beta(α,β) distr with mea = α/(α β) cojugate prior for Beroulli distr.

9 For a Beta (,) prior ad data: =0 ad, 5, 9 successes

10 For a Beta (,) prior ad data: =00 ad 0, 50, 90 successes

11 For a Beta (0,0) prior ad data: =0 ad, 5, 9 successes

12 For a Beta (0,0) prior ad data: =00 ad 0, 50, 90 successes

13 For a Beta (,5) prior ad data: =0 ad, 5, 9 successes

14 For a Beta (,5) prior ad data: =00 ad 0, 50, 90 successes

15 A priori ad a posteriori distributios: examples (). Let X,..., X be IID r.v. from N(θ, ), ad kow; θ~n(m, ) for m, kow. The the posterior distributio for θ: cojugate prior for a ormal distr., m X N

16 Bayesia Statistics Based o the Bayes approach, we ca fid estimates fid a equivalet of cofidece itervals verify hypotheses make predictios

17 Bayesia Most Probabale (BMP) / Maximum a posteriori Probability (MAP) estimate Similar to ML estimatio: the argumet which maximizes the posterior distributio: π( ˆ θbmp x,..., x) = max π( θ x,..., x θ ) i.e. BMP( θ) = ˆ θbmp = argmaxθπ( θ x,..., x )

18 BMP: examples.let X,..., X be IID r.v. from a Beroulli distr. with α β prob. of success θ; for θ (0,) θ ( θ) π( θ) = We kow the a posteriori distributio: B( α, β) β) Beta( x α, i= i x i= i we have max for ( ) = x α i= i BMP θ β α i.e. for 5 successes i 0 trials for a a priori U(0,) (i.e. Beta(,) distr.), we have BMP(θ)=5/0 = ½ ad for 9 successes i 0 trials for the same a priori distr., we have BMP(θ)=9/0 Beta(α,β) distr; the mode of this distr = (α-)/(α β-) for α>, β>

19 BMP: examples (). Let X,..., X be IID r.v. from N(θ, ), with kow; θ~n(m, ) for m, kow. The the a posteriori distr for θ: so i.e. if we have sa sample of 5 obs.;.7 ;.9 ;.; 3. from distr. N(θ, 4) ad the a priori distr is θ~n(, ), the BMP(θ) = (5 /4 * )/(5/4 ) = 4/9.56 ad if the a priori distr were θ~n(3, ), the BMP(θ) = (5 /4 * *3)/(5/4 ) = /9.44, m X N ) ( θ = m X BMP

20 Bayes Estimator A estimatio rule which miimizes the posterior expected value of a loss fuctio L(θ, a) loss fuctio,depeds o the true value of θad the decisio a. e.g. if we wat to estimate g(θ): L(θ, a) = (g(θ)-a) quadratic loss fuctio L(θ, a) = g(θ) -a module loss fuctio

21 Bayes Estimator cot. We ca also defie the accuracy of a estimatefor a give loss fuctio : acc( Π, gˆ( x)) = E ( L( θ, gˆ( x)) X = x) = Θ L( θ, gˆ( x)) π( θ x) dθ (the average loss of the estimator for a give a priori distributio ad data, i.e. for a specific posterior distributio)

22 Bayes Estimator cot. () The Bayes Estimatorfor a give loss fuctio L(θ, a) is such that x ĝ B acc( Π, gˆ ( x)) = mi acc( Π, a) For a quadratic loss fuctio (θ a) : ˆ θ B = E( θ For a module loss fuctio θ a : B X = x) ˆ θb = Med( Π x = ) a E( Π x ) more geerally: E(g(θ) x)

23 Bayes Estimator: Example ().Let X,..., X be IID r.v. from a Beroulli distr. with α β prob. of success θ; for θ (0,) θ ( θ) π( θ) = We kow the poster distributio: B( α, β) Beta( i= i x = x α, β) i i so the Bayes Estimator is ˆ θ B α = x i= i β α i.e. for 5 successes i 0 trials for a a priori U(0,) (i.e. Beta(,) distr.), we have =6/ = ½ ad for 9 successes i 0 trials for the same a priori distr., we have =0/ = 5/6 θˆb θˆb Beta(α,β) distr with mea = α/(α β)

24 BMP: examples.let X,..., X be IID r.v. from a Beroulli distr. with α β prob. of success θ; for θ (0,) θ ( θ) π( θ) = We kow the poster distributio: B( α, β) Beta( we have max for i= i x = x α, β) i i ( ) = x α i= i BMP θ β α i.e. for 5 successes i 0 trials for a a priori U(0,) (i.e. Beta(,) distr.), we have BMP(θ)=5/0 = ½ ad for 9 successes i 0 trials for the same a priori distr., we have BMP(θ)=9/0 Beta(α,β) distr; the mode of this distr = (α-)/(α β-) for α>, β>

25 Bayes Estimator: examples (). Let X,..., X be IID r.v. from N(θ, ), with kow; θ~n(m, ) for m, kow. The the a posteriori distr for θ: so i.e. if we have sa sample of 5 obs.;.7 ;.9 ;.; 3. from distr. N(θ, 4) ad the a priori distr is θ~n(, ), the = (5 /4 * )/(5/4 ) = 4/9.56 ad if the a priori distr were θ~n(3, ), the = (5 /4 * *3)/(5/4 ) = /9.44, m X N ˆ θ = m X B θˆb θˆb

26 BMP: examples (). Let X,..., X be IID r.v. from N(θ, ), with kow; θ~n(m, ) for m, kow. The the a posteriori distr for θ: so i.e. if we have sa sample of 5 obs.;.7 ;.9 ;.; 3. from distr. N(θ, 4) ad the a priori distr is θ~n(, ), the BMP(θ) = (5 /4 * )/(5/4 ) = 4/9.56 ad if the a priori distr were θ~n(3, ), the BMP(θ) = (5 /4 * *3)/(5/4 ) = /9.44, m X N ) ( θ = m X BMP

27 Highest Posterior Desity Credible Iterval A -α HPD(Highest Posterior Desity) credible iterval (Bayesia Cofidece Iterval) for parameter θ is a set A Θsuch that A = { θ : π( θ x) > k } α ad Π( A x) α k α for highest such that the secod coditio is fulfilled The HPD credible iterval has the ituitive property of iclusio which the frequetist CI does ot have

28 HPD Credible Iterval: example Let X,..., X be IID r.v. from N(θ, ), with kow; θ~n(m, ) for m, kow. The the a posteriori distr for θ: so for α= 0.05 we get a HPD CI: i.e. if we have sa sample of 5 obs from distr. N(θ, 4) with mea ad the a priori distr is θ~n(, ), the due to the fact that u we have a HPD CI:, m X N.96,.96 m X m X ( ) ( ).5.50,.96, =

29 Example problem Let 0.38, 0.65, 0.7,.00 be idepedet realizatios of a radom variable from a uiform distributio over the iterval (0,θ), where θ>0 is a ukow parameter. We iitially assume that θ is uiformly distributed over the iterval [/, ]. Fid the posterior distributio. Fid the Bayesia most probable estimator. Fid the Bayes estimator for a quadratic loss fuctio. Fid the Bayes estimator for a modulus loss fuctio.

30 Example exam problem

31