STK4 ad STK9 Autum 6 Pot estmato Covers (most of the followg materal from chapter 7: Secto 7.: pages 3-3 Secto 7..: pages 3-33 Secto 7..: pages 35-3 Secto 7..3: pages 34-35 Secto 7.3.: pages 33-33 Secto 7.3.: pages 334-339 Ørulf Borga Departmet of Mathematcs Uversty of Oslo Estmato Let, be a radom sample from the populato f ( x, so, are d ad ther pmf or pdf s f ( x A pot estmator W (, s a statstc that we use to guess the value of the populato parameter (or a fucto τ ( of the parameter Whe we evaluate the estmator at the observed values x, x,..., x of the radom varables we obta a estmate of We wll frst cosder methods for fdg estmators, ad the dscuss how the estmators may be evaluated Method of momets Assume that (,..., k The momet estmators of,..., k are obtaed by the equatos oe obtas by equatg the frst k sample momets ad populato momets Sample momets: m j j Populato momets: µ (,..., E j j k The momet estmators solve the equatos µ (,..., m j,..., k j k j 3 Example gamma dstrbuto Let, be d wth pdf x/ β f ( x β, x e for x> β Γ( We have µ (, E β β µ ( β, E The momet estmators are gve by the equatos β ad β + ( β Ths gves ( ˆ ad ˆ β ( 4 Var + (E β + ( β
Maxmum lkelhood Let, be a d sample from a populato wth pmf or pdf f ( x,..., The the lkelhood s gve by L( x L(,..., x,..., x f ( x,..., k k k For each sample pot x, let ˆ( x be the parameter value at whch L ( x attas ts maxmum as a fucto of wth x held fxed The ˆ( s the maxmum lkelhood estmator (MLE based o the sample (, Usually oe fds the MLE by maxmzg the log-lkelhood log ( Example gamma dstrbuto Let, be d wth pdf x/ β f ( x β, x e for x> β Γ( The lkelhood s gve by x / β x e β ( Γ L( β, x f ( x β, ad the log-lkelhood becomes log L(, β x log β log Γ ( + ( log x x β L x 5 6 Dfferetatg the log-lkelhood we obta ( log (, β log β Γ + L x log x Γ( log (, β + L x x β β β If we set the partal dervatves equal to zero ad solve the equatos, we fd that the MLEs are gve by ˆ β ˆ ad Γ ( ˆ log ˆ log + log ( ˆ Γ 7 A mportat property of ML-estmators s that they are varat the followg sese: Theorem 7.. (varace property of MLEs If ˆ s the MLE of, the τ ( ˆ s the MLE of τ ( Assume that the fucto η τ ( s oe-to-oe Let τ ( η be the verse fucto The L ( η x L( τ ( η x ad sup L ( η x sup L( τ ( η x sup L( x η η Thus the maxmum of L ( η x s attaed at ˆ η τ ( ˆ The result s also vald whe η τ ( s ot oe-to-oe (cf. page 3 8
Bayes estmators I ths course we maly focus o the classcal frequetst approach to statstcs, but we wll ow have a bref look at the Bayesa approach I the classcal approach a populato parameter s a fxed quatty, but ts value s ukow to us I the Bayesa approach s assumed to be a quatty whose varato may be descrbed by the pror dstrbuto π ( Whe a sample s take from the populato, we may update the pror dstrbuto ad obta the posteror dstrbuto ( The jot pmf or pdf of ad s f ( x, f ( x π ( The margal pmf or pdf of s m( x f ( x π ( d 9 π x (wth sum stead of tegral f π ( s a pmf Hece the posteror pmf or pdf of becomes π ( x f ( x π ( m( x The posteror s used to make statemets about We may e.g. use the posteror mea as a pot estmate of Example 7..4 bomal Bayes estmator Let Y bomal(, p : ( y f ( y p p ( p y y Assume that the pror dstrbuto of p s beta( β, : Γ ( + β π( p p ( p Γ( Γ( β β The the jot dstrbuto of Y ad p s y yγ ( + β β ( y f ( y, p p ( p p ( p Γ( Γ( β Γ ( + β ( p ( p y ( ( The margal dstrbuto of Y s y+ y+ β ΓΓβ p Y y beta( y+, y+ β ( y Γ ( + β y+ y+ β m( y p ( p dp ( y Γ( Γ( β Γ ( + β ( Γ( Γ( β ( y y+ y+ β p p dp Γ ( + β Γ ( y+ Γ( y+ β Γ( Γ( β Γ ( + + β The posteror dstrbuto of p becomes f ( p y Thus f ( y, p m( y Γ ( + + β p ( p Γ ( y+ Γ( y+ β y+ y+ β
We may use the posteror mea to estmate p,.e. y+ pˆ B + + β Note that we may wrte the estmator as pˆ B y + β + + + β + + β + β I ths example both the pror ad the posteror dstrbutos are beta-dstrbutos The reaso for ths s that the beta dstrbuto s a cojugate famly for the bomal 3 Evaluato of estmators Oe crtero for evaluatg estmators s the mea squared error (MSE The MSE of a estmator W W (,..., of s the fucto of gve by E ( W The bas of W s gve as Bas W E W The estmator s ubased f the bas s zero for all Note that the MSE may be wrtte ( W W W E ( Var + Bas For a ubased estmator, MSE equals ts varace Example 7.3.5 MSE of bomal Bayes estmator Let Y bomal(, p The MLE of p s pˆ Y / We kow that ths s ubased, so ts MSE becomes p( p E ˆ ˆ p( p p Varp p Y+ The cosder the Bayes estmator pˆ B + + β The varace of the Bayes estmator s Var pˆ Y + Var + + β p B p p( p ( + + β 5 The bas of the Bayes estmator s Bas pˆ Y + p + E p p + + β + + β p B p Hece the MSE of the Bayes estmator s ( ˆ ˆ + ˆ p pb p p pb p pb E ( Var Bas p( p p + + p + + + + β ( β If we chose β / 4 the MSE wll ot deped o p 6
The we obta pˆ B Y+ + / 4 ad E( pˆ p B 4( + Best ubased estmators We are ot able to fd a estmator that mmzes the MSE for all values of If we wat to fd «the best» estmator, we eed to restrct the class of estmators we cosder MSE( pˆ MSE( pˆ If we restrct ourselves to the ubased estmators, the best estmator s the oe wth smallest varace 7 Defto 7.3.7 A estmator W s a best ubased estmator of τ ( f t satsfes E W τ ( for all ad, for ay other estmator W wth E W τ (, we have Var W Var W for all. We also say that W s a uform mmum varace ubased estmator (UMVUE for τ ( Example 7.3. Posso ubased estmato Let,,..., be d Posso( Further let ad S be the sample mea ad the sample varace The E ad ES Thus, S ad a+ ( a S are all ubased estmators of Whch oe s best, ad are there better ubased estmators? Iformato Cosder a sample (,,..., wth jot pdf (or pmf f ( x Note that we do ot assume that the are d We assume that expressos of the form W ( x f ( x may be dfferetated wth respect to by chagg the order of dfferetato ad tegrato (summato the case of pmf 9
The the formato umber or Fsher formato of the sample s I( E log f ( log f ( x f ( x For the specal case where,,..., are d wth pdf (or pmf f ( x the formato the sample s gve by I ( I (, where I ( s the formato oe observato ad s gve by I ( E log f ( log f ( x f ( x If expressos of the form W ( x f ( x may be dfferetated twce wth respect to by chagg the order of dfferetato ad tegrato (summato for pmf, the formato the sample may also be gve as I( E log f ( log f ( x f ( x whle the case of d observatos, the formato oe observato may be gve as I( E log f ( log f ( x f ( x The Cramér-Rao equalty Cosder a sample (,,..., wth jot pdf (or pmf f ( x ad assume that expressos of the form W ( x f ( x may be tegrated wth respect to by chagg the order of dfferetato ad tegrato (summato the case of pmf. The for ay estmator W ( wth fte varace we have d E W ( d Var W ( E log f ( 3 Example 7.3.8 Posso UMVUE Let,,..., be d Posso( Here x log f ( x log e x! x log log x! log f ( x x x log f ( x Hece the formato oe observato s ( E I ad the formato the sample s I( / 4
By the Cramér-Rao equalty, we have for ay ubased estmator W for that Var W I ( / Now s a ubased estmator for, ad Var Hece s a UMVUE for 5