Week 4 Spring Lecture 7. Empirical Bayes, hierarchical Bayes and random effects

Size: px

Start display at page:

Download "Week 4 Spring Lecture 7. Empirical Bayes, hierarchical Bayes and random effects"

Beverly Lynch
6 years ago
Views:

1 Week 4 Spring 9 Lecture 7 Empirical Bayes, hierarchical Bayes and random effects Empirical Bayes and Hierarchical Bayes eview: Let X N (; ) Consider a normal prior N (; ) Then the posterior distribution of is jx N( + ( + ) (X ), ( + ) ) Empirical Bayes Let = I, = and () = = I Then + = + : Since E p kk = +, we may estimate + by p to yield the James-Stein estimator kk Hierarchical Bayes Let = I, = I and H It is easy to see that g () = = kk p Z _ p e kk d _ Z ' I () d, then the harmonic prior corresponds h () Theorem Let h () _ ( + ) a The the generalized Bayes estimators (i) eist if a < + p=; (ii) are admissible if p 3 and 3 p= a ; (iii) are minima if they eist and p 3 and 3 p= a Proof of the theorem: (i) It is easy to see (a ie, a < + p= ) (p ) d is nite if and only if (a ) < p, (ii) We have Z g () _ ( + ) a ep kk p d then kk p+ a g () c as kk

2 It can be shown that the growth condition holds Z g () kk (log kk) d < kk> and the atness condition holds when 3 p= a (leave it to you) (iii) It can be shown that r kk (p ) () A kk where r kk = p 4 p + a vp= e kk = a e vkk From Baranchik (97) the estimator is minima if p + a e kk = p 4 vp= a e vkk Let s just look at the case a = The general case follows similarly Write then g () _ g () = = kk p Z _ ' I () d Z Z ' (+ )I () d _ v (p 4)= ep kk v, thus rg g = vp= ep kk v vp= ep kk v = p vp= e kk = = e vkk vp= d ep vp= ep kk v kk kk v kk emark When p 5, there eist proper Bayes minima estimators, since h is the density of a nite measure for a < For p = 3; 4 there are no proper Bayes minima estimators from Brown (97) Hierarchical Bayes vs Empirical Bayes These two forms of analysis are closely related The hierarchical formulation X N (; I) ; N ; I

3 is common to both of them The empirical Bayes method uses the data to produce some heuristic estimator of Hierarchical Bayes methods treat the hierarchical parameter,, in a Bayesian fashion There is an additional heuristic connection between the two methodologies Note that the hierarchical Bayes estimator can be written as E (j) = E E j; j The inner epectation on the right hand side of the equation can be considered to be an estimator such as the one that appears in the empirical Bayes derivation Hence the hierarchical Bayes estimator, hier, say is the mean of these with respect to the Bayesian conditional distribution of given Write hier = b In this way the hierarchical Bayes estimator can also be viewed as an empirical Bayes estimator 3

4 Lecture 8 Empirical Bayes, hierarchical Bayes and random effects (Cont) obbins (956): An empirical Bayes approach to statistics, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Jerzy Neyman, ed, vol, Berkeley, California: University of California Press, 956, pp Selected writings of obbins: What is Mathematics?: An elementary approach to ideas and methods, with ichard Courant, London: Oford University Press, 94 A stochastic approimation method, with Sutton Monro, Annals of Mathematical Statistics, 3, 95, pp obbins (956) above 4 Asymptotically subminima solutions of compound decision problems, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 95, pp 3-48 Eample Observe X i P oisson ( i ) P (X i = i j i ) = ep ( i) i i i We want to estimate the unknown parameter i Assume that ; ; : : : ; n are iid with distribution G The generalized Bayes wrt squared error loss is ep( ) i i ( i ) = i ep( ) i i ep( ) i + ( = ( i + ) i+) ep( ) i i = ( i + ) G ( i + ) G ( i ) where G is the marginal distribution of X i We know for every ed i ; number of terms X ; X,, X n which are equal to i + G ( i + ) number of terms X ; X,, X n which are equal to i G ( i ) Eample (read Efron, 3, Ann Stat) Applications to missing species problem Estimate Shakespeare s vocabulary Etension to eponential family Let X i f (j i ) = ep ( i ( i )) h () and i G, then ep ( ()) h () () = ep ( ()) h () = d d (G () =h ()) G () =h () 4

5 Then the question now is how to estimate G () Connection to compound decision theory obbins (95): Asymptotically subminima solutions of compound decision problems, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 95, pp 3-48 Let X i f (j i ) and write (; ) = n nx EL ( i ; i (X)) i= For separable decision rules of the form i (X) = t (X i ), the compound risk is equal to the average risk Z Z (; ) = L (; t (X)) f (j) d where G (A) = n P n i= f Ag obbin s proposal is to seek asymptotically minima procedure satisfying (; ) = (; G ) + o () as n 5

Week 2 Spring Lecture 3. The Canonical normal means estimation problem (cont.).! (X) = X+ 1 X X, + (X) = X+ 1

Week 2 Spring Lecture 3. The Canonical normal means estimation problem (cont.).! (X) = X+ 1 X X, + (X) = X+ 1 Week 2 Spring 2009 Lecture 3. The Canonical normal means estimation problem (cont.). Shrink toward a common mean. Theorem. Let X N ; 2 I n. Let 0 < C 2 (n 3) (hence n 4). De ne!! (X) = X+ 1 Then C 2 X