Unobserved Heterogeneity in Longitudinal Data An Empirical Bayes Perspective

Transcription

1 Unobserved Heterogeneity in Longitudinal Data An Empirical Bayes Perspective Roger Koenker University of Illinois, Urbana-Champaign University of Tokyo: 25 November 2013 Joint work with Jiaying Gu (UIUC) µ^ σ 2 ^ Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

2 An Empirical Bayes Homework Problem Suppose you observe a sample {Y 1,..., Y n } and Y i N(µ i, 1) for i = 1,..., n, and would like to estimate all of the µ i s under squared error loss. We might call this incidental parameters with a vengence. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

3 An Empirical Bayes Homework Problem Suppose you observe a sample {Y 1,..., Y n } and Y i N(µ i, 1) for i = 1,..., n, and would like to estimate all of the µ i s under squared error loss. We might call this incidental parameters with a vengence. Not knowing any better, we assume that the µ i are drawn iid-ly from a distribution F so the Y i have density, g(y) = φ(y µ)df(µ), the Bayes rule is then given by Tweedie s formula: δ(y) = y + g (y) g(y) Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

4 An Empirical Bayes Homework Problem Suppose you observe a sample {Y 1,..., Y n } and Y i N(µ i, 1) for i = 1,..., n, and would like to estimate all of the µ i s under squared error loss. We might call this incidental parameters with a vengence. Not knowing any better, we assume that the µ i are drawn iid-ly from a distribution F so the Y i have density, g(y) = φ(y µ)df(µ), the Bayes rule is then given by Tweedie s formula: δ(y) = y + g (y) g(y) When F is unknown, one can try to estimate g and plug it into the Bayes rule. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

5 Stein Rules I Suppose that the µ i s were iid N(0, σ 2 0 ), so the Y i s are iid N(0, 1 + σ 2 0 ), the Bayes rule would be, ( δ(y) = 1 1 ) 1 + σ 2 y. 0 When σ 2 0 is unknown, S = Yi 2 (1 + σ2 0 )χ2 n, and recalling that an inverse χ 2 n random variable has expectation, (n 2) 1, we obtain the Stein rule in its original form: ( ˆδ(y) = 1 n 2 ) y. S Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

6 Stein Rules II More generally, if µ i N(µ 0, σ 2 0 ) we shrink instead toward the prior mean, ( δ(y) = µ ) 1 + σ 2 (y µ 0 ), 0 estimating the prior mean parameter costs us one more degree of freedom, and we obtain the celebrated James-Stein (1960) estimator, ( ˆδ(y) = Ȳ n + 1 n 3 ) (y Ȳ n ), S with Ȳ n = n 1 Y i and S = (Y i Ȳ n ) 2. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

7 Nonparametric Empirical Bayes Rules Brown and Greenshtein (Annals, 2009) propose estimating g by standard fixed bandwidth kernel methods and they compare performance of their estimated Bayes rule with various other methods including the various parametric empirical Bayes methods investigated by Johnstone and Silverman in their Needles and Haystacks paper. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

8 Nonparametric Empirical Bayes Rules Brown and Greenshtein (Annals, 2009) propose estimating g by standard fixed bandwidth kernel methods and they compare performance of their estimated Bayes rule with various other methods including the various parametric empirical Bayes methods investigated by Johnstone and Silverman in their Needles and Haystacks paper. A drawback of the kernel approach is that it fails to impose a monotonicity constraint that should hold for the Gaussian problem, or indeed for any similar problem in which we have iid observations from a mixture density, g(y) = ϕ(y, θ)df(θ) and ϕ is an exponential family density with natural parameter θ R. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

9 Back to the Homework When ϕ is an exponential family density we may write, ϕ(y, θ) = m(y)e yθ h(θ) Quadratic loss implies that the Bayes rule is a conditional mean: δ G (y) = E[Θ Y = y] = θϕ(y, θ)df/ ϕ(y, θ)df = θe yθ h(θ)df/ e yθ h(θ)df = d dy log( e yθ h(θ)df = d dy log(g(y)/m(y)) Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

10 Monotonicity of the Bayes Rule When ϕ is of the exponential family form, δ G (y) = d dy [ ] θϕdf θ 2 ( ) 2 ϕdf θϕdf = ϕdf ϕdf ϕdf = E[Θ 2 Y = y] (E[Θ Y = y]) 2 = V[Θ Y = y] 0, implying that δ G must be monotone, or equivalently that, K(y) = log ĝ(y) log m(y) is convex. Such problems are closely related to recent work on estimating log-concave densities, e.g. Cule, Samworth and Stewart (JRSSB, 2010), Koenker and Mizera (Annals, 2010), Seregin and Wellner (Annals, 2010). Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

11 Standard Gaussian Case In our homework problem, ϕ(y, θ) = φ(y θ) = K exp{ (y θ) 2 /2} = Ke y2 /2 e yθ e θ2 /2 So m(y) = e y2 /2 and the logarithmic derivative yields our Bayes rule: δ G (y) = d [ ] 1 dy 2 y2 + log g(y) = y + g (y) g(y). Estimating g by maximum likelihood subject to the constraint that K(y) = 1 2 y2 + log ĝ(y) is convex is discussed in Koenker and Mizera (2013). Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

12 Nonparametric MLE Kiefer and Wolfowitz (1956) reconsidering the Neyman and Scott (1948) problem showed that non-parametric maximum likelihood could be used to establish consistent estimators even when the number of incidental parameters tended to infinity. Laird (1978) and Heckman and Singer (1984) suggested that the EM algorithm could be used to compute the MLE in such cases. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

13 Nonparametric MLE Kiefer and Wolfowitz (1956) reconsidering the Neyman and Scott (1948) problem showed that non-parametric maximum likelihood could be used to establish consistent estimators even when the number of incidental parameters tended to infinity. Laird (1978) and Heckman and Singer (1984) suggested that the EM algorithm could be used to compute the MLE in such cases. Jiang and Zhang (Annals, 2009) adapt this approach for the empirical Bayes problem: Let u i : i = 1,..., m denote a grid on the support of the sample Y i s, then the prior (mixing) density f is estimated by the EM fixed point iteration: ˆf (k+1) j = n 1 n ˆf (k) m i=1 l=1 j φ(y i u j ) (k) ˆf l φ(y i u l ), Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

14 Nonparametric MLE Kiefer and Wolfowitz (1956) reconsidering the Neyman and Scott (1948) problem showed that non-parametric maximum likelihood could be used to establish consistent estimators even when the number of incidental parameters tended to infinity. Laird (1978) and Heckman and Singer (1984) suggested that the EM algorithm could be used to compute the MLE in such cases. Jiang and Zhang (Annals, 2009) adapt this approach for the empirical Bayes problem: Let u i : i = 1,..., m denote a grid on the support of the sample Y i s, then the prior (mixing) density f is estimated by the EM fixed point iteration: ˆf (k+1) j = n 1 n ˆf (k) m i=1 l=1 j φ(y i u j ) (k) ˆf l φ(y i u l ), and the implied Bayes rule becomes at convergence: ˆδ(Y i ) = m j=1 u jφ(y i u j )ˆf j m j=1 φ(y i u j )ˆf j. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

15 The Incredible Lethargy of EM-ing Unfortunately, EM fixed point iterations are notoriously slow and this is especially apparent in the Kiefer and Wolfowitz setting. Solutions approximate discrete (point mass) distributions, but EM goes ever so slowly. (Approximation is controlled by the grid spacing of the u i s.) f(x) GMLEBEM: m=10 2 GMLEBEM: m=10 4 GMLEBEM: m= x Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

16 Accelerating EM There is a large literature on accelerating EM iterations, but none of the recent developments seem to help very much. However, the Kiefer-Wolfowitz problem can be reformulated as a convex maximum likelihood problem and solved by standard interior point methods: can be rewritten as, max f F min{ n m log( φ(y i u j )f j ), i=1 j=1 n log(g i ) Af = g, f S}, i=1 where A = (φ(y i u j )) and S = {s R m 1 s = 1, s 0}. So f j denotes the estimated mixing density estimate ˆf at the grid point u j, and g i denotes the estimated mixture density estimate, ĝ, at Y i. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

17 Interior Point vs. EM g(x) GMLEBIP GMLEBEM: m=10 2 GMLEBEM: m=10 4 GMLEBEM: m= x Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

18 Interior Point vs. EM In the foregoing test problem we have n = 200 observations and m = 300 grid points. Timing and accuracy is summarized in this table. Estimator EM1 EM2 EM3 IP Iterations , , Time L(g) Comparison of EM and Interior Point Solutions: Iteration counts, log likelihoods and CPU times (in seconds) for three EM variants and the interior point solver. Scaling problem sizes up, the deficiency of the EM approach is even more serious. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

19 Johnstone and Silverman Simulation Design Data is generated from 12 distinct models, all of the form: Y i = µ i + u i, u i N(0, 1), i = 1,..., Of the n = 1000 observations n k of the µ i = 0, and the remaining k take one of the four values {3, 4, 5, 7}. There are three choices of k: {5, 50, 500}. There are 50 replications for each of the 12 experimental settings and 18 different competing estimators. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

20 Johnstone and Silverman Simulation Design Data is generated from 12 distinct models, all of the form: Y i = µ i + u i, u i N(0, 1), i = 1,..., Of the n = 1000 observations n k of the µ i = 0, and the remaining k take one of the four values {3, 4, 5, 7}. There are three choices of k: {5, 50, 500}. There are 50 replications for each of the 12 experimental settings and 18 different competing estimators. Performance is measured by the mean (over replications) of the sum (over the n = 1000 observations) of squared errors, so a score of 500 means that the mean squared prediction error is 0.5, or half of what the naïve prediction ˆµ i = Y i would yield if the µ i were all zero. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

21 Johnstone and Silverman Simulation Results Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

22 Performance of the NP-MLE Bayes Rule In the (now familiar) Johnstone and Silverman sweepstakes we have the following comparison of performance. Estimator k = 5 k = 50 k = ˆδ MLE IP ˆδ MLE EM ˆδ δ J-S Min Here MLE-EM is Jaing and Zhang s (2009) Bayes rule with their suggested 100 EM iterations. It does somewhat better than the shape constrained estimator, but the interior point version MLE-IP does even better. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

23 The Castillo and van der Vaart Experiment The setup is quite similar to the first earlier ones, Y i = θ i + u i, i = 1, n the θ i are most zero, but s of them take one of the values from the set {1, 2,, 5}. The sample size is n = 500, and s {25, 50, 100} and θ a takes five possible values: The first 8 rows of the Table are taken directly from Table 1 of Castillo and van der Vaart (2012). s = 25 s = 50 s = PM PM EBM PMed PMed EBMed HT HTO EBMR EBKM MSE based on 1000 replications Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

24 But How Does It Work in Theory? For the Gaussian location mixture problem empirical Bayes rules based on the Kiefer-Wolfowitz estimator are adaptively minimax. Theorem: Jiang and Zhang For the normal location mixture problem, with a (complicated) weak pth moment restriction on Θ, the approximate non-parametric MLE, ˆθ = ˆδˆF n (Y) is adaptively minimax, i.e. sup θ E n,θ L n (ˆθ, θ) inf θ sup θ Θ E n,θ L n ( θ, θ) 1. The weak pth moment condition encompasses a much broader class of both deterministic and stochastic classes Θ. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

25 Gaussian Mixtures with Longitudinal Data Model: y it = µ i + θ i u it, t = 1,, m i, 1,, n, u it N(0, 1) Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

26 Gaussian Mixtures with Longitudinal Data Model: y it = µ i + θ i u it, t = 1,, m i, 1,, n, u it N(0, 1) Sufficient Statistics: ˆµ i = m 1 i m i y it N(µ i, θ i /m i ) t=1 m i ˆθ i = (m i 1) 1 (y it ˆµ i ) 2 Γ(r i, θ i /r i ), r i = (m i 1)/2 t=1 Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

27 Gaussian Mixtures with Longitudinal Data Model: y it = µ i + θ i u it, t = 1,, m i, 1,, n, u it N(0, 1) Sufficient Statistics: ˆµ i = m 1 i m i y it N(µ i, θ i /m i ) t=1 m i ˆθ i = (m i 1) 1 (y it ˆµ i ) 2 Γ(r i, θ i /r i ), r i = (m i 1)/2 Likelihood L(F y) = t=1 n φ((ˆµ i µ i )/ θm i )/ θ i m i γ(ˆθ i r i, θ i /r i )df µ (µ)df θ (θ) i=1 Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

28 A Toy Example Model y it = µ i + θ i u it, t = 1,, m i, 1,, n, u it N(0, 1) µ i 1 3 (δ δ 1 + δ 3 ) θ i 1 3 (δ δ 2 + δ 4 ) Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

29 A Toy Example Model y it = µ i + θ i u it, t = 1,, m i, 1,, n, u it N(0, 1) µ i 1 3 (δ δ 1 + δ 3 ) θ i 1 3 (δ δ 2 + δ 4 ) Mean Mixing Distribution Variance Mixing Distribution Mixture Distribution Bayes Rule f(µ) f(θ) g(µ) δ(x) µ θ µ x Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

30 Contour Plot for Joint Bayes Rule: δ(ˆµ, ˆθ) = E(µ ˆµ, ˆθ) µ^ σ 2 ^ Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

31 Empirical Bayesball Using (ESPN) data we have constructed an unbalanced panel, 10,575 observations, on 1072 players from Following standard practice, Brown (2009, AoAS) and Jiang and Zhang (2010, Brown Festschrift) we transform batting averages to (approximate) normality: ( ) H i1 + 1/4 Ŷ i = asin N(asin( ρ), 1/(4N i1 )) N i1 + 1/2 Treating these observations as approximately Gaussian, we compute sample means and variances for each player through 2011, and estimate our independent prior model. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

32 Prior Estimates on the Gaussian Scale f(µ) f(θ) µ θ Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

33 Prior Estimates on the Batting Average Scale Gaussian Model Binomial Model f(p) f(p) p p Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

34 Dirichlet Prior Estimates on the Batting Average Scale α = 1 α = 0.01 f(µ) f(µ) µ µ Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

35 Bayes Rule Predictions Binomial Model Prior Batting Average 2012 Batting Average Gaussian Model Prior Batting Average 2012 Batting Average Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

36 Covariate Effects The location-scale mixture model is really just a starting point for more general panel data models with covariate effects and unobserved heterogeneity estimable by profile likelihood. Given the model, y it = x it β + α i + σ i u it, and a fixed β R p, we have sufficient statistics ȳ i x i β, for α i and S i = 1 m i 1 m i (y it x it β (ȳ i x i β)) 2 t=1 for σ 2 i. Clearly, ȳ i α i, β, σ 2 i N(α i + x i β, σ 2 i ) and S i β, σ 2 i Γ(r i, σ 2 i /r i), where, r i = (m i 1)/2. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

37 Profile Likelihood for Covariate Effects Reducing the likelihood to sufficient statistics we have (almost) a decomposition in terms of within and between information: L(α, β, σ) = = = K n g((α, β, σ) y i1,..., y imi ) i=1 n mi σ 1 i φ((y it x it β α i )/σ i )h(α i, σ i )dα i dσ i i=1 n i=1 t=1 S 1 r i i σ 1 i where R i = r i S i /σ 2 i, r i = (m i 1)/2, and K = n i=1 φ((ȳ i x i β α i )/σ i ) e R i R r i i S i Γ(r i ) h(α i, σ i )dα i dσ i ( Γ(ri ) r r i (1/ ) 2π) mi 1 ). i Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

38 Profile Likelihood for Covariate Effects Reducing the likelihood to sufficient statistics we have (almost) a decomposition in terms of within and between information: L(α, β, σ) = = = K n g((α, β, σ) y i1,..., y imi ) i=1 n mi σ 1 i φ((y it x it β α i )/σ i )h(α i, σ i )dα i dσ i i=1 n i=1 t=1 S 1 r i i σ 1 i where R i = r i S i /σ 2 i, r i = (m i 1)/2, and K = n i=1 φ((ȳ i x i β α i )/σ i ) e R i R r i i S i Γ(r i ) h(α i, σ i )dα i dσ i ( Γ(ri ) r r i (1/ ) 2π) mi 1 ). i But note that the likelihood doesn t factor so the between and within information isn t independent. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

39 Age and Batting Ability There is considerable controversy about the relationship between player s age and their batting ability. To explore this we collected (reported) birth years for each of the players and reestimated the model including both linear and quadratic age effects using the profile likelihood method. We evaluate the profile likelihood on a grid of parameter values, but as you will see the likelihood is quite smooth and well behaved so higher dimensional problems could be done with standard optimization software. Evaluations of the profile likelihood are quick, a few seconds for our application, with grids of a few hundred points for the mixing distributions. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

40 Profile Likelihood for the Linear Age Effect b Profile Likelihood Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

41 6 7 Contour Plot of the Quadratic Age Effect β β 1 Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

42 The Mixing Densities at the Profile MLE h(α) h(σ) α σ Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

43 The Estimated Quadratic Age Effect Age Effect Age Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

44 Predictive Performance Gaussian model Prior Batting Avergae 2012 Batting Average Naive Gaussx Gauss Binomial model Prior Batting Avergae 2012 Batting Average Naive Bmix Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

45 Predictive Performance Root Mean Squared Prediction Error Gaussian Gaussian Binomial Dirichlet Dirichlet naive Age Effects (α = 1) (α = 0.01) Dismal performance due to multimodality of the estimated mixing distribution not much shrinkage compared to naive estimator. Prior performance is not very useful for predicting future performance Model with age covariates performs slightly better. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

46 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

47 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Kernel based empirical Bayes rules can be improved with shape constrained MLEs and are computationally very efficient, but Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

48 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Kernel based empirical Bayes rules can be improved with shape constrained MLEs and are computationally very efficient, but Kiefer-Wolfowitz type non-parametric MLEs perform even better. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

49 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Kernel based empirical Bayes rules can be improved with shape constrained MLEs and are computationally very efficient, but Kiefer-Wolfowitz type non-parametric MLEs perform even better. There are many opportunities for linking such methods to various semi-parametric estimation problems a la Heckman and Singer (1983) and van der Vaart (1996) as for the baseball problem, Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

50 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Kernel based empirical Bayes rules can be improved with shape constrained MLEs and are computationally very efficient, but Kiefer-Wolfowitz type non-parametric MLEs perform even better. There are many opportunities for linking such methods to various semi-parametric estimation problems a la Heckman and Singer (1983) and van der Vaart (1996) as for the baseball problem, It is all downhill after 27 in mathematics and baseball, Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36

51 Conclusions and Extrapolations Empirical Bayes methods, employing maximum likelihood, offer some advantages over other thresholding and kernel methods, Kernel based empirical Bayes rules can be improved with shape constrained MLEs and are computationally very efficient, but Kiefer-Wolfowitz type non-parametric MLEs perform even better. There are many opportunities for linking such methods to various semi-parametric estimation problems a la Heckman and Singer (1983) and van der Vaart (1996) as for the baseball problem, It is all downhill after 27 in mathematics and baseball, Be cautious about predicting baseball batting averages, or anything else about baseball. Roger Koenker (UIUC) Empirical Bayes Tokyo: / 36