Bayesian Empirical Likelihood: a survey

Transcription

1 Bayesian Empirical Likelihood 1 Bayesian Empirical Likelihood: a survey Art B. Owen Stanford University

2 Bayesian Empirical Likelihood 2 These are the slides I presented at BNP 11 in Paris, a very fine meeting. I have added a few interstitial slides like this one to include comments I made while presenting the slides and to give some background thoughts in this area. The session included a delightful presentation by Nils Lid Hjort on some not yet published theorems where he connects Bayesian empirical likelihood to Dirichlet process Bayesian nonparametrics. I was pleased to see our session got a positive review on Xian s Og (written by our session chair, Christian Robert). Also, the conference included two very strong posters by students working on Bayesian empirical likelihood. Frank Liu (Monash) has a nice way to robustify BETEL. Laura Turbatu (Geneva) took a close look at high order asymptotics to compare frequentist coverage of Bayes with various kinds of nonparametric priors.

3 Bayesian Empirical Likelihood 3 The goal Science driven prior Data driven likelihood

4 Bayesian Empirical Likelihood 4 This goal is not everybody s goal. But this goal means you don t have to make up a likelihood which could bring unwanted and consequential restrictions. If the data deliver the likelihood and the prior is chosen flat or by previously known science, then what is there left for the statistician to do?

5 Bayesian Empirical Likelihood 5 The goal Science driven prior Data driven likelihood The ask IID data and an estimand defined by estimating equations. X 1, X 2,..., X n iid F m(x, θ) df (x) = 0, θ R p

6 Bayesian Empirical Likelihood 6 We have IID X and an estimating equation. If you picture the plate diagram, it is one of the simplest possible ones. It covers an important set of problems, but Bayes includes much else as well.

7 Bayesian Empirical Likelihood 7 Outline 1) Empirical Likelihood 2) Bayes EL Lazar (2003) 3) Exponential tilting, and ETEL Schennach (2005,2007) 4) ABC and EL Mengersen, Pudio, Robert (2013) 5) BEL with Hamiltonian MCMC Chaudhuri et al (2017) 6) Some other recent works Notes Apologies about necessarily omitted papers. Motivation: bioinformatics.

8 Bayesian Empirical Likelihood 8 In the bioinformatics motivation, the data are counts. They are assuredly not Poisson. So a common response is to model them as negative binomial. Of course they are not that either. This is work in progress. SAMSI is having a year on QMC. The original slides touted the opening workshop. However the application deadline will almost surely have passed by the time you re reading this.

9 Bayesian Empirical Likelihood 9 Empirical likelihood For X i iid F observed Xi = x i R d, define n L(F ) = F ({x i }). i=1 There are many other NPMLEs. The nonparametric MLE (NPMLE) is F arg max F E.g., Kaplan-Meier for right censored survival, L(F ) = 1 n Log concave densities: Walther, Dumbgen, Rufibach, Cule, Samworth, Stewart n i=1 δ xi

10 Bayesian Empirical Likelihood 10 Basic notions Model: x i iid F True value: F = F0 NPMLE: F Estimand T (F ) E.g., population mean, median, regression coefficient etc. True value: θ 0 = T (F 0 ) NPMLE: ˆθ = T ( F )

11 Bayesian Empirical Likelihood 11 Empirical likelihood ratios Point est. Interval / Test Parametric ˆθ = arg maxθ R(θ) 2 log R(θ 0 ) χ 2 Non-parametric T ( F ) The purpose of empirical likelihood ratios is to fill in the, getting likelihood ratio confidence intervals and tests without parametric assumptions. I.e., a nonparametric Wilks theorem O (2000) Chapman & Hall, Monograph

12 Bayesian Empirical Likelihood 12 Let w i = w i (F ) = F ({x i }) Then w i 0 and n i=1 w i 1. Empirical likelihood ratio R(F ) = R(F ) R( F ) = n i=1 n w i 1/n = i=1 (nw i ) Literal EL confidence regions {T (F ) R(F ) c} Profile likelihood R(θ) = max{r(f ) T (F ) = θ} Conf set = {θ R(θ) c}. We still have to choose c

13 Bayesian Empirical Likelihood 13 Usual EL It is usual to force i w i = i F ({x i}) = 1. R(F ) c already implies 1 F ({x 1,..., x n }) = O(1/n) Supported on x 1,..., x n It is a multinomial likelihood There are other choices

14 Bayesian Empirical Likelihood 14 Empirical likelihood for the mean T (F ) = x df (x) { n R(µ) = max i=1 Profile likelihood nw i n i=1 w i x i = µ, w i 0, Confidence set {µ R(µ) c} n i=1 } w i = 1 To pick c 2 log(r(µ 0 )) d χ 2 (k), ( c = exp 1 ) 2 χ2,1 α (k) k = rank(var(x))

15 Bayesian Empirical Likelihood 15 Dipper, Cinclus cinclus Eats larvae of Mayflies, Stoneflies, Caddis flies, other Photo By Mark Medcalf. From Wikipedia. Lic under CC by

16 Bayesian Empirical Likelihood 16 Dipper diet means Stonefly larvae Other invertebrates Caddis fly larvae Mayfly larvae Stonefly larvae Other invertebrates Caddis fly larvae Top row shows EL; bottom Hotelling s T 2 ellipses Data from Iles (1993) 22 sites in Wales. Hall (1990) quantifies correctness of shape Mayfly larvae

17 Bayesian Empirical Likelihood 17 Computing EL for the mean Easy by convex duality. The dual is self-concordant = truncated Newton assured of convergence. O (2013) Canadian Journal of Statistics R code online

18 Bayesian Empirical Likelihood 18 Estimating equations θ solves E(m(x, θ)) = 0 Examples m(x, θ) x θ Estimand θ E(x) 1 x<θ 0.5 median(x) (x θ) 1 z A E(x z A) x(y x T β) regression θ log f(x, θ) MLE

19 Bayesian Empirical Likelihood 19 Note that some of these estimands can be well defined without making any assumptions about the likelihood. For instance, we could well want a median without wanting to assume a double exponential distribution or any other whose MLE is the median. Similarly, least squares estimates are interpretable as sample based counterparts to population least squares quantities. Other cases, like logistic regression, make it harder to interpret what you get if the motivating parametric model does not hold. That said, you might still prefer reliable to unreliable uncertainty quantification for such an estimand.

20 Bayesian Empirical Likelihood 20 EL properties High power Kitamura (2001), Lazar & Mykland (1998) Comparable to a true parametric model (when there is one) Bartlett correctable DiCiccio, Hall & Romano (1991) Coverage errors O(1/n) O(1/n 2 ) Challenges Convex hull issue, fixable by pseudo-data, Chen & Variyath (2008), Tsao & Wu (2013/14), Emerson & O (2009) Computing max{r(θ 1,..., θ p ) θ j = θ j0 } can be hard O (2000) used expensive NPSOL solver Possibility Sampling might work more easily than optimization

21 Bayesian Empirical Likelihood 21 Overdetermined estimating equations θ R p, m(x, θ) R q, q > p 1 n q equations in p < q unknowns: m(x i, θ) = 0 n i=1 Popular in econometrics E.g., generalized method of moments (GMM) Hansen (1982) E.g., regression through (0, 0) E(y xβ) = E(x(y xβ)) = 0 Misspecification Sometimes no θ R p gives E(m(x, θ)) = 0 R q

22 Bayesian Empirical Likelihood 22 EL for overdetermined Get n more multinomial parameters w 1,..., w n NPMLE max θ,w n i=1 w i subject to n w i m(x i, θ) = 0, i=1 n w i = 1, w i > 0 i=1 Then solve i ŵim(x i, θ) = 0 for ˆθ Newey & Smith (2004): same variance & less bias than GMM

23 Bayesian Empirical Likelihood 23 Bayesian EL p(θ x) p(θ)r(θ x) Lazar (2003) justifies by a Bayesian CLT, like Boos & Monahan (1992) Also Hjort (Today!) Is that Bayesian enough?

24 Bayesian Empirical Likelihood 24 Cressie-Read likelihoods There are numerous alternative nonparametric likelihoods. Chang & Mukerjee (2008) give conditions for them to have a very accurate BCLT for very general priors. EL is uniquely able to do so. Some others will work with a flat prior. Poster by Turbatu (yesterday) Also

25 Bayesian Empirical Likelihood 25 Least favorable families Pr EL (x i ) = w i = 1 n p-dimensional θ = p-dimensional λ(θ). There is a least favorable p-dimensional family for θ. Any other family makes the inference artificially easy λ(θ) T m(x i, θ) EL family is asymptotically the LFF DiCiccio & Romano (1990) The connection p(θ) induces a prior on the LFF, and R(θ x) =. likelihood on LFF = p(θ) R(θ x) =. posterior on LFF = posterior on θ

26 Bayesian Empirical Likelihood 26 To my knowledge nobody has tracked down exactly how the approximations work out for the least favorable family motivation of Bayesian EL.

27 Bayesian Empirical Likelihood 27 Schennach (2005) Exponential tilting (ET) To get an empirical probability model, for θ Θ R p, partition Θ = N l=1 Θ l p(θ) is multinomial on N disjoint cells. Now let N for universal approximation. Ultimately n p(θ x) p(θ) wi (θ), s.t. i=1 w i maximize n wi x i = θ, where n w i log(w i ) i=1 i=1 i w i = 1

28 Bayesian Empirical Likelihood 28 Exponential tilting EL Schennach (2007) Form parametric family via exponential tilting w i (θ) = e λ(θ)t m(x i,θ) n j=1 eλ(θ)t m(x j,θ), where 0 = n e λ(θ)t m(x i,θ) m(x i, θ) i=1 Likelihood L(θ) = n nwi (θ) i=1 ETEL ˆθ ETEL = arg max θ L(θ) Works better on misspecified overdetermined models, i.e., no θ R p makes E(m(x, θ)) = 0 R q Reason: at least there is an estimand.

29 Bayesian Empirical Likelihood 29 If we know that no θ can make E(m(x, θ)) = 0 then I m not sure why we still want an estimand. It seems like we should instead change the choice of m.

30 Bayesian Empirical Likelihood 30 BETEL Chib, Shin, Simoni (2016,2017) Use a slack parameter V k. V k 0 for non-truthfulness of moment condition k. At most m p nonzero V k. Metropolized Independence Sampling from a t distribution. Asymptotic model selection Use log marginal likelihood. If there is a true model then it will be selected over any false one. Among true models: most equations holding wins. poster by Liu (yesterday) Robust BETEL

31 Bayesian Empirical Likelihood 31 EL with ABC Mengersen, Pudlo & Robert (2013) Sample θ i p(θ) Compute w i = R(θ i x) Normalize the weights Basic EL-ABC Advanced EL-ABC Use adaptive multiple importance sampling (AMIS) Algorithm has 5 loops Max depth 3

32 Bayesian Empirical Likelihood 32 Hamiltonian MCMC Chaudhuri, Mondal & Yin (2017) Using p(θ)r(θ x) they report that: Gibbs is not applicable Random walk Metropolis has problems with irregular support of R(θ x) Use gradients, Hamiltonian MCMC. Gradient gets steep just where you need it. Their solution

33 Bayesian Empirical Likelihood 33 Vexler, Tao & Hutson (2014) For x i R Posterior moments x(n) x (1) x(n) x (1) θ k R(θ x)p(θ) dθ R(θ x)p(θ) dθ Nonparametric James-Stein For x i = (x i1,..., x ik ) R K, get R j (θ j ) = R(θ j x 1j,..., x nj ) ˆθ j = θj R j (θ)ϕ( θj θ 0 σ ) dθ j Rj (θ)ϕ((θ j θ 0 )/σ ) dθ j σ 2 = arg max σ 2 K j=1 ( 1 log 2πσ R j (θ j )e θ2 j /(2σ2) dθ j )

34 Bayesian Empirical Likelihood 34 Quantiles Vexler, Yu & Lazar (2017) EL Bayes factors for quantile θ Q α = Q 0 vs Q α Q 0 They overcome a technical challenge handling H A R(θ x)p(θ) dθ because R has jump discontinuities. Also two sample Bayes factors. If scientific or regulalatory interest is in a specific quantile then the user does not have to pick a whole parametric family.

35 Bayesian Empirical Likelihood 35 Quantile regression Pr(Y α + βx x) = τ, e.g., τ = 0.9 Estimating equations 0 = E ( 1 y α+βx τ ) 0 = E ( x ( 1 y α+βx τ )) Lancaster & Jun (2010) BETEL for quantiles similar to Jeffrey s prior Yang & He (2012) BCLT accounting for non smoothness of estimating equations Shrinking priors pool β for multiple τ Both use MCMC

36 Bayesian Empirical Likelihood 36 Summary We can do Bayes without requiring a parametric family for the observations. It is the other nonparametric Bayes.

37 Bayesian Empirical Likelihood 37 Thanks Luke Bornn, invitation Sanjay Chaudhuri, many references Scientific and organizing committees Especially Judith Rousseau NSF DMS , DMS