Prior information, but no MCMC:

Size: px

Start display at page:

Download "Prior information, but no MCMC:"

Nathaniel Malone
5 years ago
Views:

Physikalisch-Technische Bundesanstalt Braunschweig and Berlin National Metrology Institute Prior information, but no MCMC: A Bayesian Normal linear regression case study

1 Physikalisch-Technische Bundesanstalt Braunschweig and Berlin National Metrology Institute Prior information, but no MCMC: A Bayesian Normal linear regression case study K. Klauenberg, G. Wübbeler, B. Mickan, P. Harris and C. Elster Physikalisch-Technische Bundesanstalt, Berlin and Braunschweig National Physical Laboratory, Teddington, UK

2 Content Linear Regression Regression problems frequent task in statistics and sciences similar, previous regr. often ˆ= prior information Bayesian inference beneficial BUT MCMC: no-go for practitioners Linear regression extensive analytical possibilities illustrate and exemplify Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 2/21

3 Content Content Linear Regression Linear regression: Notation, running example : Prior information prior distribution Analytic calculations: Posterior distr., estimates, regression, prediction Assessment of model assumptions (partially analytic) for practitioners Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 3/21

4 Linear Regression: Notation, Example Content Linear Regression Normal linear regression y = X θ + ε, ε N ( 0, σ 2 ) I dependent variable y = (y 1,..., y n ) covariates x j in design matrix X = ( x 1,..., x p) error ε: additive, iid Normal with variance σ 2 Likelihood l(θ, σ; y) = 1 (2πσ 2 ) estimate parameters θ (and σ 2 ) 1 e 2σ n/2 2 (y X θ) (y X θ) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 4/21

5 Linear Regression: Notation, Example Content Linear Regression Normal linear regression Example y = X θ + ε, ε N ( 0, σ 2 ) I dependent variable y = (y 1,..., y n ) covariates x j in design matrix X = ( x 1,..., x p) error ε: additive, iid Normal with variance σ 2 straight line model y i = θ 1 + θ 2 x i + ε i sonic nozzle: [ISO 9300, 2015] physics: linear: effective throat area, 1/Reynolds # θ 1, θ 2 characteristic values (calibration) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 4/21

6 Prior Information NIG Prior Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 5/21

7 Prior Information Prior Information NIG Prior Similar, previous regressions e.g. experiment rarely performed in isolation usually: no exact repetition, different sources,... (e.g. experim. conditions changed) e.g. [O Hagan et al., 2006] prior posterior of past regression Example Calibration of sonic nozzle effective area in mm Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 6/21

8 Prior Information Prior Information NIG Prior Similar, previous regressions e.g. experiment rarely performed in isolation usually: no exact repetition, different sources,... (e.g. experim. conditions changed) e.g. [O Hagan et al., 2006] prior posterior of past regression Example effective area in mm Calibration of sonic nozzle Pool posteriors & approximate depends on shape/appl. average posterior (linear pooling, [Genest and Zidek, 1986]) fit parametric family (conjugate: NIG distr.) Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 6/21

9 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) that is σ 2 IG(α 0, β 0 ), θ σ 2 N(θ 0, σ 2 V0) NIG Prior Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

10 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information NIG Prior (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) that is σ 2 IG(α 0, β 0 ), θ σ 2 N(θ 0, σ 2 V0) ) marginally θ t 2α0 (θ 0, β 0 α 0 V0 ) prior regression: X θ t 2α0 (X θ 0, β 0 α XV0X 0 posterior from same family for Normal likelihood Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

11 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information NIG Prior Parameters of NIG prior (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) e.g. marginal fits by method of moments Example density in 1 mm past knowledge past estimates NIG prior intercept in mm 2 slope in mm 2 σ in mm 2 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

12 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information NIG Prior Parameters of NIG prior (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) e.g. marginal fits by method of moments if average of the marginals has means, (co)variances µ IG, µ t, s 2 IG, S t θ 0 = µ t, V 0 = α0 1 β S 0 t, α 0 = µ2 IG + 2, β sig 2 0 = (α 0 1) µ IG if each past regression yielded NIG(m i, S i, a i, b i ) µig = 1 b i k a i 1, µt = 1 m i, k i i sig 2 = 1 bi 2 k (a i 1) 2 (a i 2) + 1 ( ) 2 bi k a i 1 µig, i i S t = 1 b i k a i 1 Si + 1 (mi µt)(mi µt)t k i Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

13 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information NIG Prior Parameters of NIG prior (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) e.g. marginal fits by method of moments Example Calibration of sonic nozzle Check adequacy of prior e.g. graphically effective area in mm slope, intercept, regression: ok yellow series: small variability unlikely under prior Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

14 Normal Inverse Gamma Prior Conjugate prior distribution Prior Information NIG Prior Parameters of NIG prior (θ, σ 2 ) NIG(θ 0, V0, α 0, β 0 ) e.g. marginal fits by method of moments Example density in 1 mm past knowledge past estimates NIG prior intercept in mm 2 slope in mm 2 σ in mm 2 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 7/21

15 Normal Inverse Gamma Prior II Prior Information NIG Prior Similarly also other sources of information (e.g. expert knowledge [O Hagan et al., 2006, Garthwaite et al., 2005]) Advantages conjugacy: no MCMC special cases known variance σ 2 BUT vague and certain non-informative priors (later) NIG prior: no default, caution needed check for unexpected effects by sensitivity analysis Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 8/21

16 Posterior Distribution Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 9/21

17 Posterior Distribution Posterior Distribution Posterior is normal inverse Gamma distr. with parameters θ 1 = V1 = θ, σ 2 y NIG(θ 1, V1, α 1, β 1 ) ( V X X ) 1 ( V X X ) 1 ( V 1 ) 0 θ 0 + X y α 1 = α n β 1 = β ( ) θ0 2 V 0 1 θ 0 + y y θ1 V 1 1 θ 1 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 10/21

18 Posterior Distribution Posterior Distribution Posterior is normal inverse Gamma distr. θ, σ 2 y NIG(θ 1, V1, α 1, β 1 ) ) marginally θ y t 2α1 (θ 1, β 1 α 1 V1 posterior regression: X θ y t 2α1 (X θ 1, β 1 α 1 XV1X ) prediction y new y t 2α1 (X new θ 1, β1 α 1 ( I + X new V 1 X new )) for y new = X new θ + ε new, ε new N(0, σ 2 I ) estimates, uncertainties, credible intervals (analytic and look-up tables) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 10/21

19 Posterior Distribution Posterior Distribution Posterior is normal inverse Gamma distr. θ, σ 2 y NIG(θ 1, V1, α 1, β 1 ) Example Calibration of sonic nozzle effective area in mm small range of regression: due to restrictions (e.g. on-site calibr. check) Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 10/21

20 Posterior Distribution Posterior Distribution Posterior is normal inverse Gamma distr. Example effective area in mm θ, σ 2 y NIG(θ 1, V1, α 1, β 1 ) Calibration of sonic nozzle Data Posterior regression Posterior prediction Re If data sparse/noisy: prior influential (+ vice versa) beneficial, but explore sensitivity Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 10/21

21 Posterior Distribution Posterior is normal inverse Gamma distr. Example θ, σ 2 y NIG(θ 1, V1, α 1, β 1 ) Posterior Distribution density in 1 mm NIG prior NIG posterior intercept in mm slope in mm 2 σ in mm 2 If data sparse/noisy: prior influential (+ vice versa) beneficial, but explore sensitivity Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 10/21

22 Prior Uncertainty Model Adequacy Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 11/21

23 Assessment of Model Prior Uncertainty Model Adequacy Uncertainty remains in approximation: prior knowledge by distribution [Robert, 2007] assumptions of statistical model data acquisition (e.g. mistakes) evaluate influence on inference by assessment by experts in field varying prior within uncertainty model checking and comparison (Prior, model, data not separable) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 12/21

24 Prior Uncertainty By varying parameters of (NIG) prior distribution (analytic) increase vagueness (Prior A) Prior Uncertainty Model Adequacy α 0 β 0 θ 0 V0 Prior A θ 0 2V0 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

25 Prior Uncertainty By varying parameters of (NIG) prior distribution (analytic) increase vagueness (Prior A) noninformative prior as reference e.g. indep. Jeffreys prior ([Jeffreys, 1967], Prior C) Prior Uncertainty Model Adequacy α 0 β 0 θ 0 V0 Prior A θ 0 2V0 Prior C 1 0 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

26 Prior Uncertainty Prior Uncertainty Model Adequacy By varying parameters of (NIG) prior distribution (analytic) increase vagueness (Prior A) noninformative prior as reference e.g. indep. Jeffreys prior ([Jeffreys, 1967], Prior C) prior info enables prediction effective area in mm Calibration of sonic nozzle * Post. regr. Post. regr. (Prior A) Post. regr. (Prior C) Data Post. pred. effective area in mm Calibration of sonic nozzle Re Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

27 Prior Uncertainty By varying parameters of (NIG) prior distribution (analytic) increase vagueness (Prior A) noninformative prior as reference e.g. indep. Jeffreys prior ([Jeffreys, 1967], Prior C) prior info enables prediction Prior Uncertainty Model Adequacy density in 1 mm NIG prior Prior A NIG posterior Post. (Prior A) Post. (Prior C) intercept in mm slope in mm 2 σ in mm 2 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

28 Prior Uncertainty Prior Uncertainty Model Adequacy By varying parameters of (NIG) prior distribution (analytic) increase vagueness (Prior A) noninformative prior as reference e.g. indep. Jeffreys prior ([Jeffreys, 1967], Prior C) prior info enables prediction α 0 β 0 θ 0 V0 Prior A θ 0 2V0 Prior B α 0 β 0 θ 0 V0 Prior C 1 0 distributional family (usually requires MCMC) e.g. independent Normal & inverse Gamma (Prior B) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

29 Prior Uncertainty By varying parameters of (NIG) prior distribution (analytic) prior info enables prediction Prior Uncertainty Model Adequacy distributional family (usually requires MCMC) e.g. independent Normal & inverse Gamma (Prior B) effective area in mm Calibration of sonic nozzle * Post. regr. Post. regr. (Prior B) Data Post. pred Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

30 Prior Uncertainty Prior Uncertainty Model Adequacy By varying parameters of (NIG) prior distribution (analytic) prior info enables prediction distributional family (usually requires MCMC) e.g. independent Normal & inverse Gamma (Prior B) effective area in mm Calibration of sonic nozzle * Post. regr. Post. regr. (Prior B) Data Post. pred judge relevance of influence for inference 1 Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 13/21

31 Adequacy of the Statistical Model Adequacy of regr. function & sampling distr. model comparison (usually requires MCMC) (e.g. competing physical expl., more robust distr.) e.g. ε t 4 ( 0, σ 2 I /2 ) more formal: Bayes factors, model selection, averaging [Kass and Raftery, 1995, George and McCulloch, 1997, Hoeting et al., 1999] Calibration of sonic nozzle Prior Uncertainty Model Adequacy effective area in mm * Post. regr. Post. regr. (t model) Data Post. pred Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 14/21

32 Adequacy of the Statistical Model Adequacy of regr. function & sampling distr. model comparison (usually requires MCMC) predictive model checks (if alternatives absent)[gelman et al., 1996] use post. predictive distr. y new y (analytic) Prior Uncertainty Model Adequacy Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 14/21

33 Adequacy of the Statistical Model Prior Uncertainty Model Adequacy Adequacy of regr. function & sampling distr. effective area in mm 2 model comparison (usually requires MCMC) predictive model checks (if alternatives absent)[gelman et al., 1996] use post. predictive distr. y new y (analytic) compare with additional / parts of / original data discrepancies w.r.t. data (y new vs. y)? * Calibration of sonic nozzle * * ** Additional data Post. prediction Re good agreement with add. measurements model, prior reasonable (in this respect) extrapolation permissible in this range Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 14/21

34 Adequacy of the Statistical Model Prior Uncertainty Model Adequacy Adequacy of regr. function & sampling distr. (unstandardized) residuals in mm 2 model comparison (usually requires MCMC) predictive model checks (if alternatives absent)[gelman et al., 1996] use post. predictive distr. y new y (analytic) compare with additional / parts of / original data discrepancies w.r.t. residuals? realized residuals predicted residuals (y new X i θ 1 ) vs. (y i X i θ 1 ) classical residual diagn. (ignores uncertainty of σ) Re Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 14/21

35 Adequacy of the Statistical Model Prior Uncertainty Model Adequacy Adequacy of regr. function & sampling distr. (unstandardized) residuals in mm 2 model comparison (usually requires MCMC) predictive model checks (if alternatives absent)[gelman et al., 1996] use post. predictive distr. y new y (analytic) compare with additional / parts of / original data discrepancies w.r.t. residuals? realized residuals predicted residuals (y new X i θ 1 ) vs. (y i X i θ 1 ) classical residual diagn. (ignores uncertainty of σ) Re explore/detect discrepancies of data & model (incl. prior) Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 14/21

36 Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 15/21

37 indep. implemented in MATLAB and R runs example, but easily adaptable new data, regr. functions: alter first 15 lines of code practitioners can do Bayesian linear regression gives regr. parameters, function, predictions & their estimates, uncertainties and CIs graphical display options: support elicitation/interpretation process available at PTB website with [Klauenberg et al., 2015] NormLinRegr Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 16/21

38 and Conclusions Acknowledgements References and Conclusions Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 17/21

39 and Conclusions and Conclusions Acknowledgements References and Conclusions highlighted analytical possibilities in linear regression illustrated how to include additional information NIG prior may represent knowledge from previous regressions benefit of prior info: may save money if trustworthy by compensating data enables practitioners to profit from Bayesian lin. regr.: no MCMC, analytic formulas, graphical tools and software Klauenberg, K., Wübbeler, G., Mickan, B., Harris, P., and Elster, C. (2015). A tutorial on Bayesian Normal linear regression. Metrologia developed guide on Bayesian regr. for metrologists [Elster et al., 2015] Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 18/21

Acknowledgements Part of this work has been funded by EMRP-project NEW 04 Novel mathematical and statistical approaches to uncertainty evaluation.

40 Acknowledgements Part of this work has been funded by EMRP-project NEW 04 Novel mathematical and statistical approaches to uncertainty evaluation. The EMRP is jointly funded by the EMRP participating countries within EURAMET and the European Union. and Conclusions Acknowledgements References and Conclusions Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 19/21

41 Elster, C., Klauenberg, K., Walzel, M., Wübbeler, G., Harris, P., Cox, M., and et al. (2015). A Guide to Bayesian Inference for Regression Problems. Deliverable of EMRP project NEW04 Novel mathematical and statistical approaches to uncertainty evaluation. Garthwaite, P. H., Kadane, J. B., and O Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470): and Conclusions Acknowledgements References and Conclusions Gelman, A., Meng, X. L., and Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6: Genest, C. and Zidek, J. V. (1986). Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1(1): George, E. I. and McCulloch, R. E. (1997). Approaches for Bayesian variable selection. Statistica Sinica, 7: Hoeting, J. A., Madigan, D., Raftery, A. E., and Volinsky, C. T. (1999). Bayesian model averaging: A tutorial. Statistical Science, 14: ISO 9300 (2015). Measurement of Gas Flow by Means of Critical Flow Venturi Nozzles, volume ISO 9300:2005(en). International Standards Organization. Jeffreys, H. (1967). Theory of probability. International series of monographs on physics. Oxford: Clarendon Press, 3rd edition. Kass, R. E. and Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430): Klauenberg, K., Wübbeler, G., Mickan, B., Harris, P., and Elster, C. (2015). A tutorial on Bayesian Normal linear regression. Metrologia. O Hagan, A., Buck, C., Daneshkhah, A., Eiser, J., Garthwaite, P., Jenkinson, D., Oakley, J., and Rakow, T. (2006). Uncertain Judgements: Eliciting Experts Probabilities. Statistics in Practice. Wiley. Robert, C. P. (2007). The Bayesian choice: From Decision-Theoretic Foundations to Computational Implementation. Springer Texts in Statistics. New York: Springer. Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 20/21

42 and Conclusions Thank you! and Conclusions Acknowledgements References and Conclusions highlighted analytical possibilities in linear regression illustrated how to include additional information NIG prior may represent knowledge from previous regressions benefit of prior info: may save money if trustworthy by compensating data enables practitioners to profit from Bayesian lin. regr.: no MCMC, analytic formulas, graphical tools and software Klauenberg, K., Wübbeler, G., Mickan, B., Harris, P., and Elster, C. (2015). A tutorial on Bayesian Normal linear regression. Metrologia developed guide on Bayesian regr. for metrologists [Elster et al., 2015] Bayesian Normal Linear Regression ENBIS-16, Sheffield, 13/09/2016 p. 21/21

Traceability of on-machine measurements under a wide range of working conditions

Traceability of on-machine measurements under a wide range of working conditions Frank Keller, Matthias Franke, Norbert Gerwien, Lisa Groos, Christian Held, Klaus Wendt Traceable in-process dimensional