Bayesian Modeling Using WinBUGS

Size: px

Start display at page:

Download "Bayesian Modeling Using WinBUGS"

Beverley Hensley
5 years ago
Views:

1 Bayesian Modeling Using WinBUGS

2 WILEY SERIES IN COMPUTATIONAL STATISTICS Consulting Editors: Paolo Giudici University of Pavia, Italy Geof H. Givens Colorado State University, USA Bani K. Mallick Texas A&M University, USA Wiley Series in Computational Statistics is comprised of practical guides and cutting edge research books on new developments in computational statistics. It features quality authors with a strong applications focus. The texts in the series provide detailed coverage of statistical concepts, methods and case studies in areas at the interface of statistics, computing, and numerics. With sound motivation and a wealth of practical examples, the books show in concrete terms how to select and to use appropriate ranges of statistical computing techniques in particular fields of study. Readers are assumed to have a basic understanding of introductory terminology. The series concentrates on applications of computational methods in statistics to fields of bioinformatics, genomics, epidemiology, business, engineering, finance and applied statistics. A complete list of titles in this series appears at the end of the volume.

3 Bayesian Modeling Using WinBUGS Ioannis Ntzoufras Department of Statistics Athens University of Economics and Business Athens, Greece WILEY A JOHN WILEY & SONS, INC., PUBLICATION

4 Copyright Q 2009 by John Wiley & Sons, Inc. All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 1 River Street, Hoboken, NJ 07030, (201) , fax (201) , or online at Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) , outside the United States at (317) or fax (317) Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at Library of Congress Cataloging-in-Publication Data is available. Ntzoufras, Ioannis, Bayesian modeling using WinBUGS / Ioannis Ntzoufras. p. cm. Includes bibliographical references and index. ISBN (pbk.) 1. Bayesian statistical decision theory. 2. WinBUGS. I. Title. QA279.5.N '42Ac Printed in the United States of America

5 To Ioanna and our baby daughter

6 CONTENTS Preface Acknowledgments Acronyms xvii xix xxi 1 Introduction to Bayesian Inference 1.1 Introduction: Bayesian modeling in the 2 1 st century 1.2 Definition of statistical models 1.3 Bayes theorem 1.4 Model-based Bayesian inference 1.5 Inference using conjugate prior distributions Inference for the Poisson rate of count data Inference for the success probability of binomial data Inference for the mean of normal data with known variance Inference for the mean and variance of normal data Inference for normal regression models Other conjugate prior distributions Illustrative examples 1.6 Nonconjugate analysis Problems vii

7 viii CONTENTS 2 Markov Chain Monte Carlo Algorithms in Bayesian Inference 2.1 Simulation, Monte Carlo integration, and their implementation in Bayesian inference 2.2 Markov chain Monte Carlo methods The algorithm Terminology and implementation details 2.3 Popular MCMC algorithms The Metropolis-Hastings algorithm Componentwise Metropolis-Hastings The Gibbs sampler Metropolis within Gibbs The slice Gibbs sampler A simple example using the slice sampler 2.4 Summary and closing remarks Problems 3 WinBUGS Software: Introduction, Setup, and Basic Analysis 3.1 Introduction and historical background 3.2 The WinBUGS environment Downloading and installing WinBUGS A short description of the menus 3.3 Preliminaries on using WinBUGS Code structure and type of parametersinodes Scalar, vector, matrix, and array nodes 3.4 Building Bayesian models in WinBUGS Function description Using the for syntax and array, matrix, and vector calculations Use of parentheses, brackets and curly braces in WinBUGS Differences between WinBUGS and R/Splus syntax Model specification in WinBUGS Data and initial value specification An example of a complete model specification Data transformations Compiling the model and simulating values Basic output analysis using the sample monitor tool 3.7 Summarizing the procedure 3.8 Chapter summary and concluding comments Problems WinBUGS Software: Illustration, Results, and Further Analysis 4.1 A complete example of running MCMC in WinBUGS for a simple model

8 CONTENTS ix The model Data and initial values Compiling and running the model MCMC output analysis and results 4.2 Further output analysis using the inference menu Comparison of nodes Calculation of correlations Using the summary tool Evaluation and ranking of individuals Calculation of deviance information criterion 4.3 Multiple chains Generation of multiple chains Output analysis The Gelman-Rubin convergence diagnostic 4.4 Changing the properties of a figure General graphical options Special graphical options 4.5 Other tools and menus The node info tool Monitoring the acceptance rate of the Metropolis-Hastings algorithm Saving the current state of the chain Setting the starting seed number Running the model as a script 4.6 Summary and concluding remarks Problems 5 introduction to Bayesian Models: Normal Models 5.1 General modeling principles 5.2 Model specification in normal regression models Specifying the likelihood Specifying a simple independent prior distribution Interpretation of the regression coefficients A regression example using WinBUGS 5.3 Using vectors and multivariate priors in normal regression models Defining the model using matrices Prior distributions for normal regression models Multivariate normal priors in WinBUGS Continuation of Example Analysis of variance models The one-way ANOVA model Parametrization and parameter interpretation

9 X CONTENTS One-way ANOVA model in WinBUGS A one-way ANOVA example using WinBUGS Two-way ANOVA models Multifactor analysis of variance Problems Incorporating Categorical Variables in Normal Models and Further Modeling Issues Analysis of variance models using dummy variables Analysis of covariance models Models using one quantitative variable and one qualitative variable The parallel lines model The separate lines model 6.3 A bioassay example Parallel lines analysis Slope ratio analysis: Models with common intercept and different slope Comparison of the two approaches 6.4 Further modeling issues Extending the simple ANCOVA model Using binary indicators to specify models in multiple regression Selection of variables using the deviance information criterion PIC) 6.5 Closing remarks Problems Introduction to Generalized Linear Models: Binomial and Poisson Data 7.1 Introduction The exponential family Common distributions as members of the exponential family Link functions Common generalized linear models Interpretation of GLM coefficients 7.2 Prior distributions 7.3 Posterior inference The posterior distribution of a generalized linear model GLM specification in WinBUGS 7.4 Poisson regression models Interpretation of Poisson log-linear parameters A simple Poisson regression example

10 CONTENTS xi A Poisson regression model for modeling football data binomial response models Interpretation of model parameters in binomial response models A simple example Models for contingency tables 269 Problems Models for Positive Continuous Data, Count Data, and Other GLM- Based Extensions Models with nonstandard distributions Specification of arbitrary likelihood using the zeros-ones trick The inverse Gaussian model 8.2 Models for positive continuous response variables The gamma model Other models An example 8.3 Additional models for count data The negative binomial model The generalized Poisson model Zero inflated models The bivariate Poisson model The Poisson difference model 8.4 Further GLM-based models and extensions Survival analysis models Multinomial models Additional models and further reading Problems Bayesian Hierarchical Models 9.1 Introduction A simple motivating example Why use a hierarchical model? Other advantages and characteristics 9.2 Some simple examples Repeated measures data Introducing random effects in performance parameters Poisson mixture models for count data The use of hierarchical models in meta-analysis 9.3 The generalized linear mixed model formulation A hierarchical normal model: A simple crossover trial Logit GLMM for correlated binary responses

11 xii CONTENTS Poisson log-linear GLMMs for correlated count data Discussion, closing remarks, and further reading Problems The Predictive Distribution and Model Checking 10.1 Introduction Prediction within Bayesian framework Using posterior predictive densities for model evaluation and checking Cross-validation predictive densities 10.2 Estimating the predictive distribution for future or missing observations using MCMC A simple example: Estimating missing observations An example of Bayesian prediction using a simple model 10.3 Using the predictive distribution for model checking Comparison of actual and predictive frequencies for discrete data Comparison of cumulative frequencies for predictive and actual values for continuous data Comparison of ordered predictive and actual values for continuous data Estimation of the posterior predictive ordinate Checking individual observations using residuals Checking structural assumptions of the model Checking the goodness-of-fit of a model 10.4 Using cross-validation predictive densities for model checking, evaluation, and comparison Estimating the conditional predictive ordinate Generating values from the leave-one-out cross-validatory predictive distributions 10.5 Illustration of a complete predictive analysis: Normal regression models Checking structural assumptions of the model Detailed checks based on residual analysis Overall goodness-of-fit of the model Implementation using WinBUGS An Illustrative example Summary of the model checking procedure 10.6 Discussion Problems Bayesian Model and Variable Evaluation 389

12 CONTENTS xiii 11.1 Prior predictive distributions as measures of model comparison: Posterior model odds and Bayes factors 11.2 Sensitivity of the posterior model probabilities: The Lindley-Bartlett paradox 11.3 Computation of the marginal likelihood Approximations based on the normal distribution Sampling from the prior: A naive Monte Carlo estimator Sampling from the posterior: The harmonic mean estimator Importance sampling estimators Bridge sampling estimators Chib s marginal likelihood estimator Additional details and further reading 11.4 Computation of the marginal likelihood using WinBUGS A beta-binomial example A normal regression example with conjugate normal-inverse gamma prior 11.5 Bayesian variable selection using Gibbs-based methods Prior distributions for variable selection in GLM Gibbs variable selection Other Gibbs-based methods for variable selection Posterior inference using the output of Bayesian variable selection samplers Implementation of Gibbs variable selection in WinBUGS using an illustrative example The Carlin-Chib method reversible jump MCMC (RJMCMC) Using posterior predictive densities for model evaluation Estimation from an MCMC output A simple example in WinBUGS Information criteria The Bayes information criterion (BIC) The Akaike information criterion (AIC) Other criteria Calculation of penalized deviance measures from the MCMC output Implementation in WinBUGS A simple example in WinBUGS Discussion and further reading 432 Problems 432 Appendix A: Model Specification via Directed Acyclic Graphs: The DOODLE Menu 435 A. 1 Introduction: Starting with DOODLE 435

13 xiv CONTENTS A.2 Nodes A.3 Edges A.4 Panels A.5 A simple example Appendix B: The Batch Mode: Running a Model in the Background Using Scripts 443 B. 1 Introduction 443 B.2 Basic commands: Compiling and running the model 444 Appendix C: Checking Convergence Using CODNBOA C. 1 Introduction C.2 A short historical review C.3 Diagnostics implemented by CODMBOA C.3.1 The Geweke diagnostic C.3.2 The Gelman-Rubin diagnostic C.3.3 The Raftery-Lewis diagnostic C.3.4 The Heidelberger-Welch diagnostic C.3.5 Final remarks (2.4 A first look at CODMBOA C.4.1 CODA C.4.2 BOA C.5 A simple example C.5.1 Illustration in CODA C.5.2 Illustration in BOA Appendix D: Notation Summary D.l MCMC D.2 Subscripts and indices D.3 Parameters D.4 Random variables and data D.5 Sample estimates D.6 Special functions, vectors, and matrices D.7 Distributions D.8 Distribution-related notation D.9 Notation used in ANOVA and ANCOVA D. 10 Variable and model specification D. 11 Deviance information criterion (DIC) D. 12 Predictive measures

14 CONTENTS XV References Index

15 PREFACE Since the mid- 1980s, the development of widely accessible powerfbl computers and the implementation of Markov chain Monte Carlo (MCMC) methods have led to an explosion of interest in Bayesian statistics and modeling. This was followed by an extensive research for new Bayesian methodologies generating the practical application of complicated models used over a wide range of sciences. During the late 199Os, BUGS emerged in the foreground. BUGS was a free software that could fit complicated models in a relatively easy manner, using standard MCMC methods. Since 1998 or so, WinBUGS, the Windows version of BUGS, has earned great popularity among researchers of diverse scientific fields. Therefore, an increased need for an introductory book related to Bayesian models and their implementation via WinBUGS has been realized. The objective of the present book is to offer an introduction to the principles of Bayesian modeling, with emphasis on model building and model implementation using WinBUGS. Detailed examples are provided, ranging from very simple to more advanced and realistic ones. Generalized linear models (GLMs), which are familiar to most students and researchers, are discussed. Details concerning model building, prior specification, writing the WinBUGS code and the analysis and interpretation of the WinBUGS output are also provided. Because of the introductory character of the book, I focused on elementary models, starting from the normal regression models and moving to generalized linear models. Even more advanced readers, familiar with such models, may benefit from the Bayesian implementation using WinBUGS, Basic knowledge ofprobability theory and statistics is assumed. Computations that could not be performed in WinBUGS are illustrated using R. Therefore, a minimum knowledge of R is also required. xvii

16 xviii PREFACE This manuscript can be used as the main textbook in a second-level course of Bayesian statistics focusing on modeling and/or computation. Alternatively, it can serve as a companion (to a main textbook) in an introductory course of a Bayesian statistics. Finally, because of its structure, postgraduate students and other researchers can complete a self-taught tutorial course on Bayesian modeling by following the material of this book. All datasets and code used in the book are available in the book s Webpage: www. stat-athens.aueb.gr/ jbn/winbugs-book. Athens, Greece June IOANNIS NTZOUFRAS

17 ACKNOWLEDGMENTS I am indebted to the people at Wiley publications for their understanding and assistance during the preparation of the manuscript. Acknowledgments are due to the anonymous referees. Their suggestions and comments led to a substantial improvement of the present book. I would particularly like to thank Dimitris Fouskakis, colleague and good friend, for his valuable comments on an early version of chapters 1-6 and I am also grateful to Professor Brani Vidakovic for proposing and motivating this book. Last but not least, I wish to thank my wife Ioanna for her love, support, and patience during the writing of this book as well as for her suggestions on the manuscript. I. N. xix

18 ACRONYMS ACF AIC ANOVA ANCOVA AR BF BIC BOA BP BOD BUGS CDF COD CODA CPO CR cv Autocorrelation Akaike information criterion Analysis of variance Analysis of covariance Attributable risk Bayes factor Bayes information criterion Bayesian output analysis (R package) Bivariate Poisson Biological oxygen demand (data variable in example 6.3) Bayesian inference using Gibbs (software) Cumulative distribution function Chemical oxygen demand (data variable in example 6.3) Convergence diagnostics and output analysis software for Gibbs sampling analysis (R package) Conditional Predictive Ordinate corner (constraint) Cross-validation xxi

19 xxii Acronyms cv- 1 DAG DI DIBP DIC GLM GP GVS ICPO i.i.d. LS MAP MP model MCMC MCE ML MLE NB OR PBF PD p.d.f. PO PPO RJMCMC RR SD SE ssvs STZ TS TVS WinBUGS ZI ZID ZIP ZINB Leave-one-out cross-validation Directed acyclic graph Dispersion index Diagonal inflated bivariate Poisson distribution Deviance information criterion Generalized linear model Generalized Poisson Gibbs variable selection Inverse conditional predictive ordinate Independent identically distributed Logarithmic score Maximum a posteriori Median probability Markov chain Monte Carlo Monte Carlo error Maximum likelihood Maximum-likelihood estimate/estimator Negative binomial Odds ratio Posterior Bayes factor Poisson difference Probability density function Posterior model odds Posterior predictive ordinate Reversible jump Markov chain Monte Carlo Relative risk Standard deviation Standard error Stochastic search variable selection sum-to-zero (constraint) Total solids(data variable in example 6.3) Total volatile solids (data variable in example 6.3) Windows version of BUGS (software) Zero inflated Zero inflated distribution Zero inflated Poisson distribution Zero inflated negative binomial distribution

20 Acronyms xxiii ZIGP ZIBP Zero inflated generalized Poisson distribution Zero inflated bivariate Poisson distribution

AN INTRODUCTION TO PROBABILITY AND STATISTICS

AN INTRODUCTION TO PROBABILITY AND STATISTICS WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Noel A. C. Cressie, Garrett M.