Bayesian Models for Categorical Data

Size: px

Start display at page:

Download "Bayesian Models for Categorical Data"

Erick Kennedy
5 years ago
Views:

1 Bayesian Models for Categorical Data PETER CONGDON Queen Mary, University of London, UK Chichester New York Weinheim Brisbane Singapore Toronto

3 Bayesian Models for Categorical Data

4 WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Peter Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Geert Molenberghs, Louise M. Ryan, David W. Scott, Adrian F. M. Smith, Jozef L. Teugels; Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall A complete list of the titles in this series appears at the end of this volume.

5 Bayesian Models for Categorical Data PETER CONGDON Queen Mary, University of London, UK Chichester New York Weinheim Brisbane Singapore Toronto

6 Copyright # 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) (for orders and customer service enquiries): cs-books@wiley.co.uk Visit our Home Page on All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or ed to permreq@wiley.co.uk, or faxed to (+44) Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA , USA Wiley-VCH Verlag GmbH, Boschstr. 12, D Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop # 02-01, Jin Xing Distripark, Singapore John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Congdon, P. Bayesian models for categorical data/peter Congdon. p. cm. (Wiley series in probability and statistics) Includes bibliographical references and index. ISBN (cloth : alk. paper) 1. Bayesian statistical decision theory. 2. Monte Carlo method. 3. Markov processes. 4. Multivariate analysis. I. Title. II. Series. QA279.5.C dc British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN (HB) ISBN (HB) Typeset in 11/13pt Times by Thomson Press (India) Limited, New Delhi, India. Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

7 Contents Preface xi Chapter 1 Principles of Bayesian Inference Bayesian updating MCMC techniques The basis for MCMC MCMC sampling algorithms MCMC convergence Competing models Setting priors The normal linear model and generalized linear models Data augmentation Identifiability Robustness and sensitivity Chapter themes 23 References 23 Chapter 2 Model Comparison and Choice Introduction: formal methods, predictive methods and penalized deviance criteria Formal Bayes model choice Marginal likelihood and Bayes factor approximations Predictive model choice and checking Posterior predictive checks Out-of-sample cross-validation Penalized deviances from a Bayes perspective Multimodel perspectives via parallel sampling Model probability estimates from parallel sampling Worked example 49 References 50

8 vi CONTENTS Chapter 3 Regression for Metric Outcomes Introduction: priors for the linear regression model Regression model choice and averaging based on predictor selection Robust regression methods: models for outliers Robust regression methods: models for skewness and heteroscedasticity Robustness via discrete mixture models Complete data representation Identification issues Dirichlet process mixture models Non-linear regression effects via splines and other basis functions Penalized random effects for spline coefficients Basis function regression Special spline functions Dynamic linear models and their application in non-parametric regression Some common forms of DLM Robust errors General additive models Alternative smoothness priors 100 Exercises 105 References 106 Chapter 4 Models for Binary and Count Outcomes Introduction: discrete model likelihoods vs. data augmentation Count data Binomial and binary data Estimation by data augmentation: the Albert Chib method Other augmented data methods Model assessment: outlier detection and model checks Model assessment: predictive model selection and checks Predictor selection in binary and count regression Contingency tables Semi-parametric and general additive models for binomial and count responses Robust and adaptive non-parametric regression Other approaches to non-linearity 144 Exercises 149 References 150

9 CONTENTS Chapter 5 Further Questions in Binomial and Count Regression Generalizing the Poisson and binomial: overdispersion and robustness Continuous mixture models Modified exponential families Discrete mixtures Hurdle and zero-inflated models Modelling the link function Discrete (DPP mixture) Parametric link transformations Beta mixture on cumulative densities Multivariate outcomes The factor analysis approach Specific modelling of marginal and odds ratios Multivariate Poisson data 182 Exercises 189 References 192 vii Chapter 6 Random Effect and Latent Variable Models for Multicategory Outcomes Multicategory data: level of observation and relations between categories Multinomial models for individual data: modelling choices Multinomial models for aggregated data: modelling contingency tables Conditional contingency tables: histogram smoothing and multinomial logit via Poisson regression The multinomial probit Non-linear predictor effects Heterogeneity via the mixed logit Aggregate multicategory data: the multinomial Dirichlet model and extensions Multinomial extra variation Uncertainty in Dirichlet parameters Latent class analysis 226 Exercises 230 References 231 Chapter 7 Ordinal Regression Aspects and assumptions of ordinal data models Latent scale and data augmentation 236

10 viii CONTENTS 7.3 Assessing model assumptions: non-parametric ordinal regression and assessing ordinality Location-scale ordinal regression Structural interpretations with aggregated ordinal data Log-linear models for contingency tables with ordered categories Multivariate ordered outcomes 258 Exercises 263 References 264 Chapter 8 Discrete Spatial Data Introduction Univariate responses: the mixed ICAR model and extensions Spatial robustness Multivariate spatial priors Varying predictor effect models 279 Exercises 285 References 286 Chapter 9 Time Series Models for Discrete Variables Introduction: time dependence in observations and latent data Observation-driven dependence Binary data and Markov Chain models Observation-driven models for individual multicategory data Time series of aggregate multinomial data Parameter-driven dependence via DLMs Measurement error models Conjugate updating Parameter-driven dependence via autocorrelated error models Integer autoregressive models Hidden Markov models 313 Exercises 315 References 317 Chapter 10 Hierarchical and Panel Data Models Introduction: clustered data and general linear mixed models Hierarchical models for metric outcomes 322

11 CONTENTS 10.3 Hierarchical generalized linear models Augmented data sampling for hierarchical GLMs Random effects for crossed factors The general linear mixed model for panel data Time dependence Longitudinal categorical data Conjugate panel models Growth curve analysis Multivariate panel data Robustness in panel and clustered data analysis APC and spatio-temporal models Space time and spatial APC models 364 Exercises 371 References 374 ix Chapter 11 Missing-Data Models Introduction: types of missing data Density mechanisms for missing data Auxiliary variables Predictors with missing values Multiple imputation Several responses with missing values Non-ignorable non-response models for survey tabulations The differential non-response model Alternative parameterizations of the differential non-response model Ecological inference: imputing missing interior data in R C tables Cross-tabulations with missing data on any factor Recent developments 409 Exercises 410 References 412 Index 415

13 Preface This book continues the themes in my two earlier Wiley books in seeking to make modern Bayesian methods accessible via a practically oriented exposition, with statistical computing and applied data analysis at the forefront. As before, I have focused on the WINBUGS package, which has now reached a wide degree of acceptance in application to the ever expanding corpus of methodological work and applications based on Monte Carlo Markov Chain techniques. I hope that the applied focus will help students and researchers alike, including those with primary disciplinary interests outside statistics (e.g. psychology, geography and epidemiology). Nevertheless a wide range of simple or more advanced modelling techniques are discussed and I hope the approach is also helpful for courses in Bayesian data analysis and statistical computing. I have sought to review recent Bayesian methodology for categorical outcomes (binary, count and multinomial data) but my take on this will obviously emphasize some themes more than others: particular aspects that I have focused on include non-parametric and non-linear regression models, model choice, time series and spatio-temporal models. Missing data models are also considered. As with my earlier Wiley books, a set of worked examples with documented code forms part of the book s presentation and can be downloaded from the publisher s website ftp:// the WINBUGS site or STATLIB. Peter Congdon

15 CHAPTER 1 Principles of Bayesian Inference 1.1 BAYESIAN UPDATING Bayesian inference differs from classical inference in treating parameters as random variables and using the data to update prior knowledge about parameters and functionals of those parameters. We are also likely to need model predictions and these are provided as part of the updating process. Prior knowledge about parameters and updated (or posterior) knowledge about them, as well as implications for functionals and predictions, are expressed in terms of densities. One of the benefits of modern Monte Carlo Markov Chain (MCMC) sampling methods (e.g. Chib and Greenberg, 1995; Tierney, 1994; Gelfand and Smith, 1990; Gilks et al., 1996a; Smith and Roberts, 1993) is the ease with which full marginal densities of parameters may be obtained. In a regression model the parameters would be regression coefficients and possible variance parameters, and functionals of parameters might include elasticities (in econometrics) or effective dose (in biometrics). The new Bayesian sampling-based estimation techniques obtain samples from the posterior density, either of parameters themselves, or functionals of parameters. They improve considerably on multiple integration or analytical approximation methods that are infeasible with large numbers of parameters. Nevertheless many issues remain in the application of sampling-based techniques, such as obtaining convergence, and choice of efficient sampling method. There are also more general Bayesian Models for Categorical Data # 2005 John Wiley & Sons, Ltd P. Congdon

Field Geophysics THIRD EDITION. John Milsom University College London

Field Geophysics THIRD EDITION John Milsom University College London Field Geophysics The Geological Field Guide Series Basic Geological Mapping, Third edition John Barnes The Field Description of Metamorphic