November 2002 STA Random Effects Selection in Linear Mixed Models
|
|
- Leonard Goodwin
- 6 years ago
- Views:
Transcription
1 November 2002 STA216 1 Random Effects Selection in Linear Mixed Models
2 November 2002 STA216 2 Introduction It is common practice in many applications to collect multiple measurements on a subject. Linear mixed models (Laird and Ware, 1982; Longford, 1993) attempt to account for within-subject dependency in the multiple measurements by including one or more subject-specific latent variables (i.e., random effects) in the regression model. An important practical problem in applying linear mixed models is how to choose the random effects component. Use AIC or BIC? Likelihood ratio test? Score test?
3 November 2002 STA216 3 Bayesian Hierarchical Approach We propose an approach for selecting random effects using a hierarchical Bayesian model. A key step: D = ΛΓΓ T Λ, (1) We allow elements of Λ to have positive probability of being zero so that random effects can have zero variances, effectively dropping out of the model. Conditionally, the parameters in either Λ or Γ can be regarded as regression coefficients in a normal linear model.
4 November 2002 STA216 4 Linear Mixed Models n subjects, with subject i contributing n i observations For subject i at observation j, let y ij denote a response variable, let x ij denote a p 1 vector of predictors, and let z ij denote a q 1 vector of predictors. In general, the linear mixed effects model is written as y i = X i α + Z i β i + ε i, (2) where y i = (y i1,..., y ini ) T, X i = (x T i1,..., xt in i ) T, Z i = (z T i1,..., zt in i ) T, α is a p 1 vector of unknown population parameters, β i is a q 1 vector of unknown subject-specific random effects with β i N(0, D), and the elements of the residual vector, ε i, are N(0, σ 2 I). Integrating out the random effects β i, the marginal distribution of y i is
5 November 2002 STA216 5 N(X i α, Z i DZ T i ). Heterogeneity among subjects is accommodated by allowing the linear predictor conditional on the covariates to vary. When z ij is a subvector of x ij, the model allows the regression coefficients for the covariates included in z ij to vary among subjects, while assuming that the remaining coefficients are fixed for all subjects. In Bayesian estimation of mixed models: inverse-wishart prior for D. The inverse-wishart density tends to be restrictive, however, since it prescribes a common degrees of freedom for all the diagonal entries of D. In addition, it is only useful if the random effects component is known, since it restricts all random effect variances to be positive.
6 November 2002 STA216 6 Reparameterization Starting with the model that has a random coefficient for each of the elements of z ij, we adaptively select models having some random effects excluded. From model (2), it is clear that selecting a subset of random effects is equivalent to setting to 0 the variances of the nonselected random effects. Let d lm denote the (l, m)th entry of D, for l, m = 1,..., q. The lth random effect β il is excluded if d ll = 0 and is included if d ll > 0. Let L be the lower triangular Cholesky decomposition of D. We assume that L has nonnegative diagonal elements so that it is unique (Seber, 1977, p388). Given L, the linear mixed model (2) can be reexpressed as y i = X i α + Z i Lb i + ε i,
7 November 2002 STA216 7 where b i = (b i1,..., b iq ) T is a vector of independent standard normal latent variables. We further let L = ΛΓ, where Λ = diag(λ 1,..., λ q ) and Γ is a q q matrix with the (l, m)th element denoted by γ lm. As minimal conditions on Λ and Γ so that they are uniquely defined, we assume that λ l 0, γ ll = 1, and γ lm = 0 for l = 1,..., q, m = l + 1,..., q. (3) Specifically, we choose Λ to be a nonnegative q q diagonal matrix, and Γ to be a lower triangular matrix with 1 s in the diagonal entries. This leads to the decomposition of D in (1), and to the reparameterized linear mixed model, y i = X i α + Z i ΛΓb i + ε i. (4)
8 November 2002 STA216 8 Implications of the Reparameterization Following straightforward matrix algebra, the diagonal elements of D are d ll = λ 2 l ( 1 + l 1 r=1 γ 2 lr ) The off-diagonal elements are d lm = d ml = λ l λ m (γ ml + for l = 1,..., q, (5) l 1 r=1 γ lr γ mr ) for l = 1,..., q; m = l + 1,..., q. In the case where λ l = 0, var(β il ) = 0 and the lth random effect, β il, is effectively dropped. The parameters γ R q(q 1)/2 measure the degree of within-subject dependency in the random-effects, β i, as is clear from the expression for the correlation coefficient
9 November 2002 STA216 9 between β il and β im, for l m, ρ(β im, β il ) = γ ml + l 1 r=1 γ lrγ mr ( 1 + )( l 1 r=1 γ2 lr 1 + ), m 1 r=1 γ2 mr which does not depend on λ. As functions of elements of the covariance matrix D, λ and γ are not independent. In particular, if λ l = 0, γ ml = γ lm = 0 for all m {l + 1,..., q} and m {1,..., l 1}. For later use, we define { R λ = γ : γ ml = γ lm = 0 if λ l = 0, } l = 1,..., q, m = l + 1,..., q, m = 1,..., l 1. (6)
10 November 2002 STA Prior Specification Our model is completed with a prior density for θ = (α, λ, γ, σ 2 ) T. First, we assume p(θ) = p(λ, γ)p(α)p(σ 2 ), Following standard convention, we choose conjugate priors, with N(α 0, A 0 ) for α and G(c 0, d 0 ) (σ 2 ) c 0 1 exp{ d 0 σ 2 } for σ 2. In choosing priors for Λ and Γ, and hence for D, we wish to allocate positive probability to zero values for the random effects variances. In addition, motivated by practical considerations, we want to choose priors that facilitate posterior computation. For this reason, prior distributions that are conditionally conjugate are desirable. We assume that p(λ, γ) = p(γ λ)p(λ) N(γ; γ 0, R 0 )1(γ R λ )p(λ),
11 November 2002 STA We further assume that the λ s are independent so that p(λ) = q l=1 p(λ l). Let ZI-N + (π, µ, σ 2 ) denote the density of a zero inflated half normal distribution consisting of a point mass at zero (with probability π) and a N(µ, σ 2 ) density truncated below by zero. To specify a model selection prior, we choose p(λ l ) = d ZI-N + (p l0, m l0, s 2 l0 ) for each l, where p l0, m l0, and s 2 l0 are hyperparameters to be specified by the investigators. The prior probability that the lth random effect is excluded (i.e., its variance is zero) is p l0, and the overall prior probability of excluding all the random effects is q l=1 p l0.
12 November 2002 STA Posterior Computation Letting b = (b 1,..., b n ) T and y = (y 1,..., y n ) T, the likelihood is given by ( n exp σ 2 i=1 n i j=1 l(θ, b; y) = (2πσ 2 ) n i=1 n i/2 ) (y ij x T ijα z T ijλγb i ) 2 /2. The posterior distribution is obtained by combining priors and the likelihood in the usual way. However, directly evaluation of the posterior distribution seems to be difficult. Instead we employ a Gibbs sampler (Gelfand and Smith, 1990) which works by alternately sampling from the full conditional distributions of the parameters (α, σ 2, λ, γ) and latent variables b. Bayesian linear model theory (Lindley and Smith, 1972) applies when deriving the full
13 November 2002 STA conditional distributions of α, σ 2, and b p(α λ, γ, σ 2, b, y) = d N( α, Â), with ( Â = σ 2 n ) 1 ni i=1 j=1 x ijx T ij + A 1 0 and { α = Â σ 2 n ni i=1 j=1 x ij(y ij z T ij ΛΓb i) + } A 1 0 α 0. For σ 2, the full conditional distribution is given by p(σ 2 α, λ, γ, b, y) = d G(ĉ, d) where ĉ = c 0 + n i=1 n i/2 and d = d 0 + n ni i=1 j=1 (y ij x T ij α zt ij ΛΓb i) 2 /2. Similar to α, the full conditional distribution of the latent normal variable b is n p(b λ, γ, σ 2, α, y) = p(b i λ, γ, σ 2, α, y i ), i=1 with p(b i λ, γ, σ 2, α, y i ) = d N(ĥi, ( Ĥi), where Ĥi = σ 2 ) n 1, i j=1 v ijvij T + I ĥ i = σ 2 Ĥ i ni j=1 v ij(y ij x T ij α), and v ij = z T ij ΛΓb i.
14 November 2002 STA FCDs of λ and γ The full conditional distributions of λ and γ seem to be complex, given the likelihood form in (7). However, upon rewriting expression (4) with constraint (3) as q q ) y ij = x T ijα + b il (λ l z ijl + λ m z ijm γ ml + ε ij, l=1 m=l+1 we obtain two equations that characterize λ and γ as regression coefficients in a normal linear model. First define the q(q 1)/2 1 vector ( T u ij = b il λ m z ijm : l = 1,..., q, m = l + 1,..., q). Then expression (7) implies y ij x T ijα = u T ijγ + ε ij. Since the error term is normally distributed and γ has a multivariate normal prior
15 November 2002 STA distribution after setting elements equal to zero to ensure that γ R λ, the full conditional distribution for γ is easy to derive. The full conditional distribution of γ is given by p(γ α, λ, b, σ 2, y) N( γ, R)1(γ R λ ), where R ( = σ 2 n ) 1 ni i=1 j=1 u iju T ij + R 1 0 and ( γ = R σ 2 n i=1 ) ni j=1 u ij(y ij x T ijα) + R 1 0 γ 0. Similarly, on defining the q 1 vector t ij = ( z ijl (b il + l 1 m=1 b im γ ml ) : l = 1,..., q) T, it is easy to verify that (7) implies y ij x T ijα = t T ijλ + ε ij. Letting η ijl = y ij x T ij α m l t ijmλ m for each λ l, we have η ijl = t ijl λ l + ε ij. It follows from straightforward (but lengthy) algebra
16 November 2002 STA that the full conditional distribution of λ l is p(λ l λ (l), α, β, γ, σ 2, y) d = ZI-N + ( p l, λ l, σ 2 l ), (8) where p l = P (λ l = 0 λ (l), α, β, γ, σ 2, y) is the conditional posterior probability that λ l = 0, and λ l and σ l 2 are the updated mean and variance in the normal component of the ZI-N + density. To derive the expressions for p l, λ l and σ l 2, first let ω 2 l = n ni i=1 j=1 t2 ijl /σ2, and let λ l be the maximum likelihood estimate of λ l so that λ l = n i=1 Then, λ l = σ 2 (ω 2 l σ 2 l = (ω 2 l ni j=1 t ijlη ijl / n i=1 λ l + s 2 l0 m l0) and + s 2 l0 ) 1. Define ni j=1 t2 ijl.
17 November 2002 STA a = exp{ n i=1 ni j=1 η2 ijl /2σ2 } and b = σ l 1 Φ( m l0 /s l0 ) s l0 1 Φ( λ l / σ l ) { } n n i exp (η ijl λ l t ijl ) 2 /2σ 2 exp { i=1 j=1 ( λ 2 l /2ω 2 l + m 2 l0/2s 2 l0 λ 2 /2 σ 2 l ) }. Then, p l = p l0 a p l0 a + (1 p l0 )b. Distribution (8) is conditionally conjugate, following the same form as the prior for λ l. Sampling from expression (8) can be implemented by (i) sampling δ l from Bernoulli( p l ); and (ii) setting λ l = 0 if δ l = 1 and otherwise sampling λ l from N( λ l, σ 2 l ) truncated below by zero. Given repeated samples from the posterior distribution, inference about the model
18 November 2002 STA parameters α, γ, λ, and σ 2 proceeds as usual. In particular, one can report posterior means, posterior standard deviations, and highest posterior density (HPD) intervals. To compute the posterior probabilities of each of the 2 q models, we simply add up the number of occurrences of each model and divide by the number of iterations. The prior and posterior probabilities can then be used to calculate Bayes factors for comparing individual models. Refer to Kass and Raftery (1995) for a review of the Bayes factor.
Bayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationBayesian Inference. Chapter 9. Linear models and regression
Bayesian Inference Chapter 9. Linear models and regression M. Concepcion Ausin Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master in Mathematical Engineering
More informationDefault Priors and Effcient Posterior Computation in Bayesian
Default Priors and Effcient Posterior Computation in Bayesian Factor Analysis January 16, 2010 Presented by Eric Wang, Duke University Background and Motivation A Brief Review of Parameter Expansion Literature
More informationLatent Variable Models for Binary Data. Suppose that for a given vector of explanatory variables x, the latent
Latent Variable Models for Binary Data Suppose that for a given vector of explanatory variables x, the latent variable, U, has a continuous cumulative distribution function F (u; x) and that the binary
More informationBayesian Hypothesis Testing in GLMs: One-Sided and Ordered Alternatives. 1(w i = h + 1)β h + ɛ i,
Bayesian Hypothesis Testing in GLMs: One-Sided and Ordered Alternatives Often interest may focus on comparing a null hypothesis of no difference between groups to an ordered restricted alternative. For
More informationGibbs Sampling in Linear Models #2
Gibbs Sampling in Linear Models #2 Econ 690 Purdue University Outline 1 Linear Regression Model with a Changepoint Example with Temperature Data 2 The Seemingly Unrelated Regressions Model 3 Gibbs sampling
More informationBayesian Linear Regression
Bayesian Linear Regression Sudipto Banerjee 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. September 15, 2010 1 Linear regression models: a Bayesian perspective
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationFixed and random effects selection in linear and logistic models
Fixed and random effects selection in linear and logistic models Satkartar K. Kinney Institute of Statistics and Decision Sciences, Duke University, Box 9051, Durham, North Carolina 7705, U.S.A. email:
More informationBayesian Inference. Chapter 4: Regression and Hierarchical Models
Bayesian Inference Chapter 4: Regression and Hierarchical Models Conchi Ausín and Mike Wiper Department of Statistics Universidad Carlos III de Madrid Advanced Statistics and Data Mining Summer School
More informationLecture 16: Mixtures of Generalized Linear Models
Lecture 16: Mixtures of Generalized Linear Models October 26, 2006 Setting Outline Often, a single GLM may be insufficiently flexible to characterize the data Setting Often, a single GLM may be insufficiently
More informationMotivation Scale Mixutres of Normals Finite Gaussian Mixtures Skew-Normal Models. Mixture Models. Econ 690. Purdue University
Econ 690 Purdue University In virtually all of the previous lectures, our models have made use of normality assumptions. From a computational point of view, the reason for this assumption is clear: combined
More informationA Fully Nonparametric Modeling Approach to. BNP Binary Regression
A Fully Nonparametric Modeling Approach to Binary Regression Maria Department of Applied Mathematics and Statistics University of California, Santa Cruz SBIES, April 27-28, 2012 Outline 1 2 3 Simulation
More informationGraphical Models for Collaborative Filtering
Graphical Models for Collaborative Filtering Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Sequence modeling HMM, Kalman Filter, etc.: Similarity: the same graphical model topology,
More informationBayesian Inference. Chapter 4: Regression and Hierarchical Models
Bayesian Inference Chapter 4: Regression and Hierarchical Models Conchi Ausín and Mike Wiper Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative
More informationSTA 216, GLM, Lecture 16. October 29, 2007
STA 216, GLM, Lecture 16 October 29, 2007 Efficient Posterior Computation in Factor Models Underlying Normal Models Generalized Latent Trait Models Formulation Genetic Epidemiology Illustration Structural
More informationvariability of the model, represented by σ 2 and not accounted for by Xβ
Posterior Predictive Distribution Suppose we have observed a new set of explanatory variables X and we want to predict the outcomes ỹ using the regression model. Components of uncertainty in p(ỹ y) variability
More informationThe linear model is the most fundamental of all serious statistical models encompassing:
Linear Regression Models: A Bayesian perspective Ingredients of a linear model include an n 1 response vector y = (y 1,..., y n ) T and an n p design matrix (e.g. including regressors) X = [x 1,..., x
More informationFixed and Random Effects Selection in Linear and Logistic Models
Biometrics 63, 690 698 September 2007 DOI: 10.1111/j.1541-0420.2007.00771.x Fixed and Random Effects Selection in Linear and Logistic Models Satkartar K. Kinney Institute of Statistics and Decision Sciences,
More informationMULTILEVEL IMPUTATION 1
MULTILEVEL IMPUTATION 1 Supplement B: MCMC Sampling Steps and Distributions for Two-Level Imputation This document gives technical details of the full conditional distributions used to draw regression
More informationPMR Learning as Inference
Outline PMR Learning as Inference Probabilistic Modelling and Reasoning Amos Storkey Modelling 2 The Exponential Family 3 Bayesian Sets School of Informatics, University of Edinburgh Amos Storkey PMR Learning
More informationMixed models in R using the lme4 package Part 4: Theory of linear mixed models
Mixed models in R using the lme4 package Part 4: Theory of linear mixed models Douglas Bates 8 th International Amsterdam Conference on Multilevel Analysis 2011-03-16 Douglas Bates
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationGibbs Sampling in Endogenous Variables Models
Gibbs Sampling in Endogenous Variables Models Econ 690 Purdue University Outline 1 Motivation 2 Identification Issues 3 Posterior Simulation #1 4 Posterior Simulation #2 Motivation In this lecture we take
More informationThe joint posterior distribution of the unknown parameters and hidden variables, given the
DERIVATIONS OF THE FULLY CONDITIONAL POSTERIOR DENSITIES The joint posterior distribution of the unknown parameters and hidden variables, given the data, is proportional to the product of the joint prior
More informationST 740: Linear Models and Multivariate Normal Inference
ST 740: Linear Models and Multivariate Normal Inference Alyson Wilson Department of Statistics North Carolina State University November 4, 2013 A. Wilson (NCSU STAT) Linear Models November 4, 2013 1 /
More informationBayesian non-parametric model to longitudinally predict churn
Bayesian non-parametric model to longitudinally predict churn Bruno Scarpa Università di Padova Conference of European Statistics Stakeholders Methodologists, Producers and Users of European Statistics
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Department of Forestry & Department of Geography, Michigan State University, Lansing Michigan, U.S.A. 2 Biostatistics, School of Public
More informationPart 8: GLMs and Hierarchical LMs and GLMs
Part 8: GLMs and Hierarchical LMs and GLMs 1 Example: Song sparrow reproductive success Arcese et al., (1992) provide data on a sample from a population of 52 female song sparrows studied over the course
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee September 03 05, 2017 Department of Biostatistics, Fielding School of Public Health, University of California, Los Angeles Linear Regression Linear regression is,
More informationHierarchical Linear Models. Hierarchical Linear Models. Much of this material already seen in Chapters 5 and 14. Hyperprior on K parameters α:
Hierarchical Linear Models Hierarchical Linear Models Much of this material already seen in Chapters 5 and 14 Hierarchical linear models combine regression framework with hierarchical framework Unified
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department
More informationBayes methods for categorical data. April 25, 2017
Bayes methods for categorical data April 25, 2017 Motivation for joint probability models Increasing interest in high-dimensional data in broad applications Focus may be on prediction, variable selection,
More informationg-priors for Linear Regression
Stat60: Bayesian Modeling and Inference Lecture Date: March 15, 010 g-priors for Linear Regression Lecturer: Michael I. Jordan Scribe: Andrew H. Chan 1 Linear regression and g-priors In the last lecture,
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationCross-sectional space-time modeling using ARNN(p, n) processes
Cross-sectional space-time modeling using ARNN(p, n) processes W. Polasek K. Kakamu September, 006 Abstract We suggest a new class of cross-sectional space-time models based on local AR models and nearest
More informationSupplementary Material for Analysis of Job Satisfaction: The Case of Japanese Private Companies
Supplementary Material for Analysis of Job Satisfaction: The Case of Japanese Private Companies S1. Sampling Algorithms We assume that z i NX i β, Σ), i =1,,n, 1) where Σ is an m m positive definite covariance
More informationSparse Linear Models (10/7/13)
STA56: Probabilistic machine learning Sparse Linear Models (0/7/) Lecturer: Barbara Engelhardt Scribes: Jiaji Huang, Xin Jiang, Albert Oh Sparsity Sparsity has been a hot topic in statistics and machine
More informationBayesian shrinkage approach in variable selection for mixed
Bayesian shrinkage approach in variable selection for mixed effects s GGI Statistics Conference, Florence, 2015 Bayesian Variable Selection June 22-26, 2015 Outline 1 Introduction 2 3 4 Outline Introduction
More informationAn Introduction to Bayesian Linear Regression
An Introduction to Bayesian Linear Regression APPM 5720: Bayesian Computation Fall 2018 A SIMPLE LINEAR MODEL Suppose that we observe explanatory variables x 1, x 2,..., x n and dependent variables y 1,
More informationBayesian Graphical Models for Structural Vector AutoregressiveMarch Processes 21, / 1
Bayesian Graphical Models for Structural Vector Autoregressive Processes Daniel Ahelegbey, Monica Billio, and Roberto Cassin (2014) March 21, 2015 Bayesian Graphical Models for Structural Vector AutoregressiveMarch
More informationAccounting for Complex Sample Designs via Mixture Models
Accounting for Complex Sample Designs via Finite Normal Mixture Models 1 1 University of Michigan School of Public Health August 2009 Talk Outline 1 2 Accommodating Sampling Weights in Mixture Models 3
More informationMarkov Chain Monte Carlo methods
Markov Chain Monte Carlo methods By Oleg Makhnin 1 Introduction a b c M = d e f g h i 0 f(x)dx 1.1 Motivation 1.1.1 Just here Supresses numbering 1.1.2 After this 1.2 Literature 2 Method 2.1 New math As
More information1 Data Arrays and Decompositions
1 Data Arrays and Decompositions 1.1 Variance Matrices and Eigenstructure Consider a p p positive definite and symmetric matrix V - a model parameter or a sample variance matrix. The eigenstructure is
More informationSparse Factor-Analytic Probit Models
Sparse Factor-Analytic Probit Models By JAMES G. SCOTT Department of Statistical Science, Duke University, Durham, North Carolina 27708-0251, U.S.A. james@stat.duke.edu PAUL R. HAHN Department of Statistical
More informationLikelihood-Based Methods
Likelihood-Based Methods Handbook of Spatial Statistics, Chapter 4 Susheela Singh September 22, 2016 OVERVIEW INTRODUCTION MAXIMUM LIKELIHOOD ESTIMATION (ML) RESTRICTED MAXIMUM LIKELIHOOD ESTIMATION (REML)
More informationLarge-scale Ordinal Collaborative Filtering
Large-scale Ordinal Collaborative Filtering Ulrich Paquet, Blaise Thomson, and Ole Winther Microsoft Research Cambridge, University of Cambridge, Technical University of Denmark ulripa@microsoft.com,brmt2@cam.ac.uk,owi@imm.dtu.dk
More informationIndex. Pagenumbersfollowedbyf indicate figures; pagenumbersfollowedbyt indicate tables.
Index Pagenumbersfollowedbyf indicate figures; pagenumbersfollowedbyt indicate tables. Adaptive rejection metropolis sampling (ARMS), 98 Adaptive shrinkage, 132 Advanced Photo System (APS), 255 Aggregation
More informationBayesian inference. Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark. April 10, 2017
Bayesian inference Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark April 10, 2017 1 / 22 Outline for today A genetic example Bayes theorem Examples Priors Posterior summaries
More informationNon-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
More informationYounshik Chung and Hyungsoon Kim 968). Sharples(990) showed how variance ination can be incorporated easily into general hierarchical models, retainin
Bayesian Outlier Detection in Regression Model Younshik Chung and Hyungsoon Kim Abstract The problem of 'outliers', observations which look suspicious in some way, has long been one of the most concern
More informationBayesian data analysis in practice: Three simple examples
Bayesian data analysis in practice: Three simple examples Martin P. Tingley Introduction These notes cover three examples I presented at Climatea on 5 October 0. Matlab code is available by request to
More informationKatsuhiro Sugita Faculty of Law and Letters, University of the Ryukyus. Abstract
Bayesian analysis of a vector autoregressive model with multiple structural breaks Katsuhiro Sugita Faculty of Law and Letters, University of the Ryukyus Abstract This paper develops a Bayesian approach
More informationNonparameteric Regression:
Nonparameteric Regression: Nadaraya-Watson Kernel Regression & Gaussian Process Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationHierarchical Linear Models
Hierarchical Linear Models Statistics 220 Spring 2005 Copyright c 2005 by Mark E. Irwin The linear regression model Hierarchical Linear Models y N(Xβ, Σ y ) β σ 2 p(β σ 2 ) σ 2 p(σ 2 ) can be extended
More informationUnivariate Normal Distribution; GLM with the Univariate Normal; Least Squares Estimation
Univariate Normal Distribution; GLM with the Univariate Normal; Least Squares Estimation PRE 905: Multivariate Analysis Spring 2014 Lecture 4 Today s Class The building blocks: The basics of mathematical
More informationThe Wishart distribution Scaled Wishart. Wishart Priors. Patrick Breheny. March 28. Patrick Breheny BST 701: Bayesian Modeling in Biostatistics 1/11
Wishart Priors Patrick Breheny March 28 Patrick Breheny BST 701: Bayesian Modeling in Biostatistics 1/11 Introduction When more than two coefficients vary, it becomes difficult to directly model each element
More informationFactorization of Seperable and Patterned Covariance Matrices for Gibbs Sampling
Monte Carlo Methods Appl, Vol 6, No 3 (2000), pp 205 210 c VSP 2000 Factorization of Seperable and Patterned Covariance Matrices for Gibbs Sampling Daniel B Rowe H & SS, 228-77 California Institute of
More informationVariational Inference (11/04/13)
STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further
More informationPartial factor modeling: predictor-dependent shrinkage for linear regression
modeling: predictor-dependent shrinkage for linear Richard Hahn, Carlos Carvalho and Sayan Mukherjee JASA 2013 Review by Esther Salazar Duke University December, 2013 Factor framework The factor framework
More informationWeb Appendix for Hierarchical Adaptive Regression Kernels for Regression with Functional Predictors by D. B. Woodard, C. Crainiceanu, and D.
Web Appendix for Hierarchical Adaptive Regression Kernels for Regression with Functional Predictors by D. B. Woodard, C. Crainiceanu, and D. Ruppert A. EMPIRICAL ESTIMATE OF THE KERNEL MIXTURE Here we
More informationSteven L. Scott. Presented by Ahmet Engin Ural
Steven L. Scott Presented by Ahmet Engin Ural Overview of HMM Evaluating likelihoods The Likelihood Recursion The Forward-Backward Recursion Sampling HMM DG and FB samplers Autocovariance of samplers Some
More informationLinear Models A linear model is defined by the expression
Linear Models A linear model is defined by the expression x = F β + ɛ. where x = (x 1, x 2,..., x n ) is vector of size n usually known as the response vector. β = (β 1, β 2,..., β p ) is the transpose
More informationA Bayesian Treatment of Linear Gaussian Regression
A Bayesian Treatment of Linear Gaussian Regression Frank Wood December 3, 2009 Bayesian Approach to Classical Linear Regression In classical linear regression we have the following model y β, σ 2, X N(Xβ,
More informationAn exploration of fixed and random effects selection for longitudinal binary outcomes in the presence of non-ignorable dropout
Biometrical Journal 0 (2011) 0, zzz zzz / DOI: 10.1002/ An exploration of fixed and random effects selection for longitudinal binary outcomes in the presence of non-ignorable dropout Ning Li,1, Michael
More informationSTA 216: GENERALIZED LINEAR MODELS. Lecture 1. Review and Introduction. Much of statistics is based on the assumption that random
STA 216: GENERALIZED LINEAR MODELS Lecture 1. Review and Introduction Much of statistics is based on the assumption that random variables are continuous & normally distributed. Normal linear regression
More informationIntroduction into Bayesian statistics
Introduction into Bayesian statistics Maxim Kochurov EF MSU November 15, 2016 Maxim Kochurov Introduction into Bayesian statistics EF MSU 1 / 7 Content 1 Framework Notations 2 Difference Bayesians vs Frequentists
More informationPrinciples of Bayesian Inference
Principles of Bayesian Inference Sudipto Banerjee University of Minnesota July 20th, 2008 1 Bayesian Principles Classical statistics: model parameters are fixed and unknown. A Bayesian thinks of parameters
More informationGibbs Sampling in Linear Models #1
Gibbs Sampling in Linear Models #1 Econ 690 Purdue University Justin L Tobias Gibbs Sampling #1 Outline 1 Conditional Posterior Distributions for Regression Parameters in the Linear Model [Lindley and
More informationAn Extended BIC for Model Selection
An Extended BIC for Model Selection at the JSM meeting 2007 - Salt Lake City Surajit Ray Boston University (Dept of Mathematics and Statistics) Joint work with James Berger, Duke University; Susie Bayarri,
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationStat260: Bayesian Modeling and Inference Lecture Date: February 10th, Jeffreys priors. exp 1 ) p 2
Stat260: Bayesian Modeling and Inference Lecture Date: February 10th, 2010 Jeffreys priors Lecturer: Michael I. Jordan Scribe: Timothy Hunter 1 Priors for the multivariate Gaussian Consider a multivariate
More informationLecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Jonathan Marchini (University of Oxford) BS2a MT / 15
Lecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Jonathan Marchini (University of Oxford) BS2a MT 2013 1 / 15 Contingency table analysis North Carolina State University
More informationDefault Bayesian Model Determination Methods for Generalised Linear Mixed Models
Default Bayesian Model Determination Methods for Generalised Linear Mixed Models Antony M. Overstall 1, Jonathan J. Forster 2 Abstract A default strategy for fully Bayesian model determination for GLMMs
More informationWill Penny. SPM for MEG/EEG, 15th May 2012
SPM for MEG/EEG, 15th May 2012 A prior distribution over model space p(m) (or hypothesis space ) can be updated to a posterior distribution after observing data y. This is implemented using Bayes rule
More informationBayesian inference on dependence in multivariate longitudinal data
Biometrics 000, 000 000 DOI: 000 000 0000 Bayesian inference on dependence in multivariate longitudinal data Hongxia Yang a, Fan Li a, Enrique Schisterman b and David Dunson a a Department of Statistical
More informationWill Penny. DCM short course, Paris 2012
DCM short course, Paris 2012 Ten Simple Rules Stephan et al. Neuroimage, 2010 Model Structure Bayes rule for models A prior distribution over model space p(m) (or hypothesis space ) can be updated to a
More informationDynamic System Identification using HDMR-Bayesian Technique
Dynamic System Identification using HDMR-Bayesian Technique *Shereena O A 1) and Dr. B N Rao 2) 1), 2) Department of Civil Engineering, IIT Madras, Chennai 600036, Tamil Nadu, India 1) ce14d020@smail.iitm.ac.in
More informationChapter 4: Factor Analysis
Chapter 4: Factor Analysis In many studies, we may not be able to measure directly the variables of interest. We can merely collect data on other variables which may be related to the variables of interest.
More informationSTAT Advanced Bayesian Inference
1 / 32 STAT 625 - Advanced Bayesian Inference Meng Li Department of Statistics Jan 23, 218 The Dirichlet distribution 2 / 32 θ Dirichlet(a 1,...,a k ) with density p(θ 1,θ 2,...,θ k ) = k j=1 Γ(a j) Γ(
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More informationClassical and Bayesian inference
Classical and Bayesian inference AMS 132 January 18, 2018 Claudia Wehrhahn (UCSC) Classical and Bayesian inference January 18, 2018 1 / 9 Sampling from a Bernoulli Distribution Theorem (Beta-Bernoulli
More information2 Bayesian Hierarchical Response Modeling
2 Bayesian Hierarchical Response Modeling In the first chapter, an introduction to Bayesian item response modeling was given. The Bayesian methodology requires careful specification of priors since item
More informationLecture 5: Spatial probit models. James P. LeSage University of Toledo Department of Economics Toledo, OH
Lecture 5: Spatial probit models James P. LeSage University of Toledo Department of Economics Toledo, OH 43606 jlesage@spatial-econometrics.com March 2004 1 A Bayesian spatial probit model with individual
More informationMultivariate Normal & Wishart
Multivariate Normal & Wishart Hoff Chapter 7 October 21, 2010 Reading Comprehesion Example Twenty-two children are given a reading comprehsion test before and after receiving a particular instruction method.
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationStat260: Bayesian Modeling and Inference Lecture Date: March 10, 2010
Stat60: Bayesian Modelin and Inference Lecture Date: March 10, 010 Bayes Factors, -priors, and Model Selection for Reression Lecturer: Michael I. Jordan Scribe: Tamara Broderick The readin for this lecture
More informationBayesian inference for sample surveys. Roderick Little Module 2: Bayesian models for simple random samples
Bayesian inference for sample surveys Roderick Little Module : Bayesian models for simple random samples Superpopulation Modeling: Estimating parameters Various principles: least squares, method of moments,
More informationBayesian Inference for the Multivariate Normal
Bayesian Inference for the Multivariate Normal Will Penny Wellcome Trust Centre for Neuroimaging, University College, London WC1N 3BG, UK. November 28, 2014 Abstract Bayesian inference for the multivariate
More informationComputational methods for mixed models
Computational methods for mixed models Douglas Bates Department of Statistics University of Wisconsin Madison March 27, 2018 Abstract The lme4 package provides R functions to fit and analyze several different
More informationSTAT 425: Introduction to Bayesian Analysis
STAT 425: Introduction to Bayesian Analysis Marina Vannucci Rice University, USA Fall 2017 Marina Vannucci (Rice University, USA) Bayesian Analysis (Part 3) Fall 2017 1 / 40 Part 3: Hierarchical and Linear
More informationGeneralized Linear Models. Kurt Hornik
Generalized Linear Models Kurt Hornik Motivation Assuming normality, the linear model y = Xβ + e has y = β + ε, ε N(0, σ 2 ) such that y N(μ, σ 2 ), E(y ) = μ = β. Various generalizations, including general
More informationMISCELLANEOUS TOPICS RELATED TO LIKELIHOOD. Copyright c 2012 (Iowa State University) Statistics / 30
MISCELLANEOUS TOPICS RELATED TO LIKELIHOOD Copyright c 2012 (Iowa State University) Statistics 511 1 / 30 INFORMATION CRITERIA Akaike s Information criterion is given by AIC = 2l(ˆθ) + 2k, where l(ˆθ)
More informationVector Autoregressive Model. Vector Autoregressions II. Estimation of Vector Autoregressions II. Estimation of Vector Autoregressions I.
Vector Autoregressive Model Vector Autoregressions II Empirical Macroeconomics - Lect 2 Dr. Ana Beatriz Galvao Queen Mary University of London January 2012 A VAR(p) model of the m 1 vector of time series
More informationLabor-Supply Shifts and Economic Fluctuations. Technical Appendix
Labor-Supply Shifts and Economic Fluctuations Technical Appendix Yongsung Chang Department of Economics University of Pennsylvania Frank Schorfheide Department of Economics University of Pennsylvania January
More informationGibbs Sampling in Latent Variable Models #1
Gibbs Sampling in Latent Variable Models #1 Econ 690 Purdue University Outline 1 Data augmentation 2 Probit Model Probit Application A Panel Probit Panel Probit 3 The Tobit Model Example: Female Labor
More informationMath 423/533: The Main Theoretical Topics
Math 423/533: The Main Theoretical Topics Notation sample size n, data index i number of predictors, p (p = 2 for simple linear regression) y i : response for individual i x i = (x i1,..., x ip ) (1 p)
More informationThe Normal Linear Regression Model with Natural Conjugate Prior. March 7, 2016
The Normal Linear Regression Model with Natural Conjugate Prior March 7, 2016 The Normal Linear Regression Model with Natural Conjugate Prior The plan Estimate simple regression model using Bayesian methods
More informationBayesian Regressions in Experimental Design
Bayesian Regressions in Experimental Design Stanley Sawyer Washington University Vs. April 9, 8. Introduction. The purpose here is to derive a formula for the posterior probabilities given observed data
More informationA Derivation of the EM Updates for Finding the Maximum Likelihood Parameter Estimates of the Student s t Distribution
A Derivation of the EM Updates for Finding the Maximum Likelihood Parameter Estimates of the Student s t Distribution Carl Scheffler First draft: September 008 Contents The Student s t Distribution The
More information