Stat260: Bayesian Modeling and Inference Lecture Date: March 10, 2010
|
|
- Monica Rose Lawson
- 5 years ago
- Views:
Transcription
1 Stat60: Bayesian Modelin and Inference Lecture Date: March 10, 010 Bayes Factors, -priors, and Model Selection for Reression Lecturer: Michael I. Jordan Scribe: Tamara Broderick The readin for this lecture is Lian et al. (008). 1 Bayes Factors We bein by clarifyin a point about Bayes factors from the previous lecture. Consider data x X and parameter θ Θ. Take H 0 : θ Θ 0 and H 1 : θ Θ 1 for some partition {Θ 0, Θ 1 } of Θ. Θ 0 Θ 1 Fiure 1: Cartoon of the partition of Θ space into reions Θ 0 and Θ 1. Then the Bayes factor BF for comparin models H 0 and H 1 is BF = p(x H 0) p(x H 1 ) And we have the followin relation between the posterior odds and prior odds. p(h 0 x) p(h 1 x) = BF p(h 0) p(h 1 ) Θ Note that the probabilities p(h 0 ) and p(h 1 ) are hyperpriors on the models themselves and are not expressible in terms of π(θ H 0 ) or π(θ H 1 ). Indeed, the overall prior on θ is a mixture distribution where p(h 0 ) and p(h 1 ) are the mixture probabilities. π(θ) = p(h 0 )π(θ H 0 ) + p(h 1 )π(θ H 1 ) Once we know BF, we can solve for the posterior probability of H 0. p(h 0 x) 1 p(h 0 x) p(h 0 x) = = BF p(h 0) p(h 1 ) BF p(h0) p(h 1) 1 + BF p(h0) p(h 1) We can see from the last line that an arbitrary constant in the Bayes factor does not factor out when calculatin the posterior of H 0. Rather, an arbitrary constant can be used to obtain an arbitrary posterior. Lettin the Bayes factor constant decrease to zero yields p(h 0 x) = 0 in the limit, and lettin the Bayes factor increase to infinity yields p(h 0 x) = 1 in the limit. 1
2 Bayes Factors, -priors, and Model Selection for Reression y y x x Fiure : Left: In one-dimensional linear reression, sets of points with more scatter alon the onedimensional predictor x are more informative about the slope and intercept. It is easy to rouhly predict these parameters from the fiure. Riht: Data with less scatter in the predictor variable are less informative about the slope and intercept. -priors We consider a reression model y = Xβ + ǫ, ǫ N(0, σ I) (1) with a Jeffreys prior on σ π(σ ) 1 σ () and a prior called a -prior on the vector of reression coefficients β β N(β 0, σ (X T X) 1 ) (3) In Eq. (3), is a scalar that we will determine later, and the σ factor in the covariance is necessary for conjuacy. Generally in reression models, we make the assumption that the desin matrix X is known and fixed, so we are free to use it in our prior. The particular form in which X appears in Eq. (3) represents a kind of dispersion matrix. For instance, recall that for the usual maximum likelihood estimator ˆβ of β, we have Var(ˆβ) = (X T X) 1 {an estimate of σ } Alternatively, consider a principal component analysis on X (and inore the response variable y for the moment). The eienvalues of X T X ive the directions of the new coordinates. Althouh the -prior is not a reference prior, we can further motivate the use of X in Eq. (3) by the same intuitive considerations we used for reference priors, namely placin more prior mass in areas of the parameter space where we expect the data to be less informative about the parameters. In the one-dimensional classical reression case (Fiure ), hih scatter alon the predictor axis corresponds to hih leverae. The data in this case are more informative about the intercept and slope (β) than in the case of less scatter and low leverae.
3 Bayes Factors, -priors, and Model Selection for Reression 3 Posterior With this enerative model (Eqs. 1,, 3) in hand, we can calculate the full posterior. π(β, σ y, X) (σ ) ( n +1) exp{ 1 σ (y X ˆβ) T (y X ˆβ) Recall that the maximum likelihood estimator ˆβ satisfies ˆβ = (X T X) 1 X T y, 1 σ (β ˆβ) T X T X(β ˆβ) 1 σ (β β 0) T X T X(β β 0 )} (4) and note that the first term in the sum is proportional to the classical residual s. s = (y X ˆβ) T (y X ˆβ) From Eq. (4), we obtain a Gaussian posterior on β and an inverse amma posterior on σ. ( ( ) β σ, y, X = d β0 N ˆβ σ ), + 1 (XT X) 1 ( ) σ y, X = d n IG, s + 1 ( + 1) (ˆβ β 0 )X T X(ˆβ β 0 ) (5) (6) Posterior means From Eqs. (5, 6), we can find each parameter s posterior mean. First, E(β y, X) = E ( E(β σ, y, X) y, X ) by the tower rule ( ( ) β0 = E ˆβ y, X) ( ) β0 = ˆβ Lettin increase to infinity, we recover the frequentist estimate of β: E(β y, X) = ˆβ. Moreover, as increases to infinity, we approach a flat prior on β. We have previously seen that such a prior can be useful for estimation but becomes problematic when performin hypothesis testin. Next, usin the fact that the mean of an IG(a, b)-distributed random variable is b/(a 1) for a > 1, we have E(σ y, X) = s (ˆβ β 0 ) T X T X(ˆβ β 0 ) n Lettin yields E(σ y, X) = s /(n ). Compare this result to the classical frequentist unbiased estimator s /(n p) for σ. Here, p is the number of derees of freedom, or equivalently the number of components of the β vector. One miht worry that s /(n ) is overly conservative for p >, but note that the σ -estimate does not provide the error bars on β for, e.., confidence intervals. We will see how to calculate confidence intervals for β next.
4 4 Bayes Factors, -priors, and Model Selection for Reression (1 α/) α/ St 1 n (1 α/) Fiure 3: Illustration of the quantile function St 1 n. Mass α/ of the Student s t-density appears to the riht of St 1 n (1 α/) and is shown in ray. The remainin mass is on the left. Hihest posterior density reion for β i In order to find the HPD reion for β i, we first calculate the posterior of β i with σ interated out. β i y, X d = St(T i, K ii, n), where St represents Student s t-distribution, which has three parameters: a mean, a spread, and a number of derees of freedom. In our case, we further have ( ) β0i T i := ˆβ i ( ) K ii := s + (ˆβ β 0 ) T X T X(ˆβ β 0 ) ω ii n ω ii := (X T X) 1 ii Here, β 0i and ˆβ i are the i th components of β 0 and ˆβ, respectively. It follows that β i T i Kii d = Stn, where St n is the standard Student s t-distribution (zero mean, unit variance) with n derees of freedom. It follows that, if we let St 1 be the quantile function for the standard Student s t-distribution with n derees of freedom (Fiure 3), the HPD interval at level α is T i ± ( K ii St 1 n 1 α ) How to set We ve already seen that lettin is not an option if we want to perform hypothesis testin. Moreover, lettin 0 takes the posterior to the prior distribution, also reducin our ability to perform inference. Some other options for choosin include usin BIC, empirical Bayes, and full Bayes. BIC The Bayesian Information Criterion (BIC) is an approximation to the marinal likelihood obtained by a Laplace expansion. We can use the BIC to determine a value for, but the resultin model is not necessarily consistent (e.. for the model selection in Section 3).
5 Bayes Factors, -priors, and Model Selection for Reression 5 Empirical Bayes Empirical Bayes works by drawin a line in the hierarchical description of our model. y β, σ β σ, σ Then we take a Bayesian approach to estimation above the line and a frequentist approach below the line. While a full frequentist would consider a profile likelihood for, empirical Bayes methodoloy prescribes interatin out other parameters and maximizin the marinal likelihood. Thus, the empirical Bayes estimate for is ĝ EB : ĝ EB = armaxp(y X, ) Inference under the prior with set to ĝ EB is, in eneral, consistent. We can calculate the marinal likelihood for our case as follows. Interatin out β and σ yields p(y X, ) = Γ ( ) n 1 π n+1 n 1 p n y (1 + ) 1/ ȳ (n 1) (1 + (1 R )) n 1 In the formula above, ȳ is the rand mean, is a vector norm, and R is the usual coefficient of determination. R = 1 (y X ˆβ) T (y X ˆβ) (y ȳ) T (y ȳ) Full Bayes A full Bayesian puts a prior distribution on. Not only is a hierarchy of this type consistent, but the converence will have the best rate (of the three methods considered here) in the frequentist sense. 3 Model Selection for Reression Let γ be an indicator vector encodin which components of β are used in a model. When p is the dimensionality of β, we miht have, for example, γ = p elements {}}{ (0, 0, 1, 1, 0, 1,..., 0) Further define X γ to be the matrix that has only those columns of X correspondin to ones in the γ vector, and let β γ be the vector that has only those components of β correspondin to ones in γ. Then there are p models M γ of the form y = X γ β γ + ǫ Model selection refers to the problem of pickin one of these models. That is, we wish to choose γ based on data (X, y). The problem is like hypothesis testin but with many simultaneous hypotheses. The issue we encounter is that p can be very lare. We need a way to select γ based on priors on the parameters in each model, but with potentially very many models, assinin separate priors for each model may be infeasible. We consider instead a Bayesian approach utilizin nested models. Recall that, when we wish to use the Bayes factor, we may use improper priors when the same parameter appears in both models.
6 6 Bayes Factors, -priors, and Model Selection for Reression Then we miht, for instance, use a -prior on the remainin parameters. This pairwise concept extends to model selection by comparin all models to the null model or the full model. That is, γ null = (0,..., 0) γ full = (1,..., 1) For some base model M b, we can calculate for each model M γ, Then for any two models M γ and M γ, we have BF(M γ, M b ) = p(y M γ) p(y M b ) BF(M γ, M γ ) = BF(M γ, M b ) BF(M γ, M b ) If the base model is null, then the only factor in common between any model and M null is σ (and often the intercept α). We can place a Jeffreys prior separately on each of these parameters. In particular, α is a scale parameter, so π(σ, α) 1 σ For the case where the base model is the full model, see Lian et al. (008) for more information. How to set, continued Choosin -priors for the remainin components of β yields a closed-form expression for BF(M γ, M γ) via BF(M γ, M null ) = (1 + )(n pγ 1)/ (1 + (1 Rγ ))(n 1)/ Here, R γ is the coefficient of determination for the model M γ, and p γ is the remainin number of parameters of β γ. We can see directly now that if we send, then BF(M γ, M null ) 0. That is, for any value of n, the null model is always preferred in the limit. This behavior is an instance of Lindley s paradox. On the other hand, consider choosin a fixed, e.. by usin BIC, and lettin n. Then BF(M γ, M). That is, no matter the data, the alternative to the null is always preferred in the limit n. We will see a resolution to this quandary in the next lecture. References Lian, F., Paulo, R., Molina, G., Clyde, M. A., and Berer, J. O. (008). Mixtures of -priors for Bayesian variable selection. Journal of the American Statistical Association, 103(481):
g-priors for Linear Regression
Stat60: Bayesian Modeling and Inference Lecture Date: March 15, 010 g-priors for Linear Regression Lecturer: Michael I. Jordan Scribe: Andrew H. Chan 1 Linear regression and g-priors In the last lecture,
More informationBayesian Linear Regression
Bayesian Linear Regression Sudipto Banerjee 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. September 15, 2010 1 Linear regression models: a Bayesian perspective
More informationSparse Linear Models (10/7/13)
STA56: Probabilistic machine learning Sparse Linear Models (0/7/) Lecturer: Barbara Engelhardt Scribes: Jiaji Huang, Xin Jiang, Albert Oh Sparsity Sparsity has been a hot topic in statistics and machine
More informationHypothesis Testing. Econ 690. Purdue University. Justin L. Tobias (Purdue) Testing 1 / 33
Hypothesis Testing Econ 690 Purdue University Justin L. Tobias (Purdue) Testing 1 / 33 Outline 1 Basic Testing Framework 2 Testing with HPD intervals 3 Example 4 Savage Dickey Density Ratio 5 Bartlett
More informationStat260: Bayesian Modeling and Inference Lecture Date: February 10th, Jeffreys priors. exp 1 ) p 2
Stat260: Bayesian Modeling and Inference Lecture Date: February 10th, 2010 Jeffreys priors Lecturer: Michael I. Jordan Scribe: Timothy Hunter 1 Priors for the multivariate Gaussian Consider a multivariate
More informationThe linear model is the most fundamental of all serious statistical models encompassing:
Linear Regression Models: A Bayesian perspective Ingredients of a linear model include an n 1 response vector y = (y 1,..., y n ) T and an n p design matrix (e.g. including regressors) X = [x 1,..., x
More informationBayesian Inference: Concept and Practice
Inference: Concept and Practice fundamentals Johan A. Elkink School of Politics & International Relations University College Dublin 5 June 2017 1 2 3 Bayes theorem In order to estimate the parameters of
More information1 Hypothesis Testing and Model Selection
A Short Course on Bayesian Inference (based on An Introduction to Bayesian Analysis: Theory and Methods by Ghosh, Delampady and Samanta) Module 6: From Chapter 6 of GDS 1 Hypothesis Testing and Model Selection
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Department of Forestry & Department of Geography, Michigan State University, Lansing Michigan, U.S.A. 2 Biostatistics, School of Public
More informationStatistics Hotelling s T Gary W. Oehlert School of Statistics 313B Ford Hall
\ C f C E A tatistics 5041 11 Hotellin s T Gary W ehlert chool of tatistics 313B Ford Hall 612-625-155 ary@statumnedu Let s think about the univariate -test First recall that there are one-sample tests,
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationAssociation studies and regression
Association studies and regression CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar Association studies and regression 1 / 104 Administration
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department
More informationBayesian Linear Models
Eric F. Lock UMN Division of Biostatistics, SPH elock@umn.edu 03/07/2018 Linear model For observations y 1,..., y n, the basic linear model is y i = x 1i β 1 +... + x pi β p + ɛ i, x 1i,..., x pi are predictors
More informationAn Introduction to Bayesian Linear Regression
An Introduction to Bayesian Linear Regression APPM 5720: Bayesian Computation Fall 2018 A SIMPLE LINEAR MODEL Suppose that we observe explanatory variables x 1, x 2,..., x n and dependent variables y 1,
More informationMultiple Linear Regression
Multiple Linear Regression Simple linear regression tries to fit a simple line between two variables Y and X. If X is linearly related to Y this explains some of the variability in Y. In most cases, there
More informationOverall Objective Priors
Overall Objective Priors Jim Berger, Jose Bernardo and Dongchu Sun Duke University, University of Valencia and University of Missouri Recent advances in statistical inference: theory and case studies University
More informationA Very Brief Summary of Bayesian Inference, and Examples
A Very Brief Summary of Bayesian Inference, and Examples Trinity Term 009 Prof Gesine Reinert Our starting point are data x = x 1, x,, x n, which we view as realisations of random variables X 1, X,, X
More informationCaterpillar Regression Example: Conjugate Priors, Conditional & Marginal Posteriors, Predictive Distribution, Variable Selection
Caterpillar Regression Example: Conjugate Priors, Conditional & Marginal Posteriors, Predictive Distribution, Variable Selection Prof. Nicholas Zabaras University of Notre Dame Notre Dame, IN, USA Email:
More informationLinear Models A linear model is defined by the expression
Linear Models A linear model is defined by the expression x = F β + ɛ. where x = (x 1, x 2,..., x n ) is vector of size n usually known as the response vector. β = (β 1, β 2,..., β p ) is the transpose
More informationSCHOOL OF MATHEMATICS AND STATISTICS. Linear and Generalised Linear Models
SCHOOL OF MATHEMATICS AND STATISTICS Linear and Generalised Linear Models Autumn Semester 2017 18 2 hours Attempt all the questions. The allocation of marks is shown in brackets. RESTRICTED OPEN BOOK EXAMINATION
More informationMeasuring the fit of the model - SSR
Measuring the fit of the model - SSR Once we ve determined our estimated regression line, we d like to know how well the model fits. How far/close are the observations to the fitted line? One way to do
More informationSimple Linear Regression
Simple Linear Regression Reading: Hoff Chapter 9 November 4, 2009 Problem Data: Observe pairs (Y i,x i ),i = 1,... n Response or dependent variable Y Predictor or independent variable X GOALS: Exploring
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationBayesian Regression (1/31/13)
STA613/CBB540: Statistical methods in computational biology Bayesian Regression (1/31/13) Lecturer: Barbara Engelhardt Scribe: Amanda Lea 1 Bayesian Paradigm Bayesian methods ask: given that I have observed
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationBayesian Model Averaging
Bayesian Model Averaging Hoff Chapter 9, Hoeting et al 1999, Clyde & George 2004, Liang et al 2008 October 24, 2017 Bayesian Model Choice Models for the variable selection problem are based on a subset
More informationSimple and Multiple Linear Regression
Sta. 113 Chapter 12 and 13 of Devore March 12, 2010 Table of contents 1 Simple Linear Regression 2 Model Simple Linear Regression A simple linear regression model is given by Y = β 0 + β 1 x + ɛ where
More informationNovember 2002 STA Random Effects Selection in Linear Mixed Models
November 2002 STA216 1 Random Effects Selection in Linear Mixed Models November 2002 STA216 2 Introduction It is common practice in many applications to collect multiple measurements on a subject. Linear
More informationFoundations of Statistical Inference
Foundations of Statistical Inference Julien Berestycki Department of Statistics University of Oxford MT 2015 Julien Berestycki (University of Oxford) SB2a MT 2015 1 / 16 Lecture 16 : Bayesian analysis
More informationInferences for Regression
Inferences for Regression An Example: Body Fat and Waist Size Looking at the relationship between % body fat and waist size (in inches). Here is a scatterplot of our data set: Remembering Regression In
More informationA Bayesian Treatment of Linear Gaussian Regression
A Bayesian Treatment of Linear Gaussian Regression Frank Wood December 3, 2009 Bayesian Approach to Classical Linear Regression In classical linear regression we have the following model y β, σ 2, X N(Xβ,
More informationModel comparison and selection
BS2 Statistical Inference, Lectures 9 and 10, Hilary Term 2008 March 2, 2008 Hypothesis testing Consider two alternative models M 1 = {f (x; θ), θ Θ 1 } and M 2 = {f (x; θ), θ Θ 2 } for a sample (X = x)
More informationModule 22: Bayesian Methods Lecture 9 A: Default prior selection
Module 22: Bayesian Methods Lecture 9 A: Default prior selection Peter Hoff Departments of Statistics and Biostatistics University of Washington Outline Jeffreys prior Unit information priors Empirical
More informationST 740: Linear Models and Multivariate Normal Inference
ST 740: Linear Models and Multivariate Normal Inference Alyson Wilson Department of Statistics North Carolina State University November 4, 2013 A. Wilson (NCSU STAT) Linear Models November 4, 2013 1 /
More informationSTAT 425: Introduction to Bayesian Analysis
STAT 425: Introduction to Bayesian Analysis Marina Vannucci Rice University, USA Fall 2017 Marina Vannucci (Rice University, USA) Bayesian Analysis (Part 1) Fall 2017 1 / 10 Lecture 7: Prior Types Subjective
More informationSubject CS1 Actuarial Statistics 1 Core Principles
Institute of Actuaries of India Subject CS1 Actuarial Statistics 1 Core Principles For 2019 Examinations Aim The aim of the Actuarial Statistics 1 subject is to provide a grounding in mathematical and
More informationSTAT5044: Regression and Anova. Inyoung Kim
STAT5044: Regression and Anova Inyoung Kim 2 / 47 Outline 1 Regression 2 Simple Linear regression 3 Basic concepts in regression 4 How to estimate unknown parameters 5 Properties of Least Squares Estimators:
More informationSupplementary materials for Scalable Bayesian model averaging through local information propagation
Supplementary materials for Scalable Bayesian model averaging through local information propagation August 25, 2014 S1. Proofs Proof of Theorem 1. The result follows immediately from the distributions
More informationSTAT763: Applied Regression Analysis. Multiple linear regression. 4.4 Hypothesis testing
STAT763: Applied Regression Analysis Multiple linear regression 4.4 Hypothesis testing Chunsheng Ma E-mail: cma@math.wichita.edu 4.4.1 Significance of regression Null hypothesis (Test whether all β j =
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationRobust Semiparametric Optimal Testing Procedure for Multiple Normal Means
Veterinary Dianostic and Production Animal Medicine Publications Veterinary Dianostic and Production Animal Medicine 01 Robust Semiparametric Optimal Testin Procedure for Multiple Normal Means Pen Liu
More informationStat 5102 Final Exam May 14, 2015
Stat 5102 Final Exam May 14, 2015 Name Student ID The exam is closed book and closed notes. You may use three 8 1 11 2 sheets of paper with formulas, etc. You may also use the handouts on brand name distributions
More informationMixtures of g-priors for Bayesian Variable Selection
Mixtures of g-priors for Bayesian Variable Selection Feng Liang, Rui Paulo, German Molina, Merlise A. Clyde and Jim O. Berger August 8, 007 Abstract Zellner s g-prior remains a popular conventional prior
More informationPower-Expected-Posterior Priors for Variable Selection in Gaussian Linear Models
Power-Expected-Posterior Priors for Variable Selection in Gaussian Linear Models Ioannis Ntzoufras, Department of Statistics, Athens University of Economics and Business, Athens, Greece; e-mail: ntzoufras@aueb.gr.
More informationBayesian Classification Methods
Bayesian Classification Methods Suchit Mehrotra North Carolina State University smehrot@ncsu.edu October 24, 2014 Suchit Mehrotra (NCSU) Bayesian Classification October 24, 2014 1 / 33 How do you define
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2008 Prof. Gesine Reinert 1 Data x = x 1, x 2,..., x n, realisations of random variables X 1, X 2,..., X n with distribution (model)
More informationIntroduction. Start with a probability distribution f(y θ) for the data. where η is a vector of hyperparameters
Introduction Start with a probability distribution f(y θ) for the data y = (y 1,...,y n ) given a vector of unknown parameters θ = (θ 1,...,θ K ), and add a prior distribution p(θ η), where η is a vector
More informationGeneralized Linear Models
Generalized Linear Models Lecture 3. Hypothesis testing. Goodness of Fit. Model diagnostics GLM (Spring, 2018) Lecture 3 1 / 34 Models Let M(X r ) be a model with design matrix X r (with r columns) r n
More informationStatistical Methods in Particle Physics
Statistical Methods in Particle Physics Lecture 11 January 7, 2013 Silvia Masciocchi, GSI Darmstadt s.masciocchi@gsi.de Winter Semester 2012 / 13 Outline How to communicate the statistical uncertainty
More informationLecture 14 Simple Linear Regression
Lecture 4 Simple Linear Regression Ordinary Least Squares (OLS) Consider the following simple linear regression model where, for each unit i, Y i is the dependent variable (response). X i is the independent
More informationInference for a Population Proportion
Al Nosedal. University of Toronto. November 11, 2015 Statistical inference is drawing conclusions about an entire population based on data in a sample drawn from that population. From both frequentist
More informationLinear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
More informationST440/540: Applied Bayesian Statistics. (9) Model selection and goodness-of-fit checks
(9) Model selection and goodness-of-fit checks Objectives In this module we will study methods for model comparisons and checking for model adequacy For model comparisons there are a finite number of candidate
More informationBayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationEco517 Fall 2004 C. Sims MIDTERM EXAM
Eco517 Fall 2004 C. Sims MIDTERM EXAM Answer all four questions. Each is worth 23 points. Do not devote disproportionate time to any one question unless you have answered all the others. (1) We are considering
More informationHypothesis Testing. Part I. James J. Heckman University of Chicago. Econ 312 This draft, April 20, 2006
Hypothesis Testing Part I James J. Heckman University of Chicago Econ 312 This draft, April 20, 2006 1 1 A Brief Review of Hypothesis Testing and Its Uses values and pure significance tests (R.A. Fisher)
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Mixed Effects Estimation, Residuals Diagnostics Week 11, Lecture 1
MA 575 Linear Models: Cedric E Ginestet, Boston University Mixed Effects Estimation, Residuals Diagnostics Week 11, Lecture 1 1 Within-group Correlation Let us recall the simple two-level hierarchical
More informationLECTURE 5 NOTES. n t. t Γ(a)Γ(b) pt+a 1 (1 p) n t+b 1. The marginal density of t is. Γ(t + a)γ(n t + b) Γ(n + a + b)
LECTURE 5 NOTES 1. Bayesian point estimators. In the conventional (frequentist) approach to statistical inference, the parameter θ Θ is considered a fixed quantity. In the Bayesian approach, it is considered
More informationBayesian methods in economics and finance
1/26 Bayesian methods in economics and finance Linear regression: Bayesian model selection and sparsity priors Linear Regression 2/26 Linear regression Model for relationship between (several) independent
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7
MA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7 1 Random Vectors Let a 0 and y be n 1 vectors, and let A be an n n matrix. Here, a 0 and A are non-random, whereas y is
More informationTHE SIGNAL ESTIMATOR LIMIT SETTING METHOD
' THE SIGNAL ESTIMATOR LIMIT SETTING METHOD Shan Jin, Peter McNamara Department of Physics, University of Wisconsin Madison, Madison, WI 53706 Abstract A new method of backround subtraction is presented
More informationMixtures of g-priors for Bayesian Variable Selection
Mixtures of g-priors for Bayesian Variable Selection January 8, 007 Abstract Zellner s g-prior remains a popular conventional prior for use in Bayesian variable selection, despite several undesirable consistency
More informationAMS-207: Bayesian Statistics
Linear Regression How does a quantity y, vary as a function of another quantity, or vector of quantities x? We are interested in p(y θ, x) under a model in which n observations (x i, y i ) are exchangeable.
More informationChapter 14. Linear least squares
Serik Sagitov, Chalmers and GU, March 5, 2018 Chapter 14 Linear least squares 1 Simple linear regression model A linear model for the random response Y = Y (x) to an independent variable X = x For a given
More informationModule 4: Bayesian Methods Lecture 5: Linear regression
1/28 The linear regression model Module 4: Bayesian Methods Lecture 5: Linear regression Peter Hoff Departments of Statistics and Biostatistics University of Washington 2/28 The linear regression model
More informationLecture 6 Multiple Linear Regression, cont.
Lecture 6 Multiple Linear Regression, cont. BIOST 515 January 22, 2004 BIOST 515, Lecture 6 Testing general linear hypotheses Suppose we are interested in testing linear combinations of the regression
More informationAsymptotic Multivariate Kriging Using Estimated Parameters with Bayesian Prediction Methods for Non-linear Predictands
Asymptotic Multivariate Kriging Using Estimated Parameters with Bayesian Prediction Methods for Non-linear Predictands Elizabeth C. Mannshardt-Shamseldin Advisor: Richard L. Smith Duke University Department
More informationSimple Linear Regression
Simple Linear Regression ST 370 Regression models are used to study the relationship of a response variable and one or more predictors. The response is also called the dependent variable, and the predictors
More informationSummer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.
Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall
More informationMinimum Message Length Analysis of the Behrens Fisher Problem
Analysis of the Behrens Fisher Problem Enes Makalic and Daniel F Schmidt Centre for MEGA Epidemiology The University of Melbourne Solomonoff 85th Memorial Conference, 2011 Outline Introduction 1 Introduction
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationStatistical Inference
Statistical Inference Liu Yang Florida State University October 27, 2016 Liu Yang, Libo Wang (Florida State University) Statistical Inference October 27, 2016 1 / 27 Outline The Bayesian Lasso Trevor Park
More informationSimple Linear Regression
Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.
More informationStat 535 C - Statistical Computing & Monte Carlo Methods. Arnaud Doucet.
Stat 535 C - Statistical Computing & Monte Carlo Methods Arnaud Doucet Email: arnaud@cs.ubc.ca 1 CS students: don t forget to re-register in CS-535D. Even if you just audit this course, please do register.
More informationProblem 1 (20) Log-normal. f(x) Cauchy
ORF 245. Rigollet Date: 11/21/2008 Problem 1 (20) f(x) f(x) 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 4 2 0 2 4 Normal (with mean -1) 4 2 0 2 4 Negative-exponential x x f(x) f(x) 0.0 0.1 0.2 0.3 0.4 0.5
More informationModule 17: Bayesian Statistics for Genetics Lecture 4: Linear regression
1/37 The linear regression model Module 17: Bayesian Statistics for Genetics Lecture 4: Linear regression Ken Rice Department of Biostatistics University of Washington 2/37 The linear regression model
More informationReview of Classical Least Squares. James L. Powell Department of Economics University of California, Berkeley
Review of Classical Least Squares James L. Powell Department of Economics University of California, Berkeley The Classical Linear Model The object of least squares regression methods is to model and estimate
More informationThe Normal Linear Regression Model with Natural Conjugate Prior. March 7, 2016
The Normal Linear Regression Model with Natural Conjugate Prior March 7, 2016 The Normal Linear Regression Model with Natural Conjugate Prior The plan Estimate simple regression model using Bayesian methods
More informationPhysics 403. Segev BenZvi. Parameter Estimation, Correlations, and Error Bars. Department of Physics and Astronomy University of Rochester
Physics 403 Parameter Estimation, Correlations, and Error Bars Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Review of Last Class Best Estimates and Reliability
More informationFirst Year Examination Department of Statistics, University of Florida
First Year Examination Department of Statistics, University of Florida August 19, 010, 8:00 am - 1:00 noon Instructions: 1. You have four hours to answer questions in this examination.. You must show your
More informationSTAT 3A03 Applied Regression With SAS Fall 2017
STAT 3A03 Applied Regression With SAS Fall 2017 Assignment 2 Solution Set Q. 1 I will add subscripts relating to the question part to the parameters and their estimates as well as the errors and residuals.
More informationData Mining Chapter 4: Data Analysis and Uncertainty Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
Data Mining Chapter 4: Data Analysis and Uncertainty Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Why uncertainty? Why should data mining care about uncertainty? We
More informationCh 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
More informationMotivation Scale Mixutres of Normals Finite Gaussian Mixtures Skew-Normal Models. Mixture Models. Econ 690. Purdue University
Econ 690 Purdue University In virtually all of the previous lectures, our models have made use of normality assumptions. From a computational point of view, the reason for this assumption is clear: combined
More informationSeminar über Statistik FS2008: Model Selection
Seminar über Statistik FS2008: Model Selection Alessia Fenaroli, Ghazale Jazayeri Monday, April 2, 2008 Introduction Model Choice deals with the comparison of models and the selection of a model. It can
More informationINTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y
INTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y Predictor or Independent variable x Model with error: for i = 1,..., n, y i = α + βx i + ε i ε i : independent errors (sampling, measurement,
More informationMachine Learning. Gaussian Mixture Models. Zhiyao Duan & Bryan Pardo, Machine Learning: EECS 349 Fall
Machine Learning Gaussian Mixture Models Zhiyao Duan & Bryan Pardo, Machine Learning: EECS 349 Fall 2012 1 The Generative Model POV We think of the data as being generated from some process. We assume
More informationPart III. A Decision-Theoretic Approach and Bayesian testing
Part III A Decision-Theoretic Approach and Bayesian testing 1 Chapter 10 Bayesian Inference as a Decision Problem The decision-theoretic framework starts with the following situation. We would like to
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Empirical Bayes, Hierarchical Bayes Mark Schmidt University of British Columbia Winter 2017 Admin Assignment 5: Due April 10. Project description on Piazza. Final details coming
More informationBayesian Variable Selection Under Collinearity
Bayesian Variable Selection Under Collinearity Joyee Ghosh Andrew E. Ghattas. June 3, 2014 Abstract In this article we provide some guidelines to practitioners who use Bayesian variable selection for linear
More information5.2 Expounding on the Admissibility of Shrinkage Estimators
STAT 383C: Statistical Modeling I Fall 2015 Lecture 5 September 15 Lecturer: Purnamrita Sarkar Scribe: Ryan O Donnell Disclaimer: These scribe notes have been slightly proofread and may have typos etc
More informationBayesian inference. Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark. April 10, 2017
Bayesian inference Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark April 10, 2017 1 / 22 Outline for today A genetic example Bayes theorem Examples Priors Posterior summaries
More informationLecture 7 October 13
STATS 300A: Theory of Statistics Fall 2015 Lecture 7 October 13 Lecturer: Lester Mackey Scribe: Jing Miao and Xiuyuan Lu 7.1 Recap So far, we have investigated various criteria for optimal inference. We
More information01 Probability Theory and Statistics Review
NAVARCH/EECS 568, ROB 530 - Winter 2018 01 Probability Theory and Statistics Review Maani Ghaffari January 08, 2018 Last Time: Bayes Filters Given: Stream of observations z 1:t and action data u 1:t Sensor/measurement
More informationNon-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
More informationLecture 2. The Simple Linear Regression Model: Matrix Approach
Lecture 2 The Simple Linear Regression Model: Matrix Approach Matrix algebra Matrix representation of simple linear regression model 1 Vectors and Matrices Where it is necessary to consider a distribution
More informationBayesian Regression Linear and Logistic Regression
When we want more than point estimates Bayesian Regression Linear and Logistic Regression Nicole Beckage Ordinary Least Squares Regression and Lasso Regression return only point estimates But what if we
More informationLECTURE 5 HYPOTHESIS TESTING
October 25, 2016 LECTURE 5 HYPOTHESIS TESTING Basic concepts In this lecture we continue to discuss the normal classical linear regression defined by Assumptions A1-A5. Let θ Θ R d be a parameter of interest.
More information