Bayesian Inference. Chapter 4: Regression and Hierarchical Models
|
|
- Julius Logan
- 5 years ago
- Views:
Transcription
1 Bayesian Inference Chapter 4: Regression and Hierarchical Models Conchi Ausín and Mike Wiper Department of Statistics Universidad Carlos III de Madrid Advanced Statistics and Data Mining Summer School 29th June - 10th July, 2015 Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 1 / 34
2 Objective AFM Smith Dennis Lindley We analyze the Bayesian approach to fitting normal and generalized linear models and introduce the Bayesian hierarchical modeling approach. Also, we study the modeling and forecasting of time series. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 2 / 34
3 Contents 1 Normal linear models 1.1. ANOVA model 1.2. Simple linear regression model 2 Generalized linear models 3 Hierarchical models 4 Dynamic models Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 3 / 34
4 Normal linear models A normal linear model is of the following form: y = Xθ + ɛ, where y = (y 1,..., y n ) is the observed data, X is a known n k matrix, called the design matrix, θ = (θ 1,..., θ k ) is the parameter set and ɛ follows a multivariate normal distribution. Usually, it is assumed that: ( ɛ N 0 k, 1 ) φ I k. A simple example of normal linear model is the simple linear regression model ( ) T where X = and θ = (α, β) x 1 x 2... x T. n Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 4 / 34
5 Normal linear models Consider a normal linear model, y = Xθ + ɛ. A conjugate prior distribution is a normal-gamma distribution: θ φ N (m, 1φ ) V ( a φ G 2, b ). 2 Then, the posterior distribution given y is also a normal-gamma distribution with: m = ( X T X + V 1) 1 ( X T y + V 1 m ) V = ( X T X + V 1) 1 a = a + n b = b + y T y + m T V 1 m m T V 1 m Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 5 / 34
6 Normal linear models The posterior mean is given by: E [θ y] = ( X T X + V 1) 1 ( X T y + V 1 m ) = ( X T X + V 1) ( 1 X T X ( X T X ) ) 1 X T y + V 1 m = ( X T X + V 1) ( ) 1 X T Xˆθ + V 1 m where ˆθ = ( X T X ) 1 X T y is the maximum likelihood estimator. Thus, this expression may be interpreted as a weighted average of the prior estimator, m, and the MLE, ˆθ, with weights proportional to precisions since, conditional on φ, the prior variance is 1 φv and that the distribution of the MLE from the classical viewpoint is ˆθ ) φ N (θ, 1φ (XT X) 1 Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 6 / 34
7 Normal linear models Consider a normal linear model, y = Xθ + ɛ, and assume the limiting prior distribution, Then, we have that, θ y, φ N φ y G p(θ, φ) 1 φ. ( ˆθ, 1 ( X T X ) ) 1, φ n k 2, yt y ˆθ T ( X T X ) ˆθ 2. Note that ˆσ 2 = yt y ˆθ T (X T X)ˆθ n k is the usual classical estimator of σ 2 = 1 φ. In this case, Bayesian credible intervals, estimators etc. will coincide with their classical counterparts. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 7 / 34
8 ANOVA model The ANOVA model is an example of normal lineal model where: y ij = θ i + ɛ ij, where ɛ ij N (0, 1 φ ), for i = 1,..., k, and j = 1,..., n i. Thus, the parameters are θ = (θ 1,..., θ k ), the observed data are y = (y 11,..., y 1n1, y 21,..., y 2n2,..., y k1,..., y knk ) T, the design matrix is: X = n n Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 8 / 34
9 ANOVA model Assume conditionally independent normal priors, θ i N i = 1,..., k, and a gamma prior φ G( a 2, b 2 ). ( ) 1 m i, α i φ, for This corresponds to a normal-gamma ( prior distribution for (θ, φ) where 1 m = (m 1,..., m k ) and V = diag α 1,..., 1 α k ). Then, it is obtained that, and θ y, φ N φ y G n 1ȳ 1 +α 1m 1 n 1+α 1. n 1ȳ 1 +α 1m 1 n 1+α 1, 1 φ 1 α 1+n 1... ( a + n 2, b + k ni i=1 j=1 (y ij ȳ i ) 2 + k 2 1 α k +n k n i i=1 n i +α i (ȳ i m i ) 2 ) Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 9 / 34
10 ANOVA model If we assume alternatively the reference prior, p(θ, φ) 1 φ, we have: θ y, φ N ( n k φ G, 2 ȳ 1.. ȳ k, 1 φ (n k) ˆσ2 2 ), 1 n n k, where ˆσ 2 = 1 n k k i=1 (y ij ȳ i ) 2 is the classical variance estimate for this problem. A 95% posterior interval for θ 1 θ 2 is given by: 1 ȳ 1 ȳ 2 ± ˆσ + 1 t n k (0.975), n 1 n 2 which is equal to the usual, classical interval. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 10 / 34
11 Example: ANOVA model Suppose that an ecologist is interested in analysing how the masses of starlings (a type of birds) vary between four locations. A sample data of the weights of 10 starlings from each of the four locations can be downloaded from: Assume a Bayesian one-way ANOVA model for these data where a different mean is considered for each location and the variation in mass between different birds is described by a normal distribution with a common variance. Compare the results with those obtained with classical methods. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 11 / 34
12 Simple linear regression model Another example of normal linear model is the simple regression model: for i = 1,..., n, where ɛ i N Suppose that we use the limiting prior: y i = α + βx i + ɛ i, ( ) 0, 1 φ. p(α, β, φ) 1 φ. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 12 / 34
13 Simple linear regression model Then, we have that: ( α β y, φ N ˆαˆβ where: φ y G ), ( n 2 2, s y n 1 xi 2 i=1 φns x n x ( ) ) 1 r 2 2 n x n ˆα = ȳ ˆβ x, ˆβ = s xy s x, s x = n i=1 (x i x) 2, s y = n i=1 (y i ȳ) 2, s xy = n i=1 (x i x) (y i ȳ), r = s xy sx s y, ˆσ 2 = s ( ) y 1 r 2. n 2 Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 13 / 34
14 Simple linear regression model Thus, the marginal distributions of α and β are Student-t distributions: α ˆα ˆσ 2 n n i=1 x 2 i s x β ˆβ ˆσ 2 s x y t n 2 y t n 2 Therefore, for example, a 95% credible interval for β is given by: ˆβ ± equal to the usual classical interval. ˆσ sx t n 2 (0.975) Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 14 / 34
15 Simple linear regression model Suppose now that we wish to predict a future observation: Note that, y new = α + βx new + ɛ new. E [y new φ, y] = ˆα + ˆβx new V [y new φ, y] = 1 ( n i=1 x i 2 + nx 2 ) new 2n xx new + 1 φ ns x = 1 ( sx + n x 2 + nx 2 ) new 2n xx new + 1 φ ns x Therefore, y new φ, y N ( ˆα + ˆβx new, 1 φ ( ( x x new ) 2 s x + 1 n + 1 )) Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 15 / 34
16 Simple linear regression model And then, y new ˆα + ˆβx new ( ) y t n 2 ˆσ ( x xnew ) n + 1 s x leading to the following 95% credible interval for y new : ( ) ˆα + ˆβx new ± ˆσ ( x x new ) s x n + 1 t n 2 (0.975), which coincides with the usual, classical interval. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 16 / 34
17 Example: Simple linear regression model Consider the data file prostate.data that can be downloaded from: This includes, among other clinical measures, the level of prostate specific antigen in logs (lpsa) and the log cancer volume (lcavol) in 97 men who were about to receive a radical prostatectomy. Use a Bayesian linear regression model to predict the lpsa in terms of the lcavol. Compare the results with a classical linear regression fit. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 17 / 34
18 Generalized linear models The generalized linear model generalizes the normal linear model by allowing the possibility of non-normal error distributions and by allowing for a non-linear relationship between y and x. A generalized linear model is specified by two functions: 1 A conditional, exponential family density function of y given x, parameterized by a mean parameter, µ = µ(x) = E[Y x] and (possibly) a dispersion parameter, φ > 0, that is independent of x. 2 A (one-to-one) link function, g( ), which relates the mean, µ = µ(x) to the covariate vector, x, as g(µ) = xθ. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 18 / 34
19 Generalized linear models The following are generalized linear models with the canonical link function which is the natural parameterization to leave the exponential family distribution in canonical form. A logistic regression is often used for predicting the occurrence of an event given covariates: Y i p i Bin(n i, p i ) p i log = x i θ 1 p i A Poisson regression is used for predicting the number of events in a time period given covariates: Y i p i P(λ i ) log λ i = x i θ Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 19 / 34
20 Generalized linear models The Bayesian specification of a GLM is completed by defining (typically normal or normal gamma) prior distributions p(θ, φ) over the unknown model parameters. As with standard linear models, when improper priors are used, it is then important to check that these lead to valid posterior distributions. Clearly, these models will not have conjugate posterior distributions, but, usually, they are easily handled by Gibbs sampling. In particular, the posterior distributions from these models are usually log concave and are thus easily sampled via adaptive rejection sampling. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 20 / 34
21 Example: A logistic regression model The O-Ring data consist of 23 observations on Pre-Challenger Space Shuttle Launches On each launch, it is observed whether there is at least one O-ring failure, and the temperature at launch The goal is to model the probability of at least one O-ring failure as a function of temperature. Temperatures were 53, 57, 58, 63, 66, 67, 67, 67, 68, 69,70, 70, 70, 70, 72, 73, 75, 75, 76, 76, 78, 79, 81 Failures occurred at 53, 57, 58, 63, 70, 70, 75 Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 21 / 34
22 Hierarchical models Suppose we have data, x, and a likelihood function f (x θ) where the parameter values θ = (θ 1,..., θ k ) are judged to be exchangeable, that is, any permutation of them has the same distribution. In this situation, it makes sense to consider a multilevel modeling assuming a prior distribution, f (θ φ), which depends upon a further, unknown hyperparameter, φ, and use a hyperprior distribution, f (φ). In theory, this process could continue further, using hyperhyperprior distributions to estimate the hyperprior distributions. This is a method to elicit the optimal prior distributions. One alternative is to estimate the hyperparameter using classical methods, which is known as empirical Bayes. A point estimate ˆφ is then obtained to approximate the posterior distribution. However, the uncertainty in φ is ignored. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 22 / 34
23 Hierarchical models In most hierarchical models, the joint posterior distributions will not be analytically tractable as it will be, f (θ, φ x) f (x θ)f (θ φ)f (φ) However, often a Gibbs sampling approach can be implemented by sampling from the conditional posterior distributions: f (θ x, φ) f (x θ)f (θ φ) f (φ x, θ) f (θ φ)f (φ) It is important to check the propriety of the posterior distribution when improper hyperprior distributions are used. An alternative (as in for example Winbugs) is to use proper but high variance hyperprior distributions. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 23 / 34
24 Hierarchical models For example, a hierachical normal linear model is given by: ( x ij θ i, φ N θ i, 1 ), i = 1,..., n, j = 1,..., m. φ Assuming that the means, θ i, are exchangeable, we may consider the following prior distribution: ( θ i µ, ψ N µ, 1 ), ψ where the hyperparameters are µ y ψ. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 24 / 34
25 Example: A hierarchical one-way ANOVA Suppose that 5 individuals take 3 different IQ test developed by 3 different psychologists obtaining the following results: Test Test Test Then, we can assume that: ( X ij θ i, φ N θ i, 1 ), φ ( θ i µ, ψ N µ, 1 ), ψ for i = 1,..., 5, and j = 1, 2, 3, where θ i represents the true IQ of subject i and µ the mean true IQ in the population. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 25 / 34
26 Example: A hierarchical Poisson model The number of failures, X i at a power plant i is assumed to follow a Poisson distribution: X i λ i P(λ i t i ), para i = 1,..., 10, where λ i is the failure rate for pump i and t i is the length of operation time of the pump (in 1000s of hours). It seems natural to assume that the failure rates are exchangeable and thus we might assume: λ i γ E(γ), where γ is the prior hyperparameter. The observed data are: Pump t i x i Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 26 / 34
27 Dynamic models The univariate normal dynamic linear model (DLM) is: y t = F t θ t + ν t, ν t N (0, V t ) θ t = G t θ t 1 + ω t, ω t N (0, W t ). These models are linear state space models, where x t = F t θ t represents the signal, θ t is the state vector, F t is a regression vector and G t is a state matrix. The usual features of a time series such as trend and seasonality can be modeled within this format. If the matrices F t, G t, V t and W t are constants, the model is said to be time invariant. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 27 / 34
28 Dynamic models One of the simplest DLMs is the random walk plus noise model, also called first order polynomial model. It is used to model univariate observations and the state vector is unidimensional: y t = θ t + ν t, ν t N (0, V t ) θ t = θ t 1 + ω t, ω t N (0, W t ). This is a slowly varying level model where the observations fluctuate around a mean which varies according to a random walk. Assuming known variances, V t and W t, a straightforward Bayesian analysis can be carried out as follows. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 28 / 34
29 Dynamic models Suppose that the information at time t 1 is y t 1 = {y 1, y 2,..., y t 1 } and assume that: θ t 1 y t 1 N (m t 1, C t 1 ). Then, we have that: The prior distribution for θ t is: θ t y t 1 N (m t 1, R t ) where R t = C t 1 + W t The one step ahead predictive distribution for y t is: where Q t = R t + V t. y t y t 1 N (m t 1, Q t ) Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 29 / 34
30 Dynamic models The joint distribution of θ t and y t is: ( ) ( θt y t 1 mt 1 N m t 1 ( Rt, R t y t R t Q t )) The posterior distribution for θ t given y t = { y t 1, y t } is: θ t y t N(m t, C t ), m t = m t 1 + A t e t, A t = R t /Q t, e t = y t m t 1, C t = R t A 2 t Q t. where Note that e t is simply a prediction error term. The posterior mean formula could also be written as: m t = (1 A t ) m t 1 + A t y t. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 30 / 34
31 Example: First order polynomial DLM Assume a slowly varying level model for the water level in Lake Huron with known variances: V t = 1 and W t = 1. 1 Estimate the filtered values of the state vector based on the observations up to time t from f (θ t y t ). 2 Estimate the predicted values of the state vector based on the observations up to time t 1 from f (θ t y t 1 ). 3 Estimate the predicted values of the signal based on the observations up to time t 1 from f (y t y t 1 ). 4 Compare the results using e.g: Vt = 10 and W t = 1. Vt = 1 and W t = 10. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 31 / 34
32 Dynamic models When the variances are not known, the Bayesian inference for the system is more complex. One possibility is the use of MCMC algorithms which are usually based on the so-called forward filtering backward sampling algorithm. 1 The forward filtering step is the standard normal linear analysis to give f (θ t y t ) at each t, for t = 1,..., T. 2 The backward sampling step uses the Markov property and samples θ T from f (θ T y T ) and then, for t = T 1,..., 1, samples from f (θ t y t, θ t+1) Thus, a sample from the posterior parameter structure is generated. However, MCMC may be computationally very expensive for on-line estimation. One possible alternative is the use of particle filters. Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 32 / 34
33 Dynamic models Other examples of DLM are the following: A dynamic linear regression model is given by: y t = F t θ t + ν t, ν t N (0, V t ) θ t = θ t 1 + ω t, ω t N (0, W t ). The AR(p) model with time-varying coefficients takes the form: y t = θ 0t + θ 1t y t θ pt y t p + ν t, θ it = θ i,t 1 + ω it, This model can be expressed in state space form by setting θ = (θ 0t,..., θ pt ) and F = (1, y t 1,..., y t p). Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 33 / 34
34 Dynamic models The additive structure of the DLMs makes it easy to think of observed series as originating form the sum of different components,e.g., y t = y 1t +..., y h,t where y 1t might represent a trend component, y 2t a seasonal component, and so on. Then, each component, y it, might be described by a different DLM: y t = F it θ it + ν it, ν it N (0, V it ) θ it = G it θ t 1 + ω it, ω it N (0, W it ). By the assumption of independence of the components, y t is also a DLM described by: F t = (F 1t... F ht ), V t = V 1t V ht, and G t = G 1t... W 1t, W t =.... G ht W ht Conchi Ausín and Mike Wiper Regression and hierarchical models Advanced Statistics and Data Mining 34 / 34
Bayesian Inference. Chapter 4: Regression and Hierarchical Models
Bayesian Inference Chapter 4: Regression and Hierarchical Models Conchi Ausín and Mike Wiper Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative
More informationBayesian Inference. Chapter 9. Linear models and regression
Bayesian Inference Chapter 9. Linear models and regression M. Concepcion Ausin Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master in Mathematical Engineering
More information10. Exchangeability and hierarchical models Objective. Recommended reading
10. Exchangeability and hierarchical models Objective Introduce exchangeability and its relation to Bayesian hierarchical models. Show how to fit such models using fully and empirical Bayesian methods.
More informationINTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y
INTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y Predictor or Independent variable x Model with error: for i = 1,..., n, y i = α + βx i + ε i ε i : independent errors (sampling, measurement,
More informationBayesian Inference. Chapter 2: Conjugate models
Bayesian Inference Chapter 2: Conjugate models Conchi Ausín and Mike Wiper Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master in
More informationBayesian Regression Linear and Logistic Regression
When we want more than point estimates Bayesian Regression Linear and Logistic Regression Nicole Beckage Ordinary Least Squares Regression and Lasso Regression return only point estimates But what if we
More informationBayesian Linear Regression
Bayesian Linear Regression Sudipto Banerjee 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. September 15, 2010 1 Linear regression models: a Bayesian perspective
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationST 740: Linear Models and Multivariate Normal Inference
ST 740: Linear Models and Multivariate Normal Inference Alyson Wilson Department of Statistics North Carolina State University November 4, 2013 A. Wilson (NCSU STAT) Linear Models November 4, 2013 1 /
More informationRonald Christensen. University of New Mexico. Albuquerque, New Mexico. Wesley Johnson. University of California, Irvine. Irvine, California
Texts in Statistical Science Bayesian Ideas and Data Analysis An Introduction for Scientists and Statisticians Ronald Christensen University of New Mexico Albuquerque, New Mexico Wesley Johnson University
More informationBayesian Inference in GLMs. Frequentists typically base inferences on MLEs, asymptotic confidence
Bayesian Inference in GLMs Frequentists typically base inferences on MLEs, asymptotic confidence limits, and log-likelihood ratio tests Bayesians base inferences on the posterior distribution of the unknowns
More informationBayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationIntroduction to Bayesian Statistics with WinBUGS Part 4 Priors and Hierarchical Models
Introduction to Bayesian Statistics with WinBUGS Part 4 Priors and Hierarchical Models Matthew S. Johnson New York ASA Chapter Workshop CUNY Graduate Center New York, NY hspace1in December 17, 2009 December
More informationDefault Priors and Effcient Posterior Computation in Bayesian
Default Priors and Effcient Posterior Computation in Bayesian Factor Analysis January 16, 2010 Presented by Eric Wang, Duke University Background and Motivation A Brief Review of Parameter Expansion Literature
More informationNovember 2002 STA Random Effects Selection in Linear Mixed Models
November 2002 STA216 1 Random Effects Selection in Linear Mixed Models November 2002 STA216 2 Introduction It is common practice in many applications to collect multiple measurements on a subject. Linear
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationLinear Mixed Models. One-way layout REML. Likelihood. Another perspective. Relationship to classical ideas. Drawbacks.
Linear Mixed Models One-way layout Y = Xβ + Zb + ɛ where X and Z are specified design matrices, β is a vector of fixed effect coefficients, b and ɛ are random, mean zero, Gaussian if needed. Usually think
More informationeqr094: Hierarchical MCMC for Bayesian System Reliability
eqr094: Hierarchical MCMC for Bayesian System Reliability Alyson G. Wilson Statistical Sciences Group, Los Alamos National Laboratory P.O. Box 1663, MS F600 Los Alamos, NM 87545 USA Phone: 505-667-9167
More informationSimple Linear Regression
Simple Linear Regression Reading: Hoff Chapter 9 November 4, 2009 Problem Data: Observe pairs (Y i,x i ),i = 1,... n Response or dependent variable Y Predictor or independent variable X GOALS: Exploring
More informationθ 1 θ 2 θ n y i1 y i2 y in Hierarchical models (chapter 5) Hierarchical model Introduction to hierarchical models - sometimes called multilevel model
Hierarchical models (chapter 5) Introduction to hierarchical models - sometimes called multilevel model Exchangeability Slide 1 Hierarchical model Example: heart surgery in hospitals - in hospital j survival
More informationLinear Models A linear model is defined by the expression
Linear Models A linear model is defined by the expression x = F β + ɛ. where x = (x 1, x 2,..., x n ) is vector of size n usually known as the response vector. β = (β 1, β 2,..., β p ) is the transpose
More informationTopic 12 Overview of Estimation
Topic 12 Overview of Estimation Classical Statistics 1 / 9 Outline Introduction Parameter Estimation Classical Statistics Densities and Likelihoods 2 / 9 Introduction In the simplest possible terms, the
More informationAdvanced Statistics I : Gaussian Linear Model (and beyond)
Advanced Statistics I : Gaussian Linear Model (and beyond) Aurélien Garivier CNRS / Telecom ParisTech Centrale Outline One and Two-Sample Statistics Linear Gaussian Model Model Reduction and model Selection
More informationLecture 2: Priors and Conjugacy
Lecture 2: Priors and Conjugacy Melih Kandemir melih.kandemir@iwr.uni-heidelberg.de May 6, 2014 Some nice courses Fred A. Hamprecht (Heidelberg U.) https://www.youtube.com/watch?v=j66rrnzzkow Michael I.
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Department of Forestry & Department of Geography, Michigan State University, Lansing Michigan, U.S.A. 2 Biostatistics, School of Public
More informationThe linear model is the most fundamental of all serious statistical models encompassing:
Linear Regression Models: A Bayesian perspective Ingredients of a linear model include an n 1 response vector y = (y 1,..., y n ) T and an n p design matrix (e.g. including regressors) X = [x 1,..., x
More informationPOSTERIOR PROPRIETY IN SOME HIERARCHICAL EXPONENTIAL FAMILY MODELS
POSTERIOR PROPRIETY IN SOME HIERARCHICAL EXPONENTIAL FAMILY MODELS EDWARD I. GEORGE and ZUOSHUN ZHANG The University of Texas at Austin and Quintiles Inc. June 2 SUMMARY For Bayesian analysis of hierarchical
More informationABC methods for phase-type distributions with applications in insurance risk problems
ABC methods for phase-type with applications problems Concepcion Ausin, Department of Statistics, Universidad Carlos III de Madrid Joint work with: Pedro Galeano, Universidad Carlos III de Madrid Simon
More informationStat 579: Generalized Linear Models and Extensions
Stat 579: Generalized Linear Models and Extensions Linear Mixed Models for Longitudinal Data Yan Lu April, 2018, week 15 1 / 38 Data structure t1 t2 tn i 1st subject y 11 y 12 y 1n1 Experimental 2nd subject
More informationProblem Selected Scores
Statistics Ph.D. Qualifying Exam: Part II November 20, 2010 Student Name: 1. Answer 8 out of 12 problems. Mark the problems you selected in the following table. Problem 1 2 3 4 5 6 7 8 9 10 11 12 Selected
More informationBayesian Linear Models
Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationPart 8: GLMs and Hierarchical LMs and GLMs
Part 8: GLMs and Hierarchical LMs and GLMs 1 Example: Song sparrow reproductive success Arcese et al., (1992) provide data on a sample from a population of 52 female song sparrows studied over the course
More informationNow consider the case where E(Y) = µ = Xβ and V (Y) = σ 2 G, where G is diagonal, but unknown.
Weighting We have seen that if E(Y) = Xβ and V (Y) = σ 2 G, where G is known, the model can be rewritten as a linear model. This is known as generalized least squares or, if G is diagonal, with trace(g)
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationGeneralized Linear Models Introduction
Generalized Linear Models Introduction Statistics 135 Autumn 2005 Copyright c 2005 by Mark E. Irwin Generalized Linear Models For many problems, standard linear regression approaches don t work. Sometimes,
More informationOverall Objective Priors
Overall Objective Priors Jim Berger, Jose Bernardo and Dongchu Sun Duke University, University of Valencia and University of Missouri Recent advances in statistical inference: theory and case studies University
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationPrinciples of Bayesian Inference
Principles of Bayesian Inference Sudipto Banerjee University of Minnesota July 20th, 2008 1 Bayesian Principles Classical statistics: model parameters are fixed and unknown. A Bayesian thinks of parameters
More informationWeb Appendix for Hierarchical Adaptive Regression Kernels for Regression with Functional Predictors by D. B. Woodard, C. Crainiceanu, and D.
Web Appendix for Hierarchical Adaptive Regression Kernels for Regression with Functional Predictors by D. B. Woodard, C. Crainiceanu, and D. Ruppert A. EMPIRICAL ESTIMATE OF THE KERNEL MIXTURE Here we
More informationMarkov Chain Monte Carlo methods
Markov Chain Monte Carlo methods Tomas McKelvey and Lennart Svensson Signal Processing Group Department of Signals and Systems Chalmers University of Technology, Sweden November 26, 2012 Today s learning
More information[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationFinal Exam November 24, Problem-1: Consider random walk with drift plus a linear time trend: ( t
Problem-1: Consider random walk with drift plus a linear time trend: y t = c + y t 1 + δ t + ϵ t, (1) where {ϵ t } is white noise with E[ϵ 2 t ] = σ 2 >, and y is a non-stochastic initial value. (a) Show
More informationLatent Variable Models for Binary Data. Suppose that for a given vector of explanatory variables x, the latent
Latent Variable Models for Binary Data Suppose that for a given vector of explanatory variables x, the latent variable, U, has a continuous cumulative distribution function F (u; x) and that the binary
More informationA Bayesian Treatment of Linear Gaussian Regression
A Bayesian Treatment of Linear Gaussian Regression Frank Wood December 3, 2009 Bayesian Approach to Classical Linear Regression In classical linear regression we have the following model y β, σ 2, X N(Xβ,
More informationMCMC algorithms for fitting Bayesian models
MCMC algorithms for fitting Bayesian models p. 1/1 MCMC algorithms for fitting Bayesian models Sudipto Banerjee sudiptob@biostat.umn.edu University of Minnesota MCMC algorithms for fitting Bayesian models
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More informationIntroduction to Bayesian Methods
Introduction to Bayesian Methods Jessi Cisewski Department of Statistics Yale University Sagan Summer Workshop 2016 Our goal: introduction to Bayesian methods Likelihoods Priors: conjugate priors, non-informative
More informationAn Introduction to Bayesian Linear Regression
An Introduction to Bayesian Linear Regression APPM 5720: Bayesian Computation Fall 2018 A SIMPLE LINEAR MODEL Suppose that we observe explanatory variables x 1, x 2,..., x n and dependent variables y 1,
More informationGibbs Sampling in Linear Models #2
Gibbs Sampling in Linear Models #2 Econ 690 Purdue University Outline 1 Linear Regression Model with a Changepoint Example with Temperature Data 2 The Seemingly Unrelated Regressions Model 3 Gibbs sampling
More informationGAUSSIAN PROCESS REGRESSION
GAUSSIAN PROCESS REGRESSION CSE 515T Spring 2015 1. BACKGROUND The kernel trick again... The Kernel Trick Consider again the linear regression model: y(x) = φ(x) w + ε, with prior p(w) = N (w; 0, Σ). The
More informationClassical and Bayesian inference
Classical and Bayesian inference AMS 132 January 18, 2018 Claudia Wehrhahn (UCSC) Classical and Bayesian inference January 18, 2018 1 / 9 Sampling from a Bernoulli Distribution Theorem (Beta-Bernoulli
More informationPrinciples of Bayesian Inference
Principles of Bayesian Inference Sudipto Banerjee and Andrew O. Finley 2 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department
More informationCross-sectional space-time modeling using ARNN(p, n) processes
Cross-sectional space-time modeling using ARNN(p, n) processes W. Polasek K. Kakamu September, 006 Abstract We suggest a new class of cross-sectional space-time models based on local AR models and nearest
More informationA few basics of credibility theory
A few basics of credibility theory Greg Taylor Director, Taylor Fry Consulting Actuaries Professorial Associate, University of Melbourne Adjunct Professor, University of New South Wales General credibility
More informationPenalized Loss functions for Bayesian Model Choice
Penalized Loss functions for Bayesian Model Choice Martyn International Agency for Research on Cancer Lyon, France 13 November 2009 The pure approach For a Bayesian purist, all uncertainty is represented
More informationA Bayesian perspective on GMM and IV
A Bayesian perspective on GMM and IV Christopher A. Sims Princeton University sims@princeton.edu November 26, 2013 What is a Bayesian perspective? A Bayesian perspective on scientific reporting views all
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
More informationwhere x and ȳ are the sample means of x 1,, x n
y y Animal Studies of Side Effects Simple Linear Regression Basic Ideas In simple linear regression there is an approximately linear relation between two variables say y = pressure in the pancreas x =
More informationThe Bayesian approach to inverse problems
The Bayesian approach to inverse problems Youssef Marzouk Department of Aeronautics and Astronautics Center for Computational Engineering Massachusetts Institute of Technology ymarz@mit.edu, http://uqgroup.mit.edu
More informationPartial factor modeling: predictor-dependent shrinkage for linear regression
modeling: predictor-dependent shrinkage for linear Richard Hahn, Carlos Carvalho and Sayan Mukherjee JASA 2013 Review by Esther Salazar Duke University December, 2013 Factor framework The factor framework
More informationHierarchical Models & Bayesian Model Selection
Hierarchical Models & Bayesian Model Selection Geoffrey Roeder Departments of Computer Science and Statistics University of British Columbia Jan. 20, 2016 Contact information Please report any typos or
More informationBayesian Interpretations of Regularization
Bayesian Interpretations of Regularization Charlie Frogner 9.50 Class 15 April 1, 009 The Plan Regularized least squares maps {(x i, y i )} n i=1 to a function that minimizes the regularized loss: f S
More informationBayesian inference. Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark. April 10, 2017
Bayesian inference Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark April 10, 2017 1 / 22 Outline for today A genetic example Bayes theorem Examples Priors Posterior summaries
More information2 Bayesian Hierarchical Response Modeling
2 Bayesian Hierarchical Response Modeling In the first chapter, an introduction to Bayesian item response modeling was given. The Bayesian methodology requires careful specification of priors since item
More informationParameter Estimation. William H. Jefferys University of Texas at Austin Parameter Estimation 7/26/05 1
Parameter Estimation William H. Jefferys University of Texas at Austin bill@bayesrules.net Parameter Estimation 7/26/05 1 Elements of Inference Inference problems contain two indispensable elements: Data
More information1 Mixed effect models and longitudinal data analysis
1 Mixed effect models and longitudinal data analysis Mixed effects models provide a flexible approach to any situation where data have a grouping structure which introduces some kind of correlation between
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationLecture 4: Dynamic models
linear s Lecture 4: s Hedibert Freitas Lopes The University of Chicago Booth School of Business 5807 South Woodlawn Avenue, Chicago, IL 60637 http://faculty.chicagobooth.edu/hedibert.lopes hlopes@chicagobooth.edu
More informationSTA 2201/442 Assignment 2
STA 2201/442 Assignment 2 1. This is about how to simulate from a continuous univariate distribution. Let the random variable X have a continuous distribution with density f X (x) and cumulative distribution
More informationPart 4: Multi-parameter and normal models
Part 4: Multi-parameter and normal models 1 The normal model Perhaps the most useful (or utilized) probability model for data analysis is the normal distribution There are several reasons for this, e.g.,
More informationBayesian Inference for Normal Mean
Al Nosedal. University of Toronto. November 18, 2015 Likelihood of Single Observation The conditional observation distribution of y µ is Normal with mean µ and variance σ 2, which is known. Its density
More informationPart IB Statistics. Theorems with proof. Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua. Lent 2015
Part IB Statistics Theorems with proof Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly)
More informationIntegrated Non-Factorized Variational Inference
Integrated Non-Factorized Variational Inference Shaobo Han, Xuejun Liao and Lawrence Carin Duke University February 27, 2014 S. Han et al. Integrated Non-Factorized Variational Inference February 27, 2014
More informationModels for spatial data (cont d) Types of spatial data. Types of spatial data (cont d) Hierarchical models for spatial data
Hierarchical models for spatial data Based on the book by Banerjee, Carlin and Gelfand Hierarchical Modeling and Analysis for Spatial Data, 2004. We focus on Chapters 1, 2 and 5. Geo-referenced data arise
More informationINTRODUCTION TO BAYESIAN STATISTICS
INTRODUCTION TO BAYESIAN STATISTICS Sarat C. Dass Department of Statistics & Probability Department of Computer Science & Engineering Michigan State University TOPICS The Bayesian Framework Different Types
More informationRecall that the AR(p) model is defined by the equation
Estimation of AR models Recall that the AR(p) model is defined by the equation X t = p φ j X t j + ɛ t j=1 where ɛ t are assumed independent and following a N(0, σ 2 ) distribution. Assume p is known and
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationLecture 2: Linear Models. Bruce Walsh lecture notes Seattle SISG -Mixed Model Course version 23 June 2011
Lecture 2: Linear Models Bruce Walsh lecture notes Seattle SISG -Mixed Model Course version 23 June 2011 1 Quick Review of the Major Points The general linear model can be written as y = X! + e y = vector
More informationBayesian Model Comparison:
Bayesian Model Comparison: Modeling Petrobrás log-returns Hedibert Freitas Lopes February 2014 Log price: y t = log p t Time span: 12/29/2000-12/31/2013 (n = 3268 days) LOG PRICE 1 2 3 4 0 500 1000 1500
More informationSTA414/2104 Statistical Methods for Machine Learning II
STA414/2104 Statistical Methods for Machine Learning II Murat A. Erdogdu & David Duvenaud Department of Computer Science Department of Statistical Sciences Lecture 3 Slide credits: Russ Salakhutdinov Announcements
More informationAreal data models. Spatial smoothers. Brook s Lemma and Gibbs distribution. CAR models Gaussian case Non-Gaussian case
Areal data models Spatial smoothers Brook s Lemma and Gibbs distribution CAR models Gaussian case Non-Gaussian case SAR models Gaussian case Non-Gaussian case CAR vs. SAR STAR models Inference for areal
More informationSubjective and Objective Bayesian Statistics
Subjective and Objective Bayesian Statistics Principles, Models, and Applications Second Edition S. JAMES PRESS with contributions by SIDDHARTHA CHIB MERLISE CLYDE GEORGE WOODWORTH ALAN ZASLAVSKY \WILEY-
More informationPh.D. Qualifying Exam Friday Saturday, January 6 7, 2017
Ph.D. Qualifying Exam Friday Saturday, January 6 7, 2017 Put your solution to each problem on a separate sheet of paper. Problem 1. (5106) Let X 1, X 2,, X n be a sequence of i.i.d. observations from a
More informationHypothesis Testing. Econ 690. Purdue University. Justin L. Tobias (Purdue) Testing 1 / 33
Hypothesis Testing Econ 690 Purdue University Justin L. Tobias (Purdue) Testing 1 / 33 Outline 1 Basic Testing Framework 2 Testing with HPD intervals 3 Example 4 Savage Dickey Density Ratio 5 Bartlett
More informationNuisance parameters and their treatment
BS2 Statistical Inference, Lecture 2, Hilary Term 2008 April 2, 2008 Ancillarity Inference principles Completeness A statistic A = a(x ) is said to be ancillary if (i) The distribution of A does not depend
More informationMFM Practitioner Module: Risk & Asset Allocation. John Dodson. February 3, 2010
MFM Practitioner Module: Risk & Asset Allocation Estimator February 3, 2010 Estimator Estimator In estimation we do not endow the sample with a characterization; rather, we endow the parameters with a
More informationBayesian methods in economics and finance
1/26 Bayesian methods in economics and finance Linear regression: Bayesian model selection and sparsity priors Linear Regression 2/26 Linear regression Model for relationship between (several) independent
More informationContents. Part I: Fundamentals of Bayesian Inference 1
Contents Preface xiii Part I: Fundamentals of Bayesian Inference 1 1 Probability and inference 3 1.1 The three steps of Bayesian data analysis 3 1.2 General notation for statistical inference 4 1.3 Bayesian
More informationStatistical Inference and Methods
Department of Mathematics Imperial College London d.stephens@imperial.ac.uk http://stats.ma.ic.ac.uk/ das01/ 31st January 2006 Part VI Session 6: Filtering and Time to Event Data Session 6: Filtering and
More informationg-priors for Linear Regression
Stat60: Bayesian Modeling and Inference Lecture Date: March 15, 010 g-priors for Linear Regression Lecturer: Michael I. Jordan Scribe: Andrew H. Chan 1 Linear regression and g-priors In the last lecture,
More informationExample using R: Heart Valves Study
Example using R: Heart Valves Study Goal: Show that the thrombogenicity rate (TR) is less than two times the objective performance criterion R and WinBUGS Examples p. 1/27 Example using R: Heart Valves
More informationMultivariate Survival Analysis
Multivariate Survival Analysis Previously we have assumed that either (X i, δ i ) or (X i, δ i, Z i ), i = 1,..., n, are i.i.d.. This may not always be the case. Multivariate survival data can arise in
More informationA Very Brief Summary of Bayesian Inference, and Examples
A Very Brief Summary of Bayesian Inference, and Examples Trinity Term 009 Prof Gesine Reinert Our starting point are data x = x 1, x,, x n, which we view as realisations of random variables X 1, X,, X
More informationA short introduction to INLA and R-INLA
A short introduction to INLA and R-INLA Integrated Nested Laplace Approximation Thomas Opitz, BioSP, INRA Avignon Workshop: Theory and practice of INLA and SPDE November 7, 2018 2/21 Plan for this talk
More informationCentral Limit Theorem ( 5.3)
Central Limit Theorem ( 5.3) Let X 1, X 2,... be a sequence of independent random variables, each having n mean µ and variance σ 2. Then the distribution of the partial sum S n = X i i=1 becomes approximately
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More informationLinear Dynamical Systems
Linear Dynamical Systems Sargur N. srihari@cedar.buffalo.edu Machine Learning Course: http://www.cedar.buffalo.edu/~srihari/cse574/index.html Two Models Described by Same Graph Latent variables Observations
More information