Peter Hoff Linear and multilinear models April 3, GLS for multivariate regression 5. 3 Covariance estimation for the GLM 8
|
|
- Rachel Stephens
- 5 years ago
- Views:
Transcription
1 Contents 1 Linear model 1 2 GLS for multivariate regression 5 3 Covariance estimation for the GLM 8 4 Testing the GLH 11 A reference for some of this material can be found somewhere. 1 Linear model Recall that the linear model is y = Xβ + Σ 1/2 z E[z] = 0 Var[z] = I, where X R n p, β R p and Σ S n +. The term linear here typically refers to the mean parameters: The expectation of any element y i of y is linear in the parameters. If the parameter space for Σ is unrestricted, then the covariance model is linear as as well. If the covariance is restricted, 1
2 say Σ = Σ(θ) for θ Θ, then typically the covariance model is nonlinear in its parameters (e.g. autoregressive covariance). A surprising number of other models can be viewed as submodels of the linear model: 1. Row and column effects models: Let y i,j = µ + a i + b j + σ i ψ j z i,j Y = µ11 + a1 + 1b + Σ 1/2 ZΨ 1/2 y = 1µ + (1 I)a + (I 1)b + (Ψ Σ) 1/2 z µ = [1 (1 I) (I 1)] a + Ψ Σ) 1/2 z. b 2. Random effects models: 3. Independent multivariate sample: Let y 1,..., y n i.i.d. N p (µ, Σ). Then Y = 1µ + ZΣ 1/2 y = (I 1)µ + (Σ 1/2 I)z 4. General linear model: Let y i N(B x i, Σ), independently for i = 1,..., n. Then Y = XB + ZΣ 1/2 y = (I X)b + (Σ 1/2 I)z 5. Correlated rows: Suppose Cov[Y [i,] ] Σ, Cov[Y [,j] ] Ψ. Then Y = XB + Ψ 1/2 ZΣ 1/2 y = (I X)b + (Σ 1/2 Ψ 1/2 )z. 2
3 6. Row and column covariates: Suppose E[y i,j ] = x i Bw j, for example, Y = XBW + Ψ 1/2 ZΣ 1/2 y = (W X)b + (Σ 1/2 Ψ 1/2 )z. So we begin by studying the deceptively simple general form of the linear model. OLS: For a given value of β, the RSS is given by RSS(β) = (y Xβ) (y Xβ). Differentiation or completing the square shows that the minimizer in β is ˆβ = (X X) 1 X y. It is straightforward to show that the OLS estimator is unbiased, E[ˆβ] = β, and that the variance of this estimator is Var[ˆβ] = (X X) 1 (X ΣX )(X X) 1. If Σ = σ 2 I then the OLS estimator is optimal in terms of variance: Theorem 1 (Gauss-Markov). Suppose Var[y] = σ 2 I and let ˇβ = Ay be an unbiased estimator for β. Then Var[ˇβ] = Var[ˆβ] + E[(ˇβ ˆβ)(ˇβ ˆβ) ] If Σ σ 2 I then the OLS estimator does not have minimum variance. 3
4 Exercise 1. Obtain the form of β, ŷ = Xβ, and Var[ˆβ] in terms of components of the SVD of X. Comment on the following intuitive statement: If y Xβ then X 1 y β. GLS: If we knew Σ we could obtain a decorrelated vector ỹ = Σ 1/2 y. The model for ỹ is ỹ = Σ 1/2 y = Σ 1/2 Xβ + Σ 1/2 Σ 1/2 z ỹ = Xβ + z. The OLS estimator based on the decorrelated data and corresponding regressors is ˆβ OLS = ( X X) 1 X ỹ. By the Gauss-Markov theorem, this estimator is optimal among all linear unbiased estimators, that is unbiased estimators of the form Aỹ. Obviously, this includes unbiased estimators of the form Ay, since ỹ is a linear transformation of y. What is this estimator in terms of the original data and covariates? ˆβ GLS = ˆβ OLS = ( X X) 1 X ỹ = (X Σ 1 X) 1 X Σ 1 y. The variance of this estimator is easily shown to be Var[ˆβ GLS ] = (XΣ 1 X) 1. This is the smallest variance attainable among unbiased linear estimators, since the GLS estimate is the BLUE - best linear unbiased estimator. 4
5 Furthermore, suppose we assume normal errors, z N n (0, I). For known Σ, the MLE of β is the minimizer of 2 log p(y X, β) = (y Xβ) Σ 1 (y Xβ). Taking derivatives, the MLE is the GLS estimator. Big picture: If y N n (Xβ, Σ) and Σ is known, ˆβ GLS = (X Σ 1 X) 1 X Σ 1 y is the GLS estimator; the ML estimator; the OLS estimator based on the decorrelated data Σ 1/2 y; the BLUE based on y. Now the problem is that Σ is rarely known. In some cases, particularly if Σ = Σ(θ) for some low dimensional parameter θ, we can jointly estimate Σ and β. This is known as feasible GLS. 2 GLS for multivariate regression Perhaps the most commonly used model is multivariate regression, a.k.a. the general linear model: Y = XB + ZΣ 1/2 y = (I X)b + (Σ 1/2 I)z 5
6 GLS: Using the vector form for the model, the GLS estimator of b = vec(b) is ˆB GLS = ( (I X) (Σ I) 1 (I X) ) 1 ( (I X) (Σ 1 I)y ) = (Σ 1 X X) 1 (Σ 1 X )y = (I (X X) 1 X )y Now unvectorize : ˆB GLS = (I (X X) 1 X )y = (X X) 1 X Y. Notice that Σ does not appear in formula for the GLS estimator. about why this is: Think The multivariate regression model is just a regression model for each variable: y j = Xb j + e j So the jth column of B R q p is b j,the vector of regression coefficients for the jth variable. In this model, there is no correlation among the elements of y j /rows of Y. We already knew that in this case the BLUE for b j based on y j is the OLS estimator: ˆβ j = (X X) 1 X y j. Now is this BLUE among estimators based on Y? The question comes down to whether or not observing some variables Y [, j] correlated with y j helps with estimating b j. The answer given by the above calculation is no. 6
7 Binding the OLS estimators columnwise gives the multivariate OLS/GLS estimator: ˆB = [ˆβ1 ˆβ p ] To summarize estimator: ˆBGLS = (X X) 1 X Y is the OLS estimator; the BLUE; the MLE under normal errors. the column-bound univariate OLS estimators. What is the covariance of ˆB? How do we even write this out, since ˆB is a matrix? One approach is to vectorize. Recall the vector form for the GLM: Y = XB + ZΣ 1/2 y = (I X)b + (Σ 1/2 I)z = Xb + Σ 1/2 z Now we already calculated the variance of a GLS estimator: Var[ ˆB] Var[ˆb] = ( X Σ 1 X) 1 = ( (I X )(Σ 1 I)(I X) ) 1 = Σ (X X) 1. = ( (Σ 1 X X) 1) This means that Cov[ ˆB [,j], ˆB [,j ]] = σ j,j (X X) 1 Cov[ ˆB [k,], ˆB [k,]] = s k,k Σ where s k,k = (X X) 1 k,k. In particular, Cov[ˆb k,j, ˆb k,j ] = σ j,j s k,k. So in a sense, Σ is the column covariance of ˆB, and (X X) 1 is the row covariance. 7
8 3 Covariance estimation for the GLM Since the OLS/GLS estimator of B is (X X) 1 X Y, the fitted values for Y are Ŷ = X ˆB = X(X X) 1 X Y = P X Y. So the fitted values of the matrix Y are just the column-wise projections of Y on to the space spanned by the columns of X. Obvious but important is that ˆB itself is a function of Ŷ: ˆB = (X X) 1 X Y = (X X) 1 (X X)(X X) 1 X = (X X) 1 X ( X(X X) 1 X ) Y = (X X) 1 X P X Y = (X X) 1 X Ŷ The residuals are the projection onto the null space of X Ê = Y Ŷ = (I P X )Y = (I X(X X) 1 X)Y = P 0 Y. This gives the decomposition Y = Ŷ + Ê. 8
9 Is Ŷ correlated with Ê? Let s find out. Let ŷ and ê be the vectorizations of Ŷ and Ê respectively. The covariance between these two matrices is the covariance between their vectorizations. Cov[ŷ, ê] = E[(ŷ E[ŷ])ê ] Now what are these items? Ŷ E[Ŷ] = P XY E[P X Y] = P X (Y E[Y]) = P X E, ŷ E[ŷ] = (I P X )e. Ê = P 0 Y = P 0 (XB + E) = P 0 E, ê = (I P 0 )e. Therefore, Cov[ŷ, ê] = E[(ŷ E[ŷ])ê ] = E[(I P X )ee (I P 0 )] = 0. So as with ordinary regression, the residuals are uncorrelated with the fitted values, and hence also uncorrelated with ˆB. This is useful, as it allows us to obtain an estimate of Σ that is independent of our estimate of B if the errors are normal. 9
10 Ok, let s consider estimating Σ from Ê. Now Ê = P 0E, and under the model E = ZΣ 1/2. Therefore, Ê = P 0ZΣ 1/2, and S = Ê Ê = Σ 1/2 Z P 0 P 0 ZΣ 1/2 E[S] = Σ 1/2 E[Z P 0 Z]Σ 1/2 = Σ 1/2 (tr(p 0 )I)Σ 1/2 = Σ (n q) Therefore, ˆΣ = Ê Ê/(n q) is an unbiased estimator of Σ. However, it is not the MLE: Exercise 2. Show that the MLE of (B, Σ) is ( ˆB, S/n). If the errors are Gaussian then we can say more. Suppose Y = XB + ZΣ 1/2 Z N n p (0, I I), or equivalently, Y N n p (XB, Σ I). Then Ê = P 0 ZΣ 1/2. Now recall from a few months ago the following: P 0 is a rank-(n q) idempotent projection matrix; P 0 = GG T for some G V n q,n R n (n q) ; If Z N(0, I I) then G Z N n q (0, I I). 10
11 So we have S = Ê Ê = Σ 1/2 Z P 0 P 0 ZΣ 1/2 = Σ 1/2 Z P 0 ZΣ 1/2 = Σ 1/2 Z GG ZΣ 1/2 = W W, where W = G ZΣ 1/2 N (n q) p (0, Σ). Therefore S = Ê Ê = W W Wishart (n q) (Σ). We ve seen a simpler version of this before: Suppose Y N n p (1µ, Σ I). Although it doesn t look like it, this is a multivariate linear regression model where the regression model for each column j is y j = 1µ j + e j. That is there is q = 1 predictor and it is constant for all rows : X = 1 and B = µ ; P X = 1(1 1) 1 1 = 1 /n; P 0 = I 11 /n = C; S = Ê Ê = Y CC Wishart n 1 (Σ). 4 Testing the GLH We now derive some classical testing and confidence procedures for the regression matrix B of the Gaussian GLM: Y N(XB, Σ I). Some results we ve obtained so far: 11
12 ˆB = (X X) 1 X Y N q p (B, Σ (X X) 1 ); S = Ê Ê Wishart (n q) (Σ); ˆB and Ê are uncorrelated and so independent. Consider first inference for B [k,] = b k, the (transposed) kth row of the matrix B. This p 1 vector represents the effects of the kth predictor X [,k] on Y: Y = XB + E = q X [,q] b k + E. k=1 In the context of making inference about b j, our results above tell us that ˆb j N p (b j, h jj Σ); ˆb j is independent of S. Our objective is to make a confidence region or test a hypothesis about b j based on ˆb j. Recall that a particularly simple case of this is when q = 1 and X = 1. This is just the multivariate normal model Y N n p (1µ, Σ I). Recall our results, both old and new, imply that ȳ N(µ, Σ/n) ȳ is independent of S. The point is that a statistical method developed to infer µ from (ȳ, S) can be used to infer b j from (ˆb j, S). So we proceed with the former since the notation is simpler. 12
13 Suppose Σ were known and we wanted to test H : E[y] = µ. We then might construct a test statistic as follows: y N(µ, Σ/n) n(y µ) N(0, Σ) (µ) = nσ 1/2 (y µ) N(0, I) z 2 (µ) = n(ȳ µ) Σ 1 (ȳ µ) χ 2 p. So in this case the null distribution is χ 2 p, and of course does not depend on the parameters. In this case that Σ is known, this statistic is a pivot statistic, that is, has a distribution that doesn t depend on any unknown parameters. We use such statistics to test and infer values of µ Since Σ is unknown we might try replacing it with ˆΣ the unbiased estimator: t 2 (ȳ, ˆΣ) = n(ȳ µ) Σ 1 (ȳ µ). Can we use this for testing and confidence region construction? To do so requires the following: show that t 2 is pivotal, that is, doesn t depend on unknown parameters; derive the distribution of t 2. 13
STAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini June 9, 2018 1 / 56 Table of Contents Multiple linear regression Linear model setup Estimation of β Geometric interpretation Estimation of σ 2 Hat matrix Gram matrix
More informationLinear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
More informationMa 3/103: Lecture 24 Linear Regression I: Estimation
Ma 3/103: Lecture 24 Linear Regression I: Estimation March 3, 2017 KC Border Linear Regression I March 3, 2017 1 / 32 Regression analysis Regression analysis Estimate and test E(Y X) = f (X). f is the
More informationIntroduction to Estimation Methods for Time Series models. Lecture 1
Introduction to Estimation Methods for Time Series models Lecture 1 Fulvio Corsi SNS Pisa Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 1 / 19 Estimation
More informationMultivariate Regression Analysis
Matrices and vectors The model from the sample is: Y = Xβ +u with n individuals, l response variable, k regressors Y is a n 1 vector or a n l matrix with the notation Y T = (y 1,y 2,...,y n ) 1 x 11 x
More informationSTAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method.
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method. Rebecca Barter May 5, 2015 Linear Regression Review Linear Regression Review
More informationProperties of the least squares estimates
Properties of the least squares estimates 2019-01-18 Warmup Let a and b be scalar constants, and X be a scalar random variable. Fill in the blanks E ax + b) = Var ax + b) = Goal Recall that the least squares
More informationSummer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.
Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall
More informationLecture 13: Simple Linear Regression in Matrix Format. 1 Expectations and Variances with Vectors and Matrices
Lecture 3: Simple Linear Regression in Matrix Format To move beyond simple regression we need to use matrix algebra We ll start by re-expressing simple linear regression in matrix form Linear algebra is
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7
MA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7 1 Random Vectors Let a 0 and y be n 1 vectors, and let A be an n n matrix. Here, a 0 and A are non-random, whereas y is
More information2. A Review of Some Key Linear Models Results. Copyright c 2018 Dan Nettleton (Iowa State University) 2. Statistics / 28
2. A Review of Some Key Linear Models Results Copyright c 2018 Dan Nettleton (Iowa State University) 2. Statistics 510 1 / 28 A General Linear Model (GLM) Suppose y = Xβ + ɛ, where y R n is the response
More informationData Mining Stat 588
Data Mining Stat 588 Lecture 02: Linear Methods for Regression Department of Statistics & Biostatistics Rutgers University September 13 2011 Regression Problem Quantitative generic output variable Y. Generic
More informationEconometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018
Econometrics I KS Module 2: Multivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: April 16, 2018 Alexander Ahammer (JKU) Module 2: Multivariate
More informationInverse of a Square Matrix. For an N N square matrix A, the inverse of A, 1
Inverse of a Square Matrix For an N N square matrix A, the inverse of A, 1 A, exists if and only if A is of full rank, i.e., if and only if no column of A is a linear combination 1 of the others. A is
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationST 740: Linear Models and Multivariate Normal Inference
ST 740: Linear Models and Multivariate Normal Inference Alyson Wilson Department of Statistics North Carolina State University November 4, 2013 A. Wilson (NCSU STAT) Linear Models November 4, 2013 1 /
More informationRegression Review. Statistics 149. Spring Copyright c 2006 by Mark E. Irwin
Regression Review Statistics 149 Spring 2006 Copyright c 2006 by Mark E. Irwin Matrix Approach to Regression Linear Model: Y i = β 0 + β 1 X i1 +... + β p X ip + ɛ i ; ɛ i iid N(0, σ 2 ), i = 1,..., n
More information3 Multiple Linear Regression
3 Multiple Linear Regression 3.1 The Model Essentially, all models are wrong, but some are useful. Quote by George E.P. Box. Models are supposed to be exact descriptions of the population, but that is
More informationQuick Review on Linear Multiple Regression
Quick Review on Linear Multiple Regression Mei-Yuan Chen Department of Finance National Chung Hsing University March 6, 2007 Introduction for Conditional Mean Modeling Suppose random variables Y, X 1,
More informationIEOR 165 Lecture 7 1 Bias-Variance Tradeoff
IEOR 165 Lecture 7 Bias-Variance Tradeoff 1 Bias-Variance Tradeoff Consider the case of parametric regression with β R, and suppose we would like to analyze the error of the estimate ˆβ in comparison to
More informationReview of Classical Least Squares. James L. Powell Department of Economics University of California, Berkeley
Review of Classical Least Squares James L. Powell Department of Economics University of California, Berkeley The Classical Linear Model The object of least squares regression methods is to model and estimate
More informationLecture 15. Hypothesis testing in the linear model
14. Lecture 15. Hypothesis testing in the linear model Lecture 15. Hypothesis testing in the linear model 1 (1 1) Preliminary lemma 15. Hypothesis testing in the linear model 15.1. Preliminary lemma Lemma
More informationHomoskedasticity. Var (u X) = σ 2. (23)
Homoskedasticity How big is the difference between the OLS estimator and the true parameter? To answer this question, we make an additional assumption called homoskedasticity: Var (u X) = σ 2. (23) This
More informationSTAT5044: Regression and Anova. Inyoung Kim
STAT5044: Regression and Anova Inyoung Kim 2 / 51 Outline 1 Matrix Expression 2 Linear and quadratic forms 3 Properties of quadratic form 4 Properties of estimates 5 Distributional properties 3 / 51 Matrix
More informationSimple Linear Regression
Simple Linear Regression ST 430/514 Recall: A regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates)
More informationMaximum Likelihood Estimation
Maximum Likelihood Estimation Merlise Clyde STA721 Linear Models Duke University August 31, 2017 Outline Topics Likelihood Function Projections Maximum Likelihood Estimates Readings: Christensen Chapter
More informationLinear models. Linear models are computationally convenient and remain widely used in. applied econometric research
Linear models Linear models are computationally convenient and remain widely used in applied econometric research Our main focus in these lectures will be on single equation linear models of the form y
More informationLecture 3: Linear Models. Bruce Walsh lecture notes Uppsala EQG course version 28 Jan 2012
Lecture 3: Linear Models Bruce Walsh lecture notes Uppsala EQG course version 28 Jan 2012 1 Quick Review of the Major Points The general linear model can be written as y = X! + e y = vector of observed
More informationMIT Spring 2015
Regression Analysis MIT 18.472 Dr. Kempthorne Spring 2015 1 Outline Regression Analysis 1 Regression Analysis 2 Multiple Linear Regression: Setup Data Set n cases i = 1, 2,..., n 1 Response (dependent)
More informationLinear Model Under General Variance
Linear Model Under General Variance We have a sample of T random variables y 1, y 2,, y T, satisfying the linear model Y = X β + e, where Y = (y 1,, y T )' is a (T 1) vector of random variables, X = (T
More informationLecture 2: Linear Models. Bruce Walsh lecture notes Seattle SISG -Mixed Model Course version 23 June 2011
Lecture 2: Linear Models Bruce Walsh lecture notes Seattle SISG -Mixed Model Course version 23 June 2011 1 Quick Review of the Major Points The general linear model can be written as y = X! + e y = vector
More informationLECTURE 2 LINEAR REGRESSION MODEL AND OLS
SEPTEMBER 29, 2014 LECTURE 2 LINEAR REGRESSION MODEL AND OLS Definitions A common question in econometrics is to study the effect of one group of variables X i, usually called the regressors, on another
More informationThis model of the conditional expectation is linear in the parameters. A more practical and relaxed attitude towards linear regression is to say that
Linear Regression For (X, Y ) a pair of random variables with values in R p R we assume that E(Y X) = β 0 + with β R p+1. p X j β j = (1, X T )β j=1 This model of the conditional expectation is linear
More information[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
More informationSTA 302f16 Assignment Five 1
STA 30f16 Assignment Five 1 Except for Problem??, these problems are preparation for the quiz in tutorial on Thursday October 0th, and are not to be handed in As usual, at times you may be asked to prove
More informationAssociation studies and regression
Association studies and regression CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar Association studies and regression 1 / 104 Administration
More informationChapter 5 Matrix Approach to Simple Linear Regression
STAT 525 SPRING 2018 Chapter 5 Matrix Approach to Simple Linear Regression Professor Min Zhang Matrix Collection of elements arranged in rows and columns Elements will be numbers or symbols For example:
More information3. For a given dataset and linear model, what do you think is true about least squares estimates? Is Ŷ always unique? Yes. Is ˆβ always unique? No.
7. LEAST SQUARES ESTIMATION 1 EXERCISE: Least-Squares Estimation and Uniqueness of Estimates 1. For n real numbers a 1,...,a n, what value of a minimizes the sum of squared distances from a to each of
More informationProbability and Statistics Notes
Probability and Statistics Notes Chapter Seven Jesse Crawford Department of Mathematics Tarleton State University Spring 2011 (Tarleton State University) Chapter Seven Notes Spring 2011 1 / 42 Outline
More informationWeighted Least Squares
Weighted Least Squares The standard linear model assumes that Var(ε i ) = σ 2 for i = 1,..., n. As we have seen, however, there are instances where Var(Y X = x i ) = Var(ε i ) = σ2 w i. Here w 1,..., w
More informationRandom vectors X 1 X 2. Recall that a random vector X = is made up of, say, k. X k. random variables.
Random vectors Recall that a random vector X = X X 2 is made up of, say, k random variables X k A random vector has a joint distribution, eg a density f(x), that gives probabilities P(X A) = f(x)dx Just
More informationStatement: With my signature I confirm that the solutions are the product of my own work. Name: Signature:.
MATHEMATICAL STATISTICS Homework assignment Instructions Please turn in the homework with this cover page. You do not need to edit the solutions. Just make sure the handwriting is legible. You may discuss
More informationLinear Algebra Review
Linear Algebra Review Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Linear Algebra Review 1 / 45 Definition of Matrix Rectangular array of elements arranged in rows and
More informationStatistics 910, #15 1. Kalman Filter
Statistics 910, #15 1 Overview 1. Summary of Kalman filter 2. Derivations 3. ARMA likelihoods 4. Recursions for the variance Kalman Filter Summary of Kalman filter Simplifications To make the derivations
More informationMultiple Linear Regression
Multiple Linear Regression Simple linear regression tries to fit a simple line between two variables Y and X. If X is linearly related to Y this explains some of the variability in Y. In most cases, there
More informationSTAT 540: Data Analysis and Regression
STAT 540: Data Analysis and Regression Wen Zhou http://www.stat.colostate.edu/~riczw/ Email: riczw@stat.colostate.edu Department of Statistics Colorado State University Fall 205 W. Zhou (Colorado State
More informationCh 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
More informationReview of Econometrics
Review of Econometrics Zheng Tian June 5th, 2017 1 The Essence of the OLS Estimation Multiple regression model involves the models as follows Y i = β 0 + β 1 X 1i + β 2 X 2i + + β k X ki + u i, i = 1,...,
More informationXβ is a linear combination of the columns of X: Copyright c 2010 Dan Nettleton (Iowa State University) Statistics / 25 X =
The Gauss-Markov Linear Model y Xβ + ɛ y is an n random vector of responses X is an n p matrix of constants with columns corresponding to explanatory variables X is sometimes referred to as the design
More informationMatrix Approach to Simple Linear Regression: An Overview
Matrix Approach to Simple Linear Regression: An Overview Aspects of matrices that you should know: Definition of a matrix Addition/subtraction/multiplication of matrices Symmetric/diagonal/identity matrix
More informationRestricted Maximum Likelihood in Linear Regression and Linear Mixed-Effects Model
Restricted Maximum Likelihood in Linear Regression and Linear Mixed-Effects Model Xiuming Zhang zhangxiuming@u.nus.edu A*STAR-NUS Clinical Imaging Research Center October, 015 Summary This report derives
More informationBias Variance Trade-off
Bias Variance Trade-off The mean squared error of an estimator MSE(ˆθ) = E([ˆθ θ] 2 ) Can be re-expressed MSE(ˆθ) = Var(ˆθ) + (B(ˆθ) 2 ) MSE = VAR + BIAS 2 Proof MSE(ˆθ) = E((ˆθ θ) 2 ) = E(([ˆθ E(ˆθ)]
More informationMatrices and vectors A matrix is a rectangular array of numbers. Here s an example: A =
Matrices and vectors A matrix is a rectangular array of numbers Here s an example: 23 14 17 A = 225 0 2 This matrix has dimensions 2 3 The number of rows is first, then the number of columns We can write
More informationMultivariate Regression
Multivariate Regression The so-called supervised learning problem is the following: we want to approximate the random variable Y with an appropriate function of the random variables X 1,..., X p with the
More informationL2: Two-variable regression model
L2: Two-variable regression model Feng Li feng.li@cufe.edu.cn School of Statistics and Mathematics Central University of Finance and Economics Revision: September 4, 2014 What we have learned last time...
More informationStatistics 910, #5 1. Regression Methods
Statistics 910, #5 1 Overview Regression Methods 1. Idea: effects of dependence 2. Examples of estimation (in R) 3. Review of regression 4. Comparisons and relative efficiencies Idea Decomposition Well-known
More informationMLES & Multivariate Normal Theory
Merlise Clyde September 6, 2016 Outline Expectations of Quadratic Forms Distribution Linear Transformations Distribution of estimates under normality Properties of MLE s Recap Ŷ = ˆµ is an unbiased estimate
More informationFENG CHIA UNIVERSITY ECONOMETRICS I: HOMEWORK 4. Prof. Mei-Yuan Chen Spring 2008
FENG CHIA UNIVERSITY ECONOMETRICS I: HOMEWORK 4 Prof. Mei-Yuan Chen Spring 008. Partition and rearrange the matrix X as [x i X i ]. That is, X i is the matrix X excluding the column x i. Let u i denote
More informationAdvanced Econometrics
Based on the textbook by Verbeek: A Guide to Modern Econometrics Robert M. Kunst robert.kunst@univie.ac.at University of Vienna and Institute for Advanced Studies Vienna May 16, 2013 Outline Univariate
More informationLinear Regression. September 27, Chapter 3. Chapter 3 September 27, / 77
Linear Regression Chapter 3 September 27, 2016 Chapter 3 September 27, 2016 1 / 77 1 3.1. Simple linear regression 2 3.2 Multiple linear regression 3 3.3. The least squares estimation 4 3.4. The statistical
More informationSensitivity of GLS estimators in random effects models
of GLS estimators in random effects models Andrey L. Vasnev (University of Sydney) Tokyo, August 4, 2009 1 / 19 Plan Plan Simulation studies and estimators 2 / 19 Simulation studies Plan Simulation studies
More informationLecture 11: Regression Methods I (Linear Regression)
Lecture 11: Regression Methods I (Linear Regression) Fall, 2017 1 / 40 Outline Linear Model Introduction 1 Regression: Supervised Learning with Continuous Responses 2 Linear Models and Multiple Linear
More informationGeneral Linear Model: Statistical Inference
Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter 4), least
More information1 Data Arrays and Decompositions
1 Data Arrays and Decompositions 1.1 Variance Matrices and Eigenstructure Consider a p p positive definite and symmetric matrix V - a model parameter or a sample variance matrix. The eigenstructure is
More information4 Multiple Linear Regression
4 Multiple Linear Regression 4. The Model Definition 4.. random variable Y fits a Multiple Linear Regression Model, iff there exist β, β,..., β k R so that for all (x, x 2,..., x k ) R k where ε N (, σ
More informationLecture 11: Regression Methods I (Linear Regression)
Lecture 11: Regression Methods I (Linear Regression) 1 / 43 Outline 1 Regression: Supervised Learning with Continuous Responses 2 Linear Models and Multiple Linear Regression Ordinary Least Squares Statistical
More informationTHE ANOVA APPROACH TO THE ANALYSIS OF LINEAR MIXED EFFECTS MODELS
THE ANOVA APPROACH TO THE ANALYSIS OF LINEAR MIXED EFFECTS MODELS We begin with a relatively simple special case. Suppose y ijk = µ + τ i + u ij + e ijk, (i = 1,..., t; j = 1,..., n; k = 1,..., m) β =
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) The Simple Linear Regression Model based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #2 The Simple
More informationBusiness Statistics. Tommaso Proietti. Linear Regression. DEF - Università di Roma 'Tor Vergata'
Business Statistics Tommaso Proietti DEF - Università di Roma 'Tor Vergata' Linear Regression Specication Let Y be a univariate quantitative response variable. We model Y as follows: Y = f(x) + ε where
More informationEstimation of the Response Mean. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 27
Estimation of the Response Mean Copyright c 202 Dan Nettleton (Iowa State University) Statistics 5 / 27 The Gauss-Markov Linear Model y = Xβ + ɛ y is an n random vector of responses. X is an n p matrix
More information2.1 Linear regression with matrices
21 Linear regression with matrices The values of the independent variables are united into the matrix X (design matrix), the values of the outcome and the coefficient are represented by the vectors Y and
More informationMotivation for multiple regression
Motivation for multiple regression 1. Simple regression puts all factors other than X in u, and treats them as unobserved. Effectively the simple regression does not account for other factors. 2. The slope
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini April 27, 2018 1 / 1 Table of Contents 2 / 1 Linear Algebra Review Read 3.1 and 3.2 from text. 1. Fundamental subspace (rank-nullity, etc.) Im(X ) = ker(x T ) R
More informationMatrix Factorizations
1 Stat 540, Matrix Factorizations Matrix Factorizations LU Factorization Definition... Given a square k k matrix S, the LU factorization (or decomposition) represents S as the product of two triangular
More informationLecture 6: Linear models and Gauss-Markov theorem
Lecture 6: Linear models and Gauss-Markov theorem Linear model setting Results in simple linear regression can be extended to the following general linear model with independently observed response variables
More informationLinear Models and Estimation by Least Squares
Linear Models and Estimation by Least Squares Jin-Lung Lin 1 Introduction Causal relation investigation lies in the heart of economics. Effect (Dependent variable) cause (Independent variable) Example:
More informationInference in Regression Analysis
Inference in Regression Analysis Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 4, Slide 1 Today: Normal Error Regression Model Y i = β 0 + β 1 X i + ǫ i Y i value
More informationQuantitative Analysis of Financial Markets. Summary of Part II. Key Concepts & Formulas. Christopher Ting. November 11, 2017
Summary of Part II Key Concepts & Formulas Christopher Ting November 11, 2017 christopherting@smu.edu.sg http://www.mysmu.edu/faculty/christophert/ Christopher Ting 1 of 16 Why Regression Analysis? Understand
More informationThe linear model is the most fundamental of all serious statistical models encompassing:
Linear Regression Models: A Bayesian perspective Ingredients of a linear model include an n 1 response vector y = (y 1,..., y n ) T and an n p design matrix (e.g. including regressors) X = [x 1,..., x
More informationEstimating Estimable Functions of β. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 17
Estimating Estimable Functions of β Copyright c 202 Dan Nettleton (Iowa State University) Statistics 5 / 7 The Response Depends on β Only through Xβ In the Gauss-Markov or Normal Theory Gauss-Markov Linear
More informationEcon 620. Matrix Differentiation. Let a and x are (k 1) vectors and A is an (k k) matrix. ) x. (a x) = a. x = a (x Ax) =(A + A (x Ax) x x =(A + A )
Econ 60 Matrix Differentiation Let a and x are k vectors and A is an k k matrix. a x a x = a = a x Ax =A + A x Ax x =A + A x Ax = xx A We don t want to prove the claim rigorously. But a x = k a i x i i=
More informationMeasuring the fit of the model - SSR
Measuring the fit of the model - SSR Once we ve determined our estimated regression line, we d like to know how well the model fits. How far/close are the observations to the fitted line? One way to do
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Mixed Effects Estimation, Residuals Diagnostics Week 11, Lecture 1
MA 575 Linear Models: Cedric E Ginestet, Boston University Mixed Effects Estimation, Residuals Diagnostics Week 11, Lecture 1 1 Within-group Correlation Let us recall the simple two-level hierarchical
More informationLecture 14 Simple Linear Regression
Lecture 4 Simple Linear Regression Ordinary Least Squares (OLS) Consider the following simple linear regression model where, for each unit i, Y i is the dependent variable (response). X i is the independent
More informationLecture 3: Multiple Regression
Lecture 3: Multiple Regression R.G. Pierse 1 The General Linear Model Suppose that we have k explanatory variables Y i = β 1 + β X i + β 3 X 3i + + β k X ki + u i, i = 1,, n (1.1) or Y i = β j X ji + u
More informationMS&E 226: Small Data. Lecture 11: Maximum likelihood (v2) Ramesh Johari
MS&E 226: Small Data Lecture 11: Maximum likelihood (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 18 The likelihood function 2 / 18 Estimating the parameter This lecture develops the methodology behind
More information11 Hypothesis Testing
28 11 Hypothesis Testing 111 Introduction Suppose we want to test the hypothesis: H : A q p β p 1 q 1 In terms of the rows of A this can be written as a 1 a q β, ie a i β for each row of A (here a i denotes
More informationRegression Analysis. y t = β 1 x t1 + β 2 x t2 + β k x tk + ϵ t, t = 1,..., T,
Regression Analysis The multiple linear regression model with k explanatory variables assumes that the tth observation of the dependent or endogenous variable y t is described by the linear relationship
More informationAdvanced Statistics I : Gaussian Linear Model (and beyond)
Advanced Statistics I : Gaussian Linear Model (and beyond) Aurélien Garivier CNRS / Telecom ParisTech Centrale Outline One and Two-Sample Statistics Linear Gaussian Model Model Reduction and model Selection
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationAdvanced Quantitative Methods: ordinary least squares
Advanced Quantitative Methods: Ordinary Least Squares University College Dublin 31 January 2012 1 2 3 4 5 Terminology y is the dependent variable referred to also (by Greene) as a regressand X are the
More informationBIOS 2083 Linear Models c Abdus S. Wahed
Chapter 5 206 Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter
More informationLikelihood Methods. 1 Likelihood Functions. The multivariate normal distribution likelihood function is
Likelihood Methods 1 Likelihood Functions The multivariate normal distribution likelihood function is The log of the likelihood, say L 1 is Ly = π.5n V.5 exp.5y Xb V 1 y Xb. L 1 = 0.5[N lnπ + ln V +y Xb
More informationEconometrics II - EXAM Answer each question in separate sheets in three hours
Econometrics II - EXAM Answer each question in separate sheets in three hours. Let u and u be jointly Gaussian and independent of z in all the equations. a Investigate the identification of the following
More informationMath 423/533: The Main Theoretical Topics
Math 423/533: The Main Theoretical Topics Notation sample size n, data index i number of predictors, p (p = 2 for simple linear regression) y i : response for individual i x i = (x i1,..., x ip ) (1 p)
More informationSCHOOL OF MATHEMATICS AND STATISTICS. Linear and Generalised Linear Models
SCHOOL OF MATHEMATICS AND STATISTICS Linear and Generalised Linear Models Autumn Semester 2017 18 2 hours Attempt all the questions. The allocation of marks is shown in brackets. RESTRICTED OPEN BOOK EXAMINATION
More informationRegression and Statistical Inference
Regression and Statistical Inference Walid Mnif wmnif@uwo.ca Department of Applied Mathematics The University of Western Ontario, London, Canada 1 Elements of Probability 2 Elements of Probability CDF&PDF
More informationLECTURE 5 HYPOTHESIS TESTING
October 25, 2016 LECTURE 5 HYPOTHESIS TESTING Basic concepts In this lecture we continue to discuss the normal classical linear regression defined by Assumptions A1-A5. Let θ Θ R d be a parameter of interest.
More informationEconometrics Multiple Regression Analysis: Heteroskedasticity
Econometrics Multiple Regression Analysis: João Valle e Azevedo Faculdade de Economia Universidade Nova de Lisboa Spring Semester João Valle e Azevedo (FEUNL) Econometrics Lisbon, April 2011 1 / 19 Properties
More information