STAT5044: Regression and Anova. Inyoung Kim
|
|
- Godfrey Welch
- 6 years ago
- Views:
Transcription
1 STAT5044: Regression and Anova Inyoung Kim
2 2 / 47 Outline 1 Regression 2 Simple Linear regression 3 Basic concepts in regression 4 How to estimate unknown parameters 5 Properties of Least Squares Estimators: Gauss-Markov theorem
3 3 / 47 Regression A way to model the relationship between dependent ( ) variable Y and independent ( ) variable X.
4 4 / 47 Regression A way to model the relationship between dependent ( ) variable Y and independent ( ) variable X. The goal of regression is to understand how the values of Y change as X is varied over its range of possible values.
5 5 / 47 Regression A way to model the relationship between dependent (response) variable Y and independent (explanatory) variable X. The goal of regression is to understand how the values of Y change as X is varied over its range of possible values and also predict Y using X. It is used to answer questions: Does changing class size affect success of students? Can we predict the time of the next eruption of old Faithful Geyser from the length of the most recent eruption? Do changes in diet result in changes in cholesterol level, and if so, do the results depend on other characteristic such as age, sex, and amount of exercise?
6 6 / 47 Regression Simple linear regression Polynomial regression Multiple linear regression let us start from simple linear regression
7 7 / 47 Simple Linear regression We have one response variable (Y) and one explanatory variable (X). Regression analysis was first developed by Sir Francis Galton Galton had studied the relation between heights of father and sons Galton had noted that the heights of sons of both tall and short fathers appeared to revert or regress to the mean of the group. Galton developed a mathematical description of this regression tendency, the precursor of today s regression models. The term regression persists to this day to describe statistical relations between variables.
8 8 / 47 Basic concepts in regression A regression model is a formal means of expressing the two essential ingredients of a statistical relation: A tendency of the dependent variable Y to vary with the independent variable in a systematic fashion A scattering of points around the curve of statistical relationship
9 9 / 47 Basic concepts in regression A regression model is a formal means of expressing the two essential ingredients of a statistical relation: A tendency of the dependent variable Y to vary with the independent variable in a systematic fashion There is a probability distribution of Y for each level of X A scattering of points around the curve of statistical relationship
10 10 / 47 Basic concepts in regression A regression model is a formal means of expressing the two essential ingredients of a statistical relation: A tendency of the dependent variable Y to vary with the independent variable in a systematic fashion There is a probability distribution of Y for each level of X A scattering of points around the curve of statistical relationship The means of these probability distributions vary in some systematic fashion with X.
11 11 / 47 What might be of interest in regression? Regression is a statistical method to estimate relationship between a response and explanatory variable using linear model
12 12 / 47 What might be of interest in regression? Regression is a statistical method to estimate relationship between a response and explanatory variable using linear model Is there a linear relationship? How to describe the relationship How to predict a new value How to predict the value of the explanatory variable that causes a specified response
13 13 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n Y: response variable/dependent variable X: explanatory variable/independent variable ε i : random error with mean E(ε i ) = 0 and variance Var(ε i ) = σ 2, and covariance COV(ε i,ε j ) = 0
14 14 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n Y: response variable/dependent variable X: explanatory variable/independent variable ε i : random error with mean E(ε i ) = 0 and variance Var(ε i ) = σ 2, and covariance COV(ε i,ε j ) = 0 There is one more random variable. What is it?
15 15 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n Y: response variable/dependent variable X: explanatory variable/independent variable ε i : random error with mean E(ε i ) = 0 and variance Var(ε i ) = σ 2, and covariance COV(ε i,ε j ) = 0 There is one more random variable. What is it? Y E(Y i ) =? and Var(Y i ) =?
16 16 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n Y: response variable/dependent variable X: explanatory variable/independent variable ε i : random error with mean E(ε i ) = 0 and variance Var(ε i ) = σ 2, and covariance COV(ε i,ε j ) = 0 There is one more random variable. What is it? Y E(Y i ) = β 0 + β 1 X i and Var(Y i ) = Var(ε i ) = σ 2
17 17 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n E(Y i ) = β 0 + β 1 X i and Var(Y i ) = Var(ε i ) = σ 2 β 0, β 1 : regression coefficient parameters. β 1 : the slope of the regression line which indicates the change in the mean of the probability distribution of Y per unit increase in X. β 0 : the intercept of the regression line. If the slope of the model includes X = 0, β 0 gives the mean of the probability distribution of Y at X = 0. What are known values?
18 18 / 47 Simple linear regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n E(Y i ) = β 0 + β 1 X i and Var(Y i ) = Var(ε i ) = σ 2 β 0, β 1 : regression coefficient parameters. β 1 : the slope of the regression line which indicates the change in the mean of the probability distribution of Y per unit increase in X. β 0 : the intercept of the regression line. If the slope of the model includes X = 0, β 0 gives the mean of the probability distribution of Y at X = 0. What are known values? What are unknown values?
19 19 / 47 Simple regression model To estimate the linear relationship between Y and X, what do we need to do?
20 20 / 47 Simple regression model To estimate the linear relationship between Y and X, what do we need to do? How to estimate unknown?
21 21 / 47 Simple regression model Goal: fit a straight line to points on a scatterplot; find intercept and slope such that ŷ i = b 0 + b 1 x i fits the data as well as possible. find b 0 and b 1 using least square estimation method (LSE) find b 0 and b 1 to minimize i e 2 i, Notation: residual e i = y i ŷ i
22 22 / 47 Notation and Definition Fitted value: Ŷ i = b 0 + b 1 X i b 0 = ˆβ 0, b 1 = ˆβ 1 Residual: e i = Y i Ŷ i S xy : i (x i x)(y i ȳ) S xx : i (x i x) 2 S yy : i (y i ȳ) 2
23 23 / 47 Notation and Definition Fitted value: Ŷ i = b 0 + b 1 X i b 0 = ˆβ 0, b 1 = ˆβ 1 Residual: e i = Y i Ŷ i S xy : i (x i x)(y i ȳ) S xx : i (x i x) 2 S yy : i (y i ȳ) 2 what is the difference between residual (e i ) and error (ε i )?
24 24 / 47 Interpreting a regression line or least squares line: 1.The slope of the line estimates the average increase in y for each one unit increase in x. 2.The intercept of the line is the value of y when x=0, but interpreting the intercept in the context of data only makes sense if 0 is included in the range of measured x-values. 3. Estimates the average y for specific value of x. It also can be as a prediction of the value of y for an individual with a specific value of x.
25 25 / 47 Interpreting a regression line or least squares line: Note: regression line is created based on the least squares criterion: When we use a line to predict the values of y, the sum of squared differences between the observed values of y and the predicted value is smaller for the least squares line than it is for any other line.
26 26 / 47 How to estimate parameters The method of Least Square ( or method of Ordinary Least Square): Estimate β 0 and β 1 to minimize Q = (Y i β 0 β 1 X i ) 2 i The values of β 0 and β 1 that minimize Q can be derived by differentiating Q with respect to β 0 and β 1 Q β 0 = Q β 1 = We then set these partial derivatives equal to zeros using b 0 and b 1 to denote the particular values of β 0 and β 1, respectively, that minimize Q.
27 27 / 47 How to estimate parameters We then finally obtain the following equation, call normal equation, Y i = nb 0 + b 1 X i X i Y i = b 0 i X i + b 1 X 2 i
28 28 / 47 Least Square Estimators ˆβ 0 = b 0 = Ȳ b 1 X ˆβ 1 = b 1 = (X i X)(Yi Ȳ )) (X i X) 2
29 29 / 47 Notation and Definition Fitted value: Ŷ i = b 0 + b 1 X i Residual: e i = Y i Ŷ i S xy : i (x i x)(y i ȳ) S xx : S yy :
30 30 / 47 Notation and Definition Fitted value: Ŷ i = b 0 + b 1 X i Residual: e i = Y i Ŷ i S xy : i (x i x)(y i ȳ) S xx : i (x i x) 2 S yy : i (y i ȳ) 2
31 31 / 47 Least Square Estimators ˆβ 0 = b 0 = Ȳ b 1 X = Ȳ ˆβ 1 X ˆβ 1 = b 1 = (X i X)(Yi Ȳ )) = S xy (X i X) 2 S xx
32 32 / 47 Properties of Least Squares Estimators Gauss-Markov theorem: Under the conditions of regression model, the least squares estimators b 0 and b 1 are unbiased and have minimum variance among all unbiased linear estimators. This theorem means that NOTE: unbiased estimator= a statistic is an unbiased estimation of the parameter if its expectation equals the parameter. i.e., E(b 0 ) = β 0 and E(b 1 ) = β 1
33 33 / 47 Properties of Least Squares Estimators Gauss-Markov theorem: Under the conditions of regression model, the least squares estimators b 0 and b 1 are unbiased and have minimum variance among all unbiased linear estimators. This theorem means that Among all linear estimators that are unbiased, b 0 and b 1 have the smallest variability in repeated samples in which the X levels remain unchanged. NOTE: unbiased estimator: E(b 0 ) = β 0 and E(b 1 ) = β 1
34 34 / 47 Properties of Fitted Regression Line E(Ŷ ) = β 0 + β 1 X = E(Y ) i e i = 0 Y i = Ŷi
35 35 / 47 Estimation of σ 2 Define SSE = e 2 i i = (y i ŷ i ) 2 i which is a sum of squares error (SSE) and has n-2 (why??) degree of freedom SSE/(n 2) is an unbiased estimator of σ 2 Notation: MSE = SSE/n 2 which call mean squares error.
36 36 / 47 Simple linear regression with normal error assumption Model Y i = β 0 + β 1 X i + ε i Y i : response variable/dependent variable X i : explanatory variable/independent variable ε i : are iid N(0, σ 2 ), i=1,...,n, COV(ε i,ε j ) = 0 β 0, β 1 : regression coefficient parameters. β 1 : the slope of the regression line which indicates the change in the mean of the probability distribution of Y per unit increase in X. β 0 : the Y intercept of the regression line. If the slope of the model includes X = 0, β 0 gives the mean of the probability distribution of Y at X = 0. Y i N(, ) Y i (β 0 + β 1 X i ) N(, )
37 Simple linear regression with normal error assumption Model Y i = β 0 + β 1 X i + ε i Y i : response variable/dependent variable X i : explanatory variable/independent variable ε i : are iid N(0, σ 2 ), i=1,...,n, COV(ε i,ε j ) = 0 β 0, β 1 : regression coefficient parameters. β 1 : the slope of the regression line which indicates the change in the mean of the probability distribution of Y per unit increase in X. β 0 : the Y intercept of the regression line. If the slope of the model includes X = 0, β 0 gives the mean of the probability distribution of Y at X = 0. Y i N(, ) Y i (β 0 + β 1 X i ) N(, ) Question: how to estimate β 0 and β 1? 37 / 47
38 38 / 47 Simple linear regression with normal error assumption We estimate β 0 and β 1 using maximum likelihood estimation (MLE). How to calculate MLE?
39 39 / 47 Simple linear regression with normal error assumption We estimate ˆβ 0 and β 1 using maximum likelihood estimation (MLE). How to calculate MLE? calculate Likelihood function Take the first derivative of the likelihood function with respect to β 0 and β 1 Check wheter the second derivative of the likelihood function is less than zero.
40 40 / 47 Maximum Likelihood Estimation Likelihood function is the function of parameters given data. Likelihood function L(,, ) given the sample observation Y 1,...,Y n L(,, ) = n i=1 1 (2πσ 2 ) exp[ (Y i β 0 β X i ) 2 ] 2σ 2
41 41 / 47 Maximum Likelihood Estimation Log-Likelihood is
42 42 / 47 MLE and LSE Parameter LSE MLE Normal Assumption. β 0 β 1 σ 2 SSE? SSE?
43 43 / 47 MLE and LSE Parameter LSE MLE Normal Assumption. NO Yes β 0 = β 1 = σ 2 SSE SSE n 2 n unbiased biased
44 What assumptions we have in simple linear regression 44 / 47
45 45 / 47 What assumptions we have in simple linear regression Independent
46 46 / 47 What assumptions we have in simple linear regression Independent Equal variance
47 47 / 47 What assumptions we have in simple linear regression Independent Equal variance Normal assumption to make inference including test and confidence interval
Regression Models - Introduction
Regression Models - Introduction In regression models there are two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent
More informationRegression Models - Introduction
Regression Models - Introduction In regression models, two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent variable,
More informationChapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression
BSTT523: Kutner et al., Chapter 1 1 Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression Introduction: Functional relation between
More informationChapter 1 Linear Regression with One Predictor
STAT 525 FALL 2018 Chapter 1 Linear Regression with One Predictor Professor Min Zhang Goals of Regression Analysis Serve three purposes Describes an association between X and Y In some applications, the
More informationSTAT5044: Regression and Anova
STAT5044: Regression and Anova Inyoung Kim 1 / 25 Outline 1 Multiple Linear Regression 2 / 25 Basic Idea An extra sum of squares: the marginal reduction in the error sum of squares when one or several
More informationChapter 1. Linear Regression with One Predictor Variable
Chapter 1. Linear Regression with One Predictor Variable 1.1 Statistical Relation Between Two Variables To motivate statistical relationships, let us consider a mathematical relation between two mathematical
More informationSimple Linear Regression
Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.
More informationCh 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
More informationMeasuring the fit of the model - SSR
Measuring the fit of the model - SSR Once we ve determined our estimated regression line, we d like to know how well the model fits. How far/close are the observations to the fitted line? One way to do
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) The Simple Linear Regression Model based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #2 The Simple
More informationSimple and Multiple Linear Regression
Sta. 113 Chapter 12 and 13 of Devore March 12, 2010 Table of contents 1 Simple Linear Regression 2 Model Simple Linear Regression A simple linear regression model is given by Y = β 0 + β 1 x + ɛ where
More informationTMA4255 Applied Statistics V2016 (5)
TMA4255 Applied Statistics V2016 (5) Part 2: Regression Simple linear regression [11.1-11.4] Sum of squares [11.5] Anna Marie Holand To be lectured: January 26, 2016 wiki.math.ntnu.no/tma4255/2016v/start
More informationLecture 14 Simple Linear Regression
Lecture 4 Simple Linear Regression Ordinary Least Squares (OLS) Consider the following simple linear regression model where, for each unit i, Y i is the dependent variable (response). X i is the independent
More informationSimple Linear Regression
Simple Linear Regression ST 430/514 Recall: A regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates)
More informationwhere x and ȳ are the sample means of x 1,, x n
y y Animal Studies of Side Effects Simple Linear Regression Basic Ideas In simple linear regression there is an approximately linear relation between two variables say y = pressure in the pancreas x =
More informationStatistical View of Least Squares
Basic Ideas Some Examples Least Squares May 22, 2007 Basic Ideas Simple Linear Regression Basic Ideas Some Examples Least Squares Suppose we have two variables x and y Basic Ideas Simple Linear Regression
More informationInference for Regression
Inference for Regression Section 9.4 Cathy Poliak, Ph.D. cathy@math.uh.edu Office in Fleming 11c Department of Mathematics University of Houston Lecture 13b - 3339 Cathy Poliak, Ph.D. cathy@math.uh.edu
More informationInference in Regression Analysis
Inference in Regression Analysis Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 4, Slide 1 Today: Normal Error Regression Model Y i = β 0 + β 1 X i + ǫ i Y i value
More informationChapter 14 Simple Linear Regression (A)
Chapter 14 Simple Linear Regression (A) 1. Characteristics Managerial decisions often are based on the relationship between two or more variables. can be used to develop an equation showing how the variables
More informationMultivariate Regression (Chapter 10)
Multivariate Regression (Chapter 10) This week we ll cover multivariate regression and maybe a bit of canonical correlation. Today we ll mostly review univariate multivariate regression. With multivariate
More information1. Simple Linear Regression
1. Simple Linear Regression Suppose that we are interested in the average height of male undergrads at UF. We put each male student s name (population) in a hat and randomly select 100 (sample). Then their
More informationQuantitative Analysis of Financial Markets. Summary of Part II. Key Concepts & Formulas. Christopher Ting. November 11, 2017
Summary of Part II Key Concepts & Formulas Christopher Ting November 11, 2017 christopherting@smu.edu.sg http://www.mysmu.edu/faculty/christophert/ Christopher Ting 1 of 16 Why Regression Analysis? Understand
More informationMathematics for Economics MA course
Mathematics for Economics MA course Simple Linear Regression Dr. Seetha Bandara Simple Regression Simple linear regression is a statistical method that allows us to summarize and study relationships between
More informationStatistics for Engineers Lecture 9 Linear Regression
Statistics for Engineers Lecture 9 Linear Regression Chong Ma Department of Statistics University of South Carolina chongm@email.sc.edu April 17, 2017 Chong Ma (Statistics, USC) STAT 509 Spring 2017 April
More informationWell-developed and understood properties
1 INTRODUCTION TO LINEAR MODELS 1 THE CLASSICAL LINEAR MODEL Most commonly used statistical models Flexible models Well-developed and understood properties Ease of interpretation Building block for more
More informationInferences for Regression
Inferences for Regression An Example: Body Fat and Waist Size Looking at the relationship between % body fat and waist size (in inches). Here is a scatterplot of our data set: Remembering Regression In
More informationMS&E 226: Small Data. Lecture 11: Maximum likelihood (v2) Ramesh Johari
MS&E 226: Small Data Lecture 11: Maximum likelihood (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 18 The likelihood function 2 / 18 Estimating the parameter This lecture develops the methodology behind
More informationSimple Linear Regression
Simple Linear Regression MATH 282A Introduction to Computational Statistics University of California, San Diego Instructor: Ery Arias-Castro http://math.ucsd.edu/ eariasca/math282a.html MATH 282A University
More informationOct Simple linear regression. Minimum mean square error prediction. Univariate. regression. Calculating intercept and slope
Oct 2017 1 / 28 Minimum MSE Y is the response variable, X the predictor variable, E(X) = E(Y) = 0. BLUP of Y minimizes average discrepancy var (Y ux) = C YY 2u C XY + u 2 C XX This is minimized when u
More informationCategorical Predictor Variables
Categorical Predictor Variables We often wish to use categorical (or qualitative) variables as covariates in a regression model. For binary variables (taking on only 2 values, e.g. sex), it is relatively
More informationSTA121: Applied Regression Analysis
STA121: Applied Regression Analysis Linear Regression Analysis - Chapters 3 and 4 in Dielman Artin Department of Statistical Science September 15, 2009 Outline 1 Simple Linear Regression Analysis 2 Using
More informationSF2930: REGRESION ANALYSIS LECTURE 1 SIMPLE LINEAR REGRESSION.
SF2930: REGRESION ANALYSIS LECTURE 1 SIMPLE LINEAR REGRESSION. Tatjana Pavlenko 17 January 2018 WHAT IS REGRESSION? INTRODUCTION Regression analysis is a statistical technique for investigating and modeling
More informationSTAT 4385 Topic 03: Simple Linear Regression
STAT 4385 Topic 03: Simple Linear Regression Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso xsu@utep.edu Spring, 2017 Outline The Set-Up Exploratory Data Analysis
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationMultivariate Regression
Multivariate Regression The so-called supervised learning problem is the following: we want to approximate the random variable Y with an appropriate function of the random variables X 1,..., X p with the
More informationEstimating σ 2. We can do simple prediction of Y and estimation of the mean of Y at any value of X.
Estimating σ 2 We can do simple prediction of Y and estimation of the mean of Y at any value of X. To perform inferences about our regression line, we must estimate σ 2, the variance of the error term.
More informationSimple Linear Regression Analysis
LINEAR REGRESSION ANALYSIS MODULE II Lecture - 6 Simple Linear Regression Analysis Dr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur Prediction of values of study
More informationChapter 2 Multiple Regression I (Part 1)
Chapter 2 Multiple Regression I (Part 1) 1 Regression several predictor variables The response Y depends on several predictor variables X 1,, X p response {}}{ Y predictor variables {}}{ X 1, X 2,, X p
More informationMa 3/103: Lecture 24 Linear Regression I: Estimation
Ma 3/103: Lecture 24 Linear Regression I: Estimation March 3, 2017 KC Border Linear Regression I March 3, 2017 1 / 32 Regression analysis Regression analysis Estimate and test E(Y X) = f (X). f is the
More informationProblems. Suppose both models are fitted to the same data. Show that SS Res, A SS Res, B
Simple Linear Regression 35 Problems 1 Consider a set of data (x i, y i ), i =1, 2,,n, and the following two regression models: y i = β 0 + β 1 x i + ε, (i =1, 2,,n), Model A y i = γ 0 + γ 1 x i + γ 2
More informationChapter 12 - Lecture 2 Inferences about regression coefficient
Chapter 12 - Lecture 2 Inferences about regression coefficient April 19th, 2010 Facts about slope Test Statistic Confidence interval Hypothesis testing Test using ANOVA Table Facts about slope In previous
More informationSimple Linear Regression
9-1 l Chapter 9 l Simple Linear Regression 9.1 Simple Linear Regression 9.2 Scatter Diagram 9.3 Graphical Method for Determining Regression 9.4 Least Square Method 9.5 Correlation Coefficient and Coefficient
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #3 1 / 42 Outline 1 2 3 t-test P-value Linear
More informationScatter plot of data from the study. Linear Regression
1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25
More informationAdvanced Quantitative Methods: ordinary least squares
Advanced Quantitative Methods: Ordinary Least Squares University College Dublin 31 January 2012 1 2 3 4 5 Terminology y is the dependent variable referred to also (by Greene) as a regressand X are the
More informationSimple linear regression
Simple linear regression Prof. Giuseppe Verlato Unit of Epidemiology & Medical Statistics, Dept. of Diagnostics & Public Health, University of Verona Statistics with two variables two nominal variables:
More informationSimple Linear Regression for the MPG Data
Simple Linear Regression for the MPG Data 2000 2500 3000 3500 15 20 25 30 35 40 45 Wgt MPG What do we do with the data? y i = MPG of i th car x i = Weight of i th car i =1,...,n n = Sample Size Exploratory
More informationSTAT 511. Lecture : Simple linear regression Devore: Section Prof. Michael Levine. December 3, Levine STAT 511
STAT 511 Lecture : Simple linear regression Devore: Section 12.1-12.4 Prof. Michael Levine December 3, 2018 A simple linear regression investigates the relationship between the two variables that is not
More informationEstadística II Chapter 4: Simple linear regression
Estadística II Chapter 4: Simple linear regression Chapter 4. Simple linear regression Contents Objectives of the analysis. Model specification. Least Square Estimators (LSE): construction and properties
More informationRegression Analysis Chapter 2 Simple Linear Regression
Regression Analysis Chapter 2 Simple Linear Regression Dr. Bisher Mamoun Iqelan biqelan@iugaza.edu.ps Department of Mathematics The Islamic University of Gaza 2010-2011, Semester 2 Dr. Bisher M. Iqelan
More informationLecture 10 Multiple Linear Regression
Lecture 10 Multiple Linear Regression STAT 512 Spring 2011 Background Reading KNNL: 6.1-6.5 10-1 Topic Overview Multiple Linear Regression Model 10-2 Data for Multiple Regression Y i is the response variable
More informationCorrelation Analysis
Simple Regression Correlation Analysis Correlation analysis is used to measure strength of the association (linear relationship) between two variables Correlation is only concerned with strength of the
More informationIntroduction to Simple Linear Regression
Introduction to Simple Linear Regression Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Introduction to Simple Linear Regression 1 / 68 About me Faculty in the Department
More information2.4.3 Estimatingσ Coefficient of Determination 2.4. ASSESSING THE MODEL 23
2.4. ASSESSING THE MODEL 23 2.4.3 Estimatingσ 2 Note that the sums of squares are functions of the conditional random variables Y i = (Y X = x i ). Hence, the sums of squares are random variables as well.
More informationFormal Statement of Simple Linear Regression Model
Formal Statement of Simple Linear Regression Model Y i = β 0 + β 1 X i + ɛ i Y i value of the response variable in the i th trial β 0 and β 1 are parameters X i is a known constant, the value of the predictor
More informationLecture 15 Multiple regression I Chapter 6 Set 2 Least Square Estimation The quadratic form to be minimized is
Lecture 15 Multiple regression I Chapter 6 Set 2 Least Square Estimation The quadratic form to be minimized is Q = (Y i β 0 β 1 X i1 β 2 X i2 β p 1 X i.p 1 ) 2, which in matrix notation is Q = (Y Xβ) (Y
More informationstatistical sense, from the distributions of the xs. The model may now be generalized to the case of k regressors:
Wooldridge, Introductory Econometrics, d ed. Chapter 3: Multiple regression analysis: Estimation In multiple regression analysis, we extend the simple (two-variable) regression model to consider the possibility
More informationLinear Models and Estimation by Least Squares
Linear Models and Estimation by Least Squares Jin-Lung Lin 1 Introduction Causal relation investigation lies in the heart of economics. Effect (Dependent variable) cause (Independent variable) Example:
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini June 9, 2018 1 / 56 Table of Contents Multiple linear regression Linear model setup Estimation of β Geometric interpretation Estimation of σ 2 Hat matrix Gram matrix
More informationSTAT 540: Data Analysis and Regression
STAT 540: Data Analysis and Regression Wen Zhou http://www.stat.colostate.edu/~riczw/ Email: riczw@stat.colostate.edu Department of Statistics Colorado State University Fall 205 W. Zhou (Colorado State
More informationApplied Regression Analysis
Applied Regression Analysis Chapter 3 Multiple Linear Regression Hongcheng Li April, 6, 2013 Recall simple linear regression 1 Recall simple linear regression 2 Parameter Estimation 3 Interpretations of
More informationSimple Linear Regression
Simple Linear Regression September 24, 2008 Reading HH 8, GIll 4 Simple Linear Regression p.1/20 Problem Data: Observe pairs (Y i,x i ),i = 1,...n Response or dependent variable Y Predictor or independent
More informationAn Introduction to Bayesian Linear Regression
An Introduction to Bayesian Linear Regression APPM 5720: Bayesian Computation Fall 2018 A SIMPLE LINEAR MODEL Suppose that we observe explanatory variables x 1, x 2,..., x n and dependent variables y 1,
More informationCh 3: Multiple Linear Regression
Ch 3: Multiple Linear Regression 1. Multiple Linear Regression Model Multiple regression model has more than one regressor. For example, we have one response variable and two regressor variables: 1. delivery
More informationMAT2377. Rafa l Kulik. Version 2015/November/26. Rafa l Kulik
MAT2377 Rafa l Kulik Version 2015/November/26 Rafa l Kulik Bivariate data and scatterplot Data: Hydrocarbon level (x) and Oxygen level (y): x: 0.99, 1.02, 1.15, 1.29, 1.46, 1.36, 0.87, 1.23, 1.55, 1.40,
More informationSTAT Chapter 11: Regression
STAT 515 -- Chapter 11: Regression Mostly we have studied the behavior of a single random variable. Often, however, we gather data on two random variables. We wish to determine: Is there a relationship
More informationBusiness Statistics. Chapter 14 Introduction to Linear Regression and Correlation Analysis QMIS 220. Dr. Mohammad Zainal
Department of Quantitative Methods & Information Systems Business Statistics Chapter 14 Introduction to Linear Regression and Correlation Analysis QMIS 220 Dr. Mohammad Zainal Chapter Goals After completing
More informationMath 3330: Solution to midterm Exam
Math 3330: Solution to midterm Exam Question 1: (14 marks) Suppose the regression model is y i = β 0 + β 1 x i + ε i, i = 1,, n, where ε i are iid Normal distribution N(0, σ 2 ). a. (2 marks) Compute the
More informationSummer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.
Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall
More informationInference for Regression Simple Linear Regression
Inference for Regression Simple Linear Regression IPS Chapter 10.1 2009 W.H. Freeman and Company Objectives (IPS Chapter 10.1) Simple linear regression p Statistical model for linear regression p Estimating
More informationSample Problems. Note: If you find the following statements true, you should briefly prove them. If you find them false, you should correct them.
Sample Problems 1. True or False Note: If you find the following statements true, you should briefly prove them. If you find them false, you should correct them. (a) The sample average of estimated residuals
More informationThe Simple Regression Model. Part II. The Simple Regression Model
Part II The Simple Regression Model As of Sep 22, 2015 Definition 1 The Simple Regression Model Definition Estimation of the model, OLS OLS Statistics Algebraic properties Goodness-of-Fit, the R-square
More informationLecture 2. The Simple Linear Regression Model: Matrix Approach
Lecture 2 The Simple Linear Regression Model: Matrix Approach Matrix algebra Matrix representation of simple linear regression model 1 Vectors and Matrices Where it is necessary to consider a distribution
More informationRegression Analysis. Regression: Methodology for studying the relationship among two or more variables
Regression Analysis Regression: Methodology for studying the relationship among two or more variables Two major aims: Determine an appropriate model for the relationship between the variables Predict the
More informationMaking sense of Econometrics: Basics
Making sense of Econometrics: Basics Lecture 2: Simple Regression Egypt Scholars Economic Society Happy Eid Eid present! enter classroom at http://b.socrative.com/login/student/ room name c28efb78 Outline
More informationSimple Linear Regression. Part
Simple Linear Regression Part I Chapter1 Linear Regression with One Predictor Variable Regression analysis is a statistical methodology that utilizes the relation between two or more quantitative variables
More informationStatistical Techniques II EXST7015 Simple Linear Regression
Statistical Techniques II EXST7015 Simple Linear Regression 03a_SLR 1 Y - the dependent variable 35 30 25 The objective Given points plotted on two coordinates, Y and X, find the best line to fit the data.
More informationINTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y
INTRODUCING LINEAR REGRESSION MODELS Response or Dependent variable y Predictor or Independent variable x Model with error: for i = 1,..., n, y i = α + βx i + ε i ε i : independent errors (sampling, measurement,
More informationECON The Simple Regression Model
ECON 351 - The Simple Regression Model Maggie Jones 1 / 41 The Simple Regression Model Our starting point will be the simple regression model where we look at the relationship between two variables In
More informationChapter 14 Student Lecture Notes Department of Quantitative Methods & Information Systems. Business Statistics. Chapter 14 Multiple Regression
Chapter 14 Student Lecture Notes 14-1 Department of Quantitative Methods & Information Systems Business Statistics Chapter 14 Multiple Regression QMIS 0 Dr. Mohammad Zainal Chapter Goals After completing
More information9. Linear Regression and Correlation
9. Linear Regression and Correlation Data: y a quantitative response variable x a quantitative explanatory variable (Chap. 8: Recall that both variables were categorical) For example, y = annual income,
More information27. SIMPLE LINEAR REGRESSION II
27. SIMPLE LINEAR REGRESSION II The Model In linear regression analysis, we assume that the relationship between X and Y is linear. This does not mean, however, that Y can be perfectly predicted from X.
More informationMath 1710 Class 20. V2u. Last Time. Graphs and Association. Correlation. Regression. Association, Correlation, Regression Dr. Back. Oct.
,, Dr. Back Oct. 14, 2009 Son s Heights from Their Fathers Galton s Original 1886 Data If you know a father s height, what can you say about his son s? Son s Heights from Their Fathers Galton s Original
More informationLinear Regression Model. Badr Missaoui
Linear Regression Model Badr Missaoui Introduction What is this course about? It is a course on applied statistics. It comprises 2 hours lectures each week and 1 hour lab sessions/tutorials. We will focus
More informationSimple Linear Regression. (Chs 12.1, 12.2, 12.4, 12.5)
10 Simple Linear Regression (Chs 12.1, 12.2, 12.4, 12.5) Simple Linear Regression Rating 20 40 60 80 0 5 10 15 Sugar 2 Simple Linear Regression Rating 20 40 60 80 0 5 10 15 Sugar 3 Simple Linear Regression
More informationThe Multiple Regression Model
Multiple Regression The Multiple Regression Model Idea: Examine the linear relationship between 1 dependent (Y) & or more independent variables (X i ) Multiple Regression Model with k Independent Variables:
More informationWeighted Least Squares
Weighted Least Squares The standard linear model assumes that Var(ε i ) = σ 2 for i = 1,..., n. As we have seen, however, there are instances where Var(Y X = x i ) = Var(ε i ) = σ2 w i. Here w 1,..., w
More informationLecture 34: Properties of the LSE
Lecture 34: Properties of the LSE The following results explain why the LSE is popular. Gauss-Markov Theorem Assume a general linear model previously described: Y = Xβ + E with assumption A2, i.e., Var(E
More informationHomework 2: Simple Linear Regression
STAT 4385 Applied Regression Analysis Homework : Simple Linear Regression (Simple Linear Regression) Thirty (n = 30) College graduates who have recently entered the job market. For each student, the CGPA
More informationLectures on Simple Linear Regression Stat 431, Summer 2012
Lectures on Simple Linear Regression Stat 43, Summer 0 Hyunseung Kang July 6-8, 0 Last Updated: July 8, 0 :59PM Introduction Previously, we have been investigating various properties of the population
More informationSimple Linear Regression. Material from Devore s book (Ed 8), and Cengagebrain.com
12 Simple Linear Regression Material from Devore s book (Ed 8), and Cengagebrain.com The Simple Linear Regression Model The simplest deterministic mathematical relationship between two variables x and
More informationLecture 2 Simple Linear Regression STAT 512 Spring 2011 Background Reading KNNL: Chapter 1
Lecture Simple Linear Regression STAT 51 Spring 011 Background Reading KNNL: Chapter 1-1 Topic Overview This topic we will cover: Regression Terminology Simple Linear Regression with a single predictor
More informationMaster s Written Examination
Master s Written Examination Option: Statistics and Probability Spring 05 Full points may be obtained for correct answers to eight questions Each numbered question (which may have several parts) is worth
More informationBNAD 276 Lecture 10 Simple Linear Regression Model
1 / 27 BNAD 276 Lecture 10 Simple Linear Regression Model Phuong Ho May 30, 2017 2 / 27 Outline 1 Introduction 2 3 / 27 Outline 1 Introduction 2 4 / 27 Simple Linear Regression Model Managerial decisions
More informationRegression Analysis for Undergraduates. By Dr. Anju Gupta. Ms. Shweta Jain
Regression Analysis for Undergraduates By Dr. Anju Gupta Director, CWEB, University of Delhi Ms. Shweta Jain Assistant Professor, CWEB, University of Delhi Introduction The statistical tool correlation
More informationLinear Regression. 1 Introduction. 2 Least Squares
Linear Regression 1 Introduction It is often interesting to study the effect of a variable on a response. In ANOVA, the response is a continuous variable and the variables are discrete / categorical. What
More informationMultiple Linear Regression
Multiple Linear Regression Simple linear regression tries to fit a simple line between two variables Y and X. If X is linearly related to Y this explains some of the variability in Y. In most cases, there
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7
MA 575 Linear Models: Cedric E. Ginestet, Boston University Midterm Review Week 7 1 Random Vectors Let a 0 and y be n 1 vectors, and let A be an n n matrix. Here, a 0 and A are non-random, whereas y is
More informationMultiple Regression. Inference for Multiple Regression and A Case Study. IPS Chapters 11.1 and W.H. Freeman and Company
Multiple Regression Inference for Multiple Regression and A Case Study IPS Chapters 11.1 and 11.2 2009 W.H. Freeman and Company Objectives (IPS Chapters 11.1 and 11.2) Multiple regression Data for multiple
More informationCorrelation and Regression
Correlation and Regression October 25, 2017 STAT 151 Class 9 Slide 1 Outline of Topics 1 Associations 2 Scatter plot 3 Correlation 4 Regression 5 Testing and estimation 6 Goodness-of-fit STAT 151 Class
More information