22s:152 Applied Linear Regression. Chapter 5: Ordinary Least Squares Regression. Part 1: Simple Linear Regression Introduction and Estimation

Transcription

1 22s:152 Applied Linear Regression Chapter 5: Ordinary Least Squares Regression Part 1: Simple Linear Regression Introduction and Estimation Methods for studying the relationship of two or more quantitative variables Examples: predict salary from years of experience find effect of lead exposure on school performance predict force at which a metal alloy rod bends based on iron content 1

2 Simple Linear Regression Linear regression model The basic model Y i = β 0 + β 1 x i + ɛ i Y i is the response of dependent variable x i is the observed predictor, explanatory variable, independent variable, covariate x i is treated as a fixed quantity (or if random it is conditioned upon) ɛ i is the error term ɛ i are iid N(0, σ 2 ) So, E[Y i ] = β 0 + β 1 x i + 0 = β 0 + β 1 x i 2

3 Simple Linear Regression Linear regression model Key assumptions (will check these later) linear relationship (between Y and x) *we say the relationship between Y and x is linear if the means of the conditional distributions of Y x lie on a straight line independent errors (independent observations in SLR) constant variance of errors normally distributed errors 3

4 Simple Linear Regression Interpreting the model Model can also be written as: Y i X i = x i N(β 0 + β 1 x i, σ 2 ) mean of Y given X = x is β 0 + β 1 x (known as conditional mean) β 0 + β 1 x is the mean value of all the Y s for the given value of x β 0 is conditional mean when x=0 β 1 is slope, change in mean of Y per 1 unit change in x σ 2 is the variation of responses at x (i.e. dispersion around conditional mean) 4

5 Simple Linear Regression Estimation of β 0 andβ 1 We wish to use the sample data to estimate the population parameters: the slope β 1 and the intercept β 0 Least squares estimation choose ˆβ 0 = b 0 and ˆβ 1 = b 1 such that we minimize the sum of the squared residuals, i.e. minimize n i=1 (Y i Ŷi) 2 minimize g(b 0, b 1 ) = n i=1 (Y i (b 0 + b 1 x i )) 2 Take derivative of g(b 0, b 1 ) with respect to b 0 and b 1, set equal to zero, and solve 5

6 Results: b 0 = Ȳ b 1 x b 1 = ni=1 (x i x)(y i Ȳ ) ni=1 (x i x) 2 the point ( x, Ȳ ) will always be on the least squares line b 0 and b 1 are best linear unbiased estimators (best meaning smallest variance estimator) Notation for fitted line: or Ŷ i = ˆβ 0 + ˆβ 1 x i Ŷ i = b 0 + b 1 x i or in the text Ŷ i = A + Bx i 6

7 predicted (fitted) value: Ŷ i = b 0 + b 1 x i residual: e i = Y i Ŷi Y X The least squares regression line minimizes the residual sums of squares (RSS)= n i=1 (Y i Ŷi) 2 7

8 Example: Cigarette data Measurements of weight and tar, nicotine, and carbon monoxide content are given for 25 brands of domestic cigarettes. VARIABLE DESCRIPTIONS: Brand name Tar content (mg) Nicotine content (mg) Weight (g) Carbon monoxide content (mg) Mendenhall, William, and Sincich, Terry (1992), Statistics for Engineering and the Sciences (3rd ed.), New York: Dellen Publishing 8

9 Do a scatterplot, fit the best fitting line according to least squares estimation. > cig.data=as.data.frame(read.delim("cig.txt",sep=" ", header=false)) > dim(cig.data) [1] 25 5 ## This data set had no header, so I will assign ## the column names here: > dimnames(cig.data)[[2]]=c("brand","tar","nic", "Weight","CO") > head(cig.data) Brand Tar Nic Weight CO 1 Alpine Benson-Hedges BullDurham CamelLights Carlton Chesterfield

10 > plot(cig.data$tar,cig.data$nic) cig.data$nic cig.data$tar ## Fit a simple linear regression of Nicotine on Tar. > lm.out=lm(nic~tar,data=cig.data) ## Get the estimated slope and intercept: > lm.out$coefficients (Intercept) Tar You can do this manually too... 10

11 b 1 = ni=1 (x i x)(y i Ȳ ) ni=1 (x i x) 2 R easily works with vectors and matrices. > numerator=sum((cig.data$tar-mean(cig.data$tar))* (cig.data$nic-mean(cig.data$nic))) > denominator=sum((cig.data$tar-mean(cig.data$tar))^2) > b1=numerator/denominator > b1 [1] b 0 = Ȳ b 1 x > b0=mean(cig.data$nic)-mean(cig.data$tar)*b1 > b0 [1] The fitted line for this data: Ŷ i = x i 11

12 ## Add the fitted line to the original plot: > plot(cig.data$tar,cig.data$nic) > abline(lm.out) cig.data$nic cig.data$tar 12

13 13

14 Simple Linear Regression Estimating σ 2 One of the assumptions of linear regression is that the variance for each of the conditional distributions of Y x is the same at all x-values. Y X In this case, it makes sense to pool all the error information to come up with a common estimate for σ 2 14

15 Recall the model: Y i = β 0 + β 1 x i + ɛ i with ɛ i iid N(0, σ 2 ) We use the sum of the squares of the residuals to estimate σ 2 Acronyms: RSS Residual sum of squares SSE Sum of squared errors RSS SSE ˆσ 2 = RSS n 2 = ni=1 (Y i Ŷi) 2 n 2 RSS = n i=1 (Y i Ŷi) 2 E[ RSS n 2 ] = σ2 ˆσ = S E = SE 2 is called the standard error for the regression (a phrase used by this author) 15

16 2 is subtracted from n in the denominator because we ve used 2 degrees of freedom for estimating the slope and intercept (i.e. there were 2 parameters estimated in the mean structure). When we estimate σ 2 in a 1-sample population, we divide n i=1 (Y i Ŷi) 2 by (n 1) because we only estimate 1 parameter in the mean structure, namely µ. 16

17 Simple Linear Regression Total sums of squares (TSS) Total sums of squares (TSS) quantifies the overall squared distance of the Y -values from the overall mean of the responses Ȳ x y Y bar= TSS= n i=1 (Y i Ȳ )2 17

18 For regression, we can decompose this distance and write: Y i Ȳ = (Y i Ŷi) + (Ŷi Ȳ ) }{{}}{{} distance from observation to fitted line distance from fitted line to overall mean Which leads to the equation 1 : n (Y i Ȳ )2 = i=1 n (Y i Ŷi) 2 + i=1 n (Ŷi Ȳ )2 i=1 or T SS = RSS + RegSS where RegSS is the regression sum of squares 1 (a + b)2 a 2 + b 2. You must square both sides, then include the summation terms and then the cross terms will cancel out due to properties of the fitted line. 18

19 Total variability has been decomposed into explained and unexplained variability In general, when the proportion of total variability that is explained is high, we have a good fitting model The R 2 value (coefficient of determination): the proportion of variation in the response that is explained by the model R 2 = RegSS T SS R 2 = 1 RSS T SS also stated as r 2 in simple linear regression the square of the correlation coefficient r 0 R 2 1 R 2 near 1 suggests a good fit to the data if R 2 = 1, ALL points fall exactly on the line different disciplines have different views on what is a high R 2 = 1, in other words what is a good model 19

20 social scientists may get excited about an R 2 near 0.30 a researcher with a designed experiment may want to see an R 2 near

21 Simple Linear Regression Analysis of Variance (ANOVA) The decomposition of total variance into parts is part of ANOVA. As was stated before: T SS = RSS + RegSS Example: cigarette data cig.data$nic cig.data$tar 21

22 Look at the ANOVA table: You can get these sums of squares manually too... > sum((lm.out$fitted.values-mean(cig.data$nic))^2) [1] > sum(lm.out$residuals^2) [1] > sum((cig.data$nic-mean(cig.data$nic))^2) [1] Get the R2 value (2 ways shown): > summary(lm.out) look for... Multiple R-Squared: > summary(lm.out)$r.squared [1]

23 Example: Lifespan and Thorax of fruitflies LONGEVITY Lifespan, in days THORAX Length of thorax, in mm n= data$thorax data$longevity Sexual Activity and the Lifespan of Male Fruitflies by Linda Partridge and Marion Farquhar. Nature, 294, ,

24 The data and the variables: > ff.data=as.data.frame(read.delim("/fruitfly.txt", sep="\t",header=false)) > dimnames(ff.data)[[2]]=c("id","partners","type", "Longevity","Thorax","Sleep") > head(ff.data) ID Partners Type Longevity Thorax Sleep See how many different Partner values there are: > unique(ff.data$partners) [1]

25 Fit the simple linear regression model: > lm.fruitflies=lm(ff.data$longevity~ff.data$thorax) > summary(lm.fruitflies)... Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-06 *** ff.data$thorax e-15 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 13.6 on 123 degrees of freedom Multiple R-Squared: ,Adjusted R-squared: F-statistic: on 1 and 123 DF, p-value: 1.497e-15 Slope interpretation: For every 1 mm increase in the thorax length of a fruitfly, the average lifespan increases by days. 25

26 Intercept interpretation: When the thorax length is 0 mm, the average lifespan is days (*#??*). This doesn t make sense for this data set for 2 reasons... a fruitfly wouldn t have a 0 mm thorax, and x=0 is far outside of the range of observed x-values. > plot(ff.data$thorax,ff.data$longevity) > abline(lm.fruitflies,lwd=2) ff.data$thorax ff.data$longevity 26

27 > anova(lm.fruitflies) Analysis of Variance Table Response: ff.data$longevity Df Sum Sq Mean Sq F value Pr(>F) ff.data$thorax e-15 *** Residuals Signif. codes: 0 *** ** 0.01 * Regression sum of squares RegSS Residual sum of squares RSS Total sum of squares TSS R 2 = RegSS T SS = =

28 > summary(lm.fruitflies)$r.squared [1] R 2 interpretation: 40.5% of the total variability in lifespan for fruitflies is explained by the length of the thorax. 28

29 Simple Linear Regression Correlation coefficient The correlation coefficient r measures the strength of a linear relationship r = ni=1 (X i X)(Y i Ȳ ) ni=1 (X i X) 2 n i=1 (Y i Ȳ )2 ni=1 (X i X) 2 = n 1 ni=1 (Y i Ȳ )2 n 1 b 1 = S X SY b 1 it is the standardized slope, a unitless measure can be thought of as the value we would get for the slope if the standard deviations of X and Y were equal (similar spreads) would be the slope if X and Y had been standardized before fitting the regression 29

30 1 r 1 r near -1 or +1 shows a strong linear relationship a negative (positive) r is associated with an estimated negative (positive) slope the sample correlation coefficient r estimates the population correlation coefficient ρ r is NOT used to measure strength of a curved line 30

31 Common mistake People often think that as the estimated slope of the regression line, ˆβ 1, gets larger (steeper), so does r. But r really measures how close all the data points are to our estimated regression line. You could have a steep fitted line with a small r (noisy relationship), or a fairly flat fitted line with large r (less noisy relationship). This can be confusing because when the estimated slope is actually 0, then r is 0 no matter how close the points are to the regression line (see formulas for r on the previous pages). 31

32 Example: Cigarette data > cor(cig.data$tar,cig.data$nic) [1] cig.data$nic cig.data$tar Standardize the dependent and independent variables and fit the simple linear regression model to the standardized variables... 32

33 Standardizing the variables: > std.y=(cig.data$nic-mean(cig.data$nic))/sqrt(var(cig.data$nic)) > std.x=(cig.data$tar-mean(cig.data$tar))/sqrt(var(cig.data$tar)) Fitting the model to the standardized variables: > (lm(std.y~std.x))$coefficients (Intercept) std.x e e-01 The slope in the standardized regression is , which is the correlation between the original two variables (as we saw on the previous slide). Standardized Y vs. Standardize X std.y std.x 33

34 A very strong curved relationship can have an r value near 0. > plot(x,y) x y > abline(lm(y~x)) > cor(x,y) [1] The correlation coefficient measures the strength in a linear relationship. 34