Linear Regression. Anna Leontjeva

Transcription

1 Linear Regression Anna Leontjeva

2 Which of the following is most related to linear regression? 1) Information Gain 2) Linear Atavism 3) Regression to Mean 4) Method of Least Squares East West North st Qtr 2nd Qtr 3rd Qtr 4th Qtr

3 Introduction to Linear Regression Linear regression is an approach to modeling the relationship between a response variable Y and one or more explanatory variables denoted X (predictors), e.g regression is the study of dependence. A response variable Y must be continuous. The case of one explanatory variable is called simple regression. More than one explanatory variable is multiple regression.

4 Scatterplot

5 Scatterplot

6 Short Quiz 1) 2) 3) 4) Sketch on each plot what you think is the best-fitting line for predicting y from x.

7 Short quiz pic 1 y prediction Residual sum of squares 2 3 4

8 Cross at the average y-value for each x and draw the best-fitting line to the crosses Re-compute the y prediction and sum of squared errors. 1) 2) 3) 4)

9 Linear Regression Function

10 Linear Regression Function Mean function Intercept Slope

11 Linear Regression Function Mean function Intercept Slope Intercept and slope are unknown, want to estimate

12 Linear regression function

13 Residuals (Errors)

14 Objective function residual sum of squares (RSS, SSE): Ordinary Least Squares (OLS)

15 Minimization

16 Example b1 = sum((x-mean(x))*(y - mean(y))) / sum((x-mean(x))^2) [1] b0 = mean(y) - b1*mean(x) [1]

17 Example b1 = sum((x-mean(x))*(y - mean(y))) / sum((x-mean(x))^2) [1] b0 = mean(y) - b1*mean(x) [1] b1 = cov(x,y)/var(x)

18 Example lm(y ~ x)

19 Example

20 Example y = M_height_cm

21 Multiple regression Usually we have more than one variable: or in matrix notation:

22 Matrix notation n observations, p explanatory variables, dim(y) = n 1, dim(x) = n (p+1), dim(ß) = (p+1) 1, dim(e) = n 1

23 OLS for multiple regression

24 Example b = solve(t(x) %*% X) %*% t(x) %*% y

25 Example b = solve(t(x) %*% X) %*% t(x) %*% y b = ginv(x) %*% y lm(y ~ X) lm(y ~ x1 + x2)

26 Types of predictors The intercept (model can be with or without); lm(y ~ x1 + x2 1) Transformations of predictors lm(y ~ x1 + log(x2)) Polynomials lm(y ~ x1 + I(x2^2)) Interactions and other combinations of predictors lm(y ~ x1/x2) Dummy variables and factors lm(y ~ is_male)

27 Polynomials

28 Polynomials m2 <- lm(salary ~ Experience + I(Experience^2), data = prof)

29 Quiz: What does it mean: linear? In which case we cannot use linear regression?

30 Quiz: What does it mean: linear? East West North st Qtr 2nd Qtr 3rd Qtr 4th Qtr

31 Dummy variables Are binary variables (i.e 0 or 1) created from a variable with the higher level of measurement (categorical variable): Eye color Code Brown 1 Blue 2 Grey 3 Eye color Is _Brown Is_Blue Is_Grey Brown Blue Grey 0 0 1

32 Dummy variables Are binary variables (i.e 0 or 1) created from a variable with the higher level of measurement (categorical variable): Eye color Code Brown 1 Blue 2 Grey 3 Eye color Is _Brown Is_Blue Is_Grey Brown Blue Grey 0 0 1

33 Example Salary for males: yrs.since.phd * 1 = yrs.since.phd Salary for females: yrs.since.phd * 0 = yrs.since.phd

34 Diagnostics

35 Leverage points Demo:

36 a leverage point is an observation that has an extreme value on one or more explanatory variables. a point is a bad leverage point if its Y -value does not follow the pattern set by the other data points. a bad leverage point is a leverage point which is also an outlier.

37 Standardized residuals

38 Goodness-of-fit-measures R-squared (square of the sample correlation coefficient between the outcomes and their predicted values) Coefficient Significance: (used to test the hypothesis that the true value of the coefficient is non-zero, in order to confirm that the independent variable really belongs in the model) Measures on the test set (RSS, R-squared)

39 Over- and underfitting

40 Regularization Simple objective function: min(error) with regularization: min(error + ʎ Complexity)

41 Regularization Simple objective function: min(error) with regularization: min(error + ʎ Complexity) Penalty for more complex models: with larger values of lambda, greater penalty more compact model

42 Regularization OLS objective function: min( e 2 ) OLS with regularization (Ridge regression): min( e 2 + λ β i 2 )

43 Regularization OLS objective function: min( e 2 ) OLS with regularization (Ridge regression): min( e 2 + λ β i 2 )

44 Literature A modern approach to Regression with R, Simon Sheather; Applied linear regression, Weisberg