Introduction to Linear regression analysis. Part 2. Model comparisons

Transcription

1 Introduction to Linear regression analysis Part Model comparisons 1

2 ANOVA for regression Total variation in Y SS Total = Variation explained by regression with X SS Regression + Residual variation SS Residual Full model y i = + 1 x i + i Unexplained variation in Y from full model = SS Residual ( y y ) i i

3 Reduced model (H true) Reduced model (H : 1 = true): y i = + i (Mean and error) Unexplained variation in Y from reduced model = SS Total ( y y) i Model comparison Difference in unexplained variation between full and reduced models: SS Total -SS Residual = SS Regression ( y y) i Variation explained by including 1 in model 3

4 Explained variation Proportion of variation in Y explained by linear relationship with X Termed r, coefficient of determination: SS Regression SS Total r is simply square of correlation coefficient (r) between X and Y. Which is the better model?? Y1 3 Y X X 4

5 Which is the better model?? Y X Dep Var: Y1 N: 6 Multiple R: Squared multiple R: Adjusted squared multiple R: Standard error of estimate: Effect Coefficient Std Error Std Coef Tolerance t P( Tail) CONSTANT X Analysis of Variance Source Sum-of-Squares df Mean-Square F-ratio P Regression.39543E E Residual E Which is the better model?? Y X Dep Var: Y N: 5 Multiple R: Squared multiple R: Adjusted squared multiple R: Standard error of estimate: Effect Coefficient Std Error Std Coef Tolerance t P( Tail) CONSTANT X Analysis of Variance Source Sum-of-Squares df Mean-Square F-ratio P Regression E E Residual

6 Which is the better model?? Y1 3 Y X n = 6 P =.9 r = X n = 5 P =.386 r =.94 Which is the better model?? 95% Confidence bands (for slope) Y1 3 Y X n = 6 P =.9 r = X n = 5 P =.386 r =.94 6

7 Assumptions Normality Y normally distributed at each value of X: Boxplot of y should be symmetrical - watch out for outliers and skewness Transformations often help 7

8 Homogeneity of variance Variance (spread) of Y should be constant for each value of x i (homogeneity of variance): Very difficult to assess usually (for models with only one value of y per x). Y x y i 1 i Y1 Y x 1 x X 8

9 Homogeneity of variance Variance (spread) of Y should be constant for each value of x i (homogeneity of variance): Very difficult to assess usually (for models with only one value of y per x). Spread of residuals should be even when plotted against x i or predicted y i s Transformations often help Transformations that improve normality of Y will also usually make variance of Y more constant Tests of Homogeneity of variance Y Standardized Residual 1-1 RESIDUAL X X ESTIMATE 9

10 Independence Values of y i are independent of each other: watch out for data which are a time series on same experimental or sampling units should be considered at design stage Linearity True population relationship between Y and X is linear: scatterplot of Y against X watch out for asymptotic or exponential patterns transformations of Y or Y and X often help Always look at residuals 1

11 EDA and regression diagnostics Check assumptions Check fit of model Warn about influential observations and outliers EDA Boxplots of Y (and X): check for normality, outliers etc. Scatterplot of Y and X: check for linearity, homogeneity of variance, outliers etc. 11

12 Anscombe (1973) data set R =.667, y = *x, t = 4.4, P =

13 Limited or weighted data Smoothers (for data exploration especially useful for model fitting) Nonparametric description of relationship between Y and X unconstrained by specific model structure Useful exploratory technique: is linear model appropriate? are particular observations influential? Limited or weighted data Smoothers Each observation replaced by mean or median of surrounding observations or predicted value of regression model through surrounding observations Surrounding observations in window (or band) covers range along X-axis size of window (number of observations) determined by smoothing parameter 13

14 Limited or weighted data Smoothers Adjacent windows overlap resulting line is smooth smoothness controlled by smoothing parameter (width of window) Any section of line robust to extreme values in other windows Types of limited or weighted data smoothers (examples) Running (moving) means or averages: means or medians within each window Lo(w)ess: locally weighted regression scatterplot smoothing observations within a window weighted differently observations replaced by predicted values from local regression line 14

15 Residuals very useful for examining regression assumptions Difference between observed value and value predicted or fitted by the model Residual for each observation: difference between observed y and value of y predicted by linear regression equation: ( y y ) i i Studentised residuals residual / SE residuals follow a t-distribution studentised residuals can be compared between different regressions Observations with large residual (or studentised residual) are outliers from fitted model. 15

16 Plot residuals against predicted y i Residual y +se -se x Predicted y i Even spread of Y around line No pattern in residuals Indicates assumptions OK y x Uneven spread of Y around line Residual +se -se Predicted y i Increasing spread of residuals, ie. wedge-shape Unequal variance in Y Skewed distribution of Y Transformation of Y helps 16

17 Violations of assumptions may not always lead to non-significant result Relationship between birth rate and Per capita income.6 Birthrate per female (per year) Per Capita Income (thousands) Violations of assumptions may not always lead to non-significant result Relationship between birth rate and Per capita income Birthrate per female (per year) Dep Var: BIRTHRATE N: 57 Multiple R: Squared multiple R:.5761 Adjusted squared multiple R: Standard error of estimate:.8899 Analysis of Variance Source Sum-of-Squares df Mean-Square F-ratio P Regression Residual Per Capita Income (thousands) 17

18 Violations of assumptions may not always lead to non-significant result Perhaps consider transformation Birthrate per female (per year) Per Capita Income (thousands) RESIDUAL ESTIMATE Violations of assumptions may not always lead to non-significant result Log transform of Per Capita Income Birthrate per female (per year) Per Capita Income (log (thousands)) RESIDUAL ESTIMATE 18

19 Comparison of models Birthrate per female (per year) Per Capita Income (thousands) Dep Var: BIRTHRATE N: 57 Multiple R: Squared multiple R:.5761 Adjusted squared multiple R: Standard error of estimate:.8899 Analysis of Variance Source Sum-of-Squares df Mean-Square F-ratio P Regression Residual Birthrate per female (per year) Per Capita Income (log (thousands)) Dep Var: BIRTHRATE N: 57 Multiple R: Squared multiple R:.851 Adjusted squared multiple R: Standard error of estimate:.6444 Analysis of Variance Source Sum-of-Squares df Mean-Square F-ratio P Regression Residual Outliers Leverage Influence Other indicators 19

20 Outliers Observations further from fitted model than remaining observations might be different from sample outliers in boxplots Large residual outlier Use robust estimator.syz Leverage How extreme observation is for X-variable Measures how much each x i influences predicted y i Large leverage

21 Influence Cook s D statistic: incorporates leverage & residual observations with large influence on estimated slope observations with D near or greater than 1 should be checked Y 1 Observation 1 is X and Y outlier but not influential Observation has large residual - outlier Observation 3 is very influential (large Cook s D) - also outlier X 3 1

22 Transformations If Y (and error terms) is skewed: log or power transformation of Y improves homogeneity of variance can reduce influence of outliers If nonlinear relationship: linearize by transformation of Y and/or X Transformed variables must make biological sense Robust regression Help with outliers in Y Least Absolute Deviation (LAD) regression Minimizes sum of residuals (instead of squares) M (maximum likelihood) regression Many types often based on iteratively reweighted least squares, not useful for issues with leverage. Converges on OLS when assumptions are met Help with outliers in Y and X Least Median of Squares (LMS) regression Minimizes median of squares of the residuals Least Trimmed Squares (LTS) regression Uses a trimmed set of observations and operates on sums of residuals Rank regression Uses ranks of residuals rather than values

23 Regression through origin Fit model: y i = 1 x i + i Problems: extrapolation below smallest x i ANOVA partition of SS no longer additive r difficult to interpret Always test H that = first Recommendation - Only do this if there is a compelling reason to do so (usually there is not) 3