Interpreting the coefficients

Transcription

1 Lecture Week 5 Multiple Linear Regression Interpreting the coefficients Uses of Multiple Regression Predict for specified new x-vars Predict in time. Focus on one parameter Use regression to adjust variation elsewhere... control for (complicating) variation Summarise by finding a pattern Disentangle the variation; Interpret the coefficients Criticise the model 1 of 30

2 Interpreting the coefficients Do more x-variables mean better models? Better? Bigger R 2, Smaller S, smaller SumSq, Fewer Coeffs Simplest coefficients have value 0 - large T-values! But - dropping one var impacts the coeffs of others No issues to discuss unless > 1 predictor (Co)variation in at least 3 dimensions 2 of 30

3 Outline Examples, mostly in more than two dims Theory for correlated x-vars Sums of Squares R 2, multiple correlation and simple corr Changes in R 2 - partial R 2 Changes depend on ORDER in x-vars MTB Coefficients - nor T or P values are NOT a measure of importance 3 of 30

4 Ex 1. Hidden/Lurking variables Too few predictors A subset of the gas data set Knowledge about insulation NOT available to analysis 4 of 30

5 M.Stuart Ex 2. Too? many predictors Relating to Respiratory Muscle Strength other measures of lung function in patients suffering from cystic fibrosis, adjusting for sex and body size. 5 of 30

6 M.Stuart PEmax FEV 1 RV FRC TLC Sex Height Weight BMP The variables Maximal static expiratory pressure a measure of expiratory muscle strength Forced expiratory volume in 1 second Residual volume (after 1 second) Functional residual capacity Total lung capacity 0 = Male, 1 = Female cms. kg. Body mass (percent of median of normal cases) 6 of 30

7 M.Stuart Too many vars; all coeffs small The regression equation is PEmax = FEV RV FRC TLC Sex Height Weight BMP Predictor Coef SE Coef T P Constant FEV RV FRC TLC Sex Height Weight BMP No variables important? Problem: no theoretical guidance S = R-Sq = 63.1% R-Sq(adj) = 44.6% 7 of 30

8 Ex 3: Trees: guidance from geometry Estimating the Vol from Height, Diameter and Ht*Diam^2 Questions 3 better than 1? How much? Coeffs? Central Issue Predictors correlated 8 of 30

9 Using Diameter and Height The regression equation is Volume = Diameter Height Predictor Coef SE Coef T P Constant Diameter Height S = R-Sq = 94.8% R-Sq(adj) = 94.4% 9 of 30

10 Are Ht and Diam Important? The regression equation is Volume = Diameter Height Ht*Diam^2 Volume = Diameter Height Predictor Coef SE Coef T P Constant Diameter (0.000) Height (0.014) Ht*Diam^ S = R-Sq = 97.8% R-Sq(adj) = 97.5% S = R-Sq = 94.8% R-Sq(adj) = 94.4% 10 of 30

11 Direct and Indirect Predictors Ht Tree Vol Ht Ht* Diam 2 Diam OR? Diam Ht* Diam 2 Tree Vol Theory Causal model 11 of 30

12 Ex 4. Math Marks guidance from theory Butterfly Network Model Mechanics Analysis Algebra Vectors Statistics Causation involves hidden variables 12 of 30

13 Direct and Indirect Predictors Correlation mx R Variable MeanStdDev Mech Vect Alg Anal Stat Mech Mech Vect Vect Alg Alg Anal Anal Stat Stat Butterfly model Impossible to deduce directly from correlations Difficult to deduce from several regression analyses 13 of 30

14 Predicting Statistics Performance The regression equation is Stat = Anal Alg Vect Mech Predictor Coef SE Coef T P Constant Anal Alg Vect Mech Mechanics Vectors Algebra Analysis Statistics S = R-Sq = 47.9% R-Sq(adj) = 45.4% 14 of 30

15 Alternative Predictions Stat = Anal Alg Predictor Coef SE Coef T P Constant Anal Alg S = R-Sq = 47.9% R-Sq(adj) = 46.6% Stat = Anal Vect Predictor Coef SE Coef T P Constant Anal Vect Mechanics Vectors Algebra Analysis Statistics S = R-Sq = 39.5% R-Sq(adj) = 38.1% 15 of 30

16 Ex 5a. Uncorrelated x-variables Artificial data x1 x2 e Y Data Generating Model Y x x ; ~ N 0, The regression equation is Y = x x2 Predictor Coef SE Coef T P Constant x x S = R-Sq = 97.9% R-Sq(adj) = 97.3% 16 of 30

17 Ex 5b. Correlated x-variables Artificial data x1 x2 e Y Data Generating Model Y x x ; ~ N 0, The regression equation is Y = x x1 Predictor Coef SE Coef T P Constant x x Corr(X 1,X 2 )= 0.76 S = R-Sq = 75.5% (cf 97.3% despite same generating mechanism) R-Sq(adj) = 67.3% 17 of 30

18 Interpreting the coefficients Do more x-variables mean better models? Bigger R 2, Smaller S, Fewer Coeffs Key issue: correlated and/or missing x-variables Theory Coefficients indirectly reflect correlation High correlation does not imply big coeff Low coeff does not imply low correlation 18 of 30

19 R-squared and Sums of Squares Artificial example y vs x 1 The regression equation is Y = x1 Predictor Coef SE Coef T P Constant x S = R-Sq = 63.5% R-Sq(adj) = 58.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Y Y-meanY Res.x meany SSQ R % of 30

20 R-squared Artificial example y vs x 1,x 2 The regression equation is Y = x x2 Predictor Coef SE Coef T P Constant x x S = R-Sq = 75.5% R-Sq(adj) = 67.3% Y-meanY Res.x1x2 Res.x Analysis of Variance Source DF SS MS F P Regression Residual Error Total R % of 30

21 Partial R-squared y vs x yx R y vs x, x % reduction using 2 y x, x R % reduction using, x 1 x x 1 2 Incremental Reduction R[ y x 1 ], x ( )( ) 2 2 y x x R R[ ], partial 32.1%further reduction on x, 1 when also using x 21 of 30 2

22 Reduction in SSQ The regression equation is Y = x x2 Predictor Coef SE Coef T P Constant x x S = R-Sq = 75.5% R-Sq(adj) = 67.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Y Y-meanY Res.x1 Res.x1x meany SSQ Incremental reduction Source DF Seq SS x x when first use x1and then x2 22 of 30

23 Is order important? The regression equation is Y = x x2 Predictor Coef SE Coef T P Constant x x The regression equation is Y = x x1 No: Coefficients not impacted by ordering Predictor Coef SE Coef T P Constant x x S = R-Sq = 75.5% R-Sq(adj) = 67.3% Analysis of Variance S = R-Sq = 75.5% R-Sq(adj) = 67.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS x x R y x x when first use x1and then x2 [ ], 32.1% 1 2 Source DF SS MS F P Regression Residual Error Total Source DF Seq SS x x Yes: partial R 2 impacted by ordering 2 R y x x when first use x1and then x2 [ ], 19.5% of 30

24 Review R-squared R as a correlation coefficient 2 S S 1 r one x-var; r Corr( x, y) y If yˆ ˆ x ˆ x ˆ x Then S S 1 R where R Corr( y, yˆ ) y yˆ is that linear combination of x, x, x which best predicts y Artificial data Y FITS Corr = Corr 2 = R 2 = 75.5% 24 of 30

25 Review R-squared S y S Var of Var of 2 2 y x, x y 1 y x 1 y x, x about its mean about best linear reg predictor Var of residuals about their mean, 0 S S 1 R y S S R R S S 1 r one x-var; r Corr( x, y) y y y But order x 1,x 2,.. can be arbitrary. One view of importance order is ordering by partial R 2 Experiments with regression 25 of 30

26 When one predictor Review Coefficients x y x 2 2 proportional to r Corr( y, x); R r NB Symmetry r Corr( x, y) When one predictor y x a by b proportional to r Corr( y, x) and hence to "Proportional to" in a theoretical sense where artificial data are created with different degrees of correlation. Then both coeffs will increase with correlation. 26 of 30

27 Review Coefficients Coefficients are not impacted by order When multiple predictors x, x, x y x x x not proportional to r Corr( y and x ) i i i In fact i reflects Corr x and best predictor of x using other x vars AN Dy i i 27 of 30

28 The regression equation is Example: Trees Volume = Diameter Height Ht*Diam^2 Predictor Coef SE Coef T P Constant Diameter Height Ht*Diam^ Ht S = R-Sq = 97.8% R-Sq(adj) = 97.5% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Diameter Height Ht*Diam^ Source DF Seq SS Ht*Diam^ Height Diameter Diam Alternative orderings Ht* Diam 2 Source DF Seq SS Ht*Diam^ Diameter Height Theory Tree Vol 28 of 30

29 Challenges with coeffs To be able to interpret coefficients, ideally Choose x variables that are complementary and measure quite different aspects of the system Organise the data such that it does not inadvertently give the impression that these are correlated, despite their selection In other words, design an experiment 29 of 30