Predict y from (possibly) many predictors x. Model Criticism Study the importance of columns

Transcription

1 Lecture Week Multiple Linear Regression Predict y from (possibly) many predictors x Including extra derived variables Model Criticism Study the importance of columns Draw on Scientific framework Experiment; find simplest & best predictor Look for important rows Diagnostics Outliers and Influence 12/02/201 1

2 Interpreting the coefficients MLR Experiment with models Dropping/Adding vars impacts coeffs of others No issues to discuss unless > 1 predictor Co-variation in at least 3 dimensions Do more x-variables mean better models? More Coeffs bigger R 2, smaller S & SumSq Simplest coefficients have value 0 large T-values! Simple models Best science 12/02/201 2

3 Multiple Linear Regression Predict y from (possibly) many predictors x Including extra derived variables Experiment with models Dropping/Adding Vars Noting change in R 2 Noting change in fitted coeffs Check diagnostics VIF Find simplest & best predictor Understand how y interacts with predictors x 12/02/201 3

4 How important is predictor x k? Fitted coeff b k = avg inc/dec in y when x k increases by one unit and all other predictors unchanged Big numerical value? Big T ratio? Small p? 12/02/201 4

5 How important is predictor x k? What if : An important predictor is not available? Some X vars are highly inter-correlated? What are implications for interpreting b k? changes in b k when other variables added/dropped? 12/02/201

6 Trees: A simple case Linear Model regressing Vol on Height Diam and Ht Diam 2 simple theory available 12/02/201 6

7 Diameter and Height important The regression equation is Volume = Diameter Height Predictor Coef SE Coef T P Constant Diameter Height S = R-Sq = 94.8% R-Sq(adj) = 94.4% 12/02/201 7

8 Diameter and Height not important The regression equation is Volume = Diameter Height Ht*Diam^2 Predictor Coef SE Coef T P Constant Diameter Height Ht*Diam^ S = R-Sq = 97.8% R-Sq(adj) = 97.% 12/02/201 8

9 Strategies with correlated predictors Regression a device: to think about Rel Importance of X vars Correlation important Proceed with care; need reg theory! Transformations Derived variables Modify models; use VIF to predict Correlation relatively unimportant? Semi-automatic options Incl Best Subsets/Stepwise 12/02/201 9

10 Outline Examples, mostly in more than two dims Theory for correlated x-vars Sums of Squares R 2, multiple correlation and simple corr Changes in R 2 - partial R 2 Changes depend on ORDER in x-vars MTB Coefficients - nor T or P values are NOT always a measure of importance 12/02/201 10

11 Technical Material MLR as a sequence of SLR Partial R 2 Correlated predictors Variance inflation Multi-collinearity Use of Intercept term with indicator variables 12/02/201 11

12 Extreme case: Tree Vol: x 1 =x 2 = Ht Regress Vol on x 1 Vol = x 1 = x 1 +0 x 2 Regress Vol on x 2 Vol = x x 2 Regress Vol on both 12/02/201 Vol = (b) x 1 +(1.43-b) x 2 for any arbitrary value of b!! Infinity of identical solutions All equally good for predicting MINITAB notes and takes action Extra tech material online 12

13 Common case: x 1 x 2 Infinity of nearly identical solutions Many pairs of coeffs almost equivalent More generally at least one x User notes and takes action nearly perfectly predictable from other x vars Many sets of coeffs almost equivalent 12/02/201 13

14 M.Stuart PEMax: Too many predictors? Obj: Relate Respiratory Muscle Strength (PEMax) To other measures of lung function in patients suffering from cystic fibrosis, adjusting for sex and body size. 12/02/201 14

15 Controlling for external variation Observation al data are often unbalanced eg age, gender Ideally data collection designed equal numbers M/F similar age dist in each group Regression often used to control for such variation 12/02/201 1

16 PEmax FEV 1 RV FRC TLC Sex Height Weight BMP PEMax: The variables Maximal static expiratory pressure a measure of expiratory muscle strength Forced expiratory volume in 1 second Residual volume (after 1 second) Functional residual capacity Total lung capacity 0 = Male, 1 = Female cms. kg. Sub Age Sex Ht Wt BMP FEV1 RV FRC TLC PEmax Body mass (percent of median of normal cases) 12/02/201 16

17 PEmax: Too many vars? all coeffs small The regression equation is PEmax = FEV RV FRC TLC Sex Height Weight BMP Predictor Coef SE Coef T P Constant FEV RV FRC TLC Sex Height Weight BMP No variables important? Issue here: Too many correlated variables Challenge here: Poor theor. guidance Return later S = R-Sq = 63.1% R-Sq(adj) = 44.6% 12/02/201 17

18 Some simple cases Scientific Framework Networks 12/02/201 18

19 Direct and Indirect Importance Ht Tree Vol Ht Ht* Diam 2 Diam OR? Diam Ht* Diam 2 Tree Vol Theory Scientific Framework Causal model 12/02/201 19

20 Math Marks Marks on 88 students in maths exams. How to predict Stats mark from others? Correlation mx R Variable MeanStdDev Mech Vect Alg Anal Stat Mech Mech Vect Vect Alg Alg Anal Anal Stat Stat What can we learn from the coeffs in the best predictor? Correlation mx R Variable MeanStdDev Mech Vect Alg Anal Stat Mech Mech Vect Vect Alg Alg Anal Anal Stat Stat /02/201 20

21 Math Marks: guidance from theory Mechanics Analysis Algebra Vectors Statistics 12/02/201 21

22 Predicting Statistics Performance The regression equation is Stat = Anal Alg Vect Mech Predictor Coef SE Coef T P Constant Anal Alg Vect Mech Mechanics Vectors Algebra Analysis Statistics S = R-Sq = 47.9% R-Sq(adj) = 4.4% 12/02/201 22

23 Alternative Predictions Stat = Anal Alg Predictor Coef SE Coef T P Constant Anal Alg S = R-Sq = 47.9% Stat = Anal Vect Predictor Coef SE Coef T P Constant Anal Vect Mechanics Vectors Algebra Analysis Statistics S = R-Sq = 39.% 12/02/201 23

24 Theory 12/02/201 24

25 Ex a. Uncorrelated x-variables Artificial data x1 x2 e y SS(total) 1.87 Corr x1 x2 x y Data Generating Model Y x x ; ~ N 0, ; 1; 1; The regression equation is ybal = x x2bal Predictor Coef SE Coef T P Constant x x2bal S = R-Sq = 93.9% SS Total /02/201 2

26 x1unbal Ex b. Correlated x-variables 1.0 Artificial data x1 x2 e y SS(total) 40.1 Scatterplot of x1unbal vs x2unbal Corr x1 x2 x y Data Generating Model Y x x ; ~ N 0, ; 1; 1; The regression equation is y = x x2 Predictor Coef SE Coef T P Constant x x S = R-Sq = 98.4% Coeffs smaller SE(Coeffs) larger x2unbal /02/201 26

27 MLR as successive SLR Additional Info from x2 not in x1 1. Regress y on x1 Store Resids RESy.x1 1a Regress x2 on x1 Store Resids RESx2.x1 Thus RESy.x1 and RESx2.x1 represent those aspects of y and x2, that DO NOT depend on x1 2. Regress RESy.x1 on RESx2.x1 24/02/201 27

28 x2unbal RESy.x y Residuals the same Models identical MLR stepwise by SLR Residuals (y.x1) x1 x2 y y.x1 x2.x1.(x2.x1) y.x1x ; R 2 =97.2% Fitted Line Plot y = x1 ; unbalanced case x S R-Sq 97.2% R-Sq(adj) 96.8% Fitted Line Plot x2 = x1; unbalanced case 1. S a R-Sq 97.2% R-Sq(adj) 96.8% 2; R 2 =43.6% Fitted Line Plot RES.x1 = RESx2.x1 unbalanced case S R-Sq 43.6% R-Sq(adj) 3.% x1unbal RESx2l.x /02/201 28

29 Reduction in SSQ Partial R 2 The regression equation is y = x x2 Predictor Coef SE Coef T P Constant X X S = R-Sq = 98.4% Analysis of Variance Source DF SS MS F P Regression (small rounding error 43.6%) e Residual Error Total Source DF Seq SS X X /02/201 Note that MINITAB 17 uses a different layout for the SS than that shown here Total SS X1 explains % X2 explains 0.48 Total % Unexplained by X Of this explained by x ie 43% y x [ y x ] [ x x ] R 1 R 1 R Here using R as in (0,1); ie %age R /100 29

30 Is Order Important? 12/02/201 30

31 Is order important? The regression equation is y = x x2 Predictor Coef SE Coef T P Constant X X The regression equation is y = x x1 No: Coefficients not impacted by ordering Predictor Coef SE Coef T P Constant x x S = R-Sq = 98.4% Analysis of Variance S = R-Sq = 98.4% Analysis of Variance Source DF SS MS Regression Residual Error Total Source DF Seq SS X X first use x and then x 1 2 Source DF SS MS Regression Residual Error Total Source DF Seq SS x X Yes: partial R 2 impacted by ordering first use x and then x 2 1 Note that MINITAB 17 uses a different layout for the SS than that shown here 24/02/201 31

32 Is order important? Uncorrelated Preds No: Coefficients not impacted by ordering Analysis of Variance Analysis of Variance Source DF SS MS Regression Residual Error Total Source DF SS MS Regression Residual Error Total Source DF Seq SS x x Source DF Seq SS x x Partial R 2 not impacted by ordering if predictors uncorrelated Note that MINITAB 17 uses a different layout for the SS than that shown here 24/02/201 32

33 Is order important? No No Yes For prediction If predictor variables uncorrelated even if correlated For coeffs and for SE( Coeff), T ratios, p values for teasing out aspects of relative importance 12/02/201 33

34 Is correlation in predictors important? No even if correlated For prediction - if n is large Yes For coeffs and for SE( Coeff), T ratios, p values Seek simplest model you can get away with But no simpler Seek and drop redundant variables 12/02/201 34

35 Variance Inflation Factors 12/02/201 3

36 2 x 2 j SE b s R j SE b j Variance Inflation Factor 2 s 1 ( n 1) s 1 R Variance of the 2 2 x j 2 x 2 j values % of var of x when regressed on all other preds j large when j j x j s R j small large Implications: If have control over study design spread out the predictors Else Here using R as in (0,1); arrange preds to be uncorrelated coeffs can be individually small, SEs can be large ie %age R / If Coeff interpretation important be careful with too many derived variables 12/02/201 36

37 Trees Regression Analysis: Vol versus Ht x Diam^2, Diam, Ht The regression equation is Vol = Ht x Diam^ Diam Ht Predictor Coef SE Coef T P VIF Constant Ht x Diam^ Diam Ht S = R-Sq = 97.8% Note that MINITAB 17 uses a different layout for the SS than that shown here Source DF Seq SS Ht x Diam^ Diam Ht /02/201 37

38 Trees Regression Analysis: Vol versus Ht x Diam^2, Diam, Ht The regression equation is Vol = Ht x Diam^ Diam Ht Predictor Coef SE Coef T P VIF Constant Ht x Diam^ Diam Ht S = R-Sq = 97.8% Note that MINITAB 17 uses a different layout for the SS than that shown here Source DF Seq SS Ht x Diam^ Diam Ht /02/201 38

39 The regression equation is Trees Vol = Ht x Diam^2 Predictor Coef SE Coef T P VIF Constant Ht x Diam^ Cf Ht x Diam^ S = R-Sq = 97.8% VIF = 1 1< VIF < VIF > to 10 Not correlated Mod. correlated Highly correlated VIF values greater than 10 may indicate multicollinearity is unduly influencing your regression results. In this case, you may want to reduce multicollinearity by removing unimportant predictors from your model. 12/02/201 39

40 Example: Trees Ht Theory Ht* Diam 2 Tree Vol Diam Source DF Seq SS Diameter Height Ht*Diam^ Alternative orderings Source DF Seq SS Ht*Diam^ Height Diameter Note that MINITAB 17 uses a different layout for the SS than that shown here Source DF Seq SS Ht*Diam^ Diameter Height /02/201 40

41 PE Max revisited The regression equation is PEmax = Age Sex Height Weight BMP FEV RV FRC TLC Predictor Coef SE Coef T P VIF Constant Age Sex Height Weight BMP FEV RV FRC TLC Which to drop? What s the objective? S = R-Sq = 63.8% 12/02/201 41

42 PEmax PEmax PE Max revisited The regression equation is PEmax = FRC Fitted Line Plot PEmax = FRC S R-Sq 17.4% R-Sq(adj) 13.8% 10 Predictor Coef SE Coef T P Constant FRC FRC S = R-Sq = 17.4% Scatterplot of PEmax vs FRC Sex FRC /02/201 42

43 PE Max revisited The regression equation is PEmax = Sex FRC Predictor Coef SE Coef T P Constant Sex FRC Source DF Seq SS Sex FRC S = R-Sq = 22.1% R-Sq(adj) = 1.0% Analysis of Variance Source DF SS MS F P Regression Residual Error Total /02/201 43

44 The regression equation is PE Max revisited PEmax = Sex Age FRC Predictor Coef SE Coef T P Constant Sex Age FRC Source DF Seq SS Sex Age FRC S = R-Sq = 41.2% Analysis of Variance Source DF SS MS F P Regression Residual Error Total /02/201 44

45 PE Max revisited Corr(Ht,Wt)=0.921 %age of variation in Ht explained by Wt = 100(0.921) 2 =8% The regression equation is PEmax = Height Weight FRC Predictor Coef SE Coef T P VIF Constant Height Weight FRC /02/201 4

46 PE Max revisited The regression equation is PEmax = Height FRC Predictor Coef SE Coef T P VIF Constant Height FRC The regression equation is PEmax = Height Weight FRC Predictor Coef SE Coef T P VIF Constant Height Weight FRC /02/201 46

47 Interpreting the coefficients Do more x-variables mean better models? Bigger R 2, Smaller S, Fewer Coeffs Key issue: correlated and/or missing x-variables Theory Coefficients indirectly reflect correlation High correlation does not imply big coeff Low coeff does not imply low correlation 12/02/201 47

48 Review R-squared R as a correlation coefficient 2 S S 1r one x-var; r Corr( x, y) y If yˆ b x b x b x Then S S 1 R where R Corr( y, yˆ ) y yˆ is that linear combination of x, x, x which best predicts y 12/02/201 48

49 SSTotal S y S Review R-squared Var of y about its mean Var of y about best linear reg predictor Var of residuals about their mean, 0 S S 1R y S S R R y x, x y y x y x, x yx y x, x 1 R 1 R 1 R S S 1 r one x-var; r Corr( x, y) y 2 2 Here using R as in (0,1); ie %age R /100 12/02/201 49

50 Review Coefficients When one predictor x y x 2 2 simply related to r Corr( y, x); R r NB Symmetry r Corr( x, y) When one predictor y x a by b simply to r Corr( y, x) and hence to 12/02/201 0

51 Review Coefficients Coefficients are not impacted by order When multiple predictors x, x, x y x x x not proportional to r Corr( y and x ) i i i In fact i reflects Corr x i and best predictor of x i using other x vars AN D y 12/02/201 1

52 Strategies for correlated x-vars Redundancy in an extreme case If two or more vars contain exactly one piece of information, use only one of them Partial redundancy If two or more vars contain much the same information for the purposes in hand, use one (possibly composite) variable. More generally, can the important info in K variables be reduced to a few (possibly composite) variables? 12/02/201 2

53 Other Strategies Best Subsets and Stepwise Regression Select Best in a predictive sense Modern methods Very large data sets n and/or p Computationally intensive Data Mining literature/software Penalise models with many variables Note that many models nearly as good 12/02/201 3

54 Challenges with coeffs To be able to interpret coefficients, ideally Choose x variables that are complementary and measure quite different aspects of the system Organise the data such that it does not inadvertently give the impression that these are correlated, despite their selection In other words, design an experiment 12/02/201 4

55 Technicality 12/02/201

56 Extreme case Exact multi-collinearity x 2 perfectly correlated with x 1 x k perfectly predicted by others SE(coeff) = can t be computed No single best set of parameters 2 j SE b R j 2 s 1 ( n 1) s 1 R 2 2 x j j % of var of x when regressed on all other preds j MINITAB refuses to proceed Often an error Same var entered twice! Always arises with sets of indicators 12/02/201 6

57 Exact multi-collinearity Many pairs of coeffs give same predicted values No unique solution slope 1.86 Pred from Preds using both x1 and x2 intercept 1.36 X1 coeffs x y x1 x2 x /02/201 7

58 Exact multi-collinearity Many pairs of coeffs give same predicted values No unique solution slope 1.86 Pred from Preds using both x1 and x2 intercept 1.36 X1 coeffs x y x1 x2 x /02/201 8

59 Indicator Vars: Exact multi-collinearity Regression Analysis: Comps versus Time since 1978, Q1, Q2, Q3, Q4 * Q4 is highly correlated with other X variables * Q4 has been removed from the equation. The regression equation is Comps = Time since Q Q2-78 Q3 12/02/201 9

60 Indicator Vars: Exact multi-collinearity Model has no intercept Regression Analysis: Comps versus Time since 1978, Q1, Q2, Q3, Q4 The regression equation is Comps = 986 Time since Q Q Q3-942 Q4 12/02/201 60

61 Indicator Vars: Exact multi-collinearity Models 1 Comps = 986 Time since Q Q Q3-942 Q4 2 Comps = Time since Q Q2-78 Q3 Time since Indicator vars Predictions 1978 Comps Q1 Q2 Q3 Q4 Model 1 Model /02/201 61

62 Indicator Vars: Exact multi-collinearity Models 1 Comps = 986 Time since Q Q Q3-942 Q4 2 Comps = Time since Q Q2-78 Q3 Time since Indicator vars Predictions 1978 Comps Q1 Q2 Q3 Q4 Model 1 Model /02/201 62