Analysis of variance and regression. April 17, Contents Comparison of several groups One-way ANOVA. Two-way ANOVA Interaction Model checking

Transcription

1 Analysis of variance and regression Contents Comparison of several groups One-way ANOVA April 7, 008 Two-way ANOVA Interaction Model checking ANOVA, April 008 Comparison of or more groups Julie Lyng Forman, Dept. of Biostatistics, Institute of Public Health, University of Copenhagen number different same of groups individuals individual unpaired paired t-test t-test oneway twoway analysis of variance analysis of variance One-way analysis of variance: Do the distributions differ between the groups? Do the levels differ between the groups?

2 ANOVA, April 008 ANOVA, April Example: bypass-patients, 3 different kinds of ventilation during anaesthesia, randomized Group I Group II Group III 50% N O, 50% O for hours 50% N O, 50% O during operation 30 50% O (no N O) for hours Gr.I Gr.II Gr.III n Mean SD ANOVA, April 008 ANOVA, April One-way ANOVA one-way: because we only have one critera for subdividing the observations, here ventilation method ANalysis Of VAriance: because we compare the variance between groups with the variance within groups Model: j th observation in group no. i Y ij = µ i + ε ij mean of group no. i individual deviation Observations are assumed be independent and to follow a normal distribution (within each group) with the same variance. ε ij N(0, σ ) or equivalently Y ij N(µ i, σ ) Model assumptions must be checked! Usual hypothesis: group means are equal

3 ANOVA, April 008 ANOVA, April ANOVA math: Sums of squares Decomposition of deviation from grand mean : y ij ȳ i ȳ. y ij ȳ = (y ij ȳ i ) + (ȳ i ȳ ) j th observation in i th group average in i th group total average Decomposition of variation (sums of squares): (y ij ȳ ) = (y ij ȳ i ) + (ȳ i ȳ ) i,j i,j i,j }{{}}{{}}{{} total variation within groups between groups Decomposition of variation: F-test statistic: total = between + within SS total = SS between + SS within (n ) = (k ) + (n k) F = MS between = SS between/(k ) MS within SS within /(N k) Reject the null hypothesis if F is large, i.e. if the variation between groups is too large compared to the variation within groups. ANOVA, April ANOVA, April Analysis of variance table - Anaestesia example Usually the analysis is summarized in an Analysis of variance table Variation df SS MS F P Between k SS b SS b /df b MS b /MS w P(F(df b, df w ) > F obs ) Within n k SS w SS w /df w Total n SS tot df SS MS F P Between Within Total F = 3.7 F(, 9) P = 0.0 Weak evidence of non-equality of the three means

4 ANOVA, April ANOVA, April 008 Analysis of variance in SAS To define the anaestesia data in SAS, we write data ex_redcell; input grp redcell; cards; ; The variable redcell contains all the measurements of the outcome and grp contains the method of ventilation for each individual. Analysis of variance program: proc glm data=ex_redcell; class grp; model redcell=grp / solution; General Linear Models Procedure Dependent Variable: REDCELL Sum of Mean Source DF Squares Square F Value Pr > F Model Error Corrected Total R-Square C.V. Root MSE REDCELL Mean Source DF Type I SS Mean Square F Value Pr > F GRP Source DF Type III SS Mean Square F Value Pr > F GRP ANOVA, April 008 ANOVA, April The option solution outputs parameter estimates: T for H0: Pr > T Std Error of Parameter Estimate Parameter=0 Estimate INTERCEPT B GRP B B B... NOTE: The X X matrix has been found to be singular and a generalized inverse was used to solve the normal equations. Estimates followed by the letter B are biased, and are not unique estimators of the parameters. Group 3 (the last group) is the reference group The estimates for the other groups refer to differences to this reference group Multiple comparisons The F-test show, that there is a difference but where? Pairwise t-tests are not suitable due to risk of mass significance Recall a significance level of α = 0.05 means 5% chance of wrongfully rejecting a true hypothesis (type I error) The chance of at least one type I error goes up with the number of tests (for k groups, we have m = k(k )/ possible tests, the actual significance level can be as bad as: ( α) m, e.g. for k=5: 0.0)

5 ANOVA, April 008 ANOVA, April There is no completely satisfactory solution. Approximative solutions:. Select a (small) number of relevant comparisons in the planning stage.. Make a graph of the average ± SEM and judge visually (!), perhaps supplemented with F-tests on subsets of groups. 3. Modify the t-tests by multiplying the P-values with the number of tests, the socalled Bonferroni correction (conservative) or another form of correction (Dunnett, Tukey). Tukey multiple comparisons in SAS: proc glm data=ex_redcell; class grp; model redcell=grp / solution; lsmeans grp / adjust=tukey pdiff cl; The GLM Procedure Least Squares Means Adjustment for Multiple Comparisons: Tukey-Kramer Least Squares Means for effect grp Pr > t for H0: LSMean(i)=LSMean(j) Dependent Variable: redcell i/j Least Squares Means for Effect grp Difference Simultaneous 95% Between Confidence Limits for i j Means LSMean(i)-LSMean(j) ANOVA, April 008 ANOVA, April Visual assessment: the bars represent confidence intervals for the means. proc gplot data=ex_redcell; plot redcell*grp / haxis=axis vaxis=axis frame; axis order=( to 3 by ) offset=(8,8) label=(h=3 gruppe nr. ) value=(h=) minor=none; axis offset=(,) value=(h=) minor=none label=(a=90 R=0 H=3 red cell foliate ); symbol v=circle i=stdmjt l= h= w=; Model checking Check if the assumptions are reasonable: (If not the analysis is unreliable!) Variance homogeneity may be checked by performing Levenes test (or Bartletts test). In case of variance inhomogeneity, we may also perform a weighted analysis (Welch s test), just as in the T-test Normality may be checked through probability plots (or histograms) of residuals, or by a numerical test on the residuals. In case of non-normality, we may use the nonparametric Kruskal-Wallis test Transformation (often logarithms) may help to achieve variance homogeneity as well as normality

6 ANOVA, April ANOVA, April Check of variance homogeneity and normality in SAS proc glm data=ex_redcell; class grp; model redcell=grp; means grp / hovtest=levene welch; output out=model p=predicted r=residual; Store residuals in a dataset for further model checking proc univariate normal data=model; var residual; Output from proc glm: Test for variance homogeneity Levene s Test for Homogeneity of redcell Variance ANOVA of Squared Deviations from Group Means Sum of Mean Source DF Squares Square F Value Pr > F grp Error and weighted anova in case of variance heterogeneity: Welch s ANOVA for redcell Source DF F Value Pr > F grp Error.0 So we are not too sure concerning the group differences... ANOVA, April ANOVA, April 008 Output from proc univariate: Test for normality: Tests for Normality Test --Statistic p Value---- Shapiro-Wilk W Pr < W 0.88 Kolmogorov-Smirnov D Pr > D >0.500 Cramer-von Mises W-Sq 0.03 Pr > W-Sq >0.500 Anderson-Darling A-Sq Pr > A-Sq >0.500 The tests focus on different aspects of non-normality. For small data sets, we rarely get significance For large data sets, we almost always get significance Could look at a probability plot instead Non-parametric ANOVA, the Kruskal-Wallis test: proc nparway wilcoxon; exact; class grp; var redcell; Again, we have lost the significance... Wilcoxon Scores (Rank Sums) for Variable redcell Classified by Variable grp Sum of Expected Std Dev Mean grp N Scores Under H0 Under H0 Score Kruskal-Wallis Test Chi-Square.85 DF Asymptotic Pr > Chi-Square 0.3 Exact Pr >= Chi-Square 0.33

7 ANOVA, April 008 ANOVA, April Two-way analysis of variance Two criterias for subdividing observations, A og B Data in two-way layout: (not for analysis!!) B A c... r. Effect of both factors Perhaps even interaction (effect modification) One factor may be individuals or experimental units (e.g. different treatments tried on same person) Repeated measurements Example: Short term effect of enalaprilate on heart rate Time Subject average average ANOVA, April 008 ANOVA, April Line plot ( Spaghettiogram ) Additive model: Y st = µ + α s + β t + ε st The two effects (s and t) work in an additive way. The ε st s are assumed to be independent, normally distributed with mean 0, and identical variances, ε st N(0, σ ) (check this!) Variational decomposition: SS total = SS subject + SS time + SS residual Ideally the time courses are parallel.

8 ANOVA, April 008 ANOVA, April Analysis of variance table - enalaprilate example Two-way ANOVA in SAS: df SS MS F P Subjects <0.000 Times Residual 9.8. Total Highly significant difference between subjects (not very interesting) Significant time differences. proc glm data=ex_pulse; class subject times; model hrate=subject times / solution; General Linear Models Procedure Class Level Information Class Levels Values SUBJECT TIMES Number of observations in data set = 3 ANOVA, April ANOVA, April Analysis of variance table from output: Parameter estimates from output: General Linear Models Procedure Dependent Variable: HRATE Sum of Mean Source DF Squares Square F Value Pr > F Model Error Corrected Total R-Square C.V. Root MSE HRATE Mean Source DF Type I SS Mean Square F Value Pr > F SUBJECT TIMES Source DF Type III SS Mean Square F Value Pr > F SUBJECT TIMES T for H0: Pr > T Std Error of Parameter Estimate Parameter=0 Estimate INTERCEPT 0.9 B SUBJECT B B B B B B B B B... TIMES 0. B B B B... NOTE: The X X matrix has been found to be singular and a generalized inverse was used to solve the normal equations. Estimates followed by the letter B are biased, and are not unique estimators of the parameters. subject 9 at time 0 minutes is the reference

9 ANOVA, April ANOVA, April Expected values for subject=3, times=30: ŷ st = ˆµ + ˆα s + ˆβ t = = 85. Residuals r st = observed expected = y st ŷ st ε st Residual for subject 3, time 30: r 3 = = 0.8 Look for: Model checking differences in variances (systematic?) Non-normality. Lack of additivity (interaction). Can only be tested if there is more than one observation for each combination Serial correlation? (Neighboring observations look more alike) ANOVA, April ANOVA, April Enalaprilate example: Use the residuals for model checking: Probability plot of residuals. Plot residuals vs expected values. Plot residuals vs group. Look for outliers (a large residual means observed and expected values deviate a lot). No systematic patterns should be present.

10 ANOVA, April ANOVA, April Interaction Example of two criterias for subdividing individuals: sex and smoking habits Outcome: FEV Possible explanations for interaction: biologically different effects of smoking on males and females perhaps the women do not smoke as much as the men perhaps the effect is relative (to be expressed in %) Here, we see an interaction between sex and smoking. ANOVA, April ANOVA, April Example: The effect of smoking on birth weight

11 ANOVA, April ANOVA, April Interaction: There is an effect of smoking, but only for those who have been smoking for a long time. There is an effect of duration, and this effects increases with amount of smoking The effect of duration depends upon... amount of smoking and the effect of amount depends upon... duration of smoking Example: Fibrinogen after spleen operation 3 rats are randomized, in ways 7 have their spleen removed (splenectomy=yes/no) 8/7 in each group are kept in high altitude (place=altitude/control) Outcome: Fibrinogen level in mg at day ANOVA, April ANOVA, April 008 The usual additive model: Y spr = µ + α s + β p + ε spr, ε spr N(0, σ ) splenectomy (s=yes/no) and place (p=altitude/control) have an additive effect. Model with interaction Y spr = µ + α s + β p + γ sp + ε spr, ε spr N(0, σ ) Here, we specify an interaction between splenectomy and place, i.e. the effect of living in a high altitude may be thought to depend upon whether or not you have an intact spleen. and vice versa..

12 ANOVA, April 008 ANOVA, April Two-way ANOVA with interaction in SAS: Dependent Variable: fibrinogen proc glm data=ex_fibrinogen; class splenectomy place; model fibrinogen=place splenectomy place*splenectomy / solution; output out=model p=predicted r=residual; Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE fibrinogen Mean Source DF Type I SS Mean Square F Value Pr > F The GLM Procedure Class Level Information Class Levels Values splenectomy no yes place altitude control Number of observations 3 place splenectomy splenectomy*place Source DF Type III SS Mean Square F Value Pr > F place splenectomy splenectomy*place ANOVA, April 008 ANOVA, April Standard Parameter Estimate Error t Value Intercept.7 B place altitude B place control B.. splenectomy no 0. B splenectomy yes B.. splenectomy*place no altitude B splenectomy*place no control B.. splenectomy*place yes altitude B.. splenectomy*place yes control B.. Parameter Pr > t Intercept <.000 place altitude 0.00 place control. splenectomy no splenectomy yes. splenectomy*place no altitude splenectomy*place no control. splenectomy*place yes altitude. splenectomy*place yes control. The reference levels are place=control, splenectomy=yes (they come last in the alphabet) so the expected fibrinogen level for these animals is intercept=.7 For all other groups, we have to add one or more extra estimates, as shown in the table below: NOTE: The X X matrix has been found to be singular, and a generalized inverse was used to solve the normal equations. Terms whose estimates are followed by the letter B are not uniquely estimable.

13 ANOVA, April 008 ANOVA, April place splenectomy control altitude.7.7 yes = no = 3. = 0.87 Model checking Variance homogeneity may be judged from a one-way anova: The GLM Procedure Class Level Information Class Levels Values group no_altitude no_control yes_altitude yes_control Number of observations 3 Levene s Test for Homogeneity of fibrinogen Variance ANOVA of Squared Deviations from Group Means Sum of Mean Source DF Squares Square F Value Pr > F group 3.39E Error 30.53E No reason to suspect inhomogeneity ANOVA, April ANOVA, April Normality assumption for residuals (proc univariate normal) In the two-way anova, the interaction was not significant (P=0.77), so we omit it from the model: Tests for Normality Test --Statistic p Value Shapiro-Wilk W 0.99 Pr < W Kolmogorov-Smirnov D 0.88 Pr > D >0.500 Cramer-von Mises W-Sq Pr > W-Sq 0.35 Anderson-Darling A-Sq Pr > A-Sq proc glm data=ex_fibrinogen; class splenectomy place; model fibrinogen=place splenectomy / solution clparm; Dependent Variable: fibrinogen Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE fibrinogen Mean No reason to suspect non-normality Source DF Type III SS Mean Square F Value Pr > F place splenectomy

14 ANOVA, April ANOVA, April Standard Parameter Estimate Error t Value Pr > t Intercept B <.000 place altitude 8.73 B place control B... splenectomy no B splenectomy yes B... Residual plots Normality: Variance homogeneity: Removal of spleen leads to a decrease in fibronogen of approx mg at day Placing in altitude leads to an increase in fibronogen of approx 8.7 mg at day ANOVA, April ANOVA, April More complicated analyses of variances Three- or more-sided analysis of variance. Latin squares 3 I A B C II B C A III C A B (Cochran & Cox (957): Experimental Designs,.ed., Wiley) Cross-over designs Variance component models Example of a latin square: A rabbit experiment rabbits Vaccination at different spots on the back different orders of vaccination Swelling is area of blister (cm ) spot rabbit order swelling

15 ANOVA, April ANOVA, April Some illustrations: ANOVA, April ANOVA, April Fit 3-way analysis of variance, with additive effects proc glm; class rabbit spot order; model swelling=rabbit spot order; Dependent Variable: swelling Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE swelling Mean Source DF Type III SS Mean Square F Value Pr > F The GLM Procedure Class Level Information Class Levels Values rabbit 3 5 spot a b c d e f order 3 5 Number of observations 3 rabbit spot order The design is balanced, so the test of the effect of one variable (covariate) does not depend on which of the others are still in the model.

16 ANOVA, April How about possible interactions? proc glm; class rabbit spot order; model swelling=rabbit spot order spot*order; Dependent Variable: swelling Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total Source DF Type I SS Mean Square F Value Pr > F rabbit spot order spot*order There is no room for interaction, since there is only one observation for each combination of spot and order!