Analysis of variance and regression. November 22, 2007

Transcription

1 Analysis of variance and regression November 22, 2007

2 Parametrisations: Choice of parameters Comparison of models Test for linearity Linear splines

3 Lene Theil Skovgaard, Dept. of Biostatistics, Institute of Public Health, University of Copenhagen

4 Parametrisations, November Parameter: unknown quantity that we want to estimate (provide a good guess) the decrease in blood pressure following treatment A or the difference in decrease for treatment A and placebo the increase in insulin growth factor (IGF-1) with age Parametrisation: choice of which parameters are to enter the model description Re-parametrisation: shift to a new set of parameters

5 Parametrisations, November Most well known choice of parametrisation: Change of scale/units Do we measure height in cm or m? Take the relation of lung capacity versus height: fev1= α + β height If we change from measuring height in cm to m, we also change the regression coefficient (the parameter) from β to β = 100β Change of origin/intercept choice of another reference group in ANOVA subtracting e.g. 170cm from all height measurements Re-parametrisations do not change the model as such! same fitted values same confidence- and prediction limits but a possibility for interpretations of specific interest

6 Parametrisations, November What makes us choose a specific parametrisation? Ease - the program has some default parametrisations Estimation of specific quantities: - the potency of a drug, ED 50 or ED 90 Test of specific hypotheses - difference between treatment and placebo - difference in height for boys and girls at the age of 14

7 Parametrisations, November In the more advanced situations (beyond linearity) non-linear regression, logistic regression, correlated observations: Knowledge of distributional assumptions: - Some parameter estimates may be more normally distributed than others (and we like to be able to construct symmetric confidence intervals, using the standard error) In linear models the estimates have exact normal distributions (provided the model assumptions are met, of course...)

8 Parametrisations, November Example: A group consisting of 45 patients with Reumatoid Arthritis are randomised to one out of 6 possible treatments (treat): Placebo Aspirin One of 4 doses (dose) of an active anti-inflammatory drug which we shall denote X. Outcome: An index (Index) summing up the effectiveness of the treatment (decrease in various symptoms)

9 Parametrisations, November Outcome: Index-values: Reference: Woolson, R.F. & Clarke, W.R.: Statistical methods for the analysis of biomedical data. 2ed., Wiley, (Exercise 10.4 page 409)

10 Parametrisations, November How do we represent these data in SAS? Obs group type dose index 1 placebo placebo placebo placebo placebo placebo placebo placebo placebo placebo placebo placebo placebo placebo x20 active x20 active x20 active x25 active x25 active x25 active x25 active x25 active

11 Parametrisations, November Summary statistics The MEANS Procedure Analysis Variable : index N group Obs N Mean Std Dev Minimum Maximum aspirin placebo x x x x

12 Parametrisations, November We start by looking at the 4 X-groups only: Below, the outcome Index is plotted against Dose group.

13 Parametrisations, November Comparison of 4 dose groups: One-way ANOVA Model written as a multiple regression: Y = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3 + ǫ where the x s are socalled dummy variables: x 1 is 1 if subject i belongs to the first group, and 0 otherwise x 2 is 1 if subject i belongs to the second group, and 0 otherwise x 3 is 1 if subject i belongs to the third group, and 0 otherwise With this parametrisation, β 0 will correspond to the level for the last group (the reference group, here group 4); β 1 will be the difference in level between group 1 and group 4 β 2 will be the difference in level between group 2 and group 4 and so on...

14 Parametrisations, November Traditional One-way ANOVA in SAS: proc glm data=drug; where type= active ; class group; model index=group / solution; run; which yields the output: The GLM Procedure Class Level Information Class Levels Values group 4 x10 x15 x20 x25 Number of Observations Used 25

15 Parametrisations, November The GLM Procedure Dependent Variable: index Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total R-Square Coeff Var Root MSE index Mean Source DF Type I SS Mean Square F Value Pr > F group <.0001 Source DF Type III SS Mean Square F Value Pr > F group <.0001

16 Parametrisations, November Standard Parameter Estimate Error t Value Pr > t Intercept B <.0001 group x B <.0001 group x B <.0001 group x B <.0001 group x B... NOTE: The X X matrix has been found to be singular, and a generalized inverse was used to solve the normal equations. Terms whose estimates are followed by the letter B are not uniquely estimable. The B to the right of the estimates is explained in the NOTE It simply means: By renaming the group levels/names, we may get a different parametrisation!

17 Parametrisations, November We here disregard the problem of variance heterogeneity: proc glm data=drug; where type= active ; class group; model index=group / noint solution; means group / hovtest=levene; run; from which we get Levene s Test for Homogeneity of index Variance ANOVA of Squared Deviations from Group Means Sum of Mean Source DF Squares Square F Value Pr > F group Error A clear indication that the variance increases with dose. Logarithms?

18 Parametrisations, November Same model, now parametrised with one level for each group: proc glm data=drug; where type= active ; class group; model index=group / noint solution; run; now yielding instead: Dependent Variable: index Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Uncorrected Total

19 Parametrisations, November R-Square Coeff Var Root MSE index Mean Source DF Type I SS Mean Square F Value Pr > F group <.0001 Source DF Type III SS Mean Square F Value Pr > F group <.0001 Standard Parameter Estimate Error t Value Pr > t group x <.0001 group x <.0001 group x <.0001 group x <.0001 The tests now refer to the hypothesis of a zero level (which is not interesting)

20 Parametrisations, November Parametrisations in One-way ANOVA One level (µ 4 ) for the reference group (the last, numerically or alphabetically), supplemented with differences from this reference group to each of the remaining groups (β 1, β 2, β 3 ) Y gi = µ 4 + β g + ε gi, good for testing of identity and certain pairwise comparisons β i = µ i µ 4 One level for each group Y gi = µ g + ε gi good for estimation, not suited for testing!!

21 Parametrisations, November Estimate statements in GLM If we want to compare dose 10 with dose 15: proc glm data=drug; where type= active ; class group; model index=group / noint solution; estimate dose 15 vs. dose 10 group ; run; from which we get Standard Parameter Estimate Error t Value Pr > t dose 15 vs. dose <.0001

22 Parametrisations, November We return to the scatter plot, now with a linear regression line Can we use a simple model, saying that the dose effect is linear?

23 Parametrisations, November We look at the simple linear regression model: Y gi = α + βx gi + ε gi, ε gi N(0, σ 2 ) where Y gi denotes Index, X gi denotes Dose, α is the intercept (with the vertical axis), β is the slope of the line and ε gi is the distance from observation to line, measured vertically (the residual), for individual i in drug group g Program statements: proc reg data=drug; where type= active ; model index=dose / clb; run;

24 Parametrisations, November The output becomes: The REG Procedure Dependent Variable: index Number of Observations Used 25 Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var

25 Parametrisations, November Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept <.0001 dose <.0001 Parameter Estimates Variable DF 95% Confidence Limits Intercept dose Is this a reasonable description? Can we test the linearity? Is this model almost as good as a quadratic model the ANOVA model

26 Parametrisations, November Quadratic fit: Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept dose <.0001 dose

27 Parametrisations, November Model reduction - F test for comparison of explained variation We have to compare two models: Model 1 The ANOVA model with 4 separate groups Model 2 The linear regression model with actual dose as covariate Note: The models have to be nested, i.e. one is derived from the other, typically be demanding some parameters to be equal to each other or equal to zero (e.g. remove effects ). Indeed, the linear regression model is a special case of the ANOVA model, in which all contrast (µ 2 µ 1, µ 3 µ 2, µ 4 µ 3 ) are equal.

28 Parametrisations, November Sums of squares Model Sum Sq(Model): i (ŷ i ȳ) 2 Explained variation: How much do the predicted values vary? (the bigger the better) Residuals Sum Sq(Residual): i (y i ŷ i ) 2 Unexplained variation (residual variation): How large are the deviations from the model? (the smaller the better)

29 Parametrisations, November Look at changes in the explained variation Sum Sq(Model). How much less is explained by the simpler model? Sum Sq = Sum Sq(Model 1 ) Sum Sq(Model 2 ) > 0, always (since more parameters can always explain more of the variation).

30 Parametrisations, November Mean Sq = Sum Sq/ df F = Mean Sq Mean Sq(Residual) How large can this get, just by chance/coincidence? F F(df 2 df 1, df 1 ) where df 1 and df 2 denote the degrees of freedom for the Mean Sq(Residual) for the two models.

31 Parametrisations, November Direct calculation of F-test statistic: Sum Sq(Model) for ANOVA model: , with 3 df Mean Sq(Residual) for ANOVA model: , with 21 df (df 1 = 21) Sum Sq(Model) for linear model: , with 1 df > (( )/2)/ [1] F(2, 21) corresponds to a P-value of 0.06 If we have many groups, the test is not very powerful

32 Parametrisations, November We may trick SAS into performing our comparison directly by including dose simultaneously as a linear effect and as a class effect: proc glm data=drug; where type= active ; class group; model index=group dose / solution; run; In this way, the term group represents the variation of dose group means around a straight line

33 Parametrisations, November The output becomes: Dependent Variable: index Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Source DF Type I SS Mean Square F Value Pr > F dose <.0001 group

34 Parametrisations, November Source DF Type III SS Mean Square F Value Pr > F dose group Standard Parameter Estimate Error t Value Pr > t Intercept B dose B <.0001 group x B group x B group x B... group x B... Note: The effect group now has only 2 degrees of freedom, since we are not testing equality, but linearity!

35 Parametrisations, November Contrast statement in GLM for test of linearity (a bit tricky) proc glm data=drug; where type= active ; class group; model index=group / solution; contrast dev linearity group , group ; run; gives an extra line of output: Contrast DF Contrast SS Mean Square F Value Pr > F dev linearity

36 Parametrisations, November Explanation for this contrast: The linear regression model is a special case of the ANOVA model, in which all contrast (µ 2 µ 1, µ 3 µ 2, µ 4 µ 3 ) are equal. µ 2 µ 1 = µ 3 µ 2 (µ 3 µ 2 ) (µ 2 µ 1 ) = µ 1 2µ 2 + µ 3 = 0 µ 3 µ 2 = µ 4 µ 3 (µ 4 µ 3 ) (µ 3 µ 2 ) = µ 2 2µ 3 + µ 4 = 0

37 Parametrisations, November Example with comparison of regression lines Analysis of covariance, ANCOVA (just a name used for a model with one Class variable and one quantitative variable.) Relate weight to age and gender, in the well known Juul-data, tanner=1 The GLM Procedure Dependent Variable: weight Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total

38 Parametrisations, November R-Square Coeff Var Root MSE weight Mean Source DF Type I SS Mean Square F Value Pr > F sex <.0001 age <.0001 age*sex Source DF Type III SS Mean Square F Value Pr > F sex age <.0001 age*sex

39 Parametrisations, November Standard Parameter Estimate Error t Value Pr > t Intercept B sex female B sex male B... age B <.0001 age*sex female B age*sex male B... NOTE: The X X matrix has been found to be singular, and a generalized inverse was used to solve the normal equations. Terms whose estimates are followed by the letter B are not uniquely estimable. Interpretation: Boys increase their weight more rapidly than girls: approx. 692g more per year, CI=(216, 1167)

40 Parametrisations, November Weight related to age: Again: Problems with variance homogeneity...

41 Parametrisations, November Estimate the weight difference at the age of 14: proc glm data=juul; where tanner=1; class sex; model weight=sex age age*sex / solution clparm; estimate difference at 14 sex -1 1 age*sex ; run; with the output: Standard Parameter Estimate Error t Value Pr > t difference at Parameter 95% Confidence Limits difference at

42 Parametrisations, November Simultaneous test of both sex-effects proc glm data=juul; where tanner=1; class sex; model weight=sex age age*sex / solution clparm; contrast sex and sex*age sex -1 1, age*sex -1 1; run; which yields: Contrast DF Contrast SS Mean Square F Value Pr > F sex and sex*age

43 Parametrisations, November SAS-computation of both lines simultaneously Subgroup analysis Keep the interaction term sex*age Leave out the marginal effect age this will merge the marginal effect into the interaction term Leave out the intercept (use option noint in textsfmodel) this will merge the intercept into the sex effect proc glm data=juul; where tanner=1; class sex; model weight=sex age*sex / noint solution clparm; run;

44 Parametrisations, November Output: The GLM Procedure Dependent Variable: weight Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Uncorrected Total R-Square Coeff Var Root MSE weight Mean Source DF Type I SS Mean Square F Value Pr > F sex <.0001 age*sex <.0001

45 Parametrisations, November Source DF Type III SS Mean Square F Value Pr > F sex <.0001 age*sex <.0001 Standard Parameter Estimate Error t Value Pr > t sex female <.0001 sex male age*sex female <.0001 age*sex male <.0001 Parameter 95% Confidence Limits sex female sex male age*sex female age*sex male

46 Parametrisations, November Same model, 2 different parametrisations: proc glm data=juul; where tanner=1; class sex; model weight=sex age age*sex / solution; run; An intercept for the reference group (sex= male ) A difference from sex= female to sex= male An effect of age (slope) for the reference group A difference in slopes from sex= female to sex= male proc glm data=juul; where tanner=1; class sex; model weight=sex age*sex / noint solution; run; An intercept for each group (sex) An effect of age (slope) for each group (sex)

47 Parametrisations, November Now we include the Placebo group as Dose=0. Does the Placebo treatment fit in here? No, obviously not. Either we have a placebo effect or a threshold effect. But how could we make a formal test?

48 Parametrisations, November data drug; set drug; active_drug=(dose>0); run; proc glm data=drug; where group ne aspirin ; class active_drug; model index= dose active_drug / solution; run; Class Level Information Class Levels Values active_drug Number of Observations Used 34

49 Parametrisations, November Dependent Variable: index Sum of Source DF Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total R-Square Coeff Var Root MSE index Mean Source DF Type I SS Mean Square F Value Pr > F dose <.0001 active_drug <.0001 Source DF Type III SS Mean Square F Value Pr > F dose <.0001 active_drug <.0001

50 Parametrisations, November Standard Parameter Estimate Error t Value Pr > t Intercept B <.0001 dose <.0001 active_drug B <.0001 active_drug B... NOTE: The X X matrix has been found to be singular, and a generalized inverse was used to solve the normal equations. Terms whose estimates are followed by the letter B are not uniquely estimable. Interpretation: The placebo group (active_drug=0) lies an estimated above the expected 0-dose level.

51 Parametrisations, November Example: Serum IGF-1 for boys We have seen different age dependencies in separate Tanner stages. Is this simply due to a nonlinear age effect?

52 Parametrisations, November How can we model non-linear effects? Polynomials tend to be very wiggly (use i=rq or i=rc in the symbol-statement) Splines piecewise interpolations, often linear or cubic

53 Parametrisations, November Separate lines for each age group: Subdivide age into groups, using appropriate thresholds Fit linear effect of age in each age group The result is a 5 unconnected lines:

54 Parametrisations, November Linear splines: Subdivide age into groups, using appropriate thresholds Fit linear effect of age in each age group Make the linear pieces meet at the thresholds The result is a bended line:

55 Parametrisations, November data juul; set juul; /* define new intercept value */ age5=age-5; /* define number of years above certain threshold ages */ extra_age1=max(age-10,0); extra_age2=max(age-12,0); extra_age3=max(age-13,0); extra_age4=max(age-15,0); run; proc sort data=juul; by sex; run; proc reg data=juul; where age ge 5 and age le 20; by sex; model ssigf1=age5 extra_age10 extra_age12 extra_age13 extra_age15; run;

56 Parametrisations, November Definition af extra_age: age_ extra_ extra_ extra_ extra_ Obs age group age10 age12 age13 age

57 Parametrisations, November sex=male The REG Procedure Dependent Variable: ssigf1 Number of Observations Read 506 Number of Observations Used 362 Number of Observations with Missing Values 144 Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model <.0001 Error Corrected Total Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept <.0001 age extra_age extra_age extra_age extra_age <.0001

58 Parametrisations, November Can we reduce the model to a simpler one? To simple linearity... not reasonable Quadratic age effect... Why? proc glm data=juul; where age ge 5 and age le 20; by sex; model ssigf1=age5 extra_age10 extra_age12 extra_age13 extra_age15 / solution; contrast all extra_age10 1, extra_age12 1, extra_age13 1, extra_age15 1; run; Contrast DF Contrast SS Mean Square F Value Pr > F all <.0001

59 Parametrisations, November Test of adequacy of linear spline? We have to test it against a more complicated model, e.g. separate regressions for each age group Inclusion of tanner as a Class-variable...

60 Parametrisations, November Dose-response curves Example of a typical dose-response relation, for moderate doses We have almost linearity in this dose range:

61 Parametrisations, November For extreme doses we see a clear deviation from linearity and: smaller variation in the ends

62 Parametrisations, November Y axis: Logit transformed response, i.e. log(response/(100 response)) X axis: Logarithmic transformed dose proc gplot; plot logit*dose / haxis=axis1 vaxis=axis2 frame; axis1 logbase=2 logstyle=expand value=(h=2) minor=none label=(h=3 Dose on log scale ); axis2 value=(h=2) minor=none label=(a=90 R=0 H=3 Logit transformed response ); symbol v=circle i=sm60 h=3 c=black l=1 w=2; run; /* i=sm60 indicates a 60% smoothing */ We get a reasonable linearity on this scale

63 Parametrisations, November Theoretical dose response relation: This looks like a cumulative normal distribution a sigmoid shape a logistic curve

64 Parametrisations, November group=halothane Example from anaesthesia: 47 patients to be operated with two different anesthetics Halothane Neurolept Y: Twitch response at the ulnar nerve (at the thumb), in % X: Dose of muscle relaxantia patient dose depression group=neurolept patient dose depression

65 Parametrisations, November

66 Parametrisations, November Neurolept

67 Parametrisations, November Transformation to linearity in the usual way: ( ) twitch y : logit_twitch = log 100 twitch x : logdose = log(dose) Linear relation: logit_twitch = α + β logdose produces estimates of α and β, with corresponding standard errors (s.e.)

68 Parametrisations, November group=neurolept The REG Procedure Dependent Variable: logity Number of Observations Used 32 Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var

69 Parametrisations, November Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept <.0001 logdose <.0001 Parameter Estimates Variable DF 95% Confidence Limits Intercept logdose

70 Parametrisations, November We get estimates of α and β from the equation logit_twitch = α + β logdose But: What about the parameters of interest, i.e. ED 50 and ED 90? ( ED ˆ 50 = exp ˆαˆβ ) How do we calculate s.e.( ˆ ED 50 )? Reparametrisation: γ 1 = log(ed 50 ) γ 2 = log(ed 90 )

71 Parametrisations, November The model may then be written as: y= logit_twitch = logit(0.9) x γ 1 γ 2 γ 1 = x γ 1 γ 2 γ 1 This function is nonlinear in γ 1 and γ 2! Direct estimation of γ 1 and γ 2 using non-linear regression... more about this next week