General Linear Models. with General Linear Hypothesis Tests and Likelihood Ratio Tests

Transcription

1 General Linear Models with General Linear Hypothesis Tests and Likelihood Ratio Tests 1

2 Background Linear combinations of Normals are Normal XX nn ~ NN μμ, ΣΣ AAAA ~ NN AAμμ, AAAAAA A sum of squared, standardized Normals is Chi-square XX μμ ΣΣ 11 XX μμ ~ χχ nn 22 Regression estimates have a Normal dist n bb ~ NN ββ, σσ 2 XX XX 1 2

3 Background A General Linear Hypothesis Test (GLHT) can be written as: LLLL = hh Often, with hh = 00 NB: LL rr xx pp has row rank rr Recall nn pp MMMMMM σσ 2 = SSSSSS σσ 2 = ee ee σσ 2 ~ χχ 2 nn pp 3

4 Background CCCCCC ee, bb = E eebb E ee E bb = E eebb = 00 Zero covariance implies independence, for MVN only bb ee bb ee ee bb SSSSSS Just like XX ss 2 4

5 GLHT Contrast distribution is also Normal LLbb ~ NN hh, σσ 2 LL XX XX 1 LL Consider test statistic LLbb hh σσ 2 LL XX XX 1 LL 1 LLbb hh ~ χχ r 22 5

6 GLHT LLLL hh σσ 2 LL XX XX 1 LL 1 LLLL hh /rr nn pp MMMMMM σσ 2 /(nn pp) = LLLL hh LL XX XX 1 LL 1 LLLL hh rrmmmmmm χχ 22 r /rr ~ 22 χχ n p /(nn pp) = FF rr,nn pp Provided those two χχ 2 s are independent They are, since one a function of SSE and the other a function of the regression estimates 6

7 tl;dr You can test a set of contrasts with the following statistic FF oooooo = LLLL hh LL XX XX 1 LL 1 LLLL hh rrmmmmmm Under the null hypothesis, it has an FF rr,nn pp distribution 7

8 To R Studio

9 Applied Stats Algorithm Scientific question? Classify Study Response Variable Multi-Var No Unacceptable Univariate Numerical Categorical Censored Complete Predictor(s) Numerical Both Categorical Today 9

10 ANCOVA Model Often, we d like to conduct an ANOVA, but control for some numerical predictor as well This could be a blocking variable like age If the numerical predictor xx is correlated with YY, then xx will likely explain some of the variation in YY and MMSS EE will be reduced These are efficient designs Also a good technique for non-experimental studies 10

11 ANCOVA Model When we re interested in Factor A but controlling for covariate xx, we call this Analysis of Covariance Assumptions: Linear relationship between Y and xx All the usual other ANOVA assumptions Homogeneity of regression slopes We can use interactions to relax the last one and allow different slopes for different groups 11

12 Drug Testing Example Suppose we have NN patients, and we randomize into 3 treatment arms Drug A, Drug B, Placebo II AA = 1 if Drug A, 0 otherwise II BB = 1 if Drug B, 0 otherwise Reference coding scheme We also want to control for age, a known covariate that affects the response xx is age 12

13 Make a Table EE YY XX = xx = ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB A 1 0 (ββ 0 + ββ 2 ) + ββ 1 xx B 0 1 ββ 0 + ββ 3 + ββ 1 xx Placebo 0 0 ββ 0 + ββ 1 xx The effect of Drug A (or B) is a change to the intercept of ββ 2 (or ββ 3 ) We saw this already, but the same holds true when controlling for age 13

14 ANCOVA Can test contrasts controlling for covariates Valuable Sometimes very easy, sometimes can require a bit of algebra An easy example: Are responses to Drug A and B different, controlling for age?

15 ANCOVA Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB A 1 0 (ββ 0 + ββ 2 ) + ββ 1 xx B 0 1 ββ 0 + ββ 3 + ββ 1 xx Placebo 0 0 ββ 0 + ββ 1 xx HH 0 : ββ 2 = ββ 3 We saw this already, but the same holds true when controlling for age Q: Why is this (probably) better than ANOVA model without age? 15

16 ANCOVA Is the average response to Drugs A and B different from response to the placebo, controlling for age? Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB A 1 0 (ββ 0 + ββ 2 ) + ββ 1 xx B 0 1 ββ 0 + ββ 3 + ββ 1 xx Placebo 0 0 ββ 0 + ββ 1 xx HH 0 : ββ 2 + ββ 3 = 0 16

17 Show your work We want to avoid this kind of thing

18 Research Questions Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB A 1 0 (ββ 0 + ββ 2 ) + ββ 1 xx B 0 1 ββ 0 + ββ 3 + ββ 1 xx Placebo 0 0 ββ 0 + ββ 1 xx What null hypothesis would you test to answer: Is Drug A equivalent to Placebo, controlling for age? Are all three treatments arms equivalent, controlling for age? 18

19 Drug Testing Example With reference coding, regression coefficients are differences between group means and the reference group mean Conditional on the covariate With cell means coding, regression coefficients are the (conditional) group means With effect coding, regression coefficients are deviations from the average (conditional on age) population mean Use whichever scheme is most convenient 19

20 Cell means coding Covariate: Age = x Drug II AA II BB II PP ββ 1 II AA + ββ 2 II BB + ββ 3 II PP + ββ 4 xx A ββ 1 + ββ 4 xx B ββ 2 + ββ 4 xx Placebo ββ 3 + ββ 4 xx Parallel regression lines Equivalent to the model with intercept Regression coefficients for the dummy vars are the intercepts Easy to specify contrasts

21 Effect coding Covariate: Age = x Drug xx 2 xx 3 ββ 0 + ββ 1 xx + ββ 2 xx 2 + ββ 3 xx 3 A 1 0 ββ 0 + ββ 1 xx + ββ 2 B 0 1 ββ 0 + ββ 1 xx + ββ 3 Placebo -1-1 ββ 0 + ββ 1 xx ββ 2 ββ 3 Regression coefficients are deviations from the average conditional population mean (conditional on x) If the regression coefficients for all the dummy variables are zero, the categorical IV is unrelated to the DV, controlling for the covariates

22 ANCOVA We just fit a parallel lines model Or more generally, a parallel planes model What does that look like? Back to the soil 22

23 Farming Example (with a covariate) Suppose we also collect information on the amount of rainfall for each field Field is the experimental unit We can t control this, and we won t know the value when the experiment is set up So no possibility for RCB But it s easy to measure after setup Can we correct or control for the effect of rain? What effect will this have on our power to detect a 23 fertilizer effect?

24 Farming Example (with a covariate) x <- c(110,98,75,90, 85,80,110,130, 68,80,100,125, 72,90,85,115) farm <- data.frame(yield=y, fertil=a, rain=x) plot(yield ~ rain, data=farm) 24

25 Farming Example (with a covariate) require(ggplot2) qplot(rain, yield, data=farm, colour=fertil) + geom_smooth(method="lm", se=f) 25

26 Farming Example (Parallel planes model) It looks like the fertilizers have a different effect on crop yield, and have roughly the same relationship with rain Let s fit a parallel planes model and compare to vanilla ANOVA without the covariate 26

27 Farming Example (Parallel planes model) > anova(lm(yield ~ fertil, data=farm)) Analysis of Variance Table Response: yield Df Sum Sq Mean Sq F value Pr(>F) fertil * Residuals > anova(lm(yield ~ rain + fertil, data=farm)) Analysis of Variance Table Response: yield Df Sum Sq Mean Sq F value Pr(>F) rain e-05 *** fertil e-06 *** Residuals

28 Building to a GLM What if we want to allow non-parallel lines? Just fit an interaction between the covariate and the treatment groups EE YY XX = xx = ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB + ββ 4 xxii AA + ββ 5 xxii BB Make a table: Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB + ββ 4 xxii AA + ββ 5 xxii BB A 1 0 (ββ 0 + ββ 2 ) + (ββ 1 + ββ 4 )xx B 0 1 ββ 0 + ββ 3 + (ββ 1 + ββ 5 )xx Placebo 0 0 ββ 0 + ββ 1 xx 28

29 Interactions Interaction between independent variables means It depends. Relationship between one IV and the DV depends on the value of another IV. Can have Quantitative by quantitative Quantitative by categorical Categorical by categorical (Done)

30 General principle Interaction between A and B means Relationship of A to Y depends on value of B Relationship of B to Y depends on value of A The two statements are formally equivalent

31 Quantitative by Quantitative For fixed x 2 Both slope and intercept depend on x 2 And for fixed x 1, slope and intercept relating x 2 to E(Y) depend on the value of x 1

32 First Order Model Y X 2 X 1 32

33 Interaction Effects EY { } = 3+ 2X1+ 2X2 3XX EE YY XX 1 = 2 3XX 2 EE YY XX 2 = 2 3XX 1 Y X X

34 Quantitative by Categorical Separate regression line for each value of the categorical independent variable. Interaction means slopes of regression lines are not equal.

35 One regression Model Form a product of quantitative variable times each dummy variable for the categorical variable For example, three treatments and one covariate: x 1 is the covariate and x 2, x 3 are dummy variables

36 What to do if H 0 : β 4 =β 5 =0 is rejected How do you test Group controlling for x 1? A popular choice is to set x 1 to its sample mean, and compare treatments at that point Or, test equal regressions, in which mean response is the same for all values of the covariate(s). Why not set x 1 to sample mean of each group? 3 different values

37 Test for differences at xx 1

38 Hypothesis Tests Drug II AA II BB ββ 0 + ββ 1 xx + ββ 2 II AA + ββ 3 II BB + ββ 4 xxii AA + ββ 5 xxii BB A 1 0 (ββ 0 + ββ 2 ) + (ββ 1 + ββ 4 )xx B 0 1 ββ 0 + ββ 3 + (ββ 1 + ββ 5 )xx Placebo 0 0 ββ 0 + ββ 1 xx What null hypothesis would you test for: Equal slopes for all groups? Equal slopes for Drugs A and B? Equal slopes for Drugs A and Placebo? Interaction between group and covariate? Equal regression surfaces between groups? 38

39 Farming Example (Interaction model) > anova(lm(yield ~ rain + fertil, data=farm)) Analysis of Variance Table Df Sum Sq Mean Sq F value Pr(>F) rain e-05 *** fertil e-06 *** Residuals > anova(lm(yield ~ rain*fertil, data=farm)) Analysis of Variance Table Df Sum Sq Mean Sq F value Pr(>F) rain *** fertil e-05 *** Rain:fertil Residuals

40 Farming Example (Conclusions) Given amount of rainfall, we have very strong evidence that the 4 types of fertilizers do not have the same effect on crop yield Follow-up tests Given fertilizer type, we have very strong evidence that additional rainfall leads to an increased yield Is this test important? The effect of fertilizer type does not seem to depend on rainfall Can we visualize these effects? 40

41 Partial Boxplots Regular boxplots plot the distribution of Y for each level of a factor, irrespective of any other variables you may have measured If the variation in Y due to these variables is large, these differences will be masked If I want to partial out the effect of a covariate xx, can fit a regression model E[YY] ~ ββ 0 + ββ 1 xx A plot of the residuals of this model against the factor A is called a partial boxplot 41

42 Farming Example (Boxplots) A set of side-by-side boxplots for fertilizer doesn t show that much difference 42

43 Farming Example (Partial boxplots) The partial boxplots highlight the differences between fertilizers in a more obvious way 43

44 Added-Variable Plots We can do the same thing to determine the relationship between Yield and Rain, conditional on the fertilizer type Fit model RRRRRRRR ~ FFFFFFFFFFFF and obtain the residuals Fit model YYYYYYYYYY ~ FFFFFFFFFFFF and obtain the residuals Plot these residuals against each other Such a plot is called an Added-variable plot 44

45 Farming Example (Added-variable plot) Scatterplot plot(yield ~ rain, data=farm, xlab="rain", ylab="crop Yield") Added-Variable Plot fitx <- lm(rain ~ fertil, data=farm) fity <- lm(yield ~ fertil, data=farm) plot(fitx$residuals, fity$residuals) 45

46 > anova(lm(fity$residuals ~ fitx$residuals)) Analysis of Variance Table Response: fity$residuals Df Sum Sq Mean Sq F value Pr(>F) fitx$residuals e-07 *** Residuals > anova(lm(yield ~ fertil + rain, data=farm)) Analysis of Variance Table Response: yield Df Sum Sq Mean Sq F value Pr(>F) fertil e-05 *** rain e-06 *** Residuals

47 It s a Linear Model There is really no difference between the model we just developed and the one you learned in your regression course yy = XXXX + εε Both are considered linear models, and the design matrix XX can contain numerical regressors or indicator variables, in whatever parametrization is convenient 47

48 General Linear Models We can generalize even further (though we won t) to a multivariate response case YY = XXXX + UU This is the General Linear Model of statistics GLM, not to be confused with the other GLM Stick to univariate (general) linear models 48

49 Centering IVs Subtract mean (for entire sample) from each quantitative independent variable.

50 Properties of Centering When independent variables are centered, estimates and tests for intercepts are affected Relationships between independent variables and dependent variables are unaffected Estimates and tests for slopes are unaffected R 2 is unaffected Predictions and prediction intervals are unaffected

51 More Properties Suppose regression model has an intercept Residuals add up to zero Then if each independent variable is set to its sample mean value, Y-hat equals Y-bar, the sample mean of all the Y values. In this case, if all independent variables are centered by subtracting off their means, then the intercept equals Y-bar, exactly.

52 Comments Often, X=0 is outside the range of independent variable values, and it is hard to say what the intercept means in terms of the data. When independent variables are centered, the intercept is the average Y value for average value(s) of X. If there are both quantitative variables and categorical variables (represented by dummy variables), it can help to center just the quantitative variables.

53 Centering just the quantitative IVs Subtract mean (for entire sample) from each quantitative independent variable. Then, comparing intercepts is the same as comparing expected values for average X values. It s more convenient than testing linear combinations.

54 For Example Suppose you want to test for differences among population mean Y values when x 1 equals its sample mean value You could test H 0 : Or, center x 1 and test H 0 : β 2 = β 3 = 0

55 ANCOVA x 1 = Age x 2 = 1 if Drug A, Zero otherwise x 3 = 1 if Drug B, Zero otherwise Parallel regression lines (equal slopes): ANCOVA

56 What do you report? x 1 = Age x 2 = 1 if Drug A, Zero otherwise x 3 = 1 if Drug B, Zero otherwise

57 Set all covariates to their sample mean values And compute Y-hat for each group Call it an adjusted mean, or something like average university GPA adjusted for High School GPA. SAS calls it a least squares mean (lsmeans)

58 Significance Testing Overall F test for all the IVs at once T-tests for each regression coefficient: Controlling for all the others, does that IV matter? Test a collection of IVs controlling for another collection Most general: Testing whether sets of linear combinations of regression coefficients differ from specified constants.

59 Significance Testing Controlling for mother s education and father s education, are (any of) total family income, assessed value of home and total market value of all vehicles owned by the family related to High School GPA?

60 Statistical Models Starting with the basics and building towards the GLM 60

61 t-test One-sample Used to test if a mean equals some value Two-sample Used to test if the difference between groups equals some value Most commonly used to test if two populations have the same mean Paired Really a one-sample test, and a special case of a Repeated Measures design

62 One-way ANOVA Generalized two-sample t-test Used to test if k groups have the same mean Q: What can you do with a t-test that you cannot do with ANOVA? A: Account for different variances within groups 1-way ANOVA t-test

63 Factorial ANOVA Generalized one-way ANOVA Used to test if k groups have the same mean across multiple (crossed) factors Can also test for interactions between factors ANOVA 1-way ANOVA

64 Simple Linear Regression (SLR) Fits a straight line through a scatterplot Used to test if two numerical variables have a linear correlation It is linear in the parameters, not the variables Can account for transformations this way (log, etc.) ANOVA SLR

65 Multiple Linear Regression (MLR) Generalization of regression with several predictors Fit a hyperplane through a multi-dimensional scatterplot Used to test if any numerical variables have a linear relationship with the response Model building and interactions MLR ANOVA SLR

66 Analysis of Covariance (ANCOVA) Generalized ANOVA Used to test if k groups have the same mean across multiple factors, controlling for a numerical covariate Typically used when the covariate itself is not of interest ANCOVA ANOVA MLR

67 Multivariate ANOVA (MANOVA) Generalized ANOVA Used to test if k groups have the same mean across multiple factors AND multiple responses Not studied in this course MANOVA ANOVA

68 General Linear Model (GLM) Generalization of ANOVA, ANCOVA, and regression (and MANOVA, MANCOVA) Used to test whether factors or numerical predictors explain the variation in a response (responses), and interactions between either Many statistical problems can be solved with GLMs GLM ANCOVA MLR

69 General Linear Model (Univariate) yy = XXXX + εε yy is a vector of responses XX is the design matrix ββ is a vector of unknown parameters To be estimated εε is a vector of unknown random errors Usually considered Normal

70 General Linear Model (Multivariate) YY = XXXX + UU YY is a matrix of responses XX is the design matrix Both XX and YY could involve transformations BB is a matrix of unknown parameters To be estimated UU is a matrix of unknown random errors Usually considered multivariate Normal

71 Generalized Linear Models (GLM) Suppose we want to model our random vector with some distribution other than Normal Can use a GLM to do this Logistic regression Poisson regression Can keep going with this hierarchy GLMM, GAM, GEE, We re getting ahead of ourselves here 71