Dr. Junchao Xia Center of Biophysics and Computational Biology. Fall /8/2016 1/38

Transcription

1 BIO5312 Biostatistics Lecture 11: Multisample Hypothesis Testing II Dr. Junchao Xia Center of Biophysics and Computational Biology Fall /8/2016 1/38

2 Outline In this lecture, we will continue to discuss more topics in Chapter 12: One-way ANOVA Overall Comparison of Group Means Comparisons of Specific Groups Multiple Comparisons Bonferroni Approach False Discovery Rate Relationship Between Multiple Testing and Regression Analysis Two-way ANOVA 11/8/2016 2/38

3 One-Way ANOVA Fixed-Effects Model Suppose there are k groups with n i observations in the ith group. The jth observation in the ith group will be denoted by y ij. Let us assume the following model. y ij = µ + i + e ij Where µ is a constant, i is a constant specific to the ith group, and e ij is an error term, which is normally distributed with mean 0 and variance 2. A typical observation from the ith group is normally distributed with mean µ+ i and variance 2. i will be a random variable for the random-effects model. NS: Nonsmokers PS: Passive Smokers NI: Noninhaling Smokers LS: Light Smokers MS: Moderate Smokers HS: Heavy Smokers Pulmonary function was assessed by force mid-expiratory Flow (FEF). How do you compare the means of the six groups? 11/8/2016 3/38

4 One-Way ANOVA: Assumptions In a one-way analysis of variance, or a one-way ANOVA model, the means of an arbitrary number of groups, each of which follows a normal distribution with the same variance, can be compared. Whether the variability in the data comes mostly from variability within groups or can truly be attributed to variability between groups can also be determined. Interpretation of the parameters of a one-way ANOVA fixed-effects model 1.µ represents the underlying mean of all groups taken together. 2. i represents the difference between the mean of the ith group and the overall mean. 3.e ij represents random error about the mean µ+ i for an individual observation from the ith group. 11/8/2016 4/38

5 One-Way ANOVA: Decomposition of Variability y ij y ( y ij y i ) ( y i y), k Total SS Within SS Between SS n j i 1 j 1 ( y y) j i 1 j 1 j i 1 j 1 Short computational form for the Between SS and Within SS ij 2 k n ( y ij y ) i 2 k n ( y i y) where y..= sum of the observations across all groups, i.e., the grand total of all observations over all groups and n = total number of observations over all groups. 11/8/2016 5/38 2

6 Overall Comparison of Group Means: F Test Between Mean Square = Between MS = Between SS/(k-1) Within Mean Square = Within MS = Within SS/(n-k) The significance test is based on the ratio of Between MS to Within MS. If this ratio is large, then we reject H 0 ; if it is small, we accept H 0. Under H 0, the ratio follows an F distribution with k - 1 and n - k degrees of freedom. Overall F test for One-way ANOVA To test the hypothesis H 0 : i = 0 for all i vs. H 1 : at least one i 0, use the following procedure. 1.Compute the Between SS, Between MS, Within SS, and Within MS. 2.Compute the test statistic F = Between MS/Within MS, which follows an F distribution with k - 1 and n - k df under H 0. If F > F k-1,n-k,1- then reject H 0. If F F k-1,n-k,1- then accept H 0. 3.The exact p-value is given by the area to the right of F under an F k-1,n-k distribution = Pr(F k-1,n-k > F) 11/8/2016 6/38

7 Comparisons of Specific Groups: t Test for Comparison of Pairs Overall F test: H 0 : all group means are equal vs. H 1 : at least two group means are different. This test lets us detect when at least two groups have different underlying means, but it does not let us determine which of the groups have means that differ from each other. t Test for Comparison of Pairs of Groups on One-Way ANOVA (Least Significant Difference (LSD) Method) If we want to test whether groups 1 and 2 have means that are significantly different from each other. Under either hypothesis, Y 1 ~N(µ+ 1, 2 /n 1 ), And Y 2 ~N(µ+ 2, 2 /n 2 ) The difference of the sample means will be used as a test criterion H 0 : 1-2 =0 t n-k distribution under H 0. If 2 is generally unknown, the best estimate of it, denoted by s 2, is substituted by S 2 = [(n 1-1)s (n 2-1)s 22 ]/(n 1 +n 2-2), or by pooled estimate of the variance for oneway ANOVA: 11/8/2016 7/38

8 Comparisons of a Collection of Specific Groups: Linear Contrasts A linear contrast (L) is any linear combination of the individual group means such that the linear coefficients add up to 0. Specifically, t Test for Linear Contrasts in One-Way ANOVA Suppose we want to test the hypothesis H 0 : µ L = 0 vs. H 1 : µ L 0, using a two-sided test with significance level = where y ij ~ N(µ+ i, 2 ), 1.Compute the pooled estimate of the variance s 2 = Within MS from the one-way ANOVA. 2.Compute the linear contrast 3.Compute the test statistic with t n-k distribution, 11/8/2016 8/38

9 Multiple Comparisons Bonferroni Approach Comparison of interested groups can not be specified before looking at the actural data. Several procedures referred to as multiple-comparisons procedures, ensure that too many falsely significant difference are not declared. It ensures that the overall probability of declaring any significant differences between all possible pairs of groups is maintained at some fixed significance level. One of the simplest such procedure is the Bonferroni adjustment. Suppose we want to compare two specific groups, arbitrarily labeled as group 1 and group 2, among k groups. To test the hypothesis H 0 : 1 = 2 vs. H 1 : 1 2, use the following procedure 1.Compute the pooled estimate of the variance s 2 = Within MS from the one-way ANOVA. 2.Compute the test statistic 3.For a two-sided level test, let * = /( k 2) If t > t n-k,1- */2 or t < t n-k, */2 then reject H 0. If t > t n-k, */2 t t n-k,1- */2 then accept H 0. This test is called the Bonferroni multiple-comparisons procedure. defined a significance level for the family of m=( k 2) tests. 11/8/2016 9/38

10 Multiple Comparisons Bonferroni Approach In a study with k groups, there are c=( k 2) possible two-group comparisons. Suppose each two-group comparison is conducted at the * level of significance. Let E be the event that at least one of the two-group comparisons is statistically significant. Pr(E) = Pr(none of the two-group comparisons is statistically significant) = 1-. If each of the two-group comparisons were independent, then from the multiplication law of probability, Pr(E) = (1- *) c where c = ( k 2) When * is small, (1- *) c 1-c *. Therefore, 1- = (1- *) c 1-c *. 11/8/ /38

11 Multiple Comparisons Bonferroni Approach The results of the multiple-comparisons procedure are typically displayed as shown. A line is drawn between the names or numbers of each pair of means that is not significantly different. This plot allows us to visually summarize the results of many comparisons of pairs of means in one concise display. Multiple-comparisons procedures are more strict than ordinary t tests if more than two means are being compared. As k increases, c = ( k 2) increases and therefore * = /c decreases. The critical value, t n-k,1- */2, therefore increases because as k increases, the degrees of freedom (n-k) decreases and the percentile 1- */2 increases, both of which result in a larger critical value. 11/8/ /38

12 Multiple Comparisons.vs. LSD procedure When is multiple-comparisons procedure used over LSD procedure? It is recommended that multiple-comparisons procedures should be used if there are many groups and not all comparisons between individual groups have been planned in advance. If there are relatively few groups and only specific comparisons of interest are intended, which have been planned in advance, preferably stated in a protocol, then ordinary t tests (i.e., the LSD procedure) is used. Multiple-comparisons procedure is applicable for comparing pairs of means. In some situations, linear contrasts involving more complex comparisons than simple contrasts based on pairs of means are of interest. If linear contrasts, which have not been planned in advance, are suggested by looking at the data, then a multiple comparisons procedure might be used to ensure that under H 0, the probability of detecting any significant linear contrast is no larger than. 11/8/ /38

13 Multiple-Comparisons for Linear Contrasts: Scheffé s Procedure Suppose we want to test the hypothesis H 0 : µ L = 0 vs. H 1 : µ L 0, at significance level, where We have k groups, with n i subjects in the ith group and a total of subjects overall. To use Scheffé s multiple-comparisons procedure in this situation, perform the following steps: 1.Compute the test statistic 2.If t > c 2 = (k-1) F k-1,n-k,1- or t < c 1 = - (k-1) F k-1,n-k,1- then reject H 0. If c 1 t c 2, then accept H 0. 11/8/ /38

14 General Procedure for Comparing the Means of k Groups of Samples 11/8/ /38

15 The False Discovery Rate Cardiovascular Disease, Genetics A subsample of 520 cases of cardiovascular disease (CVD) and 1100 controls was obtained among men in a prospective cohort study. This type of study is called a nested case control study. Baseline blood samples were obtained from men in the subsample and analyzed for 50 candidate singlenucleotide polymorphisms (SNPs). Each SNP was coded as 0 if homozygous wild type (the most common), 1 if heterozygote, and 2 if homozygous mutant. The association of each SNP with CVD was assessed using contingency-table methods. A chi-square test for trend was run for each SNP. This yielded 50 separate p - values. If the Bonferroni approach in Equation were used, then α* =.05/50 =.001. Thus, with such a low value for α* it is likely that very few of the hypotheses would be rejected, resulting in a great loss in power. Instead, an alternative approach based on the false-discovery rate (FDR) was used to control for the problem of multiple testing. 11/8/ /38

16 The False Discovery Rate In some settings, particularly in genetic studies with many hypotheses, control of the experiment-wise type I error sometimes does not seem a reasonable approach to controlling for multiple comparisons because it results in very conservative inferential procedures. The FDR approach was developed by Benjamini and Hochberg. The primary goal is not to control the overall experiment-wise type I error rate. The FDR attempts to control the proportion of false-positive results among reported statistically significant results. An advantage of the FDR approach is that is less conservative than the Bonferroni procedure and as a result yields more power to detect genuine positive effects. 11/8/ /38

17 False-Discovery-Rate (FDR) Testing Procedure 1. Suppose we have conducted k separate tests with p-values = p 1,, p k 2. For convenience we will renumber the tests so that p 1 p 2 p k 3. Define q i = kp i /i, i = 1,,k where i = rank of the p-values among the k tests. 4. Let FDR i = false-discovery rate for the ith test be defined by min (q i,, q k ) 5. Find the largest i such that FDR i < FDR 0 = critical level for the FDR (usually 0.05). 6. Reject H 0 for the hypotheses 1,, i, and accept H 0 for the remaining hypotheses. 11/8/ /38

18 False-Discovery-Rate (FDR): Example Stem-and-leaf plot and box plot of the p-values from the tests of 50 SNPs. 11/8/ /38

19 False-Discovery-Rate (FDR): Example 11/8/ /38

20 Case Study: Effects of Lead Exposure The mean and standard deviation of mean finger-wrist tapping score (MAXFWT) for each group are given. MAXFWT scores are similar in the currently exposed and previously exposed groups and are lower than the corresponding mean score in the control group. 11/8/ /38

21 Case Study: Box Plots 11/8/ /38

22 Case Study: Effects of Lead Exposure Overall F test p > 0.05: The results are considered not statistically significant. 0.01< p < 0.05:* The results are statistically significant < p < 0.01: ** The results are highly significant. p < 0.001: *** The results are very highly significant. 11/8/ /38

23 Case Study: Comparison Group Means (LSD) 11/8/ /38

24 Case Study: Use of Dummy Use of dummy variables to represent a categorical variable with k categories Suppose we have a categorical variable C with k categories. To represent that variable in a multiple-regression model, we construct k -1 dummy variables of the form: The category omitted (category 1) is referred to as the reference group. 11/8/ /38

25 Case Study: Multiple Regression Model The subjects in each category of C have a unique profile in terms of x 1,, x k-1. To relate the categorical variable C to an outcome variable y, we use the multiple regression model y = + 1 x x k-1 x k-1 + e The average value of y for subjects in category 1 (the reference category) =, the average value of y for subjects in category 2 = represents the difference between the average value of y for subjects in category 2 vs. the average value of y for subjects in the reference category. j represents the difference between the average value of y for subjects in category (j+1) vs. the reference category, j = 1,.., k-1. One-way ANOVA hypothesis H 0 : all underlying group means are the same vs. H 1 : at least two underlying group means are different. Multiple-regression Hypothesis: H 0 : all j =0 vs. H 1 : at least one of j 0. 11/8/ /38

26 Case Study: Relationship between Multiple Linear Regression and ANOVA Suppose we wish to compare the underlying mean among k groups where the observations in group j are assumed to be normally distributed with mean = µ j = µ+ j and variance = 2. To test the hypothesis H 0 : j = 0 for all j = 1,,k vs. H 1 : at least two j are different, we can use one o the two equivalent procedures: (1) We can perform the overall F test for one-way ANOVA. (2) Or we can set up a multiple-regression model of the form Where y is the outcome variable and x j = 1 if a subject is in group (j+1) and = 0 otherwise, j = 1,, k-1. The Between SS and Within SS for the one-way ANOVA model in procedure 1 are the same as the Regression SS and residual SS for the multiple linear-regression model in procedure 2. The F statistics and p- values are the same as well. 11/8/ /38

27 Case Study: Relationship between Multiple Linear Regression and ANOVA To compare the underlying mean of the (j+1)th group vs. the reference group, we can use one of the two equivalent procedures: (3) We can use the LSD procedure based on one-way ANOVA, where we compute the t statistic And s 2 = Within MS, (4) Or we can compute the t statistic The test statistic and the p-values are the same under procedures 3 and 4. 11/8/ /38

28 Case Study: Effects of Lead Exposure Relationship between Multiple Linear Regression and ANOVA To compare the underlying mean of the (j + 1)th and the (l+1)th groups, we can use one of two equivalent procedures: (5) We can use the LSD procedure based on one-way ANOVA, whereby we compute the t statistic and s 2 = Within MS (6) Or The standard error of b j b l and test statistic t can usually be obtained as a linear-contrast option for multiple-regression programs available in most statistical packages. Another way to compute se(b j b l ) is to print out the variance-covariance matrix of the regression coefficients, which is an option in most statistical packages. If there are k regression coefficients, then the (j, j)th element of this matrix is the variance of b j =Var(b j ). The (j, l)th element of this matrix is the covariance between b j and b l = Cov(b j, b l ). Then, we have Var(b j b l ) = Var(b j ) + Var(b l ) 2 Cov(b j, b l ) and se(b j b l ) = Var(b j -b l ). 11/8/ /38

29 Case Study: Relationship between Multiple Linear Regression and ANOVA 11/8/ /38

30 Two-Way ANOVA In one-way ANOVA, only one variable is used to defined the groups. In some instances, the groups being considered can be classified by two variables, arranged in the form of an RxC contingency table. To assess the effects of each variable after controlling for the effects of the other variable we use a technique called the two-way ANOVA. An interaction effect between two variables is defined as one in which the effect of one variable depends on the level of the other variable. SV: Strict Vegetarians, eat no animal products LV: Lactovegetarians, eat dairy products but no other animal foods NOR: Normals eat a standard US diet Hyertension, Nutrition Case: mean systolic blood pressure(sbp) by dietary group and sex 11/8/ /38

31 Two-way ANOVA General model can be written as Where y ijk = µ + i + j + ij + e ijk Y ijk is the SBP of the kth person in the ith dietary group and the jth sex group µ is a constant Two-Way ANOVA: General Model i is a constant representing the effect of dietary group j is a constant representing the effect of sex group ij is a constant representing the interaction effect between dietary group and sex e ijk is an error term, which is assumed to be normally distributed with mean 0 and variance 2 By convention 11/8/ /38

32 y y i Decomposition of Variability : overall mean from all data : mean of the ith row y j y ij : mean for theith row and : mean of the jth column jth column error term row effect column effect interaction effect y ijk y ( y ijk y ij ) ( y i y ) ( y j y ) ( y ij y i y j y ) r 2 T i, j i 1 j 1 k 1 S r c, Total SS c n n ( y i, j i 1 j 1 k 1 S ijk (( y 2 I y ij y ) S 2 i 2 B1 ) ( y, SS interaction,between SS for G1 r i, j i 1 j 1 k 1 j c n y ( y )) i 2 y r ) c 2 i, j i 1 j 1 k 1 S 2 E n S r 2 B2 ( y, Between SS for G2 c i, j i 1 j 1 k 1 ijk n y ( y ij, SS Error ) i 2 y ) 2 11/8/ /38

33 Hypothesis Testing in Two-Way ANOVA To test the following hypotheses 1. Test for the presence of row effects: H 0 : all i = 0 vs. H 1 : at least one i 0. This is at least for the effect of dietary group on SBP level after controlling for the effect of sex. 2. Test for the presence of column effects: H 0 : all j = 0 vs. H 1 : at least one j 0. this is a test for the effect of sex on SBP level after controlling for the effect of dietary group. 3. Test for the presence of interaction effects: H 0 : all ij = 0 vs. H 1 : at least one ij 0. this is a test of whether or not there is a differential effect of dietary group between males and females. 11/8/ /38

34 Two dummy or indicator variables were set up to represent study group (x1,x2) where x1 = 1 if a person is in the first (SV) group = 0 otherwise x2 = 1 if a person is in the second (LV) group = 0 otherwise Normal group is the reference group. A variable x3 represents sex x3 = 1 if male = 0 if female. Two-Way ANOVA: Example Then, the multiple regression model can be written as y = + 1 x x x 3 + e 11/8/ /38

35 Two-Way ANOVA: Example 11/8/ /38

36 The Kruskal-Wallis Test for Non-Normal Distribution Data When we want to compare means among more than two samples, but either the underlying distribution is far from being normal or we have ordinal data. In such situations, a nonparametric alternative to the oneway ANOVA is used. 11/8/ /38

37 The Kruskal-Wallis test To compare the means of k samples (k > 2) using nonparametric methods, use the following procedure 1. Pool the observations over all samples, thus constructing a combined sample of size N = n i 2. Assign ranks to the individual observations, using the average rank in the case of tied observations 3. Compute the rank sum R i for each of the k samples. 4. If there are ties or not, compute the test statistic respectively, Where t j refers to the number of observations (i.e., the frequency) with the same value in the jth cluster of tied observations and g is the number of tied groups. 5. For a level test, if H > 2 k-1,1- then reject H 0. Otherwise accept H To assess statistical significance, the p-value is given by p =Pr( 2 k-1 > H) This test procedure should be used only if minimum n i 5. 11/8/ /38

38 Summary In this lecture for Chapter 12, we discussed One-way ANOVA methods enable us to relate a normally distributed outcome variable to the levels of a single categorical independent variable. Two different ANOVA models based on a fixed- or random-effects model were considered. In a fixed-effects model, the levels of categorical variable are determined in advance. A major objective of this design is to test the hypothesis that the mean level of the dependent variable is different for different groups defined by the categorical variable. More complex comparisons such as testing for dose-response relationships involving more than two levels of the categorical variable can also be accomplished using linear-contrast methods. In two-way ANOVA we jointly compare the mean levels of an outcome variable according to the levels of two categorical variables. 11/8/ /38

39 The End 11/8/ /38