Analysis of Variance and Contrasts

Transcription

1 Analysis of Variance and Contrasts Ken Kelley s Class Notes 1 / 103

2 Lesson Breakdown by Topic 1 Goal of Analysis of Variance A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares 2 A Worked Example 3 Example: Weight Loss Drink ANOVA Using SPSS 4 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error 5 of the ANOVA What You Learned Notations 2 / 103

3 What You Will Learn from this Lesson You will learn: How to compare more than two independent means to assess if there are any differences via an analysis of variance (ANOVA). How the total sums of squares for the data can be decomposed into a part that is due to the mean differences between groups and to a part that is due to within group differences. Why doing multiple t-tests is not the same thing as ANOVA. Why doing multiple t-tests leads to a multiplicity issue, in that as the number of tests increases, so to does the probability of one or more error. How to correct for the multiplicity issue in order for a set of contrasts/comparisons has a Type I error rate for the collection of tests at the desirable (e.g.,.05) level. How to use SPSS and R to implement an ANOVA and follow-up tests. 3 / 103

4 Motivation Goal of Analysis of Variance When looking at different allergy medicines, there are numerous options. So how can it be determined which brand will work best when they all claim to do so? Data could be collected to determine the outcomes from each product among numerous individuals randomly assigned to different brands. An ANOVA could be run to infer if there is a performance difference between these different brands. If there are no significant results, evidence would not exist to suggest there are differences in performance among the brands. If there are significant results, we would infer that the brands do not perform the same, but further tests would have to be conducted so as to infer where the differences are. 4 / 103

5 Goal of Analysis of Variance A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares The goal of ANOVA is to detect if mean differences exist among m groups. Recall the independent groups t-test is designed to detect differences between two independent groups. The t-test is a special case of ANOVA when m = 2 (tdf 2 equals the F (1,df ) from ANOVA for two groups). 5 / 103

6 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Obtaining a statistically significant result for ANOVA conveys that not all groups have the same population mean. However, a statistically significant ANOVA with more than two groups does not convey where those differences exist. Follow-up tests (contrasts/comparisons) can be conducted to help discern specifically where group means differ. 6 / 103

7 Consumer Preference A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Consider the overall perception of how consumers regard different companies. An experiment was done in which 30 individuals were randomly assigned into one of three groups. All participants saw (almost) the same commercial advertising a new Android smart phone. The difference between the groups was that the commercial attributed the phone to either (a) Nokia, (b) Samsung, or (c) Motorola. Of interest is in whether consumers tend to rate the brands differently, even for the same cell phone. 7 / 103

8 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares What are other examples in which ANOVA would be useful? 8 / 103

9 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Consider the null hypotheses of equal variances: H 0 : σ 2 1 = σ / 103

10 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Consider the null hypotheses of equal variances: H 0 : σ 2 1 = σ 2 2. The F -statistic is used to evaluate the above null hypothesis, and is defined as the ratio of two independent variances: F (df1,df 2 ) = s2 1 s2 2, where df 1 and df 2 are the degrees of freedom for s 2 1 and s2 2, respectively. 10 / 103

11 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Consider the null hypotheses of equal variances: H 0 : σ 2 1 = σ 2 2. The F -statistic is used to evaluate the above null hypothesis, and is defined as the ratio of two independent variances: F (df1,df 2 ) = s2 1 s2 2, where df 1 and df 2 are the degrees of freedom for s 2 1 and s2 2, respectively. Notice that F cannot be negative and is unbound on the high side. F -is a positively skewed distribution. 11 / 103

12 Examples Goal of Analysis of Variance A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares We have previously asked questions about the mean difference, but the F -distribution allows us to ask questions about variability. Is the variability of user satisfaction of Gmail users different than the variability of user satisfaction of Outlook.com? Does Mars and their M&M s production have better control (i.e., smaller variance) than Wrigley s Skittles? For a given item, are Wal-Mart prices across the country more stable than Kroger s (for like items)? Does a particular machine (or location/worker/shift) produce more variable products than a counterpart? 12 / 103

13 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares The standard deviation of Gmail user satisfaction was 6.35 based on a sample size of 55. The standard deviation of Outlook.com user satisfaction was 8.90 based on a sample size of 42. For an F -test of this sort addressing any differences in the variance (e.g., is there more variability in user satisfaction in one group), there are two critical values, one at the α/2 value and one at the 1 α/2 value. The critical values are and for the.025 and.975 quantiles (i.e., when α =.05). The F -statistic for the test of the null hypothesis is F = = =.509. The conclusion is:. 13 / 103

14 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Thus far, we have talked only about the idea of comparing two variances. But, what does this have to do with comparing means, which is the question we are interested in addressing? 14 / 103

15 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Analysis of variance (ANOVA) considers two variances: 15 / 103

16 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Analysis of variance (ANOVA) considers two variances: one variance calculates the variance of the group means; another variance is the (weighted) mean of within group variances (recall s 2 p from the two group t-test). 16 / 103

17 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Analysis of variance (ANOVA) considers two variances: one variance calculates the variance of the group means; another variance is the (weighted) mean of within group variances (recall s 2 p from the two group t-test). We thus consider the variability of the group means to assess if the population group means differ. 17 / 103

18 Conceptual Underpinnings of ANOVA A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares The null hypothesis in an ANOVA context is that all of the group means are the same: µ 1 = µ 2 =... = µ m = µ, where m is the total number of groups. When the null hypothesis is true, we can estimate the variance of the scores with two methods, both of which are independent of one another. 18 / 103

19 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares If the ratio of variances (i.e., F -test) is so much larger than 1 that it seems unreasonable to have happened by chance alone, then the null hypothesis can be rejected. Of course, so much larger than 1 that it seems unreasonable is defined in terms of the p-value (compared to α). If the p-value is smaller than α, the null hypothesis of equal population means is rejected. The variance of the scores can be calculated from within each group and then pooled across the groups (in exactly the same manner as was done for the independent groups t-test). 19 / 103

20 Mean Square Within Goal of Analysis of Variance A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Recall that the best way to arrive at a pooled within group variance is to calculate a weighted mean of the variances: s 2 Pooled = m (n j 1)sj 2 j=1 m n j m j=1 = m SS j j=1 N m = s2 Within = MS Within, where SS is sum of squares, MS is mean square (i.e., a variance), m is the number of groups, n j is the sample size in the jth group (j = 1,..., m), and N is the total sample size (N = m n j ). j=1 20 / 103

21 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares In the special case where n 1 = n 2 =... = n m = n, the equation for the pooled variance reduces: m sj 2 j=1 m = s2 Within = MS Within. Notice that the degrees of freedom here are N m. The degrees of freedom are N m because there are N independent observations yet m sample means estimated. 21 / 103

22 Mean Square Between A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares Recall from the single group situation that the variance of the mean is equal ( to the variance of the scores divided by the sample size i.e.,s 2 = s2 Y j Ȳ j n j ). That is, the variance of the sample means is the variance of the scores divided by the sample size. 22 / 103

23 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares However, under the null hypothesis, we can calculate the variance of the sample means directly by using the m means as if they were individual scores. Then, an estimate of the variance of the scores could be obtained by multiplying the variance of the means by sample size (s 2 Between = ns2 Ȳ ). If the F -statistic is statistically significant, the conclusions is that the variance of the means is larger than it should have been, if in fact the null hypothesis was true. Notice that the degrees of freedom here are m / 103

24 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares There are thus two variances that estimate the same value under the null hypothesis. One (σwithin 2 ) calculated by pooling within group variances. The other (σbetween 2 ) by calculating the variance of the means and multiplying by the within group sample size. If the null hypothesis is exactly true, σ2 Between σ 2 Within = 1. If the null hypothesis is false and mean differences do exist, sbetween 2 will be larger than would be expected under the null hypothesis, then s2 Between s 2 Within > / 103

25 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares If F = s2 Between s 2 Within (i.e., F = MS Between MS Within ) is statistically significant, we will reject the null hypothesis and conclude that H 0 : µ 1 = µ 2 =... = µ m = µ is false. 25 / 103

26 A Conceptual Example Appropriate for ANOVA Example F -Test for Independent Variances Conceptual Underpinnings of ANOVA Mean Squares If F = s2 Between s 2 Within (i.e., F = MS Between MS Within ) is statistically significant, we will reject the null hypothesis and conclude that H 0 : µ 1 = µ 2 =... = µ m = µ is false. Thus, we are comparing means based on variances! 26 / 103

27 The ANOVA Model Goal of Analysis of Variance A Worked Example The ANOVA assumes that the score for the ith individual in the jth group is a function of some overall mean, µ, some effect for being in the jth group exists, τ j, and some uniqueness exists, ε ij. Such a scenario implies that where Y ij = µ + τ j + ε ij, τ j = µ j µ, with τ j being the treatment effect of the jth group. 27 / 103

28 A Worked Example When the null hypothesis is true, the sum of the τs squared m equals zero: τj 2 = 0. j=1 When the null hypothesis is false, the sum of the τs squared m equals some number larger than zero: τj 2 > 0. j=1 28 / 103

29 A Worked Example Thus, we can formally write the null and alternative hypotheses for ANOVA as H 0 : and H a : m j=1 m j=1 τ 2 j = 0 τ 2 j > 0, respectively. Note that H 0 : m j=1 τ 2 j H 0 : µ 1 = µ 2 =... = µ m = µ. = 0 is equivalent to 29 / 103

30 A Worked Example The null hypothesis can be evaluated by determining, probabilistically, if the sum of the estimated τ s squared is greater than zero by more than what would be expected by chance alone. The hard to believe part is evaluated by the specified α value. 30 / 103

31 A Worked Example The sums of squares are defined as follows: and m SS Between = SS Treatment = SS Among = n j (Ȳj Ȳ..) 2 ; j=1 n m j SS Within = SS Error = SS Residual = (Y ij Ȳj) 2 ; j=1 i=1 SS Total = SS Between + SS Within n m j SS Total = (Y ij Ȳ..) 2. j=1 i=1 31 / 103

32 A Worked Example Like usual, we divide the sums of squares by the appropriate degrees of freedom in order to obtain a variance. In the ANOVA context, the sums of squares divided by its degrees of freedom is called a mean square: SS df = MS. Mean squares are so named because when the sums of squares is divided by its degrees of freedom, the resultant value is the mean of the squared deviations (i.e., the mean square). Mean square simply means variance. 32 / 103

33 A Worked Example In general, the source table is defined as: Source SS df MS F p-value Between m n j (Ȳj Ȳ..)2 m 1 p Within Total j=1 n j SS Between m 1 m (Y ij Ȳ.j) 2 N m SS Within j=1 i=1 n j m (Y ij Ȳ..) 2 N 1 j=1 i=1 N m MS Between MS Within 33 / 103

34 A Worked Example It can also be shown that the expected values of the mean squares are given as E [MS Between ] = σ 2 Within + m n j τj 2 m 1, j=1 E [MS Within ] = σ 2 Within, When all of the population means are equal, the second component of the MS Between and the expectation of the two mean squares is the same. When any population mean difference exists, E [MS Between ] > E [MS Within ]. 34 / 103

35 Worked Example Raw Data Nokia Samsung Motorola Ratings e i1 = y i1 y 1 e 2 i1 = ( y i1 y 1 ) 2 Ratings e i2 = y i2 y 2 e i2 ( y y i2 2 ) 2 Ratings e i3 = y i3 y 3 e 2 i 3 = y i 3 y Σ Mean SD Variance Grand=Mean=(======;=y1bar=dot=dot==)= y.. ===(4.50*10=+=8.00*10=+=7.00*10)/30===6.50 The=grand=mean=is=the=(weighted)=mean=of=the=sample=means=(here=it=is=simply=equal=to=the=mean=of=the=means=due=to=equal=group=sample=sizes. Sums%of%Squares Between=Sum=of=Squares ===10*( ) 2 =+=10*( ) 2 =+=10*( ) 2 ===65.00===SS Between This=is=the=weighted=(because=each=score=in=a=group=has=the=same=sample=mean,=of=course)=sum=of=squared=deviation=between=the=group=means=and=the=grand=mean. Within=Sum=of=Squares ===9*3.17=+=9*9.11=+=9*11.56===28.5=+=82=+=104===214.50===SS Within This=is=the=sum=of=each=of=the=within=group=sum=of=squares. Mean%Squares Now,=to=obtain=the=mean=squares,=divide=the=sums=of=squares=by=their=appropriate=degrees=of=freedom: Mean=Square=Between ===65.00/(311)===32.50===MS Between ( ) 2 Mean=Square=Within= ===214.50/=27===7.94===MS Within Inference Now,=to=obtain=the=F1statistic,=divide=the=Mean=Square=Between=by=the=Mean=Square=Within: F"=" 32.50/7.94===4.091 To=obtain=the=p1value,=use=the="F.Dist.RT"=formula=for=finding=the=area=in=the=right=tail=that=exceeds=the=F=value=of=4.091 p"=" Now,=because=the=p1value=is=less=than=α=(.05=being=typical),=we=reject=the=null=hypothesis.=We=infer=that=the=population=group=mean=are=not=all=equal.= Thus,=the=same=phone=commercial,=as=attributed=to=different=brands,=had=an=effect=on=the=ratings=of=the=phone.= The=conclusion=is=that=there=is=an=effect=of=brand=on=consumer=sentiment=1=consumers=rate=the=same=thing=differently=depending=on=the=brand=attribution.= The data are available here: nd.edu/~kkelley/teaching/data/phone_commercial_preference.sav.

36 Worked Example Summary Statistics Summary8Statistics8from8the8Phone8Evaluation Nokia Samsung Motorola Mean y j Standard8deviation s j Sample8size y Grand8mean y.. 8=8(4.50* * *10)/(30)8=86.50 Rather8than8using8the8full8data8set,8only8the8summary8statistics8are8actually8needed.8The8reason8is8because8we8can8determine8the8 sums8of8squares8from8the8summary8data.8the8within8sum8of8squares8is8literally8the8sum8of8the8degrees8of8freedom8multiplied8by8the8variance8from8each8group. Sums%of%Squares Between8Sum8of8Squares 8=810*(4.50N6.50) *(8.00N6.50) *(7.00N6.50) 2 8= =8SS Between This8is8the8weighted8(because8each8score8in8a8group8has8the8same8sample8mean,8of8cousre)8sum8of8squared8deviation8between8the8group8means8and8the8grand8mean. Within8Sum8of8Squares 8= *(10N1) *(10N1) *(10N1)8= ,8which8in8terms8of8variances8(instead8of8standard8deviations)8can8be8written8as: 8=83.17*(10N1)8+9.11*(10N1) *(10N1)8= =8SS Within Recall8that8the8sums8of8squares8divided8by8its8degree8of8freedom8is8a8variance8Correspdongly,8a8variance8multiplied8by8its8degrees8 of8freedom8is8a8sum8of8squares.8thus,8we8are8able8to8find8the8sum8of8squares8by8multiplying8the8variances8by8their8degrees8of8freedom.8 Mean%Squares Now,8to8obtain8the8mean8squares,8divide8the8sums8of8squares8by8their8appropriate8degrees8of8freedom: Mean8Square8Between 8=865.00/(3N1)8= =8MS Between Mean8Square8Within8 8= /8278=87.948=8MS Within Inference Now,8to8obtain8the8FNstatistic,8divide8the8Mean8Square8Between8by8the8Mean8Square8Within: F"=" 32.50/7.948= To8obtain8the8pNvalue,8use8the8"F.Dist.RT"8formula8for8finding8the8area8in8the8right8tail8that8exceeds8the8F8value8of p"=" Now,8because8the8pNvalue8is8less8than8α8(.058being8typical),8we8reject8the8null8hypothesis.8We8infer8that8the8population8group8mean8are8not8all8equal.8 Thus,8the8same8phone8commercial,8as8attributed8to8different8brands,8had8an8effect8on8the8ratings8of8the8phone.8 The8conclusion8is8that8there8is8an8effect8of8brand8on8consumer8sentiment8N8consumers8rate8the8same8thing8difference,8depending8on8the8brand8attribution.8 The data are available here: nd.edu/~kkelley/teaching/data/phone_commercial_preference.sav.

37 Example: Weight Loss Drink ANOVA Using SPSS Product Effectiveness: Weight Loss Drinks Over a two month period in the early spring, 99 participants from the midwest were randomly assigned to one of three groups (33 each) to assess the effectiveness of meal replacement weight loss drink. Study was conducted and analyzed by an independent firm. The three groups were a (a) control group, (b) SF, and (c) TL. All participants were encouraged to exercise and given running shoes, workout outfit, and a pedometer. 37 / 103

38 Example: Weight Loss Drink ANOVA Using SPSS The summary statistics for weight change in pounds (before breakfast) are given as: Control SF TL Total Ȳ s n As can be seen, 22 participants did not compete the study. Implications? 38 / 103

39 Example: Weight Loss Drink ANOVA Using SPSS The following table is the ANOVA source table: Source SS df MS F p Between <.001 Within Total / 103

40 Example: Weight Loss Drink ANOVA Using SPSS The critical F -value at the.05 level for 2 and 69 degrees of freedom is F (2,74;.95) = So, given the information, the decision is to. The one-sentence interpretation of the results is: 40 / 103

41 Performing an ANOVA in SPSS Example: Weight Loss Drink ANOVA Using SPSS 41 / 103

42 Example: Weight Loss Drink ANOVA Using SPSS 42 / 103

43 ANOVA Output from SPSS

44 Example: Weight Loss Drink ANOVA Using SPSS Suggestions when Performing ANOVA in SPSS Start with Analyze Descriptives Explore. Analyze Compare Means One-Way ANOVA for ANOVA procedure. In the One-Way ANOVA specification, request a Means Plot (via Options). Consider using Analyze General Linear Model Univariate for a more general approach. 44 / 103

45 Omnibus Versus Targeted Tests Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Procedures such as the t-test are targeted, and thus test specific hypotheses. For example, the independent groups t-test evaluates the hypothesis that µ 1 = µ 2. Thus, after an ANOVA is performed, oftentimes we want to know where the mean differences exist. However, a rationale of ANOVA was not to perform many significance tests. 45 / 103

46 An Analogy Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Consider an airline scheduling system at the gate of departure. 46 / 103

47 An Analogy Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Consider an airline scheduling system at the gate of departure. This system requires all five processes to simultaneously function: 1 live feed from the corporate server; 2 live feed to the corporate server; 3 live feed to the departing airport; 4 live feed to the arrival airport; 5 computer terminal to function property (e.g., no software glitches, no power loss). 47 / 103

48 An Analogy Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Consider an airline scheduling system at the gate of departure. This system requires all five processes to simultaneously function: 1 live feed from the corporate server; 2 live feed to the corporate server; 3 live feed to the departing airport; 4 live feed to the arrival airport; 5 computer terminal to function property (e.g., no software glitches, no power loss). Suppose that the uptime or reliability of each of these independent systems is.95, meaning at any given time there is a 95% chance each process is working. 48 / 103

49 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? 49 / 103

50 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? Recalling the rule of independent events, the probability that the system can be used is =.95 5 = / 103

51 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? Recalling the rule of independent events, the probability that the system can be used is =.95 5 = Thus, even though each piece of the system has a 95% chance of working properly, there is only a 77.38% chance that the system itself can be used. 51 / 103

52 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? Recalling the rule of independent events, the probability that the system can be used is =.95 5 = Thus, even though each piece of the system has a 95% chance of working properly, there is only a 77.38% chance that the system itself can be used. The implication here is that an error occurring somewhere in the set of processes ( =0.2262) is much higher than for any given system (1-.95=.05). 52 / 103

53 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? Recalling the rule of independent events, the probability that the system can be used is =.95 5 = Thus, even though each piece of the system has a 95% chance of working properly, there is only a 77.38% chance that the system itself can be used. The implication here is that an error occurring somewhere in the set of processes ( =0.2262) is much higher than for any given system (1-.95=.05). Note that the rate of errors in the system is (.2262/.05) times higher than in a given process! 53 / 103

54 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error What is the probability that the system can be used when needed (i.e., that all five systems working properly)? Recalling the rule of independent events, the probability that the system can be used is =.95 5 = Thus, even though each piece of the system has a 95% chance of working properly, there is only a 77.38% chance that the system itself can be used. The implication here is that an error occurring somewhere in the set of processes ( =0.2262) is much higher than for any given system (1-.95=.05). Note that the rate of errors in the system is (.2262/.05) times higher than in a given process! This is the multiplicity issue an error somewhere among a set of tests is higher than for any given test. 54 / 103

55 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Why Multiplicity Matters Multiple Testing The probability of making a Type I error out of C (i.e., independent) comparisons is given as p(at least one Type I error) = 1 p(no Type I errors) = 1 (1 α) C, where C is the number of independent comparisons to be performed (based on rules of probability). If C = 5, then p(at least one Type I error) =.2262! Note that this is the same probability that 1 or more confidence intervals when 5 are computed, each at the 95% level, do not bracket the population quantity. The scenario here is analogous to the airline scheduling system. 55 / 103

56 Types of Error Rates Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error There are three types of error rates that can be considered: 1 Per comparison error rate (α PC ). Analogous to the per process failure rate (5%) in the the airline system example. 2 Familywise error rate (α FW ). Analogous to the system failure rate (22.62%) of the airline system example above. 3 Experimentwise error rate (α EW ). Analogous to the multiple systems being required to fly the airplane (e.g., not only the scheduling system, but also that the plan functions properly, the flight team arrives on time, etc.), which can be much higher than α FW (if there are multiple families). 56 / 103

57 Per Comparison Error Rate Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error α PC : the probability that a particular test (i.e., a comparison) will reject a true null hypothesis. This is the Type I error rate with which we have always used (as we only worked with a single test at a time). 57 / 103

58 Familywise Error Rate (α FW ) Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error α FW : the probability that one or more tests will reject a true null hypothesis somewhere in the family. Defining exactly what a family is can be difficult and open to interpretation. As an aside, there are many statistical issues open to interpretation. Reasonable people can disagree on how to handle various issues. Openness about the methods, it s assumptions, and limitations is key. 58 / 103

59 Experimentwise Error Rate (α EW ) Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error α EW : the probability that one or more tests will reject a true null hypothesis somewhere in the experiment (or study more generally). Modifying the significance criterion so that α FW is the probability of a Type I error out of the set C significance tests is the same as forming C simultaneous confidence intervals. We do not focus on the experiment wise error rate, as we will assume a single family for our set of tests. 59 / 103

60 THING EXPLAINER IS AVAILABLE AT: AMAZON, BARNES & NOBLE, INDIE BOUND, HUDSON ABOUT A Hypothesis, A Result SIGNIFICANT < < PREV RANDOM NEXT > > From XKCD:

61 Tests, tests, tests,... From XKCD:

62 ... and more tests... kcd.com/882/ From XKCD:

63 ... and more tests xkcd: Significant From XKCD:

64 ... and yet more tests... From XKCD:

65 THING EXPLAINER IS AVAILABLE AT: AMAZON, BARNES & NOBLE, INDIE BOUND, HUDSON ABOUT A Type I Error (It Seems) SIGNIFICANT < < PREV RANDOM NEXT > > From XKCD:

66 After Many Tests, A Finding From XKCD: < < PREV RANDOM NEXT > > PERMANENT LINK TO THIS COMIC: IMAGE URL (FOR HOTLINKING/EMBEDDING):

67 Error Rate < < PREV RANDOM NEXT > > PERMANENT LINK TO THIS COMIC: IMAGE URL (FOR HOTLINKING/EMBEDDING): The probability of a Type I error for 20 independent tests, which the jelly bean comparisons were, is 2/4 1 (1.05) 20 = = Thus, there is a 64% chance of a Type I error in such a case! From XKCD:

68 A Summary... Multiplicity Matters!

69 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Linear Comparisons: Specifying Contrasts of Interest Suppose a question of interest is the contrast of group 1 and group 3 in a three group design. That is, we are interested in the following effect: Ȳ 1 Ȳ3. The above is equivalent to: (1) Ȳ1 + (0) Ȳ2 + ( 1) Ȳ3. 69 / 103

70 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Suppose a question of interest is the mean of group 1 and group 2 (i.e., the mean of the two group means) and group 3 in a three group design. That is, we are interested in the following effect: Ȳ 1+Ȳ 2 2 Ȳ3. The above is equivalent to: Ȳ 1+Ȳ ( 1) Ȳ3. The above is equivalent to: ( 1 2 ) Ȳ1 + ( 1 2 ) Ȳ2 + ( 1) Ȳ3. We could also write the above as: (.5) Ȳ1 + (.5) Ȳ2 + ( 1) Ȳ3. 70 / 103

71 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Consider a situation in which group 1 receives one type of allergy medication, group 2 receives another type of allergy medication, and group 3 receives a placebo (i.e., no medication). The question here is does taking medication have an effect over not taking medication on self reported allergy symptoms. 71 / 103

72 Forming Linear comparisons Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error In the population, the value of any contrast of interest is given as m Ψ = c 1 µ 1 + c 2 µ 2 + c 3 µ c m µ m = c j µ j, where c j is the comparison weight for the jth group and Ψ is the population value of a particular linear combination of means. An estimated linear comparisons is of the form m ˆΨ = c 1 Ȳ 1 + c 2 Ȳ 2 + c 3 Ȳ c m Ȳ m = c j Ȳ j, where c j is the comparison weight for the jth group and ˆΨ is the particular linear combination of means. j=1 j=1 72 / 103

73 Forming Linear comparisons Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The first example from above was comparing the mean of group 1 versus group 2. In c-weight form the c-weights are [1, 0, 1]: (1) Ȳ1 + (0) Ȳ2 + ( 1) Ȳ3. Comparing one mean to another (i.e., using a 1 and -1 c-weight with the rest 0 s) is called a pairwise comparisons (as the comparison only involves a pair). 73 / 103

74 Forming Linear comparisons Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The second example was comparing the mean of groups 1 and 2 versus group 3. In c-weight form the c-weights are [.5,.5, 1]: (.5) Ȳ 1 + (.5) Ȳ 2 + ( 1) Ȳ 3. Comparing weightings of two or more groups to one or more other groups is called a complex comparison. That is, if the c-weights are something other than 1 and -1 and the rest 0 s, it is a complex comparison. 74 / 103

75 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error It is required that m c j = 0. j=1 For example, setting c 1 to 1 and c 2 to 1 yields the pair-wise comparison of Group 1 and Group 2: ˆΨ = (1 Ȳ 1 ) + ( 1 Ȳ 2 ) = Ȳ 1 Ȳ / 103

76 Rules for c-weights Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The sum of the c-weights for a comparison that are positive should sum to 1. The sum of the c-weights for a comparison that are negative should sum to -1. By implication of the two rules above, sum of all c-weights for a comparison should sum to 0 (i.e., c j = 0). Otherwise, the corresponding confidence interval is not as intuitive. However, any rescaling of such c-weights produces the same t-test. The confidence interval will have a different interpretation than usual, as the effect will be for a specific linear combination (e.g., ˆΨ = 2Ȳ1 Ȳ2 Ȳ3). 76 / 103

77 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Thus, for the mean of Groups 1 and 2 compared to the mean of Group 3, the contrast is ˆΨ = (.5 Ȳ 1 ) + (.5 Ȳ 2 ) + ( 1 Ȳ 3 ) = Ȳ1 + Ȳ 2 2 Ȳ / 103

78 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Consider a situation in which one wants to weight the groups based on the relative size of an outside factor, such as marketshare, profit-per-segment, number of users, et cetera. Suppose that interest is in comparing teens versus a weighted average of 20 year olds and 30 year olds in an online community, where among the 20 and 30 year olds the proportion of users is 70 percent and 30 percent, respectively. ˆΨ = 1 ȲTeens + (.70 Ȳ20s) + (.30 Ȳ30s). 78 / 103

79 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error There are technically an infinite number of comparisons that can be formed, but only a few will likely be of interest. The comparisons are formed so that targeted research questions about population mean differences can be addressed. But, recall that in general, the sum of the c-weights that are positive should sum to 1 and the sum of the c-weights that are negative should sum to -1 so as to have a more interpretable confidence interval. 79 / 103

80 A More Powerful t-test Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The t-test corresponding to a particular contrast it given as t = cj Ȳ j MS Within m j=1 ( c 2 j n j ) = ˆΨ SE( ˆΨ), where the MS Within is from the ANOVA and is the best estimate of the population variance. Importantly, this t-test has N m degrees of freedom (i.e., the MS Within degrees of freedom). Note that the denominator is simply the standard error of the contrast, which is used for the corresponding confidence interval. 80 / 103

81 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Recall that when the homogeneity of variance assumption holds, there are m different estimates of σ 2. For homogeneous variances, the best estimate of the population variance for any group is the mean square error (MS Within ), which uses information from all groups. Thus, the independent groups t-test can be given as t = Ȳ j Ȳ k MS Within ( 1 n j + 1 n k ), with degrees of freedom based on the mean square within (N m), which provides more power. 81 / 103

82 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The above two-group t-test is still addresses the question does the population mean of Group 1 differ from the population mean of Group 2? However, there is more information is used because the error term is based on N m degrees of freedom instead of n 1 + n / 103

83 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The MS Within Even for a Single Group Even if we are interested in testing or forming a confidence interval for a single group, the mean square within can (and usually should) be used again, due to having a better estimate of σ 2 : t = Ȳ j µ 0 MS Within ( 1nj ). The two-sided confidence interval is thus: ( ) Ȳ j ± MS Within t (1 α/2,n m). 1nj The degrees of freedom for the above test and confidence interval is, because MS Within is used as the estimate of σ 2, N m. 83 / 103

84 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Thus, using MS Within is one way to have more power to test the null hypothesis concerning a single group or two groups, even when more than two groups are available. Additionally, precision is increased because the confidence interval will be narrower (due to the smaller standard error and smaller critical value). 84 / 103

85 The Bonferroni Procedure Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The Bonferroni Procedure is also called Dunn s procedure. Good for a few pre-planned targeted tests, but doing too many leads to conservative critical values. Conservative critical values are those that are bigger (i.e., harder to achieve significance) than would be the case ideally. Liberal critical values are those that are smaller (i.e., easier to achieve significance) than would be the case ideally. 85 / 103

86 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error It can be shown that α PC α FW Cα PC, where C is the number of comparisons. The per comparison error rate can be manipulated by dividing the desired familywise (or experimentwise) Type I error rate by C, the number of comparisons: α PC = α FW C. The standard t-test formula is used, but the obtained t value is compared to a critical value based on α/c: t (1 (α/c)/2,df ). The observed p-values (e.g., from SPSS) can be corrected for multiplicity by multiplying the C p-values by C. If the corrected p value is less than α FW, then the test is statistically significant in the context of a correct familywise Type I error rate. 86 / 103

87 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The critical value is what changes in the context of a Bonferroni test, not the way in which the t-test and/or confidence intervals are calculated. Incorporating MS Within into the denominator of the t-test is not really a change, as MS Within is just a pooled variance based on m (rather than 2) groups. Recall this is just an extension of s 2 Pooled when information on more than two groups is available. 87 / 103

88 Tukey s Test Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Tukey s test is used when all (or several) pairwise comparisons are to be tested. For comparing all possible pair-wise comparisons, Tukey s test provides the most powerful multiple comparison procedure. There is a Tukey-b in SPSS I recommend Tukey. The p-values and confidence intervals given by SPSS already yields, for the Tukey procedure, corrected p-values and confidence intervals. 88 / 103

89 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error Pairwise comparisons compare the means of two groups (i.e., a pair; µ 1 µ 3 ) without allowing any other complex comparisons (e.g., (Ȳ 1 + Ȳ 2 )/2 Ȳ 3 ). The observed test statistic is compared to the tabled values of the Studentized range distribution. This is the distribution that the Tukey procedure uses to obtain confidence intervals and p-values. 89 / 103

90 The Scheffé Test Goal of Analysis of Variance Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error For any number of post hoc tests with any linear combination of means, the Scheffé Test is generally optimal. Although the Scheffé Test is conservative for a small number of comparisons, any number of comparisons can be conducted while controlling the Type I error rate. 90 / 103

91 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error We compute the F -value (just a t-value squared) in accord with some linear combination of means, and a critical value is determined for the specific context. The Scheffé critical F -value (take the square root for the critical t-value) is given as (m 1)F (m 1,N m;α), which is m 1 times larger than the critical ANOVA value. 91 / 103

92 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error The Scheffé procedure should not be done for all pairwise comparisons (it is not as powerful as Tukey s Test for pairwise comparisons). If many complex and other (e.g., pairwise) are to be done, usually the Scheffé procedure is optimal. 92 / 103

93 Flowchart for Begin Testing all pairwise and no complex comparisons (either planed or post hoc) or choosing to test only some pairwise comparisons post hoc? Yes Use Tukey s method No Are all comparisons planned? No Use Scheffé s method Yes Is Bonferroni critical value less than Scheffé critical value? Yes Use Bonferroni s method No Use Scheffé s method (or, prior to collecting the data, reduce the number of contrasts to be tested)

94 Why Multiplicity Matters Error Rates Linear Combinations of Means Controlling the Type I Error SPSS does not make it easy to get the appropriate p-values and confidence intervals for complex comparisons. The Bonferroni and Scheffé procedures in SPSS are for pair-wise comparisons, which are not of interest because Tukey is almost always preferred for pair-wise. For the specified contrasts, SPSS reports only the standard output (i.e., not controlling the Type I error rate). Thus, users need to be really careful they are appropriately controlling the Type I error rate appropriately. 94 / 103

95 of the ANOVA of the ANOVA What You Learned Notations The assumptions of the ANOVA are the same as for the two-group t-test. 1 The population from which the scores were sampled is normally distributed. 2 The variances for each of the m groups is the same. 3 The observations are independent. Recall that multiple regression assumes homoscedasticity, which is just an extension of homogeneity of variance. 95 / 103

96 of the ANOVA What You Learned Notations Also like the independent groups t-test, the first two assumptions become less important as sample size increases. This is especially when the per group sample sizes are equal or nearly so. Thus, the larger the sample size, the more robust the model to these two assumption violations. 96 / 103

97 of the ANOVA What You Learned Notations Again, like the t-test, the ANOVA is very sensitive (i.e., it is not robust) to violations of the assumption of independence. Observations that are not independent can make the empirical α rate much different than the nominal α rate. 97 / 103

98 of the ANOVA What You Learned Notations Analysis of variance procedures test an omnibus (i.e., an overall) hypothesis. More specifically, ANOVA models test the hypothesis that µ 1 = µ 2 =... = µ m. In many situations, primary interest concerns targeted null hypotheses (not just the omnibus hypothesis). Thus, additional analyses may be necessary. 98 / 103

99 of the ANOVA What You Learned Notations A Summary from Designing Experiments and Analyzing Data This discussion focuses on special methods that are needed when the goal is to control α FW instead of to control α PC. Once a decision has been made to control α FW, further consideration is required to choose an appropriate method of achieving this control for the specific circumstance. One consideration is whether all comparisons of interest have been planned in advance of collecting the data. If so, the Bonferroni adjustment is usually most appropriate, unless the number of planned comparisons is quite large. Statisticians have devoted a great deal of attention to methods of controlling α FW for conducting all pairwise comparisons, because researchers often want to know which groups differ from other groups. We generally recommend Tukey?s method for conducting all pairwise comparisons. Neither Bonferroni nor Tukey is appropriate when interest includes complex comparisons chosen after having collected the data, in which case Scheffé s method is generally 99 / 103

100 What You Learned from this Lesson of the ANOVA What You Learned Notations You learned: How to compare more than two independent means to assess if there are any differences via analysis of variance (ANOVA). How the total sums of squares for the data can be decomposed to a part that is due to the mean differences between groups and to a part that is due to within group differences. Why doing multiple t-tests is not the same thing as ANOVA. Why doing multiple t-tests leads to a multiplicity issue, in that as the number of tests increases, so to does the probability of one or more error. How to correct for the multiplicity issue in order for a set of contrasts/comparisons has a Type I error rate for the collection of tests at the desirable (e.g.,.05) level. How to use SPSS to implement an ANOVA and follow-up tests. 100 / 103

101 Notations Goal of Analysis of Variance of the ANOVA What You Learned Notations H 0 : σ 2 1 = σ2 2 - The null hypothesis of equal variances F (df1,df 2 ) - The F -statistic with df 1 and df 2 as the degrees of freedom s 2 1 and s2 2 s 2 Pooled - The variances for group 1 and group 2, respectively - Pooled within group variance m - Number of groups n j - Sample size in the jth group (j = 1,..., m) N - Total sample size N = m n j j=1 101 / 103

102 Notations Continued Goal of Analysis of Variance of the ANOVA What You Learned Notations SS - Sum of squares This can be for the Between, Treatment, Among, Within, Error, or Total Sum of Squares MS - Mean square (i.e., a variance) MS Within is the mean square within a group Y ij - The score for the ith individual in the jth group τ j - The treatment effect of the jth group ε ij - Some uniqueness for the ith individual in the jth group E[MS Within ] - The expected value of the mean squares within a group 102 / 103

103 Notations Continued Goal of Analysis of Variance of the ANOVA What You Learned Notations C - The number of independent comparisons to be performed α PC - Per comparison error rate α FW - Familywise error rate α EW - Experimentwise error rate ˆΨ - The particular linear combination of means c j - Comparison weight for the jth group 103 / 103