Technical Report that Accompanies Relational and arelational confidence intervals: A comment on Fidler et al. (2004)

Transcription

1 Technical Report that Accompanies Relational and arelational confidence intervals: A comment on Fidler et al. (2004) Richard D. Morey and Jeffrey N. Rouder University of Missouri-Columbia Abstract Fidler, Thomason, Cumming, Finch, and Leeman (2004) criticized psychologists for not using confidence intervals (CIs) for inference. We suggest that this is wise, noting that, first, CIs most commonly used are not appropriate for group comparisons, and second, there is insufficient standardization and statistical development of CIs which can be used for inference (termed arelational CIs; Rouder & Morey, 2005). In this technical report we outline the methods we used to calculate confidence intervals in a particular research design and estimate the CIs coverage probability using a Monte Carlo simulation. Confidence intervals (CIs) are used in a variety of ways when reporting the results of psychological experiments. Arelational are CIs that are not intended for group comparison, such as typical 95% CIs drawn around sample means. Relational CIs, on the other hand, are CIs that are designed to allow evaluation of group differences (Rouder & Morey, 2005). In our comment regarding Fidler, Thomason, Cumming, Finch, and Leeman (2004), we recommend the use of arelational CIs in conjunction with standard null hypothesis significance tests (Rouder & Morey, 2005). We recommend this for several reasons. First, the construction of relational CIs is not as standardized as null hypothesis significance tests. Researchers reporting relational confidence intervals must document the construction of the CIs as well as interpret them, because the interpretation of relational CIs may not be as well understood by the reader as would be null hypothesis significance tests. Second, there has been insufficient statistical development of relational CIs. While several authors document the construction of relational CIs (Cumming & Finch, 2001; Fidler & Thompson, 2001; Masson & Loftus, 2003), only a handful report coverage probabilities for relational CIs (cf. Algina & Keselman, 2003). In addition, relational CIs documented only for a few, limited research designs and are not built into common statistical packages. We do not dispute that relational CIs can be a useful way of presenting data, and we acknowledge that they have advantages over standard null hypothesis significance tests This research is supported by NSF grant SES to J. Rouder, D. Sun, and P. Speckman. Address correspondence to jeff@banta.psyc.missouri.edu or morey@banta.psyc.missouri.edu.

2 CONFIDENCE INTERVALS 2 Table 1: Puzzle Completion Times as a function of puzzle shape and color scheme. Monochromatic Colored Participant Round Square Round Square Participant s Mean Condition Means ( X i ) (Reichardt & Gollob, 1997). In order to demonstrate one method of constructing relational CIs, we adapted a well-known example from Hays (1994). The data are presented in Table 1 along with condition means and subject means. The dependent variable was the time it takes to solve a jigsaw puzzle. The independent variables were the overall shape of the puzzle (round vs. square) and the color scheme (either monochromatic or colored). Each participant completed four puzzles with one in each condition. Repeated-measure ANOVA analysis reveals a significant main effect of color scheme (F (1,11) = 13.9, MSE=0.86) and shape (F (1,11) = 7.5, MSE=1.59), but not a significant interaction (F (1,11) 0, MSE=2.77). To construct CIs on Cohen s d for each main effect and the interaction, we followed Cumming and Finch s (2001) method using the noncentral t distribution. Below we document the construction of these confidence intervals and use a Monte Carlo simulation to test the coverage probability. Construction of CIs We are interested in constructing CIs for the main effect of shape, the main effect of color, and the interaction of shape and color. Our choice of effect size measure is Cohen s d, due to its straightforward relationship to the noncentrality parameter of the noncentral t distribution. In a situation without random effects, the derivation of the confidence interval is straightforward. Following Hays (1994), we first estimate the population comparison ψ corresponding to the effect: J ˆψ = c j Xj j=1

3 CONFIDENCE INTERVALS 3 Table 2: Comparison weights for the calculation of the three effects in data adapted from Hays (1994). Monochromatic Colored Effect Round Square Round Square Color Shape Interaction where ˆψ is the estimate of the appropriate population comparison, c j is approprite comparison weight for the condition j, J is the number of conditions, and X j is the sample mean for condition j. In our example, we have three vectors of comparison weights, shown in Table 2. It can be shown that n ˆψ t = Jj=1 s c 2 j where s is a pooled standard deviation estimate and n is the number of participants in a condition, is distributed as a noncentral t with n 1 degrees of freedom and noncentrality parameter δ, given by nψ δ = Jj=1 σ c 2 j where σ is the within-group standard deviation. From the definition of δ we can see that Cohen s d is related in a straightforward way; in fact, Cohen s d is d = δ n In our example data adapted from Hays (1994), however, there is an random subject effect to be taken into account. In the random effects case we can run a random effects ANOVA and use the appropriate MSE in place of s. Doing this ensures we use the proper standard errors for each ψ; we account for unwanted variability introduced by the random effect. Using this method, we find point estimates of d around which we can build confidence intervals. Point estimates for the adapted Hays data are given in Table 3. S-plus code for finding these values is given in the appendix. In order to find the proper bounds for the confidence intervals, we followed Cumming and Finch s (2001) recommendation of interating bounds with the noncentral t distribution. In order to use this method, it is necessary to find the δ parameters of two noncentral t distributions. These both depend on our desired type I error rate α and our calculated t value for each effect. The first is the noncentral t distribution with its 1 α 2 quantile at the t value found for the effect; this is used to find the lower bound. The second is the noncentral t distribution with its α 2 quantile at the t value found for the effect. this is used to find upper bound. After these two values are found, the two resulting δ parameters can be converted to estimates of d as shown above. These give the lower and upper 1 α

4 CONFIDENCE INTERVALS 4 Table 3: Calculated 95% CIs on Cohen s d for the adapted Hays data. Color Shape Interaction Upper limit Estimate Lower limit Table 4: Coverage probabilities based on 100,000 iterations of the CI simulation. A effect B effect Interaction True d Coverage Lower tail Upper tail confidence interval for Cohen s d, as shown in Table 3. The appendix gives S-plus code for finding these values. Simulation of Coverage Probability of CIs Methods of finding relational CIs is an area in which there has been insufficient development and standardization (Rouder & Morey, 2005). We suggest that authors, if they use relational CIs, document their construction and justification of their beliefs in their coverage probability. In order to show that coverage probability may not be what one may think based on the construction of the CIs, we have run a simulation. In this simulation, we generated random data with known population effect sizes and found t for each effect size from the data. We then calculated CIs for the value of t 1. If the CIs have the properties we expect, the true population value of t, δ, will be inside the confidence interval 100(1 α)% of the time. Also, the value will tend to be below the lower bound and above the upper bound 100 α 2 % of the time. We ran the simulation based on the same design as the modified Hays data. There were two fixed effects and one random effect. We considered the random effect a nuisance effect and did not calculate CIs on its effect. The result of 100,000 iterations of the simulation can be found in Table 4. Code for the simulation is given in the appendix. A slight bias toward overcoverage can be seen in the results of the simulation. In addition, where we would expect the errors to be evenly distributed between the upper and lower tail based on the construction of the CIs, this is not what we found; errors are found in the upper and lower tails an unequal proportion of times. Similar results were found by Algina and Keselman (2003) with their calculation of effect size. As they point out, it is better to have overcoverage than undercoverage, but researchers should be aware of the reduced type I error rate associated with overcoverage. 1 It is unecessary to convert to Cohen s d, because the conversion from t to d is a monotonic tranform. Consequently, for any value inside the CI for t, the corresponding value of d will be inside the CI for d.

5 CONFIDENCE INTERVALS 5 References Algina, J., & Keselman, H. J. (2003). Approximate confidence intervals for effect sizes. Educational and Psychological Measurement, 63, Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidence intervals, but can t make them think: statistical reform lessons from medicine. Psychological Science, 12, Fidler, F., & Thompson, B. (2003). Computing correct confidence intervals for ANOVA fixed- and random-effects effect sizes. Educational and Psychological Measurement, 61, Hays, W. L. (1994). Statistics, 5 th edition. Ft. Worth, T.X.: Harcourt Brace. Masson, E. J., & Loftus, G. R. (2003). Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology, 57, Reichardt, C. S., & Gollob, H. F. (1997). When confidence intervals should be used instead of statistical tests, and vice versa. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (p ). Mahwah, New Jersey, USA: Lawrence Erlbaum Associates. Rouder, J., & Morey, R. D. (2005). Relational and arelational confidence intervals: A comment on fidler et al. (2004). Psychological Science, 16, Thompson, B. (2002). What future quantative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, Appendix This appendix has two parts; the first gives S-plus code for finding the 95% confidence intervals for the three effects in the adapted Hays data. The second part gives the S-plus code for the simulation we ran to estimate coverage probability of confidence intervals constructed in this manner. S-plus Code for CIs on the Adapted Hays Data For finding the CIs in the manner described above it is necessary to define several functions. findncpt() is essentially a quantile function for the noncentral t distribution. Given a t value, a quantile, and degrees of freedom, the function will return the noncentrality parameter of the distribution with its qvalue quantile at t. After defining the functions, we define some necessary variables. #define necessary functions findncpt<-function(t,qvalue=.975,df=1) optimize(myncpt,interval=c(-20,20),df=df,myt=t,qvalue=qvalue)$minimum #$ myncpt<-function(ncp,myt,df=1,qvalue=.975) (qvalue-pt(myt,df,ncp))^2

6 CONFIDENCE INTERVALS 6 #Read in our data data=matrix(data=scan(),nrow=12,ncol=4,byrow=t) n=12 #number in each group alpha=.05 #alpha level upperq=(1-alpha/2) lowerq=alpha/2 ###These MS error terms are from the ANOVA table. mserr.color= mserr.shape= mserr.interaction= ###Define our contrast coefficients color.contrast=c(.5,.5,-.5,-.5) shape.contrast=c(.5,-.5,.5,-.5) interaction.contrast=c(.5,-.5,-.5,.5) With these variables declared, we can now find ˆψ and t for each effect. Then, using our t values, we build the CIs as outlined in Cumming and Finch (2001) and convert then to Cohen s d. ###Calculate psi for each effect psi.color=sum(colmeans(data)*color.contrast) psi.shape=sum(colmeans(data)*shape.contrast) psi.interaction=sum(colmeans(data)*interaction.contrast) ###find t for each effect t.color=sqrt(n)*psi.color/sqrt(mserr.color*sum(color.contrast^2)) t.shape=sqrt(n)*psi.shape/sqrt(mserr.shape*sum(shape.contrast^2)) t.interaction=sqrt(n)*psi.interaction/ sqrt(mserr.interaction*sum(interaction.contrast^2)) ###Calculate d for each effect

7 CONFIDENCE INTERVALS 7 d.color=t.color/sqrt(n) d.shape=t.shape/sqrt(n) d.interaction=t.interaction/sqrt(n) ###Calculate the confidence intervals upper.d.color=findncpt(t.color,lowerq,n-1)/sqrt(n) lower.d.color=findncpt(t.color,upperq,n-1)/sqrt(n) upper.d.shape=findncpt(t.shape,lowerq,n-1)/sqrt(n) lower.d.shape=findncpt(t.shape,upperq,n-1)/sqrt(n) upper.d.interaction=findncpt(t.interaction,lowerq,n-1)/sqrt(n) lower.d.interaction=findncpt(t.interaction,upperq,n-1)/sqrt(n) The values obtained by this method are shown in Table 3. S-plus Code for the Simulation of Coverage Probabilities The code for the simulation described above is given here. First, variables including the true effect size and variances are defined; then, data is generated. From this data, CIs are built, and the relation between the true value and the CIs is recorded. Data is then generated again, and the precess repeats for the given number of iterations. Once this is done, the code outputs the coverage probabilities. #Declare the between function between = function(x,lower,upper) ifelse(x<lower,-1,ifelse(x>upper,1,0)) #Be sure to declare the two functions findncpt() and myncpt() above, as well grand.mean = 0 #Effects. These should each sum to 0. a.effect = c(-.3,.3) b.effect = c(.5,-.5) int.effect = matrix(nrow = 2, ncol = 2, data = c(0,0,0,0)) fixed.effects = grand.mean + outer(a.effect, b.effect, "+") + int.effect #number of subjects in a condition n = 12 #Variance for the residuals and the random subject effect variance.residual =.5 variance.subject = 1 #contrast coefficients for each of the comparisons contrast.a = c(-1,-1,1,1)

8 CONFIDENCE INTERVALS 8 contrast.b = c(-1,1,-1,1) contrast.int = c(-1,1,1,-1) #Calculate the true values of the quantities we will estimate later truepsi.a = sum(as.vector(t(fixed.effects))*contrast.a) truepsi.b = sum(as.vector(t(fixed.effects))*contrast.b) truepsi.int = sum(as.vector(t(fixed.effects))*contrast.int) truencp.a = sqrt(n)*truepsi.a/sqrt((variance.residual)*sum(contrast.a^2)) truencp.b = sqrt(n)*truepsi.b/sqrt((variance.residual)*sum(contrast.b^2)) truencp.int = sqrt(n)*truepsi.int/sqrt((variance.residual)*sum(contrast.int^2)) #Number of iterations to run N = #Reserve space for recording the coverage coverage.a = (1:N) * NA coverage.b = (1:N) * NA coverage.int = (1:N) * NA #################Start stimulation for(i in 1:N){ #Here we generate our data, including our subject effects and residuals subject.effects = rnorm(n, 0,sqrt(variance.subject)) residuals = rnorm(n*length(fixed.effects), 0, sqrt(variance.residual)) data = outer(fixed.effects, subject.effects, + ) + residuals #Calculate values from ANOVA (From Hays, p.506) SS.a = sum(apply(data, 1, sum)^2/(n*2)) - (sum(data)^2/(n*4)) SS.b = sum(apply(data, 2, sum)^2/(n*2)) - (sum(data)^2/(n*4)) SS.subject = sum(apply(data, 3, sum)^2/4) - (sum(data)^2/(n*4)) SS.int = (sum(apply(data, c(1,2), sum)^2/n) - (sum(data)^2/(n*4)) - SS.b - SS.a) SS.abccell = sum(data^2) - (sum(data)^2/(n*4)) MSE.a = (sum(apply(data, c(1,3), sum)^2/2) - (sum(data)^2/(n*4))- SS.a - SS.subject)/(n-1) MSE.b = (sum(apply(data, c(2,3), sum)^2/2) - (sum(data)^2/(n*4)) - SS.b - SS.subject)/(n-1) MSE.int = (SS.abccell - SS.int - (MSE.a*(n-1)) - (MSE.b*(n-1)) - SS.a - SS.b - SS.subject)/(n-1) means = as.vector(apply(data, c(2,1),mean)) #estimate psi and the noncentrality parameter psi.hat.a = sum(means*contrast.a) psi.hat.b = sum(means*contrast.b)

9 CONFIDENCE INTERVALS 9 psi.hat.int = sum(means*contrast.int) t.a = sqrt(n)*psi.hat.a/sqrt(mse.a*sum(contrast.a^2)) t.b = sqrt(n)*psi.hat.b/sqrt(mse.b*sum(contrast.b^2)) t.int = sqrt(n)*psi.hat.int/sqrt(mse.int*sum(contrast.int^2)) #upncp is the upper bound, loncp is the lower bound of the confidence interval upncp.a = findncpt(t.a,.025, (n-1)) loncp.a = findncpt(t.a,.975, (n-1)) upncp.b = findncpt(t.b,.025, (n-1)) loncp.b = findncpt(t.b,.975, (n-1)) upncp.int = findncpt(t.int,.025, (n-1)) loncp.int = findncpt(t.int,.975, (n-1)) #the between function returns -1 if the true value is below #the confidence interval, 0 if within, #and 1 if above the interval coverage.a[i] = between(truencp.a, loncp.a, upncp.a) coverage.b[i] = between(truencp.b, loncp.b, upncp.b) coverage.int[i] = between(truencp.int, loncp.int, upncp.int) }###############End simulation #these are the coverage probabilities for the three effects. table(coverage.a)/n table(coverage.b)/n table(coverage.int)/n The values we obtained with 100,000 iterations of the simulation can be found in Table 4.