Increasing Power in Paired-Samples Designs. by Correcting the Student t Statistic for Correlation. Donald W. Zimmerman. Carleton University

Transcription

1 Power in Paired-Samples Designs Running head: POWER IN PAIRED-SAMPLES DESIGNS Increasing Power in Paired-Samples Designs by Correcting the Student t Statistic for Correlation Donald W. Zimmerman Carleton University Key words: Student t test, Wilcoxon signed-ranks test, rank transformation, skewed distribution, symmetric distribution, nonparametric test, correlation, paired samples, repeated measures, Type I error, Type II error, power Send correspondence to: Donald W. Zimmerman 978 4A Street Surrey, BC V4A 6B6 Canada Phone: (64) 5-9 Fax: (64 ) dwzimm@telus.net

2 Power in Paired-Samples Designs Abstract Increasing Power in Paired-Samples Designs by Correcting the Student t Statistic for Correlation Donald W. Zimmerman Carleton University In order to circumvent the influence of correlation in paired-samples and repeated measures experimental designs, researchers typically perform a one-sample Student t test on difference scores. This procedure entails some loss of power, because it employs n degrees of freedom instead of the n degrees of freedom of the independent-samples t test. In the case of non-normal distributions, researchers typically substitute the Wilcoxon signed-ranks test for the one-sample t test. The present study explored an alternate strategy, using a modified two-samples t test with a correction for correlation. For non-normal distributions, the same modified t test was performed on rank-transformed data. Simulations disclosed that this procedure, which retains n degrees of freedom, protects the Type I error rate for moderate and large sample sizes, maintains power for normal distributions, and substantially increases power for many non-normal distributions.

3 Power in Paired-Samples Designs Increasing Power in Paired-Samples Designs by Correcting the Student t Statistic for Correlation Donald W. Zimmerman Carleton University Statistical analysis of paired-samples and repeated-measures experimental designs typically makes use of the one-sample Student t test on difference scores. In the first part of the last century, before modern statistical methods began to be used widely by researchers in psychology, education, and social sciences, paired-samples data sometimes were handled in a different way. Many introductory textbooks in this early period, focusing mainly on largesample studies, presented methods for dealing with what were called correlated samples, using a modification of the familiar two-sample z test. These textbooks calculated the standard deviation of a difference between means using the formula σ = σ + σ ρ σ σ, () X X X X X X where ρ is the correlation between X and X, and based the standard error on this value when calculating the large-sample z statistic (see, example, Guilford & Fruchter, 97; p. 54; McNemar, 955, p. 85). Modern textbooks (e.g., Hays, 988) occasionally include these formulas, although they now recommend the paired-samples t test, not the z test for correlated samples, as a practical method. Furthermore, after nonparametric methods became widely used to circumvent non-normality, the Wilcoxon signed-ranks test typically was used in place of the t test on difference scores when the normality assumption was questionable. The present study re-examined some two-sample significance tests for paired samples data, using formulas containing correlation coefficients. Instead of the large-sample z test, however, it employed a version of the two-sample Student t tests modified to allow for correlation. Thus, tests were based on n degrees of freedom, instead of the n degrees

4 Power in Paired-Samples Designs 4 of freedom of the one-sample t test. And in the case of non-normal distributions, it employed the same modified two-sample t test on rank-transformed data, instead of the Wilcoxon signed-ranks test. It turned out that, for a wide variety of non-normal distributions, this strategy brought about improvements in control of Type I error rates, as well as an increase in power to detect differences. A Modified Two-Sample t Test with a Correction for Correlation It is possible to derive a t test for correlated samples analogous to the obsolete z test for correlated samples. The derivation incorporates the more general standard deviation of a difference between means given by equation (), instead of the conventional formula for the standard error of a difference (Zimmerman, Williams, & Zumbo, 99). The result is t = x X + X nn ( ) x ( ρ ), () where n is the number of pairs, x and x are sums of squares, and ρ is the population correlation. Otherwise expressed, if t is the conventional statistic based on two independent samples of n observations each, then t = t/c, where c = ρ. However, in many, if not most, research studies that analyze paired data, the value of a population correlation ρ is not known. In order to make use of equation () in practical significance testing, it is necessary to substitute an obtained sample correlation coefficient, r, for the population correlation. The present simulation study investigated how this modified test compares to the paired-samples t test with regard to Type I error rates and power, and how substitution of a sample estimate, r, for the population correlation, ρ, affects the accuracy of the result.

5 Power in Paired-Samples Designs 5 Method The present study compared Type I error rates and power of several significance tests performed on correlated samples from normal and non-normal distributions. Three of the nonnormal distributions were symmetric and 7 were skewed. The significance tests were () the usual independent-samples Student t test, () the usual paired-samples Student t test, () the Wilcoxon signed-ranks test, (4) the modified t test, using sample correlation coefficients, r, in place of the population correlation ρ, and (5) the modified t test, using a fixed value of ρ. The random number generator in this study was introduced by Marsaglia, Zaman, and Tsang (99) and has been described by Pashley (99, pp ). Normal variates, N(,), were generated by the rejection method of Marsaglia and Bray (964) and were transformed to have various distribution shapes using inverse distribution functions. The sample values were further transformed to have mean and standard deviation. In some cases, constants were added to all the scores in one sample in increments of.σ,.σ or.5σ to produce systematic differences in means and to determine the power of the tests. The algorithms for generating the various non-normal distributions are described in Table A in the appendix. For further details concerning simulation of non-normal variates, see, for example, Evans, Hastings, & Peacock (), Gentle (998), Morgan (99), and Patel, Kapadia, & Owen (976). There were 5, iterations of the sampling procedure for each condition in the study, except for the ones in Table, where there were, iterations. All significance tests were non-directional. Results of Simulations Table compares Type I error rates of 4 significance tests performed on samples from normal distributions. Sample sizes ranged from 5 to and population correlations were

6 Power in Paired-Samples Designs Insert Table about here ,.5, and.5. The first column in each section of the table, labeled t, shows results of the independent-samples t test and the second column, labeled t P, shows results of the paired-samples t test, that is, the one-sample t test on difference scores. As expected, the paired-samples test maintained the Type I error rate close to the significance level for all sample sizes and all degrees of correlation, while the Type I error rate of the independent-samples test declined below the nominal significance level as the correlation increased. The same was true for both the.5 and. significance levels. These results are consistent with many previous studies. The next column in each section, t C (r), gives results for the modified two-sample t test given by equation (), using the correlation coefficients between sample values (r) obtained on each iteration of the sampling procedure. For relatively small sample sizes 5,, and 5 the probabilities of Type I errors were noticeably inflated. However, the discrepancy became progressively less as the sample size increased. The next column, t C (ρ), gives results for the same modified two-sample test, using the fixed population correlation (ρ) in the formula. In this case, the control of the Type I error rate improved to some extent, although for the smaller sample sizes there still was some discrepancy. For the larger sample sizes, or more, there was little or no difference in the performance of t P, t C (r), and t C (ρ) for all degrees of correlation. A different picture emerges when the power of the tests to detect differences is considered. Table presents probabilities of rejecting H for normal and various non-normal Insert Table about here

7 Power in Paired-Samples Designs 7 distributions, as the difference between population means varied from to.5σ in increments of.5σ. Population correlations were.5 and.5, sample sizes were 5 and 5, and significance levels were.5 and.. The first column in each section, t P, shows results for the paired-samples t test. The next column, W, shows results for the large-sample normal-approximation version of the Wilcoxon signed-ranks test. The next two columns, t C (r), and t C (ρ), are results of the modified t test applied to rank-transformed data. That is, scores in the two samples were combined and transformed to ranks, after which the modified t test given by equation () was performed on the ranks replacing the original scores in their respective groups. As before, the column labeled t C (r) is the result of using sample correlation coefficients in the equation, and the column labeled t C (ρ) is the result of using the population ρ. Comparing the first two columns in each section, it is clear that the outcomes for the paired-samples t test and the Wilcoxon signed-ranks test were consistent with many previous studies using non-normal distributions. For all distributions, the Wilcoxon test protected the Type I error rate just as well as the t test, and for many of the non-normal distributions, especially skewed distributions, the Wilcoxon test was more powerful than the t test. However, the last two columns indicate that the modified t test on rank-transformed data in many instances was even more powerful than the Wilcoxon test. Furthermore, this difference is noticeable for the same distributions where the Wilcoxon test was a significant improvement over the t test. In the case of the 4 symmetric distributions in Table the normal, Laplace, rectangular, and bimodal distributions there are minimal differences among t P, t C (r), and t C (ρ). However, for the other 7 distributions, which are skewed, the power differences are more evident. The last column of Table A contains coefficients of skewness and kurtosis. Figures and give a more detailed picture of some power functions for 4 skewed

8 Power in Paired-Samples Designs Insert Figures and about here distributions (exponential, lognormal, mixed-normal, and truncated normal), where differences between means increased in increments of.σ. It is evident that the power function of each version of the modified t test dominated both the Wilcoxon signed-ranks test and the paired samples t test. Figures and 4 show how these power differences depend on sample sizes for Insert Figures, 4, 5, and 6 about here several distributions. In the simulations represented by these figures, there was a fixed difference between means of.σ. In all cases the power of the tests increased as sample size increased, as expected. In addition, the power differences of the various tests increased as sample size increased from about 75 to and then declined somewhat as sample sizes increased further to 5. Furthermore, the rank order of the four significance tests, from least powerful to most powerful, remained the same over the entire range of sample sizes, and the modified t test on rank-transformed data dominated in the case of the four skewed distributions (exponential, lognormal, mixed-normal and truncated normal in Figures,,, and 5), where there is considerable separation of the curves, and slightly dominated in the case of the Laplace distribution (Figure 6). For the normal distribution (Figure 4 ), the paired-samples t test was slightly superior, as expected. Further Comments Researchers typically employ the one-sample Student t test on difference scores when they believe data obtained from paired-samples or repeated-measures designs to be normally

9 Power in Paired-Samples Designs 9 distributed. This test protects the Type I error rate and has good power to detect differences. The present study revealed that, for large samples say 5 or larger from normal distributions, the modified t test using a correction for correlation performs very nearly the same as the pairedsamples t test. This is true when a known population correlation, ρ, is substituted in equation () and also when a sample correlation, r, obtained on each replication is used. On the other hand, the test with a sample r does not perform well for small sample sizes 5,, 5, or, where the Type I error rates are seriously inflated. This result can be attributed to the sampling variability of r for these small n s. As sample size increases, the variability of r declines, and the t statistic becomes more stable. The test based on a known population correlation, ρ, performs well even for the small sample sizes. In the case of non-normal distributions, the modified t test on rank-transformed data showed a substantial power gain relative to the paired-samples t test. The Wilcoxon signed-ranks test, the paired-samples t test, and the modified t test on rank-transformed data, all had about the same power for the symmetric non-normal distributions in the present study. This was true for both degrees of correlation and both significance levels. However, for the 7 skewed non-normal distributions, the modified t test on rank-transformed data was substantially more powerful than the Wilcoxon signed-ranks test. Comparisons of the paired-samples t test and the Wilcoxon test were consistent with many previous studies (see, for example, Andrews, et. al., 97; Hodges & Lehmann, 956; Randles & Wolfe, 979; Zimmerman & Zumbo, 99). After conversion of scores to ranks, the modified t test on ranks replacing scores displayed the same advantages possessed by the Wilcoxon signed-ranks test for symmetric distributions, and it performed better than the Wilcoxon test for skewed distributions. The superiority of the modified t test can be explained in part because it is evaluated at

10 Power in Paired-Samples Designs n degrees of freedom, instead of n degrees of freedom. Figures, 4, 5, and 6 indicate that the power difference of the modified test, for skewed distributions, is somewhat greater for the smaller sample sizes. For the small n s, a given difference in degrees of freedom corresponds to a rather large difference in critical values of the t statistic. However, for larger sample sizes, the difference in critical values is smaller, so the gain is somewhat less. Under some conditions, therefore, the modified t test on rank-transformed data can be useful in practical significance testing. In paired-samples designs, when the number of pairs is, say 5 or larger, and the population distribution is known or suspected to be non-normal, this test is a good choice. When samples are correlated, the modified t test on rank-transformed data and the Wilcoxon signed-ranks test both effectively protect the Type I error rate. If the distribution is symmetric, the two tests are about equally powerful, and if the distribution is skewed, the modified t test is substantially more powerful. For non-normal distributions, the modified t test retains the advantage of familiar nonparametric rank methods and, at the same time, the greater number of degrees of freedom of a two-sample test.

11 Power in Paired-Samples Designs References Andrews, D.F., Bickel, P.J., Hampel, F.R., Huber, P.J., Rogers, W.H., & Tukey, J.W. (97). Robust estimates of location: Survey and advances. Princeton, NJ: Princeton University Press. Evans, M., Hastings, N., & Peacock, B. (). Statistical distributions (rd ed.). New York: Wiley. Gentle, J.E. (998). Random number generation and Monte Carlo methods. New York: Springer. Guilford, J. P., & Fruchter, B. (97). Fundamental statistics in psychology and education (5th ed.). New York: McGraw-Hill. p. 54 Hays, W.L. (988). Statistics (4th ed.). New York: Holt, Rinehart, & Winston. Hodges, J. & Lehmann, E. (956). The efficiency of some nonparametric competitors of the t test. Annals of Mathematical Statistics, 7, 4-5. Marsaglia, G., Zaman, A., & Tsang, W.W. (99). Toward a universal random number generator. SIAM Review, 6, McNemar, Q. (955). Psychological statistics (nd ed.). New York: Holt, Rinehart, & Winston. p. 85 Mehrens, W.A., & Lehmann, I.J. (99). Measurement and evaluation in education and psychology (4th ed.). Chicago: Holt, Rinehart, & Winston. Morgan, B.J.T. (99). Elements of Simulation. London: Chapman & Hall. Pashley, P.J. (99). On generating random sequences. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues. Hillsdale, NJ: Lawrence Erlbaum Associates. (pp ) Patel, J.K., Kapadia, C.H., & Owen, D.B. (976). Handbook of statistical distributions. Randles, R.H., & Wolfe, D.A. (979). Introduction to the theory of nonparametric statistics. New York: Wiley.

12 Power in Paired-Samples Designs Zimmerman, D.W., & Zumbo, B.D. (99). In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues. Hillsdale, NJ: Lawrence Erlbaum Associates. (pp ) Zimmerman, D.W., Williams, R.H., & Zumbo, B.D. (99). Effect of nonindependence of sample observations on parametric and nonparametric statistical tests. Communications in Statistics: Simulation and Computation,,

13 Power in Paired-Samples Designs Appendix A Let U be a unit rectangular variate and Y a unit normal variate. The distributions in the study were generated using the transformations in Table A (inverse distribution functions). After these transformations, X was standardized by subtracting the mean and dividing by the standard deviation, resulting in a common mean of and standard deviation for all distributions. Table A. Transformations (inverse distribution functions) used to generate variates. Distribution Transformation Coefficients of skewness and kurtosis exponential λ = X = log(u) 9 lognormal shape parameter scale parameter X = exp(y) chi-square shape parameter 4 X = X 4 = Yi, Y is N(,).44 i= 6 Gumbel (extreme value) location parameter scale parameter power function shape parameter.5 scale parameter X = log( log U) X = U.64.4 truncated normal X = N(,), if X >.5 then X = mixed-normal with outliers (skewed) X = N(,) with probability.75 X=abs(X) with probability

14 Power in Paired-Samples Designs 4 Table A (continued). Distribution Laplace (double exponential) location parameter scale parameter Transformation X = log(u /U ), where U and U are unit rectangular Coefficients of Skewness and Kurtosis 6 rectangular X = U.8 bimodal (symmetric) X = N(,) with probability.5 X = N(,) with probability.5.9

15 Power in Paired-Samples Designs 5 Footnote. The computer program used in this study, written in PowerBasic, version.5, PowerBasic, Inc., Carmel, CA, can be obtained by writing to the author.

16 Power in Paired-Samples Designs 6 Table. Probability of rejecting H for various sample sizes (n) and population correlations (ρ) with significance levels of.5 and. (normal distribution). α =.5 ρ = ρ =.5 ρ =.5 n t t P t C (r) t C (ρ) t t P t C (r) t C (ρ) t t P t C (r) t C (ρ) α =

17 Power in Paired-Samples Designs 7 Table. Probability of rejecting H as a function of difference in population means (α =.5, ρ =.5). n = 5 n = 5 Distribution μ μ t P W t C (r) t C (ρ) t P W t C (r) t C (ρ) exponential lognormal chi-square Gumbel power function truncated normal mixed-normal normal Laplace rectangular bimodal ,

18 Power in Paired-Samples Designs 8 Table (continued) α =.5 ρ =.5 n = 5 n = 5 Distribution μ μ t P W t C (r) t C (ρ) t P W t C (r) t C (ρ) exponential lognormal chi-square Gumbel power function truncated normal mixed-normal normal Laplace rectangular bimodal

19 Power in Paired-Samples Designs 9 Table (continued) α =. ρ =.5 n = 5 n = 5 Distribution μ μ t P W t C (r) t C (ρ) t P W t C (r) t C (ρ) exponential lognormal chi-square Gumbel power function truncated normal mixed-normal normal Laplace rectangular bimodal

20 Power in Paired-Samples Designs Table (continued). α =. ρ =.5 n = 5 n = 5 Distribution μ μ t P W t C (r) t C (ρ) t P W t C (r) t C (ρ) exponential lognormal chi-square Gumbel power function truncated normal mixed-normal normal Laplace rectangular bimodal

21 Power in Paired-Samples Designs Figure Captions Figure. Power functions of four significance tests on paired data for exponential and mixednormal distributions. Figure. Power functions of four significance tests on paired data for lognormal and truncated normal distributions. Figure. Power of four significance tests on paired data, for a fixed difference between population means, as a function of sample size (exponential distribution). Figure 4. Power of four significance tests on paired data, for a fixed difference between population means, as a function of sample size (normal distribution). Figure 5. Power of four significance tests on paired data, for a fixed difference between population means, as a function of sample size (lognormal distribution). Figure 6. Power of four significance tests on paired data, for a fixed difference between population means, as a function of sample size (Laplace distribution).

22 Power in Paired-Samples Designs Probability of Rejecting H Exponential Distribution n = α =.5 Student t Wilcoxon modified t, sample r modified t, population ρ Difference in Standard Units. Mixed-Normal Distribution n = α =. Probability of Rejecting H Student t Wilcoxon modified t, sample r modified t, population ρ Difference in Standard Units

23 Power in Paired-Samples Designs..9 Lognormal Distribution ρ =.5 n = α =.5 Probability of Rejecting H Student t Wilcoxon modified t, sample r modified t, population ρ Difference in Standard Units. Truncated Normal Distribution ρ =.5 n = α =. Probability of Rejecting H Student t Wilcoxon modified t, sample r modified t, population ρ Difference in Standard Units

24 Power in Paired-Samples Designs 4..9 Exponential Distribution ρ =.5 α =.5 μ μ =. Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)..9 Exponential Distribution ρ =.5 α =. μ μ =. Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)

25 Power in Paired-Samples Designs 5. Normal Distribution ρ =.5 α =.5 μ μ =..9 Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)..9 Normal Distribution ρ =.5 α =. μ μ =. Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)

26 Power in Paired-Samples Designs 6. Lognormal Distribution ρ =.5 α =.5 μ μ =..9 Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)..9 Lognormal Distribution ρ =.5 α =. μ μ =. Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)

27 Power in Paired-Samples Designs 7. Laplace Distribution ρ =.5 α =.5 μ μ =..9 Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)..9 Laplace Distribution ρ =.5 α =. μ μ =. Probability of Rejecting H paired samples t Wilcoxon signed ranks modified, sample modified, population Sample Size (n)