Contrasts and Correlations in Theory Assessment

Transcription

1 Journal of Pediatric Psychology, Vol. 27, No. 1, 2002, pp Contrasts and Correlations in Theory Assessment Ralph L. Rosnow, 1 PhD, and Robert Rosenthal, 2 PhD 1 Temple University and 2 University of California, Riverside Objective: To describe a systematic quantitative approach to assessing the predictions made by competing theories using contrasts and correlational indices of effect sizes. Methods: We illustrate the use of the contrast F and t to compare and combine predictions when the raw data are continuous scores, and s when working with frequencies in 2 k tables of counts. Results: The traditional effect size correlation indicates the magnitude of the effect on individual scores of participants assignment to particular conditions. The contrast correlation obtained from the contrast F or t is, in some cases, the easiest way of estimating the effect size correlation in designs using more than two groups. The alerting correlation is another way of appraising the predictive power of a contrast and can be used to compute the contrast F from published results when all we have are condition means and the omnibus F from an overall analysis of variance. Omnibus Fs, those with more than 1 df in the numerator, are rarely useful in data analytic work since they address unfocused questions, yielding only vague answers. Conclusions: Asking focused questions using contrasts increases the clarity of our questions and the clarity and statistical power of our answers. Key words: correlation; contrast correlation; theory assessment; effect size. How can we tell good from not-so-good theories in behavioral science? The traditional answer is that good theoretical propositions are grounded in credible ideas and facts, are stated in a precise and focused way, and are empirically falsifiable. However, historians and philosophers of science recognize that the very assumption of critical falsifying tests is an arguable proposition. In behavioral science, for example, McGuire (1986) noted that researchers do not readily abandon their theories when confronted with unfriendly evidence; instead, they think about what might have gone wrong, or how to adjust their theory so it implies higher-order interactions. Thus, if a refutation occurs, the research- All correspondence should be sent to Ralph L. Rosnow, 177 Biddulph Road, Radnor, Pennsylvania rosnow@temple.edu. ers may argue that the test was not properly conducted, or that the particular prediction did not accurately represent their theory, or that the results were analyzed incorrectly, or that vital contextual boundaries need to be specified. Nonetheless, as McGuire wisely observed, theoretical propositions must always be subjected to empirical jeopardy, not merely to test their validity but to clarify and expand their meaning and reach. The purpose of this article is to illustrate the use of contrasts and three correlational indices of effect sizes in this evaluation and adjudication process. Contrasts are focused statistical procedures for asking precise questions of two or more groups. We present two illustrative cases, one in which the metrics of measurement are unit scores on an underlying continuum and the other, frequencies in a 2002 Society of Pediatric Psychology

2 60 Rosnow and Rosenthal 2 k table of counts. In each case, two competing propositions are to be addressed. We describe how to evaluate the alternative predictions by t, F, or z contrasts and a family of conceptually related indices, which we have called the effect size correlation (symbolized as r effect size ), the alerting correlation ( ), and the contrast correlation ( ). We also illustrate how to combine the alternative predictions in order to assess how the competing propositions might fare together, as indicated by r effect size,, and. First, r effect size reflects the magnitude of the effect on each score of the participants assignment to particular conditions, with membership in the conditions represented by lambda () weights based on predictions or theoretical hunches. The special characteristic of r effect size is that any disagreement between the predicted and obtained values of the means is considered to be noise or error and is added to the level of the noise or error found within conditions. Next, reflects the aggregate relationship between the group means and weights; it takes its name from the idea that it may alert the researcher who routinely calculates omnibus F tests (i.e., F with numerator df 1) not to be too hasty in embracing the null hypothesis just because the omnibus F failed to reach p.05. The special characteristic of is that it regards as noise or error only the disagreement between the predicted and obtained values of the means. That is, the level of noise or error found within conditions is simply set aside. Finally, reflects the partial correlation between the outcome scores and s after removal of all the noncontrast variation. In certain specifiable instances (discussed later), may be the only effect size correlation we can compute from other people s data in designs with more than two groups. The special characteristic of is that it regards as noise or error only the level of noise or error found within conditions. To recap these three correlational indices of effect size, we can describe all three in terms of what each regards as noise or error: (1) for, only within group noise contributes to error; (2) for, only between group noise contributes to error; and () for r effect size, both within and between group noise contribute to error. These three indices, and other aspects of the correlational approach, are discussed more fully in Rosenthal, Rosnow, and Rubin (2000); other related discussions can be found in Rosnow and Rosenthal (1996) and Rosnow, Rosenthal, and Rubin (2000). Table I. Hypothetical Results Showing the Effects of the Number of Counseling Sessions on Parental Functioning Number of sessions in four groups S 1 0 S 4 2 S 7 2 S 10 6 S 2 2 S 5 S 8 4 S 11 7 S 4 S 6 4 S 9 6 S 12 8 M S n The aggregated S These four S 2 values are quite homogeneous, with the ratio of the largest to the smallest S 2 of 4 being less than % of the F max of 142 required for heterogeneity at p.05. Case 1: Continuous Raw Scores Computing t and F Contrasts To illustrate the use of these procedures, suppose a pediatric researcher is interested in evaluating two theories, A and B, each of which implies a specific prediction about how many counseling sessions it will take to improve the psychological functioning of parents of children with serious illness. Theory A predicts a minimum of four sessions to produce any benefit and implies that fewer than four sessions will be fruitless. Theory B predicts small benefits as early as the first session, with gradual improvement continuing throughout all four sessions. To assess these competing predictions, the researcher designs a fully randomized experiment consisting of four groups, corresponding to 1, 2,, or 4 sessions of counseling, with participants in each group. Table I shows the participants scores (higher scores implying beneficial effects), the group means (M), the variance (S 2 ) in each group, and the number of participants (n) in the group. Table II shows the overall analysis of variance computed by the researcher and the reported omnibus F. Regrettably, many researchers feel the need to compute an omnibus F test before looking more closely at their data, as if science were a Simon says game in which it was necessary to seek permission from the p value associated with some vague statistical result before addressing the question of interest. The df-f in Table II is too imprecise to be informative, as the omnibus F would be the same whether we are interested in the prediction implied by Theory A or Theory B. Contrasts, on the other hand, allow us to address the competing predictions in a precise way. To do so, we begin by

3 Contrasts and Correlations 61 Table II. Overall Analysis of Variance on Data in Table I Sum of Sum of Source squares df squares F (,8) p Between Within expressing the predictions as integer lambda values that sum to zero (i.e., 0). Theory A anticipates no benefits prior to four sessions, but a substantial benefit after Session 4, which we express by A weights of 1, 1, 1,. Theory B anticipates a continuous linear increase of benefits, which we express by B weights of, 1, 1,. Incidentally, an easy way to create such weights is, first, to make a guess about the mean outcome score in each group and, second, to subtract the overall mean from each group mean to create s. Suppose, in the case of Theory A, we predicted group means of 0, 0, 0, 4 for sessions 1, 2,, 4, respectively. Subtracting the overall mean of 1 from those group means gives us A weights of 1, 1, 1,. For Theory B, suppose we predicted group means of 1, 2,, 4 for the four dosage levels. Now, subtracting the overall mean of 2.5 yields 1.5, 0.5, 0.5, 1.5, which we multiply by 2 to create whole number B weights of, 1, 1,. A convenien formula for testing each of the two competing predictions using the t statistic is as follows: (M) 2 ns 2 pooled, (1) where M group mean, S 2 pooled the mean square (MS) error (shown in Table II), n number of observations in group, and contrast weight assigned to group. Substituting in equation 1 to assess Theory A s prediction using A weights of 1, 1, 1,, and S 2 within 2.5 from Table 2, we find 2(1) (1) 4(1) 7() (1)2 (1) ,.162 (1)2 ()2 2.5 which, with 8 df (i.e., the degrees of freedom for S 2 within), has an associated one-tailed p Using equation 1 to assess Theory B s prediction using B weights of, 1, 1, (and the same means, sample sizes, and pooled error term), we find 2() (1) 4(1) 7() ()2 (1) , (1)2 ()2 2.5 which (again with 8 df) has an associated one-tailed p If we were interested in reporting F instead of t, we would simply square the contrast t (since t 2 F). It should be noted that equation 1 is simply a test of significance of ous; Equations 2,, and 4 will apply to the computation of effect size correlations. Equation 1 replaces the need to compute the omnibus F(,8 ) of Table II and, unlike that omnibus F, yields a focused answer to a precise question with a p value 10 times more significant. So far, we have been working with the original raw data. However, suppose we wanted to work with someone else s published data and all we had were the reported omnibus F and the group means. We could still calculate the t or F statistic fos to assess both theoretical predictions, for all we need is the squared alerting correlation (r 2 alerting). Multiplying r 2 alerting omnibus F df for omnibus F gives us the 1 df contrast F. To illustrate in the case of Theory A, correlating the group means and A weights yields.9258, and thus r 2 alerting The contrast F(1, 8) (.8571)(5.60)() 14.99, p.005 which can be alternatively expressed as (8).795, one-tailed p For Theory B, we find.9562; thus r 2 alerting.914. Using the same procedure shown above, we find F(1, 8) (.914)(5.60)() 15.60, p.0044, and (8).919 (one-tailed p.0022). Another way of thinking about the squared alerting r is that it immediately tells us the proportion of SS between accounted for by the particula weights. Here, given k 4 groups (and, therefore, df between groups), we see at once that both contrasts far exceeded the % of the SS between (i.e., % the reciprocal of the df) that we might have expected from any randomly drawn contrast among these four means. Contrast and Effect Size Correlations The contrast correlation ( ), or partial r between the contrast weights and participants scores after removal of all other between-group variation (i.e., removing the SS noncontrast ) can be obtained from the sums of squares by

4 62 Rosnow and Rosenthal ss contrast, (2) ss contrast SS within where SS contrast r 2 alerting SS between (and SS between 42 in Table II). Therefore, SS contrast for Theory A is , while SS contrast for Theory B is A more convenient formula for obtaining when is reported by others, or computed by us from our own data, is, () t 2 contrast df where df degrees of freedom for S 2. If there were only two groups to be compared, there would be no noncontrast variation, in which case r effect size. In the same way that we might use the contrast sums of squares and within sums of squares to obtain the contrast r, we can find the effect size r from the contrast sums of squares and total sums of squares by r effect size SS contrast SS total, (4) where SS total SS contrast SS within SS noncontrast, and SS noncontrast SS between SS contrast. For Theory A, substitution in equation 4 yields r effect size 6/ Using equation 2 in the case of Theory A yields 6/(6 20).8018, as does equation, where we find.795/(.795) , a value not too much greater than that of r effect size (.762). For Theory B, equation 2 gives us 8.4/(8.4 20).8109, as does equation, where.919/(.919) , and equation 4 gives r effect size 8.4/ Comparing Competing Contrasts Both theories fared well, but suppose we wanted to evaluate the accuracy or predictive power of the contrast for Theory A relative to the contrast for Theory B. To do so, we compute anothe on the difference between the weights of the two competing predictions. When contrast weights are added or subtracted, their sums and differences are influenced more by the contrast weights with larger variance than by the weights with smaller variance. Thus, to be sure that the comparison is fair (i.e., not simply reflecting the contrast with greater variance), we will standardize the weights. This is done by dividing the weights of each contrast by the standard deviation () of the weights, defined as 2 k, (5) where the numerator ( 2 ) is the sum of the squared lambda weights, and the denominator (k) is the number of groups or conditions. For the contrast used to evaluate Theory A, the original A weights are 1, 1, 1,, and thus substitution in equation 5 yields (1) 2 (1) 2 () (1)2 4 Dividing the original A weights by yields new standardized (z-scored) A weights of 0.577, 0.577, 0.577, We now do the same thing for the contrast to assess Theory B, in which the original B weights are, 1, 1,. Using equation 5 gives us () 2 (1) 2 (1) 2 () , 4 and dividing the original B weights by gives us standardized B weights of 1.42, 0.447, 0.447, Subtracting the z-scored A weights from the z- scored B weights gives us the precise weights we need for our difference contrast: 0.765, 0.10, 1.024, With M, n, and S 2 defined as before, we now substitute in equation 1 to find (8) 2(0.765) (0.10) 4(1.024) 7(0.90) (0.765)2 (0.10) , (.6009)(2.5) (1.024)2 (0.90)2 2.5 which has an associated one-tailed p.4. The expected value of is zero, and this is not much larger than zero. Moreover, the alerting correlation is.0458, and thus r 2 alerting.0021, implying that there is virtually no superiority of one theory over another. After calculating SS contrast r 2 alerting SS between , we can substitute in equation 2, which yields , while using equation (with rounding differences) gives (.1844) 2 8 The effect size r, using equation 4, is

5 Contrasts and Correlations 6 r effect size , 62 which, not surprisingly, is also small. We conclude that neither theory was noticeably superior to the other, whereas both fared well on their own. In some cases, results from might differ more from the results from r effect size than they do for this example. Whenever it is possible to use both and r effect size, it seems wise to use both. Combining Competing Contrasts Both theories did so well individually that we wonder how they would do together. To find out, we begin by summing the standardized weights. We recall that the z-transformed A weights are 0.577, 0.577, 0.577, 1.72; and the z-transformed B weights are 1.42, 0.447, 0.447, Summing these values gives us combined s of1.919, 1.024, 0.10,.074. As both theories contributed equally to the combined weights, the combined s should correlate equally with the weights of each theory, and indeed we find that the combined weights correlate.9420 with the A weights and B weights. We now use the combined weights and equation 1 to find 2(1.919) (1.024) 4(0.10) 7(.074) (1.919) ,.497 (1.204)2 (0.10)2 (.074)2 2.5 which has an associated one-tailed p Routinely repeating all the other calculations we did previously, we start with the alerting correlation, which is now The large size of the squared alerting correlation (r 2 alerting.9980) assures us that the combined prediction did exceedingly well in accounting for between-group variation. Multiplying r 2 alerting.9980 SS between 42 gives us SS contrast 41.9, which we substitute in equation 2 to find Alternatively, using equation we find (4.096) 2 8 Given the size of the squared alerting r, it is safe to assume that the contrast correlation is similar in magnitude to the effect size correlation, as there is hardly any noncontrast variability to worry about. Equation 4 confirms this expectation: 41.9 r effect size In sum, Theory A and Theory B are about equally good, and each fared quite well, but combining them provided the most accurate prediction, even though the increase over the two individual theories was not spectacular. If combining the theories makes sense conceptually, the researcher s next task is to articulate a logical argument that connects the two theories in terms of their explanatory statements and assumptions. Case 2: Frequency Data Computing z Contrasts For this next example, suppose the researcher were interested in the effects of four levels of medication to control hyperactivity on the proportion of children reading at grade level. Once again, there are two competing theories, now designated as X and Y. Theory X predicts that, as dosage level increases, a higher proportion of children will achieve reading at grade level. We can express this prediction in terms of X weights of, 1, 1, (where 0). Theory Y, on the other hand, predicts that intermediate dosage levels will be superior to very low or very high levels, which we express using integer Y weights of 1, 1, 1, 1. Table III shows the results of an experiment in which 200 subjects were assigned in equal numbers (n 50) to four levels of medication, ranging from low to very high. The frequency data in rows 1 and 2 represent the number of participants at each medication level who ended up reading at grade level or below grade level. The omnibus chi-square computed on this 2 4 table of counts is (df ), p.190. Row 4 transforms the row 1 frequencies into proportions (P), and rows 5 and 6 are self-explanatory. Row 7 shows the variance (S 2 ) in each column, which is the squared standard error of each proportion, obtained by dividing row 6 by the column ns (i.e., sums) in row, that is, S 2 P(1 P). (6) n

6 64 Rosnow and Rosenthal Table III. Raw Data and Values Needed for z Contrasts Medication level Row Reading level Low Medium High Very high Sum 1 At grade level Below grade level Sum Proportion (P) reaching grade level 5 1 P (P)(1 P) S X weights Y weights To test the competing theories, X and Y, we use the following formula to calculate for each theory: (P) (S 2 2 ), where the numerator is the sum of proportions after each has been multiplied by its corresponding contrast weight, and the denominator is the square root of the sum of the variances after each has been multiplied by its corresponding squared contrast weight. Substituting in equation 7 using X weights (to assess Theory X) gives (.20)() (.0)(1) (.40)(1) (.0)() (.002)(9) (.0042)(1) (.0048)(1) (.0042)(9) ,.0756 which has an associated p.07. Substituting in equation 7 using Y weights (to assess Theory Y) gives (.20)(1) (.0)(1) (.40)(1) (.0)(1) (.002)(1) (.0042)(1) (.0048)(1) (.0042)(1) ,.0164 and p.059. Alerting and Contrast Correlations We again get an estimate of how well the two contrasts did by computing thei correlations either for our own data or in other people s data. However, we now think of the proportions (P) in row 4 as analogous to column means, redefining as the correlation between the proportions and weights. Correlating the proportions and X weights gives us.62, while correlating the same proportions with the Y weights gives us.707. Squaring these alerting correlations tells us in SS terms that Theory X accounted for 40% and Theory Y, 50% of the between-condition sum of squares. In other words, there is a substantial amount of residual noncontrast variation that cannot be accounted for by either competing prediction. With large N (e.g., large enough that the expected frequency of each cell of the table of counts is at least 4 or 5), we can also approximate from the square root of r 2 alerting omnibus 2 (i.e., 2 with df 1), that is, (r 2 alerting )(2 overall ). (8) In this case, Theory X yields (.400)(4.762) 1.802, p.084, whereas Theory Y yields (.500)(4.762) 1.540, p.061. The contrast correlation, in the case of proportions in 2 k frequency tables, would be the partial correlation between the dummy-coded scores (e.g., grade level 1, and below grade level 0) and weights after removing all the noncontrast variation. With large N, the contrast r can be found from N, (9) where N total number of observations. For the data of Table III, therefore, for Theory X 1.802/ , and for Theory Y 1.540/ Equation 9 can be used to obtain a lower limit estimate of from a p value reported only as significant at.05 (or.01 or.001, etc.). Suppose a researcher reported that, using a focused statistical test and N 70, the effect of a

7 Contrasts and Correlations 65 The two cases that we have described used data on two different levels of measurement and in each case illustrated the statistical procedure that seemed to extend most naturally to those results. However, the metric of measurement in which a dependent variable comes to us usually makes little difference as to allowable statistical procedures. For example, whether we think of a variable as nominal, ordinal, interval, or ratio really depends on the underlying construct that the variable is supposed to reflect (Rosenthal et al., 2000). Suppose the metric of measurement were grade levels of students in kindergarten through eighth grade. If the underlying construct were a categorization of the children, then grade level would be viewed as a nominal variable. If the underlying construct were the highest grade level yet attained, then grade level would be seen as ordinal. If the underlying construct were exposure to formal educational material, then grade level could even be considered interval or ratio (i.e., ignoring the prekindergarten formal educational material). Moreover, even when a variable bears the desired relation to the underlying construct, the traditional restriction on which computations we can (or may) perform is seldom justified, since even the traditional approach sometimes instructs us to do things that might be contradictory. For example, one is told that with ordinal scales, multiplication and addition are not allowed. Also, summaries like the product-moment r are not allowed, and instead one should use the rank-order correlation. But the rank-order correlation is the Pearson productmoment correlation between the two sets of ranked scores (Rosenthal & Rosnow, 1991), which themselves are presumably only ordinal (but not necesspecified pediatric treatment was significant at p.05 (but gave no further details). Turning to a table of tail areas of the normal curve, we find one-tailed p.05 has an associated z Equation 9 tells us that the lower limit of would be 1.645/ Comparing and Combining the Competing Contrasts Neither theory did extremely well alone, but suppose we were nevertheless interested in comparing them. The contrast weights comparing these competing theories are again given by the differences between the corresponding contrast weights in z- score form. To obtain these new standardized weights, we begin by substituting in equation 5, with k 4, and the lambda weights listed in rows 8 and 9 of Table III. For Theory X we find (1) 2 (1) 2 () , ()2 4 and then dividing the X weights by gives us new standardized X weights of 1.42, 0.447, 0.447, and For Theory Y we find (1) 2 (1) 2 (1) 2 1.0, (1)2 4 so the Y weights are unchanged, remaining at 1, 1, 1, 1. Subtracting the z-scored X weights from the unchanged Y weights gives us difference weights of 0.42, 1.447, 0.55, and 2.42, which substituted in equation 7 yields (.20)(0.42) (.0)(1.447) (.40)(0.55) (.0)(2.4) (.002)(0.1170) (.0042)(2.098) (.0048)(0.058) (.0042)(5.4850) , 0.7 with an associated p.454. The contrast correlation for this comparison contrast, obtained from equation 9, is.115/ As we expected, equation 9 reveals little difference in the predictive power of the two competing theories. Neither of the two contrasts did all that well alone, but we wonder whether they would do better if we combined them. We can examine their combined effect by summing the z-scored s, which gives us new weights of 2.42, 0.55, 1.447, The combined s are correlated.707 with the contrast weights of Theory X and with the contrast weights of Theory Y. (Since X and Y contributed equally, they should be correlated equally.) Substituting in equation 7 for the combined contrast yields (.20)(2.42) (.0)(0.55) (.40)(1.447) (.0)(0.42) (.002)(5.4850) (.0042)(0.058) (.0048)(2.098) (.0042)(0.1170) ,.0294 with an associated p.014. Substituting in equation 9, we find 2.211/ , while was Both and, therefore, were noticeably larger for the combined than for the individual theories. Conclusions

8 66 Rosnow and Rosenthal sarily if the investigators felt those ranks were interval with respect to the relevant underlying construct). Furthermore, the computation of r involves multiplication and addition of these ranks. To sum up, we began by alluding to the habit of many researchers of consulting omnibus F tests that are only vaguely related to their question of interest. Johnny Weissmuller, who played Tarzan in the movies, once described his philosophy of life as not letting go of the vine. This maxim is also sound advice for researchers who let go of predictions of interest without ever realizing it, distracted by omnibus F tests or phantom limitations of the metric of measurement. In the end, the important thing is to hang on to the prediction at least long McGuire, W. J. (1986). A perspectivist looks at contextualism and the future of behavioral science. In R. L. Rosnow & M. Georgoudi (Eds.), Contextualism and understanding in behavioral science: Implications for research and theory (pp ). New York: Praeger. Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York: McGraw-Hill. Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Conenough to evaluate it. The contrast and correlational procedures we have described are ideal in this respect, because they encourage researchers to be precise about what it is they want to know and to provide a systematic approach for assessing, comparing, or combining alternative predictions. Acknowledgments The first author thanks Temple University for support through the Thaddeus Bolton Professorship. Received September 1, 1999; revisions received May 1, 2000; accepted June 1, 2000 References trasts and effect sizes in behavioral research: A correlational approach. New York: Cambridge University Press. Rosnow, R. L., & Rosenthal, R. (1996). Computing contrasts, effect sizes, and counternulls on other people s published data: General procedures for research consumers. Psychological Methods, 1, Rosnow, R. L., Rosenthal, R., & Rubin, D. B. (2000). Contrasts and correlations in effect size estimation. Psychological Science, 11,