AN INDEX OF THE DISCRIMINATING POWER OF A TEST. Richard Levine and Frederic M. Lord

Transcription

1 RB ~ [ s [ B A U ~ L t L I-t [ T I N AN INDEX OF THE DISCRIMINATING POWER OF A TEST AT DIFFERENT PARTS OF THE SCORE RANGE Richard Levine and Frederic M. Lord This Bulletin is a draft for interoffice circulation. Corrections and suggestions for revision are solicited. The Bulletin should not be cited as a reference without the specific permission of the author. It is automatically superseded upon formal publication of the material. Educational Testing Service Princeton, New Jersey September 1958

2 AN INDEX OF THE DISCRIMINATING POWER OF A TEST AT DIFFERENT PARTS OF THE SCORE RANGE Abstract An important characteristic of any test is its ability to discriminate among eaminees who differ in ability. The conventional reliability and validity coefficients are indices of discrimination for the test as a whole; however, ecept under certain limited conditions, these over-all indices do not apply at all points along the score scale. The purpose of the present paper is to provide such an inde and illustrate its use.

3 I AN INDEX OF THE DISCRIMINATING, POWER OF A TEST AT DIFFERENT PARTS OF THE SCORE RANGE An important characteristic of any test is its ability to discriminate among eaminees who differ in ability. The conventional reliability and validity coefficients are indices of discrimination for the test as a whole; however, ecept under certain limited conditions, these over-all indices do not apply at all points along the score scale. The literature on measurement provides no satisfactory practical inde of a test's discriminating power at different score levels. The purpose of the present paper is to provide such an inde and illustrate its use. A basic logical necessity for the development of an inde of discriminating power is a definition of the characteristic(s) the "operational" test is supposed to measure either in abstract terms or in terms of empirical meaiurements. For one-factor tests the common factor of the test items has many theoretical advantages, but the computational requirementa usually make it an impractical criterion. If there is a single clear-cut criterion that the operational tes'g is supposed to predict, the scatterplot of test scores versus criterion contains all the information necessary to evaluate the effectiveness of the test at different score levels. When, as is usually the case, there are many possible criteria, then a reasonable epedient' is to substitute the scores on a' "criterion test" designed to measure the same trait as the test under study. If we use a criterion test, we must be willing to accept its score scale as constituting a valid metric for our purpose. Otherwise, low discrimination indices might be attributable to a faulty criterion scale rather than to the operational test's poor discriminating power. A minimum essential characteristic of the criterion test is that it should

4 -2~ be of about medium difficulty for eaminees whose scores on the operational test are in the range where discrimination is being evaluated. This implies that if we are interested in discriminating power at both etremes of the operational score scale, then two criterion tests will probably be needed. Given a scatterplbt between a test and an acceptable criterion} it is a problem in descriptive statistics to epress numerically the effectiveness of the test at different score levels. No statistical or psychometric assumptions are needed in order to obtain the desired inde. The Inde In a given scatterplot the discriminating power of the predictor variable ( ) at any point in its own scale depends both on the standard error of estimate (s y. ) at that point and on the slope of the regression curve ( dm y. = b I ) at the same point. a y A steep slope and a small standard error of estimate are associated with effective discrimination} and, conversely} a gentle slope and large standard error are associated with poor discrimination. Thus, the proposed inde of discriminating power at score on the operational test is } (1) represents the ratio of the quantity (amount of overlap in the conditional distributions of y tor two neighboring values of ) to the quantity (distance between the two neighboring values of ). ~ord, F. M. "A Theory of Test Scores," Psychometric Monograph No.7, The William Byrd Press, pp

5 -3- ~nthe special case when the scatterplot is homoscedastic ~nd the regression is linear, it is readily shown that (2) for all values of ; that 1,8, the test is equally eff'icient at all values' of. This implies that in situations where the discriminating power of the test is epected to differ at different values of' ) the regression is likely to be curvilinear and the scatterplot is likely to be heteroscedastic. Computational Methods The actual values of D at various score points are estimated from the means and standard deviations of arrays of criterion scores ( y ) associated with a fied operational test raw score, X. In order to determine the, slope, b yj,at raw score X,Y~~Y,c?mpute the means of crite:l:'ionscores as'socia;te'd with 'scores' '+"'1 'S::ti-dX..;:l. Thedif.. ference between means associated with X + 1 and X-I is proportional to the estimated slope at X. To convert to epressed in standarddeviation units, each such raw-score difference is multiplied by s~/2 (i.e., multiplied by s and divided by the difference between the two values of X used -- in this case by (X + 1) - (X - 1) = 2). The standard error of estimate, s,at score X is the conventional stany. dard deviation of criterion scores f'or eaminees with score X. The ratio b I Is is the value of D at score X. Y y. The epected value of D for a homoscedastic linear scatterplot serves as a standard against which we may judge the computed values at

6 -5- the operational test. Because there were only a few eaminees at each. score point, a rolling average of three score points was used to compute the means and standard d.eviations of criterion scores for fied predictor scores. As an eample of the method of computing the rolling averages we may consider the reported data for.the score 15. First, an average criterion score was computed for each of two samples, one with operational test scores 15, 16, and 17 and one with operational test scores 13, 14, and 15. These averages were considered the mean criterion scores for scores 16 and 14, respectively. The difference between them was multiplied by 8 /2 = 7.98 to obtain b I ' the estimate of the slope of the regression y line, at score 15. Net a variance of criterion scores was computed for each of the three samples with test scores 14, 15, and 16. Each variance was then multiplied by the number of cases in the sample it represented and the resulting products were summed and divided by 3 less than the total number of cases in the three samples. The square root of the resulting unbiased estimate of the variance is the reported value of s y for score 15. Finally each is the ratio of the appropriate pair of values of byl and Sy.' In the total group, only the lower tail of which is treated in Fig. 1 and Table 1, s = and r y =.78. The "standard" value for D is found from (2) to be 1.3. The values in Table 1 are not systematically different from 1.3. For raw scores below 5, there are more values of D that eceed the standard than there are for raw scores from 10 to 19. However, this difference is doubtless attributable to random fluctuations.

7 -6- Table 1. Values of b I J 6,and D for the Scores -6 y y. Through 19 on a 107-Item Test Raw Score b s D (X) yl y ; a ll

8 One characteristic of these data should be noted. As can be seen in Figure 1, the number of cases at each score point is small. Although the precise standard error of D is not known, it seems clear that several times as many cases as were available would be needed to obtain stable values of D'. A coarser grouping for the computation of the rolling averages would also help to produce greater stability. Despite th~l1mitations imposed by small sample size, there is one striking finding. It seems that discrimination eists in the negative score range. In fact, there 1s no appreciable difference between the discrimination indices in the negative score range and those for the rest of the score range studied. The available sample is not sufficiently large to warrant any conclusion that the test measures equally well in the negative and positive ranges, however. Negative scores are bound to; have some discriminating power, if only because an eaminee whose true score is +10, say, is less likely to obtain an actual score of -3 than is an eaminee whose true score is Furthermore, because of careful test construction, the distractors in the test items consistently tend to appear to some people to be more plausible than the correct answers. A 1'urther possible eplanation is that negative scores represent lack of test sophistication rather than unlucky guessing. In the present sample, the most popular wrong answers on 1; few difficult reading passages were numbers or phrases te:ken directly from the test, whereas among more sophisticated eaminees these alternatives were not unusually attractive. It seems possible that unsophisticated test behavior might be sufficiently consistent to produce a correlation between below-zero scores on one test and near-zero scores ona second test.

9 One characteristic of these data should be noted. As can be seen in Figure 1, the number of cases at each score point is small. Although the precise standard error of D is not known, it seems clear that several times as many cases as were available would be needed to obtain stable values of D A coarser grouping for the computation of the rolling averages would also help to produce greater stability. Despite the~limitations imposed by small sample size, there 1s one striking finding. It seems that discrimination eists in the.negative score ~ange. In fact, there is no appreciable difference between the discrimination indices in the negative score range and those for the rest of the score ra.nge studied. The available sample is not sufficiently large to warrant any conclusion that the test measures equally well in the negative a.nd positive ranges, however. Negative scores are bound to have some discriminating power, if only because an eaminee whose true score is +10, say, is ~esb likely to obtain an actual score of -3 tha.n 1s an eaminee whose true acore is Furthermore, because of careful test construction, the diatractors in the test items cons~stently tend to appear to some people to be more plausible than the correct answers. A,further possible eplanation is that negative scores represent lack of test sophistication rather than unlucky guessing. In the present sample, the most popular wrong answers on ~ few difficult reading passages were numbers or phrases taken directly from the test, whereas among more sophisticated eaminees these alterna~ives were not unusually attractive. It seems possible that unsophisticated test behavior might be sufficiently consistent to produce a correlation between below-zero scores on one test and near-zero scores on a second test.

10 ' ~ U} ~ a $ H I l.f\ CIJ >t :3 2 2., 8 1 : J :; II Item Test Fig. l. Relationship between scores on the 25-item and 107-item tests.