Small-Sample Equating With Prior Information

Transcription

1 Research Report Sall-Saple Equatng Wth Pror Inforaton Sauel A Lvngston Charles Lews June 009 ETS RR-09-5 Lstenng Learnng Leadng

2 Sall-Saple Equatng Wth Pror Inforaton Sauel A Lvngston and Charles Lews ETS, Prnceton, New Jersey June 009

3 Abstract Ths report proposes an eprcal Bayes approach to the proble of equatng scores on test fors taken by very sall nubers of test takers The equated score s estated separately at each score pont, akng t unnecessary to odel ether the score dstrbuton or the equatng transforaton Pror nforaton coes fro equatngs of other test fors, wth an approprate adjustent for possble dfferences n test length Key words: Equatng, equpercentle equatng, sall saple, eprcal Bayes, collateral nforaton

4 The Proble Often, a new for of a test s taken at ts frst adnstraton by a sall nuber of test takers For soe tests, the nuber s usually saller than 00, frequently saller than 50, and soetes saller than 0 Equatng scores on the bass of such a sall saple of test takers s lkely to produce a poor estate of the equatng relatonshp n the target populaton Yet, the statstcans responsble for deternng the raw-to-scale score converson need an estate of the equatng relatonshp n te to report scores for that sall group of test takers That estate needs to be the best estate of the equatng relatonshp n the target populaton that can be obtaned wth the nforaton avalable Soothng the score dstrbutons before equatng tends to prove the accuracy of the equatng (see, for exaple, Lvngston, 993), but f the saples are very sall, the score dstrbutons n the saples ay dffer fro the dstrbutons n the populaton n ways that soothng wll not correct One possblty for provng the stablty and accuracy of an estate s to ncorporate collateral nforaton nto the estaton process The use of collateral nforaton to prove the accuracy of an estate n educatonal testng dates back at least to Kelley (947, quoted n Lord & Novck, 968, p 65), who proposed usng nforaton fro a group of test takers n the estaton of the true score of an ndvdual eber of the group More recently, statstcans have used collateral nforaton to prove the accuracy of predctons based on test scores (Rubn, 980), ncludng at least one applcaton nvolvng very sall saples (Braun, Jones, Rubn, & Thayer, 983) However, we are not aware of any prevous attept to prove the for-to-for equatng of test scores by usng collateral nforaton fro the equatng of other test fors Those test fors can be prevous fors of the sae test or, alternatvely, fors of other tests that are (n ways that are portant to the equatng process) slar to the test for to be equated The portant slartes ght nclude the type of tes, the extent to whch the tes are nterdependent, the approxate length of the test, and (for anchor equatng desgns) the characterstcs of the equatng anchor Relevant pror nforaton s often avalable The queston s how to ncorporate t nto the process of estatng the equatng relatonshp A practcal procedure need not be theoretcally optal, but t ust prove the accuracy of the estate f not n every case, then at least often enough to justfy ts use It ust be capable of beng pleented farly easly And t should be capable of estatng a curvlnear equatng relatonshp Two unequally dffcult

5 test fors, adnstered to the sae populaton, tend to produce dfferently skewed dstrbutons of scores, resultng n a curvlnear equatng relatonshp An equatng ethod that requres the equatng transforaton to be lnear s lkely to be naccurate n ths coon stuaton Pont by Pont Incorporatng collateral nforaton nto an estate of the equatng transforaton s a coplex proble The approach proposed here s to splfy the proble by reducng t to a seres of replcatons of a uch spler task estatng the equated score that corresponds to a sngle possble score on the new for If ths task can be accoplshed for every possble raw score on the new for, the result wll be an estate of the equatng transforaton (If the resultng estate of the equatng functon s not sooth, the estate can be proved by soothng t ) The pont-by-pont approach to equatng s not a new concept; t s actually the bass for equpercentle equatng (Angoff, 984, pp 97 0; Kolen & Brennan, 004, pp 36 46) In ths respect, t dffers not only fro lnear equatng but also fro kernel equatng (von Daver, Holland, & Thayer, 004) 3 Gven a possble raw score x on the new for, the correspondng equated score on the reference for can be estated by a weghted average of a current value and a pror value The current value s the equated score fro the current equatng, denoted here as y current The pror value s y pror The weght of y current should depend on the confdence t s approprate to place n the results of the current equatng The larger the saple, the ore stable those results are lkely to be, and the ore weght t s approprate to gve to y current The weght of y pror should depend on how any pror equatngs contrbute to t, on the stablty of the nforaton those equatngs provde, and on the extent to whch the nforaton they provde s consstent All of these factors affect the stablty and the usefulness of y pror as a pror estate The ore stable the pror equatngs are and the ore consstent they are wth each other, the ore weght t s approprate to gve to y pror A soluton that has these desred propertes s the eprcal Bayes (EB) estate

6 yˆ EB ycurrent + ypror var( ycurrent ) var( ypror ) ycurrent var( ypror ) + ypror var( ycurrent ) = =, () + var( ypror ) + var( ycurrent ) var( y ) var( y ) current pror wth approprate estates for the varances To adopt ths approach s to treat the current equatng as f t had been sapled at rando fro a large doan of possble equatngs, each wth ts own new for, reference for, and saples of test takers For each cobnaton of new for and reference for n ths doan, there s a true equated score y the equated score that would result fro averagng over any replcatons of the equatng procedure, each wth a dfferent par of saples of test takers The current equatng s a eber of that doan Therefore, ts value of y wll enter nto any estates of quanttes that refer to the entre doan of possble equatngs If there have been any prevous fors of the test that s beng equated, the doan of possble equatngs can be restrcted to nclude only fors of that test Often, however, there wll be few prevous fors to nclude possbly none at all It wll be necessary to specfy the doan of possble equatngs so as to nclude other tests that are slar to the test to be equated In ths case, usng collateral nforaton fro other tests can provde a better estate of the heterogenety of the doan To obtan a value for y pror and estate ts varance, consder the doan of all possble equatngs that are relevant to the current equatng (ncludng the current equatng tself) In each of these possble equatngs, for any gven new-for score x, there s a correspondng referencefor score y Those y values for a dstrbuton, and that dstrbuton has a ean and a varance If the ean of ths dstrbuton were known, t could be used as the pror value Ths dstrbuton cannot be observed, but a sall saple of t can be the values of y that correspond to x n the pror equatngs dentfed as relevant The ean of the values that can be observed s an estate of the ean of the entre dstrbuton, and that ean, denoted here as y, can be used as the pror value For an estate of var( y current ), use the square of the condtonal standard error of equatng, as estated by a procedure approprate for the equatng ethod used For an estate of var( y pror ), use the followng expresson, 4 f ts value s greater than 0 (and 0 f t s not): 3

7 ˆ σ ( ) ˆ t = y y σ = = () In ths notaton, ndexes the equatngs, wth = for the current equatng and = to for the pror equatngs used as collateral nforaton The observed y value n equatng s denoted as y, and the ean of the y values s denoted as y The ter ˆ σ ndcates an estate of the square of the condtonal standard error of the th equatng, at the relevant score level (Often an estate s avalable fro the equatng software) If the condtonal standard error of equatng at a partcular score level s large (as t often s, n the tals of the score dstrbuton), the second ter of Equaton ay well be larger than the frst ter In that case, the resultng estate of 0 for the pror varance wll cause Equaton to splfy to yˆ EB = y pror The EB estate of the equated score wll be the ean of the values observed n all the relevant equatngs Ths estate ncludes the current equatng but gves t no ore weght than any of the others A Coplcaton One coplcaton that users of ths approach are lkely to encounter s that the test fors n the doan of equatngs can dffer n length Even f the doan s lted to fors of a sngle test, soe fors ay have one or ore tes excluded fro scorng (because of probles wth content, prntng errors, etc) Therefore, a prelnary step before applyng the proposed procedure s to transfor, nto percentage ters, the scores on the new for and the reference for, n the current equatng and n all the equatngs to be ncluded as collateral nforaton Ths transforaton conssts of subtractng the lowest possble score, dvdng by the range of possble scores, and ultplyng by 00 One consequence of ths transforaton wll be that fors that dffer n length wll have dfferent sets of possble scores For exaple, a score of 50% s possble on a for wth an even nuber of tes but not on a for wth an odd nuber of tes If the new for n the current equatng has an even nuber of tes, one of the raw scores to be equated wll be 50% But f n one of the pror equatngs the new for has an odd nuber of tes, the table of equated scores for that pror equatng wll not nclude a new-for raw score of 50% It wll be necessary to nterpolate 4

8 The Proposed Procedure The sequence of steps n the proposed procedure s as follows, for a partcular raw score x * on the new for n the current equatng: x* x n Transfor x * to xpct *, where xpct* = 00 xax xn In the current equatng, fnd the equated raw score y current correspondng to x * Transfor y current to ypct current and label t ypct for use n the steps that follow 3 In the frst pror equatng, fnd x, the new-for raw-score value for that equatng for whch xpct = xpct * In any cases, x wll not be a possble raw score, and t wll be necessary to nterpolate between two possble new-for raw scores These possble scores can be denoted as x + and x, chosen so that xpct+ and values of xpct above and below xpct * xpct are the nearest 4 Repeat Step 3 for the pror equatngs, to deterne ypct, ypct 3, and so on 5 Copute ypct pror = = ypct, the ean, over the current and pror equatngs, of the ypct value correspondng to x * 6 Use Equaton to estate var( ypct pror ) fro the values of ypct, ypct, ypct 3, and so on, and the estates of the condtonal standard error of equatng Note that these standard errors wll need to be nterpolated and rescaled to atch the unts of the correspondng ypct values 7 Use Equaton to estate ypct ˆ EB 8 Transfor ypct ˆ EB back to the score scale of the reference for n the current equatng If the functon defned by the successve values of ypct ˆ for ncreasng values of x * s not sooth, apply a soothng procedure EB 5

9 Specfyng the Doan of Collateral Inforaton An portant queston n pleentng ths procedure s how broadly to defne the doan of pror equatngs to be used as collateral nforaton Should they be lted to prevous fors of the test to be equated? If equatngs of fors of other tests are to be ncluded, n what ways ust those tests be slar to the test to be equated? Ths queston s one that can best be answered eprcally, and the answer s lkely to depend on the set of tests beng consdered At a selected percent-correct score level on the new for (eg, 60%), record the equated scores n all the pror equatngs of test fors beng consdered for ncluson n the doan possbly several fors of each test Copare the varaton between tests wth the varaton between dfferent fors of the sae test Repeat ths procedure at several selected score levels If the results do not ndcate a systeatc dfference between tests, t sees reasonable to conclude that pror equatngs of other tests wll provde useful collateral nforaton In decdng whch tests to consder for possble ncluson n the doan of collateral nforaton, the ost portant factor would be the extent to whch the test fors tend to dffer n dffculty Soe tests are constructed fro tes that have been thoroughly pretested on representatve saples of the test-taker populaton Fors of those tests are lkely to show only sall dfferences n dffculty Other tests are constructed fro tes for whch no pretest nforaton s avalable Fors of those tests are lkely to dffer uch ore n dffculty Another portant queston s whether the doan of collateral nforaton should nclude both possble equatngs of each par of fors (e, the equatng of X to Y and the equatng of Y to X) If both are ncluded, the dfferences n dffculty wll tend to cancel each other out n the estaton of y pror n Equaton The pror estate of the equatng transforaton wll be very close to the dentty, and the value of y wll be very close to that of x, when both x and y are expressed as percentages One way to thnk about ths queston s to ask, If n the past, the new fors n the doan have tended to be ore dffcult (or, alternatvely, easer) than the reference fors they were equated to, does that tendency represent a genune trend? If so, s t realstc to expect ths trend to contnue n the future? If the answer to ether of these questons s no, t ay be best to use the dentty as the pror estate In that case, the an functon of the collateral nforaton wll be to estate var( y pror ) 6

10 Advantages and Ltatons Repeatng the proposed procedure for any gven score on the new for to be equated wll produce an estate of the equated score that would result fro equatng n a very large saple fro the target populaton Ths estate should be, n ost cases, a better estate than the equated score pled by the current sall-saple equatng It should also be, n ost cases, a better estate than could be obtaned by dsregardng the current equatng and usng, nstead, the score pled by a prevous equatng or by the average of several prevous equatngs Because the proposed procedure estates the equated score separately for each rawscore value, t does not requre constrants on the for of the equatng transforaton In partcular, t s not constraned to produce an equatng transforaton wth a partcular slope, or even a transforaton wth a constant slope It can use collateral nforaton fro equatngs coputed by dfferent ethods, even f soe of those equatngs were constraned to be lnear and others were not One ltaton of the proposed procedure, fro a theoretcal pont of vew, s that t s not syetrc wth respect to the new for and reference for Therefore, t s not, strctly speakng, an equatng procedure Instead, t s an estaton procedure a procedure for estatng the results that would be obtaned f the current equatng could be perfored wth data fro the full target populaton Even though the functon to be estated s syetrc n X and Y, the best avalable estate of t fro sall-saple data ay not be syetrc n X and Y A ore portant ltaton, fro a practcal pont of vew, s the dffculty of estatng var( y current ) n Equaton The coonly used forulas ay not yeld accurate results when used wth sall-saple data Resaplng studes ay be necessary to develop a predcton forula for var( y current ) as a functon of saple sze, so as not to requre any paraeters to be estated fro the sall-saple data The greatest drawback to the proposed procedure s that there are stuatons n whch t can produce a result that s less accurate than the current equatng If there s reason to beleve, a pror, that the current for dffers n dffculty fro ts reference for n a way that the new fors n pror equatngs dd not, t would be unwse to use the procedure presented here Such a stuaton could arse f the new for were delberately constructed to be easer or harder than prevous fors of the sae test It also could arse f the new for were beng equated to a reference for known to be unusually easy or unusually dffcult 7

11 In conductng resaplng studes to evaluate the proposed procedure, the authors colleague Sooyeon K encountered a stuaton n whch several pror fors of a test were slar n dffculty to the reference fors they were equated to, but one subsequent for was uch harder than ts reference for The close agreeent aong the pror equatngs gave the collateral nforaton a heavy weght n the EB estate, pullng the estate toward an ncorrect value (K, Lvngston, & Lews, 008) The portant practcal queston s, Whch s the greater danger beng sled by the collateral nforaton or beng sled by an anoalous sall-saple equatng result? Soe prevous wrters have taken an extree pont of vew, suggestng that when the avalable saples for equatng are saller than a specfed sze, the rght thng to do s to dsregard the data entrely and assue the new for and reference for to be of equal dffculty at all score levels (Kolen & Brennan, 004, pp 89 90; Skaggs, 005, p 309) But there s an alternatve that sees preferable: usng avalable data to ndcate how uch weght to gve to the sall-saple equatng results and how uch to another estate of the equatng transforaton 8

12 References Angoff, W H (984) Scales, nors, and equvalent scores Prnceton, NJ: ETS Braun, H I, Jones, D H, Rubn, D B, & Thayer, D T (983) Eprcal Bayes estaton of coeffcents n the general lnear odel fro data of defcent rank Psychoetrka, 48, 7 8 K, S, Lvngston, S A, & Lews, C (008) Investgatng the effectveness of collateral nforaton on sall-saple equatng (ETS Research Rep No RR-08-5) Prnceton, NJ: ETS Kolen, M J (984) Effectveness of analytc soothng n equpercentle equatng Journal of Educatonal Statstcs, 9, 5 44 Kolen, M J, & Brennan, R L (004) Test equatng, scalng, and lnkng (nd ed) New York: Sprnger Lvngston, S A (993) Sall-saple equatng wth log-lnear soothng Journal of Educatonal Measureent, 30(), 3 9 Lord, F M, & Novck, M R (968) Statstcal theores of ental test scores Readng, MA: Addson-Wesley Rubn, D B (980) Usng eprcal Bayes technques n the law school valdty studes Journal of the Aercan Statstcal Assocaton, 75, Skaggs, G (005) Accuracy of rando groups equatng wth very sall saples Journal of Educatonal Measureent, 4, van der Lnden, W J (006) Equatng error n observed-score equatng Appled Psychologcal Measureent, 30, von Daver A A, Holland, P W, & Thayer, D T (004) The kernel ethod of test equatng New York: Sprnger 9

13 Notes If the nuber of possble scores on the new for s very large, t ay be necessary to select a subset of the possble raw scores, copute the equated scores for those selected raw scores, and nterpolate for the raw scores not selected See, for exaple, the ethod descrbed by Kolen (984) Ths technque s typcally referred to as postsoothng to dstngush t fro the presoothng of the score dstrbutons before equatng 3 It s also qute dfferent fro the approach taken by van der Lnden (006) n whch unlke equpercentle equatng, there s a dfferent functon for each test taker (p 359) 4 The dervaton s shown n the appendx 0

14 Appendx Dervaton of Estates for y pror and var(y pror ) Let ndex the equatngs to be used n estatng y and var( y ), fro to Each pror pror of these equatngs has ts own new for and ts own reference for Gven a specfed raw-score value x on the new for n equatng, let y represent the correspondng equated score That s, y s the score on the reference for n equatng that corresponds to the specfed raw score x on the new for Suppose t were possble to repeat, nfntely any tes, the process of saplng test takers to take the new for n equatng, saplng test takers to take the reference for n equatng, and perforng the equatng of the new for to the reference for n those saples to observe a value of y If the resultng values could then be averaged over all those replcatons of equatng, there would be a quantty that could reasonably be consdered the true equated score for equatng, at new-for score-level x Let that quantty be denoted by a avg ( y ) =, usng avg to ean the average over nfntely any replcatons of equatng Let e = y a represent the aount by whch a partcular observed value of y s hgher or lower than the average value avg e = 0 and, because the replcatons of equatngs and are a Then ( ) ndependent, for any two dfferent equatngs and, ( ' ) avg ee, ' = 0 Let σ represent the varance of y over the any replcatons of the saplng and equatng procedure of equatng (n other words, the square of the condtonal standard error of equatng at new-for raw score x) Snce Therefore, ( ) Let a does not vary over the replcaton process, var ( a ) = 0 and ( ) var e = var y = σ and, f ' avg e = ( ) ( ) y = wll be the value for cov e, a = 0 =, avg ( ', ') avg ( ) ee = e = σ y =, the average y-value over the equatngs ndexed by Ths quantty y pror n Equaton

15 sa = a a Let ( ) = var( y pror ) n Equaton One plausble estate of Ths s the quantty to be estated, to provde a value for s s ( ) a = y y However, ths estate ay requre a correcton for bas To deterne the correcton needed, frst rewrte the su of squares: ( y y) = ( y a ) + ( a a) + ( a y) = = ( y a ) ( a a) ( a y) = + + = = = ( y a )( a a) ( a y) ( y a ) ( a y) ( a a) = = = Now y a = e for all, and therefore y a = e Consequently, the above expresson can be rewrtten as = = ( ) ( ) e + a a + e ( e )( a a) ( e) ( e ) ( e) ( a a) = = = Snce e e = =, ths expresson can be rewrtten agan as e + a a + e ( ) = = = + ( e)( a a) e ( e) e ( a a), = = = = = and agan, as

16 e + a a + ee ' = = = ' = ( ) ( ) + ( e)( a a) ( ee ' ) e ( a a) = = ' = = = Now the average over replcatons for each of the equatngs can be taken (Note that a does not vary over replcatons of equatng ) Ths average s: avg ( e ) + ( a a) + avg ( ee ', ' ) = = = ' = + avg ( e )( a a) avg ( ee ', ' ) avg ( e ) ( a a) = = ' = = = To splfy ths expresson further, use the followng results developed earler: ( e ) =, avg ( e ) = σ, ( ' ) avg 0 = ' avg ee, ' = 0 for ' These results ake the expresson equal to σ = = = σ + ( a a) +, and avg ( ', ') avg ( ) = = = = ( a a) σ ( a a) + σ + a a = ( ) = = σ + a a = = = ( ) Ths quantty s the expectaton of ( ) = Therefore, the expectaton of ( ) = y y y y s ee = e = σ for over replcatons of all the equatngs 3

17 + σ = = ( a a) Suppose there are unbased estates ˆ σ for each of the σ Then, over replcatons of all the equatngs ( = to ), the expected value of s equal to σ ( ) y y ˆ = = = ( ) a a = s a Thus an unbased estate for s a s ( ) ˆ = s ˆ a y y = σ = 4