Table III.1: Probabilities for a Two-Item Test. Item 1

Transcription

1 The Dsappearng Beta Trck The essental attrbutes of a Rasch model are suffcent statstcs and separable parameters, whch allow, but don t guarantee, specfc objectvty Well, actually suffcent statstcs come pretty close f they really are suffcent to capture all the relevant nformaton n the data We wll come back to ths n the dscusson of what Rasch called control of the model and most of us call goodness of ft The current topc s a demonstraton, more ntutve than mathematcal, of how to manpulate the model to estmate tem dffcultes The process begns wth the basc Rasch model for how lkely the person wns when one person takes one dchotomous tem: 5 Prob(Correct Response ) B represents the ablty of person and Δ the dffculty of tem The complementary expresson for when the tem wns nstead of the person s: 6 Prob(Incorrect Response ) And, of course, the two outcomes, rght or wrong, cover the unverse of possbltes so the probabltes add to one If one person takes two tems, there are four possble outcomes: both rght, both wrong, the frst rght and the second wrong, or the frst wrong and the second rght If the responses are ndependent, then the probabltes for the four outcomes can be computed wth the product of the ndvdual, ndependent probabltes The four possbltes, n Table III, cover all the possble outcomes of the two tem test so these probabltes must also sum to one but that takes a lttle more algebra to demonstrate The upper left (both wrong) and the lower rght (both rght) provde no nformaton about the relatve dffcultes of the two tems; they only say that both are ether too easy or too hard for our person; nothng about whch of the two mght be easer or harder We are only nterested n the two possbltes wth one tem correct and the other not, whch are the upper rght and lower left cells So our unverse has gotten smaller We are makng our analyss condtonal on beng n one of the two shaded cells, e, condtonal on a raw score of exactly one tem correct ITEM Table III: Probabltes for a Two-Item Test Incorrect Correct Incorrect Item Correct The condtonal probablty that tem s the correct one gven a total score of s the probablty n the upper rght cell dvded by the sum of the upper rght and lower left Ths

2 forces the probabltes for our new unverse to agan sum to one and the result reduces mraculously 7 Prob(Item Correct or Correct) Of course, there s an analogous expresson for the probablty that tem s the one correct gven a total score of 8 Prob(Item Correct or Correct) The person s ablty has dsappeared completely from the expresson for tem dffculty Ths was possble because the parameters were separated and s also a somewhat backdoor demonstraton that raw score s the suffcent Person ablty has statstc The suffcency argument follows because we restrcted completely vanshed ourselves to the cases wth a raw score of one Ths s a very small from the expressons llustraton of Rasch s very Bg Idea for tem dffculty Specfc objectvty (aka, person-freed tem estmaton) means that, whle t s necessary to nvolve people to obtan data, t does not matter what people Wthn reason Dong the Math: Connectng the Model to Data If many people take the same two tems, counts can be tabulated for the four possble outcomes Ths provdes very straghtforward estmates for the probabltes n the Table III and for expressons 7 and 8 The probablty n the upper rght can be estmated by: 9, P and smlarly for the lower left cell, where s the count of people who answered tem correctly and tem ncorrectly, s the count of people who answered tem ncorrectly and tem correctly, and s the total number tested From expresson 7, D D D The atn D replaced the Greek Δ once we had brought real data nto the equaton to ndcate estmates of parameters rather than parameters themselves A smlar expresson can be wrought from equaton 8 At frst glance, they seem to provde two equatons wth two unknowns, but they reduce to the same thng D, or ts recprocal D

3 Ths expresson makes ntutve sense D s the dffculty of tem and s the count of people who found tem harder than tem ; s the count of people who found tem easer than tem If the two counts are equal, then the tems are equally dffcult and D equals D If s large compared to, then D s large compared to D The abltes of the people n the sample have no effect on ether of these ratos and for good reason Whether or not an tem s easer or harder than some other tem has nothng to do wth the ablty of any people who mght or mght not take the tems There are some loose ends that should be tded up Frst, t may not be obvous that expresson for a group of people follows mmedately from expresson 7 for one person We just need to nvoke the mathematcal face of a Rasch model: Separablty If the probablty of Item correct and Item ncorrect for some arbtrary person s gven by the upper rght cell of Table III, the expected number of people wth that pattern s found by summng the probabltes for everyone n the group: v P v v ( v )( v ) Because does not have the subscrpt for the people, t can be moved n front of the summaton symbol; every term nvolvng the people wll cancel out as before whether we are talkng about one person or everyone n the US, leavng expressons 7, 8, and ext, the equatons have been kept uncluttered by expressng the parameters and ther estmates n the exponental metrc rather than logts The model would look more famlar, although a v lttle messer and a lot harder to type, f e were substtuted for Bν and e were substtuted for Δ Usng logts, expresson becomes: 3 d d ln or ts negatve ln Fnally, n order to actually attach numbers to d and d wth one equaton and two unknowns, we need to adopt some conventon for where zero s It can take many forms; eg, d = k, or d = k, or more generally cd + cd = k, where c and k are convenent constants The most convenent s d = The most common s d + d = where c and k take the very convenent values of and respectvely Then we can fnsh up wth: 4 d ln ln and d ln ln d Ths s one possble conventon that gves actual values to d and d Another popular choce s to anchor one of the tems, say, d = x Then from (3), 5 d x ln ln and d = x Wth ether conventon, the relatve dffcultes are the same and both preserve the relatonshp of expresson 3: d d ln ln Any other choce for c and k would be just as legtmate and useable; perhaps not as convenent nor conventonal

4 If You Ever Use Tests onger than Two Items The prevous secton would be cute but not be very profound f t dd not hold for tests longer than two tems Choppn s Parwse algorthm extends the logc to all possble pars of tems For each par and j, count the people who passed but faled j and vce versa, just lke we dd for tems and n the precedng sectons Modfyng that notaton a lttle, nj s the count of the people who attempted both and j and who passed j but faled In the old notaton, we had n nstead of nj f tem s and tem s j; and nj replaces n If there are a total of tems, the nj can be arranged n an x matrx, call t *, n whch row contans the counts where tem was ncorrect and column j contans counts where tem j was correct The matrx * of these counts s converted to a matrx R of logarthms of ratos by dvdng nj by nj and takng the natural log More succnctly, 6 rj ln( nj / n j) ln( nj ) ln( n j) d d j If all the off-dagonal elements of * contan non-zero entres, then every element of R wll contan a value rj = d dj Ths s exactly the same (wth slghtly dfferent notaton) as expresson 3 As a matter of housekeepng, every dagonal element of R should be defned as zero, whch n fact represents the case dk dk Obtanng the total for each row, ncludng the zero on the dagonal, 7 rj d d j j d j j d j Imposng the generalzed conventon that d j j and rearrangng a lttle: 8 rj j d Extractng the estmate for any s trval n the complete case where * and R are full: the estmate d s the average of row of R Once agan, problem solved wth no mathematcal gymnastcs Completng the Sum: A on-teratve Soluton Expresson 7 apples to the stuaton n whch all off-dagonal elements of * are non-zero Ths smply means that every tem was pared wth every other tem and that the tem sometmes won and sometmes lost compared to the other member of the par The world s not always so accommodatng The soluton of the prevous secton can be expressed succnctly, f somewhat more erudtely, n matrx notaton as: 9 Ad = S, The dagonal, where = j, wll always be zero because the two condtons cannot hold smultaneously

5 where A s an x matrx of known (but not yet dsclosed) coeffcents; S s an x vector contanng the sum of each row of the matrx R; and d s an x vector of the tem dffculty estmates d that we are pursung In the complete case, A s an x dagonal matrx wth the constant along the dagonal Ths s just a more pedantc way of expressng the answer of the last secton, whch was, fnd the row averages If the matrces * and consequently R are not complete, expresson 9 s stll the answer but the matrx A s not a smple dagonal For example, f element (rk) s mssng because ether nk or nk or both were zero, then the sums of both rows and k are defcent Part of equaton 7 s mssng Because the element (rk)=(ddk) s not there, the row sum s changed: 3 rj d d j; jk j j ( d d ), k whch starts wth expresson 7 and subtracts the pece that sn t there Wrtng the expresson n ths form makes t easy to adopt the same conventon, e, verson of the A matrx 3 ( ) d d k rj, j d j j, and to envson a modfed whch compares to expressons 7 and 9 More mssng elements would mean a smaller coeffcent for d and addtonal dk to be added back n To repeat n general terms, the matrx A requred by expresson 9 begns wth an x matrx wth for each dagonal element and n each off-dagonal element After the A-matrx has been constructed assumng no mssng cells, the off-dagonal elements of the *-matrx are scanned row-by-row for stuatons where nj s zero, and f one s found, Subtract from the dagonal element (,) of A, and Add to the off-dagonal element (,j) of A The rows of A wll always sum to Then the estmates of the tem dffcultes can be obtaned wth expresson 9, whenever a soluton exsts Typcally, a soluton wll exst unless a row (and column) contans no non-zeros or there are blocks of tems that are not connected to other blocks Ths s the same as sayng two forms are not lnked through common tems or common persons Demonstraton of Parwse Calculatons The data presented n Table III demonstrate the requred calculatons for a smple case The data were smulated usng 5 examnees wth logt ablty equal to zero and fve tems wth logt dffcultes of (-3, -,,, 3) wth no random component The *-matrx s the number expected to pass one and fal one tem n each parng The value 7 n the second column of the frst row means 7 examnees faled tem and passed tem The value 8 n the frst column of the As descrbed here, the procedure scans every row n ts entrety More effcent, but more dffcult to descrbe, algorthms can be readly devsed that would only scan the upper or lower trangle of the matrx The small gan n computng effcency hardly makes t worth the effort

6 second row ndcates 8 examnees passed tem but faled tem There are no empty cells so the soluton s easy Table III: Demonstraton of Parwse Calculatons ( = 5; Ablty = ) ogt Dffcultes *-matrx of counts The R-matrx below s computed from the *-matrx above For example, the value n the frst row, second column of R s ln (n / n) = ln (7 / 8) = -9 Analogously, the value n the second row, frst column s ln (n / n) = ln (8 / 7) = 9 We now have data ndcatng that tem s two logts easer than tem The fnal two steps n the calculaton are to sum the fve values n each row and, snce there are no mssng values, dvde by 5 Table III3: Demonstraton of Parwse Calculatons ( = 5; Ablty = ) Row Recovered R-Matrx of og Ratos Sum Parameters The values nsde Table III3 are the comparsons of each par of tems; exactly what we would have gotten f we had treated ths as two-tem tests The row averages n the last column consoldate all the nformaton and express the dffculty for each tem as ts dstance from the center Ths happened because we chose the conventon d j j conventon the numbers would be dfferent but the relatonshps the same ; f we choose a dfferent To llustrate the estmates really do not depend on the ablty dstrbuton of the sample, the demonstraton was repeated for four levels of ablty (,, and 3) The recovered parameters are shown n Table III4 for the four cases Table III4: Recovered Parameters for Four Abltes Orgnal Ablty Dffculty Wth a sample mean of 35 logts, the stuaton became unbalanced enough to leave one cell (e, cell,5) of the *-matrx empty Ths n turn leaves two cells (,5 and 5,) of the R-matrx undefned It s stll possble to obtan estmates but t nvolves a lttle more effort

7 Table III5: Demonstraton wth Ablty = 35 ( = 5) ogt Dffculty *-Matrx of Counts R-Matrx of og Ratos Row Sum Recovered Dffcultes When some counts are zero, at least some of the fve equatons must be solved smultaneously and so the A-matrx of coeffcents contans some non-zero off-dagonal elements Completely flled rows are as easy as ever For example from table III4, d = -558 / 5 = -6 The A-matrx of coeffcents, expresson 8, needs to reflect the empty cells and the reduced number of terms n the summatons Usng the data from Table III5: d d d d 4 d The equatons nvolvng tems and 5 are readly solved wth any of a varety of tools for solvng smultaneous equatons Ths example s smple enough to do by hand Because the estmaton procedure s based on condtonng out ablty, one mght ask f the demonstraton works only f t s based on groups wth homogeneous abltes Table III6 demonstrates that usng a sample wth a mxture of abltes does not affect the result Ths s a more general demonstraton of Rasch s Specfc Objectvty Table III6: Demonstraton of Parwse Calculatons for Mxed Ablty Sample (,,, 3, & 35) Generatng Total Generatng *-Matrx of Counts Dffcultes Count Abltes ,,, 3, &

8 R-Matrx of og Ratos Row Sum Recovered Dffcultes All these demonstratons were done wth no random error component n the smulatons, so one mght expect to recover the generatng parameters exactly The ssue preventng ths s counts of examnees must be whole numbers Had we used the theoretcal p unrounded for j, the recovery would have been perfect Whle we keep nsstng that the dstrbuton of ablty does not matter, the *-matrx n Table III5 llustrates the neffcency of an off-target sample Whle the estmates are fully condtonal so that they do not depend on the ablty dstrbuton of the examnees, the estmates wll be based on fewer examnees wth an off-target sample, mplyng poorer estmates Whle tables III3 and III5 began wth the same number (5), approxmately 9% provded useful data comparng tems and 5 n the frst case and less than 4% dd n the second case Only sx of the 5 cases provded data for the comparson of tems and n Table III5 The very low counts n the frst row of Table III5 contrbuted to sgnfcant roundng error and to a recovered parameter that dffered by 7 logts from the orgnal The soluton s, don t gve people napproprate tests