A speeded item response model with gradual process change

Transcription

1 A speeded tem response model wth gradual process change Yur Goegebeur Paul De Boeck James A. Wollack Allan S. Cohen June 16, 2005 Abstract An tem response theory model for dealng wth test speededness s proposed. The model conssts of two random processes, a Rasch process and a random guessng process, wth the random guessng gradually takng over from the Rasch process. The nvolved change pont and change rate are consdered random parameters n order to model examnee dfferences n both respects. The proposed model s evaluated on smulated data and a case study. Key words: Rasch model, local tem dependence, test speededness. 1 Introducton Let Y p denote the bnary response (ncorrect/correct, coded Y p = 0 and Y p = 1, respectvely) of examnee p, p = 1,..., P, to tem, = 1,..., I. In the classcal one-parameter Rasch model (1PL) (Rasch, 1960) Y p depends on the examnee ablty θ p and tem dffculty β n the followng way Y p θ p Bern(P (θ p )) wth P (θ p ) = exp(θ p β ) 1 + exp(θ p β ) (1) and θ p N(0, σθ 2 ) f the margnal maxmum lkelhood formulaton s chosen. Moreover, condtonal on θ p all responses of subject p are assumed ndependent; ths s the so-called local tem ndependence condton. Formally, denotng Y p = (Y p1,..., Y pi ), P (Y p = y p θ p ) = I [P (θ p )] y p [1 P (θ p )] 1 y p. =1 The Rasch model has been extended n several ways. In the two-parameter logstc model (2PL) (Brnbaum, 1968) the dfference θ p β s weghted by an tem dscrmnaton parameter α : P (θ p ) = exp(α (θ p β )) 1 + exp(α (θ p β )), (2) 1

2 so that the nfluence of examnee ablty on outcome depends on the tem. The three-parameter logstc model (3PL) (Brnbaum, 1968) extends the 2PL wth an tem specfc guessng parameter c : exp(α (θ p β )) P (θ p ) = c + (1 c ) 1 + exp(α (θ p β )). The parameter c clearly reflects the probablty of a correct answer under random guessng. For further nterpretatons of the 3PL, we refer to Hutchnson (1991). The Rasch model, and tem response theory models n general, are not robust wth respect to volatons of the local tem ndependence assumpton. The ncluson of tems wth local tem dependence may result n contamnated estmates of test relablty, person and tem parameters, standard errors and equatng coeffcents, see for nstance Yen (1984), Thssen et al. (1989), Srec et al. (1991), Yen (1993), Waner and Thssen (1996) and Lee et al. (2001). Yen (1993) and Ferrara et al. (1999) provde a detaled taxonomy of possble reasons for the exstence of local tem dependency. One of the most prevalent causes n educatonal testng s test speededness. Test speededness refers to testng stuatons n whch some examnees do not have ample tme to answer all questons. Speededness effects are often detrmental to the ntended functonng of the test n that the speed wth whch one responds s usually not an mportant part of the construct of nterest. Examnees affected by test speededness hurry through, randomly guess on or even fal to complete tems, usually at the end of the test, and hence receve ablty estmates that underestmate ther capactes. On the other hand, the tem dffculty parameters of tems admnstered late n the test tend to be overestmated (Douglas et al., 1998 and Oshma, 1994). Item response theory models dealng wth test speededness are relatvely new. The hybrd model of Yamamoto and Everson (1997) uses multple tem response theory models to descrbe the behavor of examnees. A classcal tem response model s vald throughout most of the test but end-of-test tems are answered randomly by some subset of examnees. The model dentfes M possble response patterns, one for whom an tem response model s vald for all tems, and M 1 patterns wth an tem response model descrbng answers to the frst I m tems and random guessng on the last m tems, m = 1,..., M 1. Formally, P (m) (θ p (m) ) = exp(α (θ p (m) β )) 1+exp(α (θ p (m) β )) I m, c, > I m, wth m = 0,..., M 1. Clearly, speededness s unlkely to be so straghtforward, as students do not swtch mmedately to random guessng beyond some pont. Bolt et al. (2002) extend the mxture Rasch model proposed by Rost (1990) to dstngush latent classes of examnees accordng to the exstence of speededness n ther tem response 2

3 patterns. Ordnal constrants are mposed on the tem dffculty parameters across classes so as to dstngush a class havng no speededness effects from a class whose responses are affected by speededness. In partcular, for tems early n the test, the tem dffculty parameters are constraned to be equal n the two classes; however, the tem dffculty parameters of end-oftest tems n the speeded class are constraned to be larger than the respectve tem dffculty parameters n the nonspeeded class. Let g denote a class ndcator wth g = 0, 1 referrng to the nonspeeded and speeded class respectvely and let k denote the frst tem where the examnees experence the effects of test speededness. The mxture Rasch model can then be stated as wth P (g) (θ p (g) ) = exp(θ(g) p 1 + exp(θ (g) p β (g) ) β (g) ), β (0) = β (1) for < k, β (0) < β (1) for k. The tem dffculty estmates obtaned n the nonspeeded class provde more sutable estmates of the Rasch dffcultes of end-of-test tems than the dffcultes estmated usng all examnees. Although ths model has worked qute well at dentfyng test speededness, t s lkely overly smplstc as t does not allow for dfferent examnees becomng speeded at dfferent ponts n the test. The remander of ths paper s organzed as follows. In the next secton we propose an tem response model that accommodates the dsadvantages of the hybrd model and the mxture Rasch model. The model can be seen as consstng of two random processes, a Rasch process and a random guessng process, wth the random guessng gradually takng over from the Rasch process. The nvolved change pont and change rate are consdered random parameters n order to model examnee dfferences n both respects. The model was frst formulated by Wollack and Cohen (2004) as a model to smulate speededness data, but t wll be treated here as a full-fledged model for test data whch can also be estmated. In secton 3 we evaluate the performance of the model on the bass of a smulaton study. The fnal secton reports the results of applyng the model to a mathematcs placement test. 2 A model for speeded test data wth gradual process change In ths secton we propose a new tem response model for dealng wth speeded test data. Under the model, responses to tems early n the test are governed by a Rasch model. Beyond some pont the success probablty gradually decreases and eventually reduces to the success probablty under random guessng. Both change pont and change rate are examnee specfc. Usng the same notaton as before, the model can be stated as Y p θ p, η p, λ p Bern(π p ) 3

4 wth π p = c + (1 c )P (θ p ) mn { 1, [ ( )] } λp 1 I η p, (3) where P (θ p ) s gven by (1) or (2), η p (η p [0, 1]) represents the speededness pont and λ p (λ p 0) the speededness rate of examnee p. The speededness pont parameter η p dentfes the pont n the test, expressed as a fracton of the number of tems, where examnee p frst experences an effect due to speedng. For tems wth η p I there s no effect of speedng. Once the examnee passes hs/her speededness pont, /I η p s postve, resultng n a decrease of π p. The rate of decrease of π p s controlled by the parameter λ p, wth larger λ p values resultng n a faster decrease. In Fgure 1 we llustrate the role of η and λ by plottng the decay functon mn{1, [1 (x η)] λ } for some values of η and λ. (a) (b) x x Fgure 1: (a) mn{1, [1 (x η)] λ } for λ = 5, η = 0.5 (sold lne) and η = 0.75 (broken lne), (b) mn{1, [1 (x η)] λ } for η = 0.25, λ = 1 (sold lne), λ = 2 (broken lne) and λ = 0.5 (broken-dotted lne). The ratonale for the proposed model s as follows. Denote P (η p, λ p ) = mn{1, [1 (/I η p )] λp }. When examnee p encounters tem, he/she answers accordng to ether a Rasch process or a random guessng process, wth probabltes P (η p, λ p ) and 1 P (η p, λ p ) respectvely. Under random guessng the answer s correct wth probablty c. Under the Rasch process the examnee knows the answer wth probablty P (θ p ); f gnorant the examnee guesses at random. In Fgure 2 we vsualze the model wth a decson tree. Clearly, P (Y p = 1 θ p, η p, λ p ) = P (η p, λ p )P (θ p ) + P (η p, λ p )[1 P (θ p )]c + [1 P (η p, λ p )]c whch smplfes to (3). Model (3) has some nterestng lmtng cases: 4

5 p, P (η p, λ p ) 1 P (η p, λ p ) Rasch P (θ p ) 1 P (θ p ) guess guess c 1 c c 1 c Fgure 2: Decson tree representaton of speededness model. f [1 (/I η)] λ = 0 for /I > η (ths corresponds to the lmtng case λ + ) then (3) reduces to one of the speeded classes n the hybrd model, and speededness s modeled as random guessng, n case λ = 0 or η = 1, the proposed model reduces to 1PL extended wth random guessng or 3PL, n case η = 0 and λ > 0 the examnee guesses at random at least to some degree for the frst tem up to the fnal tem, smlarly to 3PL, c s the lower asymptote for θ. As s usual n tem response theory, the person ablty parameter s assumed to be normally dstrbuted wth mean zero and varance σ 2 θ. Concernng the parameters η p and λ p we make, wthout loss of generalty, the followng dstrbutonal assumptons: η p Beta(α, β), λ p log N(µ λ, σ 2 λ ). For estmaton we restrct the dscusson to the margnal maxmum lkelhood method. If the model of nterest s gven by (3)-(1) wth a common unknown random guessng parameter c, then the parameters to be estmated are (β 1,..., β I, c, σθ 2, α, β, µ λ, σλ 2 ), whereas under (3)-(2) the parameters to be estmated are (α 1,..., α I, β 1,..., β I, c, α, β, µ λ, σλ 2). In the latter case σ2 θ has to be fxed at some postve constant for dentfcaton purposes. For convenence the vector of unknown parameters wll be denoted by ξ. In the margnal maxmum lkelhood method the random effects are ntegrated out and the resultng lkelhood s maxmzed wth respect to be 5

6 unknown parameters. Under (3) and denotng jont densty functon of θ p, η p and λ p by g the margnal lkelhood functon s smply L(ξ) = P p=1 R I P (Y p = y p θ p, η p, λ p )g(θ p, η p, λ p )dλ p dη p dθ p. (4) =1 The ntegrals nvolved n (4) can be numercally approxmated by a quadrature method and the optmzaton can be performed usng a standard Newton-Raphson algorthm. The SAS NLMIXED procedure fts nonlnear mxed models wth multvarate normal random effect dstrbutons. However, as long as g n (4) s characterzed by a normal dependence structure (copula) NLMIXED can be used to ft model (3), whatever the functonal form of the (contnuous) margnal random effect dstrbuton functons. Indeed, as shown n Proposton 1 (see Appendx 2), n case of a normal dependence functon, approprately chosen compostons of probablty ntegral transforms and nverse probablty ntegral transforms of the margnal dstrbutons yeld a multvarate normal dstrbuton for the transformed random effects. In some cases, besdes ξ also the person specfc effects θ p, η p and λ p are of specal nterest. Estmates of these parameters can be obtaned from an emprcal Bayes analyss of the postulated model. 3 Smulaton study In ths secton we dscuss the results of a small smulaton study. Three data sets each contanng responses of 2000 examnees on 40 tems were generated. Sample 1 was generated under model (3)-(1) wth moderately hgh speededness (α = 9 and β = 2). Sample 2 was generated under model (3)-(1) wth moderately low speededness (α = 20 and β = 9). Fnally, a thrd sample was generated from 1PL wth random guessng. The complete lst of parameter values s gven n Table 1. The random effects are assumed to be ndependent. Table 1: Parameter values for smulaton study. Parameter Sample 1 Sample 2 Sample 3 β 1 - β c σθ α β µ λ σλ The effect of test speededness s llustrated n Fgure 3 (a), Fgure 4 (a) and Fgure 5 (a) where 6

7 we plot the emprcal proportons correct answers together wth the theoretcal ones, gven by E(Y p ) = E[E(Y p θ p, η p, λ p )] = E(π p ) = c + (1 c) n case of (3) and by R P (θ p )dg 1 (θ p ) mn E(Y p ) = c + (1 c) P (θ p )dg 1 (θ p ) R { 1, [ ( )] } λp 1 I η p dg 3 (λ p )dg 2 (η p ), n case of 1PL wth random guessng, where G 1, G 2 and G 3 denote the dstrbuton functons of θ p, η p and λ p respectvely, versus tem number. Snce all β are equal, these proportons should not depend on tem number n the absence of test speededness (see Fgure 5 (a)). Clearly, test speededness decreases the probablty of a correct answer for end-of-test tems. The ultmate effect depends on the dstrbuton of the speededness pont and rate. To do: graphcal presentaton of estmaton results test characterstc curves?? (cf Bolt, Cohen, Wollack) 4 Applcaton to mathematcs placement test 5 Concluson Acknowledgements References [1] Brnbaum, A., Some latent trat models and ther use n nferrng an examnee s ablty. In Lord, F.M. and Novck, M.R. (Eds), Statstcal Theores of Mental Test Scores, pp Addson-Wesly. [2] Bolt, D.M., Cohen, A.S. and Wollack, J.A., Item parameter estmaton under condtons of test speededness: applcaton of a mxture Rasch model wth ordnal constrants. Journal of Educatonal Measurement, 39, [3] De Boeck, P. and Wlson, M., Explanatory Item Response Models - A Generalzed Lnear and Nonlnear Approach. Sprnger. [4] Douglas, J., Km, H.R., Habng, B. and Gao, F., Investgatng local dependence wth condtonal covarance functons. Journal of Educatonal and Behavoral Statstcs, 23,

8 [5] Ferrara, S., Huynh, H. and Mchaels, H., Contextual explanatons of local dependence n tem clusters n a large-scale hands-on scence performance assessment. Journal of Educatonal Measurement, 36, [6] Hutchnson, T.P., Ablty, Partal Informaton, Guessng: Statstcal Modellng Appled to Multple-Choce Tests. Rumsby Scentfc Publshng. [7] Joe, H., Multvarate Models and Dependence Concepts. Chapman & Hall. [8] Lee, G., Kolen, M.J., Frsbe, D.A. and Ankenmann, R.D., Comparson of dchotomous and polytomous tem response models n equatng scores from tests composed of testlets. Appled Psychologcal Measurement, 25, [9] Nelsen, R.B., An Introducton to Copulas. Sprnger. [10] Oshma, T.C., The effect of speededness on parameter estmaton n tem response theory. Journal of Educatonal Measurement, 31, [11] Rasch, G., Probablstc Models for Some Intellgence and Attanment Tests. Dansh Insttute for Educatonal Research, Copenhagen, Denmark. [12] Rost, J., Rasch models n latent classes: an ntegraton of two approaches to tem analyss. Appled Psychologcal Measurement, 14, [13] Srec, S.G., Thssen, D. and Waner, H., On the relablty of testlet-based tests. Journal of Educatonal Measurement, 28, [14] Sklar, A., Fonctons de repartton a n dmensons et leurs marges. Publcatons de l Insttut de Statstque de l Unversté de Pars, 8, [15] Thssen, D., Stenberg, L. and Mooney, J., Trace lnes for testlets: a use of multplecategorcal response models. Journal of Educatonal Measurement, 26, [16] Waner, H. and Thssen, D., How s relablty related to the qualty of test scores? What s the effect of local dependence on relablty? Educatonal Measurement: Issues and Practce, 15, [17] Wollack, J.A. and Cohen, A.S., A model for smulatng speeded test data. Techncal report. [18] Yamamoto, K. and Everson, H., Modelng the effects of test length and test tme on parameter estmaton usng the hybrd model. In Rost, J. and Langehene, R. (Eds.), Applcatons of Latent Trat and Latent Class Models n the Socal Scences, pp Waxmann, New York. [19] Yen, W.M., Effects of local tem dependence on the ft and equatng performance of the three-parameter logstc model. Appled Psychologcal Measurement, 8, [20] Yen, W.M., Scalng performance assessments: strateges for managng local tem dependence. Journal of Educatonal Measurement, 30,

9 Appendx 1: Example SAS code data smdata1; nfle c:\rm\speeded\paper\smdata1.txt ; nput y nr person x1-x40; nr_n=nr/40; run; proc nlmxed data=smdata1 method=gauss noad technque=newrap maxter=500 maxfu=5000 qponts=5; parms b1-b40=-1 c=.2 s2t=1 a=9 b=2 ml=0 s2l=1 ; beta = b1*x1+b2*x2+b3*x3+b4*x4+b5*x5+b6*x6+b7*x7+b8*x8+b9*x9+b10*x10+ b11*x11+b12*x12+b13*x13+b14*x14+b15*x15+b16*x16+b17*x17+b18*x18+b19*x19+b20*x20+ b21*x21+b22*x22+b23*x23+b24*x24+b25*x25+b26*x26+b27*x27+b28*x28+b29*x29+b30*x30+ b31*x31+b32*x32+b33*x33+b34*x34+b35*x35+b36*x36+b37*x37+b38*x38+b39*x39+b40*x40; eta=betanv(probnorm(et),a,b); lambda=exp(la); r=exp(theta-beta)/(1+exp(theta-beta)); s=(1-(nr_n-eta))**lambda; f (s >=1) then pr=c+(1-c)*r; else pr=c+(1-c)*r*s; model y ~ bnary(pr); random theta la et ~ normal([0,ml,0],[s2t,0,s2l,0,0,1]) subject=person; run; 9

10 Appendx 2 Defnton 1 A n-copula s a functon C : [0, 1] n [0, 1] wth the followng propertes 1. for every u [0, 1] n wth at least one coordnate equal to 0, C(u) = 0, 2. f all coordnates of u are 1 except u k then C(u) = u k, 3. for all a, b [0, 1] n wth a b the volume of the hyperrectangle wth corners a and b s postve,.e. 2 1 =1 where u 1 = a and u 2 = b. 2 ( 1) 1+ + n C(u 1,..., u n ) 0 n =1 So essentally a n-copula s a n-dmensonal dstrbuton functon on [0, 1] n wth standard unform margnal dstrbutons. The next theorem, due to Sklar, s central to the theory of copulas and forms the bass of the applcatons of that theory to statstcs. Theorem 1 Sklar (1959) Let X = (X 1,..., X n ) be a random vector wth jont dstrbuton functon F X and margnal dstrbuton functons F, = 1,..., n. Then there exsts a copula C such that for all x R n FX(x) = C(F1(x1),..., Fn(xn)). (5) If F 1,..., F n are all contnuous then C s unque, otherwse C s unquely determned on Ran F 1 Ran F n. Conversely, gven a copula C and margnal dstrbuton functons F 1,..., F n, the functon F X as defned by (5) s a jont dstrbuton functon wth margns F 1,..., F n. As s clear, Sklar s theorem separates a jont dstrbuton nto a part that descrbes the dependence structure (the copula) and parts that descrbe the margnal behavor (the margnal dstrbutons). For further detals on copula functons we refer to Joe (1997) and Nelsen (1999). Proposton 1 Consder a n-dmensonal random vector X wth jont dstrbuton functon G and contnuous margnal dstrbuton functons G 1,..., G n. Assume that G s characterzed by a normal dependence functon (copula) C.e. wth C(u 1,..., u n ) = G(x 1,..., x n ) = C(G 1 (x 1 ),..., G n (x n )) Φ 1 (u 1 ) Φ 1 (u n ) 1 (2π) n/2 R 1/2 e 1 2 z R 1z dz n whch R denotes a (postve defnte) correlaton matrx and Φ 1 s the nverse standard normal dstrbuton functon. Then the random varables Y = Φ 1 (G (X )), = 1,..., n, are jontly dstrbuted as multvarate normal. 10

11 Proof: Denote the jont dstrbuton functon of Y 1,..., Y n by H. Then H(y 1,..., y n ) = P (Y 1 y 1,..., Y n y n ) = P (Φ 1 (G 1 (X 1 )) y 1,..., Φ 1 (G n (X n )) y n ) = P (X 1 G 1 1 (Φ(y 1)),..., X n G 1 n (Φ(y n ))) = C(Φ(y 1 ),..., Φ(y n )) y1 yn 1 = (2π) n/2 R 1/2 e 1 2 z R 1z dz, whch s the dstrbuton functon of a multvarate normal dstrbuton. 11

12 (a) (b) Proporton correct Beta Item Item (c) (d) Dfference Item Theta (e) (f) Eta Lambda Fgure 3: Results for sample 1 (a) proporton correct versus tem number: emprcal (sold lne), theoretcal wth true parameter values (broken lne), theoretcal wth estmated parameter values (broken-dotted lne), (b) estmated tem dffculty parameters under (3)-(1) (sold lne) and 1PL wth guessng (broken lne), (c) dfference between tem dffculty estmates, (d) dstrbuton of θ: theoretcal (sold lne) and ftted (broken lne), (e) dstrbuton of η: theoretcal (sold lne) and ftted (broken lne) and (f) dstrbuton of λ: theoretcal (sold lne) and ftted (broken lne). 12

13 (a) (d) Proporton correct Item Theta (e) (f) Eta Lambda Fgure 4: Results for sample 2 (a) proporton correct versus tem number: emprcal (sold lne), theoretcal wth true parameter values (broken lne), theoretcal wth estmated parameter values (broken-dotted lne), (b) estmated tem dffculty parameters under (3)-(1) (sold lne) and 1PL wth guessng (broken lne), (c) dfference between tem dffculty estmates, (d) dstrbuton of θ: theoretcal (sold lne) and ftted (broken lne), (e) dstrbuton of η: theoretcal (sold lne) and ftted (broken lne) and (f) dstrbuton of λ: theoretcal (sold lne) and ftted (broken lne). 13

14 (a) Item Fgure 5: Results for sample 3 (a) proporton correct versus tem number: emprcal (sold lne), theoretcal wth true parameter values (broken lne), theoretcal wth estmated parameter values (broken-dotted lne), (b) estmated tem dffculty parameters under (3)-(1) (sold lne) and 1PL wth guessng (broken lne), (c) dfference between tem dffculty estmates, (d) dstrbuton of θ: theoretcal (sold lne) and ftted (broken lne), (e) dstrbuton of η: theoretcal (sold lne) and ftted (broken lne) and (f) dstrbuton of λ: theoretcal (sold lne) and ftted (broken lne). 14