Qiong (Joan) Wu Harvard Center for Population and Development Studies. INDEPTH-SAGE WORKSHOP April 20, 2010

Transcription

1 Qong Joan Wu Harvard Center for Populaton and Development Studes INDEPTH-SAGE WORKSHOP Aprl 20,

2 IRT vs Classcal test theory CTT CTT: focuses test scores observed score = true score + error O=T+E IRT: focuses on ndvdual tem characterstcs IRT s a scalng method Assgns numercal scores based on a set of tem responses IRT s an tem analyss tool Evaluates qualty of ndvdual tems based on estmated tem parameters 2

3 Also called latent trat models Many varables n health assessments cannot be measured drectly, such as physcal functonng ablty, fatgue, depresson etc. Walkng more than 1 mle Clmbng flghts of stars Physcal functonng ablty Extendng arms above shoulder Stoopng/kneelng Pckng up a con 3

4 Raw total/raw percentage Items weghted equally Item dependent Factor scores Usually assume contnuous observed varables 4

5 Desgned for dchotomous or polytomous tems Pattern scorng nstead of number scorng When the assumptons are met Scores are tem nvarant Consdered equal-nterval, so preferred scores for longtudnal analyss 5

6 Person parameter Latent scores, theta scores mean=0, SD=1 Item parameters Item dffculty/locaton b, usually -3<b<3 Item dscrmnaton a, >=0 Pseudo-guessng c, 0<=c<=1 Item dffculty/locaton and theta scores on the same scale 6

7 One-parameter IRT Rasch model Two-parameter IRT Three-parameter IRT b P P = 1 log b b e e P + = 1 1 b a b a e e P + = 1 log b a P P = 1 1 b a b a e e c c P + + = 1 log b a P c P = ablty score tem dffculty tem dscrmnaton 7

8 Durng the past week, dd you ever feel sad? Probablty of a yes response P=0.5 a = 2 b = 0 Theta scores 8

9 Easer tem More dffcult tem 9

10 10

11 Deals wth tems wth more than 2 response optons Model probablty of each response opton condtonal on latent trat 11

12 Parameters Latent trat: theta Dscrmnaton parameters Threshold parameters Dchotomous IRT Theta Dscrmnaton par Dffculty/locaton par P X = k = 1 1+ exp α δ Measurement questons x 1+ 1 exp α δ x1 How many categores are needed? Are response categores really dfferent one another? 12

13 1: rarely 2: sometmes 3: most or all of the tme How often do you feel sad? 13

14 1: rarely or none of the tme; 2: some or a lttle of the tme 3: occassonally or a moderate amount; 4: most or all of the tme 14

15 Undmensonalty Local ndependence resdual covarance=0 Undmensonalty mples LI, not vce versa. Physcal functon ng ablty Walkng more than 1 mle Walkng more than 1 block Bathng/dressng oneself Dong laundry Feedng self 15

16 Test constructon and tem analyss Computerzed adaptve testng CAT and tem bankng Test equatng/lnkng Dfferental tem functonng/tem bas 16

17 Basc rule Select tems wth approprate b & hgh a for dchotomous tems Check ICCs for polytomous tems 17

18 Basc dea Items talored to ndvduals trat levels Why do we need t? Reduce test length, mnmze fatgue Mnmze floor/celng effects Challenges Large tem pool Large sample of subjects for ntal tem calbraton 18

19 Dfferent types of adaptve testng Two-stage testng 19

20 Dfferent types of adaptve testng Mult-stage testng Dffculty ncreases 20

21 Defnton Settng a common metrc for scores from tests composed of dfferent sets of tems When do we need t? When two groups of subjects take dfferent test forms How do we do t? Needs a vald lnk, ether through common-tem desgn, or common subject sample desgn 21

22 Lnkng test scores from two test forms common tem desgn Group A common tems unque tems Group B unque tems common tems Fxng tem parameters to be equal across two forms 22

23 Defnton dfferent groups of subjects dsplay dfferent probabltes of endorsng a response opton condtonal on latent trat Why do we care? Test/survey bas when those tems are ncluded Anchor tems Use anchor tems to equate the groups on the latent trat. IRT often use rest tems 23

24 Do you have any dffcultes wth makng phone calls? Functonal ablty 24

25 Hgh correlaton between number rght scores and IRT scaled scores IRT estmates can be sample dependent DIF Large-sample technque >200 for 1PL models; >400 for 2PL models; >600 for 3PL models 25

26 Computerzed adaptve testng and test constructon A large tem pool AND A large number of subjects Test scorng when tem dscrmnaton vary a lot Cross-cultural comparson Longtudnal analyss 26

27 Multdmensonal IRT Specalzed software program BILOG, MULTILOG, Mplus R module 27