EDMS Modern Measurement Theories. Explanatory IRT and Cognitive Diagnosis Models. (Session 9)

Transcription

1 EDMS Modern Measurement Theories Explanatory IRT and Cognitive Diagnosis Models (Session 9) Spring Semester 2008 Department of Measurement, Statistics, and Evaluation (EDMS) University of Maryland Dr. André A. Rupp, (301) , ruppandr@umd.edu

2 The Idea of Explanatory IRT Using a formulation of IRT models as hierarchical generalized linear and non-linear mixed models (e.g., Bryk & Raudenbush, 2002; Kamata, 2001), one gets that: (a) Item parameters are fixed effects (level 1) (b) Persons are random effects (error at level 2) (c) School and other context effects can be modeled at higher levels (d) Predictors of item difficulty can be incorporated (e) Predictors of proficiency differences can be incorporated (f) Broad classes of IRT models can be formulated within this framework EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

3 A Simple Example: Rasch Model vs. LLTM Rasch Model P ij exp( θi β j ) ( θ ) = 1+ exp( θ β ) i j LLTM LLTM with Error P P ij ij ( θ ) ( θ ) exp( θ ( i k = 1 = K 1+ exp( θ ( exp( θ ( 1+ exp( θ ( i K k = 1 i k = 1 = K i K k = 1 w q k k jk w q w q k k jk w q )) jk )) + ε )) jk j + ε )) j

4 Fit Assessment: Rasch vs. LLTM EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

5 The Context for CDA and CDMs

6 Cognitive Diagnostic Assessment Cognitive diagnostic assessment (CDA) is designed to measure specific knowledge structures and processing skills in students so as to provide information about their cognitive strengths and weaknesses. (Leighton & Gierl, 2007)

7 Cognitive Diagnostic Assessment Skills diagnosis, sometimes referred to as skills assessment, skills profiling, profile scoring, or cognitive diagnosis, is an application of psychometric theory and methods to the statistically rigorous process of (a) evaluating each examinee on the basis of level of competence on a user-developed array of skills, and (b) evaluating the skills estimation effectiveness of a test by assessing the strength of the relationship between the individual skills profiles and the observed performance on individual test items.

8 Cognitive Diagnostic Assessment Such examinee evaluation, ideally periodic throughout the teaching and learning process, can provide valuable information to enhance teaching and learning. It is primarily intended for use as formative assessment, in contrast to summative assessment. [ ] Skills diagnosis can often be considered relatively low stakes from the perspective of the individual test taker. [ ] The skills diagnostic score results, at least in principle, are well integrated within the classroom environment and constitute just one complementary element of an extensive store of teacher, peer assessment, and instructional information. (Roussos et al., 2007) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

9 The Use of Diagnostic Information A recent survey in the US indicated that: 60% and 42 % of teachers use results from commercial large-scale assessments and state-mandated large-scale assessments, respectively, not more than a few times a year to inform their instruction. Between 13% and 29% of teachers never use results from state-mandated large-scale assessments to inform their practice and between 32% and 49% of teachers never use results from commercial large-scale assessments to inform their practice. Only between 72% and 78% of teachers believe that these assessments are at least moderately appropriate for collecting diagnostic information while 93% think this about teacher-created classroom assessments. (Huff & Goodman, 2007)

10 What Types of Information are Diagnostic? white area: agree gray area: strongly agree

11 At Which Levels Do You Desire Different Information? white area: agree gray area: strongly agree

12 What Kinds Of Information Do You Desire? white area: agree gray area: strongly agree

13 Describing Response Processes

14 Investigating Response Processes Developing a plausible model for the cognitive response processes is a difficult task that requires (e.g., Leighton, 2004; Leighton & Gierl, 2007; Gorin, 2007): A strong domain-specific cognitive model (a) Tables of specification (cognitively coarse) (b) Curricular specifications (behavioral) (c) Process specifications (cognitively fine-grained) that is validated via qualitative investigations of response processes that utilize (a) Verbal reports (exploratory approach to develop computational models of information processing for rule-based problem-solving tasks) (b) Protocol studies (confirmatory appraoch to validate computational models of information-processing for knowledge-based problemsolving tasks) as well as quantitative investigations of response processes that may draw on (a) response time measurements (b) eye-tracking EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

15 Eye-tracking Research EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

16 Participant A EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

17 Participant B EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

18 Participant C EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

19 Attributes and Attribute Hierarchies

20 Attribute Types EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

21 Q-matrices Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Item Item Item Item Item Item Item Item Item Item

22 Algebra Example EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp) Yan & Mislevy (2003)

23 Attribute Hierarchies EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

24 Attribute Patterns for Algebra Hierarchy Prerequisite Skills: Meaning of Symbols and Conventions 1 Comprehension of Text 2 Tabulation 7 Algebraic Manipulation Plot 3 8 Linear Function 4 Solution of Quadratic Simultaneous Equations 5 6

25 Hierarchy Types Leighton, Gierl, & Hunka (2007)

26 Q-matrix for Linear Hierarchy EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

27 Q-matrix for Convergent Hierarchy Leighton, Gierl, & Hunka (2007)

28 Q-matrix for Divergent Hierarchy EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

29 Q-matrix for Unstructured Hierarchy EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

30 Multivariate Profiles for Reporting

31 CDA in Clinical Diagnosis (Pathological Gambling) Templin & Henson (2006)

32 CDA in Clinical Diagnosis (Pathological Gambling) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

33 CDA in Clinical Diagnosis (Pathological Gambling) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

34 CDA in Clinical Diagnosis (Pathological Gambling) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

35 CDA in the Classroom EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

36 CDA in the Classroom EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp) Jang (2005)

37 A Taxonomy of CDMs

38 Cognitive Diagnosis Models (CDMs) A variety of labels have appeared for these models in the literature including: Cognitive diagnosis models (e.g., Nichols, Chipman, & Brennan, 1995) Cognitive psychometric models (e.g., Rupp, 2007) Latent response models (e.g., Maris, 1995) Restricted latent class models (e.g., Haertel, 1989) Multiple classification (latent class) models (e.g., Maris, 1999) Structured item response theory models (e.g., Rupp & Mislevy, 2007) Similarly, several extensive reviews of such models have appeared in the literature including Junker (1999), Hartz (2002), Roussos (1994), Roussos, dibello, and Stout (2007), Rupp and Templin (2007), as well as Fu and Li (2007).

39 Definition Cognitive diagnosis models (CDMs) are probabilistic confirmatory multidimensional latent-variable models with a complex loading structure. They are suitable for modelling categorical response variables and contain categorical latent predictor variables that generate latent classes to classify learners. They enable multiple criterion-referenced interpretations and feedback for diagnostic purposes that are referenced to a cognitively-grounded theory of response processes at a fine grain size.

40 Definition Cognitive diagnosis models (CDMs) are probabilistic confirmatory multidimensional latent-variable models with a complex loading structure. They are suitable for modelling categorical response variables and contain categorical latent predictor variables that generate latent classes to classify learners. They enable multiple criterion-referenced interpretations and feedback for diagnostic purposes that are referenced to a cognitively-grounded theory of response processes at a fine grain size.

41 The Confirmatory Nature of CDMs Substantive Perspective of Use They are used to confirm a theory-driven hypothesis about a cognitive response process. Statistical Perspective of Model Structure They are psychometric models with restrictions on model parameters. These restrictions are predominantly loadings constrained to 0 in the loading structure or Q-matrix. These translate to constraints on other parameters across latent classes. They are psychometric models whose structure (i.e., number, type, and interaction of latent variables) represents a mapping of the substantive hypothesis onto the model space.

42 Definition Cognitive diagnosis models (CDMs) are probabilistic confirmatory multidimensional latent-variable models with a complex loading structure. They are suitable for modelling categorical response variables and contain categorical latent predictor variables that generate latent classes to classify learners. They enable multiple criterion-referenced interpretations and feedback for diagnostic purposes that are referenced to a cognitively-grounded theory of response processes at a fine grain size.

43 Multidimensional Factor-analytic Models ATT 1 ATT 2 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp) (simple structure)

44 Multidimensional Factor-analytic Models ATT 1 ATT 2 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 (complex structure)

45 Definition Cognitive diagnosis models (CDMs) are probabilistic confirmatory multidimensional latent-variable models with a complex loading structure. They are suitable for modelling categorical response variables and contain categorical latent predictor variables that generate latent classes to classify learners. They enable multiple criterion-referenced interpretations and feedback for diagnostic purposes that are referenced to a cognitively-grounded theory of response processes at a fine grain size.

46 Multidimensional IRT Models ATT 1 ATT 2 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp) (simple structure)

47 Multidimensional IRT Models ATT 1 ATT 2 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 (complex structure)

48 Multidimensional IRT Compensatory Multidimensional 3-parameter IRT Model P ij ( θ) = γ j + K exp( k (1 γ j ) 1+ exp( = 1 K α k = 1 jk α θ jk ik θ + δ ) ik j + δ ) j Non-compensatory Multidimensional 3-parameter IRT Model P ij ( θ) = γ j + (1 γ ) j K k 1 exp( α jk 1+ exp( α ( θ ik ( θ β = jk ik jk β )) jk ))

49 Definition Cognitive diagnosis models (CDMs) are probabilistic confirmatory multidimensional latent-variable models with a complex loading structure. They are suitable for modelling categorical response variables and contain categorical latent predictor variables that generate latent classes to classify learners. They enable multiple criterion-referenced interpretations and feedback for diagnostic purposes that are referenced to a cognitively-grounded theory of response processes at a fine grain size.

50 Cognitive Diagnosis Models ATT 1 ATT 2 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9

51 Loading Structure (Q-matrix) Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Item Item Item Item Item Item Item Item Item Item

52 Combinations of Latent Predictor Variables While common FA and IRT models are compensatory in nature, many CDMs combine their latent variables in a non-compensatory fashion. This distinction is related to the notions of conjunctive and disjunctive condensation functions (e.g., Maris, 1992, 1995, 1999): ξ ij = K k= 1 α q ik jk ξ ij K = 1 (1 α k = 1 ik ) q jk ξ ij = K k = 1 α q ik jk compensatory compensatory / disjunctive non-compensatory / conjunctive Of course, CDMs with interactions between latent variables can be conceived of similar to such models in IRT (e.g., Jannarone, 1997) but these cannot necessarily be estimated.

53 Higher-order Structures in CDMs ATT 1 ATT 2 ATT 3 ATT 4 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 I 10 I 11 I 12 I 13 CDM with two levels of latent variables (see de la Torre & Douglas, 2004)

54 A Taxonomy of CDMs (Part I) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

55 A Taxonomy of CDMs (Part II) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

56 Software programs for CDMs Software Buglib AHM DINA in Ox CDM in R CDM Arpeggio MDLTM StatShop WINBUGS Type of Software (Contact) Research License (tatsuoka@prodigy.net) Research License (mark.gierl@ualberta.edu) Freeware (j.delatorre@rutgers.edu) Freeware (alexander.robitzsch@iqb.hu-berlin.de) Freeware for Mplus (jtemplin@uga.edu) Research License (dlembeck@ets.org) Research Licencse (dlembeck@ets.org) Research License (ralmond@ets.org) Freeware (available online) Estimated Models RSM AHM DINA, HO-DINA, MS-DINA DINA, NIDA, DINO DINA, NIDA, DINO, NIDO, Reduced NC-RUM, C-RUM Full NC-RUM, Reduced NC-RUM GDM BIN BIN

57 An Overview of Selected CDMs

58 Rule-space Methodology Step 1 Calibrate data with (unidimensional) IRT model Step 2 Compute atypicality index Step 3 Classify learners in rule-space = = + = + = = J j j j j J j j j T P P X T P 1 1 )) ( ) ( )( ( )) ( ) ( ( )] ( ) ( ), ( [ )] ( ) (, [ )] ( ) (, ) ( [ ) ( θ θ θ θ θ θ θ θ θ θ θ θ θ ζ T P P T X P T X P P X

59 Rule-space Methodology EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

60 CDMs and Latent Class Models CDMs can be viewed as restricted latent class models (e.g., Haertel, 1989): C ij P( X = x ) = η π (1 π i i c= 1 c J j= 1 x jc jc ) 1 x ij where C c= 1 η c = 1 CDMs provide different parameterizations of the response probabilities.

61 DINA model π jc = (1 s j ) ξ jc g (1 ξ j jc ) s g j = Y jc jc P( = 0ξ = 1) j = Y jc jc P( = 1ξ = 0) ξ jc = K k = 1 α q ck jk

62 Parameters (DINA model) Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Parameter Item Item Item Item Item Item Item Item Item s s 1 g 1 s 2 g 2 s 3 g 3 s 4 g 4 s 5 g 5 s 6 g 6 s 7 g 7 s 8 g 8 g 9 Item s 10 g 10 Parameter

63 Equivalence Classes in the DINA Model EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

64 Command File (CDM in Mplus) *Qmatrix file = qmatrix.txt attributes = 4 format = 4i1 *DataFile file = data.txt items = 15 format = 15i1 *CDM model = DINA startmethod = subscore *Program mpluspath = C:\Programme\Mplus\Mplus.exe

65 Input Files (CDM in Mplus) qmatrix.txt data.txt EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

66 Output Files (CDM in Mplus) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

67 Output Files (CDM in Mplus) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

68 NIDA Model π jc K ij ck ck jk = k k k = 1 x [ ] q s g [ s g ] α 1 α α 1 α (1 ) 1 (1 ) K k = 1 k ck k ck q jk 1 x ij s k g k = jck ck jk P( ζ = 0α = 1, q = 1) = jck ck jk P( ζ = 1α = 0, q = 1)

69 Parameters (NIDA model) Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Parameter Item Item Item Item Item Item Item Item Item Item Parameter s 1 g 1 s 2 g 2 s 3 g 3 s 4 g 4 s 5 g 5 s 6 g 6

70 Equivalence Classes in the NIDA Model EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

71 The RUM / Fusion model π jc = π * j K k = 1 r *(1 α jk ck ) q jk exp 1+ exp ( ) τ + β c j ( ) τ + β c j π * j = K k = 1 π q jk jk π jk = P( ξ jck = 1 αck = 1) = (1 s jk ) r * jk = r π jk jk r = P( ζ = 1 α = 0) = jk jck ck g jk

72 Parameters (RUM / Fusion Model) Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Parameter Item r * π * 1 Item r * 24 r * 25 0 π * 2 Item 3 r * r * 34 0 r * 36 π * 3 Item 4 r * 41 r * 42 r * 43 r * π * 4 Item 5 r * r * 55 π * 5 Item 6 0 r * 62 0 r * 64 r * 65 Item 7 r * 71 r * 72 r * π * 6 π * 7 Item 8 0 r * 82 0 r * 84 0 r * 86 π * 8 Item 9 0 r * 92 0 r * 94 r * 95 r * 96 π * 9 Item r * 10,6 π * 10 Parameter

73 Equivalence Classes in the RUM / Fusion Model EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

74 Command File (RUM, Arpeggio) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

75 Command File (RUM, Arpeggio) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

76 Input Files (RUM, Arpeggio) Booklet 2 Reduced Q-matrix (5 Attributes) 20 5 ItemId,NumSkills,S1SG,S2VL,S3VK,S4MO,S5GK, D079012,2,0,2,0,0,5, D079022,3,1,0,3,4,0, D079032,2,0,2,0,0,5, D079042,2,1,0,3,0,0, D079052,2,1,0,0,0,5, D079062,1,0,0,3,0,0, D079072,1,0,0,0,0,5, D079082,1,0,0,0,0,5, D079092,3,1,2,0,4,0, D079102,0,0,0,0,0,0, D079112,1,0,0,3,0,0, D079122,3,0,2,3,0,5, D079132,1,0,2,0,0,0, D079142,2,0,0,3,0,5, D079152,2,1,0,0,4,0, D079162,2,0,0,3,0,5, D079172,1,0,0,3,0,0, D079182,2,0,2,0,4,0, D079192,2,0,0,3,0,5, D079202,3,0,2,3,0,5, qmatrix.in orthodata2.in EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

77 General diagnostic model π xjc = 1+ exp β xj + M j exp β K k = 1 + x ij K γ mj m= 1 k = 1 jk α mγ ck jk q α jk ck q jk von Davier (2005), Xu & von Davier (2006); see von Davier (2007)

78 Parameters (GDM) Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Attribute 6 Parameter Item γ β 1 Item γ 24 γ 25 0 β 2 Item 3 γ γ 34 0 γ 36 β 3 Item 4 γ 41 γ 42 γ 43 γ β 4 Item 5 γ γ 55 β 5 Item 6 0 γ 62 0 γ 64 γ 65 Item 7 γ 71 γ 72 γ β 6 β 7 Item 8 0 γ 82 0 γ 84 0 γ 86 β 8 Item 9 0 γ 92 0 γ 94 γ 95 γ 96 β 9 Item γ 10,6 β 10 Parameter

79 A Comparison of Compensatory CDMs NIDO model π jc = exp xij 1 + exp K k = 1 K k = 1 ( β + γ α ) k ( β + γ α ) q k k k ik ik q jk jk Compensatory RUM π jc = exp xij β j exp β j + K k = 1 K k = 1 γ γ jk α q jk ik α q ik jk jk General diagnostic model π jc = exp β xj + M j 1+ exp β K k = 1 + x K mj m= 1 k = 1 ij γ jk α q mγ ik jk jk ik α q jk

80 Command File (MDLTM) EDMS 724 Modern Measurement Theories (Spring 2008, Dr. André A. Rupp)

81 Open Questions for Future Research

82 Structuring Research into CDAs and CDMs Guide to investigating the conceptual foundations and statistical properties of CDMs (Rupp, 2007)

83 Open Questions for CDA and CDMs 1. What kinds of constructs can be measured with cognitive models and used to guide our diagnostic inferences? How can these constructs be identified and developed for CDA across grade levels and content areas? How do we develop instructional strategies to remediate deficiencies for CDA-based instruction? 2. Will principled test desing practices become more common in educational and psychological testing, particularly when cognitive models have such an important role in this approach? 3. Will test users continue to demand CDAs when they realize that these assessments yield less construct representation and content coverage in order to get the benefits associated with penetrating deeply into the student s understanding of the construct? 4. How can we communicate sophisticated, psychologically based assessment information in a timely manner to a diverse audience of test users? 5. Can our current CDA models and methods truly satisfy [the cognitive diagnosis] characteristics in formative classroom assessment that are deemed to be so important to teachers? Can our future CDA models and methods ever hope to meet the requirements of formative classroom assessments? (Leighton & Gierl, 2007)

84 A Final Word CDAs are different from more traditional forms of assessment, but need to be put constantly under the cognitive microscope by pursuing a strong program of validity. This could yield far more information about why students succeed and fail during learning, as well as what students do and do not understand during instruction, than has ever been possible. (Leighton & Gierl, 2007)