A Note on a Tucker-Lewis-Index for Item Response Theory Models. Taehun Lee, Li Cai University of California, Los Angeles

Transcription

1 A Note on a Tucker-Lewis-Index for Item Response Theory Models Taehun Lee, Li Cai University of California, Los Angeles 1

2 Item Response Theory (IRT) Models IRT is a family of statistical models used to analyze item response pattern data. The frequencies of each of the item response patterns are summarized in a multi-way contingency table. An IRT model imposes restrictions on the probabilities of each of the possible response patterns of n items. In other words, the cell probabilities in the multi-way contingency table are expressed as a function of the item parameters. 2

3 Model-Fit Testing for IRT Models The most widely available statistics to assess the goodness-of-fit of the model are the likelihood ratio test statistic G 2 or Pearson s X 2 statistic. When the proposed IRT model holds in the population, both statistics are asymptotically equivalent, and asymptotically chi-square distributed. 3

4 Model-Fit Testing for IRT Models Statistical issues of G 2 and Pearson s X 2 statistics. When the sample size N is small and the number of items large, then the table of frequencies for the response patterns become so sparse that the asymptotic results do not give a good approximation to the distribution of the test statistics. Practical issues of G 2 and Pearson s X 2 statistics The LR chi-square test of a given model represents an implicit model comparison against the saturated model. The given model is rejected on its inability to account for every bit of data. Practical significance of the remaining increment in fit may be only very small. 4

5 Model-Fit Testing for IRT Models Similar problems of the LR chi-square tests have been recognized in the maximum likelihood exploratory factor analysis (Tucker and Lewis, 1973). The factor analytic model with a scientifically desirable number of common factors would not exactly represent data for a population of objects [p.1]. This proposition raises questions as to the use of the LR test statistics. Tucker-Lewis-Index (TLI) was proposed to evaluate whether the improvement in fit of the k-factor model over the zero-factor model can be practically significant. 5

6 TLI in IRT models To evaluate an IRT model with respect to its improvement in fit over the baseline model, we need to resolve the following two issues. Issues 1) What is the approppriate baseline-model in IRT? 2) Is the Pearson X 2 or G 2 statistic appropriate for? 6

7 The M 2 Statistics (Maydeu-Olivares and Joe, 2005) The Pearson X 2 and G 2 statistics can be misleading or unavailable due to the sparseness of the data. As an improvement for handling sparseness in the multidimensional contingency table, classes of quadratic form statistics, termed Mr, are proposed based on multivariate moments up to order r (Maydeu-Olivares and Joe, 2005,2006). The Mr Statistics are shown to have better small-sample properties and are asymptotically more powerful than the existing X 2 and G 2 statistics. Expected frequencies for the first- and second-order margins are rarely small. Therefore, we can employ M 2 Statistics for the chi-square variate in computing TLI for IRT models. 7

8 Baseline model in the context of IRT The zero-factor model can serve as the baseline model in IRT. 2PL-IRT models Graded Response Models (GRM) Nominal IRT models 8

9 Baseline model in the context of IRT The zero-factor model can serve as the baseline model in IRT. 2PL-IRT models Graded Response Models (GRM) Nominal IRT models 9

10 TLI in IRT models To evaluate an IRT model with respect to its improvement in fit over the baseline model, we need to resolve the following two issues. Issues 1) A baseline-model in IRT? Zero-factor model 2) An appropriate statistics for? M 2 statistics 10

11 Examples Quality of Life (Lehman, 1988) N=586; n=35 (dichotomized) Seven sub-domains (Family, Finance, Health, Leisure, Living, Safety, and Social) Model M 2 df p TLI 0 Zero- factor PL Bi- factor The chi-square tests reject both 2PL and bi-factor models. TLI shows that the improvement in fit of the bi-factor model can be practically significant. 11

12 Is the zero-factor model acceptable? Observed- and Residual 1 st order margins (Quality of Life (Lehman, 1988)) Observed Zero- factor 2PL bi- factor Item Item Item Item Item Item The zero-factor baseline model fits better than the 2PL- and bi-factor models insofar as the 1 st -order marginal is concerned. 12

13 Is the zero-factor model acceptable? The zero-factor model fits the1 st -order margins perfectly, and fits better than any of the substantive IRT models in the 1 st -order margins. It is incontrovertible that an appropriate baseline model be a most restrictive model that is nested within the sequence of substantive IRT models (Widaman & Thompson, 2003). A unidimensional IRT model with equality constraint imposed on slopes can serve as a baseline model in IRT (Single-slope model, hereafter). 13

14 Examples Quality of Life (Lehman, 1988) Model M 2 df p TLI 0 TLI 1 Zero- factor Single- slp PL Bifactor The chi- square tests reject the bifactor model. TLI shows the improvement in fit of the bi- factor model can be praclcally significant. The use of the Single- slope baseline model makes it clear that the improvement in fit comes mostly from modeling the bifactor structure of the scale. 14

15 Examples PISA mathemalcs (2000); N=358; n=14; 5 Testlets. Model M 2 df p TLI 0 TLI 1 Zero- factor Single- slp GRM Testlet The chi- square test does not reject the testlet model. TLI shows the improvement in fit of the testlet model can be praclcally significant. The use of the Single- slope baseline model makes it clear that the improvement in fit comes mostly from modeling the testlet structure of the scale. 15

16 Summary and Discussion We propose the Tucker-Lewis-Index in the context of IRT modeling (TLIRT) by using The M 2 statistics as the chi-square variate for computing TLIRT The Single-slope model, instead of the zero-factor model, as the baseline model (further research is required, however). The real data were used for the illustrations of the calculation and the utility of the TLIRT. Further research is required to develop a way to conduct formal inferences or acceptable cut-off value(s) for the TLIRT. 16

17 Acknowledgement This research is supported by grants from the Institute of Education Sciences (R305B and R305D100039) and the National Institute on Drug Abuse (R01DA and R01DA030466). 17