A Note on Item Restscore Association in Rasch Models

Transcription

1 Brief Report A Note on Item Restscore Association in Rasch Models Applied Psychological Measurement 35(7) ª The Author(s) 2011 Reprints and permission: sagepub.com/journalspermissions.nav DOI: / Svend Kreiner 1 Keywords Rasch model, item analysis, IRT model fit To rule out the need for a two-parameter item response theory (IRT) model during item analysis by Rasch models, it is important to check the Rasch model s assumption that all items have the same item discrimination. Biserial and polyserial correlation coefficients measuring the association between items and restscores are often used in an informal way for this purpose, with the expectation that correlations will be stronger for items with a high degree of item discrimination and weaker for items where item discrimination is low. Assessment of these coefficients is subjective and often difficult because Rasch models are nonlinear models that do not require biserial and polyserial correlations to be constant across items. To make assessment easier and less uncertain, we propose an alternative way to measure the association between item scores and restscores under the Rasch model and show how to assess the significance of the difference between observed and expected associations. Let Y i be an item score and S = Pk Y i the total score on all items. The restscore without Y i is R i i=1 ¼ S Y i.asy i and R i are discrete ordinal variables, we measure the marginal association between Y i and R i by Goodman and Kruskal s (1954) g. The distribution of g depends on the marginal probabilities p yr ¼ P(Y i ¼ y, R i ¼ r). The sufficiency of S in the Rasch model implies that p yr is given by p yr = Ð P(Y i = y, R i) = rjs = r + y)p(s = r + yjy)f (y) dy = P(Y i = y, R i = rjs = r + y) R P(S = r + yjy)f (y) dy = P(Y i = y, R i = rjs = r + y)p(s = r + y), ð1þ where f(y) is the density of y. In Formula 1, P(Y i ¼ y S ¼ s) depends on the item parameters but not on y because S is sufficient (see Andersen, 1995, Formulas and for details on this distribution). The distribution of S depends on the distribution of y, but assumptions concerning f(y) are not 1 University of Copenhagen, Denmark Corresponding Author: Svend Kreiner, Department of Biostatistics, University of Copenhagen, Oster Farimagsgade 5, B, P.O. Box 2099, Copenhagen K, Denmark s.kreiner@biostat.ku.dk

2 558 Applied Psychological Measurement 35(7) Table 1. Observed and Expected Association Among Responses to Item 3 and the Restscore Summarizing Responses to Other Items Restscore Walking outside in poor weather Total (a) Observed Cannot do it without getting tired Can do it without getting tired Total (b) Expected Cannot do it without getting tired Can do it without getting tired Total Note: Observed g ¼ 0.994; Expected g ¼ 0.948; SE ¼ required because P(S ¼ s) can be consistently estimated by data. From this and Formula 1 it follows that consistent estimates of p yr under the Rasch model are easily obtained as a function of estimates of item parameters and estimates of P(S ¼ r + y). Goodman and Kruskal s g is similar to Kendall s rank correlation. Let n yr be the number of persons with Y i = y and R i = r, andletc and D be the number of concordant and discordant pairs defined by pairwise comparison of persons outcomes on Y i and R i : C = P i < k, j < l n ijn kl and D = P i < k, j > l n ijn kl,(i,k) {0,1,..., max(y i )} {0,1,..., max(y i )} and (j,l) {0,1,..., max (R i )} {0,1,...,max(R i )}. The g coefficient is g = ðc DÞ ðc + DÞ : ð2þ The g coefficient is an estimate of the difference between two conditional probabilities: the conditional probability of concordance given either concordance or discordance and the conditional probability of discordance. To calculate the expected g, we first calculate the probabilities that pairwise comparison of persons results in either concordance or discordance, P C = 2 P i < k, j < l p ijp kl, P D = 2 P i < k, j > l p ijp kl, and then calculate E(g) = ð P C P D Þ ðp C + P D Þ : ð3þ It is well known that g is asymptotically normal with a standard error that is easy to calculate (see Agresti, 1984, Appendix C.3 for details). For this reason, it is straightforward to test whether ^g departs from Eð^g Þ. An Example Avlund, Kreiner, and Schultz-Larsen (1993) fitted a Rasch model to six dichotomous items measuring mobility in a population of 70-year-old Danes. There was no evidence of differential item functioning (DIF), and the conditional likelihood ratio test by Andersen (1973) comfortably accepted the model. Table 1 shows the observed and expected association between the third item ( Walking outside in poor weather ) and the restscore, summarizing responses to the other five items. The observed g is significantly larger than the expected (observed g ¼ 0.994, expected g ¼ 0.948, p <.00005).

3 Kreiner 559 Table 2. Assessment of Item Restscore Association for Six Mobility Items Items Conditional ML estimate of the item location Observed item restscore g Expected item restscore g SE p Walking indoors Walking outdoors in nice weather Walking outdoors in poor weather Managing stairs Getting outdoors Transfer from chair or bed to standing position Note: ML = maximum likelihood. Item responses were coded 0 ¼ cannot do it or cannot do it without getting tired,1 ¼ can do it. Benjamini and Hochberg (1995) accepts hypotheses with p values above.025 to control the false discovery rate at 5% of the multiple tests. The observed and expected g coefficients for all items are summarized in Table 2. The item restscore association was significantly stronger than expected by the Rasch model for two items and significantly weaker for two items. The fit of the Rasch model to item responses is therefore less than adequate. Large Sparse Tables The expected table of item restscore counts (Table 1) indicates a difficulty that will turn into a problem as the number of items and the number of response categories increase. The problem is that some of the cells of the item restscore table will have many cells with very small expected counts resulting in a large sparse table of observed counts where it is known that the asymptotic distributions of test statistics provide very bad approximations to exact distributions. For dichotomous items, a solution to this problem is provided by Besag and Clifford s (1989) algorithm for Markov chain Monte Carlo tests in the conditional distribution of test statistics given person scores and item margins. Another option is parametric bootstrapping from the conditional distribution of the item score and the restscore given S, which works for both dichotomous and polytomous items. Item Restscore Association in Log-Linear Rasch Models One convenient feature of the g coefficient as a measure of item restscore association in Rasch models is that it generalizes to loglinear Rasch models, where local dependence (LD) and DIF are permitted and where LD and DIF are uniform in the sense that they are defined by loglinear interaction parameters that do not depend on y (Kelderman, 1984; Kreiner & Christensen, 2002). In these models, the total score is also sufficient for y, and the derivation of the distribution of the g coefficients proceeds in the same way as under the Rasch models. Consider, for instance, a model where Y i and Y j are uniformly locally dependent. To derive the distribution of (Y i, R i ), we first find the marginal distribution of (Y i,y j, R ij ) where R ij ¼ S Y i Y j. This distribution is given by P(Y i ¼ y i, Y j ¼ y j, R ij ¼ r S ¼ r + y i + y j ) P(S ¼ r + y i + y j ), where the conditional distribution depends on item parameters and

4 560 Applied Psychological Measurement 35(7) Table 3. Reassessment of Item Restscore Association Under (a) A Log-Linear Rasch Model With LD and (b) After Elimination of the Third Item Log-linear Rasch model Rasch model without Item 3 Items Obs. g Exp. g p Obs. g Exp. g p Walking indoors Walking outdoors in nice weather Walking outdoors in poor weather Managing stairs Getting outdoors Transfer from chair or bed to standing position Note: LD ¼ local dependence; Obs. ¼ observed; Exp. ¼ expected. P(S ¼ s) can be estimated from data. Marginalizing over Y j finally gives P(Y i ¼ y i, R i ¼ r) ¼ y P(Y i ¼ y i,y j ¼ y, R ij ¼ r). Similar arguments apply for models with uniform DIF except that the marginal distribution of S has to be replaced by the joint distribution of S and the covariates causing DIF among some items. The Example Revisited The evidence against the Rasch model in Table 2 suggests that a two-parameter model is appropriate. In this example, this may not be warranted. Recall that positive local response dependence will cause some item characteristic curves to be steeper than expected by the Rasch model. In the mobility example, local response dependence among the second and third item ( Walking outside in good weather and Walking outside in poor weather ) is to be expected. There are several ways to address the issue of LD. One is to fit a log-linear Rasch model with an interaction between the two items. Another is purification by eliminating one of the two items. In this example, it is natural to eliminate the second of the two dependent items. Table 3 reports the item restscore associations for both models. The fit between observed and expected g coefficients is perfect for both models. Concluding Remarks The main purpose of an analysis of the item restscore association in Rasch models is to assess the fit of items to the model. When misfit is disclosed, there are at least two ways to proceed: by purification or by modeling. Purification is a stepwise procedure that eliminates items until only items that fit the Rasch model remains. If the number of items is large, this is often the recommended strategy. If the number of items is small and if an analysis of the content of items does not disagree with the notion of LD, it may be preferable to model departures from the Rasch model to save items. The example in this note illustrates such a case. Computer Programs The analysis of item restscore association in Rasch models and log-linear Rasch models was implemented in DIGRAM (Kreiner, 2003).

5 Kreiner 561 Declaration of Conflicting Interests The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding The author received no financial support for the research, authorship, and/or publication of this article. References Agresti, A. (1984). Analysis of ordinal categorical data. New York, NY: Wiley. Andersen, E. B. (1973). A goodness of fit test for the Rasch model. Psychometrika, 38, Andersen, E. B. (1995). Polytomous Rasch models and their estimation. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments and applications (pp ). New York, NY: Springer. Avlund, K., Kreiner, S., & Schultz-Larsen, K. (1993). Construct validation and the Rasch model: Functional ability of healthy elderly people. Scandinavian Journal of Social Medicine, 21, Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical, 57, Besag, J., & Clifford, P. (1989). Generalized Monte Carlo Significance Tests. Biometrika, 76, Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, Kelderman, H. (1984). Loglinear Rasch model tests. Psychometrika, 49, Kreiner, S. (2003). Introduction to DIGRAM and SCD (Research report 03/10). Department of Biostatistics, University of Copenhagen, Denmark. Kreiner, S., & Christensen, K. B. (2002). Graphical Rasch models. In M. Mesbah, F.C. Cole, & M. T. Lee (2002). Statistical methods for quality of life studies (pp ). Dordrecht, Netherlands: Kluwer Academic.