Quasi-Exact Tests for the dichotomous Rasch Model

Transcription

1 Introduction Rasch model Properties Quasi-Exact Tests for the dichotomous Rasch Model Y unidimensionality/homogenous items Y conditional independence (local independence) Ingrid Koller, Vienna University Reinhold Hatzinger, WU-Vienna Y specific objectivity/sample independence Y strictly monotone increasing item characteristic function Y sufficient statistics Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 1 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 2 Introduction Quasi-exact tests Why quasi-exact tests? Quasi-exact tests? Parametric methods need large samples because of Y consistency and unbiasedness of parameter estimates Y assumption of asymptotic distribution of test statistics Y higher power of test-statistics mall samples in practise Y large samples often not available (e.g., clinical studies) Y complex study designs (e.g., experiments) Y smaller costs and less time-consuming Y possibility to test the quality of items also in small samples (e.g., stepwise test-construction) with quasi-exact tests it is possible to test the Rasch-model (RM) also with small samples ampling binary matrices description of MCMC method: Kathrin Gruber development of test-statistics (T) for the dichotomous RM Y Ponocny (1996, 2001) Y Chen & mall (2005) Y Verhelst (2008) Y Koller & Hatzinger (in prep.) Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 3 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 4

2 Procedure General procedure for the T-statistics Y A 0 is the observed matrix with the margins r v and c i where r v P i x vi (person score) and c i P v x vi (item score) Y Σ rc is the set of all matrices with fixed r and c (sample space) Algorithm Y sample,..., matrices A s from Σ rc Y calculate T 0 for the observed matrix A 0 Y calulate T 1,..., T for all sampled matrices A 1,..., A Y determine your p-value by p Q t s ~ where t s 1, T sˆa s C T 0ˆA 0 0, else T 11 : large inter-item correlations T 11ˆA Qr Çr where Çr P r r... the inter-item-correlation for item i and item j Çr... mean of r from all simulated matrices 1, T p Q t s ~ where t s sˆa s C T 0ˆA 0 0, else if r in A 0 is large, then the difference r Çr is also large only a few T s show the same or a higher difference than T 0 highly correlated items indicate violation of conditional independence Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 5 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 6 & T 11m : small inter-item-correlations same equation as for T 11, but modified test: 1, T p Q t s ~ where t s sˆa s B T 0ˆA 0 0, else if r in A 0 is small, then the difference r Çr is also small only a few T s show the same or a smaller difference than T 0 small correlations between items indicate multidimensionality T 1 : many equal responses Y count the number of 00 and 11 patterns in items i and j Y how many T s have same or a higher value than T 0 1, x T 1ˆA Q δ where δ vi x vj v 0, x vi x x vj many equal responses indicate violation of conditional independence : T 1m : few equal responses Y how many T s have same or a lower value than T 0 few equal responses indicate that the correlation between items is too small, unidimensionality assumption may be violated Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 7 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 8

3 & Learning & T 1l : many 11 patterns (e.g.,koller & Hatzinger) Y count only 11 patterns as opposed to T 1 T 1lˆA Q v δ where δ 1, x vi x vj 1 0 else Y how many T s have same or a higher value than T 0 if person has learned from one item (x vi 1) then the probability pˆx vj 1 is increased for a positive reponse to another item j T 2 : high dispersion of rawscore r v for a set of items Y if items are dependent, the variance of r v is large Y because of varˆz varˆx varˆy 2 covˆx, y Y define a set of items I and calculate rˆi v Y count how many T s CT 0 T 2ˆA var vˆrˆi v where rˆi v Q x vi i>i other possibilities: range, mean absolute deviation, median absolute deviation. : T 2m : low dispersion of rawscore r v for a set of items Y count how many T s BT 0 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 9 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 10 ubgroup-invariance ubgroup-invariance T MU : correlation of rawscore for item subsets (Koller & Hatzinger) Y if two sets of items I are unidimensional, r I v of set I and r J v of set J should be positiv correlated Y with increasing r I v also r J v should be increasing Y count the number of correlations T s BT 0 T MUˆA corˆr I v, r J v r I v Q i>i x vi T 10 : based on counts on certain item responses Y n ~n ji is proportional to the ratio of expˆβ i ~ expˆβ j Y no parameter differences for focal-group f oc and referencegroup ref: n ref ~n ref ji n foc ~nfoc ji Y sum of differences for all pairs of items Y counts of T s CT 0 T 10ˆA Q n ref n foc ji n ref ji n foc Y if the parameter differ across groups, the difference should be increasing Note: Y external criterion (e.g., gender): uniform DIF Y internal criterion (e.g., rawscore-median): discrimination, guessing, falsity Y split on specified item: conditional dependence Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 11 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 12

4 ubgroup-invariance ubgroup-invariance ubgroup-invariance ubgroup-invariance T 4 :counts of positive responses in person subgroups Y assumption: in one group of persons G one or more items are easier/more difficult as expected in the RM Y count the number of persons who solved these items Y easier: counts of T s CT 0 Y more difficult:counts of T s BT 0 T 4ˆA Q x vi v>g Note: Y tests the same assumptions as T 10 T DT F : based on item differences on item sumscores (Koller & Hatzinger) Y similar to T 4, but with the possibility to test DTF (all items in a test shows subgroup-invariance) Y calculate the sumscores (c) for one item (or a group of items) for the reference group c ref i and for the focal group c foc i Y calculate the difference of c between focal and reference group. Y easier: counts of T s CT 0 Y more difficult:counts of T s BT 0 T DT F ˆA Qˆc ref i c foc i 2 i>i Note: Y tests the same assumptions as T 10 & T 4 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 13 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 14 Unfolding response structure Unfolding response structure - monotonicity Item discrimination Item discrimination T 6 : responses in three person subgroups Y similar to T 4. Y split the sample in three rawscore groups and count the number of positive responses only in the middle group G m T 5 : rawscore for persons with x vi 0 for a certain item Y include persons who answer with 0 to a certain item Y sum r v of the remaining items for group x vi 0 Y counts of T s CT 0 Y easier (reversed U-shape): counts of T s CT 0 Y more difficult (U-shape): counts of T s BT 0 T 6ˆA Q v>g m x vi T 5 ˆA Q r v vx vi 0 Y if persons with high ability (r v high) fail to solve a certain item, this item may show too low discrimination, falsity, or indicate multidimensionality Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 15 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 16

5 New test statistics Constructing new test statistics Monotone transformation Example: point-biserial correlation Y based on substantive considerations. r pbis r 0 r 1 s r ¾ n0 n 1 nˆn 1 Œ P r 0 n 0 P r 1 n 1 n 0 n 1 Y based on statistics, where the approximation to the asymptotic distribution is questionable. monotone transformations example: point-biserial correlation simplification example: Mantel-Haenszel statistic remove s r and n (constant) remove º is a monotone function (n 0, n 1 C 0) T pbisˆa n 1P r 0 n 0 P r 1 n 0 n 1 n 1 Q r 0 n 0 Q r 1 n 0 n 1 Y counts T s CT 0 Y if persons with high ability (r v high) fail to solve a certain item, this item may show too low discrimination, falsity, or indicate multidimensionality Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 17 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 18 implification Example: Mantel-Haenszel statistic tests for conditional independence of two nominal variables across several strata (e.g., 2 2 C tables) MH P c N 11c P c EˆN 11c n c Ž 2 V ar P c N 11c n c Ž can be used to test various RM violations. as. χ 2 df 1 Verguts & DeBoeck (2001): sufficiency, unidimensionality, item dependence Mantel-Haenszel statistic may be simplified to T MH Q c n 11c Q c ñ 11c Ž 2 where ñ 11c Q n 11c ~ Implementation in R: erm & Raschampler Implementation in R: erm & Raschampler some statistics in erm Y subgroup-invariance T 10 based on counts on certain item responses (global test) T 4 counts of positive responses in person subgroups (test on item level) Y conditional independence T 11 large inter-item correlations (global test) T 1 many equal responses (test on item level) T 2 high dispersion of rawscore r v for a set of items (test on item level) Raschampler Y supply user defined function for arbitrary T-statistics Verhelst, Hatzinger, & Mair (2007) Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 19 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 20

6 Application in erm Example: conditional dependence T 1 : many equal responses (test on item level) > library(erm) > library(raschampler) > t1 <- NPtest(raschdat1,n=500,burn_in=500,step=32,seed=123,method="T1") > print(t1,alpha=0.05) Nonparametric RM model test: T1 (local dependence - increased inter-item correlations) (counting cases with equal responses on both items) Number of sampled matrices: 500 Number of Item-Pairs tested: 435 Item-Pairs with one-sided p < 0.05 (1,9) (1,26) (4,12) (4,26) (4,28) (8,13) (9,16) (10,24) (18,28) (21,22) (25,29) Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 21