The Use of Quadratic Form Statistics of Residuals to Identify IRT Model Misfit in Marginal Subtables

Transcription

1 The Use of Quadratic Form Statistics of Residuals to Identify IRT Model Misfit in Marginal Subtables 1 Yang Liu Alberto Maydeu-Olivares 1 Department of Psychology, The University of North Carolina at Chapel Hill Department of Psychology, University of Barcelona IMPS 01 Lincoln, NE 1/0

2 /0 Acknowledgment Dr. David Thissen Other folks from the Thurstone lab

3 Overview: goodness-of-fit test of IRT model specification parameter estimation overall GOF test reject H 0 fail to reject H 0. remove items change model... 3/0

4 Overview: goodness-of-fit test of IRT model specification parameter estimation overall GOF test reject H 0 fail to reject H 0 marginal subtable GOF test remove items change model... 3/0

5 4/0 Marginal subtable GOF test Consider A, a subset of r items (e.g. r = for pairs, 3 for triplets) Principles of testing marginal fit 1 Do not want to reject well-fitting items r should not be too large, usually r 3 Null hypothesis H 0 : π A = π A (θ A ) π A : probabilities of cells in subtable A θ A : item parameters in A Use test statistics based on residuals ê A := p A π A (ˆθ A ) p A : observed proportion π A (ˆθ A ): model-implied probability ˆθA : some estimator of θ A

6 4/0 Marginal subtable GOF test Consider A, a subset of r items (e.g. r = for pairs, 3 for triplets) Principles of testing marginal fit 1 Do not want to reject well-fitting items r should not be too large, usually r 3 Null hypothesis H 0 : π A = π A (θ A ) π A : probabilities of cells in subtable A θ A : item parameters in A Use test statistics based on residuals ê A := p A π A (ˆθ A ) p A : observed proportion π A (ˆθ A ): model-implied probability ˆθA : some estimator of θ A

7 4/0 Marginal subtable GOF test Consider A, a subset of r items (e.g. r = for pairs, 3 for triplets) Principles of testing marginal fit 1 Do not want to reject well-fitting items r should not be too large, usually r 3 Null hypothesis H 0 : π A = π A (θ A ) π A : probabilities of cells in subtable A θ A : item parameters in A Use test statistics based on residuals ê A := p A π A (ˆθ A ) p A : observed proportion π A (ˆθ A ): model-implied probability ˆθA : some estimator of θ A

8 4/0 Marginal subtable GOF test Consider A, a subset of r items (e.g. r = for pairs, 3 for triplets) Principles of testing marginal fit 1 Do not want to reject well-fitting items r should not be too large, usually r 3 Null hypothesis H 0 : π A = π A (θ A ) π A : probabilities of cells in subtable A θ A : item parameters in A Use test statistics based on residuals ê A := p A π A (ˆθ A ) p A : observed proportion π A (ˆθ A ): model-implied probability ˆθA : some estimator of θ A

9 5/0 Asymptotic tests based on residuals Proposition 1 (Maydeu-Olivares & Liu, 01): Let ˆθ A be the full table MLE corresponding to item subset A, under H 0 : N êa d N (0, ΣA ) where Σ A = Γ A A I 1 [A] A, Γ A = D A π A (θ A )π A (θ A ), D A = Diag(π A (θ A )), A = π A / θ A, I [A] is the block of Fisher information matrix I corresponding to θ A. Standardized residuals (Reiser, 1996; Maydeu-Olivares & Joe, 006) Quadratic form statistics of residuals

10 5/0 Asymptotic tests based on residuals Proposition 1 (Maydeu-Olivares & Liu, 01): Let ˆθ A be the full table MLE corresponding to item subset A, under H 0 : N êa d N (0, ΣA ) where Σ A = Γ A A I 1 [A] A, Γ A = D A π A (θ A )π A (θ A ), D A = Diag(π A (θ A )), A = π A / θ A, I [A] is the block of Fisher information matrix I corresponding to θ A. Standardized residuals (Reiser, 1996; Maydeu-Olivares & Joe, 006) Quadratic form statistics of residuals

11 5/0 Asymptotic tests based on residuals Proposition 1 (Maydeu-Olivares & Liu, 01): Let ˆθ A be the full table MLE corresponding to item subset A, under H 0 : N êa d N (0, ΣA ) where Σ A = Γ A A I 1 [A] A, Γ A = D A π A (θ A )π A (θ A ), D A = Diag(π A (θ A )), A = π A / θ A, I [A] is the block of Fisher information matrix I corresponding to θ A. Standardized residuals (Reiser, 1996; Maydeu-Olivares & Joe, 006) Quadratic form statistics of residuals

12 6/0 Quadratic form statistics of residuals 1 Pearson s X XA = N ê 1 A ˆD A êa W statistic (Reiser, 1996) W A = N ê A ˆΣ + AêA where ˆΣ + A is the Moore-Penrose pseudoinverse of ˆΣ A 3 M n statistic (Maydeu-Olivares & Joe, 005, 006) M A = N ê AÛAê A where ÛA = ˆD 1 A ˆD 1 A ˆ A ( ˆ A ˆD 1 A ˆ A ) 1 ˆ A ˆD 1 A

13 Asymptotic distributions under H 0 Proposition (Cochran, 1934; Box, 1954; Tan, 1977): Let p-dimensional random vector z N (0, Ω), and W is any p p real symmetric matrix, then z Wz p λ i χ 1 i=1 where each chi-squared random variable is distributed independently of every other, and λ i s are eigenvalues of matrix ΩW. 1 W A d χ s where s is the rank of Σ A (use ˆΣ A in practice) M A d χ t where t = dim(π A ) dim(θ A ) 1 3 X A d λ i χ 1, where λ i s are eigenvalues for matrix Σ A D 1 A (use ˆΣ A ˆD 1 A in practice) 7/0

14 Asymptotic distributions under H 0 Proposition (Cochran, 1934; Box, 1954; Tan, 1977): Let p-dimensional random vector z N (0, Ω), and W is any p p real symmetric matrix, then z Wz p λ i χ 1 i=1 where each chi-squared random variable is distributed independently of every other, and λ i s are eigenvalues of matrix ΩW. 1 W A d χ s where s is the rank of Σ A (use ˆΣ A in practice) M A d χ t where t = dim(π A ) dim(θ A ) 1 3 X A d λ i χ 1, where λ i s are eigenvalues for matrix Σ A D 1 A (use ˆΣ A ˆD 1 A in practice) 7/0

15 Asymptotic distributions under H 0 Proposition (Cochran, 1934; Box, 1954; Tan, 1977): Let p-dimensional random vector z N (0, Ω), and W is any p p real symmetric matrix, then z Wz p λ i χ 1 i=1 where each chi-squared random variable is distributed independently of every other, and λ i s are eigenvalues of matrix ΩW. 1 W A d χ s where s is the rank of Σ A (use ˆΣ A in practice) M A d χ t where t = dim(π A ) dim(θ A ) 1 3 X A d λ i χ 1, where λ i s are eigenvalues for matrix Σ A D 1 A (use ˆΣ A ˆD 1 A in practice) 7/0

16 Asymptotic distributions under H 0 Proposition (Cochran, 1934; Box, 1954; Tan, 1977): Let p-dimensional random vector z N (0, Ω), and W is any p p real symmetric matrix, then z Wz p λ i χ 1 i=1 where each chi-squared random variable is distributed independently of every other, and λ i s are eigenvalues of matrix ΩW. 1 W A d χ s where s is the rank of Σ A (use ˆΣ A in practice) M A d χ t where t = dim(π A ) dim(θ A ) 1 3 X A d λ i χ 1, where λ i s are eigenvalues for matrix Σ A D 1 A (use ˆΣ A ˆD 1 A in practice) 7/0

17 8/0 Computing asymptotic p-values 1 W statistic (requires Î) In general we do not know the rank of Σ A use the rank of ˆΣ A set some cutoff for zero EV (e.g ) M n statistic (does not require Î) Model should be identified in the subtable (dim(θ A ) < dim(π A ) 1) When r =, graded model is identified, PL is not 3 X statistic Computing/approximating the tail probabilities (requires Î) Satterthwaite s approximation (Satterthwaite, 1941) Inversion method (Imhof, 1961) Heuristical chi-squared reference distributions (does not require Î) Subtable MLE: d.f. = dim(π A ) dim(θ A ) 1 Chen-Thissen (1997): d.f. of independence model test

18 8/0 Computing asymptotic p-values 1 W statistic (requires Î) In general we do not know the rank of Σ A use the rank of ˆΣ A set some cutoff for zero EV (e.g ) M n statistic (does not require Î) Model should be identified in the subtable (dim(θ A ) < dim(π A ) 1) When r =, graded model is identified, PL is not 3 X statistic Computing/approximating the tail probabilities (requires Î) Satterthwaite s approximation (Satterthwaite, 1941) Inversion method (Imhof, 1961) Heuristical chi-squared reference distributions (does not require Î) Subtable MLE: d.f. = dim(π A ) dim(θ A ) 1 Chen-Thissen (1997): d.f. of independence model test

19 8/0 Computing asymptotic p-values 1 W statistic (requires Î) In general we do not know the rank of Σ A use the rank of ˆΣ A set some cutoff for zero EV (e.g ) M n statistic (does not require Î) Model should be identified in the subtable (dim(θ A ) < dim(π A ) 1) When r =, graded model is identified, PL is not 3 X statistic Computing/approximating the tail probabilities (requires Î) Satterthwaite s approximation (Satterthwaite, 1941) Inversion method (Imhof, 1961) Heuristical chi-squared reference distributions (does not require Î) Subtable MLE: d.f. = dim(π A ) dim(θ A ) 1 Chen-Thissen (1997): d.f. of independence model test

20 8/0 Computing asymptotic p-values 1 W statistic (requires Î) In general we do not know the rank of Σ A use the rank of ˆΣ A set some cutoff for zero EV (e.g ) M n statistic (does not require Î) Model should be identified in the subtable (dim(θ A ) < dim(π A ) 1) When r =, graded model is identified, PL is not 3 X statistic Computing/approximating the tail probabilities (requires Î) Satterthwaite s approximation (Satterthwaite, 1941) Inversion method (Imhof, 1961) Heuristical chi-squared reference distributions (does not require Î) Subtable MLE: d.f. = dim(π A ) dim(θ A ) 1 Chen-Thissen (1997): d.f. of independence model test

21 Simulation A: dichotomous items 9 items, fitted model is always PL, 1000 reps for each subcondition Condition A1: PL model Slopes β = (.7, 1.67, 1.8,.7, 1.67, 1.8) Intercepts α = (1.4, 0, 1.06, 1.4, 0, 1.06, 1.4, 0, 1.06) Condition A: Mixture model LV 1 N (0, 1) + 1 N (3, 1) Condition A3: 3-dimensional (independent cluster) model LVs N 0, β = /0

22 Simulation A: dichotomous items 9 items, fitted model is always PL, 1000 reps for each subcondition Condition A1: PL model Slopes β = (.7, 1.67, 1.8,.7, 1.67, 1.8) Intercepts α = (1.4, 0, 1.06, 1.4, 0, 1.06, 1.4, 0, 1.06) Condition A: Mixture model LV 1 N (0, 1) + 1 N (3, 1) Condition A3: 3-dimensional (independent cluster) model LVs N 0, β = /0

23 Simulation A: dichotomous items 9 items, fitted model is always PL, 1000 reps for each subcondition Condition A1: PL model Slopes β = (.7, 1.67, 1.8,.7, 1.67, 1.8) Intercepts α = (1.4, 0, 1.06, 1.4, 0, 1.06, 1.4, 0, 1.06) Condition A: Mixture model LV 1 N (0, 1) + 1 N (3, 1) Condition A3: 3-dimensional (independent cluster) model LVs N 0, β = /0

24 Simulation A: Statistics Subtable Statistics Reference distribution Pair W (jk) χ s ; s = # EV of ˆΣ (jk) 10 5 (j, k) X(jk) Chen-Thissen χ 1 Satterthwaite s approximation Inversion method Triplet W (jkl) χ s ; s = # EV of ˆΣ (jkl) 10 5 (j, k, l) M (jkl) χ 1 X(jkl) Subtable MLE χ 1 Chen-Thissen χ 4 Satterthwaite s approximation Inversion method 10/0

25 11/0 Result A1: PL (true), PL (fitted) Pair (, 3) Triplet (1,, 3) Rejection rate W(3) X (3) (Chen Thissen) X (3) (Satterthwaite) X (3) (Inversion) Rejection rate W(13) M(13) X (13) (Subtable MLE) X (13) (Chen Thissen) X (13) (Satterthwaite) X (13) (Inversion) α level α level

26 1/0 Result A: Mixture (true), PL (fitted) Pair (, 3) Triplet (1,, 3) Rejection rate W(3) X (3) (Satterthwaite) X (3) (Inversion) Rejection rate W(13) M(13) X (13) (Satterthwaite) X (13) (Inversion) α level α level

27 13/0 Result A3: 3-dim (true), PL (fitted) Pair (, 3) Triplet (1,, 3) Rejection rate W(3) X (3) (Satterthwaite) X (3) (Inversion) Rejection rate W(13) M(13) X (13) (Satterthwaite) X (13) (Inversion) α level α level

28 14/0 Condition B: 3-category items 6 items, fitted model is always graded, 1000 reps for each subcondition Condition B1: Graded model Condition B: Mixture model β = (.7, 1.67, 1.8,.7, 1.67, 1.8) [ α = LV 1 N (0, 1) + 1 N (3, 1) Condition B3: -dimensional (independent cluster) model ([ ] [ ]) LVs N, [ β = ] ]

29 14/0 Condition B: 3-category items 6 items, fitted model is always graded, 1000 reps for each subcondition Condition B1: Graded model Condition B: Mixture model β = (.7, 1.67, 1.8,.7, 1.67, 1.8) [ α = LV 1 N (0, 1) + 1 N (3, 1) Condition B3: -dimensional (independent cluster) model ([ ] [ ]) LVs N, [ β = ] ]

30 14/0 Condition B: 3-category items 6 items, fitted model is always graded, 1000 reps for each subcondition Condition B1: Graded model Condition B: Mixture model β = (.7, 1.67, 1.8,.7, 1.67, 1.8) [ α = LV 1 N (0, 1) + 1 N (3, 1) Condition B3: -dimensional (independent cluster) model ([ ] [ ]) LVs N, [ β = ] ]

31 Condition B: Statistics Subtable Statistics Reference distribution Pairs W (jk) χ s ; s = # EV of ˆΣ (jk) 10 5 (j, k) M (jk) χ X(jk) Subtable MLE χ Chen-Thissen χ 4 Satterthwaite s approximation Inversion method 15/0

32 16/0 Result B1: graded (true), graded (fitted) Pair (1, ) Pair (, 3) Rejection rate W(1) M(1) X (1) (Subtable MLE) X (1) (Chen Thissen) X (1) (Satterthwaite) X (1) (Inversion) Rejection rate W(3) M(3) X (3) (Subtable MLE) X (3) (Chen Thissen) X (3) (Satterthwaite) X (3) (Inversion) α level α level

33 17/0 Result B: mixture (true), graded (fitted) Pair (1, ) Pair (, 3) Rejection rate W(1) M(1) X (1) (Satterthwaite) X (1) (Inversion) Rejection rate W(3) M(3) X (3) (Satterthwaite) X (3) (Inversion) α level α level

34 18/0 Result B3: -dim (true), graded (fitted) Pair (1, ) Pair (, 3) Rejection rate W(1) M(1) X (1) (Satterthwaite) X (1) (Inversion) Rejection rate W(3) M(3) X (3) (Satterthwaite) X (3) (Inversion) α level α level

35 19/0 Concluding remarks When the model is correctly specified Empirical distributions of W and M n closely match their asymptotic reference distributions Heuristical chi-squared reference distributions for X works poorly p-values based on the correct asymptotic distribution of X works well; the Satterthwaite s approximation works as well as the inversion method Among W, M n and X, we suggest the use of W statistics for pairs (and triplets, if necessary) which always have high power Future direction Long tests where exact Fisher information cannot be computed: Use OPG (a.k.a. XPD) estimator of information ÎOPG = ˆ Ŵ ˆ where Ŵ = Diag{p c /πc} C c=1

36 19/0 Concluding remarks When the model is correctly specified Empirical distributions of W and M n closely match their asymptotic reference distributions Heuristical chi-squared reference distributions for X works poorly p-values based on the correct asymptotic distribution of X works well; the Satterthwaite s approximation works as well as the inversion method Among W, M n and X, we suggest the use of W statistics for pairs (and triplets, if necessary) which always have high power Future direction Long tests where exact Fisher information cannot be computed: Use OPG (a.k.a. XPD) estimator of information ÎOPG = ˆ Ŵ ˆ where Ŵ = Diag{p c /πc} C c=1

37 19/0 Concluding remarks When the model is correctly specified Empirical distributions of W and M n closely match their asymptotic reference distributions Heuristical chi-squared reference distributions for X works poorly p-values based on the correct asymptotic distribution of X works well; the Satterthwaite s approximation works as well as the inversion method Among W, M n and X, we suggest the use of W statistics for pairs (and triplets, if necessary) which always have high power Future direction Long tests where exact Fisher information cannot be computed: Use OPG (a.k.a. XPD) estimator of information ÎOPG = ˆ Ŵ ˆ where Ŵ = Diag{p c /πc} C c=1

38 19/0 Concluding remarks When the model is correctly specified Empirical distributions of W and M n closely match their asymptotic reference distributions Heuristical chi-squared reference distributions for X works poorly p-values based on the correct asymptotic distribution of X works well; the Satterthwaite s approximation works as well as the inversion method Among W, M n and X, we suggest the use of W statistics for pairs (and triplets, if necessary) which always have high power Future direction Long tests where exact Fisher information cannot be computed: Use OPG (a.k.a. XPD) estimator of information ÎOPG = ˆ Ŵ ˆ where Ŵ = Diag{p c /πc} C c=1

39 19/0 Concluding remarks When the model is correctly specified Empirical distributions of W and M n closely match their asymptotic reference distributions Heuristical chi-squared reference distributions for X works poorly p-values based on the correct asymptotic distribution of X works well; the Satterthwaite s approximation works as well as the inversion method Among W, M n and X, we suggest the use of W statistics for pairs (and triplets, if necessary) which always have high power Future direction Long tests where exact Fisher information cannot be computed: Use OPG (a.k.a. XPD) estimator of information ÎOPG = ˆ Ŵ ˆ where Ŵ = Diag{p c /πc} C c=1

40 Thanks! 0/0