Evaluating sensitivity of parameters of interest to measurement invariance using the EPC-interest

Transcription

1 Evaluating sensitivity of parameters of interest to measurement invariance using the EPC-interest Department of methodology and statistics, Tilburg University WorkingGroupStructuralEquationModeling , FU Berlin

2 Conclusion This talk discusses the ``EPC-interest'' EPC-interest is like SEM's expected parameter change (``EPC-self'') but instead of measuring the change in the restricted parameter, it measures the change in the parameter of interest; Introduced for measurement invariance evaluation in continuous data SEM by Oberski (2014) and for categorical data by Oberski et al. (frth) Implemented in Latent Gold 5.0, experimental version of, working on main branch.

3 The problem of measurement invariance

4 p(y) = ξ J p(ξ x) p(y j ξ) j=1

5 p(y) = ξ J p(ξ x) p(y j ξ, x) j=1

6 Measurement invariance: the problem Problem of measurement invariance: we want to know γ, but δ 0 might bias this parameterofinterest.

7 Preceding solutions to the problem 1. Selecting one indicator as a reference indicator (``anchor item'') apriori; 2. Imposing a strong prior on differential functioning (Muthén&Asparouhov2012) 3. Test null hypothesis of full or partial invariance.

8 Preceding solutions to the problem 1. Selecting one indicator as a reference indicator (``anchor item'') apriori; + Don't need further restrictions; - No way of testing reference indicator. E.g.: Setting loadings to 1 in each group, ``alignment method''. 2. Imposing a strong prior on differential functioning (Muthén&Asparouhov2012) 3. Test null hypothesis of full or partial invariance.

9 Preceding solutions to the problem 1. Selecting one indicator as a reference indicator (``anchor item'') apriori; 2. Imposing a strong prior on differential functioning (Muthén&Asparouhov2012) + Attractive data-driven solution when prior is neither too informative nor too weak; - More research needed to figure out when prior is neither too informative nor too weak. 3. Test null hypothesis of full or partial invariance.

10 Preceding solutions to the problem 1. Selecting one indicator as a reference indicator (``anchor item'') apriori; 2. Imposing a strong prior on differential functioning (Muthén&Asparouhov2012) 3. Test null hypothesis of full or partial invariance. + When high-powered test of full measurement invariance is not rejected, may be safe to simply continue without the need for further modeling; - Rarely happens in practice. + Partial invariance looks at size and significance of δ, but not all big δ's are important nor are small ones necessarily unimportant. So does not guarantee that he parameter of interest in free of measurement differences (Oberski,2014) 4. Sensitivity analysis: allow partial violations when they matter.

11 Preceding solutions to the problem 1. Selecting one indicator as a reference indicator (``anchor item'') apriori; 2. Imposing a strong prior on differential functioning (Muthén&Asparouhov2012) 3. Test null hypothesis of full or partial invariance. 4. Sensitivity analysis: allow partial violations when they matter.

12 Measurement invariance: the problem and a solution Problem of measurement invariance: we want to know γ, but δ 0 might bias this parameterofinterest. Solution: Use EPC-interest: the expected change in γ when freeing δ. If EPC-interest is big (e.g. can change sign), incorporate δ; if EPC-interest is small, ignore it.

13 EPC-interest

14 EPC-interest, also for categorical data ( ) θ ( ˆδ) EPC-interest = P δ δ = PĤ 1 θθ Ĥθδ EPC-self = ˆγ a ˆγ + O(ψ ψ), where P selects the parameters of interest γ from the parameter vector θ, H is a Hessian, and O(ψ ψ) is an approximation term depending on the overall amount of misspecification (parameter differences).

15 Simulation: how good is the approximation? Setup: P(Y j = 1 x) = [1 + exp( x)] 1, with j {2, 3, 4}, and structural model x = γz + ϵ with γ = 1 and ϵ N(0, 1). We then introduced a violation of measurement invariance for the first indicator, P(Y 1 = 1 x) = [1 + exp( x δz)] 1. Nine conditions varied sample size, n {250, 500, 1000}, and the size of the invariance violation: δ = 0 (no violation), 0.5 (moderate), or 1 (extreme). Data were generated using R and analyzed using Latent GOLD

16 n True δ Est. ˆγ Bias ˆγ EPC-int Table : Simulation study of EPC-interest. Shown is the average point estimate for the γ parameter of interest under full measurement invariance (``Est''), its difference from the true value γ = 1 (``Bias''), and the average EPC-interest.

17 Example with categorical data See

18 Figure : Graphical representation of the multilevel latent class regression model for (post)materialism measured by three partial ranking tasks. Observed variables are shown in rectangles while unobserved (``latent'') variables are shown in ellipses.

19 Multilevel latent class model w/ covariates for rankings L(θ) = P(A 1, A 2, B 1, B 2, C 1, C 2 Z 1, Z 2 ) = C n c P(G c ) P(X ic Z 1ic, Z 2ic, G c ) c=1 G i=1 X P(A 1ic, A 2ic X ic )P(B 1ic, B 2ic X ic )P(C 1ic, C 2ic X ic ), Goal: estimate γ (especially its sign). Possibleproblem: Violations of scalar and metric measurement invariance (DIF), parameterized respectively as τ and λ. Solution: See if these matter for the sign of γ.

20 Table : Full invariance multilevel latent class model: EPC-interest values. λ jkxg Estimates EPC-interest Est. s.e. Task 1 Task 2 Task 3 Class 1 GDP (0.007) Class 2 GDP (0.012) Class 1 Women (0.001) Class 2 Women (0.001) Free ``loadings'' for task 1 and task 2.

21 Table : Partially invariant multilevel latent class model: EPC-interest values. λ jkxg Estimates EPC-interest Est. s.e. Task 1 Task 2 Task 3 Class 1 GDP (0.008) Class 2 GDP (0.011) Class 1 Women (0.001) Class 2 Women (0.001) 0.007

22 WhathasbeengainedbyusingEPC-interest: I am fairly confident here that there truly is "approximate measurement invariance", in the sense that any violations of measurement invariance do not bias the primary conclusions. I think attaining this goal is the main purpose of model fit evaluation.

23 Conclusion

24 Whatistheproblem? We do latent variable modeling with a goal in mind. But the latent variable model might be misspecified. The appropriate question: "will that affect my goal?" The actual question: "do the data fit the model in the population" (LR) or "are the model and the data far apart relative to model complexity" (RMSEA etc.) Whatisthesolution? Evaluatedirectlywhateffectpossiblemisspecifications haveonthegoaloftheanalysis.

25 References Oberski, D. (2014). Evaluating sensitivity of parameters of interest to measurement invariance in latent variable models. Political Analysis, 22(1): Oberski, D. and Vermunt, J. (2013). A model-based approach to goodness-of-fit evaluation in item response theory. Measurement: InterdisciplinaryResearch&Perspectives, 11: Oberski, D., Vermunt, J., and Moors, G. (frth). Evaluating measurement invariance in categorical data latent variable models with the EPC-interest. Vermunt, J. K. and Magidson, J. (2013). TechnicalguideforLatent GOLD 5.0: Basicandadvanced. Statistical Innovations Inc., Belmont, MA.

26 SEM regression coefficient example Two problems with invariance testing Conclusions Unaffected by misspecification Affected by misspecification Misspecified invariance model fit ``Good'' fit ``Bad'' fit (1) (2) Overparameterization or unnecessarily d i s c a r d e d i t e m, group, or scale. (3) Non-invariance invalidates conclusions. (4)

27 SEM regression coefficient example European Sociological Review 2008, 24(5),

28 SEM regression coefficient example Poland Czech Republic Greece United Kingdom Slovenia Belgium France Portugal Finland Hungary Norway Spain Ireland Germany Netherlands Switzerland Austria Danmark Sweden Poland Czech Republic Greece United Kingdom Slovenia Belgium France Portugal Finland Hungary Norway Spain Ireland Germany Netherlands Switzerland Austria Danmark Sweden Conservation Self transcendence Regression coefficient ALLOW NOCOND

29 SEM regression coefficient example EPC-interest statistics of at least 0.1 in absolute value with respect to the latent variable regression coefficients. Metric invariance (loading) restriction ``Conditions Work skills'' in... Slovenia France Hungary Ireland EPC-interest w.r.t.: Conditions Self-transcendence Conservation SEPC-self

30 SEM regression coefficient example What has been gained by using EPC-interest Full metric invariance model: "close fit"; EPC-interest still detects threats to cross-country comparisons of regression coefficients;

31 SEM regression coefficient example What has been gained by using EPC-interest Full metric invariance model: "close fit"; EPC-interest still detects threats to cross-country comparisons of regression coefficients; MI and EPC-self do not detect these particular misspecifications;

32 SEM regression coefficient example What has been gained by using EPC-interest Full metric invariance model: "close fit"; EPC-interest still detects threats to cross-country comparisons of regression coefficients; MI and EPC-self do not detect these particular misspecifications; MI and EPC-self detect other misspecifications;

33 SEM regression coefficient example What has been gained by using EPC-interest Full metric invariance model: "close fit"; EPC-interest still detects threats to cross-country comparisons of regression coefficients; MI and EPC-self do not detect these particular misspecifications; MI and EPC-self detect other misspecifications; Looking at EPC-interest reveals that these do not affect the cross-country comparisons of regression coefficients.

34 SEM regression coefficient example What has been gained by using EPC-interest Full metric invariance model: "close fit"; EPC-interest still detects threats to cross-country comparisons of regression coefficients; MI and EPC-self do not detect these particular misspecifications; MI and EPC-self detect other misspecifications; Looking at EPC-interest reveals that these do not affect the cross-country comparisons of regression coefficients.

35 Example Goal: Estimate gender differences in "valuing Stimulation": (1) Very much like me; (2) Like me; (3) Somewhat like me; (4) A little like me; (5) Not like me; (6) Not like me at all. (S)he looks for adventures and likes to take risks. (S)he wants to have an exciting life. (S)he likes surprises and is always looking for new things to do. He thinks it is important to do lots of different things in life. Tool: Structural Equation Model for European Social Survey data (n = men and women). (OriginalstudybySchwarzetal. 2005)

36 Assume: Butreally(?): What difference does it make for the goal: true gender differences in values? (re-analysisofdatabyoberski2014) Latent mean difference estimate ± 2 s.e Men value more Women value more Model Scalar invariance Free intercept 'Adventure' ACPO ST SD HE COTR SE UN BE "Human value" factor

37 PROBLEM The original authors found that the conditional independence model fit the data "approximately" (p. 1013)... "Chi-squaredeterioratedsignificantly, χ 2 (19) = 3313, p <.001, butcfi didnotchange. Changeinchi-squareis highlysensitivewithlargesamplesizesandcomplex models. Theotherindicessuggestedthatscalarinvariance mightbeaccepted(cfi =.88, RMSEA =.04, CI = , PCLOSE =1.0).''

38 PROBLEM The original authors found that the conditional independence model fit the data "approximately" (p. 1013)... "Chi-squaredeterioratedsignificantly, χ 2 (19) = 3313, p <.001, butcfi didnotchange. Changeinchi-squareis highlysensitivewithlargesamplesizesandcomplex models. Theotherindicessuggestedthatscalarinvariance mightbeaccepted(cfi =.88, RMSEA =.04, CI = , PCLOSE =1.0).''... but unfortunately this "acceptable" misspecification could reversetheirconclusions!

39 EPC-interest applied to Stimulation example After fitting the full scalar invariance model, Effect size estimate of sex difference in Stimulation is (s.e ).

40 EPC-interest applied to Stimulation example After fitting the full scalar invariance model, Effect size estimate of sex difference in Stimulation is (s.e ). But EPC-interest of equal "Adventure" item intercept is

41 EPC-interest applied to Stimulation example After fitting the full scalar invariance model, Effect size estimate of sex difference in Stimulation is (s.e ). But EPC-interest of equal "Adventure" item intercept is So EPC-interest suggests conclusioncanbereversed by freeing a misspecified scalar invariance restriction

42 EPC-interest applied to Stimulation example After fitting the full scalar invariance model, Effect size estimate of sex difference in Stimulation is (s.e ). But EPC-interest of equal "Adventure" item intercept is So EPC-interest suggests conclusioncanbereversed by freeing a misspecified scalar invariance restriction Actualchange when freeing this intercept is very close to EPC-interest:

43 EPC-interest applied to Stimulation example After fitting the full scalar invariance model, Effect size estimate of sex difference in Stimulation is (s.e ). But EPC-interest of equal "Adventure" item intercept is So EPC-interest suggests conclusioncanbereversed by freeing a misspecified scalar invariance restriction Actualchange when freeing this intercept is very close to EPC-interest: