Using SEM to detect measurement bias in dichotomous responses An application to the measurement of intelligence in higher education S.Jak (), C.V.Dolan (2), C.W.M. van Zoelen (3), F.J. Oort () () Educational Sciences, University of Amsterdam (2) Psychological Methods, University of Amsterdam (3) Research & Development, Meurs HRM, Woerden, The Netherlands Working Group Structural Equation Modeling Annual Meeting 2009, February 27, Berlin
Introduction As part of research on the reliability and validity of the Dutch ability test Q000 capaciteiten hoog we investigated: The factorial structure of the test The unidimensionality of the subtests Measurement bias with respect to sex and age (using both the MG and RFA method)
Objective Detect measurement bias in dichotomous item responses. Find an appropriate measurement model 2. Investigate bias via the multi group (MG) method 3. Investigate bias via the restricted factor analysis (RFA) method
The Q000 capaciteiten hoog Ability test for highly educated people Seven subtests, 36 items in total. Scored 0 or (wrong/right). For this presentation we limit ourselves to the subtest: Math problems with 2 items.
Example math problem The monetary value of machine M is 2 times the mean of 8 other machines. Which part of the total value of the 9 machines is the value of machine M?. /20 2. /9 3. /8 4. /0
Method Respondents N = 67 highly educated 96 male, 656 female Three age groups for the MG method: group : age 5 35 (N = 586) group 2: age 35 45 (N = 645) group 3: age 45 65 (N = 386)
Method: Analysis Thresholds and tetrachoric correlations (instead of means, variances and covariances) Weighted least squares with adjusted mean and variance (WLSMV) as the estimation method
Procedure Find an appropriate measurement model Investigate bias via the multi group (MG) method Investigate bias via the restricted factor analysis (RFA) method
Measurement model for Math problems () χ 2 (48) = 329.56, p <.0 RMSEA =.06 NNFI =.8 ability v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Measurement model for Math problems (2) ability v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Measurement model for Math problems (2) χ 2 (43) = 63.50, p =.023 RMSEA =.07 NNFI =.98 0 Difference: χ 2 (6) = 229.48, p <.0 ability speed v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Procedure MG Multigroup sex: Separate groups (2 sex groups) Equality constraints on factor loadings and thresholds Start with a full constrained model, use MI s to decide to free thresholds (first) and factorloadings (second) Multigroup age: Separate groups (3 age groups) Equality constraints on factor loadings and thresholds Start with a full constrained model, use MI s to decide to free thresholds (first) and factorloadings (second)
Multigroup model sex τ men = τ women Λ men = Λ women Men 0 Women 0 ability speed ability speed v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2 v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Multigroup results: Sex bias Item 2 Female Threshold -.48 Factor loading.93 Model fit: χ 2 (89) = 08.78, p =.076 RMSEA =.07 Male -.2.93 NNFI =.98 Item 8 Threshold Factor loading (ability) Factor loading (speed) Difference: χ 2 (2) = 8.65, p <.0 Female.27.07 2.4 Male.63.07 2.4
Multigroup model age τ = τ 2 = τ 3 Λ = Λ 2 = Λ 3. Age 5 35 2. Age 35 45 3. Age 45 65 0 0 0 ability speed ability speed ability speed v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2 v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2 v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Multigroup results: Age bias Model fit: χ 2 (3) = 47.73, p =.5 RMSEA =.05 NNFI =.99 Difference: χ 2 () = 7.73, p <.0 Item 2 Group (5 35) Group 2 (35 45) Group 3 (45 65) Threshold - 0.6-0.30-0.30 Factor loading 0.78 0.78 0.78
Single group Procedure RFA Sex and age as exogenous variables (simultaneous) Use MI s to test direct effects of sex and age on indicators.
RFA method: sex and age bias χ 2 (60) = 44.97, p <.0 RMSEA =.03 NNFI =.95 -.9 -.0 (ns).06(ns).3 0. age sex 0 = female = male ability speed v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
RFA method: sex and age bias χ 2 (58) = 03.79, p <.0 RMSEA =.022 NNFI =.97 -.20 Difference: χ 2 (3) = 42.26, p <.0 -.0 (ns).07.34 0. age sex 0 = female = male ability speed.2 -.2 -.3 v v2 v3 v4 v5 v6 v7 v8 v9 v0 v v2
Summarized results RFA method and MG method are both suitable for detecting bias in dichotomous data. Item 2 is detected to be biased with respect to age and sex using both methods. Item is found to be biased with respect to age using the RFA method (not the MG) Item 8 is found to be biased with respect to sex using the MG method (not the RFA)
Can we explain the bias? Item 2 (biased against men and older people): At an office people can vote about what to do at the upcoming office party. There are 2 proposals: dancing or bowling. The company consists of 36 employees, whereof 20 females. Given that 22 employees vote for dancing, and 5 women for bowling, how many men voted for dancing?. 7 2. 9 3. 4 4. 5
Discussion Comparison MG and RFA method Further research