Moving beyond testing means: Using MACS modeling to test group differences in variances and covariances

Transcription

1 crmda.ku.edu Moving beyond testing means: Using MACS modeling to test group differences in variances and covariances Todd D. Little University of Kansas & Hal S. Shorey Widener University crmda.ku.edu Symposium talk presented SPSP in NOLA Special Thanks to: Sandy Sola, Steve Short, & Alexander Schoemann

2 Example 1: A 3 x 2 Factorial Design Test the effects of method of assessment (online, group paper & pencil, vs. individual paper & pencil) on the responses to various measures. A multiple assessment, twowave (repeated-measures) longitudinal design was used Positive and Negative affect were among the constructs assessed (same I Feel instrument and items as we ve been working with). 100 Assessed in Crowded Classroom TIME P s Total Assessed Online TIME Assessed in Lab Separated by Cubicles TIME Assessed Online Again TIME 2 P s completed Time 2 seven days after completing Time 1 online

3 CFA: Group x Time Online Pos T1 Neg T1 Pos T2 Neg T2 Note: Indicators not shown Allow for all within-group correlations Lab Class Pos T1 Neg T1 Pos T1 Neg T1 Pos T2 Neg T2 Pos T2 Neg T2

4

5 Factorial Invariance Configural invariance: Same factor loading pattern across groups, no constraints. ( g) ( g) ( g) ( g) ( g) ( g) ( g) ( g) ( g) x x Weak (metric) invariance: Factor loadings proportionally equal across groups. ( g) ( g) ( g) ( g) ( g) ( g) x x Strong (scalar) invariance: Loadings & intercepts proportionally equal across groups. ( g) ( g) ( g) ( g) ( g) x x Strict invariance: Add unique variances to be exactly equal across groups. -BAD ( g) ( g) ( g) ( g) x x

6 Establish Measurement Invariance Model χ2 df p RMSEA 90% CI TLI/NNFI CFI Constraint Tenable Null < Congfig < Invariance Loading < Yes Invariance Intercept < Yes Invariance Note: For loading and intercept invariance we are simultaneously testing across time and across groups. One could do this testing sequentially testing first for time invariance and then for group invariance, for example. (L7.0.CarpThesis.Null) (L7.1.CarpThesis.Configural) (L7.2.CarpThesis.Weak-Loadings) (L7.3.CarpThesis.Strong-Intercepts)

7 Unconstrained Estimates for Online Condition (latent correlations) (L7.6.0.CarpThesis.create.phantom.xside.LS8) POSA_T1 NEGA_T1 POSA_T2 NEGA_T2 POSA_T NEGA_T (0.108) POSA_T (0.046) (0.108) NEGA_T (0.108) (0.048) (0.109) (latent means) (0.232) (0.186) (0.232) (0.174) (latent Standard Deviations) (0.071) (0.071) (0.071) (0.071)

8 Unconstrained Estimates for Lab Condition (latent correlations) (L7.6.0.CarpThesis.create.phantom.xside.LS8) POSA_T1 NEGA_T1 POSA_T2 NEGA_T2 POSA_T NEGA_T (0.101) POSA_T (0.075) (0.107) NEGA_T (0.109) (0.056) (0.107) (latent means) (0.247) (0.189) (0.242) (0.168) (latent Standard Deviations) (0.071) (0.071) (0.071) (0.071)

9 Unconstrained Estimates for Class Condition (latent correlations) (L7.6.0.CarpThesis.create.phantom.xside.LS8) POSA_T1 NEGA_T1 POSA_T2 NEGA_T2 POSA_T NEGA_T (0.107) POSA_T (0.040) (0.112) NEGA_T (0.106) (0.043) (0.114) (latent means) (0.236) (0.177) (0.233) (0.157) (latent Standard Deviations) (0.071) (0.072) (0.070) (0.071)

10

11 Follow-up Tests of Variances Model χ 2 df p Δχ 2 Δdf p decide? Strong Invariance <.001 Interaction < <.001 diff (via Contrast Codes) Pos Aff < diff Neg Aff < <.001 diff Group < same Time < <.001 diff Tests of the variances indicate an interaction for both positive and negative affect.

12 Results for Variances The classroom group differs across time in the latent variances of both positive and negative affect. The variances shrink between Time 1 and Time 2, but only for the classroom condition. This finding represents a group by time interaction on the variance in the responses (an homogenizing effect).

13 crmda.ku.edu Thank You! Questions? crmda.ku.edu

14 Closer look at just the Means (L7.6.0.CarpThesis.create.phantom.xside.LS8) POSA_T1 NEGA_T1 POSA_T2 NEGA_T2 (latent means Online Condition) (0.232) (0.186) (0.232) (0.174) (latent means Lab condition) (0.247) (0.189) (0.242) (0.168) (latent Means Class Condition) (0.236) (0.177) (0.233) (0.157)

15 Test of the Latent Means Model χ 2 df p Δχ 2 Δdf p Constraint Tenable Latent Mean Invariance < <.001 No Interaction < <.001 No (via Contrast Codes) Group < Yes Time < <.001 No Pos Aff < Yes Neg Aff < <.001 No Online < No Lab < <.001 No Classroom < <.001 No Tests of the means indicate a main-effect of time for negative affect. Negative affect went down at Time 2, but this was true for all three conditions (no interaction or group main-effect was found).

16 Conclusions The effects of test-taking environment do not appear to influence the mean-levels of the affect variables, or the intercorrelation between the two affect variables. Compared to online and the individual lab environment, taking the I FEEL in the context of a crowded classroom reduces the variability among the respondents. All three conditions were equally sensitive to the drop in negative affect between the initial online assessment and the follow-up assessment one week later. The results suggest that test-taking format has some influence on the quality of data collected at least with regard to emotionrelated constructs such as positive and negative affect.