Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Transcription

1 Multivariate Behavioral Research, 9 (), Copyright 004, Lawrence Erlbaum Associates, Inc. Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling Jonathan Nevitt Department of Agriculture and Applied Economics Virginia Polytechnic Institute and State University Gregory R. Hancock Department of Measurement, Statistics and Evaluation University of Maryland, College Park Through Monte Carlo simulation, small sample methods for evaluating overall data-model fit in structural equation modeling were explored. Type I error behavior and power were examined using maximum likelihood (ML), Satorra-Bentler scaled and adjusted (SB; Satorra & Bentler, 1988, 1994), residual-based (Browne, 1984), and asymptotically distribution free (ADF; Browne, 198, 1984) test statistics. To accommodate small sample sizes the ML and SB statistics were adjusted using a k-factor correction (Bartlett, 1950); the residual-based and ADF statistics were corrected using modified and F statistics (Yuan & Bentler, 1998, 1999). Design characteristics include model type and complexity, ratio of sample size to number of estimated parameters, and distributional form. The k-factor-corrected SB scaled test statistic was especially stable at small sample sizes with both normal and nonnormal data. Methodologists are encouraged to investigate its behavior under a wider variety of models and distributional forms. Structural equation modeling (SEM) has become a versatile and widely used data analytic method for evaluating causal and predictive hypotheses in the behavioral sciences. Historically it has been a large sample technique, with minimum sample size guidelines ranging from five to ten cases per estimated model parameter (e.g., Bentler & Chou, 1987), depending upon the method of estimation employed. Unfortunately, practitioners are often unable to obtain sufficient numbers of cases to meet such minimum guidelines, let alone satisfy an estimation method s distributional assumptions. Thus, there has been increased demand for methods that perform optimally at smaller sample sizes and under varied distributional conditions. The methodological community has responded accordingly, with several major SEM software Correspondence concerning this article should be directed to the first author at Department of Agricultural and Applied Economics, Virginia Tech, Blacksburg, VA , or to jnevitt@vt.edu. MULTIVARIATE BEHAVIORAL RESEARCH 49

2 programs incorporating techniques that could be more viable under suboptimal conditions. While practitioners might be eager to employ such robust methods, their inclusion in software packages is often intended initially to facilitate methodologists scrutiny. The purpose of the current investigation is to provide just such scrutiny within the context of a factorially designed Monte Carlo investigation. Background For a system of p measured variables [yielding p* = p(p + 1)/ unique variances and covariances], let X i = (x i1,..., x ip ) for i = 1,..., n be a sample from X = (x 1,..., x p ), with sample mean vector X, sample covariance matrix S, population mean vector, and population covariance matrix 0. Then, a covariance structure model represents the elements of 0 as functions of q free model parameters in vector, with null hypothesis H 0 : 0 = ( ). An hypothesized model may be fit to a p p sample covariance matrix, and for any vector of model parameter estimates ( û ) the hypothesized model can be used to evaluate the model-implied covariance matrix, ( û ) = Ŝ. The goal in parameter estimation is to obtain a vector of parameter estimates such that a function of the discrepancy between Ŝ and S is minimized. The maximum likelihood (ML) function is the most commonly employed discrepancy function and is defined as: -1 (1) ˆF ML = ln Ŝ ln S + tr(s Ŝ ) p, with associated test statistic () T ML = (n 1) ˆF ML, asymptotically distributed as a central with p* q degrees of freedom (df) under multivariate normality and when H 0 is true (see, e.g., Hayduk, 1987). With respect to issues of sample size and normality in SEM, empirical research has generated a large body of literature and an understanding of the behavior of ML estimation (and its associated test statistic) with realistic data forms (Bentler & Yuan, 1999; Boomsma, 198, 1985; Curran, West, & Finch, 1996; Fouladi, 1998, 1999, 000; Gerbing & Anderson, 1985; Hu & Bentler, 199, 1995; Hu, Bentler, & Kano, 199; Tanaka, 1984, 1987; Yuan & Bentler, 1997, 1998, 1999). Key findings indicate that at small sample sizes: (a) rates of non-convergence and/or improper solutions can be high; (b) parameter estimates exhibit only marginal bias; (c) parameter standard errors become attenuated; (d) T ML, and hence Type I error rates based on T ML, become 440 MULTIVARIATE BEHAVIORAL RESEARCH

3 inflated; and (e) nonnormality exacerbates inflation in Type I error rates based on T ML. Findings also commonly suggest that sample size adequacy is best measured by the ratio of subjects-to-estimated parameters (n: q) rather than by sample size in an absolute sense. To combat distortion in the model test statistic, two fundamentally divergent approaches have emerged and are examined in this investigation: abandoning ML for distribution-free estimation methods, and using ML for parameter estimation but then adjusting the test statistic to account for the effects of sample size and nonnormality. Each of these approaches is discussed in turn below. 1 Browne s (198, 1984) asymptotically distribution-free (ADF) method estimates model parameters by minimizing () ˆF ADF = (s ŝ ) 1 Ĝ (s ŝ ), where s = vech(s) (i.e., a p* 1 column vector of the unique elements of S), where ŝ = vech( Ŝ ), and where (4) =Cov{vech[(X )(X ) ]}, is a symmetric p* p* population fourth-order moment weight matrix. Browne (198) proposed estimating using only the sample data [requiring estimation of p*(p* + 1)/ unique elements in the weight matrix]. Let (5) Y i = vech[(x i X )(X i X ) ]. Then an estimator for is (6) Ĝ = S Y = Cov(Y i ), the sample covariance matrix of Y i. Under the null hypothesis, the associated test statistic (7) T ADF = (n 1) ˆF ADF, asymptotically follows a central distribution with p* q df. Monte Carlo experiments have demonstrated that at large sample sizes (e.g., n 5,000) T ADF yields Type I error rates at the nominal level (Chou, Bentler, & Satorra, 1 Bootstrap resampling is another approach for obtaining robust statistics in SEM. While bootstrapping appears to be a viable alternative under nonnormal data conditions, it is not directly intended to address the problem of small samples and thus was not pursued in this investigation. MULTIVARIATE BEHAVIORAL RESEARCH 441

4 1991; Curran et al., 1996; Hu et al, 199; Muthén & Kaplan, 199; Tanaka, 1984). However, with large models and small to moderate sample sizes ADF is problematic, leading to high rates of non-convergence and improper solutions, and to inflated Type I error rates associated with inflated T ADF values when models do converge. ADF s ability to yield a correct test statistic under nonnormal conditions at large sample sizes inspired Yuan and Bentler (1997, 1999) to develop corrections to T ADF for small sample sizes. They noted that in the regression literature cross products of model residuals are often used when estimating asymptotic covariances, and proposed an estimator for the weight matrix in ˆF ADF as n (8) Gˆ = [ 1/ ( n 1 )] ( ˆ)( ˆ i i ) Y s Y s. i = 1 Incorporating this estimator into ˆF ADF is tantamount to rescaling the original ADF test statistic such that (9) T ADF1 = [1/{1 + [T ADF /(n 1)]}]T ADF, which follows a central distribution with p* q df (when H 0 is true). Further motivated to improve small sample performance associated with T ADF, Yuan and Bentler (1999) proposed another modification to the ADF test statistic, appealing to Fisher s F distribution. They offered a transformation of T ADF based upon the logic of the transformation applied to Hotelling s T statistic in MANOVA (see Stevens, 1996, p. 155, for a review of the T statistic). Observing that T is a quadratic form, similar in structure to the ADF fit function, they proposed rescaling T ADF to an F-distributed statistic, (10) T ADF = [n (p* q)]/[(n 1)(p* q)]t ADF, with numerator and denominator df of p* q and n (p* q), respectively. Yuan and Bentler (1999) used Monte Carlo simulation to investigate the small sample performance of T ADF, T ADF1, and T ADF, systematically varying distributional forms and sample sizes. They reported that over the range of sample sizes investigated T ADF1 and T ADF maintained adequate control of Type I error rates as compared to T ADF, and yielded adequate power (for rejecting misspecified models) with both normal and nonnormal data. Inspired by these limited, yet promising, results, the adjusted ADF test statistics have been incorporated into recent releases of SEM software: T ADF1 is currently available in EQS 5.7 (Bentler, 1996), and the F-distributed statistic T ADF will be incorporated into EQS MULTIVARIATE BEHAVIORAL RESEARCH

5 Another strategy to control for nonnormality and potentially small samples is to estimate model parameters using ML and then assess datamodel fit using a test statistic that has been corrected. These corrections take two basic forms: (a) adjusting T ML for sample size and nonnormality, and (b) constructing distribution-free test statistics based on ML-estimated model parameters. Both approaches are reviewed as follows. The most well known corrections to T ML were developed by Satorra and Bentler (1988, 1994), with two modifications to T ML (generically referred to as SB statistics here) that make adjustments based on the degree of nonnormality in the sample data. Define ŝ& as the p* q matrix of partial derivatives of the p* elements in ŝ with respect to the q model parameters (i.e., the Jacobian matrix), evaluated at the final model parameter estimates. Let W be the symmetric p* p* matrix of unique fourth-order moments 1 1 obtained by ˆ ˆ S S, and let Uˆ = W ˆ ˆ ˆ ˆ Wss & & Ws& s& W, (11) ( ) 1 be the residual weight matrix of those inverted fourth-order moments. Then, (1) = ( * )/ ( ˆ ) TSB1 p q tr US Y TML. The asymptotic distribution of T SB1 is generally unknown; however, when H 0 is true its first moment matches a central distribution with p* q df. The T SB statistic adjusts the model df to yield (1) d tr( ˆ ) / tr( ˆ ˆ ) = USY USYUS Y, for the model df. Then as with T SB1, T ML is scaled as (14) / ( ˆ ) TSB = d tr US Y TML. Like T SB1, the asymptotic distribution of T SB is generally unknown; however, both its first and second moments match that of a central distribution with d df (when H 0 is true). Simulation research has indicated that both SB adjustments to T ML can be effective in controlling Type I error rates under some experimental conditions (Chou et al., 1991; Curran et al., 1996; Fouladi, 1998, 1999, 000; Hu et al., 199; Nevitt & Hancock, 001; Yuan & Bentler, 1998). The scaled T SB1 is currently available in EQS 5.7 (Bentler, 1996) and LISREL 8. (Jöreskog & Sörbom, 1996), while variants of T SB1 and T SB have been incorporated into Mplus 1.0 (Muthén & Muthén, 1998). MULTIVARIATE BEHAVIORAL RESEARCH 44

6 Fouladi (1998, 1999, 000) proposed another correction to T ML, combining a small-sample correction developed by Bartlett (1950) with the SB corrections for nonnormality. Within the context of exploratory factor analysis, Bartlett (1950) offered a k-factor correction to the ML test statistic (where k represents the number of latent factors in the model), suggesting that at small sample sizes (15) T = (n p/ k/ 11/6) ˆF ML, more closely follows a central distribution (with p* q df) than the usual T ML statistic. This adjusted statistic is equivalent to applying a multiplicative correction to T ML (or to any test statistic) of the form (16) c = 1 [(p + 4k + 5)/6(n 1)]. Fouladi (1998, 000) applied this correction factor to T ML to improve its small sample performance with normal data. She also investigated the k-factor correction applied to T SB1 and T SB with nonnormal data and reported that this scaling correction can be effective in controlling Type I error rates under some experimental conditions (Fouladi, 1999). Interestingly, for all three studies she only examined the measured variable path analysis case which has no latent factors (i.e., k = 0). While ADF estimation and T ML scaling adjustments appear to provide some protection against distortion in data-model fit statistics when modeling nonnormal data at small sample sizes, both approaches demonstrate either practical or theoretical limitations. For example, ADF estimation can be carried out only when sample sizes are above an absolute lower bound of n = p* (see, e.g., Bentler & Yuan, 1999). With respect to T ML scaling, the SB adjustments to T ML have been criticized for lacking known asymptotic results with nonnormal data; the k-factor correction as proposed by Fouladi (1998, 1999, 000) is mostly uninvestigated. To address the potential shortcomings of ADF estimation and T ML scaling adjustments, Yuan and Bentler (1998) turned to a mostly ignored residual-based test statistic offered by Browne (1984), noting that there exists a p* (p* q) matrix s& ˆ c whose columns are orthogonal to ŝ& (and to each other). Then the residual-based test statistic (asymptotically distributed as a central with p* q df when H 0 is true) proposed by Browne (1984) is (17) T B ˆ ˆ ˆ 1 ˆ Y = ( n 1) eˆ s& ( ) ˆ c s& cs s& c s& ce, 444 MULTIVARIATE BEHAVIORAL RESEARCH

7 where eˆ = s s ˆ is a p* 1 column vector of residual variances and covariances. Yuan and Bentler (1998) contended that T B could be applied to any consistent estimator for, even when data are not normally distributed, and argued that ML is a prudent choice. The T B test statistic is available to practitioners within LISREL 8. (Jöreskog & Sörbom, 1996). The advantage to T B is that it is an asymptotically distribution-free test statistic with theoretical elegance not found with T SB1, T SB, or the k-factor corrected test statistics in that it follows a known sampling distribution without assuming multivariate normality of the data. Moreover, Bentler and Yuan (1999) indicated that T B has smaller minimum sample size requirements than T ADF with a lower bound of n = df + 1. Unfortunately, these authors found that T B has a propensity to over-reject true models at n < 5000, thus showing Type I error performance remarkably similar to that of the unadjusted T ADF test statistic. The poor performance of T B at small samples led Yuan and Bentler (1998) to propose corrections to the test statistic that are analogous to the small sample corrections to adjust T ADF. First, they proposed replacing the fourth-order moment matrix in T B (i.e., S Y ) with the improved weight matrix given in Equation 8. Incorporating this estimator into T B is equivalent to rescaling it such that (18) T B1 = {1/[1 + (nt B )/(n 1) ]}T B, with T B1 following a central distribution with p* q df when H 0 is true. Yuan and Bentler (1998) also noted that the equation for T B takes on a quadratic form also reminiscent of Hotelling s T statistic. They proposed rescaling T B to an F-distributed statistic, (19) T B = {[n (p* q)]/[(n 1)(p* q)]}t B, with numerator and denominator df of p* q and n (p* q), respectively. The residual-based test statistics will be incorporated into EQS 6.0. Yuan and Bentler (1998) examined the residual-based statistics in a small Monte Carlo experiment, varying combinations of distributional form and sample size. T B showed a strong tendency to over-reject true models at all but the largest sample size with the F-distributed T B demonstrating the best overall performance. The strong performance of T B evidenced by Yuan and Bentler (1998) at the smallest sample size investigated (n = 150, n:q = 4.5:1) led Bentler and Yuan (1999) to conduct an experiment designed to push the test statistics to their sample size lower bounds. Some important findings from their study are: (a) consistent with previous investigations, T ML yielded MULTIVARIATE BEHAVIORAL RESEARCH 445

8 inflated Type I error rates with normally distributed data at reduced sample sizes (even at their largest sample size of n = 10 yielding n:q =.6:1); (b) T B and T B1 performed extremely poorly under every condition T B yielded 100% model rejections while T B1 yielded 0% model rejections; (c) T SB1 was not competitive under the study conditions, yielding inflated Type I error rates ranging from 10-7%; and, (d) T B performed quite well over the range of experimental conditions. Based on these findings, Bentler and Yuan recommended that practitioners turn to the T B F-distributed statistic regardless of distributional form when sample sizes are at least n df + 1. The Current Study While the studies reviewed above yielded promising results, there is clearly the need to characterize more fully the nature of small sample test statistics in SEM. First, no studies have investigated the k-factor correction with latent variable systems (for which k > 0), and thus it is unknown if the improvement afforded to T ML for normal data and the improvements to T SB1 and T SB with nonnormal data using the k-factor multiplier will hold under such models. If an improvement with latent variable models is realized using this correction, then determining a lower bound on sample size requirements for the effective use of the k-factor correction could yield improved recommendations to practitioners. Second, much remains to be learned regarding the behavior of the ADF and residual-based adjusted test statistics, T ADF1, T ADF, and T B1, T B. Bentler and Yuan (1999) noted that much of the characterization of these test statistics focuses only on a specialized model (a three-factor confirmatory factor model) under a very narrow range of conditions. They called for continued research, investigating these test statistics under a greater variety of conditions, including variations in the number of variables in the system and the type of model examined. Third, for sample sizes in which ADF estimation is possible, no research exists that directly compares the relative performance of T ADF1 and T ADF against their respective residual-based analogs, T B1 and T B, leaving practitioners with no clear choice among the ADF-based and residual-based ML test statistics. This current investigation seeks to address these important issues, contributing to our understanding of small sample test statistics in SEM. 446 MULTIVARIATE BEHAVIORAL RESEARCH

9 Method J. Nevitt and G. Hancock Test Statistics Examined This study examined test statistics for evaluating global fit in SEM that, based on previous investigations, have shown promise for practitioner use under commonly encountered data conditions. The following statistics were investigated: T SB1, T SB, T ML k, T SB1 k, T SB k (a k suffix denotes the k-factor correction was applied), T ADF1, T ADF, T B1, and T B. The uncorrected T ML, T ADF, and T B test statistics were also analyzed under all conditions (given sample size constraints), providing benchmarks against which to judge their adjusted forms. Population Models Two population models were developed from which simulated samples of data were drawn. Population Model A (Figure 1) is a five-factor latent variable path (LVP) model with three indicators per factor; Population Model Figure 1 Latent Variable Path (LVP) Population Model A MULTIVARIATE BEHAVIORAL RESEARCH 447

10 B (Figure ) is a seven-factor confirmatory factor analysis (CFA) model, also with three indicators per factor. We chose models that would replicate previous methodological findings as well as extend knowledge to different model types. Model B includes features that have not been examined in previous methodological investigations a substantially larger model requiring large numbers of parameters to be estimated, and a model with correlated error terms, two aspects that are commonly found in applied modeling scenarios. We established non-zero error covariances among all residuals within three groups: among all factors first indicators, among all factors second indicators, among all factors third indicators. This yielded 6 non-zero error covariances, each set to a population value of 0.1. Three additional non-zero error covariances, also set to 0.1, were included between residuals 1 and, between 10 and 11, and between 19 and 0. Thus, a total of 66 non-zero error covariances were established in Population Model B. This population model reflects applied scenarios such as when constructs have indicators that are parallel questionnaire items differing only in content, such as those assessing self concept in multiple subject matter domains (e.g., mathematics, science, and reading). Figure Confirmatory Factor Analysis (CFA) Population Model B 448 MULTIVARIATE BEHAVIORAL RESEARCH

11 Model Specifications for Type I Error Analyses J. Nevitt and G. Hancock When fitting simulated samples of data drawn from each population, model specifications (all properly specified) were implemented that examined both constrained and unconstrained models. Again, we incorporated design elements that would replicate previous simulation studies, and examine previously unanalyzed model specifications found in applied modeling scenarios. Complete details for model specifications are given in Appendix A (using LISREL notation). Specifications A1 and B1 were unconstrained versions of Models A and B, respectively, which required larger numbers of parameters to be estimated. Specifications A and B had loading and error variance constraints imposed on the measurement portion of Models A and B, respectively, and thus required fewer parameters to be estimated (see Appendix A). Such constraints are similar to those found in longitudinal models in which the same constructs are measured at multiple time points. When the equality forced by these constraints holds true in the populations, as is the case in the current simulations, then these constrained models are properly specified as well (see, e.g., Yuan & Bentler, 1998). Table 1 summarizes, for each model specification, p, q, and df, and sample size information (discussed in the next section). Table 1 Simulation Conditions for Properly Specified Models Sample Size Condition Model p p* q df 1:1 :1 5:1 10:1 df + 1 p* A b 50 b 86 a 10 b A b 10 a 10 b B a 58 b 645 b 190 b 10 a 1 b B b 197 a 1 b Note. Sample size conditions reflect subject-to-estimated parameter ratios, with the exception of the last two columns which are the sample size minima for the residual-based and ADF test statistics, respectively. a indicates sample size sufficiently large to compute residual-based test statistics. b indicates sample size sufficiently large to compute residual-based and ADF test statistics. MULTIVARIATE BEHAVIORAL RESEARCH 449

12 Sample Size Conditions For all four models, ratios of n:q = 1:1, :1, 5:1, and 10:1 were examined. In addition each model was also inspected under a sample size condition at which the residual-based and ADF test statistics are at their respective minima. Thus, each model was examined under a total of six sample size conditions. Test statistics based on T ML are not limited to any sample size constraints per se and were examined across all six conditions. The ADF and residual-based test statistics were only examined under sample size conditions that were sufficiently large to support these methods. Table 1 presents each combination of model and sample size condition and the associated absolute sample size for that cell in the design of the experiment. Data Generation and Modeling; Computation and Verification of Test Statistics Three multivariate distributions were examined, each established through the manipulation of univariate skew and kurtosis of the measured variables. All manifest variables were drawn from the same univariate distribution for each data condition. Distribution 1 is multivariate normal with univariate skew and kurtosis both set equal to 0. (Note that normality is defined here, as is commonly done, by using a shifted kurtosis value of 0 rather than a value of.) Distribution represents an elliptical distribution data are nonnormal but symmetric with univariate skew of 0 and kurtosis of 6.0. Distribution is nonnormal and asymmetric with univariate skew of.0 and kurtosis of 1.0. Simulated data matrices were generated in GAUSS (Aptech Systems, 1996) using the programming described by Nevitt and Hancock (1999) that follows the algorithm developed by Vale and Maurelli (198). All modeling of simulated data was performed using EQS 5.7b (Bentler, 1996). The T ML and T ADF test statistics were captured directly from EQS, as were the Jacobian matrix of partial derivatives and the residual covariance matrix. All other test statistics were calculated using GAUSS. The duplication matrix requisite for calculating the SB test statistics was constructed as described by Magnus and Neudecker (1986, pp ); the algorithm for obtaining the Jacobian complement requisite for the residualbased test statistics followed Gill, Murray, and Wright (1981, pp. 7-40, 16-16). The fourth-order moment weight matrix, S Y, requisite for both the SB and the residual-based test statistics, was constructed from the unbiased sample variances and covariances of the Y i data (i.e., an n 1 divisor was used when computing the variances and covariances). 450 MULTIVARIATE BEHAVIORAL RESEARCH

13 Because most of the test statistics in the present investigation were constructed outside of EQS, careful steps were taken to verify the accuracy of intermediate matrices and test statistics. The (biased) weight matrix S Y (before computing the unbiased S Y with n 1 as divisor), and the T SB1 and T ADF1 test statistics were verified using EQS 5.7b (Bentler, 1996). Accuracy of the residual-based test statistic T B was verified using LISREL 8. (Jöreskog & Sörbom, 1996); T ADF statistics were matched with output from a pre-release of EQS 6.0. Design and Execution For the investigation of Type I error rates, the study fully crossed four model specifications with six sample size conditions with three distributional forms, yielding 7 between cells. For each cell, independent data sets were drawn from the associated population model, distributional form, and sample size, and fit to the appropriate model specification using EQS. The same simulated data sets were not used to fit different model specifications (i.e., across Models A1 and A, or B1 and B) even though they might have been drawn from the same population, distributional form, and sample size. Such an approach was adopted to prevent any potential dependencies among the results across Models A1 and A, or across Models B1 and B. The number of iterations to convergence was set at 00 for ML estimation; EQS 5.7b (Bentler, 1996) limits the maximum number of iterations for ADF estimation to 0. Simulated data samples were subjected to parameter estimation using ML and ADF to calculate the test statistics. Start values were assigned using the true population parameter values. For each data set all test statistics were computed (given adequate n). If a given estimation method failed, or a test statistic could not be computed (i.e., a matrix was not invertible), then only those test statistics associated with that sample of data that failed were discarded and replaced, not the sample of data itself. Other test statistics that could be constructed from that sample of data were used in the replication count. The SB and residual-based test statistics were flagged as failing (and discarded) whenever their necessary product matrices could not be inverted. Additionally, the residual-based test statistics were flagged and discarded whenever T B exceeded a clearly inappropriate value of 900,000. For each cell in the design we obtained 000 successful replications; this number of replications yields high power for detecting a departure from Bradley s (1978) liberal criterion of robustness at the =.05 level of significance (Robey & Barcikowski, 199), as described in the next section. MULTIVARIATE BEHAVIORAL RESEARCH 451

14 Summarizing Results and Defining Test Statistic Robustness Study results were analyzed in terms of empirical Type I error rates across each cell s 000 replicates. As Yuan and Bentler (1998, 1999) noted, although adherence to the referenced sampling distribution is important, the primary concern for hypothesis testing is with respect to the tail-behavior of the test statistic (i.e., in terms of model rejection rates). As such, the percentage of model rejections across each cell s replications is reported for each study condition and is used to gauge test statistic robustness. All hypothesis testing was performed at the.05 level. Type I error robustness is evaluated using both an expected 95% confidence interval (CI) about =.05 and Bradley s liberal criterion (Bradley, 1978). Given the 000 replications per condition in this investigation and =.05, the 95% CI is [ (.05.95/000) 1/ ] * 100% = (4.04%, 5.96%); the robustness interval corresponding to Bradley s liberal criterion is (.5, 1.5 ) * 100% = (.5%, 7.5%). Power Analysis While the Type I error propensity of the test statistics investigated is the central focus in our study, the power of a test statistic to reject an improperly specified model also is considered. This investigation takes a two-step approach to evaluating test statistic utility. In the first step, Type I error propensity is evaluated. Using Bradley s (1978) liberal criterion, test statistics under study conditions that yielded empirical Type I error rates above the upper bound of 7.5% were eliminated from subsequent power analyses under those same conditions. Such screening is deemed reasonable because liberal Type I error rates are indicative of inflated test statistics, which in turn would be expected under nonnull conditions to precipitate power estimates that are artificially inflated and thus not comparable with power estimates from test statistics that do maintain reasonable Type I error control. Test statistics with model rejection rates within the robustness interval, or below the lower bound of.5%, were examined in a follow-up power analysis. Test statistics that yielded empirical Type I error rates below the lower bound of robustness were retained for power analysis, rather than being eliminated, because these test statistics (albeit at a seeming disadvantage) could potentially yield acceptable power with respect to rejecting misspecified models. The objective of the power analyses is to provide secondary information regarding the relative performance of the test statistics in this study, allowing test statistics to be compared against one another. These analyses are not meant to estimate the power of the test statistics investigated in any absolute sense. 45 MULTIVARIATE BEHAVIORAL RESEARCH

15 For each test statistic that yielded an observed Type I error rate that was within or below the robustness interval, a new series of 1000 replicates was generated using the same population model for that cell in the study, and at the same distributional form and sample size condition. These new data sets were then fit to an improperly specified model. Sample data drawn from the fivefactor LVP population model were fit to a three-factor LVP model; sample data drawn from the seven-factor CFA population model were fit to a fourfactor CFA model. Complete model (mis)specifications are given in Appendix A. These model misspecifications are reflective of situations in which models are under-parameterized, an aspect common in applied modeling scenarios. The relative degree of noncentrality across all misspecified models (under multivariate normality) was fairly consistent, as indicated by the population root mean-square error of approximation (ε) values (see, e.g., Browne & Cudeck, 199). Specifically, fitting misspecified models to the population covariance matrices yielded a nonzero F ML for each model, from which ε = (F ML /df) 1/ is determined: ε A1 =.144, ε A =.145, ε B1 =.117, and ε B =.1. Because the misspecified models did not perfectly match in model df against their companion properly specified models it was not possible to maintain the exact same absolute sample sizes and n:q ratios in the cells of the power analysis as compared to the Type I error analysis. Thus, the absolute sample sizes examined in the Type I error analysis were preserved in the power analysis (rather than the n:q ratios). Results Non-convergence, Improper Solutions, and Test Statistic Failure Rates Rates of non-convergence and improper solutions for the ML and ADF estimation methods, as well as failure rates for the SB and residual-based test statistics, were tracked for all cells in the Type I error portion of the study. Improper solution and test statistic failure rates are based only on those data sets that yielded converged solutions. Non-convergence, improper solutions, and test statistic failure rates are summarized here briefly; complete tables are given in Appendix B. Overall, for the properly specified models, non-convergence and improper solutions for ML and ADF were a function of distributional form and sample size, with increasing departure from multivariate normality and decreasing sample size systematically leading to increasing rates of non-convergence and improper solutions. While population ε values may be used to determine power to reject misspecified models (e.g., MacCallum, Browne, & Sugawara, 1996), such power determinations are not reported here as they only apply under multivariate normality and with normal theory estimators. MULTIVARIATE BEHAVIORAL RESEARCH 45

16 For ML estimation, non-convergence was rare with the highest rate at %. With respect to improper solutions, ML was problematic only at n:q 1:1, with rates as high as 6%. At all larger n:q ratios, improper solution rates for ML estimation were usually not more than 0%. Unlike ML, ADF estimation frequently yielded high rates of model non-convergence, most particularly at the sample size lower bound n = p* where non-convergence rates ranged from 0-40%. At n > p* ADF yielded non-convergence rates around 1-15%. ADF was also more problematic than ML estimation, with rates of improper solutions that ranged from 1-87% at n = p*, were as high as % at n:q = 5:1, and were up to 1% for some cells at n:q = 10:1. With respect to test statistics, the SB test statistics never failed, delivering 0% failure rates across the board (i.e., the triple-product matrix requisite for computing the SB-based test statistics was successfully inverted every time). The T B test statistic also never failed with respect to inversion of its necessary product matrix; however, there were replicates at the lower bound n = df + 1 sample size in which computed values of T B exceeded a cut-off value of 900,000 and were discarded and replaced. Such failure rates ranged from 7-44% and appeared to be somewhat model-dependent, with unconstrained models showing minimal failure (7-9%) while constrained models tended to yield aberrant values for T B more frequently (10-44%). Type I Error Rates Results for the unconstrained LVP model (A1) are given in Table. With normal data at small sample size conditions T ML delivered inflated model rejection rates that monotonically decreased with increasing sample size, as expected. Using Bradley s (1978) liberal criterion, T ML was robust only at n:q = 10:1. The Bartlett k-factor correction to T ML showed improvement with normal data with Type I error rates within the robustness interval at sample size conditions of n:q 5:1. With nonnormal data, one finds in Table (as expected) that Type I error rates associated with T ML and T ML k are inflated, although the k-factor correction to T ML did provide some improvement when data were symmetric nonnormal. The first SB test statistic, T SB1, provided some control over Type I error rates, demonstrating performance with asymmetric nonnormal data that was noticeably superior to T ML. However, Type I error rates fell within the robustness range only at the largest sample size condition examined. Applying the k-factor correction to T SB1 yielded a test statistic that was robust at all combinations of sample sizes and distributional forms investigated, with model rejection rates even within the 95% CI for most sample size ratios above n:q = :1. In contrast to the inflated rejection rates 454 MULTIVARIATE BEHAVIORAL RESEARCH

17 Table Type I Error Rates (%) for Model A1 (85 df) J. Nevitt and G. Hancock condition: 1:1 :1 5:1 10:1 df + 1 p* n: D a T ML b.9 a.9 a.1 a T ML k b a 5.5 b 6.0 b a T SB a a.8 a 5.1 b 4. b 4. b.7 a T SB1 k.7 a 4.8 b 4.8 b 4.4 b 4.8 b 4.6 b 5.1 b 6.8 a 6.0 b 5.4 b 6.7 a 6.8 a a 4.5 b.7 a.8 a.9 a T SB 5. b a a T SB k T B b T B a T B 6. a 5.7 b a.9 a.0 a a T ADF b 0.0 T ADF a a T ADF b Note. Type I error rates are computed across 000 successful replications at the.05 level of significance. D = distributional form; D = 1, multivariate normal; D =, elliptically symmetric nonnormal; D =, asymmetric nonnormal. a indicates rejection rate within the robustness interval. b indicates rejection rate within the 95% CI. MULTIVARIATE BEHAVIORAL RESEARCH 455

18 associated with T SB1, model rejection rates for the second SB test statistic, T SB, tended to be attenuated. Applying the k-factor correction to T SB pushed model rejection rates even lower to near zero under most of the conditions examined. For the residual-based and ADF test statistics in Table, performance of the unadjusted T B and T ADF mirrored one another closely, with near 100% Type I error rates at all sample sizes and distributional forms investigated. The adjustments T B1 and T ADF1 also were close in performance to one another with near-zero rejection rates at n:q < 10:1 and showing monotonically decreasing Type I error rates with increasing departure from multivariate normality. The F-distributed adjustments, T B and T ADF, mirrored one another in performance at the test statistics minimum sample sizes with Type I error rates at or near zero; at larger sample size conditions T B generally maintained robustness while T ADF yielded suppressed model rejection rates. Type I error rates under the constrained LVP model (A) are shown in Table. Overall test statistic performance was similar to that seen under the unconstrained model, with the caveat that performance associated with all of the test statistics examined under the constrained model deteriorated to some extent. Applying the Bartlett k-factor correction to these test statistics led to improved Type I error performance in the constrained model, although at the smallest sample size conditions T ML k and T SB1 k exerted too much correction yielding Type I error rates below the.5% lower bound of the robustness criterion. The behavior of T SB and T SB k was again similar to the unconstrained model; the correction pushed Type I error rates to zero or near-zero values (in particular with nonnormal distributional forms). The T B and T ADF test statistics, and their adjusted forms, paralleled one another with respect to Type I error propensities in the constrained model. Table shows 100% model rejection rates for T B and T ADF, with zero or near-zero Type I error rates for all of the adjusted forms of T B and T ADF at their minimum sample size conditions. At other sample size conditions, model rejection rates were less than % for the -distributed T B1 and T ADF1 ; Type I error rates for the F-distributed T B and T ADF were within or slightly above the robustness interval. Table 4 presents Type I error results for the unconstrained CFA model (B1). With multivariate normal data, T ML was robust at n:q = 5:1 and 10:1; at smaller n:q ratios model rejection rates were inflated. The k-factor corrected form T ML k maintained Type I error rates (with normal data) to within the robustness interval at sample size conditions of n:q = 1:1 and larger. T ML and T ML k became inflated with nonnormal distributional forms with Type I error rates climbing as high as 98%. 456 MULTIVARIATE BEHAVIORAL RESEARCH

19 Table Type I Error Rates (%) for Model A (10 df) J. Nevitt and G. Hancock condition: 1:1 :1 5:1 10:1 df + 1 p* n: D T ML a 4.5 b.6 a. a T ML k T SB a.7 a 5.5 b 4.7 b 4.5 b T SB1 k b 5.7 b 4.8 b 5.6 b. 4.4 b 6.7 a a.6 a.6 a. a.8 a T SB T SB k T B T B T B T ADF T ADF T ADF 7. a b 0.7 Note: Type I error rates are computed across 000 successful replications at the.05 level of significance. D = distributional form; D = 1, multivariate normal; D =, elliptically symmetric nonnormal; D =, asymmetric nonnormal. a indicates rejection rate within the robustness interval. b indicates rejection rate within the 95% CI. MULTIVARIATE BEHAVIORAL RESEARCH 457

20 Table 4 Type I Error Rates (%) for Model B1 (10 df) condition: 1:1 :1 5:1 10:1 df + 1 p* n: D a 6. a T ML a 4. b 4.7 b 5.4 b a T ML k 7.0 a b a 6. a T SB a 5.6 b a a 4.8 b 4.8 b 5.6 b.4.9 a T SB1 k 5.1 b 4. b 5.1 b 4.4 b.6 a 5.4 b a 6.0 b b 5.4 b 4.8 b 5.6 b 4.1 b 4.4 b T SB a.8 a a 4.7 b T SB k a T B b 4.6 b T B a.4 a a 6.4 a 5.8 b 5.5 b a T B 4.1 b 4. b 4.1 b.7 a b 6.5 a.9 a. a.1 a a T ADF b 4.5 b 0.5 T ADF a a 1.4 a 5.8 b 5.5 b.1 T ADF a.5 a Note. Type I error rates are computed across 000 successful replications at the.05 level of significance. D = distributional form; D = 1, multivariate normal; D =, elliptically symmetric nonnormal; D =, asymmetric nonnormal. a indicates rejection rate within the robustness interval. b indicates rejection rate within the 95% CI. 458 MULTIVARIATE BEHAVIORAL RESEARCH

21 T SB1 showed robustness with most distributional forms at n:q = 5:1 and 10:1 (with the exception of n:q = 5:1 and asymmetric nonnormal data), but yielded inflated Type I error rates at smaller n:q ratios. The k-factor corrected form, T SB1 k, maintained Type I error rates to within the robustness interval at sample size conditions of n:q = 1:1 and larger, although error rates exceeded the upper bound of the robustness interval with asymmetric nonnormal data and n:q < 5:1. The T SB test statistic showed robustness with normal data, but then yielded attenuated model rejection rates with nonnormal distributional forms; the k-factor adjusted T SB k over-corrected, delivering Type I error rates near zero. Model rejection rates for T B and T ADF, T B1 and T ADF1, and T B and T ADF again mostly mirrored one another, with some local distinctions. The unadjusted test statistics yielded inflated Type I error rates that monotonically decreased with increasing sample size; note that even at the n:q = 10:1 ratio model rejection percentages associated with T B and T ADF were at 18%. The adjusted forms of T B and T ADF performed optimally under several of the sample size conditions investigated but suppressed model rejections with increasing nonnormality and decreasing sample size. The residual-based F- distributed statistic, T B, maintained robustness under all conditions for this model, with the exception of the n = df + 1 sample size minimum for this test statistic in which model rejection rates were at zero. Results for the constrained CFA model (B) are presented in Table 5. T ML produced inflated Type I error rates at every sample size and distributional form combination examined (even at the largest sample size condition and multivariate normal data). T SB1 yielded inflated Type I error rates that decreased with increasing sample size, approaching the 7.5% upper bound of the robustness criterion at the largest sample size conditions. Applying the k-factor correction to these test statistics improved Type I error control for T ML k with normal data and for T SB1 k with all distributional forms, albeit evidencing some degree of over-correction at the smallest sample size conditions. The T SB statistic controlled Type I error rates with normal data at all but the smallest sample size condition; error rates with nonnormal distributional forms were mostly attenuated. Applying the Bartlett k-factor correction, T SB k yielded near-zero model rejection rates for all conditions investigated. Results for T B and T ADF in Table 5 are similar to the patterns seen under the constrained LVP model. The unadjusted forms of the test statistics yielded 100% model rejection rates; the -adjusted T B1 and T ADF1 were robust only at the largest sample size condition and with normal data; the F-distributed statistics T B and T ADF yielded inflated Type I error rates ranging from % across the conditions examined here. MULTIVARIATE BEHAVIORAL RESEARCH 459

22 Table 5 Type I Error Rates (%) for Model B (196 df) condition: 1:1 :1 5:1 10:1 df + 1 p* n: D T ML a. a.9 a T ML k T SB a 4.5 b.9 a.7 a T SB1 k a 5.1 b 4.1 b 5.0 b b 5.6 b 5.6 b 5.5 b b.6 a.1 a.1 a. T SB T SB k T B a T B T B T ADF a 0.0 T ADF T ADF Note. Type I error rates are computed across 000 successful replications at the.05 level of significance. D = distributional form; D = 1, multivariate normal; D =, elliptically symmetric nonnormal; D =, asymmetric nonnormal. a indicates rejection rate within the robustness interval. b indicates rejection rate within the 95% CI. 460 MULTIVARIATE BEHAVIORAL RESEARCH

23 Empirical Power J. Nevitt and G. Hancock Results for follow-up power analyses are given in Tables 6-9, with each table paralleling one of the four models investigated in the Type I error analysis. As noted earlier, empirical power estimates presented in these tables are not considered absolute, but rather are relative estimates for the purpose of comparing performance of the test statistics investigated. Power was not analyzed for the residual-based and ADF test statistics at their respective sample size minima. The high frequency of non-convergence, improper solutions, and test statistic failure rates and the poor Type I error performance associated with these test statistics at their sample size lower bounds logically precluded them from further investigation. A general pattern is evidenced across all of the power tables, whereby power systematically increased with increasing sample size (as would be expected) and uniformly decreased with increasing departure from multivariate normality. Table 6 presents empirical power estimates for the misspecified unconstrained LVP Model. Note that for n = 50 the power estimate associated with every test statistic under all distributional forms was 100%, and as such is not useful for discriminating test statistic performance. At n < 50, however, test statistic performance with respect to power diverged. The Bartlett k-factor correction to T ML, T ML k, yielded empirical power at % with multivariate normal data. The SB test statistics and their k-factorcorrected forms generally yielded power at or above 80%. Of note is the power associated with T SB1 k which delivered model rejection rates near 90% at n = 5. In contrast T SB and T SB k delivered low power at the smallest sample size conditions power fell to about 5% for T SB k at the n = 5 sample size. Empirical power estimates for the residual-based and ADF test statistics in Table 6 suggest some noteworthy features. The residual-based test statistics generally exhibited greater power than the ADF test statistics; the F-distributed forms of T B and T ADF (T B and T ADF ) delivered greater power as compared to the -corrected forms of these test statistics (T B1 and T ADF1 ). Under the misspecified constrained LVP model, Table 7 reports empirical power estimates that are noticeably lower than those found in Table 6. For the T ML k test statistic empirical power was reported at 0% at the smallest sample size condition with multivariate normal data; at all larger sample sizes power was at or near 100%. At n = 18 (with all distributional forms) empirical power associated with T SB1 k dropped off, and went to near-zero for T SB k. Notice the 0% model rejection rate with all distributional forms for T B1 at n = 10. Again, the F-distributed correction to ADF yielded greater power than the - corrected form (e.g., 14% versus 80% with asymmetric nonnormal data). MULTIVARIATE BEHAVIORAL RESEARCH 461