Power of Modified Brown-Forsythe and Mixed-Model Approaches in Split-Plot Designs

Transcription

1 Original Article Power of Modified Brown-Forsythe and Mixed-Model Approaches in Split-Plot Designs Pablo Livacic-Rojas, 1 Guillermo Vallejo, 2 Paula Fernández, 2 and Ellián Tuero-Herrero 2 1 Universidad de Santiago de Chile, Chile 2 Universidad de Oviedo, Spain Abstract: Low precision of the inferences of data analyzed with univariate or multivariate models of the Analysis of Variance (ANOVA) in repeated-measures design is associated to the absence of normality distribution of data, nonspherical covariance structures and free variation of the variance and covariance, the lack of knowledge of the error structure underlying the data, and the wrong choice of covariance structure from different selectors. In this study, levels of statistical power presented the Modified Brown Forsythe (MBF) and two procedures with the Mixed-Model Approaches (the Akaike s Criterion, the Correctly Identified Model [CIM]) are compared. The data were analyzed using Monte Carlo simulation method with the statistical package SAS 9.2, a split-plot design, and considering six manipulated variables. The results show that the procedures exhibit high statistical power levels for within and interactional effects, and moderate and low levels for the between-groups effects under the different conditions analyzed. For the latter, only the Modified Brown Forsythe shows high level of power mainly for groups with 30 cases and Unstructured (UN) and Autoregressive Heterogeneity (ARH) matrices. For this reason, we recommend using this procedure since it exhibits higher levels of power for all effects and does not require a matrix type that underlies the structure of the data. Future research needs to be done in order to compare the power with corrected selectors using single-level and multilevel designs for fixed and random effects. Keywords: statistical power, modified Brown Forsythe, Mixed Linear Model and split-plot designs Difficulties found when testing the hypotheses of a design that has one factor with J-levels for independent groups and a second factor with dependent K-levels include noncompliance with the homogeneity assumption of the variance-covariance matrices (Wilcox, 2012), the lack of independence between values (Liu, Rovine, & Molenaar, 2012), and the relevance of using a statistical model based on the classical linear model or the Mixed Linear Model (MLM) to evaluate the different design effects (Ato, Vallejo, & Palmer, 2013; Stroup, 2013). Liu et al. (2012) suggest that among the procedures for analyzing data using the MLM are repeated-measures Analysis of Variance (ANOVA), covariance pattern models, and growth curve models, which analyze behavioral change considering as an assumption the existence of different patterns of covariance between the residuals of the main effects. Moreover, they point out that when the covariance structures of the data are heterogeneous, the performance of the Akaike Information Criterion (AIC; Akaike, 1974) and the Bayesian Information Criterion (BIC; Schwarz, 1978) has a lower efficiency than expected in correctly selecting the covariance structure, this being observed most clearly with the structure of Random Coefficients (RC). Additionally they indicate that the AIC and BIC select the covariance structure correctly when sample sizes are moderate and these two criteria have a good fit in the selection of the matrix when these are CS, MA (1), and AR (1). As regards the application of ANOVA for repeatedmeasures data, different authors suggest the existence of difficulties in the precision of the inferences for assessing the fixed effects of the design, based on repeated-measures univariate models or multivariate models of joint normality (with a general covariance structure) since they are associated with the presence of nonspherical covariance structures and free variation of the variance and covariance with the cost of estimating a large number of parameters (Kowalchuk, Keselman, Algina, & Wolfinger, 2004; Vallejo & Ato, 2006), unbalanced designs (Keselman, Algina, & Kowalchuk, 2001), moderate sample sizes (Davidson, 1972), low control of Type I error rates, noncompliance of parametric assumptions (Livacic-Rojas, Vallejo, & Fernández, 2010; Vallejo, Ato, Fernández, & Livacic-Rojas, 2013), and a lack of knowledge of the error structure Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22 DOI: / /a000124

2 10 P. Livacic-Rojas et al., Statistical Power, Covariance underlying the data, which, in turn, affect the statistical power of the contrasts in the design hypotheses. As Livacic-Rojas, Vallejo, Fernández, and Tuero-Herrero (2013) pointed out that various studies have evaluated the performance of information criteria based on the likelihood of selecting the most correct repeated-measures model and using the MLM in three different scenarios: its ability to select the correct mean model, the correct covariance structure, and to select both structures simultaneously. In the same context, different studies show that the performance of AIC is the 48% and BIC the 42% in correctly selecting the covariance structure on average, respectively. See too, Vallejo, Fernández, Livacic-Rojas, and Tuero-Herrero (2011a, 2011b). Moreover, they pointed out that AIC selects the 2% for the version of the original covariance structure and the 48% for the heterogeneous version of the same structure. Along the same line, with respect to the Type I error rates, Livacic-Rojas et al. (2013) have pointed out that AIC yields higher Type I error rates (on 7.41% ofanalyzed conditions it exceeds the Bradley s Liberal Criterion) than the Correctly Identified Model (CIM, on 2.47%). AIC shows a performance associated to main and interactional effects, ARH (Autoregressive Heterogeneity) and RC (null and positive types of relation between the sizes of group and type of matrices). In turn, Vallejo and Ato (2006) analyzed the Type I error rates between the Modified Brown Forsythe (MBF) and EGLS (empirical form of the generalized leastsquares method modified to fit Kenward-Roger-based AIC) and pointed out that both procedures were robust to violations of homogeneity of variances and non-normally distributional data on unbalanced designs. In another study, Vallejo, Arnau, and Ato (2007) compared the SAS Proc Mixed (Market Mix Modeling [MMM]) and MBF so as to detect the effects in a split-plot design containing multiple variables when multivariate normality and covariance heterogeneity assumptions were violated. They indicated that these effects are comparable in the frequency of Type I errors. As regards statistical power of different procedures, Vallejo, Fernández, and Livacic-Rojas (2007) evaluated comparatively the effectiveness of MBF and EGLS for interactional effects in repeated-measures designs, and evidenced that none demonstrated a clearly superior performance when data were non-normally distributed and matrices were heterogeneous (RC, ARH1, and UN). Specifically, MBF shows higher levels of statistical power in comparison to EGLS when the vector of means different of zero is associated with the largest variance. By contrast, in most of the analyzed conditions, EGLS shows higher power levels when the nonzero means vector is associated with the smallest variance. In turn, when ELGS was used with AIC, it rarely yields low levels of statistical power. However, EGLS yields lower levels of power when the covariance matrix is specified correctly. Similarly, the authors note that if one could establish a procedure to model the covariance structure (instead of taking one without any structure), MBF would be less efficient than EGLS in situations where the covariance structure has an important role since more parameters are needed for such an estimation. In cases where the covariance matrix is misspecified, ELGS yields higher Type I error rates and wrong inferences. In this case, it is necessary to maintain a compromise between bias and precision (for more information, check Fitzmaurice, Laird, & Ware, 2004; Lix& Lloyd, 2006). On the other hand, Vallejo, Fernández, Herrero, and Livacic-Rojas (2007) compared the sensitivity of MMM and MBF in order to detect the effects of a multivariate design partially repeated measures when data are deviated from the normal distribution and the scattering matrices are heterogeneous. In general terms, the results indicate that none of the approaches proved uniformly most powerful. Specifically, the empirical levels of statistical power averaged were: MMM = and MBF = Successively, interactional effects were: MMM = and MBF = Finally, the interaction of groups and repeated measures were: MMM = and MBF = Based on the aforementioned evidence, the authors conclude that the matrix structure used was not the most favorable for MMM as the results were limited to the conditions examined. They also add that MBF can be inefficient in situations in which some sampling units have incomplete vectors and the shape of the matrix plays an important role in estimating and/or changing when covariates exist. These results lead them to corroborate that in these cases, the MMM is efficient and adequate. Moreover, the researchers recommend keeping the necessary balance between flexibility and parsimony criteria in order to choose the covariance structure or model that best describes the data. In this context, studies carried out by Fitzmaurice et al. (2004) state that an excessively flexible model (e.g., MBF) can produce inefficient estimates, while an excessively parsimonious model (the generalization of the mixed model proposed by Scheffé) can produce biased estimates of the effects corresponding to the structure of means. Along the same line, Stroup (2013) pointed out that if a researcher chooses a wrong model of covariance structure, the misspecification could inflate the Type I error rate (e.g., CS) or reduce the statistical power (e.g., UN), which reduces the chance of identifying the treatment effects. Considering different studies, Vallejo, Arnau, Bono, Fernández, and Tuero-Herrero (2010) note that comparing the effectiveness of AIC and BIC with those of other procedures, selection of the covariance structure improves as its complexity decreases and the sample size increases. Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing

3 P. Livacic-Rojas et al., Statistical Power, Covariance 11 At the same time, Vallejo, Tuero-Herrero, Núñez, and Rosário (2014) indicated that when it comes to the performance of selection criteria based on the Maximum Likelihood (ML) or Restricted ML (REML), REML works the same or better than ML when the researcher selects the mean and covariance structures. To overcome the above limitations, they recommended the use of different informational criteria [Akaike Information Criterion Corrected (AICC), the Consistent Akaike Information Criterion (CAIC), the BIC, and the Deviance Information Criterion (DIC, Spiegelhalter, Best, Carlin, & Van der Linde, 2002)]; however, the appropriate use of the selection criteria is a subject of ongoing debate (see too, Greven & Kneib, 2010; Hamaker, Van Hattum, Kuiper, & Hoijtink, 2011; Srivastava & Kubokawa, 2010; Vaida & Blanchard, 2005). Based on the difficulties for different covariance selector structures such as the wrong choice of covariance structure, the Type I error rates, and the yield of moderate levels of statistical power, this study intents to compare the levels of statistical power presented by three selection criteria, namely the AIC, the CIM, and the MBF procedure, using a split-plot design for between-group effects, within-group effects, and interaction in different data conditions. Based on the aforementioned evidence the mixed model with the AIC was used (instead of another criterion such as the BIC or the CAIC) because, despite selecting the correct model at a low frequency, it is the criterion that exhibits the greatest efficiency. Similarly, the CIM, the procedure that represents the true structure of the data, allows a more realistic comparison of the specific functioning of the different procedures (see Keselman, Algina, Kowalchuk, & Wolfinger, 1998; Livacic-Rojas et al., 2013). And MBF, being a procedure that takes an unstructured matrix (UN), is more realistic in estimating a larger number of parameters with data taken at different points in time and it is more robust to the heterogeneity of the data. With regard to the latter procedure, Vallejo, Fidalgo, and Fernández (2001), Vallejo and Livacic-Rojas (2005), Vallejo et al. (2006), and Vallejo, Arnau, et al. (2007) extended the approximate BF procedure to the univariate and multivariate repeated-measures context, in order to avoid the negative impact that the heterogeneity of covariance matrices has on multivariate test criteria. Although it is believed that the MLM method is generally more powerful than the MBF test, this question should be investigated before researchers adopt the MLM method (see too, Vallejo, Ato, & Valdés, 2008). To date, no such comparison has been undertaken. Description of the Procedures to be Compared in This Study Let y ijk, i = 1,..., n j ; j = 1,..., p; k = 1,..., q, be the response for the ith participant in the jth group at the kth occasion, and let y ij =(y ij1,..., y ijq ) 0 be the random vector of responses for the ith participant in the jth group. Then, by stacking the subvectors y 0 11,..., y 0 np, a multivariate linear model can be written as Y ¼ XB þ E; where Y is an n q matrix of observed data, X is an n p design matrix with full column rank p < n, B is a p q matrix that contains the unknown fixed effects to be estimated from the data, and E is an n q matrix of unknown random errors. We assume that the rows of Y are normally and independently distributed within each level j, with mean vector μ j and variance-covariance matrix Σ j. The unbiased estimators of Σ j are ^Σ j ¼ð1=n j 1ÞE j ; where E j ¼ Y 0 j Y j ^B j X 0 j Y j are distributed independently as Wishart W q (n j 1, Σ j ) and ^B j ¼ðX 0 j X j Þ 1 X 0 j Y j is the maximum likelihood estimator of matrix B j (Nel, 1997). We also assume that n j 1 q, such that ^Σ 1 j exists, j = 1, 2,..., p, with probability one. The hypotheses tested under a multivariate model are linear combinations of rows and columns of B. Mostof the hypotheses of interest for the fixed-effects model can be defined as ð1þ H 0 : C 0 BA ¼ 0 versus the alternative H A : C 0 BA 6¼ 0; ð2þ where C 0 =(I h... 1) isanh p matrix of between-subjects contrasts with full row rank h p, anda =(I m... 1) 0 is a q m matrix of within-subjects contrasts with full column rank m q. It can be readily verified that with the structure defined in (2) it is not possible to test any linear hypothesis concerning the elements of B. Nevertheless, it is indeed possible to define specific contrasts for testing the hypotheses of principal interest. Under the assumption that the errors follow a multivariate normal distribution, the hypotheses of the type formulated in (2) can be tested using any of several standard multivariate tests (see Timm, 2002). MBF Procedure Practical implementation of the MBF procedure requires estimation of the degree of freedom (df) of the approximate central q-dimensional Wishart distribution, which can be easily derived by equating the first two moments (i.e., expectation and dispersion matrix) of the quadratic form associated with mth source of variation in model (1) to those of the central Wishart distribution. A detailed explanation of the multivariate Satterthwaite s approximation can be found in Vallejo et al. (2006). Applying the approach of these authors, the Wilks likelihood ratio criterion for testing the interaction effect is given by the determinant of E*(H + E*) 1,wherethe Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22

4 12 P. Livacic-Rojas et al., Statistical Power, Covariance hypothesis matrix, H, and the error matrix, E*, are determined by and H ¼ðC 0 ^BAÞ 0 ½C 0 ðx 0 XÞ CŠ 1 ðc 0 ^BAÞ; X p E ¼ ν e =ν h c j A0 Σ j A; j¼1 ð3þ ð4þ where ν e and ν h are the approximate df for E* and H, respectively, c j ¼ 1 c j ; and c j ¼ n j =n: Using results due to Nel and Van der Merwe (1986) and Krishnamoorthy and Yu (2004), the approximations to the df can be written as: ν e ¼ ðq 1Þþðq 1Þ 2 P p ; 2 1 n j tr 2 c 1 j A0 ^Σ j^ξ 1 A þ tr c j A0 ^Σ j^ξ 1 A and j¼1 ν h ¼ ðq 1Þþðq 1Þ 2! P p P p fwgþ tr 2 c j A 0 ^Σ j^ξ 1 A þ tr j¼1 j¼1 P p j¼1 c j A 0 ^Σ j^ξ 1 A ð5þ! 2 ; ð6þ where W = [tr 2 ða 0^Σ j^ξ 1 AÞþtrðA 0^Σ j^ξ 1 AÞ 2 Š 2c j ½tr 2 ða 0^Σ j^ξ 1 AÞþtrðA 0 ^Σ j^ξ 1 AÞ 2 Š; Ξ ¼ðc 1 Σ 1 þ...þ c p Σ pþ; and tr() denotes the trace of the matrix. Using the transformation of Wilks s Λ to F-statistic, the usual test of H 0 versus H A in (2) rejects approximately if F MBF ¼ 1 Λ1=s v Λ 1=s 2 F v 1 α ; ðv 1 ; v 2 Þ where S ¼ l 2 v 2 h 4 = 1 l 2 þv 2 h 5 Š 1=2 ; v 1 ¼ lv h and v 2 ¼ v e ðl v h þ 1Þ=2 S ðlv h 2Þ=2. Univariate Linear Mixed Model The linear mixed models are increasingly used in studies of growth and change for fitting and analyzing repeatedmeasures designs. This family of models defined by Laird and Ware (1982) and Jennrich and Schluchter (1986) extends the usual general linear model to cases where standard assumptions of independence and homogeneity are not required. Suppose that model (1) contains fixed as well as random effects, then for the complete response vectors considered in this article, the univariate linear mixedmodelcanbewrittenas y ¼ Xβ þ Zu þ e; ð7þ where y is an nq 1 vector of observed data, X is an nq k fixed design matrix with full column rank k(=1 + p + q + pq) <nq, β is a k 1 vector that contains the unknown fixed effects common to all participants, Z is an nq nh random design matrix with full column rank nh < nq, u is an nh 1 unknown vector of random effects, and e is an nq 1 vector of random errors whose elements need not be independent and homogeneous. For the ith participant in the jth group, it is assumed that the random vectors e i and u i are independently distributed as N(0, Ω j ) and N(0, G j ), respectively. For the combined model these assumptions imply that, marginally, E(y) = XB and Var(y) = V, where V ¼ Varðvec y 0 p Þ¼Z I nj G j Z 0 p þ ðinj Ω j Þ; G j is j¼1 j j¼1 an h h positive-definite matrix, Ω is a q q positivedefinite covariance matrix, is the Kronecker product function, and denotes the matrix direct sum. Both moments of y can be modeled separately and distinctly. For known V, statistical inference about fixed-effects parameters can be obtained by using the generalized least squares (GLS) estimator of β given by and its variance ^β GLS ¼ðX 0 V 1 XÞ X 0 V 1 y; Varð^βÞ ¼ðX 0 V 1 XÞ : ð8þ ð9þ If V is unknown, the EGLS estimator of β ðor ~ β EGLS Þ is obtained by replacing V by its estimate ^V in (8), which is V with its parameters G and Ω replaced by their maximum likelihood estimators. Likewise, the variance of ~ β is usually estimated by replacing V by its estimate ^V in (9). Though a number of estimation strategies are available, the current manuscript uses REML estimation as implemented through SAS Institute s (2005) Proc Mixed Program. In the mixed model, any specific hypothesis of the form H 0 : C 0 β ¼ 0 versus H A : C 0 β 6¼ 0; where C is a matrix of contrasts of rank ν 1 ; can be tested using Wald s F statistic approximation F ¼ ν 1 1 ðc 0 ~ βþ 0 ½C 0 Varð ~ βþcš 1 ðc 0 ~ βþ; ð10þ where the Varð ~ βþ is the covariance matrix of β; which usually underestimates the true sampling variability of ~ β due to not accommodating the variability in the estimate of VarðyÞ when estimating the covariance structure of ~ β and when computing associated Wald-type statistics, particularly when the number of subjects is not sufficiently large to support likelihood-based inference. To circumvent this difficulty, Kenward and Roger (1997) provide a method that involves inflating the conventionally estimated covariance matrix of ~ β; to derive an appropriate F-test statistic by replacing in (10) Varð ~ βþ by an adjusted estimator of the covariance matrix of ~ β; and Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing

5 P. Livacic-Rojas et al., Statistical Power, Covariance 13 Figure 1. Formal description to the type of covariance matrices for generating simulated data. estimating the denominator df for the generalized F-statistic based on it. The procedure to determine the power for a given design (which determines X and Z), covariance matrix, and treatment differences (included within ^βþ can be expressed as Pr½Fðν 1 ; ν 2 ; λþ F 1 α ; ν 1 ; ν 2 ÞjH A Š¼1 β; ð11þ where F(ν 1, ν 2, λ) represents the noncentral F distribution function with numerator df ν 1 = Rank(C), denominator df ν 2 appropriately estimated with the method proposed by Kenward and Roger (1997), the noncentrality parameter λ ¼ðC 0^βÞ 0 ½C 0 Varð^βÞCŠ 1 ðc 0^βÞ; ð12þ and F 1 α ; ν 1, ν 2 is the critical value for testing the null hypothesis at a Type I error of α. For details, see Stroup (2002). More information about the Akaike s Criterion and the Correctly Identified Model is provided in Livacic-Rojas et al. (2013). Method To evaluate the statistical power levels of the AIC, BF, and CIM with the mixed-model approach with KR solution, we conducted a Monte Carlo simulation study using a split-plot design with one between-subjects factor (p = 3) and one within-subjects factor (k = 4) using the SAS 9.1 Proc Mixed program (SAS Institute, 2005). Six variables were manipulated to investigate the performance: (a) the total sample size, (b) the type of covariance matrix, (c) the pairing of group sizes and covariance matrices, (d) the form of the distribution, (e) the trimmed means at 10% and 20%, and (f) three patterns of means (see, e.g., Algina & Keselman, 1997; Keselman et al., 1998). When the designs were balanced, the relationship between group size and dispersion matrix size was null. When the designs were unbalanced, the relationship could be positive (smaller group was associated with the smaller dispersion matrix) or negative (the smaller group was associated with the larger dispersion matrix). For each sample size condition, both a null and a moderate degree of cell size inequality were explored, as indexed by a coefficient of sample size variation (Δ), where h i 2 1=2, Δ ¼ ð1=n Þ Σ j n p n =p n being the average group size. The unequal group sizes were, respectively: (a) 6, 10, 14 (n = 30), (b) 9, 15, 21 (n = 45), and (c) 12, 20, 28 (n = 60). The degree of heterogeneity of the dispersion matrices was Σ 1 ¼ 1 Σ 3 2 and Σ 3 ¼ 5 Σ 3 2: The covariance structures generating the simulated data in the proportional case were: RC, ARH (1), and UN. The value of the sphericity parameter was held constant at 0.75 (see e.g., Keselman et al., 1998]. Figure 1 presents a formal description of these structures for a situation in which we have four measurement occasions. The form of the mean distribution had four levels: normal, slightly skewed, moderately skewed, and severely skewed. The values of the indices of asymmetry (c1) and direction (c2) selected for generating non-normal multivariate distributions were: slightly skewed (c1 = 1, c2 = 0.75), moderately skewed (c1 = 1.75, c2 = 3.00), and severely skewed (c1 = 3.00, c2 = 21.00) (see, e.g., Berkovits, Hancock, & Nevitt, 2000; Micceri, 1989). The truncated mean procedure has been used to deal with the heterogeneity of variance in nested designs with more robust methods according to the recommendations given by Wilcox (2012). Additionally, the permutation of the configuration of the mean vector pattern was selected at maximum range (Algina & Keselman, 1998). According to Ramsey (1978), this entails that the first mean of the vector takes the smallest value, the last takes the biggest value, and the middle two take the average of the previous two. To detect the sensitivity of measurement occasions, the following permutations were included within each of the three design groups: ( 1, 0, 0, 1), ( 1, 0, 1, 0), ( 1, 1, 0, 0). For the analysis of the power levels, a thousand replicated data sets were created with the Interactive Matrix Language (IML) of the SAS (SAS Institute, 2005), using the multivariate extension that Vale and Maurelli (1983) developed from the power method proposed by Fleishman (1978). The classification criteria of the statistical power were made on the basis of the proposals made by Cohen (1992). He specifies that a high level is 0.80 or higher, whereas moderate power levels are and low power levels, Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22

6 14 P. Livacic-Rojas et al., Statistical Power, Covariance Some of the reasons Cohen indicates are lower levels that yield the likelihood of increasing Type II error rates (For further details see also Vallejo, Fernández, Herrero, et al., 2007). Results Normally and Non-Normally Distributed Data Table 1 shows the power levels averaged of the three procedures for normal and non-normal distributions (including skewness and kurtosis with light, moderate, and strong levels). (See, for additional information the Electronic Supplementary Material, ESM 1.) Normally Distributed Data For the Between-Group Effects (BGE), three procedures yield high power levels averaged for all analyzed conditions with only slight differences between them (MBF = 45.39; AIC = 40.55; CIM = 42.30). In specific terms, the procedures yield: For BGE, with n = 30, MBF displays high power levels: 22.22% of the analyzed conditions and ARH and UN matrices (negative relation). AIC and CIM, respectively, show only moderate power levels in the 77.78% and 88.89% range and low levels at 22.22% and 11.11% of the analyzed conditions with RC (positive and negative types of relation). With n = 45, BF and CIM show moderate levels at 77.78% and AIC at 66.67% of the analyzed conditions. The low levels occur with BF at 11.11% (RC and null relation), AIC at 33.33% withrc(threetypes of relation), and CIM at 22.22% with RC (null and positive types of relation). With n = 60, BF and CIM exhibit moderate power levels at 66.67% and low levels at 33.33% with RC (the three types of relation). AIC shows moderate levels on 55.56% and low levels at 44.44% with UN (negative relation) and RC (all three types of relation). For Within-Group Effects (WGE) the three procedures yield high power levels averaged for all analyzed conditions with only slight differences between them (MBF = 95.24; AIC = 94.83; CIM = 94.94). For Interactional Effects (INE) the procedures yield similar levels of power (MBF = 95.40; AIC = 94.43; CIM = 94.93). Non-Normally Distributed Data For BGE, three procedures yield low power levels averaged for all analyzed conditions with only slight differences between them (MBF = 28.85; AIC = 26.99; CIM = 27.09). In specific terms, the procedures yield: For BGE and n = 30, BF shows moderate levels at 77.78% and low levels of power, at 22.22%, respectively, of the analyzed conditions with RC (with null and positive types of relation). On the other hand, AIC and CIM show moderate levels in the 66.67% and 77.78%, respectively, and low levels between 33.33% and 22.22% with RC (positive and negative types of relation) and UN (positive relation). With n = 45, BF displays moderate levels at 55.56% of the analyzed conditions and low levels at 44.44% with ARH and RC (with null and positive types of relation). In line with the former, AIC and CIM show moderate levels: 33.33% of the analyzed conditions and low levels: 66.67% associated with ARH and RC (the three types of relation). With n = 60, the three procedures yield moderate levels at 33.33% of the analyzed conditions and the low levels at 66.67% associated to ARH and RC (the three types of relation). For Within-Group Effects (WGE) the three procedures yield high power levels averaged for all analyzed conditions with only slight differences between them (MBF = 95.22; AIC = 93.59; CIM = 93.60). For Interactional Effects (INE) the procedures yield similar levels of power (MBF = 96.51; AIC = 95.24; CIM = 95.65). Average Levels of Statistical Power at Normally Distributed Data Table 2 shows the average levels of statistical power for the three procedures when data are Normally Distributed with trimmed means at 10% and 20%. Trimmed Means at 10% For BGE with trimmed means at 10%, three procedures yield moderate power levels averaged for all analyzed conditions with only slight differences between them (MBF = 45.39; AIC = 40.55; CIM = 42.30). In specific terms, the procedures yield: For BGE and n = 30, BF shows high power levels at 22.22% of the analyzed conditions with ARH and UN (negative relation). The moderate levels are observed at 55.56% and the low levels at 22.22% associated with RC (null and positive types of relation). AIC and CIM show moderate levels at 77.78% and 88.89% of the analyzed conditions and low levels at 22.22% and 11.11% associated to RC (for null and positive types of relation). With n = 45, BF and CIM exhibit moderate levels at 77.78% and 22.22% of the analyzed conditions and low levels associated with RC (for null and positive types of relation). AIC shows moderate levels at 66.67% of the analyzed conditions and low levels at 33.33% associated with RC (three types of relation). With n = 60, BFandAICexhibit moderate levels at 66.67% of the analyzed conditions and low levels at 33.33% associated with RC (null and positive types of relation). CIM shows moderate levels at 66.67% of the analyzed conditions and lower levels at 33.33% Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing

7 P. Livacic-Rojas et al., Statistical Power, Covariance 15 Table 1. Percentages of Average Statistical Power of three procedures of covariance structure selectors with normally and non-normally distributed data, with mean patterns far and near each other ( 1, 0, 0, 1; 1, 0, 1, 0; 1, 1, 0, 0) and untrimmed means MBF AIC CIM N R OCS D BGE WGE INE BGE WGE INE BGE WGE INE 30 = ARH (1) N RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) NN RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN (Continued on next page) Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22

8 16 P. Livacic-Rojas et al., Statistical Power, Covariance Table 1. (Continued) MBF AIC CIM N R OCS D BGE WGE INE BGE WGE INE BGE WGE INE + ARH (1) RC UN ARH (1) RC UN Notes. Sample size (n); Relationship between sample and dispersion matrices (R); Null relation between sample and dispersion matrix size (=); Positive relation between group sample and dispersion matrix size (+); Negative relation between sample and dispersion matrix size ( ); OCS = Original Covariance Structure; ARH (1) = Heterogeneous First-Order Autoregressive; RC = Random Coefficients; UN = Unstructured; D = Distribution of Data; N = Normal Distribution; Non-Normal Distribution (Light, Moderate, and Strong bias); MBF = Modified Brown-Forsythe Procedure; AIC = Akaike s Information Criterion; CIM = Correctly Identified Model; BGE = Between-Group Effects; WGE = Within-Group Effects; INE = Interaction Effects; Power levels equal to or greater than 0.80 (boldface). Table 2. Percentages of Average Statistical Power of three procedures of covariance structure selectors with normally distributed data, with mean patterns far and near each other ( 1, 0, 0, 1; 1, 0, 1, 0; 1, 1, 0, 0) and 10 and 20% trimmed means MBF AIC CIM N R OCS TM BGE WGE INE BGE WGE INE BGE WGE INE 30 = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN (Continued on next page) Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing

9 P. Livacic-Rojas et al., Statistical Power, Covariance 17 Table 2. (Continued) MBF AIC CIM N R OCS TM BGE WGE INE BGE WGE INE BGE WGE INE + ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN Notes. Sample size (n); Relationship between sample and dispersion matrices (R); Null relation between sample and dispersion matrix size (=); Positive relation between group sample and dispersion matrix size (+); Negative relation between sample and dispersion matrix size ( ); OCS = Original Covariance Structure; ARH (1) = Heterogeneous First-Order Autoregressive; RC = Random Coefficients; UN = Unstructured; TM = Trimmed Mean; (0.1) = 10% Trimmed Mean; (0.2) = 20% Trimmed Mean; MBF = Modified Brown-Forsythe Procedure; AIC = Akaike s Information Criterion; CIM = Correctly Identified Model; BGE = Between-Group Effects; WGE = Within-Group Effects; INE = ; Power levels equal to or greater than 0.80 (boldface). associated with ARH (positive relation) and RC (null and positive types of relation). For Within-Group Effects (WGE) the three procedures yield high power levels averaged for all analyzed conditions with only slight differences between them (MBF = 95.24; AIC = 94.83; CIM = 94.94). For Interactional Effects (INE) the procedures yield similar levels of power (MBF = 95.40; AIC = 94.43; CIM = 94.93). Trimmed Means at 20% For BGE with trimmed means at 20%, three procedures yield moderate power levels average for all analyzed conditions with only slight differences between them (MBF = 60.67;AIC =59.83; CIM = 59.79). In specific terms, the procedures yield: For BGE and n = 30, BF exhibits high levels of statistical power at 22.22% of the analyzed conditions associated with ARH and UN (negative relation), moderate levels at 66.67%, and low levels at 11.11% with RC (positive relation). AIC and CIM show moderate levels at 88.89% and low levels at 11.11% associated with RC (positive relation). For n = 45, BF, AIC, and CIM show moderate levels of statistical power in 77.78% of the analyzed conditions and low levels at 22.22% associated with RC (for null and positive types of relation). With n = 60, the three procedures yield moderate levels of statistical power at 44.44% of the analyzed conditions and low levels at 55.56% associated with UN (null relation) and ARH (negative relation). For WGE, three procedures yield high power levels for all analyzed conditions with only slight differences between them (MBF = 95.21; AIC = 94.91; CIM = 94.96). For INE, the procedures yield similar levels of power (MBF = 95.47; AIC = 93.81; CIM = 94.48). Average Levels of Statistical Power at Non-Normally Distributed Data Table 3 shows the average levels of statistical power for the three procedures when data are Non-Normally Distributed with trimmed means at 10% and 20%. Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22

10 18 P. Livacic-Rojas et al., Statistical Power, Covariance Table 3. Percentages of Average Statistical Power of three procedures of covariance structure selectors with non-normally distributed data, with mean patterns far and near each other ( 1, 0, 0, 1; 1, 0, 1, 0; 1, 1, 0, 0) and 10 and 20% trimmed means MBF AIC CIM N R OCS TMBGE WGE INE BGE WGE INE BGE WGE INE BGE 30 = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN ARH (1) RC UN ARH (1) RC UN = ARH (1) RC UN (Continued on next page) Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing

11 P. Livacic-Rojas et al., Statistical Power, Covariance 19 Table 3. (Continued) MBF AIC CIM N R OCS TMBGE WGE INE BGE WGE INE BGE WGE INE BGE + ARH (1) RC UN ARH (1) RC UN Notes. Sample size (n); Relationship between sample and dispersion matrices (R); Null relation between sample and dispersion matrix size (=); Positive relation between group sample and dispersion matrix size (+); Negative relation between sample and dispersion matrix size ( ); OCS = Original Covariance Structure; ARH (1) = Heterogeneous First-Order Autoregressive; RC = Random Coefficients; UN = Unstructured; TM = Trimmed Mean; 0.1 = 10% Trimmed Mean; 0.2 = 20% Trimmed Mean; MBF = Modified Brown-Forsythe Procedure; AIC = Akaike s Information Criterion; CIM = Correctly Identified Model; BGE = Between-Group Effects; WGE = Within-Group Effects; INE = Interaction Effects; Power levels equal to or greater than 0.80 (boldface). Trimmed Means at 10% For BGE with trimmed means at 10%, the MBF yields moderate levels of statistical power on average (MBF = 31.76) while AIC and CIM yield low levels on average for all analyzed conditions with only slight differences between them (AIC = 29.12; CIM = 29.52). In specific terms, these procedures yield: For BGE and n = 30, the three procedures show moderate levels of statistical power at 77.78% of the analyzed conditions and low levels at 22.22% associated with RC (null and positive relation). For n = 45, BFshows moderate levels in the 55.56% and low levels at 44.44% of the analyzed conditions associated with RC (all three types of relation) and UN (positive relation). AIC and CIM display moderate levels at 44.44% and low levels at 55.56% associated with RC (all three types of relation) and ARH (positive relation). For n = 60, the three procedures exhibit moderate levels at 33.33% of the analyzed conditions and low levels at 66.67%, associated with ARH, RC, and UN (all three types of relation). For Within-Group Effects (WGE), the three procedures yield high power levels averaged for all analyzed conditions with only slight differences between them (MBF = 95.08; AIC = 93.62; CIM = 93.66). For Interactional Effects (INE) the procedures yield similar levels of power (MBF = 96.55; AIC=95.16; CIM=95.48). Trimmed Means at 20% For BGE with trimmed means at 20%, the three procedures yield moderate levels of statistical power on average for all analyzed conditions unlike 8% average in favor of MBF over AIC and CIM (MBF = 41.56; AIC = 31.31; CIM = 32.08). Specifically, the procedures yield: For BGE and n = 30, BF exhibits high levels of statistical power at 22.22% of the analyzed conditions associated to ARH and UN (negative relation), moderate levels at 66.67%, and low levels at 11.11% with RC (positive relation). AIC and CIM show moderate levels at 77.78% and low levels at 22.22% of the analyzed conditions to RC (null and positive types of relation). With n = 45, the three procedures exhibit moderate power levels at 66.67% and low levels at 33.33% associated to RC (the three types of relation). For n = 60, BF exhibits moderate levels at 66.67% of the analyzed conditions and low levels at 33.33% to RC (the three types of relation). AIC and CIM show moderate levels at 22.22% and low levels at 77.78% (except ARH and UN; negative relation). For WGE, three procedures yield high power levels for all analyzed conditions with only slight differences between them (MBF = 93.58; AIC = 93.77; CIM = 93.76). For INE, the procedures yield similar levels of power (MBF = 97.52; AIC = 95.32; CIM = 95.35). Discussion and Conclusions This study analyzed the power levels for three procedures used for selecting covariance structures. The overall results show high power levels only for within-group and interaction effects for the three procedures in the different conditions analyzed. In turn, for the between-group effects high power levels were observed, mainly associated with the MBF procedure for distributions with normal and non-normal data, with and without 10% and 20% trimmed means in groups of 30 cases, also for ARH and UN matrices (negative relation). Along the same line, the three procedures tend to show lower levels of statistical power when the size of sample data is 60, mainly associated with the RC (types of relation null and positive) when data are non-normally distributed. In addition, the trimmed mean comparatively yields a better increase on levels of statistical power when data are non-normally distributed. The results of this study are consistent with those reported by Vallejo et al. (2001, 2006), Vallejo, Arnau, et al. (2007), Vallejo, Fernández, and Livacic-Rojas (2007), Vallejo, Fernández, Herrero, et al. (2007) and Ó 2017 Hogrefe Publishing Methodology (2017), 13(1), 9 22

12 20 P. Livacic-Rojas et al., Statistical Power, Covariance Livacic-Rojas et al. (2010, 2013) in which MBF shows higher power levels than AIC and CIM for data that vary in different conditions. These findings could occur with MBF since it does not require a known covariance structure underlying the data. Therefore, it is more realistic in estimating a larger number of parameters with data taken at different points in time and it is more robust to the heterogeneity of the data. Similarly, when data are heterogeneous with an unstructured covariance matrix (UN), the procedure of BF modified by Vallejo et al. (2006) provides degrees of freedom (df) similar to those obtained using SAS PROC MIXED with METHOD = REML options and DDFM = KR. Furthermore, df calculated via two procedures are fully matched and mismatched when compared to contrast product for interaction (see Stroup, 2013 for more information). This information can be checked by running the MBF with the SAS PROC MIXED of the data taken from Nunez, Rosario, Vallejo, and González-Pienda (2013). Additionally, the results of lower sensitivity for AIC and MCI, associated with the presence of heterogeneity in the data, could also generate lower efficiency in the correct choice of the covariance matrix (associated with RC with greater visibility for the between-group effects), this observation correlates in part with the studies of Vallejo et al. (Vallejo, Arnau, et al., 2007; Vallejo, Fernández, & Livacic-Rojas, 2007; Vallejo, Fernández, Herrero, et al., 2007), Liu et al., (2012), Wilcox (2012), and Livacic-Rojas et al. (2013) since the conditions mentioned can affect the accuracy of the inferences in the testing of the design hypotheses, specifically, that of the AIC. Liu et al. (2012) pointed out that it is a criterion of low effectiveness in the presence of heterogeneous data associated with RC matrix. As Stroup (2013) stated, identifying an adequate covariance model is essential to analyze data, control the type I error rates, and detect correctly the effects of treatment (when statistical power increase) on repeated-measures designs. Furthermore, the results reported here are also consistent with those presented by Livacic-Rojas et al. (2010) and Stroup (2013) in which substantial importance was given, in the context of longitudinal designs, to the observation of higher power levels for the interaction effects. By contrast, the results of this study do not match those of Vallejo et al. (2008, 2010, 2011b), since the power levels of the covariance structure selectors are not high when the sample size increases and the complexity of the matrix decreases. Similarly, and as indicated by Vallejo et al. (2014), it is important for the researcher to consider that even when the analysis of covariance structure selectors show significant results, they may have different effect sizes depending on the number of groups and subjects within them, which may affect the power levels observed and the interpretation of the results. The results of this research are useful for the readers of Methodology journal because it is necessary to previously know the performance of covariance structure selectors on different conditions as they yield the Type I error rates and moderate levels of statistical power on main effects. Consequently, it may affect the accuracy of the results and conclusions of their research because it reduces the chance to identify the treatment effects (see too, Stroup, 2013). Moreover, the mixed model utilizes AIC instead of another criterion such as the BIC or the CAIC because, despite selecting the correct model at a low frequency, it is the criterion that exhibits the greatest efficiency and its use by researchers is highly frequent. Finally, in addition to considering the existing evidence (Fernández, Livacic-Rojas, Vallejo, & Tuero-Herrero, 2014) regarding the performance of covariance structure selectors for single-level and multilevel designs, it is necessary for future studies to compare and analyze the power levels of the three covariance structure selectors used and corrected ones (CAIC, AICC, and DIC) in one level and multilevel designs (fixed and random effects) since the lowest power levels are observed in association with the RC matrix (which represents hierarchical models) and because of the difficulty in treating the error structure (Liu et al., 2012) equally at all levels (Vallejo et al., 2013, 2014). What is more, it is necessary to analyze how these procedures work using the adjustment of syntax of generalized linear mixed models as Stroup(2013) has proposed. Acknowledgments This research was funded by the Chilean National Fund for Scientific and Technological Development (FONDECYT. Ref.: ) and the Spanish Ministry of Economy and Competitiveness (Grants PSI and PSI P). Electronic Supplementary Material The electronic supplementary material is available with the online version of the article at /a ESM 1. Tables (PDF). Supplementary tables of the research. References Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on automatic Control, AC-19, doi: /TAC Algina, J., & Keselman, H. J. (1997). Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the Welch-James test. Multivariate Behavioral Research, 32, Methodology (2017), 13(1), 9 22 Ó 2017 Hogrefe Publishing