Reliability Estimates for Three Factor Score Estimators

Transcription

1 International Journal of Statistics and Probability; Vol. 5, No. 6; November 206 ISSN E-ISSN Published by Canadian Center of Science and Education Reliability Estimates for Three Factor Score Estimators University of Bonn, Institute of Psychology, Germany AndréBeauducel, Christopher Harms & Norbert Hilger Correspondence: AndréBeauducel, University of Bonn, Institute of Psychology, Kaiser-Karl-Ring 9, 53 Bonn, Germany. Received: September 8, 206 Accepted: October, 206 Online Published: October 26, 206 doi:0.5539/ijsp.v5n6p94 URL: Abstract Estimates for the reliability of Thurstone s regression factor score estimator, Bartlett s factor score estimator, and McDonald s factor score estimator were proposed. Moreover, conditions for equal reliability of the factor score estimators were presented and the reliability estimates were compared by means of simulation studies. Under conditions inducing unequal reliabilities, reliability estimates were largest for the regression score estimator and lowest for McDonald s factor score estimator. We provide an R-script and an SPSS-script for the computation of the respective reliability estimates. Keywords: factor analysis, reliability, factor score estimator. Introduction Factor score estimators are computed when individual scores on the factors are of interest. If, for example, decisions are made on the individual level (e.g., in personnel selection) an individual score is needed. It should, however, be noted that the estimation of factor scores does not refer to the estimation of population parameters from a sample. Even in the population, the individual scores on the factors cannot be computed because the number of common and unique factors exceeds the number of observed variables (McDonald & Burr, 967). In this sense, the factor scores are indeterminate. Therefore, linear composites of the observed variables (e.g., sum scales) are often formed in order to provide factor score estimates. Meanwhile, several factor score predictors with different properties have been proposed (Thurstone, 935; Bartlett, 937; McDonald, 98). When factor score estimators are computed, their reliability and validity is relevant. The coefficient of determinacy, i.e., the correlation of the factor score estimator with the factor (Grice, 200) has been related to the validity of factor score estimators (Gorsuch, 983). However, the reliability of factor score estimators has rarely been investigated. Indexes for the reliability of scores in the context of factor analysis have been proposed, but these coefficients have not been related to the available factor score predictors (McDonald, 985, 999; Revelle, 979; Revelle & Zinbarg, 2009; Zinbarg, Revelle, Yovel, & Li, 2005). A reliability estimate for Harman s ideal variable factor score estimator (Harman, 976) has already been proposed (Beauducel, 203). The present paper aims at proposing reliability estimates for Thurstone s regression factor score estimator (Thurstone, 935), Bartlett s factor score estimator (Bartlett, 937), and McDonald s correlation-preserving factor score estimator (McDonald, 98). Moreover, the effect of the size of loadings, the number of variables, the inter-correlation of the factors, and sampling error on the reliability estimates for the three factor score estimators will be investigated by means of a simulation study. It is, however, possible that at the population level the factor model does not perfectly represent the real-world relations between the measured variables, which is typically referred to as model error (MacCallum, 2003; MacCallum & Tucker, 99). Accordingly, the effect of model error on the reliability estimates of the factor score estimators is also investigated by means of a simulation study. Finally, an R-script and an SPSS-script are presented that allows for the computation of the reliability estimates for the factor score estimators starting from the loading pattern, the factor inter-correlations, and the item covariances. 2. Method In this section, we provide the definition of the factor model and the relevant reliability estimators. 2. Definitions In the population, the common factor model can be defined as x Λf e, () where x is the random vector of observations or items of order p. There are p observed variables and f is the random vector 94

2 of common factor scores of order q, e is the random error vector or unique vector of order p, and is the factor pattern matrix of order p by q. The factors f, and the unique or error vectors e are assumed to have an expectation zero ([x] = 0, [f] = 0, [e] = 0). The covariance between the factors and the error scores is assumed to be zero (Cov[f, e] = [fe ] = 0). The covariance matrix of observed variables can be decomposed into Σ ΛΦΛ Ψ 2, (2) where represents the q by q factor correlation matrix and 2 is a p by p diagonal matrix representing the expected covariance of the error scores e (Cov[e,e] = [ee ] = 2 ). It is assumed that 2 is positive definite and that the expectation of the non-diagonal elements is zero. 2.2 Reliability of Factor Score Estimators Cliff (988, p. 277; Eq. 4) presented a formula for the reliability of weighted composites which is based on two sets of parallel observed variables. For the population of individuals, this formula can be written as R B Σ B B Σ B B Σ B (3) /2 /2 = diag(diag( ) diag( ) ). ttc 2 22 where matrix B contains the weights, is the covariance matrix of the first set of observed ariables, 22 is the covariance matrix of the second set of observed variables, and 2 is the covariance matrix of the first with the second set of observed variables Thurstone s Regression Factor Score Estimator For Thurstone s regression factor score estimator the weights are B r = -. Entering these weights into Equation 4 and adding subscripts indicating the two sets of observed variables yields R = Cor( fˆ, fˆ ) r 2r diag(φ Λ Σ Λ Φ ) diag(φ Λ Σ Σ Σ Λ Φ ) diag(φ Λ Σ Λ Φ ) - -/ / Inserting Σ x x, x = Λ f +Ψ e, and x = Λ f +Ψ e into Equation 4 and some transformation yields / 2 - R = diag(φ Λ Σ Λ Φ ) diag(φ Λ Σ (Λ ffλ Λ feψ Ψ efλ + Ψ eeψ ) Σ Λ Φ ) diag(φ Λ Σ Λ Φ ) - - -/ It is assumed that the same factors are measured (f = f 2 ) and it is assumed that the same factor model holds in the population of the two sets of observed variables (,,, ). Λ = Λ Φ Φ Ψ = Ψ Σ = Σ This also implies f e = 0 and f e = 0. When these conditions hold and when there is no systematic unique or error variance, i.e., when 2 2 there is a zero covariance of the error scores across measurement occasions ( ε[ e e ] 0 ), Equation 5 can be transformed into R = (Φ Λ Σ Λ Φ ) (Φ Λ Σ Λ Φ Λ Σ Λ Φ ) (Φ Λ Σ Λ Φ ) (6) Bartlett s Factor Score Estimator Entering B b - -/ / 2 diag diag diag Ψ Λ(Λ Ψ Λ) for Bartlett s factor score estimator into Equation 4 and introducing the subscripts yields R = Cor( f ˆ, fˆ ) ttb b 2b = diag((λ Ψ Λ ) Λ Ψ Σ Ψ Λ (Λ Ψ Λ ) ) diag / 2-2 -? ((Λ Ψ Λ ) Λ Ψ xxψ Λ (Λ Ψ Λ ) ) diag ((Λ Ψ Λ ) Λ Ψ Σ Ψ Λ (Λ Ψ Λ ) ) / According to Λ = Λ, Φ Φ, Ψ = Ψ, Σ = Σ, f e = 0, f e = 0, and e e 0 Equation 7 can be transformed into (4) (5) (7) R diag((λ Ψ Λ ) Λ Ψ Σ Ψ Λ (Λ Ψ Λ ) ) / 2 ttb diag(φ ) diag((λ Ψ Λ ) Λ Ψ Σ Ψ Λ (Λ Ψ Λ ) ) / 2 It follows from diag(φ ) I that Equation 8 can be transformed into (8) 95

3 R = diag ((Λ Ψ -2 Λ ) - Λ Ψ -2 Σ Ψ -2 Λ (Λ Ψ -2 Λ ) - ) -. (9) ttb Entering Λ Φ Λ Ψ 2 for Σ into Equation 9 yields R = diag ((Λ Ψ -2 Λ ) - Λ Ψ -2 ( Λ Φ Λ Ψ 2 ) Ψ -2 Λ (Λ Ψ -2 Λ ) - ) -, (0) and, after some transformation, McDonald s Factor Score Estimator ttb R = ((Λ Ψ Λ ) Φ ) () diag ttb /2 Entering B Ψ ΛN(N Λ Ψ ΣΨ ΛN) where N is a q q matrix with m preserving factor score estimator into Equation 3 and introducing subscripts and assuming, Λ = Λ, Φ Φ, Ψ = Ψ, Σ = Σ, f e = 0, f e = 0, and e e 0 yields R = Cor( f ˆ,fˆ ) ttm m 2m NN = Φ for McDonald s correlation 2 2 / /2 = diag (( N Λ Ψ Σ Ψ Λ N ) N Λ Ψ Λ Φ Λ Ψ Λ N (N Λ Ψ Σ Ψ Λ N) ) Thus, only the parameters of the factor model are necessary in order to calculate the reliabilities, when the hypothetical item set is equivalent. 2.3 Comparing Reliability Estimates For Different Factor Score Estimators Since reliability estimates are based on Λ = Λ, Φ Φ, Ψ = Ψ, Σ = Σ, f e = 0, f e = 0, and e e 0, all true variance and all reliability is due to the amount of variance of f. Therefore, the factor score estimator with the highest correlation with f has the highest reliability. Thurstone s regression factor score estimator has the highest correlation with f (Krijnen, et al., 996), and Cor( fˆ, f ) Cor( f ˆ,f ) implies R R and Cor( fˆ, f ) Cor( f ˆ,f ) implies r b ttb r m R R Although the regression factor score estimator has the same or a larger reliability than the other two factor ttm score estimators, the conditions for having an equal reliability are also of interest. Theorem shows that the reliabilities of the regression factor score estimator and the Bartlett factor score estimator are equal when the condition Λ Σ - Λ diag(λ Σ - Λ ) holds for orthogonal factor models (i.e., Φ I ). The conditions Λ Σ - Λ diag(λ Σ - Λ ) and (2) Φ I hold for one-factor models, since q = implies Φ and that there - is only one resulting number for Λ Σ Λ.Moreover, the conditions Λ Σ - Λ diag(λ Σ - Λ ) and Φ hold for orthogonal factor models with only one non-zero factorloading of each variable. Theorem. If, Λ = Λ, Φ Φ I, Ψ = Ψ, Σ = Σ and Λ Σ - Λ diag(λ Σ - Λ ) then R = R Proof. From Jöreskog (969; Equation 0) we get Σ Λ Ψ Λ I Φ Λ Ψ Λ (3) = ( ) Premultiplication with Λ and some transformation yields Λ Σ Λ (( Λ Ψ Λ ) Φ ) which is entered into Equation 6. This yields R = diag (Φ (( Λ Ψ Λ ) Φ ) Φ ) diag (Φ (( Λ Ψ Λ ) Φ ) -2 -/ 2-2 Φ (( Λ Ψ Λ ) Φ ) Φ ) diag (Φ (( Λ Ψ Λ ) Φ ) Φ ) / 2 According to the conditions of Theorem, Equation 4 can be transformed into R = diag ((( Λ Ψ Λ ) I ) ) diag ((( Λ Ψ Λ ) I ) ) diag ((( Λ Ψ Λ ) I ) ) (5) -2 - / /2 Since Λ Σ - Λ diag(λ Σ Λ ) and Φ I implies (( Λ Ψ Λ ) I ) = diag(( Λ Ψ Λ ) I) Equation 5 can be transformed into This completes the proof. -2 ttb (4) R = diag (( Λ Ψ Λ ) I ) (6) Theorem 2 shows that the reliabilities of the regression factor score estimator and the McDonald factor score estimator are equal when the condition Λ Σ - Λ diag(λ Σ - Λ ) holds for orthogonal factor models ( Φ I). 96

4 Theorem 2. If Λ = Λ, Φ Φ I, Ψ = Ψ, Σ = Σ, and Λ Σ - Λ diag(λ Σ - Λ ) then R = R Proof. For Φ Φ I Equation 4 can be written as / / 2 R = diag (( Λ Ψ Σ Ψ Λ ) Λ Ψ Λ Λ Ψ Λ (Λ Ψ Σ Ψ Λ ) ) (7) ttm 2 Entering Λ Λ Ψ for Σ into Equation 7 and some transformation yields This completes the proof. R Λ Ψ Λ Λ Ψ Λ Λ Ψ Λ Λ Ψ Λ Λ Ψ Λ = diag (( ( ) ( ) ) ) ttm 2 = diag ((( Λ Ψ Λ ) I) ) 2 = diag (( Λ Ψ Λ ) I). Thus, the three factor score estimators considered here have the same reliability for q = and for orthogonal models with q > and only one non-zero factor loading of each variable. 2.4 Reliability of the Regression Score Estimator and the Coefficient of Determinacy In the following, the reliability estimate for the regression score estimator is compared with the determinacy coefficient (Grice, 200), which represents the validity of the factor score predictor. The covariances of the regression factor score estimator with the corresponding common factor are the diagonal elements of diag([f r f ]) = diag([ - xf ]) = diag( - ). (9) The standard deviation of the factor is one and the standard deviation of the regression factor score estimator is diag( - ) -/2. Accordingly, the factor score determinacy, i.e., the correlation of the regression score estimator with the corresponding common factors (Grice, 200) is diag(cor[f r, f]) = diag( - ) diag( - ) -/2 = diag( - ) /2. (20) When the common variance of the factor and the regression factor score estimator is computed for the factor models considered above, this yields, f 2 = diag diag cor - (Φ Λ Σ Λ Φ r - - ttm (8) f ). (2) For orthogonal factor models with Φ I and Λ Σ Λ diag(λ Σ Λ ) Equation 6 can be transformed into R = diag(λ Σ Λ ) diag(λ Σ Λ Λ Σ Λ ) diag(λ Σ Λ ) - -/ / 2-2 diag(λ Σ Λ) d iag cor f, f. r Thus, for orthogonal factor models with only one loading of each variable on one factor, the reliability estimate of the regression score estimator corresponds to the coefficient of determinacy. Since it has been shown that the reliability estimates of the regression score estimator, Bartlett s factor score estimator, and McDonald s factor score estimator are equal under these conditions, it follows that the abovementioned reliability estimates of the factor score estimators are equal to the (squared) determinacy coefficient for Φ I and - diag - Λ Σ Λ (Λ Σ Λ ). Theorem 3 describes the relation between the reliability estimate of the regression factor score estimator and factor score determinacy for orthogonal factor models that are identical across measurement occasions when Λ Σ - Λ diag(λ Σ - Λ ). Theorem 3. If Λ = Λ, Φ Φ I, Ψ = Ψ, Σ = Σ, and then R diag co f, f r. Proof. For simplification we introduce Accordingly, Equation 6 can be written as r diag( - Λ Σ Λ ) D and Λ Σ - Λ diag(λ Σ - Λ ) - Λ Σ Λ D H. / 2 / 2 R D diag( HH + HD DH + DD) D (23) Since H has a zero-diagonal, pre- and post-multiplication of H with the diagonal matrix D does not alter the diagonal elements, so that the diagonal elements in HD and DH are zero. Therefore, Equation 23 can be written as Since these diagonal elements are squared elements, it follows that (22) /2 /2 R D diag( HH + DD) D (24) diag(hh) 0 and diag(dd) 0. (25) 97

5 For orthogonal models Equation 2 can be written as co, It follows from diag(hh) 0 that diagcor f, f 2 This completes the proof. 2 /2 /2 diag r f f D D DD D. (26) r R. r To summarize, the determinacy coefficient corresponds to the reliability of the three factor score estimators for orthogonal factor models with only one loading of each variable on one factor and the determinacy coefficient is a lower-bound estimate of the reliability of the regression score estimator for orthogonal factor models when - - Λ Σ Λ diag(λ Σ Λ ), which can occur when there are non-zero secondary loadings. 3. Results However, the abovementioned considerations do not allow for a quantification of the relative differences of the reliability estimates of the factor score estimators. Therefore, three simulation studies were performed in order to give an account of the reliabilities of the three factor score estimators under different conditions. First, a simulation study was performed at the level of the population for sets of observed variables for which the factor model holds in the population. 3. Simulation Study The first short simulation study describes the effects of different population parameters on the reliability estimates. The simulation study was performed with IBM SPSS Version 22 and gives an account of the reliability estimates for the three factor score estimators for q = 6, depending on the number of main loadings per factor p/q (5, 0), the size of main loadings l (.40,.50,.60,.70,.80), the size of secondary loadings sl (.00,.0), and the size of the factor inter-correlations r (.00,.30). This results in (2 levels of p/q 5 levels of l 2 levels of sl 2 levels of r) 40 population models, for which population correlation matrices of observed variables were generated according to Equation 2. The models with p/q = 5 were based on 30 observed variables and the models with p/q = 0 were based on 60 observed variables. The reliability estimates for the factor score estimators were computed from the population parameters of the factor model ( Λ, Φ, Ψ ) and the corresponding item covariances () by means of Equations 6,, and 2. The results are summarized in Figure. No pronounced reliability differences occurred when the secondary loadings (sl) were zero, especially, when only reliabilities greater than.70 are considered. For sl =.0 and factor inter-correlations of.30, the regression score estimator had the largest reliability estimates. For factor inter-correlations of.30, McDonald s factor score estimator had the lowest reliability estimates. 3.2 Simulation Study 2 The next simulation was based on samples that were drawn from populations with the same model parameters as in the previous simulation. The simulation study was again performed with IBM SPSS Version 22. For each of the 40 population models of the previous simulation study,000 samples with n = 500 cases and,000 samples with n =,000 cases were drawn. Random numbers for the samples of factor scores were generated by means of the SPSS Mersenne Twister pseudo-random number generator. The corresponding samples of observed variables were generated from the common and unique factor scores by means of Equation 2. Maximum-likelihood factor analysis with subsequent Varimax-rotation for orthogonal population factor models and with Promax-rotation (kappa=4) for correlated factor models was performed in each sample of observed variables and the corresponding factor score reliabilities were computed from Equations 8, 3, and 4. The results can be found in Figure 2. The results of the simulation study for the samples were essentially the same as the results for the population parameters with the highest reliability of the regression factor score estimator. The main difference to the results of the simulation study for the population was that the Bartlett factor score estimator was substantially more reliable than the McDonald factor score estimator when the factor inter-correlations were substantial and when there were non-zero secondary loadings. 98

6 R tt sl r 5.30 p/q.00 r 0.30 l Figure. Reliability estimates for the regression factor score estimator, Bartlett s factor score estimator, and McDonalds factor score estimator for population models with q = 6. The horizontal line marks a reliability of.70 (R tt = Reliability estimate, l = salient loadings, sl = secondary loadings, r = factor inter-correlations). 99

7 n Mean R tt sl sl r 5.30 p/q.00 r 0.30 l Figure 2. Reliability estimates for the regression factor score estimator, Bartlett s factor score estimator, and McDonalds factor score estimator for samples based on population models with q = 6. The horizontal line marks a reliability of.70 (R tt = Reliability estimate, l = salient loadings, sl = secondary loadings, r = factor inter-correlations). 3.3 Simulation Study 3 The third simulation study was again based on the population parameters of the first and second simulation study. The only difference was that the simulation study was based on imperfect models, thus, on population models that were hypothesized according to the common factor model, but that do not fit exactly to the population covariance matrix (MacCallum & Tucker, 99; MacCallum, 2003). Imperfect models were generated as proposed by MacCallum and Tucker (99). The population correlation matrices were generated from the loadings of the major factors corresponding to the factors in the simulation studies and 2 as well as from the loadings of 00 minor factors and from the corresponding uniquenesses. Minor factors have very small nonzero population loadings and represent the many minor influences, which are thought to affect the values of the observed scores in the real world. Again, maximum-likelihood factor analysis with subsequent Varimax-rotation for orthogonal population factor models and with Promax-rotation (kappa=4) for correlated factor models was performed in each sample of observed variables and the corresponding factor score reliabilities were computed. The results for the imperfect models were extremely similar to those presented in simulation study 2, so that an additional figure was not necessary. Thus, imperfect models did not affect the reliability estimates substantially. 3.4 Standard Deviations and Effects of Factor Rotation Overall, the standard deviations of the reliability estimates were extremely small in simulation studies 2 and 3. The mean standard deviations were between.004 and.008 for n = 500 and they were between.002 and.007 for n =,000. However, the standard deviations were slightly larger for the correlated factor condition with non-zero secondary loadings. Therefore, the simulation study 2 was repeated for n = 250 for the correlated factor condition with non-zero secondary loadings. The mean standard deviations were greater than.0 for models with small salient loadings, especially for the Bartlett and McDonald factor score predictor (see Table ). Overall, the mean standard deviations for the regression factor 00

8 score estimator were smaller than the mean standard deviations of the remaining factor score estimators. The results indicate that sampling error may substantially affect reliability estimates when the sample size and salient loadings are small. Table. Standard deviations of R tt (no model error, q = 6, sl =.0, r =.30,,000 samples per condition) p/q l N Regression Bartlett McDonald Regression Bartlett McDonald Regression Bartlett McDonald Note. p = number of variables, q = number of factors, l = salient loadings, sl = secondary loadings, r = factor inter-correlations. In order to investigate the effects of factor rotation on the reliabilities of the factor score estimates, the simulation studies 2 and 3 were repeated with Equamax-rotation for the orthogonal factor models and with Oblimin-rotation (delta=0) for correlated factor models. Again, the reliability estimates were similar for the orthogonal models whereas the reliability estimates were highest for the regression factor score estimator when the analyses were performed for correlated factor models with non-zero secondary loadings. The results were very similar to those presented for the simulation studies 2 and 3 so that an additional figure was not necessary. 4. Discussion Based on the reliability definition for weighted composites (Cliff, 988), reliability estimates for Thurstone s regression factor score estimator, Bartlett s factor score estimator, and McDonald s factor score estimator were proposed. It was shown that the reliability estimates are equal for the three factor score estimators when they are based on a one-factor model or when there are orthogonal factors with only one non-zero factor loading of each observed variable (Theorem and 2). Moreover, it was shown that the reliability estimates for the regression factor score estimator are equal to the (squared) determinacy coefficient for the one-factor model or when there are orthogonal factors with only one non-zero loading of the items on a factor (Theorem 3). Thus, for one-factor models as well as for orthogonal models with only one non-zero loading of each variable, the determinacy coefficient represents a reliability estimate for the regression score estimator, for Bartlett s factor score estimator, as well as for McDonald s factor score estimator. Since some software (e.g. Mplus 7; Muthén & Muthén, 202) calculates the determinacy coefficient, it might be interesting to use these coefficients as reliability estimates for these models. Mplus users should be aware that they should square the determinacy coefficient computed by Mplus in order to get the reliability coefficient. For orthogonal factor models with more than one non-zero loading of the items on a factor, the determinacy coefficient is a lower-bound estimate of the reliability of the regression factor score estimator. The reliability estimates of the three factor score estimators were compared by means of a simulation study for the population and by means of a simulation study for samples drawn from a population in which the factor model holds as well as for samples drawn from a population in which the factor model does not hold. The population based simulation study revealed that under conditions where different reliability estimates can occur the reliability estimates were largest for the regression factor score estimator and that they were typically lowest for McDonald s factor score estimator, especially when the factor inter-correlations were substantial. In contrast, for orthogonal factors and when only substantial reliabilities (>.70) were considered, the differences between the reliability estimates for all three factor score estimators 0

9 were small. The results of the simulation studies for the samples were very similar to the results for the population based simulation study. Thus, computing the regression score predictor results in the highest reliability estimates and computing McDonald s factor score estimator typically results in considerably larger losses of reliability than computing Bartlett s factor score estimator, especially when the factor inter-correlations are substantial. It follows that McDonald s factor score estimator should only be computed when the salient factor loadings are very large. From a practical point of view it might be reasonable to compute the reliability estimates of the three factor score predictors and to only use Bartlett s or McDonald s factor score predictor when the corresponding losses of reliability can be neglected. This might be of interest because Bartlett s and McDonald s factor score predictor have properties that might convince applied researchers. For example, McDonald s factor score predictor is correlation-preserving. This means that the inter-correlations between McDonald s factor score predictors are the same as the inter- correlations between the factors. This property may facilitate the interpretation of McDonald s factor score predictor, but this property is lost with Bartlett s score predictor and with the regression score predictor. Moreover, we found that using imperfect factor models for the simulation study did not affect the results. This implies that the reliability estimates can also be used when the corresponding factor model does not fit perfectly to the data. Finally, we found that the standard deviations of the reliability estimates can be substantial in correlated factor models when the sample size is about n = 250 and when there are small salient loadings. Thus, caution with reliability estimates is warranted when sample sizes are about 250 cases or below and when salient loadings are small. Factor rotations (Equamax versus Varimax for orthogonal models and Promax versus Oblimin for correlated factor models) did not alter the results of the simulation study. From a practical point of view, it should be noted that whenever a specific score is computed and interpreted, the reliability estimate of the respective score should be known and substantial. Thus, when unit-weighted scales are computed, it might be reasonable to compute Cronbach s alpha as a reliability estimate. When a factor score estimate is computed and used in the context of assessment, the reliability of the factor score estimate should be evaluated. Accordingly, an R-script (Appendix A) as well as an SPSS-script (Appendix B) is presented that allows for the respective calculations of the reliability estimates from the loading pattern and factor inter-correlations. The R and SPSS scripts are also available in an online repository at References Bartlett, M. S. (937). The statistical conception of mental factors. British Journal of Psychology, 28, Beauducel, A. (203). Taking the error-term of the factor model into account: The factor score estimator interval. Applied Psychological Measurement, 37, Cliff, N. (988). The eigenvalues-greater-than-one rule and the reliability of components. Psychological Bulletin, 03, Gorsuch, R. L. (983). Factor analysis (2 nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Grice, J. W. (200). Computing and evaluation factor scores. Psychological Methods, 6, Harman, H. H. (976). Modern factor analysis (3rd ed.). Chicago, IL: University of Chicago Press. Jöreskog, K. G. (969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34, Krijnen, W. P., Wansbeek, T. J., & Ten Berge, J. M. F. (996). Best linear predictors for factor scores. Communications in Statistics: Theory and Methods, 25, MacCallum, R. C. (2003). Working with imperfect models. Multivariate Behavioral Research, 38, MacCallum, R. C. & Tucker, L. R. (99). Representing sources of error in the common-factor model: Implications for theory and practice. Psychological Bulletin, 09, McDonald, R. P. (98). Constrained least squares estimators of oblique common factors. Psychometrika, 46, McDonald, R. P. (985). Factor Analysis and Related Methods. Hillsdale, NJ: Erlbaum. McDonald, R. P. (999). Test Theory: A Unified Treatment. Mahwah, NJ: Erlbaum. McDonald, R. P., Burr, E. J. (967). A comparison of four methods of constructing factor scores. Psychometrika, 32,

10 Muthén, L. K., & Muthén, B. O. ( ). Mplus User ś Guide (Seventh Edition). Los Angeles, CA: Muthén & Muthén. Revelle, W. (979). Hierarchical cluster-analysis and the internal structure of tests. Multivariate Behavioral Research, 4, Revelle, W. & Zinbarg, R. E. (2009). Coefficients Alpha, Beta, Omega, and the GLB: Comments on Sijtsma. Psychometrika, 74, Thurstone, L. L. (935). The vectors of mind. Chicago, IL: University of Chicago Press. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbachs, Revelles, McDonalds H : Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, Appendix A R-script for reliabilities of factor score estimators ## This function computes and returns reliability estimates for three commonly used ## Factor Score Estimators in Factor Analyses. ## ## Explanations of the algebraic formulas are presented in the manuscript. ## Function for calculating reliability estimates for factor score estimators Lambda a \code{matrix} containing the loadings of items on the factors Phi a \code{matrix} containing the factor intercorrelations Estimators a \code{vector} to select the estimators for which the reliability ## estimates should be calculated. Available values: \code{regression}, \code{bartlett}, ## \code{mcdonald} Returns a two-dimensional list containing the reliability estimates for each ## factor. Depending on the \code{estimators} parameter, the list contains the values ## only for the selected Estimators. AndréBeauducel (\ {beauducel@uni-bonn.de}) Christopher Harms (\ {christopher.harms@uni-bonn.de}) Norbert Hilger (\ {nhilger@uni-bonn.de}) ## factor.score.reliability <- function(lambda, Phi, Estimators=c("Regression", "Bartlett", "McDonald")) { # Helper functions for frequently used matrix operations Mdiag <- function(x) return(diag(diag(x))) inv <- function(x) return(solve(x)) # If a loadings class is provided for lambda, we can easily convert it if (is(lambda, "loadings")) Lambda <- Lambda[,] # Perform several validity checks of the provided arguments if (any(missing(lambda), missing(phi), is.null(lambda), is.null(phi))) stop("missing argument(s).") if (any(nrow(phi) == 0, nrow(lambda) == 0, ncol(phi) == 0, ncol(lambda) == 0)) 03

11 stop("some diemension(s) of Phi or Lambda seem to be empty.") if (nrow(phi)!= ncol(phi)) stop("phi has to be a q x q matrix.") if (ncol(lambda)!= nrow(phi)) stop("phi and Lambda have a different count of factors.") if (any(round(min(phi)) < 0, round(max(phi)) > )) stop("phi contains invalid values (outside [0; ]).") Estimators.Allowed <- c("regression", "Bartlett", "McDonald") if (is.null(estimators)) { message("no Estimators defined, use Regression as default.") Estimators <- c("regression") } Estimators <- match.arg(estimators, Estimators.Allowed, several.ok = TRUE) # Regenerate covariance matrix from factor loadings matrix Sigma <- (Lambda %*% Phi %*% t(lambda)) Sigma <- Sigma - Mdiag(Sigma) + diag(nrow(lambda)) # Calculate uniqueness/error of items Psi <- Mdiag(Sigma - Lambda %*% Phi %*% t(lambda))^0.5 if (round(min(diag(psi))) < 0) stop("the diagonal of Psi contains negative values.") ret <- list() if ("Regression" %in% Estimators) { Rtt.Regression <- # Reliability of Thurstones Regression Factor Score Estimators # cf. Equation 6 in manuscript inv( Mdiag( Phi %*% t(lambda) %*% inv(sigma) %*% Lambda %*% Phi ) )^0.5 %*% Mdiag( Phi %*% t(lambda) %*% inv(sigma) %*% Lambda %*% Phi %*% t(lambda) %*% inv(sigma) %*% Lambda %*% Phi) %*% inv( Mdiag( Phi %*% t(lambda) %*% inv(sigma) %*% Lambda %*% Phi ) )^0.5 } ret$regression <- diag(rtt.regression) if ("Bartlett" %in% Estimators) { # Reliability of Bartletts Factor Score Estimators # cf. Equation in manuscript Rtt.Bartlett <- inv( Mdiag( inv(t(lambda) %*% inv(psi)^2 %*% Lambda) + Phi ) ) } ret$bartlett <- diag(rtt.bartlett) if ("McDonald" %in% Estimators) { # Reliability of McDonalds correlation preserving factor score estimators # cf. Equation 2 in manuscript 04

12 Decomp <- svd(phi) N <- Decomp$u %*% abs(diag(decomp$d))^0.5 sub.term <- t(n) %*% t(lambda) %*% inv(psi)^2 %*% Sigma %*% inv(psi)^2 %*% Lambda %*% N Rtt.McDonald <- Decomp <- svd(sub.term) sub.term <- Decomp$u %*% (diag(decomp$d)^0.5) %*% t(decomp$u) Mdiag( inv(sub.term) %*% t(n) %*% t(lambda) %*% inv(psi)^2 %*% Lambda %*% Phi %*% t(lambda) %*% inv(psi)^2 %*% Lambda %*% N %*% inv(sub.term)) } ret$mcdonald <- diag(rtt.mcdonald) # Return reliabilities as list, so it can be accessed via e.g. factor.score.reliability(l, P)$Regression } return(ret) ## Example : ## Users may just enter their respective values for Loadings and InterCorr. Loadings <- matrix(c( 0.50,-0.0, 0.0, 0.50, 0.0, 0.0, 0.50, 0.0,-0.0, -0.0, 0.50, 0.5, 0.5, 0.50, 0.0, -0.5, 0.50, 0.0, 0.0, 0.0, 0.60, 0.0,-0.0, 0.60, 0.0, 0.0, 0.60 ), nrow=9, ncol=3, byrow=true) InterCorr <- matrix(c(.00, 0.30, 0.20, 0.30,.00, 0.0, 0.20, 0.0,.00 ), nrow=3, ncol=3, byrow=true) reliabilities <- factor.score.reliability(lambda = Loadings, Phi = InterCorr, Estimators = c("regression", "Bartlett", "McDonald")) lapply(reliabilities, round, 3) 05

13 Appendix B SPSS-script for reliabilities of factor score estimators * This function computes and returns reliability estimates for three commonly used Factor Score Estimators in Factor Analyses, Explanations of the algebraic formulas are presented in the manuscript AndréBeauducel (\ {beauducel@uni-bonn.de}) Christopher Harms (\ {christopher.harms@uni-bonn.de}) Norbert Hilger (\ {nhilger@uni-bonn.de}) /*. MATRIX. * Users may enter their respective numbers into the loading matrix:. compute L={ 0.50,-0.0, 0.0; 0.50, 0.0, 0.0; 0.50, 0.0,-0.0; -0.0, 0.50, 0.5; 0.5, 0.50, 0.0; -0.5, 0.50, 0.0; 0.0, 0.0, 0.60; 0.0,-0.0, 0.60; 0.0, 0.0, 0.60 }. print L/format=F5.2. * Enter respective numbers into factor inter-correlations. compute Phi={.00, 0.30, 0.20; 0.30,.00, 0.0; 0.20, 0.0,.00 }. print Phi/format=F5.2. * Reproduce the observed covariances from the parameters of the factor model. compute Sig=L*Phi*T(L). compute Sig=Sig-Mdiag(diag(Sig))+ident(nrow(L),nrow(L)). * Calculate specificity/uniqueness/error of items. compute Psi=Mdiag(diag(Sig-L*Phi*T(L)))&**0.5. * Equation 9. 06

14 compute Rtt_r = INV( Mdiag(diag( Phi*T(L)*INV(Sig)*L*Phi )) )&**0.5 * Mdiag(diag(Phi*T(L)*INV(Sig)*L*Phi*T(L)*INV(Sig)*L*Phi)) * INV(Mdiag(diag(Phi*T(L)*INV(Sig)*L*Phi)))&**0.5. * Equation. compute Rtt_b=INV( Mdiag(diag(INV(T(L)*INV(Psi)&**2*L) + Phi)) ). * Equation 2. CALL svd(phi, QQ, eig, QQQ). compute N=QQ*abs(eig)&**0.5. compute help=t(n)*t(l)*inv(psi)&**2*sig*inv(psi)&**2*l*n. CALL svd(help, QQ, eig, QQQ). compute help2=qq*((eig)&**0.5)*t(qq). compute Rtt_m=Mdiag(diag( INV(help2)*T(N)*T(L)*INV(Psi)&**2*L*Phi*T(L)*INV(Psi)&**2*L*N*INV(help2) )). print/title "Reliabilities for Regression factor score estimators:". print {T(diag(rtt_r))}/Format=F6.3. print/title "Reliabilities for Bartlett factor score estimators:". print {T(diag(rtt_b))}/Format=F6.3. print/title "Reliabilities for McDonald factor score estimators:". print {T(diag(rtt_m))}/Format=F6.3. END MATRIX. Copyrights Copyright for this article is retained by the author(s), with first publication rights granted to the journal. This is an open-access article distributed under the terms and conditions of the Creative Commons Aibution license ( 07