On identification of multi-factor models with correlated residuals

Transcription

1 Biometrika (2004), 91, 1, pp Biometrika Trust Printed in Great Britain On identification of multi-factor models with correlated residuals BY MICHEL GRZEBYK Department of Pollutants Metrology, INRS, Avenue de Bourgogne, F Vandœuvre lès Nancy Cedex, France PASCAL WILD AND DOMINIQUE CHOUANIÈRE Department of Epidemiology, INRS, Avenue de Bourgogne, F Vandœuvre lès Nancy Cedex, France SUMMARY We specify some conditions for the identification of a multi-factor model with correlated residuals, uncorrelated factors and zero restrictions in the factor loadings. These conditions are derived from the results of Stanghellini (1997) and Vicard (2000) which deal with single-factor models with zero restrictions in the concentration matrix. Like these authors, we make use of the complementary graph of residuals and the conditions build on the role of odd cycles in this graph. However, in contrast to these authors, we consider the case where the conditional dependencies of the residuals are expressed in terms of a covariance matrix rather than its inverse, the concentration matrix. We first derive the corresponding condition for identification of single-factor models with structural zeros in the covariance matrix of the residuals. This is extended to the case where some factor loadings are constrained to be zero. We use these conditions to obtain a sufficient and a necessary condition for identification of multi-factor models. Some key words: Complementary graph; Covariance graph; Odd cycle; Structural constraint. 1. INTRODUCTION Our work is motivated by neurotoxicology. Several test variables coming from different neurobehavioural tests were referred to a limited number of unmeasured mental functions, or factors, each influencing some but not all of these variables. However, other factors of no scientific interest, such as the experimental conditions, may also influence some test variables. This situation can be expressed as a multi-factor model with some factor loadings constrained to zero, expressing the conditional independence between some test variables and some mental functions represented as latent factors. These zero factor loadings are termed structural zero factor loadings. For example, if variables exploring long-term memory have been measured along with others, like the simple reaction time which explores nervous motor function, the reaction time performance may be considered independent of long-term memory, given the latent factor characterising nervous motor function. We suppose here that, if all factors are included, the test variables are independent conditionally on the factors.

2 142 M. GRZEBYK, P. WILD AND D. CHOUANIÈRE However, we may want to marginalise over the factors with no scientific content, such as the level of mental concentration of the experimental subject on the neurobehavioural tests. This implies some marginal correlations among the test variables with a common factor over which we marginalised, conditional on the remaining factors. The problem we consider in this paper is the problem of identification. We do not consider of the problem of the existence of model in the words of Anderson & Rubin (1956) nor the problem of estimation. Anderson & Rubin (1956) gave some conditions for identification of multi-factor models with uncorrelated residuals and Giudici & Stanghellini (2001) gave a sufficient condition for identification of a subclass of such multi-factor models in which each observed variable is a response to one factor only and the dependencies between the residuals are expressed in terms of zero coefficients in the concentration matrix, i.e. the inverse of the covariance matrix. In this paper we build mostly on the work by Vicard (2000), who gave a necessary and sufficient condition for identification of such single-factor models. The main difference is that the independence structure in the residuals we shall consider is expressed in terms of zero restrictions on the correlation coefficients. We further restrict ourselves to the case of uncorrelated factors. After we introduce notation in 2, 3 gives a necessary and sufficient condition for identification of single-factor models with correlated residuals when the dependencies between the residuals are expressed in terms of zero covariance coefficients. Section 4 gives both a necessary and a sufficient condition for the identification of the considered multi-factor models with uncorrelated factors and correlated errors. These results are illustrated by a series of examples showing when our results can or cannot be applied. 2. NOTATION AND DEFINITIONS 2 1. T he problem We consider the multi-factor model X=lj+d, (1) where X is a vector of q observed variables X (u=1,...,q) with mean 0 and covariance u matrix S, j is a vector of p independent normally distributed factors j (i=1,...,p) each i with mean 0 and variance 1, l is a q p matrix of factor loadings and d is a vector of q normally distributed residuals with mean 0. The covariance matrix of d, denoted by H, is the covariance matrix of X conditional on j. It is assumed that cov (j, j)=i and p cov (j, d)=0. Furthermore, the factor loadings l contain structural zeros l =0 for some u,i pairs (X, j ). These structural constraints are the expression of the substantive knowledge u i about the independence of some X s from some factors, conditional on the other factors. We u further assume that H comprises structural zeros, cov (X, X j)=0, according to substantive u v knowledge. All the other parameters are assumed nonzero. According to (1), the implied covariance model of the observed variables expressed as a function of the parameters l and H is then S(l, H)=llT+H. (2) The model is said to be identified if, for two sets of parameters (l, H) and (l, H ) satisfying the structural constraints, S(l, H)=S(l, H ) implies that l=l and H=H. A well-known first necessary condition for identification of such a model is that the number of parameters be less than or equal to the number of nonnull relationships in equation (2).

3 Multi-factor models 143 As the identification issues are based solely on the moments, normality assumptions are not central to our considerations Graphical representation Such factorial models can be considered as chain graphs, denoted by G=(K, E), organised in two boxes. The set of vertices, K, consists of the factors j in the first box and the observed variables X in the second box. The set of edges, E, consists of the subset of the directed edges E X j ={(u, i):l ui N0} from the box of factors to the box of variables and the subset of the undirected edges E ={(u, v) : h uv N0 and unv} within the box of variables. The subgraph G =(X, E ) is the covariance graph (Cox & Wermuth, 1996, p. 30) of the observed variables conditional on the factors, that is the covariance graph of the errors d, which from now on we call the conditional covariance graph. We define its complementary graph as G9 =(X, E9 ) with E9 ={(u, v) : h uv =0}. Following the convention defined by Cox & Wermuth (1996), the covariance graph G is visualised with undirected dashed lines within the box of the observed variables since an absence of edge means a zero covariance between observed variables, conditional on the factors j. Likewise, chain graphs are read from right to left; directed edges are visualised as directed full arrows pointing from the factors on the right to the observed variables on the left, indicating a dependency of a variable from a factor conditional on all other factors but not on the other variables. Examples of chain graphs and their graphical representation are given throughout the text. 3. SINGLE-FACTOR MODELS 3 1. Preamble Vicard (2000) gave a necessary and sufficient condition for identification of single-factor models with correlated errors. This condition is based on the complementary graph of the conditional concentration graph. The result holds also when the dependencies are expressed in terms of a covariance graph, and if structural zero constraints on the factor loadings are allowed Structural zeros in the covariance matrix In this subsection, we suppose that all the factor loadings are nonzero. THEOREM 1. A necessary and suycient condition for a single-factor model to be identified is that the complementary graph of the conditional covariance graph G9 satisfies the following two conditions: (i) each connectivity component contains at least one odd cycle, (ii) the sign of one factor loading per connectivity component is given. Proof. Let (L, H) be the set of parameters representing the factor loadings and the conditional covariance matrix. The parameters L are nonzero whereas some H are zero if i uv and only if (u, v)µe9. According to (2), to establish the identification of the model, we have to check for the uniqueness of the solution to the system L L =l l ((u, v)µe9 ), (3) u v u v H +L L =h +l l ((u, v)µe or u=v), (4) uv u v uv u v in which (L, H) are constants and (l, h) are the unknowns.

4 144 M. GRZEBYK, P. WILD AND D. CHOUANIÈRE According to the structure of the system, each solution for l leads to a unique solution for h. If G9 contains more than one connectivity component, the system (3) can be split into disjoint subsystems, each one corresponding to a single connectivity component. The identification of l can be established independently in each connectivity component. Subsequently, we suppose that G9 is connected and the proof applies to each connectivity component. Let X and X be two distinct but arbitrary variables. Then there exists at least one 1 2 path connecting the corresponding nodes in G9 and (3) leads to l = 2 ql 1 L 2 /l, if the number of edges in the path is odd, (5) 1 l L /L, if the number of edges in the path is even. (6) We first prove sufficiency. Suppose there exists an odd cycle in G9 and suppose that X 1 and X 2 belong to this cycle. Then there exist two distinct paths from X 1 to X 2 through this cycle. As the cycle is odd, one of the paths is odd and the other is even, so that both (5) and (6) apply and give l2 1 =L2 1. (7) Thus l 1 =al 1 with a=±1 for any nodes in the odd cycle. It follows that, if X 1 is in the odd cycle and X 2 is not in the odd cycle, then both equations (5) and (6) reduce to l 2 =al 2, whatever the length of the path. If the sign of one factor loading is fixed a priori then a=1 and l u =L u for all vertices. Substituting l u by L u in (4) gives h uv =H uv for all (u, v) ine or u=v. Hence the model is identified. We next prove the necessity. Suppose that there is no odd cycle in G9. Two cases are possible. First, if there is no cycle, there exists only a single path between two vertices. Therefore, for any choice of l 1, either (5) or (6) provides a solution for any l 2, which means that the model is not identified. Secondly, suppose that G9 contains even cycles and assign an arbitrary value to l 1. Given any other distinct node X 2, all the paths connecting X 1 and X 2 have the same parity because G9 does not contain an odd cycle. Then all the paths connecting X 1 and X 2 lead to the same equation which expresses l 2 as a function of l 1 either through (5) or through (6). Thus the model is not identified. % Note that the necessary condition has been derived before by Marchetti & Stanghellini (1996) Structural zeros in the factor loadings of a single-factor model We suppose that the first q* factor loadings are nonzero and that the last q q* loadings are zeros. This situation is obtained after marginalising over some factors of no substantive interest. Furthermore, the results presented in this section will be used for the identification of multi-factor models. Let X(j)={X 1,...,X q* } be the subset of children of j; this is the subset of observed variables on which the latent factor loads. Let G* X(j).j denote the subgraph of G induced by X(j) and let G9 * X(j).j denote its complement. The following corollary results from Theorem 1.

5 Multi-factor models 145 COROLLARY 1. A necessary and suycient condition for a single-factor model with zero factor loadings to be identified is that the graph G9 * X(j).j satisfies the following conditions: (i) each connectivity component contains at least one odd cycle, (ii) the sign of one factor loading per connectivity component is given. Proof. The vector of factor loadings l can be written lt=(l*t, 0,..., 0), where l* is the q*-vector of nonzero factor loadings. Then llt is a block-diagonal matrix, llt= Al*l*T 0 0 0B. The expression of S(l, H) in (2) leads to the system of equations, for u>q* or v>q*, (8) uv s =Gh l l, for (u, uv u v v)µe9 *, (9) X(j).j h +l l, for (u, v)µe* or u=v, (10) uv u v X(j).j where s uv is the (u, v) entry in S(l, H). Equation (8) identifies h uv for u>q* orv>q*. Equations (9) and (10) correspond to the equations of identification of the single-factor model without zero constraints on the factor loadings, induced by the children of j. The complementary graph of the conditional covariance graph of this single-factor model is G9 * X(j).j, to which Theorem 1 applies. % A similar result was proved by Anderson & Rubin (1956, Theorem 5 4) for uncorrelated residuals. 4. MULTI-FACTOR MODELS 4 1. Introduction In this section, we give conditions for identification of multi-factor models using the conditions for identification of single-factor models. Submodels with fewer factors can be derived from a multi-factor model by marginalising over and conditioning on subsets of factors. The associated graphs of these submodels are called summary graphs (Cox & Wermuth, 1996, p. 196). Let j, j and j be a partition of the set of factors j. We consider the submodel derived s m c from (1) by conditioning on j and marginalising over j. c m Thus, only the observed variables X and the subset of factors j remain. The edges of the s graph of this model can easily be deduced from those of the complete graph: the set of the directed edges E is obtained by removing from E the edges coming from a X js X j factor either over which we marginalise or on which we condition; its set of undirected edges E is obtained by adding to E the edges connecting the pairs of nodes with a s common factor in j. m In particular, we focus on submodels where j is a single factor in the sequence. As s indicated in 3 3, the conditional covariance graphs induced by the children of the single factor are of particular interest; they are termed induced conditional covariance graphs of X(j ) conditional on j marginalised over j. s c m

6 146 M. GRZEBYK, P. WILD AND D. CHOUANIÈRE 4 2. A necessary condition for identification of a multi-factor model THEOREM 2. A necessary condition for a multi-factor model with uncorrelated factors to be identified is that all the single-factor models be identified which are obtained by conditioning on all factors but one. Equivalently, if at least one single-factor model, obtained by conditioning on all factors but one, is not identified then the multi-factor model is not identified. Proof. As these two conditions are equivalent, we only prove the second. Let G be a multi-factor model with uncorrelated factors and parameters {L, H}. Suppose that the single-factor model obtained by conditioning on all but factor j i is not identified. Let L i be the vector of factor loadings of j i and let Li be the q ( p 1) submatrix of the factor loadings of the remaining factors. Then {L i, H} is a set of parameters of a single-factor model. As it is not identified, there exists another set of parameters {l i, h} such that L i LT i +H=l i lt i +h. Then LiLiT+L i LT i +H=LiLiT+l i lt i +h, which means that {l i, Li, h} is another solution for the set of parameters of the initial multi-factor model. % This condition is illustrated in the following example. Figure 1(a) represents the chain graph of a multi-factor model with seven observed variables and two factors. The induced conditional covariance graphs of the two single-factor models are represented in Figs 1(b), (c). As the properties of the complementary graphs are the most important, we draw the edges of the complementary graph with full lines together with the dashed lines of the covariance graph. With this convention, we can easily check the existence of odd cycles in the full-lines graph. Fig. 1. A multi-factor model with 7 observed variables and 2 factors. The chain graph is shown in (a). ( b) and (c) show the induced conditional covariance graphs (dashed lines) and their complementary graphs (full lines) of the two single-factor models obtained by conditioning on j 1 and j 2 respectively. The single-factor model induced by conditioning on j 1 is identified but the single-factor model induced by conditioning on j 2 is not identified. Thus this model is not identified according to Theorem 2.

7 Multi-factor models 147 For example, in Fig. 1(c), the complementary graph of the induced conditional covariance graph of X(j ) does not contain an odd cycle, so that the multi-factor model is not identified. 1 This condition is not sufficient, as the following example shows. The chain graph of a multi-factor model with six observed variables and two factors is presented in Fig. 2(a). Both complementary graphs of the induced conditional covariance graphs of X(j ) conditional 1 on {j, j } and X(j ) conditional on {j, j } contain odd cycles, see Figs 2(b), (c), but the multi-factor model is not identified. To show this, we compare the number of unknowns and the number of equations. The number of nonzero covariances in S(l, H) is called the apparent number of equations. This is not always the actual number of independent equations as some equations may be redundant. Fig. 2. A multi-factor model with 6 observed variables and 2 factors. The chain graph is shown in (a). ( b) and (c) show the induced conditional covariance graphs (dashed lines) and their complementary graphs (full lines) of the two single-factor models obtained by conditioning on j 1 and j 2 respectively. Both single-factor models obtained by conditioning on one factor are identified, but the multi-factor model is not identified. The model in Fig. 2 has 17 parameters and the apparent number of equations is 18. However, two equations are redundant because the model imposes the two tetrad conditions, cov (X,X ) cov (X,X ) cov (X,X ) cov (X,X ) =1, cov (X,X ) cov (X,X ) cov (X,X ) cov (X,X ) = Thus, the actual number of equations is 16, which is lower than the number of unknowns. COROLLARY 2. A necessary condition for identification of a multi-factor model with uncorrelated factors is that each factor have at least three children. Furthermore, if there exists at least a residual correlation within the set of the children of any given factor, the number of children must be at least four.

8 148 M. GRZEBYK, P. WILD AND D. CHOUANIÈRE Proof. Suppose that a factor has fewer than three children. Then the single-factor model obtained by conditioning on all but this factor is not identified since the complementary graph of its conditional covariance graph has only one or two nodes and thus its associated graph cannot contain any odd cycle. If there exists a correlation within the children of a factor, the complementary graph of the induced conditional covariance graph of the single-factor model obtained by conditioning on all but this factor cannot have any odd cycle if the number of children is either 1, 2 or 3. % This result was proved by Anderson & Rubin (1956) in the case of uncorrelated errors A suycient condition for identification of a multi-factor model THEOREM 3. AsuYcient condition for identification of a multi-factor model with uncorrelated factors is that there exists at least one sequence of factors, j, such that each single-factor p(i) model obtained by conditioning on {j,j<i} and marginalising on {j,j>i} is identified. p(j) p(j) Proof. We show that, if such a permutation exists, the identification is reached successively for each vector of factor loadings l. In order to simplify notation, the factors are reordered p(i) such that p(i)=i. Let {L, H} be a set of parameters of a multi-factor model. Let {l, h} be a second set of parameters of the same model. By (2) the equation of identification of the model is LLT+H=llT+h. (11) In the proof, we denote by L the ith column of the matrix L and make use of the i general property that LLT=W L LT. i=1 i i First we consider identification of l. Let H =W L LT+H and h =W l lt+h. 1 i=2 i i i=2 i i Equation (11) can be rewritten as L LT+H =l lt+h. (12) This equation corresponds to the identification equation of the single-factor model obtained by marginalising over all but factor j. As it is supposed to be identified, l =L Next we consider identification of l. We suppose that l =L for all i<j. Then (11) is j i i reduced to L LT+H= l lt+h. (13) i i i i i=j i=j This equation is the identification equation of the factor model with p j+1 factors obtained by conditioning on the j 1 first factors. As for the identification of l,welet 1 H =W L LT+H and h =W l lt+h. Then (13) can be rewritten as i=j+1 i i i=j+1 i i L LT+H =l lt+h. (14) j j j j This equation corresponds to the identification equation of the single-factor model obtained by conditioning on {j,j<i} and marginalising on {j,j>i}. Again, as this single-factor p(j) p(j) model is supposed to be identififed, l =L. j j The identification process continues until j=p 1. Then l =L for i=1,...,p 1 i i and (11) is reduced to L LT+H=l lt+h. (15) p p p p This equation is the equation of identification of the single-factor model obtained by conditioning on all but factor j, which is supposed to be identified. Thus L =l and p p p H=h. %

9 Multi-factor models 149 This condition is illustrated by the following example. We consider a multi-factor model with six observed variables and three factors whose chain graph is given in Fig. 3(a). This multi-factor model is identified by Theorem 3; the sequence (j, j, j ) identifies the parameters. The induced conditional covariance graphs of the successive single-factor models resulting from the sequence are given in Fig. 3(b). Fig. 3. A multi-factor model with 6 observed variables and 3 factors. The chain graph is shown in (a). ( b) shows the induced conditional covariance graphs (dashed lines) and their complementary graphs (full lines) of the sequence of single-factor models illustrating the operations that prove the identification of the model using the sequence (j 2, j 1, j 3 ). Fig. 4. A multi-factor model with 7 observed variables and 2 factors. The chain graph is shown in (a). ( b) shows the induced conditional covariance graph (dashed lines) and its complementary graph (full lines) of the single-factor model obtained by marginalising over j 1 for the tested sequence (j 2, j 1 ). (c) shows the corresponding graph of the single-factor model obtained by marginalising over j 2 for the tested sequence (j 1, j 2 ). None of these single-factor models is identified. However this multi-factor model is identified.

10 150 M. GRZEBYK, P. WILD AND D. CHOUANIÈRE The following example proves that this condition is not necessary. We consider a multi-factor model with seven observed variables and two factors. The structural constraints are presented in the chain graph in Fig. 4(a). Neither sequence (j, j ) nor (j, j ) follows the condition of Theorem 3 since none of the single-factor models obtained by marginalising over j or j is identified. However, the identification has been checked 1 2 algebraically. Interested readers can contact M. Grzebyk for the full proof. 5. DISCUSSION An issue with these conditions is their computational complexity. We note first that Vicard s (2000) algorithm is linear in E*, where E* is the number of edges in G*. For the necessary condition of identification of a multi-factor model, the algorithm builds the induced conditional covariance graphs of all the single-factor submodels obtained by conditioning on all but one factor and checks that they follow the conditions of identification of Theorem 1, using the core algorithm. Thus the computational complexity of the algorithm is linear in p ( E +K), where K is the complexity of the algorithm that derives the induced conditional covariance graphs of a single-factor submodel obtained by conditioning on all but one factor. In the case of the sufficient condition for identification of a multi-factor model, the algorithm explores the set of permutations to look for a path among the latent variables which identifies the model. Thus, if the sufficient condition is not fulfilled, the parsing of all paths involves p! individual searches each of which consists of at most p generations of induced conditional covariance graphs of a single-factor model and then checks for odd cycles. If a particular graph is to be checked for identifiability on the basis of the present results, the necessary condition should be checked first as it is simpler and shorter. If all these single-factor sub-models are identified and no permutation of the factors is found by which the identification can be shown through Theorem 3, the identification status of the model is still pending since it was shown that none of these conditions is both necessary and sufficient. Note that, in the example presented in Fig. 2, the apparent number of equations was sufficient for identification but the model was not identified through Theorem 2, which at least in this case is a stronger criterion. The second part of the condition in Theorem 1 indicates that fixing one sign constraint for each factor is not sufficient for solving the invariance problem when there is more than one connectivity component in the graph E9 of a single-factor model. In this case, the sign of the factor loadings can be fixed independently in each connectivity component; these alternative choices lead to different solutions for the covariance matrix H, more precisely for the residual covariances between each pair of elements of X belonging to different connectivity components. Note that, in contrast to the factor loadings, these different solutions for H are not equal up to a sign-change: they may correspond to truly different values. Thus the sign constraints have to be chosen carefully. This phenomenon is crucial too in the case of a multi-factor model. Suppose there exists a sequence such that Theorem 3 applies. If the complementary graph of the induced conditional covariance graph of some single-factor model obtained by conditioning on {j,j<i} and marginalising over p(j) {j,j>i} contains more than one connectivity component, choosing different sign p(j) constraints leads to different solutions for the factor loadings of the subsequent factors as well as different solutions for the residual covariances. Note that this problem does not arise in uncorrelated factor models since then the complementary graph of the conditional covariance graph is connected.

11 Multi-factor models 151 Beyond the diagnosis of the identification of a given model, we saw in Corollary 2 that a first necessary condition for identification of a latent variable in particular studies is that each of these latent variables must be characterised by at least three observed variables; that is, if one wants to explore a latent trait in a real-world study, for instance in neurotoxicology, it must be explored through at least three observed variables. Furthermore, if these observed variables are still correlated, conditionally on the latent trait, more than three observed variables are needed. ACKNOWLEDGEMENT We acknowledge the help of the editor and of the two anonymous referees whose advice improved our paper significantly. REFERENCES ANDERSON, T. W. & RUBIN, H. (1956). Statistical inference in factor analysis. In Proc. 3rd Berkeley Symp. Math. Statist. Prob. 5, Ed. J. Neyman, pp Berkeley, CA: Univ. of California Press. COX, D. & WERMUTH, N. (1996). Multivariate Dependencies. Models, Analysis and Interpretation. London: Chapman and Hall. GIUDICI, P. & STANGHELLINI, E. (2001). Bayesian inference for graphical factor analysis models. Psychometrika 66, MARCHETTI, G. M. & STANGHELLINI, E. (1996). Alcune osservazioni sui modelli grafici in presenza di variabili latenti. In Atti della XXXVIII Riunione Scientifica 2, Ed. Società Italiana di Statistica, pp Rome, Italy: Maggioli Editore. STANGHELLINI, E. (1997). Identification of a single-factor model using graphical Gaussian rules. Biometrika 84, VICARD, P. (2000). On identification of a single-factor model with correlated residuals. Biometrika 87, [Received April Revised September 2003]