Estimation in SEM: A Concrete Example

Transcription

1 Journal of Educational and Behavioral Statistics March 2007, Vol. 32, No. 1, pp DOI: / AERA and ASA. Estimation in SEM: A Concrete Example John M. Ferron Melinda R. Hess University of South Florida A concrete example is used to illustrate maximum likelihood estimation of a structural equation model with two unknown parameters. The fitting function is found for the example, as are the vector of first-order partial derivatives, the matrix of second-order partial derivatives, and the estimates obtained from each iteration of the Newton-Raphson algorithm. The goal is to provide a concrete illustration to help those learning structural equation modeling bridge the gap between the verbal descriptions of estimation procedures and the mathematical definition of these procedures provided in the technical literature. Keywords: structural equation modeling; teaching statistics; maximum likelihood; Newton-Raphson; parameter estimation Concrete examples are often used in the teaching and learning of mathematics and are advocated at virtually all levels of mathematics education, from K-12 instruction, as evidenced by the standards of the National Council of Teacher's of Mathematics (National Council of Teachers of Mathematics, 1989) to acquisition of higher level mathematics skills. One can easily extend the reasoning behind the need to provide students with concrete examples and representations beyond the purely mathematical realm into the world of statistics. For basic statistical concepts, like the standard deviation, worked-out examples using a definitional formula are provided in most introductory texts, and additional examples can be readily generated. For more advanced analyses, like multiple regression, canonical correlation, and multivariate analysis of variance, workedout examples can be found in Pedhazur (1997), Tatsuoka and Lohnes (1988), and Stevens (2002), respectively. As statistical analyses get more complicated, the availability of concrete examples diminishes. In the area of structural equation modeling, it is relatively difficult to find or generate concrete examples illustrating parameter estimation algorithms. Such examples, however, are valuable in helping learners bridge the gap between the verbal descriptions of estimation procedures and the mathematical definition of these procedures provided in the technical literature. We have found the example illustrating maximum likelihood estimation of a single parameter model 110

2 SEM Concrete Example FIGURE 1. Path diagram for computational example. given by Bollen (1989) to be useful to learners. Our purpose is to provide a more complex example for learners who benefit from concrete illustrations. We have chosen a two-parameter model for the example. Having multiple parameters enables us to maintain vectors and matrices in the computations, whereas limiting our model to two parameters keeps the computations relatively simple. Maximum likelihood estimation is illustrated because this is the method most commonly used to estimate structural equation models and the first method learners typically try to understand. Context for the Example In this example, we consider a single latent exogenous variable, ý, being used to predict a single latent endogenous variable, il, both of which have singleindicator variables (X and Y, respectively). The model is shown in a path diagram in Figure 1, where ovals indicate the latent variables of interest, rectangles indicate observed variables used as indicators of the latent variables, circles represent errors, single-headed arrows represent effects, and double-headed arrows originating and ending on the same variable indicate variances. For illustrative purposes, we have assumed known values for five of the model parameters and unknown values for the other two. The measurement error (6) of indicator variable X is assumed to have a variance of 2, and the measurement error (a) of indicator variable Y is assumed to have a variance of 4. Other parameters that we have assumed a value for include the variance of latent variable ý to be 8 and the coefficients from both latent variables ý -and ri to their single indicators variables, X and Y respectively, to be 1. The two parameters that we are estimating in this example are the coefficient from latent variable ý to latent variable 11, represented by gamma (y) and the variance of the disturbance of the endogenous latent variable il, represented by psi (*). This model can be mathematically represented by a series of matrices. Using the eight-matrix notation that over the years has been made familiar by the LISREL 111

3 Ferron and Hess (Linear Structural RELationships) software package (J6reskog & Stbrbom, 2004), the model parameters can be represented by the following series of matrices: A. [1] Ay =[11 0z= [2] 0= [4] 9= [81 T [,] P = [0] r =[y] where A, contains the coefficient from, to its indicator variable X, A, contains the coefficient from il to its indicator variable Y, 08 contains the variance in the measurement error of X, 0, contains the variance in the measurement error of Y, 9 contains the variance in, f becomes zero because this model has no effects between latent endogenous variables, W contains the variance in the disturbance of il, and r contains the coefficient from ý to rt. Implied Covariance Matrix If a structural equation model was properly specified, and if we knew all the parameter values, we could compute the model's implication for the covariance between all pairs of variables. Put another way, the model and parameter values, 0, imply a covariance matrix, E(O). The general form of the implied covariance matrix for structural equation modeling has been discussed (Hayduk, 1987) and derived (Bollen, 1989) elsewhere, and is given by lay(lf' )(raf, )((I E() Ax '[(/-- +)-]'A axa " + 0}'F A ' (1) where I is the identity matrix. By substituting the parameters from our example, this general equation can be simplified, yielding Z [ 1 8 =82+*+ 8,y 4 8y] 10J" (2) Observed Covariance Matrix Assume the observed covariance matrix is [ 20 (3) where 20 is the observed variance of X, 10 is the observed variance of Y, and 5 is the observed covariance between X and Y. 112

4 Maximum Likelihood Fitting Function SEM Concrete Example When we estimate parameter values for a structural equation model, we are trying to pick values for the parameters that make the implied covariance matrix as close as possible to the observed covariance matrix, or in other words, we try to minimize a discrepancy function between the model implied covariance matrix and the observed covariance matrix. There are a variety of different ways to define the discrepancy function, and thus alternative ways to estimate the parameters. The most common estimation method, maximum likelihood, is based on a normality assumption that allows the likelihood of the observed covariance matrix to be considered for a given set of parameter values. Different sets of values are considered, and then the set that maximizes the likelihood of the observed covariance matrix is selected. It can be shown that the values that maximize the likelihood of the data are the values that minimize the maximum likelihood fitting function, FML = log[f,(0)i + tr(sz-' (0)) - log[si - (p + q), (4) where p + q is the number of variables, S is the observed covariance matrix, and F,(O) is the implied covariance matrix. Bollen (1989) provides detailed derivation of the likelihood function and shows its relationship to the fitting function. The fitting function can be simplified for our example by substituting our implied and observed covariance matrices, Fmf = log 8[ 8 2 +*I+4 8y ([tr20 5 ][8y 2 ±4*±4 8y]-]) I y 1011 (5 10J[8y 10j log 5-1(p + q). (5) This equation can be simplified to Equation 6 and further simplified to Equation 7 through matrix algebra (see Hayduk, 1987, or Bollen, 1989, for details about inverses, determinants, traces, and multiplication of matrices in the context of structural equation modeling). The minimization process is unaffected by the constants in the function [log(175) and 2], thus these could be dropped from the equation. FML = log(80y2 + 10*' y2) + 5 tr y (6) -log(175)- 2 -o8gy '-4 113

5 Ferron and Hess FML = log(16y * + 40) + (16y2 + -0,+ 40) (80y 2-80y + 10* + 240) - log(175) - 2 (7) Minimizing the Fitting Function Numerical methods are used to minimize the fitting function. The algorithm begins with an initial guess for each of the parameters to be estimated. These guesses become the start values, which are then adjusted by considering how the fitting function is changing at these particular values. The adjusted values are then considered and also adjusted. The iterative process of making adjustments continues until it appears that the minimum of the fitting function has been reached. There are a variety of algorithms for minimizing the fitting function (e.g., Newton-Raphson, Fletcher-Powell, Gauss-Newton, Levenberg-Marquardt). The algorithms differ from each other in the details of how the adjustments, or steps, in the process are defined. We will illustrate the Newton-Raphson algorithm because it is relatively straightforward, but complex enough to facilitate thinking about the first- and second-order partial derivatives of the fitting function. A step in the Newton-Raphson algorithm is generally defined by 6(i+) = 6()_ ra2fml]-1 [@FML] L=6050 ao[ J (8) where the vector of parameter estimates at iteration i + 1, 0 (i+1), involves an adjustment of the parameter estimates from the previous iteration, V). The size of the adjustment depends on the gradient, 3Lf-, and the Hessian matrix, a2fail Maoa The gradient contains the partial slopes for the surface of the fitting function. As we get closer to the minimum, these slopes approach zero. The Hessian matrix contains information about the rate the partial slopes are changing. Using the gradient and inverse Hessian to adjust the estimates allow relatively large adjustments to be made when we are far from the minimum, and finer and finer adjustments to be made as we get closer to the minimum. To illustrate the use of the Newton-Raphson algorithm in this example, we must first find the first-order and second-order partial derivatives of the fitting function. In the next section, we will find the first-order partial derivatives of the fitting function for our example, and in the following section, we will find the second-order partial derivatives. Once the partial derivatives have been found, we will return to the Newton-Raphson algorithm and illustrate its application to our example. 114

6 Vector of First-Order Partial Derivatives of FML SEM Concrete-Example An element of the vector of first-order partial derivatives is found by finding the derivative of the fitting function with respect to a particular parameter. For our example, the gradient, or vector of first-order partial derivatives, is W,FML a y A- d afml " (9) L 8t) j The process of finding the first-order partial derivative of the fitting function with respect to y for our example is illustrated in Equations 10 through 12. Equation 10 shows that we intend to take the derivative of the fitting function. Equation 11 shows the result of taking the derivative with respect to Y, and Equation 12 shows the result after simplifying the expression. afml [Ilog(16y2 + 10* + 40) + ( (80y2-80y + 10*+ 240) - log(175) -2] (10) afml =(6204±)32y &-- = 16y`2 + 1'0* ( *f + 40)(160y - 80) - ( y + 10qf + 240) (32y) (11) ( * + 40)2 8 FML 512,y ý y* (12) Zy (16y2 + 10* + 40) 2 We then find the first-order partial derivative of the fitting function with respect to *' for our example, which is illustrated in Equations 13, 14, and 15. afml =_ [log( * + 40) + (16y ) (80y2-80y + 10*, + 240) - log(175) - 2 (13) 115

7 Ferron and Hess afml = 10 M-- (T6y2T+ 1o0l (16y )(10) - (80y2-80y )(10) (14) (16y ) 2 afml (15) a* ( )2 Matrix of Second-Order Partial Derivatives of FML The matrix of second-order partial derivatives of FML, the Hessian matrix, for our example is given by [a2fml a2fml1 ' FAfL -76 aya'l (16) FML a2fml Note that each element of the Hessian matrix is found by taking the derivative of one of the first-order partial derivatives with respect to one of the parameters. The first element of the first row of this matrix is obtained by taking the derivative of 8FmL with respect to 7. This process is illustrated starting with Equation 17, which indicates that we will be taking the derivative with respect to y of the result from Equation 12. The result of taking this partial derivative is shown in Equation 18 and then simplified to obtain Equation FML a [512y y4-8004* (17) 72 by L ( )2 ( )2 ( y ) - (512y3 8 2 FML y ) (2)( ) (32y) (18) (16y )4-8192,y y y a 2 FML y ,4y (19) _6T ( )3 The second element of the second row is obtained by taking the derivative of 8FmL with respect to *. This process is shown starting with Equation 20, which 116

8 SEM Concrete Example relies on the result from Equation 15, then proceeds by finding the partial derivative (Equation 21), and concludes by simplifying (Equation 22). a 2 FML - [--480y y + 100t) ] (20) ( * + 40) 2 (100) - (-480y y 6 2 FML + 100*)- 1600)(2)(1672 +t 10* + 40)(10) (21) a*)2 (16y2 + 10* + 40)4 a 2 FML 11200y y * _ (16y2 + 10* + 40)3 The off-diagonal elements can be found by either taking the derivative of with respect to y or the derivative of am with respect to 4i. The process of taking the derivative of a Equation 25. with respect to y is shown in Equations 23 through a 2 FML a [ZFML,] 8 [-480y y + 100* (23) a 7 r ao_ - 7 L a j j a 7 [ (16y2 + 10*i + 40)2 j ( * + 40)2(-960y + 800) - (-480y y' a 2 FML +1001J- 1600)(2)(1612+ lo4'40)( 32 'y) (24) a'ya*v (16y2 + 10* + 40)4 a 2 FML 15360y y y y*'y *l (25) a 7 asii -- (16y2 + 10qf + 40)3 Results of Each Iteration Now that we have solved for the first-order and second-order partial derivatives, we can illustrate the use of the Newton-Raphson algorithm for our example, where each step is defined by [i+1). - L a) "Ea 2 FML a 2 FML a F[FML W ay0 32 aya*) a- (26 2 J FML a 2 FML afml a*a 7 ay L- 0* 117

9 Ferron and Hess TABLE 1 Summary of the Results of Each Iteration of the Maximum Likelihood Estimation Iteration j(3) ý(3) 2E2EM FML ay E-6a E E E a E-6 is the scientific notation for To start the estimation process, we need some initial estimates for y and 4. There are alternative ways of doing this, ranging from subjective estimates based on the researcher's understanding of the model to noniterative estimation approaches like the instrumental variables method (e.g., Bollen, 1989). For simplicity, we chose to provide subjective estimates for the start values, (1) =.5 and 4(1) = 5. By substituting these values into '(0, -(-), OFML a 2 FML a 2 FML and 'FML and solving, Equation 26 becomes [,(2)7 [ ]-1 [ ] =( [ ' which leads to values of for,(2) and for j(2). These estimates are then substituted into (),, ary2 I 'FML and -5 in Equation 26, which leads to E 1 = [ ] o(3) _ [ ]-' [ ,i(3)j [ [ [ J ( The estimates are now = -(3) and j(3) = To conserve space, the results of the iterations are suimmarized in Table 1. In this table, we list the estimates of 'y and *', the elements of the gradient, and the value of the maximum likelihood fitting function for each iteration. Note that the changes in y' and *' are relatively large during the first few iterations and get smaller as the iterations continue. Also notice the elements of the gradient and the maximum likelihood fitting function are getting smaller as the iterations increase. For this example, the elements of the gradient become zero on the seventh iteration. In addition, the value for the fitting function becomes zero, which implies 118

10 SEM Concrete Example the estimates of for y and for * lead to an implied covariance matrix, El(0), that does not differ from the sample covariance matrix, S. Alternative Estimation Methods For this small example, we could have arrived at estimates for y and *, by equating the implied covariance matrix to the sample covariance matrix, [8Y(2+) i 8,yl _20 51 ] (29) L 8y 10o= 5 1 = Separate equations can be written for each element, leading to two equations containing two unknowns, 8y = 5 (30) 8y2 + * + 4 = 20. (31) Solving Equation 30 for y leads to 0.625, and substituting for y in Equation 31 and solving for 4, gives In more complex situations, there are typically more equations than unknowns. Attempts to solve the equations for the unknowns will often lead to multiple estimates for the same unknown because sampling error keeps the elements of S from being identical to the elements of E (0). It is these situations that motivate more complex estimation methods. Different methods (e.g., maximum likelihood, weighted least squares) are based on different assumptions (e.g., maximum likelihood assumes multivariate normality) and will often lead to different estimates, so care should be taken to choose an estimation method appropriate for the application. Finally, it should be noted that iterative methods will often not reach a fitting function value of zero. We say that we have met the convergence criterion when all elements of the gradient become less than some set value (e.g., ), which implies the estimates are changing trivially from one iteration to the next. Closing Comments For the purpose of illustrating maximum likelihood estimation in the context of structural equation modeling it was useful to focus on a relatively simple model. As models and data become more complex, researchers should be aware that estimation does not always proceed as smoothly as it did with this example. If estimation problems are encountered, it may be useful to consider the following strategies: (a) changing the start values, (b) changing the estimation algorithm, (c) increasing the maximum number of iterations, (d) screening the data for outliers and other anomalies, (e) using boundary constraints, (f) rescaling any variable with a variance many orders of magnitude larger than the variances of the 119

11 Ferron and Hess other variables, (g) checking the identification status of the model, (h) checking the consistency between the model specified in the structural equation modeling software and the path diagram, and (i) reexamining the consistency between the path diagram and theoretical expectations. For details on the problems that can be encountered, potential solutions, and limitations to these potential solutions, see Chen, Bollen, Paxton, Curran, and Kirby (2001); Wothke (1993); and Bentler and Chou (1987). References Bentler, P. M., & Chou, C. P. (1987). Practical issues in structural equation modeling. Sociological Methods and Research, 16, Bollen, K. A. (1989). Structural equation modeling with latent variables. New York: John Wiley. Chen, F., Bollen, K. A., Paxton, P., Curran, P. J., & Kirby, J. B. (2001). Improper solutions in structural equation models-causes, consequences, and strategies. Sociological Methods and Research, 29, Hayduk, L. (1987). Structural equation modeling with LJSREL: Essentials and advances. Baltimore: John Hopkins University Press. J6reskog, K. G., & Sbrbom, D. (2004). LISREL (Version 8.7) [Computer Software]. Lincolnwood, IL: Scientific Software International. National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author. Pedhazur, E. J. (1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). Fort Worth, TX: Harcourt Brace. Stevens, J. P. (2002). Applied multivariate statistics for the social sciences (4th ed.). Mahwah, NJ: Lawrence Erlbaum. Tatsuoka, M. M., & Lohnes, P. R. (1988). Multivariate analysis: Techniques for educational and psychological research (2nd ed.). New York: Macmillan. Wothke, W. (1993). Nonpositive definite matrices in structural, equation modeling. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (chap. 11, pp ). Newbury Park, CA: Sage. Authors JOHN M. FERRON is Professor of Educational Measurement and Research at the University of South Florida, 4202 East Fowler Ave. EDU 162, Tampa, FL 33620; His areas of interest include structural equation modeling, multilevel modeling, and the analysis of interrupted time-series data. MELINDA R. HESS is Director of the Center for Research, Evaluation, Assessment, and Measurement at the University of South Florida, 4202 East Fowler Ave. EDU 162, Tampa, FL 33620; Her areas of interest include multilevel modeling, structural equation modeling, research reporting protocols and practices, and program assessment. Manuscript received December 1, 2004 Accepted November 21,

12 COPYRIGHT INFORMATION TITLE: Estimation in SEM: A Concrete Example SOURCE: Journal of Educational and Behavioral Statistics 32 no1 Mr 2007 PAGE(S): The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited. To contact the publisher: