Latent variable models: a review of estimation methods

Transcription

1 Latent variable models: a review of estimation methods Irini Moustaki London School of Economics Conference to honor the scientific contributions of Professor Michael Browne

2 Outline Modeling approaches for ordinal variables Review methods of estimation for latent variable models Look at a special case: longitudinal data Goodness-of-fit tests and Model selection criteria Applications

3 Notation and general aim Manifest variables are denoted by: x 1, x 2,..., x p. Latent variables are denoted by: z 1, z 2,..., z q. One wants to find a set of latent variables z 1,..., z q, fewer in number than the observed variables (q < p), that contain essentially the same information

4 Estimation Methods Full Maximum Likelihood estimation (Bock and Aitkin, 1981; Bartholomew and Knott, 1997) E-M algorithm Newthon-Raphson maximization Limited Information Methods or Composite likelihood: lower order margins Markov Chain Monte Carlo methods (Albert 1992; Albert and Chib, 1993; Baker, 1998; Patz and Junker, 1999a,b; Dunson 2000, 2003; Fox and Glas, 2001; Shi and lee, 1998; Lee and Song, 2003; Lee, 2007.)

5 Full Maximum Likelihood Since all variables are random and only x is observed: f(x) = g(x z)h(z)dz where p g(x z) = g(x i z) i=1 (z 1,..., z q ) N(0, 1) For a random sample of size n the loglikelihood is written as: L = n log f(x h ) h=1

6 Notes q-dimension integration (Laplace, quadrature points, adaptive, Monte Carlo) Maximization (E-M, Newthon-Raphson)

7 Composite likelihoods a categorization based on Varin and Vidoni, 2005 Omission methods: remove terms that make the evaluation of the full likelihood complicated. (Besag, 1974; Azzalini, 1983) Subsetting methods: likelihoods composed of univariate, bivariate, trivariate,... margins (Cox and Reid 2004). They all fall within the context of pseudo likelihood methods (Godambe, 1960) or misspecified models (White, 1982)

8 Composite likelihoods based on marginal densities Key idea: The model still holds marginally for a specific set of variables. Use only information from the univariate or/and bivariate margins. e.g.: maximize the sum of all the log bivariate likelihoods. The estimator obtained for large n is consistent and asymptotically normal. Aim: decrease the number of integrals required without loosing too much precision.

9 Bayesian framework estimation Let us denote by v = (z, α ) the vector with all the unknown parameters The joint posterior distribution of the parameter vector v: h(v x) = g(x v)ψ(v)... g(x v)ψ(v)dv g(x v)ψ(v), (1) which is the likelihood of the data multiplied by the prior distribution (ψ(v)) of all model parameters including the latent variables and divided by a normalizing constant. Calculating the normalizing constant requires multi-dimensional integration that can be very heavy computationally if not infeasible.

10 The main steps of the Bayesian approach are: 1. Inference is based on: h(v x). 2. The mean, mode or any other percentile vector and the standard deviation of h(v x) can be used as an estimator of v and its corresponding standard error. 3. Analytic evaluation of the above expectation is impossible. Alternatives include numerical evaluation, analytic approximations and Monte Carlo integration. 4. MCMC methods are used for sampling from the posterior distribution (Metropolis Hastings algorithm, Gibbs sampling)

11 General Framework for ordinal variables Let x 1, x 2,, x p denote the observed ordinal variables and let m i denote the number of response categories of variable i. We write x i = s to mean that x i belongs to the ordered category s, s = 1, 2,..., m i. p i=1 m i: possible response patterns. Let x r = (x 1 = s 1, x 2 = s 2,..., x p = s p ) represent anyone of these. The model specifies the probability π r = π r (θ) > 0 of x r. πr = 1, for all θ where θ is the vector with all model parameters. The different approaches differ in the way π r (θ) is specified and in the way the model is estimated.

12 Modelling approaches Generalized latent variable framework (Samejima 1969, Moustaki, 2003) To take into account the ordinality property of the items we model the cumulative probabilities: γ i,s (z) = P (x i s z) = π i1 (z) + π i2 (z) + + π is (z) The response category probabilities are denoted by: π i,s (z) = γ i,s (z) γ i,s 1 (z), s = 1,..., m i Proportional odds model ln [ γi,s (z) ] 1 γ i,s (z) = α is q α ij z j, i = 1,..., p j=1

13 α i1 < α i2 < α imi 1 < α imi = Let x r = (x 1 = s 1, x 2 = s 2,..., x p = s p ) represent a full response pattern. Under the assumption of conditional independence, the conditional probability, for given z, of the response pattern x r is g(x r z) = π r (z) = p π i,s (z) = i=1 p [γ i,s (z) γ i,s 1 (z)]. (2) i=1 The unconditional probability π r of the response pattern x r is obtained by integrating π r (z) over the q-dimensional factor space: f(x r ) = π r = + where h(z) is the density function of z. + g(x r z)h(z)dz, (3)

14 Full Information ML, FIML ln L = r n r ln π r = N r p r ln π r, (4) where n r is the frequency of response pattern r, N = r n r is the sample size and p r = n r /N is the sample proportion of response pattern r. The Maximization can be done with E-M or N-R. Full ML computationally intensive for a large number of factors. Adaptive methods should be used. Pairwise likelihood does not have an advantage here.

15 MCMC estimation The Gibbs sampling (Geman and Geman, 1984; Gelfand and Smith, 1990) is a way for sampling from complex joint posterior distributions. Partition the vector of unknown parameters v into two components, v 1 = z and v 2 = α. Simulate from h(z 1 α 0, x) and h(α 1 z 1, x). The Gibbs sampler eventually produces a sequence of iterations v 0, v 1,... that form a Markov Chain that converges into the desired posterior distribution and it can be summarized in the following two steps:

16 1. Start with initial guesses of z 0, α Then simulate in the following order: Draw z 1 from h(z α 0, x). Draw α 1 from h(α z 1, x). The conditional distributions are: h(z α, x) = g(x z, α)h(z, α) g(x z, α)h(z, α)dz (5) h(α z, x) = g(x z, α)h(z, α) g(x z, α)h(z, α)dα (6)

17 When the normalizing constant is not in closed form a M-H algorithm within Gibbs is needed Computationally heavy Convergence criteria should be used Model selection criteria such as the Bayes factor require the calculation of the normalizing constant. MCMC methods give similar results with FIML (Kim, 2001; Wollack and ea., 2002; Moustaki and Knott, 2005)

18 Example: compare E-M and MCMC Social Life Feelings scale (Bartholomew and Knott, 2007) MCMC solution obtained with BUGS Two-parameter model is fitted E-M MCMC Item ˆα i0 ˆα i1 ˆα i0 ˆα i (0.13) 1.20(0.14) -2.37(0.14) 1.21(0.15) (0.06) 0.71(0.09) 0.79(0.06) 0.70(0.09) (0.09) 1.53(0.17) 0.99(0.09) 1.52(0.17) (0.13) 2.55(0.41) -0.74(0.20) 2.86(0.81) (0.07) 0.92(0.10) -1.10(0.07) 0.92(0.11)

19 Underlying variable approach assumes that the observed categorical variables are generated by a set of underlying unobserved continuous variables (Muthén, 1984; Jöreskog, 1994; Lee, Poon, and Bentler, 1990, 1992). This approach employs the classical factor analysis model: x i = λ i1 z 1 + λ i2 z λ iq z q + u i, i = 1, 2,..., p, (7) x i is x i = a τ (i) a 1 < x i τ (i) a, a = 1, 2,..., m i, (8) τ (i) 0 =, τ (i) 1 < τ (i) 2 <... < τ (i) m i 1, τ (i) m i = +, z 1,..., z q, u 1,..., u p are independent and normally distributed with z j N(0, 1) and u i N(0, ψ 2 i ).

20 ψ 2 i = 1 k j=1 λ2 ij. x 1,..., x p MV N(0, 1, P = (ρ ij )), where ρ ij = q l=1 λ ilλ jl. It follows that the probability π r (θ) of a general p-dimensional response pattern π r (θ) = P r(x 1 = s 1, x 2 = s 2,..., x p = s p ) = τ (1) s 1 τ (1) s 1 1 τ (2) s 2 τ (2) s 2 1 τ (p) sp τ (p) sp 1 φ p (u 1, u 2,... u p P)du 1 du 2 du p, (9) where the integral (9) is over the p-dimensional normal density function.

21 Estimation methods within the UVA 1. Full ML: ln L = r n r ln π r = N r p r ln π r, (10) If there is no model so that the π r are unconstrained, the maximum of ln L is ln L 1 = r n r ln p r = N r p r ln p r. Instead of maximizing (10) it is convenient to minimize the fit function F (θ) = r p r [ln p r ln π r (θ)] = r p r ln[p r /π r (θ)]. (11)

22 2. Three-stage procedures leading to limited information methods. 3. Composite likelihood methods: Use information from the univariate and bivariate margins to estimate the model. (Jöreskog and Moustaki, 2001) From the multivariate normality of x 1,..., x p, it follows π (g) a (θ) = τ (g) a τ (g) a 1 φ(u)du, (12) π (gh) ab (θ) = τ (g) a τ (g) a 1 τ (h) b τ (h) b 1 φ 2 (u, v ρ gh )dudv, (13) where φ 2 (u, v ρ) is the density function of the standardized bivariate normal distribution with correlation ρ.

23 Minimizes the sum of all univariate and bivariate fit functions F (θ) = p m g g=1 a=1 p (g) a ln[p (g) a /π (g) a (θ)]+ p g 1 m g m h g=2 h=1 a=1 b=1 p (gh) ab ln[p (gh) ab /π (gh) ab (θ)], Only data in the univariate and bivariate margins are used. This approach is quite feasible in that it can handle a large number of variables as well as a large number of factors. The asymptotic covariance matrix is often unstable in small samples, particularly if there are zero or small frequencies in the bivariate margins. The bivariate approach estimates the thresholds and the factor loadings in one single step from the univariate and bivariate margins without the use of a weight matrix.

24 Simulation study: small population p = 4 satisfying a one-factor model. m i = 5, i = 1,..., p There are Q = 625 possible response pattern. The model has 20 parameters. The sample size is N = 800. In the probit-generated data there are 163 distinct response patterns in the sample (NOR). In the logit-generated data there are 305 distinct response patterns in the sample (POM).

25 Logit gives more probability to a wider set of response patterns, whereas Probit concentrates more probabilities to a smaller set of response patterns. Table 1: Small Population: Estimated Factor Loadings NOR-Generated Data POM-Generated Data True UBN NOR POM UBN NOR POM

26 Multivariate longitudinal ordinal data Study the evolvement in time of traits, attitudes, ability, etc. Characteristics of longitudinal data: Extra dependencies to account for (within time/ between time). Factor changes over time are modelled through a multivariate normal distribution. The covariance matrix imposed on the time-related latent variables accounts for the dependence of the item responses across time. A GLLVM that accounts for dependencies within time and across time using time specific factors and a heteroscedastic item-dependent random effect term (Dunson 2003, JASA and Cagnone, Moustaki, Vasdekis, 2009, BJMSP). An autoregressive model is used to model the structural part of the model.

27 Within Structural Equation Modelling (SEM) framework longitudinal data are treated in different ways: Individual response curves over time by means of latent variable growth models. The main feature of this class of models is that the parameters of the curve, random intercept and random slope, can be viewed as latent variables. Allow for correlated errors.

28 Notation for longitudinal data Let x 1t, x 2t,..., x pt be the ordinal observed variables measured at time t, (t = 1,..., T ). The m i ordered categories of the ith item measured at time t have response category probabilities π it(1) (z t, u i ), π it(2) (z t, u i ),, π it(mi )(z t, u i ) The time-dependent latent variable z t account for the dependencies among the p items measured at time t The random effect u i accounts for the dependencies among the item i measured at different time points.

29 Model specification and assumptions The systematic component: η it(s) = τ it(s) α it z t + u i, i = 1,..., p; s = 1,..., m i ; t = 1,..., T. (14) The link between the systematic component and the conditional means of the random component distributions is η it(s) = v it(s) (γ it(s) ) where γ it(s) = P (x it s z t, u i ) and v it(s) (.) is the link function.

30 The associations among the items measured at time t are explained by the latent variable z t. Cov(η it, η i t) = α it α i tv ar(z t ), i i (15) with V ar(z t ) = φ 2(t 1) σ I(t 2) t 1 k=1 φ2(k 1) The associations among the same item measured across time (x i1,..., x it ) are explained by the item-specific random effect u i. Cov(η it, η it ) = α it α it Cov(z t, z t ) + σ 2 ui, t < t (16) Cov(z t, z t ) = φ t+t 2 σ I(t 2) t 2 k=0 φt t+2k, where I(.) is the indicator function.

31 The within-individual correlations are accounted for through modelling the covariance between latent variables z t and z t. z t = φz t 1 + δ t (17) δ t N(0, 1) and z 1 N(0, 1). u i N(0, σ 2 u) Cov(η it, η i t ) = α itα i t Cov(z t, z t ), i i, t < t (18)

32 Full ML Estimation with the EM-algorithm For a random sample of size n the complete log-likelihood is written as: n L = log f(x m, z m, u m ) m=1 = log n [log g(x m z m, u m ) + log h(z m, u m )] (19) m=1 g(x m z m, u m ) = T p g(x mit z mt, u mi ) (20) t=1 i=1

33 Let us define with ζ m = (z m, u m ) and with h(z m, u m ) = h(ζ m, φ, σ 2 u). Assuming that their common distribution function is multivariate normal their log-likelihood is, apart from a constant log h(ζ m, φ, σ 2 u) = 1 2 ln Φ 1 2 ζ mφ 1 ζ m (21) Φ = [ Γ 0 0 Ω ] (22) where Ω = diag i=1,...,p {σui 2 } and the elements of Γ are such that its inverse has

34 a well known special pattern as Γ 1 = 1 σ φ 2 φ φ 1 + φ 2 φ φ 1 + φ φ 1 + φ 2 φ φ 1

35 EM-algorithm E-step: the expected score function of the model parameters is computed. The expectation is with respect to the posterior distribution of (z m, u m ) given the observations (h(z m, u m x m )) for each individual. M-step: updated parameter estimates are obtained. The full ML is very computationally intensive as the number of items and factors increase.

36 Composite likelihood estimation Vasdekis, Cagnone, Moustaki (working paper) The model specification remains the same. S 1 = {(i, j, t); 1 i < j p; t = 1,..., T } S 2 = {(i, i, t, t ); 1 i p; 1 i p; 1 t < t T }

37 π (a it,a i t ) it,i t,z t,z t,u i,u i (θ) the joint density of (x it, x i t, z t, z t, u i, u i ) π (a it,a i t ) it,i t (θ) = P (x it = a it, x i t = a i t ; θ) the marginal probability density of any pair of observations. The bivariate probability for a pair of response (x it, x i t) is: π (a it,a i t ) it,i t (θ) = π (a it,a i t ) it,i t,z t,u i,u i (θ)dz tdu i du i and for a pair of response (x it, x i t ) π (a it,a i t ) it,i t (θ) = π (a it ) it (z t, u i )π (a i t ) (z t, u i )h(z t, z t, u i, u i )dz t dz t du i d i t

38 The latter can be three dimensional if it happens that i = i. In any case, the maximum number of integrations we have to deal with, is four. The contribution of any given individual to the log pairwise likelihood is therefore: pl(θ; y) = log π (a it,a i t ) it,i t (θ) + log π (a it,a i t ) it,i t (θ) (23) S1 S 2 Since each component of equation (23) is a likelihood object we can claim that pl(θ) = 0 is unbiased under the usual regularity conditions. The maximum pairwise likelihood estimator ˆθ P L is consistent and asymptotically normally distributed (Arnold and Strauss, 1991) under regularity conditions found in Lehman (1983), Crowder (1986) or Geys et. al. (1997). The maximization of the likelihood in (23) is done with the E-M algorithm.

39 Standard Errors For the pairwise likelihood defined in (23), the maximum pairwise likelihood estimator ˆθ P L = (ˆτ P L, ˆψ P L ) converges in probability to θ 0 and ˆθ P L N q ( θ0, J 1 (θ 0 )H(θ 0 )J 1 (θ 0 ) ) where J(θ) = E y { 2 pl(θ; y) } and H(θ) = var( pl(θ; y)) Heagerty and Lele (1998) and Varin and Vidoni (2005) give expressions for the estimation of J(θ) and H(θ).

40 Model selection criteria and Goodness-of-fit tests Varin and Vidoni (2005) propose also a composite likelihood information criterion based on the expected Kullback-Leibler information between the true density of the data and the density estimated via pairwise likelihood: pl(ˆθ P L ; y) + trĵĥ 1 (24) Limited information goodness-of-fit tests developed by Reiser (1996), Olivares and Joe (2005), Olivares and Joe (2006). X 2 e = e ˆΣ+ e e (25) where ˆΣ + e is the Moore-Penrose inverse of ˆΣe. X 2 e has an asymptotic χ 2 distribution with degrees freedom given by the rank of ˆΣ e

41 Application: British Household Panel Survey BHPS: study social and economic change at the individual and household level in Britain. The original data set consists of 6 ordinal items on social and political attitudes measured at 5 different waves (1992, 1994, 1996, 1998, 2000). The sample size was A random sample of 500 is considered. The following three items for three waves (1994=1,1996=2, 1998=3) are used: Private enterprise solves economic problems [Enterp] Govt. has obligation to provide jobs [Gov] Strong trade unions protect employees [Trunion]

42 The response alternatives are: disagree, strongly disagree. Strongly agree, Agree, Not agree/disagree, We collapsed the first two categories and the last two categories. A full ML and pairwise method are used to estimate the model. Five quadrature points are used

43 Results Model 1: No equality constraints Model 2: Thresholds equality constraints Model 3: Loadings equality constraints Model 4: Thresholds and loadings equality constraints Model selection criteria FIML Pairwise Models AIC CLIC Model Model Model Model

44 Results Parameter estimates with standard errors in brackets, BHPS FIML Pairwise Items ˆα i ˆσ 2 ui ˆα i ˆσ 2 ui Enterp (0.73) (0.52) Govern 0.86 (0.12) 5.09 (3.41) 1.05(0.09) 5.54(1.98) TrUnion 1.01 (0.18) 5.45 (3.91) 1.28(0.15) 5.63(1.43) Estimated covariance matrix of the time-dependent attitudinal latent variables ˆΓ F IML = ˆΓP air = ˆφ F IML = 0.95(0.01) ˆφP air = 0.86(0.04)

45 Simulation: Full vs. Pairwise Comparison between the two estimation methods under the following conditions of the study Setting as in the BHPS application 3 ordinal variables, 3 waves (with 4 items FIML was not able to converge) different sample sizes (200; 1000), quadrature points equal to 5, 200 replications

46 Results for loadings (variances in parentheses) Full ML Pairwise ML Mean MSE Mean MSE n=200 α 1 = 1.00 α 2 = (0.05) (0.05) 0.05 α 3 = (0.10) (0.07) 0.07 n=1000 α 1 = 1.00 α 2 = (0.01) (0.01) 0.01 α 3 = (0.02) (0.02) 0.02

47 Results for variances of item random effects Full ML Pairwise ML Mean MSE Mean MSE n=200 φ = (0.01) (0.01) 0.01 σu1 2 = (0.69) (0.54) 0.54 σu2 2 = (2.56) (1.73) 1.76 σu3 2 = (2.51) (2.06) 2.08 σ1 2 = (2.96) (1.43) 1.43 n=1000 φ = (0.00) (0.00) 0.00 σu1 2 = (0.11) (0.12) 0.12 σu2 2 = (0.38) (0.31) 0.33 σu3 2 = (0.40) (0.40) 0.41 σ1 2 = (0.35) (0.34) 0.34

48 Conclusions Full ML vs Composite likelihoods method: Full ML is in some cases not possible even for moderate size problems, composite likelihoods seem promising. Efficiency of estimators to be studied (computationally demanding). Weights either on separate marginal components or inclusion of weighted version of univariate densities to be checked. Limited information goodness-of-fit test under investigation. MCMC methods work for all models. Caution needs to be given on the convergence, parameterization, goodness-of-fit..