Multilevel Structural Equation Models for the Analysis of Comparative Data on Educational Performance

Transcription

1 Jounal of Educational and Behavioal Statistics Septembe 2007, Vol. 32, No. 3, pp DOI: / Ó AERA and ASA. Multilevel Stuctual Equation Models fo the Analysis of Compaative Data on Educational Pefomance Havey Goldstein Univesity of Bistol Ge ad Bonnet Thiey Roche Ministe`e de l Education Nationale, de l Enseignement Supe ieu et de la Recheche, Diection de l E valuation et de la Pospective, Pais The Pogamme fo Intenational Student Assessment compaative study of eading pefomance among 15-yea-olds is eanalyzed using statistical pocedues that allow the full complexity of the data stuctues to be exploed. The aticle extends existing multilevel facto analysis and stuctual equation models and shows how this can extact iche infomation fom the data and povide bette fits to the data. It shows how these models can be used fully to exploe the dimensionality of the data and to povide efficient, single-stage models that avoid the need fo multiple imputation pocedues. Makov Chain Monte Calo methodology fo paamete estimation is descibed. Keywods: intenational compaisons; facto analysis model; educational assessment; stuctual equation model; item esponse model; multilevel model; Makov Chain Monte Calo The pincipal pupose of this aticle is to exploe methodological issues in the analysis of lage-scale data of compaative educational pefomance using multilevel stuctual equation models. To illustate ou appoach, we analyze data fom the Pogamme fo Intenational Student Assessment (PISA) suvey of eading pefomance, which epesents a vey ambitious and wide-anging attempt to measue and compae 15-yea-olds in 32 counties and which employs pocedues and models used widely in analyzing educational pefomance. We begin by descibing the data and aising some peliminay methodological issues. This eseach was patly suppoted by the Ministèe de l Education Nationale, de l Enseignement supéieu et de la Recheche, diection de l évaluation et de la pospective, Pais, and by a eseach gant fom the Economic and Social Reseach Council (RES ). We ae vey gateful to Fiona Steele, William Bowne, and David Thissen fo helpful comments and to anonymous efeees. 252

2 Multilevel Stuctual Equation Models Unde the auspices of the Oganization fo Economic Coopeation and Development (OECD), the testing fo PISA was conducted in the fist half of 2000, and the study was intended to be the fist of a seies. Although PISA concentates on eading, it also has mathematics and science components. The second study, conducted in 2003, concentated on mathematics, and the thid, conducted in 2006, concentated on science. The sampling design selected schools as fiststage units and sampled 15-yea-old pupils within schools with a maximum of 35 students in each school. Extensive piloting of test items and geneal pocedues, including tanslations, was caied out. The fist compehensive epot (OECD, 2001) appeaed in 2001, and an extensive (300-page) technical epot (Adams & Wu, 2002) povides detail about the pocedues used. In addition, data ae available fo seconday analysis fom the OECD Web site ( The PISA 2000 (OECD, 2001) analyses have concentated on computing student poficiencies and county means fo the thee eading poficiency subscales, Retieving Infomation, Intepeting Texts, and Reflection and Evaluation, as well as a combined scale. Each subscale is defined by a diffeent set of items. In this aticle, we analyze data fom the fist subscale, Retieving Infomation, containing 35 items. Details of this subscale can be found in the PISA 2000 technical epot (Adams & Wu, 2002). The full scale contains 36 items, but one of these (R076Q03) was eliminated fom the England file as dodgy because it did not fit well using the one-dimensional scaling pocedue applied in the study. Two counties, Fance and England, wee chosen fo this pupose. In PISA itself, Wales did not paticipate, and accoding to the technical epot (Adams & Wu, 2002, p. 191), Scotland did not popely follow the sampling pocedues. Unfotunately, the main OECD epoting only efes to the United Kingdom, that is, the aveage ove England, Scotland, and Nothen Ieland; because these have vey diffeent educational systems, intepetation is complicated. Thee is a sepaate county epot (Office fo National Statistics [ONS], 2002), howeve, that does allow diect compaisons with ou analysis. The data used in the pesent aticle consist of 326 schools (141 in England and 185 in Fance) and 8,299 students (4,070 in England and 4,229 in Fance); futhe details can be found in Adams and Wu (2002). One poblem that aises in compaing Fance and England (as well as in othe county compaisons) is that students move though the systems in diffeent ways. PISA samples by age, namely, all childen bon in In England, most childen stat school in Septembe of the school yea in which they each the age of 5. Thee is almost no epeating of yeas, so that a 15-yea-old at the time of the PISA suvey in Apil/May 2000, bon in August 1984, will stat school in Septembe 1988 and be in Gade 11 at the time of the PISA suvey and in a class whee thee ae a numbe of olde childen (not in PISA) bon in Septembe 1983 to Decembe Howeve, the fist yea of schooling is designated as eception, so that, in fact, that child will have been in fomal schooling fo 12 yeas. A child bon in Septembe 1984 will stat school 1 yea late and be in Gade 10, 253

3 Goldstein, Bonnet, and Roche and this latte child is about the same age as the fome but has had 1 yea less schooling. In Fance, on the othe hand, childen stat school in Septembe of the calenda yea in which they each 6 yeas. Thus, a child bon in August 1984 who does not epeat a yea will be in Gade 10, as will be a child bon in Septembe, and both will have eceived the same amount of schooling. In Fance, the 1st yea in school is counted as Gade 1. Any child who epeats a yea (appoximately one thid do so) will be in Gade 9. Because the nomal tansition fom collège to lycée occus afte Gade 9, these childen will be in collège along with childen who have not epeated, that is, those bon in Thus, fo the childen bon between Septembe 1984 and Decembe 1984, the Fench and English students will have been in fomal schooling fo the same length of time in tems of gades, although if eception is counted, the English will have been in school 1 yea longe. Fo those bon between Januay 1984 and August 1984, the Fench students will have been in schooling 1 futhe yea less than the English, whethe o not they epeat. Howeve, 100% of Fench childen ae in peschool povision (école matenelle) fo 2 yeas pio to fomal schooling and 94% of Fench childen 3 yeas pio. In England, about 80% of 3-yea-olds ae in pattime nusey education. This makes compaisons vey difficult, and we discuss late how we attempt, at least patially, to take account of this. The fist section of this aticle pefoms some simple analyses, effectively eplicating those of the OECD, and goes on to pefom some elational analyses. The second section intoduces the multilevel (school and student) stuctue of the data and shows how a valid analysis can be pefomed. The thid section exploes the dimensionality of the data at both the student level and the school level. The fouth section shows how a constained multilevel model can be fitted to make compaisons that have a consistent intepetation. The final section discusses some implications of the findings fo intenational compaisons. One-Dimensional Latent Vaiable Models fo Student Pefomance The standad psychometic pocedue fo the modeling of test item esponses is commonly known as item esponse theoy (Lod, 1980). A simple, basic, latent-tait model of this type elates the esponses on a set of test items to one o moe undelying latent abilities. A basic vesion can be expessed as follows. Fo a student (iþ who esponds to item (Þ, the pobability (p i Þ of a coect esponse is given by gðp i Þ = b 0 + λ y i y i Nð0, σ 2 y Þ y i binomialð1, p i Þ, ð1þ whee g is a link function, typically the logit, and the esponse, y i, is 1 if the item is coectly answeed and 0 if not and the y i ae mutually independent. 254

4 Multilevel Stuctual Equation Models This is just a binay facto model with a single facto (yþ and a set of loadings (λ Þ. We efe to the tem b 0, often efeed to as the facility fo Item, as belonging to the fixed pat of the model, which will late be augmented with futhe pedictos. In the PISA data, we have some gaded o patial-cedit items whee the coect esponses ae eithe patially coect (coded 1) o fully coect (coded 2). In this case, fo such an item, the fist line of the model can be witten, fo a esponse coded s (s = 0, 1Þ,as gðg s i Þ = b 0 + a s + λ y i g s i = Xs p f i, a 0 = 0, f =0 ð2þ whee p f i is the pobability of a esponse in Categoy f : We hee model the cumulative pobabilities whee cumulation is fom Categoy 0 (incoect) athe than in the binay case whee, by convention, the fist categoy is the coect esponse. Thus, in the fixed pat of the model, the pobability of an incoect esponse is simply g 1 ðb 0 Þ, and the pobability of an incoect o patially coect esponse is g 1 ðb 0 + a 1 Þ. An impotant assumption in Models 1 and 2 is that the esponses y i ae conditionally independent. Because some of the items involve esponses to the same text o figue, it is possible that this assumption will be violated, as the (conditional) pobability of a coect esponse to one item may depend on the outcome with espect to an ealie item. Thus, fo example, Items R104Q01, R104Q02, R104Q05, and R104Q06 all efe to a passage about telephone use and featue the same peson (Pedo) in each question. 1 To avoid this poblem, an altenative is to conside the complete set of item esponses, teating the numbe of coect esponses to this set of fou binay items as a single-item-gaded esponse. Thissen, Steinbeg, and Mooney (1989) discussed this, and Steinbeg and Thissen s (1996) study pattened item combinations as elementay esponse units (testlets). In the above example, we could, fo instance, compute a total scoe anging fom 0 to 4. Scott and Ip (2002) consideed an altenative appoach to what they tem the item clusteing effect. Fo each specified item cluste, they added an individual-level andom effect designed to identify an individual s additional esponse to each item as a membe of that cluste (see below). We have not pusued these possibilities hee, but they ae an inteesting aea fo futhe wok. A special case of Model 1 is the so-called Rasch model, whee the loadings λ ae constained to be equal. This is the model, with a logit link, used in the PISA analyses. In this aticle, we use a pobit link athe than a logit (see, e.g., Lod & Novick, 1968, chap. 16). The two link functions ae, in fact, vey simila so that we can expect esulting estimated pobabilities to be vey close. The pobit, howeve, has cetain advantages computationally and also has a useful 255

5 Goldstein, Bonnet, and Roche intepetation in tems of an undelying nomal popensity distibution fo the esponses. Thus, fo a binay esponse, we can suppose that thee is an undelying continuous esponse fo an item with a theshold value (XÞ such that esponses above that value ae coect and those below ae incoect. Fomally, we wite the pobability of a coect esponse as Z X fðxþdx, whee f is the standad nomal density. This model is discussed, among othes, by Fox and Glas (2001) and by Goldstein and Bowne (2004) and is hee extended to the odeed categoy case given by Model 2 (see the appendix fo a complete specification). The above model can be extended in seveal ways. Fist, we can add futhe fixed-pat explanatoy vaiables such as gende, county, age, and so on. Second, we can make the model multilevel by ecognizing between-school vaiation and explicitly incopoating school-level latent vaiables o factos. Thid, we can add futhe facto dimensions along which student esponses can vay. Fouth, we can allow the loadings to be functions of explanatoy vaiables. Finally, we can allow the facto values o scoes also to be functions of explanatoy vaiables. It is possible to extend the model to conside moe geneal stuctual equation models, but we shall not pusue this hee. Steele and Goldstein (2006) gave a futhe discussion and an application to women s status data. Genealizing the notation of Goldstein and McDonald (1988) and McDonald and Goldstein (1989), a basic multidimensional two-level model fo a continuous nomal esponse is given by Model 3. In the case of binay o odeed esponses, this models the undelying popensity as defined above, and ou exposition thus will be expessed in tems of the following basic model. y ij = X h b h x hij + XF λ ð2þ f νð2þ fj + XG f =1 g=1 λ ð1þ g νð1þ gij + u j + e ij u j Nð0, σ 2 u Þ, e ij Nð0, σ 2 eþ, νð2þ j MVN F ð0, 2 Þ, ν ð1þ ij MVN G ð0, 1 Þ = 1,..., R, i = 1,..., n j, j = 1,..., J, XJ n j = N, j=1 ð3þ whee h indexes the fixed-pat explanatoy vaiables, R is the numbe of esponses, F the numbe of Level 2 factos, and G the numbe of Level 1 factos. The v ð2þ fj, v ð1þ gij ae espectively sets of common factos at Levels 2 and 1 with coesponding uniquenesses u j, e ij : Note that coelations between the factos ae allowed, although we do not fit coelated factos in the pesent aticle. Whee the esponse is binay o odeed, then, fo the undelying popensity distibution, we have e ij Nð0, 1Þ. The subscipt efes to the item, i to the student, and j to 256

6 Multilevel Stuctual Equation Models the school. In this aticle, we shall assume that the diagonal tems of 1, 2 ae 1. The altenative is to fix one o moe loadings and allow 1, 2 to be geneal covaiance matices to be estimated. Goldstein and Bowne (2004) gave a futhe discussion, and the steps needed fo both fomulations ae descibed in the appendix. Seveal othe authos have studied Model 3 and extensions to it. Zhu and Lee (1999) and Fox and Glas (2001) used Makov Chain Monte Calo (MCMC) estimation, the fome based on Gibbs sampling fo a single-level facto model, wheeas the latte authos conside the binay esponse two-level model with a single facto at Level 1 and use Gibbs sampling with a pobit link function. A numbe of authos have extended Model 3, using maximum likelihood pocedues, to include categoical esponses and moe geneal stuctual equation fomulations. Thus, Muthen (1997) consideed applications to latent gowth cuve modeling, and a moe geneal discussion of these models is also given by Muthen (1989, 2002). Moe ecently, a vey geneal famewok fo multilevel stuctual equation modeling is povided by Rabe-Hesketh, Skondal, and Pickles (2004) that includes most of the models to be discussed below. These authos obtain maximum likelihood estimates, typically based on quadatue, although othe authos (e.g., Raudenbush, 1995) use an expectation maximization (EM) algoithm. Song and Lee (2004) fit this model fo mixtues of nomal, binay, and odeed esponses using a mixtue of Gibbs and Metopolis-Hastings sampling. An advantage of caying out the estimation using MCMC methods, as in the pesent aticle, is the ability to incopoate pio infomation in a Bayesian sense and to povide exact inteval estimates fo paametes o functions of paametes. Also, because of the modulaity of the algoithm, it is possible to add additional complexity elatively staightfowadly, including the possibility of incopoating distibutional assumptions othe than nomality. The algoithm descibed in the appendix assumes diffuse pios but is eadily extended to incopoate infomative pio distibutions. It extends pevious wok in paticula by allowing fo missing data and paamete constaints among fixed coefficients. It also poposes the use of the Deviance Infomation Citeion (DIC; Spiegelhalte, Best, Calin, & Van de Linde, 2002) that povides a measue of model complexity and can be used fo compaing nonnested models. In the single-level case, the DIC is analogous to the Akaike Infomation Citeion (AIC) and can be consideed a genealization of this. Unlike othe model fit pocedues such as Bayes factos, it does not equie impope (diffuse) pios such as ae used fo some of the paametes. The pocedues have been implemented using MATLAB (Mathwoks, 2004) and MLwiN softwae (Rasbash, Bowne, & Steele, 2004). Although MLwiN has some basic facilities fo multilevel facto modeling, the algoithm as witten in MATLAB is moe flexible, although computationally slow. In the next section, we descibe the analysis models used in PISA and conduct some simple compaisons using the above fomulation, befoe descibing ou moe detailed analyses. 257

7 Goldstein, Bonnet, and Roche The OECD PISA Models The unidimensionality assumption was used by PISA to detemine the stuctue of the tests as well as the subsequent modeling. The analysis essentially involves two stages (Adams & Wu, 2002). The fist stage consists of fitting Models 1 and 2 to the complete data set using equal loadings. This povides estimates of the intecept paametes b 0. Teating these estimates as known paamete values, a futhe unidimensional Rasch model is fitted to the esponses, but this time including as fixed explanatoy vaiables (conditioning vaiables) a set of scales fomed fom a pincipal components analysis of the data in the student questionnaie. These data include items elated to both the school and social backgound of the students. This is done sepaately fo each county. To take account of the uncetainty in the facto scoes, an estimate of the posteio distibution of the facto scoe fo each student is obtained, and five values ae sampled andomly fom this distibution fo each student. Fo example, if MCMC wee used fo the analysis, these could be five (appoximately) independent values fom the chain of facto scoes fo each student, obtained by selecting values suitably fa apat in the chain. Altenatively, values can be selected fom paallel chains. These plausible values ae then used in subsequent analyses to compae counties and so on. Essentially, five analyses ae pefomed, each one using just one plausible value fo each student, and the analysis estimates ae then combined to obtain infeences. Mislevy (1991) descibed the pocedue. Cetain poblems aise with this appoach. The fist is that although the plausible values may be expected to pefom easonably well fo models that use a subset of the conditioning data, this will not geneally be tue fo vaiables not in the conditioning data set. This includes school-level vaiables and in paticula applies to multilevel analyses whee school is the highe level unit (see Mislevy, 1991). The second poblem is that the uncetainty attached to the intecept paamete estimates is not taken into account, although it is not clea whethe this is a seious poblem. A elated issue is that the intecept paamete estimates (item difficulties) ae based on a model that does not include the conditioning vaiables. The PISA analyses use sampling weights that eflect the achieved sample chaacteistics. In the pesent aticle, we shall ignoe these because they appea to make only small diffeences. In subsequent sections, we show how a fully efficient analysis, avoiding the use of plausible values, can be pefomed. Ou fist simple analysis, howeve, looks at the assumption of equal loadings. We fitted Model 1 acoss the whole data set with and without the equal loadings constaint, and the esults ae given in Tables 1 and 2, with the loading and intecept estimates togethe with thei standad eos. Fo these and subsequent analyses, we used a bun in of 1,000 and a subsequent 5,000 iteations fo the chain. Fo the main paametes of inteest, the fixed-pat coefficients and the loadings, the mixing of the chains is easonable, as it is fo the esidual vaiance estimates. The mixing fo the theshold 258

8 TABLE 1 Compaisons of Intecept Paamete Estimates Multilevel Stuctual Equation Models Equal Loadings (Rasch Model) Unequal Loadings Question Estimate SE Estimate SE R040Q R040Q03A R070Q R070Q R076Q R077Q R083Q R083Q R088Q (1.39) (0.10) 1.00 (1.35) 0.08 (0.13) R091Q R100Q R104Q R104Q R104Q (1.73) 0.04 (0.15) 0.11 (1.91) 0.03 (0.17) R104Q R110Q R110Q R111Q R119Q R122Q03T 0.56 (0.84) 0.05 (0.09) 0.60 (0.89) 0.05 (0.08) R216Q R219Q01E R220Q R225Q R225Q R227Q02T 0.48 (0.91) 0.04 (0.07) 0.46 (0.87) 0.04 (0.07) R227Q R234Q R234Q R237Q R238Q R239Q R245Q R246Q R246Q Note: Theshold paamete estimates fo fou odeed categoy items ae in paentheses. Bun in = 1,000; sample = 5,000. Single-level facto model with 8,299 pupils. 259

9 Goldstein, Bonnet, and Roche TABLE 2 Compaisons of Paamete Estimates Loadings Question Equal Loadings (Rasch Model = 1) Unequal Loadings Estimate SE R040Q R040Q03A R070Q R070Q R076Q R077Q R083Q R083Q R088Q R091Q R100Q R104Q R104Q R104Q R104Q R110Q R110Q R111Q R119Q R122Q03T R216Q R219Q01E R220Q R225Q R225Q R227Q02T R227Q R234Q R234Q R237Q R238Q R239Q R245Q R246Q R246Q Facto vaiance (SE) (0.012) 1.0 DIC (PD) 89,793.4 (5,478) 89,129.7 (5,580) Note: DIC = deviance infomation citeion; PD = effective numbe of paametes. 260

10 Multilevel Stuctual Equation Models paametes is poo using the Gibbs sampling appoach but is satisfactoy when Metopolis-Hastings sampling is used (see appendix, Step 1, Odeed Responses of the algoithm). Facto scoes ae estimated by computing the mean of the chain values fo each student. The vaiances of these means ae also estimated fom the chain, and the inveses of these vaiance estimates ae used as weights in Table 3 to povide a valid analysis compaing the Fench and English mean scoes; standad eos of paametes ae computed using sandwich estimatos. This analysis and that in Table 4 ae thus just weighted least squaes egession models whee we have followed the PISA pocedue of computing a facto scoe fo each student and then using these in subsequent analyses. It is evident fom inspecting the estimates and thei standad eos that thee is stong evidence fo diffeential loadings. This is confimed by compaing the values of the DIC. In the pesent case, thee is a eduction of 664 fo the model with unconstained loadings. The two assumptions fo the loadings also povide somewhat diffeent fixed-pat paamete estimates fo the diffeence between Fance and England, although in both cases the 95% inteval includes zeo. The point estimates fo the equal loadings model have a vaiance about two thids that fo the unequal loadings model, although the esults ae simila in tems of effect sizes to those obtained using PISA pocedues (ONS, 2002). Table 4 extends the compaison by including a gende effect and an inteaction between gende and county. This shows a slightly smalle diffeence in favo of gils in England than in Fance (although not eaching the fomal 5% significance level) with simila esults in tems of effect sizes fo both models. We now compae this pocedue with a single-model appoach whee the explanatoy vaiables ae included in the facto model. In this and subsequent analyses, we use the model with unconstained loadings. Thus, fo a model with the single additional explanatoy vaiable county (denoted by the dummy vaiable x 1 Þ, we have the single-level model y ij = b 0 + b 1 x 1ij + λ ð1þ v ð1þ ij + e ij : ð4þ Note that if we wee to adopt the item cluste effect model of Scott and Ip (2002), Model 4 would become, fo cluste k, y ij = b 0 + b 1 x 1ij + λ ð1þ g kij Nð0; σ 2 k Þ: ðv ð1þ ij + g kij Þ + e ij Instead of a constant b 1, we could allow a diffeent county coefficient fo each esponse (b 1 Þ, but inteest hee lies in an oveall compaison between counties so that these ae constained to be equal. We discuss how to intepet this paamete below. 261

11 Goldstein, Bonnet, and Roche TABLE 3 Single-Level Weighted Least Squaes Models fo County Compaisons Estimate (SE) Paamete Equal Loadings (Rasch Model) Unequal Loadings Intecept 0.05 (0.008) (0.012) County (England Fance) (0.011) (0.018) TABLE 4 Single-Level Weighted Least Squaes Models fo County and Gende Compaisons Estimate (SE) Paamete Equal Loadings (Rasch Model) Unequal Loadings Intecept (0.011) (0.012) County (England Fance) (0.016) (0.024) Gende (male female) (0.016) (0.023) County Gende (0.023) (0.034) The OECD analyses essentially fit the equivalent of the following model: y ij = b 0 + ðx ðcþ b ðcþ Þ ij + λ ð1þ v ð1þ ij + e ij, v ð1þ ij = a 1 x 1ij + v ij v ð1þ ij Nð0, 1Þ, v ij Nð0, σ2 v Þ, ð5þ albeit with equal loadings, whee the tem (X ðcþ b ðcþ Þ ij epesents the effect of the conditioning vaiables. Howeve, the two distibutional assumptions in Model 5 will not in geneal both be satisfied; one case whee they ae satisfied is whee fν, x, ν g have a joint multivaiate nomal distibution. Instead, the following model avoids this poblem: y ij = b 0 + λ ð1þ δ ij Nð0, σ 2 δ Þ, v ð1þ ij + e ij, v ð1þ ij = a 1 x 1ij + δ ij ð6þ which on substitution gives y ij = b 0 + a 1 λ ð1þ x 1ij + λ ð1þ δ ij + e ij : ð7þ This is a stuctual equation model, and Model 6 is not equivalent to Model 4. It is an example of a Multiple Indicato Multiple Indicato Cause (MIMIC) model (Muthen, 1989). Moe geneally, the stuctual pat of the model will include 262

12 futhe vaiables of inteest at individual and school levels. A choice between Models 6 and 4 in any paticula case can be detemined by the model that has the bette fit to the data. We note, howeve, that the intepetation of Model 4 is not completely staightfowad because the esponses have diffeent vaiances; only the Level 1 esidual vaiances ae equal. Thus, although the coefficients ae equal acoss esponses in Model 4, the esponse distibutions themselves ae not standadized. We theefoe use models with stuctual pedictos based on Model 6 in the following analyses. In the pesent case, fitting Model 4 to the data gives an estimate fo b 1 of (0.019); with equal loadings, this is (0.011). Fitting Model 6 gives an estimate fo a 1 of (0.026) with a DIC value of 89,145.5, compaed with a athe simila value of 89,142.8 fo Model 4. Note that in the (Rasch) case of equal loadings, Models 4 and 6 do become equivalent. We also note that fo the case of moe than one facto at a level and also fo factos at moe than one level, we may wish to model each facto scoe as a function of explanatoy vaiables, in which case Model 4 is inadequate and we must use models that ae an extension of Model 6. We now look at the case of the two-level model, whee Model 4 becomes y ij = b 0 + b 1 x 1ij + λ ð1þ v ð1þ ij Multilevel Stuctual Equation Models + λ ð2þ v ð2þ j + u j + e ij, ð8þ and the PISA analyses fit essentially the (vaiance components) model, y ij = b 0 + λ ð1þ v ð1þ ij + e ij, v ð1þ ij = a 1 x 1ij + u j + v ij v ij Nð0; 1Þ, v ij Nð0, σ2 v Þ, u j Nð0, σ 2 u Þ, ð9þ whee again we have the poblem that the distibutional assumptions cannot in geneal simultaneously be satisfied. An altenative two-level model is y ij = b 0 + λ ð1þ v ð1þ ij δ ij Nð0, σ 2 v Þ, u j Nð0, σ 2 u Þ, + e ij, v ð1þ ij = a 1 x 1ij + u j + δ ij which becomes, on substitution, y ij = b 0 + a 1 λ ð1þ x 1ij + λ ð1þ δ ij + λ ð1þ u j + e ij : ð10þ Compaing Model 10 with Model 4, we see that effectively the Level 1 and Level 2 loading vectos ae constained to be equal and thee is no additional Level 2 esidual tem in Model 10. Thus, Model 10 becomes a highly constained model. We show below, howeve, that an extension of Model 10 does povide a useful model. Thus, Table 5 shows a simple two-level model fit fom which it is clea that the Level 1 and Level 2 loading vectos ae vey diffeent. Futhemoe, even with equal loadings, Models 8 and 10 ae not equivalent. 263

13 Goldstein, Bonnet, and Roche TABLE 5 Two-Level Model With One Facto at Each Level Level 1 Loading Level 2 Loading Question Intecept (Theshold) Estimate SE Estimate SE R040Q R040Q03A R070Q R070Q R076Q R077Q R083Q R083Q R088Q (1.33) R091Q R100Q R104Q R104Q R104Q (1.94) R104Q R110Q R110Q R111Q R119Q R122Q03T 0.60 (0.92) R216Q R219Q01E R220Q R225Q R225Q R227Q02T 0.46 (0.89) R227Q R234Q R234Q R237Q R238Q R239Q R245Q R246Q R246Q DIC (PD) 88,579.5 (6,148) Note: Bun in = 1,000; sample = 5,000. DIC = deviance infomation citeion; PD = effective numbe of paametes. 264

14 Multilevel Stuctual Equation Models In the next section, we develop Model 7, intoducing futhe explanatoy vaiables, and in a late section, we look at the dimensionality stuctue of the data. One-Dimensional Models With Seveal Explanatoy Vaiables In view of the poblems of diffeential lengths of schooling discussed in the intoduction, we fit in ou initial models the inteactions between age and county. Age is categoized as a dummy vaiable Januay to August vesus Septembe to Decembe biths. The additional use of gade is poblematic. Fo example, if we compae the Septembe to Decembe biths in the two counties, all the English ae in Gade 10, wheeas the epeating Fench ae in Gade 9. If we only compae Gade 10 pupils, then the Fench will tend to do elatively bette because of the stong negative association between pefomance and epetition. Fo the childen bon in Januay to August, all the English ae in Gade 11, wheeas the Fench nonepeates ae in Gade 10, and the epeates ae in Gade 9. Fo these easons, we have not used gade in ou compaisons, but a special analysis of the effect of gade in the 2003 PISA suvey in Fance will be epoted elsewhee. Table 6 extends Model 6 by including age-goup, gende, and the fist-ode inteactions of age-goup, gende, and county; inteactions between age tends and county ae negligible and not displayed. We see that thee is an advantage to the olde pupils and a negative tend with month of bith fo the peiod Januay to August, with the olde childen scoing highe and a gende effect in favo of gils. Thee is little evidence fo a tend fo the peiod Septembe to Decembe, o fo a county diffeence, o fo any inteactions, except possibly fo county by gende. Two-Level Models We now fit the full two-level facto model with stuctual pedictos and a single facto at each level togethe with just gende and age tems. No inteactions ae significant, and fitting a coefficient fo county also gave a high standad eo fo the England Fance diffeence, as well as a athe badly mixing chain with vey high seially coelated values. Running the chain fo longe povided no evidence that the coefficient was significant. The esults ae pesented in Table 7. Thee emains a lage gende diffeence in favo of gils and an advantage to the olde pupils. Thee is a negative tend with month of bith fo the peiod Januay to August and also evidence fo a positive tend fo the peiod Septembe to Decembe. The latte seems difficult to explain. Exploing Dimensionality We now exploe the dimensionality stuctue of the data. We have pefomed a seies of analyses at a single level that establishes the existence of at least two dimensions. In the PISA analyses, items wee a pioi selected fo membeship of the thee sepaate poficiencies, with each item identified with just one poficiency. 265

15 Goldstein, Bonnet, and Roche TABLE 6 Single-Level Facto Model fo County and Age-Goup Compaisons Paamete Estimate (SE) County (England Fance) (0.031) Gende (male female) (0.042) Age goup (Septembe Decembe = 1) (0.054) County Age Goup (0.047) County Gende (0.044) Age Goup Gende (0.056) Age coefficient (Januay August) (0.007) Age coefficient (Septembe Decembe) (0.020) DIC (PD) 89,136.9 (5,570) Note: Stuctual model. Intecepts ae not shown. Bun in = 1,000; sample = 5,000. Unequal facto loadings. Age (Januay August) is defined as bon in Januay (0), Febuay (1),... August (7), Septembe Decembe (0). Age (Septembe Decembe) is defined as bon in Septembe (0), Octobe (1),...Decembe (3), Januay August (0). Age is measued in months. TABLE 7 Two-Level Facto Model fo Gende and Age-Goup Compaisons Paamete Estimate (SE) Gende (male female) (0.031) Age goup (Septembe Decembe = 1) (0.055) Age coefficient (Januay August) (0.009) Age coefficient (Septembe Decembe) (0.020) DIC (PD) 88,592.2 (6,223) Note: Stuctual model. Intecepts not shown. Bun in = 1,000; sample = 5,000. Unequal loadings. DIC = deviance infomation citeion; PD = effective numbe of paametes. The items used in ou analyses of the Retieving Infomation poficiency subscale ae theefoe unique to that scale. In the pesent analyses, we fit othogonal factos so that we can detect dimensions along which counties may diffe meaningfully (see Steele & Goldstein, 2006, fo an example with coelated factos). In common with all facto models, we have choices to make to ensue identifiability. In the models of this aticle whee 1, 2 ae identity matices, a simple pocedue at any given level is to set, fo the jth facto (j > 1Þ, λ k = 0, k = 1,...j 1 (Goldstein & Bowne, 2004). In Table 8, we do this to fit two Level 1 factos, setting the fist item of the second facto to zeo. We stat with a model including just the intecepts and a fixed effect fo county vaying acoss esponses. This model (DIC = 87,028.9) povides a bette fit and somewhat diffeent loading estimates compaed with a model fitting intecepts only (DIC = 87,687.3) and a much bette fit than the basic model fo one facto (DIC = 88,471.3). 266

16 Multilevel Stuctual Equation Models TABLE 8 Single-Level Facto Model Loading Estimates With Two Uncoelated Factos at Level 1 Level 1 Facto 1 Level 1 Facto 2 Question Estimate SE Estimate SE R040Q R040Q03A R070Q R070Q R076Q R077Q R083Q R083Q R088Q R091Q R100Q R104Q R104Q R104Q R104Q R110Q R110Q R111Q R119Q R122Q03T R216Q R219Q01E R220Q R225Q R225Q R227Q02T R227Q R234Q R234Q R237Q R238Q R239Q R245Q R246Q R246Q DIC 87,028.9 Note: Intecept and county fixed-pat pedictos without equality constaints. DIC = deviance infomation citeion. 267

17 Goldstein, Bonnet, and Roche We see clea evidence in Table 8 fo at least two factos. If we fix all the loadings below 0.2 to 0 and eestimate, we obtain the esults in Table 9, with a somewhat highe value of DIC (87,312.6). The intepetation of factos estimated in this way is poblematic because a diffeent choice of zeo loading will, in geneal, lead to diffeent loading pattens. In fact, using diffeent stating values, we find that the loadings ae not stable, moving fom one facto to the othe. To exploe the vaious possibilities is time-consuming, and we have not done this because ou pincipal aim is to see whethe moe than one dimension exists; futhe factos can be fitted in simila ways, howeve. Anothe appoach would be to fit simple-stuctue models whee each item loads on only one facto at each level, but the factos ae allowed to be coelated. This involves choosing appopiate subsets of items, and we have not pusued this. An exploation of the facto space will need to make choices about the loadings to be fixed based on substantive consideations of item fomats, positioning, and content. In addition, when caying out such an exploation, we should fit a two-level model and fit explanatoy vaiables such as age and county and also allow fo the possibility that facto stuctues may vay acoss counties. It may also be useful to cay out exploatoy analyses sepaately at each level based on a sepaate modeling of estimated Level 1 and Level 2 esidual covaiance matices (see, e.g., Rowe, 2003). A Constained Multilevel Stuctual Model Rathe than fitting the following Model 11 and estimating the loadings fo each change in model paametes, we can conside fitting a standadizing model such as Model 6 and subsequently teating the posteio mean estimates of the loadings as fixed in futhe analyses. The advantage of this appoach is that we ae dealing with essentially the same factos, as defined by the loadings, in each analysis. Reestimating the loadings fo each fitted model will complicate intepetation. Clealy, vaious choices fo the standadizing model ae possible fo example, fitting a 2-level stuctue with the loadings fo each esponse constained to be equal acoss levels. In pactical applications, sensitivity analyses can be pefomed to see whethe infeences ae stongly affected by diffeent choices. We pesent hee only the esults fom fitting a single facto, but the model can be extended by fitting stuctual paametes in the case of moe than one facto. y ij = b 0 + Xq h=1 b h x hij + λ ð1þ v ð1þ ij = Xp a k z kij + v ij + u j, k=1 v ð1þ ij + e ij ð11þ 268

18 Multilevel Stuctual Equation Models TABLE 9 Single-Level Facto Model Loading Estimates With Two Uncoelated Factos at Level 1, Setting Loadings < 0.2 in 7 to Zeo Level 1 Facto 1 Level 1 Facto 2 Question Estimate Estimate R040Q R040Q03A R070Q R070Q R076Q R077Q R083Q R083Q R088Q R091Q R100Q R104Q R104Q R104Q R104Q R110Q R110Q R111Q R119Q R122Q03T R216Q R219Q01E R220Q R225Q R225Q R227Q02T R227Q R234Q R234Q R237Q R238Q R239Q R245Q R246Q R246Q DIC 87,312.6 Note: Intecept and county fixed-pat pedictos without equality constaints. DIC = deviance infomation citeion. 269

19 Goldstein, Bonnet, and Roche whee the stuctual pedictos z k ae distinct fom the fixed-pat pedictos x h and the Level 2 andom effect is incopoated into the Level 1 stuctual model. We may allow diffeent vaiances fo diffeent goups at both Level 1 and Level 2, and in the pesent case, we fit diffeent county vaiances at both Level 1 and Level 2. The second line of Model 11 becomes v ð1þ ij = Xp a k z kij + v 1ij z 1j + v 2ij z 2j + u 1j z 1j + u 2j z 2j k=1 z 1j = 1 if England, 0 if Fance, z 2j = 1 z 1j ð12þ v 1 Nð0, σ2 v1 Þ, v 2 Nð0, σ2 v2 Þ, u 1 Nð0; σ2 u1 Þ, u 2 Nð0, σ2 u2 Þ: When Models 11 and 12 ae combined, because the Level 1 loadings ae assumed known, we have the andom coefficient facto model y ij = b 0 + Xq h=1 + v 1ij λð1þ b h x hij + Xp k=1 z 1j + v 2ij λð1þ a k z kij λ ð1þ z 2j + u 1j λð1þ z 1j + u 2j λð1þ z 2j + e ij : ð13þ We fit Model 13 with the Level 1 loadings of Table 5 and the intecept and stuctual pedictos of Table 7 without the gende and county inteactions (which ae not significant at the 5% level), and we obtain the esults in Table 10. We have also fitted Model 13 whee the pedictos ae in the fixed pat of the model athe than the stuctual pat (Table 11). We note that fo the stuctual model, the coefficients tend to be smalle compaed to thei standad eos than fo the fixed-pat pedicto coefficient model, and the latte is also a bette fit with a DIC of 88,559.2, compaed with 88,573.1 in the stuctual model. We see that the atio of Level 2 to Level 1 plus Level 2 facto vaiances, the vaiance patition coefficient (VPC; Goldstein, Bowne, & Rasbash, 2002), is 21% fo England and 49% fo Fance. These values ae simila to those pesented in PISA (Adams & Wu, 2002). Goldstein (2004) suggested that the explanation fo the high value fo Fance is that the data contain a mixtue of pupils fom Gades 9 and 10. As pointed out above, epetition implies a geate vaiation among schools. We have theefoe conducted an analysis fo Fance only using Model 13 and fitting only intecept tems, and we find that fo Gade 9 (collège) pupils, the Level 2 vaiance estimate is 0.19, and fo Level 1, the vaiance estimate is 0.79, giving a VPC of 19%. Fo lycée pupils, the vaiances ae 0.14 and 0.54 with a VPC of 21%; thus, the VPC estimate fo each school type is close to the English estimate. This explanation fo the appaently high between-school vaiation also accounts fo esults fom the Tends in Intenational Mathematics and Science Study (TIMSS; Mullis et al., 2001), which show simila values fo the two counties and whee the sampling fo Fance was caied out only in collège. 270

20 TABLE 10 Two-Level (Stuctual) Facto Model fo County, Gende, and Age-Goup Compaisons Multilevel Stuctual Equation Models Paamete Estimate (SE) County (England Fance) (0.072) Gende (male female) (0.035) Age goup (Septembe Decembe = 1) (0.044) Age coefficient (Januay August) (0.008) Age coefficient (Septembe Decembe) (0.021) s 2 v1 (Fance Level 2 vaiance) 0.70 (0.08) s 2 v2 (Fance Level 1 vaiance) 0.74 (0.04) s 2 u1 (England Level 2 vaiance) 0.35 (0.05) s 2 u2 (England Level 1 vaiance) 1.30 (0.05) DIC (PD) 88,573.1 (6,146) Note: Intecepts not shown. Bun in = 1,000; sample = 5,000. Loadings fixed to Level 1 loadings in Table 5. DIC = deviance infomation citeion; PD = effective numbe of paametes. TABLE 11 Two-Level Facto Model fo County, Gende, and Age-Goup Compaisons Paamete Estimate (SE) County (England Fance) (0.032) Gende (male female) (0.017) Age goup (Septembe Decembe = 1) (0.026) Age coefficient (Januay August) (0.004) Age coefficient (Septembe Decembe) (0.011) s 2 v1 (Fance Level 2 vaiance) 0.70 (0.08) s 2 v2 (Fance Level 1 vaiance) 0.75 (0.04) s 2 u1 (England Level 2 vaiance) 0.36 (0.05) s 2 u2 (England Level 1 vaiance) 1.29 (0.05) DIC (PD) 88,559.2 (6,196) Note: Explanatoy vaiables in fixed pat. Intecepts ae not shown. Bun in = 1,000; sample = 5,000. Loadings fixed to Level 1 loadings in Table 5. DIC = deviance infomation citeion; PD = effective numbe of paametes. Discussion Ou analyses and discussion have shown that compaisons between two educational systems with diffeent pupil pogession stuctues ae complex. The combination of diffeent ages of stating school and diffeent allocation to yea goups on the basis of bith date and epetition of gades makes any meaningful 271

21 Goldstein, Bonnet, and Roche compaison extemely difficult. Although we have hee compaed only England and Fance, ou view is that the same poblems occu when nonepetition systems ae compaed with those that have impotant pecentages of epetition, such as those of Spain, Potugal, and Belgium. We have demonstated that, even within a single poficiency domain, the data stuctue appeas to contain at least two dimensions, although we have not conducted a full multilevel analysis of the dimensionality stuctue, no have we attempted to identify and label factos as such. Nevetheless, even in the onedimensional case, the (Rasch) assumption of equal item loadings is not suppoted by the data. We have shown how a valid multilevel facto model can be fitted and, in paticula, how to stuctue the facto vaiances at each level in ode to popely study between-school vaiability. Model 13 is an example of a andom coefficient facto model that can eadily be extended to include futhe explanatoy vaiables such as gende o age and also, fo example, to coss-classifications. Thus, the full ange of multilevel modeling pocedues can be incopoated into these analyses, and such analyses will often lead to infeences that diffe fom those based on single-level models. The pocedue is also much simple than the plausible value pocedue poposed by the OECD because it equies only a single fitting of a multilevel model. An issue with this appoach is the choice of loadings to use. In ou case, we have chosen a set of loadings fom a two-level model with a single facto at each level. Othe choices ae possible, such as including fixed pedictos in the initial model. In geneal, it would be useful to pefom sensitivity analyses to detemine whethe such choices substantially affect infeences. Once the loadings ae chosen, they effectively define the latent factos, and it is meaningful to make compaisons acoss subgoups only if we then use the same set of loadings in all analyses. We have not taken into account the uncetainty in the estimates of the loadings. Rathe, we take the view that the fist stage that detemines the values of these loadings povides a pactically useful metic fo futhe analysis. Nevetheless, it is impotant to have a suitably lage sample to ensue that sampling vaiability is small. If necessay, we can incopoate pio infomation, fo example, fom pevious studies, into the estimation of these paametes. Finally, although the main thust of this aticle is to pesent a methodology fo handling complex multilevel data in compaative studies, we should not ignoe the seious dawback of a lack of longitudinal data in suveys such as PISA and othe simila suveys such as TIMSS. Without such measues of pio pefomance on the same sample of students, it is not possible to ovecome the compaability poblems that aise fom the diffeent ways in which educational systems ae oganized, as we have descibed. Likewise, without such pio measues, it is not possible to attibute any obseved diffeences between systems o subgoups to the education systems pe se athe than, fo example, social, cultual, o economic factos. Goldstein (2004) discussed this issue in moe detail in the context of the stated aims of the PISA study. 272

22 Multilevel Stuctual Equation Models Appendix A Makov Chain Monte Calo Algoithm fo Two-Level Facto Analysis With Extension to a Stuctual Equation Model The basic steps of this algoithm ae given by Goldstein and Bowne (2004): They ae extended hee to include odeed categoical esponses, constaints among fixed paametes, and stuctual model pedictos. We wite a basic two-level facto model fo nomal esponses as y ij = X h b h x hij + XF λ ð2þ f νð2þ fj + XG f =1 g=1 u j Nð0, σ 2 u Þ, e ij Nð0, σ 2 e Þ, νð2þ j λ ð1þ g νð1þ gij + u j + e ij MVN F ð0, 2 Þ, ν ð1þ ij MVN G ð0, 1 Þ ða1þ = 1,..., R, i = 1,..., n j, j = 1,..., J, XJ j=1 n j = N, whee the data stuctue is that fo a multivaiate two-level model with esponses nested within individuals within schools. The subscipt indexes the esponses. We have F factos at Level 2 and G factos at Level 1 with coesponding coefficients o loadings. Whee F o G is > 1, we must intoduce constaints on the loadings fo second and subsequent factos. A common choice is to set, fo the jth facto (j > 1Þ, λ k = 0, k = 1,...j 1. In the standad implementation, we assume independent factos with known vaiance matices = I: The following steps genealize this to allow facto vaiances and covaiances to be estimated. Gibbs sampling is used, except fo facto covaiances whee Metopolis-Hastings sampling is used. The esponse vaiables can be nomally distibuted, binay, o odeed categoical, with any mixtue of these. In addition, we allow a stuctual equation model of the following type to be fitted. Fo a set of factos, say ν = fν 1,...ν G g at Level 1 (dopping the supescipt), we can wite the following model fo a set of stuctual explanatoy vaiables fz k g: v gij = X k g gk z gkij + v gij, v ij MVN Gð0, 1 Þ, ða2þ whee we efe to the coefficients λ gk as stuctual paametes. Afte substitution, the Level 1 component of the fist line of Model A1 becomes X by ij = X b h x hij + X h k y ij =by ij + e ij, v ij MVN Gð0, 1 Þ: g λ g g gk z gkij + X g λ g v gij ða3þ 273

23 Goldstein, Bonnet, and Roche In the following steps, we give details of how to implement the algoithm. The basic code is witten in MATLAB (Mathwoks, 2004) and is being incopoated into MLwiN (Bowne, 2004; Rasbash et al., 2004) by extending the existing facto-fitting pocedues. Default diffuse pios ae assumed thoughout (Bowne, 2004). Fom suitable stating values, the following steps ae caied out. Default stating values ae to set facto scoes and facto loadings to 1. Fixed coefficient stating values ae estimated fom oveall esponse popotions assuming a model with intecept tems only. Step 1 In this step, any binay o odeed esponses ae eplaced by a value sampled fom a nomal distibution, conditional on cuent paamete values as follows. Binay Response Fo each binay esponse, we sample fom a standad nomal distibution. Whee a binay esponse is missing, the nomal value is imputed in the next step. 1. Compute the cuent pedicted value fo binay esponse vaiable ^y ij = X T ^b + XF ^λ ð2þ f ^νð2þ fj + XG f =1 g=1 ^λ ð1þ g ^νð1þ gij + ^u j; whee ^ denotes the cuent value. 2. Compute, fo all i, j, P = Z ^y fðtþdt: 3. Geneate N unifom andom numbes (0, 1) into R, whee N is the numbe of Level 1 units. 4. Calculate T = YððJ P ÞR + P Þ + ðj YÞP R ; whee J is an (N 1Þ vecto of ones. This povides a set of unifom andom numbes fom (0; P Þ o (P ; 1Þ, depending on Y, the vecto of obseved esponses. 5. Choose e, the equied daw fom N(0, 1) and hence Y = ^y + e, fom the invese nomal distibution, given T. Note that this constains the Level 1 vaiance to be equal to 1. Odeed Responses Suppose we have a p-categoy esponse, numbeed 1,...p. As above, we conside the pobit link popotional odds model: 274