Center for Advanced Studies in Measurement and Assessment. CASMA Research Report. Using G Theory to Examine Confounded Effects: The Problem of One

Transcription

1 Cente fo Advaned Studies in Measuement and Assessment CASMA Reseah Repot Numbe 51 Using G Theoy to Examine Confounded Effets: The Poblem of One Robet L. Bennan 1 Januay Robet L. Bennan is E. F. Lindquist Chai in Measuement and Testing and Founding Dieto, Cente fo Advaned Studies in Measuement and Assessment (CASMA). Bennan is also a onsultant to the College Boad.

2 Confounded Effets Cente fo Advaned Studies in Measuement and Assessment (CASMA) College of Eduation Univesity of Iowa Iowa City, IA 54 Tel: Web: All ights eseved ii

3 Confounded Effets Contents Abstat iv 1 A Single Pompt 1.1 The p Design with a Single Pompt Complexities and Potential Misundestandings Stutual vs. Statistial Bias Notational and Vebal Complexity Repliations do Not Eliminate Confounding Multiple G Studies A Single Pompt fo Diffeent Pompt Types 5.1 UAO, G Study, and Pojeted D study Results D Studies with (Z, A) Random D Studies with (Z, A) Fixed Results fo n = Results fo n = Coeffiient Alpha and (R, Z, A) A Single Rate (e.g., Automated Soing Engine) Example R Random and (Z, A) Teated as Random R Random and (Z, A) Teated as Fixed Results fo n = Othe Issues Single Random vs. Fixed ASE Taining ASEs Compaing Multiple ASEs Compaisons Involving Human Rates and an ASE Conluding Comments 0 5 Refeenes 1 6 Appendix: Eo Vaiane Relationships in Tables 1 and 4 iii

4 Confounded Effets Abstat Using the oneptual famewok of G theoy, this pape addesses the issue of onfounded effets that aise when the data olletion design fo an opeational measuement poedue has a single ondition fo one o moe faets. In this ase, effets fo at least two faets will be onfounded, whih leads to ambiguities in intepeting esults. We all this the the poblem of one. Ambiguities ae patiulaly salient when one of the onfounded effets is andom and one is fixed. Confounding aises fequently in pefomane testing (e.g., essay testing), although onfounding tends to be negleted o, wose still, uneognized. Initial attention fouses on using a single pompt suh as an essay, followed by bief disussions of impotant mattes that elate to onfounding, in geneal. Then onsideation is given to designs in whih single pompts ae ompletely onfounded with multiple fixed pompt types. The thid majo setion onsides onfounding assoiated with using a single ate suh as an automated soing engine. This pape is foused mainly on oneptual issues and how single onditions of faets almost always lead to bias in eo vaianes and oeffiients. Undestanding of these issues neessitates aeful distintions between: (a) the univese of admissible obsevations and the atual measuement poedue; and (b) whih faets ae fixed and whih ae andom. In a sense, this pape hallenges etain aspets of the onventional wisdom in psyhometis iv

5 Confounded Effets Genealizability (G) theoy offes an extensive oneptual famewok and a poweful set of statistial poedues fo addessing numeous measuement issues. The defining teatment of G theoy is a monogaph by Conbah, Glese, Nanda, and Rajaatnam (197). Bennan (001) povides the most extensive uent teatment of G theoy. It is assumed hee that eades have some familiaity with G theoy. The pape addesses the issue of onfounded effets in G theoy that aise when a data olletion design fo an opeational measuement poedue has a single ondition fo one o moe faets. In this ase, effets fo at least two faets will be onfounded, whih leads to ambiguities in intepeting esults. We all this the the poblem of one. Ambiguities ae patiulaly salient when one of the onfounded effets is andom and one is fixed. Confounding aises fequently in pefomane testing (e.g., essay testing), although onfounding tends to be negleted o, wose still, uneognized. In thei oiginal teatment of G theoy, Conbah et al. (197) aknowledged the issue of onfounded effets and even outinely used a elatively omplex notational system to denote them. Bennan (001, see espeially pp. 6 63) also deals with onfounded effets. Both books, howeve, teat onfounded effets pimaily fom the point of view of identifying the effets that ae onfounded in a nested effet. Hee, we ae onsideing onfounded effets that aise pimaily beause of single onditions of faets in an atual measuement poedue. This is losely assoiated with the issue of hidden faets (see, fo example, Bennan (001, pp ), but the teatment of onfounded effets in this pape is muh moe extensive than that in Conbah et al. (197) o Bennan (001). This pape is foused mainly on oneptual issues and how single onditions of faets almost always lead to bias in eo vaianes and oeffiients. Undestanding of these issues neessitates aeful distintions between: (a) the univese of admissible obsevations (UAO) and the atual measuement poedue; and (b) whih faets ae fixed and whih ae andom. Failue to eognize these issues is muh moe pevasive that is geneally aknowledged, whih auses onsideable ambiguity in the intepetation of many psyhometi analyses. It is impotant to note, howeve, that G theoy does not intodue o ause these ambiguities; athe, G theoy povides the only uent famewok fo oheently examining these issues. Setion 1 fouses on onfounding assoiated with using a single pompt suh as an essay. Setion 1 also inludes disussions of impotant mattes that elate to onfounding, in geneal. Setion extends Setion 1 to designs that involve multiple pompt types. Setion 3 fouses pimaily on onfounding assoiated with using a single ate (e.g., and automati soing engine). The distintion between a single pompt and a single ate is somewhat atifiial sine both of them ae faets, and many of the oneptual and tehnial issues assoiated with a single pompt ould apply to a single ate, and vie-vesa. Sine onfounding is suh a hallenging topi, howeve, it is useful to disuss mattes in familia Fo example, if i is ossed with h in the univese (i.e., i h), but i is nested within h in a design (i.e., i:h), then we say that i:h involves the onfounding of i and ih fom the ossed univese. 1

6 Confounded Effets ontexts. (Gao, Bennan, & Guo, 015, onside some types of onfounding that ou with single onditions of an oasion faet, as does Bennan, 001, pp ) 1 A Single Pompt We begin with a seemingly simple example fo a single pompt that tuns out to be moe omplex than might be expeted. Then, in Setion we tun to a moe ompliated example in whih pompt and pompt-types ae onfounded. 1.1 The p Design with a Single Pompt Suppose the UAO has two ossed faets, ates () and pompts (z), that ae ossed with pesons. Futhe, suppose that p z, with the population of pesons being lage and the two faets having lage numbes of onditions. By ontast, suppose the G study data ae olleted using multiple pesons and multiple ates, but only one pompt. The G study data, then, onstitute a p design, but the data ae ompletely blind to the existene of multiple pompts in the UAO. In fat, the single pompt is hidden in the data (i.e., onfounded with evey single data element), whih means that any analysis of the data effetively teats pompt as a single fixed ondition. A typial G theoy analysis of the data fo the p design would yield estimates of thee vaiane omponents that ae usually designated (p), (i), and (pi). Then, typially, vaious D study statistis would be omputed, suh as elative eo vaiane and a genealizability oeffiient: and (p) (5) = (1) n (p) Ep =. () (p) + (5) Fo the UAO onsideed hee, howeve, these epesentations of (5) and Ep ae quite misleading, beause they do not eflet the influene on the G study data of the single level of the z faet. In this ase, theefoe, a moe desiptive vesion of Equations 1 and is and (p z) (5) = (3) n (p z) Ep =, (4) (p z) + (5 z) whee z designates that thee is a single fixed level of z in the G study data.

7 Confounded Effets and Using poedues disussed in Bennan (001, pp ), it follows that (p) + (pz) (5) = n (5) (p) + (pz) Ep =, (6) (p) + (pz) + (p)/n + (pz)/n whee the vaiane omponents in Equations 5 and 6 ae fo the UAO in whih z is a andom faet. Equations 5 and 6 ae misleading elative to the UAO, howeve. In patiula, sine the investigato has speified that the z faet is andom in the UAO, and sine n z = n z = 1, the equations fo (5) and Ep that eflet the investigato s intent ae and (p) + (pz) (5) = (pz) + (7) n (p) Ep =. (8) (p) + (pz) + (p)/n + (pz)/n Clealy, (5) in Equation 5 will be an undestatement of the intended value given by Equation 7. By ontast, Ep in Equation 6 will be an ovestatement of the intended value given by Equation 8. In shot, the G study p design based on a single ondition of the z faet leads to bias in eo vaianes and oeffiients. The fundamental ause of this bias is that the G study p design has a single fixed level of z, whih makes it impossible to disentangle (p) and (pz). 1. Complexities and Potential Misundestandings In the above disussion and thoughout this pape, aeful attention needs to be given to vaious issues, omplexities, and potential misundestandings that an aise when onfounding is pesent. Again, these issues aise essentially beause thee is a mismath between a UAO and the design of an opeational measuement poedue Stutual vs. Statistial Bias In taditional statistis and psyhometis liteatue, almost always bias efes to bias in numeial estimates of paametes aused by the use of etain estimatos that ae known to have the popety that the expeted value of the estimato does not equal the paamete (e.g., Bayesian estimatos ae usually biased). 3 In this pape, any efeene to bias has nothing to do with statistial bias. 3 As disussed by Bennan (001), in G theoy the usual ANOVA estimatos of vaiane omponents ae unbiased. 3

8 Confounded Effets Rathe, hee bias is to be undestood in a stutual sense aused by the fat that the design of an opeational measuement poedue fails to mio the intended UAO. That is, the design of the opeational measuement poedue is naowe that the UAO and/o ambiguous with espet to one o moe haateistis of the UAO. When suh a mismath ous between the UAO and the measuement poedue, we sometimes efe to it as bias, but we avoid using the phase biased estimates beause it is so losely tied to notions of statistial bias. Fo the same eason, we sometimes use tems suh as ovestate o undestate, athe than oveestimate o undeestimate. Thee ae similaities between esults fo the onfounding disussed in this setion and Kane s (198) onsideation of the eliability-validity paadox, as summaized by Bennan (001, pp ). Fo example, in Kane s famewok Ep in Equation 6 is fo a estited univese and Ep in Equation 8 is fo an unestited (o taget) univese of genealization. Kane s teatment, howeve, does not fous pimaily on onfounding as disussed in this pape. This pape and Kane s ae not ontaditoy, howeve; indeed, they ae moe popely viewed as omplementay. 1.. Notational and Vebal Complexity Confounding is a oneptually hallenging topi that is vey diffiult to epesent fully and unambiguously in notation and/o wods, without using expessions that an be ompliated o onfusing. In this pape, we usually opt fo expessions that ae easonably simple, even though suh expessions ould be misundestood if taken out of ontext. Fo example, it was stated above that: It is evident that, using the G study data, (5) in Equation 5 will be an undestatement of the intended value given by Equation 7. Stitly speaking, the G study data do not povide the paametes in Equation 5. We ould say something like the G study data an be used to estimate (5) in Equation 5, but then the wod estimate would likely tigge notions of statistial basis, whih is not a matte disussed in this pape. The above statement is also a bit misleading beause the (5) statisti is typially viewed as a D study statisti, but dawing that distintion in the above disussion seems unneessaily speifi. In this pape we typially use G study to efe to the set of data fo an opeational measuement poedue, as opposed to the UAO whih may ontain moe faets than those that ae identifiable in a G study. Ou pimay onen is to examine the onsequenes of having faets in the UAO that ae onfounded in a patiula G study. 4

9 Confounded Effets In papes that addess othe featues of G theoy, it is often moe natual to assoiate the phase measuement poedue with a D study. Hee, we geneally efe to D study wheneve we want to fous attention on paametes suh as eo vaianes and oeffiients that ae based on speifi D study sample sizes, with expliit onsideation of whih faets ae fixed and whih ae andom. Note that we do not onside ases in whih the G and D study ae based on diffeent sets of data Repliations do Not Eliminate Confounding In statistis and psyhometis, doubts about the edibility of esults fo a single study get edued if sample sizes inease and/o the study is epliated and the esults fo the two studies ae simila. This is not tue when both studies involve the same type of onfounding. Confounding is a stutual poblem that pesists with epliated studies and/o lage sample sizes. A ommon example of this misundestanding ous when an entity laims that its essay soing poedues ae highly eliable based on the fat that oelations ae all quite high fo a lage numbe of p analyses with two ates, eah of whih uses a single, difeent pompt. Sine evey one of these analyses uses a single pompt, the oelation is given by Equation 6 with n = 1 (see Bennan, 001, p. 19). The uial point to note is that the numeato of Equation 6 involves (pz) fo evey one of these analyses. 4 This inflates the oelation elative to Equation 8 with n = 1, whih is fo the intended UAO in whih pompts ae andom. In shot, aveaging ove multiple analyses, all of whih involve a diffeent, single pompt simply popagates the onfounding ove the multiple analyses. Doing so does not eliminate, o even mitigate, the onfounding Multiple G Studies In the above disussion of the G study p design with a single fixed level of z, it was noted that it is impossible to disentangle (p) and (pz). That is tue if the only available G study is fo the p design. This poblem an be iumvented, howeve, if thee is an auxillay G study that involves at least two levels of z and that has p ossed with z. Suh an auxillay study may have fewe examinees and/o have othe limitations, but the study may still help in disentangling (p) and (pz), at least appoximately. Gao et al. (015) disuss the use of multiple G studies to appoximate eo vaiane and oeffiients unde omplex measuement onditions. A Single Pompt fo Diffeent Pompt Types Suppose an essay test onsists of two types of essay pompts (e.g., naative and pesuasive), and suppose that: 4 The vaiane omponent a (pz) eflets the peson-pompt inteation, whih is almost always quite lage. 5

10 Confounded Effets the two types of pompts (a) ae fixed in the sense that diffeent foms of the assessment would ontain the same two types; and the atual essay pompts (z) ae andom in the sense that diffeent foms of the assessment would ontain diffeent pompts. Stutually, we say that z is nested within a in the UAO). We denote this nesting as z : a. Suppose, as well, that all examinees o pesons (p) take the same pompts, whih ae soed by human ates () suh that: diffeent ates ae used fo eah pompt; and diffeent ates ae used ove foms. This means that ates ae andom, and they ae nested within pompts in the UAO. It follows that : z : a. Assuming all examinees o pesons (p) espond to all pompts, we denote the full design as p (: z :a). G theoy an handle the above design in a elatively staightfowad manne povided thee ae at least two onditions (levels o instanes) of eah of the faets, z, and a in a G Study. Confusion and omplexities aise, howeve, when thee is only a single ondition fo and/o z in the G Study. (We assume hee that both fixed onditions of a ae always pesent.) Suppose, fo example, that in the G study thee is only one pompt fo eah type of pompt. In suh a ase, we say that pompt z and type of pompt a ae ompletely onfounded, whih we denote (z, a). This means that piking a speifi level of z involves simultaneously piking a speifi level of a, and vie-vesa. The ompletely onfounded notation notation (z, a) needs to be distinguished fom: z :a, whih means z is nested within a; and z a (as in setion 1.1), whih means that eah level of z involves the same single, fixed level of a. Clealy, z a involves onfounding, but hee we geneally eseve the tem ompletely onfounded fo (z, a). We onside the hypothetial essay-test example outlined above fom seveal diffeent pespetives that foe us to distinguish aefully between diffeent univeses and designs, between whih faets ae fixed and whih ae andom, and whethe o not a D study sample size is one. We begin with a aeful onsideation of the vaiane omponents and elated statistis that we would like to know. Subsequently, we onside the esults that ae available to us in etain opeational testing iumstanes. The emaining subsetions (.1.6) in this setion ae somewhat ompliated. Fo some eades, skimming these setions may be suffiient. 6

11 Confounded Effets Table 1: Paameti Values fo Vaiane Components in the UAO based on the p (: z :a) Design; and fo D Study Vaiane Components, Eo Vaianes, and Coeffiients with A Fixed, Z Random, R Random, and n = 1 D Studies with A Fixed G Study Vaiane Components n n 1 1 A Random A Fixed n 1 (p) =.5 (p A) =.9 (p A) (a) =.03 (z :a) =.06 (z :a A) =.06 (Z :A A) (: z :a) =.0 (: z :a A) =.0 (R: Z :A A) (p a) =.08 (p z :a) =.10 (p z :a A) =.10 (p Z :A A) (p : z :a) =.1 (p : z :a A) =.1 (p R: Z :A A) (δ) (5) ( ) Ep < Note. a (a A) is not listed in the seond olumn beause it is a quadati fom. a (A A) disappeas in the last two olumns whih involve the aveage ove the two fixed levels of a; i.e., a (A A) = 0. a z z.1 UAO, G Study, and Pojeted D study Results The seond olumn of Table 1 povides hypothetial vaiane omponents fo the UAO. The esults in the seond olumn ae fo type-of-pompt being fixed, whih is denoted by appending A to the notation fo eah vaiane omponent. Often, in patie, the esults in the seond olumn ae obtained using the andom effets vaiane omponents in the fist olumn, whih assume that type of pompt is andom. Fo example, (pa) (p A) = (p) +, (9) whee vaiane omponents to the ight of the equal sign ae fo the andom model in the fist olumn. 5 Equation 9 is a speial ase of the geneal poedue disussed by Bennan (001, pp ) fo obtaining mixed-model vaiane omponents fom andom model vaiane omponents. Although the vaiane omponents in the seond olumn ae hypothetial, they ae not entiely uneasonable. Fo example, (z : a) =.06 is thee times 5 As noted ealie, the notation in this pape does not distinguish between paametes and estimates; doing so seems unneessay hee, and would likely add moe onfusion than laity fo most eades. 7

12 Confounded Effets lage than the vaiane omponent (: z : a) =.0, whih means that pompts ae onsideably moe vaiable than ates. This often happens fo testing pogams that have good ubis and well-tained ates. It is geneally muh easie to get ates to behave similaly than it is to get passages to be equally diffiult. The thid olumn povides the notational onventions fo denoting D study vaiane omponents and othe esults; uppease lettes denote mean soes. At the bottom of the olumn, (δ) is univese soe vaiane, (5) is elative eo vaiane, ( ) is absolute eo vaiane, Ep denotes a genealizability oeffiient, and < denotes a phi oeffiient. Fomulas fo obtaining these esults ae povided by Bennan (001). The fouth olumn povides pojeted D study esults fo the univese of genealization (UG) in whih A is fixed with a sample size of n a =, Z is andom with a sample size of n z = 1, and R is andom with a sample size of n =. It is impotant to undestand that these esults ae not based on a data olletion design in whih thee is only one pompt (n z = 1) fo eah of the two levels of the a faet. Rathe, the esults in the fouth olumn ae based on the UAO vaiane omponents in the seond olumn, whih we assume ae known o ould be estimated using a G study with at least two levels of all faets. If all the esults in olumns 1 o wee atually available, then onfounded effets would not be poblemati. Beause the esults in the fouth olumn ae not based on an atual D study with n z = 1, we efe to them above as pojeted D study esults. They ae the esults of inteest fo a D study with n z = 1. As disussed below and illustated in the next two subsetions, these esults annot be obtained using an opeational assessment with n z = 1. The statements in the above paagaph may appea ontaditoy o inonsistent. Afte all, how an pojeted D study esults be bette than esults fo an opeational assessment? The explanation fo this appaent ontadition ests on eognizing the following fats: 1. the pompt faet is intended to be andom, while the pompt-type faet is intended to be fixed;. when n z = 1, pompt and pompt type ae ompletely onfounded; 3. any analysis of an opeational assessment in whih () holds must teat the pompt and pompt-type faets as both fixed o both andom, whih ontadits (1); 4. the pojeted D study esults in the fouth olumn ae entiely onsistent with both (1) and (). If available data ontain esults fo only one pompt fo eah pompt type, then the effets fo the two faets annot be distinguished in any manipulation of the data. Still we must deide whethe to teat the onfounded effet as andom o fixed, and the hoie makes a diffeene that an be substantial. 8

13 Confounded Effets Table : Vaiane Components fo p ( :(z, a)) design, and D Study Results fo p (R:(Z, A)) Design with R Random and (Z, A) Random Vaiane Components fo Single Conditions of Faets D Studies fo Thee Random Faets n (z,a) n (p) =.5 (p) (z, a) =.09 ( :(z, a)) =.0 (Z, A) (R:(Z, A)) (p (z, a)) =.18 (p :(z, a)) =.1 (p (Z, A)) (p R :(Z, A)) (δ) (5) ( ) Ep < D Studies with (Z, A) Random Table povides esults fo the D study p (R :(Z, A)) design when R is andom and (Z, A) is teated as andom. The vaiane omponents in the fist olumn of Table an be obtained easily fom the vaiane omponents in the fist olumn of Table 1. Speifially, (p) =.5 is the same in both olumns, and (z, a) = (a) + (z :a) = =.09 ( :(z, a)) = (: z :a) =.0 (10) (p (z, a)) = (pa) + (p z : a) = =.18 (p :(z, a)) = (p : z : a) =.1 (11) The sum of the vaiane omponents in olumn 1 of Tables 1 and is the same, namely.66, but the seven vaiane omponents in Table 1 ollapse to five in Table. The sum is the same beause all faets ae andom in both tables. 6 Of ouse, if the available data fom an opeational administation of the assessment involve only one level of z, not all of the vaiane omponents in Table 1 would be known (o estimable). That does not mean, howeve, that these unknown vaiane omponents in Table 1 do not exist; they ae meely invisible to the investigato who analyzes data fom the opeational administation. Hee, we ae simply assuming that the vaiane omponents in Table 1 ae known, and unde this assumption we would expet to obtain the numeial esults in Table. 6 In Equation 10, a ( :(z, a)) = a (: z : a) beause with n z = 1 thee is no stutual (i.e., design) diffeene between (z, a) and z : a; a simila statement holds fo Equation 11. 9

14 Confounded Effets The thid olumn in Table povides D study esults fo n = and n (z,a) =, whee n (z,a) = means that we have two levels of the (pompt, pompt-type) onfounded effet. 7 Note, in patiula, that the esults in the thid olumn of Table ae fo (Z, A) andom. These esults an be ompaed to the values in the fouth olumn of Table 1 whee Z is andom and A is fixed. It is lea that univese soe vaiane is smalle in Table, eo vaianes ae lage in Table, and oeffiients ae substantially smalle in Table. All of these esults ae dietly attibutable to the fat that A is teated as andom in Table when it should be teated as fixed. Consequently, in Table, univese soe vaiane and oeffiients ae biased downwad, and eo vaianes ae biased upwad. Fo those who wish a moe detained explanation, the following disussion may help. Univese soe vaiane is biased downwad beause, fom Equation 9, (δ ) with (Z, A) andom is (pa) (p) = (p A) = (p A) (pa). That is, (pa) gets subtated fom (p A). In addition, (pa) gets added to (5) and ( ), whih auses these eo vaianes to get lage. ( ) also gets lage by the addition of (a)/..3 D Studies with (Z, A) Fixed Table 3 povides esults when R is andom and (Z, A) is teated as fixed. The vaiane omponents in the fist olumn of Table 3 an be obtained fom the vaiane omponents in the seond olumn of Table 1. Speifially, and it an be shown that Altenatively, ( :(z, a) (Z, A)) = (: z :a A) =.0, (1) (p :(z, a) (Z, A)) = (p : z :a A) =.1, (13) (p z :a A).10 (p (Z, A)) = (p A) + =.9 + =.34. (14) (pa) (p z :a) (p (Z, A)) = (p) + + = = The fouth olumn in Table is onsideed late in Setion.6. 10

15 Confounded Effets Table 3: Vaiane Components fo p ( :(z, a)) design, and D Study Results fo p (R:(Z, A)) Design with R Random and (Z, A) Fixed Vaiane Components fo Single Conditions of Faets D Studies fo One Random Faet n (z,a) n (p (Z, A)) =.34 (p (Z, A)) ( :(z, a) (Z, A)) =.0 (R:(Z, A) (Z, A)) (p :(z, a) (Z, A)) =.1 (p R :(Z, A) (Z, A)) (δ) (5) ( ) Ep < The last equation esults fom an appliation of a the geneal poedue disussed by Bennan (001, pp ) fo obtaining mixed-model vaiane omponents fom andom model vaiane omponents. 8 The thid olumn in Table 3 povides the D study esults fo n = and n (z,a) =, with (Z, A) fixed. These esults an be ompaed to the values in the fouth olumn of Table 1 with Z andom and A fixed. It is lea that univese soe vaiane is lage in Table 3, eo vaianes ae smalle in Table 3, and oeffiients ae substantially lage in Table 3. All of these esults ae dietly attibutable to the fat that Z is teated as fixed in Table 3 when it should be teated as andom. Consequently, in Table 3, univese soe vaiane and oeffiients ae biased upwad, and eo vaianes ae biased downwad. Fo those who wish a moe detained explanation, the following disussion may help. Univese soe vaiane fo (Z, A) fixed is lage than fo A fixed and Z andom. This ous beause, with (Z, A) fixed, as indiated in Equation 14, (p z : a A)/ gets added to (p A), whih is univese soe vaiane fo A fixed. Note also that (p z : a A)/ gets subtated fom (5) and ( ), 8 In Equation 1, a ( :(z, a) (Z, A)) = a (: z : a A) beause the vaiane omponent is fo ates, and it does not matte what faets o types of faets the ates ae nested within. A simila statement holds fo peson-ate ombinations in Equation

16 Confounded Effets whih auses these eo vaianes to get smalle. The eo vaiane ( ) also gets smalle by the subtation of (z :a A)/. Tables 1 3 povide esults fo n = 1 and n =, whih ae the most ommon sample sizes in most pefomane assessments. That is, typially thee is only one o two atings of the pefomane of any single peson on any single pompt..4 Results fo n = The bulleted esults in Setions. and.3 lealy indiate that the (Z, A) andom esults and the (Z, A) fixed esults baket the paamete-of-inteest values in the fouth olumn of Table 1. Speifially, n = (Z, A) Random A Fixed (Z, A) Fixed (δ).500 <.900 <.3400 (5).100 >.0800 >.0300 ( ).1700 >.1150 >.0350 Ep.6757 <.7838 <.9189 <.595 <.7160 <.9067 One impliation of the above esults is that, if the only available data ae fo a study in whih the pompt and pompt-type faets ae ompletely onfounded, then it is appopiate (and indeed desiable) to pefom analyses with (Z, A) andom and with (Z, A) fixed. 9 Then, an inteval fo eah paamete-of-inteest (i.e., a lowe and uppe bound) an be epoted..5 Results fo n = 1 The last olumns in Tables 1 3 povide the same types of esults disussed peviously, with the one diffeene being that n = 1. That is, these esults ae fo a measuement poedue in whih thee ae two fixed types of pompts, with a single andom pompt fo eah type, and with the esponses to eah pompt ated by a single andom ate. Again, we see that the paamete-of-inteest esults in Tables 1 ae baketed by the esults in Tables and 3. Speifially, n = 1 (Z, A) Random A Fixed (Z, A) Fixed (δ).500 <.900 <.3400 (5).1500 >.1100 >.0600 ( ).050 >.1500 >.0700 Ep.650 <.750 <.8500 <.5495 <.6591 < The expeted obseved soe vaiane, a (T ) + a (J), is.37 fo all thee analyses, beause expeted obseved soe vaiane is unaffeted by whih faets ae fixed and whih ae andom. 1

17 Confounded Effets As must be the ase, Ep, and < fo n = 1 ae smalle than fo n =, and eo vaianes ae lage. Othewise, howeve, the onlusions hee mio those in the pevious subsetion. Note that the widths of the intevals fo (δ), (5), and ( ) (.09,.09, and.135, espetively) ae the same fo n = 1 and n =. 10 This ous be ause, fo eah of these statistis, going fom (Z, A) andom to (Z, A) fixed leads to the addition o subtation of a onstant that is unelated to the sample size fo ates. Fo Ep and <, the widths diffe solely beause the eo vaianes z ae lage fo n = 1 than fo n =. When both n = 1 and n = 1, it is obvious that ates, pompts, and pompt types ae all onfounded, whih we an denote (, z, a), o (R, Z, A). Fo the example onsideed hee, this tiple onfounding implies that thee ae only two obsevations fo eah peson, sine n a = n a = and the othe two faets have a sample size of 1. Unde these iumstanes, it is not sensible to teat (, z, a) as fixed, beause then thee ae no andom faets, whih means that eo vaianes ae all 0 and oeffiients ae all 1. So, in this ase, the only analysis that an be done teats (, z, a) as andom, whih is disussed next..6 Coeffiient Alpha and (R, Z, A) If (, z, a) is viewed as andom, then the analysis will poeed as if thee is one andom faet that is ossed with pesons. This is the single-faet ossed design disussed extensively by Bennan (001, hap. ). The esulting genealizability oeffiient is idential to Coeffiient alpha. 11 Fo ou hypothetial example, the value of Coeffiient alpha that we would expet to obtain is the genealizability oeffiient in the last olumn of Table, namely,.650, whih is less than the paamete value of.750 in the last olumn of Table 1 with the pompt-type faet fixed. The fat that the Coeffiient alpha value of.650 is less that the paamete value of.750 has nothing to do with the mathematial poof in Lod and Novik (1968, pp ) that alpha is a lowe limit to eliability. In effet, thei poof is based on the assumption that Coeffiient alpha and the paamete involve the same soues of eo, all of whih ae onfounded in a single lump of andom eo. Fo the example onsideed hee, the value of alpha that would be expeted based on the (D study) data is a lowe limit beause this value (.650) impliitly involves thee andom faets, wheeas the paamete value (.750) involves only two andom faets; i.e., the paamete uses a moe estitive definition of eo than does alpha. Note that Ep =.6757 in the thid olumn of Table does not have a Coeffiient-alpha type of intepetation. In that olumn, Ep is based on fou obsevations pe examinee: two ates who evaluate one (pompt, pompt type) and a diffeent two ates who evaluate the seond (pompt, pompt type). That 10 The fat that the widths of the a (T ) and a (J) intevals ae the same (.09) is puely an atifat of these patiula, hypothetial data. 11 Bennan (001, p. 18) povides anothe pespetive on Coeffiient alpha and onfounding due to the fat that data fo omputing Coeffiient alpha usually ome fom a single oasion. 13

18 Confounded Effets is, the design in Table is p (R:(Z, A)); and with n =, the ate faet is not onfounded with (pompt, pompt type). It follows that the elative eo vaiane in Ep =.6757 is (p (z, a)) (p :(z, a)).18.1 (5) = + = + = = By ontast, if the data wee analyzed as if eah of the fou obsevations fo eah peson wee independent (the assumption fo Coeffiient alpha), then the eo vaiane would be (p (z, a)) + (p :(z, a)) = =.0750, 4 4 and Coeffiient alpha would be.5./( ) =.769. That is, in this ase, Coeffiient alpha would be an uppe limit to eliability. In shot, this setion illustates that whethe Coeffiient alpha is a lowe limit o an uppe limit to eliability (in the sense of a genealizability oeffiient) depends on whih faets ae fixed, whih faets ae andom, the D study design, and the D study sample sizes. Wheneve onditions of a andom and fixed faet ae onfounded in the data fo a measuement poedue, Coeffiient alpha is an inappopiate estimate of eliability. 3 A Single Rate (e.g., Automated Soing Engine) Thus fa in this pape, we have onsideed a UAO and designs in whih ates ae nested within anothe faet. Now let us onside a UAO in whih ates ae ossed with othe faets and with pesons. When the numbe of pesons is vey lage, it is highly unlikely that a G study design with human ates would faithfully mio this UAO, beause evey ate would have to evaluate evey esponse o pefomane fo evey examinee. Cuently, howeve, thee is onsideable inteest in the use of automated soing engines to ate examinee esponses. Sine any patiula automated soing engine employs a patiula algoithm, that algoithm funtions like a ate that is the same fo all examinees and othe faets. Of ouse, thee ae diffeent automated soing engines, eah of whih uses a diffeent algoithm. Unless othewise noted, in this setion we assume the UAO ontains a faet onsisting of numeous possible automated soing engines and, fo onsisteny with the pevious setions, we efe to this faet as the ate faet. 3.1 Example Fo the p (z :a) design, Table 4 extends the hypothetial example oiginally intodued in Table 1. With espet to olumn one in both tables, the eade an veify that (: z :a) = () + (a) + (z :a) = =.00, 14

19 Confounded Effets Table 4: Paameti Values fo Vaiane Components in the UAO fo the p (z : a) Design, and fo D Study Vaiane Components, Eo Vaianes, and Coeffiients with A Fixed, Z Random, R Random, and n = 1 D Studies with A Fixed Vaiane Components in UAO n n 1 1 A Random A Fixed n 1 (p) =.50 (p A) =.90 (p A) () =.010 ( A) =.013 (R A) (a) =.030 (z :a) =.060 (z :a A) =.060 (Z : A A) (p ) =.040 (p A) =.050 (p R A) (p a) =.080 (p z :a) =.100 (p z :a A) =.100 (p Z : A A) ( a) =.006 (z :a) =.004 (z :a A) =.004 (RZ : A A) (pa) =.00 (pz :a) =.060 (pz :a A) =.060 (prz : A A) (δ) (5) ( ) Ep < Note. a (a A) is not listed in the seond olumn beause it is a quadati fom. a (A A) disappeas in the last two olumns whih involve the aveage ove the two fixed levels of a; i.e., a (A A) = 0. a z z and (p: z :a) = (p) + (pa) + (pz :a) = =.10 whee vaiane omponents to the ight of the equal sign (fo the p (z : a) design) povide a deomposition of the vaiane omponents to the left of the equal sign (fo the p (: z : a) design). The numeial values fo the vaiane omponents in Table 4 ae hypothetial, but they bea some similaities with those in Table 1. The seond olumn in Table 4 povides vaiane omponents fo the UAO in whih A is fixed. Note that: (pa) (p A) = (p) +, (a) ( A) = () +, 15

20 Confounded Effets and (pa) (p A) = (p) +. The last two olumns povide D study esults fo the sample sizes onsideed thoughout this pape. Note that the next-to-the-last olumn povides esults fo n = fo onsisteny with Table 1 (and othe tables in Setion 1), but ou pinipal fous hee is esults fo n = 1 in the last olumn. Of patiula impotane is the fat that the eo vaianes ae lage and the oeffiients ae smalle in Table 4, whee ates ae ossed with (z, a), than in Table 1, whee ates ae nested within (z, a). A somewhat heuisti explanation fo this is that in Table 4 thee ae only n ates involved in the design, wheeas in Table 1 thee ae n ates involved (e.g., when n = 1 thee is one ate fo (z 1, a 1 ) and a diffeent ate fo (z, a )). In geneal, fo a given value of n, (5) in Table 4 is lage than (5) in Table 1 by (p)/(n ), and ( ) in Table 4 is lage that ( ) in Table 1 by ( () + (p))/(n ). (See the Appendix fo futhe, moe mathematial explanations.) So, all othe things being equal, the ossed design has lage eo vaianes (and smalle oeffiients) than the nested design R Random and (Z, A) Teated as Random Suppose the only available data ae fom the opeational administation of a pefomane assessment that uses the p (z :a) design with the pompt and pompt-type faets being ompletely onfounded. Table 5 povides esults fo the D study p R (Z :A) design when R is andom and (Z, A) is teated as andom. The vaiane omponents in the fist olumn of Table 5 an be obtained fom the vaiane omponents in the fist olumn of Table 4. Speifially, (p), (), and (p) ae unhanged, and (z, a) = (a) + (z :a) = =.090 (p (z, a)) = (pa) + (p z :a) = =.180 ((z, a)) = (a) + (z :a) = =.010 (15) (p(z, a)) = (pa) + (pz :a) = =.080, (16) whee vaiane omponents to the left of the equal sign ae fo Table 5. The sum of the vaiane omponents in olumn 1 of Tables 4 and 5 is the same, namely.66, but the 11 vaiane omponents in Table 4 ollapse to seven in Table 5. The sum is the same beause all faets ae andom in both tables. 1 The vaiane omponents in the fist olumn of Table 5 ae elated to the vaiane omponents in the fist olumn of Table. Speifially, (p), (z :a), 1 In Equation 15, a (z : a) = a ((z, a)) beause: (a) a (z : a) is an abbeviated notation fo a ((z : a)), and (b) with n z = 1 thee is no stutual (i.e., design) diffeene between (z : a) and (z, a); similaly, in Equation 16 a (pz :a) = a (p(z, a)). 16

21 Confounded Effets Table 5: Vaiane Components fo p (z : a) design, and D Study Results fo the p R (Z : A) Design with R Random and (Z, A) Random Vaiane Components fo Single Conditions of Faets (p) () (z, a) (p) (p (z, a)) ((z, a)) (p(z, a)) = = = = = = = D Studies fo Thee Random Faets n (z,a) n (p) (R) (Z, A) (pr) (p (Z, A)) (R(Z, A)) (pr(z, A)) (δ) (5) ( ) Ep <.500, Table 6: Vaiane Components fo p (z, a) design, and D Study Results fo p R (Z, A)) Design with R Random and (Z, A) Fixed Vaiane Components fo Single Conditions of Faets D Studies fo One Random Faet n (z,a) n (p (Z, A)) =.340 (p (Z, A)) ( (Z, A)) =.015 (R:(Z, A) (Z, A)) (p (Z, A)) =.080 (p R :(Z, A) (Z, A)) (δ) (5) ( ) Ep <

22 Confounded Effets and (p (z, a)) ae unhanged. Also, ( :(z, a)) = () + ((z, a)) = =.00 and (p :(z, a))) = (p) + (p(z, a)) = =.10, whee vaiane omponents to the left of the equal signs ae fo Table. These equations ould be used to obtain esults fo the nested p (: z :a) design fom the ossed p (z :a) design R Random and (Z, A) Teated as Fixed Table 6 povides esults fo the D study p R (Z :A) design when R is andom and (Z, A) is teated as fixed. The vaiane omponents in the fist olumn of Table 6 an be obtained fom the vaiane omponents in the fist olumn of Table 5. Speifially, (p (z, a)) (p (Z, A)) = (p) +, ((z, a)) ( (Z, A)) = () +, and (p(z, a)) (p (Z, A)) = (p) +, whee vaiane omponents to the ight of the equal sign ae in Table Results fo n = 1 The last olumns of Tables 4, 5, and 6 povide esults fo the ase of n = 1, whih means that the single ate (automated soing engine) faet is onfounded with pesons and the othe faets. These esults ae summaized below: n = 1 (Z, A) Random A Fixed (Z, A) Fixed (δ).500 <.900 <.3400 (5).1700 >.1300 >.0800 ( ).300 >.1750 >.0950 Ep.595 <.6905 <.809 <.508 <.637 <.7816 Compaing these esults with those in Setion.5, it is evident that fo a single ate nested within faets, oeffiients ae lage and eo vaianes ae smalle, elative to what they ae fo a single ate (automated soing engine) ossed with all pesons and othe faets. This is tue whethe o not the pompt faet (z) and the pompt-type faet (a) ae onfounded. The numeial esults disussed above ae fo hypothetial data, but the onlusions hold in geneal povided the UAO s ontain the same faets and the population is the same. 18

23 Confounded Effets 3. Othe Issues The pevious setion by no means exhausts the oneptual, design, o omputational omplexities involved in estimating eo vaianes and eliability-like oeffiients when an ASE is used. A few additional omplexities ae biefly disussed next Single Random vs. Fixed ASE The disussion in Setion 3, as it elates to n = 1, is elevant fo a single andom ASE, but estimating esults pesumes that a G study is onduted that employs at least two ASEs. If a G study inludes only one ASE, then the ate (ASE) faet will be onfounded with all othe faets as well as the peson faet. This means that any D study esults will teat the ASE as a single level of a fixed faet, whethe o not that is the intent of the investigato. Fo the owne of an ASE whose only onen is that patiula ASE, the above paagaph may be intepeted as meaning that Setion 3 is ielevant sine it elates to a single andom ASE. An ASE use, howeve, may have a legitimate onen about how any single ASE might pefom. 3.. Taining ASEs When an ASE is used, it must be tained using a set (s) of papes that ae epesentative of the esponses by the examinees. In the absene of evidene to the ontay, thee is no guaantee that the ASE will funtion the same way if a diffeent epesentative set of papes wee used and/o the taining papes wee fo diffeent subpopulations of examinees (e.g., non-minoities and minoities). So, esults fo the patiula ASE ae onfounded with the set of papes hosen fo taining. Consequently, it seems lea that esults fo any speifi ASE should be examined based on: 1. diffeent sets of papes sampled fom the the same population of examinees, and. diffeent sets of papes sampled fom diffeent subpopulations of examinees. Diffeenes in esults fo #1 eflet andom eo attibutable to sampling of papes. Diffeenes in esults fo # eflet one type of diffeential ASE funtioning elevant to subpopulations Compaing Multiple ASEs Thee ae at least two geneal appoahes that might be taken to ompaing esults fo k fixed ASEs. Fist, ondut k sepaate univaiate G theoy (UGT) analyses. Seond, ondut a single multivaiage G theoy (MGT) analysis 19

24 Confounded Effets A simple example of the UGT appoah is to apply the basi methodology in Setion 1.1 to eah of the k ASEs by intehanging the oles of ates and pompts. Reall that Setion 1.1 onsides a p design with a single pompt. Hee, howeve, we want an analysis fo a p z design with a single ASE (whih plays the ole of a ate). Equations fo doing so ae obtained by intehanging z and a eveywhee in Setion 1.1. Obviously, it must be tue that n z?, and the ompaisons ae sensible only if the pesons and pompts ae the same fo all k ASEs. A moe geneal (and infomative) MGT appoah would use the p z design, as disussed in Bennan (001, hap. 9). An analysis using this design yields thee k k symmeti vaiane-ovaiane maties (Σ p, Σ z, and Σ pz ). The diagonal elements ae the vaiane omponents fo the k univaiate analyses in the UGT appoah. The off-diagonal elements ae ovaiane omponents fo the k(k 1)/ ompaisons of pais of ASEs Compaisons Involving Human Rates and an ASE A vey ommon appoah to justifying the use of a patiula ASE is to ompae the ASE atings fo a single pompt to atings fo human ates, and delae that the ASE is woking well if the ASE atings ae simila to those fo human ates, in some sense. Pesumably, any epliated analysis would use the same ASE but a diffeent set of human ates. (Atually, in most ases, the ASE/ate ompaisons ae even moe ompliated beause diffeent human ates ae often used with diffeent sets of examinees, but we ovelook this omplexity hee.) A substantial poblem with suh ASE/ate ompaisons is that the ASE is almost always viewed as fixed but the human ate faet is viewed as andom. Unde these iumstanes, any ASE/ate ompaisons ae at best ambiguous. Stated diffeently, in a sense, ASE/ate ompaisons give the ASE atings a pivileged status in that ASE atings ae not subjet to any a pioi expliitly epesented soues of eo. One way to inopoate potential eo in ASE atings is to ondut a study that inludes a faet fo diffeent sets of taining papes. Then, even fo a patiula ASE, thee will be potentially diffeent atings fo eah examinee s esponse to a pompt. Unde these iumstanes, thee ae many speifi designs and analyses that might be onduted. (Speifis ae outside the intended sope of this pape.) The basi piniple, howeve is lea, namely that a fai ompaison of human atings and ASE atings equies that ASE atings inopoate the existene of potential eo in suh atings. 4 Conluding Comments The pimay fous of this pape is on onfounded effets that aise when, fo one o moe faets, thee ae many possible onditions in a UAO, but a measuement poedue inludes only a single ondition fo one o moe of these faets. 0

25 Confounded Effets We all this the poblem of one. Among the impotant take-home messages fom this pape ae the following. 1. When thee ae many onditions fo a single faet in a UAO, but only a single ondition in the opeational measuement poedue, the faet is hidden in the measuement poedue, whih leads to undestating eo vaiane and ovestating oeffiients;. When a onfounded effet in an opeational measuement poedue involves a mixtue of andom and fixed effets, any analysis using the opeational data will esult in ambiguous esults fo eo vaianes and oeffiients. 3. When (1 o ) ous, onduting a G study with at least two onditions fo all faets is almost always neessay to esolve onfounded-effets ambiguities. 4. When () ous but no suh G study is onduted, epoting the end points of a ange of possible values fo eo vaianes and oeffiients seems sensible. The is onsistent with the geneal position that a fuzzy answe to a meaningful, foused question is bette than a peise answe to an unimpotant o fuzzy question. Impliit in these take-home messages is the impotane of distinguishing between whih faets ae fixed and whih ae andom in a UAO, whih is a hallmak of G theoy, but a negleted topi in most othe measuement theoies. These messages also emphasize that single onditions of faets in a measuement poedue lead to ambiguous esults that annot be esolved by any manipulation of the data. Disentangling these ambiguous esults equies additional studies. This pape does not ove all issues elated to onfounding. Indeed, as Conbah et al. (197) noted deades ago, G theoy has a potean quality to it. Many, if not most, in-depth genealizability analyses involve diffeent mixes of omplex issues, inluding onfounding. Unfotunately, suh issues ae fa too often negleted. In patiula, onfounded effets almost always lead to epoted statistis (e.g, eliabilities and eo vaianes) that ae biased, and failue to eognize this fat an easily lead to flawed soe intepetations. In a sense, this pape hallenges etain aspets of the onventional wisdom in psyhometis. 5 Refeenes Bennan, R. L. (001). Genealizability theoy. New Yok: Spinge-Velag. Conbah, L. J., Glese, G. C., Nanda, H., & Rajaatnam, N. (197). The dependability of behavioal measuements: Theoy of genealizability fo soes and pofiles. New Yok: Wiley. 1

26 Confounded Effets Gao, X., Bennan, R. L., & Guo, F. (015). Modeling Measuement Faets and Assessing Genealizability in a Lage-Sale Witing Assessment. GMAC Reseah Repot RR Gaduate Management Admission Counil, P. O. Box 969, Restin, Viginia Kane, M. T. (198). A sampling model fo validity. Applied Psyhologial Measuement, 6, Appendix: Eo Vaiane Relationships in Tables 1 and 4 Reall that Table 1 is fo the ase when ates ae nested within (z, a), wheeas Table 4 is fo the ase in whih ates ae ossed with (z, a). Fo Table 4, (p A) (p z :a A) (p z :a A) (5) = + + n n n n n n = a z a z (p) + (pa)/n a (p z : a) (p z :a) + + n n n n n n a z a z (p) (pa) (p z :a) (p z :a) = + + +, (17) n n n n n n n n a a z a z whee the last two equalities ae expessed in tems of the UAO andom effet vaiane omponents in Table 4. Fo Table 1, (p z : a A) (p : z :a A) (5) = + n n n n n a z a z (p z : a) (p) + (pa) + (p z :a) = + n n n n n a z a z (p z : a) (p) (pa) (p z :a) = + + +, (18) n n n n n n n n n n n a z a z a z a z whee the last two equalities ae expessed in tems of the UAO andom effets vaiane omponents in Table 4. Subtating Equation 18 fom Equation 17 gives [ ] [ ] (p) (pa) (p) (pa) + + = n n n n n n n n n a a z a z ( ) ( ) (p) 1 (pa) , n n n n n z a a a z whih is (p)/(n ) when n = 1 and n =, as is the ase in the body of this pape. A simila appoah shows that ( ) in Table 4 is lage that ( ) in Table 1 by ( () + (p))/(n ).