Psychometric Methods: Theory into Practice Larry R. Price

Transcription

1 ERRATA Psychometic Methods: Theoy into Pactice Lay R. Pice Eos wee made in Equations 3.5a and 3.5b, Figue 3., equations and text on pages 76 80, and Table 9.1. Vesions of the elevant pages that include the coected mateial follow. Reades ae encouaged to the publishe at to epot any futhe eata.

2 74 PSYCHOMETRIC METHODS Equation 3.5a. Semipatial coelation coefficient YX 1 X = 1 YX YX1 X1X X1X YX 1 X = semipatial coelation coeffecient. YX = coelation between citeion Y and pedicto X. 1 = coelation between pedicto X 1 and pedicto X. X X YX X1X = squae of the coelation between citeion Y and pedicto X. = squae of the coelation between X 1 and pedicto X. XY = - ance accounted fo in Y by X. YX 1 X = - ance accounted fo in Y by X 1 afte contolling fo X. Note. The vaiable following the multiplication dot ( ) is the vaiable being patialed. (Y) accounted fo by language development (X 1 ) afte the effect of gaphic identification (X ) is patialed o contolled. Applying the coelation coefficients fom ou example data, we have the esult in Equation 3.5b. Note that the esult below agees with Equation 3.4b. Theefoe, we have illustated a second way to aive at the same conclusion but the semipatial coelation povides a slightly diffeent way to isolate o undestand the unique and nonunique elationships among the pedicto vaiables in elation to the citeion. Figue 3. povides a Venn diagam depicting the esults of ou analysis in Equation 3.5b. Equation 3.5b. Semipatial coelation coefficient with example data (.39).1148 = = = =.15 =.0156 YX YX1 X1X YX 1 X YX 1 X 1.90 X1X and (.0156)(100) = 1.56%

3 Y-HVSCI Pat of vaiance accounted fo by language development afte the effect of gaphic identification is patialled out o emoved fom language development only.15 and.15² 100 = 1.56% X Gaphic identification Common vaiance in Y accounted fo by gaphic identification and language development X1 Language development FIGURE 3.. Venn diagam illustating the semipatial coelation. The cicles epesent pecentages (e.g., each cicle epesents 100% of each vaiable). This allows fo convesion of coelation coeffecients into the popotion of vaiance metic,. The metic can then be conveted to pecentages to aid intepetation. 75

4 76 PSYCHOMETRIC METHODS TABLE 8.8a. Repeated Measues ANOVA Output fo the Peson Rate Design Measue: MEASURE_1 Souce ates ates * pesons (Residual) Tests of Within-Subjects Effects Type III Sum of Squaes df Mean Squae F Sig. Spheicity Assumed Spheicity Assumed Note. Pats of the output have been omitted fo ease of intepetation. TABLE 8.8b. Repeated Measues ANOVA Output fo the Peson Rate Design Tests of Between-Subjects Effects Measue: MEASURE_1 Tansfomed Vaiable: Aveage Souce Type III Sum of Squaes df Mean Squae F Sig. Intecept pesons Eo Next, the vaiance components ae calculated using mean squaes fom the ANOVA esults. The vaiance component estimate fo pesons is povided in Equation 8.9, and the estimate fo ates is povided in Equation The vaiance component estimate fo eo o the esidual is povided in Equation To illustate how the genealizability coefficient obtained in ou G-study can be used within a D-study, let s assume that the ates used in ou G-study ae epesentative of the ates in the univese of genealization. Unde this assumption, ou best estimate is the Equation 8.9. Vaiance component fo pesons MS MS s = = = = Equation Vaiance component fo ates ates esidual MS MS ates = = = = n p

5 Genealizability Theoy 77 Equation Popotion of vaiance fo esidual E ŝ = = RESIDUAL aveage obseved scoe vaiance fo all the ates in the univese. The aveage scoe vaiance is captued in the sum of s P +s E. Because we ae willing to assume that ou ates ae epesentative of the univese of ates we can estimate the coefficient of genealizability in Equation 8.1 fom ou sample data. An impotant point hee is that ates ae not usually andomly sampled fom all possible ates in the univese of genealization, leading to one difficulty with this design. The value of.89 indicates that the ates ae highly eliable in thei atings. Using this infomation, we can plan a D-study in a way that ensues that ate eliability will be adequate by changing the numbe of ates. Fo example, if the numbe of ates is educed to two in the D-study, the vaiance component fo the esidual changes to.7. Using the new vaiance component fo the esidual in Equation 8.13 yields a genealizability coefficient of.85 (which is still acceptably high). Next, we tun to the popotion of vaiance as illustated in Equation 8.14 as a way to undestand the magnitude of the effects. In G theoy studies, the popotion of vaiance povides a measue of effect size that is compaable acoss studies. The popotion of vaiance is epoted fo each facet in a study. Fo example, the popotion of vaiance fo pesons is povided in Equation Equation 8.14 shows that the peson effect accounts fo appoximately 3% of the vaiability in ating scoes among pesons. Next, in Equation 8.15 we calculate the popotion of vaiance fo the ate effect. We see fom Equation 8.15 that the ate effect accounts fo appoximately 56% of the vaiability in memoy scoe pefomance atings. Fom this infomation we conclude Equation 8.1. Genealizability coefficient fo ating data with esidual aveaged ove ates σˆ p ates* = = = = = =.89 σˆ e p ' 3 n i Note. The asteisk (*) signifies that the G coefficient can be used fo a D-study with pesons cossed with ates (i.e., the measuement conditions). Notation is fom Cocke and Algina (1986, p. 167).

6 78 PSYCHOMETRIC METHODS Equation Revised genealizability coefficient fo ating data σˆ p ates* = = = = = =.85 σˆ e p ' n i Note. The asteisk (*) signifies that the G coefficient can be used fo a D-study with pesons cossed with the aveage numbe of ates (i.e., the measuement conditions). Equation Popotion of vaiance fo pesons σˆ p = = = σˆ p esidual Equation Popotion of vaiance fo ates σˆ.6.6 = = = σˆ p esidual that the ate effect is modeate (i.e., ates account fo o captue a medium amount of vaiability among the ates). Anothe way of intepeting this finding is that the ates ae modeately simila o consistent in thei atings. 8.1 DESIGN 3: TWO-FACET DESIGN WITH THE SAME RATERS ON MULTIPLE OCCASIONS In Design 3, we cove a G-study whee the atings ae aveaged, a stategy used to educe the eo vaiance in the measuement condition. We can aveage ove ates because the same obseves ae conducting the atings on each occasion fo pesons (i.e., ates ae not diffeent fo pesons). Aveaging ove ates involves dividing the appopiate eo component by the numbe of ates and occasions. Fo example, in Equation 8.16 the eo

7 Genealizability Theoy 79 Equation Genealizability coefficient fo two-facet design same 3 ates and occasions ˆ = = = = =.88 ˆ ˆ σ p RATERS σates+ σeo p + ' nates Note. The asteisk (*) signifies that the G coefficient can be used fo a D-study with pesons cossed with the aveage numbe of ates (i.e., the measuement conditions). Capital notation fo RATERS signifies that the eo vaiance is divided by 3, the numbe of ates in a D-study. The symbol N RATERS signifies the numbe of atings to fom the aveage. Notation is fom Cocke and Algina (1986, p. 167). vaiance component is divided by 3 ([ ]/3). In ou example data, the change ealized in the G coefficient by aveaging ove ates is fom.89 to.88 (Equation 8.16). Thee is little decease in the G coefficient (i.e., fom.89 in Design to.88 in Design 3), telling us that when it is easonable to do so, aveaging ove ates is an acceptable stategy DESIGN 4: TWO-FACET NESTED DESIGN WITH MULTIPLE RATERS In Design 3, we illustated the situation in which each peson is ated by the same ates on multiple occasions. In Design 4, each peson has thee atings (on thee occasions), but each peson is ated by a diffeent ate. Fo example, this may occu in the event that a lage pool of ates is available fo use in a G-study. In this scenaio, ates ae nested within pesons. Symbolically, this nesting effect is expessed as : p o (p). In this design, diffeences among pesons ae influenced by (1) ate diffeences plus () univese scoe diffeences fo pesons and (3) eo vaiance. To captue this vaiance, the obseved scoe vaiance fo this design is σ P +σ RATERS +σ E, whee the vaiance component symbols ae the same as in Design. Using the same mean squae infomation in Equations 8.9, 8.10, and 8.11, we find that the G coefficient fo Design 4 is povided in Equation We see that thee is substantial eduction in the G coefficient fom.89 (Design ) o.88 (Design 3) to.70 (Design 4). Knowing this infomation about the eduction of the G coefficient to an unacceptable level, we can plan accodingly by using Design o 3 athe than Design 4.

8 80 PSYCHOMETRIC METHODS Equation Genealizability coefficient fo Design 4 σ p RATERS = = = = = σˆ p ates esidual Note. No asteisk (*) is included in the equation afte ates, signifying that this is a D-study and the measuement condition of atings is nested within pesons DESIGN 5: TWO-FACET DESIGN WITH MULTIPLE RATERS RATING ON TWO OCCASIONS In Design 4, the scenaio was illustated whee diffeent ates ate each peson and each peson is ated on thee occasions. Ou stategy in Design 5 with multiple ates and occasions of measuement is to aveage ove atings. The G coefficient fo Design 5 is povided in Equation Table 8.9 summaizes the fomulas fo the fou G coefficients based on the designs coveed to this point (excluding Design 5, which is a modification of Design 4). Equation Genealizability coefficient fo Design 5 σˆ p 1.5 RATERS = = ates + σˆ esidual p ' n 3 ates = = = Note. The wod RATERS in capital lettes signifies that the measuement condition, atings, ae aveaged ove ates. The symbol N RATERS signifies the numbe of atings to fom the aveage. Notation is fom Cocke and Algina (1986, p. 167).

9 94 PSYCHOMETRIC METHODS table 9.1. subtest Vaiables in the GfGc dataset Name of subtest Numbe of items Scoing Fluid intelligence (Gf) Quantitative easoning sequential Fluid intelligence test /1/ Quantitative easoning abstact Fluid intelligence test 0 0/1 Quantitative easoning induction and deduction Fluid intelligence test 3 0 0/1 Cystallized intelligence (Gc) Language development Cystallized intelligence test 1 5 0/1/ Lexical knowledge Cystallized intelligence test 5 0/1 Listening ability Cystallized intelligence test /1/ Communication ability Cystallized intelligence test /1/ Shot-tem memoy (Gsm) Recall memoy Shot-tem memoy test 1 0 0/1/ Auditoy leaning Shot-tem memoy test 10 0/1//3 Aithmetic Shot-tem memoy test /1/ deductive easoning) does not coelate at even a modeate level with gaphic oientation and gaphic identification. Additionally, inspection of the unshaded cells in Table 9. eveals that the subtests in the theoetical clustes also coelate modeately (with the exception of subtest 10 on inductive and deductive easoning) with subtests that ae not pat of thei theoetical cluste. 9.4 estimating factos And facto loadings At the heat of FA is the elationship between a coelation matix and a set of facto loadings. The intecoelations among the vaiables and the factos shae an intimate elationship. Although facto(s) ae unobsevable vaiables, it is possible to calculate the coelation between factos and vaiables (e.g., subtests in ou GfGc example). The coelation between factos and the GfGc subtests ae called facto loadings. Fo example, conside questions 1 4 oiginally given in Section What ole does the patten of intecoelations among the vaiables o subtests play in identifying the numbe of factos?. What ae the geneal steps in conducting a facto-analytic study? 3. How ae factos estimated? 4. How ae facto loadings intepeted? Though these questions, we seek to know (1) how the patten of coelations among the vaiables infom what the facto loadings ae, () how the loadings ae estimated; and