Applications of latent trait theory to the development of norm-referenced tests.

Transcription

1 University f Massachusetts Amherst SchlarWrks@UMass Amherst Dctral Dissertatins February Applicatins f latent trait thery t the develpment f nrm-referenced tests. Linda L. Ck University f Massachusetts Amherst Fllw this and additinal wrks at: Recmmended Citatin Ck, Linda L., "Applicatins f latent trait thery t the develpment f nrm-referenced tests." (1979). Dctral Dissertatins February This Open Access Dissertatin is brught t yu fr free and pen access by SchlarWrks@UMass Amherst. It has been accepted fr inclusin in Dctral Dissertatins February 2014 by an authrized administratr f SchlarWrks@UMass Amherst. Fr mre infrmatin, please cntact schlarwrks@library.umass.edu.

2

3 APPLICATIONS OF LATENT TRAIT THEORY TO THE DEVELOPMENT OF NORM-REFERENCED TESTS A Dissertatin Presented By LINDA LEE OK Submitted t the Graduate Schl f the University f Massachusetts in partial fulfillment f the requirements fr the degree f DOCTOR OF EDUCATION September 1979 EDUCATION

4 LINDA LEE OK 1979 All Rights Reserved

5 applicatins f latent trait thery t the DEVELOPMENT OF NORM-REFERENCED TESTS A Dissertatin Presented By LINDA LEE OK Apprved as t style and cntent by: Dr. Rnald K. Hambletn, Chairpersn Dr. Hariharan Swaminathan, Member A iii

6 TO: Peter and Perry

7 ACKNOWLEDGEMENTS Withut the assistance, encuragement, and supprt f many individuals this dissertatin culd never have been cmpleted. I wish t cnvey my deep appreciatin t my three cmmittee members. Dr. Rnald Hambletn, Dr. Hariharan Swaminathan, and Dr. Harry Schumer. Special recgnitin must be extended t my chairman. Dr. Hambletn, fr his intellectual guidance and emtinal supprt; fr the true friendship he s willingly extended; fr his unselfish gift f his persnal time; and mst f all fr his patience. Dr. Swaminathan deserves special thanks fr his guidance, friendship, and encuragement during my early years as a graduate student. Dr. Schumer was the prfessr f the first subject I studied at the University f Massachusetts many years ag as an undergraduate and fr this and ther reasns assumes particular imprtance t me as a member f my dissertatin cmmittee. I am indebted t my graduate clleagues in the Labratry f Psychmetric and Evaluative Research wh have been an unflagging surce f encuragement and friendship, particularly t Ms. Janice Giffrd wh has been a valuable friend and cunselr but als assisted greatly in the many cmputer simulatins that were required by this study. I als wish t express my gratitude t Ms. Bernadette McDnald wh, thrugh her skillful typing and editing added immeasurably t the quality f this dissertatin. I am deeply indebted t my parents. V

8 Mrs. Jessie Dean and Mr. Hward Dean fr their sustained supprt and caring. Finally, I wuld like t acknwledge the encuragement and understanding I have received frm my gd friend and fellw graduate student, Daniel Eignr wh with a delicate cmbinatin f humr and criticism, always managed t help me restre perspective at times when I felt truly defeated. vi

9 ABSTRACT Applicatins f Latent Trait Thery t the Develpment f Nrm-Referenced Tests (September 1979) Linda Lee Ck, B.S., Ursinus Cllege M.E.D., Smith Cllege Ed.D., University f Massachusets, Amherst Directed by: Rnald K. Hambletn Latent trait thery ffers several advantages t the psychmetrician interested in develping tests: (1) invariant item parameters that facilitate the test develpment prcess as well as make pssible the develpment f tests fr a variety f applicatins, (2) a mathematical functin that can be manipulated t prvide valuable insights int hw examinees perfrm n specific test items, and (3) added infrmatin abut examinee ability derived frm new test scring methds. Because f these and ther prperties f latent trait mdels, item selectin and item analysis prcesses differ substantially frm thse emplyed when using standard testing tehcnlgy. Althugh the abve mentined advantages have been dcumented in the literature, t date, n specific methdlgy has been set frth fr the develpment f nrm-referenced tests utilizing latent trait thery. In part this is because there still exist a number f significant prblem areas requiring reslutin. Fr example, befre vii

10 latent trait thery can be used successfully in test develpment wrk, mre needs t be knwn abut: (1) the rbustness f the mdels, (2) the prperties f infrmatin functins, and (3) hw best t use these functins in the test develpment prcess. This study had three purpses. The first was t study, systematically, the "gdness f fit" f the ne-, tw-, and threeparameter lgistic mdels emplying a practical criterin fr assessment. Using cmputer-simulated test data, the effects f fur variables were studied: (1) variatin in item discriminatin parameters, (2) the average value f the psued-chance level parameters, (3) test length, and (4) the shape f the ability distributin. The secnd purpse f the study was t address tw practical questins which are f imprtance and interest t test develpers: 1. What are the effects f examinee sample size and test length n the precisin f the standard errr f ability estimatin curves? 2. What effects d the statistical characteristics f an item pl have n the precisin f standard errr f ability estimatin curves? As in the previus study, cmputer-simulated test data was used t study the prblem. The third purpse f the study was t investigate the fllwing questins related t the develpment f item selectin methdlgies: 1. Using a typical item pl (where items are described by parameters in the three-parameter lgistic test mdel) hw des ne develp alternate item selectin methdlgies and hw d the scre infrmatin curves that result frm these methdlgies cmpare? 2. Given a specific testing purpse such as prducing a schlarship exam r a test t ptimally separate examinees int three ability categries, hw des ne develp alternate item selectin methdlgies and hw d the scre infrmatin curves resulting frm these methdlgies cmpare? viii

11 The results f the rbustness studies revealed that there are sme sizeable gains t be expected with mdest length tests (n=20) in the crrect rdering f examinees at the lwer end f the ability cntinuum when three-parameter mdel estimates are used (as ppsed t number right scre). The gains were cut rughly in half when the tests were dubled (n=40) in length. It was als nted that item discriminatin parameters as weights had little effect n the results. Results frm the secnd part f the study indicated that: (1) bth test length and sample size are extremely imprtant factrs in the precisin f SEE curves; (2) the precisin f SEE curves at the extremes f an ability cntinuum is very pr, even with large examinee sample sizes, hwever, the results are substantially better when tests are lengthened, even if sample size is small; (3) the precisin f SEE curves wuld be acceptable in mst instances if the curves are based n 200 r mre examinees with tests with at least 20 items, and; (4) the mst sizeable imprvements in the precisin f SEE curves ccur when examinee sample size is increased frm 50 t 200 and when test length is increased frm 10 t 20 items. The third part f the study revealed that in all cases, item selectin methds based n either randm selectin f items r the use f classical item statistics prduced results inferir t thse prduced by methds utilizing latent trait mdel item parameters. The study als indicated that methds must be develped with a specific testing purpse in mind. If maximum infrmatin is required at nly ne pint n an ability cntinuum, it is clear that ix

12 a methd that chses items that maximize infrmatin at this particular pint will be the best. If infrmatin is required ver a wider range f abilities, methds invlving averaging the infrmatin values acrss ability levels f interest r chsing items in sme systematic way that cnsiders each pint f interest n the ability cntinuum appear t be quite prmising. X

13 TABLE OF NTENTS Page DEDICATION ACKNOWLEDGEMENTS ABSTRACT LIST OF TABLES LIST OF FIGURES v vii xiii xv CHAPTER I INTRODUCTION AND STATEMENT OF THE PROBLEM Backgrund and Review f the Literature Statement f the Prblems Purpses Organizatin f the Study 9 II LATENT TRAIT MODELS AND RELATED NCEPTS Intrductin Features f Latent Trait Mdels Cmmn Frms f Item Characteristic Curves 2.3 The Ability Scale and Its Meaning Test Infrmatin and Efficiency The Classical Test Mdel Versus Latent Trait Mdels 29 III ROBUSTNESS OF LATENT TRAIT MODELS Intrductin Methd f Investigatin Simulating the Test Data Gdness-f-Fit 3.3 Results 3.4 Cnclusins xi

14 CHAPTER Page IV EFFECTS OF TEST LENGTH AND SAMPLE SIZE ON THE ESTIMATES OF PRECISION OF LATENT ABILITY SRES Intrductin Methd f Investigatin Descriptin f the Variables Simulatin f Data 4.3 Results and Discussin Effects f Sample Size and Test Length f the Precisin f Standard Errr f Ability Estimatin Curves Effects f Statistical Characteristics f an Item Pl n Precisin f SEE Curves Relatinships Between Test Length and SEE Curves in Tw Typical Item Pls 4.4 Cnclusins 75 V A MPARATIVE STUDY OF ITEM SELECTION METHODS UTILIZING LATENT TRAIT THEORETIC MODELS AND NCEPTS 77 PART A Cmparisn f Five Item Selectin Methds 5.2 Purpse 5.3 Methd f Investigatin Generatin f the Item Pl Item Selectin Methd 5.4 Results PART B Selecting Test Items t "Fit Target Curves 5.5 Purpse 5.6 Methd f Investigatin Case I Case II 5.7 Results Case I Case II 5.8 Cnclusins VI SUMMARY, NCLUSIONS, AND IMPLICATIONS FOR FURTHER RESEARCH REFERENCES xii

15 LIST OF TABLES Table Page Summary f the Gdness-f-Fit Results (Unifrm Ability Distributin, 6 = -2.5 t 0.0) Summary f the Gdness-f-Fit Results (Unifrm Ability Distributin, 6 = -0.0 t +2.5).. 41 Summary f the Gdness-f-Fit Results (Unifrm Ability Distributin, 0 = -2.5 t +2.5).. 42 Summary f the Gdness-f-Fit Results (Lwer Half f Nrmal Ability Distributin, X0 = 0.00, SDe = 1.00) 43 Summary f the Gdness-f-Fit Results (Upper Half f Nrmal Ability Distributin, X0 = 0.00, SD0 = 1.00) 44 Summary f the Gdness-f-Fit Results (Nrmal Ability Distributin, X0 = 0.00, SD 0 = 1.0). 45 Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hetergeneus Item Pl (Test Length = 10 Items).. 53 Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hetergeneus Item Pl (Test Length = 20 Items).. 54 V Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hetergeneus Item Pl (Test Length = 80 Items).. 55 Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hetergeneus Item Pl (Sample Size = 50 Examinees) 56 Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hetergeneus Item Pl (Sample Size = 200 Examinees) xiii

16 Table A. 3. A. 3. A. 3.8 A. 3. A A A A Page Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hetergeneus Item Pl (Sample Size = 1000 Examinees).. 58 Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hmgeneus Item Pl (Test Length = 10 Items) 60 Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hmgeneus Item Pl (Test Length = 20 Items) 61 Summary f Standard Errr Estimates fr Varius Sample Sizes and Ability Levels with a Hmgeneus Item Pl (Test Length = 80 Items) 62 Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hmgeneus Item Pl (Sample Size = 50 Examinees) 63 Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hmgeneus Item Pl (Sample Size = 200 Examinees) 6A Summary f Standard Errr Estimates fr Varius Test Lengths and Ability Levels with a Hmgeneus Item Pl (Sample Size = 1000 Examinees) 65 Variatin f Standard Errrs f Estimates at Several Ability Levels fr Different Test Lengths and Examinee Sample Sizes (Hetergeneus Item Pl) A.3.1A A.1 5. A Variatin f Standard Errrs f Estimates at Several Ability Levels fr Different Test Lengths and Examinee Sample Sizes (Hetergeneus Item Pl) Item Pl Parameters and Item Infrmatin at Five Ability Levels Test Cmpsitin and Infrmatin Using Five Item Selectin Methds Overlap f Test Items Selected Using the Five Item Selectin Methds Target and Scre Infrmatin Curves fr the Tw Test Develpment Prjects xiv A

17 LIST OF FIGURES Figure Page Six exaitiples f item characteristic curves , Graphical representatin f five item characteristic curves [b=-2.0, -1.0, 0.0, 1.0, 2.0; a=.59; c=.00] 23 Graphical representatin f five item infrmatin curves [b=-2.0, -1.0, 0.0, 2.0; a=.59; c=.00] 24 Graphical representatin f five item characteristic curves [b=-2.0, -1.0, 0.0, 1.0, 2.0; a=.59; c=.25] 25 Graphical representatin f five item infrmatin curves [b=-2.0, -1.0, 0.0, 1.0, 2.0; a=.59; c=.25] 26 Graphical representatin f five item characteristic curves [b=-2.0, -1.0, 0.0, 1.0, 2.0; a=1.39; c=.25] 27 Graphical representatin f five item infrmatin curves [b=-2.0, -1.0, 0.0, 1.0, 2.0; a=1.39; c=.25] 28 Standard errrs f estimatin assciated with three test lengths (10, 20 and 80 test items) at five ability levels and reprted fr three sample sizes (50, 200 and 1000 examinees) 59 Standard errrs f estimatin assciated with three test lengths at five ability levels and reprted fr tw item pls 72 Standard errrs f estimatin assciated with tw item pls at five ability levels and reprted fr three test lengths 74 Test infrmatin curves prduced with five item selectin methds [30 test items] 89 XV

18 Figure Page Schlarship test infrmatin curves prduced with five item selectin methds Bimdal test infrmatin curves prduced with fur item selectin methds 97 xvi

19 CHAPTER I INTRODUCTION AND STATEMENT OF THE PROBLEM 1.1 Backgrund and Review f the Literature There are many well-dcumented shrtcmings f standard testing and measurement technlgy. Fr ne, the values f standard item parameters (item difficulty and item discriminatin) are nt invariant acrss grups f examinees that differ in ability. This means that standard item statistics are nly useful in test cnstructin fr examinee ppulatins very similar t the sample f examinees in which the item statistics were btained. Anther shrtcming f standard testing technlgy is that estimates f an examinee's ability depend n the specific set f test items administered t that examinee. Therefre, cmparisns f examinee ability are nly meaningful in situatins where examinees are administered the same test items, parallel test items, r items that have been carefully equated. The fact that tests that have been develped emplying standard testing technlgy prduce ability estimates that depend n a specific set f items presents a particular prblem fr thse interested in tailred testing. Tailred tests are designed such that test items are administered t examinees that are carefully selected t "match" their ability levels (Lrd, 1970, 1974a; Weiss, 1976; Wd, 1973). In "tailred testing," it is likely that n tw examinees will take the same set 1

20 2 f test Items (r even the same number f test items). Since sme examinees will be administered mre difficult sets f test items than ther examinees, the usual examinee test scres d nt prvide an adequate basis fr ranking examinees n the ability measured by the test items. Besides the tw shrtcmings f standard testing technlgy mentined abve, standard testing technlgy has failed t prvide satisfactry slutins t many testing prblems (fr example, test design, test scre equating, and item bias). Fr these and ther reasns, many psychmetricians have been investigating and develping mre apprpriate theries f mental measurement. Cnsequently, cnsiderable attentin is being currently directed tward the field f latent trait thery. Latent trait thery can be traced back t the wrk f Lawley (1943, 1944). Lazarsfeld (1950) was perhaps the first t intrduce the term "latent traits." The wrk f Lrd (1952, 1953a, 1953b), hwever, is generally regarded as the "birth" f latent trait thery (r mdern test thery as it is smetimes called). Prgress in the w 1950 s and 60 s was painstakingly slw, in part due t the mathematical cmplexity f the field, the lack f cnvenient and efficient cmputer prgrams t analyze the data accrding t latent trait thery, and the general skepticism abut the gains that might accrue frm this particular line f research. Hwever, imprtant breakthrughs recently in prblem areas such as test scre equating (Lrd, 1975a; Rentz & Bashaw, 1975), tailred testing (Lrd, 1974a; Weiss, 1976), test and design and test evaluatin (Wright, 1968) thrugh

21 3 applicatins f latent trait thery, have attracted cnsiderable interest frm measurement specialists. Other factrs that have cntributed t the current interest in latent trait thery include the availability f a number f useful cmputer prgrams and publicatin f a variety f successful applicatins in measurement jurnals (Bck, 1972; Lrd, 1968, 1974a, 1975c; Samejima, 1969, 1972; Whitely & Dawis, 1974; Wright & Panchapakesan, 1969). A thery f latent traits suppses that in testing situatins, examinee perfrmance n a test can be predicted (r explained) by defining characteristics f examinees, referred t as traits, and using the scres t predict r explain test perfrmance (Lrd & Nvick, 1968). Since the traits are nt directly measurable and therefre "unbservable, they are ften referred t as latent traits r abilities. A latent trait mdel specifies a relatinship between bservable examinee test perfrmance and the unbservable traits r abilities assumed t underlie perfrmance n the test. The relatinship between the "bservable" quantites is described by a mathematical functin. Fr this reasn, latent trait mdels are miathematical V mdels. Latent trait mdels are based n a number f assumptins cncerning the test data. When selecting a particular latent trait mdel t apply t ne's test data, it is necessary t cnsider whether the test data satisfy the assumptins f the mdel. If they d nt, different test mdels shuld be cnsidered. Alternately, sme psychmetricians (fr example, Wright, 1968) have recmmended that test develpers design their tests s as t satisfy the assumptins f the particular latent trait mdels they are interested in using.

22 4 If latent trait thery is t fulfill the ptential it hlds fr the field f educatinal and psychlgical measurement, a methd fr develping tests by applying the thery must be established. As the state f the art exists, there is nly ne well-defined and field tested methdlgy fr develping tests, i.e., the applicatin f classical test thery methds t the develpment f tests. Many thereticians have been advcating the use f latent trait thery in the develpment f these types f tests. Hwever, n specific methdlgy that can be fllwed by the practitiner exists. Latent trait thery ffers several advantages t the psychmetrician interested in develping tests: Fr example, (1) invariant item parameters that facilitate the test develpment prcess as well as make pssible the develpment f tests fr a variety f applicatins, (2) a mathematical functin that can be manipulated t prvide valuable insights int hw examinees perfrm n specific test items, and (3) added infrmatin abut examinee ability derived frm new test scring methds (Hambletn, Swaminathan, Ck, Eignr, & Giffrd, 1977). Because f these and ther prperties f latent trait mdels, item selectin, and item analysis prcesses differ substantially frm thse emplyed when using standard testing technlgy. The fllwing is a brief descriptin f sme f the imprtant ways in which latent trait thery can facilitate the test develpment prcess: (1) statistics used t describe test items will nt depend n the ability distributin f the specific grup used t calibrate them (Lrd and Nvick, 1968) ; (2) when latent trait item parameters are knwn, the psychmetrician can examine the cntributin

23 5 f each test item t the test infrmatin curve, thus enabling the test develper t build a test which precisely fulfills a set f desired test specificatins; (3) a psychmetrician can frm different cmbinatins f items (tentative tests) in the initial stages f test develpment and cmpare the infrmatin curves f different sets f items at specific ability levels thus allwing him/her t chse the set f items mst suited fr the intended purpse f the test (Marc, 1977); (4) latent trait thery prvides a methd f examining item bias (Pine, 1976; Wright, Mead & Draba, 1976) which unlike classical test thery methds fr studying the prblem, is nt affected by the difference in the ability levels f examinee grups being investigated. 1.2 Statement f the Prblems In view f the many successful applicatins f latent trait thery t a variety f mental measurement prblems, the issue f whether r nt t use latent trait thery seems t be reslved. Hwever, latent trait thery is still relatively nev.* and hence there remain many prblem areas that need t be researched s as t increase the chance f successful applicatin f the thery t test develpment. Three f the prblem areas mst imprtant t the test develpment prcess fcus n (1) the rbustness f latent trait mdels; (2) the stability f item infrmatin functins; and (3) the use f item and test infrmatin functins fr item selectin.

24 6 The results f several studies have been reprted that relate t the questin f mdel rbustness (Diner & Haertel, 1977; Hambletn, 1969; Hambletn & Traub, 1976; Panchapakesan, 1969; Ck & Eignr, 1979). The findings have been quite cntradictry, perhaps in sme instances because f the cnfunding effects f sample size. The basic prblem with mst f the gdness-f f it and rbustness studies that have been cnducted recently is that they d nt prvide the practitiner with infrmatin that he/she may use when applying latent trait thery t the test develpment prcess. It is imprtant fr practitiners t see cmparisns f the fit f latent trait mdels t varius data sets using a criterin measure that has sme practical meaning t them. T date there have been n cmparative studies f the varius latent trait mdels using practical criteria t judge the results. The increasing use f test infrmatin functins as a means f cnstructing and evaluating tests, ^has been dcumented in mst f the current literature cncerning educatinal and psychlgical measurement. Hwever, sme imprtant questins remain t be answered befre these functins can be ptimally applied t prduce the desired results. These questins address the stability f the functins under varying circumstances. Variables unique t each testing situatin, such as the characteristics f the item pl, the number f examinees used t estimate the parameters f the items cntained in the pl and the number f items cmprising the test will be reflected in the accuracy f the estimate f test infrmatin. Therefre, it seems apparent that an investigatin f the influence f

25 . 7 these variables n the stability f infrmatin functins wuld be useful t thse interested in using these functins as part f the test develpment prcess. Lrd (1977) discussed a prcedure, utlined by Birnbaum (1968), fr building a test utilizing item infrmatin functins. This prcedure cnsists f the fllwing steps: 1. Decide n the purpse f the test. Based n this purpse, determine the standard errr f estimate required at each ability level and cnsequently the target infrmatin curve 2. Select items with item infrmatin curves that fill hard t fill areas under the target infrmatin curve. 3. Cntinue t select items until the test infrmatin curve apprximates the target infrmatin curve with the desired degree f accuracy. The majr prblem with this prcedure cncerns the fact that it is nt sufficiently peratinalized t be applied by the practitiner interested in using infrmatin functins t build tests. Fr example, a specific methdlgy must be develped fr: (1) establishing target infrmatin curves that are suitable fr varius testing purpses, and (2) selecting items such that the fewest number f items will be selected in the mst efficient manner. It seems apparent that the peratinalizatin f Birnbaum s prcedure, including the develpment f algrithms fr item selectin, wuld greatly expedite the applicatin f latent trait thery t the test develpment prcess.

26 8 1.3 Purpses The previus sectin f this chapter has delineated a number f areas that require research befre latent trait thery can be successfully applied t the develpment f nrm-referenced tests. The research presented in this dissertatin cncentrates n three specific areas. The fcus f the first area f research that is described in this thesis was the rbustness f latent trait mdels. The purpse f this research was t study the "gdness-f-f it" f the ne-, tw-, and three-parameter mdels emplying practical criteria fr assessment. The secnd part f the study investigated the stability f test infrmatin functins. The cncerns f this study were t systematically investigate: 1. The effects f examinee sample size and test length n the precisin f test infrmatin functins. 2. The effects f the statistical characteristics f an item pl n test infrmatin functins. The purpse f this part f the study was t prvide guidelines t aid test develpers in determining the cnfidence they shuld have in the test infrmatin functins that they utilize in their wrk. The third area f research addressed in this thesis was the peratinalizatin f Birnbaum s prcedure fr the use f item infrmatin functins fr item selectin. The purpse f this research was t develp a set f guidelines t be used by the practitiner when making decisins cncerning a number f practical prblems that arise when emplying infrmatin functins t build

27 9 tests. In rder t accmplish this purpse, studies were carried ut that fcused n the develpment and cmparisn f item selectin algrithms suited fr specific test cnstructin purpses. This study investigated hw best t establish a target infrmatin curve and cmpared several algrithms fr selecting items t fit infrmatin curves. 1.4 Organizatin f the Study The next five chapters f this thesis are rganized in the fllwing manner. Chapter II prvides the theretical framewrk fr the research. Chapter III presents the rbustness studies. Chapter IV cntains the studies cncerning the stability f item infrmatin curves, and Chapter V cntains the studies related t the peratinalizatin f Birnbaum s prcedure. Chapters III thrugh V are self-cntained and share the fllwing frmat: 1. Intrductin 2. Methds f Investigatin 3. Results and Discussin 4. Cnclusin The sixth and final chapter in this thesis is devted t a summary f the studies as well as cnclusins and suggestins fr further research.

28 CHAPTER II LATENT TRAIT MODELS AND RELATED NCEPTS 2.1 Intrductin Ths purpse f this chapter is t intrduce the tpic f latent trait mdels. First, a brief nn mathematical intrductin t the thery f latent traits will be prvided. Secnd, the features f three latent trait mdels that seem t be particularly apprpriate fr use with mental test data will be reviewed. Third, the classical test mdel will be cmpared with latent trait mdels. 2.2 Features f Latent Trait Mdels There are at least three fundamental ntins in the general thery f latent traits: The dimensinality f the latent spaae^ lcal independence 3 and item characteristic curves. Each f these ntins will be discussed briefly belw. The dimensinality f the latent space refers t the number f latent traits that underlie examinee test perfrmance. It is typical t assume that the latent space is unidimensinal; that is, assume that the items in a test are hmgeneus in the sense f measuring nly a single ability r latent trait. Latent trait mdels in which the unidimensinal assumptin is nt made are cmplex, and t date, nt well develped. Accrding t Lrd (1968), the assumptin 10

29 11 cncerning the unidimensinal nature f a set f items is nt strictly true fr mst tests. Hwever, he adds that it may prvide a tlerably gd apprximatin in sme instances. The apprpriateness f the assumptin f unidimensinality fr any set f mental test data can be partially studied thrugh a factr analysis f the test items. (Fr details f ne attempt at this, the reader is referred t Hambletn and Traub [1973].) When the items in a test measure mre than a single ability, the items can be clustered int hmgeneus grups n the basis f the results frm a factr-analytic study. Then, a latent trait analysis can be applied t each hmgeneus cluster f items. All further discussins f latent trait mdels in this Chapter will be restricted t mdels that assume a single ability underlying test perfrmance. The secnd ntin is the principle f lcal independence. There are tw frms f this principle, referred t as the strng and weak frm f the principle f lcal independence. The strng frm f the principle states that the test item respnses f each examinee are statistically independent. This means, fr example, that the prbability f any examinee respnse pattern acrss a set f test items is given by the prduct f prbabilities representing success n each item fr that examinee. Als, it means that examinee perfrmance n ne test item des nt affect the examinee s success r failure n any ther item in the test. The weak frm f the principle is btained by substituting uncrrelated fr "statistically independent" in the statement f the principle. The distinctin between the tw frms f the principle

30 12 is the same ne we ften make in crrelatinal research: we distinguish between variables being statistically independent and variables being uncrrelated; the first cnditin being a strnger statement abut the relatinship between tw variables than the secnd. It is relatively easy t see that the assumptin f lcal independence and the assumptin f a unidimensinal latent space are identical. T say that a single ability accunts fr examinee perfrmance n a set f test items is the same thing as saying that fr examinees at the same ability level, their item respnses are statistically independent, i.e., satisfy the principle f lcal independence. If this were nt the case (i.e., if examinee item respnses were statistically dependent), then it wuld fllw that at least ne mre ability was being measured by the test items. The interested reader is referred t Lrd and Nvick (1968) fr further clarificatin f this pint. The principle f lcal independence represents a restrictive assumptin and s may nt be satisfied with many sets f mental test data. Because f the equivalence f the principle f lcal independence and the assumptin f unidimensinality, the apprpriateness f the principle f lcal independence with any data set can als be tested, in part, using factr analytic techniques. It shuld be recgnized that the principle f lcal independence des nt imply that test items are uncrrelated ver the ttal grup f examinees (Lrd & Nvick, 1968, p. 361). Psitive crrelatins between pairs f items will result whenever there is variatin amng the examinees n the ability measured by the test items.

31 13 The third ntin is that f an item aharacteristic curve (smetimes referred t as a trace line, r an item characteristic functin when the latent space is multidimensinal, i.e., when the number f latent traits underlying test perfrmance exceeds ne). An item characteristic curve is a mathematical functin that relates the prbability f success n an item t the ability measured by the test. A primary distinctin amng varius latent trait mdels is in the mathematical frm f the item characteristic curve. Examples f the mathematical frms f item characteristic curves f six latent trait mdels are shwn in Figure Each item characteristic curve fr a particular latent trait mdel is a member f a family f curves f the same general frm. Fr example, the item characteristic curve f the latent linear mdel (Figure 2.2.1, C) has the general frm Pg(0) = bg + ag0, where Pg(0) designates the prbability f a crrect respnse by an examinee with ability level 6, n an item g that is described by tw parameters, dented a and b. An item characteristic S C curve is specified cmpletely when the general frm is specified and the parameters f the curve fr a particular item are knwn. The number f parameters required t describe an item characteristic curve will depend n the particular latent trait mdel. It is cmmn thugh fr the number f parameters t be ne, tw, r three. While item characteristic curves in the latent linear mdel are all straight lines (a restrictin placed n us when we select the latent linear mdel), acrss different items in the test, the "curves (r lines in this particular case) will vary in their intercepts and slpes t reflect the fact that the test items vary in "difficulty" and "discriminating pwer."

32 14 (a) perfect scale carves (b) latent distance curves (c) latent linear curves (d) ne-parameter lgistic curves (e) tw-parameter lgistic curves (f) three-parameter lgistic cirves Figure Six examples f item characteristic curves.

33 15 In any practical applicatin f latent trait mdels, it is usually necessary t specify the mathematical frm f the item characteristic curves and btain estimates f the item parameters needed t describe the curves. Readers are referred t Bck (1972), Lrd (1968, 197Aa), Whitely and Dawis (1974), Wright and Panchapakesan (1969), and Hambletn, Swaminathan, Ck, Eignr, and Giffrd (1979) fr details n sme f the current methds fr estimating item characteristic curve parameters (and ability estimates as well) f sme f the mre ppular latent trait mdels. An item characteristic curve represents the prbability f a crrect answer t an item expressed as a functin f ability. Hwever, the prbability f a crrect answer t an item is bviusly independent f the distributin f examinee ability in the ppulatin f examinees f interest. Clearly, the prbability f a crrect respnse fr an examinee will nt depend n hw many ther examinees are lcated at the same lcatin n the ability cntinuum. Therefre, the shape f an item characteristic curve des nt depend n the distributin f ability in the examinee ppulatin. In sme V sense then, the shape f the curve will be invariant acrss different samples f examinees frm that ppulatin, regardless f hw the sample f examinees is selected. This imprtant pint will be expanded n later Cmmn Frms f Item Characteristic Curves In this sectin, three mathematical functins that are cmmnly used t represent item characteristic curves will be intrduced. All three functins can be applied t binary scred items administered

34 under nn-speeded cnditins. (When the items are administered 16 under speeded cnditins, it becmes necessary t distinguish between "mitted items and "nt reached" items by examinees s as t prperly estimate examinee ability scres.) (a) Tw-Parameter Lgistic Curves Birnbaura (1968) prpsed a latent trait mdel in which the item characteristic curve takes the frm f a tw-parameter lgistic distributin functin, P^(e)= edag(e-bg) (g = 1, 2,..., n) [ ] In this equatin, P (9) is the prbability that an examinee with O ability 9 answers item g crrectly, a and b are parameters fr O O item g(g=l, 2,...,n) and n is the number f items in the test. The parameter bg is usually referred t as the index f item difficulty It represents the pint n the ability scale at which the slpe f the item characteristic curve is a maximum. The parameter, a^, called item discriminatin, is prprtinal t the slpe f Pg(9) at the pint 9 ^)^. The cnstant D is a scaling factr. Usually we take D=1.7, t maximize the agreement between the lgistic mdel and the nrmal-give mdel, a mdel that was riginally studied by Lrd (1952) but is mathematically incnvenient t wrk with. The item difficulty parameter, bg, is defined n the same scale as ability [-«>, +«>]. In practice thugh the range f b^ is frm abut -2 t +2 (assuming the ability distributin is centered with a mean equal t zer and standard deviatin equal t ne).

35 17 As bg takes n values frm -2 t +2, the items mve frm being very easy t very difficult fr the grup f examinees. The item discriminatin parameter, ag, is defined, theretically, n the scale [-<», +=0 ]. Hwever, negatively discriminating items are discarded frm ability tests, and it therefre is unusual t btain ag values larger than tw. High values f a result in item char- O acteristic curves that are very "steep." Lw values f a lead t O item characteristic curves that increase gradually as a functin f ability. Careful inspectin f the tw-parameter lgistic mdel reveals an additinal implicit assumptin characteristic f mst latent trait mdels: guessing des nt ccur. That this must be s is apparent frm the fact that as lng as a >0 (that is, as lng as O there is a psitive relatinship between perfrmance n the test item and the ability measured by the test), the prbability f a crrect respnse t an item decreases t zer as ability decreases. (b) Three-Parameter Lgistic Mdel The three-parameter mdel is btained frm the tw-parameter mdel by adding a third parameter, dented c. The mathematical O frm f the three-parameter lgistic curve is written Pg(e) = cg + (1 Cg) ^Dag(e-bg) l+edag(e-bg) (g 1, 2,..., n). [ ] The parameter c is the lwer asymptte f the item characteristic O curve and represents the prbability f lw ability examinees crrectly answering a questin. The purpse f including a paramter Cg

36 18 int the mdel is t attempt t accunt fr the misfit f item characteristic curves at the lw end f the ability cntinuum, where amng ther things, guessing is a factr in test perfrmance. It has been cmmn t refer t the parameter c as the guessing O parameter in the mdel. It is perhaps surprising t nte then that typically the parameter Cg takes a value smaller than the value crrespnding t the prbability f a crrect answer t a test item frm randm guessing. As Lrd (1974b) has nted, this event is prbably due t the ingenuity f item writers in develping "attractive" but incrrect chices. Fr this reasn, discntinuatin f the label "guessing parameter" t describe the parameter Cg wuld seem t be desirable. (c) One-Parameter Lgistic Mdel (Rasch Mdel) Many researchers have becme aware f the wrk f Gerg Rasch, a Danish mathematician, in the area f latent trait mdels (Rasch, 1966), bth thrugh his wn publicatins and the papers f thers advancing his wrk (Andersn, Kearney, & Everett, 1968; Wright, 1968; Wright & Panchapakesan, 1969). Althugh the Rasch mdel was V develped independently f ther latent trait mdels and alng quite different lines, Rasch* s mdel can be viewed as a latent trait mdel in which the item characteristic curve is a ne-parameter lgistic functin. Cnsequently, Rasch* s mdel is a special case f Birnbaum s tw-parameter lgistic mdel, in which all items are assumed t have equal discriminating pwer and vary nly xn terms f difficulty. The frm f the item characteristic curve fr this mdel can then be written as

37 Pg(e) = ^Da(0-bg) l+edace-bg) (g - 1, 2,..., n), [ ] 19 in which a, the nly term nt previusly defined, is the cmmn level f discriminatin fr all the items and is ften set equal t ne. 2.3 The Ability Scale and Its Meaning That there is a mre basic scale f ability than the true scre scale fr a test is bvius when it is recgnized that the true scre distributins (and bserved scre distributins) f nnparallel measures f a cmmn ability will differ. The ability scale fr a particular latent trait mdel is defined such that the distributin f abilities in a grup f examinees will be identical regardless f the particular test measuring the ability (Lrd, 1975a). The ability scale is chsen s that the relatinship between ability scres and item respnses can be represented by item characteristic curves f sme specified mathematical frm. The ability scale is "stretched" and "cmpressed" at different pints s as t maximize the "fit" between the item respnses, item characteristic curves, and the ability scres. The resultant ability scale is unique up t the rigin and unit f measurement which are arbitrary. 2.4 Test Infrmatin and Efficiency Once a latent trait mdel is specified, the precisin with which it estimates examinee ability can be determined. Of curss, the validity f the results will depend n the match between the mdel

38 20 and the test data. Fllwing Sir Rnald Fisher's imprtant statistical wrk in the 1920' s, Birnbaum (1968) defined the ntin f infrmatin as a quantity inversely prprtinal t the squared length f the cnfidence interval arund an estimate f an examinee's ability. The standard errr f estimate f ability is equal t 1/ /infrmatin. Wlien infrmatin at an ability level is high, we have narrw cnfidence bands. Because the infrmatin functin varies with ability level, it has been suggested that test infrmatin curves ught t replace the use f classical reliability estimates and standard errrs f measurement in test scre interpretatins. In mathematical terms, Birnbaum (1968) gives the infrmatin curve f a given scring frmula by 2 Iy(e) - I g g [2.4.1] I g=l w^gpgqg In the expressin abve, Iy(9) is the amunt f infrmatin at ability level 6 prvided by the scring frmula y, where the variable X is 0 r 1 depending n whether r nt item g is O answered crrectly; Pg is the prbability f a crrect answer t item g by an examinee with ability level 0; Qg is equal t 1 - Pg; P' is the slpe f the item characteristic curve at ability level 0; and the item scring weights are w^, g = 1, 2,..., n.

39 ,, 21 Birnbaum (1968 has shwn that the maximum value f ly(0), referred t as the test infrmatin curve, is given by n p ' 2 1 ( 9 ) = I g g=l [2.4.3] The maximum value f the infrmatin curve f a given scring frmula is btained when the scring weights are chsen, such that w = g PV PpQ g^g [4.4.4] S, in rder t btain the test infrmatin curve, and cnsequently minimize the widths f cnfidence bands abut examinee ability estimates under the ne-, tw-, and three-parameter lgistic mdels, the scring weights shuld be chsen t be 1, Dag> ^nd Dag ^g/ (l+e^^g^ ~^g^ ^ 8 (g = 1, 2,... n) respectively. (Infrmatin curves and the best scring weights fr ther latent trait mdels are given by Samejima [1969, 1973].) Only fr the three-parameter mdel are the scring weights a functin f ability level. The scring system in the three-parameter mdel has the effect f reducing the weight assigned t crrect answers n items where the lwer asympttes (cg) f the item characteristic curves are large. Als, the weights fr such items are smaller fr lw-ability examinees than fr either middle- r high-ability examinees, t reflect the fact that lw-ability examinees are mst likely t be answering the items by guessing. Fr high-ability examinees, the ptimum scring v. eights f the items apprach the quantity Dag(g=l, 2,..., n).

40 The quantity P'g^/PgQg in Equatin [2.5.2] is the tntributin 22 f Item g t the infrmatin functin f the test. Fr this reasn It is called the item infrmatin functin. Item infrmatin functins have an imprtant rle in determining the accuracy with which ability is estimated at different levels f 0. Each item infrmatin curve depends n the slpe f the particular item characteristic curve and the cnditinal variance f test scres at each level 9. The higher the slpe f the item characteristic curve and the smaller the cnditinal variance, the higher will be the item infrmatin curve at that particular ability level. The height f the item infrmatin curve at a particular ability level is a direct measure f the usefulness f the item fr precisely measuring ability at that level. Figures (frm Hambletn, 1979) prvide three sets f typical item characteristic curves (Figures 2.4.1, and 2.4.5) and crrespnding item infrmatin curves (Figures 2.4.2, and 2.4.6). The effects f increasing the values f the item discriminatin and pseud-chance curves are clear. High item discriminatin indices result in "steeper item characteristic curves and higher amunts f infrmatin acrss the ability cntinuum than lw item discriminatin indices. In additin, when item pseud-chance level indices exceed zer, the lwer asympttes f the item characteristic curves are different frm zer, and the test items prvide less infrmatin, especially at the lw end f the ability cntinuum, than test items with the pseud-chance level values clse t zer.

41 characteristic representatin 23 curves c=.ol. item five f a=.59: 2.0: 1.0, 0.0, Graphical rb=-2.0,-1.0, Of 1.0 AiiiiqeqJd 'j- CN cu v-i D 60 H Pm

42 J O ( cvi 0 s;* cu 00 >^ K»IM evm < CM I O CMT-OCncOh-lO':^-CMT-0 uijeiujiu} Figure Graphical representatin f five item infrmatin curves [b= -2.0, -1.0, 0.0, 1.0, 2.0; a=.59; c=.ool

43 25 If) 0 > Z3 O 4-i U) 0 +-* 0 r ' LT) ' CNJ E 0 4-J MW II O CD 0 lo > Vt II»«0 m c: O cvi j-j c c f- 0 (f) 0 CL 0 k. *0 O r- 1 *% a 1 in i 0 II O. JD 1 m 1.00 'Cf CM QJ 3 bo H

44 representatin 26 curves I c\i infrmatin c=.25.. item 2. Cv3 d > < five f a=.59; 2.0; -1.0,!,1.0, 0 cm 1 Graphical [b=-2.0, CMT-OC3^I^CDtO^CMT-0 I Figure uijeuujiu]

45 characteristic representatin 27 curves c=.25. item five f a=1.39; 2.0; 1.0, 0.0, -1.0, -2.0, Graphical \h = Figure

46 OOr^LO'^tCM-r-O UOP.SUJJO^UI ^ 0 01 CM I O I Ct3.< (/) ; > 23 O c -t-t Q E c E 0 I" T" CNi II CD 0 > T- c 11 O CM O V c 0 O cn 0 Q. 0 1 "c ' CM x: 1 D- r k. O 1 V.O <r CM cu V-i 3 bo 'H Cii ll Xi 1

47 29 Frm Equatin [2.4.3] it is clear that items cntribute independently t the test infrmatin functin. Birnbaum (1968) has als shwn that with his three-parameter mdel, an item prvides maximum infrmatin at an ability level 6, where T:^ l e 1/2(1 + A + 8cg). [2.4.5] If guessing is minimal, then c = 0, and 9 = b. When c >0, the s s pint f maximum infrmatin is shifted t the right f the item difficult value, b. O If nn-ptimal scring weights are used with a particular item characteristic curve mdel, the infrmatin curve derived frm Equatin [2.4.1] will be lwer, at all ability levels, than ne that wuld result frm the use f ptimal weights. Birnbaum (1968) used the term eff'i'iency t refer t the infrmatin lss due t the use f less than ptimal scring weights. Efficiency is studied by calculating the rati f the values f the actual infrmatin curve and the test infrmatin curve at each ability level. 2.5 The Classical Test Mdel Versus Latent Trait Mdels In view f the cmplexities invlved in applying the latent trait mdels, and the restrictiveness f the assumptins underlying the mdels, ne may ask: Why bther? After all, classical test mdels are well-develped, have lead t many imprtant and useful results, and they are based n weak assumptins. Therefre, the classical test mdels can be applied t mst (if nt all) sets f mental test data.

48 30 In cntrast, latent trait mdels are based n stvng assumptins which limit their applicability t many mental test data sets. On the ther hand, strng assumptins imply strng results. Perhaps the mst imprtant advantage f latent trait mdels (Bck & Wd, 1971) is that given a set f test items that have been fitted t a latent trait mdel (that is, the item parameters are knwn), it is pssible t estimate an examinee's ability n the same ability scale frm any subset f items in the dmain f items measuring the ability. (Of curse, the dmain f items needs t be hmgeneus in the sense f measuring a single ability. If the dmain f items is t hetergeneus, the ability estimates will have little meaning.) In fact, regardless f the number f items administered, r the statistical characteristics f the items, the ability estimate fr each examinee will be an unbiased estimate f true ability. Ability estimatin independent f the particular chice (and number) f items represents ne f the majr advantages f latent trait mdels. Hence, latent trait mdels prvide a way f cmparing examinees even thugh they may have taken quite different subsets f the test items. It is this feature that makes the latent trait mdel mst useful in the field f tailred testing. In tailred testing, examinees receive test items that are matched t their ability level. Nevertheless, the ability estimates fr examinees are n a cmmn ability scale and therefre examinees can be cmpared. Clearly, the usual test scre metric will nt permit meaningful cmparisns f examinees when the tests taken by the examinees are nt matched n difficulty. In latent trait mdexs

49 . 31 the difficulty f items is accunted fr by the mdel and reflected in the ability estimates. Thus, tw students, dented A and B, receiving identical scres n an easy and difficult subset f the test items, respectively, will differ in their ability estimates (B will receive higher ability scre than A) Tw ther prblems that can be reslved thrugh the applicatin f latent trait mdels are the prblems f develping parallelfrms f a test and equating scres frm ne test t anther that measure the same ability. Bth prblems can be directly reslved thrugh fitting the test data t a latent trait mdel. Anther advantage f latent trait mdels is that the item parameters are invariant acrss sub-grups f examinees frm the examinee ppulatin. In principle, the item parameters shuld remain the same regardless f the sub-grup tested. Invariant item parameters have been sught by measurement specialists fr a lng perid f time; the advantages f which are bvius fr test develpment wrk. Certainly classical item statistics such as item difficulty and discriminatin d nt qualify. Fr example, it is well-knwn that item difficulty will vary frm grup t grup depending upn the average ability f the grup being tested. Yet anther desirable prperty f the latent trait mdels is the prvisin f a measure f the precisin f ability/ estimatin fr each ability level. Thus, instead f a single estimate f the size f errrs in individual examinee scres prvided by the standard errr f measurement, the latent trait mdels make it pssible t

50 32 prvide separate estimates f errr fr each examinee that are specific t each ability level.

51 CHAPTER III ROBUSTNESS OF LATENT TRAIT MODELS 3.1 Intrductin While the ptential usefulness f latent trait mdels is great, there remain many practical prblems t address at the applicatin stage. Fr ne, hw des a user g abut selecting a latent trait mdel? One might be tempted t say that the user shuld always wrk with the mre general mdels since these mdels will prvide the "best" fits t the available test data. Unfrtunately, the mre general latent trait mdels (fr example, the three-parameter lgistic test mdel) require mre cmputer time t btain satisfactry slutins, require larger samples f examinees and lnger tests, and are mre difficult fr practitiners t wrk with. Clearly, mre needs t be knwn abut the "gdness-f-f it" and "rbustness" f latent trait mdels. Such infrmatin wuld aid practitiners in the imprtant step f selecting a test mdel. There has been sme wrk n the "gdness-f-f it" between latent trait mdels and a variety f test data sets (see fr example, Lrd, 1975; Tinsley & Dawis, 1977; Wright, 1968). Hambletn, Swaminathan, Ck, Eignr, and Giffrd (1978) have reviewed these as well as ther studies and have nted that almst all f the studies use a chi-square statistic as the criterin measure. The prblems related t using this statistic t assess gdness-f-f it will be discussed shrtly. 33