Assessing the Reliability of Rating Data

Transcription

1 Paul Baett age 1 Assessing the Reliability of Rating Data Ratings ae any kind of coding (qualitative o quantitative) made concening attitudes, behavious, o cognitions. Hee, I am concened with those kinds of atings made by thidaties of a aticula individual s attitudes, behaviou, o cognitions. These might be fom ating scales, obsevational check-lists, o symtom check-lists etc. The incile aim of eliability analysis is to detemine the degee of ageement between ates when using a aticula ating scheme. If the eliability is low, then the scheme itself may be at fault, o the ates, o both! I am not going to ty and descibe all ossible kinds of designs and analyses, but only those that might be most common within the mental health setting. As always, the quantitative oeties of the atings must be consideed fist. Then, an aoiate statistic might be chosen to summaise the degee of ageement between ates. Fist an imotant distinction between inte-ate and inta-class coelations. Inteate coelation (inteate ). This is whee the similaity between atings is exessed as a coelation coefficient geneally using a Peason oduct-moment tye coefficient. In x tables (fo comaison of just ates), it is ossible to use a ange of measues of ageement, anging fom the hi coefficient though to say Jaccad s coefficient that excludes all non-occuences fom the calculations). See the DICHOT 3.0 ogam (downloadable fom : htt:// fo the imlementation of seveal of these coefficients. Fo examle, woking fom a x table with cell IDs as : Rate 1 - Yes Rate 1 - No Rate - Yes A B Rate - No C D We could comute, fo examle, the following measues of ageement each of which takes into account the maginal fequencies in secific ways ( A* D B * C) Phi the Peason Poduct Moment ( A + B) *( C + D) *( A + C) *( B + D) Yule s Q (o gamma) (A*D - B*C)/(A*D + B*C) The Jaccad, J A/(A+B+C) The G-Index, G ((A+D)-(B+C))/N Bennett s B index, B ((A*D-xkon )/((A+xkon)*(D+xkon)) whee: xkon (B+C)/ and the Hams and Ihm (1981) adjustment is made (to guad against A o D fequencies 0) A A+1, B B+1, C C+1, DD+1 The outut fom DICHOT 3.0 shows how they comae (the ogam includes detailed exlanations of the logic of each coefficient in its online hel you aleady have this as a handout). Assessing the Reliability of Rating Data- Revised Mach 001

2 Paul Baett age Let us take the examle whee we ae looking at the amount of ageement between two ates, on an item fom the VRAG Item 1: Lived with both biological aents to age 16 Raw Rating Table Hee we have a simle x table layout, which we can ente into DICHOT 3.0 fo a comlete analysis. 4 atients have been ated, and Rate 1 agees with Rate on (1 + 16) 8 atients. On 14, thee is disageement. Assessing the Reliability of Rating Data- Revised Mach 001

3 Paul Baett age 3 DICHOT 3.0 Analysis Assessing the Reliability of Rating Data- Revised Mach 001

4 Paul Baett age 4 As can be seen, thee is consideable vaiance between the values of the vaious coefficients. This is mild comaed to some diffeences that may be obseved. What is imotant is that you undestand the ationale behind the coefficient being used, and ae thus able to inteet its value accodingly. Play aound with DICHOT 3.0 to see just how fa the values can sometimes vay. Fo examle, take the table below Rate 1 - Yes Rate 1 - No Rate - Yes Rate - No 10 8 Whee 64 atients ae ated on a Yes/No ating vaiable. They agee on Yes fo 33 atients, and on No fo 8, the emaining atients ae classified diffeentially by the ates. The esults: Kaa ageement (Cohen s Kaa) Kaa was designed secifically as a measue of ageement between judges, whee atings ae categoical, and whee a coection fo chance ageement is made. This coefficient thus diffes fom the ecent ageement aoach adoted by some, because this simle calculation does not take into account what the chance-level ageement between judges would be alone, assuming they both guessed andomly. The fomula fo kaa comuted fo any numbe of atings categoies used by two ates/judges is: κ f o N f e f e whee fo obseved fequencies in the diagonal fe exected fequencies in the diagonal N Numbe of Patients The exected fequencies ae the same as those calculated fo the Peason Chi-Squae calculation, excet we use just the diagonal values (A and D) fo both obseved and exected fequencies. In contast to this fomula, we might conside use of the Jaccad coefficient, which is anothe measue of inteate ageement, but one that excludes joint-negatives fom its calculation. A useful oint is that both kaa and the Jaccad coefficients can be inteeted as % values. Kaa can be inteeted as the % ageement afte coecting fo chance. The Jaccad coefficient can be inteeted as the % ageement afte excluding joint negative ais. Both coefficients vay between 0 and 1 (o 0 to 100%). DICHOT 3.0 comutes this coefficient fo x tables. Assessing the Reliability of Rating Data- Revised Mach 001

5 Paul Baett age 5 If we extend ou examle to an analysis of the eliability of a 3-oint ating, we might have as an examle Rate 1 - High Rate 1 - Med Rate 1- Low Rate - High Rate - Med Rate - Low Hee we have 5 atients ated by two ates, using a high-medium-low ating fame. The diagonal exected fequencies geneated unde a hyothesis of indeendence ae: Rate 1 - High Rate 1 - Med Rate 1- Low Rate - High.4 Rate - Med 4 Rate - Low 1. Ou fomula is: κ So.. f o N f e f e whee f o obseved fequencies in the diagonal f e exected fequencies in the diagonal N Numbe of Patients κ f o N f e f e ( )-( ) 5-( ) kaa fo these data Assessing the Reliability of Rating Data- Revised Mach 001

6 Paul Baett age 6 Intaclass Coelation (Intaclass ) This coefficient coects fo a fatal flaw with inteate coelation comuted using oduct-moment coelations. That is, inteate takes no account of the vaiance between the ates. Remembe that oduct-moment coelations use standadized data, which effectively emoves the comonent of individual ate vaiability. Essentially, oduct moment coelations ae insensitive to scale, but sensitive to monotonicity elations between data. A simle examle to how misleading inteate coelations can be is given below: Atificial Data file 10 atients, 3 ates (100 oint ating scale) Comuting the inteate (eason coelation) between ates 1 and, we get 1.00 (even though the atings diffe dastically) The Intaclass (Shout and Fleiss model ) assumes that each atient is ated by two o moe ates. These ates ae andomly selected fom a lage oulation of ates. Each ate ates all atients. (In effect, a two-way ANOVA andom effects model) is Comuting the inteate (eason coelation) between ates 1 and 4, we also get 1.00 (now the atings tuly ae identical). The Intaclass fo these data is also 1.00 This simle examle indicates why the intaclass is always to be efeed to inteate. Befoe we delve into the comutations and comute-file layouts fo thee tyes of intaclass coelation (the Shout and Fleiss models 1,, and 3), it is wothwhile to mention two othe methods of assessing inteate eliability. Fo inteval-level data, we might use coefficient alha, and if ou atings ae to be consideed odinal, we would use Kendall s Coefficient of Concodance (I have ovided the elevant ages fom Siegel and Castellan s textbook fo Kendall s coefficient). When using the alha coefficient, we ae making a measue of the intenal consistency between ates. It is in fact algebaically equivalent to the intaclass coelation coefficient whee thee is only one ating (deendent) vaiable (o item) being ated and IF we assume that the judges atings ae to be aveaged to oduce a Assessing the Reliability of Rating Data- Revised Mach 001

7 Paul Baett age 7 comosite ating. Essentially, this coefficient tells you how eliable that atings ae as a whole (how intenally consistent ae the judges atings). Howeve, because of this aveaging of atings, we educe the vaiability of the judges atings such that when we aveage all judges atings, we effectively emove all the eo vaiance fo judges. Take a look at the ANOVA fomula below ic whee n n j jav + (( n j n jav ) ) / n mean squae effect fo Patients/Pesons mean squae esidualeffect the numbe of ates/judges the numbes of judges to be aveaged jav Now, when n j n jav we have ic ic + (( n j n jav ) ) / n jav + (( n) ) / n jav which in fact is an altenative fomula fo coefficient alha, the measue of intenalconsistency that we ae familia with in questionnaie sychometics. Assessing the Reliability of Rating Data- Revised Mach 001

8 Paul Baett age 8 Out of inteest, let s look at a oblem whee we comute ou inteate eliability using coefficient alha. The data file looks like: Each atient is ated by a judge, on a 1-10 oint ating scale. Assuming the data ae equalinteval, we comute coefficient alha as In essence, we have teated the judges as items in a questionnaie, and ou atients ae the obsevations on these items. Thus, we ae in effect doing an item analysis. The conventional (one that might look familia to you!) fomula fo alha we ae using is: α k k 1 k si i 1 1 ST whee k the numbe of items (judges) s i item (judge) vaiance i of k S T the total test scoe vaiance and α k 1 k 1 k si i 1 ST If we comute a -way ANOVA on the data file, with Judges as the eeated measues facto, we obtain Assessing the Reliability of Rating Data- Revised Mach 001

9 Paul Baett age 9 Statistica ANOVA setu sceen, with Patients as andom effects And Which, if we now use the ANOVA fomula fo alha gives us ic α The SPSS 9/10 commands to geneate these data ae via the Analyze Menu, then Geneal Linea Model, with submenu Reeated Measues. Then setu the Rates facto Assessing the Reliability of Rating Data- Revised Mach 001

10 Paul Baett age 10 Pess Define and make the selections so as to look like this Then OK and these ae the esults And Assessing the Reliability of Rating Data- Revised Mach 001

11 Paul Baett age 11 Anyway, afte that digession, let s go back to the thee main designs that encomass Intaclass Coelation eliability designs. Assessing the Reliability of Rating Data- Revised Mach 001

12 Paul Baett age 1 Hee, I am following the teatment outlined in the excellent chate by Owin (1996) who eots the seminal wok by Shout and Fleiss (1979). Some of the below can also be easily ecast within genealizability theoy aoaches (see Cocke and Algina (1986) but this is so confusingly demonstated that I much efe the claity of Owin and Shout and Fleiss. Essentially, thee ae thee models that concen us: Model 1: Each atient to be ated is ated by a unique ate, with each ate andomly selected fom a lage oulation (a one-way ANOVA andom effects model). Secifically, fo evey atient vaiable o item to be ated, thee is a unique ate. Each ate makes only one ating decision. This model assumes you have a lage ool of ates, who ae andomly assigned to make one ating e atient e vaiable. So, fo a study in which we ate 10 atients on 5 vaiables, we would need 50 ates. The ANOVA fomula is: 1 W + ( n 1)* W whee with W Between Patients mean squae n numbe of ates and n numbe of atients es Residual mean squae W Within Patients mean squae Between Rates ("measues") mean squae ( *( n 1) ) + ( es* ( n 1) *( n 1) ) n *( n 1) Model : Evey atient is ated by each ate. We assume the ates ae andomly selected fom some oulation of ates (a two-way andom effects model). In essence, each ate ates all atients on all vaiables. This is the default model that coves most ating situations. Fo examle, fo a study in which we ate 10 atients on 5 vaiables, we would need at least ates in ode to assess inteate eliability. Each ate would make (10*5)50 ating judgements. The ANOVA fomula is: es * ( 1) * n + n es + n whee Between Patients mean squae es ( ) Residual(inteaction) mean squae es Assessing the Reliability of Rating Data- Revised Mach 001

13 Paul Baett age 13 Model 3: Evey atient is ated by each ate, BUT, in contast to Model, we assume the ates ae THE oulation of ates (a two-way, fixed-effects model). In essence, each ate ates all atients on all vaiables. Fo examle, fo a study in which we ate 10 atients on 5 vaiables, we would select say ates in ode to assess inteate eliability. Each ate would make (10*5)50 ating judgements. Howeve, it is assumed that these ae the only two ates who will eve make atings no genealizability assumed to othe ates. The ANOVA fomula is: 3 es + ( n 1)* whee n es es Between Patients mean squae numbe of ates Residual mean squae Let us take an examle dataset fom Owin (1994) Whee we have atings made on the quality of 5 studies on a 3-oint ating scale. Assessing the Reliability of Rating Data- Revised Mach 001

14 Paul Baett age 14 In Statistica, the ANOVA esults fo these data with this setu: Ae: If we assumed that each ating fo each study was given by a unique ate (andom ates), we have Model 1 intaclass W 1 whee now studies being ated ( n 5) + ( n 1)* W W ( *( n 1) ) + ( es*( n 1)*( n 1) ) *( n 1) ((3.9*1) + (0.95* 4*1) ) W * (1)*0.44 n Assessing the Reliability of Rating Data- Revised Mach 001

15 Paul Baett age 15 If we assume that two ates (assumed to be a samle fom some oulation of ates) ovided atings of each of the 5 studies, then we have Model intaclass : + ( n 1) * es n + es * ( ) n es * (1)* ( ) Howeve, if we assumed that the ates wee the only ones we could eve use, essentially the oulation of ates, then we have Model 3 intaclass 3 + ( n es 1)* es (1)*0.95 Ou thee Intaclass s ae: Model Model 0.35 Model The examle on age 5 is actually these data tansfomed into a table suitable fo Kaa whee we assumed the atings wee categoical. The value comuted was: Kaa 0.43 A Peason coelation fo the same data 0.45 Kendall s W (coefficient of Concodance assuming odinal categoies Assessing the Reliability of Rating Data- Revised Mach 001

16 Paul Baett age 16 So, etuning to ou 4 judges data With ANOVA esults as: Ou thee Intaclass s ae: Model Model 0.9 Model The Mean Inte-Judge Peason coelation 0.76 And now look at how the judges have used thei ating scales Assessing the Reliability of Rating Data- Revised Mach 001

17 Paul Baett age 17 SPSS Windows v.9/10 GUI examles fo all thee models It is instuctive to comae the teminology and use of SPSS 9/10 to comute the Intaclass coefficients fo Models 1,, and 3 above. Peole Effects in the SPSS dialogs equate to Patients in my dialog. Item Effects in the SPSS dialogs equate to Rates in my dialog SPSS can diectly comute the Intaclass coelation using the Reliability otion fom the Scale otion on the Analyze main menu. Using the 6 atient x 4 judges dataset as befoe Model 1: Each atient to be ated is ated by a unique ate, with each ate andomly selected fom a lage oulation (a one-way ANOVA andom effects model). Secifically, fo evey atient vaiable o item to be ated, thee is a unique ate. Each ate makes only one ating decision. This model assumes you have a lage ool of ates, who ae andomly assigned to make one ating e atient e vaiable. So, fo a study in which we ate 10 atients on 5 vaiables, we would need 50 ates. Assessing the Reliability of Rating Data- Revised Mach 001

18 Paul Baett age 18 The Reliability analysis sceen looks like Select the 4 judges as items with Model Alha Then click the Statistics button Assessing the Reliability of Rating Data- Revised Mach 001

19 Paul Baett age 19 Then select Intaclass coelation coefficient and One Way Random Model (note that the tye box is geyed out). Then click Continue Then OK to oduce the esults. Assessing the Reliability of Rating Data- Revised Mach 001

20 Paul Baett age 0 Thus Shout and Fleiss Model 1 SPSS One-Way Random model Model : Evey atient is ated by each ate. We assume the ates ae andomly selected fom some oulation of ates (a two-way andom effects model). In essence, each ate ates all atients on all vaiables. This is the default model that coves most ating situations. Fo examle, fo a study in which we ate 10 atients on 5 vaiables, we would need at least ates in ode to assess inteate eliability. Each ate would make (10*5)50 ating judgements. Assessing the Reliability of Rating Data- Revised Mach 001

21 Paul Baett age 1 Do eveything as befoe until Then select Intaclass coelation coefficient and TwoWay Random Model, and Tye Absolute Ageement Continue and OK fo the esults Thus Shout and Fleiss Model SPSS Two-Way Random model with Absolute Ageement Assessing the Reliability of Rating Data- Revised Mach 001

22 Paul Baett age Model 3: Evey atient is ated by each ate, BUT, in contast to Model, we assume the ates ae THE oulation of ates (a two-way, fixed-ate effects model). Each ate ates all atients on all vaiables. Fo examle, fo a study in which we ate 10 atients on 5 vaiables, we would select say ates in ode to assess inteate eliability. Each ate would make (10*5)50 ating judgements. HOWEVER, it is assumed that these ae the only two ates who will eve make atings no genealizability is assumed to othe ates. Do eveything as befoe until Then select Intaclass coelation coefficient and TwoWay Mixed Model, and Tye Consistency Continue and OK fo the esults Thus Shout and Fleiss Model 3 SPSS Two-Way Mixed Model with Tye Consistency Assessing the Reliability of Rating Data- Revised Mach 001

23 Paul Baett age 3 Fom the SPSS 10.0 Base Manual Reliability ICC section Peole Effects in the SPSS dialogs equate to Patients in my dialog. Item Effects in the SPSS dialogs equate to Rates in my dialog * Note*..The between measues vaiance efeed to in the aagah above on Tye is the Between Rates comonent that aeas in the denominato of Model 1 and Model calculations but not Model 3. Assessing the Reliability of Rating Data- Revised Mach 001

24 Paul Baett age 4 What levels of Inteate/Intaclass ae consideed accetable? Fleiss (1981) and Cicchetti and Saow (1981) fom the medical fatenity state: < 0.40 Poo fai good > 0.74 Excellent Howeve, given an alha intenal consistency coefficient of < 0.70 is consideed unaccetable fo alied sychometic eliability indices, and alha is elated to intaclass, then I can only conclude that the medical fatenity ae setting limits fa too low. Realistically, values above about ae accetable fo alied tests. Below this value, and we have eal oblems using ating data. Remembe, the unconditional standad eo of measuement fo a ating scale is conventionally given by: SEM x st (1 xx) whee SEM x the standad eo of measuement fo test scoe X st the standad deviation of the test scoes (fom a nomative gou) the eliability coefficient xx Let s take some eal UK PCL-R data. If ou ate eliability is say 0.45, with a test standad deviation of 7, (a maximum scoe of 40), and a mean scoe of 17, and an obseved scoe of 5, we have a SEM of 5.19, with a 95% confidence inteval of ou tue scoe of between 10 and 30. If we had an inteate eliability of 0.80, with all othe factos the same, then ou SEM is 3.13, with a 95% confidence inteval of ou tue scoe of between 17 and 30. If we had an inteate eliability of 0.90, with all othe factos the same, then ou SEM is.1, with a 95% confidence inteval of ou tue scoe of between 0 and 9. By the way, the tue-scoe confidence intevals ae asymmetic as e Nunnally (1978). See the TRUESCORE ogam (available fom htt:// Fo the alication of confidence intevals in change-scoe analysis. Assessing the Reliability of Rating Data- Revised Mach 001

25 Paul Baett age 5 Key Refeences Cicchetti D.V., and Saow, S.S.(1981) Develoing citeia fo establishing the inteate eliability of secific items in a given inventoy. Ameican Jounal of Mental Deficiency, 86, Fleiss, J.L. (1981) Statistical Methods fo Rates and Pootions, nd. Edition. New Yok: Wiley. Owin, R.G. (1994) Evaluating Coding Decisions. In H. Cooe and L.V. Hedges (eds.) The Handbook of Reseach Synthesis. Russell Sage Foundation, Howell, D.C. (1997) Statistical Methods fo Psychology, 4th Edition. Duxbuy, Shout, P. E., Fleiss, J. L. (1979) Intaclass coelations: Uses in assessing ate eliability. Psychological Bulletin, 86,, Assessing the Reliability of Rating Data- Revised Mach 001