A-Kappa: A measure of Agreement among Multiple Raters

Transcription

1 Jounal of Data Scence (04), A-Kappa: A measue of Ageement among Multple Rates Shva Gautam Beth Isael Deaconess Medcal Cente, Havad Medcal School Abstact: Medcal data and bomedcal studes ae often mbalanced wth a majoty of obsevatons comng fom healthy o nomal subjects. In the pesence of such mbalances, ageement among multple ates based on Fless Kappa (FK) poduces countentutve esults. Smulatons suggest that the degee of FK s msepesentaton of the obseved ageement may be dectly elated to the degee of mbalance n the data. We popose a new method fo evaluatng ageement among multple ates that s not affected by mbalances, A-Kappa (AK). Pefomance of AK and FK s compaed by smulatng vaous degees of mbalance and llustate the use of the poposed method wth eal data. The poposed ndex of ageement may povde some nsght by elatng ts magntude to a pobablty scale. Exstng ndces ae ntepeted abtaly. Ths new method not only povdes a measue of oveall ageement but also povdes an ageement ndex on an ndvdual tem. Computaton of both AK and FK may futhe shed lght nto the data and be useful n the ntepetaton and pesentng the esults. Key wods: Fless Kappa, A-Kappa. Intoducton. The poblem Bomedcal, socal, behavoal and othe studes outnely nclude statstcal evaluatons of ageement among multple ates o condtons (Fensten et al, 985) and Fless Kappa (FK) s wdely used to evaluate ageements (Fless, 98). Ths wo stems fom a eal poblem whch aose when examnng ageement among adologsts who wee evaluatng mammogaphc beast mages. Table shows the esults of an expement whee 0 adologsts at Bgham and Women s Hosptal, Boston, Massachusetts, ndependently evewed 0 beast MRIs obtaned between Septembe 004 and Apl 008. Inceased beast densty may be assocated wth nceased beast cance s, and theefoe, classfcaton of mammogaphy epots s mpotant both scentfcally and clncally. The BI- RADS algothm developed by Amecan College of Radology was used to classfy beast Coespondng autho.

2 698 A-Kappa: A measue of Ageement among Multple Rates composton nto fou categoes. One of the categoes ndcates whethe o not an mage exhbts a Fatty (< 5% glandula) patten. Table collapses the thee non-fatty categoes nto one and denotes them as a : A fatty mage s denoted as a 0. Each lne n the table dsplays a sequence of s and 0 s whch shows the scong patten fo each of the 0 ates. Table : Classfcaton of 0 mages by 0 ates nto on-fatty (= ) vesus Fatty (= 0) beast composton categoes Yes =, o = 0 umbe of Fequency Pecent Yes (= ) Total All 0 ates classfed 85 (83.33%) of the 0 mages nto non-fatty categoy ndcatng complete ageement among the ates on these mages. An addtonal 0 mages wee classfed as non-fatty by 9 (90%) of the ates so that moe than 93% of the mages wee classfed nto the non-fatty categoy by at least 90% of the ates. Despte such a lage degee of consensus seen among the ates, Fless Kappa (FK) fo these data s only 0.9 (95% CI: 0.090, 0.48) ndcatng a poo o no ageement. An altenate ageement ndex developed n ths pape called A-Kappa (AK) yelds a value of (95% CI: 0.889, 0.93) fo the same data ndcatng a hgh ageement among ates. Thus, the wdely used method of evaluatng ageement ndex FK yelds a counte ntutve esult n ths nstance. It wll be shown late that a data set consstng of a lage numbe of postve (o negatve) events may yeld a poo FK despte hgh obseved ageement. In the context of Table, all the ates classfy 83.33% mages nto the postve (scoe ) categoy. Thee s not a sngle mage whee all the ates classfed an mage nto the negatve (scoe 0) categoy. Smlaly, thee ae 5 mages wth eght s, but thee s not a sngle mage wth eght 0s. Ths type of data set s efeed to as unbalanced o asymmetcal data set n ths pape. Ths ssue s futhe addessed

3 Shva Gautam 699 n Secton.3 and then shown that the poposed measue AK s not nfluenced by the mbalance n a data set. Medcal data ae pone to mbalance due to hgh o low pevalence of a gven chaactestcs o dsease (L, Lu and Hu, 00). Fo example, n a sceenng settng t s lely to have ae moe healthy ndvduals, whle n a specalty cae cente thee may be a lage numbe of subjects wth a dsease. Hence, fo many bomedcal data Fless Kappa (FK) s lely to msepesent o completely mss the ageement pesent n a data set. A-Kappa (AK) developed n ths pape could be an altenate tool to evaluate ageement n such data sets as t s not nfluenced by the asymmety o mbalance. In two ates case the phenomenon of hgh obseved ageement wth low Cohen s Kappa often found wth asymmetcal o unbalanced data has been studed and some emedes have been suggested (Fensten et al, 990; Ccchett et al, 990; Lantz et al, 990). These emedes pmaly suggest epotng some altenate ndces along wth Cohen s Kappa. Lantz and ebenzahl (990) mantan that Kappa alone has lttle ntepetve value and ecommend epotng altenatve ndces along wth Kappa. In ths atcle we show that Fless Kappa, the most wdely used ageement ndex among multple ates, shows the hgh ageement low Kappa behavo smla to that of Cohen s Kappa. A new method fo evaluatng ageement among multple ates, A-Kappa (AK), s poposed n ths atcle. Ths method s not affected by the type of mbalances descbed above and s able to captue the obseved ageement. Futhemoe, n the case of balanced data set t educes to FK. It s poposed that both FK and AK be epoted wth the esults of data analyss.. An Altenate Measue of Ageement. Poposed ageement ndex: A-Kappa ( ndependent ates classfes the th mage (,,..., ) Suppose that each of the ) nto one of two categoes by assgnng a scoe of o 0 to ndcate pesence o absence of a dsease, espectvely. When all the ates agee on a gven mage, we wll have ethe all s o all 0s. Smlaly, when the sequence conssts of an equal numbes of s and 0s (50% of each) t s consdeed a stuaton of complete absence of ageement. Let a denote the numbe of s n the sequence fo the th mage. In othe wods, a ates out of total ates classfy the th mage nto the dsease categoy and emanng - non-dsease categoy. We assume that each mage s ead by all of the eades. The poposed measue of ageement A-Kappa (AK) s defned as a nto the AK [( a ) ] [ ( )] (.)

4 700 A-Kappa: A measue of Ageement among Multple Rates The devatonal agument behnd the defnton AK n equaton (.) s povded n Secton 3. whee moe than two categoes ae addessed. Moe specfcally when =, equaton (3.5) educes to equaton (.). The followng esults follow fom the defnton of AK gven by equaton (.). Poofs ae gven n the Appendx. Poposton. When thee ae two ates, AK educes to Maxwell s Random Eo (RE) coeffcent (Maxwell, 977): AK P 0 RE, (.) whee P 0 s the popoton of mages on whch the two ates agee. ote that, RE was ognally poposed as an altenate measue of ageement between two ates to addess the ssue of hgh ageement and low Cohen s appa. Poposton. If s the popoton of pas of ates who agee and s the popoton of pas who dsagee on the th mage, then w AK ( w v ) / Poposton 3. AK fo multple ates s the aveage of AK fo all possble pas of ates. Let denote AK obtaned fom the th and jth ates. Smlaly, let denote the AK j popotons of mages on whch both of these ates agee. Then 0, j j v P,j 0 (.3) AK P / C(,), whee C (,) [ ( )] (.4). Relatonshp between A-Kappa (AK) and Fless Kappa (FK) In the secton, elatonshp between AK s developed and shown that fo balanced o symmetcal data these ndces ae dentcal. Concept of balanced o symmetcal data was ntoduced at the begnnng of ths pape. It wll be evsted hee to establsh equalty between AK and FK. Poposton 4. AK 4pq( FK) whee s the popoton of one postve classfcatons (o popotons of s) n the ente data set, and q p. Poof: Recall that,, and a (,,..., ) denote the total numbe of mages (o objects to be evaluated), total numbe of ates and numbe of ates who classfy the th mage as postve (o who assgn ), espectvely. Let p a / denote the popoton ates who classfy the th mage nto the postve categoy. Smlaly, let p a /( ) a / denote the oveall popoton of postve esponses, and q p ( a ). Then the Fless Kappa (FK) s defned as (Fless, 98) p /

5 Shva Gautam 70 Multplyng both sdes of t by FK= a ( a ) / ( pq 4 pq (.5) ) a ( a ) / ( ) 4 pqfk 4 pq 4 = 4 pq 4 a ( a ) / ( ) (.6) Upon smplfcaton, 4 a ( a ) / ( ) = ((a ) ) / ( ) the defnton of AK) Theefoe, fom (.6).. Symmetcal o balanced data =AK(fom AK 4pq( FK) (.7) In the pesent context, symmety may be defned n seveal ways. At the basc level, f 50 % of obsevatons (e.g. mages) n a data set come fom one populaton (say healthy) and emanng 50% come fom a second populaton (e.g. dsease) then such a data set could be consdeed a balanced data set. Even wth expeenced ates, t s lely that thee wll be nstances of msclassfcatons, and theefoe, t s unlely that an mage wll be assgned ethe o 0 by all ates. But wth lage data sets one can expect that they wll be evenly msclassfed. In othe wods, one would expect that fo evey msclassfcaton nto postve categoy thee wll be a msclassfcaton nto the negatve categoy. Imbalance n data set may be due to desgn (e.g. moe postve mages n the collected data) o pevalence (e.g. ae dsease). Fo example, f a data set contans consdeably moe postve (o negatve) mages, then the data set wll be mbalanced. Hence, a data set n whch thee s an mage wth a gven numbe of s fo each mage wth the same numbe of zeos wll be consdeed a symmetcal o balanced data. Lac of ths gves se to an unbalanced data set. Accodng to ths defnton data set pesented n Table s an unbalanced data set. Poposton 5. AK FK. When the data set s balanced then AK= FK fo balanced o symmetcal data set. Poof: In the case of balanced data (see defnton above) the ente data set be can be pesented n tems of / pas of mage such that fo a pa consstng of and ' th mages, we have p a and p a so that, p p '. Theefoe, n a balanced / ' / data set, p a /( ) a / /() /. Hence, p / q. Theefoe, fom (.7), n balanced (o symmetcal) data set, AK=FK. ext, AK< FK 4pq( FK) FK fom equaton (.7) 4pq FK( 4pq) th FK. But n fact FK. Theefoe, AK FK.

6 70 A-Kappa: A measue of Ageement among Multple Rates So, Fless Kappa (FK) yelds a smalle value than AK n an unbalanced o asymmetcal data set. It s woth notng that f the numbe of s s equal to the numbe of 0s n data set then also AK= FK whethe o not the data set meets the defnton of symmety. If s and 0 s ae assgned completely andomly then both AK and FK wll be equal to zeo ndcatng lac of ageement..3 Smulatons We conducted seveal smulatons to gan some nsght nto A-Kappa s pefomance and to compae t wth Fless Kappa, and ad n ts ntepetaton. We based ou smulatons on 0 ates, categoes and 0,000 mages. Let βdenote the popoton of dseased mages. Let π denote the pobablty of coectly classfyng an mage by a ate and s assumed to be the same fo π = each ate. Each ate s assumed to assgn a scoe Table : Compason of A-Kappa and Fless Kappa usng 0,000 mages (samples) Pecent mages Fless Kappa wth gven β 9+ ates agee A-Kappa 0% π = Pobablty of classfyng an mage coectly β = Popoton of mages fom nomal subjects of to a dseased mage and 0 to a healthy mage (fom nomal subjects) accodng to a bnomal pobablty = P[X = dseased mage] = P[X =0 healthy mage]. One can vsualze the ente data set composed of two subsets one consstng of healthy mages only and dsease mage. Smulated data sets wee geneated wth = 0.50, 0.60, 0.70, 0.80, 0.90, 0.95 and 0.99, and β = 0.50, 0.70, 0.90 and 0.95 whee β = popoton of mages fom the nomal (healthy) subjects. It s not necessaly tue tah pobablty = P[X = dseased mage] = P[X =0 healthy mage] fo each ate, but even ths smple assumpton can be used to geneate mbalanced data. Ou goal hee s to smply demonstate the vulneablty of FK and obustness of AK n cetan types of data. All smulatons and subsequent computatons wee caed out usng SAS 9.. ote that, ths s not the unque way to geneate columns (o ows) of 0 and n ode to show that AK may fal to eflect the obseved ageement n unbalanced data.

7 Shva Gautam Pefomance of A-Kappa vs Fless KappaTable pesents esults of smulatons descbed above. The fst column of Table shows values of π, the second column shows pecent to mages on whch 9 (90%) o moe ate agee. Ths s taen as a measue of cude o obseved ageement A quc glance at Table shows that when π = 0.5 both AK and FK ndcate absence of ageement. Ths s a stuaton when ates classfy mages andomly. Howeve, when s dffeent fom 0.5 then FK vaes wth the popoton of postve mages whle AK emans the same fo a gven π. When the popoton of samples of dseased mages s equal to the samples of healthy mages then both AK and FK yeld the same value fo all. On the othe hand, when a majoty of mages ae fom dseased patents (o fom healthy patents) then FK s futhe fom the obseved (o cude) ageement than AK. In such stuatons AK captues the obseved ageement bette than FK. Fo example, when π = 0.90, then obseved ageement = (at least 90% of ates agee on 73.5% mages) and A-Kappa = 0.646, but Fless Kappa could be as low as 0.03 when almost all mages ae ethe postve o negatve and could be as hgh as (value of AK) when 50% mages ae postve. The above esults showng dffeence between AK and FK ae also depcted n Fgue. When the undelyng popoton of postve mages s oughly 0.5 ( = 0.5), the lnes fo AK and FK ae the same fo all values of π. That s why lnes fo AK and FK50 (.e. FK values when 50% mages ae postve) ae not dstngushable n Fg.. Also, AK and FK ae the same (and nea zeo) espectve the popoton of dseased mages when ates assgn scoes of 0 o to an mage andomly (.e. when π = 0.50). Fgue

8 704 A-Kappa: A measue of Ageement among Multple Rates.5 An Intepetaton of A-Kappa (AK) Lands and Koch (977) povded an ntepetaton of Kappa statstc. Howeve, those ntepetatons ae consdeed abtay. Except fo Kappa = and 0 mplyng pefect and chance ageement, espectvely, othe value fal to convey the degee of ageement n tems of an ntepetable scale. The followng dscusson may help shed some lght nto ntepetaton of AK. Assume that the same mage s evaluated by a set of ates by assgnng 0 o to the mage fo the pesence and absence of a dsease. Also, assume that some tme has elapsed between two evaluatons. Poposton 6. Let a and pˆ a denote the numbe and popoton of ates who assgned dung the th evaluaton. If t s assumed that the sequence of 0 and ae geneated by an undelyng Benoull dstbuton wth a paamete, then p E pˆ ] [ E (.8) [ AK ] ( p ) (See appendx fo a poof of ths esult). Equaton (.8) may shed lght nto the ntepetaton of an AK value as p ( AK ) /. (.9) Thus fo an AK value, we can say one would have obtaned the same AK f each ate would classfy a dseased mage coectly wth the pobablty gven by equaton (.9). Howeve, t s not necessaly tue that the data n hand was geneated wth ths pobablty. Equaton (.8) ndcates that when 90% of the ates assgn o 90% ates assgn 0 (say, to all the mages), then AK s about 0.64). In the case of two ates the popoton of mages on whch two ates agee s expected to be = = 0.8 so that AK = (0.8) - = 0.64 (see equaton.). On the othe hand, f data yelds AK = 0.64, then one would obtan the same AK fom data set whee pobablty of coectly classfyng an mage by each ate s o such ntepetaton exsts fo FK whee the ntepetaton s completely abtay (Lands et al 977). 3. Multple Rates and Multple Categoes Although the man focus of the pape s evaluaton of ageement among multple of ates (o stuatons) on two possble classfcaton (o categoes), esults pesented n pevous sectons ae befly extended to multple categoy stuaton n the followng sectons. Some addtonal nsghts on AK ae also pesented. 3. Devaton of A-Kappa (AK) fo multple categoes and multple ates

9 Shva Gautam 705 Assume that each ate classfes an mage nto one of the categoes. Let numbe of ates who classfed the th (,,..., ) j a j denote the mage nto the jth ( j,,..., ) categoy so that a j fo each mage. In case of complete ageement on the th mage, all ates wll classfy the mage nto the same categoy. In the case of complete lac of ageement the th mage wll be categozed nto each categoy by an equal numbe of ates. Theefoe n ths case, / ates ae expected to classfy such an mage nto each of the categoes. One can thn of ageement as the dscepancy o dstance fom complete dsageement. Ths dscepancy may then be expessed as a j j / a / (3.) Ths quantty can be escaled by dvdng t by ts maxmum possble value so that the dstance between obseved data and the state of complete dsageement les between 0 and. The maxmum value of the expesson gven n (3.) s gven by j j / ( ) / Theefoe, the escaled dstance fo the th mage s a j / /[( ( )] G (3.) j Hence, the mean of ates. Ths s gven by G acoss mages can be consdeed as the cude ageement among G G / aj /( ( )) /( ) (3.3) j Howeve, some of ths ageement may be due to chance. Opnons dffe egadng the defnton of chance nduced ageement. Fo example, Maxwell uses 0.5 as a chance nduced ageement and Cohen uses the magnal pobabltes of the table unde consdeaton. Anothe way to quantfy the ageement due to chance mght be to estmate the ageement expected n a sample that comes fom a populaton lacng ageement among ates. In tems of the notaton used above, eo due to chance may be gven by the expected value of G gven that thee s an absence of ageement n the populaton. Poposton 7. The expected value of G gven that thee s an absence of ageement n the populaton s gven by (See appendx fo a poof) E [ G ] / (3.4)

10 706 A-Kappa: A measue of Ageement among Multple Rates A-Kappa (AK) s the measue G adjusted fo ageement due to chance and escaled to yeld the maxmum possble value of s, gven by : AK ( G / ) /( / ) ( G ) /( ) (3.5) The functonal foms of AK fo two ates and multple ates seem dffeent, but n fact they ae the same as shown below. A-Kappa fo two categoes was developed fst and then t was shown as an extenson of Maxwell s Random Eo (RE). Snce the eo due to chance was aleady mbedded n Maxwell RE, no eo adjustment was dscussed. Poposton 8. When =, then ( G ) /( ) [(a ) ] / ( whch s the same as equaton. (A poof s gven n the appendx). ) (3.6) 3.. Ageement on ndvdual tem (mage) ote that, A-Kappa poposed n ths atcle s the aveage of ( G ) ( ) acoss the obsevatons (mages), whee G s defned by equaton (3.). Theefoe, ths quantty could be consdeed as a measue of ageement among ates on the th obsevaton (mage). Let AK ( G ) /( ), then AK AK / (3.7) Ths chaactestc of AK s smla to the ageement ndex poposed by O'Connell and Dobson (984), but AK s much smple to compute and uses a dffeent stategy to estmate chance nduced ageement. One advantage of obtanng ageement on an ndvdual mage (obsevaton) s that the nvestgatos could dentfy and nvestgate mages wth hgh dsageement. Ths could especally be useful when tanng novce ates. Equaton (3.6) also ponts that AK fom two o moe data sets could be easly combned to yeld the oveall AK fom the combned data set as shown below. Let a data set of sample sze ) be and ( pattoned nto data sets havng sample szes. If AK, AK and ae AK fom the fst set, second set and combned set of data, espectvely, then AK c = ( AK )/( + ) + ( AK )/( + ). Thus, AK fom a combned data set s the weghted aveage of AKs fom the component data sets. Ths s not necessaly tue fo FK. 3. Asymptotc dstbuton of A-Kappa (AK) Followng the notatons of the pevous sectons, suppose ates classfy each of the mages nto one of the categoes. Suppose that the numbe of atngs a a,..., a AK c,

11 Shva Gautam 707 coespondng to the th mage (obsevaton) have a multnomal dstbuton wth pobabltes π,,..., ) (. Let a a... a, and let p (,,..., ) p p p denote the sample popoton (popotons of ates), whee p a. j j / Poposton 9. The Asymptotc vaance of AK s gven by 3 V ( AK ) 4 p ( p ) /[ ( ) ( ) ]. j j (A poof s gven n the appendx). j 3.3 Smulatons fo Multple Categoes Smulatons esults fom eale secton have shown that n the case of two categoes, FK and AK ae equvalent when thee s absence of ageement o when the data ae symmetcal. Othewse FK may fal to eflect the hgh degee of obseved ageement. Hee, we pesent a few smulatons usng multple ates and multple categoes. These smulatons show that as wth two categoes, FK may fal to eflect a hgh obseved ageement n case of multple categoes. We geneated 0,000 tems (mages) and assumed that each of the 0 ates classfed each mage nto one of fve categoes. Let denote the pobablty wth whch a ate assgns an mage nto the th (,,3,4,5 ) j andomly assgned nto one of these categoes,.e., when p categoy. When each mage s p = 0. fo all, then AK= FK = In ths case, both ndces tuly eflect the absence of ageement among the ates. ext, suppose that 0000 mages of categoy 4 ae evaluated by 0 ates, and each ate can coectly classfy the mages wth a pobablty 0.9. Fo a smple example, let p = p = p 3 = p 5 = 0.05 and p , then FK = 0.03 and AK = Unde ths smulated scenao, at least 8 ates (80% o moe) ae found to classfy 9,433 (94.33%) mages nto the 4 th categoy. Theefoe, FK fals to eflect hgh degee of ageement among the ates. ext, we smulated that whee wth 50% of the mages 4 and 50% wee of categoy wee of categoy, and the ates can coectly classfy the mage wth pobablty of 0.9 nto these two categoes (wth emanng pobablty equally dstbuted ove emanng categoes). In ths case, the smulated data showed that at least 8 ates (80% o moe ates) classfed 4,486 mages n categoy and 4,699 mages n categoy 4 so that at least 80% of the ates ageed on 9,84 (9.84%) mages. FK fo ths data tuned out to be whle AK = Thus ths balancng n the data bought FK value almost to the level of AK, whle AK emaned the same. Hence, n the case of moe than two categoes and multple ates FK may fal to eflect the hgh degee of obseved ageement n asymmetc data, whle AK may not be nfluenced by such asymmety

12 708 A-Kappa: A measue of Ageement among Multple Rates 3.4 Real Example (multple categoes) Consde the study descbed n the ntoducton secton. Ten ates wee ased to classfy the beast composton of 0 mages nto the fou categoes: the beast s almost entely fat (< 5% glandula), SFD: scatteed fboglandula denstes (appoxmately 5-50% glandula), HD: the beast tssue s heteogeneously dense (appoxmately 5% 75% glandula), ED: the beast tssue s extemely dense ( > 76% glandula). Table 3 shows AK and FK among multple ates and multple categoes. ote that, FK ndcates that the ates have poo ageement on whethe the mages ae fatty o not. AK on the othe hand shows thee s an excellent ageement among the ates. Most of the ates classfy mages nto non-fatty categoes. In Table 3, both AK and FK ae when classfyng an mage nto heteogeneously dense (HD). Table 3: Ageement ates on Ageement Index Classfcaton A-Kappa Fless Kappa Fatty SFD HD ED Oveall composton classfcatons among beast Ths ndcates that f the data wee e-aanged nto HD vesus non-hd categoy, then t ndcates pehaps the numbes HD and non-hd mages ae smla. In all othe stuatons we have AK > FK ndcatng lac of such symmety. Howeve, the asymmety s not substantal except fo Categoy (Fatty vs non-fatty). In concluson, f only FK was used we mght have been msnfomed about the ageement among the ates 4. Dscusson

13 Shva Gautam 709 Cohen s Kappa (CK) s used outnely to evaluate ageement between two ates o two condtons, but has been ctczed fo beng smply a functon of pevalence, and countentutve by seveal nvestgatos. Usng smulatons, t s shown n ths atcle that Fless Kappa (FK) a measue of ageement among multple ates nhets some of these shotcomngs of CK. A new and smple method fo evaluatng ageement A-Kappa (AK) among multple ates s poposed. Ths method educes to Maxwell s Random Eo (RE) poposed to addess the hgh ageement low appa paadox n case of two ates. In ths atcle t s shown, by smulatons, that Fless appa (FK) may also yeld low appa although thee s a hgh degee of ageement among the ates. Ths s especally tue n the case of mbalanced data whee one class of tems s elatvely less than the othe. AK, poposed n ths pape, may be used as an altenate o an addtonal ndex n the case of multple ates. The poposed measue does not have the seemngly paadoxcal chaactestc of FK. Computng both AK and FK may povde addtonal nsght. The dffeence between the two values may ndcate whethe the data ae domnated by one nd of classfcaton of mage. As ndcated by smulatons, FK concdes wth AK when popotons of postve and negatve mage ae the same. A small FK may not eally be an ndcaton of low ageement, whle a small AK s ndcaton of low ageement. Also, AKs fom two data sets can be easly combned to yeld the AK fom the combned data set. We ecommend calculatng both AK and FK. In the case of two categoes, A-Kappa may have a meanngful ntepetaton n a moe famla scale of pobablty as dscussed n ths pape. Exstng ntepetaton of Kappa values ae consdeed somewhat abtay. Computaton of both AK and FK may futhe shed lght nto the data and be useful n the ntepetaton and pesentng the esults. Ths s smla to ecommendaton by of computng both maxmum and mnmum Kappa n two ates two categoes stuaton. Though not the focus of the pape, thee exsts a body lteatue wth model based appoaches to evaluate ageement (Agest, 99 and 00; Tanne et al 985). Smla to the appa-le ndces, most of the model based methods have also dealt wth the stuaton of two ates, and the numbe of paametes to be estmated nceases exponentally wth numbe of ates ceatng computatonal challenges. Estmatng equaton appoaches ae also poposed to model ageement n data wth multple ates havng bnay and multple categoes (Wllamson et al, 000, Kla et al, 000). AK wll also be examned fom epeated measues vewpont n a futue study. Howeve, nvestgatos especally n bomedcal studes stll outnely use CK and FK to evaluate ageement. Ths pape hghlghts some stuatons whee these methods may fal to captue the ageement and popose an altenatve method whch educes to exstng methods poposed n two ates stuatons. Cuent lmtatons of AK nclude ts nablty to ncopoate ates chaactestcs. Howeve, ths s also the case wth FK and CK. Evaluaton AK wth epeated measues (o heachcal) appoach s beng exploed. Ths wll allow us to adjust fo confoundng factos. Despte such lmtatons ts smplcty and ntutve natue, t povdes some nsghts nto the natue of ageement and the data set tself. It s vey smple to calculate.

14 70 A-Kappa: A measue of Ageement among Multple Rates Appendx A Poposton. In the case of two ates, AK P 0 RE. Poof: Fom equaton., n the case of two categoes and ates, AK [( a ) ] [ ( )] otng that when = then, [( ) ]/( ) a ( ( a ) a Also, when thee ae only two ates, then taes values 0, o. So, the tem when the two ates dsagee. Theefoe, when =, whee P 0 Poposton. If AK [(a ( a ) ) a (numbe of ates who agee on the th mage) = when the two ates agee, and ]/[ ( 0 )] = P AE s the popoton of mages on whch both ates agee. w s the popoton of pas of ates who agee and pas who dsagee on the th mage, then AK w v / Poof: Assume that assgnng a scoe of ) and emanng - a 0). Let C ( x,) x( x ) /. Then both membes of v ( a ) = 0 s the popoton of ates out of classfy the th mage nto dsease categoy (by a C a nto the non-dsease categoy (assgnng a scoe of pas of ates wll assgn and both membes of a C pas wll assgn 0 to the th mage. Smlaly, one membe of a a ) pas wll assgn whle the othe membe wll assgn 0 to the th mage. Theefoe, ( w [ C( a,) C( a,)]/ C(,) and v [ a a ]/ C(,) Hence dffeence n popoton of mages on whch thee s pa-wse ageement and dsageement on the th mage s gven by w v [ C( a,) C( a,) a ( a )] / C(,) [ a ( a ) ( a )( a ) a ( a )]/[ ( )] [(a ) ]/[ ( )]

15 Shva Gautam 7 Thus, w v / [(a ) ]/[ ( )] AK Poposton 3. AK fo multple ates s the aveage of RE fo all possble pas of ates. Let denote RE j denote Maxwell s Random Eo fom the th and jth ates. Smlaly, let the popotons of mages on whch both ates agee. Then P,j 0 AK ( 0, j j j j P ) / C(,) RE / C(,), whee C (,) ( ) /. Poof: Assume that eadngs of mages by ates ae pesented n ows and columns. Let the columns be denoted by X X,... X. Also assume (epesentng the hth ate), taes values ethe o 0. ext consde ( ) / X h vaables V, V3,... V such that, V st X s X t ( X s)( X t ), t>s ote that, V st = when X s and coeffcent fom the sth and tth ate s then ( V ) whee acoss mages (o obsevatons), and can be expessed as V X t We need to show that ( V ) / C(,) Snce ts V ts st both tae o both tae 0 othewse V 0. The AK st V st st st / st s the aveage of V. V st AK [(a ) ]/[ ( )] st ( Vst ) / ( Vst ) / [ C( a,) C( a )]/ ts ts a ( a ) ( a )( a ) / We have, ( V ) / C(,) [4 V ]/[ ( )] ts st 4 /[ ( )] [ a ( a ts st ) ( a )( a )]/ taen [(a ) ]/[ ( )] [{(a ) }/{ ( ( )} ]

16 7 A-Kappa: A measue of Ageement among Multple Rates [(a ) Poposton 6. ]/[ ( E [ AK ] ( p ) )] AK Poof: ote that, by defnton, AK [(a ) ]/[ ( )]. Also, [(a ) ]/( ) = [ /( )][( a / ) /( ) = [ /( )](4a 4a ) /( ) Unde the assumpton that each ate wll assgn a scoe of wth pobablty p, a ~B(, p). Theefoe,, Va a ) E[ a ] { E[ a ]} so that E[ a ] p( p) ( p). E a p ( Usng ths nfomaton, we have E [ AK ] ( p ). Poposton 7. The expected value of G gven that thee s an absence of ageement n the populaton s gven by E [ G ] / Poof: Fom equaton (3.), G [ aj ]/[ ( )] /( ) ( aj ) /[ ( )]. So that ( ) G ( a ) j /( / ) ote that, unde the assumpton of andom classfcaton of mages by the ates, th mage wll be classfed nto each categoy by ates. Theefoe, ( ) G s dstbuted as wth ( ) degees of feedom, and ts expected value wll be ( ). Hence, / E[ ( ) G ] whch mples that E[ G ] /. Theefoe, E[ G ] E[ G ]/ / Poposton 8. When =, ( G ) /( ) [(a. ) ] / ( ), j. j whee G G / a /( ( )) /( )

17 Shva Gautam 73 Poof: ote that, fom equaton (3.), G a j / /[( ( )]. j When =, then G ( a a) /. otng that a a, and doppng the second subscpt, we have So that, G [ a ( a ) ]/ (a ) /. ( G ) /( ) [(a ) ]/ ( ). Theefoe, ( G ) /( ) [(a Poposton 9. Asymptotc vaance of AK s gven by ) ]/ ( ). 3 V ( AK ) 4 [ pj ( p j j j ) ]/[ ( ) ( ) ] Poof: Recall that ates classfy each of the mages nto one of the categoes. Suppose that the numbe of atngs a, a,..., a coespondng to the th mage (obsevaton) have a multnomal dstbuton wth pobabltes π,,..., ). Let = a + a + + a, and let p p, p,..., p ) denote the sample popoton (popotons of ates), whee p a j j / (. Then t follows that ( d π ) [ 0, dag( π ) π π], ( p whee dag ( π ) s a dagonal matx wth matx wth elements of π on the man dagonal. Unde the assumpton that mages ae ndependent, the vaance covaance matx fo the ente sample wll be a bloc dagonal matx wth each bloc beng of the fom expessed as n (3.6). The followng esults can be used to estmate asymptotc vaance of AK. Fom equaton (3.), fo the th mage, G f p ( aj ) /[ ( p p ) /( ) f ( p p π ( π ) /( ) ( ] /( ) ( ) p j ) /( ) /( )

18 74 A-Kappa: A measue of Ageement among Multple Rates Theefoe, fom the Delta method, ( G f ( π d )) [0,{4 /( ) }{ π(dag( π ) π π) π }] Hence the vaance of G denoted as V ( G ) {4 /( ) }{ π(dag( π ) π π) } π 3 O V G ) 4 /( ) [ j ( ( )] j Theefoe, the vaance (asymptotc) A-Kappa statstc fo the th mage s gven by V ( AK ) ( G ) /( ) 3 V ( G )[ /( ) 4 [ j ( j ) ]/[( ) ( ) ] j j j Assumng ndependent mages and notng that that the oveall A-Kappa fo the gven data set s the aveage of AK acoss the mages (obsevatons) we have, p Snce paametes V ( AK ) V ( AK ) / V ( ( AK ) / j 3 4 [ pj ( p a n the above equaton. j j / Refeences j j j ) ]/[ ( ) ( ) ] ae geneally unnown, they ae eplaced by the sample estmates [] Agest, A. (99). Modelng pattens of ageement and dsageement. Statstcal Methods n Medcal Reseach, 0-8. [] Agest, A. (00). Categoy Data Analyss (Second Edton). Wley: ew Yo. [3] Ccchett, D.V. and Fensten, A.R. (990). Hgh ageement but low appa: II. Resolvng the paadoxes. Jounal Clncal Epdemology 43, [4] Fensten, A.R. (985). A bblogaphy of publcatons on obseve vaablty. J Chon Ds 38, [5] Fensten, A.R. and Ccchett, D.V. (990). Hgh ageement but low appa: I. The poblems of two paadoxes. Jounal Clncal Epdemology 43,

19 Shva Gautam 75 [6] Fless, J.L. (98) Statstcal methods fo ates and popotons. Wley: ew Yo, pp [7] Kla,.. Lpstz, S.R. and Ibahm, J. (000). Estmatng equaton appoach fo modelng appa. Bometcal Jounal 4, [8] Lands, J.R. and Koch, G.G. (977). The measuement of obseve ageement fo categocal data. Bometcs 33, [9] Lanz, C.A. and ebenzahl, E. (996) Behavo and ntepetaton of the appa statstc: esoluton of the two paadoxes. Jounal Clncal Epdemology 49, [0] L, D.C. Lu,C.W. and Hu, S.C. (00). A leanng method fo the class mbalance poblem wth medcal data sets. Comput Bol Med 40, [] Maxwell, A.E. (977). Coeffcents of ageement between obseves and the ntepetaton. The Btsh Jounal of Psychaty 30, [] O'Connell, D. L. and Dobson, A.J. (984). Geneal Obseve-Ageement Measues on Indvdual Subjects and Goups of Subjects. Bometcs 40, [3] Tanne, M.A. and Young, M.A. (985). Modelng ageement among ates. Jounal of the Amecan statstcal Assocaton 80, [4] Wllamson, J.M., Manatunga, A.K. and Lptstz, S.R.(000). Modelng appa fo measung dependent categocal ageement data. Bostatstcs,9-0. Receved Octobe 4, 03; accepted Apl 6, 04. Shva Gautam Beth Isael Deaconess Medcal Cente Havad Medcal School 330 Boolne Avenue, Boston, MA 05 sgautam@bdmc.havad.edu

20 76 A-Kappa: A measue of Ageement among Multple Rates