Computing Krippendorff 's Alpha-Reliability

Transcription

1 Uivrsity f Psylvaia ShlarlyCmms partmtal Paprs (ASC) Abrg Shl fr Cmmuiati -5-0 Cmputig Krippdrff 's Alpha-Rliability Klaus Krippdrff Uivrsity f Psylvaia, kkrippdrff@as.up.du Fllw this ad additial wrks at: Part f th Cmmuiati Cmms Rmmdd Citati Krippdrff, K. (0). Cmputig Krippdrff's Alpha-Rliability. Rtrivd frm Pstprit vrsi. This papr is pstd at ShlarlyCmms. Fr mr ifrmati, plas tat libraryrpsitry@pbx.up.du.

2 Cmputig Krippdrff 's Alpha-Rliability Abstrat Krippdrff s alpha (α) is a rliability ffiit dvlpd t masur th agrmt amg bsrvrs, drs, judgs, ratrs, r masurig istrumts drawig distitis amg typially ustruturd phma r assig mputabl valus t thm. α mrgd i tt aalysis but is widly appliabl whrvr tw r mr mthds f gratig data ar applid t th sam st f bjts, uits f aalysis, r itms ad th qusti is hw muh th rsultig data a b trustd t rprst smthig ral. isiplis Cmmuiati Sial ad Bhaviral Sis Cmmts Pstprit vrsi. This wrkig papr is availabl at ShlarlyCmms:

3 Cmputig Krippdrff s Alpha-Rliability Klaus Krippdrff kkrippdrff@as.up.du 0..5 Krippdrff s alpha () is a rliability ffiit dvlpd t masur th agrmt amg bsrvrs, drs, judgs, ratrs, r masurig istrumts drawig distitis amg typially ustruturd phma r assig mputabl valus t thm. mrgd i tt aalysis but is widly appliabl whrvr tw r mr mthds f gratig data ar applid t th sam st f bjts, uits f aalysis, r itms ad th qusti is hw muh th rsultig data a b trustd t rprst smthig ral. s gral frm is: whr is th bsrvd disagrmt amg valus assigd t uits f aalysis: k mtrik k ad is th disagrmt wuld xpt wh th dig f uits is attributabl t ha rathr tha t th prprtis f ths uits: k mtrik ( ) k Th argumts i th tw disagrmt masurs, k,, k ad, rfr t th frquis f valus i iid matris, t b dfid blw. Algbraially, wh bsrvrs agr prftly, bsrvd disagrmt =0 ad =, whih idiats prft rliability. Wh bsrvrs agr as if ha had prdud th rsults, = ad =0, whih idiats th abs f rliability. =0 urs wh bsrvrs ar uabl t distiguish amg uits r assig valus t thm draw radmly frm a lltiv stimat f th ppulati f data. T rly data gratd by ay mthd, ds t b far frm ths tw xtrm ditis, idally =. Fr rliability sidratis, s rag is: 0 Systmati disagrmt Samplig rrrs Ulik thr spializd ffiits, is a gralizati f svral kw rliability idis. It abls rsarhrs t judg a varity f data with th sam rliability stadard. applis t: Ay umbr f bsrvrs, t just tw Ay umbr f atgris, sal valus, r masurs Ay mtri r lvl f masurmt (mial, rdial, itrval, rati, ad mr) Implt r missig data Larg ad small sampl sizs alik, t rquirig a miimum valuats rliability variabl at a tim. It ffrs thr aalytial pssibilitis t prstd hr.

4 Rliability data dupliat th prss f gratig data whs rliability is i qusti. Giv suh data, -rliability a b mputd i fur mputatial stps, graphd blw. Rliability ata Rliability ata Matrix Obsrvd iffr -Agrmt O Variabl Ciids Futi u iu k k kju mu Ths fur mputatial stps will b dfid ad illustratd with fur kids f data f irasig mplxity: A. Biary r dihtmus data, tw bsrvrs, missig data Pag B. Nmial data, tw bsrvrs, missig data Pag 3 C. Nmial data, ay umbr f bsrvrs, missig data Pag 4. All mtris, ay umbr f bsrvrs, missig data Pag 5 Fially, E. A gral mputatial frm is prstd, bypassig iid matris Pag 9 A. Biary r dihtmus data, tw bsrvrs, missig data Cstrut a rliability data matrix; hr, a bsrvrs-by-n uits matrix, taiig N valus, ad k: Uits: u N Obsrvrs: i: i i iu in j: j j ju jn Fr xampl, wh tw bsrvrs judg N=0 uits, th -by-0 data matrix tais 0 valus: Itms judgd: Mg: Ow: Tabulat iids withi uits. Ciid matris aut fr all valus taid i a rliability data matrix. Thy diffr frm th familiar tigy matris, whih aut fr uits i tw dimsis, t valus. Th imprta f this diffr bms appart i C. It a -by- iid matrix, uits ar trd twi, as -k pairs ad as k- pairs. I th xampl, uit is trd as a 0- pair f valus ad as a -0 pair f valus. Uit is trd as tw - pairs f valus, t.: Valus: Numbr f Valus: 0 =N 4 6 0

5 Ardigly, 00 rprsts th t 0-0 pairs withi uits 4, 5, 7, 8, ad 0. 0 rprsts th fur 0- pairs i uits, 3, 6, ad 9, ad 0 rprsts th fur -0 pairs i th fur sam uits. rprsts th tw - pairs fud ly i uit. 0 =4 is th umbr f 0s i th rliability data matrix, =6 is th umbr f s, ad =N=0 is th ttal umbr f valus paird. Fr ths biary data, mismathig iids ur i tw lls 0, 0 f qual frquy, 4. skip Cmput -rliability (mst simpl frm): biary ( 0 ) 0 4 I th xampl: biary (0 ) B. Nmial data, tw bsrvrs, missig data Cstrut a rliability data matrix just as i A abv. Fr a -by- xampl: Itms judgd: B: a a b b d d d a Grry: b a b b b d d d Tabulat iids withi uits. Th gral frm f a iid matrix ad with frquis frm th abv xampl trd ar: Valus:. k.. a b d. k.. a b k.. = k k d k.. = k k Whr k = u Numbr f -k pairs i uit u spifially: ab = a-b pair i uit ba = b-a pair i uit aa = a-a pairs i uit bb = 4 = b-b pairs i uit 3 + b-b pairs i uit 4 ad s frth. a =4 is th umbr f as b =6 is th umbr f bs ad s frth. =4 is th ttal umbr f valus fr tw bsrvrs: = N skip Cmput -rliability (mst simpl frm): mial A A A ( ) ( ) ( ) ( ) 3

6 Whri A is th prt f bsrvd maths i uits u ad A is th prt f maths btaiabl by ha. Th mputatial frm furthr simplifis th dd mputatis. I th xampl: mial (4 )( ) [4(4 ) 6(6 ) 6(6 ) 6(6 ) ( )] (4 ) [4(4 ) 6(6 ) 6(6 ) 6(6 ) ( )] C. Nmial data, ay umbr f bsrvrs, missig data Cstrut a rliability data matrix just as i A ad i B abv, but fr m bsrvrs: Uits u:... u..... N Obsrvrs:... u..... N i i i... iu..... in j j j... ju..... jn..... m m m... mu..... mn Numbr f bsrvrs valuig u: m m... m u..... m N Wh data ar missig, this matrix tais lss tha mn tris ad m u is variabl. Fr xampl, a 4 bsrvrs-by- uits rliability data matrix: Uits u: Obsrvr A: Obsrvr B: Obsrvr C: Obsrvr : Numbr m u f valus i uit u: Nt that 7 ut f th 48 pssibl valus i this matrix ar missig. m u varis frm t 4. Tabulat iids withi uits. Th iid matrix appars as i B: Valus:. k k.. 7 4/3 /3 / /3 0 4/3 / /3 4/3 8 /3. 0. k.. = k k 4 /3 /3 / k.. = k k Numbr f - k pairsi uit u Ulik i th tw-bsrvr as i B: k u mu Nt that ah uit tais m u (m u ) iids. A iid matrix auts fr all pairs f valus fud i u. Uit tais 3(3 )=6 pairs f mathig s. It tributs 4

7 6/(3-)=3 t th ll, fr ah valu. Uit tais 4(4-)= pairs, 6 mathig - pairs, 3 mismathig -3 pairs, ad 3 mismathig 3- pairs. It adds 6/(4-)= t, 3/(4- )= t 3, t 3, ad 4 t th ttal, thus fully autig fr its 4 valus. Uit 6 tais 4(4-)= pairs f mismathig valus, ah adds /(4-)=/3 t a diffrt ll. Th l valu 3 i uit affrds mpariss, (-)=0 pairs ad ds t add t this aut. Thus, th margis f iid matris d t rprst all valus that ur i a rliability data matrix, ly ths that a b paird withi uits, hr =40 pairabl valus vr all uits. Skip Cmput -rliability just as i B mial A A A ( ) ( ) ( ) ( ) I th xampl: (40 )( ) [9(9 ) 3(3 ) 0(0 ) 5(5 ) 3(3 )] mial (40 ) [9(9 ) 3(3 ) 0(0 ) 5(5 ) 3(3 )]. Ay mtri, ay umbr f bsrvrs, missig data Cstrut a rliability data matrix just as i C Tabulat iids withi uits just as i C Isrt th diffr futi mtri k that is apprpriat t th mtri f th giv data it th tw disagrmts ad dfid i th bgiig f this dumt. Nt that auts fr diffrt mtris r lvls f masurmt by wighig th bsrvd ad xptd iids by th squard diffr btw th iidig valus. iffrs a b xprssd as mathmatial futis ad i th frm f a tabl. Th lattr maks thir rlativ magituds traspart. Itrval ad rati mtri diffrs ar futis f th valus big paird. Ordial diffrs dpd thir frquis f usig valus. Ad mial diffrs ar addd hr t graliz stp. Nmial mtri diffrs Tw valus ithr math, r thy d t: Nmial atgris, ams: a b d f a 0 b 0 0 iff = k mial k 0 iff k d 0 0 f 0 5

8 Ordial mtri diffrs Valus hav th maig f raks ad diffrs btw raks dpd hw may raks thy ar apart frm ah thr. Fr xampl, with frquis frm data i C (ad uusd rak addd t shw that it ds t mattr): rdial k g k g g Raks: st d 3 rd 4 th 5 th 6 th k st st = 9 d d = 3 3 rd rd = 0 4 th th = 5 5 th th = 0 6 th th = 3 st,3 rd=.5 4 th,6 th=4 Ordial mtri diffrs may b stadardizd: 0 rdial k by: rdial k g k g g max mi whr max is th largst ad mi th smallst rak amg all raks usd. Stadardizati ds t afft, hwvr. k Itrval mtri diffrs Valus diffr algbraially: itrval k Itrval valus: k Itrval mtri diffrs may b stadardizd as wll: 0 itrval k by: itrvall k k max whr max is th largst ad mi is th smallst valu urrig i th data. This stadardizati ds t afft ithr. mi 6

9 Rati mtri diffrs Algbrai diffrs btw tw valus ar xprssd rlativ t a abslut zr pit. Thy ar prprtial t th magitud f thir valus: rati k Rati valus: k k () () () 3 () 4 () 5 0 () () 3 () 4 () () () 3 () () () () Cirular mtri diffrs Shrtst diffrs btw ay tw valus a irular sal with arbitrary dpits but a fixd irumfr U = th umbr f qual itrvals a irl. Cirular valus: With th si futi xprssd i dgrs: k irular k si 80 U With th si futi xprssd i radia: irular k k si U Biplar mtri diffrs Algbrai diffrs ar xprssd rlativ t th tw dpits, mi ad max, f th sal. Nar th tr, a biplar mtri bhavs lik a itrval mtri ad ar th pls it bhavs lik a rati mtri. plar k ( k ) ( k )( mi Biplar valus: max k )

10 rdial Cmput -rliability (th mputatially mst ffiit frm): k mtri k k mtri k k mtri Nt that th sums i this frm umrat ly f th tw symmtrial ff-diagal triagls f a iid matrix. Its tris, k as wll as th prduts k ar wightd by a apprpriatly hs diffr futi mtri k. Cmputatis ar illustratd with th umrial data i C abv, itrprtd as rdial, itrval ad rati data rsptivly. Zr frquis appar as 0 i th list f multipliatis: k With data i C as rdial data: rdial k k rdial k k rdialk k I th Exampl: itrval k With data i C as itrval data: itrval k itrval k itrvalk k I th xampl: (40 ) k rati k With data i C as rati data: rati I th xampl: k rati k k rati k k

11 E. A gral mputatial frm, bypassig iid matris: Start frm a rliability data matrix as i C abv: Uits u:... u..... N Obsrvrs:... u..... N i i i... iu..... in j j j... ju..... jn..... m m m... mu..... mn Wh data ar missig, this matrix will tai fwr tha mn valus. Fr th 4 bsrvrs-by- uits xampl f rliability data usd i C ad abv: Uits u: Obsrvrs: A: B: C: : Nt that Out f th mn=4=48 pssibl valus i this matrix, 7 ar missig. Th valu 3, assigd by bsrvr B t th th uit at b paird with xistig valus i that uit, at tribut t bsrvd agrmts r disagrmts, drps ut wh strutig a iid matrix, ad has t b igrd. Thus, this matrix tais a ttal f..= 40 pairabl valus. Istad f ad, umrat th valus fud i uits ad rat a valus-by-uits matrix: Uits:... u Valus:... u u k k k... uk k..... Ttals:..... u Whr u = th umbr f valus assigd t uit u. u m bsrvrs. uk by aalgy u. = u = th umbr f valus assigd t uit u. = u u. u = th umbr f pairabl valus urrig i th rliability data (mittig all uits with l r valus: u.).. = u u. u. = th ttal umbr f all pairabl valus i th rliability data (mittig all uits with l r valus: u.);.. mn 9

12 it rval Fr th abv xampl Uits u : Valus, k : = = = = =. 5 Ttals u. : =.. Nt that th margial sum. 3 f pairabl valus 3 mits th ly l valu i uit,.=. Cmput with th apprpriat mtri diffr futi as dfid i abv: mtri (.. ). u k u k.. u k mtri uk mtri If th abv xampl sists f mial data (prft agrmt s as 0 disagrmt): mi al k If th abv xampl sists f itrval data (prft agrmt s as 0 disagrmt): k Mst mputatis f a b prfrmd with SPSS r SAS mars writt by Adrw Hays. Availabl at G t SPSS ad SAS Mars th t KALPHA. Rfrs: Klaus Krippdrff (004). Ctt Aalysis, a Itrduti t Its Mthdlgy, d Editi. Thusad Oaks, CA: Sag Publiatis spially Chaptr, pags -56. Klaus Krippdrff (004). Rliability i Ctt Aalysis: Sm mm Misptis ad Rmmdatis. Huma Cmmuiati Rsarh 30,3: Last assd 0..5 Adrw F. Hays, ad Klaus Krippdrff (007). Aswrig th all fr a stadard rliability masur fr dig data. Cmmuiati Mthds ad Masurs : Last assd