Task 1: Repetition - Precision & Recall

Transcription

1 SS 208 Exrcis 9 - July 28, 208 Task : Rptition - Prcision & Rcall Considr an information nd for which thr ar 5 rlvant documnts in th collction. Givn is th following list of rlvant ( R ) and non-rlvant ( N ) documnts rturnd for a qury (th lftmost is th top rankd sarch rsult): R N R N N N N N R R Task.: Comput th prcision, rcall and F scor of th ranking. P rcision = 0 = 0. R call = 5 = 0.8 F = 2P R P +R = = 8 5 = 0.53 Task.2: Comput th prcision@, rcall@, prcision@5 and rcall@5. = =, = 5 = 0.2 = 2 5 = 0., = 2 5 = 0. Task.3: Crat th prcision/rcall graph. Rank Rcall Prcision

2 SS 208 Exrcis 9 - July 28, 208 Task 2: Rptition - Avrag Prcision Considr an information nd for which thr ar rlvant documnts in th collction. Contrast two systms run on this collction. Thir top 0 rsults ar judgd for rlvanc as follows: Systm R N R N N N N N R R Systm 2 N R N N R R R N N N Task 2.: Which of th two systm rturns a bttr ranking according to Prcision@8? ) = 2 8 = ) = 8 = 0.5 According to this masur, Systm rtrivs th bttr ranking. Task 2.2: Which of th two systm rturns a bttr ranking according to thir avrag prcision? rlvant documnt Prcision (Systm ) Prcision (Systm 2) A P (Systm ) = = 3 5 = 0.6 A P (Systm 2) = = According to Prcision@8, Systm 2 outpfrmorm Systm. Task 3: Cohn s Kappa 2

3 SS 208 Exrcis 9 - July 28, 208 Two ratrs A and B agr in a classification task as follows: B Ys No A Ys 20 5 No 0 5 Task 3.: Comput Cohn s Kappa for this agrmnt p a = = p = = 0.5 pa p p = = 0. Task 3.2: What do you xpct if you swap 20 and 5? Comput Cohn s Kappa for this. As th agrmnt dcrass, a dcras of Cohn s Kappa valu is xpctd. Nw computation with swappd valus: 5+5 p a = = p = = 0.5 pa p p = = 0.2 3

4 SS 208 Exrcis 9 - July 28, 208 Task : Intr-ratr agrmnt: Fliss Kappa Tn ratrs ( n = 0 ) assign fiv subjcts ( N = 5 ) to a total of thr catgoris ( k = 3 ). Th catgoris ar prsntd in th columns, whil th subjcts ar prsntd in th rows. Each cll lists th numbr of ratrs who assignd th indicatd (row) subjct to th indicatd (column) catgory. Catgory j Subjct id 2 3 P i = = = = p j 25 = = = 0.3 Task.: Comput th p j valus and P. p = i= n ij = 50 ( ) = 9 25 = P = p 2 j = 0.36² + 0.3² + 0.3² = j= P i Task.2: Comput th valus and P. P = 0 (0 ) 3 j= (( ) 3 ) = 90 ((² + 2 ² + 7²)² 0) = 22 5 = 0.8 n 2 j P = 5 5 P i = 5 ( ) = 5 9 = 0.5 i= Task.3: Comput Fliss Kappa valu. P P = P Task.: What dos this Fliss Kappa valu tll us about th ratr agrmnt? According to th intrprtation tabl by Landis and Koch, this Kappa valu indicatd fair agrmnt btwn th ratrs. Howvr, this tabl should not b accptd univrsally.

5 SS 208 Exrcis 9 - July 28, 208 Task 5: Graphical Modl for Truth Infrnc Fiv workrs assignd on out of fiv catgoris to tn diffrnt s. Apply th graphical modl for truth infrnc to dtrmin th labl of ach. 2 3 Workr Workr Workr 3 Workr Workr catgory 5

6 SS 208 Exrcis 9 - July 28, 208 Scors at R Workr Workr 2 Workr 3 Workr Workr = 3 Vots at R Labls at R Scors at R2 Workr 0.9 Workr Workr 3 0. Workr 0.5 Workr In R2, Workr agrs with th labl of R in 9 out of 0 cass =.9 Vots at R Labls at R Scors at R3 Workr Workr 2 0. Workr Workr 0. Workr = 2 Vots at R Labls at R3 ( convrgd )

7 SS 208 Exrcis 9 - July 28, 208 Appndix Prcision, Rcall & F P rcision = #(rtrivd s) #(rlvant s rtrivd) R call = #(rlvant s) F = 2P R P +R #(rlvant s rtrivd) Avrag Prcision If th st of rlvant documnts for an information nd is { d,..., d mj } and R jk is th st of rankd rtrival rsults from th top rsult until you gt to documnt, thn th avrag prcision is computd m j as A P = m rcision(r ). j Cohn s Kappa P jk k= d k A B Ys No Ys a b No c d p a = p = a+d a+b+c+d a+b a+b+c+d p p a p a+c a+b+c+d + c+d a+b+c+d b+d a+b+c+d Fliss Kappa n numbr of ratrs, N numbr of subjcts, k numbr of catgoris P P Fliss Kappa P p j = N Nn n ij th proportion of all assignmnts which wr to th j -th catgory i= P i = n(n )(( k nij) ) 2 n th xtnt to which ratrs agr for th i -th subjct j= P = N N P i i= k P = p2 j j= Truth Infrnc s cor(usr) = Σ c catgoris,i s (vot(usr, c, i ) stimatdlabl(c, i )) c catgoris,i s stimatdvot(c, i ) = Σ usr usrs scor(usr) vot(usr, c, i ) c catgoris,i s stimatdlabl(c, i ) =, if cx catgorism AX(stimatdscor(cx, i )) = stimatdscor(c, i ), 0 othrwis Nots & Sourcs - Schütz, Hinrich, Christophr D. Manning, and Prabhakar Raghavan. Introduction to Information Rtrival. Vol. 39. Cambridg Univrsity Prss, https ://nlp.stanford.du/ir-book/information-rtrival-book.html _kappa - s_kappa 7