Variance Estimation of the Survey-Weighted Kappa Measure of Agreement

Transcription

1 NSDUH Reliability Study (2006) Cohen s kappa Variance Estimation Acknowledg e ments Variance Estimation of the Survey-Weighted Kappa Measure of Agreement Moshe Feder 1 1 Genomics and Statistical Genetics Statistics and Epidemiology RTI International mfeder@rti.org Joint Statistical Meetings 2006 M. Feder JSM August 2006, Seattle

2 Structure of the Talk The NSDUH Reliability Study (2006)

3 Structure of the Talk The NSDUH Reliability Study (2006) Cohen s kappa (κ) Measure of Reliability

4 Structure of the Talk The NSDUH Reliability Study (2006) Cohen s kappa (κ) Measure of Reliability Taylor-linearization Variance Estimation

5 Structure of the Talk The NSDUH Reliability Study (2006) Cohen s kappa (κ) Measure of Reliability Taylor-linearization Variance Estimation Simulation Results

6 NSDUH Reliability Study (2006) The National Survey of Drug Use and Health (NSDUH, n 70, 000 annually) is currently conducting a study (n 3, 000) to assess the reliability of responses.

7 NSDUH Reliability Study (2006) The National Survey of Drug Use and Health (NSDUH, n 70, 000 annually) is currently conducting a study (n 3, 000) to assess the reliability of responses. An interview/re-interview method is employed where individuals are interviewed on two occasions, T1 and T2. The reliability of the responses is assessed by comparing the T1 and T2 responses.

8 NSDUH Reliability Study (2006) The National Survey of Drug Use and Health (NSDUH, n 70, 000 annually) is currently conducting a study (n 3, 000) to assess the reliability of responses. An interview/re-interview method is employed where individuals are interviewed on two occasions, T1 and T2. The reliability of the responses is assessed by comparing the T1 and T2 responses. To measure the reliability of categorical responses, Cohen s kappa (κ) index of inter-rater reliability is used (Cohen, 1960).

9 NSDUH Reliability Study (2006) The National Survey of Drug Use and Health (NSDUH, n 70, 000 annually) is currently conducting a study (n 3, 000) to assess the reliability of responses. An interview/re-interview method is employed where individuals are interviewed on two occasions, T1 and T2. The reliability of the responses is assessed by comparing the T1 and T2 responses. To measure the reliability of categorical responses, Cohen s kappa (κ) index of inter-rater reliability is used (Cohen, 1960). This measure is the statistic most-often used to assess inter-rater reliability of categorical variables.

10 The Set-up Consider a binary response (Y = 0, 1).

11 The Set-up Consider a binary response (Y = 0, 1). The four possible combinations of T1 and T2 responses divide the population into four cells: Y T 1 = 0 Y T 1 = 1 Y T 2 = 0 p 00 p 01 p 0+ Y T 2 = 1 p 10 p 11 p 1+ p +0 p +1 p ++ = 1

12 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11

13 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1

14 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1 Cohen s kappa corrects for chance agreement and is defined as κ = p e p c = (p 00 + p 11 ) (p 0+ p +0 + p 1+ p +1 ) 1 p c 1 (p 0+ p +0 + p 1+ p +1 )

15 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1 Cohen s kappa corrects for chance agreement and is defined as κ = p e p c = (p 00 + p 11 ) (p 0+ p +0 + p 1+ p +1 ) 1 p c 1 (p 0+ p +0 + p 1+ p +1 ) 1 κ 1

16 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1 Cohen s kappa corrects for chance agreement and is defined as κ = p e p c = (p 00 + p 11 ) (p 0+ p +0 + p 1+ p +1 ) 1 p c 1 (p 0+ p +0 + p 1+ p +1 ) 1 κ 1 p 00 + p 11 = 1 = κ = 1 (perfect agreement)

17 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1 Cohen s kappa corrects for chance agreement and is defined as κ = p e p c = (p 00 + p 11 ) (p 0+ p +0 + p 1+ p +1 ) 1 p c 1 (p 0+ p +0 + p 1+ p +1 ) 1 κ 1 p 00 + p 11 = 1 = κ = 1 (perfect agreement) p 00 + p 11 = 0 = κ = 1 (complete disagreement)

18 Cohen s kappa First measure that comes to mind is percent agreement: p e = p 00 + p 11 However agreement could be by chance alone: p c = p 0+ p +0 + p 1+ p +1 Cohen s kappa corrects for chance agreement and is defined as κ = p e p c = (p 00 + p 11 ) (p 0+ p +0 + p 1+ p +1 ) 1 p c 1 (p 0+ p +0 + p 1+ p +1 ) The survey-weighted estimate is ˆκ = (ˆp 00 + ˆp 11 ) (ˆp 0+ ˆp +0 + ˆp 1+ ˆp +1 ) 1 (ˆp 0+ ˆp +0 + ˆp 1+ ˆp +1 ) 1 κ 1 p 00 + p 11 = 1 = κ = 1 (perfect agreement) p 00 + p 11 = 0 = κ = 1 (complete disagreement)

19 Variance Estimation The common variance estimation approach Fleiss et al. (1969) asymptotic variance formula (see also Agresti, 2002)

20 Variance Estimation The common variance estimation approach Fleiss et al. (1969) asymptotic variance formula (see also Agresti, 2002) Assumes an independent sample, with equal probabilities of inclusion

21 Variance Estimation The common variance estimation approach Fleiss et al. (1969) asymptotic variance formula (see also Agresti, 2002) Assumes an independent sample, with equal probabilities of inclusion The NSDUH sample design is complex

22 Variance Estimation The common variance estimation approach Fleiss et al. (1969) asymptotic variance formula (see also Agresti, 2002) Assumes an independent sample, with equal probabilities of inclusion The NSDUH sample design is complex This may have significant effect on the variance estimates

23 Variance Estimation The common variance estimation approach Fleiss et al. (1969) asymptotic variance formula (see also Agresti, 2002) Assumes an independent sample, with equal probabilities of inclusion The NSDUH sample design is complex This may have significant effect on the variance estimates We present a Taylor linearization derivation, along with numerical results

24 Taylor Linearization Denote a = ˆp 00, b = ˆp 01 and c = ˆp 10. (Note: ˆp 11 = 1 (a + b + c).)

25 Taylor Linearization Denote a = ˆp 00, b = ˆp 01 and c = ˆp 10. (Note: ˆp 11 = 1 (a + b + c).) Also, denote U = 1 ˆκ and F = U a, G = U b, H = U (More details at c.

26 Taylor Linearization Denote a = ˆp 00, b = ˆp 01 and c = ˆp 10. (Note: ˆp 11 = 1 (a + b + c).) Also, denote U = 1 ˆκ and F = U a, G = U b, H = U (More details at First-order Taylor linearization: U F a + G b + H c. c. Let r be the population mean r = Fp 00 + Gp 01 + Hp 10 and its estimate ˆr = F ˆp 00 + G ˆp 01 + Hˆp 10

27 Taylor Linearization Denote a = ˆp 00, b = ˆp 01 and c = ˆp 10. (Note: ˆp 11 = 1 (a + b + c).) Also, denote U = 1 ˆκ and F = U a, G = U b, H = U (More details at First-order Taylor linearization: U F a + G b + H c. c. Let r be the population mean r = Fp 00 + Gp 01 + Hp 10 and its estimate ˆr = F ˆp 00 + G ˆp 01 + Hˆp 10 Then, Var(ˆκ) Var(ˆr).

28 PROCEDURE Calculate a = ˆp 00, b = ˆp 01 and c = ˆp 10. Calculate F = U a, G = U b, H = U c.

29 PROCEDURE Calculate a = ˆp 00, b = ˆp 01 and c = ˆp 10. Calculate F = U a, G = U b, H = U c. Calculate a new variable x i = F I [T1=0,T2=0] + G I [T1=0,T2=1] + H I [T1=1,T2=0] for every i s

30 PROCEDURE Calculate a = ˆp 00, b = ˆp 01 and c = ˆp 10. Calculate F = U a, G = U b, H = U c. Calculate a new variable x i = F I [T1=0,T2=0] + G I [T1=0,T2=1] + H I [T1=1,T2=0] for every i s Calculate the estimated variance of the survey-weighted mean of x i. That s our estimate of the variance of ˆκ.

31 Simulation Set-up Subjects within clusters are more alike than ones in different clusters.

32 Simulation Set-up Subjects within clusters are more alike than ones in different clusters. Assume the following model x h,i,j = φ φ ε h,i+ 1 φ ε h,i,j ε h,i N(0, 1) ε h,i,j N(0, 1)

33 Simulation Set-up y Subjects within clusters are more alike than ones in different clusters. Assume the following model x h,i,j = φ φ ε h,i+ 1 φ ε h,i,j ε h,i N(0, 1) ε h,i,j N(0, 1) and then let (Y T 1, Y T 2 ) be defined by rho x phi

34 Simulation Results φ = 1 Taylor Linearization L=5 n.sim=1000 phi=1 n.psu=20 sizepsu=5 Fleiss L=5 n.sim=1000 phi=1 n.psu=20 sizepsu=5 Frequency Frequency

35 Simulation Results φ = 0 Taylor Linearization L=5 n.sim=1000 phi=0 n.psu=20 sizepsu=5 Fleiss L=5 n.sim=1000 phi=0 n.psu=20 sizepsu=5 Frequency Frequency

36 Simulation Results bias(phi) phi relative bias Taylor Linearization Fleiss

37 Conclusions The standard (Fleiss et al.) ASE can be seriously biased (negatively).

38 Conclusions The standard (Fleiss et al.) ASE can be seriously biased (negatively). When the clustering is unrelated to the kappa measure, the ASE is valid and more efficient.

39 Acknowledgments This research was developed with the support of the Substance Abuse and Mental Health Services Administration, Office of Applied Studies, under Contract no The views expressed in this presentation do not necessarily reflect the official policies of the Department of Health and Human Services; nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government