McNemar s Test and Introduction to ANOVA

Transcription

1 McNemar Tet ad Itroductio to ANOVA Recall from the lat lecture o oparametric method we ued the eample of reductio of forced vital capacity. FVC Reduc (ml) Subj Placebo Drug At firt bluh thi look like a two-ample problem. However the FVC reductio uder placebo ad uder drug are both oberved for each ubject. We have 6 pair of meauremet. Thu by lookig at the differece placebo miu drug we reduce it to a oe-ample problem. FVC Reduc (ml) Subj Placebo Drug Differece Author: Blume Greevy BIOS 3 Page of 5

2 McNemar Tet ad Itroductio to ANOVA Aumig you had paired outcome X ad Y like with the FVC reductio ad that the mea were roughly ormally ditributed would it be wrog to ue a two-ample t-tet o the differece i mea? Uequal variace t-tet: T * X Y S X X SY m Y If X ad Y are ot idepedet Var( X Y ) X m Y Cov( X Y ) What will happe with the two-ample t-tet whe X ad Y are ot idepedet? Author: Blume Greevy BIOS 3 Page of 5

3 McNemar Tet ad Itroductio to ANOVA A Illutrative Eample i R: et.eed(7) # et to ay fied radom umber for reproducible reult optio(cipe=0) # prevet cietific otatio c < # chooe correlatio v = 4*c^ / ( - c^) # variace eeded for deired correlatio PoCorPvalample <- c() PoCorPvalPaired <- c() NegCorPvalample <- c() NegCorPvalPaired <- c() im <- 0^4 for(i i :im){ <- rorm(00 0 qrt(v)) y <- + rorm(00 0 ) z <- - + rorm(00 0 ) PoCorPvalample <- c(pocorpvalample t.tet(y)$p.value) PoCorPvalPaired <- c(pocorpvalpaired t.tet(-y)$p.value) NegCorPvalample <- c(negcorpvalample t.tet(z)$p.value) NegCorPvalPaired <- c(negcorpvalpaired t.tet(-z)$p.value) } plot( y) um( PoCorPvalample<0.05 )/im um( PoCorPvalPaired<0.05 )/im plot( z) um( NegCorPvalample<0.05 )/im um( NegCorPvalPaired<0.05 )/im Type I Error Rate Correlatio two-ample uequal variace ample tet paired oe Author: Blume Greevy BIOS 3 Page 3 of 5

4 McNemar Tet ad Itroductio to ANOVA McNemar tet for Paired Dichotomou data McNemar tet i aalogou to the paired t-tet ecept that i applied to oly dichotomou data. We leared that whe we wat to compare the mea i two group of paired obervatio the tadard t-tet are ot valid. Thi wa becaue the variace of the differece i mea wa derived uder the aumptio that the two group are idepedet ad o it igore the covariacecorrelatio term (which ca icreae or decreae that term). So i thi cae both the hypothei tet ad cofidece iterval that are baed o thi aumptio will ot provide the correct awer. Side ote: By correct I mea that tet will reject more or le ofte that the pecified type I error. I tatitic thi i what we mea by correct: that the tet ha the propertie we deiged it to. To olve thi problem we reduced the data from two dimeio dow to oe by ubtractig the obervatio withi a pair ad aalyzig the differece with a oe ample t-tet ad cofidece iterval o the differece. Thi approach avoid the correlatio problem becaue the etimated variace o the differece accout for the correlatio. Author: Blume Greevy BIOS 3 Page 4 of 5

5 McNemar Tet ad Itroductio to ANOVA With dichotomou data the problem i more complicated but a imilar approach eit. It i called McNemar tet for paired biomial data. The baic idea i till to aalyze the differece (becaue the differece i mea or i thi cae proportio i till the average of the differece) ad the ue the variace of the differece itead of tryig to etimate the correlatio. But rather tha do thi directly o the differece we arrage everythig i a table. The catch i that we ue a differet tet-tatitic for thi table. Setup: N pair of (zero or oe) repoe (X i Y i ) i= N X i i a Beroulli (θ ) radom variable Y i i a Beroulli (θ ) radom variable Here θ i the probability of ucce o the firt obervatio withi the pair ad the θ i the probability of ucce o the ecod obervatio withi the pair. A importat ote i that the probability of ucce θ doe ot deped o i. Ad we wat to tet if H: θ = θ i.e. i the probability of ucce differet o the firt trial tha o the ecod. Author: Blume Greevy BIOS 3 Page 5 of 5

6 McNemar Tet ad Itroductio to ANOVA Eample: A recet creeig tudy of 4958 wome (the DMIST trial) compared two differet imagig modalitie (mammogram digital mammogram) for detectig breat cacer. Both modalitie were preformed o each woma. If both eam were egative the wome were followed for oe year to be ure cacer wa ot preet ad if either eam wa poitive a biopy wa preformed. (Thi wa deiged ad aalyzed at Brow Uiverity Ceter for Statitical Sciece; Referece i Piao et. al. NJEM 005) Out of the 4958 erolled wome oly 4555 were eligible completed all eam ad had pathology or follow-up iformatio (called referece tadard iformatio). I tudie of diagotic tet oe hould alway eparate the true poitive cae ad the true egative cae a determied by referece tadard iformatio ad aalyze them eparately. I thi cae there were 334 wome with breat cacer ad 4 wome without breat cacer. Author: Blume Greevy BIOS 3 Page 6 of 5

7 McNemar Tet ad Itroductio to ANOVA Below i the data from the 334 wome with breat cacer. Scree film mammogram detected 36 wome digital mammogram detected 38 but oly 84 of thee wome were detected by both modalitie. The data are diplayed i thi table: Data o poitive cae from DMIST trial. Scree Mammogram Detected Mied Digital Detected Mammo Mied We wat to ee if the proportio of detected wome differ betwee the modalitie. It might be temptig to ue a imply Chi-quare tet for thi cotigecy table but that would be wrog becaue that tet i built to eamie the aumptio that the row ad colum are idepedet which they are clearly ot (becaue the ame wome are i both group). So the tet we ue i Mcemar tet. Author: Blume Greevy BIOS 3 Page 7 of 5

8 McNemar Tet ad Itroductio to ANOVA McNemar tet: Time Succe Failure Time Succe a b a+b Failure c d c+d a+c b+d N Let θ = P( Succe Time ) = (a+b)/n θ = P( Succe Time ) = (a+c)/n The ull hypothei i H 0 : θ = θ which implie that E[ (a+b)/n ]= E[ (a+c)/n ] or E[b] = E[c]. Aother form of the ull hypothei i H 0 : θ / θ = or H 0 : θ /(- θ )/ θ /(- θ )= (Odd ratio of detectio for Digital over cree i oe) b c with df= b c Notice that (θ - θ ) = ( (b-c)/n ) o we ee that the differece i proportio i ideed the top of the tet tatitic keepig the coectio betwee the Chi-quare tet ad the Z-tet for differece i proportio. So McNemar tet i thi form i a approimate tet that require large ample to be valid. Author: Blume Greevy BIOS 3 Page 8 of 5

9 McNemar Tet ad Itroductio to ANOVA Back to our DMIST eample comparig eitivity: Digital Mammo Scree Mammogram Detected Mied Detected Mied Here i the Stata output for our data:. mcci Cotrol Cae Epoed Uepoed Total Epoed Uepoed Total McNemar' chi() = 0.04 Prob > chi = Eact McNemar igificace probability = 0.97 Proportio with factor Cae Cotrol [95% Cof. Iterval] differece ratio rel. diff odd ratio Author: Blume Greevy BIOS 3 Page 9 of 5

10 McNemar Tet ad Itroductio to ANOVA Three commet: ) Some book like Pagao ad Gauvreau ugget a lightly differet verio of thi tet to help whe ome cell cout are mall. Cotiuity correctio: b c with df= b c Opiio differ o whe to ue thi but typically it i ued whe ay cell cout are le tha 5. Same reaoig applie for the cotiuity corrected verio of the Chi-quare tet. ) There i o eact aalytical formula for the variace of the differece of paired proportio. So the eaiet way to get a cofidece iterval i to imply perform a hypothei tet for every ull hypothei (differece i zero differece i etc.) ad ue the et of ull hypothee that DO NOT reject a your cofidece iterval. Thi procedure i kow a ivertig the hypothei tet ad thi i how Stata get the cofidece iterval for the differece i paired proportio. 3) You hould igore the proportio for cae ad cotrol that Stata provide. What you really are comparig here are 38/334 veru 36/334 which ha a relative rik of 38/36=.04 ad rik differece of /334 = (See output). Author: Blume Greevy BIOS 3 Page 0 of 5

11 McNemar Tet ad Itroductio to ANOVA We ca do a imilar aalyi for the egative cae: Data o egative cae from DMIST trial. Scree Mammogram Detected Mied Digital Detected Mammo Mied mcci Cotrol Cae Epoed Uepoed Total Epoed Uepoed Total McNemar' chi() = 7.63 Prob > chi = Eact McNemar igificace probability = Proportio with factor Cae Cotrol [95% Cof. Iterval] differece ratio rel. diff odd ratio Notice that the p-value i igificat (le tha 0.05) but the etimated differece i tiy ad of o cliical coequece (pecificity i the ame). Author: Blume Greevy BIOS 3 Page of 5

12 McNemar Tet ad Itroductio to ANOVA Baed o thee reult both creeig modalitie appear to have the ame overall performace. Iteretigly a imilar aalyi wa doe o the ubgroup of wome le tha 50 year old: 7 of thee wome had breat cacer ad 403 did ot. The data for youg poitive wome are a follow:. mcci Cotrol Cae Epoed Uepoed Total Epoed Uepoed Total McNemar' chi() = 4.7 Prob > chi = 0.04 Eact McNemar igificace probability = Proportio with factor Cae.486 Cotrol.347 [95% Cof. Iterval] differece ratio rel. diff odd ratio (eact) So it appear the eitivity of thee tet are differet by about 4%. Notice that the eact ad approimate p-value are differet ad that the cofidece iterval for the odd ratio iclude but the cofidece iterval for the relative rik ( ratio ) doe ot. Dicu! Author: Blume Greevy BIOS 3 Page of 5

13 McNemar Tet ad Itroductio to ANOVA Lookig at the proportio of poitive cae that were detected compare the eitivity. To eamie pecificity we look at the group of egative cae. The data for youg egative wome are a follow:. mcci Cotrol Cae Epoed Uepoed Total Epoed Uepoed Total McNemar' chi() = 3.44 Prob > chi = Eact McNemar igificace probability = Proportio with factor Cae Cotrol [95% Cof. Iterval] differece ratio rel. diff odd ratio So it appear that the two tet differ lightly with repect to pecificity although the differece i mall ad likely uiteretig. Digital mammogram appear to have better eitivity (ad the ame pecificity) for wome uder 50 ad would therefore be a (lightly) better creeig tet. Author: Blume Greevy BIOS 3 Page 3 of 5

14 McNemar Tet ad Itroductio to ANOVA Eact p-value for McNemar tet Becaue McNemar tet i baed o the iformatio oly i the dicordat pair (the b ad c off diagoal cell) the calculatio of eact p-value i quite imple. If the ull hypothei i true the it implie that the b ad c cell hould be equal very iformally H 0 : b=c. Thee cell are jut cout of people o the uderlyig ditributio ha to be Biomial where ½ of the cout hould be i each cell. That i uder the ull hypothei b~biomial(b+c0.5) So a eact p-value i P(b>b ob =b+c ad θ=/). Eample: i the ubgroup of wome le tha 50 year old 7 of thee wome had breat cacer. Here b=7 ad c=7 o P( X 7 4 / ) two ided p - value i7k (.0396) mcci (See page 0) McNemar' chi() = 4.7 Prob > chi = 0.04 Eact McNemar igificace probability = Author: Blume Greevy BIOS 3 Page 4 of 5

15 McNemar Tet ad Itroductio to ANOVA Oe Way Aalyi of Variace (ANOVA) We kow how to tet the equality of two ormal mea: Two ample havig ad. Model (aumptio): Idepedet obervatio from two ormal ditributio with mea ad commo variace. To tet the hypothei H 0 : (mea are equal) v. H A : we ue the tet tatitic p + We reject H 0 if the oberved value of the tet tatitic eceed the critical value foud i t-table uig + - degree of freedom. Eample: For =0 ad =8 the two-ided 5% critical value i.0. Author: Blume Greevy BIOS 3 Page 5 of 5

16 McNemar Tet ad Itroductio to ANOVA Thi i the ame thig if we checked to ee if the quare of the oberved tet tatitic i bigger tha (.0) = Mathematically we have ( p ) + = ( ) + p ( ) = ( ) + ( ( ) +( ) + ) >( t / ) where the ample mea without a ubcript i the overall mea obtaied from the combied ample. Thi i ometime called the grad mea. Thi quared form of the tet tatitic i importat becaue it how u how to geeralize the tet for more tha two group. Author: Blume Greevy BIOS 3 Page 6 of 5

17 McNemar Tet ad Itroductio to ANOVA Author: Blume Greevy BIOS 3 Page 7 of 5 For three group ad we have the ame aumptio: Idepedet obervatio from three ormal ditributio with mea 3 ad commo variace. To tet the hypothei 3 0 = = : H (all mea equal or o differece amog mea) veru H A : ot all mea are equal we ue the followig tet tatitic: = ) + ( ) + ( ) ( ) ( + ) ( + ) ( w b where i the grad mea of all + + = 3 obervatio: = Uder H 0 thi tatitic ha a "F-ditributio with ad -3 degree of freedom." The F-ditributio i the quare of the t-ditributio jut like the Chiquare i the quare of the Z-ditributio.

18 McNemar Tet ad Itroductio to ANOVA More geerally whe there are k group: k k k The tet tatitic i F k k = k i = i ( k i ) k ( i ) k i = b / w Clever tatiticia have proved that E( w ) = ad that whe H0 i true E( b ) = a well. But whe HA i true E( ) >. b The bigger the F-tatitic the troger the evidece that the populatio mea are ot all equal. Moreover whe H0 i true the ratio b / w ha a F probability ditributio with k ad k degree freedom. Author: Blume Greevy BIOS 3 Page 8 of 5

19 McNemar Tet ad Itroductio to ANOVA To tet H 0 at level 5% fid the critical value i the F- table ad reject if the oberved value b / w eceed the critical value. Or fid the p-value i the table p = P( F k k > F oberved ) ad reject if it i maller tha 5%. T Whe k i two the F. For eample we foud that at 5% the critical value of T with 6 df i.0 o the critical value of T i i the ame a Table 9 how that thi (4.49) i ideed the critical value of F with ad 6 df. Author: Blume Greevy BIOS 3 Page 9 of 5

20 McNemar Tet ad Itroductio to ANOVA ANOVA i built from the ame material a the two ample t-tet ad the ame aumptio are required: ormal ditributio with equal variace. A with the t-tet the ANOVA tet are robut (relatively ieitive) to failure of the ormality aumptio. There i a oparametric alterative tet (the Krukal-Walli tet) that i baed o the rak of the obervatio ad doe ot require that the uderlyig ditributio be ormal. What do you do after the F-tet reject the hypothei of o differece amog the k populatio mea ad you wat to kow which pair of mea are differet? There are variou complicated approache: The implet i to tet all of the poible pair uig twoample t-tet (with w replacig p o you have -k df) performig all k tet at the level α/ kow a a Boferroi-adjuted α-level. k. Thi i Author: Blume Greevy BIOS 3 Page 0 of 5

21 McNemar Tet ad Itroductio to ANOVA Adjutig the alpha (α) level There are time whe it i eceary to cotrol the overall Type I error ad keep it from iflatig. For eample if you deig a tudy of a ew drug with two primary edpoit ad you coider the tet a ucce if the drug perform better o either edpoit. You may cotrai the overall type I error to a α-level by tetig each edpoit at the α/ level o that the total chace of makig a Type I error i thi tudy would be α. There are a variety of opiio about whether thi make ee. The baic coflict i i figurig out which error you wat to cotrol: the error for a idividual edpoit or the overall error for a tudy. Cotrollig the overall error ca lead to weird reult becaue ow rejectio of oe edpoit deped o how may other edpoit you decide to tet. For eample oe edpoit might yield a p-value of 0.04 which would reject the ull at the 5% level ad coclude the drug work. But if you have two edpoit ad cotrai the overall error to 5% by tetig each edpoit a.5% level the you would fail to reject ad coclude the drug doe ot work. Author: Blume Greevy BIOS 3 Page of 5

22 McNemar Tet ad Itroductio to ANOVA To make matter wore both procedure have a overall 5% error rate. So you ca oly claim rejectio at the 5% level i either cae. Procedure like thi make people woder if tatiticia really have their head crewed o traight: The drug work if you did ot tet the other edpoit but it doe ot work if you did. Thi i a faciatig but edle debate. The problem i that moder tatitic ue oe quatity (the tail area either a a p-value or type I error) to do two thig: () meaure the tregth of evidece agait the ull hypothei ad () tell me how ofte I make a mitake. Ad it make ee to adjut # but ot #. The oly way thi ca be reolved i to trah thi approach ad try omethig ew (like uig a likelihood ratio to meaure the tregth of evidece ad calculatig it Type I ad Type II error). Blume ad Peipert What your tatiticia ever told you about p-value (003) ha a ice dicuio of thi poit. I fact a a matter of priciple the ifrequecy with which i particular circumtace deciive evidece i obtaied hould ot be cofued with the force or cogecy of uch evidece. [Fiher 959] Author: Blume Greevy BIOS 3 Page of 5

23 McNemar Tet ad Itroductio to ANOVA Sigle edpoit: Frequetit error rate (Type I ad Type II; reject whe i the tail) alog with likelihood error rate (reject whe the likelihood ratio i greater tha ). Adjutig the Type I error to keep it at 5%. Frequetit propertie('error' rate) Idividual Error Rate Type II Error () Likelihood ( both error ) Traditioal ( 0 Saftey edpoit) Traditioal ( 3 Saftey edpoit) Type I Error () Sample Size** *Likelihood i ot affected by multiple edpoit **Sample Size i relative to thi eample; but orderig hold i geeral Author: Blume Greevy BIOS 3 Page 3 of 5

24 McNemar Tet ad Itroductio to ANOVA Overall edpoit: Frequetit error rate (Type I ad Type II; reject whe i the tail) alog with likelihood error rate (reject whe the likelihood ratio i greater tha ). Adjutig the Type I error to keep it at 5%. Frequetit propertie('error' rate) Idividual Error Rate Type II Error () Likelihood ( both error ) Traditioal ( 0 Saftey edpoit) Traditioal ( 3 Saftey edpoit) Type I Error () Sample Size** *Likelihood i ot affected by multiple edpoit **Sample Size i relative to thi eample; but orderig hold i geeral Author: Blume Greevy BIOS 3 Page 4 of 5

25 McNemar Tet ad Itroductio to ANOVA Plot of the average error rate ((type I+ type II)/) for hypothei tetig ad likelihood iferece. Multiple edpoit are icluded alog with the adjuted ad ot adjuted reult for hypothei tetig. Probability of idetifyig the Fale hypothei (with multiple edpoit) Overall Eperimetal Error Rate Oe Edpoit Four Edpoit Likelihood Traditioal Traditioal w/ adjutmet* Sample Size** *Adjutmet for multiple edpoit create additioal problem ad i ot uiformly recommeed **Sample Size i relative to thi eample; but orderig hold i geeral Author: Blume Greevy BIOS 3 Page 5 of 5