SEMIPARAMETRIC INFERENTIAL PROCEDURES FOR COMPARING MULTIVARIATE ROC CURVES WITH INTERACTION TERMS

Transcription

1 Sttistic Sinic ), SEMIPARAMETRIC INFERENTIAL PROCEDURES FOR COMPARING MULTIVARIATE ROC CURVES WITH INTERACTION TERMS Linsheng Tng 1 nd Xio-Hu Zhou 2,3 1 George Mson University, 2 VA Puget Sound Helth Cre System nd 3 University of Wshington Abstrct: Multivrite ROC curve models tht include n interction term between biomrker type nd flse positive rte re importnt in comprtive biomrker studies, becuse such interction llows ROC curves of different biomrkers to cross ech other. However, there hs been limited work in drwing inference for compring multivrite ROC curves, especilly when interction terms re present. In this rticle we derive the symptotic covrince of three estimtors for multivrite ROC models. These covrince estimtes hve not been redily vilble in the literture, nd bootstrp methods hve been used to obtin them. With the redily vilble vrince estimtes, we cn esily perform hypothesis testing mong ROC curves, while bootstrp tests re not so esily performed. The symptotic results re pplied to compre ROC curves nd their res under ROC curves. Moreover, we derive simultneous confidence bnds for multivrite ROC curves. We evlute nd compre the finite smple performnce of our symptotic covrince estimtors. We lso discuss the dvntge of using our symptotic results over bootstrp procedures. Finlly, we illustrte our pproch through well-known pncretic cncer study. Key words nd phrses: Bootstrp, dignostic ccurcy, simultneous confidence bnd. 1. Introduction Reserch in erly cncer detection involves developing dignostic tools, such s biomrker, to distinguish disesed ptients from non-disesed ptients. Biomrkers often yield continuous mesurements. A populr tool to evlute nd compre the ccurcy of biomrkers is receiver operting chrcteristic ROC) curve Zhou, McClish nd Obuchowski 2002)), which is plot of true positive rtes vs flse positive rtes cross ll thresholds. Estimting single binorml ROC curve of continuous-scle biomrker hs been well studied in the literture Metz, Hermn nd Shen 1998) nd Ci nd Moskowitz 2004)). However, inferentil procedures for compring multivrite ROC curve models

2 LIANSHENG TANG AND XIAO-HUA ZHOU Figure 1. Empiricl ROC curves for CA 19 9 nd CA 125: solid line, CA 19 9; dshed line, CA 125. with interction terms between biomrker type nd flse positive rtes FPRs) hve not been well studied, minly becuse it is sometimes difficult to drw inferences the presence of these interction terms. However, such interctions re importnt becuse they llow ROC curves to cross ech other. For exmple, Wiend, Gil, Jmes nd Jmes 1989) studied pncretic cncer biomrkers, CA 19 9 nd CA 125, which were mesured on 51 pncretitis ptients nd 90 pncretic cncer ptients. The empiricl ROC curves were generted from this dt set nd their plots re shown in Figure 1. It is cler tht two ROC curves cross ech other when FPR gets close to 1. This shows the existence of interction terms between biomrker type nd FPRs. Metz, Wng nd Kronmn 1984) proposed mximum likelihood estimtor MLE) for estimting bivrite binorml models from ordinl dt, but their method requires estimting correltion prmeters, besides the loction nd scle prmeters in mrginl norml distributions. It would be difficult to extend their MLE method to more thn two ROC curves when mny more correltion prmeters re to be estimted. As the number of biomrkers gets lrge, the MLE method becomes inpplicble. For multiple independent ROC curves, Zhng 2004) nd Zhng nd Pepe 2005) proposed n intuitive lest squres LS) method, nd Pepe 2000) presented n elegnt generlized liner model GLM) pproch. Since it is complicted to derive symptotic results, the uthors did not consider lrge smple inference for the LS nd GLM estimtors with clustered d t. Ci nd Pepe 2002) considered n interesting semiprmetric

3 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1205 generlized estimting eqution GEE) method to llow n unknown bseline function when estimting ROC curves from correlted biomrker dt. They derived symptotic results for their estimtors, but they did not consider how to compre multiple ROC curves from clustered dt with the presence of interctions between biomrker type nd FPRs. In this rticle we consider the LS method for clustered ROC curve dt with the presence of interctions between biomrker type nd FPRs. We derive explicit covrince structures between empiricl ROC curves nd use our results to derive the symptotic covrinces of the LS estimtor. We lso dpt the GLM nd GEE methods to this type of biomrker dt nd derive their symptotic sndwich covrince estimtors. These covrinces hve not been redily vilble in the literture, nd bootstrp methods hve been used to obtin them. With redily vilble vrince estimtes we cn esily perform hypothesis testing mong ROC curves, while bootstrp tests re not so esily performed. For exmple, if we wnt to test whether two ROC curves vry by certin mount δ t specified FPR u 0, i.e., H 0 : ROC 1 u 0 ) ROC 2 u 0 ) = δ, it is not cler how to bootstrp from the null distribution, while it is strightforwrd to perform such hypothesis tests using our symptotic results nd the delt method tht is introduced in Section 4. We derive inferentil procedures for compring multivrite ROC curves tht include interction terms, which hve multivrite binorml ROC curves s specil cse. In prticulr, we develop methods for compring ROC curves nd the res under these ROC curves. We derive symptotic simultneous confidence bnds for ROC curves. Such symptotic results of simultneous confidence bnds of ROC curves re rrely studied. Insted, computer intensive methods re often employed to construct confidence bnds Ci nd Pepe 2002)). Our confidence bnds provide n esy-to-use tool to illustrte the smpling vribility of the ROC curve estimtes. In ddition, we develop method for compring multiple ROC curves t some specified FPR. This rticle is orgnized s follows. In Section 2 we consider LS, GEE nd GLM methods for multivrite ROC curves. In Section 3 we derive the symptotic results for the LS method when estimting multivrite ROC models; the symptotic results re lso derived for GLM nd GEE methods. In Section 4 we pply the results to compre ROC curves nd the res under them. In ddition, symptotic simultneous confidence bnds re derived for multivrite ROC curves. We crry out lrge scle simultion studies to evlute nd compre the finite smple performnce of our covrince estimtors from the LS, GLM nd GEE methods. We lso crry out simultion studies to evlute the dvntges of using symptotic results over using bootstrp procedures. The simultion results re summrized in Section 5. A comprtive pncretic cncer dignostic tril serves s n illustrtive exmple in Section 6, nd some discussion is presented in Section 7.

4 1206 LIANSHENG TANG AND XIAO-HUA ZHOU 2. Three Estimtors of Multivrite ROC Curves In this section, we dpt the LS nd GLM methods to clustered ROC dt nd give simplified version of the GEE method for estimting multivrite ROC curves. Let X l = X l,1,..., X l,m ) denote mesurements of the lth biomrker on m disesed subjects, nd Y l = Y l,1,..., Y l,n ) denote mesurements of the lth biomrker on n helthy subjects, where l, l = 1,..., K. For the lth nd lth different biomrkers mesured on the ith disesed subject, i = 1,..., m, the mesurements X l,i nd X l,i follow bivrite survivl function F l, l with the mrginl distributions F l nd F l, respectively, nd the mesurements of the jth helthy subject, Y l,j nd Y l,j with j = 1,..., n, follow bivrite survivl function Ḡl, l with the mrginl distributions Ḡl nd Ḡ l, respectively. The ROC curve of the lth biomrker is then given by Q l u) = F l Ḡ 1 l u)), nd its empiricl form is Q l u) = F l Ḡ 1 l u)), where F l nd Ḡ l re empiricl functions of F l nd Ḡ l, respectively. Let Z k be dummy vrible for the kth biomrker, k = 2,..., K. The multivrite ROC curves re given by nd Q 1 u) = gθ 10 + θ 11 hu)}, [ K ] Q k u) = g θ 10 + θ 11 hu) + θ k0 Z k + θ k1 Z k hu)}, 2.1) for 0 < u b < 1, where g is some specified link function nd h is some specified bseline function. In the regression ROC modeling, θ 10 + θ 11 hu) is the bseline function, usully denoted s h 0 u). In this rticle we tke this bseline function to be known function, up to two unknown prmeters θ 10 nd θ 10, s in Pepe 2000), Zhng 2004) nd Zhng nd Pepe 2005). The importnt components of the model 2.1) re the θ k1 Z k hu), which re the interction terms between FPRs nd dummy vribles indicting biomrker types. Such interction terms ply n importnt role in estimting ROC curves, especilly when estimting the multivrite binorml ROC curves. With these terms, the 2.1) includes the multivrite binorml ROC model s the specil cse tht is commonly seen in the literture Metz, Wng nd Kronmn 1984)). Specificlly, let g be Φ, let h be Φ 1, nd let Z k be n indictor vrible for the kth biomrker, so 2.1) cn be written s k=2 Q 1 u) = Φθ 10 + θ 11 Φ 1 u)} nd Q k u) = Φθ 10 + θ 11 Φ 1 u) + θ k0 + θ k1 Φ 1 u)}, 2.2) for k = 2,..., K. When there is only one biomrker, 2.2) reduces to the commonly used binorml model Zhou, McClish nd Obuchowski 2002)).

5 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1207 Let u l = u l,1,..., u l,pl ) T be some fixed prtition points in [, b] on the lth empiricl ROC curve. Here P l is rbitrrily chosen for the lth ROC curve where 0 < = u l,1 < < u l,pl = b < 1. For exmple, if we choose 50 jump points for the 1st ROC curve, we cn choose u 1 = 1/51,..., 50/51). Also, for simplicity, we write Lu l ) = Lu l,1 ),..., Lu l,pl )) T, where l = 1,..., K, for ny process or function L. Zhng 2004) nd Zhng nd Pepe 2005) proposed lest squres LS) pproch to estimte multiple ROC curves. They let Z k be the indictor vribles for the kth biomrker. In the LS estimting procedure, the ROC curve corresponding to reference biomrker is chosen s the reference ROC curve. For the lth empiricl ROC curve, the prtition points, u l = u l,1,, u l,pl ) T, re chosen within intervl boundries [, b]. When we re interested in the entire ROC curve, nd b cn be chosen to be close to 0 nd 1, respectively. If prtil ROC curve is of interest, nd b cn be chosen ccordingly. By plugging in the empiricl functions F 1 l nd Ḡ l, the lth empiricl ROC curve is Q l u l ) = F l Ḡ 1 l u l )). Let Ỹl = g 1 Q l u l )). We combine Ỹ1,..., ỸK to get the liner regression eqution Ỹ = Mθ + ɛ, 2.3) where Ỹ = Ỹ 1 T,..., Ỹ K T )T is l P l) 1 vector with elements Ỹl = g 1 Q l u l )). The l P l) 2K) design mtrix M is M M 2 M M = M 3 0 M3 0, with its P l 2 submtrices 1 1 M l = hu l,1 ) hu l,pl ) M K 0 0 M K ) T ) T, nd Mk Zk = Z k. Z k hu l,1 ) Z k hu l,pl ) Also, the error term ɛ hs multivrite norml distribution given by ɛ N0, Σ ɛ ). The detiled proof of its multivrite normlity is given in the online version of the pper t Bsed on the regression equtions in 2.3), the LS estimtor ˆθ LS of θ is given by ˆθ LS = M T M) 1 M T Ỹ. There re severl other estimting eqution methods for estimting ROC prmeters. Pepe 2000) observed tht the expected vlue of indictor vribles

6 1208 LIANSHENG TANG AND XIAO-HUA ZHOU 1 IX l,i Ḡ l u l,p )} converges to the true ROC curve of the lth biomrker nd proposed GLM method to estimte the ROC curve model 2.1). If prtil ROC curves on [, b] re of interest, the u l,p re chosen within this rnge. The GLM pproch estimtes prmeters in the model 2.1) by the estimting equtions K P l m i=1 or g M l,p T θ) 1 } } Ml,p g M l,p T θ)1 g M l,p T θ)) I X l,i Ḡ l u l,p ) g M l,p T θ) = 0, K P l w l u l,p ) M l,p Ql u l,p ) g M l,p T θ)} = 0, for l = 1,..., K nd p = 1,..., P l, with θ = θ 10, θ 11, θ 20, θ 21,..., θ K0, θ K1 ) T nd weight function w l u l,p ) = g M l,p T θ)}/g M l,p T θ)1 g M l,p T θ))}. Here M 1,p is 1 2K vector with the first two elements being 1 nd hu 1,p ), respectively, nd the rest of the elements zero. The M k,p hve the first two elements being 1 nd hu k,p ), respectively, the 2k + 1)th nd 2k + 2)th elements being Z k nd Z k hu k,p ), respectively, nd the rest of the elements being zero. The symptotic results of the prmeter estimtor not provided in Pepe 2000) re given in Section 3. The GEE method of Ci nd Pepe 2002) lso used the indictor vribles 1 IX l,i Ḡ l u l,p )}, nd relied on the fct tht points on ROC curves cn be interpreted s conditionl expecttions of these indictor vribles. Their method is flexible enough to llow n unknown bseline function h 0 in the model 2.1). When the bseline function hs the form of h 0 u) = θ 10 + θ 11 hu), the GEE method is similr to the GLM method. Specificlly, the GEE method solves the estimting equtions K P l } M l,p Ql u l,p ) g M l,p T θ)) = 0. It is observed tht if the bseline function hs known form, the GLM method differs from the GEE method by including the weight function w l. 3. Lrge Smple Theory of ROC Estimtors We derive the symptotic covrince structure of multiple empiricl ROC curves for clustered dt. The result is then pplied to derive symptotic covrinces of the LS estimtor. We lso derive symptotic sndwich covrinces of GLM nd GEE estimtors. Asymptotic results of the LS nd GLM estimtors for clustered dt hve not been provided in the literture Pepe 2000),

7 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1209 Zhng 2004) nd Zhng nd Pepe 2005)). Although Ci nd Pepe 2002) hve studied the lrge smple theory of the GEE estimtor, their covrince estimtor hs complicted form. For the multivrite ROC models we consider here, the covrince estimtor of resulting estimtor hs simplified covrince estimtor tht is esy to pply. These covrince results re essentil for drwing inference on compring ROC curves nd constructing simultneous confidence ROC bnds, s discussed in Section 4. We derive the covrince structure between empiricl ROC curves, nd summrize the results in Theorem 1. The proof of Theorem 1 is given in the online version of the pper t Theorem 1 provides bsis for deriving symptotic covrinces of the LS estimtor. Theorem 1. If 1) F l nd Ḡl hve continuous densities F l nd Ḡ l, respectively, nd 2) the first derivtive Q l of Q l is bounded in, b), when m/n λ s m, n, cov[ m Q l s) Q l s)}, m Q lt) Q lt)}] converges in distribution to [ Ḡ 1 Fl, l l for s, t in, b). } ] [ s), Ḡ 1 t) Q l l s)q lt) +λ Q Ḡl, lḡ 1 l s)q lt) l 3.1. Asymptotic covrince of the LS estimtor s), t)} st} ] Ḡ 1, l Let θ l1 = θ 11 when l = 1, nd θ l1 = θ 11 + θ l1 when l 2. Consider the 2K 2K squre mtrix 1 KD Z 2 D Z K D I 2 I 2 I 2 Z 2 D Z2 2 J = D O O I 2 O , Z K D O ZK 2 D O O I 2 where b b D = hu)du, ) b hu)du b h2 u)du for 0 < < b < 1, nd I 2 is 2 2 identity mtrix. Let V l s, t) = where l = 1,..., K, nd Q l s t) Q l s)q l t) g g 1 Q l s)))g g 1 Q l t))) + λ θ 2 l1 h s)h t)s t st), Ṽ l, ls, t) = F l, lḡ 1 l s), Ḡ 1 t)) Q l l s)q lt) g [g 1 Q l s)}])g g 1Q lt)))

8 1210 LIANSHENG TANG AND XIAO-HUA ZHOU +λ θ l1 θ l1 h s)h t)ḡl, l Ḡ 1 l s), Ḡ 1 t)) st}, l for l, l = 1,..., K, nd l l, where λ = lim m,n m/n. Theorem 2 gives results for the LS estimtor. The proof of Theorem 2 is given in the online version of the pper t Theorem 2. Under the regulrity conditions of Theorem 1, when m/n λ s m, n, nd P l, the regression prmeter estimtor ˆθ LS stisfies mˆθls θ) D N0, Σ LS = JΣ y J T ). Here Σ y is the mtrix Σ y 11 Σ y 12 Σ y 1K Σ y Σ y 21 Σ y 22 Σ y 2K =... Σ y K1 Σy K2 Σ y KK whose 2 2 digonl symmetric submtrices, Σ y ll hve elements σ 1,1) ll = b b σ 1,2) ll = σ 2,1) ll = V l s, t)dsdt, σ 2,2) b b b b ll = hs)v l s, t)dsdt, hs)ht)v l s, t)dsdt, nd whose 2 2 off-digonl symmetric submtrices, Σ y, hve elements l, l σ 1,1) l, l σ 1,2) l, l = b b = σ 2,1) l, l = Ṽ l, ls, t)dsdt, σ 2,2) l, l b b = hs)ṽl, l s, t)dsdt. b b hs)ht)ṽl, l s, t)dsdt In prctice, Fl, l, Ḡ l, l, Fl, nd Ḡl re unknown, nd re estimted by their respective empiricl functions. If the ROC dt re unclustered, the off-digonl submtrices, Σ y s, re zero mtrices. If we further let h = l, l g 1 nd Z k be indictor vribles in the model 1), we get the sme symptotic result s in Zhng 2004) Asymptotic covrinces of the GLM nd GEE estimtors The GLM estimtor, ˆθ L, is obtined by solving the estimting equtions Uθ) = K P l } w l u l,p ) M l,p Ql u l,p ) g M l,p T θ) = 0,

9 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1211 where M l,p is defined in Section 2.1. To solve these equtions for ˆθ L, we need the Newton-Rphson method. Then covˆθ L ) = Uθ) ) 1 K vr θ l=1 Uθ) ) 1, θ P ) w l u l,p ) M l,p Q l u l,p ) g M l,p T θ)} where Uθ)/ θ is the prtil derivtive of Uθ) with regrd to θ. Let U 1i = nd U 2j = K P l K P p=1 w l u l,p ) M l,p IX li Ḡ 1 l u l,p )) g M T l,p θ) }, w l u l,p ) M l,p Q l u l,p) IY lj Ḡ 1 l u l,p )) u l,p }. The result of Theorem 1 gives us tht, under mild regulrity conditions, when m/n λ s m, n, the GLM estimtor ˆθ L stisfies mˆθl θ) D N0, Σ L ), where K P l ) 1 Σ L = w l u l,p ) M l,p g M l,p T θ) K P l ) 1 w l u l,p ) M l,p g M l,p T θ). lim m,n m U 1i U1i T + λ i=1 n } U 2j U2j T The symptotic property of the modified GEE estimtor, ˆθ E, cn be similrly derived. Let U 1i = nd U 2j = K P l K P l M l,p IY li Ḡ 1 l u l,p )) g M T l,p θ) }, v=1 M l,p Q l u l,p) IY lj Ḡ 1 l u l,p )) u l,p }. When m/n λ s m, n, the GEE estimtor ˆθ E stisfies mˆθe θ) D N0, Σ E ),

10 1212 LIANSHENG TANG AND XIAO-HUA ZHOU where K P l ) 1 Σ E = M l,p g M l,p T θ) K P l ) 1 M l,p g M l,p T θ). lim m,n m r=1 U 1iU T 1i + λ n v=1 } U2jU 2j T These covrince estimtors re sndwich estimtors. finite smple performnce in our simultion studies. We will evlute their 4. Multivrite ROC Anlysis Estimted ROC curves, denoted by Q l, re obtined by replcing θ with the estimtor ˆθ in the model 2.1). ˆθ is estimted vi either of three forementioned methods. Here ˆθ is generl nottion nd cn be found vi other methods s well. But, since symptotic results re derived for the three methods, it is convenient to utilize these results. Our symptotic results give the corresponding covrince mtrix estimtor ˆΣ of Σ for ˆθ. For further nottionl convenience, let Σ l, l be the 2 2 submtrices of Σ 11 Σ 12 Σ 1K Σ = Σ 21 Σ 22 Σ 2K.... Σ K1 Σ K2 Σ KK 2K 2K The estimtor ˆΣ l, l of Σ l, l is the corresponding submtrix of ˆΣ. In this section we derive methods for pirwise comprison of ROC curves. We lso develop methods for compring more thn two res under ROC curves under multivrite binorml ssumptions. In ddition, we derive inferentil procedures for simultneous confidence ROC bnds, nd for compring multiple ROC curves t some specified FPR Pirwise comprison of ROC curves It is often of interest to compre ROC curves to investigte the ccurcy of biomrkers. The reference ROC curve, Q 1 u), nd the kth ROC curve, Q k u), in [, b] only differ by prmeter vector θ k = θ k0, θ k1 ) T. Consequently, testing the equlity of these two ROC curves is equivlent to testing H 0 : θ k = 0, 0) T. It follows from Section 3 tht the symptotic distribution of the test sttistic, κ k = ˆθ k θ k ) T Σ 1 kk ˆθ k θ k ) is χ 2 2. Similrly, testing the equlity of two ROC curves of the ωth nd νth different biomrkers in [, b], ω, ν = 2,..., K, lso

11 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1213 reduces to χ 2 test. The null hypothesis becomes H 0 : θ ω0 θ ν0, θ ω1 θ ν1 ) = 0. Let θ ω,ν = θ ω0, θ ω1, θ ν0, θ ν1 ) T. The resulting chi-squre sttistic is ˆθω0 ˆθ ) T ν0 ) θ ω0 θ ν0 ) κ ω,ν = ˆθ ω1 ˆθ AΣ ω,νa T ) 1 ˆθω0 ˆθ ) ν0 ) θ ω0 θ ν0 ) ν1 ) θ ω1 θ ν1 ) ˆθ ω1 ˆθ, ν1 ) θ ω1 θ ν1 ) where A = I 2, I 2 ) with 2 2 identity mtrix I 2, nd Σ ω,ν is the covrince mtrix of θ ω,ν Compring the res under multivrite binorml ROC curves Our symptotic results in Section 3 cn be used to compre res under multivrite ROC curves, especilly multivrite binorml ROC curves. Let A = A 1,..., A K ), where A l is the re under the lth ROC curve. It is well known tht under the binorml ssumption, θ 10 A 1 θ 10, θ 11 ) = Φ }, 1 + θ 2 11 θ 10 + θ k0 nd A k θ 10, θ 11, θ k0, θ k1 ) = Φ }. 4.1) 1 + θ11 + θ k1 ) 2 Let q be second-order differentible nd rel-vlued function of A. It follows tht when m/n λ s m, n, mq Â) qa)} D N0, σ 2 q), where σ 2 q = lim m m,n + K K k=2 k=2 A 1 A k A 1 vra 1 ) + 2 A k cova k, A k) K A 1 k=2 }. A k cova 1, A k ) By Tylor expnsions on A l nd our symptotic results, we find nd vra 1 ) = B T 1 Σ 11 B 1, cova 1, A k ) = B T 1 Σ 11 + Σ k1 )B k, cova k, A k) = B T k Σ 11 + Σ k k)b k, where B 1 = φ θ10 ) 1, φ 1 + θ θ 2 11 θ θ 2 11 ) θ θ 2 11 )3/2 } T,

12 1214 LIANSHENG TANG AND XIAO-HUA ZHOU [ } θ 10 + θ k0 1 B k = φ 1 + θ11 + θ k1 ) θ11 + θ k1 ), 2 } ] T θ 10 + θ k0 θ φ 11 + θ k1 1 + θ11 + θ k1 ) 2, 1 + θ 11 + θ k1 ) 2 } 3/2 nd Σ l l, for l, l = 1,..., K, is estimted from symptotic results. DeLong, DeLong nd Clrke-Person 1988) gve similr formul for compring the res under nonprmetric ROC curves. Although their pproch is robust, semiprmetric pproches my be more ppeling for deriving smooth ROC curves for continuous biomrker dt. If q is some liner function, the theoreticl result is simplified to similr formul in their pper. A simple exmple hs E s vector with the lth element 1, lth element 1 nd other elements zero. Then qâ) = E corresponds to Âl  l, whose vrince estimtor follows from the vrince result. Inference is then esily drwn for testing H 0 : A l = A l, nd for constructing confidence intervl Simultneous confidence ROC bnds The vrince of estimted ROC curves t ech FPR cn be derived from the prmeter estimtor ˆθ nd its covrince mtrix estimtor ˆΣ. Let H = 1, hu)), H = H, H). Corollry 2. The vrince of estimted ROC curves t u re nd σ1u) 2 = g [g 1 Q 1 u)}] 2 HΣ 11 H T ) σk 2 u) = g [g 1 Q k u)}] 2 Σ11 Σ 1k H Σ k1 Σ kk H T, respectively, for k = 2,..., K. The 1 α)100% pointwise confidence intervl of Q l u) is then given by Q l u) ± z α/2ˆσ l u), 0 u 1. In Theorem 3 below, we give explicit expressions for simultneous bnds for multivrite ROC curves. The proof of Theorem 3 is given in the online version of the pper t Theorem 3. Under the regulrity conditions of Theorem 1, the 1 α)100% simultneous confidence bnds for multivrite ROC curves in [, b] re g H ˆθ } 1 ± χ 2 2,α HΣ 11H T,

13 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1215 nd ) g Hˆθ 1 T, ˆθ k T )T ± Σ11 Σ χ 2 1k 4,α H Σ k1 Σ kk H T }, respectively, for k = 2,..., K. Note the reson why simultneous confidence bnds for ROC curves hve such simple expressions is tht we ssume tht g nd h re known. The estimted ROC curves re fully determined by two prmeters for the reference biomrker nd four prmeters for other biomrkers. Therefore, χ 2 2 nd χ2 4 distributions rise, nd the derivtion of simultneous bnds is nturlly simplified Compring multiple ROC curves t some specified FPR Let Qu) = Q 1 u),..., Q K u)) T. Similr to Section 4.2, suppose tht q is rel-vlued nd second-order differentible function of the vector Q. We hve tht q Qu 0 )), t specified FPR u 0, converges to norml distribution with men zero nd vrince σ 2 qu 0 ) = lim m m,n + K K k=2 k=2 [ Q 1 Q k Q 1 vrq 1 u 0 )} + 2 Q k K Q 1 ] k=2 covq k u 0 ), Q ku 0 )}. Q k covq 1 u 0 ), Q k u 0 )} Here we hve vrq 1 u 0 )} = C 1 u 0 ) T Σ 11 C 1 u 0 ), covq 1 u 0 ), Q k u 0 )} = C 1 u 0 ) T Σ 11 + Σ k1 )C k u 0 ), covq k u 0 ), Q ku 0 )} = C k u 0 ) T Σ 11 + Σ k k)c k u 0 ), where C 1 u) = g θ 10 + θ 11 hu)}, θ 11 g θ 10 + θ 11 hu)}) T, C k u) = [g θ 10 + θ 11 hu) + θ k0 + θ k1 hu)}, θ 11 + θ k1 )g θ 10 + θ 11 hu) + θ k0 + θ k1 hu)}] T. If q is liner function, the result cn be gretly simplified. 5. Simultion Studies 5.1. Finite smple performnce of hypothesis testing We rn lrge set of simultion studies to evlute nd compre the finite smple performnce of our symptotic covrince estimtors for the LS, GLM nd GEE methods. The bivrite norml dt were N1, 1), Σ 0 ) for the disesed nd N0, 0), Σ 0 ) for the helthy, where Σ 0 hs the vrinces 1 nd 2 with correltion prmeter ρ. True ROC curves of tests 1 nd 2 hve the sme form given by Q 1 u) = Q 2 u) = Φ1/ 2 + 1/ 2Φ 1 u)}, for 0 u 1. Thus, the true vlue of the prmeter vector is θ = 1/ 2, 1/ 2, 0, 0). We fit bivrite binorml model to the dt; tht is, we let K = 2 in the model 2.2). The null hypothesis of equl ROC curves, H 0 : θ 20, θ 21 ) = 0, 0), cn be tested

14 1216 LIANSHENG TANG AND XIAO-HUA ZHOU by the χ 2 test sttistic κ = ˆθ 20, ˆθ 21 )ˆΣ 1 22 ˆθ 20, ˆθ 21 ) T with 2 degrees of freedom, where ˆθ 20, ˆθ 21 ) s covrince mtrix estimte, ˆΣ 22, is clculted using symptotic results in Theorem 2. We simulted 1,000 dt sets under the null hypothesis with m = 50, 100, 200) nd n = 50, 100, 200) under ρ = 0, 0.25, 0.5, 0.75). The nominl rejection rte ws set t 5%. In the simultion, the vrinces of LS estimtors were estimted using symptotic results developed in Theorem 2. The vrinces of GLM nd GEE s estimtors were obtined using symptotic results in Section 3. For smll smple size such s 50, the GEE method sometimes did not converge nd we hd to run more thn 1,000 simultions in order to obtin 1,000 vlid estimtes. Since the LS method does not require itertions, the computtion time is gretly reduced compred to tht of GLM or GEE. Under the sme circumstnces, the computing time of LS is less thn hlf tht of GLM or GEE. In prticulr, we conducted our simultion study on the sme Unix mchine. It took 30 seconds for LS to estimte prmeters from 1,000 simulted dt sets when m = n = 200, while the computtionl times of GLM nd GEE were 870 nd 348 seconds, respectively. When m = n = 50, the computtion time of LS ws 24 seconds, while times of GLM nd GEE were 210 nd 98 seconds, respectively. Tble 1 presents rejection rtes from the three pproches. As shown in Tble 1, the symptotic results for LS work well for ll combintions of smple sizes s the rejection rtes re close to the nominl level of 5%. Even for smll smple size 50 for both disesed nd helthy groups, rejection rtes do vry much from the nominl level. Moreover, rejection rtes of LS re not ffected by vlues of the correltion prmeter ρ. From Tble 1, GEE nd GLM hve much in common. Both hve over-rejection rtes when smple sizes re smll. This is minly due to the vribility of their sndwich vrince estimtors. Reders re referred to Kuermnn nd Crroll 2001) for more detils. It is lso noticeble in Tble 1 tht s smple sizes for the helthy get lrger, rejection rtes of GEE nd GLM get closer to the nominl level even when smple sizes for the disesed re smll. However, rejection rtes for the disesed, given smll smple sizes for the helthy, re not improved with lrge smple sizes. We tried bootstrp methods in these situtions. The bootstrp method performed similrly to the LS method on the rejection rtes regrdless of smple sizes. The bootstrp performed better thn GLM nd GEE when smple sizes were smll, nd similrly to GLM nd GEE when smple sizes were lrge Finite smple performnce of point nd intervl estimtes We used the sme setting s in the previous section to evlute nd compre estimtion precision of the methods in the simultion study. We gin simulted 1,000 dt sets under smple sizes m = n = 50, 200, 400) with ρ = 0.5. The nominl coverge probbility of confidence intervls ws 95%. We pplied LS,

15 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1217 Tble 1. Rejection rtes in %) with the nominl level α = 0.05, from symptotic results. LS GEE GLM m n ρ = ρ = ρ = The rejection rte with 1,000 reliztions of norml model. The 95% prediction intervl of the rejection rte is 5.0% ± 1.4%). GEE nd GLM to get estimtes of the ROC prmeter vector θ 10, θ 11, θ 20, θ 21 ). Confidence intervls for the prmeters were clculted bsed on symptotic results. We then compred these methods bsed on bis, squre root of MSE RMSE), nd the coverge probbilities of confidence intervls. The results re shown in Tble 2. All three methods hve good ccurcy for estimting the prmeters, while coverge probbilities differ mong them. Our simultion results show tht LS hs the nice finite smple property tht the coverge probbilities of ll prmeters re close to the nominl level for smll smple sizes. When smple sizes re smll, confidence intervls computed from sndwich covrince estimtors for GLM nd GEE cover the intercept prmeters properly, but these confidence intervls over-cover slope prmeters. As smple sizes pproch 400, the coverges of GLM nd GEE estimtors get closer to the nominl level for slope prmeters The dvntge of our symptotic results over bootstrp procedures Mny uthors hve pplied bootstrp methods to estimte covrince mtrices for the LS nd GLM estimtors when the symptotic results were not vilble Pepe 2000) nd Zhng nd Pepe 2005)). However, it cn tke much more computtion time to do bootstrpping thn to use the symptotic results. In this simultion study, we compred coverge percentges of bootstrp covrince estimtes with those of symptotic covrince estimtes for the LS pproch. We used the sme setting in Section 5.1 with m = n = 50, 200, 400). Under ech combintion of smple sizes, we simulted 1,000 dt sets with ρ = 0.5. For ech dt set we pplied bootstrp procedures to get covrince estimtes of the LS

16 1218 LIANSHENG TANG AND XIAO-HUA ZHOU Tble 2. Bis, RMSE, coverge probbility CP) of prmeter estimtors. LS GEE GLM m Bis RMSE CP BT Bis RMSE CP Bis RMSE CP n) in %) in %) in %) in %) in %) in %) in %) 50 θ θ θ θ θ θ θ θ θ θ θ θ CP is the coverge percentge for 95% confidence intervls using symptotic stndrd errors with norml quntile. BT is the coverge percentge for 95% confidence intervls using the bootstrp. Results re bsed on 1,000 reliztions of the bivrite norml model. pproch. The number of bootstrp ws 1,000. We then used covrince estimtes to get confidence intervls nd their coverge percentges. The coverge percentges of bootstrp methods were shown in Tble 2. As cn be seen there, our symptotic results were s good s bootstrp results in tht coverge percentges were very close. More importntly, computtion time ws much shorter. For exmple, when using the LS method with m = n = 400, it took 30 seconds to obtin bootstrp covrince estimte for one dt set on PC, it took only 5 seconds to obtin n symptotic covrince estimte for the sme dt set on the sme PC. 6. Appliction to Pncretic Cncer Biomrkers The min interest in the forementioned biomrker exmple is to determine whether ROC curves generted by two biomrkers re equl. If not, one would wish to sy which biomrker cn better distinguish the disesed from the helthy. We pplied the LS estimtion procedure to this dt set. With the probit link, g = Φ, nd h = Φ 1, ROC curves of these biomrkers hve the sme structure s those in 2.2) when K = 2. We got the estimte of the prmeter vector s ˆθ 10 LS, ˆθ LS 11, ˆθ LS 20, ˆθ LS 21 ) = 1.18, 0.47, 0.49, 0.55). Its covrince mtrix estimte ws clculted bsed on the symptotic result in Theorem 2. The GLM nd GEE prmeter estimtes were very close to the LS estimte, nd thus were not listed. The χ 2 ws then clculted to be with the p-vlue , indicting significnt difference in dignostic ccurcy between two biomrkers. We lso

17 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1219 Figure 2. Estimted ROC curves nd their 95% confidence bnds for CA 19-9 nd CA 125: dshed lines, empiricl ROC; solid lines, estimted ROC; shded regions, 95% confidence bnd; dotted lines, confidence bnd boundries. Figure 3. Overlpping 95% confidence bnds for CA 19 9 nd CA 125: solid lines, estimted ROC; shded regions, 95% confidence bnd; dotted lines, confidence bnd boundries. clculted the difference between the two res under estimted ROC curves using the procedure in Section 4.2, nd found significnt difference between them, p-vlue To visulize the smpling vribility of estimted ROC curves, simultneous ROC bnds were constructed using the result in Theorem 3. Figure 2 shows the estimted ROC curves nd their simultneous bnds. These ROC curves were very close to the empiricl curves. Although our results on ROC curves nd their res show tht the two biomrkers re different, it is cler in Figure 3 tht two ROC curves intersect. The simultneous bnds cn help us determine the region of FPRs where two ROC curves re different; Figure 3 shows overlpping confidence bnds. It is obvious tht the ROC curve for CA 19 9 is significntly better thn tht for CA 125 when the flse positive rte is less thn round 0.17, nd tht the two ROC curves do not differ much elsewhere.

18 1220 LIANSHENG TANG AND XIAO-HUA ZHOU 7. Discussion The interction terms in our multivrite ROC model re completely different from the interctions of the K dignostic tests on the mesurement level. In fct, the interctions we refer to re between test types nd FPRs. Multivrite ROC models such s the multivrite binorml ply n importnt role in ROC nlysis of clustered biomrkers. This rticle derived symptotic covrinces of the LS, GLM nd GEE estimtors for the multivrite ROC models with the presence of interction terms between biomrker type nd FPRs. To our knowledge, such symptotic properties with correlted dt hve not been ddressed in empiricl process theory. Our new contributions include procedures to compre AUCs nd ROC curves, especilly when the interction terms between FPR s nd biomrker type re present. We drew inference for pirwise comprison between ROC curves, multiple comprisons of res under ROC curves under binorml ssumptions. The inferentil procedures in Section 4 for compring ROC curves re very generl for multivrite ROC models. Besides the three estimtors discussed, if other estimtors nd their covrinces were vilble, they cn lso be used in these procedures. Acknowledgement The uthors would like to thnk n ssocited editor nd two referees for their constructive comments nd suggestions. This work is supported in prt by NIH grnt R01EB Xio-Hu Zhou, Ph.D., is presently Core Investigtor nd Biosttistics Unit Director t the Northwest HSR&D Center of Excellence, Deprtment of Veterns Affirs Medicl Center, Settle, WA. The views expressed in this rticle re those of the uthors nd do not necessrily represent the views of the Deprtment of Veterns Affirs. References Ci, T. nd Moskowitz, C. 2004). Semiprmetric estimtion of the binorml ROC curve. Biosttistics 5, Ci, T. nd Pepe, M. S. 2002). Semi-prmetric ROC nlysis to evlute biomrkers for disese. J. Amer. Sttist. Assoc. 97, DeLong, E. R., DeLong, D. M. nd Clrke-Person, D. L. 1988). Compring the res under two or more correlted receiver operting chrcteristic curves: nonprmetric pproch. Biometrics 44, Kuermnn, G. nd Crroll, R. J. 2001). A note on the efficiency of sndwich covrince mtrix estimtion. J. Amer. Sttist. Assoc. 96, Metz, C. E., Wng, P. nd Kronmn, H. B. 1984). A new pproch for testing the significnce of differences between ROC curves mesured from correlted dt. In Informtion Processing in Medicl Imging Edited by F. Deconinck),

19 SEMIPARAMETRIC INFERENTIAL ROC PROCEDURES 1221 Metz, C. E., Hermn, B. A. nd Shen, J-H. 1998). Mximum-likelihood estimtion of ROC curves from continuously-distributed dt. Sttist. Medicine 17, Pepe, M. S. 2000). An interprettion for the ROC curve nd inference using GLM procedures. Biometrics 56, Ro, C. R. 2002). Liner Sttisticl Inference nd Its Appliction. John Wiley nd Sons: London. Vn der Vrt, A. nd Wellner, J. 1996). Wek Convergence nd Empiricl Processes: With Applictions to Sttistics. Springer-Verlg, New York. Wiend, S., Gil, M. H., Jmes, B. R. nd Jmes, K. L. 1989). A fmily of non-prmetric sttistics for compring dignostic mrkers with pired or unpired dt. Biometrik 76, Zhng, Z. 2004). Semiprmetric lest squres nlysis of the receiver operting chrcteristic curve. University of Wshington Doctorl Disserttion. Zhng, Z. nd Pepe, M. S. 2005). A liner regression frmework for receiver operting chrcteristic ROC) curve nlysis. Working Pper 253, UW Biosttistics Working Pper Series. Zhou, X. H., McClish, D. K. nd Obuchowski, N. A. 2002). Sttisticl Methods in Dignostic Medicine. Wiley, New York. Deprtment of Sttistics, The Volgenu School of Informtion Technology nd Engineering, George Mson University, Science & Tech II, Room 155, 4400 University Drive, MS 4A7, Firfx, VA , U.S.A. E-mil: ltng1@gmu.edu Deprtment of Biosttistics, School of Public Helth nd Community Medicine, University of Wshington, U.S.A. E-mil: zhou@u.wshington.edu Received July 2007; ccepted Februry 2008)