Model misspecication sensitivity analysis in estimating causal eects of interventions with non-compliance

Transcription

1 STATISTICS IN MEDICINE Statist. Med. 2002; 21: (DOI: /sim.1267) Model misspeiation sensitivity analysis in estimating ausal eets of interventions with non-ompliane Booil Jo ; Division of Soial Researh Methodology; Graduate Shool of Eduation & Information Studies; University of California; Los Angeles; CA; U.S.A. SUMMARY Randomized trials often fae ompliations in assessing the eet of treatment beause of study partiipants non-ompliane. If ompliane type is observed in both the treatment and ontrol onditions, the ausal eet of treatment an be estimated for a targeted subpopulation of interest based on ompliane type. However, in pratie, ompliane type is not observed ompletely. Given this missing ompliane information, the omplier average ausal eet (CACE) estimation approah provides a way to estimate dierential eets of treatments by imposing the exlusion restrition for non-ompliers. Under the exlusion restrition, the CACE approah estimates the eet of treatment assignment for ompliers, but disallows the eet of treatment assignment for non-ompliers. The exlusion restrition plays a key role in separating outome distributions based on ompliane type. However, the CACE estimate an be substantially biased if the assumption is violated. This study examines the bias mehanism in the estimation of CACE when the assumption of the exlusion restrition is violated. How ovariate information aets the sensitivity of the CACE estimate to violation of the exlusion restrition assumption is also examined. Copyright? 2002 John Wiley & Sons, Ltd. KEY WORDS: randomized trial; non-ompliane; CACE; exlusion restrition; bias mehanism; sensitivity analysis 1. INTRODUCTION Non-ompliane is a ommon problem in randomized trials involving human partiipants. Sine both the standard ITT (intent-to-treat) analysis and as-treated analysis may provide biased estimates of treatment eets for ompliers in the presene of non-ompliane, the possibility of estimating treatment eets only for ompliers (CACE, omplier average ausal eet) has been explored [1 5]. Correspondene to: Booil Jo, Division of Soial Researh Methodology; Graduate Shool of Eduation & Information Studies; University of California; Los Angeles; CA , U.S.A. booil@ula.edu Contrat=grant sponsor: NIMH; ontrat=grant number: P30 MH38330, P50 MH38725 Contrat=grant sponsor: NIAAA; ontrat=grant number: K02 AA Contrat=grant sponsor: NIDA; ontrat=grant number: RO1 DA A1 Reeived September 2000 Copyright? 2002 John Wiley & Sons, Ltd. Aepted February 2002

2 3162 B. JO Under a series of statistial assumptions, CACE estimation provides an unbiased treatment eet estimate for ompliers. In the estimation of CACE, the exlusion restrition is one of the ritial underlying assumptions. This assumption provides the basis for identiability in CACE models, given that ompliane status is not observed ompletely. Under this assumption, the dierene in outome between the treatment and the ontrol ondition is allowed for ompliers, but is not allowed for never-takers (individuals who would not reeive the treatment regardless of whether it is oered) or for always-takers (individuals who would reeive the treatment regardless of whether it is oered). The assumption of the exlusion restrition plays a ritial role in simplifying methodologial diulties involved in CACE approahes. However, this assumption an often be unrealisti in pratie [6 8]. When the exlusion restrition is violated, the ausal eet of treatment not only an be understated, but also an be exaggerated depending on how the assignment of treatment aets non-ompliers. Hirano et al. [6] demonstrated the impat of the violation of the exlusion restrition in appliation to a study of the eet of an inuenza vaine in an enouragement design [9]. Their ndings showed little evidene of benet from the vaine after taking into aount the eet of treatment assignment (that is, enouragement) on always-takers. In ontrast, positive eet of the vaine was found when the eet of treatment assignment on always-takers was ignored. In the Job Searh Intervention Study [10, 11], shown as an example in this study, the assumption of the exlusion restrition is more likely to be violated for never-takers. That is, never-takers in the treatment ondition ould be demoralized by failing to take the intervention opportunity. This negative psyhologial eet would not our for never-takers in the ontrol ondition, sine the treatment is not oered. In this situation, the CACE estimate an be understated by ignoring the eet of treatment assignment on never-takers. This study examines the bias mehanism in the estimation of CACE when the assumption of the exlusion restrition is violated. It is demonstrated how the magnitude of bias is aeted by ompliane rate and the eet of treatment assignment on non-ompliers. The study also examines how ovariate information aets the sensitivity of the CACE estimate to violation of the exlusion restrition assumption. Simulation studies demonstrate that bias in the CACE estimate due to model misspeiation an be redued substantially by inluding ovariates that are good preditors of ompliane. To demonstrate CACE estimation when the assumption of the exlusion restrition is potentially violated, the Job Searh Intervention Study for unemployed workers is presented as an example. Maximum likelihood estimation using the EM algorithm is employed in the estimation of CACE in the study. 2. DATA: THE JOB SEARCH INTERVENTION STUDY The Job Searh Intervention Study (JOBS II) is a randomized eld experiment intended to prevent poor mental health and to promote high-quality re-employment. The outome measure that will be the fous in the urrent study is depression of study partiipants. Among the mental health problems assoiated with job loss, depressive symptoms are the most ommonly reported [12, 13]. The experimental ondition onsisted of ve half-day training sessions, whih inluded the appliation of problem-solving and deision-making proesses, inoulation against setbaks, provision of soial support and positive regard from the trainers, and learning and pratising job searh skills. The ontrol ondition onsisted of a booklet briey desribing

3 MODEL MISSPECIFICATION IN CACE ESTIMATION 3163 job searh methods and tips. Although the eet of reeiving the booklet was expeted to be very small, the booklet was also mailed to treatment ondition individuals, inluding people who did not show up at the intervention seminars, to prevent possible dierenes in outome between the treatment and ontrol onditions due to the booklet. The problem of non-ompliane arises in JOBS II beause a substantial proportion of individuals who were assigned to the intervention ondition did not show up to the intervention. Among study partiipants assigned to the intervention ondition, 55 per ent attended at least one session. Among attendees, 82 per ent attended four or ve sessions (mean =4:3 sessions, median =5 sessions). A previous study [14] using the instrumental variable approah [2], whih normally yields a very similar outome to CACE analysis, showed that the hoie of threshold made little dierene in the JOBS Intervention Study, possibly beause individuals assigned to the intervention ondition either attended most sessions or did not attend any. Little and Yau [5] also employed the binary ompliane approah in their CACE analysis of JOBS II using the ML-EM method. In line with previous analyses of JOBS II, the urrent study also denes ompliane as having attended at least one out of ve total sessions. However, note that sensitivity of the CACE estimate to the hoie of threshold may vary in other situations and needs to be arefully examined. Given the rate of non-ompliane in JOBS II, one an expet that treatment eay an be better estimated for ompliers if CACE estimation is applied instead of standard ITT analysis. However, speial attention is needed in applying the CACE estimation method to JOBS II, beause the assumption of the exlusion restrition is questionable. One possible explanation for this phenomenon is that never-takers in the treatment ondition beome demoralized by failing to take the intervention opportunity. Considering that the major outome analysed in this study is depression mainly aused by unemployment, this interpretation seems plausible in JOBS II. However, this explanation is not denitive and may not apply to dierent situations. As in JOBS II, blinding or plaebo-ontrol is hard to implement in most randomized eld experiments. Although the exlusion restrition assumption is always questionable in this situation, little has been studied about how treatment assignment inuenes study partiipants who deide not to omply with the treatment. 3. CACE ESTIMATION UNDER THE ASSUMPTION OF THE EXCLUSION RESTRICTION Assume the simplest experimental setting where there is only one outome measure (Y ), treatment assignment (Z) is binary (1=treatment, 0=ontrol), and the treatment reeived (D) has only two levels (1=reeived, 0=not reeived). The behaviour types (C i ) of the subjets based on ombinations of Z and D an be lassied into four ategories based on Rubin s ausal model approah, where the possibility of statistial ausal inferene is built at the individual level [15 18]. Angrist et al. [1] labelled the four ategories as omplier, never-taker, deer and always-taker. Let D i (1) denote the potential treatment reeipt status for individual i when assigned to the treatment ondition, and D i (0) denote the potential treatment reeipt status for individual i when assigned to the ontrol ondition. Compliers are subjets who do what they are assigned to do (D i (1)=1 and D i (0)=0). Never-takers are subjets who do not reeive the treatment even if they are assigned to the treatment ondition (D i (1)=0 and D i (0)=0). Deers are the subjets who do the opposite of what they are assigned to do

4 3164 B. JO (D i (1)=0 and D i (0)=1). Always-takers are the subjets who always reeive the treatment, no matter whih ondition they are assigned to (D i (1)=1 and D i (0)=1). Among these four types of subjets, the CACE approah fouses on the estimation of ausal eet of treatment assignment for ompliers. The following assumptions 1 to 5 are ritial in estimating CACE: Assumption 1 (randomization). Treatment assignment is random. Assumption 2 (stable unit treatment value, SUTVA). Potential outomes for eah person are unrelated to the treatment status of other individuals [17 19]. Assumption 3 (exlusion restrition). For never-takers and always-takers, the distributions of the potential outomes are independent of the treatment assignment [1]. That is, Y i (0;D i (0))=Y i (1;D i (1)) for units with D i (0)=D i (1)=0 or D i (0)=D i (1)=1. Assumption 4 (monotoniity). There are no deers [3]. Assumption 5 (non-zero average ausal eet of Z on D). The average ausal eet of Z on D is not equal to zero [1]. In addition, the urrent study also assumes that there are no always-takers (assumption 6) based on the JOBS II example. In JOBS II, neither deer nor always-taker was a likely ompliane option, sine study partiipants were prohibited from reeiving a dierent intervention ondition than the one that they were assigned to. However, unlike monotoniity, the assumption of having no always-takers is not ritial in estimating the CACE, and an be relaxed depending on the situation. Assumption 6. There are no always-takers. Under assumptions 4 and 6, the possible ompliane behaviour types (C i ) an be redued to { (omplier) if Di (1)=1 and D i (0)=0 C i = n (never-taker) if D i (1)=0 and D i (0)=0 Let C(t)={i C i =t} for t {; n}. The dierential average ausal eet of treatment assignment based on ompliane type an be dened as ITT t = [Y i (1;D i (1)) Y i (0;D i (0))]=N t (1) i C(t) where Y i (1;D i (1)) denotes the potential outome for individual i with treatment reeipt status D i when Z i =1, and Y i (0;D i (0)) denotes the potential outome for individual i with treatment reeipt status D i when Z i =0. N t is the number of individuals of ompliane type t. The ausal eet of treatment assignment annot be estimated for individual i, sine two potential outomes (that is, Y i (1;D i (1)) Y i (0;D i (0))) annot be jointly observed. However, the ausal eet of treatment assignment an be estimated at the average level. The average ausal eet of treatment assignment for ompliers (ITT =CACE) an be dened as CACE= 1 0 (2) where 1 denotes population mean potential outome for ompliers if Z =1, and 0 denotes population mean potential outome for ompliers if Z =0.

5 MODEL MISSPECIFICATION IN CACE ESTIMATION 3165 This study assumes that the average ausal eet of treatment assignment (ITT t ) does not vary aross dierent values of ovariates. Under the exlusion restrition, assumption 7 is not ritial in the estimation of CACE and an be relaxed depending on situations and researh questions. Assumption 7 (additivity). The average ausal eet of treatment assignment is onstant regardless of varying values of ovariates. The urrent study employs a maximum likelihood estimation approah, whih is known to be often more eient than the traditional IV (instrumental variable) approah in the estimation of CACE [4, 5]. Given that ompliane type C i annot be observed in the ontrol ondition, the observed-data likelihood funtion assuming a normally distributed outome is L( data) i {Z i=1;d i=0} i {Z i=0;d i=0} n f(y i 1n ; 2 ) i {Z i=1;d i=1} f(y i 1 ; 2 ) [ n f(y i 0n ; 2 )+ f(y i 0 ; 2 )] (3) where =( n ; ; 1n ; 1 ; 0n ; 0 ; 2 ) is the set of parameters in the model, and f(y i ; 2 ) denotes the probability density of a normal distribution with mean and variane 2. n is the proportion of never-takers in the population, and is the proportion of ompliers in the population. 1n denotes population mean potential outome for never-takers if Z =1, and 0n denotes population mean potential outome for never-takers if Z =0. By maximizing the likelihood in equation (3) with respet to the parameters of interest, ML estimates are obtained. The unknown ompliane status (C) in the ontrol ondition is handled as missing data via the EM algorithm [20 23]. and n (=1 ) are parameters that determine the distribution of C. The E-step omputes the expeted values of the omplete-data suient statistis given data y and urrent parameter estimates. The M-step omputes the omplete-data ML estimates with omplete-data suient statistis replaed by their estimates from the E-step. This proedure ontinues until it reahes optimal status. In the urrent study, ML-EM estimation of CACE was arried out by the Mplus program [24]. Parametri standard errors are omputed from the information matrix of the ML estimator using both the rst- and the seond-order derivatives under the assumption of normally distributed outomes. Based on equation (3), three diretly estimable population means an be expressed in terms of model parameters as 1n = n + n (4) 1 = + (5) 0 = n n + (6) where n orresponds to 0n, orresponds to 0, and 0 is the overall population mean potential outome if Z =0. n represents the average ausal eet of treatment assignment for never-takers (ITT n ), and represents the average ausal eet of treatment assignment for

6 3166 B. JO ompliers (ITT =CACE). Under the assumption of the exlusion restrition, n =0. Therefore, n is diretly identied as 1n from equation (4). From equations (4) and (6), an be identied as = 0 n 1n (7) From equations (5) and (7), an then be identied as = 1 0 n 1n = 1 0 (8) where assumption 5 exludes the possibility of a zero denominator (that is, 0). Under assumptions 1 to 6, the approximately unbiased estimator of the average ausal eet of treatment assignment for ompliers an then be dened as ˆ =y 1 y 0 p n y 1n = y 1 y 0 p p (9) where y 1 is the sample mean outome of the treatment group ompliers, y 0 is the sample mean outome of the ontrol group, y 1 is the sample mean outome of the treatment group, y 1n is the sample mean outome of the treatment group never-takers, and p is the proportion of ompliers in the treatment ondition (1 p =p n ) Bias mehanism in a misspeied model It is shown in the previous setion that the assumption of the exlusion restrition plays a ritial role in providing identiability in CACE models, given that ompliane information is missing in the ontrol ondition. However, assuming the exlusion restrition may ause bias in the CACE estimate, if the assumption does not hold. In line with Angrist et al. [1], this setion examines the bias mehanism in CACE estimation fousing on ompliane rate ( ) and the eet of treatment assignment on non-ompliers (ITT n = n ). If n is not zero, n annot be diretly identied as 1n. Instead, orret speiation of n from equation (4) is n = 1n n (10) Equation (10) shows that the estimator of n will be biased as muh as n,if n is misspeied as zero. For example, if true n is 1.0, true 1n is 1.2 and true n is 0.2, then the estimator of n will be fored to be 1.2 if the exlusion restrition is imposed (that is, n =0). The bias in the estimation of n is transferred to through equation (6). The orret speiation of from equations (4) and (6) is = 0 n 1n + n n (11) Equation (11) shows that the estimator of will be biased as muh as n n =,if n is misspeied as zero (ompare equations (7) and (11)). For example, if true is 1.5, = n =0:5, and true n is 0.2, then the estimator of will be fored to be 1.3 by imposing the exlusion restrition.

7 MODEL MISSPECIFICATION IN CACE ESTIMATION 3167 The bias in the estimation of n and due to misspeiation of n is then transferred to. The orret speiation of from equations (5) and (11) is = 1 0 n 1n n n (12) Equation (12) shows that the estimator of CACE ( ) will be biased as muh as n n =, if n is misspeied as zero. The magnitude of bias is aeted by n, and n. The bias in the CACE ( ) estimate inreases if the average ausal eet of treatment assignment for never-takers ( n ) inreases and ompliane rate ( ) dereases. In some situations, n may have the opposite diretion to that of CACE. Being assigned to the treatment ondition may have a negative psyhologial impat on never-takers, sine they failed to take the given treatment. The diretions of n and CACE are opposite in this ase, assuming that treatment has a positive impat on ompliers. Assume that true CACE is 0:6 and true n is 0.2, whih is about one-third of the magnitude of CACE. If =0:5, the CACE estimate will be biased as muh as 0:2 ( n n = =0:5 0:2=0:5) by imposing the exlusion restrition. Therefore, the estimated CACE will be around 0:4, whih is twothirds of the magnitude of true CACE. If =0:3, the CACE estimate will be biased as muh as 0:47 ( n n = =0:7 0:2=0:3) by imposing the exlusion restrition. In this ase, the estimated CACE will be around 0:13, whih is less than one-third of the magnitude of true CACE. This implies that if ompliane rate is very low, violation of the exlusion restrition assumption an ause a substantial bias in the CACE estimate even when the eet of treatment assignment on never-takers is trivial. In some situations, n may have the same diretion as CACE. Being assigned to the treatment ondition may have a positive psyhologial impat on never-takers, even though they deide not to reeive the treatment. The diretions of n and CACE are the same in this situation, assuming that treatment has a positive impat on ompliers. If true CACE is 0:6, true n is 0:2 and =0:5, the CACE estimate will be biased as muh as 0:2 ( n n = =0:5 0:2=0:5) by imposing the exlusion restrition. Therefore, the estimated CACE will be around 0:8, whih implies that the CACE estimate an be also exaggerated when the exlusion restrition is violated. The bias mehanism disussed in this setion an be generalized to situations where there are both never-takers and always-takers (see Appendix). However, note that the impat of ombined bias on the CACE estimate an be small when there are two types of nonompliers, although the exlusion restrition is substantially violated for both types of nonompliers, beause the diretion of bias an be dierent depending on the type of non-ompliane The role of ovariates in a misspeied model It is demonstrated in the previous setion that the size of bias in the CACE estimate due to violation of the exlusion restrition is aeted by the ompliane rate and the eet of treatment assignment on non-ompliers. Given that these fators are not easily ontrollable in pratie, the urrent study fouses on the use of ovariate information to redue the bias due to violation of the exlusion restrition. This setion demonstrates how ovariate information aets the sensitivity of the CACE estimate to violation of the exlusion restrition.

8 3168 B. JO Let i =0 and n i =1 if i C(n), and i =1 and n i =0 if i C(). Consider a ontinuous outome variable Y for individual i with ompliane status i and n i Y i = n n i + i + n n i Z i + i Z i + x i + i (13) where x is a vetor of pretreatment ovariates that predit both the outome measure y and ompliane status C. n + x i represents the potential mean outome for ontrol ondition never-takers with ovariates x i, and +x i represents the potential mean outome for ontrol ondition ompliers with ovariates x i. For simpliity, it is assumed that ovariate eets on outome is the same for ompliers and non-ompliers (that is, = = n ). n represents the average ausal eet of treatment assignment for never-takers (ITT n ), and represents the average ausal eet of treatment assignment for ompliers (ITT =CACE). i is a normally distributed residual with zero mean and variane 2. The logisti regression of on x is desribed as P( i =1 x i )= i P( i =0 x i )=1 i = ni logit( i )= x i (14) where i denotes the probability of being a omplier, ni denotes the probability of being a never-taker, 0 represents a logit interept, and 1 is a vetor of logit oeients. Note that the probability of being a omplier ( i ) varies depending on x i (equation (14)), but the average ausal eet of treatment assignment ( t ) does not vary aross dierent values of x i (equation (13)). Assumption 7 is tested in the JOBS II example through the likelihood ratio test between models with and without imposing the assumption (no signiant dierene was found in model t). Assume that there is only one binary ovariate X that predits both C and y. The observeddata likelihood in equation (3) an be modied as L( data) i {Z i=1;d i=0;x i=0} i {Z i=1;d i=0;x i=1} i {Z i=1;d i=1;x i=0} i {Z i=1;d i=1;x i=1} i {Z i=0;d i=0;x i=0} i {Z i=0;d i=0;x i=1} n;x=0 f(y i 1n;X=0 ; 2 ) n;x=1 f(y i 1n;X=1 ; 2 ) ;X=0 f(y i 1;X=0 ; 2 ) ;X=1 f(y i 1;X=1 ; 2 ) [ n;x=0 f(y i 0n;X=0 ; 2 )+ ;X=0 f(y i 0;X=0 ; 2 )] [ n;x=1 f(y i 0n;X=1 ; 2 )+ ;X=1 f(y i 0;X=1 ; 2 )] (15)

9 MODEL MISSPECIFICATION IN CACE ESTIMATION 3169 where the binary ovariate X has two values (X=0 or X=1). The proportions of ompliers and never-takers in the population vary aross dierent values of X. The population mean potential outomes for ompliers and never-takers vary aross dierent values of X. Based on equation (15), the average ausal eet of treatment assignment for ompliers adjusted for ovariate X an be dened as CACE= 1;X=0 0;X=0 = 1;X=1 0;X=1 (16) where 0;X=0 and 0;X=1 are not diretly estimable quantities. Based on equations (13) and (15), six diretly estimable population means an be expressed in terms of model parameters as 1n;X=0 = n + n (17) 1n;X=1 = n + n + (18) 1;X=0 = + (19) 1;X=1 = + + (20) 0;X=0 = n;x=0 n + ;X=0 (21) 0;X=1 = n;x=1 n + ;X=1 + (22) where n orresponds to 0n;X=0, and orresponds to 0;X=0. 0;X=0 is the population mean potential outome if Z=0 and X=0, and 0;X=1 is the population mean potential outome if Z=0 and X=1. Equations (17) to (22) show that there are two dierent soures of information that ontribute to the identiation of CACE: the exlusion restrition and the ovariate. Although both soures of information aet the CACE estimate simultaneously, how ovariate information moderates bias due to violation of the exlusion restrition an be better understood by looking at the two soures of information separately. First, n, and an be identied based on diretly estimable quantities relying on the exlusion restrition assumption, but ignoring the assoiation between ompliane and ovariate. Let us all these tentative estimators ˆ te n, ˆ te and ˆ te. Aording to the exlusion restrition assumption, n =0. Therefore, ˆ te n an be dened from equation (17) as ˆ te n =ˆ 1n;X=0 (23) where ˆ 1n;X=0 is an ML estimate of 1n;X=0. If the exlusion restrition holds, ˆ te n unbiased estimator of n. Based on equation (21), ˆ te is dened as ˆ te is an = ˆ 0;X=0 ˆ n;x=0 ˆ te n (24) ˆ ;X=0 where ˆ 0;X=0, ˆ n;x=0 and ˆ ;X=0 are ML estimates of 0;X=0, n;x=0 and ;X=0. ˆ te n in equation (23). is dened

10 3170 B. JO Then, ˆ te an be dened from equation (19) as ˆ te =ˆ 1;X=0 ˆ te (25) where ˆ 1;X=0 is an ML estimate of 1;X=0, and ˆ te is dened in equation (24). The estimator of an be dened based on observable quantities from equations (17) and (18) (or from (19) and (20)) as ˆ= ˆ 1n;X=1 ˆ 1n;X=0 (26) where ˆ 1n;X=1 and ˆ 1n;X=0 are ML estimates of 1n;X=1 and 1n;X=0. Seond, n, and an be identied based on diretly estimable quantities relying on the assoiation between ompliane and ovariate, but ignoring the exlusion restrition. Let us all these tentative estimators ˆ tx n, ˆ tx and ˆ tx. The key to the identiation of ˆ tx n, ˆ tx and ˆ tx is the presene of ovariate X, whih has non-zero 1 in equation (14) (that is, n;x=0 n;x=1 and ;X=0 ;X=1 ). From equations (21) and (22), ˆ tx n an be dened as ˆ tx n = ˆ ;X=0 ˆ 0;X=1 ˆ ;X=1 ˆ 0;X=0 ˆ ;X=0 ˆ ˆ ;X=0 ˆ ;X=1 (27) where ˆ ;X=0, ˆ ;X=1 and ˆ 0;X=1 are ML estimates of ;X=0, ;X=1 and 0;X=1. ˆ is dened in equation (26). From equation (21), ˆ tx an be dened as ˆ tx = ˆ 0;X=0 ˆ n;x=0 ˆ tx n (28) ˆ ;X=0 where ˆ tx n is dened in equation (27). From equation (19), ˆ tx an then be dened as ˆ tx =ˆ 1;X=0 ˆ tx (29) where ˆ tx is dened in equation (28). If there is no assoiation between ompliane and ovariate (that is, 1 =0), n;x=0 = n;x=1 and ;X=0 = ;X=1 in equations (21) and (22). In other words, ovariate X does not arry information to identify n and. In this ase, identiation of CACE is ompletely dependent on the assumption of the exlusion restrition. Therefore, the CACE estimate is the same as ˆ te, whih is an unbiased estimator of CACE if the exlusion restrition assumption holds. If the exlusion restrition is violated, ˆ te is a biased estimate of CACE, where the degree of bias depends on the size of true n and the ompliane rate (see bias mehanism in Setion 3.1). If the ovariate is a perfet preditor of ompliane, the CACE estimate is the same as ˆ tx, whih is an unbiased estimator of CACE regardless of whether the exlusion restrition holds. That is, the estimator of will not be biased due to the violation of the exlusion restrition, although the estimator of n an be still distorted due to the restrition in equations (17) and (18). Consequently, the CACE estimate will not be biased despite the model misspeiation. However, ovariates that are perfet preditors of ompliane usually do not exist in pratie, whih exludes the possibility of omplete elimination of bias based on ovariate information.

11 MODEL MISSPECIFICATION IN CACE ESTIMATION 3171 Coverage (Power) of the CACE estimate Coverage Power True ITTn Figure 1. Simulation: sensitivity of the CACE estimate to violation of the exlusion restrition assumption when there is no ovariate. In most situations, where the assoiation between ompliane and ovariates is neither zero nor perfet, both the exlusion restrition assumption and ovariate information aet the identiation of CACE. When the assumption of the exlusion restrition is violated, ovariate information plays a ritial role in reduing the bias in the CACE estimate. As the level of assoiation between ompliane and the ovariate inreases, the bias due to the violation of the exlusion restrition dereases, beause parameter estimates are adjusted more towards ˆ tx n, ˆ tx and ˆ tx, whih are not aeted by the exlusion restrition. Even when the assumption of the exlusion restrition holds, having ovariates that are good preditors of ompliane is important, beause this inreases the preision of the CACE estimate [25]. 4. SIMULATION STUDIES 4.1. CACE estimation with no ovariate This setion demonstrates the quality of the CACE estimate and statistial power when the exlusion restrition assumption is violated at dierent levels. The simulation results presented in Figure 1 and Table I are based on 500 repliations with a sample size of 500. The true values of key parameters were hosen based on the JOBS II Study example. Equal probability of treatment=ontrol assignment and 50 per ent ompliane rate were used as true values for all simulation settings in this study. Without ovariates, the ontinuous outome Y generated for CACE estimation an be desribed as Y i = n n i + i + n n i Z i + i Z i + i (30)

12 3172 B. JO Table I. Simulation: estimation of CACE with no ovariate (true CACE = 0:6, true n =1:0, true =1:5). ITT n CACE n True Estimate SE Estimate SE Estimate SE 0.0 0: : : : : : : Under the assumption of the exlusion restrition, ITT n ( n ) is xed at zero in the estimation of CACE. How seriously the exlusion restrition assumption is violated depends on the true value of ITT n, whih was set at various values in the data generation. Opposite diretions are hosen for the true values of CACE and ITT n in simulation settings based on JOBS II. The true treatment eet for ompliers (CACE) is 0:6, whih is approximately 0:6 in terms of eet size, assuming that derease in the outome is desirable (for example, depression in JOBS II). The true treatment eet for never-takers (ITT n ) ranges from 0:0 to0:6, whih is approximately 0:0 to 0:6 in terms of eet size, assuming that the positive value of ITT n represents a negative eet of the intervention on never-takers. The true residual variane ( 2 ) is 1:0, the true ontrol group omplier mean ( )is1:5, and the true ontrol group never-taker mean ( n )is1:0. Figure 1 shows the sensitivity of the CACE estimate to violation of the exlusion restrition. In this study, overage is dened as the proportion of repliations out of 500 repliations that are overed by the 95 per ent ondene interval. Power is dened as the proportion of repliations out of 500 repliations where the CACE estimate is signiantly dierent from zero (=0:05). Coverage and power at ITT n =0 show the quality of the CACE estimate when the exlusion restrition is not violated (overage=0:940, power=0:860). Coverage and power of the CACE estimate are still aeptable when the exlusion restrition is minimally violated (that is, ITT n =0:1). However, the violation of the exlusion restrition starts to aet the quality of the CACE estimate substantially as the size of ITT n inreases. When the absolute magnitude of ITT n is one-half that of CACE (that is, ITT n =0:3), overage is 0:624 and power is 0:308. When the absolute magnitude of ITT n is the same as that of CACE (that is, ITT n =0:6), overage dereases to 0:094. When a large eet size is expeted for ompliers in interventions, it will be realisti to assume that the size of ITT n will be relatively small and the impat of misspeiation also will be small. However, if a small CACE is expeted, and if it is questionable whether the exlusion restrition will hold, one needs to be more areful in interpreting the CACE estimate. As shown above, violation of the exlusion restrition may eliminate or double the CACE estimate. Table I shows the estimates of CACE, n and that deviate from true values as the exlusion restrition assumption is more severely violated. In Setion 3.1, it was demonstrated that the estimate of n will be biased as muh as n (ITT n ), the estimate of will be biased as muh as n n =, and the estimate of CACE will be biased as muh as n n = by

13 MODEL MISSPECIFICATION IN CACE ESTIMATION 3173 misspeifying ITT n as zero. Sine true is 0:5, the size of bias in the estimate of will be approximately n, and the size of bias in the CACE estimate will be approximately n in this simulation setting. It is shown in Table I that the bias in the estimates of CACE, n and inreases proportionally aording to the size of ITT n. The quality of the CACE estimate reported in Figure 1 and Table I will vary in dierent settings. That is, ompliane rate, and n values, and other available auxiliary information suh as from ovariates aet the sensitivity of the CACE estimate to violation of the exlusion restrition assumption. Compliane rate has a diret inuene on the quality of the CACE estimate; however, it is often very diult to ontrol ompliane behaviour of human partiipants. and n values also aet the quality of the CACE estimate. If outome distributions of ompliers and never-takers are distant from eah other, distinguishing two distributions will be relatively easier than when the distributions are lose to eah other. If the assumption of the exlusion restrition holds, preision in the CACE estimate will inrease as the distane between and n inreases [25]. However, if the assumption does not hold, systemati inuene of and n values annot be expeted, sine the estimates of and n will also be distorted, as shown in Table I. In addition, it is not pratial to depend on information from unknown outome distributions in the ontrol ondition. In ontrast, the assoiation between ompliane and ovariates an be estimated based on information from the treatment ondition without relying on the exlusion restrition. As a way to improve the quality of the CACE estimate, this study fouses on the use of ovariate information, whih is also a relatively ontrollable fator in randomized trial pratie CACE estimation with a ovariate This setion demonstrates how ovariate information aets the sensitivity of CACE estimation due to violation of the exlusion restrition. The simulation setting is the same as in the previous setion exept that a ovariate is added in the model. The model used for data generation and CACE estimation is desribed in equations (13) and (14). For simpliity, one ontinuous ovariate X is used in the simulation study, where X i N(0; 1). Both the outome and ompliane are regressed on the same ovariate X. The true diret eet of X on the outome () is 0:3. Two dierent levels of assoiation between ompliane and the ovariate ( 1 ) are onsidered in the simulation. The two assoiation levels studied are 0:5 and 0:3 in terms of the odds ratio (OR) in the logisti regression of on X. As desribed in equation (13), onstant eet of CACE that does not vary aross dierent values of X i is assumed. Figure 2 shows the sensitivity of the CACE estimate to the violation of the exlusion restrition when there is a ovariate. The assoiation between ompliane and the ovariate is 0:5 in terms of the odds ratio. Coverage and power at ITT n =0 show the quality of CACE estimate when the exlusion restrition is not violated (overage=0:964, power=0:948). Coverage and power remain more stable than in the model without ovariates (see Figure 1) as the size of ITT n inreases. When the absolute magnitude of ITT n is one-half that of CACE (that is, ITT n =0:3), overage is 0:742 and power is 0:470. When the absolute magnitude of ITT n is the same as that of CACE (that is, ITT n =0:6), overage is 0:192 and power is 0:106. Figure 3 shows the sensitivity of the CACE estimate to the violation of the exlusion restrition when there is a ovariate that is highly assoiated with ompliane (OR =0:3). Coverage and power at ITT n =0 show the quality of the CACE estimate when the exlusion restrition is not violated (overage=0:964, power=0:970). Coverage and power show

14 3174 B. JO Coverage (Power) of the CACE estimate Coverage Power True ITTn Figure 2. Simulation: sensitivity of the CACE estimate to violation of the exlusion restrition assumption when there is a ovariate with OR =0:5. Coverage (Power) of the CACE estimate Coverage Power True ITTn Figure 3. Simulation: sensitivity of the CACE estimate to violation of the exlusion restrition assumption when there is a ovariate with OR =0:3. notieable improvement ompared to Figure 2 (that is, OR=0:5). The improvement is even more substantial ompared to the model without ovariates (see Figure 1). When the absolute magnitude of ITT n is one-half of that of CACE (that is, ITT n =0:3), overage is 0:834 and power is 0:694. When the absolute magnitude of ITT n is the same as that of CACE (that is, ITT n =0:6), overage is 0:396 and power is 0:278.

15 MODEL MISSPECIFICATION IN CACE ESTIMATION 3175 Table II. Simulation: estimation of CACE with a ovariate (OR =0:3, true CACE= 0:6, true n =1:0, true =1:5). ITT n CACE n True Estimate SE Estimate SE Estimate SE 0.0 0: : : : : : : Table II shows the estimates of CACE, n and when there is a ovariate that is highly assoiated with ompliane (OR=0.3). It is shown in Table II that the bias in the estimates of n and inreases at a slower rate than in the model without ovariates (see Table I) as the size of ITT n inreases. Consequently, the CACE estimate is muh less biased in the model with a ovariate (Table II) ompared to the CACE estimate without ovariates (Table I). For example, when the absolute magnitude of ITT n is one-half that of CACE (that is, ITT n =0:3), the size of the average CACE estimate in the model without ovariates is about two-thirds that in the model with a ovariate. 5. APPLICATION TO JOBS II This setion demonstrates the estimation of CACE in pratie when the assumption of the exlusion restrition is possibly violated. In CACE analysis examples using JOBS II, the level of depression six months after the intervention is used as the outome (depression at T6). Depression was measured with a subsale of 11 items based on the Hopkins symptom heklist [26]. The present study foused on the high-risk status group based on previous studies [10, 27], whih indiated that the job searh intervention had its primary impat on high-risk respondents. Risk sore was omputed based on risk variables (depression, nanial strain and assertiveness) in the sreening data [27] prediting depressive symptoms at followup. A total sample size of 486 was analysed in this study after deleting ases that had missingness in ovariates and outome variables. Among 486 individuals, 328 are in the treatment ondition and 158 are in the ontrol ondition. The response rate at follow-up six months after the intervention was 87 per ent. The variables used in the urrent study are shown in Table III. In addition to demographi information, all partiipants in JOBS II were asked before randomization how highly they expeted they would omply with intervention ativities, whih was intended to better predit atual ompliane. Table IV shows the responses from treatment group individuals to the question How likely or unlikely is it that you would partiipate in the one-week job seminar if you were oered the opportunity during the next three weeks? Sine ompliane behaviour is atually observed in the treatment ondition, expeted ompliane an be ompared to atual ompliane (that is, showing up). It is shown that the atual ompliane rate is sharply disriminated between the individuals who answered that it is

16 3176 B. JO Table III. JOBS II: sample statistis. Variable Control group No-shows (Z =1) Shows (Z =1) Mean SD Mean SD Mean SD Depression at T Depression at T Expeted ompliane (0=1) Age in years Shool grade ompleted Assertiveness at T Not married (0=1) Eonomi hardship at T Non-White (0=1) Male (0=1) Table IV. JOBS II: expeted versus atual ompliane (Z =1). Expeted ompliane (n) No Atual ompliane Extremely unlikely (1) 1 0 Very unlikely (1) 0 1 Unlikely (2) 2 0 Neither nor (26) Likely (93) Very likely (93) Extremely likely (112) Yes extremely likely that they would attend the intervention seminars and those who answered dierently. A dihotomous variable (expeted ompliane: 0=low, 1=high) is reated based on these two ategories of individuals. Table V shows the results from CACE estimation with various pretreatment ovariates inluding expeted ompliane. A slightly dierent analysis using JOBS II has been previously presented by Little and Yau [5]. The eet size of the dierential treatment eet estimate is alulated by dividing the outome dierene in treatment and ontrol ondition means after treatment by the square root of the variane pooled aross the ontrol and treatment groups. The CACE estimate is signiant and has a meaningful level of eet size (CACE= 0:338, eet size=0:466). The level of depression was lower for ompliers in the intervention ondition ompared to that for ontrol ondition individuals who ould have omplied if they have had been assigned to the intervention ondition. represents the diret eet of ovariates on the outome, where =( 1 ; 2 ; 3 ; 4 ; 5 ; 6 ; 7 ; 8 ; 9 ). Eonomi hardship before the intervention was found to be a signiant preditor of the outome. Depression level dereased more among individuals who did not have eonomi hardship. The same ovariates that are used as preditors of the outome are also used as preditors of ompliane. For nine ovariates, logit oeients 1 =( 11 ; 12 ; 13 ; 14 ; 15 ; 16 ; 17 ; 18 ; 19 ). Table V shows that expeted ompliane has high assoiation with ompliane ( 12 =1:264). The estimated odds of being a omplier are 3:54 times higher for individuals who expeted

17 MODEL MISSPECIFICATION IN CACE ESTIMATION 3177 Table V. JOBS II: CACE estimation with various ovariates. Parameter Estimate SE CACE 0: n Y on x Depression at T0 ( 1) Expeted ompliane ( 2) 0: Age in years ( 3) 0: Shool grade omp ( 4) 0: Assertiveness at T0 ( 5) 0: Not married ( 6) 0: Eonomi hardship ( 7) Non-White ( 8) Male ( 9) 0: on x Interept ( 0) 4: Depression at T0 ( 11) 0: Expeted ompliane ( 12) Age in years ( 13) Shool grade omp ( 14) Assertiveness at T0 ( 15) 0: Not married ( 16) Eonomi hardship ( 17) 0: Non-White ( 18) 0: Male ( 19) Table VI. JOBS II: sensitivity of the CACE estimate to ovariate information. Present ovariates CACE estimate SE Eet size No ovariates 0: Depression T0 0: Depression T0, expeted ompliane 0: Depression T0, expeted ompliane, age 0: Depression T0, expeted ompliane, age, grade 0: All nine ovariates 0: that it was extremely likely for them to attend the job seminar than for individuals who expeted a lower possibility of attending the seminar. In addition to expeted ompliane, several ovariates were found to be signiant preditors of ompliane behaviour. Individuals omplied more if they were older, more eduated less assertive and not married. The summary results from six CACE analyses are shown in Table VI. The same CACE model used in Table V is used, but dierent numbers of ovariates are present in six analyses. In eah analysis, the same ovariates that are used as preditors of the outome are also used as preditors of ompliane. The results demonstrate the sensitivity of the CACE estimate in JOBS II to ovariate information.

18 3178 B. JO When there are no ovariates present in the model, the magnitude of the CACE estimate is about 70 per ent of that when all nine ovariates are present in the model (see Table V for full results). Adding baseline depression (depression T0), whih is the weakest preditor of ompliane, has little impat on the CACE estimate. However, adding ovariates that are strong preditors of ompliane results in a notieable hange in the CACE estimate. Given randomization of intervention assignment, the utuation shown in Table VI an be onsidered as substantial, implying possible violation of the exlusion restrition. As demonstrated in previous setions, the magnitude of bias in the CACE estimate due to violation of the exlusion restrition may be redued if ovariates are good preditors of ompliane. However, ovariate information would not ompletely eliminate bias due to violation of the exlusion restrition. Therefore, the CACE estimate adjusted for ovariates should be onsidered as less biased, but not ompletely unbiased. 6. DISCUSSION This study examined the bias mehanism in the estimation of CACE when the assumption of the exlusion restrition is violated. The magnitude of bias is aeted mainly by ompliane rate and the size of treatment assignment eet on non-ompliers (ITT n ). The magnitude of bias in the CACE estimate inreases as the size of ITT n inreases and ompliane rate dereases. If the ompliane rate is very low, violation of the exlusion restrition assumption an ause substantial bias in the CACE estimate even when the eet of treatment assignment on non-ompliers is trivial. When the exlusion restrition is violated, the CACE estimate not only an be understated, but also an be exaggerated, depending on how the assignment of treatment aets non-ompliers. Compliane rate and the eet of treatment assignment on non-ompliers diretly aet bias in the CACE estimate; however, these fators are not easily ontrollable in pratie. The urrent study foused on the use of ovariate information to test and to redue bias due to violation of the exlusion restrition. It was demonstrated that bias in the CACE estimate due to model misspeiation an be redued substantially by inluding ovariates that are good preditors of ompliane. In JOBS II, one kind of ovariate information olleted before the intervention assignment was about study partiipants expeted level of ompliane, whih turned out to be a good preditor of atual ompliane. To inrease the preision in lassifying individuals based on ompliane types, one an use more aggressive ways of monitoring ompliane (for example, by blood test in pharmaeutial trials), or have a run-in period before the main trial to selet potential ompliers [28]. However, these methods are not always appliable, espeially in soial-behavioural eld experiments. Given that, readily observable ovariates suh as expeted ompliane, motivation, baseline outome measures and bakground variables are valuable soures of information that an be aquired at a relatively low ost. The limitation of using ovariate information without relaxing the exlusion restrition is that the bias due to violation of the assumption annot be ompletely eliminated unless ovariates are perfet preditors of ompliane. To test the ultimate magnitude of the bias and to estimate the assignment eet of treatment on non-ompliers, the assumption of the exlusion restrition needs to be relaxed. Without assuming the exlusion restrition, the identiability of CACE models relies on auxiliary information suh as from ovariates [7]

19 MODEL MISSPECIFICATION IN CACE ESTIMATION 3179 and proper priors [4, 6]. Pratiality of the CACE estimation methods relying on auxiliary information is still under investigation. In examining the bias mehanism in CACE estimation, this study assumed an additive eet of treatment assignment (that is, no interation eets) for both ompliers and non-ompliers. However, in pratie the eet of treatment assignment may vary depending on ovariate values. The presene of interation eets further ompliates the bias mehanism. Also, ovariate information may have weaker and inonsistent eets on treatment eet estimates. More investigation is needed to identify the bias mehanism with interation eets. This study examined the bias mehanism in CACE estimation fousing on intervention settings where the intensity of ompliane is the same among ompliers. However, in pratie, ompliane often has varying intensity [29, 30]. Further study is needed to larify the relationship between ausal eet estimates and model misspeiation in more general situations. APPENDIX: BIAS MECHANISM IN A MISSPECIFIED MODEL WITH NEVER-TAKERS AND ALWAYS-TAKERS If there are both never-takers and always-takers, ompliane type an be observed only for always-takers among individuals assigned to the ontrol ondition, and only for never-takers among individuals assigned to the treatment ondition. Given that, the observed-data likelihood funtion is L( data) i {Z i=1;d i=0} i {Z i=0;d i=1} i {Z i=1;d i=1} i {Z i=0;d i=0} n f(y i 1n ; 2 ) a f(y i 0a ; 2 ) [ f(y i 1 ; 2 )+ a f(y i 1a ; 2 )] [ n f(y i 0n ; 2 )+ f(y i 0 ; 2 )] (A1) where a is the proportion of always-takers in the population, 1a denotes population mean potential outome for always-takers if Z =1, and 0a denotes population mean potential outome for always-takers if Z =0. Based on equation (A1), four observable population means an be expressed in terms of model parameters as 1n = n + n 0a = a 1 = n ( n + n )+ ( + )+ a ( a + a ) 0 = n n + + a a (A2) (A3) (A4) (A5) where a is the average ausal eet of treatment assignment for always-takers.

20 3180 B. JO If the exlusion restrition assumption holds, n =0 and a =0. Therefore, an be identied from equations (A2), (A3) and (A5) as = 0 n 1n a 0a Then, (CACE) an be identied from equations (A2), (A3), (A4) and (A6) as (A6) = 1 0 (A7) If the assumption of the exlusion restrition is violated for both never-takers and alwaystakers, n 0 and a 0. In this ase, the orret speiation of from equations (A2), (A3) and (A5) is = 0 n 1n a 0a + n n (A8) Then, the orret speiation of (CACE) from equations (A2), (A3), (A4) and (A8) is = 1 0 n n a a (A9) Equation (A9) shows that the estimator of CACE ( ) will be biased as muh as n n =, if n is misspeied as zero, and as muh as a a =,if a is misspeied as zero. If the exlusion restrition is violated for both never-takers and always-takers, CACE ( ) will be biased as muh as n n = + a a =. ACKNOWLEDGEMENTS This study was supported by NIMH grant P30 MH38330, Mihigan Prevention Researh Center at the Institute for Soial Researh, Rihard H. Prie, P.I., University of Mihigan. This study was also supported by grant KO2 AA from NIAAA and by NIMH grant P50 MH38725, Epidemiologi Prevention Center for Early Risk Behaviors, Philip Leaf, P.I., and grant RO1 DA11796 from NIDA, Follow-up of Two Preventive Intervention Trials, Niholas Ialongo, P.I., Department of Mental Hygiene, Bloomberg Shool of Publi Health, Johns Hopkins University. I would like to thank Amiram Vinokur for providing data and stimulating input. I also thank Guido Imbens, Keisuke Hirano and Bengt Muthen for helpful advie. REFERENCES 1. Angrist JD, Imbens GW, Rubin DB. Identiation of ausal eets using instrumental variables. Journal of the Amerian Statistial Assoiation 1996; 91: Bloom HS. Aounting for no-shows in experimental evaluation designs. Evaluation Review 1984; 8: Imbens GW, Angrist J. Identiation and estimation of loal average treatment eets. Eonometria 1994; 62: Imbens GW, Rubin DB. Bayesian inferene for ausal eets in randomized experiments with non-ompliane. Annals of Statistis 1997; 25: Little RJA, Yau LHY. Statistial tehniques for analyzing data from prevention trials: treatment of no-shows using Rubin s ausal model. Psyhologial Methods 1998; 3: Hirano K, Imbens GW, Rubin DB, Zhou XH. Assessing the eet of an inuenza vaine in an enouragement design. Biostatistis 2000; 1: Jo B. Estimation of intervention eets with non-ompliane: alternative model speiations. Journal of Eduational and Behavioral Statistis (in press).