Statistical Treatment Choice Based on. Asymmetric Minimax Regret Criteria

Transcription

1 Statistical Treatment Coice Based on Asymmetric Minimax Regret Criteria Aleksey Tetenov y Deartment of Economics ortwestern University ovember 5, 007 (JOB MARKET PAPER) Abstract Tis aer studies te roblem of treatment coice between a status quo treatment wit a known outcome distribution and an innovation wose outcomes are observed only in a nite samle randomized exeriment. I evaluate statistical decision rules, wic are functions tat ma samle outcomes into te lanner s treatment coice for te oulation, based on regret, wic is te exected welfare loss due to assigning inferior treatments. I extend revious work tat alied te minimax regret criterion to treatment coice roblems by considering decision criteria tat asymmetrically treat Tye I regret (due to mistakenly coosing an inferior new treatment) and Tye II regret (due to mistakenly rejecting a suerior innovation). I derive exact nite samle solutions to tese roblems for exeriments wit normal, Bernoulli, or bounded distributions of individual outcomes. In conclusion, I discuss aroaces to te roblem for oter classes of distributions. Along te way, te aer comares asymmetric minimax regret criteria wit statistical decision rules based on classical yotesis tests. I tank Cuck Manski, Elie Tamer, Ben Jones, and seminar articiants at ortwestern for very elful comments. y Deartment of Economics, ortwestern University, 00 Seridan Rd, Evanston, IL 6008, USA. address: a-tetenov@nortwestern.edu

2 Introduction Consider a lanner wo as to coose wic one of two mutually exclusive treatments sould be assigned to members of a oulation. One treatment is te status quo, wose e ects are well known. Te oter is a romising innovation, wose exact e ects ave yet to be determined. Te treatments in question may be, for examle, two alternative drugs or teraies for a medical condition, or two di erent unemloyment assistance rograms. Suose tat a randomized clinical trial or some oter exeriment will be conducted and its results will be used to coose wic treatment oulation members will receive. Te lanner faces two roblems. First, se as to know wat exeriment (in articular, wat samle size) sould be cosen to get a su ciently accurate estimate of te treatment e ect. Second, se as to select ow treatment coices will be determined based on te statistical evidence obtained from te exeriment. Often, treatment coice is based on te results of a statistical yotesis test, wic is constructed to kee te robability of mistakenly assigning an inferior innovation (a Tye I error) below a seci ed level (usually.05 or.0). Ten, te samle size is selected to obtain a ig robability (usually.8 or.9) tat te innovation will be cosen if its ositive e ect exceeds some value of interest. Following Wald s (950) formulation of statistical decision teory, I analyze te erformance of alternative statistical metods based on teir exected welfare over di erent realizations of te samling rocess, rater tan just teir robabilities of error. In articular, I continue a recent line of work advocating and investigating treatment coice rocedures tat minimize maximum regret by Manski (004, 005, 007a, 007b), Hirano and Porter (006), Stoye (006, 007a, 007b) and Sclag (006, 007). Regret is te di erence between te maximum welfare tat could be acieved given full knowledge of te e ects of bot treatments (by assigning te treatment tat is actually better) and te exected welfare of treatment coices based on exerimental outcomes. Te latter is smaller, because exerimental outcomes generally do not allow te decision maker to coose te best treatment 00 ercent of te time. Tis aer s main dearture from revious literature on te subject is asymmetric consideration of Tye I regret (due to mistakenly using an inferior new treatment) and Tye II regret (due to missing out on using a suerior innovation). Te ersistent use in treatment coice roblems of

3 te yotesis testing aroac, wic allows Tye II errors to occur wit iger robability tan Tye I errors, suggests tat many decision makers want to lace te burden of roof on te new treatment. Most do so by selecting a low yotesis test level, suc as = :05. It is not clear wat rinciles, besides convention, are tere to guide te selection of yotesis test level for te circumstances of a articular decision roblem. Values of maximum Tye I and maximum Tye II regret of a statistical rocedure could rovide te decision maker wit more relevant caracteristics of its erformance tan te traditional yotesis testing measures (test level and ower), since regret takes into account bot te robability of making an error and its economic magnitude. How to balance Tye I and Tye II regret in a articular roblem is u to te decision maker. In tis aer I consider tree criteria. First, te traditional minimax regret criterion gives equal consideration to Tye I and Tye II regret. It seeks to minimize te larger of te two, tus minimax regret solutions ave equal maximum Tye I and Tye II regret. Te second criterion is minimax regret wit an asymmetric linear reference-deendent welfare function. Tis criterion gives larger weigt to maximum Tye I regret, tus te maximum Tye II regret of asymmetric minimax regret solutions is larger tan teir maximum Tye I regret by a given factor. Wen te treatment e ect estimate is normally distributed, yotesis test based solutions wit a given level corresond to asymmetric minimax regret solutions for some asymmetry factor K () for any samle size and variance. In a sense, te minimax regret criterion wit an asymmetric welfare function rovides a decision-teoretic rationalization of yotesis tests tat is based on exected welfare. Te tird criterion is limited Tye I regret. Many decision makers face te roblem of making treatment coices based on existing statistical evidence, witout any control over its samle size and recision. Symmetric and asymmetric minimax regret, as well as yotesis testing, often lead to solutions wose maximum Tye I regret is roortional to te standard error of te estimate of average treatment e ect. Tis may not be aealing to decision makers rimarily interested in "safety" of new treatments, wic I interret as low Tye I regret. Te limited Tye I regret criterion seeks to minimize maximum regret subject to an exlicit constraint tat maximum Tye I regret sould not exceed a given accetable level. Tis aroac guarantees limited exected welfare losses due to Tye I errors regardless of te decision rocess tat underlies samle size selection. 3

4 Instead of looking at maximum regret values, a Bayesian decision maker would assert a subjective robability distribution over te set of feasible treatment outcome distributions, use samle realizations to derive an udated osterior robability distribution, and maximize exected welfare wit regard to tat osterior distribution (wic is equivalent to minimizing exected regret). Unfortunately, in many situations decision makers do not ave any information tat would form a reasonable basis for asserting a rior distribution. In grou decision making, members of te grou may disagree in teir rior beliefs. Tese roblems lead to frequent use of conventional rior distributions in alied Bayesian analysis. Bayesian treatment coice based on a conventional rior distribution, rater tan on a subjective distribution re ecting te decision maker s rior information, does not ave a clear economic justi cation. Decision making based on maximum regret is a conservative aroac to dealing wit te lack of reasonable rior beliefs, since maximum regret is te sar uer bound on exected regret for decision makers wit any rior distributions. Te aer roceeds in te following order. Section exosits te decision-teoretic formulation of te roblem and te criteria used to address it. In section 3, I consider a simle but instructive case were te exeriment generates a normally distributed random variable wit known or bounded variance. I analyze conventional treatment coice rules based on yotesis testing and samle size coice based on ower analysis in ligt of teir maximum regret and comare tem wit minimax regret, asymmetric minimax regret and limited Tye I regret solutions. Section 4 analyzes treatment coice wen treatment outcomes are eiter binary or bounded random variables. Exact mimimax regret results were obtained for tese roblems by Stoye (006) and Sclag (007). I extend teir results to derive asymmetric minimax regret and limited Tye I regret solutions using a di erent tecnique. I also demonstrate tat te minimax-regret solution roosed by tese autors for bounded outcomes does not minimize maximum regret if te decision maker can lace an informative uer bound on te variance of te outcome distribution, wic is te case in many alications. In te concluding section 5, I discuss te use of aroximations, bounds, and numerical metods for roblems tat do not yet ave convenient analytical solutions and illustrate teir erformance in a yotetical clinical trial roblem wit rare dangerous side e ects. 4

5 Statistical treatment rules, welfare and regret Te basic setting is te same as in Manski (004, 005) and in Manski and Tetenov (007). Te lanner s roblem is to assign members of a large oulation to one of two available treatments t T; T = f0; g. Let t = 0 denote te status quo treatment and t = te innovation. Eac member j of te oulation, denoted J, as a resonse function y j (t) describing tat individual s otential outcome under eac treatment t. Te oulation is a robability sace (J; ; P ) and te robability distribution P [y ()] of te random function y () describes treatment resonse across te oulation. Te oulation is "large," in te sense tat J is uncountable and P (j) = 0; j J. Te lanner does not know te robability distribution P, but knows tat it belongs to a set of feasible treatment resonse distributions fp ; g. will be called te state of te world. I assume tat average treatment outcomes E [y (t)] are nite for all t and. All oulation members are observationally identical to te lanner, tus te lanner s treatment assignment decision can be fully described by an action a A; A = [0; ], were a denotes te roortion of te target oulation assigned by te lanner to te innovative treatment t =. Proortion a, ten, is assigned to te status quo treatment t = 0. I assume tat fractional treatment assignment (0 < a < ) is carried out randomly. I consider lanners wose welfare from taking action a in state of te world is te average treatment outcome across te oulation: U (a; ) ( a) E [y (0)] + a E [y ()] = E [y (0)] + a. Te second line exresses te welfare function in terms of te average treatment e ect E [y ()] E [y (0)], wic is te rimary oulation statistic of interest to te lanner. Te lanner conducts an exeriment and observes its outcome a random vector X X. Te robability distribution of X deends on te unknown state of te world and will be denoted by Q. A (random) function maing feasible exerimental outcomes from X into actions from 5

6 A will be called a statistical treatment rule (or simly a decision rule). Te action cosen by a lanner wit statistical treatment rule wen X is observed will be denoted by (X). Te set of all suc functions (feasible statistical treatment rules) will be labeled D. I follow Wald s (950) aroac and evaluate alternative statistical treatment rules based on te exected welfare tey yield across reeated samles in eac state of te world. If te lanner s welfare function is U (a; ), ten te exected welfare from using statistical treatment rule in state of te world equals () W (; ) Z XX U ( (X) ; ) dq = E [y (0)] + E [ (X)], were E [(X)] denotes R XX (X) dq. Statistical treatment rule dominates if W ( ; ) W ( ; ) for all wit strict inequality at least for one value of. Statistical treatment rule is said to be admissible if tere does not exist any D tat dominates, oterwise is called inadmissible. Te analysis of tis aer is based on a normalization of te exected welfare called regret. Te regret of statistical treatment rule is te di erence between te igest exected welfare acievable by any feasible statistical treatment rule in state of te world and te exected welfare of statistical treatment rule : R (; ) su W 0 ; W (; ). 0 D Te igest welfare in state of te world is acieved by statistical treatment rule (X) = j > 0j tat selects te otimal (in state ) treatment regardless of exerimental outcomes. Te regret function, ten, equals 8 () R (; ) = W ; >< ( E [ (X)]) if > 0 W (; ) = >: E [ (X)] if 0. Te regret of a statistical treatment rule, tus, is te roduct of te robability of making an error (assigning an individual to te wrong treatment) and te magnitude of te welfare loss su ered 6

7 from tat error.. Treatment coice based on yotesis testing Te most common framework used for treatment coice between a status quo treatment and an innovation is yotesis testing. Tyically, te researcer oses two mutually exclusive statistical yoteses a null yotesis H 0 : 0; tat te innovation is no better tan te status quo treatment, and an alternative yotesis H : > 0; tat te innovation is suerior. If te null yotesis is rejected, ten treatment t = is assigned to te oulation. If it is not rejected, te status quo treatment t = 0 is assigned. Rejecting te null yotesis wen it is, in fact, true (assigning an inferior innovation t = to te oulation) is called a Tye I error. ot rejecting te null yotesis wen it is, in fact, false (assigning te status quo treatment instead of te suerior innovation) is called a Tye II error. Hyotesis testing rocedures are designed to ave a certain signi cance level, wic is te robability of making a Tye I error (te maximum robability over states of te world tat fall under te null yotesis). Te signi cance level (also called -level) is usually set at conventional values = 0:05 or = 0:0. Te robability of not making a Tye II error (assigning an innovation wen it is suerior to te status quo treatment) is called te ower of te test. Te ower of te test is usually calculated for some seci c value > 0. Te samle size of an exeriment is selected so tat a yotesis test wit a cosen signi cance level would ave te desired ower (tyically :8 or :9) at.. Treatment coice based on maximum regret Savage (95) introduced te criterion of minimizing maximum di erence between otential and realized welfare (now called regret) in a review of Wald (950) as a clari cation of Wald s minimax rincile. Under te minimax regret criterion, statistical treatment rule 0 is referred to if max R 0 ; < max R (; ). A lanner wo accets te minimax-regret criterion sould select a statistical treatment rule 7

8 tat satis es (3) M arg min max D R (; ) and select a samle size suc tat te maximum regret max R ( M; ) is accetable..3 Asymmetric reference-deendent welfare As a way to exress te lanner s desire to lace te burden of roof on te innovation, I will also consider asymmetric reference-deendent welfare functions. For an asymmetry coe cient K > 0, let te welfare function U A(K) be linear in te average treatment outcomes wit te same sloe as U above te reference oint E [y (0)] and a K times steeer sloe below te reference oint. Formally, de ne U A(K) as: 8 >< (U (a; ) E [y (0)]) if U (a; ) > E [y (0)], U A(K) (a; ) E [y (0)] + >: K (U (a; ) E [y (0)]) if U (a; ) E [y (0)], 8 >< a if > 0, = E [y (0)] + >: K a if 0. Te exected welfare for tis kinked linear welfare function equals (4) W A(K) (; ) Z U A(K) ( (X) ; ) dq XX 8 >< E [ (X)] if > 0, = E [y (0)] + >: K E [ (X)] if 0. Ordinal relationsis between exected welfare of two statistical decision rules do not deend on te asymmetry factor K > 0. For any ; D and : W ( ; ) T W ( ; ) () W A(K) ( ; ) T W A(K) ( ; ). Tus, te set of admissible statistical treatment rules is te same for all asymmetrical linear welfare functions (4) and for te standard linear welfare (). 8

9 Te regret function for exected welfare (4) equals R A(K) (; ) su W A(K) 0 ; W A(K) (; ) = = 0 D 8 >< >: 8 >< >: ( E [ (X)]) if > 0; K E [ (X)] if 0, R (; ) if > 0; KR (; ) if 0. Te only di erence between tis regret function and te regret function for standard linear welfare () is te factor K for 0. Maximum regret under te asymmetrical welfare function can be exressed troug te regret function for linear welfare as max R A(K) (; ) = max K R T ye I () ; R T ye II (), were R T ye I () max R (; ) : 0 is te maximum Tye I regret (maximum regret across states of te world in wic te innovation is inferior) under te linear welfare function and R T ye II () max R (; ) : >0 is te maximum Tye II regret (maximum regret across states of te world in wic te innovation is suerior). Te names Tye I and Tye II regret are given in analogy to Tye I and Tye II errors in yotesis testing. Tye I regret is te welfare loss due to Tye I errors, wile Tye II regret is te welfare loss due to Tye II errors under te null yotesis H 0 : 0. Since te asymmetry factor K does not a ect admissibility, I will only consider asymmetrical welfare functions indirectly, by solving te weigted minimax regret roblem (5) min D max K R T ye I () ; R T ye II () 9

10 for te linear exected welfare (). In roblem (5) te lanner gives K times greater weigt to regret from Tye I errors..4 Treatment rules wit limited Tye I regret Using minimax regret treatment rules may ose a articular roblem for decision makers wo do not ave a coice over te recision of statistical evidence on wic tey ave to base teir decisions. Consider an extreme examle. Suose tat a medical regulatory agency (te Food and Drug Administration in te United States or te Euroean Agency for te Evaluation of Medicinal Products) as to coose weter to arove an innovative treatment for a common disease Z. Te status quo medical treatment for disease Z as a roven record of curing te disease wit robability.5. Proonents of te innovative treatment rovide te regulator wit results of a clinical trial in wic ve randomly selected atients wit disease Z received te innovative treatment and in all ve cases te disease as been cured. Te minimax regret and yotesis testing (at.05 level) statistical treatment rules bot rescribe tat all oulation members sould be assigned to te innovation based on tis exerimental result. Bot rules, owever, imly muc iger exected welfare losses if te innovation is inferior tan clinical trials of usual size; te decision maker may not nd tis accetable. For decision makers wo are rimarily concerned wit welfare loss due to mistakenly assigning an inferior innovation and cannot control te recision of exerimental evidence, I roose te limited Tye I regret criterion: (6) L(r) arg min max D R (; ) ; s:t: R T ye I () r: Te criterion selects a statistical treatment rule wit minimal maximum regret, subject to an exlicit constraint tat Tye I regret (regret from mistakenly assigning te innovation) sould not exceed a given value r. Tis criterion is similar to te classical yotesis testing criterion in tat bot aim to limit te damage from Tye I errors. Limited Tye I regret, owever, exresses te desired level of "safety" in terms of te maximum ossible welfare loss from Tye I errors, rater tan just te maximum robability of making tem. 0

11 3 Simle normal exeriment I will rst consider a very simle exeriment wose outcome X R is a scalar normally distributed random variable wit unknown mean R and known variance : X ( ; ). Wile X is a scalar, it need not originate from an exeriment wit samle size one. For examle, X could be a samle average X = P n i= x i of indeendent random observations. If observations (x ; :::; x ) all ave a normal distribution ; 0, ten X is a su cient statistic for (x ; :::; x ) wit variance = 0. Comaring single normal draw exeriments wit di erent values of, ten, is equivalent to comaring exeriments wit di erent samle sizes. More imortantly, te robability distribution of many commonly used statistical estimators of average treatment e ect converges to a normal distribution as samle size grows ^ D! (0; 0 ). Ten te asymtotic distribution of ^ is said to be ( ; 0 ). Heuristically, studying exeriments wit a single normally distributed outcome for di erent values of will suggest wat e ect di erent tyes of decision rules and samle sizes ave on regret in more general settings. It follows from te results of Karlin and Rubin (956, Teorem ) tat if te distribution of X exibits te monotone likeliood ratio roerty (wic is true for normal and binomial distributions) and te welfare function is (), ten te class of monotone decision rules 8 >< T; (X) >: X > T X = T ; [0; ]; T R, 0 X < T is essentially comlete (for any decision rule 0 tere exists T; suc tat W 0 ; W ( T; ; ) in all states of te world). Since te robability of observing X = T equals zero for te normal distribution, it follows tat a smaller class of tresold decision rules T (X) jx > T j ; T R is also essentially comlete. Tus, considering oter rules is not necessary in tis roblem.

12 Given tat X is normally distributed, te regret of a tresold decision rule T in state of te world equals 8 >< R( T ; ) = >: T P (X T ) = P (X > T ) = T if > 0, if 0, wic is te robability of making an incorrect decision multilied by j j, te magnitude of te loss incurred from te mistake. denotes te standard normal cumulative distribution function. Maximum Tye I and Tye II regret equal (7) R T ye I ( T ) = max : 0 R T ye II ( T ) = max : >0 T T = max 0 = max >0 T T : ; Te rigt-and equalities are derived by substituting =. Tese functions ave nite ositive values for every T R. Since R ( T ; ) = R ( T ; ), it follows tat R T ye II ( T ) = R T ye I ( T ). Lemma sows tat te decision maker faces a trade o between maximum Tye I and maximum Tye II regret. Higer tresold values imly lower Tye I regret, but necessarily iger Tye II regret. Lemma a) R T ye I ( T ) is a continuous, strictly decreasing function of T; lim R T ye I ( T ) = and T! lim R T ye I ( T ) = 0; T! b) R T ye II ( T ) is a continuous, strictly increasing function of T, lim R T ye II ( T ) = 0 and T! lim R T ye II ( T ) = : T! Figure dislays te maximum Tye I and maximum Tye II regret as functions of te decision rule tresold T. Te scale of bot axes is normalized by. Te maximum regret max R ( T ; ) = max R T ye I ( T ) ; R T ye II ( T ) is minimized wen R T ye I ( T ) = R T ye II ( T ), wic aens only at T = 0. Te minimax regret treatment rule in tis roblem is 0. Tis is sometimes called te lug-in rule (a lug-in rule takes te estimated value of te average treatment e ect and assigns

13 treatments as if it were te true value). Similarly, te minimax regret statistical treatment rule under asymmetric welfare function W A(K) is uniquely caracterized by te equation K R T ye I ( T ) = R T ye II ( T ). By substituting rigt-and exressions from (7), tis caracterization can be rewritten as K max 0 T T = max >0. Since only one value of T solves te equation for a given K, te tresold of te minimax regret statistical treatment rule is roortional to. A conventional one-sided yotesis test wit signi cance level rejects te null yotesis ( 0) and assigns te innovative treatment if X > ( ). Tis critical value guarantees tat te robability of a Tye I error does not exceed for any 0. Since X normal distribution, as a standard P X > ( ) X = P ( ) = ( ) ( ) = : = Te statistical treatment rule based on results of a yotesis test wit level is a tresold rule H() wit tresold H () ( ). For a given test level, te tresold T is roortional to te standard error. Tus a yotesis test based treatment rule can be rationalized as a solution to an asymmetrical minimax regret roblem wit asymmetry factor K () = max f (H () = )g >0 = f ( H () =)g max 0 max ( ) >0 max f ( ( ))g. 0 K () is te ratio of maximum Tye II to maximum Tye I regret of te yotesis test based decision rule, wic deends only on te test level. In tis normal model, te corresondence 3

14 Maximum Tye I regret Asymmetric Max Tye I regret, K=3 Maximum Tye II regret Maximum regret / σ Tresold T / σ Figure : Maximum Tye I and Tye II regret as functions of te decision rule tresold Minimax regret rule Hyotesis test rule (α=.05) Regret R(δ,γ) / σ θ γ / σ Figure : Regret functions of minimax regret and yotesis test based decision rules. 4

15 Test signi cance level Tresold Max Tye I regret Max Tye II regret K () = :5 (minimax regret) T = 0 :7 :7 = :5 T = :6745 :0608 :374 6:5 = : T = :8 :0877 : :5 = :05 T = :645 :00878 :837 0:4 = :05 T = :96 : :06 79:9 = :0 T = :36 :00304 :64 969:6 Table : Maximum Tye I and Tye II regret of statistical treatment rules induced by yotesis tests based on a normally distributed estimate wit variance. between a yotesis test based rule wit level and an asymmetric minimax regret rule wit level K () does not deend on te standard error of, and tus on samle size. Tis feature is seci c to te normal examle. For examle, if X is a binomial variable, ten yotesis test based rules wit te same level corresond to di erent asymmetric minimax regret treatment rules for di erent samle sizes. Table rovides maximum Tye I and II regret values and te asymmetry factors corresonding to commonly used yotesis test levels. Decision rules based on te one-sided = :05 level yotesis test minimize maximum regret for decision makers wo lace 0 times greater weigt on Tye I regret tan on Tye II regret. Decision rules based on = :0 level tests are minimax regret for decision makers wo lace nearly 970 times greater weigt on Tye I regret. Te trade o between Tye I and Tye II regret is markedly di erent from te trade o between raw Tye I and Tye II error rates (an = :05 level test as a 95% maximum robability of Tye II error, wic is 9 times iger tan te maximum robability of te test s Tye I error). Figure comares te regret functions of te minimax regret treatment rule 0 and te treatment rule H(:05) induced by a yotesis test wit signi cance level = :05 over a range of feasible values of. Te scale of bot axes is normalized by. Te maximum regret of te yotesis test rule is aroximately :837; wic is nearly ve times iger tan te maximum regret of te minimax regret treatment rule (aroximately :7). Te yotesis test rule as lower regret over 0, but it can only acieve it by greatly increasing te regret for > 0. Te greatest exected welfare losses from using a yotesis test rule occur wen te innovative treatment is moderately e ective. 5

16 3. Limited Tye I regret Comared to te minimax regret rule, yotesis testing wit signi cance level = :05 as a clear advantage in lower regret over 0. Tis can make minimax regret unattractive for decision makers wo are more concerned about negative consequences of acceting a otentially inferior new treatment tan about its otential foregone bene ts. I do not tink, owever, tat yotesis testing ractices adequately address suc concerns. It is common to see tests wit te same signi cance level = :05 alied to treatment e ect estimates wit di erent variance and samle size. Wile suc tests always limit te robability of Tye I error to :05, te maximum Tye I regret ( :008) is roortional to. Many decision makers, no doubt, would like to sensibly adjust te test level to te circumstances of a articular roblem. Considering maximum Tye I regret of a tresold rule instead of its te maximum robability of Tye I error simli es tis task. Table rovides maximum Tye I and Tye II regret values for tresold rules corresonding to yotesis tests wit di erent signi cance levels. Instead of imosing a limit on te robability of Tye I errors, te decision maker could directly imose a limit r on maximum accetable Tye I regret and use te limited Tye I regret criterion (6). Te limited Tye I statistical treatment rule coincides wit te minimax regret rule if its maximum regret :7 does not exceed r. Oterwise, it selects a treatment rule wit te smallest tresold T > 0 tat ensures tat maximum Tye I regret does not exceed r. If te estimator as ig variance, reducing maximum Tye I regret comes at a rice of iger Tye II regret. For examle, if te decision maker nds tat a tresold value T = :645 is required to bring maximum Tye I regret to an accetable level r, se as to accet tat suc statistical treatment rule imlies a maximum Tye II regret tat is over 00 times larger tan r. Tis underscores te imortance of using estimators of treatment e ect wit low variance (ig samle size), wic allow te decision maker to attain accetable maximum Tye I regret wit statistical treatment rules tat ave lower Tye II regret. 6

17 3. Samle size selection I will illustrate samle size selection based on maximum regret by comaring it wit one of te conventional metods. Te International Conference on Harmonisation formulated "Guideline E9: Statistical Princiles for Clinical Trials" (998), adoted by te US Food and Drug Administration and te Euroean Agency for te Evaluation of Medicinal Products. Te guideline rovides researcers wit te values of Tye I and Tye II errors tyically used for yotesis testing and samle size selection in clinical trials. For yotesis testing, te limit on te robability of Tye I errors is traditionally set at 5% or less. Te trial samle size is tyically selected to limit te robability of Tye II errors to 0-0% for a minimal value of te treatment e ect tat is deemed to ave "clinical relevance" or at te anticiated value of te e ect of te innovative treatment. Suose tat a researcer considers bearable te loss of ublic welfare due to a 0% robability tat er innovative treatment could be rejected if its actual treatment e ect equals > 0. Following te convention, se selects te samle size for wic te variance of X equals, were satis es te condition tat X will fall under te 5% yotesis test tresold H (:05) = (:95) wit robability 0% if = : P X H (:05) j = = (:95) = :, = (:95) (:) = :96. Te value of regret tat te researcer nds accetable at = tus equals :. Tis rocedure does not make aarent to te researcer tat a muc larger welfare loss will be su ered at a twice smaller value of = :46 :5, were te regret function acieves its maximum of :837 = :86. Consider now ow te samle size would di er if it were selected by te researcer wit an exlicit objective tat maximum regret sould equal : in two scenarios. First, suose tat te researcer lanning te exeriment as to take for granted tat te decision making will be carried out using a 5% yotesis test rule. SInce its maximum regret equals :837, se would select 7

18 samle size suc tat :837 = : = : : :96 = :837 :837 = :35, wic imlies samle size tat is over 8 times larger tan te one selected by ower calculations in te examle above. In a second scenario, suose tat te researcer as control over treatment assignment and lans to use te minimax-regret decision rule 0. Since te maximum regret of te minimax-regret decision rule equals :7, te samle size sould be suc tat :7 = : = :7, wic imlies samle size tat is almost 3 times smaller tan te one selected by ower calculations. 3.3 ormally distributed outcomes wit unknown variance So far in tis section I ave assumed tat te lanner knows te variance of te normally distributed average treatment e ect estimate X. Suose now, instead, tat te data (x ; :::; x ) consists of indeendent normally distributed observations wit unknown mean and unknown variance. Let te set of feasible states of te world be : R; ; ; were > 0 and < and let : R; = denote te subset of states of te world wit te igest feasible outcome variance. Let X P i= x i be te samle mean and S P i= x i X te samle variance. It is well known (cf. Berger, 985) tat te air X; S is a su cient statistic for (x ; :::; x ), tus only decision rules tat are functions of X and S need to be considered. It turns out, owever, tat decision 8

19 rules satisfying criteria based on maximum Tye I and Tye II regret could often be found witin a smaller class of tresold decision rules tat deend only on te samle mean X. Proosition Let T X > T be a tresold statistical treatment rule suc tat T satis es te condition (8) max (0;T ) (T ) max f (T )g ; T jt j ten a) maximum Tye I and Tye II regret of T over te set is te same as over te set : max R ( T ; ) = max R ( T ; ) ; : 0 : 0 max R ( T ; ) = max R ( T ; ), : >0 : >0 b) tere is no statistical treatment rule 0 X; S tat as bot lower maximum Tye I regret and lower maximum Tye II regret tan T. Condition (8) ensures tat te tresold decision attains maximum Tye I and maximum Tye II regret on te subset. If it is not satis ed, te maximum Tye I or maximum Tye II regret of T could be iger on te set tan on, ten tere maybe exists a non-tresold decision rule tat as bot lower Tye I and lower Tye II regret tan T. It follows from Proosition tat tresold decision rules tat satisfy minimax regret, asymmetric minimax regret, and limited Tye I regret criteria for outcomes wit xed variance (set of feasible states of te world ) also satisfy te corresonding criteria for outcomes wit bounded variance (set of feasible states ) if teir tresold values satisfy condition (8). Te range of tresolds for wic condition (8) olds deends on te ratio. For =, it olds if jt j :5. In te oosite extreme case wen!, it olds if jt j :. 9

20 4 Exact statistical treatment rules for binary and bounded outcomes Exact solutions to te minimax regret and limited tye I regret roblems and exact maximum regret values are available wen te data X consists of indeendent random outcomes of treatment t =, rovided tat te outcomes are binary or ave bounded values. I will rst consider te case of binary outcomes and ten its extension to outcomes wit bounded values. 4. Binary outcomes Let te treatment outcomes of te innovative treatment t = be binary, w.l.o.g. let y () f0; g, and let te known average outcome of te status quo treatment t = 0 equal 0 E [y (0)] (0; ). Let te set of feasible robability distributions of y () be a set of Bernoulli distribution wit means [a; b] ; 0 a < 0 < b (if 0 is outside of te interval [a; b], ten te treatment coice roblem is trivial). Te exerimental data consists of indeendent random outcomes (x ; :::; x ), eac aving a Bernoulli distribution wit mean. Te sum of outcomes X = P n i= x i as a binomial distribution wit arameters and. X is a su cient statistic for (x ; :::; x ), so it is su cient to consider statistical treatment rules tat are functions of X. It follows from te results of Karlin and Rubin (956, Teorems and 4) tat monotone statistical treatment rules 8 >< T; (X) = >: X > T X = T 0 X < T ; T f0; :::; g; [0; ] are admissible and form an essentially comlete class, tus it is su cient to consider only monotone rules. Te regret of a monotone rule T; equals 8 >< R ( T ; ; ) = >: ( B (T; ; ) + P ) B (n; ; ) ( T <n!) B (T; ; ) + P B (n; ; ) T <n if > 0, if 0, 0

21 were B (n; ; ) denotes te binomial robability density function wit arameters and and 0. It will be convenient to use a one-dimensional index for monotone rules D ( T; ) T + ( ). Tere is a one to one corresondence between index values D [0; + ] and te set of all distinct monotone decision rules. D = 0 corresonds to te decision rule tat assigns all oulation members to te innovation, no matter wat te exerimental outcomes are. D = + corresonds to te most conservative decision rule tat always assigns te status quo treatment. Lemma 3 establises roerties of maximum Tye I and Tye II regret of monotone statistical treatment rules for binomially distributed X tat lead to simle caracterisations of minimax regret, asymmetric minimax regret, and limited Tye I regret rules. As before, maximum Tye I regret is R T ye I () max R (; ) and maximum Tye II regret is R T ye II () max R (; ). : [a; 0 ] : ( 0 ;b] Lemma 3 If X as a binomial distribution, ten a) R T ye I () is a continuous and strictly decreasing function of D () wit R T ye I () = 0 for D () = +. b) R T ye II () is a continuous and strictly increasing function of D () wit R T ye II () = 0 for D () = 0. It follows from lemma 3 tat tere is a unique value of D ( M ) suc tat R T ye I ( M ) = R T ye II ( M ). M is te minimax regret treatment rule. Wile its caracterisation is imlicit, monotonicity and continuity of te maximum Tye I and Tye II regret as functions of D () makes comutation very straigtforward. Te same caracterisation of te minimax regret treatment rule for [0; ] was derived by Stoye (006, Proosition (iii)) using game teoretic metods. Likewise, tere is a unique value D A(K) suc tat K RT ye I A(K) = RT ye II A(K). A(K) is te minimax regret statistical treatment rule for asymmetric reference deendent welfare function W A(K). Limited Tye I regret statistical treatment rule wit Tye I regret tresold r is also easily caracterized. If r max R ( M; ), ten tere is a unique value D L(r) suc tat R T ye I L(r) = r, ten L(r) is te limited Tye I regret treatment rule. If r < max R ( M; ), ten te Tye I regret constraint is not binding and te limited Tye I regret treatment rule is te same as te minimax regret treatment rule M. Te following roosition derives te exact large samle limit of maximum regret of minimax-

22 regret statistical treatment rules. Unlike in te normal case covered in Section 3, te minimax-regret rule in te Bernoulli case does not generally coincide wit te lug-in rule: P X > 0 : In large samles, owever, te di erence between M and P as little e ect on maximum regret. Proosition 4 sows tat as samle size grows, te maximum regret of minimax regret rules and lug-in rules (normalized by ) converge to te same limit. Tat limit is te same as minimax regret in a roblem wit normally distributed outcomes wit xed variance 0 ( 0 ). Proosition 4 lim! s 0 ( 0 ) max R ( P ; ) = lim! s 0 ( 0 ) max R ( M; ) = max [ ( )] :7. >0 4. Bounded outcomes ow consider a more general setting. Let te outcomes of treatment t = be bounded variables y () [0; ]. Let 0 E [y (0)] (0; ) denote te known average treatment outcome of te status quo treatment t = 0. Let fp ; g be te set of robability distributions P [y ()] tat te lanner considers feasible. Assume tat E [y ()] [a; b] ; 0 a < 0 < b. Also, let fp ; B g denote te set of all Bernoulli distributions wit E [y ()] [a; b] and assume tat B. Te tecnique outlined below relies on including all te Bernoulli distributions in te feasible set. Sclag (007) roosed an elegant tecnique, wic e calls te binomial average, tat extends statistical treatment rules de ned for samles of Bernoulli outcomes to samles of bounded outcomes. Te resulting statistical treatment rules inerit imortant roerties of teir Bernoulli ancestors. Let : f0; :::; g! [0; ] be a statistical treatment rule de ned for te sum of i.i.d. Bernoulli distributed outcomes (as in te revious subsection). Let X = (x ; :::; x ) be an i.i.d. samle of bounded random variables wit unknown distribution P [y ()] and let Z = (z ; :::; z ) be a samle of i.i.d. uniform (0; ) random variables indeendent of X. Ten te binomial average

23 extension of is de ned as X (X) EZ [z k x k ]. k=0 Verbally, tis extension can be described as a simle rocess: a) randomly relace eac bounded observation x i [0; ] wit a Bernoulli observation ~x i = wit robability x i and wit ~x i = 0 wit robability x i, b) aly statistical treatment rule to (~x ; :::; ~x ). Te random variables [z k x k ] ; k = 0; :::; are i.i.d. Bernoulli wit exectation E [y ()], tus P k=0 [z k x k ] as a Binomial distribution wit arameters and E [y ()]. For any state of te world, let be te state of te world in wic P [y ()] is a Bernoulli distribution wit te same mean E [y ()]. Ten E ( ~ ) = E () and R( ~ ; ) = R (; ). Te regret of a binomial average treatment rule ~ in state of te world is te same as te regret of in a Bernoulli state of te world wit te same mean treatment outcomes. It follows tat maximum Tye I (II) regret of ~ in te roblem wit bounded outcomes ( ) is equal to maximum Tye I (II) regret of in te roblem wit Bernoulli outcomes ( B ). If statistical treatment rule satis es some decision criterion based on maximum Tye I and maximum Tye II regret for te feasible set of Bernoulli outcome distributions, ten its binomial average extension ~ satis es te same criterion for te feasible set of bounded outcome distributions. Suose, for examle, tat M minimizes maximum regret for Bernoulli distributions. Suose tere was a treatment rule ~ 0 for bounded distributions tat ad lower maximum regret tan ~ M. Ten 0 would ave to ave lower maximum regret over B tan M, wic would imly tat M does not minimize maximum regret for te roblem wit Bernoulli distributions. Binomial average extension yields exact minimax regret, asymmetric minimax regret and limited Tye I regret statistical treatment rules if te set of feasible outcome distributions includes te set of Bernoulli outcome distributions wit te same means B. In many alications, owever, te lanner knows tat Bernoulli outcome distributions are not feasible. If te outcome variable is annual income of a articiant in a job training rogram, te lanner may assume not only tat te variable is bounded, but also tat it s variance is muc smaller tan te variance of a Bernoulli distribution wit te same mean. If Bernoulli outcome distributions are excluded, ten binomial average based treatment rules may not be otimal. Te following roosition sows tat a lug-in 3

24 statistical treatment rule P X x i > 0 i= as lower asymtotic maximum regret tan a binomial average extension of M, a minimax regret statistical treatment rule in te Bernoulli case. Proosition 5 Let 0 = E [y (0)] and let fp ; g be te set of feasible robability distributions of y () suc tat E (y () E [y ()]) <, were < 0 ( 0 ). Let (x ; :::; x ) be i.i.d. random outcomes of treatment t =. Ten su R ( P ; ) max [ ( )] + o (). >0 Maximum regret of binomial average extension ~ M is by design te same as te maximum regret of te minimax regret treatment rule M in te Bernoulli case. As long as for some > 0; contains distributions wit all ossible means in a -neigborood of 0 8 [ 0 ; 0 + ] ; 9 : E [y ()] =, te results of roosition 4 aly and lim max R(~ M ; ) = 0 ( 0 ) max [ ( )] > max [ ( )].! >0 >0 Tus, for large enoug, max R(~ M ; ) > su R ( P ; ). Tis underscores te imortance of lacing aroriate restrictions on te set of feasible treatment outcome distributions before looking for minimax regret or asymmetric maximum regret based treatment rules. 5 Evaluating regret using aroximations and bounds In conclusion, I would like to discuss metods for dealing wit statistical roblems wic do not ave neat nite samle solutions suc as described in te revious sections and give an examle illustrating teir roerties. I will restrict attention to te case wen te data consists of i.i.d. observations (x ; :::; x ) suc tat E [x i ] =, were E [y ()] E [y (0)] is te average 4

25 treatment e ect. For many sets of feasible distributions of fx i g, tere aren t roven comlete class teorems tat justify restricting attention to a small class of decision rules. Considering all feasible statistical treatment rules tat are functions of (x ; :::; x ) can be roibitively di cult, but rogress can be made by considering a suitable subset of feasible decision rules. Based on teir su ciency in an idealized roblem wit normally distributed outcomes considered in Section 3, te class of tresold decision rules T X > T based on te samle mean X P i= x i is a reasonable and tractable candidate class of statistical treatment rules to consider. Te regret of a tresold decision rule T equals 8 >< R( T ; ) = >: P ( X T ) if > 0, P ( X > T ) if 0. To evaluate maximum Tye I and Tye II regret of T, R T ye I ( T ) = su 0 R T ye II ( T ) = su >0 ( ( su P ( X > T ) : = ) su P ( X T ) : = te lanner needs to know, for eac value of, te range of feasible robabilities tat te samle mean X exceeds te tresold T. ote tat for eac, P ( X > T ) is a non-increasing function of T and P ( X T ) is non-decreasing. It follows tat R T ye I ( T ) is non-increasing and R T ye II ( T ) is non-decreasing in T, tus solutions to minimax regret, asymmetric minimax regret, and limited Tye I regret roblems can be easily found if te researcer as a way to evaluate R T ye I ( T ) and ) ; ; R T ye II ( T ). Te roblem of evaluating P X Q T for distributions of xi tat do not yield a convenient closed-form exression is well studied in statistics. I will consider tree main aroaces: brute force calculation or simulation, asymtotic aroximation, and large deviation bounds. Brute force calculation or simulation Te main callenge for calculation or simulation is in selecting a nite set of feasible distributions tat reliably aroximates su : = P ( X T ) or P ( X > T ) for di erent values of. For some distributions (e.g. for discrete distributions su : = wit small nite suort) suc a set is easily constructed by creating a "grid" of distributions 5

26 wit di erent arameter values. In nonarametric roblems, owever, it may be di cult to construct a nite set of distributions tat will be certain to reliably aroximate su : = P ( X T ) or P ( X > T ) for eac. If an insu ciently ric set of distributions is cosen, te aroximation su : = will be lower tan actual maximum regret. Asymtotic aroximation Wit te knowledge of E [x i ] R and V [x i ] R, te lanner can use te asymtotic normal aroximation P ( X T )! (T ). To evaluate maximum Tye I and Tye II regret of a tresold decision rule it is su cient to know minimum and maximum feasible variance for eac feasible value of. ormal aroximations of tail robabilities of X could be eiter iger or lower tan te actual values, tus aroximate values of maximum Tye I/II regret could also be eiter above or below actual values. Large deviation bounds Tere are a number of inequalities for tail robabilities of te distribution of samle mean X. Using tese inequalities allows te statistician to construct nite samle uer bounds on maximum Tye I and Tye II regret. Unlike asymtotic aroximations, bounds constructed using large deviation inequalities are guaranteed not to be lower tan actual maximum Tye I/II regret values, wic may be useful for conservative decision making. Te simlest large deviation bound is given by te one-sided Cebysev s inequality, wic requires only tat x 0 is ave bounded variance: T < ) P ( X T ) T > ) P ( X > T ), + (T ). + (T ) If outcome variables are bounded x i [a; b], ten Hoe ding s exonential inequality (963, 6

27 Teorem ) alies to te tail robabilities of X: T < 8 ) P ( X < T ) ex : 8 < T > ) P ( X > T ) ex :! 9 = b a (T ) ;,! 9 = b a (T ) ;. Hoe ding s inequality was used by Manski (004) to comute bounds on maximum regret of lug-in (emirical success) treatment rules. If a feasible distribution as nite absolute tird moment E jx i j 3 R, ten bounds on P X T could be derived from te Berry-Esseen inequality: P X T (z) min C 0 ; C ( + jzj) 3 3 ; were z (T ). Lowest roven values for te constants C 0 and C are C 0 0:7975 (van Beek, 97) and C 3 (Paditz, 989). For large enoug samle sizes, te Berry-Esseen inequality could sow tat te tail robabilities are arbitrarily close to teir normal aroximation, wic is signi cantly smaller tan te Cebysev s and Hoe ding s bounds. 5. A numerical examle I will illustrate ow te di erent metods of evaluating maximum regret of tresold rules may erform in ractice on a simle examle insired by te roblem of rare side e ects in clinical trials. Let te average outcome of te status quo treatment t = 0 be E [y (0)] = :5 (outcome values refer to individual welfare of clinical outcomes). Suose tat a new treatment as been assigned to = 000 randomly selected atients. Te treatment as tree otential outcomes: y () = and y () = 0 corresond to te ositive and negative outcomes of te treatment on te condition tat it is intended to treat, wile y () = 00 corresonds to a rare, dangerous side e ect. Te set of feasible treatment outcome distributions includes all robability distributions wit te suort f 00; 0; g tat ave a limited robability of te rare side e ect P [y () = 00] 000. Let X be te samle average of te 000 trial outcomes of te new treatment. First, let s consider ow well te di erent metods aroximate te regret of a lug-in statistical 7

28 Maximum Tye I/II regret Regret Large deviation bound umerical evaluation ormal aroximation Average treatment effect θ γ Figure 3: Evaluation of maximum regret of te lug-in (T = :5) statistical tretment rule Large deviation bound umerical evaluation ormal aroximation Decision rule tresold T Figure 4: Maximum Tye I and Tye II regret aroximations for a range of tresold statistical treatment rules. 8

29 treatment rule P X > :5, wic assigns te oulation to new treatment if it outerforms te status quo treatment in te trial by any margin. Figure 3 dislays te maximum regret of P for a range of feasible values of te average treatment e ect E [y ()] E [y (0)]. Tere are multile feasible outcome distributions wit te same, so te lines reresent te maximum arroximated regret among tose distributions. Figure 4 sows maximum Tye I and Tye II regret aroximations for tresold decision rules wit tresolds ranging from T = :45 to T = :55. Te to lines sow te best uer bounds on maximum regret derived from large deviation bounds. Tat is, te best of te bounds derived from Cebysev s, Hoe ding s, or Berry-Esseen inequalities. Eac inequality is alied to all feasible values of distribution moments for a given. Cebysev s inequality rovides te smallest bounds in tis examle desite fairly large samle size because some of te feasible outcome distributions ave large range [ 00; ] and large tird moments. It rovides an uer bound of.0508 for bot maximum Tye I ( 0) and maximum Tye II ( > 0) regret. Te lower dotted lines sow maximum regret comuted using te normal aroximation to te distribution of X based on te feasible values for te variance of outcome distributions. Te normal aroximation suggests tat bot maximum Tye I and maximum Tye II regret of P equal to.073. Te tick solid lines in Figures 3 and 4 sow te maximum regret evaluated numerically. Te set of feasible distributions in tis roblem is simle enoug (two-dimensional and continuous) to be reliably aroximated by a nite set of distributions. For tis examle, te robabilities P X :5 and corresonding regret values were evaluated on a grid of 60,000 distributions. Tese calculations sow tat maximum Tye I regret of te lug-in rule equals.06, wile te maximum Tye II regret equals.005. Figure 4 sows tat among tresold decision rules, minimax regret is attained by te decision rule wit tresold T = :5; rater tan by te lug-in rule, and its maximum regret equals.030. In tis examle, te large deviation bounds on maximum regret are muc iger tan its actual values, wile normal aroximations are signi cantly lower. Bot of tem suggest tat te lug-in decision rule minimizes maximum regret, even toug its maximum regret is % iger tan te minimum attainable by a di erent tresold decision rule. Te di erence between tese aroximations and actual maximum regret resents a bigger roblem for te selection of 9