Decision Oriented Bayesian Design of Experiments

Transcription

1 Decision Oriented Bayesian Design of Experiments Farminder S. Anand*, Jay H. Lee**, Matthew J. Realff*** *School of Chemical & Biomoleclar Engineering Georgia Institte of echnology, Atlanta, GA 3332 USA (el: ; farminder.anand@ chbe.gatech.ed). **( jay.lee@chbe.gatech.ed) *** ( matthew.realff@chbe.gatech.ed)} Abstract: Experimental design is a fndamental problem in science and engineering. raditional Design of Experiment (DOE) approaches focs on minimization of variance. In this work, we propose a new decision-oriented DOE approach, which takes into accont how the generated data, and sbseqently the developed model, will be sed in decision making. By doing so, the variance will be distribted in a manner sch that its impact on the targeted decision making will be minimal. Or reslts show that the new decision-oriented experiment design approach significantly otperforms the standard D-optimal design approach. he new design method shold be a valable tool when experiments are condcted for the prpose of making R&D decisions. Keywords: Decision making, optimal experiment design. 1. INRODUCION Design of experiments (DOE) as a field has evolved over the period of last few decades. Its importance has grown significantly becase of the increasing need to redce the resorce reqirement for achieving the target. he targets historically perceived by the scientists in performing experiments have been driven towards nderstanding the nderlying phenomenon or estimating the parameters. Conseqently, the traditional DOE tools have been geared towards imization of some measre of information or towards the minimization of the variance in the parameter estimates. It is or opinion that this way of thinking over a long period of time has led the field to lose sight on the ltimate prpose of experiments in many applications. If one looks back into the history of the evoltion of design of experiments one finds the answers in Bernardo (1979): Scientists typically does not have, nor can be normally expected to have, a clear idea of the tility of his reslts. An alternative is to design an experiment to imize the expected information to be gained from it. Bernardo (1979), frther goes on proving that any f (fnction of the parameters, ), in informational theoretical terms is garbling of. Hence follows the conclsion that imization of information of is better than imizing information on f. his practice, while seeming logical, does not directly address the intended prpose of the experiments in many engineering applications. oday mch of indstrial research is driven by investment decisions, i.e., experiments are condcted with a specific objective in mind. For example, experiments can be condcted to aid decisions for the imization of revene fnction when investigating a new process or for the selection of a few processes among the large alternatives. In sch scenarios, following the traditional rote for design of experiments may be significantly sboptimal. 2. BACKGROUND raditionally there have been two major classes for the design of experiments (DOE) approaches: Classical approach and Bayesian approach. Historically, Classical DOE approaches like the factorial design have been more poplar de to the comptational complexities of the Bayesian approach. Bt recent developments in sampling techniqes sch as Markov Chain Monte Carlo (MCMC) (Kass (1998), Cowles (1996)) have rejvenated the interest in the Bayesian approaches. In addition, the Bayesian approaches provide an added advantage of enabling the designer to incorporate the prior expert opinions. Hence, we will focs on the Bayesian approaches for design of experiments from here on. o elaborate on the traditional Bayesian design strategies, we wold follow Chaloner s (1995) approach, as it does jstice to the inherent decision aspect hidden in the Bayesian approach. he idea of Bayesian DOE has evolved from information acqisition concepts in decision theory. Raiffa (1961) presented a decision theoretic approach for optimal information acqisition strategy sing Expected Vale of Information (EVOI) approach for investment decision problems. EVOI is defined as the expected difference between the expected posterior and prior tility, if one is to acqire information. Lindley (1956) introdced his seminal work on the se of Shannon information as a measre of information provided by an experiment. Following this, several athors (Stone (1959), DeGroot (1962) and Bernardo (1979)) presented a decision theoretic approach to experimental design, which was basically the imization of EVOI with the tility fnction being replaced by Shannon information.

2 Consider a tility fnction (U), optimal decision () nder posterior distribtion, design matrix (), parameters () and observations (Y). Application of Lindley s EVOI imization approach reslts in the imization of expected pre-posterior tility as the expected vale of the prior tility fnction is constant. he optimal expected preposterior tility is given in (1). Fig.1 demonstates how (1) can be solved nmerically. Based on a given design () and the prior distribtion of the parameters ( prior ), potential observations (Y) are fond via Monte Carlo simlation. For each of these potential realizations, posterior estimates (and covariances) for the parameters are obtained ( Posterior ) and corresponding to these posterior parameter estimates optimal decision variable () is estimated. Next step is to calclate the posterior expected tility(u) vale corresponding to each of these potential realizations (Y). he average of the posterior tility vales for each potential realization (Y) gives a tility of the design (). he design that imises this average tlity vale is the optimal design. U ( ) = U (,, Y, ) p( Y, ) p( Y ) y Θ * ddy (1) Fig. 1. Demonstrating the calclation of optimal design based on Lindley s EVOI concept. Now, if one considers the Shannon information as the tility fnction, as sggested by Lindley, the above simplifies to (2). As both Fig. 2 and (2) show the calclations become mch more tangible as the step drops ot. U ( ) = log{ p Θ ( Y, ) } p( Y, ) y * ddy (2) prior Y 1 Y 2 Y k Y n-1 Y n Posterior Evalate Utility he rest of the Bayesian DOE methodologies, follow a similar line as (2) with some small changes to the tility fnction. In a broad sense, there exist three categories of Bayesian DOE approaches. First is the information imization approach, which consists of imizing the Kllback - Leibler distance between the prior and the posterior distribtion. his approach consists of D-optimal and Ds-optimal designs. he second category is the set of designs, where the objective is to obtain a point estimate of the parameter vales. his category consists of A-optimal and C-optimal designs. he third category is the mini type of designs, where the objective is to minimize the imm Fig. 2. Demonstrating the calclation of optimal design based on Shannon information criterion. possible variance for all the linear combinations of the parameters nder consideration. hese varios designs are frther explained in details as follows: D - Optimal: Maximize information gain for the parameters (Uses Kllback-Leibler distance between the prior and posterior distribtion as a measre of gain in information). Ds-optimality: Maximize gain in Shannon information of (= S ), where S is a known constant vector. A- Optimal: he objective of the experiment is to obtain a point estimate of the parameters. A design is chosen to imize the following tility fnction: U( ) = ( ˆ) ( ˆ A ) p( y, ) ddy (3) Here A is a symmetric nonnegative definite matrix. his design minimizes expected sqared error of loss for estimating C or Minimizing sqare error of predicting at C, where C is not necessarily a fixed and a distribtion is specified on it. C- Optimal: Special case of A-optimality, where C is a constant. E- Optimal: It is a mini approach for variance. he imm posterior variance of all possible normalized linear combinations of parameter estimates is minimized. An, E- optimal design minimizes: sp ( c R) c ( + = ωλ + R) (4) c = ω prior Y 1 Y 2 Y k Y n-1 Y n Posterior Evalate Utility G- Optimal: Closely related to E-optimal deign is G-optimal 1 design, which minimizes sp x D x ( + R) x. An eqivalence theorem [see Atkinson (1992)] states that continos G-optimal designs are nmerically identical to a corresponding continos D-optimal design. It is important to note that, among the above mentioned designs, D-, Ds- A- and C-optimal design have a tility fnction, which jstifies its decision theoretic sense. On the other hand, E- and G-optimal designs thogh are considered

3 Bayesian design don t have any decision-theoretic sense, Chaloner (1995). he rest of the docment is strctred as follows: Section 3 discsses in more detail the setp for the decision oriented design, section 4 presents the nmerical reslts, and section 5 concldes the paper. 3. SELECION/REJECION DESIGN As elaborated in the previos section the traditional design criterions either try to imize the information gain or minimize the variance. Consider the case when the objective of the experimentation is to select/reject processes from a large set of potential processes. In this scenario traditional overall variance redction design techniqes may not be the optimal soltion. For example, assme that the selection criterion is based on a ct-off vale of operating profit margins, say $1M/yr and processes that have operating profit margins eqal or above the ct-off are worth prsing. he qestion at this jnctre may be: Shold one be more focsed towards redcing the overall parameter ncertainty or towards designing experiment strategies that directly target this objective? In order to design experiments focsed on this target, we propose to design experiments that imize the expected operating profit margin. he premise here is that the designs that try to obtain the imm operating profit margins wold inherently be able to obtain vales closer to the tre optimal operating profit margin vales. In order to obtain sch a DOE we sbstitte the operating profit margin fnction in place of the tility fnction U in (1). 3.1 Problem Formlation Assme an initial model strctre and prior estimates for the process models are available from the prior experimental reslts. he decision-maker wants to perform more experiments to select the few processes with the most potential. Assme the yield (Y 1 ) of the process has a linear model, Y1 = X 1+ ε, where X is the vector of the operating conditions to be optimized and ε is the Gassian noise, N(,) with known variance( 2 ). We assme that the qality of the prodct also varies linearly, Y, with the 2 = X 2 + ε operating conditions and the target qality is. We consider the operating profit margin fnction (f) in Eq. (5), which depends linearly on the yield vale, has a qadratic penalty for the qality deviation, and a qadratic penalty (Q) for higher operating conditions. 2 1 f = α* Y (5) 1 β* Y2 μ X * Q* X 2 o obtain a DOE which imizes the operating profit margins, we sbstitte f, operating profit margin fnction in place of the tility U in (1) and follow the algorithm as explained in Fig.1 and section 2. o evalate the new design criterion, we consider X = [ x1, x2] to be a two dimensional vector and hence both the prior parameter estimates 1 = 1,1, 1,2 & 2 =, 2,2 are also two dimensional vectors. We consider the range of the operating conditions to be in the range of [1e-5, 1]. We consider the prior estimates of the parameters ( 1 and 2 ) to be normal distribtions with mean 1= 1,1, 1,2, 2 =, 2,2 and covariance matrices 1 and 2 respectively. In order to statistically evalate the performance of or DOE approach against the traditional D-optimal DOE approach, we consider the following distribtions for the parameter vales: 1,1 ~ U[-1, 1] (6) 1,2 ~ U[ (-1, - 1,1 ),1] (7) ~ U[-1,1] (8) 2,2 ~ U[ (-1, - ),1] (9) 1 = [ (.1* 1,1 ) 2 ; (.1* 1,2 ) 2 ] (1) 2 = [ (.1* ) 2 ; (.1* 2,2 ) 2 ] (11) = min((.1* 1,1 ) 2, (.1* 1,2 ) 2 ) (12) =U[.5*1e-5*min(, 2,2 ), 1.5*1*(, 2,2 )] (13) he idea behind choosing the above parameter space is not only to have a sfficiently broad range of the parameter space bt also to have some realistically sensible parameter vales. he qadratic penalty matrix (14), Q, for higher operating conditions is chosen appropriately so that it is both positive definite and a practically reasonable vale. Q = [q 11 q 12 ; q 21 q 22 ], where (14) q 11 = U[1e -5, * ( 1,1 + 1,2 )/2 ] (15) q 22 = U[1e -5, * ( + 2,2 )/2 ] (16) q 12 = U[1e -5, (q 11 *q 22 ) ] (17) q 21 = U[1e -5, q 11 *q 22 /q 12 ] (18) Lastly the tre parameter vales, nknown to the decisionmaker, are considered to be drawn randomly from the prior parameter distribtions.

4 3.2 Soltion Approach o obtain the optimal design soltion for the above mentioned problem, we need to solve (19). In (19) Y is the two dimensional vector [Y 1, Y 2 ], each term corresponding to the yield and the qality vale. And is the vector of the corresponding parameters for the yield ( 1 = 1,1, 1,2 ) and qality ( 2 =, 2,2 ) respectively. Algorithm to calclate the optimal design via (19) is shown in Fig. (3). ( * ) = (,,, ) (, ) ( ) (19) U f Y p Y p Y d dy E Y = + ε y Θ ( E i i f, ) i = 1, i = 1,, E i i, i = Calclate Optimal Expected fnction Vale * f i = Calclate Posterior Parameter Distribtions, E, i i i i i,, i i i i, N N Y = + ε Y = + ε N N Y = + ε Y = + ε Y = + ε i i 2 2 * f * i f i = 1 i = i = N Simlate potential realizations of Y 1 and Y 2 from the priors 2 2,, N N N N, N N, i = N * Fig. 3. Algorithm to calclate the optimal decision oriented design of experiment. he calclation algorithm consists of two stages of optimization. he oter optimization is for selecting the optimal design and the inner optimization is for obtaining the optimal posterior operating conditions. he details for evalating a given design are explained as follows: Step : Assme an initial design Step 1: Based on the given design and the prior distribtions for the parameters 1 = 1,1, 1,2 and 2 =, 2,2 generate E i = N f i i potential realizations of Y 1 and Y 2 for i = 1, 2,..., N (we consider N = 5). Step 2: For each given realization estimate the posterior mean and covariance matrix for the parameters and 1 2. Step 3: For each posterior distribtion estimate, obtain the optimal operating condition and hence the optimal fnction f vale. Since the fnction f has nice strctre in or case, the optimization has an analytical soltion. Bt de to the constraints on the operating conditions, the optimal operating conditions are either at the bondary of the constraints or are given by the analytical soltion. Step 4: Calclate the expected vale of the optimal fnction f vale for each of the distribtions. Step 5: Calclate the average of all the optimal expected fnction vales calclated in Step 4. he vale obtained in Step 5 is the vale signifies potential of the given design. In order to obtain the optimal design, imization is performed over the design space. his imization is performed sing the inbilt fnction fmincon in MALAB. 4. RESULS o compare the reslts given by or new design approach and the D-optimal design approach, we took 1, rns for different randomly sampled parameter vales. o measre the performance of different designs, we measre the closeness of the predicted operating profit margin vale to the tre optimal operating profit margin vale. he percentage of times the tre vale is closer to the predicted vale by a design is reported as the Performance Index of that design. able 1 Comparison of performance of new- and D-optimal designs for the 1% noise case. ype of Prior Performance Index Distribtion D Design Strongly Mildly Un In order to check if the kind of prior distribtion has an effect on the performance of the new-design approach, we measre for levels of prior distribtions. An prior distribtion is the one with sqare root of the diagonal elements of the prior covariance matrix being 1% of the prior mean of the respective parameter. his kind of prior distribtion is the one we have shown in (1) and (11). A Strongly prior distribtion is the one with a small covariance and we depict it by replacing the.1 vales

5 by.5 in (1) and (11). A Mildly prior distribtion is the one with a relatively high variance and is depicted by replacing.1 vales by.15 in (1) and (11). A Un- prior distribtion is the one with a relatively high variance and is depicted by replacing.1 vales by.3 in (1) and (11). he comparison of the performance measre for or decision-oriented and the D-optimal design is shown in able 1. he above reslts clearly show ~2-35% improvement in the prediction power of the Decisionoriented design compared to the D-optimal design of experiments. Freqency rve Vale Similarly Fig. 5, Fig. 6 and Fig. 7 show the histogram plots for the, Strongly and Un- prior distribtions respectively. Fig 4, 5, 6 & 7 clearly demonstrate the better performance of the decisionoriented DOE strategy compared to traditional D-optimal design strategy. able 3 Comparison of performance of the Decision-oriented design and the D-optimal designs for the 15% noise case ype of Prior Performance Index Distribtion D Design Strongly Mildly Un Objective fnction Vale Fig. 4. Histogram comparing the prediction of the Decision oriented and D-optimal design to the tre objective vale for a Mildly prior distribtion. Freqency rve Vale able 2 Comparison of performance of the Decision-oriented design and the D-optimal designs for the 5% noise case ype of Prior Performance Index Distribtion D Design Strongly Mildly Un o give more insight to the reslts we plot the histogram of the optimal operating margin vales for the decision oriented and D-optimal design along with the optimal operating margin vales (determined assming that the tre parameter vales are known), for a particlar set of parameter vales with 5 different tre parameter vales being sampled from the prior distribtion. o be precise, these are the operating margins for the tre plant (with tre parameter vales) with the optimal operating conditions determined based on the parameter estimates reslting from the respective DOEs. Fig 4 shows the histogram for a Mildly prior distribtion, with 5 tre parameter vales sampled from the prior distribtion Objective fnction Vale Fig. 5. Histogram comparing the prediction of the Decision oriented and D-optimal design to the tre objective vale for an prior distribtion. Freqency rve Vale Objective fnction Vale Fig. 6. Histogram comparing the prediction of the Decision oriented and D-optimal design to the tre objective vale for a Strongly prior distribtion.

6 Freqency rve Vale Objective fnction Vale Fig. 7. Histogram comparing the prediction of the Decision oriented and D-optimal design to the tre objective vale for a Un- prior distribtion. In order to check if noise has any significant impact on the performance of the decision-oriented designs, we vary the noise measred by the variance of the Gassian noise in (12). In comparison to the initial noise of 1% as depicted by the vale.1 in (12) we test two other levels of noise 5% and 15%, which correspond to changing.1 vale in (12) to.5 and.15 respectively. he reslts for the 5% and the 15% noise cases are shown in the able 2 and able 3 respectively. he reslts clearly demonstrate that the decision-optimal design otperforms the D-optimal design regardless of the noise level. 5. CONCLUSIONS We have introdced a new decision oriented design of experiment strategy, which significantly improves the prediction of a process s optimal objective fnction vale compared to that of a D-optimal design of experiment strategy. hese types of DOE strategies are expected to be of significant importance in improving the R&D decisions, especially in bio-fel related research where one faces mltiple process alternatives. Moreover, in addition to the design criterion considered in this work, one can consider alternative Acceptance/Rejection design criterion. For example, in the problem discssed in this work, we were mainly concerned with the mean vale of f, bt an alternat- Fnction Vale Ctoff 2f +k* 2f 2f P1 2f -k* 2f 1f +k* 1f 1f P2 1f -k* 1f Ctoff -ive design criterion can be based on both the mean and the variance of f along with a ct-off vale. Consider two processes P1 and P2, shown in Fig. 8. he selection criterion of the decision maker for these processes is that k* be greater than the Ctoff and the rejection criterion being that +k* be less than the Ctoff, where is the posterior mean and is the posterior standard deviation of the objective fnction f. o design experiments for selection/rejection of processes based on this type of criterion can be done by imising i, where i is defined as follows: + 1, if μfi k * σi > Ctoff δi = 1, if μfi + k * σi < Ctoff, Otherwise he sbscript i represents the potential random samples with vale ranging form i = 1, 2.., N, as explained earlier in section 3.2. Similarly varios other design criterions can be created based on the decision maker s objective fnction. We will evalate these and similar acceptance/rejection decision criterions in or ftre work. REFERENCES Atkinson, A.C. and Donev, A.N. (1992). Optimm Experimental Designs, Pg. 114, Ch.1. Oxford Science Pblications, United States Bernardo, J.M.(1979). Expected information as expected tility. Annals of Statistics, Chaloner, K. and Verdinelli, I. (1995). Bayesian Experimental Design: A Review, Statistical Science, Cowles, Mary K. and Carlin, Bradley P. (1996), Markov Chain Monte Carlo Convergence Diagnostics: A comparative Review, Jornal of the American Statistical Association, DeGroot, M.H. (1962). Uncertainty, information and seqential experiments, he Annals of Mathematical Statistics, Kass, Robert E., Carlin, Bradley P., Gelman, Andrew, and Neal, Radford M., Markov Chain Monte Carlo in Practice: A Rondtable Discssion, he American Statistician, Lindley, D.V. (1956). On the measre of information and provided by an experiment. Annals of Statistics, Raiffa, H. and Schlaifer, R. (1961). Applied statistical decision theory, Division of Research, Gradate School of Bsiness Administration, Harvard University, Boston, United States Stone, M. (1959). Application of a measre of information to the design and comparison of regression experiment. he Annals of Mathematical Statistics, Fig. 8. An alternative Acceptance/Rejection criterion