for Adequate Precision via Fraction of Design Space (FDS) Pat Whitcomb Stat-Ease, Inc. 61.746.036 036 fax 61.746.056 pat@statease.com Gary W. Oehlert School of Statistics University it of Minnesota 61.65.1557 gary@stat.umn.edu 1 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 1
Sizing Factorial Designs During screening and characterization the emphasis is on identifying factor effects. What are the important design factors? For this purpose power is an ideal metric to evaluate design suitability. Screening Known Factors Unknown Factors Screening Ti Trivial i many Characterization Vital few Factor effects and interactions no Curvature? yes Optimization Response Surface methods Verification Confirm? no Backup Celebrate! yes 3 Copyright 009 Stat-Ease, Inc. Factorial Design Power 3 Full Factorial Δ= and σ=1 One Factor 1 1 11 R1 10 9 8 11 1.00 1.00 0.50 0.50 B: B 0.00-0.50-1.00-1.00-0.50 0.00 A: A R1 10 Δ 9 8-1.00-0.50 0.00 0.50 1.00 A: A 4
Factorial Design Power 3 Full Factorial Δ= and σ=1 Power is reported at a 5.0% alpha level to detect the specified signal/noise ratio. Recommended power is at least 80%. R1 Signal (delta) =.00 Noise (sigma) = 1.00 Signal/Noise (delta/sigma) =.00 A B C 57. % 57. % 57. % The following three slides dig into how power is computed. 5 Copyright 009 Stat-Ease, Inc. One Replicate of 3 Full Factorial C = (X T X) -1 matrix The design determines the standard error of the coefficient: 0.15 0 0 0 0 0 0 0 0 0.15 0 0 0 0 0 0 0 0 0.15 0 0 0 0 0 0 0 0 0.15 0 0 0 0 C = 0 0 0 0 0.15 0 0 0 0 0 0 0 0 0.15 0 0 0 0 0 0 0 0 0.15 0 0 0 0 0 0 0 0 0.15 β β β = = i i i t-value i = SE ( βi ) ciiσˆ ( 0.15) σˆ 6 3
NonCentrality Parameter 3 Full Factorial Δ= and σ=1 The reference t distribution assumes the null hypothesis of Δ = 0. The noncentrality parameter (.88) defines the t distribution under the alternate hypothesis of Δ =. noncentrality = i = β ii i cˆ σ Δi cˆ σ ii 1 = ( 015 0.15)( 1 ) 1 = =.88 0.3536 7 Copyright 009 Stat-Ease, Inc. Factorial Design Power 3 Full Factorial Δ= and σ=1 noncentral t α=0.05,df=4 with noncentrality parameter of.88.88 Power = 57.% 8 4
Factorial Design Power Two Replicates of 3 Full Factorial Δ= and σ=1 Power is reported at a 5.0% alpha level to detect the specified signal/noise ratio. Recommended power is at least 80%. R1 Signal (delta) =.00 Noise (sigma) = 1.00 Signal/Noise (delta/sigma) =.00 A B C 95.6 % 95.6 % 95.6 % 9 Copyright 009 Stat-Ease, Inc. Two Replicates of 3 Full Factorial C = (X T X) -1 matrix The design determines the standard error of the coefficient: 0.065 0 0 0 0 0 0 0 0 0.065 0 0 0 0 0 0 0 0 0.065 0 0 0 0 0 0 0 0 0.065 0 0 0 0 C = 0 0 0 0 0.065 0 0 0 0 0 0 0 0 0.065 0 0 0 0 0 0 0 0 0.065 0 0 0 0 0 0 0 0 0.065 β β β i i i t-value i = = = SE ( βi ) ciiσˆ ( 0.065) σˆ 10 5
NonCentrality Parameter Two Replicates of 3 Full Factorial Δ= and σ=1 noncentrality = i = Δ i i = iiσ cˆ iiσ β cˆ 1 ( 0.065)( 1) 1 = = 40 4.0 0.5 11 Copyright 009 Stat-Ease, Inc. Factorial Design Power Two Replicates of 3 Full Factorial Δ= and σ=1 noncentral t α=0.05,df=1 with noncentrality parameter of 4.0 4.0 Power = 95.6% 1 6
Conclusions Factorial DOE During screening and characterization (factorials) emphasis is on identifying factor effects. What are the important design factors? For this purpose power is an ideal metric to evaluate design suitability. 13 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 14 7
Sizing Response Surface Designs Screening Known Factors Unknown Factors Screening Ti Trivial i many When the goal is optimization emphasis is the fitted surface. How well does the surface represent true behavior? For this purpose precision (FDS) is an good metric to evaluate design suitability. (Assuming model adequacy; i.e. insignificant lack of fit.) Vital few Characterization Factor effects and interactions no Curvature? yes Optimization Response Surface methods Verification no Confirm? Backup Celebrate! yes 15 Copyright 009 Stat-Ease, Inc. Design Properties 1. Estimate the designed for polynomial well.. Give sufficient information to allow a test for lack of fit. Have more unique design points than coefficients i in the model. Provide an estimate of pure error. 3. Remain insensitive to outliers, influential values and bias from model misspecification. 4. Be robust to errors in control of the component levels. 5. Provide a check on model assumptions, e.g., normality of errors. 6. Generate useful information throughout the region of interest, i.e., provide a good distribution of Var Yˆ σ. ( ) 7. Do not contain an excessively large number of trials. 16 8
FDS Fraction of Design Space Fraction of Design Space: Calculates the volume of the design space having a prediction variance (PV) less than or equal to a specified value. The ratio of this volume to the total volume of the design volume is the fraction of design space. Produces a single plot showing the cumulative fraction of the design space on the x-axis (from zero to one) versus the PV on the y-axis. 17 Copyright 009 Stat-Ease, Inc. FDS PV Fraction of Design Space Prediction Variance: ( ) var yˆ ( ) 0 PV x = = x T ( X T X ) 1 x s PV is a function of: 0 0 0 x 0 the location in the design space (i.e. the x coordinates for all model terms). X the experimental design (i.e. where the runs are in the design space). 18 9
FDS StdErr Fraction of Design Space Prediction standard error of the expected value: var ( y ˆ 0 ) 1 T T PV ( x ) = = x ( X X ) x s 0 0 0 StdErr x s yˆ 0 ( ) = = PV ( x ) 0 0 s 19 Copyright 009 Stat-Ease, Inc. FDS Plot Fraction of Design Space 1. Pick random points in the design space.. Calculate the standard error of the expected value SEŷ0 T T 1 = x0 ( X X) x0 s 3. Plot the standard error as a fraction of the design space. StdErr Mean 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.0 0.10 0.00 FDS Graph 0.00 0.5 0.50 0.75 1.00 Fraction of Design Space 0 10
Simplex-Lattice Augmented and 4 Replicates A: A 1.00 0.57 1.00 FDS Graph 0.75 0.47 0.00 0.57 0.57 0.00 StdErr Mean 0.50 0.47 0.57 0.57 0.5 0.57 0.57 0.47 0.47 0.57 0.00 1.00 B: B StdErr of Design 1.00 C: C 0.00 0.00 0.5 0.50 0.75 1.00 Fraction of Design Space Mix section 1 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 11
Two Component Mixture Linear Model Two Component Mix 7 6 R1 5 4 3 Actual A 0 0.5 0.5 0.75 1 Actual B 1 0.75 0.5 0.5 0 3 Copyright 009 Stat-Ease, Inc. Two Component Mixture Linear Model The solid center line is the fitted model; ŷ is the expected value or mean prediction. The curved dotted lines are the computer generated confidence limits, or the actual precision. d Is the half-width of the desired confidence interval, or the desired precision. It is used to create the outer straight lines. R1 7 6 5 4 Two Component Mix y ˆ ± d Note: The actual precision of the fitted value depends on where we are predicting. 3 Actual A 0 0.5 0.5 0.75 1 Actual B 1 0.75 0.5 0.5 0 4 1
Two Component Mixture Linear Model Evaluate the standard error as a fraction of design space for this design. 7 Two Component Mix 6 R1 5 4 3 Actual A 0 Actual B 1 0.5 0.75 0.5 0.5 0.75 0.5 1 0 Sizing Mixture (RSM) Designs 5 Copyright 009 Stat-Ease, Inc. Click on Evaluate and then Graphs : Two Component Mixture Linear Model 6 13
Two Component Mixture Linear Model Want a linear surface to represent the true response value within ± 0.64 with 95% confidence. The overall standard deviation for this response is 0.55. Enter: d =064 0.64 s = 0.55 a (α) = 1 0.95 = 0.05 7 Copyright 009 Stat-Ease, Inc. Two Component Mixture Linear Model Only 53% of the design space is precise enough to predict the mean within ± 0.64. 8 14
Define Precision Half Width of the Confidence Interval Statistical detail Confidence interval on the expected value: The mean response is estimated and the precision of the estimate is quantified by a confidence interval: α, df ( y ) y ˆ ± t sˆ We will use the half-width of the confidence interval (d) to define the precision p desired: d d = tα, ( sˆ) or sˆ = df y y t α, df Mix section 4 9 Copyright 009 Stat-Ease, Inc. What Precision is Needed? Mean Confidence Interval Half-Width Statistical detail Input Half-width of confidence interval: d Estimate of standard deviation: s yˆ ± d d t s ( ˆ ) = α, df y ( ) 1 T T ˆ = 0 X X y 0 s s x X X x syˆ 1 T T StdErr Mean( FDS) = = x ( X X ) x s 0 0 Mix section 4 30 15
Two Component Mixture Linear Model Statistical detail Want a linear surface to represent the true response value within ± 0.64 with 95% confidence. The overall standard deviation for this response is 0.55. For 95% confidence t =.571, d = 0.64 & s = 0.55 s yˆ.05,5 d 0.64 = = = 0.5.571 t α, df ( ) StdErr Mean FDS s y ˆ 0.5 = = = 0.45 s 0.55 Mix section 4 31 Copyright 009 Stat-Ease, Inc. FDS StdErr Mean Plot Fraction of Design Space Statistical detail 1. Pick random points in the design space.. Calculate the standard error of the expected value SE S ( ) ŷ 1 0 T T = x0 X X x0 3. Plot the standard error as a fraction of the design space. StdErr 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.0 0.10 0.00 FDS Graph 0.00 0.5 0.50 0.75 1.00 Mix section 4 Fraction of Design Space 3 16
Two Component Mixture Linear Model 53% is precise enough, i.e. 53% has a StdErr 0.45 d = 0.64 s = 0.55 α = 0.05 33 Copyright 009 Stat-Ease, Inc. Two Component Mixture Linear Model Two Component Mix 53% of the design space has the desired precision, i.e. is inside the solid straight (red) lines. 7 6 53% R1 5 y ˆ ± d 4 3 Actual A 0 0.5 0.5 0.75 1 Actual B 1 0.75 0.5 0.5 0 34 17
Sizing for Precision FDS? How good is good enough? Rules of thumb: For exploration want FDS 80% For verification want FDS 90% or higher! What can be done to improve precision? Manage expectations; i.e. increase d Decrease noise; i.e. decrease s Increase risk of Type I error; i.e. increase alpha Increase the number of runs in the design 35 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 36 18
Mixture Simplex Quadratic Model ŷ = 4A + 4B+ 8C+ 16AC Most of the action occurs on the A-C edge because of the AC coefficient of 16. The quadratic coefficient of 16 means that the response is 4 units higher at A=0.5, C=0.5 than one would expect with linear blending. R1 14 1 10 8 6 4 0 A (1) B (0) C (1) C (0) A (0) B (1) 37 Copyright 009 Stat-Ease, Inc. Mixture Simplex Quadratic Model ŷ = 4A + 4B+ 8C+ 16AC This illustrates the quadratic blending along the A-C edge as C increases from zero to one. 14 1 10 8 6 4 β = 4 A 4 1 4 β = 4 AC β C = 8 0 0 0.1 0. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 C 38 19
Mixture Constrained Design for Quadratic Model 0.0 A 1.0 0.0 B 1.0 0.0 C 0.1 Total = 1.0 A: A 1.000 0.000 0.000 1.000 B: B 0.000 StdErr of Design 1.000 C: C 39 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model (1 replicate) Power at 5 % alpha level for effect of Term StdErr* VIF R i -Squared 1 s s 3 s A 0.87 3.44 0.7094 5.0 % 5.0 % 5.0 % B 0.87 3.44 0.7094 5.0 % 5.0 % 5.0 % C 33.34 7.9 0.9996 5.0 % 5.0 % 5.0 % AB.86.19 0.544.9 % 67.3 % 94.7 % AC 58.98 109.98 0.9991 5.0 % 5.0 % 5.0 % BC 58.98 109.98 0.9991 5.0 % 5.0 % 5.0 % *Basis Std. Dev. = 1.0 Power is bottomed out at alpha! 40 0
Define Precision Half Width of the Confidence Interval 6. Generate useful information throughout the region of interest. Question: Will predictions, using the quadratic model from this design, be precise enough for our purposes? To know the truth requires an infinite number of runs; most likely this will exceed our budget. So the question is how precisely do we need to estimate the response? The trade off is more precision requires more runs. 41 Copyright 009 Stat-Ease, Inc. Define Precision Half Width of the Confidence Interval Confidence interval on the expected value: The mean response is estimated and the precision of the estimate is quantified by a confidence interval: α, df ( y ) y ˆ ± t sˆ We will use half-width of the confidence interval (d) to define the precision desired: d d = tα, ( sˆ) or sˆ = df y y t α, df 4 1
Input Half-width of confidence interval: d Estimate of standard deviation: s What Precision is Needed? Confidence Interval Half-Width yˆ d = t α, df T ( ) 1 T s = s x X X x yˆ 0 0 syˆ 1 T T StdErr Mean( FDS) = = x ( X X ) x s s 0 0 43 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model (1 replicate) Want quadratic surface to represent the true response value within ±10 with 95% confidence. The standard deviation for this response is 7.8. For 95% confidence t =.365, d = 10 & s = 7.8 s yˆ t α, df.05,7 d 10 = = = 4.3.365 ( ) StdErr Mean FDS s y ˆ 4.3 = = = 0.54 s 7.8 44
Mixture Constrained Quadratic Model (1 replicate) Only 58% of the design space has StdErr 0.54 d = 10 s= 7.8 α = 0.05 45 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model ( replicates) Power at 5 % alpha level for effect of Term StdErr* VIF R i -Squared 1 s s 3 s A 0.6 3.44 0.7094 5.0 % 5.0 % 5.0 % B 0.6 3.44 0.7094 5.0 % 5.0 % 5.0 % C 164.99 7.9 0.9996 5.0 % 5.0 % 5.0 % AB.0.19 0.544 47.1 % 96.5 % 99.9 % AC 183.1 109.98 0.9991 5.0 % 5.0 % 5.0 % BC 183.1 109.98 0.9991 5.0 % 5.0 % 5.0 % *Basis Std. Dev. = 1.0 Adding a replicate does little to increase power! 46 3
Mixture Constrained Why Does Power Not Increase? Quadratic, linear or combinations of them model response. Model Coefficients are Correlated! 14 1 Linear 10 8 Quadratic 6 4 0 Constraint 0 0.1 0. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 C Individual coefficients can not be resolved! 47 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model ( replicates) Want quadratic surface to represent the true response value within ± 10 with 95% confidence. The overall standard deviation for this response is 7.8. For 95% confidence t =.086, d = 10 & s = 7.8 s yˆ t α, df.05, 0 d 10 = = = 4.79.086 ( ) StdErr Mean FDS s y ˆ 4.79 = = = 0.61 s 7.8 48 4
Mixture Constrained Quadratic Model ( replicates) 100% of the design space has StdErr 0.61 49 Copyright 009 Stat-Ease, Inc. Precision Depends On The size of the confidence interval half-width (d): A larger half-width (d) increases the FDS. The size of the experimental error σ: A smaller s increases the FDS. The α risk chosen: A larger α increases the FDS. Choose design appropriate to the problem: Size the design for the precision required. 50 5
Conclusions Factorial DOE During screening and characterization (factorials) emphasis is on identifying factor effects. What are the important design factors? For this purpose power is an ideal metric to evaluate design suitability. Mixture Design and Response Surface Methods When the goal is optimization (usually the case for mixture design & RSM) emphasis is on the fitted surface. How well does the surface represent true behavior? For this purpose precision (FDS) is a good metric to evaluate design suitability. 51 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 5 6
Detecting a Difference of Δ wherever it may occur Δ What is the probability of finding a difference Δ if it occurs between any two points in the design space? 53 Copyright 009 Stat-Ease, Inc. FPDS StdErr Diff Plot Fraction of Paired Design Space 1. Pick random pairs of points in the design space.. Calculate the standard error of the difference SE ( ) ( ) ( ) ( ) yˆ ˆ 1 1 1 1 y T T T T T T = x1 X X x1 + x X X x x1 X X x s 3. Plot the standard error as a fraction of the design space..00 FPDS Graph 1.80 1.60 1.40 StdErr Diff 1.0 1.00 0.80 0.60 0.40 0.0 0.00 0.000 0.50 0.500 0.750 1.000 Fraction of Paired Design Space 54 7
Revisit Mixture Constrained Design for Quadratic Model 0.0 A 1.0 0.0 B 1.0 0.0 C 0.1 Total = 1.0 A: A 1.000 0.000 0.000 1.000 B: B 0.000 StdErr of Design 1.000 C: C 55 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model (1 replicate) Want to detect a difference of 10 on a quadratic surface with 95% confidence. The overall standard deviation this for response is 7.8. For 95% confidence t =.365, Δ= 10 & s = 7.8 s Δ.05,7 Δ 10 = = = 4.3 t 365.365 α, df ( ) StdErr Diff FPDS sδ 4.3 = = = 0.54 s 7.8 56 8
Mixture Constrained Quadratic Model (1 replicate) Only 37% of the design space has StdErr 0.54 Δ = 10 s= 7.8 α = 0.05 57 Copyright 009 Stat-Ease, Inc. Mixture Constrained Quadratic Model ( replicates) Want to detect a difference of 10 on a quadratic surface with 95% confidence. The overall standard deviation this for response is 7.8. For 95% confidence t =.086, Δ= 10 & s = 7.8 s Δ t α, df.05, 0 Δ 10 = = = 4.79.086 ( ) StdErr Diff FPDS sδ 4.79 = = = 0.61 s 7.8 58 9
Mixture Constrained Quadratic Model ( replicates) 86% of the design space has StdErr 0.61 59 Copyright 009 Stat-Ease, Inc. Detecting a Difference of Δ Depends On The size of the difference (Δ): A larger difference (Δ) increases the FPDS. The size of the experimental error σ: A smaller σ increases the FPDS The α risk chosen: A larger α increases the FPDS. Choose design appropriate to the problem: Size the design to detect t a difference of interest. t 60 30
Conclusions Factorial DOE During screening and characterization (factorials) emphasis is on identifying factor effects. What are the important design factors? For this purpose power is an ideal metric to evaluate design suitability. Mixture Design and Response Surface Methods When the goal is to detect a change in the response emphasis is on differences over the fitted surface. Does a difference D occur in the design space? For this purpose precision (FPDS) is a good metric to evaluate design suitability. 61 Copyright 009 Stat-Ease, Inc. Review Power to size factorial designs Precision in place of power Introduce FDS Sizing designs for precision Two component mixture Three component constrained mixture Sizing designs to detect a difference Summary 6 31
Summary Two criteria covered: 1. Precision of expected value, StdErr Mean: Use the StdErr Mean when the purpose of the DOE is to quantify shape; i.e. functional design (e.g. optimization).. Precision to detect a difference, StdErr Diff: Use the StdErr Diff when the purpose of the DOE is to detect t change in the design space. 63 Copyright 009 Stat-Ease, Inc. Summary Two criteria not covered: 1. Precision of predicted value, StdErr Pred: Use the StdErr Pred when the purpose of the DOE is to develop a model for prediction.. Precision to form a tolerance interval, TI Multiplier: Use the TI Multiplier when the purpose of the DOE is to form a tolerance interval at a point in the design space. (new in Design-Expert version 8) 64 3
References [1] Gary W. Oehlert (000), A First Course in Design and Analysis of Experiments, W. H. Freeman and Company. [] Douglas C. Montgomery (005), 6 th edition, Design and Analysis of Experiments, John Wiley and Sons, Inc. [3] Gary Oehlert and Patrick Whitcomb (001), Sizing Fixed Effects for Computing Power in Experimental Designs, Quality and Reliability Engineering International, July 7, 001 [4] Alyaa R. Zahran, Christine M. Anderson-Cook and Raymond H. Myers, Fraction of Design Space to Assess Prediction, Journal of Quality Technology, Vol. 35, No. 4, October 003 [5] Heidi B. Goldfarb, Christine M. Anderson-Cook, Connie M. Borror and Douglas C. Montgomery, Fraction of Design Space plots for Assessing Mixture and Mixture-Process Designs, Journal of Quality Technology, Vol. 36, No., October 004. [6] Steven DE Gryze, Ivan Langhans and Martina Vandebroek, Using the Intervals for Prediction: A Tutorial on Tolerance Intervals for Ordinary Least-Squares Regression, Chemometrics and Intelligent Laboratory Systems, 87 (007) 147 154 [7] Raymond H. Myers, Douglas C. Montgomery and Christine M. Anderson-Cook (009), 3 rd edition, Response Surface Methodology, John Wiley and Sons, Inc. [8] John A. Cornell (00), 3 rd edition, Experiments with Mixtures, John Wiley and Sons, Inc. [9] Wendell F. Smith (005), Experimental Design for Formulation, ASA-SIAM Series on Statistics and Applied probability. 65 Copyright 009 Stat-Ease, Inc. Thank You for Attending To obtain a copy of my slides: Give me your business card Send a request to pat@statease.com t t 66 33