Design of Experiments SUTD - 21/4/2015 1
Outline 1. Introduction 2. 2 k Factorial Design Exercise 3. Choice of Sample Size Exercise 4. 2 k p Fractional Factorial Design Exercise 5. Follow-up experimentation (folding over) with factorial design Exercise 6. More about DOE SUTD - 21/4/2015 2
1. Introduction Observing a system or process while it is in operation is an important part of the learning process, and is an integral part of understanding and learning about how systems and processes work. To understand what happens to a process when you change certain input factors, you have to do more than just watch you actually have to change the factors. This means that to really understand cause-and-effect relationships in a system you must deliberately change the input variables to the system and observe the changes in the system output that these changes to the inputs produce. In other words, you need to conduct experiments on the system. SUTD - 21/4/2015 3
1. Introduction Investigators perform experiments in virtually all fields of inquiry, usually to discover something about a particular process or system. Each experimental run is a test. Experiment: A test or series of runs in which purposeful changes are made to the input variables of a process or system so that we may observe and identify the reasons for changes that may be observed in the output response. SUTD - 21/4/2015 4
1. Introduction x i Controllable factors Inputs Process Output y Uncontrollable factors z j SUTD - Capstone - 6/2/2015 5
1. Introduction Objectives of the experiment: Determining which variables are most influential on the response y. Determining where to set the influential x so that y is almost always near the desired nominal value. Determining where to set the influential x so that variability in y is small. Determining where to set the influential x so that the effects of the uncontrollable variables z are minimized. SUTD - 21/4/2015 6
1. Introduction You are a golf player who doesn t enjoy practicing. But you also want to lower your score. SUTD - 21/4/2015 7
1. Introduction Factors that may affect your score: Oversized vs regular-sized driver Balata vs three-piece ball Walk vs ride Water vs sth else Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD - 21/4/2015 8
1. Introduction Preliminary elimination: Use your experience to eliminate the factors that will not affect the response (output) significantly. Engineers, scientists, and business analysts, often must make these types of decisions about some of the factors they are considering in real experiments. Based on your long experience with golf, you decide that the effects of the last four factors on your score are very small. SUTD - 21/4/2015 9
1. Introduction Factors that may affect your score: Oversized vs regular-sized driver O vs R Balata vs three-piece ball B vs T Walk vs ride W vs R Water vs sth else W vs SE Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD - 21/4/2015 10
1. Introduction Suppose that a maximum of eight rounds of golf can be played over the course of the experiment (resource constraints). You decide to start with the following combination: O B R W resulting score: 87 During the round, you notice several wayward shots with the big driver, so you decide to play another round with the regular-sized driver. R B R W This approach could be continued almost indefinitely. SUTD - 21/4/2015 11
1. Introduction Best-guess approach is frequently used in practice by engineers and scientists. It often works reasonably well, too, because the experimenters often have a great deal of technical or theoretical knowledge of the system they are studying, as well as considerable practical experience. The best-guess approach has at least two disadvantages: 1. Bad guess: Suppose the initial best-guess does not produce the desired results. Now the experimenter has to take another guess at the correct combination of factor levels. This could continue for a long time, without any guarantee of success. 2. Good guess: Suppose the initial best-guess produces an acceptable result. Now the experimenter is tempted to stop testing, although there is no guarantee that the best solution has been found. SUTD - 21/4/2015 12
1. Introduction One-factor-at-a-time (OFAT): The OFAT method consists of selecting a starting point, or baseline set of levels, for each factor, and then successively varying each factor over its range with the other factors held constant at the baseline level. After all tests are performed, a series of graphs are usually constructed showing how the response variable is affected by varying each factor with all other factors held constant. SUTD - 21/4/2015 13
1. Introduction Baseline: O B W W Optimal combination wrt OFAT: R (B or T) R - W SUTD - 21/4/2015 14
1. Introduction The major disadvantage of the OFAT strategy is that it fails to consider any possible interaction between the factors. An interaction is the failure of one factor to produce the same effect on the response at different levels of another factor. OFAT experiments are always less efficient than other methods based on a statistical approach to design. OFAT solution: R (B or T) R - W SUTD - 21/4/2015 15
1. Introduction Correct approach? 2 4 factorial design (4 factors each having 2 levels) This experiment requires 16 runs. A 10-factor experiment would require 1024 runs. Fortunately, we can use fractional factorial designs. SUTD - 21/4/2015 16
1. Introduction 2 2 factorial design with 2 replications: 2 2 2 = 8 rounds D = B = 92 + 94 + 93 + 91 4 88 + 91 + 92 + 94 4 88 + 91 + 88 + 90 4 88 + 90 + 93 + 91 4 = 3.25 = 0.75 DB = 92 + 94 + 88 + 90 4 88 + 91 + 93 + 91 4 = 0.25 Statistical tests show that: D is significant, B and DB are not. SUTD - 21/4/2015 17
1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD - 21/4/2015 18
2. 2 k Factorial Design Geometric Coding and Labels (2 3 factorial design) 8 combination of the factors, or treatments. Run A B C Labels 1 (1) 2 + a 3 + b 4 + + ab 5 + c 6 + + ac 7 + + bc 8 + + + abc SUTD - 21/4/2015 26
2. 2 k Factorial Design Algebraic Signs for Calculating Effects Labels I A B AB C AC BC ABC (1) + + + + a + + + + b + + + + ab + + + + c + + + + ac + + + + bc + + + + abc + + + + + + + + A B = AB, AB B = AB 2 = A A = 1 4n 1 + a b + ab c + ac bc + abc SUTD - 21/4/2015 27
2. 2 k Factorial Design Geometric View SUTD - 21/4/2015 28
2. 2 k Factorial Design A chemical product is produced in a pressure vessel. A factorial experiment is carried out in the pilot plant to study the factors thought to influence the filtration rate (FR) of this product. The four factors are: A. Temperature (Temp) B. Pressure (Pres) C. Concentration of formaldehyde (Conc) D. Stirring rate (SR) Objectives: The process engineer is interested in maximizing the filtration rate. Current rates are around 75 gal/h. The process currently uses the conc. of formaldehyde, at the high level. The engineer would like to reduce the formaldehyde concentration as much as possible but has been unable to do so because it always results in lower filtration rates. SUTD - 21/4/2015 29
2. 2 k Factorial Design Each factor has 2 levels. We use a 2 4 factorial design. We carry out a single run for each of the 16 combinations. SUTD - 21/4/2015 30
2. 2 k Factorial Design Regression Model y = β 0 + β A x A + β B x B + β C x C + β D x D + β AB x A x B + β AC x A x C + β AD x A x D + β BC x B x C + β B x B x D + β CD x C x D + β ABC x A x B x C + β ABD x A x B x D + β ACD x A x C x D + β BCD x B x C x D + β ABCD x A x B x C x D SUTD - 21/4/2015 31
2. 2 k Factorial Design Minitab Exercise: Section 2 in DOE-Minitab.pdf SUTD - 21/4/2015 32
2. 2 k Factorial Design Pareto Chart Normal Plot SUTD - 21/4/2015 33
2. 2 k Factorial Design Fitted Regression Model Try all factors at high levels: Temp = Press = Conc = SR = +1 FR = 99.556 Observed value: 96 SUTD - 21/4/2015 34
2. 2 k Factorial Design Main Effects Plot If we consider only the main effects, we would run all factors at high levels. SUTD - 21/4/2015 35
2. 2 k Factorial Design Interaction Plot SUTD - 21/4/2015 36
2. 2 k Factorial Design Temp (A) effect is: very small when C is high, very large when C is low. best: C = 1, A = +1 SR (D) effect is: little when A is low, large when A is high. Best combination: A = +1, C = 1, D = +1 Note that B is not significant. SUTD - 21/4/2015 37
2. 2 k Factorial Design Risks with Unreplicated Factorial Design Error variance is not estimable. We can never be entirely certain that the experimental error is small. A good practice in these types of experiments is to spread out the factor levels aggressively. One approach to the analysis of an unreplicated factorial is to assume that certain highorder interactions are negligible and combine their mean squares to estimate the error. This is an appeal to the sparsity of effects principle; that is, most systems are dominated by some of the main effects and low-order interactions, and most high-order interactions are negligible. Appropriate when you are confident that experimental error is small: Computer simulations with no random numbers: when same inputs always result in the same outputs (experimental error = 0). SUTD - 21/4/2015 38
3. Choice of Sample Size Choice of Sample Size What is the appropriate sample size (number of replications)? Depends on desired power of the experimentation, overall variance of the response, desired accuracy. SUTD - 21/4/2015 39
3. Choice of Sample Size Choice of Sample Size Power: β = Pr Type II error = Pr fail to reject H 0 H 0 is false power = 1 β As power increases, the chance to catch a significant effect increases. Effects: The minimum difference between ± levels of the factors that you want to detect. Minimum detectable change (MDC) size. SUTD - 21/4/2015 40
3. Choice of Sample Size Choice of Sample Size As the effects increases, the power increases. As the sample size get larger, the power of the test gets larger. Minitab Exercise: Section 3 in DOE-Minitab.pdf SUTD - 21/4/2015 41
3. Choice of Sample Size SUTD - 21/4/2015 42
4. 2 k p Fractional Factorial Design 2 3 1 Fractional Design with defining relation I = ABC Labels I A B AB C AC BC ABC (1) + + + + a + + + + b + + + + ab + + + + c + + + + ac + + + + bc + + + + abc + + + + + + + + SUTD - 21/4/2015 43
4. 2 k p Fractional Factorial Design 2 3 1 Fractional Design with defining relation I = ABC Labels I A B C a + + b + + c + + abc + + + + Number of runs is reduced from 8 to 4, with the cost of aliasing some effects. Resolving the aliases: A = A I = A ABC = A 2 BC = BC B = B I = B ABC = AB 2 C = AC C = C I = C ABC = ABC 2 = AB This means: A = A + BC B = B + AC C = C + AB I = I + ABC SUTD - 21/4/2015 44
4. 2 k p Fractional Factorial Design Geometric View If the experimenter removes any of the three factors from the analysis, the 2 3 1 design will project into a full factorial 2 2 design. SUTD - 21/4/2015 45
4. 2 k p Fractional Factorial Design Improving Online Learning A university that specializes in online learning wants to improve the effectiveness of its 8-week course. Statistics group identifies 7 factors to test at 2 levels each: Code Factor - + A Textbook Current New B Readings No Yes C Homework 3 Hours 5 Hours D Software Current New E Sessions 3 per week 4 per week F Review No Yes G Lecture Notes No Yes SUTD - 21/4/2015 46
4. 2 k p Fractional Factorial Design Improving Online Learning Full factorial design requires 2 7 = 128 runs for a single replication. You decide to use 2 7 4 III saturated design. 8 runs for each replication. Each of the 8 runs defines characteristics of a section. 10 students are randomly assigned to each of the 8 sections. At the end of the course, each student takes a final exam (response). SUTD - 21/4/2015 47
4. 2 k p Fractional Factorial Design Improving Online Learning Regression Model: y = β 0 + β A x A + β B x B + β C x C + β D x D + β E x E + β F x F + β G x G Only the main effects can be estimated. Actually, they are confounded with interactions, and we know the alias structure. SUTD - 21/4/2015 48
4. 2 k p Fractional Factorial Design Improving Online Learning Generators: D = AB, E = AC, F = BC, G = ABC Or equivalently: I = ABD, I = ACE, I = BCF, I = ABCG Two-level aliases of A: A = B AB = BD (first generator) A = C AC = CE (second generator) A = BC ABC = FG (third and fourth generators) A = A + BD + CE + FG + higher-order-interactions SUTD - 21/4/2015 49
4. 2 k p Fractional Factorial Design Improving Online Learning SUTD - 21/4/2015 50
4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes 3 hr New 3 /wk No No 5 Current No 5 hr New 3 /wk No Yes 6 New No 5 hr Current 4 /wk No No 7 Current Yes 5 hr Current 3 /wk Yes No 8 New Yes 5 hr New 4 /wk Yes Yes SUTD - 21/4/2015 51
4. 2 k p Fractional Factorial Design Minitab Exercise: Section 4 in DOE-Minitab.pdf SUTD - 21/4/2015 52
4. 2 k p Fractional Factorial Design Improving Online Learning SUTD - 21/4/2015 53
4. 2 k p Fractional Factorial Design Model Adequacy Checking Normal Probability Plot should resemble a straight line. Histogram should resemble normal cdf. Versus Fits should not be structureless. Versus Order should not have a trend. SUTD - 21/4/2015 54
4. 2 k p Fractional Factorial Design SUTD - Capstone - 6/2/2015 55
4. 2 k p Fractional Factorial Design Best option: Use new textbook (A = +1), make 3 sessions per week (E = 1). SUTD - 21/4/2015 56
4. 2 k p Fractional Factorial Design However note that: A = A + BD + CE + FG + E = E + AC + BG + DF + The conclusion (best option is A = +1, E = 1) relies on the assumption that: BD CE FG AC BG DF 0 What if some of the listed two-level interactions are not negligible? SUTD - 21/4/2015 57
5. Folding Over You are suspicious with these interpretations and the course is about to be offered again. You plan to run a follow-up experiment. Again with 8 sections, and 10 students (replications) in each section. However, this time you change the settings in the sections. In particular, you fold over factor A (textbook), which appeared to be the most significant factor in the initial analysis. SUTD - 21/4/2015 58
5. Folding Over Folding over factor A means reversing the signs in column A while leaving the other columns unchanged. + + + + SUTD - 21/4/2015 59
4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes Section 3 hr Book New Read 3 /wk HW No SW No Sess Rev LecN 5 Current No 9 5 hr New 3 No /wk 3 No hr New Yes 4 /wk Yes No 6 New No 10 5 hr Current 4 No /wk 3 No hr Current No 3 /wk Yes Yes 7 Current Yes 11 5 hr Current New 3 Yes /wk 3 Yes hr Current No 4 /wk No Yes 8 New Yes 12 5 hr Current New 4 Yes /wk 3 Yes hr New Yes 3 /wk No No 13 New No 5 hr New 3 /wk No Yes 14 Current No 5 hr Current 4 /wk No No 15 New Yes 5 hr Current 3 /wk Yes No 16 Current Yes 5 hr New 4 /wk Yes Yes SUTD - 21/4/2015 60
5. Folding Over By folding over factor A, we isolate the main effect A from all two and threelevel interactions. Moreover, all two-level interactions including A will be isolated from other twolevel interactions. We will be more confident with the estimates of the main effect of A and all its twolevel interactions. Minitab Exercise: Section 5 in DOE-Minitab.pdf SUTD - 21/4/2015 61
5. Folding Over SUTD - 21/4/2015 62
5. Folding Over Model Adequacy SUTD - 21/4/2015 63
5. Folding Over Previous interpretation: The alias structure was: Book (A) and Session (E) are significant factors. E = E + AC + BG + DF SUTD - 21/4/2015 64
5. Folding Over SUTD - 21/4/2015 65
5. Folding Over SUTD - 21/4/2015 66
5. Folding Over With the current book, more homework is better. With the new one, less homework results in higher scores. New book performs always better than the current one. Best option: use new textbook (A = +1), assign less homework (C = 1). SUTD - 21/4/2015 67
6. More about DOE a) Model Adequacy Checking (unusual observations) b) Quadratic Curvature c) Blocking d) Response Surface Methods (Optimization) SUTD - 21/4/2015 68
6. a) Model Adequacy Checking Normal Probability Plot SUTD - 21/4/2015 69
6. a) Model Adequacy Checking Residuals versus Fits SUTD - 21/4/2015 70
6. a) Model Adequacy Checking Residuals versus Observation (Run) Order SUTD - 21/4/2015 71
6. b) Quadratic Curvature 2-level factorial design assumes linearity. Perfect linearity is unnecessary, and the 2 k system will work quite well even when the linearity assumption holds only very approximately. In fact, the model is capable of representing some curvature when interaction terms are added to the model (results from the twisting of the plane induced by the interaction terms). First-order model: SUTD - 21/4/2015 72
6. b) Quadratic Curvature In some situations, the curvature in the response function will not be adequately modeled by the first-order model. In such cases, a logical model to consider is the second-order response surface model: where β jj represents quadratic effects. SUTD - 21/4/2015 73
6. b) Quadratic Curvature Checking the existence of quadratic effects There is a method of replicating certain points in a 2 k factorial that will provide protection against curvature from second-order effects as well as allow an independent estimate of error to be obtained. The method consists of adding center points to the 2 k design. SUTD - 21/4/2015 74
6. b) Quadratic Curvature Checking the presence of quadratic effects Adding replicates at the center points: If y F y C the center points lie on or near the plane passing through the factorial points, there is no quadratic curvature. If y F y C quadratic curvature is present. SUTD - 21/4/2015 75
6. b) Quadratic Curvature Fitting the second-order model A simple and highly effective solution to this problem is to augment the 2 k with axial runs (± 2). The resulting design, called a central composite design, can now be used to fit the second-order model. SUTD - 21/4/2015 76
6. b) Quadratic Curvature Fitting the second-order model: Central Composite Design SUTD - 21/4/2015 77
6. c) Blocking Suppose an experimenter is investigating the effect of the concentration of the reactant factor A with levels 15% and 25% the amount of catalyst factor B with levels 1 pound and 2 pounds on the yield (response) in a chemical process. SUTD - 21/4/2015 78
6. c) Blocking 2 2 factorial design with 3 replicates is used (4 3 = 12 runs). Suppose only 4 experimental trials can be made from a single batch of raw material: the experimenter will use 3 batches to complete the experiment. What if the raw material affects the yield? Raw material is not a design factor since the experimenter does not want to measure its affect on the yield. However, existence of its effect may be misleading in the analysis. In such cases, we use blocking, to estimate the effect of the blocks (raw material) and cancel out its effect from our factor analysis. SUTD - 21/4/2015 79
6. d) Response Surface Methods Response surface methodology, or RSM, is a collection of mathematical, and statistical techniques useful for the modeling and analysis of problems in which a response of interest is influenced by several variables and the objective is to optimize this response. SUTD - 21/4/2015 80
6. d) Response Surface Methods For example, suppose that a chemical engineer wishes to find the levels of temperature x 1, and pressure x 2 that maximize the yield y of a process. The process yield is a function of the levels of temperature and pressure, say y = f x 1, x 2 + ε If we denote the expected response by E y = f x 1, x 2 represented by = η, then the surface η = f x 1, x 2 is called a response surface. SUTD - 21/4/2015 81
6. d) Response Surface Methods SUTD - 21/4/2015 82
6. d) Response Surface Methods RSM is a sequential procedure Often, when we are at a point on the response surface that is remote from the optimum, such as the current operating conditions. There is little curvature in the system and the first-order model will be appropriate. Our objective here is to lead the experimenter rapidly and efficiently along a path of improvement toward the general vicinity of the optimum. SUTD - 21/4/2015 83
6. d) Response Surface Methods RSM is a sequential procedure Once the region of the optimum has been found, a more elaborate model, such as the second-order model, may be employed, and an analysis may be performed to locate the optimum. The analysis of a response surface can be thought of as climbing a hill, where the top of the hill represents the point of maximum response. SUTD - 21/4/2015 84
References Montgomery, D.C., (2013). Design and Analysis of Experiments - 8th Edition, John Wiley and Sons Inc. Ledolter, J. and Swersey, A.J., (2007). Testing 1 2 3: Experimental Design with Applications in Marketing and Service Operations, Stanford University Press. SUTD - 21/4/2015 85
Exercise List the design variables and response(s) related with your project. Considering your constraints (time, budget, etc), try to identify the maximum number of experiments you can carry out. Identify a 2-level factorial design (full factorial, fractional) and the number of replicates that best fits your situation. Check the power of your experimental design using an arbitrary estimate on the standard deviation. Try to identify any factors that need to be blocked. If you aim to optimize the factor levels, suppose that you will need to fit at least one first-order, and one second-order model. SUTD - 21/4/2015 86