Design of Experiments SUTD - 21/4/2015 1

Size: px

Start display at page:

Download "Design of Experiments SUTD - 21/4/2015 1"

Monica Wiggins
5 years ago
Views:

1 Design of Experiments SUTD - 21/4/2015 1

2 Outline 1. Introduction 2. 2 k Factorial Design Exercise 3. Choice of Sample Size Exercise 4. 2 k p Fractional Factorial Design Exercise 5. Follow-up experimentation (folding over) with factorial design Exercise 6. More about DOE SUTD - 21/4/2015 2

3 1. Introduction Observing a system or process while it is in operation is an important part of the learning process, and is an integral part of understanding and learning about how systems and processes work. To understand what happens to a process when you change certain input factors, you have to do more than just watch you actually have to change the factors. This means that to really understand cause-and-effect relationships in a system you must deliberately change the input variables to the system and observe the changes in the system output that these changes to the inputs produce. In other words, you need to conduct experiments on the system. SUTD - 21/4/2015 3

4 1. Introduction Investigators perform experiments in virtually all fields of inquiry, usually to discover something about a particular process or system. Each experimental run is a test. Experiment: A test or series of runs in which purposeful changes are made to the input variables of a process or system so that we may observe and identify the reasons for changes that may be observed in the output response. SUTD - 21/4/2015 4

5 1. Introduction x i Controllable factors Inputs Process Output y Uncontrollable factors z j SUTD - Capstone - 6/2/2015 5

6 1. Introduction Objectives of the experiment: Determining which variables are most influential on the response y. Determining where to set the influential x so that y is almost always near the desired nominal value. Determining where to set the influential x so that variability in y is small. Determining where to set the influential x so that the effects of the uncontrollable variables z are minimized. SUTD - 21/4/2015 6

7 1. Introduction You are a golf player who doesn t enjoy practicing. But you also want to lower your score. SUTD - 21/4/2015 7

8 1. Introduction Factors that may affect your score: Oversized vs regular-sized driver Balata vs three-piece ball Walk vs ride Water vs sth else Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD - 21/4/2015 8

9 1. Introduction Preliminary elimination: Use your experience to eliminate the factors that will not affect the response (output) significantly. Engineers, scientists, and business analysts, often must make these types of decisions about some of the factors they are considering in real experiments. Based on your long experience with golf, you decide that the effects of the last four factors on your score are very small. SUTD - 21/4/2015 9

10 1. Introduction Factors that may affect your score: Oversized vs regular-sized driver O vs R Balata vs three-piece ball B vs T Walk vs ride W vs R Water vs sth else W vs SE Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD - 21/4/

11 1. Introduction Suppose that a maximum of eight rounds of golf can be played over the course of the experiment (resource constraints). You decide to start with the following combination: O B R W resulting score: 87 During the round, you notice several wayward shots with the big driver, so you decide to play another round with the regular-sized driver. R B R W This approach could be continued almost indefinitely. SUTD - 21/4/

12 1. Introduction Best-guess approach is frequently used in practice by engineers and scientists. It often works reasonably well, too, because the experimenters often have a great deal of technical or theoretical knowledge of the system they are studying, as well as considerable practical experience. The best-guess approach has at least two disadvantages: 1. Bad guess: Suppose the initial best-guess does not produce the desired results. Now the experimenter has to take another guess at the correct combination of factor levels. This could continue for a long time, without any guarantee of success. 2. Good guess: Suppose the initial best-guess produces an acceptable result. Now the experimenter is tempted to stop testing, although there is no guarantee that the best solution has been found. SUTD - 21/4/

13 1. Introduction One-factor-at-a-time (OFAT): The OFAT method consists of selecting a starting point, or baseline set of levels, for each factor, and then successively varying each factor over its range with the other factors held constant at the baseline level. After all tests are performed, a series of graphs are usually constructed showing how the response variable is affected by varying each factor with all other factors held constant. SUTD - 21/4/

14 1. Introduction Baseline: O B W W Optimal combination wrt OFAT: R (B or T) R - W SUTD - 21/4/

15 1. Introduction The major disadvantage of the OFAT strategy is that it fails to consider any possible interaction between the factors. An interaction is the failure of one factor to produce the same effect on the response at different levels of another factor. OFAT experiments are always less efficient than other methods based on a statistical approach to design. OFAT solution: R (B or T) R - W SUTD - 21/4/

16 1. Introduction Correct approach? 2 4 factorial design (4 factors each having 2 levels) This experiment requires 16 runs. A 10-factor experiment would require 1024 runs. Fortunately, we can use fractional factorial designs. SUTD - 21/4/

17 1. Introduction 2 2 factorial design with 2 replications: = 8 rounds D = B = = 3.25 = 0.75 DB = = 0.25 Statistical tests show that: D is significant, B and DB are not. SUTD - 21/4/

18 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD - 21/4/

19 2. 2 k Factorial Design Geometric Coding and Labels (2 3 factorial design) 8 combination of the factors, or treatments. Run A B C Labels 1 (1) 2 + a 3 + b ab 5 + c ac bc abc SUTD - 21/4/

20 2. 2 k Factorial Design Algebraic Signs for Calculating Effects Labels I A B AB C AC BC ABC (1) a b ab c ac bc abc A B = AB, AB B = AB 2 = A A = 1 4n 1 + a b + ab c + ac bc + abc SUTD - 21/4/

21 2. 2 k Factorial Design Geometric View SUTD - 21/4/

22 2. 2 k Factorial Design A chemical product is produced in a pressure vessel. A factorial experiment is carried out in the pilot plant to study the factors thought to influence the filtration rate (FR) of this product. The four factors are: A. Temperature (Temp) B. Pressure (Pres) C. Concentration of formaldehyde (Conc) D. Stirring rate (SR) Objectives: The process engineer is interested in maximizing the filtration rate. Current rates are around 75 gal/h. The process currently uses the conc. of formaldehyde, at the high level. The engineer would like to reduce the formaldehyde concentration as much as possible but has been unable to do so because it always results in lower filtration rates. SUTD - 21/4/

23 2. 2 k Factorial Design Each factor has 2 levels. We use a 2 4 factorial design. We carry out a single run for each of the 16 combinations. SUTD - 21/4/

24 2. 2 k Factorial Design Regression Model y = β 0 + β A x A + β B x B + β C x C + β D x D + β AB x A x B + β AC x A x C + β AD x A x D + β BC x B x C + β B x B x D + β CD x C x D + β ABC x A x B x C + β ABD x A x B x D + β ACD x A x C x D + β BCD x B x C x D + β ABCD x A x B x C x D SUTD - 21/4/

25 2. 2 k Factorial Design Minitab Exercise: Section 2 in DOE-Minitab.pdf SUTD - 21/4/

26 2. 2 k Factorial Design Pareto Chart Normal Plot SUTD - 21/4/

27 2. 2 k Factorial Design Fitted Regression Model Try all factors at high levels: Temp = Press = Conc = SR = +1 FR = Observed value: 96 SUTD - 21/4/

28 2. 2 k Factorial Design Main Effects Plot If we consider only the main effects, we would run all factors at high levels. SUTD - 21/4/

29 2. 2 k Factorial Design Interaction Plot SUTD - 21/4/

30 2. 2 k Factorial Design Temp (A) effect is: very small when C is high, very large when C is low. best: C = 1, A = +1 SR (D) effect is: little when A is low, large when A is high. Best combination: A = +1, C = 1, D = +1 Note that B is not significant. SUTD - 21/4/

31 2. 2 k Factorial Design Risks with Unreplicated Factorial Design Error variance is not estimable. We can never be entirely certain that the experimental error is small. A good practice in these types of experiments is to spread out the factor levels aggressively. One approach to the analysis of an unreplicated factorial is to assume that certain highorder interactions are negligible and combine their mean squares to estimate the error. This is an appeal to the sparsity of effects principle; that is, most systems are dominated by some of the main effects and low-order interactions, and most high-order interactions are negligible. Appropriate when you are confident that experimental error is small: Computer simulations with no random numbers: when same inputs always result in the same outputs (experimental error = 0). SUTD - 21/4/

32 3. Choice of Sample Size Choice of Sample Size What is the appropriate sample size (number of replications)? Depends on desired power of the experimentation, overall variance of the response, desired accuracy. SUTD - 21/4/

33 3. Choice of Sample Size Choice of Sample Size Power: β = Pr Type II error = Pr fail to reject H 0 H 0 is false power = 1 β As power increases, the chance to catch a significant effect increases. Effects: The minimum difference between ± levels of the factors that you want to detect. Minimum detectable change (MDC) size. SUTD - 21/4/

34 3. Choice of Sample Size Choice of Sample Size As the effects increases, the power increases. As the sample size get larger, the power of the test gets larger. Minitab Exercise: Section 3 in DOE-Minitab.pdf SUTD - 21/4/

35 3. Choice of Sample Size SUTD - 21/4/

36 4. 2 k p Fractional Factorial Design Fractional Design with defining relation I = ABC Labels I A B AB C AC BC ABC (1) a b ab c ac bc abc SUTD - 21/4/

37 4. 2 k p Fractional Factorial Design Fractional Design with defining relation I = ABC Labels I A B C a + + b + + c + + abc Number of runs is reduced from 8 to 4, with the cost of aliasing some effects. Resolving the aliases: A = A I = A ABC = A 2 BC = BC B = B I = B ABC = AB 2 C = AC C = C I = C ABC = ABC 2 = AB This means: A = A + BC B = B + AC C = C + AB I = I + ABC SUTD - 21/4/

38 4. 2 k p Fractional Factorial Design Geometric View If the experimenter removes any of the three factors from the analysis, the design will project into a full factorial 2 2 design. SUTD - 21/4/

39 4. 2 k p Fractional Factorial Design Improving Online Learning A university that specializes in online learning wants to improve the effectiveness of its 8-week course. Statistics group identifies 7 factors to test at 2 levels each: Code Factor - + A Textbook Current New B Readings No Yes C Homework 3 Hours 5 Hours D Software Current New E Sessions 3 per week 4 per week F Review No Yes G Lecture Notes No Yes SUTD - 21/4/

40 4. 2 k p Fractional Factorial Design Improving Online Learning Full factorial design requires 2 7 = 128 runs for a single replication. You decide to use III saturated design. 8 runs for each replication. Each of the 8 runs defines characteristics of a section. 10 students are randomly assigned to each of the 8 sections. At the end of the course, each student takes a final exam (response). SUTD - 21/4/

41 4. 2 k p Fractional Factorial Design Improving Online Learning Regression Model: y = β 0 + β A x A + β B x B + β C x C + β D x D + β E x E + β F x F + β G x G Only the main effects can be estimated. Actually, they are confounded with interactions, and we know the alias structure. SUTD - 21/4/

42 4. 2 k p Fractional Factorial Design Improving Online Learning Generators: D = AB, E = AC, F = BC, G = ABC Or equivalently: I = ABD, I = ACE, I = BCF, I = ABCG Two-level aliases of A: A = B AB = BD (first generator) A = C AC = CE (second generator) A = BC ABC = FG (third and fourth generators) A = A + BD + CE + FG + higher-order-interactions SUTD - 21/4/

43 4. 2 k p Fractional Factorial Design Improving Online Learning SUTD - 21/4/

44 4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes 3 hr New 3 /wk No No 5 Current No 5 hr New 3 /wk No Yes 6 New No 5 hr Current 4 /wk No No 7 Current Yes 5 hr Current 3 /wk Yes No 8 New Yes 5 hr New 4 /wk Yes Yes SUTD - 21/4/

45 4. 2 k p Fractional Factorial Design Minitab Exercise: Section 4 in DOE-Minitab.pdf SUTD - 21/4/

46 4. 2 k p Fractional Factorial Design Improving Online Learning SUTD - 21/4/

47 4. 2 k p Fractional Factorial Design Model Adequacy Checking Normal Probability Plot should resemble a straight line. Histogram should resemble normal cdf. Versus Fits should not be structureless. Versus Order should not have a trend. SUTD - 21/4/

48 4. 2 k p Fractional Factorial Design SUTD - Capstone - 6/2/

49 4. 2 k p Fractional Factorial Design Best option: Use new textbook (A = +1), make 3 sessions per week (E = 1). SUTD - 21/4/

50 4. 2 k p Fractional Factorial Design However note that: A = A + BD + CE + FG + E = E + AC + BG + DF + The conclusion (best option is A = +1, E = 1) relies on the assumption that: BD CE FG AC BG DF 0 What if some of the listed two-level interactions are not negligible? SUTD - 21/4/

51 5. Folding Over You are suspicious with these interpretations and the course is about to be offered again. You plan to run a follow-up experiment. Again with 8 sections, and 10 students (replications) in each section. However, this time you change the settings in the sections. In particular, you fold over factor A (textbook), which appeared to be the most significant factor in the initial analysis. SUTD - 21/4/

52 5. Folding Over Folding over factor A means reversing the signs in column A while leaving the other columns unchanged SUTD - 21/4/

4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3

5 hr Current 4 No /wk 3 No hr Current No 3 /wk Yes Yes 7 Current Yes 11 5 hr Current New 3 Yes /wk 3 Yes hr Current No 4 /wk No Yes 8 New Yes 12 5 hr Current New 4 Yes /wk 3 Yes hr

53 4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes Section 3 hr Book New Read 3 /wk HW No SW No Sess Rev LecN 5 Current No 9 5 hr New 3 No /wk 3 No hr New Yes 4 /wk Yes No 6 New No 10 5 hr Current 4 No /wk 3 No hr Current No 3 /wk Yes Yes 7 Current Yes 11 5 hr Current New 3 Yes /wk 3 Yes hr Current No 4 /wk No Yes 8 New Yes 12 5 hr Current New 4 Yes /wk 3 Yes hr New Yes 3 /wk No No 13 New No 5 hr New 3 /wk No Yes 14 Current No 5 hr Current 4 /wk No No 15 New Yes 5 hr Current 3 /wk Yes No 16 Current Yes 5 hr New 4 /wk Yes Yes SUTD - 21/4/

54 5. Folding Over By folding over factor A, we isolate the main effect A from all two and threelevel interactions. Moreover, all two-level interactions including A will be isolated from other twolevel interactions. We will be more confident with the estimates of the main effect of A and all its twolevel interactions. Minitab Exercise: Section 5 in DOE-Minitab.pdf SUTD - 21/4/

55 5. Folding Over SUTD - 21/4/

56 5. Folding Over Model Adequacy SUTD - 21/4/

57 5. Folding Over Previous interpretation: The alias structure was: Book (A) and Session (E) are significant factors. E = E + AC + BG + DF SUTD - 21/4/

58 5. Folding Over SUTD - 21/4/

59 5. Folding Over SUTD - 21/4/

60 5. Folding Over With the current book, more homework is better. With the new one, less homework results in higher scores. New book performs always better than the current one. Best option: use new textbook (A = +1), assign less homework (C = 1). SUTD - 21/4/

61 6. More about DOE a) Model Adequacy Checking (unusual observations) b) Quadratic Curvature c) Blocking d) Response Surface Methods (Optimization) SUTD - 21/4/

62 6. a) Model Adequacy Checking Normal Probability Plot SUTD - 21/4/

63 6. a) Model Adequacy Checking Residuals versus Fits SUTD - 21/4/

64 6. a) Model Adequacy Checking Residuals versus Observation (Run) Order SUTD - 21/4/

65 6. b) Quadratic Curvature 2-level factorial design assumes linearity. Perfect linearity is unnecessary, and the 2 k system will work quite well even when the linearity assumption holds only very approximately. In fact, the model is capable of representing some curvature when interaction terms are added to the model (results from the twisting of the plane induced by the interaction terms). First-order model: SUTD - 21/4/

66 6. b) Quadratic Curvature In some situations, the curvature in the response function will not be adequately modeled by the first-order model. In such cases, a logical model to consider is the second-order response surface model: where β jj represents quadratic effects. SUTD - 21/4/

67 6. b) Quadratic Curvature Checking the existence of quadratic effects There is a method of replicating certain points in a 2 k factorial that will provide protection against curvature from second-order effects as well as allow an independent estimate of error to be obtained. The method consists of adding center points to the 2 k design. SUTD - 21/4/

68 6. b) Quadratic Curvature Checking the presence of quadratic effects Adding replicates at the center points: If y F y C the center points lie on or near the plane passing through the factorial points, there is no quadratic curvature. If y F y C quadratic curvature is present. SUTD - 21/4/

69 6. b) Quadratic Curvature Fitting the second-order model A simple and highly effective solution to this problem is to augment the 2 k with axial runs (± 2). The resulting design, called a central composite design, can now be used to fit the second-order model. SUTD - 21/4/

70 6. b) Quadratic Curvature Fitting the second-order model: Central Composite Design SUTD - 21/4/

71 6. c) Blocking Suppose an experimenter is investigating the effect of the concentration of the reactant factor A with levels 15% and 25% the amount of catalyst factor B with levels 1 pound and 2 pounds on the yield (response) in a chemical process. SUTD - 21/4/

72 6. c) Blocking 2 2 factorial design with 3 replicates is used (4 3 = 12 runs). Suppose only 4 experimental trials can be made from a single batch of raw material: the experimenter will use 3 batches to complete the experiment. What if the raw material affects the yield? Raw material is not a design factor since the experimenter does not want to measure its affect on the yield. However, existence of its effect may be misleading in the analysis. In such cases, we use blocking, to estimate the effect of the blocks (raw material) and cancel out its effect from our factor analysis. SUTD - 21/4/

73 6. d) Response Surface Methods Response surface methodology, or RSM, is a collection of mathematical, and statistical techniques useful for the modeling and analysis of problems in which a response of interest is influenced by several variables and the objective is to optimize this response. SUTD - 21/4/

74 6. d) Response Surface Methods For example, suppose that a chemical engineer wishes to find the levels of temperature x 1, and pressure x 2 that maximize the yield y of a process. The process yield is a function of the levels of temperature and pressure, say y = f x 1, x 2 + ε If we denote the expected response by E y = f x 1, x 2 represented by = η, then the surface η = f x 1, x 2 is called a response surface. SUTD - 21/4/

75 6. d) Response Surface Methods SUTD - 21/4/

76 6. d) Response Surface Methods RSM is a sequential procedure Often, when we are at a point on the response surface that is remote from the optimum, such as the current operating conditions. There is little curvature in the system and the first-order model will be appropriate. Our objective here is to lead the experimenter rapidly and efficiently along a path of improvement toward the general vicinity of the optimum. SUTD - 21/4/

77 6. d) Response Surface Methods RSM is a sequential procedure Once the region of the optimum has been found, a more elaborate model, such as the second-order model, may be employed, and an analysis may be performed to locate the optimum. The analysis of a response surface can be thought of as climbing a hill, where the top of the hill represents the point of maximum response. SUTD - 21/4/

78 References Montgomery, D.C., (2013). Design and Analysis of Experiments - 8th Edition, John Wiley and Sons Inc. Ledolter, J. and Swersey, A.J., (2007). Testing 1 2 3: Experimental Design with Applications in Marketing and Service Operations, Stanford University Press. SUTD - 21/4/

79 Exercise List the design variables and response(s) related with your project. Considering your constraints (time, budget, etc), try to identify the maximum number of experiments you can carry out. Identify a 2-level factorial design (full factorial, fractional) and the number of replicates that best fits your situation. Check the power of your experimental design using an arbitrary estimate on the standard deviation. Try to identify any factors that need to be blocked. If you aim to optimize the factor levels, suppose that you will need to fit at least one first-order, and one second-order model. SUTD - 21/4/

Design of Experiments SUTD 06/04/2016 1

Design of Experiments SUTD 06/04/2016 1 Outline 1. Introduction 2. 2 k Factorial Design 3. Choice of Sample Size 4. 2 k p Fractional Factorial Design 5. Follow-up experimentation (folding over) with factorial