Design of Experiments SUTD 06/04/2016 1

Transcription

1 Design of Experiments SUTD 06/04/2016 1

2 Outline 1. Introduction 2. 2 k Factorial Design 3. Choice of Sample Size 4. 2 k p Fractional Factorial Design 5. Follow-up experimentation (folding over) with factorial design 6. More about DOE SUTD 06/04/2016 2

3 1. Introduction Observing a system or process while it is in operation is an important part of the learning process, and is an integral part of understanding and learning about how systems and processes work. To understand what happens to a process when you change certain input factors, you have to do more than just watch you actually have to change the factors. This means that to really understand cause-and-effect relationships in a system you must deliberately change the input variables to the system and observe the changes in the system output that these changes to the inputs produce. In other words, you need to conduct experiments on the system. SUTD 06/04/2016 3

4 1. Introduction Investigators perform experiments in virtually all fields of inquiry, usually to discover something about a particular process or system. Each experimental run is a test. Experiment: A test or series of runs in which purposeful changes are made to the input variables of a process or system so that we may observe and identify the reasons for changes that may be observed in the output response. This module is about planning and conducting experiments; and about analyzing the resulting data so that valid and objective conclusions are obtained. SUTD 06/04/2016 4

5 1. Introduction x i Controllable factors Inputs Process Output y Uncontrollable factors z j SUTD 06/04/2016 5

6 1. Introduction Objectives of the experiment: Determining which variables are most influential on the response y. Determining where to set the influential x so that y is almost always near the desired nominal value. Determining where to set the influential x so that variability in y is small. Determining where to set the influential x so that the effects of the uncontrollable variables z are minimized. SUTD 06/04/2016 6

7 1. Introduction You are a (lazy) golf player who doesn t enjoy practicing. But you also want to lower your score. SUTD 06/04/2016 7

8 1. Introduction Factors that may affect your score: Oversized vs regular-sized driver Balata vs three-piece ball Walk vs ride Water vs sth else Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD 06/04/2016 8

9 1. Introduction Preliminary elimination Use your experience to eliminate the factors that will not affect the response (output) significantly. Engineers, scientists, and business analysts, often must make these types of decisions about some of the factors they are considering in real experiments. Based on your long experience with golf, you decide that the effects of the last four factors on your score are very small. SUTD 06/04/2016 9

10 1. Introduction Factors that may affect your score: Oversized vs regular-sized driver O vs R Balata vs three-piece ball B vs T Walk vs ride W vs R Water vs sth else W vs SE Morning vs afternoon game Cool vs hot day Metal vs soft golf shoe spike Windy vs calm day SUTD 06/04/

11 1. Introduction A. Best Guess Approach Suppose that a maximum of eight rounds of golf can be played over the course of the experiment (resource constraints). You decide to start with the following combination: O B R W resulting score: 87 During the round, you notice several wayward shots with the big driver, so you decide to play another round with the regular-sized driver. R B R W resulting score: 89 This approach could be continued almost indefinitely. SUTD 06/04/

12 1. Introduction A. Best Guess Approach Best-guess approach is frequently used in practice by engineers and scientists. It often works reasonably well, because the experimenters often have a great deal of technical or theoretical knowledge of the system they are studying, as well as considerable practical experience. The best-guess approach has at least two disadvantages: 1. Bad guess: Suppose the initial best-guess does not produce the desired results. Now the experimenter has to take another guess at the correct combination of factor levels. This could continue for a long time, without any guarantee of success. 2. Good guess: Suppose the initial best-guess produces an acceptable result. Now the experimenter is tempted to stop testing, although there is no guarantee that the best solution has been found. SUTD 06/04/

13 1. Introduction B. One-factor-at-a-time (OFAT) The OFAT method consists of selecting a starting point, or baseline set of levels, for each factor, and then successively varying each factor over its range with the other factors held constant at the baseline level. After all tests are performed, a series of graphs are usually constructed showing how the response variable is affected by varying each factor with all other factors held constant. SUTD 06/04/

14 1. Introduction Baseline: O B W W Optimal combination wrt OFAT: R (B or T) R - W SUTD 06/04/

15 1. Introduction The major disadvantage of the OFAT strategy is that it fails to consider any possible interaction between the factors. An interaction is the failure of one factor to produce the same effect on the response at different levels of another factor. OFAT experiments are always less efficient than other methods based on a statistical approach to design. SUTD 06/04/

16 1. Introduction Correct approach? 2 4 factorial design (4 factors each having 2 levels) This experiment requires 16 runs. A 10-factor experiment would require 1024 runs. Fortunately, we can use fractional factorial designs. SUTD 06/04/

17 1. Introduction 2 2 factorial design with 2 replications: = 8 rounds D = B = = 3.25 = 0.75 DB = = 0.25 Statistical tests show that: D is significant, B and DB are not. SUTD 06/04/

18 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

19 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design Factor screening or characterization Optimization Confirmation Discovery Robustness 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

20 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data The experimenter should be certain that this variable really provides useful information about the process under study. It is usually critically important to identify issues related to the method to measure the response variables before conducting the experiment. 7. Conclusions and recommendation SUTD 06/04/

21 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment Design factors vs Nuisance factors Nuisance factors: Controllable (batches) Uncontrollable (humidity in environment) Noise 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

22 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design Consideration of Sample size (number of replicates) Selection of a suitable run order Determination of whether or not blocking or other randomization restrictions are involved. 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

23 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem Errors in experimental procedure at this stage will usually destroy experimental validity. 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

24 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges 4. Choice of experimental design 5. Performing the experiment Statistical methods should be used to analyze the data so that results and conclusions are objective rather than judgmental in nature. Often we find that simple graphical methods play an important role in data analysis and interpretation. 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

25 1. Introduction Guidelines for designing an experiment: 1. Recognition of and statement of the problem 2. Selection of the response variable 3. Choice of factors, levels, and ranges Once the data have been analyzed, the experimenter must draw practical conclusions about the results and recommend a course of action. 4. Choice of experimental design 5. Performing the experiment 6. Statistical analysis of the data 7. Conclusions and recommendation SUTD 06/04/

26 2 k Factorial Design SUTD 06/04/

27 2. 2 k Factorial Design Geometric Coding and Labels (2 3 factorial design) 8 combination of the factors, or treatments. Run A B C Labels 1 (1) 2 + a 3 + b ab 5 + c ac bc abc SUTD 06/04/

28 2. 2 k Factorial Design Algebraic Signs for Calculating Effects Labels I A B AB C AC BC ABC (1) a b ab c ac bc abc A B = AB, AB B = AB 2 = A A = 1 4n 1 + a b + ab c + ac bc + abc SUTD 06/04/

29 2. 2 k Factorial Design Geometric View SUTD 06/04/

30 2. 2 k Factorial Design A chemical product is produced in a pressure vessel. A factorial experiment is carried out in the pilot plant to study the factors thought to influence the filtration rate (FR) of this product. The four factors are: A. Temperature (Temp) B. Pressure (Pres) C. Concentration of formaldehyde (Conc) D. Stirring rate (SR) Objectives: The process engineer is interested in maximizing the filtration rate. Current rates are around 75 gal/h. The process currently uses the conc. of formaldehyde, at the high level. The engineer would like to reduce the formaldehyde concentration as much as possible; but has been unable to do so because it always results in lower filtration rates. SUTD 06/04/

31 2. 2 k Factorial Design Each factor has 2 levels. We use a 2 4 factorial design. We carry out a single run for each of the 16 combinations. SUTD 06/04/

32 2. 2 k Factorial Design Regression Model y = β 0 + β A x A + β B x B + β C x C + β D x D + β AB x A x B + β AC x A x C + β AD x A x D + β BC x B x C + β B x B x D + β CD x C x D + β ABC x A x B x C + β ABD x A x B x D + β ACD x A x C x D + β BCD x B x C x D + β ABCD x A x B x C x D SUTD 06/04/

33 2. 2 k Factorial Design Minitab Exercise: Section 2 in DOE-Minitab.pdf SUTD 06/04/

34 2. 2 k Factorial Design Pareto Chart Normal Plot SUTD 06/04/

35 2. 2 k Factorial Design Fitted Regression Model Try all factors at high levels: Temp = Press = Conc = SR = +1 FR = Observed value: 96 SUTD 06/04/

36 2. 2 k Factorial Design Main Effects Plot If we consider only the main effects, we would run all factors at high levels. SUTD 06/04/

37 2. 2 k Factorial Design Interaction Plot SUTD 06/04/

38 2. 2 k Factorial Design Temp (A) effect is: very small when C is high, very large when C is low. best: C = 1, A = +1 SR (D) effect is: little when A is low, large when A is high. Best combination: A = +1, C = 1, D = +1 Note that B is not significant. SUTD 06/04/

39 2. 2 k Factorial Design Risks with Unreplicated Factorial Design Error variance is not estimable. We can never be entirely certain that the experimental error is small. A good practice in these types of experiments is to spread out the factor levels aggressively. One approach to the analysis of an unreplicated factorial is to assume that certain highorder interactions are negligible and combine their mean squares to estimate the error. This is an appeal to the sparsity of effects principle; that is, most systems are dominated by some of the main effects and low-order interactions, and most high-order interactions are negligible. Appropriate when you are confident that experimental error is small: Computer simulations with no random numbers: same inputs always result in the same outputs (experimental error = 0). SUTD 06/04/

40 Choice of Sample Size SUTD 06/04/

41 3. Choice of Sample Size What is the appropriate sample size (number of replications)? Depends on desired power of the experimentation, overall variance of the response, desired accuracy. SUTD 06/04/

42 3. Choice of Sample Size Power: β = Pr Type II error = Pr fail to reject H 0 H 0 is false power = 1 β As power increases, the chance to catch a significant effect increases. Effects: The minimum difference between ± levels of the factors that you want to detect. Minimum detectable change (MDC) size. SUTD 06/04/

43 3. Choice of Sample Size As the effects increase, the power increases. As the sample size get larger, the power of the test increases. Minitab Exercise: Section 3 in DOE-Minitab.pdf SUTD 06/04/

44 3. Choice of Sample Size SUTD 06/04/

45 2 k p Fractional Factorial Design SUTD 06/04/

46 4. 2 k p Fractional Factorial Design Fractional Design with defining relation I = ABC Labels I A B AB C AC BC ABC (1) a b ab c ac bc abc SUTD 06/04/

47 4. 2 k p Fractional Factorial Design Fractional Design with defining relation I = ABC Labels I A B C a + + b + + c + + abc Number of runs is reduced from 8 to 4, with the cost of aliasing some effects. Resolving the aliases: A = A I = A ABC = A 2 BC = BC B = B I = B ABC = AB 2 C = AC C = C I = C ABC = ABC 2 = AB This means: A = A + BC B = B + AC C = C + AB I = I + ABC SUTD 06/04/

48 4. 2 k p Fractional Factorial Design Geometric View If the experimenter removes any of the three factors from the analysis, the design will project into a full factorial 2 2 design. SUTD 06/04/

49 4. 2 k p Fractional Factorial Design Improving Online Learning A university that specializes in online learning wants to improve the effectiveness of its 8-week course. Statistics group identifies 7 factors to test, each at 2 levels: Code Factor - + A Textbook Current New B Readings No Yes C Homework 3 Hours 5 Hours D Software Current New E Sessions 3 per week 4 per week F Review No Yes G Lecture Notes No Yes SUTD 06/04/

50 4. 2 k p Fractional Factorial Design Improving Online Learning Full factorial design requires 2 7 = 128 runs for a single replication. Assigning students to 128 sections does not seem possible. You decide to use III saturated design. 8 runs for each replication. Each of the 8 runs defines characteristics of a section. 10 students are randomly assigned to each of the 8 sections. At the end of the course, each student takes a final exam (response). SUTD 06/04/

51 4. 2 k p Fractional Factorial Design Improving Online Learning Regression Model: y = β 0 + β A x A + β B x B + β C x C + β D x D + β E x E + β F x F + β G x G Only the main effects can be estimated. Actually, they are confounded with interactions, and we know the alias structure. SUTD 06/04/

52 4. 2 k p Fractional Factorial Design Improving Online Learning Generators: D = AB, E = AC, F = BC, G = ABC Or equivalently: I = ABD, I = ACE, I = BCF, I = ABCG Two-level aliases of A: A = B AB = BD (first generator) A = C AC = CE (second generator) A = BC ABC = FG (third and fourth generators) A = A + BD + CE + FG + higher-order-interactions SUTD 06/04/

53 4. 2 k p Fractional Factorial Design Improving Online Learning SUTD 06/04/

54 4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes 3 hr New 3 /wk No No 5 Current No 5 hr New 3 /wk No Yes 6 New No 5 hr Current 4 /wk No No 7 Current Yes 5 hr Current 3 /wk Yes No 8 New Yes 5 hr New 4 /wk Yes Yes SUTD 06/04/

55 4. 2 k p Fractional Factorial Design Minitab Exercise: Section 4 in DOE-Minitab.pdf SUTD 06/04/

56 4. 2 k p Fractional Factorial Design Improving Online Learning SUTD 06/04/

57 4. 2 k p Fractional Factorial Design Model Adequacy Checking Normal Probability Plot should resemble a straight line. Histogram should resemble normal cdf. Versus Fits should not be structureless. Versus Order should not have a trend. SUTD 06/04/

58 4. 2 k p Fractional Factorial Design SUTD 06/04/

59 4. 2 k p Fractional Factorial Design Best option: Use new textbook (A = +1), make 3 sessions per week (E = 1). SUTD 06/04/

60 4. 2 k p Fractional Factorial Design However note that: A = A + BD + CE + FG + E = E + AC + BG + DF + The conclusion (best option is A = +1, E = 1) relies on the assumption that: BD CE FG AC BG DF 0 What if some of the listed two-level interactions are not negligible? SUTD 06/04/

61 Folding Over SUTD 06/04/

62 5. Folding Over You are suspicious with these interpretations and the course is about to be offered again. You plan to run a follow-up experiment. Again with 8 sections, and 10 students (replications) in each section. However, this time you change the settings in the sections. In particular, you fold over factor A (textbook), which appeared to be the most significant factor in the initial analysis. SUTD 06/04/

63 5. Folding Over Folding over factor A means reversing the signs in column A while leaving the other columns unchanged SUTD 06/04/

64 4. 2 k p Fractional Factorial Design Improving Online Learning Section Book Read HW SW Sess Rev LecN 1 Current No 3 hr New 4 /wk Yes No 2 New No 3 hr Current 3 /wk Yes Yes 3 Current Yes 3 hr Current 4 /wk No Yes 4 New Yes Section 3 hr Book New Read 3 /wk HW No SW No Sess Rev LecN 5 Current No 9 5 hr New 3 No /wk 3 No hr New Yes 4 /wk Yes No 6 New No 10 5 hr Current 4 No /wk 3 No hr Current No 3 /wk Yes Yes 7 Current Yes 11 5 hr Current New 3 Yes /wk 3 Yes hr Current No 4 /wk No Yes 8 New Yes 12 5 hr Current New 4 Yes /wk 3 Yes hr New Yes 3 /wk No No 13 New No 5 hr New 3 /wk No Yes 14 Current No 5 hr Current 4 /wk No No 15 New Yes 5 hr Current 3 /wk Yes No 16 Current Yes 5 hr New 4 /wk Yes Yes SUTD 06/04/

65 5. Folding Over By folding over factor A, we isolate the main effect A from all two and threelevel interactions. Moreover, all two-level interactions including A will be isolated from other twolevel interactions. We will be more confident with the estimates of the main effect of A and all its twolevel interactions. Minitab Exercise: Section 5 in DOE-Minitab.pdf SUTD 06/04/

66 5. Folding Over SUTD 06/04/

67 5. Folding Over Model Adequacy SUTD 06/04/

68 5. Folding Over Previous interpretation: The alias structure was: Book (A) and Session (E) are significant factors. E = E + AC + BG + DF SUTD 06/04/

69 5. Folding Over SUTD 06/04/

70 5. Folding Over SUTD 06/04/

71 5. Folding Over With the current book, more homework is better. With the new one, less homework results in higher scores. New book performs always better than the current one. Best option: use new textbook (A = +1), assign less homework (C = 1). SUTD 06/04/

72 More about DoE SUTD 06/04/

73 6. More about DOE a) Model Adequacy Checking (Bad Examples) b) Quadratic Curvature c) Blocking d) Response Surface Methods (Optimization) SUTD 06/04/

74 6. a) Model Adequacy Checking Normal Probability Plot SUTD 06/04/

75 6. a) Model Adequacy Checking Residuals versus Fits SUTD 06/04/

76 6. a) Model Adequacy Checking Residuals versus Observation (Run) Order SUTD 06/04/

77 6. b) Quadratic Curvature 2-level factorial design assumes linearity. Perfect linearity is unnecessary, and the 2 k system will work quite well even when the linearity assumption holds only very approximately. In fact, the model is capable of representing some curvature when interaction terms are added to the model (results from the twisting of the plane induced by the interaction terms). First-order model: SUTD 06/04/

78 6. b) Quadratic Curvature In some situations, the curvature in the response function will not be adequately modeled by the first-order model. In such cases, a logical model to consider is the second-order response surface model: where β jj represents quadratic effects. SUTD 06/04/

79 6. b) Quadratic Curvature Checking the existence of quadratic effects There is a method of replicating certain points in a 2 k factorial that will provide protection against curvature from second-order effects as well as allow an independent estimate of error to be obtained. The method consists of adding center points to the 2 k design. SUTD 06/04/

80 6. b) Quadratic Curvature Checking the presence of quadratic effects Adding replicates at the center points: If y F y C the center points lie on or near the plane passing through the factorial points, there is no quadratic curvature. If y F y C quadratic curvature is present. SUTD 06/04/

81 6. b) Quadratic Curvature Fitting the second-order model A simple and highly effective solution to this problem is to augment the 2 k with axial runs (± 2). The resulting design, called a central composite design, can now be used to fit the second-order model. SUTD 06/04/

82 6. b) Quadratic Curvature Fitting the second-order model: Central Composite Design SUTD 06/04/

83 6. c) Blocking Suppose an experimenter is investigating the effect of the concentration of the reactant (factor A with levels 15% and 25%), and the amount of catalyst (factor B with levels 1 pound and 2 pounds) on the yield (response) in a chemical process. SUTD 06/04/

84 6. c) Blocking 2 2 factorial design with 3 replicates is used (4 3 = 12 runs). Suppose only 4 experimental trials can be made from a single batch of raw material (the experimenter will use 3 batches to complete the experiment). What if the raw material affects the yield? Raw material is not a design factor since the experimenter does not want to measure its affect on the yield. However, existence of its effect may be misleading in the analysis. In such cases, we use blocking, to estimate the effect of the blocks (raw material) and cancel out its effect from our factor analysis. SUTD 06/04/

85 6. d) Response Surface Methods Suppose that two factors that affect the response y of your design: the width of the product w, the thickness of your product t. These are your design variables. You have the constraints that: w 10, 100 centimeters t 5, 10 millimetres How do you find w and t that maximizes y? SUTD 06/04/

86 6. d) Response Surface Methods Response surface methodology, or RSM, is a collection of mathematical, and statistical techniques useful for the modeling and analysis of problems in which a response of interest is influenced by several variables and the objective is to optimize this response. SUTD 06/04/

87 6. d) Response Surface Methods For example, suppose that a chemical engineer wishes to find the levels of temperature x 1, and pressure x 2 that maximize the yield y of a process. The process yield is a function of the levels of temperature and pressure, say y = f x 1, x 2 + ε If we denote the expected response by E y = f x 1, x 2 represented by = η, then the surface η = f x 1, x 2 is called a response surface. SUTD 06/04/

88 6. d) Response Surface Methods SUTD 06/04/

89 6. d) Response Surface Methods RSM is a sequential procedure Often, when we are at a point on the response surface that is remote from the optimum, such as the current operating conditions; there is little curvature in the system and the first-order model will be appropriate. Our objective here is to lead the experimenter rapidly and efficiently along a path of improvement toward the general vicinity of the optimum. SUTD 06/04/

90 6. d) Response Surface Methods RSM is a sequential procedure Once the region of the optimum has been found, a more elaborate model, such as the second-order model, may be employed, and an analysis may be performed to locate the optimum. The analysis of a response surface can be thought of as climbing a hill, where the top of the hill represents the point of maximum response. SUTD 06/04/

91 References Montgomery, D.C., (2013). Design and Analysis of Experiments - 8th Edition, John Wiley and Sons Inc. Ledolter, J. and Swersey, A.J., (2007). Testing 1 2 3: Experimental Design with Applications in Marketing and Service Operations, Stanford University Press. SUTD 06/04/

92 Final Exercise List the design variables, factors and response(s) related with your project. Identify also the alternative values that each design variable can attain. Considering your constraints (time, budget, etc), try to identify the maximum number of experiments you can carry out. Identify a 2-level factorial design (full factorial, fractional) and the number of replicates that best fits your situation. Check the power of your experimental design using an arbitrary estimate on the standard deviation. Try to identify any factors that need to be blocked. SUTD 06/04/