An Introduction to Design of Experiments

Transcription

1 An Introduction to Design of Experiments Douglas C. Montgomery Regents Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University Bradley Jones JMP Division of SAS Cary, North Carolina One-Day DOX Course 1

2 Reference Design and Analysis of Experiments, 9 th edition (2017), D.C. Montgomery, Wiley, Hoboken NJ Website: Resources for students Data (Excel, JMP, Minitab Design-Expert) Supplemental material for each chapter Resources for instructors (pwrd required) Student resources plus Power Point slides Solutions to end-of-chapter problems One-Day DOX Course 2

3 Goos, P. and Jones, B. (2011), Optimal Design of Experiments: A Case Study Approach, Wiley, UK Reference One-Day DOX Course 3

4 And this Journal of Quality Technology paper (2011)

5 Design of Engineering Experiments Part 1 Introduction Chapter 1, Text Why is this trip necessary? Goals of the course An abbreviated history of DOX Some basic principles and terminology The strategy of experimentation Guidelines for planning, conducting and analyzing experiments One-Day DOX Course 5

6 Introduction to DOX An experiment is a test or a series of tests Experiments are used widely in the engineering world Process characterization & optimization Evaluation of material properties Product design & development Component & system tolerance determination All experiments are designed experiments, some are poorly designed, some are well-designed One-Day DOX Course 6

7 Engineering Experiments Reduce time to design/develop new products & processes Improve performance of existing processes Improve reliability and performance of products Achieve product & process robustness Evaluation of materials, design alternatives, setting component & system tolerances, etc. One-Day DOX Course 7

8 Four Eras in the History of DOX The agricultural origins, s W.S. Gossett and the t-test (1908) R. A. Fisher & his co-workers Profound impact on agricultural science Factorial designs, ANOVA The first industrial era, 1951 late 1970s Box & Wilson, response surfaces Applications in the chemical & process industries The second industrial era, late 1970s 1990 Quality improvement initiatives in many companies Taguchi and robust parameter design, process robustness The modern era, beginning circa 1990 One-Day DOX Course 8

9 William Sealy Gosset ( ) Gosset's interest in barley cultivation led him to speculate that design of experiments should aim, not only at improving the average yield, but also at breeding varieties whose yield was insensitive (robust) to variation in soil and climate. Gosset was a friend of both Karl Pearson and R.A. Fisher, an achievement, for each had a monumental ego and a loathing for the other. Gosset was a modest man who cut short an admirer with the comment that Fisher would have discovered it all anyway. One-Day DOX Course 9

10 R. A. Fisher ( ) George E. P. Box ( ) One-Day DOX Course 10

11 The Basic Principles of DOX Randomization Running the trials in an experiment in random order Notion of balancing out effects of lurking variables Replication Sample size (improving precision of effect estimation, estimation of error or background noise) Replication versus repeat measurements? (see pages 12, 13) Blocking Dealing with nuisance factors One-Day DOX Course 11

12 Strategy of Experimentation Best-guess experiments Used a lot More successful than you might suspect, but there are disadvantages One-factor-at-a-time (OFAT) experiments Sometimes associated with the scientific or engineering method Devastated by interaction, also very inefficient Statistically designed experiments Based on Fisher s factorial concept One-Day DOX Course 12

13 Factorial Designs In a factorial experiment, all possible combinations of factor levels are tested The golf experiment: Type of driver Type of ball Walking vs. riding Type of beverage Time of round Weather Type of golf spike Etc, etc, etc One-Day DOX Course 13

14 Factorial Design (a 2 2 factorial) One-Day DOX Course 14

15 These are least squares estimates you ll do them by computer One-Day DOX Course 15

16 Factorial Designs with Several Factors One-Day DOX Course 16

17 One-Day DOX Course 17

18 Factorial Designs with Several Factors A Fractional Factorial One-Day DOX Course 18

19 Planning, Conducting & Analyzing an Experiment 1. Recognition of & statement of problem 2. Choice of factors, levels, and ranges 3. Selection of the response variable(s) 4. Choice of design 5. Conducting the experiment 6. Statistical analysis 7. Drawing conclusions, recommendations One-Day DOX Course 19

20 Planning, Conducting & Analyzing an Experiment Get statistical thinking involved early Your non-statistical knowledge is crucial to success Pre-experimental planning (steps 1-3) vital Think and experiment sequentially (use the KISS principle) See Coleman & Montgomery (1993) Technometrics paper + supplemental text material One-Day DOX Course 20

21 Design of Engineering Experiments The 2 k Factorial Design Text reference, Chapter 6 Special case of the general factorial design; k factors, all at two levels The two levels are usually called low and high (they could be either quantitative or qualitative) Very widely used in industrial experimentation Form a basic building block for other very useful experimental designs (DNA) Special (short-cut) methods for analysis We will make use of software One-Day DOX Course 21

22 One-Day DOX Course 22

23 The Simplest Case: The and + denote the low and high levels of a factor, respectively Low and high are arbitrary terms Geometrically, the four runs form the corners of a square Factors can be quantitative or qualitative, although their treatment in the final model will be different One-Day DOX Course 23

24 Chemical Process Example A = reactant concentration, B = catalyst amount, y = recovery One-Day DOX Course 24

25 Analysis Procedure for a Factorial Design Estimate factor effects Formulate model With replication, use full model With an unreplicated design, use normal probability plots Statistical testing (ANOVA) Refine the model Analyze residuals (graphical) Interpret results One-Day DOX Course 25

26 Estimation of Factor Effects A y y A ab a b (1) 2n 2n [ ab a b (1)] 1 2n B y y B ab b a (1) 2n 2n [ ab b a (1)] 1 2n ab (1) a b AB 2n 2n [ ab (1) a b] 1 2n A B See textbook, pg for manual calculations The effect estimates are: A = 8.33, B = -5.00, AB = 1.67 Practical interpretation? One-Day DOX Course 26

27 Statistical Testing - ANOVA The F-test for the model source is testing the significance of the overall model; that is, is either A, B, or AB or some combination of these effects important? One-Day DOX Course 27

28 JMP output, full model One-Day DOX Course 28

29 JMP output, reduced model One-Day DOX Course 29

30 Residuals and Diagnostic Checking One-Day DOX Course 30

31 The Response Surface One-Day DOX Course 31

32 One-Day DOX Course 32

33 One-Day DOX Course 33

34 Software can perform these calculations. Some JMP output is on the next slide. Also see: Jones, B. and Montgomery, D.C. (2017), Partial Replication of Small Two-Level Factorial Designs, Quality Engineering, Vol. 29, No. 3, pp One-Day DOX Course 34

35 One-Day DOX Course 35

36 The 2 3 Factorial Design One-Day DOX Course 36

37 Effects in The 2 3 Factorial Design A y y A B y y B C y y C etc, etc,... A B C These are least squares estimates Analysis done via computer One-Day DOX Course 37

38 An Example of a 2 3 Factorial Design A = gap, B = Flow, C = Power, y = Etch Rate One-Day DOX Course 38

39 Table of and + Signs for the 2 3 Factorial Design (pg. 218) One-Day DOX Course 39

40 Properties of the Table Except for column I, every column has an equal number of + and signs The sum of the product of signs in any two columns is zero Multiplying any column by I leaves that column unchanged (identity element) The product of any two columns yields a column in the table: A B AB 2 AB BC AB C AC Orthogonal design Orthogonality is an important property shared by all factorial designs One-Day DOX Course 40

41 Estimation of Factor Effects One-Day DOX Course 41

42 ANOVA Summary Full Model One-Day DOX Course 42

43 JMP Output for the full model One-Day DOX Course 43

44 Refine Model Remove Nonsignificant Factors One-Day DOX Course 44

45 Model Interpretation Cube plots are often useful visual displays of experimental results One-Day DOX Course 45

46 One-Day DOX Course 46

47 How Much Replication? Full factorial model, α = 0.05, and an effect size of two standard deviations Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 47

48 Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 48

49 The General 2 k Factorial Design Section 6-4, pg. 253, Table 6-9, pg. 25 There will be k main effects, and k two-factor interactions 2 k three-factor interactions 3 M 1 k factor interaction One-Day DOX Course 49

50 6.5 Unreplicated 2 k Factorial Designs These are 2 k factorial designs with one observation at each corner of the cube An unreplicated 2 k factorial design is also sometimes called a single replicate of the 2 k These designs are very widely used Risks if there is only one observation at each corner, is there a chance of unusual response observations spoiling the results? Modeling noise? One-Day DOX Course 50

51 Spacing of Factor Levels in the Unreplicated 2 k Factorial Designs If the factors are spaced too closely, it increases the chances that the noise will overwhelm the signal in the data More aggressive spacing is usually best One-Day DOX Course 51

52 Unreplicated 2 k Factorial Designs Lack of replication causes potential problems in statistical testing Replication admits an estimate of pure error (a better phrase is an internal estimate of error) With no replication, fitting the full model results in zero degrees of freedom for error Potential solutions to this problem Pooling high-order interactions to estimate error Normal probability plotting of effects (Daniels, 1959) Other methods Lenth s method (also see text) One-Day DOX Course 52

53 Example of an Unreplicated 2 k Design A 2 4 factorial was used to investigate the effects of four factors on the filtration rate of a resin The factors are A = temperature, B = pressure, C = mole ratio, D= stirring rate Experiment was performed in a pilot plant One-Day DOX Course 53

54 The Resin Plant Experiment One-Day DOX Course 54

55 The Resin Plant Experiment One-Day DOX Course 55

56 The image part with relationship ID rid2 was not found in the file. Estimates of the Effects One-Day DOX Course 56

57 The image part with relationship ID rid2 was not found in the file. Estimates of the Effects One-Day DOX Course 57

58 The Half-Normal Probability Plot of Effects One-Day DOX Course 58

59 The image part with relationship ID rid2 was not found in the file. Design Projection: ANOVA Summary for the Model as a 2 3 in Factors A, C, and D One-Day DOX Course 59

60 The image part with relationship ID rid2 was not found in the file. The Regression Model One-Day DOX Course 60

61 Model Residuals are Satisfactory One-Day DOX Course 61

62 Model Interpretation Main Effects and 2FI Interactions One-Day DOX Course 62

63 Model Interpretation Response Surface Plots With concentration at either the low or high level, high temperature and high stirring rate results in high filtration rates One-Day DOX Course 63

64 One-Day DOX Course 64

65 The 2 k design and design optimality The model parameter estimates in a 2 k design (and the effect estimates) are least squares estimates. For example, for a 2 2 design the model is y x x x x (1) ( 1) ( 1) ( 1)( 1) a (1) ( 1) (1)( 1) b ( 1) (1) ( 1)(1) ab (1) (1) (1)(1) The four observations from a 2 2 design (1) a y = Xβ + ε, y 1 2, X, β, ε b ab One-Day DOX Course 65

66 The least squares estimate of β is ˆ -1 β = (X X) X y (1) a b ab a ab b (1) b ab a (1) (1) a b ab (1) a b ab ˆ 4 0 (1) a b ab a ab b ( ˆ 1 1 a ab b (1) 1) 4 ˆ I 4 4 b ab a (1) 2 b ab a (1) ˆ (1) a b ab 4 12 (1) a b ab 4 The usual contrasts The X X matrix is diagonal consequences of an orthogonal design The regression coefficient estimates are exactly half of the usual effect estimates One-Day DOX Course 66

67 The matrix X X has interesting and useful properties: V ˆ 2 1 ( ) (diagonal element of ( X X) ) 2 4 Minimum possible value for a four-run design ( X X) 256 Maximum possible value for a four-run design Notice that these results depend on both the design that you have chosen and the model What about predicting the response? One-Day DOX Course 67

68 2-1 V[ yˆ ( x, x )] x (X X) x 1 2 x [1, x, x, x x ] V[ yˆ ( x1, x2)] (1 x1 x2 x1 x2 ) 4 The maximum prediction variance occurs when x 1, x 1 2 V[ yˆ ( x, x )] The prediction variance when x is V[ yˆ ( x1, x2)] 4 What about average prediction variance over the design space? x One-Day DOX Course 68

69 Average prediction variance I V yˆ x x dx dx A A [ ( 1, 2) 1 2 = area of design space = (1 x x x x ) dx dx One-Day DOX Course 69

70 One-Day DOX Course 70

71 One-Day DOX Course 71

72 For the 2 2 and in general the 2 k The design produces regression model coefficients that have the smallest variances (D-optimal design) The design results in minimizing the maximum variance of the predicted response over the design space (Goptimal design) The design results in minimizing the average variance of the predicted response over the design space (I-optimal design) One-Day DOX Course 72

73 Optimal Designs These results give us some assurance that these designs are good designs in some general ways Factorial designs typically share some (most) of these properties There are excellent computer routines for finding optimal designs (JMP is outstanding) One-Day DOX Course 73

74 Design of Engineering Experiments The 2 k-p Fractional Factorial Design Text reference, Chapter 8 Motivation for fractional factorials is obvious; as the number of factors becomes large enough to be interesting, the size of the designs grows very quickly Emphasis is on factor screening; efficiently identify the factors with large effects There may be many variables (often because we don t know much about the system) Almost always run as unreplicated factorials, but often with center points DOX Short Course 74

75 DOX Short Course 75

76 Why do Fractional Factorial Designs Work? The sparsity of effects principle There may be lots of factors, but few are important System is dominated by main effects, low-order interactions The projection property Every fractional factorial contains full factorials in fewer factors Sequential experimentation Can add runs to a fractional factorial to resolve difficulties (or ambiguities) in interpretation DOX Short Course 76

77 The One-Half Fraction of the 2 k Section 8.2, page 321 Notation: because the design has 2 k /2 runs, it s referred to as a 2 k-1 Consider a really simple case, the Note that I =ABC DOX Short Course 77

78 The One-Half Fraction of the 2 3 For the principal fraction, notice that the contrast for estimating the main effect A is exactly the same as the contrast used for estimating the BC interaction. This phenomena is called aliasing and it occurs in all fractional designs Aliases can be found directly from the columns in the table of + and - signs DOX Short Course 78

79 Aliasing in the One-Half Fraction of the 2 3 A = BC, B = AC, C = AB (or me = 2fi) Aliases can be found from the defining relation I = ABC by multiplication: AI = A(ABC) = A 2 BC = BC BI =B(ABC) = AC CI = C(ABC) = AB Textbook notation for aliased effects: [ A] A BC, [ B] B AC, [ C] C AB DOX Short Course 79

80 The Alternate Fraction of the I = -ABC is the defining relation Implies slightly different aliases: A = -BC, and C = -AB B= -AC, Both designs belong to the same family, defined by I ABC Suppose that after running the principal fraction, the alternate fraction was also run The two groups of runs can be combined to form a full factorial an example of sequential experimentation DOX Short Course 80

81 Design Resolution Resolution III Designs: me = 2fi example Resolution IV Designs: 2fi = 2fi example Resolution V Designs: 2fi = 3fi example III IV V DOX Short Course 81

82 Construction of a One-half Fraction The basic design; the design generator DOX Short Course 82

83 Projection of Fractional Factorials Every fractional factorial contains full factorials in fewer factors The flashlight analogy A one-half fraction will project into a full factorial in any k 1 of the original factors DOX Short Course 83

84 Example 8.1 DOX Short Course 84

85 Example 8.1 Interpretation of results often relies on making some assumptions Ockham s razor Confirmation experiments can be important Adding the alternate fraction see page 322 DOX Short Course 85

86 The AC and AD interactions can be verified by inspection of the cube plot DOX Short Course 86

87 Confirmation experiment for this example: see page 332 Use the model to predict the response at a test combination of interest in the design space not one of the points in the current design. Run this test combination then compare predicted and observed. For Example 8.1, consider the point +, +, -, +. The predicted response is Actual response is 104. DOX Short Course 87

88 Possible Strategies for Follow-Up Experimentation Following a Fractional Factorial Design DOX Short Course 88

89 The One-Quarter Fraction of the 2 k DOX Short Course 89

90 The One-Quarter Fraction of the Complete defining relation: I = ABCE = BCDF = ADEF DOX Short Course 90

91 The One-Quarter Fraction of the Uses of the alternate fractions E ABC, F BCD Projection of the design into subsets of the original six variables Any subset of the original six variables that is not a word in the complete defining relation will result in a full factorial design Consider ABCD (full factorial) Consider ABCE (replicated half fraction) Consider ABCF (full factorial) DOX Short Course 91

92 The General 2 k-p Fractional Factorial Design Section 8.4, page k-1 = one-half fraction, 2 k-2 = one-quarter fraction, 2 k-3 = one-eighth fraction,, 2 k-p = 1/ 2 p fraction Add p columns to the basic design; select p independent generators Important to select generators so as to maximize resolution, see Table 8.14 Projection a design of resolution R contains full factorials in any R 1 of the factors Blocking DOX Short Course 92

93 DOX Short Course 93

94 Plackett-Burman Designs These are members of a class of fractional factorials designs called non-regular designs The number of runs, N, need only be a multiple of four and the designs are resolution III N = 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, The designs where N = 12, 20, 24, etc. are called nongeometric PB designs DOX Short Course 94

95 Plackett-Burman Designs DOX Short Course 95

96 This is a nonregular design because there is partial aliasing of main effects and two-factor interactions DOX Short Course 96

97 Plackett-Burman Designs Projection of the 12-run design into 3 and 4 factors All PB designs have projectivity 3 (contrast with other resolution III fractions) The partial aliasing may allow the estimation of main effects and a few two-factor interactions DOX Short Course 97