An Introduction to Design of Experiments

An Introduction to Design of Experiments Douglas C. Montgomery Regents Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University Bradley Jones JMP Division of SAS Cary, North Carolina One-Day DOX Course 1

Reference Design and Analysis of Experiments, 9 th edition (2017), D.C. Montgomery, Wiley, Hoboken NJ Website: www.wiley.com/college/montgomery Resources for students Data (Excel, JMP, Minitab Design-Expert) Supplemental material for each chapter Resources for instructors (pwrd required) Student resources plus Power Point slides Solutions to end-of-chapter problems One-Day DOX Course 2

Goos, P. and Jones, B. (2011), Optimal Design of Experiments: A Case Study Approach, Wiley, UK Reference One-Day DOX Course 3

And this Journal of Quality Technology paper (2011)

Design of Engineering Experiments Part 1 Introduction Chapter 1, Text Why is this trip necessary? Goals of the course An abbreviated history of DOX Some basic principles and terminology The strategy of experimentation Guidelines for planning, conducting and analyzing experiments One-Day DOX Course 5

Introduction to DOX An experiment is a test or a series of tests Experiments are used widely in the engineering world Process characterization & optimization Evaluation of material properties Product design & development Component & system tolerance determination All experiments are designed experiments, some are poorly designed, some are well-designed One-Day DOX Course 6

Engineering Experiments Reduce time to design/develop new products & processes Improve performance of existing processes Improve reliability and performance of products Achieve product & process robustness Evaluation of materials, design alternatives, setting component & system tolerances, etc. One-Day DOX Course 7

Four Eras in the History of DOX The agricultural origins, 1908 1940s W.S. Gossett and the t-test (1908) R. A. Fisher & his co-workers Profound impact on agricultural science Factorial designs, ANOVA The first industrial era, 1951 late 1970s Box & Wilson, response surfaces Applications in the chemical & process industries The second industrial era, late 1970s 1990 Quality improvement initiatives in many companies Taguchi and robust parameter design, process robustness The modern era, beginning circa 1990 One-Day DOX Course 8

William Sealy Gosset (1876-1937) Gosset's interest in barley cultivation led him to speculate that design of experiments should aim, not only at improving the average yield, but also at breeding varieties whose yield was insensitive (robust) to variation in soil and climate. Gosset was a friend of both Karl Pearson and R.A. Fisher, an achievement, for each had a monumental ego and a loathing for the other. Gosset was a modest man who cut short an admirer with the comment that Fisher would have discovered it all anyway. One-Day DOX Course 9

R. A. Fisher (1890 1962) George E. P. Box (1919 2013) One-Day DOX Course 10

The Basic Principles of DOX Randomization Running the trials in an experiment in random order Notion of balancing out effects of lurking variables Replication Sample size (improving precision of effect estimation, estimation of error or background noise) Replication versus repeat measurements? (see pages 12, 13) Blocking Dealing with nuisance factors One-Day DOX Course 11

Strategy of Experimentation Best-guess experiments Used a lot More successful than you might suspect, but there are disadvantages One-factor-at-a-time (OFAT) experiments Sometimes associated with the scientific or engineering method Devastated by interaction, also very inefficient Statistically designed experiments Based on Fisher s factorial concept One-Day DOX Course 12

Factorial Designs In a factorial experiment, all possible combinations of factor levels are tested The golf experiment: Type of driver Type of ball Walking vs. riding Type of beverage Time of round Weather Type of golf spike Etc, etc, etc One-Day DOX Course 13

Factorial Design (a 2 2 factorial) One-Day DOX Course 14

These are least squares estimates you ll do them by computer One-Day DOX Course 15

Factorial Designs with Several Factors One-Day DOX Course 16

One-Day DOX Course 17

Factorial Designs with Several Factors A Fractional Factorial One-Day DOX Course 18

Planning, Conducting & Analyzing an Experiment 1. Recognition of & statement of problem 2. Choice of factors, levels, and ranges 3. Selection of the response variable(s) 4. Choice of design 5. Conducting the experiment 6. Statistical analysis 7. Drawing conclusions, recommendations One-Day DOX Course 19

Planning, Conducting & Analyzing an Experiment Get statistical thinking involved early Your non-statistical knowledge is crucial to success Pre-experimental planning (steps 1-3) vital Think and experiment sequentially (use the KISS principle) See Coleman & Montgomery (1993) Technometrics paper + supplemental text material One-Day DOX Course 20

Design of Engineering Experiments The 2 k Factorial Design Text reference, Chapter 6 Special case of the general factorial design; k factors, all at two levels The two levels are usually called low and high (they could be either quantitative or qualitative) Very widely used in industrial experimentation Form a basic building block for other very useful experimental designs (DNA) Special (short-cut) methods for analysis We will make use of software One-Day DOX Course 21

One-Day DOX Course 22

The Simplest Case: The 2 2 - and + denote the low and high levels of a factor, respectively Low and high are arbitrary terms Geometrically, the four runs form the corners of a square Factors can be quantitative or qualitative, although their treatment in the final model will be different One-Day DOX Course 23

Chemical Process Example A = reactant concentration, B = catalyst amount, y = recovery One-Day DOX Course 24

Analysis Procedure for a Factorial Design Estimate factor effects Formulate model With replication, use full model With an unreplicated design, use normal probability plots Statistical testing (ANOVA) Refine the model Analyze residuals (graphical) Interpret results One-Day DOX Course 25

Estimation of Factor Effects A y y A ab a b (1) 2n 2n [ ab a b (1)] 1 2n B y y B ab b a (1) 2n 2n [ ab b a (1)] 1 2n ab (1) a b AB 2n 2n [ ab (1) a b] 1 2n A B See textbook, pg. 235-236 for manual calculations The effect estimates are: A = 8.33, B = -5.00, AB = 1.67 Practical interpretation? One-Day DOX Course 26

Statistical Testing - ANOVA The F-test for the model source is testing the significance of the overall model; that is, is either A, B, or AB or some combination of these effects important? One-Day DOX Course 27

JMP output, full model One-Day DOX Course 28

JMP output, reduced model One-Day DOX Course 29

Residuals and Diagnostic Checking One-Day DOX Course 30

The Response Surface One-Day DOX Course 31

One-Day DOX Course 32

One-Day DOX Course 33

Software can perform these calculations. Some JMP output is on the next slide. Also see: Jones, B. and Montgomery, D.C. (2017), Partial Replication of Small Two-Level Factorial Designs, Quality Engineering, Vol. 29, No. 3, pp. 190-195. One-Day DOX Course 34

One-Day DOX Course 35

The 2 3 Factorial Design One-Day DOX Course 36

Effects in The 2 3 Factorial Design A y y A B y y B C y y C etc, etc,... A B C These are least squares estimates Analysis done via computer One-Day DOX Course 37

An Example of a 2 3 Factorial Design A = gap, B = Flow, C = Power, y = Etch Rate One-Day DOX Course 38

Table of and + Signs for the 2 3 Factorial Design (pg. 218) One-Day DOX Course 39

Properties of the Table Except for column I, every column has an equal number of + and signs The sum of the product of signs in any two columns is zero Multiplying any column by I leaves that column unchanged (identity element) The product of any two columns yields a column in the table: A B AB 2 AB BC AB C AC Orthogonal design Orthogonality is an important property shared by all factorial designs One-Day DOX Course 40

Estimation of Factor Effects One-Day DOX Course 41

ANOVA Summary Full Model One-Day DOX Course 42

JMP Output for the full model One-Day DOX Course 43

Refine Model Remove Nonsignificant Factors One-Day DOX Course 44

Model Interpretation Cube plots are often useful visual displays of experimental results One-Day DOX Course 45

One-Day DOX Course 46

How Much Replication? Full factorial model, α = 0.05, and an effect size of two standard deviations Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 47

Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 48

The General 2 k Factorial Design Section 6-4, pg. 253, Table 6-9, pg. 25 There will be k main effects, and k two-factor interactions 2 k three-factor interactions 3 M 1 k factor interaction One-Day DOX Course 49

6.5 Unreplicated 2 k Factorial Designs These are 2 k factorial designs with one observation at each corner of the cube An unreplicated 2 k factorial design is also sometimes called a single replicate of the 2 k These designs are very widely used Risks if there is only one observation at each corner, is there a chance of unusual response observations spoiling the results? Modeling noise? One-Day DOX Course 50

Spacing of Factor Levels in the Unreplicated 2 k Factorial Designs If the factors are spaced too closely, it increases the chances that the noise will overwhelm the signal in the data More aggressive spacing is usually best One-Day DOX Course 51

Unreplicated 2 k Factorial Designs Lack of replication causes potential problems in statistical testing Replication admits an estimate of pure error (a better phrase is an internal estimate of error) With no replication, fitting the full model results in zero degrees of freedom for error Potential solutions to this problem Pooling high-order interactions to estimate error Normal probability plotting of effects (Daniels, 1959) Other methods Lenth s method (also see text) One-Day DOX Course 52

Example of an Unreplicated 2 k Design A 2 4 factorial was used to investigate the effects of four factors on the filtration rate of a resin The factors are A = temperature, B = pressure, C = mole ratio, D= stirring rate Experiment was performed in a pilot plant One-Day DOX Course 53

The Resin Plant Experiment One-Day DOX Course 54

The Resin Plant Experiment One-Day DOX Course 55

The image part with relationship ID rid2 was not found in the file. Estimates of the Effects One-Day DOX Course 56

The image part with relationship ID rid2 was not found in the file. Estimates of the Effects One-Day DOX Course 57

The Half-Normal Probability Plot of Effects One-Day DOX Course 58

The image part with relationship ID rid2 was not found in the file. Design Projection: ANOVA Summary for the Model as a 2 3 in Factors A, C, and D One-Day DOX Course 59

The image part with relationship ID rid2 was not found in the file. The Regression Model One-Day DOX Course 60

Model Residuals are Satisfactory One-Day DOX Course 61

Model Interpretation Main Effects and 2FI Interactions One-Day DOX Course 62

Model Interpretation Response Surface Plots With concentration at either the low or high level, high temperature and high stirring rate results in high filtration rates One-Day DOX Course 63

One-Day DOX Course 64

The 2 k design and design optimality The model parameter estimates in a 2 k design (and the effect estimates) are least squares estimates. For example, for a 2 2 design the model is y x x x x 0 1 1 2 2 12 1 2 (1) ( 1) ( 1) ( 1)( 1) 0 1 2 12 1 a (1) ( 1) (1)( 1) 0 1 2 12 2 b ( 1) (1) ( 1)(1) 0 1 2 12 3 ab (1) (1) (1)(1) 0 1 2 12 4 The four observations from a 2 2 design (1) 1 1 1 1 0 1 a 1 1 1 1 y = Xβ + ε, y 1 2, X, β, ε b 1 1 1 1 2 3 ab 1 1 1 1 12 4 One-Day DOX Course 65

The least squares estimate of β is ˆ -1 β = (X X) X y 1 4 0 0 0 (1) a b ab 0 4 0 0 a ab b (1) 0 0 4 0 b ab a (1) 0 0 0 4 (1) a b ab (1) a b ab ˆ 4 0 (1) a b ab a ab b ( ˆ 1 1 a ab b (1) 1) 4 ˆ I 4 4 b ab a (1) 2 b ab a (1) ˆ (1) a b ab 4 12 (1) a b ab 4 The usual contrasts The X X matrix is diagonal consequences of an orthogonal design The regression coefficient estimates are exactly half of the usual effect estimates One-Day DOX Course 66

The matrix X X has interesting and useful properties: V ˆ 2 1 ( ) (diagonal element of ( X X) ) 2 4 Minimum possible value for a four-run design ( X X) 256 Maximum possible value for a four-run design Notice that these results depend on both the design that you have chosen and the model What about predicting the response? One-Day DOX Course 67

2-1 V[ yˆ ( x, x )] x (X X) x 1 2 x [1, x, x, x x ] 1 2 1 2 1 2 2 2 2 2 2 V[ yˆ ( x1, x2)] (1 x1 x2 x1 x2 ) 4 The maximum prediction variance occurs when x 1, x 1 2 V[ yˆ ( x, x )] The prediction variance when x 1 2 0 is 1 2 2 V[ yˆ ( x1, x2)] 4 What about average prediction variance over the design space? x One-Day DOX Course 68

Average prediction variance 1 1 1 I V yˆ x x dx dx A A 1 1 1 1 1 1 2 [ ( 1, 2) 1 2 = area of design space = 2 4 1 1 (1 x x x x ) dx dx 4 4 2 4 9 2 2 2 2 2 1 2 1 2 1 2 One-Day DOX Course 69

One-Day DOX Course 70

One-Day DOX Course 71

For the 2 2 and in general the 2 k The design produces regression model coefficients that have the smallest variances (D-optimal design) The design results in minimizing the maximum variance of the predicted response over the design space (Goptimal design) The design results in minimizing the average variance of the predicted response over the design space (I-optimal design) One-Day DOX Course 72

Optimal Designs These results give us some assurance that these designs are good designs in some general ways Factorial designs typically share some (most) of these properties There are excellent computer routines for finding optimal designs (JMP is outstanding) One-Day DOX Course 73

Design of Engineering Experiments The 2 k-p Fractional Factorial Design Text reference, Chapter 8 Motivation for fractional factorials is obvious; as the number of factors becomes large enough to be interesting, the size of the designs grows very quickly Emphasis is on factor screening; efficiently identify the factors with large effects There may be many variables (often because we don t know much about the system) Almost always run as unreplicated factorials, but often with center points DOX Short Course 74

DOX Short Course 75

Why do Fractional Factorial Designs Work? The sparsity of effects principle There may be lots of factors, but few are important System is dominated by main effects, low-order interactions The projection property Every fractional factorial contains full factorials in fewer factors Sequential experimentation Can add runs to a fractional factorial to resolve difficulties (or ambiguities) in interpretation DOX Short Course 76

The One-Half Fraction of the 2 k Section 8.2, page 321 Notation: because the design has 2 k /2 runs, it s referred to as a 2 k-1 Consider a really simple case, the 2 3-1 Note that I =ABC DOX Short Course 77

The One-Half Fraction of the 2 3 For the principal fraction, notice that the contrast for estimating the main effect A is exactly the same as the contrast used for estimating the BC interaction. This phenomena is called aliasing and it occurs in all fractional designs Aliases can be found directly from the columns in the table of + and - signs DOX Short Course 78

Aliasing in the One-Half Fraction of the 2 3 A = BC, B = AC, C = AB (or me = 2fi) Aliases can be found from the defining relation I = ABC by multiplication: AI = A(ABC) = A 2 BC = BC BI =B(ABC) = AC CI = C(ABC) = AB Textbook notation for aliased effects: [ A] A BC, [ B] B AC, [ C] C AB DOX Short Course 79

The Alternate Fraction of the 2 3-1 I = -ABC is the defining relation Implies slightly different aliases: A = -BC, and C = -AB B= -AC, Both designs belong to the same family, defined by I ABC Suppose that after running the principal fraction, the alternate fraction was also run The two groups of runs can be combined to form a full factorial an example of sequential experimentation DOX Short Course 80

Design Resolution Resolution III Designs: me = 2fi example Resolution IV Designs: 2fi = 2fi example Resolution V Designs: 2fi = 3fi example 3 1 2 III 4 1 2 IV 5 1 2 V DOX Short Course 81

Construction of a One-half Fraction The basic design; the design generator DOX Short Course 82

Projection of Fractional Factorials Every fractional factorial contains full factorials in fewer factors The flashlight analogy A one-half fraction will project into a full factorial in any k 1 of the original factors DOX Short Course 83

Example 8.1 DOX Short Course 84

Example 8.1 Interpretation of results often relies on making some assumptions Ockham s razor Confirmation experiments can be important Adding the alternate fraction see page 322 DOX Short Course 85

The AC and AD interactions can be verified by inspection of the cube plot DOX Short Course 86

Confirmation experiment for this example: see page 332 Use the model to predict the response at a test combination of interest in the design space not one of the points in the current design. Run this test combination then compare predicted and observed. For Example 8.1, consider the point +, +, -, +. The predicted response is Actual response is 104. DOX Short Course 87

Possible Strategies for Follow-Up Experimentation Following a Fractional Factorial Design DOX Short Course 88

The One-Quarter Fraction of the 2 k DOX Short Course 89

The One-Quarter Fraction of the 2 6-2 Complete defining relation: I = ABCE = BCDF = ADEF DOX Short Course 90

The One-Quarter Fraction of the 2 6-2 Uses of the alternate fractions E ABC, F BCD Projection of the design into subsets of the original six variables Any subset of the original six variables that is not a word in the complete defining relation will result in a full factorial design Consider ABCD (full factorial) Consider ABCE (replicated half fraction) Consider ABCF (full factorial) DOX Short Course 91

The General 2 k-p Fractional Factorial Design Section 8.4, page 340 2 k-1 = one-half fraction, 2 k-2 = one-quarter fraction, 2 k-3 = one-eighth fraction,, 2 k-p = 1/ 2 p fraction Add p columns to the basic design; select p independent generators Important to select generators so as to maximize resolution, see Table 8.14 Projection a design of resolution R contains full factorials in any R 1 of the factors Blocking DOX Short Course 92

DOX Short Course 93

Plackett-Burman Designs These are members of a class of fractional factorials designs called non-regular designs The number of runs, N, need only be a multiple of four and the designs are resolution III N = 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, The designs where N = 12, 20, 24, etc. are called nongeometric PB designs DOX Short Course 94

Plackett-Burman Designs DOX Short Course 95

This is a nonregular design because there is partial aliasing of main effects and two-factor interactions DOX Short Course 96

Plackett-Burman Designs Projection of the 12-run design into 3 and 4 factors All PB designs have projectivity 3 (contrast with other resolution III fractions) The partial aliasing may allow the estimation of main effects and a few two-factor interactions DOX Short Course 97