Two-Factor Full Factorial Design with Replications

Size: px

Start display at page:

Download "Two-Factor Full Factorial Design with Replications"

Adelia Hubbard
6 years ago
Views:

1 Two-Factor Full Factorial Design with Replications Dr. John Mellor-Crummey Department of Computer Science Rice University COMP 58 Lecture 17 March 005

2 Goals for Today Understand Two-factor Full Factorial Design with Replications motivation & model model properties estimating model parameters estimating experimental errors allocating variation to factors and interactions analyzing significance of factors and interactions confidence intervals for effects confidence intervals for interactions

3 Two-factor Design with Replications Motivation two-factor full factorial design without replications helps estimate the effect of each of two factors varied assumes negligible interaction between factors effects of interactions are ignored as errors two-factor full factorial design with replications enables separation of experimental errors from interactions Example: compare several processors using several workloads factor A: processor type factor B: workload type no limits on number of levels each factor can take full factorial design study all workloads x processors Model: y ijk = µ + α j + β i + γ ij + e ijk y ijk - response with factor A at level j and factor B at level i µ - mean response α j - effect of level j for factor A β i - effect of level i for factor B γ ij - effect of interaction of factor A at level j with B at level i e ijk - error term 3

4 Model Properties Effects sums are 0 #" j = 0, # $ i = 0 Interactions j row sums are 0 a a a #" 1 j = #" j =... = #" bj = 0 j=1 j=1 column sums are 0 b Errors errors in each experiment sum to 0 i j=1 #" i1 = #" i =... = #" ia = 0 i=1 b i=1 r " e ijk = 0, #i, j k=1 b i=1 4

5 Estimating Model Parameters I Organize measured data for two-factor full factorial design as b x a matrix of cells: (i,j) = factor B at level i and factor A at level j columns = levels of factor A each cell contains r replications Begin by computing averages observations in each cell rows = levels of factor B y ij. = µ + " j + # i + $ ij ( errors = 0) each row y i.. = µ + " i ( column effects, errors, and interactions = 0) each column y. j. = µ + " j ( row effects, errors, and interactions = 0) overall y... = µ ( row & column effects, errors, and interactions = 0) 5

6 Estimating Model Parameters II Estimate effects from equations for averages mean µ = y... effect of factor A at level j y. j. = µ + " j # " j = y. j. $ y... effect of factor B at level i y i.. = µ + " i # " i = y i.. $ y... overall y ij. = µ + " j + # i + $ ij % $ ij = y ij. & y. j. & y i.. + y... How to compute model parameter estimates? use table as for two-factor full factorial design without replications, except use cell means to compute row and column effects 6

7 Example: Workload Code Size vs. Processor Code size for 5 different workloads on 4 different processors Codes written by 3 programmers with similar backgrounds Workload W X Y Z I J K L M What kind of model is needed: additive or multiplicative? 7

8 Example: Workload Code Size vs. Processor Transformed data (log 10 ) Workload W X Y Z I J K L M

9 Example: Workload Code Size vs. Processor Computing effects with replications Workload W X Y Z row sum row mean row effect I J K L M column sum column mean column effect Each entry in pale yellow is the average of cell observations Row and column effects computed as before from cell avgs Interpretation code size avg workload on avg processor = 8756 ( ) processor W requires.304 less log code size (factor of 1.70) processor Y requires.3603 more log code size (factor of.9) difference between avg code size for W and Y is factor of

10 Example: Workload Code Size vs. Processor Workload W X Y Z row sum row mean row effect I J K L M column sum column mean column effect Compute interactions as " ij = y ij. # (µ + $ j + % i ) Workload W X Y Z I J K L M Interaction constraints: row and column sums are 0 Interpretation = 1.05 interaction of I & W requires factor of < code than avg workload on W interaction of I & W requires factor of < code than I on avg processor 10

11 Estimating Experimental Errors Predicted response of (i,j) th experiment y ˆ ij = µ + " j + # i + $ ij = y ij. Prediction error = observation - cell mean e ijk = y ijk " y ˆ ij = y ijk " y ij. 11

12 Allocating Variation Total variation of y can be allocated to the two factors their interactions errors Square both sides of model equation " y ijk = abrµ + br # j i, j,k j SSY = SS0 + SSA + SSB + SSAB + SSE Total variation Compute SSE (easiest from other sums of squares) Percent variation for each factor, interaction and errors factor A: 100(SSA/SST) interactions: 100(SSAB/SST) " + ar " $ i + r " % ij + " e ijk + SST = SSY - SS0 = SSA + SSB + SSAB + SSE SSE = SSY - SS0 - SSA - SSB - SSAB i i, j i, j,k factor B: 100(SSB/SST) error: 100(SSE/SST) cross product terms (all = 0) 1

13 Allocating Variation for Code Size Study Workload W X Y Z I J K L M column effect SSY = % workload = 100*SSA/SST = 100*.93/4.44 = 66.0% row effect " y ijk = (3.8455) + (3.8191) (4.0100) = i, j,k SS0 = abrµ = (5)(4)(3)(3.943) = SST = SSY " SS0 = " = 4.44 SSA = br #" j = (5)(3)(($.304) + ($.00) + (.3603) + ($.1096) ) =.93 j SSB = ar #" j = (4)(3)((.150) + ($.475) + (.0047) + ($.0599) + (.1507) ) =1.33 j SSAB = r #" ij = (3)(($.01) + (.0399) (.0066) ) =.1548 i, j Workload W X Y Z I J K L M % interaction = 100*SSAB/SST = 100*.1548/4.44 = 3.48% % procs = 100*SSB/SST = 100*1.33/4.44 = 9.9% 13

14 Degrees of Freedom Degrees of freedom SSY = SS0 + SSA + SSB + +SSAB + SSE abr = 1 + (a-1) + (b-1) + (a-1)(b-1) + ab(r-1) sum of ( ijk ) : ab(r-1) independent sum of abr obs. all independent single term µ : repeated abr times sum of (β i ) : b-1 independent ( β i = 0) sum of (α j ) : a-1 independent ( α j = 0) sum of (γ ij ), (a-1)(b-1) independent 0 across row, down column 14

15 Analyzing Significance Divide each sum of squares by its DOF to get mean squares MSA = SSA/(a-1) MSB = SSB/(b-1) MSAB = SSAB/((a-1)(b-1)) s e = MSE = SSE/(ab(r-1)) Are effects of factors and interactions significant at α level? Test MSA/MSE against F-distribution F [1-α; a-1; ab(r-1)] Test MSB/MSE against F-distribution F [1-α; b-1; ab(r-1)] MSE Test MSAB/MSE against F-distribution F [1-α; (a-1)(b-1); ab(r-1)] Sum of Squares % var DOF Mean Square F computed F table processors workloads interactions errors for α=.1 15

16 Variance for µ Expression for µ in terms of random variables y ij s µ = 1 abr b " i=1 a " j=1 r " k=1 y ijk # Var(µ) = Var % $ 1 abr b " i=1 a " j=1 r " k=1 y ijk Assuming errors normally distributed zero mean, variance (σ e ) What is the variance for µ? & ( ' # # 1 & " µ = % abr% ( $ $ abr' = 1 abr " e & ( " e ' 16

17 Variance for α j Expression for α j in terms of random variables y ij s " j = y. j. # y... = 1 br b r $ $ y ijk # 1 i=1 k=1 abr b $ i=1 a $ l=1 r $ k=1 What is the coefficient a ikj for each y ilk for α j? a ilk = # 1 % br " 1 $ abr % " 1 & abr " # j l = j otherwise % % = br 1 br $ 1 ( ' ' * & & abr) = (a $1) abr " e y ilk Assuming errors normally distributed zero mean, variance (σ e ) What is the variance for α j? ( ) + abr $ br % ' & 1 abr ( * ) ( * " e ) 17

18 a & j=1 b & i=1 Parameter Estimation for Two Factors w/ Repl Parameter Estimate Variance µ y... s e /abr " j y.j. # y... s e (a #1) /abr $ i y i.. # y... s e (b #1) /abr % ij y ij. - y i.. # y.j. + y... s e (a #1)(b #1) /abr a & h j " j, h j = 0 b & j=1 h i $ i, h i = 0 i=1 s e a & h j y.j. s e j=1 b & h i y i.. s e i=1 (& e ) ijk /(ab(r #1)) a & b & h j=1 j h i i=1 /br /ar DOF of errors = ab(r-1) 18

19 Confidence Intervals for Effects Variance of processor effects s " j = s e (a #1) abr = #1 4 $ 5 $ 3 =.0060 Error degrees of freedom = ab(r-1) = 5 x 4 x = 40 can use unit normal variate rather than t-variate since DOF large For 90% confidence level: z.95 = " j ± z.95 s " j for processor W example: " 1 ± z.95 s "1 = #.304 ± (1.645)(.0060) = (#.406,#.03) does not include 0: effect is significant 19

20 Confidence Intervals for Effects parameter mean effect std deviation confidence interval Mu ( , ) processors W ( , ) X ( , ) Y ( , ) Z ( , ) workloads I ( 0.140, ) J ( , ) K ( , ) L ( , ) M ( , ) pink cells are not significant 0

21 Confidence Intervals for Interactions 90% confidence interval for interaction effect s " ij = s e (a #1)(b #1) /abr γ ij ± z.95 s γij Workload W X Y Z I ( , ) ( , ) ( , ) ( , 0.03 ) J ( 0.000, ) ( , ) ( , ) ( , ) K ( , ) ( 0.076, ) ( , ) ( , 0.0 ) L ( , ) ( , ) ( , 0.15 ) ( , ) M ( , ) ( , ) ( , 0.03 ) ( , ) pink cells are not significant 1

22 Prediction Error for Code Size Study

23 Quantile-Quantile Plot for Code Size Study 3

Fractional Factorial Designs

k-p Fractional Factorial Designs Fractional Factorial Designs If we have 7 factors, a 7 factorial design will require 8 experiments How much information can we obtain from fewer experiments, e.g. 7-4 =