EEC 66/75 Modelng & Performance Evaluaton of Computer Systems Lecture 3 Department of Electrcal and Computer Engneerng Cleveland State Unversty wenbng@eee.org (based on Dr. Ra Jan s lecture notes) Outlne Revew of lecture k-p Fractonal Factoral Desgns One factor experment 3 4 k r Factoral Desgns wth Replcaton k r Factoral Desgns wth Replcaton r replcatons of k experments k r observatons Allows estmaton of expermental errors Model: y q 0 + q A x A + q B x B + q AB x A x B +e e expermental error Computaton of effects: use sgn table Estmaton of errors: e y yˆ y q + q x + q x + q x x 0 A A B B AB A B Allocaton of Varaton: SST ( y y ) rq + rq + rq + e.. A B AB, SST SSA + SSB + SSAB + SSE Effects are random varables Errors ~ N(0, σ e) y ~ N( y.., σ e) q 0 s normal wth varance /( r ) σ e
Confdence Intervals for Effects 5 Multplcatve Models for r Experments 6 Varance of errors: Estmated varance of q 0 : Smlarly, s q A s q B SSE s e ee MSE ( r ) s ( r ) Δ s /( r q ) 0 e sq se r Confdence ntervals (CI) for the effects: q t [ r ] s α / ; ( ) q CI does not nclude a zero > sgnfcant AB Addtve model. Not vald f effects do not add E.g., executon tme of workloads Executon tme y v w The two effects multply. Logarthm > addtve model: log( y ) log( v ) + log( w ) Correct model: where, y' log( y ) Takng an antlog of addtve effects q. to get the multplcatve effects u. y' q+ qx + qx + q xx + e 0 A A B B AB A B k-p Fractonal Factoral Desgns 7 7-4 Desgn Large number of factors Large number of experments Full factoral desgn too expensve Use a fractonal factoral desgn Study 7 factors wth only experments! k-p desgn allows analyzng k factors wth only k-p experments k- desgn requres only half as many experments k- desgn requres only one quarter of the experments
7-4 Desgn Full factoral desgn s easy to analyze due to orthogonalty of sgn vectors Fractonal factoral desgns also use orthogonal vectors The sum of each column s zero: x 0, th varable, th experment The sum of the products of any two columns s zero x xl 0 l The sum of the squares of each column s 7-4 x 9 7-4 Desgn Model: y q + q x + q x + q x + q x + q x + q x + q x 0 A A B B Effects can be computed usng nner products q q C C y + y y + y y + y y + y y x y y + y + y y y + y + y y x A A B B D D 3 4 5 6 7 3 4 5 6 7 E E F F G G 0 7-4 Desgn Preparng Sgn Table for k-p Desgn Factors A through G explans 37.6%, 4.74%, 43.40%, 6.75%, 0%,.06%, and 0.03% of varaton, respectvely > Use only factors C and A for further expermentaton Prepare a sgn table for a full factoral desgn wth k-p factors Mark the frst column I Mark the next k-p columns wth the k-p factors Of the ( k-p k + p ) columns on the rght, choose p columns and mark them wth the p factors whch were not chosen n step 3
Example: 7-4 Desgn 3 Example: 4- Desgn 4 Start wth 3 desgn Step : k7, p4 (3 factors), sgn table> Step : mark st column I Mark next 3 columns (.e., A, B, C) wth 3 factors (.e., A, B, C) Mark AB, AC, BC, ABC wth factors D, E, F, G Start wth 3 desgn Step : Step : Step 3: Step 4: choose the rghtmost column and mark t D > can study effects q A, q B, q C, and q D along wth nteractons q AB, q AC, and q BC Confoundng: 4- Desgn Confoundng: only the combned nfluence of two or more effects can be computed q q q + q D q A D ABC q q D ABC y + y y3 + y4 y5 + y6 y7 + y yxa y + y + y3 y4 + y5 y6 y7 + y yxd y + y + y3 y4 + y5 y6 y7 + y yxaxbxc ABC y x x x A B C y + y + y3 y4 + y5 y6 y + y Effects of D and ABC are confounded. Not a problem f q ABC s neglgble 7 5 Confoundng: 4- Desgn Confoundng representaton: D ABC Other confoundngs: y + y y3 + y4 y5 + y qa qbcd yxa > A BCD A BCD, B ACD, C ABC, AB CD, AC BD, BC AD, ABC D, and I ABCD y + y I ABCD > confoundng of ABCD wth the mean 6 7 6 4
7 Other Fractonal Factoral Desgns Algebra of Confoundng A fractonal factoral desgn s not unque p dfferent fractonal factoral desgns possble Confoundng: IABD, ABD, BAD, CABCD, DAB, ACBCD, BCACD, ABCCD Not as good as the prevous desgn Gven ust one confoundng, t s possble to lst all other confoundngs Rules: I s treated as unty Any term wth a power of s erased 9 0 Algebra of Confoundng Algebra of Confoundng I ABCD Multplyng both sdes by A : A A BCD BCD Multplyng both sdes by B, C, D,and AB : B AB CD ACD C ABC D ABD D ABCD ABC AB A B CD CD and so on. generator polynomal: I ABCD For the second desgn : I ABC In a k-p desgn, p effects are confounded together In the desgn: DAB, EAC, FBC, GABC > IABD, IACE, IBCF, IABCG >IABDACEBCFABCG Usng products of all subsets: IABDACEBCFABCGBCDEACDFCDG ABEFBEGAFGDEFADEGBDFG CEFGABCDEFG Other confoundngs: ABDCEABCFBCGABCDECDFACDGBEF ABEGFGADEFDEGABDFGACEFGBCDEFG 5
Desgn Resoluton Desgn Resoluton Order of an effect number of factors ncluded n t Order of ABCD4, order of I0 Order of a confoundng sum of the order of two terms E.g., ABCDE s of order 5 Example : 4 I ABCD R IV Resoluton IV IV A BCD, B ACD, C ABD, AB CD, AC BD, BC AD, ABC D,and I ABCD Resoluton of a desgn mnmum of orders of confoundngs Notaton: Resoluton III k p R III III Example : I ABD R III desgn Desgn Resoluton 3 Case Study: Latex vs. troff 4 Example 3: I ABD ACE BCF ABCG BCDE ACDF CDG ABEF BEG AFG DEF ADEG BDFG CEFG ABCDEFG Ths s a resoluton-iii desgn A desgn of hgher resoluton s consdered a better desgn Desgn: 6- wth IBCDEF 6
Case Study: Latex vs. troff Conclusons Over 90% of the varaton s due to: bytes, program, and equatons and a second order nteracton Text fle szes were sgnfcantly dfferent makng ts effect more than that of the programs Hgh percentage of varaton explaned by the program equaton nteracton > Choce of the text formattng program depends upon the number of equatons n the text. troff not as good for equatons 5 Case Study: Latex vs. troff Conclusons (contnued) Low program bytes nteracton > Changng the fle sze affects both programs n a smlar manner In next phase, reduce range of fle szes. Alternately, ncrease the number of levels of fle szes 6 Case Study: Scheduler Desgn 7 Case Study: Scheduler Desgn Three classes of obs: word processng, data processng, and background data processng Desgn: 5- wth IABCDE Measured throughputs T W : for word processng T I : for nteractve data processng T B : for batch data processng 7
Case Study: Scheduler Desgn 9 Case Study: Scheduler Desgn 30 Effects and varaton explaned Conclusons For work processng throughput (T W ): A (preempton), B (tme slce), and AB are mportant For nteractve obs: E (farness), A (preempton), BE, and B (tme slce) For background obs: A (preempton), AB, B (tme slce), E (farness) May use dfferent polces for dfferent classes of workloads Factors C (queue assgnment) or any of ts nteracton do not have any sgnfcant mpact on the throughput Case Study: Scheduler Desgn 3 One Factor Experments 3 Conclusons (contnued): Factor D (requeung) s not effectve Preempton (A) mpacts all workloads sgnfcantly Tme slde (B) mpacts less than preempton Farness (E) s mportant for nteractve obs and slghtly mportant for background obs Used to compare alternatves of a sngle categorcal varable For example, several processors, several cachng schemes Model: y μ + α + e r number of replcatons y th response wth th alternatve μ mean response α effect of alternatve e error term α 0
Computaton of Effects r a a r a y arμ+ r α + e arμ+ 0+ 0 r a μ y y ar r r y y ( μ+ α + e ). r r r r r e 0 r μ+ α + μ+ α + α y μ y y...... 33 Example: Code Sze Comparson Entres n a row are unrelated. Otherwse, need a two factor analyss 34 Analyss of Code Sze Comparson Data 35 Interpretaton 36 Average processor requres 7.7 bytes of storage The effects of the processors R, V, and Z are - 3.3, -4.5, and 37.7, respectvely. That s R requres 3.3 bytes less than an average processor V requres 4.5 bytes less than an average processor Z requres 37.7 bytes more than an average processor 9
Estmatng Expermental Errors 37 Example 3 Estmated response for th alternatve Error: ˆ y μ + α e y yˆ Sum of squared errors (SSE): SSE r a e 44 0 30 7.7 7.7 7.7 0 44 0 7.7 7.7 7.7 76 4 7.7 7.7 7.7 374 7.7 7.7 7.7 44 7 30 7.7 7.7 7.7 3.3 4.5 37.7 30.4 6. 95.4 3.3 4.5 37.7 54.4 9. 45.4 + 3.3 4.5 37.7+.6 47. 4.4 3.3 4.5 37.7 3.6 4. 4.6 3.3 4.5 37.7 30.4 9. 76.6 SSE + + + ( 30.4) ( 54.4) (76.6) 94365.0 Allocaton of Varaton 39 Example 40 y μ + α + e + μα + μe + α e,,,, y μ+ α + e y μ + α + e + cross product terms SSY SS0+ SSA + SSE r a r a a μ μ α α SS0 ar SSA r Total varaton of y (SST): SST ( y,..,, y ) y ary SSY SS0 SSA + SSE.. SSY 44 + 0 + + 30 633639 SS0 arμ 3 5 (7.7) 5.7 SSA r α 5[( 3.3) + ( 4.5) + (37.6) ] 099. SST SSY SS0 633639.0 5.7 05357.3 SSE SST SSA 05357.3 099. 94365. Percent varaton explaned by processors 00 099.3/05357.30.4% 9.6% of varaton n code sze s due to expermental errors (programmer dfferences). Is 0.4% statstcally sgnfcant? 0
Analyss of Varance (ANOVA) 4 F-Test 4 Importance sgnfcance Important > explans a hgh percent of varaton Sgnfcance > hgh contrbuton to the varaton compared to that by errors Degree of freedom number of ndependent values requred to compute SSY SS0 + SSA + SSE ar + ( a ) + a( r ) Note that the degrees of freedom also add up Purpose: to check f SSA s sgnfcantly greater than SSE Errors are normally dstrbuted > SSE and SSA have ch-square dstrbutons The rato (SSA/ν A )/ (SSE/ν e ) has an F dstrbuton. Where ν A a degree of freedom for SSA ν e a(r ) degree of freedom for SSE Computed rato > F[ α ; ν A, ν e] > SSA s sgnfcantly hgher than SSE SSA/ν A s called mean square of A or (MSA). Smlarly, MSE SSE/ ν e ANOVA Table for One Factor Experment 43 Example: Code Sze Comparson 44 Computed F-value < F from table > the varaton n the code szes s mostly due to expermental errors and not because of any sgnfcant dfference among the processors
Vsual Dagnostc Tests 45 Example 46 Assumptons: Factors effects are addtve Errors are addtve Errors are ndependent of factor levels Errors are normally dstrbuted Errors have the same varance for all factor levels Vsual tests: Normal quantle-quantle plot: Lnear > Normalty Resduals versus predcted response: No trend > ndependence Scale of errors << scale of response > gnore vsble trends Horzontal and vertcal scales smlar Resduals are not small Varaton due to factors s small compared to the unexplaned varaton No vsble trend n the spread Example 47 Confdence Intervals for Effects 4 S-shape > shorter tals than normal Estmates are random varables For the confdence ntervals, use t values at r(a-) degree of freedom Mean response: ŷ μ + α Contrasts h α : e.g., to compare alternatves, α α
Example: Code Sze Comparson 94365. Error varance: Se 763. Std dev of errors (var. of errors).7 Std dev of μ s / ar.7 / 5.9 e Std dev of α s / {( a ) /( ar)}.7 / ( /5) 3.4 e For 90% confdence, t [0.95;].7 49 Example: Code Sze Comparson 90% confdence ntervals: μ 97.7 (.7)(.9) (46.9,.5) α 3.3 (.7)(3.4) ( 7.0, 44.4) α 4.5 (.7)(3.4) (.,33.) α 37.6 (.7)(3.4) ( 0.0,95.4) 3 The code sze on an average processor s sgnfcantly dfferent from zero Processor effects are not sgnfcant 50 Example: Code Sze Comparson 5 Example: Code Sze Comparson 5 Usng h, h -, h 3 0, ( h 0): Mean α α y y 74.4 63.... se.7 Std dev of α α 56. (/5) ( h / ar) 90% CI for α α. (.7)(56.) (.7,.) CI ncludes zero > one sn t superor to other Smlarly, 90% CI for α α 3 (74.4 5.4) (.7)(56.) ( 50.9, 4.9) 90% CI for α α 3 (63. 5.4) (.7)(56.) ( 6.,37.7) Any one processor s not superor to another 3
Unequal Sample Szes 53 Parameter Estmaton 54 Model: y By defnton: μ + α + e a r α 0 Here, r s the number of observatons at th level (alternatve) Total number of observatons: a N r Analyss of Varance 55 Example: Code Sze Comparson 56 All means are obtaned by dvdng by the number of observatons added The column effects are.5, 3.75, and -.9 4
Analyss of Varance 57 Example: Code Sze Comparson 5 44 0 30 7.5 7.5 7.5 0 44 0 7.5 7.5 7.5 76 4 7.5 7.5 7.5 7.5 7.5 44 7.5.5 3.75.9 30.4 5.00 0.33.5 3.75.9 54.4 4.00 9.67 +.5 3.75.9+.6 5.00 9.33.5 3.75 3.6 0.00.5 30.4 Sums of Squares: SSY y SS 397375 0 N 356040.75 μ SSA 5α + 4α + 3α 0.3 3 SSE + + + ( 30.40) ( 54.40) ( 9.33) 393.7 SST SSY SS0 4334.5 Degrees of freedom: SSY SS0 + SSA + SSE N + ( a ) + N a + + 9 ANOVA Table: Code Sze Comparson 59 Dervaton of Standard Devaton 60 Concluson: varaton due to processors s nsgnfcant as compared to that due to modelng errors Consder the effect of processor Z: Snce, α y y 3.3.. ( y3 + y3 + y33) ( y + y + + y3 + y4 + y3 + y3 + y33) 3 ( y3 + y3 + y33) ( y + y + + y3 + y4) 4 5
Dervaton of Standard Devaton 6 Error n α 3 errors n terms on the rght hand sze: eα ( e 3 3 + e3 + e33) ( e + e + + e3 + e4 ) 4 e 's are normally dstrbuted > α3 s normal wth sα 3s 9 06.36 3 e + s e 4 6