EEC 686/785 Modlng & Prformanc Evaluaton of Computr Systms Lctur Dpartmnt of Elctrcal and Computr Engnrng Clvland Stat Unvrsty wnbng@.org (basd on Dr. Ra Jan s lctur nots) Outln Rvw of lctur k r Factoral Dsgns wth Rplcaton 8 Octobr 005 EEC686/785
Trmnology Rspons varabls: outcom Factors: varabls that affct th rspons varabl Lvls: th valus that a factor can assum Prmary factors and scondary factors Rplcaton: rptton of all or som prmnts Dsgn: th numbr of prmnts, th factor lvl and numbr of rplcatons for ach prmnt Eprmntal Unt Intracton: ffct of on factor dpnds upon th lvl of th othr 3 8 Octobr 005 EEC686/785 Typs of Eprmntal Dsgns Smpl Dsgns: vary on factor at a tm Full Factoral Dsgn: all combnatons Fractonal Factoral Dsgns: us only a fracton of th full factoral dsgn 8 Octobr 005 EEC686/785
k Factoral Dsgns 5 k factors, ach at two lvls Easy to analyz Hlps n sortng out mpact of factors Good at th bgnnng of a study Vald only f th ffct of a factor s undrctonal,.., th prformanc thr contnuously dcrass or contnuously ncrass as th factor s ncrasd from mn to ma E.g., mmory sz, th numbr of dsk drvs 8 Octobr 005 EEC686/785 Factoral Dsgns: Modl 6 y 0 5 0 5 0 5 0 75 0 Unu soluton for and : y 0 0 0 5 Intrprtaton: Man prformanc 0 MIPS Effct of mmory 0 MIPS Effct of cach 0 MIPS Intracton btwn mmory and cach 5 MIPS 8 Octobr 005 EEC686/785 3
llocaton of Varaton 7 Importanc of a factor proporton of th varaton pland ( ) Sampl Varanc of y y y sy Varaton of y Δ Numrator For a dsgn ( y y) Varaton du to SS, tc. sum of suars total ( SST ) SST 8 Octobr 005 EEC686/785 Drvaton 8 Modl: y Notc 0 Th sum of ntrs n ach column s zro Th sum of th suars of ntrs n ach column s Th columns ar orthogonal (nnr product of any two columns s zro): 0; ( ) 0; ( ) 0 8 Octobr 005 EEC686/785
5 8 Octobr 005 EEC686/785 9 Drvaton Varaton of y 0 ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( trms product y y 8 Octobr 005 EEC686/785 0 k r Factoral Dsgns wth Rplcaton r rplcatons of k prmnts k r obsrvatons llows stmaton of prmntal rrors Modl: y 0 prmntal rror
Computaton of Effcts Mmory-Cach Study: Smply us mans of r masurmnts Effcts: 0,.5, 9.5, 5 8 Octobr 005 EEC686/785 Estmaton of Eprmntal Errors Estmatd Rspons: y ˆ 0 Eprmntal rror stmatd masurd y yˆ 0 Sum of suard rrors: y 8 Octobr 005 EEC686/785 0 SSE r 6
Eprmntal Errors: Eampl 3 Estmatd Rspons: yˆ 0.5 9.5 5 5 Eprmntal rrors: y yˆ 5 5 0 8 Octobr 005 EEC686/785 llocaton of Varaton y.. dnots th man of rsponss from all rplcatons of all prmnts Th dots n th subscrpt ndcat th dmnson along whch th avragng s don dnots th mans of rsponss n all rplcatons of th th prmnt y. Total varaton or total sum of suars (SST): ( y SST y ).. SST y ( y 0 r SS y ).. r SS r SS SSE 8 Octobr 005 EEC686/785 7
Modl: Drvaton y 0 y 0 Snc s, thr products, and all rrors add to 0 y 0 r0 Man rspons: y.. y 0 r Suarng both sds of th modl and gnorng cross product trms: y 0 SSY SS0 SS SS SS SSE 5 8 Octobr 005 EEC686/785 Drvaton Total varaton: SST ( y y y ) y.. SSY SS0 SS SS.. SS SSE 6 SSE SSE SSY r( 0 ) 8 Octobr 005 EEC686/785 8
Eampl: Mmory-Cach Study 7 SSY 5 SS0 r SS r SS r SS r 8 0 (.5) (9.5) 5 5 07 083 300 557 SSE 70 3(.5 9.5 75 SST SSY SS0 70 07 703 8 5 ) 0 70 SSSSSSSSE 5570833000 703 SST 75 8 Octobr 005 EEC686/785 Eampl: Mmory-Cach Study 8 Factor plans 557/703 or 78.88% Factor plans 5.0% Intracton plans.7%.5% s unpland and s attrbutd to rrors 8 Octobr 005 EEC686/785 9
Confdnc Intrvals for Effcts Effcts ar random varabls Errors ~ N( 0, σ ) y ~ N( y, σ ) 0 r y.. 9 0 Lnar combnaton of normal varats > 0 s normal wth varanc /( r ) σ 8 Octobr 005 EEC686/785 Confdnc Intrvals for Effcts 0 Varanc of rrors: SSE s ΔMSE ( r ) ( r ) Man Suar of Errors (MSE): uantty at th rght hand sd Dnomnator (r-) # of ndpndnt trms n SSE > dgrs of frdom SSE has 8 Octobr 005 EEC686/785 0
Confdnc Intrvals for Effcts Estmatd varanc of 0 : Smlarly, s s s s s /( r ) 0 s Confdnc ntrvals (CI) for th ffcts: t [ r ] s α / ; ( ) CI dos not nclud a zro > sgnfcant r 8 Octobr 005 EEC686/785 Eampl For mmory-cach study: Standard dvaton of rrors: s SSE 0.75 3.57 ( r ) 8 Standard dvaton of ffcts: s s 3.57.03 ( r) For 90% confdnc: t [0.95,8].86 8 Octobr 005 EEC686/785
Eampl 3 Confdnc ntrvals: ± (.86)(.03) ±.9 0 (39.08,.9) (9.58, 3.) (7.58,.) (3.08, 6.9) No zro crossng > all ffcts ar sgnfcant 8 Octobr 005 EEC686/785 Confdnc Intrvals for Contrasts It s also possbl to comput varanc and confdnc ntrvals for any contrast of ffcts. contrast s any lnar combnaton whos coffcnts add up to zro Varanc of h, whr h 0, s: s s h r h For 00(-α)% confdnc ntrval, us t α [ / ; ( r ) ] 8 Octobr 005 EEC686/785
Eampl 8.5 5 Mmory-cach study: th confdnc ntrval for u - s calculatd as follows: Coffcnts 0,,, and - > Contrast Man u.5 9.5 5 Varanc s 6 s u 6.375 3 Standard dvaton t [0.95;8].86 s u 90% confdnc ntrval for u: u ts u 6.375.5.86.5 (6.3, 5.69) 8 Octobr 005 EEC686/785 CI for Prdctd Rsponss 6 Man rspons y ˆ 0 Th standard dvaton of th man of m rsponss: s n ˆ y m ff s nff m ffctv dgr of frdom total numbr of runs sum of DFs of r 5 / params usd n yˆ 8 Octobr 005 EEC686/785 3
CI for Prdctd Rsponss 7 00(-α)% confdnc ntrval: yˆ t [ r ] s α / ; ( ) y ˆm sngl run (m): Populaton man (m ): 5 syˆ s r s yˆ s / 5 r / 8 Octobr 005 EEC686/785 Eampl: Mmory-Cach Study 8 For - and -: sngl confrmaton prmnt: yˆ 0.5 9.5 5 5 Standard dvaton of th prdcaton: / 5 5 sy s 3.57.5 ˆ r Usng t [0.95;8].86, th 90% confdnc ntrval s: 5±.86.5(8.09,.9) 8 Octobr 005 EEC686/785
Eampl: Mmory-Cach Study 9 For - and - (contnud): Man rspons for 5 prmnts n futur: / 5 5 sy s 3.57.80 ˆ r m 5 Th 90% confdnc ntrval s: 5±.86.80(9.79, 0.9) 8 Octobr 005 EEC686/785 Eampl: Mmory-Cach Study 30 For - and - (contnud): Man rspons for a larg numbr of prmnts n futur: / 5 5 syˆ s 3.57.30 r Th 90% confdnc ntrval s: 5±.86.30(0.7, 9.8) 8 Octobr 005 EEC686/785 5
Eampl: Mmory-Cach Study 3 For - and - (contnud): Currnt man rspons: not for futur (us th formula for contrasts): s h.75 syˆ.06 r Th 90% confdnc ntrval s: 5±.86.06(.7, 8.83) Notc: Confdnc ntrvals bcom narrowr 8 Octobr 005 EEC686/785 3 Vsual Tsts for Vrfyng th ssumptons In drvng th prssons for ffcts, w mad ssntally th sam assumptons as n rgrsson analyss: Errors ar statstcally ndpndnt Errors ar addtv Errors ar normally dstrbutd Errors hav a constant standard dvaton Effcts of factors ar addtv > Obsrvatons ar ndpndnt and normally dstrbutd wth constant varanc 8 Octobr 005 EEC686/785 6
33 Vsual Tsts for Vrfyng th ssumptons Vsual tsts for ndpndnt rrors: Scattr plot of rsduals vrsus th prdctd rspons Magntud of rsduals < magntud of rsponss/0 > gnor trnds Plot th rsduals as a functon of th prmnt numbr Trnd up or down > othr factors or sd ffcts Vsual tsts for Normally dstrbutd rrors: Prpar normal uantl-uantl plot of rrors If th plot s appromatly lnar, th assumpton s satsfd 8 Octobr 005 EEC686/785 3 Vsual Tsts for Vrfyng th ssumptons Vsual tsts for constant standard dvaton of rrors: Scattr plot of y for varous lvls of th factor Sprad at on lvl sgnfcantly dffrnt than that at othr > ssumpton of constant varanc s not vald, nd transformaton 8 Octobr 005 EEC686/785 7
Vsual Tsts Eampl: Mmory-Cach 35 Rsdus an ordr of magntud smallr than th rsponss Rsduals appar to b appromatly normally dstrbutd 8 Octobr 005 EEC686/785 36 Multplcatv Modls for r Eprmnts ddtv modl: y 0 Not vald f ffcts do not add E.g., cuton tm of workloads th procssor spd v nstructons/scond th workload sz w nstructons Ecuton tm y v w Th two ffcts multply. Logarthm > addtv modl: log( y ) log( v ) log( w ) 8 Octobr 005 EEC686/785 8
37 Multplcatv Modls for r Eprmnts Corrct modl: y ' whr, y' log( y ) 0 Takng an antlog of addtv ffcts. to gt th multplcatv ffcts u.: u 0, u 0, and u 0 u rato of MIPS ratng of th two procssors u rato of th sz of th two workload ntlog of addtv man 0 > gomtrc man y 0 ( y y y ) n r 0 / n n 8 Octobr 005 EEC686/785 Eampl: Ecuton Tms 38 ddtv modl s not vald bcaus of physcal consdraton Effcts of workload and procssors do not add. Thy multply ddtv modl s not vald bcaus of larg rang for y y ma /y mn 7.90/0.08 or,53 > log transformaton Takng an arthmtc man of.7 and 0.03 s napproprat 8 Octobr 005 EEC686/785 9
Eampl: Ecuton Tms 39 ddtv modl s not vald bcaus: Th rsduals ar not small as compard to th rspons Th sprad of rsduals s larg at largr valu of th rspons > log transformaton 8 Octobr 005 EEC686/785 Eampl: Ecuton Tms 0 ddtv modl s not vald bcaus: Th rsdual dstrbuton has a longr tal than normal 8 Octobr 005 EEC686/785 0
nalyss Usng Multplcatv Modl Transformd data for multplcatv modl ampl 8 Octobr 005 EEC686/785 nalyss Usng Multplcatv Modl Prcntag of varaton pland by th two modls Wth multplcatv modl Intracton s almost zro Unpland varaton s only 0.% 8 Octobr 005 EEC686/785
Usng Multplcatv: Vsual Tsts 3 Concluson: multplcatv modl s bttr than th addtv modl 8 Octobr 005 EEC686/785 Usng Multplcatv: Intrprtaton of Rsults log( y) 0 y 0 0 0 0.03 0 0 0 0.97.07 0.07 0 0 0.97 Th tm for an avrag procssor on an avrag bnchmark s.07 Th tm on procssor s 9 tms (0.07 - ) that on an avrag procssor. Th tm on s on nnth (0.07 ) of that on an avrag procssor 0 0.07 0.03 0 0.07 0 8 Octobr 005 EEC686/785
5 Usng Multplcatv: Intrprtaton of Rsults MIPS rat for s 8 tms that of nchmark cuts 8 tms mor nstructons than Th ntracton s nglgbl > Rsults apply to all bnchmarks and procssors 8 Octobr 005 EEC686/785 Transformaton Consdratons 6 y ma /y mn small > multplcatv modl rsults smlar to addtv modl Many othr transformaton possbl 8 Octobr 005 EEC686/785 3
Transformaton Consdratons o-co famly of transformatons: a y, a 0 w a ag (ln y) g, a 0 Whr g s th gomtrc man of th rsponss: / n g ( y y y n ) w has th sam unts as y a can hav any ral valu, postv, ngatv, or zro Plot SSE as a functon of a > optmal a Knowldg about th systm bhavor should always tak prcdnc ovr statstcal consdratons 7 8 Octobr 005 EEC686/785 Gnral k r Factoral Dsgn 8 Modl: y 0 k Paramtr stmaton: S k S ( )th ntry n th sgn tabl Sum of suars: SSY k k SS0 r 0 SST SSY SS0 k k SS r SS r y y 8 Octobr 005 EEC686/785
Gnral k r Factoral Dsgn 9 Prcntag of y s varaton pland by th ffct (SS/SST) 00% SSE Standard dvaton of rrors: s k ( r ) Standard dvaton of ffcts: s s s s s 0 Varanc of contrast h, whr h 0, s: s s h Standard dvaton of th man of m futur rsponss: k r r h s 8 Octobr 005 EEC686/785 ˆ y m s k r k m / Gnral k r Factoral Dsgn 50 Confdnc ntrvals ar calculatd usng Modlng assumptons Errors ar IID (ndpndntly and dntcally dstrbutd) normal varats wth zro man Errors hav th sam varanc for all valus of th prdctors Effcts and rrors ar addtv t α k [ / ; ( r ) ] 8 Octobr 005 EEC686/785 5
Gnral k r Factoral Dsgn 5 Vsual tsts Th scattr plot of rrors vrsus prdctd rsponss should not hav any trnd Th normal uantl-uantl plot of rrors should b lnar Sprad of y valus n all prmnts should b comparabl If any of th abov vsual tsts fal or f th rato y ma /y mn s larg, a multplcatv modl should b nvstgatd 8 Octobr 005 EEC686/785 Eampl: 3 3 Dsgn 5 8 Octobr 005 EEC686/785 6
Eampl: 3 3 Dsgn 53 Sums of suars 8 Octobr 005 EEC686/785 Eampl: 3 3 Dsgn 5 Confdnc ntrvals of ffcts Th rrors hav 3 (3-) or 6 dgrs of frdom. Standard dvaton of rrors: SSE 6 s 3.0 k ( r ) 6 Standard dvaton of ffcts: s s t [0.95,6].337 3 3 3.0 0.65 8 Octobr 005 EEC686/785 7
Eampl: 3 3 Dsgn 55 90% confdnc ntrvals for paramtrs: ± (.337)(0.65) ± 0.87 0 (39.00, 0.7) (7.50, 9.5) (.50, 6.5) C (8.50, 0.) (.00, 3.75) C (.50, 3.5) C (.00,.75) C (-.00, 0.75) ll ffcts cpt C ar sgnfcant 8 Octobr 005 EEC686/785 Eampl: 3 3 Dsgn Prdcaton For a sngl confrmaton prmnt (m) Wth C -: 56 yˆ / / 5 5 syˆ s 3. 3.5 k r m 90% confdnc ntrval: ±.337 3.5 ±.70 (9.30, 8.70) 8 Octobr 005 EEC686/785 8
Cas Study: Garbag Collcton 57 8 Octobr 005 EEC686/785 Cas Study: Garbag Collcton 58 Masurd Data 8 Octobr 005 EEC686/785 9
Cas Study: Garbag Collcton 59 Effcts and varaton pland 8 Octobr 005 EEC686/785 Cas Study: Garbag Collcton 60 Conclusons Most of th varaton s pland by factors (workload), D (chunk sz), and th ntracton D btwn th two Th varaton du to prmntal rror s small > Svral ffcts that plan lss than 0.05% of varaton (lstd as 0.0% ) ar statstcally sgnfcantly Only ffcts, D, and D ar both practcally sgnfcant and statstcally sgnfcant 8 Octobr 005 EEC686/785 30