Outline. Types of Experimental Designs. Terminology. EEC 686/785 Modeling & Performance Evaluation of Computer Systems. Lecture 12

Transcription

1 EEC 686/785 Modlng & Prformanc Evaluaton of Computr Sytm Lctur Outln Rvw of lctur r Factoral Dgn wth Rplcaton Dpartmnt of Elctrcal and Computr Engnrng Clvland Stat Unvrty wnbng@.org (bad on Dr. Ra Jan lctur not 8 Octobr 5 EEC686/785 Trmnology Rpon varabl: outcom Factor: varabl that affct th rpon varabl Lvl: th valu that a factor can aum Prmary factor and condary factor Rplcaton: rptton of all or om prmnt Dgn: th numbr of prmnt, th factor lvl and numbr of rplcaton for ach prmnt Eprmntal Unt Intracton: ffct of on factor dpnd upon th lvl of th othr 3 Typ of Eprmntal Dgn Smpl Dgn: vary on factor at a tm Full Factoral Dgn: all combnaton Fractonal Factoral Dgn: u only a fracton of th full factoral dgn 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

2 Factoral Dgn 5 Factoral Dgn: Modl 6 factor, ach at two lvl Eay to analyz Hlp n ortng out mpact of factor Good at th bgnnng of a tudy Vald only f th ffct of a factor undrctonal,.., th prformanc thr contnuouly dcra or contnuouly ncra a th factor ncrad from mn to ma E.g., mmory z, th numbr of d drv y q 5 q q q 5 q q 5 q q q 75 q Unqu oluton for q and q : y 5 Intrprtaton: Man prformanc MIPS Effct of mmory MIPS Effct of cach MIPS Intracton btwn mmory and cach 5 MIPS 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 llocaton of Varaton 7 Drvaton 8 Importanc of a factor proporton of th varaton pland ( Sampl Varanc of y y y y Varaton of y Δ Numrator For a dgn um of ( y y Varaton du to SS q, tc. quar total SST ( SST q q q Modl: y q Notc Th um of ntr n ach column zro Th um of th quar of ntr n ach column Th column ar orthogonal (nnr product of any two column zro: ; ( ; ( 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

3 Drvaton 9 r Factoral Dgn wth Rplcaton Varaton of y q q ( y y ( q ( q ( ( q ( ( q ( product trm r rplcaton of prmnt r obrvaton llow tmaton of prmntal rror Modl: y q prmntal rror 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Computaton of Effct Etmaton of Eprmntal Error Mmory-Cach Study: Smply u man of r maurmnt Effct: q, q.5, q 9.5, q 5 8 Octobr 5 EEC686/785 Etmatd Rpon: y ˆ q Eprmntal rror tmatd maurd y yˆ y Sum of quard rror: 8 Octobr 5 EEC686/785 q SSE r 3

4 Eprmntal Error: Eampl 3 llocaton of Varaton Etmatd Rpon: yˆ q q q Eprmntal rror: y yˆ 5 5 y.. dnot th man of rpon from all rplcaton of all prmnt Th dot n th ubcrpt ndcat th dmnon along whch th avragng don dnot th man of rpon n all rplcaton of th th prmnt y. Total varaton or total um of quar (SST: SST ( y y SST y q.. rq rq rq SS ( y y.. SS SS SSE 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Drvaton 5 Drvaton 6 Modl: y q q q q y q Snc, thr product, and all rror add to y q rq Man rpon: y.. y q r Squarng both d of th modl and gnorng cro product trm: y q q q q SSY SS SS SS SS SSE Total varaton: SST SSE ( y y y.. y.. SSY SS SS SS SS SSE SSE SSY r( q 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

5 Eampl: Mmory-Cach Study 7 Eampl: Mmory-Cach Study 8 SSY SS rq SS rq SS rq SS rq 7 ( ( SSE 7 3( SST SSY SS Factor plan 557/73 or 78.88% Factor plan 5.% Intracton plan.7%.5% unpland and attrbutd to rror SSSSSSSSE SST 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Confdnc Intrval for Effct 9 Confdnc Intrval for Effct Effct ar random varabl Error ~ N(, σ y ~ N( y, σ q r y q Lnar combnaton of normal varat > q normal wth varanc /( r.. σ Varanc of rror: SSE ΔMSE ( r ( r Man Squar of Error (MSE: quantty at th rght hand d Dnomnator (r- # of ndpndnt trm n SSE > dgr of frdom SSE ha 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 5

6 Confdnc Intrval for Effct Eampl Etmatd varanc of q : Smlarly, q q /( r q Confdnc ntrval (CI for th ffct: q t [ r ] α / ; ( q CI do not nclud a zro > gnfcant q r For mmory-cach tudy: Standard dvaton of rror: SSE ( r 8 Standard dvaton of ffct: 3.57 q.3 ( r For 9% confdnc: t [.95,8].86 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Eampl Confdnc ntrval: q ± (.86(.3 q ±.9 q (39.8,.9 q (9.58, 3. q (7.58,. q (3.8, 6.9 No zro crong > all ffct ar gnfcant 3 Confdnc Intrval for Contrat It alo pobl to comput varanc and confdnc ntrval for any contrat of ffct. contrat any lnar combnaton who coffcnt add up to zro Varanc of h q, whr h, : q h For (-α% confdnc ntrval, u r h t α [ / ; ( r ] 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 6

7 Eampl CI for Prdctd Rpon 6 Mmory-cach tudy: th confdnc ntrval for u q -q calculatd a follow: Coffcnt,,, and - > Contrat Man u Varanc 6 u Standard dvaton u t [.95;8].86 9% confdnc ntrval for u: u t u.86.5 (6.3, 5.69 Man rpon y ˆ q Th tandard dvaton of th man of m rpon: n yˆ m ff nff m ffctv dgr of frdom um of r 5 / total numbr of DF of param ud n run yˆ 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 CI for Prdctd Rpon 7 Eampl: Mmory-Cach Study 8 (-α% confdnc ntrval: yˆ t [ r ] α / ; ( y ˆm ngl run (m: Populaton man (m : 5 yˆ r yˆ / 5 r / For - and -: ngl confrmaton prmnt: yˆ q q q Standard dvaton of th prdcaton: / 5 5 y ˆ r Ung t [.95;8].86, th 9% confdnc ntrval : 5±.86.5(8.9,.9 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 7

8 Eampl: Mmory-Cach Study 9 Eampl: Mmory-Cach Study 3 For - and - (contnud: Man rpon for 5 prmnt n futur: / 5 5 y ˆ r m 5 Th 9% confdnc ntrval : 5±.86.8(9.79,.9 For - and - (contnud: Man rpon for a larg numbr of prmnt n futur: / 5 5 yˆ r Th 9% confdnc ntrval : 5±.86.3(.7, Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Eampl: Mmory-Cach Study 3 Vual Tt for Vrfyng th umpton 3 For - and - (contnud: Currnt man rpon: not for futur (u th formula for contrat: h.75 yˆ.6 r Th 9% confdnc ntrval : 5±.86.6(.7, 8.83 Notc: Confdnc ntrval bcom narrowr In drvng th pron for ffct, w mad ntally th am aumpton a n rgron analy: Error ar tattcally ndpndnt Error ar addtv Error ar normally dtrbutd Error hav a contant tandard dvaton Effct of factor ar addtv > Obrvaton ar ndpndnt and normally dtrbutd wth contant varanc 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 8

9 33 3 Vual Tt for Vrfyng th umpton Vual tt for ndpndnt rror: Scattr plot of rdual vru th prdctd rpon Magntud of rdual < magntud of rpon/ > gnor trnd Plot th rdual a a functon of th prmnt numbr Trnd up or down > othr factor or d ffct Vual tt for Normally dtrbutd rror: Prpar normal quantl-quantl plot of rror If th plot appromatly lnar, th aumpton atfd Vual Tt for Vrfyng th umpton Vual tt for contant tandard dvaton of rror: Scattr plot of y for varou lvl of th factor Sprad at on lvl gnfcantly dffrnt than that at othr > umpton of contant varanc not vald, nd tranformaton 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/ Vual Tt Eampl: Mmory-Cach Multplcatv Modl for r Eprmnt Rdu an ordr of magntud mallr than th rpon Rdual appar to b appromatly normally dtrbutd ddtv modl: y q Not vald f ffct do not add E.g., cuton tm of worload th procor pd v ntructon/cond th worload z w ntructon Ecuton tm y v w Th two ffct multply. Logarthm > addtv modl: log( y log( v log( w 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 9

10 Multplcatv Modl for r Eprmnt 37 Eampl: Ecuton Tm 38 Corrct modl: y' q whr, y' log( y Tang an antlog of addtv ffct q. to gt th multplcatv ffct u.: q q q u, u, and u u rato of MIPS ratng of th two procor u rato of th z of th two worload ntlog of addtv man q > gomtrc man y ( y y y n r q / n n ddtv modl not vald bcau of phycal condraton Effct of worload and procor do not add. Thy multply ddtv modl not vald bcau of larg rang for y y ma /y mn 7.9/.8 or,53 > log tranformaton Tang an arthmtc man of.7 and.3 napproprat 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Eampl: Ecuton Tm 39 Eampl: Ecuton Tm ddtv modl not vald bcau: Th rdual ar not mall a compard to th rpon Th prad of rdual larg at largr valu of th rpon > log tranformaton ddtv modl not vald bcau: Th rdual dtrbuton ha a longr tal than normal 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

11 naly Ung Multplcatv Modl naly Ung Multplcatv Modl Tranformd data for multplcatv modl ampl Prcntag of varaton pland by th two modl Wth multplcatv modl Intracton almot zro Unpland varaton only.% 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Ung Multplcatv: Vual Tt Concluon: multplcatv modl bttr than th addtv modl 3 Ung Multplcatv: Intrprtaton of Rult log( y q q y.3 q q q.97.3 Th tm for an avrag procor on an avrag bnchmar.7 Th tm on procor 9 tm (.7 - that on an avrag procor. Th tm on on nnth (.7 of that on an avrag procor Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

12 Ung Multplcatv: Intrprtaton of Rult 5 Tranformaton Condraton 6 MIPS rat for 8 tm that of nchmar cut 8 tm mor ntructon than Th ntracton nglgbl > Rult apply to all bnchmar and procor y ma /y mn mall > multplcatv modl rult mlar to addtv modl Many othr tranformaton pobl 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Tranformaton Condraton 7 Gnral r Factoral Dgn 8 o-co famly of tranformaton: a y, a w a ag (ln y g, a Whr g th gomtrc man of th rpon: / n g ( y y y n w ha th am unt a y a can hav any ral valu, potv, ngatv, or zro Plot SSE a a functon of a > optmal a Knowldg about th ytm bhavor hould alway ta prcdnc ovr tattcal condraton Modl: y q Paramtr tmaton: q S S ( th ntry n th gn tabl Sum of quar: SSY SS rq SST SSY SS SS rq r y SS y 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

13 Gnral r Factoral Dgn 9 Gnral r Factoral Dgn 5 Prcntag of y varaton pland by th ffct (SS/SST % SSE Standard dvaton of rror: ( r Standard dvaton of ffct: q q q q Varanc of contrat h q, whr h, : r h q h r Confdnc ntrval ar calculatd ung Modlng aumpton t α [ / ; ( r ] Error ar IID (ndpndntly and dntcally dtrbutd normal varat wth zro man Error hav th am varanc for all valu of th prdctor Effct and rror ar addtv Standard dvaton of th man of m futur rpon: yˆ m r m / 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Gnral r Factoral Dgn 5 Eampl: 3 3 Dgn 5 Vual tt Th cattr plot of rror vru prdctd rpon hould not hav any trnd Th normal quantl-quantl plot of rror hould b lnar Sprad of y valu n all prmnt hould b comparabl If any of th abov vual tt fal or f th rato y ma /y mn larg, a multplcatv modl hould b nvtgatd 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 3

14 Eampl: 3 3 Dgn 53 Eampl: 3 3 Dgn 5 Sum of quar Confdnc ntrval of ffct Th rror hav 3 (3- or 6 dgr of frdom. Standard dvaton of rror: SSE 6 3. ( r 6 Standard dvaton of ffct: q t [.95,6] Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Eampl: 3 3 Dgn 55 Eampl: 3 3 Dgn 56 9% confdnc ntrval for paramtr: q ± (.337(.65 q ±.87 q (39.,.7 q (7.5, 9.5 q (.5, 6.5 q C (8.5,. q (., 3.75 q C (.5, 3.5 q C (.,.75 q C (-.,.75 ll ffct cpt q C ar gnfcant Prdcaton For a ngl confrmaton prmnt (m Wth C -: yˆ / / 5 5 yˆ r m 9% confdnc ntrval: ± ±.7 (9.3, Octobr 5 EEC686/785 8 Octobr 5 EEC686/785

15 Ca Study: Garbag Collcton 57 Ca Study: Garbag Collcton 58 Maurd Data 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 Ca Study: Garbag Collcton 59 Ca Study: Garbag Collcton 6 Effct and varaton pland Concluon Mot of th varaton pland by factor (worload, D (chun z, and th ntracton D btwn th two Th varaton du to prmntal rror mall > Svral ffct that plan l than.5% of varaton (ltd a.% ar tattcally gnfcantly Only ffct, D, and D ar both practcally gnfcant and tattcally gnfcant 8 Octobr 5 EEC686/785 8 Octobr 5 EEC686/785 5