Analyzing Simulation Results

Transcription

1 Analyzing Siulation Results Dr. John Mellor-Cruey Departent of Coputer Science Rice University COMP 528 Lecture March 2005

2 Topics for Today Model verification Model validation Transient reoval Terinating siulations Stopping criteria Understand 2

3 Model Goodness Fidelity to odeled syste Measuring goodness validation: are assuptions reasonable? verification: does odel ipleent assuptions correctly? Possible odel states invalid, unverified valid, unverified invalid, verified valid, verified correctly ipleents bad assuptions incorrectly ipleents good assuptions correctly ipleents good assuptions 3

4 Model Verification Techniques I Strategies for avoiding bugs software engineering top-down design layered (hierarchical) syste structure odularity well-defined interfaces unit testing assertions to check invariants e.g., # packets received = # packets sent - # packets lost - # in flight entity accounting structured walk through Deterinistic odels run siulation with known distributions for rando variates Siplified test cases with easily analyzed results Tracing: events, procedures, variables 4

5 Model Verification Techniques II On-line graphical visualizations convey progress of siulation Continuity test test siulation with slightly different paraeters investigate sudden changes in output Degeneracy tests check odel works for extree cases e.g. networking: no routers, no router delays, no sources, 5

6 Model Verification Techniques III Consistency tests siilar results for paraeters that should have siilar effects e.g. router siulation: 2 sources, rate r ~ 1 source, rate 2r Seed independence siilar results for different seed values 6

7 Model Validation Techniques I What to check assuptions input paraeter values and distributions output values and conclusions How expert intuition: ost coon and practical n (o i " e i ) 2 2 # < $ [%;k"1] e i k=1 easureents of real syste are siulation results and easureents distinguishable? can use statistical tests, e.g. paired observations verify input distributions, e.g. chi-square test 7

8 Model Validation Techniques II How (continued) theoretical results, e.g. queueing odel siplifying assuptions helps validate a few siple cases of theoretical odel with siulation or intuition use analytical odel to predict coplex cases Caution: yth of a fully-validated odel generally possible only to prove odel not wrong for soe cases ore coparisons increase confidence, but prove nothing! 8

9 Transient Reoval Transient state: prefix of siulation before steady state Steady state perforance is usually that of interest e.g. cache perforance after cache is war Goal: results exclude transient state before steady state Proble: identifying end of transient state Heuristic approaches for reoving transient state long runs proper initialization truncation initial data deletion oving average of independent replications batch eans 9

10 Transient Reoval: Long Runs Long run = steady state results long enough to doinate effects of initial transients Disadvantages wastes resources (coputer tie and real tie) difficult to ensure length of run is long enough Recoendation: avoid this ethod 10

11 Transient Reoval: Proper Initialization Proper initialization = starting siulation in state close to expected steady state e.g. start CPU scheduling siulation with non-epty job queue e.g. start WWW cache trace-driven siulation with ost frequently referenced files in cache Effect: reduces length of transient behavior 11

12 Transient Reoval: Truncation Assuption: variability of steady state < transient state Truncation ethod assues variability = range Truncation algorith input: n observations {x 1, x 2,, x n } for k = 2, n in k = in ({x k,, x n }) ax k = ax ({x k,, x n }) if in k x k && ax k x k break post condition: if k n then k - 1 = length of transient state 12 is there a flaw? can we fix it? Value transient state Observation nu ber 12

13 Terinating Siulations: Initial Data Deletion Conceptual idea copute average after soe of initial observations oitted during steady state average does not change uch as additional observations are deleted Proble randoness in observations causes avg to change even in SS Solution average across several replications replication: sae paraeter values; only seed values differ rationale: sooths trajectory Input: replications, each of length n 13

14 Initial Data Deletion: First Steps Copute ean trajectory by averaging across replications x j = 1 " x ij, j =1,2,...,n i=1 Copute overall ean x = 1 n n " j=1 x j 14

15 Initial Data Deletion: Reaining Steps for k = 1, n - 1 assue transient state is of length k delete first k observations fro ean trajectory copute overall ean fro reaining n - k values x = 1 n " k copute relative change in overall ean n # j= k +1 x j Relative change = x k " x x find knee in a curve showing the relative change in overall ean 15

16 Initial Deletion: Putting it all Together transient interval knee 16

17 Moving Average of Independent Replications Copute ean trajectory by averaging across replications x j = 1 " x ij, j =1,2,...,n i=1 for k = 1 to n plot trajectory of oving average of successive 2k+1 values x j = 1 2k +1 k # x j +l, j = k +1,k + 2,...,n " k l="k if trajectory is sufficiently sooth, break find the knee in the curve. j at the knee gives the length of the transient phase 17

18 Moving Average of Independent Replications transient interval knee 18

19 Batch Means Run a very long siulation Afterward, divide it into several parts of equal duration Each part is a batch Batch ean = ean of observations in each batch Input: batches of floor(m/n) Algorith for each batch, copute a batch ean copute the overall ean across all batches x = 1 copute variance of batch eans repeat for increasing n=3,4,5, plot variance as function of batch size " i=1 x i x i = 1 n n " x ij, i =1,2,..., j=1 Var(x ) = 1 #(x i " x ) 2 "1 length of transient interval is length at which variance starts decreasing i=1 19

20 Terinating Siulations Most siulations reach a steady state, but soe don t Exaple network traffic consists of xfer of sall files (1-3 packets each) steady state siulations using large files give results of no interest to typical user Necessary to study such systes in transient state Terinating siulations: ones that don t reach steady state Other terinating siulations one that shuts down at 10PM every day systes with paraeters that change over tie Terinating siulations don t require transient reoval Final conditions ay not be typical. can reove like initial conditions 20

21 Stopping Criteria: Variance Estiation Choosing proper siulation length is iportant too short: results highly variable too long: wastes tie and resources Siulation should be run until confidence interval for ean response narrows to desired width x ± z 1"# / 2 Var(x ) Proble: how to estiate the variance observations in siulation are not independent e.g. waiting tie for job I+1 depends on tie for job I 21

22 Variance Estiation: Independent Replications Replications obtained by repeating siulation with different seed Method assuption: eans of independent replications are independent even though observations within a replication are correlated Input: replications of size n + n o (n o is size transient phase) Algorith copute ean for each replication, excluding transient phase copute overall ean for all replications calculate variance of replicate eans confidence interval is then x ± z 1"# / 2 Var(x ) = 1 #(x i " x ) 2 "1 i=1 Var(x ) Note : conf interval inversely proportional to n x waste less by increasing n rather than 22

23 Variance Estiation: Batch Means Run long siulation; reove transient & divide into batches Algorith copute ean for each batch copute overall ean for all batches calculate variance of batch eans confidence interval is then Notes increase confidence by increasing # batches () or batch size (n) batch size ust be large so batch eans have little correlation finding correct n increase batch size until autocovariance between batch eans is sall w.r.t. variance autocovariance = Var(x ) = 1 #(x i " x ) 2 "1 i=1 x ± z 1"# / 2 Var(x ) Cov(x i,x i+1 ) = x 1 " 2 #(x i " x )(x i+1 " x ) i=1 23

24 Variance Estiation: Batch Means Run long siulation; reove transient & divide into batches Algorith copute ean for each batch copute overall ean for all batches calculate variance of batch eans confidence interval is then Notes increase confidence by increasing # batches () or batch size (n) batch size ust be large so batch eans have little correlation finding correct n increase batch size until autocovariance between batch eans is sall w.r.t. variance autocovariance = Var(x ) = 1 #(x i " x ) 2 "1 i=1 x ± z 1"# / 2 Var(x ) Cov(x i,x i+1 ) = x 1 #(x i " x )(x i+1 " x ) " 2 i=1 24

25 Variance Estiation: Method of Regeneration Consider CPU scheduling algorith every tie queue is epty, it is like a fresh start for the siulation trajectory in interval after epty state does not depend on prior trajectory this phenoenon called regeneration Regeneration point: when a siulation enters an independent phase regeneration points Regenerative period: duration between 2 regeneration points Not all systes are regenerative syste with any queues regenerates only when all are epty 25

26 Variance Estiation: Method of Regeneration Algorith copute cycle sus y i = copute the overall ean calculate difference between expected and observed cycle sus w i = y i "n i x, i =1,2,..., (w i IID ean 0) Var(w) = 1 "1 calculate variance of differences copute the ean cycle length confidence interval for ean response x ± z 1"# / 2 1 n n i " j=1 x ij " " x = y i / n i n = 1 Var(w) cycles of size n 1,n 2,,n " i=1 n i # i=1 w i 2 26