Chapter 11 Estimation of Absolute Performance

Transcription

1 Chapter 11 Estimation of Absolute Performance

2 Purpose Objective: Estimate system performance via simulation If q is the system performance, the precision of the estimator qˆ can be measured by: The standard error of qˆ. The width of a confidence interval (CI) for q. Purpose of statistical analysis: To estimate the standard error or CI. To figure out the number of observations required to achieve desired error/ci. Potential issues to overcome: Autocorrelation, e.g. inventory cost for subsequent weeks lack statistical independence. Initial conditions, e.g. inventory on hand and # of backorders at time 0 would most likely influence the performance of week 1. 2

3 Outline Distinguish the two types of simulation: transient vs. steady state. Illustrate the inherent variability in a stochastic discreteevent simulation. Cover the statistical estimation of performance measures. Discusses the analysis of transient simulations. Discusses the analysis of steady-state simulations. 3

4 Introduction to simulation From System to Model: Simulation based problem solving DISCRETE EVENT SIMULATION WITH APPLICATION TO COMPUTER COMMUNICATION SYSTEMS PERFORMANCE Introduction to Simulation Helena Szczerbicka, Kishor S. Trivedi and Pawan K. Choudhary

5 Type of Simulations Terminating verses non-terminating simulations Terminating simulation: Runs for some duration of time T E, where E is a specified event that stops the simulation. Starts at time 0 under well-specified initial conditions. Ends at the stopping time T E. T E might be random/unknown or predetermined Bank example: Opens at 8:30 am (time 0) with no customers present and 8 of the 11 teller working (initial conditions), and closes at 4:30 pm (Time T E = 480 minutes). The simulation analyst chooses to consider it a terminating system because the object of interest is one day s operation. 5

6 Type of Simulations Non-terminating simulation: Runs continuously, or at least over a very long period of time. Examples: Earth Simulator, assembly lines that shut down infrequently, telephone systems, networks, hospital emergency rooms. Initial conditions defined by the analyst. Runs for some analyst-specified period of time T E. Study the steady-state (long-run) properties of the system, properties that are not influenced by the initial conditions of the model. Whether a simulation is considered to be terminating or non-terminating depends on both The objectives of the simulation study and The nature of the system. 6

7 Stochastic Nature of Output Data Model output consists of one or more random variables because the model is an input-output transformation and the input variables are RVs M/M/1 queueing example: Poisson arrival rate = 5.0 per second (r i = 0.2 second); service time ~ Exp(6.67 or 0.15 second) System performance: utilization, time-avg queue length, L Q,avg. response time, & avg queueing delay Using OMNeT++, we run 5 statistically independent replications Each replication/run uses a distinct stream of random numbers Facilitated by OMNET++ ``repeat = 5 statement 7

8 Stochastic Nature of Output Data M/M/1 queueing example (cont.): Run/Replication (r) Uilization L_Q w d Average

9 Stochastic Nature of Output Data M/M/1 queueing example (cont.): U L_Q w d

10 Measures of performance Consider the estimation of a performance parameter, q (or f), of a simulated system in a single replication Discrete time data: [Y 1, Y 2,, Y n ], with ordinary mean: q Continuous-time data: {Y(t), 0 t T E } with time-weighted mean: f Point estimation for discrete time data. The point estimator: qˆ = 1 n n i= 1 Y i Is unbiased if its expected value is q, that is if: Is biased if: E( ˆ) q q E( ˆ) q =q Desired 10

11 Point Estimator [Performance Measures] Point estimation for continuous-time data. The point estimator: Is biased in general where: E( ˆ) f f. An unbiased or low-bias estimator is desired. Usually, system performance measures can be put into the common framework of q or f: e.g., the proportion of customers whose queueing delays equal, less, or more than 2 seconds % of time queue > 7 fˆ = ìï Y i = í îï 1 T E Y( t) dt T 0 E 1, if d i = {>,<} 2 0, otherwise ì ï 1, if L Q (t) > 7 Y(t) = í îï 0, otherwise 11

12 Point Estimator [Performance Measures] Performance measure that does not fit: quantile or percentile: P[Y q]= 0.85 E.g., the queueing delay for which 85% of customers will experience it or less than it Estimating quantiles: the inverse of the problem of estimating a proportion or probability. Consider a histogram of the observed values Y: Find qˆ such that 100p% of the histogram is to the left of (smaller than) qˆ. 12

13 Confidence-Interval Estimation [Performance Measures] To understand confidence intervals fully, it is important to distinguish between measures of error, and measures of risk e.g., confidence interval versus prediction interval. Suppose the model is the normal distribution with mean q, variance s 2 (both unknown). Let Y i be the average cycle time for parts produced on the i th replication of the simulation (its mathematical expectation is q). Average cycle time will vary from day to day, but over the long-run the average of the averages will be close to q. Sample variance across R replications: S 2 = 1 R 1 R i= 1 ( Y i Y )... 2 Y.. = 1 R R å i=1 Y i. 13

14 Confidence-Interval Estimation [Performance Measures] Confidence Interval (CI): Y.. is not q; Y.. is an estimate of q based on the sample; & it has error A CI is measure of error between q and the estimator Y.. Where Y i. are normally distributed. S Y.. t / 2, R 1 R We cannot know for certain how far Y.. is from q but CI attempts to bound that error. A CI may be wrong So, a CI, such as 95%, tells us how much we can trust the interval to actually bound the error between Y.. and q. The more replications we make, the less error there is in (converging to 0 as R goes to infinity). Y.. 14

15 Confidence-Interval Estimation [Performance Measures] Prediction Interval (PI): A measure of risk. A good guess for the average cycle time on a particular day is our estimator (Y.. ) but it is unlikely to be exactly right. PI is designed to be wide enough to contain the actual average cycle time on any particular day with high probability. Normal-theory prediction interval: Y.. t / 2, R 1S 1 1 R The length of PI will not go to 0 as R increases because we can never simulate away risk. PI s limit is: q z / 2s 15

16 CI & PI: An Example [Performance Measures] M/M/1 System Run/Replication (r) Uilization L_Q w d Average Standard Deviation t * S /SQRT(R) CI lower CI upper t * S * SQRT(1 + 1/R) PI lower PI upper Theoretical

17 CI & PI: An Example [Performance Measures] M/M/1 System Replications (R) = 5 A 95% CI.t 0.05/2, 5-1 = t 0.025, 4 = 2.78 A CI for Utilization = ± 2.78(0.1242/sqrt(5)) = ± = [0.5505, ] Our estimate of the long-run utilization is , but there could be an error as much as ± in this estimate A PI for Utilization = ± 2.78(0.1242/sqrt(1+1/5)) = ± = [0.3897, ] In a particular run, we are 95% confident that the utilization will be = ±

18 t-distribution Table [Performance Measures] 18

19 CI & PI: An Example [Performance Measures] M/M/1 System: CIs Avg Lo Hi U 1 L_Q 2 w 3 4 d 19

20 Output Analysis for Terminating Simulations A terminating simulation: runs over a simulated time interval [0, T E ]. A common goal is to estimate: 1 q = E n 1 f = E TE n i= 1 0 Better, we use independent replications/runs To eliminate the bias introduced by a particular random number stream Each run using a different random number stream Independently chosen initial conditions Y T E i, Y ( t) dt, for discrete output for continuous output Y ( t),0 t T E 20

21 Statistical Background [Terminating Simulations] Across Replication: For example: the response time averages (discrete time data) R The average: 1 Y.. = Y i. R i= 1 R The sample variance: S = ( Y i. Y.. ) R 1 i= 1 The confidence-interval half-width: S H = t / 2, R 1 R Within replication: For example: the L Q (a continuous time data) The average: 1 T Ei Yi. = Yi ( t) dt T 0 The sample variance: Ei S 2 i = 1 T Ei T Ei 0 Y ( t) Y i i. 2 dt 21

22 Statistical Background [Terminating Simulations] Important to distinguish within-replication data from across-replication data. For example, simulation of a computer system Two performance measures of that system: response time and length of the queue at the CPU. Let Y ij be the response time for the j th process executed in the i th replication. Across-replication data are formed by summarizing withinreplication data. Y i. 22

23 Statistical Background [Terminating Simulations] Within-Replication Data Across-Replication Data Y 11 Y 12 Y 1n1 Y 1. S 1 2 H 1 Y 21 Y 22 Y 2n2 Y 2. S 2 2 H 2 Y R1 Y R2 Y RnR Y R. S 2 R H R Y.. S 2 H Within-Replication Data Across-Replication Data Y 1 (t), 0 < t < T E1 Y 1. S 1 2 H 1 Y 2 (t), 0 < t < T E2 Y 2. S 2 2 H 2 Y R (t), 0 < t < T ER Y R. S R 2 H R Y.. S 2 H 23

24 Statistical Background [Terminating Simulations] Overall sample average, Y.., and the individual replication sample averages,, are always unbiased estimators. Y i. Across-replication data are Independent (different random numbers), and Identically distributed (same model). But, within-replication data do not have these properties. 24

25 Output Analysis for Terminating Simulations Example: the M/M/1 model previously discussed See Slides # 7 9, And, the example(s) in the textbook 25

26 C.I. with Specified Precision [Terminating Simulations] The half-length H of a 100(1 )% confidence interval for a mean q, based on the t distribution, is given by: H = t / 2, R 1 R is the # of replications Suppose that an error criterion e is specified with probability 1 -, a sufficiently large sample size should satisfy: P What is this sufficiently large R? Problem: Both S & H are not-so-straightforward fns of R S R q < e 1 Y.. S 2 is the sample variance 26

27 C.I. with Specified Precision [Terminating Simulations] Make an initial sample of size R 0 independent replications. Based on R 0, obtain an initial estimate S 0 2 of the population variance s 2. Then, choose a sample size R such that R R 0 : Since t /2, R-1 z /2, an initial estimate of R: 2 z / 2S0 R, z / 2 is the standard normal distribution. e t / 2, 1 0 R is the smallest integer satisfying R R 0 and R S R e Collect/conduct R - R 0 additional observations/replications. Form the 100(1- )% C.I. for q based on R: S Y.. t / 2, R 1 R Since S could differ than S 0, H would differ (<, >) than e 2 27

28 C.I. with Specified Precision [Terminating Simulations] Density of the t-distribution for 1, 2, 3, 5, 10 (all in green), and 30 (in red) degrees of freedom vs. standard normal distribution (blue) 28

29 C.I. with Specified Precision: An Example [Terminating Simulations] M/M/1 Example Objective: estimate utilization r to within ±0.04 with probability 0.95 Initial sample of size R 0 = 5 is taken and an initial estimate of the population variance is S 0 = The error criterion is e = 0.04 and confidence coefficient is 1- = 0.95, hence, the final sample size must be at least: æ z S 0 ö 2 æ ö 2 ç = ç = è e ø è 0.04 ø Next, we test the possible candidates (R = 37, 38, 39, 40, ) with the inequality of t /2, R-1 for the final sample size: R t 0.025, R R = 40 is the smallest integer satisfying the error criterion So, R - R 0 = 35 additional replications are needed. After obtaining additional outputs, H should be checked. 29

30 z-distribution Table [Performance Measures] 30

31 Output Analysis for Steady-State Simulation Consider a single run of a simulation model to estimate a steady-state or long-run characteristics of the system. The single run produces observations Y 1, Y 2,... (generally the samples of an autocorrelated time series). Steady-state/long-run Performance measure: q = f = lim 1 n n 1 T lim 0 T E E n å Y i, i=1 T E Y( t) dt, for discrete measure for continuous measure (with probability 1) (with probability 1) Independent of the initial conditions. 31

32 Output Analysis for Steady-State Simulation The sample size is a design choice, with several considerations in mind: Any bias in the point estimator that is due to artificial or arbitrary initial conditions (bias can be severe if run length is too short). Desired precision of the point estimator. Budget constraints on computer resources. Notation: the estimation of q from a discrete-time output process. One replication (or run), the output data: Y 1, Y 2, Y 3, With several replications, the output data for replication r: Y r1, Y r2, Y r3, 32

33 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) 20 Experiments (i = 1,, 20); each begins with empty & idle state Simulation runs until 2 i jobs departs the system (each 1 replication) Experiment # No. of Jobs Uilization L_Q w d

34 Initialization Bias [Steady-State Simulations] Utilization

35 Initialization Bias [Steady-State Simulations] 4.50 L Q

36 Initialization Bias [Steady-State Simulations] w

37 Initialization Bias [Steady-State Simulations] d

38 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Five initial states: X(t = 0) = 0, 1, 2, 3, 4, 5, 10, 20 X(t): # of jobs in system at t 20 Experiments (i = 1,, 20) for each initial state Results only for L Q Green colored cells correspond for values within [Theoretical value 5%, Theoretical value + 5%] 38

39 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) X(0) = 0 X(0) = 1 X(0) = 2 X(0) = 3 X(0) = 4 X(0) = 5 X(0) = 10 X(0) =

40 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d)

41 Initialization Bias [Steady-State Simulations] Methods to reduce the point-estimator bias caused by using artificial and unrealistic initial conditions: Intelligent initialization. Divide simulation into an initialization phase and data-collection phase. Intelligent initialization Initialize the simulation in a state that is more representative of long-run conditions. If the system exists, collect data on it and use these data to specify more nearly typical initial conditions, e.g., measure/sample a router at some time. If the system can be simplified enough to be mathematically solvable, solve the simplified model to find long-run expected conditions. E.g. queueing models 41

42 Initialization Bias [Steady-State Simulations] Divide each simulation into two phases: An initialization phase, from time 0 to time T 0. A data-collection phase, from T 0 to the stopping time T 0 +T E. The choice of T 0 is important: After T 0, system should be more nearly representative of steady-state behavior. System has reached steady state: the probability distribution of the system state is close to the steady-state probability distribution (bias of response variable is negligible). In OMNeT++ terminology, warmup-period E.g., warmup-period = 20s 42

43 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Three initial states: X(t = 0) = 0, 3 & 10 X(t): # of jobs in system at t 5 Experiments for each initial state # of replications in each experiment: 5 & 20 Results only for d 95% confidence interval: Y.. t / 2, R 1 S R 43

44 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Initial states: X(t = 0) = 0 # of replications in each experiment: 5 X(0) = 0 R Average Standard Deviation t * S /SQRT(R) CI lower CI upper Theoretical

45 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Initial states: X(t = 0) = 0 & 3 # of replications in each experiment:

46 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Initial states: X(t = 0) = 10 & 3 # of replications in each experiment:

47 Initialization Bias [Steady-State Simulations] M/M/1 Example (cont d) Initial states: X(t = 0) = 3 # of replications in each experiment: 5 &

48 Replication Method [Steady-State Simulations] Use to estimate point-estimator variability and to construct a confidence interval. Approach: make R replications, initializing and deleting from each one the same way. Important to do a thorough job of investigating the initial-condition bias: Bias is not affected by the number of replications, instead, it is affected only by deleting more data (i.e., increasing T 0 ) or extending the length of each run (i.e. increasing T E ). Basic raw output data {Y rj, r = 1,..., R; j = 1,, n} is derived by: Individual observation from within replication r. 48

49 Replication Method [Steady-State Simulations] Each replication is regarded as a single sample for n 1 estimating q. For replication r: Y ( n, d) = The overall point estimator: Y R.. n, d) = Yr.( n, d) and E[ Y.. ( n, d)] R r= 1 If d and n are chosen sufficiently large: q n,d ~ q. Y ( n, ) is an approximately unbiased estimator of q... d To estimate standard error of Y.., the sample variance and standard error: S 2 Y r. n d rj j= d 1 1 ( = q R R = ( Yr. Y.. ) = Yr. RY.. and s. e.( Y.. ) = R 1 r= 1 R 1 r= 1 n, d S R 49

50 Replication Method [Steady-State Simulations] Length of each replication (n) beyond deletion point (d): (n - d) > 10d Number of replications (R) should be as many as time permits, up to about 25 replications. For a fixed total sample size (n), as fewer data are deleted ( d): C.I. shifts: greater bias. Standard error of Y ( n,.. d ) decreases: decrease variance. Reducing bias Trade off Increasing variance 50

51 Summary Stochastic discrete-event simulation is a statistical experiment. Purpose of statistical experiment: obtain estimates of the performance measures of the system. Purpose of statistical analysis: acquire some assurance that these estimates are sufficiently precise. Distinguish: terminating simulations and steady-state simulations. Steady-state output data are more difficult to analyze Decisions: initial conditions and run length Possible solutions to bias: deletion of data and increasing run length Statistical precision of point estimators are estimated by standarderror or confidence interval Method of independent replications was emphasized. 51