Performance Evaluation

Transcription

1 Performance Evaluation Chapter 5 Short Probability Primer & Obtaining Results (Acknowledgement: these slides have mostly been compiled from [Kar04, Rob0, BLK0]) otivation! Recall:! Usually, a simulation program is build in order to help answer a series of (quantitative) questions about a system under study! For this, a model (the simulation program) is build! This model needs to represent the system under study with sufficient accuracy concerning the characteristics of the system that are relevant for the questions of interest! Typically, this requires modeling of the following:! System! Load! Faults

2 System, Load, and Fault odels ()! System model:! Describes the composition of an entire system out of simpler subsystems! Represents the behavior, the way a system works! In communication networks:! Entities communicate by exchanging messages over links! Protocols implemented in entities, can have parameters like processing delays, limited queue length! Links can have parameters like bandwidth and delay (compiled from [Kar04]) 3 System, Load, and Fault odels ()! Load model:! Describes the pattern with which requests are made to the system to perform different kinds of activities When do such requests occur? What is the time between requests? What are the parameters of such a request?! In communication networks:! Load is the desire of a user to send a packet to another user Example parameter: How big is the packet?! On a coarser abstraction level: How often are connections established? How much data is transmitted within a connection? What is the mix between different types of connections (QoS) within a session?... 4

3 System, Load, and Fault odels (3)! Fault model:! Describes which parts of a system can deviate from their proscribed/ desired behavior, and in which form When do such deviations occur? Can they be repeated (are faulty entities repaired)? How long do deviations last? Which kind of faulty behavior occurs?! In communication networks:! Entities can be faulty (e.g., a node can crash, often considered a permanent error)! Communication links can be faulty (e.g., some bits are not transmitted correctly, due to electromagnetic noise, usually considered a transient error) 5 System, Load, and Fault odels (4)! Neither the arrival of requests nor the occurrence of faults can be described deterministically! Random distributions needed to model these events along with their parameters:! What distributions are available, appropriate and easy to use in simulations?! How can random numbers be generated such that the simulated events occur according to these distributions? (Based on general-purpose random number generators?)! How well do standard distributions match observed behavior of real (communication) systems? How to choose parameters for distributions to model real systems?! What to do if no simple standard distributions can be found that match a real system s behavior?! Is it sufficient to just look at distributions?! By the way, what exactly is a distribution???! We will first review some basics of probability... 6

4 Short Review of Some Probability Basics! Probability is a numerical measure of the likelihood that an event will occur.! Probability values are always assigned on a scale from 0 to.! A probability near 0 indicates an event is very unlikely to occur.! A probability near indicates an event is almost certain to occur.! A probability of 0.5 indicates the occurrence of the event is just as likely as it is unlikely. Increasing Likelihood of Occurrence Probability: 0.5 The occurrence of the event is just as likely as it is unlikely. (compiled from [Rob0]) 7 An Experiment and Its Sample Space! An experiment is any process that generates well-defined outcomes.! The sample space for an experiment is the set of all experimental outcomes.! A sample point is an element of the sample space, any one particular experimental outcome.! Examples: Experiment! Draw a card from a pack! Telephone sales call! First number drawn in National Lottery Outcomes! {Ace hearts, hearts,, King of spades}! Sale or no sale! {,,3,,49} 8

5 Constructing Sample Spaces! A good way to construct the sample space is to write down examples of typical outcomes and try to identify the complete set! Example: Toss a coin four times One typical outcome is four consecutive heads (H,H,H,H), another is a head, tail, head and head (H,T,H,H) A little thought results in identifying the sample space as the set of all such 4-tuples S={ (H,H,H,H),(H,H,H,T), (H,H,T,H), (H,T,H,H), (T,H,H,H), (H,H,T,T), (H,T,H,T), (H,T,T,H), (T,H,T,H), (T,T,H,H), (T,H,H,T), (H,T,T,T), (T,H,T,T), (T,T,H,T), (T,T,T,H), (T,T,T,T) } 9 A Counting Rule for ultiple-step Experiments! If an experiment consists of a sequence of k steps in which there are n possible results for the first step, n possible results for the second step, and so on, then the total number of experimental outcomes is given by (n )(n )... (n k )! A helpful graphical representation of a multiple-step experiment is a tree diagram 0

6 Counting Rule for Combinations Another useful counting rule enables us to count the number of experimental outcomes when n objects are to be selected from a set of N objects! Number of combinations of N objects taken n at a time N Cn ' = % & N n $ " # = N! n!( N n)! where N! = N(N - )(N - )... ()() n! = n(n - )( n - )... ()() 0! = Counting Rule for Permutations A third useful counting rule enables us to count the number of experimental outcomes when n objects are to be selected from a set of N objects where the order of selection is important! Number of permutations of N objects taken n at a time: N Pn ' N $ = n! % " & n # = N! (N n)!

7 Assigning Probabilities! Classical ethod:! Assigning probabilities based on the assumption of equally likely outcomes! Relative Frequency ethod:! Assigning probabilities based on experimentation or historical data! Subjective ethod:! Assigning probabilities based on the assignor s judgment! Applied in economics and related sciences 3 Classical ethod If an experiment has n possible outcomes, this method would assign a probability of /n to each outcome.! Example: Experiment: Rolling a die Sample Space: S = {,, 3, 4, 5, 6} Probabilities: Each sample point has a /6 chance of occurring. 4

8 Relative Frequency of an Outcome! Suppose that, in a large number of repetitions, N, of the experiment, the outcome O, occurs The relative frequency of O is, n O times. n O N We can think of the probability of O as the value to which the relative frequency settles down as N gets larger and larger. 5 Events and Their Probability! An event is a collection of sample points! The probability of any event is equal to the sum of the probabilities of the sample points in the event! If we can identify all the sample points of an experiment and assign a probability to each, we can compute the probability of an event! There are some basic probability relationships that can be used to compute the probability of an event without knowledge of all the sample point probabilities:! Complement of an Event! Union of Two Events! Intersection of Two Events! utually Exclusive Events 6

9 Complement of an Event! The complement of event A is defined to be the event consisting of all sample points that are not in A! The complement of A is denoted by A c! The Venn diagram below illustrates the concept of a complement Sample Space S Event A A c 7 Union of Two Events! The union of events A and B is the event containing all sample points that are in A or B or both! The union is denoted by A B! The union of A and B is illustrated below Sample Space S Event A Event B 8

10 Intersection of Two Events! The intersection of events A and B is the set of all sample points that are in both A and B! The intersection is denoted by A Β! The intersection of A and B is the area of overlap in the illustration below Intersection Sample Space S Event A Event B 9 Addition Law! The addition law provides a way to compute the probability of event A, or B, or both A and B occurring! It is written as: P(A B) = P(A) + P(B) - P(A B) 0

11 utually Exclusive Events! Two events are said to be mutually exclusive if the events have no sample points in common! That is, two events are mutually exclusive if, when one event occurs, the other cannot occur Sample Space S Event A Event B! Addition Law for utually Exclusive Events: P(A B) = P(A) + P(B) Conditional Probability! The probability of an event given that another event has occurred is called a conditional probability! The conditional probability of A given B is denoted by P(A B)! A conditional probability is computed as follows: P( A B) = P( A B) P( B)

12 ultiplication Law! The multiplication law provides a way to compute the probability of an intersection of two events! The law is written as: P(A B) = P(B)P(A B) 3 Independent Events! Events A and B are independent if P(A B) = P(A)! ultiplication Law for Independent Events P(A B) = P(A)P(B)! The multiplication law also can be used as a test to see if two events are independent 4

13 Random Variables! A random variable is a numerical expression of the outcome of an experiment! Example : toss a die twice; count the number of times the number 4 appears (0, or times)! Example : toss a coin; assign $0 to head and -$30 to a tail! Discrete random variable:! Can only take discrete values! Obtained by counting (0,,, 3, etc.)! Usually a finite number of different values! E.g., toss a coin 5 times; count the number of tails (0,,, 3, 4, or 5 times) (compiled from [BLK0]) 5 Discrete Probability Distribution! A discrete probability distribution is the list of all possible [ j, P( j )] pairs, with:! j = Value of random variable! P( j ) = Probability associated with value! This list can be visualized with a histogram! utually Exclusive (Nothing in Common)! Collective Exhaustive (Nothing Left Out) ( j) P( j) 0 P = 6

14 Basic Summary easures ()! Expected Value (The ean):! Weighted average of the probability distribution µ = E( ) = jp( j) j! E.g., toss coins, count the number of tails, compute expected value: µ = j j ( j) P ( )( ) ( )( ) ( )( ) = = 7 Basic Summary easures ()! Variance:! Weighted average squared deviation about the mean ( ) ( j ) ( j) E" # P % & σ = µ = µ! E.g., Toss coins, count number of tails, compute variance: ( j ) P( j) σ = µ ( 0 ) (.5) ( ) (.5) ( ) (.5) = + + =.5! The standard deviation is the square root of the variance: σ = σ 8

15 Binomial Probability Distribution! n Identical Trials! E.g., 5 tosses of a coin; 0 light bulbs taken from a warehouse! utually Exclusive Outcomes on Each Trial! E.g., Heads or tails in each toss of a coin; defective or not defective light bulb! Trials are Independent! The outcome of one trial does not affect the outcome of the other! Constant Probability for Each Trial! E.g., Probability of getting a tail is the same each time we toss the coin! Sampling ethods! Infinite population without replacement! Finite population with replacement 9 Binomial Probability Distribution Function P( ) = P n!!( n )! p p ( ) n ( ) : probability of successes given n and p ( ) : number of "successes" in sample = 0,,,n p : the probability of each "success" n : sample size Tails in Tosses of Coin P() 0 /4 =.5 /4 =.50 /4 =.5 30

16 Binomial Distribution Characteristics! ean! E.g., ( ) µ = E = np ( ) µ = np = 5. =.5! Variance and Standard Deviation!! E.g., σ ( p) = np ( p) σ = np P() ( p) ( )( ) σ = np = 5.. =.6708 n = 5 p = Poisson Distribution Siméon Poisson 3

17 Poisson Distribution! Discrete events ( successes ) occurring in a given area of opportunity ( interval )! Interval can be time, length, surface area, etc.! The probability of a success in a given interval is the same for all the intervals! The number of successes in one interval is independent of the number of successes in other intervals! The probability of two or more successes occurring in an interval approaches zero as the interval becomes smaller! E.g., # customers arriving in 5 minutes! E.g., # defects per case of light bulbs 33 Poisson Probability Distribution Function ( ) P ( ) P e λ λ =! : probability of "successes" given λ : number of "successes" per unit λ : expected (average) number of "successes" e :.788 (base of natural logs) E.g., Find the probability of 4 customers arriving in 3 minutes when the mean is 3.6. ( ) P e 3.6 = =.9 4! 34

18 Poisson Distribution Characteristics! ean i=! Standard Deviation and Variance N ( ) µ = E = λ i ( ) P() = P λ = 0.5 i σ = λ σ = λ P() λ = Continuos Random Variables! Continuous random variable:! Values from interval of numbers (absence of gaps)! This implies, that for every single value, the probability that the variable takes on this value is 0! Continuous probability distribution:! Distribution of continuous random variable! Transition from a histogram to a continuous function! The probability that the random variable takes on a value in the interval [a, b] is the area under the function between the values a and b! Probability density function:! The derivation of the probability distribution! Important continuous probability distributions:! The uniform distribution! The exponential distribution! The normal distribution 36

19 The Uniform Distribution! Properties:! The probability of occurrence of a value is equally likely to occur anywhere in the range between the smallest value a and the largest value b! Also called the rectangular distribution! ean: µ = ( a+ b)! Variance: σ = ( b a) 37 The Uniform Distribution! The probability density function of the uniform distribution f = if a b ( ) ( b a)! Application: Selection of random numbers! E.g., A wooden wheel is spun on a horizontal surface and allowed to come to rest. What is the probability that a mark on the wheel will point to somewhere between the North and the East? P o 90 0 < < 90 = = 0.5 o 360 o o ( ) 38

20 Exponential Distributions ( ) P arrival time < = e λ : any value of continuous random variable λ : the population average number of arrivals per unit of time / λ: average time between arrivals e =.788 E.g., Drivers arriving at a toll bridge; customers arriving at an AT 39 Exponential Distributions! Describe Time or Distance between Events! Used for queues! Density Function! x f ( x) = e λ λ f() λ = 0.5 λ =.0! Parameters! µ = λ σ = λ 40

21 Exponential Distributions: Example! Customers arrive at the checkout line of a supermarket at the rate of 30 per hour.! What is the probability that the arrival time between consecutive customers will be greater than 5 minutes? λ = 30 = 5/ 60 hours ( arrival time > ) = ( arrival time ) P P = =.08 30( 5/ 60) ( e ) 4 The Beta distribution! Often use in the absence of better knowledge! Distributed between two distinct points! PDF: f(x) = x ( x) B(, ) B(, )! The function only normalizes the function = =! For uniform distribution [Stolen from Wikipedia] 4

22 The Normal Distribution! Properties:! Bell Shaped! Symmetrical! ean, edian and ode are Equal! Random Variable Has Infinite Range f() µ ean edian ode 43 The athematical odel f ( ) = e πσ ( ) ( ) (/ ) $ & µ / σ %' f : density of random variable π 3.459; e.788 µ : population mean σ : population standard deviation : value of random variable ( < < ) 44

23 any Normal Distributions There are an Infinite Number of Normal Distributions Varying the Parameters σ and µ, We Obtain Different Normal Distributions 45 The Standardized Normal Distribution When is normally distributed with a mean and a standard deviation, Z = µ σ µ σ follows a standardized (normalized) normal distribution with a mean 0 and a standard deviation. f(z) f() σ σ = Z µ µ Z = 0 Z 46

24 Finding Probabilities Probability is the area under the curve! Pc ( d) =? f() c d 47 Which Table to Use? Infinitely any Normal Distributions eans Infinitely any Tables to Look Up! 48

25 Solution: The Cumulative Standardized Normal Distribution Cumulative Standardized Normal Distribution Table (Portion) Z µ = 0 σ = Z Z Probabilities 0 Z = 0. Only One Table is Needed 49 Standardizing Example Normal Distribution σ =0 µ 6. 5 Z = = = 0. σ 0 Standardized Normal Distribution σ = Z Z µ = 5 µ Z = 0 50

26 Example: P ( ).9 7. =.664 Z µ.9 5 µ 7. 5 = = =. Z = = =. σ 0 σ 0 Normal Distribution σ =0 Standardized Normal Distribution.083 σ = Z Z µ = 5 µ = 0. 0 Z 5 Example: P( ).9 7. =.664 Cumulative Standardized Normal Distribution Table (Portion) Z µ = 0 σ = Z 0 Z = 0. Z.583 5

27 Example: P( ).9 7. =.664 Cumulative Standardized Normal Distribution Table (Portion) Z µ = 0 σ = Z Z = -0. Z 53 Back to Simulation...! When performing a simulation study, we are usually interested in obtaining answers to some quantitative questions! One very common question in this respect is:! What is the mean value of some defined metric of the system under study?! Example:! Packets arrive with an average inter-arrival time of / λ at a router! The router has two outgoing links and arriving packet joins link i with probability φ i! The service time on link i is µ i λ µ µ (mostly compiled from [Tow04]) 54

28 Gathering Performance Statistics! Average delay at queue i:! Record D ij : delay of customer j at queue i! Let N i be # customers passing through queue i T i N i j= = N D i ij! Throughput at queue i, γ i = N i total simulated time! Average queue length at i: N i = γ T i i Little s Law (not treated here) 55 Analyzing Output Results ()! Each time we run a simulation (using different random number streams), we will get different output results! distribution of random numbers to be used during simulation (interarrival, service times) random number sequence input simulation output output results random number sequence input simulation output output results input output random number sequence simulation output results 56

29 Analyzing Output Results () λ µ µ! Example: Delay experienced by each customer in queue! W,n : delay of nth departing customer from queue 57 Analyzing Output Results (3) λ µ µ! Each run shows variation in customer delay! One run different from next! Statistical characterization of delay must be made! Expected delay of n-th customer! Behavior as n approaches infinity! Average of n customers 58

30 Transient Behavior! Simulation outputs that depend on initial condition (i.e., output value changes when initial conditions change) are called transient characteristics! Early part of simulation! Example: The first customer entering a supermarket in the morning will always find an empty queue at the counter! Later part of simulation less dependent on initial conditions λ µ µ 59 Effect of Initial Conditions! Histogram of delay of 0 th customer, given initially empty (000 runs)! Histogram of delay of 0 th customer, given non-empty conditions 60

31 Steady state behavior! Output results may converge to limiting steady state value if simulation run long enough avg delay of packets [n, n+0] Knee avg of 5 simulations! Discard statistics gathered during transient phase, e.g., ignore first k measurements of delay at queue T i = N i j= k N i D ij k! Pick k so statistic is approximately the same for different random number streams and remains same as n increases 6 Estimating the Transient Phase ()! Perform m independent replications containing n observations each: { } m i, j, j =,..., n i! Calculate mean across all replications: = m j = i, j, j =,..., n m i=! Calculate overall mean across all replications: = n n! Remove first k observations when calculating overall mean: k n = n k j= k+ j j=, j k =,..., n 6

32 Estimating the Transient Phase ()! Compute and plot as a function of k: k! Identify the position k 0 of the knee and discard the first k 0 observations in subsequent runs! In order to facilitate computation of, it is better to record the running sum of all prior values instead of recording the values themselves:! Individual values can still be easily obtained by simple subtraction! With this we get: k k S i =, k j= i, j = n m m i= S i, n n k m m and k = ( Si, n Si, k ) i= 63 Confidence Intervals ()! Run simulation: get estimate as estimate of performance metrics of interest! Repeat simulation times (each with new set of random numbers), get, all different!! Which of, is right?! Intuitively, average of samples should be better than choosing any one of samples j How confident j= = are we in? 64

33 Confidence Intervals ()! We can not get perfect estimate of true mean, µ, with finite # samples! Instead, we look for bounds: find c and c such that: [c,c]: confidence interval 00(-α)%: confidence level Probability(c < µ < c) = α! One approach for finding c, c (suppose α=.)! Take k samples (e.g., k independent simulation runs)! Sort! Find largest value is smallest 5% -> c! Find smallest value in largest 5% -> c 65 Confidence Intervals The Central Limit Theorem! Central Limit Theorem: If samples, are independent and from same population (independent identically distributed, i.i.d.) with population mean µ and standard deviation σ, then the sample mean: j= = is approximately normally distributed with mean µ and standard deviation j σ 66

34 Confidence Intervals (3)! However, we usually do not know the population s standard deviation! So, we estimate it using the sample s (observed) standard deviation: s = ( m )! Given, s we can now find upper and lower tails of normal distributions containing α00% of the mass m= 67 Interpretation of Confidence Intervals! If we calculate confidence intervals as in the above recipe for α = 0.05, 95% of the confidence intervals thus computed will contain the true (unknown) population mean 68

35 Computing a Sample s Variance Simplified! Computing the sum of squared samples leads to simpler computation: s = i= = ( i ) = # % $ i= i= ( i i + ) & i i + ( ' i= = # & % i + ( $ ' = i= i= i i= But: This may lead to numerical problems, if numbers get big and variance is small! 69 Computing a Sample s Variance (Numerically Stable)! In 96, B. P. Welford proposed a method in an article that has been incorporated into Donald Knuth s book The Art of Programming ( nd volume, page 3, 3 rd edition): = x ; S = 0 ( ) k = k + x k k k S k = S k + ( x k k ) x k k ( ) For k n, the k th estimate of the variance is: s = S k k ( ) (see also: 70

36 Confidence Intervals: The Recipe ()! Given samples,, (e.g., having repeated simulation times), compute j= = j σ = m= ( m ) 95% confidence interval: ±.96σ 7 Confidence Intervals: The Recipe ()! Why does this work?! If the i have been observed running the same simulation program with different seed values for the random number generation (with enough distance between the seed values), then they are independent and identically distributed (i.i.d.) with some mean µ and variance σ! With = i i= we have: E & # & # [ ] = E i = E i! " $ % i= = E i=! " $ % i= [ ] = µ = µ i 7

37 Confidence Intervals: The Recipe (3)! Furthermore, and VAR& ( ' i= % # = VAR[ i ] σ $ i = # σ i! = σ = i= & VAR $ = VAR % " [ ]! Because of the central limit theorem, for large the are normal distributed with expected value µ and standard deviation σ! In practice, this gives a good estimation already for >30 73 Confidence Intervals: The Recipe (4)! Consider now the random variable Z = σ µ! Z is normal distributed with mean 0 and variance Z α! We now look for and so that! From the table of the N(0,) normal distribution we obtain the value for a given α (~ the amount of confidence we are aiming at )! For α = 0.05, we obtain Z α P( Zα Z Z % α = α &' #$ Z α =.96 Z α 74

38 Confidence Intervals: The Recipe (5) P α '( ) Z Z Z ) P ' Z ' '( P ) '( P ) '( α Z Z α α σ σ α & $% µ Z σ = α α & $ = α $ $ % µ µ + Z α + Z σ α σ & $% & $% = α = α.96σ ±! This leads to our formula for the confidence interval 75 Generating Confidence Intervals for Steady State easures! In order to obtain enough i.i.d. observations, we have two simple alternatives (others exist):! Independent replications: perform independent runs with different seeds (also remember to delete k observations from transient phase): However, if we want to use the central limit theorem, we should at least perform 30 independent simulation runs... Alternatively, we can use a student-t distribution instead of the normal distribution if <30 (see next slides)! Batch means: take a single run, delete first k observations, divide remainder into n groups and obtain i for i-th group, i =,,n Follow procedure for independent replications Complication due to non-independence of i s Potential efficiency gain due to deletion of only k observations 76

39 Confidence Intervals with Student-T Distribution ()! When the sample size is relatively small, however, the approximation of the confidence interval with the normal distribution is poor! In this case, one can make use of the student-t distribution! If is the sample mean of a random sample of size from a random population having the mean µ and standard deviation S, then the random variable T = µ S has student-t distribution with - degrees of freedom! Thus, the 00(-α)% confidence interval of µ can be obtained as t ; α S < µ < + t ; α S (source: [Tri0]) 77 78

40 Confidence Intervals with Student-T Distribution ()! In the above formula, ; is defined such that the area under the t probability density function to its right is equal to α, that is P( T > t ) ; α = α n! This value can be read from a table (see next slide)! Example: t n α! Estimate the average execution time of a program, that was run six time with randomly chosen data sets, obtaining sample mean = 30 ms and a sample standard deviation of s = 4! To obtain a 98% confidence interval of the true mean execution time µ, we read t 5;0.0 from the table of student-t distribution with n- = 5 degrees of freedom to be 3.365, leading to the following confidence interval: < µ < or < µ < (with 98% confidence) 79 Important Values of the Student-T Distribution df a = degrees of freedom 80

41 Additional References [BLK0]. L. Berenson, D.. Levine, T. K. Krehbiel. Basic Business Statistics. course slides to the ninth edition of the book. Prentice-Hall, [Jain9] R. Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, easurement, Simulation, and odeling. Wiley, 99. [Karl04] H. Karl. Praxis der Simulation. course slides, Technische Universität Berlin. [Rob0] T. Robinson. Probability. course slides, University of Bath, ~masar/univ0037/probability.ppt [Tow04] Don Towsley. Network Simulation. slides for course Advanced Foundations of Computer Networks, University of assachusetts, USA, [Tri0] K. S. Trivedi. Probability and Statistics with Reliability, Queueing and Computer Science Applications. Wiley, 00. [Var04] A. Varga. ONeT++: Object-Oriented Discrete Event Simulator