http://www.xelca.l/articles/ufo_ladigsbaa_houte.aspx

imulatio Output aalysis 3/4/06

This lecture Output: A simulatio determies the value of some performace measures, e.g. productio per hour, average queue legth etc... If your model cotais radom iput values e.g. customers iterarrival times, your output, i.e., performace measures, are stochastic variables as well I this lecture you lear basic statistical priciples to aalyse the output values a simulatio 3 3/4/06

4 http://www.xelca.l/articles/ufo_ladigsbaa_houte.aspx

Output aalysis Quote from Law simulatio book: `imulatio is computer-based statistical samplig experimet 5 3/4/06

Output aalysis: Idepedece across rus ru : ru :,,,,,, j, j, ru i : i, i,, ij, j, j,, ij are idepede t 6 3/4/06

Types of simulatio w.r.t. output aalysis Termiatig: Edpoit of simulatio ru is defied by your model. No-termiatig 7 3/4/06

teady state example M M queue sigle server queue with expoetial iter arrival ad service times ad ρ=0.9 D i : waitig time of customer i s umber of customers preset at time 0 8 3/4/06

teady state Y i, i-th realizatio withi a simulatio ru I: iitial coditios Cosider the coditioal distributio fuctio of Y i I a steady state we have: F i y I P Y i y I F i y I i F y for all y, I 9 3/4/06

0 3/4/06 teady state

If you do ot have a steady state you might have: Time axis ca be divided ito time iterval cycles. Oe week i call ceter Oe week i a emergecy departmet Y C i radom variable o i-th cycle e.g. umber of calls with a waitig time loger tha 5 miutes i week i a call ceter Y C Y C Y C 3 has a steady state distributio F C. teady cycle 3/4/06

No-termiatig simulatio teady state teady cycle Others 3/4/06

We start doig some statistics i geeral Exercise: take cards from a deck of cards without returig ay card that has take = umber of aces, Y=umber of kigs Idepedet or ot? how why. 3 3/4/06

4 Idepedece Two stochastic variables ad Y are idepedet if: Cotiuous Discrete B A B Y P A P B Y A P, sets ad y x y Y P x P y Y x P, ad imulatio, lecture 5

Idepedece If ad Y idepedet stochastic variables: EY EEY cov,y 0 var Y var vary 5 imulatio, lecture 5

Estimators ubiased Assume,, IID stochastic variables ample mea: estimates μ i i ample variace: i i 6 3/4/06

trog law of large umbers,, IID stochastic variables, E = μ with probability if 7 3/4/06

8 3/4/06 Cetral limit theorem,,, IID, average µ, variace σ if the Z ormally distributed N0, Z i i

,, IID stochastic variables Cofidece iterval t Follows studet s t-distributio with - degrees of freedom Note σ replaced by estimate Assumptio ot too strict: i are ormally distributed 9 3/4/06

tudet s t-distributio http://www.statsoft.com/textbook/distributio-tables/#t 0 3/4/06

Cofidece iterval t, t,, 3/4/06

Cofidece iterval: example How may hours do computer sciece studets sped o gamig? 8, 5, 8,, 3, 8, 8, 6, 5, t9.3 0 3.34 0.05.6 95% cofidece iterval: [.3.6 3.34 0,.3.6 3.34 0 Questio: This meas that with 95 % probability. ] [9.69,4.9]

Cofidece iterval t What does this mea?, t,, -α00 % cofidece, µ is i the iterval with probability -α, t -,-α/ coverges to z -α/ for large Ed some statistics i geeral 3 3/4/06

Aalysis: first cosider termiatig simulatio ru : ru ru : :,,,,,,, j j,,,, j, avg avg avg 4 3/4/06

Termiatig simulatio i output result, value of a certai performace measure, of simulatio ru i E.g. average waitig time of patiet i hospital departmet i ru i The i s ca be cosidered as Idepedet Idetically Distributed IID stochastic variables We wat to have iformatio μ = E i tatistical theory applies 5 3/4/06

6 3/4/06 Termiatig simulatio 3 t t,,, -α00 % cofidece, µ is i the iterval with probability -α For estimatig average i i

No termiatig simulatio eparate ru Batch meas 7 3/4/06

No-termiatig simulatio Replicatio/deletio approach, i.e. separate rus: Iitializatio effect i.e. warm-up period Either very large rus or Kow cofidece iterval from: i K K jk 0 K i, j 0 8 3/4/06

No-termiatig simulatio Batch meas method sub rus: Correlatio Either very large rus or Assume Covariace statioary: Weak idepedece: Cofidece iterval from cov i, ik idepedet of cov i, i 0 k i K j K i, j 9 3/4/06

30 3/4/06 No-termiatig simulatio 3 with replaced by is i i i C C,,, C t C t

3 3/4/06 Warm up period ->teady state,,,, : average,,,, : ru,,,, : ru,,,, ru : j j j

Warm up period ->teady state ij j-th observatio i ru i j i ij Movig average should coverge j w w sw w js 3 3/4/06

Warm-up period: example Expoetial iterarrival times with mea miute Machie processig times uiform [0.65,0.7] miutes Ispectio times uiform [0.75,0.8] miutes Machie: life time exp6 hours ad repair times uiform 8 to miutes. 33 3/4/06

N i productio i hour i N i Average over 0 rus: 34 3/4/06

N i 0 0 N i N i 30 35 3/4/06

Comparig systems: example Arrival: Poisso per miute Two types of ATM s: Zippy: service time exp0.9 mi Kluky: service time exp.8 mi Oe Zippy or Klukies? Cost are equal Average customer delay matters 36 3/4/06

Comparig systems Zippy: j j=,.., average delay ru j Klukies: j j=,.., average delay ru j Compare: perform the followig experimet 00 times: Compare average delay of rus with Zippy to average delay of rus with Klukies. Vote Zippy/Klukies # rus 5 5 43 0 38 0 34 % Zippy 37 3/4/06

38 3/4/06

39 Comparig two systems: use paired t- cofidece iterval 3/4/06 t Z Z Z Z Z Z Z Z Z j j Z j j j j j Cofidece iterval : df - distributio with follows t ] [ EZ /,

tudet s t-distributio α=0.05 tn- -α from table 95% = - α 0.5% = α/ -t crit = -tn- -α/ t crit = tn- -α/ 40 Oderzoeksmethode: statistiek 3

Comparig two systems: use paired t- cofidece iterval Z j j j cofidece iterval: Z t, / If 0 i cofidece iterval, o sigificat differece Z If If left side of iterval Z t right side iterval Z t, /, / Z Z 0, the 0 the larger tha smaller tha 4 3/4/06

4 3/4/06 Backgroud: Paired t-test two-sided ] [ 0 : 0 : 0 Z Z Z Z H H Z E Z j j Z j j j j j Used i course Evolutioary Computig

Paired t-test t obs Z Z We wat cofidece level o we accept follows a t - distributio with df H 0 whe t, t obs t, 43 3/4/06

Paired t-test: p-value p-value or sigificace: Idicates how extreme t obs p - value mi P T t obs, P T t, where T follows a t - distributio with df. obs We reject H if p 0 0.05 44 3/4/06

tudet s t-distributio p value t obs 45

Relatio to paired-t cofidece iterval Accept H 0 : μ=ez=0 if ad oly if t, if ad oly if Z, Z t 0 is i the paired t-cofidece iterval A cofidece iterval gives more iformatio 46 3/4/06

Variace reductio Use commo radom umbers for ad 47 3/4/06

48 3/4/06 A modified two sample-t cofidece iterval Welsch cofidece iterval observatios j, observatios j If both samples do ot have the same variace ] [ ] [, ] [ with : iterval cofidece df o with distributi - t q q t q