A COMPARISON OF OUTPUT-ANALYSIS METHODS FOR SIMULATIONS OF PROCESSES WITH MULTIPLE REGENERATION SEQUENCES. James M. Calvin Marvin K.

Transcription

1 Proceedings of the 00 Winter Simulation Conference E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, eds. A COMPARISON OF OUTPUT-ANALYSIS METHODS FOR SIMULATIONS OF PROCESSES WITH MULTIPLE REGENERATION SEQUENCES James M. Calvin Marvin K. Nakayama Department of Computer Science New Jersey Institute of Technology Newark, NJ 070, U.S.A. ABSTRACT We compare several simulation estimators for a performance measure of a process having multiple regeneration sequences. We examine the setting of two regeneration sequences. We compare two existing estimators, the permuted estimator and the semi-regenerative estimator, and two new estimators, a type of U-statistic estimator and a type of V -statistic estimator. The last two estimators are obtained by resampling trajectories without and with replacement, respectively. The permuted estimator and the U-statistic estimator turn out to be equivalent, but the others are in general different. We show that when estimating the second moment of a cumulative cycle reward, the semi-regenerative and V -statistic estimators have non-negative bias, with the semi-regenerative bias being larger. The permuted estimator was previously shown to be unbiased. Although some of the estimators have different small-sample properties, they all satisfy central limit theorems with the same asymptotic variance constant. INTRODUCTION A regenerative process is a process that has an infinite sequence of stopping times, known as regeneration points, at which times the process probabilistically restarts. The evolution of the process between two successive regeneration points is called a regenerative cycle, and cycles are i.i.d. e.g., see Shedler 99). For example, for an irreducible, positiverecurrent Markov chain, the successive hitting times to a fixed state constitute a regeneration sequence. The standard regenerative method Crane and Iglehart 975) exploits the i.i.d. cycle structure in regenerative processes to construct asymptotically valid confidence intervals for a large class of performance measures. Many regenerative processes possess several different regeneration sequences. For example, for a Markov chain, each fixed state yields a regeneration sequence, but there are many choices for the state to use. The standard regenerative method only uses one of these sequences, and in this paper, we examine several estimators that exploit more than one sequence. Throughout the paper, we consider estimating the second moment of the cumulative cycle reward of a discretetime Markov chain on a discrete state space, but the ideas can also be applied to other performance measures and more general processes having multiple regeneration sequences. Also, we consider only two regeneration sequences. Several methods to exploit multiple regeneration sequences have been proposed. 998, 000) developed permuted regenerative estimators, which can be described in the context of regeneration sequences as follows. Run a simulation of a fixed number of cycles from one sequence, and based on the resulting sample path, calculate an estimate of the performance measure. Construct a new sample path from the original one by permuting the cycles from both sequences, and calculate an estimate based on the new path. Averaging over all permutations yields the permuted estimator. Calculation of this estimator does not require constructing all permutations since a simple closedform representation for the estimator can be derived. Calvin and Nakayama 000) show that the permuted estimator always has the same expectation as the standard regenerative estimator, so if the standard estimator is unbiased, so is the permuted estimator. Moreover, compared to the standard estimator, the permuted estimator always has no larger, and typically strictly smaller, variance. Calvin, Glynn, and Nakayama 00) developed another approach, the semi-regenerative method, which gets its name from its close connection to semi-regenerative processes see Section 0.6 of Çinlar 975). The idea is to fix a set A of states, and to break up the sample path into trajectories determined by successive entrances into A. Using the semi-regenerative structure, one derives an alternative representation of the performance measure in terms of expectations of random quantities defined over trajectories, and then constructs an estimator by replacing each

2 expectation by its natural estimator. Calvin, Glynn, and Nakayama 00 also present a stratified semi-regenerative estimator in which trajectories are sampled in an i.i.d. fashion, but we will not consider that approach in this paper.) Other methods for simulating processes with multiple regeneration sequences include the almost regenerative method Gunther and Wolff 980) and A-segments Zhang and Ho 99). In this paper, we present two new ways to construct estimators. Both approaches start with a sample path of a fixed number of cycles with respect to one regeneration sequence. The path is then broken into trajectories determined by visits into a set A, and the new estimators are obtained by resampling the trajectories and averaging over all possible resamples. One estimator considers resampling without replacement, which is a type of U-statistic, and the other resamples with replacement, which is a type of V -statistic. See Serfling 980, Chapter 5, for details on U- and V -statistics.) We compare four estimators: the permuted, semiregenerative, U-statistic, and V -statistic estimators. We show that the U-statistic and permuted estimators are equivalent. The other two estimators are different, and we derive the exact differences of all the estimators based on a sample path of a fixed number m of cycles from one regeneration sequence. It turns out that the semi-regenerative and V - statistic estimators are biased high for any m, with larger bias for the semi-regenerative estimator. Asymptotically, all four estimators are equivalent and satisfy central limit theorems with the same asymptotic variance constant. The rest of the paper has the following organization. In Section we describe the mathematical model and the performance measure considered. We derive the semiregenerative estimator in Section, and we present the permuted estimator from 000) in Section 4. Section 5 contains the resampled estimators, and we compare the four estimators in Section 6. We state some conclusions in Section 7. All proofs of theorems stated in this paper are given in 00). FRAMEWORK For simplicity, we specialize the development to discretetime Markov chains, but the results hold for more general processes with multiple regeneration sequences. Let X = X j : j = 0,,,...) be an irreducible positive-recurrent discrete-time Markov chain on a discrete state space S = {,,,...}. For any state x S, we can define a sequence of regeneration points Tx) = T k x) : k = 0,,,...), where T 0 x) = inf{j 0 : X j = x} and T k x) = inf{j > T k x) : X j = x}, k. We call the sample-path segment X j : T k x) j<t k x)) the kth x-cycle. Fix a state w S. Forx S, define αx) = τ fx k ), where τ = inf{k : X k = w} and f : S Ris a reward function. Our goal is to estimate αw). We will compare the estimator of αw) using the semi-regenerative method and the permuted estimator of 000), and we will present two other estimators of αw) based on resampling. Fix a set A S with w A. Define T 0 = inf{j : X j A} and T k = inf{j T k : X j A} for k. We will carry out our analysis with A ={, } and w =. Also, we will assume that we simulate one sample path of a fixed number m of -cycles, and let the sample path be X m = X j : j = 0,,...,T m )), where X 0 =. Note that τ = T ). Also, let W = W k : k = 0,,,...) with W k = X Tk for k 0, which is the embedded chain of X on visits to the set A. Let M = m {T k ), k 0 : T k ) <T m )}, which is the number of returns to the set A up to time T m ), sot M = T m ). The standard estimator of α) based on the sample path X m is where for k =,,...,m. t STD X m ) = m Y k = T k ) j=t k ) m Yk, ) k= fx j ) SEMI-REGENERATIVE ESTIMATOR We now derive the semi-regenerative estimator for α). The key to applying the semi-regenerative method is deriving an alternative representation for α) based on returns to the set A. Let T = inf{k : X k A}. Forx A, note that [ T αx) = fx k ) τ Iτ > T,X T = y) fx k ) y A

3 Expanding this formula for αx), we see that it is composed of three terms: αx) = T fx k ) τ T Iτ > T,X T = y) fx k ) fx j ) y A j=0 τ Iτ > T,X T = y) fx k ). ) y A We now derive new expressions for the second and third terms in ). Each summand in the second term in ) satisfies τ Iτ > T,X T = y) = Fx,y)ey), T fx k ) j=0 fx j ) where ] T Fx,y) = [Iτ > T,X T = y) fx k ), Also so letting [ τ ] ey) = E y fx k ). [ T ] ex) = fx k ) y A [Iτ > T,X T = y)] ey), [ T ] qx) = fx k ), and Gx, y) = [Iτ > T,X T = y)],e= ex) : x A), q = qx) : x A), and G = Gx, y) : x,y A), we see that e = q Ge, ore = I G) q. Thus, each summand in the second term in ) satisfies τ Iτ > T,X T = y) = Fx,y)I G) q)y). T fx k ) j=0 X j For the summand in the third term in ), we make the observation that τ Iτ > T,X T = y) fx k ) Now define = Gx, y) αy). cx) = T fx k ). Let α = αx) : x A), c = cx) : x A), and F = F x, y) : x,y A). Then, putting this all together, we get α = c FI G) q Gα, so α = I G) c FI G) q). Now suppose A = {, } and w =. Note that Gx, ) = Fx,) = 0 for x A. Let J = J x, y) : x,y A) with J = I G), and note that Thus, J = = ) G, ) 0 G, ) G, )/ G, )) 0 / G, )) )[ ) J, ) c) α = 0 J, ) c) ) 0 F, ) J, ) 0 F, ) 0 J, ) ) c) J, )c) = J, )c) so ). ) q) q) F, ) J, )F, ))J, )q) J, F, )q) )] ), α) = c) J, )c) [F, ) J, )F, )]J, )q). ) To derive an estimator based on ) from the sample path X m of m -cycles, we need estimates of the quantities on the right-hand side of ). Recall that M was defined such that T M = T m ). Forx,y A, we call a trajectory that begins in state x and ends in state y an x, y)-trajectory, and define hx, y) = M IW k = x,w k = y) as the number of such trajectories along the sample path X m. Let Hx) =

4 M IW k = x), which is the number of trajectories in X m that begin in state x. Note that hx, )hx, ) = Hx) for x A. Also, h, ) = h, ) since X m is a path of a fixed number of -cycles, and H) = m. For x,y A, let T x, y) = inf{t k : k 0, W k = x, W k = y} and T i x, y) = inf{t k T i x, y) : k 0, W k = x, W k = y} for i. Also, let T x, y) = inf{t k : k 0, W k = x, W k = y} and T i x, y) = inf{t k : k 0, T k T i x, y), W k = x, W k = y} for i. Thus, for k =,,...,hx,y), the kth trajectory starting in state x A and ending in state y A begins at time T k x, y) and finishes at time T k x, y), and we define Y k x, y) as the sum of the rewards along that trajectory; i.e., Y k x, y) = T k x,y) j=t k x,y) fx j ). Define S j x, y) = hx,y) k= Y k x, y) j. Now for x A, observe that S x, )S x, ))/H x) and S x, ) S x, ))/H x) are the natural estimators of cx) and qx), respectively. Also, we use hx, )/H x) and S x, )/H x) as estimators for Gx, ) and Fx,), respectively. Our estimator of J, ) = G, )/ G, )) is then ˆ J, ) = = ) h, ) h, ) H) H) ) h, ) H) = H) H) h, ) H) since h, ) h, ) = H) and h, ) = h, ). Similarly, our estimator of J, ) = / G, )) is J, ˆ ) = H)/h, ). Thus, the semi-regenerative estimator of α) based on the sample path X m is t SR X m ) = H) S, )S, )) ) H) S, ) S, ) H) H) [ ) ] S, ) H) S, ) H) H) H) ) H) S, ) S, ) h, ) H) = S, ) S, ) S, ) S, ) m h, ) [S, )S, ) S, )S, ) S, )S, ) S, ]) 4) since H) = m. To simplify the comparison with other estimators, we define so Q = m S, ) S, ) S, ) S, ) [ S, )S, ) S, )S, ) h, ) S, )S, ) ]), 5) t SR X m ) = Q 4 PERMUTED ESTIMATOR mh, ) S,. 6) The permuted estimator of α) based on the sample path X m is obtained as follows. Recall we defined the standard estimator of α) as t STD X m ) in ). The permuted estimator of α) is obtained by permuting -cycles and -cycles to obtain a new sample path X m, and averaging t STD X m ) over all possible permuted paths X m. Calvin and Nakayama 000) showed that the permuted estimator is t P X m ) m S, ) S, ) S, ) [ S, )S, ) h, ) ] S, )S, ) S, )S, ) h, ) h, ) S, ) h, ) S, = Q mh, ) ) where Q is defined in 5). 5 RESAMPLED ESTIMATORS S, S, ) ) ), 7) We now present two estimators obtained via resampling the x, y)-trajectories for x,y A. One estimator is based on resampling the x, y)-trajectories without replacement i.e., a permutation of trajectories), and the other resamples with replacement. Fix a sample path X m. For x,y A, define x, y) as the set of permutations of,,...,hx,y)), and let = x,y A x, y). Let x, y) ={i,i,...,i hx,y) ) : i j hx, y) for j =,,...,hx,y)}, which is the set of all hx, y)-dimensional vectors in which the components are selected with replacement from

5 {,,...,hx,y)}. Let = x,y A x, y). Define = {i,i,...,i h,) ) : i =, i h,) h, ), i j i j for j =,,...,h, ) }. Let Ɣ = and Ɣ =, and let K, D) denote a generic element of Ɣ, with K = Kx, y) : x,y A), Kx,y) = K i x, y) : i =,,...,hx,y)) x, y), and D = D,D,...,D h,) ). Also, define D h,) = h, ) and the function g : R as gk, D) = [ h,) Y Ki,), m i= Y Ki,), ) Y Ki,), ) h,) i= D i ] Y Kj,), ) j=d i = [ h,) Y Ki,), m i= Y Ki,), Y Ki,), h,) i= D i Y Kj,), ) j=d i Y Ki,), )Y Ki,), ) { Y Ki,), ) Y Ki,), ) } D i )] Y Kj,), ). 8) j=d i Observe that the function g is conditional on the original sample path X m. Note that x, y) resp., x, y)) is the set of all possible orderings of hx, y) x, y)-trajectories, where the x, y)- trajectories are chosen without resp., with) replacement, and in 8), Kx,y) x, y) is one such ordering of hx, y) x, y)-trajectories. Also, note that is the set of vectors specifying which, )-trajectories fall in between each pairing of, )-trajectory and, )-trajectory determined by K, ) and K, ); i.e., for D,...,D h,) ), slots D j to D j are the locations of the, )-trajectories in the ordered list K, ), ) of, )-trajectories that fall between the, )-trajectory indexed by K j, ) and the, )-trajectory indexed by K j, ). Observe that if D j = D j, then there are no, )-trajectories that fall between the, )-trajectory indexed by K j, ) and the, )-trajectory indexed by K j, ). Suppose we set K x, y) =,,...,hx,y)) for all x,y A, and let K = K x, y) : x,y A). Note that K x, y) x, y) for all x,y A, and K. Define D = D,D,...,D h,) ) with D i = min{j : T j, ) >T i, )} for i. Note that D. Then gk,d ) = t STD X m ), the standard estimator of α). We now define the resampled estimators and t U X m ) = Ɣ t V X m ) = Ɣ K,D) Ɣ K,D) Ɣ gk, D) 9) gk, D), 0) where t U X m ) is the estimator based on resampling without replacement, and t V X m ) is the estimator with replacement. In addition to resampling trajectories, both estimators also average over all possible allocations of h, ), )- trajectories to the -cycles containing a hit to state, which is accomplished in 9) and 0) by averaging over D. Note that t U X m ) is a type of U-statistic and t V X m ) is a type of V -statistic. Figure presents an example of a sample path X m top) and a path middle) obtained from X m by resampling without replacement the x, y)-trajectories, for x, y A. State corresponds to the horizontal axis, and state corresponds to the dashed horizontal line. The original path X m, which is shown on top, has m = 5 -cycles. Also, the original path has h, ) =, )-trajectories, h, ) =, )- trajectories, h, ) =, )-trajectories, and h, ) =, )-trajectories. Moreover, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; D =, D =, D = 4, D 4 = 4. The resampled path shown on the bottom has K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; D =, D =, D =, D 4 = 4. The bottom path in Figure is one possible path obtained from X m by resampling with replacement the trajectories. The resampled path has K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; K, ) =, K, ) =, K, ) = ; D =, D =, D =, D 4 = 4. In the resampled path, Y, ) appears twice, and Y, ) does not appear. Also, for the, )-trajectories, Y, ) appears once, Y, ) does not appear, and Y, ) appears twice. For the, )-trajectories, each Y k, ) appears once. For

6 Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Y,) Figure : A Sample Path top) and Paths Obtained by Resampling Trajectories Without Replacement middle) and With Replacement bottom)

7 the, )-trajectories, Y, ) does not appear, Y, ) appears once, and Y, ) appears twice. Theorem For all X m and m, t U X m ) = t P X m ) and t V X m ) = Q h, ) ) mh, ) h, ) ) S,, ) where Q is defined in 5). Thus, since t U X m ) = t P X m ), we see that permuting trajectories and reallocating, )-trajectories to the -cycles leads to the same estimator as permuting cycles. 6 COMPARING THE ESTIMATORS We start by comparing the semi-regenerative estimator t SR X m ) in 6) and the permuted estimator t P X m ) in 7). In the second term in the right-hand side of 7), the sum of the squares of the Y k, ) i.e., S, )) is subtracted from the square of the sum i.e., S, ), whereas the second term in the right-hand side of 6) only has the square of the sum. A similar situation occurs when comparing the V -statistic estimator t V X m ) in ) and the permuted estimator t P X m ). The estimators of α) satisfy a complete ordering. Theorem For all X m and m, where t SR X m ) t V X m ) t U X m ) = t P X m ), ) the first inequality in ) is strict if and only if S, ) = 0; the second inequality in ) is strict if and only if not all the Y k, ), k =,,...,h, ), are the same i.e., there exists i, j {,,..., h, )} such that Y i, ) = Y j, )); t SR X m )>t P X m ) if and only if Y k, ) = 0 for some k =,,...,h, ). Moreover, where E[t SR X m )] E[t V X m )] E[t U X m )] ) = E[t P X m )] = α), the first inequality in ) is strict if and only if P [S, ) = 0] < ; the second inequality in ) is strict if and only if Var[Y, )] > 0; E[t SR X m )] > E[t P X m )] if and only if P [Y, ) = 0] <. We now compare some asymptotic properties of our D estimators. Let denote convergence in distribution and Na,b) denote a normal distribution with mean a and variance b. Theorem Assume E[ T ) j=0 fx j ) ) 4 ] <. Then for t X m ) defined as either t P X m ), t SR X m ), or t V X m ), we have that t X m ) converges with probability one to α) while a suitably standardized version of t X m ) converges in distribution to a normal distribution with mean zero and variance σα, i.e., and t X m ) α), a.s., m [ t X m ) α)] D N0,σ α ) as m, for some constant σα > 0, where σ α is the same for all three estimators and is given in Calvin and Nakayama 000). For all of the estimators of α, one can estimate the asymptotic variance constant σα from one simulation run. Thus, one does not need to run multiple replications to obtain estimates of the asymptotic variance. See Calvin and Nakayama 000) for details. 7 CONCLUSIONS In this paper we showed that a wide spectrum of estimators can be obtained in simulations of processes having multiple regeneration sequences. We compared some of these estimators, two of which were proposed previously the permuted and semi-regenerative estimators) and two are new the U- and V -statistic estimators). We showed that the U-statistic and permuted estimators are equivalent. All of the other estimators are in general different, but they are asymptotically equivalent and they all satisfy the same central limit theorem. When estimating the second moment of a cycle reward, we showed that the estimators satisfy a complete ordering, with the semi-regenerative estimator being the largest, the V -statistic estimator being next largest, and the U-statistic and permuted estimators being the smallest. More work is needed to analyze the behavior of these estimators for other performance measures and to determine which is the best perhaps in terms of smallest mean squared error) for a specific measure in the small-sample context. ACKNOWLEDGMENTS This work was supported by the National Science Foundation under grants DMI and DMI Thus, t SR X m ) and t V X m ) are biased high.

8 REFERENCES Calvin, J. M. and M. K. Nakayama Using permutations in regenerative simulations to reduce variance. ACM Transactions on Modeling and Computer Simulation 8: 5 9. Calvin, J. M. and M. K. Nakayama Simulation of processes with multiple regeneration sequences. Probability in the Engineering and Informational Sciences, 4: Calvin, J. M. and M. K. Nakayama. 00. Resampled regenerative estimators. Forthcoming. Calvin, J. M., P. W. Glynn and M. K. Nakayama. 00. The semi-regenerative method of simulation output analysis. Forthcoming. Çinlar, E Introduction to Stochastic Processes. Englewood Cliffs, N.J.: Prentice-Hall. Crane, M. and D. L. Iglehart Simulating stable stochastic systems, III: Regenerative processes and discrete-event simulations. Operations Research : 45. Gunther, F. L. and R. W. Wolff The almost regenerative method for stochastic system simulations. Operations Research 8: Serfling, R.J Approximation Theorems of Mathematical Statistics. New York: Wiley. Shedler, G. S. 99. Regenerative Stochastic Simulation. San Diego: Academic Press. Zhang, B. and Y.-C. Ho. 99. Improvements in the likelihood ratio method for steady-state sensitivity analysis and simulation. Performance Analysis 5: interests include applied probability, statistics, simulation and modeling. AUTHOR BIOGRAPHIES JAMES M. CALVIN is an associate professor in the Department of Computer Science at the New Jersey Institute of Technology. He received a Ph.D. in operations research from Stanford University and is an associate editor for ACM Transactions on Modeling and Computer Simulation. Besides simulation output analysis, his research interests include global optimization and probabilistic analysis of algorithms. MARVIN K. NAKAYAMA is an associate professor in the Department of Computer Science at the New Jersey Institute of Technology. He received a Ph.D. in operations research from Stanford University. He won second prize in the 99 George E. Nicholson Student Paper Competition sponsored by INFORMS and is a recipient of a CAREER Award from the National Science Foundation. He is the area editor for the Stochastic Modeling Area of ACM Transactions on Modeling and Computer Simulation and an associate editor for Informs Journal on Computing. His research