Technical Report: CS Monte-Carlo Approximations to Continuous-time Semi-Markov Processes

Size: px

Start display at page:

Download "Technical Report: CS Monte-Carlo Approximations to Continuous-time Semi-Markov Processes"

Easter West
6 years ago
Views:

1 Technical Report: CS Monte-Carlo Approximations to Continuous-time Semi-Markov Processes Milos Hauskrecht Department o Computer Science niversity o Pittsburgh milos@cs.pitt.edu Abstract Continuous-time semi-markov processes (SMP) represent a very useul extension o Markov process models to dynamic systems whose behaviors are more naturally described in terms o time proiles. The semi-markov model however comes with a limitation: individual transitions in the model can occur at dierent times which means that or a speciic time interval the system modeled as an SMP can go through exponentially many transitions. This aects adversely the worst-case perormance o Monte Carlo methods designed to support various inerence and decision tasks. In this work we propose and analyze modiications o Monte Carlo methods that rely on simulation trajectories o a limited length. We prove that or any practical time models the probability o not reaching the target time by a trajectory o length decays exponentially ast. sing this result we analyze extensions o some o the Monte Carlo inerence methods to semi-markov settings and derive sample complexity bounds or additive estimates o desired quantities. 1 Introduction Markov processes (MPs) orm the oundations o work on modeling stochastic dynamic systems in Artiicial Intelligence. The AI research has traditionally ocused on discrete-time Markov process models their eicient representations and 1

2 inerence solutions. The Markov property enables the time decomposition o the MP model and the same time-invariant behavior in every transition. This greatly simpliies the analysis o the model and associated inerences. However Markov models are not it well to represent behaviors that are more naturally described in terms o time proiles. As has been pointed out by some researchers such processes are common in many application areas including medicine [12] transportations [2] robotics [13 7] and others [5 3]. Semi-Markov processes (SMPs) [ ] alleviate the time-invariance problem and extend Markov processes to time-dependent transitions. In continuous-time SMPs state transitions can occur at any time. Similarly to discrete-time Markov processes continuous-time SMPs allow us to decompose the process to independent time segments bounded by state transitions. However a complication is that durations o time segments can vary. As a result inerences in semi-markov processes must consider cases in which a large number o transitions occurs within a time segment. This problem can aect the worst case perormance o Monte Carlo (MC) methods that perorm the inerence by simulating many dierent state trajectories over a speciic time segment. To address the problem o long trajectories we propose and analyze a simple modiication o Monte Carlo sampling methods or SMPs. The idea o the modiication is to use only trajectories with up to transitions all other (longer) trajectories are replaced with a random guess. We show that or any practical time density model (a density model with a inite mean) the probability o generating trajectories o length more than decays exponentially ast. This is the key result that allows us to modiy many Monte Carlo algorithms developed or discrete-time models and derive sample complexity bounds or additive approximations. We illustrate the results on the analysis o algorithms or state prediction but we expect a similar transer to apply also to decision-making problems. In the ollowing we irst introduce continuous-time semi-markov processes and discuss their compact representations. Next we consider the problem o state prediction and propose a simple Monte Carlo algorithm that works with truncated state-time trajectories. We prove the key theorem o the paper the exponential decay theorem or truncated simulation trajectories and illustrate it on semi-markov models with Gamma time densities. Finally we use the theorem to analyze and derive sample complexity bounds o some MC state-prediction algorithms. 2

3 ' 2 Continuous-time Semi-Markov Process A semi-markov process (SMP) [9 10 1] extends discrete-time Markov process models to time-dependent stochastic behaviors. A semi-markov process is like a Markov process except that transition probabilities depend on the amount o time elapsed since the last change in the state. The Markov property holds as a consequence o the reset property; once a change occurs (and we know the state) the uture becomes independent o the past. More ormally a (inite-state) semi-markov process (SMP) is deined by a tuple where: is a inite set o states and is a stochastic transition model such that %! represents the probability o a transition occurring in time interval "$# & ; and is the initial state. In most cases the transition model o an SMP is not provided directly. Instead transition probabilities are deined as: ')(+* -./ (5*./0 21 where./036 3 is the probability o a transition as used in a Markov process and models the distribution o stochastic transition times. A semi-markov model approximates the dynamics o a continuous-time system by decomposing the process along times in which the state change occurs and its time model relects the luctuation o these changes in time. Compact Parameterizations. In general a semi-markov process permits models o arbitrary complexity. However or practical purposes we are interested in models with eicient parameterizations. We ocus on two aspects o parameterization: (1) parameterization o transition time densities./7 ; (2) parameterizations o state transitions./ Compact Parameterizations o Time Densities.80! Time dependencies in SMPs are modeled through time densities relecting the luctuation o transition times during the transition between states and :;%. Time models are deined over interval "9#. For the sake o representational eiciency we are interested primarily in densities with compact parameterizations. A good time density :;% model candidates are: Gamma lognormal or a Gaussian density rectiied to "9# range. Figure 1 gives examples o time models based on Gamma distribution and its mixtures. Gamma distribution belongs to the amily o 3

4 Gamma density a=5 b=10 a=5 b=5 a=2 b=5 a=5 b= Mixture o Gammas time t time t Figure 1: (a) Gamma density or 4 dierent parameter settings. (b) Mixture o 4 Gammas. exponential distributions. The density unction or Gamma distribution is: < 03=4>/'? >@ K = > AL=M where is a shape and is a scale parameter and is a Gamma unction. The./036 mixture o Gammas model is a convex combination o multiple Gamma models: B'ON;PSR EFT. 7 where #V T NWPSR? are weight parameters such that E T '?. 06 and is a Gamma model in the mixture. The advantage o the mixture model is that it gives us more lexibility in representing more complex time proiles. 2.2 Factored-state Semi-Markov Models Much o the recent research work in modeling Markov processes ocuses structured representations based on dynamic belie networks (DBNs) [6 11]. Such models allows us to represent processes over large state spaces more eiciently. In particular 'ZY[ the state o the process is actored into a set o state variables X E ]\ ^ ^^ `_6a./ and transition probabilities decompose to local transition probabilities./.4=`& 2.b=F& such that denote state variables the state variable depends on. Our goal is to extend actored representations to semi-markov processes. The key is to model appropriately time dependent interactions between an arbitrary state variable and all state variables it depends on. In particular it is necessary to deine a clock or each state variable and rules or its resets. Nodelman et al [16] 4

5 e e proposed one such model or a special class o homegeneous Markov processes. We extend the model to a more general semi-markov settings. Let c be a set o state variables the variable depends on. Our transition model or actored semi-markov processes is deined by: d Assumption 1: only one state variable can change at any instance o time; d Assumption 2: a change in values o or c resets the time or to # ; d The transition model or and ixed values o its conditioning set e is a semi-markov process model deined in terms o two probabilities: & the probability o a transition./7 set o values e o the conditioning set. 2 - time density or under e. Basically the time changes in state variables in X given c proile resets o are changes in conditioning variables and in under the ixed are conditionally independent o the rest o the. In addition the only events aecting the time itsel. 3 MC Algorithms or State Predictions in SMPs Probabilistic inerence in context o stochastic dynamic models covers predictions diagnosis and other probabilistic queries. In this work we ocus on the state g'ih`& prediction problem where we are interested in computing the probability o a uture state o the system in time given the initial state or given the distribution o initial states. In general no closed orm solution or computing the probability o j'khf& or the distribution &l or an arbitrary time-model exists. Exceptions are special cases such as homogeneous Markov processes [1 16] with exponentially distributed transition times. 3.1 Basic Monte Carlo Inerence Algorithm Monte-Carlo (MC) approximations are a natural choice when closed orm inerence solutions do not exist. The basic Monte-Carlo algorithm or state value predictions is simple: 1. generate m samples o states at time by sampling sequentially transitions (states and times) o a semi-markov model (starting rom the initial state [ ); 5

6 q % w E \ 2. estimate the target probability & o a state using the re- samples. quency o occurrences o value h in m n'oh`& Complexity analysis. The total number o simulation steps depends on the number o samples m and lengths o simulation trajectories in every sample. Assuming the same time model or every transition the dependence between the error conidence parameter and the number o samples m is captured by the ollowing (rather trivial) theorem. hm Theorem 1 Let prq Then: 2 hmuwv++')h` Mw G D \{z0 p`q yx " trajectories that guarantees the 2 approximation is m be the sample average approximation o Proo. Direct application o Hoeding s inequality [8 15]. s'thf3. The number o sample \}z ~S. To ully analyze the complexity o the Monte Carlo algorithm we need to consider the number o state transitions in every simulation trajectory reaching time. The problem is that transitions in a semi-markov model may occur at dierent times and trajectories with (exponentially) many transitions (in terms o states) may be occasionally generated. This aects negatively the worst case complexity o the basic MC solution. However we note that in the average-case analysis the expected number o transitions in]' a trajectory depends only on and the mean o the time ƒ. model and equals v+ 3.2 Modiied MC Algorithm * { The problem with the basic Monte Carlo algorithm is the worst case complexity since some trajectories may become very long. To remedy the problem we propose a modiication o the MC algorithm that simulates trajectories or at most steps. The algorithm works as ollows: d generate m samples o states at time by sampling sequentially transitions (states and times) o a semi-markov model or at most randomly the result or samples not reaching time ; d estimate the target probability samples. s'th` steps guess o a state &s'th using m We will show that or an appropriate choice o that is logarithmic in? ˆ we obtain an additive 2 approximation that is very similar in terms o the sample 6.

7 c E w complexity o m to the basic MC algorithm. The crucial thing is that the result generalizes to any practical transition time model with a inite mean or combinations o such models. Probability o Incomplete Simulation Trajectories To analyze the complexity o the modiied MC algorithm we irst prove the key result on probabilities o incomplete trajectories that is simulation trajectories that were not able to reach time in steps. We show that or a suiciently large that depends only on the characteristics o the time model density the probability o obtaining incomplete trajectories decreases exponentially ast in. The result holds or an arbitrary time density model with a inite mean. To simpliy the initial analysis we irst consider the case in which every transition in the model ollows the same time density. is a sum o independent random variables representing transition times o an arbitrary time density model with a inite mean. Theorem 2 Let r O'Š EŒ {\ Ž Then or ˆ v 7 the probability o not reaching time. ' decays exponentially ast in the number o steps in particular. ' / G D š œ Ũ 2 / 0 / where s# used the target time and a variational parameter ž#. # are constants that depend only on the time model Proo. To bound the dierence between and we build upon Cherno s exponential bound v+l [4] (see also [14]). Let be a real-valuedwtv L random variable with Žw a inite mean G D œ z v+ G œhÿ ^. Then ` or any value t# and it holds: I is a sum o independent random variables we can rewrite the bounding expression as: ` sw G D z œ v+ G œ6ÿ ' G D œ z SR v+ G œ[ ^. ' We want to bound the probability ` o not reaching time in.. steps. First can be bounded as v+r ` v+r J ' v+` ü ` swžv ` 6u ^ Assuming. v ` u that # holds Cherno s exponential bound [4] applies and can be bounded as: ª5Ç 6ª «W ±Mª ²b³ ±Mªž µ ±Mª ²b³ ²]«- C º¹$»¼¹ ½¾] 0 À Á º¹9»]¹ ½¾] 7 Â½¾` ²`ÃW À5Ä! Á M Å ²Æ 7

8 w * * * * ' ^ c *? ^ ^ = * u > G D œ The expectation v+ G Disœ over time. Since * expectation? holds or any # and # the is bounded strictly rom above by the distribution unction: (jè G D œ Thus there exists a constant? ˆ6Ê.80Éº21 0 / (-È./7Éº21/' such that v+ G D œ?. Substituting the bound into the bound or. we get: 0 / ` As Ê given? ˆ6Ê 0 8. G œ D VËÌ2Í P!œ?. y' the probability ` 2 8 G D š œ Ž 2 / 0 /8' with Ž# ~!Î6Ï Ê 0 / ˆ v 7. the probability o decays exponentially in.. y' can be bounded by Ž#. Thus The exponential decay result is critical or the analysis o Monte Carlo algorithms or continuous-time semi-markov models. Constants determining the decay rate are density speciic and need to be derived. Take or example Gamma time model. The mean o Ð This leads to a bound:. =MÑ+Ñ+=`0=b> G œ ÓËÔ E Õ œ[ö is v+0{'ò=m> and v+ G D œ G œ G ËÔ E{Õ œºö 't? /> ^ Figure 2 shows exponential bounds or a Gamma distribution and three dierent.4 values o and compares them to its empirical. The -' best value o ound by dierentiating the bound and setting the result to 0 is ˆ? ˆ. Multiple dierent time models. The assumption used in the proo o Theorem 2 was that the transition. time model is ixed or every transition in the SMP. To obtain the bound on or an SMP with many dierent time models we can substitute all model speciic quantities in the theorem with their worst case values rom among all time models. Assuming that there are dierent transition time models we E ÚÛÛÛÚ Ü v+0 % : can derive or ˆ "9ØVÙ SR. Ũ where? ˆ6Ê-à 0 /lw SR ØâáHã E ÚÛÛÛÚ Ü v / G D ÝËÌ2Í PßÞ!œ G D œ /8' Complexity o the Modiied MC Algorithm and G œ #. sing theorem 2 we can to derive the complexity o the modiied MC algorithm introduced in Section 3.2. Once again the result applies to any time density with a inite mean. 8

9 ' ' p w p z p Ũ x Ç w q 2.5 Exponential bound on pn β= β= probability β= empirical pn 0 =ä'tå >æ' 'tå Figure 2: Gamma time model example with ( x ) and #6#. An empirical.4 estimate o based on? #6##6#. trajectories and bounds on or 3 values o are shown. N ' Theorem hf& 3 Let ç be a modiied MC algorithm hm that estimates the probability using the sample average p based on m truncated simulation trajec- Ú q tories o the maximum length. Then or a ixed time density model p gives an 2 approximation 2 Ú hm/uwv+& 'khf& Mw q Øâáºã è v+0 Mé? ~ ÎÏíì x cëê Mî lïfð and m where cn # is a constant speciic to the time model. Proo. 2 Ú q hc8uòv+& 'khf&mw L' 2 Ú uwv+ ')h`º q 2 p Ú hm8uwv 'khf&[ q v+ ºw ')h`&6 u+. ž when: x \ ~Sñì x `î Ú q ]uäv +')hf3 Mw G D \ " z D6ó % ^ Last step ollows rom Hoeding s inequality. sing the bound on.4 (rom theo-. rem W' 2) G with D š?. to assure that accounts or at most hal o the error we get: \ ^. Substituting and expressing both m and we obtain: óô õ êáö! ø ì]ù îûú ï and ü ù 2ý ö!þ We note that the sample complexity result in theorem 3 is not the tightest possible and can be improved or example by optimizing the variational parameter 9 ì/ù î hm

10 Ê? w E. However the theorem illustrates clearly two very important points. First the worst-case sample complexity result or the modiied MC is comparable in terms o m to the expected sample complexity o the basic MC algorithm (see Theorem 1). Second the proo and the result show that the sample complexity bounds or and m are coupled only through error parameter. Thus using a ixed proportion o. to cover is able to ully absorb the eect o continuous time at a very modest (logarithmic in?ˆ ) expense. This separation is very convenient or extending MC algorithms developed or discrete-time Markov processes into continuous-time semi-markov settings and their subsequent sample complexity analysis. MC Algorithm or Factored SMPs The modiied MC algorithm and its analysis or lat-state SMPs (Theorem 3) can be easily extended to actored settings. In terms o complexity bounds only will change. Trajectories in actored SMPs are deined by ÿ state variable subprocesses and each o these trajectories must reach time to make a simulation run successul. Things are urther complicated by dependences among state variables. An occurrence o a transition aects all the variables that are conditioned on it and resets their clock. The reset eect may cascade through multiple dependency relations. To analyze MC algorithms or actored SMPs we consider two extreme cases: d Fully independent semi-markov subprocesses. In this case each state variable is independent o other state variables and we have ÿ independent subprocesses. The probability o not reaching the target time with at least one.4 o the Gsubpro- cesses can be bound through the union (Boneroni) bound that is: Ũ ÿ. D š Then using the same assumptions as in theorem 3 the number o steps \{_ or every š trajectory necessary to assure 2 approximation is: ~!Î6Ï z. Note that m stays unchanged. d Fully dependent semi-markov subprocesses. In this case every state variable E 2 \ process is aected by changes in all other state variables. Assuming that 2 P are random variables representing transitions times or state variables E ]\ P the time o occurrence o a transition is ØVÙ R E2ÛÛÛ P. I the time model used or each state variable is the same and ollows the den-. 77 sity the min statistic is:./0{ u 0{2 P DFE. 0 ß 0{ where is the distribution. 0{ unction o. Nevertheless Theorem 3 applies directly to any time model. But then m stays unchanged and the only dierence is the density 10

11 ê speciic constant c aecting the bound on. Extensions o the Modiied MC Method A similar disassociation pattern between m and also transers to extensions o the modiied MC algorithm to more complex inerence tasks. An example is an./ 3 MC algorithm approximating the distribution o a state at time with its empirical distribution p q Ú based on truncated simulation trajectories. In such a case the lower bound on the length o simulation trajectories or an accuracy under the? norm becomes: w? ~!Î6Ï ì x ï c î while the sample complexity bound or u the number o samples m discrete-time case by the standard ˆ x correction. 4 Conclusions diers rom the Continuous-time semi-markov processes allows us to model stochastic systems with continuous time-dependent transition proiles. This property is crucial or modeling many applications domains [12 2 3]. This advantage however comes with trade-os: (1) time-dependencies in the semi-markov process model must be explicitly represented which itsel may translate to the increase in the complexity o the stochastic model (2) closed-orm inerence solutions are typically not available. To address these problems we have presented and analyzed (1) compact parameterizations o semi-markov processes based on parametric time models and actorizations and (2) MC inerence algorithms. We showed that time and simulations over time or MC state predictions do not prevent eicient worst-case inerences and a simple modiication o the MC algorithm successully resolves the concern o exponentially long simulation trajectories or any practical time model. We gave the detailed analysis o the modiied MC solution and discussed its extensions to more complex inerences and models. We expect that the modiication and its analysis will transer in a rather straightorward way also to MC methods or semi-markov decision processes. 11

12 Reerences [1] Dimitri P. Bertsekas. Dynamic programming and optimal control. Athena Scientiic [2] Justin A. Boyan and Michael L. Littman. Exact solutions to time-dependent MDPs. In NIPS pages [3] John Bresina Richard Dearden Nicolas Meuleau Sailesh Ramakrishnan David Smith and Ric h Washington. Planning under continuous time and resource uncertainty. In Proceedings o the Eighteen Conerence on ncertainty in Artiicial Intelligence pages [4] H. Cherno. A measure o the asymptotic eiciency o tests o a hypothesis based on the sum o observations. Annals o Mathmematical Statistics 23: [5] Robert H. Crites and Andrew G. Barto. Improving elevator perormance using reinorcement learning. In David S. Touretzky Michael C. Mozer and Michael E. Hasselmo editors Advances in Neural Inormation Processing Systems volume 8 pages The MIT Press [6] Thomas Dean and K. Kanazawa. A model or reasoning about persistence and causation. Computational Intelligence 5: [7] Mohammad Ghavamzadeh and Sridhar Mahadevan. Continuous-time hierarchical reinorcement learning. In Proceedings o the 18th International Conerence on Machine Learning [8] W. Hoeding. Probability inequalities or sums o bounded random variables. Journal o the American Statistical Association 58: [9] Ronald A. Howard. Semi-markovian decision processes. In Proceedings o International Statistical Inst. Ottawa Canada [10] W.S. Jewell. Markov renewal programming: I. ormulations inite return models ii. ininite return models example. Operations Research 11: [11] e Kjaerul. A computational scheme or reasoning in dynamic probabilistic networks. In Proceedings o the Eighth Conerence on ncertainty in Artiicial Intelligence pages [12] T.-Y. Leong. An integrated approach to dynamic decision making under uncertainty. PhD thesis Massachusetts Institute o Technology [13] Sridhar Mahadevan Nicholas Marchalleck Tapas K. Das and Abhijit Gosavi. Selimproving actory simulation using continuous-time average-reward reinorcement learning. In Proc. 14th International Conerence on Machine Learning pages Morgan Kaumann [14] David McAllester and Luis Ortiz. Concentration inequalities or the missing mass and or histogram rule error. In Neural Inormation Processing Systems

13 [15] Colin McDiarmid. Concentration. In Probabilistic methods or algorithmic discrete mathematics pages Springer [16] ri Nodelman Christian Shelton and Daphne Koller. Continuous time bayesian networks. In Proceedings o the Eighteen Conerence on ncertainty in Artiicial Intelligence pages [17] Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley New York

Analysis Scheme in the Ensemble Kalman Filter

JUNE 1998 BURGERS ET AL. 1719 Analysis Scheme in the Ensemble Kalman Filter GERRIT BURGERS Royal Netherlands Meteorological Institute, De Bilt, the Netherlands PETER JAN VAN LEEUWEN Institute or Marine