Stability of Model Predictive Control using Markov Chain Monte Carlo Optimisation

Transcription

1 Stability of Model Predictive Control sing Markov Chain Monte Carlo Optimisation Elilini Siva, Pal Golart, Jan Maciejowski and Nikolas Kantas Abstract We apply stochastic Lyapnov theory to perform stability analysis of MPC controllers for nonlinear deterministic systems where the nderlying optimisation algorithm is based on Markov Chain Monte Carlo (MCMC) or other stochastic methods. We provide a set of assmptions and conditions reqired for employing the approximate vale fnction obtained as a stochastic Lyapnov fnction, thereby providing almost sre closed loop stability. We demonstrate convergence of the system state to a target set on an example, in which simlated annealing with finite time stopping is sed to control a nonlinear system with non-convex constraints. I. INTRODUCTION Model predictive control (MPC) solves an open loop optimal control problem at each time step [7]. The nderlying optimisation problems that mst be solved in MPC schemes for nonlinear systems with arbitrary constraints and objective fnctions defined on continos spaces are in general nonconvex. The existence of mltimodalities in the objective presents frther challenges in solving for global optima. In sch instances, options for the optimisation method are restricted to stochastic optimisation techniqes. This paper is concerned with the application of Markov Chain Monte Carlo (MCMC) methods to solving for MPC control laws for deterministic nonlinear systems. Simlated annealing can admit an MCMC formlation for expected tility optimisation [9], [1]. This has previosly been employed by [6] for nonlinear MPC with arbitrary distrbances, with the incorporation of probabilistic constraint satisfaction. Problems of feasibility and stability were not addressed there. This paper addresses the closed-loop stability of an MPC control law implemented via the repeated application of an MCMC optimiser, specifically simlated annealing, in a receding horizon fashion. Probabilistic finite time garantees for general simlated annealing methods have been obtained for instance by [5], in which the desired precision of the approximate optimm [14] obtained is controlled by the choice of stopping temperatre. The MPC control laws obtained with finite time stopping simlated annealing are sboptimal. Conditions nder which sboptimal control of deterministic systems is stabilising are examined by [8] and [1]. A modified version of the standard Lyapnov stability theorem allowing for nonniqeness and discontinity in the control law is presented in [1]. One condition for convergence is This work is spported by the EPSRC grant EP/C014006/1 and EC project TREN/07/FP6AE/S / IFLY. E. Siva, J. Maciejowski and N. Kantas are at Cambridge University Engineering Department es403,jmm,nk34@eng.cam.ac.k and P. Golart is at Imperial College, London p.golart@imperial.ac.k that the cost fnction redces at each time step. This is achieved by enforcing a terminal set constraint, in which a locally asymptotically stabilizing control law exists. Key conditions on the terminal set and controller are presented in [8] to ensre that the vale fnction is Lyapnov. The conditions ensre that it is sfficient to find a feasible soltion at each timestep to garantee stability. The se of stochastic optimisation for obtaining MPC control laws introdces probabilistic ncertainty into the system, even when the system dynamics are deterministic. A need for establishing stability in this context therefore exists. It is not possible to meet the stability reqirement in [8] and ensre redction of the approximate vale fnction at each time step. The standard notion of Lyapnov stability is inapplicable, so we consider stochastic Lyapnov fnctions, which are processes having spermartingale properties [3] in the neighborhood of the stable point. Or contribtion is the application of stochastic stability theory [13], [3] to obtain an extension of the existing stability reslt obtained by [8] when approximate optima are obtained with MCMC optimisation. We obtain conditions on the approximate vale fnction to be a stochastic Lyapnov fnction. Specifically, we show that this is achieved by making the correct choice of stopping temperatre for the simlated annealing optimisation at each time step. An MPC strategy with MCMC optimisation is otlined and its stabilising properties discssed. Some of the notation sed in this paper is otlined here. The state and control inpts are denoted x and v respectively. We se x k to depict the actal measred state at time k. At the time instant k, the prediction of the state i steps in the ftre is denoted x k+i k. The joint process (x k, v k ) is represented as φ k. We denote the expectation of a real valed fnction of a random variable Z π as E π [h(z)] = h(z)π(z)dz. A fnction α( ), defined on nonnegative reals, is a K-fnction if it is continos and strictly increasing with α(0) = 0. The paper is organised as follows. In Section II, the problem formlation and backgrond stochastic stability theory is presented. The MPC formlation and proposed MCMC optimisation procedre are otlined in section III. In section IV, the stabilising properties of the proposed MPC strategy are proved. An illstrative example is presented in Section V and conclding remarks are made in Section VI. II. PROBLEM FORMULATION AND BACKGROUND We consider controlling a discrete time deterministic system whose dynamics can be described as x k+1 = f(x k, v k ), (1) where x R n and v R m and f : R n R m R n is a nonlinear mapping of the states and inpts to the sccessor state.

2 The inpts and state mst satisfy the following constraints: v k U R m () x k X R n. (3) where U is compact and X is closed. The control objective is to steer the state to the origin. Given the difficlty in obtaining a closed form soltion for stabilising controllers for nonlinear systems with nonconvex constraint sets, we consider randomised sampling of control inpts. We seek to obtain control laws which give rise to almost sre asymptotic stability [3] of the system in (1) as defined below: Definition.1: Asymptotic Stability 1) The origin is stable with probability one if and only if, for any ρ > 0, ɛ > 0, there is a δ > 0 sch that, for all x 0 δ. P r(sp x k ɛ) ρ k ) The origin is asymptotically stable with probability 1 if and only if it is stable w.p.1., and x n 0 w.p.1 for all x 0 in some neighborhood of the origin. At time k, a measrement of the crrent state x k is made, and control inpts v k are sampled according to some distribtion π k (v k x k ) dependent on x k. The control law v k is applied, giving rise to a sccessor state generated by (1). The joint process φ k evolves according to Φ k+1 p k+1 ( Φ k ), with the conditional distribtion p k+1 given by 1 p k+1 (φ k+1 φ k ) = δ(x k+1 f(x k, v k ))π k+1 (v k+1 x k+1 ). (4) The origin is asymptotically stable according to Definition.1 if the conditions of the following theorem are satisfied: Theorem.1: Assme the following conditions hold: (i.) There is a fnction V : R n R m R continos at the origin with V (0, 0) = 0, and a K-fnction α, sch that for all x R n, v R m, V (x, v) α( x ) (5) (ii.) There is a compact set Q R n that contains the origin, and given the initial condition x 0 Q, realisations of the controlled system trajectory (x k, v k ) satisfy x k Q with probability 1. (iii.) There exists a K-fnction η( ) sch that in Q, along the controlled system trajectory E pk+1 [V (Φ k+1 ) Φ k = (x, v)] V (x, v) η( x, v ) (6) (iv.) For any λ > 0, there exists a δ > 0 sch that E π0 [V (x 0, v 0 )] < λ for all x 0 δ. 1 Althogh the state transition from x k to x k+1 is deterministic, we shall treat x k+1 as an extreme case of a random variable drawn from a delta fnction distribtion as this is more convenient for or analysis. Then, the origin is asymptotically stabilising for all x 0 Q with probability 1. Proof: Stability of the origin follows from Theorem 1, [3]. Convergence: In view of Theorem 1, [4, Ch.6, p.71], we have η( x k, v k ) 0 with probability 1. Since η( ) is a class K-fnction, this implies x, v 0 with probability 1. In the next sections we define a cost fnction V and inpt sampling distribtions π k, sing a combination of MPC and MCMC to ensre V is a stochastic Lyapnov fnction, and the conditions of Theorem.1 are satisfied. A. MPC III. MPC AND MCMC OPTIMISATION We consider controlling the system in (1) via the Model Predictive Control (MPC) approach. The MPC formalism reqires an explicit model of the process. The crrent control action is obtained via optimisation of an open-loop objective over a finite prediction horizon, which is solved in a receding horizon manner. The soltion is performed online, nlike other control techniqes where precompted control laws are applied. The prediction horizon is of length N. The state prediction x k+i k is generated by the model in (7), where v k+i k is the predicted open loop inpt: x k+i+1 k := f(x k+i k, v k+i k ), (7) for i = 0,..., N 1 x k k := x k. For the state x k at time k, the cost is defined as: V (x k, k ) := k+n 1 j=k L(x j k, v j k ) + F (x k+n k ) (8) with stage cost L : R n R m R, L(x, ) > l( x, ), (x, ) (0, 0), where l( ) is a K-fnction and L(0, 0) = l( 0, 0 ) = 0. The terminal cost is F : R n R +..A terminal constraint is enforced: x k+n k X f X (9) We reqire f( ), L(, ) and F ( ) to be continos, U compact and X and X f to be closed to ensre that a minimm for V (x k, ) exists [8]. The finite inpt seqence of predicted ftre controls is denoted: k := {v k k, v k+1 k,..., v k+n 1 k }. The feasible set of inpt seqences satisfying the control, state and terminal constraints is defined as U(x k ), and in general depends on the initial state x k. The optimisation problem at each time instant k is to minimise V (x k, ), yielding the vale fnction V (x k ) := min V (x k, k ) (10) k U(x k )

3 by obtaining the optimal seqence of inpts k from the feasible set U(x k ). k = argmin V (x k, k ) =: {vk k,..., v k+n 1 k }. The MPC control law applied to the state x k at time k is the first inpt v k k of the actal seqence k obtained by the optimisation, v k = v k k, giving the closed loop system B. MCMC Optimisation x k+1 = f(x k, v k ). (11) Here we describe the randomised sampling techniqe for minimising the objective fnction V (x k, ) at each time k. A key property of the approximate optimm obtained this way is then derived which ensres the cost is a stochastic Lyapnov fnction. The sbscript k denoting system time is dropped from the notation in this section for clarity. A sitable transformation of variables is applied to convert the original minimisation problem to that of maximisation. An example wold be g := V + sp,x V (x, ). Note that g is a fnction of both the state and inpt seqence x and, bt the optimisation variable of interest is. At each time, we wold like to sample control inpt seqences according to a distribtion whose modes coincide with the maximisers of g, and which enables satisfaction of the latter two reqirements of Theorem.1. We propose sampling inpts according to a distribtion π proportional to the objective fnction raised to a positive exponent J: π( x) g J(x, ). The higher the vale of J, the more concentrated abot the optimm vales of g(x, ) the distribtion π becomes. Sampling from this distribtion is enabled by constrcting a sitable Markov chain in, with stationary distribtion π. This is achieved by the following simlated annealing algorithm. We begin with a proposal distribtion q, which can be chosen freely by the ser bt is reqired to be spported on the whole space. We also reqire a nondecreasing seqence of inverse temperatres {J n } with J n J, known as a cooling schedle, with n denoting the chain iteration index. Algorithm 3.1: Simlated annealing with finite time stopping Initialise: At n = 0 1) Select q 0 ( ). Initialise J = J 0. ) Extract a sample U 0 q 0. Set U = U 0. Repeat: 1) Draw a new inpt sample seqence Ũ q( U); ) Record Ũ as a proposed new state of the Markov chain; 3) Evalate the acceptance probability [ ] ρ J = min 1, gj (x, Ũ)q(U Ũ) g J (x, U)q(Ũ U). Accept the proposal Ũ with probability ρ J and set U = Ũ. Otherwise, with probability 1 ρ J, leave U nchanged; 4) If J < J, set J to J n+1. Else keep J fixed at J; Repeat 1 4 ntil convergence of the chain to the stationary distribtion π is achieved. Discard the initial samples obtained with J < J and select the final sample û of those samples remaining obtained with J = J. Constraints are handled by rejecting infeasible points. Other options inclde appending the cost with a parameter representing a reward for constraint satisfaction. Inpts û chosen according to Algorithm 3.1 are distribted approximately according to the stationary distribtion of the Markov chain in with the inverse temperatre [10] capped at J, given by: û π = g J(x, ) g J(x, )d We consider now a featre of the approximate optimm obtained by implementation of Algorithm 3.1 which is necessary for implementing the MPC scheme presented later. The closeness of the expected vale of the approximate optimm to the global optimm of the cost fnction is determined by the choice of capping parameter J. The expected vale of the approximate optimm is: E π [g(x, U)] = g(x, )π()d g J+1 (x, )d = g J(x, )d, since π( x) = g J(x, ) g J(x, )d. Writing g J(x, )d as g J(x, ) 1, since g(x, ) 0, the expected difference between the global and approximate optimm is given by E π [sp g(x, ) g(x, )] = sp g(x, ) g J+1 (x, ) 1 g J(x, (1) ) 1 The following reslt shows that this expected difference can be made arbitrarily small. Lemma 3.1: For any π-measrable fnction g, assme that g : R m R is sch that g() 0 for all and ess sp g() = sp g() =: g. For any ɛ > 0, there exists a finite M sch that for all n > M, Proof: g gn+1 1 g n 1 ɛ. g n 1 = g n+1 g n 1 1

4 From Holder s ineqality we have g n+1 g n 1 1 g n+1 g n 1 g n+1 g n 1 1 g n+1 g n 1 = g n+1 1 g n 1 1 g n 1 g n+1 1 g n 1 1 g n+1 1 g n 1 gn 1 g n 1 1 (13) From [11], Chapter 6, p.71, we have The seqence { g n+1 1 lim n g n = sp g = g 1 } sp g() gn+1 () 1 g n () 1 n 0 is ths monotonically non-increasing for all positive n, and converges to 0. It follows that for any ɛ > 0, there exists an M sch that for all n > M, g gn+1 1 g n 1 ɛ. If we make the transformation g := V sp V, where V sp := sp x, V, we have From Lemma 3.1, we have Combining with (14) gives g := sp g = V sp inf V (14) E π [g g] ɛ E π [V sp inf V (V sp V (x, U))] ɛ E π [V (x, U) V (x, )] ɛ (15) It follows that the expected difference between the global optimm of V (x k, ) and the expected cost can be made arbitrarily small by sitable choice of stopping inverse temperatre at each time step. This featre is reqired for implementation of the proposed MPC strategy detailed next, and is also necessary for the stability analysis in Section IV. As the stopping temperatre need not be the same at each time step k, we will denote it J(k) henceforth. C. MCMC with MPC We now detail the overall receding horizon MPC scheme with the MCMC optimisation. The control inpt seqence k (x k ) obtained by the simlated annealing optimisation at each time step k with state x k mst satisfy the state and inpt constraints. Additionally, we place a reqirement on the soltion qality at each time step in terms of the expected difference between the global optimm and approximate soltion. The reqirement is that at each time step, the expected difference between the global optimm and approximate soltion is no greater than some fraction of the first stage cost at the previos timestep. This fraction is determined by the K-fnction l( ) that bonds this stage cost from below, as defined in Section III. We present now the algorithm: Algorithm 3.: MPC with Simlated Annealing 1) At time k = 0, state x 0, find a control seqence 0 = {v 0 0, v 1 0,..., v N 1 0 } which satisfies (), (3), (7) and (9). Set v 0 = v 0 0. ) At system time k, measre state x k, choose an inverse stopping temperatre J(k) large enogh which satisfies E πk [V (x k, U k ) V (x k, k)] L(x k 1, v k 1 k 1 ) l( x k 1, v k 1 k 1 ) (16) and implement Algorithm 3.1 to obtain a control seqence k = {v k k, v k+1 k,..., v k+n 1 k } = û that satisfies (), (3), (7) and (9). Set v k = v k k. From Lemma 3.1, setting the right hand side of (16) to be the ɛ term in (15), it follows that a vale for J(k) exists at each timestep which satisfies the stiplated reqirement on the soltion qality. The bonds on soltion qality obtained for simlated annealing with finite-time stopping in [5] provide a sefl starting point in determining J(k). The stabilising properties of the reslting sboptimal MPC law are explored in Section IV. Remark 3.1: Care mst be taken to distingish between the state of Markov chain generated by Algorithm 3.1 and the controlled system chain state, which evolves within a different time frame; at every time instant k, with respect to the system time reference, a Markov chain is constrcted whose state evolves with iteration n, (with the system time still fixed at k) according to an inhomogeneos Markov Chain transition kernel [10] generated by Algorithm 3.1. Note we make the key assmption in the closed loop stability analysis that the extractions from the chain are generated from the stationary distribtion. Criteria for diagnosing convergence to stationarity of the chain are presented in [10]. In [], [9] it is shown that nder certain conditions, inclding employing a logarithmic cooling schedle for incrementing J, in the limit with infinite J, the distribtion of the Markov chain converges to a niform distribtion over the set of minimisers of V (x, ). In practice only a finite vale of J is reqired, and in the next section it is shown how closed loop stability is achieved with J chosen to meet the reqirements of Algorithm 3.. IV. STABILITY IN MPC WITH SIMULATED ANNEALING We seek to establish almost sre stability [3] of the closed loop system in (11). Before proceeding with establishing the stabilising properties of the control law that arise by implementation of Algorithm 3., we first reqire some ancillary reslts and assmptions. Appropriate conditions on the terminal set X f and terminal cost F ( ) need to be determined. We detail them next: Assmptions 4.1: (i) The terminal set X f X, X f is closed, 0 X f.

5 (ii) There exists a controller f (x) U, x X f. (iii) f(x, f (x)) X f, x X f (X f is positively invariant nder f ( )). (iv) F (f(x, f (x))) + L(x, f (x)) F (x) 0, x k X f. (v) There exists a K-fnction σ( ), sch that {x 0, 0} satisfies 0 σ( x 0 ) (17) The first for conditions in Assmption 4.1 have been identified by [8] as key ingredients for closed loop stability when employing their strategy. As in [8], we reqire the existence of a terminal controller f in the terminal set X f satisfying the properties in Assmption 4.1. The control law that is actally applied when the state is in the terminal set is obtained from the optimisation. The third condition in Theorem.1 for almost sre stability concerns the initial expected approximate vale fnction and initial state. The proof of Theorem 4.1, showing that this condition is satisfied on carrying ot Algorithm 3., reqires the following reslt abot the initial state and initial expectation of the approximate soltion: Lemma 4.1: For any λ > 0, given that V is continos at the origin and V (0, 0) = 0, there exists a J and δ > 0 sch that E π0 [V (x 0, U 0 )] < λ V (x 0, 0) (18) for all x 0 δ. Proof: For all r 0, n 1, define B r := {x R n : x r}. As V is continos at the origin, with V (0, 0) = 0, there exists a constant r 1 > 0 and a K-fnction β( ) sch that V (x, ) β( (x, ) ) x B r1. For any λ > 0, there exists a δ > 0 sch that δ < r 1 and σ(δ) r 1. Let x 0 < δ. Now 0 < σ(δ) and V (x 0, 0) β( x 0, 0 ) β(δ + σ(δ)). As δ 0, σ(δ) 0, and therefore δ + σ(δ) 0 and β(δ + σ(δ)) 0. Given λ > 0, a δ > 0 satisfying λ V (x 0, 0) > 0 exists. From eqation (1) and Lemma 3.1 it follows that there exists a J sch that (18) holds. We are now ready to establish the convergence properties of the control law that arise by implementation of Algorithm 3.. Theorem 4.1: Let Q represent the set of states for which there exists a control seqence that satisfies (), (3), (7) and (9) and (16). Given satisfaction of Assmptions 4.1, the MPC law arising from implementation of Algorithm 3. is asymptotically stabilising with region of attraction Q. Proof: The algorithm ensres that along trajectories of the controlled system we have x k Q, satisfying condition (ii) of Theorem.1 by assmption, and E πk+1 [V (x k+1, U k+1 ) V (x k+1, k+1)] L(x k, v k k ) l( x k, v k k ). (19) The following analysis shows the eqivalence of this relation and the stochastic Lyapnov condition (iii) in Theorem.1. The model predictive controller v k = v k k steers x k to x k+1. V (x k+1, {v k+1 k,..., f (x k+n k )}) = V (x k, {v k k, v k+1 k,... v k+n 1 k }) L(x k, v k k ) F (x k+n k ) + L(x k+n k, f (x k+n k )) + F (f(x k+n k, f (x k+n k ))) (0) The sm of the last three terms is less than or eqal to zero, from Assmption 4.1(iv), giving L(x k, v k k ) V (x k, k ) V (x k+1, {v k+1 k,..., f (x k+n k )}) (1) Sbstittion of (1) into (19) yields E πk+1 [V (x k+1, U k+1 ) V (x k+1, k+1)] V (x k, {v k k, v k+1 k,... v k+n 1 k }) V (x k+1, {v k+1 k,..., f (x k+n k )}) l( x k, v k k ) () The cost associated with {v k+1 k,..., f (x k+n k )} and state x k+1 is an pper bond for the cost yielded on application of the optimal seqence of inpts k+1. V (x k+1, {v k+1 k,..., f (x k+n k )}) V (x k+1, k+1) It follows that E πk+1 [V (x k+1, U k+1 ) V (x k+1, k+1)] V (x k, k ) V (x k+1, k+1) l( x k, v k k ) (3) Since V (x k+1, k+1 ) is not a random variable, E πk+1 [V (x k+1, U k+1 )] V (x k, k ) l( x k, v k k ). Noting also that E πk+1 [V (x k+1, U k+1 )] = E pk+1 [V (X k+1, U k+1 ) X k = x k, U k = k ] (4) with p k+1 defined in (4), we see that condition (iii) of Theorem.1 is satisfied. Almost sre convergence of the state to the origin follows. From Lemma 4.1 it follows that for any λ > 0 there exists a J and δ > 0 sch that E π0 [V (x 0, U 0 )] < λ V (x 0, 0) for all x 0 δ. Condition (iv) of Theorem.1 is therefore satisfied. Almost sre asymptotic stability follows from Theorem.1. V. SIMULATION EXAMPLE We now present an example to illstrate application of simlated annealing to nonlinear MPC. It is shown that an inverse stopping temperatre can be determined empirically by simlation. The objective is to drive a constant speed particle in D to a target set centred at O T whilst avoiding an obstacle of radis R centred at O X = [O x O y ]. The control inpt v k is the change in particle heading, θ k θ k 1. The

6 state at time k is x k = [x k y k θ k ]. The system is nonlinear in state and inpt and described by: x k+1 = x k + V cos(θ k + k ) V sin(θ k + k ) v k where V = 5. The stage cost is given by: L(x, ) = [ x [O T 0] ] (5) The horizon length N = 5, and the constraint sets for the controls and states are U = { R : 0 π} and X = {x R n : (x O x ) + (y O y ) > R }. The initial state x 0 is [10 15] and the target set is centred at O T = [50 50]. The problem is solved in a receding horizon manner. We implement Algorithm 3. withot identifying a distinct J(k) at each time k, bt instead keeping J fixed for the entire seqence of optimisations. Control inpts yielding infeasible soltions are rejected. For the cooling schedle, we increment J n by 1 every 5 iterations ntil J = J. For the proposal distribtion, we se a Gassian random walk centred on the crrent vale of the chain. Figre 1 depicts sample trajectories obtained with variable obstacle widths R = {5, 10, 15} and fixed horizon length N = 5. It can Fig. 1. Representative particle trajectories obtained with obstacles of radii 5, 10 and 15, and inverse stopping temperatre J = 40. be seen that in each instance, the target set is reached. The total nmber of iterations per stage reqired for the simlations, n, were 195, 3897 and 5755 respectively, and took 3.3, 5.7 and 8.1 seconds on a networked.4 GHz Dal Processor. A brn-in period of 500 iterations was sed, and convergence of the chain was jdged to be achieved after a certain nmber of states (1000) were accepted. As the obstacle radis is increased, a greater extent of resampling is reqired as the infeasible region increases, reslting in an increased nmber of iterations reqired. As the vale of J was fixed for the entirety of each simlation, and not tned for each optimisation at k, there is scope for a redction in the nmber of iterations reqired. VI. CONCLUDING REMARKS In this paper we consider application of simlated annealing to the MPC optimisation problems that arise for nonlinear, deterministic discrete time systems. We present a sboptimal MPC scheme with almost sre stabilising properties. This is a sefl extension to the deterministic stability reslt in [8], [1], for cases where near optimality is a soltion reqirement, and the optimisation method is stochastic. As in [8], we are reqired to enforce a terminal set constraint with similar conditions on the terminal set and terminal controller which is assmed to exist. As a monotonic cost redction constraint cannot be imposed when a randomised optimiser is sed, a condition is placed on the soltion qality in terms of its expected vale. Althogh we do not explictly state how to determine a stopping temperatre at each time step to achieve this, we prove that sch a vale exists, and it is shown by simlation that it can be empirically determined for or example. We are crrently extending this work to establish almost sre stability of MPC schemes for stochastic nonlinear systems with an expected objective minimisation formlation. REFERENCES [1] A. Docet, S. J. Godsill, and C. Robert, Marginal maximm a posteriori estimation sing Markov Chain Monte Carlo, Statisitcs and Compting, vol. 1, pp , 00. [] Geman and Geman, Stochastic relaxation, gibbs distribtion and the bayesian restoration of images, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 6, pp , [3] H. Kshner, On the stability of stochastic dynamical systems, in Proceedings of the National Academy of Sciences, [4], Introdction to Stochastic Control. Holt, Rinehart and Winston, [5] A. Lecchini-Visintini, J. Lygeros, and J. Maciejowki, Simlated annealing: Rigoros finite-time garantees for optimisation on continos domains, in Advances in Neral Information Processing 0, 007. [6] J. M. Maciejowki, A. Lecchini-Visintini, and J. Lygeros, NMPC for complex stochastic systems sing Markov Chain Monte Carlo, in Assessment and Ftre Directions of Nonlinear Model Predictive Control. Springer Berlin / Heidelberg, 007. [7] J. M. Maciejowski, Predictive Control with Constraints. Prentice Hall, 001. [8] D. Mayne, J. Rawlings, C. Rao, and P. Scokaert, Constrained model predictive control:stability and optimality, Atomatica, vol. 36, pp , 000. [9] P. Mller, B. Sanso, and M. D. Iorio, Optimal Bayesian design by inhomogeneos Markov Chain simlation, Jornal of the American Statistical Association, vol. 99, pp , 004. [10] C. Robert and G. Casella, Monte Carlo Statistical Methods. Springer, 004. [11] W. Rdin, Real and Complex Analysis. McGraw-Hill, [1] P. Scokaert, D. Mayne, and J. Rawlings, Sboptimal Model Predictive Control (Feasibility Implies Stability), IEEE Transactions on Atomatic Control, vol. 44, pp , [13] S.P.Meyn, Ergodic theorems for discrete time stochastic systems sing a stochastic Lyapnov fnction, SIAM Jornal of Control and Optimisation, vol. 7, pp , [14] M. Vidyasagar, Randomised algorithms for robst controller synthesis sing statistical learning theory, Atomatica, vol. 37, pp , 001.