Dynamic Consistency for Stochastic Optimal Control Problems

Size: px
Start display at page:

Download "Dynamic Consistency for Stochastic Optimal Control Problems"

Transcription

1 Dynamic Consistency for Stochastic Optimal Control Problems Cadarache Summer School CEA/EDF/INRIA 2012 Pierre Carpentier Jean-Philippe Chancelier Michel De Lara SOWG June 2012

2 Lecture outline Introduction Distributed formulation SOC with constraints Reduction to finite-dimensional problem

3 Informal definition of dynamically consistent problems Informal definition For a sequence of dynamic optimization problems, we aim at discussing a notion of consistency over time. At the very first time step t 0, formulate an optimization problem that yields optimal decision rules for all the forthcoming time steps t 0,t 1,...,T; At the next time step t 1, formulate a new optimization problem starting at time t 1 that yields a new sequence of optimal decision rules. This process can be continued until the final time T is reached. A family of optimization problems is said to be dynamically consistent if the optimal strategies obtained when solving the original problem remain optimal for all subsequent problems.

4 Informal definition of dynamically consistent problems Decision at time t 0 State u 0,0 u 1 0,k u 2 0,k u 3 0,k depends on decisions computed at later times for possible future states. t 0 t k T Time

5 Informal definition of dynamically consistent problems Time consistent decision at time t k State u 0,0 t 0 u 0,k t k u k,k T Time Decision u k,k computed using a problem formulated at time t k knowing where I am at time t k or decision u 0,k. computed at time t 0 and to be applied at time t k for the same position should be the same.

6 Informal definition of dynamically consistent problems An example: deterministic shortest path If A B C D E is optimal then C D E is optimal. Arriving at C, If I behave rationally I should not deviate from the originally chosen route starting at A. t=1 B t=2 C E A t=0 D

7 Dynamic Programming and dynamically consistent problems Dynamic Programming and dynamically consistent problems Dynamic Programing gives dynamically consistent controls It gives a family of problems indexed by time: the solution of problem at time t uses the solution of problem at time t +1: V k x = min E [ L k x,u,wk+1 +Vk+1 ftk x,u,w u k+1 ] Choose control at time t k optimizing the sum of a current cost and of a running cost. The running cost has an interpretation as the optimal cost function of a problem at time t k+1.

8 Dynamic Programming and dynamically consistent problems Dynamic Programming and dynamically consistent problems 2 The problem at time t k+1 is the following: V k+1 x = min X,U T 1 E L t Xt,U t,w t+1 +K XT Xtk+1 = x, t=t k+1 s.t. X t+1 = f t Xt,U t,w t+1, t = tk+1,...,t 1, U t X t t = t k+1,...,t 1.

9 Deterministic case Deterministic case State u 0,k u 0,0 u k,k Exact state match at time t k t 0 t k T Time

10 Deterministic case Deterministic case State u 0,k u 0,0 u k,k Independence of the initial condition. t 0 t k T Time

11 Deterministic case Deterministic case State u 0,0 u 0,k u k,k Taking care of all possible initial states at time t k through state feedback. t 0 t k T Time

12 Deterministic case A first example in the deterministic case 1 min u t0,...,u T 1,x t0,...,x T T 1 t=t 0 L t xt,u t +K xt, Dt0 subject to x t+1 = f t xt,u t, xt0 given. Suppose a solution to this problem exists: u t 0,...,u T 1 : controls indexed by time t, the state variable. x t0,xt 1,...,x T : optimal path for No need for more information since the model is deterministic. these controls depend on the hidden parameter x t0, these controls are usually not optimal for x t 0 x t0.

13 Deterministic case A first example in the deterministic case 2 Consider the natural subsequent problems for every t i t 0 : min u ti,...,u T 1,x ti,...,x T T 1 t=t i L t xt,u t +K xt, Dti subject to x t+1 = f t xt,u t, xti given. One makes the two following observations. 1. Independence of the initial condition. In the very particular case where the solution to Problem D ti does not depend on x ti, Problems {D ti } ti are dynamically consistent. 2. True deterministic world. Suppose that the initial condition for Problem D ti is given by xt i = f ti x ti 1,ut i 1 exact model, then Problems {D ti } ti are dynamically consistent. Otherwise, adding disturbances to the problem brings inconsistency.

14 Deterministic case A first example in the deterministic case 3 Solve now Problem D t0 using Dynamic Programming DP: φ t 0,...,φ T 1 : controls depending on both t and x. The following result is a direct application of the DP principle. 3. Right amount of information. Suppose that one is looking for strategies as feedback functions φ t depending on state x. Then Problems {D ti } ti are dynamically consistent. As a first conclusion, time consistency is recovered provided we let the decision rules depend upon a sufficiently rich information.

15 Deterministic case Independence of the initial condition for every t = t 0,...,T 1, functions l t : U t R and g t : U t R, and assume that x t is scalar. Let K be a scalar constant and consider the following deterministic optimal control problem: min x,u T 1 t=t 0 l t u t x t +Kx T, s.t. x t0 given, x t+1 = g t u t x t, t = t 0,...,T 1.

16 Deterministic case Independence of the initial condition 2 Variables x t can be recursively replaced using dynamics g t leading to: min u T 1 t=t 0 l t u t g t 1 u t 1...g t0 u t0 x t0 +Kg T 1 u T 1...g t0 u t0 x t0. The optimal cost is linear with respect to x t0. Suppose that x t0 only takes positive values. Then x t0 has no influence on the minimizer. Remains true at subsequent time steps provided that dynamics are such that x t remains positive for every time step. The dynamic consistency property holds true. This example which may look very special but we will see later on that it is analogous to familiar SOC problems.

17 Deterministic case The guiding principle of the lecture To obtain Dynamical Consistency, let the decision rules depend upon a sufficiently rich information set.

18 Introduction Informal definition of dynamically consistent problems Dynamic Programming and dynamically consistent problems Deterministic case Distributed formulation A classical SOC Problem Equivalent distributed formulation SOC with constraints Constraints typology A toy example with finite probabilities Back to time consistency Reduction to finite-dimensional problem An equivalent problem with added state and control Dynamic programing

19 A classical SOC Problem A classical SOC Problem Control variables U = U t t=t0,...,t 1. Noise variables W = W t t=t1,...,t. Markovian setting: noise variables X t0,w t1,...,w T are independent. The problem S t0 starting at t 0 writes: [ T 1 min X,U E s.t. X t0 given, ] L t Xt,U t,w t+1 +K XT, t=t 0 X t+1 = f t Xt,U t,w t+1, U t X t0,w t1,...,w t, t = t 0,...,T 1, We know that there is no loss of optimality in looking for the optimal strategy as a state feedback control U t = φ t X t. 1

20 A classical SOC Problem Stochastic optimal control: the classical case 2 Can be solved using Dynamic Programming: V T x = Kx, Vtx = min E L t x,u,w u U t+1 +Vt+1 ft x,u,w t+1. It is clear while inspecting the DP equation that optimal strategies {φ t} t t0 remain optimal for the subsequent optimization problems: min U ti,...,u T 1,X ti,...,x T T 1 E L t X t,u t,w t+1 +KX T, t=t i subject to: X ti given, S ti X t+1 = f t X t,u t,w t+1, U t X ti,w ti +1,...,W t.

21 Equivalent distributed formulation An equivalent distributed formulation Ξ t a linear space of R-valued measurable functions on X t. Υ t space of signed measures on X t. We consider a dual pairing between Ξ t and Υ t by considering the bilinear form: ξ,µ = ξxdµx for ξ Ξ t and µ Υ t. X t When X t is a random variable taking values in X t and distributed according to µ t. Then ξ t,µ t = E ξ t X t.

22 Equivalent distributed formulation Fokker-Planck equation: state probability law dynamics For a given feedback laws φ t : X t U t we define T φt t : Ξ t+1 Ξ t : T φ t t ξ t+1 x def = E ξ t+1 X t+1 X t = x x X t. Using state dynamics and Markovian setting we obtain: T φ t t ξ t+1 x = E ξ t+1 ft x,φ t x,w t+1 x X t. Denoting by T φt t the adjoint operator of T φ t t T φ t ξ,µ = ξ, T φ µ t The state probability law driven by the chosen feedback follows: µ t+1 = T φt µt t t = t 0,...,T 1, µ t0 given.

23 Equivalent distributed formulation Revisiting the expected cost Next we introduce the operator Λ φt t : X t R: Λ φt t x def = E L t x,φt x,w t+1 meant to be the expected cost at time t for each possible state value when feedback function φ t is applied. Using Λ φt t we have: E L t Xt,φ t Xt,Wt+1 = E Λ φ t t X t = Λ φt t,µ t.

24 Equivalent distributed formulation Equivalent deterministic infinite-dimensional problem We obtain an equivalent deterministic infinite-dimensional optimal control problem D t0 : min Φ,µ T 1 t=t 0 s.t. µ t0 given, µ t+1 = Λ Φt t,µ t + K,µ T, T Φt t µt, t = t 0,...,T 1, Very similar to the Independence of the initial condition example

25 Equivalent distributed formulation Solving D t0 using Dynamic Programming V T µ = K,µ. A V T 1 µ = min Λ φ φ T 1,µ φ µ +V T T 1. Optimal feedback Γ T 1 : µ φ µ a priori depends on x and µ. V T 1 µ = min Λ φ φ T 1 +Aφ T 1 K,µ, = min Λ φ T 1 φ +Aφ T 1 K xµdx. X

26 Equivalent distributed formulation Solving D t0 using Dynamic Programming 2 Interchanging minimization and expectation operators leads to: optimal Γ T 1 does not depend on µ : Γ T 1 φ T 1, V T 1 again depends on µ in a multiplicative manner.

27 Introduction Informal definition of dynamically consistent problems Dynamic Programming and dynamically consistent problems Deterministic case Distributed formulation A classical SOC Problem Equivalent distributed formulation SOC with constraints Constraints typology A toy example with finite probabilities Back to time consistency Reduction to finite-dimensional problem An equivalent problem with added state and control Dynamic programing

28 Constraints typology Constraints in stochastic optimal control Different kinds of constraints in stochastic optimization: almost-sure constraint : gx T a P-a.s., chance constraint : P gx T a p, expectation constraint : E gx T a,... A chance constraint can be modelled as an expectation constraint: P gx T a = E 1 X adx T, with X ad = { x X, gx a }. Chance constraints bring both theoretical and numerical difficulties, especially convexity [Prékopa, 1995]. However the difficulty we are interested in is common to chance and expectation constraints. In the sequel, we concentrate on adding an expectation constraint.

29 Constraints typology Probability constraint on state trajectories Water level P { X t x t,t [0,T] } a Minimal water level Time

30 Constraints typology Transformed [ ] into a Probability constraints on final state : E g X T a Motivated by a joint probability constraint: P { γ t Xt bt, t = t 0,...,T } a. Introducing a new binary state variable Y t {0,1}: Y t0 = 1, Y t+1 = Y t 1 { γ t+1 X t+1 b t+1 }, E [ Y T ] = 1 P { γt Xt bt, t = t 0,...,T } a

31 Constraints typology Problem setting with expectation constraint Control variables U = U t t=t0,...,t 1. Noise variables W = W t t=t1,...,t. Markovian setting: noises variables X t0,w t1,...,w T are independent. The problem starting at t 0 writes: min X,U [ T 1 ] E L t Xt,U t,w t+1 +K XT, t=t 0 s.t. X t0 given, X t+1 = f t Xt,U t,w t+1, U t X t0,w t1,...,w t, t = t 0,...,T 1, E [ g X T ] a, a R

32 Constraints typology Problem at time t i min X,U [ T 1 ] E L t Xt,U t,w t+1 +K XT, t=t i s.t. X t+1 = f t Xt,U t,w t+1, Xti given, U t X ti,w ti+1,...,w t, t = t i,...,t 1, E [ g X T ] a. Not dynamically consistent with the usual state variable. With new appropriate state variable, one regains dynamical consistency. Optimal strategy at time t is a function of Φ t0,t : X U since information structure is not modified by the last constraint.

33 Constraints typology Equivalent deterministic infinite-dimensional problem We obtain an equivalent deterministic infinite-dimensional optimal control problem. min Φ,µ T 1 t=t 0 s.t. µ t0 given, µ t+1 = Λ Φt t,µ t + K,µ T +χ { g,µ a} µ T T Φt t g,µ T a. µt, t = t 0,...,T 1, The last constraint implies that the optimal feedback laws depend on the initial condition µ t0 and we do not have dynamical consistency.

34 A toy example with finite probabilities Introduction Informal definition of dynamically consistent problems Dynamic Programming and dynamically consistent problems Deterministic case Distributed formulation A classical SOC Problem Equivalent distributed formulation SOC with constraints Constraints typology A toy example with finite probabilities Back to time consistency Reduction to finite-dimensional problem An equivalent problem with added state and control Dynamic programing

35 A toy example with finite probabilities Simple case-study: a discrete controlled Markov chain The state space X = {1,2,3}: Discrete possible values of a reservoir level. x = 1 resp. x = 3 the lower resp. upper level of the reservoir. The control action u takes values in U = {0,1}, the value 1 corresponding to using some given water release to produce some electricity. The noise variable: stochastic inflow takes its values in W = { 1,0,1}. The discrete probability law of each noise variable W t being characterized by the weights {, 0, +}. Finally, the reservoir dynamics is: X t+1 = min x,maxx,x t U t +W t+1.

36 A toy example with finite probabilities Simple case-study: a discrete controlled Markov chain Let µ t be the discrete probability law associated with the state random variable X t, the initial probability law µ t0 being given. In such a discrete case, it is easy to compute the transition matrix P u giving the Markov chain transitions for each possible value u of the control, with the following interpretation: P u ij = P X t+1 = j X t = i,u t = u = P {min x,maxx,i u +Wt+1 = j }.

37 A toy example with finite probabilities Simple case-study: a discrete controlled Markov chain We obtain for the reservoir problem: P 0 = 0 +, P 1 = and we denote by P u i the i th row of matrix P u.

38 A toy example with finite probabilities Simple case-study: a discrete controlled Markov chain Let φ t : X U be a feedback law at time t. We denote by Φ t the set of admissible feedbacks at time t. In our discrete case, cardφ t = cardu cardx = 8 for all t. The transition matrix P φt associated with such a feedback is obtained by properly selecting rows of the transition matrices P u, namely: P φt1 1 P φt = P φt2 2. P φt3 3 Then the dynamics given by the Fokker-Planck equation writes: µ t+1 = P φt µt, and the state µ t involved in this dynamic equation is a three-dimensional column vector µ t = µ 1,t,µ 2,t,µ 3,t

39 A toy example with finite probabilities The cost at time t is supposed to be linear, equal to p t U t, p t being a negative deterministic price. No final cost K 0. The reservoir level at final time T must be equal to x with a probability level at least equal to π. The distributed formulation associated with the reservoir control problem is: min {φ t Φ t} T 1 t=t 0 T 1 t=t 0 p t φt,µ t, 6a subject to: µ t+1 = P φt µt, µ t0 given, 6b 1{x},µ T π, 6c with 1 {x} x = 1 if x = x, and 0 otherwise.

40 A toy example with finite probabilities The previous problem can be solved using DP: the state equation is a two-dimensional µ 1,t +µ 2,t +µ 3,t = 1 Suppose now that the number of reservoir discrete levels is n rather than 3, with n big enough: the resolution of the distributed formulation suffers from the curse of dimensionality n 1-dimensional state, The ultimate curse being to consider that the level takes values in the interval [x,x], so that the µ t s are continuous probability laws.

41 Back to time consistency Back to Dynamical Consistency We can use Dynamic Programing: Let V t µ t be the optimal cost of the problem starting at time t with initial condition µ t. { K,µ if g,µ a, V T µ = + otherwise, and, for every t = t 0,...,T 1 and every probability law µ on X: V t µ = min Λ Φt t,µ +V t+1 Φ t T Φt t µ. Optimal feedback functions Φ t depend on µ t. Consistency recovered for strategies depending on X t and µ t.

42 Back to time consistency Back to Dynamical Consistency 2 Optimal feedback functions Φ t depend on µ t µ min Λ Φt t,µ +V t+1 T Φt t. Φ t But it also depends on X t because of the distributed formulation U t = φ t X t! Consistency recovered for strategies depending on X t and µ t. The new state variable to consider X t,µ t is an infinite dimensional object!

43 Introduction Informal definition of dynamically consistent problems Dynamic Programming and dynamically consistent problems Deterministic case Distributed formulation A classical SOC Problem Equivalent distributed formulation SOC with constraints Constraints typology A toy example with finite probabilities Back to time consistency Reduction to finite-dimensional problem An equivalent problem with added state and control Dynamic programing

44 An equivalent problem with added state and control Equivalent problem: added state process Z and control V min U,V,X,Z T 1 E L t X t,u t,w t+1 +KX T, t=t 0 X t0 = x t0, X t+1 = f t X t,u t,w t+1, Z t0 = 0, Z t+1 = Z t +V t, Almost sure final constraint: gx T Z T a.. Measurability constraints: U t F t, V t F t+1. Additional time constraints: E V t F t = 0.

45 An equivalent problem with added state and control Equivalent problem 2 Let U t0,...,u T 1 and X t0,...,x T satisfying the constraints of Initial problem. We define V t0,...,v T 1 and Z t0,...,z T as follow: V t = E gx T F t+1 E gxt F t, Zt+1 = Z t +V t. Noting that the σ-field F t0 is the minimal σ-field and that the random variable X T is F T -measurable, we have: Z T = gx T E gx T. Hence, U, X, V and Z satisfy the set of constraints of the new problem.

46 An equivalent problem with added state and control Equivalent problem 3 Let U t0,...,u T 1, X t0,...,x T, V t0,...,v T 1 and Z t0,...,z T satisfying the constraints of V,Z-problem be given. Then, Z T = T 1 t=t 0 V t. Thus we have: E T 1 Z T = E E V t F t = 0. t=t 0 If follows that gx T Z T a E gx T a, and therefore the processes U t0,...,u T 1 and X t0,...,x T satisfy the constraints of Initial problem To conclude, since the two problems share the same sets as admissible constraints and share the same criteria to optimize, they are equivalent.

47 An equivalent problem with added state and control Equivalent problem 4 1. Dynamic Programming will give feedbacks as functions of X t,z t. 2. The new problem is more intricate: added state and added control, Vt depends on W t+1 Hazard Decision some new constraints on the controls are to be taken into account. 3. E gx T F t : perception of the risk constraint at time t. 4. V t = E gx T F t+1 E gxt F t : variation between time t and time t +1 of this perception. 5. Z t: cumulative variation over time of the risk constraint.

48 Dynamic programing Dynamic programing In order to use Dynamic programming on problem 7, we will first recall on a simplified model how Dynamic Programming works on a classical SOP. min X,U E ] L t Xt,U t,w t+1 +K XT, t=t 0 [ T 1 s.t. X t0 given, X t+1 = f t Xt,U t,w t+1, U t F t t = t 0,...,T 1, Then we will show the modifications induced by hazard-decision control or E V t F t = 0.

49 Dynamic programing Dynamic programing steps in classical framework 1. Rewrites criterion using conditional expectations: E E L 0 X 0,U 0 +E L 1 X 1,U 1 +KX 2 F 1 F0 2. minimization with respect to U t and X t enters up to the conditional expectation with respect to F t. 3. Recursive computation from the most internal problem to the outer one t = t 0. The most internal t = T 1: min E hx T 1,U T 1,W T F T 1. U F T 1 T 1 hx,u,w = L T 1 x,u+k f T 1 x,u,w

50 Dynamic programing Hazard Decision framework Hazard Decision: U t F t+1 Interchange of expectation and minimization can be done one step further. E min hx T 1,U T 1,W T F T 1. U F T 1 T with hx,u,w = L T 1 x,u,w+k f T 1 x,u,w Optimal feedback U T 1 can be chosen as a X T 1,W T -measurable function.

51 Dynamic programing Hazard Decision framework with constraint Control constraint: E U t F t = 0. We are led to the following problem at time t = T 1: min E hx T 1,U T 1,W T F T 1, U F T 1 T subject to: E U T 1 F T 1 = 0. Optimal feedback U T 1 can still be chosen as a X T 1,W T -measurable function? Yes and the proof is actually available when F T 1 is generated by finite valued random variables.

52 Dynamic programing Back to the equivalent problem The Hazard-Decision V and Decision-Hazard U controls. Initialization of the Bellman function at time T: V T x,z = Kx+χ G ad x,z, χ G ad characteristic function of Gad = { x,z gx z a } ; Bellman function at time t: V t x,z = min u U min E hx,u,z,v,w V W t+1, t+1 subject to: E V = 0. hx,u,z,v,w def = L t x,u,w+v t+1 ft x,u,w,z +v U t X t,z t and V t X t,z t,w t+1.

arxiv: v1 [math.oc] 20 May 2010

arxiv: v1 [math.oc] 20 May 2010 DYNAMIC CONSISTENCY FOR STOCHASTIC OPTIMAL CONTROL PROBLEMS PIERRE CARPENTIER, JEAN-PHILIPPE CHANCELIER, GUY COHEN, MICHEL DE LARA, AND PIERRE GIRARDEAU arxiv:1005.3605v1 [math.oc] 20 May 2010 Abstract.

More information

Deterministic Dynamic Programming

Deterministic Dynamic Programming Deterministic Dynamic Programming 1 Value Function Consider the following optimal control problem in Mayer s form: V (t 0, x 0 ) = inf u U J(t 1, x(t 1 )) (1) subject to ẋ(t) = f(t, x(t), u(t)), x(t 0

More information

Exam 2, Solution 4. Jordan Paschke

Exam 2, Solution 4. Jordan Paschke Exam 2, Solution 4 Jordan Paschke Problem 4. Suppose that y 1 (t) = t and y 2 (t) = t 3 are solutions to the equation 3t 3 y t 2 y 6ty + 6y = 0, t > 0. ( ) Find the general solution of the above equation.

More information

7. Shortest Path Problems and Deterministic Finite State Systems

7. Shortest Path Problems and Deterministic Finite State Systems 7. Shortest Path Problems and Deterministic Finite State Systems In the next two lectures we will look at shortest path problems, where the objective is to find the shortest path from a start node to an

More information

Fenchel-Moreau Conjugates of Inf-Transforms and Application to Stochastic Bellman Equation

Fenchel-Moreau Conjugates of Inf-Transforms and Application to Stochastic Bellman Equation Fenchel-Moreau Conjugates of Inf-Transforms and Application to Stochastic Bellman Equation Jean-Philippe Chancelier and Michel De Lara CERMICS, École des Ponts ParisTech First Conference on Discrete Optimization

More information

Alternative Characterization of Ergodicity for Doubly Stochastic Chains

Alternative Characterization of Ergodicity for Doubly Stochastic Chains Alternative Characterization of Ergodicity for Doubly Stochastic Chains Behrouz Touri and Angelia Nedić Abstract In this paper we discuss the ergodicity of stochastic and doubly stochastic chains. We define

More information

Kalman Filter and Parameter Identification. Florian Herzog

Kalman Filter and Parameter Identification. Florian Herzog Kalman Filter and Parameter Identification Florian Herzog 2013 Continuous-time Kalman Filter In this chapter, we shall use stochastic processes with independent increments w 1 (.) and w 2 (.) at the input

More information

Introduction - Motivation. Many phenomena (physical, chemical, biological, etc.) are model by differential equations. f f(x + h) f(x) (x) = lim

Introduction - Motivation. Many phenomena (physical, chemical, biological, etc.) are model by differential equations. f f(x + h) f(x) (x) = lim Introduction - Motivation Many phenomena (physical, chemical, biological, etc.) are model by differential equations. Recall the definition of the derivative of f(x) f f(x + h) f(x) (x) = lim. h 0 h Its

More information

1 Markov decision processes

1 Markov decision processes 2.997 Decision-Making in Large-Scale Systems February 4 MI, Spring 2004 Handout #1 Lecture Note 1 1 Markov decision processes In this class we will study discrete-time stochastic systems. We can describe

More information

Stochastic and Adaptive Optimal Control

Stochastic and Adaptive Optimal Control Stochastic and Adaptive Optimal Control Robert Stengel Optimal Control and Estimation, MAE 546 Princeton University, 2018! Nonlinear systems with random inputs and perfect measurements! Stochastic neighboring-optimal

More information

4th Preparation Sheet - Solutions

4th Preparation Sheet - Solutions Prof. Dr. Rainer Dahlhaus Probability Theory Summer term 017 4th Preparation Sheet - Solutions Remark: Throughout the exercise sheet we use the two equivalent definitions of separability of a metric space

More information

Positive Markov Jump Linear Systems (PMJLS) with applications

Positive Markov Jump Linear Systems (PMJLS) with applications Positive Markov Jump Linear Systems (PMJLS) with applications P. Bolzern, P. Colaneri DEIB, Politecnico di Milano - Italy December 12, 2015 Summary Positive Markov Jump Linear Systems Mean stability Input-output

More information

ENSC327 Communications Systems 19: Random Processes. Jie Liang School of Engineering Science Simon Fraser University

ENSC327 Communications Systems 19: Random Processes. Jie Liang School of Engineering Science Simon Fraser University ENSC327 Communications Systems 19: Random Processes Jie Liang School of Engineering Science Simon Fraser University 1 Outline Random processes Stationary random processes Autocorrelation of random processes

More information

HJB equations. Seminar in Stochastic Modelling in Economics and Finance January 10, 2011

HJB equations. Seminar in Stochastic Modelling in Economics and Finance January 10, 2011 Department of Probability and Mathematical Statistics Faculty of Mathematics and Physics, Charles University in Prague petrasek@karlin.mff.cuni.cz Seminar in Stochastic Modelling in Economics and Finance

More information

Game Theory Extra Lecture 1 (BoB)

Game Theory Extra Lecture 1 (BoB) Game Theory 2014 Extra Lecture 1 (BoB) Differential games Tools from optimal control Dynamic programming Hamilton-Jacobi-Bellman-Isaacs equation Zerosum linear quadratic games and H control Baser/Olsder,

More information

Solution of Stochastic Optimal Control Problems and Financial Applications

Solution of Stochastic Optimal Control Problems and Financial Applications Journal of Mathematical Extension Vol. 11, No. 4, (2017), 27-44 ISSN: 1735-8299 URL: http://www.ijmex.com Solution of Stochastic Optimal Control Problems and Financial Applications 2 Mat B. Kafash 1 Faculty

More information

Stochastic Shortest Path Problems

Stochastic Shortest Path Problems Chapter 8 Stochastic Shortest Path Problems 1 In this chapter, we study a stochastic version of the shortest path problem of chapter 2, where only probabilities of transitions along different arcs can

More information

CHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME. Chapter2 p. 1/67

CHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME. Chapter2 p. 1/67 CHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME Chapter2 p. 1/67 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME Main Purpose: Introduce the maximum principle as a necessary condition to be satisfied by any optimal

More information

Static and Dynamic Optimization (42111)

Static and Dynamic Optimization (42111) Static and Dynamic Optimization (421) Niels Kjølstad Poulsen Build. 0b, room 01 Section for Dynamical Systems Dept. of Applied Mathematics and Computer Science The Technical University of Denmark Email:

More information

On the Convergence of Optimistic Policy Iteration

On the Convergence of Optimistic Policy Iteration Journal of Machine Learning Research 3 (2002) 59 72 Submitted 10/01; Published 7/02 On the Convergence of Optimistic Policy Iteration John N. Tsitsiklis LIDS, Room 35-209 Massachusetts Institute of Technology

More information

Matthew Zyskowski 1 Quanyan Zhu 2

Matthew Zyskowski 1 Quanyan Zhu 2 Matthew Zyskowski 1 Quanyan Zhu 2 1 Decision Science, Credit Risk Office Barclaycard US 2 Department of Electrical Engineering Princeton University Outline Outline I Modern control theory with game-theoretic

More information

QUALIFYING EXAM IN SYSTEMS ENGINEERING

QUALIFYING EXAM IN SYSTEMS ENGINEERING QUALIFYING EXAM IN SYSTEMS ENGINEERING Written Exam: MAY 23, 2017, 9:00AM to 1:00PM, EMB 105 Oral Exam: May 25 or 26, 2017 Time/Location TBA (~1 hour per student) CLOSED BOOK, NO CHEAT SHEETS BASIC SCIENTIFIC

More information

Uncertainty quantification and systemic risk

Uncertainty quantification and systemic risk Uncertainty quantification and systemic risk Josselin Garnier (Université Paris Diderot) with George Papanicolaou and Tzu-Wei Yang (Stanford University) February 3, 2016 Modeling systemic risk We consider

More information

Gaussian, Markov and stationary processes

Gaussian, Markov and stationary processes Gaussian, Markov and stationary processes Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ November

More information

14 - Gaussian Stochastic Processes

14 - Gaussian Stochastic Processes 14-1 Gaussian Stochastic Processes S. Lall, Stanford 211.2.24.1 14 - Gaussian Stochastic Processes Linear systems driven by IID noise Evolution of mean and covariance Example: mass-spring system Steady-state

More information

EN Applied Optimal Control Lecture 8: Dynamic Programming October 10, 2018

EN Applied Optimal Control Lecture 8: Dynamic Programming October 10, 2018 EN530.603 Applied Optimal Control Lecture 8: Dynamic Programming October 0, 08 Lecturer: Marin Kobilarov Dynamic Programming (DP) is conerned with the computation of an optimal policy, i.e. an optimal

More information

Lecture 6: Bayesian Inference in SDE Models

Lecture 6: Bayesian Inference in SDE Models Lecture 6: Bayesian Inference in SDE Models Bayesian Filtering and Smoothing Point of View Simo Särkkä Aalto University Simo Särkkä (Aalto) Lecture 6: Bayesian Inference in SDEs 1 / 45 Contents 1 SDEs

More information

MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016

MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016 MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016 Lecture 14: Information Theoretic Methods Lecturer: Jiaming Xu Scribe: Hilda Ibriga, Adarsh Barik, December 02, 2016 Outline f-divergence

More information

Stochastic Composition Optimization

Stochastic Composition Optimization Stochastic Composition Optimization Algorithms and Sample Complexities Mengdi Wang Joint works with Ethan X. Fang, Han Liu, and Ji Liu ORFE@Princeton ICCOPT, Tokyo, August 8-11, 2016 1 / 24 Collaborators

More information

Decomposability and time consistency of risk averse multistage programs

Decomposability and time consistency of risk averse multistage programs Decomposability and time consistency of risk averse multistage programs arxiv:1806.01497v1 [math.oc] 5 Jun 2018 A. Shapiro School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta,

More information

Z i Q ij Z j. J(x, φ; U) = X T φ(t ) 2 h + where h k k, H(t) k k and R(t) r r are nonnegative definite matrices (R(t) is uniformly in t nonsingular).

Z i Q ij Z j. J(x, φ; U) = X T φ(t ) 2 h + where h k k, H(t) k k and R(t) r r are nonnegative definite matrices (R(t) is uniformly in t nonsingular). 2. LINEAR QUADRATIC DETERMINISTIC PROBLEM Notations: For a vector Z, Z = Z, Z is the Euclidean norm here Z, Z = i Z2 i is the inner product; For a vector Z and nonnegative definite matrix Q, Z Q = Z, QZ

More information

Optimal Control. McGill COMP 765 Oct 3 rd, 2017

Optimal Control. McGill COMP 765 Oct 3 rd, 2017 Optimal Control McGill COMP 765 Oct 3 rd, 2017 Classical Control Quiz Question 1: Can a PID controller be used to balance an inverted pendulum: A) That starts upright? B) That must be swung-up (perhaps

More information

The Social Cost of Stochastic & Irreversible Climate Change

The Social Cost of Stochastic & Irreversible Climate Change The Social Cost of Stochastic & Irreversible Climate Change Thomas S. Lontzek (University of Zurich and RDCEP) joint work with Yongyang Cai and Kenneth L. Judd (both at Hoover Institution, RDCEP and NBER)

More information

2.3 Terminology for Systems of Linear Equations

2.3 Terminology for Systems of Linear Equations page 133 e 2t sin 2t 44 A(t) = t 2 5 te t, a = 0, b = 1 sec 2 t 3t sin t 45 The matrix function A(t) in Problem 39, with a = 0 and b = 1 Integration of matrix functions given in the text was done with

More information

Linear Quadratic Zero-Sum Two-Person Differential Games Pierre Bernhard June 15, 2013

Linear Quadratic Zero-Sum Two-Person Differential Games Pierre Bernhard June 15, 2013 Linear Quadratic Zero-Sum Two-Person Differential Games Pierre Bernhard June 15, 2013 Abstract As in optimal control theory, linear quadratic (LQ) differential games (DG) can be solved, even in high dimension,

More information

An Uncertain Control Model with Application to. Production-Inventory System

An Uncertain Control Model with Application to. Production-Inventory System An Uncertain Control Model with Application to Production-Inventory System Kai Yao 1, Zhongfeng Qin 2 1 Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China 2 School of Economics

More information

LECTURE 3. Last time:

LECTURE 3. Last time: LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate

More information

Uniformly Uniformly-ergodic Markov chains and BSDEs

Uniformly Uniformly-ergodic Markov chains and BSDEs Uniformly Uniformly-ergodic Markov chains and BSDEs Samuel N. Cohen Mathematical Institute, University of Oxford (Based on joint work with Ying Hu, Robert Elliott, Lukas Szpruch) Centre Henri Lebesgue,

More information

Markov Chain Monte Carlo Methods for Stochastic

Markov Chain Monte Carlo Methods for Stochastic Markov Chain Monte Carlo Methods for Stochastic Optimization i John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge U Florida, Nov 2013

More information

Markov Chain BSDEs and risk averse networks

Markov Chain BSDEs and risk averse networks Markov Chain BSDEs and risk averse networks Samuel N. Cohen Mathematical Institute, University of Oxford (Based on joint work with Ying Hu, Robert Elliott, Lukas Szpruch) 2nd Young Researchers in BSDEs

More information

Region of attraction approximations for polynomial dynamical systems

Region of attraction approximations for polynomial dynamical systems Region of attraction approximations for polynomial dynamical systems Milan Korda EPFL Lausanne Didier Henrion LAAS-CNRS Toulouse & CTU Prague Colin N. Jones EPFL Lausanne Region of Attraction (ROA) ẋ(t)

More information

Extended Formulations, Lagrangian Relaxation, & Column Generation: tackling large scale applications

Extended Formulations, Lagrangian Relaxation, & Column Generation: tackling large scale applications Extended Formulations, Lagrangian Relaxation, & Column Generation: tackling large scale applications François Vanderbeck University of Bordeaux INRIA Bordeaux-Sud-Ouest part : Defining Extended Formulations

More information

Exercises. T 2T. e ita φ(t)dt.

Exercises. T 2T. e ita φ(t)dt. Exercises. Set #. Construct an example of a sequence of probability measures P n on R which converge weakly to a probability measure P but so that the first moments m,n = xdp n do not converge to m = xdp.

More information

Lecture 5 Linear Quadratic Stochastic Control

Lecture 5 Linear Quadratic Stochastic Control EE363 Winter 2008-09 Lecture 5 Linear Quadratic Stochastic Control linear-quadratic stochastic control problem solution via dynamic programming 5 1 Linear stochastic system linear dynamical system, over

More information

Modeling and Analysis of Dynamic Systems

Modeling and Analysis of Dynamic Systems Modeling and Analysis of Dynamic Systems Dr. Guillaume Ducard Fall 2017 Institute for Dynamic Systems and Control ETH Zurich, Switzerland G. Ducard c 1 / 57 Outline 1 Lecture 13: Linear System - Stability

More information

Hamilton-Jacobi-Bellman Equation Feb 25, 2008

Hamilton-Jacobi-Bellman Equation Feb 25, 2008 Hamilton-Jacobi-Bellman Equation Feb 25, 2008 What is it? The Hamilton-Jacobi-Bellman (HJB) equation is the continuous-time analog to the discrete deterministic dynamic programming algorithm Discrete VS

More information

Discrete-Time Finite-Horizon Optimal ALM Problem with Regime-Switching for DB Pension Plan

Discrete-Time Finite-Horizon Optimal ALM Problem with Regime-Switching for DB Pension Plan Applied Mathematical Sciences, Vol. 10, 2016, no. 33, 1643-1652 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/10.12988/ams.2016.6383 Discrete-Time Finite-Horizon Optimal ALM Problem with Regime-Switching

More information

LMI Methods in Optimal and Robust Control

LMI Methods in Optimal and Robust Control LMI Methods in Optimal and Robust Control Matthew M. Peet Arizona State University Lecture 15: Nonlinear Systems and Lyapunov Functions Overview Our next goal is to extend LMI s and optimization to nonlinear

More information

Module 05 Introduction to Optimal Control

Module 05 Introduction to Optimal Control Module 05 Introduction to Optimal Control Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html October 8, 2015

More information

Filtrations, Markov Processes and Martingales. Lectures on Lévy Processes and Stochastic Calculus, Braunschweig, Lecture 3: The Lévy-Itô Decomposition

Filtrations, Markov Processes and Martingales. Lectures on Lévy Processes and Stochastic Calculus, Braunschweig, Lecture 3: The Lévy-Itô Decomposition Filtrations, Markov Processes and Martingales Lectures on Lévy Processes and Stochastic Calculus, Braunschweig, Lecture 3: The Lévy-Itô Decomposition David pplebaum Probability and Statistics Department,

More information

ACM/CMS 107 Linear Analysis & Applications Fall 2016 Assignment 4: Linear ODEs and Control Theory Due: 5th December 2016

ACM/CMS 107 Linear Analysis & Applications Fall 2016 Assignment 4: Linear ODEs and Control Theory Due: 5th December 2016 ACM/CMS 17 Linear Analysis & Applications Fall 216 Assignment 4: Linear ODEs and Control Theory Due: 5th December 216 Introduction Systems of ordinary differential equations (ODEs) can be used to describe

More information

Chain differentials with an application to the mathematical fear operator

Chain differentials with an application to the mathematical fear operator Chain differentials with an application to the mathematical fear operator Pierre Bernhard I3S, University of Nice Sophia Antipolis and CNRS, ESSI, B.P. 145, 06903 Sophia Antipolis cedex, France January

More information

Formula Sheet for Optimal Control

Formula Sheet for Optimal Control Formula Sheet for Optimal Control Division of Optimization and Systems Theory Royal Institute of Technology 144 Stockholm, Sweden 23 December 1, 29 1 Dynamic Programming 11 Discrete Dynamic Programming

More information

Game Theory with Information: Introducing the Witsenhausen Intrinsic Model

Game Theory with Information: Introducing the Witsenhausen Intrinsic Model Game Theory with Information: Introducing the Witsenhausen Intrinsic Model Michel De Lara and Benjamin Heymann Cermics, École des Ponts ParisTech France École des Ponts ParisTech March 15, 2017 Information

More information

1 Stochastic Dynamic Programming

1 Stochastic Dynamic Programming 1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future

More information

16.30 Estimation and Control of Aerospace Systems

16.30 Estimation and Control of Aerospace Systems 16.30 Estimation and Control of Aerospace Systems Topic 5 addendum: Signals and Systems Aeronautics and Astronautics Massachusetts Institute of Technology Fall 2010 (MIT) Topic 5 addendum: Signals, Systems

More information

Nonlinear Observers. Jaime A. Moreno. Eléctrica y Computación Instituto de Ingeniería Universidad Nacional Autónoma de México

Nonlinear Observers. Jaime A. Moreno. Eléctrica y Computación Instituto de Ingeniería Universidad Nacional Autónoma de México Nonlinear Observers Jaime A. Moreno JMorenoP@ii.unam.mx Eléctrica y Computación Instituto de Ingeniería Universidad Nacional Autónoma de México XVI Congreso Latinoamericano de Control Automático October

More information

Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters

Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 204 Emo Todorov (UW) AMATH/CSE 579, Winter

More information

Management of dam systems via optimal price control

Management of dam systems via optimal price control Available online at www.sciencedirect.com Procedia Computer Science 4 (211) 1373 1382 International Conference on Computational Science, ICCS 211 Management of dam systems via optimal price control Boris

More information

Homogeneous Linear Systems and Their General Solutions

Homogeneous Linear Systems and Their General Solutions 37 Homogeneous Linear Systems and Their General Solutions We are now going to restrict our attention further to the standard first-order systems of differential equations that are linear, with particular

More information

Math 249B. Nilpotence of connected solvable groups

Math 249B. Nilpotence of connected solvable groups Math 249B. Nilpotence of connected solvable groups 1. Motivation and examples In abstract group theory, the descending central series {C i (G)} of a group G is defined recursively by C 0 (G) = G and C

More information

Brownian Motion. 1 Definition Brownian Motion Wiener measure... 3

Brownian Motion. 1 Definition Brownian Motion Wiener measure... 3 Brownian Motion Contents 1 Definition 2 1.1 Brownian Motion................................. 2 1.2 Wiener measure.................................. 3 2 Construction 4 2.1 Gaussian process.................................

More information

Markov decision processes and interval Markov chains: exploiting the connection

Markov decision processes and interval Markov chains: exploiting the connection Markov decision processes and interval Markov chains: exploiting the connection Mingmei Teo Supervisors: Prof. Nigel Bean, Dr Joshua Ross University of Adelaide July 10, 2013 Intervals and interval arithmetic

More information

The first order quasi-linear PDEs

The first order quasi-linear PDEs Chapter 2 The first order quasi-linear PDEs The first order quasi-linear PDEs have the following general form: F (x, u, Du) = 0, (2.1) where x = (x 1, x 2,, x 3 ) R n, u = u(x), Du is the gradient of u.

More information

6.231 DYNAMIC PROGRAMMING LECTURE 13 LECTURE OUTLINE

6.231 DYNAMIC PROGRAMMING LECTURE 13 LECTURE OUTLINE 6.231 DYNAMIC PROGRAMMING LECTURE 13 LECTURE OUTLINE Control of continuous-time Markov chains Semi-Markov problems Problem formulation Equivalence to discretetime problems Discounted problems Average cost

More information

MATH 220 solution to homework 5

MATH 220 solution to homework 5 MATH 220 solution to homework 5 Problem. (i Define E(t = k(t + p(t = then E (t = 2 = 2 = 2 u t u tt + u x u xt dx u 2 t + u 2 x dx, u t u xx + u x u xt dx x [u tu x ] dx. Because f and g are compactly

More information

3 Compact Operators, Generalized Inverse, Best- Approximate Solution

3 Compact Operators, Generalized Inverse, Best- Approximate Solution 3 Compact Operators, Generalized Inverse, Best- Approximate Solution As we have already heard in the lecture a mathematical problem is well - posed in the sense of Hadamard if the following properties

More information

MEAN FIELD GAMES WITH MAJOR AND MINOR PLAYERS

MEAN FIELD GAMES WITH MAJOR AND MINOR PLAYERS MEAN FIELD GAMES WITH MAJOR AND MINOR PLAYERS René Carmona Department of Operations Research & Financial Engineering PACM Princeton University CEMRACS - Luminy, July 17, 217 MFG WITH MAJOR AND MINOR PLAYERS

More information

Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Process Approximations

Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Process Approximations Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Process Approximations Simo Särkkä Aalto University Tampere University of Technology Lappeenranta University of Technology Finland November

More information

CONTROL SYSTEMS, ROBOTICS AND AUTOMATION Vol. XI Stochastic Stability - H.J. Kushner

CONTROL SYSTEMS, ROBOTICS AND AUTOMATION Vol. XI Stochastic Stability - H.J. Kushner STOCHASTIC STABILITY H.J. Kushner Applied Mathematics, Brown University, Providence, RI, USA. Keywords: stability, stochastic stability, random perturbations, Markov systems, robustness, perturbed systems,

More information

Robustness and bootstrap techniques in portfolio efficiency tests

Robustness and bootstrap techniques in portfolio efficiency tests Robustness and bootstrap techniques in portfolio efficiency tests Dept. of Probability and Mathematical Statistics, Charles University, Prague, Czech Republic July 8, 2013 Motivation Portfolio selection

More information

Advanced Computer Networks Lecture 2. Markov Processes

Advanced Computer Networks Lecture 2. Markov Processes Advanced Computer Networks Lecture 2. Markov Processes Husheng Li Min Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville Spring, 2016 1/28 Outline 2/28 1 Definition

More information

Conservation Laws: Systematic Construction, Noether s Theorem, Applications, and Symbolic Computations.

Conservation Laws: Systematic Construction, Noether s Theorem, Applications, and Symbolic Computations. Conservation Laws: Systematic Construction, Noether s Theorem, Applications, and Symbolic Computations. Alexey Shevyakov Department of Mathematics and Statistics, University of Saskatchewan, Saskatoon,

More information

Chapter 0 of Calculus ++, Differential calculus with several variables

Chapter 0 of Calculus ++, Differential calculus with several variables Chapter of Calculus ++, Differential calculus with several variables Background material by Eric A Carlen Professor of Mathematics Georgia Tech Spring 6 c 6 by the author, all rights reserved - Table of

More information

6.241 Dynamic Systems and Control

6.241 Dynamic Systems and Control 6.241 Dynamic Systems and Control Lecture 7: State-space Models Readings: DDV, Chapters 7,8 Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology February 25, 2011 E. Frazzoli

More information

Stochastic optimal control with rough paths

Stochastic optimal control with rough paths Stochastic optimal control with rough paths Paul Gassiat TU Berlin Stochastic processes and their statistics in Finance, Okinawa, October 28, 2013 Joint work with Joscha Diehl and Peter Friz Introduction

More information

Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Approximations

Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Approximations Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Approximations Simo Särkkä Aalto University, Finland November 18, 2014 Simo Särkkä (Aalto) Lecture 4: Numerical Solution of SDEs November

More information

DEPARTMENT OF MATHEMATICS

DEPARTMENT OF MATHEMATICS DEPARTMENT OF MATHEMATICS Ma322 - Final Exam Spring 2011 May 3,4, 2011 DO NOT TURN THIS PAGE UNTIL YOU ARE INSTRUCTED TO DO SO. Be sure to show all work and justify your answers. There are 8 problems and

More information

Chap. 3. Controlled Systems, Controllability

Chap. 3. Controlled Systems, Controllability Chap. 3. Controlled Systems, Controllability 1. Controllability of Linear Systems 1.1. Kalman s Criterion Consider the linear system ẋ = Ax + Bu where x R n : state vector and u R m : input vector. A :

More information

Kernels for Dynamic Textures

Kernels for Dynamic Textures Kernels for Dynamic Textures S.V.N. Vishwanathan SVN.Vishwanathan@nicta.com.au http://web.anu.edu.au/~vishy National ICT Australia and Australian National University Joint work with Alex Smola and René

More information

Linear Algebra Review (Course Notes for Math 308H - Spring 2016)

Linear Algebra Review (Course Notes for Math 308H - Spring 2016) Linear Algebra Review (Course Notes for Math 308H - Spring 2016) Dr. Michael S. Pilant February 12, 2016 1 Background: We begin with one of the most fundamental notions in R 2, distance. Letting (x 1,

More information

AC&ST AUTOMATIC CONTROL AND SYSTEM THEORY SYSTEMS AND MODELS. Claudio Melchiorri

AC&ST AUTOMATIC CONTROL AND SYSTEM THEORY SYSTEMS AND MODELS. Claudio Melchiorri C. Melchiorri (DEI) Automatic Control & System Theory 1 AUTOMATIC CONTROL AND SYSTEM THEORY SYSTEMS AND MODELS Claudio Melchiorri Dipartimento di Ingegneria dell Energia Elettrica e dell Informazione (DEI)

More information

Linear-Quadratic Optimal Control: Full-State Feedback

Linear-Quadratic Optimal Control: Full-State Feedback Chapter 4 Linear-Quadratic Optimal Control: Full-State Feedback 1 Linear quadratic optimization is a basic method for designing controllers for linear (and often nonlinear) dynamical systems and is actually

More information

6.231 DYNAMIC PROGRAMMING LECTURE 6 LECTURE OUTLINE

6.231 DYNAMIC PROGRAMMING LECTURE 6 LECTURE OUTLINE 6.231 DYNAMIC PROGRAMMING LECTURE 6 LECTURE OUTLINE Review of Q-factors and Bellman equations for Q-factors VI and PI for Q-factors Q-learning - Combination of VI and sampling Q-learning and cost function

More information

Chapter 7. Canonical Forms. 7.1 Eigenvalues and Eigenvectors

Chapter 7. Canonical Forms. 7.1 Eigenvalues and Eigenvectors Chapter 7 Canonical Forms 7.1 Eigenvalues and Eigenvectors Definition 7.1.1. Let V be a vector space over the field F and let T be a linear operator on V. An eigenvalue of T is a scalar λ F such that there

More information

MATH The Chain Rule Fall 2016 A vector function of a vector variable is a function F: R n R m. In practice, if x 1, x n is the input,

MATH The Chain Rule Fall 2016 A vector function of a vector variable is a function F: R n R m. In practice, if x 1, x n is the input, MATH 20550 The Chain Rule Fall 2016 A vector function of a vector variable is a function F: R n R m. In practice, if x 1, x n is the input, F(x 1,, x n ) F 1 (x 1,, x n ),, F m (x 1,, x n ) where each

More information

Control Theory in Physics and other Fields of Science

Control Theory in Physics and other Fields of Science Michael Schulz Control Theory in Physics and other Fields of Science Concepts, Tools, and Applications With 46 Figures Sprin ger 1 Introduction 1 1.1 The Aim of Control Theory 1 1.2 Dynamic State of Classical

More information

Sequential State Estimation (Crassidas and Junkins, Chapter 5)

Sequential State Estimation (Crassidas and Junkins, Chapter 5) Sequential State Estimation (Crassidas and Junkins, Chapter 5) Please read: 5.1, 5.3-5.6 5.3 The Discrete-Time Kalman Filter The discrete-time Kalman filter is used when the dynamics and measurements are

More information

Linear Programming Methods

Linear Programming Methods Chapter 11 Linear Programming Methods 1 In this chapter we consider the linear programming approach to dynamic programming. First, Bellman s equation can be reformulated as a linear program whose solution

More information

Uncertainty quantification and systemic risk

Uncertainty quantification and systemic risk Uncertainty quantification and systemic risk Josselin Garnier (Université Paris 7) with George Papanicolaou and Tzu-Wei Yang (Stanford University) April 9, 2013 Modeling Systemic Risk We consider evolving

More information

Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation

Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation M.-O. Hongler, O. Gallay, R. Colmorn, P. Cordes and M. Hülsmann Ecole Polytechnique Fédérale de Lausanne (EPFL), (CH)

More information

Introduction to Nonlinear Control Lecture # 3 Time-Varying and Perturbed Systems

Introduction to Nonlinear Control Lecture # 3 Time-Varying and Perturbed Systems p. 1/5 Introduction to Nonlinear Control Lecture # 3 Time-Varying and Perturbed Systems p. 2/5 Time-varying Systems ẋ = f(t, x) f(t, x) is piecewise continuous in t and locally Lipschitz in x for all t

More information

UC Berkeley Summer Undergraduate Research Program 2015 July 8 Lecture

UC Berkeley Summer Undergraduate Research Program 2015 July 8 Lecture UC Berkeley Summer Undergraduate Research Program 25 July 8 Lecture This lecture is intended to tie up some (potential) loose ends that we have encountered on the road during the past couple of weeks We

More information

Prof. Erhan Bayraktar (University of Michigan)

Prof. Erhan Bayraktar (University of Michigan) September 17, 2012 KAP 414 2:15 PM- 3:15 PM Prof. (University of Michigan) Abstract: We consider a zero-sum stochastic differential controller-and-stopper game in which the state process is a controlled

More information

Mean-Field Games with non-convex Hamiltonian

Mean-Field Games with non-convex Hamiltonian Mean-Field Games with non-convex Hamiltonian Martino Bardi Dipartimento di Matematica "Tullio Levi-Civita" Università di Padova Workshop "Optimal Control and MFGs" Pavia, September 19 21, 2018 Martino

More information

Utility Maximization in Hidden Regime-Switching Markets with Default Risk

Utility Maximization in Hidden Regime-Switching Markets with Default Risk Utility Maximization in Hidden Regime-Switching Markets with Default Risk José E. Figueroa-López Department of Mathematics and Statistics Washington University in St. Louis figueroa-lopez@wustl.edu pages.wustl.edu/figueroa

More information

6 Reinforcement Learning

6 Reinforcement Learning 6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,

More information

Abstrct. In this paper, we consider the problem of optimal flow control for a production system with one machine which is subject to failures and prod

Abstrct. In this paper, we consider the problem of optimal flow control for a production system with one machine which is subject to failures and prod Monotonicity of Optimal Flow Control for Failure Prone Production Systems 1 Jian-Qiang Hu 2 and Dong Xiang 3 Communicated By W.B. Gong 1 This work is partially supported by the national Science Foundation

More information

Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation Logistics

Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation Logistics Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation Logistics O. Gallay, M.-O. Hongler, R. Colmorn, P. Cordes and M. Hülsmann Ecole Polytechnique Fédérale de Lausanne

More information

LECTURES 2-3 : Stochastic Processes, Autocorrelation function. Stationarity.

LECTURES 2-3 : Stochastic Processes, Autocorrelation function. Stationarity. LECTURES 2-3 : Stochastic Processes, Autocorrelation function. Stationarity. Important points of Lecture 1: A time series {X t } is a series of observations taken sequentially over time: x t is an observation

More information