Stochastic and Adaptive Optimal Control
|
|
- Earl Patterson
- 5 years ago
- Views:
Transcription
1 Stochastic and Adaptive Optimal Control Robert Stengel Optimal Control and Estimation, MAE 546 Princeton University, 2018! Nonlinear systems with random inputs and perfect measurements! Stochastic neighboring-optimal control! Linear-quadratic (LQ) optimal control with perfect measurements! Adaptive Critic Control! Information Sets and Expected Cost Copyright 2018 by Robert Stengel. All rights reserved. For educational use only Nonlinear Systems with Random Inputs and Perfect Measurements Inputs and initial conditions are uncertain, but the state can be measured without error = f x( t),u t z( t) = x( t) x t E x( 0) = x( 0) x ( 0 ) T { x( 0) } = 0 E x 0 x 0,w( t),t Assume that random disturbance effects are small and additive = f x t x t,u( t),t E w( t)w T τ + L( t)w( t) E w( t) = 0 = W t δ t τ 2
2 Cost Must Be an Expected Value Deterministic cosunction cannot be minimized because disturbance effect on state cannot be predicted state and control are random variables min u(t ) min u(t ) J = φ x( ) + J = E φ x( ) + t o t o L [ x(t),u(t) ] However, the expected value of a deterministic cost function can be minimized L [ x(t),u(t) ] dt dt 3 Stochastic Euler-Lagrange Equations? There is no single optimal trajectory Expected values of Euler-Lagrange necessary conditions may not be well defined 1) E λ( ) = E φ[x(t )] f x 2) E λ(t) = E H[x(t),u(t),λ(t),t] x T T 3) E H[x(t),u(t),λ(t),t] u = 0 4
3 Stochastic Value Function for a Nonlinear System Expected values of terminal and integral cost are well defined Apply Hamilton-Jacobi-Bellman (HJB) to expected terminal and integral costs Principle of Optimality Optimal expected value function at t 1 V * t 1 = E φ x *( ) = min u t 1 [ ] L x *(τ ),u*(τ ) E t 1 φ x *(t ) f L x *(τ ),u(τ ) [ ] dτ dτ 5 Rate of Change of the Value Function Total time-derivative of V* dv * = E L x * (t 1 ),u *(t 1 ) dt t =t 1 { [ ]} x(t) and u(t) can be known precisely; therefore dv * dt t =t 1 = L x * (t 1 ),u * (t 1 ) [ ] 6
4 ΔV* = dv * dt Incremental Change in the Value Function Apply chain rule to total derivative dv * dt = E V * + V * t x x Δt = E V * Δt + V * t x!xδt V * 2!xT!x x 2 Δt 2 +" = E V * Δt + V * t x f. Incremental change in value function, Expand to second degree ( + Lw (.)) Δt Next: cancel Δt ( f (.) + Lw (.)) T 2 V * f. x 2 ( + Lw (.)) Δt 2 +" 7 dv * dt Tr x T Qx Introduction of the Trace Trace of a matrix product is scalar Tr( ABC) = Tr( CAB) = Tr( BCA) = Tr( xx T Q) = Tr( Qxx T ) dim Tr E V * + V * t x f. = E V * + V * t x f. ( + Lw (.)) Tr f (.) + Lw (.) ( + Lw (.)) Tr 2 V * = 1 1 T 2 V * f. x 2 f (.) + Lw (.) x 2 f. + Lw (.) + Lw (.) Δt T Δt 8
5 dv * dt = E V * t Toward the Stochastic HJB Equation + V * x = V * + V * t x f. Because x(t) and u(t) can be measured, f (.) + Lw (.) Tr 2 V * f (.) + Lw (.) x 2 f. + E V * x Lw Tr 2 V * f (.) + Lw (.) x 2 f. + Lw (.) + Lw (.) T Δt 1 st two terms can be taken outside the expectation T Δt 9 Toward the Stochastic HJB Equation Disturbance is assumed to be zero-mean white noise E w( t) = 0 w T ( τ ) E f. = 0 E w( t)w T ( τ ) = W( t)δ t τ E w( t)w T ( τ ) Δt W t Δt 0 dv * dt = V * + V * t x f (.) lim = V * t Uncertain disturbance input can only increase the value function rate of change Tr 2 V * E( f (.)f (.) T )Δt + LE( w( t)w( τ ) T )L T Δt 0 x 2 Δt 0 ( t) + V * ( x t)f (.) Tr 2 V * ( t)l( t)w( t)l( t) T x 2 10
6 Stochastic Principle of Optimality (Perfect Measurements) dv * dt min u = V * t E dv * dt ( t) + V * ( x t)f. + V * x Tr 2 V * V * t ( t) = x 2 ( t)l( t)w( t)l t! Substitute for total derivative, dv*/dt = L(x*,u*)! Solve for the partial derivative, V*/ t! Expected value of HJB Equation -> Stochastic HJB Equation = min u T ( t)f x *( t),u( t),t Tr 2 V * ( t)l( t)w( t)l t x 2 E L x *( t),u( t),t + V * ( x t)f x *( t),u( t),t Tr 2 V * ( t)l( t)w( t)l t x 2 Boundary (terminal) condition : V * = E φ ( ) T T 11 min u Observations of Stochastic Principle of Optimality (Perfect Measurements) V * t ( t) = E L x *( t),u( t),t + V * ( x t)f x *( t),u( t),t Tr 2 V * ( t)l( t)w( t)l t x 2! Control has no effect on the disturbance input! Criterion for optimality is the same as for the deterministic case! Disturbance uncertainty increases the magnitude of the total optimal value function, E[V*(0)] T 12
7 Adaptive Critic Controller 13 Adaptive Critic Controller Nonlinear control law, c, takes the general form x(t) : = c x(t),a,y * ( t) u t a : y *(t) : state [ ] parameters defining an operating point command input On-line adaptive critic controller Nonlinear control law ( action network ) Criticizes non-optimal performance via critic network Adapts control gains to improve performance, respond to failures, and accommodate parameter variation Adapts cost model to improve performance evaluation 14
8 Adaptive Critic Iteration Cycle Ferrari, Stengel, Model-Based Adaptive Critic Designs, in Learning and Approximate Dynamic Programming, Si et al, ed., Overview of Adaptive Critic Designs 16
9 Modular Description of Typical Iteration Cycle 17 Gain Scheduled Controller Design PI-LQ controllers (LQR with integral compensation) that satisfy requirements at N operating points Scheduling variable, a, identifies each operating point u( t k ) = C F ( a i )y *( t k ) + C B ( a i )Δx( t k ) + C I ( a i ) Δy τ c x(t k ),a i,y *( t k ), i = 1, N t k t k 1 dτ 18
10 Replace Gain Matrices by Neural Networks Replace control gain matrices by sigmoidal neural networks = u t k NN F y *( t k ),a( t k ) + NN B x( t k ),a( t k ) + NN I Δy( τ )dτ,a( t k ) = c x(t k ),a,y *( t k ) 19 Sigmoidal Neuron Input-Output Characteristic Logistic sigmoid function Sigmoid with two inputs, one output u = s(r) = or 1 1+ e r u = s(r) = er 1 e r +1 = tanh r 2 u = s(r) = 1 1+ e w 1r 1 +w 2 r 2 +b 20
11 Algebraically Trained Neural Control Law Algebraic training of neural networks produces exacit of PI- LQ control gains and trim conditions at N operating points Interpolation and gain scheduling via neural networks One node per operating point in each neural network See Ferrari, Stengel, JGCD, 2002 [on Blackboard] 21 On-line Optimization of Adaptive Critic Neural Network Controller Critic adapts neural network weights to improve performance using approximate dynamic programming Ferrari, Stengel, JGCD,
12 Heuristic Dynamic Programming Adaptive Critic Dual Heuristic, Dynamic Programming Adaptive Critic for receding-horizon optimization problem V x a ( t k ) = L x a ( t k ),u( t k ) + V x a ( t k+1 ) [ ] V u = L u + V x a x a u = 0 V x a t x a t [ ] = NN C x a ( t),a t Critic and Action (i.e., Control) networks adapted concurrently LQ-PI cosunction applied to nonlinear problem Modified resilient backpropagation for neural network training on 2 nd Time Step Ferrari, Stengel, JGCD, Action Network On-line Training Train action network, at time t, holding the critic parameters fixed x a (t) a(t) NN A Aircraft Model Transition Matrices State Prediction Utility Function Derivatives NN A Target Optimality Condition NN C Target Generation 24
13 Critic Network On-line Training Train critic network, at time t, holding the action parameters fixed x a (t) a(t) NN A Aircraft Model Transition Matrices State Prediction NN C Utility Function Derivatives NN C (old) NN C Target Target Cost Gradient Target Generation 25 Adaptive Critic Flight Simulation 70-deg Bank Angle Command Multiple Failures (Thrust, Stabilator, Rudder) Ferrari, Stengel, JGCD,
14 Stochastic Linear-Quadratic Optimal Control 27 Stochastic Principle of Optimality Applied to the Linear-Quadratic (LQ) Problem Quadratic value function = E φ x( ) V t o t o L [ x(τ ),u(τ ) ] dτ = 1 t 2 E o Q(t) M(t) xt ( )S( )x( ) x T (t) u T (t) M T (t) R(t) x(t) u(t) dt Linear dynamic constraint x ( t) = F(t)x(t) + G(t)u(t) + L(t)w(t) 28
15 Components of the LQ Value Function Quadratic value function has two parts V ( t) = 1 2 xt (t)s(t)x(t) + v( t) Certainty-equivalent value function V CE ( t) 1 2 xt (t)s(t)x(t) Stochastic value function increment v( t) = 1 2 t Tr S( τ )L( τ )W( τ )L τ T dτ 29 Value Function Gradient and Hessian Certainty-equivalent value function V CE ( t) 1 2 xt (t)s(t)x(t) Gradient with respect to the state V x t = xt (t)s(t) Hessian with respect to the state 2 V x 2 ( t) = S(t) 30
16 Linear-Quadratic Stochastic Hamilton-Jacobi-Bellman Equation (Perfect Measurements) V * t Certainty-equivalent plus stochastic terms = min u = min u 1 2 E x *T Qx *+2x * T Mu + u T Ru 1 2 x *T Qx *+2x * T Mu + u T Ru + x * T S Fx *+Gu + x * T S Fx *+Gu Terminal condition V ( ) = 1 2 xt ( )S( )x( ) + Tr SLWL T + Tr SLWL T 31 Optimal Control Law (Perfect Measurements) Differentiate right side of HJB equation w.r.t. u and set equal to zero ( V t ) u u t + x T SG = 0 = x T M + u T R Solve for u, obtaining feedback control law = R 1 ( t) G T t! C( t)x( t) S t + M T ( t) x( t) 32
17 LQ Optimal Control Law u t (Perfect Measurements) = R 1 ( t) G T t C( t)x( t) S t + M T ( t) x( t) Zero-mean, white-noise disturbance has no effect on the structure and gains of the LQ feedback control law 33 Matrix Riccati Equation for Control S t S t Substitute optimal control law in HJB equation 1 Sx 2 xt + v = 1 2 xt Q + MR 1 M T Matrix Riccati equation provides S(t) = Q(t) + M(t)R 1 (t)m T (t) F(t) G(t)R 1 (t)m T (t) ( F GR 1 M T ) T S S( F GR 1 M T ) + SGR 1 G T S + 1 Tr u ( SLWLT ) ( t) = R 1 t 2 G T t S t F(t) G(t)R 1 (t)m T T (t) S t + S( t)g(t)r 1 (t)g T (t)s( t), S + M T ( t) = φ xx ( )! Stochastic value function increases cost due to disturbance! However, its calculation is independent of the Riccati equation v = 1 2 Tr( SLWLT ) 34 x x t
18 Evaluation of the Total Cost (Imperfect Measurements)! Stochastic quadratic cosunction, neglecting cross terms J = 1 t f 2 Tr E xt ( )S( )x( ) + E xt (t) u T (t) t o Q(t) 0 0 R(t) x(t) u(t) dt = 1 2 Tr S(t )E x( f )xt ( ) + t o { Q(t)E x(t)xt (t) + R(t)E u(t)u T (t) }dt or where J = 1 2 Tr S( )P( ) + t o Q(t)P(t) + R(t)U( t) dt P(t) E x(t)x T (t) U t E u(t)u T (t) 35 Optimal Control Covariance U t = C t Optimal control vector = C( t) ˆx ( t) u t Optimal control covariance P( t)c T ( t) = R 1 ( t)g T ( t)s( t)p( t)s( t)g( t)r 1 t 36
19 Express Cost with State and Adjoint Covariance Dynamics J = 1 2 Tr S(t o)p t o Integration by parts to express S( )P( ) t S(t)P(t) f to = S(t)P(t) + S(t) P(t) dt S( )P( ) = S(t o )P(t o ) + + Q(t)P( t) + R(t)U( t) + S(t)P(t) + S(t) P(t) t o t o S(t)P(t) + S(t) P(t) dt Rewrite cosunction to incorporate initial cost t o dt 37 Evolution of State and Adjoint Covariance Matrices (No Control) u( t) = 0; U( t) = 0 State covariance response to random disturbance and initial uncertainty P ( t) = F( t)p( t) + P( t)f T ( t) + L( t)w( t)l T ( t), P( t o ) given Adjoint covariance response to state weighting and terminal cost weight S ( t) = F T ( t)s( t) S( t)f( t) Q( t), S given 38
20 P t Evolution of State and Adjoint Covariance Matrices (Optimal Control) = F( t) G t State covariance response to random disturbance and initial uncertainty C t P t + P( t) F( t) G t C t T + L( t)w( t)l T t Dependent on S(t) Adjoint covariance response to state weighting and terminal cost weight S ( t) = F T ( t)s( t) S( t)f( t) Q( t) S( t)g( t)r 1 ( t)g T ( t)s( t) Independent of P(t) 39 Total Cost With and Without Control With no control J no control = 1 t 2 Tr f S(t o)p( t o ) + S( t)l( t)w( t)l T ( t)dt t o With optimal control, the equation for the cost is the same J optimal control = 1 2 Tr S(t o)p t o + S t t o L( t)w( t)l T ( t)dt... but evolutions of S(t) and S(t o ) are different in each case 40
21 Information Sets and Expected Cost 41 The Information Set, I! Sigma algebra(wikipedia definitions)! The collection of sets over which a measure is defined! The collection of events that can be assigned probabilities! A measurable space! Information available at current time, t 1! All measurements from initial time, t o! All control commands from initial time [ ] = { z[ t o,t 1 ],u[ t o,t 1 ]} I t o,t 1! Plus available model structure, parameters, and statistics [ ] = { z[ t o,t 1 ],u[ t o,t 1 ],f ( ),Q,R, } I t o,t 1 42
22 A Derived Information Set, I D! Measurements may be directly useful, e.g.,! Displays! Simple feedback control!... or they may require processing, e.g.,! Transformation! Estimation! Example of a derived information set! History of mean and covariance from a state estimator [ ] = { ˆx [ t o,t 1 ],P[ t o,t 1 ],u[ t o,t 1 ]} I D t o,t 1 43 Additional Derived Information Sets! Markov derived information set! Most current mean and covariance from a state estimator I MD t 1 = ˆx t 1 {,P( t 1 ),u( t 1 )}! Multiple model derived information set! Parallel estimates of current mean, covariance, and hypothesis probability mass function = I MM t 1 ˆx A ( t 1 ),P A ( t 1 ),u( t 1 ),Pr( H A ), ˆx B ( t 1 ),P B ( t 1 ),u( t 1 ),Pr( H B ),! { } 44
23 Required and Available Information Sets for Optimal Control! Optimal control requires propagation of information back from the final time! Hence, it requires the entire information set, extending from t o to I t o,! Separate information set into knowable and future sets I t o, [ ] + I t 1, = I t o,t 1! Knowable set has been received! Future set is based on current knowledge of model, current estimates, and statistics of future uncertainties 45 Expected Values of State and Control Expected values of the state and control are conditioned on the information set E x( t) I D = ˆx ( t) E{ T x( t) ˆx ( t) x( t) ˆx ( t) I } D = P t... where the conditional expected values are estimates from an optimal filter 46
24 Dependence of the Stochastic Cost Function on the Information Set J = 1 t 2 E E Tr S( )x( )x T ( ) I D + E Tr Qx ( f { t xt t) }dt + E Tr Ru( t)u T { ( t) }dt 0 0 P t Expand the state covariance T { x( t) ˆx ( t) I D } = E x( t)x T ( t) ˆx ( t)x T ( t) x t = E x( t) ˆx ( t) { ˆx T ( t) + ˆx ( t) ˆx T ( t) I D } E { ˆx T ( t) I D } = E ˆx t x t { x T ( t) I D } = ˆx t ˆx T ( t) P t or E = E x t x( t)x T t { x T ( t) I D } ˆx t ˆx T ( t) { I D } = P( t) + ˆx ( t) ˆx T ( t)... where the conditional expected values are obtained from an optimal filter 47 Certainty-Equivalent and Stochastic Incremental Costs { + ˆx( )ˆx T ( ) } + Tr Q P t J = 1 2 E Tr S( ) P { + ˆx ( t) ˆx T ( t) }dt + Tr Ru( t)u T ( t) dt 0 0 J CE + J S! Cosunction has two parts! Certainty-equivalent cost! Stochastic increment cost J CE = 1 t 2 E Tr S( )ˆx( )ˆx T ( ) + Tr Qˆx ( t ) f { ˆx T ( t) }dt + Tr Ru( t)u T ( t) dt 0 0 J S = 1 2 E Tr S( )P + 0 Tr QP ( t ) dt 48
25 Expected Cost of the Trajectory V *( t o ) J * = E ξ I[ t o,t 1 ] E ξ E J * Optimized cosunction = E φ x *( ) Pr I t o,t 1 t F [ ] + L x *(τ ),u * (τ ) Law of total expectation { [ ]} + E ξ I t 1, = E E ( ξ I ) t 0 ( )Pr I t 1, dτ { } Because the past is established at t 1 ( [ ]) 1 = E( J* I[ t o,t 1 ]) + E J* I t 1, = E J* I t o,t 1 [ ]+ E( J* I t 1, )Pr I t 1, ( )Pr{ I t 1, } { } 49 Expected Cost of the Trajectory E J * E J* I( t 0,t 1 ) For planning = E J* I t 0, E J * For post-trajectory analysis (with perfect measurements) For real-time control, t 1 = E J * + E J* I( t 1, ) ( )Pr I( t 0, ) = E J* I t 0, { } Pr I t,t 0 f { } = E( J* I( t 0, )) 1 { } Pr I t,t 1 f [ ] Known Estimated 50
26 min u Dual Control (Fel dbaum, 1965)! Nonlinear system! Uncertain system parameters to be estimated! Parameter estimation can be aided by test inputs! Approach: Minimize value function with three increments! Nominal control! Cautious control! Probing control min u V* = ( V * nominal + V * cautious + V * probing ) Estimation and control calculations are coupled and necessarily recursive 51 Next Time: Linear-Quadratic-Gaussian Regulators 52
Linear-Quadratic-Gaussian Controllers!
Linear-Quadratic-Gaussian Controllers! Robert Stengel! Optimal Control and Estimation MAE 546! Princeton University, 2017!! LTI dynamic system!! Certainty Equivalence and the Separation Theorem!! Asymptotic
More informationStochastic Optimal Control!
Stochastic Control! Robert Stengel! Robotics and Intelligent Systems, MAE 345, Princeton University, 2015 Learning Objectives Overview of the Linear-Quadratic-Gaussian (LQG) Regulator Introduction to Stochastic
More informationDynamic Optimal Control!
Dynamic Optimal Control! Robert Stengel! Robotics and Intelligent Systems MAE 345, Princeton University, 2017 Learning Objectives Examples of cost functions Necessary conditions for optimality Calculation
More informationOPTIMAL CONTROL AND ESTIMATION
OPTIMAL CONTROL AND ESTIMATION Robert F. Stengel Department of Mechanical and Aerospace Engineering Princeton University, Princeton, New Jersey DOVER PUBLICATIONS, INC. New York CONTENTS 1. INTRODUCTION
More informationTime-Invariant Linear Quadratic Regulators!
Time-Invariant Linear Quadratic Regulators Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 17 Asymptotic approach from time-varying to constant gains Elimination of cross weighting
More informationTime-Invariant Linear Quadratic Regulators Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 2015
Time-Invariant Linear Quadratic Regulators Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 15 Asymptotic approach from time-varying to constant gains Elimination of cross weighting
More informationEN Applied Optimal Control Lecture 8: Dynamic Programming October 10, 2018
EN530.603 Applied Optimal Control Lecture 8: Dynamic Programming October 0, 08 Lecturer: Marin Kobilarov Dynamic Programming (DP) is conerned with the computation of an optimal policy, i.e. an optimal
More informationNonlinear State Estimation! Extended Kalman Filters!
Nonlinear State Estimation! Extended Kalman Filters! Robert Stengel! Optimal Control and Estimation, MAE 546! Princeton University, 2017!! Deformation of the probability distribution!! Neighboring-optimal
More informationFirst-Order Low-Pass Filter
Filters, Cost Functions, and Controller Structures Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 218! Dynamic systems as low-pass filters! Frequency response of dynamic systems!
More informationLinearized Equations of Motion!
Linearized Equations of Motion Robert Stengel, Aircraft Flight Dynamics MAE 331, 216 Learning Objectives Develop linear equations to describe small perturbational motions Apply to aircraft dynamic equations
More informationHamilton-Jacobi-Bellman Equation Feb 25, 2008
Hamilton-Jacobi-Bellman Equation Feb 25, 2008 What is it? The Hamilton-Jacobi-Bellman (HJB) equation is the continuous-time analog to the discrete deterministic dynamic programming algorithm Discrete VS
More informationSteady State Kalman Filter
Steady State Kalman Filter Infinite Horizon LQ Control: ẋ = Ax + Bu R positive definite, Q = Q T 2Q 1 2. (A, B) stabilizable, (A, Q 1 2) detectable. Solve for the positive (semi-) definite P in the ARE:
More informationOptimal Control. Quadratic Functions. Single variable quadratic function: Multi-variable quadratic function:
Optimal Control Control design based on pole-placement has non unique solutions Best locations for eigenvalues are sometimes difficult to determine Linear Quadratic LQ) Optimal control minimizes a quadratic
More informationFirst-Order Low-Pass Filter!
Filters, Cost Functions, and Controller Structures! Robert Stengel! Optimal Control and Estimation MAE 546! Princeton University, 217!! Dynamic systems as low-pass filters!! Frequency response of dynamic
More informationDeterministic Dynamic Programming
Deterministic Dynamic Programming 1 Value Function Consider the following optimal control problem in Mayer s form: V (t 0, x 0 ) = inf u U J(t 1, x(t 1 )) (1) subject to ẋ(t) = f(t, x(t), u(t)), x(t 0
More informationOptimal Control. Lecture 18. Hamilton-Jacobi-Bellman Equation, Cont. John T. Wen. March 29, Ref: Bryson & Ho Chapter 4.
Optimal Control Lecture 18 Hamilton-Jacobi-Bellman Equation, Cont. John T. Wen Ref: Bryson & Ho Chapter 4. March 29, 2004 Outline Hamilton-Jacobi-Bellman (HJB) Equation Iterative solution of HJB Equation
More informationOPTIMAL CONTROL. Sadegh Bolouki. Lecture slides for ECE 515. University of Illinois, Urbana-Champaign. Fall S. Bolouki (UIUC) 1 / 28
OPTIMAL CONTROL Sadegh Bolouki Lecture slides for ECE 515 University of Illinois, Urbana-Champaign Fall 2016 S. Bolouki (UIUC) 1 / 28 (Example from Optimal Control Theory, Kirk) Objective: To get from
More informationReturn Difference Function and Closed-Loop Roots Single-Input/Single-Output Control Systems
Spectral Properties of Linear- Quadratic Regulators Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 2018! Stability margins of single-input/singleoutput (SISO) systems! Characterizations
More informationLecture 10 Linear Quadratic Stochastic Control with Partial State Observation
EE363 Winter 2008-09 Lecture 10 Linear Quadratic Stochastic Control with Partial State Observation partially observed linear-quadratic stochastic control problem estimation-control separation principle
More informationIntroduction to Optimization!
Introduction to Optimization! Robert Stengel! Robotics and Intelligent Systems, MAE 345, Princeton University, 2017 Optimization problems and criteria Cost functions Static optimality conditions Examples
More informationOptimal Control and Estimation MAE 546, Princeton University Robert Stengel, Preliminaries!
Optimal Control and Estimation MAE 546, Princeton University Robert Stengel, 2018 Copyright 2018 by Robert Stengel. All rights reserved. For educational use only. http://www.princeton.edu/~stengel/mae546.html
More informationControlled Diffusions and Hamilton-Jacobi Bellman Equations
Controlled Diffusions and Hamilton-Jacobi Bellman Equations Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 2014 Emo Todorov (UW) AMATH/CSE 579, Winter
More informationQuadratic Stability of Dynamical Systems. Raktim Bhattacharya Aerospace Engineering, Texas A&M University
.. Quadratic Stability of Dynamical Systems Raktim Bhattacharya Aerospace Engineering, Texas A&M University Quadratic Lyapunov Functions Quadratic Stability Dynamical system is quadratically stable if
More informationAdvanced Mechatronics Engineering
Advanced Mechatronics Engineering German University in Cairo 21 December, 2013 Outline Necessary conditions for optimal input Example Linear regulator problem Example Necessary conditions for optimal input
More informationLecture 5 Linear Quadratic Stochastic Control
EE363 Winter 2008-09 Lecture 5 Linear Quadratic Stochastic Control linear-quadratic stochastic control problem solution via dynamic programming 5 1 Linear stochastic system linear dynamical system, over
More informationRobotics. Control Theory. Marc Toussaint U Stuttgart
Robotics Control Theory Topics in control theory, optimal control, HJB equation, infinite horizon case, Linear-Quadratic optimal control, Riccati equations (differential, algebraic, discrete-time), controllability,
More informationCHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME. Chapter2 p. 1/67
CHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME Chapter2 p. 1/67 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME Main Purpose: Introduce the maximum principle as a necessary condition to be satisfied by any optimal
More informationSolution of Stochastic Optimal Control Problems and Financial Applications
Journal of Mathematical Extension Vol. 11, No. 4, (2017), 27-44 ISSN: 1735-8299 URL: http://www.ijmex.com Solution of Stochastic Optimal Control Problems and Financial Applications 2 Mat B. Kafash 1 Faculty
More informationMinimization with Equality Constraints
Path Constraints and Numerical Optimization Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2015 State and Control Equality Constraints Pseudoinverse State and Control Inequality
More informationOptimal Control and Estimation MAE 546, Princeton University Robert Stengel, Preliminaries!
Optimal Control and Estimation MAE 546, Princeton University Robert Stengel, 2017 Copyright 2017 by Robert Stengel. All rights reserved. For educational use only. http://www.princeton.edu/~stengel/mae546.html
More informationLinear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters
Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 204 Emo Todorov (UW) AMATH/CSE 579, Winter
More information6.241 Dynamic Systems and Control
6.241 Dynamic Systems and Control Lecture 24: H2 Synthesis Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology May 4, 2011 E. Frazzoli (MIT) Lecture 24: H 2 Synthesis May
More informationTopic # Feedback Control Systems
Topic #17 16.31 Feedback Control Systems Deterministic LQR Optimal control and the Riccati equation Weight Selection Fall 2007 16.31 17 1 Linear Quadratic Regulator (LQR) Have seen the solutions to the
More informationStochastic optimal control theory
Stochastic optimal control theory Bert Kappen SNN Radboud University Nijmegen the Netherlands July 5, 2008 Bert Kappen Introduction Optimal control theory: Optimize sum of a path cost and end cost. Result
More informationA Tour of Reinforcement Learning The View from Continuous Control. Benjamin Recht University of California, Berkeley
A Tour of Reinforcement Learning The View from Continuous Control Benjamin Recht University of California, Berkeley trustable, scalable, predictable Control Theory! Reinforcement Learning is the study
More informationAdaptive State Estimation Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 2018
Adaptive State Estimation Robert Stengel Optimal Control and Estimation MAE 546 Princeton University, 218! Nonlinearity of adaptation! Parameter-adaptive filtering! Test for whiteness of the residual!
More informationLecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Process Approximations
Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Process Approximations Simo Särkkä Aalto University Tampere University of Technology Lappeenranta University of Technology Finland November
More informationTime Response of Dynamic Systems! Multi-Dimensional Trajectories Position, velocity, and acceleration are vectors
Time Response of Dynamic Systems Robert Stengel Robotics and Intelligent Systems MAE 345, Princeton University, 217 Multi-dimensional trajectories Numerical integration Linear and nonlinear systems Linearization
More informationNumerical Optimal Control Overview. Moritz Diehl
Numerical Optimal Control Overview Moritz Diehl Simplified Optimal Control Problem in ODE path constraints h(x, u) 0 initial value x0 states x(t) terminal constraint r(x(t )) 0 controls u(t) 0 t T minimize
More informationMinimization with Equality Constraints!
Path Constraints and Numerical Optimization Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2017 State and Control Equality Constraints Pseudoinverse State and Control Inequality
More informationOptimization-Based Control
Optimization-Based Control Richard M. Murray Control and Dynamical Systems California Institute of Technology DRAFT v1.7a, 19 February 2008 c California Institute of Technology All rights reserved. This
More informationCALIFORNIA INSTITUTE OF TECHNOLOGY Control and Dynamical Systems. CDS 110b
CALIFORNIA INSTITUTE OF TECHNOLOGY Control and Dynamical Systems CDS 110b R. M. Murray Kalman Filters 14 January 2007 Reading: This set of lectures provides a brief introduction to Kalman filtering, following
More informationControl Systems! Copyright 2017 by Robert Stengel. All rights reserved. For educational use only.
Control Systems Robert Stengel Robotics and Intelligent Systems MAE 345, Princeton University, 2017 Analog vs. digital systems Continuous- and Discretetime Dynamic Models Frequency Response Transfer Functions
More informationDepartment of Aerospace Engineering and Mechanics University of Minnesota Written Preliminary Examination: Control Systems Friday, April 9, 2010
Department of Aerospace Engineering and Mechanics University of Minnesota Written Preliminary Examination: Control Systems Friday, April 9, 2010 Problem 1: Control of Short Period Dynamics Consider the
More informationNonlinear State Estimation! Particle, Sigma-Points Filters!
Nonlinear State Estimation! Particle, Sigma-Points Filters! Robert Stengel! Optimal Control and Estimation, MAE 546! Princeton University, 2017!! Particle filter!! Sigma-Points Unscented Kalman ) filter!!
More informationNonlinear Observers. Jaime A. Moreno. Eléctrica y Computación Instituto de Ingeniería Universidad Nacional Autónoma de México
Nonlinear Observers Jaime A. Moreno JMorenoP@ii.unam.mx Eléctrica y Computación Instituto de Ingeniería Universidad Nacional Autónoma de México XVI Congreso Latinoamericano de Control Automático October
More informationObservability and state estimation
EE263 Autumn 2015 S Boyd and S Lall Observability and state estimation state estimation discrete-time observability observability controllability duality observers for noiseless case continuous-time observability
More informationHere represents the impulse (or delta) function. is an diagonal matrix of intensities, and is an diagonal matrix of intensities.
19 KALMAN FILTER 19.1 Introduction In the previous section, we derived the linear quadratic regulator as an optimal solution for the fullstate feedback control problem. The inherent assumption was that
More informationSYSTEMTEORI - KALMAN FILTER VS LQ CONTROL
SYSTEMTEORI - KALMAN FILTER VS LQ CONTROL 1. Optimal regulator with noisy measurement Consider the following system: ẋ = Ax + Bu + w, x(0) = x 0 where w(t) is white noise with Ew(t) = 0, and x 0 is a stochastic
More informationOverview of the Seminar Topic
Overview of the Seminar Topic Simo Särkkä Laboratory of Computational Engineering Helsinki University of Technology September 17, 2007 Contents 1 What is Control Theory? 2 History
More informationRobotics: Science & Systems [Topic 6: Control] Prof. Sethu Vijayakumar Course webpage:
Robotics: Science & Systems [Topic 6: Control] Prof. Sethu Vijayakumar Course webpage: http://wcms.inf.ed.ac.uk/ipab/rss Control Theory Concerns controlled systems of the form: and a controller of the
More informationAn efficient approach to stochastic optimal control. Bert Kappen SNN Radboud University Nijmegen the Netherlands
An efficient approach to stochastic optimal control Bert Kappen SNN Radboud University Nijmegen the Netherlands Bert Kappen Examples of control tasks Motor control Bert Kappen Pascal workshop, 27-29 May
More informationLecture 4 Continuous time linear quadratic regulator
EE363 Winter 2008-09 Lecture 4 Continuous time linear quadratic regulator continuous-time LQR problem dynamic programming solution Hamiltonian system and two point boundary value problem infinite horizon
More informationOptimal Control. McGill COMP 765 Oct 3 rd, 2017
Optimal Control McGill COMP 765 Oct 3 rd, 2017 Classical Control Quiz Question 1: Can a PID controller be used to balance an inverted pendulum: A) That starts upright? B) That must be swung-up (perhaps
More informationLecture 19 Observability and state estimation
EE263 Autumn 2007-08 Stephen Boyd Lecture 19 Observability and state estimation state estimation discrete-time observability observability controllability duality observers for noiseless case continuous-time
More informationSubject: Optimal Control Assignment-1 (Related to Lecture notes 1-10)
Subject: Optimal Control Assignment- (Related to Lecture notes -). Design a oil mug, shown in fig., to hold as much oil possible. The height and radius of the mug should not be more than 6cm. The mug must
More informationUCLA Chemical Engineering. Process & Control Systems Engineering Laboratory
Constrained Innite-time Optimal Control Donald J. Chmielewski Chemical Engineering Department University of California Los Angeles February 23, 2000 Stochastic Formulation - Min Max Formulation - UCLA
More informationPath integrals for classical Markov processes
Path integrals for classical Markov processes Hugo Touchette National Institute for Theoretical Physics (NITheP) Stellenbosch, South Africa Chris Engelbrecht Summer School on Non-Linear Phenomena in Field
More informationEE363 homework 2 solutions
EE363 Prof. S. Boyd EE363 homework 2 solutions. Derivative of matrix inverse. Suppose that X : R R n n, and that X(t is invertible. Show that ( d d dt X(t = X(t dt X(t X(t. Hint: differentiate X(tX(t =
More informationOptimal control and estimation
Automatic Control 2 Optimal control and estimation Prof. Alberto Bemporad University of Trento Academic year 2010-2011 Prof. Alberto Bemporad (University of Trento) Automatic Control 2 Academic year 2010-2011
More informationProblem 1 Cost of an Infinite Horizon LQR
THE UNIVERSITY OF TEXAS AT SAN ANTONIO EE 5243 INTRODUCTION TO CYBER-PHYSICAL SYSTEMS H O M E W O R K # 5 Ahmad F. Taha October 12, 215 Homework Instructions: 1. Type your solutions in the LATEX homework
More informationStatic and Dynamic Optimization (42111)
Static and Dynamic Optimization (421) Niels Kjølstad Poulsen Build. 0b, room 01 Section for Dynamical Systems Dept. of Applied Mathematics and Computer Science The Technical University of Denmark Email:
More informationCALIFORNIA INSTITUTE OF TECHNOLOGY Control and Dynamical Systems. CDS 110b
CALIFORNIA INSTITUTE OF TECHNOLOGY Control and Dynamical Systems CDS 110b R. M. Murray Kalman Filters 25 January 2006 Reading: This set of lectures provides a brief introduction to Kalman filtering, following
More informationSuppose that we have a specific single stage dynamic system governed by the following equation:
Dynamic Optimisation Discrete Dynamic Systems A single stage example Suppose that we have a specific single stage dynamic system governed by the following equation: x 1 = ax 0 + bu 0, x 0 = x i (1) where
More informationOptimal Control Theory
Optimal Control Theory The theory Optimal control theory is a mature mathematical discipline which provides algorithms to solve various control problems The elaborate mathematical machinery behind optimal
More informationLecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Approximations
Lecture 4: Numerical Solution of SDEs, Itô Taylor Series, Gaussian Approximations Simo Särkkä Aalto University, Finland November 18, 2014 Simo Särkkä (Aalto) Lecture 4: Numerical Solution of SDEs November
More informationIntroduction Optimality and Asset Pricing
Introduction Optimality and Asset Pricing Andrea Buraschi Imperial College Business School October 2010 The Euler Equation Take an economy where price is given with respect to the numéraire, which is our
More informationLinear Quadratic Zero-Sum Two-Person Differential Games Pierre Bernhard June 15, 2013
Linear Quadratic Zero-Sum Two-Person Differential Games Pierre Bernhard June 15, 2013 Abstract As in optimal control theory, linear quadratic (LQ) differential games (DG) can be solved, even in high dimension,
More informationMinimization of Static! Cost Functions!
Minimization of Static Cost Functions Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2017 J = Static cost function with constant control parameter vector, u Conditions for
More informationConditional Simulation of Random Fields by Successive Residuals 1
Mathematical Geology, Vol 34, No 5, July 2002 ( C 2002) Conditional Simulation of Random Fields by Successive Residuals 1 J A Vargas-Guzmán 2,3 and R Dimitrakopoulos 2 This paper presents a new approach
More informationFeedback Optimal Control of Low-thrust Orbit Transfer in Central Gravity Field
Vol. 4, No. 4, 23 Feedback Optimal Control of Low-thrust Orbit Transfer in Central Gravity Field Ashraf H. Owis Department of Astronomy, Space and Meteorology, Faculty of Science, Cairo University Department
More informationStatic and Dynamic Optimization (42111)
Static and Dynamic Optimization (42111) Niels Kjølstad Poulsen Build. 303b, room 016 Section for Dynamical Systems Dept. of Applied Mathematics and Computer Science The Technical University of Denmark
More informationStochastic Differential Dynamic Programming
Stochastic Differential Dynamic Programming Evangelos heodorou, Yuval assa & Emo odorov Abstract Although there has been a significant amount of work in the area of stochastic optimal control theory towards
More informationSUB-OPTIMAL RISK-SENSITIVE FILTERING FOR THIRD DEGREE POLYNOMIAL STOCHASTIC SYSTEMS
ICIC Express Letters ICIC International c 2008 ISSN 1881-803X Volume 2, Number 4, December 2008 pp. 371-377 SUB-OPTIMAL RISK-SENSITIVE FILTERING FOR THIRD DEGREE POLYNOMIAL STOCHASTIC SYSTEMS Ma. Aracelia
More informationUCLA Chemical Engineering. Process & Control Systems Engineering Laboratory
Constrained Innite-Time Nonlinear Quadratic Optimal Control V. Manousiouthakis D. Chmielewski Chemical Engineering Department UCLA 1998 AIChE Annual Meeting Outline Unconstrained Innite-Time Nonlinear
More informationFinite-Horizon Optimal State-Feedback Control of Nonlinear Stochastic Systems Based on a Minimum Principle
Finite-Horizon Optimal State-Feedbac Control of Nonlinear Stochastic Systems Based on a Minimum Principle Marc P Deisenroth, Toshiyui Ohtsua, Florian Weissel, Dietrich Brunn, and Uwe D Hanebec Abstract
More informationEE C128 / ME C134 Final Exam Fall 2014
EE C128 / ME C134 Final Exam Fall 2014 December 19, 2014 Your PRINTED FULL NAME Your STUDENT ID NUMBER Number of additional sheets 1. No computers, no tablets, no connected device (phone etc.) 2. Pocket
More informationLecture 9. Introduction to Kalman Filtering. Linear Quadratic Gaussian Control (LQG) G. Hovland 2004
MER42 Advanced Control Lecture 9 Introduction to Kalman Filtering Linear Quadratic Gaussian Control (LQG) G. Hovland 24 Announcement No tutorials on hursday mornings 8-9am I will be present in all practical
More informationECE7850 Lecture 7. Discrete Time Optimal Control and Dynamic Programming
ECE7850 Lecture 7 Discrete Time Optimal Control and Dynamic Programming Discrete Time Optimal control Problems Short Introduction to Dynamic Programming Connection to Stabilization Problems 1 DT nonlinear
More informationChapter 3. LQ, LQG and Control System Design. Dutch Institute of Systems and Control
Chapter 3 LQ, LQG and Control System H 2 Design Overview LQ optimization state feedback LQG optimization output feedback H 2 optimization non-stochastic version of LQG Application to feedback system design
More informationBayesian Decision Theory in Sensorimotor Control
Bayesian Decision Theory in Sensorimotor Control Matthias Freiberger, Martin Öttl Signal Processing and Speech Communication Laboratory Advanced Signal Processing Matthias Freiberger, Martin Öttl Advanced
More informationLinear Quadratic Optimal Control Topics
Linear Quadratic Optimal Control Topics Finite time LQR problem for time varying systems Open loop solution via Lagrange multiplier Closed loop solution Dynamic programming (DP) principle Cost-to-go function
More informationInfinite-dimensional nonlinear predictive controller design for open-channel hydraulic systems
Infinite-dimensional nonlinear predictive controller design for open-channel hydraulic systems D. Georges, Control Systems Dept - Gipsa-lab, Grenoble INP Workshop on Irrigation Channels and Related Problems,
More informationMin-Max Output Integral Sliding Mode Control for Multiplant Linear Uncertain Systems
Proceedings of the 27 American Control Conference Marriott Marquis Hotel at Times Square New York City, USA, July -3, 27 FrC.4 Min-Max Output Integral Sliding Mode Control for Multiplant Linear Uncertain
More informationLINEAR QUADRATIC GAUSSIAN
ECE553: Multivariable Control Systems II. LINEAR QUADRATIC GAUSSIAN.: Deriving LQG via separation principle We will now start to look at the design of controllers for systems Px.t/ D A.t/x.t/ C B u.t/u.t/
More informationContinuous Time Finance
Continuous Time Finance Lisbon 2013 Tomas Björk Stockholm School of Economics Tomas Björk, 2013 Contents Stochastic Calculus (Ch 4-5). Black-Scholes (Ch 6-7. Completeness and hedging (Ch 8-9. The martingale
More informationCDS 110b: Lecture 2-1 Linear Quadratic Regulators
CDS 110b: Lecture 2-1 Linear Quadratic Regulators Richard M. Murray 11 January 2006 Goals: Derive the linear quadratic regulator and demonstrate its use Reading: Friedland, Chapter 9 (different derivation,
More informationGame Theoretic Continuous Time Differential Dynamic Programming
Game heoretic Continuous ime Differential Dynamic Programming Wei Sun, Evangelos A. heodorou and Panagiotis siotras 3 Abstract In this work, we derive a Game heoretic Differential Dynamic Programming G-DDP
More informationOn Stochastic Adaptive Control & its Applications. Bozenna Pasik-Duncan University of Kansas, USA
On Stochastic Adaptive Control & its Applications Bozenna Pasik-Duncan University of Kansas, USA ASEAS Workshop, AFOSR, 23-24 March, 2009 1. Motivation: Work in the 1970's 2. Adaptive Control of Continuous
More informationMark Gales October y (x) x 1. x 2 y (x) Inputs. Outputs. x d. y (x) Second Output layer layer. layer.
University of Cambridge Engineering Part IIB & EIST Part II Paper I0: Advanced Pattern Processing Handouts 4 & 5: Multi-Layer Perceptron: Introduction and Training x y (x) Inputs x 2 y (x) 2 Outputs x
More information3 Gramians and Balanced Realizations
3 Gramians and Balanced Realizations In this lecture, we use an optimization approach to find suitable realizations for truncation and singular perturbation of G. It turns out that the recommended realizations
More informationLinear-Quadratic Optimal Control: Full-State Feedback
Chapter 4 Linear-Quadratic Optimal Control: Full-State Feedback 1 Linear quadratic optimization is a basic method for designing controllers for linear (and often nonlinear) dynamical systems and is actually
More informationLQR and H 2 control problems
LQR and H 2 control problems Domenico Prattichizzo DII, University of Siena, Italy MTNS - July 5-9, 2010 D. Prattichizzo (Siena, Italy) MTNS - July 5-9, 2010 1 / 23 Geometric Control Theory for Linear
More informationOptimization-Based Control
Optimization-Based Control Richard M. Murray Control and Dynamical Systems California Institute of Technology DRAFT v1.7b, 23 February 2008 c California Institute of Technology All rights reserved. This
More informationKalman Filter. Man-Wai MAK
Kalman Filter Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University enmwmak@polyu.edu.hk http://www.eie.polyu.edu.hk/ mwmak References: S. Gannot and A. Yeredor,
More informationEL2520 Control Theory and Practice
EL2520 Control Theory and Practice Lecture 8: Linear quadratic control Mikael Johansson School of Electrical Engineering KTH, Stockholm, Sweden Linear quadratic control Allows to compute the controller
More informationAlternative Expressions for the Velocity Vector Velocity restricted to the vertical plane. Longitudinal Equations of Motion
Linearized Longitudinal Equations of Motion Robert Stengel, Aircraft Flig Dynamics MAE 33, 008 Separate solutions for nominal and perturbation flig paths Assume that nominal path is steady and in the vertical
More informationNonlinear and robust MPC with applications in robotics
Nonlinear and robust MPC with applications in robotics Boris Houska, Mario Villanueva, Benoît Chachuat ShanghaiTech, Texas A&M, Imperial College London 1 Overview Introduction to Robust MPC Min-Max Differential
More informationLinear Systems. Manfred Morari Melanie Zeilinger. Institut für Automatik, ETH Zürich Institute for Dynamic Systems and Control, ETH Zürich
Linear Systems Manfred Morari Melanie Zeilinger Institut für Automatik, ETH Zürich Institute for Dynamic Systems and Control, ETH Zürich Spring Semester 2016 Linear Systems M. Morari, M. Zeilinger - Spring
More informationComputational Issues in Nonlinear Dynamics and Control
Computational Issues in Nonlinear Dynamics and Control Arthur J. Krener ajkrener@ucdavis.edu Supported by AFOSR and NSF Typical Problems Numerical Computation of Invariant Manifolds Typical Problems Numerical
More information