Linear-Quadratic-Gaussian Controllers!

Linear-Quadratic-Gaussian Controllers! Robert Stengel! Optimal Control and Estimation MAE 546! Princeton University, 2017!! LTI dynamic system!! Certainty Equivalence and the Separation Theorem!! Asymptotic stability of the constant-gain LQG regulator!! Coupling due to parameter uncertainty!! Robustness (loop transfer) recovery!! Stochastic Robustness Analysis and Design!! Neighboring-optimal Stochastic Control Copyright 2017 by Robert Stengel. All rights reserved. For educational use only. http://www.princeton.edu/~stengel/mae546.html http://www.princeton.edu/~stengel/optconest.html 1 The Problem: Minimize Cost, Subject to Dynamic Constraint, Uncertain Disturbances, and Measurement Error = Fx(t) + Gu(t) + Lw(t), x( 0) = x o z( t) = Hx( t) + n( t)!x t Dynamic System minv t o u = min u J ( t f ) Cost Function = 1 2 min u ( t f E * ( E! " x T (t )S(t )x(t ) I f f f D + E * ) )! ' " xt (t) u T (t) 0 +* +*! " Q 0 0 R! " x(t) u(t) dt I D,, ** --.*.* I D { } ( t) = ˆx ( t),p( t),u t 2

Initial Conditions and Dimensions E x( 0) { ˆx ( 0 ) ˆx ( 0 ) } = P( 0)!" w T ( ) = W' ( t ) T!" = ˆx o ; E!" x 0!" x 0 E w( t)!" = 0; E w t E n( t)!" = 0; E!" n( t)n T ( ) = N' t E!" w( t)n T ( ) = 0 dim!" x t = n 1 dim!" u t = m 1 dim!" w t = s 1 Dimensions dim!" z t = dim! " n t = r 1 Statistics 3 Linear-Quadratic-Gaussian Control 4

Separation Property and Certainty Equivalence Separation Property! Optimal Control Law and Optimal Estimation Law can be derived separately! Their derivations are strictly independent Certainty Equivalence Property! Separation property plus,...! The Stochastic Optimal Control Law and the Deterministic Optimal Control Law are the same! The Optimal Estimation Law can be derived separately Linear-quadratic-Gaussian (LQG) control is certainty-equivalent 5 The Equations (Continuous-Time Model) System State and Measurement Control Law = F(t)x(t) + G(t)u(t) + L(t)w(t) z( t) = Hx( t) + n( t)!x t u( t) =!C(t)ˆx(t) + C F (t)y C (t) State Estimate Estimator Gain and State Covariance Estimate = F(t)ˆx(t) + G(t)u(t) + K( t) z( t)! Hˆx ( t)! K( t)h ˆ!x t " = " F(t)! G(t)C t ˆx(t) + G( t)c F (t)y C (t) + K( t)z t = P(t)H T N!1 ( t) W( t)l T ( t)! P(t)H T N!1 t K t!p(t) = F(t)P(t) + P(t)F T (t) + L t Control Gain and Adjoint Covariance Estimate!S t C( t) = R!1 t =!Q(t)! F(t) T S( t)! S t G T ( t)s( t) F(t) + S( t)g(t)r!1 (t)g T (t)s t HP(t) 6

Estimator in the Feedback Loop Linear-Gaussian (LG) state estimator adds dynamics to the feedback signal =!C(t)ˆx(t) u t ˆ!x ( t) = Fˆx(t) + Gu(t) + K( t) z( t)! Hˆx ( t) " Thus, state estimator can be viewed as a control-law compensator Bandwidth of the compensation is dependent on the multivariable signal/noise ratio, [PH T ]N 1 K( t) = P(t)H T N!1!P(t) = FP(t) + P(t)F T + LWL T! P(t)H T N!1 HP(t) 7 Stable Scalar LTI Example of Estimator Compensation Dynamic system (stable) and measurement!x =!x + w; z = Hx + n Estimator differential equation Laplace transform of estimator Estimator transfer function Low-pass filter ˆ!x =! ˆx + K ( z! Hˆx ) = (!1! KH ) ˆx + Kz " s! (!1! KH ) ˆx ( s) = Kz( s) ˆx ( s) = ˆx s z s K z s " s!!1! KH = K s! (!1! KH ) " 8

Unstable Scalar LTI Example of Estimator Compensation Dynamic system (unstable) and measurement!x = x + w; z = Hx + n Estimator differential equation Laplace transform of estimator Estimator transfer function Low-pass filter ˆ!x = ˆx + K ( z! Hˆx ) = ( 1! KH ) ˆx + Kz " s! ( 1! KH ) ˆx ( s) = Kz( s) ˆx ( s) = ˆx s K " s! ( 1! KH ) z( s) = K s! ( 1! KH ) z s " 9 Steady-State Scalar Filter Gain P = Signal "Power" = State Estimate Variance N = Noise "Power" = Measurement Error Variance H = Projection from Measurement Space to Signal Space W = Disturbance Covariance Constant, scalar filter gain K = PH N 0 = 2P + W! P2 H 2 Algebraic Riccati equation (unstable case) P = N H ± 2! N " H 2 N ; P2! 2N H 2 P! WN H 2 = 0 2 + WN = N ' 1 ± 1+ WH 2 ) H 2 H 2 () N *, +, 10

Steady-State Filter Gain K = ') N! 1 + 1 + WH 2 ( H 2 " N *) N + ), H -) ') = ( 1 H *)! 2 WH 1+ 1+ " N + ), -) K! W!! >> N " W N 11 Dynamic Constraint on the Certainty-Equivalent Cost P(t) is independent of u(t); therefore min u J = min u J CE + J S J CE is Identical in form to the deterministic cost function Minimized subject to dynamic constraint based on the state estimate ˆ!x ( t) = Fˆx(t) + Gu(t) + K( t) z( t)! Hˆx ( t) " 12

Kalman-Bucy Filter Provides Estimate of the State Mean Value Filter residual is a Gaussian process = Fˆx(t) + Gu(t) + K( t) z( t)! Hˆx ( t) " Fˆx(t) + Gu(t) + K( t) ( t) ˆ!x t " Filter equation is analogous to deterministic dynamic constraint on deterministic cost function!x ( t) = Fx(t) + Gu(t) + Lw(t) 13 Control That Minimizes the Certainty-Equivalent Cost Optimizing control history is generated by a time-varying feedback control law =!C(t)ˆx(t) u t The control gain is the same as the deterministic gain = R!1 G T S( t) C t!s ( t) =!Q! F T S( t)! S t F + S( t)gr!1 G T S t given S t f 14

Optimal Cost for the Continuous- Time LQG Controller Certainty-equivalent cost J CE = 1 t 2 Tr! f S(0)E!" ˆx ( 0) ˆx T ( 0) + S( t)k( t)nk T ( t)dt ' " 0 ' Total cost J = J CE + J S = 1 t 2 Tr f ( S(0)E *!" ˆx ( 0) ˆx T ( 0) + S( t)k( t)nk T ( ' ( t)dt + )( 0,( + 1 t 2 Tr! f S(t )P(t ) + QP( t)dt - f f. "- 0. 15 Discrete-Time LQG Controller Kalman filter produces state estimate ˆx k (!) = "ˆx k!1 ( + ) + C k!1ˆx k!1 ( + ) ˆx k ( + ) = ˆx k (!) + K k z k! Hˆx k (!) " Closed-loop system uses state estimate for feedback control x k +1 =!x k " C k ˆx k + 16

Response of Discrete-Time 1 st -Order System to Disturbance Kalman Filter Estimate from Noisy Measurement Propagation of Uncertainty Kalman Filter, Uncontrolled System 17 Comparison of 1 st -Order Discrete- Time LQ and LQG Control Response Linear-Quadratic Control with Noise-free Measurement Linear-Quadratic-Gaussian Control with Noisy Measurement 18

Asymptotic Stability of the LQG Regulator! 19 System Equations with LQG Control With perfect knowledge of the system!x ( t) = Fx(t) + Gu(t) + Lw(t) = Fˆx(t) + Gu(t) + K( t) z( t)! Hˆx ( t) ˆ!x t State estimate error!( t) = x( t) " ˆx ( t) " State estimate error dynamics!( t) = ( F " KH)! ( t) + Lw t " Kn( t) 20

Control-Loop and Estimator Eigenvalues are Uncoupled "!x t! t " ' ' = ( F ( GC) GC 0 F ( KH " ' ' x t! t ' ' + " L 0 L (K " ' w t n t ' ' Upper-block-triangular stability matrix LQG system is stable because (F GC) is stable (F KH) is stable Estimate error affects state response = F! GC!x t x t + GC" ( t) + Lw( t) Actual state does not affect error response Disturbance affects both equally 21 Parameter Uncertainty Introduces Coupling! 22

Coupling Due To Parameter Uncertainty Actual System: { F A,G A,H A } Assumed System: { F,G,H}!x ( t) = F A x(t) + G A u(t) + Lw(t) = Fˆx(t) + Gu(t) + K( t) z( t)! Hˆx ( t) ˆ!x t " = H A x(t) + n( t) u(t) =!C( t) ˆx ( t) z t "!x t! t " ' ' = ( F A ( G A C) G A C " ( F A ( F) ( ( G A ( G)C ( K( H A ( H) " F + ( G A ( G)C ( KH " ' ' x t! t ' ' +" Closed-loop control and estimator responses are coupled 23 Effects of Parameter Uncertainty on Closed-Loop Stability si 2n! F CL = " si n! ( F A! G A C)!G A C! ( G A! G)C! K( H A! H) { }! " F A! F si n! " F + ( G A! G)C! KH = CL ( s) = 0!! Uncertain parameters affect closed-loop eigenvalues!! Coupling can lead to instability for numerous reasons!! Improper control gain!! Control effect on estimator block!! Redistribution of damping 24

Doyle s Counter-Example of LQG Robustness (1978)! " Unstable Plant!x 1!x 2 =! 1 1 " 0 1! " z =! " 1 0 x 1 x 2! " +! 0 " 1 x 1 x 2 + n! u + 1 " 1 w Design Matrices! Q = Q 1 1 " 1 1! ; R = 1; W = W 1 1 " 1 1 ; N = 1 Control and Estimator Gains C = ( 2 + 4 + Q )! " 1 1! 1 K = ( 2 + 4 + W ) " 1 =! " c c! k = " k 25 Uncertainty in the Control Effect System Matrices! F A = F; G A = " 0 µ ; H A = H Characteristic Equation ( s!1)!1 0 0 µc µc!1 ( s!1+ c) 0 s!1!k 0 s!1+ k!k 0 c + k = 0 s 4 + a 3 s 3 + a 2 s 2 + a 1 s + a 0 =! CL ( s) = 0 26

Stability Effect of Parameter Variation Routh's Stability Criterion (necessary condition) All coefficients of!(s) must be positive for stability µ is nominally equal to 1 µ can force a 0 and a 1 to change sign Result is dependent on magnitude of ck a 1 = k + c! 4 + 2 ( µ!1)ck a 0 = 1+ ( 1! µ )ck Arbitrarily small uncertainty, µ = 1 +!, could cause unstability Not surprising: uncertainty is in the control effect 27 The Counter-Example Raises a Flag Solution Choose Q and W to be small, increasing allowable range of "!! However,... The counter-example is irrelevant because it does not satisfy the requirements for LQ and LG stability!! The open-loop system is unstable, so it requires feedback control to restore stability!! To guarantee stability, Q and W must be positive definite, but! Q = Q 1 1 " 1 1! W = W 1 1 " 1 1 ; hence, Q = 0 ; hence, W = 0 28

Restoring Robustness! (Loop Transfer Recovery)! 29 Loop-Transfer Recovery (Doyle and Stein, 1979)!! Proposition: LQG and LQ robustness would be the same if the control vector had the same effect on the state and its estimate Cx( t) and Cˆx t produce same expected value of control, E!" u( t) E " u LQ t but not the same! u LQG ( t) u ( t)! u ( t) T { " LQ LQG } as ˆx ( t) contains measurement errors but x( t) does not!! Therefore, restoring the correct mean value from z(t) restores closed-loop robustness!! Solution: Increase the assumed process noise for estimator design as follows (see text for details) W = W o + k 2 GG T 30

Stochastic Robustness Analysis and Design! 31 Expression of Uncertainty in the System Model System uncertainty may be expressed as Elements of F Coefficients of!(s) Eigenvalues, " Frequency response/singular values/time response, A(j ), (j ), x(t) Variation may be! Deterministic, e.g., Upper/lower bounds ( worst-case )! Probabilistic, e.g., Gaussian distribution Bounded variation is equivalent to probabilistic variation with uniform distribution 32

Stochastic Root Locus: Uncertain Damping Ratio and Natural Frequency Gaussian Distribution of Eigenvalues Uniform Distribution of Eigenvalues 33 Probability of Instability Nonlinear mapping from probability density functions (pdf) of uncertain parameters to pdf of roots! Finite probability of instability with Gaussian (unbounded) distribution! Zero probability of instability for some uniform distributions! 34

Probabilistic Control Design Design constant-parameter controller (CPC) for satisfactory stability and performance in an uncertain environment Monte Carlo Evaluation of simulated system response with! competing CPC designs [Design parameters = d]! given statistical model of uncertainty in the plant [Uncertain plant parameters = v] Search for best CPC! Exhaustive search! Random search! Multivariate line search! Genetic algorithm! Simulated annealing 35 Design Outcome Follows Binomial Distribution Binomial distribution: Satisfactory/Unsatisfactory Confidence intervals of probability estimate are functions of! Actual probability! Number of trials Number of Evaluations 1e+09 1e+07 1e+05 1e+03 Number of Evaluations Confidence Interval Interval Width 2 5 10 20 100 1e+01 1e-06 1e-05 1e-04 0.001 0.01 0.1 1 0.5 Probability or (1 - Probability Binomial Distribution Maximum Information Entropy when Pr = 0.5 36

Example: Probability of Stable Control of an Unstable Plant X-29 Aircraft F = Longitudinal dynamics for a Forward-Swept-Wing Airplane!2gf 11 /V "V 2 f 12 / 2 "Vf 13!g!45 /V 2 "Vf 22 / 2 1 0 0 "V 2 f 32 / 2 "Vf 33 0 0 0 1 0 ( ( ( ( ( '! G = " g 11 g 12 0 0 g 31 g 32 0 0! ; x = " V, Airspeed ', Angle of attack q, Pitch rate (, Pitch angle Ray, Stengel, 1991 37 Example: Probability of Stable Control of an Unstable Plant Nominal eigenvalues (one unstable)! 1" 4 = "0.1 ± 0.057 j, " 5.15, 3.35 Air density and airspeed,! and V, have uniform distributions(±30) 10 coefficients have Gaussian distributions (! = 30) p = "! V f 11 f 12 f 13 f 22 f 32 f 33 g 11 g 12 g 31 g 32 T Environment Uncontrolled Dynamics Control Effect 38

LQ Regulators for the Example Three stabilizing feedback control laws Case a) LQR with low control weighting Q = diag( 1,1,1,0 ); R = ( 1,1);! 1"4nominal = 35, 5.1, 3.3,.02 C = 0.17 130 33 0.36 0.98 "11 "3 "1.1 Case b) LQR with high control weighting Q = diag( 1,1,1,0 ); R = ( 1000,1000);! 1"4nominal = "5.2,"3.4,"1.1,".02 C = 0.03 83 21 "0.06 0.01 "63 "16 "1.9 Case c) Case b with gains multiplied by 5 for bandwidth (loop-transfer) recovery! 1" 4nominal = "32, 5.2, 3.4, 0.01 C = ( ' ( ' 0.13 413 105 "0.32 0.05 "313 "81 "1.1" 9.5 ( ' 39 Stochastic Root Loci!! Distribution of closed-loop roots with!! Gaussian uncertainty in 10 parameters!! Uniform uncertainty in velocity and air density!! 25,000 Monte Carlo evaluations!! Probability of instability!! a) Pr = 0.072!! b) Pr = 0.021!! c) Pr = 0.0076 Case a Case b Case c Stochastic Root Locus Ray, Stengel, 1991 40

Probabilities of Instability for the Three Cases 41 Stochastic Root Loci for the Three Cases Case a: Low LQ Control Weights with Gaussian Aerodynamic Uncertainty! Case b: High LQ Control Weights Probabilities of instability with 30 uniform aerodynamic uncertainty!! Case a: 3.4 x 10-4!! Case b: 0!! Case c: 0! Case c: Bandwidth Recovery 42

American Control Conference Benchmark Control Problem, 1991 Parameters of 4 th -order mass-spring system! Uniform probability density functions for 0.5 < m 1, m 2 < 1.5 0.5 < k < 2 Probability of Instability, P i! m i = 1 (unstable) or 0 (stable) Probability of Settling Time Exceedance, P ts! m ts = 1 (exceeded) or 0 (not exceeded) Probability of Control Limit Exceedance, P u! m u = 1 (exceeded) or 0 (not exceeded) Design Cost Function 10 controllers designed for the competition J = ap i 2 + bp ts 2 + cp u 2 43 Stochastic LQG Design for Benchmark Control Problem SISO Linear-Quadratic-Gaussian Regulators (Marrison)! Implicit model following with control-rate weighting and scalar output (5 th order)! Kalman filter with single measurement (4 th order)! Design parameters Control cost function weights Springs and masses in ideal model Estimator weights! Search Multivariate line search Genetic algorithm 44

Comparison of Design Costs for Benchmark Control Problem J = P i 2 + 0.01P ts 2 + 0.01P u 2 Cost Emphasizes Instability J = 0.01P i 2 + 0.01P ts 2 + P u 2 Cost Emphasizes Excess Control J = 0.01P i 2 + P ts 2 + 0.01P u 2 Cost Emphasizes Settling-Time Exceedance Competition Designs Four SRAD LQG Designs Marrison, Stengel, 1995 45 Stochastic LQG controller more robust in 39 of 40 benchmark controller comparisons Estimation of Minimum Design Cost Using Jackknife/Bootstrapping Evaluation 1 0.9 0.8 Corresponding Weibull Distribution Pr(J < J0) 0.7 0.6 0.5 0.4 0.3 Genetic Algorithm Results 0.2 0.1 0 2.24 2.25 2.26 2.27 2.28 2.29 2.3 2.31 2.32 Design Cost Estimate x 103 Marrison, Stengel, 1995 46

Neighboring-Optimal Control with Uncertain Disturbance, Measurement, and Initial Condition! 47 Immune Response Example!! Optimal open-loop drug therapy (control)!! Assumptions!! Initial condition known without error!! No disturbance!! Optimal closed-loop therapy!! Assumptions!! Small error in initial condition!! Small disturbance!! Perfect measurement of state!! Stochastic optimal closed-loop therapy!! Assumptions!! Small error in initial condition!! Small disturbance!! Imperfect measurement!! Certainty-equivalence applies to perturbation control 48

Immune Response Example with Optimal Feedback Control Open-Loop Optimal Control for Lethal Initial Condition Open- and Closed-Loop Optimal Control for 150 Lethal Initial Condition Ghigliazza, Kulkarni, Stengel, 2002 49 Immune Response with Full-State Stochastic Optimal Feedback Control (Random Disturbance and Measurement Error Not Simulated) Low-Bandwidth Estimator ( W < N ) High-Bandwidth Estimator ( W > N )!! Initial control too sluggish to prevent divergence Ghigliazza, Stengel, 2004!! Quick initial control prevents divergence 50

Stochastic-Optimal Control (u 1 ) with Two Measurements (x 1, x 3 ) W = I 4 N = I 2 / 20 51 Immune Response to Random Disturbance with Two-Measurement Stochastic Neighboring-Optimal Control Disturbance due to! Re-infection! Sequestered pockets of pathogen Noisy measurements Closed-loop therapy is robust... but not robust enough:! Organ death occurs in one case Probability of satisfactory therapy can be maximized by stochastic redesign of controller 52

The End! Thank you! 53