Constrained Innite-time Optimal Control Donald J. Chmielewski Chemical Engineering Department University of California Los Angeles February 23, 2000
Stochastic Formulation - Min Max Formulation - UCLA Chemical Engineering Outline Background and Motivation - Objectives of Constrained Control - Finite-time Predictive Control Constrained Innite-time Optimal Control - Advantages of Innite-time Formulation - Finite-time Solution Method Constrained Disturbance Attenuation
From FCC Feed Rate (MV1+CV1) Vent Position (MV3) O 2 out (CV) CO out (CV8) Naphtha Hydrotreating Process Reactor Temperature (CV4) From Column Recycle Ratio (MV2 + CV2) Heat Exchanger Furnace Hydrotreating Reactors Fuel Feed Feed Temp Loop (MV2) MV4 Hydrogen Feed H2 Sep Clean Product
Modern Control System Architecture Real Time Optimizer (Steady State Calculations) Predictive Control Predictive Control Predictive Control PID Servo Loops Servo Loops Servo Loops PI PI PID PI PID PI PID PI Process 1 Process 2 Process 3
Objectives of Real-time Optimization Min Feed Rate Operator Optimum Max * Reactor Temperature Target Steady State Expected Domain of Operation Min Reactor Temperature * * Max CO Emissions Economic Optimum Max Feed Rate
Constrained Set Point Tracking Min Feed Rate Max Reactor Temperature Min Reactor Temperature Old Target * * OldOptimum Max CO Emissions Max Feed Rate NewTarget * * NewOptimum
Design a Feedback Controller so that the process follows - given Set Point Trajectory the UCLA Chemical Engineering Set Point Tracking y k (sp) y k + - C u k P
PI Control System Design time manipulated variable manipulated variable control variable control variable time time time Time Constant () : Small Time Constant () : Large PI Gain (K): Small PI Gain (K): Large
control variable 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 control variable 000 111 000 111 000 111 000000 111111 Quadratic Optimal Control Design time time manipulated variable 000 111 000 111 manipulated variable 000000 111111 000000 111111 0000 1111 0000 1111 0000 1111 time time Output Concern (Q oc ) : Small Output Concern (Q oc ) : Large Move Suppression (R ms ) : Large Move Suppression (R ms ) : Small
umin y k x k k 8 >< >: NX Model: x k+1 = Ax k + Bu k 0 < k < N Process k = Cx k 0 k N y 9 >= > UCLA Chemical Engineering Linear Quadratic Optimal Control Problem Formulation k y ) T N;1 X Q oc (y k ; y tgt ) + tgt ; (y (u k ; u tgt ) T R ms (u k ; u tgt ) k=0 k=0 subject to Input Constraints: u u k u 0 k < N Output Constraints: y y k y 0 < k N
min k ~x k ~u 8 >< >: ~x T k ~ Q oc ~x k + ~u T k R ms ~u k! 9 > T = Q oc ~x N > ~ N ~x + UCLA Chemical Engineering Constrained LQ Optimal Control X N;1 k=0 subject to Model: ~x k+1 = A~x k + B ~u k 0 < k < N Process 0 = ;x tgt ~x Input Constraints: u ~u k u 0 k < N Output Constraints: y C ~x k y 0 < k N where: ~ Q oc = C T Q oc C, x tgt = Ax tgt + Bu tgt, y tgt = Cx tgt, and k = x k ; x tgt y = y ; y tgt u = u ; u tgt ~x k = u k ; u tgt y = y ; y tgt u = u ; u tgt ~u
Trajectory Predictions vs. Actual Present Past Future Predicted Output y k Actual Output Δt 2Δt 3Δt 4Δt 5Δt time u k Predicted Input
^P - Process Model P - Actual Process P = P ; ^P UCLA Chemical Engineering Model Mismatch y k (sp) + - P C + + u k ΔP y k
Predictive Control Present Past Future New Predicted Output y k Δt 2Δt 3Δt 4Δt 5Δt time u k New Predicted Input
Set Point Change in Hydrotreating Process From FCC Feed Rate (MV1+CV1) Vent Position (MV3) O 2 out (CV) CO out (CV8) Reactor Temperature (CV4) From Column Recycle Ratio (MV2 + CV2) Heat Exchanger Furnace Hydrotreating Reactors Fuel Feed Feed Temp Loop (MV2) MV4 Hydrogen Feed H2 Sep Clean Product
0 0 0 b44 0 0 0 b54 0 0 0 b4 0 0 0 0 0 0 0 0 0 0 0 0 a1313 0 0 0 0 0 0 0 0 0 0 0 0 c13 UCLA Chemical Engineering State Space Model of Hydrotreating Unit 2 3 a11 0 0 0 0 0 0 0 0 0 0 0 0 0 a22 0 0 0 0 0 0 0 0 0 0 0 0 a32 a33 0 0 0 0 0 0 0 0 0 0 0 0 0 a44 0 0 0 0 0 0 0 0 0 0 0 0 0 a55 0 0 0 0 0 0 0 0 0 0 0 0 0 a 0 0 0 0 0 0 0 0 0 0 0 0 a a 0 0 0 0 0 0 A = 0 0 0 0 0 0 0 a88 0 0 0 0 0 0 0 0 0 0 0 0 a98 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1010 0 0 0 0 0 0 0 0 0 0 0 0 a1110 a1111 0 0 0 0 0 0 0 0 0 0 0 0 0 a1212 0 4 5 b11 0 0 0 2 3 b21 0 0 0 2 3 0 0 0 0 c11 0 0 0 0 0 0 0 0 0 0 0 0 0 c22 0 0 0 0 c2 0 0 0 0 0 0 0 0 c33 c34 c35 c3 0 0 0 0 0 0 0 B = 0 0 0 0 0 0 0 0 c49 0 0 0 0 0 0 0 0 C = 0 0 0 0 0 0 0 0 0 c510 0 0 0 b81 b82 0 0 0 0 0 0 0 0 0 0 0 0 c11 c12 0 0 0 0 0 4 0 b102 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 c812 0 4 0 0 b123 0 5 0 0 b133 0
Set Point Change in Hydrotreating Process New Target Point: Reactor Exit Temperature: Reduce by 4 degrees Reactor Inlet Temperature: Reduce by 5 degrees Reactant Feed Rate: Unchanged Quadratic Program Statistics: Size: N = 3 Horizon of Optimization Variables: 1 N = 48 # of Equality Constraints: 13 N = 39 # of Inequality Constraints: 12 N = 3 #
u min x k k 8 >< 9 > Q T + u T k R = msu k > ocx k k x UCLA Chemical Engineering Constrained Innite-time LQ Optimal Control 1X >: k=0 subject to Process Model: x k+1 = Ax k + Bu k k > 0 Input Constraints: u u k u k 0 Output Constraints: y Cx k y k > 0
LQ Optimal Control The Unconstrained LQ Problem, both Finite and Innite-time versions 190) (Kalman, Introduced Constrained Predictive Control to the ChE Community Group - Cutler, Prett, et. al., 1980) (Shell Guaranteed Closed-loop Stability with a Constrained Predictive Control (Rawlings and Muske, 1993) Algorithm The Constrained Innite-time LQ Problem (Sznaier and Damborg, 198 and Manousiouthakis, 199 and Scokaert and Rawlings, 1998 ) Chmielewski
If there exists any constrained stabilizing input trajectories, UCLA Chemical Engineering Main Result 1 - Constrained Stability then the solution to Innite-time Problem will be stabilizing If there does not exist any constrained stabilizing inputs, then the Innite-time Problem will be infeasible
T 0 Px 0 = min u x x k k 8 >< 9 > Q x T + u T = k R ms u k > k oc k x x k+1 = Ax k + Bu k UCLA Chemical Engineering Unconstrained Innite-time LQ Optimal Control Let P satisfy the Riccati Equation: = T 0 ; PB R ms + B T ;1 PB B T 1 P A A + Q oc @P A P Then the Solution to the Unconstrained Problem is 1X >: k=0 subject to
>: >: >< >: >< >: 1X k+1 = Ax k + Bu k k 0 x u k u k 0 u x k+1 = Ax k + Bu k > x k+1 = Ax k + Bu k x k+1 = Ax k + Bu k >= > >= > >= > UCLA Chemical Engineering Finite-time Solution to the Innite-time Problem 8 9 >< x T k Q ocx k + u T k R msu k >= 1X min s.t. k=0 y Cx k y k > 0 8 x T k Q ocx k + u T k R msu k >< X N;1 0 k < N = min s.t. u u k u 0 k < N k=0 y Cx k y 0 < k N 9 9 8 k N x T k Q ocx k + u T k R msu k k N + min s.t. k=n u u k u k > N y Cx k y 9 8 x T k Q ocx k + u T k R msu k 0 k < N X N;1 0 k < N? =? min u u k u + x T N Px N s.t. k=0 y Cx k y 0 < k N
x Finite-time Solution to Innite-time Problem k Region of Constrained Stabilizability Region of Constraint Inactivity N * time N-1 * Σ (x k=0 2 +u k 2 k ) + 8 Σ (x k=n * 2 k 2 +u ) k P x 2 N*
umin x k k 8 >< T oc x k + u T k R ms u T k + x NPx N Q k x x k+1 = Ax k + Bu k 9 >= UCLA Chemical Engineering Main Result 2 - Existence If there exists a constrained stabilizing input trajectory, then there always exists N (x0) < 1 such that X N;1 >: > k=0 subject to 0 k < N u u k u 0 k < N y Cx k y 0 < k N gives the solution to the Innite-time Problem for all N N (x0)
Z 1 1T A f(x) ; 1 4 1T A UCLA Chemical Engineering Constrained Innite-time Nonlinear Optimal Control x T Q oc (x)x + u T R ms (x)u dt min u(t) x(t) 0 subject to Process Model: _x = f(x) +g(x)u x(0) = Input Constraints: u u(t) u t 0 Output Constraints: y Cx(t) y t > 0 Unconstrained Solution: Find J(x) to satisfy the Hamilton-Jacobi-Bellman Equation 0 @J @ 0 @J @ 0 @J 1 A @ B(x)R ;1 (x)b T (x) 0 = x T Q(x)x + @x @x @x
( Z tf Input Constraints: u u(t) u 0 t t f Output Constraints: y Cx(t) y 0 t t f UCLA Chemical Engineering Constrained Innite-time Nonlinear Optimal Control ) x T Q oc (x)x + u T R ms (x)u dt + J (x (t f )) min u(t) x(t) 0 subject to Process Model: _x = f(x) + g(x)u x(0) =
Disturbance Rejection Process Disturbances Actual Output + - C P + + Measurement Noise Measured Output
Disturbance Types: Reactor Example CA in Possible Measurement of C Ain C Ain C Ain A k 1 B Possible Disturbance Models
>< >: w max v k k >: 8 >< >: NX T Qx k + T (y k ) R (y k ) k x w T k k w w T k k w v T k k v v T k k v = v 9 >= > >= v > = >= > UCLA Chemical Engineering Standard Disturbance Attenuation Formulations Min Max LQ Optimal Control - Assumes Worst Case Scenario 1 xt P Qx k + T (y k ) R (y k ) k=0 k P T 1k=0 w k + v T k v s.t. k k w k+1 = Ax k + B (y k )+w k x k = Cx k + v k y min ) ( Stochastic LQ Optimal Control - Assumes Most Likely Scenario 9 8 k+1 = Ax k + B (y k )+w k x k = Cx k + v k y 1 min ) ( s.t. lim N!1 k=0 N = w E E 9 8 >< x T k Qx k + T (y k ) R (y k ) E N!1 k+1 = Ax k + B (y k )+w x k = Cx k + v k y s.t. min = ) ( lim E = w E
Unconstrained Stochastic Optimal Control Separation Principle: A Cascaded Implementation of the Optimal Estimator followed by the Optimal Controller does not sacrice Overall Optimality. Certainty Equivalence: Feedback Implementation of the Disturbance Free Optimal Controller provides the Solution to the Stochastic Optimal Control Problem.
y k (sp) + Separation Principle Stochastic Optimal Controller w k P y k - u k + + v k y k (sp) + - Optimal State Estimator x k w k FSI Stoc Optimal Controller u k P + + y v k k
x T Q 0 ocx + u T R ms u + E [J (Ax + Bu + w)] u T R ms u + E [ J (Ax + Bu + w) ] UCLA Chemical Engineering Certainty Equivalence If J satises the Hamilton-Jacobi-Bellman Equation ( E w w T = ) (x) = J 0 (x) ; inf x J fj 0 (x)g J 0 (x) = u u u min Then the full information stochastic optimal feedback policy is (x) = arg min u u u In the Unconstrained Case, Certainty Equivalence holds ( i.e., (x) = 0 (x) ). In the Constrained Case, (x) 0 (x) only if the disturbances are small.
Stochastic Optimal Control Constrained Results Main UCLA Chemical Engineering Separation Principle Continues to hold in the Constrained Case Certainty Equivalence Does Not hold in the Constrained Case If the Open-Loop Process (i.e., the A matrix) is Stable, the Constrained Stochastic Optimal Policy Exists then If the Open-Loop Process (i.e., the A matrix) is Unstable, the Constrained Stochastic Optimal Policy Does Not Exist then Output Constraints can be Incorporated through a Soft Constraints
Disturbance Rejection: CSTR Example C A in Disturbance Model C A Measurement of (Relative Error: 0.2%) A k 1 B Operating Conditions: Flow Rate ( o ): 0.4 (m 3 /min) 0:325 o 0:45 Volumetric Rate Constant (k 1 ): 0.0 (m 3 /kmole min) Reaction Residence Time (): 5 (min) Concentration of A In (C Ain ): 10 (kmole/m 3 )
8 >< >: 8 >< >: 1X x 2 X max if if x =2 X max +1 9 >= > UCLA Chemical Engineering Constrained Worst Case Problem Formulation H(x k ) s.t. x k+1 = Ax k + B(x k ) +!(x k ) min ( ) u u max!( ) w w k=0 x T Qx + T (x)(x) ; 2! T (x)!(x) H(x) = where X max is the constrained stabilizable set for the disturbance free problem The Nonlinear Operators ( ) and!( ) are the Optimization Variables > 0 is a xed constant
4 (x) >< >: >< >: Ax + Bu + w 2 X max Ax + Bu + w 2 X max >= > >= > UCLA Chemical Engineering Constrained Min Max Optimal Control If J satises the Isaacs' Equation 8 9 x T Qx + u T u ; 2 w T w + J(Ax + Bu + w) min u u u max w w w J(x) = s.t. Then the Min Max Optimal Feedback policy is 9 8 2 3 u T u ; 2 w T w + J(Ax + Bu + w) 5 = arg min u u u max w w w!(x) s.t.
= arg max 5 k w >< >: u min x k k >< >: N;1 P k=0 T Qx k + u T k u k ; 2 w T k w k + x T N Px N k x u k u u w k w w >= > >= > UCLA Chemical Engineering A Finite-time Max Min Problem 8 8 9 9 2 4 (x 0) 3 s.t. k+1 = Ax k + Bu k + w k x Cx k y y!(x 0 ) where P = Q + A T P ;1 A and = I + BB T ; ;2 GG T P
4 Value Functions Min Max Example 3 2 Solution to Constrained Issacs' Equation Solution to Constrained Max Min Problem the One State Consider System: 1 0 x k+1 = 0:9x k + u k + w k 0.4-1.5-1.0-0.5 0.0 0.5 1.0 1.5 x Control Policies: u(x) 0.2 0.0 With Constraints: -0.2 ju k j 0:3 and jw k j 0:05-0.4-1.5-1.0-0.5 0.0 0.5 1.0 1.5 x 0.15 Disturbance Policies: w(x) 0.10 And Design Parameters: 0.05 0.00-0.05-0.10 Q = 1 and 2 = p 3-0.15-1.5-1.0-0.5 0.0 0.5 1.0 1.5 x
Conclusions The CITLQOC Controller has many Advantages over the Finite-time Controller (i.e., Tuning and Stability) Predictive The CITLQOC can be solved using a Finite-Dimensional Optimization Problem The Constrained Stochastic Optimal Controller has Performance Advantages, a Substantial Increase in Computational Cost with The Constrained Min Max Controller also has Performance Advantages, a only Moderate Increase in Computational Cost with
Acknowledgments Special Thanks to: Professor Vasilios Manousiouthakis and The Optimal Group Financial Support is Gratefully Acknowledged from: The Department of Education # P200A4032-95 and The National Science Foundation #'s GER-955450, CTS-9312489