Optimal Control of Linear Systems with Stochastic Parameters for Variance Suppression: The Finite Time Case

Size: px

Start display at page:

Download "Optimal Control of Linear Systems with Stochastic Parameters for Variance Suppression: The Finite Time Case"

Erika Byrd
5 years ago
Views:

1 Optimal Control of Linear Systems with Stochastic Parameters for Variance Suppression: The inite Time Case Kenji ujimoto Soraki Ogawa Yuhei Ota Makishi Nakayama Nagoya University, Department of Mechanical Science and Engineering, Nagoya , JAPAN Kobe Steel, LTD., Kobe , JAPAN Abstract: In this paper, we consider an optimal control problem for a linear discrete time system with stochastic parameters. Whereas traditional stochastic optimal control theory only treats systems with deterministic parameters with stochastic noises, this paper focuses on systems with both stochastic parameters and stochastic noises. We derive an optimal control law for a novel cost function by which the designer can adjust the trade-off between the average and the variance of the states. urthermore, a numerical simulation shows the effectiveness of the proposed method. Keywords: Stochastic optimal control problems; Probabilistic robustness; Linear systems 1. INTRODUCTION Stochastic and statistical methods are used in several problems in control systems theory. In particular, system identification and estimation methods rely on them. Recently, it is reported that Bayesian estimation and the related estimation tools in machine learning Bishop 26 are applied to system identification of state-space models and statistical information of the system parameter become available for control Barber and Chiappa 27; ukunaga et al. 28. There is another paper reporting that the quality of the products are estimated by those methods in process control Kano 21. So called randomized approach is also proposed to apply statistical tools to controller design problems Calafiore et al. 24. On the other hand, optimal control is an important and established control method. LQG Linear Quadratic Gaussian method can take care of optimal control problems with stochastic disturbances Aström 26; Mendel and Gieseking Stochastic control theory has been developed by many authors based on it which mainly focuses on systems with deterministic parameters and stochastic noises. In stochastic control theory, MCV Minimum Cost Variance control Sain 1966; Sain et al and RS Risk Sensitive control Whittle 1981 were proposed which can suppress the variance of the cost function with respect to the stochastic disturbance. Those existing results focus on the systems with deterministic parameters whereas they can take care of stochastic disturbances. In order to utilize the Bayesian estimation results for state space systems Barber and Chiappa 27; ukunaga et al. 28, controller design method for state space systems with stochastic parameters are needed. or this problem, De Koning proposed a controller design method for state space systems with stochastic system parameters De Koning 1982 which employs standard quadratic cost function for optimal control. Although De Koning s paper can take care of the effect of the variation of system parameters, it cannot suppress the variance of the transient of the resulting control system caused by the variation of the system parameters. The present paper proposes a novel optimal control method with variance suppression for state space systems with stochastic parameters and disturbances, in which we can suppress the variance of the trajectory of the state caused by the variation of the system parameters. urthermore, numerical simulations demonstrate the effectiveness of the proposed method. We believe that the proposed method can provide a new framework to stochastic control together with Bayesian estimation. 2. PRELIMINARIES This section gives notations and some preliminary results according to Bishop 26; De Koning The symbol R n denotes the n-dimensional Euclidean space. M mn and M n denote the space of m n real valued rectangular matrices and that of n n real valued square matrices, respectively. S n denotes the space of n n real valued symmetric matrices. The symbol vec denotes a function satisfying a 1 a 2 a 1i veca =. a 2i Rmn, a i :=. Rm. a n a mi

2 where a ij is the i, j element of the matrix A. The expectation of a time-varying stochastic parameter a t is denoted without the time parameter t as E[a] instead of E[a t ], if its statistics is time invariant. We use several transformations for symmetric matrices. Let us define a monotonic transformation. Definition 1. De Koning 1982 A transformation for symmetric matrices A : S n S n is said to be monotonic if it satisfies AX AY, for any symmetric matrices X, Y S n satisfying X Y. The following lemma holds for monotonic transformations. Lemma 1. De Koning 1982 Suppose that a transformation A : S n S n is monotonic and that AX holds for any symmetric matrix X. Then A i is monotonic for all natural number i and A i X holds for any symmetric matrix X. Next, let us consider a discrete time system x t+1 = A t x t + B t u t 1 where x t R n is the state, u t R m is the control input, A t M n, B t M nm are the system matrices costing of system parameters. Suppose that the system matrices A t and B t are random parameters with time invariant statistics. Assume also that the initial state x is deterministic. Apply a state feedback u t = Lx t 2 to the system 1 with the feedback gain L M mn. Then the resulting feedback system is described by x t+1 = Ψ L,t x t 3 with a new system matrix Ψ L,t := A t B t L M n. In order to describe the behavior of the state x t with its variance, let us define a transformation A L : S n S n describing the expectation of a quadratic function along the feedback system 3 as follows. A L X := E[Ψ T LXΨ L ], X S n 4 The transformation A L thus defined satisfies the following lemma. Lemma 2. De Koning 1982 a E[x T i Xx i x ] = x T A i L Xx holds for any natural number i and any symmetric matrix X. b The transformation A i L is linear and monotonic for any natural number i. 3. OPTIMAL CONTROL OR VARIANCE SUPPRESSION This section states the main result of the paper, optimal control for variance suppression. Consider a linear discretetime system with stochastic parameters x t+1 = A t x t + B t u t + G t ɛ t 5 where ɛ t R n is a stochastic external noise. A t M n, B t M nm, and G t M n are stochastic parameters defined by time invariant statistics. or this system, let us consider the following cost function. J N U N, x = E x T t Qx t + u T t Ru t + trs cov[x t+1 x t ] } + x TN x N x ] Here the matrices Q S n, R S m, S n, and S S n are design parameters and U N = u t, t N 1} denotes the collection of the input. The third term in the right hand side of Equation 6 reduces to tr S cov[x t+1 x t ] = E [ x T t+1sx t+1 x t ] E[xt+1 x t ] T S E[x t+1 x t ] which is the weighted sum of the covariance of the states. The coefficient matrix S can be used to select the weights between the variation and the average of the state. or this system, the optimal control problem for variance suppression is defined as follows. Definition 2. Consider the system 5 and the cost function 6. or a given initial condition x R n, find an input sequence UN = u t, t N 1} minimizing the cost function J N U N, x and the corresponding minimum value of the cost function JN x. We call this control problem as a finite time optimal control for variance suppression. Before solving the variance suppression problem, let us consider the relationship between the value of the cost function at the time t with the state x t and that at the time t 1 with the state x t 1. Suppose that a state feedback u t = Lx t is employed, then the term to evaluate the variance of the state E[trS cov[x t+1 x t ] x ] in the cost function can be calculated as 6 E [trs cov[x t+1 x t ] x ] = E [ E[x T t+1sx t+1 x t ] E[x t+1 x t ] T ] S E[x t+1 x t ] x = E [ x T t E[Ψ T L SΨ L ] E[Ψ T L]S E [Ψ L ] ] x t x = E [ x T t AL S E[Ψ T L]S E[Ψ L ] ] x t x = E [ x T t 1A L AL S E[Ψ T L]S E[Ψ L ] ] x t 1 x = E [ x T A t L S A t 1 L E[Ψ T L ]S E[Ψ L ] } ] x x = x T A t L S A t 1 L E[Ψ T L ]S E[Ψ L ] } x 7 where the function B L : S n S n is defined by B L X := A L X+Q+L T RL+A L S E[Ψ T L]S E[Ψ L ], with X S n 8 βx := E[ɛ T G T XGɛ]. 9 Then the value of the cost function 6 becomes

3 J N U N, x = E x T t Q + L T RLx t } ] + trs cov[x t+1 x t ] + x TN x N x N 1 Q + L T RL = x T A t L } + A L S E[Ψ T L]S E[Ψ L ] + A N L + β N 1 = x T B N L x + β t 1 A t L + A j L x Q + L T RL + A L S E[Ψ T L]S E[Ψ L ] } + NS NS + N 1 B t L. 1 Now we can prove a property of the function B L as in the following lemma. Lemma 3. or any natural number i, the function B i L is monotonic. In particular, for any symmetric matrix X, B i L X holds. Proof. irst of all, it will be proved that B i L = B L is monotonic for the case i = 1. Lemma 2 b implies that A L X A L Y holds for any matrices X, Y S n satisfying X Y since the function A L is linear. Hence Equation 8 proves B L X B L Y which implies the monotonicity of B L. On the other hand, A L X + Q + L T RL holds for any X. Therefore what we have to prove is the property A L S E[Ψ L ] T S E[Ψ L ] which will complete the proof by using Lemma 1. Since S S n, there exists an orthogonal matrix M M n diagonalizing S as Σ = M T SM = diags 1, s 2,..., s n. Employing a coordinate transformation y t = M T x t, the closed loop system 3 is described by y t+1 = M T Ψ L,t My t. Using the expression y t = [yt 1, yt 2,, yt n ] T, we have AL S E[Ψ T L]S E[Ψ L ] x t x T t = y T t M T A L S E[Ψ T L]S E[Ψ L ] My t = yt T M T E[Ψ T LSΨ L ] E[Ψ T L]S E[Ψ L ] My t = yt T E[M T Ψ T LMΣM T Ψ L M] E[M T Ψ T LM]Σ E[M T Ψ L M] = E[yt+1Σy T t+1 y t ] E[y t+1 y t ] T Σ E[y t+1 y t ] = tr Σ cov[y t+1 y t ] n = s i var[yt+1 y i t ]. i=1 y t Since the variances var[yt+1 y i t ] s and the coefficients s i s are positive, n i=1 s i var[yt+1 y i t ] which proves A L S E[Ψ T L ]S E[Ψ L]. This proves that B L X = A L X + Q + L T RL + A L S E[Ψ T L ]S E[Ψ L] holds for any X. It follows from Lemma 1 that the function BL i is monotonic for any natural number i and that BL i X holds for any X which completes the proof. Using this lemma, we can prove the main theorem which provides the solution to the optimal control for variance suppression. Theorem 1. The optimal control law to minimize the cost function 6 and the minimum value of the cost function J N x are given as follows. u t = L B N t 1 x t, t =,..., N 1 11 J N x = x T B N x + β NS + N 1 B t Here the function B : S n S n is defined by, x 12 B X := B LX X, X S n. 13 The function B LX is defined in Equation 8 and the gain matrix L X is defined by L X := E[B T XB]+Σ BB +R 1 E[B T XA]+Σ BA, where Σ XY := E[X T SY ] E[X] T S E[Y ]. Proof. irst of all, let us define the value function by V t x t = min u t,,u N 1 E j=t x T j Qx j + u T j Ru j X S n 14 } + trs cov[x j+1 x j ] + x T N x N x t ]. 15 which denotes the minimum value of the cost function along the system 5 for the time period from t to N 1 with the initial state x t. Let us now prove by induction that k 1 V N k x N k = x T N kb k x N k + β ks + B t holds for any natural number k1 k N. 16 The case k = : The boundary condition implies that V N x N = x T N x N = x T N B x N. The case k = 1:

4 V N 1 x N 1 [ u N 1 u N 1 E x T N 1Qx N 1 + u T N 1Ru N 1 + cov[x N x N 1 ] + x T N x N x N 1 ] x T N 1 E[A T B A] + Σ AA + Q + u T N 1 E[B T B B] + Σ BB + R + 2u T N 1 E[B T B A] + Σ BA } + E[ɛ T G T B Gɛ] + E[ɛ T G T SGɛ] x N 1 x N 1 u N holds. Hence the optimal input u N 1 minimizing the right hand side of Equation 17 is given by u N 1 = E[B T B B] + Σ BB + R 1 E[B T B A] + Σ BA x N 1 = L B x N V N l x N l u N l,,u N 1 u N l u N l E j=n l x T j Qx j + u T j Ru j + tr S cov[x j+1 x j ] } + x T N x N x N l ] x T N lqx N l + u T N lru N l + tr S cov[x N l+1 x N l ] + E[V N l+1 x N l+1 x N l ] x T N l + u T N l + 2u T N l E[A T B A] + Σ AA + Q x N l E[B T B B] + Σ BB + R u N l E[B T B A] + Σ BA x N l + E[ɛ T G T B Gɛ] + E[ɛ T G T SGɛ] } l 2 + β l 1S + B j. 2 Substituting this equation for Equation 17, we obtain V N 1 x N 1 = x T N 1 E[A T B A] + Σ AA + Q x N 1 + x T N 1L T B E[B T B B] + Σ BB + R L B x N 1 2x T N 1L T B E[B T B A] + Σ BA x N 1 + β S + B = x T N 1 E [ A BL B T B A BL B ] + Q + L T B RL B + E [ A BL B T SA BL B ] E [ A BL B ] T [ S E A BLB ]} x N 1 + β S + B = x T N 1 A LB B + Q + L T B RL B + A S E[Ψ LB L B ]T S E[Ψ LB x ] N 1 + β S + B = x T N 1B LB B x N 1 + β S + B = x T N 1B 1 x N 1 + β S + B. 19 The input u N l minimizing the right hand side of Equation 2 is u N l = E[B T B B] + Σ BB + R 1 E[B T B A] + Σ BA x N l = L B x N l. 21 Substituting this input for Equation 2, we obtain V N l x N l = x T N l E [ A BL B T B A BL B ] + Q + L T RL B B + E [ A BL B T SA BL B ] E [ A BL B ] T [ S E A BLB ]} x N l + β ls + B j = x T N l = x T N lb LB B x N l + β ls + B j Therefore, Equation 16 holds for the case k = 1. Next, suppose that Equation 16 holds in the case k = l1 l N 1 and prove 16 in the case k = l + 1: = x T N lb l x N l + β ls + B j 22 Namely, we suppose V N l+1 x N l+1 = x T N l+1 B x N l+1 + β l 1S + l 2 Bj here. Then we have which coincides with Equation 16 in the case k = l + 1. A LB B + A LB S E[Ψ LB + β ls + B j + Q + L T B RL B ] T S E[Ψ LB ] x N l

5 This implies that Equation 16 holds for any k 1 k N by induction. Substituting k = N t for Equation 16, we obtain the optimal input u t at the time t as u t = E[B T B N t 1 B] + Σ BB + R 1 E[B T B N t 1 A] + Σ BA x t = L B N t+1 x t. Next, substituting k = N for Equation 16, we have the minimum value of the cost function as J N x = V x = x T B N x + β NS + N 1 B t Conversely, if we employ the input in Equation 11, then the cost function 6 coincides with its minimum 12. This prove the theorem. Let us denote the value of the cost function at the time t by V t x t = x T Π t x. Then Equation 13 reduces to the following recursive equation similar to a Riccati equation. Π t 1 = Q + Σ AA + E[A T Π t A] E[A T Π t B] + Σ AB E[B T Π t B] + Σ BB + R 1 E[B T Π t A] + Σ BA. 23 Here Π N = and B k = Π N k, namely, u t = L Πt x t. urthermore, Equation 23 reduces to a Riccati equation for the conventional LQG problem if the parameters A t and B t are deterministic, which implies that the proposed method is a natural generalization of the conventional LQG method. 4. NUMERICAL EXAMPLE This section gives a numerical example to demonstrate the effectiveness of the proposed method in comparison to the conventional LQG method. Let us consider the plant described by Equation 5 with the terminal time N = 25, the state x R 2 and the input u R. [ ] [ ] 1.1 E[A] =, E[B] = cov [vec[a, B]]= [ ] [ ] 1 4 Q =, R = 1, = 1 4 [ ] 1 x = 1 The covariance cov [vec[a, B]] are selected in such a way that the parameters A 21, A 22 and B 2 have 1% standard deviations. ig.1 shows the time responses of 1 random samples of the feedback system with a conventional LQG controller. igs.2 and 3 show the time responses of 1 random samples by the proposed method with the design parameters S = and S = 1I, respectively. In those figures, the plus signs + denote the upper and the lower i State transition of x 1 ig. 1. State transition LQG ii State transition of x 2 line of the 1σ deviation from the average and the solid lines denote the sampled responses. igures show that both the conventional LQG controller and the proposed controller achieve stability of both the average and the variance. The parameter S = 1I in ig.3 achieves small variance as we expected whereas the convergence speed of its average is slower, which implies that there is a trade-off between the convergence speed of the expectation the that of the variance. i State transition of x 1 ii State transition of x 2 ig. 2. State transition proposed method S = i State transition of x 1 ii State transition of x 2 ig. 3. State transition proposed method S = 1I Next, igs.4 6 show the case in which the variances of the parameters are bigger than the case in igs.1 3. The covariance of the system parameters A and B is selected as follows in such a way that the parameters A 21, A 22 and B 2 have 5% standard deviations cov [vec[a, B]] = igs.4 6 show the time responses of 1 random samples by the LQG method and the proposed method with the design parameters S = and S = 1I, respectively. In those figures, the plus signs + denote the upper and the

6 ACKNOWLEDGEMENTS i State transition of x 1 ig. 4. State transition LQG i State transition of x 1 ii State transition of x 2 ii State transition of x 2 ig. 5. State transition proposed method S = i State transition of x 1 ii State transition of x 2 ig. 6. State transition proposed method S = 1I lower line of the 1σ deviation from the average and the solid lines denote the sampled responses as in igs.1 6. ig.4 shows the time responses of the state of the feedback system with the LQG controller. In the figure the states diverge, since the LQG controller does not take care of the variance of the system parameters. igs.5 and 6 shows the time responses of the states with the proposed controllers for the parameters S = and S = 1I. Both figures show that the state converge to smoothly and the variance of the state is smaller in the case S = 1I compared to the other case S =. This result shows that the controller with the bigger parameter S suppresses the variance of the state response as we desired, whereas it also let the convergence speed of the average of the state response slower as in the previous case. The authors would like to thank Professor Yasumasa ujisaki at Osaka University for his valuable suggestions. REERENCES Aström, K.J. 26. Introduction to Stochastic Control Theory. Dover Publications. Barber, D. and Chiappa, S. 27. Unified inference for variational Bayesian linear Gaussian state-space models. In Advances in Neural Information Processing Systems 19 NIPS 2, The MIT Press. Bishop, C.M. 26. Pattern Recognition and Machine Learning. Springer, New York. Calafiore, G., Tempo, R., and Dabbene,. 24. Randomized Algorithms for Analysis and Control of Uncertain Systems. Springer. De Koning, W.L Infinite horizon optimal control of linear discrete time systems with stochastic parameters. Automatica, 184, ukunaga, S., Ishihara, Y., and ujimoto, K. 28. Byesian estimation methods for state-space models. In Proc. SICE 8th Annual Conference on Control Systems. in Japanese. Kano, M. 21. Modeling from process data. Measurement and Control, 492, in Japanese. Mendel, J.M. and Gieseking, D.L Bibliography on the Linear-Quadratic-Gaussian problem. IEEE Trans. Autom. Contr., 6, Sain, M.K Control of linear systems according to the minimal variance criterion: A new approach to the disturbance problem. IEEE Trans. Autom. Contr., 111, Sain, M.K., Won, C.H., and Spencer Jr, B Cumulants in risk-sensitive contro: The full-state-feedback cost variance case risk-sensitive and MCV stochastic control. In Proc. 34th IEEE Conf. on Decision and Control, Whittle, P Risk-sensitive Linear/Quadratic/Gaussian control. Advances in Applied Probability, 13, CONCLUSION This paper proposes a novel optimal control method with variance suppression for state space systems with stochastic parameters and disturbances. We have derived a Riccati type recursive equation to solve the proposed optimal control problem which employs second order statistics of the system parameters. urther, some numerical simulations demonstrate the effectiveness of the proposed method. We believe that the proposed method provides a new stochastic control framework with Bayesian estimation.

Optimal Control of Linear Systems with Stochastic Parameters for Variance Suppression

Optimal Control of Linear Systems with Stochastic Parameters for Variance Suppression Optimal Control of inear Systems with Stochastic Parameters for Variance Suppression Kenji Fujimoto, Yuhei Ota and Makishi Nakayama Abstract In this paper, we consider an optimal control problem for a