Statistical Control for Performance Shaping using Cost Cumulants

Transcription

1 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 1 Statistical Control for Performance Shaping using Cost Cumulants Bei Kang Member IEEE Chukwuemeka Aduba Member IEEE and Chang-Hee Won Member IEEE Abstract The cost cumulant statistical control method optimizes the system performance by shaping the probability density function of the random cost function For a stochastic system a typical optimal control method minimizes the mean first cumulant of the cost function However there are other statistical properties of the cost function such as variance second cumulant skewness third cumulant and kurtosis fourth cumulant which affect system performance In this paper we extend the theory of traditional stochastic control by deriving the Hamilton-Jacobi-Bellman HJB equation as the necessary conditions for optimality Furthermore we derived the verification theorem for higher order statistical control and construct the optimal controller In addition we utilize neural networks to numerically solve HJB partial differential equations Finally we provide simulation results for an oscillator system to demonstrate our method Index Terms Cost cumulants Hamilton-Jacobi-Bellman equation neural networks statistical control stochastic optimization I INTRODUCTION In statistical control paradigm we consider the cost function as a random variable and use statistical controller to shape its probability distribution Shaping of the cost distribution is thus achieved by optimizing the cost cumulants In this way we have the opportunity to influence and optimize the distribution of the cost function which then in turn offers the possibility of controlling the probability of the occurrence of certain types of events In intuitive terms we refer to this as performance shaping One instance of the statistical control is achieved by the selection of the mean which is the first cumulant In this case the optimization of the mean of the cost for a linear system with quadratic cost and Gaussian noise leads to the well known linear-quadratic-gaussian LQG control Another instance of the choice of such a function is the selection of the cost variance which is the second cumulant This minimal cost variance MCV control is carried out while keeping the mean at a prespecified level Generalizing this idea a finite linear combination of the cost cumulants is optimized in statistical control The reader may wonder why we propose to use cost cumulants instead of cost moments Even though moments and cumulants seem to be similar mathematically in control engineering cumulant control has shown substantial promise especially in structural control 1 while cost moment control has been studied by various researchers without much promise For example even for a linear system the optimization of first two moments lead to a nonlinear controller Moreover low order cumulants are more significant than higher order cumulants thus this leads to more effective numerical approximation algorithm 3 Statistical control gives the control engineer the freedom to shape the cost distribution The cost function is related to the performance of the system such as energy fuel or time Consequently by controlling the cost distribution one can affect the performance of the system For example in satellite attitude control application the pointing This work was supported in part by the National Science Foundation AIS Grant ECCS Bei Kang is a researcher with Bloomberg LP New York and Chukwuemeka Aduba is a doctoral student with Department of Electrical and Computer Engineering Temple University Philadelphia Chang-Hee Won corresponding author is an associate professor of Electrical and Computer Engineering Department Temple University PA 191 USA phone: ; fax: ; cwon@templeedu accuracy may be represented by the variance of the cost function Consequently instead of optimizing the mean pointing accuracy one may minimize the variation using the second cumulant Statistical control allows the control engineer to make those choices and shape the performance statistically through the cost cumulants Statistical control has been researched over the past five decades and it is closely related to Linear-Quadratic-Gaussian 4 and Risk Sensitive Control 5 In typical Linear-Quadratic-Gaussian control we find the controller such that the expected value of the utility function is optimized 6 In Risk Sensitive control denumerable sum of the quadratic cost functions is optimized 7 5 This idea can be generalized to the optimization of the arbitrary order cumulants of the cost function which is the concept behind statistical optimal control 4 8 Consequently statistical optimal control is a generalization of the traditional Linear-Quadratic-Gaussian LQG and risk-sensitive control Sain solved the open-loop minimum cost variance problem which is the second cumulant of a cost function 4 In minimum cost variance control the variance of the cost function is minimized while the mean is constrained Liberty et al 9 studied minimum cost variance control for a linear system of quadratic cost function Risk Sensitive control as a special case of statistical control was studied in 7 1 Sain et al 11 analyzed the minimum cost variance control as an approximation of Risk Sensitive control where the mean and variance are optimized In 3 we derived the HJB equations for nd and 3rd cumulant control problem In 3 the method is based on iterative method and general n-th cost cumulant HJB equation which is a necessary condition for optimality is not given By using an induction method we derive n-th cost cumulant HJB equation in the current paper Moreover the optimal controller is not derived in 3 Paper 1 is a preliminary conference version of this paper There we did not have the induction proof for the HJB equation and the application was for a linear system We find the 3rd cumulant control of a linear system in 1 HJB equations are difficult to solve analytically except for a few special cases such as Linear-Quadratic-Gaussian control For a linear stochastic systems previous works in solved the HJB equation for higher order cumulants However for a nonlinear system there is no effective way to solve the corresponding HJB equations Thus we seek a method which handles both linear and nonlinear system HJB equations Werbos proposed to use neural network for optimal control of a linear system in Beard proposed a method based on Galerkin approximation to solve the generalized HJB eqauation 15 For discrete time nonlinear system Chen and Jagannathan solved the HJB equation using neural networks 16 For a deterministic continuous-time nonlinear system a neural network method is used to approximate the value function of the HJB equation in 17 The neural network approximation method is similar to power series expansion method which was introduced by This method approximates the value function in infinite-time horizon The approximated value function and the optimal controller are determined by finding coefficients of the series expansion In the authors proposed a radial basis function neural network method for the output feedback system Inspired by these advances we develop a neural network method to solve HJB equations numerically for arbitrary order cost cumulant minimization problem If we define the weighting and the basis functions as polynomials the neural network is in fact equivalent to power series expansion in that we use the coefficients of the power expansion as the weights in neural networks However neural networks method has the potential to be more than simple power series expansion by adding more sophisticated layers of neurons and algorithms This paper is organized as follows In Section II the statistical

2 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL control problem is formulated In Section III the necessary and sufficient conditions for the n-th cumulant optimization are addressed and we construct the optimal controller in Section IV In Section V we introduce the notion of neural network method as a solution to the n-th cumulant HJB equations In Section VI simulation results are reported to show the effectiveness of our proposed neural network approach to the statistical control of a nonlinear system Finally we conclude the paper in Section VII II PROBLEM FORMULATION The dynamic system that we are studying is given by the Ito-sense stochastic differential equation dxt ft xt ut xtdt σt xtdwt 1 where t T t t F xt x xt R n is the state ut kt xt U R m is defined as the system control and dwt is a Gaussian random process of dimension d with zero mean and covariance of W tdt f and σ satisfy the local Lipschitz and linear growth conditions A memoryless feedback control law is introduced as ut kt xt where k satisfies a Lipschitz condition See p 156 of 1 for details of this assumption These conditions determine the nonlinear system that behaves like a linear system If such a control law k exists then k is called an admissible control law Next we introduce a backward evolution operator Ok 11 defined by Ok n t f i t x kt x i n σt xw tσ t x ij i j i Moreover we use the Dynkin s formula 3 in HJB equation derivations as shown in the following equation tf } Φ t x E OkΦ s x ds Φ t F x t F xt x t where Φt x Cp 1 Q Q t t F R n Q is closure of Q k is admissible and E xs m xt x} is bounded for m 1 n and t T In later derivations we will use E tx instead of E xt x} to denote the conditional expectation The system cost Jt xt k tf t L s xs ks xs ds ψxt F 3 which is the quantity that we want to optimize satisfies the conditions such that Ls xs ks xs is continuous and Ls xs ks xs and ψ satisfy the polynomial growth condition These conditions allow the mean of cost function E tx J} to be finite In order to mathematically define the n-th cost cumulant minimization problem we first introduce the following definitions Definition 1 11: The i-th moment of the cost function is defined by the following Equation } M it x kt x E J i t x kt x xt x } 4 E tx J i t x kt x i 1 n Definition : A function M i : Q R is an admissible i-th moment cost function if there exists an admissible control law k such that M i t x; kt x M i t x Definition 3: Let the class of control law K M be such that for each k K M M i satisfies Definition for i 1 n 1 where n denotes a fixed positive integer Definition 4: A function V i : Q R is an admissible i-th cumulant cost function if there exists an admissible control laws k such that V it x; kt x V it x Definition 5 4: The admissible n-th cumulant of the cost function V n is defined by the following Equation V nt x M nt x n 1! Mn1it xvi1t x 5 where t T t t F xt x xt R n M i are admissible moment function for i 1 n and V i are admissible cumulant function for i 1 n 1 Consider an open set Q Q Assume the cumulant cost function V it x Cp 1 Q C Q are admissible functions for i 1 n 1 where the set Cp 1 Q C Q means that the functions satisfy polynomial growth condition and are continuous in the first and second derivatives of Q and continuous on the closure of Q This is restrictive assumption which is only true for special cases For non-smooth value function viscosity method should be used as in 3 This is reserved as the future study We assume the existence of an optimal control law k K V n M K M and an optimum value function Vn t x Cp 1 Q C Q Then the statistical control is to find the optimal control law k V n M which results in the minimal n-th cumulant cost function Vn t x and satisfies the conditions such that V n t x V n t x; k V n M t x min V n t x; kt x} 6 It should be noted that to find the optimal controller k V n M we constrain the candidates of the optimal controller to K M and the optimal V n t x is found with the assumption that lower order cumulants V i are admissible for i 1 n 1 III n-th COST CUMULANT MINIMIZATION AND HJB EQUATION In this section we mathematically analyze the n-th cost cumulant statistical control problem The goal is to minimize the n-th cost cumulant of a cost function J subject to the system dynamics in the presence of the random noise The following definition and lemmas are needed for deriving the n-th cost cumulant HJB equation Lemma 1: Let V it x V k1 t x Cp 1 Q C Q be admissible cost cumulant functions then we have the following result the arguments are suppressed k1 1 k k! j!k j! k 1! k 1 Vj σw σ Vkj Vk1i Proof: See Appendix A 5 The necessary conditions for the n-th cost cumulant minimization problem are given through HJB equations We use induction method to prove this theorem and it is a different approach than the iterative method of 6 The following theorem is the main result of this paper Theorem 1: Minimal n-th cumulant HJB equation Let V 1 t x V t x V n1 t x Cp 1 Q C Q be an admissible cumulant cost function Assume the existence of an optimal control law k V k M K M and an optimal value function Vn t x Cp 1 Q C Q Then the minimal n-th cumulant 7

3 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 3 cost function Vn t x satisfies the following HJB equation min Ok Vn t x 1 n1 n! Vs t x s!n s! s1 } σt xw σt x Vns t x 8 for t x Q with the terminal condition Vn t F x Proof: Here we present a brief proof of the theorem because this is the main result of the paper For a full version of the proof see Appendix B We use induction method to prove the theorem We assume that the theorem holds for the second third n 1-th cumulant case then we will show that the theorem also holds for the n-th cumulant case Let Vn be in the class of Cp 1 Q C Q Apply the backward evolution operator Ok to the recursive formula we have Ok V n Ok M n n 1! OkM n1iv i1 Assume that M jt x Cp 1 Q C Q is the j-th admissible moment cost function Then M j t x satisfy the following HJB equation Ok M j t x jm j1 t xlt x k 1 for t x Q where M j1t x is the j1-th admissible moment cost function and the terminal condition is given as M j t F x ψxt F 3 From 1 we have 9 OkM n nm n1 L 11 Let M it x V jt x Cp 1 Q C Q where i and j are nonnegative integers then Ok M it xv jt x Ok M i t x V j t x M i t xok V j t x Mit x Vjt σ t x W σ t x x 1 with boundary condition M it F x ψxt F V jt F x OkV υxt F 7 n 1! Vn1i n By letting i n 1 i j i 1 in 1 we obtain 18 OkM n1i V i1 OkM n1i V i1 Using 7 18 is written as M n1i OkV i1 13 OkVn 1 n1 n! n Substitute equations into 9 we have Vi t x σt xw σt x Vni t x OkVn n 1! nm n1 L OkM n1i V i1 The theorem is proved M n1iokv i1 Remarks The HJB equation 8 in Theorem 1 provides a necessary condition for the optimality of the n-th cost cumulant statistical control The optimality is achieved under the constraints that 14 Using the fact that M is an identity and OkV 1 L and the V 1 V V n1 Cp 1 Q C Q are admissible Next we will conversion formula between moment and cumulant 4 Therefore 14 becomes OkVn n 1! n 1! Mn1iOkV i1 15 Then from Theorem 1 7 and the formula in 8 M i V j j!i j! M ij equation 15 becomes OkVn n 1! n1i σw σ Vj1 n 1 j!n 1 i j! M n1ij M n1i i1 j n 1! Vj j!i j! 16 From 16 we notice that both the second and third term on the right hand side contain the moment from M x} where x 1 Therefore it is feasible to combine two summations with respect to M x and simplify the equation Thus 16 becomes OkVn n 1! 1 M n1i i1 j j!i j! n 1! i n 1 j!n 1 i j! M Vj n1ij n 1! Vn1i 17 In 17 for the second term when i varies from 1 to n the corresponding M x varies from M to M 1 In the third term when i varies from to n and j varies from 1 to n i the corresponding M x also takes the value from M 1 to M Now we combine the M x terms from the second and third terms of 17 Therefore 17 becomes the following equation demonstrate the sufficient condition of the n-th cumulant optimality Theorem : Minimal n-th cumulant HJB equation verification theorem Let V 1 t x V t x V n1 t x M 1 t x M t x M n1t x Cp 1 Q C Q be an admissible cumulant function Let Vn Cp 1 Q C Q be

4 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 4 a solution to the partial differential equation min Vi t x n1 n! n Ok Vn t x 1 σt xw σt x Vni t x } 19 with zero terminal condition Then Vn t x V nt x; k for all k K M and t x Q If in addition there is a k that satisfies the following equation OkV n t x min OkVn t x } Then Vn t x V nt x; k and k k is an optimal controller Proof: See Appendix C This theorem provides the sufficient condition of optimality for the n-th cumulant case Remarks: The control of the different cumulants leads to the different shapes of the distribution of the cost function In Theorems 1 and we generalize the traditional optimal control for the stochastic system where the first cumulant the mean value of the cost function is minimized to the n-th cumulant control method which minimizes any cumulant of a cost function Therefore we can control the distribution of the cost function and corresponding system performance by controlling specific order of the cumulant IV OPTIMAL CONTROLLER FOR THE N-TH COST CUMULANT CONTROL In this section we seek a method to construct the n-th cumulant controller From 1 we assume that ft xt kt xt gt x Bt xkt x and from 3 we assume that Ls xs ks xs ls xs k s xsrtks xs where the matrices Rt > Bt x are continuous real matrices and l : Q R is C Q and satisfies the polynomial growth conditions and g : Q R n is C 1 Q and satisfies linear growth condition This special form for the control action allows us to find a linear controller even when the state is nonlinear with nonquadratic cost function Theorem 3: Assume the conditions in Theorem are satisfied The nonlinear optimal n-th cumulant controller is of the following form k t x 1 R1 B V 1 t x V t x γ γ 3 V 3 t x γ n V n t x 1 with terminal condition Vn t F x where γ γ 3 γ n are the Lagrange multipliers Proof: See Appendix D Remarks: When we substitute k back to the HJB equation for the n-th cumulant and the partial differential equations from the first to n 1-th cumulants we obtain n partial differential equations By solving these equations we obtain the optimal statistical control problem and determine the corresponding Lagrange multipliers γ to γ n In the next section we will study the method to solve the HJB equations for the n-th cumulant statistical control problems V SOLUTION OF HJB EQUATION USING NEURAL NETWORKS In Section III we proved that the n-th cumulant minimization problem is equivalent to solving HJB equation 8 and in Section IV we constructed the n-th optimal controller However partial differential equation 8 is difficult to solve directly especially for the nonlinear systems We utilize the neural network NN method approach in 1 to approximate the value functions in 8 and then find the solution of the HJB equations Our new neural network algorithm utilized power series basis functions where the coefficients or weights of the network are tuned to solve the HJB equations In the authors utilized a radial basis functions in their network NN method based on power series function has an important property of differentiability and with timevarying weights can approximate time-varying continuous functions 9 The activation basis functions are polynomials and the NN weights associated with the basis functions are optimized by solving the system minimization problem over a compact region For detailed NN solution see 1 VI NUMERICAL EXAMPLES In this section we demonstrate our n-th cost cumulant statistical control using an example We show that our approach yields stabilizing feedback control for the nonlinear system In the example the input functions for the value functions are formulated as in 16; δ L x M n i x j where M is an even-order of the approximation L is the number of neural network functions n is the system dimension All the terms from the expansion of the polynomial are used to represent δ Lx Oscillator System Application We will study the third cumulant V 3 statistical control for a twodimensional nonlinear oscillator system given in 3 The general nonlinear dynamics equations are ẋ 1 t x t ẋ t x 1t π arctan5x1t 5x 1t 1 5x 1 t 4x t 3ut 3 where x 1 t and x t are set of state vectors and ut is the control The system in 3 is a deterministic system thus we introduce Gaussian noise into the system to add the process noise The stochastic system is represented as x dx1 t t dx t x 1 t π arctan5x 5x 1t 1t dt 1 5x 1 t dt utdt Edwt 4x t 3 4 where the state variables x are defined as: xt x 1t x t Here we minimize the third cumulant V 3 while constraining the second cumulant V and first cumulant V 1 We assume that E in 4 is a 1 constant vector of ones dwt in 4 is a Brownian motion with mean Edwt} and variance Edwtdwt } W 1

5 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 5 The cost function is quadratic and given as Jt xt ut tf t x t u t dt ψ 1 xt F 5 where ψ 1xt F is the terminal cost We will suppress the argument t for brevity We utilize the neural network function defined in Section V to approximate the value functions in the HJB equation 8 We selected a polynomial function of sixth-order in the state variable ie x is 6-th order with length of the neural network series function L 15 terms These functions are given as follows NN weights NN weights at γ 1 γ Timesec a Neural Network NN Weights V V 1 at γ from to 1 γ γ b First Cumulant Value Function - V 1 versus γ V 1L x 1 x w 1 x 4 1 w x 4 w 3 x 1 w 4 x 165 V at γ from to 1 γ V 3 at γ from to 1 γ 3 1 w 5x 1x w 6x 1x w 7x 3 1x w 8x 1x 3 w 9 x 6 1 w 1 x 6 w 11 x 5 1x w 1 x 4 1x w 13 x 3 1x 3 w 14 x 5 x 1 w 15 x 4 x V 145 V 3 V L x 1 x w 16 x 4 1 w 17 x 4 w 18 x w 19 x w x 1x w 1 x 1 x w x 3 1x γ γ w 3x 1x 3 w 4x 6 1 w 5x 6 w 6x 5 1x w 7x 4 1x w 8x 3 1x 3 w 9x 5 x 1 w 3x 4 x 1 7 V 3L x 1 x w 31 x 4 1 w 3 x 4 w 33 x 1 w 34x w 35x 1x w 36x 1x w 37x 3 1x w 38x 1x 3 w 39x 6 1 w 4x 6 w 41x 5 1x w 4 x 4 1x w 43 x 3 1x 3 w 44 x 5 x 1 w 45 x 4 x 1 8 In the simulation the asymptotic stability region for states was arbitrarily chosen as 4 x 1 4 and 3 x 3 The final time t F is 3 s and we integrate backwards in time to solve for the weights The initial state condition is given as x 3 From Fig 1a we note that the neural network weights converge to constants which are the optimal weights Figs 1b to 1d show the first second and third cumulant value functions From Fig 1b it is observed that the value function V 1 increases when the value of γ increases For V and V 3 from Figs 1c and 1d the value functions show a decrease with increase in γ These graphs show that the control engineer can choose appropriate γ to shape the distribution through the 1 st nd 3 rd cumulants Fig 1e shows the state trajectory with noise influence with variance σ 1 The states are bounded and show convergence to values close to zero Fig 1f shows the phase trajectory which indicates relatively low influence of the process noise which has a small magnitude The optimal control trajectory u is shown in Fig and converges to zero It should be noted that the optimal controller is solved by selecting γ and γ 3 where the value functions are minimum which in our case was γ 1 and γ 3 1 as shown in Figs 1b to 1d In addition we have the design freedom in γ values selection to enhance system performance VII CONCLUSION In this paper we analyzed the statistical optimal control problem using a cost cumulant approach The control of cost cumulants leads to the shaping of the distribution of the cost function which improves the system performance We developed the n-th cumulant control method which minimizes the cumulant of any order for a given stochastic system The HJB equation for the n-th cumulant minimization is derived as necessary conditions of the optimality The verification theorem which is a sufficient condition for the n-th cost cumulant case is also presented in this paper We derived the optimal statistical controller for n-th cumulant statistical control We State c Second Cumulant Value Function - V versus γ State Trajectory at γ 1 γ Time sec e State trajectory at initial condition x 3- x 1 x d Third Cumulant Value Function - V 3 versus γ x x 1 x phase portrait at γ 1 γ x 1 f Phase Trajectory at initial condition x 3- Fig 1: Oscillator System Neural Network Weight Value Functions State and Phase Trajectory FeedBack Control Input Optimal Control at γ 1 γ Time sec Fig : Control Trajectory at initial condition x 3- used neural network approximation method to find the solutions of the HJB equation Neural network method converts the HJB partial differential equation into the ordinary differential equation and solves them numerically A nonlinear oscillator system is considered as an example The results for the third cost cumulant statistical control is presented and discussed From the results we showed that a higher order cumulant statistical controller shapes the performance of the system through the cost cumulants REFERENCES 1 K D Pham M K Sain and S R Liberty Statistical Control for Smart base-isolated builings via cost cumulants and output feedback paradigm in Proc of the American Control Conference Portland OR Jun 5 pp

6 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 6 C-H Won Cost Moment Control and Verification Theorem for Nonlinear Stochastic Systems in Proc IEEE Conference on Decision and Control San Diego CA Dec 6 pp C-H Won R W Diersing and B Kang Statistical Control of Control- Affine Nonlinear Systems with Nonquadratic Cost Function: HJB and Verification Theorems Automatica vol 46 no 1 pp M K Sain Control of Linear Systems According to the Minimal Variance Criterion A New Approach to the Disturbance Problem IEEE Transactions on Automatic Control vol AC-11 no 1 pp Jan P Whittle Risk-Sensitive Linear/Quadratic/Gaussian Control Advances in Applied Probability vol 13 pp J Si A Barto W Powell and D Wunsch Eds Handbook of Learning and Approximate Dynamic Programming ser IEEE Press Series on Computational Intelligence Piscataway NJ: IEEE Press 4 7 D H Jacobson Optimal Stochastic Linear systems with Exponential Performance Criteria and Their Relationship to Deterministic Differential Games IEEE Transaction on Automatic Control vol AC-18 pp M K Sain and S R Liberty Performance Measure Densities for a Class of LQG Control Systems IEEE Transactions on Automatic Control vol AC-16 no 5 pp Oct S R Liberty and R C Hartwig On the essential quadratic nature of LQG control performance measure cumulants Information and Control vol 3 no 3 pp Oct T Runolfsson The Equivalence Between Infinite-Horizon Optimal Control of Stochastic Systems with Expoential-of-Integral Performance Index and Stochastic Differential Games IEEE Transaction on Automatic Control vol 39 no 8 pp M K Sain C-H Won B F Spencer Jr and S R Liberty Cumulants and Risk-Sensitive Control: Cost Mean and Variance Theory with Application to Seismic Protection of Structures in Advances in Dynamic Games and Applications Annals of the International Society of Dynamic Games Boston MA: Birkhauser pp B Kang and C-H Won Statistical Optimal Control using Neural Networks Proceeding of International Symposium on Neural Networks ISNN Lecture Notes in Computer Science: Advances in Neural Networks ISNN 11 Guilin China vol 6675 pp May K D Pham Statistical Control Paradigm for Structure Vibration Suppression PhD dissertation University of Notre Dame Notre Dame IN Aug P Werbos A Menu of Designs for Reinforcement Learning Over Time in Neural Networks for Control W Miller R Sutton and P Werbos Eds Cambridge MA: MIT Press 199 pp R Beard G Saridis and J Wen Galerkin Approximations of the Generalized Hamilton-Jacobi-Bellman Equation Automatica vol 33 no 1 pp Dec Z Chen and S Jagannathan Generalized Hamilton-Jacobi-Bellman Formulation-Based Neural Network Control of Affine Nonlinear Discrete-Time Systems IEEE Transactions on Neural Networks vol 19 no 1 pp 9 16 Jan 8 17 T Chen F L Lewis and M Abu-Khalaf A Neural Network Solution for Fixed-Final Time Optimal Control of Nonlinear Systems Automatica vol 43 no 3 pp E G Albrekht On the Optimal Stabilization of Nonlinear Systems PMM - Journal of Applied Mathematics and Mechanics pp W L Garrard Suboptimal Feedback Control for Nonlinear Systems Automatica vol 8 pp P V Medagam and F Pourboghrat Optimal Control of NonLinear Systems using RBF Neural Network and Adaptive Extended Kalman Filter in Proc of the American Control Conference St Louis Missouri Jun 9 pp W H Fleming and R W Rishel Deterministic and Stochastic Optimal Control New York NY: Springer-Verlag 1975 C W Gardiner Handbook of Stochastic Methods for Physics Chemistry and the Natural Sciences nd ed New York NY: Springer-Verlag W H Fleming and H M Soner Controlled Markov Processes and Viscosity Solutions New York NY: Springer-Verlag P J Smith A Recursive Formulation of the Old Problem of Obtaining Moments from Cumulants and Vice Versa The American Statistician no 49 pp B Kang Statistical Control using Neural Network Methods with Hierarchical Hybrid Systems PhD dissertation Temple University Philadelphia PA May 11 6 C-H Won Nonlinear n-th Cost Cumulant Control and Hamilton- Jacobi-Bellman Equations for Markov Diffusion Process in Proc IEEE Conference on Decision and Control Seville Spain Dec 5 pp R W Diersing H Cumulants and Control PhD dissertation University of Notre Dame Notre Dame IN Aug 6 8 A Stuart and J K Ord Eds Kendall s Advanced Theory of Statistics ser Distribution Theory New York NY: Oxford University Press I W Sandberg Notes on Uniform Approximation of Time-Varying Systems on Finite Time Intervals IEEE Transactions on Circuit and Systems-1:Fundamental Theory and Applications vol AC-45 no 8 pp Aug J A Primbs V Nevistic and J C Doyle Nonlinear Optimal Control: A Control Lyapunov Function and Receding Horizon Perspective Asian Journal of Control vol 1 no 1 pp 14 4 Mar 1999 APPENDIX A PROOF OF LEMMA 1 Let j i 1 the right hand side of 7 becomes k1 k 1! j 1!k j! Vkj 9 Then when k is odd there are even number of terms in the above summation We can rewrite the above summation in pairs For example the first term and last term the second term and second last term and so on We represent the sum of each pair as follows: k 1! Vkj j 1!k j! k 1! Vj σw σ Vkj k j 1!j! jk! Vkj kj!k j! k jk! Vj σw σ Vkj kk j!j! jk! k jk! Vkj kj!k j! kk j!j! k! Vkj j!k j! Thus for j 1 to n 1 k1 1 k! j!k j! k1 Vkj k 1! j 1!k j! Vkj 3 31 When k is even we have the same result except for the term when In this case we have k 1! k/ 1!k k/! which can be written as follows k 1! k/ 1!k k/! 1 kk 1! k/!k/! j k Vk/ Vk/ σw σ Vk/ σw σ Vk/ σw σ Vk/ Vk/ Therefore the lemma 7 is proved 5

7 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 7 APPENDIX B PROOF OF THEOREM 1 We use mathematical induction method assuming that the theorem holds for the second third n 1-th cumulant case then we will show that the theorem also holds for the n-th cumulant case Let Vn be in the class of Cp 1 Q C Q Apply the backward evolution operator Ok to the recursive formula 5 we have Ok V n Ok M n n 1! OkM n1iv i1 3 Assume that M jt x Cp 1 Q C Q is the j-th admissible moment cost function Then M j t x satisfy the following HJB equation Ok M jt x jm j1t xlt x k 33 for t x Q where M j1t x is the j1-th admissible moment cost function and the terminal condition is given as M j t F x ψxt F 3 From 33 we have OkM n nm n1 L 34 Let M it x V jt x Cp 1 Q C Q where i and j are nonnegative integers then Ok M i t xv j t x Ok M i t x V j t x M i t xok V j t x Mit x Vjt σ t x W σ t x x 35 with boundary condition M i t F x ψxt F V j t F x 7 By letting i n 1 i j i 1 in 35 we obtain OkM n1i V i1 OkM n1i V i1 M n1iokv i1 σw σ Vi1 36 Substitute equations into 3 then we have OkVn n 1! nm n1l M n1i OkV i1 37 Use 34 again for OkM n1i in 37 results in OkVn n 1! nm n1 L n 1 im iv i1l M n1iok V i1 OkM n1iv i1 38 From the third term of the right hand side of 38 we split the summation of the following term n 1! n 1 im i V i1 L into two parts one is the summation from i to i n 3 the other is the term when i n such that n 1! n 1 im iv i1l n3 n 1! M i V i1 L n 1M V n1 L n Furthermore we split the summation into two parts such that n 1! M n1iokv i1 n 1! M n1iokv i1 n 1! M n1 OkV 1 M n1i OkV i1 Using the above results the fact that M is an identity and OkV 1 L we rewrite 38 as OkV n nm n1 L n 1V n1 L n3 n 1! M n1l MiV i1l n n 1! Mn1i OkV i1 n 1! 39 The second third fourth and fifth terms of 39 can be combined as n3 n 1L M n1 n! Mi V i1 Vn1 n which equals to zero according to the conversion formula between moment and cumulant 4 Therefore 39 becomes OkVn n 1! n 1! Mn1i OkV i1 4 For the second term in the right side of 4 we assume that Theorem 1 holds from the nd to n 1-th order cumulant case We have n 1! Mn1i OkV i1 Vj n 1! j M n1i 1 i i 1! j!i 1 j! 41 Then we use 7 to the above equation The following equation is obtained n 1! Mn1iOkV i1 n 1! Using the formula in 8 that M n1i i1 1 j!i j! M i V j j!i j! M ij then we have 4

8 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 8 M i i M i V j V j i j!i j! M V j ij 43 Thus 46 becomes Then we apply 43 to M n1i M n1i n1i Applying 44 to n1i resulting in M n1i V j V j n 1 j!n 1 i j! Mn1ij V j we have the following equation n1i n 1 j!n 1 i j! M n1ij Therefore 4 becomes OkVn n1i σw σ Vj1 Vj n 1! i1 M n1i j n 1 j!n 1 i j! M n1ij n 1! Vj j!i j! 46 From 46 we notice that both the second and third term on the right hand side contain the moment from M x } where x 1 Therefore it is feasible to combine two summations with respect to M x and simplify the equation First we split the right term of 45 in 46 into two parts such that n1i i i n 1 j!n 1 i j! Mn1ij n 1 j!n 1 i j! M n1ij n 1 j!n 1 i j! Mn1ij M Vn1i i Vj Vj n 1 Vj j!n 1 i j! Mn1ij Vn1i n 1 j!n 1 i j! Mn1ij Vj Vj σw σ Vi1 jn1i 47 OkVn n 1! i1 M n1i j σw σ Vj1 j!i j! n 1! i n 1 Vj j!n 1 i j! Mn1ij n 1! Vn1i 48 In 48 for the second term when i varies from 1 to n the corresponding M x varies from M to M 1 In the third term when i varies from to n and j varies from 1 to n i the corresponding M x also takes the value from M 1 to M Now we combine the M x terms from the second and third terms of 48 From the second term of 48 we can expand the first summation in the following manner n 1! M n1i i1 j 1 1 n 1! σw σ Vj1 i1 j σw σ Vj1 j!i j! j!i j! n 1! M n1i M n1i i M n1i i i1 j j!i j! n 1! i1 j 49 Each term on the right hand side of 49 contains one of the moment term M x } where x 1 Then we can find any specific M k k 1 n by letting k n 1 i which has the following form n 1! k!n 1 k! Vn1kj k j Vn1kj k j σw σ Vj1 n 1 k! j!n 1 k j! M k n 1! k!j!n 1 k j! 1 M k j!i j! 5 We examine the third term of 48 for the same M k to see if we can combine the coefficients of M k with the right hand side of 5 For the third term of 48 we rewrite this summation in terms of M k

9 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 9 by letting k n 1 i j as follows i n 1! M n1ij Vj M k Vj n 1! n 1 j!n 1 i j! i n 1 j!k! 51 Because k n 1 i j then n 1 k i j Because i and j are not constants in 51 we can set a combinations of i and j which satisfy the condition n 1 k i j and are listed as follows i j n 1 k i 1 j n k i n 1 k j The sum of these i and j pair are equal to the same M k for 51 Then we substitute the above i and j back to the right hand side of 51 and pick up all the M k terms we have n 1! n 1! Vn1k σw σ V1!n 1! n 1 k!k! Vk σw σ V M k M k n 1! n! 1!n! n k!k! n 1! k 1! V1 σw σ Vn1k n k!k 1! 1!k! n 1! k! V σw σ Vnk M k n 1 k!k! 1!k! M k When we combine M k term it is further rewritten as n 1! Vn1k σw σ V1!n 1 k!k! n 1! Vk σw σ V 1!n k!k! } n 1! V1 σw σ Vn1k M k n k!1!k! Now we define an index p and rewrite the above equation in the following summation form k n 1! p!n 1 k p!k! σw σ Vp1 p Vn1kp M k 5 Compare the right hand side of 5 and 5 we notice that they are equal to each other except for the sign and the index character Thus when we combine M k k 1 n terms from the second and third terms of 17 they cancel with each other Therefore 48 becomes the following equation OkVn n 1! Using 7 53 is written as OkV n 1 Vit x n1 Vn1i n! n Vnit σt xw σt x x 53 The theorem is proved APPENDIX C PROOF OF THEOREM Assume there is a Vn t x Cp 1 Q C Q which satisfies the following equation min Vi t x n1 n! n Ok Vn t x 1 σt xw σt x Vni t x } Applying 7 to 54 is equivalent to min Ok Vn n 1! t x } Vn1it x Vi1t σt xw σt x x Then we substitute 9 into 55 and suppress the arguments we have n 1! min Ok M n Ok M n1iv i1 } n 1! Vn1i 56 Using the result of the second term in the curly bracket of the left hand side of 56 is written as n 1! Ok M n1i V i1 n 1! MiV i1l n n 1! Mn1i Ok V i1 n 1! 57 Then we split the first term on the right hand side of 57 into two parts one is from i to n 3 and the other is i n And we split the second term on the right hand side of 57 into two parts one is i and the other is from i 1 to n We rewrite 57 as follows n3 n 1! Ok Mn1iVi1 n 1! n M iv i1 L n 1V n1 L M n1 L n 1! Mn1iOk V i1 n 1! 58

10 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 1 Furthermore the term n 1V n1 L M n1 L of equation 57 has the following property 4 n 1V n1l M n1l MiV i1l nm n1l Therefore we have n3 n 1! Ok M n1i V i1 n3 n 1! MiV i1l n n3 n 1! n n 1! Mi V i1 L nm n1 L n n 1! Mn1i Ok V i1 n 1! n 1! nm n1 L Mn1i Ok V i1 n 1! Substituting 6 back to 56 we have min Ok M n nm n1l Mn1iOk V i1 n 1! n 1! M n1i } n 1! Vn1i From 4 and 43 in Theorem 1 we know the summation of the third and fourth term on the right hand side of 61 can be represented as the following n 1! Mn1i Ok V i1 n 1! n 1! M n1i i1 1 j!i j! j n 1! M n1ij Vj n1i σw σ Vi1 n 1 j!n 1 i j! 6 Furthermore from 47 we notice that n1i i n 1 j!n 1 i j! M n1ij Vnii n 1 j!n 1 i j! M n1ij Substitute 63 into 6 we have Vj Vj n 1! Mn1iOk V i1 n 1! n 1! i σw σ Vj1 M n1i i1 j j!i j! n 1 j!n 1 i j! M n1ij n 1! Vn1i n 1! Vj From the proof of Theorem 1 we know that the sum of the second and third term of 48 is equal to zero Furthermore the first term on the right hand side of the above equation is same as the second term in 48; and the second term on the right hand side of the above equation is same as the third term in 48 Therefore the summation of these two terms equals to zero and 64 becomes n 1! Mn1i Ok V i1 n 1! n 1! Vn1i 65 Substitute 65 back to 61 then after manipulation 61 becomes min Ok Mn t x nm n1 L } which satisfies the sufficient condition for optimality for control in 33 Thus the theorem is proved min Vs t x APPENDIX D PROOF OF THEOREM 3 From Theorem 1 the optimal controller k satisfies the following HJB equation Ok Vn t x 1 n1 n! s!n s! s1 σt xw σt x Vns t x } 66 with the constraint condition that the first second n 1-th cumulant are admissible and satisfy the following partial differential

11 SUBMITTED TO IEEE TRANSACTION ON AUTOMATIC CONTROL 11 equations Ok V 1 t x Lt x k V1t Ok V x V1t t x σt xw σt x x Ok V n1 t x 1 Vs t x n 1! s!n s 1! s1 σt xw σt x Vns1 t x Therefore we construct a new function F t x k such that n1 n! s!n s! 67 F t x k Ok Vn t x 1 s1 Vs t x σt xw σt x Vns t x λ 1 Ok V 1t x Lt x k V1 t x λ Ok V t x σt xw σt x V1 t x V1 t x λ 3 Ok V 3t x 3 σt xw σt x V t x λ n1 Ok V n1 t x 1 n 1! s!n s 1! s1 Vs t x σt xw σt x Vns1 t x 68 where λ 1 λ λ 3 λ and λ n1 are Lagrange multipliers Then by using the method of Lagrange multiplier we take the derivative of F t x k with respect to We can calculate the optimal controller k from 69 as follows k 1 R1 B V 1 t x λ V t x λ3 V 3 t x λ 1 λ 1 Now let Then we have λn1 V n1t x 1 Vn t x λ 1 λ 1 γ λ λ 1 γ 3 λ3 λ 1 γ n1 λn1 λ 1 γ n 1 λ 1 k 1 R1 B V 1 t x V t x γ γ n1 V n1t x γ n V n t x V 3 t x γ k λ 1 λ λ 3 λ λ n1 and equate result to zero We obtain the following equations F t x k k λ 3 B V 3 t x Rλ 1 k λ 1 B V 1t x λ n1 B V n1 t x F t x k Ok V 1 t x Lt x k λ 1 F t x k Ok V t x λ V1t x σt xw σt x V1t x F t x k Ok V n1 t x 1 λ n1 Vst x s1 σt xw σt x Vns1t x λ B V t x λ n B V n t x n 1! s!n s 1! 69