Optimal Switching of DC-DC Power Converters using Approximate Dynamic Programming

Transcription

1 Optimal Switching of DC-DC Power Conerters using Approximate Dynamic Programming Ali Heydari, Member, IEEE Abstract Optimal switching between different topologies in step-down DC-DC oltage conerters, with non-ideal inductors and capacitors, is inestigated in this paper. Challenges including constraint on the inductor current and oltage leakages across the capacitor due to switching) are incorporated. The objectie is generating the desired oltage with low ripples and high robustness towards line and load disturbances. A preiously deeloped tool which is based on approximate dynamic programming is adapted for this application. The scheme leads to tuning a parametric function approximator to proide optimal switching in a feedback form. No fixed cycle time is assumed, as the cycle time and the duty ratio will be adjusted on the fly in an optimal fashion. The controller demonstrates good capabilities in controlling the system een under parameter uncertainties. Finally, some modifications on the scheme are conducted to handle optimal switching problems with state jumps at the switching times. Index Terms- Step-down dc-dc oltage conerter, approximate dynamic programming, optimal switching with state jump, neural networks. I. INTRODUCTION The idea in the widely used DC-DC buck conerters is switching between different topologies inoling an inductorcapacitor filter such that a desired oltage is generated at the output. The problem, howeer, is finding the right switching time, such that the output has a desired steady state behaior e.g., in terms of low harmonic distortions) as well as a suitable dynamic performance e.g., low sensitiity towards disturbances) [] [3]. To hae a short literature surey on this rich field, one may consider state space aeraging scheme [], linear quadratic regulator LQR) control [5], [], fuzzy logic control [7], [8], adaptie control [9], nonlinear control schemes [], including sliding mode control [], [], and optimization based schemes, including model predictie control MPC) [3], [3] [], and dynamic programming [7] [9], used for control of oltage conerters. The common practice for control of the conerters is Pulse Width Modulation PWM), where a fixed cycle time is selected and the duty ratio the ratio of the cycle that a switch is closed, hence, a specific topology is selected) is adjusted, [3] [7], [], [3]. Another practice, howeer, is finding switching times without a fixed cycle, [], [7] []. The challenges in control of power conerters are numerous, including desired low sensitiity towards ariations in the source oltage line disturbances) and in the load current load disturbances), existing parameter uncertainties in modeling the system, enforcing constraints on the inductor current, A. Heydari is an Assistant Professor of Mechanical Engineering with the Southern Methodist Uniersity, Dallas, TX, aheydari@smu.edu. This material is based upon work supported by the National Science Foundation under Grant No and incorporation of oltage leaks across the capacitors after each switching. While each of the aailable method in the literature tackles some of these challenges, this study is aimed at deeloping a scheme which has the capability of addressing all of these concerns. The idea is utilizing approximate dynamic programming ADP), sometimes referred to by reinforcement learning RL) or neuro-dynamic programming NDP), for soling this problem. Interested readers are referred to the chapters in Ref. [] for reiews of the ADP/RL field from perspecties of different authors of the chapters. ADP/RL has shown to hae many adantages oer other competitors in different applications, including power systems, both in oltage conerters [7] [9] and other problems of the discipline [] []. This study is mainly focused on inestigation of an interesting application of a general purpose tool deeloped preiously by the author for switching problems, namely, [5], []. The application is the oltage conerter problem presented in [7] and []. Using the tool, the dynamic and steady state performance of the switching controller is shown to be promising in the oltage conerter under line disturbances and parameter uncertainties. Howeer, in inestigating the sensitiity to the load current, the controller is obsered to lead to an undesired steady state error. An idea is proposed for measuring and feeding the load current into the controller, to find the optimal switching schedule as a function of the current, i.e., haing both the load current and the states as the feedback signals for the controller. After demonstrating the superior performance of this controller in handling the disturbance, another challenge is inestigated, namely, oltage leakage across the capacitor after each switching. In other words, the problem is changed to a switching problem with state jump at the switches. Incorporating the state jump is shown to be a non-triial task as it leads to fundamental changes in the controller. As a theoretical contribution of this study, a new learning scheme is proposed to accommodate the state jumps and the results are applied to the problem with desired outcomes. Finally, comparing this work with [5] and [], the main difference is this study focuses on application while the cited two references are theoretical work. Also, the two papers do not address switching problems with state jump, which is addressed in here. The closest studies to this work are [7] and [8], where relaxed dynamic programming was used for finding the approximate optimal switching schedule. The differences of this work compared with them are ) proiding detailed analyses of capabilities of the ADP-based controller in facing different existing challenges in oltage conerters, ) deeloping an idea for reducing the sensitiity of the result to load disturbances, and 3) incorporating oltage leakages through deeloping a

2 noel controller, i.e., a controller for optimal switching with state jumps. The rest of this study is organized as follows. The switching problem of a power conerter is briefly discussed in Section II. Afterwards, the theory of the method selected for finding the controller for the switching problem is detailed in III, followed by its implementation and numerical analyses in Section IV. The issue of sensitiity to load disturbances is addressed in Section V. The new scheme for incorporating switching problems with state jumps is theoretically deeloped and numerically analyzed in Section VI. Comparison of the results with two other methods is presented in Section VII. Finally, some concluding remarks are gien in Section VIII. II. DC-DC CONVERTER Consider the DC-DC buck conerter inestigated in [7] and [] shown in Fig.. The switch on the left of the inductor is subject to be shifted between the zero and one positions, each of which leads to a different topology for the circuit. The inductor with the inductance of L) is assumed to be non-ideal, hence, has the resistance of r l, and the capacitor with the capacitance of C) also has the internal resistance of r c, the current source i o represents the load on the conerter, and finally, the oltage input/source is gien by oltage source V s. Denoting the oltage across the capacitor with and the current through the inductor with, the state ector is selected as x := [, ] T, motiated by [7] and []. The dynamics of the system can be modeled by the two modes based on the two alues of the switching input {, }, corresponding to the respectie positions of the switch. ẋ = d [ ] [ ] Vc C = i o ) dt L r c + r l ) + r c i o + V s ). ) Note the exogenous signals i o and V s present in the dynamics. Freezing the signals at constant alues i.e., no load and line disturbances) leads to a switching system with two autonomous subsystems. The approach for controller design in this work calls for a discrete-time dynamics. Hence, the aboementioned system may be discretized using a small enough sampling time denoted with t, such that the resulting discrete-time system approximates the continuous-time dynamics accurately. Denoting the state ector at discrete time k with x k and using forward Euler integration for discretization, the system reads x k+ = x k + tẋ k, ) where the time-deriatie of the state, i.e., ẋ k, is gien by ), based on the position of the switch. The objectie is manipulating the switch such that the oltage across the capacitor tracks some constant reference oltage V ref, where V ref < V s, i.e., a step-down in the oltage is resulted. Finding an optimal switching schedule is sought in this study, hence, a performance measure or cost function is required to be selected. Motiated by [] the following cost function is selected J = γ k k V ref ), 3) k= V Fig.. Schematic of a step-down DC-DC oltage conerter picture adapted from []). where k denotes the oltage of the capacitor at time k and the discount factor, denoted with γ, ] is incorporated to aoid an unbounded cost. Minimizing 3) while enforcing the desired behaior of tracking V ref, penalizes the oltage ripples, which is also desired. The next section briefly reiews the approach selected for soling the switching problem of minimizing 3) subject to the dynamics gien by ) and ). III. OPTIMAL SWITCHING USING ADP A. Problem formulation A discrete-time switching system with M autonomous subsystems may be presented by x k+ = f x k ), k Z +, V, ) where the dynamics of the modes are gien by f : R n R n, V := {,,..., M} and the dimension of the state ector x k is gien by positie integer n. The discrete time index is denoted by subscript k in x k and subscript in f.) corresponds to the respectie mode/subsystem. Finally, the set of non-negatie integers is denoted with Z +. A switching schedule identifies k, k Z +, where the actie mode at time k is denoted with k. The optimal solution is the switching schedule that minimizes the discounted cost function J = γ k Qx k, k ), 5) k= where function Q : R n V R +, continuous with respect to x, penalizes the state error the difference between the actual alue and the desired constant alue for the state ector) during the horizon. Function Q.,.) may generally be dependent on the actie mode, for the purpose of haing the flexibility of assigning different penalties to different modes. In 3), howeer, Qx, ) is defined as x) V ref ) for a constant V ref, where x) denotes the first state element, which is. Finally, set R + denotes the non-negatie real numbers and γ, ] denotes the discount factor. B. Proposed solution If the optimal alue function, sometimes called optimal costto-go, is known, the optimal solution to the switching problem follows. The optimal alue function is a function of state that gies the total cost of ealuating the cost function along the state trajectory propagated from the gien current state using the optimal switching schedule. Denoting the alue function with V : R n R +, one has V x ) = γ k Q x k, x k) ). ) k= V

3 where : R n V denotes the optimal feedback switching schedule the optimal switching policy). Moreoer, the state trajectory resulting from the optimal decisions is denoted with x k, k =,,,... and x := x. Denoting x k ) with k, Eq. ) leads to V x k ) = Qx k, k) + γv f k x k ) ), k Z +. 7) By the Bellman principle of optimality [8], one has Qx, ) + γv f x) )), x. 8) V x) = min and the optimal mode, in a feedback form, is gien by x) = arg min Qx, ) + γv f x) )), x. 9) The problem, hence, is finding the optimal alue function, gien by Eq. 8). An idea for soling this equation is using alue iteration VI), [5], where starting from an initial guess on V x), one iterates through V j+ x) = min Qx, ) + γv j f x) )), x X, ) until it conerges, where the iteration index is gien by j. This is done through selecting a function approximator, e.g., neural networks, for approximating the alue function within a compact set, denoted by X, which represents the domain of interest. It should be noted that the solution will be alid only if the entire state trajectory remains within X, hence, X needs to be selected carefully and large enough. The implementation of the VI is as simple as selecting multiple random sample states x X and ealuating the right hand side of ) at each sample. These alues can be used as targets for learning the left hand side, for example using least squares. Gien V j.), the iteration leads to V j+.) and this successie approximation continues until the parameters of the function approximator conerge. The following theorem proides sufficient conditions for this conergence. Before that, it is worth mentioning that exact reconstruction of the right hand side of Eq. ) with the left hand side, that is, exact approximation is generally not possible. Hence, approximation errors will be introduced into the problem. Interested readers are referred to [9] [3] for inestigation of effects of these errors on the quality of the result. Theorem. If there exists at least one switching policy using which cost function 5) remains bounded for all x X and if the initial guess V.) in the alue iteration gien by ) is selected as V x) =, x X, then, the iterations conerge to the optimal alue function gien by Eq. 8). Proof : A conergence proof was presented in [5] for undiscounted cost functions. The idea was establishing an analogy between the iterations of the VI and the horizon length of a finite-horizon optimal control problem with a fixed final time. That idea can be simply extended to the case of haing discounted cost functions subject to this study. Let the cost function with the finite horizon of N time steps be gien by J N = N k= γ k Qx k, k ). ) The finite-horizon problem is defined as minimizing J N subject to the dynamics gien by ). Once the final time is fixed, the alue function and the control policy become timedependent [8], [3], i.e., they may be denoted with V, ) and, ), respectiely, where the second argument is the number of remaining time steps, or time-to-go. Let the optimal finite-horizon alue function gien state x and time-to-go τ M := {,,..., N} be denoted by V : R n M R +, where τ V x, τ) = γ k Q x,τ k, x,τ k, τ k)), ) x,τ k k= := f x,τ k, x,τ k, τ k ))), k such that k τ, and x,τ := x, τ. In other word, the summation is ealuated along the trajectory generated by applying the time arying switching policy, τ k) at time k. Clearly V x, ) =, x, 3) and by Bellman equation for fixed-final-time problems, one has [8], [3] V x, τ + ) = min Qx, ) + γv f x), τ )), ) x, τ M {N}, and x, τ + ) = arg min Qx, )+ γv f x), τ )), x, τ M {N}. 5) Selecing V x) =, x in initiating VI, and comparing Eq. ) with ) it follows that V x, j) = V j x), x, j M. ) In other words, the immature alue function at the jth iteration of VI is identical to the optimal alue function of the fixedfinal-time problem of minimizing ) with the final time of j. On the other hand, the sequence of functions {V j.)} j= is monotonically non-decreasing. This can be seen by induction, as V x) V x), x, gien V x) =, x, and V x) = min Qx, ), x, which follows from ) after setting V x) =. Assuming V j x) V j x), x, it follows that V j x) V j+ x), x, because the argument subject to minimization in V j x) = min Qx, ) + γv j f x) )), x X, 7) will be smaller than or equal to that of Eq. ) at any gien x X. This proes the monotonicity of {V j.)} j=. This sequence is also upper bounded by the finite cost-to-go of the assumed existing switching policy as each element of the sequence is an optimal cost-to-go gien Eq. ). Hence, the sequence conerges to some V.) = V., ) as eery nondecreasing and upper bounded sequence conerges, [33]. It can be seen that V., ) = V.) by comparing ) with 5), because, J N J as N. Finally, once the function approximator is trained offline, it may be used for online feedback control. This is done through finding the approximate) optimal schedule in a feedback form through 9), where V.) is replaced with the tuned function approximator. This controller will be implemented on the 3

4 Weight Elements 8 8 Learning Iterations Fig.. Eolution of parameters/weights during offline learning. oltage conerter in the next section. Before concluding this section it is worth mentioning that each V j.) is desired to be continuous ersus its input, in order to guarantee possible uniform approximation [3] using parametric function approximators. Gien the switching between different s in the right hand side of Eq. ), this continuity is not obious, een for problems with continuous Q., ) and f.) for any gien. The desired continuity, howeer, is proed to exist, [5]. IV. IMPLEMENTATION Implementation of the scheme in practice will inole an offline training stage, where the parameters of the controller are tuned using Eq. ). This can be done by a desktop computer ahead of time. After that, the embedded microcontroller in the oltage conerter will be only in charge of finding the optimal mode using Eq. 9). The memory requirement includes storage for parameters of the function approximator, e.g., Neural Networks. The computational load will be as low as ealuating two scalar alued functions resulting from plugging = and = into the term subject to minimization in Eq. 9). Once the two scalar alues are found, comparing them the microcontroller simply outputs the minimizing, which is the optimal mode. In terms of sensors for state feedback, a oltage sensor for measuring and a current sensor for measuring will be needed to construct the state ector to be fed into Eq. 9). A. Variables and design choices The discretization of the continuous-time system ) is done using Euler integration with the sampling time of t =.5s. The alues for the parameters of the circuit are selected based on [], namely, r c =.5Ω, r l =.5Ω, L = 3/π)H, C = 7/π)F, V s =.8V, i o = A, and V ref = V. Function Qx, ) = V ref ) is selected for the state penalizing term in 5), also γ =.97 is selected as a design parameter one can choose other alues). For alue function approximation, polynomial functions made of all the possible non-repeating) combination of the input i.e., the state elements) up to the fourth order are selected. Using this function approximator, the parameters subject to training are the coefficients of the polynomial terms. The method of least squares, detailed in Appendix of [35], is utilized with selecting random states from X =,.8) 5, 5). Domain X is selected based on the expected operation enelope of the system. V) A) Time s) Fig. 3. Simulation results for x := [.5; ] T with V s changing from.8v to.v at t = 5s. B. Simulation Using the learning iteration gien by ), the eolution of the parameters/weights of the function approximator ersus the iterations of the VI is presented in Fig., which shows the desired conergence behaior. The conerged alues are then used for control of the initial condition x := [.5; ] T, with the resulting state trajectories gien in Fig. 3 the first 5s of the simulation). As seen in this figure, the controller successfully generated the desired oltage across the capacitor, with the steady state error of less than.3%. The oltage ripples across the capacitor not noticeable in the plot) were of the amplitudes of around 5 V. Gien small ripples in the current of amplitudes less than A) and the ery small resistance of the capacitor r c =.5Ω) the ripple in the oltage across the load which is r c i o ) + ) also turned out to be ery small less than V in this case). Considering the switching schedule gien in Fig. 3, this desired low ripple output is generated through a high frequency switching, as high as switching once at eery sampling time, at some instants. Finally, it should be noted that the relatiely slow transient response of the system also obsered in []) is due to the selected capacitance and inductance. These alues are selected the same as in [] for facilitating comparison of the results conducted in Section VII. C. Analyzing sensitiity towards line disturbances In order to ealuate the dynamic performance of the controller towards line disturbances, the source oltage was changed from.8v to.v in the last fie seconds of the simulation in Fig. 3. Considering the capacitor oltage, no change is obsered, which demonstrates the great dynamic performance of the controller. Note that, the controller is trained with the constant V s =.8. Howeer, in the control stage, using Eq. 9), when f.) s are supposed to be ealuated, the new V s =. is used in the last fie seconds of the simulation. This leads to a solution which is semi-feedback ersus V s, as the parameters of the controller are tuned based on V s =.8. The suitable performance of the controller in handling the line disturbance leads to the conclusion that the

5 scheduler has a descent robustness towards such disturbances, as long as the new V s is measured and used in the scheduler 9). Finally, before concluding this subsection, it is worth mentioning that this change in the V s is large enough such that if the open loop schedule generated assuming that no change would happen in the oltage source was applied, it would destabilize with oscillations about.v with an amplitude as large as.5v. It is the feedback nature of the controller that incorporated this change and proided a new schedule, as presented in Fig. 3, where the change in the switching schedule is noticeable in the last fie seconds of the simulation. V) A) D. Decreasing the switching frequency It was obsered in Fig. 3 that the controller led to high frequency switching between the modes as high as switching once at eery sampling time, at some instants). This behaior, howeer, is not desired, gien physical limitations on the switching frequency in practice. An idea for decreasing the number of switching, is using a threshold, denoted with τ R +, in online control, such that, switching from an actie mode to another mode is permitted, only if the reward, i.e., the decrease in the right hand side of the scheduler 9), is greater than τ. This is called threshold remedy in [35]. The threshold remedy seems to be similar to haing a term for incurring switching cost in the cost function, []. This change in the cost function, howeer, will lead to some important changes in the structure of the alue function as described in [] and will gie a solution which is optimal with respect to the switching cost as well. The threshold remedy, on the other hand, is much simpler to implement, as no change in the training stage will be done and the threshold will be applied solely in the online control. Selecting τ = 7, as an example, the preious case is re-simulated with the results gien in Fig.. It can be seen that the switching frequency has decreased considerably, compared with Fig. 3, while the desired reference signal is still tracked with barely noticeable ripples i.e., ripples with amplitude less than V across the capacitor and similar ripples across the load). Note that, een though the threshold was not incorporated in the offline tuning stage, the feedback nature of the controller can handle small thresholds, as they will be considered as some disturbances for the feedback controller. Howeer, selecting ery large thresholds leads to the degradation of the performance of the controller. An alternatie idea to switching cost or threshold is enforcing a minimum dwell time. Again instead of rigorously applying a minimum dwell time in soling the optimal control problem, for example using the theory presented in [3], one may enforce the dwell time after each switch on the fly. This can be done by skipping the ealuation of the scheduler 9) after each switch, until the dwell time selected as.s in here) is elapsed. The result from applying this remedy is presented in Fig. 5. These results are desirable also. It is interesting to note that the robustness towards line disturbances is still retained, as the change in V s in the last 5s discussed in subsection IV-C) is applied in both Fig. and Fig. 5, with suitable results. 8 8 Time s) Fig.. Simulation results for x := [.5; ] T with threshold of 7 and changing V s from.8v to.v at t = 5s. V) A) Time s) Fig. 5. Simulation results for x := [.5; ] T with minimum dwell time of.s and changing V s from.8v to.v at t = 5s. E. Analyzing sensitiity towards load disturbances On the topic of dynamic performance of the controller, it is of interest to check the sensitiity of the result to a change in the load current. For this purpose, the simulation is redone, with the difference that instead of changing V s, the i o is changed from A to A at t = 5s and then to 3A at t = s. The results are gien in Fig.. It can be seen that these changes hae led to some steady state errors in, namely, around 3% for each A change in the i o. This change is considerable, compared with the negligible steady state error under the case of no load disturbance, or the case of line disturbances discussed before. Section V presents an idea for improing this behaior, through reducing the sensitiity of the controller to the load current. F. Analyzing sensitiity towards parameter uncertainties While the controller is designed based on the assumption that a perfect model of the dynamics of the system is aailable, one needs to deal with modeling uncertainties in reality. In the DC-DC conerter, uncertainties are typically due to the internal resistance of the inductor and capacitor r l and r c, 5

6 V) A) V) A) Time s) Fig.. Simulation results for x := [; ] T with load current increasing for A after each 5s, without incorporating changing load current in controller design with threshold 7 ). respectiely) or the inductance and capacitance L and C. The point that the controller calculates the switching on the fly and in a feedback form leads to some interesting results in terms of its robustness towards such uncertainties. To demonstrate this performance, it is assumed that the actual internal resistance of the inductor is ten times its nominal resistance used in the offline training and also in decision making using Eq. 9). Moreoer, the actual internal resistance of the capacitor is assumed to be ten times less than the nominal model. Under these assumptions, the simulation with the threshold τ = 7 is done with the result shown in Fig. 7. It can be seen that the performance of the controller is still suitable in terms of tracking the desired oltage, een under such large parameter uncertainties. The same simulation is repeated with modeling uncertainties in the inductance and the capacitance, instead of the internal resistances. To this end, the inductance and also the capacitance are assumed to be twice their nominal sizes, used in the training and also in the decision making. The results, depicted in Fig. 8 show that the controller handles these uncertainties well also. Such low sensitiity to parameters is particularly of interest gien the fact that parameter uncertainties are unaoidable and in some cases these parameters are unobserable. Interested readers are referred to, for example, [37] where this sensitiity is considerable and some adaptie schemes are sought for lowering it. Before concluding this subsection it is important to make two points. Firstly, the actual alues of the parameters subject to uncertainty were assumed to be unknown in this section. Hence, the nominal alues were used not only in the training phase, but also in the decision making and control. Secondly, the controller is not theoretically designed to be robust towards modeling uncertainties. Hence, een though it shows great potentials in this regard, the controller should not be expected to proide reliable results when large modeling uncertainties exist. G. Incorporating oltage leakage Each switching between the topologies leads to a oltage loss in reality, [], [38], [39]. This loss is called oltage 8 8 Time s) Fig. 7. Simulation results for x := [.5; ] T with modeling uncertainties in r l and r c and changing V s from.8v to.v at t = 5s with threshold 7 ). V) A) Time s) Fig. 8. Simulation results for x := [.5; ] T with modeling uncertainties in L and C and changing V s from.8v to.v at t = 5s with threshold 7 ). leakage and is ery challenging to incorporate in the controller design. The reason is, once the leakage is considered, the switching problem conerts to a switching problem with state jumps at the switching times, leading to a more hybrid problem. Before rigorously incorporating the state jumps in the controller design in Section VI, it is interesting to see the robustness of the already designed controller under the presence of the state jumps. Motiated by [39], it is assumed that the oltage leakage can be modeled as a function of the oltage across the capacitor immediately before each switching. Let 3% oltage drop be assumed across the capacitor as an example for testing the controller. The state trajectories, gien in Fig. 9, show the poor performance of the controller under the state jump. This problem will be addressed later in Section VI. H. Changing initial condition The tuned controller handles different initial conditions as long as the respectie state trajectories remain within the domain in which the controller is tuned. To show this performance, a few different initial conditions, namely x = [, ] T,

7 V).8. V) x = [, ] T x = [, ] T x = [, ] T A) Time s) Fig. 9. Simulation results for x := [.5; ] T with oltage leakage of 3%, without incorporating state jump in controller design. A) x = [, ] T x = [, ] T x = [, ] T Time s) Fig.. Simulation results for three different initial conditions with soft constraint on, namely, < 3A. V) A) x = [, ] T x = [, ] T x = [, ] T 5 x = [, ] T x = [, ] T x = [, ] T Time s) Fig.. Simulation results for three different initial conditions. x = [, ] T, and x = [, ] T are selected and simulated. The results, presented in Fig., demonstrate this desired feature. I. Enforcing a constraint on the inductor current As seen in the preious simulation results, for example in Fig., the inductor current assumes high alues in the transient phase. In practice, howeer, one is interested in enforcing a constraint on the maximum inductor current, e.g.,,max = 3A as in [], [7]. Similar to [], this constraint may be enforced as a soft constraint in the cost function using a step-like function that changes from zero to a positie alue when exceeds,max. Function + e 5,max ) ) t, which is an approximation of a step function located at,max, is selected for this purpose. This function along with the preiously selected Qx, ), scaled with as a weight, are used in the cost function and the tuning is re-done. Gien the constraint on, the domain of interest is changed to X =,.8), ). Using the resulting controller, the initial conditions presented in Fig. are re-simulated and the results are gien in Fig.. Interestingly, the controller has noticed the constraint on the and has forced the inductor current to obsere it, as compared with the results gien in Fig. for the case of no constraint. Finally, as a limitation of the proposed method, it should be mentioned that the scheme applies a soft constraint on the inductor current. While tuning the respectie weights in the cost function, it was obsered that the desired adjustment on the inductor current was achieed, this behaior is not guaranteed. In order to hae such a guarantee, the sate inequality constraint needs to be incorporated as a hard constraint in the problem. V. REDUCING SENSITIVITY TO LOAD CURRENT The sensitiity of the result to load disturbances, unlike its insensitiity to the line disturbances, both obsered in the preious section, shows that the tuned parameters of the controller are more dependent on the selected i o. Note that, when i o and/or V s were changed in the analysis of the preious section, the new alues were used in the scheduler on the fly, through updating f.) s. Howeer, no retraining based on the new alues were done, hence, the less desirable performance in changing i o compared with changing V s, leads to the conclusion that the parameters of the controller depend more heaily on i o. Gien this obseration, the idea in this section is feeding i o to the function approximator as an exogenous signal, instead of freezing it at its nominal alue. Let the alue function be denoted with V : R n I o R +, where I o denotes the domain in which i o is expected to change. Moreoer, let the notation for the model of the dynamics be redefined as [ ẋ = f x, i o ) = C i o ) L r c + r l ) + r c i o + V s ) ], 8) i.e., the dependency of the dynamics on i o is explicitly shown in f x, i o ). Of course, the aboementioned continuous-time dynamics should be discretized as in ), in order to be compatible with the ADP approach inestigated in this study. Gien these changes, the Bellman equation reads V x, i o ) = min Qx, ) + γv f x, i o ), i o ) ), x, i o. 9) and the optimal mode, in a feedback form ersus i o, is gien by x, i o ) = arg min Qx, ) + γv f x, i o ), i o ) ), x, i o. ) Similarly, in order to find V x, i o ), one may use the VI gien by V j+ x, i o ) = min Qx, ) + γv j f x, i o ), i o ) ), x X, i o I o. ) The key point is the fact that Eq. ) will need to be alid for different i o s randomly) selected within I o. Hence, the conerged alue function will learn the entire I o, instead of a single i o, as before. This is similar to the feature that the VI gien by ) was conducted on the entire X, instead of on a single x, so that the controller is alid for different initial 7

8 Weight Elements 8 8 Learning Iterations Fig.. Eolution of parameters/weights during offline learning for the case of feeding i o to the controller V) A) Time s) Fig. 3. Simulation results for x := [; ] T with load current increasing for A after each 5s, with incorporating the changing current load in the controller design with threshold 7 ). conditions and states. Gien the new alue function, with three scalar inputs, a new function approximator is also needed, compared with the one used in Section IV. In here, the polynomials made of the three inputs up to the fourth order are selected with the coefficients being the tunable parameters. Besides the ariables and design choices gien in subsection IV-A, the interal, 3) was selected for I o. The conerged parameters and the history of the states are gien in Figs. and 3, respectiely. As seen in the history of, the controller has nicely handled the A changes in i o after each 5s, as after an initial decay in, the steady state is regulated and the desired tracking is re-achieed. This results shows the improed dynamic performance of the scheme presented in this section, for reducing the sensitiity towards the load disturbances, compared with the results gien in Fig.. As for implementation, howeer, one will need to hae a sensor to measure the load current, besides the states, in order to feed them into the scheduler gien by ) and find the approximate) optimal control. This may be considered as a limitation of the proposed method or the cost that one pays for haing the desired low sensitiity to load current. VI. OPTIMAL SWITCHING WITH STATE JUMP USING ADP AND ITS APPLICATION TO THE POWER CONVERTER It was obsered in subsection IV-G, where the oltage leakage after each switching was modeled by a state jump, that the performance of the controller in handling such jumps is poor. Howeer, such results were not surprising, as the controller was not supposed to encounter state jumps after each switching, as described in Section III. This section, howeer, is aimed at deeloping a controller which addresses these jumps, through incorporating the jumps in the design. A. Problem formulation Considering the optimal switching problem described in Section III, the state jump is added as follows. Let each switching at the current state x from mode to mode, where, V and, lead to the state-dependent) state jump modeled by x,, ) for some : R n V V R n continuous with respect to x for eery gien and and x,, ) =, x,. In other words, gien the dynamics presented by ), the current state x, and the preious actie mode, if the same mode is selected for the current time as well, then the next state will be f x). But, if mode is decided to be selected at this instant, which means switching from to, the next state will be f x) + x,, ). Gien this argument, the next state of the system will be dependent not only on the mode that is selected at the current time, but also on the mode selected at the preious time. Considering this important change compared with the case of no state jump, how is the scheduler, gien by 9), supposed to know which mode was selected at the preious time step? Note that the scheduler needs to find the proper next state, at which it needs to ealuate the V.) in the right hand side of 9). An idea for addressing this problem is presented next. B. Proposed solution The idea for addressing the problem of the dependency of x k+ on k as well as on k is ery intuitie. When the preiously selected mode plays a role in determining the next state, then, the preious actie mode is also a part of the state of the system. It should be noted that the mode that will be selected at the current time, will be the input to the system and hence, can change the next state. Howeer, just like the rest of the state elements, the preious actie mode also matters. So, gien the current time k, the state ector x k defined in subsection III-A, and the preiously selected mode k, the augmented state is defined as y k := [x k, k ] T X V. Gien Eq. ), the dynamics of the augmented state ector will be gien by [ ] f x y k+ = F y k ) := k ) + x k, k, ), ) k Z +, V, y k = [x k, k ] T X V. Gien ), the problem is now simply the optimal switching problem defined in subsection III with the only difference that the state ector is y and the dynamics of modes are gien by F.) s. Therefore, the same solution, gien in subsection III-B is alid, i.e., the VI will be gien by V j+ y) = min Qx, )+γv j F y) )), y X V, 3) starting from some V y), to find the solution to the Bellman equation V y) = min Qx, ) + γv F y) )), y X V. ) 8

9 Once the alue function V y) is found, the new scheduler will be Qx, ) + γv F y) )), y X V. y) = arg min 5) Now, the question asked at the end of subsection VI-A is addressed. This is done through noting that the scheduler, as gien by Eq. 5), will be of a feedback form based on the augmented state y, which includes the preiously selected mode as well as x. Hence, the scheduler will know what mode was selected at the preious time step. Considering the desired continuity of V j.)s, for their uniform approximation, discussed at the end of subsection III-B, the next step is analyzing this continuity. Gien the augmented state y and its last element which changes within the discrete set V, the issue is, the alue function may not be continuous ersus y. This problem can be resoled by utilizing M many function approximators, each for approximating V [., ] T ) for one specific V. This is motiated by the method presented in [] for incorporating switching costs, which led to a similar feature. Howeer, Ref. [] incorporates a switching cost, not a state jump. In other words, the current study admits a jump in the state after each switch while Ref. [] addresses a jump in the cost after each switch. What remains to show is the continuity of V [., ] T ) in X for eery fixed which, gien the state jump in the dynamics, is not obious and does not directly follow from [5]. The following theorem proes this desired continuity. Theorem. If functions f.), Q., ),.,, ), and the initial guess V [., ] T ) used in the alue iteration 3) are continuous in X for eery fixed,, and, then, V j [., ] T ) for eery gien and j is continuous in X. Proof : The proof is by induction. Function V [., ] T ),, is continuous. If continuity of V j [., ] T ),, leads to the continuity of V j+ [., ] T ),, then the proof is complete. Let the scalar function G : R n V V R + be defined as G x,, ) := Qx, )+γv j [f x)+ x,, ), ] T ), ) and function : R n V V be gien by x, ) := arg min G x,, w). 7) w V Function G x,, x, ) ) is identical to V j+ [x, ] T ) considering 3), ), and 7). Hence, continuity of G.,,., ) ) for eery gien completes the induction gien the assumed continuity of V j [., ] T )). Let x be any point in X, for any gien V set = x, ). 8) Open set α X may be selected such that x belongs to the boundary of α and limit ˆ = lim x, ), 9) x x,x α exists, where the ector norm is denoted with.. If for eery such α one has = ˆ, then an open set β X containing x exists such that x, ) is constant for all x β, because the range of x, ) is a discrete set. The continuity of V) A) Time s) Fig.. Simulation results for x := [.5; ] T with oltage leakage of 3%, with incorporating the state jump in the controller design. G.,,., ) ) at x in this case follows from the continuity of G x,, ) at x = x, for eery fixed/gien, V, as Q., ), f.),.,, ), and V j [., ] T ) are continuous. Finally, the continuity of G.,,., ) ) at eery x X, leads to the continuity in X. Now assume, for some α, one has ˆ. The continuity of Gx,, ˆ) for the fixed and ˆ leads to If one has G x,, ˆ) = lim δx G x + δx,, ˆ) 3) G x,, ) = G x,, ˆ), 3) for eery selected α, then the continuity of G.,,., )) for the gien follows. The reason is, 3) and 3) lead to G x,, ) = lim G x + δx,, ˆ) ), 3) δx and 3) gies the continuity by definition []. By contradiction, one can show that 3) holds. Assume G x,, ) > G x,, ˆ), 33) for some x and some α. The foregoing inequality leads to x, ). But, this is against 8) per the definition of.,.) gien by 7). Hence, 33) does not hold. Now, assume that G x,, ) < G x,, ˆ), 3) then, there exists an open set γ containing x, such that Gx,, ) < Gx,, ˆ), x γ. 35) because, both sides of 3) are continuous at x for the fixed, ˆ, and. Inequality 35) implies that at points which are close enough to x, one has x, ) ˆ. This, howeer, contradicts Eq. 9) which says that there always exists a point x arbitrarily close to x at which x, ) = ˆ, considering the point that.,.) only assumes integer alues. Therefore, inequality 35) also cannot hold, hence, 3) holds which leads to the continuity of G x,, x, ) ) at eery x X for eery fixed V. This completes the proof. C. Implementation on DC-DC conerter with oltage leakage The new controller synthesis which incorporates the oltage leakage modeled as state jumps is numerically analyzed here. Gien the number of modes, two function approximators are trained for approximating V [x, ] T ) and V [x, ] T ). The same parameters and ariable gien in subsection IV-A are 9

10 selected, with the difference that the domain of interest is changed to X =,.) 5, 5), for improing the accuracy of the approximation. Moreoer, the 3% oltage leakage selected in subsection IV-G is utilized for both the offline training using 3) and for online control using scheduler 5). Note that the 3% is just an example alue to show the performance of the controller in handling state jumps. The results are gien in Fig.. Compared with Fig. 9, it is eident that the controller has successfully incorporated the oltage leakage through adjusting the switching pattern such that the desired Vref = V is approximately) tracked. Finally, as a limitation of the proposed method, it may be mentioned that a model of the the oltage leakage needs to be aailable to be used in the offline training. This is due to the model-based nature of the proposed method. A future effort will be on extending the idea to the data-based case, where such models will be identified implicitly during online learning. Interested readers are referred to [] for an idea. Fig. 5. Simulation results for x = [.5,.5]T with the load current changing from.3a to.a at t = s, to.a at t = s, and back to.3a at t = 3s. Annotation ADP Controller is for the results from this paper without feeding io to the controller) and RDP Controller represents results from [7]. VII. R ESULTS C OMPARISON The presented method is compared with two other controllers in this section. The selected controllers are the Relaxed Dynamic Programming RDP) framework in [7] and the Nonlinear Programming NLP) approach presented in []. Identical parameters for the switching circuit gien by Fig., identical alues for Vs, Vref, and the same initial condition are used for the controllers to facilitate the comparison. For comparing the deeloped ADP-based controller with the RDP, the simulation presented in [7] with rc =, rl = Ω, L =.H, C = F, Vs = V, io =.3A, and Vref =.5V was selected. The controller deeloped in Section IV, i.e., without feeding load current into the controller, was used for controlling the system. The results are presented in Fig. 5 where the load current was changed from the initial alue of.3a to.a and then to.a and back to.3a, where each current was applied for s. Comparing the ADP result with the RDP one copied from [7]), it can be seen that the proposed method demonstrates a better performance een without feeding the load current to the network and also without defining an integral state for penalizing the tracking error as conducted in [7]. It is worth mentioning that the large sampling time of t =.s, used in [7], was used here in the proposed ADP-based controller also, which led to higher oltage ripples compared with the results presented in preious sections. Next, the simulation results presented in [] are compared with the results in this work. Similar soft constraints on the current are applied and starting with the same initial condition, the capacitor oltage and inductor current are plotted in Fig.. This figure also shows that the proposed method compares faorably with the one presented in []. Fig.. Simulation results for x = [, ]T with soft constraint on il, namely, il < 3A. Annotation ADP Controller corresponds to results from this paper and NLP Controller is for the results from []. parameter uncertainties were analyzed. Constraints on the inductor current and the oltage leakage of the capacitor are some other challenges inestigated in this study, in which, the proposed tool either inherently proided a suitable performance or the changes made in the controller addressed them. The proposed tools and methods including the controller for optimal switching with state jump may be utilized by different researchers and practitioner for different applications inoling state jumps. Gien the ersatility of the proposed scheme, it may also be utilized for different power conerters including switched-capacitor DC-DC conerters with more than two topologies as well as DC-AC conerters. Finally, the results presented in this work solely show the performance of the controller in simulations. Real implementation could reeal other challenges and is subject of future research. R EFERENCES [] M. Kazimierczuk and L. Starman, Dynamic performance of pwm dc-dc boost conerter with input oltage feedforward control, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, ol., pp. 73 8, Dec 999. [] T. Geyer, A comparison of control and modulation schemes for medium-oltage dries: Emerging predictie control concepts ersus pwm-based schemes, IEEE Transactions on Industry Applications, ol. 7, pp , May. [3] S. Almer, S. Mariethoz, and M. Morari, Sampled data model predictie control of a oltage source inerter for reduced harmonic distortion, IEEE Trans. Control Syst. Technol., ol., pp , 3. [] R. Middlebrook and S. Cuk, A general unified approach to modelling switching-conerter power stages, in IEEE Power Electronics Specialists Conference, pp. 8 3, June 97. [5] F. Leung, P. Tam, and C. Li, The control of switching dc-dc conertersa general lwr problem, IEEE Transactions on Industrial Electronics, ol. 38, pp. 5 7, Feb 99. [] F. Garofalo, P. Marino, S. Scala, and F. Vasca, Control of dc-dc conerters with linear optimal feedback and nonlinear feedforward, IEEE Transactions on Power Electronics, ol. 9, pp. 7 5, No 99. VIII. C ONCLUSIONS Approximate dynamic programming was shown to hae interesting potentials in handling different challenges existing in the problem of switching in a oltage conerter circuit. Sensitiity towards line and load disturbances as well as

11 [7] T. Gupta, R. Boudreaux, R. Nelms, and J. Hung, Implementation of a fuzzy controller for dc-dc conerters using an inexpensie 8-b microcontroller, IEEE Transactions on Industrial Electronics, ol., pp. 9, Oct 997. [8] M. Khayamy, H. Chaoui, A. Oukaour, and H. Gualous, Adaptie fuzzy logic control mixing strategy of dc/dc conerters in both discontinuous and continuous conduction modes, Journal of Control, Automation and Electrical Systems, ol. 7, no. 3, pp. 7 88,. [9] Q. Zhao and C. Cui, Optimal control of dc/dc conerter using improed model-free adaptie control algorithm, International Journal of Applied Mathematics and Statistics, ol. 5, no. 3, pp , 3. [] H. Sira-Ramirez, Nonlinear p-i controller design for switchmode dcto-dc power conerters, IEEE Transactions on Circuits and Systems, ol. 38, pp. 7, Apr 99. [] S.-C. Tan, Y. Lai, and C. Tse, A unified approach to the design of pwm-based sliding-mode oltage controllers for basic dc-dc conerters in continuous conduction mode, IEEE Transactions on Circuits and Systems I: Regular Papers, ol. 53, pp. 8 87,. [] R. Ling, D. Maksimoic, and R. Leya, Second-order sliding-mode controlled synchronous buck dc-dc conerter, IEEE Transactions on Power Electronics, ol. 3, no. 3, pp ,. [3] T. Geyer, G. Papafotiou, and M. Morari, Hybrid model predictie control of the step-down dc-dc conerter, IEEE Transactions on Control Systems Technology, ol., pp., No 8. [] S. Kouro, P. Cortes, R. Vargas, U. Ammann, and J. Rodriguez, Model predictie control-a simple and powerful method to control power conerters, IEEE Trans. Ind. Electron., ol. 5, pp , 9. [5] S.-K. Kim, C. R. Park, J.-S. Kim, and Y. I. Lee, A stabilizing model predictie controller for oltage regulation of a dc/dc boost conerter, IEEE Trans. Control Syst. Technol., ol., pp. 3,. [] Q. Wei, B. Wu, D. Xu, and N. R. Zargari, Model predictie control of capacitor oltage balancing for cascaded modular dc-dc conerters, IEEE Transactions on Power Electronics, ol. 3, no., pp. 75 7, 7. [7] B. Lincoln and A. Rantzer, Relaxing dynamic programming, IEEE Transactions on Automatic Control, ol. 5, pp. 9, Aug. [8] A. Wernrud, Strategies for computing switching feedback controllers, in American Control Conference, pp , July 7. [9] K. Vamoudakis and J. Hespanha, Online optimal switching of single phase dc/ac inerters using partial information, in American Control Conference ACC), pp. 3, June. [] H. Kawashima, Y. Wardi, D. Taylor, and M. Egerstedt, Optimal switching control of a step-down dc-dc conerter, in American Control Conference ACC), pp. 5 59, June. [] F. Lewis and D. Liu, Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. IEEE Press Series on Computational Intelligence, Wiley, 3. [] C. Druet, D. Ernst, and L. Wehenkel, Application of reinforcement learning to electrical power system closed-loop emergency control, in Principles of Data Mining and Knowledge Discoery D. Zighed, J. Komorowski, and J. Zytkow, eds.), ol. 9 of Lecture Notes in Computer Science, pp. 8 95, Springer Berlin Heidelberg,. [3] D. Ernst, M. Glaic, and L. Wehenkel, Power systems stability control: reinforcement learning framework, IEEE Transactions on Power Systems, ol. 9, pp. 7 35, Feb. [] S. Mohagheghi, Y. Del Valle, G. Venayagamoorthy, and R. Harley, A proportional-integrator type adaptie critic design-based neurocontroller for a static compensator in a multimachine power system, IEEE Transactions on Industrial Electronics, ol. 5, pp. 8 9, Feb 7. [5] A. Heydari, Optimal scheduling for reference tracking or state regulation using reinforcement learning, Journal of the Franklin Institute, ol. 35, no. 8, pp , 5. [] A. Heydari, Feedback solution to optimal switching problems with switching cost, IEEE Transactions on Neural Networks and Learning Systems, ol. 7, no., pp. 9 9,. [7] S. Almer, S. Mariethoz, and M. Morari, Optimal sampled data control of pwm systems using piecewise affine approximations, in IEEE Conference on Decision and Control, pp. 3 8, Dec. [8] D. E. Kirk, Optimal control theory; an introduction. Prentice-Hall, 97. pp [9] A. Heydari, Theoretical and numerical analysis of approximate dynamic programming with approximation errors, Journal of Guidance, Control, and Dynamics, ol. 39, pp. 3 3,. [3] Q. Wei, F. Wang, D. Liu, and X. Yang, Finite-approximation-errorbased discrete-time iteratie adaptie dynamic programming, IEEE Transactions on Cybernetics, ol., no., pp ,. [3] Y. Zhu, D. Zhao, H. He, and J. Ji, Conergence proof of approximate policy iteration for undiscounted optimal control of discrete-time systems, Cognitie Computation, ol. 7, no., pp , 5. [3] A. Heydari and S. N. Balakrishnan, Fixed-final-time optimal control of nonlinear systems with terminal constraints, Neural Networks, ol. 8, pp. 7, 3. [33] W. Rudin, Principles of Mathematical Analysis. McGraw-Hill, 3rd ed., 97. p. 55. [3] K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are uniersal approximators, Neural Networks, ol., no. 5, pp , 989. [35] A. Heydari and S. Balakrishnan, Optimal switching between autonomous subsystems, Journal of the Franklin Institute, ol. 35,. [3] A. Heydari, Approximate solution to discrete-time optimal switching problems with minimum dwell time constraint, in American Control Conference, pp ,. [37] M. Fairbank, S. Li, X. Fu, E. Alonso, and D. Wunsch, An adaptie recurrent neural-network controller using a stabilization matrix and predictie inputs to sole a tracking problem under disturbances, Neural Networks, ol. 9, pp. 7 8,. [38] C.-F. Ho, B.-K. Ling, Y.-Q. Liu, P.-S. Tam, and K.-L. Teo, Optimal pwm control of switched-capacitor dc-dc power conerters ia model transformation and enhancing control techniques, IEEE Trans. Circuits and Systems I: Regular Papers, ol. 55, no. 5, pp , 8. [39] R. Loxton, K. Teo, V. Rehbock, and W. Ling, Optimal switching instants for a switched-capacitor dc/dc power conerter, Automatica, ol. 5, no., pp , 9. [] W. F. Trench, Introduction to Real Analysis pp. -5. [] A. Heydari, Optimal triggering of networked control systems, aailable online at arxi:.57.