F.L. Lewis. New Developments in Integral Reinforcement Learning: Continuous-time Optimal Control and Games

Transcription

1 F.L. Lewis National Academy of Inventors Moncrief-O Donnell Chair, UTA Research Institute (UTARI) The University of Texas at Arlington, USA and Qian Ren Consulting Professor, State Key Laboratory of Synthetical Automation for Process Industries Northeastern University, Shenyang, China New Developments in Integral Reinforcement Learning: Continuous-time Optimal Control and Games Supported by : ONR US NSF Supported by : China NNSF China Project 111 Talk available online at

2 Applications of Reinforcement Learning Microgrid Control Human Robot Interactive Learning Industrial process control Mineral grinding in Gansu, China H infinity control for UAV Resilient Control to Cyber Attacks in Networked Multi agent Systems Decision & Control for Heterogeneous MAS (different dynamics)

3 Game-theoretic Control for DC Microgrids Work of Vahidreza Nasirian with Ali Davoudi 8

4 9

5 Advantages of DC Microgrids AC Microgrid: 1) Complex synchronization procedure for grid-tied operation (frequency, magnitude, and phase match is required) 2) Complex control circuitry (voltage, frequency, and active/reactive power control) 3) Unwanted transmission loss due to reactive power exchange 4) Redundant dc-ac-dc conversions for integration of renewable sources, loads, and storage units 5) Harmonic current management and phase unbalances DC Microgrid: 1) Only voltage and power control is needed 2) No reactive power flow and, thus, an improved overall efficiency 3) Converted renewable energies are basically dc and, thus, a dc distribution is more effective for integration of these sources 4) No harmonic current or phase unbalance issue 10

6 Cooperative Game-theoretic Control of Active Loads in DC Microgrids pout e pin Ling-ling Fan, V. Nasirian, H. Modares, F.L. Lewis, Y.D. Song, and A. Davoudi, Game-theoretic Control of Active Loads in DC Microgrids, IEEE Trans. Energy Conversion, vol. 31, no. 3, pp , p out e p in t 2 t 1 t 3 t 2 t 1 t 3 t 2 t 1 t 3 Power buffers in Microgrid Network Power buffer operation during a step change in power demand. r r r r Supplies excess power needed during load changes until sources can respond r 47 r r r 18 r 59 r s1 v s1 vi e i pi u i

7 Active Load Power Buffer 2 ìï vi ïe = -p i i í r, i ïr u i = ïî i Stored energy Input impedance r i Bus voltage v i Control input u i Output power = a disturbance e i Nonlinear dynamics Not obvious how to handle p i Vahid Nasirian pi

8 Solve for bus voltage to get coupled agent dynamics Linearize. Add p i as a state. Formulate as H infinity Problem. Define coupled performance indices 2 é q q e ù é ù 0 2i g ( i ) 1 ée ù é0ù é0ù i - - i ii i i r r 1 u 0 w i = i + + i + i p p 0 1 ê ë iú ê iú ê ú ê ú û ê ë ú û ë û ë û ë û x x B D i i i i Ai + é ê M+ N ij j j= M+ 1( ¹ i) q 2i ê 0 ú, i ê ë å 0 g r ù ú ú û i = M + 1,, M + N, Coupling terms Vahid Nasirian Reza Modares Dr. Ali Davoudi æ ö T 2 J = r u d t, i M 1,, M N, i ò + = + + j ij j i i ç å xqx çèjî N ø 0 i Define Communication Graph Sparse efficient topology Optimal design provides Resilience and disturbance rejection

9 Optimal Cooperative Control as a Dynamic Game Minimize the performance function for active loads J i 0 x T 2 j Q ij x j i u i dt jn i Let s define the neighborhood state vector as x i x i T, x j T T jni Graphical Game The optimal solution is in a general form of u i k i x i With such solutions, the performance function J i is quadratic in x: J i (x i ) x i T P i x i which helps to find the optimal solution by solving an algebraic Riccati equation u * i B T ii P i x 1 i i 14 14

10 Optimal Cooperative Control: Policy Iteration finds Optimal Solutions Substituting the optimal solution in Bellman equations leads to the following coupled Algebraic Riccati Equations (ARE) H i x T i Q i x T * i + i u i 2 +x T i P i A i x i B i u * i D i w i (x i ) T P i x i =0 + A i x i B i u i * D i w i (x i ) u * i u * * i, u j T jni Policy iteration (a class of reinforcement learning) is used to solve ARE and find P i and the optimal control input u * i B T ii P i x 1 i i Policy evaluation: the performance of a given control policy, u i, is evaluated using the Bellman equation, and P i are found. Policy improvement: an improved control policy, u i, is found for each agent, using P i found in the first step. Policy evaluation and improvement are repeated until no improvement in control policies, u i, of any agent is observed

11 Microgrid Setup and Cooperative Controller (a) DC microgrid system (b) Active load (c) Communication network Controller Implementation 16

12 Controller Performance with Load Change Load change in bus 5; Buffers 4 & 5 assisting Load change in bus 4; Multiple assistive buffers (a) microgrid bus voltages at the load terminals, (b) Output voltage of the power buffers, (c) output voltage across the resistive loads, (d) Source currents, (e) Stored energies in power buffers, (f) Input impedance of the power buffers, (g) Output of the active loads, (h) energy-impedance trajectory of power buffers during the load transient. 17

13 Intelligent Operational Control for Complex Industrial Processes Jinliang Ding Professor Chai Tianyou State Key Laboratory of Synthetical Automation for Process Industries Northeastern University May 20, Jinliang Ding, H. Modares, Tianyou Chai, and F.L. Lewis, "Data-based Multi-objective Plant-wide Performance Optimization of Industrial Processes under Dynamic Environments, IEEE Trans. Industrial Informatics, vol.12, no. 2, pp , April Xinglong Lu, B. Kiumarsi, Tianyou Chai, and F.L. Lewis, Data-driven Optimal Control of Operational Indices for a Class of Industrial Processes, IET Control Theory & Applications, vol. 10, no. 12, pp , 2016.

14 Manufacturing as the Interactions of Multiple Agents Each machine has it own dynamics and cost function Neighboring machines influence each other most strongly There are local optimization requirements as well as global necessities

15 Production line for mineral processing plant Mineral Processing Plant in Gansu China

16 Overall Existing Manual Control for Plant production indices, unit operational indices, and unit process control for a production line

17 Automated online reinforcement learning for determining operational indices Q, Q, Q * k kmin kmax 2 RL loops And Value Function Approximation Qˆ k () t ri, j( mt) Qk ( mt) Q, Q, Q * k kmin kmax rˆ( t) * r i, j r ~{ r } i 1, n i, j j 1, 2, 3 r r( mt ) Q, Q, Q * k kmin kmax Qk ( mt) Implemented by Jingliang Ding and Chai Tianyou s group in biggest mineral processing factory of hematite iron ore in China, Gansu Province. Savings of million RMB per year were realized by implementing this automated optimization procedure instead of the standard industry practice of human operator selection of process operational indices.

18 Jinliang Ding, H. Modares, Tianyou Chai, and F.L. Lewis, "Data-based Multi-objective Plant-wide Performance Optimization of Industrial Processes under Dynamic Environments, IEEE Trans. Industrial Informatics, vol.12, no. 2, pp , April Xinglong Lu, B. Kiumarsi, Tianyou Chai, and F.L. Lewis, Data-driven Optimal Control of Operational Indices for a Class of Industrial Processes, IET Control Theory & Applications, vol. 10, no. 12, pp , Yi Jiang, Jialu Fan, Tianyou Chai, Jinna Li, and F.L. Lewis, Data-Driven Flotation Industrial Process Operational Optimal Control Based on Reinforcement Learning, IEEE Trans. Industrial Informatics, to appear, Jinna Li, Tianyou Chai, F.L. Lewis, Jialu Fan, Zhangtao Ding, and Jinliang Ding, Off-policy Q- learning: set-point design for optimizing dual-rate rougher flotation operational processes, IEEE Trans. Industrial electronics, vol. 65, no. 5, pp , May Jinna Li, Bahare Kiumarsi, Tianyou Chai, F.L. Lewis, and Jialu Fan, Off-Policy Reinforcement Learning: Optimal Operational Control for Two-Time-Scale Industrial Processes, IEEE Trans. Cybernetics, vol. 47, no. 12, pp , Dec

19 Control of Non-affine Aerial Systems Using Off-policy Reinforcement Learning

20

21 Non affine nonlinear aerial vehicle model Xt ( ) = fxt ( ( )) + gxt ( ( )) Lu ( ) + Dwt ( ) UAV dynamics x = V cos gcos y+ dw x = V cos gsin y+ d w x =- V sin g + d w n 2 z =-a - sin g+ a -a -a z 4 2 V V g T n g g = ( n cosf-cos g) z V with n n x x TT = TT = max max cos a - D mg sin a + K mg y V g = n V cos g z sin f

22 State Dynamics where X = { x, x, x, V, gy, } T Xt ( ) = fxt ( ( )) + gxt ( ( )) Lu ( ) + Dwt ( ) é x cos( x )cos( x ) ù x cos( x )sin( x ) x sin( x ) f( X( t)) = -a x -g sin( x ) g (- cos( x ) 5 x 4 ê 0 ú ë û é ù a4 gxt ( ( )) = a -a x g ê x cos( x ) ú ë 4 5 û Lut ( ( )) él ù é u ù 1 1 L u L = u 3 = 2 L u cos( u ) êl ú êu sin( u ) ú ë 5 û ë 2 3 û

23 Optimal Control for Constrained Input Systems Murad Abu-Khalaf Control constrained by saturation function tanh(p) 1 Encode constraint into Value function u T J ( ud, ) Qx ( ) 2 ( ) d dt p u q 2 u T 2 ( ) d 0 (Used by Lyshevsky for H 2 control) This is a quasi-norm Weaker than a norm homogeneity property is replaced by the weaker symmetry property x q x q Then u V ( ) T x 1 R g x Is BOUNDED

24 H infinity Control Tracking Problem UAV dynamics Xt ( ) = fxt ( ( )) + gxt ( ( )) Lu ( ) + Dwt ( ) Desired trajectory generator X () t = h ( X ) d d d Bounded L2 norm ò ò Constrained controls 2 ( t) e - at - z() t dt t 2 - at ( - t) t e w() t dt u u u g 1 1 u where 2 T ( ) = ( - ) ( - ) + ( ( )) d d zt X X Q X X WLu Formulate as Optimal Control Problem JX ( t) T 2 T ( ) e at = - - ò é ( X - X ) QX ( - X ) + WLu ( ( ))-g wwd ù t t êë d d úû

25 Write Augmented System and Leader Dynamics Tracking error Augmented State et () = Xt ()-X () t é et () ù Zt () = X () t êë d úû d Augmented Tracking Dynamics é et () ù éfe ( X ) h ( X ()) t ù ége ( X ) ù édù + - d d d + Lu ( ) wt ( ) FZt ( ( )) GZt ( ( )) Lu ( ) Kwt ( ) X () t = + + º + + h ( X ()) t 0 0 êë d úû êë d d úû êë úû êë úû Performance Index at ( ) T 2 T JLu ( ( ), w) = e - - ò t é ZQZ+ WLu ( ( ))-g wwd ù t t êë 1 úû with Q 1 éq 0ù = 0 0 êë úû

26 Optimal H inf Tracker Bellman Equation + ( ( ))- - ( ) + ( ) = 0 1 T 2 T Z QZ W L u g w w av Z V Z HZ (, Lu ( ), wv, ) = Z QZ + W( Lu ( )) -g w w - a V( Z) + V ( FZ ( ) + GZLu ( ) ( ) + Kw) = 0 Z T 2 T T 1 Z Stationarity Condition gives Optimal Control and worst case disturbance * * L ( u) = argmin H( Z, L( u), wv, ) Lu ( ) * * w = arg max H( Z, L( u), wv, ) w So that L u L V G * T * T ( ) =- tanh (( ) ) Z 1 2 T w * = g - ( V * ) K Z 2

27 Assume L(u) is Invertible Then * - 1 T * u =-L ( Ltanh ( v ))

28 Reinforcement Learning Policy Iteration Solution é et () ù éfe ( X ) h ( X ()) t ù ége ( X ) ù édù + - d d d + Lu ( ) wt ( ) FZt ( ( )) GZt ( ( )) Lu ( ) Kwt ( ) X () t = h ( X ()) t + + º êë d úû êë d d úû êë úû êë úû Need to know input matrices G and K

29 Off Policy IRL Solution ( ) ( ( )) ( ( )) j ( ) j ( ( ))( ( ) j ( )) ( j Zt = FZt + GZt L u + Kw + GZt Lu - L u + Kw-w ) Off Policy Bellman Equation Do not need any of the dynamics of UAV or leader

30 Data Driven Real Time Solution Using VFA Approximate critic, control, disturbance Plug into Off Policy Bellman Equation to get algebraic equations for the weights

31 RL for Human-Robot Interaction (HRI) 1. H. Modares, I. Ranatunga, F.L. Lewis, and D.O. Popa, Optimized Assistive Human-robot Interaction using Reinforcement Learning, IEEE Transactions on Cybernetics, vol. 46, no. 3, pp , I. Ranatunga, F.L. Lewis, D.O. Popa, and S.M. Tousif, "Adaptive Admittance Control for Human-Robot Interaction Using Model Reference Design and Adaptive Inverse Filtering" IEEE Transactions on Control Systems Technology, vol. 25, no. 1, pp , Jan B. AlQaudi, H. Modares, I. Ranatunga, S.M. Tousif, F.L. Lewis, and D.O. Popa, Model reference adaptive impedance control for physical human robot interaction, Control Theory and Technology, vol. 14, no. 1, pp. 1-15, Feb

32 PR2 meets Isura

33 Robot dynamics Impedance Control Prescribed Error system Control torque depends on Impedance model parameters

34 Standard Robot Trajectory Tracking Controller Where is the human?

35 Human Performance Factors Studies Human task learning has 2 components: 1. Human learns a robot dynamics model to compensate for robot nonlinearities 2. Human learns a task model to properly perform a task Inner Robot Specific Control Loop INDEPENDENT OF TASK Outer Task Specific Control Loop INDEPENDENT OF ROBOT DETAILS

36 RL for Human Robot Interactions Task control outer loop Robot control inner loop

37 New Inner Robot Control Loop No task trajectory information is used in this inner loop robot controller The inner loop robot controller makes the model following error small The admittance model parameters are not needed Only the admittance model trajectories x, x, x are needed. m m m

38 Three Outer Loop Designs To appear 2016

39 2C. Outer loop Task Specific Design #3 Reinforcement Learning for minimum human effort Force exerted by human indicates his discontent A measure of Human Intent Feedforward assistive control term Work of Reza Modares l (.) Human force amplifier xd e + d ( Ks + Ks ) - p d Human 1 f h K h ( Ms + Bs + K) xm Prescribed Impedance Model Find robot impedance model parameters To minimize human force effort f h And task trajectory following error e d M, BK,

40 Robot Impedance Model Unknown Human Model ( Ks+ K ) f= ke d p e d K f + K f = k e d h p h e d f =- K K f + k K e º A f + E e -1 h d p h e d,0 d h h h d Feedback linearization loop

41 Performance index ò J = ( e T Q e + f T Q f + u T Ru ) dt t Then control is d d d h h h e e u = K e + K f e 1 d 2 h Augmented Tracker Dynamics with Human and Tracking Error Minimize human effort and tracking error e = x -x Î d d m e = [ e e ] = x -x Î T T T 2n d d d d Overall Augmented Dynamics n ò T T J = ( X QX + u Ru ) dt t e e

42 Optimal control is an offline method Based on solving ARE Knowing all the plant dynamics We want online method to learn the optimal control without knowing the System Matrix A Optimal Design Always Admits Reinforcement Learning for Real time Optimal Adaptive Control

43 OFF POLICY Reinforcement Learning Needs NO knowledge of the system dynamics Take enough data along the system trajectory To solve this equation using least squares

44

45