Robust Learning Control with Application to HVAC Systems

Transcription

1 Robus Learning Conrol wih Applicaion o HVAC Sysems Naional Science Foundaion & Projec Invesigaors: Dr. Charles Anderson, CS Dr. Douglas Hile, ME Dr. Peer Young, ECE Mechanical Engineering Compuer Science

2 Graduae Sudens Michael Anderson Chrisopher Delnero David Hodgson Mahew Krechmar Jilin Tu

3 Ouline Inroducion Experimenal HVAC Plaform PI Plus Neural Nework Conroller MIMO Robus Conroller Design Robus Reinforcemen Learning Conroller Resuls and Discussion Concluding Remarks

4 Moivaion From he Mechanical Engineering perspecive, how can neural neworks be applied o highly non-linear, ime varying HVAC sysems? From he Compuer Science poin of view, how can we rain neural neworks wih reinforcemen learning while guaraneeing sabiliy? From he Elecrical and Compuer Engineering view poin, how can neural neworks be used wih robus conrol sysems o improve performance? Inerdisciplinary!

5

6 Inroducion Characerisics of Typical HVAC Sysems Energy Transfer via Heaing/Cooling Coils Air flow Regulaion o Mainain Saic Air Pressure Cenral Waer Supply Servicing Muliple Unis Curren HVAC Sysems Perform Poorly Complex Nonlinear Time-Varying Sysem Highly Uncerain Sysem Dynamics Ineracion of Conrolled Variables Conrolled via Muliple SISO PID Conrol Loops

7 Experimenal HVAC Sysem Simple HVAC Sysem Couner Flow Ho Waer o Air Hea-Exchanger Variable Air Volume Mixing Box Elecric Ho Waer Heaer Conrolled Variables: Discharge Air Temperaure Mixed Air Temperaure Air Flow Rae Ho Waer Temperaure

8 exernal air T T T T T T T

9 PC/MATLAB Based Conrol Sysem

10 PI Plus Neural Nework Conroller

11 PI Conroller Design Nonlinear Sysem Tune a High Gain Parameers Conrolled: Waer Supply Temperaure Air Flow Rae Inpu Air Temperaure Discharge Air Temperaure Heaing Coil capaciy vrs. waer flow rae

12 PI Conrol Algorihm O τ = K e p τ + K i τ e j= 0 j O τ 1 = K e p τ 1 + K i τ 1 e j= 0 j O τ = O τ 1 + K p ( ) + e τ e τ 1 K e i τ

13 Reference PI Conroller

14 Neural Nework

15 Training Back Propagaion on: Model Daa (Seady ) Curve fi o measured flow vs. conrol signal Effeciveness hea exchanger model based on physical properies of he coil adjused based on experimen Experimenal Daa Wai for he PI conroller o achieve seady sae

16 PI Conrol Plus Neural Ne O O τ τ = 1 K e p = τ K e p + τ K 1 + i j τ = K 0 i e τ j j 1 = 0 e j O τ = O τ 1 + K p ( ) e τ e τ 1 + K e i τ O τ = NN + K e p τ + K e i τ

17 Response o Se Poin Change

18 Disurbance Rejecion

19 Seling Times

20 Advanages Improved performance compared o PI alone Simple, easy o rain neural nework The combined PI/NN conroller is comparaively easy o undersand Can be implemened in he near erm MISO Conrol

21 Muli-inpu inpu Muli- oupu (MIMO) Robus Conrol of HVAC Sysems

22 Previous Work - Conrollers Proporional-Inegral Inegral-Derivaive (PID) Low Gains, Tedious Tuning, Poor Performance Opimal Decoupling, Disurbance Rejecion Lack of Robusness Adapive Poenial o Adap, Eliminaing Need for Tuning Sabiliy Issues Robus Poenial Loss in Performance Limied o SISO Implemenaion

23 Overview MIMO Robus Conrol for HVAC Malab/Simulink Modeling and Design True MIMO Analysis and Design Improved Sysem Performance Coordinaed Acion for Independen Conrol Many Poenial Sysem/Conroller Archiecures MIMO Robus Sabiliy and Performance Experimenal Implemenaion and Tesing

24 Simulink Inerface Model

25

26 Dynamic Model for: Conroller Design HVAC Sysem Model Simulaion Tesing Nonlinear Subsysems Linearizaion for Design Combinaion of: 1s Principles Daa fiing

27 HVAC Sysem Model Model of Experimenal Sysem Dynamic Model for: Conroller Design Simulaion Tesing Nonlinear Subsysems Linearizaion for Design 1s Principles and Daa fiing Model and Experimenal Sysem Oupus

28 Conroller Design Basic Design Goals: MIMO Sabiliy and Robusness Independen Conrol of Key Sysem Variables Discharge Air Temperaure and Flow Rae Reference Convenional PI Conroller MIMO Robus Conrollers: Minimal (3x6) ~ T WS Exernally Conrolled Consrained (4x7) ~ T WS Inegraed Full (4x7) MIMO

29 Conroller K Simulaion PI

30 Conroller K Experimenal PI

31 Robus H Conroller Design Using µ-synhesis Design Objecive: Find all sabilizing K such ha:, Use augmened perurbaion srucure o es robus performance The goal of µ synhesis o minimize over all sabilizing K, he peak value of µ ( )

32 Robus H Conroller Synhesis Using D-K D K Ieraion No mehod exiss o direcly synhesize a robus H conroller, however, D-K ieraion provides a good approximaion: The upper bound on µ can be expressed as: The µ opimal conroller minimizes he peak value of µ UB over frequency (ω ), or: D-K ieraion process: DMD-1) 1) Iniialize he ransfer marix D (jω ) (o he ideniy marix) 2) Holding D fixed find K sabilizing ha solves: 3) Holding F L (G,K) fixed find D (jω ) o minimize a each frequency 4) Fi D(jω ) wih a sable minimum-phase ransfer funcion and reurn o sep 2 Ieraion coninues unil or unil H ceases o decrease

33 MIMO Robus Conroller Srucure

34 Conroller K Design R3 Diagram

35 Conroller K Design R3 Inerconnecion Srucure

36 Conroller K Design R3 Weighs

37 Conroller K Design R3 Crierion

38 Conroller K Implemenaion R3 4x7 Conroller Air flow (F a ) Inpu Air Temperaure (T ai ) Discharge Air Temperaure (T ao ao )

39 Conroller K Simulaion R3

40 Conroller K Experimenal R3

41 Comparison of Conroller Performance

42 Comparison of Conroller Archiecures

43 Facors Affecing Performance Sysem Archiecure Model Uncerainy Robus Performance Conroller Design Operaing Range

44 MIMO Conrol - Conclusions Designed and Implemened MIMO Robus Conrol on HVAC Simulaion and Experimen Coordinaed Conrol Acion Decoupling Robusness o Plan Variaion Theory and Experimen Large Performance Gains

45 Robus Reinforcemen Learning Conroller

46 Robus Reinforcemen Learning Moivaion Review of reinforcemen learning Our previous resuls of reinforcemen learning for HVAC Review of robus conrol heory Incorporaing reinforcemen learning agen in robus conrol heory Resuls, Conclusions, Planned Work

47

48 Reinforcemen Learning Agen in Parallel wih Conroller reinforcemen = e

49 Reinforcemen Learning Defines a kind of learning problem. The acion you ake now may have a delayed effec on sysem and on performance evaluaion. Mus find bes sequence of acions, defined as he sequence ha opimizes he sum of performance evaluaions, or reinforcemens. Commonly formulaed as a dynamic programming problem. Solved by esimaing he sum of expeced fuure reinforcemens for f each sae. The muli-sep problem becomes a single sep decision. Dynamic programming assumes knowledge of sae-ransiion ransiion probabiliies. Reinforcemen learning does no. Insead, akes a Mone Carlo approach. a

50 { } ), ( ), ( ), ( ), ( ), ( ), ( ), ( = = = + = + = T k k k k T k k k k a s Q a s R E a s R a s R E a s R a s R E a s Q π π π π π γ γ γ γ Reinforcemen Learning Reinforcemen Learning { } ), ( ), ( ), ( ), ( 1 1 a s Q a s Q a s R E a s Q π π π π γ + = + + Subrac righ side from lef o ge algorihm for updaing Q Subrac righ side from lef o ge algorihm for updaing Q [ ] ), ( ), ( ), ( ), ( 1 1 a s Q a s Q a s R a s Q π π π γ α + = + + Replace expecaion wih sample (Mone Carlo approach) Replace expecaion wih sample (Mone Carlo approach) Temporal Temporal-difference error difference error = = + + T k k k k a s R E a s Q 0 ), ( ), ( γ π π acion acion sae sae policy funcion policy funcion value value funcion funcion discoun facor discoun facor reinforcemen ( error ) reinforcemen ( error )

51 Reinforcemen Learning Q [ R( s, a ) + γq ( s, a ) Q ( s, a )] ( s, a ) = α π + 1 π + 1 π This converges on bes approximaion of value funcion for policy π. To also improve he policy: Q s, a ) = α R( s, a ) [ min Q ( s, a ) Q ( s, a )] + γ ( π 1 π a A π + Q implicily defines he policy: π ( s ) = arg min Q( s, a) Value funcion (Q) learned by criic nework. a A (Q-learning, Wakins, 1989) Policy funcion (π)( ) learned by acor nework.

52 Reinforcemen Learning Resuls (1996)

53

54 Robus Reinforcemen Learning? Learns improved conrol, bu no guaranee of sabiliy. Can we formulae combinaion of PI conrol and RL wihin robus conrol heory? Robus conrol heory is based on linear, ime-invarian invarian ransfer funcions. RL agens are nonlinear,, because of he unis acivaion funcions. RL agens are ime-varying, because hey updae heir parameers o produce improved behavior.

55 Neural Ne for Learning Agen W anh V anh linear Acor Nework (Criic Nework no shown) anh

56 IQCs for Neural Nework as RL Agen Nonlinear par: anh replace wih odd, bounded-slope IQC Time-varying par: weigh updaes replace wih slowly ime-varying IQC Replace wih IQCs only for sabiliy analysis, no during operaion

57 IQCs for Neural Nework as RL Agen Two-layer neural ne as acor, in parallel wih conroller. Now wih anh and varying parameers covered by IQCs.

58 Incorporaing Time-Varying IQC in Reinforcemen Learning Reinforcemen learning algorihm guides adjusmen of acor s weighs. IQC places bounding box in weigh space, beyond which sabiliy has no been verified. Sep 1 iniial guaraneedsable region Sep 4 nex guaraneed-sable region Sep 5 Now learning can coninue unil edge of new bounding box is encounered. UNSTABLE REGION! final weigh vecor Sep 0 iniial weigh vecor rajecory of weighs Sep 2 while learning mus find Sep 3 new sable region weigh rajecory wihou robus consrains weigh rajecory wih robus consrains weigh space (high-dimensional) weigh space (high-dimensional)

59 Tes on Simple Simulaed Task Reference Oupu

60 Trajecory of Weighs and Bounds on Regions of Sabiliy C E D B A iniial weigh vecor

61 Disillaion Column Example of ask for which conrol variables inerac in complex way.

62 Decoupling Conroller Nominal Good response Perurbed Terrible response

63 Robus Conroller Nominal Less aggressive response Perurbed Much improved response

64 Robus Reinforcemen Learning Perurbed case, no learning (from previous slide) Perurbed case, wih learning Sum Squared Error Nominal Conroller Robus Conroller Robus RL Conroller Through learning, conroller has been fine-uned o acual dynamics of real plan wihou losing guaranee of sabiliy!

65 Reinforcemen Learning wihou IQCs Ulimaely achieves same good performance, bu during learning periods of insabiliy occur.

66 Conclusions IQC bounds on parameers of anh and sigmoid neworks exis for which he combinaion of a reinforcemen learning agen and feedback conrol sysem saisfy he requiremens of robus sabiliy heorems. (saic and dynamic sabiliy) Resuling robus reinforcemen learning algorihm improves conrol performance while avoiding insabiliy on several simulaed problems.

67 Curren Work Applying robus reinforcemen learning o HVAC model and real HVAC sysem. Developing coninuous versions of reinforcemen learning. Coninuous sae, acion needed for high- dimensional conrol problems Invesigaing value-gradien mehod (based on Werbos heurisic dynamic programming, 1987). Uses known or learned model of sysem dynamics. Can resul in much faser learning.

68

69 Significan Projec Oucomes Buil Experimenal HVAC Sysem Developed models of HVAC Sysem Developed and Implemened PI Plus Neural Nework Conrol Improved performance Simple o implemen and rain MISO Applicable o many processes

70 Significan Projec Oucomes (con.) Designed MIMO Robus Conrollers Implemened MIMO Conrol Reduced T ao Sele Time by over 300% Decoupling of Conrolled Variables Simulaneous, Coordinaed Conrol Acion Conrollers are Insensiive o Model Uncerainy

71 Significan Projec Oucomes (con.) Designed Robus Reinforcemen Learning Conroller Tesed Conroller on Sandard Problems For he firs ime, a conrol neural nework can be rained while guaraneeing robus sabiliy A poenial breakhrough in he applicaion of neural neworks o conrol Training is by reinforcemen learning, obviaing he need for raining daa ses.

72 Impac of Projec Dramaic Improvemen versus Curren HVAC Conrol Improved Efficiency Sabiliy and Robusness Coordinaed MIMO Acion MIMO Robus Conrol Firs Guaranee of Sabiliy During Reinforcemen Learning. Poenial Cos Savings Insallaion and Mainenance 6 Publicaions Currenly Pursuing 2 Paens 4 Masers and 2 PhD Sudens

73 Fuure Direcions Disseminaion ino Indusry Implemenaion of Robus Learning Conrol on MIMO HVAC Sysem Large Scale Experimenal Plaform Gain-Scheduled Conrollers Nonlinear Modeling PDE Approach Robus Reinforcemen Learning Conrol Theoreical Advances Advanced Robus Learning Algorihms

74