INVESTIGATION OF REINFORCEMENT LEARNING FOR BUILDING THERMAL MASS CONTROL

Transcription

1 INVESTIGATION OF REINFORCEMENT LEARNING FOR BUILDING THERMAL MASS CONTROL Simeng Liu nd Gregor P. Henze, Ph.D., P.E. Universiy of Nebrsk Lincoln, Archiecurl Engineering 1110 Souh 67 h Sree, Peer Kiewi Insiue, Omh, Nebrsk U.S.A. ABSTRACT This pper describes simulion-bsed invesigion of mchine-lerning conrol for he supervisory conrol of building herml mss. Model-free reinforcemen lerning conrol is invesiged for he operion of elecriclly driven chilled wer sysems in hevy-mss commercil buildings. The reinforcemen lerning conroller lerns o precool he building nigh before he onse of occupncy bsed on he feedbck i receives from ps conrol cions. The lerning gen inercs wih is environmen by commnding he globl zone emperure sepoins nd exrcs cues bou he environmen solely bsed on he reinforcemen feedbck i receives, which in his sudy is he monery cos of ech conrol cion. No predicion or sysem model is required. Over ime nd by exploring he environmen, he reinforcemen lerning conroller esblishes sisicl summry of pln operion, which is coninuously upded s operion coninues. The conroller lerns o ccoun for he ime-dependen cos of elecriciy, he vilbiliy of pssive herml sorge invenory, nd weher condiions. This sudy reveled h lerning conrol is fesible mehodology o find ner-opiml sepoin profile for exploiing he pssive building herml sorge cpciy. The freedom from building model mkes i especilly rcive in rel-ime conrol problems, nd heoreiclly i cn rech he rue opimum evenully, no mer wh building i is deling wih, if only he environmen could be smpled for n infinie period of ime. The nlysis showed h he lerning conroller is ffeced by he dimension of he cion nd se spce, he uiliy re differenils beween on- nd off-pek, lerning re nd severl oher fcors. Moreover, lerning speed is relively low when deling wih problems wih lrge se spce nd cion spce. INTRODUCTION The moivion of his reserch sems from reserch projec funded by he U.S. Deprmen of Energy h invesiges predicive opiml conrol of cive nd pssive building herml sorge invenory. The gol of his projec is o develop model-bsed supervisory building conroller o exploi he poenil of boh cive nd pssive building herml sorge cpciy. In he firs wo yers of his sudy, boh numericl nlysis nd lborory experimen hve demonsred cos sving poenil when pplying opiml conrol o he uilizion of cive nd pssive building herml sorge invenory. However, modeling complexiy nd inccurcy of his model-bsed pproch hve been reveled s well. Menwhile, numerous pplicion of reinforcemen lerning conrol o engineering problems provides new direcion o ckle his conrol problem in more efficien mnner. To his end, n invesigion hs been crried ou o deermine he fesibiliy nd meris of lerning conrol pplied o he conrol of building cive nd pssive herml sorge invenory. As n iniil sep, simulion environmen in Simulink ws developed o invesige he fesibiliy of pplying reinforcemen lerning conrol o pssive building herml sorge only. REVIEW OF PAST WORK Previous sudies on building herml mss uilizion demonsred he poenil of pek cooling lod reducion nd ssocied elecricl demnd. However, cos svings vry widely mong he published cse sudies (Rbl nd Norford 1991; Conniff 1991; Morris e l. 1994; Keeney nd Brun 1997). In simulion sudy presened by Brun (1990), cos svings for design dy vried from 0-35% depending on sysem ype nd uiliy re. Andresen nd Brndemuehl (1992) demonsred energy nd cos svings poenil by precooling he building srucure, clling enion o he impornce of he mss of furnishing which significnly ffecs he precooling sregy. Brun e l. (2001) developed ool o evlue differen precooling sregies by compring he HVAC uiliy coss in ech pplicion. Simulion sudies hve been crried ou for seleced locions, climes, nd uiliy re srucures. A comprison showed cos svings vrying from 40% bes o zero or even excess coss for some less fvorble cses. In review ricle on lod conrol using building herml mss, Brun (2003) concluded h he svings poenil is very sensiive o he uiliy res, building nd pln chrcerisics, nd weher condiions nd occupncy schedule. The grees cos svings were relized for he cse of hevy consrucion, good pr-lod chrcerisics nd low mbien SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 1 of 11

2 emperure which enbled free cooling during nigh venilion. A simulion sudy ws crried ou by Henze e l. (2004) o invesige he combined usge of boh cive nd pssive building herml sorge invenory. The erm cive refers o herml energy sorge (TES) sysem, which is eiher bsed on chilled wer or ice, nd pssive refers o he building mss iself including srucure nd furniure. The nlysis uses model-bsed pproch o find he opiml globl zone ir emperure sepoin nd TES opering commnds. Subsnil uilizion of boh sorge medi ws observed using perfec building model. However, in rel-ime pplicions, mismch beween building model nd he cul building cn never be compleely voided. In compnion sudy by Liu nd Henze (2004) on he impc of model ccurcy on he quliy of he opimizion sysem, i ws found h mismch of inernl he gin nd building consrucion will significnly ffec he opiml zone ir emperure sepoin. I is desired o hve model clibrion procedure in plce o preven lrge deviions of he model, bu his will increse he compuionl cos nd my lso no be prcicl for rel-ime conrol. Menwhile, even hough he predicive opiml conrol lwys looks forwrd by relying on shor-erm forecss, he knowledge gined from he ps hs been ignored. Consequenly, he reinforcemen lerning conrol prdigm provides n lernive pproch h elimines he need for model nd is buil enirely upon experience while voiding he disdvnges of model-bsed conrol such s modeling complexiy, inccurcy, nd need for predicion. Alhough echniques of mchine lerning hve been widely pplied in mny indusries, he concep of lerning conrol ppers sill new in he re of HVAC conrol. Krechmr e l. (2001) employed reinforcemen lerning ssised by rificil neurl neworks o lern o improve muliple-inpu mulipleoupu (MIMO) conrol performnce of heing sysem wihin sble environmen gurneed by robus conrol. Henze nd Dodier (1997) invesiged lerning conrol of grid-independen phoovolic sysem consising of collecor, bery sorge, nd lod. Q-lerning, model-free reinforcemen lerning lgorihm ws pplied o opimize conrol performnce of he sysem. Simulion nlysis compred he performnce beween he convenionl PV-prioriy conrol nd he opiml conrol. Beer performnce ws found by pplying he reinforcemen lerning o opimize he operion of he sysem. Henze nd Schoenmnn (2003) pplied reinforcemen lerning conrol o he opimizion of cive herml energy sorge sysems. Though reinforcemen lerning conrol proved sensiive o he selecion of se vribles, level of discreizion, nd lerning re, i effecively lerns difficul sk of conrolling herml energy sorge nd displys good performnce. The cos svings compre fvorbly wih convenionl cool sorge conrol sregies, bu do no rech he level of predicive opiml conrol. INTRODUCTION TO LEARNING CONTROL Reinforcemen lerning conrol sems from he developmen of wo differen disciplines, which re psychology nd opiml conrol (Suon nd Bro 1998). I is defined s process in which n gen lerns nd improves is behvior by ril-nd-error inercions wih dynmic environmen o chieve long-erm gol. Algorihms hve been well pplied o solve sequenil decision mking problems. Se Environmen policy Agen Rewrd Acion Figure 1: Schemic of sequenil decision mking problems Figure 1 presens schemic of ypicl sequenil decision mking problem. A ech ime sep or sge, he gen will execue n cion seleced from cion spce A ccording o policy; he environmen will hen be rnsmied from se s o s +1, long wih feedbck signl r, which is defined s rewrd or reinforcemen. The gol of he sequenil decision mking problem is o find n opiml policy h mximizes he ccumuled rewrds in he fuure sring from priculr se 2 k k + 1 k = 0 R = r + r + r +... = r. (1) A policy is he gen s selecion of cions in given se, i.e., i is mpping beween se nd cions, : S A. In Eq.1, is inroduced s discouning fcor, which is used o weigh he fuure rewrds. One of he mos widely pplied pproches o solve sequenil decision mking problems is dynmic SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 2 of 11

3 progrmming (Bellmn 1957), which ssumes here is n explici model of he environmen vilble. The rnsiion of he environmen se is defined s rnsiion probbiliy funcion Pss ' = Pr( s + 1 = s ' s = s, = ) The expression of he rnsiion in erms of probbiliy densiy funcion llows for rndom effecs o be considered. Similrly, he insn rewrd is lso defined s funcion of curren se, curren cion nd nex se rss ' = E( r + 1 s = s, =, s + 1 = s '). In dynmic progrmming, he policy is usully defined s probbiliy funcion (s,) of king cion when in se s. Given policy nd cerin se, he se vlue funcion is defined s V ( s) = E R s = s { } k (2) = E r + k + 1 s = s. k = 0 Similrly, we define he vlue of king cion in se s ccording o policy s Q ( s, ) = E R s = s, = { } k (3) = E r + k + 1 s = s, = k = 0 The gol of he opimizion is o find he opiml policy o mximize he se vlue funcion. Bellmn s principle of opimliy ses h whever he iniil se nd cion re, he remining decisions mus consiue n opiml policy wih regrd o he se resuling from he firs decision. Bellmn s opimliy equion cn be wrien s * * V ( s) = mx Pss '[ rss ' + V ( s ')] (4) s ' Bsed on he recursive nure of dynmic progrmming, reinforcemen lerning lgorihms were developed o del wih problems when here is no explici model vilble of he environmen. The only ccess of he gen o he informion bou he environmen is vi he direc inercions wih he environmen including percepion of he se nd he rewrd. The vlue funcion is progressively esimed by coninuous smpling vlues ssocied wih priculr policy. This so-clled Mone Crlo mehod overcomes he necessiy of he rnsiion probbiliy funcion P. ss ' The rel power of reinforcemen lerning lies in he fc h he gen does no hve o wi unil he erminl cos is incurred o djus is policy (Kelbling e l. 1996). This is relized by he emporl difference (TD) mehod inroduced by Suon (Suon 1988), in which he vlue funcion for priculr se is upded bsed on he previous esimion, immedie rewrd nd he esimed vlue of he nex se. The simples TD mehod, known s TD(0), is defined s: V ( s ) V ( s ) + α[ r + V ( s ) V ( s )] (5) where α is defined s lerning re. The Q-Lerning Algorihm Developed by Wkins (Wkins 1992), he Q-Lerning lgorihm is one of he mos widely pplied reinforcemen lerning lgorihms. The quliy vlue Q( s, ) ssocied wih se-cion pir is inroduced, which cn be considered n esime of he cumulive rewrds by performing cion in se s. Bellmn s opimliy equion cn be formuled s Q( s, ) = Pss '[ rss ' + mx Q( s ', ')] (6) s ' The Q funcion cn be lerned by he lgorihm TD (0), which smples he rnsiion probbiliy hrough repeed cions. Given he se s he gen jus visied, he seleced cion, he nex se s ', nd he reinforcemen r( s,, s '), he vlue funcion cn be upded ccording o: Q( s, ) Q( s, ) + α (7) r( s,, s ') + mx Q( s ', ') Q( s, ) ' An imporn issue for reinforcemen lerning lgorihm is he rde-off beween explorion nd exploiion. The online smpling lgorihm relies on repeedly visiing every se-cion pir o upde he ssocied Q -vlue by mens of explorion process. On he oher hnd, o conrol effecively, he gen should pick n cion for conrol, e.g., he cion wih he highes Q vlue, which is known s greedy exploiion policy. Algorihms hve been inroduced o blnce explorion nd exploiion. One of simples pproches is clled ε-greedy mehod, in which, insed of being greedy ll of he ime, he gen kes non-greedy explorory cions wih probbiliy of ε. Anoher cegory of mehods is clled sofmx cion selecion mehods, mong which, Gibbs or Bolzmnn disribuion is one of he mos populr mehods. I defines he rule of choosing n cion wih probbiliy: Q( s, ) / τ e P( s, ) =. n Q( s, b) / τ e b = 1 The posiive prmeer τ is clled emperure, he lower he vlue of he emperure, he higher he probbiliy becomes h n cion wih high Q-vlue will be chosen. When τ 0, his becomes he greedy policy. SIMULATION ANALYSIS The gol of his nlysis is o design conroller which lerns o conrol he zone ir emperure sepoin o ' SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 3 of 11

4 reduce opering coss by uilizing he building pssive herml sorge cpciy. The Q-Lerning lgorihm will be pplied o govern he model-free lerning procedure, nd subsequenly he opiml policy will be compred wih he resul of model-bsed opiml conrol, which is found by direc serch lgorihm. Developmen of simulion environmen A commercil building nd ssocied energy sysem model ws developed in he Simulink environmen of Mlb o model he dynmic herml response of he building nd energy consumpion of he HVAC sysem. Figure 7 depics he srucure of he simulion environmen. The firs schemic provides n overview of he models. The simulion environmen is mde up of four mjor groups of componens. The firs one consiss of he exernl nd inernl he gin models including modules processing weher d, solr rdiion, nd building inernl he gin. The second group is he building envelope modules. Using se spce modeling, he rnsien he rnsfer hrough ech consrucion elemen of he building is clculed using second-order lumped cpcince model. Previous reserch shows h second-order lumped cpcince model wih one inernl nd one exernl cpcince s well s hree resisors cn dequely pproxime he herml response of building consrucion (Goud e l., 2003). The hird group shows HVAC componens modules, which includes VAV erminl box wih rehe coil, n ir hndling uni including n economizer, cooling coil, nd circulion fn, nd finlly simple pln module including n elecricl chiller, cooling ower nd chilled wer pump. The hree groups of modules re finlly linked ogeher ino he fourh group, he herml nd humidiy blnce funcion block in he fourh schemic, in which he building zone ir emperure is upded ccording o he following equion N dt surfces z Cz = Qin + hi Ai ( Tsi Tz ) + d i= 1 m c ( T T ) + m c ( T T ) inf p Z sys p Z where, Tz, T, Ts re zone ir emperure, mbien ir emperure nd supply ir emperure; C z is he room ir herml cpciy; Qin is he convecive inernl he gin; hi, Ai, T si re he inerior convecion he rnsfer coefficien, surfce re nd emperure of he inernl surfce of he envelope of he building; m m re he mss flow re of infilrion nd inf, sys supply ir, respecively. Similrly, he zone ir humidiy is upded ccording he following funcion: dgz M ( ) ( ) / z = m inf g gz + m sys gs TZ + QL hfg d where M z is he room ir mss; g z, g, gs re moisure conen of zone ir, exernl ir nd supply ir; Q L, h fg re he len he gin o he room spce nd sndrd len he of vporizion of wer in ir respecively. The model hs been vlided wih n EnergyPlus model, here is bou 7~8% discrepncy in he lod profile beween he EnergyPlus nd Simulink models, which is considered ccepble. Simulion prmeers For convenience, he building is modeled s simple box model wih ol re of 1200 m 2, nd only one herml zone is considered. Acully, ps reserch (Liu nd Henze 2004) shows h he simplificion of building geomery nd zoning only mrginlly ffecs he opimizion. Typicl meeorologicl weher d (TMY2) for Omh, Nebrsk re seleced nd lighing power densiies ypicl of office buildings re ssumed. The uiliy re srucure nd occupncy schedule re synchronized in order o simplify he problem s shown in Figure 2. BM1 BM2 BM3 BM1 BM2 BM3 Dy N Dy N+1 Time Figure 2: Profiles of he building modes Re srucure Occupncy A new erm, building mode, is inroduced o fcilie he problem represenion. Building mode describes he chrcerisic of re srucure nd occupncy of building cerin period of ime. Figure 2 presens n exmple wih hree building modes, in which, building mode one (BM1) is defined s he ime period from midnigh o he occurrence of occupncy nd on-pek uiliy re, which is 7:00 AM in our sudy; BM2 is he ime period covering ll he hours wih occupncy nd on-pek uiliy re (7:00 AM o 17:00 PM); nd BM3 covers he remining hours of he dy, which hs no occupncy nd off-pek uiliy re. The purpose of he building modes is o simplify he opimizion problem. Insed of seing he zone ir sepoin hourly, sepoins wihin sme building mode cn be se s he sme. As resul, he number of opimizion vribles is reduced SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 4 of 11

5 from he hours of dy (24) o he ol number of building modes. We cn define more hn hree building modes in ech dy. Menwhile, zone ir emperures re subjec o he consrins h hey lie in he rnge of 15 C o 30 C for unoccupied period nd 20 C o 24 C for occupied period. Implemenion of Reinforcemen Lerning Conrol Now we re ble o formule he problem in he frmework of reinforcemen lerning. Ech building mode is considered sge in he sequenil decision mking problem; he cion vrible is he sepoin of globl zone ir emperure, which will be discreized beween he emperure consrins corresponding o he building mode. We define he problem s n infinie horizon conrol problem. Since our gol is o minimize he ccumuled opering cos in he long-erm, he insn rewrd is defined s he cos of opering during he building mode muliplied by negive one. There re mny wys o selec he se vribles, which should compleely describe ll of he relevn informion of he conrol problem nd is environmen. As n iniil sep of he reserch, we simply define ech building mode s he only se vrible. A Simulink block for he Q-Lerning conroller is implemened ino he simulion environmen nd he clling sequence of he Q-Lerning conroller is presened in Figure 8. The Q-vlue of ech cion-se pir is sored in Q-ble. A he beginning of ech building mode, emperure sepoin mong he vilble opions for he curren se will be seleced ccording o eiher he sofmx or ε-greedy explorion mehod; he sepoin will hen be pplied by he simulion model nd simuled, nd he end of his building mode (beginning of nex mode), cos will be genered, which will be fed bck long wih he informion on he new se ino he lerning conroller. This informion will be used o upde he Q-ble. In our simulion sudy, we ssume h he cpciy of he HVAC sysem is lwys ble o rech he conrol sepoin, bu his my no be rue in n cul pplicion. In he cse where he HVAC sysem does no hve enough cpciy, he seleced cion cnno led o he expeced se rnsiion. However, even in his cse, s long s he se is fully observble nd he cul se cn be scerined, he Q-ble is sill going o be upded properly, nd hen he opiml policy will be idenified evenully. RESULTS Bse cse nd model-bsed opiml conrol cse In order o evlue he performnce of he reinforcemen lerning conroller, is resuls re compred o bse cse, which pplies ypicl nighime sebck conrol of zone ir emperure sepoin, nd model-bsed opiml conrol cse. The nighime sebck llows he sepoin of zone ir o flo o is upper bound (30 C for unoccupied period, 24 C for occupied period). The model-bsed opiml conrol is chieved by pplying direc serch lgorihm (Nelder-Med lgorihm) o find he opiml sepoin profile. The ol opering cos for he simulion period (one dy in his sudy) is seleced s he objecive funcion; opimizion vribles re he sepoins of ech building mode subjec o consrins. Figure 3 presens simple digrm of he model-bsed opiml conrol pproch. Direc serch lorihm Zone ir sepoins for ech building mode Building simulion Model Tol Cos Figure 3: Schemic of model-bsed opiml conrol Since here is no deviion in his sudy beween building model nd cul building s well s beween modeled weher nd cul weher, he model-bsed opiml conrol consiues ruly opiml scenrio. Five cses of model-bsed opimizions hve been crried ou wih he number of building modes vrying from 3 o 24. The rrngemen of building modes nd he opiml sepoin is presened in Tble 5. Two uiliy re srucures hve been esed; he firs one hs n onpek uiliy re of $0.25/kWh nd n off-pek uiliy re s $0.05/kWh; he second re srucure hs he sme off-pek re s he firs one, bu he on-pek re is incresed o $0.50/kWh, which is supposed o exggere he incenive for lod shifing. The resul of model-bsed opiml conrol nd bse cse under wo uiliy re schemes re summrized in Tble 5. Alhough he opiml sepoin vlues re differen given differen building mode nd uiliy re, he generl pern of precooling cn be found mong ll he simulion cses. The cos of one dy s simulions of opiml conrol cses nd bse cse re summrized in Tble 1. Tble 1: Cos of bse cse nd opiml conrol cses Cos [$] Sving [%] Uiliy rio Bse cse 3 BM 6 BM 9 BM 12 BM 24 BM $0.25/$0.05 $64 $55 $56 $56 $55 $57 $0.50/$0.05 $127 $100 $99 $101 $99 $102 $0.25/$ % 13% 13% 14% 11% $0.50/$ % 22% 21% 22% 20% SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 5 of 11

6 Resuls of reinforcemen lerning conrol Differen reinforcemen lerning conrol scenrios hve been invesiged by vrying he cion spce discreizion nd he number of building modes, which represens he dimension of se spce. The size of he Q-ble is defined by he dimension of cion spce muliplied by he dimension of he se spce. Two schemes of cion spce were invesiged in he preliminry sudy (Tble 2). Acion spce Tble 2: Acion spce Off-pek building modes I [18,21,24] On-pek building modes II [16,19,22,25,28] [20,21,22,23,24] Acion spce I hs hree vlues ([18,21,24]), nd i is sme for on-pek nd off-pek periods. The cion spce II hs five cion vlues, bu i is differen o onpek period nd off-pek period. For on-pek periods, i is [20,21,22,23,24], nd [16,19,22,25,28] for off-pek period. This is becuse he upper nd lower bound of zone ir emperure is differen for on nd off-pek periods. In ll of he simulion cses, he Q-ble is iniilized wih zeros for ll enries, nd he weher d for July dy is repeed during he simulion o simplify he rining procedure. The mximum number of rining dys is 6,000. The opiml resuls for he simuled cses re summrized in Tble 3 nd Tble 4. Tble 3: Lerning conrol resuls for uiliy cos rio of 5/1 Uiliy rio: $0.25/$0.05 Acion 3 building mode 6 building mode 9 building mode spce =0.5 =0.3 =0.5 =0.3 =0.5 = /24 I 24 18/ / / /28 22/ / II / / /25/28 22 Tble 4: Lerning conrol resuls for uiliy cos rio of 10/1 Uiliy rio: $0.50/$0.05 Acion 3 building mode 6 building mode 9 building mode spce =0.5 =0.3 =0.5 =0.3 =0.5 = I / / /25 22/ / II / / 28 19/ / / /22 From hese bles i cn be deduced, he performnce of he Q-conroller is ffeced by mny fcors including dimensions of he cion spce nd se spce, on/off pek uiliy rio nd lerning re ec. Generlly speking, he cses wih higher on/off pek uiliy rio genere beer resuls. The bes resul is obined for he cse of hree building modes under uiliy re rio of 10/1. The lerning conroller found he opiml policy wihin 2,000 rining dys, nd is resul is very close o he one found by model-bsed opiml conrol s shown in Figure 4. Sepoin [C] RL-Opiml (lph = 0.5) RL-Opiml (lph = 0.3) Model-Bsed Opiml Time [hr] Figure 4: Comprison of lerning conrol wih model-bsed opiml conrol (BM = 3, uiliy rio = 10/1) Among he cses wih lower uiliy rio, he one wih 6 building modes, cion spce-i nd lerning re α = 0.5 found he bes opiml resul (Figure 5). This gives he SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 6 of 11

7 implicion h less dimension of se spce does no necessrily led o he fser nd beer lerning. However, i is imporn o idenify he proper se spce, by which he problem cn be well represened nd he opiml conrol policy hen will be found. Sepoin [C] Time [hr] RL-Opiml Model-Bsed Opiml Figure 5: Comprison of lerning conrol wih model-bsed opiml conrol (BM = 3, uiliy rio = 10/1, α = 0.5 ) In Tble 3 nd Tble 4, i is noiceble h some opiml vlues of cerin cses hve wo or more vlues; his indices h he Q-vlues of wo or more cionse pirs re ied or very close o ech oher fer he simulion ended (6,000 dys of rining). In he exmple presened in Figure 6, se 1 (building mode 1), wo cions hve he sme Q-vlue, which re higher hn ny oher vlues. This implies h Q-conroller currenly considers h king eiher of hese wo cions would produce he sme reurn. This occurs more frequenly when higher dimensions of boh se nd cion spces re pplied. Simply sying, wih he increse of he dimension of he Q-ble, more rining is needed o le he conroller find he opiml policy. Se-Acion [1,1] Q_vlue = Se-Acion [1,5] Q_vlue = Figure 6: Lerning conrol resuls wih ied Q-vlue The phenomenon bove brings he imporn issue of he lerning conroller o enion, which cn be clled speed of lerning. As menioned before, he mximum rining dys is se 6,000 in his preliminry sudy becuse of limied compuion resources. However, i is found h in some cses he lerning conroller cnno find he opiml policy in he given rining period, such s he exmple given bove, which leds o he ied of vlues of cerin cion-se pirs in he Q- ble. Even for he cses h finlly found he opiml resul, he compuion ime is lso relively longer hn he predicive opiml conrol cses. This resul is o be expeced becuse he lerning conroller does no hve n explici building model vilble bu cn only improve incremenlly from one decision o he nex. CONCLUSIONS AND FUTURE WORK This simulion sudy shows h reinforcemen lerning conrol is fesible mehodology o find ner-opiml sepoin profile for exploiing he pssive building herml sorge cpciy. The freedom from building model mkes i especilly rcive in rel-ime conrol problems, nd heoreiclly i cn rech he rue opimum evenully, no mer wh building i is deling wih, if only he environmen could be smpled infiniely ofen. On he oher hnd, nlysis shows h he opiml policy of lerning conroller is ffeced by he dimension of he cion nd se spce, he uiliy re differenils beween on- nd off-pek, lerning re nd severl oher fcors. Lerning speed is relively low when deling wih problems wih lrge se spce nd cion spce. Fuure work is needed o ddress hese problems o exploi he power of reinforcemen lerning conrol. I ws found priculrly imporn o selec he proper cion spce nd se spce for he lerning conroller. In his sudy, he se vrible is simply defined s he building mode. I is desirble o define se vribles, which describe he herml hisory of he building nd mbien condiions. However, i is imporn o relize h here is downside ssocied wih expnding he se spce. Adding new se vribles, such s mbien condiion, cn mke he se spce represenion more pproprie, i.e., i cn represen he environmen closer. On he oher hnd, ny se vrible h is dded bu h plys no role will prolong he lerning process since explorive cions wihin irrelevn ses do no conribue o lerning he cos funcion. The resoluion of he discreizion of he coninuous cion spce lso needs o be nlyzed furher. Insed of rbirrily choosing 3 or 5 discree cion vlue, n opiml size of he cion spce needs o be idenified. One my rgue h he on/off pek uiliy rio used in his sudy is no very relisic (5/1 or 10/1). I ws emped o exggere he lod-shifing incenive from SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 7 of 11

8 he uiliy re srucure in his simulion nlysis. However, he rel cos for ech building mode my no necessrily be he insn rewrd for he lerning conroller. Pos-processing of he rel cos my be useful o mke he lerning conroller lern fser. I is poin of fuure reserch o idenify modified rewrd signl insed of using he rel cos. One disdvnge of he reinforcemen lerning conroller is he speed of lerning. Compred wih model-bsed opiml conrol, lerning conrol needs much more rining d o find he opimum, which my no be prcicl in he perspecive of compuionl resources when lrge se nd cion spces re involved. One lernive wy o overcome his problem is o implemen se of neurl neworks o replce he funcions of he Q-ble. I is lso expeced o ccelere he lerning process of he conroller. NOMENCLUTURE s R P r ss ' ss ' (s,) V ( s) Q ( s, ) discree ime sep or sge se cion cumulive discouned rewrd (or reurn) following probbiliy of rnsiion from se s o se s under cion expeced insn rewrd on rnsiion from se s o se s under cion probbiliy of king cion in se s under he policy vlue of se s under policy vlue of king cion in se s under policy V *( s ) vlue of se s under opiml policy α REFERENCES lening re discoun fcor [1] Andresen, I. nd Brndemuehl, M. J. (1992) He sorge in building herml mss: prmeric sudy. ASHRAE Trnscions 98 (1). [2] Bellmn, R.E. (1957). Dynmic Progrmming. Princeon Universiy Press, Princeon. [3] Brun, J. E. (1990) Reducing energy coss nd pek elecricl demnd hrough opiml conrol of building herml mss. ASHRAE Trnscions 96 (2): [4] Brun, J. E., Mongomery, K.W. nd Churvedi, N. (2001) Evluing he performnce of building herml mss conrol sregies, In. J. of HVAC&R Reserch, 7 (4): [5] Brun, J. E. (2003) Lod Conrol Using Building Therml Mss, Journl of Solr Energy Engineering, Vol. 125, No. 3, pp , Americn Sociey of Mechnicl Engineers, New York, New York. [6] Conniff, J. P. (1991) Sregies for reducing pek ir-condiioning lods by using he sorge in he building srucure. ASHRAE Trnscions 97 (1): [7] Goud M.M.; Underwood C.P.; Dnher S. (2003) "Modelling he robusness properies of HVAC pln under feedbck conrol." Building Services Engineering Reserch nd Technology, 1 December 2003, Vol. 24, No. 4, pp [8] Henze, G. P., Dodier, R. H. nd Krri, M. (1997) Developmen of Predicive Opiml Conroller for Therml Energy Sorge Sysems. Inernionl Journl of HVAC&R Reserch, Vol. 3, No. 3, pp [9] Henze, G.P. nd Schoenmnn, J. (2003) Evluion of Reinforcemen Lerning Conrol for Therml Energy Sorge Sysems. Inernionl Journl of HVAC&R Reserch, Americn Sociey of Heing, Refrigering, nd Air-Condiioning Engineers, Aln, Georgi. [10] Henze, G.P., Felsmnn, C. nd Knbe, G. (2004) Evluion of Opiml Conrol for Acive nd Pssive Building Therml Sorge. Inernionl Journl of Therml Sciences, Februry [11] Kelbling, L.P., Limn, M.L., nd Moore, A.W. (1996) "Reinforcemen Lerning: A Survey", Journl of Arificil Inelligence Reserch, 4: [12] Keeney, K.R. nd Brun, J.E. (1996) A simplified mehod for deermining opiml cooling conrols sregies for herml sorge in building mss. In. J. of HVAC&R Reserch, 2 (1): [13] Keeney, K. R. nd Brun, J. E. (1997) Applicion of building precooling o reduce pek cooling requiremens. ASHRAE Trnscions 103 (1): [14] Krechmr, R.M., Young, P.M., Anderson, C.W., Hile, D.C., Anderson, M. L., Delnero, C. C. (2001) Robus Reinforcemen Lerning Conrol wih Sic nd Dynmic Sbiliy. Inernionl Journl of Robus nd Nonliner Conrol. no. 11, pp [15] Liu, S. nd Henze, G.P. (2004) Impc of Modeling Accurcy on Predicive Opiml Conrol of Acive nd Pssive Building Therml Sorge Invenory." ASHRAE Trnscions, Technicl Pper No Vol. 110, Pr 1. [16] Morris, F.B., Brun, J. E. nd Tredo, S. J. (1994) Experimenl nd simuled performnce of opi- SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 8 of 11

9 ml conrol of building herml sorge. ASHRAE Trnscions 100 (1): [17] Rbl, A. nd Norford, L.K. (1991) Pek lod reducion by precondiioning buildings nigh, Inernionl Journl of Energy Reserch 15: [18] Suon, R.S. (1988). Lerning o predic by he mehods of emporl differences. Mchine Lerning 3: [19] Suon, R.S. nd Bro, A.G. (1998) Reinforcemen lerning: An inroducion. MIT Press, Cmbridge, MA. [20] Wkins, C. nd Dyn, P. (1992) Q-lerning, Mchine lerning, SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 9 of 11

10 Overll Building Model Building Envelope Model HVAC Sysem Model Therml & Humidiy Blnce Funcion Figure 7: Schemic of he simulion environmen Dy N-1 Dy N Dy N+1 Building Mode 1 Building Mode 2 Building Mode M Simulion Time Zone ir sepoin Cos Selec n cion Red Q vlue Q ble Wrie Q vlue Upde he Q vlue Q Lerning Conroller Figure 8: Schemic of clling sequence of he lerning conroller SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 10 of 11

11 Tble 5: Summry of resuls for model-bsed opiml conrol 3 building mode 6 building mode 9 building mode 12 building mode 24 building mode Time BM T_sp1 T_sp2 BM T_sp1 T_sp2 BM T_sp1 T_sp2 BM T_sp1 T_sp2 BM T_sp1 T_sp2 0: : : : : : : : : : : : : : : : : : : : : : : : T_sp1 denoe he opiml sepoin under On/Off pek uiliy rio $0.25/$0.05 T_sp2 denoe he opiml sepoin under On/Off pek uiliy rio $0.5/$0.05 SimBuild 2004, IBPSA-USA Nionl Conference Boulder, CO, Augus 4-6, Pge 11 of 11