A Learning Agent for Heat-Pump Thermostat Control

Size: px

Start display at page:

Download "A Learning Agent for Heat-Pump Thermostat Control"

Christina Dawson
5 years ago
Views:

1 A Learning Agent for Heat-Pump Thermostat Control Daniel Urieli and Peter Stone Department of Computer Science The University of Texas at Austin

3 Heating, Ventilation, and Air-conditioning (HVAC) systems

consumes But: no longer effective in freezing outdoor temperatures + Backed up by an auxiliary heater Resistive heat

4 Heat-Pump based HVAC System Part of the efforts of moving to sustainable energy Heat-pump is widely used and highly efficient Consumes renewable energy (electricity) rather than gas/oil Its heat output is up to 3x-4x the energy it consumes But: no longer effective in freezing outdoor temperatures + Backed up by an auxiliary heater Resistive heat coil Unaffected by outdoor temperatures But: consumes 2x the energy consumed by the heat-pump heater Heat pump is also used for cooling

5 Thermostat an HVAC System s Decision Maker The thermostat : Controls Comfort Significantly affects energy consumption Current interest evident from companies like NEST, BuildingIQ

6 Goal: Minimize energy consumption while satisfying comfort requirements

7 Goal: Minimize energy consumption while satisfying comfort requirements Contributions: 1. A complete reinforcement learning agent that learns and applies a new, adaptive control strategy for a heat-pump thermostat 2. Our agent achieves 7.0%-14.5% yearly energy savings

Tout HVAC Tset Qhvac UAenv Internal Gains Total Heat Tair Cair Qgains Qair UAmass Solar i i Tmass Cmass 2 a 2 n 2 a 2 n n n air mass % % air mass Va Vb Vc Vn mass air a n a n Ra Rb Rc Rn mass out

Regulator Output Wholesale Market Business Ops Generation Ops/SCADA Transmission Ops/SCADA Distribution Ops/SCADA Energy Management Control/SCADA Actual Impedance Estimated Regulation Point wholesale

loads and markets Open-source software, developed for the U.S.

8 Tout HVAC Tset Qhvac UAenv Internal Gains Total Heat Tair Cair Qgains Qair UAmass Solar i i Tmass Cmass 2 a 2 n 2 a 2 n n n air mass % % air mass Va Vb Vc Vn mass air a n a n Ra Rb Rc Rn mass out mass n n air % % env air Xa Xb Xc Xn mass env Va Vb Vc Vn n n mass Ia Ib Ic In mass 49 $ Market MW Feeder Transformer Regulator Vpri Vreg Control line Estimated Impedance R and X Regulator Relay Regulator Output Wholesale Market Business Ops Generation Ops/SCADA Transmission Ops/SCADA Distribution Ops/SCADA Energy Management Control/SCADA Actual Impedance Estimated Regulation Point wholesale cost ancillary services transmission congestion distribution congestion billing impact End of Feeder Simulation Environment GridLAB-D: A realistic smart-grid simulator, simulates power generation, loads and markets Open-source software, developed for the U.S. DOE, simulates seconds to years Realistically models a residential home Heat gains and losses, thermal mass, solar radiation and weather effects, uses real weather data recorded by NREL ( VD VFeeder VEnd Vset Vdesired VD h l if VD VD, thenvbw Vbw h h if VD VD, thenvbw Vbw if Vset Vmeasured Vbw,adjust tap Power Systems Buildings GridLAB-D Core. Control Systems Markets I rk I mk Vrk Vmk sp Pk Vrk 2 Vrk sp Pk Vmk 2 Vrk J sp Qk Vmk 2 Vmk sp Qk Vrk 2 Vmk G i1 GkiVmi BkiVri 0 i1 I mk I 1 mk Vrk J I I rk rk Vrk Power Systems n n Buildings kivri BkiVmi 0 I mk Vmk I rk Vmk GridLAB-D Core if if Qneeded Qneeded dmaxqcapacitor, dminqcapacitor, Control Systems Markets switch on switch off Qsolar Q mass dt 1 dt C T Q UA T T UA UA UA dt dt 1 C UA T T Q P Q V V S Z V cos Z S I cosi S P% cosp V V V S Z sin Z S I sini S P% sinp V V 100 Z% I% P%

9 Problem Setup Simulating a typical residential home Goal: minimize energy consumed by the HVAC, while satisfying the following comfort spec: Occupants are 12am-7am: At home. 7am-6pm: Not at home. (the don t care period) 6pm-12am: At home.

10 The Default Thermostat

11 The Default Thermostat

12 The Default Thermostat

13 The Default Thermostat

14 Can We Just Shut-Down The Thermostat During don t-care Period? Effective way to save energy Indoor temp. closer to outdoor heat dissipation slows down Simulating it In this case, the result is: Increased energy consumption Failure to satisfy the comfort spec

15 Can We Shut-Down The Thermostat During don t-care Period? Effective way to save energy Indoor temp. closer to outdoor heat dissipation slows down Simulating it In this case, the result is: Increased energy consumption Failure to satisfy the comfort spec Therefore, people frequently prefer to leave the thermostat open all day

16 Can We Shut-Down The Thermostat During don t-care Period? Effective way to save energy Indoor temp. closer to outdoor heat dissipation slows down Simulating it In this case, the result is: Increased energy consumption Failure to satisfy the comfort spec Therefore, people frequently prefer to leave the thermostat open all day However, a smarter shut-down should still be able to save energy

17 From the US Dept. of Energy s website

18 Challenges Desired behavior: Maximize shut-down time while staying above the heat-pump slope Similarly for cooling (no AUX) Challenges: The heat-pump slope: Is unknown in advance Changes every day Depends on future weather Depends on specific house characteristics Action effects are: Drifting rather than constant: since heat is being moved rather than generated, heat output strongly depends on the temperatures indoors, outdoors and along the heat path Noisy due to hidden physical conditions Delayed due to heat capacitors like walls and furniture Also, in a realistic deployment: Exploration cannot be too long or too aggressive Customer acceptance will probably depend on worst-case behavior Making decisions in continuous, high dimensional space

19 Our Problem as a Markov Decision Process (MDP) States: Actions: Transition: Reward: Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

20 Our Problem as a Markov Decision Process (MDP) States: Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: Reward: Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

21 Our Problem as a Markov Decision Process (MDP) States: Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: Reward: e a Δ 2 6pm where: Δ 2 6pm := (indoor_temp_at_6pm required_indoor_temp_at_6pm) Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

22 Our Problem as a Markov Decision Process (MDP) States:??? Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: Reward: e a Δ 2 6pm where: Δ 2 6pm := (indoor_temp_at_6pm required_indoor_temp_at_6pm) Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

23 How Should We Model State? Choosing a state representation is an important design decision. A state variable: captures what we need to know about the system at a given moment is the variable around which we construct value function approximations [Powell 2011] Definition from [Powell 2011]: A state variable is the minimally dimensioned function of history that is necessary and sufficient to compute the decision function, the transition function, and the contribution function.

24 Our Problem as a Markov Decision Process (MDP) States: <T in, Time, e a > Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: Reward: e a Δ 2 6pm where: Δ 2 6pm := (indoor_temp_at_6pm required_indoor_temp_at_6pm) Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

25 Our Problem as a Markov Decision Process (MDP) States: <T in, Time, e a > Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: Reward: e a Δ 2 6pm where: Δ 2 6pm := (indoor_temp_at_6pm required_indoor_temp_at_6pm) Terminal States: Action is taken every 6 minutes Modeling a realistic lockout of the system

26 Expanding State to Compute the Transition Function Can we predict action effects for each of the state variables? Current state representation: <T in, Time, e a > Need to be able to predict T in and e a Method: generate simulated data, use cross-validation to test for regression prediction accuracy

27 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information

28 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high

29 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action

30 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action Measured T in history of 10 temperatures: <t 0 >

31 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action Measured T in history of 10 temperatures: <t 0, t 1 >

32 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action Measured T in history of 10 temperatures: <t 0, t 1, t 2 >

33 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action Measured T in history of 10 temperatures: <t 0, t 1, t 2,, t 9 >

34 Predicting T in Prediction error is unacceptably high state <T in, Time, e a > doesn t capture enough information Add T out directly affects T in. Prediction error still unacceptably high Noise explained as hidden home state add history of observable information Previous action Measured T in history of 10 temperatures: <t 0, t 1, t 2,, t 9 > Resulting state: <T in, T out, Time, e a, prevaction, t 0,,t 9 >

35 Completing the state definition Resulting state: <T in, T out, Time, e a, prevaction, t 0,,t 9 > Can we predict the newly added variables? Trivially, except for T out Therefore, add weatherforecast to state weatherforecast doesn t need to be predicted in our transition function This completes our state definition The final resulting state is: <T in, T out, Time, e a, prevaction, t 0,,t 9, weatherforecast>

36 Our Problem as a Markov Decision Process (MDP) States: <T in, T out, Time, e a, prevaction, t 0,,t 9, weatherforecast> Actions: {COOL, OFF, HEAT, AUX} 1 : 0 : 2 : 4 consumption (e a ) proportion Transition: unknown in advance learned Reward: e a Δ 2 6pm where: Δ 2 6pm := (indoor_temp_at_6pm required_indoor_temp_at_6pm) Terminal States: {s s.time = 11:59pm} Action taken every 6 minutes Modeling a realistic lockout of the system State space is continuous and high dimensional

37 Agent Operation First 3 days: exploration Starting day 4: energy-saving setback policy Choose Random Action If Midnight? Update House Model From data (regression) Choose Best Action (TreeSearch) Record Action Effect: <s,a,s > Observe Resulting State Record Action Effect: <s,a,s > Observe Resulting State

38 Agent Operation First 3 days: exploration Starting day 4: energy-saving setback policy Choose Random Action If Midnight? Update House Model From data (regression) Choose Best Action (TreeSearch) Record Action Effect: <s,a,s > Observe Resulting State Record Action Effect: <s,a,s > Observe Resulting State

39 Exploration Random actions for 3 days Could use more advanced exploration policy However, this is still a realistic setup

40 Exploration Random actions for 3 days Could use more advanced exploration policy However, this is still a realistic setup For instance when occupants are traveling during the weekend

41 Agent Operation First 3 days: exploration Starting day 4: energy-saving setback policy Choose Random Action If Midnight? Update House Model From data (regression) Choose Best Action (TreeSearch) Record Action Effect: <s,a,s > Observe Resulting State Record Action Effect: <s,a,s > Observe Resulting State

42 Update House Model from Data Every midnight, use all the recorded data <s, a, s > to estimate the house s transition function Linear Regression to estimate <s,a> s

Update House Model From data (regression) Choose Best Action (TreeSearch)

43 Agent Operation First 3 days: exploration Starting day 4: energy-saving setback policy Choose Random Action If Midnight? Update House Model From data (regression) Choose Best Action (TreeSearch) Record Action Effect: <s,a,s > Observe Resulting State Record Action Effect: <s,a,s > Observe Resulting State

44 Choosing the Best Action Dealing with continuous high-dimensional state Impractical to compute a value function Run a tree search at every step Choose the first action of the best search as the next action

45 Safety Buffer in a Tree Search C ~ 0 C ~ 2σ

46 Results Simulate 1 year under different weather conditions 21 residential homes of sizes ft 2 Using real data weather recorded in NYC Boston Chicago Why cold cities? Since heating consumes 2x-4x more energy

47 Temperature Graphs Learned Setback Policy

48 Energy Savings

49 Comfort Performance In more than 22,000 simulated days

50 Ablation Analysis Removing features and their combinations State features: prevact: previousaction Hist: temperature history t 0,, t 9 conf: confidence buffer Setting other values to the confidence bound

51 Related Work [Rogers et al. 2011] adaptive thermostat that tries to minimize price & peak demand rather than the total amount of energy. [Hafner and Riedmiller 2011; Kretchmar 2000] use RL inside an HVAC systems, but for tuning the system itself. [T. Peffer et al. 2011] How people use thermostats in homes [Powell 2011] Approximate Dynamic Programming Learning thermostats Commercial companies: NEST, more Do not publish the algorithms used

Carefully defined the problem as an MDP Carefully chose a state representation Using an

52 Summary A complete, adaptive, RL agent for controlling a heat-pump thermostat Achieves 7%-14.5% yearly energy savings in simulation, while satisfying comfort requirements Techniques: Carefully defined the problem as an MDP Carefully chose a state representation Using an efficient, specialized tree-search Experiments run on a range of homes and weather conditions Thank you!

Better Weather Data Equals Better Results: The Proof is in EE and DR!

Better Weather Data Equals Better Results: The Proof is in EE and DR! www.weatherbughome.com Today s Speakers: Amena Ali SVP and General Manager WeatherBug Home Michael Siemann, PhD Senior Research Scientist