Uncertainty of Measured Energy Savings from Statistical Baseline Models

Transcription

1 4375 VOL. 6, NO. 1 HVAC&R RESEARCH JANUARY 2000 Uncertainty of Measured Energy Savings fro Statistical Baseline Models T. Agai Reddy, Ph.D., P.E. David E. Claridge, Ph.D., P.E. Meber ASHRAE Meber ASHRAE Baseline odels are a crucial eleent in deterining savings fro energy conserving easures. The baseline odel is obtained by regressing the energy consuption data for the period prior to the ipleentation of the energy conservation easures. The widely used criteria for deterining adequacy of a particular baseline odel is based on statistical cutoff criteria that do not give the user a knowledge of the error inherent in the savings deterination. The absolute cutoff criteria ay not be appropriate since baseline odel developent is not the desired end. It is proposed that odels be evaluated in ters of the ratio of the expected uncertainty in the savings to the total savings ( E save /E save ). This physically and financially intuitive easure perits the user to vary the criteria according to factors ost relevant for a particular energy conservation project. Siplified expressions for ( E save /E save ) appropriate for use by practitioners as applicable for cases with uncorrelated data and for use with correlated tie series data are developed and discussed in the context of case-study data. The use of this concept to logically select the ost appropriate easureent and verification protocol to verify savings is also described. INTRODUCTION The International Perforance Measureent and Verification Protocol (IPMVP 1997) proposes three onitoring and verification ethods or options for deterining energy savings fro energy conserving easures (ECM), either due to retrofits or operational and aintenance iproveents in buildings. The first (Option A) deals with one-tie easureents before and one-tie easureents after the ECM. Options B and C, however, deal with continuous easureents before and after the ECM. Option B typically involves onitoring specific end uses for a period (e.g. weeks to onths) before and after the retrofit, while Option C entails easuring whole building consuption for a baseline period of at least several onths before the retrofit and continuously following the retrofit. The discussion in this paper is liited only to Options B and C wherein baseline odel developent is explicit. Such savings will be called easured savings to distinguish the fro audit-estiated savings (or other savings based priarily on engineering analysis) and are deterined fro easured energy perforance data. With building energy use data, either utility bills or onitored data that is available, both prior to and after the ECM, the procedure used to deterine savings is to noralize the data for any discrete one-tie changes (e.g., in conditioned area or in occupancy), and to identify a pre-ecm odel using known quantifiable variables that are likely to change after the ECM is ipleented but which are not part of the ECM. These variables include cliatic and building related variables such as internal loads or changes to building or HVAC operational schedules, (Katipaula et al. 1998). In ost cases, however, a odel with the outdoor dry-bulb teperature as the only T. Agai Reddy is an associate professor in the Departent of Civil and Architectural Engineering, Drexel University, Philadelphia, Pennsylvania and David E. Claridge is a professor in the Energy Systes Laboratory, Mechanical Engineering Departent, Texas A&M University, College Station, Texas THIS REPRINT IS FOR DISCUSSION PURPOSES ONLY. It is reprinted fro the International Journal of Heating, Ventilating, Air-Conditioning and Refrigerating Research. Sold only to attendees of the 2000 ASHRAE Annual Meeting, it is not to be reprinted in whole or in part without written perission of the Aerican Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., 1791 Tullie Circle, Atlanta, GA Opinions, findings, conclusions, or recoendations expressed in this paper are those of the author(s) and do not necessarily reflect the views of ASHRAE. Written questions and coents regarding this reprint should be sent to the ASHRAE Handbook Editor.

2 4 HVAC&R RESEARCH regressor variable is adequate (Kissock et al Reddy et al. 1997a). There are, subsequently, two variants used to deterine savings depending on how the post-ecm data is used: 1. The Noralized Annual Average Savings approach, which involves developing an outdoor dry-bulb teperature-based post-ecm regression odel. Long-ter average dry-bulb teperature values over several years are used to drive this odel along with the baseline odel (Fels 1986, Ruch and Claridge 1993); and 2. The Actual Savings over a certain tie period in which the baseline odel is driven with actual onitored outdoor teperature under post-ecm conditions. The su of the differences between these values and the observed post-ecm values over that tie period yields the savings (Kissock et al. 1998, IPMVP 1997). This study is concerned priarily with (2) above. In this case, the uncertainty in the baseline consuption as deterined fro the statistical odel becoes the ajor deterinant of the uncertainty in the resulting easured savings. The uncertainty is of ajor interest to building owners and financial institutions associated with energy service contracts that are intended to reduce energy cost of operating specific buildings. The uncertainty in the baseline consuption is in turn directly related to the goodness of fit of the baseline odels. Little work has been done to establish sound statistical liits for gauging the goodness-of-fit of the baseline odel. The ost widely used criterion is that proposed by Reynolds and Fels (1988). They proposed (1) that odels with coefficient of deterination, i.e., R 2 values >0.7 and coefficient of variation (CV values) < 7%, or (2) that odels with low R 2 values and CV < 12%, be considered reliable odels. Models that do not satisfy at least one of these criteria be considered poor and that data fro buildings with poor baseline odels should be discarded while perforing deand-side anageent evaluations. These cutoffs appear arbitrary, since no clear rationale for their choice is provided. Further, the interpretation of R 2 and CV for change point odels of energy use versus outdoor dry-bulb teperature (Kissock et al. 1998) is unclear since these indices have a clear physical/intuitive eaning only when applied to linear odels. Further, we contend that there is no basis for developent of absolute statistical cutoff criteria since baseline odel developent is not an end in itself. We propose that the ore relevant criterion for use in deterining whether a baseline odel is acceptable or not is the fractional uncertainty in savings easureent E save /E save. As an exaple, consider two cases. In one, an energy audit estiates that the retrofit is likely to reduce energy use by 30%, and for the other the anticipated energy reduction is only 5%. A uch poorer baseline odel in the first case would produce a particular choice of required uncertainty in savings (e.g., E save /E save 0.1), but a very good odel would be needed in the second case to produce the sae uncertainty. A physically based criterion such as this would provide ore eaningful evaluations of deand side anageent and ECM progras. This paper presents this physical concept in statistical and atheatical ters and illustrates the approach with energy use utility bill data fro several Ary bases nation-wide as described by Reddy et al. (1997b). The logical consequence of using this criterion for selection of the ost appropriate easureent and verification ethod as described in the IPMVP (1997) to verify savings is also discussed. STATISTICAL CRITERIA The goodness-of-fit of statistical baseline odels is custoarily expressed in ters of statistical indices, the coefficient of deterination R 2 and the coefficient of variation of the root ean square error (CVRMSE). In case of a ean odel, the coefficient of variation of the standard deviation (CVSTD) is appropriate. In this study, y is the dependent variable of soe function of the predictor variable(s), y the saple arithetic ean, and ŷ the regression odel-predicted

3 VOLUME 6, NUMBER 1, JANUARY value of y. Then for linear odels with an intercept ter, these indices are defined as follows (Draper and Sith 1981): R 2 y i ŷ i ( )2 ( y i y )2 (1) and CVRMSE 1 -- y ( y i ŷ i ) n p 0.5 (2a) 1 CVSTD -- y ( y i y ) n (2b) where all suations are done fro i 1 to i n, where n is the nuber of observations, and p is the nuber of paraeters in the regression odel. R 2 values are bounded in the range {0,1} while CV values (i.e., either CVRMSE or CVSTD) have a lower bound of zero and no upper bound. The closer to unity the R 2 values and the closer to zero the CV of a linear odel, the better the odel is deeed to be. As discussed later, these indices, though representative of the goodness-of-fit of the odel, have slightly different interpretation, and situations often arise when the analyst is uncertain as to which easure to turn to in order to ake absolute evaluations of odels identified fro different energy use data. For exaple, gas use often varies widely over the year, with a very low value in suer and a very high value in winter. As a result the odels ay have, for exaple, a R 2 value of 0.98 and a CV value of over 20% (Reddy et al., 1997b). Whether this should be considered a good odel or not is abiguous since the R 2 value taken by itself would suggest a good odel, while the CV value considered alone would suggest a poor odel. This paper will discuss this issue and provide an understanding of both these indices in ters of baseline energy odels. There are fundaental differences between the R 2 and CV statistics defined by Equations (1) and (2), respectively. Both the statistics are noralized indices, but the noralization is done differently and so their interpretation is different. Both have essentially the sae nuerator, but the denoinators are different. R 2 is representative of the variation in the dependent variable data explained by the odel as copared to the variation of the data about the ean, while CV is the ean variation in the data not explained by the odel noralized by the ean value of the dependent variable. As illustrated conceptually in Figure 1a, if linear odels are fit to two sets of data, both of which have the sae ean and slope but one with greater data spread than the other (in Figure 1a the data spread of the right-hand set is twice that of the left-hand side), these odels will have R 2 and CV values as shown. There is little abiguity here because either index can be used to identify the preferred odel. However, for data sets with the sae ean and spread but different slopes (see Figure 1b), the CV values are unaltered while the R 2 values are widely different. A third set of odels with one set of data having twice the ean value and twice the variability is shown in Figure 1c. Both odels have the sae CV values but widely different R 2 values. Thus, in ters of baseline energy odeling, R 2 is of liited value when the slope of the linear odel is too high or too low, while the CV is not sensitive to the slope of the odel but it is sensitive to the data spread with respect to the ean value. Hence, if the objective is only to evaluate how well a baseline energy odel has captured the variation in the data, and to assess how

4 6 HVAC&R RESEARCH Figure 1. Conceptual plots to illustrate how different configurations of linear data affect odel R 2 and CVRMSE (a) Both odels have the sae ean values, but one has twice the data spread of the other (b) Both odels have the sae ean and data spread (c) One odel has twice the ean value and the data spread as the other

5 VOLUME 6, NUMBER 1, JANUARY different odel functional fors have fared against one another when fit to the sae data set, then R 2 is the right easure to use. However, if the uncertainty in the savings prediction is the quantity upon which we wish to base our choice of baseline odel reliability and to do so with different data sets, then CV is the ore relevant easure to use. The CV provides a ore direct correspondence between the aount of rando variation relative to the ean. If the user, were analyzing easured energy use data fro several buildings and decided (based on the particular proble) that odels with rando coponents less than 10% of the ean were acceptable, then odels with CV < 0.1 would be accepted for subsequent predictive purposes. Several studies have pointed out that building energy use data E plotted against outdoor dry-bulb teperature T often exhibits a change point behavior, i.e., a segented linear trend (Fels 1986, Kissock et al. 1992, IPMVP 1997). A four-paraeter change point (4-P) cooling energy use odel conceptually illustrated in Figure 2 would have the following functional for: E E c + a(t T c ) + b(t T c ) + (3) If the base portion (i.e., Region 1 of Figure 2) is horizontal, i.e., of zero slope, a three-paraeter (3-P) odel is obtained. The lower slope in Region 1 and higher slope in Region 2 are representative of cooling energy odels, while the trend is reversed for heating energy use odels. The theoretical fraework of linear segented odels, which are non-linear odels, has been discussed by Beale (1960) and by Hinckley (1969) and has been subsequently adapted to energy use data by Goldberg (1982) and Ruch and Claridge (1993). Reddy et al. (1998) pointed out that the non-linear behavior can be siplified to a linear proble if the uncertainty associated with the change point is overlooked. In such a case, a 4-P odel siplifies down to two siple linear odels with a known hinge or change point value T c. This assuption is iplicit throughout this paper. Figure 2. Illustration of data points fitted by a change-point four-paraeter cooling odel (4P-C)

6 8 HVAC&R RESEARCH Figure 3. Conceptual plots to illustrate additive and ultiplicative errors in linear and change-point three paraeter-cooling odels Appendix A presents an unsuccessful attept to renoralize the R 2 and CV indices, which though suitable for linear odels, has been found to be isleading for the four-paraeter change point odels. Another intuitive approach is to calculate the CV (i.e., either the CVRMSE or the CVSTD as appropriate) of each of the two segents of the change point odels separately fro Equation (2) and weight the appropriately in order to obtain a weighted CV. This is addressed later on in this paper. An underlying concept in odel prediction uncertainty has to do with the type of noise present in the energy use observations, whether additive or ultiplicative (Draper and Sith 1981). For a siple linear (2-P) odel as shown in Figure 3 (a), the odel for would be as follows: Additive noise:e (a + bt ) + αr (4) Multiplicative noise:e (a + bt )(1 + αr) (5) where R is a norally distributed rando nuber with zero ean and unit standard deviation, and α is a ultiplier, which can be used to inflate or deflate the error effect. Note that the two would be identical for a ean odel (i.e., b 0) with a 1. Consequently for a three paraeter (3-P) odel, as shown in Figure 3b, we would have the sae data spread about the ean line for the left-hand portion of the odel line for both types of rando errors. Prior studies indicate that there sees to be no consistency as to which type of error occurs in energy use data (Reddy et al. 1997a). Both these cases will be separately considered in the siplified expressions for E save /E save developed in the next section. RATIONAL METHODOLOGY FOR BASELINE MODEL EVALUATION The priary objective of this paper is to develop a procedure for assessing the soundness of a baseline odel for subsequent savings deterination. The concept of a universal or absolute statistic is extended to view the issue in ters of soe of the physical considerations of the proble. A ajor consideration is the relative ipact of the retrofit on the baseline energy use. Both issues of goodness of fit of the baseline odel and the expected fractional reduction in energy use due to the ECM are logical considerations. Though a certain aount of subjectivity is bound to be associated with the liits of odel acceptability in such a paradig, the interactions between the baseline odel identification and the subsequent objective of deterining energy retrofit savings needs to be recognized. The uncertainty arising fro a odel identified fro

7 VOLUME 6, NUMBER 1, JANUARY Figure 4. Conceptual plot showing how energy savings are deterined operational data that does not cover a whole annual cycle was not considered (Reddy et al. 1998). Hence the baseline odel is assued to be identified fro yearlong data with no odel extrapolation errors. Conceptually, as shown in Figure 4, actual savings (as against noralized savings) E save over periods (hours, days or onths, depending on the type of energy use data available) into the retrofit period are calculated as follows: j1 j j1 Ê pre, j j1 E eas, j (6) where nuber of periods (hour, day, or onth) in the post-retrofit period Ê pre pre-retrofit energy use predicted by the baseline odel per period, and E eas easured post-retrofit energy use per period. In order to ake the equations ore readable, we shall rewrite Equation (6) as: Ê pre, E eas, (7) Ê pre, where, E now denotes the energy use over periods. For exaple, denotes the su of individual odel predicted values of baseline energy use. With the assuption that odel prediction and easureent errors are independent, the total variance is the su of both: ( ) 2 ( Ê pre, ) 2 + ( E eas, ) 2 (8) The total prediction uncertainty increases with, i.e., as the post-retrofit period gets longer. However, as the aount of energy savings also increases with, a better indicator of the uncertainty is the fractional uncertainty defined as the energy savings uncertainty over periods divided by the energy savings over periods:

8 10 HVAC&R RESEARCH ( Êpre, ) 2 + ( E eas, ) ( Ê pre, ) 2 F (9) where F is the ratio of energy savings to pre-retrofit energy use, i.e., F ( ) Ê pre, Êpre, E eas, (10) Equation (9) provides a eans of calculating the fractional uncertainty in the actual savings, which consists of a ter representative of the regression odel prediction uncertainty and another ter representative of the easureent error in the post-retrofit energy use. The easureent error in the pre-retrofit energy use is inherently contained in the odel goodness-of-fit paraeter (naely, the ean square error (MSE) statistic), and should not be introduced a second tie. In case the easureent uncertainty is sall (for exaple, when electricity is the energy channel its associated error is of the order of 1 to 2%), the fractional uncertainty in our savings easureent can be siplified into E pre 0.5 ( ) E pre F (11) E pre where is the ean pre-retrofit energy use during the selected period. This expression can be cast into a ore useful for by aking certain siplifying assuptions. The effect of easureent errors in the post-retrofit data was neglected and it was assued that odel prediction errors are the only source of prediction uncertainty (the various sources of errors as applied to building energy analysis are described by Reddy et al. 1998). Two separate cases were considered: (a) odels with uncorrelated residuals which one would encounter when analyzing utility bills, and (b) odels with correlated residuals often encountered with odels based on hourly or daily data (Ruch et al. 1999). Models with Uncorrelated Residuals and Additive Errors The prediction uncertainty of a siple linear odel identified fro rando data is given in statistical textbooks (Draper and Sith 1981). The regression odel prediction uncertainty for an individual observation during the post-retrofit period, for a odel with constant variance and uncorrelated odel residual behavior, is: Ê j ( T j T ) 2 MSE n n ( T i T ) 2 i1 0.5 (12) where n MSE variance of the odel error ( ) 2 ( n p) (13) T j i 1 E i Ê i individual value of outdoor dry-bulb teperature during the prediction or post-retrofit period

9 VOLUME 6, NUMBER 1, JANUARY T ean value of T i ( i.e., ean value of the outdoor teperature) during odel identification or pre-retrofit period The retrofit savings ethodology is not based on individual predictions of Ê j, but the su over days of the Ê j values. The prediction error of a su of future observations (as is needed for deterining energy savings) is given by Theil (1971): Ê MSE 1 X post ( X X) [ ( X post + I )1] (14) where X is the atrix of regressor variables during the pre-retrofit period, X denotes the transpose of X, I is an identity atrix, and the subscript post indicates post-retrofit period. Note that pre and post ultiplication of the atrix within the square brackets by unit atrices 1 is akin to suing all the eleents of the atrix. Equation (14) involves atrix algebra and was siplified using nuerical trials to for the following equation that holds to within about 10% accuracy: ( T j T j ) MSE E pre F --- j n n ( T i T i ) 2 i MSE (15a) E pre F n n where n, nuber of observations in the baseline and the post-ecm periods, respectively T j, T i average outdoor dry-bulb teperature values during post-ecm and during odel identification (i.e., pre-ecm) periods, respectively. Finally, for baseline odels with additive errors, Equation (15a) can be expressed in ters of the standard CV statistic (either the CVRMSE or the CVSTD as appropriate) as: RMSE n E pre F 1.26CV n F (15b) 1.26CV when n is large (say, n > 60) 1 2 F (15c) Note that the above expression yields the fractional energy savings uncertainty at one standard error (i.e., at 68% confidence level). For other confidence levels, say 95%, the bounds have to be ultiplied by the student t-statistic evaluated at significance level and (n-p) degrees of freedo (Draper and Sith 1981). Models with Uncorrelated Errors and Multiplicative Errors As illustrated in Figure 3, odel residuals will provide the necessary indication as to whether the errors are additive or ultiplicative. Reddy et al. (1997a) had proposed the following epirical odification to Equation (15) for the prediction uncertainty in the case of ultiplicative errors:

10 12 HVAC&R RESEARCH 1.26 ( MSE MSE 2 2 ) + MSE n 0.5 (16) where 1 and 2 are the nuber of data points on either side of the change point during the post-retrofit period, MSE 1 and MSE 2 are the ean square errors of the data points on either side of the change point as shown in Figure 2, and MSE is given by Equation (13). MSE 1 (and MSE 2 by analogy) can be calculated fro pre-retrofit data: n 1 MSE 1 ( E i Ê i ) pre n 1 i (17) Finally, the fractional uncertainty in the annual savings is: CV MSE MSE MSE n F (18) which, to be consistent with Equation(15b), can be re-written as: WCV n F (19) where WCV is tered the weighted CV value for a change point odel with ultiplicative errors, and is defined as: WCV CV MSE MSE MSE n n 0.5 (20) Defining the WCV is this anner allows the uncertainty of change point baseline odels with either additive or ultiplicative errors to be treated in the sae anner. In the reainder of the paper, the ter CV to denote either the CV or the WCV is used as appropriate. Models with Correlated Residuals Equations (15) or (19) are appropriate for regression odels without serial correlation in the residuals. This would apply to odels identified fro utility (i.e., onthly) data. When odels are identified fro hourly or daily data, previous studies (Ruch et al. 1999) have shown that serious autocorrelation often exists. These autocorrelations ay be due to (1) pseudo patterned rando behavior due to the strong autocorrelation in the regressor variables (for exaple, outdoor teperature fro one day to the next is correlated), or (2) to seasonal operational changes in the building and HVAC syste not captured by an annual odel. Consequently, the uncertainty bands have to be widened appropriately. Accurate expressions for doing so have been proposed by Ruch et al. (1999). In the fraework of this study, a nuber of nuerical tri-

11 VOLUME 6, NUMBER 1, JANUARY als were perfored and found that the following siplified treatent yields results accurate to within 20% of those proposed by Ruch et al. (1999). Fro statistical sapling theory, the nuber of independent observations n of n observations with constant variance but having a lag 1 autocorrelation ρ equals (Neter et al. 1989): n n ρ 1 + ρ (21) For exaple, if n 365 (i.e., one year of daily data), and ρ 0.85, then n , which iplies that there were effectively only about 30 independent observations. Equation (15) in the presence of serial autocorrelation can be expressed as: 1.26CV --- n n n F (22) The CV value calculated using n degrees of freedo has been renoralized by the new degrees of freedo n. Discussion Equations (15), (19) and (22) provide a ore rational eans of evaluating the accuracy of our baseline odel to deterine savings. Figure 5 that has been generated based on Equation (15b) allows easy deterination of fractional savings uncertainty. Consider a change point baseline odel based on daily energy use easureents with a CV (or WCV) of, for exaple, 10%, we wish to assess the uncertainty in savings for 6 onths into the post-retrofit period for a retrofit easure that is supposed to save 10% of the pre-retrofit energy use (i.e., F 0.1). Then for 6 onths days, and CV 10%, the y-ordinate value fro Figure 5 is 0.01, i.e., E save /E save (0.01/0.1) %. On the other hand, if a baseline odel is relatively poor, with, for exaple, a CV of 30%, and the retrofit is supposed to save 40% of the preretrofit energy use, then fro Figure 5, E save /E save (0.028/0.4) %. Hence, the first odel, despite having a CV value three ties lower than that of the second odel, leads to a larger fractional uncertainty in the savings than the second case. The above exaple serves to illustrate how viewing the retrofit savings proble in the perspective outlined in this paper is ore relevant than erely looking at the baseline odel goodness-of-fit. This exaple is based on 68% confidence level (i.e., t-statistic 1). Suitable corrections need to be ade for different confidence levels. The variation of the fractional uncertainty E save /E save with CV, n,, and F is of interest to energy anagers and energy service copanies while negotiating energy conservation service contracts. If utility bills are the eans of savings verification and if yearlong pre- and post-retrofit data are available, then Table 1 provides an indication of how E save /E save varies with CV (or WCV) and F. For exaple, if both negotiating parties are cofortable with a fractional uncertainty in energy savings of 10% at the 68% confidence level, and if the retrofits are expected to save 20% (i.e., F 0.2), then a baseline odel with a CV of 5% or less will satisfy the expectations of savings verification when one year of pre-retrofit and one year of post-retrofit data are available. Table 2 provides the sae inforation as Table 1 but assuing that daily onitored data is available for savings verification (i.e., we now have 365 data points instead of 12 data points only during either period) and that no residual autocorrelation is present (i.e., ρ 0). For the above illustrative case, baseline odels with up to 30% CV will still prove satisfactory.

12 14 HVAC&R RESEARCH Figure 5. Variation of fractional uncertainty in savings [Given by Equation (15b) at one standard error when daily easured energy values are available for post-retrofit period assuing no residual auto-correlation] Table 1. Fractional Energy Savings Uncertainty [At 68% confidence level after one year as given by Equation (15b) with baseline odel identified fro yearlong utility bills ( n 12) and no residual autocorrelation] CV F Table 2. Fractional Energy Savings Uncertainty [At 68% confidence level after one year as given by Equation (15b) with baseline odel identified fro yearlong daily onitored data ( n 365) and no residual autocorrelation] CV F

13 VOLUME 6, NUMBER 1, JANUARY CV Table 3. Product of Fractional Energy Savings Uncertainty and Ratio of Energy Savings to Baseline Energy Use [At 68% confidence level and savings fraction after one year as given by Equation (22) with baseline odel identified fro year-long daily onitored data ( n 365)] ρ If the odel residuals exhibit auto-correlated behavior, then Equation (22) should be used. Since an extra variable, naely ρ is present, it is better to look at how the variable [( / ) F ] varies with ρ (Table 3). The nubers below ρ 0 have a direct correspondence to those given in Table 2. For exaple, if CV 0.2, fro Table 3, ( / ) F Further, if F 0.2, then / 0.013/ , which is alost identical to the value of shown in Table 2 for CV 0.2 and F 0.2. As shown in Table 3 the ter [( / ) F ] increases as ρ increases. For exaple, for the sae case of CV 0.2 and F 0.2, and for ρ 0.85, / 0.048/ , which is alost four ties the value found when ρ 0. The previous discussion illustrates how the tie scale of data and the presence of serial autocorrelation affect fractional savings uncertainty and have direct bearing on energy savings verification. APPLICATION TO DATA In order to illustrate the above concepts, six U.S. Ary bases (AB) whose onthly utility data in the for of electricity use and gas use consued by the entire base were available for a coplete year, were selected. The data for these Ary bases, which are geographically dispersed in the continental U.S., were provided by the U.S. Ary Construction Engineering Research Laboratories in Chapaign, IL. Reddy et al. (1997b) provides specific details concerning the Ary bases, the years during which data was provided, how the data was screened, and the year chosen as the baseline. Table 4 presents the salient results of the analysis. Various linear and change-point odels were fit to the utility bill data for all six bases. The odels that yielded the best fits are shown in Table 4. Other than electricity use for AB1 which is best fit by a 4-P odel and two other bases which are best odeled by ean odels, all other odels are 3-P. Whether the odel is a cooling odel or a heating odel is also indicated in colun 2 by C and H respectively. The R 2 and CV as calculated by Equations (1) and (2) are shown in coluns 3 and 4. The nuber of data points for each Ary base that fall on either side of the change point are shown under n 1 and n 2. The corresponding ean square error (MSE) calculated fro Equation (17) as well as the associated CVs are also tabulated. Other than gas use for AB3, which sees poor, all other odels see satisfactory. As discussed earlier, there are certain odels which have siilar CVs and different R 2 values, like electricity use for AB3 which has a low R 2 (of 0.66) and a CV of 10.7%, while gas use for AB4 has a siilar CV but a R 2 of Fro the point of view of odel prediction uncertainty in energy savings estiate, both odels are as good provided F values for both cases are the sae. Differences between the CV of the two segents fitted with two individual odels (i.e., CV 1 and CV 2 ) yield additional insight into whether the errors are additive or ultiplicative. The CVs

14 16 HVAC&R RESEARCH Ary Base Model Type Table 4. Application to Utility Bill Energy Use Data fro Six Ary Bases in the United States (Adapted fro Reddy et al. 1997a) R 2 CV, % n 1 n 2 MSE 1 *, + MSE 2 *,+ of the two individual segents of the change point odels ay be widely different by as uch as a factor of 3 in soe cases (Table 4). For exaple, both the CVs for gas use at AB2 are close indicating that errors are probably additive over the entire range of variation. A wide difference between both values, such as for gas use at AB3, indicates that the odel for region (2) is uch poorer than region (1) (see Figure 2) and are due to errors being probably ultiplicative. Thus, the nature of how the errors in the data affect the odel differs fro one Ary base to another. The last colun of Table 4 presents the WCV values as calculated fro Equation (20) with, 1, and 2 replaced by n, n 1, and n 2, respectively. Note that n 12 in this case because year long utility bill data are considered. The WCV values are generally slightly lower than the CV values (colun 4), but this difference is usually sall. This suggests that the user could siply use the CV value of the entire odel if a preliinary estiate of the savings uncertainty fraction is required. If the CVs of both segents are close, then the analyst considers the errors to be additive and uses Equation (15) along with the overall CV to deterine fractional uncertainty in savings. If the CVs are widely different, the WCV values have to be coputed fro Equation (20) and this value is used in Equation (15) for the CV. IMPLICATIONS TOWARDS REQUIRED LEVEL OF MONITORING AND VERIFICATION A situation ay exist where the energy anager of a certain facility wishes to reduce the energy bill by having certain ECMs perfored by an energy service copany. It is assued that utility bills of the facility are available for at least one year but that no sub-etered data is available. The statistical expression given by Equation (15) or Equation (19) depending on whether the odel residuals are additive or ultiplicative, can be used to directly provide an indication as to whether sub-etering is necessary or not. As discussed earlier, the selection of the particular value of / depends on the energy anager and the ESCO while negotiating the energy conservation contract. It is assued CV 1 % CV 2 % WCV % Electricity AB1 4P-HC AB2 Mean AB3 3P-H AB4 3P-C AB5 3P-C AB6 Mean Gas AB1 3P-H AB2 3P-H AB3 3P-H AB4 3P-H AB5 3P-H AB6 3P-H H heating only, C cooling only, HC heating and cooling * Units for electricity are in kwh/day per 1000 ft 2+ Units for gas are in ft 3 /day per 1000 ft 2

15 VOLUME 6, NUMBER 1, JANUARY that both parties have reached a utually agreeable value and that the energy savings fraction F has also been deterined by an audit. Equation (15b) is then used to deterine the axiu CV (called CV ax ) of the baseline odel. A odel is then fit to the utility bills of the past year. If the odel CV < CV ax, then no sub-etering is necessary and the required savings verification could be done fro utility bill analysis alone. If this is not the case, then soe sort of sub-onitoring is required. Let us illustrate this with the data of the Ary bases shown in Table 4. The values / 0.15, and F 0.2 are assued for illustration. Then for one year of pre-retrofit and one year of post-retrofit, n 12. These are substituted in Equation (15b), CV ax 7.6%. Fro Table 4, for only two cases (electricity use at AB1 and AB4) are the WCV values sufficiently less than 7.6% so as not to require sub-onitoring. There are five ore odels with CV values within plus/inus two percentage points, and whether sub-onitoring is required or not is negotiable. In the reaining five cases, there is a definite need to perfor sub-onitoring if the desired confidence in the resulting savings is to be satisfied. CONCLUSIONS A previous study suggested absolute criteria for the odel goodness-of-fit to ascertain whether a baseline odel is adequate to subsequently deterine savings due to energy conservation easures (Reynolds and Fels 1988). The contention in this paper is that baseline odel developent is not an end in itself, and that explicit consideration should be given to the relative ipact of the retrofit on the baseline energy use. Under this paradig, we argue that there will be no absolute statistical cutoff criteria for odel goodness-of-fit indices. Subject to certain siplifications that are intuitively appealing, expressions for the fractional uncertainty of energy savings in ters of odel CV and the relative ipact of the retrofit on the baseline energy use for both the case of a odel with uncorrelated residuals (as encountered in utility bill analysis) as well as for correlated residuals (as often encountered with odels based on hourly or daily data) are proposed. The odification to the standard CV statistic so as to be applicable to change point odels with ultiplicative errors is presented. The effect of the two different types of odel residual errors, additive and ultiplicative, on fractional uncertainty of energy savings has also been explicitly treated. How these concepts would logically lead to insights into the selection of the ost appropriate onitoring and verification protocol to verify savings has also been discussed and illustrated with energy use utility bill data fro several Ary bases nation-wide. The approach suggested in this paper for deterining the uncertainty in the savings due to an ECM should be useful to energy service copanies and professionals engaged in the ever-expanding field of energy conservation in buildings. ACKNOWLEDGMENTS This research was perfored as part of the Texas LoanSTAR Monitoring and Analysis Progra funded by the State Energy Conservation Office of the State of Texas. Stiulating discussions with Dr. J. Hebert of Sa Houston State University are acknowledged. An insightful review by G. Reeves is also highly appreciated. The Ary Base data has been taken fro a forer project funded by the U.S. Ary Construction Engineering Research Laboratories (CERL). Finally, we would like to thank S. Deng for his help in generating soe of the figures. NOMENCLATURE E F energy use, baseline energy use ratio of the energy savings to baseline energy use nuber of post-retrofit observation points (onths, days, hours,...) n p T nuber of pre-retrofit observation points (onths, days, hours,...) nuber of odel paraeters in the regression odel outdoor dry-bulb teperature

16 18 HVAC&R RESEARCH t X Xˆ ρ CV t-statistic ean value of X odel predicted value of X auto-correlation coefficient coefficient of variation (either the CVRMSE or the CVSTD) WCV weighted CV defined by Equation (20) R 2 coefficient of deterination E uncertainty in E MSE ean square error STD standard deviation Subscripts post post-retrofit pre pre-retrofit or baseline save saving REFERENCES Beale, E Confidence Regions in Non-Linear Estiation. J.R. Stat. Assoc. B-22: Draper, N., and H. Sith Applied Regression Analysis, 2nd Ed. John Wiley & Sons, New York. Fels, M.F. (Ed.) Special Issue devoted to Measuring Energy Savings, The Princeton Scorekeeping Method (PRISM). Energy and Buildings 9(1) and 9(2). Fels, M.F., K. Kissock, M. Marean, and C. Reynolds PRISM (Advanced Version 1.0) Users Guide. Center for Energy and Environental Studies, Princeton University, Princeton, NJ. Goldberg, M A Geoetric Approach to Nondifferentiable Regression Models as Related to Methods for Assessing Residential Energy Conservation. Ph.D. Thesis, Departent of Statistics, Princeton University, Princeton, NJ. Hinkley, D.V Inferences about the Intersection on Two-Phase Regression. Bioetrica 56: IPMVP International Perforance Measureent and Verification Protocol. Departent of Energy, DOE/EE-0157, Washington DC. Katipaula, S., T.A. Reddy, and D.E. Claridge Multivariate Regression Modeling. ASME Journal of Solar Energy Engineering 120(9): 177. Kissock, J.K., T.A. Reddy, and D.E. Claridge Abient-teperature Regression Analysis for Estiating Retrofit Savings in Coercial Buildings. ASME Journal of Solar Energy Engineering 120(8): 168. Neter, J., W. Wasseran, and M.H. Kutner Applied Linear Regression Models, 2nd Ed. Irwin, Hoewood. Reddy, T.A., N.F. Saan, D.E. Claridge, J.S. Haberl, W.D. Turner, and A. Chalifoux. 1997a. Baselining Methodology for Facility-Level Monthly Energy Use Part 1: Theoretical Aspects. ASHRAE Transactions 103(2). Reddy, T.A., N.F. Saan, D.E. Claridge, J.S. Haberl, W.D. Turner, and A. Chalifoux. 1997b. Baselining Methodology for Facility-Level Monthly Energy Use Part 2: Application to Eight Ary Installations. ASHRAE Transactions 103(2). Reddy, T.A., J.K. Kissock, and D.K. Ruch Uncertainty in Baseline Regression Modeling and in Deterination of Retrofit Savings. ASME Journal of Solar Energy Engineering 120(8): 185. Reynolds, C., and M. Fels Reliability Criteria for Weather Adjustent of Energy Billing Data. Proceedings of the 1988 ACEEE Suer Study on Energy Efficiency in Buildings, pp Ruch, D.K., and D.E. Claridge A Developent and Coparison of NAC Estiates for Linear and Change-Point Energy Models for Coercial Buildings. Energy and Buildings 20: Ruch, D.K., J.K. Kissock, and T.A. Reddy Prediction Uncertainty of Linear Building Energy Use Models with Autocorrelated Residuals. ASME Journal of Solar Energy Engineering 121(2): 121. Thiel, H Principles of Econoetrics. John Wiley & Sons, New York. APPENDIX A We shall present an unsuccessful attept to redefine soe of the standard goodness-of-fit criteria so that different odels could be copared on an absolute basis. In order to be able to copare odels with different slopes and agnitude, we ay consider a siple linear transforation such as: x (x x in )/(x ax x in ) and y (y y in )/(y ax y in ) (A1)

17 VOLUME 6, NUMBER 1, JANUARY where the iniu and axiu points of a linear odel are as shown in Figure A-1a. In physical ters, this noralization iplies that the data points have been collapsed to fit into a square with the origin as one end point and (1,1) as the other (see Figure A1b). If we were to replace the y ters in Equation (1) by the above definition of y, define y y ax y in and y* y in /(y ax y in ), and assue that regression is based on the least square criterion, we obtain the following: Modified R 2 1 ( y i ŷ i ) 2 ( y i y )2 y i 1 ŷ i y* y* 2 y y y i y y* y* 2 y y 1 ( y i ŷ i ) 2 ( y i y )2 (A2) This expression is the sae as Equation (1) and so we can conclude that the above linear transforation of the data does not affect the value of R 2. Thus R 2 is independent of variable rescaling when Equation (A1) is used. If we were to do the sae with Equation (2), we find that the new or noralized CV (which we shall denote by NCV) depends on variable scaling. This can be shown as follows: y i NCV ŷ i y* y* y ( n p) y y y y* [ ( y i ŷ i )2 ] 0.5 ( y y in ) ( n p) 0.5 y CV ( y ) y in (A3) Y Yax Y 1 Yin 0 0 Xin A1(a) Xax X 0 0 A1 (b) 1 X Figure 6. Transforation of linear odel to fit into square with origin (0,0) and (1,1) as other end point

18 20 HVAC&R RESEARCH In order to reove the undue sensitivity to one data point only, viz. y in, a ore stable solution is to use the value of y in as predicted by the identified odel using x x in. Though the above transforation ay have intuitive appeal, we found certain insurountable liitations to this type of noralization while considering change point odels. Two ajor liitations have been identified with the type of transforation given by Equation (A1) when applied to change point odels. For a 3-P odel, the noralization does not ake sense for the ean portion of the odel. So the CVSTD rather than the CVRMSE should be used for the base portion. For 4-P odels with one of the segents having a very low slope, the denoinator of the noralization factor for y tends to be a very sall nuber ( y y in ), which artificially inflates the noralized CVRMSE easure. Consequently correspondence between different odel types is lost and the sae easure cannot be used to evaluate different odel types in the sae fraework. Hence we do not recoend the above transforation.