Reports by the General Accounting Office, American

Transcription

1 CLINICAL PRACTICE Forecasting Daily Patient Volumes in the Emergency Department Spencer S. Jones, MStat, Alun Thomas, PhD, R. Scott Evans, PhD, Shari J. Welch, MD, Peter J. Haug, MD, Gregory L. Snow, PhD Abstract Background: Shifts in the supply of and demand for emergency department (ED) resources make the efficient allocation of ED resources increasingly important. Forecasting is a vital activity that guides decision-making in many areas of economic, industrial, and scientific planning, but has gained little traction in the health care industry. There are few studies that explore the use of forecasting methods to predict patient volumes in the ED. Objectives: The goals of this study are to explore and evaluate the use of several statistical forecasting methods to predict daily ED patient volumes at three diverse hospital EDs and to compare the accuracy of these methods to the accuracy of a previously proposed forecasting method. Methods: Daily patient arrivals at three hospital EDs were collected for the period January 1, 2005, through March 31, The authors evaluated the use of seasonal autoregressive integrated moving average, time series regression, exponential smoothing, and artificial neural network models to forecast daily patient volumes at each facility. Forecasts were made for horizons ranging from 1 to 30 days in advance. The forecast accuracy achieved by the various forecasting methods was compared to the forecast accuracy achieved when using a benchmark forecasting method already available in the emergency medicine literature. Results: All time series methods considered in this analysis provided improved in-sample model goodness of fit. However, postsample analysis revealed that time series regression models that augment linear regression models by accounting for serial autocorrelation offered only small improvements in terms of postsample forecast accuracy, relative to multiple linear regression models, while seasonal autoregressive integrated moving average, exponential smoothing, and artificial neural network forecasting models did not provide consistently accurate forecasts of daily ED volumes. Conclusions: This study confirms the widely held belief that daily demand for ED services is characterized by seasonal and weekly patterns. The authors compared several time series forecasting methods to a benchmark multiple linear regression model. The results suggest that the existing methodology proposed in the literature, multiple linear regression based on calendar variables, is a reasonable approach to forecasting daily patient volumes in the ED. However, the authors conclude that regression-based models that incorporate calendar variables, account for site-specific special-day effects, and allow for residual autocorrelation provide a more appropriate, informative, and consistently accurate approach to forecasting daily ED patient volumes. ACADEMIC EMERGENCY MEDICINE 2008; 15: ª 2008 by the Society for Academic Emergency Medicine Keywords: forecasting, patient flow, emergency department crowding From the Department of Biomedical Informatics, University of Utah (SSJ, AT, RSE, PJH), Salt Lake City, UT; Intermountain Healthcare (RSE, PJH, GLS), Salt Lake City, UT; and the Department of Emergency Medicine, LDS Hospital (SJW), Salt Lake City, UT. Received July 9, 2007; revision received August 20, 2007; accepted September 23, Spencer S. Jones was supported through a grant from the National Library of Medicine (Training Grant 2 T15 LM ). Address for correspondence and reprints: Spencer S. Jones, MStat; spencer.jones@hsc.utah.edu. Reports by the General Accounting Office, American College of Emergency Physicians, and the Institute of Medicine (IOM) describe the United States emergency care system as being in a state of crisis. 1 3 The IOM reports that in 2003 there were 425 fewer emergency departments (EDs) operating in the United States than were operating in 1993, while over the same decade the number of ED visits increased by 26%. 3 These shifts in supply and demand make the efficient allocation of ED resources increasingly important. Key to any ED resource planning strategy is a reliable ª 2008 by the Society for Academic Emergency Medicine ISSN doi: /j x PII ISSN

2 160 Jones et al. FORECASTING DAILY PATIENT VOLUMES IN THE ED means of modeling and forecasting the demand for ED services. 4 Forecasting is a widely applicable, multidisciplinary science whose contributors include statisticians, economists, and operations researchers and is a vital activity that guides decision-making in many areas of economic, industrial, and scientific planning, but has gained little traction in the healthcare industry. 5 The IOM report, Hospital-Based Emergency Care: At the Breaking Point, recommends that emergency medicine researchers explore the use of new methods to improve patient flow. 3 Forecasting is one such method; accurate forecasts of demand can influence planning and guide the allocation of human and physical resources to facilitate patient flow. Efficient patient flow has the potential to increase the capacity of the already existing system, minimize patient care delays, and improve the overall quality of care. 3 Several studies have confirmed that demand in the ED is cyclical. Daily ED patient volumes fluctuate, based on the day of the week and the time of the year. 4,6 17 Additionally, some studies have indicated that local weather and environmental factors are correlated with the demand for emergency services These same studies have alluded to the fact that knowledge of these features of ED demand can be exploited to generate forecasts of future demand. However, we could find little research describing the accuracy or generality of forecasting methods for predicting daily ED patient volumes. Important considerations, such as how far in advance forecasts of daily ED patient volumes can be made, which forecasting methods are most appropriate for forecasting demand in the ED, and to what level of accuracy can forecasts be made have not been well documented. The goals of this study are to explore and evaluate the use of several statistical forecasting methods to predict daily ED patient volumes and to compare the accuracy of these methods to the accuracy of a previously proposed forecasting method. METHODS Study Design This was a retrospective study using aggregated data extracted from clinical information systems. The local institutional review board approved this study and waived the requirement for informed consent. Study Setting and Population This study was conducted using data collected from three hospital EDs operated by Intermountain Healthcare, a nonprofit integrated delivery network that operates hospitals and clinics in Utah and southern Idaho. The three facilities were chosen because they vary in size, in setting, and in the demographics of the communities they serve. Facility 1 is a 10-bed ED for a rural hospital located in southern Idaho that treats an average of approximately 33 patients per day (mean 32.5, standard deviation [SD] ± 6.7); Facility 2 is a 25-bed ED for an urban quaternary-care Level 1 trauma center that, on average, treats 108 patients per day (mean 108.2, SD ± 12.5); and Facility 3 is a 16-bed ED for a suburban Level 3 trauma center that treats approximately 69 patients per day (mean 68.9, SD ± 11.7). At each facility, patient arrival times were recorded using an electronic patient tracking system developed by Intermountain Healthcare, and the tracking system data were archived in Intermountain Healthcare s electronic data warehouse. Aggregated daily counts of all patients presenting for service at each of the three EDs for the period January 1, 2005, to March 31, 2007, were included in this analysis. Study Protocol Initial data analysis and model fitting were conducted using the first 2 years of data (January 1, 2005, to December 31, 2006), hereafter referred to as the training set. Daily patient counts for each facility for the period January 1, 2007, to March 31, 2007, were held out of the modeling process to empirically evaluate the postsample forecast accuracy, hereafter referred to as the validation set. Forecast accuracy was assessed at horizons ranging from 1 to 30 days in advance. Data from each facility were treated as individual time series and analyzed and evaluated separately. Measures Forecast accuracy for each horizon (1 30 days in advance) was measured by calculating the mean absolute prediction error (MAPE). The MAPE is a scaleindependent statistic that expresses prediction error as a percentage. For a series of predicted values (ŷ 1, ŷ 2,..., ŷ n ) and the corresponding series of observed values (y 1, y 2,...,y n ) MAPE ¼ 1 n X n t¼1 jðy t ^y t Þ=y t j: The advantage of using a scale-independent statistic such as the MAPE is that it facilitates the direct comparison of forecast accuracy across multiple time series. 4 Data Analysis A time series is a set of chronologically ordered observations, and time series forecasting is the practice of using past and present values of one or more time series to predict future values of the time series of interest. 4 Many methods have been developed to model and forecast time series, making the choice of an appropriate method an important task. Extensive research conducted by forecasters indicates that there is no universally superior forecasting method and that theoretical arguments can be made in favor of multiple methods given the same set of data. 4,22,23 Therefore, good forecasting practice suggests that multiple forecasting methods should be compared and evaluated based on their ability to forecast postsample observations. 23 For this analysis of daily ED patient volumes, we consider a variety of forecasting methods and then evaluate them based on their ability to forecast daily patient arrivals relative to a method that has been proposed in the literature (i.e., a linear regression model based on calendar variables similar to the model proposed by Batal et al. 6 ). Descriptions of each forecasting method and the accompanying process of model

3 ACAD EMERG MED February 2008, Vol. 15, No selection follow. Additionally, Table 1 lists the strengths and weaknesses of each forecasting approach. All analysis and statistical programming was conducted with the R statistical software package (Version 2.4.1). 24 Initial Data Analysis. Before any attempt to model or forecast a given time series, it is critical to conduct preliminary descriptive analyses of the data, giving particular attention to the identification of important features such as autocorrelation, seasonal patterns, cyclical variations nested within seasonal patterns, trend, outliers, and any other noteworthy fluctuations in the series. Initial data analysis should also evaluate whether or not the time series is stationary (i.e., whether or not basic statistical properties such as the mean and variance of the series remain constant through time). Most time series methods work under the assumption of stationarity; if the time series is deemed to be nonstationary, one or more data transformations may be necessary. 4 Initial data analysis was conducted via the visual analysis of time plots and correlograms and the computation of basic descriptive statistics. Model Selection and Estimation. Multiple. Linear Regression (Benchmark). Multiple linear regression was chosen as the benchmark forecasting method. Although linear regression is not an appropriate modeling strategy for time series data, our review of the literature indicated that a linear regression model based on calendar variables such as the weekday, month, and holidays represents the stateof-the-art in forecasting daily ED patient volumes. 6 For our benchmark model, seasonal effects were accounted for by a set of categorical variables representing the month of the year. Weekly seasonal effects were incorporated via categorical variables representing the day of the week. We included a categorical variable indicating whether or not the day being modeled was a holiday. The holiday variable was based on U.S. federal holidays as indicated by the U.S. Office of Personnel Management. 25 Along with the U.S. federal holidays, July 24, a major state holiday observed in Utah, was treated as a holiday for the two facilities located in Utah. Additionally, a near-holiday term was included representing whether or not the day being modeled fell Table 1 The Strengths and Weaknesses of the Forecasting Methods Evaluated in This Study Forecasting Method Strengths Weaknesses Linear regression (benchmark) 1. Simple familiar statistical method requires only moderate level of statistical expertise. 1. Not appropriate for autocorrelated nor nonlinear data. 2. Capable of modeling seasonal variations and trend. 2. Weights all observations (recent and remote) the same. 3. Easily interpretable results. 4. Informative modeling process. 5. Statistical software widely available. 6. Provided accurate forecasts of daily ED patient volumes at each of the three facilities. 3. Multiple variables requires additional data collection and parameter estimation. SARIMA Exponential smoothing Time series regression Artificial neural network 1. Theoretically appropriate methodology for most time series. 2. Capable of modeling seasonal variation, trend, autoregressive, and moving average processes. 3. Univariate method no external data necessary. 4. Statistical software widely available. 1. Complex statistical methodology that requires a higher level of expertise and experience than linear regression. 2. The modeling process is less informative than linear regression. 3. Generally provided less accurate forecasts of daily ED volumes than the linear and time series regression models. 1. Fully automatic, low level of expertise required. 1. Not based on formal statistical model or theory. 2. Capable of modeling seasonal variation, trend, 2. The modeling process is less informative than autoregressive, and moving average processes. that of linear regression. 3. Effective when the parameters describing the 3. Generally provided less accurate forecasts of model are changing over time. daily ED volumes than the linear and time series 4. Statistical software widely available. regression models. 1. Capable of modeling seasonal variation, trend, autoregressive, and moving average processes. 2. Easily interpretable results. 3. Informative modeling process. 4. Statistical software widely available. 5. Consistently provided more accurate forecasts of daily ED patient volumes than the linear regression models. 1. Capable of modeling complex, nonlinear systems. 2. Allows for rapid adjustment to changes in the time series. ED = emergency department; SARIMA = seasonal autoregressive integrated moving average. 1. Complex statistical methodology that requires a higher level of expertise and experience than linear regression. 2. Multiple variables requires additional data collection and parameter estimation. 1. Black box modeling procedure makes the final model difficult to interpret. 2. Statistical software packages provide fewer, less mature procedures to estimate artificial neural network models. 3. Generally provided less accurate forecasts of daily ED volumes than the linear and time series regression models.

4 162 Jones et al. FORECASTING DAILY PATIENT VOLUMES IN THE ED 1 day before or after a holiday. For holidays falling on a Monday, the 2 preceding weekend days were indicated as near-holidays. Seasonal. Autoregressive Integrated Moving Average. The second modeling strategy chosen was to fit autoregressive integrated moving average (ARIMA) models to our time series. ARIMA models provide a flexible means of modeling and forecasting a wide variety of time series. Seasonal autoregressive integrated moving average (SARIMA) models extend basic ARIMA models and allow for the incorporation of a repetitive pattern, such as the weekly pattern observed in daily ED patient volumes. In the vernacular of time series analysis, the term seasonal refers to any pattern that repeats itself with a known periodicity (e.g., 7 days in a week). When working with time series data that display seasonal patterns, it is important to not only identify the correlation between current observations and their immediate predecessors, but also to determine whether or not correlation exists between current observations and their predecessors from previous seasons. This is referred to as evaluating the time series at both the nonseasonal and seasonal levels. Box and Jenkins 26 advocate an iterative, three-stage process of SARIMA model identification, estimation, and verification. During the model identification stage, we examined correlograms displaying the autocorrelation function and the partial autocorrelation function of daily patient arrivals to identify the appropriate model structure (i.e., the order of the nonseasonal and seasonal autoregressive and moving average terms), as well as determining whether or not any degree of differencing is necessary. After the appropriate model was identified, parameter estimation was carried out using maximum likelihood. The goodness of fit of each model was verified via the Ljung Box version of the Portmanteau test. 27 For ease in reporting results, we introduce the notation of Box and Jenkins for ARIMA modeling. The structure of SARIMA model is typically represented in the following way: (p,d,q) (P,D,Q), where p and P represent the order of autocorrelation at the nonseasonal and seasonal levels respectively, d and D represent the degree of nonseasonal seasonal differencing, and q and Q represent the order of the moving average process at the nonseasonal and seasonal levels. 26 Time. Series Regression. The third modeling strategy chosen was to augment the benchmark model by including interaction terms that model the multiplicative effects of certain combinations of holiday near-holiday and the day of the week or the month of the year (e.g., when there are greater than expected increases in daily patient volumes on Tuesdays following a Monday holiday or on holidays during the month of January). Model selection process was carried out by regressing daily patient volumes on all main effects (i.e., month, weekday, holiday near holiday effects), and all possible month weekday and holiday near-holiday interactions. All variables representing month, weekday, and holiday near-holiday effects were included in the final model, while only interaction terms approximately significant (p < 0.1) were retained in the final forecasting model. In addition to the inclusion of the interaction terms, the time series regression models improve upon the benchmark models by adding a parameter that accounts for autocorrelation in the residuals. This was accomplished by fitting an ARIMA model to the residuals using the same methodology for ARIMA model identification detailed earlier. Time. Series Regression with Climatic Variables. Because the predictive value added by including climatic variables in models of daily ED patient volumes is still uncertain, 7 9,18 21 we wanted to determine whether or not the incorporation of climatic data in a forecasting model adds notable predictive value. To accomplish this, we built a fourth model that enhanced the time series regression models via the incorporation of readily available climatic variables. We considered daily minimum and maximum air temperatures (in F), daily precipitation (in inches, not including snowfall), and daily snowfall (in inches) for inclusion in the model. Local climactic data for each facility were obtained from the Utah climate center. 28 The model identification process was identical to the identification process detailed for the time series regression models, aside from the addition of the four climatic variables. Only climatic variables approximately significant (p < 0.1) were retained in the final models. For each facility the time series regression models with and without climatic variables were compared in terms of goodness of fit during the model estimation stage and in terms of predictive accuracy during the postsample evaluation stage. Exponential. Smoothing. Exponential smoothing is a term that is applied to a variety of methods that generate forecasts-based formulae that weight recent observations more heavily than more remote observations. 4 Our fifth modeling strategy was to use the exponential smoothing approach proposed by Hyndman et al. 29 This method provides a theoretically robust means of generating point forecasts based on state space models. Using this approach, it is possible to consider the full range of exponential smoothing methods, and choose the one that best fits your data. A major advantage of using this approach is that it is fully automatic, which reduces the effort and expertise necessary to identify an appropriate model. 29 Artificial. Neural Networks. Artificial neural networks are designed to mimic the architecture of human brain and can be used to model complex nonlinear relationships between inputs and outputs. Artificial neural networks have been demonstrated to be highly effective in applications such as pattern recognition and classification. 30,31 Since forecasting is essentially the process of identifying patterns from observed data and extrapolating them into the future, one would expect artificial neural networks to perform well at forecasting tasks. Empirical studies have demonstrated that in some cases artificial neural networks provide accurate time series forecasts; however, other studies have shown that much simpler time series methods can be employed to obtain as good or better forecast accuracy. 4,31 34 Despite their mixed results, artificial neural networks are an appealing

5 ACAD EMERG MED February 2008, Vol. 15, No approach, have been applied to a wide variety of forecasting tasks, and represent a state-of-the-art approach to time series forecasting. 34 An artificial neural network is formed by a network of computing units, called neurons, which are connected to each other, forming a network. The strength of each connection, or weight, is updated iteratively as the network is trained, so in effect it learns to recognize patterns as it is provided with data. The neurons are organized in layers; the first layer consists of neurons that represent the inputs to the network. The second, or hidden, layer consists of an analyst-defined number of neurons that apply a nonlinear function to the inputs. The neurons in the hidden layer act as intermediaries between the input layer and the final output layer that consists of the predictions made by the artificial neural network. For our sixth and final modeling strategy, we chose to employ a single hidden layer feed-forward model, which is the most common artificial neural network approach to time series forecasting. 22 The input layer consisted of variables representing month, weekday, holiday, and near-holiday effects along with the month weekday, holiday near-holiday interactions, and once-lagged patient volumes. The connection weights were estimated via the back-propagation algorithm with a weight-updating rate of The number of neurons to include in the hidden layer was determined via an iterative process that sought to minimize the bias-corrected version of the Akaike s information criteria (AIC c ). 4,35,36 Model Evaluation We developed six forecasting models for each facility; goodness-of-model fit was initially assessed via comparison of the AIC c. 4,35,36 The AIC c is a model selection criterion that allows for the direct comparison of different classes of models with differing parameters. In the case of this analysis, comparison via the AIC c allows for the fair comparison of multivariable models (e.g., multiple regression, time series regression, and artificial neural networks) with many parameters and univariate models (e.g., SARIMA and exponential smoothing) with relatively few parameters. The primary outcome of interest was the postsample forecast accuracy relative to the accuracy of forecasts made by a simple regression model. The benchmark forecasts were based solely on the training set (i.e., after fitting the linear regression model described earlier to the training data, forecasts were generated for all 90 days in the validation set). For the time series forecasting methods considered, we simulated true postsample forecasts by incrementally expanding the training set and then generating forecasts for the next 30 days (e.g., after forecasts for the period January 1, 2007, to January 30, 2007, were generated based on the model estimated in the training set, the observed values for January 1, 2007, were added to the training set). The models were then reestimated using the expanded training set, and forecasts were generated for the next period (in the example case, January 2, 2007, to January 31, 2007). After 1- through 30-day ahead forecasts were generated for each day in the validation set, the MAPE was calculated for each forecasting method at each forecast horizon. The process described above allowed us to generate true postsample forecasts using each forecasting method with the exception of the time series regression models that incorporated climatic variables. We were unable to obtain forecasted values of the various climatic variables that we considered for inclusion in these time series regression models; therefore, we chose to use the observed values. Because perfect weather forecasts are not achievable, using observed values rather than forecasted values establishes the maximum benefit in terms of forecast accuracy that is obtainable by including basic climatic variables in a multiple regression forecasting model. RESULTS As expected, initial data analysis revealed that daily ED volumes for each facility were characterized by annual and weekly seasonality. The time plot (Figure 1) showed annual seasonal fluctuations in ED volumes and no evidence of a long-term increasing or decreasing trend. Figure 2 shows that daily patient volumes are typically elevated on weekends and Mondays. Examination of the correlograms indicated that daily ED volumes showed signs of autocorrelation at both the seasonal and the nonseasonal levels at each facility. Analysis of the time plots in conjunction with the correlograms revealed that the mean, variance, and autocorrelation structure of each time series remained consistent, indicating that each series was stationary and that aside from first-order differencing for the SARIMA models, additional data transformations were unnecessary. In-Sample Model Goodness of Fit SARIMA. Ljung Box statistics along with autocorrelation function plots and partial autocorrelation function plot for model residuals indicated that SARIMA models of order (1,0,0) (0,1,1), (0,1,1) (0,1,1), and (0,1,1) (0,1,1) provided adequate model fit for daily ED patient volumes at Facilities 1, 2, and 3, respectively. Initial goodness-of-fit statistics (Table 2) indicated that the SARIMA models provided improved in-sample goodness of fit relative to the benchmark regression models. Time Series Regression. Based on the calculated values of AIC c, the multiple regression models that included interaction terms and accounted for autocorrelated errors provided the best initial model fit at all three facilities. In addition to comparisons based on the AIC c, we calculated multiple R 2 statistics for the time series regression models. Table 3 presents R 2 statistics along with the individual parameter estimates for the variables retained in the time series regression models without climatic variables and the models that included climatic variables. The results presented in Tables 2 and 3 indicate that the inclusion of climatic variables provided only slight gains in terms of R 2 (1% 2% of the explained variance), and the lower AIC c values for the models that did not include climatic variables suggest that the small gains in explained variance do not justify

6 164 Jones et al. FORECASTING DAILY PATIENT VOLUMES IN THE ED Figure 1. Time plot for daily ED patient volumes for the period January 1, 2005, to December 31, 2006, at Facilities 1 3. ED = emergency department. Table 2 Measures of In-sample Goodness of Fit (Bias-Corrected AIC c ) for Each Forecasting Method at Facilities 1 3 Forecasting Method Facility 1 Facility 2 Facility 3 Benchmark* SARIMA Time series regression Time series regression with climatic variables Exponential smoothing Artificial neural network Lower values indicate better model fit. Time series regression models provided the best initial model fit at all three facilities. SARIMA = seasonal autoregressive integrated moving average. * Simple linear regression model. the inclusion of the additional model parameters required to model the effects of climatic variables. Table 3 lists each main effect and interaction term retained in the multiple regression models. This table indicates that annual and weekly seasonality and special day effects exist at each facility. The annual seasonal pattern appears to be fairly consistent across the three facilities; however, the weekly pattern and the impact of holidays vary. Saturday appears to be the highest volume day at a Facilities 1 and 3, while Monday is the busiest day at Facility 2. Holidays seem to have little impact on volume at Facilities 1 and 2 (an average increase of less than 1 patient at Facility 1 and an average increase of approximately 4 patients at Facility 2), but have a major impact on volumes at Facility 3 (an average increase of approximately 13 patients). There appears to be a positive correlation between maximum daily temperatures and daily ED patient volumes and an inverse correlation between daily ED patient volumes and the maximum daily temperature on the prior day. The regression coefficients estimated for the lagged daily ED patient volumes indicate that a fairly consistent ( ) first-order positive residual autocorrelation exists at each facility. The various interaction terms reported in Table 3 reveal some important phenomena that might not otherwise be expected; for instance, at Facility 2, where Monday is consistently the busiest day, we identified a statistically significant multiplicative interaction on Tuesdays after a Monday holiday. We found that daily ED patient volumes typically increase by approximately 11 patients, suggesting that in these instances the first-day-of-the-workweek effect trumps the weekly cycle and that it would be wise to staff these Tuesdays like the typical Monday. Other interesting facility-specific interactions were identified during the development of the time series regression models, such as the 13-patient increase that can be expected on holidays in September, i.e., Labor Day, at Facility 1, and the nearly 30-patient increase that can be expected on a Friday after a holiday, e.g., the day after Thanksgiving, at Facility 3. Exponential Smoothing. Using the fully automatic exponential smoothing approach proposed by Hyndman et al., 29 it was determined that exponential smoothing models with multiplicative errors, no trend, and additive seasonality were most appropriate for Facilities 1, 2, and 3. The exponential smoothing models outperformed the benchmark regression model in terms of in-sample goodness of fit for each of the three facilities considered. Artificial Neural Networks. Single hidden-layer feedforward artificial neural network models with two

7 ACAD EMERG MED February 2008, Vol. 15, No Table 3 Parameter Estimates for Time Series Regression Models with and without Climatic Variables Facility 1 Facility 2 Facility 3 TSR 0 TSR 1 TSR 0 TSR 1 TSR 0 TSR 1 Parameter Estimate Estimate Estimate Estimate Estimate Estimate Intercept Holiday Near-holiday )0.24 ) Max daily temperature NA 0.09 NA 0.24 NA 0.06 January Ref Ref Ref Ref Ref Ref February )0.05 )0.04 March )0.79 )0.89 April )0.19 ) )1.17 )1.46 May 2.42 ) ) June 2.49 ) July 4.35 ) August 2.81 ) September 0.91 ) )2.57 )1.07 )1.52 October 0.39 )1.97 )3.29 )5.09 )5.39 )5.66 November 0.48 )0.55 )3.15 )3.84 )4.1 )4.24 December Sunday Ref Ref Ref Ref Ref Ref Monday )4.28 )4.26 Tuesday )3.54 )3.55 )1.61 )1.66 )7.33 )7.34 Wednesday )2.03 )2.01 )0.23 )0.42 )7.1 )7.15 Thursday )1.05 ) )4.57 )4.58 Friday )6.97 )6.97 Saturday AR(1) Lagged max daily temperature(1) NA NA NA )0.14 NA )0.05 Interaction: July Holiday NA NA )21.45 )20.44 NA NA September Holiday NA NA NA NA October Holiday NA NA NA NA )15.14 )15.42 December Holiday NA NA NA NA Tuesday Holiday NA NA NA NA July Near-holiday NA NA NA NA September Near-holiday NA NA NA NA November Near-holiday NA NA NA NA December Near-holiday NA NA NA NA Tuesday Near-holiday NA NA NA NA Friday Near-holiday NA NA NA NA Multiple R TSR 0 = time series regression model without climatic variables; TSR 1 = time series regression model with climatic variables; AR(1) = first-order autoregressive term; NA = variable or interaction term was not retained in the model for the given facility; Ref = reference. neurons in the hidden layer were trained for each facility using the test set. The artificial neural network models provided better in-sample goodness of fit than the benchmark regression model at each of the three facilities. Inputs to the artificial neural networks were identical to the variables included in the time series regression models without climatic variables; however, the time series regression models provided better insample goodness of fit. Postsample Forecast Accuracy The primary outcome of interest for this study was the performance of the various forecasting methods in terms of postsample forecast accuracy relative to an available benchmark. Table 4 presents the MAPE for selected forecast horizons; we present only selected forecast horizons primarily because the MAPEs remain fairly constant for each forecasting method over the range of forecast horizons (1 30). The SARIMA model provided the best postsample forecast accuracy and offered improved forecast accuracy relative to the benchmark model for forecast horizons up to 30 days in advance at Facility 1, improved forecast accuracy relative to the benchmark model for forecast horizons less than 7 days in advance at Facility 2, and performed worse than the benchmark model for all forecast horizons at Facility 3. The time series regression model without climatic variables provided improved or comparable forecast accuracy for horizons up to 14 days at Facility 1, improved forecast accuracy for all forecast horizons up to 30 days at Facility 2, and improved forecast accuracy for forecast horizons up to 7 days in advance at Facility 3. The addition of climatic variables to the time series regression models led to small improvements in postsample forecast accuracy at each facility. The exponential smoothing and artificial neural network models performed inconsistently across the three facilities. The exponential smoothing models only

8 166 Jones et al. FORECASTING DAILY PATIENT VOLUMES IN THE ED Table 4 Measures of Forecasting Error of Daily Patient Arrivals Made 1, 7, 14, 21, and 30 Days in Advance for Facilities 1 3 Facility % Benchmark* MAPE at Forecast Horizon (%) Forecasting method 1 day 7 days 14 days 21 days 30 days SARIMA Time series regression Time series regression with climatic variables Exponential smoothing Artificial neural network Facility % Benchmark* Forecast Horizon (%) Forecasting method 1 day 7 days 14 days 21 days 30 days SARIMA Time series regression Time series regression with climatic variables Exponential smoothing Artificial neural network Facility % Benchmark* Forecast Horizon (%) Forecasting method 1 day 7 days 14 days 21 days 30 days SARIMA Time series regression Time series regression with climatic variables Exponential smoothing Artificial neural network MAPE = mean absolute prediction error; SARIMA = seasonal autoregressive integrated moving average. * Multiple linear regression model. provided improved forecast accuracy relative to the benchmark method for short forecast horizons at Facility 2, and the artificial neural network models provided improved forecast accuracy relative to the benchmark method at Facility 3, but performed poorly at the other two facilities. The most notable feature of the measures of MAPE presented in Table 4 is that even in the best cases, the various forecasting methods provided only slightly better (typically < 0.5%) forecast accuracy relative to the accuracy of a multiple linear regression model based solely on calendar variables. Figure 3 presents observed patient volumes for the validation period, along with 1-day-ahead forecasts generated using the time series regression model with climatic variables (the forecasting method with the lowest MAPE) and the forecasts made using the benchmark model. Figure 3 shows that the forecasts generated using a more complicated model that incorporates climatic variables along with a first-order autoregressive term are rarely very different from the forecasts using the linear regression model. This result indicates that demand for ED services is driven primarily by the annual and weekly cycles, and the incorporation of climatic variables and autoregressive terms adds only minimal predictive value. DISCUSSION We found several studies that utilize time series techniques to analyze daily demand for ED services, but to this point the majority of these studies have been descriptive in nature. 8,9,13 21 We located only two studies that report any measure of postsample forecast accuracy. The most recent study proposed a multiple linear regression model based on calendar variables (the inspiration for the benchmark forecasting method considered in this study). The investigators report that their model predicted patient volumes in the validation set within ± 11%. Using this model to optimize staffing patterns, they report that the number of patient complaints dropped by 30% and the number of patients who left without being seen decreased by 18.5%. 6 The most accurate time series forecasting methods considered in this analysis provided slightly more accurate forecasts at Facility 2 (8.91% 9.04%) and at Facility 3 (8.54% 9.06%) and less accurate forecasts at Facility 1 (13.89% 14.35%). The discrepancy in the MAPE values for Facilities 2 and 3 versus the MAPE values for Facility 1 is a mathematical artifact of small errors being magnified by the relatively low patient volumes at Facility 1. The literature on the impact of climatic variables on the demand for emergency services is mixed. Descrip-

9 ACAD EMERG MED February 2008, Vol. 15, No Figure 2. Box plots showing the distribution of daily ED patient volumes by day of the week and month of the year for Facilities 1 3. ED = emergency department. tive analyses by Alberdi et al., 18 Jones et al., 19 MacGregor, 20 and Rising et al. 21 indicate that climatic variables such as air temperature are positively correlated with the demand for ED services. On the other hand, similar studies by Batal et al., 6 Diehl et al., 8 and Zibners et al. 9 conclude that while statistically significant correlations between climatic variables and ED patient volumes exist, climatic variables add little predictive value to models of daily ED patient volumes. However, it is not possible to confirm this conclusion because studies that evaluate the impact climatic variables in terms of postsample forecast accuracy do not exist. Our analysis compares the postsample forecast accuracy of time series regression models that include climatic variables to similar models that do not include climatic variables. We found that maximum air temperature was the only

10 168 Jones et al. FORECASTING DAILY PATIENT VOLUMES IN THE ED Figure 3. One-day-ahead postsample forecasts of daily ED patient volumes at Facility 2 using a time series regression model that incorporates calendar and climatic variables for the period January 1, 2007, to March 31, 2007, compared to observed values and forecasts generated using the benchmark linear regression forecasting model. ED = emergency department. climatic variable considered that was significantly correlated with daily ED patient volumes. The inclusion of this variable in our forecasting model resulted in small reductions in terms of percentage error, but these reductions translated into unimportant improvements in terms of predicting daily ED patient volumes. Ultimately, the results of our postsample analysis support the conclusion that basic climatic variables, such as air temperature, add little predictive value to regressionbased forecasting models. The studies by Batal et al. 6 and Holleman et al. 7 were limited in that both studies evaluated the predictive accuracy of a single forecasting model (a linear regression model based on calendar variables in both cases) at a single facility. Our study fills a gap in the literature by considering several forecasting methods at multiple, diverse hospital EDs. The methods considered in this analysis range from simple (linear regression) to state-of-the-art (artificial neural networks). While our study found that a relatively simple regression model based on calendar variables is a reasonable approach to forecasting daily patient volumes, we also found that modest improvements in terms of forecast accuracy can be obtained when serial autocorrelation is accounted for. Ultimately, we felt that the time series regression class of models offered the most consistent performance in terms of postsample forecast accuracy. By accounting for the autocorrelated nature of time series data, the time series regression models offer a more technically correct approach to modeling and forecasting daily ED volumes. Additionally, we found the model development process for the time series regression models to be relatively simple and highly informative, in particular, the consideration of interaction terms allowed for the identification of site-specific patterns that were not readily apparent. LIMITATIONS We only considered data from three facilities in the same region of the United States, all of which seemed to demonstrate relatively stable patterns in daily ED patient volumes. For facilities that are located in areas where population growth is rapid, or where other hospital EDs are opening or closing, it may be necessary to incorporate additional parameters into the forecasting models that account for trends or resort to judgmental forecasting methods that project the implications of major changes to the underlying processes that drive demand for services at the facility. We considered only a few of the forecasting methods that are available. We selected the various forecasting methods considered in this analysis based on our review of the literature and our experience in other applications of time series analysis. Other methods, such as Fourier analysis and transfer function models that are well suited for time series data, could potentially be useful for forecasting daily ED patient volumes. Additionally, we consider only point forecast of daily ED census; another approach would be to consider the generation of interval forecasts that provide a range in which the daily patient volume is most likely to fall. Finally, while one study showed positive impacts on patient satisfaction as the result of using a forecasting model to optimize staffing patterns, 6 important factors other than patient census (e.g., patient acuity, minimum staffing regulations, and clinician satisfaction) must be

11 ACAD EMERG MED February 2008, Vol. 15, No considered when staffing an ED. This study does not address these issues. CONCLUSIONS Our study confirms the widely held belief that daily demand for ED services is characterized by seasonal and weekly patterns. We compared several time series forecasting methods to a benchmark multiple linear regression model. In-sample goodness-of-fit statistics suggested that each of the time series forecasting methods considered provided better model fit than the benchmark method; however, when we compared the postsample forecast accuracy of the various methods, we found that even in the best case (time series regression), these more sophisticated methods provided only small gains in forecast accuracy over the benchmark method. These results suggest that the existing methodology proposed in the literature, multiple linear regression based on calendar variables, is a reasonable approach to forecasting daily patient volumes in the ED. However, due to the fact that time series data often violate a key inference assumption associated with linear regression, and that the time series regression class of models provided slightly more accurate forecasts of daily ED patient volumes, we conclude that of the methods considered in this analysis, the regression-based models that incorporate calendar variables account for site-specific special-day effects, allow for residual autocorrelation, and provide the most informative and consistently accurate forecasts of daily ED patient volumes. References 1. General Accounting Office. Hospital Emergency Departments: Crowded Conditions Vary among Hospitals and Communities. Washington, DC: General Accounting Office, American College of Emergency Physicians. The National Report Card on the State of Emergency Medicine. January Available at: acep.org/site/pageserver?pagename=wp1_research_ overview. Accessed Aug 22, Institute of Medicine Committee on the Future of Emergency Care in the U.S. Health System. Hospital-Based Emergency Care: At the Breaking Point. Washington, DC: National Academies Press, Asplin BR, Flottemesch TJ, Gordon BR. Developing models for patient flow and daily surge capacity research. Acad Emerg Med. 2006; 13: Chatfield C. Time series Forecasting. Boca Raton, FL: Chapman & Hall CRC, Batal H, Tench J, McMillian S, Adams J, Mehler PS. Predicting patient visits to an urgent care clinic using calendar variables. Acad Emerg Med. 2001; 8: Holleman DR, Bowling RL, Gathy C. Predicting daily visits to a walk-in clinic and emergency department using calendar and weather data. J Gen Intern Med. 1996; 11: Diehl AK, Morris MD, Mannis SA. Use of calendar and weather data to predict walk-in attendance. South Med J. 1981; 74: Zibners LM, Bonsu BK, Hayes JR, Cohen DM. Local weather effects on emergency department visits: a time series and regression analysis. Pediatr Emerg Care. 2006; 22(2): Green LV, Kolesar PJ, Svoronos A. Improving the SIPP approach for staffing service systems that have cyclic demands. Operations Res. 2001; 49(4): Green LV, Soares J, Giglio JF, Green RA. Using queueing theory to increase the effectiveness of emergency department staffing. Acad Emerg Med. 2006; 13: Tandberg D, Qualls C. Time series forecasts of emergency department patient volume, length of stay, and acuity. Ann Emerg Med. 1994; 23: Rotstein Z, Wilif-Miron R, Lavi B, Shahar A, Gabbay U, Noy S. The dynamics of patient visits to a public hospital ED: a statistical model. Am J Emerg Med. 1997; 15: Brillman JC, Burr T, Forslund D, Joyce E, Picard R, Umland E. Modeling emergency department visit patterns for infectious disease complaints: results and application to disease surveillance. BMC Med Inform Decis Mak. 2005; 5: Sinreich D, Marmor Y. Emergency department operations: the basis for developing a simulation tool. IIE Transact. 2005; 37: Jiang H, Lam L, Owens B, Sier D, Westcott M. Predicting emergency department status. Proceedings of the Sixth International Conference on Optimization: Techniques and Applications. Ballarat, Australia, December 9 11, Morzuch BJ, Allen PG. Forecasting hospital emergency department arrivals. Presented at the 26th Annual Symposium on Forecasting, Santander, Spain, June 11 14, Available at: umass.edu/resec/faculty/morzuch/index.html. Accessed Sep 8, Alberdi JC, Diaz J, Lage MB, et al. Model for forecasting emergency hospital admissions: effect of environmental variables. J Environ Health. 2001; 64: Jones SA, Joy MP, Pearson J. Forecasting demand of emergency care. Health Care Manag Sci. 2002; 5: MacGregor DM. Effect of weather on attendance with injury at a paediatric emergency department. Emerg Med J. 2003; 20: Rising WR, O Daniel JA, Roberts CS. Correlating weather and trauma admissions at a level I trauma center. J Trauma. 2006; 60(5): De Gooijer J, Hyndman RJ. 25 years of time series forecasting. Int J Forecast. 2006; 22: Armstrong JS, Fildes R. Making progress in forecasting. Int J Forecast. 2006; 22: R Development Core Team (2006). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available at: Accessed Sep 28, U.S. Office of Personnel Management. Federal Holidays. Available at: Accessed May 10, 2007.