Using Ensemble Weather Forecasts for Predicting Airport Runway Configuration and Capacity

Transcription

1 AIAA Aviation June 2016, Washington, D.C. 16th AIAA Aviation Technology, Integration, and Operations Conference Approved for Public Release; Distribution Unlimited The MITRE Corporation. ALL RIGHTS RESERVED. AIAA Using Ensemble Weather Forecasts for Predicting Airport Runway Configuration and Capacity Shin-Lai (Alex) Tien 1, Christine Taylor 2 and Craig Wanke 3 The MITRE Corporation, McLean, VA, Strategic air traffic flow management requires predictions of airport departure and arrival capacities several hours into the future. It is difficult to do this well because of weather forecast uncertainty. However, if the measurement of prediction uncertainty is provided, strategies can be developed to account for it. In this paper, we examine the potential for using ensemble weather products to quantify uncertainty in airport capacity prediction. An improved prediction model from previous studies that utilizes weather forecast variables to predict runway configuration and capacity is validated and then applied to each ensemble member, so that the distribution of possible capacity outcomes and prediction confidence can be obtained. Our results suggest that the performance of the prediction model has been improved, and that the forecast variation among ensemble members effectively captures the possible variations in future arrival capacity for certain airports. I. Introduction trategic air traffic flow management (TFM) aims to address predictions of significant demand-capacity Simbalances six or more hours in the future. However, the forecast uncertainty at long look-ahead times (LATs) makes it challenging for TFM decision makers to mentally translate weather forecasts into capacity impact. To address this issue, several methods have been proposed to quantitatively and automatically estimate capacity loss for airspace and airport resources using numerical weather forecast variables. 1-7 To account for forecast uncertainty, [4] and [8] leveraged an ensemble forecast product to capture the wide range of capacity outcomes persistent in the planning horizon. Using a prototype simulation and evaluation capability, these works further highlighted the resulting capacity variation in developing traffic management strategies that arise as a result of the inherent differences among ensemble members. In this paper, we examine the utility of ensemble forecast products in depicting the prediction uncertainty of airport capacity. Specifically, when applied to an airport capacity prediction model, the individual ensemble members together generate a range of airport capacity profiles, from which not only capacity values but also uncertainty measures can be derived. Through understanding prediction uncertainty, TFM decision makers can make informed decisions about formulating traffic management initiatives, such as determining the start time, duration, and target arrival rates for a Ground Delay Program. In the rest of the paper, we review existing work in airport capacity prediction. We then improve the airport prediction model in [2] and use empirical data to validate several enhancements made to the model in Section III. In Section IV, we apply an ensemble forecast product to the airport capacity prediction model and demonstrate the viability of using ensemble forecast products in predicting airport capacity for strategic TFM. The prediction performance of using the ensemble forecast is analyzed from various quantitative measurements in Section V. II. Airport Capacity Prediction Predicting airport capacity at longer LATs is a challenging task. Weather forecasts of wind, ceiling, visibility, and convective activity are main factors in determining runway configuration, which defines the airport departure rate (ADR) and arrival rate (AAR). In recent literature, a number of airport-capacity models for strategic TFM have been proposed. Buxi and Hansen 9 generated probabilistic capacity profiles for a full day, by correlating current Terminal 1 Lead Operations Research Analyst, 7515 Colshire Drive, M/S N450, AIAA Member. 2 Principal Modeling/Simulation Engineer, 7515 Colshire Drive, M/S N450, AIAA Member. 3 Senior Principal Aviation Systems Engineer, 7515 Colshire Drive, M/S N450, AIAA Associate Fellow. Copyright 2016 by The MITRE Corporation. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

2 Aerodrome Forecasts (TAF) with historical forecasts. Provan et al. 10 derived probability distributions for airport capacity given weather forecasts and the airport state (runway configurations, demand, operational standards, procedures, etc.). Kicinger et al. 11 explored the usage of time-lagged ensemble forecasts on airport capacity prediction. One approach described in the literature develops models that leverage data-mining and statistical estimation approaches for predicting runway configuration or capacity. 3,12,13 Alternatively, DeLaura et al. 14 developed an Airport Arrival Rate (AAR) prediction model that captures in detail the facility-specific runway selection rules and spacing issues that drive AAR. Although all the existing models study sample airports to support concept development and validation, requiring detailed site-specific calibration makes these models less readily applicable to other National Airspace System (NAS) airports. Dhal et al. 2 proposed a model to predict runway configuration and capacity that aims to support the strategic TFM decision-making process. The model determines the runway configuration and associated AAR and ADR based on weather forecast variables and local preferences observed in the historical data regarding configuration selection under different weather conditions. Tien et al. 1 further validated the model by applying it systematically to the 35 Federal Aviation Administration (FAA) Operational Evolution Partnership (OEP) airports to demonstrate its applicability in strategic TFM. Several model improvement areas were also identified. This paper builds upon the work of Dhal et al. 2 and Tien et al. 1 The modeling steps in Dhal et al. 2 are illustrated in Figure 1(a), and can be summarized as follows: The first step, Identify Configuration Preferences, categorizes the historically used configurations by meteorological condition (MC), runway configurations, usage frequency, ADR, and AAR. The output is a summary table like that shown in Figure 1(b). This configuration preference table can be derived from historical data and supplemented by inputs from local facilities, if available. By relying primarily on historical data, the model can be systematically applied to key NAS airports. Although not considered in this paper, further categorization could be based on local hours, arrival/departure banks, or seasons. The second step determines the eligible configurations based on the forecast variables for the hour (ceiling, visibility, wind speed, and wind direction). Ceiling and visibility are used to determine the MC category, while wind speed and direction are used to determine the eligible configurations that meet the tailwind and crosswind requirements. The set of eligible configurations are sorted based on their frequency of use. In the third step, the model selects a configuration from the set of eligible ones identified. The selection considers several attributes, including wind preference, capacity, and the previous hour s predicted configuration. More details of the selection process can be found in Dhal et al. 2 (a) Illustration of Algorithmic Steps in Airport Capacity Prediction Model (b) Sample of the Observed Configuration Preferences Figure 1. Runway Configuration and Capacity Prediction Model 2

3 III.Model Improvement and Validation The FAA s Aviation System Performance Metrics (ASPM) database contains airport operational data (runway configuration, AAR, ADR) and recorded weather variables (meteorological condition, ceiling, visibility, wind speed, wind direction) at 75 key U.S. airports in quarter-hour updates. Using this data, Dhal et al. s model was validated for the 35 OEP airports. 2 Although the initial results were promising, two crucial improvement areas were identified: Refine ASPM s categorization of meteorological conditions Include more recently observed runway configurations in the configuration preference table ASPM records only two categories MC, visual (VMC) and instrumental (IMC) conditions. By defining additional MC categories, additional configuration preference tables can be generated. A finer MC categorization enables the model to capture local operational trends that arise in response to conditions near the boundary of VMC and IMC. As such, instead of directly using the data field of MC from ASPM, this paper introduces a marginal MC category and recategorizes MC by using ASPM s ceiling and visibility data. The thresholds of ceiling and visibility in Figure 2 are used for defining visual MC at the 35 OEP airports. When either ceiling is under 1,000 feet or visibility is under 3 statute miles, MC is categorized as IMC. Any other combination of ceiling and visibility will be considered as marginal MC. Tien et al. 1 also showed the model s poor performance at a few airports due to using the configuration preferences observed in 2013 summer to predict configurations in July Consequently, this paper uses more recent observations. Specifically, for validating the model on a particular day, the runway configuration table (shown in Figure 1b) is constructed from ASPM runway data over the past 90 days, resulting in 2,160 hourly observations. To analyze the impact of the two improvements on model accuracy, we examine performance using data for July 2014 again. We note that with the exception of these two improvements, the rest of the model settings are the same as in Tien et al. 1 Predictions are made using the configuration preference table derived from ASPM and recorded weather variables in ASPM. In order to compare with the results from Tien et al. 1, the mean absolute errors (MAE) is used to measure the model performance at each airport, which is defined as: (, )= where x is the vector of modeled AAR, and y is the vector of reported AARs. Figure 3 compares the MAEs for the 35 OEP airports using the original and improved approaches, where model performance is significantly better (lower MAE) for several airports that suffered from high MAE before the improvement. Improvements at these airports, DFW, MCO, LAX, LGA, CLT, CVG, and BOS, are primarily due to considering more recent runway configuration data when constructing the preference tables. For example: DFW, MCO, and LAX had ongoing construction in July 2014 limiting runway use, thus leading to an over-prediction of the AAR in Tien et al. 1 For CVG, a runway configuration not observed in the 2013 data was frequently used in July 2014, causing prediction errors. Figure 3 also shows several airports with poor performance under the new model (more than 2 aircraft/hour), including DEN, MEM, ATL, CLE, and PHX. Note that the runway preference table is directly derived from ASPM data-mining, which is applied systematically to each of the OEP airports. Local factors or common practice that are not revealed in the data source may include, but not limited to: Taxiway/runway outage that leads to a reduced AAR, Runway configuration dependent on neighboring airports, Airport s low demand periods that a low AAR is called. Calibrating a site-specific model is beyond the scope of this paper. For the model to be operationally ready, local inputs and adaptation shall be included to derive configuration preferences for improving the prediction performance. 3

4 Figure 2. VMC Threshold Values of FAA s ASPM Database Figure 3. Comparison of Model Validation Results (July 2014 Data) IV.Application to Ensemble Forecast An Illustrative Example After confirming the improvement, we apply the AAR prediction model to ensemble weather forecast. Short Range Ensemble Forecast (SREF) is a gridded, numerical weather forecast product composed of 21 ensemble members, each of which provides a deterministic forecast on a 40km-by-40km grid system over North America, 15,16 as of September Previous studies have treated each ensemble member as one trajectory of weather development through the NAS, assuming equal likelihoods of occurrence. 4,8 The SREF forecast variables from the grids where the OEP airports reside, including ceiling, visibility, wind, precipitation, and reflectivity, are used to predict airport AAR. Figures 4 and 5 illustrate the forecast differences among the 21 ensemble members. Figure 4 is the forecast of hourly wind speed generated on July 2, 2014, at 15:00Z at Chicago O Hare International Airport (ORD) for a 12-hour LAT. For each hour, the colored symbols represent the forecast values of wind speed. The yellow-shaded and red-shaded areas outline the uncertainty regions, which are defined by the mean of 21 forecast values within one and two standard deviations, respectively. Despite the individual differences and outliers, the forecast suggests a decreasing wind speed. The solid blue line shows the recorded wind speed during that timeframe, confirming the trend suggested by the aggregate behavior of the ensemble members. Figure 5 shows a similar result for wind direction. Applying the capacity prediction model to each of the ensemble members renders 21 profiles of airport capacity. Figure 6 shows the predicted AAR for ORD using the SREF forecast generated on July 2, 2014 at 15:00Z, where predictions are made up to 12 hours ahead. The actual AARs from ASPM (the solid blue line) are then compared with the predicted AARs by ensemble members (colored symbols). While individual ensemble members do not provide good estimate of airport capacity, their mean indicates a decrease in AARs after 2100Z. The yellow-shaded area, 4

5 which is bounded by the mean plus and minus one standard deviation, encloses the actual AARs between 1900Z and 2300Z. This example illustrates several potential usages of the ensemble-based predictions: Weather forecast ensembles may be translated into capacity forecast ensembles of a TFM resource (an airport in this case). Ensemble members together depict a range of possible capacity outcomes, which capture the trend of weather impact to airport capacity at a strategic decision-making timeframe, typically 6 to 12 hours ahead. The uncertainty of capacity prediction can be quantitatively described by common statistical measures, e.g., standard deviation, variance, etc. The uncertainty band formed by the mean and standard deviation of the SREF-predicted AARs enables TFM decision makers to visualize the predictive trend and uncertainty, although the perception of uncertainty measurement is still subjective and needs further study. Wind Speed (Nautical Mile per Hour) Figure 4. Wind Speed Ensemble Forecast vs. ASPM 5

6 Wind Direction (Degree) Airport Arrival Rate (Aircraft per Hour) Figure 5. Wind Direction Ensemble Forecast vs. ASPM Figure 6. Airport Arrival Rates Predictions of Ensemble Forecast vs. ASPM V.Application to Ensemble Forecast Prediction Performance This section examines the prediction performance of the proposed model using SREF. Metrics and comparison methods are designed to measure how SREF-derived AAR prediction captures the reported AAR of 35 OEP airports. Given data availability, the hourly data of ASPM and SREF achieved for 2015 summer months (June, July, August) are used for validation in this section. 6

7 SREF provides forecast up to 3 days into the future, and it is generated 4 times a day at 3Z, 9Z, 15Z, and 21Z. This study only uses forecast data from local time from 6:00 to 23:00 and data from look-ahead time less than 12 hours. The main questions to be answered by this validation effort are: 1) How well does the proposed model effectively capture reported AAR? 2) How well does SREF-based prediction match the reported AAR? 3) What are the source of prediction errors? A. Use of Mean Values To do a pair-wise comparison to the reported AAR, the mean of predicted AARs from 21 ensemble members is used as a single capacity estimate in this subsection. We conduct three types of pair-wise comparison: Nominal vs. reported AAR A naïve approach for AAR prediction is to use the nominal capacity derived from the most frequently used configuration under VMC of an airport. Comparing the nominal and ASPM reported AARs sets up the baseline for demonstrating any improvement to be made by any prediction model. Modeled vs. reported AAR The capacity modeled by the ASPM reported weather variables is compared against ASPM reported AAR. This comparison shows the model performance under perfect weather information. SREF-mean vs. reported AAR The mean of predicted AARs from 21 SREF ensemble members is compared against ASPM reported AAR. This comparison will show one aspect of the model performance using SREF forecast, as there might be other metrics to represent ensemble prediction such as median and quartiles. Because nominal capacity varies across airports, the tolerance to prediction error would vary across airports. For example, 5 aircraft off a reported AAR had different impact to a large airport like ATL than to a small airport like DCA. To account for the relative errors between predicted and reported values, mean percentage absolute error (MPAE) is used and defined as: (, )= 100% where x is the vector of predicted values, and y is the vector of reported values. Figure 7 illustrates the three performance comparisons for 35 OEP airports in summer Several observations are made and summarized as follows: The gray bars are MPAE of nominal AARs, where BOS, DEN, DTW, JFK, MSP, and SDF demonstrate average error rates over 20 percent for this naïve approach. These airports inherently demonstrate high variation in their AARs, which can be confirmed in Figure 8. Their coefficients of variance of AARs (standard deviation divided by mean) are higher than other airports. The yellow bars are MPAE of modeled AARs, where noticeable improvement is observed for those airports with poor performance in the naïve approach, except for SDF. The blue bars of Figure 7 are MPAE of the average of the predicted AARs from 21 SREF members. While it is expected that SREF-derived prediction has worse performance than the modeled AAR (from recorded weather variables), it shows a comparable performance for most airports. Compared to the naïve approach, there are airports that show indifferent or worse performance for modeled or SREF-mean AARs, which is unsurprising as the proposed model is systematically applied to 35 OEP airports without local operational knowledge (except for local VMC minima). The primary goal of the proposed AAR prediction model, either using ASPM or SREF weather variables, is meant to convert weather phenomena into the impact on airport arrival rate. Despite the fact that some airports show promising results, prediction errors can derive from model construct, missed forecast, or site-dependent factors that are not considered in the model, such as local common practice, runway dependency on other adjacent airports, departure or arrival push by time-of-day, runway outage or maintenance, TFM programs, etc. For example, SDF (Louisville International Airport), whose nominal rate is 58 aircraft per hour, regularly declares a low AAR (below 30 aircraft per hour) even when no prominent weather is present because its arrival demand during daytime is low. In fact, in summer 2015, SDF has 88 percent of time under VMC, during which 89 percent of time is under 20 arrivals/hour. 7

8 The three AAR prediction approaches shown in Figure 7 do not capture this non-weather related factor. However, the proposed model could use local expert opinion for modifying the runway preference table of Figure 1b in order to include non-weather consideration. Figure 7 Mean Absolute Percentage Error from Reported AARs Figure 8 Coefficient of Variation of Reported AARs B. Use of Band The mean of SREF-predicted AARs only provide point estimates of AARs. There are cases where the reported AARs are in the neighborhood of the mean. Leveraging the ensemble nature of prediction, if the standard deviation of SREFpredicted AARs is used to define an uncertainty band, the model performance on SREF can be measured in another perspective. Figure 9 summarizes the percentage of observations where the reported AARs fall within one and two standard deviations of the SREF mean. The blue bars are for one standard deviation, while the orange bars are for two standard deviations. As a rule of thumb, the band of two standard deviations contains at least 75 percent of the SREF-predicted AARs, per Chebyshev's inequality. 17 Intuitively, the percentage of the reported AARs being captured would increase when the uncertainty band becomes wider. CLT, DEN, IAH, JFK, LAS, MEM, and SFO show more than 20 percent increase from the band of one standard deviation to that of two standard deviations. This also implies that the SREF-predicted AARs for these airports have variation such that two more standard deviations away from the mean captures more reported AARs. The uncertainty band captures well the AARs of CLT, DCA, EWR, MIA, PIT, and SAN. Over 70 percent of the reported AARs fall within the band of two standard deviations. SLC and TPA demonstrate low capture rates. However, their error rates (MAPEs) in Figure 7 are around 10 percent or below, which means that the SREF-predicted AARs do not show large variation (i.e., small standard deviation), but their means are on average within 10 percent from the reported AARs. 8

9 The proposed AAR prediction model shows poor performance on DTW and SDF, measured by either MAPE or uncertainty band analyses. Figure 9 Percentage of Reported AARs within 1 and 2 Standard Deviations from SREF Mean C. Effect of Look-Ahead Times It is generally perceived that the longer the LAT, the higher the uncertainty of weather forecast. This hypothesis is also of interest to AAR prediction, i.e., whether AAR predictions also have poorer performance at longer LAT. To further examine this, Figure 10 shows the percentage of reported AARs within 2 standard deviations from the SREF mean binned by 1 to 6 look-ahead hours. By visual check, there exists difference among one to six hours ahead, but the behaviors are not consistent across 35 OEP airports. ANOVA analysis in Table 1 also supports the observation that variation exists among airports and lookahead hours. Under 95 percent confidence level, the statistical test confirms that AAR prediction variation is related to site-specific variables. The test also rejects the null hypothesis on the factor of look-ahead hours, suggesting that prediction performance does vary across future hours. However, whether there exists a pattern, such as deterioration at longer hours ahead, cannot be concluded. More site-dependent study may be conducted to examine the effect of look-ahead hours for particular airports. Figure 10 Prediction Performance by Look-Ahead Times 9

10 Table 1 ANOVA of Look-Ahead Time Source of Sum of Degree of Mean Variation Squares Freedom Square F-Dist P-value Note Airports ~0.000 P-value < 5% Reject null hypothesis at 95% confidence level Variation exists among airports Look- Ahead Hours P-value < 5% Reject null hypothesis at 95% confidence level Variation exists among lookahead hours Error Total VI.Concluding Remarks We examined the performance of an existing AAR prediction model systematically for the 35 OEP airports. We improved the model performance by refining meteorological condition categorization and by using more recent runway configuration data. Using SREF, we showed that weather forecast ensembles may be translated into capacity forecast ensembles of an airport. A range of possible capacity outcomes can capture the trend of weather impact to airport capacity at a strategic decision-making timeframe, typically 6 to 12 hours ahead. The uncertainty of capacity prediction can be quantitatively described by common statistical measures, e.g., standard deviation, variance. The performance analysis reveals that the proposed prediction model brings the most value to those airports whose arrival capacity varies significantly with weather phenomena. The SREF-mean AARs have as good prediction performance as the modeled AARs, implying that individual ensemble members collectively produce a prediction result close to those that would result using reported weather variables. The uncertainty band formed by the mean and standard deviations of the SREF-predicted AARs provides not only a visualization of uncertainty over time but also a way to measure prediction performance. Its practical meanings need further exploration. The proposed model does not predict well the AARs of a few OEP airports potentially due to local practices that are not included in the model. Future study will focus on accounting for non-weather related factors in model construct. Notice This work was produced for the U.S. Government under Contract DTFAWA-10-C and is subject to Federal Aviation Administration Acquisition Management System Clause , Rights In Data-General, Alt. III and Alt. IV (Oct. 1996). The contents of this material reflect the views of the authors and The MITRE Corporation and do not necessarily reflect the views of the FAA or the Department of Transportation (DOT). Neither the FAA nor the DOT makes any warranty or guarantee, or promise, expressed or implied, concerning the content or accuracy of these views The MITRE Corporation. All rights reserved. References 1 Tien, S., Roy, S., Taylor, C., and Wanke, C., Evaluation of an Airport Capacity Prediction Model for Strategizing Air Traffic Management, The 95th AMS Annual Meeting, Phoenix, AZ, January Dhal, R., Roy, S., Taylor, C., Tien, S., Wanke, C., An Operations-Structured Model for Strategic Prediction of Airport Arrival Rate and Departure Rate Futures, AIAA Aviation 2014, Atlanta, GA, June Dhal, R., Roy, S., Taylor, C., and Wanke, C., Forecasting Weather-Impacted Airport Capacities for Flow Contingency Management: Advanced Methods and Integration, 2013 Aviation Technology, Integration, and Operations Conference, American Institute of Aeronautics and Astronautics, Tien, S., Taylor, C., Wanke, C., Representative Weather-Impact Scenarios for Strategic Traffic Flow Planning, AIAA Aviation 2014, Atlanta, GA, June

11 5 Xue, M., Zobell, S., Roy, S., Taylor, C., Wan, Y., and Wanke, C., Using stochastic, dynamic weather-impact models in strategic traffic flow management. In Proceedings of the 91st American Meteorological Society Annual Meeting. Seattle, WA Steiner, Matthias, and J. Krozel. Translation of ensemble-based weather forecasts into probabilistic air traffic capacity impact. In Digital Avionics Systems Conference, DASC'09. IEEE/AIAA 28th, pp. 2-D. IEEE, Song, L., Greenbaum, D., and Wanke, C., The Impact of Severe Weather on Sector Capacity, The Eighth USA/Europe Air Traffic Management Research and Development Seminar, Napa Valley, CA, Taylor, C., Masek, T., Wanke, C., and Roy, S., Designing Traffic Flow Management Strategies Under Uncertainty, The 11th USA/Europe ATM R&D Seminar, Lisbon, Portugal, June Buxi, G., and Hansen, M., Generating probabilistic capacity profiles from weather forecast: a design-of-experiment approach, The 9th U.S.A./Europe ATM R&D Seminar, Berlin, Germany, Provan, Christopher A., Lara Cook, and Jon Cunningham. A probabilistic airport capacity model for improved ground delay program planning. Digital Avionics Systems Conference (DASC), 2011 IEEE/AIAA 30th. IEEE, Kicinger, R., Krozel, J. A., and Steiner, M., Airport capacity prediction incorporating ensemble weather forecasts, AIAA Guidance, Navigation, Control Conference, Los Angeles, CA, June Houston, S., and Murphy, D., Predicting Runway Configurations at Airports, Proceedings of the Annual Meeting of Transportation Research Board, Washington D.C., Ramanujam, V.; Balakrishnan, H., Estimation of Maximum-Likelihood Discrete-Choice Models of the Runway Configuration Selection Process, Proceedings of the American Control Conference (AAC 2011), San Francisco, CA. 14 DeLaura, R. A., Ferris, R. F., Robasky, F. M., Troxel, S. W., Underhill, N. K., Initial Assessment of Wind Forecasts for Airport Acceptance Rate (AAR) and Ground Delay Program (GDP) Planning, Project Report ATC-414, MIT Lincoln Laboratory, Lexington, MA, Jan Du, J., DiMego, G., Toth, Z., Jovic, D., Zhou, B., Zhu, J., Chuang, H., Wang, J., Juang, H., Rogers, E., and Lin, Y., NCEP Short-Range Ensemble Forecast (SREF) System Upgrade in 2009, The 19th Conf. on Numerical Weather Prediction and the 23rd Conf. on Weather Analysis and Forecasting, Omaha, Nebraska, June 1-5, NOAA Operational Model Archive and Distribution System (NOMADS), 17 Kvanli, Alan H.; Pavur, Robert J.; Keeling, Kellie B. (2006). Concise Managerial Statistics. Cengage Learning. pp