Forecast Confidence Haig Iskenderian 18 November 2014 Sponsor: Randy Bass, FAA Aviation Weather Research Program, ANG-C6 Distribution Statement A. Approved for public release; distribution is unlimited. This work is sponsored by the Federal Aviation Administration under Air Force Contract #FA8721-05-C-0002. Opinions, interpretations, recommendations and conclusions are those of the author and are not necessarily endorsed by the United States Government.
Traffic Flow Management (TFM) Decision Portfolio Playbook Reroutes Ground Stops Ground Delay Programs Airspace Flow Programs Arrival & Departure Fix Metering National Regional 9+ 6 3 Planning Horizon (Scale & Hours) Local 0 Forecast Confidence - 2
Sample Strategic TFM Decisions Sample Strategic Forecast Tools Strategic Decisions Manage tactically? Throttle traffic flow? Reduce airport arrivals? LAMP (Probabilistic) CCFP (Probabilistic) CoSPA (Deterministic) Strategic reroutes? TFM decisions require high-fidelity forecasts of storm structure and characteristics Forecast Confidence - 3
Prior Findings on Forecast Uncertainty (CoSPA/LAMP 2010) (CoSPA 2011) User Feedback: Participants felt uncertain about the forecast accuracy. Participants noted the inclusion of a forecast accuracy score (similar to CIWS) or trending scores would be beneficial to provide insight into forecast performance to decision making and reliance on the forecast Study Findings: (2013 & 2014) NAS users and weather information providers do understand that all convective weather observations and forecasts contain uncertainty NAS users and weather information providers do not have an equivalent understanding of terms used to express convective weather uncertainty Aviation users require a measure of forecast confidence appropriate for their decisions Forecast Confidence - 4
Examples of Providing Forecast Confidence Probability of a Weather Event Confidence inferred from probability value Provides probability of weather event, not operational impact Often converted to binary decision by human users Prior Forecast Performance Confidence inferred from prior forecast performance Cast in terms of weather, not operational impact Does not translate easily to TFM decisions Expressions of a Weather Event Range of Possible Weather Impact with Uncertainty Confidence provided by expressions like Low Confidence, and High Confidence in storm coverage Cast in terms of weather event, not operational impact Does not translate easily to TFM decisions Im mpact 0h hr Low Impact Medium Impact High Impact Most Likely Impact Time Uncertainty Confidence provided by most likely outcome along with forecast uncertainty Cast in terms of operational impact Translates easily to 8h hr TFM decisions Forecast Confidence - 5
Outline Motivation Forecast confidence capability Decision-based confidence Initial capability assessment Summary Forecast Confidence - 6
Decision-based Confidence Airspace Flow Program Example Resources Decisions Flow Constrained Area (FCA) Airspace Flow Program (AFP) FCAOB1 Used to reduce demand for airspace in the presence of impacting weather When to start the AFP? When to stop the AFP? Flow is throttled ttl across FCA What flow rate to set for the AFP? Forecast Confidence - 7
Decision-Based Confidence Define Resources Define Decisions Provide Uncertainty NY Metro FCA FCA Airspace Availability Most Likely Availability & Range FCAOB1 FCAOB1 FCABW1 FCAA01 1 hr 2 hr 3 hr 4 hr 5 hr 6 hr 7 hr 8 hr lockage Low Impact Medium Impact FCAOB1 B Low Impact High Impact Medium Impact High Impact 0 hr Time 8 hr Forecast confidence is provided in terms of airspace impact Forecast Confidence - 8
Classifying En-Route Convective Impacts Airspace Resource Definitions Airspace boundary FCA Airspace crossing Airspace traversing trajectories Weather Avoidance Field (WAF) Airspace Resource Definitions Directional flow Major ATC flow regions Constraint regions (ARTCC transitions) Airspace Classification Translate into ATM metrics for decisionmaking (weather impact on capacity) Weather Forecasts Echo Tops Trajectory Parameters (Convective Weather Avoidance Model) Pilot Deviation Probability Flight Alt. Echo Top Precipitation Intensity WAF (minutes) Encounter Length Trajectory Impact Model 0.0 0.2 0.5 1.0 Trajectory Block kage Airspace Classification Low Impact Medium Impact VIL Encounter Maximum WAF (%) High Impact Forecast Confidence - 9
Modeling Forecast Uncertainty Train a model that provides probability of forecasted impact Blockage Weather Features Deterministic Forecasts + Time Lagged Ensemble 6 Hour 5 Hour Uncertainty Model Combines blockage and weather features measured from multiple time-lagged forecasts to estimate probability of airspace impact 0.6 0.5 0.4 03 0.3 0.2 0.1 0 Probability bilit of Impact Low Medium High Impact Impact Impact 4 Hour 3H Hour Historical Forecast Performance 5 Hour Forecast Confidence - 10
FCA Blockage and Uncertainty FCA Bloc ckage 0% 20% 40% 60% 80% 100% Low Impact Medium Impact High Impact Forecast Issued 12 UTC TMI? Worst Case Scenario Best Case High Scenario Forecast Uncertainty Ensemble of CoSPA Forecasts Valid 8 Hours in the Future (20Z) Most likely blockage Time (UTC) Possible range of blockage Forecast Confidence - 11 TMI = Traffic Management Initiative
FCA Blockage and Uncertainty FCA Bloc ckage 0% 20% 40% 60% 80% 100% Low Impact Medium Impact High Impact Forecast Issued 12 UTC TMI? High Forecast Uncertainty Ensemble of CoSPA Forecasts Valid 8 Hours in the Future (20Z) FCA Blockage 0% 20% 40% 60% 80% 100% Low Impact Medium Impact High Impact Time (UTC) Forecast Issued 15 UTC TMI! Low Forecast Uncertainty Ensemble of CoSPA Forecasts Valid 5 Hours in the Future (20Z) Observed blockage Time (UTC) Regular forecast updates needed to revisit decisions Forecast Confidence - 12 TMI = Traffic Management Initiative
FCA Bloc ckage 0% 20% 40% 60% 80% 100% Forecast Event Start, Duration and Recovery Low Impact Medium Impact High Impact Forecast Issued 15 UTC Event start Ongoing g event Event Start Ongoing g Event Time (UTC) 0% Forecast Issued 22 UTC FCA Blockage 20% 40% 60% 80% Ongoing event Event recovery Event Recovery 100% Time (UTC) Forecast Confidence - 13
Mapping Blockage to Flow-Based Capacity Assembled large database of weather impacted and non-impacted days 25 Weather impacted days (10 from 2013, 15 from 2014) 31 Non-weather days (August 2013) Flow Rate (non-impact days) ZOB/ZNY Transition Airspace Historical time period of day with highest demand Nominal clear weather flow rate Goal: To supply forecast confidence in terms of flow rates Forecast Confidence - 14
Comparing Permeability and Flow Rate Observed Flow Rat te (ac/15 min) ZOB/ZNY Transition Airspace (18Z to 00Z) Maximum Clear Weather Flow Rate Nominal Clear Weather Flow Rate Modeled Permeability and Observed Flow Rates are correlated Mapping Permeability into Traffic Management Initiative programs is specific to FCA and requires additional studies Observed ed Weather e Modeled ed Permeability* eab (%) * Permeability = 1 - Blockage Flow rates provided for several FCAs Forecast Confidence - 15
Outline Motivation Forecast confidence capability Initial capability assessment Statistical Informal operational Summary Forecast Confidence - 16
Scoring Approach Blockage forecasts made in real-time for FCAs along major flows Validated in Northeast US for all forecasts from May - August 2014 Real-time FCAs Forecast Confidence - 17
Scoring Approach Blockage forecasts made in real-time for FCAs along major flows Validated in Northeast US for all forecasts from May - August 2014 FCAs Validated Forecast Confidence - 18
Forecast Confidence Assessment Observed Fre equency Reliability for Probability of Impact Categories Low Impact Medium Impact High Impact Curve along diagonal desired 6 Hour Forecasts Observed Fre equency 6 Hour Forecasts Observed Fre equency Good reliability for high impact events 6 Hour Forecasts Statistics for May to Aug 2014 Predicted Probability Predicted Probability Predicted Probability Precision of Impact Categories Midpoint-Ob bserved Blocka age Over Forecast Under Forecast Median along zero line desired Midpoint-O Observed Block kage Over Forecast Under Forecast Midpoint-O Observed Block kage Over Forecast Under Forecast Good precision for high impact events Forecast Lead Time (Hours) Forecast Lead Time (Hours) Forecast Lead Time (Hours) Forecast Confidence - 19
Outline Motivation Forecast confidence capability Initial capability assessment Statistical Informal operational Summary Forecast Confidence - 20
2014 CoSPA Operational Observations: Field Sites 3 field observation blitzes 24-25 June 14 July 12 August FAA Air Traffic Control System Command Center Targeted key strategic facilities 5 FAA facilities ARTCCs: Boston, New York, Cleveland, Washington ATCSCC Airlines: JetBlue, Delta Forecast confidence shown to limited users at ATCSCC Forecast Confidence - 21 ARTCC = Air Route Traffic Control Center ATCSCC = Air Traffic Control System Command Center
FCA Blockage/Capacity Estimation: Prototype Display Very strong positive reaction to Flow Rate addition by yplanners and ATCSCC operations manager ZNY001 Several suggestions used as input for initial webbased display Unofficially used during 12 August observation by planner to determine rate and timing of several AFPs Clicking displays FCA confidence plot Predicted blockage category I think your data yesterday was spot on. I wrote to (FAA NE Tactical Operations) and I want to make certain we discuss this after the summer. It seems to be providing really good results. Thank you! - ATCSCC Planner Forecast Confidence - 22
Future Work Expand and validate mapping of FCA blockage to flow rate to improve capability for Traffic Flow Management Develop enhancements to tighten uncertainty bounds Incorporate other models in decision-based confidence (e.g. SREF and LAMP probabilistic models) Consider confidence for broader portfolio of decisions (e.g. terminal area) Further capability and user interface assessments Forecast Confidence - 23
Summary Confidence in weather forecast information is crucial for Traffic Flow Management (TFM) How much can I trust this forecast to help make a decision? Expressions of forecast confidence are often inadequate for aviation decision making Do not translate well to managing airspace resources Developed forecast confidence capability for deterministic forecasts Provides confidence in Flow Constrained Area (FCA) blockage Translates to TFM decisions regarding Airspace Flow Programs Beginning to map FCA blockage to flow rates Forecast Confidence - 24