Using Reforecasts to Improve Forecasting of Fog and Visibility for Aviation Greg Herman

Transcription

1 Using Reforecasts to Improve Forecasting of Fog and Visibility for Aviation Greg Herman

2 Introduction Cloud ceiling and visibility have a very substantial impact on the aviation industry Federal Aviation Administration (FAA) regulations determined by Flight Rules (FRs) and Meteorological Conditions (MCs) in effect Some airports must reduce operations (stop operating on certain runways) during reduced visibility Good reason- low visibility has been attributed as the primary cause (in conjunction with pilot error) of numerous accidents and crashes throughout aviation history in both general and commercial aviation

3 Motivation Great value to having skillful Flight Rule Category (FRC) forecasts Fog and other processes affecting FRCs are often small-scale, very sensitive to small changes in wind or other fields, and require very high resolution in the vertical (near-surface grid spacing O(meters) or less) Very difficult to accurately model dynamically, especially in an operational setting Clear and relatively well understood physical relationships between fog/low clouds and other atmospheric proxy variables Statistical modeling/post-processing may yield best results for this forecast problem

4 Goal(s) Use statistical post-processing techniques to generate skillful, calibrated probabilistic FRC forecasts at several different major airports across CONUS Explore several different aspects of statistical model formulation and configuration Best algorithm(s) Most useful model predictor(s) How to best post-process an NWP ensemble Etc. Produce end products/models that could be readily expanded to more widespread use and application

5 Flight Rule Conditions (FRCs) FAA defines flight rule categories: 1. Visual Flight Rules (VFR) Ceiling above 3,000 feet AGL Visibility above 5 miles 2. Marginal Visual Flight Rules (MVFR) Ceiling between 1,000 and 3,000 feet AGL, or Visibility between 3 and 5 miles 3. Instrument Flight Rules (IFR) Ceiling between 500 and <1,000 feet AGL, or Visibility between 1 and <3 miles 4. Low Instrument Flight Rules (LIFR) Ceiling less than 500 feet AGL or Visibility less than 1 mile

6 Seattle-Tacoma International (KSEA) Stations Studied Denver International (KDEN) San Francisco International (KSFO) George Bush Intercontinental (KIAH)

7 FRC Climatologies Use archive of (human-augmented) METARs to create time series of cloud ceiling height, visibility, and FRCs for each station of study Examine as a function of month-of-year and hour-of-day

8 Station Seasonal Cycles

9 KDEN: Diurnal Cycle

10 KSEA: Diurnal Cycle

11 KSFO: Diurnal Cycle

12 KIAH: Diurnal Cycle

13 Methods: Model Data NOAA s Second-Generation Global Medium-Range Ensemble Reforecast Dataset (GEFS/R; Hamill et al. 2013) 00Z initialization; forecasts out to 384h 11 ensemble members Operational version of GEFS on 2/14/2012 T254L42 resolution (~40 km) Data from Dec Present Same model configuration throughout Updates to data assimilation over the period (CSFR 12/84-2/11; GSI 2/11-5/12; hybrid ENkF 5/12-Present)

14 Methods Use GEFS/R data from to train and configure many different statistical models Test different algorithmic configurations Test different input predictors/features Use statistical models to post-process GEFS/R forecasts and create FRC forecasts at different stations across CONUS for hour lead times Long enough that current observations of less utility, short enough that some predictability/utility Algorithms Tested: Logistic Regression (LOG_REG) Random Forest Classification (RAND_FOR) Gradient Boosting Classification (GRAD_BOOST) Support Vector Classification (SVC) K-Nearest Neighbors (KNN) Weighted Average of all Retained Classifiers (WAVG_ALL) Algorithms implemented based on Python s Scikit-Learn Package (Pedregosa et al. 2011)

15 Methods: Evaluation Four FRCs; Ordinal, not nominal (IFR closer to LIFR than VFR) Use Rank Probability Score (RPS) Where K is the number of forecast categories, P j is the forecast probability for each category, and O j denotes whether category j was observed For reference, can convert to a skill score (RPSS or ARPSS) using an informed climatology reference with forecast probabilities based on the climatological FRC frequencies for the given forecast month and hour in a 24x12(x4) climatology table

16 Sensitivity Tests 12 years of data ( ) used to train the statistical model(s) and determine best configuration Evaluated using cross validation with four chunks Pick an algorithmic configuration (set of parameters) Train model four times on 9 years of data ( ; & ; & ; ) Evaluate each trained model on the three years of held out data, for a total of twelve years of verification Use Rank Probability Skill Score (RPSS) for verification to determine optimal configuration Due to the large dimensionality of the parameter space, experiments are largely performed greedily, rather than exhaustively

17 Sensitivity Tests: Atmospheric Field Selection Somewhat primitive method of feature selection Test a set of atmospheric proxy variables believed a priori to conceivably have a physical relationship with FRCs Retain those variables whose inclusion as features of the statistical models improves RPSS

18 Sensitivity Tests: Ensemble Information Five configurations explored: 1. Control (CTRL): Use only forecast information from the GEFS/R control member 2. Mean & Spread (MNSPRD): For any given atmospheric field and location, use the ensemble mean & spread as candidate predictors/features 3. Confidence Bounds (CNFDB): For any given atmospheric field and location, use the median, 2 nd -from-lowest, and 2 nd -from-highest ranking values from the ensemble as input features 4. All Members (MEMS): Use the control member s forecast values, perturbation 1 s forecast values, perturbation 2 s forecast values, etc. as input features 5. Ranked Members (RMEMS): For a given field and location, use the ensemble minimum, 2 nd -from-lowest, 3 rd -from-lowest,,ensemble median,,ensemble maximum as input features

19 Sensitivity Tests: Predictor Radius By default, use forecast values from nearest outputted grid point from GEFS/R to the forecast station as input features No additional interpolation of GEFS/R data to forecast station, etc. Can also use forecast values from surrounding points- does one gain much from doing so?

20 Sensitivity Tests: Predictor Radius and Ensemble Information

21 Sensitivity Tests: Parameter Tuning Random Forests Forest Size (B) Findings support previous literature: bigger is better, diminishing returns Splitting Criterion (H) Gradient Boosting Gini Impurity vs. Entropy; didn t really matter, entropy slightly better Ensemble Size (B) Same as for Forest Size Maximum Tree Depth (D MAX ) 1 (stumps) best; deeper trees performed moderately worse Fraction of Features Considered per Estimator (α features ) 0.5 best; little difference Fraction of Training Examples to Sample per Estimator (α subsamp ) 1 best; little difference KNN Number of Nearest Neighbors (K) SVC Same as for B above Distance Metric (D) Euclidean or Mahalonobis; Euclidean was better, but didn t spend too much time on tuning Mahalonobis Neighbor Weighting (W) Uniform vs. Inversely proportional to distance; uniform was better Choice of kernel function (k) Linear vs. Quadratic vs. Cubic vs. Radial Basis Function (RBF) vs. Sigmoid; RBF best, moderately sensitive Kernel Coefficient (γ) 1/N best; fairly insensitive Regularization Coefficient (C) 1 best; fairly insensitive

22 Sensitivity Tests: Training Data Length

23 Results: Cross-Validation

24 Results: Test After optimal configurations determined, re-train models using all 12 years of training data Evaluate over to make final determination of statistical model performance in an operational setting (Weights for WAVG_ALL based on cross-validation results)

25 KDEN Feature Importances

26 KSEA Feature Importances

27 KSFO Feature Importances

28 KIAH Feature Importances

29 Results: Test

30 Real Time Forecasts Now creating forecasts in real-time for the four stations examined in the study Available at: colostate.edu/gherman/aviation.php

31 Conclusions Statistical post-processing of long record of reforecasts yielded significant improvement in FRC forecast skill relative to both climatology and operational forecast guidance at all stations studied. Probabilistic forecasts add substantial value over deterministic ones Often far-field values, especially immediately offshore where applicable, have stronger correspondence to the FRC values than more local forecast values Dew point depression, cloud cover, and to a lesser extent latent heat fluxes, low level lapse rates, and surface winds are most important predictors (among those examined) of fog and degraded FRCs

32 Future Work Expand to more stations Explore application to other forecast lead times Explore using more physically relevant derived variables, particularly those based on local relationships (e.g. pressure gradients) Explore training based on season (increase relevance of training examples) Explore training based on meteorological context Both improving forecast skill and diagnosing physical relationships in specific meteorological situations Apply these methods to other relevant forecasting problems

33 Backup

34 Notable Examples Tenerife Disaster 3/27/77 Explosion at nearby Gran Canaria Airport diverted many planes to small Los Rodeos airport Insufficient capacity at Los Rodeos forced many planes to park on taxiway Taxiing had to be done on runway Dense fog formed reducing visibility to near zero Miscommunication resulted in KLM pilot taking off without clearance Pan-Am flight still on runway, neither could see the other 583 killed- deadliest non-terrorist aviation accident in history

35 Notable Examples Madrid disaster 12/7/83; 93 killed Similar circumstances to Tenerife Linate disaster 10/8/01; 118 killed Similar circumstances to Tenerife Air China 129 and China Air 676 4/15/02; 2/16/98 129; 203 killed Poor visibility due to fog, pilot mis-navigated on landing JFK Jr. Crash 7/16/99; 3 killed Reduced visibility due to late evening haze Many more

36 Complexity: Number of Features N = #Features = (2r + 1) 2 EF r: Predictor radius E: Ensemble multiplier factor 1 CTRL E = 2 MNSPRD 3 CNFDB 11 MEMS 11 RMEMS F: Number of atmospheric fields