Denis Nikolayevich Botambekov. Master of Science In Hydrometeorology Kazakh National University 2006

Transcription

1 Statistical Evaluation of the National Hurricane Center s Tropical Cyclone Wind Speed Probability Forecast Product by Denis Nikolayevich Botambekov Master of Science In Hydrometeorology Kazakh National University 2006 A thesis submitted to the College of Engineering/ Department of Marine and Environmental Systems at Florida Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Meteorology Melbourne, Florida May, 2011

2 Copyright 2011 Denis Nikolayevich Botambekov All Rights Reserved The author grants permission to make singe copies

3 We the undersigned committee hereby recommend that the attached document be accepted as fulfilling in part the requirements for the degree of Master of Science of Meteorology Statistical Evaluation of the National Hurricane Center s Tropical Cyclone Wind Speed Probability Forecast Product, a thesis by Denis Nikolayevich Botambekov Steven M. Lazarus, PhD. Associate Professor Marine & Environmental Systems Committee Chair Syamal K. Sen, PhD. Professor Mathematical Sciences Committee Member Michael E. Splitt, MS Research Professor Marine & Environmental Systems Committee Member George A. Maul, PhD. Professor and Department Head Marine & Environmental Systems

4 Abstract Statistical Evaluation of the National Hurricane Center s Tropical Cyclone Wind Speed Probability Forecast Product by Denis Nikolayevich Botambekov Committee Chair: Steven M. Lazarus, PhD. The Wind Speed Probability Forecast Product (WPFP) was developed at Cooperative Institute for Research in the Atmosphere (CIRA) to predict categorical wind speed likelihood due to tropical cyclones DeMaria (2009). The National Hurricane Center (NHC) has adapted this product and provides this information to the general public, government agencies and private companies. The 45 th Weather Squadron (45 WS) is responsible for interpreting and relaying critical weather information to Patrick Air Force Base, NASA s Kennedy Space Center (KSC), Cape Canaveral Air Force Station (CCAFS) and other local companies Shafer (2008). The WPFP is a useful tool to evaluate significant threats to these agencies when a TC approaches their area of operations. The previous work of Shafer (2008) and Splitt et al. (2009) was limited to the tropical cyclone seasons and spatially limited to the U.S. coastline from Brownsville, TX to Bar Harbor, ME. Here, the product is evaluated for all threatening TC events in Atlantic basin over seasons. As in the previous work, the discernment of risk is associated with both the interval probability (IP) and the cumulative probability (CP) forecasts for the three wind speed categories ( 34-kt, 50-kt, and 64-kt). A quantitative assessment of the WPFP is performed for the seven IP (0-12, 12-24, 24-36, 36-48, 48-72, 72-96, and h) and six CP iii

5 forecast time intervals (0-24, 0-36, 0-48, 0-72, 0-96, and h). The area of interest includes all International, U.S. and open ocean locations for which the NHC WPFP issues forecasts (150 locations total). Reliability diagrams are commonly used to directly assess the probabilistic forecasts. Overall, the IP reliability diagram indicates that the WPFP over-forecasts for forecast probabilities less than 60%, nearly perfectly reliable for forecast probabilities between 61% and 80%, and under-forecasts for probabilities between 81% and 100%. The CP reliability diagram indicates the WPFP over-forecasts for probabilities up to 40%, under-forecasts for probabilities between 41% - 90%, and has low resolution for probabilities between %. The attribute diagram, which is a modification of the standard reliability diagram, was used for additional direct evaluation of the WPFP forecasts. The attribute diagram has all benefits of a reliability diagram but with the additional ability to assess forecast skill. Both WPFP IP and CP attribute diagrams indicate for the most part that the forecast system has positive skill. Additional methods to indirect measures of forecast skills are available for forecast verification, but require the probabilities to be converted into binary (yes/no) forecasts with the use of a decision threshold. The optimal thresholds, those that provide the greatest forecast skill, allow for evaluation of the performance of the forecasting system. Also, decision makers can use the decision thresholds as an aide to determining when appropriate actions should be taken such for the safety of personal and equipment. Decision thresholds were selected using maximum values from the true skill statistic (TSS) and the Heidke skill score (HSS) methods. IP decision thresholds range from 1% to 37%; while CP range from 2% to 43% depending on the threshold selection method (i.e. TSS, HSS), wind speed category and time interval. More statistically relevant decision thresholds are obtained from bootstrap resampling methods. These results suggest iv

6 that decision thresholds should be determined from an average (or median) of the resampled data. Indirect evaluation of the NHC WPFP shows that it performs reasonably well. Both TSS and HSS IP statistics demonstrate significant forecast skill up to the 72h forecast interval, while the CP has significant skill for all time intervals. The bias score indicates slight under-forecasting by the WPFP with using decision thresholds based on the HSS, CP are not affected by the bias. Bias scores for the WPFP IP and CP, based on TSS, indicate a tendency for the WPFP to overforecast. WPFP null forecasts, i.e. less than 0.5% probability ( X ), can affect the value of the decision thresholds. Sensitivity of the threshold selection methods to null forecasts is tested by using a buffer zone to filter varying amounts of null forecasts. The sensitivity tests show that the TSS based thresholds are more sensitive to the inclusion of the null forecasts, while the HSS based thresholds are not. The use of a 400 km buffer zone is recommended for filtering null forecasts. The WPFP forecasts are segregated into the following TC intensity categories to test for performance differences: intensifying, no-trend (no change) and weakening. The performance of these categories is different with statistical significance. However, the forecast system performs well for any stage of TC intensification, and the best for weakening TCs. In order to evaluate WPFP performance differences prior to and after storm closest passage of a TC, data are divided into two separate sets by the time of closest approach (before and after), i.e. when the distance between the center of TC and station is minimal. Both IP and CP attribute diagrams, for before the approach, are similar to those for the full dataset, but t-test shows significant, but small differences. Results for After the closest approach are impacted by sample size which has approximately 10 times fewer samples than the before data set. Evaluation of the WPFP was conducted for two sub-regions of interest to the 45 WS, the Cape Canaveral region in Central Florida and Antigua area in v

7 Atlantic Ocean in order to assess if the WPFP performs differently in these subregions. Statistically significant differences for the WPFP performance are observed for both sub-regions. At the same time, the small size of the Antigua subsample does not give statistically reliable results on WPFP performance. vi

8 Table of Contents List of Figures... ix List of Tables... xiv List of Abbreviations/Acronyms... xvi Chapter 1 Introduction... 1 Chapter 2 Data and Methods NHC WPFP data HURREVAC Reliability and frequency distributions Attribute diagram Verification statistics Decision thresholds Confidence intervals and t-test Null forecasts and the use of a buffer zone TC Intensity Closest approach of TC Cape Canaveral and Antigua regions Chapter 3 Results Reliability and frequency Attribute diagrams vii

9 3.3 Heidke skill score and true skill statistic The bias score Null forecasts and the use of a buffer zone Maximum forecast probability WPFP performance based on the TC intensity Closest approach of TC Cape Canaveral region Antigua region Chapter 4 Summary References viii

10 List of Figures Figure 2.1 Location of the 150 stations evaluated for the years 2005 to Figure 2.2 Location of the 43 stations evaluated for the 2004 year... 6 Figure 2.3. Example of HURREVAC wind speed radii for hurricane Frances, September 4th, 2004 at 1800 UTC... 8 Figure 2.4 Examples of reliability diagram performance Figure 2.5 Example of an attribute diagram Figure 2.6 Verification statistics for IP (including null forecasts), for all wind categories and forecast time intervals for the TC seasons Figure 2.7 HSS-based IP, with all null forecasts excluded for 50-kt wind speed and 7 time intervals with 95% confidence intervals Figure 2.8 Example of a 100 km radius buffer zone Figure 2.9 Stations used for WPFP evaluation of Cape Canaveral area Figure 2.10 Stations used for WPFP evaluation of Antigua area Figure 3.1 IP (a) reliability diagram and associated (b) frequency distribution of full data set using all wind speed categories and forecast time intervals Figure 3.2 CP (a) reliability diagram and associated (b) frequency distribution of full data set using all wind speed categories and forecast time intervals Figure 3.3 IP attribute diagram with all wind speed categories and forecast time intervals included ix

11 Figure 3.4 CP attribute diagram with all wind speed categories and forecast time intervals included Figure 3.5 IP (a) and CP (b) TSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with 95% confidence intervals, chosen based on the maximum TSS skill value (with null forecasts excluded) Figure 3.6 IP (a) and CP (b) TSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with 95% confidence intervals, chosen based on the maximum TSS skill value (with null forecasts included) Figure 3.7 IP (a) and CP (b) HSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with 95% confidence intervals, chosen based on the maximum HSS skill value (with null forecasts excluded) Figure 3.8 IP (a) and CP (b) HSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with 95% confidence intervals, chosen based on the maximum HSS skill value (with null forecasts included) Figure 3.9 IP true skill statistic (TSS) values with null forecasts excluded, based on (a) TSS-based decision thresholds method with 95% confidence intervals, (b) HSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.10 IP true skill statistic (TSS) values with null forecasts included, based on (a) TSS-based decision thresholds method with 95% confidence intervals, (b) HSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.11 CP true skill statistic (TSS) values with null forecasts excluded, based on (a) TSS-based decision thresholds method with 95% confidence intervals, (b) HSS-based decision thresholds method, versus forecast time interval for each wind speed criteria x

12 Figure 3.12 CP true skill statistic (TSS) values with null forecasts included, based on (a) TSS-based decision thresholds method with 95% confidence intervals, (b) HSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.13 IP Heidke skill score (HSS) values with null forecasts excluded, based on (a) HSS-based decision thresholds method with 95% confidence intervals, (b) TSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.14 IP Heidke skill score (HSS) values with null forecasts included, based on (a) HSS-based decision thresholds method with 95% confidence intervals, (b) TSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.15 CP Heidke skill score (HSS) values with null forecasts excluded, based on (a) HSS-based decision thresholds method with 95% confidence intervals, (b) TSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.16 CP Heidke skill score (HSS) values with null forecasts included, based on (a) HSS-based decision thresholds method with 95% confidence intervals, (b) TSS-based decision thresholds method, versus forecast time interval for each wind speed criteria Figure 3.17 IP frequency distribution for 12h, 24h and 36h Figure 3.18 Sensitivity of the decision threshold (%) to the change of the buffer zone size in IP data set for 34-kt 12h forecast Figure 3.19 Sensitivity of the decision threshold (%) to the change of the buffer zone size in IP data set for 34-kt 120h forecast Figure 3.20 Total number of forecasts used at each IP data set for 34-kt 12 and 120h Figure 3.21 Attribute diagram of IP forecasts of intensifying TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone xi

13 Figure 3.22 Attribute diagram of IP forecasts of no-trend intensity TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone Figure 3.23 Attribute diagram of IP forecasts of weakening TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone Figure 3.24 Attribute diagram of CP forecasts of intensifying TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone Figure 3.25 Attribute diagram of CP forecasts of no-trend intensity TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone Figure 3.26 Attribute diagram of CP forecasts of weakening TCs, for all wind speed categories and forecast time intervals included with use of a 400 km radius buffer zone Figure 3.27 TSS skill score for different stage of TC intensity with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.28 HSS skill score for different stage of TC intensity with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.29 IP Attribute diagram before the time of closest approach with use of a 400 km buffer zone Figure 3.30 IP Attribute diagram after the time of closest approach with use of a 400 km buffer zone Figure 3.31 CP Attribute diagram before the time of closest approach with use of a 400 km buffer zone Figure 3.32 CP Attribute diagram after the time of closest approach with use of a 400 km buffer zone Figure 3.33 TSS skill score before and after the time of closest approach with use of a 400 km radius buffer zone and 95% confidence intervals xii

14 Figure 3.34 HSS skill score before and after the time of closest approach with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.35 IP attribute diagram of Cape Canaveral region with use of a 400 km buffer zone Figure 3.36 CP attribute diagram of Cape Canaveral region with use of a 400 km buffer zone Figure 3.37 TSS skill score of WPFP for Cape Canaveral region with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.38 HSS skill score of WPFP for Cape Canaveral region with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.39 IP attribute diagram of Antigua region with use of a 400 km buffer zone Figure 3.40 CP attribute diagram of Antigua region with use of a 400 km buffer zone Figure 3.41 TSS skill score of WPFP for Antigua region with use of a 400 km radius buffer zone and 95% confidence intervals Figure 3.42 HSS skill score of WPFP for Antigua region with use of a 400 km radius buffer zone and 95% confidence intervals xiii

15 List of Tables Table 2.1 Example of the text wind speed probability forecast for tropical storm Fay, 0300 UTC August 17th, Table 2.2 Classifications assigned to the probability forecasts... 7 Table 2.3 Contingency for classification of probability forecasts Table 2.4 Verification statistics Table 3.1 TSS- and HSS-based decision thresholds (%) and skill Table 3.2 TSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) Table 3.3 HSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) Table 3.4 Bias score for the IP and CP data sets with included and excluded null forecasts, based on TSS and HSS methods Table 3.5 TSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with use of a 400 km buffer zone Table 3.6 HSS-based decision thresholds (%) for each wind speed criteria (kt) and forecast time interval (h) with use of a 400 km buffer zone Table 3.7 T-test results of HSS decision thresholds (400 km buffer) at 95% confidence level Table 3.8 IP maximum probability (%) forecast as a function of wind speed criteria (kt) and time interval (h) Table 3.9 TSS- and HSS-based decision thresholds (%) for different stages of TC intensity with use of a 400 km radius buffer zone xiv

16 Table 3.10 TSS- and HSS-based decision thresholds (%) before and after the time of closest approach with use of a 400 km radius buffer zone Table 3.11 TSS- and HSS-based decision thresholds (%) for Cape Canaveral region with use of a 400 km radius buffer zone Table 3.12 TSS- and HSS-based decision thresholds (%) for Antigua region with use of a 400 km radius buffer zone xv

17 List of Abbreviations/Acronyms NWS: NHC: WPFP: 45WS: SMG: TC: KSC: CCAFS: IP: CP: HSS: TSS: FA: CN: PoD: PoFD: FAR: TS: FD: National Weather Service National Hurricane Center Wind Speed Probability Forecast Product 45 th Weather Squadron Spaceflight Meteorology Group Tropical Cyclone Kennedy Space Center Cape Canaveral Air Force Station Interval Probability Cumulative Probability Heidke Skill Score True Skill Statistic False Alarm Correct Negative Probability of Detection Probability of False Detection False Alarm Ratio Threat Score Frequency Distribution xvi

18 Chapter 1 Introduction The Wind Speed Probability Forecast Product (WPFP) provided by the National Hurricane Center (NHC) gives a forecast probability for three wind categories at predefined locations for threatening tropical cyclones (TC). Patrick Air Force Base, NASA s Kennedy Space Center (KSC), Cape Canaveral Air Force Station (CCAFS) and other local companies in Central East Florida receive comprehensive weather information from the 45 th Weather Squadron (45 WS). The WPFP helps to make the decisions about the safety of personal and as well as decisions on launch cancellations (Shafer et al. 2007; Shafer 2008; Splitt et al. 2009). The Spaceflight Meteorology Group (SMG) provides weather information for Space Shuttle operations including the Johnson Space Center. Both the 45 WS and SMG provide detailed information (Winters et al. 2006) including track, timing, intensity and size of threatening tropical cyclones (TCs). Previous evaluation of the NHC product was conducted to address two key questions 1.) How well does the product perform and 2.) What probability values are significant for making a yes/no decision? In this earlier work the reliability diagrams indicated that the WPFP has a tendency, for IP forecasts of 60% or more, to overforecast TC events (Splitt et al. 2009). In particular, it tends to overforecast the 34-kt, but underforecasts the 64-kt wind speed category. The 50-kt results are mixed, but show a tendency of the WPFP to under-forecast for latter time intervals. 1

19 Evaluating the WPFP indirectly requires selection of decision thresholds to convert probability forecasts into binary (yes/no) forecasts. The decision threshold, are selected by maximizing the skill of a particular forecast evaluation metric. The true skill statistic (TSS) and Heidke skill score (HSS) were used to select decision thresholds. These thresholds vary in range from 1% to 55% depending on the selection method, wind speed category, and time interval (Splitt et al. 2009). Based on its reduced sensitivity to the filtering of correct negatives and better resultant WPFP bias scores, the Heidke skill score (HSS) was recommended over true skill statistic (TSS) for determining thresholds. The results of the CP evaluation (Shafer, 2008) indicated that the product performed well, but has a tendency to underforecast, specifically for the 50-kt and 64-kt wind speed intervals. The decision thresholds ranged from 11% to 51%. In addition to the two key questions mentioned before, several other questions are addressed in this work. Does skill vary by time or wind speed category? Does the performance of the system vary by the type of TC intensification? Do the results vary by the TC approach? Does the system perform equally for geographical sub-regions? The full data set is segregated by the change of TC intensity (intensifying, no change, and weakening), time of the closest approach (before and after), and by the geographic regions Cape Canaveral in Central Florida (6 forecasting locations) and Antigua (12 stations) to test for performance differences. In addition to evaluating the interval probability (IP) forecasts this work also evaluates the skill of the cumulative probability (CP) product. The WPWF is evaluated for the Atlantic hurricane seasons for all 150 stations that are forecast for in the WPFP. The verification of the WPFP includes reliability diagrams, frequency distribution, as well as additional statistics designed to quantitatively measure the WPFP performance. The statistics are computed for each wind category ( 34-kt, 50-kt, 64-kt); 7 IP (0-12, 12-24, 24-36, 36-48, 48-2

20 72, 72-96, h), and 6 CP time intervals (0-24, 0-36, 0-48, 0-72, 0-96, h). In order to assess the impact of WPFP null forecasts (less than 0.5% probability) on performance evaluation, statistics are calculated for all intervals including varying numbers of null forecasts with the use of a buffer zone (Section 3.4) The statistical technique bootstrap re-sampling with replacement is used to estimate confidence intervals for each decision threshold and skill value. Also t- tests are conducted to test for difference between subsample populations. 3

21 Chapter 2 Data and Methods 2.1 NHC WPFP data When a TC is a potential threat to the United States or countries of the Atlantic basin the NHC issues a wind speed probability product (at least one city must have a forecast probability > 0%). The wind speed probabilities are based on errors during recent years in the official track and intensity forecasts issued by the NHC. Variability in tropical cyclone size (wind radii) is also incorporated into the location-specific probabilities (DeMaria 2007). There are two types of the WPFP, graphical and the text, in this work the latter is evaluated. The text product, for a single storm, includes for each city a wind category and probability forecast issued by the NHC. The full list of NHC locations is used in this work, and includes 142 U.S. and international cities in the Atlantic basins; and 8 points in the Gulf of Mexico (Fig. 2.1) for the hurricane seasons. The 2004 season had only 43 U.S. locations (Fig. 2.2) available and this data set was provided in a hind-cast mode. The hurricane center provides the WPFP every six hours for wind speeds of at least 34-kt (17 m/s), 50-kt (26 m/s), 64-kt (33 m/s) at intervals of 0-12, 12-24, 24-36, 36-48, 48-72, 72-96, hours, and cumulative time intervals 0-24, 0-36, 0-48, 0-72, 0-96, hours. Hereafter, the intervals are referred to by the end of the interval hour, for example 0-12 h will be 12 h. Table 2.1 shows an example of the WPFP text product for tropical storm Fay on August 17 th, Probabilities are given for each wind speed category and time interval (Table 2.1). Each time interval, except 0-12 h, consists of two columns. The left column is the interval probability (IP); it is the forecast of the event beginning during a specific time 4

22 interval. The right (parenthetical) column is the cumulative probability (CP); which is the summation of the interval probabilities. In this study, both the IP and CP probability for each storm from 2004 to 2009 hurricane season are evaluated. Table 2.1 Example of the text wind speed probability forecast for tropical storm Fay, 0300 UTC August 17 th, The interval probability (IP) is in the left column, cumulative (CP) is in the right in parenthesis WIND SPEED PROBABILITIES FOR SELECTED LOCATIONS FROM FROM FROM FROM FROM FROM FROM TIME 00Z SUN 12Z SUN 00Z MON 12Z MON 00Z TUE 00Z WED 00Z THU PERIODS TO TO TO TO TO TO TO 12Z SUN 00Z MON 12Z MON 00Z TUE 00Z WED 00Z THU 00Z FRI FORECAST HOUR (12) (24) (36) (48) (72) (96) (120) LOCATION KT MIAMI FL 34 X 1( 1) 13(14) 16(30) 13(43) 4(47) X(47) MIAMI FL 50 X X( X) 1( 1) 3( 4) 7(11) 1(12) 1(13) MIAMI FL 64 X X( X) X( X) X( X) 3( 3) 1( 4) X( 4) All forecast probabilities are given in percent, while X indicates probabilities between 0.0 and 0.5 %, the so called null forecasts. The skill of the forecast system can be influenced by null forecasts. For example, if all the null forecasts are included, it is possible to saturate the data set with correct negative (CN) forecasts (i.e., did not occur events that were correctly forecast, Table 2.2) (Doswell et al. 1990). However, if null forecasts are entirely excluded from the data set, some missed forecasts would be eliminated (i.e., event occurred but was not predicted). To mitigate this problem a buffer zone was tested and implemented in which all of the null forecasts are retained (see Sections 2.8 and 3.4). 5

23 Figure 2.1 Location of the 150 stations evaluated for the years 2005 to Figure 2.2 Location of the 43 stations evaluated for the 2004 year. 6

24 Table 2.2 Classifications assigned to the probability forecasts. Classification Definition Hit Event Forecast to Occur, Event Occurred Miss Event Forecast Not to Occur, Event Occurred False Alarm (FA) Event Forecast to Occur, Event Did Not Occur Correct Negative (CN) Event Forecast Not to Occur, Event Did Not Occur 2.2 HURREVAC The HURREVAC (FEMA 1995; Sea Island Software Inc. 2006) is a GIS hurricane decision assistance program for emergency managers (referred to as a Hurricane Evacuation). Here this software is used to verify the WPFP. To maintain consistency with previous studies (Shafer et al. 2007; Shafer 2008; Splitt et al. 2009) the HURREVAC radii for 34 -, 50-, and 64-kt wind speed for each storm advisory from 2004 to 2009 hurricane season were used. The wind speed radii in the HURREVAC are based on information from the NHC. The main difference in the verification procedures from the previous work is that the verification wind radii from HURREVAC were exported into shape files for objective rather than manual assessment. Using the Interactive Data Language (IDL) all locations from the NHC station list are tested to see if they fall within each wind speed radii for a given advisory. This is done by comparison of each station s geographical coordinates with the wind speed radii position. To identify if the event occurred, for each advisory time interval to each location assigned a number: 0 if it was out of the shape file area; 1 if located within a 34-kt radius (Miami, Fig. 2.3); 2 if located within 50-kt (Fort Pierce, Fig. 2.3); and 3 if it was within the 64-kt shape file area (Palm Beach, Fig. 2.3). The use of shape files helped prevent making visual classification mistakes. The IP verification is different for the first forecast interval (the 12h forecast), because each new issued WPFP advisory does not consider the past 7

25 conditions (i.e. starts fresh each forecast cycle). The 12h forecast interval verification is based on the occurrence of the TC during this interval, i.e. even if the on-set of the wind speed categories at some location happened before the current advisory. For all other time intervals the IP product verifies as a hit only if the forecast wind condition started during the forecast interval. If the condition onsets before or after the interval, the forecast would be considered a miss. Figure 2.3 Example of HURREVAC wind speed radii for hurricane Frances, September 4 th, 2004 at 1800 UTC. 2.3 Reliability and frequency distributions The reliability diagram is a common diagnostic graph used to summarize and evaluate probabilistic forecasts (Wilks 2006). The reliability diagram consists of a plot of the observed relative frequency on the Y axis against the predicted 8

26 probability X axis (Fig. 2.4). If the curve connecting these points is close to the prefect reliability line (a line of slope 1 and intercept 0), the forecast system is considered to be reliable. For over-forecasting systems the empirical reliability line lays under the perfect reliability line, opposite is true for the under-forecasting systems. In this study the reliability diagrams are plotted using an interval bin width of 10% to help increase the statistical significance of the sub-samples. The forecast probabilities are defined using a bin mean based on the data distribution rather than the bin center. Frequency distributions (FD), also known as sharpness diagrams (Jolliffe and Stephenson 2003), are constructed in order to help assess if the estimates of the reliability are robust. 9

27 Figure 2.4 Examples of reliability diagram performance. (Figure from Daniel S. Wilks Statistical Methods in the Atmospheric Sciences ) 2.4 Attribute diagram Reliability is only one of the desired properties of a probabilistic forecast system. Other relevant attributes include skill and resolution. The attribute diagram, a modification of the reliability diagram, includes assessments of these aspects (Hsu & Murphy 1986). 10

28 All attribute diagrams in this research are obtained by using R, a functional language and environment to statistically explore data sets. In particular, the Verification package is used which allows verifying discrete, continuous, probabilistic forecasts and forecast expressed as parametric distributions. As with the reliability diagram, the attribute diagram is a two dimensional Cartesian coordinate system with forecast probability as the abscissa and the observed relative frequency as the ordinate. Figure 2.5 is an example of an attribute diagram. As with the reliability diagram, the diagonal line represents a perfectly reliable probability forecast, when forecast and observation frequencies are equal. Geometrically, the closer each sub-sample point located to the ideal line the better forecast probability. A relative frequency numbers is plotted on the diagram by each sub-sample which tells about the distribution of forecasts. The horizontal dashed line represents no resolution, when the subsample relative frequency is equal to overall sample frequency. One can think of this line as the climatological frequency of occurrence, but in this study this only represents a climate frequency for when TCs are present in the basin. The no skill line is equidistant between the diagonal perfect reliability line and the horizontal no resolution line. If any empirical point lies below this line the Brier skill score is negative. The shaded area represents positive skill. Also, each subsample point has 95% confidence intervals drawn calculated using a boot strap replacement method. 11

29 Observed relative frequency, o Attribute Diagram Under-forecast No skill Over-forecast No resolution Forecast Probability Forecast probability, y i Figure 2.5 Example of an attribute diagram 2.5 Verification statistics An indirect method of probability forecast evaluation is used to calculate an array of other verification statistics. In this method the probability forecast is converted to a binary (yes/no) forecast by use of a forecast threshold and the observation is already binary (occurred/did not occur). First, the WPFP probabilities and observational data (see Section 2.2) for each forecast location and advisory time interval are classified via the contingency table (Table 2.3) as a hit, 12

30 miss, false alarm (FA), and correct negative (CN, Table 2.2). The classification depends on the threshold which is used to determine whether or not an event is forecast to occur and whether the event was observed to occur using HURREVAC. Table 2.3 Contingency for classification of probability forecasts. False alarm (FA), correct negative (CN). Contingency tables are calculated for both IP and CP probabilities (from 1% to 100%) in 1% increments for the hurricane seasons. To verify performance of the WPFP the following verification statistics were calculated: probability of detection (PoD), probability of false detection (PoFD), false alarm ratio (FAR), threat score (TS), Heidke skill score (HSS), true skill statistic (TSS) and bias score (Table 2.4). A weakness with the accuracy, PoD, PoFD, FAR and TS statistics is that one can get favorable skill by intentionally forecasting a certain category of the contingency table (in another words cheat the system ). For example, to get a high value of accuracy one can easily increase the number of Correct Negatives (CNs) by forecasting only low probability of strong wind occurrence. Accuracy does not take into account Misses or Falls Alarms (FA). One advantage of the TSS and HSS skill scores is that they take into account all categories of contingency table, and are hence more robust. The decision threshold 13

31 is chosen as a threshold which represents the highest value of Heidke skill score (HSS-based) or true skill statistic (TSS-based). Table 2.4 Verification statistics. Statistic Formula Range Definition 0 to 1; Fraction of events that were Accuracy (Hits + CN) / Total Perfect is 1 correctly forecast Probability of 0 to 1; Fraction of observed events that Hits / (Hits + Misses) Detection Perfect is 1 were correctly forecast Probability of 0 to 1; A measure of the product s Falls FA / (CN + FA) Perfect is 0 ability to forecast non-events Detection False Alarm Ratio Threat Score Bias Score True Skill Statistic Heidke Skill Score FA / (Hits + FA) Hits / (Hits + Misses + FA) (Hits + FA) / (Hits + Misses) PoD PoFD 2C ratio (PoD-PoFD)/ (((C ratio *PoFD)+PoD) (C ratio -1)+ C ratio +1) Where, C ratio =Total Observed No/ Total Observed Yes 0 to 1; Perfect is 0 0 to 1; Perfect is 1 0 to Infinity; Perfect is 1-1 to 1; Perfect is 1, 0 indicates no skill -1 to 1; Perfect is 1, 0 indicates no skill A measure of the product s ability to forecast events A measure of how well the forecast Yes events correspond to the observed events Indicates if the product under forecasts (Bias <1) or over forecasts (Bias >1) A measure of how well the product distinguishes observed events from non-observed events Presents the skill as a percentage improvement over the skill expected due to random chance An example of these verification statistics for the IP data set, including the null forecasts, is shown in Figure 2.6. Some of the statistics, such as TS, TSS and HSS, have easily identifiable maximum values (asterisks) which can be used as a decision threshold. Other statistics (Accuracy, PoD, PoFD, and FAR) are not useful because they always have a maximum at the 1% or 100% threshold. 14

32 Statistic's value Decision Thresholds Probability forecast, % Accuracy PoD PoFD FAR TS TSS HSS Figure 2.6 Verification statistics for IP (including null forecasts), for all wind categories and forecast time intervals for the TC seasons. (maximum statistic s value marked by the asterisk) 2.6 Decision thresholds The maximum values of the verification statistics for a given subsample are used to choose decision thresholds. As it is mentioned above (see Section 2.5), the verification statistics are calculated for each threshold from 1% to 100%. Decision thresholds are the threshold that produces a maximum skill in one of the verification statistics. In this work maximum values of the TSS or HSS are used to determine decision thresholds for each wind speed category and forecast time interval (see Section 3.3). These will be referred to as the TSS method and the HSS method hereafter, or as TSS-based and HSS-based derived statistics. 15

33 2.7 Confidence intervals and t-test Bootstrap re-sampling, a widely used statistical technique (Efron 1982; Efron 1987; Mooney 1993; Simon 1997), is applied to obtain confidence interval estimates for the TSS- and HSS-based thresholds and skills. Randomly choosing and recording paired (forecast and observation) data from the original sample with replacement, a new data set is constructed. The total amount of pairs in the new random data was set to equal the original. TSS and HSS are calculated for each threshold from 1 to 100% for the sample population and decision thresholds and skills are found (using the same procedures as outlines in Sections 2.5 and 2.6). After repeating the resampling procedure 100 times, skills and thresholds are sorted in increasing order. Using the 2.5 and the 97.5 percentiles of the bootstrap distribution, the limits of the 95% confidence intervals are obtained. Figure 2.7 shows 95% confidence intervals for IP HSS-based thresholds for 50-kt wind speed and 7 time intervals. The decision thresholds are indicated by the small circles. Note that segments of the 12h and 24h skill curves become relatively flat and the confidence intervals for these time intervals are wider than for the latter intervals. As forecast time increases, the skill curves extremes become more definitive and the confidence intervals shrink. Sides of the confidence intervals are not always equally extended from the decision threshold point (which is defined by the maximum skill), the mean values (red asterisks, Fig. 2.7) for each confidence interval (of 100 random data sets) are calculated to find more statistically relevant decision thresholds. The mean threshold values and standard deviations obtained from the bootstrap re-sampling are used for significance testing between forecast time and speed categories. The t-test is used to test the null hypothesis between two samples at the 5% significance level. An unpaired or independent samples t-test is 16

34 HSS value applied to separate, independent and identically distributed samples, i.e. to make sure that samples are non-overlapping (Wilks 2006) Threshold, % 12h 24h 36h 48h 72h 96h 120h Figure 2.7 HSS-based IP, with all null forecasts excluded for 50-kt wind speed and all 7 time intervals with 95% confidence intervals. (Decision thresholds are colored circles. Red asterisks are confidence mean.) 2.8 Null forecasts and the use of a buffer zone Forecasts of less than 0.5% probability are considered here as null forecasts (X, Table 2.1) for any location, wind speed or forecast interval. Null forecasts can overwhelm verifications measures of forecast systems (Doswell et al. 1990) and thus influence some of the verification statistics, including those that are used for converting the probability forecast into a binary forecast. The buffer zone is used to filter null forecasts and to keep Misses (Miss, Table 2.2) in the data sets that are related to null forecasts. The sensitivity of the 17

35 various statistics [bias score, true skill statistic (TSS), and Heidke skill score (HSS)] to the size of the buffer zone (i.e. the number of null forecasts in the data) is evaluated. The buffer radius is systematically changed from 100 to 500 kilometers with interval of 100 km. If, for any particular location, the forecast probability was greater than 0.5% all surrounding stations with null forecast within the buffer zone from that location were included (Fig. 2.8). If a station was out of all buffer zones, it was excluded from the data set. Using the bias score, TSS and HSS methods the respective decision thresholds were calculated for each data set (see Section 3.5). Figure 2.8 Example of a 100 km radius buffer zone. Null forecasts within the buffer zone are retained for statistical evaluation of the WPFP. 18

36 2.9 TC Intensity For the last couple of decades improvement of the TC track forecast has been a success (DeMaria 2007) and since 2003 the NHC has extended the track forecast from 72h to 120h. Despite the improvement in TC track forecasts, the intensity forecast improvement was not significant, and according to DeMaria (2007) the track forecast improvement is an order of magnitude larger than that for the intensity forecast. With that being said one might expect a difference in the performance of the WPFP for different TC stages such as the intensity trend. The WPFP data set was divided into three groups based on the type of TC intensity change: increasing, no trend (same) and weakening. Change in the intensity is determinate by calculating the backward difference in the maximum sustained wind between two consecutive advisories. If the difference is greater/less than 5-kt the storm intensity was categorized as increasing/decreasing, Absolute differences less than or equal to 5-kt are categorized as no trend. The described criteria applied to all forecasts from the particular advisory. All first advisories of TC are categorized as increasing intensity. Then, each group of TC trend forecasts is evaluated with use of attribute diagrams and the calculation of decision thresholds and verification skills (see Section 3.7) Closest approach of TC The closest approach of a TC to a given verification location is defined as a time when the distance between the center of the TC and each station is minimal. The forecast system performance might be expected to be different up to the time of closest approach and after. For example, TC re-curvature (often defining a closest approach) may indicate shear impacting the TC and perhaps affect WPFP performance. 19

37 To achieve the answers for these questions, the distance between the center of the TC and each evaluating station for each TC (for the 2004 to 2009 hurricane seasons) is calculated. The time when the TC distance is minimal to a location is considered to be the closest approach for the evaluated station. This time was used to break the dataset into two subsets, before and after the closest approach. The results of statistical tests are presented in Section Cape Canaveral and Antigua regions The Cape Canaveral area in East Central Florida and the Antigua region are areas of special interest for the 45WS. The Kennedy Space center, Patrick Air base and other companies which receive a meteorological support that are located in the East Central Florida; the tracking radar in Antigua provides data to support launches. Evaluation of the WPFP for these regions and comparison with the full data set results might give some insights for these decision makers and help them in the practical application of NHC s wind probability forecasts. For the Cape Canaveral area 6 stations (Jacksonville, Daytona Beach, Cocoa Beach, Fort Pierce, West Palm Beach and Miami, Fig. 2.9) along the Florida s Atlantic coast are chosen to be consistent to the past studies (Shafer et al. 2007; Shafer 2008; Splitt et al. 2009). The results of the previous research show that IP forecasts tend to over-forecast and CP tends to under-forecast. The decision thresholds range from 2% to 51% depending on the wind speed and forecast time interval. 12 stations represent the Antigua region (St. Maarten, Barbuda, Antigua, St. Kitts-Nevis, Guadeloupe, Aves, Dominica, Martinique, Saint Lucia, Barbados, St. Vincent and Grenada, Fig 2.10). Attribute diagrams, decision thresholds and skills statistics for these regions are generated and compared to the full data sets (Sections 3.9 and 3.10). 20

38 Figure 2.9 Stations used for WPFP evaluation of Cape Canaveral area Figure 2.10 Stations used for WPFP evaluation of Antigua area 21

39 Chapter 3 Results Results of the WPFP forecast verification are presented. Sections 3.1 and 3.2 include the results of direct evaluation of the WPFP system (reliability and attribute diagrams). Sections 3.3 and 3.4 provide indirect evaluation results, obtained by using thresholds and resultant statistical skills. The issue of null forecasts is addressed in Section 3.5, where data are filtered a varying size buffer zone. The maximum forecast probabilities by the WPFP for specific time and wind category intervals provide context for the decision thresholds and are discussed in Section 3.6. The performance of the WPFP system is also evaluated by segregating data based on the TC intensity trend and is discussed in Section 3.7. Section 3.8 is devoted to the subject of the TC closest approach. The point of Sections 3.9 and 3.10 is to discover differences or similarities in the wind speed probability forecasts for the Cape Canaveral and Antigua geographic sub-regions, respectively, in comparison with full data set. 3.1 Reliability and frequency A forecast which is perfectly reliable by definition will result in a forecast frequency equal to an observed frequency (Hartmann et al. 2002). Here, the reliability diagrams are constructed for both IP and CP forecast products. The diagram is also a convenient tool to examine whether the WPFP contains bias. Figure 3.1 (a) shows the IP reliability for the full data set ( ) including all forecast time intervals and wind speed categories, but excluding all null forecasts. The associated frequency diagram (Fig. 3.1 b) depicts sample size versus 22

40 probability. The frequency diagram is a rough measure of the statistical significance of the reliability curve. The reliability of the IP WPFP shows overforecasting at probabilities of less than 60%; nearly perfect reliability between 61% and 80%, and under-forecasting at probabilities between 81% - 90% and perfect reliability at probabilities greater than 91%. The % sub-sample point lies on the perfect line. Similarly, Figures 3.2 (a, b) shows CP reliability and frequency diagrams for the WPFP. The reliability curve for the CP product (Fig. 3.2 a) shows overforecasting between 11% and 20%; nearly perfect reliability from 21% to 50%; under-forecasting between 51% - 80%, and lower resolution for probabilities greater than 91%. Overall, both interval and cumulative forecast systems are reliable with poor resolution for CP forecast probabilities greater than 91%. 23

41 Number of forests Observed frequency Perfect Reliability Probability bin, % Probability (%) Figure 3.1 IP (a) reliability diagram and associated (b) frequency distribution of full data set using all wind speed categories and forecast time intervals. 24

42 Number of forests Observed frequency Perfect Reliability Probability bin, % Probability (%) Figure 3.2 CP (a) reliability diagram and associated (b) frequency distribution of full data set using all wind speed categories and forecast time intervals 25

43 3.2 Attribute diagrams The attribute diagram in R allows for evaluation of other characteristics of a probability forecast, such as skill and confidence intervals of the reliability curve. Figure 3.3 is an attribute diagram of IP data including all wind speed and time intervals, but excluded all null forecasts. Observed relative frequency, o Sample Attribute size Diagram = 201,881 No skill No resolution Forecast probability, Probability y i Figure 3.3 IP attribute diagram with all wind speed categories and forecast time intervals included (all null forecasts excluded). 26

44 That attribute diagram indicates that for the most part the reliability curve is located in the shaded (positive skill) area, except for the 1%-10% forecast probability bin which is located very close to the climate occurrence and the 11%- 20% bin which lays right on the no skill line (Brier skill = 0). More than 87% of total number of forecasts is for forecast probabilities under 10%. Observed relative frequency, o Sample Attribute size Diagram = 284,029 No skill No resolution Forecast Probability Forecast probability, y i Figure 3.4 CP attribute diagram with all wind speed categories and forecast time intervals included (all null forecasts excluded). The attribute diagram indicates (Fig. 3.4) skill in the CP WPFP forecasts with only the 11% - 20% subsample point indicating negative Brier skill, but it is 27

45 noted that it is located very close to the climate occurrence point which is hard to beat in performance at that forecast probability. Both attribute diagrams indicate good reliability and positive skill of IP and CP WPFP with slight under- or over-forecasting. The resolution is good except for the 91%-100% forecast bin for the cumulative probabilities. 3.3 Heidke skill score and true skill statistic Two commonly used verification statistics, the HSS and TSS, are used to evaluate skill in the NHC forecast product. The TSS, sometimes referred to as the Hanssen-Kuiper skill score, assesses whether a forecast system (particularly a rare events) (Wilks 2006). In our case, TSS indicates how well the WPFP distinguishes observed events from non-observed events (Flueck 1987) while the HSS is a measure of the improvement over the skill expected due to random chance (Stephenson 2000). Both vary between -1 and +1, with 0 indicating no skill and +1 perfect skill (Table 2.4). Table 3.1 TSS- and HSS-based decision thresholds (%) and skill. (left numbers based on maximum skill, right on the mean bootstrap value) Statistic TSS HSS Range -1 to 1-1 to 1 Perfect 1 1 Data Set Threshold (%) Skill Threshold (%) Skill IP excluded null forecasts 7/ / IP included null forecasts 4/ / CP excluded null forecasts 20/ / CP included null forecasts 13/ / To evaluate the impact of including and excluding null forecasts from the full data set, verification statistics are calculated for all forecast time intervals and 28