North Carolina State University Land Grant University founded in undergraduate fields of study 80 master's degree areas 51 doctoral degree

Transcription

1

2

3 NC State University North Carolina State University Land Grant University founded in undergraduate fields of study 80 master's degree areas 51 doctoral degree programs Enrollment is 33,819 students (~19% graduate students) Raleigh is the state capital (population ~390,000) Research Triangle Park 3 Corners: NCSU, Duke, and UNC

4 35 faculty; 100 grad students; 240 undergrads Only Ph.D. atmospheric sciences program in the Carolinas One of largest interdisciplinary environmental sciences departments in U.S. High level of sponsored research (~$6M/year) Faculty participation in IPCC assessment.

5

6

7 Lorenz Attractor Slight change in ICs ends up in a same wing target Prediction event Lorenz Attractor

8 Methods for Estimating Quality of Ensemble Climate Prediction Systems Root Mean Square Error (RMSE) Brier Skill Score (BSS) Heidke Skill Score (SS) Kuipers Skill Score (KS) Relative Operating Characteristics (ROC)

9 Criteria for making decision The user of an ensemble of (N) forecasts has (N) options for use as a decision criteria with respect to his/her climate-related action. (i) He or she can chose to take action only if all (N) forecasts predict the adverse climate (ii) He or she can chose to take action only if at least (N-1), (n-2),, forecasts predict the adverse climate, OR (iii) He or she can chose to take action even if only one member predicts the adverse climate

10 Quality of Ensemble Climate Prediction Systems Root Mean Square Error (RMSE) Brier Skill Score (BSS) Heidke Skill Score (SS) Kuipers Skill Score (KS) Relative Operating Characteristics (ROC) also recommended by WMO

11 Decision to issue a forecast of (E) to occur is probabilistically based on the criteria: (N): size of the ensemble (n): number of the runs in the ensemble for which (E) actually occurs (p): probability given by the ratio (n/n) this is the threshold fraction above which the event (E) is predicted to occur based on the model forecast

12 What do these variables mean? ROC and other skill evaluation tools rely on identifying a particular event (E, e.g. high/low precipitation events) Historical record of E as forecasted by an EPS Observational record of E

13 Forecast-Model Contingence Matrix A decision maker becomes a user of weather forecasts if he/she alters his/her actions based on forecast information Over a large sample of independent forecasts we construct a 2x2 matrix in which we evaluate the skill of a probabilistic forecast EPS Forecast No Yes Observations No No cost ( ) False Alarm (γ) Yes Miss (β) Hit (δ) Hit Rate: H pt = δ/(β +δ) False Alarm Rate: F pt = γ/(γ + )

14 Hit Rate & False Alarm Hit rate = δ/(β+δ) observations forecast i.e. fraction of correct forecasts to total observed occurrence False alarm = γ/(γ+ ) i.e. fraction of cases when E was predicted to occur but did not occur of the total number of cases when the event E was observed not to occur

15 Quality of Ensemble Climate Prediction Systems

16 Relative Operating Characteristics (ROC) RFor each there is a corresponding pair of H(t) & F(t) which may be plotted on a ROC plot +1 Hit rate * *** *** False Alarm ratio 1. No relative skill i.e. climatology H=F, the chances for a Hit are as good as the chances for a False Alarm. 2. Total area under curve is usually used to provide a measure of the skill of the model to predict event E. 3. For a skillful forecast: the hit rate is >the false alarm. +1

17 Relative Operating Characteristics (ROC) Relative operating area (ROC area) is one of the summary measure of ensemble forecast performance ROC area (ROCA) is defined by the points (0,0), (1,1), and the points representing the forecast system

18 Relative Operating Characteristics (ROC) The closer a curve is to the upper-left-hand corner, the more skillful is the forecast system A perfect forecast system would have a ROCA of 1 A system with no capability of distinguishing in advance between different climate events has a score of 0.5, i.e. lying on the diagonal defined by (0,0) and (1,1)

19 Estimation of Economic Value (Development of Methodology) Based on trade-off between cost in taking action by the user and the loss when no action is taken & the event occurs Let $C = cost of protection (See examples) Let $L = Loss for not protecting when E occurs Occurrence of Event No Yes EXPENSE No O L CONTINGENCY Action Yes C C MATRIX User to take action that will minimize expense over a large number of cases Occurrence of Event No Yes No O βl USER Action Yes Cγ Cδ EXPENSE MATRIX

20 Observations EPS Forecast No Yes No No cost ( ) False Alarm (Cγ) Yes Miss (Lβ) Hit (Cδ) We can now represent the expected user expense using the variables with the following equation: (3)

21 From the relationships created by (3), we can develop and derive further associations among the variables presented in the tables Over the course of time, the climatological occurrence of the event becomes: (4) The definition of also allows us to derive further identities: (5) For eq. 6 to be true, # of events has to be sufficiently large, End user can now make informed decisions concerning impending expenses. (6)

22 Using eq. 1 and eq. 4 we find that: so that, when multiplying both sides by, we get: Cost of E occurring equals climatology multiplied by the hit rate (H) Similarly, employing eq. 2 and the identity in eq. 6, we find that: (7) when variables are rearranged, we get the following formula: this identity allows us to gauge the interconnection between F and (8)

23 Using equations 5 and 7 and substituting δ with, we find: which allows us to represent β as the hit rate-climatology relationship subtracted from climatology itself in order to analyze the loses due to errant forecasts relative to the climatology. (9)

24 Now that we have derived all variables in terms of their relationship with (H, F, ) the average frequency of the event,, we rewrite (3) as: where the Ls cancel out and the equation reduces to The C/L derived here now represents a hypothetical user for which there is a particular C/L ratio that they are willing to work with when utilizing EPS forecasts.

25 Redistributing the C/L we get: which becomes: (10) Substituting µ for C/L we can rewrite our equation as follows: (11) This is equation (4) in Semazzi and Mera (2006) and defines the value of an EPS for a range of mitigation options (or users) µ.

26 One special case of the M equation includes the perfect forecast, where Hpt = 1 and Fpt = 0. Thus, (11) becomes: Which becomes: (12)

27 Another special case for (11) occurs when we define the value for climatology, so that M has two possible outcomes: One is H = 0 and F = 0, when the user never takes action and the other occurs when H = F = 1 (when a user always takes action). If all you have is climatology then there is no basis for a user to change take action to take action and vice versa. When H = F = 0 we get: Which reduces to: (13) In the case where H = F = 1, we get: Which reduces to: (14)

28 Special Case: Perfect Prediction Model Perfect Forecast H=1 & F=0

29 Special Case Climatology: Poor Man s Prediction Model Only Climatology is known (Has Important Economic Benefits) Option 1: Always Take Action (H=F=1) Option 2: Never take action (H=0, F=0)

30 Option 1: Always Take Action [H=F=1] ` O observed N Y Forecast N 0 0 Y γ δ Always forecasts E to occur but it is equally likely that E will not occur

31 F Option 2: Never take action (H=0, F=0) O N Y N β Y O O Comes from Always take action Comes from never take action If L very large If C very small If L is very small

32 NOTE

33 Formula for Computing Value Define as reduction of M over normalized by maximum possible reduction I.e. so that O<V<1 V=0 for Climatology V=1 for perfect prediction model

34 Thus, we have arrived at two possible outcomes of M cli, which is defined by the hypothetical user. We now have the ability to define the value of an EPS forecast for a hypothetical user by utilizing the following formula: (15) This formula can be employed for every point in the ROC graph, that is, every Hpt and Fpt have a corresponding value curve on the EV plot and using eq. (15) and combining it with (11), (12), (13) and (14) we define specific Vpt for every H and F on the ROC figure: (16)

35 Assessment of Economic Benefits of Cli.Prediction USER SECTOR MODEL HINDCAST MET. OBS Define (E) Identity C & L Forecast (E) Specify (Pt) Obs (E) Compute OCCURANCE Fst No β Yes γ δ No Yes Region 1 Region 2 Region 9 ROC (Perfect) H F (Climatology) See example figure See example figure DECISION IF $ IS IMPORTANT TO SECTOR

36 The envelope curve may be obtained by plotting V against µ for discrete values of pt appropriately covering the possible range of 0 1. Palmer et al. (2000) have shown that the end user with should expect the maximum value V=Vmax. This relationship allows us to construct EV

37

38

39 Required Input from User Sectors: Identification of Important Climatic Event e.g. Temperature below which crop fails. E Identification of Loss L$ which occurs when no action is taken to prevent loss. Identification of Cost C$ required to prevent loss. User Benefits: Level I: Criteria either always action or never to take action when only climatology is known. Level II: Estimate of $ saved over using only climatology information Model Development Benefits: To Access/Identify model improvements which result in added value Model Intercomparison

40 Summary Diagrams of yields estimates of Eco. Ben Methodology provide means of determining when model has achieved usable range. ROC>1/2 not adequate if C/L is out of range. Ensemble approach provides objective way for combining probabilistic forecasts from different model Method is ideal for investigating added value due to downscaling based on regional climate model Application of the knowledge of climatology alone has economic benefits User need to be prepared to rely on forecast for V Based on Based on Based on Optimal (C/L) is given by. Therefore qualitative potential benefits of climate prediction can be assessed by comparing (C/L) & even before using model output. Specific sector could explore adjusting (C/L) to increased V. C/L

41 Main Conclusion The evaluation may be customized for a specific end-user application and the results are in a form that is understandable by the user & the producer of the forecasts

42 The EROC Method The traditional ROC method is widely used and indeed highly recommended by WMO. Therefore, it is more desirable to express Relative Economic Value in terms of ROC metrics if at all possible

43 The EROC Method EROC merges the information for a range of hypothetical users within the ROC plot itself. Using (16), we recognize that the V will change for each µ and that a non-optimal user would shift from its position at H = F (climatology). It is desirable for a user to set a threshold of the minimum added value that make it worth investing in an EPS. Thus, we find that V >V min and we rewrite (16): (17)

44 In order to construct our baselines where Hpt is a function of Fpt we divide both sides of (17) by and re-arranging we obtain: Further dividing both sides by we get: (18) We wish to construct mathematical forms where H is a linear function of F, thus be able to apply similar interpretation of ROC

45 If (18) becomes: With further simplification yields, (19)

46 If (18) the function becomes: where the (20)

47 Note that both (19) and (20) have (i.e a function of F) in common and constant term (i.e not a function of F). The two equations are reproduced below for convenience: (19) (20) Thus, we can combine (19) and (20) into one equation that defines H as an inequality function of F: (21)

48 where and for for (22) (23) We note that H=F is a special case of (21), when and V min =0. Therefore the traditional ROC condition for using the condition as a metric for measuring the skill on an EPS is a very special case of a user where.

49 Bounds for Our µ range is thus bounded by the two inequalities so that: (24) Outside this range the EPS has no value for the user µ

50 EROC & EV Relationship We now know that ROC is a special case of EROC for an optimal user. We found that when,, (21) reduces to H = F, or the H=F diagonal in ROC plots. We generalize this by the following equation: (25) so that all baselines can be derived from (25) as a function of a particular µ. In the case of, (25) reduces to: or simply, climatology

51 Let us now calculate a baseline given a specific µ where H is now the point corresponding to the baseline and also a function of F. To obtain the difference between ROC and the new baseline we set: (26) where H is the baseline. Combining (25) with (26), we get Again the ideal user: (27) (28)

52 Combining the equation for V in (16) and in (27) yields: for for (29)

53 The Semazzi-Mera Skill Score We have shown that there is one-to-one relationship between EROC and EV. We also showed that the base line H=F, common to the widely-used ROC plots, is a special case of (21), and as thus, a special case of EROC. Another important use of ROC in EPS skill measurement is the Area Skill Score (ASS) or ROC Skill Score (RSS), as shown in Richardson (2000a) and references therein: (30)

54 The area A under the ROC is used as an index of the accuracy of the forecast system (Mason 1982; Buizza et al. 1998, 1999). A perfect system would have A = 1.0, while no-skill systems (H = F) would have A = 0.5. Substituting A per =1, and A clim =0.5, we have Which reduces to: (31) The skill score as determined by Richardson (2000a), Stanski, and Wilks (2006), corresponds to the optimal user where M clim in the equation for V is given by H = F. Adopting a more general baseline the Semazzi-Mera Skill Score (SMSS) is given by: (32)

55 In SMSS where A is the area under the ROC curve, A per = 1, and Aµ is the appropriate area between the baseline and the ROC graph. In this case, Aµ is determined by the µ. It is the climatological baseline for a hypothetical user. (33)

56 The Application Meningitis is a serious infectious disease affecting 21 countries 300 million people at risk 700,000 cases in the past 10 years % case fatality rates

57 Meningitis-Climate link Outbreaks coincide with dry, dusty conditions over the Sahel due to the Harmattan winds Largest correlation occurs between humidity and disease ( 2003 al., outbreaks (Molesworth et Disease occurrence drops dramatically with the onset of humidity SH L SH L January July

58 Our Study: Predictability of Atmospheric Moisture What are the variables important for the prediction of the moisture regime? What are the sources of moisture in West Africa during the Boreal Spring? Can regional climate models forecast moisture transport dynamics?

59 The Variables Variables related to the West Africa monsoon have the highest correlation with meningitis We use relative humidity (RH) as an indicator in our study From Yaka et al (2008)

60 Intraseasonal Variability Kano Short-term events

61 Ensemble Dynamical Downscaling Analysis of added value of dynamical downscaling We use the Weather Research and Forecasting WRF Model as both a predictive and analytical tool for High resolution reanalysis Real-time forecasts Sensitivity experiments Comparison with large-scale models Scale of relevance

62 Advantages of Dynamical Downscaling Ghana Ghana WRF at 30km resolution NCEP/NCAR Reanalysis at 2.5

63 Advantages of Dynamical Downscaling Kano

64 Lessons

65 References Richardson, D. S., 2001: Measures of skill and value of ensemble prediction systems, their interrrelationship and the effect of ensemble size. Quart. J. Rol. Meteor. Soc., 127, Zhu et al (2002): The economic value of ensemble-based weather forecasts. BAMS, January 2002, Semazzi, F.H.M., and R. Mera, 2006: An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method. J. Appl. Meteor. Clim., 45, Publications cited in Richardson (2001), Zhu(2002), & Semazzi and Mera (2006).

66 More Information Updates on our work: twitter.com/climlab

67 Acknowledgements