BAYESIAN APPROACH TO DECISION MAKING USING ENSEMBLE WEATHER FORECASTS

Transcription

1 BAYESIAN APPROACH TO DECISION MAKING USING ENSEMBLE WEATHER FORECASTS Rick Katz Institute for Study of Society and Environment National Center for Atmospheric Research Paper: R.W. Katz & M. Ehrendorfer, 2006: Wea. Fore., 21, ( Talk:

2 Quote -- Lao Tzu, Chinese Philosopher He who knows does not predict. He who predicts does not know.

3 Outline (1) Background (2) Bayesian Approach (3) Skill of Ensemble Forecasts (4) Value of Ensemble Forecasts (5) Discussion

4 (1) Background Ensemble Forecasts -- How to produce probability forecasts (take at face value )? Skill of Ensemble-based Probability Forecasts -- Effect of ensemble size on reliability & Brier score Value of Ensemble-based Probability forecasts -- Effect of ensemble size on economic value

5 (2) Bayesian Approach (2.0) Bayesian Inference (2.1) Bayesian Model for Bernoulli Process (2.2) Bayesian Ensemble Probability Forecasts (2.3) Probability Model for Ensemble Generation

6 Quote -- I. J. Good, British/American statistician (1973) The subjectivist states his judgements, whereas the objectivist sweeps them under the carpet by calling assumptions knowledge, and he basks in the glorious objectivity of science.

7 (2.0) Bayesian Inference Alternative to Classical/Frequentist Approach -- Incorporate prior information Combination of different sources of information -- Interpretation Tests of significance, Confidence intervals -- Computations Markov Chain Monte Carlo

8 (2.1) Bayesian Model for Bernoulli Process Bernoulli Process -- Sequence of n Bernoulli trials p probability of event occurrence on given trial Assume trials are (conditionally) independent given p -- Prior distribution Probability density function f(p) Assume p has beta distribution with parameters r = r 0, s = s 0

9 -- Likelihood function L(x 1, x 2,..., x n p) where x 1, x 2,..., x n denotes data Let k denote no. of events that occur (out of n trials), 0 k n Given p, distribution of k is binomial with parameters n, p -- Posterior distribution Probability density function g(p x 1, x 2,..., x n ) Bayes s Theorem: g(p x 1, x 2,..., x n ) % L(x 1, x 2,..., x n p) f(p)

10 Given prior beta(r 0, s 0 ) & data k, conditional distribution of p is beta with parameters: r = r 0 + k, s = s 0 + n k Posterior mean: E(p x 1, x 2,..., x n ) = (r 0 + k) / (r 0 + s 0 + n) Differs from classical/frequentist estimate: p^ = k/n

11 Prior & Posterior Distributions: k = 0, n = Probability density function Prior(1, 1) Posterior(1, 11) Probability

12 (2.2) Bayesian Ensemble Probability Forecasts Ensemble Forecasting -- Notation n is ensemble size k is number of ensembles for which event occurs p is actual forecast probability of event Obtain from infinitely many ensembles (from perfect numerical weather prediction model)

13 -- View weather event as a single future Bernoulli trial Posterior predictive probability is posterior mean: p^ = (r 0 + k) / (r 0 + s 0 + n) p^ denotes Bayesian forecast probability estimator

14 Bayesian Interpretation of Existing Schemes -- Face value p^ = k/n r 0 = 0, s 0 = 0 Only plausible for perfect forecasting system (not just perfect model) -- Fictitious ensemble p^ = (k + 0.5) / (n + 1) r 0 = 0.5, s 0 = 0.5 Only plausible for forecasting system with moderate skill

15 Probability density function Prior Beta Distributions Event probability r = 1/2, s = 1/2 r = 1, s = 1 r = 0, s = 0 r = 10, s = 20

16 (2.3) Probability Model for Ensemble Generation Ensemble Forecasting -- Assumptions Numerical weather prediction model is perfect Generates probability forecasts from beta distribution with parameters: r = r, s = s

17 -- Stochastic simulation of ensembles (i) Generate actual forecast probability p from beta distribution(r, s ) (ii) Given p, generate n ensembles (n Bernoulli trials, parameter p), say X 1, X 2,..., X n (iii) Independently of X 1, X 2,..., X n generate weather observation as additional Bernoulli variable, X n+1 say, with same probability parameter p Note: Observed weather X n+1 is (unconditionally) correlated with ensembles X 1, X 2,..., X n

18 (3) Skill of Ensemble Forecasts (3.1) Reliability (3.2) Brier Skill Score

19 (3.1) Reliability Reliability diagram Plot of observed probability vs. forecast probability Probability model for ensemble generation: Plot of (r + k) / (r + s + n) vs. (r 0 + k) / (r 0 + s 0 + n) -- Perfect reliability Obtained for prior distribution with r 0 = r, s 0 = s

20 1.0 Reliability Diagram (r = 2, s = 2, n = 10) Observed probability Forecast probability Face value Perfect reliability Fictitious ensemble Uniform prior

21 (3.2) Brier Skill Score Brier Score (BS) -- Mean square error of probability forecasts X n+1 denotes observation (i. e., X n+1 = 0 or 1) Viewed as (n+1)th ensemble member p^ denotes ensemble probability forecast BS = E[(X n+1 p^ ) 2 ]

22 Brier Skill Score (BSS) -- BSS = 0 for climatology, BSS = 1 for perfect forecasts -- Infinite Ensemble Size BSS( ) = 1 / (r + s + 1) -- Finite Ensemble Size n Perfect reliability BSS(n) = [n / (r + s + n)] BSS( ) Face value BSS(n) = ([n (r + s )] / n) BSS( )

23 Brier skill score (r = 1.5, s = 1.5) Brier skill score Perfect reliability Face value Infinite ensemble size Ensemble size

24 (4) Value of Ensemble Forecasts (4.0) Value of Imperfect Information (4.1) Cost-Loss Decision-Making Model (4.2) Economic Value of Ensemble Forecasts

25 Quote -- Lou Gerstner, former IBM CEO You don t get points for predicting rain. You get points for building arks.

26 (4.0) Value of Imperfect Information Decision-Theoretic Concept -- Maximize expected utility Take expectation with respect to probability of future weather event -- Prescriptive framework Value of information cannot decrease as quality increases Resources -- Book (Katz and Murphy, Cambridge Univ. Press, 1997) -- Case studies (

27 (4.1) Cost-Loss Decision-Making Model Expense Matrix -- Cost C incurred if take protective action -- Loss L incurred if do not protect and adverse weather occurs No Event Event Do Not Protect 0 L Protect C C

28 -- Optimization criterion Decision maker chooses action that minimizes expected expense Optimal strategy: p denotes (generic) probability of adverse weather Expected expense (Protect): C Expected expense (Do Not Protect): pl Protect if probability of adverse weather p > C/L Do not protect if p < C/L

29 (4.2) Economic Value of Ensemble Forecasts Value of Ensemble Forecasts -- Definition of economic value of forecasts Reduction in expected expense -- Use climatology as standard of comparison Difference between minimal expected expense for climatology and minimal expected expense for forecasts

30 -- Rescale economic value of forecasts Focus on effect of finite ensemble size: (i) Climatological information alone Economic Value = 0 (ii) For perfect model forecasting system (i.e., infinite ensemble size) Economic Value = 1

31 r = 0.75, s = 0.75, C/L = Economic value Face value Bayesian Ensemble size

32 1.0 r = 0.75, s = 0.75, C/L = 0.3 Economic value Face value Bayesian Ensemble size

33 (5) Discussion Prior Information -- What is reasonable assumption? Seems plausible that decision maker would have some idea about parameters of beta distribution (r, s ) for numerical prediction model (not necessarily equivalent to recalibration ) Imperfect Numerical Prediction Model -- Underdispersed ensembles