BAYESIAN APPROACH TO DECISION MAKING USING ENSEMBLE WEATHER FORECASTS Rick Katz Institute for Study of Society and Environment National Center for Atmospheric Research Email: rwk@ucar.edu Paper: R.W. Katz & M. Ehrendorfer, 2006: Wea. Fore., 21, 220-231 (www.isse.ucar.edu/hp_rick/pdf/waf2006.pdf) Talk: www.isse.ucar.edu/hp_rick/pdf/sbensem.pdf
Quote -- Lao Tzu, Chinese Philosopher He who knows does not predict. He who predicts does not know.
Outline (1) Background (2) Bayesian Approach (3) Skill of Ensemble Forecasts (4) Value of Ensemble Forecasts (5) Discussion
(1) Background Ensemble Forecasts -- How to produce probability forecasts (take at face value )? Skill of Ensemble-based Probability Forecasts -- Effect of ensemble size on reliability & Brier score Value of Ensemble-based Probability forecasts -- Effect of ensemble size on economic value
(2) Bayesian Approach (2.0) Bayesian Inference (2.1) Bayesian Model for Bernoulli Process (2.2) Bayesian Ensemble Probability Forecasts (2.3) Probability Model for Ensemble Generation
Quote -- I. J. Good, British/American statistician (1973) The subjectivist states his judgements, whereas the objectivist sweeps them under the carpet by calling assumptions knowledge, and he basks in the glorious objectivity of science.
(2.0) Bayesian Inference Alternative to Classical/Frequentist Approach -- Incorporate prior information Combination of different sources of information -- Interpretation Tests of significance, Confidence intervals -- Computations Markov Chain Monte Carlo
(2.1) Bayesian Model for Bernoulli Process Bernoulli Process -- Sequence of n Bernoulli trials p probability of event occurrence on given trial Assume trials are (conditionally) independent given p -- Prior distribution Probability density function f(p) Assume p has beta distribution with parameters r = r 0, s = s 0
-- Likelihood function L(x 1, x 2,..., x n p) where x 1, x 2,..., x n denotes data Let k denote no. of events that occur (out of n trials), 0 k n Given p, distribution of k is binomial with parameters n, p -- Posterior distribution Probability density function g(p x 1, x 2,..., x n ) Bayes s Theorem: g(p x 1, x 2,..., x n ) % L(x 1, x 2,..., x n p) f(p)
Given prior beta(r 0, s 0 ) & data k, conditional distribution of p is beta with parameters: r = r 0 + k, s = s 0 + n k Posterior mean: E(p x 1, x 2,..., x n ) = (r 0 + k) / (r 0 + s 0 + n) Differs from classical/frequentist estimate: p^ = k/n
Prior & Posterior Distributions: k = 0, n = 10 10 Probability density function 8 6 4 2 Prior(1, 1) Posterior(1, 11) 0 0.0 0.2 0.4 0.6 0.8 1.0 Probability
(2.2) Bayesian Ensemble Probability Forecasts Ensemble Forecasting -- Notation n is ensemble size k is number of ensembles for which event occurs p is actual forecast probability of event Obtain from infinitely many ensembles (from perfect numerical weather prediction model)
-- View weather event as a single future Bernoulli trial Posterior predictive probability is posterior mean: p^ = (r 0 + k) / (r 0 + s 0 + n) p^ denotes Bayesian forecast probability estimator
Bayesian Interpretation of Existing Schemes -- Face value p^ = k/n r 0 = 0, s 0 = 0 Only plausible for perfect forecasting system (not just perfect model) -- Fictitious ensemble p^ = (k + 0.5) / (n + 1) r 0 = 0.5, s 0 = 0.5 Only plausible for forecasting system with moderate skill
Probability density function 5 4 3 2 1 0 Prior Beta Distributions 0.0 0.2 0.4 0.6 0.8 1.0 Event probability r = 1/2, s = 1/2 r = 1, s = 1 r = 0, s = 0 r = 10, s = 20
(2.3) Probability Model for Ensemble Generation Ensemble Forecasting -- Assumptions Numerical weather prediction model is perfect Generates probability forecasts from beta distribution with parameters: r = r, s = s
-- Stochastic simulation of ensembles (i) Generate actual forecast probability p from beta distribution(r, s ) (ii) Given p, generate n ensembles (n Bernoulli trials, parameter p), say X 1, X 2,..., X n (iii) Independently of X 1, X 2,..., X n generate weather observation as additional Bernoulli variable, X n+1 say, with same probability parameter p Note: Observed weather X n+1 is (unconditionally) correlated with ensembles X 1, X 2,..., X n
(3) Skill of Ensemble Forecasts (3.1) Reliability (3.2) Brier Skill Score
(3.1) Reliability Reliability diagram Plot of observed probability vs. forecast probability Probability model for ensemble generation: Plot of (r + k) / (r + s + n) vs. (r 0 + k) / (r 0 + s 0 + n) -- Perfect reliability Obtained for prior distribution with r 0 = r, s 0 = s
1.0 Reliability Diagram (r = 2, s = 2, n = 10) Observed probability 0.8 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Forecast probability Face value Perfect reliability Fictitious ensemble Uniform prior
(3.2) Brier Skill Score Brier Score (BS) -- Mean square error of probability forecasts X n+1 denotes observation (i. e., X n+1 = 0 or 1) Viewed as (n+1)th ensemble member p^ denotes ensemble probability forecast BS = E[(X n+1 p^ ) 2 ]
Brier Skill Score (BSS) -- BSS = 0 for climatology, BSS = 1 for perfect forecasts -- Infinite Ensemble Size BSS( ) = 1 / (r + s + 1) -- Finite Ensemble Size n Perfect reliability BSS(n) = [n / (r + s + n)] BSS( ) Face value BSS(n) = ([n (r + s )] / n) BSS( )
Brier skill score (r = 1.5, s = 1.5) 0.4 0.2 Brier skill score -0.0-0.2-0.4 Perfect reliability Face value Infinite ensemble size -0.6 0 10 20 30 40 50 Ensemble size
(4) Value of Ensemble Forecasts (4.0) Value of Imperfect Information (4.1) Cost-Loss Decision-Making Model (4.2) Economic Value of Ensemble Forecasts
Quote -- Lou Gerstner, former IBM CEO You don t get points for predicting rain. You get points for building arks.
(4.0) Value of Imperfect Information Decision-Theoretic Concept -- Maximize expected utility Take expectation with respect to probability of future weather event -- Prescriptive framework Value of information cannot decrease as quality increases Resources -- Book (Katz and Murphy, Cambridge Univ. Press, 1997) -- Case studies (www.isse.ucar.edu/hp_rick/esig.html)
(4.1) Cost-Loss Decision-Making Model Expense Matrix -- Cost C incurred if take protective action -- Loss L incurred if do not protect and adverse weather occurs No Event Event Do Not Protect 0 L Protect C C
-- Optimization criterion Decision maker chooses action that minimizes expected expense Optimal strategy: p denotes (generic) probability of adverse weather Expected expense (Protect): C Expected expense (Do Not Protect): pl Protect if probability of adverse weather p > C/L Do not protect if p < C/L
(4.2) Economic Value of Ensemble Forecasts Value of Ensemble Forecasts -- Definition of economic value of forecasts Reduction in expected expense -- Use climatology as standard of comparison Difference between minimal expected expense for climatology and minimal expected expense for forecasts
-- Rescale economic value of forecasts Focus on effect of finite ensemble size: (i) Climatological information alone Economic Value = 0 (ii) For perfect model forecasting system (i.e., infinite ensemble size) Economic Value = 1
r = 0.75, s = 0.75, C/L = 0.2 1.0 Economic value 0.5 0.0-0.5-1.0-1.5 Face value Bayesian 2 4 6 8 10 Ensemble size
1.0 r = 0.75, s = 0.75, C/L = 0.3 Economic value 0.8 0.6 0.4 0.2 Face value Bayesian 0.0 2 4 6 8 10 Ensemble size
(5) Discussion Prior Information -- What is reasonable assumption? Seems plausible that decision maker would have some idea about parameters of beta distribution (r, s ) for numerical prediction model (not necessarily equivalent to recalibration ) Imperfect Numerical Prediction Model -- Underdispersed ensembles