Sharp statistical tools Statistics for extremes

Transcription

1 Sharp statistical tools Statistics for extremes Georg Lindgren Lund University October 18, 2012 SARMA

2 Background Motivation We want to predict outside the range of observations Sums, averages and proportions Normal distribution Extremes Central limit theorem Successes in large experiments Normal distribution inappropriate Bulk of data may be misleading Extremes are rare Sparsity of data may be compensated by proper (extreme value) theory

3 Background Overview of contents Topics Extreme value distributions Block maxima POT - Peaks Over Threshold Estimation, return period, uncertainties Extremes with cyclic or linear trend Extremes with other covariates CO2, NAO, AO,... Useful statistics packages The R environment - need packages extremes and evd for the computer experiments The WAFO package in Matlab and Python (Downloadable from code.google.com)

4 Background A typical example: annual maximum of daily precipitation 100 years of yearly maximum of daily precipitation in Fort Collins precipitation (in)

5 Background A second typical example Monthly maximum precipitation values has 12 times as many data but season Precipitation (in)

6 Background Some basic facts about statistical extremes Notation: M n = max(x 1, X 2,..., X n ) is the maximum of n observations of a random quantity. Example: X k = maximum precipitation year k Problem: Find the probability Prob(M n > x) for large n and reasonable x. In particular when x > largest observed data!

7 Background Return period return level X 1, X 2,... a sequence of random quantities with common distribution, e.g. one observation per year (a mean, a yearly maximum,..., anything) 100 year return value x 100 is the value that is exceeded on average once in 100 years. The same as Prob(X 1 > x 100 ) = 1/100 ( Prob(M 100 > x 100 ) = ) /e = The probability that the 100-year return level is exceeded at least one time during a 100 year period is 63%. The 1000-year value exceeded with the same probability at least once during 1000 years, etc. IF YEARS ARE INDEPENDENT OF EACH OTHER. USUALLY,

8 Background Wave data from the Pacific Ocean 582 measurements of H s = Significant wave-height at buoy in the Pacific. About 14 data per month, Dec-Feb, during 7 years. Hs t

9 Background Histogram of Pacific wave height Histogram of Hs Hs F(x) CDF of Hs exceedances over 7m CDF of exponential distribution +7m x Conclusion: the tail of the distribution has a more standardized distribution than the central part.

10 GEV and GPD Goals for statistical extreme value analysis One-dimensional: From a series of observations (often in time) estimate the probability of high values, of the order of the largest observation; OR HIGHER, even MUCH HIGHER Make a statement about how uncertain the prediction is Identify possible non-stationarity in extremes can be different from non-stationarity in averages or standard deviation Find covariates that explain the occurrence of extreme values Utilize data as much as possible balance between number of data available and bias Multi-dimensional: find probabilities of combinations of extreme (or not so extreme) values in two or more series (research is going on) Extremes in random fields (even more research is going on)

11 GEV and GPD The GEV and the GPD Block maxima: Take one extreme value per time unit (e.g. day, month, year,...). Ideal situation: stationary conditions, no trend. Exceedance analysis: Take all values over a certain threshold gives more data that are more representative for the global maximum that the bulk data. Statistical extreme value analysis relies on two mathematical facts Block maxima: The maximum of many observations has a GEV distribution. Exceedance analysis: The exceedances have a GPD distribution.

12 GEV and GPD The Extremal types theorem An almost true theorem: When n is large, the distribution of the maximum M n is a GEV = Generalized Extreme Value distribution: { ( Prob(M n x) exp 1 + ξ x µ ) } 1/ξ ψ where ξ = shape (type), ψ = scale, µ = location ξ < 0 = an upper limit exists, ξ > 0 = a lower limit exists, ξ = 0 = Gumbel distribution, no limits exist +

13 GEV and GPD Light, Gumbel, or Heavy tails Light tail ξ < 0 log( log(f)) Gumbel Probability Plot X Gumbel Probability Plot 5 10 Double tail ξ = Heavy 200 tail ξ > log( log(f)) log( log(f)) X Gumbel Probability Plot X

14 GEV and GPD The Tail theorem Another almost true theorem: Almost all statistical distribution have a tail that is GPD they have a Generalized Pareto Distribution. Take only observations that above a fixed threshold u and let Y = X u be the exceedance. ( Prob(Y y) ξ y ) 1/ξ σ + where ξ = shape (type), σ = scale, (µ = location = 0). ξ = 0 = exponential tail, ξ > 0 = heavy tail, ξ < 0 = limited tail

15 GEV and GPD GPD-tail in normal distribution Normal tail is GPD with ξ = Normal distribution The tail > 2 of a normal distribution 0.8 F(x) Red = empirical cdf of exceedances over 2 Blue = estimated GPD x 1.8

16 Block maxima Block maxima with block = one year Many observations over a year (hourly, semi-daily, daily) take the maximum. Assume a GEV distribution for yearly maximum. Estimate parameters in GEV by Maximum likelihood method in R-package extremes (or WAFO) fit <- gev.fit(data)

17 Block maxima 100 years of precipitation in Fort Collins, CO Fort Collins, CO, daily precipitation: odie/rain.html Time series of daily precipitation, Strong annual cycle wet in late spring, dry in winter No long-term trend Recent flood, July 28, 1997

18 Block maxima Annual daily maximum precipitation, Fort Collins precipitation (in) year

19 Block maxima Seasons can be handled by making separate analyses for each month Precipitation (in)

20 Block maxima Some R-code data <- read.table(file="ftcprec.prn", header=true) plot(data$mn,data$prec/100,ylab="precipitation (in)", xlab="month") YearlyMax = aggregate(prec Year,data=data,max) YearlyMax[,2]= YearlyMax[,2]/100 plot(yearlymax,type= l,col= blue ) library(extremes) ftcanmax <- read.table(file="ftcanmax.prn", header=true) fit <- gev.fit(ftcanmax$prec/100) gev.diag(fit)

21 Block maxima Estimated parameter values + standard errror Parameter Estimate Standard error Location µ Scale ψ Shape ξ

22 Block maxima Is the GEV a Gumbel distribution? Is the shape parameter ξ = > 0 really significantly different from 0? Can be tested by a likelihood ratio test: Dev = 2 log max likelihood under restricted model (ξ = 0) max likelihood under full model If restricted model is correct, this has a chi-squared distribution; d.f. = # parameters in full model - # parameters in restricted model Test by extremes (ξ = 0 = Gumbel): fit0 <- gum.fit(ftcanmax$prec/100) Dev <- 2*(fit0$nllh - fit$nllh) pval = pchisq( Dev, 1, lower.tail=f) (= 0.038)

23 Block maxima A 95% confidence interval for shape estimate is (0.009, 0.369) gev.profxi(fit, fit$mle[3] -0.25, fit$mle[3] +0.25) Profile Log likelihood

24 Block maxima Diagnostics and conclusions We want return values for N = 10, 100, 100, 1000 years in GEV: x N = µ ψ {1 ( ln(1 1/N)) ξ} ξ Estimated standard errors gives confidence limits for return levels. gev.diag(fit) An example for the exercises; fit.rl <- return.level(fit,rperiods=c(10,100), conf = 0.05)

25 Block maxima GEV summary Probability Plot Quantile Plot Model Empirical Empirical Model Return Level Plot Density Plot Return Level f(z) e 01 1e+00 1e+01 1e+02 1e+03 Return Period z

26 Block maxima Trends in extremes Water level in the Japan Sea 20 Water level 15 (m) Maximum 5 min water level (m) Time (h)

27 Block maxima Alt I: Normalize (simplistic) Take average and standard deviation for each 5-minute period, subtract and divide. Take maximum over 5-minute period, and fit a GPD. Diagnostic plot: 1 Probability plot Density plot F(x) x Residual Quantile Plot x Residual Probability Plot Model (gev) Model (gev) Empirical Fit method: PWM, Fit p value: Empirical

28 Block maxima Alt II: fit with trend yura5fit<-gev.fit(y5max[,1]) Est.: S.e.: yura5fit.mul<-gev.fit(y5max[,1], ydat=as.matrix(1:length(y5max[,1])),mul=1) Est S.e.: yura5fit.sigl<-gev.fit(y5max[,1], ydat=as.matrix(1:length(y5max[,1])),sigl=1) Est.: S.E.:

29 Exceedance analysis The dilemma of statistical extremes We want to predict events that have never been observed! From 20 years data can one say something about the 100 year return value?

30 Exceedance analysis 20 years of monthly data 5 20 years of monthly data

31 Exceedance analysis Use more high values Waste of data to use only yearly maximum 20 years of data = 240 monthly data The smallest yearly maximum (year 7) is X 7 = There are 42 monthly values greater than 1.67 Can one use all 42? Or all those 48 greater than 1.5? Or the 84 values greater than 1?

32 Exceedance analysis High values are rare and occur randomly Take a reasonably high level u try many! Estimate the rate by which this level is exceeded per time unit (e.g. per year), λ = λ u : λ = Observed number of exceedances over u Total observation time u = 1.5 gives λ 1.5 = 48/20 = 2.4 Reasonable to assume that N = the number of exceedances over 1.5 any year has a Poisson distribution with mean λ = 2.4: Prob(N = k) = e λ λ k /k!

33 Exceedance analysis Generaliserad Pareto fördelning - GPD Exceedances over a high threshold are more representative for the global extremes than the buld data Almost all distributions have a Generalized Pareto-tail, GPD With Y = X u = exceedance over level u: ( Prob(Y y) ξ y ) 1/ξ σ + Exponential tail: ξ = 0; Heavy tail: ξ > 0; Limited tail: ξ < 0

34 Exceedance analysis GPD-tail in normal distribution Normal tail is GPD with ξ = Normal distribution The tail > 2 of a normal distribution 0.8 F(x) Red = empirical cdf of exceedances over 2 Blue = estimated GPD x 1.8

35 Exceedance analysis Poisson + GPD = GEV N = number of exceedances Y j = X j u over u is (approximately) Poisson distributed with expectation λ The size of exceedances Y 1,..., Y N, have (approximately) a GPD If M = yearly maximum = u + max(y 1,..., Y N ), then for x > u: Prob(M x) = Prob(N = 0) + =... = exp { ( λ 1 + ξ x u σ Prob(N = n, Y 1,..., Y n x u) n=1 ) 1/ξ + } (1)

36 Exceedance analysis Poisson + GPD = GEV, contd (1) is a GEV distribution Prob(M x) = exp Translation from Poisson+GPD to GEV: { ψ = σ λ ξ ( 1 + ξ x µ ψ µ = u + ψ σ ξ ) 1/ξ To get maximum over n years, replace λ with nλ in (1) + }

37 Exceedance analysis Choice of threshold How to choose u? Note: assume GPD above threshold u Diagnostic: A GPD has linear mean excess E(X u X > u) = σ + ξu 1 ξ Plot the mean of all excesses over u as function of u. Take u as the smallest threshold for which the curve to the right is linear The slope is ξ/(1 ξ) om ξ < 1.

38 Exceedance analysis Mean excess plot Plot of E(X u X > u) for 20 years of monthly data: 1.2 Mean exceedance over threshold

39 Exceedance analysis Diagnostics in GPD analysis A plot of mean excess can be had to interpret Alternative: Estimate a full GPD for different thresholds If the tail above u 0 is GPD then all exceedances over u > u 0 are also GPD with the same shape parameter ξ but with different scal parameter σ u = σ u0 + ξ (u u 0 ) Modified scale = σ u ξ u should be constant, independent of u, when GPD is appropriate

40 Exceedance analysis Estimated CDF for yearly maximum From 20 yearly maxima: c = 0.14, µ = 2.81, a = Empirical and GEV estimated cdf (PWM method) True CDF for yearly maximum F(x) CDF for estimated GEV x

41 Exceedance analysis Estimated CDF by POT method 84 exceedances over u = 1 and GPD estimate gives ξ = 0.04, b = 2.38, a = Tail probability 10 1 Tail by POT method True CDF 10 2 Tail by direct GEV estimation

42 Exceedance analysis Fatalities in English coal mines Time and number of fatalities

43 Exceedance analysis GEV? We try GEV on all data (not really motivated!): Empirical and GEV estimated cdf (PWM method) F(x) x

44 Exceedance analysis Consider only big accidents POT There are 25 accidents with > 100 dead. Fit GPD to data > % of these exceed 350. CDF for deaths > 100 and GPD F(x) x

45 Exceedance analysis Vargas, Venezuela, catastrophe 1999 In December 15, 1999, the Vargas province was hit by 410 mm rain.

46 Exceedance analysis Rain in Venezuela - GEV or Gumbel? Gumbel distribution is GEV with shape parameter ξ = 0. Based on rain statistics the maximum daily rain distribution was estimated with GEV: scale = 19.9, location = 49.2, shape = The shape parameter ξ is not significantly different from 0, so Gumbel could perhaps be used instead of GEV. Estimated year return value for daily rain is x = 249 mm. In December 1999 the Vargas province was hit by 410 mm rain during one day. From the full GEV the estimate of the year value had been x = 468 mm, much closer to the real value. A 95% one-sided confidence interval is x < 1030 mm.

47 Multivariate extremes General recipe for bivariate extremes Assume, for a sequence of days, X 1, X 2,... is the wind speed and Y 1, Y 2,... is the wave height measured at a North Sea platform. The platform is designed for extreme 2 waves. It is also designed for extreme 1.8 Extremes come togehter winds. What about the combined 1.6 effect of wind and waves (and current)? Make marginal EVA and transform 1 Extremes just add x and y to standard scale. Plot 0.8 pairs of transformed extremes to examine joint extreme behavior. Three Extremes come alone main types of dependence/nondependence:

48 Some literature Some literature Beirlant, J., Goegebeur, Y., Segers, J., Teugels, J. (2004), Statistics of Extremes: Theory and Applications. Wiley. Coles S. (2001), An Introduction to Statistical Modeling of Extreme Values. Springer-Verlag. Gilleland, E., Katz, R., Young, G. (Feb. 2012), Package extremes. Nelsen, R.B. (2006), An Introduction to Copulas. Wiley. The WAFO group (2011), WAFO a MATLAB toolbox for analysis of random waves and loads. Lund univ. and