Homogenization of monthly and daily climatological time series

Transcription

1 Homogenization of monthly and daily climatological time series Petr Štěpánek Czech Hydrometeorological Institute, Czech Republic Latsis Foundation 1st International Summer School Environment: Climate Climate change - Impacts

2 Climatological studies Measuring and collecting data

3 Climatological studies Measuring data Data quality control and 0.8 Homogenization

4 Climatological studies Measuring data Homogenization Data Analysis

5 Processing before any data analysis Software AnClim, ProClimDB

6 Data Quality Control Finding Outliers Two main approaches: Using limits derived from interquartile ranges (time series) -2.0 comparing values to values of -4.0 neighbouring stations (spatial analysis) Quality control Homogenization Data Analysis

7 Homogenization Change of measuring conditions inhomogeneities

8 Inhomogeneity Detection Absolute Homogeneity Testing 3,0 2,0 Praha - Klementinum 1,0 0,0-1,0-2,0-3,0-4,

9 Inhomogeneity Detection Absolute Homogeneity Testing Relative Homogeneity Testing (differences with reference series) 1,5 1,0 0,5 0,0-0,5-1,0-1,5-2,0 Diference Praha - Klementinum a průměrovaná řada okolí ČR

10 Relative Homogeneity Testing Creating reference Series

11 Relative Homogeneity Testing Creating reference Series Tests of homogeneity

12 Relative Homogeneity Testing Creating reference Series Tests of homogeneity Assessing homogeneity - Metadata - Physically justified - undoubted inhomogeneity

13 Relative Homogeneity Testing Creating reference Series Tests of homogeneity Assessing homogeneity - Metadata - Physically justified - undoubted inhomogeneity Adjusting Series

14 Inhomogeneity Detecting by SNHT (p=0.05, 950 series) Inhomogeneities detected/ % Error /years > Amount of change in level / C

15 Assessing Homogeneity Problems: most of metadata incomplete we depend upon statistical tests results

16 Assessing Homogeneity Problems: most of metadata incomplete we depend upon statistical tests results uncertainty in test results - right inhomogeneity detection is problematic (for smaller amount of change)

17 Proposed solution To get as many test results for each candidate series as possible Ensemble approach - processing of big amount of test results for each individual series

18 Adventages of the Ensemble approach we know relevance (probability) of each inhomogeneity we can easily assess quality of measurements for series as a whole

19 How to increase number of test results Monthly, Seasonal and How to increase Data Processing number of test Annual results Averages Quality Control - Outliers Interquartile Range Comparing to Neighbours Combining Near Stations Homogeneity Testing Alexandersson test Bivariate Test t-test Mann-Whitney-Pettit from Correlations Reference Series from Distances Several Iterations Hom. Assessment Probability Adjusting Data Filling Miss. Values Days, Months, seasons, year

20 Creating Reference Series for monthly, daily data (each month individually) weighted/unweighted mean from neighbouring stations criterions used for stations selection (or combination of it): best correlated / nearest neighbours (correlations from the first differenced series) limit correlation, limit distance limit difference in altitudes neighbouring stations series should be standardized to test series AVG and / or STD (temperature - elevation, precipitation - variance) - missing data are not so big problem then

21 Relative homogeneity testing Available statistical tests: Alexandersson SNHT Bivariate test of Maronna and Yohai Mann Whitney Pettit test t-test Easterling and Peterson test Vincent method 20 year parts of the daily series (40 for monthly series with 10 years overlap), in SNHT splitting into subperiods in position of detected significant changepoint (30-40 years per one inhomogeneity)

22 Homogeneity Tests Alexandersson s SNHT Alexandersson Standart Normal Homogeneity Test (Single shift test) Reference series: k q i = Y /{[ ρ X Y / X j ]/ ρ } i j= 1 k 2 j ji k j= 1 2 j q i = Yi { ρ j [ X ji X j + Y ]/ ρ j } j= 1 2 k j= Null and alternative hypothesis: H 0 : z i N(0,1), i {1,..,n}. H 1 : z i N(μ 1,1), i {1,..,a}, z i N(μ 2,1), i {a+1,..,n}, for 1? a < n a μ 1? μ 2. z i = (q i - q )/s q, z i N(0,1) Test statistic: 2 2 T 0 = max {Ta} 1 a< n 1 = max { az1 + ( n a) z 2} 1 a< n 1 where z a 1 1 = z i, ( z 1? μ a 1), Quality control í = 1 n z 1 Homogenization 2 = z i, ( z 2? μ ( n a) 2). í = a+ 1 Data Analysis

23 Homogeneity Tests Bivariate Test of Maronna and Yohai Bivariate Test Null and alternative hypothesis: H 0 : vectors {x i,y i } bivariate normal distributed N(μ x, μ y, σ 2 x, σ 2 y, ρ) H 1 : pro 0<i 0 <n a d? 0 - N(μ x, μ y, σ 2 x, σ 2 y, ρ) pro i? i 0 N(μ x, μ y+d, σ 2 x, σ 2 y, ρ) pro i > i 0. Test statistic: T 0 = max{t } i<n i i where: X i = 1 / i x j, Y i = 1 / i y j, X = X n, Y = Yn n j= 1 i j= S x = ( x j X ), S y = ( y j Y), S xy = ( x j X )( y j Y), j = 1 n j= 1 2 F i = S x ( X i X ) ni /( n i), i<n, D i = S x ( Y Yi ) S xy( X X i ) n /[( n i) Fi ], 2 i 2 xy T i =[ i( n i) D F ]/( S S S ) i x y n j= 1

24 Homogeneity Tests Two-phase linear regression (Vincent Technique, Easterling and Peterson test) Easterling and Peterson Test statistic: U = [(RSS 1 -RSS 2 )/3]/[RSS 2 /(n-4)]? F(3,n-4) t-test: differences of levels before and after a discontinuity

25 Homogeneity assessment Various outputs created for better inhomogeneities assessment Combining results with information from metadata whenever possible Decision about undoubted inhomogeneities (without metadata) coincidence of test results

26 Homogeneity assessment Output example: Station Čáslav, 3rd segment, , n=40 Test Ref I II III IV V VI VII VIII IX X XI XII Win Spr Sum Aut Year A avg A 1930 A corr A A dist A B avg B 1922 B corr B B 1937 B dist B V corr V V 1937 V dist V 1918

27 Homogeneity assessment, Output II example: Begin End Length InHomogen eity Number % detected inhom % possible inhom End Missin g Summed numbers of detections for individual years

28 Homogeneity assessment combining several outputs (sums of detections in individual years, metadata, graphs of differences/ratios, ) ID ELYEAR_BEGINEND YEAR_COUNY_POSSIBL YEAMIS X_BEGIN_DX_END_DATX_X_L L ABREMARKC C_ x B1BOJK01 x # # Bchange B1BOJK01 x # # obs V B B1BYSH01 x ? B1BYSH01 x ? B1BYSH01 x ? B1HLHO01 x B1HOLE01 x B1KROM01 x xb1rade01x # # Rchange B1RADE01 x # # obs JoB x B1RYCH01 x # # Vchange B1RYCH01 x # # obs MB xx? B1STRZ01 x B1STRZ01 x B1UHBR01 x # # Uchange B1UHBR01 x # # obs JoB x B1UHBR01 x # # Uchange B1UHBR01 x # # obs JoB B1VELI01 x ? B1VELI01 x ? B1VKLO01 x x B1VYSK01 x # # Vchange B1VYSK01 x # # obs V B B2BOSK01_rx B2BREC01 x B2BRUM01 x # # Bchange B2BRUM01 x # # obs MB

29 Adjusting monthly data using reference series based on correlations adjustment: from differences/ratios 20 years before and after a change, monhtly smoothing monthly adjustments (low-pass filter for adjacent values) I II III IV V VI VII VIII IX X XI XII

30 Iterative homogeneity testing several iteration of testing and results evaluation several iterations of homogeneity testing and series adjusting (3 iterations should be sufficient) question of homogeneity of reference series is thus solved: possible inhomogeneities should be eliminated by using averages of several neighbouring stations if this is not true: in next iteration neighbours should be already homogenized

31 Filling missing values Before homogenization: influence on right inhomogeneity detection After homogenization: more precise - data are not influenced by possible shifts in the series Dependence of tested series on reference series

32 Example: CZ, air temperature (200 stations, ) Number of significant inhomogeneities before and after homogenization (p=0.05) 2500 Before After I II III IV V VI VII VIII IX X XI XII Before After 0 Win Spr Sum Aut Year

33 Amount of adjustments for homogenised series (absolute values) - median C I II III IV V VI VII VIII IX X XI XII Correlation coefficients between candidate and reference series before and after homogenization (median) 1.00 Before After I II III IV V VI VII VIII IX X XI XII

34 Example: CZ, precipitation (800 stations, ) 4 tests, 4 reference series, 12 months + 4 seasons and year Number of detected inhomogeneities (significant) 6000 Number of detections I II III IV V VI VII VIII IX X XI XII Month

35 Amount of change (ratios standardized to be >1.0), precipitation (reference series calculation based on correlations) Amount of change (standardized) I II III IV V VI VII VIII IX X XI XII Boxplots: -Median - Upper and lower quartiles (for 589 testes series) Correlation improvement Correlation increase I II III IV V VI VII VIII IX X XI XII

36 Inhomogeneities in summer versus in winter, Air temperature Change of measuring conditions at the station (relocation etc.) is manifested in the series mainly in summer in winter: active surface role is diminished, prevailng circulation factors, in summer: active surface role increases, prevailing radiation factors

37 Inhomogeneities in summer versus in winter, Precipitation Change of measuring conditions at the station (relocation etc.) is manifested in the series mainly in winter in winter: errors of measurement (solid precipitation -wind, )

38 Homogenization Final remarksr emarks, recommendations 1/3 data quality control before homogenization is of very importance (if it is not part of it) Using series of observation hours (complementarily to daily AVG) is highly recommended (different manifestation of breaks) be aware of annual cycle of inhomogeneities, adjustments, to know behavior of spatial correlations (of element being processed) to be able to create reference series of sufficient quality

39 Homogenization Final remarksr emarks, recommendations 2/3 Because of Noise in the time series it makes sense: - Ensemble approach to homogenization (combining information from different statistical tests, time frames, overlapping periods, reference series, meteorological elements, ) - more information for inhomogeneities assessment higher quality of homogenization in case metadata are incomplete

40 Homogenization of daily values, remarks 3/3 Correlation coefficients (tested versus reference series) are slightly lower (compared to monthly data), but still high enough (around 0.9 even in case precipitation) Advantage: reliable inhomogeneities detection near the ends of series Complementary information to monthly and seasonal values detections (but problems with distribution, autocorrelations, ) Correction of daily data: delta method, if applied, it should be discriminated with regard to other parameters like cloudiness, Variable correction (such as HOM) seems to be a good choice (preserving CDF)

41 Software used for data processing LoadData - application for downloading data from central database (e.g. Oracle) ProClimDB software for processing whole dataset (finding outliers, combining series, creating reference series, preparing data for homogeneity testing, extreme value analysis, RCM outputs validation, correction, ) AnClim software for homogeneity testing limahom.eu

42 AnClim software AnClim software

43 Examples of Data formats AnClim, monthly data

44 Examples of Data formats AnClim, daily data

45 ProClimDB software ProcData software

46 limahom.eu

47 ACTION COST-ES0601: Advances in homogenisation methods of climate series: an integrated approach (HOME) 03/05/ End date: 02/05/2011, Year: 3 Inventory of existing detection and correction methods Compilation of a benchmark dataset with (un)known inhomogeneities Selection, comparison and evaluation of existing detection and correction (including those not traditionally used in climatology) Objective analysis of advantages and/or disadvantages of existing methods (benchmark) Investigation in further improvements of methods Documentation of practical recommendations Presentation and release of a new common method