Venugopal, Basu, and Foufoula-Georgiou, 2005: New metric for comparing precipitation patterns

Transcription

1 Venugopal, Basu, and Foufoula-Georgiou, 2005: New metric for comparing precipitation patterns Verification methods reading group April 4, 2008 D. Ahijevych

2 Forecast Quality Index useful for ensembles uses surrogate fields accounts for close forecasts One number

3 Outline Paper overview universal image quality index (UIQI) and modified UIQI components of forecast quality index (FQI) Geometric examples (from Sukanta and Efi) Perturbed fake examples (also from S and E) Cases from SPC Spring 2005 surrogates traditional skill scores expert rankings

4 Paper overview forecast ensembles filter out similar members, and keep just enough to characterize the probability structure of forecast find best member and propagate it forward single measure (like RMSE and EqTh) but has important additional information

5 Paper overview - UIQI R1 and R2 are fields being compared 3 terms: covariance means standard deviations 3 properties: correlation brightness (bias) distortion (variability) UIQI ( R, R ) = 1 2! 2µ µ 2!! " "!! µ + µ! +! R, R R R R R R R R R R R

6 Paper overview UIQI, Hausdorff UIQI entirely amplitude-based measure not efficient at telling difference between displaced patterns and amplitude error Distance-based measures Hausdorff distance

7 Paper Overview - Hausdorff ( ) h( A, B) = max min a " b a! A b! B A h(a,b) forward distance B

8 Paper Overview - Hausdorff ( ) h( B, A) = max min a " b b! B a! A A h(b,a) backward distance B

9 Paper Overview - Hausdorff ( ( ) ( )) H ( A, B) = max h A, B, h B, A A h(b,a) backward distance h(a,b) forward distance B

10 Paper Overview - Hausdorff ( ( ) ( )) H ( A, B) = max h A, B, h B, A A H(A,B) B

11 Paper Overview partial Hausdorff ( ) h( A, B) = k th percentile min a " b a! A b! B A h(a,b)? B

12 Paper overview - Hausdorff ( ) h( A, B) = max min a " b a! A b! B a1 h(a,b) forward distance A b1 B a2 a3 b2

13 Paper overview - FQI (, ) FQI R R 1 2 = ( R, R ) k 1 2 ( R, Surrogates of R ) PHD Mean " PHD # $ k 1 1 % 2µ µ 2!! & µ µ!! R R R R R R R R

14 Paper overview - FQI FQI = normalized PHD [0,!] k modified UIQI [0,1]

15 Paper Overview - surrogates

16 Paper overview illustrative example 2 RMSE EqTh FQI 1 0 vs vs

17

18 Geometric examples O F O F O F O F O F CSI = 0 for first 4; CSI > 0 for the 5th

19 PHD 75 mod. UIQI, including zero pixels mod. UIQI

20

21

22

23

24 when I did 10 surrogates <H S > = 271 +/-27

25 Perturbed fake cases 3 pts right, -5 pts up 6 pts right, -10 pts up 12 pts right, -20 pts up 24 pts right, -40 pts up 48 pts right, -80 pts up 12 pts right, -20 pts up, times pts right, -20 pts up, minus 0.05

26

27

28

29

30

31

32

33

34 Spring 2005 SPC cases surrogates pictures example of distribution of forward and backward Hausdorff distances comparison to traditional methods comparison to expert scores

35

36 100 surrogates distribution of Hausdorff distance, solid/forward, dash/backward count Hausdorff distance (in grid spacing units) 75 th percentile

37 surrogate mean PHD 75 PHD 75 standard error mod. UIQI FQI: FQI = normalized PHD [0,!] k modified UIQI [0,1]

38

39

40

41

42

43

44

45

46

47 expert score expert score Apr 26 Apr 13 May 13 May 14 May 14 May 18 May 18 May 19 May 19 May wrf2caps 25 1 Jun 3 Jun 4 Jun May wrf4ncar 25 1 Jun 3 Jun 4 Jun May FQI FQI expert score r = w/o 1 st case expert score FQI expert score FQI Apr May 14 May 18 May 19 May wrf4ncep 25 1 Jun 3 Jun 4 Jun May FQI expert score FQI first case really bad; experts start out too generous?

48 expert scores vs grid stats grid stats agree: first case was bad

49 Pearson correlation coefficient and Spearman rank correlation coefficient

50 FQI Discussion application to ensembles adding to MET...