Decomposition and Attribution of Forecast Errors

Transcription

1 Decomositio d ttributio o orecst rrors gli Yg virometl Modelig Ceter tiol Ceters or virometl Predictio College Prk, Mryld, US gli.yg@o.gov 7 th Itertiol eriictio Methods Worksho Berli, Germy, My 3-, 07

2 Outlie Itroductio to RMS i model evlutio Decomosig RMS or sclr vribles Decomosig RMS or vector wids revised RMS veriictio

3 RMS Hs log bee used s erormce metric or model evlutio. Smller HGT RMS, ex is better th os Smller wid RMS, ex is better th os Curret os GS Q3Y7 ex MS GS Curret os GS Q3Y7 ex MS GS

4 RM S CP-MC GS eriictio Scorecrd Mgemet ote relies o the scorecrd to mke decisio o model imlemettio I this resettio I will demostrte tht RMS c t times misrereset model erormce.

5 ( ) Root-Me Squred rror () Where, is orecst, is either lysis or observtio, is the totl umber o oits i temorl or stil domi, or stil-temorl combied sce. ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) R ( ) ( ) R omlous tter correltio where ( ) ( ) rices o orecst & lysis Me squred error

6 m Me Squred rror: MS m ( ) MS by Me Dierece R MS by Ptter ritio Totl MS c be decomosed ito two rts: the error due to diereces i the me d the error due to diereces i tter vritio, which deeds o stdrd devitio over the domi i questio d omlous tter correltio to observtio/lysis. I orecst hs lrger me bis th the other, its MS c still be smller i it hs much smller error i tter vritio, d vice vers. I the ollowig we discuss the chrcteristics o tter vritio

7 Geerl Percetio: models with strog diusio roduce smoother ields, d hece hve smller RMS. The swer is: ot lwys true R R 0 mi i R Cse ) R, erect tter correltio (mi) 0 whe Oe c see tht i orecst hvig either too lrge or too smll vrice wy rom the lysis vrice, its error o tter vritio icreses. R ( ) I R, does ot wrd smooth orecsts tht hve smller vrices. It is ot bised. 7

8 Cse ) R 0.5, imerect tter correltio (mi) 0 whe 0. 5 I this cse, i oe orecst hs better vrice ( ) th the other ( ), 0. 5 the ormer will hve lrger th the ltter. Good orecsts re ctully elized! worse orecst better orecst I geerl, i 0 < R <, wrds smoother orecsts which hve smller vrices close to R. 8

9 Cse 3) or cses where R 0, (mi) whe 0 Icrese mootoiclly with R R I this cse, lwys wrds smoother orecsts tht hve smller vrices. Good orecst is eelized! 9

10 Will MS ormlized by lysis vrice be ubised? m Rλ λ λ λ R λ λ ssume m 0 R c Idelly, or give correltio R, the ormlized error should lwys decrese s the rtio o orecst vrice to lysis vrice reches to oe rom both sides. I the bove tble oly whe R is close to oe (highly corrected tters) does this eture exist. or most other cses, esecilly whe R is egtive, the ormlized error decreses s the vrice rtio decrese rom two to zero. I other words, the ormlized error still vors smoother orecsts tht hve vrice smller th the lysis vrice (the truth). 0

11 Is Me-Squred-rror Skill Score (Murhy, MWR, 988, 49) Ubised? m Rλ λ λ MSSS λr λ m R ssume m λ λ MSSS λ MSSS R c The best cse is MSSS whe R d Lmbd. or most cses, esecilly whe R is egtive, MSSS decreses mootoiclly with Lmbd. Thereore, MSSS still vors smoother orecsts tht hve vrice smller th the lysis vrice.

12 Summry I Covetiol RMS c be decomosed ito rror o Me Dierece (m) d rror o Ptter ritio () is ubised d c be used s objective mesure o model erormce oly i the omlous tter correltio R betwee orecst d lysis is oe (or very close to oe) I R <, is bised d vors smoother orecsts tht hve smller vrices. ormlized by lysis vrice is still bised d vors orecsts with smller vrice i omlous tter correltio is ot erect. comlete model veriictio should iclude omlous Ptter Correltio, Rtio o orecst rice to lysis rice, rror o Me Dierece, d rror o Ptter ritio. RMS c t times be misledig, esecilly whe the omlous tter correltio betwee orecst d lysis is smller. R m ( ) m

13 Decomosig RMS o ector Wid

14 ector Wid Stts So r the devitios re or sclr vribles. or vector wid, the corresodig stts re deied i the ollowig wy. Deie u i ρ v ρ j u i ρ v ρ j The MS: ( ) ( ) ( ) ( u v ) ( u v ) ( u u v v ) B - C where ( u v ) B ( u v ) C ( u ) u v v, B, d C re rtil sums i CP MC SDB dtbse omlous Ptter Correltio: R [( u u )( u u ) ( v v )( v v )] [( ) ( ) ] u u v v ( u u ) ( v v ) [ ] 4

15 5 R ( ) ( ) m v v u u R ector Wid Stts where ( ) ( ) m v v u u ( ) ( ) [ ] v v u u ( ) ( ) [ ] v v u u MS by Me Dierece MS by Ptter ritio rice o orecst rice o lysis

16 Demostrtio Decomosed RMS o Sclr d ector ribles lictio to Comlete Objective Model vlutio

17 Decomosig MS o Sclr ribles The ollowig ive comoets will be exmied. ll orecsts re veriied gist the sme lysis, i.e., the me o the two exerimets rur d re3d. m Totl MS m ( ) MS by Me Dierece R MS by Ptter ritio λ Rtio o Stdrd Devitio: cst/l MSSS λr λ m Murhy s Me-Squred rror Skill Score R ( ) ( ) omlous Ptter Correltio ( ) ( ) 7

18 Decomosig RMS o ector Wid The ollowig ive comoets will be exmied. ll orecsts re veriied gist the sme lysis, i.e., the me o the two exerimets rur d re3d. m Totl MS m ( u ) ( ) u v v MS by Me Dierece R MS by Ptter ritio λ Rtio o Stdrd Devitio: cst/l MSSS λr λ m Murhy s Me-Squred rror Skill Score R [( u u )( u u ) ( v v )( v v )] [( ) ( ) ] u u v v omlous Ptter Correltio ( u u ) ( v v ) [ ] 8

19 Decomosig H HGT MS, T38L64 GS Totl MS MS by Me Dierece MS by Ptter ritio Rtio o Stdrd Devitio omlous Ptter Correltio Totl RMS is rimrily comosed o MD i the lower strtoshere d P i the trooshere. HGT geerlly hs high omlous tter correltio. The orecst vrice is lower th tht o lysis i the lower trooshere d strtoshere, d lrger er the troouse. orecst vrice er troouse icreses with orecst led time. 9

20 Decomosig Troicl ector Wid RMS^, T38L64 GS Totl MS MS by Me Dierece MS by Ptter ritio Rtio o Stdrd Devitio omlous Ptter Correltio or troicl Wid, both MD d P re cocetrted er the troouse, d icrese with orecst led time. T38 GS is ot ble to miti wid vrice er the troouse, d hs stroger vrice everywhere else. Wid omlous tter correltio is much oorer th tht o HGT, d its quickly with orecst led time, esecilly i the lower trooshere. 0

21 Troicl ector Wid RMS, T574GS - T38GS, Q3Y00 Imlemettio Totl MS MS by Me Dierece MS by Ptter ritio Rtio o Stdrd Devitio omlous Ptter Correltio T475 hs weker wids due to stroger bckgroud diusio. RMS rom tter vritio is reduced becuse wid is smoother. Thus overll smller RMS (misledig?)

22 Summry RMS/MS c be t times misledig. Its iress s erormce metric deeds o the goodess o me dierece, stdrd devitio, d tter correltio. I tter correltio is low, RMS teds to wrd orecsts with smoother ields. The imlictio is tht RMS should ot be used or exteded WP orecsts d sesol orecsts either. RMS hs ote bee used s the oly metric to mesure model orecst erormce i the troics, esecilly or wid orecst. more comrehesive veriictio should t lest iclude MS, MS by Me Dierece, omlous Ptter Correltio, d Rtio o orecst rice to lysis rice.