Forecast Performance Assessment of a Kinematic and a Magnetohydrodynamic Solar Wind Model [Simulation meets Reality] 09 October 202 Integrity Service Ecellence Donald C. orquist AFRL/RVBXS Kirtland AFB, M
Presentation Outline Motivation for this study Models considered in this study Data for model initialization and forecast verification Method of forecast verification Results of forecast verification Conclusions reached from this study Acknowledgements 2
Motivation Comprehensive forecast verification informs forecasters who use the products as guidance Provides feedback to model authors on performance strengths and weaknesses Gives quantitative skill measures to modelers considering use of forecast products as forcing data what to epect Guides decision-makers in operational centers regarding incorporation or continuation of the model 3
Models Wang-Sheeley-Arge (WSA) solar wind model WSA corona model: potential field source surface model inputs photosphere Br maps, computes 2.5 Rs Br grid Schatten Current Sheet model etends Br to 5 Rs Vr specified from Br field at 5.0 Rs empirically -D kinematic code propagates Vr, Br polarity (±) to AU Enlil solar wind model 3-D magnetohydrodynamic model from OAA/SWPC Computational domain: ± 60 latitude, 2.5 Rs to. AU Computational grid: 2 lat.-lon., 0.67 Rs radial 2.5 Rs inner boundary specification: radial components of mag. field Br from WSA corona model, also empirical Vr 4
Data Model initialization Global Oscillation etwork Group (GOG) magnetogram Br maps from ational Solar Observatory lat.-lon., daily at 0 UTC, 2007-20 as available Uncorrected (Unc) and zero-point corrected (Cor) maps WSA corona model used to process maps to 5 Rs, 2.5 Rs Forecast verification Advanced Composition Eplorer (ACE) observations at L from ACE Science Center at Cal Tech Bmag, B from MAG used Level 2 (quality checked) for entire period Proton speed from SWEPAM used Level 2 through 0 Jun, Level 0 (unchecked) Jun end of period 5
Method Model / Init. Cond. Pairs: Enl-Unc, Enl-Cor, WSA-Unc, WSA-Cor Forecast variables verified: Vr (models) vs. proton speed (obs): solar wind (SW) speed Interplanetary magnetic field (IMF) polarity (models) vs. sign of B (obs): IMF polarity WSA forecast data used at 4.55 h output intervals (Δ t W ) Hourly ACE averaged to Δ t W centered at WSA output time ominal Enlil interval of 3.64 h rounded to 4 h interval (Δ t E ) Enlil 2.73 min interval output averaged to Δ t E Hourly ACE averaged to Δ t E centered at Δ t E / 2 ACE ave. polarity: if B /Bmag <., set to -999; ± otherwise 24 h obs set to -999 starting interval prior to shock arrival at L Seven-day forecasts verified separately by forecast days - 7 6
umber of Forecast Times Verified Daily forecasts by year: 2007 (363), 2008 (355), 2009 (357), 200 (358), 20 (355); 788 X 2 (Unc, Cor) 3576 in all umber of Forecast Times Verified by Forecast Day WSA/Enl Day Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 2007-SW 87/238 888/238 869/238 879/238 878/238 877/693 858/238 2007-IMF 698/95 73/95 697/97 69/97 687/97 693/97 688/99 2008-SW 852/207 840/207 867/207 848/207 85/207 846/207 83/207 2008-IMF 688/937 675/94 709/94 663/938 684/94 677/942 648/93 2009-SW 853/204 860/204 838/204 855/206 845/204 856/204 825/204 2009-IMF 63/89 634/89 622/85 62/88 60/87 634/82 599/820 200-SW 806/208 833/208 834/2084 827/2082 827/208 829/208 822/208 200-IMF 647/924 697/92 694/922 706/924 698/928 708/938 684/929 20-SW 740/2000 758/2004 755/200 750/995 740/995 767/995 746/2005 20-IMF 599/835 623/837 632/832 65/828 68/827 628/826 600/836 7
Forecast Verification Metrics Fcst Obs Difference (ifcst time inde): Mean: Mean Square: Standard Deviation: Absolute Mean: Skill Score (Recurrence): X X σ X SS X i i 2 2 X i i X R i X i i X 2 ( X R i 2 X ; R i X 2 ) i F i O i O i O 0 27 D i Skill Score (Persistence): SS P X 2 P 2 ; P i O i O i 0 8
Results Annual Statistics from ACE Observations 2007 2008 2009 200 20 Mean SW Speed (km/s) 44 450 365 406 422 Std Dev SW Speed (km/s, 4-h ave) 3 65 88 89 Mean IMF magnitude (nt) 4.50 4.25 3.94 4.75 5.29 o. of Shocks (24h pds removed) 6 3 4 5 2 o. of High Speed Events (HSEs)* 26 26 9 3 8 o. of IMF Polarity Changes (IPCs)* 58 48 69 7 47 *as determined from h ACE observations using Maceice (2009) algorithm 9
SW Speed Fcst Obs Difference Mean (% of Obs Mean) SW Speed Fcst - Obs Difference Mean (% of Obs Mean) 30 25 20 5 0 5 0 Enl-Unc Enl-Cor WSA-Unc WSA-Cor -5 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day 0
SW Speed Fcst Obs Difference Absolute Mean (% of Obs Mean) 32 28 24 SW Speed Fcst - Obs Difference Absolute Mean (% of Obs Mean) 20 6 Enl-Unc Enl-Cor WSA-Unc WSA-Cor 2 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day
SW Speed Skill Score Based on Recurrence SW Speed Skill Score Based on Recurrence 0.5 0-0.5 - -.5-2 -2.5-3 -3.5 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor 2
SW Speed Fcst Obs Difference Standard Deviation (km/s) 20 SW Speed Fcst - Obs Difference Standard Deviation (km/s) 0 00 90 80 Enl-Unc Enl-Cor WSA-Unc WSA-Cor 70 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day 3
4 Place Proper DISTRIBUTIO STATEMET Here Forecast and Observation Means, Standard Deviations, and Correlation Means: Standard Deviations: Correlation: Alternatively, we compute correlation of day-average (DA) forecasts and observations i Oi O i F i F, i O Oi i F F i O F 2 2 ) (, ) ( σ σ O F i i O F i O O F F r σ σ ) )( ( Distribution A: Approved for Public Release
Forecast and Observation SW Speed Standard Deviation (km/s) Forecast and Observation SW Speed Standard Deviation (km/s) 20 0 00 90 80 70 60 50 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor Obs 5
Day-Average SW Speed Forecast-Observation Correlation Day-Average SW Speed Forecast-Observation Correlation 0.8 0.7 0.6 0.5 0.4 0.3 0.2 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor 6
Percentage of Correct IMF Polarity Forecast Intervals Percentage of Correct IMF Polarity Forecast Intervals 84 8 78 75 72 69 Enl-Unc Enl-Cor WSA-Unc WSA-Cor 66 07: 2 3 4 5 6 7 08: 2 3 4 5 6 7 09: 2 3 4 5 6 7 0: 2 3 4 5 6 7 : 2 3 4 5 6 7 Year: Forecast Day 7
Analysis of High Speed Events (HSEs) and IMF Polarity Changes (IPCs) Owens et al. (2005) HSE criteria A net solar wind increase of 00 km/s in 48 hours HSEs within two days of each other considered one event Maceice (2009) HSE criteria Identify pds of fcst intervals w/ -day increase of 50 km/s A contiguous set is an HSE pd, pds < 8 h apart combined Accept HSE pds w/ min V sw 500 km/s, ma V sw 500 km/s Accept HSE pds that maintain 50 km/s day - for 2 h Accept HSE pds w/ ma min difference 200 km/s Maceice (2009) IPC criteria Convert B pol to ϕ-45, B pol - to ϕ35 Apply 60-h running mean to each ϕ in 7-day fcst period Set filtered ϕ to 35 if > 45, to -45 otherwise ID st, last fcst time, polarity of constant polarity pds (CPP) Iteratively remove CPP of τ.,.2,.3,.4,.5 days & separated from nearest CPP of like polarity by 3τ days 8
Seven-Day Enlil Forecast Initialized 00 UTC 2007055 vs. ACE Obs 700 600 Enlil-Unc SW Spd Enlil-Cor SW Spd ACE SW Spd SW Speed (km/s) 500 400 IPCs: 20070550 20070580 200705208 IMF Polarity (+/-) 300 200. 0 -. HSE 200705502 200705506 20070550 20070554 20070558 200705522 200705602 200705606 20070560 20070564 20070568 200705622 200705702 200705706 20070570 20070574 20070578 200705722 200705802 200705806 20070580 20070584 20070588 200705822 200705902 200705906 20070590 20070594 20070598 200705922 2007052002 2007052006 200705200 200705204 200705208 2007052022 200705202 200705206 20070520 20070524 20070528 200705222 Date/Time (YYYYMMDDHH) 200705502 200705506 20070550 20070554 20070558 200705522 200705602 200705606 20070560 20070564 20070568 200705622 200705702 200705706 20070570 20070574 20070578 200705722 200705802 200705806 20070580 20070584 20070588 200705822 200705902 200705906 20070590 20070594 20070598 200705922 2007052002 2007052006 200705200 200705204 200705208 2007052022 200705202 200705206 20070520 20070524 20070528 200705222 Date/Time (YYYYMMDDHH) Enlil IMF Polarity ACE IMF Polarity 9
a b 2007-20 HSE Contingency Tables Values shown are % of all forecast periods c d Enl-Unc Enl-Cor WSA-Unc WSA-Cor HSEs (Owens et al., 2005) HSEs (Maceice, 2009) Predicted Predicted Yes o Yes o O Yes 24 36 3 8 b o 0 30 3 76 s Yes 22 39 3 7 e o 8 3 2 78 r Yes 47 2 4 5 v o 25 6 9 72 e Yes 44 5 5 4 d o 22 9 6 75 Contingency Table Attributes HSEs (Owens, 2005) HSEs (Maceice, 2009) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) Enl-Unc 0.34 0.40 (2/5) 0.57 0.3 0.4 (/7) 0.29 Enl-Cor 0.32 0.36 0.49 0.3 0.5 0.25 WSA-Unc 0.56 0.80 (4/5).22 0.4 0.2 (/5) 0.68 WSA-Cor 0.54 0.75.2 0.20 0.26 0.58 20
Model - ICs Model - ICs # Y/Y FPs # Y/Y FPs HSE Arrival Time and Peak Speed for 2007-20 Y/Y Forecast Periods Owens, 2005 HSE Arrival Time Obs-Fcst (h) HSE Arrival Time Obs Fcst (h) HSE Peak Speed Fcst-Obs (km/s) Mean Abs Mean RMS Mean Abs Mean RMS Enl-Unc 437 4 28 40 9 76 96 Enl-Cor 385-2 30 4-27 79 96 WSA-Unc 862 4 28 39-7 8 0 WSA-Cor 794 28 38-33 75 93 Maceice, 2009 HSE Peak Speed Fcst Obs (km/s) Mean Abs Mean RMS Mean Abs Mean RMS Enl-Unc 57 4 22 29-28 48 62 Enl-Cor 59 6 9 24-39 53 65 WSA-Unc 66 3 27 36-3 56 66 WSA-Cor 84 2 28-30 59 75 2
2007-20 IPC Contingency Tables IPCs (Maceice, 2009) Values shown are % of all forecast periods Yes Predicted o Enl-Unc O Yes 36/3 28 b o 9 24 Enl-Cor s Yes 34/3 29 e o 7 27 WSA-Unc r Yes 36/2 26 v o 8 28 WSA-Cor e Yes 34/3 27 d o 7 29 Contingency Table Attributes a c IPCs (Maceice, 2009) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) Enl-Unc 0.49 0.56 (5/9) 0.70 Enl-Cor 0.49 0.54 0.65 WSA-Unc 0.5 0.58 0.7 WSA-Cor 0.50 0.56 0.67 b d 22
Count and Average Duration of All CPPs Predicted and Observed in 2007-20 o. Positive CPPs o. egative CPPs Ave. Duration (h) Enl-Unc 446 453 69 Enl-Cor 4 406 67 Obs (4h) 8 838 56 WSA-Unc 44 444 66 WSA-Cor 409 435 67 Obs (4.55 h) 806 827 56 23
Conclusions COR magnetograms reduced small (ecept 2009) pos. F-O diff. WSA V sw fcsts had 2-3% smaller abs mean F-O diff. than Enlil COR magnetograms reduced stnd dev of V sw fcsts in years with greatest observed stnd dev They increased fcst stnd dev when obs stnd dev was least Smaller F-O diff. for smoothed fcsts, larger for amplified fcsts F-O diff. were least at 3-5 days and increased greatly later V sw skill < recurrence in more variant years, > in less variance Models/ICs: no significant difference in IMF polarity fcst skill # FPs with predicted HSE / # FPs with obs HSE: in Owens, 2/5 (Enlil), 4/5 (WSA); in Maceice, /7 (Enlil), /5 (WSA) IPCs: ~ 5/9 (Enlil, WSA); 45-50% fewer CPPs predicted than obs; predicted CPPs average 0-2 h longer than obs 24
Acknowledgements ick Arge, David MacKenzie and Christina Lee of AFRL solar section for their assistance with the models Sean McManus of ational Solar Observatory for the corrected magnetograms ACE Science Center at Cal Tech for the ACE Level 3 observations DoD High Performance Computing Modernization Program for computer processing time on Cray, IBM supercomputers AFRL applied research program and the Space Weather Forecasting Laboratory for funding 25