Forecast Performance Assessment of a Kinematic and a Magnetohydrodynamic Solar Wind Model [Simulation meets Reality]

Size: px

Start display at page:

Download "Forecast Performance Assessment of a Kinematic and a Magnetohydrodynamic Solar Wind Model [Simulation meets Reality]"

Samantha Norris
6 years ago
Views:

meets Reality] 09 October 202 Integrity Service

1 Forecast Performance Assessment of a Kinematic and a Magnetohydrodynamic Solar Wind Model [Simulation meets Reality] 09 October 202 Integrity Service Ecellence Donald C. orquist AFRL/RVBXS Kirtland AFB, M

2 Presentation Outline Motivation for this study Models considered in this study Data for model initialization and forecast verification Method of forecast verification Results of forecast verification Conclusions reached from this study Acknowledgements 2

3 Motivation Comprehensive forecast verification informs forecasters who use the products as guidance Provides feedback to model authors on performance strengths and weaknesses Gives quantitative skill measures to modelers considering use of forecast products as forcing data what to epect Guides decision-makers in operational centers regarding incorporation or continuation of the model 3

4 Models Wang-Sheeley-Arge (WSA) solar wind model WSA corona model: potential field source surface model inputs photosphere Br maps, computes 2.5 Rs Br grid Schatten Current Sheet model etends Br to 5 Rs Vr specified from Br field at 5.0 Rs empirically -D kinematic code propagates Vr, Br polarity (±) to AU Enlil solar wind model 3-D magnetohydrodynamic model from OAA/SWPC Computational domain: ± 60 latitude, 2.5 Rs to. AU Computational grid: 2 lat.-lon., 0.67 Rs radial 2.5 Rs inner boundary specification: radial components of mag. field Br from WSA corona model, also empirical Vr 4

5 Data Model initialization Global Oscillation etwork Group (GOG) magnetogram Br maps from ational Solar Observatory lat.-lon., daily at 0 UTC, as available Uncorrected (Unc) and zero-point corrected (Cor) maps WSA corona model used to process maps to 5 Rs, 2.5 Rs Forecast verification Advanced Composition Eplorer (ACE) observations at L from ACE Science Center at Cal Tech Bmag, B from MAG used Level 2 (quality checked) for entire period Proton speed from SWEPAM used Level 2 through 0 Jun, Level 0 (unchecked) Jun end of period 5

6 Method Model / Init. Cond. Pairs: Enl-Unc, Enl-Cor, WSA-Unc, WSA-Cor Forecast variables verified: Vr (models) vs. proton speed (obs): solar wind (SW) speed Interplanetary magnetic field (IMF) polarity (models) vs. sign of B (obs): IMF polarity WSA forecast data used at 4.55 h output intervals (Δ t W ) Hourly ACE averaged to Δ t W centered at WSA output time ominal Enlil interval of 3.64 h rounded to 4 h interval (Δ t E ) Enlil 2.73 min interval output averaged to Δ t E Hourly ACE averaged to Δ t E centered at Δ t E / 2 ACE ave. polarity: if B /Bmag <., set to -999; ± otherwise 24 h obs set to -999 starting interval prior to shock arrival at L Seven-day forecasts verified separately by forecast days - 7 6

7 umber of Forecast Times Verified Daily forecasts by year: 2007 (363), 2008 (355), 2009 (357), 200 (358), 20 (355); 788 X 2 (Unc, Cor) 3576 in all umber of Forecast Times Verified by Forecast Day WSA/Enl Day Day 2 Day 3 Day 4 Day 5 Day 6 Day SW 87/ / / / / / / IMF 698/95 73/95 697/97 69/97 687/97 693/97 688/ SW 852/ / / /207 85/ /207 83/ IMF 688/ /94 709/94 663/ /94 677/ / SW 853/ / / / / / / IMF 63/89 634/89 622/85 62/88 60/87 634/82 599/ SW 806/ / / / / / / IMF 647/ /92 694/ / / / / SW 740/ / / / / / / IMF 599/ / /832 65/828 68/ / /836 7

8 Forecast Verification Metrics Fcst Obs Difference (ifcst time inde): Mean: Mean Square: Standard Deviation: Absolute Mean: Skill Score (Recurrence): X X σ X SS X i i 2 2 X i i X R i X i i X 2 ( X R i 2 X ; R i X 2 ) i F i O i O i O 0 27 D i Skill Score (Persistence): SS P X 2 P 2 ; P i O i O i 0 8

9 Results Annual Statistics from ACE Observations Mean SW Speed (km/s) Std Dev SW Speed (km/s, 4-h ave) Mean IMF magnitude (nt) o. of Shocks (24h pds removed) o. of High Speed Events (HSEs)* o. of IMF Polarity Changes (IPCs)* *as determined from h ACE observations using Maceice (2009) algorithm 9

10 SW Speed Fcst Obs Difference Mean (% of Obs Mean) SW Speed Fcst - Obs Difference Mean (% of Obs Mean) Enl-Unc Enl-Cor WSA-Unc WSA-Cor -5 07: : : : : Year: Forecast Day 0

11 SW Speed Fcst Obs Difference Absolute Mean (% of Obs Mean) SW Speed Fcst - Obs Difference Absolute Mean (% of Obs Mean) 20 6 Enl-Unc Enl-Cor WSA-Unc WSA-Cor 2 07: : : : : Year: Forecast Day

12 SW Speed Skill Score Based on Recurrence SW Speed Skill Score Based on Recurrence : : : : : Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor 2

13 SW Speed Fcst Obs Difference Standard Deviation (km/s) 20 SW Speed Fcst - Obs Difference Standard Deviation (km/s) Enl-Unc Enl-Cor WSA-Unc WSA-Cor 70 07: : : : : Year: Forecast Day 3

14 4 Place Proper DISTRIBUTIO STATEMET Here Forecast and Observation Means, Standard Deviations, and Correlation Means: Standard Deviations: Correlation: Alternatively, we compute correlation of day-average (DA) forecasts and observations i Oi O i F i F, i O Oi i F F i O F 2 2 ) (, ) ( σ σ O F i i O F i O O F F r σ σ ) )( ( Distribution A: Approved for Public Release

15 Forecast and Observation SW Speed Standard Deviation (km/s) Forecast and Observation SW Speed Standard Deviation (km/s) : : : : : Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor Obs 5

16 Day-Average SW Speed Forecast-Observation Correlation Day-Average SW Speed Forecast-Observation Correlation : : : : : Year: Forecast Day Enl-Unc Enl-Cor WSA-Unc WSA-Cor 6

17 Percentage of Correct IMF Polarity Forecast Intervals Percentage of Correct IMF Polarity Forecast Intervals Enl-Unc Enl-Cor WSA-Unc WSA-Cor 66 07: : : : : Year: Forecast Day 7

18 Analysis of High Speed Events (HSEs) and IMF Polarity Changes (IPCs) Owens et al. (2005) HSE criteria A net solar wind increase of 00 km/s in 48 hours HSEs within two days of each other considered one event Maceice (2009) HSE criteria Identify pds of fcst intervals w/ -day increase of 50 km/s A contiguous set is an HSE pd, pds < 8 h apart combined Accept HSE pds w/ min V sw 500 km/s, ma V sw 500 km/s Accept HSE pds that maintain 50 km/s day - for 2 h Accept HSE pds w/ ma min difference 200 km/s Maceice (2009) IPC criteria Convert B pol to ϕ-45, B pol - to ϕ35 Apply 60-h running mean to each ϕ in 7-day fcst period Set filtered ϕ to 35 if > 45, to -45 otherwise ID st, last fcst time, polarity of constant polarity pds (CPP) Iteratively remove CPP of τ.,.2,.3,.4,.5 days & separated from nearest CPP of like polarity by 3τ days 8

19 Seven-Day Enlil Forecast Initialized 00 UTC vs. ACE Obs Enlil-Unc SW Spd Enlil-Cor SW Spd ACE SW Spd SW Speed (km/s) IPCs: IMF Polarity (+/-) HSE Date/Time (YYYYMMDDHH) Date/Time (YYYYMMDDHH) Enlil IMF Polarity ACE IMF Polarity 9

20 a b HSE Contingency Tables Values shown are % of all forecast periods c d Enl-Unc Enl-Cor WSA-Unc WSA-Cor HSEs (Owens et al., 2005) HSEs (Maceice, 2009) Predicted Predicted Yes o Yes o O Yes b o s Yes e o r Yes v o e Yes d o Contingency Table Attributes HSEs (Owens, 2005) HSEs (Maceice, 2009) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) Enl-Unc (2/5) (/7) 0.29 Enl-Cor WSA-Unc (4/5) (/5) 0.68 WSA-Cor

21 Model - ICs Model - ICs # Y/Y FPs # Y/Y FPs HSE Arrival Time and Peak Speed for Y/Y Forecast Periods Owens, 2005 HSE Arrival Time Obs-Fcst (h) HSE Arrival Time Obs Fcst (h) HSE Peak Speed Fcst-Obs (km/s) Mean Abs Mean RMS Mean Abs Mean RMS Enl-Unc Enl-Cor WSA-Unc WSA-Cor Maceice, 2009 HSE Peak Speed Fcst Obs (km/s) Mean Abs Mean RMS Mean Abs Mean RMS Enl-Unc Enl-Cor WSA-Unc WSA-Cor

22 IPC Contingency Tables IPCs (Maceice, 2009) Values shown are % of all forecast periods Yes Predicted o Enl-Unc O Yes 36/3 28 b o 9 24 Enl-Cor s Yes 34/3 29 e o 7 27 WSA-Unc r Yes 36/2 26 v o 8 28 WSA-Cor e Yes 34/3 27 d o 7 29 Contingency Table Attributes a c IPCs (Maceice, 2009) CSIa/(a+b+c) PODa/(a+c) Bias(a+b)/(a+c) Enl-Unc (5/9) 0.70 Enl-Cor WSA-Unc WSA-Cor b d 22

23 Count and Average Duration of All CPPs Predicted and Observed in o. Positive CPPs o. egative CPPs Ave. Duration (h) Enl-Unc Enl-Cor Obs (4h) WSA-Unc WSA-Cor Obs (4.55 h)

24 Conclusions COR magnetograms reduced small (ecept 2009) pos. F-O diff. WSA V sw fcsts had 2-3% smaller abs mean F-O diff. than Enlil COR magnetograms reduced stnd dev of V sw fcsts in years with greatest observed stnd dev They increased fcst stnd dev when obs stnd dev was least Smaller F-O diff. for smoothed fcsts, larger for amplified fcsts F-O diff. were least at 3-5 days and increased greatly later V sw skill < recurrence in more variant years, > in less variance Models/ICs: no significant difference in IMF polarity fcst skill # FPs with predicted HSE / # FPs with obs HSE: in Owens, 2/5 (Enlil), 4/5 (WSA); in Maceice, /7 (Enlil), /5 (WSA) IPCs: ~ 5/9 (Enlil, WSA); 45-50% fewer CPPs predicted than obs; predicted CPPs average 0-2 h longer than obs 24

25 Acknowledgements ick Arge, David MacKenzie and Christina Lee of AFRL solar section for their assistance with the models Sean McManus of ational Solar Observatory for the corrected magnetograms ACE Science Center at Cal Tech for the ACE Level 3 observations DoD High Performance Computing Modernization Program for computer processing time on Cray, IBM supercomputers AFRL applied research program and the Space Weather Forecasting Laboratory for funding 25

A COMPARATIVE VERIFICATION OF FORECASTS FROM TWO OPERATIONAL SOLAR WIND MODELS (POSTPRINT)

AFRL-RV-PS- TP-2012-0006 AFRL-RV-PS- TP-2012-0006 A COMPARATIVE VERIFICATION OF FORECASTS FROM TWO OPERATIONAL SOLAR WIND MODELS (POSTPRINT) Donald C. Norquist and Warner C. Meeks 8 February 2012 Technical