PUBLICATIONS. Space Weather. GCR environmental models III: GCR model validation and propagated uncertainties in effective dose

Transcription

1 PUBLICATIONS RESEARCH ARTICLE Companion to Slaba and Blattnig [2014] doi: /2013sw and Slaba and Blattnig [2014] doi: /2013sw Key Points: BON2011 over estimates heavy ion flux at relevant energies BON2010 and Matthia models have similar overall uncertainty statements GCR model uncertainty may be reduced by calibrating to high energy data Correspondence to: T. C. Slaba, Citation: Slaba, T. C., X. Xu, S. R. Blattnig, and R. B. Norman (2014), GCR environmental models III: GCR model validation and propagated uncertainties in effective dose, Space Weather, 12, , doi:. Received 23 DEC 2013 Accepted 16 MAR 2014 Accepted article online 19 MAR 2014 Published online 7 APR 2014 GCR environmental models III: GCR model validation and propagated uncertainties in effective dose Tony C. Slaba 1, Xiaoing Xu 2, Steve R. Blattnig 1, and Ryan B. Norman 1 1 NASA Langley Research Center, Hampton, Virginia, USA, 2 Science Systems and Applications, Inc., Hampton, Virginia, USA Abstract This is the last of three papers focused on quantifying the uncertainty associated with galactic cosmic rays (GCR) models used for space radiation shielding applications. In the first paper, it was found that GCR ions with Z > 2 and boundary energy below 500 MeV/nucleon induce less than 5% of the total effective dose behind shielding. This is an important finding since GCR model development and validation have been heavily biased toward Advanced Composition Explorer/Cosmic Ray Isotope Spectrometer measurements below 500 MeV/nucleon. Weights were also developed that quantify the relative contribution of defined GCR energy and charge groups to effective dose behind shielding. In the second paper, it was shown that these weights could be used to efficiently propagate GCR model uncertainties into effective dose behind shielding. In this work, uncertainties are quantified for a few commonly used GCR models. A validation metric is developed that accounts for measurements uncertainty, and the metric is coupled to the fast uncertainty propagation method. For this work, the Badhwar-O Neill (BON) 2010 and 2011 and the Matthia GCR models are compared to an extensive measurement database. It is shown that BON2011 systematically overestimates heavy ion fluxes in the range GeV/nucleon. The BON2010 and BON2011 also show moderate and large errors in reproducing past solar activity near the 2000 solar maximum and 2010 solar minimum. It is found that all three models induce relative errors in effective dose in the interval [ 20%, 20%] at a 68% confidence level. The BON2010 and Matthia models are found to have similar overall uncertainty estimates and are preferred for space radiation shielding applications. 1. Introduction Quantifying the astronaut exposure from galactic cosmic rays (GCR) behind shielding is a key component of risk assessment [Cucinotta et al., 2013], especially for long-duration missions to the Moon or Mars. This requires computational models of the ambient GCR environment beyond the geomagnetic field, along with models for nuclear physics, particle transport, and biological response. The uncertainty of each of these components needs to be well understood in order to reliably assess the exposure level and associated risk. Several GCR models were reviewed in National Council on Radiation Protection and Measurements 153 [2006], and it was determined that the uncertainty of the available models was generally within 15%. This assessment is consistent with other studies [O Neill, 2006, 2010; Mrigakshi et al., 2012] and has been used in cancer risk models [Cucinotta et al., 2013]. However, it has been recently shown that if two models are evaluated over a common time period and transported through to effective dose, differences can easily exceed 50% [Mrigakshi et al., 2013; Slaba and Blattnig, 2014a]. This type of inconsistency indicates the need for a more rigorous validation approach with uncertainty metrics that are better tied to exposure quantities of interest for space radiation shielding applications. This is the last of three papers focused on addressing this need. In the first paper [Slaba and Blattnig, 2014a], a sensitivity analysis was performed to quantify the extent to which each GCR ion and energy, prior to entering any shielding material or tissue, contribute to effective dose behind shielding. It was found that GCR ions with Z > 2 and boundary energy below 500 MeV/nucleon induce less than 5% of the total effective dose behind shielding. Given that most of the GCR models are heavily calibrated and validated against measurements taken below 500 MeV/nucleon by the Advanced Composition Explorer/Cosmic Ray Isotope Spectrometer (ACE/CRIS) instrument [Stone et al., 1998], it is plausible for two GCR models to accurately reproduce the ACE/ CRIS data and have similar overall uncertainty statements while inducing very different effective dose values behind shielding. Weights were also provided that quantify the relative contribution of specific GCR ions and SLABA ET AL American Geophysical Union. All Rights Reserved. 233

2 energy groups to effective dose behind shielding. These values can be used to calibrate GCR model free parameters to reduce uncertainties in effective dose and provide guidance for future GCR measurements, nuclear cross-section measurements, and radiobiology experiments. In the second paper [Slaba and Blattnig, 2014b], a simple and computationally efficient method, utilizing the weights derived in the sensitivity analysis [Slaba and Blattnig, 2014a], was presented for propagating GCR model relative uncertainty distributions into effective dose behind shielding. This approach allows error bars and probability density functions (PDFs), representing GCR model uncertainties, to be easily placed around a point estimate of effective dose. The approach may alternatively be used to quantify the uncertainty in effective dose or other exposure quantities caused by uncertainty in the GCR boundary condition. This is particularly important in the present work, as it ties GCR model uncertainties directly to exposure quantities of interest for space radiation shielding applications. It would also be useful in transport code validation, as the method could be extended to place error bars around model results compared to integrated measurements [Zeitlin et al., 2013a, 2013b; Hassler et al., 2014]. Further, for validation purposes, it allows average GCR model uncertainty estimates to be computed according to the importance of each energy region (for effective dose) as opposed to regions for which there are the most measurements (i.e., ACE/CRIS energy region below 500 MeV/nucleon). All that is needed to perform the calculation are the weights which can be computed using the method developed by Slaba and Blattnig [2014a] and the GCR model relative uncertainty distributions in each energy group. GCR model relative uncertainty distributions are currently not available in the literature. Consequently, in the previous work addressing uncertainty propagation methods, assumed distributions were used (standard normal) with shape parameters corresponding to the often used 15% uncertainty estimate. In this work, those assumptions are replaced with distributions derived from comparisons between GCR model results and an extensive measurement database. The GCR models considered in this work are the Badhwar- O Neill 2010 [O Neill, 2010] and 2011 [Cucinotta et al., 2013] models, denoted as BON2010 and BON2011, respectively. The Matthia model is also included [Matthia et al., 2013]. In order to develop the relative uncertainty distributions for each GCR model, a detailed validation comparison is required. Often, the validation metrics that have been utilized to communicate overall model uncertainty were based on mean or root-mean-square (RMS) relative differences. Also, a recent independent evaluation of GCR models by Mrigakshi et al. [2012] utilized a chi-square test. This validation approach has the advantage of providing a single number that is easily understood and communicated. However, other issues must also be considered in the analysis and interpretation: 1. Distribution of uncertainties. Any average model uncertainty is determined from a distribution of uncertainties. If an uncertainty distribution is broad, the average value will include cancelation (i.e., a mix of positive and negative uncertainties). If an uncertainty distribution is shifted or heavily skewed, it may indicate a systematic model error (i.e., general overprediction of underprediction). 2. Measurement uncertainty. Model uncertainties must be greater than the error associated with the measurements used for validation [Trucano et al., 2006]. ACE/CRIS measurement errors are ~10% [CRIS], but high-energy balloon measurement error can be larger. A combined model uncertainty less than 15% would therefore necessitate exceedingly good agreement across all levels of solar activity to compensate for the inherent measurement uncertainty. 3. Biased experimental database. ACE/CRIS measurements dominate the available measurements used for GCR model development and validation. Average model uncertainty estimates and model free parameters are therefore heavily biased toward a region (Z > 5 and E < 500 MeV/nucleon) that induces less than 5% of the total effective dose behind shielding [Slaba and Blattnig, 2014a]. 4. Gaps in experimental data. Reported GCR model uncertainties are derived by comparing model results to available measurements. The model uncertainty for ions and energy regions with little or no measurements is not well characterized at this time. There is currently a lack of time-resolved continuous measurements for the ions and energy groups that are most important for integrated exposure quantities. It should also be pointed out that most of the GCR models used for radiation shielding applications have free parameters that are calibrated using all, or a large subset of, the available measurement data. In particular, the ACE/CRIS data set is used for parameter calibration in the BON2010, BON2011, and Matthia models. A subsequent comparison of these GCR models to the ACE/CRIS data will therefore inform users about how well the models fit data but communicates only limited information about the actual predictive capability of the SLABA ET AL American Geophysical Union. All Rights Reserved. 234

3 model. The latter can only be well determined by comparing to measurements not used in parameter calibration [Trucano et al., 2006]. Presently, there simply are not enough data to support a true independent validation since the models have been fit to most of the available data. It should therefore be recognized that current uncertainty estimates (as well as the estimates provided in this work) are based on data that are directly used in model parameter calibration. This paper is organized as follows. First, the collection of measurements used to validate GCR models is described, with particular attention given to the treatment of ACE/CRIS data. Second, an interval-based validation metric is developed that accounts for measurement uncertainty and allows for uncertainty distributions to be determined. Third, the metric is applied to the BON2010, BON2011, and Matthia models, and results are discussed. Fourth, validation results are weighted according the relative importance of each energy region to effective dose, and point estimates of effective dose from nominal GCR model results are modified to include error bars, as discussed previously [Slaba and Blattnig, 2014b]. 2. Measurement Database The data used for this study include measurements of ion-specific differential flux rates up to 1 TeV/nucleon. Higher-energy measurements are available [e.g., Derbina et al. [2005]] but were neglected because these energies contribute negligibly to effective dose behind shielding. Measurements that did not clearly exclude low-energy anomalous cosmic rays (ACRs) [e.g., Garcia-Munoz et al. [1975]] were neglected because the models considered here do not include ACR; also, these low-energy ions are readily stopped within a few g/cm 2 of shielding and contribute negligibly to exposure quantities considered herein. A summary of the compiled measurement database is given in Table 1. The measurement error, given in the rightmost column is defined as the full width of the error bar. For example, a reported measurement error of +4.5% is represented as a 9% error in Table 1. Most of the measurements included in the database have a full error less than 30%. With the exception of ACE/CRIS, all of measurement data were obtained directly from the published reference. TheACE/CRIS datawereobtained from[ace/cris database, 2013] using the 27 day average. Data for the year 1997 were not included because of calibration issues between August and December of that year reported in the documentation. An estimate for the measurement error was obtained from the documentation for each ion and energy band. After viewing the 27 day average data, it was clear that moderately high statistical errors, presumably associated with low count rates, were present for less abundant ions such as Co, Mn, and others. Direct inclusion of these data into the validation approach would inflate reported model uncertainties; therefore, further data processing was performed. For each ion, the data for successive 27 day periods were integrated together until the statistical variation in the time period was less than 25%. This reduced the overall number of data points, but the resulting data were more statistically meaningful. An example of selected ion fluxes drawn directly from the database along with the processed data and model results is given in Figure 1. It is also interesting to note that the ACE/CRIS data make up 82% of the entire database, while measurements in the energy region greater than 500 MeV/nucleon make up only 16% of the database. Clearly, model uncertainties derived from straightforward averaging will be heavily weighted by ACE/CRIS comparisons. 3. Interval-Based Validation Metrics In this section, the metrics used to quantify model uncertainty are described. There are two main classifications of uncertainty: variability and incertitude [Sentz and Ferson, 2011]. Variability arises from the stochastic nature of a system and may only be better understood, but not generally reduced, with further measurements [Ferson and Haagos, 2004]. Incertitude, or epistemic uncertainty, arises from a lack of knowledge due to measurement uncertainty, sparse measurement data, or an incomplete understanding of physical mechanisms [Ferson and Haagos, 2004]. GCR model uncertainty for a specified historical period in time is largely driven by epistemic uncertainty, and consequently, an interval-based approach for uncertainty quantification and propagation is appropriate [Sentz and Ferson, 2011; Ferson et al., 2007]. By way of comparison, past validation studies have typically compared model results directly to nominal measurement values without regard for the range of measured values that may occur within reported error bars. An interval-based approach disregards the nominal measurement value and compares model results to the range of measured values that may occur. No assumption is made about the distribution of measured values represented by the error bars [Sentz and Ferson, 2011]. SLABA ET AL American Geophysical Union. All Rights Reserved. 235

4 Table 1. Summary of Collected Measurements Name a Flight Time Ions (Z) Energy (GeV/nucleon) Data Points Median Error ACE/CRIS Satellite 1998 to present % AMS STS , % ATIC-2 Balloon , 2, 6, 8, 10,,14, % BESS Balloon , , % CAPRICE Balloon 1994, , % CREAM-II Balloon , 10, 12, 14, % HEAO-3 Satellite % IMAX Balloon , % IMP-8 Satellite , 8, 10, 12, % LEAP Balloon , % MASS Balloon , % PAMELA Satellite , % TRACER Balloon , 10, 12,,20, % Lezniak Balloon , 16, 20, % Minagawa Balloon , % Muller STS , 8, 10, 12, % Simon Balloon % a Each name (from top to bottom) corresponds to the following references (separated by semicolons), respectively: Stone et al. [1998]; Alcaraz et al. [2000a, 2000b]; Panov et al. [2009]; Shikaze et al. [2008]; Boezio et al. [1999, 2003]; Ahn et al. [2009]; Engelmann et al. [1990]; Menn et al. [2000]; Garcia-Munoz et al. [1977]; Seo et al. [1991]; Bellotti et al. [1999]; Adriani et al. [2011, 2013]; Ave et al. [2008]; Lezniak and Webber [1978]; Minagawa [1981]; Muller et al. [1991]; Simon et al. [1980]. The interval analysis used in this work is an extension of the metrics developed by Norman and Blattnig [2010, 2013]. A graphical representation of the symbols and nomenclature used to represent experimental data is shown in Figure 2. Consider a measurement taken over the energy interval (E 0 ΔE, E 0 + ΔE + ) such that the nominal measured value, M 0, is expected to fall within the interval (M 0 ε, M 0 + ε + ). The average model result over the energy interval is given by F ¼ ΔEþ ΔE ϕ ð E 0 þ E ÞdE ΔE þ þ ΔE ; (1) where ϕ(e) is the ion differential flux. The model relative difference compared to the nominal measurement is defined as R D ¼ F M 0 M 0 : (2) Figure 1. (left) Mg and (right) Ca differential flux rates from ACE/CRIS measurements along with BON2010, BON2011, and Matthia model results. The red symbols with error bars were obtained by integrating the 27 day avg. meas. data (green dashed curves) obtained from ACE/CRIS database. SLABA ET AL American Geophysical Union. All Rights Reserved. 236

5 Figure 2. Graphical representation of typical measurement for GCR differential flux. Equation (2) has been typically used in past validation studies [Matthia et al., 2013; Mrigakshi et al., 2012; O Neill, 2006, 2010; National Council on Radiation Protection and Measurements, 2006] and compares model results directly to the nominal measurement value without regard for the range of measured values that may occur within the error bars (i.e., it does not account for measurement error). The interval-based metric is given by the equations U lower ¼ F ½ M 0 þ ε þ Š M 0 þ ε þ ; (3) U upper ¼ F ½ M 0 ε Š M 0 ε : (4) The quantities U lower and U upper are the relative differences between the model result and the bottom and top of the error bars shown in Figure 2, respectively, and define an interval of model uncertainties. The width of the interval may be approximately interpreted as the measurement uncertainty. Throughout the rest of the text, the interval defined by [U lower, U upper ] is referred to as the model uncertainty interval and may be written in usual bracketed interval notation. The measurement uncertainty, considered later in this paper, is given by U exp ¼ ε þ ε þ : (5) M 0 The following example illustrates the usefulness of the interval-based analysis provided by equations (3) and (4). Suppose that a given model result is identical to a nominal measurement value and that the experimental uncertainty is +10%. Equation (2), and most of the past validation studies, would indicate that the model uncertainty is zero, since the model result and nominal measurement value are identical. However, the actual quantity being measured may fall anywhere within the experimental error, and consequently, the model could actually be off by +10%. One arrives at a similar situation in the case where a model result falls outside the measurement error bars. Either way, it is clear that if the experimental error is not included, model uncertainties may be significantly underestimated. Equations (2) (4) may all be averaged over a set or subset of measurements to express model uncertainties in a single number or an interval. The distribution of these values may also be determined if enough data are available. 4. Validation Results and Discussion 4.1. Model Relative Uncertainty In this section, validation results are given using the metrics defined in section 3. The average model relative difference and uncertainty interval using the entire measurement database are given in Table 2. These values show general trends of each model with respect to Table 2. Average GCR Model Relative Difference and all measurements but should not be taken directly Uncertainty Intervals Using All Measurements a as the expected model uncertainty. The average Model R D U lower ; U upper values are heavily biased toward regions that BON % [ 15%, 3%] currently have the most measurements. As already BON2011 4% [ 2%, 11%] mentioned, the measurement database is Matthia 1% [ 5%, 8%] dominated by the ACE/CRIS data set covering an a These values are heavily biased toward the low-energy region covered by ACE/CRIS. The average experimental uncertainty, evaluated using equation (5), is 12%. energy region that contributes little to exposure quantities behind shielding. The range of uncertainties, expressed by a distribution, is also SLABA ET AL American Geophysical Union. All Rights Reserved. 237

6 not represented by average values alone. Nonetheless, useful information can be derived from Table 2. First, the average relative difference metric, R D,isin reasonable agreement with previous assessments of the models [National Council on Radiation Protection and Measurements, 2006; O Neill, 2010; Matthia et al., 2013]. Second, the BON2010 model appears to systematically underestimate the available measurements. Last, the average measurement error of 12%, computed directly with equation (5), is consistent with the width of the uncertainty intervals (right column). The cumulative distribution functions (CDFs) for the upper and lower bounds, sometimes referred to as p-boxes [Ferson and Haagos, 2004], are shown in Figure 3 and highlight some Figure 3. Cumulative distribution of GCR model uncertainties. general trends of the models and validation metrics. First, since nominal measurement values typically appear at the midpoint of the error bars, the relative difference CDF (red dashed curve) falls in the middle of the model uncertainty interval (shaded region). The horizontal width of the shaded region, which may be approximately interpreted as the measurement uncertainty, stays near 10%. Second, even though the average relative differences associated with the BON2011 and Matthia models are small, it is clear in Figure 3 that the distribution of errors is broad, and errors in excess of 15% can occur with moderate probability. For example, 22% of the BON2011 model uncertainties and 28% of the Matthia model uncertainties exceed +15%. This could also be restated as the Matthia and BON2011 model uncertainties are within +15% at an approximate 75% confidence level (CL). For the BON2010 model, the average relative difference is slightly larger than the other two models, and the CDF is shifted to the left of zero on the horizontal axis, indicating a systematic model underprediction. These findings are consistent with past validation studies [Mrigakshi et al., 2012; Cucinotta et al., 2013]. Figure 4. Average relative model uncertainties separated into energy groups. The relationship between R D, U lower, and U upper is shown for a single data point. The boxed number indicates the number of measurements in each energy bin. In Figure 4, the average relative model uncertainties are separated into the energy groups previously defined by Slaba and Blattnig [2014a]. The boxed number appearing along the horizontal axis indicates the number of data points in each energy group. The energy region below 500 MeV/nucleon has by far the largest number of data points, comprising 84% of the entire database. It is not surprising that the results in the low-energy bins are highly correlated with the results shown in Table 2 and Figure 3 and seem to indicate that BON2010 is less accurate than the other SLABA ET AL American Geophysical Union. All Rights Reserved. 238

7 Figure 5. Average relative model uncertainty separated into energy and charge groups. The symbols and error bars have the same meaning as in Figure 4. two models. However, in the energy regions above 500 MeV/nucleon that contribute the most to effective dose, it appears that the BON2010 is more accurate than the other two models and that the BON2011 has the largest uncertainty of all three models. This is further supported by the results shown in Figure 5, where the uncertainties have been separated according to energy group and charge group. In the highest two energy bins, the BON2010 is clearly more accurate than the other two models, and in the energy region between 500 MeV/nucleon and 4 GeV/nucleon, the BON2011 is clearly the least accurate model. It is also interesting to note that even though the BON2011 appears to be more accurate than BON2010 below 500 MeV/nucleon in Figure 4, Figure 5 suggests that this may be due in part to competing errors for Z = 1 and Z =2. The data presented in Table 2 and Figures 3 5 do not directly address the ability of the models to reproduce past solar modulation (time dependence). The ACE/CRIS data set covers an energy region that is heavily influenced by solar activity. It is also the only data set included in the validation database that continuously spans a time period between solar minimum and solar maximum. Comparing the models only to the ACE/ CRIS data as a function of time will therefore give some indication of how well past solar modulation (time dependence) is represented in the models. It is important to note that these comparisons only quantify the ability of models to reproduce past solar activity as measured by the ACE/CRIS data; the predictive capability of the models for future solar activity is not being addressed and would depend on the ability to predict future solar activity. The average relative model uncertainties, compared only to ACE/CRIS data, are shown in Figure 6 as a function of time from 1998 to The values in parentheses shown in the legend of Figure 6 are the timeaveraged relative model uncertainties. The largest uncertainties for the BON2010 model occur near the 2000 solar maximum and solar minimum. The discrepancies shown near the solar minimum are consistent with those already observed by Mrigakshi et al. [2012]. The discrepancies near the 2000 solar maximum are not unexpected and may be somewhat attributed to sporadic solar activity in that period. In fact, even the measurement uncertainty (error bars) is slightly amplified in that region due to statistical fluctuations. The overall time-averaged relative model uncertainties shown in parentheses in the legend of Figure 6 indicate that the BON2010 and BON2011 are comparable, with each model showing significant uncertainties in specific time periods. Overall, it appears that the Matthia model is the most accurate in this comparison, with average model uncertainties in each year within +10% Weighted Model Relative Uncertainty Slaba and Blattnig [2014a] developed a set of weights quantifying the relative contribution of each boundary energy and ion to effective dose behind shielding. It was subsequently shown [Slaba and Blattnig, 2014b] that these weights could be easily used to propagate GCR model uncertainties into effective dose values behind shielding. These weights may be combined with the GCR model uncertainty distributions discussed in the previous section to place error bars and probability distributions around nominal effective dose values. In this section, both sets of results are given and discussed. SLABA ET AL American Geophysical Union. All Rights Reserved. 239

8 Using the energy groups shown in Figure 4, the relative uncertainty on effectivedoseinducedbyuncertainties in the GCR boundary condition is given by [Slaba and Blattnig, 2014b] Figure 6. Average relative model uncertainty as a function of time. Only ACE/CRIS data were used in this comparison. The time-averaged relative model uncertainties are shown in parentheses in the legend. The symbols and error bars have the same meaning as in Figure 4. U EffDose ½uŠ ¼ 1 X Z X Z X 5 ¼1 X 5 ¼1 w ðzþ ð Þ 1 þ u Z w ðzþ ð Þ 1 þ u Z ; (6) where w ðzþ is the weight for ion Z in energy group [Slaba and Blattnig, 2014a] and u ðzþ is a sampled uncertainty value for boundary ion Z in energy group. The sum of weights over all energy groups and ions is unity. Equation (6) would need to be evaluated for many sampled uncertainty values in order to construct a meaningful distribution for U EffDose. Similarly, the distribution of effective dose values associated with uncertainties in the GCR boundary condition may be rapidly evaluated using X ed ½uŠ ¼ D 0 Z X 5 ¼1 w ðzþ ð Þ ; (7) 1 þ u Z where D 0 is the effective dose induced by the nominal GCR model result. The functional dependence, [u], is included in equations (6) and (7) to provide clarity in subsequent plots and discussion. Recall from the previous section that three uncertainty metrics were defined (R D, U lower, and U upper ) and evaluated along with a definition of experimental uncertainty (U exp ). The distribution of any of these metrics could be used for sampling u ðzþ. For example, U EffDose [R D ] indicates that the distribution of R D values was used to sample u ðzþ. Similarly, U EffDose [U exp ] indicates that the distribution of U exp values was used and would directly quantify the uncertainty in effective dose induced by GCR experimental uncertainty. While one would like to use ion-specific weights in the evaluation of equation (6), there are not enough measurements to support this approach, and underlying uncertainty distributions for specific ions and energy groups may not be well defined or even exist. Therefore, the charge groups used by Slaba and Blattnig [2014a] (Z =1,2,3 10, 11 20, and 21 28) are used in the present analysis. This ensured that for each energy group and charge group, there was sufficient measurement data to make the uncertainty distributions meaningful. The same problem occurs when one attempts to apply time-specific weights to time-specific measurements. For example, the solar minimum weights from Slaba and Blattnig [2014a] should ideally be used with uncertainty distributions derived from measurements taken over the same solar minimum time period. In general, this is not possible. While ACE/CRIS covers both a solar minimum and solar maximum time period, there are measurement gaps in these same periods for protons, alphas, and high-energy heavy ions. Consequently, one is only able to combine time-specific weights with uncertainty distributions derived by comparison to the entire measurement database (i.e., covering all time periods). This is partially compensated by the fact that higher-energy regions that contribute to effective dose are less heavily influenced by solar modulation. SLABA ET AL American Geophysical Union. All Rights Reserved. 240

9 Figure 7. Impact of correlated and uncorrelated sampling on PDF of U EffDose [R D ] values for the BON2010 model. Weights for solar minimum conditions and 20 g/cm 2 of aluminum shielding were used. The uncertainty values in equation (6) are sampled from a distribution, and consequently, correlations between ion and energy groups must be considered as previously discussed [Slaba and Blattnig, 2014b]. Uncorrelated sampling allows for cancelation to occur, resulting in a narrow distribution of uncertainties (or exposure quantities), as shown in Figure 7. Further, results from uncorrelated sampling inherently depend on the manner in which the sampling domain is subdivided. For example, greater cancelation would occur if the energy domain were separated into 100 groups instead of only 5. In order to compute correlation coefficients between each charge and energy group, timeresolved measurements are required. The ACE/ CRIS data set provides this time dependence for heavy ions in the lowest energy groups, but there is insufficient data for protons and alphas (over all energies) and high-energy heavy ions to completely determine all correlations. Based on comparisons to ACE/CRIS data, it appears reasonable to assume that model uncertainties in specific ion and energy groups will have a nonnegligible degree of correlation, as shown in Table 3. Note that fully correlated data have a coefficient of 1, uncorrelated data have a coefficient of 0, and anticorrelated data have a coefficient of 1. Fully correlated sampling will be used in all subsequent results. In Figure 8, the relative uncertainty on effective dose induced by GCR model uncertainties is shown. The weights provided in Slaba and Blattnig [2014a, Table 2] for each energy and charge group were used. These weights correspond to solar minimum conditions and 20 g/cm 2 aluminum shielding. The BON2011 and Matthia models are noticeably shifted to the right of zero, indicating that these models will lead to systematic overpredictions of effective dose. This finding is consistent with the results in Figures 4 and 5, showing that the BON2011 and Matthia models systematically overpredict measurements in the energy regions of importance for effective dose. The BON2010, on the other hand, is slightly shifted to the left, indicating a small systematic underprediction of effective dose. These results are in contrast to those shown in Figure 3 where the BON2010 model appears to noticeably underestimate measurements (CDF shifted to left), and the BON2011 and Matthia models show minimal systematic error. The difference in these two plots is that the latter accounts for the relative contribution of each GCR ion and energy group to effective dose. Therefore, uncertainties obtained through comparison to ACE/CRIS data dominate the CDFs in Figure 3 but have only a small impact on the CDFs in Figure 8. Specific values from Figure 8 have been summarized in Table 4. The median values were drawn from the U EffDose [R D ] distributions. The lower bound of each confidence level is drawn from the U EffDose [U lower ] distribution, and the upper bound is drawn from the U EffDose [U upper ] distribution. Reporting the Table 3. Correlation Coefficients Between First Two Energy Groups for Each Heavy Ion Charge Group a Charge Group BON2010 BON2011 Matthia Z = Z = Z = a These coefficients were derived from comparisons to ACE/CRIS data. confidence levels in this way ensures that the measurement error is included. In Figure 9, effective dose is plotted as a function of aluminum shielding thickness for the October 1976 solar minimum. The red line in the plot is the effective dose induced by the nominal GCR model boundary condition. The blue line with error bars shows the distribution SLABA ET AL American Geophysical Union. All Rights Reserved. 241

10 median and standard deviation evaluated with equation (7). The percent value explicitly shown above and below the error bars at 10 g/cm 2 and 50 g/cm 2 is the percent difference between the extreme value of the error bar and the median value. Figure 8. Relative uncertainty on effective dose behind 20 g/cm 2 during solar minimum conditions induced by GCR model uncertainties. Note that the data given in Table 4 will not be the same as the percentages shown in Figure 9. The data in Table 4 are representative values drawn from a distribution of relative uncertainties. These values include systematic errors which may shift the CL intervals toward positive or negative values. The percentages in Figure 9 are relative differences between representative values drawn from a distribution of effective dose values. These values simply measure the relative distance between the CL bounds and the median value of the distribution. At large shielding thicknesses, beyond 40 g/cm 2, the effective dose induced by all three GCR models are in reasonable agreement. At these depths, the effective dose is mainly delivered by GCR protons with high incident energy [Slaba and Blattnig, 2014a] where all three GCR models perform reasonably well, as showninfigure5.forthinnershields,lessthan10g/cm 2,asmuchas40%oftheeffectivedose,is delivered by high-energy GCR ions with Z > 2ions[Slaba and Blattnig, 2014a]. Figure 5 shows that the BON2011 model noticeably overestimates heavy ion fluxes at high energies, which leads directly to an overestimate of effective dose (and increased uncertainty) at small shielding thicknesses, as shown in Figure 9. In general, Table 4 and Figures 8 and 9 suggest that the BON2010 and Matthia models induce relative errors on effective dose in the interval [ 20%, 20%] at a 68% CL. The BON2011 model induces slightly larger uncertainties. These assessments make it difficult to distinguish between the models, and therefore, results in previous figures also need to be considered. For example, Figure 5 shows that the BON2011 model has noticeable errors for heavy ions in the 500 MeV/nucleon to 4 GeV/nucleon energy range, and Figure 6 shows that the model has large errors in reproducing past solar activity. Based on the current available data, and the present validation approach which ties model uncertainties directly to exposure quantities of interest, the BON2010 and Matthia models are preferred for space radiation shielding applications. It is expected that recalibration of free parameters in the Matthia and BON2010 and BON2011 models away from ACE/CRIS measurements and toward energy regions and ions that contribute heavily to effective dose behind shielding would directly reduce the propagated uncertainties. Table 4. Relative Uncertainty on Effective Dose Behind 20 g/cm 2 During Solar Minimum Conditions as a Result of GCR Model Uncertainty a Model Median 68% CL 95% CL BON2010 6% [ 23%, 17%] [ 34%, 55%] BON2011 9% [ 13%, 32%] [ 32%, 74%] Matthia 8% [ 10%, 29%] [ 23%, 62%] a These values were drawn from the distributions shown in Figure 8. The median values are drawn from the UEffDose [R D ] distribution. The lower bound of each confidence level (CL) is drawn from the U EffDose [U lower ] distribution, and the upper bound is drawn from the U EffDose [U upper ] distribution. SLABA ET AL American Geophysical Union. All Rights Reserved. 242

11 Figure 9. Effective dose versus aluminum shielding thickness for October 1976 solar minimum. The red line is the effective dose induced by the nominal GCR model boundary condition. The blue line with error bars is the distribution median and standard deviation (68% CL) evaluated with equation (7). 5. Summary and Conclusions This is the last of three papers focused on rigorously quantifying GCR model uncertainty and the impact of those uncertainties on effective dose behind shielding. The first paper presented results from a sensitivity analysis establishing that measurements from ACE/CRIS cover an energy region that contributes little to exposure quantities behind shielding, yet these measurements are heavily used in model development and validation. The use of these data is necessary at this time due to the lack of time-resolved measurements for protons and alphas at higher energies; however, it has now been demonstrated that care needs to be taken to avoid heavily biasing model parameters and validation metrics toward these measurements. An important product from the sensitivity analysis is the set of weights quantifying the relative contribution of boundary ions and energies to effective dose behind shielding. It was later shown that these weights can be used to efficiently propagate GCR model uncertainties into effective dose behind shielding; they may also be an important tool in better calibrating free model parameters. The second paper discussed two methods for propagating GCR model uncertainties into effective dose behind shielding. Uncertainty propagation quantifies the expected variation in exposure quantities that may occur as a result of boundary condition uncertainties. A simple and efficient propagation method was developed utilizing the weights developed in the first work for propagating boundary condition uncertainties into effective dose values. This allowed a distribution of effective dose values to be determined, along with the uncertainty distribution in effective dose induced by boundary condition uncertainties. The final product of the first two papers is a framework that can be applied to any GCR model for rigorously and efficiently quantifying the expected variation or uncertainty induced in effective dose or other exposure quantities behind shielding. In this way, GCR model uncertainties have been directly tied to exposure quantities of interest for space radiation shielding applications. In this final work, the BON2010, BON2011, and Matthia GCR models were considered for validation. First, the measurements collected for the validation study were described. In general, the open literature was surveyed and measurements below 1 TeV/nucleon were assembled. Extra processing was performed on the ACE/CRIS data set to control statistical variation associated with low count rates for less abundant ions. Second, the interval-based validation metrics applied in this study were described. The interval approach inherently accounts for measurement uncertainty. Inclusion of the measurement uncertainty in the analysis is important since models cannot be more accurate than the measurements used for validation. Finally, the results from the validation study were presented. The average relative uncertainties computed for the BON2010, BON2011, and Matthia models was found to be consistent with previously published uncertainty estimates [Matthia et al., 2013; O Neill, 2010; National Council on Radiation Protection and Measurements, 2006]. The cumulative distribution of uncertainties was shown for all three models, and it was SLABA ET AL American Geophysical Union. All Rights Reserved. 243

12 found that even though average uncertainty estimates fall within +15%, the distribution of uncertainties is quite broad. Approximately 22% of the BON2011 model uncertainties and 28% of the Matthia model uncertainties exceed +15%. A systematic underprediction of the measurements by the BON2010 model was also observed. Validation results were then segregated into the energy groups and charge groups. It was found that the BON2011 and Matthia models are most accurate for heavy ions in the energy region below 500 MeV/nucleon, indicating close agreement to the ACE/CRIS data set. The BON2011 model was found to be less accurate than the other two models in the higher-energy regions that contribute to effective dose. The BON2010 and BON2011 both showed noticeable errors in reproducing past solar activity. The Matthia model was clearly superior in this regard, showing average relative errors within +10% when compared against ACE/CRIS data between 1998 and As a final comparison, the uncertainty distributions for each model were adusted using the weights derived from the sensitivity analysis [Slaba and Blattnig, 2014a]. The adustment removes the inherent bias associated with the large number of ACE/CRIS measurements, as compared to the number of measurements in highenergy regions. The adusted distributions incorporate the relative importance of the ion and energy region of each measurement to effective dose. After applying the weights using the uncertainty propagation equation developed previously [Slaba and Blattnig, 2014b], it was found that the BON2010 and Matthia models induce uncertainty in effective dose in the interval [ 20%, 20%] a 68% CL. The BON2011 model induces slightly larger uncertainties. Based on the variation induced in effective dose by the models, it is hard to distinguish between the models. Consequently, this information should be considered simultaneously with the other validation results presented herein. The BON2011 model shows clear and systematic errors for heavy ions at energies that are important for effective dose. This model is also lacking in reproducing past solar activity. The BON2010 and Matthia models are hard to distinguish in most of the present comparisons. Future GCR model development could recalibrate free parameters away from ACE/CRIS data and toward measurements that cover energy regions of importance to effective dose behind shielding. Such an effort would directly reduce the error bars shown in Figure 9. Acknowledgments This work was supported by the Human Research Program under the Human Exploration and Operations Mission Directorate of NASA and by NASA grant NNL11AA00B. The authors would like to thank Pat O Neill for providing the BON2011 source code and Daniel Matthia for providing the Matthia source code. The data presented in this paper may be obtained by contacting the authors. References ACE/CRIS database (2013), 23 Jan. [Available at Adriani, O., et al. (2011), PAMELA measurements of cosmic-ray proton and helium spectra, Science, 332, Adriani, O., et al. (2013), Time dependence of the proton flux measured by PAMELA during the 2006 July 2009 December solar minimum, Astrophys. J., 765, 1 8. Ahn, H. S., et al. (2009), Energy spectra of cosmic-ray nuclei at high energies, Astrophys. J., 707, Alcaraz, J., et al. (2000a), Cosmic protons, Phys. Lett. B, 490, Alcaraz, J., et al. (2000b), Helium in near Earth orbit, Phys. Lett. B, 494, Ave, M., P. J. Boyle, F. Gahbauer, C. Hoppner, J. R. Horandel, M. Ichimura, D. Muller, and A. Romero-Wolf (2008), Composition of primary cosmic-ray nuclei at high energies, Astrophys. J., 678, Bellotti, R., et al. (1999), Balloon measurements of cosmic ray muon spectra in the atmosphere along with those of primary protons and helium nuclei over midlatitude, Phys. Rev. D, 60, Boezio, M., et al. (1999), The cosmic-ray proton and helium spectra between 0.4 and 200 GV, Astrophys. J., 518, Boezio, M., et al. (2003), The cosmic-ray proton and helium spectra measured with the CAPRICE98 balloon experiment, Astroparticle Physics, 19, CRIS documentation (2013), 23 Jan. [Available at Cucinotta, F. A., M. Y. Kim, and L. J. Chappell (2013), Space radiation cancer risk proections and uncertainties NASA Technical Paper Derbina, V. A., et al. (2005), Cosmic-ray spectra and composition in the energy range of TeV per particle obtained by the RUNJOB experiment, Astrophys. J., 628, L41 L44. Engelmann, J. J., et al. (1990), Charge composition and energy spectra of cosmic-ray nuclei for elements from Be to Ni. Results from HEAO-3-C2, Astron. Astrophys., 233, Ferson, S., and J. G. Haagos (2004), Arithmetic with uncertain numbers: Rigorous and (often) best possible answers, Reliability Eng. Syst. Safety, 85, Ferson, S., V. Kreinovich, J. G. Haagos, W. Oberkampf, and L. Ginzburg (2007), Experimental uncertainty estimation and statistics for data having interval uncertainty. Sandia Report SAND Garcia-Munoz, M., G. M. Mason, and J. A. Simpson (1975), The anomalous 4 He component in the cosmic-ray spectrum at <50 MeV per nucleon during , Astrophys. J., 202, Garcia-Munoz, M., G. M. Mason, J. A. Simpson, and J. P. Wefel (1977), Charge and energy spectra of heavy cosmic rays at intermediate energies, 15th International Cosmic Ray Conference, Plovdiv, Bulgaria. Hassler, D. M., et al. (2014), Mars surface radiation environment measured with the Mars Science Laboratory s curiosity rover, Science, 343, , doi: /science SLABA ET AL American Geophysical Union. All Rights Reserved. 244

13 Lezniak, J. A., and W. R. Webber (1978), The charge composition and energy spectra of cosmic-ray nuclei from 3000 MeV per nucleon to 50 GeV per nucleon, Astrophys. J., 233, Matthia, D., D. Berger, A. I. Mrigakshi, and G. Reitz (2013), A ready-to-use galactic cosmic ray model, Adv. Space Res., 51, Menn, W., et al. (2000), The absolute flux of protons and helium at the top of the atmosphere using IMAX, Astrophys. J., 533, Minagawa, G. (1981), The abundances and energy spectra of cosmic ray iron and nickel at energy from 1 to 10 GeV per amu, Astrophys. J., 248, Mrigakshi, A. I., D. Matthia, T. Berger, G. Reitz, and R. W. Wimmer-Schweingruber (2012), Assessment of galactic cosmic ray models, J. Geophys. Res., 117, A08109, doi: /2012ja Mrigakshi, A. I., D. Matthia, T. Berger, G. Reitz, and R. W. Wimmer-Schweingruber (2013), How galactic cosmic ray models affect the estimation of radiation exposure in space, Adv. Space Res., 51, Muller, D., S. P. Swordy, P. Meyer, M. L Heureux, and J. M. Grunsfeld (1991), Energy spectra and composition of cosmic rays, Astrophys. J., 374, National Council on Radiation Protection and Measurements (2006), Information needed to make radiation protection recommendations for space missions beyond low-earth orbit, NCRP Report 153, Bethesda, MD. Norman, R. B., and S. R. Blattnig (2010), A comprehensive validation methodology for sparse experimental data. NASA Technical Paper Norman, R. B., and S. R. Blattnig (2013), Validation of Nuclear Models used in Space Radiation Shielding Applications, J. Comput. Phys., 233, O Neill, P. M. (2006), Badhwar-O Neill galactic cosmic ray model updated based on Advanced Composition Explorer (ACE) energy spectra from 1997 to present, Adv. Space Res., 37, O Neill, P. M. (2010), Badhwar-O Neill galactic cosmic ray flux model Revised, IEEE Trans. Nucl. Sci., 57, Panov, A. D., et al. (2009), Energy spectra of abundant nuclei of primary cosmic rays from the data of ATIC-2 experiment: Final results, Bull. Russ. Acad. Sci. Phys., 73, Sentz, K., and S. Ferson (2011), Probabilistic bounding analysis in the quantification of margins and uncertainties, Reliability Eng. Syst. Safety, 96, Seo, E. S., J. F. Ormes, R. E. Streitmatter, S. J. Stocha, W. V. Jones, S. A. Stephens, and T. Bowen (1991), Measurement of cosmic-ray proton and helium spectra during the 1987 solar minimum, Astrophys. J., 378, Shikaze, Y., et al. (2008), Measurements of GeV/n cosmic-ray proton and helium spectra from with the BESS spectrometer, Astroparticle Phys., 28, Simon, M., J. F. Ormes, V. K. Balasubrahmanyan, and J. F. Arens (1980), Energy spectra of cosmic-ray nuclei to above 100 GeV per nucleon, Astrophys. J., 239, Slaba, T. C., and S. R. Blattnig (2014a), GCR environmental models I: Sensitivity analysis for GCR environments, Space Weather, doi: / 2013SW001025, in press. Slaba, T. C., and S. R. Blattnig (2014b), GCR environmental models II: Uncertainty propagation methods for GCR environments, Space Weather, doi: /2013sw001026, in press. Stone, E. C., A. M. Frandsen, R. A. Mewaldt, E. R. Christian, D. Margolies, J. F. Ormes, and F. Snow (1998), The advanced composition explorer, Space Sci. Rev., 86, Trucano, T. G., L. P. Swiler, T. Igusa, W. L. Oberkampf, and M. Pilch (2006), Calibration, validation, and sensitivity analysis: What s what, Reliability Eng. Syst. Safety, 91, Zeitlin, C., et al. (2013a), Measurements of galactic cosmic ray shielding with the CRaTER instrument, Space Weather, 11, , doi: /swe Zeitlin, C., et al. (2013b), Measurements of energetic particle radiation in transit to Mars on the Mars Science Laboratory, Science, 340, , doi: /science SLABA ET AL American Geophysical Union. All Rights Reserved. 245