Sensitivity of oscillation experiments to the neutrino mass ordering emb@kth.se September 10, 2014, NOW2014
1 Our usual approach
1 Our usual approach 2 How much of an approximation?
1 Our usual approach 2 How much of an approximation? 3 Summary and conclusions
1 Our usual approach 2 How much of an approximation? 3 Summary and conclusions
Parameter estimation sensitivity Define the test statistic χ 2 [ ] χ 2 L(θ d) (θ) = 2 log sup θ L(θ d) Assume it is χ 2 distributed with n degrees of freedom Use the data set without statistical fluctuations (Asimov data) Quote result
The interpretation Probability to observe a non-zero θ 13 at 99.73% CL in T2K 2π χ 2 is asymptotically χ 2 (Wilks theorem) The Asimov data is representative Several requirements, not always fulfilled 3π/2 π π/2 0 3.10-3 10-2 7.10-2 true sin 2 2θ 13 Schwetz, Phys.Lett. B648 (2007) 54 true δ CP < 20% 20%-50% 50%-90% 90%-99% > 99%
Not a nested hypothesis Mass ordering is not nested Wilks theorem not applicable Some different choices of test statistic ) 2 PDF( χ 0.1 0.08 0.06 0.04 NH MC IH MC NH Norm. Approx. IH Norm. Approx. χ 2 = χ 2 NO min χ 2 T = χ 2 IO χ 2 NO T = χ 2 (θ) min χ 2 All have different distributions We concentrate on T T 0 = value for Asimov data 0.02 T N (T 0, 2 T 0 ) 0 40 20 0 20 40 2 χ Qian, et al., Phys.Rev. D86 (2012) 113011
1 Our usual approach 2 How much of an approximation? 3 Summary and conclusions
Back to basics Test hypothesis NO, alternative hypothesis IO Can we reject NO if IO is true? Critical region defined by T c Reject NO if T < T c Test significance (CL): 1 α = 1 P(T < T c NO) α: Probability to reject NO if NO is true Test power: 1 β = P(T < T c IO) β: Probability of failing to reject false NO if IO is true Both α and β are relevant
What is sensitivity? Other Sensitivity (median) What is the expected rejection of a false ordering? (Given a parameter set) What is the probability of ruling out a false hypothesis at xσ? At what CL is exactly one ordering ruled out?
Simple hypotheses 6 Distribution of test statistic independent of parameter values Straight forward Applicable to reactor experiments sensitivity (σ) 5 4 3 2 median (2 sided) median (1 sided) crossing (2 sided) crossing (1 sided) 1 0 0 10 20 30 T 0 MB, Coloma, Huber, Schwetz, JHEP 03(2014)028
Composite hypotheses 150 NOvA 100 NO rejected 50 at 95 CL 0 50 IO rejected at 95 CL 100 150 10 5 0 5 0 T 150 LBNE 34kt Distributions have parameter dependence To reject = Rejecting all parameter sets (θ 13, m31 2, δ, etc) Distribution of T is parameter dependent 100 50 0 50 100 150 IO rejected at 95 CL NO rejected at 95 CL 30 20 10 0 10 20 0 T MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 T c = min T c (θ) θ Extensive Monte Carlo simulations Approximation (must check validity) T (θ) = N (T 0 (θ), 2 T 0 (θ))
Accelerator experiments - results MC Β 0.5 NOvA MC Β 0.5 LBNE 10kt Sensitivity Σ 6 4 2 Gaussian 1 sided Gaussian 2 sided Α Β Sensitivity Σ 6 4 2 Gaussian 1 sided Gaussian 2 sided Α Β 0 150 100 50 0 50 100 150 0 150 100 50 0 50 100 150 MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 Green: 68 % of outcomes Yellow: 95 % of outcomes
Comparison of experiments - specific β 7 6 True NO LBNE 34kt 7 6 True IO LBNE 34kt Sensitivity for Β 0.5 Σ 5 4 3 2 1 NOvA PINGU INO JUNO LBNE 10kt Sensitivity for Β 0.5 Σ 5 4 3 2 1 NOvA PINGU INO JUNO LBNE 10kt 0 2015 2020 2025 2030 Date MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 Note: Bands have different meanings! 0 2015 2020 2025 2030 Date
Comparison of experiments - specific α 1.0 0.8 True NO LBNE 34kt 1.0 0.8 True IO LBNE 34kt p 1 Β for 3Σ 0.6 0.4 PINGU JUNO LBNE 10kt p 1 Β for 3Σ 0.6 0.4 PINGU JUNO LBNE 10kt 0.2 INO 0.2 INO NOvA NOvA 0.0 2015 2020 2025 2030 Date MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 Note: Bands have different meanings! 0.0 2015 2020 2025 2030 Date
How to interpret the median sensitivity It is representative for how well the experiment will do 50 % probability of not reaching it 50 % probability of doing better Not 50 % probability of being wrong Not the only relevant quantity, distribution matters (do Brazilian bands!) Personal preference: Quote the power 1 β for a target sensitivity
Testing both orderings Depending on the significance it may be possible to: Reject exactly one hypothesis Reject both hypotheses Not reject any hypothesis It is natural, the true hypotheses should be rejected with probability α rejection probability α 10 0 both rejected 10-1 10-2 10-3 10-4 NO rejected IO rejected both accepted -20-10 0 10 20 α critical value T c MB, Coloma, Huber, Schwetz, JHEP 03(2014)028
What about δ CP? λ(α) 10 χ 2 for 2 dof 9 8 δ CP = 0 7 δ CP = π χ 2 for 1 dof 6 5 4 3 10-4 10-3 10-2 sin 2 2θ 13 Schwetz, Phys.Lett. B648 (2007) 54 The test statistic may be very different from a χ 2 distribution Checked with full Feldman Cousins approach Several devious details, caveat venditor! See talk by Enrique Fernandez Martinez
What about δ CP? Minakata, Nunokawa, Parke, PLB537 (2002) 249 The test statistic may be very different from a χ 2 distribution Checked with full Feldman Cousins approach Several devious details, caveat venditor! See talk by Enrique Fernandez Martinez
1 Our usual approach 2 How much of an approximation? 3 Summary and conclusions
Summary and conclusions Wilks theorem is not applicable to the neutrino mass ordering, the test statistic is not χ 2 distributed Regardless, T 0 is still a good approximation of the (median) sensitivity Frequentist (as well as Bayesian not discussed) methods are perfectly applicable Critical values will depend on the experiment and underlying assumptions May be relevant for δ, please stay for Enrique s talk
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods 4 Backup frequentist 5 Backup distributions and results 6 Backup Bayesian basics 7 Bayesian methods
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods At the end of the day For the simple hypotheses: Two Gaussians, H ± : N (±T 0, 2 T 0 ) For H +, typical (median) result is +T 0 +T 0 is T 0 ( T 0 ) = 2T 0 away from the expected H result 2T 0 2T 0 /(2 T 0 ) = T 0 See also: Vitelis, Read, 1311.4076
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Interpolating parameters Introduce an interpolating continuous parameter α such that P(α = ±1, θ) = P(θ, NO/IO) see, e.g., Capozzi, Lisi, Marrone, arxiv:1309.1638 Wilks theorem now applies Put bounds on α If α = 1 ( 1) is allowed, NO (IO) is allowed 2.0 2 1.5 1.01 0.5 0.00 α 0.5 1.0 1.5 h Entries 1764 Mean x 2.43 3 Mean y 10 0.9697 RMS x 0.006387 NH TRUE RMS y 0.5427 2.0 0 2.40 2.41 2.42 2.43 2.44 2.45 2.46 2 3 2 mee /10 [ev ] Capozzi, Lisi, Marrone, arxiv:1309.1638 Personal comment: α = 0 is not special, it being allowed a priori does not affect the rejection of NO/IO 1200 1000 800 600 400 200
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods CP violation sensitivity Distribution under CP conserving hypothesis: High precision: Slightly shifted to higher χ 2 Low precision: Shifted to lower χ 2 Median values: High precision: Do not change much Low precision: Shifted to lower χ 2 1 CDF 0.1 0.01 1 Σ 2 Σ 3 Σ NOΝA LBNE Χ 2 T2HK 0.001 0 2 4 6 8 0 MB,Coloma,Fernandez-Martinez,1407.3274 ESS
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods CP violation sensitivity Distribution under CP conserving hypothesis: High precision: Slightly shifted to higher χ 2 Low precision: Shifted to lower χ 2 Median values: High precision: Do not change much Low precision: Shifted to lower χ 2 Σ 3 2 1 T2HK Sc Median ESS LBNE NOΝA S Asimov 0 0 45 90 135 180 225 270 315 0 MB,Coloma,Fernandez-Martinez,1407.3274
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Octant sensitivity NuFIT 1.3 (2014) Degeneracies closer Wilk s theorem still violated A priori, a dedicated study is needed χ 2 χ 2 20 15 10 5 0 20 15 10 5 10 8 6 4 2 0 10 8 6 4 2 Global w/o ATM Global with SK1 4 (old) Global with SK1 4 (new) Normal Ordering Inverted Ordering 0 0 0.3 0.4 0.5 0.6 0.7 0 90 180 270 360 www.nu-fit.org sin 2 θ 23 δ CP
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods LBNO predictions p = 1-β 1 0.8 3σ 0.6 0.4 0.2 5σ Test power for NH 0 0 1 2 3 4 5 Exposure (/1e20 POT) LBNO collaboration, arxiv:1312.6520
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods 4 Backup frequentist 5 Backup distributions and results 6 Backup Bayesian basics 7 Bayesian methods
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Reactor experiments Distribution of T essentially parameter independent Simple hypotheses Gaussian limit well satisfied Median sensitivity nσ: n [ ( )] 2 erfc 1 1 2 erfc T0 2 JUNO, 4320 kt GW yr, 3% E-resol. true IO 0.01 T c,no true NO 0.01 T c,io -30-20 -10 0 10 20 30 T MB, Coloma, Huber, Schwetz, JHEP 03(2014)028
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Atmospheric experiments PINGU true IO true NO -30-20 -10 0 10 20 30 T MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 Distribution of T mainly depends on θ 23 Gaussianity is still well satisfied for each θ 23 Analytic as function of T 0 (θ 23 ) Boils down to evaluating the Asimov data: T 0
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Accelerator experiments - distributions NOνA Not always Gaussian! Typical for low-sensitivity experiments Need to perform Monte Carlo studies for accuracy Rejection power depends on the true parameters LBNE 90 90 true IO true IO true NO true NO 20 10 0 10 20 T MB, Coloma, Huber, Schwetz, JHEP 03(2014)028 50 0 50 T
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods 4 Backup frequentist 5 Backup distributions and results 6 Backup Bayesian basics 7 Bayesian methods
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Bayesian basics Assign a degree of belief in each hypothesis P(H) Update the degree of belief depending on observations Bayes theorem P(A, B) = P(A; B)P(B) = P(B; A)P(A) P(A; B) = Take A = hypothesis H, B = data d P(B; A)P(A) P(B) P(H; d) = L H(d)P(H) P(d)
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods 4 Backup frequentist 5 Backup distributions and results 6 Backup Bayesian basics 7 Bayesian methods
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Bayesian hypothesis testing Study the relative degrees of belief in two hypotheses P(H 1 ; d) P(H 2 ; d) = L H 1 (d) P(H 1 ) L H2 (d) P(H 2 ) Strength of the evidence for H 1 : [ ] P(H1 ; d) κ = 2 log P(H 2 ; d) Kass-Raftery scale: Strength of evidence for H κ Posterior odds Degree of belief Barely worth mentioning 0 to 2 ca 1 to 3 < 73.11% Positive 2 to 6 ca 3 to 20 > 73.11% Strong 6 to 10 ca 20 to 150 > 95.26% Very strong > 10 150 > 99.33%
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods What can be said about the future? Can compute probability of obtaining evidence at least strength κ 0 for the true ordering P(κ 0 ) = P(κ > κ 0 ; H 1 )P(H 1 ) + P(κ < κ 0 ; H 2 )P(H 2 ) Typical choice P(H 1 ) = P(H 2 ) = 0.5 Takes into account information on oscillation parameters L H (d) = L H(θ) (d)π(θ)dθ Compactifies all of the available information to one number Easy to simulate through Monte Carlo methods Prior dependent
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Example: NOνA 1 0.8 CDF(T) 0.6 0.4 0.2 0 20 15 10 5 0 5 10 15 20 T PDF(T) NOνA experiment GLoBES implementation Only δ and m 2 31 varying Flat and 10 % Gaussian priors, respectively 20 15 10 5 0 5 10 15 20 T MB, JHEP 01(2014)139
Backup frequentist Backup distributions and results Backup Bayesian basics Bayesian methods Example: NOνA, results 1 CDF(K) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 False Positive p [%] 25 50 75 90 95 99 99.7 Positive Strong Very strong 0 4 2 0 2 4 6 8 10 12 K MB, JHEP 01(2014)139 Probability of obtaining evidence with at least strength K for the correct ordering