Parameter Estimation and Hypothesis Testing Parameter estimation Maximum likelihood Least squares Hypothesis tests Goodness-of-fit Elton S. Smith Jefferson Lab Con ayuda de Eduardo Medinaceli y Cristian Peña 1
Dictionary / Diccionario systematic sistematico spin espin background trasfondo scintillator contador de centelleo random aleatorio (al azar) scaler escalar histogram histograma degrees of freedom grados de libertad signal sen~al power potencia maximum likelihood maxima verosimilitud test statistic prueba estadistica least squares minimos cuadrados goodness bondad/fidelidad chi-square ji-cuadrado estimation estima confidence intervals and limits intervalos de confianza y limites significance significancia frequentist frecuentista? conditional p p condicional nuissance parameters parametros molestosos? (A given B) (A dado B) outline indice fit ajuste (encaje) asymmetry asimetria parent distribution distr del patron variance varianza signal-to-background fraction sen~al/trasfondo root-mean-square raiz quadrada media (rms) sample muestra biased sesgo flip-flopping indeciso counting experiments multiple scattering correlations weight experimentos de conteo dispersion multiple correlaciones pesadas
Parameter estimation 3
Properties of estimators 4
An estimator for the mean 5
An estimator for the variance 6
The Likelihood function via example We have a data set given by N data pairs (x i, y i ±σ i ) graphically represented below. The goal is to determine the fixed, but unknown, µ = f(x). σ is known or estimated from the data. 7
Gaussian probabilities (least-squares) We assume that at a fixed value of x i, we have made a measurement y i and that the measurement was drawn from a Gaussian probability distribution with mean y(x i ) = a + bx i and variance σ i. f (y i ;a,b) = e (y i y(x i )) 1 σ i L = N 1 i=1 πσ i πσ i e (y i a bx i ) σ i y(x i ) = a + bx i χ = ln L + k = N i=1 (y i a bx i ) σ i 8
χ minimization χ a = N (y i a bx i ) = 0 i=1 σ i χ b = N x i (y i a bx i ) = 0 i=1 σ i a N i=1 1 σ i + b N i=1 x i σ i = N i=1 y i σ i a N i=1 x i σ i + b N i=1 x i σ i = N i=1 x i y i σ i 9
Solution for linear fit For simplicity, assume constant σ = σ i. Then solve two simultaneous equations for unknowns: a = y i x i x i x i y i ( ) N x i x i b = N x y x i i i y i N x i x i ( ) Parameter uncertainties can be estimated from the curvature of the χ function. V[a] χ a V[b] χ b 1 θ = ˆ θ 1 = = θ = ˆ θ σ x i ( ) N x i x i Nσ ( ) N x i x i 10
Parameter uncertainties In a graphical method the uncertainty in the parameter estimator θ 0 is obtained by changing χ by one unit. χ (θ 0 ±σ θ ) = χ (θ 0 ) + 1 In general, using maximum likelihood lnl(θ 0 ±σ θ ) = lnl(θ 0 ) 1/ 11
The Likelihood function via example What does the fit look like? ROOT fit to pol1 Additional information about the fit: χ and probability Were the assumptions for the fit valid? We will return to this question after a discussion of hypothesis testing 1
More about the likelihood method Recall likelihood for least squares: L = N 1 i=1 πσ i e (y i a bx i ) σ i But the probability density depends on application L = N i=1 f (y i ; parameters) Proceed as before maximizing lnl (χ has minus sign). The values of the estimated parameters might not be very different, but the uncertainties can be greatly affected. 13
Applications L = N 1 i=1 πσ i e (y i a bx i ) σ i a) σ i = constant b) σ i = y i c) σ i = Y(x) Poisson distribution (see PDG Eq. 3.1) L = N i=1 ( a bx i) n i e (a bx i ) n i! Stirling s approx ln(n!) ~ n ln(n) - n ln L = N i 1 [( a + bx ) i n i + n i ln[n i /(a + bx i )]] 14
Statistical tests In addition to estimating parameters, one often wants to assess the validity of statements concerning the underlying distribution Hypothesis tests provide a rule for accepting or rejecting hypotheses depending on the outcome of an experiment. [Comparison of H 0 vs H 1 ] In goodness-of-fit tests one gives the probability to obtain a level of incompatibility with a certain hypothesis that is greater than or equal to the level observed in the actual data. [How good is my assumed H 0?] Formulation of the relevant question is critical to deciding how to answer it. 15
Selecting events background signal 16
Other ways to select events 17
Test statistics 18
Significance level and power of a test 19
Efficiency of event selection 0
Linear test statistic 1
Fisher discriminant
Testing significance/goodness of fit Quantify the level of agreement between the data and a hypothesis without explicit reference to alternative hypotheses. This is done by defining a goodness-of-fit statistic, and the goodness-of-fit is quantified using the p-value. For the case when the χ is the goodness-of-fit statistic, then the p-value is given by p = f (χ ;n d )dχ χ obs The p-value is a function of the observed value χ obs and is therefore itself a random variable. If the hypothesis used to compute the p-value is true, then p will be uniformly distributed between zero and one. 3
χ distribution Gaussian-like 4
p-value for χ distribution 5
Using the goodness-of-fit Data generated using Y(x) = 6.0 + 0. x, σ = 0.5 Compare three different polynomial fits to the same data. y(x) = p 0 + p 1 x y(x) = p 0 y(x) = p 0 + p 1 x + p x 6
χ /DOF vs degrees of freedom 7
Summary of second lecture Parameter estimation Illustrated the method of maximum likelihood using the least squares assumption Reviewed how hypotheses are tested Use of goodness-of-fit statistics to determine the validity of underlying assumptions used to determine parent parameters 8
Constructing a test statistic 9