Lecture 20: Hypothesis testing

Transcription

1 Lecture : Hpothess testng Much of statstcs nvolves hpothess testng compare a new nterestng hpothess, H (the Alternatve hpothess to the borng, old, well-known case, H (the Null Hpothess or, decde whether to reject a sngle hpothess, H, even wthout a competng hpothess 449

2 Hpothess testng A hpothess ma be ether Smple: completel defned wthout an free parameters Composte: havng free parameters to be determned from the data 45

3 Hpothess testng Let W be the space of all possble phscal observatons (expermental outcomes We want to dvde ths nto The crtcal regon : w, the set of all observatons for whch we reject H The acceptance regon : W w, the set of observatons for whch we accept H To do ths, we defne a functon x(w called the test statstc. The art of hpothess testng amounts to makng a good choce of x(w 45

4 Hpothess testng To defne a good test statstc we need to understand two quanttes: The sgnfcance α of the test s the probablt that a true null hpothess wll (erroneousl fal the test (a Tpe I error: α = P(x w H The power of the test, β, s the probablt that the null hpothess wll fal the test f the alternatve hpothess H s true β = P(x w H 45

5 Hpothess testng Alternatvel, β = power = P(x w H = P(x W w H Ths s the probablt that the Null Hpothess wll be (erroneousl accepted even f the Alternatve Hpothess s correct (Tpe II error We want to choose a test statstc (and defne a crtcal regon whch mnmzes α and β 453

6 Example We detect a photon and measure ts energ E: was t emtted b atom A (whch emts at energ E A call ths the Null Hpothess, H or atom B (whch emts at energ E B alternatve hpothess H E C Probablt Crtcal regon Acceptance regon P(E H P(E H o α β E B E A Measured energ, E 454

7 Example An deal test mmmzes BOTH α and β Tpcall, we can mprove one at the cost of worsenng the other b varng a parameter (E C β Increase E C Good test Better Even better α 455

8 Tpe I and Tpe II errors The tradeoff we choose between Tpe I and Tpe II errors depends upon whch we consder more serous, e.g. n a crmnal tral, we have: Null Hpothess, H : the defendant s nnocent Alternatve hpothess, H : the defendant s gult Tpe I error: nnocent defendant wrongl convcted probablt = α Tpe II error: gult defendant wrongl acqutted probablt = β Tpe I error s consdered worse: set a ver small α (gult must be proved beond a reasonable doubt even though β s thereb reduced (the gult are more lkel to be acqutted 456

9 Neman Pearson test Regardless of how we wegh the serousness of Tpe and Tpe II errors, we alwas want to mnmze β for gven α, (or mnmze α for gven β. In the example consdered prevousl, there was onl one reasonable wa of defnng the crtcal regon (b means of a cut at E C 457

10 Neman Pearson test But suppose we have a more complcated stuaton P(t H Probablt P(t H Measured quantt, t 458

11 Neman Pearson test We should reject the hpothess when P(t H s large and P(t H s small P(t H Probablt P(t H Crtcal regons, w Measured quantt, t 459

12 Neman Pearson test The optmal test alwas rejects the hpothess when the lkelhood rato P(t H /P(t H exceeds some constant Where we set the threshold lkelhood rato depends on how we set the tradeoff between mnmzng α and β (.e. whether we are more averse to makng Tpe or Tpe II errors 46

13 Goodness of ft Another tpe of test smpl consders whether a sngle Null Hpothess H should be rejected We can stll fnd the probablt of rejectng H when t s true But we cannot evaluate a probablt of acceptng H when t s false Most famous test of ths knd s called the Ch-Squared test 46

14 Ch-squared Consder a set of N ndependent measurements wth =, N at N values of a dependent varable x Suppose the measurements are normall dstrbuted wth standard devatons σ. Jont N-dmensonal probablt dstrbuton for obtanng results,,. N s P(,,.. N d d.. d N = N = exp( [ ( x πσ ] / σ d 46

15 463 Ch-squared Defne Then ( ( N x σ = = = = = N N N N N P d x d d d P exp(,.., ( / ] ( [ exp(..,.., ( πσ πσ σ

16 Ch-squared Our Ch-squared test wll reject H f exceeds some threshold value If H s true, the probablt that exceeds s α = P( > H ( d The element of hpervolume d N s proportonal to N d = N d N / d exp( d 464

17 Ch-squared Hence the sgnfcance of the test becomes α = P( > H = k ( N exp( d The normalzaton constant k s obtaned from = P( > H = k t N exp( t dt = k. N Γ ( N 465

18 Ch-squared The gamma functon s defned (Recpes, Ch 6. b Γ( a = t e a t dt For ntegral a, Γ(a = (a!, and Γ( = π The ncomplete Gamma functon s gven b γ ( a, x x = t a e t dt 466

19 Ch-squared Hence, the probablt of exceedng f H s true can be wrtten α = P( > Q( N, H = N γ (N, Γ(N 467

20 468 Ch-squared Check for N =, for whch Γ(N = Γ( = π = = = = = > = exp( exp( ( exp( ( ( exp( ( (, (, ( π π π π γ π α d d d dt t t Q H P

21 for a composte hpothess Suppose now that H s composte, contanng m free parameters, a j wth j =,m Recpe: determne set of a j that mnmzes and call the resultant mnmum mn The probablt that ths mnmum exceeds s Q(ν,, where ν = N m s the number of degrees of freedom 469

22 Event Countng Man experments (especall n partcle phscs count events: Examples: number of events n an accelerator satsfng a partcular crteron number of X-ra photons detected n a partcular photon energ bn 47

23 Event Countng The observed number of photons, n, n energ bn centered at energ x, has a Posson dstrbuton In the lmt of large n, ( wll usuall suffce, ths can be approxmated adequatel b a Gaussan dstrbuton wth varance n Suppose the Null Hpothess predcts an expectaton value g (x for bn g (x = f(x H dx bn 47

24 Event Countng Our test statstc s then N ( n g ( x n = Even wthout a predcton for g (x, we can test a null hpothess that two samples, n and m, are drawn from the same dstrbuton (e.g. test the hpothess that the X-ra spectrum s the same now as t was ear ago If the total number of events are the same, N ( n m = n + m 47

25 Event Countng Summar: Ch-squared works well provded the number of events s suffcent to justf the approxmaton to a Gaussan It conssts of two parts: ( a test statstc and ( a probablt functon It s dstrbuton-free (doesn t depend on f(x It needs lots of data (want man bns, each wth events It depends on bnnng Queston: Are there bn-independent tests that work on smaller samples? (Answer: In -D, es 473