STATS 00: Itroductio to Statistical Iferece Autum 06 Lecture 6 Simple alteratives ad the Neyma-Pearso lemma Last lecture, we discussed a umber of ways to costruct test statistics for testig a simple ull hypothesis, ad we showed how to use the ull distributio of the statistic to determie the rejectio regio so as to achieve a desired sigificace level The goal of today s lecture is to aswer the followig questio: Which test statistic should we use? The aswer depeds o the alterative hypothesis that we wish to distiguish from the ull 6 The Neyma-Pearso lemma Let s focus o the problem of testig a simple ull hypothesis H 0 agaist a simple alterative hypothesis H We deote by β = P H [accept H 0 ] the probability of type II error acceptig the ull H 0 whe i fact the alterative H is true (Here, P H [E] deotes probability of a evet E if H is true) Equivaletly, β = P H [reject H 0 ] is the probability of correctly rejectig H 0 whe H is true, which is called the power of the test agaist H Whe desigig a hypothesis test for testig H 0 versus H, we have the followig goal: maximize: the power of the test agaist H subject to: the sigificace level of the test uder H 0 is at most α This is a example of a costraied optimizatio problem, which we ca reaso about i the followig way: Suppose we observe data which are realizatios of radom variables X,, X For otatioal coveiece, let us deote by X = (X,, X ) the etire data vector, ad by x = (x,, x ) a vector of possible values for X I the discrete case, suppose the hypotheses are H 0 : X is distributed with joit PMF f 0 (x) := f 0 (x,, x ), H : X is distributed with joit PMF f (x) := f (x,, x ) Let X deote the set of all possible values of X uder f 0 ad f To defie the hypothesis test, for each x X, we must specify whether to accept or reject H 0 if the observed data is x I other words, we specify a rejectio regio R X such that we reject H 0 if the observed data belogs to R ad we accept H 0 otherwise The the probability of rejectig Cautio: Some books/papers use opposite otatio ad let β deote the power ad β deote the probability of type II error Make sure to double-check the meaig of the otatio 6-
H 0 if H 0 were true would be f 0(x), the probability of rejectig H 0 if H were true would be f (x), ad the above optimizatio problem is formalized as choosig the rejectio regio R X with the goal maximize f (x) subject to f 0 (x) α The cotiuous case is similar: suppose the hypotheses are H 0 : X is distributed with joit PDF f 0 (x) := f 0 (x,, x ), H : X is distributed with joit PDF f (x) := f (x,, x ) We defie a hypothesis test by defiig the regio R R such that we reject H 0 if ad oly if the observed data x belogs to R The above optimizatio problem is to choose R R with the goal maximize f (x) dx dx R subject to f 0 (x) dx dx α R I either the discrete or cotiuous case, what are the best poits x to iclude i this rejectio regio R? A momet s thought should covice you that R should cosist of those poits x correspodig to the smallest values of f 0(x) f, as these give the smallest icrease i (x) type I error per uit icrease of power Aother iterpretatio is that these are the poits providig the strogest evidece i favor of H over H 0 The statistic L(X) = f 0(X) f (X) is called the likelihood ratio statistic, ad the test that rejects for small values of L(X) is called the likelihood ratio test The Neyma-Pearso lemma shows that the likelihood ratio test is the most powerful test of H 0 agaist H : Theorem 6 (Neyma-Pearso lemma) Let H 0 ad H be simple hypotheses (i which the data distributios are either both discrete or both cotiuous) For a costat c > 0, suppose that the likelihood ratio test which rejects H 0 whe L(x) < c has sigificace level α The for ay other test of H 0 with sigificace level at most α, its power agaist H is at most the power of this likelihood ratio test Proof Cosider the discrete case, ad let R = {x : L(x) < c} be the rejectio regio of the likelihood ratio test Note that amog all subsets of X, R maximizes the quatity (cf (x) f 0 (x)), 6-
because cf (x) f 0 (x) > 0 for x R ad cf (x) f 0 (x) 0 for x / R Hece for ay other test with sigificace level at most α, say with rejectio regio R, (cf (x) f 0 (x)) (cf (x) f 0 (x)) Rearragig the above, this implies ( c f (x) ) f (x) f 0 (x) f 0 (x) = α f 0 (x) 0, where the last iequality follows because f 0 (x) is the sigificace level of the test that rejects for x R The f (x) f (x), ie the likelihood ratio test has power at least that of this other test The proof i the cotiuous case is exactly the same, with all sums above replaced by itegrals over R ad R 6 Examples Let s work out what the likelihood ratio test actually is for two simple examples Example 6 Cosider data X,, X ad the followig ull ad alterative hypotheses: H 0 : X,, X IID N (0, ) N (µ, ) Here we assume µ is a kow, specified value (ot equal to 0), so that H alterative hypothesis The joit PDF of (X,, X ) uder H 0 is f 0 (x,, x ) = e x π i= i = ( ) ( ) exp x + + x π is a simple The joit PDF uder H is f (x,, x ) = i= ( ) e (x i µ) = exp π π ( (x µ) + + (x µ) Thus, the likelihood ratio statistic is L(X,, X ) = f ( 0(X,, X ) f (X,, X ) = exp X + + X + (X ) µ) + + (X µ) Expadig the squares ad simplifyig, we obtai ( ) µ(x + + X ) + µ L(X,, X ) = exp ) 6-3
Suppose first that µ > 0 The L(X,, X ) is a strictly decreasig fuctio of the sample mea X = (X + + X ) Hece, rejectig for small values of L(X,, X ) is the same as rejectig for large values of X So the Neyma-Pearso lemma tells us that the most powerful test should reject whe X > c, for some threshold c We pick c to esure that the sigificace level is α uder H 0 : Sice the ull distributio of X is X N (0, ), c should be the z(α) where z(α) is the upper α poit of the stadard ormal distributio Now suppose that µ < 0 The L(X,, X ) is strictly icreasig i X, so rejectig for small L(X,, X ) is the same as rejectig for small X By the same argumet as above, to esure sigificace level α, the most powerful test rejects whe X < z(α) Remark 63 The most powerful test agaist the alterative H : X,, X N (µ, ) is the same for ay µ > 0 (rejectig whe X > z(α)), ad either the test statistic or the rejectio regio deped o the specific value of µ This meas that, i fact, this test is uiformly most powerful agaist the (oe-sided) composite alterative N (µ, ) for some µ > 0 O the other had, the most powerful test is differet for µ > 0 versus for µ < 0: oe test rejects for large positive values of X, ad the other rejects for large egative values of X This implies that there does ot exist a sigle most powerful test for the (two-sided) composite alterative N (µ, ) for some µ 0 Example 64 Let X,, X {0, } be the results of flips of a coi, ad cosider the followig ull ad alterative hypotheses: H 0 : X,, X IID Beroulli ( ) Beroulli(p) Here we assume that p is a kow ad specified value, so H is simple The joit PMF of (X,, X ) uder H 0 ad H are, respectively, f 0 (x,, x ) = f (x,, x ) = =, ( ) x ++x p x i ( p) x i = p x ++x ( p) x x p = ( p) p i= i= Thus, the likelihood ratio statistic is L(X,, X ) = f 0(X,, X ) f (X,, X ) = ( p) ( ) X ++X p First suppose p > The L(X,, X ) is a decreasig fuctio of S = X + + X, so rejectig for small values of L(X,, X ) is the same as rejectig for large values of S 6-4 p
Hece, by the Neyma-Pearso lemma, the most powerful test rejects whe S > c for a costat c We choose c to esure sigificace level α: Uder H 0, S Biomial(, ), so c should be the α quatile of the Biomial(, ) distributio This test is the same for all p >, so it is i fact uiformly most powerful agaist the composite alterative Beroulli(p) for some p > For p <, L(X,, X ) is icreasig i S, so the most powerful test rejects for S < c ad some costat c To esure sigificace level α, c should be the α quatile of the Biomial(, ) distributio This test is the same for all p <, so it is uiformly most powerful agaist the compositive alterative Beroulli(p) for some p < Remark 65 We have glossed over a detail, which is that whe the distributio of the likelihood ratio statistic L(X) is discrete uder H 0, it might ot be possible to choose c so that the sigificace level is exactly α For istace, i the previous example, suppose we wish to achieve sigificace level α = 005, ad = 0 For S Biomial(0, ), we have P[S 5] = 00 ad P[S 4] = 0058 So if we reject H 0 whe S 4, we do ot achieve sigificace level α, ad if we reject H 0 whe S 5, the we are too coservative The theoretically correct solutio is to perform a radomized test: Always reject H 0 whe S 5, always accept H 0 whe S 3, ad reject H 0 with a certai probability whe S = 4, where this probability is chose to make the sigificace level exactly α A more complete statemet of the Neyma-Pearso lemma shows that this type of (possibly radomized) likelihood ratio test is most powerful amog all radomized tests I practice, it might ot be acceptable to use a radomized test (We foud the effects of this drug to be statistically sigificat because our statistical procedure told us to flip a coi, ad our coi laded heads) So we might take the more coservative optio of just rejectig H 0 whe S 5 6-5