Theoretical Statistics. Lecture 25.

Transcription

1 Theoretical Statistics. Lecture 25. Peter Bartlett 1. Relative efficiency of tests [vdv14]: Rescaling rates. 2. Likelihood ratio tests [vdv15]. 1

2 Recall: Relative efficiency of tests Theorem: Suppose that (1) T n, µ, and σ are such that, for all h and θ n = θ 0 +h/ n, n(tn µ(θ n )) σ(θ n ) θ n N(0,1), (2) µ is differentiable at 0, (3) σ is continuous at 0. Then a test that rejects H 0 : θ = θ 0 for large values oft n and is asymptotically of level α satisfies, for all h, π n (θ n ) 1 Φ ( ) z α h µ (θ 0 ). σ(θ 0 ) So the slope µ (θ 0 )/σ(θ 0 ) determines the asymptotic power. 2

3 Rescaling rates So far, we ve considered alternatives of the form θ n = θ 0 + h n. This corresponds to choosing a sequence θ n such that the difference, θ n θ 0, when appropriately rescaled, approaches a constant: n(θn θ 0 ) h. This rescaling rate is appropriate for regular cases. But other rates are possible. 3

4 Rescaling rates: L 1 -distance Definition: The L 1 -distance [not total variation] between two distributions P and Q with densities p = dp/dµ and q = dq/dµ is P Q = p q dµ. Lemma: For a sequence of modelsp n,θ with null hypothesish 0 : θ = θ 0 and alternatives H 1 : θ = θ n, the power function of any test satisfies π n (θ n ) π n (θ 0 ) 1 2 P n,θ n P n,θ0. Furthermore, there is a test for which equality holds. 4

5 Rescaling rates: L 1 -distance Consequences: 1. If P n,θn P n,θ0 2: Some sequence of tests is perfect, that is, π n (θ n ) 1 and π n (θ 0 ) If P n,θn P n,θ0 0: Any sequence of tests is worthless, because π n (θ n ) π n (θ 0 ) If P n,θn P n,θ0 is bounded away from 0 and 2: There is no perfect sequence of tests, but not all tests are worthless. This result reveals the appropriate rescaling rate: we need θ n to approach θ 0 at a rate than ensures an intermediate value of P n,θn P n,θ0. 5

6 Rescaling rates: L 1 -distance Proof: First, for any densities p andq, 0 = (p q)dµ = (p q)dµ+ = p>q p>q p q dµ p<q p<q so [notice relationship with total variation distance] p q dµ = p q dµ+ p>q = 2 p>q p q dµ. (p q)dµ p q dµ, p<q p q dµ 6

7 Rescaling rates: L 1 -distance So we have π n (θ n ) π n (θ 0 ) = = 1[T n K n ](p n,θn p n,θ0 )dµ n 1[p n,θn > p n,θ0 ](p n,θn p n,θ0 )dµ n 1[p n,θn > p n,θ0 ] p n,θn p n,θ0 dµ n = 1 2 P n,θ n P n,θ0, where the upper bound is achieved by the test 1[T n K n ] = 1[p n,θn > p n,θ0 ]. 7

8 Rescaling rates: Hellinger distance It s convenient to relate thel 1 -distance to Hellinger distance (because then product measures are easy to deal with). Definition: The Hellinger distance between P and Q (which have densities p and q) is ( 1 ( h(p,q) = p 1/2 q 1/2) ) 2 1/2 dµ. 2 (The 1/2 ensures 0 h(p,q) 1. It is defined without it in vdv.) 8

9 Rescaling rates: Hellinger distance Theorem: nh 2 (P θn,p θ0 ) P n θ n P n θ 0 2, nh 2 (P θn,p θ0 ) 0 Pθ n n Pθ n 0 0, ( ) 1 h 2 (P θn,p θ0 ) = Θ Pθ n n n Pθ n 0 {0,2}. 9

10 Rescaling rates: Hellinger distance Proof: Useful properties: Also, 2h 2 (P,Q) P Q 2 2h(P,Q). A(P n,q n ) = A n (P,Q), Where A(P,Q) = 1 h 2 (p,q) = p 1/2 q 1/2 dµ is the Hellinger affinity. 10

11 Rescaling rates: Hellinger distance Proof (continued): nh 2 (P θn,p θ0 ) A(P θn,p θ0 ) = 1 ω A(Pθ n n,pθ n 0 ) 0 h 2 (Pθ n n,pθ n 0 ) 1 Pθ n n Pθ n 0 2. ( ) 1 n 11

12 Rescaling rates: Hellinger distance Proof (continued): nh 2 (P θn,p θ0 ) 0 A(P θn,p θ0 ) = 1 o A(P n θ n,p n θ 0 ) 1 h 2 (P n θ n,p n θ 0 ) 0 P n θ n P n θ 0 0. ( ) 1 n 12

13 Rescaling rates: Hellinger distance Thus, ifh 2 (P θ,p θ0 ) = Θ( θ θ 0 α ), then the critical quantity is the limit of (( ) α ) nh 2 (P θn,p θ0 ) = Θ n 1/α θ n θ 0. If P θ is QMD at θ 0, then h 2 (P θ,p θ0 ) = Θ( θ θ 0 2 ), that is,α = 2, so we consider a shrinking alternative with n(θn θ 0 ) h. 13

14 Rescaling rates: Hellinger distance Definition: The root density θ p θ (for θ R k ) is differentiable in quadratic mean at θ if there exists a vector-valued measurable function l θ : X R k such that, forh 0, ( pθ+h p θ 1 ) 2 2 ht lθ pθ dµ = o( h 2 ). Theorem: If P θ is QMD at θ andi θ = P θ lθ lt θ exists, then h 2 (P θ+h,p θ ) = 1 8 ht I θ h+o( h 2 ). 14

15 Rescaling rates: Hellinger distance Proof: 2h 2 (P θ+h,p θ ) = ( pθ+h p θ ) 2 dµ But QMD implies p θ+h p θ ht lθ pθ and 1 2 ht lθ pθ = p θ+h p θ 2 L 2 (µ). L 2 (µ) 2 L 2 (µ) = o( h 2 ), = 1 4 ht P θ ( lθ lt θ )h = 1 4 ht I θ h = O( h 2 ). 15

16 Rescaling rates: Hellinger distance So 2h 2 (P θ+h,p θ ) = p θ+h p θ 2 = 1 2 ht lθ pθ + L 2 (µ) ( pθ+h p θ 1 2 ht lθ pθ = 1 4 ht I θ h+o ( h 2) + ( o( h 2 )O( h 2 ) ) 1/2 = 1 4 ht I θ h+o ( h 2). ) 2 L 2 (µ) (Cauchy-Schwarz) 16

17 Rescaling rates: Hellinger distance Consider P θ uniform on[0,θ]. Recall that this model is not QMD. A straightforward calculation shows that h 2 (P θ,p θ0 ) = θ θ 0 θ θ 0. So the appropriate shrinking alternative hasn(θ n θ 0 ) h. 17

18 Likelihood ratio tests Suppose we observe X 1,...,X n, with density p θ, H 0 : θ Θ 0 versus H 1 : θ Θ 1. ForΘ 0 = {θ 0 } andθ 1 = {θ 1 }, the optimal test statistic is log n i=1 p θ1 (X i ) p θ0 (X i ). If we have composite hypotheses, we could instead use Λ n = log sup θ Θ 1 n i=1 p θ(x i ) sup θ Θ0 n i=1 p θ(x i ). 18

19 Likelihood ratio tests Notice that, for a minimal sufficient statistic T, we can write Λ n = log sup θ Θ 1 n i=1 h(x i)f θ (T(X i )) sup θ Θ0 n i=1 h(x i)f θ (T(X i )) = log sup θ Θ 1 n i=1 f θ(t(x i )) sup θ Θ0 n i=1 f θ(t(x i )), so Λ n depends only on the minimal sufficient statistic. Since the critical value will be positive, it will not change the test if we replace this statistic by Λ n 0. We will also scale it by a factor of 2. (We ll see that this gives a neater test.) 19

20 Likelihood ratio tests Define Λ n = 2( Λ n 0) ( n supθ Θ1 i=1 p θ(x i ) ) ( n sup θ Θ0 i=1 p θ(x i ) ) = 2log sup θ Θ0 n i=1 p θ(x i ) = 2log sup n θ Θ 0 Θ 1 i=1 p θ(x i ) n sup θ Θ0 i=1 p θ(x i ) n ( ) = 2 (X lˆθn i (X ) lˆθn,0 i ), i=1 where ˆθ n is the maximum likelihood estimator forθ over Θ = Θ 0 Θ 1, and ˆθ n,0 is the maximum likelihood estimator over Θ 0. 20

21 Likelihood ratio tests We ll focus on cases where Θ = Θ 0 Θ 1 is a subset ofr k, and where Θ and Θ 0 are locally linear spaces. Then under H 0, we ll see that Λ n is asymptotically chi-square distributed withmdegrees of freedom, where m = dim(θ) dim(θ 0 ). So we can get a test that is asymptotically of level α by comparing Λ n to the upper α-quantile of a chi-square distribution. 21