4. Hypothesis testing (Hotelling s T 2 -statistic)

Transcription

1 4. Hypothesis testig (Hotellig s T -statistic) Cosider the test of hypothesis H 0 : = 0 H A = 6= 0 4. The Uio-Itersectio Priciple W accept the hypothesis H 0 as valid if ad oly if H 0 (a) : a T = a T 0 is accepted for all liear compouds a: [i some sese the uio of all such hypotheses] For give a; we set y = a T x so that i the populatio uder H 0 ; ad i our sample E (y) = a T 0 V ar (y) = a T a y = a T x s s s y s:e: (y) = = a T S u a The uivariate t-statistic for testig H 0 (a) agaist the alterative (y) 6= a T 0 is t (a) = y at 0 s:e: (y) p a T (x = 0 ) p a T Sa where S is uderstood to deote the ubiased estimator S u for the rest of this subsectio. The acceptace threshold for H 0 (a) takes the form t (a) R for some R. The multivariate acceptace regio is the itersectio \ t (a) R (4.) which is true if ad oly if max a t (a) R: Therefore we adopt T = max t (a) a as the test statistic for H 0 : Write d = x 0 ; the t (a) = at dd T a a T Sa which is the ratio of two quadratic forms. This is a classical extremal problem kow as the geeralized eigevalue problem As this ratio is idepedet of the scale of a we ca set the deomiator arbitrarily to have the value.

2 This gives a equivalet formulatio Maximize a T (x 0 ) (x 0 ) T a subject to a T Sa = (4.) i.e. of the form: max f (a) subject to g (a) = 0. We itroduce a Lagragea multiplier ad seek to determie ad a to satisfy d h i a T (x da 0 ) (x 0 ) T a a T Sa = 0 dd T a Sa = 0 (4.3a) S dd T I a = 0 (4.3b) js dd T Ij = 0 (4.3c) (4.3b) ca be writte Ma = a showig that a is a eigevector of M = S dd T. (4.3c) is the determiatal equatio satis ed by the eigevalues of S dd T. Premultiplyig (4.3a) by a T gives a T dd T a a T Sa = 0 = at dd T a a T Sa = t (a) Therefore i order to maximize t (a) we choose to be the largest eigevalue of M = S dd T : This is a rak matrix with the sigle o-zero eigevalue tr S dd T = d T S d ad the maximum of (4.) is kow as Hotellig s T statistic T = (x 0 ) T S (x 0 ) (4.4) which is the sample Mahalaobis distace betwee x ad 0. Note that M = uv T u = S d ad v = d are rak matrices (vectors). Mu = u v T u = u where so u = S d is the correspodig eigevector of M: Thus a = u = S d de es the liear compoud y = a T x givig the maximum discrepacy from the ull hypothesis H 0 : 4. Distributio of T Uder H 0 it ca be show that T s p p F p; p (4.5)

3 where F p; p is the F distributio o p ad p degrees of freedom. The correspodig F statistic is therefore F = p : T p : If H 0 is true the F has a F distributio o p ad p degrees of freedom (d.f.), while if H 0 is ot true the the distributio has a o-cetral F distributio. NB. depedig o the covariace matrix used, T has slightly di eret forms ( T ( ) (x = 0 ) T Sm (x 0 ) (x 0 ) T Su (x 0 ) where S m is the ML estimate of (with divisor ) ad S u is the ubiased estimator of (with divisor ). Example I a ivestigatio of adult itelligece, scores were obtaied o two tests "verbal" ad "performace" for = 0 subjects aged 60 to 64. Doppelt ad Wallace (955) reported the followig mea score ad covariace matrix: x 55:4 = x 34:97 0:54 6:99 S u = 6:99 9:68 At the = :0 (%) level, test the hypothesis that 60 = 50 ad We rst compute S u = :039 :0400 :0400 :03 d = x 0 = 4:76 5:03 T The T statistic is the T = 4:76 5:03 :039 :0400 4:76 0 :0400 :03 5:03 4:76 = :039 4:76 5:03 : :03 :03 = 3:54 This gives F = p : T p = 99 3:54 = 76:9 3

4 The earest tabulated % value correspods to F ;60 ad is Therefore we coclude the ull hypothesis should be rejected. The sample probably arose from a populatio with a much lower mea vector, rather closer to the sample mea. Example The chage i levels of free fatty acid (FFA) were measured o 5 hypotised subjects who had bee asked to experiece fear, depressio ad ager e ects while uder hyposis. The mea FFA chages were x = :699 x = :78 x 3 = :558 Give that the covariace matrix of the stress di ereces y i = x i x i ad y i = x i x i3 is :7343 :666 S u = :666 :7733 Su 0:804 0:338 = 0:338 :7733 test at the 0.05 level of sigi cace, whether each e ect produced the same chage i FFA. [T = :68 ad F = :4 with degrees of freedom,3. Do ot reject the hypothesis "o emotio e ect" at the = :05 level] 4.3 Ivariace of T T is ua ected by chages i the scale or origi of the (respose) variables. Cosider where C is (p p) ad o-sigular. y = Cx + d The ull hypothesis H 0 : x = 0 is equivalet to H 0 : y = C 0 + d. trasformatio Uder this liear y = C x + d S y = CSC T so that T y = y y T S y y y = (x 0 ) T C T CSC T C (x 0 ) = (x 0 ) T C T C T S C C (x 0 ) = (x 0 ) T S (x 0 ) which demostrates ivariace. 4

5 4.4 Co dece iterval for a mea A co dece regio for ca be obtaied give the distributio of T by substitutig the data values x ad S : I Example above we have (x ) T S (x ) s p p F p; p (4.6) x = (55:4; 34:97) T 00S :3 :40 = :40 :3 ad F ;99 (:0) is approximately 4.83 (by iterpolatio). A 99% co dece regio for is therefore the set of ( ; ) satisfyig :3 ( 55:4) :80 ( 5:4) ( 34:97) + :3 ( 34:97) 00 4:83 = 9:76: 99 This represets the iterior of a ellipse i p = dimesios which ca be plotted. I higher dimesios a ellipsoidal co dece regio is obtaied. 4.5 Likelihood ratio testig Give a data matrix X of observatios o a radom vector x whose distributio depeds o a vector of parameters, the likelihood ratio for testig the ull hypothesis H 0 : 0 H : agaist the alterative is de ed as = sup 0 L sup L where L = L (; X) is the likelihood fuctio. I a likelihood ratio test (LRT) we reject H 0 for low values of ; i.e. if < c where c is chose so that the probability of Type I error is : If we de e l0 = log L 0 where L 0 is the value of the umerator ad similarly l = log L, the rejectio criterio takes the form L log = log 0 Result L (4.7) = l 0 l > k (4.8) Whe H 0 is true ad for large the log likelihood ratio (4.8) has the -distributio o r degrees of freedom, r, where r equals the umber of free parameters uder H mius the umber of free parameters uder H 0 : 5

6 4.6 LRT for a mea whe is kow H 0 : = 0 a give value whe is kow Give a radom sample from N (; ) resultig i x ad S the likelihood give i (3.8b) is (to withi a additive costat) l (; ) = log jj + tr S o + (x ) T (x ) (4.9) Uder H 0 the value of is kow ad l0 = l ( 0 ; ) = log jj + tr S o + (x 0 ) T (x 0 ) Uder H with o restrictio o ; the m.l.e. of is ^ = x: Thus l = log jj + tr S Therefore log = l 0 l = (x 0 ) T (x 0 ) (4.0) which is times the Mahalaobis distace of x from 0. Note the similarity with Hotellig s T statistic. Give the distributio of x uder H 0 is x s N p 0 ; ad (4.0) may be writte usig the trasformatio y = idepedet N (0; ) variates as (x 0 ) to a stadard set of px log = y T y = yi (4.) we have the exact distributio showig that i this case the asymptotic distributio of Example log s p (4.) log is exact for the small sample case. Measuremets of the legth of skull were made o a sample of rst ad secod sos from 5 families. 85:7 x = 83:84 9:48 66:88 S = 96:78 Assumig that i fact =

7 test at the = :05 level the hypothesis H 0 : = 8 8 T Solutio log = 5 3:7 :84 :0 0 3:7 0 :0 :84 = 0:5 3:7 + :84 = 4:3 Sice (:05) = 5:99 do ot reject H LRT for whe is ukow (Hotellig0s) Cosider the LRT test of the hypothesis H 0 : = 0 whe is ukow. H : 6= 0 I this case must be estimated uder H 0 ad also uder H : Uder H 0 : ( = 0 ) l ( 0 ; ) = log jj + tr S o + (x 0 ) T (x 0 ) = log jj + tr S + d T 0 d 0 = log jj + tr S + d 0 d T 0 (4.3a) (4.3b) writig d 0 for x 0 : Note that (as before) we used the followig: d T 0 d 0 = tr d T 0 d 0 = tr d 0 d T 0 Uder H : l (^; ) = log jj + tr S o + (x ^) T (x ^) = log jj + tr S (4.4a) Substitutig m.l.e. s ^ = x ad ^ = S obtaied previously, gives l ^; ^ = log jsj + tr S S = flog jsj + tr (I p )g l = log jsj + p (4.4b) 7

8 Comparig (4:3b) with (4:4b) it is clear that the m.l..e. of uder H 0 must be ^ = S + d 0 d T 0 ad that the correspodig value of l = log L is l 0 = log js + d 0 d T 0 j + p (4.4c) The log likelihood ratio is l 0 l = log js + d 0 d T 0 j log jsj = log js j + log js + d 0 d T 0 j = log js S + d 0 d T 0 j = log ji p +S d 0 d T 0 j = log + d T 0 S d 0 (4.5) makig use of the useful matrix result proved i (:8:3) that ji p +uv T j = + v T u : Sice log = log + T (4.6) we see that ad T are mootoically related. Therefore we ca coclude that the LRT of H 0 : = 0 whe is ukow is equivalet to use of Hotellig s T statistic. 4.8 LRT for = 0 with ukow H 0 : = 0 whe is ukow. H : 6= 0 Uder H 0 we substitute ^ = x ito o l (^; 0 ) = log j 0 j + tr 0 S + (x ^) T 0 (x ^) givig l 0 = log j 0 j + tr 0 S (4.7) Uder H we substitute the urestricted m.l.e. s ^ = x ad ^ = S givig as i (4:4e) l = log jsj + p (4.8) l0 l = log j 0 j + tr 0 S log jsj p = log j0 Sj+ tr 0 S p (4.9) This statistic depeds oly o the eigevalues of the positive de ite matrix 0 S ad has the property that l0 l = log! 0 as S approaches 0: 8

9 Let A be the arithmetic mea ad G the geometric mea of the eigevalues of 0 S the tr 0 = pa Sj = Gp j 0 log = fpa p log G pg The geeral result for the distributio of (4:0) for large gives = p fa log G g (4.0) l 0 l s r (4.) where r = p (p + ) is the umber of idepedet parameters i : 4.0 Test for sphericity A covariace matrix is said to have the property of "sphericity" if = ki p (4.) for some k: We see that this is a special case of the more geeral situatio = k 0 treated i Sectio (3.3.3). The same procedure ca be applied. The geeral likelihood expressio for a sample from the MVN distributio is: log L = log jj + tr S + dd T Uder H 0 : = ki p ad ^ = x so log L = log jki p j + tr k S = p log k + k tr S [ log L] = 0 at a miimum p k k tr S = 0 ^k = tr S p (4.4) which is i fact the arithmetic mea A of the eigevalues of S: Substitute back ito (4.3) gives l0 = p (log A + ) Uder H : ^ = x ad ^ = S l = log jsj + p = p (log G + ) 9

10 thus log = l 0 l = p log A G (4.5) The umber of free parameters cotaied i is uder H 0 ad p (p + ) uder H : Hece the appropriate distributio for comparig log is r where r = p (p + ) = (p ) (p + ) (4.6) 4. Test for idepedece Idepedece of the variables x ; :::; x p is maifest by a diagoal covariace matrix = diag ( ; :::; pp ) (4.7) We cosider H 0 : is diagoal H :. is urestricted agaist the geeral alterative Uder H 0 it is clear i fact that we will d ^ ii = s ii because the estimators of ii for each x i are idepedet. We ca also show this formally = log jj + tr S + dd T ( px = log ii + px s ii ii ii ( log L) = 0 ii s ii ii = 0 b ii = s ii Therefore ( px ) = log s ii + p = flog jdj + pg where D = diag (s ; :::; s pp ) : Uder H as before we d l = log jsj + p 0

11 Therefore l 0 l = [log jdj log jsj] = log jd Sj = log jd SD j = log jrj (4.8) The umber of free parameters cotaied i is p uder H 0 ad p (p + ) uder H : Hece the appropriate distributio for comparig log is r where r = p (p + ) p = p (p ) (4.9) 4. Simultaeous co dece itervals (Sche e, Roy & Bose) The uio-itersectio method for derivig Hotellig s T statistic provides "simultaeous co - dece itervals" for the parameters whe is ukow. Followig Sectio 4. let T = ( ) (x ) T S (x ) (4.30) where is the ukow (true) mea. Let t (a) be the uivariate t liear compoud y = a T x: The max a t (a) = T ad for all p where vectors a statistic correspodig to the t (a) T (4.3) t (a) = y y s y = p p a T (x ) = p a T Sa (4.3) so From Sectio 4. the distributio of T is T s p p F p; p Pr T ( ) p p F p; p () = therefore from (4.3), for all p vectors a Pr t (a) ( ) p p F p; p () = (4.33) Substitutig from (4.3), the co dece statemet i (4.33) is:

12 With probability for all p vectors a where K is the costat ( ) p ja T x a T j p F p; p () s a = K T Sa = s a T Sa say, (4.34) ( ) p = K = p F p; p () (4.35) A 00 ( ) % co dece iterval for the liear compoud a T is therefore s a T a x K T Sa (4.36) How ca we apply this result? We might be iterested i a de ed set of liear combiatios (liear compouds) of : The i th compoet of is for example the liear compoud de ed by a T = (0; :::; ; :::0) the uit vector with a sigle i the i th positio. For a large umber of such sets of CI s we would expect 00 ( ) % to cotai o mis-statemets while 00% would cotai at least oe mis-statemet. We ca relate the T co dece itervals to the T test of H 0 : = 0. If this H 0 is rejected at sigi cace level the there exists at least oe vector a such that the iterval (4.36) does ot iclude the value a T 0 : NB. If the covariace matrix S u (with deomiator ) is supplied, the i (4.36) r a T S u a may be replaced by : r a T Sa 4.3 The Boferroi method This provides aother way to costruct simultaeous CI s for a small umber of liear compouds of whilst cotrollig the overall level of co dece. Cosider a set of evets A ; A ; :::; A m Pr (A \ ::: \ A m ) = Pr A [ ::: [ A m From the additive law of probabilities X m Pr A [ ::: [ A m Pr A i Therefore Pr (A \ ::: \ A m ) mx Pr A i (4.37)

13 Let C k deote a co dece statemet about the value of some liear compoud a T k with Pr [C k true] = k : Pr (all C k true) ( + ::: + m ) (4.38) Therefore we ca cotrol the overall error rate give by + ::: + m = say. For example, i order to costruct simultaeous 00 ( ) % CI s for all p compoets k of we could choose k = p (k = ; :::; p) leadig to if s ii derives from S u : Example x t x p t p. p r s r spp Itelligece scores data o = 0 subjects: x x = = x S U = 55:4 34:97 0:54 6:99 6:99 9:68. Costruct 99% simultaeous co dece itervals for ; ad : For take a T = (; 0) a T x = 0 55:4 = 55:4 34:97 a T S u a = 0:54 Now take = :0 K = ( ( ) p p) F p; p () = 00 F ;99 (:0) 99 = 3: takig F ;99 (:0) = 4:83 (approx). Therefore the CI for is r 0:54 55:4 3: 0 = 55:4 4:50 givig a iterval = (50:7; 59:7) For we already have K, take a T = (0; ) the a T x = 34:97 a T S u a = 9:68 3

14 The CI for is r 9:68 34:97 3: = 34:97 3:40 0 givig a iterval = (3:6; 38:4) For take a T = [; ] a T 55:4 x = [; ] = 0:7 34:97 a T 0:54 6:99 S u a = [; ] 6:99 9:68 = 0:54 6:9 + 9:68 = 76:4 CI for is 0:7 3: r 76:4 0 = 0:7 :7 = (7:6; 3:0). Costruct CI s for ; by Boferroi method. Use = :0: Idividual CI s are costructed usig k = :0 = :005 (k = ; ) : The k t 00 = t 00 (:005) ' (:9975) CI for is = :8 55:4 :8 r 0:54 0 = 55:4 4:06 = (5:; 59:3) ad for is 34:97 :8 r 9:68 0 = 34:97 3:06 = (3:9; 38:0) Comparig CI s obtaied by the two methods we see that the simultaeous CI s for ad ad are 8.7% wider tha the coirrespodig Boferroi CI s. NB. If we had required 99% Boferroi CI s for ; ad the m = 3 i (4.38) ad m = :0 = :007: The correspodig percetage poit of t would be 6 t 00 (:007) ' (:9983) = :93 leadig to a slightly wider CI Tha obtaied above. 4

15 4.4 Two sample procedures Suppose we have two idepedet radom samples fx ; :::; x g fx ; :::; x g of size ; from two populatios. : x s N p ( ; ) : x s N p ( ; ) givig rise to sample meas x ; x ad sample covariace matrices S ; S. Note the assumptio of a commo covariace matrix : We cosider testig H 0 : = agaist H : 6= Let d = x x : Uder H 0 d s N 0; + (a) Case of kow Aalogously to the oe sample case d + s N (0; Ip ) dt d s p where = + (b) Case of ukow We have the Wishart distributed quatities S s W p (; ) S s W p (; ) Let S p = S + S be the pooled estimator of the covariace matrix : The from the additive properties of the Wishart distributio ( ) S p has the Wishart distributio W p (; ) ad d s N (0; ) It may be show that T = d T Sp d (4.39) has the distributio of a Hotellig s T statistic. I fact T s ( ) p p F p; p (4.40) 5

16 4.5 Multi-sample procedures (MANOVA) We cosider the case of k samples from populatios ; :::; k : The sample from populatio i is of size i : By aalogy with the uivariate case we ca decompose the SSP matrix ito orthogoal parts. This decompositio ca be represeted as a Multivariate Aalysis of Variace (MANOVA) table. The MANOVA model is x ij = + i + e ij j = ; :::; i ad i = ; :::; k (4.4) where e ij are idepedet N p (0; ) variables. Here the parameter vector is the overall (grad) mea ad the i is the i th treatmet e ect with kx i i = 0 De itios Let the i th sample mea be with = X k x i = i i: The Grad Mea is x = X i x ij j= X k ix i. The Betwee Groups sum of squares ad cross-products (SSP) matrix is ad the Total SSP matrix is B = T = kx i (x i x) (x i x) T (4.4) kx X i (x ij x) (x ij x) (4.43) j= It ca be show algebraically that T = B + W where W is the Withi Groups (or residual) SSP matrix give by kx X i W = (x ij x i ) (x ij x i ) T (4.44) The MANOVA table is j= Source Matrix of SS ad Degrees of of variatio cross-products (SSP) freedom (d.f.) Treatmet Residual B = X k i (x i x) (x i x) T k W = X k X i j= (x ij x i ) (x ij x i ) T X k i k Total (corrected for the mea) T = X k X i (x ij j= x) (x ij x) 6 X k i

17 We are iterested i testig the hypothesis H 0 : = = ::: = k (4.4) whether the samples i fact come from the same populatio agaist the geeral alterative H : 6= 6= ::: 6= k (4.43) 4.5. Wilk s statistic We ca derive a likelihood ratio test statistic kow as Wilk s : Uder H 0 the m.l.e. s are ^ = x ^ = S leadig to the maximized log likelihood (miimum of log L) where Uder H the m.l.e. s are This follows from l = mi ;d i = mi ( ( W = l 0 = p + log jsj (4.44) ^ i = x i ^ = W kx W i = log jj + kx i S i kx i tr log jj + tr S i + d i d T ) i!) kx i S i sice ^d i = x i ^ i = 0. Hece ^ = W ad l = p + log W (4.45) Therefore sice T = S jw j l0 l = log jt j = log (4.46) where is kow as Wilk s statistic. We reject H 0 for small values of or large values of log : Asymptotically, the rejectio regio is the upper tail of a p(k ). Uder H 0 the ukow has p parameters ad uder H the umber of parameters for ; :::; k is pk: Hece the d.f. of the is p (k ). Apart from this asymptotic result, other approximate distributios (otably Bartlett s approximatio) are available, but the details are outside the scope of this course. 7

18 4.5. Calculatio of Wilk s Result Let ; :::; p be the eigevalues of W B the Proof py = ( + j ) (4.47) j= = T W = (W + B) W = W (W + B) = I + W B py = ( + j ) (4.48) j= by the useful idetity proved earlier i the otes Case k = We show that use of Wilk s for k = groups is equivalet to usig Hotellig s T statistic. Speci cally, we show that is a mootoic fuctio of T. Thus to reject H 0 for < is equivalet to rejectig H 0 for T > (for some costats ; ): Proof For k = we ca show (Ex.) that where d = x x. The B = ddt (4.49) I + W B = I + W dd T = + dt W d Now W is just ( ) S p where S p is the pooled estimator of : Thus comparig with (4.39) where T (two sample case) is give we see = + T (4.50) 8