8 Statistical Analysis of Multivariate Data

Transcription

1 8 Statstcal Aalyss of Multvarate Data 8.0 Itroducto So far, the course has cosdered oly data-aalytc techques: methods of dmesoalty reducto data dsplay, vestgato of subgroup structure etc whch deped oly upo the structure of the data themselves ad ot upo assumptos o the form of the geeratg process of the data. Such methods may be a useful [prelmary??] step the aalyss they may smplfy later aalyses ad are ot proe to falures assumptos (sce oe are made!) ad they provde a valuable tutve uderstadg of the data. Ths secto cosders more formal statstcal models ad techques for the aalyss of multvarate data. The secto starts wth the defto of the p-dmesoal multvarate ormal dstrbuto ad ts basc propertes (mea, varace ad samplg propertes such as maxmum lkelhood estmato). Ths allows the costructo of lkelhood rato tests ad thus the exteso to several dmesos of the route oe-dmesoal tests such as t-tests ad aalyss of varace. N. B. If you wat to check the basc deas of maxmum lkelhood estmato ad lkelhood rato tests the you should read Appedx 0 (Backgroud Results). NRJF, Uversty of Sheffeld, 011/1 Semester 1 197

2 Also cosdered are tests of more complex hypotheses, whch ca oly arse multdmesos, such as whether the populato mea s somewhere o the ut sphere. Ths requres use of Lagrage multplers to maxmze lkelhoods subject to costrats. For ths secto you should read the materal o Geeralzed Lkelhood Rato Tests gve the Backgroud Results The secod mportat topc of the secto s the troducto of a ew method for costructg hypothess tests (the Uo-Itersecto prcple) whch some crcumstaces ca gve a dfferet form of test from a lkelhood rato test ad others provde a useful addtoal terpretato of lkelhood rato tests. Ths topc lks wth the dea of projectg data to oe dmeso, choosg the dmeso approprately, whch was ecoutered the costructo of prcpal compoets ad crmcoords. NRJF, Uversty of Sheffeld, 011/1 Semester 1 198

3 8.1 The Multvarate Normal Dstrbuto Defto The radom p-vector x has a p-dmesoal Multvarate Normal dstrbuto (wth mea ad varace, a colum p-vector, a pp symmetrc o-sgular postve defte matrx) f the probablty desty fucto (p.d.f.) of x s 1 f (x) exp{ (x ) (x )} (where =det() ) x ( ) 1 1 p 1 ad we wrte x~n p (,) Stadardzato Suppose x~n p (,) ad y= ½ (x-), where ½ s as defed earler (see 4.5, eqs & 4.5.), the (x-) 1 (x-) = yy = y. Now the desty of y s f (y) exp{ y' y}.j y ( ) 1 1 p 1 xy p 1 where J xy s the Jacobea of the trasformato gve by dx dy where dx dy s the pp matrx wth (,j) th elemet dx. dy j Now y= ½ (x-) so x= ½ y+ so dx dy ( 1 ) j j ad so dx 1 dy ad J xy = ½ gvg f(y) exp{ y} y 1 p 1 ( ) 1 p Thus, the y s are depedet uvarate N(0,1); [& otce therefore that f y (y) s > 0, tegrates to 1 ad so s a geue p.d.f. ad so therefore f x (x) s a p.d.f. also] NRJF, Uversty of Sheffeld, 011/1 Semester 1 199

4 8.1.3 Mea & Varace Now f x~n p (,) ad y= ½ (x-) the E[y]= ½ (E[x] ) ad var(y)= ½ var(x) ½ but E[y]=0 ad var(y)=i p, so E[x]= ad var(x)= Radom Samples x~n p (,); observatos x 1,x,...,x of x. Defe x 1 1 x 1 ad S ( )(X X)(X X) 1 ( ) xx xx The E[ x ]= ad var( x ) = 1 1 var(x ) 1 Also S= ( ) (x )(x ) (x )(x ) 1 ( ) j 1 j (see Notes 0.9) ad E[(x )(x j )] = 0 f j = f =j ad so E[S] =,.e. the sample mea ad varace are ubased for the populato mea ad varace. (ote that as yet we have ot used ay assumpto of ormalty) Further, usg the fact that f x~n p (,) the the characterstc fucto of t x x s x (t)=e[ e ]=exp{t-½tt} [where =( 1) ½ ] we ca shew that x ~N p (, 1 ). NRJF, Uversty of Sheffeld, 011/1 Semester 1 00

5 8. Maxmum Lkelhood Estmato x 1, x,..., x depedet observatos of x~n p (, ) 1 The Lk(, ; X) p exp 1 1 (x ) (x ) ( ) 1 so (, ; X)=log e (Lk(, ; X)) = = (x ) (x ) ½plog() ½log( ) { (x x) (x x) (x ) (x )} ½plog() ½log( ) So, 1 (x ) ad thus ˆ x. Further, f we set T= 1 t ca be shew that T { ( 1 )S (x )(x ) } 1 dag{ ( 1 )S (x )(x ) } (where the dervatve of the scalar wth respect to the pp matrx T s the pp matrx formed by the partal dervatves of wth respect to each of the elemets t j of T, ad where dag{a pp } s the dagoal matrx formed by just the dagoal elemets of the pp matrx A, zeroes elsewhere. Some detals of ths are gve the appedx.) NRJF, Uversty of Sheffeld, 011/1 Semester 1 01

6 So T =0 whe ˆ 1 S (x ˆ)(x ˆ) ad whe ˆ x ths gves the [urestrcted] maxmum lkelhood estmates of ad ˆ x, ˆ 1 S More geerally, whatever the mle of s, f d x ˆ the we have ˆ Sdd 1 [Ths form s sometmes useful costructg lkelhood rato tests of hypotheses that put some restrcto o ad so uder the ull hypothess the maxmum lkelhood estmate of s ot x. I these cases we ca easly obta the maxmum lkelhood estmate of ad thus the value of the maxmzed lkelhood uder the ull hypothess.] NRJF, Uversty of Sheffeld, 011/1 Semester 1 0

7 8..1 The Maxmzed Log Lkelhood For the costructo of lkelhood rato tests we eed the actual form of the maxmzed lkelhood uder ull ad alteratve hypotheses. Typcally, the alteratve hypothess gves o restrctos o ad ad so the mles uder the alteratve hypothess are as gve earler (.e. ˆ x& ˆ S ). The ull hypothess wll ether mpose some 1 costrat o (e.g. H 0 : = 0 ) or some costrat o (e.g. H 0 : = 0 or H 0 : =1). I these cases we obta the estmate of ad the use the more geeral form gve above. For example, uder H 0 : = 0 we have ˆ= 0 ad so ths gves ˆ S (x )(x ) (x )(x ) 0 = S NRJF, Uversty of Sheffeld, 011/1 Semester 1 03

8 To calculate the actual maxmzed lkelhood ether case usually requres the use of a slck trck mapulatg matrces. Ths s the followg: Frst ote that a vector product such as yay where y s a p-vector ad A s pp s a scalar (.e. 11) Next ote that sce ths s a scalar we have trace(yay)=yay (oly oe dagoal elemet a 11 matrx). Next, applyg the rule that trace(bc)=trace(cb), f both products are defed, gves yay=trace(ayy) Next, otg that trace(b+c)=trace(b)+trace(c) gves yay trace{a yy} 1 1 The advatage of ths s that the matrx product o the rght had sde mght reduce to the detty matrx whose trace s easy to calculate. NRJF, Uversty of Sheffeld, 011/1 Semester 1 04

9 Now we have (, ; X)=log e (Lk(, ; X)) = = (x ) (x ) ½plog() ½log( ) { (x x) (x x) (x ) (x )} ½plog() ½log( ) ad 1 1 (x x) (x x) = trace{ 1 1 (x x) (x x) } 1 =trace{ (x x)(x x) } = trace{ 1 ( 1)S} 1 = ( 1)trace{ 1 S} So (,;X) = { (x x) (x x) (x ) (x )} ½plog() ½log( ) = ½( 1)trace{ 1 S} trace{ 1 (x )(x ) } ½plog() ½log( ) NRJF, Uversty of Sheffeld, 011/1 Semester 1 05

10 ad so max (, ;X) ( ˆ, ˆ;X), = ½( 1)tr{ 1 S} 0 ½plog() ½log( ) = ½( 1)tr{S 1 S/( 1)}) ½log ( 1)S/ = ½tr{I p } ½plog( (-1) / ) ½log S ½plog() ½plog( = ½p ½plog( (-1) / ) ½log S ½plog() More geerally, whatever the mle of s, f d= x, ˆ the we have max (, ;X) ( ˆ, ˆ;X) ˆ 1 Sdd ad = ½tr{ ˆ 1 [( 1)S dd ]} ½plog() ½log( ˆ ) = ½p ½plog() ½log( ˆ ) = ½p ½plog() ½log( 1 S dd ) NRJF, Uversty of Sheffeld, 011/1 Semester 1 06

11 8.3 Related Dstrbutos Itroducto Ths secto troduces two dstrbutos related to the multvarate ormal dstrbuto. The destes are ot gve (they ca be foud stadard texts) but some basc propertes of them are outled. Ther use s the costructo of tests, specfcally determg the dstrbuto of test statstcs. They are geeralzatos of famlar uvarate dstrbutos ad ther propertes match those of ther uvarate specal cases The Wshart Dstrbuto If X=(x 1, x,..., x ), ad f M pp =XX wth x ~N p (0,), (..d) the M~W p (,) the Wshart dstrbuto wth scale matrx ad degrees of freedom. Ths s a matrx geeralzato of the -dstrbuto : 1 f p=1 the M= x wth x ~N(0, ) Note that t s a ½p(p+1) dmesoal dstrbuto (M s symmetrc). Its stadard form s whe =I p. NRJF, Uversty of Sheffeld, 011/1 Semester 1 07

12 Its propertes are geeralzatos of those of the -dstrbuto, e.g. addtvty o the degrees of freedom parameter: f U~ W p (,m) ad V~W p (,) depedetly the U+V~W p (,m+) Its key use s as a termedate step dervg the dstrbuto of thgs of real terest. I partcular, f S= (x x)(x x), the sample varace, the (-1)S~W p (, -1) depedetly of x NRJF, Uversty of Sheffeld, 011/1 Semester 1 08

13 8.3. Hotellg s T Dstrbuto Ths s a uvarate dstrbuto of a scalar radom varable, t s a geeralzato of studet s t-dstrbuto or Sedecor s F-dstrbuto. Defto: If d~n p (0,I p ) ad M~W p (I p,) depedetly the dm 1 d~t (p,) Hotellg s T, parameter p, degrees of freedom. I partcular, f x~n p (, ) ad M~W p (, ) the we have (x )M 1 (x )~T (p,) ( prove by wrtg d = ½ (x ) ad M = ½ M ½ ) ad especally; ( x )S 1 ( x )~T (p, 1) (Notg the depedece of x ad S). Ths s the bass of oe ad two sample tests. To evaluate p-values we use the followg Theorem: T (p,) p p 1 Fp, p 1 Proof: Not gve see ay stadard text. Ths allows us to calculate a T value, multply by ( p+1)/p ad the refer the result to F tables wth p ad ( p+1) degrees of freedom. NRJF, Uversty of Sheffeld, 011/1 Semester 1 09

14 8.4 Smple oe & two sample tests Oe sample tests If x 1, x,..., x are depedet observatos of x~n p (, ) the we have ( x )S 1 ( x )~T (p, 1), so ( 1 ) p ( 1)p 1 ( x )S 1 ( x )~F p, p.e. p 1 p ( x )S 1 ( x )~F p, p So we ca test e.g. H 0 : = 0 vs 0 sce uder H 0 we have p 1 p ( x 0 )S 1 ( x 0 )~F p, p ad so we reject H 0 whe ths s mprobably large whe referred to a F-dstrbuto wth (p, p) degrees of freedom. NRJF, Uversty of Sheffeld, 011/1 Semester 1 10

15 8.4. Two sample tests The Mahalaobs dstace betwee two populatos wth meas 1 ad ad commo varace s defed as where = ( 1 ) 1 ( 1 ) If we have samples of szes 1 ad ; meas x 1 ad x ; varaces S 1 ad S the we defe the sample Mahalaobs dstace as D = ( x 1 x )S 1 ( x 1 x ) where S=[( 1 1)S 1 +( 1)S ]/( ) (.e. the pooled varace [or pooled varace-covarace]); = 1 + Now f 1 = the D ~ 1 T (p, ) sce we have x ~N p (, -1 ) ad ( 1)S ~W p (, 1); =1, so x 1 x ~N p ( 1, 1 ) ad ( )S~W p (, ) ad hece the result follows. The use s to test H 0 : 1 = sce we ca reject H 0 f 1 (p 1) D ( )p s mprobably large whe compared wth F p, p 1. NRJF, Uversty of Sheffeld, 011/1 Semester 1 11

16 8.4.3 Notes These oe ad two sample tests are easy to compute R, S-PLUS or MINITAB by drect calculato usg ther matrx arthmetc facltes. The two-sample test ca be calculated usg the geeral MANOVA facltes, see below. The lbrary ICSNP cotas a fucto HotellgsT(.) whch provdes oe ad two sample tests. Note that oe dmeso the best practce s always to use the separate varace verso of the two-sample t-test. I prcple t would be good to do the same hgher dmesos but there s o avalable equvalet of the Welch approxmato to obta approxmate degrees of freedom for the T -dstrbuto so the pooled varace verso s just a pragmatc expedet. NRJF, Uversty of Sheffeld, 011/1 Semester 1 1

17 8.5 Lkelhood Rato Tests Itroducto The oe ad two sample test statstcs for = 0 ad 1 = gve above are easly shew to be lkelhood rato statstcs (.e. optmal ). LRTs are a useful geeral procedure for costructg tests ad ca ofte be mplemeted umercally usg geeral purpose fucto maxmzato routes eve whe aalytc closed forms for maxmum lkelhood estmates are ot obtaable. Suppose data are avalable from a dstrbuto depedg o a parameter, where may be a vector parameter,.e. cosst of several separate parameters, (e.g. =(,), the parameters of a uvarate ormal dstrbuto whch has separate parameters, or e.g. =(,), the parameters of a p-dmesoal ormal dstrbuto has p+½p(p+1)=½p(p+3) separate parameters). Typcally, the ull hypothess H 0 wll specfy the values of some of these, e.g. the frst case H 0 : =0 specfes 1 parameter ad the secod t specfes p of them. NRJF, Uversty of Sheffeld, 011/1 Semester 1 13

18 The geeral procedure for costructg a lkelhood rato test (.e. fdg the LRT statstc) of H 0 versus H A s: 1. Fd the maxmum lkelhood estmates of all parameters assumg H 0 s true to get ˆ 0, e.g. wth N(, ) or N p (,), f H 0 : =0 the estmate or assumg =0 gvg ˆ x ˆ 1 xx ad 1 or the ˆ 0 =(0, 1 ) or ˆ =(0, 0 xx x 1 ). Fd the maxmum value of the log lkelhood uder H 0 (.e. substtute the mles of the parameters uder H 0 to the log lkelhood fucto) to get max (H ) ( ˆ ) Fd the maxmum lkelhood estmates of all parameters assumg H A s true, ˆ A. Typcally these wll be the ordary mles NRJF, Uversty of Sheffeld, 011/1 Semester 1 14

19 4. Fd the maxmum value of the log lkelhood uder H A (.e. substtute the mles of the parameters uder H A to the log lkelhood fucto) to (H ) ( ˆ ) get max A A 5. Calculate twce the dfferece maxmzed log lkelhoods, { max(h A ) max(h0 )}. 6. Use Wlks Theorem whch says that uder H 0 ths statstc s approxmately dstrbuted as wth degrees of freedom gve by the dfferece the umbers of estmated parameters uder H 0 ad H A,.e. ~ k where k=dm(h A ) dm(h 0 ) or 7. Fd some mootoc fucto of whch has a recogsable dstrbuto uder H 0. NRJF, Uversty of Sheffeld, 011/1 Semester 1 15

20 8.5.1 LRT of H 0 : = 0 vs. H A : 0 wth = 0 kow x 1, x,..., x depedet observatos of x~n p (, 0 ). To test H 0 : = 0 vs. H A : 0 wth 0 kow (.e. ot estmated). Now (; X)= log lk(; X) = ½plog() ½log 0 ½( 1)tr{ 0 1 S} ½( x ) 0 1 ( x ) = K ½(x ) 0 1 ( x ) So uder H 0 we have ( 0 ; X, H 0 ) = K ½( x 0 ) 1 0 ( x 0 ).e. max(h 0 ) = K ½( x 0 ) 1 0 ( x 0 ) Uder H A the mle of s x gvg max (H A ) = K So the LRT statstc s ={ max (H A ) max (H 0 )} = ( x 0 ) 0 1 ( x 0 ) ad the test s to reject ths f t s mprobably large whe compared wth p, otg that there are p parameters to be estmated uder H A but oe uder H 0. Also ote that ths s a exact result (.e. ot a Wlks Theorem approxmato) sce =yy=y wth y ~N(0,1) where y= ½ 0 ½ ( x - 0 )~N p (0,I p ). NRJF, Uversty of Sheffeld, 011/1 Semester 1 16

21 8.5. LRT of H 0 : = 0 vs H A : 0 ; ukow. x 1, x,..., x depedet observatos of x~n p (, ). To test H 0 : = 0 vs. H A : 0. Uder H 0 we have Uder H A we have ˆ x ad = 0. ˆ x ad ˆ 1 S = S say. Thus max (H 0 ) = ½tr{ -1 0 S } ½plog() ½log( 0 ) ad max (H A ) = ½p ½plog() ½log( S ) So ={ max (H A ) max (H 0 )}=tr{ -1 0 S } log( -1 0 S ) p ad the test s to reject H 0 f s mprobably large whe compared wth a dstrbuto o ½p(p+1) degrees of freedom (usg the asymptotc result of Wlks Theorem). Notce that tr{ -1 0 S p } = ad -1 0 S = 1 p 1 where are the egevalues of -1 0 S ad so we ca express as =p( log( ) 1) where ad are the arthmetc ad geometrc meas respectvely of the. NRJF, Uversty of Sheffeld, 011/1 Semester 1 17

22 LRT of =1 wth kow =I p x 1, x,..., x depedet observatos of x~n p (, I p ). To test H 0 : =1 vs. H A : 1. Let (; X) be the [urestrcted] lkelhood of, the (; X)= ½( 1)trace(S) ½( x )( x ) ½plog() To maxmze () uder H 0 we eed to mpose the costrat =1 ad so troduce a Lagrage multpler ad let = () ( 1). The (x ) ad dfferetatg w.r.t. gves =1. So we requre ˆ x ad the =1 mples (+) = x x So ˆ ad x xx max (H 0 )= ½( 1)trace(S) ½( x x xx = ½( 1)trace(S) ½ x (1 1 xx =½( 1)trace(S) ½( Uder H A we have )(1 1 xx )( x x xx xx 1) ½plog(). ˆ x ad so max (H A ) = ½( 1)trace(S) ½plog() gvg ={ max(h A ) max(h 0 )}= ( xx 1) ) ½plog() ) x ½plog() ad the test s to reject H 0 whe ths s mprobably large whe referred to a 1 dstrbuto. Note oly 1 degree of freedom sce has p depedet parameters so p are estmated uder H A ad uder H 0 wth oe costrat we have effectvely p 1 parameters, p (p 1)=1. NRJF, Uversty of Sheffeld, 011/1 Semester 1 18

23 8.5.4 Test of = 0 ; = 0 kow, 0 kow. x 1, x,..., x depedet observatos of x~n p ( 0, ). To test H 0 : = 0 vs. H A : 0 where both 0 ad 0 are kow. (Note that uder H 0, s the oly ukow parameter but uder H A all ½p(p+1) parameters of are ukow). Uder H 0 (; X)= ½tr{( 0 ) 1 S } ½log( 0 ) ½plog() (where S = (x )(x ) ) = ½ 1 tr{ 0 1 S } ½plog() ½log( 0 ) ½plog() so ½ tr{ 1 0 S } ½p 1 gvg ˆ tr( S ) 1 1 p 0 so max(h 0 )= ½p ½plog( ˆ ) ½log( 0 ) ½plog() Uder H A we have ˆ = S ad so max (H A ) = ½p ½log( S ) ½plog() The the LRT statstc s { max(h A ) max (H 0 )} ad ths would be compared wth r where r=½p(p+1) 1, usg Wlks Theorem. Although ths s ot a smple algebrac form, t ca be calculated umercally practce ad the test evaluated. NRJF, Uversty of Sheffeld, 011/1 Semester 1 19

24 8.5.5 Commets Examples ad 8.5. are multvarate geeralzatos of equvalet uvarate hypotheses, ad a useful check s to put p=1 ad verfy that the uvarate test s obtaed. (I Example ote that the egevalue of a 11 matrx (scalar) s the scalar tself). Examples ad llustrate the more structured hypotheses that ca be tested multvarate problems; they have o couterpart uvarate models. Such LRTs are a powerful all-purpose method of costructg tests ad ca ofte be mplemeted umercally eve f algebrac aalyss caot produce mles closed form. Further, they are a elegat applcato of geeral statstcal theory ad have varous desrable propertes they are guarateed to be [asymptotcally] most powerful,.e. they are more lkely tha ay other test to be able to detect successfully that the ull hypothess s false provded that the sample s large eough ad provded that the paret dstrbuto of the data s deed that presupposed (e.g. multvarate ormal). NRJF, Uversty of Sheffeld, 011/1 Semester 1 0

25 However, a dffculty multvarate problems volvg hypothess testg s that whe a hypothess s rejected t may ot be apparet just why t s false. Ths s ot so uvarate problems; f we have a model that uvarate x~n(, ) ad we reject H 0 : = 0 the we kow whether x> 0 or x< 0 ad hece why H 0 s false. I cotrast, f we have a model that multvarate x~n p (,) ad we reject H 0 : = 0 the all we kow s that there s evdece that ( 1,,..., p ) ( 01, 0,..., 0p ). [For multvarate we caot say > 0 ]. It may be that oly oe compoet 0 s ot correct ad that = 0 for all the others. That s we do ot kow the drecto of departure from H 0. That s, a lkelhood rato test may be able to reject a hypothess but ot actually reveal aythg terestg about the structure of the data, e.g. kowg that H 0 was early correct ad oly oe compoet was wrog could provde a useful sght to the data but ths mght be mssed by a LRT. Ths leads to cosderg a dfferet strategy for costructg tests whch mght provde more formato for data aalyss, though f they actually produce a dfferet test from the LRT the they may ot be so powerful (at least for suffcetly large data sets). NRJF, Uversty of Sheffeld, 011/1 Semester 1 1

26 8.6 Uo Itersecto Tests Itroducto Uo-Itersecto Tests (UITs) provde a dfferet strategy for costructg multvarate tests. They are ot avalable all stuatos (ulke LRTs), they do ot have ay geeral statstcal optmal propertes (aga ulke LRTs) ad sometmes they produce test statstcs that ca oly be assessed for statstcal sgfcace by smulato or Mote Carlo or Bootstrap procedures. However, they wll automatcally provde a dcato of the drecto of departure from a hypothess (just as uvarate problems t s apparet whether the sample mea s too bg or too small). The method s to project the data to oe dmeso (just as wth may multvarate exploratory data aalytc techques) ad test the hypothess that oe dmeso. The partcular dmeso chose s that whch shews the greatest devato from the ull hypothess, aga there are close aaloges wth multvarate EDA. The valdty of the procedure reles o the Cramér Wold Theorem whch establshes the coecto betwee the set of all oe-dmesoal projectos ad the multvarate dstrbuto. NRJF, Uversty of Sheffeld, 011/1 Semester 1

27 8.6.1 The Cramér Wold Theorem The dstrbuto of a p-vector x s completely determed by the set of all 1-dmesoal dstrbutos of 1-dmesoal projectos of x, tx, where t{all fxed p-vectors} Proof: Let y=tx, the, for ay t, the dstrbuto (ad hece the characterstc fucto) of y s kow ad s, say, y (s) = E[e sy st x ] = E[ e ] t x Puttg s=1 gves y (1) = E[ e ] s kow for all t p. t x But E[ e ]= x (t), the characterstc fucto of x,.e. x (t) s kow for all t p,.e. the dstrbuto of x s determed by specfyg the dstrbutos of tx for all t p. Importace: s that ay multvarate dstrbuto ca be defed by specfyg the dstrbuto of all of ts lear combatos (ot just the p margal dstrbutos), e.g. f we specfy that the mea of all oedmesoal projectos of x s 0, the ecessarly the mea of the p- dmesoal dstrbuto must be 0 (the coverse s true also of course). Note that specfyg that the p margals have a zero mea s ot suffcet to esure that the p-dmesoal dstrbuto s zero. NRJF, Uversty of Sheffeld, 011/1 Semester 1 3

28 8.6. A Example of a UIT Suppose x~n p (,I p ). The for ay p-vector we have that f y =x the y ~N(, I p ),.e. y ~N(, ) [ad ote that the C W theorem shews the coverse s true.] Suppose that we wat to test the hypothess H0: =0, based just o the sgle observato x. The, uder H 0, we have that for all, H 0 : y ~N(0,) s true..e. H 0 true H 0 true for all ad (by the C W theorem) H 0 true for all H 0 true..e. H 0 = H 0 H 0 s the tersecto of all uvarate hypotheses H 0. For ay, H 0 s a compoet of H 0. Now for ay specfc, H 0 s the hypothess that the mea of a ormal dstrbuto wth kow varace = s zero, ad we would reject H 0 at level f y c, for some sutable c (actually the upper 100 ½% pot of N(0,1).).e. the rejecto rego for H 0 s {y : c } = {x: x c }= R say y ad we reject H 0 f xr. NRJF, Uversty of Sheffeld, 011/1 Semester 1 4

29 Further: H 0 s true f ad oly f every H 0 s true..e. f ay of the H 0 s false the H 0 s false. So a sesble rejecto rego for H 0 s the uo of all the rejecto regos for the compoet hypotheses H 0,.e. reject H 0 f x. R.e. reject H 0 : =0 f ay oe-dmesoal projecto of x, x, s suffcetly dfferet from 0. x opt x opt x x x 0 opt opt 0 f H 0 s rejected the we kow whch (or s) cause the rejecto, ad hece the drecto of devato from H 0. [c.f. a -sded test the uvarate case, the we kow whether the mea s large or small] NRJF, Uversty of Sheffeld, 011/1 Semester 1 5

30 8.6.3 Defto A uo tersecto test of a multvarate hypothess s a test whose rejecto rego ca be wrtte as a uo of rejecto regos R, where R s the rejecto rego of a compoet hypothess H 0, where H 0 s the tersecto of the H 0. Ex cotued I the above case we reject H 0 f for ay we have c. y.e. H 0 s ot rejected (.e. accepted ) ff y c for all,.e. ff max y c,.e. ff max c y.e. ff max c yy.e. ff max xx ' c NRJF, Uversty of Sheffeld, 011/1 Semester 1 6

31 Now xx ' s varat uder scalar multplcato of, so we ca mpose the [o-restrctve] costrat =1 ad maxmze xx subject to ths costrat. Itroducg a Lagrage multpler gves the problem: maxmze =xx ( 1) w.r.t. ad. Dfferetatg w.r.t. gves xx = 0 so s a egevector of xx Now xx s of rak 1 ad so has oly oe o-zero egevalue. Ths egevector of xx s x wth egevalue xx: Check: (xx)x x=0 f =xx (sce (xx)x=x(xx), otg xx s a scalar)..e. max xx ' xxxx xx xx So the UIT of H 0 s to reject H 0 f xx>c, c chose to gve the desred sze of test. Now xx~ uder H p 0, so for a sze test take c=upper 100% pot of p. Ths s actually the same as the LRT. For ths problem ad clearly tellg the drecto of devato from =0 s ot dffcult wth just a sgle observato. The followg examples llustrate cases where more formato s obtaed from the UIT over ad above that gaed from the LRT. NRJF, Uversty of Sheffeld, 011/1 Semester 1 7

32 8.6.4 UIT of H 0 : = 0 vs. H A : 0, ukow. x 1, x,..., x depedet observatos of x~n p (, ). To test H 0 : = 0 vs. H A : 0 wth ukow (.e. to be estmated). Let be ay p-vector, ad y =x the y ~N(, ),.e. y ~N y, y) say. A compoet hypothess s H 0 : y = 0y ( 0y = 0 ) Ths eeds a test of a uvarate ormal mea, wth ukow varace usual oe-sample t-test ad we look at t y 0y where 1 sy 1 1 y s (y y) ( xx) = (x x)(x x) S Also y 0y = ( x 0 ) ad (y 0y ) = {( x 0 )} = ( x 0 )( x 0 ) so t (x )(x ) S 0 0 ad the compoet hypothess H 0 s rejected f ths s large. The uo tersecto test statstc s obtaed by maxmzg respect to :.e. t s t = max t (x )(x ) max S 0 0 t wth NRJF, Uversty of Sheffeld, 011/1 Semester 1 8

33 Now t s varat uder scalar multplcato of so mpose the [orestrctve] costrat S=1 ad maxmze stead = ( x 0 )( x 0 ) (S 1) w.r.t. ad. Dfferetatg w.r.t. shews that satsfes ( x 0 )( x 0 ) S = 0.e. S 1 ( x 0 )( x 0 ) = 0.e s the egevector of the [rak 1 pp matrx] S 1 ( x 0 )( x 0 ) correspodg to the oly o-zero egevalue. Now ths egevector s S 1 ( x 0 ) (or more exactly a scalar multple of t to satsfy S=1) Check: [S 1 ( x 0 )( x 0 )].[ S 1 ( x 0 )] S 1 ( x 0 ) = 0 for = ( x 0 )S 1 ( x 0 ) So t = ( x 0 )S 1 ( x 0 ) whch s Hotellg s T ad thus the UIT s detcal to the LRT. Further, f H 0 s rejected the ths shews that the drecto of devato s alog the vector S 1 ( x 0 ), ad we ca terpret ths drecto by lookg at the magtude of the loadgs o the dvdual compoets, just as PCA ad LDA..e. ot just alog the dfferece ( x 0 ) but adjusted to take accout of the dfferg varaces of the compoets of S. If S= I p the the drecto of devato s alog ( x 0 ). NRJF, Uversty of Sheffeld, 011/1 Semester 1 9

34 8.6.5 UIT of H 0 : = 0 vs H A : 0 ; ukow. x 1, x,..., x depedet observatos of x~n p (, ). To test H 0 : = 0 vs. H A : 0. [N.B. The LRT for ths problem was cosdered 4.5.] H 0 : y = 0y, tested by U =( 1)s y / 0y (~ -1 uder H 0 ) rejectg f ether U <c 1, or U >c,. So, the UIT s obtaed by rejectg H 0 f m {U } < c 1 or f max {U } > c (where c 1 ad c are chose to gve the test the desred sze). Now U =( 1)S/ 0 whch s max/mmzed whe ( 1) 1 0 S = 0 ad 0 =1 We have that f ( 1) 1 0 S = 0 ad 0 =1 the (pre-multplyg by 0 ) ( 1)S = 0 = Ad so max {U }= 1 ad m {U }= p where 1 > >...> p are the egevalues of 1 0 S. Thus the test s: ot the same as the LRT dcates that the drecto of devato s alog oe or other of the frst or last egevectors (ad whch t s wll be evdet from whether t s 1 that s too bg or p that s too small) requres smulato to apply practce sce there are o geeral results for UITs comparable to Wlks Theorem for LRTs NRJF, Uversty of Sheffeld, 011/1 Semester 1 30

35 8.7 Multsample Tests Multvarate Aalyss of Varace Setup: k depedet samples from N p (, ) of szes To test H 0 : 1 = =... = k (= say) vs H A : at least oe Lkelhood Rato Approach: Wlks test ( 1,,..., k,; X) = k 1 log log( ) ( 1)tr( S ) tr (x )(x ) p (.e. the sum of the k separate log-lkelhoods of the dvdual samples) Uder H 0 we have a sample of sze = k from N p (,), so mles are ˆ x, ˆ S ad so p p (H ) log( S ) max 0 log( ) Uder H A we have ˆ x the th sample mea, ad k k 1 1 ˆ W ( 1)S (W as defed 3.0) (H ) max A log W log( ) p p ad so k 1 S k W ad thus [ max(h A ) max(h 0 )]=log ad so a lkelhood rato test statstc for H 0 s ths s mprobably large. S 1 W W S, rejectg H 0 f NRJF, Uversty of Sheffeld, 011/1 Semester 1 31

36 Now k (k 1)B (x x)(x x) ( 1)S ( k)w 1 ad so a equvalet test statstc s W 1 [(k-1)b+(-k)w] or equvaletly I p + k 1 W 1 B rejectg f ths s large k or equvaletly = I p + k 1 W 1 B 1, rejectg f ths s small. k s sad to have a Wlks -dstrbuto (p, k,k 1) whch for some values of p,, k ( partcular k= or 3) s closely related to a F-dstrbuto. Addtoally, for other values of p, ad k, F ad approxmatos are avalable ad Bometrka Tables, vol, gve percetage pots. For k= ths test reduces to the -sample Hotellg s T test (see 8.3.) Computatoal Note I R ad S-PLUS the fucto maova(.) provdes facltes for multvarate aalyss of varace. MINITAB provdes Wlks test (complete wth p-values, exact for k3, approxmate otherwse) for oe-way multvarate aalyss of varace the meu Stat>ANOVA>Balaced MANOVA...) I MINITAB the meu Stat>ANOVA>Geeral MANOVA... provdes the same faclty. NRJF, Uversty of Sheffeld, 011/1 Semester 1 3

37 8.7.3 The Uo-Itersecto Test Followg the usual procedure, f s ay vector the the test statstc for testg H 0 s F =B/W whose maxmum value s the largest egevalue of W 1 B (see 4.3 o Crmcoords). For k= ths reduces to the -sample Hotellg s T test (whch s the same as the LRT) but for k> the UIT ad LRT are dfferet. The ull dstrbuto of ths largest egevalue s closely related to Roy s Greatest Root Dstrbuto, see Bometrka Tables Vol Further Notes R, MINITAB ad S-PLUS provde Roy s statstc as well as Wlks statstc the routes referred to I addto they produce two further statstcs: Plla s Trace ad the Lawley-Hotellg Trace. The frst of these s the trace of the matrx B(B+W) 1 ad the secod s the trace of W 1 B. Wlks test statstc ca be expressed as the product of all the egevalues of W(B+W) 1. All four of these statstcs measure or reflect the magtude of the matrx W 1 B whch s the obvous multvarate geeralzato of the F-statstc uvarate 1-way aalyss of varace. Geerally, all four tests should lead to equvalet coclusos f they do ot the there s somethg very uusual about the data whch eeds further vestgato. NRJF, Uversty of Sheffeld, 011/1 Semester 1 33

38 Hotellg s T statstc s most easly computed as ( ) Lawley- Hotellg Trace, usg the MANOVA opto descrbed above, the total umber of observatos. I prcple further extesos of MANOVA (e.g. -way or Geeral Multvarate Lear Model) are possble. MINITAB does ot provde these f you specfy a two factor model the Balaced Aova or Geeral Lear Model meu ad the ask for multvarate tests t wll gve you oly two separate 1-way MANOVAs, eve though t gves the full -way uvarate ANOVAs for each compoet. MANOVA s rarely the oly stage the aalyss, ot least because the terpretato of the results s ofte dffcult. It s always useful to look at the separate uvarate ANOVAs, supplemeted by the frst egevector of W 1 B. NRJF, Uversty of Sheffeld, 011/1 Semester 1 34

39 A key advatage of MANOVA over p separate uvarate ANOVAs s whe a expermet cossts of measurg lots of varables o the same dvduals the hope that at least oe (or eve some) wll shew dffereces betwee the groups, but t s ot kow whch of the p varables wll do so. Ths s a multple comparso problem whch s partally overcome by performg a tal MANOVA to see whether there are ay dffereces at all betwee the groups. If the MANOVA fals to reveal ay dffereces the there s lttle pot vestgatg dffereces o separate varables further. If there s some overall dfferece betwee the groups the examato of the coeffcets the frst egevector of W 1 B, together wth formal examato of the dvdual ANOVAs wll dcate whch varables or combato of varables (.e. drectos) cotrbute to the dffereces. NRJF, Uversty of Sheffeld, 011/1 Semester 1 35

40 8.8 Assessg Multvarate Normalty If a p-dmesoal radom varable has a multvarate ormal dstrbuto the t follows that the p oe dmesoal margal compoets must be uvarate ormal. However, the coverse does ot follow, t s possble that a p-dmesoal vable has uvarate Normal compoets but s ot multvarate ormal. Ths pecularty meas that although t s sesble to check each margal compoet of sample data for Normalty (e.g. by probablty plottg) t does ot follow that the multvarate data are satsfactorly multvarate Normally dstrbuted for the statstcal tests ad other procedures to be approprate. A further check s provded by the fact that the squared Mahalaobs dstaces of each observato from the mea D (x x)s (x x) 1 have approxmately a ch-squared dstrbuto wth p degrees of freedom, p. These dstaces wll ot actually be depedet but are early so, cosequetly a test of Normalty s provded by assessg the D as a sample of observatos from a p -dstrbuto. Evertt provdes a fucto chsplot() for producg a ch-squared probablty plot (.e. ordered observatos agast quatles of p ). As a example, cosder Evertt s ar polluto data arpoll ad the varables Educato ad Nowhte cosdered 0.8. Frst there are the two Normal probablty plots of the margal compoets (whch gve clear cause for cocer) followed by the chsquared plot whch s also ot very satsfactory: NRJF, Uversty of Sheffeld, 011/1 Semester 1 36

41 > attach(arpoll) > par(mfrow=c(1,)) > X<-cbd(Educato,Nowhte) > qqorm(x[,1],ylab="ordered observatos") > qqle(x[,1]) > qqorm(x[,],ylab="ordered observatos") > qqle(x[,]) > Ordered observatos Ordered observatos Quatles of Stadard Normal Quatles of Stadard Normal > par(mfrow=c(1,1)) > chsplot(x) Ordered dstaces Ch-square quatle NRJF, Uversty of Sheffeld, 011/1 Semester 1 37

42 8.9 Summary ad Coclusos Ths chapter has llustrated the exteso of basc uvarate results to multvarate data. Multvarate Normal, Wshart ad Hotellg s T -dstrbutos were troduced. The sample mea ad varace are ubased estmates of the populato mea ad varace. If addtoally, the data are Multvarate Normal the the sample mea s also Normal, the varace s Wshart ad they are depedet. Oe ad two-sample T -tests are drect geeralzatos of uvarate t-tests. Geeralzed lkelhood rato tests ca be costructed of hypotheses whch caot arse oe dmeso. Uo-Itersecto tests provde a alteratve strategy for costructg tests. These have smlartes wth multvarate EDA techques such as PCA ad LDA costructo ad terpretato of drectos. All stadard tests ca be performed stadard packages. NRJF, Uversty of Sheffeld, 011/1 Semester 1 38

43 NRJF, Uversty of Sheffeld, 011/1 Semester 1 39