Chater Geeral Lear Hyothess ad Aalyss of Varace Regresso model for the geeral lear hyothess Let Y, Y,..., Y be a seuece of deedet radom varables assocated wth resoses. The we ca wrte t as EY ( ) = β x, =,,...,, j=,,..., Var Y j j j= ( ) =. Ths s the lear model the exectato form where β, β,..., β are the ukow arameters ad x s are the kow values of deedet covarates j X, X,..., X. Alteratvely, the lear model ca be exressed as Y = β x, + ε, =,,..., ; j =,,..., j j j= where ε s are detcally ad deedetly dstrbuted radom error comoet wth mea ad varace,.e., E( ε ) = Var( ε ) = ad Cov( ε, ε ) = ( j). j I matrx otatos, the lear model ca be exressed as Y = Xβ + ε where Y ( Y, Y,..., Y )' s a vector of observatos o resose varable, = the matrx X X X... X X X... X = s a matrx of observatos o deedet X X... X covarates X, X,..., X, β = ( β, β,..., β ) s a vector of ukow regresso arameters (or regresso coeffcets) β, β,..., β assocated wth X, X,..., X, resectvely ad ε = ( ε, ε,..., ε ) s a vector of radom errors or dsturbaces. We assume that E( ε ) =, covarace matrx V = E = I rak X =. ( ε ) ( εε '), ( ) Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
I the cotext of aalyss of varace ad desg of exermets, the matrx X s termed as desg matrx; ukow β, β,..., β are termed as effects, the covarates X, X,..., X, are couter varables or dcator varables where x couts j the umber of tmes the effect β occurs the th observato x j. xj mostly takes the values or but ot always. The value x j = dcates the resece of effect x j = dcates the absece of effect β x. j β x j ad Note that the lear regresso model, the covarates are usually cotuous varables. Whe some of the covarates are couter varables ad rest are cotuous varables, the the model s called as mxed model ad s used the aalyss of covarace. Relatosh betwee the regresso model ad aalyss of varace model The same lear model s used the lear regresso aalyss as well as the aalyss of varace. So t s mortat to uderstad the role of lear model the cotext of lear regresso aalyss ad aalyss of varace. Cosder the multle lear model Y = β + X β + X β + + X β + ε.... I the case of aalyss of varace model, the oe-way classfcato cosders oly oe covarate, two-way classfcato model cosders two covarates, three-way classfcato model cosders three covarates ad so o. If βγ, ad δ deote the effects assocated wth the covarates X, Z ad W whch are the couter varables, the Oe-way model: Y = α + Xβ + ε Two-way model: Y = α + Xβ + Zγ + ε Three-way model : Y = α + Xβ + Zγ + Wδ + ε ad so o. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
Cosder a examle of agrcultural yeld. The study varable Y deotes the yeld whch deeds o varous covarates X, X,..., X. I case of regresso aalyss, the covarates X, X,..., X are the dfferet varables lke temerature, uatty of fertlzer, amout of rrgato etc. Now cosder the case of oe-way model ad try to uderstad ts terretato terms of multle regresso model. The covarate X s ow measured at dfferet levels, e.g., f X s the uatty of fertlzer the suose there are ossble values, say Kg., Kg.,,..., Kg. the X, X,..., X deotes these values the followg way. The lear model ow ca be exressed as Y = β + β X + β X + + β X + ε by defg. o... X X X f effect of Kg.fertlzer s reset = f effect of Kg.fertlzer s abset f effect of Kg.fertlzer s reset = f effect of Kg.fertlzer s abset f effect of Kg.fertlzer s reset = f effect of Kg.fertlzer s abset. If effect of Kg. of fertlzer s reset, the other effects wll obvously be abset ad the lear model s exressble as Y = β + β ( X = ) + β ( X = ) +... + β ( X = ) + ε = β + β+ ε If effect of Kg. of fertlzer s reset the Y = β + β ( X = ) + β ( X = ) +... + β ( X = ) + ε = β + β + ε If effect of Kg. of fertlzer s reset the Y = β + β ( X = ) + β ( X = ) +... + β ( X = ) + ε = β + β + ε ad so o. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
If the exermet wth Kg. of fertlzer s reeated umber of tmes the observato o resose varables are recorded whch ca be rereseted as Y Y = β + β. + β. +... + β. + ε = β + β. + β. +... + β. + ε Y = β + β. + β. +... + β. + ε If X = s reeated tmes, the o the same les umber of tmes the observato o resose varables are recorded whch ca be rereseted as Y Y Y = β + β. + β. +... + β. + ε = β + β. + β. +... + β. + ε = β + β. + β. +... + β. + ε The exermet s cotued ad f X = s reeated tmes, the o the same les Y Y Y = β + β. + β. +... + β.+ ε P = β + β. + β. +... + β.+ ε P = β + β. + β. +... + β.+ ε All these,,.., observatos ca be rereseted as y ε y ε y ε y ε y β ε β = + y ε β y ε y ε y ε or Y = Xβ + ε. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 4
I the two-way aalyss of varace model, there are two covarates ad the lear model s exressble as Y = β + β X + β X +... + β X + γ Z + γ Z +... + γ Z + ε where X, X,..., X deotes, e.g., the levels of uatty of fertlzer, say Kg., Kg.,..., Kg. ad Z, Z,..., Z deotes, e.g., the levels of level of rrgato, say Cms., Cms., Cms. etc. The levels X, X,..., X, Z, Z,..., Z are the couter varable dcatg the resece or absece of the effect as the earler case. If the effect of X ad Z are reset,.e., Kg of fertlzer ad Cms. of rrgato s used the the lear model s wrtte as Y = β + β. + β. +... + β. + γ. + γ. +... + γ. + ε = β + β + γ + ε. If X = ad Z = s used, the the model s Y = β + β + γ + ε. The desg matrx ca be wrtte accordgly as the oe-way aalyss of varace case. I the three-way aalyss of varace model Y = α + β X +... + β X + γ Z +... + γ Z + δw +... + δ W + ε r r The regresso arameters β 's ca be fxed or radom. If all β 's are ukow costats, they are called as arameters of the model ad the model s called as a fxed effect model or model I. The objectve ths case s to make fereces about the arameters ad the error varace. If for some j, x j = for all =,,..., the β s termed as addtve costat. I ths case, j occurs wth every observato ad so t s also called as geeral mea effect. If all β s are observable radom varables excet the addtve costat, the the lear model s termed as radom effect model, model II or varace comoets model. The objectve ths case s to make fereces about the varaces of varace ad/or certa fuctos of them.. β 's,.e., β j,,..., ad error β β β If some arameters are fxed ad some are radom varables, the the model s called as mxed effect model or model III. I mxed effect model, at least oe β j s costat ad at least oe β j s radom varable. The objectve s to make ferece about the fxed effect arameters, varace of radom effects ad error varace. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 5
Aalyss of varace Aalyss of varace s a body of statstcal methods of aalyzg the measuremets assumed to be structured as y = βx + βx +... + βx + ε, =,,..., where x are tegers, geerally or dcatg usually the absece or resece of effects j β j ; ad ε s are assumed to be detcally ad deedetly dstrbuted wth mea ad varace. It may be oted that the ε s ca be assumed addtoally to follow a ormal dstrbuto N(, ). It s eeded for the maxmum lkelhood estmato of arameters from the begg of aalyss but the least suares estmato, t s eeded oly whe coductg the tests of hyothess ad the cofdece terval estmato of arameters. The least suares method does ot reure ay kowledge of dstrbuto lke ormal uto the stage of estmato of arameters. We eed some basc cocets to develo the tools. Least suares estmate of β : Let y, y,..., y be a samle of observatos o Y, Y,..., Y. The least suares estmate of β s the values ˆβ of β for whch the sum of suares due to errors,.e., = S = ε = εε ' = ( yxβ)( yxβ) = yy X' y+ β XX β s mmum where y = ( y, y,..., y ). Dfferetatg S wth resect to β ad substtutg t to be zero, the ormal euatos are obtaed as ds = XX β Xy = dβ or XX β = Xy. If X has full rak, the ( XX ) has a uue verse ad the uue least suares estmate of β s ˆ β = ( XX) Xy whch s the best lear ubased estmator of β the sese of havg mmum varace the class of lear ad ubased estmator. If rak of X s ot full, the geeralzed verse s used for fdg the verse of ( XX ). Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 6
If L β s a lear arametrc fucto where L = (,,..., ) s a o-ull vector, the the least suares estmate of L β s L ˆ. β A uesto arses that what are the codtos uder whch a lear arametrc fucto L β admts a uue least suares estmate the geeral case. The cocet of estmable fucto s eeded to fd such codtos. Estmable fuctos: A lear fucto λβ of the arameters wth kow λ s sad to be a estmable arametrc fucto (or estmable) f there exsts a lear fucto LY of Y such that b E( LY ) = λβ for all β R. Note that ot all arametrc fuctos are estmable. Followg results wll be useful uderstadg the further tocs. Theorem : A lear arametrc fucto L β admts a uue least suares estmate f ad oly f L β s estmable. Theorem (Gauss Markoff theorem): If the lear arametrc fucto L β s estmable the the lear estmator L ˆ β where ˆβ s a soluto of XX ˆ β = XY s the best lear ubased estmator of L ' β the sese of havg mmum varace the class of all lear ad ubased estmators of L β. ' ' ' Theorem 3: If the lear arametrc fucto φ = lβ, φ = l β,..., φ = l β are estmable, the ay lear combato of φ, φ,..., φ k s also estmable. k k Theorem 4: All lear arametrc fuctos β are estmable f ad oly f X has full rak. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 7
If X s ot of full rak, the some lear arametrc fuctos do ot admt the ubased lear estmators ad othg ca be ferred about them. The lear arametrc fuctos whch are ot estmable are sad to be cofouded. A ossble soluto to ths roblem s to add lear restrctos o β so as to reduce the lear model to a full rak. Theorem 5: Let ' Lβ ad ther least suares estmators. The ' ˆ ' Var( Lβ) = L( X X ) L ' ˆ ' ˆ ' Cov( L β, L β) = L ( X X ) L ' Lβ be two estmable arametrc fuctos ad let ' ˆ L β ad ' ˆ L β be assumg that X s a full rak matrx. If ot, the geeralzed verse of XX ca be used lace of uue verse. Estmator of based o least suares estmato: Cosder a estmator of as, ˆ β ˆ β ˆ = ( yx )( yx ) [ ( ) ' ][ ( ) y X XX X y Y X XX Xy ] = = y'[ IX( XX ) X'][ IX( XX ) X ] y = y'[ I X( XX ) X ] y where the hat matrx s a demotet matrx wth ts trace as [ I X( XX) X ] = tr( X X ) X X (usg the result tr( AB) = tr( BA)) tr[ I X ( X X ) ] = tri trx ( X X ) X = tr I =. Note that usg E( y ' Ay) = µ ' Aµ + tr( AΣ ), we have E( ˆ ) = tr[ I X ( X X ) X ] = ad so ˆ s a ubased estmator of. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 8
Maxmum Lkelhood Estmato The least suares method does ot uses ay dstrbuto of the radom varables case of the estmato of arameters. We eed the dstrbutoal assumto case of least suares oly whle costructg the tests for hyothess ad the cofdece tervals. For maxmum lkelhood estmato, we eed the dstrbutoal assumto from the begg. Suose y, y,..., y are deedetly ad detcally dstrbuted followg a ormal dstrbuto wth mea E( y ) = β x ad varace Var( y ) = ( =,,, ). The the lkelhood fucto j j j= of y, y,..., y s L y y X y X ( π) ( ) ( β, ) = ex ( )( ) β β where y = ( y, y,..., y ). The L= L y = yx y X l ( β, ) log π log ( β) ( β). Dfferetatg the log lkelhood wth resect to β L = XX β = Xy β L = = yx β yx β ( )( ) ad, we have Assumg the full rak of X, the ormal euatos are solved ad the maxmum lkelhood estmators are obtaed as β = ( XX ) Xy = ( yx β )( yx β ) = y I X( XX ) X y. The secod order dfferetato codtos ca be checked ad they are satsfed for ˆβ ad to be the maxmum lkelhood estmators. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 9
Note that the maxmum lkelhood estmator β s same as the least suares estmator β ad β s a ubased estmator of β,.e., E( β) = β ulke the least suares estmator but s ot a ubased estmator of,.e., E( ) = estmator. lke the least suares Now we use the followg theorems for develog the test of hyothess. Theorem 6: Let Y ( Y, Y,..., Y ) = follow a multvarate ormal dstrbuto N ( µ, Σ ) wth mea vector µ ad ostve defte covarace matrx Σ. The Y AY follows a ocetral ch-suare dstrbuto wth degrees of freedom ad ocetralty arameter µ Aµ,.e., oly f Σ A s a demotet matrx of rak. χ (, µ Aµ ) f ad Theorem 7: Let Y ( Y, Y,..., Y ) = follows a multvarate ormal dstrbuto N ( µ, ) wth mea vector µ ad ostve defte covarace matrx Σ. Let YAY follows χ (, µ Aµ ) ad YAY follows χ (, µ A µ ). The YAY ad YAY are deedetly dstrbuted f AΣ A =. Theorem 8: Let Y = ( Y, Y,..., Y ) follows a multvarate ormal dstrbuto N (, I) µ, the the maxmum lkelhood (or least suares) estmator L ˆ β of estmable lear arametrc fucto s deedetly dstrbuted of ˆ ; L ˆ β follow ad ˆ N Lβ, L( XX) L follows χ ( ) where rak( X ) =. Proof: Cosder ˆ ( ) β = XX XY, the E Lˆ β = L XX XE Y ( ) ( ) ( ) = L( XX) XXβ = Lβ Var( L ˆ β) = L Var( ˆ β) L = LE ( ˆ β β)( ˆ β β) L = L ( XX ) L. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
Sce ˆβ s a lear fucto of y ad L ˆ β s a lear fucto of ˆβ, so L ˆ β follows a ormal dstrbuto. Let A= I X( XX ) X ad N Lβ, L( XX) L B L XX X = ( ), the ˆ L β = L ( XX ) XY = BY ad ˆ ad = ( Y Xβ)' ( ) I X XX X ( Y Xβ) = Y' AY. So, usg Theorem 6 wth rak(a) = -, = ( ) ( ) ( ) BA L XX X L XX XX XX X =. ˆ follows a χ ( ). Also So usg Theorem 7, Y ' AY ad BY are deedetly dstrbuted. Tests of Hyothess the Lear Regresso Model Frst we dscuss the develomet of the tests of hyothess cocerg the arameters of a lear regresso model. These tests of hyothess wll be used later the develomet of tests based o the aalyss of varace. Aalyss of Varace The techue the aalyss of varace volves the breakg dow of total varato to orthogoal comoets. Each orthogoal factor reresets the varato due to a artcular factor cotrbutg the total varato. Model Let Y, Y,..., Y be deedetly dstrbuted followg a ormal dstrbuto wth mea EY ( ) = β x ad varace. Deotg Y = ( Y, Y,..., Y ) a colum vector, such j j j= assumto ca be exressed the form of a lear regresso model Y = Xβ + ε where X s a matrx, β s a vector ad ε s a vector of dsturbaces wth E ( ε ) = Cov( ε) = I ad ε follows a ormal dstrbuto. Ths mles that EY ( ) = Xβ Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
E Y X Y X = I ( β)( β). Now we cosder four dfferet tyes of tests of hyothess. I the frst two cases, we develo the lkelhood rato test for the ull hyothess related to the aalyss of varace. Note that, later we wll derve the same test o the bass of least suares rcle also. A mortat dea behd the develomet of ths test s to demostrate that the test used the aalyss of varace ca be derved usg least suares rcle as well as lkelhood rato test. Case : Cosder the ull hyothess for testg H β = β : where β = ( β, β,..., β ), β = ( β, β,..., β )' s secfed ad s ukow. Ths ull hyothess s euvalet to H : β = β, β = β,..., β = β. Assume that all β ' s are estmable,.e., rak( X ) = (full colum rak). We ow develo the lkelhood rato test. The ( + ) dmesoal arametrc sace Ω s a collecto of ots such that { β β } Ω= (, ); < <, >, =,,.... Uder H, all β s s are kow ad eual, say β dmesoal sace gve by {(, ); } ω = β >. The lkelhood fucto of y, y,..., y s = L( y β, ) ex ( y Xβ )( y Xβ ) π all are kow ad the Ω reduces to oe The lkelhood fucto s maxmum over Ω whe β ad are substtuted wth ther maxmum lkelhood estmators,.e., ˆ β = ( XX ) Xy ˆ β ˆ β ˆ = ( yx )( yx ). Substtutg ˆβ ad ˆ Ly ( β, ) gves Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
( β, ) = ex ( ˆ β)( ˆ β) Max L y y X y X Ω πˆ ˆ = ex. π( y X ˆ β)( y X ˆ β) Uder H, the maxmum lkelhood estmator of s The maxmum value of the lkelhood fucto uder H s ( β, ) = ex ( β )( β) Max L y y X y X ω πˆ ˆ The lkelhood rato test statstc s (, ) Max L y β ω λ = Max L y β Ω (, ) ( yx ˆ β)( yx ˆ β) = ( yxβ )( yxβ ) = ex π( yxβ )( yxβ ) ˆ ˆ ( yxβ)( yxβ) = ' ˆ ˆ ˆ ˆ ( y Xβ) + ( Xβ Xβ ) ( y Xβ) + ( Xβ Xβ ) ( ˆ β β )' XX ( ˆ β β ) = + ( y X ˆ β)( y X ˆ β) = + where ˆ ˆ = ( yxβ) ( yxβ) = ˆ β β XX ˆ β β ad ( ) ( ). β β ˆ = ( yx )( yx ). The exresso of ad ca be further smlfed as follows. Cosder Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
= ˆ β β XX ˆ β β ( ) ( ) = ( XX) Xy β XX ( XX ) Xy β = ( XX) X ( y Xβ ) XX ( XX ) X ( yxβ ) = ( yxβ ) X( XX ) XX ( XX ) X ( yxβ ) = β ( y X ) X( XX) X ( y X ) = ( yx ˆ β)( yx ˆ β) = y X( XX) Xy yx( XX ) Xy = y I X( XX ) X y = y Xβ + Xβ I X XX X y Xβ + Xβ [( ) ] [ ( ) '][( ) ] = yxβ I X XX X yxβ ( )[ ( ) ]( ) β Other two terms become zero usg = [ I X( XX) X ] X I order to fd out the decso rule for H based o λ, frst we eed to fd f λ s a mootoc creasg or decreasg fucto of. So we roceed as follows: Let g =, so that λ = + = ( + g) the dλ = dg ( + g ) + So as g creases, λ decreases. Thus λ s a mootoc decreasg fucto of. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur The decso rule s to reject H f λ λ where λ s a costat to be determed o the bass of sze of the test α. Let us smlfy ths our cotext. 4
λ λ or or or or + λo ( + g ) λ o ( + ) λ g g λ or g C where C s a costat to be determed by the sze α codto of the test. So reject H wheever Note that the statstc C. ca also be obtaed by the least suares method as follows. The least suares methodology wll also be dscussed further lectures. ˆ ˆ = ( β β ) XX ( β β ) = M( yxβ)( yxβ) M( yxβ)( yxβ) ω sum of suares sum of suares sum of suares due to due to Ho due to error devato from Ho OR OR sum of suares due Totalsum of suares to β It wll be see later that the test statstc wll be based o the rato. I order to fd a arorate dstrbuto of, we use the followg theorem: Ω Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 5
Theorem 9: Let Q The Z = Y Xβ Q = ZX XX X Z ( ) ' Q = Z [ I X( XX ) X ] Z.. Q ad are deedetly dstrbuted. Further, whe Q H s true, the ~ χ ( ) Q ad ~ χ ( ) where χ ( m ) deotes the Proof: Uder H, ( ) EZ = Xβ Xβ = ( ) ( ). Var Z = Var Y = I χ dstrbuto wth m degrees of freedom. Further Z s a lear fucto of Y ad Y follows a ormal dstrbuto. So Z N I ~ (, ) The matrces X( XX) X ad are demotet matrces. So [ I X( XX) X ] tr X X X X = tr X X X X = tr I = tr[ I X ( X X ) X ] = tr I tr[ X ( X X ) X ] = [ ( ) ] [( ) ] ( ) So usg theorem 6, we ca wrte that uder H Q ~ χ ( ) Q ad ~ χ ( ) where the degrees of freedom ad ( ) are obtaed by the trace of X( XX) X ad trace of I X( XX) X, resectvely. Sce I X XX X X XX X = ( ) ( ), So usg theorem 7, the uadratc forms Q ad Q are deedet uder H. Hece the theorem s roved. Sce Q ad Q are deedetly dstrbuted, so uder H Q / Q /( ) follows a cetral F dstrbuto,.e. Q F(, ). Q Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 6
Hece the costat C the lkelhood rato test statstc λ s gve by C= F α (, ) where F α (, ) deotes the uer α % ots of F-dstrbuto wth ad degrees of freedom. The comutatos of ths test of hyothess ca be rereseted the form of a aalyss of varace table. ANOVA for testg Source of Degrees Sum of Mea F-value varato of freedom suares suares Due to β Error ( ) Total β β ( y X )( y X ) Case : Test of a subset of arameters H β = β k = r < whe βr+, βr+,..., β ad are ukow. :,,,.., k k I case, the test of hyothess was develoed whe all β s are cosdered the sese that we test for each β = β, =,,...,. Now cosder aother stuato, whch the terest s to test oly a subset of β, β,..., β,.e., ot all but oly few arameters. Ths tye of test of hyothess ca be used, e.g., the followg stuato. Suose fve levels of voltage are aled to check the rotatos er mute (rm) of a fa at 6 volts, 8 volts, volts, volts ad 4 volts. It ca be realzed ractce that whe the voltage s low, the the rm at 6, 8 ad volts ca be observed easly. At ad 4 volts, the fa rotates at the full seed ad there s ot much dfferece the rotatos er mute at these voltages. So the terest of the exermeter les testg the hyothess related to oly frst three effects, vz., β, for 6 volts, β for 8 volts ad β 3 for volts. The ull hyothess ths case ca be wrtte as: Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 7
H : β = β, β = β, β = β 3 3 whe β4, β 5 ad are ukow. Note that uder case, the ull hyothess wll be H : β = β, β = β, β = β, β = β, β = β. 3 3 4 4 5 5 Let β, β,..., β be the arameters. We ca dvde them to two arts such that out of β, β,..., βr, βr,..., β ad we are terested + testg a hyothess of a subset of t. Suose, We wat to test the ull hyothess H β = β k = r < whe βr+, βr+,..., β ad are ukow. : k k,,,.., The alteratve hyothess uder cosderato s H : β β for at least oe k =,,.., r <. I order to develo a test for such a hyothess, the lear model Y = Xβ + ε uder the usual assumtos ca be rewrtte as follows: Partto X = ( X X ), where () β() β = β() β = ( β, β,..., β ), β() = ( β, β,..., β ) r r+ r+ wth order as X : r, X : ( r), β() : r ad β : ( r). () k k The model ca be rewrtte as Y = Xβ + ε β() = ( X X) + ε β() = X β + X β + ε () () The ull hyothess of terest s ow H : β() = β() = ( β, β,..., β r ) where β () ad are ukow. The comlete arametrc sace s {( β, ); β,,,,..., } Ω= < < > = ad samle sace uder H s { () () r r } ω = ( β, β, ); < β <, >, = +, +,...,. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 8
The lkelhood fucto s ( β, ) = ex ( β)( β) L y y X y X π. The maxmum value of lkelhood fucto uder Ω s obtaed by substtutg the maxmum lkelhood estmates of β ad,.e., ˆ β = ( XX) Xy = ˆ β ˆ β ˆ ( y X )( y X ) as ( β, ) = ex ( ˆ β)( ˆ β) MaxL y y X y X Ω πˆ ˆ = ex. ˆ ' π( y Xβ)( y X ˆ β) Now we fd the maxmum value of lkelhood fucto uder H. The model uder H becomes Y = X β + X β + ε. The lkelhood fucto uder H s () = () () () () L( y β, ) ex ( y X β X β )( y X β X β ) π ex ( * y X β() )( y* X β() ) = π where () y* = y X β(). Note that () β ad are the ukow arameters. Ths lkelhood fucto looks lke as f t s wrtte for y N X * ~ ( β(), ). Ths hels s wrtg the maxmum lkelhood estmators of β () ad drectly as ˆ β = ( X X ) X y* ' ' () ˆ β ˆ β ˆ = ( y* X ())( y* X ()). Note that. ' ' XX s a rcal mor of XX. Sce XX s a ostve defte matrx, so XX s ' also ostve defte. Thus ( X X ) exsts ad s uue. Thus the maxmum value of lkelhood fucto uder H s obtaed as Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 9
ˆ = ˆ ˆ () () MaxL( y* β, ) ex ( y* X β )( y* X β ) ω πˆ ˆ = ex ( y* X ˆ ˆ π β() )'( y* X β() ) The lkelhood rato test statstc for H : β = β s () () λ = max Ly ( β, ) ω max Ly ( β, ) Ω ( yx ˆ β)( yx ˆ β) = ( * ˆ ˆ y X β() )( y* X β() ) - ( y* X ˆ β )( y* X ˆ β ) ( yx ˆ β)( y X ˆ β) + ( yx ˆ β)( yx ˆ β) ( y X ˆ β)( y X ˆ β) () () = - ( y* X ˆ ˆ ˆ ˆ β() )( y* X β() ) ( yxβ)( yxβ) = + ( y X ˆ β)( y X ˆ β) - = + = ( y* X ˆ β )( y* X ˆ β ) ( yx ˆ β)( yx ˆ β) ad ˆ ˆ = ( yxβ)( yx β). where () () Now we smlfy ad. Cosder ( y* X ˆ β )( y* X ˆ β ) = () () ' ' ' ' ' ' *' ( ) * = ( y* X ( X X ) X y*) ( y* X ( X X ) X y*) = y I X X X X y ' ' ' = ( yx β() X β() ) + X β () I X( XX) X ( yx β() X β() ) + X β () ' ' = ( y X β() X β() ) I X ( X X ) X ( y X β() X β() ). The other terms becomes zero usg the result X ' ' ' I X ( X X ) X =. Note that uder H, X β + X β ca be exressed as () () ( X X )( β β )', () () Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
Cosder ( yx ˆ β)( y X ˆ β) = = ( y X( X ' X) X ' y)( y X( X ' X) X ' y) y I X XX X y = ( ) ' ˆ = ( y X β() X β() ) X β ˆ () X β() ) I X( X ' X) X ( y X β() X β() ) X β() X β() ) + + + + ( yxβ() X β() ) IX( XX ) X ( yx β() X β() ) = ( yx β() X β() )' I X( XX ) X ( yx β() X β() ) ad other term becomes zero usg the result X I X XX X = ' ( ). Note that uder H, the term X β + X β ca be exressed as () () ( X X )( β β )'. () () Thus = ( y* X ˆ β )( y* X ˆ β ) ( yx ˆ β)( yx ˆ β) () () = y*' I X( X X) X y* y' I X( XX ) X y ' ' = ( yx β() X β() ) I X( X X) X ( yx β() X β() ) ( yx β() X β() )' IX( XX ) X ( yx β X β () () ' ' = ( yx β() X β() ) X( XX ) X X( X X) X ( yx β() X β() ) ) = ( yx ˆ β)( yx ˆ β) [ ( ) ] = y I X XX X y ' [ ] = ( yx β() X β() ) + ( X β() + X β() ) IX( XX) X ( yx β() X β() ) + ( X β() + X β() ) = ( yx β() X β() )' I X( XX ) X ( yx β() X β() ). Other terms become zero. Note that smlfyg the terms ad, we tred to wrte them the uadratc form wth same varable ( yx β X β ). () () Usg the same argumet as the case, we ca say that sce λ s a mootoc decreasg fucto of, so the lkelhood rato test rejects H wheever Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
> C where C s a costat to be determed by the sze α of the test. The lkelhood rato test statstc ca also be obtaed through least suares method as follows: ( + ) : Mmum value of ( yxβ)( yx β) whe : Sum of suares due to H H : β = β holds true. () () : Sum of suares due to error. : Sum of suares due to the devato from H or sum of suares due to β adjusted () for β. () If β () = the = ( yx ˆ β() ) '( yx ˆ ˆ ˆ β() ) ( yxβ) '( yxβ) ˆ' ' = ( yy β ˆ () Xy) ( yy β Xy ) ˆ ˆ ' ' = β Xy β X y. Reducto sum of suares or sum of suares due to β () sum of suares due to β() gorg β () Now we have the followg theorem based o the Theorems 6 ad 7. Theorem : Let Z = Y X β X β Q Q = Z AZ = Z BZ () () where A= X( XX ) X' X ( X X ) X B= I X XX X ( ) '. ' ' Q The ad Q Q are deedetly dstrbuted. Further ~ () Q χ r ad ~ χ ( ). Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur
Thus uder H, Q / r Q = Q /( ) r Q follow a F-dstrbuto Fr (, ). Hece the costat C λ s C= F α ( r, ), where F (, r ) deotes the uer α % ots o F-dstrbuto wth r ad ( ) degrees of freedom. α The aalyss of varace table for ths ull hyothess s as follows: ANOVA for testg H : β = β () () Source of Degrees Sum of Mea F-value Varato of suares suares Freedom Due to β r () Error Total ( r) + r ( ) r Case 3: Test of H : Lβ = δ Let us cosder the test of hyothess related to a lear arametrc fucto. Assumg that the lear arameter fucto L β s estmable where L = (,,..., ) s a vector of kow costats ad β = ( β, β,..., β ). The ull hyothess of terest s H : Lβ = δ. where δ s some secfed costat. Cosder the set u of lear model Y = Xβ + ε where Y = ( Y, Y,..., Y ) follows N( Xβ, I). The maxmum lkelhood estmators of β ad are ˆ β = ( XX ) Xy ad ˆ = ( yx ˆ β)( y X ˆ β) resectvely. The maxmum lkelhood estmate of estmable L β s L ˆ β, wth Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
EL ( ' ˆ β) = L β ( ˆ β) = ( ) ˆ β ~ β, ( ) Cov L L X X L L N L L XX L ad ˆ χ ~ ( ) assumg X to be the full colum rak matrx. Further, dstrbuted. Uder H : Lβ δ, = the statstc L ˆ β ad ˆ are also deedetly t = ( )( L ˆ β δ) ˆ ( ) L XX L follows a t-dstrbuto wth ( ) degrees of freedom. So the test for H : Lβ = δ agast H : Lβ δ rejects H wheever t t α ( ) where t α ( ) deotes the uer α % ots o t-dstrbuto wth degrees of freedom. Case 4: Test of H : φ = δ, φ = δ,..., φk = δk Now we develo the test of hyothess related to more tha oe lear arametrc fuctos. Let the th estmable lear arametrc fucto s φ = L β ad there are k such fuctos wth L ad β both beg vectors as the Case 3. ' Our terest s to test the hyothess H : φ = δ, φ = δ,..., φk = δk where δ, δ,..., δ k are the kow costats. Let φ = ( φ, φ,..., φ ) k ad δ = ( δ, δ,.., δ ) k. The H s exressble as H : φ = L β = δ where L s a k matrx of costats assocated wth L, L,..., L k. The maxmum lkelhood estmator of φ s : The ˆ φ = ( ˆ φ ˆ ˆ ˆ, φ,..., φk ) = Lβ. Also E( ˆ φ) = φ ˆ φ L ˆ = β where ˆ ( ) β = XX Xy. ' Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 4
ˆ Cov( φ) = V where V L XX L where ' = (( ( ) j)) L XX L s the (, j ) th elemet of V. Thus ' ( ( ) j) ˆ φ φ ˆ φ φ ( ) V ( ) follows a χ dstrbuto wth k degrees of freedom ad ˆ follows a χ dstrbuto wth ( ) degrees of freedom where ˆ ( ˆ)( ˆ = yxβ y Xβ) s the maxmum lkelhood estmator of. Further ˆ φ φ ˆ φ φ ( ) V ( ) Thus uder H :φ = δ or ˆ V k ˆ ( φ δ) ( φ δ) ˆ ( ) ad ˆ φδ V ˆ φδ k ˆ ( ) ( ) ˆ are also deedetly dstrbuted. follows F- dstrbuto wth k ad ( ) degrees of freedom. So the hyothess H : φ = δ s rejected agast H : At least oe φ δ for = ì,,..., k wheever F F α ( k, ) where F α ( k, ) deotes the α% ots o F-dstrbuto wth k ad ( ) degrees of freedom. Oe-way classfcato wth fxed effect lear models of full rak: The objectve the oe-way classfcato s to test the hyothess about the eualty of meas o the bass of several samles whch have bee draw from uvarate ormal oulatos wth dfferet meas but the same varaces. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 5
Let there be uvarate ormal oulatos ad samles of dfferet szes are draw from each of the oulato. Let y ( j =,,..., ) be a radom samle from the th ormal oulato wth mea β ad varace j, =,,...,,.e. j ~ ( β, ), =,,..., ; =,,...,. Y N j The radom samles from dfferet oulato are assumed to be deedet of each other. These observatos follow the set u of lear model Y = Xβ + ε where Y = ( Y, Y,..., Y, Y,..., Y,..., Y, Y,..., Y )' y = ( y, y,..., y, y,..., y,..., y, y,..., y )' β = ( β, β,..., β ) ε = ( ε, ε,..., ε, ε,..., ε,..., ε, ε,..., ε )' X... values... values =...... values... x th f β occurs the j observato = or f effect β s reset x f effect β s abset x j j j =. = x j So X s a matrx of order, β s fxed ad - frst rows of ε are ε ' = (,,,...,), ' - ext rows of ε are ε = (,,,...,) Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 6
- ad smlarly the last rows of ε are ε = (,,...,,). ' Obvously, ( ), ( ) rak X = E Y = X β ad Cov Y ( ) = I. Ths comletes the reresetato of a fxed effect lear model of full rak. The ull hyothess of terest s H : β = β =... = β = β (say) ad H : At least oe β β ( j) j where β ad are ukow. We would develo here the lkelhood rato test. It may be oted that the same test ca also be derved through the least suares method. Ths wll be demostrated the ext module. Ths way the readers wll uderstad both the methods. We already have develoed the lkelhood rato for the hyothess H : β = β =... = β the case. The whole arametrc sace Ω s a ( + ) dmesoal sace {( β, ) : β,,,,..., } Ω = < < > =. Note that there are ( + ) arameters are β, β,..., β ad. Uder H, Ω reduces to two dmesoal sace ω as { } ω= ( β, ); < β<, >.. The lkelhood fucto uder Ω s = y j β π = j= Ly ( β, ) ex ( ) L= l Ly ( β, ) = l ( π ) ( y β ) L ˆ = β = y = y β L j = j= j o j= = = ˆ ( yj yo ). = j= Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 7
The dot sg ( o ) y o dcates that the average has bee take over the secod subscrt j. The Hessa matrx of secod order artal dervato of l L wth resect to β ad s egatve defte at β = values. y ad = ˆ whch esures that the lkelhood fucto s maxmzed at these o Thus the maxmum value of Lyβ (, ) over Ω s ˆ = y j β πˆ = j= Max L( y β, ) ex ( ) Ω The lkelhood fucto uder ω s / = ex. π ( yj yo ) = j= = y j β π = j= Ly ( β, ) ex ( ) l Ly ( β, ) l( π ) ( β ) = yj = j= The ormal euatos ad the least suares are obtaed as follows: l Ly ( β, ) β = ˆ β = yj = y = j= l Ly ( β, ) = ˆ = yj yoo = j= ( ). oo The maxmum value of the lkelhood fucto over ω uder H s ˆ = j ˆ y β π ˆ = j= / Max L( y β, ) ex ( ) ω = ex. π ( yj yoo) = j= The lkelhood rato test statstc s Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 8
Max L y ω λ = Max L y Ω ( β, ) ( β, ) / ( yj yo ) = j= = ( yj yoo) = j= We have that ( y ) j yoo = ( yj yo ) + ( yo yoo) = j= = j= ( yj yo ) ( yo yoo) = j= = = + Thus ( yj y ) + ( yo yoo) = = = λ = j I ( yj yo ) = j= = + where = ( yo y oo), ad = = ( y j y o). = j= Note that f the least suares rcal s used, the : sum of suares due to devatos from H or the betwee oulato sum of suares, : sum of suares due to error or the wth oulato sum of suares, + : sum of suares due to H or the total sum of suares. Usg the theorems 6 ad 7, let = ( o oo), = = = Q Y Y Q S where Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 9
= ( j o) = S Y Y Y Y oo o = = = j= j= Y j Y the uder H j Q Q ~ χ ( ) ~ χ ( ) Q Q ad ad are deedetly dstrbuted. Thus uder H Q ~ F(, ). Q The lkelhood rato test reject H wheever > C where the costat C = F (, ). α The aalyss of varace table for the oe-way classfcato fxed effect model s Source of Degrees of Sum of Mea sum F Varato freedom suares of suares Betwee Poulato Wth Poulato Total +. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
Note that Q = E ( β β) Q = E = + ; = β β = Case of rejecto of H If F > F α (, ), the H : β = β =... = β s rejected. Ths meas that at least oe β s dfferet from other whch s resosble for the rejecto. So objectve s to vestgate ad fd out such β ad dvde the oulato to grou such that the meas of oulatos wth the grous are same. Ths ca be doe by arwse testg of β '. s Test H : ( ) β = β k agast H : β β. k Ths ca be tested usg followg t-statstc t = Yo Yko s + k whch follows the t dstrbuto wth ( ) degrees of freedom uder H ad k s = Thus. the decso rule s to reject H at level α f the observed dfferece ( y y ) t s k o ko > α +, The uatty t α, Thus followg stes are followed : s + s called the crtcal dfferece. k. Comute all ossble crtcal dffereces arsg out of all ossble ar ( β, βk), k =,,...,.. Comare them wth ther observed dffereces 3. Dvde the oulatos to dfferet grous such that the oulatos the same grou have same meas. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
The comutato are smlfed f (CCD) s = for all. I such a case, the commo crtcal dfferece CCD = t α, s ad the observed dfferece ( y y ), k are comared wth CCD. If y y > CCD o ko o ko the the corresodg effects/meas y o ad y ko are comg from oulatos wth the dfferet meas. Note: I geeral we say that f there are three effects β, β, β 3 the H : β = β ( deote as evet A) s acceted f ad f H : β = β3( deote as evet B) s acceted the H3 : β = β( deote as evet C) wll be acceted. The uesto arses here that what sese do we coclude such statemet about the accetace of H 3. The reaso s as follows: Sce evet A B C, so PA ( B) PC ( ) I ths sese f the robablty of a evet s hgher tha the tersecto of the evets,.e., the robablty that H 3 s acceted s hgher tha the robablty of accetace of H ad H both, so we coclude, geeral, that the accetace of H ad H mly the accetace of H 3. Multle comarso test: Oe terest the aalyss of varace s to decde whether oulato meas are eual or ot. If the hyothess of eual meas s rejected the oe would lke to dvde the oulatos to subgrous such that all oulatos wth same meas come to the same subgrou. Ths ca be acheved by the multle comarso tests. A multle comarso test rocedure coducts the test of hyothess for all the ars of effects ad comare them at a sgfcace level α.e., t works o er comarso bass. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 3
Ths s based maly o the t-statstc. If we wat to esure that the sgfcace level α smultaeously for all grou comarso of terest, the aroxmate multle test rocedure s oe that cotrols the error rate er exermet bass. There are varous avalable multle comarso tests. We wll dscuss some of them the cotext of oe-way classfcato. I two-way or hgher classfcato, they ca be used o smlar les.. Studetzed rage test: It s assumed the Studetzed rage test that the samles, each of sze, have bee draw from ormal oulatos. Let ther samle meas be y,..., o, y o y These meas are raked ad arraged o a ascedg order as y, y,..., y where * * * y M y ad * = o y = Max y, =,,...,. * o Fd the rage as R= y y. * * The Studetzed rage s defed as, = R s where α,, γ avalable. s the uer α % ot of Studetzed rage whe γ =. The tables for α,, γ are The testg rocedure volves the comarso of, γ wth the usual way as-,, β = β =... = β f, < the coclude that. α,, α γ f, > the all α,, β s the grou are ot the same.. Studet-Newma-Keuls test: The Studet-Newma-Keuls test s smlar to Studetzed rage test the sese that the rage s comared wth α % ots o crtcal Studetzed rage W P gve by W s = α,, γ. The observed rage R= y y s ow comared wth W. * * If R< W the sto the rocess of comarso ad coclude that β = β =... = β. If R> W the Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 33
() dvde the raked meas * * * y, y,..., y to two subgrous cotag - ( y, y,..., y ) * * * ad ( y, y,..., y ) * * * () Comute the rages R = y y ad * * R = y y. The comare the rages R ad R * * wth W. If ether rage ( R or R ) s smaller tha W, the the meas (or β s) each of the grous are eual. If R ad/or R are greater the W, the the ( ) meas (or β s) the grou cocered are dvded to two grous of ( ) meas (or β s) each ad comare the rage of the two grous wth W. Cotue wth ths rocedure utl a grou of meas (or β s) s foud whose rage does ot exceed W. By ths method, the dfferece betwee ay two meas uder test s sgfcat whe the rage of the observed meas of each ad every subgrou cotag the two meas uder test s sgfcat accordg to the studetzed crtcal rage. Ths rocedure ca be easly uderstood by the followg flow chart. Arrage y o ' s creasg order y y... y * * * Comute Comare wth W R= y y * * = α,, γ s If R< W Sto ad coclude β = β =... = β If R> W cotue Dvde raked mea grous ( y,..., y ) ad ( y,..., y ) * * * *,. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 34
Comute R = y y * * R = y y * * Comare R adr wth W 4 ossbltes of R ad R wth W R R < W < W β = β =... = β ad 3 β = β =... = β β = β =... = β R R β = β =... = β 3 3 β β, j =,,..., < W > W ad at least oe whch s β j oe subgrou s ( β, β,..., β ) ad aother grou has oly β R R β = β =... = β β β, j =,3,..., > W < W ad at least oe whch s β j ( β, β,..., β ) Oe subgrou has ad aother has oly β R R. y,..., y > W > W Dvde raked meas four grous * * 3. y,..., y * * 3. y,..., y (same as ) * * 4. y,..., y * * Comute R = y y * * 3 3 R = y y * * 4 R = y y * * 5 ad comare wth W Cotue tll we get subgrous wth same β 's Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 35
3. Duca s multle comarso test: The test rocedure Duca s multle comarso test s the same as the Studet-Newma- Keuls test excet the observed rages are comared wth Duca s α % crtcal rage D = where * α,, γ s α = ( α), α γ deotes the uer α % ots of the Studetzed rage based o Duca s rage. *,, Tables for Duca s rage are avalable. Duca feels that ths test s better tha the Studet-Newma-Keuls test for comarg the dffereces betwee ay two raked meas. Duca regarded that the Studet-Newma-Keuls method s too strget the sese that the true dffereces betwee the meas wll ted to be mssed too ofte. Duca otes that testg the eualty of a subset k,( k ) meas through ull hyothess, we are fact testg whether ( ) orthogoal cotrasts betwee the β 's dffer from zero or ot. If these cotrasts were tested searate deedet exermets, each at level α, the robablty of correctly rejectg the ull hyothess would be ( α). So Duca roosed to use ( α) lace of α the Studet-Newma-Keuls test. [Referece: Cotrbutos to order statstcs, Wley 96, Chater 9 (Multle decso ad multle comarsos, H.A. Davd, ages 47-48)]. Case of ueual samle szes: Whe samle meas are ot based o the same umber of observatos, the rocedures based o Studetzed rage, Studet-Newma-Keuls test ad Duca s test are ot alcable. Kramer roosed that Duca s method, f a set of meas s to be tested for eualty, the relace * s * * by * s α,, γ α,, γ + U L where U ad L are the umber of observatos corresodg to the largest ad smallest meas the data. Ths rocedure s oly a aroxmate rocedure but wll ted to be coservatve, sce meas based o a small umber of observatos wll ted to be overrereseted the extreme grous of meas. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 36
Aother oto s to relace by the harmoc mea of,,...,,.e., 4. The Least Sgfcat Dfferece (LSD): =. I the usual testg of H : β = β agast H :, β β the t-statstc k k t = y o y ko Var ( y y ) o ko s used whch follows a t-dstrbuto, say wth degrees of freedom ' df '. Thus H s rejected wheever t > t α df, ad t s cocluded that β ad β are sgfcatly dfferet. The eualty t > t ca be α df, euvaletly wrtte as y o yko > t α Var( yo yko)., df If every ar of samle for whch y o yko exceeds t α Var( yo yko), df the ths wll dcate that the dfferece betwee β ad β k s sgfcatly dfferet. So accordg to ths, the uatty t α Var( yo yko) would be the least dfferece of y o ad, df y ko for whch t wll be declared that the dfferece betwee β ad ooled varace of the two samle ( ) defed as o ko βk s sgfcat. Based o ths dea, we use the Var y y as s ad the Least Sgfcat Dfferece (LSD) s LSD = t s + α df, k. If = =, the s LSD = t. Now all α df, ( ) ars of y o ad y, ( k =,,..., ) are comared wth LSD. Use of LSD ko crtero may ot lead to good results f t s used for comarsos suggested by the data Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 37
(largest/smallest samle mea) or f all arwse comarsos are doe wthout correcto of the test level. If LSD s used for all the arwse comarsos the these tests are ot deedet. Such correcto for test levels was cororated Duca s test. 5. Tukey s Hoestly sgfcat Dfferece (HSD) I ths rocedure, the Studetzed rak values α,, γ are used lace of t-uatles ad the stadard error of the dfferece of ooled mea s used lace of stadard error of mea commo crtcal dfferece for testg H : β = βk agast H : β βk ad Tukey s Hoestly Sgfcat Dfferece s comuted as HSD MS error = α.,, γ assumg all samles are of the same sze. All If yo yko > HSD the β ad ( ) β k are sgfcatly dfferet. ars yo yko are comared wth HSD. We otce that all the multle comarso test rocedure dscussed u to ow are based o the testg of hyothess. There s oe-to-oe relatosh betwee the testg of hyothess ad the cofdece terval estmato. So the cofdece terval ca also be used for such comarsos. Sce H : β = βk s same as H : β β = k so frst we establsh the relatosh ad the descrbe the Tukey s ad Scheffe s rocedures for multle comarso test whch are based o the cofdece terval. We eed the followg cocets. Cotrast: A lear arametrc fucto L= l 'β = β where β = ( β, β,..., β P ) ad = (,,..., ) are = the vectors of arameters ad costats resectvely s sad to be a cotrast whe = =. For examle β β =, β+ β β3 β =, β+ β 3β3 = etc. are cotrast whereas β + β =, β + β + β + β =, β β 3β = etc. are ot cotrasts. 3 4 3 Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 38
Orthogoal cotrast: If L = 'β = β ad = 'β = L m m β are cotrasts such that m = or = = = m = the L ad L are called orthogoal cotrasts. For examle, L = β+ β β3 β4 ad L = β β + β3 β4 are cotrasts. They are also the orthogoal cotrasts. The codto m = esures that L ad L are deedet the sese that = (, ) = m =. = Cov L L Mutually orthogoal cotrasts: If there are more tha two cotrasts the they are sad to be mutually orthogoal, f they are arwse orthogoal. It may be oted that the umber of mutually orthogoal cotrasts s the umber of degrees of freedom. Comg back to the multle comarso test, f the ull hyothess of eualty of all effects s rejected the t s reasoable to look for the cotrasts whch are resosble for the rejecto. I terms of cotrasts, t s desrable to have a rocedure () that ermts the selecto of the cotrasts after the data s avalable. () wth whch a kow level of sgfcace s assocated. Such rocedures are Tukey s ad Scheffe s rocedures. Before dscussg these rocedure, let us cosder the followg examle whch llustrates the relatosh betwee the testg of hyothess ad cofdece tervals. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 39
Examle: Cosder the test of hyothess for H : β = β j ( j =,,..., ) or H : β β j = or H : cotrast = or H : L=. The test statstc for H : β = β s j ( ˆ β ˆ ˆ β j) ( β β j) L L t = = Var ( y y ) Var ( Lˆ ) o ko where ˆβ deotes the maxmum lkelhood (or least suares) estmator of β ad t follows a t- dstrbuto wth df degrees of freedom. Ths statstc, fact, ca be exteded to ay lear cotrast, say e.g., L= β+ β β ˆ ˆ ˆ ˆ ˆ 3 β4, L = β+ β β3 β4. The decso rule s reject H : L= agast H : L. If Lˆ > t Var ( Lˆ). df The ( α) % cofdece terval of L s obtaed as Lˆ L P tdf t df = α Var ( Lˆ ) or P Lˆt ( ˆ) ˆ ( ˆ df Var L L L + tdf Var L) = α so that the ( α) % cofdece terval of L s Lˆ t ( ˆ), ˆ ( ˆ df Var L L + tdf Var L) ad Lˆt Var ( Lˆ) L Lˆ+ t Var ( Lˆ) df df If ths terval cludes L = betwee lower ad uer cofdece lmts, the H : L= s acceted. Our objectve s to kow f the cofdece terval cotas zero or ot. Suose for some gve data the cofdece tervals for β β ad β β3 are obtaed as 3 ββ ad β β3 4. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 4
Thus we fd that the terval of β β cludes zero whch mles that H : β β= s acceted. Thus β = β. O the other had terval of β β 3 does ot clude zero ad so H : β β = s ot acceted. Thus β β. 3 3 If the terval of β β 3 s ββ3 the H : β = β3 s acceted. If both H : β = β ad H : β = β3 are acceted the we ca coclude that β = β = β3. Tukey s rocedure for multle comarso (T-method) The T-method uses the dstrbuto of studetzed rage statstc. (The S-method (dscussed ext) utlzes the F-dstrbuto). The T-method ca be used to make the smultaeous cofdece statemets about cotrasts ( β β ) error varace f certa restrctos are satsfed. amog a set of arameters j {,,..., } β β β ad a estmate s of These restrctos have to be vewed accordg to the gve codtos. For examle, oe of the restrctos s that all ˆ β ' s have eual varaces. I the setu of oe-way classfcato, ˆβ has ts mea Y ad ts varace s. Ths reduces to a smle codto that all ' s are same,.e., = for all. so that all the varaces are same. Aother assumto s to assume that ˆ β ˆ ˆ, β,..., β are statstcally deedet ad the oly cotrasts cosdered are the ( ) We make the followg assumtos: dffereces { β β j, j,,..., } =. () The ˆ β ˆ ˆ, β,..., β are statstcally deedet () ˆ ~ (, N a ),,,...,, a β β = > s a kow costat. () s s a deedet estmate of wth γ degrees of freedom (here γ = ),.e., γ s ~ χ ( γ ) ad (v) s s statstcally deedet of ˆ β ˆ ˆ, β,..., β. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 4
The statemet of T-method s as follows: Uder the assumtos ()-(v), the robablty s ( α) that the values of cotrasts L= Cβ ( C = ) smultaeously satsfy = = ˆ ˆ L Ts C L L + Ts C = = where Lˆ = Cβˆ, ˆ β s the maxmum lkelhood (or least suares) estmate of β, T a,,, = = α γ wth α,, γ beg the uer α ot of the dstrbuto of studetzed rage. Note that f L s a cotrast lke β β j( j) the a = ad the terval smlfes to ( ˆ β ˆ β ) Ts β β ( ˆ β ˆ β ) + Ts j j j C = ad the varace s so that = where T = α,, γ. Thus the maxmum lkelhood (or least suares) estmate Lˆ = ˆ β ˆ β of L = β β s sad to be sgfcatly dfferet from zero accordg to T-crtero f the terval j ( ˆ β ˆ β Ts, ˆ β ˆ β + Ts) does ot cover β β =,.e., j j f ˆ β ˆ β > j Ts or more geeral f ˆ L > Ts C. = The stes volved the testg ow volve the followg stes: - Comute L ( ˆ β ˆ β j) ˆ or. - Comute all ossble arwse dffereces. - Comare all the dffereces wth j j α,, γ. s = C - If ˆ or ( ˆ ˆ L β β j) > Ts C = the ˆβ ad ˆβ j are sgfcatly dfferet where Tables for T are avalable. T = α,, γ. Whe samle szes are ot eual, the Tukey-Kramer Procedure suggests to comare ˆL wth Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 4
α,, γ s + C j = or T + C. j = The Scheffe s method (S-method) of multle comarso S-method geerally gves shorter cofdece tervals the T-method. It ca be used a umber of stuatos where T-method s ot alcable, e.g., whe the samle szes are ot eual. A set L of estmable fuctos { ψ } s called a -dmesoal sace of estmable fuctos f there exsts learly deedet estmable fuctos ( ψ, ψ,..., ψ ) such that every ψ L s of the form ψ = Cy where C, C,..., C are kow costats. I other word, L s the set of all = lear combatos of ψ, ψ,..., ψ. Uder the assumto that the arametrc sace Ω s Y ~ N( Xβ, I) wth rak( X ) =, β = ( β,..., β ), X s matrx, cosder a -dmesoal sace L of estmable fuctos geerated by a set of learly deedet estmable fuctos { ψ, ψ,..., ψ }. For ay ψ L, Let ψˆ = Cy be ts least suares (or maxmum lkelhood) estmator, Var( ψˆ ) = = C = ψ ( say) = ad ˆ ψ = s C = where s s the mea suare due to error wth ( ) degrees of freedom. The statemet of S-method s as follows: Uder the arametrc sace Ω, the robablty s ( α) that smultaeously for all ψ L ψˆ S ˆ ψ ψˆ + S ˆ where the costat S = F (, )., ψˆ ψˆ Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur α 43
Method: For a gve sace L of estmable fuctos ad cofdece coeffcet ( α), the least suare (or maxmum lkelhood) estmate ˆ ψ of ψ L wll be sad to be sgfcatly dfferet from zero accordg to S-crtero f the cofdece terval ( ψˆ S ˆ ψ ψˆ + S ˆ ) ψˆ does ot cover ψ =,.e., f ψˆ > S ˆ. ψˆ ˆψ The S-method s less sestve to the volato of assumtos of ormalty ad homogeety of varaces. Comarso of Tukey s ad Scheffe s methods:. Tukey s method ca be used oly wth eual samle sze for all factor level but S-method s alcable whether the samle szes are eual or ot.. Although, Tukey s method s alcable for ay geeral cotrast, the rocedure s more owerful whe comarg smle arwse dffereces ad ot makg more comlex comarsos. 3. It oly arwse comarsos are of terest, ad all factor levels have eual samle szes, Tukey s method gves shorter cofdece terval ad thus s more owerful. 4. I the case of comarsos volvg geeral cotrasts, Scheffe s method teds to gve marrower cofdece terval ad rovdes a more owerful test. 5. Scheffe s method s less sestve to the volatos of assumtos of ormal dstrbuto ad homogeety of varaces. Aalyss of Varace Chater Geeral Lear Hyothess ad Aova Shalabh, IIT Kaur 44