Understand the properties of the parameter estimates, how we compute confidence intervals for them and how we test hypotheses about them.

Size: px

Start display at page:

Download "Understand the properties of the parameter estimates, how we compute confidence intervals for them and how we test hypotheses about them."

Nicholas Green
5 years ago
Views:

1 9.07 Itroducto to Statstcs for Bra ad Cogtve Sceces Emery N. Brow I. Objectves Lecture 13 Lear Model I: Smple Regresso Uderstad the smple lear regresso model ad ts assumptos. Uderstad the relato betwee fttg the smple lear regresso model by maxmum lkelhood ad the method-of-least squares. Uderstad the propertes of the parameter estmates, how we compute cofdece tervals for them ad how we test hypotheses about them. Uderstad how we assess model goodess-of-ft. Uderstad the Pythagorea formulato of regresso aalyss. The lear model s the most used tool moder statstcal aalyss. It also provdes a mportat coceptual framework for costructg more complex models. For example, euroscece these methods are at the heart of most fuctoal euromagg data aalyss paradgms. I the ext three lectures we wll preset the lear model by studyg smple regresso, multple regresso ad aalyss of varace (ANOVA). II. Smple Lear Regresso Model A. Motvato for the Smple Lear Regresso Model To motvate the smple lear regresso problem, we cosder the followg examples from the euroscece lterature. Fgure Relato betwee Coducto Velocty ad Axo Dameter. Replotted from Hursh (1939).

2 page : 9.07 Lecture 14: Smple Lear Regresso Example The Relato betwee Axo Coducto Velocty ad Axo Dameter. Hursh (1939) preseted data from a adult cat relatg euro coducto veloctes to the dameters of axos. Hursh reported 67 velocty ad dameter measuremet pars. He measured maxmal velocty amog fbers several erve budles ad the measured the dameter of the largest fber the budle. The data are replotted Fgure 9.1. Dameter s reported mcros ad velocty s reported meters per secod. We see that there s a strog postve relato that as the axo dameter creases, so does the coducto velocty. Possble questos we may wsh to aswer clude: How strog s the relato betwee coducto velocty ad axo dameter? Ca we quatfy how coducto velocty chages as a fucto of dameter? Fgure 14.. Tme-Seres Plot of frst 500 observatos of the MEG sesor backgroud ose measuremets. Example 3. (cotued). I ths example, we have utl ow oly cosdered the dstrbutoal propertes of the MEG measuremets. I so dog, we have treated these observatos as f they were depedet. If we plot the frst 500 observatos of the tme-seres (~ 833 msec), we see that the measuremets do ot appear to be depedet (Fgure 14.). Ideed, there seems to be almost a oscllatory patter the tme-seres. Furthermore, f we plot x t versus x t 1, we see that there s a strog lear relato betwee adjacet measuremets (Fg. 14.3). Whe x t 1 s large, x t s also large ad whe x t 1 s small, x t also teds to be small. I fact, the correlato coeffcet betwee x t ad x t 1, whch we defe below, s Is ths relato really there ad f so, could ths represet a systematc dstorto the local magetc feld?

3 page 3: 9.07 Lecture 14: Smple Lear Regresso Fgure Plot of MEG backgroud ose values x t versus x t 1. Courtesy of Scece. Used wth permsso. Fgure Plot of Chage Neural Actvty durg Learg Expermets versus Learg Tral (Wrth et al. 003). Example 14.. Defg Neural Correlates of Behavoral Learg. Wrth et al. (003) studed the relato betwee a mokey s performace learg a locato-scee assocato task ad chages eural actvty the amal s hppocampus (Fgure 14.4). They were terested the questo of how the tmg of chages the amal s eural actvty durg the task related to the tral the expermet whe the amal leared the task. Chage eural actvty was defed as a sgfcat crease eural actvty above basele. Learg was defed by performace better tha chace wth a hgh degree of certaty. The 45 degree le Fgure 14.4 s the le whch would mea that the eural chage ad the behavoral chage occurred

4 page 4: 9.07 Lecture 14: Smple Lear Regresso o the same tral. Is ths the most accurate descrpto of the relato betwee eural chage ad learg for these data? B. Smple Lear Regresso Model Assumptos To formulate a statstcal framework to study these problems, we assume we have data cosstg of pars of observatos that we deote as ( x1, y 1),...,( x, y ). For example, Example 14.1, the varable y s axo velocty ad the varable x s axo dameter. Let us assume that there s a lear relato betwee x ad y ad wrte t as y = α+ βx. (14.1) To make the relato Eq to a statstcal model, we wll make the followg assumptos: ) E[ y x ] = α+ βx for = 1,....,. ) The x 's are fxed o-radom measuremets termed covarates, regressors or carrers. ) The y 's are depedet Gaussa radom varables wth mea α βx ad varace σ +. Equato 14.1 ad these three assumptos are ofte summarzed as y = α+ βx + ε (14.) where the ε 's are depedet Gaussa radom varables wth mea 0 ad varace σ. Equato 14. defes a smple lear regresso model. It s smple because oly oe varable x s beg used to descrbe or predct y ad t s lear because the relato betwee y ad x s assumed to be lear. C. Model Parameter Estmato Our objectve s to estmate the parameters α, β ad σ. Because y s assumed to have a Gaussa dstrbuto codtoal o x, a logcal approach s to use maxmum lkelhood estmato. For these data the jot probablty desty (lkelhood) s L( αβσ,, y, x) = f ( y αβσ,,, x) ( + x,σ ) = f y α β (14.3) 1 1 ( y α βx ) = exp πσ σ where y = ( y,..,. y ) ad x = ( x,..., x ). The log lkelhood s ( y α βx ) logf ( y x, α, βσ, ) = lo g( πσ ). (14.4) σ

5 page 5: 9.07 Lecture 14: Smple Lear Regresso To compute the maxmum lkelhood estmates of, σ α β ad we dfferetate Eq wth respect to α ad β ad σ (, αβ α βx = (14.5) α σ log f y x,,σ ) ( y ) log f y x,,,σ ( x x = β σ ( αβ ) y α β ) (14.6) = + σ ( ) logf ( y,,, x α βσ ) 1 ( y α βx ) σ σ. (14.7) Settg the dervatves equal to zero Eqs ad 14.6 yelds the ormal equatos α + β x = y (14.8) I matrx form Eqs ad 14.9 become x = x y. (14.9) α x + β x y α =. (14.10) β x x x y Solvg for α ad β yelds 1 x y α =. (14.11) β x x x y The solutos for βˆ ad αˆ, whch are the maxmum lkelhood estmates of β ad α are respectvely

6 page 6: 9.07 Lecture 14: Smple Lear Regresso x y = x ( x x ) x xy ( x x )( y y ) ˆ β = (14.1) αˆ = y βˆx. (14.13) We may wrte the estmate of y as ˆ ˆ x = ˆ x + ˆ x y βˆ ( ) ŷ = α + β y β β = + x x, (14.14) for = 1,...,. If we retur to Eq we ca compute the maxmum lkelhood estmate of σ by frst substtutg βˆ ad αˆ for β ad α. Settg the left had sde of Eq to zero we obta the maxmum lkelhood estmate of σ 1 ˆ σˆ = (y αˆ β x ) (14.15) whch s what we would have predcted based o our aalyses Lecture 9. Remark Estmatg α ad β by maxmum lkelhood uder a assumpto of Gaussa errors s equvalet to the method of least-squares. I least-squares estmato, we fd the values of α ad β whch mmze the sum of the squared devatos of the y 's from the estmated regresso le. The method of least-squares s a form of method-of-momets estmato as we wll show below. Remark 14.. Other metrcs could be used to estmate α ad β, such as mmzg the sum of the absolute devatos. Ths would be defed as 1 y α β x. Remark The estmate αˆ ad Eq show that every regresso le goes through the pot ( x, y. ) Remark The resduals, y ŷ, are the compoets the data whch the model does ot expla. They are estmates of the ε s ad we ofte wrte ˆ ε = y ŷ. We ote that ad summg we have ŷ = y +βˆ (x x ), (14.16) ( y ŷ ) = ( y y ) βˆ(x x ) = 0. (14.17)

7 page 7: 9.07 Lecture 14: Smple Lear Regresso Remark There s a mportat Pythagorea relato betwee the sum of squared devato the data about ts mea, the sum of squared devatos of the regresso estmates about the mea of the data, ad the sum of squared resduals. We derve t ow. By Remark 14.4, the th resdual s Squarg ad summg both sdes of Eq , we obta y y ˆ = ( y y ) ( yˆ y). (14.18) (y ŷ ) = {( y y ) ( ŷ y )} (14.19) = ( y y ) + ( ŷ y ) ( y y )( ŷ y ). (14.0) Now f we aalyze the last term Eq usg Eq we fd that ( y y ) βˆ (x x ) = βˆ ( y y )(x x ) (14.1) = βˆ (x x ) ad Eq becomes = ( ŷ y ), ( y ŷ ) = ( y y ) ( ŷ y ), (14.) whch yelds the desred Pythagorea Relato It states that ( y y ) = ( ŷ y ) + ( y ŷ ). (14.3) Total sum of = Explaed sum + Resdual sum of squares of squares squares (TSS) (ESS) (RSS) We wll make extesve use of ths relato our aalyses of our regresso fts.

8 page 8: 9.07 Lecture 14: Smple Lear Regresso D. The Relato betwee Correlato ad Regresso Let us defe the sample correlato coeffcet betwee x ad y as ( x x )( y y ) r xy =. (14.4) ( x x ) ( y y ) 1 It s the method-of-momets estmate of the theoretcal correlato coeffcet betwee x ad y whch s defed as Cov ( X, Y ) ρ xy =, (14.5) [ Var ( X ) Var( Y )] 1 where Cov ( X, Y ) = E [( X E( X ))(Y E( Y ))] ad where the expectato s take wth respect to f( X, Y ), the jot dstrbuto of X ad Y. If X ad Y have a bvarate Gaussa dstrbuto, the t s easy to show that r xy s also the maxmum lkelhood estmate of ρ xy. It s also straght forward to show that we always have 1< r xy <1. Recall we showed Lecture 5 1< ρ xy <1. As we suggested Lecture 4, whe we use the term correlato we wll be usg t the techcal sese defed by ether Eqs or To work out the relato betwee the sample correlato coeffcet ad the slope parameter estmate, we recall that the slope parameter estmate (Eq. 14.1) ca be expressed as Hece, ˆ ( x x )( y y ) β =. (14.6) ( x x ) ( y y )( x x ) σˆ ˆ xy β = = σˆ ( x x ) x ( y y ) σˆ y = r xy = r xy σˆ x ( x x ) 1 (14.7)

9 page 9: 9.07 Lecture 14: Smple Lear Regresso where σˆxy = 1 ( y y )(x x ). We see that the regresso coeffcet s a scaled verso of the sample correlato coeffcet or smply the sample covarace of x ad y dvded by the sample varace of x. I ths way, we see that the least-squares estmates are method-ofmomets estmates. E. Dstrbutos of the Parameter Estmates The parameter estmates αˆ ad βˆ are fuctos of the data. Therefore, they are radom varables ad have dstrbutos. It s straght forward to show that these dstrbutos are Gaussa. Therefore, t suffces to specfy ther meas ad varaces because ths s the mmal descrpto eeded to defe Gaussa radom varables. It s straght forward to show that E( βˆ ) = β ad E( αˆ ) = α. The varaces of the regresso parameter estmates are σ Var ( βˆ ) = (14.8) ( x x ) 1 x Var ( αˆ ) = σ +. (14.9) ( x x ) 1 If we estmate σ by ŝ = ( ) σˆ (Eq ) the followg the logc we used Lecture 8, the 100%(1 δ ) cofdece tervals for the parameters based o the t dstrbuto wth degrees of freedom are t,1 δ / sˆ βˆ ± (14.30) 1 ( x x ) 1 1 x αˆ ± t,1 δ / + ŝ (14.31) ( x x ) There are degrees of freedom for ths t-dstrbuto because we estmated two parameters ths aalyss. Remark We ca also vert the above cofdece tervals as we dscussed Lecture 1 to test hypotheses about the regresso coeffcets by costructg a t test.

10 page 10: 9.07 Lecture 14: Smple Lear Regresso Remark The varace of βˆ decreases as ( x x ) creases. Hece, f we ca desg a expermet whch we ca choose the value of the x k 's, we wll spread them out as far as possble across the relevat rage to decrease the varace of the estmated slope. To costruct a cofdece terval for a predcted y k value at a gve value, x k, we ote that the ( x k x ) σ Var ( y ) = + k σ (x x ) (14.3) ad hece, the 100%(1 δ ) cofdece terval s defed by 1 1 (x k x ) ŷ k ± t,1 δ / + ŝ. (14.33) (x x ) Example (cotued). We ft the model Eq to the axo dameter ad velocty data Fg The estmated regresso le, alog wth the local 95% cofdece tervals (Eq ), s show Fg The estmated regresso le s cosstet wth the strog lear relato see the orgal data Fg Most of the data le close to the estmated regresso le. Parameter Estmate Stadard Error t-statstc αˆ = βˆ = Table Parameter Estmate Summary The parameter estmates ad ther correspodg stadard errors ad t statstcs are show Table We reject the ull hypothess that there s o lear relato sce the t-statstc for βˆ s 43.5 ad has a p value<<0.05. As there are 67 observatos ths t statstc s essetally a z statstc whch has a value of 43.5!! Hece, ths result s hghly sgfcat. More mportatly, the 95% cofdece terval for βˆ s approxmately [5.79, 6.35]. The estmate of βˆ has uts of meters per secod per mcro. Ths meas that for every oe mcro crease dameter, there s approxmately a 6.07 meter per secod crease velocty. We also reject the ull hypothess that there s a zero tercept for the regresso le sce the t statstc for αˆ s -.61 ad has a p-value<0.05. More mportatly, the 95% cofdece terval for α s approxmately [-5.1, -.33]. We coclude that the tercept s most lkely egatve gve the lmts of the cofdece tervals. The tercept has uts of meters per secod. If ths were a physcal model, t would mea that at a zero dameter the velocty would be egatve. Ths does

11 page 11: 9.07 Lecture 14: Smple Lear Regresso ot make physcal sese ad shows the mportace of ot usg a regresso model beyod the rage of the data where t s estmated. The smallest dameter our sample s approxmately mcros ad the largest s approxmately 0 mcros. Fgure Ft ad Cofdece Itervals for a Smple Lear Regresso Model of the Axo Coducto Velocty as a fucto of Axo Dameter. F. Model Goodess-of-Ft A crucal f ot the most crucal step a statstcal aalyss s measurg goodess-of ft. That s, how well does the model agree wth the data. We prevously used Q-Q plots to assess the degree to whch elemetary probablty destes descrbed sets of data. We dscuss two statstcal measures ad three graphcal measures of goodess-of-ft. 1. F-test Gve the ull hypothess H 0 : β = 0,.e. that there s o lear relato x ad y, we ca test ths hypothess explctly usg a F test. The F statstc wth p 1 ad p degrees of freedom s 1 ( p 1) ( y y ) 1 ( p 1) ESS F p 1, p 1 ( p ) RSS 1 ( p ) ( y y ˆ ) = =, (14.34) where ESS s the explaed sum of squares, RSS s the resdual sum, p s the umber of parameters the model ad s the umber of data pots. We reject ths ull hypothess for large values of the F statstc. Ths suggests that f the amout of the varace the data that the regresso explas s large relatve to the amout whch s uexplaed, the we reject the ull hypothess of o lear relato.

12 page 1: 9.07 Lecture 14: Smple Lear Regresso. R-Squared Aother measure of goodess-of-ft s the Square of the Multple Correlato Coeffcet or R. It s defed as N ( ŷ y ) ESS = R = = 1. (14.35) N TSS ( y y ) The R measures the fracto of varace the data explaed by the regresso equato. By the Pythagorea relato Eq. 14.3, we see that 0 < R 1. For the smple lear regresso model R = r R xy. The ad the F statstc are related as F( p 1 )/( p ) R = (14.36) 1+ F( p 1 )/( p) As the F statstc creases, the R F statstc, the greater the R. creases. Hece, the greater the magtude of the Example 14.1 (cotued). We have from the Pythagorea relato Eq ESS 65,041 RSS,33 TSS 67,74 ad that F = 1,895 ad R 1,65 = Therefore, we reject the ull hypothess of o lear relato the data wth a p-value << Furthermore, the R suggests that the regressor, axo dameter ca expla 97% of the varace the axo coducto velocty. Remark 8. The square of the t statstc wth p degrees of freedom s the F statstc wth 1 ad p degrees of freedom. To verfy ths we fd Table 1 that the ( t statstc) = (43.5) = 1,893 whch s the value of F 1, Graphcal Measures of Goodess-of-Ft a. Plot the raw data. Is the relato lear? (Fg. 14.1). b. Plot the resduals versus the covarate x. Is there lack of ft of the model? (Fg. 14.6) c. Plot the resduals versus the predcted values ŷ. Is there a relato? (Fg. 14.7)

13 page 13: 9.07 Lecture 14: Smple Lear Regresso Fgure Plot of Resduals versus Axo Dameter for the Smple Lear Regresso Model of the Axo Dameter ad Coducto Velocty. The resduals have o real dscerble structure except that they seem to grow wth the dameter of the axo suggestg that the assumpto of homoscedastcty,.e., that all of the observatos have the same varace may be correct. Whe observatos have dfferet varaces they are sad to be heteroscedastc. We mght model ths by makg the varace proportoal to the dameter. Fgure Plot of the Resduals agast the Predcted Velocty. The plot of the resduals versus the predcted velocty also shows the heteroscedastcty the observatos. Ths plot s performed because by the Pythagorea relato, the resduals ad the predcted velocty estmates are orthogoal. Hece, f there s ay relato betwee the two t suggests a way whch the model may be msspecfed. Iferece. Based o ths regresso aalyss axo coducto velocty s strogly assocated wth (predcted by) axo dameter. Ths aalyss also reveals that a oe mcro crease

14 page 14: 9.07 Lecture 14: Smple Lear Regresso dameter s assocated wth a 6 meter per secod chage velocty for axo dameters betwee ad 0 mcros. Remark Correlato does ot mea causato. G. The Geometry of Regresso Aalyss (Method of Least-Squares) As we have show, regresso aalyss has a tutvely appealg geometrc terpretato defed prmarly be Eq I partcular, we ca uderstad ths geometry by cosderg two cases: ) Small Resdual Error If the resdual error s small t follows from the Pythogorea relato that the geometry of the aalyss must have a small RSS relatve to ESS. ) Large Resdual Error Smlarly, f the resdual error s large, t follows from the Pythogorea relato that the geometry of the aalyss must have a large RSS relatve to ESS. III. Summary The smple lear regresso model ca be used to study how well oe varable may be descrbed as a lear fucto of aother varable. The Pythagorea relato s a useful costruct for uderstadg how the aalyss framework for the smple lear regresso model s costructed. Ths Pythagorea relato wll also be cetral to our aalyses of multple lear regresso ad ANOVA models. Ackowledgmets I am grateful to Ur Ede for makg the fgures, to Jule Scott for techcal assstace ad to Jm Mutch for careful proofreadg ad commets. The data from the Hursh (1993) were provded by Rob Kass at Carege Mello. Text Refereces DeGroot MH, Schervsh MJ. Probablty ad Statstcs, 3rd edto. Bosto, MA: Addso Wesley, 00. Draper NR, Smth H. Appled Regresso Aalyss, d ed. New York: Wley, Rce JA. Mathematcal Statstcs ad Data Aalyss, 3 rd edto. Bosto, MA, 007. Lterature Refereces Hursh JB. The propertes of growg erve fbers. Amerca Joural of Physology, 1939, 17: Wrth S, Yake M. Frak LM, Smth AC, Brow EN, Suzuk WA. Sgle euros the mokey hppocampus ad learg of ew assocatos. Scece, 003, 300:

15 MIT OpeCourseWare Statstcs for Bra ad Cogtve Scece Fall 016 For formato about ctg these materals or our Terms of Use, vst:

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The