F. Tibaldi Limburgs Universitair Centrum, Hasselt University, Belgium

Size: px

Start display at page:

Download "F. Tibaldi Limburgs Universitair Centrum, Hasselt University, Belgium"

Stephanie Holmes
6 years ago
Views:

1 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald LINEAR REGRESSION MODELS F Tbald Lmburgs Uverstar Cetrum, Hasselt Uversty, Belgum Keywords: Smple lear regresso, multple regresso, estmato, ferece, dagostcs Cotets Itroducto Smple Lear Regresso model The Model Estmato 3 Iferece 3 Ifereces about the Regresso Coeffcets 3 Dagostcs ad Remedal Measures 4 Multple Lear Regresso Model 4 Estmato of Regresso Coeffcets 4 Ifereces About Regresso Coeffcets 5 Model Adequacy ad Dagostcs 6 Commets o Iterpretg Regresso Aalyss Glossary Bblography Bographcal Sketch Summary The basc lear regresso model s troduced, followed by estmato ad feretal methods Dagostc ad remedal measures are dscussed The, the method ad ts feretal procedures s geeralzed to the multple lear regresso settg, e, the cotext where the respose varable s explaed terms of several rather tha oe explaatory varable Itroducto Uderstadg relatoshps amog sets of varables s a basc problem statstcal scece I the late eteeth cetury, Sr Fracs Galto made a fudametal cotrbuto to uderstadg multvarate relatoshps by troducg regresso aalyss I oe dataset, descrbed hs 885 presdetal address before the Athropologcal Secto of the Brtsh Assocato of the Advacemet of Sceces, Galto lked the dstrbuto of chldre's heghts to ther parets' Galto showed ot oly that each dstrbuto was approxmately ormal but also that the jot dstrbuto could be descrbed as a bvarate ormal Thus, the codtoal dstrbuto of adult chldre's heght, could also be descrbed by usg a ormal dstrbuto As a byproduct of hs aalyss, Galto observed that tall parets ted to have tall chldre although ot as tall as the parets (ad vce versa for short chldre) From ths, he correctly ferred that chldre would regress to medocrty subsequet Ecyclopeda of Lfe Support Systems (EOLSS)

2 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald geeratos, hece suggestg the term that has become kow as regresso aalyss Several authors have gve sghtful ad etertag accouts of the termology use of Galto as well as of other cotrbutors to statstcal scece Regresso aalyss has developed to the most wdely appled statstcal methodology It s a mportat compoet of multvarate aalyss because t allows researchers to focus o the effects of explaatory varables To llustrate, the Galto dataset of famly heghts, regresso allows the aalyst to descrbe the effect of parets' heght o a chld's adult heght Smple Lear Regresso model Regresso ts smplest form, s a techque for modelg a relatoshp betwee two varables Ths, of course, ca be exteded to multple varables The Model The smple lear regresso model ca be stated as follows Y β + β x + ε,,, () where Y s the respose (depedet varable) for the th tral (subject, sample, ); x s the value of the depedet varable (predctor, regressor, ) the th tral; the error term ε represets the resduals, assumed to be depedet radom varables havg a ormal dstrbuto wth mea zero ad costat varace σ I other words, o E( ε ), o Var( ε ) σ (homoscedastc errors), o Cov( ε, ε j) wth j (ucorrelated errors), the ukow parameters β ad β, also called the regresso coeffcets, eed to be estmated I Secto we wll outle a method to obta estmates ˆ β ad ˆβ for β ad β The smple lear regresso model () s called a statstcal model ad eeds to be dstgushed from a so-called determstc model The law of gravty physcs, for example, s a determstc model that assumes a deal settg where the respose varable vares a completely prescrbed way accordg to a perfect mathematcal fucto of the depedet varables Statstcal models allow for the possblty of error (varablty) descrbg a relatoshp We also eed to dstgush betwee observatoal data ad expermetal data The frst type of data s obtaed wthout cotrollg the depedet varable A major lmtato of ths kd of data s that they ofte do ot provde adequate formato about causal relatoshps Oe always should vestgate whether other depedet varables mght expla causal relatoshps more drectly Whe cotrol s exercsed over the depedet varable, Ecyclopeda of Lfe Support Systems (EOLSS)

3 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald the resultg expermetal data provde much stroger formato about causal relatoshps I a completely radomzed desg, treatmets are assged to each of the expermetal uts completely at radom Radomzato teds to balace out the effects of ay other varable that mght affect the respose Estmato The regresso coeffcets, β ad β, are tradtoally estmated usg least squares Such estmators, usually deoted by ˆ β ad ˆβ, are defed as the mmzer of ( β, β) ( β β ) Q Y x () Dfferetatg Q( β, β ) wth respect to β ad β ad settg these partal dervatves equal to zero leads to the ormal equatos that, oce solved, yeld: ˆ β Y ˆ β x ad ( Y Y )( x x) ( x x) ˆ β (3) We ca use the estmators ˆ β ad ˆ β to estmate the regresso fucto E( Y) β + βx by Yˆ ˆ β ˆ + βx We call Y ˆ ˆ ˆ β + βx the th ftted value ad e ˆ Y Y the th resdual Resduals play a very mportat role studyg whether a gve regresso model s approprate for the data at had Next, we propose a estmator for the varace parameter σ Recall that, based o a sample of depedet ormally dstrbuted radom varables Z,, Z, S ( Z Z) /( ) s a ubased estmator for σ Now, the regresso model (), each Y has ts ow mea β + βx, whch ca be estmated by the ftted value Y ˆ Hece, the devato from the mea s ow represeted by the resdual e ad the approprate sum of squares SSE e wth degrees of freedom s used to obta a ubased estmator of σ by MSE SSE /( ) 3 Iferece To set up terval estmates ad test procedures, we eed to specfy the error dstrbuto I the ormal error regresso model we exted () wth the assumpto that ε are depedet zero-mea ormally dstrbuted wth varace σ Therefore, the ormal regresso model ca be formulated as Y are depedet N( β + β x, σ ) Ecyclopeda of Lfe Support Systems (EOLSS)

4 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald I the ext secto we wll expla how fereces ca be made uder the assumpto of the lear regresso model To do so, we wll use the followg results If SSE s the sum of squares defed Secto, the SSE / σ ~ χ( ) SSE ad ( ˆ β ˆ, β) are depedet ˆ ˆ β ~ N ( β, σ ( β )) where ˆ x σ ( β) σ + ( x ) x ˆ β ~ N β, σ ( β) where ˆ σ σ ( β) ( x x) ( ) The varaces ˆ σ ( β ) ad σ ( ˆ β ) cota the ukow parameter have to be estmated Usg the fact that ˆ σ MSE we have ˆ ( ˆ β) MSE + ( x ) x σ The, j j ˆ ( ˆ β j ) ˆ β β ~ t ( ), j, σ x 3 Ifereces about the Regresso Coeffcets ad σ ad therefore ˆ ( β) ( x ) x σ ˆ MSE Maly hypothess tests regardg β are of mportace, wth partcular emphass o H : β versus H : β Ideed, β dcates that there s o lear assocato betwee the respose Y ad the depedet varable X If β the lear model smplfes to Y beg depedet N( β, σ ), whch mples ot oly that there s o lear assocato betwee respose ad depedet varable but also that there s ot relato of ay type betwee them I cotrast there are oly frequet occasos whe we wsh to make fereces cocerg β Usg the dstrbutoal results we ca the costruct ( α ) % cofdece tervals for β ad β the followg way: ˆ β ± t( α/ ; ) ˆ σ( ˆ β ) ad Ecyclopeda of Lfe Support Systems (EOLSS)

5 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald ˆ β ± t( α/ ; ) ˆ σ( ˆ β ) Test cocerg the parameters of ths model ca be set up a stadard fasho usg the t dstrbuto The decso rule for the two-sded alteratve H β s ˆ β f t( α/ ; ) do ot reject H : β, ˆ( σ ˆ β) : ˆ β f > t( α/ ; ) reject H : β, ˆ( σ ˆ β) Ths rule ca also be used for the oe-sded alteratve If the respose dstrbuto s ot exactly ormal but does ot serously depart from ormal, the dstrbutos of ˆ β ad ˆ β wll stll be approxmately ormal If the respose dstrbuto s far from ormal, oe ca use the asymptotc ormalty of ˆ β ad ˆβ : ther dstrbutos approach ormalty as the sample sze creases Thus, wth suffcetly large samples, the cofdece tervals ad decso rules gve earler stll apply wth t -percetles replaced by ormal percetles Bblography TO ACCESS ALL THE PAGES OF THIS CHAPTER, Vst: Berry, WD ad Feldma, S (99) Multple Regresso I Practce Quattatve Applcatos Socal Sceces Beverly Hlls: Sage Uversty Paper [Basc text o regresso] Draper, N (998) Appled Regresso Aalyss New York: Joh Wley [Stadard text o regresso] Fox, J (997) Appled Regresso Aalyss, Lear Models, ad Regresso Methods Thousad Oaks, Calfora: Sage Publcatos [Referece text o regresso] Galto, F (885) Regresso towards medocrty heredty stature Joural of Athropologcal Isttute, 5, [Hstorc perspectve o regresso] Mosteller, F ad Tukey, JW (977) Data Aalyss ad Regresso Readg, MA: Addso-Wesley [Perspectve o regresso] Neter, J, Kuter, M, ad Nachtshem, C (996) Appled Lear Statstcal Models (4th ed) Irw: Chcago [Classcal troductory text to the subject] Ecyclopeda of Lfe Support Systems (EOLSS)

6 BIOMETRICS - Vol I - Lear Regresso Models - F Tbald Roald, C (998) Aalyss of Varace, Desg ad Regresso: Appled Statstcal Methods Boca Rato: Chapma ad Hall [Stadard lear models text] Sokal, RR ad Rohlf, FJ (993) Bometry: The Prcples ad Practce of Statstcs Bologcal Research (3rd ed) New York: Freema [Partcular focus o bology] Stgler, SM (986) The Hstory of Statstcs: The Measuremet of Ucertaty Before 9 Cambrdge: Harvard Uversty Press [Etertag perspectve o the hstory of statstcs] Wesberg, S (98) Appled Lear Regresso New York: Joh Wley [Regresso from a appled pot of vew] Bographcal Sketch Fabá S Tbald holds a BS degree mathematcs from the Uversty of Bueos Ares (Argeta) ad Master of Scece ad PhD degrees Bostatstcs from the Lmburgs Uverstar Cetrum (Belgum) Hs research terests focus o methods for correlated ad herarchcal ormally dstrbuted ad survval data, wth applcatos the evaluato of surrogate edpots cotrolled clcal trals ad populato geetcs He has extesve experece survey methodology ad worked offcal statstcs, both Argeta ad Belgum Ecyclopeda of Lfe Support Systems (EOLSS)

Ordinary Least Squares Regression. Simple Regression. Algebra and Assumptions.

Ordinary Least Squares Regression. Simple Regression. Algebra and Assumptions. Ordary Least Squares egresso. Smple egresso. Algebra ad Assumptos. I ths part of the course we are gog to study a techque for aalysg the lear relatoshp betwee two varables Y ad X. We have pars of observatos