Simple Linear Regression and Correlation.

Size: px

Start display at page:

Download "Simple Linear Regression and Correlation."

Duane Horton
6 years ago
Views:

1 Smple Lear Regresso ad Correlato. Correspods to Chapter 0 Tamhae ad Dulop Sldes prepared b Elzabeth Newto (MIT) wth some sldes b Jacquele Telford (Johs Hopks Uverst)

2 Smple lear regresso aalss estmates the relatoshp betwee two varables. Oe of the varables s regarded as a respose or outcome varable (). The other varable s regarded as predctor or eplaator varable (). Sometmes t s ot clear whch of two varable should be the respose (e.g. heght ad weght). I ths case, correlato aalss ma be used. Smple lear regresso estmates relatoshps of the form a + b.

3 Scatter plot of ozoe cocetrato b temperature ar$ozoe ar$temperature Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 3

4 A Probablstc Model for Smple Lear Regresso Let,,..., be specfc settgs of the predctor varable. Let,,..., be the correspodg values of the respose varable. Assume that s the observed value of a radom varable (r.v.) Y, whch depeds X o accordg to the followg model: Y β 0 + β + ε (,,, ) Here ε s the radom error wth E(ε )0 ad Var(ε )σ. Thus, E(Y ) µ β 0 + β (true regresso le). The s usuall are assumed to be fed (ot radom varables). 4

5 A Probablstc Model for Smple Lear Regresso See Fgure 0., p. 348 ad also see page 348 for the four assumptos of a smple lear regresso model. 5

6 Least Square Le Mathematcs (veted b Gauss) Fd the le,.e., values of β 0 ad β that mmzes the sum of the squared devatos: Q How? [ ( β0 + β )] Solve for values of β 0 ad β for whch Q β 0 0 ad Q β 0 6

7 Fdg Regresso Coeffcets )] ( [ )] ( [ Q Q β β β β β β + + 7

8 Normal Equatos β β β β 8

9 Soluto to Normal Equatos ˆ ˆ S S ) ( ) )( ( ˆ 0 β β β )., ( Note that least squares le goes through 9

10 Ftted regresso le ar$ozoe ar$temperature Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 0

11 Ftted values of : ˆ ˆ β ˆ 0 + β,,,..., Resduals : e ˆ ˆ ˆ ( β0 + β ),,,..., temperature ozoe ftted resd Ths code was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato.

12 Matr Approach to Smple Lear Regresso (what our regresso package s reall dog) The model: Xβ + ε s b X s b β s b ε s b

13 YXβ + ε ε ε ε ε β β 3

14 Soluto of lear equatos I lear algebra: Fd whch solves Ab. I regresso aalss: Fd β whch solves Xβ Wh ca t we do ths? 4

15 Least Squares Q(-Xβ) (-Xβ) β X Xβ + β X Xβ β X + β X Xβ Q/ β -X + X Xβ Q/ β 0 X X Xb, where b βˆ 5

16 Least Squares cotued For smple lear regresso: X X X' ' 6

17 Least Squares cotued X Xb X b The Normal Equatos as before 7

18 Least Squares cotued X Xb X b (X X) - X (f X has learl depedet colums) Soluto b QR decomposto XQR, Q orthoormal, R upper tragular ad vertble b(x X) - X (R Q QR) - R Q (R R) - R Q R - Q 8

19 The Hat Matr b(x X) - X ŷxb X(X X) - X H H ( b ) s the Hat matr Takes to ŷ H s smmetrc ad dempotet HHH Dagoal elemets of the hat matr are useful detectg fluetal observatos. 9

20 Epected value of b E(b) E((X X) - X ] E[(X X) - X (Xβ+ε)] E[(X X) - X X β+ (X X) - X ε] β Hece b s a ubased estmator of β. 0

21 Covarace of b The covarace matr of s σ I b(x X) - X A (where A s k b ) Cov(b) A Var() A A σ I A σ AA σ (X X) - X X(X X) - σ (X X) -

22 Covarace of b For smple lear regresso, σ (X X) - - ) ( σ σ S S b SD ) SD(b ; ) ( 0 σ σ

23 Estmato of σ s e ( ˆ ) Note: The deomator s - sce two parameters are beg estmated (β 0 ad β ). E[S ]σ (See proof Seber, Lear Regresso Aalss) 3

24 Statstcal Iferece for βo ad β SE ( ˆ β 0 ) s ad SE( ˆ β) S s S For ozoe eample: Coeffcets: Value Std. Error t value Pr(> t ) (Itercept) temperature Ths code was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 4

25 Sums of Squares Sum of Squares Total (SST) : ( ) Sum of Squares for Error (SSE) : e ( ˆ ) Sum of Squares for Regresso (SSR) : ( ˆ ) 5

26 Geometr of the Sums of Squares ( ˆ ) + ( ˆ ) SST SSR + SSE, see dervato o p. 354 J. Telford 6

27 Coeffcet of Determato (R-squared) r SSR SST SSE SST proporto of the varace that s accouted for b the regresso o square of correlato betwee ad ŷ For ozoe eample: Multple R-Squared:

28 Aalss of Varace (ANOVA) H : β 0 vs. H : β F SSR/ MSR SSE/( - ) MSE t For ozoe eample: summar.aov(tmp) Df Sum of Sq Mea Sq F Value Pr(F) temperature Resduals Ths code was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 8

29 Regresso Dagostcs Resdual vs. observato umber resd(ozoe.lm) Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 9

30 Regresso Dagostcs resdual vs. ftted value resd(ozoe.lm) ftted(ozoe.lm) Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 30

31 Regesso Dagostcs resdual vs. resd(ozoe.lm) ar$temperature Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 3

32 Regresso Dagostcs qq plot of resduals resd(ozoe.lm) Quatles of Stadard Normal Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 3

33 Hat Matr Dagoals hat(model.matr(ozoe.lm)) Ths graph was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 33

34 Some useful S-Plus commads m.lm <- lm(~, datamdata, a.actoa.omt) cludes tercept term b default summar(m.lm) gves coeffcets, correlato of coeffcets, R-square, F- statstc, resdual stadard error summar.aov(m.lm) gves ANOVA table resd(m.lm) gves resduals ftted(m.lm) gves ftted values model.matr(m.lm) gves model matr Ths code was created usg S-PLUS(R) Software. S-PLUS(R) s a regstered trademark of Isghtful Corporato. 34

12.2 Estimating Model parameters Assumptions: ox and y are related according to the simple linear regression model

12.2 Estimating Model parameters Assumptions: ox and y are related according to the simple linear regression model 1. Estmatg Model parameters Assumptos: ox ad y are related accordg to the smple lear regresso model (The lear regresso model s the model that says that x ad y are related a lear fasho, but the observed