ENGI 3423 Simple Linear Regression Page 12-01

Size: px

Start display at page:

Download "ENGI 3423 Simple Linear Regression Page 12-01"

Ursula Brooks
5 years ago
Views:

1 ENGI 343 mple Lear Regresso Page - mple Lear Regresso ometmes a expermet s set up where the expermeter has cotrol over the values of oe or more varables X ad measures the resultg values of aother varable Y, producg a feld of observatos. The questo the arses: What s the best le (or curve to draw through ths feld of pots? Values of X are cotrolled by the expermeter, so the o-radom varable x s called the cotrolled varable or the depedet varable or the regressor. Values of Y are radom, but are flueced by the value of x. Thus Y s called the depedet varable or the respose varable. We wat a le of best ft so that, gve a value of x, we ca predct the value of Y for that value of x. The smple lear regresso model s that the predcted value of y s y β + β x ad that the observed value of Y s Y + β x β + ε where ε s the error. It s assumed that the errors are ormally dstrbuted as ε ~ N(, σ, wth a costat varace σ. The pot estmates of the errors ε are the resduals e y yˆ. Wth the assumptos Y β + β x + ε x x Y ~ N( β + β x, σ [ (3 V[Y] s d t of x ] place, t the follows that ˆβ ad are ubased estmators of the coeffcets β ad β. ( ote lower case E ˆ β ˆ βx + β + βx x Methods for dealg wth o-lear regresso are avalable the course text, but are beyod the scope of ths course.

2 ENGI 343 mple Lear Regresso Page - Examples llustratg volatos of the assumptos the smple lear regresso model:.. ( volated ot lear (3 volated varace ot costat OK ( & (3 volated If the assumptos are true, the the probablty dstrbuto of Y x s N( β + β x, σ.

3 ENGI 343 mple Lear Regresso Page -3 Example. Gve that Y.5 x + ε, where ε ~ N(,, fd the probablty that the observed value of y at x 8 wll exceed the observed value of y at x 7. Y ~ N(.5 x, Let Y 7 the observed value of y at x 7 ad Y 8 the observed value of y at x 8, the Y 7 ~ N(6.5, ad Y 8 ~ N(6, Y 8 Y 7 ~ N(6 6.5, + μ.5 σ 4 (.5 P[ Y8 Y7 > ] P Z > Φ(.5.43 Despte β <, P[Y 8 > Y 7 ] > 4%! For ay x the rage of the regresso model, more tha 95% of all Y wll le wth σ ( ether sde of the regresso le.

4 ENGI 343 mple Lear Regresso Page -4 Dervato of the coeffcets ˆβ ad ˆβ of the regresso le y ˆ β ˆ + β x : We eed to mmze the errors. Each error s estmated by the observed resdual e y yˆ. Mmze errors.? NO e Use the E (sum of squares due to errors e ( y ˆ β ˆ β x f ( ˆ β, ˆ β o Fd ˆβ ad ˆβ such that. ˆ β ˆ β [Note: ˆ β ˆ, β are varables, whle x, y are costats.] ˆ β ( ˆ ˆ ( ˆ ˆ β β β β ( y x + x y ad ( y ˆ ˆ β β ( ˆ ˆ ˆ x x β x + β x x y ( β x ˆ β or, equvaletly, x x ˆ β ˆ β x y ˆ β x x y x y x x y x y x (3 (4

5 ENGI 343 mple Lear Regresso Page -5 The soluto to the lear system of two ormal equatos ( ad ( s: from the lower row of matrx equato (4: ˆβ x y, (where x y x x y x x y ad x ( x x x ( or, equvaletly, ˆ sample covarace of x, y β ; sample varace of x ad, from equato (: ˆ β ( y ˆ β x. A form that s less susceptble to roud-off errors (but less coveet for maual computatos s ( x x( y y ( x x ˆβ ad ˆ β ˆ β x. y The regresso le of Y o x s y y ( x x ˆβ. Equato ( guaratees that all smple lear regresso les pass through the cetrod ( x, y of the data. It turs out that the smple lear regresso method remas vald eve f the values of the regressor x are also radom. However, ote that terchagg x wth y, (so that Y s the regressor ad X s the respose, results a dfferet regresso le (uless X ad Y are perfectly correlated..

ENGI 343 mple Lear Regresso Page -6 Example. (the same data set as Example.6: pared two sample t test Ne voluteers are tested before ad after a trag programme.

6 ENGI 343 mple Lear Regresso Page -6 Example. (the same data set as Example.6: pared two sample t test Ne voluteers are tested before ad after a trag programme. Fd the le of best ft for the posteror (after trag scores as a fucto of the pror (before trag scores. Voluteer: After trag: Before trag: Let Y score after trag ad X score before trag. I order to use the smple lear regresso model, the assumptos Y β + β x + ε x x Y ~ N( β + β x, σ must hold. From a plot of the data ( ad oe ca see that the assumptos are reasoable. catter plot Normal probablty plot of resduals

7 ENGI 343 mple Lear Regresso Page -7 Calculatos: x y x x. y y um: x y x y x y [Note: ( * sample covarace of (X, Y ] x ( x x x [Note: ( * sample varace of X ] x y 776 ˆ β ˆ x x y ˆ ad β ( β ( x Each predcted value ŷ of Y s the estmated usg yˆ ˆ β + ˆ β x x ad the pot estmates of the ukow errors ε are the observed resduals e y yˆ. [Use u-rouded values ad to fd resduals.] A measure of the degree to whch the regresso le fals to expla the varato Y s the sum of squares due to error, E e ( y ˆ β ˆ β x whch s gve the adjog table. x y ŷ e e E 3.84

8 ENGI 343 mple Lear Regresso Page -8 A Alteratve Formula for E: ˆ β y βˆ x E y ( y ˆ β x ˆ x y y ˆ β β x x ( y y ˆ ( x x( y y + ˆ β β ( x x yy ˆ β ˆ + β But ˆ β ( ( ( E yy βˆ or E E yy or ( ( ( yy ( I ths example, E However, ths formula s very sestve to roud-off errors: If all terms are rouded off prematurely to three sgfcat fgures, the E 5.85 ( d.p. 9 4 ( y ˆ T ( y y E e y

9 ENGI 343 mple Lear Regresso Page -9 The total varato Y s the T (sum of squares - total: yy T ( y y (whch s ( the sample varace of y. I ths example, T / The total varato (T ca be parttoed to the varato that ca be explaed by the regresso le ( R ( yˆ y ad the varato that remas uexplaed by the regresso le (E. T R + E ˆ β yy The proporto of the varato Y that s explaed by the regresso le s kow as the coeffcet of determato r R T E T I ths example, r ( / Therefore the regresso model ths example explas 99.% of the total varato y. Note: R ˆ β ad T yy r x x x y y y The coeffcet of determato s just the square of the sample correlato coeffcet r. Thus r r.996. It s o surprse that the two sets of test scores ths example are very strogly correlated. Most of the pots o the graph are very close to the regresso le y.87 x +.34.

10 ENGI 343 mple Lear Regresso Page - A pot estmate of the ukow populato varace σ of the errors ε s the sample varace or mea square error s ME E / (umber of degrees of freedom. But the calculato of s cludes two parameters that are estmated from the data: ˆβ E ad ˆβ. Therefore two degrees of freedom are lost ad ME. I ths example, ME.973. A cocse method of dsplayg some of ths formato s the ANOVA table (used Chapters ad of Devore for aalyss of varace. The f value the top rght corer of the table s the square of a t value that ca be used a hypothess test o the value of the slope coeffcet β. equece of maual calculatos: {, x, y, x,, y } {,, yy } ˆ β ˆ β, R, T } { R, E } { MR, ME } f t {, ource Degrees of ums of quares Mea quares f Freedom Regresso R MR R / MR/ME Error E ME E / ( Total T To test H o : β (o useful lear assocato agast H a : β (a useful lear assocato exsts, we compare t f to t α/, (.

11 ENGI 343 mple Lear Regresso Page - I ths example, t >> t.5, 7 (the p-value s < 7 so we reject H o favour of H a at ay reasoable level of sgfcace α. ˆ β The stadard error s b of ˆβ s s / so the t value s also equal to. ME x x Yet aother alteratve test of the sgfcace of the lear assocato s a hypothess test o the populato correlato coeffcet ρ, (H o : ρ vs. H a : ρ, usg the test statstc above. Example.3 t r r, whch s etrely equvalet to the other two t statstcs (a Fd the le of best ft to the data x 3 4 y (b Estmate the value of y whe x. (c Why ca t the regresso le be used to estmate y whe x? (d Fd the sample correlato coeffcet. (e Does a useful lear relatoshp betwee Y ad x exst? (a A plot of these data follows. The Excel spreadsheet fle for these data ca be foud at /~ggeorge/343/demos /regress3.xls.

12 ENGI 343 mple Lear Regresso Page - The summary statstcs are Σ x 6 Σ y 38 Σ x 4 Σ 45.6 Σ y 63.6 From whch Σ Σ x Σ y 5 Σ x (Σ x 44 yy Σ y (Σ y 86.6 ˆ β ˆ y ˆ β ad β 5.48 o the regresso le s x y x (3 d.p. (b x y ( d.p. (c x y <! ( d.p. Problem: x s outsde the sample rage for x. LR model may be vald. I oe word: EXTRAPOLATION. (d (e x y 5 r x x y y ( ( 5 R 6.4 ( 44 T yy ( 86.6 / 8.66 ad E T R

13 ENGI 343 mple Lear Regresso Page -3 The ANOVA table s the: ource d.f. M f R E T from whch t f 7.5 (3 d.p. But t.5, t > t.5, 8 Therefore reject H o : β favour of H a : β at ay reasoable level of sgfcace α. [p-value...] r OR t 7. 5 r reject H o : ρ favour of H a : ρ (a sgfcat lear assocato exsts. R 6.4 [Also, from the ANOVA table, r.8598 T 8.66 Therefore the regresso le explas ~86% of the varato y. r r.97, as before.] Cofdece ad Predcto Itervals The smple lear regresso model Y β + β x + ε leads to a le of best ft the least squares sese, whch provdes a expected value of Y gve a value for x : yˆ ˆ β + ˆ β x E[ Y x ] μ Y x. The ucertaty ths expected value has two compoets: the square of the stadard error of the scatter of the observed pots about the regresso le ( σ /, ad the ucertaty the posto of the regresso le tself, whch creases wth the dstace of the chose x from the cetrod of the data but decreases wth creasg ( x x spread of the full set of x values: σ. The ukow varace σ of dvdual pots about the true regresso le s estmated by the mea square error s ME.

14 ENGI 343 mple Lear Regresso Page -4 Thus a ( α% cofdece terval for the expected value of edpots at Y at x x o has ( ˆ ˆ ( xo x β β x ± t s + + o α /, ( The predcto error for a sgle pot s the resdual E Y ŷ, whch ca be treated as the dfferece of two depedet radom varables. The varace of the predcto error s the ( x x V[ E] V[ Y] V[ yˆ ] σ σ Thus a ( α% predcto terval for a sgle future observato of Y at x x o has edpots at ( ˆ ˆ ( xo x β + β xo ± tα /, ( s + + The predcto terval s always wder tha the cofdece terval. Example.3 (cotued (f Fd the 95% cofdece terval for the expected value of Y at x ad x 5. (g Fd the 95% predcto terval for a future value of Y at x ad at x 5. (f α 5% α/.5 Usg the varous values from parts (a ad (e: t.5, s x ˆβ ˆβ x o the 95% CI for μ Y s ( ˆ ˆ ( xo x β x ± t s + β + o α /, ( ± ± E[ Y ] < 3.8 (to 3 s.f.

15 ENGI 343 mple Lear Regresso Page -5 x o 5 the 95% CI for μ Y 5 s ( ˆ ˆ ( xo x β x ± t s + β + o α /, (... ± ± E[ Y 5 ].46 (to 3 s.f. (g x o the 95% PI for Y s ( ˆ ˆ ( xo x β x ± t s + + β + o α /, ( ± ± Y < 4.77 (to 3 s.f. at x x o 5 the 95% PI for Y s ( ˆ ˆ ( xo x β x ± t s + + β + o α /, (... ± ± < Y <.3 (to 3 s.f. at x 5 Note how the cofdece ad predcto tervals both become wder the further away from the cetrod the value of x o s. The two tervals at x 5 are wde eough to cross the x-axs, whch s a llustrato of the dagers of extrapolato beyod the rage of x for whch data exst. ketch of cofdece ad predcto tervals for Example 3 (f ad (g: (f 95% Cofdece Itervals (g 95% Predcto Itervals

Chapter 13 Student Lecture Notes 13-1

Chapter 13 Student Lecture Notes 13-1 Chapter 3 Studet Lecture Notes 3- Basc Busess Statstcs (9 th Edto) Chapter 3 Smple Lear Regresso 4 Pretce-Hall, Ic. Chap 3- Chapter Topcs Types of Regresso Models Determg the Smple Lear Regresso Equato