Linear Regression Siana Halim

Size: px

Start display at page:

Download "Linear Regression Siana Halim"

Thomas Fisher
5 years ago
Views:

1 Lear Regresso Saa Halm Draper,N.R; Smth, H.; Appled Regresso Aalyss,3rd Edto, Joh Wley & Sos, Ic. 998 Motgomery, D.C; Peck, E.A; Itroducto to Lear Regresso Aalyss, d Edto, 99

2 Outle Itroducto Fttg a Straght Le y Least Squares Measures of Model Adequacy

3 The Need for Statstcal Aalyss I research laoratores, expermets are eg performed daly. These are usually small, carefully plaed studes ad result sets of data modest sze. The ojectve s ofte a quck yet accurate aalyss, ealg the expermeter to move to etter expermetal codtos, whch wll produce a product wth desrale characterstcs.

4 The Need for Statstcal Aalyss A Ph.D researcher may travel to a Afrca jugle for a oe-year perod of tesve data-gatherg o plats or amals. She wll retur wth the raw materal for her thess ad wll put much-effort to aalyzg the data she has, searchg for the messages that they cota. It wll ot e easy to ota more data oce her trp s completed, so she must carefully aalyze every aspect of what data she has.

5 The Need for Statstcal Aalyss Regresso aalyss s a techque that ca e used ay of these stuatos. I ay system whch varale quattes chage; t s of terest to exame the effect that some varale exert (or appear to exert) o others. We use the followg ames : Predctor varale put varales puts varales regressors depedet varales Respose varale output varales outputs Y varales depedet varales Respose varale Model fucto + Radom error

6 Straght Le Relatoshp etwee Two Varales I much expermetal work we wsh to vestgate how the chages oe varale affect aother varales. I ths example, for ay gve heght there s a rage of oserved weghts, ad vce versa. Ths varato wll e partally due to measuremet errors ut prmarly due to varato etwee dvduals. Thus o uque relatoshp etwee actual heght ad weght ca e expected. Whe we are cocered wth the depedece of a radom varale Y o a quatty,.e., a varale ut ot radomly varale, a equato that relates Y o s usually called a regresso equato.

7 Fttg a Straght Le y Least Squares 5 oservatos of varale Y : pouds of steam used per moth : average atmospherc temperature degrees Fahrehet. 3 Y The lear frst-order model Y β + β + ε () Parameters Errors

8 Meag of Lear Model Whe we say that a model s lear or olear, we are referrg to learty or olearty the parameters. The value of the hghest power of a predctor varale the model s called the order of the model. For example Y β + β + β + ε Is a secod-order ( ) lear the (β s) regresso model.

9 Least Square Estmato Model ˆ estmate + Y + () Suppose we have sets of oservatos (,Y ),...,(,Y ) the we ca wrte () as Y β + β + ε,,..., (3) Y so that the sum of squares of devato from the true le s S ε ( Y β β ) Sum of squares fucto (4)

10 Least Square Estmato Least Square Estmato We ca determe ad y dfferetatg Eq. (4) frst w.r.t to β ad From (6) we have g q ( ) β the w.r.t β ad settg the result to zero ( ), Y S β β β (7) Y Y (5) ( ), Y S β β β β (7) + Y or So that the estmates ad are solutos of the two equatos : (8) Y (6) ( ) ( ) Y Y These equatos are called the ormal equatos (orthogoal). where we susttute (, ) for (β,β )

11 Least Square Estmato The soluto of (8) S Y Y Y [( )( Y )] ( ) ( )( Y Y ) ( ) ( )( Y Y ) Y Y Y Yˆ S S YY ( ) ( Y Y ) Y Y

12 The Aalyss of Varace We ow tackle the questo of how much of the varato the data has ee explaed y the regresso le. Cosder the followg detty e Y Yˆ Y Y ( Yˆ Y ) Y Y Yˆ Y Yˆ + Y Y ˆ Yˆ Y e Y Y The resdual e s the dfferece etwee two quattes : ()the devato of the oserved Y from the overall mea ()ad the devato of the ftted from the overall mea ( Y Y ) ( Yˆ Y ) + ( Y Yˆ ) If we square oth sdes of ths ad sum from,...,, we ota ( Y Y ) ( Yˆ Y ) + ( Y Y ˆ )

13 Sum of Squares Sum of squares Sum of squares Sum of squares + aout the mea due to regresso aout regresso ANOVA Tale Source of Varato Degree of Freedom (df) Sum of Squares (SS) Due to regresso SS ( ˆ ) R Y Y Aout regresso (Resdual) Total Corrected for mea Y - - SS E ( Y Yˆ ) ( Y Y ) Mea Square (MS) MS R SSE s MS E

14 R Statstc R SS due to regresso gve Total SS,corrected ( Yˆ Y ) ( Y Y ) for the mea Y Adjusted R ( R Where umer of samples, p umer of regressors the lear models ) p R measures the proporto of total varato aout the mea explaed y the regresso. I fact, R s the correlato etwee Y ad ad s usually called the multple correlato coeffcet. R s the the square of the multple correlato coeffcet. Yˆ Y

15 Ifereces The asc assumptos the model Y β + β,,.., : A. ε s a radom varale wth mea zero ad varace σ (ukow); that s E(ε ), V(ε ) σ A. ε are ucorrelated, j, so that cov(ε,ε j ). Thus E(Y ) β + β, V(Y ) σ Y ad Y j, j, are ucorrelated. A3. ε s a ormally dstruted radom varale, wth mea zero ad varace σ ;.e.ε ~ N(, σ ) Uder A3, ε,, ε j are ot oly ucorrelated ut ecessarly depedet.

16 Ifereces Varace ad stadard Devato of ( σ σ V ) S sd( ) ( ) estsd. ( ) σ ( ) s / σ S / s / / S ( )

17 Ifereces Cofdece Iterval for β Test for H : β β vs H : β β If we assume that the varatos of the oservatos aout the le are ormal, (-α)% cofdece lmts for β y calculatg ± t, α s ( ) / where t(-,/α) /α) s the (-/α) /α) percetage pot of a t-dstruto, wth (-) degrees of freedom. t ( ) β se( ) Compare t wth t (-, -/ / α) from t-tale. If t had happeed that the oserved t value had ee smaller tha the crtcal value, we could ot reject the hypothess.

18 Ifereces Stadard Devato of sd( ) ( ) / (-α) Cofdece Iterval for β ± t (, α ) ) σ ( ) / s A t- test for H : β β vs H : β β wll e rejected f β fall outsde the cofdece terval, ad vce versa, or compare t β ( ) Wth t(-, -/α). / s

19 F-Test for Sgfcace of Regresso The rato : F MS MS follows a F-dstruto wth (,-) degrees of freedom provded that β R E Ths fact ca e used as a test of H :β versus H : β. We compare the rato F wth the (-α)% pot of the taulated F(, -) dstruto. Reject H f F > F(, (, -) )

20 Ifereces Stuatos where the hypothess H :β s ot rejected y y Stuatos where the hypothess H :β s rejected y x y x x x

21 Regresso Aalyss : Y vs The regresso equato s Y Predctor Coef SE Coef T P Costat S.895 R-Sq 7.4% R-Sq(adj) 7.% Aalyss of Varace Source DF SS MS F P Regresso Resdual Error Total

22 Measures of Adequacy The major assumptos that we have made so far our study of regresso aalyss are as follows :. The relatoshp etwee y ad x s lear, or at least t s well approxmated y a straght le. The error term ε has zero mea 3. The error term ε has costat t varace σ 4. The errors are ucorrelated 5. The errors are ormally dstruted.

23 Resdual Aalyss explaed y the regresso We have df defed d the resduals as model. It s also coveet to thk of resduals as the realzed e y yˆ,,..., or oserved values of the errors. The resduals have several Oservato Ftted value mportat propertes. They have mea zero, ther h approxmate Resdual may e vewed as the average varace s devato etwee the data ad the ft. It s a measure of the ( e e ) e varalty ot SSE MS E

Normal Proalty Plot Although small departures from ormalty do ot effects the model greatly, gross o ormalty s potetally more serous as the t- or F statstcs, ad cofdece ad predcto tervals deped d o

24 Normal Proalty Plot Although small departures from ormalty do ot effects the model greatly, gross o ormalty s potetally more serous as the t- or F statstcs, ad cofdece ad predcto tervals deped d o the ormalty assumpto. Furthermore f the errors come from a dstruto wth thcker or heaver tals tha the ormal, the least squares ft may e sestve to a small suset of the data To check the ormalty, we use the QQ (Quatle-Quatle) Quatle) plot of resdual

25 Normal Proalty Plot Ideal heavy taled lght taled rght skew left skew

26 Plot of Resdual agast ŷy A plot of the resduals e versus the correspodg ftted values ad x s useful for detectg several commo types of model adequaces satsfactory fuel doule ow o lear

27 Other Resdual Plots If the tme are sequece whch the data were collected s kow, t may e structve t to plot the resduals agast tme order. The tme sequece plot of resduals may dcate that the errors at oe tme perod are correlated wth those at other tme perods (autocorrelato) Postve autocorrelate Negatve autocorrelate

28 Testg Homogeety of Pure Error (Optoal). Bartlett s Test Let υ υ + υ + + υ m ad Let s,s,,s m e the estmate of σ from the m groups of repeats wth υ,υυ,,υυ m degrees of freedom, respectvely, where υ j j ad s s ( Y Y ) C + m υ υ 3m Ad m s the umer of groups wth repeat rus. The test statstc s the m B υ l se υ j l s j Whe the varace of the groups are all the same, B s dstruted as χ m-. A sgfcat B value could dcate homogeeous varaces. j u ju j j j m υ s e m υ It could also dcate o-ormalty. ormalty. j C

29 Testg Homogeety of Pure Error (Optoal). Levee s Test usg Meas The approprate F- statstc s the Cosder, the jth group of repeats, the asolute devatos z ju Y ju Y j, u,,..., Of the Y s from the meas of ther repeats group. Cosder ths as a oe-way classfcato ad compare the etwee groups mea square wth the wth groups mea square va a F-test. where z j m j m j j ( z j z) ( m ) m ( z z ) ( ) ju j j u j j z u ju j, z j m j z ju j u j The F-value s referred to F, usg oly the upper tal. j { m, ( ) } m j j

30 Dur-Watso Test The Dur Watso test It ca e show that : checks for a sequetal. d 4 always depedece whch each error. If successve resduals are (o so resdual) s correlated postvely serally correlated, wth those efore ad after t that s, postvely correlated the sequece. ther sequece, d wll e ear 3. If succesve resduals are ( ) d eu eu eu egatvely correlated, d wll e u u ear 4, so that 4 d, wll e ear 4. The dstruto of d s symmetrc aout.

31 Dur-Watso Test The test s coducted as follows : Compare d (or 4-d, d whchever h s closer to zero) wth d L da d u the followg tale. If d < d L, coclude that postve seral correlato s a posslty. If d > d u, coculde that o seral correlato s dcated. If 4-d < d L, coclude that egatve seral correlato s a posslty f 4-d d > d u, coclude that o seral correlato s dcated. f the d (or 4-d) value les etwee d L ad d u, the test s coclusve. A dcato of postve or egatve seral correlato would e cause for the model to e reexamde.

32 Dur-Watso Test %.5% 5% d L d u d L d u d L d u Iterpolate learly for termedate - values

33 Detecto ad Treatmet of Outlers A outler s a extreme oservato. Resduals that are cosderaly larger asolute value tha the others, say three or four stadard devatos from the mea, are potetal outlers. Outlers are data pots that are ot typcal of the rest of the data.

Detecto ad Treatmet of Outlers Outlers should e carefully vestgated to see f a reaso for ther uusual ehavor ca e foud. Sometmes outlers are ad values, occurg as a result of uusuaal ut explaale evets.

34 Detecto ad Treatmet of Outlers Outlers should e carefully vestgated to see f a reaso for ther uusual ehavor ca e foud. Sometmes outlers are ad values, occurg as a result of uusuaal ut explaale evets. Examples clude faulty measuremet or aalyss, correct recordg of data ad falure of a measurg strumet. If ths s the case, the the outler should e corrected (f possle) or deleted from the data set. Sometmes we fd that the outler s a uusual ut perfectly plausle oservato. Deletg these pots to mprove the ft equato ca e dagerous, as t ca gve the user a false sese of precso estmato or predcto.

35 Cook s Dstace Cook (977) proposed that the fluece of the th data pot e measured y the squared scaled dstace D Yˆ D ( Yˆ Yˆ( )) ( Yˆ Yˆ( )) ( ps ), Yˆ( ) ( ), Yˆ Yˆ( ) { ( ) } { ( ) } ps { - () } s the vector of predcted values from a least square ft whe the th data pot s deleted. p s ay power, s s the sample of stadard devato.

36 Resdual Plots Resdual Plots for Y 99 9 Normal Proalty Plot of the Resduals Resduals Versus the Ftted Values Percet 5 Resdual Resdual Ftted Value Freq quecy Hstogram of the Resduals Res sdual Resduals Versus the Order of the Data Resdual Oservato Order 4

Chapter 13 Student Lecture Notes 13-1

Chapter 13 Student Lecture Notes 13-1 Chapter 3 Studet Lecture Notes 3- Basc Busess Statstcs (9 th Edto) Chapter 3 Smple Lear Regresso 4 Pretce-Hall, Ic. Chap 3- Chapter Topcs Types of Regresso Models Determg the Smple Lear Regresso Equato