COURSE: Applied Regression Analysis. Lecture 1: Review Simple linear regression.

Transcription

1 COURSE: Appled Regresso Aalyss Lecture : Revew Smple lear regresso.

2 Fudametal elemets of statstcs: Populato: set of uts Sample: a subset of the populato Varable of terest: systolc blood pressure cotuous umber of errors o a exam dscrete dabetc/o-dabetc categorcal bary martal status: marred, dvorced, sgle omal degree of pa: mmal/moderate/severe ordal Statstcal ferece: estmate, predcto or other geeralzato about the populato based o the formato cotaed the sample Relablty of statstcal ferece: degree of ucertaty assocated wth statstcal ferece

3 Types of varables regresso Depedet respose varable: affected by oe or more depedet varables, assumed to have a probablty dstrbuto at each value of the depedet varable Idepedet explaatory, covarate, predctor, regressor varable: ca be set to a desred level f ts values are recorded as they occur a populato Example: cosumpto of saturated fatty acds ad plasma cholesterol levels, plasma cholesterol levels ad heart dsease Types of relatoshps betwee varables: assocato ad causal 3

4 Dstrbutos: Probablty dstrbuto s specfed mathematcally ad s used to calculate the theoretcal probablty of dfferet values occurrg. It s descrbed by a mathematcal formula wth parameters. Examples of probablty dstrbutos: ormal, log-ormal, bomal, t-dstrbuto, etc. 4

5 Some graphs of ormal dstrbutos 5

6 Normal desty f y exp y f y y exp{ } 6

7 T-dstrbuto T-dstrbuto s lke ormal but wth heaver tals As the degrees of freedom of the t-dstrbuto crease t more closely resembles the ormal dstrbuto For df=30 or hgher the two are almost dstgushable 7

8 Parameters Mea: sum of measuremets dvded by the total umber of measuremets Populato mea: E N N Varace: sum of squared devatos from the mea Populato varace: N Var N Populato stadard devato: SD Var 8

9 Other mportat parameters Meda the mddle value whe the observatos are ordered from the lowest to the hghest Mode the measuremet that occurs most ofte p th percetle: the value such that at most p% of the measuremets fall below t ad 00-p% above t whe the measuremets are ordered from lowest to hghest Skewess 9

10 Estmates ad samplg Estmate: a quatty computed from the sample whch s teded to estmate a populato parameter Example: Sample mea Varablty of a estmate: Example: Stadard error of the mea If σ s ukow t s estmated by the sample stadard devato s s Samplg dstrbuto of a estmate: f may samples of fxed sze ca be obtaed what would the hstogram of the estmator look lke Example: The dstrbuto of s ether stadard ormal or t s s 0

11 Cofdece tervals Defto: a rage of values whch we ca be cofdet 90%, 95%, 99%, ever 00%! cludes the true value Basc dea: the cofdece terval covers a large proporto of the samplg dstrbuto of the statstc of terest Example: To obta a cofdece terval for a populato mea take estmator +/- *stadard error approxmately 95% CI Iterpretato: About 95 out of 00 cofdece tervals based o dfferet radom samples from the same populato wll clude the true mea. 5% wll ot clude the true mea.

12 Oe sample cofdece terval CI: t /, s 95% CI meas α=0.05 Iterpretato of cofdece terval: We are 95% cofdet that the terval cotas the true populato mea. That s, f we were to costruct may 95% CI based o dfferet radom samples from the same populato, 95% of these tervals wll cota the true mea, 5% wo t. Our partcular terval ether does or does ot cota the true mea.

13 Hypothess tests Null hypothess Ho: usually the opposte of the research hypothess geeratg data o dfferece betwee treatmets, o lear relatoshp betwee ad Alteratve hypothess Ha: research hypothess treatmet A s better tha treatmet B, creases wth Alpha level: tolerace for mstakely declarg Ha to be true usually set at 0.05, but ca use 0.0, 0.0. Ths type of mstake s called type I error. Test statstc: umercal summary of evdece cotaed the sample favor of Ha uder the assumpto that Ho s true. P-value: the probablty of obtag as much or more evdece as cotaed the test statstc favor of Ha uder the assumpto that Ho s true. 3

14 Hypothess testg Ho: = 0 vs Ha: 0 0 TS: t* = s Test statstc has t dstrbuto wth - degrees of freedom Rejecto rego t* > t-α/,- or t* < -t-α/,- p-value = the probablty to get a more extreme value tha the observed value of the test statstc based o t-dstrbuto wth - degrees of freedom. Note: ths probablty s computed uder Ho. Reject Ho f p-value < α 0.05 or equvaletly f t* fall RR. 4

15 Rejecto regos crtcal values of t-dstrbuto 5

16 Fahrehet Restg Metabolc Rate kcal/4 hrs Relatoshp betwee varables Fuctoal Statstcal Celsus Weght kg 6

17 Fuctoal vs Statstcal Fuctoal: - = f - relatoshp s perfect - data pots fall exactly o the curve of the relatoshp - systematc part oly Statstcal: - = f + ε - relatoshp s ot perfect - data pots are scattered aroud the curve - systematc plus radom part 7

18 Model defto depedet varable for subject depedet varable for subject The regresso equato s the: 0,,... - β 0 s ukow tercept ad β s ukow slope β 0 ad β are called regresso parameters - ε are depedet, detcally dstrbuted errors wth mea 0 ad varace σ Eε =0, Varε = σ >0, Covε, ε j = 0 8

19 Propertes of SLR The model s lear the parameters There s a sgle depedet varable The model s also lear the depedet varable The mea of the depedet varable depeds o the predctor accordg to the systematc part of the model E = β 0 + β The varace of the depedet varable s costat Var = σ 9

20 Estmato of regresso parameters Least squares s the stadard method. The least squares regresso le mmzes the sums of squares of the vertcal dstaces from the observatos to the le resduals. The obtaed estmates of the regresso parameters are called ordary least squares estmates These are the values that mmze Q 0 0

21 Estmato of regresso parameters cot The estmates of the regresso parameters are obtaed by solvg the ormal equatos ad are as follows: b b0 b b s terpreted as the estmated mea chage the respose varable per ut chage the predctor b 0 s terpreted as the estmated mea respose at 0 value of the predctor. Note that f 0 s ot the rage of values of the predctor varable the ths estmate s meagless.

22 Regresso le b 0 b or equvalet ly b

23 Estmatg the varace Ftted respose: ˆ b0 b Resdual: e ˆ Error sum of squares: Mea squared error: The resdual varace σ s estmated by the MSE SSE SSE MSE ˆ ˆ e 3

24 Propertes of least squares estmates Ubased o average do ot overestmate or uderestmate the true value: Eb 0 =β 0, Eb =β, EMSE=σ Estmated regresso coeffcets have mmum varace amog all ubased lear estmators,.e. amog all estmators that are lear fuctos of the s For ferece about the regresso parameters we eed addtoal assumpto: ε ~ N0, σ 4

25 Maxmum lkelhood estmato Ths s a alteratve method to ordary least squares to obta estmates of the regresso parameters ad the error varace It requres that the errors ε ~..d. N0, σ Based o ths method we ca also do ferece hypotheses tests, cofdece tervals for the parameters of terest The ma dea s to fd the values of the parameters that maxmze the lkelhood of the observed data 5

26 Normal desty f y exp y f y y exp{ } 6

27 Lkelhood fucto The desty of each dvdual s: The lkelhood s the product of the dvdual destes: } exp{ 0 f L 0 0 } exp{,, 7

28 Maxmum lkelhood estmates The estmates of the regresso parameters are exactly the same as the OLS estmates b 0 ad b The estmate of the varace s dfferet based estmate: MSE ˆ ˆ ˆ b ˆ 0 SSE ˆ 8

29 Iterpretato of SLR The tercept s the value of E for =0 The regresso le cossts of the estmated mea resposes systematc part over the rage, of the fxed s Regresso le passes through the pot Sum of resduals s zero e ˆ 0 Therefore ˆ Sum of squared resduals Q s mmum Also e ˆe 0 9

30 Data example Assessmet of the stregth of assocato betwee body weght kg ad restg metabolc rate RMR kcal/4hr. Ca body weght be used to predct RMR? Now please look at attached SAS code 30

31 Results from data example b 0 = 8.3 b = 7.06 Regresso equato: RMR= *weght SSE = , MSE = Pot estmate of the mea RMR for weght = 70 kg: = Resdual for the frst data pot: e = Note: sum of resduals s 0 3

32 Dstctos betwee regresso ad correlato models Both smple lear regresso ad correlato ca be used to assess the lear relatoshp betwee two cotuous varables. I regresso oe of the varables s a respose, the other oe s a predctor. Correlato treats the varables equally. Correlato assumes to be a radom varable Regresso s more formatve tha correlato. It allows to predct values of the respose depedet varable from values of the depedet varable. 3

33 Jot ad margal dstrbutos of ad The bvarate ormal dstrbuto descrbes the jot dstrbuto of two cotuous varables ~ N, μ ad σ are the mea ad stadard devato of the margal dstrbuto of μ ad σ are the mea ad stadard devato of the margal dstrbuto of ρ s the correlato coeffcet betwee ad Margally ~ Nμ,σ ad ~ Nμ,σ 33

34 Graph of bvarate ormal desty 34

35 Codtoal dstrbuto of gve ad SLR ~ Nμ +ρσ /σ - μ, σ -ρ,.e. for every value of the codtoal dstrbuto of gve s ormal wth mea ad varace that deped o the parameters of the jot dstrbuto The mea of the dstrbuto of s a LINEAR fucto of For all values of the varace of the codtoal dstrbuto s the same Same observatos ca be made for the dstrbuto of For makg fereces for codtoal o codtoal o the ormal regresso model s approprate Eve f s ot ormally dstrbuted, but are depedet ad the dstrbuto of does ot volve the regresso parameters, we ca stll use smple lear regresso 35

36 Estmato of correlato coeffcet Pearso product-momet correlato coeffcet r - ρ, ρ=0 whe ad are depedet Testg H 0 : ρ=0 versus H a : ρ 0 s equvalet to testg whether the slope of the regresso le of or s 0 Settg up cofdece terval for ρ s based o the ormal dstrbuto ad Fsher s z-trasform 36

37 Noparametrc estmato of the correlato coeffcet Whe the jot dstrbuto of ad s ot bvarate ormal ad ca ot be trasformed to ormalty, the oparametrc rak correlato procedure ca be used Spearma rak correlato coeffcet: R R R R rs R R R R That s, we frst obta the raks of the s R ad s R separately ad the compute the Pearso correlato coeffcet o the raks 37

38 COURSE: Appled Regresso Aalyss Lecture : Iferece Smple Lear Regresso 38

39 Ifereces cocerg β The test of lear assocato betwee predctor ad respose varable: H 0 : β = 0 vs H a : β 0 Cofdece terval for β 39

40 Samplg dstrbuto of the estmator of β Remember b ˆ Sce the estmated slope s a lear combato of the s t s Nβ, σ b, where b Hece, z* b ~ N0, b The varace σ s ukow but we have a ubased estmate of t 40

41 Samplg dstrbuto of the estmator of β cot the MSE Hece the estmated varace wll be: s b MSE ad we ca use the followg statstc: b t* s b Sce we are estmatg the varace, the samplg dstrbuto s o loger ormal t* ~ t-,.e. the samplg dstrbuto of b s t wth - degrees of freedom 4

42 Two-sded hypothess test for β H 0 : β = 0 vs H a : β 0 Compute TS uder H 0 : t* b 0 s b If t* > t-α/,- or f t* < -t-α/,- the reject H 0 favor of H a, otherwse fal to reject H 0 Equvaletly compute p-value as *Pt- > t* ad reject H 0 f p-value < α 4

43 Rejecto regos crtcal values of t-dstrbuto 43

44 Oe-sded hypotheses tests for β Test for egatve slope H 0 : β 0 vs H a : β < 0 Compute TS uder H 0 If t* < -t-α,- the reject H 0 favor of H a, otherwse fal to reject H 0 Equvaletly compute p-value as Pt- < t* ad reject H 0 f p-value < α Test for postve slope H 0 : β 0 vs H a : β > 0 Compute TS uder H 0 If t* > t-α,- the reject H 0 favor of H a, otherwse fal to reject H 0 Equvaletly compute p-value as Pt- > t* ad reject H 0 f p-value < α 44

45 Cofdece terval for β Cosder α = The wth probablty 0.95 Therefore a 95% cofdece terval s as follows:,, t b s b t 45,,,, b s t b b s t b b s t b

46 Iferece cocerg β 0 Estmator: Samplg dstrbuto of b 0 s Nβ 0,σ b 0 Varace: Estmated varace: Therefore we have: b b 0 0 ˆ ~ t b s b 46 ] [ 0 b ] [ 0 MSE b s

47 Iferece cocerg β 0 CI: b t, s 0 0 b HT: H 0 : β 0 =0 vs H a :β 0 0 TS: t*= b 0 0 ~ t s b 0 RR: t* > t-α/,- 47

48 RMR data example cot d from lecture Assessmet of the stregth of assocato betwee body weght kg ad restg metabolc rate RMR kcal/4hr. Costruct 90% CI for the slope. Test at alpha=0.05 whether the slope s sgfcatly postve. 48

49 RMR data example cot d 90% CI: b ± t-0.0/,4s{b } s {b } = MSE/[-s x ]= /43*4.63 =0.956 s{b }= ± =5.35,8.7 We are 90% cofdet that the mea crease RMR per kg crease body weght s betwee 5.35 ad 8.7 kcal/4hrs HT: H 0 : β = 0 vs H a : β > 0 TS: t* = b -0/s{b }=7.06/0.98=7.0 RR: t*>.68 Cocluso: t* >.68 ad hece we reject H 0 ad coclude that average RMR sgfcatly creases o average as body weght creases 49

50 Iferece cocerg E h Pot estmator: Expectato: Varace: Samplg dstrbuto: ] [ ˆ h h ] [ ˆ MSE s h h 50 h h b b 0 ˆ ~ ˆ ˆ t s E h h h ˆ h h E E

51 Iferece cocerg E h cot d CI: HT: ˆ t, s h h ˆ H 0 : E h = E 0 vs H a : E h E 0 TS: ˆ h E0 t* ~ t s ˆ h RR: t* > t-α/,- 5

52 Predcto of a ew observato hew Pot estmator s the same as mea estmato: Expectato s the same but ca t be used ferece sce t s ukow: Rather we base ferece o the followg dstrbuto: Varace of predcto: Estmated varace of predcto: ˆ h ew h ew b 0 ˆ s pred b h ew h ew ~ t ˆ h ew h ew h [ s E ˆ h ew E h ew h ew pred MSE[ ] ] 5

53 Iferece for hew Predcto terval for hew : ˆ t, s ˆ h ew h ew Note that ths terval s wder tha the terval aroud the estmated mea at the same value of 53

54 RMR example: mea estmato ad predcto of a ew observato 95% CI for mea RMR at weght = 90kg: CI = Ŷ 90 ± t0.975,4s{ŷ 90 } Ŷ 90 = b 0 +b *90= = s ˆ h MSE[ h [ 44 s{ŷ 90 } = 8.0 CI = ± = , % predcto terval for RMR for a dvdual who weghs 90kg: CI = Ŷ 90 ± t0.975,4s{pred}, Ŷ 90 = b 0 +b *90= s ˆ h MSE[ h ] s{pred} = CI = ± =.9, 77.9 ] [ ] ]

55 Aalyss of varace approach to regresso Total varato varato aroud the mea of the respose varable: SSTO SSTO Varato aroud the ftted regresso le: SSE SSE ˆ Varato of ftted regresso le from the mea of the respose varable: SSR SSR ˆ 55

56 Parttog of total sum of squares ad degrees of freedom SSTO = SSE + SSR That s, ˆ ˆ Total df = - Regresso df = Error df = - Total df = Error df + Regresso df - =

57 Mea squares ad ANOVA table MSE = SSE/- MSR = SSR/ = SSR Source of varato df SS MS F p Regresso SSR MSR MSR/MSE Error - SSE MSE Total - SSTO 57

58 F-test for β =0 H 0 : β = 0 vs H a : β 0 TS: F* = MSR/MSE ~F,- If F* > F-α;,- reject H 0 favor of H a, otherwse fal to reject H 0 Ituto: E MSE E MSR Uder Ho the rato s, uder the alteratve t s > Note: F* = t*, hece the F-test s equvalet to the two-sded t-test 58

59 Geeral Lear Test Full model: = β 0 + β + ε Reduced model: = β 0 + ε H 0 : β = 0 vs H a : β 0 SSEF = SSE SSER = SSTO TS: [ SSE R SSE F] [ df R F* SSE F / df F df F] If F* > F-α;df R -df F,df F the reject the ull ad assume that the full model fts the data better tha the reduced model 59

60 RMR example: ANOVA table Source of varato df SS MS F p Regresso SSR <.000 Error 4 SSE Total 43 SSTO 60

61 Measures of assocato betwee ad Coeffcet of determato: SSR SSE r SSTO SSTO r s terpreted as the proportoate reducto of total varato assocated wth the use of predctor 0 r r = whe all the pots fall o the ftted regresso le ad the regresso le s ot horzotal r = 0 whe the ftted regresso le s horzotal 6

62 Measures of assocato betwee ad cot d Correlato coeffcet Perso momet-product correlato coeffcet - r r > 0 dcate postve lear assocato betwee ad r < 0 dcate egatve lear assocato betwee ad Note that small r does ot ecessarly mea o relatoshp betwee ad t meas o LINEAR relatoshp Hgh r does ot ecessarly dcate good ft ad that good predctos ca be made r r 6

63 Relatoshp betwee r ad the estmated regresso slope b x y s s r r b / 63 s x ad s y are the sample stadard devatos of ad respectvely Note that b = 0 mples r=0 ad vce versa The sgs of b ad r are also the same The value of r s affected by the spacg of the values

64 Results from data example Testg b =0: t*=7., p<.000 F*=5.5,p< % CI for b : 6.09,9.03 R = % CI for mea RMR for weght = 90 kg: 390.0, % CI for dvdual RMR for weght = 90kg:.9,

65 COURSE: Appled Regresso Aalyss Lecture 3: Dagostcs ad remedal measures 65

66 Departures from model No-lear regresso fucto No-ormal errors No-costat error varace No-depedet error terms Presece of outlers Omsso of mportat predctor varables Remedal measures Noparametrc regresso Quadratc or hgher order polyomal Trasformatos Weghted least squares Models for correlated data mxed models, tme seres models Robust regresso Multple regresso 66

67 Cosequeces of model departures Nolearty ad/or omsso of predctor varables lead to based estmates of the parameters serous No-costat error varace leads to less effcet estmates ad to vald error varace estmates less serous Presece of outlers may or may ot be serous. Depeds o how fluetal outlers are for the regresso estmato ad o the sze of the data set. No-depedece of errors results based varaces estmates are ubased. May be serous. 67

68 Dagostcs for predctor varable Dot plot: useful whe umber of observatos the data set s ot large. Helps detfy outlyg cases. Sequece plot: useful whe data o the predctor varable are obtaed sequece. Helps detfy patters. Stem-ad-Leaf Plot: provdes formato smlar to a frequecy hstogram. Useful to detfy outlers ad skewess. Box plot: shows mmum, maxmum value, frst, secod ad thrd quartles. Helps detfy skewess ad outlers. Most useful for large data sets. 68

69 Resduals e ˆ Remember Basc dea: If the assumptos of the regresso model are satsfed the dstrbuto of the resduals should resemble the error dstrbuto e The mea of the resduals s zero e 0 The varace of the resduals s approxmated by the MSE The resduals are ot depedet because the ftted values are based o the same regresso le. The resduals are subject to costrats: that ther sum s zero ad that the products e sum to 0. However whe s large the depedecy ca be gored. 69

70 Stadardzed resduals The dea s to stadardze the resduals by ther stadard devato However the latter s ukow We ca use the MSE Semstudetzed resduals: Studetzed resduals: use a dfferet deomator e * e e MSE e MSE 70

71 Several prototype stuatos for resdual vs predctor plot Fgure 3.4 o p.06 textbook 7

72 No-lear regresso fucto Ca be assessed va plot of the resduals vs predctor varable or equvaletly resduals vs ftted values 7

73 No-ormal error terms To detect gross departures from ormalty for a relatvely large sample a hstogram, dot plot, box plot or stem-ad-leaf plot of the resduals ca be helpful Otherwse a ormal probablty plot of the resduals s more formatve 73

74 Normal probablty plot Order the resduals Plot each resdual agast ts expected value uder ormalty Expected value of the k th smallest resdual s k.375 MSE z.5 za s the a00 percetle of the stadard ormal dstrbuto Plot should be early lear 74

75 Good ormal probablty plot of resduals 75

76 Examples of ot so good ormal probablty plots 76

77 Cauto assessg ormalty Normal probablty plots may reflect other volatos of the assumptos: For example a wrog choce of regresso fucto Or o-costat error varace Ivestgate other possble volatos frst! 77

78 No-costat error varace Ca be detected usg plot of resduals versus values of the predctor varable Equal varablty aroud zero dcates costat varaces Icreasg spread of the resduals aroud zero dcates that varace creases wth the mea megaphoe type Plot of absolute resduals or squared resduals agast may be eve more useful to detect ocostacy of error varace 78

79 Brow-Forsythe test Approprate for SLR Ca be used eve whe errors are o-ormal Requres large sample sze to be able to gore depedecy amog resduals Ma dea s to separate the resduals two groups ad compare the average absolute devatos from the ceter the two groups 79

80 Brow-Forsythe test cot d ~ ~ ~ groups resdual ad of medas - ~, ~ * * d st t t s d d t d d d d s e e d e e d e e BF BF 80

81 No-depedet error terms Whe the data are recorded a temporal or spatal fasho a sequece plot may be useful to detect o-depedet error terms Ths plot may reveal tred effect or cyclcal odepedece 8

82 Correlated errors 8

83 Correlated errors 83

84 Presece of outlers Outlers are extreme observatos Ca be detfed based o plots of resduals agast or ftted values, box plots, stem-ad-leaf plots or dot plots of resduals It s better to plot semstudetzed resduals Semstudetzed resduals wth values of four or more are cosdered outlers Outlers may result from error recordg, malfucto, mscalculato, etc ad may eed to be dscarded Otherwse they may provde mportat formato Note that the estmates of regresso parameters may or may ot be greatly affected by outlers 84

85 Omsso of mportat predctor varables Plot resduals agast varables omtted from the model that mght have mportat effects o the respose If a patter emergece t may lead to a substatally better ft f we clude oe or more of these addtoal varables the model 85

86 F test for lack of ft Tests whether the chose regresso fucto adequately fts the data Assumes that ~ depedet ad ormally dstrbuted ad that the error varaces are costat Requres repeat observatos o some 86

87 F-test for lack of ft cot d, c levels of the predctor, c replcates at each level j observed respose for the th replcate at the j th level of Full model: Let SSPE sum of squares for pure error deote the SSE for the full model Reduced model: j j j, j ~ dep. N0, j 0 j j, j ~ dep. N0, 87

88 F-test of lack of ft cot d 0, ; * * * H reject c c F F If MSPE MSLF c SSPE c SSLF F SSPE SSE SSLF c SSPE c SSPE SSE F 88

89 Trasformatos Trasformato o should be attempted whe the error terms are approxmately ormally dstrbuted wth costat varace but the relatoshp betwee ad s o-lear Otherwse trasformato o s more approprate For some trasformatos log, sqrt, verse sqrt addg a costat may be ecessary to make umbers postve 89

90 Prototype plots for trasformato of Use = or = exp f the - plot suggests a arc from lower left to upper rght wth bulge below the straght le st plot o rght Use = square root of or = log 0 f the - plot suggests a arc from lower left to upper rght wth bulge above the straght le Use = / or = exp- f the - plot suggests a arc from upper left to lower rght wth bulge below the straght le d plot o rght 90

91 Prototype plots suggestg trasformato o 9

92 Trasformatos o The choce of a trasformato of may be suggested by examg the plot of resduals agast ftted values. If ths appears lear, but the varace of the resduals creases as ftted creases, suggestg a wedge or megaphoe shape, the takg square roots, logarthms, or recprocals of the values may promote homogeety of varace Note that a smultaeous trasformato o may be ecessary 9

93 Trasformatos o Use = square root of f there s a arc from lower left to upper rght wth bulge below the straght le, ad the varace of the resduals creases as ftted creases Use = log f there s a arc from upper left to lower rght, ad the varace of the resduals creases as ftted creases Use = / f varace of the resduals creases as ftted creases 93

94 Box-Cox trasformatos Automatcally detfes a trasformato from the famly of power trasformatos = λ, where λ s detfed from the data λ = correspods to = λ = ½ correspods to = λ = 0 correspods to = log e λ = -½ correspods to = / λ = - correspods to = / Regresso model: 0 0 The Box-Cox trasformato fds the MLE of λ 94

95 Commets regardg trasformatos Theoretcal cosderatos may preval Resdual plots ad tests eeded to ascerta approprateess of trasformato Iterpretato ad propertes of regresso coeffcets are wth respect to the trasformed scale A more coveet value of λ may be selected for terpretato purposes Whe the cofdece terval for λ cludes t may be better to stay wth the orgal scale 95

96 Noparametrc regresso Idea: ft a smoothed curve to the data to explore or cofrm regresso relatoshp For tme-seres data popular methods are: the method of movg averages, the method of rug medas For regresso data popular methods are bad regresso ad locally weghted regresso scatter plot smoothg Lowess 96

97 Lowess method Obtas a smoothed curve by fttg successve lear regresso fuctos local eghborhoods The smoothed value at a gve s equal to the ftted value for the regresso that local eghborhood For example f eghborhood of 3 values s used, the smoothed value of at s the ftted value for at based o the regresso ftted to,,, 3, 3 Smlarly the smoothed value at 3 wll be the ftted value for at 3 based o the regresso ftted to, 3, 3 ad 4, 4 97

98 Steps obtag fal smoothed curve. The lear regresso s weghted wth smaller weghts gve to values further from the mddle level each eghborhood. Lear regresso fttg s repeated wth revsed weghts so that cases wth large resduals the frst fttg receve smaller weghts the secod fttg 3. Addtoal teratos of step may be eeded 98

99 Choces lowess method Sze of successve eghborhoods the larger the sze the smoother the fucto but essetal features may be lost I SAS PROC LOESS a smoothg parameter s s chose. Whe s < the s fracto of values closest to are chose each eghborhood Weght fucto for values SAS PROC LOESS uses a trcube weght fucto: w = [3/5] -[d /d q ] 3 3 where d,, d q are the dstaces from the st, d, qth closest value to Weght fucto for resduals 99

100 Importat pots regardg the lowess method No aalytcal expresso s provded for the fuctoal form of the regresso relatoshp Hgher order polyomals may be used to smooth out local eghborhoods If lowess curve falls cofdece bads of regresso le the t ca be cosdered cofrmatory of the chose regresso relatoshp 00

101 COURSE: Appled Regresso Aalyss Lecture 4: Smultaeous ferece ad other topcs 0

102 Jot estmato of β 0 ad β Statemet cofdece coeffcet: reflects the proporto of cofdece tervals that cota the true value of a parameter repeated samplg Settg two separate cofdece tervals for the slope ad the tercept SLR assures that each of the two statemet cofdece coeffcets s correct However, the probablty that both cofdece tervals cota ther respectve parameters s less tha 95% Famly cofdece coeffcet correspods to the proporto of repeated samples whch both the true tercept ad the true slope fall ther respectve cofdece tervals 0

103 Boferro jot cofdece tervals for tercept ad slope Separate cofdece tervals: Let A deote the evet that β 0 does ot belog to the frst CI, ad A deote the evet that β does ot belog to the secod CI. Therefore PA = PA =α We wat But } { ; / } { ; / 0 0 b s t b b s t b A A P 03 A P A P A A P A P A P A A P A A P

104 Boferro jot cofdece tervals cot d Therefore, f we costruct two 00-α/% cofdece tervals we wll get at least 00- α% famly cofdece coeffcet for both parameters The jot cofdece tervals are as follows: b b 0 t / 4; t / 4; s{ b s{ b 0 } } 04

105 A example Remember for RMR data we had b 0 = 8.3, sb 0 =76.98 b = 7.06, sb =0.98 Therefore jot 90% cofdece tervals wll be: b 0 ± t-0.0/4;4s{b 0 }=8.3 ± = , b ± t-0.0/4;4s{b }=7.06 ± = 5.08,9.04 Iterpretato: we are at least 90% cofdet that both the tercept ad the slope fall ther respectve cofdece terval above 05

106 Commets The Boferro famly cofdece coeffcet s ot a exact coeffcet but s rather a lower boud to the desred probablty The Boferro equalty s exteded to more tha two evets P g A g Useful whe umber of cofdece tervals s ot too large j Jot cofdece tervals ca be used for testg for example of whether both slope ad tercept are equal to zero Each cofdece terval ca have ts ow cofdece level to reflect ts relatve mportace 06

107 Smultaeous estmato of mea resposes Separate estmates of the mea respose at dfferet levels of the predctor eed ot be smultaeously rght or smultaeously wrog. It s possble that the cofdece terval for the mea respose s rght oly over some part of the rage We ll cosder two procedures: Workg-Hotellg procedure Boferro procedure 07

108 Workg-Hotellg procedure The cofdece bouds at each value of are the values of cofdece bad for the etre regresso le: Here W =F-α;,-} ˆ Ws{ ˆ h h } 08

109 RMR example h h S{ h } Workg-Hotellg 90% CI for mea: W = F0.9,,4=4.87 W = ±.7.90= 73.4, ±.4.80 =335.30, ±.36.37= ,

110 Boferro procedure Same dea as for smultaeous cofdece tervals for the tercept ad slope For smultaeous cofdece tervals o g meas: ˆ h Bs{ ˆ } h B t /g; 0

111 RMR example h h S{ h } Boferro 94% CI for mea: -α/g=0.99 B = t0.99,4= ± =67.8, ± =330.09, ± =450.34,66.38

112 orkg-hotellg vs Boferro procedures For large g WH provdes tghter bouds ad hece s preferred Both WH ad Boferro provde lower bouds Whe the levels of the predctor varable at whch mea estmato s of terest are ot kow a pror WH s more approprate sce WH provdes smultaeous protecto for all levels of

113 Regresso through org The regresso le may be forced to go through the org or a partcular value Regresso model: The the pot estmates are: b s MSE ˆ 3

114 Regresso through org cot d CI for the slope the s: where MSE s b CI for the mea respose s: where CI for a predcted respose s: where s ˆ s h MSE h b t / ; s{ } b ˆ t / ; s{ ˆ h h ˆ ˆ h ew h ew MSE t / ; s{ ˆ } h ew h ew } 4

115 Example: Typographcal errors problem 4. umber of galleys for a mauscrpt total dollar cost of correctg typographcal errors Estmated regresso fucto: = % CI for the slope: 8.03±.00.08= 7.85,8. 95% predcto terval for =0: 70.,90.36 Note resduals do ot sum to 0 5

116 Typographcal errors example cot d Test for lack of ft of regresso through the org: full model SLR, reduced model regresso through the org SSE R SSE F * SSE R SSE F F SSE F MSE F If F* F ;, reject F*= /.00=0.6 F* s ot the rejecto rego for ay meagful test ad hece ok to use reduced model H 0 6

117 Caveats of regresso through the org Sum of resduals s ot zero SSE may exceed SSTO R may be egatve The tervals for mea respose ad ew predcto wde at far away from the org Ucorrected total ad corrected total sums of squares are related as follows: SSTOU = SSRU+ SSE where SSTOU ˆ SSRU b, SSE, b 7

118 Effects of measuremet error Measuremet errors do ot affect the estmates as log as these errors are ucorrelated ad ot based Measuremet error does affect the estmate of the slope by atteuatg t except Berkso s case. The magtude of the bas depeds o the relatve szes of the errors ad. Whe measuremet error s preset a specal approach such as the use of strumetal varable approach s eeded 8

119 Berkso s model The observato * s a fxed quatty whle the uderlyg s a radom varable. I that case the errors have expectato 0 ad the predctor varable s a costat therefore the errors are ucorrelated wth t ad hece SLR s vald ad ca be used. 9

120 Choce of levels Whe the levels of are uder the cotrol of the expermeter the followg cosderatos should be used: If the ma purpose of the regresso aalyss s to estmate the slope use well spread levels of If the ma purpose s to estmate the tercept the mea of values should be 0. If the ma purpose s to predct a ew observato at hew the mea of the values should be at hew Choose as may levels as eeded to estmate shape for example at least for straght le, at least 3 for quadratc tred, etc 0

121 Formulae to ad uderstadg of selecto of levels 0 } { ˆ } { ˆ } { } { b b ew h ew h h h

122 COURSE: Appled Regresso Aalyss Lecture 5: Matrx represetato of smple lear regresso

123 Matrces Matrx: a rectagular array of elemets Dmeso: r x c meas r rows by b colums Example: A = [a j ], =,,3; j=, I geeral: A = [a j ], =, r; j=,...c A a a a 3 A a a a 3 a a... ar a a... a r a c a c... arc 3

124 Types of matrces A square matrx r=c Symmetrc matrx square ad upper rght tragle s a mrror mage of lower rght tragle A a a a 3 A a a a 3 a a a

125 Types of matrces cot d Row vector: r= Colum vector: c= Traspose of a matrx: A or A T A = [a j ], A = [a j ] If A s r by c matrx the A s c by r matrx Symmetrc matrx: A = A A A'

126 Types of matrces cot d Dagoal matrx: a square matrx wth all off-dagoal elemets equal to 0 Idetty matrx: a dagoal matrx wth all dagoal elemets equal to Note that A*I=I*A=A Scalar matrx: a dagoal matrx wth all dagoal elemets equal to a scalar 6

127 Examples: A I I ~ ~ 7

128 Types of matrces cot d Vector of oes Vector of zeros J deotes the matrx formed by multplcato of a traspose of a vector of oes ad tself J ' '

129 Equalty of matrces Two matrces are equal oly f they have the same dmesos ad all correspodg elemets are the same That s, f A=[a j ], =, r; j=, c ad B=[b j ], =, k; j=, l for A to be equal to B we eed: K=r, l=c ad a j = b j for all ad j 9

130 Operatos o matrces: addto A 0 B B A 30

131 Operatos o matrces: subtracto A 0 B B A 3

132 Operatos o matrces: multplcato A 0 B A* B 3

133 Cautos about matrx operatos For addto ad subtracto umber of rows of the matrces must be the same ad umber of colums must be the same For multplcato umber of colums of the frst matrx left multpler must be the same as umber of rows of secod matrx rght multpler Note that A + B = B + A but A*B B*A geeral 33

134 Iverse of a matrx Iverse for a square matrx A s A - such that: A A - = A - A = I A - s uque ad has the same rak as A To have a verse a matrx eeds to be full rak or osgular To check where a matrx s of full rak we check whether ts determat s 0 34

135 Fdg the verse For a x matrx: A D A bc ad D a c b d D A d c b a A 35

136 Smple lear regresso matrx terms I Var E E 0 36

137 Regresso parameter estmate matrx otato b b b b ' ' ' ' ' ' ' ' 37

138 Ftted values matrx otato ' ' ' ' ˆ... ˆ ˆ ˆ H H b The H matrx s called the hat matrx H s symmetrc ad dempotet HH=H H s very useful resdual dagostcs 38

139 Resduals matrx otato ˆ... H I MSE e s H I e H I H e e e e I-H s also symmetrc ad dempotet 39

140 Aalyss of varace results matrx otato J I J SSTO H I b SSE J H J b SSR ] '[ ' ' ' ' ' ' ] '[ ' ' ' 40

141 Iferece about regresso coeffcets, mea respose ad predcto of a ew observato ' ' ' ' MSE b s b b 4 ' ' } { ˆ ' ' } { ˆ ' ' } { ˆ ' ˆ h h ew h h h h h h h h h h h MSE s MSE s b ad

142 Example: Flavor deterorato problem 5.4 o p.0 Wll work out class 4

143 COURSE: Appled Regresso Aalyss Lecture 6: Multple Lear Regresso 43

144 Model formulato s the respose for the th subject,,,p- are the values of the predctor varables for the th subject β, β, β p- are ukow parameters to be estmated from the data they are also called partal regresso coeffcets Regresso respose surface: Eε =0, Covε, ε j =0 for j, Varε =σ >0 p p,...,..., 0, 0... p p E 44

145 Commets o model formulato Whe p- = multple lear regresso reduces to smple lear regresso If we assume 0 = the we ca wrte the regresso equato as follows: p, p p k0 β k s terpreted as the mea chage per ut chage the k th predctor whe the remag predctors are held costat The predctors eed ot be all dfferet varables Whe there are two dfferet predctors the respose surface s a plae For ferece we requre ε ~N0,σ k k 45

146 Frst order regresso model All predctors represet dfferet varables Some of the predctors ca be qualtatve dummy varables: For example let be age ad The a frst order regresso model s: female f male f 0 males for E ad females for E

147 Polyomal regresso All varables are powers of the same varable Ths mples curvlear relatoshp betwee the predctor ad the respose, 0...,... p p p p 47

148 Regresso wth trasformed varables, teracto effects Ex: Trasformato of the respose Frst order two-way teracto Ay possble combatos of the above as log as lear the parameters Example of olear regresso: 0 ' log p k k k p p c c c c, exp 0

149 Example: Lug trasplatato data Altma et al, p.364 It s dffcult to measure total lug capacty TLC ad hece t s useful to be able to predct TLC from other formato. The data set cotas measuremets of pre-trasplat TLC of 3 recpets of heart-lug trasplat, obtaed by whole-body plethysmography, ad ther age, sex ad heght. Sample data: Age Sex Heghtcm TLCl 35 F F M

150 Example cot d: TLC heght sex age

151 Model formulato matrx otato,...,,...,...,..., ' 0 ', 0 p p p p ad where as rewrtte be ca 5

152 Model formulato matrx otato cot d ,,, p p subjects all For subject for T T T p p p p th p p T 5

153 Model formulato matrx otato cot d p p vector of resposes β- vector of parameters matrx of costats desg matrx ε ~ N0,σ I ad hece ~ Nβ,σ I 53

154 Estmato of regresso coeffcets Least square estmates are obtaed by mmzg the sum of dstaces from the pots to the regresso plae: Deote the vector of the least squares estmated regresso coeffcets as b: Least squares ormal equatos: Least squares estmates!mstake textbook eq 6.5: Maxmum-lkelhood estmates are the same p p Q, p p b b b b 54 b ' ' ' ' p p p p b

155 Ftted values ad resduals } {, } { ˆ... ' ' ' ' ˆ... ˆ ˆ ˆ H I MSE e s H I e H I b e e e e H H b p p p p 55

156 Sums of squares ad mea squares J I J SSTO p SSE MSE H I b SSE p SSR MSR J H J b SSR ] '[ ' ' ' ' ' ' ] '[ ' ' ' 56

157 ANOVA table Source of varato df SS MS F Regresso p- SSR MSR MSR/MSE Error -p SSE MSE Total - SSTO 57

158 F-test for regresso relato H 0 : β = β = = β p- =0 H a : ot all β k k=, p- equal to 0 TS: F* = MSR/MSE Rejecto rego: If F* > F-α;p-,-p reject H 0 Note that uder H 0 EMSR=EMSE=σ Ad uder Ha EMSR>σ ad hece the test has tutve sese Whe p-= the test reduces to the F-test SLR 58

159 Coeffcets of multple determato ad correlato Usual defto: SSTO Note that the coeffcet s betwee 0 ad ad creases as varables are added to the model Modfed measure that adjusts for umber of varables the model: Adjusted R-square: Adjusted R-square ca decrease as umber of varables crease R a R SSR SSE / p SSTO / SSE SSTO p SSE SSTO Coeffcet of multple correlato: R= R 59

160 Ifereces about regresso parameters The LSE of the regresso parameters are ubased: Eb=β Varace s σ {b}=σ - Estmated varace s s {b}=mse - 00-α% CI for β k s b k ±t- α/;-ps{b k } HT: H 0 : β k =0 vs H a : β k 0 TS: t* = b k /s{b k } Rejecto rego: t* >t- α/;-p Jot fereces o several β k s: b k ±Bs{b k } where B=t- α/g;-p 60

161 Estmato of mea respose } ps{ˆ - α/; ˆ : α00% CI ' } { ˆ : Estmated varace ' } { } { ˆ : Varace } { } { ˆ Expectato : ˆ Pot estmator : } { M ea respose:... Let ' ' ' ' ' ', h h h h h h h h h h h h h h h h h p h h p t MSE s b E E b E h 6

162 Other ferece regardg mea respose Cofdece rego for regresso surface a exteso of Workg-Hotellg bad. Also used for smultaeous cofdece terval for several mea resposes: ˆ h W Ws{ ˆ h } pf ; p, p Boferro smultaeous cofdece tervals for several mea resposes: ˆ h Bs{ ˆ h } B t /g; p 6

163 Predcto of ew observato Idvdual respose: Pot estmator : ˆ h ew Estmated varace : s h ew ' h ew ' h ew b { pred} MSE[ h ' h ' h ] α00% CI : ˆ h ew t α/; - ps{ pred} 63

164 Cauto about hdde extrapolatos Pcture from textbook 64

165 Dagostc ad remedal measures Most of the dagostcs ad remedal procedures from smple lear regresso carry over to multple lear regresso Scatter plot matrx: a collecto of scatter plots betwee predctors ad respose varables. It s useful to assess bvarate relatoshps ad detfy outlers. Correlato matrx: a matrx of correlato coeffcets 65

166 Resdual plots Plot of resduals agast ftted values s useful for assessg the approprateess of the multple lear regresso fucto, the costacy of the varaces of the error terms ad the presece of outlers Normal probablty plots of the resduals ca help assess ormalty of errors. Resdual plots agast each predctor varable ca help assess the adequacy of the regresso model wth respect to that varable. Resduals ca also be plotted agast varables, or teractos of varables ot the model to assess whether these varables/teractos are eeded. Brow-Forsythe test ad Breusch-Paga test ca be appled for a partcular predctor varable suspected to be assocated wth crease or decrease of error varace. F-test for lack of ft s also appled as SLR except that we ow requre replcatos over all predctor varables smultaeously. 66

167 Remedal measures Remedal measures MLR are appled as SLR: Defto of more complex polyomal or teracto models Trasformatos o respose ad predctor varables Box-Cox trasformato 67

168 COURSE: Appled Regresso Aalyss Lecture 8: Statstcal ferece multple regresso 68

169 Extra sums of squares Due to the relatoshp amog predctor varables ad betwee the predctors ad the respose, t s possble that the relatoshp betwee oe predctor ad the respose s affected by other predctors the model SSE decreases as more ad more predctors are added to the model The extra sum of squares measures such margal reducto SSE 69

170 TLC example extra sums of squares Suppose we frst cosder geder as a predctor of TLC: The SSE =56.40, SSR =5.3 The suppose we add heght to the model: The SSE, =38.9, SSR, =4.79 We see that SSE > SSE, ad SSR < SSR,, that s resdual varablty decreases by addg a varable to the model whle systematc varablty creases SSR s called the extra sum of squares whe addg to the model that already cotas : SSR = SSR, - SSR = =7.48 SSR = SSE - SSE, = =

171 Extra sums of squares cot d SSR Sce SSTO SSR SSE the equvalet ly SSR SSE SSR SSE, SSR SSR measures the reducto errors varablty SSE whe s added to the model that already cludes Thus t measures the margal effect of addg to the model that already cludes, 7

172 Extra Sums of Squares cot d Addtoally: SSR Sce SSR SSTO SSR,, 3 3 SSE SSE SSR the equvalet ly SSR, 3 measures the reducto errors varablty SSE whe ad 3 are added to the model that already cludes, SSE, 3,, SSR 3 7

173 Extra Sums of Squares Also: SSR 3, measures the reducto errors varablty SSE whe 3 s added to the model that already cludes ad,,,,,,,, SSR SSR SSR SSE SSE SSR 73

174 Decomposto of SSR to extra sum of squares Dfferet decompostos ca be cosdered: freedom of wth q degrees assocated s varables for q addtoal squares extra sum of That s,...,,,,...,,,,,,,,, SSR MSR SSR MSR SSR SSR SSR SSR SSR SSR SSR SSR SSR SSR SSR 74

175 ANOVA table wth decomposto of sums of squares Source of varato df SS MS F Regresso p- SSR MSR MSR/MSE SSR MSR MSR /MSE SSR MSR MSR /MSE 3, SSR 3, MSR 3, MSR 3, /MSE p-, p- SSR p-, p- MSR p-, p- MSR p- /MSE Error -p SSE MSE Total - SSTO 75

176 Tests of multple regresso coeffcets, ~,...,,...,,..., * /,...,, /,...,,..., * ] [ - SSEF] [SSER F* statstc: Test F :... R : : model vs full to testg the followg reduced equvalet Ths s ot 0 s,..., At least oe of : 0... :,,, 0, 0 0 p q p F MSE MSR F p SSE q p SSE SSE F F df F SSE F df R df H H p q p q p p q p p q q q q q q p q q a p q q 76

177 Example TLC data Testg whether age cotrbutes to the model after accoutg for heght ad sex: Ca do t - test or F - test : T - test : H 0 : 3 0 vs 3 0 Test statstc: t* b 3 / s{ b 3 } 0.05/ F - test : R : TLC F : TLC Test statstc: F* RR : F* heght 0 0 heght F0.95,,8 sex sex age MSR age heght, sex MSE heght, sex, age 3.5/ 37.40/8 4.;Cocluso : Fal to reject H

178 Example TLC data cot d Testg whether age ad sex cotrbute to the model after accoutg for heght: Oly F- test s approprate : R : TLC F : TLC Test statstc: F* RR : F* heght 0 0 heght sex age MSR sex, age heght MSE heght, sex, age F0.95,,8 3.3;Cocluso : Fal to reject H / 37.40/

179 Type I ad Type III sums of squares Type I sums of squares are sequetal ad order-depedet, they adjust oly for varables already the model I TLC example Type I SS SAS table are as follows: SSRheght, SSRsex heght, SSRage heght,sex If we had specfed the varable dfferet order the model statemet we would have gotte dfferet type I SS: for example model TLC= heght age sex leads to SSRheght, SSRage heght, SSRsex age,heght Type III sums of squares are order-depedet they adjust for all of the remag varables the model I TLC example the type III sums of squares are SSRheght sex,age, SSRsex age,heght, SSRage heght,sex 79

180 Coeffcets of partal determato Recall that coeffcet of multple determato R measures the proportoate reducto the varato of acheved by all varables Coeffcets of partal determato measure the cotrbuto of oe varable oce all the remag varables are already the model Model: R 0 SSE SSE, SSE The coeffcet of partal determato betwee ad gve that s the model measures: the relatve reducto SSE whe s added to the model that already cotas the relatve margal reducto the varato assocated wth whe s already the model SSR SSE 80

181 Coeffcets of partal determato cot d The coeffcet of partal determato betwee ad ca be terpreted as a coeffcet of smple determato betwee the resduals from the regresso of o ad the resduals from the regresso of o 8

182 Coeffcets of partal determato ad correlato Model: R k 0,,... k, k,... p SSR SSE,...,... k Coeffce tsof partal correlato : k... k, p, k, p k,...,... p p r k,,... k, k,... p / R k,,... k, k,... p I TLC example R age heght,sex=.5/38.9=0.04 r age heght,sex = 0.04 = 0. 8

183 Stadardzed multple regresso model It s desrable to be able to compare the magtude of the regresso coeffcets ad based o that to judge the relatve mportace of the predctor varables However the magtude of the regresso coeffcets depeds o the scale of ad hece rescalg s order Also, stadardzg the regresso coeffcets mproves umercal stablty 83

184 Correlato trasformato All ad varables are stadardzed by subtractg the sample mea ad dvdg by the sample stadard devato:,...,,, * * p k s s s s k k k k k k k 84

185 Stadardzed regresso model Regresso model terms of the stadardzed varables Note that there s o tercept ths model Coeffcets SRM are terpreted as estmated chage stadard devatos for oe stadard devato chage 0 * k * *, * * * *...,..., be show that It ca... p p k k p p p k s s 85

186 Polyomal regresso oe varable All varables are powers of the same varable Ths mples curvlear relatoshp betwee the predctor ad the respose whe p > 0...,... p p p p 86

187 Polyomal regresso: applcablty ad cauto Polyomal regresso models are useful whe the true curvlear fucto s deed polyomal, or whe polyomal s a good approxmato Extrapolato outsde of the rage of the data s dagerous, especally wth hgh-order polyomals Data that cossts of dstct values ca always be ftted perfectly wth a polyomal of degree - Polyomal regresso models may cota oe, two or more cotuous predctor varables ad each predctor may be preset at dfferet power Each predctor s ofte cetered to remove hgh correlato betwee lear ad quadratc terms e.g. usually hgh correlato exsts betwee ad 87

188 Polyomal regresso models for a sgle predctor varable 88

189 Polyomal models for a sgle predctor varable cot d x x x x x x x where : Thrd - order model where : - order model Secod

190 Secod-order polyomal regresso models wth predctors a coc secto defes ad where 0 0 x x x x x x E x x x x x x x x 90

191 Secod-order polyomal regresso models wth 3 predctors ad, where x x x x x x x x x x x x x x x 9

192 Herarchcal approach to fttg polyomal models Polyomal models are specal cases of MLR ad hece ca be ftted usg the usual approach If a hgher order term s preset a regresso model, the lower order terms eed also to be preset e.g. quadratc model should have a lear term It s usually of terest to test whether a smpler polyomal model adequately represets the data. Ths ca be acheved usg the extra sums of squares approach 9

193 Herarchcal approach to fttg polyomal models cot d For example cosder testg whether the cubc term s eeded a polyomalregresso model : Model: x 0 SSR SSR x x SSR x Tocheck whether Tocheck whether x 3 x SSR x 3 x, x 3 SSR x x, x 0 use F* MSE 3 SSR x, x 0 ad 0 use F* MSE x 93

194 Commets of polyomal regresso Expesve terms of degrees of freedom Collearty may stll exst ad the orthogoal polyomals ca be used Test of lower order effect s meagless whe a hgher order term s preset 94

195 Muscle mass example To explore the relatoshp betwee muscle mass ad age wome, a utrtost radomly selected 5 wome from each 0-year age group, begg wth age 40 ad edg wth age 79. s age ad s a measure of muscle mass Cosder polyomal regresso 95

196 Sterod example cot d We fd terms of cetered age = age : = R-squared =SSR/SSTO= 830.6/ = Test whether or ot there s a regresso relato α=.0. Ho: β = β =0 Ha: at least oe s ot zero TS: F*= MSR/MSE=595.3 / = 9.84 p-value <

197 Muscle mass example cotued 97

198 Muscle mass example cot d Test whether the quadratc term ca be dropped from the model R: = ß 0 + ß F: = ß 0 + ß + ß TS: F* = MSR /MSE, = 03.35/ = 3.54 RR: F* > F0.95,, 57 = 4.0, R model s adequate Express the ftted regresso fucto terms of age = ß 0 + ß age ß age We wat = ß 0* + ß * age + ß * age ß 0* = ß 0 -ß ß = * * = ß * = ß -ß = -.84-*0.05* = ß * = ß =

199 Lecture 8: Multcollearty ad teracto models 99

200 Multcollearty I MLR frequetly asked questos are:. What s the relatve mportace of the effects of dfferet predctor varables?. What s the magtude of the effect of a gve predctor varable o the respose varable 3. Ca ay predctor varable be dropped from the model because t has lttle or o effect o the respose varable? 4. Should ay predctor varable ot yet cluded the model be cosdered for cluso? These questos have easy aswers whe the predctor varables are ot tercorrelated correlated wth oe aother 00

201 Ucorrelated predctor varables Whe the predctors are ucorrelated the estmated effects are the same o matter what other varables are the model. Also SSR =SSR ad SSR =SSR That s, type III SS = type I SS So drect comparsos of stadardzed regresso coeffcets ad smple t-tests for each predctor varable ca help aswer questos through 4. 0

202 Effects of multcollearty Our ablty to obta a good ft, to estmate ad predct mea ad dvdual respose respectvely are ot hbted However estmated regresso coeffcets have large samplg varablty Commo terpretato of regresso coeffcets as the effect of oe predctor whle holdg the others costat may ot be realstc, sce we may ot be able to vary oe varable whle keepg aother varable hghly correlated wth t costat. 0

203 Body fat example from textbook Study of the relatoshp betwee amout of body fat ad several possble predctors: trceps skfold thckess, thgh crcumferece, ad mdarm crcumferece 3. r R =0.998 whe 3 regressed o ad Regresso coeffcets vary wdely accordg to what other predctors are the model. Varables model b b , ,,

204 Body fat example cot d Type I ad Type III SS ca be very dfferet: SSR =35.7, SSR =3.47. The terpretato ths example s that by tself trceps skfold thckess s a mportat predctor of body fat but t does ot add much ew formato after thgh crcumferece s accouted for ad vce versa 04

205 Body fat example cot d Multcollearty also affects stadard errors of the regresso coeffcets Varables model s{b } s{b } , ,,

206 Body fat example cot d Multcollearty does ot sgfcatly affect the ftted values ad predctos Varables model Ftted at =5.0 =50.0, 3 =9.0 s{ftted } at =5.0 =50.0, 3 = , ,,

207 Iteracto models wth quattatve varables } { } { Eg : teractos: M odel wth multplcatve effects } { :... : M odel wth addtve effects E E E Eg f f f E{} p p 07

208 Iterpretato of regresso coeffcets teracto models wth quattatve varables Model: 0 3 Itercept for the relatoshp betwee ad : Slope for the relatoshp betwee ad : 0 3 Itercept for the relatoshp betwee ad : Slope for the relatoshp betwee ad :

209 Example: } { } {. 5 0 } {. E c E b E a 09

210 Commets o teractve effects for quattatve varables Curvlear effects may be preset Sce some of the predctors may be hghly correlated wth the teracto terms, t s a good dea to ceter the predctor varables Whe the umber of predctor varables the regresso model s large, the potetal umber of teractos may be large. Ether a pror kowledge or resdual plots of ftted values based o the ma effects model vs teracto terms ca be used the to gude the choce of teracto terms to clude the model. F-tests based o extra sums of squares ca be used to test whether teracto effects are eeded the model. 0

211 Models wth qualtatve predctors Qualtatve predctors wth classes bary: Qualtatve predctors wth 3 or more classes categorcal omal: Nomal categorcal varable wth c classes s represeted by c- dcator dummy varables f f f f female male more tha hgh school educato otherwse less tha hgh school educato otherwse

212 Iterpretato of regresso coeffcets Example: - age, geder = f female β commo slope β 0 tercept for males β 0 + β tercept for females Frst-order model mples parallel regresso les at each level of the categorcal predctor E 0 0 for males ad E 0 for females

213 Iterpretato of regresso coeffcets cot d Example: - age, ad 3 educato level = f > HS, 3 = f < HS β commo slope β 0 tercept for HS β 0 + β tercept for > HS β 0 + β 3 tercept for < HS Frst-order model mples parallel regresso les at each level of the categorcal predctor HS for E HS for E HS for E

214 Cosderatos usg dcator varables Allocato codes may be used some cases stead of dcator varables: e.g. educato defed as + f >HS, 0 f HS, - f <HS ad the treatg the varable as cotuous for the purposes of regresso. That however mples a metrc whch may ot correspod to realty Sometmes quattatve varables may be used based o tervals defed for qualtatve varables Dfferet type of codg may be used: for example = f female, - f male Itercept may be dropped ad c dcator varables may be used for a categorcal varable wth c classes 4

215 Iteractos betwee qualtatve ad quattatve predctors 0 E 0 for males ad E 0 for females Ths mples that theregresso les for the twolevels of the quattatve varable tersect are ot parallel Testg 0 s equvalet to testg for addtve effects 5

216 Commets We ca have models wth ay combato of quattatve ad qualtatve varables If teractos are preset the correspodg ma effects should also be cluded 6

217 Comparso of two or more regresso fuctos Soap producto le example: amout of scrap le speed producto le f le, 0 f le Model: 0 Ftted model: ˆ

218 Soap producto example cot d 8

219 Soap producto example cot d Oe mportat assumpto to be able to estmate the regresso relatoshp for both producto les smultaeously s that the resdual varaces for the two producto les are the same Hece perform Brow-Forsythe test for equalty of varace 9

220 Soap producto example cot d 0

221 Soap producto example cot d H 0 Ha : uequal,95.0,045.8 s 7 s 4.39 t * BF RR : t Fal : Equal * BF to reject H varaces varaces 5 0 t0.975;5 for the producto les

222 Soap producto example cot d

223 Soap producto example cot d Test whether the two regresso les are detcal H H 0 a : 0 : at least oe of or s ot zero TS: F* SSR, SSE,, 4 RR : F* F0.99;, ,694 80/ Sce the example F* 9,904/3 Reject H that theles are detcal

224 Soap producto example cot d Test whether the slopes of the two regresso les are the same H H 0 a : 0 3 : 0 3 TS: F* RR : F* SSR, SSE,, 4 F0.99;, Sce the example F*.88 9,904/3 Fal to reject H that theles are parallel 0 4

225 Lecture 0: Model selecto ad valdato Appled Regresso Aalyss BIS63a Fall 005 Istructor: Raltza Gueorgueva 5

226 Model-buldg process Data collecto ad preparato Reducto of explaatory varables Model refemet ad selecto Model valdato 6

227 Data collecto ad preparato Data collecto vares by type of study: - Cotrolled expermets - Cotrolled expermets wth covarates - Cofrmatory observatoal studes - Exploratory observatoal studes - exclude exploratory varables ot fudametal to the problem - may exclude varables subject to large measuremet error - exclude duplcate varables Data preparato volves edt checks, plots to detfy gross errors Prelmary model vestgato: detfy fuctoal form of explaatory varables, mportat teractos, may rely o pror kowledge 7

228 Reducto of explaatory varables Omttg mportat varables may bas estmates ad may damage exploratory power of the model Overftted model may result large varaces of estmates of parameters Varable subset should be maageable ad large eough for adequate descrpto 8

229 Model refemet ad selecto Tetatve regresso model or several good regresso models eed to be checked for curvature ad teracto effects. Resdual plots, formal tests for volatos of assumptos ad lack of ft ca be used Remedal actos may eed to be appled 9

230 Model valdato Model valdty refers to: - the stablty ad reasoableess of the regresso coeffcets - usablty of the regresso fucto - ablty to geeralze fereces 30

231 Crtera for model selecto Examato of all models s vrtually mpossble For p- predctors there are p- possble models ot cosderg teractos or hgher order terms amog the predctors Model selecto procedures are used to detfy a R p, Ra, p, Cp, AIC p, SBC p, small group of regresso models that are good accordg to a specfed crtero These good models are the examed detal to come up wth best models We wll cosder 6 crtera: PRESS p 3

232 Notato ad assumptos Number of potetal varables: P- All models cota a tercept β 0 The umber of varables a subset s deoted by p-, p P Number of observatos s greater tha maxmum umber of potetal parameters >P It s hghly desrable that s much larger tha P >>P 3

233 R p or SSE p crtero R based o p parameters, p- varables The goal s to detfy models wth hgh coeffcet of determato R always creases as we add varables to the model but a small crease may ot be worthwhle R p - SSE p SSTO 33

234 R a,p or MSE p crtero R p does ot take to accout the umber of varables the model The adjusted coeffcet of determato R a,p s a better choce The adjusted coeffcet of determato does ot always crease It creases oly f MSE p decreases. R a, p p SSE p SSTO MSE p SSTO 34

235 Mallows C p crtero,... ˆ { ˆ} ˆ { ˆ} ˆ bas squared ˆ of varace ˆ { ˆ} ˆ ] ˆ ˆ ˆ [ ˆ ˆ ˆ ˆ ˆ ˆ respose: from mea ftted values Devatos of model subset regresso for each ftted values the squared error of the total mea to mmze : Goal p MSE SSE C μ E μ E μ E μ E μ E μ E E μ bas error radom μ E E μ μ P p p p 35

236 Mallows C p crtero cot d E C p whe E{Ŷ } p Models wthout bas wll fall ear the le C p =p Models above the le show substatal bas Models below the le are there due to radom error Hece we are lookg for values of C p that are small ad ear the le C p =p C P = P The choce of the P- potetal varables s very mportat sce o that choce depeds whether the MSE p s ubased estmator of the error varace 36

237 AIC p ad SBC p crtera Akake s formato crtero AIC p ad Schwartz Bayesa crtero SBC p also pealze for the umber of varables the model. We look for models wth small AIC p ad SBC p SBC p favors more parsmoous models AIC p l SSE p l p SBC p l SSE p l [l ] p 37

238 PRESS p crtero Predcto sum of squares crtero s a measure of how well the use of the ftted values for a subset model ca predct the observed resposes ˆ PRESS predcto error for case : ˆ ˆ predcted value for the case based o the ftted regresso le whe the case was deleted PRESS p ˆ ˆ Models wthsmall PRESS PRESS p p values are preferred values ca be calculated from a sgle regresso ru 38

239 Surgcal ut example A hosptal surgcal ut s terested predctg survval patets udergog a lver operato. A radom sample of 08 patets was avalable for aalyss. Potetal predctor varables: blood clottg score, progostc dex, ezyme fucto test score 3, lver fucto test score 4, age 5, geder 6, dcator varables for hstory of alcohol use 7, 8 Demostrates the utlty of the 6 dfferet crtera for model selecto graph o p

240 40