PubH 7405: REGRESSION ANALYSIS SLR: PARAMETER ESTIMATION

Transcription

1 PuH 745: REGRESSION ANALYSIS SLR: PARAMETER ESTIMATION

2 REGRESSION MODEL Model: Y + + ε where and are two new parameters called regresson coeffcents, the Intercept and the Slope, respectvel. The last term, ε, s the error representng the random fluctuaton of -values around ther mean, +, when X. The presence of the error term s an mportant characterstc of a statstcal relatonshp; the ponts on a scatter dagram do not fall perfectl on the lne. The scatter dagram s an useful dagnostc tool for checkng out the Model e.g. to see f t s lnear.

3 Y + ε N,σ + ε The "normal" assumpton can sometmes e weakenedto Eε and Varε σ The Normal Error Regresson Model

4 REGRESSION COEFFICIENTS The error term ε - wth varance σ -would tell how spread the dots are around the regresson lne. The regresson coeffcents, and, determne the poston of the lne and are mportant quanttes n the analss process. In correlaton analss, we need to know onl the coeffcent of correlaton r whch s proportonal to the slop we ll see; ut n a regresson analss, wth new emphass on predcton, so we need them oth, and. As parameters, oth and are unknown; ut the can e estmated statstcs from data

5 Y + ε N,σ + ε The Varance σ around the regresson lne s the thrd parameter : It s hdden, ut has a specfc role & ver mportant too!

6 THE INTERCEPT If the scope of the model nclude X, gves the Mean of Y when X ; otherwse, t does not have an partcular meanng as a separate term. If the scope of the model does not nclude X, we ma choose a transformaton such as: New - Under ths transformaton, gves the Mean of Y when X,.e. a tpcal suject value

7 * * * * * * * * * X Y E X Y E X Y E X Y E X X X X Y E X Y E + + d Model: Transforme Model: Orgnal

8 THE SLOPE The Slope s a more mportant parameter: If X s nar / representng an eposure, represents the ncrease n the mean of Y assocated wth the eposure or a decrease f s negatve; If X s on a contnuous scale, represents the ncrease n the mean of Y assocated wth one unt ncrease n the value of X, X+ vs. X, or a decrease f s negatve.

9 Bnar Independen t Varale X : EY X + EY X EY X + EY X - EY X The change n the mean of Y assocated wth the eposure

10 Contnuous Independen t Varale EY X EY X + EY X EY X + - EY X + X : The change n the mean of Y assocated wth one unt ncrease n the value of X

11 EXAMPLE For eample, let X e a mother s weght gan durng her pregnanc and Y the rth weght of the neworn. When X, the rth weghts BW of all nfants form certan normal dstruton. The Mean of that Normal Dstruton depends on the weght gan: Mean of BW Intercept + Slope The slope represents the average ncrease n rth weght for ever pound the mother ganed ; In ths case, slope>

12 The Regresson Model Y ε + N, σ + ε : THE NEED Use the "oserved data": to estmate parameters the "model" three :,, and σ {, } n

13 SUM OF SQUARED ERRORS B the Model, when X, the Mean of Y s +. Let and are estmates of and, respectvel; an estmate of + s consdered as a sample of sze. The error of that estmate s [ - + ] so that QΣ [ - + ] represents a form of the total errors not dstngushng an under-estmaton from an over-estmaton; called the sum of squared errors The method of least squares requres that we fnd good estmates of and the values of and so as to mnmze ths sum of squared devatons.

14 METHOD OF LEAST SQUARES PROCESS: We take the two partal dervatves of Q wth respect to and, set each equal to zero, and solve a sstem of two equatons for two unknowns and ; the solutons are and.

15 { }, : Data n n n Q Q δ δ δ δ n Q

16 Pont estmators/estmates Results n Equatons Normal : "page 7 : " called + + n n Q Q δ δ δ δ

17 RESULTS Gven the estmates of the Intercept and of the Slope, Estmate of for the mean or a new value of X s Ŷ + ; ths s called ftted value n n, The Least Squares Estmates are:

18 ] ][ [ r

19 r s s s r s s s s From ths smple result, we can see that and r are of the same sgn and oth are equal to zero at the same tme; the measure the same thng ut on dfferent scale.

20 Model: Y ε + N, σ Method: Mnmze Q Note : We do not needthe "normal" assumpton to otan the estmates + ε and However, later, we do needt for nferences concernng the parameters n,, σ

21 n n EXAMPLE #, Totals the estmates of the Slope and the Intercept are: For eample, for new suject wth X5, t s predcted that ts -value would e:

22 EXAMPLE #:, ,3,7 3,56 975, Brth weght data: oz % Note:f the rth weght s 95 ounces, t s predcted that the ncrease etween das 7 & would e %

23 Age SBP EXAMPLE #: Age and SBP 984,93 46, ,954 5, Note: for a 6-ear-old woman, t s predcted that her sstolc lood pressure would e mmhg.

24 EXAMPLE #3: Toluca Compan Data LotSze WorkHours Descrpton on page 9 of Tet Suppose we are nterested n the mean numer of work hours requred when the lot sze s X 65; our pont estmate s: hours See tetook, page

25 SCOPE OF THE MODEL In formulatng a regresson model, we need to restrct the coverage of the model to some nterval of values of the ndependent varale X; ths s determned ether the desgn or the avalalt of data at hand. The shape of the regresson functon outsde ths range would e n dout ecause the nvestgaton provded no evdence as to the nature of the statstcal relaton outsde ths range. In short, one should not do an etrapolaton.

26 Oserved SUM OF SQUARED ERRORS QΣ [ - + ] s the sum of squared errors Snce + s an estmate of the mean of Y, e [ - + ] represents the error of our predcton; so that SSE Σe Σ [ - + ] s the oserved sum of squared errors ver much lke the numerator of the sample varance s.

27 ESTIMATING THE VARIANCE In the Regresson Model, the error term ε s assumed to have a Normal Dstruton wth mean and varance σ. ε s lke a varale of whch we have a sample wth sample mean zero: {e };,,n Varance σ s estmated MSESSE/n-; two degrees of freedom were lost due to the need to estmate the ntercept and slope.

28 ^ e + MSE e σ ^ n

29 CHARACTERISTICS OF PREDICTION ERRORS e e n n Q Q δ δ δ δ

30 INTERPRETATION e e Average error s zero, Error & Predctor are not correlated 3 As a result of, error and ftted value are not correlated

31 lne regresson the around wth and on scatter dagram form a Dots : ] ][ [ constant wdth n Implcato n e e n n e e r e e

32 UNBIASED ESTIMATES E E E MSE σ The are correct on the average; we ll prove at least the frst two - later

33 MORE ON THE SLOPE Data ponts wth -values at oth ends are nfluental

34 } { Var Var σ

35 s n MSE MSE SE MSE Var ^ σ

36 } { n n Var Var Var σ σ σ

37 MSE SE + n s

38 DESIGN IMPLICATION + n σ σ σ σ These varances, for gven n and σ, are affected the spacng of the X s levels n the data. The larger the sum of squares of X, the more precse the estmates of the Slope and the Intercept.

39 FITTED VALUE & RESIDUAL From the model ^ e + Resdual + : ^ Ftted value : :

40 Across the sample, we have: Var Y ^ s r s s s r s

41 r r Result : + ^ ^ s r s r s Y Var Y Var e Var e Var Y Var Y Var

42 Recall that, across the sample, we have: ^ Y Var r s r s s s r s Y Var VarŶ s the eplaned varance; so r s the fracton or proporton of the total varance that s eplaned the regresson. We call t Coeffcent of Determnaton.

43 Besdes Least Squares, parameters can e estmated usng the method of Mamum Lkelhood ; results are called MLE mamum lkelhood estmators/estmates.

44 MAXIMUM LIKELIHOOD ESTIMATION Suppose that we can assume a parametrc Model for the Dependent Varale Y whch s characterze a Denst Functon ft; θ sa, normal dstruton - nvolvng a parameter or parameters θ whch s fed ut unknown. Gven a random sample {}n; the Lkelhood Functon for θ s gven : L Πf; θ, and data can e analzed standard methods assocated wth largesample Mamum Lkelhood Theor Mamum Lkelhood Estmator- MLE- and ts asmptotc normalt, Score statstc, Lkelhood Rato statstc

45 + + / ep f for Y : Functon Denst, : Model N Y σ πσ σ ε ε

46 n n / n / / ep ep L : Functon Lkelhood ep f for Y : Denst Functon σ πσ σ πσ σ πσ

47 RESULTING MLEs The mamum lkelhood estmates of the Intercept & Slope are dentcal to the Least Squares estmates. The Varance Estmate s slghtl dfference; snce the MLE varance estmator s ased, we prefer and use the Least Squares estmator MSE. The MLE varance estmator s: ^ n n n MSE

48 INTERCEPT & SLOPE Snce the MLEs of Intercept and Slope are the same as the least squares estmates, the have the propertes of least squares estmates: the are unased, and the are mnmum varance unased that s, the have mnmum varance n the class of all unased estmators. In addton, as MLEs for the normal error regresson model: 3 the are consstent, and 4 the are suffcent.

49 Readngs & Eercses Readngs: A thorough readng of the tet s Chapter s hghl recommended. Eercses: The followng eercses are good for practce, all from chapter of tet:.9,.,.,.,.7,.3, and.35.

50 Due As Homework #5. We have a data set on 86 smokers Fle: Cgarettes; three outcome or response varales are Caron monode, Cotnne a dervatve of Ncotne, and NNAL a dervatve of NNN, a ton onl comes from toacco products. Data for 3 other eplanator varales are also ncluded: Age, Gender female, and Cgarettes per Da CPD. Let Y lognnal & XCPD: a Otan Least Squares estmates of and, then state/epress the estmated regresson functon.e. the mean of the dependent varale, the ftted value. Plot the estmated regresson functon on the same plot wth our scatter dagram; does the lnear relatonshp appear to ft the data? Does plot support the antcpaton that the average urne lognnal ncreases wth ncreasng CPD? Is the lnear relatonshp strong? c Gve an estmate of mean NNAL when CPA 3. d What s the pont estmate of the change n the mean lognnal when CPD ncreases cg? cgs? e Does an data pont appear to e out of ts place?

51 #5. It has een generall known that resprator functon ma declne wth age. To stud ths posslt, We consder a data set consstng of age ears and vtal capact VC, lters for each of 44 men workng n the cadmum ndustr ut have not een eposed to cadmum fumes Fle: Vtal Capact. Let X Age and Y Vtal Capact: a Otan Least Squares estmates of and, then state/epress the estmated mean of the dependent varale. Plot the estmated regresson functon on the same plot wth our scatter dagram; does the lnear relatonshp appear to ft the data? Does plot support the antcpaton that the average vtal capact decreases wth ncreasng Age? Is the lnear relatonshp strong? c Gve an estmate of mean EY when Age 35 ears. d What s the pont estmate of the change n the mean EY when Age ncreases ear? ears? e What would e the values of the Intercept, Slope, and MSE f Vtal Capact s use as the dependent varale nstead of Y; not run the computer program wth the new response varale