S136_Reviewer_002 Statistics 136 Introduction to Regression Analysis Reviewer for the 2 nd Long Examination

Transcription

1 w UP School of Statstcs Student Councl Educaton and Research erho.weebly.com 0 erhomyhero@gmal.com f /erhosmyhero S136_Revewer_00 Statstcs 136 Introducton to Regresson Analyss Revewer for the nd Long Examnaton Dummy Varables ncorporates qualtatve explanatory varable/s m categores m 1 dummy varables all that are left out are lumped nto a BASELINE category Interpretaton of regresson parameters Y = β 0 + β 1 X 1 + β X + ε X 1 = sze of frm X = { stock - 1 mutual - 0 Stock: Y = β 0 + β 1 X 1 + ε E(Y ) = β 0 + β 1 X 1 Mutual: Y = β 0 + β + β 1 X 1 + γ, γ s an error term E(Y ) = (β 0 + β ) + β 1 X 1 β ndcates how much hgher or lower the response functon for stock frm s than the one for mutual frm. β measures the dfferental effect of the type of frm. Interacton between a quanttatve regressor varable and a qualtatve regressor varable dfferent categores have dfferent slopes and ntercepts Note: All other models are compared wth the baselne when nterpretng ntercepts and slopes. Why not ft separate regressons for the dfferent categores?. The model assumes equal slopes and the same constant error varance σ for each category, so the common slope β 1 can be best estmated by poolng the categores.. Inferences pertanng to the parameters assocated wth the dummy varables and β 0 can be made more precsely by workng wth one regresson model contanng the dummy varable snce more degrees of freedom wll be assocated wth the mean square error. 1

2 Regme Swtchng The regresson functon of Y on X follows a partcular lnear relaton n some range of X, but follows a dfferent lnear relaton elsewhere. We can consder usng ndcator varables to ft pecewse lnear regresson. We take up the case where x p, the pont where the slope changes, s known. Dagnostc Checkng Resdual Plots 1. ^Y on horzontal axs () no rregularty () varance not constant weghted least squares or transformaton on the Y s needed () Systematc departure from ftted. Ths may be due to wrongly omttng β 0 from the model. (v) Model s nadequate or s nonlnear.

3 . Regressor on horzontal axs () no rregularty () varance not constant weghted least squares or prelmnary transformaton of Y s () Lnear effect of X not removed. Ths may be due to multcollnearty. (v) Need for extra terms 3. Tme on horzontal axs () no rregularty () varance not constant weghted least squares () need for a lnear term n tme n the model (v) need for lnear or quadratc terms n tme n the model Partal Regresson Plot/ Added varable plot Regress Y on all X s except X j Regress X j on all other X s Plot the resduals from these regressons The resultng plot s the partal regresson plot of Y aganst X j Partal Resdual Plot Estmated regresson functon: ^Y = β 0 + β ^ 1 X 1 + β ^ X + + β ^ k X k ( j) Y = e + ^β j X j s Y wth the lnear effects of the other varables removed. e = Y ^Y Y Y ( j) = e + ^Y = e + (β 0 + ^ β 1 X 1 + ^ β X + + ^ β k X k ) = e + (β 0 + β ^ 1 X 1 + β ^ X + + β ^ k X k ) ( β ^ β ^ j 1 X j 1 + β ^ j + 1 X j β ^ k X k ) Y ( j) = e + ^β j X j Example: Y () = e + β ^ X = e β ^ 0 β ^ 1 X 1 β ^ 3 X 3 β ^ k X k The lnear effects of the other X s except X were removed. The plot of Y ( j) aganst X j s called a partal resdual plot or component plus resdual plot On Nonlnearty Indvdual scatter plots between Y and one regressor are almost useless when several regressors are mportant or are correlated. 3

4 Resdual based plots (resdual plots, partal regresson plots, and partal resdual plots) are more relevant Lack-of-ft appears to be present f the resduals depart from 0 n a systematc fashon. Resdual plots ensure that the e s are lnearly uncorrelated wth each X thus, they can not dstngush between monotone and non-monotone nonlnearty Monotone nonlnearty frequently can be corrected by smple transformatons Partal regresson plots can reveal nonlnearty and suggest whether a relatonshp s monotone they cannot, however, specfy the type of transformaton useful n fndng outlers and nfluental observatons adjusts X j for the other X s, but t s the unadjusted X j that s transformed n re-specfyng the model Partal resdual plots an alternatve to partal regresson plots n revealng monotone nonlnearty and specfyng transformaton/s not as sutable as the partal regresson plot n detectng outlers and nfluental observatons Nonnormalty and Heteroskedastcty Valdaton of assumptons: ε N (0, σ ), Snce the ε s are unobservable, we look at the behavor of the resduals, snce the resduals are pseudo-estmates of the error terms (whch are unknown and are random varables). Recall: e s correlated wth Y e s uncorrelated wth ^Y On Nonnormalty Normal Probablty plot Tests 1. χ Goodness of Ft Test. Kolmogorov-Smrnov One Sample Test 4

5 3. Shapro-Wlk Test 4. Anderson-Darlng Test one of the most powerful tests for detectng departures from normalty 5. Cramer-Von-Mses Crteron 6. Jarque-Bera Test uses skewness and kurtoss On Heteroskedastcty Resdual plots are usually funnel-shaped or damond-shaped Reasons for Non-constancy of varances 1. Error-learnng models as people learn, ther error of behavor tends to decrease over tme; σ s expected to decrease. As ncome grows, people have more dscretonary ncome and hence, more scope of choce about the dsposton of ther ncome; σ s expected to ncrease wth ncome 3. As data collecton technques mprove, σ s expected to decrease. 4. Heteroskedastcty can also arse as a result of the presence of outlers 5. The regresson model s ncorrectly specfed. Implcatons of Heteroskedastcty Y = X β + ε, ε N n (0, V ), V σ I, V s p.d. 1. Ordnary Least Squares (OLS) estmators are stll lnear and unbased. ^β = ( X ' X) 1 X ' Y E( ^β ) = ( X ' X) 1 X ' E(Y ) = ( X ' X) 1 X ' X β E( ^β ) = β. OLS estmators are consstent. 3. OLS estmators are not effcent. The OLS estmators are no longer BLUE. Moreover, they are also not asymptotcally effcent, or they do not become effcent even f the sample sze ncreases. Var ( ^β ) = [(X ' X ) 1 X ' Y ] = (X ' X ) 1 X ' Var (Y ) X (X ' X ) 1 The varances of the OLS estmators are not provded by the usual OLS formula. Tests for Heteroskedastcty 1. Two-sample F-test Ft separate regressons to each half of the observatons arranged by the level of X Ho: σ 1 = σ Ha: σ 1 σ 5

6 (n 1 1)S 1 (n F c = 1)σ (n 1)S (n 1 1)σ 1 = S 1 S F (n 1 1, n 1) α / C.R.: Reject Ho f F c > F (n1 1, n 1). Goldfeld-Quandt Test 1 α / or F c < F (n1 1, n 1) a) Plot the resduals aganst each X j and choose the X j wth the most notceable relaton to error varaton (funnel-shaped or damond-shaped plot) b) Order the entre data set by magntude of the X j chosen n (a). c) Omt the mddle d observatons (d s chosen arbtrarly but usually taken to be 1/5 of the total observatons) d) Ft two separate regressons:. One for the porton of the data assocated wth low resdual varance.. One for the porton assocated wth hgh resdual varance. e) Calculate SSE 1 and SSE f) Calculate SSE 1 SSE. Assumng the error terms are normally dstrbuted, SSE 1 F SSE ( n d p n d p, ) g) Reject the Ho of constant varance f SSE 1 α > F SSE df for a specfed α and df = ( n d p, 3. Whte s Heteroskedastcty Test n d p ) Ho: Ha: varances are constant varances are not constant a) Run the regresson equaton and obtan the resduals, e. b) Regress e on the followng regressors: constant, orgnal regressors, ther squares, and cross-products (optonal). c) Obtan R and k, the number of regressors excludng the constant. d) Test stat: nr, n s the number of observatons nr χ k e) Choose α and reject Ho based on the p-value or C.R. 6

7 4. Breusch-Pagan Test Ho: Ha: varances are constant varances are not constant a) Run the regresson equaton and obtan the resduals, e. b) Regress e on the followng regressors: constant and ^Y c) Obtan R and k, the number of regressors excludng the constant. d) Test stat: nr, n s the number of observatons nr χ k e) Choose α and reject Ho based on the p-value or C.R. Remedes for Heteroskedastcty a) Transformaton of varables b) Generalzed Least Squares c) Weghted Least Squares Dualty of Nonnormalty and Heteroskedastcty a) Presence of one s usually assocated wth the presence of the other. b) Soluton for one usually solves the other. Transformatons a) To enhance lnearty b) To mprove data confguraton c) Note: A model s lnear f parameters appear as lnear components regardless of the complexty of the predctor varables. Autocorrelaton Usually occurs n tme seres data Reasons for occurrence 1. Adjacent resduals tend to be smlar n both temporal and spatal dmensons.. In tme seres economc data, resduals tend to be postvely correlated. 3. Large postve errors are followed by other postve errors and large negatve errors are followed by other negatve errors. 4. For cross secton data: Observatons sampled from adjacent expermental plots are correlated due to external factors. 5. An autocorrelated ndependent varable s deleted. 6. Seasonalty n ether or both the dependent and ndependent varables. Detecton Durbn-Watson Test 7

8 Recall: Y = X β + ε, ε NID(0, σ I) Possble autocorrelaton model: ε t = ρ ε t 1 + ω t, ω t N (0, σ ω ), ρ < 1 Hypotheses: Ho: ρ = 0 vs Ha: ρ 0 Test stat: d = n t = (e t e t 1 ) n e t t = 1 ^ρ = n e t e t 1 t = n e t t = 1 Decson crteron: For Ha: ρ > 0 d (1 ^ρ ) Range on d s from 0 to 4 ρ = 1 means d = 0and ρ = 0 means d s close to d < d L, Reject Ho d > d U, do not reject Ho d L < d < d U, test s nconclusve If correlaton s negatve, compute 4 d, smlar process above. Effects 1. OLS estmators of the regresson coeffcents are no longer effcent n the sense that they no longer have mnmum varance.. Estmate of σ and the standard errors of the regresson coeffcents may be serously understated, gvng a spurous mpresson of accuracy 3. The confdence ntervals and the varous tests of sgnfcance commonly employed would no longer be strctly vald. Remedes 1. Re-specfcaton f the cause of seral correlaton s ncorrect specfcaton, a re-specfcaton can remedy the problem ncorrect specfcaton mght come from deletng necessary ndependent varable/s. Generalzed Least Squares Model at tme t : y t = β 0 + β 1 X t + ε t (1) ε t = ρ ε t 1 + δ t, δ t N (0, σ ) At tme (t 1) : y t 1 = β 0 + β 1 X t 1 + ε t 1 Multply by ρ : ρ y t 1 = ρ β 0 + ρ β 1 X t 1 + ρ ε t 1 () Subtract () from (1) 8

9 y t ρ y t 1 * y t = β 0 ρ β 0 * β 0 X t * + β 1 ( X t ρ X t 1 ) + (ε ρ ε t 1 ) (3) δ t Equaton (3) s called the generalzed dfference equaton and can be expressed as: y t * = β 0 * + β 1 X t * + δ t, whch s free of the autocorrelaton problem. Problem of GLS: When ρ s unknown. 3. Cochrane-Orcutt Procedure Provdes an estmate for ρ for GLS Assume the model: y t = β 0 + β 1 X t + ε t [same as (1) above] a) Estmate the parameters usng OLS. Obtan the e s Multcollnearty b) ^ρ = n t = n t = e t 1 e t e t 1 * c) Usng ^ρ, regress y t = β * 0 + β 1 X * t + δ t to obtan β ^* 0, ^β 1 and β ^ 0 d) Calculate ^y t = β ^ 0 + β ^ 1 X t and get the new resduals a t = y t ^y t e) Go back to (b) and estmate a new ρ usng the new set of resduals. a) Contnue untl the dfference between two successve ρ s s small (0.01 or 0.005) If X s are lnearly dependent, rk( X ' X) < p and consequently, (X ' X ) 1 does not exst If there s near dependence, rk( X ' X ) s barely p and (X ' X ) 1 becomes unstable. Multcollnearty exsts when the jont assocaton of the ndependent varables affects the model process Parwse correlaton of ndependent varables wll not necessarly lead to multcollnearty. Absence of parwse correlaton wll not necessarly ndcate absence of multcollnearty. Jont correlaton of the ndependent varables wll not be a problem f t s weak to affect modelng Prmary Sources of the Problem 1. Data collecton method employed.. Constrants n the model or n the populaton. 3. Model specfcaton. 4. Over-defned model (overparametrzed) Multcollnearty s a data problem, not a statstcal problem Remarks on the Practcal Consequences of Multcollnearty 1. Larger varance and standard errors of the OLS estmators.. Wder confdence ntervals 3. Insgnfcant t ratos 4. Hgh R but few sgnfcant t ratos 5. Unstable (X ' X ) 1 [Ill-condtonng problem] 9

10 6. Wrong sgns for regresson coeffcents 7. Dffculty n assessng the ndvdual contrbutons of explanatory varables to the expected sum of squares or R Detecton and Analyss 1. Sgns of the coeffcents are reversed. Correlaton matrx (lmted though) 3. Varance Inflaton Factors (VIF) VIF j s the j th dagonal element of (X ' X ) 1. It ndcates whch term s much affected by multcollnearty. VIF >> 10 ndcates severe varance nflaton for the parameter estmator assocated wth X j. If X s centered, X ' X = R xx ( X ' X) 1 1 = R xx VIF j = 1 1 R, where R j = coeffcent of determnaton obtaned when regressng X j on the other j k 1 ndependent varables. Some software use 1, called the tolerance value, nstead of the VIF. Tolerance lmts frequently VIF used are 0.01, 0.001, VIFs cannot dstngush between several smultaneous multcollneartes. 4. Condton number 5. Suppose the egenvalues of X ' X are gven by λ 1, λ,, λ k. If λ max = max {λ } and λ mn = mn {λ }, then the condton number s k = λ max. Note that f there s ll-condtonng, some λ mn egenvalues of X ' X are near zero. Range of k k < 100 State of Multcollnearty no serous problem 100 k 1000 moderate to strong k > 1000 severe 6. Condton ndces May compute the rato of the square root of maxmum value to the square root of each of the other egenvalues. ( λ j λ max), j = 1,,, k s.t. λ j λ max Ths gves a clarfcaton as to whether one or several dependences are present among the X s. Indces greater than 30 could ndcate presence of dependences. Ths can help n formulatng a possble smultaneous system of equatons (other than theory of course!). 10

11 Outlers 7. Varance proportons A multcollnearty problem occurs when a component assocated wth a hgh condton ndex contrbutes strongly (varance proporton greater than about 0.5) to the varance of two or more varables. Remedes 1. Centerng of observatons (especally effectve f complex functons of X s are present n the desgn matrx). Deleton of unmportant varables (backward selecton) 3. Imposng constrants 4. Reparametrzaton (Prncpal Component Regresson) 5. Shrnkage Estmaton (Rdge Regresson) How are they generated? (Possble Sources) 1. Part of the populaton (hgh degree of stochastcty of the populaton).. Measurement problems. 3. Recordng/encodng problems. 4. Contamnaton/mxture populatons (large data sets). 5. Non-lnearty. There s the nevtable occurrence of mproperly recorded data, ether at ther source or n transcrpton to computer-readable form. Observatonal errors are often nherent n the data. Outlyng data ponts may be legtmately occurrng extreme observatons. Such data often contan valuable nformaton that mproves estmaton effcency by ts presence. Snce the data could have been generated by a model(s) other than that specfed, dagnostcs may reveal patterns suggestve of these alternatves. Detecton 1. Scatter plots or resdual plots can clearly reflect outlyng/nfluental observatons. Wth several varables smultaneously enterng nto the model, graphcal dsplays may not be suffcent.. Although much can be learned through such methods, they nevertheless fal to show us drectly what the estmated model would be f a subset of the data were modfed or set asde. 3. Even f we are able to detect suspcous observatons by these methods, we stll wll not know the extent to whch ther presence has affected the estmated coeffcents, standard errors, and test statstcs. 4. Influental Observaton: ts deleton sngly or n combnaton wth other observatons causes substantal changes n the ftted model (ncludng all statstcs). 5. Outlers: observatons dfferent from the rest. 6. Outlers n Dependent Varable: observatons wth large standardzed resduals; far from the rest of responses; observatons wth standardzed resduals that are standard devatons away from 0 are consdered outlers. e 7. Standardzed Resdual: MSE 11

12 8. Outlers n X: can also affect regresson results; measured n terms of leverage (weght, contrbuton through the desgn matrx) of the pont to the confguraton of the desgn matrx X. 9. Hat Matrx: Recall, H = X (X ' X) 1 X '. The th dagonal element of H s h =x ' (X ' X ) 1 x, where x corresponds to vector of measurements for the th sampled observaton. h s the LEVERAGE (n terms of the X values) of the th observaton to ts own ft h s n the range [0,1], where 1 hgh leverage, 0 low leverage h = p A large leverage value ndcates that the observaton s dstant from the center of the x observatons A leverage value s usually consdered to be large f t s more than twce as large as the mean leverage value; that s, leverage values greater than p/n are consdered by ths rule to ndcate outlyng observatons wth respect to the x values 10. Studentzed Resduals (Internal): s = MSE(1 h ) 11. Studentzed (Deleted) Resduals (External): e s * = MSE()(1 h ) where MSE() s computed from the model where th observaton was deleted. The studentzed deleted resdual, to be denoted by s * s s * n p 1 = e SSE(1 h ) e t (n p 1) based on n observatons To dentfy outlyng observatons, we examne the studentzed deleted resduals for large absolute values and use the approprate t-dstrbuton to ascertan how far n the tals such outlyng values fall. (usually, compare the t value to ) Measurng Influence 1. Cook s Dstance. DFFITS D = e p MSE[ e h ] (1 h ) F ( p, n p ) an overall measure of the mpact of the th observaton on the estmated regresson coeffcents measures the change n the parameter estmates caused by deletng each observaton DFFITS = D p MSE, where D MSE () s the th Cook s Dstance a scaled measure of the change n the predcted value for the th observaton and s calculated by deletng the th observaton. A large value ndcates that the observaton s very nfluental a general cutoff to consder s a sze-adjusted cutoff recommended s p n 1

13 3. DFBETAS DFBETAS j = ^β j ^β j () s() (X ' X ) jj where s() s the estmate of the error varance wthout the th observaton and (X ' X ) jj s the j th dagonal element of X ' X. the DFBETAS statstcs are the scaled measures of the change n each parameter estmate when the th observaton s deleted. n general, large values of DFBETAS ndcate observatons that are nfluental n estmatng a gven parameter a general cutoff value to ndcate nfluental observatons s and s the sze-adjusted cutoff n Effects 1. Dfference n some or all statstcs before and after an outler or nfluental observaton s removed.. Hgh leverage observatons should also be examned for nfluence to the general ft of the model. 3. Hgh leverage observatons may hde outlers. Ths s so because hgh leverage = good ft for the pont => small resdual (non-outler). 4. Outlers may be masked f other outlers also exst. Ths s so because presence of several outlers could ncrease standard devaton of resduals, hence, the 3 SD band could be wde enough. Solutons Outlers should not routnely be deleted or down-weghed because they are not necessarly bad. If they are corrected, they are the most nformatve ponts n the data. Do not drop t wthout any justfcaton. Why are they outlyng or nfluental? Correctve actons: correct the data deleton/down-weghng transformaton consder a dfferent model re-desgn experment/survey collect more data Procedure for f Outlers Frst, check f the extreme observatons are due to errors n computaton or some other explanable cause. If they are, you can drop these outlers; postulated model stll apples to the populaton as a whole. If they are not due to measurement error, ft a new regresson lne to the other (n-f) observatons (exclude the f outlers). The outlers that are not used n fttng the new regresson lne can now be regarded as new observatons (x 0, Y 0 ). For each outler, obtan a (1- )100% predcton nterval for Y h(new) gven x = x 0 from the new regresson lne and see f Y 0 les n the nterval. If Y 0 falls n the nterval, the outler s due to chance and the postulated model s stll vable. Outlers may or may not be retaned. If Y 0 does not fall n the nterval, some sutable restatement of the model s requred. Do not drop the outler. 13

14 Addtonal Notes 1. An outlyng nfluental observaton may be dscarded or deleted from the set of observatons f the crcumstances surroundng the data provde explanaton of the unusual observaton that ndcates an exceptonal stuaton not to be covered by the model.. If the outlyng nfluental observaton s accurate, t may not represent an unlkely event but rather a falure of the model. Often, dentfcaton of outlyng nfluental observatons leads to valuable nsghts for strengthenng the model. 3. To dampen the nfluence of an nfluental observaton, a dfferent method of estmaton may be used (e.g., method of least absolute devaton). 14