Problem of Estimation. Ordinary Least Squares (OLS) Ordinary Least Squares Method. Basic Econometrics in Transportation. Bivariate Regression Analysis

Transcription

1 1/60 Problem of Estmaton Basc Econometrcs n Transportaton Bvarate Regresson Analyss Amr Samm Cvl Engneerng Department Sharf Unversty of Technology Ordnary Least Squares (OLS) Maxmum Lkelhood (ML) Generally, OLS s used extensvely n regresson analyss: It s ntutvely appealng Mathematcally much smpler than the MLE. Both methods generally gve smlar results n the lnear regresson context. Prmary Source: Basc Econometrcs (Gujarat) 2/60 3/60 Ordnary Least Squares Method To odetermne e PRF: Y 1 2 Populaton Regresson Functon We estmate t from the SRF: Sample Regresson Functon X u We would lke to determne the SRF n a way that t s as close as possble to the actual Y. Y ˆ ˆ 1 2 X.e. sum of the resduals û s as small as possble. uˆ Ordnary Least Squares Method 2 uˆ f ( ˆ, ˆ 1 2) OLS fnds unque estmates of β 1 and β 2 that gve the smallest possble value of the functon above. ˆ 2 ˆ Y ˆ 2 X 1 ( X X )( Y Y ) 2 ( X X ) x y 2 x Why square? More weght to large resduals. Sgn of the resduals. Devaton form: y Y Y x X X

2 4/60 5/60 OLS Propertes: Sample Mean OLS Propertes: Lnearty Regresson lne passes through the sample mean Lnearty It s a lnear functon (a weghted average) of Y. X and thus k are nonstochastc and: Note: 6/60 7/60 OLS Propertes: Unbasedness OLS Propertes: Mean of Estmated Y Unbasedness ess Mean value of the estmated Y s equal to the mean value of the actual Y: Sum both sdes over the sample values and dvde by sample sze

3 8/60 9/60 OLS Propertes: Mean of Resduals OLS Propertes: Uncorrelated Resduals Y The mean value of the resduals s zero: The resduals are uncorrelated wth the predcted Y : 10/60 11/60 OLS Propertes: Uncorrelated Resduals X OLS Assumptons The resduals s are uncorrelated wth X : Our objectve s not only to estmate some coeffcents c e but also to draw nferences about the true coeffcents. Thus certan assumptons are made about the manner n whch Y are generated. Y = β 1 + β 2 X + u Unless we are specfc about how X and u are created, there s no way we can make any statstcal nference about the Y, β 1 and β 2. The Gaussan standard, or classcal lnear regresson model (CLRM), makes 10 assumptons that are extremely crtcal to the vald nterpretaton of the regresson estmates.

4 12/60 13/60 Assumpton 1 Assumpton 2 The regresson model s lnear n the parameters. E(Y X ) = β 1 + β 2 X 2 s a lnear model. E(Y X ) = β 1 + β 22 X s not a lnear model. X values are fxed n repeated samplng. X s assumed to be non-stochastc. Our regresson analyss s condtonal regresson analyss, that s, condtonal on the gven values of the regressor(s) X. 14/60 15/60 Assumpton 3 Assumpton 4 Zero mean value of dsturbance u E(u X ) = 0 Homoscedastcty oscedast c ty or equal varance a of u Factors not explctly ncluded n the model, and therefore subsumed n u, do not systematcally affect the mean value of Y.

5 16/60 17/60 Assumpton 4 Assumpton 5 Heteroscedastcty No autocorrelaton (seral correlaton) between the dsturbances. 18/60 19/60 Assumpton 6 Assumpton 7 Dsturbance u and dexplanatory ato varable ab X are uncorrelated. n must be greater than the number of explanatory ato varables. ab Obvously, we need at least two pars of observatons to estmate the two unknowns! If X and u are correlated, ther ndvdual effects on Y may not be assessed.

6 20/60 21/60 Assumpton 8 Assumpton 9 Varablty n X values. Mathematcally, f all the X values are dentcal, t mpossble to estmate β 2 (the denomnator wll be zero) and therefore β 1. Intutvely, t s obvous as well. The regresson model s correctly specfed. Important questons that arse n the specfcaton of a model: What varables should be ncluded n the model? What s the functonal form of the model? Is t lnear n the parameters, the varables, or both? What are the probablstc assumptons made about the Y, the X, and the u enterng the model? 22/60 23/60 Assumpton 10 Assumptons There e s no perfect ect multcollnearty. ty. No perfect lnear relatonshps among the explanatory varables. Wll be further dscussed n multple regresson models. How realstc are all these assumptons? s? We make certan assumptons because they facltate the study, not because they are realstc. Consequences of volaton of CLRM assumptons wll be examned later. We wll look nto: Precson of OLS estmates, and Statstcal propertes of OLS.

7 24/60 25/60 Precson of OLS Estmates Standard Error of Precson of an estmate s measured by ts standard error. Precson of an estmate s measured by ts standard error. 26/60 27/60 Homoscedastc Varance of u Features of the Varances How to estmate 2 The varance a of s drectly proportonal o to but nversely proportonal to sum of. As n ncreases, the precson wth whch β 2 can be estmated also ncreases. If there s substantal varaton n X, β 2 can be measured more accurately. The varance of s drectly proportonal to σ 2 and X 2, but nversely proportonal to x 2 2 and the sample sze n. Note:. If the slope coeffcent s overestmated, the ntercept wll be underestmated.

8 28/60 29/60 Gauss Markov Theorem Goodness of Ft Gven the assumptons of the classcal lnear regresson model, the OLS estmators, n the class of unbased lnear estmators, have mnmum varance, that s, they are BLUE. The coeffcent of determnaton R 2 s a summary measure that tells how well the sample regresson lne fts the data. The theorem makes no assumptons about the probablty dstrbuton of u, and therefore of Y. Total Sum of Squares (TSS) = Explaned Sum of Squares (ESS) + Resdual Sum of Squares (RSS) 30/60 31/60 Consstency Before Hypothess Testng An asymptotc property. An estmator s consstent f t s unbased and ts varance tends to zero as the sample sze n tends to nfnty. Unbasedness s already proved. Usng the method of OLS we can estmate β 2 1, β 2, and σ. Estmators ( ) are random varables. To draw nferences about PRF, we must fnd out how close s to the true. We need to fnd out PDF of the estmators.

9 32/60 33/60 Probablty Dstrbuton of Dsturbances Why the Normalty Assumpton? s ultmately a lnear functon of the random varable u, whch s random by assumpton. The nature of the probablty dstrbuton of u plays an extremely mportant role n hypothess testng. It s usually assumed that u NID(0, σ 2 ) NID: Normally and Independently Dstrbuted. We hope that the nfluence of these omtted or neglected varables s small and at best random. We show by the CLT, that f there are a large number of IID random varables, the dstrbuton of ther sum tends to a normal dstrbuton as the number of such varables ncrease ndefntely. Central lmt theorem (CLT): Let X 1, X 2,..., X n denote n ndependent random varables, all of whch have the same PDF wth mean = μ and varance = σ 2. Let : 34/60 35/60 Why the Normalty Assumpton? Normalty Test A varant a of the CLT states that, even f the number of varables ab s not very large or f these varables are not strctly ndependent, ther sum may stll be normally dstrbuted. Wth ths assumpton, PDF of OLS estmators can be easly derved, as any lnear functon of normally dstrbuted varables s tself normally dstrbuted. The normal dstrbuton s a comparatvely smple dstrbuton nvolvng only two parameters. It enables us to use the t, F, and χ 2 tests for regresson models. The normalty assumpton plays a crtcal role for small sample sze data. In reasonably large sample sze, we may relax the normalty assumpton. Snce we are mposng the normalty assumpton, t behooves us to fnd out n practcal applcatons nvolvng small sample sze data whether the normalty assumpton s approprate. Later, we wll ntroduces some tests to do just that. We wll come across stuatons where the normalty assumpton may be napproprate. Untl then we wll contnue wth the normalty assumpton.

10 36/60 37/60 Estmators Propertes wth Normalty Assumpton Method of Maxmum Lkelhood They are unbased, wth mnmum varance (effcent), and consstent.,, and s dstrbuted as the χ 2 wth (n 2) df. Wll help us to draw nferences about the true σ 2 from the estmated σ 2.. The mportance of ths wll be explaned later. Estmators have mnmum varance n the entre class of unbased estmators, whether lnear or not. Best Unbased Estmators (BUE) If u are assumed to be normally dstrbuted, the ML and OLS estmators of the regresson coeffcents, are dentcal. The ML estmator of σ 2 s based for small sample sze. Asymptotcally, the ML estmator of σ 2 s unbased. ML method can be appled to regresson models that are nonlnear n the parameters, n whch OLS s generally not used. 38/60 39/60 ML Estmaton ML Estmaton Assume ethe etwova two-varable ab model Y = β 1 + β 2 X + u n whch Havng ndependent Y s, jont PDF of Y 1,...,Y n, can be wrtten as: The method of maxmum lkelhood conssts sts n estmatng the unknown parameters n such a manner that the probablty of observng the gven Y s s as hgh as possble. Where: β 1, β 2, and σ 2 are unknowns n lkelhood functon:

11 40/60 41/60 ML Estmaton Interval Estmaton From the frst-order condton for optmzaton: How relable are the pont estmates? We try to fnd out two postve numbers δ and α, such that: Note how ML underestmates the true σ 2 n small samples. Probablty of constructng an nterval that contans β 2 s 1 α. Such an nterval s known as a confdence nterval. α (0 < α < 1) s known as the level of sgnfcance. How are the confdence ntervals constructed? If the probablty dstrbutons of the estmators are known, the task of constructng confdence ntervals s a smple one. 42/60 43/60 Confdence Intervals for Confdence Intervals for It tcan beshown that atthe ett varable ab follows ows the t dstrbuton wth n 2 df. It can be shown that under the normalty assumpton, followng ow varable follows χ 2 dstrbuton wth n 2 df. Wdth of the confdence nterval s proportonal to the standard error of the estmator. Same for 1 Interpretaton of ths nterval: If we establsh 95% confdence lmts on σ 2 and f we mantan a pror that these lmts wll nclude true σ 2,we shall be rght n the long run 95 percent of the tme.

12 44/60 45/60 Hypothess Testng Confdence Interval Approach Is a gven observaton compatble wth some stated hypothess? In statstcs, the stated hypothess s known as the null hypothess or H 0 (versus an alternatve hypothess or H 1 ). Hypothess testng s developng rules for rejectng or acceptng the null hypothess. Confdence nterval approach, Test of sgnfcance approach. Most of the statstcal hypotheses of our nterest make statements about one or more values of the parameters of some assumed probablty dstrbuton such as the normal, F, t, or χ 2. Decson Rule: Construct a 100(1 α)% confdence nterval for β 2. If the β 2 under H 0 falls wthn ths nterval, do not reject H 0, but f t falls outsde ths nterval, reject H 0. Note: There s a 100α percent chance of commttng a Type I error. If α = 0.05, there s a 5 percent chance that we could reject the null hypothess even though t s true. When we reject the null hypothess, we say that our fndng s statstcally sgnfcant. One-tal or two-tal test: Sometmes we have a strong expectaton that the alternatve hypothess s onesded rather than two-sded. 46/60 47/60 Test of Sgnfcance Approach Practcal Aspects In the econfdence confdence-nterval nterval procedure e we try to establsh s a range that has a probablty of ncludng the true but unknown β 2. In the test-of-sgnfcance approach we hypothesze some value for β 2 and try to see whether the estmated β 2 les wthn confdence lmts around the hypotheszed value. A large t value wll be evdence aganst the null hypothess. Acceptng null hypothess: s: All we can say: based on the sample evdence we have no reason to reject t. Another null hypothess may be equally compatble wth the data. 2-t rule of thumb If df >20 and α = 0.05, then the null hypothess β 2 = 0 can be rejected f t > 2. In these cases we do not even have to refer to the t table to assess the sgnfcance of the estmated slope coeffcent. Formng the null hypotheses Theoretcal expectatons or pror emprcal work can be reled upon to formulate hypotheses. p value The lowest sgnfcance level at whch a null hypothess can be rejected.

13 48/60 49/60 Analyss of Varance Analyss of Varance A study of two components of TSS (= ESS + RSS) s known as ANOVA from the regresson vewpont. If we assume that the dsturbances u are normally dstrbuted, whch we do under the CNLRM, and f the null hypothess (H 0 ) s that β 2 = 0, then t can be shown that the F follows the F dstrbuton wth 1 df n the numerator and (n 2) df n the denomnator. What use can be made of the precedng F rato? 50/60 51/60 F-rato Example It tcan beshown that Compute F rato and obtan p value of the computed F statstc. Note that β 2 2 and σ are the true parameters. If β 2 s zero, both equatons provde us wth dentcal estmates of true σ 2. Thus, X has no lnear nfluence on Y. F rato provdes a test of the null hypothess H 0 : β 2 = 0. p value of ths F statstc wth 1 and 8 df s Therefore, f we reject the null hypothess, the probablty of commttng a Type I error s very small.

14 52/60 53/60 Applcaton Of Regresson Analyss Mean Predcton One use s to predct or forecast the future consumpton expendture Y correspondng to some gven level of ncome X. Now there are two knds of predctons: Predcton of the condtonal mean value of Y correspondng to a chosen X. Predcton of an ndvdual Y value correspondng to a chosen X. Estmator of E(Y X 0 ): It can be shown that: Ths statstc follows the t dstrbuton wth n 2 df and may be used to derve confdence ntervals 54/60 55/60 Indvdual Predcton Confdence Bands Estmator of E(Y ( X 0 0): It can be shown that: Ths statstc follows the t dstrbuton wth n 2 df and may be used to derve confdence ntervals

15 56/60 57/60 Indvdual Versus Mean Predcton Reportng the Results Confdence nterval for ndvdual Y 0 s wder than that for the mean value of Y 0. The wdth of confdence bands s smallest when X 0 = X. 58/60 59/60 Evaluatng the Results Normalty Tests How good s the ftted model? Any standard? d? Are the sgns of estmated coeffcents n accordance wth theoretcal or pror expectatons? How well does the model explan varaton n Y? One can use r 2 Does the model satsfes the assumptons of CNLRM? For now, we would lke to check the normalty of the dsturbance term. Recall that the t and F tests requre that the error term follow the normal dstrbuton. Several tests n the lterature. We look at: Hstogram of resduals: A smple graphc devce to learn about the shape of the PDF. Horzontal axs: the values of OLS resduals are dvded nto sutable ntervals. Vertcal axs: erect rectangles equal n heght to the frequency n that nterval. From a normal populaton we wll get a bell-shape PDF. Normal probablty yp plot (NPP): A smple graphc devce. Horzontal axs: plot values of OLS resduals, Vertcal axs: show expected value of varable f t were normally dstrbuted. From a normal populaton we wll get a straght lne. The Jarque Bera test. An asymptotc test, wth ch-squared dstrbuton and 2 df:

16 60/60 Homework 2 Basc Econometrcs (Gujarat, 2003) 1. Chapter 3, Problem 21 [10 ponts] 2. Chapter 3, Problem 23 [30 ponts] 3. Chapter 5, Problem 9 [30 ponts] 4. Chapter 5, Problem 19 [30 ponts] Assgnment weght factor = 1