Amherst College Department of Economics Economics 60 Fall 2012 Wednesday, September 26 Handout: Estimating the Variance of an Estimate s Probability Distribution Preview: Review: Ordinary Least Squares (OLS) Estimation Procedure o General Properties of the Ordinary Least Squares (OLS) Estimation Procedure o Importance of the Coefficient Estimate s Probability Distribution Mean (Center) of the Coefficient Estimate s Probability Distribution Variance (Spread) of the Coefficient Estimate s Probability Distribution Estimating the Variance of the Coefficient Estimate s Probability Distribution o Step 1: Estimate the Variance of the Error erm s Probability Distribution First Attempt: Variance of the Error erm s Numerical Values Second Attempt: Variance of the Residual s Numerical Values hird Attempt: Adjusted Variance of the Residual s Numerical Values o Step 2: Use the Estimated Variance of the Error erm s Probability Distribution to Estimate the Variance of the Coefficient Estimate s Probability Distribution Degrees of Freedom Summary: he Ordinary Least Squares (OLS) Estimation Procedure o hree Important Parts Value of the Coefficient Variance of the Error erm s Probability Distribution Variance of the Coefficient Estimate s Probability Distribution o Regression Printouts Review: General Properties of the Ordinary Least Squares (OLS) Estimation Procedure When the standard ordinary least squares premises are met, the following equations describe the coefficient estimate s general properties, the estimate s probability distribution: Var[e] Mean[b x ] β x Var[b x ] Σ t1 (xt x ) 2 Importance of the Probability Distribution s Mean (Center) and Variance (Spread) Mean: When the mean of the estimate s probability distribution, Mean[b x ], equals the actual value of the coefficient, β x, the estimation procedure is unbiased. he estimation procedure does not systematically underestimate or overestimate the actual value. Variance: When the estimation procedure is unbiased, the variance of the estimate s probability distribution, Var[b x ], determines the reliability of the estimate. As the variance decreases, the probability distribution becomes more tightly cropped around the actual value making it more likely for the coefficient estimate to be close to the actual value. Var[e] Mean[b x ] β x Var[b x ] Σ t1 (xt x ) 2 Estimation Procedure Determines the Reliability As Var[b x ] Decreases Is Unbiased of the Estimate Reliability of b x Increases he Problem: But there is a problem here, isn t there?
2 We need to know the variance of the error term s probability distribution to calculate the variance of the coefficient estimate s probability distribution. Unfortunately, the variance of the error term s probability distribution is unobservable. In reality, we can never know the actual variance of the error term s probability distribution. How can Clint proceed? Econometrician s Philosophy: If you lack the information to determine the value directly, estimate the value to the best of your ability using the information you do have. Estimating the Variance of the Coefficient Estimate s Probability Distribution Strategy Step 1: Estimate the variance of the error term s Step 2: Apply the relationship between the probability distribution from the available variances of the coefficient estimate s and information information from the first quiz: the error term s probability distributions: Var[e] EstVar[e] Var[b x ] Σ t1 (xt x ) 2 EstVar[e] EstVar[b x ] Σ t1 (xt x ) 2 wo Steps Step 1: Estimate of the variance of the error term s probability distribution. Step 2: Use the estimate of the variance of the error term s probability distribution to estimate the variance of the coefficient estimate s probability distribution. Step 1: Estimating the Variance of the Error erm s Probability Distribution We will now describe three attempts to estimate the variance using the results of Professor Lord s first quiz by calculating the: 1. Variance of the error term s numerical values from the first quiz. 2. Variance of the residual s numerical values from the first quiz. Adjusted variance of the residual s numerical values from the first quiz. We shall use simulations to assess these attempts by exploiting the relative frequency interpretation of probability: Relative Frequency Interpretation of Probability: After many, many repetitions of the experiment, the distribution of the numerical values from the experiments mirrors the random variable s probability distribution; the two distributions are identical: Applying this to the variance Distribution of the Numerical Values Variance of the Numerical Values After many, many repetitions Probability Distribution Variance of the Probability Distribution Preview: While the first two attempts fail for different reasons, they provide the motivation for the third attempt which succeeds. herefore, it is useful to explore the first two attempts even though they will fail.
Estimating the Variance of the Error erm s Probability Distribution, Var[e] First Attempt Strategy: Use the variance of the three error terms numerical values from the first quiz, Var[e 1, e 2, and e 1 st Quiz], to estimate the variance of the error term s probability distribution, Var[e]. First Quiz: Variance of Error erms Numerical Values. Var[e 1, e 2, and e 1 st Quiz] o Estimate Variance of Error erm s Probability Distribution, Var[e] Using the regression model, solve for the error term: y t β Const + β x x t + e t e t y t (β Const + β x x t ) Recall that the variance is the average of the squared deviations from the mean: Compute the deviations from the mean; Square the deviations; Calculate the average of the squared deviations. We now can calculate the variance of the error terms numerical values from the first quiz: Var[e 1, e 2, and e 1 st Quiz] (e 1 - Mean[e])2 + (e 2 - Mean[e]) 2 + (e - Mean[e]) 2 Since the error term reflects random influences: Mean[e]. First Quiz β Const 50 β x 2 e t y t (β Const + β x x t ) Student x t y t β Const + β x x t 50 + 2x t e t y t (50 + 2x t ) 2 e t 1 5 66 50 + 2 2 15 87 50 + 2 25 90 50 + 2 Var[e 1, e 2, and e 1 st Quiz] e 2 1 + e 2 2 + e 2 1 st Quiz SSE 1st Quiz SSE Question: As a consequence of random influences, can we expect variance of the numerical values from one repetition, the first quiz to equal the actual variance of the coefficient estimate s probability distribution? Answer: Question: What can we hope for then? Answer:
4 Econometrics Lab Simulation Unbiased Estimation Procedure: After many, many repetitions of the experiment the average (mean) of the estimates equals the actual value. Mean (average) of the error term s numerical values from all repetitions. Sum of Squared Residuals Sum of Squared Errors Divide by sample size or degrees of freedom? Use errors or residuals? Estimate of the variance for the error s term probability distribution calculated from this repetition, EstVar[e] Repetition Error erms Mean Var SSE Divide by 2 Use Err Res Error Var Est Mean Act Err Var 200 50 500 Does the error term represent a random influence? Actual Variance of Error erm s Probability Distribution, Var[e] Does the simulation represent the variance of the error term s probability distribution accurately? Variance of the error term s numerical values from all repetitions. Is the estimation procedure for the variance of the error term s probability distribution unbiased? Average of the variance estimates from all repetitions. Estimate for the Variance Actual Value of the Error erm s Var[e] Repetition e 1 e 2 e SSE Probability Distribution 500 1 500 2 500 Observations Can we expect the estimate to equal the actual value?. In fact, we all but certain that the estimate will not equal the actual value. Sometimes the estimate is than the actual value and sometimes it is.
5 Question: What is the best we can hope for? An unbiased estimation procedure does not systematically underestimate or overestimate the actual value. When the experiment is repeated many, many times, the average of the numerical values of the estimates will equal the actual value. Question: How can we determine whether or not the estimation procedure for variance of the error term s probability distribution unbiased? Answer: Compare the actual variance of the error term s probability distribution and the mean (average) of the variance estimates after many, many repetitions. Mean (Average) of the Estimates for the Variance of the Error erm s Actual Value Simulation Probability Distribution Var[e] Repetitions SSE Divided by 500 200 50 Good news: his procedure is. Bad news: Does this procedure help Clint?. Why? NB: Nevertheless, keep in mind that the sum of squared errors based on the actual constant and coefficient, β Const or β x, provides an unbiased estimate of variance the error term s probability distribution. Sum of Squared Errors (SSE) Versus Sum of Squared Residuals () Sum of Squared Errors (SSE) Sum of Squared Residuals () Based on the error terms Based on the residuals y t β Const + β x x t + e t Res t y t Esty t where Esty t b Const + b x x t e t y t (β Const + β x x t ) Res t y t (b Const + b x x t ) Need the actual constant and coefficient, β Const and β x, to calculate the sum or squared errors But β Const and β x are unobservable; that is the whole problem Use the OLS procedure to calculate the estimates of the constant and coefficient, b Const and b x Use the estimates to calculate the sum of squared residuals We can think of the sum of squared residuals () as an estimate of the sum of squared errors (SSE).
6 Estimating the Variance of the Error erm s Probability Distribution, Var[e] Second Attempt Calculate the variance of the three residuals, Res 1, and Res ; that is, the variance of the actual value of y less the estimated value of y. Use this variance to estimate Var[e]: First Quiz: Variance of Residuals Numerical Value, Var[Res 1, and Res 1 st Quiz] Estimates Variance of Error erm s Probability Distribution: Var[e] First, we calculate the residuals from the first quiz Res t y t Est t y t (b Const + b x x t ) NB: Here we use the estimated constant and coefficient. and then their variance: Var[Res 1, and Res 1 st Quiz] (Res 1 - Mean[Res])2 + (Res 2 - Mean[Res]) 2 + (Res - Mean[Res]) 2 First Quiz b Const 6 b x 6 5 1.2 Res t y t Esty t Student x t y t Est t b Const + b x x t 6 + 6 5 x t Res t y t (6 + 6 5 x t ) Res 2 t 1 5 66 6 + 6 5 2 15 87 6 + 6 5 25 90 6 + 6 5 Sum Mean[Res] Mean[Res 1, and Res 1 st Quiz] Res 1 + Res 2 + Res In fact, we can provide that the mean of the residuals will always equal 0. Var[Res 1, and Res 1 st Quiz] Res 2 1 + Res 2 2 + Res 2 1 st Quiz 1st Quiz Econometrics Lab Simulation Estimate for the Variance Actual Value of the Error erm s Var[e] Repetition Res 1 Res 2 Res Probability Distribution 500 1 500 2 500 Observations Can we expect the estimate to equal the actual value?. In fact, we all but certain that the estimate will not equal the actual value. Sometimes the estimate is than the actual value and sometimes it is.
7 Question: Is this estimation procedure unbiased? Mean (Average) of the Estimates for the Variance of the Error erm s Actual Value Simulation Probability Distribution Var[e] Repetitions Divided by 500 200 50 Good news: Clint the information to perform this calculation. Bad news: he procedure is. It systematically the variance of the error term s probability distribution. Why Is Our Second Attempt Biased? Recall the difference between the error terms and the residuals: Error term Residual e t y t (β Const + β x x t ) Res t y t Res t y t (b Const + b x x t ) Var[e 1, e 2, and e 1 st Quiz] SSE Var[Res 1, and Res 1 st Quiz] Estimation procedure based on Estimation procedure based on the SSE s is. the s is. SSE 1 st 2 2 2 Quiz e 1 + e 2 + e [y 1 (β Const + β x x 1 )] 2 + [y 2 (β Const + β x x 2 )] 2 + [y (β Const + β x x )] 2 1 st Quiz Res 2 1 + Res 2 2 + Res 2 [y 1 (b Const + b x x 1 )] 2 + [y 2 (b Const + b x x 2 )] 2 + [y (b Const + b x x )] 2 Difference between the equations: SSE uses β Const and β x. uses b Const and b x. Question: How were b Const and b x chosen? Answer: o minimize the sum of squared residuals,. he sum of squared residuals will equal the sum of squared errors only if the estimates equal the actual values of β Const and β x. But we can never expect the estimates to equal the actual values: Sum using b s Sum using β s In all likelihood, β Const b Const and β x b x. SSE SSE When the actual Var[Res 1, and Res 1 st Quiz] Var[e 1, e 2, and e 1 st Quiz] constant and coefficient are used Procedure systematically Unbiased estimation the procedure is the variance of the procedure. error term s probability distribution
8 Estimating the Variance of the Error erm s Probability Distribution, Var[e] hird Attempt It can be shown that an unbiased estimate of the variance error term s probability distribution results when we divide the sum of squared residuals by the degrees of freedom rather than the sample size: Res 2 AdjVar[Res 1, and Res 1 st 1 + Res 2 2 2 + Res Quiz] Degrees of Freedom Degrees of Freedom Number of Degrees of Freedom Sample Size Estimated 2 1 Parameters AdjVar[Res 1, and Res 1 st Quiz] 1st Quiz 1 1 Econometrics Lab Simulation Estimate for the Variance Actual Value of the Error erm s Var[e] Repetition Res 1 Res 2 Res Probability Distribution 500 1 500 2 500 Observations Can we expect the estimate to equal the actual value?. In fact, we all but certain that the estimate will not equal the actual value. Sometimes the estimate is than the actual value and sometimes it is. Question: Is this estimation procedure unbiased? Mean (Average) of the Estimates for the Variance of the Error erm s Actual Value Simulation Probability Distribution Var[e] Repetitions Divided by 2 500 200 50 Good news: Clint the information to perform this calculation. he procedure is.
9 Step 2: Use the estimate for the variance of the error term s probability distribution to estimate the variance for the coefficient estimate s probability distribution. Step 1: Estimate the variance of the error term s Step 2: Apply the relationship between the probability distribution from the available variances of the coefficient estimate s and information information from the first quiz: the error term s probability distributions: EstVar[e] AdjVar[Res] Degrees of Freedom Var[b x ] Var[e] Σ t1 (xt x ) 2 EstVar[e] EstVar[b x ] Σ t1 (xt x ) 2 x 1 5 x 2 15 x 25 Σ t1 (xt x ) 2 + + SE[b x ] EstVar[b x ] NB: he square root of the estimated variance is called the standard error. What do we know about the estimation procedure for the error term s probability distribution?. What can we hope to be able to say about the estimation procedure for the coefficient estimate s probability distribution?.
10 Econometrics Lab Simulation Unbiased estimation procedure: After many, many repetitions of the experiment the average of the estimates equals the actual value. Estimated coefficient value from this repetition: Σ t1 (yt y )(x t x ) b x Σ t1 (xt x ) 2 EstVar[e] Act Coef 2 0 2 Degrees of Freedom EstVar[e] EstVar[b x ] Σ t1 (xt x ) 2 Estimate of the variance for the coefficient estimate s probability distribution calculated from this repetition Repetition Coef Value Est Mean Var Sum Sqr XDev Coef Var Est Mean Act Err Var 200 50 500 Variance of the estimated coefficient values from all repetitions. Actual Variance of Error erm s Probability Distribution: Var[e] Actual Variance of Coefficient Estimate s Probability Distribution: Var[b x ] Var[e] Σ t1 (xt x ) 2 Is the estimation procedure for the variance of the coefficient estimate s probability distribution unbiased? Average of the variance estimates from all repetitions. Estimate for the Variance Actual Value of the Coefficient Estimate s Var[e] Repetition Probability Distribution 500 1 500 2 500 Is the estimation procedure for the variance of the coefficient estimate s probability distribution unbiased? Variance of the Coefficent Mean (Average) of the Estimates Actual Estimate s Probability for the Variance of the Coefficient Var[e] Distribution: Var[b x ] Estimate s Probability Distribution 500 200 50
11 Degrees of Freedom Recall Attempts 2 and to estimate the variance of the error terms probability distribution Clint is trying to estimate the variance of the error term from the residual information: Error terms Residuals e t y t (β Const + β x x t ) Res t y t (b Const + b x x t ) We can interpret the residuals as the estimated error terms. Strategy: Use the residuals ( estimated errors ) to estimate the variance of the error term s probability distribution. Question: Why does dividing by the sample size fail, but dividing by the degrees of freedom succeed? Attempt 2: We divided by the sample size: Var[Res 1, and Res ] (Res 1 - Mean[Res])2 + (Res 2 - Mean[Res]) 2 + (Res - Mean[Res]) 2 Sample Size Since Mean[Res] 0: Var[Res 1, and Res ] Res 2 1 + Res 2 2 2 + Res Sample Size Sample Size Attempt : We divided by the degrees of freedom rather than the sample size: Res 2 1 + Res 2 2 2 + Res AdjVar[Res 1, and Res ] Degrees of Freedom Degrees of Freedom 1 Degrees of Freedom Sample Size Number of Estimated Parameters 2 1 Dividing by the degrees of freedom rather than the sample size worked. Question: Why does dividing by 1 rather than work? How Do We Calculate an Average? Year Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 1901 2.09 0.56 5.66 5.80 5.12 0.75.77 5.75.67 4.17 1.0 8.51 1902 2.1.2 5.47 2.92 2.42 4.54 4.66 4.65 5.8 5.59 1.27 4.27............. 2000.00.40.82 4.14 4.26 7.99 6.88 5.40 5.6 2.29 2.8 4.24.75 + 4.54 +... + 7.99 Mean (Average) for June 100 77.76 100.78 Each of the 100 Junes in the twentieth century provide one piece of information in calculating the average. Consequently, to calculate an average we divide the sum by the number of pieces of information. Key Principle: o calculate an average we divide the sum by the number of pieces of information: Sum Mean (Average) Number of Pieces of Infomration
12 Claim: he degrees of freedom equal the number of pieces of information that are available to estimate the error term s variance. y Question: Why does subtracting 2 from the sample size make sense? o understand why, suppose that the sample size were 2. Plot the scatter diagram. With only two observations, we only have points. he best fitting line passes through each of the points on the scatter diagram. Consequently, the two residuals, estimated errors, for each observation must always equal when the sample size is 2 regardless of what the variance of the error term s probability distribution actually equals: Sample Size of 2: Res 1 and Res 2 Question: Do the first two residuals, the first two estimated errors, provide regardless of what Var[e] equals information about the actual variance of error term s probability distribution? x Question: Which observation provides the first piece of information about of the actual variance of the error term s probability distribution? Summary he first two observations provide no information about the variance. he third observation provides the piece of information about the variance. y Res Res 2 y Consequently, in Clint s case, Res 1 when there are three observations, we should divide by to calculate the average of the squared Suggests large error term variance x deviations because we really only have piece of information. x Suggests small error term variance In general, we should divide by the degrees of freedom, the sample size less the number of estimated parameters: Degrees of Freedom Sample Size Number of Estimated Parameters
1 Summary of Ordinary Least Squares (OLS) Calculations and the Regression Printout he ordinary least squares (OLS) estimation procedure actually includes three procedures: A Procedure to Estimate the Value of the Parameters Σ t1 (yt y )(x t x ) o b x Σ t1 (xt x 240 ) 2 200 6 5 1.2 o b Const y b x x 81 6 15 6 5 A Procedure to Estimate the Variance of the Error erm s Probability Distribution o Σ 2 t1 Rest Σ t1 (yt Esty t ) 2 Σ t1 (yt b Const b x x t ) 2 54 o EstVar[e] AdjVar[Res 1, and Res ] Degrees of Freedom 54 2 54 1 54 o S.E. of regression EstVar[e] 54 7.48 he square root of the estimated value of the error terms probability distribution is call the standard error of the regression A Procedure to Estimate the Variance of the Coefficient Estimate s Probability Distribution EstVar[e] o EstVar[b x ] Σ t1 (xt x 54 ) 2 200.27 o SE[b x ] EstVar[b x ].27.5196 Good News: When the standard ordinary least squares (OLS) premises are satisfied: Each of these procedures is unbiased. he procedure to estimate the value of the parameters is the best linear unbiased estimation procedure. EViews performs these calculations for us thereby saving us the laborious task of performing all the arithmetic: Dependent Variable: Y Included observations: Variable Coefficient Std. Error t-statistic Prob. X 1.200000 0.519615 2.09401 0.2601 C 6.00000 8.874120 7.099296 0.0891 S.E. of regression 7.48469 Akaike info criterion 7.061582 Sum squared resid 54.00000 Schwarz criterion 6.460657