Introduction to Estimation Methods for Time Series models Lecture 1 Fulvio Corsi SNS Pisa Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 1 / 19
Estimation Methods We will briefly review the following estimation methods: LS: Least Square OLS: Ordinary Least Square NLS: Nonlinear Least Square (idea) ML: Maximum Likelihood GMM: Generalized Method of Moments (idea) Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 2 / 19
Econometric model Econometrics: intersection of Economics and Statistics Econometric model = association between y i and x i E.g.: personal income y i and personal QI x i stock return y i and market return x i current return y t and past returns y t h Econometric model provides approximate i.e. probabilistic description of the association. The relation will be stochastic and not deterministic. Econometrics provides estimation methods for parametric model Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 3 / 19
Estimators Given a parametric model X f(x;θ 0 ) with θ 0 Ω(θ) and a random sample {x 1, x 2,...,x n} Estimator: is any function T() of the random sample {x 1, x 2,...,x n}, i.e. ˆθ T(x 1, x 2,...,x n) Ex: if x i.i.d.(µ,σ) then X n = 1 n n t=1 xt is an estimator for µ Sampling Distribution: the estimator, being a function of the random sample, is also a random variable. Ex: X n N(µ,σ/n) Estimate: a single realization of the statistics on a particular sample. Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 4 / 19
Estimators: Finite Sample Properties Unbiased Estimator E[ˆθ] = θ 0 or Bias[ˆθ] E[ˆθ θ 0 ] = 0 Efficient Unbiased Estimator Var[ˆθ 1 ] < Var[ˆθ 2 ] Mean Square Error MSE[ˆθ] E[(ˆθ θ 0 ) 2 ] = Var[ˆθ]+Bias[ˆθ] 2 Best Linear Unbiased Estimator (BLUE): linear function of the data with minumum variance among linear unbiased estimators. Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 5 / 19
Ordinary Least Square (OLS): Linear model Linera model y i = f(x i1, x i2,..., x ik )+ɛ i = β 0 +β 1 x i1 +β 2 x i2 + +β K x ik +ɛ i i = 1,...,N where - y i : dependent or explained variable (observed) - x i : regressors or covariates or explanatory variables (observed) - ɛ i : error term or random disturbance (unobserved) - β i : unknown parameters or regression coefficient (unobserved) Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 6 / 19
Ordinary Least Square (OLS): Vector notation can be written in vector notation and in the even more compact matrix notation with Y = y 1 y 2.. y N }{{} N 1 y i = β 0 +β 1 x i1 +β 2 x i2 + +β K x ik +ɛ i y i = x i β + ɛ i }{{}}{{} 1 K K 1 Y }{{} N 1 = X }{{} N K β + ɛ }{{} K 1 x 1 1 x 1,1 x 2,1... x 1,K x 2 1 x X =... = 2,1 x 2,2... x 2,K....... x N } 1 x N,1 x N,2 {{... x N,K } N K ɛ 1 ɛ 2 ɛ =.. ɛ N }{{} N 1 Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 7 / 19
Standard OLS Assumptions Standard OLS Assumptions: H.1 Strict exogeneity of regressors: E[ɛ X] = 0 Note: ɛ i does not depend on any x j, neither past nor future xs E[ɛ X] = 0 E[ɛ] = 0 E[ɛ X] = 0 E[y X] = Xβ i.e. Xβ is the conditional mean of y X. H.2 Identification: X is N K with rank K with probability 1 H.3 Spherical errors Var[ɛ X]= σ 2 I N homoscedastic: Var[ɛ i X] = σ 2, i = 1,..., n and uncorrelated errors: Cov[ɛ i ɛ j X] = 0 i j Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 8 / 19
Univariate OLS E.g., univariate regression: y i = α+βx i +ɛ i with ɛ i N(0,σ 2 ) i Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 9 / 19
Ordinary Least Square Idea: minimize the square of the estimation errors e Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 10 / 19
OLS Estimator Goal: statistical inference on β, e.g. estimate β Least Square find β that minimize the sum of squared residuals in Y = Xβ +ɛ: N SS = ɛ 2 i = ɛ ɛ i=1 = (Y Xβ) (Y Xβ) = YY 2X Yβ +β X Xβ F.O.C. : 2X Y + 2X Xβ = 0 X (Y Xβ) = 0 X Xβ = X Y OLS estimator: ˆβ = (X X) 1 X Y ( N ) 1 ( N ) = x i x i x i y i i=1 i=1 Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 11 / 19
Finite sample Properties Unbiasedness: E[ˆβ X] = β ˆβ = (X X) 1 X (Xβ +ɛ) = β +(X X) 1 X ɛ Then E[ˆβ X] = β +(X X) 1 X E[ɛ X] = β }{{} =0 (H.1) Variance: Var(ˆβ X) = σ 2 (X X) 1 Var[ˆβ X] = (X X) 1 X Var[ɛ X] }{{} σ 2 I N (H.3) X(X X) 1 = σ 2 (X X) 1 Efficiency (Gauss-Markov Theorem): ˆβ is BLUE, i.e. Var(ˆβ X) Var( β X), β linear unbiased estimator Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 12 / 19
Population and estimated coefficient Notation true values y α β ɛ fitted/estimated ŷ i a b e Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 13 / 19
Projections being we have that Y = Xβ }{{} + ɛ }{{} explained by the model random/unexplained Ŷ ˆβ = (X X) 1 X Y = Xˆβ = X(X X) 1 X Y = P xy P x = X(X X) 1 X is called the OLS projection matrix. Moreover, e = Y Xˆβ = Y X(X X) 1 X Y = (I X(X X) 1 X )Y = (I P x)y = M xy M x = I P x is called the residual Maker matrix as M xy = e Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 14 / 19
Orthogonal projection M x and P x are - symmetric P = P - idempotent P 2 = PP = P - orthogonal PM = MP = 0 Hence, OLS partitions Y in two orthogonal parts: Y = P xy + M xy = Ŷ + e = projection + residual Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 15 / 19
Orthogonal projection Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 16 / 19
Goodness of fit being Ŷ e then Var(Y) = Var(Ŷ) + Var(e) TSS = ESS n n + RSS n Total Var = Explained Var + Residual Var A common measure of goodness of fit is the coefficient of determination R 2 : R 2 = Explained Var Total Var = 1 Residual Var Total Var = 1 RSS TSS since R 2 always increases when a regressor is added (even if uncorrelated) Adjusted R 2 = 1 Residual Var/(n K) Total Var/(n 1) Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 17 / 19
Goodness of fit of a linear regression Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 18 / 19
OLS with Normality if H.4: ɛ X N(0,σ 2 I N ) then ˆβ N(β,σ 2 (X X) 1 ) Rao-Blackwell Theorem: ˆβ is the ML estimator (i.e. most efficient unbiased estimator) Hypothesis Testing to make inference on ˆβ N(β,σ 2 (X X) 1 ) we need an estimator for σ 2. Using e y ŷ with ŷ Xˆβ we can define: we can prove that ˆσ 2 = N i=1 e2 N K = e e N K = RSS N K E[ σ 2 ] = σ 2 and RSS/σ 2 χ 2 N K Hence, denoting Var( β) = σ 2 (X X) 1 we have that β β N(0, 1) t N K Var( β) χ 2 N K /(N K) Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 1 SNS Pisa 19 / 19