7. Introduction to Large Sample Theory

Transcription

1 7. Introuction to Large Samle Theory Hayashi / Avance Econometrics I, Autumn 2010, Large-Samle Theory 1

2 Introuction We looke at finite-samle roerties of the OLS estimator an its associate test statistics These are base on assumtions that are violate very often The finite-samle theory breaks own if one of the following three assumtions is violate: - the exogeneity of regressors - the normality of the error term, an - the linearity of the regression equation Avance Econometrics I, Autumn 2010, Large-Samle Theory 2

3 Introuction (cont ) Asymtotic or large-samle theory rovies an alternative aroach when these assumtions are violate It erives an aroximation to the istribution of the estimator an its associate statistics assuming that the samle size is sufficiently large Rather than making assumtions on the samle of a given size, largesamle theory makes assumtions on the stochastic rocess that generates the samle. Avance Econometrics I, Autumn 2010, Large-Samle Theory 3

4 Introuction (cont ) The two main concets in asymtotics relate to consistency an asymtotic normality. Some intuition: Consistency: the more ata we get, the closer we get to knowing the truth (or we eventually know the truth) Asymtotic normality: as we get more an more ata, averages of ranom variables behave like normally istribute ranom variables. Examle: Establishing consistency an asymtotic normality of an i.i.. ranom samle X 1,..., X N with E(X i ) = µ an var(x i ) = σ 2. Avance Econometrics I, Autumn 2010, Large-Samle Theory 4

5 Introuction (cont ) The main robability theory tools for asymtotics: The robability theory tools for establishing consistency of estimators are: Laws of Large Numbers (LLNs) A LLN is a result that states the conitions uner which a samle average of ranom variables converges to a oulation exectation. LLNs concern conitions uner which the sequence of samle mean converges either in robability or almost surely There are many LLN results (eg. Chebychev s LLN, Kolmongorov s/khinchine s LLN, Markov s LLN) Avance Econometrics I, Autumn 2010, Large-Samle Theory 5

6 Introuction (cont ) The robability tools for establishing asymtotic normality are: Central Limit Theorems (CLTs) CLTs are about the limiting behaviour of the ifference between a samle mean an its execte value There are many CLTs (eg. Lineberg-Levy CLT, Lineberg-Feller CLT, Liaounov s CLT) Avance Econometrics I, Autumn 2010, Large-Samle Theory 6

7 Basic concets of large samle theory Using large samle theory, we can isense with basic assumtions from finite samle theory 1.2 E(ε i X) = 0: strict exogeneity 1.4 V ar(ε X) = σ 2 I: homosceasticity 1.5 ε X N(0, σ 2 I n ): normality of the error term Aroximate/assymtotic istribution of b, an t- an the F-statistic can be obtaine Avance Econometrics I, Autumn 2010, Large-Samle Theory 7

8 Moes of convergence - Convergence in robability {z n }: sequence of ranom variables {z n }: sequence of ranom vectors Convergence in robability: A sequence {z n } converges in robability to a constant α if for any ε > 0 lim P ( z n α > ε) = 0 n Short-han we write: lim n Extens to ranom vectors: z n = α or z n α or z n α 0 If lim n P ( z kn α k > ε) = 0 k = 1, 2,..., K, then z n α where z nk is the k-th element of z n an α k the k-th element of α Avance Econometrics I, Autumn 2010, Large-Samle Theory 8

9 Moes of convergence - Almost Sure Convergence Almost Sure Convergence: A sequence of ranom scalars {z n } converges almost surely to a constant α if: ( ) Prob lim z n = α = 1 n We write this as z n a.s. α. The extension to ranom vectors is analogous to that for convergence in robability. Note: This concet of convergence is stronger than convergence in robability if a sequence converges a.s., then it converges in robability. Avance Econometrics I, Autumn 2010, Large-Samle Theory 9

10 Moes of convergence - Convergence in mean square Convergence in mean square: lim n E [ (z n α) 2] = 0 or z n m.s. α The extension to ranom vectors is analogous to that for convergence in robability: z n m.s. α if each element of z n converges in mean square to the corresoning comonent of α. Convergence in mean square exten to ranom vectors Avance Econometrics I, Autumn 2010, Large-Samle Theory 10

11 Moes of convergence - Convergence to a Ranom Variable In the above efinitions of convergence, the limit is a constant. However, the limit can also be a ranom variable. We say that a sequence of K-imensional ranom variables {z n } converges to a K-imensional ranom variable z an write z n z if {z n z} converges to 0: Similarly, z n z is the same as z n z 0. z n a.s. z is the same as z n z a.s. 0, z n m.s. z is the same as z n z m.s. 0. Avance Econometrics I, Autumn 2010, Large-Samle Theory 11

12 Moes of convergence - Convergence in istribution Convergence in istribution: Let {z n } be a sequence of ranom scalars an F n istribution function (c..f.) of z n. be the cumulative We say that {z n } converges in istribution to a ranom scalar z if the c..f. F n of z n converges to the c..f. F of z at every continuity oint of F. We write z n z or z n L z an call F the asymtotic (or limit or limiting) istribution of z n. Avance Econometrics I, Autumn 2010, Large-Samle Theory 12

13 Moes of convergence - Convergence in istribution Convergence in robability is stronger than convergence in istribution, i.e., z n z z n z. A secial case of convergence in istribution is that z is a constant (a trivial ranom variable). The extension to a sequence of ranom vectors is immeiate: z n z if the joint c..f. F n of the ranom vector z n converges to the joint c..f. F of z at every continuity oint of F. Note: For convergence in istribution, unlike the other concets of convergence, element-by-element convergence oes not necessarily mean convergence for the vector sequence. Avance Econometrics I, Autumn 2010, Large-Samle Theory 13

14 Weak Law of Large Numbers (WLLN) accoring to Khinchine {z i } i.i.. with E(z i ) = µ, then z n = 1 n n i=1 z i we have: z n µ or lim n P ( z kn µ > ε) = 0 or lim z n = µ Avance Econometrics I, Autumn 2010, Large-Samle Theory 14

15 Extensions of the Weak Law of Large Numbers (WLLN) The WLLN hols for: Extension (1): Multivariate Extension (sequence of ranom vectors {z i }) Extension (2): Relaxation of ineenence Extension (3): Functions of ranom variables h(z i ) Extension (4): Vector value functions f(z i ) Avance Econometrics I, Autumn 2010, Large-Samle Theory 15

16 Central Limit Theorems (Lineberg-Levy) {z i } i.i.. with E(z i ) = µ an V ar(z i ) = σ 2. Then for z n = 1 n n i=1 z i: n(zn µ) N ( 0, σ 2) or z n µ a N ( 0, σ2 n ) or z n a N ( µ, σ2 n ) Remark: Rea a aroximately istribute as CLT also hols for multivariate extension: sequence of ranom vectors {z i } Avance Econometrics I, Autumn 2010, Large-Samle Theory 16

17 Useful lemmas of large samle theory Lemma 1: z n α with a as a continuous function which oes not een on n then: a(z n ) a(α) or lim n Examles: x n α ln(x n ) ln(α) ( ) a(z n ) = a lim (z n ) n x n β an y n γ x n + y n β + γ Y n Γ Y 1 n Γ 1 Avance Econometrics I, Autumn 2010, Large-Samle Theory 17

18 Useful lemmas of large samle theory (continue) Lemma 2: z n z then: a(z n ) a(z) Examles: z n z, z N(0, 1) z 2 χ 2 (1) z n N(0, 1) z 2 χ 2 (1) Avance Econometrics I, Autumn 2010, Large-Samle Theory 18

19 Useful lemmas of large samle theory (continue) Lemma 3: x n x an y n α then: x n + y n x + α Examles: x n N(0, 1), y n α x n + y n N(α, 1) x n x, y n 0 x n + y n x Lemma 4: x n x an y n 0 then: x n y n 0 Avance Econometrics I, Autumn 2010, Large-Samle Theory 19

20 Useful lemmas of large samle theory (continue) Lemma 5: x n x an A n A then: A n x n A x Examle: x n MV N(0, Σ) A n x n MV N(0, AΣA ) Lemma 6: x n x an A n A then: x na 1 n x n x A 1 x Avance Econometrics I, Autumn 2010, Large-Samle Theory 20

21 8. Time Series Basics (Stationarity an Ergoicity) Hayashi Avance Econometrics I, Autumn 2010, Large-Samle Theory 21

22 Deenence in the ata Certain egree of eenence in the ata in time series analysis; only one realization of the ata generating rocess is given CLT an WLLN rely on i.i.. ata, but eenence in real worl ata Examles: Inflation rate Stock market returns Stochastic rocess: sequence of r.v.s. inexe by time {z 1, z 2, z 3,...} or {z i } with i = 1, 2,... A realization/samle ath: One ossible outcome of the rocess Avance Econometrics I, Autumn 2010, Large-Samle Theory 22

23 Deenence in the ata - theoretical consieration If we were able to run the worl several times, we ha ifferent realizations of the rocess at one oint in time We coul comute ensemble means an aly the WLLN As the escribe reetition is not ossible, we take the mean over the one realization of the rocess Key question: Does 1 T T t=1 x t E(x) hol? Conition: Stationarity of the rocess Avance Econometrics I, Autumn 2010, Large-Samle Theory 23

24 Definition of stationarity Strict stationarity: The joint istribution of z i, z i1, z i2,..., z ir eens only on the relative osition i 1 i, i 2 i,..., i r i but not on i itself In other wors: The joint istribution of (z i, z ir ) is the same as the joint istribution of (z j, z jr ) if i i r = j j r Weak stationarity: - E(z i ) oes not een on i - Cov(z i, z i j ) eens on j (istance), but not on i (absolute osition) Avance Econometrics I, Autumn 2010, Large-Samle Theory 24

25 Ergoicity A stationary rocess is also calle ergoic if lim E [f(z i, z i+1,..., z i+k ) g(z i+n, z i+n+1,..., z i+n+l )] = n E [f(z i, z i+1,..., z i+k )] E [g(z i+n, z i+n+1,..., z i+n+l )] Ergoic Theorem: Sequence {z i } is stationary an ergoic with E(z i ) = µ, then z n 1 n n i=1 z i a.s. µ Avance Econometrics I, Autumn 2010, Large-Samle Theory 25

26 Martingale ifference sequence Stationarity an Ergoicity are not enough for alying the CLT. To erive the CAN roerty of the OLS-estimator we assume: {g i } = {x i ε i } {g i } is a stationary an ergoic martingale ifference sequence (m..s.): E(g i g i 1, g i 2,..., g i j ) = 0 E(g i ) = 0 Imlications of m..s. when 1 x i : ε i an ε i j are uncorrelate, i.e. Cov(ε i, ε i j ) = 0 Avance Econometrics I, Autumn 2010, Large-Samle Theory 26

27 Large samle assumtions for the OLS estimator (2.1) Linearity: y i = x i β + ε i i = 1, 2,..., n (2.2) Ergoic Stationarity: the (K + 1)-imensional vector stochastic rocess {y i, X i } is jointly stationary an erogoic (2.3) Orthogonality/reetermine regressors: E(x ik ε i ) = 0 If x ik = 1 E(ε i ) = 0 Cov(x ik, ε i ) = 0 This can be written as E[x i (y i x i β)] = 0 or E(g i) = 0, where g i x i ε. (2.4) Rank conition: E(x i x i) Σ XX is non-singular KxK Avance Econometrics I, Autumn 2010, Large-Samle Theory 27

28 Large samle assumtions for the OLS estimator (cont ) (2.5) Martingale Difference Sequence (M.D.S): g i is a martingale ifference sequence with finite secon moments. It follows that; i. E(g i ) = 0, ii. The K K matrix of cross moments E(g i g i ) is nonsingular iii. S Avar(ḡ) = E(g i g i ), where (ḡ) 1 n of the asymtotic istribution of nḡ) i g i. (Avar(ḡ) is the variance See Hayashi Avance Econometrics I, Autumn 2010, Large-Samle Theory 28

29 Large samle istribution of the OLS estimator We get for b = (X X) 1 X y: }{{} b n = [ 1 n n i=1 x ix i] 1 1 n n i=1 x iy i n inicates the eenence on the samle size Uner WLLN an lemma 1: b n β ) n(bn β) MV N (0, Avar(b)) or b a MV N (β, Avar(b) n b n is consistent, asymtotically normal (CAN) Avance Econometrics I, Autumn 2010, Large-Samle Theory 29

30 How to estimate Avar(b) Avar(b) = Σ 1 xx E(g i g i )Σ 1 xx with g i = X i ε i 1 n n i=1 x ix i E(x ix i ) Estimation of E(g i g i ): Ŝ = 1 n e 2 i x i x i E(g ig i ) [ Avar(b) 1 = n 1 n x i x i] Ŝ i=1 [ 1 n n i=1 x i x i] 1 Avar(b) = E(x i x i) 1 E(g i g i)e(x i x i) 1 Avance Econometrics I, Autumn 2010, Large-Samle Theory 30

31 Develoing a test statistic uner the assumtion of conitional homoskeasticity Assumtion: E(ε 2 i x i) = σ 2 Avar(b) = [ 1 n ] 1 n x i x i ˆσ 2 1 n i=1 = ˆσ 2 [ 1 n n x i x i i=1 ] 1 n x i x i i=1 [ 1 n n x i x i i=1 ] 1 with Ŝ = 1 n n i=1 e2 i 1 n n i=1 x ix i Note: 1 n n i=1 e2 i is a biase estimate for σ2 Avance Econometrics I, Autumn 2010, Large-Samle Theory 31

32 White stanar errors Ajusting the test statistics to make them robust against violations of conitional homoskeasticity t-ratio t k = b k β k [[ 1 ni=1 n x i x i] 1 1 ni=1 n e 2 i x ix i[ n 1 ni=1 ] x i x i] 1 n kk a N(0, 1) Hols uner H 0 : β k = β k F-ratio [ W = (Rb r) Avar(b) R ] 1 R (Rb r) a χ 2 (#r) n Hols uner H 0 : Rβ r = 0; allows for nonlinear restrictions on β Avance Econometrics I, Autumn 2010, Large-Samle Theory 32

33 We show that b n = (X X) 1 X y is consistent b n = [ 1 n n i=1 x ix i ] 1 1 n n i=1 x iy i b } n {{ β } = [ 1 n xi x i] 1 1 n xi ε i samling error We show: b n β When sequence {y i, x i } allows alication of WLLN 1 n n i=1 x ix i E(x ix i ) 1 n n i=1 x iε E(x i ε i ) 0 Avance Econometrics I, Autumn 2010, Large-Samle Theory 33

34 We show that b n = (X X) 1 X y is consistent (continue) Lemma 1 imlies: b n β = [ 1 n xi x i] 1 1 n E(x i x i) 1 E(x i ε i ) E(x i x i) 1 0 = 0 xi ε i b n = (X X) 1 X y is consistent Avance Econometrics I, Autumn 2010, Large-Samle Theory 34

35 We show that b n = (X X) 1 X y is asymtotically normal Sequence {g i } = {x i ε i } allows alying CLT for 1 n xi ε i = g n(g E(gi )) MV N(0, Σ 1 xx E(g i g i )Σ 1 xx ) n(bn β) = [ 1 n xi x i] 1 n g Alying lemma 5: A n = [ ] 1 n xi x 1 i A = Σ 1 xx x n = n g x MV N(0, E(g i g i )) n(b n β) MV N(0, Σ 1 xx E(g i g i )Σ 1 xx ) b n is CAN Avance Econometrics I, Autumn 2010, Large-Samle Theory 35

36 9. Generalize Least Squares Hayashi Avance Econometrics I, Autumn 2010, Large-Samle Theory 36

37 Assumtions of GLS Linearity: y i = x i β + ε i Full rank: rank(x) = K Strict exogeneity: E(ε i X) = 0 E(ε i ) = 0 an Cov(ε i, x ik ) = E(ε i x ik ) = 0 NOT assume: V ar(ε X) = σ 2 I n Instea: V ar(ε X) = E(εε X) = V ar(ε 1 X) Cov(ε 1, ε 2 X)... Cov(ε 1, ε n X) Cov(ε 1, ε 2 X) V ar(ε 2 X). Cov(ε 1, ε 3 X). Cov(ε 2, ε 3 X) V ar(ε 3 X).... Cov(ε 1, ε n X)... V ar(ε n X) V ar(ε X) = E(εε X) = σ 2 V(X) Avance Econometrics I, Autumn 2010, Large-Samle Theory 37

38 Deriving the GLS estimator Derive uner the assumtion that V(X) is known, symmetric an ositive efinite V(X) 1 = C C Transformation: ỹ = Cy X = CX y = Xβ + ε Cy = CXβ + Cε ỹ = Xβ + ε Avance Econometrics I, Autumn 2010, Large-Samle Theory 38

39 Least squares estimation of β using transforme ata ˆβ GLS = ( X 1 X) X ỹ = (X C CX) 1 X C Cy = (X 1 σ 2V 1 X) 1 X 1 σ 2V 1 y = [ X [V ar(ε X)] 1 X ] 1 X [V ar(ε X)] 1 y GLS estimator is the best linear unbiase estimator (BLUE) Problems: Difficult to work out the asymtotic roerties of ˆβ GLS In real worl alications V ar(ε X) not known If V ar(ε X) is estimate the BLUE-roerty of ˆβ GLS is lost Avance Econometrics I, Autumn 2010, Large-Samle Theory 39

40 Secial case of GLS - weighte least squares E(εε X) = V ar(ε X) = σ 2 V 1 (X) V 2 (X) V N (X) = σ 2 V As V(X) 1 = C C C = 1 V1 (X) 0 1 V2 (X) V n(x) = 1 s s s n argmin ( n y1 i=1 s i ˆβ 1 s 1 i ˆβ 2 x i2 s i... ˆβ K x ik si ) 2 Observations are weighte by stanar eviation Avance Econometrics I, Autumn 2010, Large-Samle Theory 40

41 10. Multicollinearity Avance Econometrics I, Autumn 2010, Large-Samle Theory 41

42 Exact multicollinearity Exressing a regressor as linear combination of (an)other regressor(s) rank(x) K: No full rank Assumtion 1.3 or 2.4 is violate (X X) 1 oes not exist Often economic variables are correlate to some egree BLUE result is not affecte Large samle results are not affecte relative results V ar(b X) is affecte in absolute terms Avance Econometrics I, Autumn 2010, Large-Samle Theory 42

43 Effects of Multicollinearity an solutions to the roblem Effects: - Coefficients may have high stanar errors an low significance levels - Estimates may have the wrong sign - Small changes in the ata rouces wie swings in the arameter estimates Solutions: - Increasing recision by imlementing more ata. (Costly!) - Builing a better fitting moel that leaves less unexlaine. - Excluing some regressors. (Dangerous! Omitte variable bias!) Avance Econometrics I, Autumn 2010, Large-Samle Theory 43