Large Sample Theory Covergece Covergece i Probability Covergece i Distributio Cetral Limit Theorems Asymptotic Distributio Delta Method
Covergece i Probability A sequece of radom scalars {z } = (z 1,z, ) coverges i probability to z (a costat or a radom variable) if, for ay e>0, lim Prob( z - z >e) = 0. z is the probability limit of z ad is writte as: plim z = z or z p z. Extesio to a sequece of radom vectors or matrices: elemet-by-elemet covergece i probability, z p z.
Covergece i Probability A special case of covergece i probability is mea square covergece: if E(z ) = m ad Var(z ) = s such that m z ad s 0, the z coverges i mea square to z, ad z p z. What is the differece betwee E(z ) ad plim z or z p z? Mea square covergece is sufficiet (ot ecessary) for covergece i probability.
Covergece i Probability Example: Sample: x { x, x, } ~ ( ms, ) 1 1 Sample Mea: x x i 1 i Sequece of Sample Meas: { x}, E( x ) m m x m p s Var( x ) 0
Almost Sure Covergece A sequece of radom scalars {z } = (z 1,z, ) coverges almost surely to z (a costat or a radom variable) if, Prob(lim z =z) = 1. Write: z as z. If a sequece coverges almost surely, the it coverges i probability. That is, z as z z p z. Extesio to a sequece of radom vectors or matrices: elemet-by-elemet almost sure covergece. I particular, z as z z p z.
Let Laws of Large Numbers LLN cocer coditios uder which the sequece coverges i probability. Chebychev s LLN: [ lim E(z) m, lim Var(z ) 0] z p z 1 z i i 1 { z } μ
Laws of Large Numbers Kolmogorov s LLN: Let {z i } be i.i.d. with E(z i ) = m (the variace does ot eed to be fiite). The z as μ This implies z p μ
Speed of Covergece Order of a Sequece Little oh o(.): Sequece z is o( ) (order less tha ) if ad oly if - z 0. Example: z = 1.4 is o( 1.5 ) sice -1.5 z = 1 /.1 0. Big oh O(.): Sequece z is O( ) if ad oly if - z a fiite ozero costat. Example 1: z = ( + + 1) is O( ). Example : i x i is usually O( 1 ) sice this is the mea of x i ad the mea of x i geerally coverges to E[x i ], a fiite costat. What if the sequece is a radom variable? The order is i terms of the variace. Example: What is the order of the sequece i radom samplig? Because Var[ ] = σ / which is O(1/) x x
Covergece i Distributio Let {z } be a sequece of radom scalars ad F be the c.d.f. of z. {z } coverges i distributio to a radom scalar z if the c.d.f. F of z coverges to the c.d.f. F of z at every cotiuity poit of F. That is, z d z. F is the asymptotic or limitig distributio of z. z p z z d z, or z a F(z)
Covergece i Distributio The extesio to a sequece of radom vectors: z d z if the joit c.d.f. F of the radom vector z coverges to the joit c.d.f. F of z at every cotiuity poit of F. However, elemet-by-elemet covergece does ot ecessarily mea joit covergece.
Cetral Limit Theorems CLT cocer about the limitig behavior of z m Note: m blow up by E(z ). E(z i.i.d. Lideberg-Levy CLT (multivariate): Let {z i } be i.i.d. with E(z i ) = m ad Var(z i ) =. The i ) if z i d i 1 i is 1 z ( μ) ( z μ) N (0, Σ)
Cetral Limit Theorems Lideberg-Levy CLT (uivariate): If z ~ (m,s ), ad {z 1,z,...,z } are radom sample. Defie z z 1 i 1 z, the ( m) d N(0, s ) i
Cetral Limit Theorems Lideberg-Feller CLT (uivariate): If z i ~ (m i,s i ), i=1,,,. Let m 1 1 m s s s, 1 i ad i i 1 i If o sigle term domiates this average variace, the z ( m) d N(0, s )
Asymptotic Distributio A asymptotic distributio is a fiite sample approximatio to the true distributio of a radom variable that is good for large samples, but ot ecessarily for small samples. Stabilizig trasformatio to obtai a limitig distributio: Multiply radom variable x by some power, a, of such that the limitig distributio of a x has a fiite, ozero variace. x Example, has a limitig variace of zero, sice the variace is σ /. But, the variace of x is σ. However, this does ot stabilize the distributio because E( x ) m. The stabilizig trasformatio would be x ( m)
Asymptotic Distributio Obtaiig a asymptotic distributio from a limitig distributio: Obtai the limitig distributio via a stabilizig trasformatio. Assume the limitig distributio applies reasoably well i fiite samples. Ivert the stabilizig trasformatio to obtai the asymptotic distributio.
Asymptotic Distributio Example: Asymptotic ormality of a distributio. From (x m) / s N[0,1] s (x m) N[0, s ] a (x m) N[0, s / ] a x N[ m, s / ] Asymptotic distributio. a / asymptotic variace of x. d
Asymptotic Efficiecy Compariso of asymptotic variaces How to compare cosistet estimators? If both coverge to costats, both variaces go to zero. Example: Radom samplig from the ormal distributio, Sample mea is asymptotically N[μ,σ /] Media is asymptotically N[μ,(π/)σ /] Mea is asymptotically more efficiet.
Covergece: Useful Results Multivariate Covergece i Distributio Let {z } be a sequece of K-dimesioal radom vectors. The: z d z l z d l z for ay K-dimesioal vector of l real umbers.
Covergece: Useful Results Slutsky Theorem: Suppose a(.) is a scalaror vector-valued cotiuous fuctio that does ot deped o : z p a a(z ) p a(a) z d z a(z ) d a(z) x d x, y p a x +y d x+a x d x, y p 0 y x p 0
Covergece: Useful Results Slutsky results for matrices: A p A (plim A = A), B p B (plim B = B), (elemet by elemet) plim (A -1 ) = [plim A ] -1 = A -1 plim (A B ) = (plima )(plimb ) = AB
Covergece: Useful Results x d x, A p A A x d Ax I particular, if x~n(0, ), the A x d N(0,A A ) x d x, A p A x A -1 x d x A -1 x
Delta Method Suppose {x } is a sequece of K- dimesioal radom vector such that x b ad (x -b) d z. Suppose a(.): R K R r has cotiuous first derivatives with A(b) defied by a(β) A( β) β The [a(x )-a(b)] d A(b)z '
Delta Method (x -b) d N(0, ) [a(x )-a(b)] d N(0, A(b) A(b) )
Delta Method Example a x N[ m, s / ] What is the asymptotic distributio of f(x )=exp(x ) or f(x )=1/x (1) Normal sice x is asymptotically ormally distributed () Asymptotic mea is f( m)=exp( m) or 1/ m. (3) For the variace, we eed f'( m) =exp( m) or -1/ m 4 Asy.Var[f(x )]= [exp( m)] s / or [1/ m ] s /