Maximum Likelihood Asymptotic Theory. Eduardo Rossi University of Pavia
|
|
- Shavonne Allen
- 5 years ago
- Views:
Transcription
1 Maximum Likelihood Asymtotic Theory Eduardo Rossi University of Pavia
2 Slutsky s Theorem, Cramer s Theorem Slutsky s Theorem Let {X N } be a random sequence converging in robability to a constant a, and let g( ) be a function continuous at a. Then g(x N ) g(a) Cramer s Theorem Let {Y N } and {X N } be random sequences, with Y N d Y and XN a (a constant). Then 1. X N + Y N d a + Y 2. X N Y N d ay 3. Y N X N d Y a when a 0. Continuous Maing Theorem If X N continuous, then g(x N ) d g(x). d X and g( ) is Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 2
3 Cramér-Wold device Cramér-Wold device If {X N } is a sequence of random vectors, X N D X if and only if for all conformable fixed vectors λ, λ X N D λ X. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 3
4 The Weak Law of Large Numbers, Kinchine s Theorem Cramér s Theorem Assume z N N(µ,Σ) and A N is a conformable matrix with A N A. Then A N z N d N(Aµ,AΣA ) The Weak Law of Large Numbers It is the result that given a samle of z 1, z 2,...,z N of random variables with E(z t ) = µ the samle mean z N µ The strong law of large numbers is the corresonding result with a.s. convergence. Khinchine s Theorem Let {z t } be i.i.d. and integrable with E(z t ) = µ <. Then z N µ. d Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 4
5 Chebyshev s Theorem Chebyshev s Theorem If E(z t ) = 0, t and lim N z N 0, then 0. z N Assuming E(z t ) = 0 and E(z 2 t ) = σ 2 t, if the terms of the sequence are indeendent they are also uncorrelated, then E(z 2 N) = 1 N 2 N t=1 σ 2 t Given that E(z 2 N) 0 as N imlies that z N alying the Chebyshev s inequality 0. In fact, by P( z N > ǫ) E(z2 N ) ǫ 2 0 ǫ > 0. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 5
6 The Central Limit Theorem The Central Limit Theorem The CLT concers the convergence of scaled sums of random variables, but the means of the variables must be zero, and the scale factor is N 1/2 instead of N 1. If then V ar(z N ) = O(N 1 ) V ar(n 1/2 z N ) = O(1) The random sequence of normalized samle means {N 1/2 x N }, having E(N 1/2 x N ) = 0, neither degenerates to 0 in the limit nor diverge to infinite values, excet with robability 0. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 6
7 Lindberg-Lévy Theorem Lindberg-Lévy Theorem Let {z t } be i.i.d., with E(z t ) = 0 and E(zt 2 ) = σ 2 <. If then ω N = NzN σ ω N d ω N(0, 1). The variance must be finite. This is always a requirement for the CLT to hold. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 7
8 Asymtotic analysis of the MLR Let θ N an estimator, alied to a samle of size N, of a vector of arameters θ 0. θ N, θ 0 Θ R k. For technical reasons, Θ must be a comact subset (The set is bounded and closed). It is required that θ 0 be an interior oint of Θ: θ 0 int(θ). We say that θ 0 int(θ) if there exists a real number δ > 0 such that θ Θ whenever θ θ 0 < δ. This exclude θ 0 be on the boundary of the set. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 8
9 Consistency Definition (Consistency) θ N is said to be consistent for θ 0 if lim θ N = θ 0 ( θ N θ 0 ). Sketch of roof If the sequence E( θ N ) has a limit and lim V ar( θ N ) = 0 N then θ N converges in robability to this limit. If lim N E( θ N ) = θ 0 is asymtotically unbiased. Asymtotically unbiasedness and zero limiting variance are jointly sufficient for consistency. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 9
10 CAN estimator Suose θ N θ 0, and that N k ( θ N θ 0 ) = O (1) for some K > 0, and has a non-degenerate distribution as N. This distribution is called the asymtotic distribution of θ N. Definition (CAN estimator) θ N is said to be consistent and asymtotically normal (CAN) for θ 0 int(θ) if there exists k > 0 such that N k ( θ N θ 0 ) d N(0,V) where V is a finite variance matrix. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 10
11 CAN estimator In most alications K = 1/2 although it can be larger than this in models containing trend terms. Sometimes with stochastic trends, k > 1/2 but the limiting distribution is not Gaussian. Definition (Class of CAN estimators of θ 0 ) θ N C is said to be the best asymtotically normal estimator of θ 0 (BAN) in the class C if Avar( θ N ) Avar( θ N ) is.s.d. for θ N C. This roerty is called asymtotic efficiency. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 11
12 MLE Asymtotics The MLE is an imlicit function of the random samle. MLE is not a function of samle averages of the data. But the samle log-likelihood is a sum of i.i.d. random variables. Because the (U t, V t ) i.i.d. so are any such transformations as the L(θ) L(θ; U t V t ), t = 1, 2,...,N. The LLN can aly to the samle average log-likelihood function itself E N [L(θ)] for any fixed θ. E[L(θ)] Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 12
13 Consistency Under the assumtions 1. Distribution 2. Dominance 3. Global Identification 4. Comactness of Θ The MLE is consistent θ N θ0 Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 13
14 Consistency The samle average log-likelihood converges to the exected log-likelihood for any value of θ: E N [L(θ)] E[L(θ)] θ N = arg max θ Θ E N[L(θ)] θ 0 = arg max θ Θ E[L(θ)] by construction by strict log-likelihood inequality As a result, θ N θ0, rovided that the relationshis are continuous. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 14
15 Consistency The argument of arg max θ Θ is a function of θ, E N [L(θ)]. arg max θ Θ must be a continuous function of its functional argument. The distance between two functions over a set containing an infinite number of ossible comarisons at different values of θ: Uniform Convergence in Probability: The sequence of real-valued functions {g N (θ)} converges in robability to the limit function {g 0 (θ)} if su g N (θ) g 0 (θ) 0 θ Θ we say g N (θ) g 0 (θ) uniformly. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 15
16 Consistency We use the Uniform Convergence in Probability in order to define the robability limit of a sequence of random functions. Uniform LLN. g(θ, U) continuous function over θ Θ, where Θ R K is closed and bounded, {U t } is a sequence of i.i.d. r.v. with c.d.f. F U (u). If E[su θ Θ g(θ; U) ] exists, then 1. E[g(θ; U)] is continuous over θ Θ 2. E N [g(θ; U)] E[g(θ; U)] Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 16
17 Consistency We aly the uniform LLN to the samle average log-likelihood. Consistency of Maxima. If there is a sequence of functions Q N (θ) that converges in robability uniformly to a function Q 0 (θ) on the closed and bounded Θ and if Q 0 (θ) is continuous and uniquely maximized at θ 0, then θ N = arg max θ Θ Q N(θ) θ 0 Comactness and differentiability guarantee that E N [L(θ)] has a maximum. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 17
18 Consistency Let g(θ; U) L(θ; U V ) the conditional likelihood function for θ evaluated at the r.v. (U, V ). The conditions for uniform convergence are satisfied: Differentiability imlies continuity of L(θ) Comactness of Θ. (U t, V t ) are i.i.d. with c.d.f. F U V (u v; θ) Dominance states that E[su θ Θ L(θ) ] exists Then E[L(θ)] is continuous and uniformly. E N [L(θ)] E[L(θ)] Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 18
19 Consistency For the Consistency of Maxima Q N (θ) = E N [L(θ)] andq 0 (θ) = E[L(θ)]. Under the assumtions: From Likelihood Identification: if θ 1 Θ, θ 0 θ 1 imlies Pr{L(θ 0 ) L(θ 1 )} > 0 we have the Strict Exected Log-likelihood Inequality: θ θ 0 imlies E[L(θ)] < E[L(θ 0 )] Hence E[L(θ)] is uniquely maximized at θ 0. Therefore θ N = arg max θ Θ E N[L(θ)] θ 0 = arg max E[L(θ)] Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 19
20 Asymtotic Normality Assumtion: There is an oen subset of Θ that contains the oulation arameter value θ 0. θ 0 is not on the boundary of Θ. Assumtion: E N [L θ ( θ N )] = 0 the MLE solves the normal equations. First-order Taylor series exansion: E N [L θ ( θ N )] = 0 = E N [L θ (θ 0 )] + E N [L θθ (θ N )]( θ N θ 0 ) θ N = α N θn + (1 α N )θ 0 α N [0, 1] Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 20
21 Asymtotic Normality N( θn θ 0 ) = { E N [L θθ (θ N )]} 1 NE N [L θ (θ 0 )] NE N [L θ (θ 0 )] E N [L θθ (θ N )] d N(0, I(θ 0 )) (by CLT) I(θ 0 ) (by LLN) then, N( θ N θ 0 ) d N(0, I(θ 0 ) 1 ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 21
22 Asymtotic Normality NEN [L θ (θ 0 )] d N(0, I(θ 0 )) Proof. Since NEN [L θ (θ 0 )] = N 1 N L(θ) θ = 1 N L θ (θ 0 ; U t V t ) θ=θ0 N t=1 is a sum of i.i.d. r.v.. Under the assumtions of i.i.d. (U t, V t ) and differentiability, E[L θ (θ 0 ) V = v] = 0 so that E[c L θ (θ 0 )] = 0 c R K V ar[c L θ (θ 0 )] = c I(θ 0 )c exists c R K Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 22
23 Asymtotic Normality The CLT imlies NEN [c L θ (θ 0 )] d N(0,c I(θ 0 )c) by Cramér-Wold device: d NEN [L θ (θ 0 )] N(0, I(θ 0 )) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 23
24 Asymtotic Normality E N [L θθ (θ N )] where θ N θ0. I(θ 0 ) While θ N θ0 imlies g(θ N ) g(θ 0 ) for continuous g. g is a function of the random samle, therefore is random. Lemma 1. g N (θ) g 0 (θ), θ Θ, closed and bounded subset of R K. 2. g 0 (θ) is continuous 3. θ N θ0 Θ then g N (θ N ) g 0 (θ 0 ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 24
25 Asymtotic Normality Assumtion. (Dominance II) E[su θ Θ L θθ (θ) ] exists. Lemma. Under the assumtions 1. Differentiability 2. Finite Information: V ar[l θ (θ 0 )] exists. 3. Comactness: Θ is bounded and closed. 4. Dominance II: E[su θ Θ L θθ (θ) ] exists. E N [ Lθθ (θ N ) ] I(θ0 ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 25
26 Asymtotic Normality Proof 1. To establish that E N [L θθ (θ)] E[L θθ (θ)] uniformly. We note that Differentiability imlies that L θθ (θ) is continuous. (U t, V t ) are i.i.d. Θ is comact. E[su θ Θ L θθ (θ) ] exists. Then we can invoke the uniform LLN: E N [L θθ (θ)] E[L θθ (θ)] uniformly in θ Θ. 2. θ N θ0. Alying the Lemma E N [L θθ (θ N )] E[L θθ (θ 0 )]. The assumtions of Distribution, Differentiability and Finite Information are met so E[L θθ (θ 0 )] = I(θ 0 ). Then E N [L θθ (θ N )] I(θ 0 ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 26
27 Asymtotic Normality Now, N( θn θ 0 ) = { E N [L θθ (θ N )]} 1 NE N [L θ (θ 0 )] d I(θ 0 ) 1 N[0, I(θ 0 )] N(0, I(θ 0 ) 1 ) by Slutsky Square roots of nonsingular matrices are continuous functions of the elements of the matrix { E N [L θθ ( θ N )]} 1/2 I(θ 0 ) 1/2 Then by Slutsky s Theorem { E N [L θθ ( θ N )]} 1/2 N( θ N θ 0 ) d I(θ 0 ) 1/2 N[0, I(θ 0 ) 1 ] N[0,I K ]. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 27
28 Variance Estimation Three consistent estimators of the I(θ 0 ): E N [ L θθ ( θ N )] V ar N [L θ ( θ N )] E N [I( θ N )] To use the emirical information estimator the oulation information function I(θ) is known only when the log-likelihood is unconditional. When the log-likelihood is conditional then the conditional information function is I(θ 0 V ) = E[L θ (θ 0 ; U V )L θ (θ 0 ; U V ) V ] because the marginal distribution of V is unsecified I(θ) is unknown. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 28
29 Variance Estimation The unconditional information matrix is obtained using the LIE: I(θ 0 ) = E[I(θ 0 V )] the emirical information matrix estimator is E N [I( θ N V )]. Consistency of the estimators Î(θ) I(θ) uniformly. Because θ N θ0 Î( θ N ) I(θ 0 ) since the inverse is a continuous function Î( θ N ) 1 I(θ 0 ) 1. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 29
30 Efficiency The asymtotic distribution of MLE has no bias and the variance matrix of its asymtotic distribution equals the Cramér-Rao Lower Bound. The MLE and the efficient (but infeasible) Cramer-Rao estimator are asymtotically equivalent: N( θn θ 0 ) N(θ θ 0 ) = N( θ N θ ) 0 because {E N [L θθ (θ N )]} 1 I(θ 0 ) 1 d NEN [L θ (θ 0 )] N(0, I(θ 0 )) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 30
31 Efficiency θ θ 0 + [I(θ 0 )] 1 E N [L θ (θ 0 )] N( θn θ 0 ) N(θ θ 0 ) = N( θ N θ ) = N( θ N θ 0 ) N{E N [I(θ 0 )]} 1 E N [L θ (θ 0 )] = [ {E N [L θθ (θ N )]} 1 I(θ 0 ) 1] NEN [L θ (θ 0 )] 0 using the equivalence of convergence in distribution to a constant and convergence in robability. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 31
32 Invariance ML estimator is invariant to nonsingular transformations of the arameters. If γ = g(θ) is a one-to-one rearameterization then the MLE for γ is γ = arg max γ Γ E N {L[g 1 (γ)]} where Γ is the arameter sace {γ = g(θ) θ Θ}. Because the rearameterization is one-to-one max γ Γ E N {L[g 1 (γ)]} = max θ Θ E N[L(θ)] γ = g( θ) Invariance: Rearameterization does not alter the location of the MLE. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 32
33 Delta Method For finding the asymtotic distribution of a trasformation. Given a consistent estimator of the aroximate variance of N( θ N θ 0 ), Ω, the aroximate variance of N(g( θn ) g(θ 0 )) is g θ ( θ N ) Ωg θ ( θ N ). Delta Method: If N( θ N θ 0 ) d N(0,Ω) and g(θ) is continuous at θ 0 then N(g( θn ) g(θ 0 )) d N(0,J 0 ΩJ 0) J 0 g(θ) θ, J 0 J(θ 0 ). Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 33
34 Delta Method Proof: Exanding g( θ N ) around θ 0 we obtain N(g( θn ) g(θ 0 )) = J(θ N ) N( θ N θ 0 ) for some θ N = α N θn + (1 α N )θ 0 α N [0, 1] θ N θ 0 J(θ N ) J 0 (Slutsky s Theorem) N(g( θn ) g(θ 0 )) d J 0 N(0,Ω) N(0,J 0 ΩJ 0). Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 34
35 Asymtotics MLE Normal Linear Regression Model The emirical exectation of the log-likelihood E N [L(θ)] = 1 2 log (2πσ2 ) E N[(y t x tβ) 2 ] 2σ 2 = 1 2 log (2πσ2 ) (y Xβ) (y Xβ)/N 2σ 2 The log-lik is differentiable. F.O.C s: E N [L β (θ)] = 1 σ 2 E N[x t (y t x tβ)] = 1 Nσ 2 [X (y Xβ)] E N [L σ 2(θ)] = 1 2σ 4 {σ2 E N [(y t x tβ) 2 ]} = 1 [ σ 2 1 ] 2σ 4 N (y Xβ) (y Xβ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 35
36 Asymtotics MLE Normal Linear Regression Model Solutions: 1 Nσ 2 [X (y X β)] = 0 β = (X X) 1 X y σ 2 = 1 N (y Xβ) (y Xβ) The Hessian matrix: E N [L θθ (θ)] = 1 σ 2 N X X (y Xβ) X σ 4 N X (y Xβ) σ 4 N 1 2σ 1 4 σ 6 N (y Xβ) (y Xβ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 36
37 Asymtotics MLE Normal Linear Regression Model E N [L θθ ( θ N )] = = 1 σ 2 N X X X (y X β) σ 4 N (y X β) X σ 4 N 1 2 σ 1 4 σ 6 N (y X β) (y X β) 1 σ 2 N X X σ 1 4 σ 6 N (y X β) (y X β) which is negative definite. The MLE of σ 2 is σ N 2 = ǫ ǫ N = N K N s2 The second-order necessary condition for a oint to be the local maximum of a twice continuously differentiable function is that the Hessian be negative semidefinite at the oint. Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 37
38 Asymtotics MLE Normal Linear Regression Model E N [ L θθ ( θ N )] = 1 σ 2 N NX X σ 4 N = E N [I( θ N x t )] V ar N [L θ ( θ N )] = ¾ 1 σ 4 N ( ) 1/2 1 X X σ N N N( βn β) 1 2 σ 4 N( σ 2 N σ 2 ) N E N [x t (y t x t β N ) 2 x t] 1 σ 8 N d N(0,I K+1 ) 1 2 σ N 6 E N [x t (y t x t β N ) 3 ] {E N [(y t x t β N ) 4 ] σ 4 N} and N( βn β) d N(0, σ 2 E(x t x t) 1 ) Eduardo Rossi - Econometria er i mercati finanziari (avanzato) 38
Maximum Likelihood Estimation
University of Pavia Maximum Likelihood Estimation Eduardo Rossi Likelihood function Choosing parameter values that make what one has observed more likely to occur than any other parameter values do. Assumption(Distribution)
More information1 Extremum Estimators
FINC 9311-21 Financial Econometrics Handout Jialin Yu 1 Extremum Estimators Let θ 0 be a vector of k 1 unknown arameters. Extremum estimators: estimators obtained by maximizing or minimizing some objective
More informationLECTURE 7 NOTES. x n. d x if. E [g(x n )] E [g(x)]
LECTURE 7 NOTES 1. Convergence of random variables. Before delving into the large samle roerties of the MLE, we review some concets from large samle theory. 1. Convergence in robability: x n x if, for
More informationElements of Asymptotic Theory. James L. Powell Department of Economics University of California, Berkeley
Elements of Asymtotic Theory James L. Powell Deartment of Economics University of California, Berkeley Objectives of Asymtotic Theory While exact results are available for, say, the distribution of the
More informationUniversity of Pavia. M Estimators. Eduardo Rossi
University of Pavia M Estimators Eduardo Rossi Criterion Function A basic unifying notion is that most econometric estimators are defined as the minimizers of certain functions constructed from the sample
More informationElements of Asymptotic Theory. James L. Powell Department of Economics University of California, Berkeley
Elements of Asymtotic Theory James L. Powell Deartment of Economics University of California, Berkeley Objectives of Asymtotic Theory While exact results are available for, say, the distribution of the
More informationLecture 2: Consistency of M-estimators
Lecture 2: Instructor: Deartment of Economics Stanford University Preared by Wenbo Zhou, Renmin University References Takeshi Amemiya, 1985, Advanced Econometrics, Harvard University Press Newey and McFadden,
More informationChapter 7: Special Distributions
This chater first resents some imortant distributions, and then develos the largesamle distribution theory which is crucial in estimation and statistical inference Discrete distributions The Bernoulli
More informationIntroduction to Estimation Methods for Time Series models Lecture 2
Introduction to Estimation Methods for Time Series models Lecture 2 Fulvio Corsi SNS Pisa Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 2 SNS Pisa 1 / 21 Estimators:
More informationGARCH Models Estimation and Inference. Eduardo Rossi University of Pavia
GARCH Models Estimation and Inference Eduardo Rossi University of Pavia Likelihood function The procedure most often used in estimating θ 0 in ARCH models involves the maximization of a likelihood function
More informationLecture 3 January 16
Stats 3b: Theory of Statistics Winter 28 Lecture 3 January 6 Lecturer: Yu Bai/John Duchi Scribe: Shuangning Li, Theodor Misiakiewicz Warning: these notes may contain factual errors Reading: VDV Chater
More informationEconometrics I. September, Part I. Department of Economics Stanford University
Econometrics I Deartment of Economics Stanfor University Setember, 2008 Part I Samling an Data Poulation an Samle. ineenent an ientical samling. (i.i..) Samling with relacement. aroximates samling without
More informationEstimation of Dynamic Regression Models
University of Pavia 2007 Estimation of Dynamic Regression Models Eduardo Rossi University of Pavia Factorization of the density DGP: D t (x t χ t 1, d t ; Ψ) x t represent all the variables in the economy.
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationProblem Set 2 Solution
Problem Set 2 Solution Aril 22nd, 29 by Yang. More Generalized Slutsky heorem [Simle and Abstract] su L n γ, β n Lγ, β = su L n γ, β n Lγ, β n + Lγ, β n Lγ, β su L n γ, β n Lγ, β n + su Lγ, β n Lγ, β su
More informationEstimation, Inference, and Hypothesis Testing
Chapter 2 Estimation, Inference, and Hypothesis Testing Note: The primary reference for these notes is Ch. 7 and 8 of Casella & Berger 2. This text may be challenging if new to this topic and Ch. 7 of
More informationMATH 829: Introduction to Data Mining and Analysis Consistency of Linear Regression
1/9 MATH 829: Introduction to Data Mining and Analysis Consistency of Linear Regression Dominique Guillot Deartments of Mathematical Sciences University of Delaware February 15, 2016 Distribution of regression
More informationReview of Probability Theory II
Review of Probability Theory II January 9-3, 008 Exectation If the samle sace Ω = {ω, ω,...} is countable and g is a real-valued function, then we define the exected value or the exectation of a function
More information7. Introduction to Large Sample Theory
7. Introuction to Large Samle Theory Hayashi. 88-97/109-133 Avance Econometrics I, Autumn 2010, Large-Samle Theory 1 Introuction We looke at finite-samle roerties of the OLS estimator an its associate
More informationGeneral Linear Model Introduction, Classes of Linear models and Estimation
Stat 740 General Linear Model Introduction, Classes of Linear models and Estimation An aim of scientific enquiry: To describe or to discover relationshis among events (variables) in the controlled (laboratory)
More informationLecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN
Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and
More informationLecture 3 Consistency of Extremum Estimators 1
Lecture 3 Consistency of Extremum Estimators 1 This lecture shows how one can obtain consistency of extremum estimators. It also shows how one can find the robability limit of extremum estimators in cases
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2009 Prof. Gesine Reinert Our standard situation is that we have data x = x 1, x 2,..., x n, which we view as realisations of random
More informationTheory of Maximum Likelihood Estimation. Konstantin Kashin
Gov 2001 Section 5: Theory of Maximum Likelihood Estimation Konstantin Kashin February 28, 2013 Outline Introduction Likelihood Examples of MLE Variance of MLE Asymptotic Properties What is Statistical
More informationECON 4130 Supplementary Exercises 1-4
HG Set. 0 ECON 430 Sulementary Exercises - 4 Exercise Quantiles (ercentiles). Let X be a continuous random variable (rv.) with df f( x ) and cdf F( x ). For 0< < we define -th quantile (or 00-th ercentile),
More informationHypothesis Testing. 1 Definitions of test statistics. CB: chapter 8; section 10.3
Hypothesis Testing CB: chapter 8; section 0.3 Hypothesis: statement about an unknown population parameter Examples: The average age of males in Sweden is 7. (statement about population mean) The lowest
More informationOutline for today. Maximum likelihood estimation. Computation with multivariate normal distributions. Multivariate normal distribution
Outline for today Maximum likelihood estimation Rasmus Waageetersen Deartment of Mathematics Aalborg University Denmark October 30, 2007 the multivariate normal distribution linear and linear mixed models
More information1 Probability Spaces and Random Variables
1 Probability Saces and Random Variables 1.1 Probability saces Ω: samle sace consisting of elementary events (or samle oints). F : the set of events P: robability 1.2 Kolmogorov s axioms Definition 1.2.1
More informationSums of independent random variables
3 Sums of indeendent random variables This lecture collects a number of estimates for sums of indeendent random variables with values in a Banach sace E. We concentrate on sums of the form N γ nx n, where
More informationδ -method and M-estimation
Econ 2110, fall 2016, Part IVb Asymptotic Theory: δ -method and M-estimation Maximilian Kasy Department of Economics, Harvard University 1 / 40 Example Suppose we estimate the average effect of class size
More informationMathematical statistics
October 4 th, 2018 Lecture 12: Information Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation Chapter
More informationElementary Analysis in Q p
Elementary Analysis in Q Hannah Hutter, May Szedlák, Phili Wirth November 17, 2011 This reort follows very closely the book of Svetlana Katok 1. 1 Sequences and Series In this section we will see some
More informationSystem Reliability Estimation and Confidence Regions from Subsystem and Full System Tests
009 American Control Conference Hyatt Regency Riverfront, St. Louis, MO, USA June 0-, 009 FrB4. System Reliability Estimation and Confidence Regions from Subsystem and Full System Tests James C. Sall Abstract
More informationNotes on Instrumental Variables Methods
Notes on Instrumental Variables Methods Michele Pellizzari IGIER-Bocconi, IZA and frdb 1 The Instrumental Variable Estimator Instrumental variable estimation is the classical solution to the roblem of
More informationChapter 1: A Brief Review of Maximum Likelihood, GMM, and Numerical Tools. Joan Llull. Microeconometrics IDEA PhD Program
Chapter 1: A Brief Review of Maximum Likelihood, GMM, and Numerical Tools Joan Llull Microeconometrics IDEA PhD Program Maximum Likelihood Chapter 1. A Brief Review of Maximum Likelihood, GMM, and Numerical
More informationChapter 6. Convergence. Probability Theory. Four different convergence concepts. Four different convergence concepts. Convergence in probability
Probability Theory Chapter 6 Convergence Four different convergence concepts Let X 1, X 2, be a sequence of (usually dependent) random variables Definition 1.1. X n converges almost surely (a.s.), or with
More informationTsung-Lin Cheng and Yuan-Shih Chow. Taipei115,Taiwan R.O.C.
A Generalization and Alication of McLeish's Central Limit Theorem by Tsung-Lin Cheng and Yuan-Shih Chow Institute of Statistical Science Academia Sinica Taiei5,Taiwan R.O.C. hcho3@stat.sinica.edu.tw Abstract.
More information4. Score normalization technical details We now discuss the technical details of the score normalization method.
SMT SCORING SYSTEM This document describes the scoring system for the Stanford Math Tournament We begin by giving an overview of the changes to scoring and a non-technical descrition of the scoring rules
More informationECE531 Lecture 10b: Maximum Likelihood Estimation
ECE531 Lecture 10b: Maximum Likelihood Estimation D. Richard Brown III Worcester Polytechnic Institute 05-Apr-2011 Worcester Polytechnic Institute D. Richard Brown III 05-Apr-2011 1 / 23 Introduction So
More informationLarge Sample Theory. Consider a sequence of random variables Z 1, Z 2,..., Z n. Convergence in probability: Z n
Large Sample Theory In statistics, we are interested in the properties of particular random variables (or estimators ), which are functions of our data. In ymptotic analysis, we focus on describing the
More informationEvaluating the Performance of Estimators (Section 7.3)
Evaluating the Performance of Estimators (Section 7.3) Example: Suppose we observe X 1,..., X n iid N(θ, σ 2 0 ), with σ2 0 known, and wish to estimate θ. Two possible estimators are: ˆθ = X sample mean
More informationProbability Background
Probability Background Namrata Vaswani, Iowa State University August 24, 2015 Probability recap 1: EE 322 notes Quick test of concepts: Given random variables X 1, X 2,... X n. Compute the PDF of the second
More informationChapter 3. Point Estimation. 3.1 Introduction
Chapter 3 Point Estimation Let (Ω, A, P θ ), P θ P = {P θ θ Θ}be probability space, X 1, X 2,..., X n : (Ω, A) (IR k, B k ) random variables (X, B X ) sample space γ : Θ IR k measurable function, i.e.
More informationChapter 4: Asymptotic Properties of the MLE (Part 2)
Chapter 4: Asymptotic Properties of the MLE (Part 2) Daniel O. Scharfstein 09/24/13 1 / 1 Example Let {(R i, X i ) : i = 1,..., n} be an i.i.d. sample of n random vectors (R, X ). Here R is a response
More information1 Martingales. Martingales. (Ω, B, P ) is a probability space.
Martingales January 8, 206 Debdee Pati Martingales (Ω, B, P ) is a robability sace. Definition. (Filtration) filtration F = {F n } n 0 is a collection of increasing sub-σfields such that for m n, we have
More informationChapter 4: Asymptotic Properties of the MLE
Chapter 4: Asymptotic Properties of the MLE Daniel O. Scharfstein 09/19/13 1 / 1 Maximum Likelihood Maximum likelihood is the most powerful tool for estimation. In this part of the course, we will consider
More informationAsymptotic theory for linear regression and IV estimation
Asymtotic theory for linear regression and IV estimation Jean-Marie Dufour McGill University First version: November 20 Revised: December 20 his version: December 20 Comiled: December 3, 20, : his work
More informationEconomics 241B Review of Limit Theorems for Sequences of Random Variables
Economics 241B Review of Limit Theorems for Sequences of Random Variables Convergence in Distribution The previous de nitions of convergence focus on the outcome sequences of a random variable. Convergence
More informationMultivariate Analysis and Likelihood Inference
Multivariate Analysis and Likelihood Inference Outline 1 Joint Distribution of Random Variables 2 Principal Component Analysis (PCA) 3 Multivariate Normal Distribution 4 Likelihood Inference Joint density
More informationLecture 1: Introduction
Principles of Statistics Part II - Michaelmas 208 Lecturer: Quentin Berthet Lecture : Introduction This course is concerned with presenting some of the mathematical principles of statistical theory. One
More informationSection 8.2. Asymptotic normality
30 Section 8.2. Asymptotic normality We assume that X n =(X 1,...,X n ), where the X i s are i.i.d. with common density p(x; θ 0 ) P= {p(x; θ) :θ Θ}. We assume that θ 0 is identified in the sense that
More informationEstimating function analysis for a class of Tweedie regression models
Title Estimating function analysis for a class of Tweedie regression models Author Wagner Hugo Bonat Deartamento de Estatística - DEST, Laboratório de Estatística e Geoinformação - LEG, Universidade Federal
More informationSystem Identification, Lecture 4
System Identification, Lecture 4 Kristiaan Pelckmans (IT/UU, 2338) Course code: 1RT880, Report code: 61800 - Spring 2012 F, FRI Uppsala University, Information Technology 30 Januari 2012 SI-2012 K. Pelckmans
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationGraduate Econometrics I: Maximum Likelihood I
Graduate Econometrics I: Maximum Likelihood I Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Maximum Likelihood
More informationExtremal Polynomials with Varying Measures
International Mathematical Forum, 2, 2007, no. 39, 1927-1934 Extremal Polynomials with Varying Measures Rabah Khaldi Deartment of Mathematics, Annaba University B.P. 12, 23000 Annaba, Algeria rkhadi@yahoo.fr
More informationThe properties of L p -GMM estimators
The properties of L p -GMM estimators Robert de Jong and Chirok Han Michigan State University February 2000 Abstract This paper considers Generalized Method of Moment-type estimators for which a criterion
More informationEstimation of the large covariance matrix with two-step monotone missing data
Estimation of the large covariance matrix with two-ste monotone missing data Masashi Hyodo, Nobumichi Shutoh 2, Takashi Seo, and Tatjana Pavlenko 3 Deartment of Mathematical Information Science, Tokyo
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2008 Prof. Gesine Reinert 1 Data x = x 1, x 2,..., x n, realisations of random variables X 1, X 2,..., X n with distribution (model)
More informationElementary theory of L p spaces
CHAPTER 3 Elementary theory of L saces 3.1 Convexity. Jensen, Hölder, Minkowski inequality. We begin with two definitions. A set A R d is said to be convex if, for any x 0, x 1 2 A x = x 0 + (x 1 x 0 )
More informationEconometrics I, Estimation
Econometrics I, Estimation Department of Economics Stanford University September, 2008 Part I Parameter, Estimator, Estimate A parametric is a feature of the population. An estimator is a function of the
More informationInstrumental Variables
Università di Pavia 2010 Instrumental Variables Eduardo Rossi Exogeneity Exogeneity Assumption: the explanatory variables which form the columns of X are exogenous. It implies that any randomness in the
More informationEstimation of spatial autoregressive panel data models with xed e ects
Estimation of satial autoregressive anel data models with xed e ects Lung-fei Lee Deartment of Economics Ohio State University l eeecon.ohio-state.edu Jihai Yu Deartment of Economics University of Kentucky
More informationGraduate Econometrics I: Unbiased Estimation
Graduate Econometrics I: Unbiased Estimation Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Unbiased Estimation
More informationAdvanced Econometrics II (Part 1)
Advanced Econometrics II (Part 1) Dr. Mehdi Hosseinkouchack Goethe University Frankfurt Summer 2016 osseinkouchack (Goethe University Frankfurt) Review Slides Summer 2016 1 / 22 Distribution For simlicity,
More informationTheoretical Statistics. Lecture 1.
1. Organizational issues. 2. Overview. 3. Stochastic convergence. Theoretical Statistics. Lecture 1. eter Bartlett 1 Organizational Issues Lectures: Tue/Thu 11am 12:30pm, 332 Evans. eter Bartlett. bartlett@stat.
More informationTECHNICAL APPENDIX FOR GENERALIZED LEAST SQUARES INFERENCE IN PANEL AND MULTILEVEL MODELS WITH SERIAL CORRELATION AND FIXED EFFECTS
ECHNICAL APPENDIX FOR GENERALIZED LEA QUARE INFERENCE IN PANEL AND MULILEVEL MODEL WIH ERIAL CORRELAION AND FIXED EFFEC CHRIIAN B. HANEN Let C st = x st β + z st βs + v st, ) or, in vector notation, C
More informationECE 275A Homework 7 Solutions
ECE 275A Homework 7 Solutions Solutions 1. For the same specification as in Homework Problem 6.11 we want to determine an estimator for θ using the Method of Moments (MOM). In general, the MOM estimator
More informationEstimating Time-Series Models
Estimating ime-series Models he Box-Jenkins methodology for tting a model to a scalar time series fx t g consists of ve stes:. Decide on the order of di erencing d that is needed to roduce a stationary
More informationEconomics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
More informationNonconcave Penalized Likelihood with A Diverging Number of Parameters
Nonconcave Penalized Likelihood with A Diverging Number of Parameters Jianqing Fan and Heng Peng Presenter: Jiale Xu March 12, 2010 Jianqing Fan and Heng Peng Presenter: JialeNonconcave Xu () Penalized
More informationLower Confidence Bound for Process-Yield Index S pk with Autocorrelated Process Data
Quality Technology & Quantitative Management Vol. 1, No.,. 51-65, 15 QTQM IAQM 15 Lower onfidence Bound for Process-Yield Index with Autocorrelated Process Data Fu-Kwun Wang * and Yeneneh Tamirat Deartment
More informationParameter Estimation
Parameter Estimation Consider a sample of observations on a random variable Y. his generates random variables: (y 1, y 2,, y ). A random sample is a sample (y 1, y 2,, y ) where the random variables y
More informationStat 5101 Lecture Slides: Deck 7 Asymptotics, also called Large Sample Theory. Charles J. Geyer School of Statistics University of Minnesota
Stat 5101 Lecture Slides: Deck 7 Asymptotics, also called Large Sample Theory Charles J. Geyer School of Statistics University of Minnesota 1 Asymptotic Approximation The last big subject in probability
More informationQualifying Exam in Probability and Statistics. https://www.soa.org/files/edu/edu-exam-p-sample-quest.pdf
Part : Sample Problems for the Elementary Section of Qualifying Exam in Probability and Statistics https://www.soa.org/files/edu/edu-exam-p-sample-quest.pdf Part 2: Sample Problems for the Advanced Section
More informationMollifiers and its applications in L p (Ω) space
Mollifiers and its alications in L () sace MA Shiqi Deartment of Mathematics, Hong Kong Batist University November 19, 2016 Abstract This note gives definition of mollifier and mollification. We illustrate
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationOn-Line Appendix. Matching on the Estimated Propensity Score (Abadie and Imbens, 2015)
On-Line Aendix Matching on the Estimated Proensity Score Abadie and Imbens, 205 Alberto Abadie and Guido W. Imbens Current version: August 0, 205 The first art of this aendix contains additional roofs.
More informationOn split sample and randomized confidence intervals for binomial proportions
On slit samle and randomized confidence intervals for binomial roortions Måns Thulin Deartment of Mathematics, Usala University arxiv:1402.6536v1 [stat.me] 26 Feb 2014 Abstract Slit samle methods have
More informationLecture 4: Law of Large Number and Central Limit Theorem
ECE 645: Estimation Theory Sring 2015 Instructor: Prof. Stanley H. Chan Lecture 4: Law of Large Number and Central Limit Theorem (LaTeX reared by Jing Li) March 31, 2015 This lecture note is based on ECE
More informationUse of Transformations and the Repeated Statement in PROC GLM in SAS Ed Stanek
Use of Transformations and the Reeated Statement in PROC GLM in SAS Ed Stanek Introduction We describe how the Reeated Statement in PROC GLM in SAS transforms the data to rovide tests of hyotheses of interest.
More informationElements of statistics (MATH0487-1)
Elements of statistics (MATH0487-1) Prof. Dr. Dr. K. Van Steen University of Liège, Belgium November 12, 2012 Introduction to Statistics Basic Probability Revisited Sampling Exploratory Data Analysis -
More informationLecture 3 September 1
STAT 383C: Statistical Modeling I Fall 2016 Lecture 3 September 1 Lecturer: Purnamrita Sarkar Scribe: Giorgio Paulon, Carlos Zanini Disclaimer: These scribe notes have been slightly proofread and may have
More informationMoment Generating Function. STAT/MTHE 353: 5 Moment Generating Functions and Multivariate Normal Distribution
Moment Generating Function STAT/MTHE 353: 5 Moment Generating Functions and Multivariate Normal Distribution T. Linder Queen s University Winter 07 Definition Let X (X,...,X n ) T be a random vector and
More informationSystem Identification, Lecture 4
System Identification, Lecture 4 Kristiaan Pelckmans (IT/UU, 2338) Course code: 1RT880, Report code: 61800 - Spring 2016 F, FRI Uppsala University, Information Technology 13 April 2016 SI-2016 K. Pelckmans
More informationLecture 6. 2 Recurrence/transience, harmonic functions and martingales
Lecture 6 Classification of states We have shown that all states of an irreducible countable state Markov chain must of the same tye. This gives rise to the following classification. Definition. [Classification
More informationStat 5102 Lecture Slides Deck 3. Charles J. Geyer School of Statistics University of Minnesota
Stat 5102 Lecture Slides Deck 3 Charles J. Geyer School of Statistics University of Minnesota 1 Likelihood Inference We have learned one very general method of estimation: method of moments. the Now we
More informationOn Z p -norms of random vectors
On Z -norms of random vectors Rafa l Lata la Abstract To any n-dimensional random vector X we may associate its L -centroid body Z X and the corresonding norm. We formulate a conjecture concerning the
More informationSpring 2012 Math 541B Exam 1
Spring 2012 Math 541B Exam 1 1. A sample of size n is drawn without replacement from an urn containing N balls, m of which are red and N m are black; the balls are otherwise indistinguishable. Let X denote
More informationAsymptotically Optimal Simulation Allocation under Dependent Sampling
Asymtotically Otimal Simulation Allocation under Deendent Samling Xiaoing Xiong The Robert H. Smith School of Business, University of Maryland, College Park, MD 20742-1815, USA, xiaoingx@yahoo.com Sandee
More informationKhinchine inequality for slightly dependent random variables
arxiv:170808095v1 [mathpr] 7 Aug 017 Khinchine inequality for slightly deendent random variables Susanna Sektor Abstract We rove a Khintchine tye inequality under the assumtion that the sum of Rademacher
More informationIntroduction to Stochastic processes
Università di Pavia Introduction to Stochastic processes Eduardo Rossi Stochastic Process Stochastic Process: A stochastic process is an ordered sequence of random variables defined on a probability space
More informationSolution: (Course X071570: Stochastic Processes)
Solution I (Course X071570: Stochastic Processes) October 24, 2013 Exercise 1.1: Find all functions f from the integers to the real numbers satisfying f(n) = 1 2 f(n + 1) + 1 f(n 1) 1. 2 A secial solution
More informationELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)
1 ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE) Jingxian Wu Department of Electrical Engineering University of Arkansas Outline Minimum Variance Unbiased Estimators (MVUE)
More informationB8.1 Martingales Through Measure Theory. Concept of independence
B8.1 Martingales Through Measure Theory Concet of indeendence Motivated by the notion of indeendent events in relims robability, we have generalized the concet of indeendence to families of σ-algebras.
More informationMaximum Likelihood Estimation
Chapter 8 Maximum Likelihood Estimation 8. Consistency If X is a random variable (or vector) with density or mass function f θ (x) that depends on a parameter θ, then the function f θ (X) viewed as a function
More informationNotes on Random Vectors and Multivariate Normal
MATH 590 Spring 06 Notes on Random Vectors and Multivariate Normal Properties of Random Vectors If X,, X n are random variables, then X = X,, X n ) is a random vector, with the cumulative distribution
More informationStatistical Methods for Handling Incomplete Data Chapter 2: Likelihood-based approach
Statistical Methods for Handling Incomplete Data Chapter 2: Likelihood-based approach Jae-Kwang Kim Department of Statistics, Iowa State University Outline 1 Introduction 2 Observed likelihood 3 Mean Score
More informationColin Cameron: Asymptotic Theory for OLS
Colin Cameron: Asymtotic Theory for OLS. OLS Estimator Proerties an Samling Schemes.. A Roama Consier the OLS moel with just one regressor y i = βx i + u i. The OLS estimator b β = ³ P P i= x iy i canbewrittenas
More information