Chapter 4: Asymptotic Properties of the MLE (Part 2)
|
|
- Molly Johnston
- 5 years ago
- Views:
Transcription
1 Chapter 4: Asymptotic Properties of the MLE (Part 2) Daniel O. Scharfstein 09/24/13 1 / 1
2 Example Let {(R i, X i ) : i = 1,..., n} be an i.i.d. sample of n random vectors (R, X ). Here R is a response indicator and X is a covariate. We assume that logit P[R = 1 X ] = α + βx and assume that X is normally distributed with mean µ and variance σ 2. So, our probability model has four parameters, θ = (α, β, µ, σ 2 ). Let θ 0 = (α 0, β 0, µ 0, σ0 2 ) denote the true value of θ. 2 / 1
3 Likelihood L(θ; x, r) = n i=1 1 σ 2π exp( 1 2σ 2 (x i µ) 2 ) exp(r i(α + βx i )) 1 + exp(α + βx i ) The log-likelihood for an individual is l(θ; x, r) log(σ) 1 2σ 2 (x µ)2 +r(α+βx) log(1+exp(α+βx)) 3 / 1
4 Score The score vector for an individual is r ψ(x, r; θ) = = x(r l(θ;x,r) α l(θ;x,r) β l(θ;x,r) µ l(θ;x,r) σ 2 exp(α+βx) 1+exp(α+βx) exp(α+βx) 1+exp(α+βx) ) x µ σ (x µ)2 2σ 2 2σ 4 The score equations for the full sample are: n r i exp(α + βx i) 1 + exp(α + βx i ) = 0 (1) i=1 n i=1 n i=1 n i=1 x i (r i exp(α + βx i) 1 + exp(α + βx i ) ) = 0 (2) x i µ σ 2 = 0 (3) 1 2σ 2 + (x i µ) 2 2σ 4 = 0 (4) 4 / 1
5 MLE Note that Equations (3) and (4) can be solved explicitly to get solutions for ˆµ and ˆσ 2, i.e., ˆµ = 1 n n x i and ˆσ 2 = 1 n i=1 n (x i ˆµ) 2 i=1 The solutions for ˆα and ˆβ are obtained by solving Equations (1) and (2). Requires iterative technique such as Newton-Raphson (more later). 5 / 1
6 Observed Information Matrix Let p i (α, β) = exp(α+βx i ) 1+exp(α+βx i ) and q i(α, β) = 1 p i (α, β), Now, nj n (θ) = [ Jn1 (α, β) 0 0 J n2 (µ, σ 2 ) ] where [ n J n1 (α, β) = i=1 p i(α, β)q i (α, β) n i=1 x ip i (α, β)q i (α, β) n i=1 x ip i (α, β)q i (α, β) n i=1 x i 2p i(α, β)q i (α, β) ] and J n2 (µ, σ 2 ) = [ n 1 σ 1 2 σ 4 n i=1 (x i µ) n σ 4 i=1 (x i µ) n + 3 n 2σ 3 2σ 5 i=1 (x i µ) 2 ] 6 / 1
7 Assumptions Assume that α K 1, β K 2, µ K 3, and K 4 σ 2 K 5, where K 1,..., K 5 are finite positive quantities. Assume that θ 0 (truth) belongs to the interior of Θ. 7 / 1
8 Regularity Conditions Condition i) is satisfied by assumption. Condition ii) is satisfied since log p(x, r; θ) is continuous at each θ for all < x < and r = 0, 1. Now, we need to show condition iii) that log p(x, r; θ) d(x, r) for all θ Θ and E θ0 [d(x, R)] <. log p(x, r; θ) = 1 1 log(2π) log(σ) 2 2σ (x 2 µ)2 + r(α + βx) log(1 + exp(α + βx)) By the triangle inequality, we know that log p(x, r; θ) 1 2 log(2π) + log(σ) + 1 2σ (x 2 µ)2 + r(α + βx) + log(1 + exp(α + βx)) 8 / 1
9 Inequalities x 1 + x 2 log(1 + exp(α + βx)) 0 log(1 + exp(α + βx)) log(1 + exp( α + β x )) 1 + log(exp( α + β x )) = 1 + α + β x 1 + K 1 + K 2 (1 + x 2 ) log(σ) 1 2 max( log(k 4), log(k 5 ) ) 1 2σ 2 (x µ)2 1 2σ 2 x µ2 µ x + σ2 2σ 2 x 2 + K 3 (1 + x 2 ) + K 3 2 2K 4 K 4 2K 4 9 / 1
10 Inequalities r(α + βx) α + β x K 1 + K 2 (1 + x 2 ) log(1 + exp(α + βx)) 1 + K 1 + K 2 (1 + x 2 ) 10 / 1
11 Regularity Conditions Let d(x, r) = 1 2 log(2π) max( log(k 4), log(k 5 ) ) + x 2 2K 4 + K 3 K 4 (1 + x 2 ) + K 2 3 2K 4 + K 1 + K 2 (1 + x 2 ) K 1 + K 2 (1 + x 2 ) Now, E θ0 [X 2 ] = µ σ2 0 <. This implies that E θ0 [d(x, R)] <. 11 / 1
12 Regularity Conditions For condition iv), we have to show that p(x, r; θ) is twice-continuously differentiable and p(x, r; θ) > 0 for θ in a neighborhood of θ 0. This follows simply from the fact that the density is made up of polynomial and exponentials for which derivatives exists. The density is clearly positive over the entire parameter space. 12 / 1
13 Regularity Conditions For condition v), we have to show that p(x,r;θ) θ e(x, r), where e(x, r)is integrable. Remember, and p(x, r; θ) = 1 σ 2π exp( 1 2σ 2 (x exp(r(α + βx)) µ)2 ) 1 + exp(α + βx) p(x, r; θ) θ = 1 σ exp( 1 2π 1 σ exp( 1 2π 1 σ exp( 1 2π 2σ 2 (x µ) 2 ) 2σ 2 (x µ) 2 ) r exp(r(α+βx))+(r 1) exp((r+1)(α+βx) (1+exp(α+βx)) 2 xr exp(r(α+βx))+x(r 1) exp((r+1)(α+βx) (1+exp(α+βx)) 2 (1+exp(α+βx)) { x µ 2 σ } 2 2σ (x µ) 2 ) exp(r(α+βx)) 2 1 σ exp( 1 2π 2σ (x µ) 2 ) exp(r(α+βx)) 2 (1+exp(α+βx)) { 1 2 2σ 2 + (x µ)2 σ 2 } 13 / 1
14 Regularity Conditions Now, p(x, r; θ) θ p(x, r; θ) + α p(x, r; θ) + β p(x, r; θ) p(x, r; θ) + µ σ 2 We want to bound each of the terms in the sum by a function of x and r which is integrable. To do this, note that 1 σ 2π exp( 1 2σ 2 (x µ)2 ) e (x) where e (x) = 1 2K4π 1 exp( 1 2K4π 1 exp( 1 2K4π 2K 5 (x K 3 ) 2 ) x > K 3 K 3 x K 3 2K 5 (x + K 3 ) 2 ) x < K 3 14 / 1
15 Regularity Conditions Now, p(x, r; θ) α e 1 (x) = 2e (x) p(x, r; θ) β e 2 (x) = 2xe (x) p(x, r; θ) µ e 3 (x) = e (x)( x + K 3 K4 2 ) p(x, r; θ) σ 2 e 4 (x) = e (x)( 1 2K5 2 + x 2 (1 + 2K 3 ) + 2K 3 + K 2 3 K 2 4 We know that p(x,r;θ) θ e(x) = e 1 (x) + e 2 (x) + e 3 (x) + e 4 (x). e(x) is integrable since each e i (x) (i = 1,..., 4) is integrable. ) 15 / 1
16 Regularity Conditions For condition vi), we need to show that I (θ 0 ) = E θ0 [ψ(x, R; θ 0 )ψ(x, R; θ 0 ) ] is non-singular. It would suffice to show that I (θ 0 ) was positive definite. That is, a I (θ 0 )a > 0 for all a such that a > 0. Alternatively, we can show that Var θ0 [a ψ(x, R; θ 0 )] > 0. Now, a ψ(x, R; θ 0 ) = a 1 (R exp(α 0 + β 0 X ) 1 + exp(α 0 + β 0 X ) ) + a 2 X (R exp(α 0 + β 0 X ) 1 + exp(α 0 + β 0 X ) ) + a 3 ( X µ 0 σ0 2 ) + a 4 ( 1 2σ0 2 + (X µ 0) 2 σ0 4 ) Var θ0 [a ψ(x, R; θ 0)] = E θ0 [Var θ0 [a ψ(x, R; θ 0) X ]]+Var θ0 [E θ0 [a ψ(x, R; θ 0) X ]] 16 / 1
17 Regularity Conditions Var θ0 [a ψ(x, R; θ 0 ) X ] = (a 1 + a 2 X ) 2 exp(α 0 + β 0 X ) (1 + exp(α 0 + β 0 X )) 2 E θ0 [a ψ(x, R; θ 0 ) X ] = a 3 ( X µ 0 σ0 2 ) + a 4 ( 1 2σ0 2 + (X µ 0) 2 σ0 4 ) Var θ0 [a ψ(x, R; θ 0 )] = E θ0 [(a 1 + a 2 X ) 2 exp(α 0 + β 0 X ) (1 + exp(α 0 + β 0 X )) 2 ] + Var θ0 [a 3 ( X µ 0 σ0 2 ) + a 4 ( 1 2σ0 2 + (X µ 0) 2 σ0 4 )] = E θ0 [(a 1 + a 2 X ) 2 exp(α 0 + β 0 X ) (1 + exp(α 0 + β 0 X )) 2 ] + a3 2 σ0 2 + a2 4 4σ 4 0 The variance can only equal zero if a 1 = a 2 = a 3 = a 4 = 0. So, I (θ 0 ) is positive definite. 17 / 1
18 Regularity Conditions Conditions vii) and viii) involve showing that the norm of the matrix of second derivatives of the log likelihood and likelihood are bounded by functions of x and r which are integrable. The norm of a matrix A = [a ij ] is equal to ( i,j a2 ij )1/2. For each of these matrices, it is sufficient to show that the absolute value of each of entries is bounded by an integrable function of x and r. The bounding techniques are very similar to those used to show condition v). 18 / 1
19 Asymptotic Variance of θ n I (θ 0 ) 1 = exp(α E θ0 [ 0+β 0X ) (1+exp(α 0+β 0X )) ] 2 E θ0 [X exp(α0+β0x ) (1+exp(α 0+β 0X )) ] E θ0 [X exp(α0+β0x ) (1+exp(α 0+β 0X )) ] 2 E θ0 [X 2 exp(α 0+β 0X ) (1+exp(α 0+β 0X )) ] σ σ / 1
20 Regular Estimators Suppose we want to estimate a real-valued function of θ, say g(θ), g( ) : R k R. Assume that g has continuous partial derivatives. Consider estimating g(θ 0 ) by g(ˆθ(x n )), where ˆθ(X n ) is the MLE. Assuming that the 8 regularity conditions hold, we know that n(ˆθ(x n ) θ 0 ) D N(0, I 1 (θ 0 )). By the multivariate delta method, we know that n(g(ˆθ(x n )) g(θ 0 )) D N(0, a(θ 0 ) I 1 (θ 0 )a(θ 0 )) where a(θ 0 ) is the gradient of g(θ) at θ / 1
21 Regular Estimators Consider an estimator, T (X n ), of g(θ 0 ) which is asymptotically normal and asymptotically unbiased, i.e., n(t (Xn ) g(θ 0 )) D N(0, V (θ 0 )) (5) It turns out that under some additional regularity conditions on T (X n ), we can shown that V (θ 0 ) a(θ 0 ) I 1 (θ 0 )a(θ 0 ) (6) 21 / 1
22 Regular Estimators Definition: A regular estimator T (X n ) of g(θ 0 ) which satisfies (5) with V (θ 0 ) = a(θ 0 ) I 1 (θ 0 )a(θ 0 ) is said to be asymptotically efficient. If g(ˆθ(x n )) is regular, then we know that it is asymptotically efficient. For a long time, it was believed that the regularity conditions needed to make (6) hold involved regularity conditions on the density p(x; θ). This belief was exploded by Hodges. 22 / 1
23 Hodges Example of Super-Efficiency Suppose that X n = (X 1,..., X n ) where the X i s are i.i.d. Normal(µ 0, 1). We can show that I (µ 0 ) = 1 for all µ 0. Consider the following estimator of µ 0, { X ˆµ(X n ) = n X n n 1/4 0 X n < n 1/4 Suppose that µ 0 0. Then n(ˆµ(xn ) µ 0 ) = n(ˆµ(x n ) X n ) + (7) n( Xn µ 0 ) (8) By the CLT, we know that (8) converges in distribution to a N(0, 1) random variable. We can show that (7) converges in probability to zero. By Slutsky s theorem, we know that n(ˆµ(xn ) µ 0 ) D N(0, 1). So the estimator attains the efficiency bound. 23 / 1
24 Hodges Example of Super-Efficiency To show that (7) converges in probability to zero, note that P µ0 [ n(ˆµ(x n) X n) = 0] = P µ0 [ˆµ(X n) = X n] = P µ0 [ X n n 1/4 ] = P µ0 [ X n n 1/4 ] + P µ0 [ X n n 1/4 ] = P µ0 [ n( X n µ 0) n(n 1/4 µ 0)] + P µ0 [ n( X n µ 0) n( n 1/4 µ 0)] = Φ( n(n 1/4 µ 0)) + 1 Φ( n( n 1/4 µ 0)) = Φ(n 1/4 n 1/2 µ 0) + 1 Φ( n 1/4 n 1/2 µ 0) 1 24 / 1
25 Hodges Example of Super-Efficiency If µ 0 = 0, then we can show that n(ˆµ(x n ) µ 0 ) P 0 or n(ˆµ(xn ) µ 0 ) D N(0, 0). This violates (6). To show this result, we note that P µ0 [ n(ˆµ(x n ) µ 0 ) = 0] = P µ0 [ˆµ(X n ) = 0] = P µ0 [ X n < n 1/4 ] = P µ0 [ n 1/4 < X n < n 1/4 ] = P µ0 [ n 1/4 < n X n < n 1/4 ] 1 25 / 1
26 Remarks about Super-Efficiency The problem here is not due to irregularities of the density function, but due to the partialness of our estimator to µ 0 = 0. This example shows that no regularity conditions on the density can prevent an estimator from violating (6). This possibility can only be avoided by placing restrictions on the sequence of estimators. LeCam (1953) showed that for any sequence of estimators satisfying (5), the set of points in Θ violating (6) has Lebesgue measure zero. 26 / 1
27 Regular Estimators When you first discussed estimators, it was argued that it was important to rule out impartial estimators, i.e., estimators which favored some values of the parameters over others. In an asymptotic sense, we may want our sequence of estimators to be impartial so that we rule out estimators like the one presented by Hodges. Toward this end, we may restrict ourselves to regular estimators. A regular sequence of estimators is one whose asymptotic distribution remains the same in shrinking neighborhoods of the true parameter value. 27 / 1
28 Regular Estimators More formally. consider a sequence of estimators {T (X n )} and a sequence of parameter values {θ n } so that n(θ n θ 0 ) is bounded. T (X n ) is a regular sequence of estimators if converges to the same limit as P θn [ n(t (X n ) g(θ n )) a] P θ0 [ n(t (X n ) g(θ 0 )) a] for all a. For estimators that are regular and satisfy (2), (3) holds. When we talk about influence functions, we will formally establish this result. 28 / 1
29 Is the Hodge s Estimator Regular? Suppose that µ 0 = 0. We know that the limiting distribution of n(ˆµ(xn ) µ 0 ) is a degenerate distribution with point mass at zero. Consider the sequence µ n = τ/ n, where τ is some positive constant. Note that n(µ n µ 0 ) τ. Now, we can show that n(ˆµ(xn ) µ n ) P(µn) τ. To see this, not that P µn [ n( ˆµ(X n) µ n) = τ] = P µn [ n( ˆµ(X n) τ/ n) = τ] = P µn [ ˆµ(X n) = 0] = P µn [ X n < n 1/4 ] = P µn [ n 1/4 < X n < n 1/4 ] = P µn [ n( n 1/4 µ n) < n( X n µ n) < n(n 1/4 µ n)] = Φ( n( n 1/4 µ n)) Φ( n(n 1/4 µ n)) = Φ( n 1/4 τ) Φ(n 1/4 τ) 1 29 / 1
30 Is the Hodge s Estimator Regular? So, the limiting distribution of n(ˆµ(x n ) µ n ) is also degenerate, but with point mass at τ. This is a different limiting distribution than that of n(ˆµ(x n ) µ 0 ). Therefore, ˆµ(X n ) is not regular. 30 / 1
Chapter 4: Asymptotic Properties of the MLE
Chapter 4: Asymptotic Properties of the MLE Daniel O. Scharfstein 09/19/13 1 / 1 Maximum Likelihood Maximum likelihood is the most powerful tool for estimation. In this part of the course, we will consider
More informationSection 8.2. Asymptotic normality
30 Section 8.2. Asymptotic normality We assume that X n =(X 1,...,X n ), where the X i s are i.i.d. with common density p(x; θ 0 ) P= {p(x; θ) :θ Θ}. We assume that θ 0 is identified in the sense that
More informationMaximum Likelihood Estimation
Chapter 8 Maximum Likelihood Estimation 8. Consistency If X is a random variable (or vector) with density or mass function f θ (x) that depends on a parameter θ, then the function f θ (X) viewed as a function
More informationFor iid Y i the stronger conclusion holds; for our heuristics ignore differences between these notions.
Large Sample Theory Study approximate behaviour of ˆθ by studying the function U. Notice U is sum of independent random variables. Theorem: If Y 1, Y 2,... are iid with mean µ then Yi n µ Called law of
More informationLecture 32: Asymptotic confidence sets and likelihoods
Lecture 32: Asymptotic confidence sets and likelihoods Asymptotic criterion In some problems, especially in nonparametric problems, it is difficult to find a reasonable confidence set with a given confidence
More informationP n. This is called the law of large numbers but it comes in two forms: Strong and Weak.
Large Sample Theory Large Sample Theory is a name given to the search for approximations to the behaviour of statistical procedures which are derived by computing limits as the sample size, n, tends to
More informationStatistics 3858 : Maximum Likelihood Estimators
Statistics 3858 : Maximum Likelihood Estimators 1 Method of Maximum Likelihood In this method we construct the so called likelihood function, that is L(θ) = L(θ; X 1, X 2,..., X n ) = f n (X 1, X 2,...,
More informationMaximum Likelihood Estimation
Chapter 7 Maximum Likelihood Estimation 7. Consistency If X is a random variable (or vector) with density or mass function f θ (x) that depends on a parameter θ, then the function f θ (X) viewed as a function
More informationLecture 1: Introduction
Principles of Statistics Part II - Michaelmas 208 Lecturer: Quentin Berthet Lecture : Introduction This course is concerned with presenting some of the mathematical principles of statistical theory. One
More informationSection 10: Role of influence functions in characterizing large sample efficiency
Section 0: Role of influence functions in characterizing large sample efficiency. Recall that large sample efficiency (of the MLE) can refer only to a class of regular estimators. 2. To show this here,
More informationHT Introduction. P(X i = x i ) = e λ λ x i
MODS STATISTICS Introduction. HT 2012 Simon Myers, Department of Statistics (and The Wellcome Trust Centre for Human Genetics) myers@stats.ox.ac.uk We will be concerned with the mathematical framework
More informationMath 181B Homework 1 Solution
Math 181B Homework 1 Solution 1. Write down the likelihood: L(λ = n λ X i e λ X i! (a One-sided test: H 0 : λ = 1 vs H 1 : λ = 0.1 The likelihood ratio: where LR = L(1 L(0.1 = 1 X i e n 1 = λ n X i e nλ
More informationSpring 2012 Math 541A Exam 1. X i, S 2 = 1 n. n 1. X i I(X i < c), T n =
Spring 2012 Math 541A Exam 1 1. (a) Let Z i be independent N(0, 1), i = 1, 2,, n. Are Z = 1 n n Z i and S 2 Z = 1 n 1 n (Z i Z) 2 independent? Prove your claim. (b) Let X 1, X 2,, X n be independent identically
More informationStatement: With my signature I confirm that the solutions are the product of my own work. Name: Signature:.
MATHEMATICAL STATISTICS Homework assignment Instructions Please turn in the homework with this cover page. You do not need to edit the solutions. Just make sure the handwriting is legible. You may discuss
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2009 Prof. Gesine Reinert Our standard situation is that we have data x = x 1, x 2,..., x n, which we view as realisations of random
More information5.2 Fisher information and the Cramer-Rao bound
Stat 200: Introduction to Statistical Inference Autumn 208/9 Lecture 5: Maximum likelihood theory Lecturer: Art B. Owen October 9 Disclaimer: These notes have not been subjected to the usual scrutiny reserved
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationLecture 28: Asymptotic confidence sets
Lecture 28: Asymptotic confidence sets 1 α asymptotic confidence sets Similar to testing hypotheses, in many situations it is difficult to find a confidence set with a given confidence coefficient or level
More informationProblem Selected Scores
Statistics Ph.D. Qualifying Exam: Part II November 20, 2010 Student Name: 1. Answer 8 out of 12 problems. Mark the problems you selected in the following table. Problem 1 2 3 4 5 6 7 8 9 10 11 12 Selected
More informationStatistics & Data Sciences: First Year Prelim Exam May 2018
Statistics & Data Sciences: First Year Prelim Exam May 2018 Instructions: 1. Do not turn this page until instructed to do so. 2. Start each new question on a new sheet of paper. 3. This is a closed book
More informationGraduate Econometrics I: Maximum Likelihood I
Graduate Econometrics I: Maximum Likelihood I Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Maximum Likelihood
More informationLecture 3 September 1
STAT 383C: Statistical Modeling I Fall 2016 Lecture 3 September 1 Lecturer: Purnamrita Sarkar Scribe: Giorgio Paulon, Carlos Zanini Disclaimer: These scribe notes have been slightly proofread and may have
More informationPh.D. Qualifying Exam Friday Saturday, January 3 4, 2014
Ph.D. Qualifying Exam Friday Saturday, January 3 4, 2014 Put your solution to each problem on a separate sheet of paper. Problem 1. (5166) Assume that two random samples {x i } and {y i } are independently
More informationf(x θ)dx with respect to θ. Assuming certain smoothness conditions concern differentiating under the integral the integral sign, we first obtain
0.1. INTRODUCTION 1 0.1 Introduction R. A. Fisher, a pioneer in the development of mathematical statistics, introduced a measure of the amount of information contained in an observaton from f(x θ). Fisher
More informationOptimization. The value x is called a maximizer of f and is written argmax X f. g(λx + (1 λ)y) < λg(x) + (1 λ)g(y) 0 < λ < 1; x, y X.
Optimization Background: Problem: given a function f(x) defined on X, find x such that f(x ) f(x) for all x X. The value x is called a maximizer of f and is written argmax X f. In general, argmax X f may
More informationNotes on the Multivariate Normal and Related Topics
Version: July 10, 2013 Notes on the Multivariate Normal and Related Topics Let me refresh your memory about the distinctions between population and sample; parameters and statistics; population distributions
More informationChapter 2: Fundamentals of Statistics Lecture 15: Models and statistics
Chapter 2: Fundamentals of Statistics Lecture 15: Models and statistics Data from one or a series of random experiments are collected. Planning experiments and collecting data (not discussed here). Analysis:
More informationPh.D. Qualifying Exam Friday Saturday, January 6 7, 2017
Ph.D. Qualifying Exam Friday Saturday, January 6 7, 2017 Put your solution to each problem on a separate sheet of paper. Problem 1. (5106) Let X 1, X 2,, X n be a sequence of i.i.d. observations from a
More information[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
More informationEconometrics I, Estimation
Econometrics I, Estimation Department of Economics Stanford University September, 2008 Part I Parameter, Estimator, Estimate A parametric is a feature of the population. An estimator is a function of the
More informationSTAT 730 Chapter 4: Estimation
STAT 730 Chapter 4: Estimation Timothy Hanson Department of Statistics, University of South Carolina Stat 730: Multivariate Analysis 1 / 23 The likelihood We have iid data, at least initially. Each datum
More informationAsymptotic Statistics-III. Changliang Zou
Asymptotic Statistics-III Changliang Zou The multivariate central limit theorem Theorem (Multivariate CLT for iid case) Let X i be iid random p-vectors with mean µ and and covariance matrix Σ. Then n (
More informationBIO5312 Biostatistics Lecture 13: Maximum Likelihood Estimation
BIO5312 Biostatistics Lecture 13: Maximum Likelihood Estimation Yujin Chung November 29th, 2016 Fall 2016 Yujin Chung Lec13: MLE Fall 2016 1/24 Previous Parametric tests Mean comparisons (normality assumption)
More informationThe Expectation-Maximization Algorithm
1/29 EM & Latent Variable Models Gaussian Mixture Models EM Theory The Expectation-Maximization Algorithm Mihaela van der Schaar Department of Engineering Science University of Oxford MLE for Latent Variable
More informationIntroduction to Normal Distribution
Introduction to Normal Distribution Nathaniel E. Helwig Assistant Professor of Psychology and Statistics University of Minnesota (Twin Cities) Updated 17-Jan-2017 Nathaniel E. Helwig (U of Minnesota) Introduction
More informationGraduate Econometrics I: Maximum Likelihood II
Graduate Econometrics I: Maximum Likelihood II Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Maximum Likelihood
More informationMATH 829: Introduction to Data Mining and Analysis Graphical Models III - Gaussian Graphical Models (cont.)
1/12 MATH 829: Introduction to Data Mining and Analysis Graphical Models III - Gaussian Graphical Models (cont.) Dominique Guillot Departments of Mathematical Sciences University of Delaware May 6, 2016
More information13. Parameter Estimation. ECE 830, Spring 2014
13. Parameter Estimation ECE 830, Spring 2014 1 / 18 Primary Goal General problem statement: We observe X p(x θ), θ Θ and the goal is to determine the θ that produced X. Given a collection of observations
More informationStat 5102 Lecture Slides Deck 3. Charles J. Geyer School of Statistics University of Minnesota
Stat 5102 Lecture Slides Deck 3 Charles J. Geyer School of Statistics University of Minnesota 1 Likelihood Inference We have learned one very general method of estimation: method of moments. the Now we
More informationFinite Singular Multivariate Gaussian Mixture
21/06/2016 Plan 1 Basic definitions Singular Multivariate Normal Distribution 2 3 Plan Singular Multivariate Normal Distribution 1 Basic definitions Singular Multivariate Normal Distribution 2 3 Multivariate
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2008 Prof. Gesine Reinert 1 Data x = x 1, x 2,..., x n, realisations of random variables X 1, X 2,..., X n with distribution (model)
More informationReview and continuation from last week Properties of MLEs
Review and continuation from last week Properties of MLEs As we have mentioned, MLEs have a nice intuitive property, and as we have seen, they have a certain equivariance property. We will see later that
More informationChapter 3: Maximum Likelihood Theory
Chapter 3: Maximum Likelihood Theory Florian Pelgrin HEC September-December, 2010 Florian Pelgrin (HEC) Maximum Likelihood Theory September-December, 2010 1 / 40 1 Introduction Example 2 Maximum likelihood
More informationMathematical Statistics
Mathematical Statistics Chapter Three. Point Estimation 3.4 Uniformly Minimum Variance Unbiased Estimator(UMVUE) Criteria for Best Estimators MSE Criterion Let F = {p(x; θ) : θ Θ} be a parametric distribution
More informationwhere x and ȳ are the sample means of x 1,, x n
y y Animal Studies of Side Effects Simple Linear Regression Basic Ideas In simple linear regression there is an approximately linear relation between two variables say y = pressure in the pancreas x =
More informationLecture 1 October 9, 2013
Probabilistic Graphical Models Fall 2013 Lecture 1 October 9, 2013 Lecturer: Guillaume Obozinski Scribe: Huu Dien Khue Le, Robin Bénesse The web page of the course: http://www.di.ens.fr/~fbach/courses/fall2013/
More informationIntroduction to Estimation Methods for Time Series models Lecture 2
Introduction to Estimation Methods for Time Series models Lecture 2 Fulvio Corsi SNS Pisa Fulvio Corsi Introduction to Estimation () Methods for Time Series models Lecture 2 SNS Pisa 1 / 21 Estimators:
More informationLinear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
More informationSection 9: Generalized method of moments
1 Section 9: Generalized method of moments In this section, we revisit unbiased estimating functions to study a more general framework for estimating parameters. Let X n =(X 1,...,X n ), where the X i
More informationNotes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed
18.466 Notes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed 1. MLEs in exponential families Let f(x,θ) for x X and θ Θ be a likelihood function, that is, for present purposes,
More informationIntroduction to Machine Learning. Lecture 2
Introduction to Machine Learning Lecturer: Eran Halperin Lecture 2 Fall Semester Scribe: Yishay Mansour Some of the material was not presented in class (and is marked with a side line) and is given for
More informationProbability and Statistics qualifying exam, May 2015
Probability and Statistics qualifying exam, May 2015 Name: Instructions: 1. The exam is divided into 3 sections: Linear Models, Mathematical Statistics and Probability. You must pass each section to pass
More informationMultivariate Analysis and Likelihood Inference
Multivariate Analysis and Likelihood Inference Outline 1 Joint Distribution of Random Variables 2 Principal Component Analysis (PCA) 3 Multivariate Normal Distribution 4 Likelihood Inference Joint density
More information1 General problem. 2 Terminalogy. Estimation. Estimate θ. (Pick a plausible distribution from family. ) Or estimate τ = τ(θ).
Estimation February 3, 206 Debdeep Pati General problem Model: {P θ : θ Θ}. Observe X P θ, θ Θ unknown. Estimate θ. (Pick a plausible distribution from family. ) Or estimate τ = τ(θ). Examples: θ = (µ,
More informationSOLUTION FOR HOMEWORK 7, STAT p(x σ) = (1/[2πσ 2 ] 1/2 )e (x µ)2 /2σ 2.
SOLUTION FOR HOMEWORK 7, STAT 6332 1. We have (for a general case) Denote p (x) p(x σ)/ σ. Then p(x σ) (1/[2πσ 2 ] 1/2 )e (x µ)2 /2σ 2. p (x σ) p(x σ) 1 (x µ)2 +. σ σ 3 Then E{ p (x σ) p(x σ) } σ 2 2σ
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationTheory of Statistics.
Theory of Statistics. Homework V February 5, 00. MT 8.7.c When σ is known, ˆµ = X is an unbiased estimator for µ. If you can show that its variance attains the Cramer-Rao lower bound, then no other unbiased
More informationMaximum Likelihood Estimation
Maximum Likelihood Estimation Assume X P θ, θ Θ, with joint pdf (or pmf) f(x θ). Suppose we observe X = x. The Likelihood function is L(θ x) = f(x θ) as a function of θ (with the data x held fixed). The
More informationFinal Exam. 1. (6 points) True/False. Please read the statements carefully, as no partial credit will be given.
1. (6 points) True/False. Please read the statements carefully, as no partial credit will be given. (a) If X and Y are independent, Corr(X, Y ) = 0. (b) (c) (d) (e) A consistent estimator must be asymptotically
More information1 Likelihood. 1.1 Likelihood function. Likelihood & Maximum Likelihood Estimators
Likelihood & Maximum Likelihood Estimators February 26, 2018 Debdeep Pati 1 Likelihood Likelihood is surely one of the most important concepts in statistical theory. We have seen the role it plays in sufficiency,
More informationParametric Techniques Lecture 3
Parametric Techniques Lecture 3 Jason Corso SUNY at Buffalo 22 January 2009 J. Corso (SUNY at Buffalo) Parametric Techniques Lecture 3 22 January 2009 1 / 39 Introduction In Lecture 2, we learned how to
More informationsimple if it completely specifies the density of x
3. Hypothesis Testing Pure significance tests Data x = (x 1,..., x n ) from f(x, θ) Hypothesis H 0 : restricts f(x, θ) Are the data consistent with H 0? H 0 is called the null hypothesis simple if it completely
More informationCovariance function estimation in Gaussian process regression
Covariance function estimation in Gaussian process regression François Bachoc Department of Statistics and Operations Research, University of Vienna WU Research Seminar - May 2015 François Bachoc Gaussian
More informationUnbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.
Unbiased Estimation Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others. To compare ˆθ and θ, two estimators of θ: Say ˆθ is better than θ if it
More informationUniversity of Cambridge Engineering Part IIB Module 4F10: Statistical Pattern Processing Handout 2: Multivariate Gaussians
Engineering Part IIB: Module F Statistical Pattern Processing University of Cambridge Engineering Part IIB Module F: Statistical Pattern Processing Handout : Multivariate Gaussians. Generative Model Decision
More informationChapter 3. Point Estimation. 3.1 Introduction
Chapter 3 Point Estimation Let (Ω, A, P θ ), P θ P = {P θ θ Θ}be probability space, X 1, X 2,..., X n : (Ω, A) (IR k, B k ) random variables (X, B X ) sample space γ : Θ IR k measurable function, i.e.
More informationPOLI 8501 Introduction to Maximum Likelihood Estimation
POLI 8501 Introduction to Maximum Likelihood Estimation Maximum Likelihood Intuition Consider a model that looks like this: Y i N(µ, σ 2 ) So: E(Y ) = µ V ar(y ) = σ 2 Suppose you have some data on Y,
More informationParametric Techniques
Parametric Techniques Jason J. Corso SUNY at Buffalo J. Corso (SUNY at Buffalo) Parametric Techniques 1 / 39 Introduction When covering Bayesian Decision Theory, we assumed the full probabilistic structure
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More informationAn Empirical Characteristic Function Approach to Selecting a Transformation to Normality
Communications for Statistical Applications and Methods 014, Vol. 1, No. 3, 13 4 DOI: http://dx.doi.org/10.5351/csam.014.1.3.13 ISSN 87-7843 An Empirical Characteristic Function Approach to Selecting a
More informationInformation in a Two-Stage Adaptive Optimal Design
Information in a Two-Stage Adaptive Optimal Design Department of Statistics, University of Missouri Designed Experiments: Recent Advances in Methods and Applications DEMA 2011 Isaac Newton Institute for
More informationMath 152. Rumbos Fall Solutions to Assignment #12
Math 52. umbos Fall 2009 Solutions to Assignment #2. Suppose that you observe n iid Bernoulli(p) random variables, denoted by X, X 2,..., X n. Find the LT rejection region for the test of H o : p p o versus
More informationInformation in Data. Sufficiency, Ancillarity, Minimality, and Completeness
Information in Data Sufficiency, Ancillarity, Minimality, and Completeness Important properties of statistics that determine the usefulness of those statistics in statistical inference. These general properties
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationECE 275A Homework 7 Solutions
ECE 275A Homework 7 Solutions Solutions 1. For the same specification as in Homework Problem 6.11 we want to determine an estimator for θ using the Method of Moments (MOM). In general, the MOM estimator
More informationECE 275B Homework # 1 Solutions Version Winter 2015
ECE 275B Homework # 1 Solutions Version Winter 2015 1. (a) Because x i are assumed to be independent realizations of a continuous random variable, it is almost surely (a.s.) 1 the case that x 1 < x 2
More informationCOM336: Neural Computing
COM336: Neural Computing http://www.dcs.shef.ac.uk/ sjr/com336/ Lecture 2: Density Estimation Steve Renals Department of Computer Science University of Sheffield Sheffield S1 4DP UK email: s.renals@dcs.shef.ac.uk
More informationMISCELLANEOUS TOPICS RELATED TO LIKELIHOOD. Copyright c 2012 (Iowa State University) Statistics / 30
MISCELLANEOUS TOPICS RELATED TO LIKELIHOOD Copyright c 2012 (Iowa State University) Statistics 511 1 / 30 INFORMATION CRITERIA Akaike s Information criterion is given by AIC = 2l(ˆθ) + 2k, where l(ˆθ)
More informationMath 494: Mathematical Statistics
Math 494: Mathematical Statistics Instructor: Jimin Ding jmding@wustl.edu Department of Mathematics Washington University in St. Louis Class materials are available on course website (www.math.wustl.edu/
More informationMaximum Likelihood Asymptotic Theory. Eduardo Rossi University of Pavia
Maximum Likelihood Asymtotic Theory Eduardo Rossi University of Pavia Slutsky s Theorem, Cramer s Theorem Slutsky s Theorem Let {X N } be a random sequence converging in robability to a constant a, and
More informationChapter 4 HOMEWORK ASSIGNMENTS. 4.1 Homework #1
Chapter 4 HOMEWORK ASSIGNMENTS These homeworks may be modified as the semester progresses. It is your responsibility to keep up to date with the correctly assigned homeworks. There may be some errors in
More informationQualifying Exam in Probability and Statistics. https://www.soa.org/files/edu/edu-exam-p-sample-quest.pdf
Part : Sample Problems for the Elementary Section of Qualifying Exam in Probability and Statistics https://www.soa.org/files/edu/edu-exam-p-sample-quest.pdf Part 2: Sample Problems for the Advanced Section
More informationMaster s Written Examination
Master s Written Examination Option: Statistics and Probability Spring 05 Full points may be obtained for correct answers to eight questions Each numbered question (which may have several parts) is worth
More informationF & B Approaches to a simple model
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 215 http://www.astro.cornell.edu/~cordes/a6523 Lecture 11 Applications: Model comparison Challenges in large-scale surveys
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationMaster s Written Examination - Solution
Master s Written Examination - Solution Spring 204 Problem Stat 40 Suppose X and X 2 have the joint pdf f X,X 2 (x, x 2 ) = 2e (x +x 2 ), 0 < x < x 2
More informationLimiting Distributions
Limiting Distributions We introduce the mode of convergence for a sequence of random variables, and discuss the convergence in probability and in distribution. The concept of convergence leads us to the
More informationChapter 6: Large Random Samples Sections
Chapter 6: Large Random Samples Sections 6.1: Introduction 6.2: The Law of Large Numbers Skip p. 356-358 Skip p. 366-368 Skip 6.4: The correction for continuity Remember: The Midterm is October 25th in
More informationChapter 3 : Likelihood function and inference
Chapter 3 : Likelihood function and inference 4 Likelihood function and inference The likelihood Information and curvature Sufficiency and ancilarity Maximum likelihood estimation Non-regular models EM
More informationMATH 829: Introduction to Data Mining and Analysis Graphical Models II - Gaussian Graphical Models
1/13 MATH 829: Introduction to Data Mining and Analysis Graphical Models II - Gaussian Graphical Models Dominique Guillot Departments of Mathematical Sciences University of Delaware May 4, 2016 Recall
More informationECE 275B Homework # 1 Solutions Winter 2018
ECE 275B Homework # 1 Solutions Winter 2018 1. (a) Because x i are assumed to be independent realizations of a continuous random variable, it is almost surely (a.s.) 1 the case that x 1 < x 2 < < x n Thus,
More informationGeneralized Linear Models. Kurt Hornik
Generalized Linear Models Kurt Hornik Motivation Assuming normality, the linear model y = Xβ + e has y = β + ε, ε N(0, σ 2 ) such that y N(μ, σ 2 ), E(y ) = μ = β. Various generalizations, including general
More informationEconomics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
More informationREGRESSION WITH SPATIALLY MISALIGNED DATA. Lisa Madsen Oregon State University David Ruppert Cornell University
REGRESSION ITH SPATIALL MISALIGNED DATA Lisa Madsen Oregon State University David Ruppert Cornell University SPATIALL MISALIGNED DATA 10 X X X X X X X X 5 X X X X X 0 X 0 5 10 OUTLINE 1. Introduction 2.
More informationSTAT 135 Lab 2 Confidence Intervals, MLE and the Delta Method
STAT 135 Lab 2 Confidence Intervals, MLE and the Delta Method Rebecca Barter February 2, 2015 Confidence Intervals Confidence intervals What is a confidence interval? A confidence interval is calculated
More informationMaximum Likelihood Estimation. only training data is available to design a classifier
Introduction to Pattern Recognition [ Part 5 ] Mahdi Vasighi Introduction Bayesian Decision Theory shows that we could design an optimal classifier if we knew: P( i ) : priors p(x i ) : class-conditional
More informationStatistics. Lecture 2 August 7, 2000 Frank Porter Caltech. The Fundamentals; Point Estimation. Maximum Likelihood, Least Squares and All That
Statistics Lecture 2 August 7, 2000 Frank Porter Caltech The plan for these lectures: The Fundamentals; Point Estimation Maximum Likelihood, Least Squares and All That What is a Confidence Interval? Interval
More informationLinear Methods for Prediction
Chapter 5 Linear Methods for Prediction 5.1 Introduction We now revisit the classification problem and focus on linear methods. Since our prediction Ĝ(x) will always take values in the discrete set G we
More informationTesting Hypothesis. Maura Mezzetti. Department of Economics and Finance Università Tor Vergata
Maura Department of Economics and Finance Università Tor Vergata Hypothesis Testing Outline It is a mistake to confound strangeness with mystery Sherlock Holmes A Study in Scarlet Outline 1 The Power Function
More information2018 2019 1 9 sei@mistiu-tokyoacjp http://wwwstattu-tokyoacjp/~sei/lec-jhtml 11 552 3 0 1 2 3 4 5 6 7 13 14 33 4 1 4 4 2 1 1 2 2 1 1 12 13 R?boxplot boxplotstats which does the computation?boxplotstats
More information