Lecture 6 Ecient estimators. Rao-Cramer bound.
|
|
- Jody Butler
- 6 years ago
- Views:
Transcription
1 Lecture 6 Eciet estimators. Rao-Cramer boud. 1 MSE ad Suciecy Let X (X 1,..., X) be a radom sample from distributio f θ. Let θ ˆ δ(x) be a estimator of θ. Let T (X) be a suciet statistic for θ. As we have see already, MSE provides oe way to compare the quality of dieret estimators. I particular, estimators with smaller MSE are said to be more eciet. O the other had, oce we kow T (X), we ca discard X. How do these cocepts relate to each other? The theorem below shows that for ay estimator θˆ δ(x), there is aother estimator which depeds o data X oly through T (X) ad is at least as eciet as θˆ: Theorem 1 (Rao-Blackwell). I the settig above, dee φ(t ) E[δ(X) T ]. The θˆ2 φ(t (X)) is a estimator for θ ad MSE(θˆ2) MSE(θˆ). I additio, if θˆ is ubiased, the θˆ2 is ubiased as well. Proof. To show that θˆ2 is a estimator, we have to check that it does ot deped o θ. Ideed, sice T is suciet for θ, the coditioal distributio of X give T is idepedet of θ. So the coditioal distributio of δ(x) give T is idepedet of θ as well. I particular, the coditioal expectatio E[δ(X) T ] does ot deped o θ. Thus, φ(t (X)) depeds oly o the data X ad θˆ2 is a estimator. where i the last lie we used sice E[δ(X) T ] φ(t (X)). MSE(θˆ) E[(θˆ θˆ θˆ θ) 2 ] E[(θˆ θ ˆ 2 ) 2 ] + 2E[( θˆ θˆ 2 )(θˆ2 θ)] + E[(θˆ2 θ) 2 ] E[(θˆ θ ˆ 2 ) 2 ] + 2E[(θˆ θ ˆ 2 )(θˆ2 θ)] + MSE(θˆ2) E[(θˆ ˆ θ 2 ) 2 ] + MSE(θˆ2), E[(θˆ θ ˆ 2 )(θˆ2 θ)] E[(δ(X) φ(t (X)))(φ(T (X)) θ)] E[E[(δ(X) φ(t (X)))(φ(T (X)) θ) T ]] E[(φ(T (X)) θ)e[(δ(x) φ(t (X))) T ]] E[(φ(T (X)) θ) (E[δ(X) T ] φ(t (X)))] 0, Cite as: Aa Mikusheva, course materials for Statistical Methods i Ecoomics, Fall MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].
2 To show the last result, we have by the law of iterated expectatio. E[φ(T (X))] E[E[δ(X) T ]] E[δ(X)] θ Example Let X 1,..., X be a radom sample from Biomial(p, k), i.e. P {X j m} (k!/(m!(k m)!))p m (1 p) k m for ay iteger m 0. Suppose our parameter of iterest is the probability of oe success, i.e. θ P {X j 1} kp(1 p) k 1. Oe possible estimator is θˆ i1 I(X i 1)/. This estimator is ubiased, i.e. E[θˆ] θ. Let us d a suciet statistic. The joit desity of the data is f(x 1,..., x ) (k!/(xi!(k x i )!))p i (1 p) k x i i1 fuctio(x 1,..., x )p x i (1 p) k x i x Thus, T i1 X i is suciet. I fact, it is miimal suciet. Usig the Rao-Blackwell theorem, we ca improve θˆ by cosiderig its coditioal expectatio give T. Let φ E[θˆ T ] deote this estimator. The, for ay oegative iteger t, φ(t) E[ I(X i 1)/ X i t] i1 i1 P {X i 1 X j t}/ i1 j1 P {X 1 1 X j t} j1 P { X 1 1, j1 X j t} P { j1 X j t} P {X 1 1, j2 X j t 1} P { j1 X j t} P {X 1 1} P { j2 X j t 1} P { j1 X j t} kp(1 p) k 1 (k( 1))!/((t 1)!(k( 1) (t 1))!)p t 1 (1 p) k( 1) (t 1) (k)!/(t!(k t)!)p t (1 p) k t k(k( 1))! /((t 1)!(k( 1) (t 1))!) (k)!/(t!(k t)!) k(k( 1))!(k t)!t (k)!(k k + 1 t)! where we used the fact that X 1 is idepedet of (X 2,..., X ), i1 X i Biomial(k, p), ad i2 X i 2
3 Biomial(k( 1), p). So our ew estimator is ˆ k( k( 1))!(k i1 θ 2 φ(x 1,..., X ) X i)!( i1 X i) (k)!(k k + 1 i1 X i)! By the theorem above, it is ubiased ad at least as eciet as θˆ. The procedure we just applied is sometimes iformally referred to as Rao-Blackwellizatio. 2 Fisher iformatio Let f(x θ) with θ Θ be some parametric family. For give θ Θ, let Supp θ {x : f(x θ) > 0}. Supp θ is usually called the support of distributio f(x θ). Assume that Supp θ does ot deped o θ. As before, l(x, θ) log f(x θ) is called loglikelihood fuctio. Assume that l(x, θ) is twice cotiuously dieretiable i θ for all x S. Let X be some radom variable with distributio f(x θ). The Deitio 2. I(θ) E θ [( l(x, θ)/ θ) 2 ] is called Fisher iformatio. Fisher iformatio plays a importat role i maximum likelihood estimatio. The theorem below gives two iformatio equalities: Theorem 3. I the settig above, (1) E θ [ l(x, θ)/ θ] 0 ad (2) I(θ) E θ [ 2 l(x, θ)/ θ 2 ]. Proof. Sice l( x, θ) is twice dieretiable i θ, f(x θ) is twice dieretiable i θ as well. Dieretiatig idetity θ)dx 1 with respect to θ yields + f(x for all θ Θ. The secod dieretiatio yields + f(x θ) dx 0 θ for all θ Θ. I additio, ad The former equality yields + 2 f(x θ) dx 0 θ 2 (1) l(x, θ) log f(x θ) 1 f(x θ) θ θ f(x θ) θ 2 ( ) 2 l(x, θ) 1 f ( x θ) 1 2 f(x θ) θ 2 f 2 +. (x θ) θ f(x θ) θ 2 [ ] [ l(x, θ) 1 f(x θ) ] + 1 f(x θ) + f(x θ) E θ E θ f(x θ)dx dx 0, θ f(x θ) θ f(x θ) θ θ 3
4 which is our rst result. The latter equality yields [ 2 ( ) 2 ] l(x, θ) + 1 f(x θ) E θ θ 2 f(x θ) θ dx i view of equatio (1). So, [ ( ) ] 2 l(x, θ) I(θ) E θ θ + ( ) 2 1 f(x θ) f(x θ)dx f(x θ) θ + ( ) 2 1 f(x θ) dx f(x θ) θ [ ] 2 l(x, θ) E θ, θ 2 which is our secod result. Example Let us calculate Fisher iformatio for a N(µ, σ 2 ) distributio where σ 2 is kow. Thus, our parameter θ µ. The desity of a ormal distributio is f(x µ) exp( (x µ) 2 /(2σ 2 ))/ 2π. The loglikelihood is l(x, µ) log(2π)/2 (x µ) 2 /(2σ 2 ). So l(x, µ)/ µ (x µ)/σ 2 ad 2 l(x, µ)/ µ 2 1/σ 2. So E θ [ 2 l(x, µ)/ µ 2 ] 1/σ 2. At the same time, I(θ) E µ [( l(x, µ)/ µ) 2 ] Eµ[(X µ) /σ ] 1/σ So, as was expected i view of the theorem above, I(θ) E µ [ 2 l(x, µ)/ µ 2 ] i this example. Example Let us calculate Fisher iformatio for a Beroulli(θ) distributio. Note that a Beroulli distributio is discrete. So we use probability mass fuctio (pms) istead of pdf. The pms of Beroulli(θ) is f(x θ) θ x (1 θ) 1 x for x {0, 1}. The log-likelihood is l(x, θ) x log θ + (1 x) log(1 θ). So l(x, θ)/ θ x/θ (1 x)/(1 θ) ad 2 l(x, θ)/ θ 2 x/θ 2 (1 x)/(1 θ) 2. So E θ [( l(x, θ)/ θ) 2 ] E θ [(X/θ (1 X)/(1 θ)) 2 ] E θ [X 2 /θ 2 ] 2E θ [X(1 X)/(θ(1 θ))] + E θ [(1 X) 2 /(1 θ) 2 ] E θ [X/θ 2 ] + E θ [(1 X)/(1 θ) 2 ] 1/(θ(1 θ)) 4
5 sice x x 2, x(1 x) 0, ad (1 x) (1 x) 2 if x {0, 1}. At the same time, So I(θ) E θ [ 2 l(x, θ)/ θ 2 ] as it should be. 2.1 Iformatio for a radom sample E 2 θ[ l(x, θ)/ θ 2 ] E θ [X/θ 2 + (1 X)/(1 θ) 2 ] θ/θ 2 + (1 θ)/(1 θ) 2 1/θ + 1/(1 θ) 1/(θ(1 θ)) Let us ow cosider Fisher iformatio for a radom sample. Let X ( X 1,..., X ) be a radom sample from distributio f(x θ). The the joit pdf is f (x) i1 f(x i θ) where x (x 1,..., x ). The joit log-likelihood is l (x, θ) i1 l(x i, θ). So Fisher iformatio for the sample X is I(θ) E θ [( ( ) ) 2 ] l X, θ / θ E θ [ ( l(x i, θ)/ θ)( l(x j, θ)/ θ)] 1 i,j E θ [ ( l(x i, θ)/ θ) 2 ] + 2E θ [ ( l(x i, θ)/ θ)( l(x j, θ)/ θ)] E I( i1 1 i<j 2 θ[ ( l(x i, θ)/ θ) ] i1 θ) where we used the fact that for ay i < j, E θ [( l(x i, θ)/ θ)( l(x j, θ)/ θ)] 0 by idepedece ad rst iformatio equality. Here I(θ) deotes Fisher iformatio for distributio f(x θ). 3 Rao-Cramer boud A importat questio i the theory of statistical estimatio is whether there is a otrivial boud such that o estimator ca be more eciet tha this boud. The theorem below is a result of this sort: Theorem 4 (Rao-Cramer boud). Let X (X 1,..., X ) be a radom sample from distributio f(x θ) with iformatio I (θ). Let W (X) be a estimator of θ such that (1) de θ [W (X)]/dθ W (x)df(x, θ)/dθdx where x (x 1,...x ) ad (2) V (W ) <. The V (W ) (de θ [W (X)]/dθ) 2 /I (θ). I particular, if W is ubiased for θ, the V (W ) 1/I θ (θ). 5
6 Proof. The rst iformatio equality gives E θ [ l(x, θ)/ θ] 0. So, cov(w (X), l(x, θ)/ θ) E[W (X) l(x, θ)/ θ] W (x) l(x, θ)/ θf(x θ)dx W (x) f(x θ)/ θ (1/f(x θ))f(x θ)dx W (x) f(x θ)/ θdx de θ [W (X)]/dθ. By the Cauchy-Schwarz iequality, (cov(w (X), l(x, θ)/ θ) 2 V (W (X))V ( l(x, θ)/ θ) V (W (X))I (θ). Thus, V (W (X)) (de θ [W (X)]/dθ) 2 /I (θ). If W is ubiased for θ, the E θ [W (X)] θ, de θ [W (X)]/dθ 1, ad V (W (X)) 1/I (θ). Example Let us calculate the Rao-Cramer boud for radom sample X 1,..., X from Beroulli(θ) distributio. We have already see that I(θ) 1/(θ(1 θ)) i this case. So Fisher iformatio for the sample is I (θ) /(θ(1 θ)). Thus, ay ubiased estimator of θ, uder some regularity coditios, has variace o smaller tha θ(1 θ)/. O the other had, let θˆ X i1 X i/ be a estimator of θ. The E θ [θˆ] θ, i.e. θˆ is ubiased, ad V (θˆ) θ(1 θ)/ which coicides with the Rao-Cramer boud. Thus, X is the uiformly miimum variace ubiased (UMVU) estimator of θ. Word uiformly i this situatio meas that X has the smallest variace amog ubiased estimators for all θ Θ. Example Let us ow cosider a couterexample to the Rao-Cramer theorem. Let X 1,..., X be a radom sample from U[0, θ]. The f(x i θ) 1/θ if x i [0, θ] ad 0 otherwise. So l(x i, θ) log θ if x i [0, θ]. The l/ θ 1/θ ad 2 l/ θ 2 1/θ 2. So I(θ) 1/θ 2 while E θ [ 2 l(x i, θ)/ θ 2 ] 1/θ 2 I(θ). Thus, the secod iformatio equality does ot hold i this example. The reaso is that support of the distributio depeds o θ i this example. Moreover, cosider a estimator θˆ (( + 1)/)X () of θ. The E θ [X () ] θ ad V (θˆ) (( + 1) 2 / 2 )V (X () ) θ 2 /(( + 2)) as we saw whe we cosidered order statistics. So θˆ is ubiased, but its variace is smaller tha 1/I (θ) θ 2 / 2. Thus, the Rao-Cramer theorem does ot work i this example as well. Agai, the reaso is that Rao-Cramer theorem assumes that support is idepedet of parameter. 6
7 MIT OpeCourseWare Statistical Method i Ecoomics Fall 2013 For iformatio about citig these materials or our Terms of Use, visit:
Unbiased Estimation. February 7-12, 2008
Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom
More informationStatistical Theory MT 2008 Problems 1: Solution sketches
Statistical Theory MT 008 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. a) Let 0 < θ < ad put fx, θ) = θ)θ x ; x = 0,,,... b) c) where α
More informationStatistical Theory MT 2009 Problems 1: Solution sketches
Statistical Theory MT 009 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. (a) Let 0 < θ < ad put f(x, θ) = ( θ)θ x ; x = 0,,,... (b) (c) where
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationLecture 12: September 27
36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.
More informationDirection: This test is worth 250 points. You are required to complete this test within 50 minutes.
Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationFirst Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >
More information1.010 Uncertainty in Engineering Fall 2008
MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval
More informationLecture 23: Minimal sufficiency
Lecture 23: Miimal sufficiecy Maximal reductio without loss of iformatio There are may sufficiet statistics for a give problem. I fact, X (the whole data set) is sufficiet. If T is a sufficiet statistic
More information6. Sufficient, Complete, and Ancillary Statistics
Sufficiet, Complete ad Acillary Statistics http://www.math.uah.edu/stat/poit/sufficiet.xhtml 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > 1 2 3 4 5 6 6. Sufficiet, Complete, ad Acillary
More informationLecture 11 and 12: Basic estimation theory
Lecture ad 2: Basic estimatio theory Sprig 202 - EE 94 Networked estimatio ad cotrol Prof. Kha March 2 202 I. MAXIMUM-LIKELIHOOD ESTIMATORS The maximum likelihood priciple is deceptively simple. Louis
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationx = Pr ( X (n) βx ) =
Exercise 93 / page 45 The desity of a variable X i i 1 is fx α α a For α kow let say equal to α α > fx α α x α Pr X i x < x < Usig a Pivotal Quatity: x α 1 < x < α > x α 1 ad We solve i a similar way as
More informationMATH 472 / SPRING 2013 ASSIGNMENT 2: DUE FEBRUARY 4 FINALIZED
MATH 47 / SPRING 013 ASSIGNMENT : DUE FEBRUARY 4 FINALIZED Please iclude a cover sheet that provides a complete setece aswer to each the followig three questios: (a) I your opiio, what were the mai ideas
More informationLecture Note 8 Point Estimators and Point Estimation Methods. MIT Spring 2006 Herman Bennett
Lecture Note 8 Poit Estimators ad Poit Estimatio Methods MIT 14.30 Sprig 2006 Herma Beett Give a parameter with ukow value, the goal of poit estimatio is to use a sample to compute a umber that represets
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationExercise 4.3 Use the Continuity Theorem to prove the Cramér-Wold Theorem, Theorem. (1) φ a X(1).
Assigmet 7 Exercise 4.3 Use the Cotiuity Theorem to prove the Cramér-Wold Theorem, Theorem 4.12. Hit: a X d a X implies that φ a X (1) φ a X(1). Sketch of solutio: As we poited out i class, the oly tricky
More informationLecture 16: UMVUE: conditioning on sufficient and complete statistics
Lecture 16: UMVUE: coditioig o sufficiet ad complete statistics The 2d method of derivig a UMVUE whe a sufficiet ad complete statistic is available Fid a ubiased estimator of ϑ, say U(X. Coditioig o a
More informationExponential Families and Bayesian Inference
Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where
More informationAPPM 5720 Solutions to Problem Set Five
APPM 5720 Solutios to Problem Set Five 1. The pdf is f(x; α, β) 1 Γ(α) βα x α 1 e βx I (0, ) (x). The joit pdf is f( x; α, β) 1 [Γ(α)] β α ( i1 x i ) α 1 e β i1 x i i1 I (0, ) (x i ) 1 [Γ(α)] βα } {{ }
More informationSupport vector machine revisited
6.867 Machie learig, lecture 8 (Jaakkola) 1 Lecture topics: Support vector machie ad kerels Kerel optimizatio, selectio Support vector machie revisited Our task here is to first tur the support vector
More informationLecture 7: Properties of Random Samples
Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 3 9/11/2013. Large deviations Theory. Cramér s Theorem
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 3 9//203 Large deviatios Theory. Cramér s Theorem Cotet.. Cramér s Theorem. 2. Rate fuctio ad properties. 3. Chage of measure techique.
More informationThis exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.
Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the
More informationLecture 01: the Central Limit Theorem. 1 Central Limit Theorem for i.i.d. random variables
CSCI-B609: A Theorist s Toolkit, Fall 06 Aug 3 Lecture 0: the Cetral Limit Theorem Lecturer: Yua Zhou Scribe: Yua Xie & Yua Zhou Cetral Limit Theorem for iid radom variables Let us say that we wat to aalyze
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More informationLecture 18: Sampling distributions
Lecture 18: Samplig distributios I may applicatios, the populatio is oe or several ormal distributios (or approximately). We ow study properties of some importat statistics based o a radom sample from
More informationEconomics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More information6.041/6.431 Spring 2009 Final Exam Thursday, May 21, 1:30-4:30 PM.
6.041/6.431 Sprig 2009 Fial Exam Thursday, May 21, 1:30-4:30 PM. Name: Recitatio Istructor: Questio Part Score Out of 0 2 1 all 18 2 all 24 3 a 4 b 4 c 4 4 a 6 b 6 c 6 5 a 6 b 6 6 a 4 b 4 c 4 d 5 e 5 7
More informationSince X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain
Assigmet 9 Exercise 5.5 Let X biomial, p, where p 0, 1 is ukow. Obtai cofidece itervals for p i two differet ways: a Sice X / p d N0, p1 p], the variace of the limitig distributio depeds oly o p. Use the
More information5. Likelihood Ratio Tests
1 of 5 7/29/2009 3:16 PM Virtual Laboratories > 9. Hy pothesis Testig > 1 2 3 4 5 6 7 5. Likelihood Ratio Tests Prelimiaries As usual, our startig poit is a radom experimet with a uderlyig sample space,
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationSolutions: Homework 3
Solutios: Homework 3 Suppose that the radom variables Y,...,Y satisfy Y i = x i + " i : i =,..., IID where x,...,x R are fixed values ad ",...," Normal(0, )with R + kow. Fid ˆ = MLE( ). IND Solutio: Observe
More informationPoint Estimation: properties of estimators 1 FINITE-SAMPLE PROPERTIES. finite-sample properties (CB 7.3) large-sample properties (CB 10.
Poit Estimatio: properties of estimators fiite-sample properties CB 7.3) large-sample properties CB 10.1) 1 FINITE-SAMPLE PROPERTIES How a estimator performs for fiite umber of observatios. Estimator:
More informationTAMS24: Notations and Formulas
TAMS4: Notatios ad Formulas Basic otatios ad defiitios X: radom variable stokastiska variabel Mea Vätevärde: µ = X = by Xiagfeg Yag kpx k, if X is discrete, xf Xxdx, if X is cotiuous Variace Varias: =
More informationStatistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.
Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece
More informationAMS570 Lecture Notes #2
AMS570 Lecture Notes # Review of Probability (cotiued) Probability distributios. () Biomial distributio Biomial Experimet: ) It cosists of trials ) Each trial results i of possible outcomes, S or F 3)
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationAlgorithms for Clustering
CR2: Statistical Learig & Applicatios Algorithms for Clusterig Lecturer: J. Salmo Scribe: A. Alcolei Settig: give a data set X R p where is the umber of observatio ad p is the umber of features, we wat
More informationElement sampling: Part 2
Chapter 4 Elemet samplig: Part 2 4.1 Itroductio We ow cosider uequal probability samplig desigs which is very popular i practice. I the uequal probability samplig, we ca improve the efficiecy of the resultig
More informationAAEC/ECON 5126 FINAL EXAM: SOLUTIONS
AAEC/ECON 5126 FINAL EXAM: SOLUTIONS SPRING 2015 / INSTRUCTOR: KLAUS MOELTNER This exam is ope-book, ope-otes, but please work strictly o your ow. Please make sure your ame is o every sheet you re hadig
More informationEcon 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara
Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationDepartment of Mathematics
Departmet of Mathematics Ma 3/103 KC Border Itroductio to Probability ad Statistics Witer 2017 Lecture 19: Estimatio II Relevat textbook passages: Larse Marx [1]: Sectios 5.2 5.7 19.1 The method of momets
More informationSOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker
SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker CHAPTER 9. POINT ESTIMATION 9. Covergece i Probability. The bases of poit estimatio have already bee laid out i previous chapters. I chapter 5
More informationAsymptotics. Hypothesis Testing UMP. Asymptotic Tests and p-values
of the secod half Biostatistics 6 - Statistical Iferece Lecture 6 Fial Exam & Practice Problems for the Fial Hyu Mi Kag Apil 3rd, 3 Hyu Mi Kag Biostatistics 6 - Lecture 6 Apil 3rd, 3 / 3 Rao-Blackwell
More informationLecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
More informationJanuary 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS
Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we
More informationMaximum Likelihood Estimation and Complexity Regularization
ECE90 Sprig 004 Statistical Regularizatio ad Learig Theory Lecture: 4 Maximum Likelihood Estimatio ad Complexity Regularizatio Lecturer: Rob Nowak Scribe: Pam Limpiti Review : Maximum Likelihood Estimatio
More informationTime series models 2007
Norwegia Uiversity of Sciece ad Techology Departmet of Mathematical Scieces Solutios to problem sheet 1, 2007 Exercise 1.1 a Let Sc = E[Y c 2 ]. The This gives Sc = EY 2 2cEY + c 2 ds dc = 2EY + 2c = 0
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationThe variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.
SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample
More informationLecture 3. Properties of Summary Statistics: Sampling Distribution
Lecture 3 Properties of Summary Statistics: Samplig Distributio Mai Theme How ca we use math to justify that our umerical summaries from the sample are good summaries of the populatio? Lecture Summary
More informationGoodness-of-Fit Tests and Categorical Data Analysis (Devore Chapter Fourteen)
Goodess-of-Fit Tests ad Categorical Data Aalysis (Devore Chapter Fourtee) MATH-252-01: Probability ad Statistics II Sprig 2019 Cotets 1 Chi-Squared Tests with Kow Probabilities 1 1.1 Chi-Squared Testig................
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5
CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio
More informationLECTURE 8: ASYMPTOTICS I
LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece
More informationIIT JAM Mathematical Statistics (MS) 2006 SECTION A
IIT JAM Mathematical Statistics (MS) 6 SECTION A. If a > for ad lim a / L >, the which of the followig series is ot coverget? (a) (b) (c) (d) (d) = = a = a = a a + / a lim a a / + = lim a / a / + = lim
More information6.867 Machine learning
6.867 Machie learig Mid-term exam October, ( poits) Your ame ad MIT ID: Problem We are iterested here i a particular -dimesioal liear regressio problem. The dataset correspodig to this problem has examples
More information2. The volume of the solid of revolution generated by revolving the area bounded by the
IIT JAM Mathematical Statistics (MS) Solved Paper. A eigevector of the matrix M= ( ) is (a) ( ) (b) ( ) (c) ( ) (d) ( ) Solutio: (a) Eigevalue of M = ( ) is. x So, let x = ( y) be the eigevector. z (M
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationResampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.
Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator
More informationLecture 2: Monte Carlo Simulation
STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationECE 901 Lecture 13: Maximum Likelihood Estimation
ECE 90 Lecture 3: Maximum Likelihood Estimatio R. Nowak 5/7/009 The focus of this lecture is to cosider aother approach to learig based o maximum likelihood estimatio. Ulike earlier approaches cosidered
More informationSOLUTION FOR HOMEWORK 7, STAT np(1 p) (α + β + n) + ( np + α
SOLUTION FOR HOMEWORK 7, STAT 6331 1 Exerc733 Here we just recall that MSE(ˆp B ) = p(1 p) (α + β + ) + ( p + α 2 α + β + p) 2 The you plug i α = β = (/4) 1/2 After simplificatios MSE(ˆp B ) = 4( 1/2 +
More informationProblem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
More informationChapter 12 EM algorithms The Expectation-Maximization (EM) algorithm is a maximum likelihood method for models that have hidden variables eg. Gaussian
Chapter 2 EM algorithms The Expectatio-Maximizatio (EM) algorithm is a maximum likelihood method for models that have hidde variables eg. Gaussia Mixture Models (GMMs), Liear Dyamic Systems (LDSs) ad Hidde
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationNotes 5 : More on the a.s. convergence of sums
Notes 5 : More o the a.s. covergece of sums Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: Dur0, Sectios.5; Wil9, Sectio 4.7, Shi96, Sectio IV.4, Dur0, Sectio.. Radom series. Three-series
More informationStat 421-SP2012 Interval Estimation Section
Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible
More informationG. R. Pasha Department of Statistics Bahauddin Zakariya University Multan, Pakistan
Deviatio of the Variaces of Classical Estimators ad Negative Iteger Momet Estimator from Miimum Variace Boud with Referece to Maxwell Distributio G. R. Pasha Departmet of Statistics Bahauddi Zakariya Uiversity
More informationHOMEWORK I: PREREQUISITES FROM MATH 727
HOMEWORK I: PREREQUISITES FROM MATH 727 Questio. Let X, X 2,... be idepedet expoetial radom variables with mea µ. (a) Show that for Z +, we have EX µ!. (b) Show that almost surely, X + + X (c) Fid the
More informationEXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY
EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA, 016 MODULE : Statistical Iferece Time allowed: Three hours Cadidates should aswer FIVE questios. All questios carry equal marks. The umber
More informationEstimation of the Mean and the ACVF
Chapter 5 Estimatio of the Mea ad the ACVF A statioary process {X t } is characterized by its mea ad its autocovariace fuctio γ ), ad so by the autocorrelatio fuctio ρ ) I this chapter we preset the estimators
More informationRandom Signals and Noise Winter Semester 2017 Problem Set 12 Wiener Filter Continuation
Radom Sigals ad Noise Witer Semester 7 Problem Set Wieer Filter Cotiuatio Problem (Sprig, Exam A) Give is the sigal W t, which is a Gaussia white oise with expectatio zero ad power spectral desity fuctio
More informationLecture Stat Maximum Likelihood Estimation
Lecture Stat 461-561 Maximum Likelihood Estimatio A.D. Jauary 2008 A.D. () Jauary 2008 1 / 63 Maximum Likelihood Estimatio Ivariace Cosistecy E ciecy Nuisace Parameters A.D. () Jauary 2008 2 / 63 Parametric
More information1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable
More informationo <Xln <X2n <... <X n < o (1.1)
Metrika, Volume 28, 1981, page 257-262. 9 Viea. Estimatio Problems for Rectagular Distributios (Or the Taxi Problem Revisited) By J.S. Rao, Sata Barbara I ) Abstract: The problem of estimatig the ukow
More informationHomework for 2/3. 1. Determine the values of the following quantities: a. t 0.1,15 b. t 0.05,15 c. t 0.1,25 d. t 0.05,40 e. t 0.
Name: ID: Homework for /3. Determie the values of the followig quatities: a. t 0.5 b. t 0.055 c. t 0.5 d. t 0.0540 e. t 0.00540 f. χ 0.0 g. χ 0.0 h. χ 0.00 i. χ 0.0050 j. χ 0.990 a. t 0.5.34 b. t 0.055.753
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationBinomial Distribution
0.0 0.5 1.0 1.5 2.0 2.5 3.0 0 1 2 3 4 5 6 7 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Overview Example: coi tossed three times Defiitio Formula Recall that a r.v. is discrete if there are either a fiite umber of possible
More informationECE 901 Lecture 14: Maximum Likelihood Estimation and Complexity Regularization
ECE 90 Lecture 4: Maximum Likelihood Estimatio ad Complexity Regularizatio R Nowak 5/7/009 Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio Y i iid p θ, i,, where
More informationEE 4TM4: Digital Communications II Probability Theory
1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair
More informationQuestions and Answers on Maximum Likelihood
Questios ad Aswers o Maximum Likelihood L. Magee Fall, 2008 1. Give: a observatio-specific log likelihood fuctio l i (θ) = l f(y i x i, θ) the log likelihood fuctio l(θ y, X) = l i(θ) a data set (x i,
More informationProbability and Statistics
ICME Refresher Course: robability ad Statistics Staford Uiversity robability ad Statistics Luyag Che September 20, 2016 1 Basic robability Theory 11 robability Spaces A probability space is a triple (Ω,
More informationStat410 Probability and Statistics II (F16)
Some Basic Cocepts of Statistical Iferece (Sec 5.) Suppose we have a rv X that has a pdf/pmf deoted by f(x; θ) or p(x; θ), where θ is called the parameter. I previous lectures, we focus o probability problems
More informationSimulation. Two Rule For Inverting A Distribution Function
Simulatio Two Rule For Ivertig A Distributio Fuctio Rule 1. If F(x) = u is costat o a iterval [x 1, x 2 ), the the uiform value u is mapped oto x 2 through the iversio process. Rule 2. If there is a jump
More informationMachine Learning Brett Bernstein
Machie Learig Brett Berstei Week Lecture: Cocept Check Exercises Starred problems are optioal. Statistical Learig Theory. Suppose A = Y = R ad X is some other set. Furthermore, assume P X Y is a discrete
More informationMachine Learning Brett Bernstein
Machie Learig Brett Berstei Week 2 Lecture: Cocept Check Exercises Starred problems are optioal. Excess Risk Decompositio 1. Let X = Y = {1, 2,..., 10}, A = {1,..., 10, 11} ad suppose the data distributio
More informationMathematical Statistics - MS
Paper Specific Istructios. The examiatio is of hours duratio. There are a total of 60 questios carryig 00 marks. The etire paper is divided ito three sectios, A, B ad C. All sectios are compulsory. Questios
More informationKernel density estimator
Jauary, 07 NONPARAMETRIC ERNEL DENSITY ESTIMATION I this lecture, we discuss kerel estimatio of probability desity fuctios PDF Noparametric desity estimatio is oe of the cetral problems i statistics I
More informationCSE 527, Additional notes on MLE & EM
CSE 57 Lecture Notes: MLE & EM CSE 57, Additioal otes o MLE & EM Based o earlier otes by C. Grat & M. Narasimha Itroductio Last lecture we bega a examiatio of model based clusterig. This lecture will be
More information