# Kernel density estimator

Save this PDF as:

Size: px
Start display at page:

## Transcription

2 where h is a small umber ote that we cosider cotiuously distributed radom variables, so that P X i = x h = 0 We write h istead of just h because, typically, it will be a fuctio of the sample size such that lim h = 0 Now, defie the followig kerel fuctio: The, the kerel PDF estimator is give by u = { u } ˆf x = h 3 i= Thus, with the kerel fuctio defied accordig to, the kerel desity estimator is a average umber of observatios i the small eighborhood of x as defied by the smoothig parameter or badwidth also kerel widow The kerel fuctio i is called uiform, because it correspods to the uiform distributio we have that u du = It has a disadvatage of givig equal weights to all observatios iside the h -widow with the ceter at x, regardless of how close they are to the ceter Also, if oe cosiders ˆf x as a fuctio of x, it is rough havig jumps at the poits X i ±h, ad has a zero derivative everywhere else Those problems ca be resolved if oe cosiders alterative kerel fuctios, for example, the quadratic kerel: h u = 5 u { u } 6 The class of estimators 3 with a kerel satisfyig u du = is referred to as Roseblatt-Parze erel Estimator Small sample properties of the kerel desity estimator We will make the followig assumptio cocerig : Assumptio a u du = b u = u c is compactly supported o [, ] ad bouded d u u du 0 The kerel desity estimator is biased: Lemma Uder Assumptios a ad a, E ˆf x f x = u f x + uh f x du Proof E ˆf x = h E i= = h E = h h h u x h f u du Next, usig chage of variable y = u x /h, u = x + yh, ad du = h dy, we obtai E ˆf x = u f x + uh du, ad the result follows sice f x u du = f x by Assumptio a

3 Lemma Uder Assumptios a ad a, the variace of ˆf x is give by V ar ˆf x = h u f x + uh du u f x + uh du Proof Sice the data are iid, V ar ˆf x = V ar h h = E E h h h h By the same chage of variable argumet as i the proof of Lemma, we obtai E u x = f u du h h = h u f x + uh du From Lemma, oe ca expect that the bias icreases with h ; a bigger badwidth implies that more observatios away from x have o-zero weights which cotributes to the bias O the other had, the variace decreases with h, as the estimator averages over more observatios The theorem below establishes more formally the bias-variace trade-off for the kerel estimator Let f s deote the s-order derivative of f: f s x = ds f x dx s Theorem Suppose that h 0 ad h as The, uder Assumptios ad, a E ˆf x f x = c x h + o h, where c x = f x u u du/ b V ar ˆf x = c x / h + O /, where c x = f x u du Proof Sice the first two derivative of f exist by Assumptio b, cosider the followig expasio for f x + uh : f x + uh = f x + f x uh + f x u u h, where x u lies betwee x ad x + uh From Lemma we have E ˆf x f x = u f x uh + f x u u h du = h u f x u u du = c h + h u f x u f x u du 4 The secod equality follows because by Assumptio b, u udu = 0 We will show ext that u f x u f x u du = o, 5 3

5 Whe the optimal i the MSE sese badwidth is selected, either bias or variace compoets of the MSE domiate each other asymptotically as V ar = Bias = O 4/5 Whe the Itegrated MSE criterio is employed, MSE ˆf x dx, the optimal badwidth becomes u du /5 h = f x /5 dx u u du /5 /5 Let ˆσ deote the sample variace of the data The followig rules of thumb ofte used i practice: h = 364ˆσ u du /5 u u du /5 /5, which is optimal for f x N µ, σ, ad h = 06ˆσ /5, which is optimal for f x N µ, σ ad whe is the stadard ormal desity Cosistecy of the kerel desity estimator Cosistecy of ˆf x follows immediately from Theorem by Chebychev s iequality Corollary Suppose that h 0 ad h as The, uder Assumptios ad, ˆf x p f x Proof By Chebychev s iequality, P ˆf x f x > ε E ˆf x f x ε = c x ε + c x h 4 h ε 0, + o + h 4 h where the secod lie is by 7 A stroger result ca be give, see Newey 994 Suppose that f admits at least m cotiuous derivatives o some iterval [x, x ]; has at least m cotiuous derivatives, is compactly supported ad of order m: u j u du = 0 for all j =,, m ; u m u du 0, ad u du = The sup x [x,x ] ˆf / h x f x = O p + h m log The derivatives of f ca be estimated by the derivative of ˆf, however, with a slower rates of covergece Newey 994 shows that sup x [x,x ] ˆf k h x f k k / x = O p + h m log 5

6 Asymptotic ormality of the kerel desity estimator Write ˆf x = h i= h = v i, where v i = h i= h Note that h ad cosequetly v i deped o The collectio {{v i : i =,, } : N} is called a triagular array I our case, uder Assumptio, v i s are iid The followig CLT is available for idepedet triagular arrays Lehma ad Romao, 005, Corollary, page 47 Lemma 3 Lyapouov CLT Suppose that for each, w,, w are idepedet Assume that Ew i = 0 ad σi = Ew i <, ad defie s = i= σ i Suppose further that for some δ > 0 the followig coditio holds: lim E w i +δ = 0 8 The, i= s +δ w i /s d N 0, i= The coditio 8 is called Lyapouov s coditio Whe the data are ot just idepedet but iid, the Lyapouov s coditio ca be simplified as follows Davidso, 994, Theorem 3 o page 373 Lemma 4 The Lyapouov s coditio is satisfied whe w,, w are iid, σ = Ewi > 0 uiformly i, ad lim E w i +δ / δ/ = 0 for some δ > 0 Proof Sice the data are iid, We have i= s +δ s = σ, ad +δ s +δ = / σ = +δ/ σ +δ E w i +δ = σ δ δ/ E w i +δ i= = σ δ δ/ E w i +δ Therefore, the Lyapouov s coditio is satisfied if δ/ E w i +δ 0, sice σ is uiformly bouded away from zero by the assumptio Assumig that lim σ exists, i the iid case, the result of Lyapouov CLT ca be stated as follows Corollary Suppose that for each, w,, w are iid, Ew i = 0, lim Ewi lim E w i +δ / δ/ = 0 for some δ > 0 The, / w i d N i= 0, lim Ew i > 0 ad fiite, ad 6

7 Next, we prove asymptotic ormality of the kerel desity estimator Theorem Suppose that h ad h / h 0 Assume further that f x > 0 The, uder Assumptios ad, h / ˆf x f x d N 0, f x u du 9 Furthermore, for x x, h / ˆf x f x ad h / ˆf x f x are asymptotically idepedet Proof By Theorem a, Defie The, h / ˆf x f x = h E w i = h / / h / h i= + h / h E h h / ˆf x f x h = = h f x = O h / / h / E h h / f x E h w i + O p h / h 0 i= w i + o p, where the equality i the secod lie is by the assumptio that h / h 0 It is ow left to verify the coditios of Corollary By the defiitio of w i, Ew i = 0 Next, Ewi = E E h h h h As i the proof of Lemma ad by the domiated covergece theorem, E = h u f x + uh du h i= = O h, so that the secod summad i is O h ad asymptotically egligible For the first term i, we ca use the chage of variable argumet agai: E = u x du h h h h = u f x + uh du, f x u du, 3 7

8 where the last result is by the domiated covergece theorem The results i -3 together imply that lim Ew i = f x u du Lastly, we show that E w i +δ / δ/ 0 We will use the c r iequality Davidso, 994, Theorem 98 o page 40 i order to deal with E w i +δ : for r > 0, m r m E X i c r E X i r, i= where c r = whe r, ad c r = m r whe r Now, by the c r iequality, E w i +δ +δ E h +δ/ +δ + h h +δ/ E +δ h By, Further, h +δ/ h +δ/ E +δ h i= E +δ h = = h +δ/ h δ/ = O h δ/ = O h +δ/ u x h +δ f u du u +δ f x + uh du where the equality i the last lie is agai by the domiated covergece theorem Hece, E w i +δ δ/ = O δ/ h This completes the proof of 9 I order to show asymptotic idepedece of ˆf x ad ˆf x, cosider their asymptotic covariace: E = u x u x f u du h h h h h h = u u + x x f x + uh du h Sice the kerel fuctio is compactly supported ad lim x x /h =, lim u + x x = 0, h ad by the domiated covergece theorem, u lim u + x x h Asymptotic idepedece the follows by the Cramer-Wold device, f x + uh du = 0 8

10 so that ˆf x,, x d = ˆf x = h d i= dx i x/h Note that d udu = u du d u d du d = u du =, where the secod equality follows by Assumptio a Similar results to those show for the uivariate estimator ca be established i the multivariate case Assumptio 3 a Suppose {X i : i =,, } is a collectio of iid radom vectors draw from the distributio with a joit PDF f b I the eighborhood N x of x, f is bouded ad twice cotiuously differetiable with bouded partial derivatives Theorem 3 Suppose that h 0 ad h as The, uder Assumptios ad 3, fx x j a E ˆf x f x = fx + h u d u du j= b V ar ˆf x = fx u h d du + O /h d + / + oh Proof For part a, E ˆf x = h d u x d fudu = h d vfx + h vdv, where we used the chage of variable v = u x/h, u = x + h v, du j du = du du d, dv = dv dv d Next, = h dv j for j =,, d, ad fx + h v = fx + h v fx x + h v fx v x x v, where x v deotes the mea-value satisfyig x v x h v, ie it lies betwee x ad x + h v Sice the kerel fuctio is symmetric aroud zero, v d vdv = 0 By Assumptio 3b ad the same argumets as i the proof of Theorem a, fx v x x fx x x Hece, E ˆf x = fx + h v fx x x v dvdv + oh = fx + d d fx h v i v j d vdv + oh x i x j = fx + h = fx + h d j= i= j= fx x j v v dv d j= v j v j dv j + h fx x j d i= j i + oh, = o fx x i x j v i v i dv i v j v j dv j + oh 0

11 where the equality i the last lie holds due to the symmetry of the kerel fuctio u aroud zero For part b, V ar ˆf x = V ar d h d h [ = h d Ed h = [ h d Ed h = [ h d Ed h = h d Ed h = u x h d d h = h d dufx + h udu + O = h d du fx + h u fx x + h u fx v x x = d h h d fx u du + O h d +, ] h d E d h fx + Oh ] holds by the result i a ] f x + Oh + O fudu + O u du + O where the last lie follows sice ud udu = 0 due to the symmetry of the kerel fuctio aroud zero, ad because d udu = d u du The bias ad variace calculatios imply that ˆf x = fx + O p + h h d, ad therefore the rate of covergece slows dow with the umber of variables d I the oparametric literature, this is referred to as the curse of dimesioality Oe ca derive the MSE-poit-optimal badwidth cosiderig the leadig terms i the bias ad variace expressios implied by Theorem 3: MSE x = 4 h4 u u du Miimizig the MSE with respect to h, we obtai: h 3 u u du d j= Therefore, the MSE optimal badwidth is give by d j= fx x j fx d u du h = d u d u du j= fx x j fx x j + fx d u du h d = d fx d u du h d+ /d+4 /d+4

12 Oe ca see that the rate of the optimal badwidth, /d+4, icreases with the umber of variables d, ie oe should use larger values for the badwidth whe there are more variables Oe ca exted the uivariate CLT to the multivariate case as follows: h d / ˆf x fx d h u d fx u du x d N 0, fx u du j= j To elimiate the asymptotic bias, oe has to choose a uder-smoothig badwidth so that h d / h 0 Oe ca also icorporate differet badwidth values for differet variables: ˆf x,, x d = h h h d i= j= The bias ad variace results i this case take the followig form: d Xij x j E ˆf x = fx + d u fx u du h j + o h + + h d x j= j V ar ˆf x = fx d u du h + O + + h d + h h d h h d h j Aalogously, the CLT statemet ca be modified as h h d / ˆf x fx d u u du j= fx x j h j d N 0, fx u du d

13 Bibliography Ait-Sahalia, Y 996: Testig Cotiuous-Time Models of the Spot Iterest Rate, The Review of Fiacial Studies, 9, Davidso, J 994: Stochastic Limit Theory, New York: Oxford Uiversity Press Guerre, E, I Perrige, ad Q Vuog 000: Optimal Noparametric Estimatio of First-Price Auctios, Ecoometrica, 68, Härdle, W ad O Lito 994: Applied Noparametric Methods, i Hadbook of Ecoometrics, ed by R F Egle ad D L McFadde, Amsterdam: Elsevier, vol 4, chap 38, Lehma, E L ad J P Romao 005: Testig Statistical Hypotheses, New York: Spriger, third ed Li, Q ad J S Racie 007: Noparametric Ecoometrics: Theory ad Practice, Priceto, New Jersey: Priceto Uiversity Press Newey, W 994: erel Estimatio of Partial Meas ad a Geeral Variace Estimator, Ecoometric Theory, 0, Paga, A ad A Ullah 999: Noparametric Ecoometrics, New York: Cambridge Uiversity Press va der Vaart, A W 998: Asymptotic Statistics, Cambridge: Cambridge Uiversity Press 3

### LECTURE 2 LEAST SQUARES CROSS-VALIDATION FOR KERNEL DENSITY ESTIMATION

Jauary 3 07 LECTURE LEAST SQUARES CROSS-VALIDATION FOR ERNEL DENSITY ESTIMATION Noparametric kerel estimatio is extremely sesitive to te coice of badwidt as larger values of result i averagig over more

### LECTURE 8: ASYMPTOTICS I

LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece

### Topic 9: Sampling Distributions of Estimators

Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

### Lecture 19: Convergence

Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may

### Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.

Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator

### Convergence of random variables. (telegram style notes) P.J.C. Spreij

Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space

### Lecture 33: Bootstrap

Lecture 33: ootstrap Motivatio To evaluate ad compare differet estimators, we eed cosistet estimators of variaces or asymptotic variaces of estimators. This is also importat for hypothesis testig ad cofidece

### Let us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.

Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,

### 32 estimating the cumulative distribution function

32 estimatig the cumulative distributio fuctio 4.6 types of cofidece itervals/bads Let F be a class of distributio fuctios F ad let θ be some quatity of iterest, such as the mea of F or the whole fuctio

### Study the bias (due to the nite dimensional approximation) and variance of the estimators

2 Series Methods 2. Geeral Approach A model has parameters (; ) where is ite-dimesioal ad is oparametric. (Sometimes, there is o :) We will focus o regressio. The fuctio is approximated by a series a ite

### Sequences and Series of Functions

Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges

### Distribution of Random Samples & Limit theorems

STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to

### Lecture 8: Convergence of transformations and law of large numbers

Lecture 8: Covergece of trasformatios ad law of large umbers Trasformatio ad covergece Trasformatio is a importat tool i statistics. If X coverges to X i some sese, we ofte eed to check whether g(x ) coverges

### Topic 9: Sampling Distributions of Estimators

Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

### ECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors

ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS

MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak

### Topic 9: Sampling Distributions of Estimators

Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

### Lecture 20: Multivariate convergence and the Central Limit Theorem

Lecture 20: Multivariate covergece ad the Cetral Limit Theorem Covergece i distributio for radom vectors Let Z,Z 1,Z 2,... be radom vectors o R k. If the cdf of Z is cotiuous, the we ca defie covergece

### Kolmogorov-Smirnov type Tests for Local Gaussianity in High-Frequency Data

Proceedigs 59th ISI World Statistics Cogress, 5-30 August 013, Hog Kog (Sessio STS046) p.09 Kolmogorov-Smirov type Tests for Local Gaussiaity i High-Frequecy Data George Tauche, Duke Uiversity Viktor Todorov,

### Chapter 3. Strong convergence. 3.1 Definition of almost sure convergence

Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i

### Unbiased Estimation. February 7-12, 2008

Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom

### EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1

EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum

### Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.

Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio

### 1 The Haar functions and the Brownian motion

1 The Haar fuctios ad the Browia motio 1.1 The Haar fuctios ad their completeess The Haar fuctios The basic Haar fuctio is 1 if x < 1/2, ψx) = 1 if 1/2 x < 1, otherwise. 1.1) It has mea zero 1 ψx)dx =,

### ELEN6887 Lecture 13: Maximum Likelihood Estimation and Complexity Regularization

ELEN6887 Lecture 3: Maximum Likelihood Estimatio ad Complexity Regularizatio R Castro 3/0/00 Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio Y i iid p θ, i,,

### An Introduction to Asymptotic Theory

A Itroductio to Asymptotic Theory Pig Yu School of Ecoomics ad Fiace The Uiversity of Hog Kog Pig Yu (HKU) Asymptotic Theory 1 / 20 Five Weapos i Asymptotic Theory Five Weapos i Asymptotic Theory Pig Yu

### Notes 5 : More on the a.s. convergence of sums

Notes 5 : More o the a.s. covergece of sums Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: Dur0, Sectios.5; Wil9, Sectio 4.7, Shi96, Sectio IV.4, Dur0, Sectio.. Radom series. Three-series

### ECE 901 Lecture 14: Maximum Likelihood Estimation and Complexity Regularization

ECE 90 Lecture 4: Maximum Likelihood Estimatio ad Complexity Regularizatio R Nowak 5/7/009 Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio Y i iid p θ, i,, where

### Integrable Functions. { f n } is called a determining sequence for f. If f is integrable with respect to, then f d does exist as a finite real number

MATH 532 Itegrable Fuctios Dr. Neal, WKU We ow shall defie what it meas for a measurable fuctio to be itegrable, show that all itegral properties of simple fuctios still hold, ad the give some coditios

### 17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15

17. Joit distributios of extreme order statistics Lehma 5.1; Ferguso 15 I Example 10., we derived the asymptotic distributio of the maximum from a radom sample from a uiform distributio. We did this usig

### MA Advanced Econometrics: Properties of Least Squares Estimators

MA Advaced Ecoometrics: Properties of Least Squares Estimators Karl Whela School of Ecoomics, UCD February 5, 20 Karl Whela UCD Least Squares Estimators February 5, 20 / 5 Part I Least Squares: Some Fiite-Sample

### 1 Convergence in Probability and the Weak Law of Large Numbers

36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec

### Lecture 27: Optimal Estimators and Functional Delta Method

Stat210B: Theoretical Statistics Lecture Date: April 19, 2007 Lecture 27: Optimal Estimators ad Fuctioal Delta Method Lecturer: Michael I. Jorda Scribe: Guilherme V. Rocha 1 Achievig Optimal Estimators

### Lecture 7: Density Estimation: k-nearest Neighbor and Basis Approach

STAT 425: Itroductio to Noparametric Statistics Witer 28 Lecture 7: Desity Estimatio: k-nearest Neighbor ad Basis Approach Istructor: Ye-Chi Che Referece: Sectio 8.4 of All of Noparametric Statistics.

### This section is optional.

4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore

### ECE 901 Lecture 12: Complexity Regularization and the Squared Loss

ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality

### 6.3 Testing Series With Positive Terms

6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial

### Probability 2 - Notes 10. Lemma. If X is a random variable and g(x) 0 for all x in the support of f X, then P(g(X) 1) E[g(X)].

Probability 2 - Notes 0 Some Useful Iequalities. Lemma. If X is a radom variable ad g(x 0 for all x i the support of f X, the P(g(X E[g(X]. Proof. (cotiuous case P(g(X Corollaries x:g(x f X (xdx x:g(x

### Stat 421-SP2012 Interval Estimation Section

Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible

1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable

### Lecture 3. Properties of Summary Statistics: Sampling Distribution

Lecture 3 Properties of Summary Statistics: Samplig Distributio Mai Theme How ca we use math to justify that our umerical summaries from the sample are good summaries of the populatio? Lecture Summary

### Lecture 15: Density estimation

Lecture 15: Desity estimatio Why do we estimate a desity? Suppose that X 1,...,X are i.i.d. radom variables from F ad that F is ukow but has a Lebesgue p.d.f. f. Estimatio of F ca be doe by estimatig f.

### 1 Approximating Integrals using Taylor Polynomials

Seughee Ye Ma 8: Week 7 Nov Week 7 Summary This week, we will lear how we ca approximate itegrals usig Taylor series ad umerical methods. Topics Page Approximatig Itegrals usig Taylor Polyomials. Defiitios................................................

### 7.1 Convergence of sequences of random variables

Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

### Random Variables, Sampling and Estimation

Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig

### Chapter 6 Infinite Series

Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat

### Statistical Inference Based on Extremum Estimators

T. Rotheberg Fall, 2007 Statistical Iferece Based o Extremum Estimators Itroductio Suppose 0, the true value of a p-dimesioal parameter, is kow to lie i some subset S R p : Ofte we choose to estimate 0

### The variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.

SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample

### Definition 4.2. (a) A sequence {x n } in a Banach space X is a basis for X if. unique scalars a n (x) such that x = n. a n (x) x n. (4.

4. BASES I BAACH SPACES 39 4. BASES I BAACH SPACES Sice a Baach space X is a vector space, it must possess a Hamel, or vector space, basis, i.e., a subset {x γ } γ Γ whose fiite liear spa is all of X ad

### CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5

CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio

### Elements of Statistical Methods Lots of Data or Large Samples (Ch 8)

Elemets of Statistical Methods Lots of Data or Large Samples (Ch 8) Fritz Scholz Sprig Quarter 2010 February 26, 2010 x ad X We itroduced the sample mea x as the average of the observed sample values x

### Lecture 2: Monte Carlo Simulation

STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?

### A RANK STATISTIC FOR NON-PARAMETRIC K-SAMPLE AND CHANGE POINT PROBLEMS

J. Japa Statist. Soc. Vol. 41 No. 1 2011 67 73 A RANK STATISTIC FOR NON-PARAMETRIC K-SAMPLE AND CHANGE POINT PROBLEMS Yoichi Nishiyama* We cosider k-sample ad chage poit problems for idepedet data i a

### FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals

### Journal of Multivariate Analysis. Superefficient estimation of the marginals by exploiting knowledge on the copula

Joural of Multivariate Aalysis 102 (2011) 1315 1319 Cotets lists available at ScieceDirect Joural of Multivariate Aalysis joural homepage: www.elsevier.com/locate/jmva Superefficiet estimatio of the margials

### STA Object Data Analysis - A List of Projects. January 18, 2018

STA 6557 Jauary 8, 208 Object Data Aalysis - A List of Projects. Schoeberg Mea glaucomatous shape chages of the Optic Nerve Head regio i aimal models 2. Aalysis of VW- Kedall ati-mea shapes with a applicatio

### STAT Homework 1 - Solutions

STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better

### Exponential Families and Bayesian Inference

Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where

### Jacob Hays Amit Pillay James DeFelice 4.1, 4.2, 4.3

No-Parametric Techiques Jacob Hays Amit Pillay James DeFelice 4.1, 4.2, 4.3 Parametric vs. No-Parametric Parametric Based o Fuctios (e.g Normal Distributio) Uimodal Oly oe peak Ulikely real data cofies

### Notes On Median and Quantile Regression. James L. Powell Department of Economics University of California, Berkeley

Notes O Media ad Quatile Regressio James L. Powell Departmet of Ecoomics Uiversity of Califoria, Berkeley Coditioal Media Restrictios ad Least Absolute Deviatios It is well-kow that the expected value

### Advanced Stochastic Processes.

Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.

### Notes On Nonparametric Density Estimation. James L. Powell Department of Economics University of California, Berkeley

Notes O Noparametric Desity Estimatio James L Powell Departmet of Ecoomics Uiversity of Califoria, Berkeley Uivariate Desity Estimatio via Numerical Derivatives Cosider te problem of estimatig te desity

### MAT1026 Calculus II Basic Convergence Tests for Series

MAT026 Calculus II Basic Covergece Tests for Series Egi MERMUT 202.03.08 Dokuz Eylül Uiversity Faculty of Sciece Departmet of Mathematics İzmir/TURKEY Cotets Mootoe Covergece Theorem 2 2 Series of Real

### Output Analysis and Run-Length Control

IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%

### Chapter 2 The Monte Carlo Method

Chapter 2 The Mote Carlo Method The Mote Carlo Method stads for a broad class of computatioal algorithms that rely o radom sampligs. It is ofte used i physical ad mathematical problems ad is most useful

### 1 = δ2 (0, ), Y Y n nδ. , T n = Y Y n n. ( U n,k + X ) ( f U n,k + Y ) n 2n f U n,k + θ Y ) 2 E X1 2 X1

8. The cetral limit theorems 8.1. The cetral limit theorem for i.i.d. sequeces. ecall that C ( is N -separatig. Theorem 8.1. Let X 1, X,... be i.i.d. radom variables with EX 1 = ad EX 1 = σ (,. Suppose

### Lecture Chapter 6: Convergence of Random Sequences

ECE5: Aalysis of Radom Sigals Fall 6 Lecture Chapter 6: Covergece of Radom Sequeces Dr Salim El Rouayheb Scribe: Abhay Ashutosh Doel, Qibo Zhag, Peiwe Tia, Pegzhe Wag, Lu Liu Radom sequece Defiitio A ifiite

### Lecture 14: Maximum Likelihood and Complexity Regularization

ECE90 Sprig 007 Statistical Learig Theory Istructor: R Nowak Lecture 4: Maximum Likelihood ad Complexity Regularizatio Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio

### Large Sample Theory. Convergence. Central Limit Theorems Asymptotic Distribution Delta Method. Convergence in Probability Convergence in Distribution

Large Sample Theory Covergece Covergece i Probability Covergece i Distributio Cetral Limit Theorems Asymptotic Distributio Delta Method Covergece i Probability A sequece of radom scalars {z } = (z 1,z,

### Notes 19 : Martingale CLT

Notes 9 : Martigale CLT Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: [Bil95, Chapter 35], [Roc, Chapter 3]. Sice we have ot ecoutered weak covergece i some time, we first recall

### Expectation and Variance of a random variable

Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio

### Efficient GMM LECTURE 12 GMM II

DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet

### Lecture 3 The Lebesgue Integral

Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified

### Lecture 7: Properties of Random Samples

Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ

### Introduction to Extreme Value Theory Laurens de Haan, ISM Japan, Erasmus University Rotterdam, NL University of Lisbon, PT

Itroductio to Extreme Value Theory Laures de Haa, ISM Japa, 202 Itroductio to Extreme Value Theory Laures de Haa Erasmus Uiversity Rotterdam, NL Uiversity of Lisbo, PT Itroductio to Extreme Value Theory

### Frequentist Inference

Frequetist Iferece The topics of the ext three sectios are useful applicatios of the Cetral Limit Theorem. Without kowig aythig about the uderlyig distributio of a sequece of radom variables {X i }, for

### Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator

Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters

### Math Solutions to homework 6

Math 175 - Solutios to homework 6 Cédric De Groote November 16, 2017 Problem 1 (8.11 i the book): Let K be a compact Hermitia operator o a Hilbert space H ad let the kerel of K be {0}. Show that there

### Fall 2013 MTH431/531 Real analysis Section Notes

Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters

### Last Lecture. Wald Test

Last Lecture Biostatistics 602 - Statistical Iferece Lecture 22 Hyu Mi Kag April 9th, 2013 Is the exact distributio of LRT statistic typically easy to obtai? How about its asymptotic distributio? For testig

### 7.1 Convergence of sequences of random variables

Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

### Stochastic Simulation

Stochastic Simulatio 1 Itroductio Readig Assigmet: Read Chapter 1 of text. We shall itroduce may of the key issues to be discussed i this course via a couple of model problems. Model Problem 1 (Jackso

### Problem Set 4 Due Oct, 12

EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios

### Estimation of the Mean and the ACVF

Chapter 5 Estimatio of the Mea ad the ACVF A statioary process {X t } is characterized by its mea ad its autocovariace fuctio γ ), ad so by the autocorrelatio fuctio ρ ) I this chapter we preset the estimators

### STAT Homework 2 - Solutions

STAT-36700 Homework - Solutios Fall 08 September 4, 08 This cotais solutios for Homework. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better isight.

### Entropy and Ergodic Theory Lecture 5: Joint typicality and conditional AEP

Etropy ad Ergodic Theory Lecture 5: Joit typicality ad coditioal AEP 1 Notatio: from RVs back to distributios Let (Ω, F, P) be a probability space, ad let X ad Y be A- ad B-valued discrete RVs, respectively.

### EFFECTIVE WLLN, SLLN, AND CLT IN STATISTICAL MODELS

EFFECTIVE WLLN, SLLN, AND CLT IN STATISTICAL MODELS Ryszard Zieliński Ist Math Polish Acad Sc POBox 21, 00-956 Warszawa 10, Polad e-mail: rziel@impagovpl ABSTRACT Weak laws of large umbers (W LLN), strog

### Rates of Convergence by Moduli of Continuity

Rates of Covergece by Moduli of Cotiuity Joh Duchi: Notes for Statistics 300b March, 017 1 Itroductio I this ote, we give a presetatio showig the importace, ad relatioship betwee, the modulis of cotiuity

### 4. Partial Sums and the Central Limit Theorem

1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems

### This exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.

Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 6 9/23/2013. Brownian motion. Introduction

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 6 9/23/203 Browia motio. Itroductio Cotet.. A heuristic costructio of a Browia motio from a radom walk. 2. Defiitio ad basic properties

### Glivenko-Cantelli Classes

CS28B/Stat24B (Sprig 2008 Statistical Learig Theory Lecture: 4 Gliveko-Catelli Classes Lecturer: Peter Bartlett Scribe: Michelle Besi Itroductio This lecture will cover Gliveko-Catelli (GC classes ad itroduce

### 1.010 Uncertainty in Engineering Fall 2008

MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval

### Math 341 Lecture #31 6.5: Power Series

Math 341 Lecture #31 6.5: Power Series We ow tur our attetio to a particular kid of series of fuctios, amely, power series, f(x = a x = a 0 + a 1 x + a 2 x 2 + where a R for all N. I terms of a series

### Asymptotic Results for the Linear Regression Model

Asymptotic Results for the Liear Regressio Model C. Fli November 29, 2000 1. Asymptotic Results uder Classical Assumptios The followig results apply to the liear regressio model y = Xβ + ε, where X is

### Joint Probability Distributions and Random Samples. Jointly Distributed Random Variables. Chapter { }

UCLA STAT A Applied Probability & Statistics for Egieers Istructor: Ivo Diov, Asst. Prof. I Statistics ad Neurology Teachig Assistat: Neda Farziia, UCLA Statistics Uiversity of Califoria, Los Ageles, Sprig

### Approximations and more PMFs and PDFs

Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.