Kernel density estimator


 Norah Harrington
 10 months ago
 Views:
Transcription
1 Jauary, 07 NONPARAMETRIC ERNEL DENSITY ESTIMATION I this lecture, we discuss kerel estimatio of probability desity fuctios PDF Noparametric desity estimatio is oe of the cetral problems i statistics I ecoomics, oparametric desity estimatio plays importat roles i various areas such as, for example, idustrial orgaizatio Guerre et al, 000, empirical fiace AitSahalia, 996, ad etc These otes borrow from the followig sources: Li ad Racie 007, Paga ad Ullah 999, ad Härdle ad Lito 994 erel desity estimator Assumptio a Suppose {X i : i =,, } is a collectio of iid radom variables draw from a distributio with the CDF F ad PDF f b I the eighborhood N x of x, f is bouded ad twice cotiuously differetiable with bouded derivatives Whe discussig f x, we will implicitly assume that f x exists at x The ecoometricia s objective is to estimate f without imposig ay fuctioal form parametric assumptios o the PDF First, cosider estimatio of F Sice a estimator of F ca be costructed as F x = E {X i x}, ˆF x = {X i x} The fuctio ˆF x is called the empirical CDF of X i The WLLN implies that for all x, i= ˆF x p F x As a matter of fact, a stroger results ca be established GlivekoGatelli Theorem, see Chapter 9 of va der Vaart 998: sup ˆF x F x as 0 x R Next, by the CLT, / ˆF x F x d N 0, F x F x Furthermore, for ay x, x R, / ˆF x F x ad / ˆF x F x are joitly asymptotically ormal with mea zero ad the covariace F x x F x F x, where x x deotes the miimum betwee x ad x Sice df x F x + h F x h f x = = lim, dx h 0 h from, oe ca cosider the followig estimator for the PDF f: ˆf x = ˆF x + h ˆF x h h = {x h X i x + h }, h i=
2 where h is a small umber ote that we cosider cotiuously distributed radom variables, so that P X i = x h = 0 We write h istead of just h because, typically, it will be a fuctio of the sample size such that lim h = 0 Now, defie the followig kerel fuctio: The, the kerel PDF estimator is give by u = { u } ˆf x = h 3 i= Thus, with the kerel fuctio defied accordig to, the kerel desity estimator is a average umber of observatios i the small eighborhood of x as defied by the smoothig parameter or badwidth also kerel widow The kerel fuctio i is called uiform, because it correspods to the uiform distributio we have that u du = It has a disadvatage of givig equal weights to all observatios iside the h widow with the ceter at x, regardless of how close they are to the ceter Also, if oe cosiders ˆf x as a fuctio of x, it is rough havig jumps at the poits X i ±h, ad has a zero derivative everywhere else Those problems ca be resolved if oe cosiders alterative kerel fuctios, for example, the quadratic kerel: h u = 5 u { u } 6 The class of estimators 3 with a kerel satisfyig u du = is referred to as RoseblattParze erel Estimator Small sample properties of the kerel desity estimator We will make the followig assumptio cocerig : Assumptio a u du = b u = u c is compactly supported o [, ] ad bouded d u u du 0 The kerel desity estimator is biased: Lemma Uder Assumptios a ad a, E ˆf x f x = u f x + uh f x du Proof E ˆf x = h E i= = h E = h h h u x h f u du Next, usig chage of variable y = u x /h, u = x + yh, ad du = h dy, we obtai E ˆf x = u f x + uh du, ad the result follows sice f x u du = f x by Assumptio a
3 Lemma Uder Assumptios a ad a, the variace of ˆf x is give by V ar ˆf x = h u f x + uh du u f x + uh du Proof Sice the data are iid, V ar ˆf x = V ar h h = E E h h h h By the same chage of variable argumet as i the proof of Lemma, we obtai E u x = f u du h h = h u f x + uh du From Lemma, oe ca expect that the bias icreases with h ; a bigger badwidth implies that more observatios away from x have ozero weights which cotributes to the bias O the other had, the variace decreases with h, as the estimator averages over more observatios The theorem below establishes more formally the biasvariace tradeoff for the kerel estimator Let f s deote the sorder derivative of f: f s x = ds f x dx s Theorem Suppose that h 0 ad h as The, uder Assumptios ad, a E ˆf x f x = c x h + o h, where c x = f x u u du/ b V ar ˆf x = c x / h + O /, where c x = f x u du Proof Sice the first two derivative of f exist by Assumptio b, cosider the followig expasio for f x + uh : f x + uh = f x + f x uh + f x u u h, where x u lies betwee x ad x + uh From Lemma we have E ˆf x f x = u f x uh + f x u u h du = h u f x u u du = c h + h u f x u f x u du 4 The secod equality follows because by Assumptio b, u udu = 0 We will show ext that u f x u f x u du = o, 5 3
4 ad therefore the secod summad i 4 is o h By Assumptio c, we oly eed to cosider u ; by Assumptio b ad sice x u lies betwee x ad x + uh, f x u f f x sup z < z N x Next, sice h 0, Now, by the domiated covergece theorem, lim lim x u = x u f x u f x u du = = 0, u lim f x u f x u du which establishes 5 ad cocludes the proof of part a of the theorem For part b, u f x + uh du = f x u du h h + f x u udu + h u f x u u du = c h + O h, 6 sice u udu = 0 by symmetry Assumptio b, ad u f x u u du = O as i the proof of part a The result of part b follows from 6 ad Lemma Agai, Theorem shows the biasvariace tradeoff The optimal choice of badwidth ca be foud by miimizig some fuctio that combies bias ad variace, for example, the mea squared error MSE: MSE ˆf x = E ˆf x f x = V ar ˆf x + E ˆf x f x = c x h = c x h Miimizatio of the leadig term of MSE gives + c x h 4 + O + c x h 4 + o 4c x h 3 = c x h, or + o h 4 h + h 4 7 h = = /5 c x 4c x /5 f x u du /5 f x u u du /5 /5 4
5 Whe the optimal i the MSE sese badwidth is selected, either bias or variace compoets of the MSE domiate each other asymptotically as V ar = Bias = O 4/5 Whe the Itegrated MSE criterio is employed, MSE ˆf x dx, the optimal badwidth becomes u du /5 h = f x /5 dx u u du /5 /5 Let ˆσ deote the sample variace of the data The followig rules of thumb ofte used i practice: h = 364ˆσ u du /5 u u du /5 /5, which is optimal for f x N µ, σ, ad h = 06ˆσ /5, which is optimal for f x N µ, σ ad whe is the stadard ormal desity Cosistecy of the kerel desity estimator Cosistecy of ˆf x follows immediately from Theorem by Chebychev s iequality Corollary Suppose that h 0 ad h as The, uder Assumptios ad, ˆf x p f x Proof By Chebychev s iequality, P ˆf x f x > ε E ˆf x f x ε = c x ε + c x h 4 h ε 0, + o + h 4 h where the secod lie is by 7 A stroger result ca be give, see Newey 994 Suppose that f admits at least m cotiuous derivatives o some iterval [x, x ]; has at least m cotiuous derivatives, is compactly supported ad of order m: u j u du = 0 for all j =,, m ; u m u du 0, ad u du = The sup x [x,x ] ˆf / h x f x = O p + h m log The derivatives of f ca be estimated by the derivative of ˆf, however, with a slower rates of covergece Newey 994 shows that sup x [x,x ] ˆf k h x f k k / x = O p + h m log 5
6 Asymptotic ormality of the kerel desity estimator Write ˆf x = h i= h = v i, where v i = h i= h Note that h ad cosequetly v i deped o The collectio {{v i : i =,, } : N} is called a triagular array I our case, uder Assumptio, v i s are iid The followig CLT is available for idepedet triagular arrays Lehma ad Romao, 005, Corollary, page 47 Lemma 3 Lyapouov CLT Suppose that for each, w,, w are idepedet Assume that Ew i = 0 ad σi = Ew i <, ad defie s = i= σ i Suppose further that for some δ > 0 the followig coditio holds: lim E w i +δ = 0 8 The, i= s +δ w i /s d N 0, i= The coditio 8 is called Lyapouov s coditio Whe the data are ot just idepedet but iid, the Lyapouov s coditio ca be simplified as follows Davidso, 994, Theorem 3 o page 373 Lemma 4 The Lyapouov s coditio is satisfied whe w,, w are iid, σ = Ewi > 0 uiformly i, ad lim E w i +δ / δ/ = 0 for some δ > 0 Proof Sice the data are iid, We have i= s +δ s = σ, ad +δ s +δ = / σ = +δ/ σ +δ E w i +δ = σ δ δ/ E w i +δ i= = σ δ δ/ E w i +δ Therefore, the Lyapouov s coditio is satisfied if δ/ E w i +δ 0, sice σ is uiformly bouded away from zero by the assumptio Assumig that lim σ exists, i the iid case, the result of Lyapouov CLT ca be stated as follows Corollary Suppose that for each, w,, w are iid, Ew i = 0, lim Ewi lim E w i +δ / δ/ = 0 for some δ > 0 The, / w i d N i= 0, lim Ew i > 0 ad fiite, ad 6
7 Next, we prove asymptotic ormality of the kerel desity estimator Theorem Suppose that h ad h / h 0 Assume further that f x > 0 The, uder Assumptios ad, h / ˆf x f x d N 0, f x u du 9 Furthermore, for x x, h / ˆf x f x ad h / ˆf x f x are asymptotically idepedet Proof By Theorem a, Defie The, h / ˆf x f x = h E w i = h / / h / h i= + h / h E h h / ˆf x f x h = = h f x = O h / / h / E h h / f x E h w i + O p h / h 0 i= w i + o p, where the equality i the secod lie is by the assumptio that h / h 0 It is ow left to verify the coditios of Corollary By the defiitio of w i, Ew i = 0 Next, Ewi = E E h h h h As i the proof of Lemma ad by the domiated covergece theorem, E = h u f x + uh du h i= = O h, so that the secod summad i is O h ad asymptotically egligible For the first term i, we ca use the chage of variable argumet agai: E = u x du h h h h = u f x + uh du, f x u du, 3 7
8 where the last result is by the domiated covergece theorem The results i 3 together imply that lim Ew i = f x u du Lastly, we show that E w i +δ / δ/ 0 We will use the c r iequality Davidso, 994, Theorem 98 o page 40 i order to deal with E w i +δ : for r > 0, m r m E X i c r E X i r, i= where c r = whe r, ad c r = m r whe r Now, by the c r iequality, E w i +δ +δ E h +δ/ +δ + h h +δ/ E +δ h By, Further, h +δ/ h +δ/ E +δ h i= E +δ h = = h +δ/ h δ/ = O h δ/ = O h +δ/ u x h +δ f u du u +δ f x + uh du where the equality i the last lie is agai by the domiated covergece theorem Hece, E w i +δ δ/ = O δ/ h This completes the proof of 9 I order to show asymptotic idepedece of ˆf x ad ˆf x, cosider their asymptotic covariace: E = u x u x f u du h h h h h h = u u + x x f x + uh du h Sice the kerel fuctio is compactly supported ad lim x x /h =, lim u + x x = 0, h ad by the domiated covergece theorem, u lim u + x x h Asymptotic idepedece the follows by the CramerWold device, f x + uh du = 0 8
9 From 0, oe ca see that the assumptio h / h 0 is used to make the bias asymptotically egligible Cosequetly, there is udersmoothig relatively to the MSEoptimal badwidth, ad the bias goes to zero at a faster rate tha the variace Suppose that the badwidth is chose accordig to h = c α The, h / h / α/ α = 5α/, ad for h / h 0 to hold, we eed that 5α < 0 or α > /5 Thus, for asymptotic ormality, the badwidth is o /5, while the MSEoptimal badwidth is h = c /5 A more geeral statemet of the asymptotic ormality result that also icludes the bias result, ie without imposig udersmoothig is h / ˆf x fx 05h f x u udu d N 0, fx udu 4 The result i 4 holds provided that h ad does ot require that h / h 0 I particular, if oe chooses h = ah /5, the h / ˆf x fx d N a5/ f x u udu, fx udu, ad the kerel desity estimator is asymptotically biased Multivariate kerel desity estimatio ad the curse of dimesioality Suppose ow that {X i : i =,, } is a collectio of iid radom dvectors draw from a distributio with a joit PDF f x,, x d The uivariate kerel desity estimator ca be exteded to the multivariate case as follows: ˆf x,, x d = d i= j= h Xij x j h = h d i= j= d Xij x j ote h d i the deomiator istead of h Oe ca see that the multivariate kerel desity estimator is a extesio of uivariate kerel smoothig to d dimesios or d variables I the multivariate case, oe ca establish results similar to those of the uivariate case To simplify the otatio, let ad write Also for u = u,, u d R d, let x = x x d R d, fx,, x d = fx d u = d u j, j= h, 9
10 so that ˆf x,, x d = ˆf x = h d i= dx i x/h Note that d udu = u du d u d du d = u du =, where the secod equality follows by Assumptio a Similar results to those show for the uivariate estimator ca be established i the multivariate case Assumptio 3 a Suppose {X i : i =,, } is a collectio of iid radom vectors draw from the distributio with a joit PDF f b I the eighborhood N x of x, f is bouded ad twice cotiuously differetiable with bouded partial derivatives Theorem 3 Suppose that h 0 ad h as The, uder Assumptios ad 3, fx x j a E ˆf x f x = fx + h u d u du j= b V ar ˆf x = fx u h d du + O /h d + / + oh Proof For part a, E ˆf x = h d u x d fudu = h d vfx + h vdv, where we used the chage of variable v = u x/h, u = x + h v, du j du = du du d, dv = dv dv d Next, = h dv j for j =,, d, ad fx + h v = fx + h v fx x + h v fx v x x v, where x v deotes the meavalue satisfyig x v x h v, ie it lies betwee x ad x + h v Sice the kerel fuctio is symmetric aroud zero, v d vdv = 0 By Assumptio 3b ad the same argumets as i the proof of Theorem a, fx v x x fx x x Hece, E ˆf x = fx + h v fx x x v dvdv + oh = fx + d d fx h v i v j d vdv + oh x i x j = fx + h = fx + h d j= i= j= fx x j v v dv d j= v j v j dv j + h fx x j d i= j i + oh, = o fx x i x j v i v i dv i v j v j dv j + oh 0
11 where the equality i the last lie holds due to the symmetry of the kerel fuctio u aroud zero For part b, V ar ˆf x = V ar d h d h [ = h d Ed h = [ h d Ed h = [ h d Ed h = h d Ed h = u x h d d h = h d dufx + h udu + O = h d du fx + h u fx x + h u fx v x x = d h h d fx u du + O h d +, ] h d E d h fx + Oh ] holds by the result i a ] f x + Oh + O fudu + O u du + O where the last lie follows sice ud udu = 0 due to the symmetry of the kerel fuctio aroud zero, ad because d udu = d u du The bias ad variace calculatios imply that ˆf x = fx + O p + h h d, ad therefore the rate of covergece slows dow with the umber of variables d I the oparametric literature, this is referred to as the curse of dimesioality Oe ca derive the MSEpoitoptimal badwidth cosiderig the leadig terms i the bias ad variace expressios implied by Theorem 3: MSE x = 4 h4 u u du Miimizig the MSE with respect to h, we obtai: h 3 u u du d j= Therefore, the MSE optimal badwidth is give by d j= fx x j fx d u du h = d u d u du j= fx x j fx x j + fx d u du h d = d fx d u du h d+ /d+4 /d+4
12 Oe ca see that the rate of the optimal badwidth, /d+4, icreases with the umber of variables d, ie oe should use larger values for the badwidth whe there are more variables Oe ca exted the uivariate CLT to the multivariate case as follows: h d / ˆf x fx d h u d fx u du x d N 0, fx u du j= j To elimiate the asymptotic bias, oe has to choose a udersmoothig badwidth so that h d / h 0 Oe ca also icorporate differet badwidth values for differet variables: ˆf x,, x d = h h h d i= j= The bias ad variace results i this case take the followig form: d Xij x j E ˆf x = fx + d u fx u du h j + o h + + h d x j= j V ar ˆf x = fx d u du h + O + + h d + h h d h h d h j Aalogously, the CLT statemet ca be modified as h h d / ˆf x fx d u u du j= fx x j h j d N 0, fx u du d
13 Bibliography AitSahalia, Y 996: Testig CotiuousTime Models of the Spot Iterest Rate, The Review of Fiacial Studies, 9, Davidso, J 994: Stochastic Limit Theory, New York: Oxford Uiversity Press Guerre, E, I Perrige, ad Q Vuog 000: Optimal Noparametric Estimatio of FirstPrice Auctios, Ecoometrica, 68, Härdle, W ad O Lito 994: Applied Noparametric Methods, i Hadbook of Ecoometrics, ed by R F Egle ad D L McFadde, Amsterdam: Elsevier, vol 4, chap 38, Lehma, E L ad J P Romao 005: Testig Statistical Hypotheses, New York: Spriger, third ed Li, Q ad J S Racie 007: Noparametric Ecoometrics: Theory ad Practice, Priceto, New Jersey: Priceto Uiversity Press Newey, W 994: erel Estimatio of Partial Meas ad a Geeral Variace Estimator, Ecoometric Theory, 0, Paga, A ad A Ullah 999: Noparametric Ecoometrics, New York: Cambridge Uiversity Press va der Vaart, A W 998: Asymptotic Statistics, Cambridge: Cambridge Uiversity Press 3
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chisquare Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chisquare Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More informationIntegrable Functions. { f n } is called a determining sequence for f. If f is integrable with respect to, then f d does exist as a finite real number
MATH 532 Itegrable Fuctios Dr. Neal, WKU We ow shall defie what it meas for a measurable fuctio to be itegrable, show that all itegral properties of simple fuctios still hold, ad the give some coditios
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationThe variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.
SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample
More informationDefinition 4.2. (a) A sequence {x n } in a Banach space X is a basis for X if. unique scalars a n (x) such that x = n. a n (x) x n. (4.
4. BASES I BAACH SPACES 39 4. BASES I BAACH SPACES Sice a Baach space X is a vector space, it must possess a Hamel, or vector space, basis, i.e., a subset {x γ } γ Γ whose fiite liear spa is all of X ad
More informationA RANK STATISTIC FOR NONPARAMETRIC KSAMPLE AND CHANGE POINT PROBLEMS
J. Japa Statist. Soc. Vol. 41 No. 1 2011 67 73 A RANK STATISTIC FOR NONPARAMETRIC KSAMPLE AND CHANGE POINT PROBLEMS Yoichi Nishiyama* We cosider ksample ad chage poit problems for idepedet data i a
More informationOutput Analysis and RunLength Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad RuLegth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More informationJacob Hays Amit Pillay James DeFelice 4.1, 4.2, 4.3
NoParametric Techiques Jacob Hays Amit Pillay James DeFelice 4.1, 4.2, 4.3 Parametric vs. NoParametric Parametric Based o Fuctios (e.g Normal Distributio) Uimodal Oly oe peak Ulikely real data cofies
More informationMAT1026 Calculus II Basic Convergence Tests for Series
MAT026 Calculus II Basic Covergece Tests for Series Egi MERMUT 202.03.08 Dokuz Eylül Uiversity Faculty of Sciece Departmet of Mathematics İzmir/TURKEY Cotets Mootoe Covergece Theorem 2 2 Series of Real
More informationLecture 3 The Lebesgue Integral
Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 6 9/23/2013. Brownian motion. Introduction
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 6 9/23/203 Browia motio. Itroductio Cotet.. A heuristic costructio of a Browia motio from a radom walk. 2. Defiitio ad basic properties
More informationJoint Probability Distributions and Random Samples. Jointly Distributed Random Variables. Chapter { }
UCLA STAT A Applied Probability & Statistics for Egieers Istructor: Ivo Diov, Asst. Prof. I Statistics ad Neurology Teachig Assistat: Neda Farziia, UCLA Statistics Uiversity of Califoria, Los Ageles, Sprig
More informationAsymptotic Results for the Linear Regression Model
Asymptotic Results for the Liear Regressio Model C. Fli November 29, 2000 1. Asymptotic Results uder Classical Assumptios The followig results apply to the liear regressio model y = Xβ + ε, where X is
More informationFirst Year Quantitative Comp Exam Spring, Part I  203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I1 Part I  203A A radom variable X is distributed with the margial desity: >
More informationLaw of the sum of Bernoulli random variables
Law of the sum of Beroulli radom variables Nicolas Chevallier Uiversité de Haute Alsace, 4, rue des frères Lumière 68093 Mulhouse icolas.chevallier@uha.fr December 006 Abstract Let be the set of all possible
More informationMonte Carlo Integration
Mote Carlo Itegratio I these otes we first review basic umerical itegratio methods (usig Riema approximatio ad the trapezoidal rule) ad their limitatios for evaluatig multidimesioal itegrals. Next we itroduce
More information5.1 A mutual information bound based on metric entropy
Chapter 5 Global Fao Method I this chapter, we exted the techiques of Chapter 2.4 o Fao s method the local Fao method) to a more global costructio. I particular, we show that, rather tha costructig a local
More informationDS 100: Principles and Techniques of Data Science Date: April 13, Discussion #10
DS 00: Priciples ad Techiques of Data Sciece Date: April 3, 208 Name: Hypothesis Testig Discussio #0. Defie these terms below as they relate to hypothesis testig. a) Data Geeratio Model: Solutio: A set
More informationINFINITE SEQUENCES AND SERIES
11 INFINITE SEQUENCES AND SERIES INFINITE SEQUENCES AND SERIES 11.4 The Compariso Tests I this sectio, we will lear: How to fid the value of a series by comparig it with a kow series. COMPARISON TESTS
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More information6. Uniform distribution mod 1
6. Uiform distributio mod 1 6.1 Uiform distributio ad Weyl s criterio Let x be a seuece of real umbers. We may decompose x as the sum of its iteger part [x ] = sup{m Z m x } (i.e. the largest iteger which
More informationRegression with an Evaporating Logarithmic Trend
Regressio with a Evaporatig Logarithmic Tred Peter C. B. Phillips Cowles Foudatio, Yale Uiversity, Uiversity of Aucklad & Uiversity of York ad Yixiao Su Departmet of Ecoomics Yale Uiversity October 5,
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationProbability and Statistics
ICME Refresher Course: robability ad Statistics Staford Uiversity robability ad Statistics Luyag Che September 20, 2016 1 Basic robability Theory 11 robability Spaces A probability space is a triple (Ω,
More informationLecture 6 Ecient estimators. RaoCramer bound.
Lecture 6 Eciet estimators. RaoCramer boud. 1 MSE ad Suciecy Let X (X 1,..., X) be a radom sample from distributio f θ. Let θ ˆ δ(x) be a estimator of θ. Let T (X) be a suciet statistic for θ. As we have
More information2.2. Central limit theorem.
36.. Cetral limit theorem. The most ideal case of the CLT is that the radom variables are iid with fiite variace. Although it is a special case of the more geeral LidebergFeller CLT, it is most stadard
More informationA sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as
More informationIt should be unbiased, or approximately unbiased. Variance of the variance estimator should be small. That is, the variance estimator is stable.
Chapter 10 Variace Estimatio 10.1 Itroductio Variace estimatio is a importat practical problem i survey samplig. Variace estimates are used i two purposes. Oe is the aalytic purpose such as costructig
More informationSampling Distributions, ZTests, Power
Samplig Distributios, ZTests, Power We draw ifereces about populatio parameters from sample statistics Sample proportio approximates populatio proportio Sample mea approximates populatio mea Sample variace
More informationSTAT331. Example of Martingale CLT with Cox s Model
STAT33 Example of Martigale CLT with Cox s Model I this uit we illustrate the Martigale Cetral Limit Theorem by applyig it to the partial likelihood score fuctio from Cox s model. For simplicity of presetatio
More informationLesson 10: Limits and Continuity
www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals
More informationSingular Continuous Measures by Michael Pejic 5/14/10
Sigular Cotiuous Measures by Michael Peic 5/4/0 Prelimiaries Give a set X, a σalgebra o X is a collectio of subsets of X that cotais X ad ad is closed uder complemetatio ad coutable uios hece, coutable
More informationMath 2784 (or 2794W) University of Connecticut
ORDERS OF GROWTH PAT SMITH Math 2784 (or 2794W) Uiversity of Coecticut Date: Mar. 2, 22. ORDERS OF GROWTH. Itroductio Gaiig a ituitive feel for the relative growth of fuctios is importat if you really
More informationLECTURE 11 LINEAR PROCESSES III: ASYMPTOTIC RESULTS
PRIL 7, 9 where LECTURE LINER PROCESSES III: SYMPTOTIC RESULTS (Phillips ad Solo (99) ad Phillips Lecture Notes o Statioary ad Nostatioary Time Series) I this lecture, we discuss the LLN ad CLT for a liear
More informationStatistical Theory MT 2009 Problems 1: Solution sketches
Statistical Theory MT 009 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. (a) Let 0 < θ < ad put f(x, θ) = ( θ)θ x ; x = 0,,,... (b) (c) where
More informationStatisticians use the word population to refer the total number of (potential) observations under consideration
6 Samplig Distributios Statisticias use the word populatio to refer the total umber of (potetial) observatios uder cosideratio The populatio is just the set of all possible outcomes i our sample space
More informationIIT JAM Mathematical Statistics (MS) 2006 SECTION A
IIT JAM Mathematical Statistics (MS) 6 SECTION A. If a > for ad lim a / L >, the which of the followig series is ot coverget? (a) (b) (c) (d) (d) = = a = a = a a + / a lim a a / + = lim a / a / + = lim
More informationLecture 11 October 27
STATS 300A: Theory of Statistics Fall 205 Lecture October 27 Lecturer: Lester Mackey Scribe: Viswajith Veugopal, Vivek Bagaria, Steve Yadlowsky Warig: These otes may cotai factual ad/or typographic errors..
More informationFundamental Theorem of Algebra. Yvonne Lai March 2010
Fudametal Theorem of Algebra Yvoe Lai March 010 We prove the Fudametal Theorem of Algebra: Fudametal Theorem of Algebra. Let f be a ocostat polyomial with real coefficiets. The f has at least oe complex
More informationGamma Distribution and Gamma Approximation
Gamma Distributio ad Gamma Approimatio Xiaomig Zeg a Fuhua (Frak Cheg b a Xiame Uiversity, Xiame 365, Chia mzeg@jigia.mu.edu.c b Uiversity of Ketucky, Leigto, Ketucky 45646, USA cheg@cs.uky.edu Abstract
More informationREGRESSION WITH QUADRATIC LOSS
REGRESSION WITH QUADRATIC LOSS MAXIM RAGINSKY Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X, Y ), where, as before, X is a R d
More informationAn alternative proof of a theorem of Aldous. concerning convergence in distribution for martingales.
A alterative proof of a theorem of Aldous cocerig covergece i distributio for martigales. Maurizio Pratelli Dipartimeto di Matematica, Uiversità di Pisa. Via Buoarroti 2. I56127 Pisa, Italy email: pratelli@dm.uipi.it
More informationTHE SYSTEMATIC AND THE RANDOM. ERRORS  DUE TO ELEMENT TOLERANCES OF ELECTRICAL NETWORKS
R775 Philips Res. Repts 26,414423, 1971' THE SYSTEMATIC AND THE RANDOM. ERRORS  DUE TO ELEMENT TOLERANCES OF ELECTRICAL NETWORKS by H. W. HANNEMAN Abstract Usig the law of propagatio of errors, approximated
More informationRademacher Complexity
EECS 598: Statistical Learig Theory, Witer 204 Topic 0 Rademacher Complexity Lecturer: Clayto Scott Scribe: Ya Deg, Kevi Moo Disclaimer: These otes have ot bee subjected to the usual scrutiy reserved for
More informationThe standard deviation of the mean
Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider
More informationBinomial Distribution
0.0 0.5 1.0 1.5 2.0 2.5 3.0 0 1 2 3 4 5 6 7 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Overview Example: coi tossed three times Defiitio Formula Recall that a r.v. is discrete if there are either a fiite umber of possible
More informationSEMIPARAMETRIC SINGLEINDEX MODELS. Joel L. Horowitz Department of Economics Northwestern University
SEMIPARAMETRIC SINGLEINDEX MODELS by Joel L. Horowitz Departmet of Ecoomics Northwester Uiversity INTRODUCTION Much of applied ecoometrics ad statistics ivolves estimatig a coditioal mea fuctio: E ( Y
More informationSolution. 1 Solutions of Homework 1. Sangchul Lee. October 27, Problem 1.1
Solutio Sagchul Lee October 7, 017 1 Solutios of Homework 1 Problem 1.1 Let Ω,F,P) be a probability space. Show that if {A : N} F such that A := lim A exists, the PA) = lim PA ). Proof. Usig the cotiuity
More informationProbability and statistics: basic terms
Probability ad statistics: basic terms M. Veeraraghava August 203 A radom variable is a rule that assigs a umerical value to each possible outcome of a experimet. Outcomes of a experimet form the sample
More informationMAS111 Convergence and Continuity
MAS Covergece ad Cotiuity Key Objectives At the ed of the course, studets should kow the followig topics ad be able to apply the basic priciples ad theorems therei to solvig various problems cocerig covergece
More information71. Chapter 4. Part I. Sampling Distributions and Confidence Intervals
71 Chapter 4 Part I. Samplig Distributios ad Cofidece Itervals 1 7 Sectio 1. Samplig Distributio 73 Usig Statistics Statistical Iferece: Predict ad forecast values of populatio parameters... Test hypotheses
More information1 Covariance Estimation
Eco 75 Lecture 5 Covariace Estimatio ad Optimal Weightig Matrices I this lecture, we cosider estimatio of the asymptotic covariace matrix B B of the extremum estimator b : Covariace Estimatio Lemma 4.
More information4.1 Data processing inequality
ECE598: Iformatiotheoretic methods i highdimesioal statistics Sprig 206 Lecture 4: Total variatio/iequalities betwee fdivergeces Lecturer: Yihog Wu Scribe: Matthew Tsao, Feb 8, 206 [Ed. Mar 22] Recall
More informationLecture 01: the Central Limit Theorem. 1 Central Limit Theorem for i.i.d. random variables
CSCIB609: A Theorist s Toolkit, Fall 06 Aug 3 Lecture 0: the Cetral Limit Theorem Lecturer: Yua Zhou Scribe: Yua Xie & Yua Zhou Cetral Limit Theorem for iid radom variables Let us say that we wat to aalyze
More informationAnalytic Continuation
Aalytic Cotiuatio The stadard example of this is give by Example Let h (z) = 1 + z + z 2 + z 3 +... kow to coverge oly for z < 1. I fact h (z) = 1/ (1 z) for such z. Yet H (z) = 1/ (1 z) is defied for
More informationTechnische Universität Ilmenau Institut für Mathematik
Techische Uiversität Ilmeau Istitut für Mathematik Preprit No. M 03/14 Rates of cosistecy for oparametric estimatio of the mode i absece of smoothess assumptios Herrma, Eva; Ziegler, Klaus 2003 Impressum:
More informationo <Xln <X2n <... <X n < o (1.1)
Metrika, Volume 28, 1981, page 257262. 9 Viea. Estimatio Problems for Rectagular Distributios (Or the Taxi Problem Revisited) By J.S. Rao, Sata Barbara I ) Abstract: The problem of estimatig the ukow
More informationMathematics 170B Selected HW Solutions.
Mathematics 17B Selected HW Solutios. F 4. Suppose X is B(,p). (a)fidthemometgeeratigfuctiom (s)of(x p)/ p(1 p). Write q = 1 p. The MGF of X is (pe s + q), sice X ca be writte as the sum of idepedet Beroulli
More informationConfidence Intervals for the Population Proportion p
Cofidece Itervals for the Populatio Proportio p The cocept of cofidece itervals for the populatio proportio p is the same as the oe for, the samplig distributio of the mea, x. The structure is idetical:
More informationEksamen 2006 H Utsatt SENSORVEILEDNING. Problem 1. Settet består av 9 delspørsmål som alle anbefales å telle likt. Svar er gitt i <<.. >>.
Eco 43 Eksame 6 H Utsatt SENSORVEILEDNING Settet består av 9 delspørsmål som alle abefales å telle likt. Svar er gitt i . Problem a. Let the radom variable (rv.) X be expoetially distributed with
More informationSolutions to HW Assignment 1
Solutios to HW: 1 Course: Theory of Probability II Page: 1 of 6 Uiversity of Texas at Austi Solutios to HW Assigmet 1 Problem 1.1. Let Ω, F, {F } 0, P) be a filtered probability space ad T a stoppig time.
More informationDirichlet s Theorem on Arithmetic Progressions
Dirichlet s Theorem o Arithmetic Progressios Athoy Várilly Harvard Uiversity, Cambridge, MA 0238 Itroductio Dirichlet s theorem o arithmetic progressios is a gem of umber theory. A great part of its beauty
More informationCommutativity in Permutation Groups
Commutativity i Permutatio Groups Richard Wito, PhD Abstract I the group Sym(S) of permutatios o a oempty set S, fixed poits ad trasiet poits are defied Prelimiary results o fixed ad trasiet poits are
More informationHow to Maximize a Function without Really Trying
How to Maximize a Fuctio without Really Tryig MARK FLANAGAN School of Electrical, Electroic ad Commuicatios Egieerig Uiversity College Dubli We will prove a famous elemetary iequality called The Rearragemet
More informationB Supplemental Notes 2 Hypergeometric, Binomial, Poisson and Multinomial Random Variables and Borel Sets
B671672 Supplemetal otes 2 Hypergeometric, Biomial, Poisso ad Multiomial Radom Variables ad Borel Sets 1 Biomial Approximatio to the Hypergeometric Recall that the Hypergeometric istributio is fx = x
More informationRegression with quadratic loss
Regressio with quadratic loss Maxim Ragisky October 13, 2015 Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X,Y, where, as before,
More informationDiscrete probability distributions
Discrete probability distributios I the chapter o probability we used the classical method to calculate the probability of various values of a radom variable. I some cases, however, we may be able to develop
More informationsin(n) + 2 cos(2n) n 3/2 3 sin(n) 2cos(2n) n 3/2 a n =
60. Ratio ad root tests 60.1. Absolutely coverget series. Defiitio 13. (Absolute covergece) A series a is called absolutely coverget if the series of absolute values a is coverget. The absolute covergece
More informationLecture 1 Probability and Statistics
Wikipedia: Lecture 1 Probability ad Statistics Bejami Disraeli, British statesma ad literary figure (1804 1881): There are three kids of lies: lies, damed lies, ad statistics. popularized i US by Mark
More informationChapter 11 Output Analysis for a Single Model. Banks, Carson, Nelson & Nicol DiscreteEvent System Simulation
Chapter Output Aalysis for a Sigle Model Baks, Carso, Nelso & Nicol DiscreteEvet System Simulatio Error Estimatio If {,, } are ot statistically idepedet, the S / is a biased estimator of the true variace.
More informationParameter, Statistic and Random Samples
Parameter, Statistic ad Radom Samples A parameter is a umber that describes the populatio. It is a fixed umber, but i practice we do ot kow its value. A statistic is a fuctio of the sample data, i.e.,
More informationFUNDAMENTALS OF REAL ANALYSIS by
FUNDAMENTALS OF REAL ANALYSIS by Doğa Çömez Backgroud: All of Math 450/1 material. Namely: basic set theory, relatios ad PMI, structure of N, Z, Q ad R, basic properties of (cotiuous ad differetiable)
More informationECE 901 Lecture 4: Estimation of Lipschitz smooth functions
ECE 9 Lecture 4: Estiatio of Lipschitz sooth fuctios R. Nowak 5/7/29 Cosider the followig settig. Let Y f (X) + W, where X is a rado variable (r.v.) o X [, ], W is a r.v. o Y R, idepedet of X ad satisfyig
More informationLecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
More informationMath 21BB  Homework Set 2
Math BB  Homework Set Sectio 5.:. a) lim P k= c k c k ) x k, where P is a partitio of [, 5. x x ) dx b) lim P k= 4 ck x k, where P is a partitio of [,. 4 x dx c) lim P k= ta c k ) x k, where P is a partitio
More informationA constructive analysis of convexvalued demand correspondence for weakly uniformly rotund and monotonic preference
MPRA Muich Persoal RePEc Archive A costructive aalysis of covexvalued demad correspodece for weakly uiformly rotud ad mootoic preferece Yasuhito Taaka ad Atsuhiro Satoh. May 04 Olie at http://mpra.ub.uimueche.de/55889/
More informationSeunghee Ye Ma 8: Week 5 Oct 28
Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value
More informationSection 11.8: Power Series
Sectio 11.8: Power Series 1. Power Series I this sectio, we cosider geeralizig the cocept of a series. Recall that a series is a ifiite sum of umbers a. We ca talk about whether or ot it coverges ad i
More informationA goodnessoffit test based on the empirical characteristic function and a comparison of tests for normality
A goodessoffit test based o the empirical characteristic fuctio ad a compariso of tests for ormality J. Marti va Zyl Departmet of Mathematical Statistics ad Actuarial Sciece, Uiversity of the Free State,
More information6.867 Machine learning, lecture 7 (Jaakkola) 1
6.867 Machie learig, lecture 7 (Jaakkola) 1 Lecture topics: Kerel form of liear regressio Kerels, examples, costructio, properties Liear regressio ad kerels Cosider a slightly simpler model where we omit
More informationMachine Learning Theory Tübingen University, WS 2016/2017 Lecture 12
Machie Learig Theory Tübige Uiversity, WS 06/07 Lecture Tolstikhi Ilya Abstract I this lecture we derive risk bouds for kerel methods. We will start by showig that Soft Margi kerel SVM correspods to miimizig
More informationIn this section, we show how to use the integral test to decide whether a series
Itegral Test Itegral Test Example Itegral Test Example pseries Compariso Test Example Example 2 Example 3 Example 4 Example 5 Exa Itegral Test I this sectio, we show how to use the itegral test to decide
More informationNonparametric estimation of conditional distributions
Noparametric estimatio of coditioal distributios László Györfi 1 ad Michael Kohler 2 1 Departmet of Computer Sciece ad Iformatio Theory, udapest Uiversity of Techology ad Ecoomics, 1521 Stoczek, U.2, udapest,
More informationProbability, Expectation Value and Uncertainty
Chapter 1 Probability, Expectatio Value ad Ucertaity We have see that the physically observable properties of a quatum system are represeted by Hermitea operators (also referred to as observables ) such
More informationSolutions to Tutorial 5 (Week 6)
The Uiversity of Sydey School of Mathematics ad Statistics Solutios to Tutorial 5 (Wee 6 MATH2962: Real ad Complex Aalysis (Advaced Semester, 207 Web Page: http://www.maths.usyd.edu.au/u/ug/im/math2962/
More informationR. van Zyl 1, A.J. van der Merwe 2. Quintiles International, University of the Free State
Bayesia Cotrol Charts for the Twoparameter Expoetial Distributio if the Locatio Parameter Ca Take o Ay Value Betwee Mius Iity ad Plus Iity R. va Zyl, A.J. va der Merwe 2 Quitiles Iteratioal, ruaavz@gmail.com
More informationPRACTICE FINAL/STUDY GUIDE SOLUTIONS
Last edited December 9, 03 at 4:33pm) Feel free to sed me ay feedback, icludig commets, typos, ad mathematical errors Problem Give the precise meaig of the followig statemets i) a f) L ii) a + f) L iii)
More information18.657: Mathematics of Machine Learning
8.657: Mathematics of Machie Learig Lecturer: Philippe Rigollet Lecture 4 Scribe: Cheg Mao Sep., 05 I this lecture, we cotiue to discuss the effect of oise o the rate of the excess risk E(h) = R(h) R(h
More informationElement sampling: Part 2
Chapter 4 Elemet samplig: Part 2 4.1 Itroductio We ow cosider uequal probability samplig desigs which is very popular i practice. I the uequal probability samplig, we ca improve the efficiecy of the resultig
More information2.4.2 A Theorem About Absolutely Convergent Series
0 Versio of August 27, 200 CHAPTER 2. INFINITE SERIES Add these two series: + 3 2 + 5 + 7 4 + 9 + 6 +... = 3 l 2. (2.20) 2 Sice the reciprocal of each iteger occurs exactly oce i the last series, we would
More informationTesting Statistical Hypotheses for Compare. Means with Vague Data
Iteratioal Mathematical Forum 5 o. 3 656 Testig Statistical Hypotheses for Compare Meas with Vague Data E. Baloui Jamkhaeh ad A. adi Ghara Departmet of Statistics Islamic Azad iversity Ghaemshahr Brach
More informationBHW #13 1/ Cooper. ENGR 323 Probabilistic Analysis Beautiful Homework # 13
BHW # /5 ENGR Probabilistic Aalysis Beautiful Homework # Three differet roads feed ito a particular freeway etrace. Suppose that durig a fixed time period, the umber of cars comig from each road oto the
More informationARIMA Models. Dan Saunders. y t = φy t 1 + ɛ t
ARIMA Models Da Sauders I will discuss models with a depedet variable y t, a potetially edogeous error term ɛ t, ad a exogeous error term η t, each with a subscript t deotig time. With just these three
More informationFinal Solutions. 1. (25pts) Define the following terms. Be as precise as you can.
Mathematics H104 A. Ogus Fall, 004 Fial Solutios 1. (5ts) Defie the followig terms. Be as recise as you ca. (a) (3ts) A ucoutable set. A ucoutable set is a set which ca ot be ut ito bijectio with a fiite
More informationSUMMARY OF SEQUENCES AND SERIES
SUMMARY OF SEQUENCES AND SERIES Importat Defiitios, Results ad Theorems for Sequeces ad Series Defiitio. A sequece {a } has a limit L ad we write lim a = L if for every ɛ > 0, there is a correspodig iteger
More informationZeros of Polynomials
Math 160 www.timetodare.com 4.5 4.6 Zeros of Polyomials I these sectios we will study polyomials algebraically. Most of our work will be cocered with fidig the solutios of polyomial equatios of ay degree
More informationDimensionfree PACBayesian bounds for the estimation of the mean of a random vector
Dimesiofree PACBayesia bouds for the estimatio of the mea of a radom vector Olivier Catoi CREST CNRS UMR 9194 Uiversité Paris Saclay olivier.catoi@esae.fr Ilaria Giulii Laboratoire de Probabilités et
More information4 Conditional Distribution Estimation
4 Coditioal Distributio Estimatio 4. Estimators Te coditioal distributio (CDF) of y i give X i = x is F (y j x) = P (y i y j X i = x) = E ( (y i y) j X i = x) : Tis is te coditioal mea of te radom variable
More information