Chapter 1. Density Estimation

Transcription

1 Capter 1 Density Estimation Let X 1, X,..., X n be observations from a density f X x. Te aim is to use only tis data to obtain an estimate ˆf X x of f X x. Properties of f f X x x, Parametric metods f X x dx 1. Assume tat f X as a known distributional type e.g. Nµ, σ. Estimate parameters using e.g. Maximum Likeliood. Tis metod works very well wen te underlying assumption of normality is correct - oterwise it can perform very badly! Tis course aims to let te data speak for temselves more tan if tey are restricted to a parametric family. We will employ less rigid assumptions, e.g., we migt only assume tat f X exists and is continuous. 1

2 1.1 Histogram To start wit, we assume tat X i [, 1; te relaxation of tis condition will be discussed later. Define bins by [j 1, j were 1/. Te following estimator is called a istogram wit bandwidt, [ ˆf X x 1 1 Bj x 1 Bj X i were Statistical Properties If x, ten ˆf n, x 1 if X i 1 Bj X i oterwise 1 Bj X i B n, fxdx were Bn, p denotes te binomial distribution wit parameters n, p. Proposition 1. Let X 1,..., X n be iid wit density f and N. Denote p j f for j 1,...,. Te Mean Integrated Squared Error MISE of te istogram ˆf n, is given by 1 1 MISE f E f ˆfn, x fx dx f xdx + 1 n + 1 Proof. Using Fubini teorem wit te bias-variance decomposition, we ave 1 1 E f ˆfn, x fx dx E f ˆfn, x fx dx E f ˆfn,x fx dx { [ E f ˆfn,x p j. } fx + Varf ˆf n, x dx.

3 Since ˆf [ n, x is a Binomial variate, it is straigtforward to verify tat E ˆfn, x np j and Var f ˆf n, x np j1 p j. Terefore, [ MISE f 1 pj fx p j 1 p j dx p j n + 1 p j. f xdx + We are interested in te beaviour of te MISE wen n and lim n n. We verify tat f xdx 1 p j 1 fx 1 fudu fx fudu dx dx. Since f is twice continuously differentiable, we ave fu fx u xf a j + O for x, u, and a j is te left border of. Terefore, f Bj xdx 1 p j f a j x udu dx + O 4. Using te cange of variable x, u a j + y, a j + z, we obtain We proved tat 1 1 x udu dx 5 y zdz dy 5 1. Bj f xdx 1 p j 3 1 f a j + O Consequently, f xdx + O 4. 3

4 MISE f 1 1 f xdx 1 p j f xdx + O O1/n. p j Teorem 1. Let X 1,..., X n be iid wit density f twice continuously differentiable and equal to zero outside [, 1. Let be te bandwidt of te istogram estimator ˆf n, suc tat n as n, ten MISE f 1 1 f xdx O1/n + O3. Teorem 1 is very useful and provides us wit an explicit form of te risk. If we know te quantity 1 f xdx, ten we can compute te risk by minimising te function 1 f xdx + 1. Te minimum would be 1 opt n 6 1 f x dx 1/3. Unfortunately, 1 f xdx is unknown in general, and we can not use tis in practice. However, it provides us wit an estimate of of order n 1/3 for n. Putting opt in te expression of te MISE, we obtain 1 1/3 MISE f 3/4 /3 f x dx n /3 + On 1. Te rate of convergence of te istogram estimator is n /3, wic is worse tan te parametric rate n 1. In kernel estimation, we will see tat te rate of convergence is n 4/5, wic is a better rate tan n / Coice of te window widt wit cross-validation We ave seen in te previous subsection tat we cannot use opt in practice because it depends on f wic is unknown. To construct a metod for coosing 4

5 independent of f, we start by estimating te risk of te estimator ˆf n, using only te observations X 1,..., X n. Let Ĵ, X 1,..., X n be an unbiased estimator of MISE f f, tat is E f Ĵ, X 1,..., X n MISE f f. Once we obtain Ĵ, we can determine te value of by minimising Ĵ, X 1,..., X n wrt >. For a density f and a istogram estimator ˆf n,, and using Proposition 1 J f MISE f f 1 n n + 1 p j. 1.1 We see from 1.1 tat te unbiased estimation of J f reduces to estimating p j. A naive approac would be to estimate p j by ˆp j, were ˆp j 1 n 1 Bj X i. Since nˆp j as a binomial distribution wit parameters n, p j, we ave Varˆp j ˆp j 1 ˆp j /n, and E f ˆp j Varˆp j + E f ˆp j p j p j n n. 1. Tis equality sows tat te naive approac to estimating p j by ˆp j leads to a biased estimator. However, we can easily prove using 1. tat, ˆp j ˆp j /n is an unbiased estimator of p j1 1/n. Terefore, is an unbiased estimator of p j. p ˆp j ˆp j /n 1 1/n n n 1 ˆp j 1 n 1 ˆp j Proposition. If f is a density, square integrable and if ˆfn, is te istogram estimator wit 1/, ten Ĵ, X 1,..., X n is an unbiased estimator of MISE f f. n 1 n ˆp j

6 We can now give te cross-validation algoritm witout assuming tat te observations are in [, 1. In tis case, we put a min i X i and b max i X i and for an integer m, coose te bandwidt b a/n. We define te bins as [a + j 1, a + j for j 1,.., m and C m [b,. Data: X 1,..., X n esult: ĥcv Define: a min i X i, b max i X i ; Initialise: m 1; m CV 1; J CV 1; wile m < n do Set J if J J CV ten else end m mn+1 n 1 n m CV m; J CV J; end m m + 1; ĥ CV b a/m CV ; m 1 n n 1 C j X i ; Algoritm 1: Cross-validation algoritm for istogram estimator. 1. ernel density estimator Te estimation of te density by a istogram approac is simple and widely used. However, te istogram estimator is not appropriate wen we ave a priori information on te smootness of te density to estimate. Specifically, if it is known beforeand tat te density of te observed sample, for example, is twice continuously differentiable, one would naturally want to estimate te density by a function wic is also twice continuously differentiable. However, istograms are functions 6

7 tat are not even continuous. It is natural ten to try to smoot istograms. Te smooting not only improves te visual appearance of te estimator, but produces a closer approximation to te true density of te istogram estimator Definitions and properties Let x and >. If we suppose tat x is te center of a bin and is its lengt, te istogram estimator is ten written f n, x 1 1 Xi x / 1 1 X i x. 1 One way to generalize te istogram is to use te formula above for all x and not only for te centers of te bins. Tis general formulation is certainly useful, because it leads to an estimator wic is piecewise constant, but as te advantage of aving bins wit varying lengts. However, tis does not lead us to a continuous estimator: te discontinuity of te estimator defined above is a consequence of te discontinuous indicator function. Terefore, replacing 1 z 1/ wit any function, we obtain te estimator f n,x 1 Xi x, wic is l-continuously differentiable if is l-continuously differentiable. We call te kernel estimator. Most popular kernels are ectangular kernel: Triangular kernel: u 1 1 [ 1,1u, u 1 u 1 [ 1,1 u, Epanecnikov kernel: u u 1 [ 1,1 u, 7

8 Gaussian kernel: u 1 π e u /. Figure 1..1 sows tese kernels. 8

9 t...4 Epanecnikov t Biweigt t..6 t t Triangular t Normal t t t..3.6 ectangular t Figure 1.1: Commonly used kernels 9

10 Lemma 1. If is nonnegative function and udu 1, ten f n,. is a probability density. Moreover, f n,. is continuous if is continuous. Proof. Te kernel estimator is te sum of nonnegative functions, terefore it is nonnegative. Also f n,xdx 1 Xi x 1 Xi x 1 udu 1. dx dx 1.. Bias and variance Wen defining a kernel estimator, we coose not only te window but also te kernel. Tere are a number of conditions tat are considered usual for te kernels and for analyzing te risk of te kernel estimator. Assumption : we assume tat satisfies 1. udu 1,. is a symmetric function, in general, uudu, 3. u u du <, 4. u <. Proposition 3. If te first tree conditions of are satisfied, and if f and its second derivative are bounded, ten Bias f n,x C 1, 1

11 were C 1 1 sup z f z u du. If, moreover, te condition 4 of is satisfied, ten were C sup z fz u du. Var f n,x C, Proof. E f [ f n,x 1 Xi x E f 1 Xi x ufx + udu. dx fydx By performing a Taylor expansion of order, we ave: E f [ f n,x u[fx + uf x + u f η u du fx udx + f x uudx + Terefore Bias f n,x Ef [ f n,x fx u uf η u du u u f η u du u u du. max z f z u uf η u du To prove te second inequality, we use tat te random variables Y i X i x 11

12 are iid. We ave, Var f n,x [ 1 Var X i x 1 [ Var X i x [ 1 nvar X 1 x [ 1 E Xi x f 1 fz u du. z }{{} C ate of convergence: From Proposition 3, we can sow tat MSE f n,x C C. Terefore te optimal is opt C /4C 1 1/5 n 1/5, and MSE f n,x Const.n 4/5. Tis sows tat te rate of convergence of te kernel estimator is n. It is better tan te rate of convergence of istograms n /3. Terefore, kernel estimators are better tan istograms wen it comes to estimating a twice continuously differentiable density Cross-validation To construct a metodology to coose te optimal, we often use te crossvalidation approac. In te first instance, we find an unbiased estimator of J MISE f f, ten we minimize tis estimator Ĵ over a set of values of. 1

13 Proposition 4. Te statistic Ĵ f n, is an unbiased estimator of J. Proof. We ave E[Ĵ E f Also, we ave nn 1 [ f n, nn 1 E f [ f n, nn 1,j i,j i,j i E f [ f n, nn 1 nn 1 E f [ f n, x y Xi X j Xi X j E f [ x y x y fxfydxdy. fxfydxdy fxfydxdy Terefore, J MISE f n, f E f [ f n, f f E f [ f n, f n,, f + f f [ E f [ f n, E f f n,xfxdx E f [ f [ n, E f f n, x fxdx. J E f [ f n, E f [ f n, E[Ĵ. [ E f f n, x fxdx y x 1 fydyfxdx Terefore, Algoritm 1 also applies ere. 13