Kernel Density Estimation

Transcription

1 EECS 598: Statistical Learning Theory, Winter 2014 Topic 19 Kernel Density Estimation Lecturer: Clayton Scott Scribe: Yun Wei, Yanzhen Deng Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. 1 Introduction Let f be a density on R d, i.e. f 0 and f(x)dx = 1. Suppose X 1, X 2,..., X n iid f. Let φ be a function s.t. φ(x)dx = 1, called a kernel, and denote φ σ (x) := 1 σ d φ( x σ ) for σ > 0. σ is called the bandwidth. The kernel density estimator (KDE) is f n (x) := 1 n φ σ (x X i ). Example. 1) Gaussian kernel: φ(x) = (2π) d 2 e x 2 2 2) There are some common kernels like triangle kernel and box kernel. See Figure 1 for their graph in one dimension. 2 L p Space For f : R R and 0 < p <, define f p = ( f(x) p dx) 1 p and L p = {f f p < }. If p 1 and we identify f and g when f g p = 0 (thus defining equivalence classes) then L p is a normed vector space, where the triangle inequality is given by Minkowski s Inequality. For a full development, see [1]. Definition 1 (Convolution). Given f, g, the convolution f g is the function f g(x) = f(y)g(x y)dy = g(y)f(x y)dy. Young s Inequality shows that the convolution of L 1 functions is still an L 1 function. Lemma 1 (Young s Inequality). If f, g L 1, then f g L 1 and f g 1 f 1 g 1. 1

2 2 Figure 1: This is a picture showing some kernels. In addition to the uniform and triangular kernels, the third example is an arbitrary kernel illustrating that a kernel need not be 1) positive, or 2) symmetric. Proof. f g 1 = = = = f g(x) dx f(y)g(x y)dy dx ( f(y)g(x y) dy)dx f(y) ( g(x y) dx)dy (By Tonelli Theorem) f(y) g 1 dy (By substitution: u = x y) = f 1 g 1. We state the next result without proof. Theorem 1 (See Folland, Thm 8.14). Let f L p, and φ L 1 with φ(x)dx = a. Then for any r > 0, f φ r L p and lim r 0 f φ r af p = 0.

3 3 3 L 2 consistency Theorem 2. Let f L 2 be a density, φ L 1 L 2 with φ(x)dx = 1. Assume X 1, X 2,..., X n iid f. If σ 0 and nσ d as n, then f n f 2 i.p. 0. Proof. By the triangle inequality, f n f 2 f n f φ σ 2 + f φ σ f 2. The second term 0 as σ 0, by Theorem 1, since f L 2, φ L 1. The first term converges i.p. to zero according to Lemma 2. Lemma 2. If f is a density, φ L 2, and X 1, X 2,..., X n iid f, then provided nσ d. Proof. Observe f n f φ σ 2 i.p. 0 Pr{ f n f φ σ 2 > ɛ} = Pr{ f n f φ σ 2 2 > ɛ 2 } E{ f n f φ σ 2 2}/ɛ 2 by Markov s Inequality. So it suffices to show E{ f n f φ σ 2 2} 0. Note E is an integral operator, and therefore by Tonelli s Theorem we can interchange the order of integration: E{ f n f φ σ 2 2} = E{( f n (x) f φ σ (x)) 2 }dx. Write f n (x) f φ σ (x) = 1 n Z i, where Z i = φ σ (x X i ) f φ σ (x). Note that Z i are iid and E(Z i ) = 0 because Eφ σ (x X i ) = φ σ (x x i )f(x i )dx i = f φ σ (x). The variance of Z i is E(Z 2 i ) = Var(φ σ (x X i )) = E{(φ σ (x X i )) 2 } [E{φ σ (x X i )}] 2 E{(φ σ (x X i )) 2 } = f(y)φ σ (x y) 2 dy = f φ 2 σ(x) = 1 σ d f (φ2 ) σ (x),

4 4 where the last step follows from the fact φ 2 σ(x) = [ 1 σ d φ( x σ )]2 = 1 σ d [ 1 σ d φ 2 ( x σ )] = 1 σ d (φ 2 ) σ. Thus E{( 1 n Z i ) 2 } = 1 n E{Z2 1} 1 nσ d f (φ2 ) σ (x) and therefore E{( f n (x) f φ σ (x)) 2 }dx 1 nσ d f (φ 2 ) σ (x)dx = 1 nσ d f (φ2 ) σ (x) 1 1 nσ d f 1 (φ 2 ) σ 1 (by Young s Inequality) = 1 nσ d φ 2 2 ( f 1 = 1) 0, since nσ d and φ L 2. Remark. (1) The condition f L 2 excludes certain densities such as where 1 2 < r < 1. (2) φ L 2 is satisfied by all common kernels. f(x) = 1 1 r x r, 0 < x < 1, (3) Recall φ need not be symmetric w.r.t. the origin. Thus, the consistency result holds for where φ(x) = 1 {x B(x0,r)}, B(x 0, r) = {x : x x 0 2 r}, r is s.t. φ(x)dx = 1, and x 0 = (0, 0,, 0, ). This may seem bizarre, but as an exercise you are asked to make sense of this example. 4 L 1 -Consistency In this section we will show that the L 1 error converges to 0 in probability. To keep things simpler, we will assume f has compact support, although this is not necessary for L 1 consistency. Theorem 3. If f is a density with compact support, φ L 1 s.t. then f i.p. n f 1 0, provided that σ 0 and nσ d 0 and n. φ(x)dx = 1, and X1, X 2,..., X n i.i.d. f, Proof. Note that f n f 1 f n f φ σ 1 + f φ σ f 1. By Theorem 1, we know that f φ σ f 1 0, so it remains to show convergence to zero of f n f φ σ 1.

5 5 Let C c = {g : R d R g is bounded and has compact support}. It is a well-known fact in analysis [1] that C c is dense in L 1. Thus for any fixed ɛ > 0, we can take ψ C c s.t. φ ψ 1 < ɛ. Denote f c n(x) = 1 n n ψ σ(x X i ). Note that By Young s Inequality, The first term is bounded by f n f φ σ 1 f n f c n 1 + f c n f ψ σ 1 + f ψ σ f φ σ 1. f ψ σ f φ σ 1 = f (ψ σ φ σ ) 1 f 1 ψ σ φ σ = ψ φ 1 < ɛ. f n f c n 1 1 n ψ σ (x X i ) φ σ (x X i ) 1 < ɛ. Since ɛ is arbitrary, we only need to prove that f n c f ψ σ 1 0 i.p. Denote by S f and S ψ the supports of f and ψ, respectively. We know that S f and S ψ are both compact sets and S ψσ is also compact and shrinks as σ 0. Thus, f n c f ψ σ 1 = f n c f ψ σ dx S f S ψσ = f n c f ψ σ dx (for σ 1) S f S ψ = f n c f ψ σ 1 Sf S ψ dx f c n f ψ σ 2 1 Sf S ψ 2. (Hölder s Inequality) The second equality holds when σ < 1, which implies S f S ψσ S f S ψ. Since ψ is in L 1 and bounded with compact support, it is also in L 2. Thus by Lemma 2, f n c f ψ σ 2 0 i.p. Now 1 Sf S ψ 2 is the square root of the volume of a compact set and thus is finite. Therefore f n c f ψ σ 1 0 i.p. Remark. The reason that we care about L 1 error is the following equality called Scheffe s Identity: if f, g are densities and B is the set of Borel sets, then: f g 1 = (f g)(x)dx (g f)(x)dx f<g = (f g)(x)dx [ (g f)(x)dx (g f)(x)dx] = 2 (f g)(x)dx = 2 sup f(x)dx g(x)dx B B B B Scheffe s Identity shows that small L 1 error leads to accurate probability estimation. 5 Strong Consistency If we add the constraint that the kernel be nonnegative, then weak L 1 consistency implies strong L 1 consistency.

6 6 Theorem 4. Assume φ 0 and φ(x)dx = 1. If X 1, X 2,..., X n i.i.d. f, then f n f 1 0 i.p. implies f n f 1 0 a.s. Proof. Let S = (X 1,..., X n ) and S i = (X 1,..., X i 1, X i, X i+1,..., X n ). Write f n = f n,s, using the new subscript to indicate the sample. Denote φ n (S) = f n,s f 1. Then φ n (S) φ n (S i) f n,s f n,s i 1 (reverse triangle inequality) = 1 φ σ (x X i ) φ σ (x X n i) dx 1 φ σ (x X i ) dx + φ σ (x X n i) dx By the bounded difference inequality, = 2. (φ nonnegative) n Pr(φ n (S) E[φ n (S)] ɛ) e nɛ2 /2. Fix ɛ > 0. By weak consistency, N s.t. n N Eφ n (S) < ɛ 2. Then for n N, This upper bound decrease geometrically. Therefore and Borel-Cantelli implies φ n (S) 0 a.s. Exercises Pr(φ n (S) ɛ) Pr(φ n (S) Eφ n (S) ɛ/2) e nɛ2 /8. Pr(φ n (S) ɛ) < n=1 1. Make sense of the third remark after the proof of L 2 consistency. 2. What does Bernstein s inequality imply about 1 n Zi in the proof of Lemma 2? Is this observation useful in any way? 3. Remove the assumption in Theorem 3 that f has compact support. References [1] Gerald B. Folland, Real Analysis: Modern Techniques and Their Applications, Wiley, 1999