Local Orthogonal Polynomial Expansion (LOrPE) for Density Estimation

Transcription

1 Local Ortogonal Polynomial Expansion (LOrPE) for Density Estimation Alex Trindade Dept. of Matematics & Statistics, Texas Tec University Igor Volobouev, Texas Tec University (Pysics Dept.) D.P. Amali Dassanayake, University of Calgary (former student) July 2015 LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 1 / 17Te

2 Outline 1 Intro via Kernel Density Estimation (KDE) 2 LOrPE construction 3 Connections wit: KDE, OSDE, LLDE 4 Selection of tuning parameters 5 Simulations 6 Real Data Example 7 Summary LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 2 / 17Te

3 Kernel Density Estimation (KDE): Simple & Effective Observe: x 1,..., x n IID from PDF f ( ) on compact support [a, b]. Start wit empirical density function (EDF): ˆf EMP (x) = 1 n were δ( ) is te Dirac delta function. n δ(x x i ) i=1 Better: If can assume PDF as first few derivatives or at most a few modes, a convolution of EDF wit a kernel function K ( ) gives a weigted average of points close to x: ˆf KDE (x) 1 K ( x y ) ˆf EMP (y)dy = 1 n n ( ) 1 x i=1 K xi LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 3 / 17Te

4 Example: Explanation of KDE Density x LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 4 / 17Te

5 KDE is Optimal, But as Problems... Teorem For kernel order r, and optimal bandwidt n, KDE s (asymptotic) MISE 0 at te fastest possible rate (for any estimator): But: O(n 2r/(2r+1) ) = O(n 4/5 ), for usual Gaussian kernel Altoug consistent for a sequence of bandwidts n 0 under mild conditions, KDE suffers from boundary bias, esp. for sarply truncated supports, e.g., Exp(1). In bounded supports KDE fails to attain tis optimal convergence rate (Jones, 1993)... LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 5 / 17Te

6 Example: KDE Boundary Bias Estimates for n=100 obs from an Exp(1) Density KDE (plug in) True PDF LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 6 / 17Te

7 Solution 1: Local and/or Adaptive Metods (1980 s on) Adaptive kernels. Have local estimates in regions wit ample data, but expand te neigborood in regions were data is more scarce. (Juggle bias-variance trade-off to reduce MISE.) Acieved by: canging bandwidts at different points, or canging kernel sapes at different points (or bot). Boundary corrections. Truncation and reflection of boundary kernels (data mirroring); optimal weigting scemes; data transforms; etc. Local polynomial/likeliood. Local Likeliood Density Estimation (LLDE) models PDF locally via a polynomial; estimation via ML (Hjort & Jones, 1996; Loader, 1996, 1999). Ortogonal polynomial expansions. Ortogonal Series Density Estimation (OSDE) (Čencov, 1962; Tarter & Lock, 1993; Efromovic, 1999). LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 7 / 17Te

8 Solution 2: LOrPE (under review... ) Construct truncated ortogonal polynomial series expansion for EDF near eac grid point x fit. Like KDE, we need to coose some tings: K ( ), te kernel;, te bandwidt; M, te polynomial order. (An extra tuning parameter... ) And like KDE, K ( ) is less important tan & M. LOrPE estimate: f LOrPE (x) = M ( ) x x fit c k (x fit, )P k k=0 LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 8 / 17Te

9 LOrPE Construction f LOrPE (x) = M ( ) x x fit c k (x fit, )P k k=0 normalize ˆf LOrPE (x) Polynomials P k ( ) satisfy normalization condition: 1 b ( ) ( ) ( ) x x fit x x fit x x fit P j P k K dx = δ jk a Coefficients c k ( ) satisfy integral equation: c k (x fit, ) = 1 f (x)p k ((x x fit )/)K ((x x fit )/)dx Subs f (x) = ˆf EMP (x) in above determines coefficients: c k (x fit, ) = 1 n n P k ((x i x fit )/)K ((x i x fit )/) i=1 LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July Igor 2015Volobouev, 9 / 17Te

10 Connections Wit KDE Teorem (Teorem 1) LOrPE is linear combo of KDEs wit varying kernels K k (z) P k (z)k (z): f LOrPE (x) = Teorem (Teorem 2) M k=0 ( ) x x fit ˆf KDE (x, K k )P k Wen evaluated away from support boundaries, LOrPE is equivalent to KDE wit a ig-order (r M + 1) effective kernel: K eff (x) = M P k (0)P k ( x)k ( x) k=0 LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 10 / 17Te

11 Example: LOrPE K eff (x x fit ) Plots (a) (b) (c) Effective Kernel xfit = 0.0 M = 0 M = 2 M = 4 Effective Kernel xfit = 0.1 M = 0 M = 2 M = 4 Effective Kernel xfit = 0.5 M = 0 M = 2 M = x xfit x xfit x xfit PDF support is [0, 1] wit sarp truncation at 0, and grid points are: (a) exactly at boundary (x fit = 0) (b) close to boundary (x fit = 0.1) (c) away from boundary (x fit = 0.5) LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 11 / 17Te

12 Connections Wit OSDE For PDF supported on [a, b] wit f (x) 2 dx <, ten for ortonormal basis {φ k } classical OSDE is: ˆf OSDE (x) = J ˆθ j φ j (x), j=0 ˆθ j = 1 n n φ j (x i ) i=1 Tus LOrPE is localized version of OSDE (basis functions {P k } are not global, but adjust locally depending on x fit ). Also: Teorem (Teorem 3) As (fixed M), LOrPE reduces to OSDE wit (scaled & sifted) ortonormal Legendre polynomials on [ 1, 1] as basis functions. LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 12 / 17Te

13 Connections Wit LLDE (euristic... ) LLDE: overcomes boundary bias by matcing localized sample moments to population moments using log-polynomial PDF approx. LOrPE: matces localized moments of ortogonal polynomials to teir sample values using polynomial PDF approx. Comparison. LLDE: may be teoretically superior, but involves solving non-linear equations at every grid point. LOrPE: enjoys pragmatic advantage of computational speed & numerical stability (is linear smooter of EDF). LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 13 / 17Te

14 Selection of Tuning Parameters Plug-in approac: similar to Silverman s Rule for KDE. Cross-validation metods. Coose & M to minimize: least squares cross-validation (LSCV) LSCV (, M) = f LOrPE (x, M) 2 dx 2 n regularized (α = 0.5) likeliood cross-validation (RLCV) RLCV (, M) = { n max i=1 f ( i) (+i) LOrPE (x f i, M), n f ( i) LOrPE (x i, M), i=1 LOrPE (x i, M) n α } LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 14 / 17Te

15 Revisit Exp(1) Example Improved performance at boundary confirmed by extensive simulations; o/w competitive wit (KDE, OSDE, LLDE). Density KDE (plug in) True PDF LOrPE (LSCV) LOrPE (RLCV) LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 15 / 17Te

16 Real Data Example: n = 86 Patients (Silverman, 1986) Density KDE (plug in) Cen (1999, modified kernel) LLDE (minimum AIC) OSDE (Hart sceme) LOrPE LSCV LOrPE RLCV (α=0.5) LOrPE Lengt ( Dept. for Density oftreatment Matematics Estimation (days) & Statistics, Texas Tec University July 2015 Igor Volobouev, 16 / 17Te

17 Summary Away from boundary LOrPE acts like KDE wit a ig-order kernel. Faster asymptotic convergence rates! Close to boundary LOrPE is adaptive: effective kernels naturally cange sape to accommodate endpoint; reduces boundary bias. For fixed M, LOrPE inerits consistency from: KDE, for 0 (away from boundary); OSDE, for. LOrPE extends to multivariate settings, need only switc to multivariate ortogonal polynomial systems. Tank You! [ttp://arxiv.org/abs/ ] LOrPE ( Dept. for Density of Matematics Estimation & Statistics, Texas Tec University July 2015 Igor Volobouev, 17 / 17Te