Nonparametric estimation of log-concave densities

Transcription

1 Nonparametric estimation of log-concave densities Jon A. Wellner University of Washington, Seattle Seminaire, Institut de Mathématiques de Toulouse 5 March 2012

2 Seminaire, Toulouse Based on joint work with: Fadoua Balabdaoui Kaspar Rufibach Arseni Seregin

3 Outline A: Log-concave densities on R 1 B: C: D: E: F: Nonparametric estimation, log-concave on R Limit theory at a fixed point in R Estimation of the mode, log-concave density on R Generalizations: s concave densities on R and R d Summary; problems and open questions Seminar, Institut de Mathématiques de Toulouse; 5 March

4 Suppose that A. Log-concave densities on R 1 f(x) f ϕ (x) = exp(ϕ(x)) = exp ( ( ϕ(x))) where ϕ is concave (and ϕ is convex). The class of all densities f on R of this form is called the class of log-concave densities, P log concave P 0. Properties of log-concave densities: A density f on R is log-concave if and only if its convolution with any unimodal density is again unimodal (Ibragimov, 1956). Every log-concave density f is unimodal (but need not be symmetric). P 0 is closed under convolution. Seminar, Institut de Mathématiques de Toulouse; 5 March

5 A. Log-concave densities on R 1 Many parametric families are log-concave, for example: Normal (µ, σ 2 ) Uniform(a, b) Gamma(r, λ) for r 1 Beta(a, b) for a, b 1 t r densities with r > 0 are not log-concave Tails of log-concave densities are necessarily sub-exponential P log concave = the class of Polyá frequency functions of order 2, P F 2, in the terminology of Schoenberg (1951) and Karlin (1968). See Marshall and Olkin (1979), chapter 18, and Dharmadhikari and Joag-Dev (1988), page 150. for nice introductions. Seminar, Institut de Mathématiques de Toulouse; 5 March

6 A. Log-concave densities on R 1 Thus: log-concave = P F 2 = strongly uni-modal Seminar, Institut de Mathématiques de Toulouse; 5 March

7 B. Nonparametric estimation, log-concave on R The (nonparametric) MLE f n exists (Rufibach, Dümbgen and Rufibach). f n can be computed: and Rufibach) R-package logcondens (Dümbgen In contrast, the (nonparametric) MLE for the class of unimodal densities on R 1 does not exist. Birgé (1997) and Bickel and Fan (1996) consider alternatives to maximum likelihood for the class of unimodal densities. Consistency and rates of convergence for f n : Dümbgen and Rufibach, (2009); Pal, Woodroofe and Meyer (2007). Pointwise limit theory? Yes! Balabdaoui, Rufibach, and W (2009). Seminar, Institut de Mathématiques de Toulouse; 5 March

8 B. Nonparametric estimation, log-concave on R MLE of f and ϕ: Let C denote the class of all concave function ϕ : R [, ). The estimator ϕ n based on X 1,..., X n i.i.d. as f 0 is the maximizer of the adjusted criterion function over ϕ C. l n (ϕ) = = logf ϕ (x)df n (x) ϕ(x)df n (x) e ϕ(x) dx f ϕ (x)dx Properties of f n, ϕ n : (Dümbgen & Rufibach, 2009) ϕ n is piecewise linear. ϕ n = on R \ [X (1), X (n) ]. The knots (or kinks) of ϕ n occur at a subset of the order statistics X (1) < X (2) < < X (n). Characterized by... Seminar, Institut de Mathématiques de Toulouse; 5 March

9 B. Nonparametric estimation, log-concave on R... ϕ n is the MLE of logf 0 = ϕ 0 if and only if { Hn (x), for all x > X Ĥ n (x) (1), = H n (x), if x is a knot. where F n (x) = x X (1) f n (y)dy, Ĥn(x) = x x X (1) F n (y)dy, H n (x) = F n(y)dy. Furthermore, for every function such that ϕ n + t is concave for t small enough, (x)df n(x) (x)d F n (x). R R Seminar, Institut de Mathématiques de Toulouse; 5 March

10 B. Nonparametric estimation, log-concave on R Consistency of f n and ϕ n : (Pal, Woodroofe, & Meyer, 2007): If f 0 P 0, then H( f n, f 0 ) a.s. 0. (Dümbgen & Rufibach, 2009): If f 0 P 0 and ϕ 0 H β,l (T ) for some compact T = [A, B] {x : f 0 (x) > 0}, M <, and 1 β 2. Then sup( ϕ n (t) ϕ 0 (t)) = O p t T sup t T n (ϕ 0 (t) ϕ n (t)) = O p ( logn n ( logn n ) β/(2β+1), ) β/(2β+1) and where T n [A + (logn/n) β/(2β+1), B (logn/n) β/(2β+1) ] and β/(2β + 1) [1/3, 2/5] for 1 β 2. The same remains true if ϕ n, ϕ 0 are replaced by f n, f 0. Seminar, Institut de Mathématiques de Toulouse; 5 March

11 B. Nonparametric estimation, log-concave on R If ϕ 0 H β,l (T ) as above and, with ϕ 0 = ϕ 0( ) or ϕ 0 ( +), ϕ 0 (x) ϕ 0 (y) C(y x) for some C > 0 and all A x < y B, then sup t T n F n (t) F n (t) = O p ( logn n ) 3β/(4β+2) where 3β/(2β + 4) [1/2, 3/5] = [.5,.6] for 1 β 2.. If β > 1, this implies sup t Tn F n (t) F n (t) = o p (n 1/2 ). Seminar, Institut de Mathématiques de Toulouse; 5 March

12 B. Nonparametric estimation, log-concave on R density functions density functions log density functions log density functions CDFs CDFs hazard functions hazard functions Seminar, Institut de Mathématiques de Toulouse; 5 March

13 B. Nonparametric estimation, log-concave on R Seminar, Institut de Mathématiques de Toulouse; 5 March

14 B. Nonparametric estimation, log-concave on R Seminar, Institut de Mathématiques de Toulouse; 5 March

15 B. Nonparametric estimation, log-concave on R Seminar, Institut de Mathématiques de Toulouse; 5 March

16 C: Limit theory at a fixed point in R Assumptions: f 0 is log-concave, f 0 (x 0 ) > 0. If ϕ 0 (x 0) 0, then k = 2; otherwise, k is the smallest integer such that ϕ (j) 0 (x 0) = 0, j = 2,..., k 1, ϕ (k) 0 (x 0) 0. ϕ (k) 0 is continuous in a neighborhood of x 0. Example: f 0 (x) = Cexp( x 4 ) with C = 2Γ(3/4)/π: k = 4. Driving process: Y k (t) = t 0 W (s)ds t k+2, W standard 2-sided Brownian motion. Invelope process: H k determined by limit Fenchel relations: H k (t) Y k (t) for all t R R (H k(t) Y k (t))dh (3) k (t) = 0. H (2) k is concave. Seminar, Institut de Mathématiques de Toulouse; 5 March

17 C: Limit theory at a fixed point in R Theorem. (Balabdaoui, Rufibach, & W, 2009) Pointwise limit theorem for f n (x 0 ): ( n k/(2k+1) ( f n (x 0 ) f 0 (x 0 )) n (k 1)/(2k+1) ( f n(x 0 ) f 0 (x 0)) where c k d k ) d f 0(x 0 ) k+1 ϕ (k) 0 (x 0) (k + 2)! c kh (2) k (0) d k H (3) k (0) 1/(2k+1) f 0(x 0 ) k+2 ϕ (k) 0 (x 0) 3 [(k + 2)!] 3, 1/(2k+1). Seminar, Institut de Mathématiques de Toulouse; 5 March

18 C: Limit theory at a fixed point in R Pointwise limit theorem for ϕ n (x 0 ): ( where n k/(2k+1) ( ϕ n (x 0 ) ϕ 0 (x 0 )) n (k 1)/(2k+1) ( ϕ n(x 0 ) ϕ 0 (x 0)) C k D k ϕ (k) 0 (x 0) f 0 (x 0 ) k (k + 2)! ) d 1/(2k+1) ϕ (k) 0 (x 0) 3 f 0 (x 0 ) k 1 [(k + 2)!] 3 C kh (2) k (0) D k H (3) k (0), 1/(2k+1). Proof: Use the same perturbation as for convex - decreasing density proof with perturbation version of characterization: Seminar, Institut de Mathématiques de Toulouse; 5 March

19 C: Limit theory at a fixed point in R Τ Τ Seminar, Institut de Mathématiques de Toulouse; 5 March

20 D: Mode estimation, log-concave density on R Let x 0 = M(f 0 ) be the mode of the log-concave density f 0, recalling that P 0 P unimodal. Lower bound calculations using Jongbloed s perturbation ϕ ɛ of ϕ 0 yields: Proposition. If f 0 P 0 satisfies f 0 (x 0 ) > 0, f 0 (x 0) < 0, and f 0 is continuous in a neighborhood of x 0, and T n is any estimator of the mode x 0 M(f 0 ), then f n exp(ϕ ɛn ) with ɛ n νn 1/5 and ν 2f 0 (x 0) 2 /(5f 0 (x 0 )), lim inf n n1/5 inf max {E n T n M(f n ), E 0 T n M(f 0 ) } T n ( ) 1/5 ( 5/2 f0 (x 0 ) ) 1/ e f 0 (x 0) 2 Does the MLE M( f n ) achieve this? Seminar, Institut de Mathématiques de Toulouse; 5 March

21 D: Mode estimation, log-concave density on R Seminar, Institut de Mathématiques de Toulouse; 5 March

22 D: Mode estimation, log-concave density on R Proposition. (Balabdaoui, Rufibach, & W, 2009) Suppose that f 0 P 0 satisfies: ϕ (j) 0 (x 0) = 0, j = 2,..., k 1, ϕ (k) 0 (x 0) 0, and ϕ (k) 0 is continuous in a neighborhood of x 0. Then M n M( f n ) min{u : f n (u) = sup t f n (t)}, satisfies n 1/(2k+1) ( M n M(f 0 )) d where M(H (2) k ) = argmax(h (2) k ). ((k + 2)!)2 f 0 (x 0 ) f (k) 0 (x 0) 2 1/(2k+1) M(H (2) k ) Note that when k = 2 this agrees with the lower bound calculation, at least up to absolute constants. Seminar, Institut de Mathématiques de Toulouse; 5 March

23 D: Mode estimation, log-concave density on R f 2 8 f f Seminar, Institut de Mathématiques de Toulouse; 5 March

24 D: Mode estimation, log-concave density on R 1.5 H H 1 Y Seminar, Institut de Mathématiques de Toulouse; 5 March

25 D: Mode estimation, log-concave density on R 4 F 1 F X Seminar, Institut de Mathématiques de Toulouse; 5 March

26 D: Mode estimation, log-concave density on R f 2 8 f f Seminar, Institut de Mathématiques de Toulouse; 5 March

27 D: Mode estimation, log-concave density on R f 0 f f Seminar, Institut de Mathématiques de Toulouse; 5 March

28 D: Mode estimation, log-concave density on R When f 0 = φ, the standard normal density, M(f 0 ) = 0, f 0 (0) = (2π) 1/2, f 0 (0) = (2π) 1/2, and hence ((4)!)2 f 0 (0) f (2) 0 (x 0) 2 1/5 = ( 24 2 (2π) 1/2 (2π) 1 )1/5 = Seminar, Institut de Mathématiques de Toulouse; 5 March

29 E: Generalizations of log-concave to R and R d : Three generalizations: log concave densities on R d (Cule, Samworth, and Stewart, 2010) s concave and h transformed convex densities on R d (Seregin, 2010) Hyperbolically k monotone and completely monotone densities on R; (Bondesson, 1981, 1992) Seminar, Institut de Mathématiques de Toulouse; 5 March

30 E: Generalizations of log-concave to R and R d : Log-concave densities on R d : A density f on R d is log-concave if f(x) = exp(ϕ(x)) with ϕ concave. Some properties: Any log concave f is unimodal The level sets of f are closed convex sets Convolutions of log-concave distributions are log-concave. Marginals of log-concave distributions are log-concave. Seminar, Institut de Mathématiques de Toulouse; 5 March

31 E: Generalizations of log-concave to R and R d : MLE of f P 0 (R d ): (Cule, Samworth, Stewart, 2010) MLE f n = argmax f P0 (R d ) P nlogf exists and is unique if n d + 1. The estimator ϕ n of ϕ 0 is a taut tent stretched over tent poles of certain heights at a subset of the observations. Computable via non-differentiable convex optimization methods: Shor s (1985) r algorithm: R package LogConcDEAD (Cule, Samworth, Stewart, 2008). Seminar, Institut de Mathématiques de Toulouse; 5 March

32 E: Generalizations of log-concave to R and R d : Seminar, Institut de Mathématiques de Toulouse; 5 March

33 E: Generalizations of log-concave to R and R d : If f 0 is any density on R d with R d x f 0 (x)dx <, R d f 0 (x)logf 0 (x)dx <, and {x R d : f 0 (x) > 0} = int(supp(f 0 )), then f n satisfies: f n (x) f (x) dx a.s. 0 R d where, for the Kullback-Leibler divergence K(f 0, f) = f 0 log(f 0 /f)dµ, f = argmin f P0 (R d ) K(f 0, f) is the pseudo-true density in P 0 (R d ) corresponding to f 0. In fact: R d ea x f n (x) f (x) dx a.s. 0 for any a < a 0 where f (x) exp( a 0 x + b 0 ). Seminar, Institut de Mathématiques de Toulouse; 5 March

34 E: Generalizations of log-concave to R and R d : r concave and h transformed convex densities on R d : (Seregin, 2010; Seregin & W, 2010) Generalization to s concave densities: A density f on R d is r concave on C R d if f(λx + (1 λ)y) M r (f(x), f(y); λ) for all x, y C and 0 < λ < 1 where M r (a, b; λ) = ((1 λ)a r + λb r ) 1/r, r 0, a, b > 0, 0, r < 0, ab = 0 a 1 λ b λ, r = 0. Let P r denote the class of all r concave densities on C. For r 0 it suffices to consider C = R d, and it is almost immediate from the definitions that if f P r for some r 0, then { g(x) f(x) = 1/r }, r < 0 for g convex. exp( g(x)), r = 0 Seminar, Institut de Mathématiques de Toulouse; 5 March

35 E: Generalizations of log-concave to R and R d : Long history: Avriel (1972), Prékopa (1973), Borell (1975), Rinott (1976), Brascamp and Lieb (1976) Nice connections to t concave measures: (Borell, 1975) Known now in math-analysis as the Borell, Brascamp, Lieb inequality One way to get heavier tails than log-concave! Example: Multivariate t density with p degrees of freedom: if f(x) = f(x; p, d) = Γ((d + p)/2) 1 Γ(p/2)(pπ) d/2 ( 1 + x 2 p ) (d+p)/2 then f P 1/s for s (d, d + p]; i.e. f P r (R d ) for 1/(d + p) r < 1/d. Seminar, Institut de Mathématiques de Toulouse; 5 March

36 E: Generalizations of log-concave to R and R d : A measure µ on (R, B) is called t concave if for all A, B B and 0 λ 1 µ(λa + (1 λ)b) M t (µ(a), µ(b), λ). Theorem. (Borell, 1975) If f P r with 1/d r, then the measure P = P f defined by P (A) = A f(x)dx for Borel subsets A of R d is t concave with and conversely. t = r 1+dr, if 1/d < r <,, if r = 1/d, 1/d, if r =, Seminar, Institut de Mathématiques de Toulouse; 5 March

37 E: Generalizations of log-concave to R and R d : h convex densities: Seregin (2010), Seregin & W (2010)) f(x) = h(ϕ(x)) (1) where ϕ : R d R is convex, h : R R + is decreasing and continuous; e.g. h s (u) (1 + u/s) s with s > d. This motivates the following definition: Definition. Say that h : R R + is a decreasing transformation if, with y 0 sup{y : h(y) > 0}, y inf{y : h(y) < }, h(y) = o(y α ) for some α > d as y. If y >, then h(y) (y y ) β y y. for some β > d as If y =, then h(y) γ h( Cy) = o(1) as y for some γ, C > 0. h is continuously differentiable on (y, y 0 ). Seminar, Institut de Mathématiques de Toulouse; 5 March

38 E: Generalizations of log-concave to R and R d : Seminar, Institut de Mathématiques de Toulouse; 5 March

39 E: Generalizations of log-concave to R and R d : Let P h denote the collection of all densities on R d of the form f = h ϕ for a fixed decreasing transformation h and ϕ convex, and let f n argmax f Ph P n logf, the MLE. Theorem. f n P h exists if n n d where n d d + dγ1{y = } + βd2 α(β d) 1{y > } { d + 1, if h(y) = e y, = ), if h(y) = y s, s > d. d ( s s d Theorem. If h is a decreasing transformation as defined above, and f 0 P h, then H( f n, f 0 ) a.s. 0. Seminar, Institut de Mathématiques de Toulouse; 5 March

40 E: Generalizations of log-concave to R and R d : Questions: Rates of convergence? Limiting distribution(s) for d > 1? (n r with r = 2/(4 + d)?) MLE (rate-) inefficient for d 4? efficient rates? How to penalize to get Multivariate classes with nice preservation/closure properties and smoother than log-concave? Can we treat f n P h with miss-specification: f 0 / P h? Algorithms for computing f n P h? Related results for convex regression on R d : Seijo and Sen, Ann. Statist. (2011). Seminar, Institut de Mathématiques de Toulouse; 5 March

41 Selected references: Balabdaoui, F., Rufibach, K., and Wellner, J.A. (2009). Limit distribution theory for maximum likelihood estimation of a log-concave density. Ann. Statist. 37, Borell, C. (1975). Convex set functions in d space. Period. Math. Hungar. 6, Dümbgen, L. and Rufibach, K. (2009). Maximum likelihood estimation of a log-concave density. Bernoulli 15, Groeneboom, P., Jongbloed, G., and Wellner, J. A. (2001a). Estimation of a convex function: characterizations and asymptotic theory. Ann. Statist. 29, Groeneboom, P., Jongbloed, G., and Wellner, J. A. (2001b). The invelope of integrate Brownian motion +t 4. Ann. Statist. 29, Pal, J.K., Woodroofe, M.B., and Meyer, M.C. (2007). Estimating a Pólya frequency function. In Complex Datasets and Inverse Problems: Tomography, Networks, and Beyond. IMS Lecture Notes - Monograph Series Seijo, E. and Sen, B. (2011). Nonparametric least squares estimation of a multivariate convex regression function. Ann. Statist. 39, Seminar, Institut de Mathématiques de Toulouse; 5 March

42 Selected references: Seregin, A. and Wellner, J. A. (2010). Nonparametric estimation of multivariate convex-transformed densities. Ann. Statist. 38, Seminar, Institut de Mathématiques de Toulouse; 5 March

43 Merci! Seminar, Institut de Mathématiques de Toulouse; 5 March

44 Seminar, Institut de Mathématiques de Toulouse; 5 March