Geometric Inference on Kernel Density Estimates

Transcription

1 Geometric Inference on Kernel Density Estimates Bei Wang 1 Jeff M. Phillips 2 Yan Zheng 2 1 Scientific Computing and Imaging Institute, University of Utah 2 School of Computing, University of Utah Feb 6, 2014

2 Take home message Geometric inference from a point cloud can be calculated by examining its kernel density estimate (KDE) of Gaussians. Such an inference is made possible with provable properties through the vehicle of kernel distance. Such an inference is robust to noise and scalable. We provide an algorithm to estimate the topology of kernel distance using weighted Vietoris-Rips complexes.

3 A bit more detail... Geometric inference using the kernel distance, in place of the distance to a measure [Chazal Cohen-Steiner Merigot 2011]. 1. [Robustness] Kernel distance is distance-like: 1-Lipschitz, 1-semiconcave, proper and stable. 2. [Scalability] Kernel distance has a small coreset, making efficient inference possible on 100 million points. 3. [Relation to KDE] Geometric inference based on kernel distance works naturally via superlevel sets of KDE: sublevel sets of the kernel distance are superlevel sets of KDE. 4. [Algorithm] to approximate the sublevel set filtration of kernel distance from a point cloud sample.

4 Why kernel distance? People love and are familiar with KDE, especially with Gaussian kernel Kernel distance provides a proper way to relate KDE with properties that are crucial for geometric inference We could approximate the topology of kernel distance via point cloud samples

5 Background on kernels, KDE and kernel distance

6 Kernels A kernel is a similarity measure, more similar points have higher value, K : R d R d R + We focus on the Gaussian kernel (positive definite):. K(p, x) = σ 2 exp( p x 2 /2σ 2 ) x x x Gaussian G Triangle T Ball B

7 Kernel density estimate (KDE) A kernel density estimate represents a continuous distribution function over R d for point set P R d : kde P (x) = 1 P K(p, x) p P More generally, it can be applied to any measure µ (on R d ) as kde µ (x) = K(p, x)µ(p)dp p R d

8 Kernel distance For two point sets P and Q, define similarity κ(p, Q) = 1 1 K(p, q) P Q If Q = {x}, κ(p, x) = kde P (x). p P q Q The kernel distance (a metric between P and Q): D K (P, Q) = κ(p, P ) + κ(q, Q) 2κ(P, Q) Self similarity minus cross similarity... [Phillips, Venkatasubramanian 2011]

9 Kernel distance For two point sets P and Q, define similarity κ(p, Q) = 1 1 K(p, q) P Q If Q = {x}, κ(p, x) = kde P (x). p P q Q The kernel distance (a metric between P and Q): D K (P, Q) = κ(p, P ) + κ(q, Q) 2κ(P, Q) Self similarity minus cross similarity... [Phillips, Venkatasubramanian 2011]

10 Kernel distance For two point sets P and Q, define similarity κ(p, Q) = 1 1 K(p, q) P Q If Q = {x}, κ(p, x) = kde P (x). p P q Q The kernel distance (a metric between P and Q): D K (P, Q) = κ(p, P ) + κ(q, Q) 2κ(P, Q) Self similarity minus cross similarity... [Phillips, Venkatasubramanian 2011]

11 Kernel distance For D K (µ, ν) between two measures µ and ν, define similarity κ(µ, ν) = K(p, q)µ(p)µ(q)dpdq p R d q R d The kernel distance (a metric between µ and ν): D K (µ, ν) = κ(µ, µ) + κ(ν, ν) 2κ(µ, ν) If ν = unit Dirac mass at x, κ(µ, x) = kde µ (x), D K (µ, x) = κ(µ, µ) + κ(x, x) 2κ(µ, x) = c µ 2kde µ (x) Kernel distance (current distance or maximum mean discrepancy) is a metric, if the kernel K is characteristic (a slight restriction of being positive definite, e.g. Gaussian and Laplace kernels).

12 Geometric inference and distance to a measure: a review [Lieutier 2004] [Chazal, Cohen-Steiner, Lieutier 2009] [Merigot, Ovsjanikov, Guibas 2009] [Chazal, Cohen-Steiner, Merigot 2010] [Biau, Chazal, Cohen-Steiner, Devroye and Rodriguez 2011] [Chazal Cohen-Steiner Merigot 2011]...

13 Geometric inference Given: An unknown object (e.g. a compact set) S R d A finite point cloud P R d that comes from S under some process Aim: Recover topological and geometric properties of S from P, e.g. # of components, dimension, curvature... e.g. preserve homeomorphism, homotopy type, or homology of S from P.

14 Distance function based geometric inference Reconstructs an approximation of S by offsets from P. [Chazal, Cohen-Steiner, Lieutier 2009] Distance function: f P (x) = inf y P x y Offset: (P ) r = f 1 P ([0, r]) Hausdorff distance: d H (S, P ) := f S f P = inf x R d f S (x) f P (x)

15 Distance function based geometric inference: the intuition [Hausdorff stability w.r.t. distance functions] If d H (S, P ) is small, thus f S and f P are close, and subsequently, S, (S) r and (P ) r carry the same topology for an appropriate scale r. Theorem (Reconstruction from f P ) Let S, P R d be compact sets such that reach(s) > R and ε := d H (S, P ) R/7. Then S and (P ) r are homotopy equivalent (and even isotopic) if 4ε r R 3ε. [Chazal Cohen-Steiner Lieutier 2009] [Chazal Cohen-Steiner Merigot 2011] R ensures topological properties of S and (S) r are the same; ε ensures (S) r and (P ) r are close, ε density of the sample.

16 Distance function based geometric inference Not robust to outliers. [Chazal Cohen-Steiner Merigot 2011] If S = S x and f S (x) > R, then f S f S > R : offset-based inference methods fail...

17 Distance(-like) function that is robust to noise... Desirable properties for f S to be useful in geometric inference: (F1) f S is 1-Lipschitz: for all x, y R d, f S (x) f S (y) x y. (F2) f 2 S is 1-semiconcave: x Rd (f S (x)) 2 x 2 is concave. (F1) ensures that f S is differentiable almost everywhere and the medial axis of S has zero d-volume; (F2) is crucial, e.g. in proving the existence of the flow of the gradient of the distance function for topological inference.

18 Distance to a measure [Chazal Cohen-Steiner Merigot 2011] Intuition: W 2 distance to m 0 fraction of the space. µ: probability measure on R d m 0 > 0: a parameter smaller than the total mass of µ The distance to a measure d CCM µ,m 0 : R n R +, x R d, ( 1 m0 ) 1/2 d CCM µ,m 0 (x) = (δ µ,m (x)) 2 dm m 0 m=0 where δ µ,m (x) = inf { r > 0 : µ( B r (x)) m }. ( ) Wasserstein-2 distance W 2 (µ, ν) = inf π Π(µ,ν) R d R d x y 2 1/2 dπ(x, y)

19 Distance to a measure d CCM µ,m 0 is distance-like (D1) 1-Lipschitz (D2) 1-semiconcave (D3) [Stability] For probability measures µ and ν on R d and m 0 > 0, then d CCM µ,m 0 d CCM ν,m 0 1 m0 W 2 (µ, ν). (D4) Proper (for Groves Isotopy Lemma). [Chazal Cohen-Steiner Merigot 2011]

20 Our Main Results

21 Our results Similar properties hold for the kernel distance defined as d K µ (x) = D K (µ, x) = κ(µ, µ) + κ(x, x) 2κ(µ, x) = c 2 µ 2kde µ (x) Specifically, the following properties of d K µ reconstruction properties of d CCM µ,m 0. allow it to inherit the (K1) d K µ is 1-Lipschitz. (K2) (d K µ ) 2 is 1-semiconvave: the map x (d K µ (x)) 2 x 2 is concave. (K3) d K µ is proper. (K4) [Stability] d K µ d K ν D K (µ, ν). For the point cloud setting, d K P (x) = D K (P, x) = κ(p, P ) + κ(x, x) 2κ(P, x) = c 2 P 2kde P (x)

22 Properness of d K µ A continuous map f : X Y between two topological spaces is proper if and only if the inverse image of every compact subset in Y is compact in X. Lemma (K3) d K µ is proper (when its range is restricted to be less than c µ ). Corollary The superlevel sets of kde µ for all ranges whose lower bound a > 0 are compact.

23 Reconstruction theory for d K µ Theorem (Isotopy lemma on d K µ ) Let r 1 < r 2 be two positive numbers such that d K µ has no critical points in (d K µ ) [r 1,r 2 ]. Then all the sublevel sets (d K µ ) r := {x R d d K µ (x) r} are isotopic for r [r 1, r 2 ]. Theorem (Reconstruction on d K µ, simple version) Let d K µ and d K ν be two kernel distance functions such that d K µ d K ν ε. Suppose reach(d K µ ) R. Then r [4ε, R 3ε], and η (0, R), the sublevel sets (d K µ ) η and (d K ν ) r are homotopy equivalent for ε R/9.

24 Stability properties for d K µ : Basic result For two probability measures µ and ν on R d, Lemma (K4) d K µ d K ν D K (µ, ν). Define µ = p p µ(p)dp. We have lim σ D K(µ, ν) = µ ν and µ ν W 2 (µ, ν). Theorem lim σ D K (µ, ν) W 2 (µ, ν). Open problem: remove need for limit in σ.

25 Stability properties for d K µ : Comparing D K to W 2 Lemma (One-sided bound) There is no Lipschitz constant γ s.t. for any µ and ν, we have W 2 (µ, ν) γd K (µ, ν). Intuition, W 2 can grow arbitrarily big, there are some measures from which our bound is tighter. Lemma (Special case) Consider µ and ν where ν is represented by a Dirac mass at a point x R d. Then d K µ (x) = D K (µ, ν) W 2 (µ, ν) for any σ > 0, where the equality only holds when µ is also a Dirac mass at x, and the difference decreases for any other x as σ increases.

26 Stability properties for d K µ : Stability with Respect to σ Theorem For any x, d K µ (x) is l-lipschitz with respect to σ, for l = 18/e 3 + 8/e + 2 < 6. σ geometric notion of an outlier parameter

27 Algorithm: Approximate the persistence diagram of sublevel sets filtration of kernel distance using weighted Rips filtration

28 Power distance construction [Buchet, Chazal, Oudot and Sheehy 2013] Metric space (X, d X (, )), a set P X and a function w : P R, the (general) power distance f associated with (P, w) is f(x) = min (d X(p, x) 2 + w(p) 2 ). p P Persistence diagram of w can be approximated by weight Rips filtration based on (P, w) and f.

29 Power distance construction [Buchet, Chazal, Oudot and Sheehy 2013] Sublevel set of f, f 1 ((, α]) is the union of balls centered at points p P with radius r p (α) = α 2 w(p) 2 for each p.

30 Power distance construction [Buchet, Chazal, Oudot and Sheehy 2013] Sublevel set of f, f 1 ((, α]) is the union of balls centered at points p P with radius r p (α) = α 2 w(p) 2 for each p. Weighted Čech complex C α (P, w) for parameter α is the union of simplices s such that p s B(p, r p(α)) 0.

31 Power distance construction [Buchet, Chazal, Oudot and Sheehy 2013] Sublevel set of f, f 1 ((, α]) is the union of balls centered at points p P with radius r p (α) = α 2 w(p) 2 for each p. Weighted Čech complex C α (P, w) for parameter α is the union of simplices s such that p s B(p, r p(α)) 0. Weighted Rips complex R α (P, w) for parameter α is the maximal complex whose 1-skeleton is the same as C α (P, w).

32 Power distance construction [Buchet, Chazal, Oudot and Sheehy 2013] Sublevel set of f, f 1 ((, α]) is the union of balls centered at points p P with radius r p (α) = α 2 w(p) 2 for each p. Weighted Čech complex C α (P, w) for parameter α is the union of simplices s such that p s B(p, r p(α)) 0. Weighted Rips complex R α (P, w) for parameter α is the maximal complex whose 1-skeleton is the same as C α (P, w). Weighted Rips filtration: {R α (P, w)}.

33 Power distance using d K µ A power distance using d K µ for a measure µ is defined with a point set P R d and a metric d(, ) on R d, ( f P (µ, x) = d(p, x) 2 + d K µ (p) 2). min p P

34 Power distance using d K µ A power distance using d K µ for a measure µ is defined with a point set P R d and a metric d(, ) on R d, ( f P (µ, x) = d(p, x) 2 + d K µ (p) 2). min p P We consider d(p, x) := D K (p, x). Let p + = arg max q R d κ(µ, q) and P + = P p +. We have 1 2 d K µ (x) f k P + (µ, x) 14d K µ (x).

35 Power distance using d K µ A power distance using d K µ for a measure µ is defined with a point set P R d and a metric d(, ) on R d, ( f P (µ, x) = d(p, x) 2 + d K µ (p) 2). min p P We consider d(p, x) := D K (p, x). Let p + = arg max q R d κ(µ, q) and P + = P p +. We have 1 2 d K µ (x) f k P + (µ, x) 14d K µ (x). However, constructing p + exactly seems quite difficult. We use its approximation ˆp + s.t. D K (P, ˆp + ) (1 + δ)d K (P, p + ) (obtained through an algorithm that is polynomial in n and 1/δ under reasonable conditions). Define ˆP + = P ˆp +.

36 Constructive topological estimation using d K µ Theorem The weighted Rips filtration {R α ( ˆP +, d K µ P )} can be used to approximate the persistence diagram of d K µ P such that d ln B (Dgm(dK µ P ), Dgm({R α ( ˆP +, d K µ P )})) ln(16). Proof based on the power distance mechanisms [Buchet, Chazal, Oudot and Sheehy 2013], ɛ-interleaving of persistence modules. Open problems: Let d(p, x) = x p Calculate ˆp + without restrictions, efficiently (current works, but messy) Tighten bounds

37 Comments on relating d K µ with f S D_K(x,0), sigma=2 3 x x /2 D_K(x,0), sigma= D_K(x,0), sigma=1/ Figure: Showing that x 0 /2 D K (x, 0) x 0, where the second inequality holds for x 3σ. The kernel distance D K (x, 0) is shown for σ = {1/2, 1, 2} in purple, blue, and red, respectively.

38 Algorithmic and Approximation Observations: Advantages

39 Advantages of the kernel distance summary (I) Small coreset representation for sparse representation and efficient, scalable computation. (II) Its inference is easily interpretable and computable through the superlevel sets of a KDE. (III) It is Lipschitz with respect to the outlier parameter σ when the input x is fixed. (IV) As σ, the kernel distance is bounded by the Wasserstein distance: lim σ D K (µ, ν) W 2 (µ, ν).

40 Small coreset There exists a small ɛ-coreset Q P s.t. d K P dk Q ε and kde P kde Q ε with probability at least 1 δ. Size O(((1/ε) log(1/εδ)) 2d/(d+2) ) [Phillips 2013]. The same holds under a random sample of size O((1/ε 2 )(d + log(1/δ))) [Joshi Kommaraju Phillips 2011]. Operate with P = 100,000,000 [Zheng Jestes Phillips Li 2013]. Stability of persistence diagram is preserved: d B (Dgm(kde P ), Dgm(kde Q )) ε.

41 Small coreset There exists a small ɛ-coreset Q P s.t. d K P dk Q ε and kde P kde Q ε with probability at least 1 δ. Size O(((1/ε) log(1/εδ)) 2d/(d+2) ) [Phillips 2013]. The same holds under a random sample of size O((1/ε 2 )(d + log(1/δ))) [Joshi Kommaraju Phillips 2011]. Operate with P = 100,000,000 [Zheng Jestes Phillips Li 2013]. Stability of persistence diagram is preserved: d B (Dgm(kde P ), Dgm(kde Q )) ε.

42 Geometric inference with KDE Recall d K P c (x) = 2 P 2kde P (x) where c 2 P is a constant that depends only on P. Perform geometric inference on noisy P by considering the super-level sets of kde P, Key: {x R d kde P (x) τ} d K P ( ) is monotonic with kde P ( ); as d K P (x) gets smaller, kde P (x) gets larger. A clean and natural interpretation of the reconstruction problem through the well-studied lens of KDE. Geometric inference with sublevel sets of d K P (superlevel sets of kde P ).

43 Experiments An example with 25% of P as noise, σ = 0.05

44 Experiments An example with 25% of P as noise, σ = 0.003

45 Experiments An example with 25% of P as noise, σ = 0.001

46 Comparison with distance to a measure (a) (b) Figure: Sublevel sets of kernel distance (a), and distance to a measure (b) on a data set with two circles with different scales and densities. These plots fix a level set γ and vary (a) σ for d K µ and (b) m 0 for d CCM The parameters are chosen to provide similar appearing images. µ,m 0.

47 Future directions

48 Beyond Gaussian kernels More general theory for KDE with systematic understanding of family of kernels: distance to a measure (KNN kernel), kernel distance (a larger class of kernels, e.g. Gaussian, Laplace; triangle kernel may work OK in practice with less perfect properties).

49 Alternative KDE Laplace kernel K(p, x) = exp( 2 x y /σ)

50 Alternative KDE Triangle kernel: K(x, p) = max { } 0, 1 p x σ=0.05

51 Alternative KDE Epanechnikov kernel: (reconstruction) } K(x, p) = max {0, 1 p x 2 (σ=0.05) 2

52 Alternative KDE Ball kernel: K(x, p) = { 1 if p x σ = otherwise. α shape can be viewed as using the ball kernel with σ = α and r = 1/n.

53 Alternative KDEs

54 Multi-dim / scale space persistence? Parameter selection? Two parameters r (isolevel) and σ (outlier/bandwidth) that control the scale. Figure: Sublevel sets for the kernel distance. Left: fix σ, vary r. Right: fix r, vary σ. The values of σ and r are chosen to make the plots similar.

55 Acknowlegement The authors thank D. Sheehy, F. Chazal and the rest of the Geometrica group at INRIA-Saclay for enlightening discussions on geometric and topological reconstruction. We also thank D. Sheehy for personal communications regarding the power distance constructions.

56 References Paper available at: