Penalized Barycenters in the Wasserstein space

Transcription

1 Penalized Barycenters in the Wasserstein space Elsa Cazelles, joint work with Jérémie Bigot & Nicolas Papadakis Université de Bordeaux & CNRS Journées IOP - Du 5 au 8 Juillet 2017 Bordeaux Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 1 / 29

2 Motivation and definitions 1 Motivation and definitions 2 Convergence to a population Wasserstein barycenter 3 Stability of the minimizer 4 Application to real and simulated data Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 2 / 29

3 Motivation and definitions For a given name, an histogram is the proportion of children born with that name per year in France between 1900 and Source: INSEE Carmen Chantal Pierre Jesus Nicolas Pamela Emmanuel Yves Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 3 / 29

4 Motivation and definitions Euclidean barycenter Wasserstein barycenter Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 4 / 29

5 Motivation and definitions Figure: (a) A subset of 3 smoothed neural spike trains out of n = 60. (b) Probability density function of the empirical Wasserstein barycenter. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 5 / 29

6 Motivation and definitions 100 January Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

7 Motivation and definitions 100 February Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

8 Motivation and definitions 100 March Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

9 Motivation and definitions 100 April Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

10 Motivation and definitions 100 May Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

11 Motivation and definitions 100 June Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

12 Motivation and definitions 100 July Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

13 Motivation and definitions 100 August Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

14 Motivation and definitions 100 September Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

15 Motivation and definitions 100 October Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

16 Motivation and definitions 100 November Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

17 Motivation and definitions 100 December Figure: All crimes registered in the city of Chicago for a giver month of from the Chicago Police Department s CLEAR. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 6 / 29

18 Motivation and definitions The data The measures ν 1,..., ν n are defined by a dataset of random variables (X i,j ) 1 i n; 1 j pi organized in the form of n experimental units, that is X i,1,..., X i,pi ν i = 1 p i p i δ Xi,j j=1 iid observations sampled from a measure ν i Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 7 / 29

19 Motivation and definitions Ω R d convex space P 2 (Ω) the set of Borel probability measures supported on Ω with finite moment of order 2 Definition The 2-Wasserstein distance between two measures µ and ν in P p (Ω) is defined by W 2 (µ, ν) = ( inf π Π(µ,ν) Ω 2 x y 2 dπ(x, y) ) 1/2 where the infimum is taken over all probability measures π on the product space Ω Ω with respective marginals µ and ν. In R, W 2 2 (µ, ν) = 1 F 1 0 where F 1 is the quantile function. µ (x) Fν 1 (x) 2 dx Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 8 / 29

20 Motivation and definitions For Ω R d, P 2 (Ω) is the set of probability measures. Definition Let P ν n = 1 n ni=1 δ νi where δ νi is the dirac distribution at ν i P 2 (Ω). We define the regularized empirical barycenter of the discrete measure P ν n as µ γ P ν n = argmin µ P 2 (Ω) 1 n n W2 2 (µ, ν i ) + γe(µ) i=1 where γ > 0 is a regularisation parameter and the penalty E is a proper, differentiable, lower semicontinuous and strictly convex function. Case γ = 0 : Wasserstein barycenter of [Agueh and Carlier, 2011] ν n argmin µ 1 n n W2 2 (µ, ν i ) i=1 Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 9 / 29

21 Motivation and definitions As an example, take E the negative entropy defined as { E(µ) = Ω f (x) log(f (x))dx, if µ admits a pdf f + otherwise. Advantage: It is possible to enforce the regularized barycenter to be absolutely continuous with respect to the Lebesgue measure on Ω. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 10 / 29

22 Convergence to a population Wasserstein barycenter 1 Motivation and definitions 2 Convergence to a population Wasserstein barycenter 3 Stability of the minimizer 4 Application to real and simulated data Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 11 / 29

23 Convergence to a population Wasserstein barycenter Consistency of the regularized barycenter µ γ P n We define the population Wasserstein barycenter defined as µ 0 P argmin W2 2 (µ, ν)dp(ν), µ P 2 (Ω) and its regularized version µ γ P = argmin µ P 2 (Ω) W 2 2 (µ, ν)dp(ν) + γe(µ). where P is a probability measure on P 2 (Ω) and ν 1,..., ν n iid of law P. We recall that µ γ P ν n = argmin µ P 2 (Ω) 1 n n W2 2 (µ, ν i ) + γe(µ) i=1 Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 12 / 29

24 Convergence to a population Wasserstein barycenter The Bregman divergence D E associated to E is defined for two measures µ, ζ as D E (µ, ζ) := E(µ) E(ζ) E(ζ)(dµ dζ) Ω where E denotes the gradient of E. Thus the symmetric Bregman divergence d E is given by d E (µ, ζ) := D E (µ, ζ) + D E (ζ, µ). Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 13 / 29

25 Convergence to a population Wasserstein barycenter The Bregman divergence D E associated to E is defined for two measures µ, ζ as D E (µ, ζ) := E(µ) E(ζ) E(ζ)(dµ dζ) Ω where E denotes the gradient of E. Thus the symmetric Bregman divergence d E is given by d E (µ, ζ) := D E (µ, ζ) + D E (ζ, µ). Theorem For Ω compact in R d and E(µ 0 P ) bounded, lim D E (µ γ γ 0 P, µ0 P) = 0, which corresponds to showing that the squared bias term d 2 E (µγ P, µ0 P ) (as classically referred to in nonparametric statistics) converges to zero when γ 0. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 13 / 29

26 Convergence to a population Wasserstein barycenter Theorem If Ω is a compact of R d, then one has that where C is a positive constant, E(d 2 E (µ γ P n, µ γ P )) C I(1, H) H L 2 (P) γ 2 n H = {h µ : ν P 2 (Ω) W 2 2 (µ, ν) R; µ P 2 (Ω)} is a class of functions defined on P 2 (Ω) with envelope H, and 1 I(1, H) = sup (1 + log N(ε H L2 (Q), H, L2 (Q))) 1 2 dε Q 0 }{{} Metric entropy The metric entropy is of order 1 ε d. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 14 / 29

27 Convergence to a population Wasserstein barycenter Remarks on the metric entropy. Metric entropy of H Metric entropy of P 2 (Ω) Metric entropy of Ω Compacity of Ω Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 15 / 29

28 Convergence to a population Wasserstein barycenter Complementary result in the 1D case Theorem (1-D) When ν 1,..., ν n are iid random measures with support included in a compact interval Ω, ( ( )) E de 2 µ γ P n, µ γ P C γ 2 n. where C > 0 does not depend on n and γ. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 16 / 29

29 Convergence to a population Wasserstein barycenter Complementary result in the 1D case Theorem (1-D) When ν 1,..., ν n are iid random measures with support included in a compact interval Ω, ( ( )) E de 2 µ γ P n, µ γ P C γ 2 n. where C > 0 does not depend on n and γ. Remark: Metric entropy of the space of quantiles. It follows that if γ = γ n is such that lim n γ 2 nn = + then ( ) lim E(d 2 n E µ γ P, ν µ0 n P) = 0. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 16 / 29

30 Convergence to a population Wasserstein barycenter Complementary result with additional regularization in greater dimension When d > 1, the class of functions H is too large, so we have to add regularity. For Ω smooth and uniformly convex, and (ν i ) i=1,...,n of law P. We specify the penalty function E: { E(µ) = R d f (x) log(f (x))dx + f H k (Ω), if f = dµ dλ and f > α + otherwise. where H k (Ω) designates the Sobolev norm associated to the L 2 (Ω) space and α > 0 is arbitrarily small. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 17 / 29

31 Convergence to a population Wasserstein barycenter Sobolev embedding theorem: H k (Ω) is included in the Hölder space C m,β for β = k m d/2. Regularity a.e. on optimal maps is obtained from regularity on probability measures (e.g. [Caffarelli, De Philippis and Figalli]) Hence we can bound the metric entropy [Van der Vaart and Wellner] by ( ) 1 a K for any a d/(m + 1). ε Hence, as soon as a/2 < 1, for which k > d 1 is necessary, we get a rate of convergence. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 18 / 29

32 Stability of the minimizer 1 Motivation and definitions 2 Convergence to a population Wasserstein barycenter 3 Stability of the minimizer 4 Application to real and simulated data Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 19 / 29

33 Stability of the minimizer Stability of the minimizer for the symmetric Bregman distance d E. Theorem Let ν 1,..., ν n and η 1,..., η n be two sequences of probability measures in P 2 (Ω). Let µ γ P and ν µγ n P η be the regularized empirical barycenters n associated to the discrete measures P ν n and P η n, then ( ) d E µ γ P, ν µγ n P η 2 n γn inf σ S n n W 2 (ν i, η σ(i) ) = 2 γ T W 2 (P ν n, P η n), i=1 where S n denotes the permutation group of the set of indices {1,..., n} and T W2 (P, Q) := inf W 2 (µ, ν)dπ(µ, ν). Π P 2 (Ω) P 2 (Ω) Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 20 / 29

34 Stability of the minimizer Application: Let ν 1,..., ν n be n absolutely continuous probability measures and X = (X i,j ) 1 i n; 1 j pi a dataset of random variables such that X i,j ν i. Then ( ( )) E de 2 µ γ P, ν µγ n X 4 γ 2 n n i=1 ( ) E W2 2 (ν i, ν pi ), where µ γ X is the random measure satisfying µ γ X = argmin µ P 2 (Ω) 1 n n W2 2 µ, 1 p i p i=1 i j=1 δ Xi,j + γe(µ). Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 21 / 29

35 Application to real and simulated data Choice of the parameter γ. 1 Motivation and definitions 2 Convergence to a population Wasserstein barycenter 3 Stability of the minimizer 4 Application to real and simulated data Choice of the parameter γ. 1-D case 2-D case Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 22 / 29

36 Application to real and simulated data Choice of the parameter γ. Lepskii balancing principle [Bauer and Munk, 2007] Definition We set λ = 1/γ For σ > 1 and a threshold Λ > 0, the Look-Ahead function is l Λ,σ (λ) = min{min{κ ρ(λ) > σρ(κ)}, Λ} where ρ : λ 1 λ. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 23 / 29

37 Application to real and simulated data Choice of the parameter γ. Lepskii balancing principle [Bauer and Munk, 2007] Definition We set λ = 1/γ For σ > 1 and a threshold Λ > 0, the Look-Ahead function is l Λ,σ (λ) = min{min{κ ρ(λ) > σρ(κ)}, Λ} where ρ : λ 1 λ. For δ > 0, the balancing functional reads b Λ,σ (λ) = max { 1 λ<κ l Λ,σ (λ) 4δ d E (µ 1/λ P n, µ 1/κ P n )ρ(κ)} Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 23 / 29

38 Application to real and simulated data Choice of the parameter γ. Lepskii balancing principle [Bauer and Munk, 2007] Definition We set λ = 1/γ For σ > 1 and a threshold Λ > 0, the Look-Ahead function is l Λ,σ (λ) = min{min{κ ρ(λ) > σρ(κ)}, Λ} where ρ : λ 1 λ. For δ > 0, the balancing functional reads b Λ,σ (λ) = max { 1 λ<κ l Λ,σ (λ) 4δ d E (µ 1/λ P n, µ 1/κ P n )ρ(κ)} The data-driven choice is given by λ Λ,σ,ɛ = min{λ Λ ; B Λ,σ (λ) ɛ} where B Λ,σ (λ) = max {b Λ,σ(κ)} is the smooth balancing functional. λ<κ Λ Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 23 / 29

39 Application to real and simulated data 1-D case We consider 1 i n = 100 random Gaussian distributions N (µ i, σ 2 i ) where µ i random uniform variable on [ 2, 2] σ 2 i random uniform variable on [0, 1]. Then we generate (X ij ) 1 i n;1 j pi, 5 p i 10, random variables such that X ij N (µ i, σ 2 i ) for each 1 i j. Finally, let. ν i = 1 p i p i δ Xij j=1 for each i Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 24 / 29

40 Application to real and simulated data 1-D case Automatic selection of the parameter γ through an adaptation of Lepskii balancing principal [Bauer and Munk] Dirichlet Entropy Dirichlet + Entropy Figure: Balancing functional (times 10 6 ) in solid line and d E (µ P, µ 1/λ P n )/ min λ d E (µ P, µ 1/λ P n ) (dotted line) are plotted as functions of λ for 3 different regularizations. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 25 / 29

41 Application to real and simulated data 1-D case true barycenter kernel method regularized barycenter 0.08 true barycenter kernel method regularized barycenter 0.08 true barycenter kernel method regularized barycenter Dirichlet regularization Entropy regularization Dirichlet + Entropy regularization Dashed and black curve = density of the population Wasserstein barycenter. Blue and dotted curve = the smoothed Wasserstein barycenter obtained by a preliminary kernel smoothing step of the discrete measures ν i that is followed by quantile averaging. Warm color = regularized Wasserstein barycenters for several gamma and regularization. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 26 / 29

42 Application to real and simulated data 1-D case Regularized barycenter Regularized ideal barycenter Kernel estimator 0.9 Regularized barycenter Regularized ideal barycenter Kernel estimator Error with Bregman distance Error with Wasserstein distance Monte-Carlo simulations (200 repetitions): for a given n 0 = 10, 25, 50 and 75, we randomly draw n 0 measures ν i from the whole sample n = 100. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 27 / 29

43 Application to real and simulated data 2-D case e-4 1e-3 1.2e-3 1.4e-3 2e-3 2.9e-3 5e-3 1e-2 Figure: Smooth balancing functional associated to regularized barycenters for different value of λ for Crimes in the city of Chicago. Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 28 / 29

44 Application to real and simulated data 2-D case Regularized Wasserstein barycenter (Dirichlet) Kernel density estimator gkde2 Figure: Location of Crimes in the city of Chicago during the year 2014 Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 29 / 29

45 Application to real and simulated data 2-D case Thank you for your attention! Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP 30 / 29

46 Application to real and simulated data 2-D case [1] Martial Agueh and Guillaume Carlier. Barycenters in the Wasserstein space. SIAM Journal on Mathematical Analysis, 43(2): , [2] Martin Burger, Marzena Franek and Carola-Bibiane Schönlieb. Regularized regression and density estimation based on optimal transport. Applied Mathematics Research express, 2012(2): , [3] Thibault Le Gouic and Jean-Michel Loubes. Existence and consistency of Wasserstein barycenters. arxiv preprint arxiv: , [4] Amir Beck and Marc Teboulle. A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM Journal on Imaging Sciences, 2(1): , [5] Guido De Philippis and Alessio Figalli. The Monge Ampère equation and its link to optimal transportation. Bulletin of the American Mathematical Society, 51(4), , [6] Frank Bauer and Axel Munk. Optimal regularization for ill-posed problems in metric spaces. Journal of Inverse and Ill-posed Problems jiip, 15(2), , 2007 Elsa Cazelles Penalized Barycenters in the Wasserstein space Journées IOP