Pattern learning and recognition on statistical manifolds: An information-geometric review

Transcription

1 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 1/64 Pattern learning and recognition on statistical manifolds: An information-geometric review Frank Nielsen Sony Computer Science Laboratories, Inc. July 2013, SIMBAD, York, UK.

2 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 2/64 Praise for Computational Information Geometry What is Information? = Essence of data (datum= thing ) (make it tangible e.g., parameters of generative models) Can we do Intrinsic computing? (unbiased by any particular data representation same results after recoding data) Geometry?! Science of invariance (mother of Science, compass & ruler, Descartes analytic=coordinate/cartesian, imaginaries,...). The open-ended poetic mathematics...

3 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 3/64 Rationale for Computational Information Geometry Information is...never void! lower bounds Cramér-Rao lower bound and Fisher information (estimation) Bayes error and Chernoff information (classification) Coding and Shannon entropy (communication) Program and Kolmogorov complexity (compression). (Unfortunately not computable!) Geometry: Language (point, line, ball, dimension, orthogonal, projection, geodesic, immersion, etc.) Power of characterization (eg., intersection of two pseudo-segments not admitting closed-form expression) Computing: Information computing. Seeking for mathematical convenience and mathematical tricks (eg., kernel). Do you know the space of functions?!? (Infinity and 1-to-1 mapping, language vs continuum)

4 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 4/64 This talk: Information-geometric Pattern Recognition Focus on statistical pattern recognition geometric computing. Consider probability distribution families (parameter spaces) and statistical manifolds. Beyond traditional Riemannian and Hilbert sphere representations (p p) Describe dually flat spaces induced by convex functions: Legendre transformation dual and mixed coordinate systems Dual similarities/divergences Computing-friendly dual affine geodesics

5 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 5/64 SIMBAD 2013 Information-geometric Pattern Recognition By departing from vector-space representations one is confronted with the challenging problem of dealing with (dis)similarities that do not necessarily possess the Euclidean behavior or not even obey the requirements of a metric. The lack of the Euclidean and/or metric properties undermines the very foundations of traditional pattern recognition theories and algorithms, and poses totally new theoretical/computational questions and challenges. Thank you to Prof. Edwin Hancock (U. York, UK) and Prof. Marcello Pelillo (U. Venice, IT)

6 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 6/64 Statistical Pattern Recognition Models data with distributions (generative) or stochastic processes: parametric (Gaussians, histograms) [model size D], semi-parametric (mixtures) [model size kd], non-parametric (kernel density estimators [model size n], Dirichlet/Gaussian processes [model size D log n],) Data = Pattern ( information) + noise (independent) Statistical machine learning hot topic: Deep learning (restricted Boltzmann machines)

7 Example I: Information-geometric PR (I) Pattern = Gaussian mixture models (universal class) Statistical (dis)similarity/distance: total Bregman divergence (tbd, tkl). Invariance:..., p i (x) N(µ i,σ i ), y = A(x) = Lx +t, y i N(Lµ i +t,lσ i L ), D(X 1 : X 2 ) = D(Y 1 : Y 2 ) (L: any invertible affine transformation, t a translation) Shape Retrieval using Hierarchical Total Bregman Soft Clustering [11], IEEE PAMI, c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 7/64

8 Example II: Information-geometric PR DTI: diffusion ellipsoids, tensor interpolation. Pattern = zero-centered Gaussians Statistical (dis)similarity/distance: total Bregman divergence (tbd, tkl). Invariance:..., D(A PA : A QA) = D(P : Q), A SL(d): orthogonal matrix (volume/orientation preserving) total Bregman divergence (tbd). (3D rat corpus callosum) Total Bregman Divergence and its Applications to DTI Analysis [34], IEEE TMI c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 8/64

9 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 9/64 Statistical mixtures: Generative models of data sets GMM = feature descriptor for information retrieval (IR) classification [21], matching, etc. Increase dimension using color image patches. Low-frequency information encoded into compact statistical model. Generative model statistical image by GMM sampling. A mixture k i=1 w in(µ i,σ i ) is interpreted as a weighted point set in a parameter space: {w i,θ i = (µ i,σ i )} k i=1.

10 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 10/64 Statistical invariance Riemannian structure (M,g) on {p(x;θ) θ Θ R D } θ-invariance under non-singular parameterization: ρ(p(x;θ),p(x;θ )) = ρ(p(x;λ(θ)),p(x;λ(θ ))) Normal parameterization (µ,σ) or (µ,σ 2 ) yields same distance x-invariance under different x-representation: Sufficient statistics (Fisher, 1922): Pr(X = x t(x) = t,θ) = Pr(X = x T(X) = t) All information for θ is contained in T. Lossless information data reduction (exponential families). Markov kernel = statistical morphism (Chentsov 1972,[6, 7]). A particular Markov kernel is a deterministic mapping T : X Y with y = T(x), p y = p x T 1. Invariance if and only if g Fisher information matrix

11 f-divergences (1960 s) A statistical non-metric distance between two probability measures: I f (p : q) = f ( ) p(x) q(x)dx q(x) f: continuous convex function with f(1) = 0,f (1) = 0,f (1) = 1. asymmetric (not a metric, except TV), modulo affine term. can always be symmetrized using s = f +f, with f (x) = xf(1/x). include many well-known statistical measures: Kullback-Leibler, α-divergences, Hellinger, Chi squared, total variation (TV), etc. f-divergences are the only statistical divergences that preserves equivalence wrt. sufficient statistic mapping: I f (p : q) I f (p M : q M ) with equality if and only if M = T (monotonicity property). c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 11/64

12 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 12/64 Information geometry: Dually flat spaces, α-geometry Statistical invariance also obtained using (M,g,, ) where and are dual affine connections. Riemannian structure (M,g) is particular case for = = 0, Levi-Civita connection: (M,g) = (M,g, (0), (0) ) Dually flat space (α = ±1) are algorithmically-friendly: Statistical mixtures of exponential families Learning & simplifying mixtures (k-mle) Bregman Voronoi diagrams & dually triangulations Goal: Algorithmics of Gaussians/histograms wrt. Kullback-Leibler divergence.

13 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 13/64 Exponential Family Mixture Models (EFMMs) Generalize Gaussian & Rayleigh MMs to many usual distributions. m(x) = k i=1 w i p F (x;λ i ) with i w i > 0, k i=1 w i = 1 p F (x;λ) = e t(x),θ F(θ)+k(x) F: log-laplace transform (partition, cumulant function): F(θ) = log e t(x),θ +k(x) dx, θ Θ = { θ the natural parameter space. x X x X d: Dimension of the support X. } e t(x),θ +k(x) dx < D: order of the family (= dimθ). Statistic: t(x) : R d R D.

14 Statistical mixtures: Rayleigh MMs [33, 22] IntraVascular UltraSound (IVUS) imaging: Rayleigh distribution: p(x;λ) = x e x2 λ 2 2λ 2 x R + d = 1 (univariate) D = 1 (order 1) θ = 1 2λ 2 Θ = (,0) F(θ) = log( 2θ) t(x) = x 2 k(x) = logx (Weibull k = 2) Coronary plaques: fibrotic tissues, calcified tissues, lipidic tissues Rayleigh Mixture Models (RMMs): for segmentation and classification tasks c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 14/64

15 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 15/64 Statistical mixtures: Gaussian MMs [8, 22, 9] Gaussian mixture models (GMMs): model low frequency. Color image interpreted as a 5D xyrgb point set. Gaussian distribution p(x; µ, Σ): 1 e 1 2 D Σ 1 (x µ,x µ) Σ (2π) d 2 Squared Mahalanobis distance: D Q (x,y) = (x y) T Q(x y) x R d d (multivariate) D = d(d+3) 2 (order) θ = (Σ 1 µ, 1 2 Σ 1 ) = (θ v,θ M ) Θ = R S++ d F(θ) = 1 4 θt v θ 1 M θ v 1 2 log θ M + d 2 logπ t(x) = (x, xx T ) k(x) = 0

16 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 16/64 Sampling from a Gaussian Mixture Model To sample a variate x from a GMM: Choose a component l according to the weight distribution w 1,...,w k, Draw a variate x according to N(µ l,σ l ). Sampling is a doubly stochastic process: throw a biased dice with k faces to choose the component: l Multinomial(w 1,...,w k ) (Multinomial is also an EF, normalized histogram.) then draw at random a variate x from the l-th component x Normal(µ l,σ l ) x = µ+cz with Cholesky: Σ = CC T and z = [z 1... z d ] T standard normal random variate:z i = 2logU 1 cos(2πu 2 )

17 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 17/64 Relative entropy for exponential families Distance between features (e.g., GMMs) Kullback-Leibler divergence (cross-entropy minus entropy): KL(P : Q) = = p(x)log p(x) q(x) dx 0 p(x)log 1 q(x) dx } {{ } H (P:Q) p(x)log 1 p(x) dx } {{ } H(p)=H (P:P) = F(θ Q ) F(θ P ) θ Q θ P, F(θ P ) = B F (θ Q : θ P ) Bregman divergence B F defined for a strictly convex and differentiable function up to some affine terms. Proof KL(P : Q) = B F (θ Q : θ P ) follows from X E F (θ) = E[t(X)] = F(θ)

18 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 18/64 Convex duality: Legendre transformation For a strictly convex and differentiable function F : X R: F (y) = sup{ y,x F(x) } x X }{{} l F (y;x); Maximum obtained for y = F(x): x l F (y;x) = y F(x) = 0 y = F(x) Maximum unique from convexity of F ( 2 F 0): Convex conjugates: 2 x l F(y;x) = 2 F(x) 0 (F,X) (F,Y), Y = { F(x) x X}

19 Legendre duality: Geometric interpretation Consider the epigraph of F as a convex object: convex hull (V-representation), versus half-space (H-representation). z O Dual coordinate systems: P = x P H P : y P = F (x P ) Q F H Q : z = (x x Q )F (p)+f(x Q ) HP+ z P = F(x P ) H P : z = (x x P )F (x P )+F(x P ) 0 x P P : (x,f(x)) x (0,F(x P ) x P F (x P ) = F (y P )) Legendre transform also called slope transform. c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 19/64

20 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 20/64 Legendre duality & Canonical divergence Convex conjugates have functional inverse gradients F 1 = F F may require numerical approximation (not always available in analytical closed-form) Involution: (F ) = F with F = ( F) 1. Convex conjugate F expressed using ( F) 1 : F (y) = x,y F(x),x = y F (y) = ( F) 1 (y),y F(( F) 1 (y)) Fenchel-Young inequality at the heart of canonical divergence: F(x)+F (y) x,y A F (x : y) = A F (y : x) = F(x)+F (y) x,y 0

21 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 21/64 Dual Bregman divergences & canonical divergence [26] [ KL(P : Q) = E P log p(x) ] 0 q(x) = B F (θ Q : θ P ) = B F (η P : η Q ) = F(θ Q )+F (η P ) θ Q,η P = A F (θ Q : η P ) = A F (η P : θ Q ) with θ Q (natural parameterization) and η P = E P [t(x)] = F(θ P ) (moment parameterization). KL(P : Q) = p(x)log 1 q(x) dx p(x)log 1 p(x) dx } {{ } H (P:Q) Shannon cross-entropy and entropy of EF [26]: } {{ } H(p)=H (P:P) H (P : Q) = F(θ Q ) θ Q, F(θ P ) E P [k(x)] H(P) = F(θ P ) θ P, F(θ P ) E P [k(x)] H(P) = F (η P ) E P [k(x)]

22 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 22/64 Bregman divergence: Geometric interpretation (I) Potential function F, graph plot F : (x,f(x)). D F (p : q) = F(p) F(q) p q, F(q)

23 Bregman divergence: Geometric interpretation (III) Bregman divergence and path integrals B(θ 1 : θ 2 ) = F(θ 1 ) F(θ 2 ) θ 1 θ 2, F(θ 2 ), (1) = η = F(θ) θ1 θ 2 F(t) F(θ 2 ),dt, (2) η2 = F (t) F (η 1 ),dt, (3) η 1 = B (η 2 : η 1 ) (4) η 1 η 2 θ 2 θ 1 θ c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 23/64

24 Bregman divergence: Geometric interpretation (II) Potential function f, graph plot F : (x,f(x)). B f (p q) = f(p) f(q) (p q)f (q) F ˆq ˆp B f (p q) H q X q p B f (. q): vertical distance between the hyperplane H q tangent to F at lifted point ˆq, and the translated hyperplane at ˆp. c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 24/64

25 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 25/64 total Bregman divergence (tbd) By analogy to least squares and total least squares total Bregman divergence (tbd) [11, 34, 12] δ f (x,y) = b f (x,y) 1+ f(y) 2 Proved statistical robustness of tbd.

26 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 26/64 Bregman sided centroids [25, 21] Bregman centroids = unique minimizers of average Bregman divergences (B F convex in right argument) θ = argmin θ 1 n θ = argmin θ 1 n n B F (θ i : θ) i=1 n B F (θ : θ i ) i=1 θ = 1 n n θ i, center of mass, independent of F i=1 θ = ( F) 1 ( 1 n ) n ( F)(θ i ) i=1 Generalized Kolmogorov-Nagumo f-means.

27 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 27/64 Bregman divergences B F and F-means Bijection quasi-arithmetic means ( F) Bregman divergence B F. Bregman divergence B F F f = F f 1 = (F ) 1 f-mean (entropy/loss function F) (Generalized means) Squared Euclidean distance 1 2 x 2 x x Arithmetic mean (half squared loss) nj=1 1 n x j Kullback-Leibler divergence x log x x log x exp x Geometric mean (Ext. neg. Shannon entropy) Itakura-Saito divergence logx x 1 (Burg entropy) 1 x ( n j=1 x j ) 1 n Harmonic mean n nj=1 1 xj F strictly increasing (like cumulative distribution functions)

28 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 28/64 Bregman sided centroids [25] Two sided centroids C and C expressed using two θ/η coordinate systems: = 4 equations. C : θ, η C : θ, η C : θ = 1 n n i=1 θ i η = F( θ) C : η = 1 n n i=1 η i θ = F ( η)

29 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 29/64 Bregman centroids and Jacobian fields Centroid θ zeroes the left-sided Jacobian vector field: ( ) θ w t A(θ : η t ) t Sum of tangent vectors of geodesics from centroid to points = 0 θ A(θ : η ) = η η θ ( t w t A(θ : η t )) = η η since t w t = 1, with η = t w tη t.

30 Bregman information [25] Bregman information = minimum of loss function I F (P) = 1 n = 1 n = 1 n n B F (θ i : θ) i=1 n F(θ i ) F( θ) θ i θ, F( θ) i=1 n F(θ i ) F( θ) i=1 = J F (θ 1,...,θ n ) 1 n n θ i θ, F( θ) i=1 } {{ } =0 Jensen diversity index (e.g., Jensen-Shannon for F(x) = x logx) For squared Euclidean distance, Bregman information = cluster variance, For Kullback-Leibler divergence, Bregman information related to mutual information. c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 30/64

31 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 31/64 Bregman k-means clustering [4] Bregman k-means: Find k centers C = {C 1,...,C k } that minimizes the loss function: L F (P : C) = P PB F (P : C) B F (P : C) = min B F(P : C i ) i {1,...,k} generalize Lloyd s quadratic error in Vector Quantization (VQ) L F (P : C) = I F (P) I F (C) I F (P) total Bregman information I F (C) between-cluster Bregman information L F (P : C) within-cluster Bregman information total Bregman information = within-cluster Bregman information + between-cluster Bregman information

32 Bregman k-means clustering [4] I F (P) = L F (P : C)+I F (C) Bregman clustering amounts to find the partition C that minimizes the information loss: Bregman k-means : L F = L F (P : C ) = min C (I F (P) I F (C)) Initialize distinct seeds: C 1 = P 1,...,C k = P k Repeat until convergence Assign point Pi to its closest centroid: C i = {P P B F (P : C i ) B F (P : C j ) j i} Update cluster centroids by taking their center of mass: C i = 1 C i P C i P. Loss function monotonically decreases and converges to a local optimum. (Extend to weighted point sets using barycenters.) c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 32/64

33 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 33/64 Bregman k-means++ [1]: Careful seeding (only?!) (also called Bregman k-medians since min i B1 F (p i : x)). Extend the D 2 -initialization of k-means++ Only seeding stage yields probabilistically guaranteed global approximation factor: Bregman k-means++: Choose C = {C l } for l uniformly random in {1,...,n} While C < k Choose P P with probability B F (P:C) n i=1 B F(P i:c) = B F(P:C) L F (P:C) Yields a O(log k) approximation factor (with high probability). Constant in O( ) depends on ratio of min/max 2 F.

34 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 34/64 Anisotropic Voronoi diagram (for MVN MMs) [10, 14] Learning mixtures, Talk of O. Schwander on EM, k-mle From the source color image (a), we buid a 5D GMM with k = 32 components, and color each pixel with the mean color of the anisotropic Voronoi cell it belongs to. ( weighted squared Mahalanobis distance per center) logp F (x;θ i ) B F (t(x) : η i )+k(x) Most likely component segmentation Bregman Voronoi diagram (squared Mahalanobis for Gaussians)

35 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 35/64 Voronoi diagrams in dually flat spaces... Voronoi diagram, dual Delaunay triangulation (general position)

36 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 36/64 Bregman dual bisectors: Hyperplanes & hypersurfaces [5, 24, 27] Right-sided bisector: Hyperplane (θ-hyperplane) H F : H F (p,q) = {x X B F (x : p) = B F (x : q)}. F(p) F(q),x +(F(p) F(q)+ q, F(q) p, F(p) ) = 0 Left-sided bisector: Hypersurface (η-hyperplane) H F (p,q) = {x X B F(p : x) = B F (q : x)} H F : F(x),q p +F(p) F(q) = 0

37 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 37/64 Visualizing Bregman bisectors Primal coordinates θ Dual coordinates η natural parameters expectation parameters Gradient Space: Bernouilli p ( , ) q ( , ) D*(p,q )= D*(q,p )= q q Source Space: Logistic loss p p( , ) q( , ) D(p,q)= D(q,p)= p Gradient Space: Itakura-Saito dual p ( , ) q ( , ) D*(p,q )= D*(q,p )= p p Source Space: Itakura-Saito p( , ) q( , ) q D(p,q)= D(q,p)= q

38 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 38/64 Bregman Voronoi diagrams as minimization diagrams [5] A subclass of affine diagrams which have all non-empty cells. Minimization diagram of the n functions D i (x) = B F (x : p i ) = F(x) F(p i ) x p i, F(p i ). minimization of n linear functions: H i (x) = (p i x) T F(q i ) F(p i )

39 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 39/64 Bregman dual Delaunay triangulations Delaunay Exponential Hellinger-like empty Bregman sphere property, geodesic triangles. BVDs extends Euclidean Voronoi diagrams with similar complexity/algorithms.

40 Non-commutative Bregman Orthogonality 3-point property (generalized law of cosines): B F (p : r) = B F (p : q)+b F (q : r) (p q) T ( F(r) F(q))) P Γ PQ DF(P R) DF(P Q) ΓQR Q DF(Q R) (pq) θ Bregman orthogonal to (qr) η iff. R B F (p : r) = B F (p : q)+b F (q : r) (Equivalent to θ p θ q,η r η q = 0) Extend Pythagoras theorem (pq) θ F (qr) η F is not commutative except in the squared Euclidean/Mahalanobis case, c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 40/64

41 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 41/64 Dually orthogonal Bregman Voronoi & Triangulations Ordinary Voronoi diagram is perpendicular to Delaunay triangulation. Dual line segment geodesics: (pq) θ = {θ = θ p +(1 λ)θ q λ [0,1]} (pq) η = {η = η p +(1 λ)η q λ [0,1]} Bisectors: B θ (p,q) : x,θ q θ p +F(θ p ) F(θ q ) = 0 B η (p,q) : x,η q η p +F (η p ) F (η q ) = 0 Dual orthogonality: B η (p,q) (pq) η (pq) θ B θ (p,q)

42 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 42/64 Dually orthogonal Bregman Voronoi & Triangulations B η (p,q) (pq) η (pq) θ B θ (p,q)

43 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 43/64 Simplifying mixture: Kullback-Leibler projection theorem An exponential family mixture model p = k i=1 w ip F (x;θ i ) Right-sided KL barycenter p of components interpreted as the projection of the mixture model p P onto the model exponential family manifold E F [32]: p = arg min p E F KL( p : p) P e-geodesic p p m-geodesic p E F exponential family mixture sub-manifold manifold of probability distribution Right-sided KL centroid = Left-sided Bregman centroid

44 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 44/64 A simple proof argmin θ KL(m(x) : p θ (x)) = E m [logm] E m [logp θ ] = E m [logm] E m [k(x)] E m [ t(x),θ F(θ)] argmax θ E m [ t(x),θ ] F(θ) = argmax θ E m [t(x)],θ ] F(θ) E m [t(x)] = t w t E pθ t [t(x)] = t w t η t = η F(θ) = η = η θ = F ( η)

45 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 45/64 Left-sided or right-sided Kullback-Leibler centroids? Left/right Bregman centroids=right/left entropic centroids (KL of exp. fam.) Left-sided/right-sided centroids: different (statistical) properties: Right-sided entropic centroid: zero-avoiding (cover support of pdfs.) Left-sided entropic centroid: zero-forcing (captures highest mode). Left-side Kullback-Leibler centroid (zero-forcing) N2 = (5,0.64) Right-sided Kullback-Leibler centroid zero-avoiding N1 = N( 4,4) Symmetrized centroid

46 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 46/64 Hierarchical clustering of GMMs (Burbea-Rao) Hierarchical clustering of GMMs wrt. Bhattacharyya distance. Simplify the number of components of an initial GMM. (a) source (b) k = 48 (c) k = 16

47 Two symmetrizations of Bregman divergences Jeffreys-Bregman divergences. S F (p;q) = B F(p,q)+B F (q,p) 2 = 1 p q, F(p) F(q), 2 Jensen-Bregman divergences (diversity index). J F (p;q) = B F(p, p+q 2 )+B F(q, p+q 2 ) 2 = F(p)+F(q) ( ) p +q F 2 2 = BR F (p,q) Skew Jensen divergence [21, 29] J (α) F (p;q) = αf(p)+(1 α)f(q) F(αp+(1 α)q) = BR(α) F (p;q) (Jeffreys and Jensen-Shannon symmetrization of Kullback-Leibler) c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 47/64

48 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 48/64 Burbea-Rao centroids (α-skewed Jensen centroids) Minimum average divergence: OPT : c = argmin x Equivalent to minimize: E(c) = ( n i=1 n w i α)f(c) i=1 w i J (α) F (x,p i) = argmin x L(x) n w i F(αc +(1 α)p i ) i=1 Sum E = F +G of convex F + concave G function Convex-ConCave Procedure (CCCP) Start from arbitrary c 0, and iteratively update as: F(c t+1 ) = G(c t ) guaranteed convergence to a local minimum.

49 ConCave Convex Procedure (CCCP) min x E(x) = F(x)+G(x) F(c t+1 ) = G(c t ) c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 49/64

50 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 50/64 Iterative algorithm for Burbea-Rao centroids Apply CCCP scheme n F(c t+1 ) = w i F (αc t +(1 α)p i ) i=1 ( n ) c t+1 = F 1 w i F (αc t +(1 α)p i ) i=1 Get arbitrarily fine approximations of the (skew) Burbea-Rao centroids and barycenters. Unique GLOBAL minimum when divergence is separable [21]. Unique GLOBAL minimum for matrix mean [23] for the logdet divergence.

51 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 51/64 Information-geometric computing on statistical manifolds Cramér-Rao Lower Bound and Information Geometry [13] (overview article) Chernoff information on statistical exponential family manifold [19] Hypothesis testing and Bregman Voronoi diagrams [18] Jeffreys centroid [20] Learning mixtures with k-mle [17] Closed-form divergences for statistical mixtures [16]

52 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 52/64 Summary Information-geometric pattern recognition: Statistical manifold (M, g): Rao s distance and Fisher-Rao curved Riemannian geometry. Statistical manifold (M,g,, ): dually flat spaces, Bregman divergences, geodesics are straight lines in either θ/η parameter space. ±1-geometry, special case of α-geometry Clustering in dually flat spaces Software library: jmef [8] (Java), pymef [31] (Python)... but also many other geometries to explore: Hilbertian, Finsler [3], Kähler, Wasserstein, Contact, Symplectic, etc. (it is easy to require non-euclidean geometry but then space is wild open!)

53 Edited book (MIG) and proceedings (GSI) c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 53/64

54 Thank you. c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 54/64

55 Exponential families & statistical distances Universal density estimators [2] generalizing Gaussians/histograms (single EF density approximates any smooth density) Explicit formula for Shannon entropy, cross-entropy, and Kullback-Leibler divergence [26]: Rényi/Tsallis entropy and divergence [28] Sharma-Mittal entropy and divergence [30]. A 2-parameter family extending extensive Rényi (for β 1) and non-extensive Tsallis entropies (for β α) ( ( ) H α,β (p) = 1 p(x) dx)1 β α 1 α 1, 1 β with α > 0,α 1,β 1. Skew Jensen and Burbea-Rao divergence [21] Chernoff information and divergence [15] Mixtures: total Least square, Jensen-Rényi, Cauchy-Schwarz divergence [16]. c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 55/64

56 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 56/64 Statistical invariance: Markov kernel Probability family: p(x; θ). (X,σ) and (X,σ ) two measurable spaces. σ: A σ-algebra on X (non-empty, closed under complementation and countable union). Markov kernel = transition probability kernel K : X σ [0,1]: E σ,k(,e ) measurable map, x X,K(x, ) is a probability measure on (X,σ ). p a pm. on (X,σ) induces Kp a pm., with Kp(E ) = K(x,E )p(dx), E σ X

57 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 57/64 Space of Bregman spheres and Bregman balls [5] Dual Bregman balls (bounding Bregman spheres): Ball r F (c,r) = {x X B F(x : c) r} and Ball l F (c,r) = {x X B F(c : x) r} Legendre duality: Ball l F (c,r) = ( F) 1 (Ball r F ( F(c),r)) Illustration for Itakura-Saito divergence, F(x) = log x

58 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 58/64 Space of Bregman spheres: Lifting map [5] F : x ˆx = (x,f(x)), hypersurface in R d+1. H p : Tangent hyperplane at ˆp, z = H p (x) = x p, F(p) +F(p) Bregman sphere σ ˆσ with supporting hyperplane H σ : z = x c, F(c) +F(c)+r. (// to H c and shifted vertically by r) ˆσ = F H σ. intersection of any hyperplane H with F projects onto X as a Bregman sphere: H : z = x,a +b σ : Ball F (c = ( F) 1 (a),r = a,c F(c)+b)

59 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 59/64 Bibliographic references I Marcel R. Ackermann and Johannes Blömer. Bregman clustering for separable instances. In Scandinavian Workshop on Algorithm Theory (SWAT), pages , Yasemin Altun, Alexander J. Smola, and Thomas Hofmann. Exponential families for conditional random fields. In Uncertainty in Artificial Intelligence (UAI), pages 2 9, Marc Arnaudon and Frank Nielsen. Medians and means in Finsler geometry. LMS Journal of Computation and Mathematics, 15, Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with Bregman divergences. Journal of Machine Learning Research, 6: , Jean-Daniel Boissonnat, Frank Nielsen, and Richard Nock. Bregman Voronoi diagrams. Discrete & Computational Geometry, 44(2): , Nikolai Nikolaevich Chentsov. Statistical Decision Rules and Optimal Inferences. Transactions of Mathematics Monograph, numero 53, Published in russian in 1972.

60 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 60/64 Bibliographic references II José Manuel Corcuera and Federica Giummolé. A characterization of monotone and regular divergences. Annals of the Institute of Statistical Mathematics, 50(3): , Vincent Garcia and Frank Nielsen. Simplification and hierarchical representations of mixtures of exponential families. Signal Processing (Elsevier), 90(12): , Vincent Garcia, Frank Nielsen, and Richard Nock. Levels of details for Gaussian mixture models. In Asian Conference on Computer Vision (ACCV), volume 2, pages , François Labelle and Jonathan Richard Shewchuk. Anisotropic Voronoi diagrams and guaranteed-quality anisotropic mesh generation. In Proceedings of the nineteenth annual symposium on Computational geometry, SCG 03, pages , New York, NY, USA, ACM. Meizhu Liu, Baba C. Vemuri, Shun-ichi Amari, and Frank Nielsen. Shape retrieval using hierarchical total Bregman soft clustering. Transactions on Pattern Analysis and Machine Intelligence, 34(12): , Meizhu Liu, Baba C. Vemuri, Shun ichi Amari, and Frank Nielsen. Total Bregman divergence and its applications to shape retrieval. In International Conference on Computer Vision (CVPR), pages , 2010.

61 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 61/64 Bibliographic references III Frank Nielsen. Cramér-rao lower bound and information geometry. In Rajendra Bhatia, C. S. Rajan, and Ajit Iqbal Singh, editors, Connected at Infinity II: A selection of mathematics by Indians. Hindustan Book Agency (Texts and Readings in Mathematics, TRIM). arxiv Frank Nielsen. Visual Computing: Geometry, Graphics, and Vision. Charles River Media / Thomson Delmar Learning, Frank Nielsen. Chernoff information of exponential families. arxiv, abs/ , Frank Nielsen. Closed-form information-theoretic divergences for statistical mixtures. In International Conference on Pattern Recognition (ICPR), pages , Frank Nielsen. k-mle: A fast algorithm for learning statistical mixture models. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, preliminary, technical report on arxiv.

62 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 62/64 Bibliographic references IV Frank Nielsen. Hypothesis testing, information divergence and computational geometry. In F. Nielsen and F. Barbaresco, editors, Geometric Science of Information (GSI), volume 8085 of LNCS, pages Springer, Frank Nielsen. An information-geometric characterization of Chernoff information. IEEE Signal Processing Letters, 20(3): , Frank Nielsen. Jeffreys centroids: A closed-form expression for positive histograms and a guaranteed tight approximation for frequency histograms. Signal Processing Letters, IEEE, PP(99):1 1, Frank Nielsen and Sylvain Boltz. The Burbea-Rao and Bhattacharyya centroids. IEEE Transactions on Information Theory, 57(8): , August Frank Nielsen and Vincent Garcia. Statistical exponential families: A digest with flash cards, arxiv.org:

63 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 63/64 Bibliographic references V Frank Nielsen, Meizhu Liu, Xiaojing Ye, and Baba C. Vemuri. Jensen divergence based SPD matrix means and applications. In International Conference on Pattern Recognition (ICPR), Frank Nielsen and Richard Nock. The dual Voronoi diagrams with respect to representational Bregman divergences. In International Symposium on Voronoi Diagrams (ISVD), pages 71 78, Frank Nielsen and Richard Nock. Sided and symmetrized Bregman centroids. IEEE Transactions on Information Theory, 55(6): , June Frank Nielsen and Richard Nock. Entropies and cross-entropies of exponential families. In International Conference on Image Processing (ICIP), pages , Frank Nielsen and Richard Nock. Hyperbolic Voronoi diagrams made easy. In International Conference on Computational Science and its Applications (ICCSA), volume 1, pages 74 80, Los Alamitos, CA, USA, march IEEE Computer Society. Frank Nielsen and Richard Nock. On rényi and tsallis entropies and divergences for exponential families. arxiv, abs/ , 2011.

64 c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 64/64 Bibliographic references VI Frank Nielsen and Richard Nock. Skew Jensen-Bregman Voronoi diagrams. Transactions on Computational Science, 14: , Frank Nielsen and Richard Nock. A closed-form expression for the Sharma-Mittal entropy of exponential families. Journal of Physics A: Mathematical and Theoretical, 45(3), Olivier Schwander and Frank Nielsen. PyMEF - A framework for exponential families in Python. In IEEE/SP Workshop on Statistical Signal Processing (SSP), Olivier Schwander and Frank Nielsen. Learning mixtures by simplifying kernel density estimators. In Frank Nielsen and Rajendra Bhatia, editors, Matrix Information Geometry, pages , Jose Seabra, Francesco Ciompi, Oriol Pujol, Josepa Mauri, Petia Radeva, and Joao Sanchez. Rayleigh mixture model for plaque characterization in intravascular ultrasound. IEEE Transaction on Biomedical Engineering, 58(5): , Baba Vemuri, Meizhu Liu, Shun ichi Amari, and Frank Nielsen. Total Bregman divergence and its applications to DTI analysis. IEEE Transactions on Medical Imaging, 2011.