Diffusion Processes and Dimensionality Reduction on Manifolds

Transcription

1 Faculty of Science Diffusion Processes and Dimensionality Reduction on Manifolds ESI, Vienna, Feb Stefan Sommer Department of Computer Science, University of Copenhagen February 23, 2015 Slide 1/22

2 Outline Dimensionality Reduction Diffusion PCA Development and Anisotropic Diffusions Examples Slide 2/22

3 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: Principal Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) Slide 3/22

4 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: PGA: analysis relative to Principal the Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) data mean Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) Slide 3/22

5 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: PGA: analysis Principal Geodesic Analysis (PGA, Fletcher et al., 04) relative to the Geodesic PCA (GPCA, Huckeman et al., 10) data mean Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) data points on non-linear manifold Slide 3/22

6 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: PGA: analysis Principal Geodesic Analysis (PGA, Fletcher et al., 04) relative to Geodesic the PCA (GPCA, Huckeman et al., 10) data mean Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) intrinsic mean µ Slide 3/22

7 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: PGA: analysis relative to Principal the Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) data mean Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) tangent space T µ M Slide 3/22

8 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: PGA: analysis relative to Principal the Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) data mean Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) projection of data point to T µ M Slide 3/22

9 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data PGA: analysis Non-Euclidean generalizations of PCA: relative to the Principal Geodesic Analysis (PGA, Fletcher et al., 04) data mean Geodesic PCA (GPCA, Huckeman et al., 10) Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces Euclidean (BS, PCA Pennec, in tangent 15) space Slide 3/22

10 Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean space that preserves structure of data Non-Euclidean generalizations of PCA: Principal Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) Slide 3/22

11 PGA: Dimensionality Reduction in Non-Linear Manifolds dim. reduction and linearizations - mappings from non-linear manifolds to low dimensional Euclidean GPCA: space that preserves structure of data HCA: Non-Euclidean generalizations of PCA: Principal Geodesic Analysis (PGA, Fletcher et al., 04) Geodesic PCA (GPCA, Huckeman et al., 10) Horizontal Component Analysis (HCA, Sommer, 13) Principal Nested Spheres ((C)PNS, Jung et al., 12) Barycentric Subspaces (BS, Pennec, 15) Slide 3/22

12 PGA, GPCA, HCA, PNS,... search for explicitly constructed parametric subspaces: geodesic sprays, geodesics, iterated development,... in general manifolds, these subspaces are not totally geodesic projections to subspaces are problematic: geodesics may be dense on tori Slide 4/22

13 Generalizing Linear Statistics Euclidean Riemannian norm x y distances d(x,y) vectors v 0 for geodesics linear subspaces geodesic sprays why are geodesics fundamental when estimating covariance? Euclidean space analogies can lead to non-local constructions to goal of this talk is to get closer to constructions defined infinitesimally Slide 5/22

14 Euclidean PCA Usual formulation: eigendecomposition (u 1,λ 1 ),...,(u d,λ d ) of sample covar. matrix C principal components: x n = U T (y n µ) Probabilistic interpretation (Tipping, Bishop, 99): latent variable model y = Wx + µ + ε, x N (0,I), ε N (0,σ 2 I) marginal distribution y N (µ,c σ ), C σ = WW T + σ 2 I MLE of W : W ML = U(Λ σ 2 I) 1/2 + rotation Λ = diag(λ 1,...,λ d ) principal components: E[x n y n ] = (W T W + σ 2 I) 1 W T ML (y n µ) Slide 6/22

15 Diffusion PCA probabilistic PCA does not explicitly use subspaces on Riemannian manifolds, the Eells-Elworthy-Malliavin construction gives a map : FM Dens(M) Diff Γ Dens(M): the image Diff (FM), the set of (normalized) densities resulting from diffusions in FM µ Γ anisotropic normal distribution with µ = Diff (x,x α) = p µ µ 0, define the log-likelihood lnl(x,x α ) = lnl(µ) = N i=1 lnp µ (y i ) Diffusion PCA: maxim. lnl(x,x α ) for (x,x α ) FM MLE of data y i under the assumption y µ Γ Slide 7/22

16 Diffusion PCA all geometric complexities are hidden in the diffusion processes no parametric subspaces, no projections to dense geodesics principal components: mean sample paths reaching y i ˆx i (t) = E[x(t) x(1) = y i ] path dependency can be integrated out x i = 1 0 d dt ˆx i(t)dt = ˆx i (1) x i provides a linear view of the data Slide 8/22

17 Diffusion PCA all geometric complexities are hidden in the diffusion processes no parametric subspaces, no projections to dense geodesics principal components: mean sample paths reaching y i ˆx i (t) = E[x(t) x(1) = y i ] path dependency can be integrated out x i = 1 0 d dt ˆx i(t)dt = ˆx i (1) x i provides a linear view of the data Slide 8/22

18 Statistical Manifold: Geometry of Γ Densities Dens(M) = {µ Ω n (M) : Fisher-Rao metric: Gµ FR (α,β) = α β M µ µ µ Γ finite dim. subset of Dens(M) properties of Diff M : FM Dens(M) µ = 1,µ > 0} naturally defined on bundle of symmetric positive T 0 2 tensors Slide 9/22

19 SDEs On Manifolds stationary driftless diffusion SDE in R n : dx t = σdw t, σ M n d diffusion field σ, infinitesimal generator σσ T curvature: stationary field/generator cannot be defined due to holonomy Slide 10/22

20 SDEs On Manifolds stationary driftless diffusion SDE in R n : dx t = σdw t, σ M n d diffusion field σ, infinitesimal generator σσ T curvature: stationary field/generator cannot be defined due to holonomy Slide 10/22

21 The Frame Bundle the manifold and frames (bases) for the tangent spaces T p M F(M) consists of pairs u = (x,x α ), x M, X α frame for T x M curves in the horizontal part of F(M) correspond to curves in M and parallel transport of frames Slide 11/22

22 The Frame Bundle the manifold and frames (bases) for the tangent spaces T p M F(M) consists of pairs u = (x,x α ), x M, X α frame for T x M curves in the horizontal part of F(M) correspond to curves in M and parallel transport of frames Slide 11/22

23 The Frame Bundle the manifold and frames (bases) for the tangent spaces T p M F(M) consists of pairs u = (x,x α ), x M, X α frame for T x M curves in the horizontal part of F(M) correspond to curves in M and parallel transport of frames Slide 11/22

24 SDEs On Manifolds: Eells-Elworthy-Malliavin construction H i, i = 1...,n horizontal vector fields on F(M): SDE in R n : SDE in F(M): H i (u) = π 1 (u i ) dx t = σ(x t )dw t, σ(x t ) M n n du t = H i (U t ) dx i t = H i (U t ) σ(x t ) i j dw j t SDE on M: π F(M) (U t ) Slide 12/22

25 SDEs On Manifolds: Eells-Elworthy-Malliavin construction H i, i = 1...,n horizontal vector fields on F(M): SDE in R n : SDE in F(M): H i (u) = π 1 (u i ) dx t = σ(x t )dw t, σ(x t ) M n n du t = H i (U t ) dx i t = H i (U t ) σ(x t ) i j dw j t SDE on M: π F(M) (U t ) Slide 12/22

26 Anisotropic Diffusions Diff (x,x α ) = π F(M) (U 1 ) stochastic development / rolling without slipping in R n, sample path increments x ti = x ti+1 x ti are normally distributed N (0,( t) 1 Σ) with log-probability ln p Σ (x t ) 1 t Formally, we can set ln p Σ (x t ) N 1 xt T i Σ 1 x ti + c i=1 1 0 ẋ t 2 Σ dt + c Slide 13/22

27 Most Probable Paths Let φ be path development and U t a heat diffusion. As N, the finite path measure pullback φ v of Wiener measure v on W([0,1],M) (cont. paths from x) (Andersson, Driver, 99) geodesics and be formally viewed as most probable paths (MPPs) for the pull-back path density, i.e. MPPs for the Euclidean R n diffusion Anisotropic case: (x t,x α,t ) path in FM, X α,t represents Σ 1/2 and defines invertible map R n T xt M. Inner product on T xt M v,w Xα,t = X 1 α,t v,x 1 α,t w R n Slide 14/22

28 Sub-Riemannian Geometry optimal control problem with nonholonomic constraints let x t = arg min c t,c 0 =x,c 1 =y 1 0 ċ t 2 X α,t dt ṽ, w HFM = X 1 α,t π (ṽ),x 1 α,t π ( w) R n on H (xt,x α,t )FM. This defines a sub-riemannian metric G on TFM and equivalent problem (x t,x α,t ) = arg min (c t,c α,t ),c 0 =x,c 1 =y 1 with constraints (ċ t,ċ α,t ) H (ct,c α,t )FM 0 (ċ t,ċ α,t ) 2 HFM dt (1) Slide 15/22

29 Evolution Equations geodesics satisfy the Hamilton-Jacobi equations ẏ k t = G kj (y t )ξ t,j, ξt,k = 1 2 G pq y k ξ t,pξ t,q in coordinates (x i ) for M, X i α for X α, and W encoding the inner product W kl = δ αβ X k α X l β : ẋ i = W ij ξ j W ih Γ j β h ξ j β, Ẋα i = Γ i α h W hj ξ j + Γ i α k W kh Γ j β ξ i = W hl Γ k δ l,i ξ hξ kδ 1 ( ) Γ h γ k,i 2 W kh Γ k δ h + Γh γ k W kh Γ k δ h,i ξ hγ ξ kδ ( ξ i α = Γ h γ k,i α W kh Γ k δ h ξ h γ ξ kδ W hl,i α Γ k δ l + W hl Γ k δ l,i α )ξ h ξ kδ 1 ( ) W hk,i 2 α ξ h ξ k + Γ h γ k W kh,i α Γ k δ h ξ h γ ξ kδ h ξ j β Slide 16/22

30 S 2 (a) cov. diag(1, 1) (b) cov. diag(2,.5) (c) cov. diag(4,.25) Slide 17/22

31 S2 (d) cov. diag(1, 1) (e) cov. diag(2,.5) (f) cov. diag(4,.25) Slide 17/22

32 Landmark LDDMM Christoffel symbols (Michelli et al. 08) Γ k ij = 1 2 g ir ( ) g kl g rs,l g sl g rk,l g rl g ks,l gsj mix of transported frame and cometric: F d M bundle of rank d linear maps R d T x M, ξ, ξ T F d M, cometric ξ, ξ = δ αβ (ξ π 1 X α)( ξ π 1 X β) + λ the whole frame need not be transported ξ, ξ g R Slide 18/22

33 Landmark LDDMM Examples Slide 19/22

34 Summary Diffusion PCA: dim. reduction or linearization by MLE of densities generated by diffusion processes infinitesimal definition, no subspaces, no projections diffusion map Diff : FM Dens(M) from Eells-Elworthy-Malliavin construction of Brownian motion MPPs for anisotropic diffusions generalize geodesics 1 Sommer: Diffusion Processes and PCA on Manifolds, Oberwolfach extended abstract, Sommer: Anisotropic Distributions on Manifolds: Template Estimation and Most Probable Paths, in preparation. Slide 20/22