Introduction to Information Geometry

Transcription

1 Introduction to Information Geometry based on the book Methods of Information Geometry written by Shun-Ichi Amari and Hiroshi Nagaoka Yunshu Liu

2 Outline 1 Introduction to differential geometry Manifold and Submanifold Tangent vector, Tangent space and Vector field Riemannian metric and Affine connection Flatness and autoparallel 2 The Fisher metric and α-connection Exponential family Divergence and Geometric statistical inference Yunshu Liu (ASPITRG) Introduction to Information Geometry 2 / 79

3 Part I Introduction to differential geometry Yunshu Liu (ASPITRG) Introduction to Information Geometry 3 / 79

4 Basic concepts in differential geometry Basic concepts Manifold and Submanifold Tangent vector, Tangent space and Vector field Riemannian metric and Affine connection Flatness and autoparallel Yunshu Liu (ASPITRG) Introduction to Information Geometry 4 / 79

5 Manifold Manifold S Manifold: a set with a coordinate system, a one-to-one mapping from S to R n, supposed to be locally looks like an open subset of R n Elements of the set(points): points in R n, probability distribution, linear system. Figure : A coordinate system ξ for a manifold S Yunshu Liu (ASPITRG) Introduction to Information Geometry 5 / 79

6 Manifold Manifold S Definition: Let S be a set, if there exists a set of coordinate systems A for S which satisfies the condition (1) and (2) below, we call S an n-dimensional C differentiable manifold. (1) Each element ϕ of A is a one-to-one mapping from S to some open subset of R n. (2) For all ϕ A, given any one-to-one mapping ψ from S to R n, the following hold: ψ A ψ ϕ 1 is a C diffeomorphism. Here, by a C diffeomorphism we mean that ψ ϕ 1 and its inverse ϕ ψ 1 are both C (infinitely many times differentiable). Yunshu Liu (ASPITRG) Introduction to Information Geometry 6 / 79

7 Examples of Manifold Examples of one-dimensional manifold A straight line: a manifold in R 1, even if it is given in R k for k 2. Any open subset of a straight line: one-dimensional manifold A closed subset of a straight line: not a manifold A circle: locally the circle looks like a line Any open subset of a circle: one-dimensional manifold A closed subset of a circle: not a manifold. Yunshu Liu (ASPITRG) Introduction to Information Geometry 7 / 79

8 Examples of Manifold: surface of a sphere Surface of a sphere in R 3, defined by S = {(x, y, z) R 3 x 2 + y 2 + z 2 = 1}, locally it can be parameterized by using two coordinates, for example, we can use latitude and longitude as the coordinates. nd sphere(n-1 sphere): S = {(x 1, x 2,..., x n ) R n x x x2 n = 1}. Yunshu Liu (ASPITRG) Introduction to Information Geometry 8 / 79

9 Examples of Manifold: surface of a torus The torus in R 3 (surface of a doughnut): x(u, v) = ((a + b cos u)cos v, (a + b cos u)sin v, b sin u), 0 u, v < 2π. where a is the distance from the center of the tube to the center of the torus, and b is the radius of the tube. A torus is a closed surface defined as product of two circles: T 2 = S 1 S 1. n-torus: T n is defined as a product of n circles: T n = S 1 S 1 S 1. Yunshu Liu (ASPITRG) Introduction to Information Geometry 9 / 79

10 Coordinate systems for manifold Parametrization of unit hemisphere Parametrization: map of unit hemisphere into R 2 (1) by latitude and longigude; x(ϕ, θ) = (sin ϕ cos θ, sin ϕ sin θ, cos ϕ), 0 < ϕ < π/2, 0 θ < 2π (1) Yunshu Liu (ASPITRG) Introduction to Information Geometry 10 / 79

11 Coordinate systems for manifold Parametrization of unit hemisphere Parametrization: map of unit hemisphere into R 2 (2) by stereographic projections. 2u x(u, v) = ( 1 + u 2 + v 2, 2v 1 + u 2 + v 2, 1 u2 v u 2 + v 2 ) where u2 + v 2 1 (2) Yunshu Liu (ASPITRG) Introduction to Information Geometry 11 / 79

12 Examples of Manifold: colors Parametrization of color models 3 channel color models: RGB, CMYK, LAB, HSV and so on. The RGB color model: an additive color model in which red, green, and blue light are added together in various ways to reproduce a broad array of colors. The Lab color model: three coordinates of Lab represent the lightness of the color(l), its position between red and green(a) and its position between yellow and blue(b). Yunshu Liu (ASPITRG) Introduction to Information Geometry 12 / 79

13 Coordinate systems for manifold Parametrization of color models Parametrization: map of color into R 3 Examples: Yunshu Liu (ASPITRG) Introduction to Information Geometry 13 / 79

14 Submanifolds Submanifolds Definition: a submanifold M of a manifold S is a subset of S which itself has the structure of a manifold An open subset of n-dimensional manifold forms an n-dimensional submanifold. One way to construct m(<n)dimensional manifold: fix n-m coordinates. Examples: Yunshu Liu (ASPITRG) Introduction to Information Geometry 14 / 79

15 Submanifolds Examples: color models 3-dimensional submanifold: any open subset; 2-dimensional submanifold: fix one coordinate; 1-dimensional submanifold: fix two coordinates. Note: In Lab color model, we set a and b to 0 and change L from 0 to 100(from black to white), then we get a 1-dimensional submanifold. Yunshu Liu (ASPITRG) Introduction to Information Geometry 15 / 79

16 Basic concepts in differential geometry Basic concepts Manifold and Submanifold Tangent vector, Tangent space and Vector field Riemannian metric and Affine connection Flatness and autoparallel Yunshu Liu (ASPITRG) Introduction to Information Geometry 16 / 79

17 Curves and Tangent vector of Curves Curves Curve γ: I S from some interval I( R) to S. Examples: curve on sphere, set of probability distribution, set of linear systems. Using coordinate system {ξ i } to express the point γ(t) on the curve(where t I): γ i (t) = ξ i (γ(t)), then we get γ(t) = [γ 1 (t),, γ n (t)]. C Curves C : infinitely many times differentiable(sufficiently smooth). If γ(t) is C for t I, we call γ a C on manifold S. Yunshu Liu (ASPITRG) Introduction to Information Geometry 17 / 79

18 Tangent vector of Curves A tangent vector is a vector that is tangent to a curve or surface at a given point. When S is an open subset of R n, the range of γ is contained within a single linear space, hence we consider the standard derivative: γ(a) = lim h 0 γ(a + h) γ(a) h (3) In general, however, this is not true, ex: the range of γ in a color model Thus we use a more general derivative instead: γ(a) = n i=1 γ i (a)( ξ i ) p (4) where γ i (t) = ξ i γ(t), γ i (a) = d dt γi (t) t=a and ( ξ i ) p is an operator which maps f ( f ξ i ) p for given function f : S R. Yunshu Liu (ASPITRG) Introduction to Information Geometry 18 / 79

19 Tangent space Tangent space Tangent space at p: a hyperplane T p containing all the tangents of curves passing through the point p S. (dim T p (S) = dim S) n T p (S) = { c i ( ξ i ) p [c 1,, c n ] R n } i=1 Examples: hemisphere and color Yunshu Liu (ASPITRG) Introduction to Information Geometry 19 / 79

20 Vector fields Vector fields Vector fields: a map from each point in a manifold S to a tangent vector. Consider a coordinate system {ξ i } for a n-dimensional manifold, clearly i = ξ i are vector fields for i = 1,, n. Yunshu Liu (ASPITRG) Introduction to Information Geometry 20 / 79

21 Basic concepts in differential geometry Basic concepts Manifold and Submanifold Tangent vector, Tangent space and Vector field Riemannian metric and Affine connection Flatness and autoparallel Yunshu Liu (ASPITRG) Introduction to Information Geometry 21 / 79

22 Riemannian Metrics Riemannian Metrics: an inner product of two tangent vectors(d and D T p (S)) which satisfy D, D p R, and the following condition hold: Linearity : ad + bd, D p = a D, D p + b D, D p Symmetry : D, D p = D, D p Positive definiteness : If D 0 then D, D p > 0 The components {g ij } of a Riemannian metric g w.r.t. the coordinate system {ξ i } are defined by g ij = i, j, where i = ξ i. Yunshu Liu (ASPITRG) Introduction to Information Geometry 22 / 79

23 Riemannian Metrics Examples of inner product: For X = (x 1,, x n ) and Y = (y 1,, y n ), we can define inner product as X, Y 1 = X Y = n i=1 x iy i, or X, Y 2 = YMX, where M is any symmetry positive-definite matrix. For random variables X and Y, the expected value of their product: X, Y = E(XY) For square real matrix, A, B = tr(ab T ) Yunshu Liu (ASPITRG) Introduction to Information Geometry 23 / 79

24 Riemannian Metrics For unit sphere: x(ϕ, θ) = (sin ϕ cos θ, sin ϕ sin θ, cos ϕ), 0 < ϕ < π, 0 θ < 2π (5) we have: ϕ = (cos ϕ cos θ, cos ϕ sin θ, sin ϕ) θ = ( sin ϕ sin θ, sin ϕ cos θ, 0) g 11 = ϕ, ϕ, g 22 = θ, θ g 12 = g 21 = ϕ, θ = θ, ϕ If we define X, Y = YMX = 2 n i=1 x iy i, where M = ( ) g11 g (g i,j ) = 12 = g 21 g 22 ( 2 0 ) 0 2sin 2 ϕ ( ) 2 0, then 0 2 Yunshu Liu (ASPITRG) Introduction to Information Geometry 24 / 79 (6)

25 Figure : Translation of a tangent vector along a curve Yunshu Liu (ASPITRG) Introduction to Information Geometry 25 / 79 Affine connection Parallel translation along curves Let γ: [a, b] S be a curve in S, X(t) be a vector field mapping each point γ(t) to a tangent vector, if for all t [a, b] and the corresponding infinitesimal dt, the corresponding tangent vectors are linearly related, that is to say there exist a linear mapping Π p,p, such that X(t + dt) = Π p,p (X(t)) for t [a, b], we say X is parallel along γ, and call Π γ the parallel translation along γ. Linear mapping: additivity and scalar multiplication.

26 Affine connection Affine connection: relationships between tangent space at different points. Recall: Natural basis of the coordinate system [ξ i ]: ( i ) p = ( ξ i ) p : an operator which maps f ( f ξ i ) p for given function f : S R at p. Tangent space: T p (S) = { n i=1 c i ( ξ i ) p [c 1,, c n ] R n } Tangent vector(elements in Tangent space) can be represented as linear combinations of i. Tangent space T p Tangent vector X p Natural basis ( i ) p = ( ξ i ) p Yunshu Liu (ASPITRG) Introduction to Information Geometry 26 / 79

27 Affine connection If the difference between the coordinates of p and p are very small, that we can ignore the second-order infinitesimals (dξ i )(dξ j ), where dξ i = ξ i (p ) ξ i (p), then we can express difference between Π p,p (( j ) p ) and (( j ) p ) as a linear combination of {dξ 1,, dξ n }: Π p,p (( j ) p ) = ( j ) p i,k (dξ i (Γ k ij) p ( k ) p ) (7) where {(Γ k ij ) p; i, j, k = 1,, n} are n 3 numbers which depend on the point p. From X(t) = n i=1 Xi (t)( i ) p and X(t + dt) = n i=1 (Xi (t + dt)( i ) p ), we have Π p,p (X(t)) = ({X k (t) dt γ i (t)x j (t)(γ k ij) p }( k ) p ) (8) i,j,k Yunshu Liu (ASPITRG) Introduction to Information Geometry 27 / 79

28 Affine connection Π p,p (( j ) p ) = ( j ) p i,k (dξ i (Γ k ij) p ( k ) p ) Yunshu Liu (ASPITRG) Introduction to Information Geometry 28 / 79

29 Connection coefficients(christoffel s symbols): (Γ k ij ) p Given a connection on the manifold S, the value of (Γ k ij ) p are different for different coordinate systems, it shows how tangent vectors changes on a manifold, thus shows how basis vectors changes. In Π p,p (( j ) p ) = ( j ) p (dξ i (Γ k ij) p ( k ) p ) i,k if we let Γ k ij = 0 for i, j, k = x, y, we will have Π p,p (( j ) p ) = ( j ) p Yunshu Liu (ASPITRG) Introduction to Information Geometry 29 / 79

30 Connection coefficients(christoffel s symbols): (Γ k ij ) p Given a connection on a manifold S, Γ k i,j depend on coordinate system. Define a connection which makes Γ k i,j to be zero in one coordinate system, we will get non-zero connection coefficients in some other coordinate systems. Example: If it is desired to let the connection coefficients for Cartesian Coordinates of a 2D flat plane to be zero, Γ k ij = 0 for i, j, k = x, y, we can calculate the connection coefficients for Polar Coordinates: Γ ϕ rϕ = Γ ϕ ϕr = 1 r, Γ r ϕϕ = r, and Γ k ij = 0 for all others. Yunshu Liu (ASPITRG) Introduction to Information Geometry 30 / 79

31 Connection coefficients(christoffel s symbols): (Γ k ij ) p Example(cont.): Now if we want to let the connection coefficients for Polar Coordinates to be zero, Γ k ij = 0 for i, j, k = r, ϕ, we can calculate the connection coefficients for Polar Coordinates: Γ x xx = sin2 ϕ cos ϕ r, Γ y xx = sin ϕ(1+cos2 ϕ) r, Γ x xy = Γ x yx = sin3 ϕ r Γ y yy = sin ϕ cos2 ϕ r., Γ y xy = Γ y yx = cos3 ϕ r, Γ x yy = cos ϕ(1+sin2 ϕ) r, and Yunshu Liu (ASPITRG) Introduction to Information Geometry 31 / 79

32 Affine connection Covariant derivative along curves Derivative: dx(t) X(t+dt) X(t) dt = lim dt 0 dt, what if X(t) and X(t + dt) lie in different tangent spaces? X t (t + dt) = Π γ(t+dt),γ(t) (X(t + dt)) δx(t) = X t (t + dt) X(t) = Π γ(t+dt),γ(t) (X(t + dt)) X(t) Yunshu Liu (ASPITRG) Introduction to Information Geometry 32 / 79

33 Affine connection Covariant derivative along curves We call δx(t) dt the covariant derivative of X(t): δx(t) dt X t (t + dt) X(t) = lim = Π γ(t+dt),γ(t)(x(t + dt)) X(t) dt 0 dt dt Π γ(t+dt) (X(t + dt)) = ({X k (t + dt) + dt γ i (t)x j (t)(γ k ij) γ(t) }( k ) γ(t) (10) ) i,j,k (9) δx(t) dt = i,j,k ({Ẋ k (t) + γ i (t)x j (t)(γ k ij) γ(t) }( k ) γ(t) ) (11) Yunshu Liu (ASPITRG) Introduction to Information Geometry 33 / 79

34 Affine connection Covariant derivative of any two tangent vector Covariant derivative of Y w.r.t. X, where X = n i=1 (Xi i ) and Y = n i=1 (Yi i ): X Y = i,j,k (X i { i Y k + Y j Γ k ij} k ) (12) i j = Note: ( X Y) p = Xp Y T p (S) n Γ k ij k (13) k=1 Yunshu Liu (ASPITRG) Introduction to Information Geometry 34 / 79

35 Examples of Affine connection metric connection Definition: If for all vector fields X, Y, Z T (S), Z X, Y = Z X, Y + X, Z Y. where Z X, Y denotes the derivative of the function X, Y along this vector field Z, we say that is a metric connection w.r.t. g. Equivalent condition: for all basis i, j, k T (S), k i, j = k i, j + i, k i. Property: parallel translation on a metric connection preserves inner products, which means parallel transport is an isometry. Π γ (D 1 ), Π γ (D 2 ) q = D 1, D 2 p. Yunshu Liu (ASPITRG) Introduction to Information Geometry 35 / 79

36 Examples of Affine connection Levi-Civita connection For a given connection, when Γ k ij = Γk ji hold for all i, j and k, we call it a symmetric connection or torsion-free connection. From i j = n k=1 Γk ij k, we know for a symmetric connection: i j = j i If a connection is both metric and symmetric, we call it the Riemannian connection or the Levi-Civita connection w.r.t. g. Yunshu Liu (ASPITRG) Introduction to Information Geometry 36 / 79

37 Basic concepts in differential geometry Basic concepts Manifold and Submanifold Tangent vector, Tangent space and Vector field Riemannian metric and Affine connection Flatness and autoparallel Yunshu Liu (ASPITRG) Introduction to Information Geometry 37 / 79

38 Flatness Affine coordinate system Let {ξ i } be a coordinate system for S, we call {ξ i } an affine coordinate system for the connection if the n basis vector fields i = are all parallel on S. ξ i Equivalent conditions for a coordinate system to be an affine coordinate system: i j = n k=1 (Γk ij k) = 0 for all i and j (14) Γ k ij = 0 for all i, j and k (15) Flatness S is flat w.r.t the connection : an affine coordinate system exist for the connection. Yunshu Liu (ASPITRG) Introduction to Information Geometry 38 / 79

39 Flatness Examples: i j = n k=1 (Γk ij k) = 0 for all i and j Γ k ij = 0 for all i, j and k Yunshu Liu (ASPITRG) Introduction to Information Geometry 39 / 79

40 Flatness Curvature R and torsion T of a connection R( i, j ) k = l (R l ijk l) and T( i, j ) = k (T k ij k ) (16) where R l ijk and Tk ij can be computed in the following way: R l ijk = iγ l jk jγ l ik + Γl ih Γh jk Γl jh Γh ik (17) T k ij = Γ k ij Γ k ji (18) If a connection is flat, then T=R=0; If T= 0, Γ k ij = 0 for all i, j and k, we get the symmetry connnection. Yunshu Liu (ASPITRG) Introduction to Information Geometry 40 / 79

41 Flatness Curvature Curvature R = 0 iff parallel translation does not depend on curve choice. Curvature is independent of coordinate system, under Riemannian connection, we can calculate: Curvature of 2 dimensional plane: R = 0; Curvature of 3 dimensional sphere: R = 2 r 2. Yunshu Liu (ASPITRG) Introduction to Information Geometry 41 / 79

42 Autoparallel submanifold Equivalent condition for a submanifold M of S to be autoparallel X Y T (M) for X, Y T (M) (19) a b T (M) for all a and b (20) a b = (Γ c ab c) (21) c where a = u and a b = are the basis for submanifold M w.r.t. u b coordinate system {u i }. Examples of autoparallel submanifold: Open subsets of manifold S are autoparallel; A curve with the properity that all the tangent vector are parallel Yunshu Liu (ASPITRG) Introduction to Information Geometry 42 / 79

43 Autoparallel submanifold Geodesics Geodesics(autoparallel curves): A curve with tangent vector transported by parallel translation. Examples under Riemannian connection: 2 dimensional flat plane: straight line 3 dimensional sphere: great circle Yunshu Liu (ASPITRG) Introduction to Information Geometry 43 / 79

44 Autoparallel submanifold Geodesics The geodesics with respect to the Riemannian connection are known to coincide with the shortest curve joining two points. Shortest curve: curve with the shortest length. Length of a curve γ : [a, b] S: γ = b a dγ dt dt = b a g ij γ i γ j dt (22) Yunshu Liu (ASPITRG) Introduction to Information Geometry 44 / 79

45 Part II Geometric structure of statistical models and statistical inference Yunshu Liu (ASPITRG) Introduction to Information Geometry 45 / 79

46 Motivation Motivation Consider the set of probability distributions as a manifold. Analysis the relationship between the geometric structure of the manifold and statistical estimation. Introduce concepts like metric, affine connection on statistical models and studying quantities such as distance, the tangent space (which provides linear approximations), geodesics and the curvature of a manifold. Yunshu Liu (ASPITRG) Introduction to Information Geometry 46 / 79

47 Statistical models Statistical models P(X ) = {p : X R p(x) > 0 ( x X ), p(x)dx = 1} (23) Example Normal Distribution: X = R, n = 2, ξ = [µ, σ], Ξ = {[µ, σ] < µ <, 0 < σ < } p(x, ξ) = 1 (x µ)2 exp 2πσ 2σ 2 Yunshu Liu (ASPITRG) Introduction to Information Geometry 47 / 79

48 Basic concepts The Fisher metric and α-connection Exponential family Divergence and Geometric statistical inference Yunshu Liu (ASPITRG) Introduction to Information Geometry 48 / 79

49 The Fisher information matrix Fisher information matrix G(ξ) = [g i,j (ξ)], and g i,j (ξ) = E ξ [ i l ξ j l ξ ] = i l(x; ξ) j l(x; ξ)p(x; ξ)dx where l ξ = l(x; ξ) = log p(x; ξ) and E ξ denotes the expectation w.r.t. the distribution p ξ. Motivation: Sufficient statistic and Cramér-Rao bound Yunshu Liu (ASPITRG) Introduction to Information Geometry 49 / 79

50 The Fisher information matrix Sufficient statistic Sufficient statistic: for Y = F(X), given the distribution p(x; ξ) of X, we have p(x; ξ) = q(f(x); ξ)r(x; ξ), if r(x; ξ) does not depend on ξ for all x, we say that F is a sufficient statistic for the model S. Then we can write p(x; ξ) = q(y; ξ)r(x). A sufficient statistic is a function whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate). Fisher information matrix and sufficient statistic Let G(ξ) be the Fisher information matrix of S = p(x; ξ), and G F (ξ) be the Fisher information matrix of the induced model S F = q(y; ξ), then we have G F (ξ) G(ξ) in the sense that G(ξ) = G F (ξ) G(ξ) is positive semidefinite. G(ξ) = 0 iff. F is a sufficient statistic for S. Yunshu Liu (ASPITRG) Introduction to Information Geometry 50 / 79

51 Cramér-Rao inequality Cramér-Rao inequality The variance of any unbiased estimator is at least as high as the inverse of the Fisher information. Unbiased estimator ˆξ: E ξ [ˆξ(X)] = ξ The variance-covariance matrix V ξ [ˆξ] = [v ij ξ ] where v ij ξ = E ξ[(ˆξ i (X) ξ i )(ˆξ j (X) ξ j )] Thus Cramér-Rao inequality state that V ξ [ˆξ] G(ξ) 1, and an unbiased estimator ˆξ satisfying V ξ [ˆξ] = G(ξ) 1 is called an efficient estimator. Yunshu Liu (ASPITRG) Introduction to Information Geometry 51 / 79

52 α-connection α-connection Let S = {p ξ } be an n-dimensional model, and consider the function Γ (α) ij,k which maps each point ξ to the following value: (Γ (α) ij,k ) ξ = E ξ [( i j l ξ + 1 α i l ξ j l ξ )( k l ξ )] (24) 2 where α is an arbitrary real number. We defined an affine connection (α) which satisfy: (α) i j, k = Γ (α) ij,k (25) where g =, is the Fisher metric. We call (α) the α-connection Yunshu Liu (ASPITRG) Introduction to Information Geometry 52 / 79

53 α-connection Properties of α-connection α-connection is a symmetric connection Relationship between α-connection and β-connection: Γ (β) ij,k = Γ(α) ij,k + α β E[ i l ξ j l ξ k l ξ ] 2 The 0-connection is the Riemannian connection with respect to the Fisher metric. Γ (β) ij,k = Γ(0) ij,k + β 2 E[ il ξ j l ξ k l ξ ] (α) = 1 + α 2 (1) + 1 α ( 1) 2 Yunshu Liu (ASPITRG) Introduction to Information Geometry 53 / 79

54 Basic concepts The Fisher metric and α-connection Exponential family Divergence and Geometric statistical inference Yunshu Liu (ASPITRG) Introduction to Information Geometry 54 / 79

55 Exponential family Exponential family p(x; θ) = exp[c(x) + n θ i F i (x) ψ(θ)] [θ i ] are called the natural parameters(coordinates), and ψ is the potential function for [θ i ], which can be calculated as ψ(θ) = log i=1 exp[c(x) + n θ i F i (x)]dx The exponential families include many of the most common distributions, including the normal, exponential, gamma, beta, Dirichlet, Bernoulli, binomial, multinomial, Poisson, and so on. i=1 Yunshu Liu (ASPITRG) Introduction to Information Geometry 55 / 79

56 Exponential family Exponential family Examples: Normal Distribution p(x; µ, σ) = 1 2πσ e (x µ)2 2σ 2 (26) where C(x) = 0, F 1 (x) = x, F 2 (x) = x 2, and θ 1 = µ σ 2, θ 2 = 1 2σ 2 are the natural parameters, the potential function is : ψ = (θ1 ) 2 4θ log( π θ 2 ) = µ2 2σ 2 + log( 2πσ) (27) Yunshu Liu (ASPITRG) Introduction to Information Geometry 56 / 79

57 Mixture family Mixture family p(x; θ) = C(x) + n θ i F i (x) In this case we say that S is a mixture family and [θ i ] are called the mixture parameters. e-connection and m-connection The natural parameters of exponential family form a 1-affine coordinate system(γ (1) ij,k = 0), which means the connection is 1-flat, we call the connection (1) the e-connection, and call exponential family e-flat. The mixture parameters of mixture family form a (-1)-affine coordinate = 0), which means the connection is (-1)-flat, and we call the system(γ ( 1) ij,k connection ( 1) the m-connection and call mixture family m-flat. i=1 Yunshu Liu (ASPITRG) Introduction to Information Geometry 57 / 79

58 Dual connection Dual connection Definition: Let S be a manifold on which there is given a Riemannian metric g and two affine connection and. If for all vector fields X, Y, Z T (S), Z < X, Y >=< Z X, Y > + < X, ZY > (28) hold, we say that and are duals of each other w.r.t. g and call one the dual connection of the other. Additional, we call the triple (g,, ) a dualistic structure on S. Yunshu Liu (ASPITRG) Introduction to Information Geometry 58 / 79

59 Dual connection Properties For any statistical model, the α-connection and the ( α)-connection are dual with respect to the Fisher metric. Π γ (D 1 ), Π γ(d 2 ) q = D 1, D 2 p. where Π γ and Π γ are parallel translation along γ w.r.t. and. R = 0 R = 0 where R and R are the curvature tensors of and. Yunshu Liu (ASPITRG) Introduction to Information Geometry 59 / 79

60 Dually flat spaces and dual coordinate system Dually flat spaces Let (g,, ) be a dualistic structure on a manifold S, then we have R = 0 R = 0, and if the connnection and are both symmetric(t = T = 0), then we see that -flatness and -flatness are equivalent. We call (S, g,, ) a dually flat space if both duals and are flat. Examples: Since α-connections and α-connections are dual w.r.t. Fisher metric and α-connections are symmetry, we have for any statistical model S and for any real number α S is α flat S is ( α) flat (29) Yunshu Liu (ASPITRG) Introduction to Information Geometry 60 / 79

61 Dually flat spaces and dual coordinate system Dual coordinate system For a particular -affine coordinate system [θ i ], if we choose a corresponding -affine coordinate system [η j ] such that g = i, j = δ j i where i = and j = θ i η j. Then we say the two coordinate systems mutually dual w.r.t. metric g, and call one the dual coordinate system of the other. Existence of dual coordinate system A pair of dual coordinate system exist if and only if (S, g,, ) is a dually flat space. Yunshu Liu (ASPITRG) Introduction to Information Geometry 61 / 79

62 Legendre transformations Consider mutually dual coordinate system [θ i ] and [η i ] with functions ψ : S R and ϕ : S R satisfy the following equations: i ψ = η i i ϕ = θ i g i,j = i η j = j η i = i j ψ ϕ(η) = max θ {θ i η i ψ(θ)} ψ(θ) = max η {θ i η i ϕ(η)} Yunshu Liu (ASPITRG) Introduction to Information Geometry 62 / 79

63 Legendre transformations Geometric interpretation for f (p) = max x (px f (x)): A convex function f (x) is shown in red, and the tangent line at point (x 0, f (x 0 )) is shown in blue. The tangent line intersects the vertical axis at (0, f ) and f is the value of the Legendre transform f (p 0 ), where p 0 = ḟ (x 0). Note that for any other point on the red curve, a line drawn through that point with the same slope as the blue line will have a y-intercept above the point (0, f ), showing that is indeed a maximum. Yunshu Liu (ASPITRG) Introduction to Information Geometry 63 / 79

64 Examples of Legendre transformations Examples: The Legendre transform of f (x) = 1 p x p (where 1 < p < ) is f (x ) = 1 q x q (where 1 < q < ), The Legendre transform of f (x) = e x is f (x ) = x ln x x (where x > 0), The Legendre transform of f (x) = 1 2 xt Ax is f (x ) = 1 2 x T A 1 x, The Legendre transform of f (x) = x is f (x ) = 0 if x 1, and f (x ) = if x > 1. Yunshu Liu (ASPITRG) Introduction to Information Geometry 64 / 79

65 The natural parameter and dual parameter of Exponential family For distribution p(x; θ) = exp[c(x) + n i=1 θi F i (x) ψ(θ)], [θ i ] are called the natural parameters If we define η i = E θ [F i ] = F i (x)p(x; θ)dx, we can verify [η i ] is a (-1)-affine coordinate system dual to [θ i ], we call this [η i ] the expectation parameters or the dual parameters. Yunshu Liu (ASPITRG) Introduction to Information Geometry 65 / 79

66 The natural parameter and dual parameter of Exponential family Recall: Normal Distribution p(x; µ, σ) = 1 2πσ e (x µ)2 2σ 2 (30) where C(x) = 0, F 1 (x) = x, F 2 (x) = x 2, and θ 1 = µ σ 2, θ 2 = 1 2σ 2 are the natural parameters, the potential function is : ψ = (θ1 ) 2 4θ log( π θ 2 ) = µ2 2σ 2 + log( 2πσ) (31) The dual parameter are calculated as η 1 = ψ θ 1 = µ = θ1 2θ 2, η 2 = ψ θ 2 = µ 2 + σ 2 = (θ1 ) 2 2θ 2 4(θ 2 ) 2, It has potential function: ϕ = 1 2 (1 + log( π θ 2 )) = 1 (1 + log(2π)) + 2logσ) (32) 2 Yunshu Liu (ASPITRG) Introduction to Information Geometry 66 / 79

67 Basic concepts The Fisher metric and α-connection Exponential family Divergence and Geometric statistical inference Yunshu Liu (ASPITRG) Introduction to Information Geometry 67 / 79

68 Divergences Let S be a manifold and suppose that we are given a smooth function D = D( ) : S S R satisfying for any p, q S: D(p q) 0 with equality iff. p = q) (33) Then we introduce a distance-like measure of the separation between two points. Yunshu Liu (ASPITRG) Introduction to Information Geometry 68 / 79

69 Divergence, semimetrics and metrics A distance satisfying positive-definiteness, symmetry and triangle inequality is called a metric; A distance satisfying positive-definiteness and symmetry is called semimetrics; A distance satisfying only positive-definiteness is called a divergence. Yunshu Liu (ASPITRG) Introduction to Information Geometry 69 / 79

70 Generally, we use Kullback-Leibler divergence to measurethe difference between two probability distributions p and q. KL measures the expected number of extra bits required to code samples from p when using a code based on q, rather than using a code based on p. Typically p represents the true distribution of data, observations, or a precisely calculated theoretical distribution. The measure q typically represents a theory, model, description, or approximation of p. Yunshu Liu (ASPITRG) Introduction to Information Geometry 70 / 79 Kullback-Leibler divergence Discrete random variables p and q: D KL (p q) = i p(x)log p(x) q(x) (34) Continuous random variables p and q: D KL (p q) = p(x)log p(x) dx (35) q(x)

71 Bregman divergence Bregman divergence associated with F for points p, q is : B F (x y) = F(y) F(x) (y x), F(x), where F(x) is a convex function defined on a closed convex set. Examples: F(x) = x 2, then B F (x y) = x y 2. More generally, if F(x) = 1 2 xt Ax, then B F (x y) = 1 2 (x y)t A(x y). KL divergence:if F = i x logx x, we get KullbackLeibler divergence. Yunshu Liu (ASPITRG) Introduction to Information Geometry 71 / 79

72 Canonical divergence Canonical divergence(a divergence for dually flat space) Let (S, g,, ) be a dually flat space, and {[θ i ], [η j ]} be mutually dual affine coordinate systems with potentials {ψ, ϕ}, then the canonical divergence((g, ) divergence) is defined as: D(p q) = ψ(p) + ϕ(q) θ i (p)η j (q) (36) Yunshu Liu (ASPITRG) Introduction to Information Geometry 72 / 79

73 Canonical divergence Properties: Relation between (g, ) divergence and (g, ) divergence: D (p q) = D(q p) If M is a autoparallel submanifold w.r.t. either or, then the (g M, M )-divergence D M = D M M is given by D M (p q) = D(p q) If is a Riemannian connection( = ) which is flat on S, there exist a coordinate system which is self-dual(θ i = η i ), then ϕ = ψ = 1 2 i (θi ) 2, then the canonical divergence is D(p q) = 1 {d(p, q)}2 2 where d(p, q) = i {θi (p) θ i (q)} 2 Yunshu Liu (ASPITRG) Introduction to Information Geometry 73 / 79

74 Canonical divergence Triangular relation Let {[θ i ], [η i ]} be mutually dual affine coordinate systems of a dually flat space (S, g,, ), and let D be a divergence on S. Then a necessary and sufficient condition for D to be the (g, )-divergence is that for all p, q, r S the following triangular relation holds: D(p q) + D(q r) D(p r) = {θ i (p) θ i (q)}{η i (p) η i (q)} (37) Yunshu Liu (ASPITRG) Introduction to Information Geometry 74 / 79

75 Canonical divergence Pythagorean relation Let p, q, and r be three points in S. Let γ 1 be the -geodesic connecting p and q, and let γ 2 be the -geodesic connecting q and r. If at the intersection q the curve γ 1 and γ 2 are orthogonal(with respect to the inner product g), then we have the following Pythagorean relation. D(p r) = D(p q) + D(q r) (38) Yunshu Liu (ASPITRG) Introduction to Information Geometry 75 / 79

76 Canonical divergence Projection theorem Let p be a point in S and let M be a submanifold of S which is -autoparallel. Then a necessary and sufficient condition for a point q in M to satisfy D(p q) = min r M D(p r) (39) is for the -geodesic connecting p and q to be orthogonal to M at q. Yunshu Liu (ASPITRG) Introduction to Information Geometry 76 / 79

77 Canonical divergence Examples From the definition of exponential family and mixture family, the product of exponential family are still exponential family, the sum of mixture family are still mixture family. e-flat submanifold: set of all product distributions: E 0 = {p X p X (x 1,, x N ) = N i=1 p X i (x i )} m-flat submanifold: set of joint distributions with given marginals: M 0 = {p X X\i p X(x) = q i (x i ) i {1,, N}} Yunshu Liu (ASPITRG) Introduction to Information Geometry 77 / 79

78 Canonical divergence Examples Yunshu Liu (ASPITRG) Introduction to Information Geometry 78 / 79

79 Thanks! Thanks! Question? Yunshu Liu (ASPITRG) Introduction to Information Geometry 79 / 79