DUAL CONNECTIONS IN NONPARAMETRIC CLASSICAL INFORMATION GEOMETRY

Transcription

1 DUAL CONNECTIONS IN NONPARAMETRIC CLASSICAL INFORMATION GEOMETRY M. R. GRASSELLI Department of Mathematics and Statistics McMaster University 1280 Main Street West Hamilton ON L8S 4K1, Canada NONPARAMETRIC INFORMATION GEOMETRY Abstract. We show how to obtain the mixture connection in an infinite dimensional information manifold and prove that it is dual to the exponential connection with respect to the Fisher information. We also define the α-connections and prove that they are convex mixtures of the extremal (±1)-connections. Key words and phrases: Information geometry, statistical manifiold, Fisher metric, Orlicz spaces, Amari-Nagaoka duality 1. Introduction Information geometry is the branch of probability theory dedicated to provide families of probability distributions with differential geometrical structures. One then uses the tools of differential geometry in order to have a clear and intuitive picture, as well as rigor, in a variety of practical applications ranging from neural networks to statistical es- 1

2 timation, from mathematical finance to nonequilibrium statistical mechanics (see Sollich et al. (2001)). It was just over half a century ago that the Fisher information log p(x, θ) log p(x, θ) (1.1) g ij = p(x, θ)dx θ i θ j was independently suggested by Rao (1945) and Jeffreys (1946) as a Riemannian metric for a parametric statistical model {p(x, θ), θ = (θ 1,..., θ n )}. The Riemannian geometry of statistical models was then studied as a mathematical curiosity for some years, with an emphasis in the geodesic distances associated with the Levi-Civita connection for this metric. A greater amount of attention was devoted to the subject after Efron (1975) introduced the concept of statistical curvature, pointing out its importance to statistical inference, as well as implicitly using a new affine connection, which would be known as the exponential connection. This exponential connection, together with another connection, later to be called the mixture connection, were further investigated by Dawid (1975). The work of several years on the geometric aspects of parametric statistical models culminated with the Amari (1985) masterful account, where the whole finite dimensional differentialgeometric machinery is employed, including a one-parameter family of α-connections, the essential concept of duality and the notions of statistical divergence, projections and minimization procedures. Among the successes of the research at these early stages one could single out the rigidity of the geometric structures, like in the Čencov (1982) result concerning the uniqueness of the Fisher metric with respect to monotonicity and Amari s result concerning the uniqueness of the α-connections as introduced by invariant statistical divergences. The ideas were then extensively used in statistics, in particular higher order asymptotic inference and curved exponential models (see Kass and Vos (1997)). 2

3 A different line of investigation in Information Geometry took off in the nineties: the search for a fully-fledge infinite dimensional manifold of probability measures. As for motivations for this quest, one had, on the practical side, the need to deal with nonparametric models in statistics, where the shape of the underlying distribution is not assumed to be known. On a more fundamental level, there was the desire of having parametric statistical manifolds defined simply as finite dimensional submanifolds of a well defined manifold of all probability measures on a sample space. The motivating idea was already in Dawid (1975) and was also addressed by Amari (1985). The first sound mathematical construction, however, is due to Pistone and Sempi (1995). Given a probability space (, F, µ), they showed how to construct a Banach manifold M of all probability measures equivalent to µ. The Banach space used as generalised coordinates was the Orlicz space L Φ 1, where Φ 1 is an exponential Young function. In a subsequent work, Pistone and Rogantin (1999) analysed further properties of this manifold, in particular the concepts of orthogonality and submanifolds. In section 3, we review their construction and present an alternative proof of the main result in Pistone and Sempi (1995), namely that the collection of covering neighborhoods U p and charts e 1 p form an affine C atlas for M. The crux is proposition 3.1, where we show that the image of overlapping neighborhoods under any chart e 1 p is open in the topology of the target space L Φ 1. The next step in this development was the Gibilisco and Pistone (1998) definition of the exponential connection as the natural connection induced by the use of L Φ 1. These authors then propose a mixture connection acting on the pretangent bundle T M and prove that it is dual to the exponential connection, in the sense of duality for Banach spaces. They further define the α-connections through generalised α-embeddings and show that the formal relation between the exponential, mixture and α-connections are 3

4 the same as in the parametric case, that is (1.2) (α) = 1 + α 2 (e) + 1 α (m). 2 We argue, however, that neither of these two results (duality for the exponential and mixture connection and α-connections as convex mixture of them) is a proper generalisation of the corresponding parametric ones, the reason being twofold. First, Banach space duality is not Amari-Nagaoka duality. The latter refers to a metric being preseverd by the joint action of two parallel transports, which are then said to be dual (see (4.2)). Secondly, all the α-connections in the parametric case act on the tangent bundle, whereas in Gibilisco and Pistone (1998) each of them acts on its own bundle-connection pair, making a formula like (1.2) at least difficult to interpret. In order to solve these problems we adopted in Grasselli (2001) the smaller space M Φ as generalized coordinates (following Pistone (2001)) and showed how to meaningfully introduce the mixture connection on the tangent bundle T M. In the present paper we strengthen this result by showing that the same can be done using the original L Φ spaces as coordinates. This considerably enlarges its domain of applicability, since the space L Φ correspond to random variables with finite moment generating function on an interval around the origin, whereas M Φ correspond to random variables with finite moment generating function for all t R. We then prove that the mixture and exponential connections are dual in the sense of Amari and Nagaoka. In section 5 we show that these definitions reduce to the familiar ones for finite dimensional submanifolds and that exponential and mixture families are geodesic for the exponential and mixture connections respectively. We then move to the subject of α-connections in section 6, where we again rearrange the definitions of Gibilisco and Pistone (1998) in order to have them all acting on the same bundle and with the desired relation between them, the exponential and the mixture 4

5 connections still holding. We finalize the paper by showing that, like the ±1 geodesics, the α geodesics between any two points in the same connected component E(p) of M are also contained in E(p). 2. Orlicz Spaces We present here the aspects of the theory of Orlicz spaces that will be relevant for the construction of the information manifold. For more comprehensive accounts, as well as for the proofs of all statements in this section, the reader is referred to the monographs of Rao and Ren (1991) and Krasnosel skiĭ and Rutickiĭ (1961). The general theory of Orlicz spaces is developed around the concept of a Young function, that is, a convex function Φ : R R + satisfying (i) Φ(x) = Φ( x), x R, (ii) Φ(0) = 0, (iii) lim x Φ(x) = +. For applications in information geometry, it is enough to consider Young functions of the form (2.1) Φ(x) = x 0 φ(t)dt, x 0, where φ : [0, ) [0, ) is nondecreasing, continuous and such that φ(0) = 0 and lim φ(x) = +. Young functions of this type include the monomials x x r /r, for 1 < r <, and the following examples arising in information geometry: (2.2) (2.3) (2.4) Φ 1 (x) = cosh x 1, Φ 2 (x) = e x x 1, Φ 3 (x) = (1 + x ) log(1 + x ) x 5

6 (in the sequel, Φ 1,Φ 2 and Φ 3 will always refer to these three particular functions, with other symbols being used to denote generic Young functions). When a Young function Φ is given in the form (2.1) we can define its complementary (conjugate) function as the Young function Ψ given by (2.5) Ψ(y) = y 0 ψ(t)dt, y 0, where ψ is the inverse of φ. One can verify that (Φ 2, Φ 3 ) and ( x r /r, x s /s), with r 1 + s 1 = 1, are examples of complementary pairs. For a general Young function Φ, the complementary function Ψ is given less constructively by (2.6) Ψ(y) = sup{x 0 : x y Φ(x)}. There are many different ways of introducing a partial order on the class of Young functions. A particularly straightforward one is to say that a Young function Ψ 2 is stronger than another Young function Ψ 1, denoted by Ψ 1 Ψ 2, if there exist a constant a > 0 such that (2.7) Ψ 1 (x) Ψ 2 (ax), x x 0, for some x 0 0 (depending on a). For example, one can verify that (2.8) x Φ 3 x r r x s s Φ 2 whenever 1 < r s <. Two Young functions Ψ 1 and Ψ 2 are said to be equivalent if Ψ 1 Ψ 2 and Ψ 2 Ψ 1, that is, if there exist real numbers 0 < c 1 c 2 < and x 0 0 such that (2.9) Ψ 1 (c 1 x) Ψ 2 (x) Ψ 1 (c 2 x), x x 0. For example, the functions Φ 1 and Φ 2 are equivalent, both being of exponential type. 6

7 There are also several classifications of Young functions according to their growth properties. The only one we are going to need for the construction of the information manifold is the so called 2 -class. A Young function Φ : R R + satisfies the 2 - condition if (2.10) Φ(2x) KΦ(x), x x 0 0, for some constant K > 0. Examples of functions in this class are the monomials x r /r, r 1 and the function Φ 3. Now let (, Σ, P ) be a probability space. The Orlicz class associated with a Young function Φ is defined as (2.11) LΦ (P ) = { f : R, measurable : } Φ(f)dP <. Since P is a finite measure, the Banach space L (, Σ, P ) of essentially bounded random variables is easily seen to be a subset of L Φ (P ) for any Young function Φ. One can also immediately check that L Φ (P ) is a convex set and (with a bit more work) that it is a vector space when Φ satisfies the 2 -condition. The trouble is that, whenever the probability measure P is diffuse on at least one set D Σ of positive measure, the 2 - condition is also necessary for L Φ (P ) to be a vector space (see Rao and Ren (1991, page 46)). In contrast, the Orlicz space associated with a Young function Φ defined as (2.12) L Φ (P ) = { f : R, measurable : } Φ(αf)dP <, for some α > 0 is always vector space (see Rao and Ren (1991, page 49)). If Φ satisfies the 2 -condition then it coincides with its subset L Φ. Moreover, if we identify functions which differ only on sets of measure zero, then L Φ is a Banach space when furnished with the Luxembourg 7

8 norm (see Rao and Ren (1991, page 67)) (2.13) N Φ (f) = inf { k > 0 : Φ( f } k )dp 1, or with the equivalent Orlicz norm (see Rao and Ren (1991, page 61)) { } (2.14) f Φ = sup fg dp : g L Ψ (P ), Ψ(g)dP 1, where Ψ is the complementary Young function to Φ. Φ(f)dP 1 iff N Φ(f) 1 (see Rao and Ren (1991, page 54)). We observe for later use that A key ingredient in the analysis of Orlicz spaces is the generalised Hölder inequality (see Rao and Ren (1991, page 58)). If Φ and Ψ are complementary Young functions, f L Φ (P ), g L Ψ (P ), then (2.15) fg dp 2N Φ (f)n Ψ (g). It follows that each element f L Φ (P ) defines a continuous linear functional on L Ψ (P ), so that if we denote its topological dual by ( L Ψ) we obtain the continuous injection L Φ ( L Ψ) for any pair of complementary Young functions. If Ψ 2 Ψ 1 then there exist a constant k such that Ψ2 k Ψ1 and therefore L Ψ 1 (P ) L Ψ 2 (P ) (see Rao and Ren (1991, page 155)). For instance, due to (2.8) we obtain that for 1 < r s < (2.16) L Φ 2 L s L r L Φ 3 L 1, where L r, r 1 denote the usual Lebesgue spaces on (, Σ, P ), which coincide with the Orlicz space defined by the Young functions x r /r, r 1. If two Young functions are equivalent, the Banach spaces associated with them coincide as sets and have equivalent norms. For example, we denote this by L Φ 1 (P ) L Φ 2 (P ). 8

9 Now define (2.17) M Φ (P ) = { f L Φ : } Φ(kf)dP <, for all k > 0. In the next lemma, we collect important results concerning M Φ for the case of a continuous Young function vanishing only at the origin. Part (i) and the last statement of the lemma are adaptations of Rao and Ren (1991, proposition and corollary 3.4.4), while part (ii) is taken from Rao and Ren (1991, theorem 4.1.6). Lemma 2.1. Let (Φ, Ψ) be a complementary pair of Young functions, Φ continuous, Φ(x) = 0 iff x = 0. Then: (i) M Φ (P ) is the closure of L (, Σ, P ) in the L Φ (P ) norm. (ii) ( M Φ (P ) ) is isometrically isomorphic to L Ψ (P ). If, moreover, Φ satisfies the 2 -condition, then M Φ (P ) = L Φ (P ). Since Φ 3 satisfies the 2 condition, we have that M Φ 3 (P ) = L Φ 3 (P ) by the last statement in the lemma. But by part (ii) of the lemma (M Φ 3 (P )) is isometrically isomorphic to L Φ 2 (P ), since (Φ 2, Φ 3 ) is a complementary pair. Therefore, since Φ 1 and Φ 2 are equivalent, we have (2.18) (L Φ 3 (P )) = (M Φ 3 (P )) = L Φ 2 (P ) L Φ 1 (P ), where the first two equalities correspond to isometric isomorphisms and the third to Banach space equivalence. Similarly M Φ 1 (P ) M Φ 2 (P ) = L Φ 3 (P ). 3. The Pistone-Sempi Information Manifold We start by reviewing the construction of an infinite dimensional information manifold along the lines of Pistone and Sempi (1995); Pistone and Rogantin (1999); Gibilisco 9

10 and Pistone (1998). Consider the set M of all densities of probability measures equivalent to a reference measure µ, that is, M M(, Σ, µ) = {f : R, measurable : f > 0 a.e. and fdµ = 1}. For each point p M, let L Φ 1 (p) be the exponential Orlicz space over the probability space (Σ,, pdµ) and consider its closed subspace of p-centred random variables (3.1) B p = {u L Φ 1 (p) : updµ = 0} as the coordinate Banach space. For definiteness, we choose to work with the Orlicz norm Φ1, although everything could be done with the equivalent Luxemburg norm N Φ1, and use the notation Φ1,p when it is necessary to specify the base point p. In probabilistic terms, the set L Φ 1 (p) correspond to random variables whose moment generating function with respect to the probability pdµ is finite on a neighborhood of the origin (see Pistone and Sempi (1995, proposition 2.3)). In statistics this are exactly the random variables used to define the one dimensional exponential model p(t) associated with a point p M and a random variable u: (3.2) p(t) = etu Z p (tu) p, t ( ε, ε). In particular, if we denote by V p the unit ball in B p, then it follows that the moment generating functional Z p (u) = eu pdµ is finite on V p (see Pistone and Sempi (1995, proposition 2.4)). The underlying idea for the Pistone Sempi manifold is to parametrize the neighborhoods around points p M by all possible one dimensional exponential models passing through p. As a preliminary result, we mention that if two densities p and q are connected by a one dimensional exponential model, then L Φ 1 (p) L Φ 1 (q) (see Pistone and Rogantin (1999, proposition 5)). 10

11 Pistone and Sempi define the inverse of a local chart around p M as e p : V p M (3.3) u eu Z p (u) p. Denote by U p the image of V p under e p. We verify that e p is a bijection from V p to U p, since e u Z p (u) p = ev Z p (v) p implies that (u v) is a constant random variable, which must vanish, since both u, v have zero p expectation. Then let e 1 p be the inverse of e p on U p. One can check that (3.4) e 1 p : U p B p q log ( ) q log p ( ) q pdµ. p and also that, for any p 1, M, the transition functions are given by (3.5) e 1 e p1 : e 1 p 1 (U p1 U p2 ) e 1 (U p1 U p2 ) ( ) p1 u u + log ( u + log p ) 1 dµ. The main result of Pistone and Sempi (1995) is to show that the charts defined above lead to a well define infinite dimensional manifold. The crucial part of the proof is to show that, for any two points p 1, M, the image of the overlapping neighborhoods U p1 U p2 under e 1 p 1 is open in the topology of the model space B p1. To do so they introduce a topology induce by the notion of exponential convergence, with respect to which the sets U p1 U p2 are open, and then show that e 1 p 1 is sequentially continuous from exponential convergence to L Φ 1 convergence. In what follows, we bypass the use of exponential convergence and present a direct proof that Pistone and Sempi s construction yields a Banach manifold. We first need to establish the following proposition. 11

12 Proposition 3.1. For any p 1, M, the set e 1 p 1 (U p1 U p2 ) is open in the topology of B p1. Proof: Suppose that q U p1 U p2 for some p 1, M. Then we can write it as for some u V p1. Using (3.5), we find e 1 (q) = u + log q = ( p1 eu Z p1 (u) p 1, ) Since e 1 (q) V p2, we have that ( ) e 1 (q) Φ1, = u + log p1 ( u + log p ) 1 dµ. ( u + log p ) 1 dµ < 1. Φ1, Consider an open ball of radius r around u = e 1 p 1 (q) e 1 p 1 (U p1 U p2 ) in the topology of B p1, that is, consider the set A r = {v B p1 : v u Φ1,p 1 < r} and let r be small enough so that A r V p1. Then the image in M of each point v A r under e p1 is q = e p1 (v) = ev Z p1 (v) p 1. We claim that q U p1 U p2 if r is sufficiently small. Indeed, applying e 1 ) e 1 ( q) = v + log ( p1 ( v + log p ) 1 dµ, to it we find so ( ) ( ( q) Φ1, v u Φ1, + u + log p1 u + log p ) 1 dµ + (v u) dµ Φ1, v u Φ1, + e 1 (q) Φ1, + 1 Φ1, v u dµ e 1 = v u Φ1, + e 1 (q) Φ1, + v u 1,p2 K, Φ1, 12

13 where K = 1 Φ1, and we use the notation 1,p2 for the L 1 ( )-norm. As we have seen in the previous section, it follows from the growth properties of Φ 1 that there exists c 1 > 0 such that f 1,p2 c 1 f Φ1,. Moreover, since L Φ 1 (p 1 ) and L Φ 1 ( ) coincide as sets and have equivalent norms (since both p 1 and are connected to q by one dimensional exponential models) it follows that there exists a constant c 2 > 0 such that f Φ1, c 2 f Φ1,p 1. Therefore, the previous inequality becomes e 1 ( q) Φ1, c 2 v u Φ1,p 1 + e 1 (q) Φ1, + c 1 c 2 K v u Φ1,p 1 = c 2 (1 + c 1 K) v u Φ1,p 1 + e 1 (q) Φ1,. Thus, is we choose we will have that r < 1 e 1 (q) Φ1,, c 2 (1 + c 1 K) e 1 ( q) Φ1, < 1 which proves the claim. What we have just proved is that e 1 p 1 (U p1 U p2 ) consists entirely of interior points in the topology of B p1, so e 1 p 1 (U p1 U p2 ) is open in B p1. We then have that the collection {(U p, e 1 p ), p M} satisfies the three axioms for being a C atlas for M (see Lang (1995, 0)). Moreover, since all the spaces B p are isomorphic as topological vector spaces, we can say that M is a C manifold modeled on B p. As usual, the tangent space at each point p M can be abstractly identified with B p. A concrete realisation has been given in Pistone and Rogantin (1999, proposition 21), namely each curve through p M is tangent to a one-dimensional exponential model e tu Z p(tu) p, so we take u as the tangent vector representing the equivalence class of such a curve. 13

14 Finally, given a point p M, the connected component of M containing p coincides with the maximal exponential model obtained from p (see Pistone and Sempi (1995, theorem 4.1)): { } e u (3.6) E(p) = Z p (u) p, u B p Z p, where Z p = {f : Z p (f) < } The Fisher Information and Dual Connections In the parametric version of information geometry, Amari and Nagaoka have introduced the concept of dual connections with respect to a Riemannian metric (see Amari and Nagaoka (2000) and the references given therein to their earlier work). For finite dimensional manifolds, any continuous assignment of a positive definite symmetric bilinear form to each tangent space determines a Riemannian metric. In infinite dimensions, we need to impose that the tangent space be self-dual and that the bilinear form be bounded. Since our tangent spaces B p are not even reflexive, let alone self-dual, we abandon the idea of having a Riemannian structure on M and propose a weaker version of duality, the duality with respect to a continuous scalar product. When restricted to finite dimensional submanifolds, the scalar product becomes a Riemannian metric and the original definition of duality is recovered. Let, p be a continuous positive definite symmetric bilinear form assigned continuously to each B p T p M. A pair of connection (, ) are said to be dual with respect to, p if (4.1) τu, τ v q = u, v p for all u, v T p M and all smooth curves γ : [0, 1] M such that γ(0) = p,γ(1) = q, 14

15 where τ and τ denote the parallel transports associated with and, respectively. Equivalently, (, ) are dual with respect to, p if (4.2) v ( s 1, s 2 p ) = v s 1, s 2 p + s 1, vs 2 p for all v T p M and all smooth vector fields s 1 and s 2. We stress that this is not the kind of duality obtained when a conection on a bundle F is used to construct another connection on the dual bundle F as defined, for instance, in Gibilisco and Pistone (1998, definiton 6). The latter is a construction that does not involve any metric or scalar product and the two connections act on different bundles, while Amar-Nagaoka duality is a duality with respect to a specific scalar product (or metric, in the finite dimensional case) and the dual connections act on the same bundle, the tangent bundle. The infinite dimensional generalisation of the Fisher information is given by (4.3) u, v p = (uv)pdµ, u, v B p. This is clearly bilinear, symmetric and positive definite. Moreover, continuity follows from that fact that, since L Φ 1 (p) L Φ 2 (p) L Φ 3 (p), the generalised Hölder inequality gives (4.4) u, v p K u Φ1,p v Φ1,p, u, v B p. The use of exponential Orlicz space to model the manifold induces naturally a globally flat affine connection on the tangent bundle T M, called the exponential connection and denoted by (1). It is defined on each connected component of the manifold M, which is equivalent to saying that its parallel transport is defined between points connected by an exponential model. If p and q are two such points, then the exponential parallel transport 15

16 is given by (4.5) τ (1) pq : T p M T q M u u uqdµ. It is well defined, since T p M = B p and T q M = B q are subsets of the same set L Φ 1 (p) L Φ 1 (q), so the exponential parallel transport just subtracts a constant from u to make it centred around the right point. We now want to define the dual connection to (1) with respect to the Fisher information. In Grasselli (2001, lemma 2.3.1) we proved a result that allowed the definiton of the mixture parallel transport for tangent vectors with absolutely continuous Orlicz norm (see Rao and Ren (1991, page 84)). A modification of that argument is presented bellow and allows for the definition of the mixture parallel transport for all tangent vectors. Proposition 4.1. Let p and q be two points in the same connected component of M. Then p q u B q, for all u B p. Proof: From the hypothesis, p and q can be connected by a one dimensional exponential model, which implies that M Φ 1 (p) M Φ 1 (q), since they are the completion of the same set L under equivalent norms. Using lemma 2.1, we find that L Φ 3 (p) (M Φ 1 (p)) (M Φ 1 (q)) L Φ 3 (q), in the sense that they are the same set furnished with equivalent norms. Therefore, there 16

17 exist a constant k such that N Φ3,p( ) kn Φ3,q( ). We then obtain that p { q u Φ 1,q = sup { = sup { = k sup { k sup p q uv qdµ : v LΦ 3 (q), p q uv p ( v q u k = k u Φ1,p <, } Φ 3 (v)qdµ 1 } qdµ : v LΦ 3 (p), N Φ3,q(v) 1 ) qdµ : v L Φ 3 (q), N Φ3,q(v) 1} uf pdµ : f L Φ 3 (p), N Φ3,p(f) 1 } so that p q u LΦ 1 (q). But since p q u is centred around q we have that p q u B q. We can then define the mixture connection on T M, as τ ( 1) pq : T p M T q M (4.6) u p q u, for p and q in the same connected component of M. We notice that it is also globally flat and prove the following result. Theorem 4.1. The connections (1) and ( 1) are dual with respect to the Fisher information. Proof: We have that τ (1) u, τ ( 1) v q = = = u uqdµ, p q v u p q vqdµ ( uvpdµ = u, v p, u, v B p, q ) uqdµ p q vqdµ where, to go from the second to the third line above, we used that v is centred around p. 17

18 5. Covariant Derivatives and Geodesics We begin this section recalling that the covariant derivative for the exponential connection has been computed in Gibilisco and Pistone (1998, proposition 25) and found to be (5.1) ( (1) v s ) (p) = (d v s)(p) E p ((d v s)(p)), where s S(T M) is a differentiable vector field, v T p M is a tangent vector at p, E p ( ) denotes the expected value with respect to the measure pdµ and d v s denotes the directional derivative in L Φ 1 (p) of s composed with the patch e p as a map between Banach spaces. We first notice that this gives the usual covariant derivative for the exponential connection in parametric information geometry (see Murray and Rice (1993, pages )). For if {θ 1,..., θ n } is a coordinate system in a finite dimensional submanifold of M we can put v = θ i and s = log p θ j as the 1-representation (see Amari and Nagaoka (2000, page 41)) of the vector field θ j, then (5.1) reduces to ( ) (1) (p) = 2 log p θ i θ j θ i θ E j p ( 2 log p θ i θ j which is the classical finite dimensional result. Accordingly, parametric exponential models are flat submanifolds of M. We can also verify that one-dimensional exponential models of the form ), q(t) = are geodesics for (1), since if s(t) = d dt ( etu Z p (tu) p log q(t) p ) is the vector field tangent to q(t) at 18

19 each point t (according to Pistone and Rogantin (1999, proposition 21)), then (5.1) gives ( ) ( ) (1) q(t) s(t) (q(t)) = d2 d 2 dt (tu log Z p(tu)) E 2 q(t) dt (tu log Z p(tu)) ( 2 ) = d2 d 2 dt log Z p(tu) + E 2 q(t) dt log Z p(tu) 2 = 0. As we emphasised in the previous section, the definition given in Gibilisco and Pistone (1998, definition 22) for the mixture connection differs from ours (due to the different concepts of duality employed), so we have to compute its covariant derivative according to the definition given here, if only to have the notation right. Proposition 5.1. Let γ : ( ε, ε) M be a smooth curve such that p = γ(0) and v = (e 1 p γ) (0) and let s S(T M) be a differentiable vector field. Then (5.2) ( ( 1) v s ) (p) = (d v s)(p) + s(p)l (0), where l(t) = log(γ(t)). Proof: ( ( 1) v s ) ( (p) = ( 1) (e 1 1 = lim h 0 h 1 = lim h 0 h ) s (γ(0)) p γ) (0) [ ] τ ( 1) γ(h)γ(0) s(γ(h)) s(γ(0)) [ ] γ(h) s(γ(h)) s(γ(0)) γ(0) 1 = lim [s(γ(h)) s(γ(0))] + lim h 0 h h 0 = (d v s)(p) + s(p)l (0). [ ] 1 γ(h) γ(0) s(γ(h)) h γ(0) Again this reduces to the parametric result for the case of submanifolds of M. Put v = θ i and s = log p θ j in (5.2) to obtain ( θ j ( 1) θ i ) (p) = 2 log p θ i θ j + log p log p, θ i θ j 19

20 which is the classical finite dimensional result. The mixture connection owes its name to the fact that in the parametric version of information geometry a convex mixture of two densities describes a geodesic with respect to ( 1). To verify the same statement in the non-parametric case, we first need to check that a convex mixture of two points in a connected component of M remains in the same connected component. Proposition 5.2. If q 1 and q 2 are two points in E(p) for some p M, then q(t) = tq 1 + (1 t)q 2 belongs to E(p) for all t (0, 1). Proof: We begin by writing q 1 = eu 1 Z p (u 1 ) p and q 2 = eu2 Z p (u 2 ) p, for some u 1, u 2 B p L Φ 1 (p). Therefore, there exist constants α 1 and α 2 such that Φ 1(α 1 u 1 )pdµ < and Φ 1(α 2 u 2 )pdµ <, or in other words, α 1 u 1, α 2 u 2 L Φ 1 (p). But since L Φ 1 (p) is a circled solid set (see Rao and Ren (1991, page 49)), if we take α = min(α 1, α 2 ) we also have that αu 1, αu 2 L Φ 1 (p). To simplify the notation, let us define ũ 1 = u 1 log Z p (u 1 ) and ũ 2 = u 2 log Z p (u 2 ). We want to show that, if we write eũp = q(t) = teũ1 p + (1 t)eũ2 p, then ũ is an element of L Φ 1 (p), so that u = ũ ũpdµ B p 20

21 and q(t) = eu Z p (u) p E(p). We have that eũ = teũ1 + (1 t)eũ2 which implies e αũ = ( teũ1 + (1 t)eũ2 ) α 2 α ( t α e αũ 1 + (1 t) α e αũ 2 ). Thus (5.3) e αũ pdµ 2 α t α e αũ 1 pdµ + 2 α (1 t) α e αũ 2 pdµ < since both αũ 1 and αũ 2 are in L Φ 1 (p). Similarly, we observe that e αũ = ( teũ1 + (1 t)eũ2 ) α 1 = (teũ1 + (1 t)e ũ 2) α 1 t α e = αũ t α e αũ 1. 1 Therefore (5.4) e αũ pdµ t α e αũ 1 pdµ <, since αũ 1 L Φ 1 (p). But this completes the proof, since (5.3) and (5.4) together imply that ũ L Φ 1 (p). We can now verify that a family of the form q(t) = tq 1 + (1 t)q 2, t (0, 1) 21

22 is a geodesic for ( 1). Let s(t) = d dt ( log q(t) p ) be the vector field tangent to q(t) at each point t (see Pistone and Rogantin (1999, proposition 21)), then (5.2) gives ( ) [ ( 1) s(t) (q(t)) = d2 (e 1 p q) (t) dt 2 log tq ] 1 + (1 t)q 2 p [ log tq 1 + (1 t)q 2 p ] ] d dt [log tq 1 + (1 t)q 2 ] + d dt = d [ p (q 1 q 2 ) dt tq 1 + (1 t)q 2 p ( ) p q 1 q 2 (q1 q 2 ) + tq 1 + (1 t)q 2 p tq 1 + (1 t)q 2 ( ) 2 ( ) 2 (q1 q 2 ) (q1 q 2 ) = + tq 1 + (1 t)q 2 tq 1 + (1 t)q 2 = α-connections In this section, we define the infinite dimensional analogue of the α-connections introduced in the parametric case independently in Čencov (1982) and Amari (1985). We use the same technique proposed in Gibilisco and Pistone (1998), namely exploring the geometry of spheres in the Lebesgue spaces L r, but modified in such a way that the resulting connections all act on the tangent bundle T M. We begin with the Amari-Nagaoka α-embeddings l α : M L r (µ) (6.1) p 2 1 α p 1 α 2, α ( 1, 1), where r = 2 1 α. Observe that [ 1/r [ l α (p) r = l α (p) dµ] r = ( ) r ] 1/r 2 1 α p 1 α 2 dµ = r, so l α (p) S r (µ), the sphere of radius r in L r (µ) (we warn the reader that, throughout 22

23 this paper, the r in S r refers to the fact that this is a sphere of radius r, while the fact that it is a subset of L r is judiciously omitted from the notation). According to Gibilisco and Pistone (1998), the tangent space to S r (µ) at a point f is (6.2) T f S r (µ) = where f = sgn(f) f r 1. In our case, { } g L r (µ) : gf dµ = 0, (6.3) f = l α (p) = rp 1/r so that (6.4) f = ( rp 1/r) r 1 = r r 1 p 1 1/r. Therefore, the tangent space to S r (µ) at rp 1/r is (6.5) T rp 1/rS r (µ) = { } g L r (µ) : gp 1 1/r dµ = 0. We now look for a concrete realization of the push-forward of the map l α when the tangent space T p M is identified with B p as in the previous sections. Since ( ) d 2 dt 1 α p 1 α 2 = p 1 α 2 the push-forward can be formally implemented as d log p, dt (l α ) (p) : T p M = B p T rp 1/rS r (µ) (6.6) u p 1 α 2 u. For this to be well defined, we need to check that p 1 α 2 u is an element of T rp 1/rS r (µ). Indeed, since L Φ 1 (p) L s (p) for all s > 1, we have that p 1/r u r dµ = u r pdµ <, 23

24 so p 1 α 2 u L r (µ). Moreover which verifies that p 1/r u T rp 1/rS r (µ). p 1/r up 1 1/r dµ = updµ = 0, The sphere S r (µ) inherits a natural connection obtained by projecting the trivial connection on L r (µ) (the one where parallel transport is just the identity map) onto its tangent space at each point. For each f S r (µ), a canonical projection from the tangent space T f L r (µ) onto the tangent space T f S r (µ) can be uniquely defined, since the spaces L r (µ) are uniformly convex (see Gibilisco and Isola (1999)), and is given by (6.7) Π f : T f L r (µ) T f S r (µ) ) g g (r r gf dµ f. When f = rp 1/r and f = r r 1 p 1 1/r, the formula above gives (6.8) Π rp 1/r : T rp 1/rL r (µ) T rp 1/rS r (µ) ( ) g g gp 1 1/r dµ p 1/r. We are now ready to define the α-connections. In what follows, is used to denote the trivial connection on L r (µ). Definition 6.1. For α ( 1, 1), let γ : ( ε, ε) M be a smooth curve such that p = γ(0) and v = γ(0) and let s S(T M) be a differentiable vector field. The α-connection on T M is given by [ (6.9) ( α v s) (p) = (l α ) 1 (p) Π ] rp 1/r (lα) (p) v(l α ) (γ(t)) s. 24

25 A formula like (6.9) deserves a more wordy explanation. We take the vector field s and push it forward along the curve γ to obtain (l α ) (γ(t)) s. Then we take its covariant derivative with respect to the trival connection in the direction of (l α ) (p) v, the pushforward of the tangent vector v. The result is a vector in T rp 1/rL r (µ), so we use the canonical projection Π to obtain a vector in T rp 1/rS r (µ). Finally, since this belongs to the image of (l α ) (p) v (see the proof of the next theorem for a justification of this statement) we pull it back to T p M using (l α ) 1 (p). The next theorem shows that the relation between the exponential, the mixture and the α-connections just defined is the same as in the parametric case. Its proof resembles the calculation in the last pages of Gibilisco and Pistone (1998), except that all our connections act on the same bundle, whereas in Gibilisco and Pistone (1998) each one is defined on its own bundle-connection pair. Theorem 6.1. The exponential, mixture and α-covariant derivatives on T M satisfy (6.10) α = 1 + α 2 (1) + 1 α ( 1). 2 Proof: Let l(t) = log(γ(t)) with γ, s, p and v as in definition 6.1. Before explicitly computing the derivatives in (6.9), observe that since s(γ(t)) B γ(t) for each t ( ε, ε), we have d s(γ(t))γ(t)dµ = 0 dt ds(γ(t)) γ(t)dµ = dt ds(γ(t)) γ(t)dµ = dt s(γ(t)) dγ(t) dµ dt s(γ(t)) d log(γ(t)) γ(t)dµ. dt 25

26 In particular, for t = 0, we get (6.11) (d v s) (p)pdµ = s(p) l(0)pdµ We can now look more closely at (6.9). It reads ( α v s) (p) = p [Π 1/r ] rp 1/r p vγ(t) 1/r s(γ(t)) 1/r = p [Π 1/r d ( rp 1/r γ(t) 1/r s(γ(t)) ) ] dt t=0 ( )] 1 = p [Π 1/r d log(γ(t)) rp 1/r p1/r 1/r ds(γ(t)) r dt s(p) + p t=0 dt ( )] t=0 1 = p [Π 1/r rp 1/r r p1/r l (0)s(p) + p 1/r (d v s) (p) = 1 ( ) 1 r l (0)s(p) + (d v s)(p) r l (0)s(p) + (d v s)(p) pdµ. At this point we make use of (6.11) in the integrand above to obtain ( α v s) (p) = 1 r l (0)s(p) + (d v s)(p) + ( 1 + α = 2 = 1 + α ( (1) v s ) (p) + 1 α 2 2 ( ) 1 r 1 ) [(d v s)(p) E p ((d v s)(p))] + ( ( 1) v s ) (p). (d v s)(p)pdµ ( 1 α 2 ) [(d v s)(p) + s(p)l (0)] A direct application of (4.2) gives the following corollary. Corollary 6.1. The connections α and α are dual with respect to the Fisher information, p. The previous theorem shows that the α-connections are well defined objects at the level of covariant derivatives. We conclude this paper with an analogue of proposition 5.2, establishing that their parallel transport is well defined on M. Proposition 6.1. The connected component E(p) contains the α-geodesic between any two of its points. 26

27 Proof: Using the same notation as in proposition 5.2, we have that the α geodesic connecting q 1, q 2 E(p) is the pull back of the arc of great circle connecting their images f 1 = l α (q 1 ) and f 2 = l α (q 2 ) on the sphere S r (µ). Now if tf 1 + (1 t)f 2 is the straight line connecting f 1 and f 2 in L r (µ), then for each fixed t [0, 1] the corresponding point on the sphere of radius r is (6.12) f(t) = r k(t) [tf 1 + (1 t)f 2 ], where k(t) = tf 1 + (1 t)f 2 r. Let us write its inverse image with respect to the α embedding as eũp = (l α ) 1 (f(t)) = 1 k(t) r [tf 1 + (1 t)f 2 ] r, for some random variable ũ. Following the argument in proposition 5.2, we see that e αũ 1 = p α k(t) [tf αr 1 + (1 t)f 2 ] αr [ ] αr 2r [t αr e αu 1 + (1 t) αr e αu 2 ], k(t) so that (6.13) e αũ pdµ <, since both αu 1 and αu 2 are in L Φ 1 (p). Furthermore, e αũ = p α k(t) αr [tf 1 + (1 t)f 2 ] αr [ ] αr k(t) e αu 1, tr so that (6.14) e αũ pdµ <, which together with (6.13), imply that ũ L Φ 1. To complete the proof we can define (6.15) u = ũ ũpdµ, 27

28 to obtain that u B p and (6.16) (l α ) 1 (f(t)) = eu Z p (u) p E(p). 7. Further developments We have seen that using L Φ 1 as the coordinate space for the infinite dimensional information manifold leads to a well defined mixture connection which is dual to the exponential connection with respect to the generalised Fisher metric. Both connections are flat, and the next natural step in our programme is to prove that the Kullback- Leibler relative entropy is the statistical divergence associated with the dualistic triple ( (1), ( 1), g) (see Amari and Nagaoka (2000)). In the same vein, we have also shown how to obtain the interpolation family of α-covariant derivatives satisfying the same convex mixture structure as in finite dimensions, which prompts us to the study of the infinite dimensional analogues of the α-divergences. The completion of this circle of ideas would be an infinite dimensional generalisation of the projection theorems obtained by Amari in the finite dimensional case. Namely, one seeks to prove that, given a point p M and an α-flat submanifold S, then the point q S with minimal α-divergence from p is obtained by projecting p orthogonally (with respect to the Fisher metric) onto S following a α-geodesic. An equally ambitious result to be pursued is the infinite dimensional analogue of Centsov s theorem, which would characterise the generalised Fisher metric as the unique continuous scalar product on M which is reduced by Markov morphisms on the tangent space. 28

29 Acknowledgements I would like to acknowledge the financial suport from CAPES, Brazil, and of an ORS scholarship from the British government received while I was a PhD student at King s College London, where this research was initiated. It is a pleasure to thank R.F. Streater for suggesting the theme and thoroughly discussing the results obtained. I am also grateful to P. Gibilisco and G. Pistone for estimulating discussions during the Information Geometry and Applications conference held in Pescara, July References Amari, S.-i. (1985). Differential-geometrical Methods in Statistics. Springer-Verlag, New York. Amari, S.-i. and Nagaoka, H. (2000). Methods of Information Geometry. American Mathematical Society, Providence, RI. Translated from the 1993 Japanese original by Daishi Harada. Čencov, N. N. (1982). Statistical Decision Rules and Optimal Inference. American Mathematical Society, Providence, R.I. Translation from the Russian edited by Lev J. Leifman. Dawid, A. P. (1975). On the concepts of sufficiency and ancillarity in the presence of nuisance parameters. Journal of the Royal Statistical Society B Efron, B. (1975). Defining the curvature of a statistical problem (with applications to second order efficiency). Annals of Statistics With a discussion by C. R. Rao, Don A. Pierce, D. R. Cox, D. V. Lindley, Lucien LeCam, J. K. Ghosh, J. Pfanzagl, Neils Keiding, A. P. Dawid, Jim Reeds and with a reply by the author. 29

30 Gibilisco, P. and Isola, T. (1999). Connections on statistical manifolds of density operators by geometry of noncommutative L p -spaces. Infinite Dimensional Analysis Quantum Probability and Related Topics Gibilisco, P. and Pistone, G. (1998). Connections on non-parametric statistical manifolds by Orlicz space geometry. Infinite Dimensional Analysis Quantum Probability and Related Topics Grasselli, M. R. (2001). Classical and Quantum Information Geometry. Ph.D. thesis, King s College London. Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proceedings of Royal Society A Kass, R. E. and Vos, P. W. (1997). Geometrical Foundations of Asymptotic Inference. John Wiley & Sons Inc., New York. A Wiley-Interscience Publication. Krasnosel skiĭ, M. A. and Rutickiĭ, J. B. (1961). Convex Functions and Orlicz spaces. P. Noordhoff Ltd., Groningen. Lang, S. (1995). Differential and Riemannian Manifolds. 3rd ed. Springer-Verlag, New York. Murray, M. K. and Rice, J. W. (1993). Differential Geometry and Statistics. Chapman & Hall, London. Pistone, G. (2001). New ideas in nonparametric estimation. In Disordered and Complex Systems (P. Sollich et al., eds.). American Institute of Physics. AIP Conference Proceedings

31 Pistone, G. and Rogantin, M. P. (1999). The exponential statistical manifold: mean parameters, orthogonality and space transformations. Bernoulli Pistone, G. and Sempi, C. (1995). An infinite-dimensional geometric structure on the space of all the probability measures equivalent to a given one. Annals of Statistics Rao, C. R. (1945). Information and accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcuta Mathematical Society Rao, M. M. and Ren, Z. D. (1991). Theory of Orlicz spaces. Marcel Dekker Inc., New York. Sollich, P. et al. (eds.) (2001). Disordered and Complex Systems. American Institute of Physics. AIP Conference Proceedings