Dual connections in nonparametric classical information geometry

Transcription

1 Ann Inst Stat Math 00 6: DOI 0.007/s Dual connections in nonparametric classical information geometry M. R. Grasselli Received: 8 September 006 / Revised: 9 May 008 / Published online: 5 August 008 The Institute of Statistical Mathematics, Tokyo 008 Abstract We construct an infinite-dimensional information manifold based on exponential Orlicz spaces without using the notion of exponential convergence. We then show that convex mixtures of probability densities lie on the same connected component of this manifold, and characterize the class of densities for which this mixture can be extended to an open segment containing the extreme points. For this class, we define an infinite-dimensional analogue of the mixture parallel transport and prove that it is dual to the exponential parallel transport with respect to the Fisher information. We also define α-derivatives and prove that they are convex mixtures of the extremal ±-derivatives. Keywords Information geometry Statistical manifiold Fisher metric Orlicz spaces Amari Nagaoka duality Introduction Information geometry is the branch of probability theory dedicated to provide families of probability distributions with differential geometrical structures. One then uses the tools of differential geometry in order to have a clear and intuitive picture, as well as rigor, in a variety of practical applications ranging from neural networks to statistical estimation, from mathematical finance to nonequilibrium statistical mechanics see Sollich et al. 00. M. R. Grasselli B Department of Mathematics and Statistics, McMaster University, 80 Main Street West, Hamilton, ON L8S 4K, Canada grasselli@math.mcmaster.ca

2 874 M. R. Grasselli It was just over half a century ago that the Fisher information log px,θ g ij = θ i log px,θ θ j px,θdx was independently suggested by Rao 945 and Jeffreys 946 as a Riemannian metric for a parametric statistical model { px,θ,θ = θ,...,θ n }. The Riemannian geometry of statistical models was then studied as a mathematical curiosity for some years, with an emphasis in the geodesic distances associated with the Levi Civita connection for this metric. A greater amount of attention was devoted to the subject after Efron 975 introduced the concept of statistical curvature, pointing out its importance to statistical inference, as well as implicitly using a new affine connection, which would be known as the exponential connection. This exponential connection, together with another connection, later to be called the mixture connection, were further investigated by Dawid 975. The work of several years on the geometric aspects of parametric statistical models culminated with the masterful account in Amari 985, where the whole finite dimensional differential-geometric machinery is employed, including a one-parameter family of α-connections, the essential concept of duality and the notions of statistical divergence, projections and minimization procedures. Among the successes of the research at these early stages one could single out the rigidity of the geometric structures, such as the result concerning the uniqueness of the Fisher metric with respect to monotonicity in Čencov 98 and Amari s result concerning the uniqueness of the α-connections introduced by invariant statistical divergences. The ideas were then extensively used in statistics, in particular higher order asymptotic inference and curved exponential models see Kass and Vos 997. A different line of investigation in Information Geometry took off in the nineties: the search for a fully fledge infinite dimensional manifold of probability measures. As for motivations for this quest, one had, on the practical side, the need to deal with nonparametric models in statistics, where the shape of the underlying distribution is not assumed to be known. On a more fundamental level, there was the desire of having parametric statistical manifolds defined simply as finite dimensional submanifolds of a well defined manifold of all probability measures on a sample space. The motivating idea was already in Dawid 975 and was also addressed by Amari 985. The first sound mathematical construction, however, is due to Pistone and Sempi 995.Given a probability space, F,μ, they showed how to construct a Banach manifold M of all probability measures equivalent to μ. The Banach space used as generalized coordinates was the Orlicz space L, where is an exponential Young function. In a subsequent work, Pistone and Rogantin 999 analyzed further properties of this manifold, in particular the concepts of orthogonality and submanifolds. In Sect. 3, we review their construction and present an alternative proof of the main result in Pistone and Sempi 995, namely that the collection of covering neighborhoods U p and charts e p form an affine C -atlas for M. The crux is Proposition, where we show that the image of overlapping neighborhoods under any chart e p is open in the topology of the target space L.

3 Dual connections in nonparametric classical information geometry 875 The next step in this development was the Gibilisco and Pistone 998 definition of the exponential connection as the natural connection induced by the use of L. These authors then propose a mixture connection acting on the pretangent bundle T M and prove that it is dual to the exponential connection, in the sense of duality for Banach spaces. They further define the α-connections through generalized α-embeddings and show that the formal relation between the exponential, mixture and α-connections are the same as in the parametric case, that is α = + α e + α m. We argue, however, that neither of these two results duality for the exponential and mixture connection and α-connections as convex mixture of them is a proper generalization of the corresponding parametric ones, the reason being twofold. First, Banach space duality is not Amari Nagaoka duality. The latter refers to a metric being preserved by the joint action of two parallel transports, which are then said to be dual see 5. Secondly, all the α-connections in the parametric case act on the tangent bundle, whereas in Gibilisco and Pistone 998 each of them acts on its own bundle-connection pair, making a formula like at least difficult to interpret. In order to address these problems, we define in Sect. 4 an isomorphism τ of tangent spaces, which satisfy the Amari Nagaoka duality relation with respect to the Fisher metric when paired with the exponential parallel transport τ. However, it turns out that our map τ can only be rigorously defined between points q and in M whose ratio is a bounded random variable. Proposition 3 then characterizes the extended convex mixtures between such points. In Sect. 5, we rearrange the definitions of Gibilisco and Pistone 998 in order to have α-derivatives all acting on the same tangent bundle, but defined only for a restricted class of tangent vectors. We then show that the desired relation holds for our definitions. We then finalize the paper by showing that the α-auto-parallel curves between two points whose ratio is a bounded function belong to the connected component Ep. Orlicz spaces We present here the aspects of the theory of Orlicz spaces that will be relevant for the construction of the information manifold. For more comprehensive accounts, as well as for the proofs of all statements in this section, the reader is referred to the monographs of Rao and Ren 99 and Krasnosel skiĭ and Rutickiĭ 96. The general theory of Orlicz spaces is developed around the concept of a Young function, that is, a convex function : R R + satisfying i x = x, x R, ii 0 = 0, iii lim x

4 876 M. R. Grasselli For applications in information geometry, it is enough to consider Young functions of the form x = x 0 φtdt, x 0, 3 where φ :[0, [0, is nondecreasing, continuous and such that φ0 = 0 and lim φx =+. Young functions of this type include the monomials x x r /r, for < r <, and the following examples arising in information geometry: x = cosh x, 4 x = e x x, 5 3 x = + x log + x x 6 in the sequel,, and 3 will always refer to these three particular functions, with other symbols being used to denote generic Young functions. When a Young function is given in the form 3 we can define its complementary conjugate function as the Young function given by y = y 0 ψtdt, y 0, 7 where ψ is the inverse of φ. One can verify that, 3 and x r /r, x s /s, with r + s =, are examples of complementary pairs. For a general Young function, the complementary function is given less constructively by y = sup{x 0 : x y x}. 8 There are many different ways of introducing a partial order on the class of Young functions. A particularly straightforward one is to say that a Young function is stronger than another Young function, denoted by, if there exist a constant a > 0 such that x ax, x x 0, 9 for some x 0 0 depending on a. For example, one can verify that x 3 x r r x s s 0 whenever < r s <. Two Young functions and are said to be equivalent if and, that is, if there exist real numbers 0 < c c < and x 0 0 such that c x x c x, x x 0.

5 Dual connections in nonparametric classical information geometry 877 For example, the functions and are equivalent, both being of exponential type. Now let,, P be a probability space. The Orlicz class associated with a Young function is defined as L P = { f : R, measurable : } f dp <. Since P is a finite measure, the Banach space L,, P of essentially bounded random variables is easily seen to be a subset of L P for any Young function.it is easy to see that L P is a convex set and that h L P and f h imply that f L P. However, in general, L P is not a vector space, which leads to the definition of the Orlicz space associated with a Young function as { } L P = f : R, measurable : αf dp <, for some α>0, 3 furnished with the Luxembourg norm see Rao and Ren 99, p. 67 { } f N f = inf k > 0 : dp. 4 k or with the equivalent Orlicz norm see Rao and Ren 99, p. 6 { } f = sup fg dp : g L P, gdp, 5 where is the complementary Young function to. We observe for later use that f dp iffn f seerao and Ren 99, p. 54. A key ingredient in the analysis of Orlicz spaces is the generalized Hölder inequality see Rao and Ren 99, p. 58. If and are complementary Young functions, f L P, g L P, then fg dp N f N g. 6 It follows that each element f L P defines a continuous linear functional on L P, so that if we denote its topological dual by L we obtain the continuous injection L L for any pair of complementary Young functions. If then there exist a constant k such that N kn and therefore L P L P see Rao and Ren 99, p. 55. For instance, due to 0 we obtain that for < r s < L L s L r L 3 L, 7 where L r, r denote the usual Lebesgue spaces on,, P, which coincide with the Orlicz space defined by the Young functions x r /r, r. If two Young functions are equivalent, then the Orlicz spaces associated with them are isomorphic,

6 878 M. R. Grasselli that is, they coincide as sets and have equivalent norms. For example, we have that L P = L P. 3 The Pistone Sempi information manifold We start by reviewing the construction of an infinite dimensional information manifold along the lines of Pistone and Sempi 995; Pistone and Rogantin 999; Gibilisco and Pistone 998. Consider the set M of all densities of probability measures equivalent to a reference measure μ, that is, M M,,μ={f : R, measurable : f > 0 a.e. and f dμ = }. For each point p M, letl p be the exponential Orlicz space with norm N p over the probability space,, pdμ and consider its closed subspace of p-centred random variables B p ={u L p : updμ = 0} 8 as the coordinate Banach space. In probabilistic terms, the set L p corresponds to random variables whose moment generating function with respect to the probability pdμ is finite on a neighborhood of the origin see Pistone and Sempi 995, proposition.3. In statistics this are exactly the random variables used to define the one dimensional exponential model pt associated with a point p M and a random variable u: pt = etu p, t ε, ε. 9 Z p tu In particular, if we denote by V p the unit ball in B p, then it follows that the moment generating functional Z p u = eu pdμ is finite on V p see Pistone and Sempi 995, proposition.4. The underlying idea for the Pistone Sempi manifold is to parametrize the neighborhoods around points p M by all possible one dimensional exponential models passing through p. As a preliminary result, we mention that if two densities p and q are connected by a one dimensional exponential model, then L p = L q see Pistone and Rogantin 999, proposition 5. Pistone and Sempi define the inverse of a local chart around p M as e p : V p M u eu p. 0 Z p u

7 Dual connections in nonparametric classical information geometry 879 Denote by U p the image of V p under e p. We verify that e p is a bijection from V p to U p, since e u Z p u p = ev Z p v p implies that u v is a constant random variable, which must vanish, since both u,v have zero p-expectation. Then let e p be the inverse of e p on U p. One can check that e p : U p B p q log q q log pdμ. p p and also that, for any p, M, the transition functions are given by e e p : e p U p U p e u u + log U p U p p u + log p dμ. The main result of Pistone and Sempi 995 is to show that the charts defined above lead to a well-defined infinite dimensional manifold. The crucial part of the proof is to show that, for any two points p, M, the image of the overlapping neighborhoods U p U p under e p is open in the topology of the model space B p. To do so they introduce a topology induced by the notion of exponential convergence, with respect to which the sets U p U p are open, and then show that e p is sequentially continuous from exponential convergence to L -convergence. In what follows, we bypass the use of exponential convergence and present a direct proof that the Pistone and Sempi construction yields a Banach manifold. We first need to establish the following proposition. Proposition For any p, M, thesete p U p U p is open in the topology of B p. Proof Suppose that q U p U p for some p, M. Then we can write it as q = for some u V p.using, we find e q = u + log Since e q V p, we have that p N e q = N u + log eu Z p u p, u + log p dμ. p u + log p dμ <.

8 880 M. R. Grasselli Consider an open ball of radius r around u = e p q e p U p U p in the topology of B p, that is, consider the set A r ={v B p : N p v u <r} and let r be small enough so that A r v A r under e p is V p. Then the image in M of each point q = e p v = ev Z p v p. We claim that q U p U p if r is sufficiently small. Indeed, applying e to it we find e q = v + log p so N e q N v u + N u + log +N v u dμ v + log p dμ, p N v u + N e q = N v u + N e q + v u,p K, u + log p dμ + N v u dμ where K = N and we use the notation,p for the L -norm. As we have seen in the previous section, it follows from the growth properties of that there exists c > 0 such that f,p c N f. Moreover, since L p = L since both p and are connected to q by one dimensional exponential models it follows that there exists a constant c > 0 such that N f c N p f. Therefore, the previous inequality becomes N e q c N p v u + N e q + c c KN p v u = c + c K N p v u + N e q. Thus, if we choose r < N e q, c + c K

9 Dual connections in nonparametric classical information geometry 88 we will have that N e q < which proves the claim. What we have just proved is that e p U p U p consists entirely of interior points in the topology of B p, and is therefore open in B p. We then have that the collection {U p, e p, p M} satisfies the three axioms for being a C -atlas for M see Lang 995, p. 0. Moreover, since for each connect component all the spaces B p are isomorphic as topological vector spaces, we can say that M is a C -manifold modeled on B p. As usual, the tangent space at each point p M can be abstractly identified with B p. A concrete realization has been given in Pistone and Rogantin 999, proposition, namely each curve through p M is tangent to a one-dimensional exponential e model tu Z p tu p,sowetakeu as the tangent vector representing the equivalence class of such a curve. Finally, given a point p M, the connected component of M containing p coincides with the maximal exponential model obtained from p see Pistone and Sempi 995, Theorem 4.: { e u } Ep = Z p u p, u B p Z p, 3 where Z p ={f : Z p f < } 0. 4 The Fisher information and dual connections In the parametric version of information geometry, Amari and Nagaoka have introduced the concept of dual connections with respect to a Riemannian metric see Amari and Nagaoka 000 and the references given therein to their earlier work. For finite dimensional manifolds, any continuous assignment of a positive definite symmetric bilinear form to each tangent space determines a Riemannian metric. In infinite dimensions, we need to impose that the tangent space be self-dual and that the bilinear form be bounded. Since our tangent spaces B p are not even reflexive, let alone self-dual, we abandon the idea of having a Riemannian structure on M and propose a weaker version of duality, the duality with respect to a continuous scalar product. When restricted to finite dimensional submanifolds, the scalar product becomes a Riemannian metric and the original definition of duality is recovered. Let, p be a continuous positive definite symmetric bilinear form assigned continuously to each B p T p M. A pair of connections, are said to be dual with respect to, p if τu,τ v q = u,v p 4

10 88 M. R. Grasselli for all u,v T p M and all smooth curves γ :[0, ] M such that γ0 = p, γ = q, where τ and τ denote the parallel transports associated with and, respectively. Equivalently,, are dual with respect to, p if v s, s p = v s, s p + s, v s p 5 for all v T p M and all smooth vector fields s and s. We stress that this is not the kind of duality obtained when a connection on a bundle F is used to construct another connection on the dual bundle F as defined, for instance, in Gibilisco and Pistone 998, Definiton 6. The latter is a construction that does not involve any metric or scalar product and the two connections act on different bundles, while Amari Nagaoka duality is a duality with respect to a specific scalar product or metric, in the finite dimensional case and the dual connections act on the same bundle, the tangent bundle. The infinite dimensional generalization of the Fisher information is given by u,v p = uvpdμ, u,v B p. 6 This is clearly bilinear, symmetric and positive definite. Moreover, continuity follows from that fact that, since L p = L p L 3p, the generalized Hölder inequality gives u,v p KN p un p v, u,v B p. 7 The use of exponential Orlicz space to model the manifold naturally induces a globally flat affine connection on the tangent bundle T M, called the exponential connection and denoted by. It is defined on each connected component of the manifold M, which is equivalent to saying that its parallel transport is defined between points connected by an exponential model. If q and are two such points, then the exponential parallel transport is given by τ q : T q M T q M u u u dμ. 8 It is a well-defined isomorphism, since T q M = B q and T q M = B q are subsets of the same Orlicz space L q = L, so the exponential parallel transport just subtracts a constant from u to make it centred around the right point. We now want to obtain the dual parallel transport to τ with respect to the Fisher information, which in the parametric version of information geometry is called the mixture parallel transport since it is derived from the convex mixture of two densities. We therefore start with a result regarding such mixtures.

11 Dual connections in nonparametric classical information geometry 883 Proposition If q and are two points in U p for some p M, then belongs to Ep for all t [0, ]. Proof We begin by writing qt = tq + t q = eu Z p u p and = eu Z p u p, for some u, u V p L p. Therefore, there exist constants β > and β > such that β u pdμ < and β u pdμ <. To simplify the notation, let us define ũ = u log Z p u and ũ = u log Z p u. We want to show that, if we write eũ p = qt = teũ p + teũ p, then ũ is an element of Z p, so that u =ũ ũpdμ B p Z p and qt = eu Z p u p Ep. For this, let β = minβ,β > and observe that, on account of the inequality a + b β β a β + b β, we have that e βũ = teũ + teũ β β t β e βũ + t β e βũ. Thus e βũ pdμ β t β e βũ pdμ + β t β e βũ pdμ < 9 since both βũ and βũ are in L p. On the other hand, we observe that e βũ = te ũ + teũ β t β e βũ.

12 884 M. R. Grasselli Therefore e βũ pdμ t β e βũ pdμ <, 30 since βũ L p. But this completes the proof, since 9 and 30 together imply that ũ Z p. We now explore the possibility of extending the convex mixture between q and beyond these extreme points while maintaining positivity of qt. This depends on the relative sizes of q and, as shown in the next proposition: Proposition 3 Let q = eu Z p u p and = eu Z p u p be two points in U p. Then there exist constants ε > 0 and ε > 0 such that qt =[tq + t ] E p for all t ε, + ε if and only if u u L. Moreover, if u u L, then L 3q = L 3. Proof Suppose that qt E p for all t ε, + ε. Then since q ε 0 we have that ε q + + ε 0 q + ε ε. Similarly, since q + ε 0 we have that Therefore, the random variable u u = log + ε q ε 0 q ε + ε. q + log q p pdμ log q pdμ p is uniformly bounded from above and below. Conversely, if we have that u, u V p with u u L and K = u u, then Z p u Z p u e K q Z pu Z p u ek. We can then conclude that there exist constants 0 <ξ < and ξ > such that Then observe that for t 0wehave qt = ξ q ξ. t q + t tξ + t.

13 Dual connections in nonparametric classical information geometry 885 Therefore, provided ξ < t 0, the inequality above ensures that qt is strictly positive. Using the same notation as in the proof of Proposition, the same inequality gives e βũ pdμ = Similarly, for t 0wehave qt = t q + t Z p u β tξ + t β q p β pdμ t q + t tξ + t. e βu pdμ <. 3 Therefore, provided 0 t < ξ, this inequality shows that qt is strictly positive and that e βũ pdμ = t q + t Z p u β tξ + t β q p β pdμ e βu pdμ <. 3 Moreover, since the first part of the proof of Proposition holds provided qt is positive, we have that qt = tq + t E p for all t ξ, ξ, which completes the proof for the first statement by setting ε = ξ and ε = ξ ξ. For the second statement in the Proposition, observe that q = q + ε + ε + ε + ε 33 = q ε + ε q, + ε + ε 34 for positive densities q + ε and q ε. Therefore 3 βvq dμ< 3 βv dμ<, which implies that L 3q = L 3. Moreover, equations 33 and 34 can be used to show that the norms N 3,q and N 3, are equivalent. To see this, let u L u 3q and consider v = N 3,q u, so that N 3,q v = and consequently 3 vq dμ.

14 886 M. R. Grasselli Using 33 we see that 3 vq + ε dμ + ε 3 v dμ, + ε + ε which implies that ε + ε 3 v dμ. 35 On the other hand, it follows from convexity of 3 that ε 3 v + ε Inserting this into 35 and denoting K = +ε ε 3 v K ε + ε 3 v. gives dμ, which means that N 3, v K. Consequently we have that N 3, u = N 3, vn 3,q u KN 3,q u. Similarly, let f L f 3 and consider g = N 3, f, so that N 3, g = and consequently 3 g dμ. Using 34 we see that + ε 3 gq ε dμ + ε 3 gq dμ, + ε which implies that ε + ε 3 gq dμ. 36 Again, it follows from convexity of 3 that ε 3 g + ε ε + ε 3 g.

15 Dual connections in nonparametric classical information geometry 887 Inserting this into 36 and denoting k = +ε ε gives 3 g k q dμ, which means that N 3,q g k. Consequently we have that N 3,q f = N 3,q gn 3, f kn 3, f. 37 Proposition 4 Let q = eu Z p u p and = eu Z p u p be two points in U p such that u u L. Then the map τ q : T q M T q M u q u, 38 is an isomorphism of Banach spaces. Proof In view of Proposition 3, we have that L 3q = L 3 and that the norms N,q and N,q are equivalent, from which it follows that, for u B q, { } q u q = sup q uv, q dμ : v L 3, 3 v dμ { } = sup q uv q dμ : v L 3, N 3, v { = k sup q v u dμ : v L 3, N 3,q k v } { } k sup uf q dμ : f L 3 q, N 3,q f = k u,q <, where k is the constant appearing in 37. Thus, q u L, and since q u is centred around we have that q u B q. Therefore u q u is a well-defined continuous bijection from B q to B q whose inverse map v q v is well-defined by the same arguments.

16 888 M. R. Grasselli We denoted the map in the previous proposition by τ since it satisfies the duality relation τ u,τ v q = u u dμ, q v = = u q v dμ uvq dμ = u,v q, u,v B q, u dμ q v dμ where the third equality follows from the fact that v is centred around q. Let us now reflect on the collective results of Propositions to 4. Proposition tells us that the convex mixture of two probability densities in the same U p remains in the connected component E p of M, but not necessarily in the same neighbourhood. Proposition 3 then characterizes exactly those pairs q and for which the convex mixture can be extended beyond the extreme points while remaining in the same connected component E p. For such pairs, Proposition 4 gives an isomorphism of tangent spaces τ which satisfies the duality relation 4 with respect to the Fisher information. However, we refrain from calling τ a parallel transport, since it might fail to be well-defined when the points q and do not satisfy the conditions of Proposition 4. Nevertheless, we can still compute the derivative of τ along curves that satisfy these conditions, as is done in the next proposition. Proposition 5 Let v T p M be a tangent vector at p M and s ST M be a differentiable vector field. If there exist a differentiable curve γ : ε, ε M such that p = γ0, v = γ 0, and whose image consists entirely of points satisfying the hypotheses of Proposition 4, then D v sp := d v sp + spl 0 B p, 39 where d v sp denotes the directional derivative of s in the direction of v in the Banach space L p, and lt = logγ t. Proof For h sufficiently small, it follows from Proposition 4 that τ B γ0. The result then follows from the following calculation: γhγ 0 sγ h [ ] [ ] lim τ h 0 h γhγ 0 sγ h sγ 0 γh = lim sγ h sγ 0 h 0 h γ0 = lim h 0 h + lim h 0 h [ sγ h sγ 0 ] [ γh γ0 γ0 = d v sp + spl 0. ] sγ h

17 Dual connections in nonparametric classical information geometry 889 Despite satisfying all the usual properties of a covariant derivative, such as linearity and Leibniz rule, the differential operator D v might fail to be well-defined when no curve satisfying the conditions of Proposition 5 exists. For this reason, we simply call it the -derivative in the direction of those v for which it is well-defined. In the next section, we will see that D v is part of a one-parameter family of derivatives defined for exactly this class of tangent vectors. 5 α-derivatives In this section, we define an infinite-dimensional analogue of the α-connections introduced in the parametric case independently in Čencov 98 and Amari 985. We use the same technique proposed in Gibilisco and Pistone 998, namely exploring the geometry of spheres in the Lebesgue spaces L r, but modified in such a way that the resulting derivatives all act on sections of the tangent bundle T M. The price we pay is that our derivatives are not defined for all tangent vectors, but only those satisfying the conditions of Proposition 5. We begin with the Amari Nagaoka α-embeddings where r = α. Observe that l α : M L r μ p α p α, α [,, 40 [ /r [ l α p r = l α p dμ] r = r ] /r α p α dμ = r, so l α p S r μ, the sphere of radius r in L r μ we warn the reader that, throughout this paper, the r in S r refers to the fact that this is a sphere of radius r, while the fact that it is a subset of L r is judiciously omitted from the notation. According to Gibilisco and Pistone 998, the tangent space to S r μ at a point f is T f S r μ = where f = sgn f f r. In our case, { } g L r μ : gf dμ = 0, 4 f = l α p = rp /r 4 so that f = rp /r r = r r p /r. 43

18 890 M. R. Grasselli Therefore, the tangent space to S r μ at rp /r is T rp /r S r μ = { } g L r μ : gp /r dμ = We now look for a concrete realization of the push-forward of the map l α when the tangent space T p M is identified with B p as in the previous sections. Since d dt α p α = p α the α-push-forward can be formally implemented as dlogp, dt l α p : T p M = B p T rp /r S r μ u p α u. 45 For this to be well defined, we need to check that p α u is an element of T rp /r S r μ. Indeed, since L p L s p for all s, we have that p /r u r dμ = so p α u L r μ. Moreover p /r up /r dμ = u rf r pdμ <, updμ = 0, which verifies that p /r u T rp /r S r μ. The sphere S r μ inherits a natural connection obtained by projecting the trivial connection on L r μ the one where parallel transport is just the identity map onto its tangent space at each point. For each f S r μ, a canonical projection from the tangent space T f L r μ onto the tangent space T f S r μ can be uniquely defined, since the spaces L r μ are uniformly convex see Gibilisco and Isola 999, and is given by f : T f L r μ T f S r μ g g r r gf dμ When f = rp /r and f = r r p /r, the formula above gives f. 46 rp /r : T rp /r L r μ T rp /r S r μ g g gp /r dμ p /r. 47

19 Dual connections in nonparametric classical information geometry 89 We are now ready to introduce the α-derivative. Suppose that γ : ε, ε M is a smooth curve whose image consists entirely of points satisfying the conditions of Proposition 4. Then the α-push-forward of an arbitrary vector field s ST M along γ is l α γ t s = γt /r sγ t, while the α-push-forward of the tangent vector v = γ0 T M p is l α p v = p /r v. Therefore, the covariant derivative of l α γ t sγ t in the direction of l α p v with respect to the trivial connection on L r μ is given by lα p vl α γ t s = d γt /r sγ t dt t=0 = dlogγ t p/r /r dsγ t r dt sp + p t=0 dt = r p/r l 0sp + p /r d v s p. Using the projection rp /r to obtain a tangent vector in T rp /r S r μ we get t=0 [ rp /r lα p vl α γ t s = p /r r l 0sp + d v sp r l 0sp + d v sp ] pdμ. It then follows from Proposition 5 that r l 0sp + d v sp L p, which implies that the expression above belongs to the image of l α p. Therefore, we can pull it back to T p M using l α p, from which we obtain l α [ p rp /r lα p vl α γ t s ] = r l 0sp + d v sp r l 0sp + d v sp pdμ. 48 As this construction shows, the α-derivatives can be rigorously defined on the tangent bundle T M as follows: Definition For α [,, letγ : ε, ε M be a smooth curve such that p = γ0 and whose image consists entirely of points satisfying the conditions of Proposition 4. The α-derivative of a differentiable vector field s ST M in the direction of v = γ 0 is given by D α v s p = l α [ p rp /r lα p vl α γ t s ]. 49

20 89 M. R. Grasselli Before we proceed, observe that since sγ t B γt for each t ε, ε, we have d sγ tγ tdμ = 0 dt dsγ t γtdμ = sγ t dγt dμ dt dt dsγ t dlogγ t γtdμ = sγ t γtdμ. dt dt In particular, for t = 0, we get d v s ppdμ = s p l0 pdμ 50 Inserting this relation into 48 with α =, corresponding to r =, leads to D v s p = d v sp + spl 0, 5 which coincides with 39. Recall that the covariant derivative associated with the exponential parallel transport 8 was computed in Gibilisco and Pistone 998, proposition 5 as v s p = d v sp d v sppdμ. 5 The next proposition shows that the relation between the exponential connection and the α-derivatives just defined is the same as in the parametric case. Its proof resembles the calculation in the last pages of Gibilisco and Pistone 998, except that all our derivatives act on the same bundle, whereas in Gibilisco and Pistone 998 each one is defined on its own bundle-connection pair. Proposition 6 The exponential connection and the α-derivatives on T M satisfy D α = + α + α D. 53 Proof Let lt = logγ t with γ, s, p and v as in definition. Inserting 50 into 49 gives D α v s p = r l 0sp + d v sp + r d v sppdμ [ ] + α = d v sp d v sp

21 Dual connections in nonparametric classical information geometry Auto-parallel curves α + = + α [d v sp + spl 0] v s p + α D v s p. We now investigate some of the auto-parallel curves associated with the derivatives introduced in the previous sections. First observe that a one-dimensional exponential model of the form qt = etu Z p tu p, u B p, t ε, ε, which obviously belong to the connected component E p, is an auto-parallel curve for, since its tangent vector field st = dt d log qt p according to Pistone and Rogantin 999, proposition satisfies qt st qt = d tu log Z dt p tu E qt = d dt log Z ptu + E qt d tu log Z dt p tu d dt log Z ptu = 0. Next observe that for q and satisfying the conditions of Proposition 4, a mixture model of the form qt = tq + t, q, U p, t [0, ], which belongs to the connected component E p according to Proposition, is an autoparallel curve for D, since the tangent vector field st = dt d log qt p satisfies [ D st qt = d e p q t dt log tq ] + t p + d [ log tq ] + t d dt p dt [log tq + t ] = d [ ] p q dt tq + t p p q q + tq + t p tq + t q q = + = 0. tq + t tq + t The next theorem establishes the corresponding result for the α-derivatives.

22 894 M. R. Grasselli Proposition 7 For α,, theα-auto-parallel curves between two of points q and in U p satisfying the conditions of Proposition 4, forsomep M, belongs to the connected component E p. Proof Using the same notation as in Proposition, we have that the α-auto-parallel curve connecting q, Ep is the pull back of the arc of great circle connecting their images f = l α q and f = l α on the sphere S r μ.nowiftf + t f is the straight line connecting f and f in L r μ, then for each fixed t [0, ] the corresponding point on the sphere of radius r is f t = r kt [tf + t f ], 54 where kt = tf + t f r. Let us write its inverse image with respect to the α-embedding as eũ p = l α f t = kt r [tf + t f ] r, for some random variable ũ. Following the argument in proposition, we see that e βũ = p β kt βr [tf + t f ] βr [ ] r βr [t βr e βũ + t βr e βũ ], kt so that e βũ pdμ <, 55 since both βu and βu are in L p. Furthermore, e βũ = p β kt βr [tf + t f ] βr [ ] kt βr e βũ, tr so that e βũ pdμ <, 56 which together with 55, imply that ũ Z p. To complete the proof we can define u =ũ ũpdμ, 57

23 Dual connections in nonparametric classical information geometry 895 to obtain that u B p and l α f t = eu Z p u p Ep Further developments We have seen that using L as the coordinate space for the infinite-dimensional information manifold leads to a well-defined isomorphism τ between the tangent spaces B q and B q whenever the difference of their log-likelihoods u and u is bounded. Moreover, this isomorphism is dual to the exponential parallel transport with respect to the generalized Fisher metric. The next step in our program is to show that the Kullback Leibler relative entropy is the statistical divergence associated with the dualistic triple τ,τ, g see Amari and Nagaoka 000. In the same vein, since our interpolation family of α-derivatives satisfy the same convex mixture structure as in finite dimensions, we are led to the study of the infinite-dimensional analogues of the α-divergences. The completion of this circle of ideas would be an infinite-dimensional generalization of the projection theorems obtained by Amari in the finite dimensional case. Namely, one seeks to prove that, given a point p M and an α-flat submanifold S, then the point q S with minimal α-divergence from p is obtained by projecting p orthogonally with respect to the Fisher metric onto S following a αgeodesic. An equally ambitious result to be pursued is the infinite-dimensional analogue of Centsov s theorem, which would characterize the generalized Fisher metric as the unique continuous scalar product on M which is reduced by Markov morphisms on the tangent space. Acknowledgments I would like to acknowledge the financial support from CAPES, Brazil, and of an ORS scholarship from the British government received while I was a PhD student at King s College London, where this research was initiated. It is a pleasure to thank R.F. Streater for suggesting the theme and thoroughly discussing the results obtained. I am also grateful to P. Gibilisco and G. Pistone for estimulating discussions during the Information Geometry and Applications conference held in Pescara, July 00. Finally, I am indebted to H. Nagaoka for pointing out a crucial mistake in the previous version of this paper, and to two anonymous referees for marked improvements leading to the current version. References Amari, S.-I Differential-geometrical methods in statistics. New York: Springer. Amari, S.-I., Nagaoka, H Methods of information Geometry. Providence, RI: American Mathematical Society. Translated from the 993 Japanese original by Daishi Harada. Čencov, N.N. 98. Statistical decision rules and optimal inference. Providence, RI: American Mathematical Society. Translation from the Russian edited by Lev J. Leifman. Dawid, A.P On the concepts of sufficiency and ancillarity in the presence of nuisance parameters. Journal of the Royal Statistical Society B, 37, Efron, B Defining the curvature of a statistical problem with applications to second order efficiency. Annals of Statistics, 3, With a discussion by C. R. Rao, Don A. Pierce, D. R. Cox,

24 896 M. R. Grasselli D. V. Lindley, Lucien LeCam, J. K. Ghosh, J. Pfanzagl, Neils Keiding, A. P. Dawid, Jim Reeds and with a reply by the author. Gibilisco, P., Isola, T Connections on statistical manifolds of density operators by geometry of noncommutative L p -spaces. Infinite Dimensional Analysis Quantum Probability and Related Topics,, Gibilisco, P., Pistone, G Connections on non-parametric statistical manifolds by Orlicz space geometry. Infinite Dimensional Analysis Quantum Probability and Related Topics,, Grasselli, M.R. 00. Classical and quantum information geometry. Ph.D. thesis, King s College, London. Jeffreys, H An invariant form for the prior probability in estimation problems. Proceedings of Royal Society A, 86, Kass, R.E., Vos, P.W Geometrical foundations of asymptotic inference. New York: Wiley- Interscience. Krasnosel skiĭ, M.A., Rutickiĭ, J.B. 96. Convex functions and Orlicz spaces. Groningen: P. Noordhoff. Lang, S Differential and Riemannian manifolds 3rd ed.. New York: Springer. Murray, M.K., Rice, J.W Differential geometry and statistics. London: Chapman & Hall. Pistone, G. 00. New ideas in nonparametric estimation. In P. Sollich, et al. Eds., Disordered and complex systems. American Institute of Physics. AIP Conference Proceedings 553. Pistone, G., Rogantin, M.P The exponential statistical manifold: Mean parameters, orthogonality and space transformations. Bernoulli, 5, Pistone, G., Sempi, C An infinite-dimensional geometric structure on the space of all the probability measures equivalent to a given one. Annals of Statistics, 3, Rao, C.R Information and accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37, 8 9. Rao, M.M., Ren, Z.D. 99. Theory of Orlicz spaces. New York: Marcel Dekker. Sollich, P., et al. Eds. 00. Disordered and complex systems. American Institute of Physics. AIP Conference Proceedings 553.