COARSE-GRAINING OPEN MARKOV PROCESSES. John C. Baez. Kenny Courser

Transcription

1 COARE-GRAINING OPEN MARKOV PROCEE John C. Baez Department o Mathematics University o Caliornia Riverside CA, UA 95 and Centre or Quantum echnoloies National University o inapore inapore 7543 Kenny Courser Department o Mathematics University o Caliornia Riverside CA, UA 95 baez@math.ucr.edu, courser@math.ucr.edu March 0, 08 Abstract. Coarse-rainin is a standard method o extractin a simpler Markov process rom a more complicated one by identiyin states. Here we extend coarse-rainin to open Markov processes: that is, those where probability can low in or out o certain states called inputs and outputs. One can build up an ordinary Markov process rom smaller open pieces in two basic ways: composition, where we identiy the outputs o one open Markov process with the inputs o another, and tensorin, where we set two open Markov processes side by side. In previous work, Fon, Pollard and the irst author showed that these constructions make open Markov processes into the morphisms o a symmetric monoidal cateory. Here we o urther by constructin a symmetric monoidal double cateory where the -morphisms include ways o coarse-rainin open Markov processes. We also extend the already known black-boxin unctor rom the cateory o open Markov processes to our double cateory. Black-boxin sends any open Markov process to the linear relation between input and output data that holds in steady states, includin nonequilibrium steady states where there is a nonzero low o probability throuh the process. o extend black-boxin to a unctor between double cateories, we need to prove that black-boxin is compatible with coarse-rainin.

2 COARE-GRAINING OPEN MARKOV PROCEE. Introduction A Markov process is a stochastic model describin a sequence o transitions between states in which the probability o a transition depends only on the current state. he only Markov processes we consider here are continuous-time Markov processes with a inite set o states. hese can be drawn as labeled raphs: 3 4 where the number labelin each ede describes the probability per time o makin a transition rom one state to another. An open Markov process is a eneralization in which probability can also low in or out o certain states desinated as inputs and outputs : inputs 3 4 outputs Open Markov processes can be seen as morphisms in a cateory, since we can compose two open Markov processes by identiyin the outputs o the irst with the inputs o the second. Composition lets us build a Markov process rom smaller open parts or conversely, analyze the behavior o a Markov process in terms o its parts [4, 5, 5, ]. Coarse-rainin is a widely used method o simpliyin a Markov process by mappin its set o states X onto some smaller set X in a manner that respects, or at least approximately respects, the dynamics [, 8]. Here we introduce coarse-rainin or open Markov processes. We show how to extend this notion to the case o maps p: X X that are not surjective, obtainin a eneral concept o morphism between open Markov processes. ince open Markov processes are already morphisms in a cateory, it is natural to treat morphisms between them as morphisms between morphisms, or -morphisms. We can do this usin double cateories. hese were irst introduced by Ehresmann [3, 4], and they have lon been used in topoloy and other branches o pure mathematics [9, 0]. More recently they have been used to study open dynamical systems [] and open discrete-time Markov chains []. o, it should not be surprisin that they are also useul or open Markov processes. A -morphism in a double cateory looks like this: A C M α N B D

3 COARE-GRAINING OPEN MARKOV PROCEE 3 While a mere cateory has only objects and morphisms, here we have a ew more types o entities. We call A, B, C and D objects, and vertical -morphisms, M and N horizontal -cells, and α a -morphism. We can compose vertical -morphisms to et new vertical -morphisms and compose horizontal -cells to et new horizontal -cells. We can compose the -morphisms in two ways: horizontally by settin squares side by side, and vertically by settin one on top o the other. In a strict double cateory all these orms o composition are associative. In a pseudo double cateory, horizontal -cells compose in a weakly associative manner: that is, the associative law holds only up to an invertible -morphism, called the associator, which obeys a coherence law. his is just a quick sketch o the ideas; or ull deinitions see or example the works o Grandis and Paré [7, 8]. We construct a double cateory Mark with: (i) inite sets as objects, (ii) maps between inite sets as vertical -morphisms, (iii) open Markov processes as horizontal -cells, (iv) morphisms between open Markov processes as -morphisms. Composition o open Markov processes is only weakly associative, so this is a pseudo double cateory. he plan o the paper is as ollows. In ection we deine open Markov processes and steady state solutions o the open master equation. In ection 3 we introduce coarserainin irst or Markov processes and then open Markov processes. In ection 4 we construct the double cateory Mark described above. We prove this is a symmetric monoidal double cateory in the sense o hulman [3]. his captures the act that we can not only compose open Markov processes but also tensor them by settin them side by side. For example, i we compose this open Markov process: inputs outputs 3 with the one shown beore: inputs 3 4 outputs

4 COARE-GRAINING OPEN MARKOV PROCEE 4 we obtain this open Markov process: inputs outputs but i we tensor them, we obtain this: 3 4 inputs outputs 3 As compared with an ordinary Markov process, the key new eature o an open Markov process is that probability can low in or out. o describe this we need a eneralization o the usual master equation or Markov processes, called the open master equation [5]. In this equation the probabilities at input and output states are arbitrary speciied unctions o time, while the probabilities at other states obey the usual master equation. As a result, the probabilities are not necessarily normalized. We interpret this by sayin probability can low either in or out at both the input and the output states. I we ix constant probabilities at the inputs and outputs, there typically exist solutions o the open master equation with these boundary conditions that are constant as a unction o time. hese are called steady states. Oten these are nonequilibrium steady states, meanin that there is a nonzero net low o probabilities at the inputs and outputs. For example, probability can low throuh an open Markov process at a constant rate in a nonequilibrium steady state. In previous work, Fon, Pollard and the irst author studied the relation between probabilities and lows at the inputs and outputs that holds in steady state [5, 6]. hey called the process o extractin this relation rom an open Markov process black-boxin, since it ives a way to oret the internal workins o an open system and remember only its externally observable behavior. hey proved that black-boxin is compatible with composition and tensorin. his result can be summarized by sayin that black-boxin is a symmetric monoidal unctor. In ection 5 we show that black-boxin is compatible with morphisms between open Markov processes. o make this idea precise, we prove that black-boxin ives a map rom the double cateory Mark to another double cateory, called LinRel, which has:

5 COARE-GRAINING OPEN MARKOV PROCEE 5 (i) inite-dimensional real vector spaces U, V, W,... as objects, (ii) linear maps : V W as vertical -morphisms rom V to W, (iii) linear relations R V W as horizontal -cells rom V to W, (iv) squares R V V V V W W W W obeyin ( )R as -morphisms. Here a linear relation rom a vector space V to a vector space W is a linear subspace R V W. Linear relations can be composed in the same way as relations [3]. he double cateory LinRel becomes symmetric monoidal usin direct sum as the tensor product, but unlike Mark it is strict: that is, composition o linear relations is associative. Maps between symmetric monoidal double cateories are called symmetric monoidal double unctors []. Our main result, hm. 5.5, says that black-boxin ives a symmetric monoidal double unctor : Mark LinRel. he hardest part is to show that black-boxin preserves composition o horizontal -cells: that is, black-boxin a composite o open Markov processes ives the composite o their black-boxins. Luckily, or this we can adapt a previous arument [6]. hus, the new content o this result concerns the vertical -morphisms and especially the -morphisms, which describe coarse-rainins. An alternative approach to studyin morphisms between open Markov processes uses bicateories rather than double cateories [7, 4]. In ection 6 we use a result o hulman [3] to construct symmetric monoidal bicateories Mark and LinRel rom the symmetric monoidal double cateories Mark and LinRel. We conjecture that the black-boxin double unctor determines a unctor between these symmetric monoidal bicateories. However, double cateories seem to be a simpler ramework or coarse-rainin open Markov processes. It is worth comparin some related work. Fon, Pollard and the irst author constructed a symmetric monoidal cateory where the morphisms are open Markov processes [5, 6]. Like us, they only consider Markov processes where time is continuous and the set o states is inite. However, they ormalized such Markov processes in a slihtly dierent way than we do here: they deined a Markov process to be a directed multiraph where each ede is assined a positive number called its rate constant. In other words, they deined it to be a diaram (0, ) r E where X is a inite set o vertices or states, E is a inite set o edes or transitions between states, the unctions s, t : E X ive the source and taret o each ede, and r : E (0, ) ives the rate constant o each ede. hey explained how rom this data one can extract a matrix o real numbers (H i j ) i, j X called the Hamiltonian o the Markov process, with two amiliar properties: (i) H i j 0 i i j, s t X

6 COARE-GRAINING OPEN MARKOV PROCEE 6 (ii) i X H i j = 0 or all j X. A matrix with these properties is called ininitesimal stochastic, since these conditions are equivalent to exp(th) bein stochastic or all t 0. In the present work we skip the directed multiraphs and work directly with the Hamiltonians. hus, we deine a Markov process to be a inite set X toether with an ininitesimal stochastic matrix (H i j ) i, j X. his allows us to work more directly with the Hamiltonian and the all-important master equation dp(t) = Hp(t) dt which describes the evolution o a time-dependent probability distribution p(t): X R. Clerc, Humphrey and Pananaden have constructed a bicateory [] with inite sets as objects, open discrete labeled Markov processes as morphisms, and simulations as -morphisms. In their ramework, open has a similar meanin as it does in works listed above. hese open discrete labeled Markov processes are also equipped with a set o actions which represent interactions between the Markov process and the environment, such as an outside entity actin on a stochastic system. A simulation is then a unction between the state spaces that map the inputs, outputs and set o actions o one open discrete labeled Markov process to the inputs, outputs and set o actions o another. Another compositional ramework or Markov processes is iven by de Francesco Albasini, abadini and Walters [6] in which they construct an alebra o Markov automata. A Markov automaton is a amily o matrices with non-neative real coeicients that is indexed by elements o a binary product o sets, where one set represents a set o sinals on the let interace o the Markov automata and the other set analoously or the riht interace. Notation and erminoloy. Followin hulman, we use double cateory to mean pseudo double cateory, and use strict double cateory to mean a double cateory or which horizontal composition is strictly associative and unital. (In older literature, double cateory oten reers to a strict double cateory.) It is common to use blackboard bold or the irst letter o the name o a double cateory, and we do so here. Bicateories are written in boldace, while ordinary cateories are written in roman ont. hus, three main players in this paper are a double cateory Mark, a bicateory Mark, and a cateory Mark, all closely related.. Open Markov processes Beore explainin open Markov processes we should recall a bit about Markov processes. As mentioned in the Introduction, we use Markov process as a short term or continuous-time Markov process with a inite set o states, and we identiy any such Markov process with the ininitesimal stochastic matrix appearin in its master equation. We make this precise with a bit o terminoloy that is useul throuhout the paper. We call a unction v: X R a vector and call its values at points x X its components v x. We deine a probability distribution on X to be a vector p: X R whose components are nonneative and sum to. As usual, we use R X to denote the vector space o unctions v: X R. Given a linear operator : R X R Y we have (v) i = j X i j v j or some matrix : Y X R with entries i j. Deinition.. Given a inite set X, a linear operator H : R X R X is ininitesimal stochastic i (i) H i j 0 or i j and

7 COARE-GRAINING OPEN MARKOV PROCEE 7 (ii) i X H i j = 0 or each j X. he reason or bein interested in such operators is that when exponentiated they ive stochastic operators. Deinition.. Given inite sets X and Y, a linear operator : R X R Y is stochastic i or any probability distribution p on X, p is a probability distribution on Y. Equivalently, is stochastic i and only i (i) i j 0 or all i Y, j X and (ii) i Y i j = or each j X. I we think o i j as the probability or j X to be mapped to i Y, these conditions make intuitive sense. ince stochastic operators are those that preserve probability distributions, the composite o stochastic operators is stochastic. In Lemma 3.7 we recall that a linear operator H : R X R X is ininitesimal stochastic i and only i its exponential (th) n exp(th) = n! is stochastic or all t 0. hus, iven an ininitesimal stochastic operator H, or any time t 0 we can apply the operator exp(th): R X R X to any probability distribution p R X and et a probability distribution n=0 p(t) = exp(th)p. hese probability distributions p(t) obey the master equation dp(t) = Hp(t). dt Moreover, any solution o the master equation arises this way. All the material so ar is standard. We now turn to open Markov processes. Deinition.3. We deine a Markov process to be a pair (X, H) where X is a inite set and H : R X R X is an ininitesimal stochastic operator. We also call H a Markov process on X. Deinition.4. We deine an open Markov process to consist o inite sets X, and and injections X i toether with a Markov process (X, H). We call the the set o inputs and the set o outputs. In eneral, a diaram o this shape in any cateory: i X o o is called a cospan. he objects and are called the eet, the object X is called the apex, and the morphisms i and o are called the les. We use Finet to stand or the cateory

8 COARE-GRAINING OPEN MARKOV PROCEE 8 o inite sets and unctions. hus, an open Markov process is a cospan in Finet with injections as les and a Markov process on its apex. We oten abbreviate an open Markov process as (X, H) i o or simply i (X, H) o. Given an open Markov process we can write down an open version o the master equation, where probability can also low in or out o the inputs and outputs. o work with the open master equation we need two well-known concepts: Deinition.5. Let : A B be a map between inite sets. he linear map : R B R A sends any vector v R B to its pullback alon, iven by (v) = v. he linear map : R A R B sends any vector v R A to its pushorward alon, iven by ( (v))(b) = v(a). {a: (a)=b} Now, suppose we are iven an open Markov process (X, H) i o toether with inlows I : R R and outlows O: R R, arbitrary smooth unctions o time. We write the value o the inlow at s at time t as I s (t), and similarly or outlows and other unctions o time. We say a unction p: R R X obeys the open master equation i dp(t) = Hp(t) + i (I(t)) o (O(t)). dt his says that or any state x X the time derivative o the probability p x (t) takes into account not only the usual term rom the master equation, but also inlows and outlows. I the inlows and outlows are constant in time, a solution p o the open master equation that is also constant in time is called a steady state. More ormally: Deinition.6. Given an open Markov process i (X, H) o toether with I R and O R, a steady state with inlows I and outlows O is an element p R X such that Hp + i (I) o (O) = 0. Given p R X we call i (p) R and o (p) R the input probabilities and output probabilities, respectively. o, we deine its black- Deinition.7. Given an open Markov process i (X, H) boxin to be the set ( i (X, H) o ) R R R R consistin o all 4-tuples (i (p), I, o (p), O) where p R X is some steady state with inlows I R and outlows O R.

9 COARE-GRAINING OPEN MARKOV PROCEE 9 hus, black-boxin records the relation between input probabilities, inlows, output probabilities and outlows that holds in steady state. his is the externally observable steady state behavior o the open Markov process. It has already been shown [5, 6] that black-boxin can be seen as a unctor between cateories. Here we o urther and describe it as a double unctor between double cateories, in order to study the eect o blackboxin on morphisms between open Markov processes. 3. Morphisms o open Markov processes here are various ways to approximate a Markov process by another Markov process on a smaller set, all o which can be considered orms o coarse-rainin [8]. A common approach is to take a Markov process H on a inite set X and a surjection p: X X and create a Markov process on X. In eneral this requires a choice o stochastic section or p, deined as ollows: Deinition 3.. Given a unction p: X X between inite sets, a stochastic section or p is a stochastic operator s: R X R X such that p s = X. It is easy to check that a stochastic section or p exists i and only i p is a surjection. In Lemma 3.9 we show that iven a Markov process H on X and a surjection p: X X, any stochastic section s: R X R X ives a Markov process on X, namely H = p Hs. Experts call the matrix correspondin to p the collector matrix, and they call s the distributor matrix [8]. he names help clariy what is oin on. he collector matrix, comin rom the surjection p: X X, typically maps many states o X to each state o X. he distributor matrix, the stochastic section s: R X R X, typically maps each state in X to a linear combination o many states in X. hus, H = p Hs distributes each state o X, applies H, and then collects the results. In eneral H depends on the choice o s, but sometimes it does not: Deinition 3.. We say a Markov process H on X is lumpable with respect to a surjection p: X X i the operator p Hs is independent o the choice o stochastic section s: R X R X. his concept is not new [8]. While our deinition may be new, in hm. 3.0 we show that it is equivalent to the traditional one, and also to an even simpler one: H is lumpable with respect to p i and only i p H = H p. his equation has the advantae o makin sense even when p is not a surjection. hus, we can use it to deine a more eneral concept o morphism between Markov processes: Deinition 3.3. Given Markov processes (X, H) and (X, H ), a morphism o Markov processes p: (X, H) (X, H ) is a map p: X X such that p H = H p. here is a cateory Mark with Markov processes as objects and the morphisms as deined above, where composition is the usual composition o unctions. But what is the meanin o such a morphism? Usin Lemma 3.7 one can check that or any Markov processes (X, H) and (X, H ), and any map p: X X, we have p H = H p p exp(th) = exp(th )p or all t 0. hus, p is a morphism o Markov processes i evolvin a probability distribution on X via exp(th) and then pushin it orward alon p is the same as pushin it orward and then evolvin it via exp(th ). We can also deine morphisms between open Markov processes:

10 COARE-GRAINING OPEN MARKOV PROCEE 0 Deinition 3.4. A morphism o open Markov processes rom the open Markov process i (X, H) o to the open Markov process i (X, H ) o is a triple o unctions :, p: X X, : such that the squares in this diaram are pullbacks: i X o p i X o and p H = H p. We need the squares to be pullbacks so that in Lemma 5.3 we can black-box morphisms o open Markov processes. In Lemma 4. we show that horizontally composin these morphisms preserves this pullback property. But to do this, we need the horizontal arrows in these squares to be injections. his explains the conditions in Des..4 and 3.4. We oten abbreviate a morphism o open Markov processes as i (X, H) o p i o (X, H ) As an example, consider the ollowin open Markov process: 8 c 6 a 5 b 4 d 8 c 6 his is a way o drawin an open Markov process with state space X = {a, b, c, c, d } and ininitesimal stochastic operator H : R X R X iven by the ollowin matrix: H = Let X = {a, b, c, d}. We can deine a surjection p: X X where each element o X oes to the element o X named by the same letter. We can choose a stochastic section

11 COARE-GRAINING OPEN MARKOV PROCEE s: R X R X o p as ollows: s = 0 0 / / he ininitesimal stochastic matrix H = p Hs is iven by H = One can check that iven any other stochastic section s : R X R X we have p Hs = p Hs, so H is lumpable with respect to p. his veriication is made much easier by hm. 3.0 below. o ar we have described a morphism o Markov processes p: (X, H) (X, H ), but toether with identity unctions on the inputs and outputs this deines a morphism o open Markov processes, oin rom the above open Markov process to one that may be drawn as ollows: a b c d In ection 4 we construct a double cateory Mark with open Markov processes as horizontal -cells and morphism between these as -morphisms. his double cateory is our main object o study. First, however, we should prove the results mentioned above. For this it is helpul to recall a ew standard concepts: Deinition 3.5. A -parameter semiroup o operators is a collection o linear operators U(t): V V on a vector space V, one or each t [0, ), such that (i) U(0) = and (ii) U(s + t) = U(s)U(t) or all s, t [0, ). I V is inite-dimensional we say the collection U(t) is continuous i t U(t)v is continuous or each v V. Deinition 3.6. Let X be a inite set. A Markov semiroup is a continuous -parameter semiroup U(t): R X R X such that U(t) is stochastic or each t [0, ). Lemma 3.7. Let X be a inite set and U(t): R X R X a Markov semiroup. hen U(t) = exp(th) or a unique ininitesimal stochastic operator H : R X R X, which is iven by Hv = d dt U(t)v t=0 or all v R X. Conversely, iven an ininitesimal stochastic operator H, then exp(th) = U(t) is a Markov semiroup. Proo. his is well-known; or a proo, see or example [, hm. 7]. Lemma 3.8. Let U(t): R X R X be a dierentiable amily o stochastic operators deined or t [0, ) and havin U(0) =. hen d dt U(t) t=0 is ininitesimal stochastic.

12 COARE-GRAINING OPEN MARKOV PROCEE Proo. Let H = d dt U(t) t=0 = lim t 0 +(U(t) )/t. As U(t) is stochastic, its entries are nonneative and the column sum o any particular column is. hen the column sum o any particular column o U(t) will be 0 with the o-diaonal entries bein nonneative. hus U(t) is ininitesimal stochastic or all t 0, as is (U(t) )/t, rom which it ollows that lim t 0 +(U(t) U(0))/t = H is ininitesimal stochastic. Lemma 3.9. Let p: X X be a unction between inite sets with a stochastic section s: R X R X, and let H : R X R X be an ininitesimal stochastic operator. hen H = p Hs: R X R X is also ininitesimal stochastic. Proo. Lemma 3.7 implies that exp(th) is stochastic or all t 0. For any map p: X X the operator p : R X R X is easily seen to be stochastic, and s is stochastic by assumption. hus, U(t) = p exp(th)s is stochastic or all t 0. Dierentiatin, we conclude that d dt U(t) = d t=0 dt p exp(th)s = p exp(th)hs t=0 = p Hs t=0 is ininitesimal stochastic by Lemma 3.8. We can now ive some conditions equivalent to lumpability. he third is the most common in the literature [8] and the easiest to check in examples. It makes use o the standard basis vectors e j R X associated to the elements j o any inite set X. he surjection p: X X deines a partition on X where two states j, j X lie in the same block o the partition i and only i p( j) = p( j ). he elements o X correspond to these blocks. he third condition or lumpability says that p H has the same eect on two basis vectors e j and e j when j and j are in the same block. heorem 3.0. Let p: X X be a surjection o inite sets and let H be a Markov process on X. hen the ollowin conditions are equivalent: (i) H is lumpable with respect to p. (ii) here exists a linear operator H : R X R X such that p H = H p. (iii) p He j = p He j or all j, j X such that p( j) = p( j ). When these conditions hold there is a unique operator H : R X R X such that p H = H p, it is iven by H = p Hs or any stochastic section s o p, and it is ininitesimal stochastic. Proo. (i) = (iii). uppose that H is lumpable with respect to p. hus, p Hs: R X R X is independent o the choice o stochastic section s: R X R X. uch a stochastic section is simply an arbitrary linear operator that maps each basis vector e i R X to a probability distribution on X supported on the set { j X : p( j) = i}. hus, or any j, j X with p( j) = p( j ) = i, we can ind stochastic sections s, s : R X R X such that s(e i ) = e j and s (e i ) = e j. ince p Hs = p Hs, we have p He j = p Hs(e i ) = p Hs (e i ) = p He j. (iii) = (ii). Deine H : R X R X on basis vectors e i R X by settin H e i = p He j or any j with p( j) = i. Note that H is well-deined: since p is a surjection such j exists, and since H is lumpable, H is independent o the choice o such j. Next, note that or any j X, i we let p( j) = i we have p He j = H e i = H p e j. ince the vectors e j orm a basis or R X, it ollows that p H = H p.

13 COARE-GRAINING OPEN MARKOV PROCEE 3 (ii) = (i). uppose there exists an operator H : R X R X such that p H = H p. Choose such an operator; then or any stochastic section s or p we have p Hs = H p s = H. It ollows that p Hs is independent o the stochastic section s, so H is lumpable with respect to p. uppose that any, hence all, o conditions (i), (ii), (iii) hold. uppose that H : R X R X is an operator with p H = H p. hen the arument in the previous pararaph shows that H = p Hs or any stochastic section s o p. hus H is unique, and by Lemma 3.9 it is ininitesimal stochastic. 4. A double cateory o open Markov processes In this section we construct a symmetric monoidal double cateory Mark with open Markov processes as horizontal -cells and morphisms between these as -morphisms. ymmetric monoidal double cateories were introduced by hulman [3] and applied to various examples rom enineerin by the second author []. We reer the reader to those papers or the basic deinitions, since they are rather lon. he pieces o the double cateory Mark work as ollows: (i) An object is a inite set. (ii) A vertical -morphism : is a map between inite sets. (iii) A horizontal -cell is an open Markov process i o (X, H). In other words, it is a pair o injections i X o toether with a Markov process H on X. (iv) A -morphism is a morphism o open Markov processes i (X, H) o p i o (X, H ). In other words, it is a triple o maps, p, such that these squares are pullbacks: i X o p and H p = p H. i X o, Composition o vertical -morphisms in Mark is straihtorward. o is vertical composition o -morphisms, since we can paste two pullback squares and et a new pullback

14 COARE-GRAINING OPEN MARKOV PROCEE 4 square. Composition o horizontal -cells is a bit more subtle. Given open Markov processes i o (X, H) and i o (Y, G) U we irst compose their underlyin cospans usin a pushout: j X + Y k X Y i o i o ince monomorphisms are stable under pushout in a topos, the les o this new cospan are aain injections, as required. We then deine the composite open Markov process to be ji ko (X + Y, H G) U () where H G = j H j + k Gk. Here we use both pullbacks and pushorwards alon the maps j and k, as deined in De..5. o check that H G is a Markov process on X + Y we need to check that j H j and k Gk, and thus their sum, are ininitesimal stochastic: Lemma 4.. uppose that : X Y is any map between inite sets. I H : R X R X is ininitesimal stochastic, then H : R Y R Y is ininitesimal stochastic. Proo. Usin De..5, we see that the matrix elements o and are iven by { ( j) = i ( ) ji = ( ) i j = 0 otherwise or all i Y, j X. hus, H has matrix entries ( H ) ii = H j j. j, j : ( j)=i, ( j )=i o show that H is ininitesimal stochastic we need to show that its o-diaonal entries are nonneative and its columns sum to zero. By the above ormula, these ollow rom the same acts or H. Horizontal composition o -morphisms is even subtler: Lemma 4.. uppose that we have horizontally composable -morphisms as ollows: U i o (X, H) i o (Y, G) U p q h i o i o (X, H ) (Y, G ) U

15 COARE-GRAINING OPEN MARKOV PROCEE 5 hen there is a -morphism i 3 o 3 (X + Y, H G) U p + q h i 3 o 3 (X + U Y, H G ) whose underlyin diaram o inite sets is i j k o X X + Y Y U p + q h i j k o X X + Y Y U, where j, k, j, k are the canonical maps rom X, Y, X, Y, respectively, to the pushouts X + Y and X + Y. Proo. o show that we have deined a -morphism, we irst check that the squares in the above diaram o inite sets are pullbacks. hen we show that (p + q) (H G) = (H G )(p + q). For the irst part, it suices by the symmetry o the situation to consider the let square. We can write it as a pastin o two smaller squares: i X j X + Y p p + q i j X X + Y By assumption the let-hand smaller square is a pullback, so it suices to prove this or the riht-hand one. For this we use that act that Finet is a topos and thus an adhesive cateory [9, 0], and consider this commutative cube: i q o Y o k p X X j j X + Y p + q X + Y i Y k

16 COARE-GRAINING OPEN MARKOV PROCEE 6 By assumption the top and bottom aces are pushouts, the two let-hand vertical aces are pullbacks, and the arrows o and i are monic. In an adhesive cateory, this implies that the two riht-hand vertical aces are pullbacks as well. One o these is the square in question. o show that (p + q) (H G) = (H G )(p + q), we aain use the above cube. Because its two riht-hand vertical aces commute, we have so usin the deinition o H G we obtain By assumption we have (p + q) j = j p and (p + q) k = k q (p + q) (H G) = (p + q) ( j H j + k Gk ) = (p + q) j H j + (p + q) k Gk = j p H j + k q Gk. so we can o a step urther, obtainin p H = H p and q G = G q (p + q) (H G) = j H p j + k G q k. Because the two riht-hand vertical aces o the cube are pullbacks, Lemma 4.3 implies that Usin these, we obtain p j = j (p + q) and q k = k (p + q). (p + q) (H G) = j H j (p + q) + k G k (p + q) = ( j H j + k G k )(p + q) = (H G )(p + q) completin the proo. he ollowin lemma is reminiscent o the Beck Chevalley condition or adjoint unctors: Lemma 4.3. Given a pullback square in Finet: A B C k D h the ollowin square o linear operators commutes: R A R B h R C k R D

17 COARE-GRAINING OPEN MARKOV PROCEE 7 Proo. Choose v R B and c C. hen ( (v))(c) = (k h (v))(c) = a:(a)=c b:h(b)=k(c) v( (a)), v(b), so to show = k h it suices to show that restricts to a bijection : {a A : (a) = c} {b B : h(b) = k(c)}. On the one hand, i a A has (a) = c then b = (a) has h(b) = h( (a)) = k((a)) = k(c), so the above map is well-deined. On the other hand, i b B has h(b) = k(c), then by the deinition o pullback there exists a unique a A such that (a) = b and (a) = c, so the above map is a bijection. heorem 4.4. here exists a double cateory Mark as deined above. Proo. Let Mark 0, the cateory o objects, consist o inite sets and unctions. Let Mark the cateory o arrows, consist o open Markov processes and morphisms between these: i (X, H) o p i o (X, H ). o make Mark into a double cateory we need to speciy the identity-assinin unctor the source and taret unctors and the composition unctor hese are iven as ollows. For a inite set, u( ) is iven by u: Mark 0 Mark, s, t : Mark Mark 0, : Mark Mark0 Mark Mark. (, 0 ) where 0 is the zero operator rom R to R. For a map : between inite sets, u( ) is iven by (, 0 ) (, 0 )

18 COARE-GRAINING OPEN MARKOV PROCEE 8 he source and taret unctors s and t map a Markov process i (X, H) o to and, respectively, and they map a morphism o open Markov processes i (X, H) o p i o (X, H ) to : and :, respectively. he composition unctor maps the pair o open Markov processes to their composite i o i o (X, H) (Y, G) U ji ko (X + Y, H G) U deined as in Eq. (), and it maps the pair o morphisms o open Markov processes i o (X, H) i o (Y, G) U p q h i o i o (X, H ) (Y, G ) U to their horizontal composite as deined as in Lemma 4.. It is easy to check that u, s and t are unctors. o prove that is a unctor, the main thin we need to check is the interchane law. uppose we have our morphisms o open Markov processes as ollows: (X, H) (Y, G) U p q h (X, H ) (Y, G ) U (X, H ) (Y, G ) U p q h (X, H ) (Y, G ) U

19 COARE-GRAINING OPEN MARKOV PROCEE 9 Composin horizontally ives (X + Y, H G) U p + q h (X + U Y, H G ) (X + Y, H G ) U p + h q (X + U Y, H G ), and then composin vertically ives (X + Y, H G) U (p + q ) (p + q) h h (X + Y, H G ) U. Composin vertically ives (X, H) (Y, G) U p p q q h h (X, H ) (Y, G ) U, and then composin horizontally ives (X + Y, H G) U (p p) + ( ) (q q) h h (X + U Y, H G ). he only apparent dierence between the two results is the map in the middle: one has (p + q ) (p + q) while the other has (p p) + ( ) (q q). But these are in act the same map, so the interchane law holds. he unctors u, s, t and obey the necessary relations s u = = t u,

20 COARE-GRAINING OPEN MARKOV PROCEE 0 and the relations sayin that the source and taret o a composite behave as they should. Lastly, we have three natural isomorphisms: the associator, let unitor, and riht unitor, which arise rom the correspondin natural isomorphisms or the double cateory o inite sets, unctions, cospans o inite sets, and maps o cospans. he trianle and pentaon equations hold in Mark because they do in this simpler double cateory []. Next we ive Mark a symmetric monoidal structure. We call the tensor product addition. Given objects, Mark 0 we deine their sum + usin a chosen coproduct in Finet. he unit or this tensor product in Mark 0 is the empty set. We can similarly deine the sum o morphisms in Mark 0, since iven maps : and : there is a natural map + : + +. Given two objects in Mark : we deine their sum to be i o i o (X, H ) (X, H ) i + i o + o + (X + X, H H ) + where H H : R X +X R X +X is the direct sum o the operators H and H. he unit or this tensor product in Mark is (, 0 ) where 0 : R R is the zero operator. Finally, iven two morphisms in Mark : i o i o (X, H ) (X, H ) p p i o i o (X, H ) (X, H ) we deine their sum to be + i + i o + o (X + X, H H ) + + p + p + i + i o + o + (X + X, H H ) +. We complete the description o Mark as a symmetric monoidal double cateory in the proo o this theorem: heorem 4.5. he double cateory Mark can be iven a symmetric monoidal structure with the above properties. Proo. First we complete the description o Mark 0 and Mark as symmetric monoidal cateories. he symmetric monoidal cateory Mark 0 is just the cateory o inite sets with a chosen coproduct o each pair o inite sets providin the symmetric monoidal structure. We have described the tensor product in Mark, which we call addition, so now we need to introduce the associator, unitors, and braidin, and check that they make Mark into a symmetric monoidal cateory.

21 COARE-GRAINING OPEN MARKOV PROCEE Given three objects in Mark (X, H ) (X, H ) 3 (X 3, H 3 ) 3 tensorin the irst two and then the third results in ( + ) + 3 ((X + X ) + X 3, (H H ) H 3 ) ( + ) + 3 whereas tensorin the last two and then the irst results in + ( + 3 ) (X + (X + X 3 ), H (H H 3 )) + ( + 3 ). he associator or Mark is then iven as ollows: ( + ) + 3 ((X + X ) + X 3, (H H ) H 3 ) ( + ) + 3 a a a + ( + 3 ) (X + (X + X 3 ), H (H H 3 )) + ( + 3 ) where a is the associator in (Finet, +). I we abbreviate an object (X, H) o Mark as (X, H), and denote the associator or Mark as α, the pentaon identity says that this diaram commutes: ((X, H ) (X, H )) ((X 3, H 3 ) (X 4, H 4 )) α α (((X, H ) (X, H )) (X 3, H 3 )) (X 4, H 4 ) α (X4,H 4 ) ((X, H ) ((X, H ) (X 3, H 3 ))) (X 4, H 4 ) α (X, H ) ((X, H ) ((X 3, H 3 ) (X 4, H 4 ))) (X,H ) α (X, H ) (((X, H ) (X 3, H 3 )) (X 4, H 4 )) which is clearly true. Recall that the monoidal unit or Mark is iven by (, 0 ). he let and riht unitors or Mark, denoted λ and ρ, are iven respectively by the ollowin -morphisms: + ( + X, 0 H) + + (X +, H 0 ) + l l l r r r (X, H) (X, H) where l and r are the let and riht unitors in Finet. he let and riht unitors and associator or Mark satisy the trianle identity: ρ (X, H) (Y, G) λ ((X, H) (, 0 )) (Y, G) α (X, H) ((, 0 ) (Y, G)).

22 COARE-GRAINING OPEN MARKOV PROCEE he braidin in Mark is iven as ollows: + (X, H ) (X, H ) + b, b X,X b, + (X, H ) (X, H ) + where b is the braidin in (Finet, +). It is easy to check that the braidin in Mark is its own inverse and obeys the hexaon identity, makin Mark into a symmetric monoidal cateory. he source and taret unctors s, t : Mark Mark 0 are strict symmetric monoidal unctors, as required. o make Mark into a symmetric monoidal double cateory we must also ive it two other pieces o structure. One, called χ, says how the composition o horizontal -cells interacts with the tensor product in the cateory o arrows. he other, called µ, says how the identity-assinin unctor u relates the tensor product in the cateory o objects to the tensor product in the cateory o arrows. We now deine these two isomorphisms. Given horizontal -cells (X, H ) (Y, G ) U (X, H ) (Y, G ) U the horizontal composites o the top two and the bottom two are iven, respectively, by (X + Y, H G ) U (X + Y, H G ) U Addin the let two and riht two, respectively, we obtain + (X + X, H H ) + + (Y + Y, G G ) U + U hus the sum o the horizontal composites is + ((X + Y ) + (X + Y ), (H G ) (H G )) U + U while the horizontal composite o the sums is + ((X + X ) + + (Y + Y ), (H H ) (G G )) U + U. he required lobular -isomorphism χ between these is + ((X, H ) (Y, G )) ((X, H ) (Y, G )) U + U + ˆχ U +U + ((X, H ) (X, H )) ((Y, G ) (Y, G )) U + U where ˆχ is the bijection ˆχ: (X + Y ) + (X + Y ) (X + X ) + + (Y + Y ) obtained rom takin the colimit o the diaram

23 COARE-GRAINING OPEN MARKOV PROCEE 3 X Y X Y U U in two dierent ways. We call χ lobular because its source and taret -morphisms are identities. For the other lobular -isomorphism, i and are inite sets, then u( + ) is iven by while u( ) u() is iven by ( +, 0 + ) ( +, 0 0 ) + so there is a lobular -isomorphism µ between these, namely the identity -morphism. All the commutative diarams in the deinition o symmetric monoidal double cateory [3] can be checked in a straihtorward way. 5. Black-boxin or open Markov processes he eneral idea o black-boxin is to take a system and oret everythin except the relation between its inputs and outputs, as i we had placed it in a black box and were unable to see its inner workins. Previous work o Pollard and the irst author [6] constructed a black-boxin unctor : Dynam emialrel where Dynam is a cateory o inite sets and open dynamical systems and emialrel is a cateory o inite-dimensional real vector spaces and relations deined by polynomials and inequalities. When we blackbox such an open dynamical system, we obtain the relation between inputs and outputs that holds in steady state. A special case o an open dynamical system is an open Markov process as deined in this paper. hus, we could restrict the black-boxin unctor : Dynam emialrel to a cateory Mark with inite sets as objects and open Markov processes as morphisms. ince the steady state behavior o a Markov process is linear, we would et a unctor : Mark LinRel where LinRel is the cateory o inite-dimensional real vector spaces and linear relations [3]. However, we will o urther and deine black-boxin on the double cateory Mark. his will exhibit the relation between black-boxin and morphisms between open Markov processes. o do this, we promote LinRel to a double cateory LinRel with: (i) inite-dimensional real vector spaces U, V, W,... as objects, (ii) linear maps : V W as vertical -morphisms rom V to W, (iii) linear relations R V W as horizontal -cells rom V to W, (iv) squares R V V V V W W W W

24 COARE-GRAINING OPEN MARKOV PROCEE 4 obeyin ( )R as -morphisms. he last item deserves some explanation. A preorder is a cateory such that or any pair o objects x, y there exists at most one morphism α: x y. When such a morphism exists we usually write x y. imilarly there is a kind o double cateory or which iven any rame A M B C N D there exists at most one -morphism A C M α N B D illin this rame. For lack o a better term let us call this a deenerate double cateory. Item (iv) implies that LinRel will be deenerate in this sense. In LinRel, composition o vertical -morphisms is the usual composition o linear maps, while composition o horizontal -cells is the usual composition o linear relations. ince composition o linear relations obeys the associative and unit laws strictly, LinRel will be a strict double cateory. ince LinRel is deenerate, there is at most one way to deine the vertical composite o -morphisms R U U U U α R U U U U V V V V = βα β W W W W W W W W so we need merely check that a -morphism βα illin the rame at riht exists. his amounts to notin that ( )R, ( ) = ( )( )R.

25 COARE-GRAINING OPEN MARKOV PROCEE 5 imilarly, there is at most one way to deine the horizontal composite o -morphisms R V V V V R V V 3 V 3 R R V V 3 V V 3 α α h = α α h W W W W W W 3 W 3 W W 3 W W 3 so we need merely check that a iller α α exists, which amounts to notin that ( )R, ( h)r = ( h)(r R). heorem 5.. here exists a strict double cateory LinRel with the above properties. Proo. he cateory o objects LinRel 0 has inite-dimensional real vector spaces as objects and linear maps as morphisms. he cateory o arrows LinRel has linear relations as objects and squares R V V V V W W W W with ( )R as morphisms. he source and taret unctors s, t : LinRel LinRel 0 are clear. he identity-assinin unctor u: LinRel 0 LinRel sends a inite-dimensional real vector space V to the identity map V and a linear map : V W to the unique - morphism V V V W W W. he composition unctor : LinRel LinRel0 LinRel LinRel acts on objects by the usual composition o linear relations, and it acts on -morphisms by horizontal composition as described above. hese unctors can be shown to obey all the axioms o a double cateory. In particular, because LinRel is deenerate, all the required equations between -morphisms, such as the interchane law, hold automatically. Next we make LinRel into a symmetric monoidal double cateory. o do this, we irst ive LinRel 0 the structure o a symmetric monoidal cateory. We do this usin a speciic choice o direct sum or each pair o inite-dimensional real vector spaces as the tensor product, and a speciic 0-dimensional vector space as the unit object. hen we ive LinRel a symmetric monoidal structure as ollows. Given linear relations R V W and R V W, we deine their direct sum by R R = {(v, v, w, w ) : (v, w ) R, (v, w ) R } V V W W.

26 COARE-GRAINING OPEN MARKOV PROCEE 6 Given two -morphisms in LinRel : R V V V V R V V V V α α W W W W W W W W there is at most one way to deine their direct sum R R V V V V V V V V α α W W W W W W W W because LinRel is deenerate. o show that α α exists, we need merely note that ( )R, ( )R = ( )(R R ). heorem 5.. he double cateory LinRel can be iven the structure o a symmetric monoidal double cateory with the above properties. Proo. We have described LinRel 0 and LinRel as symmetric monoidal cateories. he source and taret unctors s, t : LinRel LinRel 0 are strict symmetric monoidal unctors. he required lobular -isomorphisms χ and µ are deined as ollows. Given our horizontal -cells R U V, R V W, U V, V W, the lobular -isomorphism χ: (R )(R ) (R R ) ( ) is the identity - morphism (R )(R ) U U W W (R R ) ( ) U U W W. he lobular -isomorphism µ: u(v W) u(v) u(w) is the identity -morphism V W V W V W V W V W V W.

27 COARE-GRAINING OPEN MARKOV PROCEE 7 All the commutative diarams in the deinition o symmetric monoidal double cateory [3] can be checked straihtorwardly. In particular, all diarams o -morphisms commute automatically because LinRel is deenerate. his sets the stae or deinin the symmetric monoidal double unctor : Mark LinRel. We start as ollows: (i) On objects: or a inite set, we deine ( ) to be the vector space R R. (ii) On horizontal -cells: or an open Markov process i (X, H) o, we deine its black-boxin as in De..7: ( i (X, H) o ) = {(i (v), I, o (v), O) : v R X, I R, O R and H(v) + i (I) o (O) = 0}. (iii) On vertical -morphisms: or a map :, we deine ( ): R R R R to be the linear map. What remains to be done is deine how acts on -morphisms o Mark. his describes the relation between steady state input and output concentrations and lows o a coarse-rained open Markov process in terms o the correspondin relation or the oriinal process: Lemma 5.3. Given a -morphism i (X, H) o p in Mark, there exists a unique -morphism i o (X, H ), ( ) ( i (X, H) o ) () ( ) () in LinRel. ( i (X, H ) o ) ( ) ( ) Proo. ince LinRel is deenerate, i there exists a -morphism o the claimed kind it is automatically unique. o prove that such a -morphism exists, it suices to prove (i (v), I, o (v), O) V = ( i (v), (I), o (v), (O)) W where V = ( i (X, H) o ) = {(i (v), I, o (v), O) : v R X, I R, O R and H(v) + i (I) o (O) = 0} and W = ( i (X, H ) o ) = {(i (v ), I, o (v ), O ) : v R X, I R, O R and H (v ) + i (I ) o (O ) = 0}.

28 COARE-GRAINING OPEN MARKOV PROCEE 8 o do this, assume (i (v), I, o (v), O) V, which implies that H(v) + i (I) o (O) = 0. () ince the commutin squares in α are pullbacks, Lemma 4.3 implies that i = i p, o = o p. hus ( i (v), (I), o (v), (O)) = (i p (v), (I), o p (v), (O)) and this is an element o W as desired i o prove Eq. (3), note that H p (v) + i (I) o (O) = 0. (3) H p (v) + i (I) o (O) = p H(v) + p i (I) p o (O) = p (H(v) + i (I) o (O)) where in the irst step we use the act that the squares in α commute, toether with the act that H p = p H. hus, Eq. () implies Eq. (3). he ollowin result is a special case o a result by Pollard and the irst author on blackboxin open dynamical systems [6]. o make this paper sel-contained we adapt the proo to the case at hand: Lemma 5.4. he black-boxin o a composite o two open Markov processes equals the composite o their black-boxins. Proo. Consider composable open Markov processes i (X, H) o compose these, we irst orm the pushout o, i (Y, G) j X + Y k o U. X Y i o i o hen their composite is ji (X + Y, H G) ko U where H G = j H j + k Gk. o prove that preserves composition, we irst show that hus, iven with we need to prove that (X + Y, H G) (Y, G) (X, H) (i (v), I, o (v), O) (X, H), (i (v ), I, o (v ), O ) (Y, G) o (v) = i (v ), O = I (i (v), I, o (v ), O ) (X + Y, H G). U

29 COARE-GRAINING OPEN MARKOV PROCEE 9 o do this, it suices to ind probabilities w R X+ Y such that (i (v), I, o (v ), O ) = (( ji) (w), I, (ko ) (w), O ) and w is a steady state o (X + Y, H G) with inlows I and outlows O. ince o (v) = i (v ), this diaram commutes: v R v X Y o i so by the universal property o the pushout there is a unique map w: X + Y R such that this commutes: R (4) v j w X + Y v k X Y o his simply says that because the unctions v and v aree on the overlap o our two open Markov processes, we can ind a unction w that restricts to v on X and v on Y. We now prove that w is a steady state o the composite open Markov process with inlows I and outlows O : i (H G)(w) + ( ji) (I) (ko ) (O ) = 0. (5) o do this we use the act that v is a steady state o i o (X, H) with inlows I and outlows O: H(v) + i (I) o (O) = 0 (6) and v is a steady state o i (Y, G) o U with inlows I and outlows O : G(v ) + i (I ) o (O ) = 0. (7) We push Eq. (6) orward alon j, push Eq. (7) orward alon k, and sum them: j (H(v)) + ( ji) (I) ( jo) (O) + k (G(v )) + (ki ) (I ) (ko ) (O ) = 0. ince O = I and jo = ki, two terms cancel, leavin us with j (H(v)) + ( ji) (I) + k (G(v )) (ko ) (O ) = 0. Next we combine the terms involvin the ininitesimal stochastic operators H and G, with the help o Eq. (4) and the deinition o H G: j (H(v)) + k (G(v )) = ( j H j + k Gk )(w) = (H G)(w). (8)

30 COARE-GRAINING OPEN MARKOV PROCEE 30 his leaves us with (H G)(w) + ( ji) (I) (ko ) (O ) = 0 which is Eq. (5), precisely what we needed to show. o inish showin that is a unctor, we need to show that o, suppose we have We need to show where and (X + Y, H G) (Y, G) (X, H). (( ji) (w), I, (ko ) (w), O ) (X + Y, H G). (( ji) (w), I, (ko ) (w), O ) = (i (v), I, o (v ), O ) (9) (i (v), I, o (v), O) (X, H), (i (v ), I, o (v ), O ) (Y, G) o do this, we bein by choosin o (v) = i (v ), O = I. v = j (w), v = k (w). his ensures that Eq. (9) holds, and since jo = ki, it also ensures that o (v) = ( jo) (w) = (ki ) (w) = i (v ). o inish the job, we need to ind an element O = I R such that v is a steady state o (X, H) with inlows I and outlows O and v is a steady state o (Y, G) with inlows I and outlows O. O course, we are iven the act that w is a steady state o (X + Y, H G) with inlows I and outlows O. In short, we are iven Eq. (5), and we seek O = I such that Eqs. (6) and (7) hold. hanks to our choices o v and v, we can use Eq. (8) and rewrite Eq. (5) as Eqs. (6) and (7) say that j (H(v) + i (I)) + k (G(v ) o (O )) = 0. (0) H(v) + i (I) o (O) = 0 G(v ) + i (I ) o (O ) = 0. () Now we use the act that j X + Y k X Y o i

31 COARE-GRAINING OPEN MARKOV PROCEE 3 is a pushout. Applyin the ree vector space on a inite set unctor, which preserves colimits, this implies that j R X+ Y k R X R Y o R is a pushout in the cateory o vector spaces. ince a pushout is ormed by takin irst a coproduct and then a coequalizer, this implies that i (o,0) R (0,i ) R X R Y j +k R X+ Y is a coequalizer. hus, the kernel o j + k is the imae o (o, 0) (0, i ). Eq. (0) says precisely that (H(v) + i (I), G(v ) o (O )) ker( j + k ). hus, it is in the imae o o i. In other words, there exists some element O = I R such that (H(v) + i (I), G(v ) o (O )) = (o (O), i (I )). his says that Eqs. (6) and (7) hold, as desired. his is the main result o this paper: heorem 5.5. here exists a symmetric monoidal double unctor : Mark LinRel with the ollowin behavior: (i) Objects: sends any inite set to the vector space R R. (ii) Vertical -morphisms: sends any map : to the linear map : R R R R. (iii) Horizontal -cells: sends any open Markov process i o (X, H) to the linear relation iven in De..7: ( i (X, H) o ) = {(i (v), I, o (v), O) : H(v) + i (I) o (O) = 0 or some I R, v R X, O R }. (iv) -Morphisms: sends any morphism o open Markov processes i (X, H) o p i o (X, H )