Uniformly Uniformly-ergodic Markov chains and BSDEs

Uniformly Uniformly-ergodic Markov chains and BSDEs Samuel N. Cohen Mathematical Institute, University of Oxford (Based on joint work with Ying Hu, Robert Elliott, Lukas Szpruch) Centre Henri Lebesgue, Rennes, 22-24 May 2013 Research supported by the Oxford Man Institute S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 1 / 47

Outline 1 Uniform Uniform-Ergodicity 2 Markov chain BSDEs 3 Markov chain Ergodic BSDEs 4 Markov chain BSDEs to stopping times S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 2 / 47

Outline 1 Uniform Uniform-Ergodicity 2 Markov chain BSDEs 3 Markov chain Ergodic BSDEs 4 Markov chain BSDEs to stopping times S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 3 / 47

Markov chains Consider a time homogenous finite/countable-state continuous-time Markov chain X. X is a process in (Ω, F, {F t } t 0, P), where {F t } is the natural filtration of X. X takes values in X, the standard basis of R N where N = number of states. X jumps from state X t to state e i X at rate e i AX t, so X t = X 0 + AX u du + M t for some R N -valued martingale M. ]0,t] A is a R N N matrix, with [A] ij 0 for i j, and 1 Ae i = j [A] ij 0. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 4 / 47

Example For N = 4, we can consider the Markov chain with matrix 3 1 0 0 A = 1 3 2 2 2 2 3 1 0 0 1 3 Which can be drawn as a graph 1 1 1 2 2 2 2 2 1 3 4 1 S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 5 / 47

Uniform ergodicity If X 0 µ, for some probability measure µ on X, we write X t P t µ. Writing µ as a vector, P t µ e At µ. We say X is uniformly ergodic if P t µ π TV Re ρt for all µ for some R, ρ > 0, some probability measure π on X. TV is the total variation norm (and writing signed measures as vectors, TV = l1 ). The measure π is the ergodic measure of the Markov chain and is unique. It also satisfies P t π = π, so it is stationary. We call (R, ρ) the parameters of ergodicity. Irreducible finite state chains are always uniformly ergodic. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 6 / 47

Stability of Uniform Ergodicity We can tie together the matrix A and the measure P. Hence a change of measure corresponds to using a different (stochastic, time varying?) matrix B. Question We say A is uniformly ergodic if X is uniformly ergodic under P A Suppose the chain is uniformly ergodic under the measure P A, and has associated parameters (R A, ρ A ). If A and B are similar, can we say the chain is uniformly ergodic under P B, and can we say anything about (R B, ρ B )? We wish to find a relationship between A and B under which we can get a uniform estimate of ρ B. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 7 / 47

Definition For γ > 0, we say A γ-controls B (and write A γ B) if B γa is also a rate matrix. If A γ B and B γ A we write A γ B. Example A = 3 1 0 0 1 3 2 2 2 2 3 1 0 0 1 3 Then for γ 1/2, A γ B. B = 4 1 0 0 2 3 2 2 2 2 3 1 0 0 1 3 S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 8 / 47

Note: 1 is a partial order. If A γ B for some γ > 0 then A and B have the same zero entries. Our key result is: Theorem Suppose A is uniformly ergodic and γ ]0, 1[. Then there exist constants R, ρ dependent on γ and A such that B is uniformly ergodic with parameters ( R, ρ) whenever A γ B. The key is that these constants are uniform in B, so these measures are uniformly uniformly-ergodic. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 9 / 47

Outline 1 Uniform Uniform-Ergodicity 2 Markov chain BSDEs 3 Markov chain Ergodic BSDEs 4 Markov chain BSDEs to stopping times S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 10 / 47

Markov chain BSDEs This result allows us to describe (uniformly) the long-run behaviour of Markov chains under a variety of measures. For infinite-horizon BSDEs, we can think of dynamically changing the measure, so such results are useful. Before we can make use of this connection, we need the basic theory of BSDEs when noise is driven by a Markov chain. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 11 / 47

Martingale Representation Recall that X t = X 0 + AX u du + M t ]0,t] M is a R N martingale with predictable quadratic variation d M = φ t dt = (diag(ax t ) + AX t X t + X t X t A )dt We have E [( Z dm ) 2 ] = E [ Z φ t Zdt] =: E [ Z 2 M t dt ] We can give a martingale representation theorem with respect to M, namely for any scalar martingale L L t = L 0 + Zt dm t ]0,t] and Z is unique up equality in z 2 M = z φ t z. Note 1 2 M t 0. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 12 / 47

BSDE Existence and Uniqueness Theorem Let f be a predictable function with f (ω, t, y, z) f (ω, t, y, z ) 2 c( y y 2 + z z 2 M t ) for some c > 0, and E[ ]0,T] f (ω, t, 0, 0) 2 dt] <. Then for any ξ L 2 (F T ) the BSDE Y t = ξ + f (ω, t, Y u, Z u )du + ZudM u ]t,t] ]t,t] has a unique adapted solution (Y, Z) with appropriate integrability. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 13 / 47

γ-balanced drivers To get a comparison theorem, we need the following definition. Definition f is γ-balanced if there exists a random field λ, with λ(,, z, z ) predictable and λ(ω, t,, ) Borel measurable, such that f (ω, t, y, z) f (ω, t, y, z ) = (z z ) (λ(ω, t, z, z ) AX t ), for each e i X, e i λ(ω, t, z, z ) e i AX t for some γ > 0, where 0/0 := 1, 1 λ(ω, t, z, z ) 0 and [γ, γ 1 ] λ(ω, t, z + α1, z ) = λ(ω, t, z, z ) for all α R. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 14 / 47

Comparison theorem Lemma If f is γ-balanced for γ > 0, then it is uniformly Lipschitz in z with respect to Mt. Lemma If f (u; ) is γ-balanced for each u, then g( ) := ess sup u {f (u; )} is γ-balanced (given it is always finite). S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 15 / 47

Comparison theorem Lemma If B γ A then f (ω, t, y, z) = z (B A)X t is γ-balanced with λ( ) = BX t, and Y t = E B [ξ F t ]. The purpose of this definition is to obtain: Theorem (Comparison theorem) Let f be γ-balanced for some γ > 0. Then if ξ ξ and f (ω, t, y, z) f (ω, t, y, z), the associated BSDE solutions satisfy Y t Y t for all t, and Y t = Y t on A F t iff Y s = Y s on A for all s t. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 16 / 47

Comparison theorem The comparison theorem is easy to derive from Lemma Let f be γ-balanced. Then there exists a measure Q P such that Y t Y t ( + f (ω, s, Ys, Z s ) f (ω, s, Y s, Z s ) ) ds ]0,t] is a Q-martingale. Proof. Q is the measure where λ(ω, t, Z t, Z t) is the compensator of X. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 17 / 47

Markovian Solutions We can also obtain a Feynman Kac type result. Theorem Let f : X [0, T] R R n, and consider the BSDE with Y t = φ(x T ) + f (X t, t, Y u, Z u )du + ZudM u ]t,t] ]t,t] for some function φ : X R. Then the solution satisfies Y t = u(t, X t ) = X t u t, Z t = u t for u solving du t = (f(t, u t ) + A u t )dt; e i u T = φ(e i ) where e i f(t, u t) := f (e i, t, e i u t, u t ). S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 18 / 47

Application to Control Consider the problem min u E u[ e rt φ(x T ) + ]0,T] ] e rt L(u t ; X t, t)dt where u is a predictable process with values in U, and E u is the expectation under which X has compensator λ t (u t ) R N. Suppose λ t ( ) satisfies e i λt( ) e i AX t [γ, γ 1 ] for some γ > 0. Define the Hamiltonian f (ω, t, y, z) = ry t + inf u U {L(u; X t, t) + z (λ t (u) AX t (ω))} The dynamic value function Y t satisfies the BSDE with driver f, terminal value φ(x t ). By the comparison theorem, we have existence and uniqueness of an optimal feedback control. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 19 / 47

Outline 1 Uniform Uniform-Ergodicity 2 Markov chain BSDEs 3 Markov chain Ergodic BSDEs 4 Markov chain BSDEs to stopping times S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 20 / 47

Ergodic BSDEs Hu, Fuhrman, Tessitore and collaborators have introduced ergodic BSDEs. These are known to relate to ergodic control problems. In a Brownian setting, existence/uniqueness of these equations depends on analysis of uniform ellipticity estimates of perturbations of the generator of the underlying X. These methods do not directly map to the Markov Chain case. Instead we will use the uniform ergodicity estimates we have derived. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 21 / 47

Infinite-Horizon BSDEs The first step is to establish existence of Infinite Horizon BSDEs Theorem Let α > 0, and consider the equation dy t = (f (ω, t, Z t ) αy t )dt + Z t dm t t [0, [. If f (ω, t, z ) < C, this admits a solution (Y, Z) with Y < C/α for t [0, [, and this is the unique bounded solution. Proof. Let Y T t solve the BSDE dy t = (f (ω, t, Z t ) αy t )dt + Z t dm t ; Y T T = 0. Then show Y T t C/α for all T and Y T Y (uniformly on compacts). S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 22 / 47

EBSDE Existence/Uniqueness Theorem (SC & Hu 2012) The ergodic BSDE dy t = (f (X t, Z t ) λ)dt + Z t dm t admits a bounded solution (Y, Z, λ) with Y t = u(x t ) and u(ˆx) = 0, whenever f is γ-balanced with f (x, 0) < C. Any other bounded solution has the same λ, any other bounded Markovian solution has Y t = Y t + c. Our Feynman Kac type result gives, with Y = u(x t ) = X t u, Z = u, f(u) λ = A u which is readily calculable for small dimensions. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 23 / 47

Proof. Denote the infinite horizon BSDE solution Yt α matrices B T γ A, = u α (X t ). Then for some rate [ ] u α (x) = lim T EBT e αu f (X u, 0)du X 0 = x ]0,T] So, with ( R, ρ) the uniform convergence bounds on P B u µ, u α (x) u α (x ) C lim e αu P BT u δ x P BT u δ x TV du T ]0,T] C R(α + ρ) 1 C R/α < A diagonal argument implies u α (x) u α (ˆx) u(x) and αu α (ˆx) λ, as α 0. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 24 / 47

Properties The comparison theorem does not hold for Y for EBSDEs. However, it does hold for λ. Theorem Let f, f be γ-balanced and f f. Then we have λ λ in the EBSDE solutions. Theorem Let π A denote the ergodic measure when X has matrix A. Then for some B u γ A, λ = f (x, u)dπ A (x) = f (x, 0)dπ Bu (x) X X Y t + c = f (x, u)µ A X 0 (x) = f (x, 0)µ Bu X 0 (x) where µ A x = R + (P A u δ x π A )du. X X S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 25 / 47

Applications Application to Ergodic control similar to the finite-horizon case, with value { λ = min lim sup E u[ 1 ]} L(u t ; X t )dt u T T ]0,T] Write f as the Hamiltonian, comparison theorem gives optimal feedback control, etc... Instead, we will look at creating spatially stable nonlinear probabilities on graphs. These attempt to generalize the ergodic distribution of a Markov chain to a nonlinear setting. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 26 / 47

Applications Consider the driver f (x, z) = I {x Ξ} + g(x, z). Then if g 0, the EBSDE solutions will be λ = π(ξ). If g is convex, we will have a convex ergodic probability, If g(x, z) = sup B B {z (B A)x}, when B has column-exchangeability, then we can show that λ Ξ = sup π B (Ξ) B The global balance equation Aπ 0 becomes the Isaac s condition { } sup inf B X B {v B x}dπ B (x) 0 for all v R N. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 27 / 47

Nonlinear Pagerank The Pagerank algorithm is modelled on a Markov chain randomly following links in a graph (e.g. the WWW) at a constant rate. Nodes are ranked according to the ergodic probability the chain is at each node. What happens if we instead dynamically take the minimum over a range of rate matrices? S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 28 / 47

Pagerank example Consider this undirected graph. 2 7 1 4 5 6 3 8 Our Markov chain moves between states randomly, at a constant (exit) rate. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 29 / 47

Pagerank example The ergodic probabilities/pagerank under this basic model can be simply calculated. 11.1% 5.6% 11.1% 16.7% 11.1% 16.7% 16.7% 11.1% We have three states which are equally highest ranked, even though they have different positions in the graph. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 30 / 47

Pagerank example Now suppose a bias can be introduced dynamically, so that the relative rate of jumping to the right/left can be doubled. Then the EBSDE can be solved with drivers f (x, z) = I {x Ξ} + min B {z (B A)x} for each Ξ X. The corresponding values λ Ξ (for Ξ singletons) are 5.5% 2.0% 4.3% 10.3% 4.8% 7.8% 11.4% 6.8% S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 31 / 47

Pagerank example 2 7 1 4 5 6 3 8 We can also assess collections of states, so if Ξ = {1, 3, 4}, then λ Ξ = 0.3067 Ξ = {2, 3, 4}, then λ Ξ = 0.3164 Ξ = {3, 4, 5}, then λ Ξ = 0.3531 Ξ = {4, 5, 6}, then λ Ξ = 0.2899. Note all these collections have ergodic measure 0.444 under the basic model. Peculiarly, λ {2} = 0.055 > 0.048 = λ {5}, but λ {2,3,4} < λ {3,4,5}. In this way we can assess the centrality of a collection of states. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 32 / 47

Outline 1 Uniform Uniform-Ergodicity 2 Markov chain BSDEs 3 Markov chain Ergodic BSDEs 4 Markov chain BSDEs to stopping times S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 33 / 47

BSDEs to stopping times Another interesting class of BSDEs is where the terminal time is a stopping time. Here we want Y t = ξ + ]t,τ] f (ω, u, Y u, Z u )du + ZudM u ]t,τ] when τ is a stopping time, ξ is F τ -measurable. If f (ω, t, y, z) f (ω, t, y, z) y y α for some α > 0, then we can use the discounted methods considered earlier. This does not cover the interesting case f (x, z) = z (B A)x. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 34 / 47

BSDEs to stopping times We will assume sufficient bounds on ξ and τ that our BSDE admits a unique solution with polynomial growth. This is similar to the backwards heat equation admitting only one polynomial growth solution. Of use will be the family Q γ of measures where X has compensator λ with e i λ(ω, t) e i AX [γ, γ 1 ] for all i. t These measures are singular on F but equivalent on F τ. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 35 / 47

BSDEs to stopping times Imagine that we can prove the following bounds. Assumption For some nondecreasing K, K : R + [1, [, constants β, β > 0, E Q [ ξ F t ] K(t), E Q [(1 + τ) 1+β F t ] K(t), E Q [K(τ) 1+ β F t ] K(t), all P-a.s. for all Q Q γ and all t. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 36 / 47

BSDEs to stopping times Theorem Let ξ, τ satisfy the previous bounds, and suppose f is γ-balanced, and such that for any y, y, z, f (ω, t, 0, 0) c(1 + t ˆβ) f (ω, t, y, z) f (ω, t, y, z) y y [ c, 0] for some c R, some ˆβ [0, β[. Then the BSDE Y t = ξ + f (ω, u, Y u, Z u )du + ]t,τ] admits a unique adapted solution satisfying the bound Y t (1 + c)k(t). ]t,τ] Z udm u S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 37 / 47

BSDEs to stopping times Proof. Approximate with finite horizon BSDE Y t = I T τ ξ + f (ω, u, Y u, Z u )du + ZudM u ]t,τ T] ]t,τ T] and let T, using Hölder and Markov inequalities to prove convergence on compacts. We can also prove the comparison theorem, which implies... Theorem If the conditions on f hold only locally, and ξ, f (ω, t, 0, 0) are bounded, by the comparison theorem everything works given conditions on τ. This isn t too interesting if we can t verify the assumptions on τ. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 38 / 47

Hitting times work! Theorem If τ is the first hitting time of a set Ξ X, and ξ = g(τ, X τ ) for some g(t, x) c(1 + t ˆβ), then the assumptions are satisfied. Proof. As we have uniform ergodicity estimates, we can show that τ admits exponential moments uniformly bounded under all Markovian measures in Q γ. By applying the comparison theorem to finite-time approximations, we can extend this to all measures in Q γ. By extension, this also holds for nth hitting times, etc... S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 39 / 47

Markovian Structure We can also give a Feynman Kac type result Theorem Let τ be the first hitting time of a set Ξ X and let f satisfy the conditions. Consider the BSDE Y t = φ(x τ ) f (X u, Y u, Z u )du + Z u dm u ]t,τ] ]t,τ] where φ is a bounded function X R. Then there exists a bounded vector u R N such that Y t = X t u, Z t u and f (x, x u, u) = u Ax for x X \ Ξ, with boundary values x u = φ(x) for x Ξ. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 40 / 47

Applications: Network Reliability Consider a model for transmission of messages over our graph. 2 7 1 4 5 6 3 8 Messages are passed randomly between nodes, we wish to assess whether a message created at node 1 will reach node 8 before being killed at the faulty node 5. This probability corresponds to a terminal value φ(x) = I {x=e8 } at a time τ = inf{t : X t = e 5 or e 8 }. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 41 / 47

Applications: Network Reliability Without control, the probability is given by 45% 50% 55% 36% 0% 50% 64% 100% With our control, we have 68% 67% 73% 59% 0% 67% 78% 100% S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 42 / 47

Applications: Non-Ohmic Circuits Markov chains can be used to model electronic circuits of resistors. If w ij = 1/R ij is the resistance of a connection, then the voltage potential v satisfies { v(i) = φ(i), i Ξ for Ξ a source set. v(i) j w ij = j w ijv(j), i / Ξ This depends on Ohm s Law and Kirtchoff s current laws. For finite circuits, the voltage potential is the expected value at the first hitting time of Ξ, for a Markov chain with A ij = w ji for i j. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 43 / 47

Applications: Non-Ohmic Circuits Diodes, Transistors, etc. do not satisfy Ohm s law. For a diode, the effective resistance is R v ij = v(j) v(i) c i (exp([v(j) v(i)]/c v ) 1) > 0 Write B v for the rate matrix with entries B v ij = 1/Rv ji. We can write the voltage potential as solving the BSDE to the first hitting time of Ξ with driver f (x, z) = z (B z A)x where A is generated from a reference resistor circuit. One can show the required conditions are (locally) satisfied, so for any finite source potential we have unique solutions. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 44 / 47

Applications: Non-Ohmic Circuits: Example Consider an AND logic gate. +5v +5v Cv=0.1 +5v Cv=0.1 +0.16v +5v R=1000 +0.1v R=1000 +0v +0v +0.1v Cv=0.1 +0.1v +0.1v R=1000 +0v S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 45 / 47

Connection to EBSDE The following theorem gives a relation between Ergodic BSDEs and BSDEs to stopping times. Theorem Let ˆf (x, z) = f (x, z) X f (x, z)dπa (x). Then the BSDE to τ = inf{t : X t = ˆx} with driver ˆf and terminal value ξ = 0 agrees with the EBSDE with driver f and u(ˆx) = 0. Hence the corresponding BSDEs to τ T converge uniformly to the EBSDE solution. This suggests an alternative Monte-Carlo approach to numerical computation of EBSDE solutions, which is of importance for large/infinite-state chains. S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 46 / 47

Conclusions We have seen that one can obtain uniform uniform-ergodicity estimates for Markov chains under perturbations of the rate matrix. With these, one can study solutions to Ergodic BSDEs and BSDEs to hitting times Solutions are readily calculable for small examples, and can be used to model various problems. Open problems include removing the γ-balanced requirement, including y in the driver of EBSDEs, better numerics... S.N. Cohen (Oxford) Uniformly-ergodic Markov chains Rennes, May 2013 47 / 47