Lecture 7. µ(x)f(x). When µ is a probability measure, we say µ is a stationary distribution.
|
|
- Jewel Goodman
- 6 years ago
- Views:
Transcription
1 Lecture 7 1 Stationary measures of a Markov chain We now study the long time behavior of a Markov Chain: in particular, the existence and uniqueness of stationary measures, and the convergence of the distribution of the Markov chain to its stationary measure as time tends to infinity. 1.1 Existence and uniqueness of the stationary measure Definition 1.1 [Stationary measure] Let X be an irreducible Markov chain with countable state space S and transition matrix Π. A measure µ on S is called a stationary measure for X if (µπ)(x) : y S µ(y)π(y, x) µ(x) for all x S, (1.1) or equivalently, µ, Πf µ, f for all bounded f, (1.2) where µ, f x S µ(x)f(x). When µ is a probability measure, we say µ is a stationary distribution. The equivalence comes from the fact that µ is uniquely determined by its action on bounded test functions, while µ, Πf µπ, f. Example 1.2 A random walk on Z d regardless of the distribution of its increment has µ 1 as a stationary measure by virtue of the translation invariance of Z d. Any irreducible finite state Markov chain admits a unique stationary distribution, which is a left eigenvector of Π with eigenvalue 1. We are interested in the long time behavior of the Markov chain. If the chain is transient, then for any x, y S, G(x, y) n 0 Πn (x, y) <. In particular, Π n (x, y) 0 as n. This rules out the existence and convergence to a stationary probability distribution. The more interesting cases are the null recurrent and positive recurrent Markov chains. Theorem 1.3 [Existence of a stationary measure for recurrent Markov chains] Let X be an irreducible recurrent Markov chain with countable state space S and transition matrix Π. Then for any x S with τ x inf{n 1 : X n x}, the measure µ(y) : n0 [ τ x 1 P x (X n y, n < τ x ) E x n0 is a stationary measure for X, and y S µ(y) E x[τ x ]. 1 {Xny} ], y S, 1
2 Remark. In words, µ(y) is the expected number of visits to y before the Markov chain returns to x. Note that µ(x) 1. This is sometimes called the cycle trick. Proof. First we show that µ(y) < for all y S. Since µ(x) 1, let y x. Since the Markov chain is irreducible and recurrent, P y (τ x < τ y ) > 0. Therefore starting from y, the number of visits to y before the chain visits x is geometrically distributed. In particular, the expected number of visits to y before τ x is finite, and so is µ(y). For each y x, µ(y) P x (X n y, n < τ x ) P x (X n 1 z, X n y, n < τ x ) z S P x (X n 1 z, n 1 < τ x )Π(z, y) z S z S µ(z)π(z, y), which verifies the stationarity of µ at all y x. On the other hand, by the recurrence of X and a similar decomposition, 1 P x (τ x n) P x (X n 1 y, n 1 < τ x, X n x) y S y S P x (X n 1 y, n 1 < τ x )Π(y, x) y S µ(y)π(y, x), which verifies the stationarity of µ at x. Theorem 1.4 [Uniqueness of stationary measures for recurrent Markov chains] Let X be an irreducible recurrent Markov chain with countable state space S. Then the stationary measure µ for X is unique up to a constant multiple. Proof. Let µ be the stationary measure defined in Theorem 1.3 with µ(x) 1. Let ν be any stationary measure with ν(x) 1. We have for any y x, ν(y) ν(x)π(x, y) + z 1 x ν(z 1 )Π(z 1, y) (1.3) ν(x)π(x, y) + z 1 x ν(x)π(x, z 1 )Π(z 1, y) + z 1,z 2 x ν(z 2 )Π(z 2, z 1 )Π(z 1, y), where we have substituted (1.3) into itself. Iterating the substitution indefinitely then gives ν(y) Π(x, y) + z 1,,z n x P x (X n y, n < τ x ) µ(y). n 1 Π(x, z 1 ) Π(z i, z i+1 ) Π(z n, y) 2
3 Now suppose that ν(y) > µ(y) for some y S. By irreducibility, there exists n N with Π n (y, x) > 0. The stationarity of µ and ν implies z S µ(z)π n (z, x) µ(x) 1 ν(x) z S ν(z)π n (z, x). Therefore 0 z S(ν(z) µ(z))π n (z, x) (ν(y) µ(y))π n (y, x) > 0, which is a contradiction. Therefore ν µ. Combining Theorems 1.3 and 1.4 with the observation that transient irreducible Markov chains do not admit stationary probability distributions, we have the following. Corollary 1.5 [Stationary distributions] An irreducible Markov chain admits a stationary probability distribution µ (which is necessarily unique) if and only if it is positive recurrent, in which case µ(x) 1 E x[τ x] for all x S. 1.2 Convergence of the Markov chain We now proceed to the study of the convergence of an irreducible Markov chain, i.e., what is the limit of the probability measure Π n (x, ) as n for each x S? When the chain is transient, we have seen that Π n (x, y) 0 for all x, y S. If the chain is null recurrent, then there is a unique (up to a constant multiple) stationary measure, which has infinite mass. Since Π n (x, ) corresponds to the Markov chain starting with unit mass at x, we expect the measure to spread out and approximate a multiple of the stationary measure, hence Π n (x, y) 0 for all x, y S. If the chain is positive recurrent, then it is natural to expect that Π n (x, y) µ(y), the mass of the unique stationary distribution µ at y. The last statement is almost true, except for the issue of periodicity. To illustrate the problem, take a simple random walk on the Torus S : {0, 1,, 2m} where 0 and 2m are identified. Clearly the Markov chain is irreducible and the uniform distribution on S is the unique stationary distribution. However, Π n (0, ) is supported on the even sites when n is even, and on the odd sites when n is odd. So Π n (0, ) does not converge to the uniform distribution on S. Therefore we first need to address the issue of periodicity. Definition 1.6 [Period of a Markov chain] Let X be an irreducible Markov chain with countable state space S and transition matrix Π. For x S, let D x : {n : Π n (x, x) > 0} and let d x be the greatest common divisor (gcd) of D x. Then d x is independent of x S, which we simply denote by d and call it the period of the Markov chain. When d 1, we say the chain is aperiodic. In the definition above, we have used part of the following result. Lemma 1.7 Let X be an irreducible Markov chain with countable state space S. Then d x d y for all x, y S. Furthermore, for any x S, D x contains all sufficiently large multiples of d x. Proof. By irreducibility, there exist K, L N with Π K (x, y) > 0 and Π L (y, x) > 0. Therefore Π K+L (x, x) Π K (x, y)π L (y, x) > 0, 3
4 and hence d x (K + L), i.e., d x divides K + L. For any m D y, Π m (y, y) > 0, therefore Π K+L+m Π K (x, y)π m (y, y)π L (y, x) > 0. So d x (K +L+m). Since d x (K +L), we have d x m for all m D y. Therefore d x d y. Similarly we also have d y d x, and hence d x d y. Since d x is the greatest common divisor of D x, it is the gcd of a finite subset n 1,, n k D x. By the properties of gcd, there exist a 1,, a k Z such that k a i n i d x. Moving the terms with negative a i to the RHS above shows that there exists m N with md x, (m + 1)d x D x. For any n m 2, we can write nd x (lm + r)d x (l r)md x + r(m + 1)d x, where r is the remainder of n after diving by m, and l m > r by assumption. Therefore nd x D x for all n m 2, which proves the lemma. We are now ready to state the convergence result for irreducible aperiodic Markov chains. Theorem 1.8 [Convergence of transition kernels] Let X be an irreducible aperiodic Markov chain with countable state space S. If the chain is transient or null recurrent, then lim n Πn (x, y) 0 x, y S. (1.4) If the chain is positive recurrent with stationary distribution µ, then Theorem 1.8 follows from the renewal theorem. lim n Πn (x, y) µ(y) x, y S. (1.5) Theorem 1.9 [Renewal Theorem] Let f be a probability distribution on N { } with mean m n nf(n) [1, ]. Assume further that D : {n 1 : f(n) > 0} has greatest common divisor 1. A renewal process (U n ) n 0 with renewal time distribution f is a homogeneous Markov chain with state space {0, 1, } { } and transition probabilities p(x, x + n) f(n) for all x 0 and p(, ) 1. Then we have lim P 0(U i n for some i N) 1 n m. (1.6) Proof of Theorem 1.8. For a Markov chain X starting from x S, if we let U 0 0 and U n be the successive return times of X to x, then clearly U n is a renewal process with f(n) P x (τ x n), m E x [τ x ], and Π n (x, x) P 0 (U i n for some i N). Equations (1.4) (1.5) with y x then follows from the renewal theorem since E x [τ x ] when X is transient or null recurrent, and µ(x) 1 E 1 x[τ x] m when the chain is positive recurrent. When x y, note that n Π n (x, y) P x (τ y i)π n i (y, y). Equations (1.4) (1.5) then follow from the case for x y and the dominated convergence theorem. 4
5 Remark 1.10 Not surprisingly, the renewal theorem can conversely be deduced from Theorem 1.8. Given a renewal process U on {0, 1, } with renewal time distribution f on N { }, we can construct an irreducible aperiodic Markov chain X on {0, 1, } { } as follows. Let Π(0, l) f(l + 1) for l {0, 1, } { }, Π(i, i 1) 1 for i 1, and Π(, ) 1. Then the successive return times of X to 0 is distributed as U, and P 0 (U i n for some i N) is precisely Π n (0, 0). Since m nf(n) E 0[τ 0 ], (1.6) follows from (1.4) (1.5). Proof of Theorem 1.9. If f( ) > 0, then τ < almost surely for the Markov chain U, and (1.6) clearly holds. From now on, we assume f( ) 0, so that n N f(n) 1. Let p(n) P 0 (U i n for some i N). By decomposing in terms of the first renewal time, p(n) satisfies the recursive relation (known as the renewal equation) n p(n) f(i)p(n i). (1.7) Summing over 1 n N, we obtain p(n) (f(1) + + f(n)) + (f(1) + + f(n 1))p(1) + + f(1)p(n 1) p(n n) n f(i) p(n n)(1 T (n + 1)), where T (n + 1) in+1 f(i). Rearranging terms then gives T (n)p(n n + 1) 1 T (N + 1) f(n). (1.8) Note that T (n) m. By dominated convergence, if lim n p(n) exists, then it must be 1 m. Let a lim sup n p(n), which is bounded by 1 since p(n) 1. By Cantor diagonalization, we can find a sequence (n j ) j N along which p(n j + i) q(i) for all i Z, with q(0) a. We claim that q a. Assuming the claim, then taking the limit N in (1.8) along the sequence n j shows that a 0 when m by Fatou s lemma, and a 1 m when m < by dominated convergence. It remains to verify q a. sequence n j + k in (1.7) gives In particular, Applying the dominated convergence theorem along the q(k) a f(i)q(k i). (1.9) f(i)q( i). Since by definition of a, q( i) a for all i Z, we have q( i) a for all i D : {n N : f(n) > 0}. The same argument applied to (1.9) shows that q( i) a for all i 2 D : {n x + y : x, y D}, and inductively, for all i k D, k N, with k D defined analogously. Since the gcd of D is 1, the proof of Lemma 1.7 shows that q( i) a for all i sufficiently large. Substituting these values of q into (1.9) shows that q a. The same argument can be used to show that lim inf p(n) 1 m when m <, which proves Theorem
Lecture 10. Theorem 1.1 [Ergodicity and extremality] A probability measure µ on (Ω, F) is ergodic for T if and only if it is an extremal point in M.
Lecture 10 1 Ergodic decomposition of invariant measures Let T : (Ω, F) (Ω, F) be measurable, and let M denote the space of T -invariant probability measures on (Ω, F). Then M is a convex set, although
More informationModern Discrete Probability Spectral Techniques
Modern Discrete Probability VI - Spectral Techniques Background Sébastien Roch UW Madison Mathematics December 22, 2014 1 Review 2 3 4 Mixing time I Theorem (Convergence to stationarity) Consider a finite
More informationLecture 7. We can regard (p(i, j)) as defining a (maybe infinite) matrix P. Then a basic fact is
MARKOV CHAINS What I will talk about in class is pretty close to Durrett Chapter 5 sections 1-5. We stick to the countable state case, except where otherwise mentioned. Lecture 7. We can regard (p(i, j))
More informationLecture 5. If we interpret the index n 0 as time, then a Markov chain simply requires that the future depends only on the present and not on the past.
1 Markov chain: definition Lecture 5 Definition 1.1 Markov chain] A sequence of random variables (X n ) n 0 taking values in a measurable state space (S, S) is called a (discrete time) Markov chain, if
More informationNote that in the example in Lecture 1, the state Home is recurrent (and even absorbing), but all other states are transient. f ii (n) f ii = n=1 < +
Random Walks: WEEK 2 Recurrence and transience Consider the event {X n = i for some n > 0} by which we mean {X = i}or{x 2 = i,x i}or{x 3 = i,x 2 i,x i},. Definition.. A state i S is recurrent if P(X n
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 2
MATH 56A: STOCHASTIC PROCESSES CHAPTER 2 2. Countable Markov Chains I started Chapter 2 which talks about Markov chains with a countably infinite number of states. I did my favorite example which is on
More informationP i [B k ] = lim. n=1 p(n) ii <. n=1. V i :=
2.7. Recurrence and transience Consider a Markov chain {X n : n N 0 } on state space E with transition matrix P. Definition 2.7.1. A state i E is called recurrent if P i [X n = i for infinitely many n]
More informationPositive and null recurrent-branching Process
December 15, 2011 In last discussion we studied the transience and recurrence of Markov chains There are 2 other closely related issues about Markov chains that we address Is there an invariant distribution?
More informationMarkov Chains and Stochastic Sampling
Part I Markov Chains and Stochastic Sampling 1 Markov Chains and Random Walks on Graphs 1.1 Structure of Finite Markov Chains We shall only consider Markov chains with a finite, but usually very large,
More informationMATH 56A SPRING 2008 STOCHASTIC PROCESSES 65
MATH 56A SPRING 2008 STOCHASTIC PROCESSES 65 2.2.5. proof of extinction lemma. The proof of Lemma 2.3 is just like the proof of the lemma I did on Wednesday. It goes like this. Suppose that â is the smallest
More informationErgodic Theorems. Samy Tindel. Purdue University. Probability Theory 2 - MA 539. Taken from Probability: Theory and examples by R.
Ergodic Theorems Samy Tindel Purdue University Probability Theory 2 - MA 539 Taken from Probability: Theory and examples by R. Durrett Samy T. Ergodic theorems Probability Theory 1 / 92 Outline 1 Definitions
More informationNecessary and sufficient conditions for strong R-positivity
Necessary and sufficient conditions for strong R-positivity Wednesday, November 29th, 2017 The Perron-Frobenius theorem Let A = (A(x, y)) x,y S be a nonnegative matrix indexed by a countable set S. We
More informationMarkov Chains CK eqns Classes Hitting times Rec./trans. Strong Markov Stat. distr. Reversibility * Markov Chains
Markov Chains A random process X is a family {X t : t T } of random variables indexed by some set T. When T = {0, 1, 2,... } one speaks about a discrete-time process, for T = R or T = [0, ) one has a continuous-time
More informationSTOCHASTIC PROCESSES Basic notions
J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving
More informationApplied Stochastic Processes
Applied Stochastic Processes Jochen Geiger last update: July 18, 2007) Contents 1 Discrete Markov chains........................................ 1 1.1 Basic properties and examples................................
More informationLecture 9 Classification of States
Lecture 9: Classification of States of 27 Course: M32K Intro to Stochastic Processes Term: Fall 204 Instructor: Gordan Zitkovic Lecture 9 Classification of States There will be a lot of definitions and
More information215 Problem 1. (a) Define the total variation distance µ ν tv for probability distributions µ, ν on a finite set S. Show that
15 Problem 1. (a) Define the total variation distance µ ν tv for probability distributions µ, ν on a finite set S. Show that µ ν tv = (1/) x S µ(x) ν(x) = x S(µ(x) ν(x)) + where a + = max(a, 0). Show that
More informationLecture 6. 2 Recurrence/transience, harmonic functions and martingales
Lecture 6 Classification of states We have shown that all states of an irreducible countable state Markov chain must of the same tye. This gives rise to the following classification. Definition. [Classification
More informationMarkov Chains, Stochastic Processes, and Matrix Decompositions
Markov Chains, Stochastic Processes, and Matrix Decompositions 5 May 2014 Outline 1 Markov Chains Outline 1 Markov Chains 2 Introduction Perron-Frobenius Matrix Decompositions and Markov Chains Spectral
More informationINTRODUCTION TO MARKOV CHAINS AND MARKOV CHAIN MIXING
INTRODUCTION TO MARKOV CHAINS AND MARKOV CHAIN MIXING ERIC SHANG Abstract. This paper provides an introduction to Markov chains and their basic classifications and interesting properties. After establishing
More informationMARKOV CHAINS AND HIDDEN MARKOV MODELS
MARKOV CHAINS AND HIDDEN MARKOV MODELS MERYL SEAH Abstract. This is an expository paper outlining the basics of Markov chains. We start the paper by explaining what a finite Markov chain is. Then we describe
More information11.4. RECURRENCE AND TRANSIENCE 93
11.4. RECURRENCE AND TRANSIENCE 93 Similar arguments show that simple smmetric random walk is also recurrent in 2 dimensions but transient in 3 or more dimensions. Proposition 11.10 If x is recurrent and
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 1
MATH 56A: STOCHASTIC PROCESSES CHAPTER. Finite Markov chains For the sake of completeness of these notes I decided to write a summary of the basic concepts of finite Markov chains. The topics in this chapter
More informationHomework set 3 - Solutions
Homework set 3 - Solutions Math 495 Renato Feres Problems 1. (Text, Exercise 1.13, page 38.) Consider the Markov chain described in Exercise 1.1: The Smiths receive the paper every morning and place it
More informationMS&E 321 Spring Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10
MS&E 321 Spring 12-13 Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10 Section 3: Regenerative Processes Contents 3.1 Regeneration: The Basic Idea............................... 1 3.2
More informationMarkov Chains for Everybody
Markov Chains for Everybody An Introduction to the theory of discrete time Markov chains on countable state spaces. Wilhelm Huisinga, & Eike Meerbach Fachbereich Mathematik und Informatik Freien Universität
More informationMARKOV CHAINS AND MIXING TIMES
MARKOV CHAINS AND MIXING TIMES BEAU DABBS Abstract. This paper introduces the idea of a Markov chain, a random process which is independent of all states but its current one. We analyse some basic properties
More informationµ n 1 (v )z n P (v, )
Plan More Examples (Countable-state case). Questions 1. Extended Examples 2. Ideas and Results Next Time: General-state Markov Chains Homework 4 typo Unless otherwise noted, let X be an irreducible, aperiodic
More informationChapter 7. Markov chain background. 7.1 Finite state space
Chapter 7 Markov chain background A stochastic process is a family of random variables {X t } indexed by a varaible t which we will think of as time. Time can be discrete or continuous. We will only consider
More informationChapter 2. Markov Chains. Introduction
Chapter 2 Markov Chains Introduction A Markov chain is a sequence of random variables {X n ; n = 0, 1, 2,...}, defined on some probability space (Ω, F, IP), taking its values in a set E which could be
More informationABSTRACT MARKOV CHAINS, RANDOM WALKS, AND CARD SHUFFLING. Nolan Outlaw. May 2015
ABSTRACT MARKOV CHAINS, RANDOM WALKS, AND CARD SHUFFLING by Nolan Outlaw May 215 Chair: Dr. Gail Ratcliff, PhD Major Department: Mathematics A common question in the study of random processes pertains
More information8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains
8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains 8.1 Review 8.2 Statistical Equilibrium 8.3 Two-State Markov Chain 8.4 Existence of P ( ) 8.5 Classification of States
More informationP(X 0 = j 0,... X nk = j k )
Introduction to Probability Example Sheet 3 - Michaelmas 2006 Michael Tehranchi Problem. Let (X n ) n 0 be a homogeneous Markov chain on S with transition matrix P. Given a k N, let Z n = X kn. Prove that
More informationCONVERGENCE THEOREM FOR FINITE MARKOV CHAINS. Contents
CONVERGENCE THEOREM FOR FINITE MARKOV CHAINS ARI FREEDMAN Abstract. In this expository paper, I will give an overview of the necessary conditions for convergence in Markov chains on finite state spaces.
More informationRANDOM WALKS. Course: Spring 2016 Lecture notes updated: May 2, Contents
RANDOM WALKS ARIEL YADIN Course: 201.1.8031 Spring 2016 Lecture notes updated: May 2, 2016 Contents Lecture 1. Introduction 3 Lecture 2. Markov Chains 8 Lecture 3. Recurrence and Transience 18 Lecture
More informationCS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions
CS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions Instructor: Erik Sudderth Brown University Computer Science April 14, 215 Review: Discrete Markov Chains Some
More informationSome Results on the Ergodicity of Adaptive MCMC Algorithms
Some Results on the Ergodicity of Adaptive MCMC Algorithms Omar Khalil Supervisor: Jeffrey Rosenthal September 2, 2011 1 Contents 1 Andrieu-Moulines 4 2 Roberts-Rosenthal 7 3 Atchadé and Fort 8 4 Relationship
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationTreball final de grau GRAU DE MATEMÀTIQUES Facultat de Matemàtiques Universitat de Barcelona MARKOV CHAINS
Treball final de grau GRAU DE MATEMÀTIQUES Facultat de Matemàtiques Universitat de Barcelona MARKOV CHAINS Autor: Anna Areny Satorra Director: Dr. David Márquez Carreras Realitzat a: Departament de probabilitat,
More information25.1 Ergodicity and Metric Transitivity
Chapter 25 Ergodicity This lecture explains what it means for a process to be ergodic or metrically transitive, gives a few characterizes of these properties (especially for AMS processes), and deduces
More informationNon-homogeneous random walks on a semi-infinite strip
Non-homogeneous random walks on a semi-infinite strip Chak Hei Lo Joint work with Andrew R. Wade World Congress in Probability and Statistics 11th July, 2016 Outline Motivation: Lamperti s problem Our
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Simulated Annealing Barnabás Póczos & Ryan Tibshirani Andrey Markov Markov Chains 2 Markov Chains Markov chain: Homogen Markov chain: 3 Markov Chains Assume that the state
More informationStochastic Processes (Week 6)
Stochastic Processes (Week 6) October 30th, 2014 1 Discrete-time Finite Markov Chains 2 Countable Markov Chains 3 Continuous-Time Markov Chains 3.1 Poisson Process 3.2 Finite State Space 3.2.1 Kolmogrov
More information2 Discrete-Time Markov Chains
2 Discrete-Time Markov Chains Angela Peace Biomathematics II MATH 5355 Spring 2017 Lecture notes follow: Allen, Linda JS. An introduction to stochastic processes with applications to biology. CRC Press,
More informationMS&E 321 Spring Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10. x n+1 = f(x n ),
MS&E 321 Spring 12-13 Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10 Section 4: Steady-State Theory Contents 4.1 The Concept of Stochastic Equilibrium.......................... 1 4.2
More informationClassification of Countable State Markov Chains
Classification of Countable State Markov Chains Friday, March 21, 2014 2:01 PM How can we determine whether a communication class in a countable state Markov chain is: transient null recurrent positive
More informationMatrix analytic methods. Lecture 1: Structured Markov chains and their stationary distribution
1/29 Matrix analytic methods Lecture 1: Structured Markov chains and their stationary distribution Sophie Hautphenne and David Stanford (with thanks to Guy Latouche, U. Brussels and Peter Taylor, U. Melbourne
More informationThree hours THE UNIVERSITY OF MANCHESTER. 24th January
Three hours MATH41011 THE UNIVERSITY OF MANCHESTER FOURIER ANALYSIS AND LEBESGUE INTEGRATION 24th January 2013 9.45 12.45 Answer ALL SIX questions in Section A (25 marks in total). Answer THREE of the
More informationA note on adiabatic theorem for Markov chains and adiabatic quantum computation. Yevgeniy Kovchegov Oregon State University
A note on adiabatic theorem for Markov chains and adiabatic quantum computation Yevgeniy Kovchegov Oregon State University Introduction Max Born and Vladimir Fock in 1928: a physical system remains in
More information1 Random walks: an introduction
Random Walks: WEEK Random walks: an introduction. Simple random walks on Z.. Definitions Let (ξ n, n ) be i.i.d. (independent and identically distributed) random variables such that P(ξ n = +) = p and
More informationChapter 8. General Countably Additive Set Functions. 8.1 Hahn Decomposition Theorem
Chapter 8 General Countably dditive Set Functions In Theorem 5.2.2 the reader saw that if f : X R is integrable on the measure space (X,, µ) then we can define a countably additive set function ν on by
More informationConvergence Rate of Markov Chains
Convergence Rate of Markov Chains Will Perkins April 16, 2013 Convergence Last class we saw that if X n is an irreducible, aperiodic, positive recurrent Markov chain, then there exists a stationary distribution
More information12 Markov chains The Markov property
12 Markov chains Summary. The chapter begins with an introduction to discrete-time Markov chains, and to the use of matrix products and linear algebra in their study. The concepts of recurrence and transience
More informationMarkov Chains on Countable State Space
Markov Chains on Countable State Space 1 Markov Chains Introduction 1. Consider a discrete time Markov chain {X i, i = 1, 2,...} that takes values on a countable (finite or infinite) set S = {x 1, x 2,...},
More informationThe Theory behind PageRank
The Theory behind PageRank Mauro Sozio Telecom ParisTech May 21, 2014 Mauro Sozio (LTCI TPT) The Theory behind PageRank May 21, 2014 1 / 19 A Crash Course on Discrete Probability Events and Probability
More informationDiscrete time Markov chains. Discrete Time Markov Chains, Limiting. Limiting Distribution and Classification. Regular Transition Probability Matrices
Discrete time Markov chains Discrete Time Markov Chains, Limiting Distribution and Classification DTU Informatics 02407 Stochastic Processes 3, September 9 207 Today: Discrete time Markov chains - invariant
More informationSome Definition and Example of Markov Chain
Some Definition and Example of Markov Chain Bowen Dai The Ohio State University April 5 th 2016 Introduction Definition and Notation Simple example of Markov Chain Aim Have some taste of Markov Chain and
More informationLecture 11: Introduction to Markov Chains. Copyright G. Caire (Sample Lectures) 321
Lecture 11: Introduction to Markov Chains Copyright G. Caire (Sample Lectures) 321 Discrete-time random processes A sequence of RVs indexed by a variable n 2 {0, 1, 2,...} forms a discretetime random process
More informationRECURRENCE IN COUNTABLE STATE MARKOV CHAINS
RECURRENCE IN COUNTABLE STATE MARKOV CHAINS JIN WOO SUNG Abstract. This paper investigates the recurrence and transience of countable state irreducible Markov chains. Recurrence is the property that a
More informationarxiv: v1 [math.pr] 6 Jan 2014
Recurrence for vertex-reinforced random walks on Z with weak reinforcements. Arvind Singh arxiv:40.034v [math.pr] 6 Jan 04 Abstract We prove that any vertex-reinforced random walk on the integer lattice
More informationStochastic Models: Markov Chains and their Generalizations
Scuola di Dottorato in Scienza ed Alta Tecnologia Dottorato in Informatica Universita di Torino Stochastic Models: Markov Chains and their Generalizations Gianfranco Balbo e Andras Horvath Outline Introduction
More informationAdvanced Computer Networks Lecture 2. Markov Processes
Advanced Computer Networks Lecture 2. Markov Processes Husheng Li Min Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville Spring, 2016 1/28 Outline 2/28 1 Definition
More informationErgodic Properties of Markov Processes
Ergodic Properties of Markov Processes March 9, 2006 Martin Hairer Lecture given at The University of Warwick in Spring 2006 1 Introduction Markov processes describe the time-evolution of random systems
More informationMarkov Chains. X(t) is a Markov Process if, for arbitrary times t 1 < t 2 <... < t k < t k+1. If X(t) is discrete-valued. If X(t) is continuous-valued
Markov Chains X(t) is a Markov Process if, for arbitrary times t 1 < t 2
More informationSTA 294: Stochastic Processes & Bayesian Nonparametrics
MARKOV CHAINS AND CONVERGENCE CONCEPTS Markov chains are among the simplest stochastic processes, just one step beyond iid sequences of random variables. Traditionally they ve been used in modelling a
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More information88 CONTINUOUS MARKOV CHAINS
88 CONTINUOUS MARKOV CHAINS 3.4. birth-death. Continuous birth-death Markov chains are very similar to countable Markov chains. One new concept is explosion which means that an infinite number of state
More informationMarkov Processes on Discrete State Spaces
Markov Processes on Discrete State Spaces Theoretical Background and Applications. Christof Schuette 1 & Wilhelm Huisinga 2 1 Fachbereich Mathematik und Informatik Freie Universität Berlin & DFG Research
More informationWinter 2019 Math 106 Topics in Applied Mathematics. Lecture 9: Markov Chain Monte Carlo
Winter 2019 Math 106 Topics in Applied Mathematics Data-driven Uncertainty Quantification Yoonsang Lee (yoonsang.lee@dartmouth.edu) Lecture 9: Markov Chain Monte Carlo 9.1 Markov Chain A Markov Chain Monte
More information1 Continuous-time chains, finite state space
Université Paris Diderot 208 Markov chains Exercises 3 Continuous-time chains, finite state space Exercise Consider a continuous-time taking values in {, 2, 3}, with generator 2 2. 2 2 0. Draw the diagramm
More informationMarkov chain Monte Carlo
1 / 26 Markov chain Monte Carlo Timothy Hanson 1 and Alejandro Jara 2 1 Division of Biostatistics, University of Minnesota, USA 2 Department of Statistics, Universidad de Concepción, Chile IAP-Workshop
More informationBirth-death chain models (countable state)
Countable State Birth-Death Chains and Branching Processes Tuesday, March 25, 2014 1:59 PM Homework 3 posted, due Friday, April 18. Birth-death chain models (countable state) S = We'll characterize the
More informationExamples of Countable State Markov Chains Thursday, October 16, :12 PM
stochnotes101608 Page 1 Examples of Countable State Markov Chains Thursday, October 16, 2008 12:12 PM Homework 2 solutions will be posted later today. A couple of quick examples. Queueing model (without
More informationProf. Ila Varma HW 8 Solutions MATH 109. A B, h(i) := g(i n) if i > n. h : Z + f((i + 1)/2) if i is odd, g(i/2) if i is even.
1. Show that if A and B are countable, then A B is also countable. Hence, prove by contradiction, that if X is uncountable and a subset A is countable, then X A is uncountable. Solution: Suppose A and
More informationMarkov Chains (Part 3)
Markov Chains (Part 3) State Classification Markov Chains - State Classification Accessibility State j is accessible from state i if p ij (n) > for some n>=, meaning that starting at state i, there is
More informationChapter 2: Markov Chains and Queues in Discrete Time
Chapter 2: Markov Chains and Queues in Discrete Time L. Breuer University of Kent 1 Definition Let X n with n N 0 denote random variables on a discrete space E. The sequence X = (X n : n N 0 ) is called
More informationMATH 564/STAT 555 Applied Stochastic Processes Homework 2, September 18, 2015 Due September 30, 2015
ID NAME SCORE MATH 56/STAT 555 Applied Stochastic Processes Homework 2, September 8, 205 Due September 30, 205 The generating function of a sequence a n n 0 is defined as As : a ns n for all s 0 for which
More informationISyE 6761 (Fall 2016) Stochastic Processes I
Fall 216 TABLE OF CONTENTS ISyE 6761 (Fall 216) Stochastic Processes I Prof. H. Ayhan Georgia Institute of Technology L A TEXer: W. KONG http://wwong.github.io Last Revision: May 25, 217 Table of Contents
More informationProbability & Computing
Probability & Computing Stochastic Process time t {X t t 2 T } state space Ω X t 2 state x 2 discrete time: T is countable T = {0,, 2,...} discrete space: Ω is finite or countably infinite X 0,X,X 2,...
More informationMarkov chains. Randomness and Computation. Markov chains. Markov processes
Markov chains Randomness and Computation or, Randomized Algorithms Mary Cryan School of Informatics University of Edinburgh Definition (Definition 7) A discrete-time stochastic process on the state space
More informationLecture 5: Random Walks and Markov Chain
Spectral Graph Theory and Applications WS 20/202 Lecture 5: Random Walks and Markov Chain Lecturer: Thomas Sauerwald & He Sun Introduction to Markov Chains Definition 5.. A sequence of random variables
More informationRENEWAL THEORY STEVEN P. LALLEY UNIVERSITY OF CHICAGO. X i
RENEWAL THEORY STEVEN P. LALLEY UNIVERSITY OF CHICAGO 1. RENEWAL PROCESSES A renewal process is the increasing sequence of random nonnegative numbers S 0,S 1,S 2,... gotten by adding i.i.d. positive random
More informationProbabilistic Aspects of Computer Science: Markovian Models
Probabilistic Aspects of Computer Science: Markovian Models S. Haddad September 29, 207 Professor at ENS Cachan, haddad@lsv.ens-cachan.fr, http://www.lsv.ens-cachan.fr/ haddad/ Abstract These lecture notes
More informationStochastic Processes
Stochastic Processes 8.445 MIT, fall 20 Mid Term Exam Solutions October 27, 20 Your Name: Alberto De Sole Exercise Max Grade Grade 5 5 2 5 5 3 5 5 4 5 5 5 5 5 6 5 5 Total 30 30 Problem :. True / False
More informationSummary of Results on Markov Chains. Abstract
Summary of Results on Markov Chains Enrico Scalas 1, 1 Laboratory on Complex Systems. Dipartimento di Scienze e Tecnologie Avanzate, Università del Piemonte Orientale Amedeo Avogadro, Via Bellini 25 G,
More informationMath 4121 Spring 2012 Weaver. Measure Theory. 1. σ-algebras
Math 4121 Spring 2012 Weaver Measure Theory 1. σ-algebras A measure is a function which gauges the size of subsets of a given set. In general we do not ask that a measure evaluate the size of every subset,
More informationMATH FINAL EXAM REVIEW HINTS
MATH 109 - FINAL EXAM REVIEW HINTS Answer: Answer: 1. Cardinality (1) Let a < b be two real numbers and define f : (0, 1) (a, b) by f(t) = (1 t)a + tb. (a) Prove that f is a bijection. (b) Prove that any
More informationPractical conditions on Markov chains for weak convergence of tail empirical processes
Practical conditions on Markov chains for weak convergence of tail empirical processes Olivier Wintenberger University of Copenhagen and Paris VI Joint work with Rafa l Kulik and Philippe Soulier Toronto,
More informationMarkov Chains. Sarah Filippi Department of Statistics TA: Luke Kelly
Markov Chains Sarah Filippi Department of Statistics http://www.stats.ox.ac.uk/~filippi TA: Luke Kelly With grateful acknowledgements to Prof. Yee Whye Teh's slides from 2013 14. Schedule 09:30-10:30 Lecture:
More informationOutlines. Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC)
Markov Chains (2) Outlines Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC) 2 pj ( n) denotes the pmf of the random variable p ( n) P( X j) j We will only be concerned with homogenous
More informationMARKOV CHAIN MONTE CARLO
MARKOV CHAIN MONTE CARLO RYAN WANG Abstract. This paper gives a brief introduction to Markov Chain Monte Carlo methods, which offer a general framework for calculating difficult integrals. We start with
More informationTransience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient:
Stochastic2010 Page 1 Long-Time Properties of Countable-State Markov Chains Tuesday, March 23, 2010 2:14 PM Homework 2: if you turn it in by 5 PM on 03/25, I'll grade it by 03/26, but you can turn it in
More informationIntroduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 Markov Chain Monte Carlo Methods Barnabás Póczos & Aarti Singh Contents Markov Chain Monte Carlo Methods Goal & Motivation Sampling Rejection Importance Markov
More informationIntroduction to Markov Chains and Riffle Shuffling
Introduction to Markov Chains and Riffle Shuffling Nina Kuklisova Math REU 202 University of Chicago September 27, 202 Abstract In this paper, we introduce Markov Chains and their basic properties, and
More information1.2. Markov Chains. Before we define Markov process, we must define stochastic processes.
1. LECTURE 1: APRIL 3, 2012 1.1. Motivating Remarks: Differential Equations. In the deterministic world, a standard tool used for modeling the evolution of a system is a differential equation. Such an
More informationSMSTC (2007/08) Probability.
SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................
More informationLecture 21. David Aldous. 16 October David Aldous Lecture 21
Lecture 21 David Aldous 16 October 2015 In continuous time 0 t < we specify transition rates or informally P(X (t+δ)=j X (t)=i, past ) q ij = lim δ 0 δ P(X (t + dt) = j X (t) = i) = q ij dt but note these
More informationIrreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1
Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate
More informationLectures on Stochastic Stability. Sergey FOSS. Heriot-Watt University. Lecture 4. Coupling and Harris Processes
Lectures on Stochastic Stability Sergey FOSS Heriot-Watt University Lecture 4 Coupling and Harris Processes 1 A simple example Consider a Markov chain X n in a countable state space S with transition probabilities
More informationA note on transitive topological Markov chains of given entropy and period
A note on transitive topological Markov chains of given entropy and period Jérôme Buzzi and Sylvie Ruette November, 205 Abstract We show that, for every positive real number h and every positive integer
More information