Homework set 3 - Solutions
|
|
- Loreen Golden
- 6 years ago
- Views:
Transcription
1 Homework set 3 - Solutions Math 495 Renato Feres Problems 1. (Text, Exercise 1.13, page 38.) Consider the Markov chain described in Exercise 1.1: The Smiths receive the paper every morning and place it on a pile after reading it. Each afternoon, with probability 1/3, someone takes all the papers in the pile and puts them in the recycling bin. Also, if ever there are at least five papers in the pile, Mr. Smith (with probability 1) takes the papers to the bin. Consider the number of papers in the pile in the evening. (a) Model this by a Markov chain by drawing its transition diagram and writing the transition matrix. (b) After a long time, what would be the expected number of papers in the pile? (c) Assume the pile starts with 0 papers. What is the expected time until the pile will again have 0 papers? (d) Do a stochastic simulation to obtain the expected number of papers in the pile using one of the Markov chain programs of the previous assignment. Does it agree to reasonable precision with the value you obtain analytically in part (b)? Solution. (a) The transition diagram is shown below. It represents my interpretation of the situation described by the problem. The transition matrix is: P = 1/3 2/ /3 0 2/ / /3 0 1/ /
2 (b) The chain is irreducible and aperiodic. So it has a unique stationary distribution π. By solving the linear system πp = π we obtain π = (81,54,36,24,16). In the long run, the number X of papers in the pile has the stationary distribution π. Therefore, E(X ) = 4 j =0 j π j = 1 ( ) = (c) The expected return time to a given state is the reciprocal of the stationary probability of that state. Therefore, E[T 0 X 0 = 0] = 1/π 0 = 211/81 = (d) For this problem we may use the Markov chain program of H.W. 2: #It is assumed that the states are {1,2,...,s} #where s is the length of pi0, and P is s-by-s. #pi0 is the probability distribution of the initial step #and P is the transition probabilities matrix. Markov=function(N,pi0,P) { #Number of states s=length(pi0) X=matrix(0,1,N) a=sample(c(1:s),1,replace=true,pi0) X[1]=a for (i in 2:N) { a=sample(c(1:s),1,replace=true,p[a,]) X[i]=a } X } The initial probability vector may be taken to be pi0 = c(1,0,0,0,0) which assumes initial state 0 with probability 1. For this problem, any other initial condition would work similarly well because we are interested in the system s long run, stationary regime. The transition probability matrix is a = 1/3 b = 2/3 P[1,]=c(a, b, 0, 0, 0) P[2,]=c(a, 0, b, 0, 0) P[3,]=c(a, 0, 0, b, 0) P[4,]=c(a, 0, 0, 0, b) P[5,]=c(1, 0, 0, 0, 0) The average number of newspapers on the pile can now be estimated by running 2
3 N= X=Markov(N,pi0,P) mean(x)-1 One run of this script gave me the value > mean(x)-1 [1] Which seems to confirm the result obtained analytically. Note that the sample standard deviation (which the problem does not ask you to find) would be > sd(x-1)/sqrt(n) [1] This suggests to me that it is reasonable to think that the first two digits at least (mean = 1.2) are correct. 2. (Text, Exercise 1.18, page 40.) Consider a deck of cards numbered 1,...,n. At each time step we shuffle the cards by drawing a card at random and placing it at the top of the deck. This can be thought of as a Markov chain on S n, the set of all permutations of n elements. If λ denotes any permutation (a one-to-one correspondence of {1,...,n} with itself), and ν j denotes the permutation corresponding to moving the j th card to the top of the deck, then the transition probabilities for this chain are given by p(λ,λν j ) = 1 n, j = 1,...,n. (Note: the notation λν j should be understood as the composition of two permutation maps.) This chain is irreducible and aperiodic. It is easy to verify that the unique invariant probability is the uniform measure on S n the measure that assigns probability 1/n! to each permutation. Therefore, if we start with any ordering of the cards, after enough moves of this kind the deck will be well shuffled. Suppose we take a standard deck of cards with 52 cards and do one move every second. What is the expected amount of time in years until the deck returns to the original order? Solution. If we accept the claim that the stationary distribution is the uniform distribution on S n, which is a set of n! elements, then π j = 1/n! for every permutation j S n, and the mean return time to any permutation is 1/π j = n! in seconds. If n = 52 and σ is the original permutation of the deck, then E[T σ X 0 = σ] = 52! seconds. To get a better sense of the number, note that 52! = 10 ln52!/ln10 = ln10 j =1 ln j seconds years. For comparison, the current measurement of the age of the universe is approximately years. 3. (Text, Exercise 2.1, page 57.) Consider the queueing model (Example 3 of Section 2.1). [Edited question: You need not do part (d); it will be explained in class. Also see below the edited part (a).] 3
4 (a) For which values of p, q is the chain null recurrent, positive recurrent, transient? Edited part (a): For which values of p, q is the chain positive recurrent? [The rest of the problem will be explained in class. Note: The necessary and sufficient condition on p, q for positive recurrent will follow naturally once you solve part (b), which you may do first.] (b) For the positive recurrent case give the limiting probability distribution π. (c) For the positive recurrent case, what is the average length of the queue in equilibrium? (d) For the transient case, give α(x) = the probability starting at x of ever reaching state 0. Solution. (a) Let z = 0 and for each x 1 define the probability α(x) that X n = z for some n 0 given that X 0 = x. It is shown on page 49 of the textbook that α(x) = y S p(x, y)α(y), x > 0. It is also stated there that the chain is transient if there exists a solution α to this equation such that α(0) = 1 and the infimum of α(x) is 0. Using the transition probabilities p(0,0) = 1 p, p(0,1) = p, p(n,n + 1) = p(1 q), p(n,n) = pq + (1 p)(1 q), p(n,n 1) = q(1 p), for n 1 we obtain the following recursive relation for α: 0 = aα(n 1) bα(n) + cα(n + 1) where a = q(1 p),b = p + q 2pq,c = p(1 q). Note that a + c = b. Equivalently, α(n + 1) = a c α(n 1) + b c α(n). The recursive relation may be written in the matrix form ( ) ( α(n + 1) b/c a/c = α(n) 1 0 )( α(n) α(n 1) ). The eigenvalues of the coefficients matrix are the roots of λ 2 b c λ + a c = 0. Using the fact that b c a c = 1 we can easily solve for the roots and find λ = 1 and a c = q(1 p) p(1 q). From our class discussion of difference equations, we know that [ ] q(1 p) x α(x) = c 0 + c 1. p(1 q) 4
5 From α(0) = 1 it follows that c 0 + c 1 = 1, so [ ] q(1 p) x α(x) = (1 c 1 ) + c 1. p(1 q) It is easily checked that any such expression satisfies the recurrence relation. We may now use the criterion of transience given on page 50 of the textbook (conditions (2.2) - (2.4).) Those conditions are satisfied by setting c 1 = 1 and q(1 p) p(1 q) < 1. The latter is equivalent to q < p. Thus we conclude: The chain is transient if and only if q < p. When q p, we need to decide when the chain is null or positive recurrent. We know that the chain is positively recurrent if and only if it admits a stationary probability measure π. Such a measure is characterized by the equation π(x)p(x, y) = π(y) for all y S. In the present example, this equation amounts to the recursive equation x q n+1 π(n + 1) (p n + q n )π(n) + p n 1 π(n 1) = 0 where p n = p(1 q), q n = q(1 q) for n 1, p 0 = p, q 0 = 0. Note that when n = 0, q 1 π(1) p 0 π(0) = 0. From these relations we derive q n+1 π(n + 1) p n π(n) = q n π(n) p n 1 π(n 1) = = q 1 π(1) p 0 π(0) = 0 so that Replacing the values of p n, q n gives π(n + 1) = p n q n+1 π(n) = = p np n 1 p 0 q n+1 q n q 1 π(0). ( ) p p(1 q) n 1 π(n) = π(0) for n 1. q(1 p) q(1 p) It is now clear that for an invariant probability measure to exist it is necessary and sufficient that the geometric series 1+u+u 2 + converges, where u = p(1 q)/q(1 p). This happens if and only if p < q. Thus we conclude: the chain is transient if q < p the chain is null recurrent if q = p the chain is positive recurrent if q > p. (b) To obtain π(n) we first need to find the normalization constant so that n 0 π(n) = 1: { 1 = π(0) + π(1) + = π(0) 1 + p q(1 p) ( 1 + u + u 2 + )} { = π(0) 1 + p 1 q(1 p) 1 u } = q q p π(0) 5
6 where u = p(1 q)/q(1 p). Thus π(0) = (q p)/q. Therefore, π(0) = q p p(q p), π(n) = q q 2 (1 p) ( ) p(1 q) n 1 for n 1. q(1 p) (c) The average length of the queue at equilibrium is p(q p) p(q p) nπ(n) = n q 2 (1 p) un 1 = q 2 nu n 1. (1 p) The infinite series can be obtained by noting that, within its radius of convergence, nx n 1 = d dx x n = d 1 dx 1 x = 1 (1 x) 2. Evaluating at u = p(1 q)/q(1 p) gives the value nu n 1 = (q(1 p)/(q p)) 2. Therefore, the mean queue length is Expected queue length = p(1 p) q p. Note that if q is only slightly larger than p, then the expected length is very large. (d) From the expression for α(x) given above and the fact that α(x) must approach 0 as x goes to we conclude that [ ] q(1 p) x α(x) =. p(1 q) 4. (Text, Exercise 2.2, page 57.) Consider the following Markov chain with state space S = {0,1,2,...}. A sequence of positive numbers p 1, p 2,... is given with i=1 p i = 1. Whenever the chain reaches state 0 it chooses a new state according to the probability p i. Whenever the chain is at a state other than 0 it proceeds deterministically, one step at a time, toward 0. In other words, the chain has transition probability p(x, x 1) = 1, p(0, x) = p x, for x > 0. This is a recurrent chain since the chain keeps returning to 0. Under what conditions on the p x is the chain positive recurrent? In this case, what is the limiting probability distribution π? [Hint: it may be easier to compute E(T ) directly where T is the time of first return to 0 starting at 0.] Solution. The (recurrent) chain is positively recurrent if and only if the expected time of recurrence to any state if finite. Consider the return time T 0 to state 0. The condition for positively recurrent chain is E[T 0 ] = (n + 1)p n = 1 + np n <. The recursive equation for the invariant probability is π(x) = y S π(y)p(y, x). Only the transition probabilities to 0 given by p(0, x) = p x and p(x + 1, x) = 1 are non-zero. Thus, for n 1, π(n) = π(n + 1) + p n π(0). 6
7 Iterating this relation gives π(1) = π(0) and π(n) = π(0) The normalization constant is obtained from 1 = [ 1 n 1 j =1 [ π(n) = π(0) 1 + ] ( p j = p j )π(0). j =n ] np n from which we obtain π(0) = 1 j =n p j, π(n) = (n + 1)p for n 1. n (n + 1)p n 7
MATH 56A: STOCHASTIC PROCESSES CHAPTER 2
MATH 56A: STOCHASTIC PROCESSES CHAPTER 2 2. Countable Markov Chains I started Chapter 2 which talks about Markov chains with a countably infinite number of states. I did my favorite example which is on
More informationMATH 56A SPRING 2008 STOCHASTIC PROCESSES 65
MATH 56A SPRING 2008 STOCHASTIC PROCESSES 65 2.2.5. proof of extinction lemma. The proof of Lemma 2.3 is just like the proof of the lemma I did on Wednesday. It goes like this. Suppose that â is the smallest
More informationMath Homework 5 Solutions
Math 45 - Homework 5 Solutions. Exercise.3., textbook. The stochastic matrix for the gambler problem has the following form, where the states are ordered as (,, 4, 6, 8, ): P = The corresponding diagram
More informationExamples of Countable State Markov Chains Thursday, October 16, :12 PM
stochnotes101608 Page 1 Examples of Countable State Markov Chains Thursday, October 16, 2008 12:12 PM Homework 2 solutions will be posted later today. A couple of quick examples. Queueing model (without
More informationHomework set 2 - Solutions
Homework set 2 - Solutions Math 495 Renato Feres Simulating a Markov chain in R Generating sample sequences of a finite state Markov chain. The following is a simple program for generating sample sequences
More informationMarkov Chains. X(t) is a Markov Process if, for arbitrary times t 1 < t 2 <... < t k < t k+1. If X(t) is discrete-valued. If X(t) is continuous-valued
Markov Chains X(t) is a Markov Process if, for arbitrary times t 1 < t 2
More informationLecture 7. µ(x)f(x). When µ is a probability measure, we say µ is a stationary distribution.
Lecture 7 1 Stationary measures of a Markov chain We now study the long time behavior of a Markov Chain: in particular, the existence and uniqueness of stationary measures, and the convergence of the distribution
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 7
MATH 56A: STOCHASTIC PROCESSES CHAPTER 7 7. Reversal This chapter talks about time reversal. A Markov process is a state X t which changes with time. If we run time backwards what does it look like? 7.1.
More informationAt the boundary states, we take the same rules except we forbid leaving the state space, so,.
Birth-death chains Monday, October 19, 2015 2:22 PM Example: Birth-Death Chain State space From any state we allow the following transitions: with probability (birth) with probability (death) with probability
More informationSTOCHASTIC PROCESSES Basic notions
J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving
More informationMS&E 321 Spring Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10. x n+1 = f(x n ),
MS&E 321 Spring 12-13 Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10 Section 4: Steady-State Theory Contents 4.1 The Concept of Stochastic Equilibrium.......................... 1 4.2
More informationPositive and null recurrent-branching Process
December 15, 2011 In last discussion we studied the transience and recurrence of Markov chains There are 2 other closely related issues about Markov chains that we address Is there an invariant distribution?
More informationStochastic Processes
Stochastic Processes 8.445 MIT, fall 20 Mid Term Exam Solutions October 27, 20 Your Name: Alberto De Sole Exercise Max Grade Grade 5 5 2 5 5 3 5 5 4 5 5 5 5 5 6 5 5 Total 30 30 Problem :. True / False
More information88 CONTINUOUS MARKOV CHAINS
88 CONTINUOUS MARKOV CHAINS 3.4. birth-death. Continuous birth-death Markov chains are very similar to countable Markov chains. One new concept is explosion which means that an infinite number of state
More informationOn random walks. i=1 U i, where x Z is the starting point of
On random walks Random walk in dimension. Let S n = x+ n i= U i, where x Z is the starting point of the random walk, and the U i s are IID with P(U i = +) = P(U n = ) = /2.. Let N be fixed (goal you want
More informationMarkov Chains (Part 3)
Markov Chains (Part 3) State Classification Markov Chains - State Classification Accessibility State j is accessible from state i if p ij (n) > for some n>=, meaning that starting at state i, there is
More informationChapter 7. Markov chain background. 7.1 Finite state space
Chapter 7 Markov chain background A stochastic process is a family of random variables {X t } indexed by a varaible t which we will think of as time. Time can be discrete or continuous. We will only consider
More informationBirth-death chain models (countable state)
Countable State Birth-Death Chains and Branching Processes Tuesday, March 25, 2014 1:59 PM Homework 3 posted, due Friday, April 18. Birth-death chain models (countable state) S = We'll characterize the
More informationThe cutoff phenomenon for random walk on random directed graphs
The cutoff phenomenon for random walk on random directed graphs Justin Salez Joint work with C. Bordenave and P. Caputo Outline of the talk Outline of the talk 1. The cutoff phenomenon for Markov chains
More informationTransience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient:
Stochastic2010 Page 1 Long-Time Properties of Countable-State Markov Chains Tuesday, March 23, 2010 2:14 PM Homework 2: if you turn it in by 5 PM on 03/25, I'll grade it by 03/26, but you can turn it in
More information8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains
8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains 8.1 Review 8.2 Statistical Equilibrium 8.3 Two-State Markov Chain 8.4 Existence of P ( ) 8.5 Classification of States
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationP i [B k ] = lim. n=1 p(n) ii <. n=1. V i :=
2.7. Recurrence and transience Consider a Markov chain {X n : n N 0 } on state space E with transition matrix P. Definition 2.7.1. A state i E is called recurrent if P i [X n = i for infinitely many n]
More informationMATH 564/STAT 555 Applied Stochastic Processes Homework 2, September 18, 2015 Due September 30, 2015
ID NAME SCORE MATH 56/STAT 555 Applied Stochastic Processes Homework 2, September 8, 205 Due September 30, 205 The generating function of a sequence a n n 0 is defined as As : a ns n for all s 0 for which
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 1
MATH 56A: STOCHASTIC PROCESSES CHAPTER. Finite Markov chains For the sake of completeness of these notes I decided to write a summary of the basic concepts of finite Markov chains. The topics in this chapter
More informationA note on adiabatic theorem for Markov chains and adiabatic quantum computation. Yevgeniy Kovchegov Oregon State University
A note on adiabatic theorem for Markov chains and adiabatic quantum computation Yevgeniy Kovchegov Oregon State University Introduction Max Born and Vladimir Fock in 1928: a physical system remains in
More informationHomework 3 posted, due Tuesday, November 29.
Classification of Birth-Death Chains Tuesday, November 08, 2011 2:02 PM Homework 3 posted, due Tuesday, November 29. Continuing with our classification of birth-death chains on nonnegative integers. Last
More informationDiscrete time Markov chains. Discrete Time Markov Chains, Limiting. Limiting Distribution and Classification. Regular Transition Probability Matrices
Discrete time Markov chains Discrete Time Markov Chains, Limiting Distribution and Classification DTU Informatics 02407 Stochastic Processes 3, September 9 207 Today: Discrete time Markov chains - invariant
More informationMarkov Chains CK eqns Classes Hitting times Rec./trans. Strong Markov Stat. distr. Reversibility * Markov Chains
Markov Chains A random process X is a family {X t : t T } of random variables indexed by some set T. When T = {0, 1, 2,... } one speaks about a discrete-time process, for T = R or T = [0, ) one has a continuous-time
More informationMarkov Chain Monte Carlo
Chapter 5 Markov Chain Monte Carlo MCMC is a kind of improvement of the Monte Carlo method By sampling from a Markov chain whose stationary distribution is the desired sampling distributuion, it is possible
More informationStochastic Processes (Week 6)
Stochastic Processes (Week 6) October 30th, 2014 1 Discrete-time Finite Markov Chains 2 Countable Markov Chains 3 Continuous-Time Markov Chains 3.1 Poisson Process 3.2 Finite State Space 3.2.1 Kolmogrov
More informationCountable state discrete time Markov Chains
Countable state discrete time Markov Chains Tuesday, March 18, 2014 2:12 PM Readings: Lawler Ch. 2 Karlin & Taylor Chs. 2 & 3 Resnick Ch. 1 Countably infinite state spaces are of practical utility in situations
More informationErgodic Theorems. Samy Tindel. Purdue University. Probability Theory 2 - MA 539. Taken from Probability: Theory and examples by R.
Ergodic Theorems Samy Tindel Purdue University Probability Theory 2 - MA 539 Taken from Probability: Theory and examples by R. Durrett Samy T. Ergodic theorems Probability Theory 1 / 92 Outline 1 Definitions
More informationLecture 7. We can regard (p(i, j)) as defining a (maybe infinite) matrix P. Then a basic fact is
MARKOV CHAINS What I will talk about in class is pretty close to Durrett Chapter 5 sections 1-5. We stick to the countable state case, except where otherwise mentioned. Lecture 7. We can regard (p(i, j))
More informationWeek 5: Markov chains Random access in communication networks Solutions
Week 5: Markov chains Random access in communication networks Solutions A Markov chain model. The model described in the homework defines the following probabilities: P [a terminal receives a packet in
More informationINTRODUCTION TO MARKOV CHAINS AND MARKOV CHAIN MIXING
INTRODUCTION TO MARKOV CHAINS AND MARKOV CHAIN MIXING ERIC SHANG Abstract. This paper provides an introduction to Markov chains and their basic classifications and interesting properties. After establishing
More informationMath 456: Mathematical Modeling. Tuesday, March 6th, 2018
Math 456: Mathematical Modeling Tuesday, March 6th, 2018 Markov Chains: Exit distributions and the Strong Markov Property Tuesday, March 6th, 2018 Last time 1. Weighted graphs. 2. Existence of stationary
More informationStochastic Processes MIT, fall 2011 Day by day lecture outline and weekly homeworks. A) Lecture Outline Suggested reading
Stochastic Processes 18445 MIT, fall 2011 Day by day lecture outline and weekly homeworks A) Lecture Outline Suggested reading Part 1: Random walk on Z Lecture 1: thursday, september 8, 2011 Presentation
More informationBudapest University of Tecnology and Economics. AndrásVetier Q U E U I N G. January 25, Supported by. Pro Renovanda Cultura Hunariae Alapítvány
Budapest University of Tecnology and Economics AndrásVetier Q U E U I N G January 25, 2000 Supported by Pro Renovanda Cultura Hunariae Alapítvány Klebelsberg Kunó Emlékére Szakalapitvány 2000 Table of
More informationStochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
More informationMarkov Chain Model for ALOHA protocol
Markov Chain Model for ALOHA protocol Laila Daniel and Krishnan Narayanan April 22, 2012 Outline of the talk A Markov chain (MC) model for Slotted ALOHA Basic properties of Discrete-time Markov Chain Stability
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 3
MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 Plan for rest of semester (1) st week (8/31, 9/6, 9/7) Chap 0: Diff eq s an linear recursion (2) n week (9/11...) Chap 1: Finite Markov chains (3) r week (9/18...)
More informationErgodic Properties of Markov Processes
Ergodic Properties of Markov Processes March 9, 2006 Martin Hairer Lecture given at The University of Warwick in Spring 2006 1 Introduction Markov processes describe the time-evolution of random systems
More informationIEOR 6711: Professor Whitt. Introduction to Markov Chains
IEOR 6711: Professor Whitt Introduction to Markov Chains 1. Markov Mouse: The Closed Maze We start by considering how to model a mouse moving around in a maze. The maze is a closed space containing nine
More informationµ n 1 (v )z n P (v, )
Plan More Examples (Countable-state case). Questions 1. Extended Examples 2. Ideas and Results Next Time: General-state Markov Chains Homework 4 typo Unless otherwise noted, let X be an irreducible, aperiodic
More information1 Continuous-time chains, finite state space
Université Paris Diderot 208 Markov chains Exercises 3 Continuous-time chains, finite state space Exercise Consider a continuous-time taking values in {, 2, 3}, with generator 2 2. 2 2 0. Draw the diagramm
More informationPower Series. Part 2 Differentiation & Integration; Multiplication of Power Series. J. Gonzalez-Zugasti, University of Massachusetts - Lowell
Power Series Part 2 Differentiation & Integration; Multiplication of Power Series 1 Theorem 1 If a n x n converges absolutely for x < R, then a n f x n converges absolutely for any continuous function
More informationMARKOV PROCESSES. Valerio Di Valerio
MARKOV PROCESSES Valerio Di Valerio Stochastic Process Definition: a stochastic process is a collection of random variables {X(t)} indexed by time t T Each X(t) X is a random variable that satisfy some
More informationEECS 495: Randomized Algorithms Lecture 14 Random Walks. j p ij = 1. Pr[X t+1 = j X 0 = i 0,..., X t 1 = i t 1, X t = i] = Pr[X t+
EECS 495: Randomized Algorithms Lecture 14 Random Walks Reading: Motwani-Raghavan Chapter 6 Powerful tool for sampling complicated distributions since use only local moves Given: to explore state space.
More informationISyE 6650 Probabilistic Models Fall 2007
ISyE 6650 Probabilistic Models Fall 2007 Homework 4 Solution 1. (Ross 4.3) In this case, the state of the system is determined by the weather conditions in the last three days. Letting D indicate a dry
More informationThe Theory behind PageRank
The Theory behind PageRank Mauro Sozio Telecom ParisTech May 21, 2014 Mauro Sozio (LTCI TPT) The Theory behind PageRank May 21, 2014 1 / 19 A Crash Course on Discrete Probability Events and Probability
More information6 Markov Chain Monte Carlo (MCMC)
6 Markov Chain Monte Carlo (MCMC) The underlying idea in MCMC is to replace the iid samples of basic MC methods, with dependent samples from an ergodic Markov chain, whose limiting (stationary) distribution
More informationHW 2 Solutions. The variance of the random walk is explosive (lim n Var (X n ) = ).
Stochastic Processews Prof Olivier Scaillet TA Adrien Treccani HW 2 Solutions Exercise. The process {X n, n } is a random walk starting at a (cf. definition in the course. [ n ] E [X n ] = a + E Z i =
More informationMarkov Chains and Stochastic Sampling
Part I Markov Chains and Stochastic Sampling 1 Markov Chains and Random Walks on Graphs 1.1 Structure of Finite Markov Chains We shall only consider Markov chains with a finite, but usually very large,
More informationMarkov Processes Hamid R. Rabiee
Markov Processes Hamid R. Rabiee Overview Markov Property Markov Chains Definition Stationary Property Paths in Markov Chains Classification of States Steady States in MCs. 2 Markov Property A discrete
More informationThe Markov Decision Process (MDP) model
Decision Making in Robots and Autonomous Agents The Markov Decision Process (MDP) model Subramanian Ramamoorthy School of Informatics 25 January, 2013 In the MAB Model We were in a single casino and the
More informationAdvanced Computer Networks Lecture 2. Markov Processes
Advanced Computer Networks Lecture 2. Markov Processes Husheng Li Min Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville Spring, 2016 1/28 Outline 2/28 1 Definition
More informationStatistics 253/317 Introduction to Probability Models. Winter Midterm Exam Friday, Feb 8, 2013
Statistics 253/317 Introduction to Probability Models Winter 2014 - Midterm Exam Friday, Feb 8, 2013 Student Name (print): (a) Do not sit directly next to another student. (b) This is a closed-book, closed-note
More informationMATH 56A SPRING 2008 STOCHASTIC PROCESSES
MATH 56A SPRING 008 STOCHASTIC PROCESSES KIYOSHI IGUSA Contents 4. Optimal Stopping Time 95 4.1. Definitions 95 4.. The basic problem 95 4.3. Solutions to basic problem 97 4.4. Cost functions 101 4.5.
More informationMarkov Chains, Stochastic Processes, and Matrix Decompositions
Markov Chains, Stochastic Processes, and Matrix Decompositions 5 May 2014 Outline 1 Markov Chains Outline 1 Markov Chains 2 Introduction Perron-Frobenius Matrix Decompositions and Markov Chains Spectral
More informationTCOM 501: Networking Theory & Fundamentals. Lecture 6 February 19, 2003 Prof. Yannis A. Korilis
TCOM 50: Networking Theory & Fundamentals Lecture 6 February 9, 003 Prof. Yannis A. Korilis 6- Topics Time-Reversal of Markov Chains Reversibility Truncating a Reversible Markov Chain Burke s Theorem Queues
More informationMarkov Chains on Countable State Space
Markov Chains on Countable State Space 1 Markov Chains Introduction 1. Consider a discrete time Markov chain {X i, i = 1, 2,...} that takes values on a countable (finite or infinite) set S = {x 1, x 2,...},
More informationChapter 2. Markov Chains. Introduction
Chapter 2 Markov Chains Introduction A Markov chain is a sequence of random variables {X n ; n = 0, 1, 2,...}, defined on some probability space (Ω, F, IP), taking its values in a set E which could be
More informationBayesian Methods with Monte Carlo Markov Chains II
Bayesian Methods with Monte Carlo Markov Chains II Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University hslu@stat.nctu.edu.tw http://tigpbp.iis.sinica.edu.tw/courses.htm 1 Part 3
More informationMarkov Chains for Everybody
Markov Chains for Everybody An Introduction to the theory of discrete time Markov chains on countable state spaces. Wilhelm Huisinga, & Eike Meerbach Fachbereich Mathematik und Informatik Freien Universität
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Simulated Annealing Barnabás Póczos & Ryan Tibshirani Andrey Markov Markov Chains 2 Markov Chains Markov chain: Homogen Markov chain: 3 Markov Chains Assume that the state
More informationNote that in the example in Lecture 1, the state Home is recurrent (and even absorbing), but all other states are transient. f ii (n) f ii = n=1 < +
Random Walks: WEEK 2 Recurrence and transience Consider the event {X n = i for some n > 0} by which we mean {X = i}or{x 2 = i,x i}or{x 3 = i,x 2 i,x i},. Definition.. A state i S is recurrent if P(X n
More information215 Problem 1. (a) Define the total variation distance µ ν tv for probability distributions µ, ν on a finite set S. Show that
15 Problem 1. (a) Define the total variation distance µ ν tv for probability distributions µ, ν on a finite set S. Show that µ ν tv = (1/) x S µ(x) ν(x) = x S(µ(x) ν(x)) + where a + = max(a, 0). Show that
More informationApplied Stochastic Processes
Applied Stochastic Processes Jochen Geiger last update: July 18, 2007) Contents 1 Discrete Markov chains........................................ 1 1.1 Basic properties and examples................................
More informationMarkov Chains and Computer Science
A not so Short Introduction Jean-Marc Vincent Laboratoire LIG, projet Inria-Mescal UniversitéJoseph Fourier Jean-Marc.Vincent@imag.fr Spring 2015 1 / 44 Outline 1 Markov Chain History Approaches 2 Formalisation
More informationInterlude: Practice Final
8 POISSON PROCESS 08 Interlude: Practice Final This practice exam covers the material from the chapters 9 through 8. Give yourself 0 minutes to solve the six problems, which you may assume have equal point
More informationMatrix analytic methods. Lecture 1: Structured Markov chains and their stationary distribution
1/29 Matrix analytic methods Lecture 1: Structured Markov chains and their stationary distribution Sophie Hautphenne and David Stanford (with thanks to Guy Latouche, U. Brussels and Peter Taylor, U. Melbourne
More informationA review of Continuous Time MC STA 624, Spring 2015
A review of Continuous Time MC STA 624, Spring 2015 Ruriko Yoshida Dept. of Statistics University of Kentucky polytopes.net STA 624 1 Continuous Time Markov chains Definition A continuous time stochastic
More informationNon-homogeneous random walks on a semi-infinite strip
Non-homogeneous random walks on a semi-infinite strip Chak Hei Lo Joint work with Andrew R. Wade World Congress in Probability and Statistics 11th July, 2016 Outline Motivation: Lamperti s problem Our
More informationQuantitative Model Checking (QMC) - SS12
Quantitative Model Checking (QMC) - SS12 Lecture 06 David Spieler Saarland University, Germany June 4, 2012 1 / 34 Deciding Bisimulations 2 / 34 Partition Refinement Algorithm Notation: A partition P over
More informationMarkov Processes on Discrete State Spaces
Markov Processes on Discrete State Spaces Theoretical Background and Applications. Christof Schuette 1 & Wilhelm Huisinga 2 1 Fachbereich Mathematik und Informatik Freie Universität Berlin & DFG Research
More informationAn Introduction to Entropy and Subshifts of. Finite Type
An Introduction to Entropy and Subshifts of Finite Type Abby Pekoske Department of Mathematics Oregon State University pekoskea@math.oregonstate.edu August 4, 2015 Abstract This work gives an overview
More informationLTCC. Exercises solutions
1. Markov chain LTCC. Exercises solutions (a) Draw a state space diagram with the loops for the possible steps. If the chain starts in state 4, it must stay there. If the chain starts in state 1, it will
More information1 Gambler s Ruin Problem
1 Gambler s Ruin Problem Consider a gambler who starts with an initial fortune of $1 and then on each successive gamble either wins $1 or loses $1 independent of the past with probabilities p and q = 1
More informationConvergence Rate of Markov Chains
Convergence Rate of Markov Chains Will Perkins April 16, 2013 Convergence Last class we saw that if X n is an irreducible, aperiodic, positive recurrent Markov chain, then there exists a stationary distribution
More information18.175: Lecture 30 Markov chains
18.175: Lecture 30 Markov chains Scott Sheffield MIT Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup Outline Review what you know
More informationLecture 10: Powers of Matrices, Difference Equations
Lecture 10: Powers of Matrices, Difference Equations Difference Equations A difference equation, also sometimes called a recurrence equation is an equation that defines a sequence recursively, i.e. each
More informationCourse 495: Advanced Statistical Machine Learning/Pattern Recognition
Course 495: Advanced Statistical Machine Learning/Pattern Recognition Lecturer: Stefanos Zafeiriou Goal (Lectures): To present discrete and continuous valued probabilistic linear dynamical systems (HMMs
More informationStochastic Models: Markov Chains and their Generalizations
Scuola di Dottorato in Scienza ed Alta Tecnologia Dottorato in Informatica Universita di Torino Stochastic Models: Markov Chains and their Generalizations Gianfranco Balbo e Andras Horvath Outline Introduction
More informationGärtner-Ellis Theorem and applications.
Gärtner-Ellis Theorem and applications. Elena Kosygina July 25, 208 In this lecture we turn to the non-i.i.d. case and discuss Gärtner-Ellis theorem. As an application, we study Curie-Weiss model with
More informationClassification of Countable State Markov Chains
Classification of Countable State Markov Chains Friday, March 21, 2014 2:01 PM How can we determine whether a communication class in a countable state Markov chain is: transient null recurrent positive
More informationIrreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1
Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate
More informationLet (Ω, F) be a measureable space. A filtration in discrete time is a sequence of. F s F t
2.2 Filtrations Let (Ω, F) be a measureable space. A filtration in discrete time is a sequence of σ algebras {F t } such that F t F and F t F t+1 for all t = 0, 1,.... In continuous time, the second condition
More informationMarkov chains. Randomness and Computation. Markov chains. Markov processes
Markov chains Randomness and Computation or, Randomized Algorithms Mary Cryan School of Informatics University of Edinburgh Definition (Definition 7) A discrete-time stochastic process on the state space
More informationSMSTC (2007/08) Probability.
SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................
More informationWinter 2019 Math 106 Topics in Applied Mathematics. Lecture 9: Markov Chain Monte Carlo
Winter 2019 Math 106 Topics in Applied Mathematics Data-driven Uncertainty Quantification Yoonsang Lee (yoonsang.lee@dartmouth.edu) Lecture 9: Markov Chain Monte Carlo 9.1 Markov Chain A Markov Chain Monte
More informationStochastic process. X, a series of random variables indexed by t
Stochastic process X, a series of random variables indexed by t X={X(t), t 0} is a continuous time stochastic process X={X(t), t=0,1, } is a discrete time stochastic process X(t) is the state at time t,
More information1.3 Convergence of Regular Markov Chains
Markov Chains and Random Walks on Graphs 3 Applying the same argument to A T, which has the same λ 0 as A, yields the row sum bounds Corollary 0 Let P 0 be the transition matrix of a regular Markov chain
More information12 Markov chains The Markov property
12 Markov chains Summary. The chapter begins with an introduction to discrete-time Markov chains, and to the use of matrix products and linear algebra in their study. The concepts of recurrence and transience
More informationMath Bootcamp 2012 Miscellaneous
Math Bootcamp 202 Miscellaneous Factorial, combination and permutation The factorial of a positive integer n denoted by n!, is the product of all positive integers less than or equal to n. Define 0! =.
More informationApproximate Counting and Markov Chain Monte Carlo
Approximate Counting and Markov Chain Monte Carlo A Randomized Approach Arindam Pal Department of Computer Science and Engineering Indian Institute of Technology Delhi March 18, 2011 April 8, 2011 Arindam
More informationModern Discrete Probability Spectral Techniques
Modern Discrete Probability VI - Spectral Techniques Background Sébastien Roch UW Madison Mathematics December 22, 2014 1 Review 2 3 4 Mixing time I Theorem (Convergence to stationarity) Consider a finite
More informationCS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions
CS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions Instructor: Erik Sudderth Brown University Computer Science April 14, 215 Review: Discrete Markov Chains Some
More informationA note on adiabatic theorem for Markov chains
Yevgeniy Kovchegov Abstract We state and prove a version of an adiabatic theorem for Markov chains using well known facts about mixing times. We extend the result to the case of continuous time Markov
More information