Reinforcement Learning and Deep Reinforcement Learning

Transcription

1 Reinforcement Learning and Deep Reinforcement Learning Ashis Kumer Biswas, Ph.D. Deep Learning November 5, / 64

2 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

3 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

4 X=42 Reinforcement Learning Interpreter Environment Reward Action 1 An agent takes an action in an environment. State Agent Figure: The perception-action-learning loop. Image source Deep Learning November 5, / 64

5 X=42 Reinforcement Learning Interpreter Environment Reward State Agent Action Figure: The perception-action-learning loop. Image source 1 An agent takes an action in an environment. 2 That action is interpreted into a reward, R and a representation of the state, S. Deep Learning November 5, / 64

6 X=42 Reinforcement Learning Interpreter Environment Reward State Agent Action Figure: The perception-action-learning loop. Image source 1 An agent takes an action in an environment. 2 That action is interpreted into a reward, R and a representation of the state, S. 3 These two are then fed back into the agent. Deep Learning November 5, / 64

7 A Crawling Robot learns to crawl Figure: A Crawling robot developed by Francis wyffels. Video Link: Deep Learning November 5, / 64

8 Introduction to Markov Decision Process, MDP MDPs formally describe an environment for reinforcement learning, RL. where the environment is fully observable. That is, the current state completely characterizes the process. Almost all RL problems can be formalized as MDPs. Deep Learning November 5, / 64

9 Markov Property The future is independent of the past given the present. Definition A state S t is Markov if and only if P[S t+1 S t ] = P[S t+1 S 1,, S t ] The state captures all relevant information from the history. Once the state is known, the history may be thrown away. i.e., the state is a sufficient statistic of the future. Deep Learning November 5, / 64

10 State Transition Matrix For a Markov state, s, and the successor state s, the state transition probability is defined as: P ss = P[S t+1 = s S t = s] State transition matrix, P defines transition probabilities from all states s to all successor states, s, P 11 P 1n P =..... P n1 P nn where each row of the matrix sums to 1. Deep Learning November 5, / 64

11 Markov Process A Markov Process is a memoryless random process, i.e., a sequence of random states S 1, S 2,, with the Markov property. Definition A Markov Process (or Markov Chain) is a tuple, S, P, such that: S is a finite set of states. P is a state transition probability matrix: P ss = P[S t+1 = s S t = s] Deep Learning November 5, / 64

12 Example: Student Markov Chain Sample episodes for Student Markov Chain starting from S 1 = C1. C1, C2, C3, Pass, Sleep Deep Learning November 5, / 64

13 Example: Student Markov Chain Sample episodes for Student Markov Chain starting from S 1 = C1. C1, C2, C3, Pass, Sleep C1, FB, FB, C1, C2, Sleep Deep Learning November 5, / 64

14 Example: Student Markov Chain Sample episodes for Student Markov Chain starting from S 1 = C1. C1, C2, C3, Pass, Sleep C1, FB, FB, C1, C2, Sleep C1, C2, C3, Pub, C2, C3, Pass, Sleep Deep Learning November 5, / 64

15 Example: Student Markov Chain Sample episodes for Student Markov Chain starting from S 1 = C1. C1, C2, C3, Pass, Sleep C1, FB, FB, C1, C2, Sleep C1, C2, C3, Pub, C2, C3, Pass, Sleep C1, FB, FB, C1, C2, C3, Pub, C1, FB, FB, FB, C1, C2, C3, Pub, C2, Sleep Deep Learning November 5, / 64

16 Example: Student Markov Chain Deep Learning November 5, / 64

17 Markov Reward Process, MRP A Markov reward process is a Markov Chain with values. Definition A Markov Reward Process is a tuple S, P, R, γ Deep Learning November 5, / 64

18 Markov Reward Process, MRP A Markov reward process is a Markov Chain with values. Definition A Markov Reward Process is a tuple S, P, R, γ S is a finite set of states. P is a state transition probability matrix, P ss = P[S t+1 = s S t = s] Deep Learning November 5, / 64

19 Markov Reward Process, MRP A Markov reward process is a Markov Chain with values. Definition A Markov Reward Process is a tuple S, P, R, γ S is a finite set of states. P is a state transition probability matrix, P ss = P[S t+1 = s S t = s] R is a reward function, R s = E[R t+1 S t = s] Deep Learning November 5, / 64

20 Markov Reward Process, MRP A Markov reward process is a Markov Chain with values. Definition A Markov Reward Process is a tuple S, P, R, γ S is a finite set of states. P is a state transition probability matrix, P ss = P[S t+1 = s S t = s] R is a reward function, R s = E[R t+1 S t = s] γ is a discount factor, γ [0, 1] Deep Learning November 5, / 64

21 The Student MRP Deep Learning November 5, / 64

22 Return Definition The return, G t is the total discounted reward from time-step t: G t = R t+1 + γr t+2 + γ 2 R t+3 = γ k R t+k+1 k=0 The discount, γ [0, 1] is the present value of future rewards. The value of receiving reward, R after k + 1 time-steps is γ k R. This values immediate reward above delayed reward: γ close to 0 leads to myopic evaluation. γ close to 1 leads to far-sighted evaluation. Deep Learning November 5, / 64

23 Why discount? Most Markov reward and decision processes are discounted. Why? Avoids infinite returns in cyclic Markov Processes. Uncertainty about the future may not be fully represented. If the reward is financial, immediate rewards may earn more interest than delayed rewards. Animal/human behavior shows preference for immediate reward. Deep Learning November 5, / 64

24 Value function The value function v(s) gives the long-term value of state s. Definition The state value function v(s) of an MRP is the expected return starting from state s: v(s) = E[G t S t = s] Deep Learning November 5, / 64

25 The Student MRP Sample returns for Student MRP, starting from S 1 = C1 with γ = 0.5 G 1 = R 2 + γr 3 + C1, C2, C3, Pass, Sleep v 1 = = 2.25 Deep Learning November 5, / 64

26 The Student MRP Sample returns for Student MRP, starting from S 1 = C1 with γ = 0.5 G 1 = R 2 + γr 3 + C1, FB, FB, C1, C2, Sleep v 1 = = Deep Learning November 5, / 64

27 The Student MRP Sample returns for Student MRP, starting from S 1 = C1 with γ = 0.5 G 1 = R 2 + γr 3 + C1, FB, FB, C1, C2, Sleep v 1 = = Deep Learning November 5, / 64

28 State-Value function for Student MRP (γ = 0) A myopic evaluation!!! Deep Learning November 5, / 64

29 State-Value function for Student MRP (γ = 0.9) Deep Learning November 5, / 64

30 State-Value function for Student MRP (γ = 1) A far-sighted evaluation!!! Deep Learning November 5, / 64

31 Bellman Equation for MRPs The value function, v(s) can be decomposed into two parts: immediate reward, R t+1. Deep Learning November 5, / 64

32 Bellman Equation for MRPs The value function, v(s) can be decomposed into two parts: immediate reward, R t+1. discounted value of successor state γv(s t+1 ) Deep Learning November 5, / 64

33 Bellman Equation for MRPs The value function, v(s) can be decomposed into two parts: immediate reward, R t+1. discounted value of successor state γv(s t+1 ) v(s) = E[G t S t = s] = E[R t+1 + γr t+2 + γ 2 R t+3 + S t = s] = E[R t+1 + γ(r t+2 + γr t+3 + ) S t = s] = E[R t+1 + γg t+1 S t = s] = E[R t+1 + γv(s t+1 ) S t = s] Deep Learning November 5, / 64

34 Bellman Equation for MRPs it can be represented as: v(s) = E[R t+1 + γv(s t+1 ) S t = s] v(s) = R s + γ s S P ss v(s ) Deep Learning November 5, / 64

35 Bellman Equation for Student MRP Deep Learning November 5, / 64

36 Bellman Equation in Matrix Form The Bellman equation can be expressed concisely using matrices, v = R + γpv where, v is a column vector with one entry per state. v(1). v(n) = R 1 P 11 P 1n v(1). + γ R n P n1 P nn v(n) Deep Learning November 5, / 64

37 Solving the Bellman Equation It is a linear equation. Deep Learning November 5, / 64

38 Solving the Bellman Equation It is a linear equation. It can be solved directly:: v = R + γpv (1 γp)v = R v = (1 γp) 1 R Deep Learning November 5, / 64

39 Solving the Bellman Equation It is a linear equation. It can be solved directly:: v = R + γpv (1 γp)v = R v = (1 γp) 1 R Computational complexity is O(n 3 ) for n states. Direct solution only possible for small MRPs. Deep Learning November 5, / 64

40 Solving the Bellman Equation It is a linear equation. It can be solved directly:: v = R + γpv (1 γp)v = R v = (1 γp) 1 R Computational complexity is O(n 3 ) for n states. Direct solution only possible for small MRPs. There are iterative solutions for large MRPs developed: Dynamic Programming Deep Learning November 5, / 64

41 Solving the Bellman Equation It is a linear equation. It can be solved directly:: v = R + γpv (1 γp)v = R v = (1 γp) 1 R Computational complexity is O(n 3 ) for n states. Direct solution only possible for small MRPs. There are iterative solutions for large MRPs developed: Dynamic Programming Monte-Carlo Evaluation Deep Learning November 5, / 64

42 Solving the Bellman Equation It is a linear equation. It can be solved directly:: v = R + γpv (1 γp)v = R v = (1 γp) 1 R Computational complexity is O(n 3 ) for n states. Direct solution only possible for small MRPs. There are iterative solutions for large MRPs developed: Dynamic Programming Monte-Carlo Evaluation Temporal Difference Learning Deep Learning November 5, / 64

43 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ Deep Learning November 5, / 64

44 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ S is a finite set of states. Deep Learning November 5, / 64

45 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ S is a finite set of states. A is a finite set of actions. Deep Learning November 5, / 64

46 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ S is a finite set of states. A is a finite set of actions. P is a state transition probability matrix, P a ss = P[S t+1 = s S t = s, A t = a] Deep Learning November 5, / 64

47 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ S is a finite set of states. A is a finite set of actions. P is a state transition probability matrix, P a ss = P[S t+1 = s S t = s, A t = a] R is a reward function, R a s = E[R t+1 S t = s, A t = a] Deep Learning November 5, / 64

48 Markov Decision Process, MDP A Markov Decision Process, MDP is a Markov Reward Process with decisions. It is an environment in which all states are Markov. Definition A Markov Decision Process is a tuple S, A, P, R, γ S is a finite set of states. A is a finite set of actions. P is a state transition probability matrix, P a ss = P[S t+1 = s S t = s, A t = a] R is a reward function, R a s = E[R t+1 S t = s, A t = a] γ is a discount factor, γ [0, 1] Deep Learning November 5, / 64

49 Student Markov Process Deep Learning November 5, / 64

50 Student Markov Reward Process, MRP Deep Learning November 5, / 64

51 Student Markov Decision Process, MDP Deep Learning November 5, / 64

52 Policies (1 of 2) Definition A policy, π is a distribution over actions given states, π(a s) = P[A t = a S t = s] A policy fully defines the behavior of an agent. MDP policies depend on the current state (and not the history). That is, policies are stationary (time independent), A t π(. s t ), t > 0 Deep Learning November 5, / 64

53 Policies (2 of 2) Given an MDP, M = S, A, P, R, γ, and a policy, π The state sequence S 1, S 2, is a Markov process S, P π The state and reward sequence S 1, R 2, S 2, is a Markov Reward Process, MRP: S, P π, R π, γ, where P π ss = a A π(a s)p a ss R π s = a A π(a s)r a s Deep Learning November 5, / 64

54 Value Functions Definition The state-value function, v π (s) of an MDP is the expected return starting from state s, and then following policy π. v π (s) = E π [G t S t = s] Deep Learning November 5, / 64

55 Value Functions Definition The state-value function, v π (s) of an MDP is the expected return starting from state s, and then following policy π. v π (s) = E π [G t S t = s] Definition The action-value function q π (s, a) is the expected return starting from state s, taking action a, and then following policy π. q π (s, a) = E π [G t S t = s, A t = a] Deep Learning November 5, / 64

56 Example: State-Value Function for Student MDP Deep Learning November 5, / 64

57 Bellman Expectation Equation The state-value function can again be decomposed into immediate reward plus discounted value of successor state, v π (s) = E π [R t+1 + γv π (S t+1 ) S t = s] The action-value function can also be decomposed as, q π (s, a) = E π [R t+1 + γq π (S t+1, A t+1 ) S t = s, A t = a] Deep Learning November 5, / 64

58 Bellman Expectation Equation for v π (.) v π (s) = a A π(a s)q π (s, a) Deep Learning November 5, / 64

59 Bellman Expectation Equation for q π (.) (1 of 2) q π (s, a) = R a s + γ s S P a ss v π(s ) Deep Learning November 5, / 64

60 Bellman Expectation Equation for v π (.) v π (s) = π(a s) R a s + γ Pss a v π(s ) a A s S Deep Learning November 5, / 64

61 Bellman Expectation Equation for q π (.) (2 of 2) q π (s, a) = R a s + γ s S Pss a π(a s )q π (s, a ) a A Deep Learning November 5, / 64

62 Bellman Expectation Equation in Student MDP Deep Learning November 5, / 64

63 Optimal Value Function Definition The optimal state-value function, v (s) is the maximum state-value function over all policies v (s) = max π v π (s) Definition The optimal action-value function, q (s, a) is the maximum action-value function over all policies q (s, a) = max π q π (s, a) The optimal value function specifies the best possible performance in the MDP. An MDP is considered solved when we know the optimal value function. Deep Learning November 5, / 64

64 Optimal State-Value function for the Student MDP Deep Learning November 5, / 64

65 Optimal Action-Value function for the Student MDP Deep Learning November 5, / 64

66 Optimal Policy It defines a partial ordering over the policies, π π if v π (s) v π (s), s Theorem For any Markov Decision Process, There exists an optimal policy, π that is better than or equal to all other policies, π π, π. All optimal policies achieve the optimal state value function, v π (s) = v (s) All optimal policies achieve the optimal action value function, q π (s, a) = q (s, a) Deep Learning November 5, / 64

67 Finding an optimal policy An optimal policy can be found by maximizing over q (s, a), 1 if a = argmax q (s, a) π (a s) = a A 0 otherwise There is always a deterinistic optimal policy for any MDP. If we know q (s, a), we immediately have the optimal policy. Deep Learning November 5, / 64

68 Example: Optimal Policy for the Student MDP Deep Learning November 5, / 64

69 Bellman Optimality Equation for v The optimal value functions are recursively related by the Bellman optimality equations: v (s) = max a q (s, a) Deep Learning November 5, / 64

70 Bellman Optimality Equation for q q (s, a) = R a s + γ s S P a ss v (s ) Deep Learning November 5, / 64

71 Bellman Optimality Equation for v (2) v (s) = max a R a s + γ Pss a v (s ) s S Deep Learning November 5, / 64

72 Bellman Optimality Equation for q (2) q (s, a) = R a s + γ s S P a ss max a q (s, a ) Deep Learning November 5, / 64

73 Bellman Optimality Equation for the Student MDP Deep Learning November 5, / 64

74 Solving the Bellman Optimality Equation Bellman Optimality Equation is non-linear. No closed form solution (in general). However, many iterative solution available: Policy gradient Q-learning SARSA Deep Learning November 5, / 64

75 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

76 The Q value How to find the optimal π? How does the agent learn by interacting with the environment? Instead of finding the policy that maximizes the state-value for all states, find the action that maximizes the Q value for all states. Deep Learning November 5, / 64

77 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

78 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

79 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

80 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

81 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

82 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

83 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

84 Outlines 1 Principles of Reinforcement Learning 2 The Q value 3 Q-learning example 4 Q-learning in Python 5 Non-deterministic Environment 6 Temporal difference Learning 7 Q-learning on OpenAI Gym 8 Deep Q-network (DQN) 9 DQN on Keras 10 Double DQN (DDQN) Deep Learning November 5, / 64

85 Thanks Questions? Deep Learning November 5, / 64