Learning in State-Space Reinforcement Learning CIS 32

Transcription

1 Learning in State-Space Reinforcement Learning CIS 32

2 Functionalia Syllabus Updated: MIDTERM and REVIEW moved up one day. MIDTERM: Everything through Evolutionary Agents. HW 2 Out - DUE Sunday before the MIDTERM. EVENING TEA: Next Monday, 5pm to 7pm, 0317 N Today: Training TLU s Recap Neural Networks Learning a Heuristic Search-Tree-Less Heuristic Learning Reinforcement Learning

3 Technique Error Correction Training TLU s Techniques Gradient Descent? No Widrow-Hoff Yes f = s = Threshold Function f = 1 if n! i=1 = 0 otherwise n x w i i # "! x i w i=1 i Generalized Delta Yes f (s) = e s

4 Weight Update Functions Technique Range of d (Desired Output) Range of f (Actual Training Output) Weight Update Error Correction 0 or 1 0 or 1 Widrow-Hoff -1 or 1 [-inf, +inf] Generalized Delta 0 or 1 [0, 1] sigmoid c is the learning rate parameter (small positive fraction)

5 Error-Correction Technique d f change c 1 0 +c Change in finite chunks. For small enough c: terminates after a finite number of steps (if the function is linearly separable) If the function is not linearly separable, does not terminate (but oscillates).

6 Example or Error Correction W 0 = 1.5 W 0 = 0.5? W 0 = 1.5 random W 0 = 0.5 W 0 = 0.5 W 1 = 1 W 2 = 1 W 1 = W1 1 = random W 1 = 1 W 1 = 1 W 1 = 1 W 2 = 1 W 2 = 1 random W 2 = 1 AND OR AND NOT Remember that the Threshold becomes rolled into the weights. We will start with random (can also be uniform) set of weights. Set our Learning Rate to 0.1

7 Example or Error Correction W 0 = 1.5 random W 0 = 0.5 V X1 X2 W 0 = 0.5 d W1 = random W 1 = 1 W 2 = 1 random AND W 1 = 1 W 2 = W 1 = OR training set NOT

8 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E random

9 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

10 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

11 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E f = 1 if n! i=1 = 0 otherwise x w i i # "

12 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E learning parameter (constant)

13 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

14 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

15 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

16 Train One Example a Time V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E Error:

17 After First Round V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

18 Second Round V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

19 Third Round V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E

20 Fourth Round V X1 X2 d w0 w1 w2 s f c d-f dw0 dw1 dw2 E Successfully Completed a Round with Changes Done

21 Gradient Descent in Weight Space Wa Gradient of error with respect to TLU s Weights (Wa, Wb) (wa0, wb0) (wa2, wb2) (wa1, wb1) Wb

22 Widrow-Hoff Technique f(s) = s Change to the weights in variable chunks. d uses -1 to represent training examples of 0 (pulls zero s below 0) threshold f(s) 1-1 training f(s) Process never terminates, but the differences in Error will be minimized.

23 After First Round V X1 X2 d w0 w1 w2 s f f c d-f dw0 dw1 dw2 E f = s Notice Wide Range in Error

24 After 10 Rounds V X1 X2 d w0 w1 w2 s f f c d-f dw0 dw1 dw2 E Good Enough

25 Round 200-something V X1 X2 d w0 w1 w2 s f f c d-f dw0 dw1 dw2 E Still Good Enough Converged And Decreasing

26 Generalized Delta Technique Steeper slope close to the threshold causes faster change near boundary Change to the weights in variable chunks. More fuzzy boundary. d uses 0 to represent training examples of 0 (instead of -1 for W-H). More modern threshold - used in multi-node networks.

27 After First Round V X1 X2 d w0 w1 w2 s f f c d-f f(1-f) dw0 dw1 dw2 E Uses larger learning rate Notice Smaller Range in Error

28 After 14 Rounds V X1 X2 d w0 w1 w2 s f f c d-f f(1-f) dw0 dw1 dw2 E Always ranges between -0.5 and 0.5

29 Network Structures Two kinds of larger Neural Network Structures: 1. feed-forward networks - acyclic contains hidden layers and inputs. 2. recurrent networks - cyclic dynamic systems - with oscillations and chaotic behavior can exhibit short-term memory

30 Hidden Units 1 W 1, 3 W 1, 4 3 W 3, W 2, 3 W 2, 4 4 W 4, 5 Activation of Unit 5 is based on the weighted outputs of Unit 3 and 4. Units 3 and 4 represent the hidden units. Activation depends on the Unit (can use the sigmoid function)

31 Layers are usually fully connected. Multi-layer Numbers of nodes typically set by hand.

32 Multilayer Feed Forward Layers are usually fully connected; numbers of nodes typically set by hand. Single Hidden Layer is Most Common. back-propagation

33 Larger hypothesis space Combine two opposite-facing threshold functions to make a ridge Combine two perpendicular ridges to make a bump Add bumps of various sizes and locations to fit any surface

34 Hopfield Networks - Recurrent Networks contain bidirectional connections (units are inputs and outputs) stimulus results in the networks settling into an activations pattern that most closely resembles a training example N units can store N training examples. Boltzmann Machines - like Hopfield Networks, but contain hidden units activation functions are stochastic (functions based on a probability that a unit exhibits a 1 based on the total weighted unit)

35 Learning in State Space We return now to heuristics (evaluation functions): used both in Search and Minimax Search. Having a good heuristics greatly improve s an agent s performance: (i.e. A* search, and in evaluating leaf nodes in Adversarial Search) Good Knowledge of Subject Domain Good heuristics No Knowledge of Subject Domain Learn the heuristic

36 more Levels of Reinforcement Learning knowledge about the problem domain less Agent knows it s actions, results, and costs; can build an explicit Search Tree to explore; has a clear short-term goal. Agent does not have a model of it s actions; can build an explicit Search Tree to explore; has a clear short-term goal. Agent does have a model of it s actions; but cannot (too large) build an explicit Search Tree to explore; has a clear shortterm goal state. Agent knows it s actions, results, and costs; cannot (too large) build an explicit Search Tree to explore; does not have a clear short-term goal. Performance based on Reward not Goals.

37 Explicit Graph Heuristic Learning Just as we did with previous searches, Agent: knows actions, their results, and costs has enough space to build an entire search tree. Set the heuristic function h(n) = 0 for all nodes, and do an A* search. Updates the h(n) once the node is expanded: Knows the goal state: h(goal) = 0 set of all children

38 Explicit Graph Learning Performance What kind of search is this - when the agent searches for the first time?

39 Explicit Graph Learning Performance Uniform Cost Search (f = g + 0)

40 Explicit Graph Learning Performance Subsequent searches zoom in on the right solution faster and faster. This happens as the true (h(n)) values propagate to the goal. h=1 2 2 h=2 h= h= h=1 h= h=1

41 Explicit Graph Learning Performance Each run propagates the true cost of getting to goal further back through the search. Eventually the minimal path can be read off the the tree. h=1 2 2 h=2 h= h= h=1 h= h=1

42 Explicit Graph Learning Performance Each run propagates the true cost of getting to goal further back through the search. Eventually the minimal path can be read off the the tree h=1 h=2 h=2 3 1 h= Agent goes through a thought experiment, uses a model of the State-Space. h=1 h= h=1

43 No Model of Action Heuristic Learning What if there is not clear model of action for state transition? Assuming agent can build, name, and store previous states......the agent can learn heuristics in the real-world. This can be perilous... Explore: A robot uses a grid to plan a route, moves randomly about the room. Exploit: Works out which runs about the room are the most optimal, and at what time were certain operations useful.

44 Updating the heuristic value of states Start Node: Agent knows the Cost of an action after taking it. States are Named and Stored, and can be Distinguished at a later State. Heuristic function for a State is updated: heuristic value of the node agent was just in cost of the transition (i.e. action) heuristic value of node transitioned to (initially 0 if not travelled to previously)

45 Choosing Actions Initially actions are chosen randomly. After some exploring, states have h(n) values ascribed to them. And there is model built of the actions: (describes the state (i.e. node n) that is reached from node ni after carrying out action a) Actions are now chosen by: Eventually the estimated minimum path to the goal is built up. Keeping some randomness allows for discovery of possibly more optimum paths to the goal.

46 Learning without a Search Graph (or Node Table) More realistic problems are so large: it is not possible to store all the states/node and build the entire search graph. Now, if we have a model of the actions, we can create and search with an evaluation function. Assemble a heuristic function out of as many sub-functions that can describe some value of a state-space. For the 8-puzzle it a list of functions could be: W(n) : number of tiles out of place P(n) : sum of distance of each tile from it s home Any other functions : usually relaxed heuristics.

47 Weighted Heuristic Function Write our heuristic function as a linear weighted combination: All we have to do now is learn which weights are the best. One way to do that, is to notice the difference in the heuristic value once we traverse from one node to another taking into consideration that cost:

48 Updating the Heuristic Learning Rate Set of Successor Nodes We modify h(ni) by adding some proportion of (controlled by ) of the difference of what we thought h(ni) was before expansion, what we think it is after. Once we know the change in h(ni), we adjust the weights similar to the Neural Networks.

49 Rewritten: Temporal Learning controls how fast the agent learns how much weight we give to the new estimate of the heuristic. Effect 0 no adjustment to h(ni) low high slow learning erratic performance 1 h(ni) is thrown away

50 Temporal Learning Called Temporal Learning - because the difference is based on the distance in one timed step. Note that this temporal difference approach can also work without a model of the effects of actions (with suitable modification).

51 Rewards not goals For many tasks agents don t have short term goals, but instead accrue rewards over a period of time. Instead of a plan, we want a policy act over time. which says how the agent should Typically this is expressed as what action should be carried out in a given state. Express the reward an agent gets as We want an optimal policy at every node. special reward for being in state nj which maximizes the (discounted) reward

52 Finding the Optimum Policy One (non-ideal) solution is to search through all policies (randomly) until a good one is discovered. Instead, given a certain policy, one can calculate the value of a node - the reward an agent will get if it starts at that node and follows the policy. Agent at ni and follows the policy to nj, then the agent can expect this reward (in the long-term): discounting factor - adds a little long-term goal

53 Value Iteration The optimum policy then gives us the action that maximizes this reward: If we knew what the values of the nodes were under easily compute the optimal policy:, then we could The problem is that we don t know these values. But we can find them out using value iteration. We start by guessing (randomly is fine) an estimated value V(n) for each node.

54 Approximating the Estimated Values Then when we are at ni we pick the action to maximize: that is the best thing given what we currently know. We then update V(ni) by: Progressive iterations of this calculation make V(n) a closer and closer approximation to Intuitively this is because we replace the estimate with the actual reward we get for the next state (and the next state and the next state).

55 Summary This lecture has looked at a number of approaches to learning heuristic functions. We started assuming that the agent knew everything but the heuristic, and progressively relaxed assumptions. This created a battery of reinforcement learning methods that can be applied in a wide variety of situations. These models also tie learning and planning together very closely, and we will revisit them as planning models later in the course.