Final Exam. CS 188 Fall Introduction to Artificial Intelligence

Transcription

1 CS 188 Fall 2018 Introduction to rtificial Intelligence Final Exam You have 180 minutes. The time will be projected at the front of the room. You may not leave during the last 10 minutes of the exam. Do NOT open exams until told to. Write your SIDs in the top right corner of every page. If you need to go to the bathroom, bring us your exam, phone, and SID. We will record the time. In the interest of fairness, we want everyone to have access to the same information. To that end, we will not be answering questions about the content. If a clarification is needed, it will be projected at the front of the room. Make sure to periodically check the clarifications. The exam is closed book, closed laptop, and closed notes except your three-page double-sided cheat sheet. You are allowed a non-programmable calculator for this exam. Turn off and put away all other electronics. The last two sheets in your exam are scratch paper. Please detach them from your exam. Mark your answers ON THE EXM IN THE DESIGNTED NSWER RES. We will not grade anything on scratch paper. For multiple choice questions: means mark LL options that apply means mark ONE choice When selecting an answer, please fill in the bubble or square COMPLETELY ( and ) First name Last name SID Student to the right (SID and Name) Student to the left (SID and Name) Q1. gent Testing Today! /1 Q2. Short Questions /14 Q3. Treasure Hunting MDPs /12 Q4. pproximate Q-learning /8 Q5. Value of symmetric Information /16 Q6. Bayes Net Modeling /12 Q7. Help the Farmer! /14 Q8. Bayes Nets and RL /15 Q9. Decision Trees /8 Total /100 1

2 THIS PGE IS INTENTIONLLY LEFT BLNK

3 Q1. [1 pt] gent Testing Today! SID: It s testing time! Not only for you, but for our CS188 robots as well! Circle your favorite robot below. ny answer was acceptable. 3

4 Q2. [14 pts] Short Questions (a) [2 pts] Which of the following properties must a set of preferences satisfy if they are rational: ( B) OR (B ) OR (B ) (B ) ND (C ) (C B) ( B) ND (B C) [p, ; 1 p, B] [q, B; 1 q, C] B C D [p, ; 1 p, C] [p, B; 1 p, D] (B ) ND (C B) (C ) The first one is the orderability axiom that all rational preferences must satisfy. The second one is false, you could consider the utility function U() = 1, U(B) = 2, U(C) = 3 which satisfies the premises but U(B) U(C). The second one is true since, B, and C are equally preferable any lottery with those preferences is also equally preferable. It follows directly from the substitutability axiom. The fourth and fifth statements follow from the transitivity axiom. (b) [2 pts] Which of the following are true? Given a set of preferences there exists a unique utility function. U(x) = x 4 is a risk prone utility U(x) = 2x is a risk prone utility For any specific utility function, any lottery can be replaced by an appropriate deterministic utility value For the lotteries = [0.8, $4000; 0.2, $0], B = [1.0, $3000; 0.0, $0] we have B Given a set of preferences there exists an infinite number of utility functions, you may consider a fixed utility function and then apply a monotonic increasing function to it. This will result in a new utility function for that set of preferences. The second statement is risk prone since U(L) > U(EMV(L)). The third statement is risk neutral U(L) = U(EMV(L)). The fourth statement is true since given a lottery and a utility function, we consider the utility of it as the expected utility value for that lottery. The last statement is false: we can consider the utility function log(x), then we have 0.8 log(4000) < log(3000). (c) [2 pts] Which of the following paths is a feasible trajectory for the gradient ascent algorithm? 4

5 SID: B C D E F is a gradient ascent path since the gradient lines are orthogonal to the contours and the point towards the maximum. B is also a gradient ascent path with a high learning rate. C is not because the path is going towards the minimum instead of the maximum. D is not a gradient ascent path since the gradient is not orthogonal to the contour lines. E is not a gradient ascent path since it starts going towards the minimum. F is not since it goes towards the minimum and the gradients are not orthogonal to the contour lines. 5

6 (d) We are given the following 5 neural networks (NN) architectures. The operation represents the matrix multiplication operation, [w i1...w ik ] and [b i1...b ik ] represents the weights and the biases of the NN, the orientation (vertical and horizontal) its just for consistency in the operations. The term [ReLU σ] in B means applying a ReLU activation to the first element of the vector and a sigmoid (σ) activation to the second element. These operations are depicted in the following figures: Which of the following neural networks can represent each function? (i) [2 pts] f I (x) : B C D E and B cannot represent this plot since the ReLU activation results in a flat semi-line. C can represent it by having w 11 = 1, w 12 = 1, w 21 = 1, w 22 = 1 and the rest of the parameters being 0. D cannot because the sigmoid activations are non-linear. E is a linear function and therefore can represent the identity. (ii) [2 pts] f II (x) : B C D E This is a piecewise linear function with 6 pieces. s a result you can represent it by lineraly combining 5 ReLU functions. The only possible graph that can represent this function is C. 6

7 (iii) [2 pts] f III (x): B C D E This is a piecewise linear function with 2 pieces. This can be obtained by linearly combining to ReLU functions. The only possible solution is then C. Note that cannot represent this function since the last ReLU of the network would result in a flat semi-line. (iv) [2 pts] f IV (x) : B C D E This function corresponds to a scaled sigmoid function., C, E cannot represent any non-linear or nonpiecewise linear functions. B can represent it by setting w 11 = b 11 = 0. D does not work because the composition of sigmoid with sigmoid is not a sigmoid. SID: 7

8 Q3. [12 pts] Treasure Hunting MDPs In each below Gridworld, Pacman s starting position is denoted as P. G denotes the goal. t the goal, Pacman can only Exit, which will cause Pacman to exit the Gridworld and gain reward The Gridworld also has K gold nuggets g 1,..., g k which will have the properties listed below in each part. Pacman automatically picks up a gold nugget if he enters a square containing one. Finally, define P 0 as Pacman s initial state, where he is in position P and has no gold nuggets. (a) (b) (c) (a) Pacman now ventures into (a). Each gold nugget Pacman holds when he exits will add +100 to his reward, and he will receive +100 when he exits from the goal G. (i) [2 pts] When conducting value iteration, what is the first iteration at which V (P 0 ) is nonzero? nswer: 5 The shortest number of iterations is 5 because it takes 4 timesteps for Pacman to go from P to G, and 1 more timestep for Pacman to exit from the goal. It takes far more iterations for Pacman to gain reward by picking up g 1 and then exiting from G. (ii) [2 pts] ssume Pacman will act optimally. What nonzero discount factor γ ensures that the policy of picking up g 1 before going to goal G and the policy of going straight to G yield the same reward? nswer: ( 1 2 )1/6 To solve for γ, we have the equation 200γ 11 = 100γ 5, where the left hand is the reward after picking up g 1, and the right hand is the reward after going straight to and exiting from the goal G. Solving this equation yields γ 6 = 1 2, γ = (b) Pacman is now at (b), which contains a Gold Store (S). He will receive +5 per nugget. When at the Store, Pacman can either Sell to sell all his gold for +5 per nugget or Exit to exit the Gridworld. Exiting from the Store yields 0 reward. Exiting from goal G will give k, where Pacman has k nuggets. Note that Pacman can also only carry one gold nugget at a time. (i) [2 pts] When conducting value iteration, what is the first iteration at which V (P 0 ) is nonzero? nswer: 7 It takes 7 steps for Pacman to pick up g 3 and carry it to the Store to sell it. ll other methods of obtaining reward, such as exiting from goal G, selling g 1, selling g 2, or selling some combination of nuggets, takes more time than to directly carry g 3 to the store and sell it. The distance between P and S is 6, as Pacman picks g 3 up on the way, and 1 more timestep is needed to do the action Sell. (ii) [2 pts] Now Pacman is in a world with a Store that is not necessarily the Gridworld (b). ssume Pacman is acting optimally, and he begins at the Store. It takes Pacman time T 1, T 2, T 3 to go from the Store, pick up the nuggets g 1, g 2, g 3 respectively, return to the store, and sell each nugget. It takes time T G to go from the Store and exit from the goal G. ssume T 1 < T 2 < T 3 < T G. What must be true such that the better policy for Pacman would be to gather and sell all nuggets and exit from the store rather than to gather all nuggets and exit from goal G? 5(γ T1 +γ T1+T2 +γ T1+T2+T3 ) > 115γ T1+T2+T3+T G 5(γ T1 + γ T2 + γ T3 ) > 100γ T G 5(γ T1 + γ T1+T2 + γ T1+T2+T3 ) > 115γ T G 5(γ T1 + γ T1+T2 + γ T1+T2+T3 ) > 100γ T G 15γ T1+T2+T3 > 115γ T G None of the above 8

9 The first solution, which was intended but incorrect, as well as None of the above were both awarded points due to unclear problem instructions. The clarification that Pacman cannot use the Store if he holds multiple gold nuggets was added during the exam. The intended solution was to select the first option. However, it is always more optimal to pick up the gold nuggets in a single round trip, rather than returning to the store each time after a nugget is picked up, so the expression 115γ T1+T2+T3+T G does not denote the reward if Pacman acts optimally and exits from the goal. Therefore, if we specify some time T G that is the optimal time to pick up all round trip gold nuggets and exit via the goal, then the reward when acting optimally and exiting from the goal is 115γ T G. However, we never specified any such value T G, and the correct answer was not one of the listed five inequalities. Because the true answer was not included in the options, None of the above is the correct answer. (c) Finally, Pacman finds himself in Gridworld (c). There is no store. However, Pacman finds that there is now a living reward! He gets the living reward for every action he takes except the Exit action. Pacman receives +0 exiting from the Door, and +100 exiting from the Goal. Once in the Door, Pacman can only Exit. (i) [2 pts] Suppose γ = 0.5. For what living reward will Pacman receive the same reward whether he exits via the Door or exits via the goal? nswer: 50 7 The rewards are the same if RL R L R L R L = R L. Solving this yields 7 8 R L = , which reduces to R L = = (ii) [2 pts] Suppose γ = 0.5. What is the living reward such that Pacman receives the same reward if he traverses the Gridworld forever or if he goes straight to and exits from the goal? Hint: t=0 rγt = r 1 γ nswer: 50 SID: R He would want to go around forever if L = R L, which can be reduced to 2R L = R L, or 1 8 R L = = 50. The 8 in the first equation is from R L R L R L R L that Pacman accumulated on the way to the goal G. 9

10 Q4. [8 pts] pproximate Q-learning (a) [2 pts] Pacman is trying to collect all food pellets, and each treasure chest contains 10 pellets but must be unlocked with a key. Pacman will automatically pick up a pellet or key when he enters a square containing them, and he will automatically unlock a chest when he enters a square with a chest and has at least one key. key can only unlock on chest; after being used, it vanishes. To finish, Pacman must exit through either goal G 1 or G 2. The keys are shown on the map as a K, treasure chests as T, and food pellets as circles. Pacman s starting position is shown as P. The goals are G 1, G 2. When calculating features for a Q-function Q(s, a), the state the features are calculated with is the state s Pacman is in after taking action a from state s. The possible features the Q-learning can use are: N keys : Number of keys Pacman holds. D m (K): Manhattan distance to closest key. N chests : Number of chests Pacman has unlocked. D m (T ): Manhattan distance to closest chest. N food : Number of food pellets Pacman has eaten. D m (F ): Manhattan distance to closest food pellet. D m (G 1 ): Manhattan distance to G 1. D m (G 2 ): Manhattan distance to G 2. Note that the approximate Q-learning here can be any function over the features, not necessarily a weighted linear sum. Suppose we finished training an agent using approximate Q-learning and we then run the learned policy to observe the following. ssume we observe all the above paths. What is the minimal set of features that could have been used to learn this policy? N keys D m (K) N chests D m (T ) N food D m (F ) D m (G 1 ) D m (G 2 ) No features Due to the phrasing of the problem, there is some ambiguity regarding the precise minimal feature set. Full credit was given to answers that chose D m (G 1 ), at least one of {N keys, D m (K)}, at least one of {N food, D m (F )}, and did not pick any of {N chests, D m (T ), D m (G 2 )}. In episodes 1 and 3 above, Pacman walks in the direction of G 1 ; of the features provided only D m (G 1 ) could be used to learn this behavior. Based on episode 3 above, Pacman is willing to move away from G 1 to gather a food pellet. Since the food pellet is just one square away, either N food or D m (F ) could be used to learn this behavior. t the same time, after collecting the first key Pacman chooses to collect the second key instead of picking up the pellet right away; this can be explained by the use of either N keys or D m (K). Moving towards/away from the treasure chest is not required to explain Pacman s behavior, so the choices N chests and D m (T ) are incorrect. The feature D m (G 2 ) is not required, either. In episode 2, Pacman begins by collecting the key and two food pellets. t that point, moving either up or right would bring Pacman closer to G 1, so Pacman could have 10

11 broken the tie in favor of G 2, at which point Pacman happens to stumble across the second goal and the episode ends. (b) Suppose Pacman is now in an empty grid of size M M. For a Q-value Q(s, a), the features are the x- and y-position of the state Pacman is in after taking action a. Select Possible with weighted sum if the policy can be expressed by using a weighted linear sum to represent the Q-function. Select Possible with large neural net if the policy can be expressed by using a large neural net to represent the Q-function. Select Not Possible if expressing the policy with given features is impossible no matter the function. (i) [2 pts] Pacman s optimal policy is always to go upwards. Possible with large neural net Possible with weighted linear sum Not Possible (ii) [2 pts] We draw a vertical line and divide the Gridworld into two halves. SID: On the left half, Pacman s optimal policy is to go upwards, and on the right half, Pacman s optimal policy is to go downwards. Possible with large neural net Possible with weighted linear sum Not Possible (iii) [2 pts] We draw a vertical line and divide the Gridworld into two equal halves. On the left half, Pacman s optimal policy is to go upwards, and on the right half, Pacman s optimal policy is to go right. Possible with large neural net Possible with weighted linear sum Not Possible The key in this part is understanding that a large enough neural net can approximate any function, so it is always possible to use a large neural net to represent the Q-functions and express a certain policy. The first is possible with a weighted linear sum, for example, 0 x + 1 y, so Q-values are larger for states with larger y values. The second is not possible with a weighted linear sum. To make going up optimal, Q-values with larger y-values must be similarly larger. In the right half, this must be the opposite. However, the x features for two positions (x, y+1) and (x, y-1), e.g., one position is higher than the other, are the same, meaning that the x feature can only create offsets between the weighted linear sums for those states and not actually reverse the magnitude to cause (x, y-1) to have a larger Q-value than (x, y+1). That is, for some weights w x, w y, the difference in Q-values between (x, y+1), (x, y-1) is w x x+w y (y+1) (w x x+w y (y 1)) = w x x+w y y+w y w x x w y y+w y = 2w y, and is independent of both x-position and w x. The third is not possible with a weighted linear sum. For two positions (x, y+1) and (x + 1, y), we can calculate the difference in Q-values for the given positions given any weights w x, w y. The difference is then w x x + w y (y + 1) (w x (x + 1) + w y y) = w x x + w y y + w y w x x w x w y y = w y w x. We see then that this is actually completely independent of the positions and the difference will therefore always be fixed; Pacman cannot prefer to go up in some situations and right in others. 11

12 Q5. [16 pts] Value of symmetric Information lice and Bob are playing an adversarial game as shown in the game tree below. lice (the MX player) and Bob (the MIN player) are both rational and they both know that their opponent is also a rational player. The game tree has one chance node E whose outcome can be either E = 100 or E = +120 with equal 0.5 probability. lice Bob -20 E Each player s utility is equal to the amount of money he or she has. The value x of each leaf node in the game tree means that Bob will pay lice x dollars after the game, so that lice and Bob s utilities will be x and x respectively. (a) [2 pts] Suppose neither lice nor Bob knows the outcome of E before playing. What is lice s expected utility? nswer: 10 E s expectation is 10. Using minimax, both lice and Bob should go left. (b) Carol, a good friend of lice s, has access to E and can secretly tell lice the outcome of E before the game starts (giving lice the true outcome of E without lying). However, Bob is not aware of any communication between lice and Carol, so he still assumes that lice has no access to E. (i) [1 pt] Suppose Carol secretly tells lice that E = 100. What is lice s expected utility in this case? nswer: -20 Here, Bob will still go left as before since he isn t aware of lice s access to E. Given this, lice should now choose to go right when E = 100. (ii) [1 pt] Suppose Carol secretly tells lice that E = What is lice s expected utility in this case? nswer: 120 Here, Bob will still go left as before since he isn t aware of lice s access to E. Given this, lice should now choose to go left when E = (iii) [1 pt] What is lice s expected utility if Carol secretly tells lice the outcome of E before playing? nswer: = E is equally likely to be 100 or veraging the two cases above, We define the value of private information V pri (X) of a random variable X to a player as the difference in player s expected utility after the outcome of X becomes a private information to player, such that has access to the outcome of X, while other players have no access to X and are not aware of s access to X. (iv) [2 pts] In general, the value of private information V pri (X) of a variable X to a player always satisfies V pri (X) > 0 in all cases. always satisfies V pri (X) 0 in all cases. always satisfies V pri (X) = 0 in all cases. can possibly satisfy V pri (X) < 0 in certain cases. Since player can always choose to ignore this information and act in the same way as if he/she doesn t know this information, player is guaranteed to obtain at least the same utility as before, so V pri 12 (X) 0.

13 (v) [1 pt] What is V pri lice (E), the value of private information of E to lice in the specific game tree above? SID: nswer: 40 Subtracting the answer of (a) from the answer of (b, iii), = 40. (c) David also has access to E, and can make a public announcement of E (announcing the true outcome of E without lying), so that both lice and Bob will know the outcome of E and are both aware that their opponent also knows the outcome of E. lso, lice cannot obtain any information from Carol now. (i) [1 pt] Suppose David publicly announces that E = 100. What is lice s expected utility in this case? nswer: -20 Using minimax with E = 100, Bob will go left and lice will go right. (ii) [1 pt] Suppose David publicly announces that E = What is lice s expected utility in this case? nswer: 60 Using minimax with E = +120, Bob will go right and lice will go left. (iii) [1 pt] What is lice s expected utility if David makes a public announcement of E before the game starts? nswer: = E is equally likely to be 100 or veraging the two cases above, We define the value of public information V pub (X) of a random variable X to a player as the difference in player s expected utility after the outcome of X becomes a public information, such that everyone has access to the outcome of X and is aware that all other players also have access to X. (iv) [2 pts] In general, the value of public information V pub (X) of a variable X to a player always satisfies V pub (X) > 0 in all cases. always satisfies V pub (X) 0 in all cases. always satisfies V pub (X) = 0 in all cases. can possibly satisfy V pub (X) < 0 in certain cases. Player s utility may decrease if the outcome of X becomes a public information, since other players can now exploit this information to better play against player, especially in an adversarial setting. (v) [1 pt] What is V pub lice (E), the value of public information of E to lice in the specific game tree above? nswer: 10 Subtracting the answer of (a) from the answer of (c, iii), = 10. (vi) [2 pts] Let a = V pub lice (E) be the value of public information of E to lice. Suppose David will publicly announce the outcome of E if anyone (either lice or Bob) pays him b dollars (b > 0), and will make no announcement otherwise. Which of the following statements are True? The value of public information of E to Bob is V pub Bob (E) = a. If b < a, then lice should pay David b dollars. If b > a, then Bob should pay David b dollars. If b < a, then Bob should pay David b dollars. If b > a, then lice should pay David b dollars. There exists some value b > 0 such that both lice and Bob should pay David b dollars. There exists some value b > 0 such that neither lice nor Bob should pay David b dollars. Since lice and Bob s utilities always sum up to zero, if lice s utility increases by a after the outcome of E becomes a public information, then Bob s utility will certainly decrease by a, so V pub Bob (E) = a. lice should pay when b < V pub pub lice (E) = a, and Bob should pay when b < VBob (E) = a, which cannot happen simultaneously since b > 0. When b is large enough (b > a ), then neither lice nor Bob should pay for the announcement. 13

14 Q6. [12 pts] Bayes Net Modeling (a) Modeling Joint Distributions For each of the Bayes Net (BN) models of the true data distribution, indicate if the new Bayes Net model is guaranteed to be able to represent the true joint distribution. If it is not able to, draw the minimal number of edges such that the resulting Bayes Net can capture the joint distribution, or indicate if it is not possible. (i) [2 pts] BN Model of True Data Distribution New Bayes Net Model Can new BN represent joint distribution of True Data Distribution? If no, draw arrows needed B C B C Yes No B C D D The two Bayes Net models make the same independence assumptions. (ii) [2 pts] BN Model of True Data Distribution B D New Bayes Net Model B Can new BN represent joint distribution of True Data Distribution? Not Possible If no, draw arrows needed B C C Yes No C D E D E Not Possible The new Bayes Net model encodes the independence assumptions B C and D E, which are not present in the BN model of the true data distribution. The edge B removes the assumption B C, and the edge D E removes the assumption D E. Using a different direction for either/both edges is also correct. (iii) [2 pts] BN Model of True Data Distribution B New Bayes Net Model B Can new BN represent joint distribution of True Data Distribution? D E If no, draw arrows needed B C D C D Yes No C D E E Not Possible The new Bayes Net model encodes the independence assumptions B C, E, C D B, E, and D B, E, which are not present in the BN model of the true data distribution. The edge B removes B C. The edge E removes E. The edge C D removes C D B, E and D B, E. Using a different edge direction E is also correct. E 14

15 (b) Bayes Nets and Classification Recall from class that we can use Bayes Nets for classification by using the conditional distribution of P (Y X 1, X 2,..., X n ), where Y is the class and each of the X i are the observed features. SID: ssume all we know about the true data distribution is that it can be represented with the True Distribution Model Bayes Net structure. Indicate if the new Bayes Net models are guaranteed to be able to represent the true conditional distribution, P (Y X 1, X 2,..., X n ). Mark Yes if it can be represented and No otherwise. For all subparts of this problem, the answer to the rightmost question (with edges X 1 Y, X 2 Y,... ) is Yes. These models contain a factor P (Y X 1, X 2,..., X n ) that can represent any conditional distribution. True Distribution Model Y Y Y (i) [2 pts] X 1 X 2 X1 X2 X3 X1 X2 X3 X 3 Yes No Yes No For the left Yes/No question: based on the true distribution model, we can conclude that P (Y X 1, X 2, X 3 ) = P (Y X 1, X 2 ). In other words, X 3 is not required to model the conditional distribution. If we set P (X 3 = x 3 ) = constant for all values x 3 in its domain, the new Bayes Net model will also have P (Y X 1, X 2, X 3 ) = P (Y X 1, X 2 ). With X 3 out of the picture, the remaining difference between the two models is the direction of a single edge between Y and X 1, which does not affect the ability to model the true conditional distribution. True Distribution Model Y Y Y (ii) [2 pts] B C X1 X2 X3 X1 X2 X3 X 1 X 2 X 3 Yes No Yes No For the left Yes/No question: starting with the true distribution model, we run variable elimination to eliminate, B, and C. The result would be exactly the model shown in the middle column, so we can conclude that the answer is Yes : the new model can represent the true conditional distribution. True Distribution Model Y Y Y (iii) [2 pts] B C X 1 X 2 X 3 X 4 X 5 X 1 X 2 X 3 X 4 X 5 X 1 X 2 X 3 X 4 X 5 Yes No Yes No 15

16 For the left Yes/No question: the true data distribution can have the property that Y = X 1 XOR X 2, constructed as follows. Let Y, X 1, and X 2 be binary random variables. Let take on values from the set {(0, 0), (0, 1), (1, 0), (1, 1)}. Let P ( = (0, 0) Y = 0) = P ( = (1, 1) Y = 0) = 0.5 and P ( = (0, 1) Y = 1) = P ( = (1, 0) Y = 1) = 0.5. Let X 1 equal the first element of s value with probability 1, and let X 2 equal the second element of s value with probability 1. However, the Naive Bayes model has X 1 X 2 Y, which can t represent the XOR function. s a result, the Naive Bayes model can t represent the true distribution model. 16

17 Q7. [14 pts] Help the Farmer! SID: Chris is a farmer. He has a hen in his barn, and it will lay at most one egg per day. Chris collects data and discovers conditions that influence his hen to lay eggs on a certain day, which he describes below. O P(O) +o 0.1 o 0.9 W P(W) +w 0.7 w 0.3 H W P(H W) +h +w 0.9 +h w 0.5 h +w 0.1 h w 0.5 S W O P(S W,O) +s +w +o 0.6 +s +w o 0.1 +s w +o 0.8 +s w o 0.1 s +w +o 0.4 s +w o 0.9 s w +o 0.2 s w o 0.9 E H S P(E H,S) +e +h +s 0.4 +e +h s 0.8 +e h +s 0.2 +e h s 0.6 e +h +s 0.6 e +h s 0.2 e h +s 0.8 e h s 0.4 E S O H W For a single hen, variables O, W, S, H, and E denote the event of an outbreak (O), sunny weather (W ), sickness (S), happiness (H), and egg being laid (E). If an event does occur, we denote it with a +, otherwise, e.g., +o denotes an outbreak having occurred and o denotes no outbreak occurred. (a) Suppose Chris wants to estimate the probability that the hen lays an egg given it s good weather and the hen is not sick, e.g., P (+e + w, s). Suppose we receive the samples: ( o, +w, s, h, +e), ( o, +w, s, +h, e), (+o, +w, s, h, e) (i) [2 pts] Similar to the likelihood weighing method, Chris weighs each of his samples after fixing evidence. However, he weighs each sample only with P ( s +w, O), i.e. he omits weighing by P (+w). Chris method results in the correct answer for the query P (+e + w, s). True False Because the probability P (+w) is constant, by not including it, the query after normalizing would be correct. (ii) [2 pts] Using likelihood weighting with the samples listed above, what is the probability the hen lays an egg given it s good weather and the hen is not sick, or P (+e + w, s)? Round your answer to the second decimal place or express it as a fraction simplified to the lowest terms The weights are for the first two and This gives us the estimation probability of (b) Chris uses Gibbs sampling to sample tuples of (O, W, S, H, E) , or a (i) [2 pts] s a step in our Gibbs sampling, suppose we currently have the assignment of ( o, w, +s, +h, +e). Then suppose we resample the sickness variable, i.e., S. What is the probability that the next assignment is the same, i.e., ( o, w, +s, +h, +e)? Round your answer to the second decimal point, or express it as a fraction simplified to the lowest terms..05 This is asking for the probability P (+s o, w, +e, +h). Mathematically, this is = This gives us P (+s, o, w,+e,+h) P (+s, o, w,+e,+h)+p ( s, o, w,+e,+h) =

18 (ii) [2 pts] What will be the most observed tuple of (O, W, S, H, E) if we keep running Gibbs sampling for a long time? Select one value from each column below to denote the assignment. +o o +w w +s s +h h +e e The most observed sample should be good weather, no outbreak, no sickness, happiness, and laying an egg. This yields probability of = (c) [3 pts] Suppose we adopt a sampling procedure where at each evidence node with probability 0.5 we fix to the evidence, otherwise we sample the outcome and reject if it doesn t match. Upon seeing an evidence node, write an expression for the value we will multiply into the weight of the sample to make this procedure consistent. The weight is initialized at the start as weight = 1. Your answer may use the variables s and p, where s is 1 if the coin flip told us to sample the evidence node, and p is the probability of the evidence given its parents. weight *= (1 s)p + s When the coin flip tells us to fix the evidence, we treat it like likelihood weighting and multiply by p. When the coin flip tells us to sample, we treat it like rejection sampling, so we don t need to change the weights. nice way of writing this is (1 s)p + s. Technically, we can also multiply by any constant (e.g..5 or p) since if every sample weight is scaled this cancels when we normalize weights to estimate probabilities. We also accepted answers written as piece-wise functions. Now, suppose there are 1000 hens, each independently modeled by the Bayes Net model below. Denote the random variables for sickness, happiness, and laying an egg as S i, H i, E i for hen i. The conditional probability tables are the same as above for each hen. E i S i O H i W (d) [3 pts] One day, Chris observed that all the hens lay eggs and the weather is bad. What s the probability of an outbreak happening? Round your answer to the second decimal point. Hint: P (O = +o, W = w, E i = +e i ) = and P (O = o, W = w, E i = +e i ) = for all i. P (+e i + o, w) = /P (+o, w) = /0.1/0.3 = 0.38 P (+e i o, w) = /P ( o, w) = /0.9/0.3 = P (O + e 1,..., +e 1000, w) P (O, +e 1,..., +e 1000, w) = 1000 i=1 P (+e i O, w)p (O)P ( w) By normalizing this, P (+o, +e 1,..., +e 1 000, w) = (0.38) (0.38) (0.66) = 1 1+(66/38)

19 Q8. [15 pts] Bayes Nets and RL SID: In this question, you will see that variable elimination can solve reinforcement learning problems. following Bayes net, where S t S, t, and R t {0, 1}: Consider the t-1 t t+1 S t S t+1 S t+2 R t R t+1 R t+2 (a) [2 pts] From the list below, select all the (conditional) independencies that are guaranteed to be true in the Bayes net above: S t+1 S t 1 S t, t R t+1 R t 1 S t, t R t+1 R t t+1 R t S t t+1 R t S t, t None of the above Let +r t:t denote the event R t = R t+1 = = R T = 1, and assume that P (a t ) = 1/. Define the following functions: β t (s t, a t ) = P (+r t:t s t, a t ), β t (s t ) = P (+r t:t s t ) = 1 β t (s t, a t ) a t Perform variable elimination to compute P ( t S t, +r t:t ). (b) [2 pts] Which of the following recursions does β t (s t, a t ) satisfy? β t (s t, a t ) = P (+r t s t ) s t+1 β t+1 (s t+1 ) β t (s t, a t ) = P (+r t s t ) s t+1 β t+1 (s t+1, a t ) β t (s t, a t ) = P (+r t+1 s t ) s t+1 β t+1 (s t+1, a t ) β t (s t, a t ) = P (+r t s t ) s t+1 P (s t+1 s t, a t )β t+1 (s t+1 ) β t (s t, a t ) = a t+1 P (+r t+1 s t+1 ) 1 None of the above s t+1 P (s t+1 s t, a t )β t+1 (s t+1 ) (c) [2 pts] Write P (a t s t, +r 1:T ) in terms of β t (s t, a t ), β t (a t ), and relevant probabilities from the Bayes net. P (a t s t, +r 1:T ) = βt(st,at) β P (+r t(s t) t s t ) P (a t s t, +r 1:T ) = βt(st,at) a t βt(st,a t ) P (a t s t, +r 1:T ) = βt(st,at) β t(s t) P (a t s t, +r 1:T ) = P (a t s t, +r 1:T ) = None of the above βt(st,at) P (+r t s t) P (+r t s t) β t(s t)β t(s t,a t) 19

20 So far, we have only discussed variable elimination in a certain Bayes net. Now, we will associate the Bayes net with an MDP with two parameters p, q (0, 1). The states are S = {x, y, z, w}, and the actions are = {a, b}. The transitions are described in this diagram: a (w/prob. p) a (w/prob. 1-p) x y z a b a, b b w a, b ll transitions are deterministic except when taking action a starting in state y this transition is determined by P (S t+1 = y S t = y, t = a) = p. The rewards are stochastic and depend only on state, taking on values in {0, 1} with probabilities P (+r t S t = x) = 1, P (+r t S t = y) = 1, P (+r t S t = z) = 0, P (+r t S t = w) = q Throughout the following questions, assume that p, q (0, 1). (d) [3 pts] Consider running the uniform policy π uniform ( t = a) = 1 2 is P ( 1 = a S 1 = x, +r 1:T +2 )? for T + 2 timesteps starting in state x. What p T +1 /(p T +1 + (2q) T +1 ) (2q) T +1 /(p T +1 +(2q) T +1 ) p T /(p T + 2 T q T +1 ) 2 T q T +1 /(p T + 2 T q T +1 ) p T /(p T + (2q) T ) (2q) T /(p T + (2q) T ) p T /(p T + q T ) q T /(p T + q T ) p T +1 /(p T +1 + q T +1 ) q T +1 /(p T +1 + q T +1 ) None of the above Since P ( 1 = a, +r 1:T +2 S 1 = x) = 1 2 x, +r 1:T +2 ) = p T /(p T + 2 T q T +1 ). ( 1 2 p) T and P (1 = b, +r 1:T +2 S 1 = x) = 1 2 qt +1, we get P ( 1 = a S 1 = (e) [2 pts] Suppose p > 2q (T +1)/T. When running π uniform, what is arg max z P ( 1 = z S 1 = x, +r 1:T +2 )? arg max z P ( 1 = z S 1 = x, +r 1:T +2 ) = a arg max z P ( 1 = z S 1 = x, +r 1:T +2 ) = b Cannot be determined If p > 2q (T +1)/T, then P ( 1 = a S 1 = x, +r 1:T +2 ) > 1/2, so a is the answer. (f) [2 pts] Suppose q > 2 T/(T +1). When running π uniform, what is arg max z P ( 1 = z S 1 = x, +r 1:T +2 )? arg max z P ( 1 = z S 1 = x, +r 1:T +2 ) = a arg max z P ( 1 = z S 1 = x, +r 1:T +2 ) = b Cannot be determined If q > 2 T/(T +1), then P ( 1 = a S 1 = x, +r 1:T +2 ) < p T /(p T + 1) < 1/2 because p < 1. So, the answer is b. (g) [2 pts] Consider running the optimal policy π for T + 1 timesteps starting in state x. When is π always guaranteed to choose b as its first action? 20

21 T > 1 (1 p)q T > 1 (1 q)p T > 1 (1 p)(1 q) T > 1 pq T < 1 (1 p)q T < 1 (1 q)p SID: T < 1 (1 p)(1 q) T < 1 pq None of the above In state y, the optimal policy will always choose action a, and its value is at most 1/(1 p). Meanwhile, in state w, the value is always qt. So, if qt > 1/(1 p), the optimal policy must go to state w, meaning that it must take action b at x. 21

22 Q9. [8 pts] Decision Trees You are given a dataset for training a decision tree. The goal is to predict the label (+ or -) given the features, B, and C. B C label First, consider building a decision tree by greedily splitting according to information gain. (a) [2 pts] Which features could be at the root of the resulting tree? Select all possible answers. B C and C yield maximal information gain at the root. (b) [2 pts] How many edges are there in the longest path of the resulting tree? Select all possible answers None of the above Regardless of the choice of the feature at the root, the resulting tree needs to consider all 3 features in a path, so there are 3 edges in that path. Now, consider building a decision tree with the smallest possible height. (c) [2 pts] Which features could be at the root of the resulting tree? Select all possible answers. B C The optimal decision tree first splits on B. For the B=0 branch, the next split is on ; for the B=1 branch, the next split is on C. (d) [2 pts] How many edges are there in the longest path of the resulting tree? Select all possible answers None of the above s can be seen from the answer to part (c), the optimal tree has two edges per path from the root to any leaf. 22

23 THIS PGE IS INTENTIONLLY LEFT BLNK

24 SCRTCH PPER INTENTIONLLY BLNK PLESE DETCH ME 24

25 SCRTCH PPER INTENTIONLLY BLNK PLESE DETCH ME 25

26 SCRTCH PPER INTENTIONLLY BLNK PLESE DETCH ME 26