Lecture 14: Introduction to Decision Making

Transcription

1 Lecture 14: Introduction to Decision Making Preferences Utility functions Maximizing exected utility Value of information

2 Actions and consequences So far, we have focused on ways of modeling a stochastic, uncertain world But intelligent agents should be not only observers, but actors I.e. they should choose actions in a rational way Most often, actions roduce as a consequence changes in the world Pearl examle: buying a baseball ticket How should we choosing between buying and not buying a ticket???

3 lotteries So choosing between actions amounts to choosing between C P A air L is called a lottery (Luce and Raiffa, 1957) one, the set of consequences C c1 cn and a robability distribution over the consequences, P, s.t. i P ci 1. In order to comare different actions we need to know, for each We will call the consequences of an action ayoffs or rizes robabilities of the consequences or value) of each consequence and weigh these by the A rational method would be to evaluate the benefit (desirability, Preferences

4 ;B In order for an agent to act rationally, its references have to obey certain constraints A A A B - B not referred to A B - A referred to B B - indifference between A and B Agents have references over ayoffs: L 1 A B or as a tree-like diagram: A L 1 A lottery can be reresented as a list of airs, e.g. Lotteries

5 C, A Examle: Transitivity Suose an agent has the following references:b C A, and it owns C. B, If B C, then the agent would ay (say) 1 cent to get B A If A B, then the agent, who now has B would ay (say) 1 cent to get A If C A, then the agent (who now has A) would ay (say) 1 cent to get C The agent looses money forever! 1c 1c B C 1c

6 A; 1 L The robability at which equivalence occurs can be used to comare the merit of B w.r.t A and C C B rizes A and C that is equivalent to receiving B for sure: 2. Continuity: If A B C, then there exists a lottery L with A Transitivity: A Linearity: B B B B A C A A B C exist between the rizes of any lottery These secify constraints over the references that a rational agent can have: 1. Orderability: A linear and transitive reference relation must The Axioms of Utility Theory

7 L1; 0 A; C1; L1; A; L1; C1; L2; L2 C2 1 1 q 1 1 q lotteries L1 and L2 C2 1, 5. Reduction of comound lotteries ( No fun in gambling ): For any B B A B 1 1 iff roducing the best rize most often is referred 4. Monotonicity: If two lotteries have the same rizes, the one L2 For any L1 L3 1 L1 L2 1 L3 1 L3 3. Substitutability: Adding the same rize with the same robability to two equivalent lotteries does not change the reference between them:

8 C1; Proof: see Pearl book and board i iu U 1 ; n where L1 Cn Ci U L1 L2 iff U L2 maximization of exected utility Theorem: (Ramsey, 1931; von Neumann and Morgenstern, 1944): Given a reference relation over lotteries satisfying the axioms of utility theory, there exists a real-valued function U on the set of rizes C such that If an agent has rational references, his behavior is describable as Maximizing exected utility (MEU)

9 Acting under Uncertainty MEU rincile: Choose the action that maximizes exected utility. Most widely acceted as a standard for rational behavior Note that an agent can be entirely rational (consistent with MEU) without ever reresenting or maniulating utilities and robabilities E.g., a looku table for erfect tic-tac-toe Random choice models: choose the action with the highest exected utility most of the time, but kee non-zero robabilities for other actions as well Avoids being too redictable If utilities are not erfect, allows for exloration Minimizing regret

10 with robability Utilities Utilities ma states to real numbers. How do we get these numbers? The roof of the utility theorem suggests that a way to obtain these numbers is by comaring a given rize A with a standard (calibration) lottery L that has best ossible rize u worst ossible catastrohe u Usually utilities are normalized: u Adjust lottery robability until A utility of A. with robability 1 0, u L. Then is used as the

11 Utility scales Note that given a reference behavior, the utility function is NOT unique E.g. behavior is invariant w.r.t. additive linear transformations: U x k1u x k2 where k1 0 With deterministic rizes only (no lottery choices), only ordinal utility can be determined, i.e., total order on rizes

12 Money Suose you had to choose between two lotteries: L1: win $1 million for sure L2: 5 million w.. 0.1, 1 million w and nothing w Suose you had to choose between two lotteries: L3: 5 million w.. 0.1, nothing w L4: 1 million w , nothing w See also Bernoulli s aradox Peole are risk-averse

13 Decision networks Add action nodes (rectangles) and utility nodes (diamonds) to belief networks to enable rational decision making Airort Site Air Traffic Deaths Litigation Noise U Construction Cost 1. For each value of action node: comute exected value of utility node given action, evidence 2. Return MEU action

14 Examle: Value of Information Buying oil drilling rights: Two blocks A and B, exactly one has oil, worth k Prior robabilities 0.5 each, mutually exclusive Current rice of each block is k Consultant offers accurate survey of A What is a fair rice for the survey? 2

15 0 = [ = 0 5 k k 2 k 2 value of buy B given no oil in A ] value of buy A given oil in A Survey may say oil in A or no oil in A, with robability 0.5 each: without information action given the information minus exected value of best action Comute exected value of information = exected value of best Solution for the examle

16 X a VPIE X X E EU k P x αx E X α E x EU we must comute exected gain over all ossible values: X is a random variable whose value is currently unknown EU αx max E x a i U ci P ci a E X x Suose we knew X x. Then we would choose αx s.t. α E EU max a i U ci P ci E Possible action outcomes ci, otential new evidence X Current evidence E, current best action α Value of Perfect Information (VPI)

17 X Y Note: when more than one iece of evidence can be gathered, maximizing VPI for each to select one myoic strategy is not always otimal evidence-gathering becomes a sequential decision roblem V PIE V PIE X Y X V PIE Y V PIE Y V PIE Order-indeendent V PIE X Y V PIE X VPIE Y Nonadditive e.g. consider obtaining X twice we find for X, there can actually be a loss ost-hoc Note that VPI is an exectation! Deending on the actual value EV PIE X 0 Nonnegative: X Proerties of VPI X

18 Qualitative behaviors There are three ossible cases: Choice is obvious, information worth little Choice is nonobvious, information worth a lot Choice is nonobvious, information worth little P ( U E j ) P ( U E j ) P ( U E j ) U U U U 2 U 1 U 2 U 1 U 2 U 1 (a) (b) (c) Information has value to the extent that it is likely to cause a change in lan, and the new lan is significantly better than the old one

19 Summary: Decision making under uncertainty To make decisions under uncertainty, we need to know the likelihood (robability) of different ossible outcomes, and have references among the outcomes: Decision Theory = Probability Theory + Utility Theory An agent with consistent references has a utility function, which associates a real number to each ossible state Rational agents try to maximize their exected utility. Utility theory allows us to determine whether gathering more information is valuable. Next time: sequential decision making (Markov Decision Processes)