Intro Utility theory Utility & Loss Functions Value of Information Summary. Decision Theory. (CIV Machine Learning for Civil Engineers)

Transcription

1 Decision Theory (CIV654 - Machine Learning for Civil Engineers) Professor: James-A. Goulet Département des génies civil, géologique et des mines Chapter 14 Goulet (219). Probabilistic Machine Learning for Civil Engineers. Chapter 16 Russell, S. and Norvig, P. (1995). Artificial Intelligence, A modern approach. Prentice-Hall. Decision Theory V1.3 Machine Learning for Civil Engineers 1 / 26

2 Intro Utility theory Utility & Loss Functions Value of Information Summary Context Making rational decisions - Fire alarm t 1 min t Pr =.99 t + 1 min Pr =.7.99 Pr =.1.1 Pr =.1.99 Pr = Pr =.9.95 Decision Theory V1.3 Machine Learning for Civil Engineers.95 Polytechnique Montr eal 2 / 26

3 Context Making rational decisions - Soil contamination We have 1 m 3 of soil form an industrial site. What should we do? Pr =.9 $ 1K$ Pr =.1 1$ 1$ E[$ ] = $.9 + 1K$.1 = 1K$ Optimal action E[$ ] = 1$.9 + 1$.1 = 1$ Decision Theory V1.3 Machine Learning for Civil Engineers 3 / 26

4 Nomenclature Nomenclature A = {a 1, a 2,, a A } x Z or R Pr(x) U(a i, x) L(a i, x) U(a i, x) A set of possible actions An outcome in a set of possible states Probability of a state x Utility given a state x and an action a i Loss given a state x and an action a i Decision Theory V1.3 Machine Learning for Civil Engineers 4 / 26

5 Nomenclature Soil contamination example a i {, } {, 1} x {, } {, 1} Pr(x) = [.9,.1] [,, U(a i, x) = U,, [,, L(a i, x) = L,, ] [ ] $ 1K $ 1$ 1$ [ $ ] 1K$ 1$ 1$ ] Decision Theory V1.3 Machine Learning for Civil Engineers 5 / 26

6 Rational decisions Rational decisions Expected utility maximization The perceived benefit of an outcome x i given an action a i is measured by the expected utility or expected loss E[U(a)] = X i=1 U(a, x i) Pr(x i ) E[L(a)] = X i=1 L(a, x i) Pr(x i ) The optimal action a is the one that maximizes the expected utility or minimizes the expected loss a = arg max a E[U(a)] = arg min a E[L(a)] Decision Theory V1.3 Machine Learning for Civil Engineers 6 / 26

7 Intro Utility theory Utility & Loss Functions Value of Information Summary Module #9 Outline Topics organization Supervised learning Decision Making Decision Theory V1.3 Machine Learning for Civil Engineers 1 Revision probability & linear algebra µ σ ( Bayesian Estimation 4 FX(x) 2 x 2 µ σ µ µ+σ x 2 4 p(b A)p(A) p(b) p(a B) = s = MCMC sampling & Newton Regression θ µ µ+σ f X(x).4 2 Probability distributions Introduction θ1 5 θ θ1 Patogens [ppm] Machine Learning Basics ( [ ] cl+ d= fx D(x d) Probability theory Classification d=1 d=2 d=3 d=4 d=5 1 P(D S) Intro Utility theory Utility & Loss Functions Value of Information.2.4 S={4,5} 1 x.6 S={2,3}.8 1 S={,1} State-space model for time-series E[U] = 3.7 E[U] = 7. 8 Decision Theory 9 AI & Sequential decision problems Polytechnique Montr eal 7 / 26

8 Section Outline Utility theory 2.1 Lotteries 2.2 Axioms of utility theory Decision Theory V1.3 Machine Learning for Civil Engineers 8 / 26

9 Lotteries Nomenclature for ordering preferences A lottery: L i = [{p 1, x 1 }, {p 2, x 2 },, {p X, x X }] L = [{1.,, }{.,, }] L = [{.9,, }{.1,, }] A decision maker L i L j L i L j L i L j prefers L i over L j is indifferent between L i and L j prefers L i over L j or is indifferent Decision Theory V1.3 Machine Learning for Civil Engineers 9 / 26

10 Axioms of utility theory Axioms of utility theory What is defining a rational behaviour? Orderability: Exactly one of (L i L j ), (L j L i ), (L i L j ) holds Transitivity: if (L i L j ) (L j L k ), then (L i L k ) Continuity: if (L i L j L k ), then p : [{p, L i }; {1 p, L k }] L j Substitutability: if (L i L j ), then [{p, L i }; {1 p, L k }] [{p, L j }; {1 p, L k }] Monotonicity: if L i L j, then (p q [{p, L i }; {1 p, L j }] [{q, L i }; {1 q, L j }]) Decomposability:...no fun in gambling Decision Theory V1.3 Machine Learning for Civil Engineers 1 / 26

11 Section Outline Utility & Loss Functions 3.1 Utility 3.2 Non-linear utility functions 3.3 Utility and Loss functions U(v) & L(v) 3.4 E[L(v(a i, X ))] and risk aversion Decision Theory V1.3 Machine Learning for Civil Engineers 11 / 26

12 Utility Axioms utility Existence of a utility function: U(L i ) U(L j ) L i L j U(L i ) = U(L j ) L i L j Expected utility of a lottery: E[U([{p 1, x 1 }, {p 2, x 2 },, {p X, x X }])] = X p i U(x i ) i=1 Decision Theory V1.3 Machine Learning for Civil Engineers 12 / 26

13 Non-linear utility functions Do you want to take the lottery? L = [{ 1 2, + }, { 1 2, - }] L = [{1, +$}] Which lottery do you choose? Why? E[$(L )] = $ $ = +5$ E[$(L )] = $ Are you being irrational? For individuals U($) and L($) are non-linear... Decision Theory V1.3 Machine Learning for Civil Engineers 13 / 26

14 1Utility, (v) Utility and Loss functions U(v) & L(v) Risk aversion and utility functions U(v) U(v): An utility function weight monetary value (v) as a function of risk aversion/propension (± 1$ not the same effet if you have 1$ or 1M$) Risk seeking Risk neutral Risk averse v U(v) = v k People/organizations are risk averse k > 1 Risk seeking k = 1 Neutral < k < 1 Risk averse Decision Theory V1.3 Machine Learning for Civil Engineers 14 / 26

15 1Loss, (v) Utility and Loss functions U(v) & L(v) Risk aversion and loss functions L(v) L(v): A loss function weight monetary value (v) as a function of risk aversion/propension Risk seeking Risk neutral Risk averse v L(v) = v k People/organizations are risk averse k > 1 Risk averse k = 1 Neutral < k < 1 Risk seeking Decision Theory V1.3 Machine Learning for Civil Engineers 15 / 26

16 1Loss, (v) Utility and Loss functions U(v) & L(v) Attitude toward risks Risk seeking Risk neutral Risk averse v k > 1 Risk averse L(v) = v k k = 1 Neutral < k < 1 Risk seeking A neutral attitude toward risks minimizes the expected costs over a multiple decisions Insurance compagnies: neutral attitude toward risks Insured people: risk averse; they pay a premium not to be in a risk neutral position (i.e. expected costs are higher over multiple decisions) Decision Theory V1.3 Machine Learning for Civil Engineers 16 / 26

17 1Loss, (v) Utility and Loss functions U(v) & L(v) Expected Loss Risk seeking Risk neutral Risk averse v Value Loss: Value x = x = a = v(, ) v(, ) a = v(, ) v(, ) Loss x = x = E[L(v(a, X ))] a = L(v(, )) L(v(, )) E[L(v(, X ))] a = L(v(, )) L(v(, )) E[L(v(, X ))] Decision Theory V1.3 Machine Learning for Civil Engineers 17 / 26

18 E[L(v(a i, X ))] and risk aversion E[L(v(a i, X ))] and risk aversion (ex. discrete) [ ] 1 E[L(v(a1, X)] = E[L(v(a2, X)] Loss, L(v) 1 p(l(v) a i, x) Action a 1 Action a 2 p(v ai, x) 1 v(a i, x) 1.5 E[v(a1, X)] = E[v(a2, X)].5 1 v(a i, x) Risk perception neutral: E[L(v(a 1, X ))] E[L(v(a 2, X ))] [CIV ML/Decision/PCgA discrete.m] Decision Theory V1.3 Machine Learning for Civil Engineers 18 / 26

19 E[L(v(a i, X ))] and risk aversion E[L(v(a i, X ))] and risk aversion (ex. discrete) [ ] 1 E[L(v(a1, X)] E[L(v(a2, X)] p(l(v) a i, x) Action a 1 Action a 2 p(v ai, x) Loss, L(v) 1 1 v(a i, x) 1.5 E[v(a1, X)] = E[v(a2, X)].5 1 v(a i, x) Risk perception neutral: E[L(v(a 1, X ))] E[L(v(a 2, X ))] [CIV ML/Decision/PCgA discrete.m] Decision Theory V1.3 Machine Learning for Civil Engineers 18 / 26

20 E[L(v(a i, X ))] and risk aversion E[L(v(a i, X ))] and risk aversion (ex. continuous) [ ] E[L(v(a1, X))] = E[L(v(a2, X))] 1 Loss, L(v) f(l(v) a i, x) 1 v(a i, x) Action a 1 Action a 2 f(v ai, x) E[v(a1, X)] = E[v(a2, X)].5 1 v(a i, x) Risk perception neutral: E[L(v(a 1, X ))] E[L(v(a 2, X ))] [CIV ML/Decision/PCgA continuous.m] Decision Theory V1.3 Machine Learning for Civil Engineers 19 / 26

21 E[L(v(a i, X ))] and risk aversion E[L(v(a i, X ))] and risk aversion (ex. continuous) [ ] 1 E[L(v(a 2, X))] E[L(v(a 1, X))] f(l(v) a i, x) Loss, L(v) 1 v(a i, x) Action a 1 Action a 2 f(v ai, x) E[v(a1, X)] = E[v(a2, X)].5 1 v(a i, x) Risk perception neutral: E[L(v(a 1, X ))] E[L(v(a 2, X ))] [CIV ML/Decision/PCgA continuous.m] Decision Theory V1.3 Machine Learning for Civil Engineers 19 / 26

22 Section Outline Value of Information 4.1 Value of perfect information 4.2 Value of imperfect information 4.3 Exemples Decision Theory V1.3 Machine Learning for Civil Engineers 2 / 26

23 Value of perfect information Expected utility of collecting information In cases where the value of a state x is imperfectly known,one possible action is to collect information about X. X E[U(a )] = max U(a, x i ) Pr(x i ) a U(a, x = y) = max U(a, x = y) a Because y has not been observed yet, we must consider all possibilities Y = X i according to their probability E[U(ã )] = X i=1 Value of perfect information i=1 max [U(a, x i )] Pr(x i ) a VPI (y) = E[U(ã )] E[U(a )] Decision Theory V1.3 Machine Learning for Civil Engineers 21 / 26

24 Value of perfect information Soil contamination example L(a, x) x = x = a = 1$ 1$ a = $ 1K$ Current expected costs conditional on actions E[L(, X )] = $.9 + 1K$.1 = 1K$ E[L(, X )] = 1$.9 + 1$.1 = 1$ = E[L(a, X )] Expected costs conditional on perfect information E[L(ã )] = X i=1 min a (L(a, x i)) Pr(x i ) = $.9 }{{} y=x= + 1$.1 }{{} y=x= = 1$ Value of perfect information VPI (y) = E[L(a )] E[L(ã )] = 9$ Decision Theory V1.3 Machine Learning for Civil Engineers 22 / 26

25 Value of perfect information Value of information The value of information represents how much you are willing to pay for an information. What if the information is not perfect? Decision Theory V1.3 Machine Learning for Civil Engineers 23 / 26

26 Value of imperfect information Value of imperfect information Discrete state case E[U(ã )] = X X i=1 j=1 max a (U(a, y i)) Pr(y i x j ) Pr(x j ) Continuous state case E[U(ã )] = max(u(a, y)) f (y x) f (x)dydx a Decision Theory V1.3 Machine Learning for Civil Engineers 24 / 26

27 Exemples Soil contamination example - Discrete case L(a, x) x = x = a = 1$ 1$ a = $ 1K$ Pr(y = ) = 1 Pr(y = ) =.95 Current expected costs E[L(a, X )] = 1$ Expected costs conditional on imperfect information E[L(ã )] = X i=1 X j=1 min a (L(a, y i)) Pr(y i x j ) Pr(x j ) = $.9 }{{} x=y= = 59.5$ Value of information + ( 1$.95 }{{} y= + 1K$.5 ).1 }{{} y= } {{ } x= VOI (y) = E[L(a )] E[L(ã )] = 4.5$ Decision Theory V1.3 Machine Learning for Civil Engineers 25 / 26

28 Summary Rational Decision: Choose the action ai which minimize the expected loss L(a, x) or maximizes the expected cost U(a, x) a i = arg min a i E[L(a i, x)] = arg max E[U(a i, x)] a i L(v(a, x)) & U(v(a, x)): Subjective weight on value as a function of the attitude toward risks (± 1$ not the same effect if you have 1$ or 1M$) 1Loss, (v) Risk seeking Risk neutral Risk averse Value of information: Value you should be willing to pay for information VOI (y) = E[U(a )] E[U(ã )] Value of perfect information: E[U(ã )] = X i=1 max [U(a, x a i )] Pr(x i ) Value of imperfect information: v k > 1 Risk averse L(v) = v k k = 1 Neutral < k < 1 Risk seeking E[U(ã )] = X X max a (U(a, y)) Pr(y x j ) Pr(x j ) i=1 j=1 Decision Theory V1.3 Machine Learning for Civil Engineers 26 / 26