Why on earth did you do that?!

Transcription

1 Why on earth did you do that?! A formal model of obfuscation-based choice Caspar Chorus Delft University of Technology Challenge the future

2 Background: models of decision-making Based on the notion that choices are based on motivations (preferences, desires, decision-rules) - Theory of Reasoned Action / Planned Behavior (Psych.) - Optimal Decision-Making (OR, Dec. Sciences) - Rational Choice Theory (Economics, Discrete Choice) - Belief-Desire-Intention framework (Multi-Agent Systems, AI) - In other words, motivations echo through in choices e.g. Revealed preference axiom Cornerstone of micro-econom(etr)ics 2

3 Background: models of decision-making (II) Based on the notion that choices are based on motivations (preferences, desires, decision-rules) - Theory of Reasoned Action / Planned Behavior (Psych.) - Optimal Decision-Making (OR, Dec. Sciences) - Rational Choice Theory (Economics, Discrete Choice) - Belief-Desire-Intention framework (Multi-Agent Systems, AI) - In other words, motivations echo through in choices This talk: decision-maker wishes to suppress the echo In other words, wishes to hide his motivations from an onlooker 3

4 Example two futures for AI Dystopia AI outsmarts us Autonomous AI AI fights us AI out of control Opaque AI Unaligned AI Utopia AI outsmarts us Autonomous AI AI supports us AI in control Interpretable AI Aligned AI 4

5 Example (ctd): AI versus supervisor Dystopia AI fights us AI outsmarts us AI out of control Opaque AI Obfuscates rules Unaligned AI Utopia AI supports us AI outsmarts us AI in control Interpretable AI Tries to infer rules Aligned AI Actions are observeable, rules are latent 5

6 AI vs supervisor (illustration): algorithmic discrimination Artificial, autonomous agent Task: select CVs for job-interview (or: determine recidivism-probability) (or: set optimal health insurance premium) Rules: may include racist, sexist biases Human supervisor Aims to find out which rules agent uses Will punish use of biased rules Agent knows this and will try to obfuscate its rules 6

7 Some other ways to frame obfuscation Moral wiggle room & Human (Moral) decision making Privacy-protection - Human: facing a choice / moral dilemma - Onlooker: punishes with contempt (based on employed principle!) - Human: creates ambiguity about employed principle(s) Legal decision making - Suspect: making choice that (may) violate(s) law - Prosecutor: punishes with legal action (based on motivation!) - Suspect: creates reasonable doubt about motivations Military decision making - Autonomous weapon: infiltrates, fights - Adversary (human, AI): wins, if it finds out strategic objectives - Autonomous weapon: creates strategic ambiguity 7

8 Relevance: variety of research fields Supervisor Obfuscator Human Artificial Human Artificial Artificial Intelligence, (moral) Psychology, Law Expert Systems, Human-Computer Interaction Artificial Intelligence, Expert Systems, Multi-agent sytems Human-Computer Interaction 8

9 Relevance: also in Transport Supervisor Obfuscator Human Artificial Human Moral decisions by travelers and citizens (e.g. taboo trade-offs) Keeping your Automated Vehicle on the moral high Artificial Travel recommender system (e.g. personal travel assistant) AV-AV interaction on multi-lane highways, ground ( moral machine ) crossroads 9

10 Intermezzo: obfuscation vs deception Why not assume that agents try to mislead? (rather than assuming that they try to create a smoke screen) Because agent may not be bad, merely afraid of punishment And creating reasonable doubt is enough to avoid it Because deceit is costly when found out (compared to smoke) And obfuscating is much harder to proove than deceit Because agent does not know which rule is considered right one And as such has no target rule for deceit 1

11 Base Model 11

12 The model basic notation Set contains actions (alternatives, options) Set contains rules (motivations) by matrix contains scores describing how an action performs on a given rule. +,, : obliged (+), permitted (), prohibited ( ) Strong rule: +, Weak rule:, 12

13 Behavior of a Rule-follower Agent follows rule. Probability that she chooses given that she follows : Strong rule = 1 if is obliged under = if is prohibited under Weak rule = 1 if is permitted (but not obliged) under, where is actions that are permitted (but not obliged) under. = if is prohibited under 13

14 Behavior of an Obfuscator : Assumptions regarding beliefs This study: Supervisor and agent share knowledge regarding - (Choice) set of actions - Set of rules - Scores of actions on rules as in Future work: Diverging ideas about,, This study: Supervisor has no idea beforehand about which rule is used by the agent ( uninformative priors ) Future work: Repeated choices, iterative learning 14

15 Behavior of an Obfuscator Agent only cares about making sure that the supervisor remains unable to infer what rule lead to agent s choice. That is: agent has no motivation, other than obfuscation Easily extended to hybrid version, where rule-following and Obfuscation (measured by ) both play a role (more realistic) = But for ease of illustration, only two extreme cases are discussed in the remainder: Rule-followers and Obfuscators 15

16 Intermezzo: learning vs finding evidence Notion of a supervisor learning an agent s rules has no meaning when the agent only cares about obfuscation (and as such strictly speaking has no rules) Notion of finding evidence that a certain action follows from a certain rule is still meaningful in that context. Supervisor wishes to link an action to a rule beyond reasonable doubt Agent wishes to create reasonable doubt (measured in Entropy) 16

17 Behavior of an Obfuscator (II) Agent knows that supervisor will update his beliefs regarding the rule presumably used by her decision making: This posterior can be modeled as follows (Bayes, 1763): = prior: 1 Where: = 1 if is obliged under strong rule = if is prohibited under = 1 if is permitted under weak rule 17

18 Behavior of an Obfuscator (III) The agent knows that the supervisor s updated beliefs after having witnessed action result in uncertainty (denoted ). How to model, quantify this uncertainty? Using the notion of information Entropy (Shannon, 1948): = $ log Agent s behavior characterized by: argmax.. 18

19 Obfuscator worked out example -. - / / e.g. -. obliged by 1., permitted by 1 /, prohibited by 1, 1 2. e.g. 1. obliges -., prohibits - /, -. 19

20 Obfuscator worked out example (II) Rule-posteriors conditional on choosing action 1, 2, 3 P a r = 1; P a r 4 =.5; P a r 6 = ; P a r 7 = = = / = = = = P a 4 r = ; P a 4 r 4 =.5; P a 4 r 6 =.5; P a 4 r 7 = / = ; 9 1 / - / = / = 1 4 ; / = = 9 1 / - = = ; = 1 2

21 Obfuscator worked out example (III) Entropy resulting from choosing action 1: = 2 3 log log 1 3 =.276 Entropy resulting from choosing action 2: 4 = 1 4 log log log 1 2 =.452 Entropy resulting from choosing action 3: 6 = 1 log 1 = 21

22 Intermezzo behavior of Hybrid -. - / / Obfuscator will prefer 4 over, 6. Hybrid following 4 will prefer 4 over Hybrid following 1 will prefer - / over - Hybrid following may even prefer 4 over, depending on 22

23 Intermezzo: Maximizing entropy Maximizing rule-compliances / / =.24 4 =.29 6 =.29 Even though all actions are permitted by two rules, making all actions equivalent to a rule-compliance-maximizer, - / and - are preferred over -. by an Obfuscator. 23

24 Model extensions 24

25 Considered extensions From passive to active supervisor choice set design Versatile agent multiple rules 25

26 Behavior of active supervisor Until now: (implicit) assumption of passive supervisor : only exists in mind of the agent Now: supervisor is able to determine the set (A) of actions from which the agent chooses. - Select a set A of a given size - Given set of a given size, select alternative to add - Given set of a given size, select alternative to remove Supervisor s objective: minimize entropy contained in set Caveat: Entropy of action is contingent on set 26

27 Intermezzo: Entropy of action is contingent on set example -. - / / B -/ -.,- / = C. 2E B -/ - /,- = C. 2D 27

28 Behavior of active supervisor (II) Objective: minimize entropy through choice set design Depends on whether or not supervisor is naive or cynical Naive supervisor: Assumes agent follows some rule, does not care about obfuscating Cynical supervisor : Assumes agent only cares about obfuscating, has no rules Hybrid version covered in the paper Disclaimer: strictly speaking, naive and cynical not right terms 28

29 Behavior of naive supervisor Cf. efficient experimental design of DCEs Objective: minimize entropy through choice set design Is a function of the alternative chosen by the agent from the set; which depends on rule followed by the agent; which is unknown to the supervisor a priori. This is captured as follows: supervisor composes A such that: $ G $ G < $ G I $ G I K L Expected entropy associated with A Expected entropy associated with A 29

30 Behavior of naive supervisor Objective: minimize entropy through choice set design Is a function of the alternative chosen by the agent from the set; which depends on rule followed by the agent; which is unknown to the supervisor a priori. This is captured as follows: supervisor composes A such that: $ G $ G < $ G I $ G I K L Entropy associated with agent choosing action from A 3

31 Behavior of naive supervisor Objective: minimize entropy through choice set design Is a function of the alternative chosen by the agent from the set; which depends on rule followed by the agent; which is unknown to the supervisor a priori. This is captured as follows: supervisor composes A such that: $ G $ G < $ G I $ G I K L Probability of agent choosing action from A 31

32 Behavior of naive supervisor Objective: minimize entropy through choice set design Is a function of the alternative chosen by the agent from the set; which depends on rule followed by the agent; which is unknown to the supervisor a priori. This is captured as follows: supervisor composes A such that: $ G $ G < $ G I $ G I K L Probability of agent following rule multiplied by probability of choosing from K conditional Regret in Traveler Decision on following Making 32

33 Naive supervisor simple example -. - / / 1 N, 4 =.4 N 4, 6 =.38 P B -., - = C. /C $ G $ G = =.4 33

34 Behavior of cynical supervisor Objective: minimize entropy through choice set design Is a function of the alternative chosen by the agent from the set; which is based on obfuscation by the agent. This is captured as follows: supervisor composes A such that: max.. G < max.. G I K L Entropy of the maximum entropy action in A Entropy of the maximum entropy action in A 34

35 Cynical supervisor simple example -. - / / 1, 4 =.45; 4, 6 =.46; B -., - = C. C, 4, 4, 6 change order, compared to naive supervisor 35

36 Agent- supervisor interaction: Stackelberg game Supervisor = Leader: selects choice set, anticipating behavior of follower (agent) Agent = Follower: chooses action, given choice set provided by leader (supervisor) Resulting B Rule-follower agent Obfuscator agent Naive supervisor B QR B QRS Cynical supervisor B AR B ARS All resulting (expected) entropies can be computed using equations presented on earlier slides. 36

37 Considered extensions From passive to active supervisor choice set design Versatile agent multiple rules 37

38 Behavior of a versatile agent: Case of Rule-follower Agent cares about multiple rules, to different degrees. Denote T the utility (to the agent) of complying with the score on rule of action. Strong rule, normalized: T U R, Weak rule, normalized: T U R, Weight U R or shorthand U represents the relative importance of rule to the agent. Then, under simple MNL-assumptions: -dimensional vector of U s V = exp T exp T Regret in Traveler Decision Making 38

39 Behavior of a versatile agent: Case of Obfuscator Agent cares about prohibiting the supervisor to learn V. Maximize = Y Z V log Z V [V \ Where: Z V = ] ^_ V ` V Y ] ^_ V ` V av b Here: V = cde h gij f _g k _ij h gij cde f _g Modeled by means of continuous versions of Bayes theorem, Shannon entropy Note: for versatile Hybrid : = 1 T + 39

40 (much) Work to be done 1. Empirical: formulate choice models, estimate, validate. Study identifiability of parameters (e.g. ), perform DCEs 2. Theoretical: relax assumptions (e.g. regarding beliefs), study model properties, also as Multi-agent-systems 3. Simulations: norm formation in societies (infering norms from actions) using Agent-based models 4. Implications: different agents (humans, state actors, AIs), different contexts (military, health, transport, ) 4