A Simple Model for Sequences of Relational State Descriptions

Transcription

1 A Simple Model for Sequences of Relational State Descriptions Ingo Thon, Niels Landwehr, Luc De Raedt Thursday 18. September 2008

2 Outline Introduce a rarely investigated setting Sequences where states are relational interpretations (simplified: graphs) Define a simple model allowing for efficient inference techniques Experimental evaluation shows 1. able to capture real worlds domain dynamics 2. scalability

3 Sequences of Relational State Descriptions

4 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games

5 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games Computer network modeling

6 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games Computer network modeling Music

7 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games Computer network modeling Music Traffic Modeling

8 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games Computer network modeling Music Traffic Modeling...

9 Sequences of Relational State Descriptions Interactions of (human/artificial) agents Business applications Computer games Computer network modeling Music Traffic Modeling... Computer games

10 Welcome to Travian Travian: A massively multiplayer real-time strategy browser game Commercial game run by TravianGames GmbH ~ players spread over different worlds ~ players in one world Classical strategy game Played on a grid map Cities, resources, buildings, technology, armies Attacks/Conquests, trading

11 City/Player/Alliance Graph Greeks Archilleus Troys Menelaos Pelion Mykena Sparta Paris Alliances: Boxes/Color-coded Players: Diamonds Cities: Circles Fat arrows Conquests Troy

12 City/Player/Alliance Graph ~10 alliances ~200 players ~600 cities Greeks Archilleus P 111 P 107 P 105 P Troys 648 P P Menelaos Alliance P P P 63 P 66 P 68 P 64 P Alliance 11P 56 P 57 P 22 P P Mykena Pelion P P 123 Alliance 10 P 36 P 45 P P P 144 P 156 P 141 P 140 P 129 P 154 P 34 P 30 P Alliance 47 P 125 P 153 P P 121 P 124 P 46 P 24 P 37 P 138 P 126 P P P P P P P P 139 P 33 Alliance P 89 P P 50 P 132 P 59 P 174 P P P P P P Alliances: Boxes/Color-coded Alliance Players: Diamonds Cities: Circles P 86 Fat arrows Conquests Sparta P 122 P 100 P 85 P 223 P 18 P 81 P 99 P 120 P 98 P 97 P 217 P P 7578 P 91 P 95 P 203 P 74 P 17 P 2220 P 77 P 199 Alliance 6 P 15 P 12 P 228 P 201 P 238 P 6 P 5 P 19 P 13 P 8 P 7 P 3 P 159 P 240 P 16 P 246 P 244 P 202 P P 21 P 212P P P 224 Alliance 7 P P P Troy P P P 51 P 61 P 233 Alliance 2 P 234 P 193 P 236 P 180 P 72 Alliance 9 P 189 P 183 P 187 P 179 P 181 P 177 P 195 P P P Paris P

13 City/Player/Alliance Graph ~10 alliances ~200 players ~600 cities Greeks Archilleus P 111 P 107 P 105 P Troys Mykena Pelion Menelaos Can we model world dynamics? P P Alliance P P How do players act, and why? P P P P 66 P 68 P 64 P Alliance 11P P 57 P 22 P P P 36 P Alliance 10 P 40 P P P 24 P P P 154 P 34 P 30 P Alliance 47 P P P 89 P 138 P 146 P P 50 P 33 P P 141 P 140 P 150 P 174 P 172 Alliance 5 P P 129 P 144 P 125 P 121 P P P P P 132 P 126 P P P 139 P P Sparta Can we predict how the world will evolve? Paris 667 Alliances: Boxes/Color-coded Alliance Players: Diamonds Cities: Circles P P P 230 Fat arrows Conquests P P 74 P 85 P 3 P P 98 P 97 P 223 Alliance P 159 P P P 99 P P 17 P P P P 120 P 217 P P 7578 P 91 P 95 P 2220 P 77 P 199 P 15 P 12 P 228 P 201 P 238 P 6 P 5 P 19 P 13 P 8 P 7 P 246 P 244 P 239 P P P 212P P P Alliance 7 P 2 P 88 Troy P P Alliance P 180 P 51 Alliance 9 P 183 P 187 P 179 P 177 P P 191 P 185 P P 233 P P P P P 236 P P P

14 World Dynamics P 10 P 6 Can players actions be explained from social context? Alliance 2 P 5 P 7 Alliance Typical patterns conquer a city which is close conquer a city of a player that has been attacked by alliance member 1024 P P tendency of alliance to conquer whole areas on the map (clusters) retaliate against attacks at an alliance level change alliance if you are under attack

15 World Dynamics P 10 Can players actions be explained from social context? Typical patterns conquer a city which is close conquer a city of a player that has been attacked by alliance member 1040 Alliance 2 P P Alliance tendency of alliance to conquer whole areas on the map (clusters) P 2 retaliate against attacks at an alliance level change alliance if you are under attack

16 World Dynamics Can players actions be explained from social context? Alliance Relations Important 713 P P 3 Statistical Relational Learning Alliance 4 Typical patterns conquer a city which is close conquer a city of a player that has been attacked by alliance member tendency of alliance to conquer whole areas on the map (clusters) retaliate against attacks at an alliance level change alliance if you are under attack P P

17 CPT-L

18 CPT-L Standard SRL: Very general, expressive models Not specifically tailored to sequences (of interpretations) No guarantee for tractability

19 CPT-L Standard SRL: Very general, expressive models Not specifically tailored to sequences (of interpretations) No guarantee for tractability CPT-L: Generative probabilistic model for sequences of interpretations Based on CP-Logic (Causal Probabilistic Logic) Efficiency by Focussing on sequences Assuming data fully observable Use of efficient data structures (BDD) for inference

20 CPT-L Idea: Model is Markov Chain P (I 0,... I T T ) = P (I 0 ) t=1...t P (I t I t 1, T )

21 CPT-L Idea: Model is Markov Chain P (I 0,... I T T ) = P (I 0 ) t=1...t P (I t I t 1, T ) P (I t I t 1, T ) Greeks Archilleus Troys Greeks Archilleus Troys Menelaos Pelion Menelaos Pelion Owner Mykena Attacker Mykena C Sparta Close Troy Paris C2 Sparta Paris Troy

22 CPT-Rules b 1,... b n h 1 : p 1... h m : p m P 5 P 3 Alliance conquer a city which is close P 2 Greeks Archilleus Troys Menelaos Mykena Pelion Sparta Paris Troy

23 CPT-Rules b 1,... b n h 1 : p 1... h m : p m cause (past) P 5 P 3 Alliance conquer a city which is close city(c, Owner), city(c2, Attacker), close(c, C2) P Greeks Archilleus Troys Owner Menelaos Mykena Pelion Attacker C Sparta Close Troy Paris C

24 CPT-Rules b 1,... b n h 1 : p 1... h m : p m cause (past) effect (future) P 5 P 3 Alliance conquer a city which is close city(c, Owner), city(c2, Attacker), close(c, C2) conquest(attacker, C) : p nil : (1 p) P Greeks Archilleus Greeks Archilleus Troys Troys Owner Menelaos Mykena Pelion Attacker Menelaos Mykena Pelion C Sparta Close Troy Paris C2 Sparta Troy Paris

25 Tasks

26 Tasks Sampling: Random sequence of states Inference: Probability I 0,..., I T P (I 0,..., I T T ) I 0,..., I T given a theory of sequence of states Parameter Estimation: Given a set of sequences of states, what is the most likely parameterization of a theory Prediction: Given a sequence of states I 0,..., I T and a formula F the probability of coming true in steps P (I t+d F I 0,..., I T, T ) F d T

27 Toy Example + Sampling Toy Example: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b

28 Toy Example + Sampling Toy Example: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Sampling successor of state {a,b}:

29 Toy Example + Sampling Toy Example: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Sampling successor of state {a,b}: 1. Select rules applicable {r1,r2,r33}

30 Toy Example + Sampling Toy Example: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Sampling successor of state {a,b}: 1. Select rules applicable {r1,r2,r33} 2. For each rule select one successor {r1:a,r2:nil,r33:a}

31 Toy Example + Sampling Toy Example: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Sampling successor of state {a,b}: 1. Select rules applicable {r1,r2,r33} 2. For each rule select one successor {r1:a,r2:nil,r33:a} 3. yield successor state {a}

32 Probability Calculation: Naive Model: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Naive Implementation of P (S t+1 S t ) e.g.. P({a, b} {a})

33 Probability Calculation: Naive Model: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Naive Implementation of Enumerate all rules with P (S t+1 S t ) S t b 1,..., b n b 1,..., b n h 1 : p 1... h m : p m e.g.. P({a, b} {a})

34 Probability Calculation: Naive Model: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Naive Implementation of Enumerate all rules with P (S t+1 S t ) S t b 1,..., b n b 1,..., b n h 1 : p 1... h m : p m Generate all selections s.t. for each rule one Probability of selection is product of p i s is select e.g.. P({a, b} {a}) {r1:a,r31:a}:0.48,{r1:a,r31:b}:0.16,{r1:a}:0.16, {r31:a}:0.12,{r31:b}:0.04,{}:0.02 h i

35 Probability Calculation: Naive Model: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b Naive Implementation of Enumerate all rules with P (S t+1 S t ) S t b 1,..., b n b 1,..., b n h 1 : p 1... h m : p m Generate all selections s.t. for each rule one Probability of selection is product of p i s is select e.g.. P({a, b} {a}) {r1:a,r31:a}:0.48,{r1:a,r31:b}:0.16,{r1:a}:0.16, {r31:a}:0.12,{r31:b}:0.04,{}:0.02 h i Probability of state is sum of probability of selections generatings t+1 {a,b}:0.16,{a}:0.76,{b}:0.04,{}:0.04

36 Probability Calculation: Efficient Algorithm Model: Query: r1 : (.8 :: a.2 :: nil) a r2 : (.9 :: b.1 :: nil) b r31 : (.6 :: a.2 :: b.2 :: nil) a, b r32 : (.2 :: a.6 :: b.2 :: nil) a, b r33 : (.4 :: a.4 :: b.2 :: nil) a, b P({a, b} {a}) applicable rules r1, r31 Min. one: (r1 : a r1 : nil) (r31 : a r31 : b r31 : nil) Only one: (r1 : a r1 : b) (r31 : a r31 : b) (r31 : a... Generate: (r1 : a r31 : a) r31 : b Calculate probability by translating into BDD

37 Efficient calculation Efficient calculation using Dynamic Programing possible selections: r1:a σ 1 = {r1:a, r2:b, r3:b} σ 2 = {r1:a, r2:b, r3:nil} σ 3 = {r1:a, r2:a, r3:b} σ 4 = {r1:nil, r2:a, r3:b} r2:b r2:a r1: nil r2:b r1: nil r2:a r2:b r3:b r3:b r3: nil r3: nil r3: nil 1 0

38 Efficient calculation Efficient calculation using Dynamic Programing possible selections: p σ 1 = {r1:a, r2:b, r3:b} r2:b + r1:a σ 2 = {r1:a, r2:b, r3:nil} p r2:a p r1: r1: 3:b nil nil σ 3 = {r1:a, pr2:a, r3:b} 3:b + p 3:nil σ 4 = {r1:nil, r2:a, r3:b} = 1 r2:b r2:a r2:b r2:a r2:b p 3:b 1 P ({a, b} true) = r3: nil r3:b r3: nil p r3:nil r3:b r3: nil 1 p r1:a (p r2:b + p r2:a p 3:b ) +p r1:nil p r2:a p r3:b 1 0

39 Learning Problem: Given observation e.g. {a},{a} Hidden Parameter: Which Rule generated which Element EM-Algorithm r1 : (p 1,a :: a p 1,nil :: nil) a r31 : (p 31,a :: a p 31,b :: b p 31,nil :: nil) a, b Expectation: Calculate Probability for each Selection, turning it into fully Observable Problem σ 1 = {r1 : a, r31 : a} p 1,a p 31,a σ 2 = {r1 : a, r31 : nil} p 1,a p 31,nil σ 3 = {r1 : nil, r31 : a} p 1,nil p 31,nil Maximization: Set Parameters according to Probability of Selections p 1,a = p 1,a p 31,a + p 1,a p 31,nil = p 1,a p 1,a p 31,a + p 1,a p 31,nil + p 1,nil p 31,nil p

40 Experiment Blocks-World: Setup Goal: show that model scales with world complexity Stochastic Blocks-World (stacks can collapse, stacking can fail,...) Complexity increases with number of blocks

41 Experiment Blocks-World: Setup Goal: show that model scales with world complexity Stochastic Blocks-World (stacks can collapse, stacking can fail,...) Complexity increases with number of blocks a c b c a b a c b

42 Experimental Evaluation I Blocks-World: convergence behaviour Log-Likelihood Log-Likelihood runtime [minutes] for 10 iterations blocks 25 blocks 50 blocks Iterations of the EM-Algorithm 10 blocks 25 blocks 50 blocks runtime Number of blocks Iterations of the EM-Algorithm

43 Experimental Evaluation I Blocks-World: runtime behavior runtime [minutes] for 10 iterations runtime Number of blocks

44 Experiment Travian: Setup Goal: show that model works on real world problems. Logged the state of a live game server: ~ players / ~3 months /snapshot of the world every 24h Projecting down to a subset of players/cities/alliances following 10 players (~50 cities) over a period of one month 30 such sequences extracted from distinct parts of the map Learn a model from December data I t+5 given I t... for January and February data (up to five steps in future) Want to predict conquests(p,c) in

45 Experimental Evaluation II Travian: predict player behavior (attack of a city) true positive rate CPT-L, k = 1 CPT-L, k = 2 CPT-L, k = 3 CPT-L, k = 4 CPT-L, k = false positive rate

46 Experimental Evaluation II Travian: predict player behavior (attack of a city) 0.9 Area under ROC curve January 2008 February Number of steps predicted (k)

47 Conclusions Sequences of relational state descriptions CPT-L: simple model for sequences of relational state descriptions addresses: 1. efficiency 2. rather than maximal expressivity Experimental evaluation of complex relational sequences 1. able to capture domain dynamics 2. scalability

48 Thanks!