Learning from paired comparisons: three is enough, two is not

Transcription

1 Learning from paired comparisons: three is enough, two is not University of Cambridge, Statistics Laboratory, UK; CNRS, Paris School of Economics, France Paris, December 2014

2 2 Pairwise comparisons A set of alternatives and, for any pair of distinct alternatives, the information that one beats the other. Such a data set is a tournament (Moon 1968). Examples: sport competition and majoritarian politics. Many sports involve by definition two players or teams. Competition among any number of players take the form of aggregation of pairwise comparisons. Majority voting: a candidate is socially preferred to another when a majority of voters prefer the former to the latter. More generally, logic traced back to specific features of efficient natural languages (Rubinstein 1996).

3 3 The pairwise comparison problem If a chess player beats all the other players, she is the best. If a candidate cannot be defeated under majority rule by any challenger, she is a Condorcet candidate. But if no alternative beats all the others, how to define the best alternatives? Choosing from pairwise comparisons has attracted attention in various fields, most often from the axiomatic, normative, point of view (David 1963; Fishburn 1977; Laslier 1997; Brandt et al. 2015). Here: an evolutionary perspective instead of an axiomatic one.

4 4 Our contribution In an urn model, we show that if one samples three alternatives (distinct or not) at each date, the process is able to discover the solution of the tournament, that is the unique probability distribution which is, in expectation, defeated by no alternative.

5 5 Tournaments Let X be a finite set. A tournament T on X is a complete and antisymmetric binary relation. For any x and y in X, one and only one of the three possibilities occurs: x = y, x T y, or y T x. When x T y we often say that x beats y. Define the sets : T + (x) = {y X : xty}, T (x) = {y X : ytx}. (1) The Top-Cycle of T is the set of alternatives which beat indirectly all other alternatives. Denoted TC(T ).

6 6 Tournament Solution A solution concept associates to any tournament T on a finite set X a non empty subset S(T, X ) of X, stable by tournament isomorphism. One usually requires that if x is a Condorcet winner, (that is if xty for all y X, y x) then S(T, X ) = {x}. The Top-Cycle is a Tournament Solution.

7 6 Tournament Solution A solution concept associates to any tournament T on a finite set X a non empty subset S(T, X ) of X, stable by tournament isomorphism. One usually requires that if x is a Condorcet winner, (that is if xty for all y X, y x) then S(T, X ) = {x}. The Top-Cycle is a Tournament Solution. Another obvious solution is the so-called Copeland solution. Define the Copeland score: s(x) = #{y X : xty} = #T + (x) Then the Copeland winners are the alternatives with largest score.

8 7 A Markov chain Let p a probability on X. Define p [t] : p [0] = p, p [t+1] (x) = p [t] (x) p ( T + (x) {x} ) + p [t] (T + (x)) p(x), (3) Interpretation: p [t] is the distribution of a random ξ(t) X s. t. ξ(0) is chosen according to p and then, given ξ(t) = x, ξ(t + 1) is the winner of the comparison between x and some y chosen in X according to p. Therefore ξ(t + 1) = x either because ξ(t) was already equal to x and y was chosen in T + (x) {x} (first term in the above formula), or because ξ(t) was in T + (x) and x was chosen according to p (second term). We call p the sampling probability. (2)

9 8 Stationary distributions This process is usually considered with p uniform on (Daniels 1969, Ushakov 1976, Levchenkov 1992, Chebotarev and Shamis 1998, Slutzky and Volij 2006, Google). Need the general version because p will be endogenous. Given p, the stationary distribution for this finite Markov chain exists and is unique. Denote it: p [ ]. It is characterized by the fact that supp(p [ ] ) supp(p) and, x supp(p), p [ ] (T + (x)) p(x) = p [ ] (x) p(t (x)). (4) Note the inclusion may be strict: p [ ] (x) = 0 when x beats no alternative in the support of p. If p has full support, for instance in the uniform case, supp(p [ ] ) = TC(T ).

10 9 The Tournament Game The tournament game is the two-player, symmetric, zero-sum game defined by the strategy set X and the payoff function g(x, y) = +1 if x T y, g(x, y) = 0 if x = y, and g(x, y) = 1 if y T x. For p, q probability distributions on X, write: g(p, q) = g(x, y)p(x)q(y). (5) x,y X By definition, g is antisymmetric: g(q, p) = g(p, q). As a model of majority voting and two-party electoral competition, tournament games are studied in Social Choice and in Political Science (Moulin 1986, Myerson 1993, Laslier 2000a,b). They have been also studied by graph theorists (Fischer and Ryan 1995a,b) and more recently by computer scientists (Rivest and Chen 2010).

11 10 Solution of the Tournament Game Remarkably, the tournament game has a unique equilibrium. Fisher and Ryan (1995a) prove this using linear algebra and Laffond, Laslier, LeBreton (1993) have a direct proof using a parity argument.) Theorem There exists a unique p such that g(p, q) 0 for all q (X ). This p, called the optimal strategy, is characterized by: x X, p (x) > 0 g(x, p (x)) = 0 p (x) = 0 g(x, p (x)) < 0. The support of the optimal strategy is called the Bipartisan Set of the tournament. In totally random tournaments, the Top Cycle contains all the alternatives and the Bipartisan Set contains only half of them (Fisher and Reeves 1995).

12 11 Properties of the Bipartisan set The Bipartisan set is a Tournament solution which refines the Top-Cycle and several other proposals: BP MC UC TC It satisfies interesting normative properties: Composition-consistency (alias Clone-independence) Strong Superset property BP(X ) Y X = BP(Y ) = BP(X ) Regularity (Copeland full tie = BP full tie)

13 12 Relation between optimal strategies and stationary probabilities The game optimal strategy p satisfies a nice fixed-point property if we take p [0] = p as the sampling probability. Only an optimal strategy can be such a fixed point. Theorem Let p be the optimal strategy for the tournament game, then (p ) [1] = (p ) [ ] = p. Conversely, let p be such that p [1] = p, then p is the optimal strategy for the tournament game restricted to the support of p. Proof: p [1] (x) = p (x)(1 + g(x, p )), so, by the previous result, either p (x) = 0 or g(x, p ) = 0. Conversely if p [1] (x) = p(x) = p(x)(1 + g(x, p)) then g(x, p) = 0 as soon as p(x) 0 and thus p is the optimal strategy on its support. QED

14 13 Random urns An urn on X is a list n of strictly positive integers. The integer n(x) is the number of balls of color x in the urn n. To each n is associated the probability distribution ñ on X defined by ñ(x) = n(x) y X n(y). When we write that the alternative x is picked in the urn n, we mean that x is picked in X according to the probability ñ. A random urn sequence is a sequence U τ of random variables such that U τ+1 is defined conditionally on U τ. Three examples:

15 14 Choice by reinforcement One-step reinforcement. An alternative x is picked in X according to ñ τ [1], and one ball of color x is added to the urn. This means that two alternatives, say a and b are picked independently in the urn n τ, and are compared according to T. The result of the comparison is x = a if a = b or if a T b and x = b if b T a. Alternative x is reinforced. Two-step reinforcement. Same thing as above, but with ñ τ [2]. Means that three alternatives, say a, b and c are picked independently in X according to n τ ; a, b and c are compared according to T in sequence and one ball of color x = max {max{a, b}, c} is added to the urn.

16 15 Choice by reinforcement Fast reinforcement. Same thing as above, with the probability distribution ñ τ [ ], the stationary distribution for T when sampling is done according to ñ τ. The first two examples can be concretely implemented easily, as described, but fast reinforcement cannot.

17 16 Main result We are now able to state the main result of this paper: with two-step reinforcement, the urn sequence converges to the optimal strategy. Theorem For any initial urn n 0, the random urn sequence obtained by two-step reinforcement is such that the realization n τ almost surely verifies: lim τ ñτ = p.

18 17 Main result: comment Optimal choice can be achieved through a learning process with two essential features. An alternative which is considered as good at some date is reinforced for the future in the sense that we (slightly, and less and less) increase the probability for this alternative to be considered: reinforcement updates the sampling, or prior probability. The test according to which an alternative is considered as a good one at time t rests on comparing three, rather than two, randomly chosen alternatives.

19 18 Idea of the proof The proof combines an intuition which can be explained on a deterministic and continuous version of the model with properties of urn processes. Deterministic and continuous approximation. Let ɛ = Pr[x is added]. On expectation, n t+1 (x) = n t (x) + ɛ thus: n t+1 (x) t + 1 n t(x) = 1 ( ) nt+1 (x) t t t + 1 ɛ and one gets slowed-down version of the replicator dynamics equation dp(x) = 1 (p(x) ɛ). dt t

20 19 Idea of the proof (2) The following function φ of p is positive and such that φ(p) = 0 p = p. φ(p) = x X p (x) log(p /p)(x). In the approximation, for two-step reinforcement: t dφ(p) dt = 1 2 g(p, p) g(w, p) 2 p (w) + v w X p(v)g(p, v)(1 + g(v, p)). And all the terms in this sum are non-negative. The sum can be 0 only if g(p, p) = 0

21 20 Idea of the proof (3) Studying the discrete stochastic model uses martingale arguments on the discrete analog of φ and features of urn models: Increments of the order 1/t : avoids lock-in because Σ1/t diverges may stabilize because increments tends to zero. (Solving the exploration-exploitation dilemma.)

22 21 Discussion What if we take only one pair of alternatives at each date? Theorem For any initial urn n 0, there is a positive probability that the random urn sequence obtained by one-step reinforcement has no limit. (We conjecture that higher-order testing leads to optimality just like two-step reinforcement.)

23 22 Cycling On the face of the simplex defined by the support of the optimal probability (the Bipartisan simplex), the surfaces φ(p) = constant are compact sets around p (level surfaces). For one-step reinforcement, almost surely: For t large, ñ t comes close to stay in a level surface of φ. From the Bipartisan simplex, it does not converges to a point. If the support of the optimal probability has dimension two, there is a limit cycle.

24 23 Evolutionary Politics The process may be a model of the long run emergence of political debate under democracy. The proposals which are rejected tend to re-appear less. Those which have been successful in the past tend to appear more and more on the agenda. This is an example where strategic reasoning (the optimal mixed strategy in a zero-sum game) game be supplemented by a myopic adaptive process.

25 References Banks Sophisticated Voting Outcomes and Agenda Control. Social Choice and Welfare Brandt, Chudnovsky, Kim, Liu, Norin, Scott, Seymour and Thomasse A counter-example to a conjecture of Schwartz. Social Choice and Welfare. Brandt, Brill and Harrenstein Tournament Solutions. Forthcoming chapter in the Handbook of Computational Social Choice. Brandt and Fischer Computing the Minimal Covering Set. Mathematical Social Sciences. Chebotarev and Shamis Characterizations of scoring methods for preference aggregation. Annals of Operation Research. Daniels Round-robin tournament scores. Biometrika.

26 Fishburn Condorcet social choice functions. SIAM Journal on Applied Mathematics. Fisher and Reeves Optimal strategies for random tournament games. Linear Algebra and its Applications. Fisher and Ryan 1995a. Tournament games and positive tournaments. Journal of Graph Theory Fisher and Ryan 1995b. Probabilities within optimal strategies for tournament games. Discrete Applied Mathematics Laffond, Laslier and Le Breton The Bipartisan set of a tournament game. Games and Economic Behavior Laslier, B and Laslier JF, Reinforcement learning from comparison: three is enough, two is not. Laslier Tournament Solutions and Majority Voting, Springer-Verlag.

27 Laslier Aggregation of preferences with a variable set of alternatives. Social Choice and Welfare Laslier Interpretation of electoral mixed strategies. Social Choice and Welfare Levchenkov Social choice theory: a new insight. Institute of Systems Analysis, Moscow. Moon Topics on Tournaments. Holt, Rinehart and Winston. Moulin Choosing from a tournament. Social Choice and Welfare Myerson Incentives to cultivate favored minorities under alternative electoral systems. American Political Science Review

28 Rivest and Shen 2010 An Optimal Single-Winner Preferential Voting System Based on Game Theory. conf.pdf Rubinstein Why are certain properties of binary relations relatively more common in natural languages? Econometrica. Slutzki and Volij 2006 Scoring of web pages and tournaments: axiomatizations. Social Choice and Welfare Ushakov 1976 The problem of choosing the preferred element: An application to sport games. In Management Science in Sports (Machol, Ladany and Morrison, eds.) North-Holland.