Ranking from Pairwise Comparisons

Transcription

1 Ranking from Pairwise Comparisons Sewoong Oh Department of ISE University of Illinois at Urbana-Champaign Joint work with Sahand Negahban(MIT) and Devavrat Shah(MIT) / 9

2 Rank Aggregation Data Algorithm Decision / Score Ranking / 9

3 Example Data Algorithm Decision kittenwar.com / 9

4 Example Data Algorithm Decision.8.. kittenwar.com / 9

5 Example Data Algorithm Decision kittenwar.com / 9

6 Example Data Algorithm Decision / 9

7 Example Data Algorithm Decision / 9

8 Comparisons data Group decisions, recommendation and advertisement, sports and game Universal (ratings can be converted into comparisons) Consistent and reliable Natural (e.g. Sports and games, MSR s TrueSkill) Aggregation is challenging / 9

9 NP-hard approach: Kemeny optimal algorithm a a Find the most consistent ranking arg min a ij I(σ(j) > σ(i)) σ i j NP-hard combinatorial optimization Optimal for a specific model Strong Weak w.p. p w.p. p / 9

10 Traditional approach: l Ranking a a Compute score: s(i) = i j i a ji a ij + a ji n: number of items m: number of samples Compute in O(m) time Works when everyone plays everyone else: m = Ω(n ) [Ammar, Shah ] / 9

11 Traditional approach: l Ranking a a Compute score: s(i) = i j i a ji a ij + a ji n: number of items m: number of samples Compute in O(m) time Works when everyone plays everyone else: m = Ω(n ) [Ammar, Shah ] Strong Weak / 9

12 Traditional approach: l Ranking a a Compute score: s(i) = i j i a ji a ij + a ji n: number of items m: number of samples Compute in O(m) time Works when everyone plays everyone else: m = Ω(n ) [Ammar, Shah ] Strong All wins are equally weighted Weak / 9

13 Traditional approach: l Ranking a a Compute score: s(i) = a ji s(j) i a ij + a ji j i n: number of items m: number of samples Compute in O(m) time Works when everyone plays everyone else: m = Ω(n ) [Ammar, Shah ] Strong All wins are equally weighted Weak / 9

14 Prior work Data Algorithm Decision l ranking l p ranking Kemeny optimal MNL Mixed MNL etc. / Score Ranking / 9

15 Challenges High-dimensional regime # of samples n(log n) Low computational complexity # of operations # of samples Model independent Makes sense 9 / 9

16 Challenges High-dimensional regime # of samples n(log n) Low computational complexity # of operations # of samples Model independent Makes sense Under BTL model, Rank Centrality achieves optimal performance 9 / 9

17 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes 0 / 9

18 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes 0 / 9

19 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes 0 / 9

20 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes 0 / 9

21 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes 0 / 9

22 Rank Centrality a a Define a random walk on G a ij P ij = d max a ij + a ji P ii = a ij d max a ij + a ji j i (unique) stationary distribution s T P = s T Random walk spends more time on stronger nodes Higher score for beating a stronger node ( s(i) = a ) ij s(i) + P ji s(j) d max a ij + a ji j i = Z i j i j i a ji a ij + a ji s(j) 0 / 9

23 Experiment: Polling public opinions Washington Post - allourideas / 9

24 Experiment: Polling Ground truth: what the algorithm produces with complete data Error = n i (σ i ˆσ i ) Error L ranking Rank Centrality Sampling rate / 9

25 Model Algorithm is model independent For experiments and analysis, need model to generate data Comparisons model Bradley-Terry-Luce (BTL) model there is a true ranking {w i } when a pair is compared, the noise is modelled by P (i < j) = w j w i + w j / 9

26 Model Algorithm is model independent For experiments and analysis, need model to generate data Comparisons model Bradley-Terry-Luce (BTL) model there is a true ranking {w i } when a pair is compared, the noise is modelled by P (i < j) = Sampling model Sample each pair with probability d/n k comparisons for each pair w j w i + w j k d k / 9

27 Experiment: BTL model k d Error= w i>j (w i w j ) I ( ) (ˆσ i ˆσ j )(w i w j )>0 0. Error 0.0 Ratio Matrix L ranking Rank Centrality ML estimate 0. Error Ratio Matrix L ranking Rank Centrality ML estimate k d/n / 9

28 Performance guarantee For d = Ω(log n) BTL model {w i } with w min = Θ(w max ) Theorem (Negahban, O., Shah, ) Rank Centrality achieves w s w C log n k d Information-theoretic lower bound: inf s sup w W w s w C k d / 9

29 Performance guarantee For d = Ω(log n) BTL model {w i } with w min = Θ(w max ) Theorem (Negahban, O., Shah, ) Rank Centrality achieves w s w C log n k d Information-theoretic lower bound: inf s sup w W w s w C k d # of samples = O(n log n) sufficies to achieve arbitrary small error / 9

30 Performance guarantee for general graphs Oftentimes we do not control how data is collected Let G denote the (undirected) graph of given data Compute spectral gap of G (cf. mixing time of natural random walk) ξ λ (G) λ (G) Theorem (Negahban, O., Shah, ) Rank Centrality achieves w s w C d max log n ξ d min k d max / 9

31 Performance guarantee for general graphs Oftentimes we do not control how data is collected Let G denote the (undirected) graph of given data Compute spectral gap of G (cf. mixing time of natural random walk) ξ λ (G) λ (G) Theorem (Negahban, O., Shah, ) Rank Centrality achieves w s w C d max log n ξ d min k d max # of samples = O(n log n) sufficies to achieve arbitrary small error / 9

32 Proof technique. Spectral analysis of reversible Markov chains Markov chain (P, π) is revisible iff π(i)p ij = π(j)p ji P ij = d max a ij a ij + a ji is not reversible, but the expectation π(i) P ij = π(j) P ji P ij = π(i) w i d max w j w i + w j / 9

33 Proof technique. Spectral analysis of reversible Markov chains For any Markov chain (P, π) and any reversible MC ( P, π) π π = P T π P T π P T π P T π + P T π P T π P T (π π) + (P }{{} P ) T (π π) + P P π Reversible ( λ ( P ) + P P ) π π + P P π π π π P P λ ( P ) P P 8 / 9

34 Proof technique. Spectral analysis of reversible Markov chains For any Markov chain (P, π) and any reversible MC ( P, π) π π = P T π P T π P T π P T π + P T π P T π P T (π π) + (P }{{} P ) T (π π) + P P π Reversible ( λ ( P ) + P P ) π π + P P π π π π P P λ ( P ) P P. Bound λ ( P ) by comparisons theorem. Bound P P by concentration of measure inequality for matrices 8 / 9

35 Conclusion Rank aggregation from comparisons High-dimensional regime nkd n log n Low complexity O(nkd log n) Model independent Optimal under BTL up to log n Other Markov chain approaches e.g. Rank Aggregation Revisited C. Dwork, R. Kumar, M. Naor, and D. Sivakumar 9 / 9