Rank and Rating Aggregation. Amy Langville Mathematics Department College of Charleston

Transcription

1 Rank and Rating Aggregation Amy Langville Mathematics Department College of Charleston Hamilton Institute 8/6/2008

2 Collaborators Carl Meyer, N. C. State University College of Charleston (current and former students) Luke Ingram Kathryn Pedings Neil Goodson Colin Stephenson Emmie Douglas

3 Outline Popular Ranking Methods Massey Colley Markov HITS New Ranking Methods Rank Differential Method Rating Differential Method Aggregation Methods of Comparison Rank Aggregation Rating Aggregation

4 Popular Ranking Methods

5 Ranking Problem given data of n items, create a ranked list of these items - (e.g., pairwise comparisons) Applications webpages (PageRank, HITS, SALSA,...) sports teams (Massey, Colley, Markov, mhits,...) recommendation systems (Netflix movies, Amazon books,...) Related Problem cluster n items into groups

6 The Data A 0 Sports Examples A = team team stats ; A = team team stat differentials team A = game wins & losses

7 Popular Ranking Methods Massey: symmetric linear system Mr = p Colley: s.p.d linear system Cr = b Markov: irreducible eigensystem Vr = r, where V 0 mhits: Sinkhorn-Knopp algorithm on P 0

8 Popular Ranking Methods Massey: symmetric linear system Mr = p Colley: s.p.d linear system Cr = b Markov: irreducible eigensystem Vr = r, where V 0 mhits: Sinkhorn-Knopp algorithm on P 0 ranking methods are adapted to fit the application (webpages, sports teams, movies, etc.)

9 Popular Ranking Methods Markov Method

10 Markov Method Voting Matrix V 0 losers vote with points given up (or some other statistic) winners and losers vote with points given up losers vote with point differentials Duke V = Duke Miami UNC UVA VT Duke Miami UNC UVA VT Miami UNC UVA VT 38 vote with multiple statistics V = α 1 V 1 + α 2 V α k V k

11 Fair Weather Fan s Random Walk row normalize to make V stochastic V = Check/enforce irreducibility Duke Miami UNC UVA VT Duke 0 45/124 3/124 31/124 45/124 Miami 1/5 1/5 1/5 1/5 1/5 UNC 0 18/ /45 UVA 0 8/48 2/ /48 VT Solve eigensystem: Vr = r r = fair weather fan s long-term visit proportions not as successful at ranking and predicting winners as mhits (coming soon)

12 Nonnegativity and Markov enforce irreducibility, aperiodicity V = V + e e T 0 P-F guarantees existence and uniqueness of r P-F guarantees convergence and rate of convergence of power method on V

13 Popular Ranking Methods mhits Method

14 mhits Method each team i gets both offensive rating o i and defensive rating d i mhits Thesis: A team is a good defensive team (i.e., deserves a high defensive rating d j ) when it holds its opponents (particularly strong offensive teams) to low scores. A team is a good offensive team (i.e., deserves a high offensive rating o i ) when it scores many points against its opponents (particularly opponents with high defensive ratings). Miami 17 UNC Duke VT UVA P= Duke Miami UNC UVA VT Duke Miami UNC UVA VT Graph Point Matrix P 0

15 Summation Notation mhits Equations d j = X i I j p ij 1 o i and o i = X j L i p ij 1 d j Matrix Notation: iterative procedure d (k) = P T 1 o (k) and o (k) = P 1 d (k 1) related to the Sinkhorn-Knopp algorithm for matrix balancing (uses successive row and column scaling to transform P 0 into doubly stochastic matrix S) P. Knight uses Sinkhorn-Knopp algorithm to rank webpages

16 mhits Results: tiny NCAA (data from Luke Ingram) Duke Miami 38 VT UNC 7 UVA 5

17 Weighted mhits Weighted Point Matrix P 0 p ij = w ij p ij (w ij = weight of matchup between teams i and j) possible weightings w ij w Linear w Logarithmic t 0 tf t t 0 tf t w Exponential w Step Function t 0 tf t t 0 t s tf t

18 mhits Results: full NCAA (image from Neil Goodson and Colin Stephenson)

19 mhits Results: full NCAA weightings can easily be applied to all ranking methods interesting possibilities for weighted webpage ranking

20 mhits Point Matrix P 0 Perron-Frobenius guarantees (for irreducible P with total support) existence of o and d uniqueness of o and d convergence of mhits algorithm rate of convergence of mhits algorithm σ2 2 (S), where S = D (1/o) P D (1/d)

21 mhits on Netflix each movie i gets a rating m i and each user gets a rating u j mhits Thesis: A movie is a good (i.e., deserves a high rating m i ) if it gets high ratings from good (i.e., discriminating) users. A user is good (i.e., deserves a high rating u j ) when his or her ratings match the true rating of a movie. mhits Netflix Algorithm u = e; for i = 1 : maxiter m = A u; m = 5(m min(m)) max(m) min(m) ; u = 1 ((R (R>0). (em T )). 2 )e ; end

22 Netflix mhits results movies,.5 million users David Gleich s subset: 1500 super users 1 st Raiders of the Lost Ark 2 nd Silence of the Lambs 3 rd The Sixth Sense 4 th Shawshank Redemption 5 th LOR: Fellowship of the Ring 6 th The Matrix 7 th LOR: The Two Towers 8 th Pulp Fiction 9 th LOR: The Return of the King 10 th Forrest Gump 11 th The Usual Suspects 12 th American Beauty 13 th Pirates of the Carribbean: Black Pearl 14 th The Godfather 15 th Braveheart (rate 1000 movies)

23 New Ranking Methods

24 Ranking Philosophies Old e.g. Data Team 1 Team 2... Team n Team Team Team n Method Massey Mr = p Rating Vector Ranking Vector

25 Ranking Philosophies Old Data Method Rating Vector Ranking Vector e.g. Team 1 Team 2... Team n Team Team Team n Massey Mr = p New e.g. Data Method Ranking Vector Team 1 Team 2... Team n Team Team Team n Rank Differential

26 New Ranking Methods Rank Differential Method

27 Ranking Vector Every ranking vector is a permutation Relative positions matter One-to-one mapping: ranking vector rank differential matrix R

28 Ranking Vector Every ranking vector is a permutation Relative positions matter One-to-one mapping: Example: ranking vector rank differential matrix R

29 Ranking Vector Every ranking vector is a permutation Differences in position have meaning One-to-one mapping: ranking vector Example: rank differential matrix R Every rank differential matrix R is a reordering of the fundamental rank differential matrix ˆR ˆR = corresponds to ranking vector

30 Rank Differential Method Input Matrix: D Output Matrix: R

31 Rank Differential Method Input Matrix: D Output Matrix: R GOAL: Find symmetric reordering of D that min q kd(q, q) ˆRk D = data differential matrix (e.g., Markov voting matrix V) q = permutation vector ˆR = fundamental rank differential matrix

32 Rank Differential Example D = Duke Miami UNC UVA VT Duke Miami UNC UVA ˆR = VT Find ordering of teams that brings D closest to ˆR

33 Rank Differential Example D = Duke Miami UNC UVA VT Duke Miami UNC UVA ˆR = VT Find ordering of teams that brings D closest to ˆR Normalize

34 Rank Differential Example D = Duke Miami UNC UVA VT Duke Miami UNC UVA ˆR = VT Find ordering of teams that brings D closest to ˆR Normalize

35 Rank Differential Example D = Duke Miami UNC UVA VT Duke Miami UNC UVA ˆR = VT Optimal ordering: q = [ ] (q, q) = Duke Miami UNC UVA VT Duke Miami UNC UVA VT ˆR =

36 Rank Differential Example ^ reordered D Reordered D R ˆR

37 Rank Differential Example ^ reordered D Reordered D R ˆR rank differential method Duke Miami UNC UVA VT 5 th 2 nd 3 rd 4 th 1 st mhits method Duke Miami UNC UVA VT 5 th 2 nd 4 th 3 rd 1 st

38 Gottlieb 1982

39 Gottlieb 1982

40 Solving the Optimization Problem min q kd(q, q) ˆRk Huge solution space: n! permutations q (related to TSP; NP-hard) Evolutionary Algorithm initialize population with k=10 solutions {x 1,x 2,...,x k } until convergence compute fitness kd(x i,x i ) ˆRk for each x i create new population by copy 3 fittest x i into next generation pair 6 fittest x i and mate with rank aggregation (coming soon) mutate next 3 x i with flip, invert, reversal operators insert 1 immigrant x i using random permutation end guaranteed to converge to global min (Fogel, Michelawicz) but slow

41 New Ranking Methods Rating Differential Method

42 Rating Differential Method Rank mapping: ranking vector rank differential matrix R Rating Mapping: rating vector rating differential matrix R

43 Rating Differential Method Rank mapping: ranking vector rank differential matrix R Rating Mapping: Example: rating vector rating differential matrix R

44 Rating Differential Method Rank mapping: ranking vector rank differential matrix R Rating Mapping: Example: rating vector rating differential matrix R No fundamental rating differential matrix BUT there is a fundamental form for rating differential matrix

45 Rating Differential Fundamental Form (R-bert form)

46 Rating Differential Method GOAL: Find symmetric reordering of D so that D(q,q) min q (# violations of fundamental form constraints)

47 Rating Differential Example D = Duke Miami UNC UVA VT Duke Miami UNC UVA VT D(q, q) = Miami VT UNC UVA Duke Miami VT UNC UVA Duke

48 Rating Differential Example D = D(q, q) = Duke Miami UNC UVA VT Duke Miami UNC UVA VT Miami VT UNC UVA Duke Miami VT UNC UVA Duke mhits method rank differential method rating differential method Duke Miami UNC UVA VT 5 th 2 nd 4 th 3 rd 1 st Duke Miami UNC UVA VT 5 th 2 nd 3 rd 4 th 1 st Duke Miami UNC UVA VT 5 th 1 st 3 rd 4 th 2 nd

49 Solving the Optimization Problem min q (# violations of fundamental form constraints) Huge solution space: n! permutations q (related to TSP; NP-hard) Evolutionary Algorithm initialize population with k=10 solutions {x 1,x 2,...,x k } until convergence compute fitness kd(x i,x i ) ˆRk for each x i create new population by copy 3 fittest x i into next generation pair 6 fittest x i and mate with rank aggregation mutate next 3 x i with flip, invert, reversal operators insert 1 immigrant x i using random permutation end guaranteed to converge to global min (Fogel, Michelawicz) but slow

50 Solving the Optimization Problem min q (# violations of fundamental form constraints) Huge solution space: n! permutations q (related to TSP; NP-hard) Evolutionary Algorithm initialize population with k=10 solutions {x 1,x 2,...,x k } until convergence compute fitness # violations for each x i create new population by copy 3 fittest x i into next generation pair 6 fittest x i and mate with rank aggregation mutate next 3 x i with flip, invert, reversal operators insert 1 immigrant x i using random permutation end guaranteed to converge to global min (Fogel, Michelawicz) but slow

51 Aggregation

52 Aggregation Methods of Comparison

53 Methods of Comparison Many methods, which is best? Qualitative < 1 plots bipartite line graphs Quantitative distance between two ranked lists * Kendall s τ * Spearman s footrule distance to aggregated list

54 Methods of Comparison < 1 plots 2008 SoCon results (Neil Goodson and Colin Stephenson) -

55 Methods of Comparison bipartite line graphs 2005 NCAA basketball

56 Methods of Comparison distance between two ranked lists Kendall s τ on full lists of length n 1 τ = n c n d n 2 1 n c = # concordant pairs n d = # discordant pairs τ = 1, lists in complete agreement τ = 1, one list is reverse of the other

57 Methods of Comparison distance between two ranked lists Kendall s τ on full lists of length n 1 τ = n c n d n 2 1 n c = # concordant pairs n d = # discordant pairs τ = 1, lists in complete agreement τ = 1, one list is reverse of the other What about partial lists, like top-k lists?

58 Methods of Comparison bipartite line graphs 2005 NCAA basketball

59 Kendall s Tau on partial lists τ = n n c n d n c = # concordant pairs u n n 2 d = # discordant pairs nu n u = # unlabeled pairs Bounds n 2 n τ 2 nu n 2 n 2 nu

60 Kendall s Tau on partial lists τ=.67 τ=.06

61 Methods of Comparison distance between two ranked lists Spearman s footrule on full lists l and l of length n nx 0 φ = l(i) l(i) φ=kl lk 1 i=1

62 Methods of Comparison distance between two ranked lists Spearman s footrule on full lists l and l of length n nx 0 φ = l(i) l(i) φ=kl lk 1 i=1 BUT disagreements in lists are given equal weight

63 Methods of Comparison distance between two ranked lists Weighted footrule on full lists l and l of length n P n i=1 φ = l(i) l(i) min{l(i), l(i)}

64 Methods of Comparison distance between two ranked lists Weighted footrule on full lists l and l of length n P n i=1 φ = l(i) l(i) min{l(i), l(i)} What about partial lists, like top-k lists?

65 Weighted Footrule on partial lists φ=.05 φ=.27

66 Weighted Footrule on partial lists

67 Weighted Footrule vs. Kendall Tau A B C D E F G H I J K L A B C D E F H G J I L K B A D C F E H G J I L K Kendall Tau: weighted footrule: τ=.95 φ=.34 τ=.95 φ=1.53

68 But which list is best? Methods of Comparison distance to aggregated list φ=.05 φ=.27 mhits Aggregate Markov

69 Aggregation Rank Aggregation

70 Rank Aggregation

71 Rank Aggregation average rank Borda count simulated data graph theory

72 Average Rank

73 Borda Count for each ranked list, each item receives a score equal to the number of items it outranks.

74 Simulated Data 3 ranked lists mhits VT beats Miami by 1 point, UVA by 2 points,... Miami beats UVA by 1 point, UNC by 2 points,... UVA beats UNC by 1 point, Duke by 2 points UNC beats Duke by 1 point repeat for each ranked list generates game scores for teams Simulated Game Data

75 Simulated Data

76 Graph Theory more voting ranked lists are used to form weighted graph possible weights * w ij = # of ranked lists having i below j * w ij = sum of rank differences of lists having i below j run algorithm (e.g., Markov, PageRank, HITS) to determine most important nodes

77 Aggregation Rating Aggregation

78 Rating Aggregation rating vectors form rating differential matrix R for each rating vector

79 Rating Aggregation rating differential matrices R 0 differing scales normalize

80 Rating Aggregation rating differential matrices

81 Rating Aggregation rating differential matrices combine into one matrix average

82 Rating Aggregation rating differential matrices combine into one matrix average

83 Rating Aggregation average rating differential matrix R average 0 run ranking method * Markov method on R T average * row sums of R average / col sums of R average * Perron vector of R average

84 Rating Aggregation average rating differential matrix R average 0 run ranking method * Markov method on R T average * row sums of R average / col sums of R average * Perron vector of R average

85 Conclusions several methods for rating and ranking items begin by building nonnegative matrices * Markov: stationary vector of V 0 * mhits: Sinkhorn-Knopp on P 0 * Rank Differential: reordering of D 0 * Rating Differential: reordering of D 0 nonnegative matrix theory many times insures existence, uniqueness, convergence. sometimes nonnegativity is forced to guarantee these properties rank and rating aggregation also build nonnegative matrices: W 0, R 0