Having Fun with PageRank and MapReduce. Hadoop User Group (HUG) UK 14 th April Paolo Castagna HP Labs, Bristol, UK

Transcription

1 Having Fun with PageRank and MaReduce Hadoo Use Gou (HUG) UK 4 th Ail 29 htt://huguk.og/ Paolo Castagna HP Labs, Bistol, UK

2 PageRank /3 3 /2 3 2? 2 3 2

3 PageRank i B i ecusive definition B i backwad links (i.e. links to i ) numbe of fowad links (i.e. links fom ) 3

4 PageRank N i B k i k i k ( i ) ageank of age i at k iteation N total numbe of ages iteative comutation 4

5 Random Sufe A sufe follows links at andom indefinitely Time sent on a given age measue the imotance of that age Poblems: ank sinks (accumulate too much) cycles (could cause eiodicity) Dangling ages? Jum to any othe age Boed? Teleotation (fixes ank sinks and eliminates cycles) 5

6 Dangling Pages if is zeo? k i B i k k N andom um indeendent fom i N total numbe of ages 6

7 Teleotation i k k B k i k N d N d if thee ae loos o someone gets boed? d=.85 duming facto andom um indeendent fom i k 7

8 PageRank N N d N d i k B k i k i 8

9 Adacency Matix

10 Hyelink Matix H

11 Sase Matices

12 Adacency List bette fo sase matices 2 { 2, 3, 4 } 2 { 5 } { 5 } { 5, 6 } 5 { } 6 { } 2

13 PageRank T T T k T k T N d N d d k T T T T T e ae H N d N d d k k k a dangling node vecto e T vecto of all 3

14 Convegence How many iteations? How to check convegence? n log d k i k i n ε = -n numbe of significant digits toleance 4

15 Convegence significant digits iteations

16 Slow to convege Powe Method Each iteation comlexity: O(N) Oveall comlexity: #iteations O(N) = O(N) Minimal stoage: H sase matix, no comletely dense matices need to be stoed 6

17 Stoage Requiements Sase hyelink matix H numbe of non zeo elements (each a double) Sase binay dangling node vecto numbe of dangling nodes (each a boolean) PageRank values fo the cuent iteation N elements (each a double) (otional) PageRank values fo the evious iteation to measue toleance eo N elements (each a double) 7

18 Imlementing PageRank with MaReduce 8

19 Adacency List

20 MaReduce ob 3 age anks fom backwad links ma inut key = value = (,, 2,..., n ) outut key = i value = /n i = (, 2,..., n ) key = value = (, 2,..., n ) educe inut key = values = ( /n )*, (, 2,..., n ) outut key = value = (,, 2,..., n ) d n d d N 2

21 ma inut MaReduce ob 2 contibution fom dangling ages key =... value = dangling age outut key = value = N total numbe of ages combine and educe inut key = values = ( )* outut key =... value = d only one value d d N 2

22 MaReduce ob total numbe of ages ma inut key = value =... outut key = value = combine and educe inut key = values = ( v )* outut key =... value = N N v 22

23 Adacency List k+ k k+ 2 k k+ 3 k k+ 4 k k+ 5 k 5 6 k+ 6 k 6 23

24 ma inut MaReduce ob 4 check fo convegence key = value = ( k+, k,, 2,..., n ) outut key = value = abs ( k+ - k ) combine and educe inut key = values = ( v )* outut v key =... value = ε ε toleance 24

25 Putting all togethe ob total numbe of ages fo max n iteations o until convegence ob 2 contibution fom dangling ages ob 3 age anks fom backwad links evey y iteations ob 4 check fo convegence Total numbe of obs <= + 2n + n/y 25

26 Having Fun with PageRank Intelligent sufe Change ows of the hyelink matix H so long they emain obability distibutions Teleotation vecto (a.k.a. esonalization vecto) instead of andom ums CiteRank to ank aes (using a time deendant decay facto to shae obability distibutions of the hyelink matix) Social netwoks Ranking schemes: evaluation and comaison techniques (without involving humans?) Ranking schemes fo diected labelled multigahs (a.k.a. RDF)? 26

27 Ranking Paes: CiteSee Dataset CiteSee () (2) PageRank Title (Yea) 3426 yes yes New Diections in Cytogahy Invited Pae (976) 549 no no.2952 Stuctue and Comlexity of Relational Queies (982) no no.2267 Comutable Queies fo Relational Data Bases (98) yes yes.4733 Otimization by Simulated Annealing (983) 567 no no.2389 Pobabilistic Methods in Combinatoics (974) yes no.8669 A Method fo Obtaining Digital Signatues and Public-Key Cytosystems (978) no no.7492 Fast Anisotoic Gauss Filteing (2) no yes.743 Scheduling Algoithms fo Multiogamming in a Had-Read-Time Envionment (973) no no.968 Disceancy in Aithmetic Pogessions (996?) yes no.946 Yacc: Yet Anothe Comile-Comile (975) 3874 no yes.975 Gah-Based Algoithms fo Boolean Function Maniulation (986) no no.949 Pivacy Enhancement fo Intenet Electonic Mail: Pat II (993) 2944 no no.848 Pivacy Enhancement fo Intenet Electonic Mail: Pat III (993) no no.8948 A Timeout-Based Congestion Contol Scheme fo Window Flow-Contolled Netwoks (986) 2336 no no Genealised Additive Models (995) yes no.7477 Imlementing Remote Pocedue Calls (984) 525 no no.7384 Congestion Avoidance and Contol (988) 3536 no no.6975 Relational Queies Comutable in Polynomial Time (986) yes no The UNIX Time-Shaing (974) 3523 no no.6744 Histoy of Cicumscition (993) CiteSee - htt://citesee.ist.su.edu/{cid} () htt://en.wikiedia.og/wiki/list_of_imotant_ublications_in_comute_science (2) htt://schola.google.com/schola?as_q=%22+%22&num=&as_sub=eng 27

28 Dity Data a b c b h k c d a a c e s f f b g Ignoe dulicate links (d a a c) and self-efeences (f f b) Imlicit dangling nodes ( h, k,, s ) If the data is dity, the Google matix will not be stochastic and a unique solution as well as convegence ae not guaanteed (with a sufficient high numbe of iteations, you might get as esult ) 28

29 Convegence in Pactice Rows: anking ositions. Columns: iteations. Cells: document ids. Red: document should not be in the fist 2 esults. Yellow: document in the fist 2 esults, but wong osition. Geen: document in the fist 2 esults, coect osition. 29

30 Convegence in Pactice 3

31 Convegence in Pactice 3

32 Convegence in Pactice 32

33 Refeences Google s PageRank and Beyond: The Science of Seach Engine Rankings Amy N. Langville and Cal D. Meye Pinceton Univesity Pess (26), ISBN htt://ess.inceton.edu/titles/826.html The anatomy of a lage-scale hyetextual Web seach engine Segey Bin and Lawence Page In Poc. of the Seventh Intenational Wold Wide Web Confeence (WWW 998) htt://ilubs.stanfod.edu:89/36/ The PageRank Citation Ranking: Binging Ode to the Web Lawence Page, Segey Bin, Raeev Motwani and Tey Winogad Technical Reot, Stanfod InfoLab (999) htt://ilubs.stanfod.edu:89/422/ The Intelligent Sufe: Pobabilistic Combination of Link and Content Infomation in PageRank Matthew Richadson and Pedo Domingos In Poc. of Advances in Neual Infomation Pocessing Systems (22) htt:// Ranking Scientific Publications Using a Simle Model of Netwok Taffic Dylan Walke, Huafeng Xie, Koon-Kiu Yan, Segei Maslov Jounal of Statistical Mechanics (27) htt://axiv.og/abs/hysics/6222v 33

34