Link Analysis and Web Search

Transcription

1 Link Analysis and Web Search Episode 11 Baochun Li Professor Department of Electrical and Computer Engineering University of Toronto

2 Link Analysis and Web Search (Chapter 13, 14)

3 Information networks and the Web Logical relationships among pieces of information Best example: the Web 1991: Tim Berners-Lee at CERN (Switzerland) created the Web provided an easy way to make documents web pages for the world to see view these pages using browsers it is based on the idea of connecting these pages using links 3

4 The idea of links is both inspired and non-obvious There are many ways of organizing information: classification (library), series of folders (files), or just alphabetically (phone book) 4

5 Modeling the web as a directed graph Objective: create a map of the web But how? 5

6 Strongly Connected Components Strongly connected component: a subset of nodes such that (1) every node in the subset has a path to every other; and (2) the subset is not part of some larger set in which every node can reach every other. 6

7

8 Now we can build a global map of the Web, using strongly connected components (Broder et al. [1999])

9 A giant strongly connected component

10 Can there be a second giant strongly connected component? 10

11 Not really it s too fragile 11

12 The bow-tie structure of the web 12

13 Tendrils 44 Million nodes IN SCC OUT 44 Million nodes 56 Million nodes 44 Million nodes Tubes Disconnected components

14 How do we find web pages using search? Up through the 1980s, very few people cared about information retrieval (search) librarians patent attorneys They are trained to formulate effective queries, and the documents they were searching for were written by professionals research articles court documents U.S. patents 14

15 The Web is entirely different Both search users and web page authors are amateurs Scale is really large Highly dynamic nature of the content to be searched Some of the authors may even optimize their content for a search engine An industry called Search Engine Optimization Millions of dollars on the line 15

16 When I search for a key phrase, what do I need? University of Washington University of Waterloo University of Wisconsin University of Windsor University of Winnipeg University of Wyoming? 16

17 Basic idea: let the links vote 17

18 Using links as more than simple votes 18

19 Searching for good museum 19

20 Link-based ranking with hubs and authorities Key idea: voting again and again 20

21 Link-based ranking with hubs and authorities Key idea: principle of repeated improvement 21

22 But why stop here? 22

23 Let s make it more formal Two kinds of quality measures for web pages Authority score Auth(p): level of endorsement Hub score Hub(p) : quality as a list Authority update rule: Auth(p) = sum of hub scores of all pages that link to p. Hub update rule: Hub(p) = sum of authority scores of all pages that p points to. Divide all scores so that they add to 1. 23

24 Using adjacency matrices to represent a graph view a set of n pages, 1, 2, n, as a set of nodes in a directed graph n x n matrix M: Mij is equal to 1 if thee is a link from node i to node j node node node 3 node

25 Hub update rule as matrix multiplication h i M i1 a 1 + M i2 a 2 + +M in a n h Ma. node node = node 3 node 4 25

26 The authority update rule as matrix multiplication a i M 1i h 1 + M 2i h 2 + +M ni h n. a M T h. 26

27 Unwinding the k-step hub-authority updates a 1 = M T h 0 h 1 = Ma 1 = MM T h 0 a 2 = M T h 1 = M T MM T h 0 h 2 = Ma 2 = MM T MM T h 0 = (MM T ) 2 h 0. a k = (M T M) k 1 M T h 0 h k = (MM T ) k h 0 27

28 Thinking about multiplication in terms of eigenvectors We wish to show that there are constants, c and d, such that the sequences of vectors, h <k> /c k and a <k> /d k converge to limits as k goes to infinity Let s consider hub vectors first After convergence: h k c k = (MMT ) k h 0 c k,whatpropertiesdo (MM T )h = ch In other words, h <*> is the eigenvector of the matrix MM T, and c is the corresponding eigenvalue. 28

29 The convergence of hub vector updates Any symmetric matrix A with n rows and n columns has a set of n eigenvectors, that are all unit vectors and all mutually orthogonal they form the basis for the space R n. Since MM T is symmetric, it has n orthogonal eigenvectors z1, z2,, zn, and their corresponding eigenvalues c1, c2,, cn (let c1 > c2 >= cn ) Given + vector + + x: (MM T )x = (MM T )(p 1 z 1 + p 2 z 2 + +p n z n ) = p 1 MM T z 1 + p 2 MM T z 2 + +p n MM T z n = p 1 c 1 z 1 + p 2 c 2 z 2 + +p n c n z n, (MM T ) k x = c k 1 p 1z 1 + c k 2 p 2z 2 + +c k n p nz n. 29

30 The convergence of hub vector updates = + + h k = (MM T ) k h 0 = c1 k q 1z 1 + c2 k q 2z 2 + +cn k q nz n, and if we divide both sides by c1 k,thenweget h k c k 1 = q 1 z 1 + ( c2 c 1 ) k q 2 z ( cn c 1 ) k q n z n. 30

31 PageRank: the core of Google search 31

32 Basic PageRank update rule 32

33

34

35 Basic PageRank update rule Assign each page p a PageRank value start with 1/n, n is the number of pages Basic PageRank update rule Each node divides its current PageRank into equal shares, and then pass them across outbound links And then use the principle of repeated improvement Fact: If the network is strongly connected, then there are unique equilibrium PageRank values. 35

36 Equilibrium PageRank values 36

37 Basic PageRank updates as matrix multiplication Each entry in the adjacency matrix Nij specifies the portion of i s PageRank that should be passed to j in one single step start with 1/Li, where Li is the number of links out of i node 1 0 1/2 0 1/ /2 1/2 node node 3 node

38 Basic PageRank updates as matrix multiplication Vector r: the PageRanks of all the nodes r i N 1i r 1 + N 2i r 2 + +N ni r n. r N T r. 38

39 One major problem with the basic update rule 39

40 One major problem with the basic update rule 40

41

42 Scaled PageRank update rule First apply the basic PageRank Update rule Then scale down all PageRank values by a factor of s The total PageRank in the network has shrunk from 1 to s We then divide the residual 1-s units of PageRank equally over all nodes, given (1 - s)/n to each 42

43 Scaled PageRank update rule Fact: Repeatedly applying the scaled PageRank update rule converges to a unique equilibrium set of PageRank values for all networks. node node node 3 node 4 here Ñ ij sn ij + (1 s)/n, r i Ñ 1i r 1 + Ñ 2i r 2 + +Ñ ni r n r Ñ T r. r k = (Ñ T ) k r 0. 43

44 Convergence to r <*> We wish to show that r <k> converges to r <*>, so that: Ñ T r = r ; ding eigenvalue Unfortunately, we cannot use the fact that the matrix (e.g. MM T ) is symmetric, since it is not But fortunately, there is a Perron s theorem stating that there is a unique vector y that remains fixed under the application of the scaled PageRank update rule, and that the update rule converges to y from any starting point. 44

45 Chapter , , 14.6