Authoritative Sources in a Hyperlinked Enviroment

Size: px

Start display at page:

Download "Authoritative Sources in a Hyperlinked Enviroment"

Angelina Stevenson
5 years ago
Views:

1 Authoritative Sources in a Hyperlinked Enviroment Dimitris Sakavalas June 15, 2010

2 Outline Introduction Problem description Authorities and Broad-topic queries Algorithm outline Construction of a Focused Subgraph of the www Expanding R into the base set S Heuristics impoving base set S Computing Hubs and Authorities Iterative Algorithm Elements of Linear Algebra Convergence of the Iterative Algorithm Similar page queries Multiple Sets of Hubs and Authorities Non-Principal Eigenvectors and Clustering of G Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 2/25

3 Introduction Problem description www-search Discover pages relevant to a given query string Discover the most authoritative pages (subjective, requires human evaluation) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 3/25

4 Introduction Problem description www-search Discover pages relevant to a given query string Discover the most authoritative pages (subjective, requires human evaluation) Queries Specic queries Scarcity Problem Broad-topic queries Abundance Problem Similar-page queries Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 3/25

5 Introduction Authorities and Broad-topic queries In search of a denition of authority Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 4/25

6 Introduction Authorities and Broad-topic queries In search of a denition of authority No endogenous measure of a page to assess authority (ex. Text-based ranking functions for queries: \Harvard", \search engines", \automobile manufacturers", images) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 4/25

7 Introduction Authorities and Broad-topic queries In search of a denition of authority No endogenous measure of a page to assess authority (ex. Text-based ranking functions for queries: \Harvard", \search engines", \automobile manufacturers", images) Solution: Analysis of the link structure Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 4/25

8 Introduction Authorities and Broad-topic queries In search of a denition of authority No endogenous measure of a page to assess authority (ex. Text-based ranking functions for queries: \Harvard", \search engines", \automobile manufacturers", images) Solution: Analysis of the link structure Pitfalls: Advertising links Navigational links Universal Popularity links Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 4/25

9 Introduction Authorities and Broad-topic queries In search of a denition of authority No endogenous measure of a page to assess authority (ex. Text-based ranking functions for queries: \Harvard", \search engines", \automobile manufacturers", images) Solution: Analysis of the link structure Pitfalls: Advertising links Navigational links Universal Popularity links Question.How reliable is page p? Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 4/25

10 Introduction Algorithm outline Hubs Pages that link to many Authorities Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 5/25

Introduction Algorithm outline Hubs Pages that link to many Authorities Outline of the algorithm 1 Create a focused subgraph of the www from the output of a text-based engine

11 Introduction Algorithm outline Hubs Pages that link to many Authorities Outline of the algorithm 1 Create a focused subgraph of the www from the output of a text-based engine (Small collection of pages likely to contain the most authoritative pages) 2 Identify hubs and authorities Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 5/25

12 Construction of a Focused Subgraph of the www Directed graph G = (V; E) of the www. V the collection of hyperlinked pages of the www V : (nodes correspond to the pages) (p; q) E There is a link from p to q G[W]. Subgraph of G induced on W V Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 6/25

13 Construction of a Focused Subgraph of the www Directed graph G = (V; E) of the www. V the collection of hyperlinked pages of the www V : (nodes correspond to the pages) (p; q) E There is a link from p to q G[W]. Subgraph of G induced on W V Goal Construct G = G[S ] such that 1 S is relatively small (computational cost) 2 S rich in relevant pages (search quality) 3 S contains most of the strongest authorities Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 6/25

14 Construction of a Focused Subgraph of the www Directed graph G = (V; E) of the www. V the collection of hyperlinked pages of the www V : (nodes correspond to the pages) (p; q) E There is a link from p to q G[W]. Subgraph of G induced on W V Goal Construct G = G[S ] such that 1 S is relatively small (computational cost) 2 S rich in relevant pages (search quality) 3 S contains most of the strongest authorities Observation S rich in relevant pages many links to authorities Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 6/25

15 Construction of a Focused Subgraph of the www Notation Γ + (p): the set of all pages p points to Γ (p): the set of all pages pointing to p E: a text-based search engine Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 7/25

16 Construction of a Focused Subgraph of the www Notation Γ + (p): the set of all pages p points to Γ (p): the set of all pages pointing to p E: a text-based search engine 1st Step Root set R ={t highest ranked pages for the query from E} R satises (i); (ii). Generally doesn't satisfy (iii) and is structureless (few links) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 7/25

17 Construction of a Focused Subgraph of the www Notation Γ + (p): the set of all pages p points to Γ (p): the set of all pages pointing to p E: a text-based search engine 1st Step Root set R ={t highest ranked pages for the query from E} R satises (i); (ii). Generally doesn't satisfy (iii) and is structureless (few links) Observation If a = R, strong authority, there is a high probability that p R such that a Γ + (p) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 7/25

18 Expanding R into the base set S 2nd Step Idea:Expand R into the Base set S by adding strong authorities and relevant pages Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 8/25

19 Expanding R into the base set S 2nd Step Idea:Expand R into the Base set S by adding strong authorities and relevant pages Subgraph(; E ; t ; d) t; d N Set S := R For each page p R Add all pages in Γ + (p) to S If Γ (p) d then, Add all pages in Γ (p) to S Else Add an arbitary set of d pages from Γ (p) to S End. Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 8/25

20 Expanding R into the base set S Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 9/25

21 Heuristics impoving S Links Transverse: link between pages with dierent domain name Intrinsic: link between pages with the same domain name (navigational) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 10/25

22 Heuristics impoving S Links Transverse: link between pages with dierent domain name Intrinsic: link between pages with the same domain name (navigational) 3rd Step:Heuristics 1 Delete all Intrinsic links from the graph G 2 Allow up to a small number m pages from a single domain to point to any given page p Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 10/25

23 Heuristics impoving S Links Transverse: link between pages with dierent domain name Intrinsic: link between pages with the same domain name (navigational) 3rd Step:Heuristics 1 Delete all Intrinsic links from the graph G 2 Allow up to a small number m pages from a single domain to point to any given page p Finally we obtain a small subgraph G, relatively focused on the query, containing many relevant pages and strong authorities. Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 10/25

24 Hubs and Authorities Purely In-degree ordering of Authorities Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 11/25

25 Hubs and Authorities Purely In-degree ordering of Authorities Problem. \Universally popular" pages, large in-degree regardless of the underlying query topic (ex.amazon Books) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 11/25

26 Hubs and Authorities Purely In-degree ordering of Authorities Problem. \Universally popular" pages, large in-degree regardless of the underlying query topic (ex.amazon Books) Observation Authorities: high in-degree, considerable overlap in the sets of pages that point to them(hubs) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 11/25

27 Hubs and Authorities Observation (mutually reinforcing relationship) Good Hub: a page that points to many good authorities Good Authority: a page that is pointed by many good hubs Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 12/25

28 Hubs and Authorities Observation (mutually reinforcing relationship) Good Hub: a page that points to many good authorities Good Authority: a page that is pointed by many good hubs Denitions p S = {1; 2; :::; n} assign: authority weight x 0 hub weight y 0 Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 12/25

29 Hubs and Authorities Observation (mutually reinforcing relationship) Good Hub: a page that points to many good authorities Good Authority: a page that is pointed by many good hubs Denitions p S = {1; 2; :::; n} assign: authority weight x 0 hub weight y 0 corresponding vectors x = x <1> x <2>. x <n> and y = y <1> y <2>. y <n> Normalized such that x = y = 1 example.for norm 1 ; n i=1 x = n i=1 y = 1 Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 12/25

30 Hubs and Authorities Denitions I(x ; y) (x -weight update): x ; x q:(q;p) E y <q> Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 13/25

31 Hubs and Authorities Denitions I(x ; y) (x -weight update): x ; x q:(q;p) E O(x ; y) (y-weight update): y ; y q:(p;q) E y <q> x <q> ; Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 13/25

32 Hubs and Authorities Denitions I(x ; y) (x -weight update): x ; x q:(q;p) E O(x ; y) (y-weight update): y ; y q:(p;q) E y <q> x <q> ; Idea Find the disired \equilibrium" values for the weights by applying I; O operations iterativelly. Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 13/25

33 Hubs and Authorities Denitions I(x ; y) (x -weight update): x ; x q:(q;p) E O(x ; y) (y-weight update): y ; y q:(p;q) E y <q> x <q> ; Idea Find the disired \equilibrium" values for the weights by applying I; O operations iterativelly. Question. Does this \equilibrium" exist? Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 13/25

34 Hubs and Authorities Denitions I(x ; y) (x -weight update): x ; x q:(q;p) E O(x ; y) (y-weight update): y ; y q:(p;q) E y <q> x <q> ; Idea Find the disired \equilibrium" values for the weights by applying I; O operations iterativelly. Question. Does this \equilibrium" exist? YES (linear algebra) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 13/25

35 Iterative algorithm Iterate(G ; k) z := (1; 1; :::; 1) R n Set x 0 := z Set y 0 := z For i = 1; 2; :::; k Apply the I(x i 1 ; y i 1 ) operation to obtain new x weights x i Apply the O(x i ; y i 1) operation to obtain new y weights y i Normalize x i, obtaining x i Normalize y i, obtaining y i End Return (x k ; y k ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 14/25

36 Iterative algorithm Iterate(G ; k) z := (1; 1; :::; 1) R n Set x 0 := z Set y 0 := z For i = 1; 2; :::; k Apply the I(x i 1 ; y i 1 ) operation to obtain new x weights x i Apply the O(x i ; y i 1) operation to obtain new y weights y i Normalize x i, obtaining x i Normalize y i, obtaining y i End Return (x k ; y k ) Filter(G ; k ; c) (x k ; y k ) :=Iterate(G ; k) Report the pages with the c largest coordinates in x k as authorities Report the pages with the c largest coordinates in y k as hubs Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 14/25

37 Elements of Linear Algebra Notation Matrix M C n n Eigenvalues i (M ) Eigenvectors! i (M ) eigenspace V subspace of C n multiplicity of : m = dim(v ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 15/25

38 Elements of Linear Algebra Notation Matrix M C n n Eigenvalues i (M ) Eigenvectors! i (M ) eigenspace V subspace of C n multiplicity of : m = dim(v ) Denitions Positive matrix M : If M i;j > 0; i ; j Primitive matrix M : If m N; M m : positive Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 15/25

39 Elements of Linear Algebra Let M R n n symmetric, nonnegative matrix (M i;j 0; i ; j ) Theorems 1 M has at most n distinct eigenvalues and = n m i Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 16/25

40 Elements of Linear Algebra Let M R n n symmetric, nonnegative matrix (M i;j 0; i ; j ) Theorems 1 M has at most n distinct eigenvalues and m i = n 2 M has only real eigenvalues i Denote them 1 (M ) 2 (M ) ::: n (M ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 16/25

41 Elements of Linear Algebra Let M R n n symmetric, nonnegative matrix (M i;j 0; i ; j ) Theorems 1 M has at most n distinct eigenvalues and m i = n 2 M has only real eigenvalues i Denote them 1 (M ) 2 (M ) ::: n (M ) 3 (Perron-Frobenius) If M is primitive then: i. Largest eigenvalue 1 (M ) > 0 and m = 1 ii. 1 > i i 1 iii. 1 has a corresponding eigenvector! 1 (M ) with all entries positive (the principal eigenvector) Consider! 1 (M ) normalized,! 1 (M ) = 1 Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 16/25

42 Elements of Linear Algebra Let M R n n symmetric, nonnegative matrix (M i;j 0; i ; j ) Theorems 1 M has at most n distinct eigenvalues and m i = n 2 M has only real eigenvalues i Denote them 1 (M ) 2 (M ) ::: n (M ) 3 (Perron-Frobenius) If M is primitive then: i. Largest eigenvalue 1 (M ) > 0 and m = 1 ii. 1 > i i 1 iii. 1 has a corresponding eigenvector! 1 (M ) with all entries positive (the principal eigenvector) Consider! 1 (M ) normalized,! 1 (M ) = 1 4 If M is primitive and vector v not orthogonal to the principal eigenvector! 1 (M ), let v k be the unit vector in the direction of M k k v. Then v k! 1 (M ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 16/25

43 Elements of Linear Algebra Observation Note that we can consider! 1 (M ) as a vector of the orthonormal base of V 1. Moreover dimv 1 = m 1 = 1 and all entries in! 1 (M ) are positive. Hence the principal eigenvector! 1 (M ) is unique. Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 17/25

44 Connection of Iterative algorithm to linear Algebra Let A be the adjacency matrix of G Observation Operations I(x ; y) and O(x ; y) can be written as x A y and y Ax respectivelly Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 18/25

45 Connection of Iterative algorithm to linear Algebra Let A be the adjacency matrix of G Observation Operations I(x ; y) and O(x ; y) can be written as x A y and y Ax respectivelly Iterate Algorithm The algorithm initializes x 0 = y 0 = z =. 1 step of thalgorithm we have the following steps: I(x i ; y i 1) x i = A y i 1 O(x i ; y i ) y i = Ax i 1. In the and i th Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 18/25

46 Connection of Iterative algorithm to linear Algebra Let A be the adjacency matrix of G Observation Operations I(x ; y) and O(x ; y) can be written as x A y and y Ax respectivelly Iterate Algorithm The algorithm initializes x 0 = y 0 = z =. 1 step of thalgorithm we have the following steps: I(x i ; y i 1) x i = A y i 1 1. In the and i th O(x i ; y i ) y i = Ax i x i = A Ax i 1 = (A A) i x 0 y i = AA y i 1 = (AA ) i y 0 } x i = (A A) i z y i = (AA ) i z ( ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 18/25

47 Convergence of the Iterative Algorithm Observation We consider x i ; y i normalized in each step as in Iterate algorithm. So in every step x i = y i = 1 For simplicity, matrices AA and A A are considered primitive Obviously z c 0; c R n 0 Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 19/25

48 Convergence of the Iterative Algorithm Observation We consider x i ; y i normalized in each step as in Iterate algorithm. So in every step x i = y i = 1 For simplicity, matrices AA and A A are considered primitive Obviously z c 0; c R n 0 Theorem The sequences {x i } and {y i } produced by Iterate Algorithm converge to limits x ; y respectively Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 19/25

49 Convergence of the Iterative Algorithm Observation We consider x i ; y i normalized in each step as in Iterate algorithm. So in every step x i = y i = 1 For simplicity, matrices AA and A A are considered primitive Obviously z c 0; c R n 0 Theorem The sequences {x i } and {y i } produced by Iterate Algorithm converge to limits x ; y respectively Proof. A A primitive, z not orthogonal to the principal eigenvector! 1 (A A) = x, x i the unit vector in the direction (A A) i z Hence from Theorem 4 {x i } x, simirarly {y i } y =! 1 (AA ) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 19/25

50 Conclusion Conclusion The process of iterated I=O operations will converge. The desired equilibrium values for the x ; y weights, are the values x ; y. Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 20/25

51 Conclusion Conclusion The process of iterated I=O operations will converge. The desired equilibrium values for the x ; y weights, are the values x ; y. Observation In practice the convergence of Iterate is quite rapid, so one can compute x ; y weights by starting from any initial vectors x 0 ; y 0, and performing a xed number of I; O operations Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 20/25

52 Conclusion Conclusion The process of iterated I=O operations will converge. The desired equilibrium values for the x ; y weights, are the values x ; y. Observation In practice the convergence of Iterate is quite rapid, so one can compute x ; y weights by starting from any initial vectors x 0 ; y 0, and performing a xed number of I; O operations Hyperlink-Induced Topic Search (HITS) algorithm (Ask.com) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 20/25

53 Similar-Page Queries Query. Find pages similar to p Idea Initiate search with the page p instead of a query string Use the link structure to infer \similarity among pages" Similar pages to p Strongest authorities in local region of p Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 21/25

54 Similar-Page Queries Query. Find pages similar to p Idea Initiate search with the page p instead of a query string Use the link structure to infer \similarity among pages" Similar pages to p Strongest authorities in local region of p Adaptation of the broad-topic query method 1 Find t pages pointing to p Assemble root set R p 2 Expand R p to base set S p as before 3 Search for authorities and hubs in the focused subgraph G p Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 21/25

55 Multiple Sets of Hubs and Authorities Situation. For a query string, relevant pages grouped into clusters. Reasons Dierent meanings of. ex. \jaguar\ String arises as term in multiple technical communities. ex. \randomized algorithms" String refers to a highly polarized issue. ex.\abortion" Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 22/25

56 Multiple Sets of Hubs and Authorities Situation. For a query string, relevant pages grouped into clusters. Reasons Dierent meanings of. ex. \jaguar\ String arises as term in multiple technical communities. ex. \randomized algorithms" String refers to a highly polarized issue. ex.\abortion" Problem. Identify the clusters Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 22/25

57 Multiple Sets of Hubs and Authorities Situation. For a query string, relevant pages grouped into clusters. Reasons Dierent meanings of. ex. \jaguar\ String arises as term in multiple technical communities. ex. \randomized algorithms" String refers to a highly polarized issue. ex.\abortion" Problem. Identify the clusters Idea Clusters represented in focused subgraph G as densely linked subgraphs(collection of hubs and authorities) Extract densely linked collections of hubs and authorities through non-principal eigenvectors of AA and A A Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 22/25

58 Non-Principal Eigenvectors and Clustering of G Proposition AA and A A have the same multiset of eigenvalues, and their eigenvectors can be chosen so that! i (AA ) = A! i (A A) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 23/25

59 Non-Principal Eigenvectors and Clustering of G Proposition AA and A A have the same multiset of eigenvalues, and their eigenvectors can be chosen so that! i (AA ) = A! i (A A) Observation For each pair of eigenvectors x i =! i (A A); y i =! i (AA ) operation I(x i ; y i ) keeps x weights parallel to x i : x = A! i (AA ) = A A! i (A A) = i x i Simirarly operation O(x i ; y i ) keeps x weights parallel to y i Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 23/25

60 Non-Principal Eigenvectors and Clustering of G Proposition AA and A A have the same multiset of eigenvalues, and their eigenvectors can be chosen so that! i (AA ) = A! i (A A) Observation For each pair of eigenvectors x i =! i (A A); y i =! i (AA ) operation I(x i ; y i ) keeps x weights parallel to x i : x = A! i (AA ) = A A! i (A A) = i x i Simirarly operation O(x i ; y i ) keeps x weights parallel to y i Non-principal eigenvectors have both positive and negative entries. Hence each pair of (x i ; y i ) provide us with two densely connected set of hubs and authorities Pages that correspond to the c coordinates with the most positive values Pages that correspond to the c coordinates with the most negative values Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 23/25

61 Non-Principal Eigenvectors and Clustering of G Description of the method For each of the st few non-principal eigenvectors (x i ; y i ) nd the two densely connected set of hubs and authorities through an algorithm similar(but less clean conceptually) to the Iterate algorithm Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 24/25

62 Non-Principal Eigenvectors and Clustering of G Description of the method For each of the st few non-principal eigenvectors (x i ; y i ) nd the two densely connected set of hubs and authorities through an algorithm similar(but less clean conceptually) to the Iterate algorithm The pages with large coordinates in the rst few nonprincipal eigenvectors tend to recur, so that essentially the same collection of hubs and authorities will often be generated by several of the strongest nonprincipal eigenvectors Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 24/25

63 Similar Concepts in Other Areas Social networks \Standing" Bibliometrics \Impact" Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 25/25

64 Similar Concepts in Other Areas Social networks \Standing" Bibliometrics \Impact" Economics (Wassily Leontief 1941-Nobel prize 1973) Dimitris Sakavalas, Authoritative Sources in a Hyperlinked Enviroment 25/25

Data Mining and Matrices

Data Mining and Matrices 10 Graphs II Rainer Gemulla, Pauli Miettinen Jul 4, 2013 Link analysis The web as a directed graph Set of web pages with associated textual content Hyperlinks between webpages