Rough incomplete sketch of history of the WorldWideWeb (Note first that WorldWideWeb Internet, WWW is rather a protocol and set of resources that is

Transcription

1 Rough ncomplete sketch of hstory of the WorldWdeWeb (Note frst that WorldWdeWeb Internet, WWW s rather a protocol and set of resources that s layered on top of the pre-exstng nternet.) 945 Memex, V.Bush. one of many hypertext forerunners 989 Berners-Lee, CERN, global hyperspace dea 990 WorldWdeweb.app on NeXT computer 99 CERN server/clent released n summer of 9, http protocol and html coded pages, also lnemode browser (lynx) growth, manly n Europe. Frst U.S. webste at Stanford Lnear Accelerator Center (992). Sprng 93 Mosac clent (NCSA), added n-lne graphcs, also produced ts own verson of httpd server software. CERN stll mantaned a lst of all webservers n world. Early 94: crawlers lke umpstaton, and WWW Worm (McBryan 994, 0,000 pages and 500 queres per day n Mar/Apr 94). Nov 97: 2M-00M docs (expected B by 2000). Altavsta handled 20M queres/day. 2000: expected 00 s of mllon/day. (Actual 2004 Google was 4.2B pages. In 2005 Yahoo and Google each clamed to have ndexed upwards of 5B pages, then stopped postng ther clamed counts.) If 0 tmes the number of pages meant every query brngs up 0 tmes as many results to sort through, then search engne methodology doesn t scale wth sze of web. Perhaps t only worked because the amount of materal on the web was stll so small? But t turned out there s a set of heurstcs for orderng the search results, so that the desred page s frequently ranked n the top ten, and t doesn t matter that there are many thousands of other pages retreved. The Page Rank methodology stems from a long hstory of ctaton analyss, where a lnk s as well some sgnal of recommendaton (or popularty). It can computed as a property of the entre graph, ndependent of any partcular query, and hence s effcent for servng a large volume of queres. The Markov process t embodes was also not new, but was appled n a partcularly powerful way, agan demonstratng the unexpected power of smple algorthms and ample computng power appled to massve datasets.

2 Brn/Page used Page Rank to measure the mportance of page, and help to order the search results. The page rank r of page s determned self-consstently by the equaton r = p + ( p) n r d, () where p s a number between 0 and (orgnally taken to be.5), the sum on s over pages pontng to, and d s the outgong degree of page. Intutvely, we see that pages wth hgh Page rank r that have low d,.e., don t pont to too many other pages, convey the most page rank to page. Formally, the above descrbes the steady state of a dffuson process, where r can be nterpreted as the probablty that one lands at page by navgatng accordng to the followng algorthm: wth probablty p one goes at random to any of the pages ponted to by page, and wth probablty p one nstead umps at random to any page anywhere on the web (so one doesn t get caught n a sngle subcomponent). Ths can also be stated as an egenvector problem. Recall the ncdence matrx A s defned by A = f ponts to and otherwse A = 0; A matrx B gvng the transton probablty from page to page can be constructed n terms of the ncdence matrx A as B = p n O + ( p) d A (2) where n = total # of pages, d s the outdegree of node, and O = (, ) s a matrx of all ones (.e., O = ). The matrx egenvector relaton r = B T r (3) s then seen to be equvalent to eqn. (), where B T s the transpose of B as defned n (2), and we have assumed that r s normalzed as a probablty, so that r O = r =. By the Perron-Frobenus theorem [detals to be added], the matrx B has a unque prncpal egenvector, correspondng to largest egenvalue, and ts components are all postve. (Snce A /d =, we fnd B = p + ( p) = and B s normalzed such that ts prncpal egenvalue s.) Thus eqn. (3) always has a soluton. To calculate the rank of all the pages, the crawler frst vsts as many pages as possble, and calculates the lnk structure of the web graph. Calculatng egenvectors of enormous matrces can be panful, but n the case of the prncpal egenvector there s a smple method. Recall that any N N matrx M has N egenvectors v () that satsfy M v () = λ v (). (4) 2

3 These egenvectors form a bass set, meanng that any other N-dmensonal vector w can be expressed as a lnear combnaton w = N = α v (), wth α constants. Now take the largest egenvalue to be λ, and consder applyng M a total of n tmes M n w = N α λ n v () = where we have used eqn (4). As n gets large we see that the term wth the largest egenvalue wll domnate, N ( M n λ ) n v() w = α v () + α α v (), λ λ n =2 snce λ /λ <. That means we can determne the prncpal egenvector smply by applyng a matrx M suffcently many tmes to any vector whch has non-zero dot product wth the prncpal egenvector (.e., non-vanshng α n the above) ths procedure effectvely proects to the egenvector of nterest. ( ) 2 Some smple 2 2 examples show how ths works. The matrx M = has 3 2 rght egenvectors v = ( ) and v2 = ( ), wth egenvalues λ = and λ 2 = /3. Consder actng on w = v + v 2 = ( 2 0) wth M many tmes: M w = ( M n w = 3 n ), M 2 w = ( ) ( 3 n + 3 n = v + 3 n v 2. ), M 3 w = ( ) , We see that as n grows larger, multplyng by M n acts to proect out the unque prncple egenvector v correspondng to the largest egenvalue, and the components of v = ( ) are all non-negatve. Ths wll not be the case ( for matrces ) that are ether not rreducble or 0 have negatve entres. For example, M = s not rreducble and has two egenvectors ( ( ) 0 ( 0) and 0 ) 2 both correspondng to egenvalue λ =. And the matrx M = 3 0 has prncple egenvector v = ( ) wth λ = 3, but v has negatve components. (The other egenvector v 2 = ( 3) has egenvalue λ =.) 3

4 Now consder the case of n = 4 web pages lnked as n the fgure below. We can go back to eqn. () and calculate r teratvely as follows: begn wth r (0) = /n, the vector of equal probablty of beng at any ste. Its dot product wth the prncpal egenvector s guaranteed to be non-vanshng, snce the prncpal egenvector has only postve components. We calculate teratvely r () = p r (0) + ( p) n d r (2) r (n) = p + ( p) n. = p + ( p) n r () d r (n ) d. The fgure shows the result of these teratons for p =.2 (and n = 4): 4

5 In the frst teraton, the rank r = /4 s multpled by.8 and then shared between pages 2 and 3, gvng. to each. Snce that s the only page pontng to 2, we see that r () 2 = =.5. For the four components of r correspondng to pages (, 2, 3, 4), we see that startng from the vector (/4, /4, /4, /4), after 9 teratons the probabltes converge to approxmately (.36,.20,.39,.05). Pages and 3 are roughly the same, page 2 s roughly half as probable as those two, and page 4 s reachable only by random umps hence has the lowest probablty. Page 3 has slghtly hgher probablty than page due to to the addtonal ncomng lnk from 4. Ths graph s also small and smple enough to work wth the transton matrx drectly. Frst we wrte the adacency matrx A for ths graph (where A = ff page ponts to page ), A =. Snce the graph has a sngle connected component, for addtonal smplcty we ll ust take p = 0, so from any page there s only a probablty of ump ng to one of the pages to whch t ponts. Then the matrx B s equal to A /d (where 0 /2 / d s the outgong degree of page ), and satsfes B = A /d = If we agan start from an equal probablty dstrbuton, n whch all four pages have = /4, and terate the equaton r (n) = r(n ) /d two tmes (.e., for n =, 2) to approxmate the soluton for the page rank, ths s equvalent to the effect of actng wth r (0) the matrx B T wth p = 0 twce on r (0) : /2 0 (B T ) /2 0 r = /2 0 / r r 2 r 4 = r /2 + r 2 + r 4 /2 r /2 + /2 0 so (B T ) 2 r (0) = (5/8, /8, /4, 0). The exact page rank for the four pages n the fgure for p = 0 can be guessed by starng long enough at them, or equvalently by drectly calculatng the prncpal egenvector r = (r, r 2,, r 4 ) of the matrx, wth components normalzed as probabltes, 4 = r =. The components of the egenvalue equaton B T r = λ r are / / mplyng the four equatons r r 2 r 4 = r /2 r /2 + r 2 + r 4 0 = λ = λr r /2 = λr 2 r /2 + r 2 + r 4 = λ 0 = λr 4. 5 r r 2 r 4

6 The prncpal egenvector, wth λ =, hence satsfes = r = 2r 2 and r 4 = 0, hence the egenvector, normalzed as a probablty s r = (2/5, /5, 2/5, 0). (The other three egenvalues are λ = 0, ( ± )/2, but only the largest one above s relevant here.) We see that page 4 has zero probablty for p = 0, snce there s no path to t. Pages and 3 have the same steady state probablty, snce page 3 conveys all of ts page rank to page, whch gves half drectly back to page 3, and the other half flows back to page 3 va page 2. Page 3 only has the hghest page rank when we take p non-zero as shown n the fgure, so that t receves extra probablty from page 4. (Incomplete notes here...) In addton to lnk analyss, other heurstcs such as locaton nformaton are used to order the search results. If the query term s a sngle word, then does t occur n the ttle of the page (metadata), or n the anchor text pontng to the page, or n the URL tself, or f t s n the text, s t n a large or small font, how many tmes does t occur, how near to the begnnng of the page does t occur? For multword queres, how close together do the query terms appear n the page (proxmty ncreases relevance). (Note that sometmes the anchor text descrbng a page <a href=... g.pg >graffe</a> gves a more accurate descrpton of page than the page tself, though can also lead to delberate collaboratve manpulaton, see google bombng ) Page rank works on the bass of lnk analyss: mportant web pages are selfconsstently defned as ones ponted to by other mportant pages. But there s potentally more structure, for example pages wth large n-degree or large out-degree mght play some specal role. Ths possblty was exploted by the HITS ( hypertext nduced topc search ) algorthm (Klenberg, 998), proposed at roughly the same tme as the Page rank algorthm. Stes wth large n-degree acqure some mplct sense of authorty, by vrtue of beng ponted to by so many other stes. They are termed authortes, and the stes wth large out-degree that pont to the most mportant authortes are termed hubs, as schematcally depcted below: In ths framework, every page has both an authorty weght a and a hub weght h, 6

7 defned to satsfy a = h, h =.e., the authorty weght of a ste s gven by the sum of the hub weghts of stes that pont to the ste, and the hub weght of a ste s gven by the sum of the authorty weghts of the stes to whch t ponts. In terms of the ncdence matrx A, these can be wrtten a = A T h, h = A a. Now we could magne startng wth some tral forms of the hub and authortes weghts, h (0) and a (0), and teratng the above equaton: a () = h (0), h () = a (0) to provde more refned guesses h () and a (), and then contnung. In matrx form, the th such teraton can be wrtten a, a () = A T h( ), h() = A a ( ). Note that the result of two teratons can be wrtten a () = A T A a ( 2), h () = AA T h( 2), so the result of 2n such teratons s a (2n) = (A T A) n a (0), h(2n) = (AA T ) n h(0). The matrces A T A and AA T are symmetrc wth non-negatve entres that sum to along any row, so for a sutably chosen subset of nodes such that they re rreducble, the Perron- Frobenus theorem wll apply. In that case, the above teratve procedure wll converge, and we see that the authorty weghts a wll be gven by the components of the prncpal egenvector of A T A, and the hub weghts h by the components of the prncpal egenvector of AA T. In the orgnal proposal, the algorthm was appled as follows: frst retreve all pages from the web that match the query terms. Then expand ths set by ncludng as well all pages that lnk to and are lnked from the pages n the orgnal set. (Ths mght nvolve a cut-off for the number of pages any gven page can brng n so that the set does not grow unmanageably large.) Then the above algorthm can be used to determne the authorty weghts for ths set and the pages wth the hghest authorty weghts can be returned frst n response to the query. Note that whle ths procedure mght produce more refned results that the Page rank algorthm, t s computatonally less effcent snce t requres calculatng a set of weghts dynamcally n response to each query. The Page rank algorthm, on the other hand, calculates the weghts once and for all for the entre drected graph of the web, and they can then be used as a statc feature to order the search results returned n response to any query. 7