Analyzing Evolution of Web Communities using a Series of Japanese Web Archives

Size: px

Start display at page:

Download "Analyzing Evolution of Web Communities using a Series of Japanese Web Archives"

Bruce Roland Griffith
5 years ago
Views:

1 {toyoda,kitsure}@tkl.iis.u-tokyo.ac.jp Analyzing Evolution of Web Communities using a Series of Japanese Web Archives Masashi TOYODA and Masaru KITSUREGAWA Institute of Industrial Science, University of Tokyo 4 6 Komaba Meguro-ku, Tokyo, Japan {toyoda,kitsure}@tkl.iis.u-tokyo.ac.jp Abstract In this paper, we analyze evolution of web communities using a series of Japanese web archives. A web community is a set of web pages created by individuals or associations with a common interest on a topic. So far various techniques have been developed to extract web communities by link analysis. We examined evolution of web communities by comparing Japanese web archives crawled four times from 999 to Several metrics are introduced for measuring the degree of web community evolution, such as growth rate, novelty, and stability. Statistics of these archives and the evolution metrics are shown, and the global behavior of evolution is described. We found that most of changes in communities follow the power-law, and the changes are symmetric between split and merge, and between growth and shrinkage. Key words Web, link analysis, web community, evolution. URL [4] WWW

2 [2], [4], [5], [7] [2] () (2) (3) [3] [3] Fig A part of our web community chart Yahoo! /../Maker/Electric PC vendor links Computer vendors IBM TOSHIBA SONY 2 Authority hub Fig. 2 Typical graph structure of hubs and authorities [3] [6], [3] () (2) authority hub (3) authority authority hub hub authority hub authority [6], [3] 2 authority hub IBM TOSHIBA SONY authority hub 2 authority hub IBM TOSHIBA SONY IN IN

3 N N a b [3] 2 () 3. t,t 2,..., t n: W (t k ): t k C(t k ): W (t k ) c(t k ),d(t k ),e(t k ),...: C(t k ) 3. (W (t ),W(t 2),..., W (t n)) () (C(t ),C(t 2),..., C(t n)) (2) t k C(t k ) 2 C(t k ) C(t k ) t k t k : c(t k ) C(t k ) URL c(t k ) C(t k ) c(t k ) URL W (t k ) : c(t k ) C(t k ) c(t k ) c(t k ) URL W (t k ) : C(t k ) c(t k ) C(t k ) c(t k ) URL 2 URL URL URL URL : c(t k ) C(t k ) URL : c(t k ) C(t k ) URL 3. 2 c(t k ) c(t k ) c(t k ) c(t k ) C(t k ) t k c(t k ) C(t k ) URL c(t k ) c(t k ) URL URL c(t k ) t k c (t k ) c (t k ) c(t k ) (c(t k ), c (t k )) c(t k ) c(t k ) N(c(t k )): N sh (c(t k ),c(t k )): N dis (c(t k )): N sp(c(t k ),c(t k )): N ap(c(t k ): N mg(c(t t ),c(t k )): c(t k ) c(t k ) c(t k ) c(t k ) c(t k ) c(t k ) c(t k ) c(t k ) c(t k ) URL : URL c(t k ) N(c(t k )) 0

4 R grow(c(t k ),c(t k )) = N(c(t k)) N(c(t k )) t k t k : URL URL URL URL : R stability (c(t k ),c(t k )) = N(c(t k )) + N(c(t k )) 2N sh (c(t k ),c(t k )) t k t k URL R novelty (c(t k ),c(t k )) = Nap(c(t k)) t k t k : URL R disappear (c(t k ),c(t k )) = N dis(c(t k )) t k t k : URL R split (c(t k ),c(t k )) = Nsp(c(t k ),c(t k )) t k t k : URL R merge(c(t k ),c(t k )) = Nmg(c(t k ),c(t k )) t k t k : : : Year Period Crawled pages Total URLs Links 999 Jul. to Aug. 7M 34M 20M 2000 Jun. to Aug. 7M 32M 2M 200 Early Oct. 40M 76M 33M 2002 Early Feb. 45M 84M 375M Table Details of our web archives (c(t i), c(t i+),..., c(t j)) j k=i R novelty (c(t i),c(t j)) = Nap(c(t k)) t j t i jp URL URL URL com edu URL URL URL URL connectivity server [] Sun Enterprise Server URL URL ( URL ) 3 URL URL URL URL URL 3 URL URL

5 8.E+07 7.E+07 6.E+07 5.E+07 4.E+07 3.E+07 2.E+07 Transition of URLs in web graphs.e+07 0.E+00 Aug-99 Oct-99 Dec-99 Feb-00 Apr-00 Jun-00 Aug-00 Oct-00 Dec-00 Feb-0 Apr-0 Jun-0 Aug-0 Oct-0 Dec-0 Feb-02 Number of URLs Year Period Seeds Communities 999 Jul. to Aug. 657K 79K 2000 Jun. to Aug. 737K 88K 200 Early Oct. 404K 56K 2002 Early Feb. 5K 70K 2 URL Table 2 The number of seeds and communities Time 3 Fig. 3 URL Transition of URLs in web graphs URL % URL % URL URL 5 URL URL () [3], [0], [2] i URL i /i k [3], [0], [2] k 2. k URL IN 3 3 URL IN N N 0 0 ( [3] ) 2 URL 5. 4 Fig. 4 An emerged community around an Islam information community URL [5] URL URL URL (

e+06 Size distribution of communities Aug 999 Aug 2000 Oct 200 Feb 2002 Number of communities 0000 000 00 0 0 00 000 Size of community 6 Fig.

6 e+06 Size distribution of communities Aug 999 Aug 2000 Oct 200 Feb 2002 Number of communities Size of community 6 Fig. 6 Size distribution of communities Transition of communities 5 Fig. 5 Communities of biotechnology companies Aug-99 Emerged Dissolved Split or merged Single Oct-99 Dec-99 Feb-00 Apr-00 Jun-00 Aug-00 Oct-00 Dec-00 Feb-0 Apr-0 Jun-0 Aug-0 Oct-0 Dec-0 Feb-02 ) URL hub hub hub hub hub 6. Time 7 Fig. 7 Transition of communities t k :Aug.99 Aug.00 Oct.0 t k :Aug.00 Oct.0 Feb.02 t k 28,467 32,490 4,50 8,060 22,299 44,425 t k 29,722 4,752 44,305 3 Table 3 Number of main lines in split or merged communities 6. ( URL ) t k t k

7 Distribution of split rate Aug Aug 2000 Aug Oct 200 Oct Feb 2002 Size distribution of emerged communities Aug Aug 2000 Aug Oct 200 Oct Feb Nsp Size of emerged community Fig. 8 8 Distribution of split rate 0 Fig. 0 Size distribution of emerged communities Distribution of merge rate Aug Aug 2000 Aug Oct 200 Oct Feb 2002 Size distribution of dissolved communities Aug 999--Aug 2000 Aug Oct 200 Oct 200--Feb Nmg Size of dissolved community Fig. 9 9 Distribution of merge rate Fig. Size distribution of dissolved communities 7 25% % ( 3. 2 ) t k 40% % 8 9 URL ( 3. 2 N sp N mg)

8 Number of mainlines Growth rate distribution Fig. 2 2 Growth rate Aug Aug 2000 Aug Oct 200 Oct Feb 2002 Distribution of growth rate t k t k 0 [] K. Bharat, A. Broder, M. Henzinger, P. Kumar, and S. Venkatasubramanian. The Connectivity Server: fast access to linkage information on the Web. In Proceedings of the 7th International World Wide Web Conference, 998. [2] K. Bharat and G. A. Mihaila. When Experts Agree: Using Non-Affiliated Experts to Rank Popular Topics. In Proceedings of the 0th World-Wide Web Conference, 200. [3] A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. Wiener. Graph Structure in the Web. In Proceedings of the 9th World- Wide Web Conference, [4] S. Chakrabarti. Integrating the Document Object Model with Hyperlinks for Enhanced Topic Distillation. In Proceedings of the 0th World-Wide Web Conference, 200. [5] S. Chakrabarti, B. Dom, P. Raghavan, S. Rajagopalan, D. Gibson, and J. Kleinberg. Automatic resource compilation by analyzing hyperlink structure and associated text. In Proceedings of the 7th International World Wide Web Conference, 998. [6] J. Dean and M. R. Henzinger. Finding related pages in the World Wide Web. In Proceedings of the 8th World-Wide Web Conference, 999. [7] G. W. Flake, S. Lawrence, and C. L. Giles. Efficient Identification of Web Communities. In Proceedings of KDD 2000, [8] D. Gibson, J. Kleinberg, and P. Raghavan. Inferring Web Communities from Link Topology. In Proceedings of Hyper- Text98, 998. [9] J. M. Kleinberg. Authoritative Sources in a Hyperlinked Environment. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms, 998. [0] R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Extracting large-scale knowledge bases from the web. In Proceedings of the 25th VLDB Conference, 999. [] R. Lempel and S. Moran. The Stochastic Approach for Link-Structure Analysis (SALSA) and the TKC Effect. In Proceedings of the 9th World-Wide Web Conference, [2] S. R. Ravi Kumar, Prabhakar Raghavan and A. Tomkins. Trawling the Web for emerging cyber-communities. In Proceedings of the 8th World-Wide Web Conference, 999. [3] M. Toyoda and M. Kitsuregawa. Creating a Web Community Chart for Navigating Related Communities. In Conference Proceedings of Hypertext 200, pp. 03 2, 200. [4] Wayback Machine, The Internet Archive. archive. org/. [5],..,

Analyzing Evolution of Web Communities using a Series of Japanese Web Archives

DEWS2003 2-P-05 53 8505 4 6 E-mail: {toyoda,kitsure}@tkl.iis.u-tokyo.ac.jp 999 2002 4 Analyzing Evolution of Web Communities using a Series of Japanese Web Archives Masashi TOYODA and Masaru KITSUREGAWA