Efficient Monitoring Algorithm for Fast News Alert

Transcription

1 Efficient Monitoring Algorithm for Fast News Alert Ka Cheung Richard Sia UCLA

2 Backgroud Goal Monitor and collect information from the Web Answer most of users queries Challenges Billions of pages to monitor Information are updated frequently Users want information fresh!

3 Information aggregator framework Server-based monitoring and dissemination

4 Overview Modeling the posting generation process Definition of delay Poisson process

5 Overview Modeling the posting generation process Definition of delay Poisson process Crawl scheduling Resource allocation (how often to contact?) Retrieval scheduling (when to contact?)

6 Overview Modeling the posting generation process Definition of delay Poisson process Crawl scheduling Resource allocation (how often to contact?) Retrieval scheduling (when to contact?) The collected data 10k RSS (since September 2004) 40k Weblogs (since April 2004)

7 New challenges Higher requirement on freshness Finer time granularity (will traditional assumption be valid?)

8 Terminology t i - posting generation time τ j - time of the j th contact D(O) = k i=1 (τ j t i ), where t i [τ j 1, τ j ]

9 Posting generation model Homogeneous Poisson model λ(t) = λ at any t Periodic inhomogeneous Poisson model λ(t) = λ(t nt ), n = 1, 2,...

10 Posting generation model Homogeneous Poisson model λ(t) = λ at any t Periodic inhomogeneous Poisson model λ(t) = λ(t nt ), n = 1, 2,... Weekly number of postings (Sep 26 Jan 8) x hours posting count (Oct 3 Oct 9) Sun Mon Tue Wed Thu Fri Sat Number of postings Oct Nov Dec Number of postings in 2 hours Week Hours (since Oct 3 12:00 am)

11 Expected retrieval delay Inhomogeneous Poisson model rate - λ(t) retrieval time - τ j 1, τ j expected delay - τ j τ j 1 λ(t)(τ j t)dt

12 Expected retrieval delay Inhomogeneous Poisson model rate - λ(t) retrieval time - τ j 1, τ j expected delay - τ j τ j 1 λ(t)(τ j t)dt Homoegeneous Poisson model expected delay - λ(τ j τ j 1 ) 2 2

13 Objective Maximize resource utlilization to provide timely informaiton.

14 Objective Maximize resource utlilization to provide timely informaiton. Resource allocation How often to contact data sources?

15 Objective Maximize resource utlilization to provide timely informaiton. Resource allocation How often to contact data sources? Retrieval scheduling When to contact data sources within a day?

16 Resource allocation Consider n data source O 1,..., O n λ i - posting rate of O i w i - weight of O i (how important) N - total number of retrievals per day m i - number of retrievals per day allocated to O i

17 Resource allocation Consider n data source O 1,..., O n λ i - posting rate of O i w i - weight of O i (how important) N - total number of retrievals per day m i - number of retrievals per day allocated to O i Optimal allocation m i w i λ i

18 Retrieval scheduling m retrieval(s) per day is allocated for data source O, how should we schedule these m retrievals? m = 1 m > 1

19 Multiple retrievals per period m retrievals per period are allocated, when scheduled at time τ 1,..., τ m, the expected delay is: D(O) = m τi+1 i=1 τ i λ(t)(τ i+1 t)dt Criteria for optimality τ m+1 = T + τ 1 λ(τ j )(τ j+1 τ j ) = τ j τ j 1 λ(t)dt

20 Multiple retrievals per period Example: λ(t) = sin(2πt) τ j 1 τ j τj 1 λ(t)dt λ(t) τ j τ j λ(τ j )(τ j+1 τ j ) time t

21 Experiment 10k RSS feeds from Sep 21 - Dec Characteristics of posting generation

22 Distribution of posting rate feeds Number of RSS feeds Number of postings 9634 RSS feeds are used Power-law distribution

23 Is posting rate stable and predictable? Posting rate on week 3 4 (Oct 17 30) Remaining top 50% top 90% top 50% Posting rate on week 1 2 (Oct 3 16) Posting rate on week (Dec 12 25) Remaining top 50% top 90% top 50% Posting rate on week 1 2 (Oct 3 16) The closer to diagonal, the more the stability and predictability red - top 50%, green - top 80%, blue - rest

24 How much history to keep? Average delay (minutes) Resource allocation only: 4 retrievals per day Size of sliding window (days) Reallocate resource everyday 2 weeks is a good choice

25 What is the posting pattern? Total number of postings between Apr 1 Dec x Changes of posting rate within one day Hours Periodic (daily pattern) inactive at night

26 What are the individual pattern? feeds 1346 feeds feeds 377 feeds feeds 305 feeds K-mean clustering Optimize for different patterns

27 Performance Average delay (minutes) Even scheduling Retrieval scheduling Resource allocation Combined Number of retrievals per day per feed 1. Even scheduling 2. Retrieval scheduling only 3. Resource allocation only 4. Combined

28 Performance strategy average delay (in min) max delay (in min) standard deviation Statistics breakdown of posting delay using one retrieval per day.

29 Summary Efficient Monitoring Resource allocation Retrieval scheduling Include user access pattern (extension) Data 1 year of weblogs and half year of RSS data For prototype testing