To pose an abstract computational problem on graphs that has a huge list of applications in web technologies

Transcription

1 Talk Objectie Large Scale Graph Algorithms A Gide to Web Research: Lectre 2 Yry Lifshits Steklo Institte of Mathematics at St.Petersbrg To pose an abstract comptational problem on graphs that has a hge list of applications in web technologies Stttgart, Spring / 34 2 / 34 Otline 1 Family of Problems: Finding Strongest Connection Problem Statement and Applications Variations of Strongest Connection Problem 2 Max-Intersection Problem Statement and Naie Soltions Hierarchical Schema Soltion 3 Conclding Remarks Oeriew of Related Research Open Problems Part I Family of Problems: Finding Strongest Connection Problem statement Applications Variations of the problem 3 / 34 4 / 34

2 Strongest Connection Problem (SCP) Homogeneos Graph / 2-Step Paths BASIC SETTINGS: a class of graphs G, a class of paths P INPUT: a graph G G Allowed time for preprocessing: o( G 2 ) Graph of coathoring QUERY: a (new) ertex TASK: to find a ertex G that has maximal nmber of P-paths from to Allowed time for qery processing: o( G ) Coathor sggest in DBLP The most common coathor of my coathors 5 / 34 6 / 34 Directed Graph / 2-Step Paths Bipartite Graph / 2-Step Paths People Graph of hyperlinks Bands Adanced option for Google search: link-based similar website The website that is most often co-cited with the gien one Last.fm similar msic bands The band that is most often co-listened with the gien one In general: any content-based similarity, keyword-similarity, any co-occrrence similarity 7 / 34 8 / 34

3 Homogeneos-Bipartite Graph / 2-Step Paths Bipartite Graph / 3-Step Paths Friendship graph New girlfriend sggest: Boys Recommended items Girls Social recommendations in networks like Facebook System recommends things that are poplar among my friends Amazon.com recommendations Sbscription recommendations for FeedBrner, Google Reader Items that hae the largest nmber of co-occrrences with my items 9 / / 34 Tripartite 3-Graph / 2-Step Paths Folksonomy is a set of triples < ser, tag, object > Users Tags Mlticolor-Mltiparty Graph / k-step Paths Semantic search: Most poplar drink that is aailable on bars that are isited by my friends Friendship graph Objects Bar isiting Similar websites in Del.icio.s, similar pictres in Flicr Largest nmber of common tags Largest nmber of common sers Largest nmber of common pairs < ser, tag > Drinks in men 11 / / 34

4 Variations of Strongest Connection Problem Claim Directed/ndirected graphs Weights on edges/ertices Task: offline, on-line, all-to-all Task: one best connection, k best connections Graph and weights are eoling with time Compting strongest connection is probably the most important algorithmic problem related to web technologies Personal opinion of Yry Lifshits 13 / / 34 Soltion Variations Usal alternaties to exact algorithm: Approximate algorithms Randomized algorithms Inpt graph (or qery) belongs to a certain distribtion. Aerage complexity analysis Introdcing additional assmptions Introdcing additional inpt-complexity parameter Modifying the comptation task Heristics Look to particlar cases (sbproblems) Part II Max-Intersection Problem Statement and naie soltions Hierarchical schema soltion This section represents a work-in-progress joint research with Benjamin Hoffmann and Dirk Nowotka 15 / / 34

5 Statement of Max-Intersection Problem In set notation: Inpt: Family F of n sets, f F f k Time for preprocessing: n polylog(n) poly(k) Qery: a set f new, f new k Task: Find f i F that maximizes f new f i Time for qery processing: polylog(n) poly(k) or at most o(n) In bipartite graph notation: Inpt: Bipartite docments-terms graph, D = n, d D d k Qery: a docment d new, d new k Task: Find d i D that has maximal nmber of common terms with d new 17 / 34 Applications of Max-Intersection (1/2) Homogeneos graphs: References in scientific papers: (1) maximal nmber of co-occrrences in reference list (2) maximal intersection of reference lists Social networks (e.g. LinkedIn): a person that has maximal connections with my direct neighborhood Collaboration networks (e.g. DBLP): gien a scientist, to find another one with maximal oerlapping of coathors-list Bipartite graphs: Websites Words graph: find a website with maximal intersection of sed terms with the gien one Msic Bands Listeners graph: find a band that has maximal intersection of listeners with the gien one 18 / 34 Applications of Max-Intersection (2/2) Inerted Index (1/2) Let s se docments-terms notation Tripartite graphs: Long Search Qeries Web Dictionary Websites: gien a qery to find a website with maximal nmber of qery terms Adertisement Description Keywords Websites (e.g. AdSense Matching): find a website with maximal nmber of terms form adertisement description PC Members Keywords Sbmissions: find a paper that has maximal nmber of terms that belong to expertise of the gien PC member Inerted index approach: Preprocessing. For eery term prodce a list of all docments that contain it Complexity: O(n k) Qery d new = {t 1,..., t k }. Retriee docment lists for all terms of qery. Check all docments in all these k lists and retrn the one with maximal intersection with d new Worst case complexity: Ω(n) Let T max be the maximal degree of terms. Then the qery complexity is O(k T max ) 19 / / 34

6 Inerted Index (2/2) Rare-Term Reqirement Inerted Set-Index Cheating: modify the Max-Intersection problem New Task: Gien the docment d new, find a docment d i sch that 1 It has a joint rare term (term that occrs in at most r docments) with d new 2 The intersection with d new is maximal among all docments satisfying (1) Obseration Inerted index can handle qeries in O(r k) time now Assme that k is extremely small, say k = O(log log n) Inerted set-index approach: Preprocessing. Write down all term sbsets of all docments. Sort all these sbsets in lexicographical order Complexity: O(n 2 k ) Qery d new = {t 1,..., t k }. For eery sbset of qery terms search it in the inerted set-index. Retrn the docment that corresponds to the maximal sbset fonded in index Complexity: O(2 k (k + log n)) 21 / / 34 Hierarchical Schema Magic Leels (1/2) Table of terms: k leels Leel i is diided to 2 i 1 cells Eery cell contains k terms Random natre of D and d new : Choose random cell on the bottom leel Mark all cells that are aboe it Choose one random term in eery marked cell Assme that there are 2 k sch random docments in D Notation: magic leels q = k, log k+1 q = k log k Theorem With ery high probability there exists d D that has the same terms from top q ε leels 23 / / 34

7 Magic Leels (2/2) Algorithm for Hierarchical Schema Theorem With ery high probability there are no d D that has at least q + ε common elements with d new Preprocessing: Encode eery docment as a 2k 1 seqence, eery odd element lies in range [1..k], eery een is 0 or 1 Constrct a lexicographic tree for all encodings Qery processing: Find the largest prefix-match between d new and docments from D By two theorems aboe with ery high probability maximal prefix-match is ery close to maximal intersection 25 / / 34 Oeriew of Related Research Part III Conclding Remarks Oeriew of related research Open problems Famos comptational problems that need scalable algorithms: Nearest neighbors in ector spaces Nearest neighbors in abstract metric spaces Connection sbgraph problem Collaboratie filtering Mining association rles Indexing with errors Common approach: heristical algorithm + experimental alidation Alternatie: randomized model of inpt + probabilistic analysis Alternatie: realistic assmption abot inpt + exact algorithm 27 / / 34

8 Algorithms for Max-Intersection Data Strctre Complexity Algorithmic open problems: 1 Max-Intersection for bonded tree-width graphs 2 Max-Intersection in configration model 3 Max-Intersection in preferential attachment model Conceptal open problem: 1 Find simple-bt-realistic assmptions allowing reqired exact soltion of Max-Intersection Long-term goal: to deelop theoretical framework for scalability analysis of algorithms On-line inclsion problem Inpt: Family F of 2 k sbsets of [1..k 2 ] Data storage after preprocessing: 2 k poly(k) Qery: a set f new [1..k 2 ] Task: decide whether f F : Time for qery processing: poly(k) f new f Conjectre: the on-line inclsion problem can not be soled within sch time/space constraints 29 / / 34 Call for participation Know a releant reference? Hae an idea? Highlights Strongest Connection family, inclding Max-Intersection Find a mistake? Soled one of these problems? Knock to my office Write to me yra@logic.pdmi.ras.r Join or informal discssions Participate in writing a follow-p paper Open problems: Max-Intersection in complex-networks models Data strctre complexity of on-line inclsion problem Vielen Dank für Ihre Afmerksamkeit! Fragen? 31 / / 34

9 References (1/2) References (2/2) Corse homepage Y. Lifshits Web research: open problems J. Zobel and A. Moffat Inerted files for text search engines C. Falotsos, K.S. McCrley, A. Tomkins Fast discoery of connection sbgraphs M.E.J. Newman The strctre and fnction of complex networks P.N. Yianilos Data strctres and algorithms for nearest neighbor search in general metric spaces J. Kleinberg Two algorithms for nearest-neighbor search in high dimensions R. Agrawal and R. Srikant Fast algorithms for mining association rles in large databases M. O Connors J. Herlocker Clstering items for collaboratie filtering m.pdf 33 / / 34