Comparative Document Analysis for Large Text Corpora Xiang Ren Yuanhua Lv Kuansan Wang Jiawei Han University of Illinois at Urbana-Champaign, Urbana, IL, USA Microsoft Research, Redmond, WA, USA {xren7, hanj}@illinois.edu {yuanhual, Kuansan.Wang}@microsoft.com arxiv:1517197v1 [cs.ir] 25 Oct 215 ABSTRACT This paper presents a novel research problem on joint discovery of commonalities and differences between two individual documents (or document sets), called Comparative Document Analysis (CDA). Given any pair of documents from a document collection, CDA aims to automatically identify sets of quality s to summarize the commonalities of both documents and highlight the distinctions of each with respect to the other informatively and concisely. Our solution uses a general graph-based framework to derive novel measures on semantic commonality and pairwise distinction, and guides the selection of sets of s by solving two joint optimization problems. We develop an iterative algorithm to integrate the maximization of commonality or distinction measure with the learning of document semantic relevance in a mutually enhancing way. Experiments on text corpora from two different domains scientific publications and news demonstrate the effectiveness and robustness of the proposed method on comparing individual documents. Our case study on comparing news articles published at different dates shows the power of the proposed method on comparing document sets. 1. INTRODUCTION Comparative text mining is concerned with identifying common and different information to distinguish text items in corpora. Instead of measuring the similarity between items or predicting sentiment rating on an item, it delves deeper into the text and focuses on extracting content units of different granularity (e.g., words, sentences) to summarize the commonalities and differences. With the rapid emergence of text-based data in many domains, automatic techniques for comparative text mining have a wide range of applications including social media analysis (e.g., opinion mining on twitter [27, 18]), business intelligence (e.g., customer review analysis [14, 12], news summarization [1, 25, 16]), and scientific literature study (e.g., patentability search [31]). Work done when author was an intern at MSR. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 2XX ACM X-XXXXX-XX-X/XX/XX $15.. [11] Scaling Personalized Commonalities Web Search PageRank vector; importance scores; PageRank algorithm; query results; web graph@@web pages; Distinctions user-specified@@web pages; personalized web search; dynamic programming; incremental computation; [6] Topic-sensitive PageRank Distinctions topic-sensitive@@ranking vector; context-specific importance score; topic-sensitive PageRank; query context; Figure 1: Example output of CDA for papers [11] and [6]. While there has been some research in comparative text mining, most of these focus on generating word-based or sentence-based summarization for sets of documents. Wordbased summarization [2, 3] suffers from limited readability as single words are usually non-informative and bag-ofwords representation does not capture the semantics of the original document well it may not be easy for users to interpret the combined meaning of the words. Sentence-based summarization [1, 29, 15], on the other hand, may be too verbose to accurately highlight the general commonalities and differences users may be distracted by the irrelevant information that it contains (see Table 4). Furthermore, previous work compares two sets of documents based on the data redundancy (e.g., word overlap) between them but the task becomes much more challenging when comparing two individual documents, since there may not exist sufficient amount of common content units between them. This paper studies a novel comparative text mining problem which leverages multi-word s (i.e., minimal semantic units) to represent the common and distinct information between two individual documents (or two sets of documents). We refer to the task as Comparative Document Analysis (CDA): Given a pair of documents from a document collection, the task is to (1) extract from each document salient s and pairs which cover its major content; (2) discover the commonalities between the document pair by selecting salient s which are representative to both of them; and (3) find the distinctions for each document by selecting salient s that are exclusively relevant to the document. CDA can benefit a variety of applications including related item recommendation and document retrieval. For example, as shown in Fig. 1, a citation recommendation system can show users the common and distinct concepts produced by CDA to help them understand the connections and differences between a query paper [11] and a recommended paper [6]. In a similar way,
Doc A Scaling Personalized Web Search Doc B Topic- Sensitive PageRank Salient generation personalize web search; user-specified@@web pages; importance score; dynamic programming; pagerank vector; pagerank vector; topic-sensitive@@ranking vectors; importance score; topic-sensitive pagerank; query topic; query context; Document collection Graph construction dynamic programming user study search engine topic-sensitive pagerank Doc A Doc B Documents to compare Salient s Document- Bipartite Graph salient document Figure 2: Overview framework of PhraseCom. Graph-based optimization for common & distinct selection personalized web search; dynamic programming; Distinction of Doc A Commonality pagerank vector; importance score; pagerank algorithm; topic-sensitive pagerank; query context; query topic; Distinction of Doc B CDA can reduce the efforts on patentbility searching [31]. To give another example, in product recommendation scenario, CDA can address a user s doubts like why Amazon recommends Canon 6D camera to me after I viewed Canon 5D3 camera by showing common aspects of the two cameras (e.g., DSLR camera, full-frame sensor ) and the distinct aspects about Canon 6D (e.g., WiFi function ). By analyzing the common/distinct aspects and user s responses, a relevance feedback system can have a better understanding of the user s purchase intent and helps recommend other cameras with similar functions. However, existing comparative summarization techniques encounter several unique challenges when solving CDA. Semantic Commonality: Commonalities between two documents can intuitively be bridged through s that do not explicitly occur in both documents, e.g., web graph @@web pages in Fig. 6. Previous methods consider only content units (e.g., words) explicitly occurring in both documents as indication of commonalities but ignore such semantic commonality (see our study in Fig. 3). Pairwise Distinction: Distinct s should be extracted based on the pair of compared documents jointly it should be exclusively relevant to its own document (see Fig. 5). Current methods select discriminative content units for each document set independently. The results so generated thus may not distinguish the two documents effectively. Data Sparsity: Most existing work relies on word overlap or sentence repetition between the items in comparison (i.e., document groups) to discover their commonalities and each item s distinctions. However, such sources of evidences may be absent when comparing two individual documents 1. We address these challenges with several intuitive ideas. First, to discover semantic commonalities between two documents, we consider s which are semantically relevant to both documents as semantic common s even they do not occur in both documents (i.e., (b) in Fig. 5). Second, to select pairwise distinct s, we use a novel measure that favors s relevant to one document but irrelevant to the other (i.e., (e) in Fig. 5). Third, to resolve data sparsity, we exploit -document co-occurrence statistics in the entire corpus to go beyond the simple inter-document content overlap to model their semantic relevance. To systematically integrate these ideas, we propose a novel graph-based framework called PhraseCom (Fig. 2) to unify the formalization of optimization problems on selecting common s and distinct s. It first segments the cor- 1 One may argue that we can expand each individual document into a pseudo document collection by retrieving its topic-related (item-related) documents. However, such a solution is expensive. pus to extract candidate s where salient s for each document are selected based on interestingness and diversity (Sec. 3.1). We then model the semantic relevance between s and documents using graph-based propagation over the co-occurrence graphs (Fig. 6) to measure commonality and distinction (Sec. 3.2). We formulate two joint optimization problems using the proposed measures to select sets of common s and distinct s for a document pair, and present an alternative minimization algorithm to efficiently solve them (Sec. 3.3-3.4). The algorithm tries to integrate the optimizing of the proposed commonality or distinction measure with learning of the semantic relevance scores, and can be flexibly extended to compare two sets of document (Sec. 3.4). The major contributions of this paper are summarized as follows. 1. We define and study a novel comparative text mining task, comparative document analysis, which uses sets of s to jointly represent the commonality and distinctions between a pair (two sets) of documents. 2. We propose a graph-based framework, PhraseCom, to model and measure the semantic commonality and pairwise distinction for s in the documents. 3. We formulate joint optimization problems to integrate the maximization of commonality or distinction with the learning of -document semantic relevance, and develop an efficient algorithm for solving them. 4. Experiments on datasets from different domains news and academic papers demonstrate that the proposed method achieves significant improvement over the stateof-the-art (e.g., 56% enhancement in F1 on the Academia dataset over the best competitor from existing work). 2. PROBLEM DEFINITION Literally, a comparison identifies the commonalities or differences among two or more objects. It consists of three components [29]: the compared objects, the scale (i.e., on what aspects the objects are measured and compared), and the result. In our task, compared objects are two given documents; scale is multi-word or pair; and the result is represented by three sets of s, i.e., a set of common s and two sets of distinct s for the two documents, respectively. The input to our proposed comparative document analysis framework is a document collection D = {d 1, d 2,..., d n}, a set of document pairs for comparison U = {(d, d )} d,d D, and a set of positive example s P + = {p + i,..., p+ n +} extracted from knowledge bases for candidate generation. In this work, we generate P + by using 5 Wikipedia article titles (see Sec. 3.1 for details).
No common 8% Contain semantic common s 48% Exact match 44% Academia Dataset No common 17% Contain semantic common s 4% News Dataset Exact match 43% Figure 3: Study on semantic commonality: we show %document pairs that contain semantic common s (i.e., which does not occur in both document but represents the commonality) on the two evaluation sets. 62.37% of the gold standard s are semantic common s. A, p, is a single-word or multi-word sequence in the text document which represents a cohesive content unit (e.g., a noun like automatic text summarization, or a verb like earthquake struck ). We further consider combining two s p a and p b into a pair, i.e., p a p b, if they tend to co-occur with each other frequently in the document 2 (e.g., web graph@@web pages in Fig. 1). Let P = {p 1,..., p m} denote m unique s extracted from the corpus D. For a pair of documents (d, d ) U in comparison, we denote the two sets of salient s extracted from them as S and S, respectively. We use a binary vector y c {, 1} m to indicate whether salient s from the two documents (i.e., S S ) are selected to form the set of common s C S S. Another two binary vectors, y, y {, 1} m, are used to indicate whether salient s from S and S are selected as the set of distinct s, i.e., Q S and Q S, for documents d and d, respectively. Obviously, C Q = C Q =. By estimating {y c, y, y }, one can generate the three sets of s for the task, i.e., C = {p p S S, yp c = 1}, Q = {p p S, yp = 1}, and Q = {p p S, y p = 1}. Formally, we define the problem of comparative document analysis (CDA) as follows. Definition 1 (Problem Definition). Given a document collection D, a set of document pairs U and a set of positive example s P +, the task CDA aims to: (1) extract salient s S for each document d D; and (2) for each pair of comparing documents (d, d ) U, estimate the indicator vectors for the common and distinct s {y c, y, y } to predict the comparison result sets {C, Q, Q }. In our study, we assume the given document pairs in U should be comparable, i.e., they share some common aspects or belong to the same concept category. For example, they can be two computer science papers or two news articles on similar topics. Such document pairs can come from document retrieval and item recommendation results. It is not the focus of this paper to generate such document pairs. 3. COMPARATIVE DOCUMENT ANALYSIS Our overall framework (shown in Fig. 2) is as follows: 1. Perform distantly-supervised segmentation on D and generate salient concepts S for d D by optimizing both interestingness and diversity. (Sec. 3.1). 2. Construct a -document co-occurrence graph to help model semantic relevance as well as to assist measuring commonality and distinction (Sec. 3.2). 3. Estimate indicator vectors {y c, y, y } for the three sets {C, Q, Q } by solving the two proposed optimization problems with the efficient algorithms. (Sec. 3.3-3.4). 2 Without explicitly mentioning, we refer both and pair as in the rest of the paper. /, /we /propose /computing /a set of /PageRank vectors/,/biased /using /a set of /representative topics/, /to /capture /the /notion /of /importance /with respect to /a /particular /topic /. /For/ordinary /keyword search /queries /, /we/ compute /the /topicsensitive PageRank /scores /for /pages /using /the /topic /of /the /query keywords/ Aggregate statistics a set of; topic; topics; queries; scores; pages; PageRank; query keywords; PageRank vectors; representative topics; topicsensitive PageRank Optimize interestingness and diversity topic-sensitive PageRank; PageRank vectors; query keywords; Segmented Document Candidate Phrases Salient Phrases Figure 4: Output for [6] in salient generation. Each step will be elaborated in the following sections. 3.1 Salient Phrase Generation To ensure the generation of cohesive, informative, salient s for each document, we introduce a scalable, datadriven approach by incorporating both local syntactic signals and corpus-level statistics. Our method first uses a distantly-supervised segmentation algorithm [17] to partition the text into non-overlapping segments for candidate generation. Then it adopts both interestingness and diversity measures in a joint optimization framework [7] to guide the filtering of low importance s and removing s with redundant meaning. Given a word sequence, the result of segmentation is a sequence of s each representing a cohesive content unit (see Fig. 4). First, we mine frequent contiguous patterns {v} up to a fixed length. A random forest classifier is then used to estimate the quality Q(v) based on multiple features and the distant supervision P + from Wikiepdia. With the estimated Q(v), we derive the optimal segmentation of each sentence using Viterbi Training [17]. The above two steps can further iterate between each other to improve the results [17]. To enhance readability, we further combine s with good concordance in the document (i.e., p a and p b co-occur more than 3 times in a window of 1 words in d) into pairs p a p b. The results from segmentation represents each document as a bag of s, but the majority are not representative for the document. To select a set of salient s for a document, we consider two different aspects to measure the salience, i.e., interestingness and diversity. The intuition behind interestingness is simple [1]: a is more interesting to the document if it appears frequently in the current document while relatively infrequently in the entire collection. Let P d denote the set of s from the segmented document d (including pairs), n(p, d) denote the frequency of p in d, and n(p, D) denote the document frequency of p in D. The interestingness measure r( ) of p in d D is defined as follows [1]. r D(p, d) = (.5 n(p, d) ) 2 ( D ).5 + log, (1) max t Pd n(t, d) n(p, D) which is the product of the square of normalized term frequency and the inverse document frequency. In particular, interestingness score of pair p a p b is computed using an intra-document point-wise mutual information and is discount by its document frequency as follows. r D(p a p b, d) = n(p a p b,d) P d n(p a,d) n(p b,d) P d P d ( log D n(p a p b, D) ). (2) Interestingness scores for both s and pairs from a document are normalized into (, 1) for comparison.
document salient A (a) B A Commonality B p A B A B f' f Distinction (b) (c) (d) (e) p p' p Figure 5: An illustration of the proposed measures. (a) Intersection model; (b) Semantic commonality model (ours): is relevant to both documents; (c) Independent distinction model; (d) Joint distinction model: similar s are excluded; (e) Pairwise distinction model (ours): is relevant to one document while irrelevant to the other. To impose good diversity on the set of selected s, we enforce them to be different from each other. We adopt edit distance-based Levenshtein similarity to measure the string similarity M(p a, p b ) between two s p a and p b. One can also apply semantic similarity measures like distributional similarity [5]. To select a subset S P d of K salient s for document d, we solve an optimization problem [7] which maximizes interestingness and diversity jointly as follows. argmax S =K H(S) = µ p' p a P d q ar a p A f p p' B f' p a,p b P d r am ab r b. (3) Here, r a = r D (p a, d) is the interestingness score, M ab = M(p a, p b ) is similarity score, and q a = M ar is the importance score of p a. A near-optimal solution of Eq. (3) can be obtained by an efficient algorithm with time cost O( P d 2 + P d K) [7]. Fig. 4 provides an illustration of the salient generation process with examples. 3.2 Commonality and Distinction Measures The salient generation step provides a set of quality salient s S for each document d D. A straightforward solution to generate {C, Q, Q } for CDA is to use s which are salient s for both documents to represent commonalities between the two documents (i.e., C = S S ) and to select the most salient s from the remaining ones to highlight the distinction (i.e., Q S \C, Q S \C) [2]. However, such a solution have several major limitations. First, the common s so obtained ignore semantic common s, i.e., s which do not occur in the salient sets of both documents but can also represent the commonalities (e.g., web graph@@web pages in Fig. 6). Ignoring such semantic common s leads to low recall (see Fig. 3). Second, the simple selection of distinct s by excluding the common s only considers its relevance to the current document while ignoring its relationship to the other document. On one hand, this may not guarantee the pairwise distinction property a salient in one document may be also relevant to the other document. On the other hand, this often misses good distinct s which, although moderately salient in the current document, are completely irrelevant to the other document. Third, the distinct s so generated may include overly specific s which are salient in the current document but irrelevant to the other one (e.g., partial vectors in Fig. 6). Recent work [31] considers pairwise distinction property by casting the problem as feature selection but it appears to produce many overly specific s. To systematically resolve these issues, we derive semantic relevance between s and documents based on their corpus-level co-occurrences statistics (Sec. 3.3), and formalize novel objectives to measure semantic commonality and pairwise distinction for s. The intuitive ideas are simple: (1) a good to represent the commonality between two documents should be semantically relevant to both documents; (2) a good to highlight the distinction of a document should be relevant to this document but irrelevant to the other document; and (3) a good should be well accepted in the corpus (i.e., with reasonable popularity). Specifically, let function f(p, d) : P D R + denote the non-negative relevance score between p P and d D, which will be elaborated in Sec. 3.3. We define the commonality score function Φ(p, d, d ) : P D D R + to measure how well p can represent the commonality between the document pair (d, d ). The following hypothesis guides our modeling of commonality score. Hypothesis 1 (Phrase Commonality). Given (d, d ), p tends to have high commonality score Φ(p, d, d ) if and only if the relevance scores between the and both documents, i.e., f(p, d) and f(p, d ), are high. In Fig. 6, for example, web graph@@web pages has high commonality score since it has high relevance score to Doc B (it occurs frequently in Doc B) and high relevance score to Doc A (it occurs frequently in Doc X and Doc X contains several that are relevant to Doc A). Formally, we define the commonality score function as follows. ( ) Φ(p, d, d ) = ln 1 + f(p, d) f(p, d ). (4) Similar to the product-of-experts model [8], it models the commonality score as the product of the two relevance scores, likes an and operation. An alternative definition for Φ( ) is the summation of two relevance scores, i.e., Φ(p, d, d ) = f(p, d) + f(p, d ). However, as an or operation, the score so modeled may be dominated by the larger relevance score among the two. For instance, the case f(p, d) = f(p, d ) =.5 and the case f(p, d) =.1, f(p, d ) =.9 share the same commonality score, but the former represents better commonality. We compare these two models in our experiments. A good for highlighting the distinction of d between (d, d ) should not only help distinguish d from d but also have good readability, i.e., not overly specific. For example, in Fig. 6, dynamic programming serves as a good distinct for Doc A it has strong association with Doc A and weak association with Doc B; meanwhile, it is fairly popular in the corpus. On the other hand, partial vector has similar association pattern with Doc A and Doc B but it is rarely mentioned in the corpus, i.e., overly specific. We use a distinction score function Π(p, d, d ) : P D D R to measure how well p can highlight the distinction of d from d, based on the following hypothesis. Hypothesis 2 (Phrase Distinction). Given (d, d ), p tends to have high distinction score Π(p, d d ) if it has relatively higher relevance score f(p, d) to document d compared with its relevance score f(p, d ) to document d. Specifically, we define the distinction score function Π(p, d d ) based on the division of relevance to d by the relevance to d, which has the following form. Π(p, d d ) = ln ( f(p, d) + γ f(p, d ) + γ ). (5)
dynamic programming partial vector web graph@@web pages topic-sensitive pagerank.9.9.7.1 5 1. Doc A Doc X Doc B Figure 6: An illustration of relevance scores to Doc A derived from the constructed bipartite graph. Our method can infer the relevance between web graph@@web pages and Doc A (i.e.,.7) even it does not occur in Doc A. Here, a smoothing parameter γ = 1 is used to avoid selecting p with too small f(p, d) or f(p, d ). In particular, the relevance score f(p, d) incorporates the popularity of p in the collection (see Sec. 3.3) and thus can filter overly specific s. A will receive high distinct score in two cases: (1) p has high relevance score to d and moderate or low relevance score to d ; and (2) p has moderate relevance score to d and low relevance score to d. The second case helps include more s to represent the distinctions even they are moderately relevant to its own document. An alternative way to define the distinction score is by the difference between two relevance scores, i.e., Π(p, d d ) = f(p, d) f(p, d ). Such score functions prefer the first case than the second one, and thus will suffer from low recall. We compare with this alternative measure in the experiments. 3.3 Comparative Selection Optimization With the proposed commonality and distinction measures, we now are concerned of the following two questions: (1) how to derive the relevance score f(p, d) between and document; and (2) how to select the sets of s {C, Q, Q }. Graph-Based Semantic Relevance. A simple idea to compute f(p, d) is to use bag-of-words similarity measures such as cosine similarity. However, the score so generated may not capture the semantic relatedness between them (see String- Fuzzy in Sec. 4). Our solution leverages graph-based semisupervised learning [32, 33] to model the semantic relevance between s and documents, and further integrates the relevance score learning with selection. It treats the target document d D as positive label and tries to rank the s P and the remaining documents D \ d based on the intrinsic manifold structure among them. By exploiting the aggregated co-occurrences between s and their supporting documents (i.e., documents where the occurs) across the corpus D, we weight the importance of different s for a document, and use their connected edge as bridges to propagate the relevance scores between s and documents. The following hypothesis guides the modeling of relevance score f(p, d). Hypothesis 3 (Document- Relevance). If a often appears in documents that are relevant to the target document, then the tends to have high relevance score to the target document; If a document contains many s that are relevant to the target document, the document is likely relevant to the target document. In Fig. 6, for example, if we know dynamic programming and partial vector have high relevance scores regarding the target document Doc A, and find that the two s have strong association with Doc X, then Doc X is likely relevant to Doc A. This reinforces the relevance score propagation that web graph@@web page is also relevant to Doc A. Specifically, we construct a bipartite graph G to capture the co-occurrences between all the s P and documents D. A bi-adjacency matrix W R +m n is used to represent the edge weights for the links where W ij is the bag-of-words ranking score BM25 [21] (k 1 = 1.2, b =.75) between p i P and d j D if p i occurs in d j, and otherwise zero. We further normalize W by the popularity of the s and documents to reduce the impact of overly popular s [33]. S = D (P) 1 2 W D (D) 1 2, (6) where we define the diagonal matrices D (P) R m m as D (P) ii = n j=1 Wij and D(D) R n n as D (D) jj = m Wij. i=1 Formally, we use function g(d a, d b ) : D D R + to denote the relevance score between two documents d a, d b D, and define vector f R +m as f i = f(p i, d) and vector g R +n as g j = g(d j, d). Following Hypothesis 3, we use target document d as positive label and model the relevance score propagation between s and documents using a graph regularization term and a supervision term [32]. d,α(f, g) = m i=1 j=1 n ( W ij f i D (P) ii g ) j 2 + α g g 2 2. D (D) jj (7) Here, we define the indicator vector g {, 1} n to impose the positive label of d in the second term, where gd = 1 and otherwise. A tuning parameter α > is used to control the strength of supervision from d on the score propagation. The Joint Optimization Problems. Relevance score f(p, d) can be directly learned by minimizing d (f, g) but the score so computed is document independent it considers only information from d while ignoring that from the other d in the compared pair (d, d ). To address this problem, we propose two joint optimization problems for selecting common s and distinct s, respectively, by incorporating information from both documents. The intuition is simple: s which are likely common (distinct) s can reinforce the propagation between relevance scores by serving as extra positive labels (i.e., besides the original document d). In our experiments, we compare with the independent optimization method (i.e., CDA-TwoStep). Formally, to discover the common s C between a document pair (d, d ) U, we propose the common selection problem which unifies two different tasks: (i) selection of C S S to maximize the overall commonality score; and (ii) minimization of the graph-based objectives to learn relevance scores. Let f R +m denote -document relevance scores f(p i, d ) for p i P, and g R +n denote document-document relevance scores g(d j, d ) for d j D, we formulate the problem as follows. min O y c,f,f,g,g α,λ = λ s.t. m yi c Φ(p i, d, d ) (8) i=1 + 1 2 d,α (f, g) + 1,α(f, g ) 2 d yi c Φ(p i, d, d ) yi c Φ(p j, d, d )/ S, p i P; y c i Φ(p i, d, d ) y c i p j S y c y S S 2 y S S 2. p j S Φ(p j, d, d )/ S, p i P; The first term in objective O aggregates the commonality scores for the selected s, and the tuning parameter
λ > controls the trade-off between it and the second and third terms which model the relevance score propagation on graph. We add the first and second constraints to automatically decide the size of C, by enforcing that the selected s should have higher commonality score than the average commonality scores computed over salient s S and S, respectively. The last constraint enforces C S S, where y S S i = 1 if p i S S, and zero otherwise. To jointly select the distinct s {Q, Q } for pair (d, d ) U, we propose the distinct selection problem. It aims to: (i) select s Q S and Q S to maximize the overall distinction scores; and (ii) derive relevance scores by minimizing the graph-based objectives. Specifically, we formulate the problem as follows. min F y,y,f,f,g,g α,λ = λ s.t. m i=1 [ ] y i Π(p i, d d ) + y i Π(p i, d d) + 1 2 d,α (f, g) + 1,α(f, g ) (9) 2 d y i Π(p i, d d ) y i Π(p j, d d )/ S, p i P; y i Π(p i, d d) y i p j S p j S Φ(p j, d d)/ S, p i P; y y S 2 y S 2, y y S 2 y S 2 y y c 2 = y 2 + y c 2, y y c 2 = y 2 + y c 2. The first term represents the aggregated distinction score over the two sets of selected distinct s, i.e., Q and Q, respectively. Similarly, the first two constraints help control the output size and the third constraint enforces Q S, Q S. In particular, the last constraint excludes the selected common s C from the results Q, Q. 3.4 An Efficient Algorithm The optimization problems in Eq. (8) and Eq. (9) are mixinteger programming and thus are NP-hard. We propose an approximate solution for each problem based on the alternative minimization framework [28]: first estimate the relevance scores {f, f, g, g } through minimizing O while fixing y c (or {y, y }); then fix {f, f, g, g } and optimize O with respect to y c (or {y, y }) by imposing the constraints; and iterate between these two steps until reaching the convergence of the objective function O. For common selection problem in Eq. (8), to estimate the relevance score vectors {f, f, g, g }, we take derivative on O with respect to each of the variables in{f, f, g, g } while fixing y c and other variables. By setting the derivatives to zero, we have the following update rules. S i g f i = 2 1 2f i + ( S ig 2 + 1 2f ) 2 + λyi c, if f i > ; i (1) S i g, if f i =. g j = (S T j f + α g j )/(1 + α) (11) { Si f i = g 1 + ( S ig + 1 ) 2 2f i 2 2f 2 + λy c i i, if f i > ; S i g, if f i =. (12) g j = (ST j f + α g j )/(1 + α), (13) Here S i and S j denote i-th row and j-th column of matrix S, respectively. With the updated relevance score vectors {f, f, g, g }, we can compute the commonality score Φ(p i, d, d ) for p i P and estimate the common indicator vector y c as follows. { yi c 1, if pi P satisfies contraints in Eq. (8); = (14), otherwise. Algorithm 1: PhraseCom Input: document pair (d, d ), salient s {S, S }, bi-adjacency matrix S, tuning parameters {α, λ} Output: common s C, distinct s {Q, Q } 1 Initialize: {g, g } by {d, d }; {y c, f, f, g, g } as zero vectors 2 g g +.1; g g +.1 3 repeat 4 repeat 5 Update {f, g} using Eq. (1) and Eq. (11) 6 Update {f, g } using Eq. (12) and Eq. (13) 7 until Eq. (7) converges 8 Update y c following Eq. (14) 9 until the objective O in Eq. (8) converges 1 return C based on the estimated y c 11 Initialize: {y, y } and {f, f, g, g } as zero vectors 12 repeat 13 repeat 14 Update {f, g} using Eq. (15) and Eq. (11) 15 Update {f, g } using Eq. (16) and Eq. (13) 16 until Eq. (7) converges 17 Update {y, y } based on C and Eq. (17) 18 until the objective F in Eq. (9) converges 19 return {Q, Q } based on the estimated {y, y } To estimate the relevance score vectors for the distinct selection problem in Eq (9), we first optimize F with respect to {f, f, g, g } while fixing {y, y }. The update rule can be derived by taking derivative of F with respect to each variable in {f, f, g, g } while fixing the other variables and setting the derivative to zero. f i = γ Sig 2 f i = γ Sig 2 ( γ + S ig ) 2 + + λ(y i y i ), (15) 2 ( γ + S ig + ) 2 + λ(y i yi), (16) 2 In particular, update rules for {g, g } are the same as Eq. (11) and Eq. (13). To ensure the existence of a solution and the non-negativity for {f, f }, the tuning parameters need to satisfy the condition λ min ( γ 2 /4, αγδ/(α + 1) ). Here, a small positive value δ =.1 is added to {g, g } to help derive stable solution. Similar to Eq. (14), the two indicator vectors {y, y } can be estimated by first computing distinction scores Π(p i, d d ) and Π(p i, d d) based on the estimated {f, f, g, g } and then enforcing constraints in Eq. (9). y i, y i = 1, if p i P satisfies the corresponding contraints in Eq. (9);, otherwise. (17) Algorithm 1 summarizes our algorithm PhraseCom. For convergence analysis, PhraseCom applies block coordinate descent on problems in Eq. (8) and Eq. (9). The proof procedure in [28] (not included for lack of space) can be adopted to prove convergence for PhraseCom (to the local minima). Extension to compare two sets of documents. PhraseCom can be easily extended to perform the comparison between two sets of documents, i.e., (D a, D b ). Let S a = d Da S d and S b = d Db S d denote the salient s extracted from the two sets of documents, respectively. Our method can directly replaces {S, S } by {S a, S b }, initializes the vectors {g, g } based on {D a, D b }, and derive the comparison results {C, Q a, Q b } following Algorithm 1. Computational Complexity Analysis. Given a corpus D with n documents and N D words, the time cost for salient
Data sets Academia News #Documents 25,484 67,89 #Candidate s 611,538 224,468 #Salient s 316,194 119,6 #Salient pairs 24,744 11,631 #Unique words 1.98M 246,66 #Links 153.19M 2.85M #Salient s per doc 18.95 17.43 #Salient pairs per doc 2.42 1.31 p P p A p B No Table 1: Statistics of the datasets. Bad () A : s which can distinguish or connect the two documents B : overly specific s C : overly general s Yes p C No Yes Perfect (2) Figure 7: Annotation criterion. Good (1) generation is O(N D + n P d 2 ) where P d denotes average number of candidate s in each document. In document graph construction, the costs for computing S is O(N D + n P d l p) where l p is the average number of words in a (l p P d ). The overall cost of data preparation is O(N D + n P d 2 ). In Algorithm 1, it takes O(n + m) time to initialize all the variables. Each iteration of the inner loop costs O(n P d + m) to update {f, f } and O(n P d ) to update {g, g }. The cost for inner loop is O(n P d t i + mt i ) supposing it stops after t i iterations (t i < 5 in our experiments). In the outer loop, update of {y c, y, y } costs O(m) time. Overall, suppose that the outer loop terminates in t o iterations, the computational complexity of PhraseCom is O ( ) n P d t i t o + mt i t o (to < 5 in our experiments). 4. EXPERIMENTS 4.1 Data Preparation Our experiments use two real-world datasets 3 : Academia: We collected 25,484 full-text papers (574M tokens and 1.98M unique words) published in a variety of venues between 199 and 215 from ACM Digital Library. News: constructed by crawling news articles published between Mar. 11 and April 11, 211 with keywords âăij- Japan TsunamiâĂİ from NewsBank. This yields a collection of 67,89 articles (29M tokens and 247k unique words). 1. Salient Phrases. We first performed lemmatization on the tokens to reduce variant forms of words (e.g., ate, eating) into their lemmatized form (e.g., eat). In segmentation, we set maximal pattern length to 5, minimum support to 1, and non-segmented ratio to 5 in SegPhrase algorithm [17], to extract candidate s from the corpus. We then used the GenDeR algorithm in [7] to solve the salient selection problem in Eq. (3). We set weighting parameter µ = 3 and maximal number of salient selected for each document K = 3. Member s in salient pairs were removed from the salient set. 3 http://dl.acm.org/; http://www.newsbank.com/ Distinct - Common - Overall - News.7951.6827.7395.7261.7769.7138 Academia 5.5.75 1 (a) Kappa coefficient Distinct - Common - Overall - News.9369.941.9251 893.9332 928 Academia 5.5.75 1 (b) Relative observed agreement Figure 8: Inter-annotator agreement. 2. Bipartite Graphs. We followed the introduction in Sec. 3.3 to construct the -document bipartite graph for each dataset. To compute the BM25 score between a pair and a document, we concatenated the two member s in the pair together. Table 1 summarizes the statistics of the constructed bipartite graphs for the two datasets. 3. Evaluation Sets. For evaluation purposes, we selected papers published in two different areas (i.e., KDD and SI- GIR) along with the News articles on Japan Tsunami to construct three evaluation sets. To generate document pairs U, we computed the document cosine similarity using TF-IDF vectors. We sampled 35 pairs of more related documents (score >.6), 35 pairs of less related documents (5 < score < ) and 35 random pairs for each evaluation set. As the number of s in each document is large, we adopted the pooling method [21] to generate gold standard s. For each document pair, we constructed the pool with the results returned by all the compared methods and the top-15 salient words and s by WordMatch and PhraseCom. Human assessment was conducted by computer science researchers and students following the criterion in Fig. 7. This yields 8,463, 8,547 and 5,985 annotated s for KDD, SIGIR and News evaluation sets, respectively. In Fig. 8, the kappa value and relative observed agreement both demonstrate that the human annotators have good interjudge agreement on both common and distinct s. 4.2 Experimental Settings In our testing of PhraseCom and its variants, we set {λ, α} = {.1, 1} based on the required condition and effectiveness study on a validation set. Empirically, our performance does not change dramatically across a wide choices of parameters. For convergence criterion, we stop the outer (inner) loops in Algorithm 1 if the relative change of O in Eq. (8) and F in (9) ( reconstruction error in Eq. (7) ) is smaller than 1 4. Compared Methods: We compared the proposed method (PhraseCom) with its variants which only model part of the proposed hypotheses. Several state-of-the-art comparative summarization methods were also implemented (parameters were first tuned on our validation sets): (1) Word- Match [2]: extracts top-2 salient words based on TF-IDF scores. It generates common set based on (a) in Fig. 5 and takes the rest as distinct sets; (2) TopicLabel [22]: TopicLabel selects salient s based on first-order relevance measure and uses same method as WordMatch to generate common and distinct sets; (3) PatentCom [31]: a stateof-the-art graph-based comparative summarization method. We adopt its common and distinct sets for comparisons; (4) StringFuzzy: follows Sec. 3.1 to extract salient s. It finds common if its BM25 scores to both documents are larger than a threshold (3.) and uses the remaining salient s to form distinct sets; and (5) ContextFuzzy: Different from StringFuzzy, it measures cosine
SIGIR (common) SIGIR (distinct) KDD (common) KDD (distinct) Method P R F1 P R F1 P R F1 P R F1 WordMatch [2] 93 52 62 63.132 81 16 5 7 35 95 49 TopicLabel [22] 2 16 18.158 27 26 1 3 4.1 48.137 PatentCom [31] 93 92.346 85.696 13.563.379 23 91.732 2 StringFuzzy.181 15 83 61 94.329 2 2.32 99.631.391 ContextFuzzy.128 39 1 59.335 81.171.796 48.31 85.338 CDA-AlterMea.16.36.157.1 37 49 88.324.131.128 29 44 CDA-NoGraph.628.637.63.67.529.562.799.75.716.756.645.676 CDA-TwoStep.711 18.749.78.55.597.721 25.749.763.72.726 CDA.752 43.778.74.596.644 7 34 13.788.711.733 Table 2: Performance comparisons on Academia dataset in terms of Precision, Recall and F1 score. Common Distinct Method P R F1 P R F1 WordMatch [2] 35 64 45 79 21.112 TopicLabel [22].327 49.363 12 51.534 PatentCom [31].358 81.399 34 77.554 StringFuzzy.18 14 45.376.735 7 ContextFuzzy.166 22 22.317.661.397 CDA-AlterMea.347.613.399 64.194 15 CDA-NoGraph.6 88.521 38.687.727 CDA-TwoStep.642 4.639 31.726.753 CDA.74 78.757 71.723.773 Table 3: Performance comparisons on News dataset in terms of Precision, Recall and F1 score. similarity between the pseudo-document of a (all contexts in a 1 words window in the corpus) and a document. For PhraseCom, besides the proposed full-fledged model, CDA, we also compare with its variants which implement our intuitions differently: (1) CDA-NoGraph: It uses BM25 scores to compute commonality and distinction measures and optimizes Eqs. (8) and (9) without graph-based regularization ; (2) CDA-AlterMea: It uses summation of relevance scores as a commonality measure and differences between relevance scores as a distinction measure; and (3) CDA-TwoStep: It first learns relevance score based on Eq. (7) and then optimizes Eqs. (8) and (9) without. Evaluation Metrics: We use F1 score computed from Precision and Recall to evaluate the performance. Given a document pair, we denote the set of system-identified common terms as I and the set of gold standard common terms (i.e., words and s which are judged as good or perfect) as G. Precision (P) is calculated by P = I G / I and Recall (R) is calculated by R = I G / G. For each document in the pair, we compute above metrics for distinct terms in a similar manner. The reported numbers are averaged over the evaluation set. For parameter study in validation set, we use the same metrics to evaluate the performance. 4.3 Experiments and Performance Study 1. Comparing CDA with the other methods. Tables 2 and 3 summarize the comparison results on the three evaluation sets. Overall, CDA outperforms others on all metrics on all evaluation sets in terms of finding commonalities, and achieves superior Precision and F1 scores on finding distinctions. In particular, CDA obtains a 125% improvement in F1 score and 188% improvement in Recall on the SIGIR dataset compared to the best baseline PatentCom on finding commonalities, and improves F1 on the News dataset by 39.53% compared to PatentCom on finding distinctions. PatentCom uses feature selection technique to jointly discover distinct words and s for a document pair but suffers from low recall on commonalities since it finds common terms simply by term overlap without considering semantic common words/s (same as WordMatch and TopicLabel). Although its recall on distinctions is high, it has low precision, since it returns a large number of distinct terms without filtering those overly specific ones. Superior performance of CDA validates the effectiveness of our salient generation (vs. WordMatch and TopicLabel) and of the proposed hypotheses on modeling semantic commonality and document- relevance. Both StringFuzzy and ContextFuzzy can find semantic common s but they suffer from low precision and instable recall (49.7% drop from SI- GIR to News) due to its sensitivity to the cut-off threshold. A one-fit-all threshold is not guaranteed to work well for different document pairs or for different domains. It is worth mentioning that CDA performs more stably because it leverages the graph-based semantic relevance and the constraints in the optimization, which can control the size of the output sets automatically like an adaptive thresholding algorithm. 2. Comparing CDA with its variants. Comparing with CDA-NoGraph and CDA-TwoStep, CDA gains performance from propagating semantic relevance on graphs and integrating relevance propagation with selection in a mutually enhancing way. It dramatically outperforms CDA- AlterMea on all metrics on all evaluation sets, which demonstrates the effectiveness of the proposed commonality and distinction measures (see Sec. 3.2). 3. More related pairs versus less related pairs. Fig. 9 compares the methods on pairs of highly similar (more related) documents and pairs of lowly similar (less related) documents, respectively. CDA and its variant outperform other methods in terms of Precision and F1 scores on both kinds of document pairs. As there exist more semantic commonalities and subtle differences between a pair of highly similar documents, CDA gains larger improvement by optimizing the proposed measures and by learning semantic relevance. The superior Recall of CDA over CDA-NoGraph (a 23% improvement on the less related pairs) mainly comes from the graph-based relevance propagation. 4.4 Case Study 1. Example output on Academic papers. Table 5 compares the output of CDA with that of Microsoft Rich-Caption system on two sample papers. Rich-Caption system provides structured information (e.g., related concepts) for papers on Bing Academic vertical (for United States users). However, it suffers from low coverage on the identified related concepts. CDA not only can highlight the subtle differences between two similar papers (first example) but also can find the semantic commonalities (e.g., EM algorithm ) between two moderately similar papers (second example). 2. Comparing two document sets. Fig. 11 presents CDA output for comparing document sets on News dataset (2 documents were sampled for each date). Common s show the connections between things happened in two dif-
1. Precision (common).6 More related Less related 1. Precision (distinct).6 More related Less related CDA CDA-NoGraph PatentCom StringFuzzy ContextFuzzy 1. Recall (common).6 More related Less related More related 1. Recall (distinct).6 More related Less related More related 1. F1 (common).6 1. F1 (distinct).6 Less related Less related Figure 9: Performance study on more related and less related document pairs on the Academia dataset. Recall on semantic common s 852 71 257.3257.575.1.3.5.6 Recall on perfect distinct terms.1367.6833.7635 415.6 1.926 (a) Semantic Commonality (b) Perfect Distinction Figure 1: Case studies on semantic commonality and perfect distinction on the News dataset. ferent dates while distinction s highlight the unique things happened in each date. In particular, distinction results demonstrate that our method can capture pairwise distinction property well by generating different distinct s for the same news set when comparing with different news sets. The comparison results provide a good overview on the event evolution of 211 Japan Tsunami. 3. Testing on semantic commonality. To study the performance on finding semantic common s (Fig. 3), we compare our method with methods that can also find such s. In Fig. 1(a). CDA achieved significant improvement in recall since it leverages the bipartite graph to derive semantic relevance (versus CDA-NoGraph, StringFuzzy, ContextFuzzy), and integrates the relevance score propagation with selection to reinforce the learning of semantic relevance (versus CDA-TwoStep). Compared with StringFuzzy and ContextFuzzy, our method does not require a unified cut-off threshold and thus is more robust. 4. Testing on perfect distinction. We consider overly general terms as positive in previous evaluations (i.e., good labels are given to overly general terms in Sec. 4.2). Next, we further test our method particularly on finding perfect distinct terms (by assigning bad label to those overly general terms). In Fig. 1(b), CDA achieved superior performance (over 9% on recall) compared with other methods. This is because that (1) is not only informative and concise enough for user to understand, but also general enough to highlight the distinctions (versus WordMatch); and (2) our relevance score learning can balance generality and discrimination well so as to be not dominated by overly general terms. (versus CDA-Graph, PatentCom, TopicLabel). 5. Comparing with word-based and sentence-based summarization. Table 4 shows the comparative analysis re- Distinctions of [11] Distinctions of [6] Keywords: search, Web graph, link structure, Keywords: web search, PageRank, search in context, PageRank personalized search hub, partial, skeleton, pages, personalized, section, basis, computation, preference The Hubs Theorem allows basis vectors to be encoded as partial vectors and a hubs skeleton. Our approach enables incremental computation, so that the construction of personalized views from partial vectors is practical at query time. personalized web search, user-specified@@web pages, dynamic programming, incremental computation, theoretical results query, rank, htm, sensitive, ranking, urls, search, topic, context, regional, kids Finally, we compute the query-sensitive importance score of each of these retrieved URLs as follows. In Section 4.3, the topicsensitive ranking vectors were chosen using the topics most strongly associated with the query term contexts. topic-sensitive PageRank, context-specific importance score, query context, topicsensitive@@ ranking vector, query topic Table 4: Distinctions for papers [11] and [6] generated by WordMatch [2] (top), Discriminative Sentence Selection [29] (middle), and CDA (bottom). sults between papers [11] and [6] generated by word-based method [2], sentence-based method [29] (top-2 sentences) and our -based method. We do not include commonality results since sentence-based summarization techniques only provide distinction results [29] for each document. We found that CDA provides sufficiently cohesive and readable results compared with word-based methods, and it keeps the overall summary concise, as compared to sentence-based methods. Furthermore, the results also show that authorgenerated keywords are often not able to highlight the distinctions when compared to other papers. 5. RELATED WORK There have been many attempts on performing comparative analysis on text data. Previous work can be categorized in terms of sources of comparison (e.g., single or multiple text collections), targets of comparison (e.g., between topics, individual documents or document sets), aspects to compare (e.g., commonality, distinction or both), and representation forms of results (e.g., words, sentences). Multi-document summarization [25, 4, 16, 3, 24] aims to generate a compressed summary to cover the consensus of
Distinct Common Japan; Tokyo; tsunami; 7.3- magnitude offshore earthquake; nuclear power plants; Fukushima; wave hitting; shook buildings; property damage; seismic disaster; shinkansen bullet trains; television channels; hydrogen explosion; radioactive disaster; partial meltdown; water cooling failure; lifethreatening injuries; Fukushima; hydrogen explosion; Japan; life-threatening injuries; nuclear power plants; fuel rods; Yukio Edano; press conference; spewing radiation into air; partial meltdown; north-eastern coast; water cooling failure caught fire; crippled nuclear plant; radiation leaked; prevent further explosions; nuclear threat; explosion; earthquake and tsunami; Fukushima; nuclear fuel; Japan; nuclear crisis; radiation leaked; caught fire; crippled nuclear plant; prevent further explosions; stay indoors to avoid; storage bond 3km northwest; high radiation levels; moved quicker in assessing; restore cooling systems; 3/11/211 3/12/211 3/16/211 3/19/211 C Q Figure 11: Compare document sets on News dataset. We use news articles published at four different dates. d: Document Summarization Based on Word Associations d : Cross-Document Summarization by Concept Classification d: A Latent Topic Model for Linked Documents d : Document clustering with cluster refinement and model selection capabilities CDA Microsoft system CDA Microsoft system multi-document summarization; text summarization; document summary sentence selection@@word associations; background corpus; association mixture text summarization; language-independent; unsupervised method; novelty detection; passage similarity@@term weights; crossdocument summarizer; clustering tech- Q niques; discourse structure; relevance measure; conceptual clustering; multi-document summarization N/A n-gram; cluster analysis document clustering; clustering task@@em algorithm; expectation-maximization; plsi model@@linked documents; latent semantic space; link information; citationtopic model; voting scheme@@document cluster labels; gaussian mixture model; discriminative feature set@@cluster label; majority voting@@feature set; clus- document tering digital library; stochastic process; topic model expectation maximization algorithm; mixture model; model selection Table 5: Example output of CDA and Microsoft Academic Rich-Caption system on two papers from the Academia Dataset. information among the original documents. It is different from commonality discovery as it focuses on providing comprehensive view of the corpus (union instead of intersection). Comparative document summarization [31, 19, 1, 29, 34, 9] focuses on the distinction aspect it summarizes the differences between comparable document sets by extracting the discriminative sentences from each set. Existing work formalizes the problem of discriminative sentence selection into different forms, including integer linear programming [1], multivariate normal model estimation [29], and group-related centrality estimation [2]. Going beyond selecting discriminative sentences from a document set, our problem aims to select quality and concise s to highlight not only differences but also commonalities between two documents. In particular, our work is related to [2] since both try to derive common and distinct terms for a pair of related documents, but their work focuses on finding topic-related terms and selecting sentences based on such terms. They assume terms appearing in both documents as the common ones, and treat the remaining terms as distinct ones. As shown in our experiments, this method (labeled as Word- Match) suffers from low recall in finding common terms and low precision in finding distinct terms since it ignores semantic common s and pairwise distinct s. Similarly, Zhang et al. [31] consider both common and distinct aspects in generating comparative summary for patent documents. They derive a term co-occurrence tree which can be used to extract sentences for summarization. However, they use all shared noun s between two documents as common terms, and apply feature selection techniques to find distinct terms. This method (see PatentCom in Sec. 4.3) demonstrates poor performance on finding common terms due to the ignorance of semantic common s; although this method performs well in terms of the recall for distinct s, it achieves low precision, since it fails to model generality and produce many overly-specific s. Another line of related work, referred to as comparative text mining [3], focuses on modeling latent comparison aspects and discovering common and collection-specific word clusters for each aspect. They adopt topic model [2, 3, 18] and matrix factorization [13] to present the common and distinct information by multinomial distributions of words. While latent comparison aspects can help enrich the comparison results, these methods still adopt bag-of-words representation which is often criticized for its unsatisfying readability [22]. Furthermore, it is difficult to apply statistical topic modeling in comparative document analysis as the data statistic between two documents is insufficient, in particular when the documents are about emergent topics (e.g., Academia, News). Finally, our work is also related to comparing reviews in opinion mining [26, 23, 14, 12] and contrastive summarization for entities [15] they also aim to find similarities and differences between two objects. However, these works are restricted to sentiment analysis. 6. CONCLUSION AND FUTURE WORK In this paper, we study the problem of -based comparative summarization for a document pair, called comparative document analysis (CDA), and propose a general graph-based approach to model semantic commonality and pairwise distinction for s. We cast the selection problems into joint optimization problems based on the proposed novel measures. Experiment results demonstrate the effectiveness and robustness of the proposed method on text corpora of different domains. Interesting future work includes extending CDA to consider different comparison aspects [3, 13] and to exploit the hierarchical semantic relations between s. CDA is general and can be applied as a primitive step for sentence-based comparative summarization [1, 29]. It can potentially benefit many other text applications such as content recommendation, document clustering and relevance feedback in information retrieval [21].