Text Mining Approaches for Surveillance

Transcription

1 Text Mining Approaches for Surveillance Document Space Workshop, IPAM/UCLA Michael W. Berry and Murray Browne Department of Computer Science, UTK January 23, / 61

2 Outline Enron Background Electronic Mail Surveillance Conclusions and References 2 / 61

3 Enron Background Collection Enron Background Collection Historical Events Electronic Mail Surveillance Conclusions and References 3 / 61

4 Enron Background Collection Collection By-product of the FERC investigation of Enron (originally contained 15 million messages). 4 / 61

5 Enron Background Collection Collection By-product of the FERC investigation of Enron (originally contained 15 million messages). This study used the improved corpus known as the Enron set, which was edited by Dr. William Cohen at CMU. 5 / 61

6 Enron Background Collection Collection By-product of the FERC investigation of Enron (originally contained 15 million messages). This study used the improved corpus known as the Enron set, which was edited by Dr. William Cohen at CMU. This set had over 500,000 messages. The majority were sent in the 1999 to 2001 timeframe. 6 / 61

7 Enron Background Historical Events Enron Background Collection Historical Events Electronic Mail Surveillance Conclusions and References 7 / 61

8 Enron Background Historical Events Enron Historical Ongoing, problematic, development of the Dabhol Power Company (DPC) in the Indian state of Maharashtra. 8 / 61

9 Enron Background Historical Events Enron Historical Ongoing, problematic, development of the Dabhol Power Company (DPC) in the Indian state of Maharashtra. Deregulation of the Calif. energy industry, which led to rolling electricity blackouts in the summer of 2000 (and subsequent investigations). 9 / 61

10 Enron Background Historical Events Enron Historical Ongoing, problematic, development of the Dabhol Power Company (DPC) in the Indian state of Maharashtra. Deregulation of the Calif. energy industry, which led to rolling electricity blackouts in the summer of 2000 (and subsequent investigations). Revelation of Enron s deceptive business and accounting practices that led to an abrupt collapse of the energy colossus in October, 2001; Enron filed for bankruptcy in December, / 61

11 Motivation Enron Background Motivation Underlying Optimization Problem MM Method (Lee and Seung) Enforcing Statistical Sparsity Hybrid NMF Approach Electronic Mail Surveillance Conclusions and References 11 / 61

12 Motivation NMF Origins NMF (Nonnegative Matrix Factorization) can be used to approximate high-dimensional data having nonnegative components. 12 / 61

13 Motivation NMF Origins NMF (Nonnegative Matrix Factorization) can be used to approximate high-dimensional data having nonnegative components. Lee and Seung (1999) demonstrated its use as a sum-by-parts representation of image data in order to both identify and classify image features. 13 / 61

14 Motivation NMF Origins NMF (Nonnegative Matrix Factorization) can be used to approximate high-dimensional data having nonnegative components. Lee and Seung (1999) demonstrated its use as a sum-by-parts representation of image data in order to both identify and classify image features. [Xu et al., 2003] demonstrated how NMF-based indexing could outperform SVD-based Latent Semantic Indexing (LSI) for some information retrieval tasks. 14 / 61

15 Motivation NMF for Image Processing A i SVD W Hi U Sparse NMF verses Dense SVD Bases; Lee and Seung (1999) Σ V i 15 / 61

16 Motivation NMF for Text Mining (Medlars) Highest Weighted Terms in Basis Vector W *1 Highest Weighted Terms in Basis Vector W *2 term 1 ventricular 2 aortic 3 septal 4 left 5 defect 6 regurgitation 7 ventricle 8 valve 9 cardiac 10 pressure weight Highest Weighted Terms in Basis Vector W *5 term 1 oxygen flow pressure blood cerebral hypothermia fluid venous arterial perfusion weight Highest Weighted Terms in Basis Vector W *6 term 1 children child autistic speech group early visual anxiety emotional autism weight term 1 kidney marrow dna cells nephrectomy unilateral lymphocytes bone thymidine rats weight Interpretable NMF feature vectors; Langville et al. (2006) 16 / 61

17 Underlying Optimization Problem Derivation Given an m n term-by-message (sparse) matrix X. 17 / 61

18 Underlying Optimization Problem Derivation Given an m n term-by-message (sparse) matrix X. Compute two reduced-dim. matrices W,H so that X WH; W is m r and H is r n, with r n. 18 / 61

19 Underlying Optimization Problem Derivation Given an m n term-by-message (sparse) matrix X. Compute two reduced-dim. matrices W,H so that X WH; W is m r and H is r n, with r n. Optimization problem: min X W,H WH 2 F, subject to W ij 0 and H ij 0, i, j. 19 / 61

20 Underlying Optimization Problem Derivation Given an m n term-by-message (sparse) matrix X. Compute two reduced-dim. matrices W,H so that X WH; W is m r and H is r n, with r n. Optimization problem: min X W,H WH 2 F, subject to W ij 0 and H ij 0, i, j. General approach: construct initial estimates for W and H and then improve them via alternating iterations. 20 / 61

21 MM Method (Lee and Seung) Multiplicative Method (MM) Multiplicative update rules for W and H (Lee and Seung, 1999): 1. Initialize W and H with non-negative values, and scale the columns of W to unit norm. 2. Iterate for each c, j, and i until convergence or after k iterations: 2.1 H cj H cj (W T X ) cj (W T WH) cj + ɛ 2.2 W ic W ic (XH T ) ic (WHH T ) ic + ɛ 2.3 Scale the columns of W to unit norm. 21 / 61

22 MM Method (Lee and Seung) Multiplicative Method (MM) Multiplicative update rules for W and H (Lee and Seung, 1999): 1. Initialize W and H with non-negative values, and scale the columns of W to unit norm. 2. Iterate for each c, j, and i until convergence or after k iterations: 2.1 H cj H cj (W T X ) cj (W T WH) cj + ɛ 2.2 W ic W ic (XH T ) ic (WHH T ) ic + ɛ 2.3 Scale the columns of W to unit norm. Setting ɛ = 10 9 will suffice [Shahnaz et al., 2006]. 22 / 61

23 MM Method (Lee and Seung) Normalization, Complexity, and Convergence Important to normalize X initially and the basis matrix W at each iteration. 23 / 61

24 MM Method (Lee and Seung) Normalization, Complexity, and Convergence Important to normalize X initially and the basis matrix W at each iteration. When optimizing on a unit hypersphere, the column (or feature) vectors of W, denoted by W k, are effectively mapped to the surface of the hypersphere by repeated normalization. 24 / 61

25 MM Method (Lee and Seung) Normalization, Complexity, and Convergence Important to normalize X initially and the basis matrix W at each iteration. When optimizing on a unit hypersphere, the column (or feature) vectors of W, denoted by W k, are effectively mapped to the surface of the hypersphere by repeated normalization. MM implementation of NMF requires O(rmn) operations per iteration; Lee and Seung (1999) proved that X WH 2 F is monotonically non-increasing with MM. 25 / 61

26 MM Method (Lee and Seung) Normalization, Complexity, and Convergence Important to normalize X initially and the basis matrix W at each iteration. When optimizing on a unit hypersphere, the column (or feature) vectors of W, denoted by W k, are effectively mapped to the surface of the hypersphere by repeated normalization. MM implementation of NMF requires O(rmn) operations per iteration; Lee and Seung (1999) proved that X WH 2 F is monotonically non-increasing with MM. From a nonlinear optimization perspective, MM/NMF can be considered a diagonally-scaled gradient descent method. 26 / 61

27 Enforcing Statistical Sparsity Hoyer s Method From neural network applications, Hoyer (2002) enforced statistical sparsity for the weight matrix H in order to enhance the parts-based data representations in the matrix W. 27 / 61

28 Enforcing Statistical Sparsity Hoyer s Method From neural network applications, Hoyer (2002) enforced statistical sparsity for the weight matrix H in order to enhance the parts-based data representations in the matrix W. Mu et al. (2003) suggested a regularization approach to achieve statistical sparsity in the matrix H: point count regularization; penalize the number of nonzeros in H rather than ij H ij. 28 / 61

29 Enforcing Statistical Sparsity Hoyer s Method From neural network applications, Hoyer (2002) enforced statistical sparsity for the weight matrix H in order to enhance the parts-based data representations in the matrix W. Mu et al. (2003) suggested a regularization approach to achieve statistical sparsity in the matrix H: point count regularization; penalize the number of nonzeros in H rather than ij H ij. Goal of increased sparsity better representation of parts or features spanned by the corpus (X ) [Shahnaz et al., 2006]. 29 / 61

30 Hybrid NMF Approach GD-CLS Hybrid Approach First use MM to compute an approximation to W for each iteration a gradient descent (GD) optimization step. 30 / 61

31 Hybrid NMF Approach GD-CLS Hybrid Approach First use MM to compute an approximation to W for each iteration a gradient descent (GD) optimization step. Then, compute the weight matrix H using a constrained least squares (CLS) model to penalize non-smoothness (i.e., non-sparsity) in H common Tikohonov regularization technique used in image processing (Prasad et al., 2003). 31 / 61

32 Hybrid NMF Approach GD-CLS Hybrid Approach First use MM to compute an approximation to W for each iteration a gradient descent (GD) optimization step. Then, compute the weight matrix H using a constrained least squares (CLS) model to penalize non-smoothness (i.e., non-sparsity) in H common Tikohonov regularization technique used in image processing (Prasad et al., 2003). Convergence to a non-stationary point evidenced (but no formal proof given to date). 32 / 61

33 Hybrid NMF Approach GD-CLS Algorithm 1. Initialize W and H with non-negative values, and scale the columns of W to unit norm. 33 / 61

34 Hybrid NMF Approach GD-CLS Algorithm 1. Initialize W and H with non-negative values, and scale the columns of W to unit norm. 2. Iterate until convergence or after k iterations: (XH 2.1 W ic W T ) ic ic (WHH T, for c and i ) ic + ɛ 2.2 Rescale the columns of W to unit norm. 2.3 Solve the constrained least squares problem: min H j { X j WH j λ H j 2 2}, where the subscript j denotes the j th column, for j = 1,..., m. 34 / 61

35 Hybrid NMF Approach GD-CLS Algorithm 1. Initialize W and H with non-negative values, and scale the columns of W to unit norm. 2. Iterate until convergence or after k iterations: (XH 2.1 W ic W T ) ic ic (WHH T, for c and i ) ic + ɛ 2.2 Rescale the columns of W to unit norm. 2.3 Solve the constrained least squares problem: min H j { X j WH j λ H j 2 2}, where the subscript j denotes the j th column, for j = 1,..., m. Any negative values in H j are set to zero. The parameter λ is a regularization value that is used to balance the reduction of the metric X j WH j 2 2 with enforcement of smoothness and sparsity in H [Shahnaz et al., 2006]. 35 / 61

36 Electronic Mail Surveillance Message Parsing Enron Background Electronic Mail Surveillance Message Parsing Term Weighting GD-CLS Benchmarks Clustering and Topic Extraction Topic Tracking (Through Time) Smoothing Effects Comparison Conclusions and References 36 / 61

37 Electronic Mail Surveillance Message Parsing inbox Collection Parsed inbox folder of all 150 accounts (users) via GTP (General Text Parser); 495-term stoplist used and extracted terms must appear in more than 1 and more than once globally. 37 / 61

38 Electronic Mail Surveillance Message Parsing private Collection Parsed all mail directories (of all 150 accounts) with the exception of all documents, calendar, contacts, deleted items, discussion threads, inbox, notes inbox, sent, sent items, and sent mail; 495-term stoplist used and extracted terms must appear in more than 1 and more than once globally. 38 / 61

39 Electronic Mail Surveillance Message Parsing private Collection Parsed all mail directories (of all 150 accounts) with the exception of all documents, calendar, contacts, deleted items, discussion threads, inbox, notes inbox, sent, sent items, and sent mail; 495-term stoplist used and extracted terms must appear in more than 1 and more than once globally. Distribution of messages sent in the year 2001: Month Msgs Terms Month Msgs Terms Jan 3,621 17,888 Jul 3,077 17,617 Feb 2,804 16,958 Aug 2,828 16,417 Mar 3,525 20,305 Sep 2,330 15,405 Apr 4,273 24,010 Oct 2,821 20,995 May 4,261 24,335 Nov 2,204 18,693 Jun 4,324 18,599 Dec 1,489 8, / 61

40 Electronic Mail Surveillance Term Weighting Term Weighting Schemes For m n term-by-message matrix X = [x ij ], define x ij = l ij g i d j, where l ij is the local weight for term i occurring in message j, g i is the global weight for term i in the subcollection, and d j is a document normalization factor (set d j = 1). 40 / 61

41 Electronic Mail Surveillance Term Weighting Term Weighting Schemes For m n term-by-message matrix X = [x ij ], define x ij = l ij g i d j, where l ij is the local weight for term i occurring in message j, g i is the global weight for term i in the subcollection, and d j is a document normalization factor (set d j = 1). Schemes used in parsing inbox and private subcollections: Name Local Global txx Term Frequency None l ij = f ij g i = 1 lex Logarithmic Entropy (Define: p ij = f ij / j f ij) l ij = log(1 + f ij ) g i = 1+ ( j p ijlog(p ij ))/ log n) 41 / 61

42 Electronic Mail Surveillance GD-CLS Benchmarks Computational Complexity Rank-50 NMF (X WH) computed on a 450MHz (Dual) UltraSPARC-II processor using 100 iterations: Mail Dictionary Time Collection Messages Terms λ (sec.) inbox 44, , , , , 521 private 65, , , , , / 61

43 Electronic Mail Surveillance Clustering and Topic Extraction private with Log-Entropy Weighting Identify rows of H from X WH or H k with λ = 0.1; r = 50 feature vectors (W k ) generated by GD-CLS: Feature Cluster Topic Dominant Index (k) Size Description Terms California ca, cpuc, gov, socalgas, sempra, org, sce, gmssr, aelaw, ci Louise Kitchen evp, fortune, named top britain, woman, woman by ceo, avon, fiorina, Fortune cfo, hewlett, packard Fantasy game, wr, qb, play, football rb, season, injury, updated, fantasy, image (Cluster size no. of H k elements > row max/10) 43 / 61

44 Electronic Mail Surveillance Clustering and Topic Extraction private with Log-Entropy Weighting Additional topic clusters of significant size: Feature Cluster Topic Dominant Index (k) Size Description Terms Texas UT, orange, longhorn longhorn[s], texas, football true, truorange, newsletter recruiting, oklahoma, defensive Enron partnership[s], collapse fastow, shares, sec, stock, shareholder, investors, equity, lay s dabhol, dpc, india, about India mseb, maharashtra, indian, lenders, delhi, foreign, minister 44 / 61

45 Electronic Mail Surveillance Topic Tracking (Through Time) 2001 Topics Tracked by GD-CLS r = 50 features, lex term weighting, λ = / 61

46 Electronic Mail Surveillance Smoothing Effects Comparison Two Penalty Term Formulation Introduce smoothing on W k (feature vectors) in addition to H k : min W,H { X WH 2 F + α W 2 F + β H 2 F }, where F is the Frobenius norm. 46 / 61

47 Electronic Mail Surveillance Smoothing Effects Comparison Two Penalty Term Formulation Introduce smoothing on W k (feature vectors) in addition to H k : min W,H { X WH 2 F + α W 2 F + β H 2 F }, where F is the Frobenius norm. Constrained NMF (CNMF) iteration [Piper et al., 2004]: H cj H cj (W T X ) cj βh cj (W T WH) cj + ɛ W ic W ic (XH T ) ic αw ic (WHH T ) ic + ɛ 47 / 61

48 Electronic Mail Surveillance Smoothing Effects Comparison Term Distribution in Feature Vectors Terms Wt Lambda Alpha Topics Blackouts Cal Stocks Collapse UT Texasfoot Chronicle Indian India Fastow Collapse Gas CFO Kitchen Californians Cal Solar Partnerships Collapse Workers Maharashtra India Mseb India Beach Ljm Collapse Tues IPPS Cal Rebates Ljm Collapse 48 / 61

49 Electronic Mail Surveillance Smoothing Effects Comparison British National Corpus (BNC) Noun Conservation In collaboration with P. Keila and D. Skillicorn (Queens Univ.) 289,695 subset (all mail folders - not just private) Smoothing solely applied to NMF W matrix (α = 0.001, 0.01, 0.1, 0.25, 0.50, 0.75, 1.00 with β = 0) Log-entropy term weighting applied to the term-by-message matrix X Monitor top ten nouns for each feature vector (ranked by descending component values) and extract those appearing in two or more features; topics assigned manually. 49 / 61

50 Electronic Mail Surveillance Smoothing Effects Comparison BNC Noun Distribution in Feature Vectors Noun GF Entropy Alpha Topic Waxman Downfall Lieberman Downfall Scandal Downfall Nominee(s) Barone Downfall MEADE Downfall Fichera California blackout Prabhu India-strong Tata India-weak Rupee(s) India-strong Soybean(s) Rushing Football - college Dlrs Janus India-weak BSES India-weak Caracas Escondido California/Blackout Promoters Energy/Scottish Aramco India-weak DOORMAN Bawdy/Real Estate 50 / 61

51 Conclusions and References Enron Background Electronic Mail Surveillance Conclusions and References Conclusions Future Work References 51 / 61

52 Conclusions and References Conclusions Conclusions GD-CLS Algorithm can effectively produce a parts-based approximation X WH of a sparse term-by-message matrix X. 52 / 61

53 Conclusions and References Conclusions Conclusions GD-CLS Algorithm can effectively produce a parts-based approximation X WH of a sparse term-by-message matrix X. Smoothing on the features matrix (W ) as opposed to the weight matrix H forces more reuse of higher weighted terms. 53 / 61

54 Conclusions and References Conclusions Conclusions GD-CLS Algorithm can effectively produce a parts-based approximation X WH of a sparse term-by-message matrix X. Smoothing on the features matrix (W ) as opposed to the weight matrix H forces more reuse of higher weighted terms. Surveillance systems based on GD-CLS could be used to monitor discussions without the need to isolate or perhaps incriminate individual employees. 54 / 61

55 Conclusions and References Conclusions Conclusions GD-CLS Algorithm can effectively produce a parts-based approximation X WH of a sparse term-by-message matrix X. Smoothing on the features matrix (W ) as opposed to the weight matrix H forces more reuse of higher weighted terms. Surveillance systems based on GD-CLS could be used to monitor discussions without the need to isolate or perhaps incriminate individual employees. Potential applications include the monitoring/tracking of company morale, employee feedback to policy decisions, and extracurricular activities 55 / 61

56 Conclusions and References Future Work Future Work Further work needed in determining effects of alternative term weighting schemes (for X ) and choices of control parameters (α, β, λ) on quality of the basis vectors W k. 56 / 61

57 Conclusions and References Future Work Future Work Further work needed in determining effects of alternative term weighting schemes (for X ) and choices of control parameters (α, β, λ) on quality of the basis vectors W k. How does document (or message) clustering change with different column ranks (r) in the matrix W? 57 / 61

58 Conclusions and References Future Work Future Work Further work needed in determining effects of alternative term weighting schemes (for X ) and choices of control parameters (α, β, λ) on quality of the basis vectors W k. How does document (or message) clustering change with different column ranks (r) in the matrix W? Use MailMiner and similar text mining software to produce a topic annotated Enron subset for the public domain. 58 / 61

59 Conclusions and References Future Work Future Work Further work needed in determining effects of alternative term weighting schemes (for X ) and choices of control parameters (α, β, λ) on quality of the basis vectors W k. How does document (or message) clustering change with different column ranks (r) in the matrix W? Use MailMiner and similar text mining software to produce a topic annotated Enron subset for the public domain. Explore use of NMF for automated gene classification; Semantic Gene Organizer (K. Heinrich, PhD Thesis 2006) 59 / 61

60 Conclusions and References References For Further Reading F. Shahnaz, M.W. Berry, V.P. Pauca, and R.J. Plemmons. Document Clustering Using Nonnegative Matrix Factorization. Information Processing & Management 42(2), 2006, pp J. Piper, P. Pauca, R. Plemmons, and M. Giffin. Object Characterization from Spectral Data using Independent Component Analysis and Information Theory. Proc. AMOS Technical Conference, Maui, HI, September W. Xu, X. Liu, and Y. Gong. Document-Clustering based on Non-Negative Matrix Factorization. Proceedings of SIGIR 03, July 28 - August 1, Toronto, CA, pp , / 61

61 Conclusions and References References SMD06 Text Mining Workshop Hyatt Regency Bethesda Bethesda, Maryland April 22, 2006 to be held in conjunction with Sixth SIAM International Conference on Data Mining (SDM 2006) and also in conjunction with SIAM's Link Analysis, Counterterrorism Security Workshop, which is also being held on April 22, Website: 61 / 61