Text Mining. Exercise: Business Intelligence (Part 7) Summer Term 2014 Stefan Feuerriegel

Size: px

Start display at page:

Download "Text Mining. Exercise: Business Intelligence (Part 7) Summer Term 2014 Stefan Feuerriegel"

Neil Pierce
5 years ago
Views:

1 Text Mining Exercise: Business Intelligence (Part 7) Summer Term 2014 Stefan Feuerriegel

2 Tday s Lecture Objectives 1 Being able t perfrm preprcessing steps fr text mining 2 Learning the representatin as a term-dcument matrix 3 Understanding hw a dictinary-based sentiment analysis wrks Text Mining 2

3 Outline 1 Recap 2 Text Mining 3 Excursus: Sentiment Analysis Text Mining 3

4 Outline 1 Recap 2 Text Mining 3 Excursus: Sentiment Analysis Text Mining: Recap 4

5 Artificial Neural Netwrks Neurns are arranged in three (r mre) layers First layer: Input neurns receive the input vectr x X Hidden layer(s): Cnnect input and utput neurns Final layer: Output neurns cmpute a respnse ỹ Y Input Hidden Output x 1 z 1 x N z M y 1 y 2 When neurns are cnnected as a directed graph withut cycles, this is called a feed-frward ANN Text Mining: Recap 5

6 Supprt Vectr Machine (SVM) Which f these linear separatrs is ptimal? Idea: Maximize separating margin (here: A) Data pints n the margin are called supprt vectrs When calculating decisin bundary, nly supprt vectrs matter; ther training data is ignred Frmulatin as cnvex ptimizatin prblem with glbal slutin y B A x Text Mining: Recap 6

7 Predictive Perfrmance Cnfusin matrix (als named cntingency table r errr matrix) displays predictive perfrmance Cnditin (as determined by Gld standard) True False Psitive Outcme True Psitive (TP) False Psitive (FP) Type I Errr False Alarm Precisin r Psitive Predictive Value = TP TP+FP Negative Outcme False Negative (FN) Type II Errr / Miss True Negative (TN) Sensitivity = TP Rate = TP TP+FN Specificity = TN Rate = TN FP+TN Accuracy = TP+TN Ttal Equivalent with hit rate and recall Text Mining: Recap 7

Sensitivity Receiver Operating Characteristic (ROC) ROC illustrates trade-ff between sensitivity and specificity Interpretatin: Curve A is randm guessing (50% crrect guesses) Curve frm mdel B perfrms

8 Sensitivity Receiver Operating Characteristic (ROC) ROC illustrates trade-ff between sensitivity and specificity Interpretatin: Curve A is randm guessing (50% crrect guesses) Curve frm mdel B perfrms better than A, but wrse than C Curve C frm perfect predictin C B A Area suth-east f curve is named area under the curve and shuld be maximized Specifity Text Mining: Recap 8

9 Predictive vs. Explanatry Pwer Significant difference between predicting and explaining: 1 Empirical Mdels fr Predictin Empirical predictive mdels (e. g. statistical mdels, methds frm data mining) designed t predict new/future bservatins Predictive Analytics describes the evaluatin f the predictive pwer, such as accuracy r precisin 2 Empirical Mdels fr Explanatin Any type f statistical mdel used fr testing causal hypthesis Use methds fr evaluating the explanatry pwer, such as statistical tests r measures like R 2 Text Mining: Recap 9

10 Overfitting When learning algrithm is perfrmed fr t lng, the learner may adjust t very specific randm features nt related t the target functin Overfitting: Perfrmance n training data (in gray) still increases, while the perfrmance n unseen data (in red) becmes wrse y Mean Squared Errr x Flexibility Text Mining: Recap 10

11 Outline 1 Recap 2 Text Mining 3 Excursus: Sentiment Analysis 11

12 Text Mining Text mining seeks patterns in textual cntent, i. e. unstructured data Idea: Impse (mathematical) structure first, then analyze it Examples: Summarizatin Categrizatin Infrmatin extractin Sentiment analysis Lad necessary library tm in R t d text mining library(tm) 12

13 Outline 2 Text Mining Creating the Crpus Transfrming the Crpus Term-Dcument Matrix 13

14 Creating the Crpus Cllectin f textual materials are called crpus Surces can vary frm XML t text files, as well as data frames Crpus(...) creates data representatin frm chsen surce Frequently anntated by additinal metadata (e. g. time stamps) inspect(crpus) displays the structure f a crpus Example: Access sample crpus cnsisting f Reuters crude il news reut21578 <- system.file("texts", "crude", package="tm") reuters <- Crpus(DirSurce(reut21578), readercntrl=list(reader=readreut21578xml)) 14

15 Outline 2 Text Mining Creating the Crpus Transfrming the Crpus Term-Dcument Matrix 15

16 Crpus Transfrmatin Additinal peratins necessary t transfrm unstructured text int a mathematical representatin Perfrm transfrmatins via tm_map(crpus, traf) 1 Remve all nn-text tkens 2 Make all letters lwer case 3 Remve redundant, nn-discriminating tkens (numbers & stpwrds) 4 Reduce all inflected wrd frms t cmmn base, i. e. the stem Example: "Details are given in Sectin 2." "detail are giv in sect" 16

17 Example: Remving HTML/XML Tags # Crpus cntains dcuments in XML frmat; remve the XML tags if (packageversin("tm")$minr <= 5) { reuters <- tm_map(reuters, as.plaintextdcument) } else { reuters <- tm_map(reuters, PlainTextDcument) } inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` DIAMOND SHAMROCK (DIA) CUTS CRUDE PRICES NEW YORK, FEB 26 - Diamnd Shamrck Crp said that effective tday it had cut its cntract prices fr crude il by 1.50 dlrs a barrel. The reductin brings its psted price fr West Texas Intermediate t dlrs a barrel, the cpany said. "The price reductin tday was made in the light f falling il prduct prices and a weak crude il market," a cmpany spkeswman said. Diamnd is the latest in a line f U.S. il cmpanies that have cut its cntract, r psted, prices ver the last tw days citing weak il markets. Reuter 17

18 Example: Stripping Whitespaces reuters <- tm_map(reuters, stripwhitespace) inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` DIAMOND SHAMROCK (DIA) CUTS CRUDE PRICES NEW YORK, FEB 26 - Diamnd Shamrck Crp said that effective tday it had cut its cntract prices fr crude il by 18

19 Example: Remving punctuatins reuters <- tm_map(reuters, remvepunctuatin) inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` DIAMOND SHAMROCK DIA CUTS CRUDE PRICES NEW YORK FEB 26 Diamnd Shamrck Crp said that effective tday it had cut its cntract prices fr crude il by 19

20 Example: Cnverting t Lwer Case reuters <- tm_map(reuters, tlwer) inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` diamnd shamrck dia cuts crude prices new yrk feb 26 diamnd shamrck crp said that effective tday it had cut its cntract prices fr crude il by 20

21 Example: Remving Numbers reuters <- tm_map(reuters, remvenumbers) inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` diamnd shamrck dia cuts crude prices new yrk feb diamnd shamrck crp said that effective tday it had cut its cntract prices fr crude il by 21

22 Stpwrds Stpwrds are shrt functin wrds Occur frequently but n deep meaning Remval f stpwrds in rder t cncentrate n mre imprtant wrds (that are unique/specific fr the text) Examples: the, is, at, which, and n Cmmn apprach is t use predefined list f stpwrds Get such a built-in list via stpwrds(language) sw <- stpwrds("english") length(sw) [1] 174 head(sw) [1] "i" "me" "my" "myself" "we" "ur" 22

23 Example: Remving Stpwrds reuters <- tm_map(reuters, remvewrds, stpwrds("english")) inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` diamnd shamrck dia cuts crude prices new yrk feb diamnd shamrck crp said effective tday cut cntract prices crude il dlrs barrel r 23

24 Stemming Stemming is the prcess f reducing inflected (r smetimes derived) wrds t their stem, base r rt frm Depending n the algrithm, the stem is nt a valid rt frm, but a shrted frm withut an ending Aims t grup wrds with (pssibly) the same meaning Examples: fishing, fished, fish, fisher fish argue, argued, argues, arguing, argus argu argument and arguments argument 24

25 Example: Stemming reuters <- tm_map(reuters, stemdcument, language = "english") inspect(reuters[1]) A crpus with 1 text dcument The metadata cnsists f 2 tag-value pairs and a data frame Available tags are: create_date creatr Available variables in the data frame are: MetaID $`reut xml` diamnd shamrck dia cut crude price new yrk feb diamnd shamrck crp said effect tday cut cntract price crude il dlrs barrel reduc 25

26 Summary: Crpus Transfrmatins Perfrm transfrmatins via tm_map(crpus, traf) R Functin PlainTextDcument stripwhitespace remvepunctuatin tlwer remvenumbers remvewrds stemdcument Transfrmatin Rule Remve HTML/XML tags Eliminate unnecessary spaces, e. g. line breaks Remve punctuatin Cnvert t lwer case letters Remve all numbers Remve stpwrds given by additinal parameter Reduce inflected wrds t stem Results can be represented as a term-dcument matrix fr further evaluatin 26

27 Outline 2 Text Mining Creating the Crpus Transfrming the Crpus Term-Dcument Matrix 27

28 Term-Dcument Matrix Term-dcument matrix is a mathematical matrix that describes the frequency f terms ccurring in dcuments Example: D 1 = "I like prgramming" D 2 = "I hate hate prgramming" Term-dcument matrix given by D 1 D 2 I 1 1 like 1 0 hate 0 2 prgramming 1 1 Term-dcument matrix is input t further machine learning prcedures, such as clustering, classificatin r predictin 28

29 Term-Dcument Matrix Create matrix via TermDcumentMatrix(crpus) frm crpus tdm <- TermDcumentMatrix(reuters) inspect(tdm[200:205, 1:5]) A term-dcument matrix (6 terms, 5 dcuments) Nn-/sparse entries: 4/26 Sparsity : 87% Maximal term length: 10 Weighting : term frequency (tf) Dcs Terms dhabi dia diamnd differenti difficulti dillard

30 Term-Dcument Matrix Use findfreqterms(tdm, n) t find terms that ccur at least n times # Retrieve wrds that ccur at least 10 times findfreqterms(tdm, 10) [1] "accrd" "analyst" "arabia" "barrel" "bpd" [6] "crude" "dlrs" "futur" "gvern" "grup" [11] "increas" "industri" "kuwait" "last" "march" [16] "market" "meet" "minist" "mln" "mnth" [21] "new" "ffici" "il" "ne" "pec" [26] "utput" "pct" "petrleum" "pst" "price" [31] "prduc" "prduct" "quta" "reprt" "reserv" [36] "reuter" "said" "saudi" "say" "sheikh" [41] "studi" "will" "wrld" "year" 30

31 Text Mining Operatins Assciatins are terms that frequently ccur tgether in dcuments Measured by crrelatin between rws in term-dcument matrix findasscs(tdm, term, p) finds assciatins with a crrelatin f at least p fr a term # Find assciatins fr the term 'pec' with a crrelatin f at least 0.8 findasscs(tdm, "pec", 0.8) meet analyst name il want emerg buyer said tri

32 Sparsity f Term-Dcument Matrix Prblem: Term-dcument matrices get very big, with many entries at zer Remval f these s-called sparse entries by deleting wrds that ccur in less than p (in %) f all dcuments remvesparseterms(tdm, p) # Remves wrds that ccur in less than 40% f dcuments tdm.rm.sparse <- remvesparseterms(tdm, 0.4) inspect(tdm.rm.sparse[, 1:5]) A term-dcument matrix (6 terms, 5 dcuments) Nn-/sparse entries: 23/7 Sparsity : 23% Maximal term length: 6 Weighting : term frequency (tf) Dcs Terms barrel march il price reuter said

33 Analyzing a Dictinary f Terms Study nly a subset f wrds f interest, specified by dictinary =... # select relevant terms f interest d <- c("price", "crude", "il") # term-dcument matrix is created nly fr thse entries tdm.small <- TermDcumentMatrix(reuters, list(dictinary = d)) inspect(tdm.small[, 1:5]) A term-dcument matrix (3 terms, 5 dcuments) Nn-/sparse entries: 12/3 Sparsity : 20% Maximal term length: 5 Weighting : term frequency (tf) Dcs Terms crude il price

34 Summary: Term-Dcument Matrix Create term-dcument matrix frm crpus via TermDcumentMatrix(crpus) R Functin findfreqterms(tdm, n) findasscs(tdm, term, p) remvesparseterms(tdm, p) dictinary =... Inspectin Terms ccurring at least n times Terms with a crrelatin f at least p Delete sparse terms with many zers Select a subset f wrds Term-dcument matrix is input t machine learning prcedures, such as clustering, classificatin r predictin 34

35 Dcument Clustering by k-means Example: Term-dcument matrix can be used t cluster dcuments accrding t cntent using k-means kmeans(t(tdm.small), 2) K-means clustering with 2 clusters f sizes 15, 5 Cluster means: crude il price Clustering vectr: Within cluster sum f squares by cluster: [1] (between_ss / ttal_ss = 50.9 %) Available cmpnents: [1] "cluster" "centers" "ttss" "withinss" [5] "tt.withinss" "betweenss" "size" 35

36 Outline 1 Recap 2 Text Mining 3 Excursus: Sentiment Analysis Text Mining: Sentiment Analysis 36

37 Frm News t Sentiment Methds that use the textual representatin f dcuments t measure the psitivity and negativity f the cntent are referred t as pinin mining r sentiment analysis Flw diagram Filtering Preprcessing Sentiment Analysis Evaluatin Crpus Stpwrds Dictinaries Text Mining: Sentiment Analysis 37

38 Sentiment Analysis Frequent apprach utilizes dictinaries cntaining wrds labeled as psitive r negative Let W ps dente the number f psitive wrds, W neg the negative and W tt the ttal number f wrds S-called Net-Optimism sentiment S NO [ 1,+1] is given by S NO = W ps W neg W tt Gives nrmalized rati between psitive and negative terms Example During the first nine mnths f 2008 KRONES remained n curse fr grwth, despite the cyclical dwnturn. On a like-fr-like basis, sales rse by 12.5 % t reach Eur 1,765.9 m. During the perid under review, the cmpany benefited frm the increasing number f clients lking fr all-inclusive jbs. Anther grwth driver during the year s first three quarters was the grup s Plastics Technlgy Divisin. KRONES is the wrld s leading vendr f machines and... Psitive wrds marked in blue Negative wrds marked in red S NO = = Text Mining: Sentiment Analysis 38

39 Sentiment Analysis in R Read dictinaries with psitive/negative wrds int data frame Create crrespnding term-dcument matrices ps <- as.data.frame(read.csv("psitivity.txt", header=false)) tdm.ps <- TermDcumentMatrix(reuters, list(dictinary = t(ps))) neg <- as.data.frame(read.csv("negativity.txt", header=false)) tdm.neg <- TermDcumentMatrix(reuters, list(dictinary = t(neg))) Text Mining: Sentiment Analysis 39

40 Sentiment Analysis in R Calculate Net-Optimism sentiment fr each dcument # Initialize empty vectr t stre results sentiment <- numeric(length(reuters)) # Iterate ver all dcuments fr (i in 1:length(reuters)) { # Calculate Net-Optimism sentiment sentiment[i] <- (sum(tdm.ps[, i]) - sum(tdm.neg[, i]))/sum(tdm[, i]) } # Output results sentiment [1] [8] [15] Sentiment scres are input t data analysis (e. g. regressin) r predictin (e. g. Supprt Vectr Machine) Text Mining: Sentiment Analysis 40

Data Mining with Linear Discriminants. Exercise: Business Intelligence (Part 6) Summer Term 2014 Stefan Feuerriegel

Data Mining with Linear Discriminants Exercise: Business Intelligence (Part 6) Summer Term 2014 Stefan Feuerriegel Tday s Lecture Objectives 1 Recgnizing the ideas f artificial neural netwrks and their