Text Analysis. Week 5

Transcription

1 Week 5 Text Analysis Reference and Slide Source: ChengXiang Zhai and Sean Massung Text Data Management and Analysis: a Practical Introduction to Information Retrieval and Text Mining. Association for Computing Machinery and Morgan & Claypool, New York, NY, USA.

2 What is Multimedia? Multiple Carriers of information

3 Sensory data (Video, Audio, etc.) Web data (Text) (OSN, News, etc.) Multimedia User data (User attributes, Preferences, etc.)

4 Main Topics Audio Text Video Information Fusion Case Studies 4

5 Text Mining in Multimedia Real world Actionable knowledge Sensor 1 Sensor 2 Sensor k Non-text data Numerical Categorical Relational Video Data mining software General data mining Video mining Text data Text mining 5

6 Humans as Sensors Weather Locations Networks Thermometer Geo sensor Network sensor 3 C, 15 F, 41 N and 120 W Perceive Human sensor Express Lorem ipsum, dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum, dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum, dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor. Excepteur sint occaecat cupidatat nonproident, sunt in culpa qui officia deserunt mollit anim id est laborum. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. 6

7 Knowledge of the observed world Events and activities Market trend Prediction of future events Product popularity Ratings/reviews 7

8 Knowledge about the Observer Preferences Sentiments Mood 8

9 Text Analysis is also called Text Mining!

10 Text Mining Information Retrieval (IR) Natural Language Processing (NLP) Concept Extraction Web Mining Document Clustering Document Classification

11 How to Capture Text?

12 1. Offline documents 2. Online on the web 3. From speech signal

13 Capturing text from the Web Web crawling/web scrapping APIs JSON/CSV

14 Challenges in Text Analysis Documents are described with many attributes The attributes are either characters, words, or phrases 14

15 How to Represent Text?

16 Digital Representation ASCII Characters American Standard Code for Information Interchange 7-bits EBCDIC Characters Extended Binary Coded Decimal Interchange Code 8-bits

17 How to Represent for Text Analysis?

18 Let s take a sentence A dog is chasing a boy on the playground.

19 Idea 1: String of characters! Most general and simple approach But semantic analysis is difficult Words have more meaning than characters

20 Idea 2: Sequence of words! A dog is chasing a boy on the playground

21 Advantages Easily discover most frequent words Most frequent words lead to topics Many other analysis possibilities

22 Challenges In some languages, e.g. Chinese, it is difficult to identify word boundaries All words are treated equally, nouns, adjectives, verbs all are same This may limit the semantic analysis

23 Idea 3: Part of Speech (POS) Tagging! +Noun, Verb, Determiners, etc.

24 A dog is chasing a boy on the playground String of characters A dog is chasing a boy on the playground Sequence of words Det Det Noun Aux Aux Verb Det Det Noun Prep Det Det Noun Noun phrase Complex verb Noun phrase Noun phrase + POS tags Verb phrase Verb phrase Prep phrase + Syntactic structures Sentence A dog Animal CHASE a boy Person ON the playground Location + Entities and relations Deeper analysis requires more efforts!

25 POS Tagging Challenges Even less robust than sequence of words Difficult to identify all entities with the right types Relations are even harder to find

26 How would you differentiate a love letter from a research paper? Love Like Feel I, you, this, that Challenge Method Application We, you, this, that

27 Idea 4: Count the word frequencies for each document! 27

28 Example 28

29 Bag-of-Words (BoW) Consider the document as a bag/set of words! 29

30 What assumptions are we making? 1. Words are mutually independent 2. Word order in text is irrelevant 30

31 BoW is a Vector Space Model! 31

32 Dimensions Each term represents one dimension of the vector space The term can be a single word or a sequence of words (phrase) The number of terms determines the dimension of the vector space 32

33 Vector Elements The vector elements are weights associated with each term These weights reflect the relevance of the term If the corpus contains n terms, a document is represented as d = {w 1, w 2, w n } 33

34 Term Document Matrix (TDM) An m x n matrix with following features: Rows (i=1,m) represent terms from the corpus Columns (j=1,n) represent documents from the corpus Cell ij stores the weight of the term i in the context of the document j 34

35 VSM: Term Document Matrix Information theory deals with uncertainty and the transfer or storage of quantified Information theory deals with uncertainty information in the form of bits. It is applied in and the transfer or storage of quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty information in the form of bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in management computer and science, the analysis, transfer mathematics, which or storage physics, we of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty introduce linguistics. management here briefly. information A few The concepts in analysis, most the important form from of information bits. It is applied in computer and science, the transfer mathematics, which storage physics, we of and quantified concept theory of information many are very theory useful is entropy, in text data introduce linguistics. here briefly. information fields, such A few The concepts in as most the electrical important form from of information bits. engi- It neering, is applied in which concept is management a building computer of information block and science, for analysis, theory many mathematics, is other which physics, we and theory many are fields, very useful such as entropy, electrical in text engi- data neering, measures. introduce linguistics. here briefly. The most important which is management a building computer A few block and science, concepts for analysis, many mathematics, from information other which physics, we and The problem concept can theory of be information formally are defined very theory useful as the is entropy, in text data measures. introduce linguistics. here briefly. A few The concepts most important from information quantified which The problem uncertainty is management a building can be in predicting block and formally defined the for analysis, value many other which we concept theory of information are very theory useful as the is entropy, in text data of a random measures. variable. introduce In the common quantified which uncertainty is management here briefly. a building in predicting block and The the for analysis, most important value many other which we example The of a of random a problem coin, concept the can variable. two of be values information formally In would defined the common be theory 1 as the is entropy, measures. introduce here briefly. The most important or 0 (depicting quantified which heads uncertainty or tails) in predicting and the the value example The of a problem coin, concept is a building the can two of be values information block for formally would defined be theory many 1 as the is other entropy, random of variable a random measures. representing variable. In these common or 0 (depicting quantified which heads uncertainty is or a building tails) in predicting and block the the for value many other outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the measures. representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In variable a other of random a coin, words, the two values would be 1 representing variable. In these common or 0 (depicting heads or tails) and the outcomes example random is X. In variable other of a coin, words, the two values would be 1 representing these or 0 (depicting heads or tails) and the outcomes random is X. In variable other words, representing these outcomes is X. In other words, D1 D2 D3 D4 D5 complexity algorithm entropy traffic network

36 Text Preprocessing Objective: to choose most relevant words for the corpus also called a dictionary! 36

37 1. Transform various forms of the terms in a common normalized form Transform all words to lower case Use a dictionary to replace synonyms with a common, general term 37

38 Examples Apple, apple, APPLE -> apple Intelligent Systems, Intelligent systems, Intelligent-systems -> intelligent systems automobile, car -> vehicle 38

39 2. Remove High Frequency Terms Very high frequency terms are semantically almost useless Examples: the, a, an, we, do, to 39

40 3. Remove Low Frequency Terms Very low frequency terms are semantically rich, but not representative of the class Examples: dextrosinistral 40

41 The rest of the words are those that represent the corpus the best! Remove words that do not bear meaning Upper cut-off Lower cut-off Frequency of words Revolving power of significant words Significant words Remove highly infrequent words Words by rank order Image s Image source: 41

42 4. Stop Words Stop-words are those words that (on their own) do not bear any information / meaning It is estimated that they represent 20-30% of words in any corpus There is no unique stop-words list Frequently used lists are available at: 42

43 Removing words causes loss of meaning and structure! this is not a good option -> option to be or not to be -> null 43

44 5. Lemmatization & Stemming to reduce the variability of words 44

45 Stemming It is a crude heuristic process that chops off the ends of words to get to a basic form Does not consider linguistic features of the words Normalizes verb tenses 45

46 Examples walking, walks, walked, walker => walk argue, argued, argues, arguing => argu apply, applications, reapplied => apply denormalization => norm

47 Lemmatization Use vocabulary and morphological analysis of words to get the base or dictionary form of a word Base form is also known as the lemma E.g., argue, argued, argues, arguing -> argue 47

48 Summary of Preprocessing Steps 1. Transform various forms of the terms in a common normalized form 2. Remove high frequency terms 3. Remove low frequency terms 4. Remove stop-words 5. Lemmatization & Stemming 48

49 How do we compute term weights? Information theory deals with uncertainty and the transfer or storage of quantified Information theory deals with uncertainty information in the form of bits. It is applied in and the transfer or storage of quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty information in the form of bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in management computer and science, the analysis, transfer mathematics, which or storage physics, we of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty introduce linguistics. management here briefly. information A few The concepts in analysis, most the important form from of information bits. It is applied in computer and science, the transfer mathematics, which storage physics, we of and quantified concept theory of information many are very theory useful is entropy, in text data introduce linguistics. here briefly. information fields, such A few The concepts in as most the electrical important form from of information bits. engi- It neering, is applied in which concept is management a building computer of information block and science, for analysis, theory many mathematics, is other which physics, we and theory many are fields, very useful such as entropy, electrical in text engi- data neering, measures. introduce linguistics. here briefly. The most important which is management a building computer A few block and science, concepts for analysis, many mathematics, from information other which physics, we and The problem concept can theory of be information formally are defined very theory useful as the is entropy, in text data measures. introduce linguistics. here briefly. A few The concepts most important from information quantified which The problem uncertainty is management a building can be in predicting block and formally defined the for analysis, value many other which we concept theory of information are very theory useful as the is entropy, in text data of a random measures. variable. introduce In the common quantified which uncertainty is management here briefly. a building in predicting block and The the for analysis, most important value many other which we example The of a of random a problem coin, concept the can variable. two of be values information formally In would defined the common be theory 1 as the is entropy, measures. introduce here briefly. The most important or 0 (depicting quantified which heads uncertainty or tails) in predicting and the the value example The of a problem coin, concept is a building the can two of be values information block for formally would defined be theory many 1 as the is other entropy, random of variable a random measures. representing variable. In these common or 0 (depicting quantified which heads uncertainty is or a building tails) in predicting and block the the for value many other outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the measures. representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In variable a other of random a coin, words, the two values would be 1 representing variable. In these common or 0 (depicting heads or tails) and the outcomes example random is X. In variable other of a coin, words, the two values would be 1 representing these or 0 (depicting heads or tails) and the outcomes random is X. In variable other words, representing these outcomes is X. In other words, complexity algorithm entropy traffic network D1 D2 D3 D4 D5 49

50 Idea 1: take the value of 0 or 1, to reflect the presence (1) or absence (0) of the term in a particular document! 50

51 Example Doc1: Text mining is to identify useful information. Doc2: Useful information is mined from text. Doc3: Apple is delicious. text information identify mining mined is useful to from apple delicious Doc Doc Doc

52 Observation: few terms occur more often in a specific class of documents! 52

53 Idea 2: measure the term frequency (TF) in a specific document! Assumption: If a term occurs more often in a doc, it measures something important for that doc! 53

54 Term Frequency (TF) Information theory deals with uncertainty and the transfer or storage of quantified Information theory deals with uncertainty information in the form of bits. It is applied in and the transfer or storage of quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty information in the form of bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified many fields, Information such as electrical theory deals engi- with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in computer and science, the transfer mathematics, or storage physics, of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty linguistics. information A few concepts in the form from of information bits. It is applied in management computer and science, the analysis, transfer mathematics, which or storage physics, we of and quantified theory many are fields, very Information useful such as electrical theory in text deals engi- data with neering, uncertainty introduce linguistics. management here briefly. information A few The concepts in analysis, most the important form from of information bits. It is applied in computer and science, the transfer mathematics, which storage physics, we of and quantified concept theory of information many are very theory useful is entropy, in text data introduce linguistics. here briefly. information fields, such A few The concepts in as most the electrical important form from of information bits. engi- It neering, is applied in which concept is management a building computer of information block and science, for analysis, theory many mathematics, is other which physics, we and theory many are fields, very useful such as entropy, electrical in text engi- data neering, measures. introduce linguistics. here briefly. The most important which is management a building computer A few block and science, concepts for analysis, many mathematics, from information other which physics, we and The problem concept can theory of be information formally are defined very theory useful as the is entropy, in text data measures. introduce linguistics. here briefly. A few The concepts most important from information quantified which The problem uncertainty is management a building can be in predicting block and formally defined the for analysis, value many other which we concept theory of information are very theory useful as the is entropy, in text data of a random measures. variable. introduce In the common quantified which uncertainty is management here briefly. a building in predicting block and The the for analysis, most important value many other which we example The of a of random a problem coin, concept the can variable. two of be values information formally In would defined the common be theory 1 as the is entropy, measures. introduce here briefly. The most important or 0 (depicting quantified which heads uncertainty or tails) in predicting and the the value example The of a problem coin, concept is a building the can two of be values information block for formally would defined be theory many 1 as the is other entropy, random of variable a random measures. representing variable. In these common or 0 (depicting quantified which heads uncertainty is or a building tails) in predicting and block the the for value many other outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the measures. representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In The variable a other of random a problem coin, words, the can two be values formally would defined be 1 as the representing variable. In these common or 0 (depicting quantified heads uncertainty or tails) in predicting and the the value outcomes example random is X. of In variable a other of random a coin, words, the two values would be 1 representing variable. In these common or 0 (depicting heads or tails) and the outcomes example random is X. In variable other of a coin, words, the two values would be 1 representing these or 0 (depicting heads or tails) and the outcomes random is X. In variable other words, representing these outcomes is X. In other words, D1 D2 D3 D4 D5 complexity algorithm entropy traffic network

55 Example Task: Separate technical and nontechnical document job and engineer both occur 100 times in a corpus of 100 documents Which term is more important? Most likely, the term job appears in most documents while engineer occurs mainly in technical documents!

56 Observation: generally words not so common in the corpus are more important! 56

57 Idea 3: assign higher weights to terms occurring in fewer documents! 57

58 Inverse Document Frequency (IDF) IDF t =1+ log N N t N = Number of documents N t = Number of documents with term t Note - IDF is computed at the corpus level for each term 58

59 A rare term has high idf and a frequent term has low idf! 59

60 Observation: terms that are not so common in the corpus, but still have same reasonable level of frequency are important! 60

61 TF-IDF Most frequently used weight matrix Generally computer as follows: TF-IDF t = TF IDF t 61

62 Text Retrieval Example Query = news about presidential campaign d 1 d 2 d 3 d 4 d 5 news about news about organic food campaign news of presidential campaign news of presidential campaign presidential candidate news of organic food campaign campaign campaign campaign 62

63 Text Retrieval Example 1. Define the dimensions 2. Define how the vectors are obtained 3. Define similarity between vectors 63

64 The goal is to find the closest document in the corpus! Programming Query q d 2? d M d 3? d 5 dential d 4 d 1? 64

65 Dot product similarity with bit vector representation q = (x 1,, x N ) d = (y 1,, y N ) x i, y i 2{0, 1} 1: word W is p where i i 2 1: word W i is present 0: word W i is absent Sim(q, d) = q.d = x 1 y x N y N = Σ N i=1 x i y i 65

66 Dot product similarity with bit vector representation Query = news about presidential campaign Similarity score d 1 d 2 d 3 d 4 d 5 news about news about organic food campaign news of presidential campaign news of presidential campaign presidential candidate news of organic food campaign campaign campaign campaign

67 Problems d 2, d 3, d 4 all are scored same d 2 should be ranked lower than d 3 and d 4 d 4 should score more than d 3 because d 4 mentioned presidential more times 67

68 Dot product similarity with Term Frequency representation q = (x 1,, x N ) d = (y 1,, y N ) x i = count of word W i in query y i = count of word W i in doc Sim(q, d) = q.d = x 1 y x N y N = Σ N i=1 x i y i 68

69 Dot product similarity with Term Frequency representation Query = news about presidential campaign Similarity score d 1 d 2 d 3 d 4 d 5 news about news about organic food campaign news of presidential campaign news of presidential campaign presidential candidate news of organic food campaign campaign campaign campaign

70 Problems d 2 and d 3 have the same scores d 2 should be ranked lower than d 3 and d 4 70

71 Dot product similarity with TF- IDF representation W 1 x i = count of word W i in query q = (x 1,, x N ) y i = c(w i, d) *IDF(W i ) d = (y 1,, y N ) W 3 W 2 71

72 Dot product similarity with TF- IDF representation Query = news about presidential campaign Similarity score d 1 d 2 d 3 d 4 d 5 news about news about organic food campaign news of presidential campaign news of presidential campaign presidential candidate news of organic food campaign campaign campaign campaign