Lesson4: Descriptive Modelling of Similarity of Text Unit3: Vector space models for similarity

Transcription

1 Lesson4: Descriptive Modelling of Similarity of Text Unit3: Vector space models for similarity Rene Pickhardt Introduction to Web Science Part 2 Emerging Web Properties CC-BY-SA Rene Institute Pickhardt for Web Science Web and Science Technologies Part2 3 Ways University to study the of Web Koblenz-Landau, Germany 22

2 Completing this unit you should Be familiar with the the vector space model for text documents Be aware of term frequency and (inverse) document frequency Have reviewed the definitions of base and dimension Realize that the angle between two vectors can be seen as a similarity measure CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 23

3 A model based on vector spaces 1) Model documents as vectors of words Vector space Model CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 24

4 A model based on vector spaces 1) Model documents as vectors of words Vector space Model 2) calculating Distance between vectors CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 25

5 A model based on vector spaces 1) Model documents as vectors of words 3) interpreting 2) calculating Vector space Model CC-BY-SA Rene Pickhardt Distance between vectors Web Science Part2 3 Ways to study the Web 26

6 A model based on vector spaces Be able to make statements about similarity of documents 1) Model documents as vectors of words 3) interpreting 2) calculating Vector space Model CC-BY-SA Rene Pickhardt Distance between vectors Web Science Part2 3 Ways to study the Web 27

7 Pay close attention to notation Words W = {w 1,w 2,...,w n } w i D j Word Vectors V =< ~w 1, ~w 2,..., ~w n > ~w i ~ dj Document D j 2 W are a sequence of words So D j = w i1 w i2...w im has a length of m Document vector ~d j = nx i=1 tf(w i,d j ) ~w i CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 28

8 Usually tf-idf is considered instead of tf! The document frequency is defined as df (w i )= {D j w i ind j } Inverse document frequency is defined as idf(w i ) = log D resulting in df (w i ) tfidf(w i,d j )=tf(w i,d j ) log D df (w i ) In the videos and slides for simplicity of numbers we will only use the term frequency CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 29

9 Example (generic language) Let us assume 3 documents D 1 = a a a b b a a b a b D 2 = a b b D 3 = a a b In our artificial language we have just two words a and b Which documents are similar to each other? CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 30

10 Choose a vector space and base Let ~a = V =< ~a, ~ b> be the vector space spanned by the words a and b 1 0 is a base vector for word a Similarly for word b we have ~ 0 b = 1 w i D j ~w i ~ dj Chosing the base was a modelling choice! CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 31

11 Calculate the modelled document vectors d 1 D 1 = a a a b b a a b a b tf(a, D 1 )=6 tf(b, D 1 )=4 Let us now create the document vectors ~d 1 = 2X i=1 tf(w i,d 1 ) ~w i = tf(a, D 1 )~a + tf(b, D 1 ) ~ b =6 +4 = CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 32

12 Calculate the modelled document vectors d 2 D 2 = a b b tf(a, D 2 )=1 tf(b, D 2 )=2 Let us now create the document vectors ~d 2 = 2X i=1 tf(w i,d 2 ) ~w i = tf(a, D 2 )~a + tf(b, D 2 ) ~ b ~d 2 =1 +2 = CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 33

13 Calculate the modelled document vectors d 3 D 3 = a a b tf(a, D 3 )=2 tf(b, D 3 )=1 Let us now create the document vectors ~d 3 = 2X i=1 tf(w i,d 3 ) ~w i = tf(a, D 3 )~a + tf(b, D 3 ) ~ b ~d 3 =2 +1 = CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 34

14 Digression: Drawing the document vectors D 1 = a a a b b a a b a b, D 2 = a b b, D 3 = a a b ~d 1 = 6, d 4 ~ 1 2 =, d 2 ~ 2 3 = 1 CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 35

15 Which vectors are closest too each other? D 1 = a a a b b a a b a b, D 2 = a b b, D 3 = a a b ~d 1 = 6, d 4 ~ 1 2 =, d 2 ~ 2 3 = 1 CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 36

16 Two ways of calculating the distance between two vectors d i and d j? Euclidean distance Take a rule and measure Take difference: ~d = d ~ i dj ~ Calculate length of difference d ~ 2 Cosine distance Take the angle between vectors CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 37

17 Calculate Euclidean distance between two document vectors d i and d j Take the difference ~d = d ~ i dj ~ Calculate the length of the difference ~ d 2 = nx (d k ) 2 = k=1 k=1 nx (( d ~ i ) k ( d ~ j ) k ) 2 CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 38

18 Calculate Euclidean distance between two document vectors d i and d j Take the difference ~d = d ~ i dj ~ Calculate the length of the difference ~ d 2 = nx (d k ) 2 = k=1 k=1 nx (( d ~ i ) k ( d ~ j ) k ) 2 The k-th component of vector ( d ~ i ) k = tf(w k,d i ) by definition ~d i CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 39

19 Calculate Euclidean distance between two document vectors d i and d j Take the difference ~d = d ~ i dj ~ Calculate the length of the difference ~ d 2 = nx (d k ) 2 = k=1 = nx (( d ~ i ) k ( d ~ j ) k ) 2 k=1 nx (tf(w k,d i ) tf(w k,d j )) 2 k=1 w k D i D j For every word we compare how often it appears in document and document CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 40

20 Calculate Euclidean distance between two document vectors d i and d j Take the difference ~d = d ~ i dj ~ Calculate the length of the difference ~ d 2 = nx (d k ) 2 = k=1 = nx (( d ~ i ) k ( d ~ j ) k ) 2 k=1 nx (tf(w k,d i ) tf(w k,d j )) 2 k=1 w k D i D j For every word we compare how often it appears in document and document CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 41

21 Euclidean distances for our example D 1 = a a a b b a a b a b, D 2 = a b b, D 3 = a a b d ~ 1 d2 ~ e =5.4 d ~ 1 d3 ~ e =5 d ~ 2 d3 ~ e =1.4 CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 42

22 Calculate Cosine distance between two document vectors d i and d j Calculate the scalar product s =< d ~ i, d ~ j > < ~ d i, ~ d j >= = nx ( d ~ i ) k ( d ~ j ) k k=1 nx k=1 tf(w k,d i )tf(w k,d j ) Remember: ( d ~ i ) k = tf(w k,d i ) is zero most of the time. CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 43

23 Calculate Cosine distance between two document vectors d i and d j Calculate the scalar product Divide it by the product of lengths of both vectors (length as in Euclidean distance) cos( ) = < ~ d i, ~ d j > ~ d i ~ d j ) = cos 1 < d ~ i, d ~! j > d ~ i d ~ j s =< ~ d i, ~ d j > CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 44

24 Cosine distances for our example D 1 = a a a b b a a b a b, D 2 = a b b, D 3 = a a b ~ d 1 ~ d2 c =0.52 ~ d 1 ~ d3 c =0.12 ~ d 2 ~ d3 c =0.64 CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 45

25 Comparing cosine and Euclidean distance D 1 = a a a b b a a b a b, D 2 = a b b, D 3 = a a b Euklid Cosine ~ d 1 ~ d2 ~ d 2 ~ d ~ d 1 ~ d Different choices of metric can yield very different results Choice of metric is part of the model! Usually cosine distance is considered CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 46

26 Thank you for your attention! Contact: Rene Pickhardt Institute for Web Science and Technologies Universität Koblenz-Landau CC-BY-SA Rene Institute Pickhardt for Web Science Web and Science Technologies Part2 3 Ways University to study the of Web Koblenz-Landau, Germany 47

27 Copyright: This Slide deck is licensed under creative commons 3.0. share alike attribution license. It was created by Rene Pickhardt. You can use share and modify this slide deck as long as you attribute the author and keep the same license. By Alecmconroy (Own work) [GFDL ( or CC BY 3.0 ( creativecommons.org/licenses/by/3.0)], via Wikimedia Commons CC-BY-SA by CSTAR & Oleg Alexandrov CC-BY-SA Rene Pickhardt Web Science Part2 3 Ways to study the Web 48