Corpora and Statistical Methods Lecture 6. Semantic similarity, vector space models and wordsense disambiguation

Transcription

1 Corpora and Statstcal Methods Lecture 6 Semantc smlarty, vector space models and wordsense dsambguaton

2 Part 1 Semantc smlarty

3 Synonymy Dfferent phonologcal/orthographc words hghly related meanngs: sofa / couch boy / lad Tradtonal defnton: w1 s synonymous wth w2 f w1 can replace w2 n a sentence, salva vertate Is ths ever the case? Can we replace one word for another and keep our sentence dentcal?

4 The mportance of text genre & regster Wth near-synonyms, there are often regster-governed condtons of use. E.g. nave vs gullble vs ngenuous You're so bloody gullble [ ] [ ] outsde on the pavement tryng to entce gullble dots n [ ] You're so ngenuous. You tackle thngs the wrong way. The commentator's ngenuous query could just as well have been prompted [ ] However, t s ngenuous to suppose that peace process [ ] (source: BNC)

5 Synonymy vs. Smlarty The contextual theory of synonymy: based on the work of Wttgensten (1953), and Frth (1957) You shall know a word by the company t keeps (Frth 1957) Under ths vew, perfect synonyms mght not exst. But words can be judged as hghly smlar f people put them nto the same lngustc contexts, and judge the change to be slght.

6 Synonymy vs. smlarty: example Mller & Charles 1991: Weak contextual hypothess:the smlarty of the context n whch 2 words appear contrbutes to the semantc smlarty of those words. E.g. snake s smlar to [resp. synonym of] serpent to the extent that we fnd snake and serpent n the same lngustc contexts. It s much more lkely that snake/serpent wll occur n smlar contexts than snake/toad NB: ths s not a dscrete noton of synonymy, but a contnuous defnton of smlarty

7 The Mller/Charles experment Subjects were gven sentences wth mssng words; asked to place words they felt were OK n each context. Method to compare words A and B: fnd sentences contanng A fnd sentences contanng B delete A and B from sentences and shuffle them ask people to choose whch sentences to place A and B n. Results: People tend to put smlar words n the same context, and ths s hghly correlated wth occurrence n smlar contexts n corpora.

8 Issues wth smlarty Smlar s a much broader concept than synonymous : Contextually related, though dfferng n meanng : man / woman boy / grl master / pupl Contextually related, but wth opposte meanngs : bg / small clever / stupd

9 Uses of smlarty Assumpton: semantcally smlar words behave n smlar ways Informaton retreval: query expanson wth related terms K nearest neghbours, e.g.: gven: a set of elements, each assgned to some topc task: classfy an unknown w by topc method: fnd the topc that s most prevalent among w s semantc neghbours

10 Common approaches Vector-space approaches: represent word w as a vector contanng the words (or other features) n the context of w compare the vectors of w1, w2 varous vector-dstance measures avalable Informaton-theoretc measures: w1 s smlar to w2 to the extent that knowng about w1 ncreases my knowledge (decreases my uncertanty) about w2

11 Vector-space models

12 Basc data structure Matrx M M j = no. of tmes w co-occurs wth w j (n some wndow). Can also have Document * word matrx We can treat matrx cells as boolean: f M j > 0, then w co-occurs wth w j, else t does not.

13 Dstance measures Many measures take a set-theoretc perspectve: vectors can be: bnary (ndcate co-occurrence or not) real-valued (ndcate frequency, or probablty) smlarty s a functon of what two vectors have n common

14 Classc smlarty/dstance measures Boolean vector (sets) Real-valued vector Dce coeffcent Jaccard Coeffcent Dce coeffcent Jaccard Coeffcent v w v w 2 v w v w n n v w v w 1 1 ), mn( 2 n n v w v w 1 1 ), max( ), mn(

15 Dce vs. Jaccard Dce (car, truck) On the boolean matrx: (2 * 2)/(4+2) = 0.66 Jaccard On the boolean matrx: 2/4 = 0.5 Dce s more generous ; Jaccard penalses lack of overlap more.

16 Classc smlarty/dstance measures Boolean vector (sets) Real-valued vector Cosne smlarty Cosne smlarty (= angle between 2 vectors) w v w v n n n w v w v w v w v

17 probablstc approaches

18 Turnng counts to probabltes P(spacewalkng cosmonaut) = ½ = 0.5 P(red car) = ¼ = 0.25 NB: ths transforms each row nto a probablty dstrbuton correspondng to a word

19 Probablstc measures of dstance KL-Dvergence: treat W1 as an approxmaton of W2 D( v w) x P( x v)log P( x v) P( x w) Problems: asymmetrc: D(p q) D(q p) not so useful for word-word smlarty f denomnator = 0, then D(v w) s undefned

20 Probablstc measures of dstance Informaton radus (aka Jenson-Shannon Dvergence) compares total dvergence between p and q to the average of p and q symmetrc! IRad ( W1, W2 ) IRad ( v, w) Dv v w Dw 2 v w 2 Dagan et al (1997) showed ths measure to be superor to KL- Dvergence, when appled to a word sense dsambguaton task.

21 Some characterstcs of vector-space measures 1. Very smple conceptually; 2. Flexble: can represent smlarty based on document cooccurrence, word co-occurrence etc; 3. Vectors can be arbtrarly large, representng wde context wndows; 4. Can be expanded to take nto account grammatcal relatons (e.g. head-modfer, verb-argument, etc).

22 Grammar-nformed methods: Ln (1998) Intuton: The smlarty of any two thngs (words, documents, people, plants) s a functon of the nformaton ganed by havng: a jont descrpton of a and b n terms of what they have n common compared to descrbng a and b separately E.g. do we gan more by a jont descrpton of: apple and char (both THINGS ) apple and banana (both FRUIT: more specfc)

23 Ln s defnton cont/d Essentally, we compare the nfo content of the common defnton to the nfo content of the separate defnton sm( a, b) nf.content of jont descrpton nf content of separate descrptons NB: essentally mutual nformaton!

24 An applcaton to corpora From a corpus-based pont of vew, what do words have n common? context, obvously How to defne context? just bag-of-words (typcal of vector-space models) more grammatcally sophstcated

25 Klgarrff s (2003) applcaton Defnton of the noton of context, followng Ln: defne F(w) as the set of grammatcal contexts n whch w occurs a context s a trple <rel,w,w >: rel s a grammatcal relaton w s the word of nterest w s the other word n rel Grammatcal relatons can be obtaned usng a dependency parser.

26 Grammatcal co-occurrence matrx for cell Source: Jurafsky & Martn (2009), after Ln (1998)

27 Example wth w = cell Example trples: <subject-of, cell, absorb> <object-of, cell, attack> <nmod-of, cell, archtecture> Observe that each trple f conssts of the relaton r, the second word n the relaton w,..and the word of nterest w We can now compute the level of assocaton between the word w and each of ts trples f: I( w, f ) log 2 P( w, f ) P( w) P( r w) P( w' w) An nformaton-theoretc measure that was proposed as a generalsaton of the dea of pontwse mutual nformaton.

28 Calculatng smlarty Gven that we have grammatcal trples for our words of nterest, smlarty of w1 and w2 s a functon of: the trples they have n common the trples that are unque to each sm Ln ( w 1, w 2 ) 2 I( F( w1 ) I( F( w2 )) I( F( w )) I( F( w )) I.e.: mutual nfo of what the two words have n common, dvded by sum of mutual nfo of what each word has 1 2

29 Sample results: master & pupl common: Subject-of: read, st, know Modfer: good, form Possesson: nterest master only: Subject-of: ask Modfer: past (cf. past master) pupl only: Subject-of: make, fnd PP_at-p: school

30 Concrete mplementaton The onlne SketchEngne gves grammatcal relatons of words, plus thesaurus whch rates words by smlarty to a head word. Ths s based on the Ln 1998 model.

31 Lmtatons (or characterstcs) Only applcable as a measure of smlarty between words of the same category makes no sense to compare grammatcal relatons of dfferent category words Does not dstngush between near-synonyms and smlar words student ~ pupl master ~ pupl MI s senstve to low-frequency: a relaton whch occurs only once n the corpus can come out as hghly sgnfcant.