Collocations. (M&S Ch 5)

Transcription

1 Colloations M&S Ch 5

2 Introdution Colloations are haraterized by limited ompositionality. Large overlap between the onepts of olloations and terms tehnial term and terminologial phrase. Colloations sometimes reflet interesting attitudes in English towards different types of substanes: strong igarettes tea offee versus powerful drug e.g. heroin

3 Definition w.r.t Computational and Statistial Literature [A olloation is defined as] a sequene of two or more onseutive words that has harateristis of a syntati and semanti unit and whose eat and unambiguous meaning or onnotation annot be derived diretly from the meaning or onnotation of its omponents. [Chouekra 988] 3

4 ther Definitions/Notions w.r.t. Linguisti Literature Colloations are not neessarily adjaent Typial riteria for olloations: nonompositionality non-substitutability nonmodifiability. Colloations annot be translated into other languages. Generalization to weaker ases strong assoiation of words but not neessarily fied ourrene. 4

5 Linguisti Sublasses of Colloations Light verbs: verbs with little semanti ontent Verb partile onstrutions or Phrasal Verbs Proper Nouns/Names Terminologial Epressions 5

6 verview of the Colloation Deteting Tehniques Surveyed Seletion of Colloations by Frequeny Seletion of Colloation based on Mean and Variane of the distane between foal word and olloating word. Hypothesis Testing Mutual Information 6

7 Frequeny Justeson & Katz 995. Seleting the most frequently ourring bigrams. Passing the results through a part-ofspeeh filter Simple method that works very well. 7

8 Mean and Variane I Smadja et al. 993 Frequeny-based searh works well for fied phrases. However many olloations onsist of two words in more fleible relationships. The method omputes the mean and variane of the offset signed distane between the two words in the orpus. If the offsets are randomly distributed i.e. no olloation then the variane/sample deviation will be high. 8

9 Mean and Variane II n = number of times two words olloate μ = sample mean d i = the value of eah sample Sample deviation: s n d i μ i n 9

10 Eample of fleible olloation knoked on the door 3 knoked at the door 3 knoked on John s door 5 knoked on the metal front door 5 μ = /4 = 4 s = sqrt / 3 =.5 If s is big => no olloation If μ is not zero => fleible olloation 0

11 Hypothesis Testing: verview High frequeny and low variane an be aidental. We want to determine whether the oourrene is random or whether it ours more often than hane. This is a lassial problem in Statistis alled Hypothesis Testing. We formulate a null hypothesis H 0 no assoiation - only hane and alulate the probability p that a olloation would our if H 0 were true and then rejet H 0 if p is too low. therwise retain H 0 as possible.

12 Hypothesis Testing: The t-test The t-test looks at the mean and variane of a sample of measurements where the null hypothesis is that the sample is drawn from a distribution with mean. The test looks at the differene between the observed and epeted means saled by the variane of the data and tells us how likely one is to get a sample of that mean and variane assuming that the sample is drawn from a normal distribution with mean. To apply the t-test to olloations we think of the tet orpus as a long sequene of N bigrams.

13 Hypothesis Testing: Formula N = number of bigrams μ = sample mean for H 0 = observed sample mean t μ s N p = probability that the event would our if H 0 were true Signifiane level p < 0.05 means 95% onfidene p < 0.0 means 99% onfidene 3

14 Eample new ompanies olloation or not? w = new w new w = ompanies = 8 = 4667 w ompanies = 580 = Pnew = / Pompanies = / H 0 : Pnew ompanies = Pnew * Pompanies = = μ = 8 / = s = p-p p t => We annot rejet null hypothesis 4

15 Hypothesis testing of differenes Churh & Hanks 989 We may also want to find words whose oourrene patterns best distinguish between two words. This appliation an be useful for leiography. The t-test is etended to the omparison of the means of two normal populations. Here the null hypothesis is that the average differene is 0. 5

16 6 Hypothesis testing of difs. II n s n t s Pw v = Cw v / N Pw v = C w v / N Eample: strong tea vs. powerful tea t w v C w v C w v C w v C

17 t-test for statistial signifiane of the differene between two systems System System sores total n Mean i s i sum ^ ij i^ df

18 t-test for differenes ontinued Pooled s = / = 3.4 t s n For rejeting the hypothesis that System is better then System with a probability level of α = 0.05 the ritial value is t=.75 from statistis table We annot onlude the superiority of System beause of the large variane in sores 8

19 Chi-Square test I: Method Use of the t-test has been ritiized beause it assumes that probabilities are approimately normally distributed not true generally. The Chi-Square test does not make this assumption. The essene of the test is to ompare observed frequenies with frequenies epeted for independene. If the differene between observed and epeted frequenies is large then we an rejet the null hypothesis of independene. 9

20 0 Chi-Square test II: Formula......;...; N X E E E N N N E E E X j i ij ij ij

21 Chi-Square test III: Appliations ne of the early uses of the Chi square test in Statistial NLP was the identifiation of translation pairs in aligned orpora Churh & Gale 99. A more reent appliation is to use Chi square as a metri for orpus similarity Kilgariff and Rose 998 Nevertheless the Chi-Square test should not be used in small orpora.

22 Eample new ompanies olloation or not? w = new w new w = ompanies = 8 = 4667 w = ompanies = 580 = E ij = marginal probabilities = totals of row i and olumn j onverted into proportions = epeted values for independene X =.55 < 3.84 needed for p < 0.05 one degree of freedom for table

23 Likelihood Ratios I: Within a single orpus Dunning 993 Likelihood ratios are more appropriate for sparse data than the Chi-Square test. In addition they are easier to interpret than the Chi-Square statisti. In applying the likelihood ratio test to olloation disovery we eamine the following two alternative eplanations for the ourrene frequeny of a bigram w w: The ourrene of w is independent of the previous ourrene of w The ourrene of w is dependent of the previous ourrene of w 3

24 4 Log likelihood distrib. binomial - b and where log log log log ; ; ; ; log log ; ; ; ; ; ; : : k n k n k L p N L p L p N L p L p N b p b p N b p b H L H L w w C w C w C N p p N p w w P p p w w P H p w w P w w P H

25 Likelihood Ratios II: Between two or more orpora Damerau 993 Ratios of relative frequenies between two or more different orpora an be used to disover olloations that are harateristi of a orpus when ompared to other orpora. This approah is most useful for the disovery of subjet-speifi olloations. 5

26 Mutual Information I An information-theoreti measure for disovering olloations is pointwise mutual information Churh et al Pointwise Mutual Information is roughly a measure of how muh one word tells us about the other. Pointwise mutual information does not work well with sparse data. 6

27 7 Mutual Information II log log log y C C N y C y PMI y P P y P y PMI y P P y P Y X P y MI PMI = E MI

28 Eample PMInew ompanies = = log 8 * / 4675 * 588 =.546 PMIhouse ommons = 4. PMIvideoasette reorder =