Statistical NLP: Lecture 7. Collocations. (Ch 5) Introduction

Transcription

1 Statistical NLP: Lecture 7 Collocations Ch 5 Introduction Collocations are characterized b limited compositionalit. Large overlap between the concepts of collocations and terms, technical term and terminological phrase. Collocations sometimes reflect interesting attitudes in English towards different tpes of substances: strong cigarettes, tea, coffee versus powerful drug e.g., heroin

2 Definition w.r.t Computational and Statistical Literature [A collocation is defined as] a sequence of two or more consecutive words, that has characteristics of a sntactic and semantic unit, and whose eact and unambiguous meaning or connotation cannot be derived directl from the meaning or connotation of its components. [Chouekra, 988] 3 Other Definitions/Notions w.r.t. Linguistic Literature Collocations are not necessaril adjacent Tpical criteria for collocations: noncompositionalit, non-substitutabilit, nonmodifiabilit. Collocations cannot be translated into other languages. Generalization to weaker cases strong association of words, but not necessaril fied occurrence. 4

3 Linguistic Subclasses of Collocations Light verbs: verbs with little semantic content Verb particle constructions or Phrasal Verbs Proper Nouns/Names Terminological Epressions 5 Overview of the Collocation Detecting Techniques Surveed Selection of Collocations b Frequenc Selection of Collocation based on Mean and Variance of the distance between focal word and collocating word. Hpothesis Testing Mutual Information 6

4 Frequenc Justeson & Katz, 995. Selecting the most frequentl occurring bigrams. Passing the results through a part-ofspeech filter Simple method that works ver well. 7 Mean and Variance I Smadja et al., 993 Frequenc-based search works well for fied phrases. However, man collocations consist of two words in more fleible relationships. The method computes the mean and variance of the offset signed distance between the two words in the corpus. If the offsets are randoml distributed i.e., no collocation, then the variance/sample deviation will be high. 8

5 Mean and Variance II n = number of times two words collocate µ= sample mean d i = the value of each sample Sample deviation: s = n i i= n d µ 9 Eample of fleible collocation knocked on the door 3 knocked at the door 3 knocked on John s door 5 knocked on the metal front door 5 µ = /4 = 4 s = sqrt / 3 =.5 If s is big => no collocation If µ is not zero => fleible collocation 0

6 Hpothesis Testing: Overview High frequenc and low variance can be accidental. We want to determine whether the cooccurrence is random or whether it occurs more often than chance. This is a classical problem in Statistics called Hpothesis Testing. We formulate a null hpothesis H 0 no association - onl chance and calculate the probabilit p that a collocation would occur if H 0 were true, and then reject H 0 if p is too low. Otherwise, retain H 0 as possible. Hpothesis Testing: The t-test The t-test looks at the mean and variance of a sample of measurements, where the null hpothesis is that the sample is drawn from a distribution with mean µ. The test looks at the difference between the observed and epected means, scaled b the variance of the data, and tells us how likel one is to get a sample of that mean and variance assuming that the sample is drawn from a normal distribution with mean µ. To appl the t-test to collocations, we think of the tet corpus as a long sequence of N bigrams.

7 Hpothesis Testing: Formula N = number of bigrams µ = sample mean for H 0 = observed sample mean t µ = p = probabilit that the event would occur if H 0 were true s N Significance level p < 0.05 means 95% confidence p < 0.0 means 99% confidence 3 Eample new companies collocation or not? w = new w new w = companies O = 8 O = 4667 w companies O = 580 O = Pnew = / Pcompanies = / H 0 : Pnew companies = Pnew * Pcompanies = = µ = 8 / = s = p-p p t = = <.576 => We cannot reject null hpothesis

8 Hpothesis testing of differences Church & Hanks, 989 We ma also want to find words whose cooccurrence patterns best distinguish between two words. This application can be useful for leicograph. The t-test is etended to the comparison of the means of two normal populations. Here, the null hpothesis is that the average difference is 0. 5 Hpothesis testing of difs. II t = s s + n n Pw v = Cw v / N Pw v = C w v / N Eample: strong tea vs. powerful tea t = C w v C w v C w v + C w v 6

9 t-test for statistical significance of the difference between two sstems Sstem Sstem scores 7,6,55,60,68,49, 4,7,76,55,64 total n Mean i si ^ = sum ij - i^ df 0 0 4,55,75,45,54,5, 55,36,58,55,67 7 t-test for differences continued Pooled s = / = 3.4 t = s n = = For rejecting the hpothesis that Sstem is better then Sstem with a probabilit level of α = 0.05, the critical value is t=.75 from statistics table We cannot conclude the superiorit of Sstem because of the large variance in scores 8

10 Chi-Square test I: Method Use of the t-test has been criticized because it assumes that probabilities are approimatel normall distributed not true, generall. The Chi-Square test does not make this assumption. The essence of the test is to compare observed frequencies with frequencies epected for independence. If the difference between observed and epected frequencies is large, then we can reject the null hpothesis of independence. 9 Chi-Square test II: Formula X E E X = i, j=, O + O = N =...; E = O O ij + O E O + O N =...; E E ij ij N OO O + O N =... O O O + O O + O 0

11 Chi-Square test III: Applications One of the earl uses of the Chi square test in Statistical NLP was the identification of translation pairs in aligned corpora Church & Gale, 99. A more recent application is to use Chi square as a metric for corpus similarit Kilgariff and Rose, 998 Nevertheless, the Chi-Square test should not be used in small corpora. Eample new companies collocation or not? w = new w new w = companies O = 8 O = 4667 w = companies O = 580 O = E ij = marginal probabilities = totals of row i and column j converted into proportions = epected values for independence X =.55 < 3.84 needed for p < 0.05, one degree of freedom for table

12 Likelihood Ratios I: Within a single corpus Dunning, 993 Likelihood ratios are more appropriate for sparse data than the Chi-Square test. In addition, the are easier to interpret than the Chi-Square statistic. In appling the likelihood ratio test to collocation discover, we eamine the following two alternative eplanations for the occurrence frequenc of a bigram w w: The occurrence of w is independent of the previous occurrence of w The occurrence of w is dependent of the previous occurrence of w 3 Log likelihood H : P w H : P w c p = ; p N w = P w w = p L H b c; c, p b c logλ = log = L H b c ; c, p b c = logl c logl c w = p, c, p + logl c, c, p logl c k wherel k, n, = c = ; p c p = P w n k c c = ; c N c w = C w ; c = C w ; c c; N c, p c ; N c, p c, N c, p and b - binomial distrib. c, N c, p = C ww ; 4

13 Likelihood Ratios II: Between two or more corpora Damerau, 993 Ratios of relative frequencies between two or more different corpora can be used to discover collocations that are characteristic of a corpus when compared to other corpora. This approach is most useful for the discover of subject-specific collocations. 5 Mutual Information I An information-theoretic measure for discovering collocations is pointwise mutual information Church et al., 89, 9 Pointwise Mutual Information is roughl a measure of how much one word tells us about the other. Pointwise mutual information does not work well with sparse data. 6

14 7 Mutual Information II, log,, log,, log,, C C N C PMI P P P PMI P P P Y X P MI = = = PMI = E MI 8 Eample PMInew, companies = = log 8 * / 4675 * 588 =.546 PMIhouse, commons = 4. PMIvideocasette, recorder = 5.94