Computer Science 786S - Statistical Methods in Natural Language Processing and Data Analysis Page 1

Transcription

1 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 1 Hypothesis Testing A statistial hypothesis is a statement about the nature of the distribution of a random variable. How an suh a hypothesis be tested? Consider the geneti relationships among ertain Indo-European Languages. Ross (1950) onstruted a table as follows: Allot a olumn to eah branh language and a row to eah of a ertain set of attested Indo-European roots; if a root appears in a partiular branh language, put an x in the appropriate ell of the table. Root-number L 1 L 2 L x x 2 x x 3 x x x : Ross suggests that the question Is L i losely related to L j? is equivalent to the question Given the number of rosses in the ith and jth olumns, what is the probability of obtaining the given number (or a greater number) of ases of a row with a ross in eah of the two olumns if the rosses were plaed in the two olumns at random? If this probability is suffiiently small, we might be tempted to infer that the two languages have some asual (geneti) relationship. Let N be the number of rows in the table (number of roots under onsideration), let n i be the number of rows whih ontain a ross in olumn i, let n j be the number of rows whih ontain a ross in olumn j, and let r be the number of rows in whih both the ith and jth olumns are marked with a ross. Our problem is to ompute the probability that r agreeing rows our by hane, given that n i and n j entries in olumn i and olumn j respetively are marked with a ross. We proeed as follows. We first determine the number of ways of marking n i entries in olumn i and n j entries in olumn j with exatly r agreements. The number of ways that n i rosses an be plaed in the ith olumn is N! ni! N n i! whih is the number of ways of seleting a subset of n i objets from a set ontaining N objets. One these have been hosen, we put r rosses in the jth olumn on r of the n i rows already marked in olumn i. This an be done in ni r

2 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 2 ways. Finally we mark at random the N-n i rows whih do not ontain a ross in the ith olumn with the n i -r remaining rosses allotted to the jth olumn. This an be aomplished in N ni nj ways. Thus the entire operation an be done in N ni r nj different ways. Now we must find the number of ways n i rosses an be plaed in olumn i and n i in olumn j without any restrition on agreements. Sine n i of the entries of olumn i an be marked with a ross in ways and nj of the entries of olumn j an be marked with a ross in n j ways, the total number of ways both olumns an be marked is N n j We let the set of outomes be the set of all these possible ways of marking the two olumns with rosses. Sine eah way is as likely to our as any other, the probability of any one suh marking is N n j and sine, as we have seen, N ni r nj ases are favourable to the event of r agreeing markings, the probability p r of r agreements is p r N ni r nj N n j

3 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 3 or p r N ni r nj n j Suppose that we fine s oinident rosses when we onsider the atual ase of Indo-European roots in L i and L j. The probability of at least that many ourring by hane is k Pm s p q q s where k is the smaller of the two numbers n i and n i. If P(mŠs) is small, say 0.05, this means that the probability of obtaining at least as many as s oinident roots by hane is Sine the probability of obtaining s or more ommon roots by hane is small indeed, one is tempted to rejet the hypothesis of random root phenomenon. Thus it would appear that some ausal fator is at work. If, on the other hand, P(mŠs) turned out to be, say 0.5, we would not be tempted to look for ausal fators, for in this ase there is a hane that s or more roots oinide. This sort of argument is basi to hypothesis testing, whih is where we turn our disussion to now. Introdution Ross argument to the effet that the hypothesis that two languages L 1 and L 2, say, are not genetially related is equivalent to the following hypothesis: (H) If, in a set of N Indo-European roots, n 1 have a ognate in L 1 and n 2 a ognate in L 2, then the distribution of the random variable R equal to the number of roots with ognates in both languages is governed by hane alone. We showed that under this hypothesis R has the hyper-geometri distribution, that is PR r n 2 N n 2 r n 1 n 1 To test this hypothesis, we an atually list the N Indo-European roots, find n 1, n 2, and r, and finally ompute P(RŠr), the probability of obtaining at least r ognates by hane. If this probability is small, we would tend to rejet H and assume that L 1 and L 2 are related. If P(RŠr), were not small, then it might be possible that this number of ognates ould have ourred by hane, and no ase ould be made on probabilisti grounds for the two languages to be any more losely related than any other pair of Indo-European languages. [This is a speial ase of Fisher s exat ase (test for independene) and is a very ommonly used test]. A few observations about the situation above may prove helpful. Two kinds of error an be made by a researher: (1) he may rejet the hypothesis when it is in fat true; and (2) he may aept the hypothesis as being true when it is indeed false. Let us see what we an do to ontrol errors of the first kind. 1

4 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 4 For our example, we may rejet the hypothesis H that there is no geneti relationship between L 1 and L 2 when there is indeed no suh relationship. This is an error of type 1. On the other hand we may aept the hypothesis of no geneti relationship when it is false; this is an error of the seond kind or a type 2 error. Of ourse aepting H when it is true and rejeting H when it is false present no problem. In our example we tend to rejet the hypothesis of hane ourrene if the number of shared ognate roots is high.if we were to hoose a ritial value and rejet the hypothesis whenever the observed value r of R is greater than or equal to, then the probability of making a type 1 error is PR PR i i probability of obtaining a value of R as high as under the given hypothesis This probability P(RŠ) of rejeting hypothesis H when it is true an be ontrolled by first hoosing, the ritial level, and then finding so that P(RŠ) Generally is hosen to be small when type 1 errors are espeially hazardous; ritial levels like 0.05, 0.01, and are ommon, so that for example, if [1] PR [2] then rejeting the hypothesis at r Š 0.01 renders the probability of a type 1 error at However, if the observed value r of R were r< 0.01, this would not neessarily mean that we ould automatially assume H to be valid. If indeed P(RŠr) 0.05 this would mean that the probability of obtaining an observed number of ognates at least as high as r is quite small, 0.05, but not as small as the ritial level, There is a distint asymmetry to the situation just desribed. The hypothesis H is an assertion that there is nothing to the laim that L 1 and L 2 are related. Thus in employing 0.01 in expression [2], if we obtain an observed r Š 0.01 and rejet the hypothesis, we are laiming that L 1 and L 2 are related, with a probability of 0.01 that we are rejeting the hypothesis when it is true. On the other hand, if r < 0.01 we are not in a position to aept H without inurring the possibility of an error of type 2. Hene in order to be able to make a ertain laim C about the values of some random variable X, we try to arrange to make the hypothesis H that there is nothing to C and then proeed to test H with an eye to rejeting it, and hene expousing C, if the value observed for X is suffiiently improbable. For this reason H is often alled a null hypothesis beause, as indiated above, it usually amounts to a denial of some partiular laim C. A null hypothesis H onerning a random vari- 1 Type 2 errors are espeially problemati for linguistis researhers for two reasons. First, s/he has less ontrol over them than s/ he does over type 1 errors and seondly, s/he is often in the position where being able to aept the statistial hypothesis yields positive sientifi results. Humanities researhers guard against this situation by forming hypothesis in suh a way that rejetion yields positive results, i.e., in this ase indiates a geneti relationship between the two languages.

5 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 5 able X is usually of the form: X has some partiular distribution F X, the null distribution, from whih suh probabilities as PX P X P X EX [3] [4] [5] an be omputed. Then a ritial or signifiane level is hosen and, depending on the requirements of the problem, a ritial value is obtained from [3], [4], and [5], et., so that the probability in question equals, is the ritial value and is the orresponding level of signifiane or ritial level. Thus a hypothesis test onsists of making the null hypothesis about the distribution of X, hoosing a level of signifiane, hoosing a probability onfiguration like [3], [4], or [5], or something else appropriate for a ritial region, obtaining a ritial value orresponding to for that partiular ritial region, and rejeting or aepting the hypothesis aording as the observed value of X lies in the ritial region or not. If the ritial region is XŠ (as in [3]) or Xð, the test is alled a one-tailed test. If, on the other hand, one wishes to use a ritial region like X Š or X-E(X) Š, the test is alled a two-tailed test. The problem at hand determines whih test is appropriate. Example 1 Ross sheme to test for signifiant relationships between languages is an example of an (upper) one-tailed hypothesis test. It is often ompliated by the amount of omputation required to obtain P(Rr). For example, Ross ites in his table 6, that in N1860 Indo-European roots there are n11184 Italo-Celti ognates and n21165 Greek ognates, and of these r783 are ommon ognates in both languages. This means that to find P(RŠr) in this ase we must ompute PR 783 whih is a prodigious task i 1184 i 1184 i 783 Sine the hyper-geometri distribution has been tabulated, in Lieberman and Owen (1961), for Nð50 and for ertain seleted values of N>50, a areful hoie of N may render Ross test easier to use. Example 2 In Hayden s study of Amerian English (1950), there were 325 ourrenes of j in 65,122 phonemes. If we assume that Robert s survey (1965) is over a suffiiently large orpus that his relative frequeny of j is a good approximation of the probability of j and that both samples are random, then we an make the null hypothesis that both samples ome from the same population for whih the probability of seleting j is p

6 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 6 Hayden s sample an be taken as 65,122 Brownell trials with probability p of obtaining j in eah trial. Thus, if the random variable X i (i1,2,...,65,122) is defined to take the value 1 if the ith phoneme is j and 0 otherwise, then with X Xi i 1 number of ourrenes of j in a phoneme running text the null hypothesis an be stated as follows: X has a binomial distribution with n65,122 and p0.0036, that is PX k k k k Sine npš5, the approximation of this binomial distribution by the normal distribution is adequate. Using this normal approximation, we an write PX k Nknp npq Nk whih aording to an be written Nx N x [A] PX k N k Let us perform a two-tailed test by hoosing the ritial value for some deviation of X from E(X) orresponding to some signifiane level, that is, let us hoose suh that P X EX [B] The expression X EX [C] is equivalent to the disjuntion of and EX + X X EX [D] [E] that is, [C] holds if and only if either [D] or [E] holds. In the present ase, the events [D] and [E] are disjoint events for >0, so P X EX PX EX + PX EX [F]

7 Computer Siene 786S - Statistial Methods in Natural Language Proessing and Data Analysis Page 7 1 PX EX + + PX EX N sine E(X) np and X Using [A], we an write P( X-E(X) Š) as follows: Beause the distribution is symmetri with respet to the vertial oordinate axis, then N(- x,0,1)1-n(x,0,1). Thus If we set the signifiane level at 0.01, we must hoose a so that P( X-E(X) Š)0.01. From a table of he normal distribution, we an obtain the result N(2.58,0,1)0.995, so that the right side of [H] equals Thus if / or 15.28* in expression [H], we have P( X Š39.42) Hayden s sample yields the value 325 for X and hene X , whih is well beyond the ritial value. Therefore in rejeting the hypothesis that X has binomial distribution with p0.0036, we have a probability of less than 0.01 of being wrong, that is, of ommitting an error of type 1. Thus there is overwhelming evidene that X is not binomially distributed with probability p This is probably beause, on the one hand, the X i s may not be independent (that is, Hayden s sample may not be random) and, on the other, Hayden s sample is a very speialized one with perhaps a different value of p. In general, when a null hypothesis takes the form the random variable X has normal distribution with mean and variane 2, then in terms of expression [5] the equation relating the ritial value to the level of signifiane is Referenes P X EX 1 N N [G] P X EX 2 1 N [H] P X EX 2 1 N Ross, A.S.C. (1950). Philologial probability problems. J. Roy. Stat. So, ser. B, 12, Lieberman, G.J., and Owen, D.B. (1961). Tables of hypergeometri probability distribution (Palo Alto: Stanford University Press). Hayden, R.E. (1950). The relative frequeny of phonemes in general-amerian English. Word 6, Roberts, A.H. (1965). A Statistial Linguisti Analysis of Amerian English (The Hague: Mouton).