Ted Pedersen. Southern Methodist University. large sample assumptions implicit in traditional goodness

Transcription

1 Appears in the Proceedings of the Soth-Central SAS Users Grop Conference (SCSUG-96), Astin, TX, Oct 27-29, 1996 Fishing for Exactness Ted Pedersen Department of Compter Science & Engineering Sothern Methodist University Dallas, TX Abstract Statistical methods for atomatically identifying dependent word pairs (i.e. dependent bigrams) in a corps of natral langage text have traditionally been performed sing asymptotic tests of signicance. This paper sggests that Fisher's exact test is a more appropriate test de to the skewed and sparse data samples typical of this problem. Both theoretical and experimental comparisons between Fisher's exact test andavariety of asymptotic tests (the t-test, Pearson's chi-sqare test, and Likelihood-ratio chi-sqare test) are presented. These comparisons show that Fisher's exact test is more reliable in identifying dependent word pairs. The seflness of Fisher's exact test extends to other problems in statistical natral langage processing as skewed and sparse data appears to be the rle in natral langage. The experiment presented in this paper was performed sing PROC FREQ of the SAS System. Introdction De to advances in compting power and the increasing availability of large amonts of on-line text the empirical stdy of hman langage has become an increasingly active area of research in both academic and commercial environments. Statistical natral langage processing (NLP) relies pon stdying large bodies of text called corpora. These methods are sefl becase the naided hman mind simply cannot notice all important lingistic featres, let alone rank them in order of importance, when dealing with large amonts of text. The diclty in stdying hman langage based pon samples from text is that despite having billions of words on-line it is still diclt to collect large samples of certain events as many lingistic events very rarely occr. Many featres of hman langage adhere to Zipf's Law (Zipf 1935). Informally stated, this law says that most events occr rarely and that a few very common events occr most of the time. Statistical NLP mst inevitably deal with a large nmber of rare events. Typical NLP data violates the This research was spported by the Oce of Naval Research nder grant nmber N large sample assmptions implicit in traditional goodness of t tests sch aspearson's X 2, the Likelihood ratio G 2 and the t-test. When this occrs the reslts obtained from these tests may be in error. An alternative to these statistics is Fisher's exact test, which assigns signicance by exhastively compting all probabilities for a contingency table with xed marginal totals. This paper presents an experiment that compares the eectiveness of X 2, G 2, the t-test and Fisher's exact test in identifying dependent bigrams. When these reslts are dierent it is shown why Fisher's exact test gives the most reliable signicance vale. All of these tests can be conveniently performed sing the SAS System (SAS Institte 1990). Lexical Relationships A common problem in NLP is the identication of strongly associated word pairs. A bigram is any two consective words that occr together in a text. The freqency with which a bigram occrs throghot a text says something abot the relationship between the words that make p the bigram. A dependent bigram is one where the two words are related in some way other than what wold be expected prely by chance. Intitively appealing examples of dependent bigrams inclde major leage, sothern baptist and ne wine. The challenge in identifying dependent bigrams is that most bigrams are relatively rare regardless of the size of the text. This follows from the the distribtional tendencies of individal words and bigrams as described in Zipf's Law (Zipf 1935). Zipf fond that if the freqencies of the words in a large text are ordered from most to least freqent, (f 1 f 2 ::: f m ), these freqencies roghly obey: f i / 1 i. The implications of Zipf's law are two-sided for statistical NLP. The good news is that a signicant proportion of a corps is made p of the most freqent words these occr freqently enogh to collect reliable statistics on them. The bad news is that there will always be a large nmber of words that occr jst a few times. As an example, in a 133,000 word sbset of the

2 % 50 % freqency freqency Figre 1: Distribtion of single words Figre 2: Distribtion of bigrams ACL/DCI Wall Street Jornal corps (Marcs et al. 1993) there are 14,319 distinct words and 73,779 distinct bigrams. Of the distinct words, 48 percent of them occr only once and 80 percent of them occr ve times or less. Of the distinct bigrams, 81 percent occr once and 97 percent of them occr ve times or less. This data is represented graphically in Figres 1 and 2. As a reslt of these distribtional tendencies, data samples characterizing specic bigrams are terribly skewed. This kind of data violates the large sample assmptions regarding the distribtional characteristics of a data sample that are made by asymptotic signicance tests. Representation of the Data To represent the data in terms of a statistical model, the featres of each object are mapped to random variables. The relevant featres of a bigram are the two words that form the bigram. If each bigram in the data sample is characterized by two featres represented by the binary variables X and Y, then each bigram will have one of for possible classications corresponding to the possible combinations of these variable vales. In this case, the data is said to be cross-classied with respect to the variables X and Y. The freqency of occrrence of these classi- cations can be shown in a sqare table having 2 rows and 2 colmns. The freqency conts of each ofthe4 possible data classications in Figre 3 are denoted by n 11, n 12, n 21, and n 22. The joint freqency distribtion of X and Y is described by the conts fn ij g for the data sample represented in the contingency table. The marginal distribtions of X and Y are the row and colmn (eqation 1) totals obtained by smming the joint freqencies. The row variable is denoted n i+ and the colmn variable n +j. The sbscript + indicates the index over which smming has occrred. n i+ = JX j=1 n ij n +j = IX i=1 n ij (1) More generally, if there are I possible vales for the rst variable and J possible vales for the second variable, then the freqency of each classication can be recorded in a rectanglar table having I rows and J colmns. Each cell of this table represents one of the I J possible combinations of the variable vales. Sch a table is called an I J contingency table. X Y indstry :indstry totals oil n 11 = 17 n 12 = 229 n 1+ = 246 :oil n 21 =935 n 22 = n 2+ = totals n +1 =952 n +2 = n ++ = Figre 3: Contingency Table As shown in Figre 3, in order to stdy the association (i.e., degree of dependence) between the words oil and indstry, the variable X is sed to denote the presence or absence of oil in the rst position of each bigram, and Y is sed to denote the presence or absence of indstry in the second position. Signicance Testing In both exact and asymptotic signicance testing, a probabilistic model is sed to describe the distribtion of the poplation from which the data sample

3 was drawn. The acceptability of a potential poplation model is postlated as a nll hypothesis and that hypothesis is tested by evalating the t of the model to the data sample. The t is considered acceptable if the model diers from the data sample by an amont consistent with sampling variation, that is, if the vale of the metric measring the t of the model is statistically signicant. The steps involved in performing a signicance test are listed below and discssed in the sbsections that follow. Both exact and asymptotic signicance tests follow steps 1 and Select an appropriate sampling plan, 2. hypothesize a poplation model, 3. select a smmary statistic to se in testing the t of the hypothesized model to the sampled data, and 4. assess the statistical signicance of the model: determine the probability that the data came from a poplation described by the model. Steps 3 and 4 are more commonly associated with an asymptotic signicance test. An exact test does not se a goodness of t statistic bt the notion of assessing signicance still remains. The dierences between the two approaches will be discssed in more detail shortly. Sampling Plan In order for a signicance test to yield valid reslts the data mst be collected from the poplation via a random sampling plan. The sampling plan assres that each object in the data sample is selected via independent and identical trials. The sampling plan together with the poplation characteristics can be sed to de- ne the likelihood of selecting any particlar sample. In the experiment for this paper a mltinomial sampling plan was sed. In mltinomial sampling the overall sample size n ++ is determined in advance and each object is randomly selected from the poplation to be stdied. Given this, the probability of observing a particlar freqency distribtion fn ij g in a randomly selected data sample is shown in eqation (2), where the p ij 's are the poplation characteristics specifying the probability of classication (i j). P (fn ij g)= Q I i=1 n ++! Q J j=1 n ij! IY JY i=1 j=1 p nij ij (2) The data in Figre 3 was sampled sing a mltinomial sampling plan. This data is sed to test the bigram oil indstry for association. When this data was sampled, the only vale that was xed prior to the beginning of the experiment was the total sample size, n ++, which was eqal to 1,382,828. Hypothesizing a Model The poplation model sed to stdy association between two words, where the two words are represented by the binary variables X and Y, is the model for independence between X and Y: P (x y) =P (x)p (y) (3) If the model for independence ts the data well as measred by its statistical signicance, then one can infer from this data sample that these two words are independent in the larger poplation. The worse the t, the more dependent the words are jdged to be. Using the notation introdced previosly, the parameters of the model for independence between two words (i.e., the words oil and indstry in Figre 3) are estimated as follows: P (x) = n i+ n ++ P (y) = n +j n ++ (4) In signicance testing, the poplation model is the nll hypothesis that is tested. This hypothesis can only be rejected or accepted, it can not be proven tre or false with absolte certainty. The signicance assigned to the hypothesis indicates how likely it is that the sample was drawn from a poplation specied by that model. Goodness of Fit Statistics A goodness of t statistic is sed to measre how closely the conts observed in a data sample correspond to those that wold be expected in a random sample drawn from a poplation where the nll hypothesis is tre. In this section, we discss three metrics that have been sed to measre the t of the models for association: the likelihood ratio statistic G 2,Pearson's X 2 statistic and the t-statistic. The distribtion of each of these statistics can be approximated when the hypothesis is tre and certain other conditions hold they therefore can be sed in asymptotic signicance testing. In Fisher's exact test a goodness of t statistic is not employed. G 2 and X 2 These statistics measre the divergence of observed (n ij ) and expected (m ij ) sample conts, where the expectation is based on a hypothetical poplation model. These statistics can be conveniently compted sing PROC FREQ of the SAS System. The rst step in calclating either G 2 or X 2 is to calclate the expected conts given that the hypothetical poplation model is correct. In the model for independence, maximm likelihood estimates of the expected conts are formlated as in eqation (5) where m ij denotes the expected cont incontingency table cell (i j). m ij = n i+n +j n ++ (5) Using this formlation, G 2 and X 2 are calclated as:

4 X G 2 =2 n ij log n X ij X 2 (n ij ; m ij ) = 2 m i j ij m i j ij (6) When the hypothetical poplation model is the tre poplation model, the distribtion of both G 2 and X 2 converges to 2 as the sample size grows large (i.e., the 2 distribtion is an asymptotic approximation for the distribtions of G 2 and X 2 ). More precisely, X 2 and G 2 are approximately 2 distribted when the following conditions regarding the random data sample hold (Read and Cressie 1988): 1. the sample size is large, 2. the nmber of cells in the contingency table representation of the data is xed and small relative to the sample size, and 3. the expected cont (nder the hypothetical poplation model) for each cell is large. (Dnning 1993) shows that G 2 holds more closely to the 2 distribtion than does X 2 when dealing with bigram data. However, as pointed ot in (Read and Cressie 1988), it is ncertain whether G 2 holds to the 2 distribtion when the minimm of the expected vales in a table is less than 1.0. Since low expected freqencies appear to be the rle in bigram data (e.g. colmn m 11 in Figre 8) we sggest that the reliability of the 2 approximation to G 2 cold be in qestion. the t-statistic The t-statistic (eqation 7) measres the dierence between the mean of a randomly drawn sample (x) and the hypothesized mean for the poplation from which that sample was drawn ( 0 ). This dierence is scaled by thevariance of the poplation. When the variance of the poplation is nknown and the sample size is large, standard statistical techniqes allow that the poplation variance can be estimated by the sample variance (s 2 ) which is in trn scaled by the sample size (n). t = x ; 0 q s 2 n (7) (Chrch et al. 1991) show how the t-statistic can be sed to identify dependent bigrams. The data sample is prodced throgh a series of Bernolli trials that record the presence or absence of a single bigram. Given this sampling plan the sample mean is dened to be the relative freqency of the bigram ( n++ n11 ) and the sample variance is roghly approximated by that same relative freqency. The t-statistic can then be rewritten as in eqation 8. t n11 n++ ; n1+n+1 n++ q 2 n11 n++ 2 = n11 ; m11 p n11 (8) In the t-test, signicance is assigned to the t-statistic sing the t-distribtion, which is eqal to the standard normal distribtion in large sample experiments. This approach to assigning signicance is based on the assmption that the sample means are normally distribted. This assmption is shown to be inappropriate for bigram data in (Dnning 1993). The formlation of (Chrch et al. 1991) is eqivalent to a one-sample t-test. PROC TTEST of the SAS System comptes a two sample t-test and was not sed to compte the t-statistic vales. Instead a separate data step was created to calclate the vale in eqation 8 and signicance was assigned to that vale sing the PROBT fnction. Assessing Statistical Signicance If the test statistic sed to evalate a model has a known distribtion when the model is correct, that distribtion can be sed to assign statistical signicance. For X 2 and G 2 the 2 distribtion is sed while the t-test ses the t-distribtion. These serve as reliable approximations of distribtions of the test statistics when certain assmptions hold. However, as has been pointed ot, these assmptions are freqently violated in bigram data. An alternative to sing a signicance test based on an approximate distribtion is to se an exact signicance test. In particlar, for bigram data Fisher's exact test is recommended. This test can be performed sing PROC FREQ in the SAS System. Fisher's Exact Test Rather than sing an asymptotic approximation of the signicance of observing a particlar table, Fisher's exact test (Fisher 1935) comptes the signicance of an observed table by exhastively compting the probability ofevery table that wold lead to the marginal totals that were observed in the sampled table. The signicance vales obtained sing Fisher's exact test are reliable regardless of the distribtional characteristics of the data sample. However, when the nmber of comparable data samples is large, the exhastive enmeration performed in Fisher's exact test becomes infeasible. In (Pedersen et al. 1996) an alternative test, the exact conditional test, is discssed for tables where Fisher's exact test is not a practical option. When performing Fisher's exact test in a 2 2contingency table the marginal totals n 1+ and n +1 and the sample size n ++ are xed at their observed vale. Given this, the vale of n 11 determines the conts in n 12, n 21 and n 22. All of the possible 2 2 tables that adhere to the xed marginal totals are generated and the probability of each table is compted sing the hypergeometric distribtion. Given that all the marginals and the sample size is xed the hypergeometric probability of observing a particlar freqency distribtion fn 11 n 12 n 21 n 22 g can be compted sing eqation 9. Hypergeometric

5 probabilities can be compted with the SAS System sing the PROBHYPR fnction. PROBHYPR was sed to compte the individal table probabilities the athor added to the PROC FREQ otpt in the appendix and the data plotted in Figre 5. P = 1 n 11!n 12!n 21!n 22! n 1+!n 2+!n +1!n +2! n ++! (9) The original problem that Fisher sed to present this test has gone down in statistical lore as the Tea Drinker's Problem. A woman claimed that by tasting a cp of tea with milk she cold determine if the milk or tea had been added rst. An experiment was designed where eight cps of tea were mixed, for cps with the milk added rst and for with the tea added rst. The eight cps of tea were presented to the woman in random order and she was asked to divide the 8 cps into two sets of 4, one set being those cps where milk was added rst and the other those where tea was added rst. Given that the lady knew that there were 8 cps of tea and that 4 of the cps had the tea added rst and 4 had the milk added rst it is clear that all marginal totals shold be xed at 4 and the sample size xed at 8. Given this there are 5 possible otcomes to the experiment (n 11 = 0, 1, 2, 3, and 4). Figre 5 shows the distribtions of the hypergeometric probabilities associated with those 5 possible tables. This problem can be represented sing a 2 2 contingency table where variable X represents the actal order of mixing and Y represents the order determined by the tea drinker. The 5 possible contingency tables and associated test vales as generated by PROC FREQ are shown in the appendix. Interpreting the Tea Drinker's Problem The probabilities that reslt from Fisher's exact test indicate how likely it is that the observed table was drawn from a poplation where the nll hypothesis is tre. In other words, the test indicates how likely it wold be to randomly sample a table more spportive of the nll hypothesis than the observed table. In the Tea Drinker's Problem the nll hypothesis is that the tea drinker is gessing and does not really know if the milk or tea was added rst. This is the hypothesis of independence and is the same hypothesis sed in the test for association that identies dependent bigrams. Fisher's exact test can be interpreted as a one sided or two sided test. PROC FREQ shows all of the possible reslts: two-sided, right-sided and left-sided. A one sided test can be either right or left sided. A right sided exact test is compted by smming the hypergeometric probabilities of all the tables with xed marginal totals n 1+ and n +1 whose cell cont inn 11 is greater than or eqal to the observed table. As an example consider the Tea Drinker's Problem where n 11 = 3. This implies that the tea drinker fond 3 of the P 1 0:9 0:8 0:7 0:6 0:5 0:4 0:3 0:2 0: n 11 Figre 5: n ++ =8,n 1+ =4,n +1 =4 4 cps where milk had been added rst. To compte the signicance of the right sided test the probabilities of the tables where n 11 = 3 (.229) and n 11 = 4 (.014) are smmed. In this interpretation it is determined how likely it is that the tea drinker cold perform more accrately in the experiment ifitwas repeated. A right sided test shows how likely it wold be to randomly sample a table where n 11 is greater than or eqal to the observed vale when sampling from a poplation where the nll hypothesis is tre. The probability of being more accrate (i.e. the right sided probability) is.243 which leads to the conclsion that she is not gessing and has some idea of whether the milk or tea was added rst. The left sided test is compted in the same fashion, except that it sms the probabilities of the tables where the cont inn 11 is less than or eqal to the observed vale. Using the same example where n 11 =3then the probabilities of the tables where n 11 = 3 (.229), n 11 = 2 (.514), n 11 = 1 (.229), and n 11 = 0 (.014) are smmed reslting in a left sided vale of.986. The left sided test tells how likely itwold be for the tea drinker to perform less accrately in the same experiment ifit was repeated. Again, this is a fairly strong indication that the tea drinker is not simply gessing. The two sided exact test is compted by smming the hypergeometric probabilities of all tables with the same xed marginals bt whose probabilities are less than or eqal to the probability of the observed table. Consider yet again the case where n 11 = 3. The probability of this table is.229. The tables that have a probability less than this are those where n 11 =1 (.229), n 11 = 0 (.014) and n 11 = 4 (.014). The sm of these probabilities is.486 which is the reslt of the two

6 1 1 0:9 0:9 0:8 0:8 0:7 0:7 0:6 0:6 P 0:5 P 0:5 0:4 0:4 0:3 0:2 0:3 0:2 0: n 11 0: n 11 Figre 6: n ++ = 10000, n 1+ =20,n +1 =10 Figre 7: n ++ = 10000, n 1+ =5000,n +1 =10 sided exact test. The qestion answered here is how probable wold it be for the tea drinker to gess less accrately than was observed. In more general terms this is asking the qestion how likely wold it be to randomly sample a table where the probability ofob- serving a table where n 11 was eqal to or less than the observed vale when sampling from a poplation where the hypothesis of independence is tre. On the srface this is a less convincing demonstration of the tea drinker's skill, however, it still provides reasonable evidence to spport the conclsion that the tea drinker is not gessing. Interpreting the Bigram Experiment The sample sizes in the bigram data are qite a bit larger than those in the Tea Drinker's Problem. In addition, the bigram data is mch more skewed. However, Fisher's exact test remains a practical option since the nmber of possible tables is bond by the smallest marginal total (i.e, the smallest row or colmn total) which for bigram data is associated with the overall cont of the rst or second word in the bigram. In the bigram data it wold be more typical to nd a sample size of 10,000 where n 1+ =20andn +1 =10. This implies that the row totals n +1 and n +2 are 10 and 9,990 respectively. In this case there are 11 possible tables where n 11 wold range from 0 to 10. The distribtion of hypergeometric probabilities for these 11 possible tables is shown in Figre 6. The practical eect of the skewed distribtion shown in Figre 6 is that the right sided and two sided exact test for association are eqivalent. The right-sided exact test vale is the probability of observing a table with n 11 greater than or eqal to the observed. The two-sided exact test vale is the probability ofobserv- ing a table with a probability vale less than or eqal to the observed. These are identical since the vale of P(n 11 ) decreases as n 11 increases. For example, if n 11 > 2 the probability of observing a table with n 11 2 eqals.000. Ths, if n 11 is greater than 2 then there is no probability that the two words in the bigram are independent. They mst be related. Notice however that if the observed cont isanything bt0avery small probability of independence is observed. Sch skewed probabilities are observed for tables where n ++ n 1+ and n ++ n +1. These sorts of tables are what was observed with the bigram data. Notice that this was not the case in the Tea Drinker's Problem and wold not be the case when the row totals are closer or eqal in vale. Consider the example shown in Figre 7wheren ++ = 10000, n 1+ = 5000 and n +1 = 10. It is easy to see that in this case n 1+ = n 2+. In this case the distribtion of the hypergeometric probabilities is symmetric and the right and two-sided tests are dierent. In the test for association the marginal row totals n 1+ and n 2+ are never very close in vale. n 1+ conts how many times the rst word in the bigram occrs in the text overall while n 2+ is the cont of all the other potential rst words in the bigram. Since n 1+ will always be mch less than n 2+ the distribtion of the hypergeometric probabilities will always be very skewed. In the test for association to determine bigram dependence Fisher's exact test is interpreted as a leftsided test. This shows how probable wold it be to see the observed bigram a fewer nmber of times in

7 another random sample from a poplation where the hypothesis of independence is tre. If this probability is high then the words that form the bigram are dependent. Experiment: Test for Association 1 There are two fndamental assmptions that nderly asymptotic signicance testing: (1) the data mst be collected via a random sampling of the poplation nder stdy, and (2) the sample mst exhibit certain distribtional characteristics. If either of these assmptions is not met then the inference procedre may not provide reliable reslts. In this experiment, we compare the signicance vales compted sing the t-test, the 2 approximation to the distribtion of both G 2 and X 2, and Fisher's exact test (left sided). Or data sample is a 1,382,828 word sbset of the ACL/DCI Wall Street Jornal corps. We chose to characterize the associations established by the word indstry as shown in bigrams of the form <word> indstry. In Figre 8, we display a sbset of 24 bigrams and their associated test reslts. As can be seen in Figre 8, there are dierences in the signicance vales assigned by the varios tests. This indicates that the assmptions reqired by certain of these tests are being violated. When this occrs, the signicance vales assigned sing Fisher's exact test shold be regarded as the most reliable since there are no restrictions on the natre of the data reqired by this test. Figre 8 displays the signicance vale assigned to the test for association between the word shown in colmn one and indstry. A signicance vale of.0000 implies that this data shows no evidence of independence. The likelihood of having randomly selected this data sample from a poplation where these words were independent is zero. This is an indication of a dependent bigram. A signicance vale of 1.00 wold indicate that there is an exact t between the sampled data and the model for independence there is no reason to dobt that this sample was drawn from a poplation in which these two words are independent. In this case the bigram is considered independent. In this gre, we show the relative rankings of the bigrams according to their signicance vales. The most independent bigram is rank 1 and the most dependent bigram is rank 24. Note that the rankings dened sing Fisher's exact test and the 2 approximation to G 2 are identical as are the rankings as determined by the 2 approximation to X 2 and the t-test. Notice frther that the signicance vales assigned by Fisher's exact test are similar to the vales as assigned by the 2 approximation to G 2 for the most dependent bigrams. However, there is some variation between the significance compted for Fisher's test and G 2 among the more independent bigrams. This conrms the observation made by (Dnning 1993) that G 2 tends to overstate independence. This indicates that the asymptotic approximation of G 2 bythe 2 distribtion is breaking down for those bigrams. In this case Fisher's test provides a more reliable signicance vale. The signicance vales assigned sing the 2 approximation to Pearson's X 2 and the t-test are very dierent from those assigned by Fisher's exact test. This indicates that neither X 2 nor the t-statistic is holding to its assmed asymptotic approximation. Conclsions In this paper we examined recent work in identifying dependent bigrams. This work has sed asymptotic signicance tests when exact ones wold have been more appropriate. When asymptotic methods are sed there are reqirements regarding both the sampling plan and the distribtional characteristics of the data that mst be met. If the distribtional reqirements are not met, as is freqently the case in NLP, then Fisher's exact test is a viable alternative to asymptotic tests of signicance. The SAS system allows for convenient comptation of Fisher's exact test sing PROC FREQ. References Chrch, K. Gale, W. Hanks, P. and Hindle, D Using statistics in lexical analysis. In Zernik, U., editor 1991, Lexical Acqisition: Exploiting On-Line Resorces to Bild a Lexicon. Lawrence Erlbam Associates, Hillsdale, NJ. Dnning, T Accrate methods for the statistics of srprise and coincidence. Comptational Lingistics 19(1):61{74. Fisher, R The Design of Experiments. Oliver and Boyd, London. Marcs, M. Santorini, B. and Marcinkiewicz, M Bilding a large annotated corps of English: The Penn Treebank. Comptational Lingistics 19(2):313{330. Pedersen, T. Kayaalp, M. and Brce, R Signicant lexical relationships. In Proceedings of the 13th National Conference on Articial Intelligence (AAAI-96), Portland, OR. 455{460. Read, T. and Cressie, N Goodness-of-t Statistics for Discrete Mltivariate Data. Springer-Verlag, New York, NY. SAS Institte, Inc SAS/STAT User's Gide, Version 6. SAS Institte Inc., Cary, NC. Zipf, G The Psycho-Biology of Langage. Hoghton Miin, Boston, MA. 1 Please contact the athor at pedersen@seas.sm.ed if yo wold like acopy of the sorce code and data.

8 <word> n 11 m 11 exact rank G 2 ~ 2 rank X 2 ~ 2 rank T rank and services nancial domestic motor recent instance engineering tility sgar glass in food newspaper drg to appliance movie of ftres the oil airline an Figre 8: test for association : <word> indstry

9 Appendix: PROC FREQ otpt for Tea Drinker's Problem The SAS System TABLE OF X BY Y Y ** Freqency Expected Deviation Percent Row Pct Col Pct milk tea Total milk X ** tea Total STATISTICS FOR TABLE OF X BY Y Statistic DF Vale Prob Chi-Sqare Likelihood Ratio Chi-Sqare Continity Adj. Chi-Sqare Mantel-Haenszel Chi-Sqare Fisher's Exact Test (Left) (Right) (2-Tail) P(n11 = 4) ** Phi Coefficient Contingency Coefficient Cramer's V Sample Size = 8 WARNING: 100% of the cells have expected conts less than 5. Chi-Sqare may not be a valid test. **: Added by the athor. Not a part of PROC FREQ otpt

10 The SAS System TABLE OF X BY Y Y ** Freqency Expected Deviation Percent Row Pct Col Pct milk tea Total milk X** tea Total STATISTICS FOR TABLE OF X BY Y Statistic DF Vale Prob Chi-Sqare Likelihood Ratio Chi-Sqare Continity Adj. Chi-Sqare Mantel-Haenszel Chi-Sqare Fisher's Exact Test (Left) (Right) (2-Tail) P(n11 = 3) ** Phi Coefficient Contingency Coefficient Cramer's V Sample Size = 8 WARNING: 100% of the cells have expected conts less than 5. Chi-Sqare may not be a valid test. **: Added by the athor. Not a part of PROC FREQ otpt

11 The SAS System TABLE OF X BY Y Y ** Freqency Expected Deviation Percent Row Pct Col Pct milk tea Total milk X** tea Total STATISTICS FOR TABLE OF X BY Y Statistic DF Vale Prob Chi-Sqare Likelihood Ratio Chi-Sqare Continity Adj. Chi-Sqare Mantel-Haenszel Chi-Sqare Fisher's Exact Test (Left) (Right) (2-Tail) P(n11 = 2) ** Phi Coefficient Contingency Coefficient Cramer's V Sample Size = 8 WARNING: 100% of the cells have expected conts less than 5. Chi-Sqare may not be a valid test. **: Added by the athor. Not a part of PROC FREQ otpt

12 The SAS System TABLE OF X BY Y Y ** Freqency Expected Deviation Percent Row Pct Col Pct milk tea Total milk X** tea Total STATISTICS FOR TABLE OF X BY Y Statistic DF Vale Prob Chi-Sqare Likelihood Ratio Chi-Sqare Continity Adj. Chi-Sqare Mantel-Haenszel Chi-Sqare Fisher's Exact Test (Left) (Right) (2-Tail) P(n11 = 1) ** Phi Coefficient Contingency Coefficient Cramer's V Sample Size = 8 WARNING: 100% of the cells have expected conts less than 5. Chi-Sqare may not be a valid test. **: Added by the athor. Not a part of PROC FREQ otpt

13 The SAS System TABLE OF X BY Y Y ** Freqency Expected Deviation Percent Row Pct Col Pct milk tea Total milk X** tea Total STATISTICS FOR TABLE OF X BY Y Statistic DF Vale Prob Chi-Sqare Likelihood Ratio Chi-Sqare Continity Adj. Chi-Sqare Mantel-Haenszel Chi-Sqare Fisher's Exact Test (Left) (Right) (2-Tail) P(n11 = 0) ** Phi Coefficient Contingency Coefficient Cramer's V Sample Size = 8 WARNING: 100% of the cells have expected conts less than 5. Chi-Sqare may not be a valid test. **: Added by the athor. Not a part of PROC FREQ otpt