13.7 ANOTHER TEST FOR TREND: KENDALL S TAU

Transcription

1 13.7 ANOTHER TEST FOR TREND: KENDALL S TAU In 1969 the U.S. government instituted a draft lottery for choosing young men to be drafted into the military. Numbers from 1 to 366 were randomly assigned to the days of the year, including February 29. Young men born in 1950 or earlier received the number corresponding to their birthday. They were then to be drafted into the military in the order given by these numbers. After the numbers were assigned, some people thought they detected a trend: namely, that people born later in the year tended to have lower lottery numbers. Figure is a plot of the lottery data, X being the day of the year (1 January 1, 2 January 2,..., 366 December 31) and Y being the lottery number assigned to that day. On the face of it, there does not appear to be much trend in this graph. However, instead of looking at individual points, one could look at the average lottery number for each month. That is what Table 13.8 provides. The plot of the monthly averages is in Figure Now there is a distinct trend down. Is it significant, or could it just be due to chance? With our knowledge of statistics, we know this is the key question! We could use the regression procedures already introduced, but instead we will use a procedure called Kendall s tau. Tau refers to the Greek letter, which we will use below. This is a nonparametric procedure because it does not make strong assumptions, such as that the data lie in a straight line (except for random noise) or that the observations have been drawn from a particularly shaped distribution or box model. Since we really have no reason to believe that the trend here should be a straight line, this nonparametric approach seems a wise choice. Kendall s tau is simpler in concept than the linear regression procedures above, and as just stated it can be used to test nonlinear trends as well as linear ones. The basic idea is to connect all pairs of points and then count how many of these line segments have negative slope and how many have positive slope. Clearly, if a large proportion have negative slopes, this suggests that the null hypothesis of

2 300 Lottery number Day of year Figure Draft lottery number versus birthday (data are from Stephen E. Fienberg, Randomization and Social Affairs: The 1970 Draft Lottery, Science, Jan. 22, 1971, pp ). Table 13.8 Average Assigned Lottery Number for Month X Month Month s number, X Average lottery number, Y January February March April May June July August September October November December

3 220 Average lottery number Month Figure month. Average lottery number versus birth no downward trend is likely not defensible. Figure is the same plot as Figure 13.16, except that now there are lines connecting all pairs of points. There are a lot of line segments: a total of 66. Most of them slope downward, but a number slope upward (such as two of the segments in the upper left). Counting carefully reveals that there are 55 segments with negative slopes and 11 segments with positive slopes. Kendall s tau is defined to be the difference between the number of positives and the number of negatives, divided by the total number of segments: number of positive slopes number of negative slopes number of segments For the draft lottery data, The variable is read somewhat like a correlation coefficient. If 1, then the points trend exactly up (although it may not be in a straight line); if 1, then the points trend exactly down; and if 0, there is no particular trend up or down. The for the draft lottery data shows a fairly strong trend down. Note: If a line segment has a slope of 0, it is counted as neither positive nor negative. If two points share the same value of x, then the slope is not

4 220 Average lottery number Figure Month Points of Figure joined by line segments. defined, so it is not counted as a positive or a negative and is not counted as a segment in the denominator of. Using as a Test Statistic The number appears to provide impressive evidence of a trend, but could it be just because of chance that it is that far below 0? The null hypothesis we wish to test is that the average lottery numbers Y are independent of the month number X: H : Yis independent of X. 0 If H0 is true, there should be no trend either up or down: should be around 0. The way we build a step 1 population from the data to formally test this hypothesis is to note that if the hypothesis is true, then any rearrangement of the given set of Y s in Table 13.8 is as plausible as any other. That is, for the 12 given X s and the 12 given Y s, if there is no trend then a particular Y is equally likely to appear with any of the X s. That is, we should be able to match the Y s with the months ( X s) in any randomly chosen order to get a typical value. This procedure we now detail using the six-step method. 1. Choice of a Model (Definition of the Population): The population consists of the 12 values of Y that we observed: 201.2, 203.0,...,

5 Table 13.9 Randomly Assigned Lottery Number for Month X Using Data of Table 13.8 Month Month s number, X Average lottery number, Y January February March April May June July August September October November December Under the null hypothesis, these are to be randomly assigned to the 12 months, because any order is as likely as any other if X and Y are indeed independent. Thus again we employ a randomization approach. 2. Definition of a Trial (Sample): A trial consists of randomly assigning the 12 Y s to the 12 months. We do this by randomly choosing the Y s, one at a time, without replacement. The first one chosen is assigned to January, the second to February, and so on, until there is just one left for December. Table 13.9 has an example. Thus the first Y we drew was 208.0, then came 121.5, and so on, and finally The Y values are the same as in Table 13.8, but in a different order. 3. Definition of the Test Statistic and Definition of a Successful Trial: Now that we have the new arrangement, we proceed to calculate Kendall s tau for it. Figure shows the points of the new arrangement and their connecting segments. This time the trend is not so obvious, and there are many segments sloping up and many sloping down. Counting shows that 44 of the slopes are positive and 22 are negative. Thus This is nearer 0 than the of the obtained data. Such a simulated test statistic produces a successful trial if Repetition of Trials: We do steps 2 and times. Each time, we end up with a new based on a new random assignment of months to the given Y s. Table shows the stem-and-leaf plot for the simulated s. The mean of the s is , and the standard deviation is

6 Simulated average lottery number Simulated average lottery number Month Month Figure Data of Table 13.9 randomly assigned and joined by line segments. 5. Estimation of the Probability of Obtaining a As Small As or Smaller Than the Observed (Probability of a Successful Trial): From the data, the was Looking at all the simulated s, we see that the lowest is Thus we estimate the chance of getting a as small as or smaller than to be about 0% if the X and Y are independent. 6. Decision: Step 5 shows that it is very unlikely to get a as low as by chance under the null hypothesis. Therefore we have to reject the null hypothesis that is, conclude that X and Y are dependent and that there is a trend to the average lottery numbers among the months. From the above analysis, we have to conclude that the 1969 draft lottery was not completely random, and that indeed, on average, people born later in the year did have lower lottery numbers to an extent not explained by chance. The personnel in the U.S. government also decided the lottery was not completely random. The following year a better randomization process was used, and there did not appear to be any trend. A z Test for Kendall s Tau As for many of the other hypothesis tests we have carried out by simulations based on bootstrapping or randomization, one can also use a test based on the normal approximation to see whether the observed is statistically significant. That this is legitimate here is suggested by the roughly belllike shape of the stem-and-leaf plot for of Table A sophisticated

7 Table One Hundred Simulated s for Lottery Data zzzzzzzz Key: 4 5 stands for probability argument can be given to justify the assumption that these sampled s are approximately normally distributed. The key to the test is to figure out what the mean and the standard deviation of are when the null hypothesis is true. The formulas are given below. They are based on the assumption that the X s are independent of the Y s. It is clear that under the null hypothesis, Population mean of 0 However, deriving the standard deviation of is beyond the scope of this book. The value is SD of 2(2n 5) 9 nn ( 1) where nis the number of pairs, 12 in our case. Then the zstatistic is mean of tau z SD of tau For our example, n 12; hence 2 (2 12 5) SD of Note that the standard deviation of the simulated s was , very close to the theoretical value. Now the z statistic: z Now we can finish up with the new step 5.

8 New Step 5. Estimation of the Probability of Obtaining a Slope As Large As or Larger Than the Observed (Probability of a Successful Trial): Looking in the normal table (Appendix E), we see that the chance of z being less than is , or about 1/10 of a percent. Thus in step 6 we conclude that the chance of getting a below is so small that we have to rule out chance. Again, we strongly reject the null hypothesis that X and Y are independent. We have made considerable progress in this chapter on one of the major problems in statistics, namely, discovering relationships between variables in the presence of random noise. We can use linear regression if it is appropriate to assume a straight-line relationship. Moreover, we can test for the presence or absence of such a relationship using the correlation coefficient. We can solve some nonlinear relationship problems by rescaling to turn them into ordinary linear regression problems. We have learned about curvilinear and multiple regression and the use of the multiple correlation coefficient R2 to measure how good the fit is in these cases. Finally, we have learned to test for the presence of either an upward or a downward trend via Kendall s tau, even if the trend is curved rather than linear. Sometimes we used our often-used six-step simulation approach, and sometimes we used the often-used normal approximation approach. SECTION 13.7 EXERCISES 1. Consider the five points in the following scat- 2. For each of the following sets of conditions, ter plot: calculate Kendall s tau: a. Seven segments have positive slopes, 14 segments have negative slopes, and a total Y 10 of seven points appear on the scatter plot. b. Twenty-one segments have positive slopes, 8 15 segments have negative slopes, and a total of nine points appear on the scatter 6 plot. 4 c. Thirteen segments have positive slopes, 32 segments have negative slopes, and a total 2 of 10 points appear on the scatter plot X a. Calculate for this set of data. b. Use the z test for Kendall s tau to test the hypothesis that the X s are independent of the Y s. 3. For each of the sets of conditions in Exercise 2, test the hypothesis that X and Y are independent. 4. Consider the six points in the following scatter plot:

9 Y X 7. Consider the data in Exercise 8 of Section a. Plot the points, and connect each pair of points with a line segment. How many of the segments are there? How many have positive slope? How many have negative slope? b. Find Kendall s for these data. c. Keeping the Xs fixed, randomly choose four values from the Y values without re- placement, and plot the results. d. Connect the pairs of points in (c) with line segments. How many are positive, and how many are negative? Find Kendall s. Is it lower than that in (b)? a. What is the value of Kendall s tau for this set of data? b. Perform the hypothesis test to determine whether X is independent of Y. 5. Describe the similarity in the interpretation of the Kendall s tau statistic and the Pearson correlation coefficient. 6. State two advantages of the use of Kendall s tau to test for trend over the use of the standard method with the least squares regression line.