A TEST OF SIGNIFICANCE OF DIFFERENCE BETWEEN CORRELATED PROPORTIONS. John A. Keats

Transcription

1 ~ E S E B A U ~ L t L I-i E TI RB A TEST OF SIGNIFICANCE OF DIFFERENCE BETWEEN CORRELATED PROPORTIONS John A. Keats N This Bulletin is a draft for interoffice circulation. Corrections and suggestions for revision are solicited. The Bulletin will be automatically superseded when a report of the research is published in the literature. Educational Testing Service Princeton, New Jersey August, 1955

2 A TEST OF SIGNIFICANCE OF DIFFERENCE BETWEEN CORREIATED PROPORTIONS Abstract This bulletin reports a likelihood ratio test of the significance of the difference between correlated proportions developed by Dr. S.S. Wilks. A test of the independence of the variables yielding these proportions on the assumption of e~ual difficulty is also presented. It is also shown that the tests of the significance of differences between correlated proportions used in the literature are in fact ways of estimating a conditional probability. Computational formulae and a computational layout as well as an illustrative example are included.

3 A TEST OF THE SIGNIFICANCE OF DIFFERENCE BETWEEN CORRELATED PROPORTIONS The problem of testing the significance of the difference between correlated proportions arose in connection with a piece of research conducted by the writer (2). As will be shown below, the solutions of this problem which appear in the psychological literature 7 see for example Edwards (1) p. 90, involve estimates of conditional probabilities. The present article reports a maximum likelihood test of the null hypothesis that these proportions, representing fractions of correct answers on two items of an examination for instance, are equal irrespective of' correlation. The proposed test is for large samples; the small sample problem is being investiga.ted. In fact, the present article will present liklihood ratio methods* of deriving test criteria for the following null hypotheses~ 1. that there is no difference between the proportions, 2. that the two items yielding the proportions are uncorrelated on the as):l.umption that they are equal. It will be shown that these two tests are independent-and combine naturally to test the hypothesis that there is no difference in difficulty and also no correlation. *The likelihood ratio solution presented in this paper is due to Dr. S. S.. Wilks (private communication).

4 -2- The situation is considered in which a sample of n cases is cross classified in two different ways. Such a situation can be represented by a two by two table in the following way: II 1 2 I 1 n l l n l o 2 n 2l n 22 n 2 n l n 2 n =n The most general statement of the population proportions is the following: I 1 2 II 1 2 Pll P12 Pl. P2l ~22 P2. P.l P.2 1 The probability (p) of obtaining the observed values with these a population proportions is as follows: where On the assumption of equality of proportions; i.e., P12 = P2l = r, the probability P b of obtaining the observed values becomes: On the further assumption of equality of proportion and zero correlation, writing Pll = u 2, r= uv and P22 = v 2 leads to the probability P c of obtaining the observed values

5 -3- Estimates of Pll 2 P12 2, obtained which maximize P a, P22 ' r, u and v are and P c separately with the obvious restriction that the proportions add to unity. When this is carried out it is found that the appropriate estimates are the sample values as might be expected. The maximum likelihood test of equality of proportions in the population irrespective of' correlation is theng + n 21 The quantity -2 log ":L is approximately distributed as Chi square with one degree of freedom in large samples if the hypothesis of equal proportions in the population is true~ The maximum likelihood test of zero correlation in the population under the assumption of equal proportions is~ P c max. '2 = p. = ~----.;...""--:" b max. 2n -sn 2+n2,2n +n +n ~2~~ n [n l l + 7" ] [n ] n n l l h 12 + n 21 + n 21 n 22 n n l l [ 2] n 22 The quantity -2 log '2 is approximately distributed as Chi square with one degree of freedom in large samples if the hypothesis of zero correlation is true (and, of course, also if the proportions in the population are equal) 0 The maximum likelihood test for zero correlation ~ equal proportions in the population is: P. A = c roax= A x'~ 3 P a max 1

6 -4- The tests "1 and "2 are independent such that -2 log "3(=:: -2 log "1 ~2 log "2) has approximately th.e Chi square distribution with two degrees of freedom under the hypothesis of equal proportion and zero correlation. It will be convenient to denote -2 log "1 ' -2 log "2 and..2 log "3 by xi,l' Xi,2 and x~,3 where the first subscript indica.tes the degrees of freedom and the second indicates the particular test being applied. Formulae suitable for computation are: -(2n l l + + n loge 21) (2n l l + + n ~ 21) (2n n 2l) - ( + n 21) IOge2J (2n n 21) loge It will be noted that certain values appearing in 2 X l,2 also appear in Comments on the exact treatment The probability of a given "Ion the assumption P12 = P2l may be written a.s~ For an exact test of whether or not the observed "1 is significant the values of P("lfp12 = P2l = z ) are summed over all nil' n 22, n l 2 and n 2l which yields values of Al less than or equal to the observed value. It may be shown that this s'uiiliijation simplifies to the following value of

7 -5- where p = Pll + P22 and k i = i + n 21i,and where summation is over all possible values of i and k i for which the corresponding Ali is ~ A to It is to be noted that: (1) this expression depends on the unknown value of p ) (2) the usual test may be Wl:"itten as E (:12) 2~ which is the conditional distribution for k. = k, the observed value a. It would be desirable to find out ho'w rapidly the exact probability approaches the Chi square approximation with increases in n and k for different p values, and perhaps a not too inefficient overstatement of the probability in a particular case. These problems are under investigation. Computational lay9ut and 'Worked. e~.mple The following computational layout 'Was found convenient by the writer: Natural Data Logarithm Products (1) Totals ~(Chi squares) loge loge n 21 etc. etc. etc E A + n 2l B A - B = -log A l n l l n n C 2n l l + + n 21 2n n E (2) D C - D = -log A 2 ~ ("hi square) 2 degrees of freedom

8 -6- Notes (1) In this column the product of the number and its logarithm is inserted except for the numbers 2 and 4 whose logarithms are multiplied by the ~umber below them, i.e., n + 21 and n respectively. (2) The values for ( + n 21) loge 2 are repeated and may be copied, of course. Worked example from Keats "Formal and Concrete Thought Processes" (2). Item 1 Right ~:,'W.igng Right Item 2 Wrong Xl,l for difference in difficulty = xi 2 for correlation on assumption equal difficulty =.66, X~,3 for difference of difficulty and correlation = The 5% and 1% values of Chi square with 1 degree of freedom are 3 8 and 6 6, with 2 degrees of freedom 6 0 and 9 2. The conclusion is that these two items differ from a pair which are equally difficult and uncorrelated in the direction of being not equally difficult.

9 References Edwards, A. L. Experimental Design in Psychological Research. New York~ Rinehart and Company, Inc., Keats, J. A. Formal and Concrete Thought Processes. Educational Testing Service Research Bulletin 55-17, 1955.