Population Invariance of Score Linking: Theory and Applications to Advanced Placement Program Examinations

Transcription

1 Research Report Population Invariance of Score Linking: Theory and Applications to Advanced Placement Program Examinations Neil J. Dorans, Editor Research & Development October 2003 RR-03-27

2

3 Population Invariance of Score Linking: Theory and Applications to Advanced Placement Program Examinations Edited by Neil J. Dorans Educational Testing Service, Princeton, New Jersey Papers by Alina A. von Davier, Neil J. Dorans, Paul W. Holland, Krishna Tateneni, Dorothy T. Thayer, and Wen-Ling Yang Educational Testing Service, Princeton, New Jersey Bert F. Green Johns Hopkins University, Baltimore, Maryland Michael L. Kolen University of Iowa, Iowa City, Iowa October 2003

4 Research Reports provide preliminary and limited dissemination of ETS research prior to publication. They are available without charge from: Research Publications Office Mail Stop 7-R Educational Testing Service Princeton, NJ 08541

5 Abstract Test equating methods are intended to produce interchangeable scores and should not be strongly influenced by the group of examinees on which they are computed. Advanced Placement Program (AP ) exams, which include both multiple-choice (MC) items and constructed-response (CR) items, provide an environment for studying the effects of subpopulations on the estimation of equating functions. The exams are equated to past forms using an internal anchor-test design in which the linking items are restricted to MC items due to the immediate disclosure of the CR sections of the tests (precluding their reuse as linking items). This collection of closely related papers uses the variety of cases provided by AP exams to assess the effects of subpopulations on various important aspects of test equating. Earlier versions of these papers were prepared for a symposium at the 2002 Annual Meetings of the National Council on Measurement in Education. Key words: Advanced Placement Program (AP ), anchor test design, score equating, gender differences, nonequivalent groups, population invariance, region effects, self-equating, test linking i

6 Acknowledgements The authors wish to thank their colleagues who contributed their own papers as well as helpful comments on the other papers in this collection. The authors particularly appreciate the comments of Bert Green and Michael Kolen. Henry Braun, Dan Eignor, and Samuel Livingston reviewed earlier versions of these papers and provided helpful comments and posed challenging questions. Rick Morgan, who moderated the symposium that predated this report, also reviewed earlier versions of some of these papers. Rick's extensive knowledge of the Advanced Placement Program has been valuable for several of the authors. Finally, the editor thanks Martha Thompson for taking a collection of papers and making them look like they were parts of a common report. This work was partially supported by ETS and the College Board. The opinions expressed herein are those of the authors and not their respective institutions. ii

7 Table of Contents Page Preface... iv Overview of Population Invariance of Test Equating and Linking by Paul Holland... 1 Population Invariance and Chain versus Post-stratification Methods for Equating and Test Linking by Alina A. von Davier, Paul W. Holland, and Dorothy T. Thayer Invariance of Linkages for Free-response, Multiple-choice, and Composite Scores on the Advanced Placement Program Physics B Examination by Krishna Tateneni and Neil J. Dorans Effect of Sample Selection on Advanced Placement Multiple-choice Score to Composite Score Linking by Wen-Ling Yang, Neil J. Dorans, and Krishna Tateneni Invariance of Score Linking Across Gender Groups for Three Advanced Placement Program Examinations by Neil J. Dorans, Paul W. Holland, Dorothy T. Thayer, and Krishna Tateneni Evaluating Population Invariance: A Discussion of Population Invariance of Score Linking: Theory and Applications to Advanced Placement Program Examinations by Michael J. Kolen Comments on Population Invariance of Score Linking by Bert F. Green References iii

8 Preface Many testing programs produce alternate forms of a test and want the scores on these forms to be comparable to each other. Equating is the process used to link the raw scores from each new test form to the raw scores from an earlier form. For decades, scores have been equated. While much time and effort has been given to how to equate, much less attention has been given to whether or not it makes sense to equate. Lord (1980, chapter 13) provided guidelines for evaluating whether or not it makes sense to equate. Unfortunately, Lord s valuable perspective on equating is often reduced to a restatement of his Theorem (p. 198) equating is either impossible or unnecessary. This pithy statement had little impact on practice. Tests continued to be built. For a variety of reasons, alternate forms have been produced, and equating procedures have been applied with the hope, if not the expectation, that interchangeable scores would be produced. Dorans and Holland (2000) revisited Lord s guidelines, questioned his equating argument, modified the framework, and proposed indices that can be used to evaluate how much a linking deviates from the ideal of perfect equatability. Population invariance plays a central role in assessing equatability. Tests are equatable to the extent that the same equating function is obtained across significant subpopulations, such as males and females. This collection of papers extends the work done by Dorans and Holland and applies it to the Advanced Placement (AP) Program. Holland (pp. 1-18, this report) provides an overview that introduces terminology and indices that appear in several of the other papers in this report. Some of Holland s material pertains to linking in general, some is specific to equating, and some is tailored to the AP setting in particular. Von Davier, Holland, and Thayer (pp , this report) examine two observed-score equating methods used for Non-Equivalent Anchor Test (NEAT) designs chain equating and post-stratification from a common statistical framework and introduce a method that can be used in the NEAT design to study the population invariance of equating methods. Tateneni and Dorans (pp , this report) investigated the population invariance of linking relationships on the AP Program Physics B exam when the anchor test is made up of only multiple-choice items, only free-response items, or a combination of both. Linking results for subgroups were compared to those for the total group. Cross-classifications of grades based iv

9 on total and subgroup linkings were also examined. Yang, Dorans, and Tateneni (pp , this report) investigates whether the multiple-choice to composite linking functions remain invariant over subgroups by region for two AP exams using 3 years of test data. The study focuses on two questions: (a) How are invariant cut scores across regions? and (b) Does the small sample size for some regional groups present particular problems for assessing linking invariance? Both equipercentile and linear linking methods are applied. The equatability index proposed by Dorans and Holland (2000) is employed to evaluate the invariance of the linking functions, and the cross-classification approach is used to evaluate the invariance of the composite cut scores. Some exams, such as AP exams, involve a mix of formats (e.g., multiple-choice questions and free-response questions) and are composed of questions that tap different dimensions. AP grade assignments are made on the basis of both multiple-choice and freeresponse sections. Males and females exhibit differential mean score differences on the freeresponse and multiple-choice sections. Do these mean score differences affect the equatability of AP scores and the invariance of AP grade assignments across gender groups? Dorans, Holland, Thayer, and Tateneni (pp , this report) examine the invariance of AP linkings across gender. Bert Green, professor emeritus at Johns Hopkins University, and Michael Kolen, professor at the University of Iowa, are well-versed in the theory and practice of linking and equating. Kolen was responsible for the scaling and equating activities of the ACT and other exams during his years with ACT. The Kolen and Brennan (1995) text, which will be issued in its second edition, is a primary text for training in the theory and practice of equating, as well as a valuable reference book. Green has been actively involved in applied testing issues since the 1950s. In addition to serving as a role model for many professionals who seek to foster high standards in a practical applied way, Green has served indefatigably on a wide variety of committees for many years. He recently served as an editor for two important monographs on test score linkage that were produced by the Board of Testing and Assessment of the National Research Council. Green s comments on the first five papers of this report appear on pp , while Kolen s comments on the same papers appear on pp Neil J. Dorans September 2003 v

10

11 Overview of Population Invariance of Test Equating and Linking Paul W. Holland Educational Testing Service, Princeton, New Jersey 1

12 Abstract The purpose of this overview is to introduce terminology and indices that appear in several of the other papers in this report. Some of the material pertains to linking in general, some is specific to equating, and some is tailored to the Advanced Placement (AP) setting in particular. The appendixes for this paper contain material that is related to equating and linking, but less immediately germane to the papers that follow. For example, I emphasize nonlinear methods here, but Appendix A shows how they are related to linear ones in a fundamental way. We used kernel equating in several of the papers as well, and this method is briefly outlined in Appendix B. 2

13 Observed Score Equating and Test Linking Methods The raw scores of a new test, X, are to be transformed to be equivalent to the raw scores on an old test, Y, on a target population, T. Observed score equating is accomplished by finding a suitable transformation, called an equating function, that is applied to each rawscore from test X and results in equivalent Y-score for each X-score. Ideally, when the scores of the tests are equated they may be used interchangeably for any purpose (von Davier, Holland, & Thayer, 2003a). The term test linking will be used to refer to the general problem of linking or connecting the scores on two different tests on a target population. Dorans and Holland (2000) comment on the five requirements of test equating. One of these requirements is the population invariance requirement, which is the focus of this collection of papers. By population invariance, we mean that the linking function is invariant across important subpopulations of the population for which the test is designed. If we ignore the population invariance requirement, then the linking relationship is sometimes called a concordance. A concordance between the scores of two tests is a transformation between the scores of the tests Y and X that is designed to hold for a specific population of examinees, and there is no claim that it holds for any other. The process that produces a concordance will be referred to as a scaling to produce concordance, or scaling for short in latter papers in this set. In the following sections, the traditional term equating is used to describe different methods of linking scores. The same statistical procedures can be used to produce concordances. The target population that is relevant will depend on the equating or linking design used (Kolen & Brennan, 1995). Many observed score equating (OSE) methods are based on the equipercentile equating function. It is defined (on the target population) as: e XY;T (x) = G -1-1 T (F T (x)) = G T F T (x), where F T (x) and G T (y) are the cumulative distribution functions, cdfs, of X and Y, respectively, on T (with the assumption that F T (x) and G T (y) have been made continuous or continuized so that the inverse functions exist for F T (x) and G T (y); see Appendix B). The NEAT Equating Design The Non-Equivalent groups Anchor Test (NEAT) design is very common. The two tests to be equated, X and Y, are given to different test subpopulations or administrations 3

14 (denoted here by the subpopulations P and Q). In addition, an anchor test, V, is given to both P and Q, resulting in the following data structure where denotes the presence of data: X V Y P X, V observed on P, Q Y, V observed on Q. The anchor test score, V, can be either a part of both X and Y (the internal anchor case) or a separate score (the external anchor case). The target population, T, for the NEAT design is a mixture of P and Q and denoted by: T = wp + (1 w)q. The mixture is determined by a weight w. When w = 1, then T = P, and when w = 0, then T = Q. Another choice is w = ½, which makes P and Q represented equally in T. Any choice of w between 0 and 1 is possible as well and reflects the amount of weight that is given to each subpopulation in determining T. Note T is a synthetic population composed of two actual subpopulations. One way to think of target populations is by simply pooling data observed on both P and Q, i.e., V, into one score variable. This is done in Tucker equating with the w determined by the relative size of the samples from P and Q, i.e., w = n P /(n P + n Q ). However, there is no reason different weights from these might be used to define T. It is often a good idea to be explicit as to what w is in the NEAT design, but the extent to which the choice of w matters for the resulting equating function depends on the data and the equating method. The scores, X and Y, are not observed in both P and Q. For this reason, assumptions must be made in order to overcome this lack of complete data in the NEAT design. The basic problem is to make sufficient assumptions that values for F T (x) and G T (y) can be found. In other equating and test linking designs, such as the Equivalent Groups or Single Group designs, there is usually less ambiguity as to what the target population is and how to estimate F T (x) and G T (y). 4

15 In our discussion of the NEAT design we will let F, G, and K denote, respectively, the cdfs (suitably continuized) of X, Y, and V, and will further specify the subpopulations on which these cdfs are determined by the subscripts P, Q, and T. The Advanced Placement Program Examples In the AP examples that will be used in the other papers of this symposium, the data collection design is slightly more complicated than a NEAT design, though closely related to it. In the AP program, there is a composite score (C) for each test. C is a weighted sum of scores from the multiple-choice (MC) part and the free-response (FR) part with weights that depend on the particular AP test. The scores are: C, MC, and EQ. (EQ is an included anchor within the MC part of the test.) The operational AP program uses only part of these data (see Tateneni & Dorans, pp , this report, for details), but we include the full design here for completeness. The AP data has the following structure where P and Q refer to two administrations, P being the older and Q being the newer: C1 MC1 EQ MC2 C2 P Q In this design, 1 and 2 refer to the forms used in P and Q, respectively. The paper by von Davier, Holland, and Thayer (pp , this report) uses the links from MC1 to EQ to MC2 to illustrate a test of different assumptions that are made when using the NEAT design. Tateneni and Dorans (pp , this report) use the data from MC2 and C2 as well as the free response part of C2. Yang, Dorans, and Tatnenei (pp , this report) use the link from MC2 to C2. Dorans, Holland, Thayer, and Tateneni (pp , this report) use the long path from MC1 to C2 because this is important for the operational work in the AP program. Chain and Post-stratification Equating Methods for the NEAT Design There are two important classes of OSE methods used in the NEAT design. The first is called chain equating, or equating two tests through the anchor or the Lindquist method. 5

16 The second has several variants including the Tucker method and frequency estimation. We will show here how these can all be put into the OSE framework outlined earlier. We will call the chain methods chain equating (CE) and the other methods post-stratification equating (PE). The reason for the latter name is that these methods all use the anchor test, V, to post-stratify both X and Y by first conditioning on V and then reweighting the conditional distributions in order to estimate F T (x) and G T (y). I will concentrate my discussion here on the equipercentile or non-linear case due to the relationship between linear and equipercentile OSE methods described in Appendix A. Chain equating (CE). This method first links X to V on P, then V to Y on Q, and then functionally composes the two linking functions to link or equate X to Y. In order for this to make sense as an observed score equating method, we must identify T, the target population, and see what assumptions are made in order for F T (x) and G T (y) to be determined. The target population turns out to be irrelevant for CE any T of the form wp + (1 w) Q will result in exactly the same equating function. There are two assumptions for CE. CE1: The link from X to V is population invariant, so that: -1-1 K P F P (x) = K T F T (x) for any T, and therefore -1 F T(C) (x) = K T K P F P (x). CE2: The link from V to Y is population invariant, so that: -1-1 G Q K Q (v) = G T K T (v) for any T, and therefore -1 G T(C) (y) = K T K Q G Q (y), or G -1-1 T(C) = G Q K Q K -1 T. We use the subscript (C) to indicate that F T(C) and G T(C) are the solutions to specifying F T and G T for CE. Now we apply these two assumptions to the computation of the composed link between X, V, and Y. It is: -1 e XY;T(C) (x) = G T(C) F T(C) (x) = G Q K Q K T K T K P F P (x) = G Q K Q K P F P (x). 6

17 Notice how T cancels out in the middle of the above composition of six cdfs and their inverses, so that the chain equipercentile equating function depends only on the four marginal cdfs, F P, K P, K Q, and G Q, and is the same for any choice of T. Post-stratification equating (PSE). This method first estimates the distributions of X and Y on T using post-stratification methods and then computes the equipercentile equating function as indicated in the first section of this paper. In order to do this, the post-stratification methods make two assumptions that are similar in spirit to CE1 and CE2 for chain equating. PSE1: The conditional distribution of X given V is population invariant, i.e., f T(PS) (x) = f P(x v)k T (v) for any T, v where f(x) denotes the score probabilities for X and k(v) the score probabilities for V. PSE2: The conditional distribution of Y given V is population invariant, i.e., g T(PS) (y) = g Q(y v)k T (v) for any T, v where g(y) denotes the score probabilities for Y. Once the score probabilities, f T(PC) (x) and g T(PC) (y), are computed by the poststratification reweighting indicated in PSE1 and PSE2, the corresponding cdfs, F T(PS) (x) and G T(PS) (y), are computed by continuizing the discrete distributions specified by f T(PC) (x) and g T(PC) (y). Thus, F T(PS) and G T(PS) are the solutions for F T and G T for PSE. Finally, the post-stratification equipercentile equating function is computed by: -1 e XY;T(PS) (x) = G T(PS) F T(PS) (x). In this case, the equating function, e XY;T(PS) (x), can depend on the choice of T through the dependence of F T(PS) and G T(PS) on T. In this way, PSE can be different from CE, though they can also be identical in certain circumstances, discussed in the von Davier, Holland, and Thayer paper (pp , this report). The Advanced Placement examples. In the operational AP procedures that involve the longer chain from MC1 to C2, more than one equating method is used and then linked together. These functions are used to transform the average of cut scores from a set of prior examinations to the current composite score, rather than the more traditional use of equating and test linking functions to map scores on the new form to the scale of an old form (see Tateneni & Dorans, 7

18 pp , this report, for more details). The AP operational procedures often involve deciding between the PSE and CE links. Part of the rationale for the studies reported in this symposium are that they give data that can inform these choices of linking functions in practice. The Dorans and Holland Measure of the Lack of Population Invariance of an Equating or Test Linking Method Dorans and Holland (2000) define the conditional root-mean-square difference, or RMSD(x), as a measure of the degree to which an equating or test linking procedure fails to be invariant across a given set of subpopulations of a base population, P. Hence, equating is a linking that is expected to be invariant across subpopulations from the same population, but not across populations. For example, linkings between AP multiple-choice scores are expected to be invariant for females and males, but are not expected to hold up in populations of students who have not taken the appropriate AP course. They develop their measure in the following setting. The subpopulations, {P j }, partition P into mutually exclusive and exhaustive subpopulations (such as males and females, or race/ethnicity groups). Each P j has a weight, w j, which could be its relative proportion in P, or some other set of weights that sum to unity. This is denoted by: P = w jp j. j Next we assume that the function, e j (x), links or equates X to Y on P j, and that e(x) links or equates X to Y on P. Then, RMSD(x) is defined as: RMSD(x) = j w (e (x)-e(x)) j σ j YP 2. In the above equation, σ YP denotes the standard deviation of Y in P. Because Y is not observed for examinees in P, the estimation of σ YP depends on assumptions of Tucker equating. In other words, at each X-score, RMSD(x) is the root-mean-square difference between the linking functions computed on each subpopulation and the linking function computed on P. It is standardized by dividing by the standard deviation of Y on P so that it is a type of effect size, and it may be described as a percent (of the Y-standard deviation on P). 8

19 Examples of subpopulations used in these symposium papers include those based on examinee characteristics such as gender and those based on regions of the world. Other examples of subpopulations that could be examined include race/ethnicity, language group, and the time of arrival of the answer sheets in the post-testing processing stream of a major test administration early returns versus later returns. Other examples include test administrations, which typically are confounded with test forms since only one test form is given at a test administration. The only restriction for computing RMSD is that the {P j } partition P and the non-negative weights {w j } sum to one. In addition to RMSD(x), which gives a value for each score point of X, Dorans and Holland (2000) also suggest a summary measure that is a type of average over the X-values. It is the root-expected-mean-square difference, or REMSD, defined as: REMSD = 2 E{ P w j(e j(x)-e(x)) } j σ YP = j w 2 je{ P (e j(x)-e(x)) } In REMSD, E P {} means average over distribution of X in P. REMSD is a double-weighted average of differences between subpopulation linking functions and the total group linking function. At each score level, the difference between each subpopulation linking function and the total group linking function is squared. These squared differences are then averaged over subpopulations weighted by the relative size of each subpopulation. Then these weighted sums of squared differences are averaged across score levels weighted by the relative number of candidates in the total population at each score level. Taking the square root of that weighted average and dividing the result by the standard deviation of the composite score in the total population gives us a measure of overall equatability in the metric of the standard deviation of the composite score. σ YP. Applying the Dorans and Holland Measure to the NEAT Design Using the notation in the previous section, we assume we have a system of subpopulations that can be identified both in P and in Q, i.e., P = w PjP j and Q = j w QjQ j, j 9

20 where P j and Q j refer to the same subpopulation (e.g.., to males or to females) of P and Q, but the sets of weights, {w Pj } and {w Qj }, may be different. Then the target populations, T j, are defined as: T j = wp j + (1 w)q j, where we use a common weight, w, to define the target populations,{t j }. In order to proceed, we simply restrict attention to each pair (P j, Q j ) and use the data structure: X V Y P j Q j Then we compute the linking functions using the data from P j and Q j. This results in the linking functions, e XY;Tj(C) (x) for CE and e XY;Tj(PS) (x) for PSE. They are given by: and e XY;Tj(C) (x) = G Qj -1 K Qj K Pj -1 F Pj (x), e XY;Tj(PS) (x) = G Tj -1 F Tj (x). We note that the subpopulation CE equating functions are not necessarily equal. However, they are all insensitive to the choice of weight, w, used to define T j. In addition, we compute the CE and PSE equating functions e XY;T(C) (x) and e XY;T(PS) (x) for the whole target population, T. There is a slight difference in the computation of RMSD(x) for the two equating methods. It has to do with the choice of weights and the standard deviation used to normalize the RMSD into an effect size measure. We discuss these issues in the next two subsections. The CE case. Here we have to make a decision as to what weights to use in RMSD(x). The choice is clear, because there is only one subpopulation where there is data for X, and that is P. So we use {wpj}. In addition, we need the standard deviation of Y, and the only place σ YQ where there is Y-data is Q, so we use. Hence, for the CE case we define RMSD(C) (x) by: RMSD (C) (x) = j w (e (x)-e (x)) Pj XY;Tj(C) XY;T(C) σ YQ 2. 10

21 This formula is used Dorans, Holland, Thayer, and Tatnenei (this report, pp ). In addition, REMSD (C) is defined as: REMSD (C) = j 2 wpje P{e XY;Tj(C) (X)-e XY;T(C) (X)) } The PSE case. Here the situation is simpler. There are natural weights given by: w j = w(w Pj ) + (1 w)w Qj, and a natural estimate of σ YT given by the one used in the Tucker method for T = wp + (1 w)q. The final equation for RMSD (PS) (x) is given by: RMSD (PS) (x) = In addition, REMSD (PS) is defined as: REMSD (PS) = j j σ YQ w (e (x)-e (x)) j XY;Tj(PS) XY;T(PS) σ YT 2 we j T{(e XY;Tj(PS) (X)-e XY;T(PS) (X))} where E T {} means averaging over the implied distribution of X in the mixture, T = wp + (1 w)q. σ YT 2.., The Advanced Placement examples. In the AP operational program, the main focus is on the final five-point AP grade scale. Thus, the values of RMSD(x) over the full range of x is of less interest than at the five cut-points. In addition, from an operational standpoint the sizes of effects that are important are those that result in changes in AP grades, and RMSD does not directly express this. To show the effect of the different subpopulations on which the linking is done in terms of changes in AP grades, the following grade comparison table, Table 1, is used in some of the papers in this report. The rows of Table 1 correspond to grades using the function determined on a particular group, P j, and the columns correspond to grades using the function determined on all the examinees, P. The entries A, H, and L denote percentages of examinees from group P j, only. The A-percentages indicate the percent agreement between the two grades; the H-percentages give the percent in group P j who get a higher grade using their P j -based function rather than the one based 11

22 on all of P; and the L-percentages are those in group P j who get a lower grade using their P j -based function. Table 1 Example of a Grade-comparison Table Cut-points determined by all cases Cut-points determined by the subgroup A H 4 L A H 3 L A H 2 L A H 1 L A 12

23 Appendix A Linear and Equipercentile Observed-score Equating The following theorem summarizes the relationship between the linear equating function, Lin XY;T (x), defined by: Lin XY;T (x) = µ YT + σ YT ((x µ XT )/σ XT ) and the equipercentile equating function, e XY;T (x). Theorem: For any target population T, e XY;T (x) = Lin XY;T (x) + R(x), where -1 R(x) = σ YT r((x µ XT )/σ XT ), and r(u) = G 0 F 0 (u) u. F 0 (u) and G 0 (u) are the cumulative distribution functions (cdfs) with mean zero and variance one that satisfy the equations: F T (x) = F 0 ((x µ XT )/σ XT ), G T (y) = G 0 ((y µ YT )/σ YT ). In this theorem, the remainder, R(x), is the non-linear part of the equipercentile equating function. When F 0 (u) and G 0 (u) are the same, R(x) is identically 0, and e XY;T (x) = Lin XY;T (x). This relationship between the linear and non-linear observed-score equating functions suggests that post-smoothing might profitably be focused on smoothing R(x) and leaving Lin XY;T (x) intact. However, I have not explored this idea further and am inclined to think of postsmoothing as less satisfactory than pre-smoothing followed by a judicious choice of bandwidth in the continuization process described in Appendix B. 13

24 Appendix B An Outline of Kernel Equating Here I give a brief outline of the Kernel method of observed-score test equating. This method is over a decade old (Holland & Thayer, 1989), but has not been widely discussed due to its apparent excess of mathematical notation and lack of a clear value-added over the existing equipercentile methods. We regard it as a useful structure in which to view all equipercentile equating methods and a unifying method for all the standard designs. The Kernel method is discussed in detail in von Davier, Holland, and Thayer (2003a) for all of the standard equating designs. The method has five basic steps, which are designed to isolate the ideas that go into the actual test equating practice, whether or not they are recognized in practice as separate steps. Step 1: Pre-smoothing In this step, the data that are collected in an equating design are pre-smoothed using standard statistical procedures designed to estimate the actual score distributions that arise in the equating design. Pre-smoothing, using various techniques, has become a standard tool in various approaches to equipercentile equating. We advocate using log-linear models for univariate and bivariate score distributions, as discussed in Holland and Thayer (2000), because of their great flexibility and ability to accommodate the many unusual features of score distributions that arise in practice. The results of this pre-smoothing process are twofold. First, the necessary smoothed score distributions that are needed for the rest of the equating process are obtained, and second, a matrix that can be used to calculate the standard error of equating later on in the process is computed. Every presmoothing method has such a matrix, but the log-linear methods have a standard way of finding it in an efficient manner. This is discussed in detail in Holland and Thayer (2000). Step 2: Estimating Score Distributions for the Target Population Once the pre-smoothing has been done, there are formulas, depending on the equating design, that use the smoothed score distribution estimates to produce estimates of the score probability distributions on T which we call r and s, where: r j = P{X = x j T}, s k = P{Y = y k T}, 14

25 and the vectors r and s are given by: r = (r 1,..., r J ), and s = (s 1,..., s K ). The score probabilities for X are associated with the X-raw scores, {x j }, and those for Y are associated with the Y-raw scores, {y k }. Depending on the equating design, the formulas for r and s range from simple identity to the complexities implicit in anchor test methods. In the NEAT Design, use of Chain equating (CE) can avoid the explicit computation of r and s. Instead, the pre-smoothed data can be used directly to move to the next step in the process. The computation of the standard error of equating for CE involves the matrices mentioned in Step 1. Step 3: Continuizing the Discrete Score Distributions This step is often overlooked in discussions of equipercentile equating methods, but it occurs in all of them. We start with discrete score distributions for X and Y on T and turn these into continuous score distributions over the whole real line. It is similar to approximating the probabilities from the discrete binomial distribution by probabilities from the continuous normal distribution. Thus, it is a step that looks like an everyday statistical method, but it is actually unusual because the entire discrete distribution is changed into a continuous one that is close to the original in a sense that is often left vague. Our approach is to make this step explicit and to make the sense of the approximation clear. Older equipercentile equating methods replace the discrete score distributions by piece-wise linear cdfs based on percentile ranks. The (Gaussian) Kernel method of continuizing r uses the formula: F T (x; h X ) = x-a x -(1-a ) µ X j X XT rjφ, j hxa X where, µ XT = xr j j, 2 XT j σ = 2 (x j- µ XT) rj, and a X = j σ /( σ + h ) XT XT X Φ(z) denotes the standard N(0, 1) cdf, x ranges over (,+ ), and h X > 0. F T (x; h X ) is the continuized cdf based on the discrete score distribution determined by r and {x j }. The continuized G T (y, h Y ) is computed in a similar way using the score probabilities from s, and the Y-scores, {y k }. An essential feature of Gaussian kernel continuization is the choice of the bandwidths, h X and h Y. We recommend using a penalty function to select the bandwidths automatically to make the density functions, f T (x; h X ) and g T (y, h Y ), derived from F T (x; h X ) and 15

26 G T (y, h Y ), both smooth and able to track the essential features of the smoothed discrete score probabilities. We have found the following penalty functions to give good results. PENALTY 1 (h) = [(r j/d j ) f T (x j ; h)] 2, j where, d j is the width of the interval associated with the score x j (often these widths are all set equal to 1). PENALTY 2 (h) = A(1-B) j j, j where A j = 1 if the derivative of f T (x; h) with respect to x, u(x; h), is less than 0 a little to the left of x j, and B j = 0 if u(x; h) > 0 a little to the right of x j. Thus, we get a penalty of 1 for every score point where the density f T (x; h) is U-shaped around it. What near means is a parameter of PENALTY 2 (h), and we can combine the two penalties with a weight, i.e., PENALTY 1 (h) + K*PENALTY 2 (h). Standard derivative-free methods can be used to minimize these penalty functions in order to choose h. Separate continuizations of the two discrete score distributions are carried out, resulting in F T (x; h X ) and G T (y; h Y ). The case of Chain equating is slightly different and requires continuizing the cdfs, F P, K P, K Q, and G Q, directly, and then proceeding to Step 4. Step 4: Computing the Equating Function Once all the above work is done, the Kernel method equipercentile equating function can be computed directly as the function composition: -1 e XY (x) = G T F T (x) = G -1 T (F T (x; h X ); h Y ) where F T (x) = F T (x; h X ), and G -1 T (p; h Y ) denotes the inverse of p = G T (y) = G T (y; h Y ). In the case of Chain equating, the formula for the equating function is given by: -1-1 e XY(C) (x) = G Q K Q K P F P (x). Step 5: Computing the Standard Error of Equating (SEE) The standard error of equating for e XY (x) depends on three factors that correspond to the above four steps pre-smoothing, computing r and s from the smoothed data, and the combination of continuization and the mathematical form of the equating function from Step 4. 16

27 Being based on analytical formulas, the Kernel method allows us to use the Taylor expansion or delta method to compute the SEE for a variety of equating designs. The main difference between the various equating designs, as far as computing the SEE for kernel equating is concerned, is Step 2. Each design requires a different formula for mapping the pre-smoothed data to the score probabilities, r and s, but the contributions of the other steps to the SEE are the same for all designs. This observation allows a general SEE computing formula to be devised for kernel equating that needs to reflect the equating design in only one place. In addition, for Chain equating the results of the Single Group design can be used along with standard formulas for the standard error of the composition of two equating functions (Braun & Holland, 1982). This shows how the standard error of equating for CE depends in very similar ways on the three factors mentioned above. This is discussed extensively in von Davier et al. (2003a). 17

28 18

29 Population Invariance and Chain versus Post-stratification Methods for Equating and Test Linking Alina A. von Davier, Paul W. Holland, and Dorothy T. Thayer Educational Testing Service, Princeton, New Jersey 19

30 Abstract The Non-Equivalent-groups Anchor Test (NEAT) design involves two subpopulations, P and Q, of test takers and makes use of an anchor test to link them. Two observed-score equating methods used for NEAT designs are those based on chain equating and those using the anchor to post-stratify the distributions of the two operational test scores to a common target population i.e., Tucker equating and frequency estimation. We introduce a method that can be used in the NEAT design to study the population invariance of equating methods. We then apply this method to study the relative population invariance of Chain and Post-stratification equating methods. Our method combines self-equating (equating a test to itself) with the rootmean-square difference (RMSD) measure of the population invariance of test linking methods introduced by Dorans and Holland (2000). We illustrate our method using data from the Advanced Placement (AP) examinations. 20

31 Introduction Test equating methods are used to produce scores that are comparable across different test forms. Weaker forms of test linking often use the same computations as test equating but do not necessarily result in scores that are comparable. In this paper, we follow the standard practice of referring to the designs and procedures described as equating designs and equating procedures, even though they can be used with any type of linking. One of the primary requirements of equating functions is that they should be population invariant, i.e. they should be the same across different subpopulations from the same population. Because strict population invariance is often impossible to achieve, Dorans and Holland (2000) introduced a measure of the degree to which an equating method is sensitive to the subpopulation on which it is computed. The measure compares equating or linking functions computed on different subpopulations with the equating or linking function computed for the whole population. Their discussion is restricted to equating designs that involve only one test administration (such as the Equivalent-groups design and the Single Group design). The Non-Equivalent-groups Anchor Test (NEAT) design involves two subpopulations (usually different test administrations), P and Q, of test takers and makes use of an anchor test to link them. We also want population invariance to hold for equating functions used in the NEAT design, but when there are two test administrations, there can be ambiguity as to which administration is the one on which the equating (or linking) is done. For the NEAT design there are several observed-score equating methods that are used in practice. Two important classes of these methods are those we will call Chain equating and Post-stratification equating, following Holland (pp. 1-18, this report). In this paper we examine the relative population invariance of Chain versus Poststratification equating methods in the NEAT design. We use the existence of two subpopulations, such as male and female examinees, to mimic a situation where a test has been reused so that it can be equated to itself and the result compared to the identity function. We use this idea to adapt the Dorans and Holland (2000) measure of root mean square difference, RMSD(x), to compare the results of Chain and Post-stratification equating methods. Data from the Advanced Placement program are used to illustrate these ideas. 21

32 The NEAT Design Holland (pp. 1-18, this report) describes the NEAT design. Here we just reiterate some of its basic features. The important idea is the data structure: X V Y P X, V observed on P, Q Y, V observed on Q. Usually, X and Y are the operational tests given to test administrations P and Q, respectively, and V is the anchor test given to both P and Q. The anchor test score, V, can be either a part of both X and Y (the internal anchor case) or a separate score (the external anchor case). The target population, T, for the NEAT design is a mixture of P and Q and denoted by T = wp + (1 w)q. The mixture is determined by a weight w. When w = 1, then T = P, and when w = 0, then T = Q. Other choices of w are often used as well. In this situation we will let the (continuized) cdfs of the score distributions of X, Y, and V be denoted by F(x), G(y), and K(v) and will append subscripts as necessary to distinguish between P, Q, and the target population, T. The two most important test scores, X and Y, are not observed in both P and Q, but only one or the other, unlike the anchor test score, V. Thus, assumptions must be made in order to overcome this aspect of the NEAT design. The different observed score equating methods used in this design each make different assumptions about the distributions of X and Y in the subpopulations where they are not observed. Chain and Post-stratification Equating Methods for the NEAT Design The Chain equating (CE) and the Post-stratification equating (PSE) methods described in Holland (pp. 1-18, this report) are two important classes of observed score equating methods used in the NEAT design. They are briefly described below for the equipercentile case. Chain equating. Chain equating uses a two-stage transformation of X scores into Y scores. First, link X to V on P and then link V to Y on Q. These two linking functions are then functionally composed to map X to Y through V. This method is a valid observed score 22

33 equating method if the following two assumptions, CE1 and CE2, hold (Holland, pp. 1-18, this report). CE1: Given any target population T, the link from X to V is population invariant, so that -1-1 K P F P (x) = K T F T (x). -1 (This makes use of the fact that K T F T (x) is the equipercentile function linking X to V on population T, for any T.) that CE2: Given any target population T, the link from V to Y is population invariant, so G Q -1 K Q (v) = G T -1 K T (v). Applying these two assumptions to the computation of the composed link from X to V to Y, we get -1-1 e XY;T(C) (x) = G Q K Q K P F P (x). See Holland (pp. 1-18, this report) for more details. We note that because the target population, T, cancels out from the formula for the composed function that equates X to Y, e XY;T(C) (x) is assumed to work for any T. In a sense, CE is defined to be population invariant, but this is only strictly true for mixtures of P and Q and not for subpopulations of P or Q. For example, if T 1 were a subpopulation of P, then CE2 would make sense as an untestable assumption, but CE1 would be directly testable and probably refuted because all of the relevant cdfs could be estimated on P (and its subpopulation T 1 ). Post-stratification equating. This method first estimates the marginal distributions of both X and Y on a target population T (that is a specific mixture of P and Q) and then computes the equipercentile equating function. In order to estimate the distribution of X in Q and the distribution of Y in P, the PSE method makes the following assumptions: the conditional distribution of X given V and the conditional distribution of Y given V are population invariant, i.e., 23

34 PSE1: Given a target population T, the conditional distribution of X given V is population invariant, i.e., f T(PS) (x) = f P(x v)k T (v), v where f(x) denotes the score probabilities for X and k(v) the score probabilities for V. PSE2: Given a target population T, the conditional distribution of Y given V is population invariant, i.e., g T(PS) (y) = g P(y v)k T (v), v where g(y) denotes the score probabilities for Y. Population invariance of conditional distributions is the same as conditional independence (given the anchor test score) of the test score and the indicator variable denoting the subpopulation, P or Q. Using PSE1 and PSE2, f T(PS) (x) and g T(PS) (y) are computed, and from these, continuous cdfs, F T(PS) (x) and G T(PS) (y) are formed. Then Y is equated to X on T through: -1 e XY;T(PS) (x) = G T(PS) F T(PS) (x). Note that the equating function, e XY;T(PS) (x), can depend on the choice of T unlike e XY;T(C) (x). Therefore, PSE can be different from CE, though they can also be identical in a particular circumstance that we now discuss. When will CE and PSE both give the same results? One of the important roles of the anchor test in NEAT design is to provide information about differences in the relevant abilities of the examinees in the two subpopulations, P and Q. This is why the anchor test should be appropriately constructed. Brennan and Kolen (1987b) discuss conditions for an appropriate anchor test. Marco, Petersen, and Stewart (1983), Petersen, Marco, and Stewart (1982), and Angoff and Cowell (1985) examined a number of equating methods, with or without an anchor test, varying the similarity of the examinee groups. Other studies focused on matching the anchor for equating (Lawrence & Dorans, 1990; Livingston, Dorans, & Wright, 1990). These empirical studies are summarized well by the observation: The general [ ] finding is that, when the anchor test design is used to equate carefully constructed alternate forms, the groups taking the old and new forms 24

35 are similar to one another, and the common set is a miniature version of the total test form, then equating methods all tend to give similar results. (Kolen, 1990, pp ) The theorem given next is an analytical proof of a part of this statement. More precisely, our first result concerns the case when the anchor test has the same distribution on both P and Q (without any additional assumptions about how similar X and Y are, or about the anchor test being a miniature version of the test ). In this situation we show that, as they are described in the previous section, both CE and PSE will result in exactly the same equating function. In practice, when the distribution of the anchor test is the same or similar in the two groups, it is usually realized that the two groups are nearly equivalent so that ignoring the anchor test and treating P and Q as equivalent will give results similar to anchor test methods. Theorem 1 formalizes this observation. Theorem 1: If, in the NEAT design, we have K P = K Q, then both CE and PSE yield the same equating function and it is e XY;T(C) (x) = e XY;T(PS) (x) = G Q -1 F P (x). Proof: The case for CE is obvious because the composition, K Q K P -1 (x), now equals the identity function, so that it cancels out, i.e., e XY;T(C) (x) = G Q -1 K Q K P -1 F P (x) = G Q -1 F P (x). For the case of PSE, suppose K P = K Q, then the score probabilities satisfy Hence, k T = wk P + (1 w)k Q = k P = k Q, for any T = wp + (1 w)q. f T(PC) (x) = g T(PC) (y) = f P(x v)k T (v)= v g Q(y v)k T (v) = v f P(x v)k P (v) = f P (x), and v g Q(y v)k Q (v) = g Q (y). v Once continuized, we must also have F T(PC) (x) = F P (x), and G T(PC) (y) = G Q (y), from which the result for PSE follows. QED Note that Theorem 1 will also hold for the Tucker and chain linear equating methods (see Holland, pp. 1-18, this report), on the relationship between linear and equipercentile equating functions). 25

36 This theorem shows that CE and PSE are closely connected when the distributions of V are similar on both P and Q, or, in other words, CE and PSE must yield nearly identical results when the two subpopulations are similar in the abilities measured by V. A case that arises in which the distribution of V is the same in both P and Q is when P and Q are the same the Equivalent Groups design with an anchor test. In this case, the use of an anchor test that is highly correlated with X and Y will reduce the standard error of equating for PSE but will have no effect whatsoever on the standard error of equating for CE. The next theorem addresses the case when the distributions of the anchor test can be very different for P and Q, but the anchor test is highly correlated with both X and Y. While the statement of Theorem 2 is probably not the best that can be formulated, we include it because we believe that it formally expresses the often stated conclusion that a high correlation between the anchor and the operational tests is important. Theorem 2: If, in the NEAT design, we have X = V = Y, so that there is a perfect correlation between the anchor test and the other two tests, then both CE and PSE give the same equating function and it is: e XY;T(C) (x) = e XY;T(PS) (x) = x, the identity. Proof: Because X = V = Y, we have F T (x) =G T (x) =K T (x) for any T. that The case for CE follows from the fact that now both G Q -1 K Q (x) = x and K P -1 F P (x) = x so e XY;T(C) (x) = G Q -1 K Q (K P -1 F P (x)) = G Q -1 K Q (x) = x, regardless of how different F P and F Q are. For the case of PSE, note that the conditional score probabilities f P (x v), f Q (x v), g P (y v) and g Q (y v) are all 0 unless x = v = y, and then they equal 1. Then the score probabilities satisfy f T(PC) (x) = f P(x v)k T (v)= k T (x) = f T (x) = wf P (x) + (1 w)f Q (x), and v g T(PC) (y) = g Q(y v)k T (v) = k T (y) = g T (y) = wg P (y) + (1 w)g Q (y) = f T (y). v Hence the two sets of score probabilities are the same. Once continuized, we must also have F T(PC) (x) = G T(PC) (x) from which the result for PSE follows. QED 26