An Ordinal Approach to Decomposing Test Score Gaps. David M. Quinn University of Southern California

Transcription

1 An Ordinal Approach to Decomposing Test Score Gaps David M. Quinn University of Southern California Andrew D. Ho Harvard Graduate School of Education

2 Background and Purpose The estimation of racial/ethnic and socioeconomic test score gaps plays an important role in monitoring educational inequality. Since the Coleman Report (Coleman et al., 1966), researchers have decomposed gaps into within-and between school portions in order to learn about the potential sources of gap formation, and where resources might be targeted to narrow gaps (Hanushek & Rivkin, 2006). A limitation of existing decomposition methods is that they are parametric in nature; as such, they rely on the interval test scale assumption, which is often questionable and difficult to verify (Reardon, 2008; Dominigue, 2014). When the interval scale assumption cannot be verified, any monotonic scale transformation is permissible (Reardon, 2008). This is problematic because analytic results can vary meaningfully across such transformations (Bond & Lang, 2013). Research into the extent to which results are sensitive to test score transformations in various settings has grown recently, as has the development of approaches for addressing this concern (Barlevy & Neal, 2012; Bond & Lang, 2013; Briggs & Domingue, 2013; Cunha & Heckman, 2008; Nielsen, 2015; Reardon, 2008). In this study, we develop and evaluate ordinal methods for decomposing test score gaps. These methods offer a transformation-invariant decomposition that can be applied in any situation in which the interval properties of the outcome cannot be established. Research Design Methods We extend the V-gap framework, a framework for estimating transformation-invariant test score gaps (Ho, 2009; Ho & Reardon, 2012), by applying ordinal techniques inspired by Reardon s (2008) three-part parametric gap decomposition, which establishes: δ = β 1(1 VR ) + β 1VR + β 2VR (1) where δ is the total Black-White gap (as an illustrative example), β 1 is the mean within-school gap (i.e., from a school fixed effects model), VR is the variance ratio index of segregation, and β 2 is the effect of school proportion Black on student test scores (see Appendix A for detail). Our decomposition method involves mapping the empirical cumulative distribution functions (ECDFs) by race within schools to the total within-school ECDF. In our simulations (described below), we take the following approach (assuming a Black/White population): 1) Divide the full sample by q quantiles (in simulations below, q=10) and assign each student the number corresponding to the bin in which they fall. 2) Within each school s, find p (all) sb,the proportion of students whose scores fall into each bin b. 3) Across all schools and bins, multiply p (all) sb by the total number of Black (White) students sampled in school s to get the (approximate) number of Black (White) students who would be in each bin b if the Black (White) distribution in school s equaled the overall distribution in school s, or n (Black) sb, n (White) sb. 4) Use n (White) sb and n (Black) sb from (3) as weights in a model estimating V. This V will represent the total between-school V gap. This is the ordinal analogue to Reardon s (2008) total between-school gap. We will call this V btwn. We also apply alternative procedures in which we map the Black to White, or White to Black, ECDF within each school in step 2. After these mappings, the resulting V gaps are ordinal analogues to the residual gap that remains after fitting a school fixed effects model, or what Reardon (2008) calls the unambiguously between-school portion of the gap. However,

3 unlike in Reardon s (2008) parametric framework, our non-parametric approach yields different unambiguously-between school gaps depending on which racial group s distribution is treated as (B to the reference. We call these V W) (W to btwn and V B) btwn, with superscripts indicating whether the Black ECDF was mapped to the White ECDF within school, or vice versa. Simulations We employ a multilevel model-based approach to simulate samples from a population with known values for parametric and non-parametric gaps and gap decompositions (see Appendix B for detail on the model and on how we calculate true population values). For each simulated sample, we: 1) Estimate the overall V total (using Stata s rocfit routine): V total = 2Φ 1 (P(b > w)), where Φ 1 is the inverse of the standard normal CDF and P(b > w) is the probability that a random black student will score higher than a random white student. For comparison, we also estimate V total (quant), or V after assigning students quantile scores as described above. 2) Generate new scores for students by mapping ECDFs within schools using the procedures described above. 3) Using the new scores from (2), obtain new V statistics, yielding estimates of the ordinal total between-school gap (V btwn ) or the ordinal unambiguously between school gaps (B to (V btwn W) (W to and V btwn B) ) 4) We also present the parametric gap estimates for each simulated data set: β 1, β 2, the total parametric population gap (δ ), and the total parametric between-school gap (δ btwn ). In Table 1, we present the parameter values we use for β 1 and β 2 across simulations, along with the parametric and non-parametric parameter values that correspond to each set of values for β 1 and β 2. Results In Table 2, we present simulation results for the range of parameter values. As seen in the top panel, V and V (quant) are unbiased across parameter values (in each simulation set, estimates are never statistically different from the true parameter values). For the ordinal decomposition elements V btwn, (B to W) V btwn, and (W to B) V btwn bias is small but increases slightly as parameter values increase; RMSD (bottom panel) also rises slightly with parameter values. The (B to W) bias-to-true-value ratio (middle panel) may increase with parameter values for V btwn. Bias in is minimal, however; as a reference point, the bias, bias-to-true-value ratio, and RMSD V btwn for V btwn are similar to the respective values for its parametric analogue, δ btwn.

4 References Barlevy, G., & Neal, D. (2012). Pay for percentile. American Economic Review, 102(5), Bond, T. N., & Lang, K. (2013). The evolution of the Black-White test score gap in Grades K 3: The fragility of results. Review of Economics and Statistics, 95(5), Briggs, D.C., & Domingue, B. (2013). The gains from vertical scaling. Journal of Educational and Behavioral Statistics, 38(6), DOI: / Coleman, James S., Ernest Q. Campbell, Carol J. Hobson, James McPartland, Alexander M. Mood, Frederic D. Weinfeld, and Robert York Equality of Educational Opportunity. Washington, D.C. Cunha, F., & Heckman, J.J. (2008). Identifying and estimating the technology of cognitive and noncognitive skill formation. Journal of Human Resources, 43, Domingue, B. (2014). Evaluating the equal-interval hypothesis with test score scales. Psychometrika, 79(1), DOI: /S Hanushek, Eric A., and Steven G. Rivkin School Quality and the Black-White Achievement Gap. (No. w12651). National Bureau of Economic Research. Ho, Andrew D A Nonparametric Framework for Comparing Trends and Gaps Across Tests. Journal of Educational and Behavioral Statistics, 34: Ho, A.D., & Reardon, S.F (2012). Estimating achievement gaps from test scores reported in ordinal proficiency categories. Journal of Educational and Behavioral Statistics, 37(4), Nielsen, E.R. (2015). Achievement estimates and deviations from cardinal comparability. Federal Reserve Working Paper. Retrieved from: Reardon, Sean F Thirteen Ways of Looking at the Black-White Test Score Gap. Working paper, Stanford University. Retrieved from: Reardon, S.F., & Ho, A.D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40(2),

5 Table 1. Parameter Values for Simulations. Determining Parameters Parametric Ordinal (unambig) δ btwn V btwn (B to W) V btwn (W to B) V btwn β 1 β 2 δ δ btwn V

6 Table 2. Results from Simulations with Varying Parameter Values. Parameters Parametric β 1 β 2 δ V β 1 β 2 δ δ btwn Simulation Results V btwn (B to W) V btwn Ordinal (W to B) V btwn V (quant) V Bias Bias/Parameter Ratio ND ND ND ND ND ND ND ND ND RMSD

7 Note. ND=not defined. RMSD=root mean square deviation. Simulated school-level sample size=500; within-school sample size=25. Each set of parameter values was run for 5000 simulations; simulated data sets that did not converge were discarded.

8 Appendix A. Reardon s (2008) Three-part Gap Decomposition Our method is an ordinal analogue to Reardon s (2008) three-part gap decomposition. Reardon showed that the overall black-white test score gap in the population, δ, could be decomposed as: δ = β 1(1 VR ) + β 1VR + β 2VR (1) In (1), β 1 and β 2 are estimated from the model (with only white and black students): Y is = β 0 + β 1 Black is + β 2 Black s + ε is, (2) where Y is is the test score of student i in school s, Black is an indicator that the student is black, and Black s is the proportion of sampled students in school s who are black. VR in (1) is the estimated variance ratio index of segregation, which can be expressed as the difference in school proportion black between the average black student and the average white student: Black s (black) Black s (white). Reardon calls the first term on the RHS of (1) the unambiguously within school gap, the center term the ambiguous gap, and the last term the unambiguously between school gap. The unambiguously between-school gap is the portion of the gap that would remain if black and white mean performance were equalized within schools without altering the relationship between school proportion black and student test scores. The unambiguously within school gap is the portion of the gap that would be closed if mean performance between black and white students were equalized within schools, without changing the overall school mean. In Reardon s (2008) three-part decomposition, anchoring the fitted lines for black and white students to the same y-intercept (without changing their slopes) results in the closure of the total within-school gap, and the remaining black-white difference is the unambiguously betweenschool gap (or what would be left over if the within-school gap from a school fixed effects model were closed). In the parametric decomposition, the resulting unambiguously between-school gap would be the same if the black fitted line were anchored to the y-intercept for the white fitted line, vice versa, or any other location.

9 Appendix B. Drawing Samples for Simulations. We begin with a school-level population distribution of school proportion black (similar to that found in the ECLS-K:99). We randomly sample 500 values from this distribution, and assign each of these schools true school-by-race test score means using: Μ s (b) ~N(μ =.11 β 1 β 2 P s (B = 1), σ =.026) Μ s (w) ~N(μ =.11 β 2 P s (B = 1), σ =.011) where Μ s (b) represents the true mean in school s for black students and Μ s (w) represents the same for white students, β 1 and β 2 are as defined in equation 1 above, and P s (B = 1) is the sampled true school proportion black. After establishing a random sample of school-by-race means in this way, we draw a random sample of 25 students from each school and assign each a race (where B i = 1 means the student is black and B i = 0 means the student is white), where each race draw is a Bernoulli trial with probability equal to the true school proportion black. Then, for each student i in school s, we draw a test score, conditional on the student s race and schoolby-race mean: Y is (B i = 1, Μ s (b) = μs (b) )~ N(μs (b), σ =.97) Y is (B i = 0, Μ s (w) = μs (w) )~ N(μs (w), σ =.89) We vary β 1 and β 2 by simulation in order to understand how the method performs under different population values. Calculating Population Values for Ordinal Decomposition For each value of school probability black in our distribution, school-by-race means are normally distributed. Therefore, assuming equally-sized schools for simplicity, the overall (i.e., across schools) white mean at a given school probability black is β 2 (E(B)) and the overall black mean is.11 + β 2 (E(B)) + β 1. By the law of total variance, the overall racespecific variances (conditional on school probability black) are sums of the within-school variances and the variance of the means. With the overall race-specific means and variances at each school probability black, we find the overall population CDF for white students by applying the formula for the CDF of the mixture of normal distributions 1 : (w) 2(w) ) σ p F white (x) = w (w) p Φ ( x μ p p (B1) where p indexes a particular school probability black, Φ is the normal CDF, w (w) p is a weight giving the proportion of the total white population that attends schools with school proportion black p, or w p (w) = 1 P(b) p (1 P(b) p ) p, μ p (w) is the true white mean across schools with probability black p, and σ p 2(w) is the true white variance across schools with probability black p. Similarly, we find the overall population PDF for black students using the formula for the PDF of a mixture of normals: f black (x) = w (b) s ϕ(x, μ (b) 2(b) p s, σ s ), (B2) where φ is the normal PDF and the weight w s (b) = P(b) s s P(b) s. This gives a total V of: 1 We used the nor1mix package in R to find all mixture PDFs and CDFs.

10 V total = 2Φ 1 (w) (w) x μ ( w s Φ ( s (b) p 2(w) )) ( w s ϕ(x, (b) 2(b) p μs, σ s ) ) dx (B3) σ s To solve for the true value of the total between-school V (V btwn ), we find the overall black PDF and the overall white CDF when the black and white CDFs at each school probability black are equal to the total CDF at that school probability black. We find the overall black PDF by applying the mixture of normal using weight P(B) P(B) to each conditional (on school 2000 probability black) black distribution (where P(B) is the school probability black, and 2000 is the normalizing constant for the distribution) and P(B) (1 P(B)) to each conditional white 2000 distribution. To find the overall white CDF, we apply weights 1 P(B) P(B) to the conditional black parameters and 1 P(B) (1 P(B)) to the conditional white parameters. This yields, for 2000 each racial group, the mixture distribution for the overall population that would result if the within-school distributions for each racial group matched the observed combined (black/white) distribution within school. We then find V as in B3, using these newly weighted overall distributions by race. Mapping Black Distributions to White Distributions Within Schools and Vice Versa When mapping the black CDF within school to the white CDF, we keep all white parameters unchanged; we also keep all black parameters unchanged for students in schools with p(b) s = 1. For black students in schools where p(b) s 1, we assign them the parameter values of white students in the same school; weighting their new parameter values by w (b) s, we apply the (B to formulas above and find V W) btwn. Following a similar but reversed procedure to map the white distribution to the black distribution within school, we find V btwn 2000 (W to B).