Redistricting Simulation through Markov Chain Monte Carlo

Transcription

1 Redistricting Simulation through Markov Chain Monte Carlo Kosuke Imai Department of Government Department of Statistics Institute for Quantitative Social Science Harvard University SAMSI Quantifying Gerrymandering Workshop October 8, 2018 Joint work with Benjamin Fifield, Michael Higgins, Jun Kawahara, and Alexander Tarr Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 1 / 27

2 Motivation and Progress of Our Team s Efforts Redistricting simulation: detect gerrymandering assess impact of constraints (e.g., population, compactness, race) In 2013 when our team started working on the project, many optimization methods existed but there were surprisingly few simulation methods no theoretical justification for standard random seed and grow algorithms Need a simulation method that: 1 samples uniformly from a target population of redistricting maps 2 incorporates common constraints 3 scales to redistricting problems of moderate and large size Paper presented at the 2014 Political Methodology Summer Meeting Open-source R package redist first published at CRAN in May 2015 Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 2 / 27

3 Overview of the Talk 1 Markov chain Monte Carlo algorithms 2 Validation studies 3 Empirical studies Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 3 / 27

4 The Random Seed-and-Grow Algorithm Cirincione et. al (2000), Altman & McDonald (2011), Chen & Rodden (2013): 1 Randomly choose a precinct as a seed for each district 2 Identify precincts adjacent to each seed 3 Randomly select adjacent precinct to merge with the seed 4 Repeat steps 2 & 3 until all precincts are assigned 5 Swap precincts around borders to achieve population parity Modify Step 3 to incorporate compactness No theoretical properties known The resulting maps may not be representative of the population Local exploration is difficult Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 4 / 27

5 Redistricting as a Graph-Cut Problem Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 5 / 27

6 Step 1: Independently Turn On Each Edge with Prob. q Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 6 / 27

7 Step 2: Gather Connected Components on Boundaries Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 7 / 27

8 Step 3: Select Subsets of Components and Propose Swaps Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 8 / 27

9 Step 4: Accept or Reject the Proposal Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 9 / 27

10 Step 4: Accept or Reject the Proposal Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 10 / 27

11 The Theoretical Property of the Algorithm We prove that the algorithm samples uniformly from the population of all valid redistricting plans An extension of the Swendsen-Wang algorithm (Barbu & Zhu, 2005) Metropolis-Hastings move from plan v t 1 vt : ( ) α(v t 1 vt ) = min 1, π(v t v t 1 ) π(v t 1 vt ) min (1, ) (1 q) B(C,v) B(C,v ) where q is the edge cut probability and B(C, v) is # of edges between connected component and its assigned district in redistricting plan v Easy to calculate Exact Metropolis ratio is too costly to evaluate, but approximation appears to work well Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 11 / 27

12 Incorporating a Population Constraint Want to sample plans where ψ(v k ) = p k p 1 ɛ where p k is population of district k, p is average district population, ɛ is strength of constraint Strategy 1: Only propose valid swaps slow mixing Strategy 2: Oversample certain plans and then reweight 1 Sample from target distribution f rather than the uniform distribution: ( ) f (v) g(v) = exp β V k v ψ(v k ) where β 0 and ψ(v k ) is deviation from parity for district V k 2 (Approximate) Acceptance probability is still easy to calculate, ( α(v v g(v ) ) ) min 1, (1,v) B(C,v ) q) B(C g(v) 3 Discard invalid plans and reweight the rest by 1/g(v) Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 12 / 27

13 Additional Constraints 1 Compactness (Fryer and Holden 2011): ψ(v k ) i,j V k,i<j p i p j d 2 ij where d ij is the distance between precincts i, j 2 Similarity to the adapted plan: ψ(v k ) = r k 1 where r k (rk ) is the # of precincts in V k (V k of the adapted plan) 3 Any criteria where constraint can be evaluated at each district g(v) = exp β (w 1 ψ 1 (V k ) + w 2 ψ 2 (V k ) + + w L ψ L (V k )) V k v r k Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 13 / 27

14 Improving the Mixing of the Algorithm Single iteration of the proposed algorithm runs very quickly But, like any MCMC algorithm, convergence may take a long time 1 Swapping multiple connected components more effective than increasing q but still leads to low acceptance ratio 2 Simulated tempering (Geyer and Thompson, 1995) Lower and raise the temperature parameter β as part of MCMC Explores low temperature space before visiting high temperature space 3 Parallel tempering (Geyer 1991) Run multiple chains of the algorithm with different temperatures Use the Metropolis criterion to swap temperatures with adjacent chains Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 14 / 27

15 Validation Studies based on Florida Data Evaluate algorithms when all valid plans can be enumerated # of precincts: 25 and 50 # of districts: 2 and 3 for the 25 set, and 2 for the 50 set With and without a population constraint of 20% within parity Also, consider simulated and parallel tempering Comparison with the standard random seed-and-grow algorithm via the BARD package (Altman & McDonald 2011) 10,000 draws for each algorithm Republican Dissimilarity Index for each simulated plan: D = 1 2 n w i R i R i=1 R(1 R) Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 15 / 27

16 Our Algorithm vs. Standard Algorithm Unconstrained Simulations Constrained Simulations 25 Precinct Set Density True Distribution Our Algorithm Standard Algorithm Density Republican Dissimilarity Index Republican Dissimilarity Index 50 Precinct Set Density Density Republican Dissimilarity Index Republican Dissimilarity Index Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 16 / 27

17 Simulated and Parallel Tempering Unconstrained Simulations Constrained Simulations (20%) Constrained Simulations (10%) Without Tempering (Algorithm 1) Density True Distribution Our Algorithm (Hard Constraint) Standard Algorithm Density Density Republican Dissimilarity Index Republican Dissimilarity Index Republican Dissimilarity Index With Tempering (Algorithms 2 & 3) Density True Distribution Simulated Tempering Parallel Tempering Density Republican Dissimilarity Index Republican Dissimilarity Index Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 17 / 27

18 More Validation Studies based on Florida Data 26.7 Fully Enumerated Validation Maps for Redistricting Simulation 25 Precinct Validation Subset 70 Precinct Validation Subset Population Latitude Longitude Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 18 / 27

19 enumpart (Kawahara et al. 2017) Quickly Computes the Total Number of Solutions 1e+09 How Solutions Change with Map Size and Constraints 2 Districts 3 Districts 4 Districts Number of Solutions 1e e+09 1e e+09 1e+06 No Constraint 5% 1% Number of Precincts Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 19 / 27

20 Total Number of Solutions and Population Parity # < 1% Parity: 8 # < 5% Parity: 192 # < 10% Parity: 927 Distribution of Population Parity for Fully Enumerated Maps 25 Precinct Validation Subset 9e+05 # < 1% Parity: # < 5% Parity: # < 10% Parity: Precinct Validation Subset Number of Plans e+05 3e e Distance from Population Parity Total number of solutions without population constraint 1 25 precinct (3 districts) example: 117, precinct (2 districts) example: 44,082,156 Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 20 / 27

21 Validation Results 1 Divide 70-precincts into 2 districts: no constraint, 5%, 1% 2 4 MCMC chains for 50,000 iterations each: with and without simulated tempering 3 Run 200,000 iterations of random seed-and-grow algorithm Validation Exercises on 70 Precinct Validation Map No Constraint 5% Parity 1% Parity Density Truth MCMC MCMC (Simulated Tempering) RSG Republican Dissimilarity Index Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 21 / 27

22 Empirical Studies Apply algorithm to state election data: New Hampshire: 2 congressional districts, 327 precincts Mississippi: 4 congressional districts, 1,969 precincts 1% (NH) and 5% (MS) deviation from population parity Convergence diagnostics: Autocorrelation Trace plot Gelman-Rubin multiple chain diagnostic New Hampshire Kosuke Imai (Harvard) Mississippi Redistricting through MCMC SAMSI (Oct. 8, 2018) 22 / 27

23 New Hampshire: Tempering Works Better Autocorrelation of a Chain Trace of a Chain Gelman Rubin Diagnostic Hard Population Constraint (Algorithm 1) Autocorrelation Lag Republican Dissimilarity (logit transformed) Iterations shrink factor last iteration in chain Parallel Tempering (Algorithm 2) Autocorrelation Lag Republican Dissimilarity (logit transformed) Iterations shrink factor last iteration in chain Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 23 / 27 median 97.5%

24 Mississippi: Parallel Tempering, More Challenging Case Autocorrelation of a Chain Trace of a Chain Gelman Rubin Diagnostic Republican Dissmilarity Autocorrelation Lag Republican Dissimilarity (logit transformed) Iterations shrink factor median 97.5% last iteration in chain African American Dissimilarity Autocorrelation Lag African American Dissimilarity (logit transformed) Iterations shrink factor last iteration in chain Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 24 / median 97.5%

25 Redistricting Plans that are Similar to the Adapted Plan Question: How does the partisan bias of the adapted plan compare with that of similar plans? Local exploration Fraction of Precincts Switched Partisan Bias towards Republicans Partisan Bias New Hampshire Fraction of Precincts Switched Absolute Deviation Absolute Deviation from Partisan Symmetry Fraction of Precincts Switched Republican Seats Number of Republican Seats Fraction of Precincts Switched Partisan Bias towards Republicans Mississippi Fraction of Precincts Switched Absolute Deviation Fraction of Precincts Switched Republican Seats Fraction of Precincts Switched Partisan Bias towards Republicans Partisan Bias New Hampshire Fraction of Precincts Switched Absolute Deviation Absolute Deviation from Partisan Symmetry Fraction of Precincts Switched Republican Seats Number of Republican Seats Fraction of Precincts Switched Partisan Bias towards Republicans Mississippi Fraction of Precincts Switched Absolute Deviation Fraction of Precincts Switched Republican Seats Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 25 / 27

26 Concluding Remarks Scholars use simulations to characterize the distribution of redistricting plans Commonly used algorithms lack theoretical properties and speed Our MCMC algorithm has: better theoretical properties superior speed better performance in validation studies can do global exploration for small states and local exploration for other states Future research: more validation studies more diffused starting maps larger states with more districts and precincts apply the method to historical redistricting data Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 26 / 27

27 References 1 Paper at 2 R package at 3 Comments and suggestions: Imai@Harvard.Edu Kosuke Imai (Harvard) Redistricting through MCMC SAMSI (Oct. 8, 2018) 27 / 27