Problems with Penalised Maximum Likelihood and Jeffrey s Priors to Account For Separation in Large Datasets with Rare Events

Transcription

1 Problems with Penalised Maximum Likelihood and Jeffrey s Priors to Account For Separation in Large Datasets with Rare Events Liam F. McGrath September 15, 215 Abstract When separation is a problem in binary dependent variable models many researchers use Firth s penalised maximum likelihood in order to obtain finite estimates (Firth, 1993; Zorn, 25). In this letter I show that this approach can lead to inferences in the opposite direction to that of the separation when the number of observations are sufficiently large and both the dependent and independent variables are rare events. As large datasets with rare events can often occur in political science, such as dyadic data measuring war and treaty formation, a lack of awareness of this problem can lead to inferential issues. Simulated data and an empirical illustration show that the use of independent prior distributions centred at zero, for example the Cauchy prior suggested by Gelman et al. (28), do not result in the problems associated with Firth s penalised maximum likelihood and are thus preferred for point estimates in all cases. Thanks to Carlisle Rainey, Jonah Gabry, and Ben Goodrich for comments and suggestions. Postdoctoral Researcher at Centre for Comparative and International Studies (CIS) and Institute for Environmental Decisions (IED), ETH Zürich. Contact liam.mcgrath@ir.gess.ethz.ch

2 1 Introduction The existence of separation in binary choice models, where an independent variable perfectly predicts a binary dependent variable, is a problem within political science. The default response to this problem suggested by Zorn (25) is the use of penalised maximum likelihood, equivalent to the use of a Jeffrey s prior (Jeffreys, 1946), developed by Firth (1993). In this letter I demonstrate that the reliability of this approach to separation is dependent upon features of the data analysed. The penalised maximum likelihood approach leads to point estimates in the opposite direction to that of the separation, when the number of observations are sufficiently large and the binary dependent and independent variables responsible for separation are rare events. Dyadic data on countries over time is such an example of a large dataset, with a number of observations that is usually in the tens to hundreds of thousands. Furthermore in these cases researchers often focus on rare events both as dependent and independent variables, such as interstate war, possession of nuclear weapons, and the signing of preferential trade agreements. This is a result of the Jeffrey s prior resulting in non-independent prior densities for parameters, which can lead to high prior density for parameters opposite the direction of separation in the case of rare events. To demonstrate these problems I use simulated data and analyse the replication of research on the effect of nuclear weapons upon war by Bell and Miller (213). In contrast to Jeffrey s prior/firth s penalised maximum likelihood which can often lead to inferences in the opposite direction to that of the separation, independent priors such as the Cauchy prior suggested by Gelman et al. (28) ensure that the point estimate remains in the correct direction for all specifications. Researchers should therefore be mindful of the potential of Firth s method to lead to incorrect inferences, and if a default estimator is required instead use the Cauchy prior approach of Gelman et. al. 2 The Problem with Penalised Maximum Likelihood and Jeffrey s Prior Suppose that we are faced with negative quasi-complete separation. That is there exist no observations in the 2 2 table, displayed in table 1, such that x = 1 y = 1. 2

3 Table 1: The Observed Data x = x = 1 y = n 1 n 2 y = 1 n 3 In this single covariate case Firth s Jeffrey s prior approach is equivalent to adding.5 to each cell in table 1 (Zorn, 25), resulting in table 2. Table 2: The Result of Using Jeffrey s Prior x = x = 1 y = n n y = 1 n This solution becomes problematic when it does not maintain the feature that there are relatively more observations where y = 1 when x = than when x = 1. For inferences about the effect of x on y to have the correct sign, it must hold that: n n 1 + n >.5 n (1) From this we can see that rare events in large datasets can violate this inequality. When y is a rare event, then n 3 is small. If x is also a rare event, then n 1 will be very large whilst n 2 remains small. This is important as if n 2 is small, then the addition of.5 to all cells will lead the relative frequency of y = 1 when x = 1 to be larger than when x =, in spite of the data suggesting the opposite. This is due to this relevant proportion being strongly affected due to n 2 being small, whilst the proportion for when x = is mostly unaffected due to n 1 being so large. Table 3: Two Nuke Dyads and War, data from Rauchhaus (29) Not Both Nuclear Both Nuclear No War 454, War 62 3

4 As an illustration consider the case of war between two nuclear powers. Table 3 displays the appropriate frequencies using data from Rauchhaus (29). 1 The relationship between these two variables shows quasi-complete separation, with no observations where both countries had nuclear weapons and went to 62 war. The relevant relative frequencies in this case are p 1 = and p 11 = =. 85 Suppose we were to use the Firth penalised maximum likelihood estimator to examine this bivariate relationship. As noted previously this is akin to adding.5 to each cell. This makes the new relative frequencies: p = and p 11 =.5.6. Thus this approach to dealing with separation 85+1 results in the relative frequency of war in nuclear dyads being six times larger than in dyads where both states do not simultaneously have nuclear weapons. The prior information supplied in the form of the Jeffrey s prior therefore leads to opposite conclusions of the relationship between nuclear weapons and interstate war. More concretely, whilst Jeffrey s prior is non-informative with regard to the baseline probabilities it is ultimately informative about the parameters. This is because the use of Jeffrey s prior results in prior distributions for the constant term and parameter for x that are not independent. Jeffrey s prior gives more (less) prior mass for large positive coefficients for x than equally large negative coefficients for x, when evaluated at a large negative (positive) value for the constant term. This ensures that the joint prior distribution remains uninformative with regard to the baseline probability. This becomes problematic because y is a rare event. As a result the likelihood is high at large negative values for the constant, the point where Jeffrey s prior assigns more mass to large positive values for the coefficient of the variable that leads to separation. Therefore the prior leads to the estimate for the coefficient on x to be positive, even though this is in the direction opposite to that of the separation. Figure 1 illustrates this for the case where n 1 = 5, n 2 = 1, n 3 = 1. From the first panel we can see that Jeffrey s prior gives more mass to positive values for the coefficient on x relative to equally negative values when the constant is negative. As the likelihood has high density at the point where the constant has a large negative value, the Jeffrey s prior pulls the posterior towards being positive even though there is negative separation. Thus the uninformative nature of the Jeffrey s prior leads to inferences opposite to the direction of the 1 This is the original data where the Kargil war is not considered a war. For problems related to this coding see Bell and Miller (213). 4

5 (b) Jeffrey's Prior and the Likelihood (a) Jeffrey's Prior 1 1 Prior Density β 1 5 β Constant Parameter 1 1 Constant Parameter Figure 1: (a) the joint prior density for Jeffrey s prior overlaid with contours of the prior. (b) the prior density for Jeffrey s prior overlaid with the contours of the likelihood. The point indicates the resulting estimate from the penalised maximum likelihood. separation. This problem is resolved if researchers instead use independent prior distributions centred at zero. The Cauchy prior suggested by Gelman et al. (28) is one such prior. 2 They advocate using independent Cauchy distributions as priors for parameters, with a location of zero and scale of 2.5 for independent variables and 1 for the constant. This retains the attractive property that the posterior point estimate of the coefficient for the variable that leads to separation is never in a direction opposite to the separation. 3 Figure 2 illustrates the use of independent Cauchy distributions. As can be seen from the prior, equally large positive and negative values for the coefficient on 2 Whilst suggested by Gelman et al. (28) as a default prior, other research suggests either assessing the sensitivity of inferences to prior choices (Rainey, 214) or instead the use of a different prior distribution as a default prior (Ghosh, Li, and Mitra, 215). In the case of the specific problem outlined here what is important is the use of prior distributions upon the parameters of interest, that are independent and centred at zero, and not on the baseline probability are used as a default. 3 The most extreme case would be as n 1, whilst n 2 and n 3 remain fixed and n 4 =. In this case the likelihood increasingly provides little information. Thus the posterior is approximately the prior, which is centred at zero. 5

6 (b) Cauchy Priors and the Likelihood (a) Cauchy Priors 1 1 Prior Density.1 β.2.3 β Constant Parameter 1 1 Constant Parameter Figure 2: (a) the joint prior density for the Cauchy priors suggested by Gelman et al. (28), overlaid with contours of the prior. (b) the joint prior density for the Cauchy priors overlaid with the contours of the likelihood. The point indicates the resulting posterior mode. x have equal prior density independent of the value of the constant. As a result at the point where the likelihood has high density for the constant, the posterior remains in the direction of the separation as movements towards positive values for the coefficient on x lead to lower prior density. Thus the use of an independent prior density centred at zero ensures that inferences in terms of point estimates remain in the direction of the separation, unlike those of the Jeffery s prior/penalised maximum likelihood. To further illustrate this problem I estimate Firth s penalised maximum likelihood estimator on hypothetical data. 4 Following the definition of cell frequencies in table 1 I create datasets from all possible combinations of: n 1 ={1, 5, 1, 15, 2, 25, 3, 35, 4, 45, 5} n 2 ={5, 1, 2} n 3 ={5, 1, 2} 4 For all estimations of this form in the paper I use the function logistf from the logistf package in R (Heinze et al., 213). 6

7 n β Firth 2 n n1 1 β Cauchy Prior 2 3 n n1 Figure 3: The estimated effects using Firth s penalised maximum likelihood and Gelman et. al s Cauchy prior when x and y are rare events and there is negative quasi-complete separation. The x-axis denotes the number of observations where x = y =, the columns indicate the number of observations where x = 1 y =, and lines denote the number of observations where x = y = 1. I also estimate models using independent prior distributions with a Cauchy prior distribution for β (with location and scale 2.5) suggested by Gelman et al. (28). 5 Figure 3 displays the results of estimating these two models on the simulated data. We can see that when y = 1 x = and y = x = 1 are rare events relative to y = x =, the penalised maximum likelihood estimator can lead to coefficients in the opposite (positive) direction to that of the separation. In contrast using the Cauchy prior ensures that the coefficient on x remains in the same direction of the separation although this parameter asymptotically 5 For all estimations of this form in the paper I use the function bayesglm from the arm package in R (Gelman and Su, 215). This function uses an approximate Expectation-Maximisation algorithm to find the posterior mode and variance. For other posterior summaries, such as the posterior mean or median, researchers should write the model in software for full Bayesian inference such as MCMC pack (Martin, Quinn, and Park, 211) or Stan (Stan Development Team, 214). The drawback in doing so for the rare events in large data sets case discussed here is that this is much slower than the EM algorithm used in the bayesglm function. 7

8 approaches zero. 6 3 Illustration - The Effect of Nuclear Weapons upon Interstate War Given the above results I use the replication of Rauchhaus (29) by Bell and Miller (213) to illustrate the problem with actual data. Bell and Miller (213) correctly note that the estimations of Rauchhaus (29) suffer from separation, and so instead estimate models using Firth s penalised maximum likelihood. Table 4 shows the results of models estimating the bivariate relationship (models 1 and 2), their main estimations (models 3 and 4), and models that do not include the major power variable (models 5 and 6). 7 Table 4: Replication of Bell and Miller (213) (1) (2) (3) (4) (5) (6) Firth Cauchy Prior Firth Cauchy Prior Firth Cauchy Prior Constant [-9.15,-8.65] [-9.15,-8.65] [-6.7,-1.51] [-6.4,-1.7] [-8.2,-3.54] [-7.94,-3.52] twonukedyad [-3.33,3.44] [-4.12,3.36] [-5.64,1.76] [-4.15,1.89] [-3.54,3.41] [-4.9,3.25] onenukedyad [.17,1.66] [.21,1.62] [1.38,2.61] [1.34,2.56] logcapabilityratio [-.9,-.4] [-.85,-.38] [-.61,-.18] [-.59,-.16] Ally [-1.17,.25] [-1.1,.25] [-1.9,.29] [-1.4,.32] SmlDemocracy [-.14,-.1] [-.13,-.1] [-.13,] [-.12,] SmlDependence [ ,-26.93] [-21.73,-2.63] [ ,14.26] [ ,23.69] logdistance [-.95,-.42] [-.94,-.43] [-.68,-.16] [-.69,-.18] Contiguity [2.16,3.71] [2.17,3.64] [2.73,4.31] [2.71,4.26] NIGOs [-.5,-.1] [-.5,-.1] [-.6,-.2] [-.6,-.2] MajorPower [1.54,3.14] [1.5,3.1] n Setting n 1 = 1 1 7, n 2 = 1, n 3 = 1 yields a coefficient of for the Cauchy prior and for the Firth penalised maximum likelihood estimator. Thus even in a case that is unlikely to occur in applied statistics, the Cauchy prior still retains the property of the coefficient for x being in the same direction of the separation. 7 To ensure consistency of observations across the different models I first listwise delete observations based upon the model with the largest number of covariates (models 3 and 4) before estimating the subsequent models. 8

9 The results show that whilst the coefficient is in the same direction of the separation in the full model (model 3), the bivariate model (model 1) and the model without the major powers variable (model 5) lead to the opposite inference. Whilst this is not a problem in this specific application, it is entirely possible that researchers are faced with a situation where no such covariate exists to ensure the coefficient is in the right direction. 8 Therefore researchers should not rely on Firth s penalised maximum likelihood as a default solution to separation in binary choice models. Rather they should use independent prior distributions centred at zero, such as the Cauchy prior suggested by Gelman et al. (28). 9 It should also be noted that despite the coefficient remaining in the same direction to that of the separation when using the Cauchy prior, the distribution of the estimated coefficient has substantial mass in the opposite direction of the separation. In models 2, 4, and 6 approximately 42%, 23% and 41% of the distribution includes positive values, the opposite direction of the observed separation. Thus whilst independent prior distributions are preferred as a default with regard to point estimates, further attention to tailoring the prior to the specific research question and quantity of interest is also desirable as noted by Rainey (214). 4 Conclusion In this letter I have shown that the commonly suggested penalised maximum likelihood approach to separation (Zorn, 25) can lead to point estimates opposite to the direction of separation dependent upon features of the data analysed. This occurs when confronted with separation in datasets with a large number of observations where the dependent and independent variables of interest are rare events. Therefore Firth s penalised maximum likelihood/jeffrey s prior is not a suitable default choice for dealing with separation. Instead the use of independent prior distributions centred at zero, such as the Cauchy prior suggested by Gelman et al. (28), which do not suffer from this problem are preferred as a default. 8 In fact this problem was discovered in a statistical analysis which led to inference opposite to the direction of separation. Importantly there did not exist an independent variable that changed this incorrect inference. 9 This is easily implemented in R using the bayesglm function within the arm package. 9

10 References Bell, Mark S., and Nicholas L. Miller Questioning the Effect of Nuclear Weapons on Conflict. Journal of Conflict Resolution. Firth, David Bias Reduction of Maximum Likelihood Estimates. Biometrika 8(1): pp Gelman, Andrew, Aleks Jakulin, Maria Grazia Pittau, and Yu-Sung Su. 28. A Weakly Informative Default Prior Distribution for Logistic and Other Regression Models. The Annals of Applied Statistics 2(4): Gelman, Andrew, and Yu-Sung Su arm: Data Analysis Using Regression and Multilevel/Hierarchical Models. Ghosh, J., Y. Li, and R. Mitra On the Use of Cauchy Prior Distributions for Bayesian Logistic Regression. ArXiv e-prints (July). Heinze, Georg, Meinhard Ploner, Daniela Dunkler, and Harry Southworth logistf: Firth s bias reduced logistic regression. Jeffreys, Harold An Invariant Form for the Prior Probability in Estimation Problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences 186(17): pp Martin, Andrew D., Kevin M. Quinn, and Jong Hee Park MCMCpack: Markov Chain Monte Carlo in R. Journal of Statistical Software 42(9): 22. Rainey, Carlisle Dealing with Separation in Logistic Regression Models. Working Paper. Rauchhaus, Robert. 29. Evaluating the Nuclear Peace Hypothesis: A Quantitative Approach. Journal of Conflict Resolution 53(2): Stan Development Team RStan: the R interface to Stan, Version Zorn, Christopher. 25. A Solution to Separation in Binary Response Models. Political Analysis 13(2):