Separate Appendix to: Semi-Nonparametric Competing Risks Analysis of Recidivism

Transcription

1 Separate Appendix to: Semi-Nonparametric Competing Risks Analysis of Recidivism Herman J. Bierens a and Jose R. Carvalho b a Department of Economics,Pennsylvania State University, University Park, PA 1682 b CAEN, Universidade Federal do Ceará, Fortaleza, Brazil May Identification In this section we will re-derive the identification results of Heckman and Honore (1989) and Abbring and Van den Berg (23) for the common heterogeneity case, as follows Parametric identification For t T, let the true conditional probability P [T t, D =1,C = X] be P [T t, D =1,C = X] (1.1) Z t = h exp exp β,1 X Λ,1 (τ)+exp β,2x Λ,2 (τ) exp exp β,1x Λ,1 (τ)+exp β,2x Λ,2 (τ) exp β,1x λ,1 (τ) dτ The paper involved was presented by the first author at the Econometric Society European Meeting 26 in Vienna. The helpful comments of Jaap Abbring are gratefully acknowledged.

2 Suppose there exist a density h on [, 1], parameter vectors β 1, β 2 and hazard functions λ 1 (t) and λ 2 (t) with corresponding integrated hazards Λ 1 (t) and Λ 2 (t) such that for all t T, P [T t, D =1,C = X] (1.2) = Z t h (exp ( (exp (β 1X) Λ 1 (τ)+exp(β 2X) Λ 2 (τ)))) exp ( (exp (β1x) Λ 1 (τ)+exp(β2x) Λ 2 (τ))) exp (β1x) λ 1 (τ) dτ as well. Taking the derivative to t, it then follows that for all t T, h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) (1.3) exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) exp β,1x λ,1 (t) = h (exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t)))) exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t))) exp (β1x) λ 1 (t) a.s. Similarly, if for all t T, is equal to then P [T t, D =2,C = X] Z t = h exp exp β,1 X Λ,1 (τ)+exp β,2x Λ,2 (τ) exp exp β,1x Λ,1 (τ)+exp β,2x Λ,2 (τ) exp β,2x λ,2 (τ) dτ P [T t, D =2,C = X] = Z t h (exp ( (exp (β 1X) Λ 1 (τ)+exp(β 2X) Λ 2 (τ)))) exp ( (exp (β1x) Λ 1 (τ)+exp(β2x) Λ 2 (τ))) exp (β2x) λ 2 (τ) dτ h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) (1.4) 2

3 exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) exp β,2x λ,2 (t) = h (exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t)))) exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t))) exp (β2x) λ 2 (t) a.s. Now suppose that h (1) = h(1) = 1, which corresponds to E[V ]=1. (See Assumption 2) Then, letting t, it follows from (1.3) that exp (β,1 β 1 ) X λ,1 (t) lim =1a.s. (1.5) t λ 1 (t) If λ,1 (t) and λ 1 (t) are Weibull baseline hazards, including scale factors, i.e., λ,1 (t) =α 1,1α 1,2t α 1,2 1, λ 1 (t) =α 1,1 α 1,2 t α 1,2 1, (1.6) where α1,1 and α 1,1 are the scale factors involved, and all the parameters involved are positively valued, then lim t λ,1 (t) λ 1 (t) = α 1,1 α 1,1 α 1,2 α 1,2 lim t t α 1,2 α 1,2 = if α1,2 > α 1,2, α 1,1 α 1,1 if α1,2 = α 1,2, if α1,2 < α 1,2. so that by (1.5), α 1,2 = α 1,2 and X (β,1 β 1 )=ln µ α1,1 α 1,1 a.s. (1.7) Because of the presence of the scale factors α1,1 and α 1,1, we cannot allow a constant in X. Next, suppose that the variance matrix Σ x = E (X E[X]) (X E[X]) is non-singular (c.f. Assumption 3). Then it follows from (1.7) that Σ x (β,1 β 1 )=, hence β,1 = β 1, and thus α 1 = α,1. Thus, in the case that the two baseline hazard functions are Weibull, Assumptions 2-3 guarantee the identification of the parameters. In the case of non-weibull baseline hazards we need more conditions. For example, suppose that λ 1 (t) = 2α 1,1t α 2 1,2 + t 2, λ,1 (t) = 3 2α 1,1t α 1,2 2 + t 2, (1.8)

4 where again α1,1 and α 1,1 are scale factors, and all the parameters are positively valued. These hazard functions are unimodal, with modes at α 1,2 > and α1,2 >, respectively. Then λ,1 (t) lim t λ 1 (t) = α 1,1 α2 1,2 α 1,1 α 2, (1.9) 1,2 hence (1.7) now becomes (β,1 β 1 ) X =ln ³ α 2 1,2 /α1,2 2 ln α 1,1 /α1,1 a.s. (1.1) Under Assumptions 2-3, (1.1) still implies that β,1 = β 1 but now only that α 2 1,2 α 2 = α 1,1. 1,2 α1,1 Thus, in the unimodal hazard case the Assumptions 2-3 do not guarantee identification of the parameters of the unimodal baseline hazard. It is easy to verify that the same problem occurs whenever lim λ,1 (t) /λ 1 (t) (, )\{1} t is possible. On the other hand, Assumptions 2-3 still guarantee that in the case D =1, β,1 = β 1, and similarly in the case D =2that β,2 = β 2. Thus, (1.3) now reads, h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) (1.11) exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) λ,1 (t) = h (exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t)))) exp exp β,1x Λ 1 (t)+exp β,2x Λ 2 (t) λ 1 (t) a.s. and (1.4) reads h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) (1.12) exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) λ,2 (t) = h (exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t)))) exp exp β,1x Λ 1 (t)+exp β,2x Λ 2 (t) λ 2 (t) a.s. 4

5 It follows from (1.11) and (1.12) that for all t<t, λ 2 (t) λ 1 (t) = λ,2 (t) λ,1 (t). (1.13) To see what this result implies for the unimodal case, let similar to (1.8), λ 2 (t) = 2α 2,1t α 2 2,2 + t 2, λ,2 (t) = 2α 2,1t α 2,2 2 + t 2, (1.14) and assume that α1,2 6= α2,2, so that λ,1 (t) and λ,2 (t) are not proportional. Then it follows straightforwardly from (1.13), (1.8) and (1.14) that α 1,2 = α1,2, which implies that λ 1 (t) and λ,1 (t) are proportional, and therefore λ 2 (t) and λ,2 (t) are proportional as well, with common proportionality factor c>, say: λ 1 (t) =c.λ,1 (t), λ 2 (t) =c.λ,2 (t). But then it follows from (1.11) and Assumption 2 that h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) c = lim t h (exp ( (exp (β1x) Λ 1 (t)+exp(β2x) Λ 2 (t)))) exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) exp exp β,1x Λ 1 (t)+exp β,2x Λ 2 (t) = 1, hence λ 1 (t) =λ,1 (t), λ 2 (t) =λ,2 (t). If α 1,2 = α 2,2, which implies that for some constant κ >, λ,2 (t) =κλ,1 (t), then (1.13) implies that λ 2 (t) =κλ 1 (t) as well, but not necessarily that (1.9) holds. Therefore, proportionality of λ,1 (t) and λ,2 (t) has to be excluded, at least for t close to zero: Assumption A.1. Ifthetruebaselinehazards λ,1 (t) and λ,2 (t) are non- Weibull, then they have to be non-proportional in the sense that there exists a small ε > such that for any constant κ > the set {t (, ε) :λ,2 (t) = κ.λ,1 (t)} has Lebesgue measure zero. In general (1.13) and Assumption A.1. are necessary but not sufficient conditions for (1.9), because we can always choose a hazards function λ 1 (t) such that λ 2 (t) defined by µ λ1 (t) λ 2 (t) λ,2 (t). (1.15) λ,1 (t) 5

6 is a valid hazard function. The reason that (1.9) holds for Weibull hazards and the unimodal hazards is that the four hazard functions involved have the same functional forms, which is such that (1.15) implies that λ 1 (t) and λ,1 (t) are proportional. Therefore, we need to require that Assumption A.2. If the baseline hazard functions in the competing risks model are non-weibull then they belong to a class of parametric hazard functions L = {λ(t α), α A} such that for any pair λ,1, λ,2 of non-proportional 1 hazard functions in L, (1.15) can only hold for a pair λ 1, λ 2 L if and only if λ 1 (t) c.λ,1 (t) for some constant c>. Summarizing, we have shown that Theorem A.1. If the baseline hazards are Weibull then under Assumptions 2-3 the parameters of the competing risks model are identified. If the baseline hazards are non-weibull then parameter identification requires the additional conditions in Assumptions A.1 and A Nonparametric identification Under Assumptions 2-4 and A.1-2 it follows now from (1.3) that for t T, h exp exp β,1 X Λ,1 (t)+exp β,2x Λ,2 (t) = h exp exp β,1x Λ,1 (t)+exp β,2x Λ,2 (t) a.s. By a similar argument it can be shown that H exp exp β,1 X Λ,1 T +exp β,2 X Λ,2 T = H exp exp β,1x Λ,1 T +exp β,2 X Λ,2 T a.s. Thus, denoting U = exp exp β,1x Λ,1 (T )+exp β,2x Λ,2 (T ), u = inf P [U u]> 1 As defined in Assumption A.1. 6

7 we have that H (u) =H (u) a.e. on (u, 1]. (1.16) Therefore, at first sight it seems that in the case of right-censoring it may not be true that H(u) =H (u) a.e. on [, 1]. (1.17) This is not a problem if we adopt Assumption 1: Theorem A.2. Given Assumption 1, let q be the smallest natural number for which there exists a δ R q such that h (u) =h q (u δ ) a.e. Then δ is unique: If for some δ R q, hq (u δ )=h q (u δ) a.e. on a set with positive Lebesgue measure, then δ = δ. Moreover, for any q>q and δ R q such that h (u) = h q (u δ) a.e. on a set with positive Lebesgue measure, we have δ =(δ, ). Proof. This result follows straightforwardly from Theorem 4 in Bierens (28). However, Assumption 1 is not necessary for nonparametric identification, but is merely adopted because it allows for standard maximum likelihood inference. In the competing risks case with common unobserved heterogeneity (1.16) does imply (1.17), because for t, H(exp( t)) = R R exp( t.v)dg(v) and H (exp( t)) = exp( t.v)dg (v) are Laplace transforms of the distributions G(v) and G (v), respectively. 2 Lemma A.1. Let H(u) = R u v dg(v) and H (u) = R u v dg (v) for u [, 1], where G(v) and G (v) are distribution functions with non-negative support. If H(u) =H (u) a.e. on an arbitrary interval (u, u) (, 1) then G(v) =G (v) a.e. on [, ), hence H(u) =H (u) a.e. on [, 1]. Proof : First observe that for u (, 1) and non-negative integers m, sup v m u v 1 <. (1.18) v Take the derivative of H(u) and H (u) to u (u, u). Then by (1.18) and dominated convergence we may take the derivatives inside the integrals involved: vu v 1 dg(v) = 2 We are indebted to Jaap Abbring for suggesting this. vu v 1 dg (v). (1.19) 7

8 Multiply (1.19) by u, and then take the derivatives to u (u, u) again, which by (1.18) implies that v 2 u v 1 dg(v) = v 2 u v 1 dg (v). Repeating this procedure it follows by induction that hence Since v m u v dg(v) = kx (t.v) m u v dg(v) = m! m= sup k 1 kx (t.v) m u v m! m= v m u v dg (v) for m =, 1, 2,... (1.2) kx (t.v) m u v dg (v) for k =, 1, 2,... (1.21) m! m= X ( t.v) m exp ( v. ln(1/u)) m! m= = exp(( t ln(1/u)).v) 1 if t < ln(1/u) it follows from (1.21) and bounded convergence that Now denote exp (t.v) u v dg(v) = F (x u) = exp (t.v) u v dg (v) for t < ln(1/u). (1.22) R x uv dg(v) R u v dg(v),f (x u) = R x uv dg (v) R u v dg (v) for u (u, u) and x>. Then it follows from (1.22) and (1.23) that exp (t.v) df (v u) = exp (t.v) df (v u) for t < ln(1/u). (1.23) Hence it follows from the uniqueness of moment-generating functions that F (x u) = F (x u) for u (u, u) and x>, and thus Z x u v dg(v) = 8 Z x u v dg (v). (1.24)

9 Moreover, similar to (1.2) it follows from (1.24) that for x>, m,k=, 1, 2,... and u (u, u), Z x Z x v m+k u v dg(v) = v m+k u v dg (v), hence Z x v m dg(v) = = = = Thus, for x> and m =, 1, 2,..., Z x X v m k= X (ln(1/u)) k k= k! X (ln(1/u)) k k= Z x (v.i(v x)) m dg(v) = k! v m dg (v). (v. ln(1/u)) k u v dg(v) k! Z x Z x v m+k u v dg(v) v m+k u v dg (v) (v.i(v x)) m dg (v), (1.25) where I(.) is the indicator function. Now use the well-known fact that distributions of bounded random variables are equal if and only if all their moments are equal. Then, with V arandom drawing from G(v) and V a random drawing from G (v), it follows from (1.25) that for x, y >, P [V.I(V x) y] =P [V.I(V x) y]. This implies that for <y<x, G (x) G(y) =G (x) G (y). Hence, letting x, it follows that G (y) =G (y) for y> and thus by right-continuity of distribution functions, G (v) =G (v) for v. Q.E.D. 9

10 2. Empirical results 2.1. Initial estimation and test results Table A.1: Initial ML results Parameters F = F =1 (i = F + 1) Estimates t-values Estimates t-values β i,1 (MALE) β i,2 (BLACK) β i,3 (RELEASE) β i,4 (AGE) β i,5 (SENT) α i, α i, q =6 N = L.L. = Table A.2: Logit results for felony arrest, F =1 P arameters Original Logit Dif f erence Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 9979 L.L. = ICM test:

11 Table A.3: ML results with state fixed effects Parameters F = F =1 (i = F + 1) Estimates t-values Estimates t-values β i,1 (MALE) β i,2 (BLACK) β i,3 (RELEASE) β i,4 (AGE) β i,5 (SENT) β i,6 (Florida) β i,7 (Illinois) β i,8 (Michigan) β i,9 (Minnesota) β i,1 (New Jersey) β i,11 (New York) β i,12 (Ohio) β i,13 (Oregon) α i, α i, q =6 N = L.L =

12 Table A.3: Logit results for felony arrest, F =1, with state fixed effects Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 β 1, SENT β 2,6 β 1, Florida β 2,7 β 1, Illinois β 2,8 β 1, Michigan β 2,9 β 1, Minnesota β 2,1 β 1, New Jersey β 2,11 β 1, New York β 2,12 β 1, Ohio β 2,13 ³ β 1, Oregon ln α2,1 α 2, α 1,1 α 1,2 n = 9979 L.L. = ICM test: Results per state Table A.4: Logit results for felony arrest, F =1: California Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1,3 N.A. N.A. N.A N.A RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 817 LL. = ICM test:

13 Table A.5: Logit results for felony arrest, F =1: Florida Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 115 LL. = ICM test: 1.31 Table A.6: Logit results for felony arrest, F =1: Illinois Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 96 L.L. = 66.4 ICM test:.93 Table A.7: Logit results for felony arrest, F =1: Michigan Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 843 L.L. = ICM test:

14 Table A.8: Logit results for felony arrest, F =1: Minnesota Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 85 L.L. = ICM test: 1.6 Table A.9: Logit results for felony arrest, F =1: New Jersey Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 941 L.L. = ICM test: 1.54 Table A.1: Logit results for felony arrest, F =1: NewYork Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 948 L.L. = ICM test:

15 Table A.11: Logit results for felony arrest, F =1: Ohio Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 736 L.L. = ICM test: 1.44 Table A.12: Logit results for felony arrest, F =1: Oregon Parameters Original Logit Difference Logit s.e. V ariables α 2,2 α 1, ln(t ) β 2,1 β 1, MALE β 2,2 β 1, BLACK β 2,3 β 1, RELEASE β 2,4 β 1, AGE β 2,5 ³ β 1, SENT ln α2,1 α 2, α 1,1 α 1,2 n = 784 L.L. = ICM test: The SNP densities The following plots of the SNP densities h q δ have the same scale, [, 9] [, 1], in order to make them comparable. The flatter the density, the less dependent the misdemeanor and felony recidivism durations are, conditional on the covariates. To explain the shape of these densities, recall that the true density h(u) is h(u) = vu v 1 dg(v), where G(v) is the distribution function of the common unobserved heterogeneity variable V. Also, recall that the identification condition E[V ]=1corresponds to h(1) = 1, which has been imposed on h q δ as well. Therefore, h q ³1 b δ =1. 15

16 Moreover, E[V ]=1and P [V =1]< 1 imply that P [V <1] >, whichinitsturn implies that lim u h(u) =. Althoughthislimitcannotbeattainedbyh q δ for finite q, itexplainstheshapeofh q δ close to u =. Moreover, it seems that for California, Illinois, Michigan and New York the density h q δ is zero for a u (, 1), whereas the true density h(u) cannot be zero, because h(u) = for some point u (, 1) implies P [V = ] = 1, which violates the condition E[V ]=1. However, in these cases the minimum value of h q δ is very small but positive. For example, in the case of Michigan u =argmin u 1 h 4 ³ δ =.18, with h 4 u b δ =.616. To explain this phenomenon, suppose that V takes only two values, P [V = λ] =p, P [V = µ] =1 p where < λ < 1 <µ<. The condition E[V ]=1implies µ = 1 λp 1 p hence h(u) =λp.u λ 1 +(1 λp)u (1 λ)p/(1 p) (2.1) The first-order condition for an extremum of h(u) in u is = λ (1 λ) p.u λ 2 p(1 λp) +(1 λ) u (1 λ)p/(1 p) 1 1 p hence µ Ã (1 p)/(1 λ) µ! 1 p 1/(1 λ) λ (1 p) 1 p u = = λ 1 p (2.2) 1 λp 1 λp Substituting this expression in (2.1) yields h (u )= µ 1 λp 1 p 1 p λ p = λ.u λ 1 (2.3) To show that h (u ) can get close to zero for some u bounded away from zero, let for a given constant c (, 1), λ = c 1/(1 p). 16

17 Then lim u = lim p 1 p 1 Ã µ 1 p 1/(1 c 1 p c! 1/(1 p) ) 1 c 1/(1 p) p = c. lim p 1 (1 p) 1 p = c and lim h (u )=limλ/c 1 λ = p 1 λ More generally, if h q δ takes a minimum in u and h q ³u b δ is small then λ and p can be chosen such that the density (2.1) takes a minimum in u, with h q ³u b δ = h(u ). Of course, this is not the complete story, because in all cases h q δ has two extrema rather than only one in this example. However, it is too difficult to construct a distribution of V that can explain the second extremum as well. 17

18 Figure 2.1: SNP density h 6 δ for California 18

19 Figure 2.2: SNP density h 3 δ for Florida 19

20 Figure 2.3: SNP density h 4 δ for Illinois 2

21 Figure 2.4: SNP density h 4 δ for Michigan 21

22 Figure 2.5: SNP density h 3 δ for Minnesota 22

23 Figure 2.6: SNP density h 5 δ for New Jersey 23

24 Figure 2.7: SNP density h 4 δ for New York 24

25 Figure 2.8: SNP density h 3 δ for Ohio 25

26 Figure 2.9: SNP density h 5 δ for Oregon 26

27 References Abbring, J. H., and G. J. van den Berg (23), The Identifiability of the Mixed Proportional Hazards Competing Risks Model, Journal of the Royal Statistical Society B, 65, Bierens, H. J. (28): Semi-Nonparametric Interval-Censored Mixed Proportional Hazard Models: Identification and Consistency Results, Econometric Theory 24, Heckman, J. J., and B. E. Honore (1989). The Identifiability of the Competing Risks Model. Biometrika 76,