DISCUSSION PAPER 2016/43

Transcription

1 I N S T I T U T D E S T A T I S T I Q U E B I O S T A T I S T I Q U E E T S C I E N C E S A C T U A R I E L L E S ( I S B A ) DISCUSSION PAPER 2016/43 Bounds on Kendall s Tau for Zero-Inflated Continuous Variables Denuit, M. and M. Mesfioui

2 BOUNDS ON KENDALL S TAU FOR ZERO-INFLATED CONTINUOUS VARIABLES MICHEL M. DENUIT Institut de statistique, biostatistique et sciences actuarielles (ISBA) Université Catholique de Louvain B-1348 Louvain-la-Neuve, Belgium MHAMED MESFIOUI Département de mathématiques et d informatique Université du Québec à Trois-Rivières Trois-Rivières (Québec) Canada G9A 5H7 October 5, 2016

3 Abstract In this short note, we derive the lower and upper bounds on the association measure for zeroinflated continuous random variables proposed by Pimentel et al. (2015). These bounds only involve the respective probability masses at the origin. This suggests simple normalizations to achieve all values in [ 1, 1]. Keywords: Association measure, bivariate zero-inflated data, Frechet-Hoeffding bounds.

4 1 Introduction Pimentel et al. (2015) proposed an estimator of Kendall s tau for bivariate zero-inflated data. This estimator possesses a natural interpretation and improved statistical properties. As pointed out by these authors, their association measure cannot attain the values -1 and +1. In this paper, we derive the lower and upper bounds on this newly proposed association measure. As expected, these bounds only depend on the respective probability masses at zero, and not on the continuous conditional distributions on R + = (0, ). Let us adopt the same notation as in Pimentel et al. (2015). In this paper, we consider non-negative random variables X having a continuous distribution on R + with probability density function h and a positive probability mass at 0, P[X = 0] = p 1 > 0. The distribution function of X is given by 0 if s < 0 F (s) = p 1 if s = 0 (1.1) F h (s) = p 1 + (1 p 1 ) s h(x)dx if s > 0. 0 Recall that the generalized inverse of F is defined by (α) = inf {x 0 F (x) α}, α [0, 1]. In our setting, it can be explicitly expressed as 0 if 0 α p (α) = (1.2) h (α) if p < α 1 where h denotes the inverse of the function F h. 2 Kendall s tau for bivariate zero-inflated data Let (X, Y ) be a pair of random variables with joint distribution function H, i.e. H(x, y) = P[X x, Y y]. Suppose further that X and Y are zero-inflated continuous and nonnegative with respective distribution functions F and G, and positive probability masses at 0. Specifically, the distribution function F of X is given in (1.1) and G is defined similarly, with P[Y = 0] = G(0) = p 2 > 0 and G h (s) = p 2 + (1 p 2 ) s 0 k(x)dx for s > 0, where k is the probability density function of Y given Y > 0. Denuit and Lambert (2005) and Mesfioui and Tajar (2005) studied general versions of the population Kendall s tau. In the context of zero-inflated continuous and non-negative 1

5 random pairs (X, Y ), Pimentel et al. (2015) have established an interesting and useful formula of the population Kendall s tau that is recalled next. Let us denote as Now, define a random variable distributed as X given that Y = 0 a random variable distributed as X given that Y > 0 Y 01 a random variable distributed as Y given that X = 0 Y 11 a random variable distributed as Y given that X > 0 X 1 a random variable distributed as X given X > 0 Y 1 a random variable distributed as Y given Y > 0. p 00 = P[X = 0, Y = 0] p 10 = P[X > 0, Y = 0] p 01 = P[X = 0, Y > 0] p 11 = P[X > 0, Y > 0] p 1 = P[ > ] p 2 = P[Y 01 > Y 11 ] τ 11 = Kendall s tau of (X 1, Y 1 ). The association measure studied in Pimentel et al. (2015) is then given by τ = p 2 11τ (p 00 p 11 p 01 p 10 ) + 2p 11 ( p10 (1 2p 1) + p 01 (1 2p 2) ) (2.1) In the next section, we derive bounds on (2.1) in terms of the marginal distributions of X and Y. 3 Bounds on the association measure (2.1) As pointed out by Pimentel et al. (2015), the association measure (2.1) cannot reach the values ±1 when p 1 > 0 or p 2 > 0. Our aim is to derive the range of admissible values for the association measure (2.1). This can be done by using the property of monotonicity of Kendall s tau with respect to the concordance order; see e.g. Mesfioui and Tajar (2005) and Denuit and Lambert (2005). This means that if H 1 and H 2 are two bivariate distribution functions such that H 1 (x, y) H 2 (x, y) for all x and y, then τ H1 τ H2. Hence, the sharp lower and upper bounds of Kendall s tau can be derived using Fréchet-Hoeffding bounds. In other words, the desired lower bound τ min and upper bound τ max on (2.1) are obtained when the random pair (X, Y ) obeys the distribution function min{f (x), G(y)} for τ max and max{f (x) + G(y) 1, 0} for τ min, respectively. The next result establishes the expression of these bounds in terms of the probabilities p 1 and p 2. 2

6 Proposition 3.1. The lower and upper bounds on the association measure (2.1) are given by τ max = 1 max{p 2 1, p 2 2} and where x + = max{x, 0}. τ min = (1 p 1 p 2 ) 2 + 2(1 p 1 )(1 p 2 ) Proof. We know from the Proposition 2.3 in Mesfioui and Tajar (2005) that the upper bound τ max on (2.1) is obtained when the random pair (X, Y ) obeys the upper Frechet-Hoeffding bound, that is, when X = (U) and Y = G 1 (U), where U is a random variable uniformly distributed over the unit interval [0, 1]. Clearly, (1.2) shows that X = (U) = h (U)I[p 1 < U 1] and Y = G 1 (U) = G 1 k (U)I[p 2 < U 1], where I[A] is the indicator of the event A, equal to 1 when A is realized and to 0 otherwise. Suppose without loss of generality that p 1 p 2 holds. Then, p 00 = P[X = 0, Y = 0] = P[U p 1 ] = p 1 p 10 = P[X > 0, Y = 0] = P[p 1 < U p 2 ] = p 2 p 1 p 01 = P[X = 0, Y > 0] = 0 p 11 = P[X > 0, Y > 0] = P[U > p 2 ] = 1 p 2. It remains to calculate p 1. To this end, we first derive the distribution function of the random variable : has no probability mass at zero as P[ = 0] = P[X = 0 Y > 0] = 0 and with F X11 (x) = P[X x Y > 0] = P[X x, Y > 0] 1 p 2 so that P[X x, Y > 0] = P[X x, U > p 2 ] F X11 (x) = = P[ h (U) x, U > p 2] = P[p 2 < U F h (x)] 0 if x h (p 2) F h (x) p 2 1 p 2 if x > h (p 2). Therefore, the quantile function of is ( ) (α) = h (1 p2 )α + p 2, α [0, 1]. (3.1) 3

7 Similar arguments imply that the distribution function and the quantile function of are respectively given by 0 if x < 0 F F X10 (x) = h (x) p 2 if 0 x h (p 2) 1 if x > h (p 2) and (α) = 0 if α [ ] 0, p 1 p 2 h (αp 2) if α [ ] p 1 p 2, 1. We see from (3.1) and (3.2) that (α) (α) for all α [0, 1], so that p 1 = P [ (U) > (U) ] = 0. (3.2) Moreover, τ 11 = 1 when the random variables X and Y are perfectly positively dependent. We then get from (2.1) that τ max = p p 00 p p 11 p 10 = (1 p 2 ) 2 + 2p 1 (1 p 2 ) + 2(1 p 2 )(p 2 p 1 ) = 1 p 2 2, as announced. Let us now establish the expression for the lower bound τ min. To that end, we first calculate the probabilities p 00, p 10, p 01 and p 11 when X and Y obey the lower Frechet- Hoeffding bound, that is, X = (U) and Y = G 1 (1 U) where U is uniformly distributed over the unit interval [0, 1]. We distinguish two cases: either 1 p 1 p 2 < 0 p 1 + p 2 > 1. In this situation, one observes It follows from (2.1) that p 00 = P[X = 0, Y = 0] = P[U p 1, 1 U p 2 ] = P[1 p 2 U p 1 ] = p 1 + p 2 1 p 10 = P[X > 0, Y = 0] = P[U > p 1, 1 U p 2 ] = P[U > p 1 ] = 1 p 1 p 01 = P[X = 0, Y > 0] = P[U p 1, 1 U > p 2 ] = P[U 1 p 2 ] = 1 p 2 p 11 = P[X > 0, Y > 0] = P[U > p 1, 1 U > p 2 ] = 0. τ min = 2p 01 p 10 = 2(1 p 1 )(1 p 2 ). (3.3) 4

8 or 1 p 1 p 2 0. Then one has p 00 = P[X = 0, Y = 0] = 0 p 10 = P[X > 0, Y = 0] = p 2 p 01 = P[X = 0, Y > 0] = p 1 p 11 = P[X > 0, Y > 0] = 1 p 1 p 2. It remains to calculate p 1 and p 2. To do so, let us derive the distribution of as follows: F X10 (x) = P[X x Y = 0] = P[X x, Y = 0] p 2 with P[X x, Y = 0] = P[X x, 1 U p 2 ] = P[X x, U 1 p 2 ] = P[ h (U) x, U 1 p 2] = P[U F h (x), U 1 p 2 ] so that and F X10 (x) = 0 if x h (1 p 2) F h (x)+p 2 1 p 2 if x > h (1 p 2) Likewise, the distribution of is given by (α) = h (p 2α + 1 p 2 ), α [0, 1]. (3.4) F X11 (x) = P[X x Y > 0] = P[X x, Y > 0] 1 p 2 with P[X x, Y > 0] = P[X x, 1 U > p 2 ] = P[X x, U < 1 p 2 ] = P[U p 1 ] + P[U F h (x), p 1 < U < 1 p 2 ] so that F X11 (x) = 0 if x < 0 F h (x) 1 p 2 if 0 x h (1 p 2) 1 if x h (1 p 2) 5

9 and (α) = 0 if α [ 0, ] p 1 1 p 2 h (α(1 p 2)) if α [ ] p 1 1 p 2, 1. From (3.4) and (3.5), we see that (α) (α) for all α [0, 1]. Thus, p 1 = P [ (U) > (U) ] = 1. (3.5) Similar arguments lead to p 2 = 1. Also, τ 11 = 1 when X and Y are perfectly negatively dependent. It then follows from (2.1) that τ min = p p 01 p 10 2p 11 (p 10 + p 01 ) = (1 p 1 p 2 ) 2 2p 1 p 2 2(1 p 1 p 2 ) (p 1 + p 2 ) = p p = (1 p 1 p 2 ) 2 2(1 p 1 )(1 p 2 ). (3.6) Finally, from (3.3) and (3.6), one deduces that { 2(1 p1 )(1 p τ min = 2 ) if 1 p 1 p 2 < 0 (1 p 1 p 2 ) 2 2(1 p 1 )(1 p 2 ) if 1 p 1 p 2 0 which can be reduced to This ends the proof. τ min = (1 p 1 p 2 ) 2 + 2(1 p 1 )(1 p 2 ). Figure 3.1 displays the lower and upper bounds on (2.1) in terms of the probabilities p 1 and p 2. We see there that the lower bound increases in terms of p 1 and p 2 and reach the minimum value 1 at (p 1, p 2 ) = (0, 0) and the maximum value 0 when p 1 = 1 or p 2 = 1. The upper bound decreases in terms p 1 and p 2 and achieves the maximum value 1 at (p 1, p 2 ) = (0, 0) and the minimum value 0 when p 1 = 1 or p 2 = 1. Considering Proposition 3.1, the measure of association (2.1) can be modified into τ τ min if τ 0 τ = τ τ max if τ > 0 to ensure that the modified τ reaches all values in [ 1, +1]. 6

10 Figure 3.1: Lower bound (left panel) and upper bound (right panel) on (2.1) in terms of (p 1, p 2 ) [0, 1] 2. Acknowledgements Michel Denuit acknowledges the financial support from the contract Projet d Actions de Recherche Concertées No 12/ of the Communauté française de Belgique, granted by the Académie universitaire Louvain. Mhamed Mesfioui acknowledges the financial support of the Natural Sciences and Engineering Research Council of Canada No References Denuit, M., Lambert, P. (2005). Constraints on concordance measures in bivariate discrete data. Journal of Multivariate Analysis 93, Mesfioui, M., Tajar, A. (2005). On the properties of some nonparametric concordance measures in the discrete case. Nonparametric Statistics 17, Pimentel, R. S., Niewiadomska-Bugaj, M., Wang, J. C. (2015). Association of zero-inflated continuous variables. Statistics and Probability Letters 96,