Comments on Detecting Outliers in Gamma Distribution by M. Jabbari Nooghabi et al. (2010)

Transcription

1 Comments on Detectng Outlers n Gamma Dstrbuton by M. Jabbar Nooghab et al. (21) M. Magdalena Lucn Alejandro C. Frery September 17, 215 arxv:159.55v1 [stat.co] 16 Sep 215 Ths note shows that the results presented by Jabbar Nooghab et al. (21) do not hold n all expected cases. Wth ths, the technque proposed by Kumar and Lalhta (212) for detectng upper outlers n Gamma samples s also not vald. Specfcally, ths note shows that the probablty densty functons (pdf) under the null hypothess of the test statstcs theren proposed are not always vald. In the aformentoned works the authors propose test statstcs to detect outlers n Gamma samples usng a test of dscordancy for outlers framework as defned n Barnett and Lews (1994). Followng the approach of Barnett and Lews (1994), the null hypothess (H ) of a test for dscordancy s a statment of an ntal probablty model that explans the data generatng process. For nstance, n the case here consdered, H states that data are generated as ndependent observatons from a common dstrbuton F. If F s a Gamma dstrbuton, as n Jabbar Nooghab et al. (21) and Kumar and Lalhta (212), then H : X 1, X 2,..., X n are n ndependent random varables, each followng a Gamma dstrbuton wth shape parameter m > and scale parameter σ >, denoted by Γ(m, σ), whose probablty densty functon (pdf) s gven by f(x; m, σ) = 1 Γ(m)σ m xm 1 exp ( x ), x >. σ As σ s a scale parameter, wthout loosng generalty, t wll be assumed from now on that these random varables are dstrbuted accordng to a Γ(m, 1) law, that s, wth pdf gven by f(x; m) = 1 Γ(m) xm 1 exp ( x), x >. The alternatve hypothess used n Jabbar Nooghab et al. (21) and Kumar and Lalhta (212) s the slppage alternatve. We are nterested n detectng 1 k < n upper outlers usng Z k, the statstc proposed by Kumar and Lalhta (212). Ths statstc, after some computatons, can be wrtten as n j=n k+1 Z k = (n j + 1)Y j n j=2 (n j + 1)Y, (1) j where Y j = X (j) X (j 1), (2) Facultad de Cencas Exactas, Naturales y Agrmensura, Unversdad Naconal del Nordeste, Av. Lbertad 546, 34 Correntes, and CONICET, Argentna, magdalenalucn@gmal.com Alejandro C. Frery s wth the LaCCAN, Unversdade Federal de Alagoas, Av. Lourval Melo Mota, s/n, Maceó AL, Brazl, acfrery@gmal.com 1

2 X (j) denotes the j-th order statstcs of the ordered sample from (X ) 1 n n nondecreasng order, that s, X (1) X (2) X (n), and k s the number of observatons suspected to be upper outlers. As n any statstcal test, once the test statstc s proposed we need to determne rejecton crtera related to a prevously specfed sgnfcance level. To do that, and to compute the p-value assocated to a sample, the dstrbuton of the test statstc under the null hypothess must be known. In Kumar and Lalhta (212) the dstrbuton of Z k under the null hypothess was obtaned based, manly, on the dstrbuton of dfferences of subsequent order statstcs from Gamma random varables,.e., the dstrbuton of the Y j gven n Eq. (2). However, when performng smulatons we observed that the emprcal pdf of Z k under the null hypothess gven by Kumar and Lalhta (212) gave a proper adjustment only for m = 1, that s, when the random varables (X ) 1 n follows an exponental law. Jabbar Nooghab et al. (21) also used the random varables Y j to fnd the pdf of the test statstc they proposed under the alternatve (Theorem 3.1) and null (Corollary 3.1) hypotheses. Kumar and Lalhta (212), followed the very same reasonng and methodology used n Theorem 3.1 of Jabbar Nooghab et al. (21) to derve the pdf of Z k under the null hypothess. A strong assumpton made n both works s that, under the null hypothess, each Y j follows a Γ(m, (n j + 1) 1 ) dstrbuton. Ths s not true when m 1, as we show n what follows. Recall that under the null hypothess of a test for dscordancy, X 1,..., X n are ndependent dentcally dstrbuted Gamma random varables. In general, f X 1,..., X n are ndependent dentcally dstrbuted random varables the pdf of Y sr = X (s) X (r) can be found by solvng the followng ntegral (Davd and Nagaraja, 23): n! f Ysr (y) = (r 1)!(s r 1)!(n s)! F r 1 (x)f(x)[f (x + y) F (x)] s r 1 f(x + y)[1 F (x + y)] n s dx, (3) where F and f are the cumulatve dstrbuton functon and the pdf, respectvely, of any of the X (wthout sortng). Replacng s by j and r by j 1 n Eq. (3), the pdf of Y j = X (j) X (j 1) can be found by solvng the followng ntegral f Yj (y) = n! (j 2)!(n j)! F j 2 (x)f(x)f(x + y)[1 F (x + y)] n j dx. (4) Let us suppose that the sample s only composed by two random varables X 1 and X 2, each Γ(m, 1) dstrbuted wth shape parameter m N. Then n = 2, and we just have to compute. Makng n = 2, and j = 2 n Eq. (4), and havng n mnd than m N, after some computatons (see Appendx) the pdf of Y 2 can be wrtten as (y) = exp( y) m 1 = ( m 1 ) Γ(2m 1) 2 2(m 1) y, y >. (5) As already mentoned, a strong assumpton made by Jabbar Nooghab et al. (21) and by Kumar and Lalhta (212) s that f X 1, X 2 are random varables dstrbuted accordng to a Γ(m, 1) law then Y 2 Γ(m, 1). But, f for nstance m = 2 and usng Eq. (5), 2

3 the pdf can be expressed as (y) = 1 2( exp( y) + y exp( y) ), y >. (6) Ths s a composton of a Γ(1, 1) and a Γ(2, 1) dstrbutons wth same probablty, and not a Γ(2, 1) dstrbuton as clamed by both Jabbar Nooghab et al. (21) and by Kumar and Lalhta (212). The dscrepancy s notorous, as wll be shown henceforth. Algorthm 1 presents the pseudocode used for the dscusson. We mplemented t n the R programmng language R Core Team (214), and run t wth R = 1 replcatons for each case of m {1, 3, 8}. Algorthm 1: Pseudocode for the analyss of Y 2. Data: Read m, R, and the pseudorandom number generator seed. Intalze Z of length R; Intalze r = 1; for 1 r R do Obtan X = (X 1, X 2 ) from the Γ(m, 1) dstrbuton; Sort X and obtan X (1) X (2) ; Compute ; Store Z(r) = Y 2 ; Update r = r + 1; Analyze Z; Fgure 1 presents the results obtaned wth ths smulaton wth R = 1 4 : hstograms of Y 2 and the denstes proposed by Jabbar Nooghab et al. (21) and Kumar and Lalhta (212) (dashed lnes), and the one we obtaned and presented n Eq. (5) (sold lnes). Case m = 1 Case m = 3 Case m = (a) X Γ(1, 1) (b) X Γ(3, 1) (c) X Γ(8, 1) Fgure 1: The pdf of Y 2 assumed by Jabbar Nooghab et al. (21) and by Kumar and Lalhta (212) n dashed lnes, and n sold lnes the pdf gven n Eq. (5) Both denstes concde n the case m = 1,.e., when X 1, X 2 follow untary mean Exponental dstrbutons; cf. Fg. 1(a). Fgures 1(b) and 1(c) show the dscrepancy between the observed data and the model clamed by Kumar and Lalhta (212). The data s well ft by the dstrbuton we obtaned, though. 3

4 Conclusons In ths work we have shown that f X 1, X 2 are ndependent random varables, each Γ(m, 1) dstrbuted, wth m N 2, then the pdf of Y 2 = X (2) X (1) s a composton of m Gamma dstrbutons, and not a Γ(m, 1) law as clamed by Jabbar Nooghab et al. (21) and then assumed by Kumar and Lalhta (212). Therefore, wth ths counterexample we conclude that f m N 2 then Y j, as n Eq. (2), does not follow a Gamma dstrbuton. Ths mples that most computatons presented by Jabbar Nooghab et al. (21) and by Kumar and Lalhta (212) are not vald, ncludng the pdf of Z k gven by Kumar and Lalhta (212). Appendx From Eq. (4) (y) = = 2 = 2! (2 2)!(2 2)! 2 f(x)f(x + y)dx = 2 exp( y) F 2 2 (x)f(x)f(x + y)[1 F (x + y)] 2 2 dx exp( x)x m 1 exp( (x + y))(x + y) m 1 dx exp( 2x)(x 2 + xy) m 1 dx. Havng n mnd that m N, expandng the bnomal (x 2 + xy) m 1 and usng that x n e ax dx = a (n+1) Γ(n + 1), follows that { [ (y) = 2 exp( y) + ( ] } m 1 exp( 2x) )y x (2(m 1) j) dx = { = 2 exp( y) ( [ m 1 + ] )y } x (2(m 1) j) exp( 2x)dx = [ = exp( y) ( ) ] m 1 Γ(2m 1) y. 2 2(m 1) = Incdentally, the expresson gven for the Dxon s D k statstc by both the artcles commented n ths work are wrong. They state that D k = (X (n) X (n k) )/X (n) when, n fact, t s D k = X (n) X (n k) X (n) X (1), the rato of the gap to the range of the data. References Barnett, V. and T. Lews (1994), Outlers n Statstcal Data, thrd edton. John Wley and Sons. 4

5 Davd, H.A. and H.N. Nagaraja (23), Order Statstcs, thrd edton. John Wley and Sons. Jabbar Nooghab, M., H. Jabbar Nooghab, and P. Nasr (21), Detectng outlers n gamma dstrbuton. Communcatons n Statstcs-Theory and Methods, 39, Kumar, N. and S. Lalhta (212), Testng for upper outlers n gamma sample. Communcatons n Statstcs-Theory and Methods, 41, R Core Team (215), R: A Language and Envronment for Statstcal Computng. R Foundaton for Statstcal Computng, Venna, Austra. 5