Statististics of the SK estimator

Transcription

1 New Jersey Institute of Technology E-mil: Dle E. Gry New Jersey Institute of Technology E-mil: Spectrl Kurtosis (SK is sttisticl pproch for detecting nd removing rdio frequency interference (RFI in rdio stronomy dt. In this study, the sttisticl properties of the SK estimtor re investigted nd ll moments of its probbility density function re nlyticlly determined. These moments provide mens to determine the til probbilities of the estimtor tht re essentil to defining the thresholds for RFI discrimintion. It is shown tht, for number of ccumulted spectr M 24, the first SK stndrd moments stisfy the conditions required by Person Type IV probbility density function (PDF, which is shown to ccurtely reproduce the observed distributions. The cumultive function (CF of the Person Type IV is then found, in both nlyticl nd numericl forms, suitble for ccurte estimtion of the til probbilities of the SK estimtor. This sme frmework is lso shown to be pplicble to the relted Time Domin Kurtosis (TDK estimtor, whose PDF corresponds to Person Type IV when the number of time-domin smples is M 46. The PDF nd CF re determined for this cse lso. RFI mitigtion workshop - RFI2010, Mrch 29-31, 2010 Groningen, the Netherlnds Speker. This work ws supported by NSF grnt AST to NJIT c Copyright owned by the uthor(s under the terms of the Cretive Commons Attribution-NonCommercil-ShreAlike Licence.

2 1. Introduction The Spectrl Kurtosis estimtor (ŜK ws originlly proposed by Nit et l. [7] s sttisticl tool for rel-time rdio frequency interference (RFI detection nd excision in Fst Fourier Trnsform (FFT rdio spectrogrph. The first spectrogrph designed for ŜK, the Koren Solr Rdio Burst Loctor [KSRBL; 1], demonstrted the effectiveness of the SK lgorithm, but lso reveled the need for more ccurte clcultion of the theoreticl RFI detection thresholds thn initilly proposed. Consequently, Nit & Gry [8] derived the exct nlyticl expressions for the sttisticl moments of ŜK nd, bsed on its first four stndrd moments, ssigned to it Person Type IV probbility curve [9], which ws shown to be in very good greement with the Monte Crlo simulted ŜK probbility distribution function (PDF, s well s with the distribution derived from direct experimentl observtions mde with the KSRBL instrument [2]. As extensively described in the previous ppers, wht mkes n SK spectrogrph with N spectrl chnnels distinct from trditionl one is the fct tht it ccumultes not only set of M instntneous power spectrl density (PSD estimtes, denoted S 1, but lso the squred spectrl power denoted S 2. These sums, which hve n implicit dependence on frequency chnnel f k, re used to compute the verged power spectrum P = S 1 /M, s well s the quntity ŜK = M + 1 ( MS2 M 1 S1 2 1, (1.1 which is cumulnt-bsed estimtor of the spectrl vribility corresponding to the signl prent popultion, Vk 2 = σ k 2 µ k 2, (1.2 where µ k nd σk 2 re the frequency-dependent PSD popultion mens nd vrinces, respectively. For normlly distributed time domin signl, i.e. n RFI-free signl, Nit & Gry [8] showed tht the estimtor given by eqution (1.1 is unbised, i.e. E(ŜK = Vk 2 = 1. This presenttion compiles the min results of Nit & Gry [8] leding to the definition of ŜK given by eqution (1.1, nd provides prcticl guide for computing the RFI thresholds with predefined flse lrm probbility (PFA level. 2. The Unbised Spectrl Kurtosis nd Time Domin Kurtosis Estimtors The unbised estimtor ŜK my be derived bsed on the properties of the generlized gmm distribution [GGD, 10] defined s f (x,,d, p = pxd 1 e ( x p d Γ(d/p, (2.1 where Γ(z = 0 tz 1 e t dt is the well known Euler s Gmm function. The expected smple moments bout origin corresponding to GGD function re given by E(x n = Γ( d+n p Γ(d/p n, (2.2 2

3 which reduces to the known result E(x n = n!µ n in the prticulr cse of n exponentil distribution, f (x, µ,1,1. As shown by Nit & Gry [8], if one considers set of M independent rndom vribles tht re re individully distributed ccording with p = 1 GGD function, f (x,, d, 1, (which is nothing else thn stndrd gmm distribution, the probbility distribution of the men x = Σ M i=1 x i is the GGD function f ( x,/m,md,1, (2.3 nd the probbility distribution of the squred men is the GGD function [ ( 2, p( x 2 = f x 2 Md, M 2, 1 ], (2.4 2 which, mking use of eqution (2.2, provides the expecttions of n rbitrry power of the squred men, E( x 2n = Γ(Md + 2n ( 2n. (2.5 Γ(Md M Although closed form for the PDF of the men of squres x 2 = Σ M i=1 x2 i hs not been found, Nit & Gry [8] proved tht the sttisticl moments of the men of squres cn be exctly computed ccording with the formul E( x 2 n = (/ M 2n [Γ(d] M [ n 1 t n r=0 r! Γ(2r + dtr] M t=0. (2.6 n For the prticulr cses of the exponentil nd χ 2 distributions, f (x, µ,1,1 nd f (x, µ,1/2,1 respectively, Nit & Gry [8] proved tht x 2 / x 2 nd x 2 re uncorrelted rndom vribles, i.e. tht they hve null covrince, which immeditely led to the exct moments of the rtio between the men of squres nd squre of men, which cn be expressed s [( x 2 n ] E x 2 = E( x2 n E( x 2n. (2.7 However, s we will show here, the null covrince of x 2 / x 2 nd x 2 is n intrinsic property of ny p = 1 GGD function f (x,,d,1, which mkes eqution (2.7 hold for ny vlue of d. To prove this key property, we mke use of the expression ( x 2 cov x 2, x2 = 2 [ E(x 3 M E(x 2 ] 2E(x2 E(x 2 + E(x2 tht is vlid for ny prticulr PDF [Eq.19, 8], in which we enter the explicit moments provided by eqution (2.2 to obtin (2.8 ( x 2 cov x 2, x2 = 0. (2.9 Therefore, provided tht the observble x is distributed ccording to GGD function f (x,,d,1, nd tking in considertion equtions (2.5 nd (2.6, the sttisticl moments of the rtio between 3

4 the men of squres nd squre of men given by eqution (2.7 my be written in the compct form [( x 2 n ] M n Γ(Md n [ n 1 E x 2 = Γ(d M Γ(Md + 2n t n r=0 r! Γ(2r + dtr] M t=0, (2.10 which is independent of the scling prmeter. Nit & Gry [8] used the prticulr form of eqution (2.10 corresponding to d = 1 to derive the SK estimtor given by eqution (1.1, nd the prticulr form corresponding to d = 1/2 to derive time domin kurtosis (TDK estimtor T DK = M + 2 ( MS2 1, (2.11 M 1 both of them being unbised estimtors of the spectrl vribility corresponding to the underlying probbility distribution f (x,,d,1, which ccording to eqution (1.2 is S 2 1 V 2 = E(x2 E(x 2 E(x 2 = 1 d, (2.12 n expression tht is 1 for the d = 1 (SK cse, nd 2 for the d = 1/2 (TDK cse. 3. The Person Type IV Approximtion of the Spectrl nd Time Domin Kurtosis Estimtors Using eqution (2.10, the first stndrd moments of the ŜK estimtor given by eqution (1.1 my be written s µ 1 = 1; µ 2 = β 1 = 4M 2 (M 1(M + 2(M + 3 4(M + 2(M + 3(5M 72 (M 1(M (M ; β 2 = 3(M + 2(M + 3(M3 + 98M 2 185M + 78, (M 1(M + 4(M + 5(M + 6(M + 7 where β 1 = µ 3 2/µ3 2 nd β 2 = µ 4 /µ 2 2 re directly relted to the more commonly used skewness, γ 1 = β 1 nd kurtosis excess, γ 2 = β 2 3. Similrly, the first stndrd moments of the T DK estimtor re µ 1 = 2; µ 2 = 24M 2 (M 1(M + 4(M + 6 β 1 = 216(M 22 (M + 4(M + 6 (M 1(M (M ; β 2 = 3(M + 4(M + 6(M M 2 474M (M 1(M + 8(M + 10(M + 12(M The pproprite PDF pproximtion for the ŜK nd T DK estimtors my be found by investigting the Person s criterion defined s [5, p. 151]: κ = (3.1 (3.2 β 1 (β (4β 2 3β 1 (2β 2 3β 1 6, (3.3 where the exct vlues of the prmeters β 1 nd β 2 re provided by equtions (3.1 nd (3.2, respectively. Person s criterion indictes tht for M 24 the ŜK estimtor my be pproximted by Person Type IV probbility curve. The sme PDF pproximtion my be used for the T DK estimtor for M 46. 4

5 Figure 1: Comprison between the SK distributions (blck lines obtined by numericl simultion for different ccumultion lengths M nd their corresponding Person Type IV pproximtions (red lines. The four Person Type IV prmeters m, ν, λ, nd re displyed on ech plot. 3.1 Person Type IV Probbility Distribution Function The most generl nlyticl form of the Person Type IV PDF originlly introduced by Person [9], including its non-trivil normliztion fctor, ws given by Nghr [6] s p(x = 1 Γ(m + i ν 2 Γ(m i ν 2 [ ( x λ π Γ(m 1 2 Γ(m ] mexp [ νarctn( x λ ], (3.4 where the four prmeters m, µ,, nd λ my be expressed in terms of the centrl moments of the distribution s [4]: r = 6(β 2 β 1 1 2β 2 3β 1 6 ; m = r ; ν = r(r 2 β1 (3.5 16(r 1 β1 (r 2 2 = 1 µ 2 (6(r 1 β 1 (r ; λ = µ 1 4 (r 2 µ 2 β 1. Figure 1 compres the Person IV pproximtions of the ŜK PDF with the Monte Crlo simulted distributions for M = 32, 1024, 4096, nd By visul inspection, we my conclude tht the Person IV pproximtions ccurtely reproduce the shpes of the numericlly simulted histogrms for different orders of mgnitude of the ccumultion length. To compute the til probbilities of the Person Type IV PDF, one hs to compute the cumultive function P(x, (CF, nd the complementry cumultive function, 1 P(x, (CCF, P(x = x p(xdx; 1 P(x = for which Heinrich [4] provided the following closed form: 1 + P 1 (m,ν,,λ,x, x < λ 3 P(x = P 2 (m,ν,,λ,x x λ < 3 1 P 1 (m, ν,, λ, x, x > λ + 3, x p(xdx, (3.6 (3.7 where ( P 1 (m,ν,,λ,x = i x λ ( F 1,m + i ν 2m 1 2,2m, 2 1 i x λ p(x 5

6 Figure 2: ŜK threshold computtion for M = The lower nd upper thresholds displyed by the two verticl lines hve been estimted s the intersection points of the horizontl nd numericl integrtion lines. Their vlues of = / 6104 nd = / 6104, respectively, hve to be compred with the symmetric thresholds of 1 ± 6/ 6104 (verticl dotted lines. Figure 3: T DK threshold computtion for M = The two solid verticl lines, hving the ordintes nd , represent the RFI detection thresholds corresponding to symmetric stndrd flse lrm probbility level of %. These thresholds hve to be compred with with the less ccurte symmetric thresholds of 2 ± 3 24/12208 = 2 ± (verticl dotted lines. nd P 2 (m,ν,,λ,x = 1 1 e (ν+i2mπ i [ 1 + iν 2m + 2 F (1,2 2m,2 m + i ν 2, 1 + i x λ 2 F(α,β,δ,z = 1 + αβ α(α + 1β(β + 1 z + 1!δ 2!δ(δ + 1 z = ( x λ 2 ] p(x, k=0 α (k β (k k!δ (k z k (3.8 is the Guss hypergeometric series. Alterntively, insted of eqution (3.7, one my use the formul provided by Willink [11] where P(m,ν,,λ,x = e [λ i(2 2m]Φ R 1 e [λ i(2 2m]π 1, (3.9 Φ = π ( x λ 2 + rctn ; u = m i 2 ν; R = F(2 2m,u,u+1,eiΦ F(2 2m,u,u+1,1. However, if one wnts to void the numericl difficulties relted to the evlution of the hypergeometric series, one my choose to perform direct numericl integrtion (eqution [3.6] of eqution (3.4, which my chieve resonble ccurcy with fr less computtionl effort, especilly if tilored integrtion methods, [e.g. 6], re employed. Figure 2 displys the numericl results for M = 6104, computed ccording to eqution (3.7, (tringulr symbols, eqution (3.9, (squre symbols, nd by direct integrtion of equtions (3.6, 6

7 (solid line. The hypergeometric series where computed using the hypergeom function in Mple 11 (MpleSoft, nd the numericl integrtion ws performed using the int_tbulted function in IDL 6.4 (ITT. The plots disply both CF (rising nd CCF (descending needed to evlute the RFI thresholds equivlent to norml distribution s ±3σ level (probbility %, horizontl solid line. It my be concluded tht, in the region of interest, ll three methods provide similr numericl results. However, it ws found tht, for SK vlues well before the distribution pek, the numericl ccurcy of eqution (3.9 is better thn tht of eqution (3.7, while the direct numericl integrtion of CF gives similr results s eqution (3.9. After the pek of the ŜK distribution, the numericl ccurcy of eqution (3.7 is better thn tht of eqution (3.9, while the numericl integrtion of CCF gives similr results s eqution (3.7. Therefore we conclude tht the numericl evlution of eqution (3.9 gives more ccurte estimtion of the CF nd the numericl evlution of eqution (3.7 gives more ccurte estimtion of the CCF, while the direct numericl integrtion of equtions (3.6 gives results of comprble ccurcy t both sides of the ŜK distribution. The lower nd higher thresholds displyed by the two verticl lines hve been estimted s the intersection points of the horizontl nd numericl integrtion lines. Their vlues of = / 6104 nd = / 6104, respectively, re compred with the symmetric thresholds of 1 ± 6/ 6104 (verticl dotted lines originlly proposed by Nit & Gry [8]. Although this correction seems smll in bsolute vlue for the lrge-m cse, e.g. M = 6104 illustrted in Figure 2, we clculte tht, compred with the symmetric thresholds, the new thresholds ccount for 67% less rejection of vlid dt s flse RFI occurrences t the upper bound of the distribution, nd provide better rejection of true RFI signls of low signl to noise rtio t the lower bound. In combintion, the result is n overll better performnce of the ŜK bsed RFI rejection lgorithm. The correction becomes more importnt for lower M. Figure 3 displys the PDF of the T DK estimtor corresponding to n ccumultion length of M = 12208, chosen to mtch the sme frequency nd time resolution of DFT-bsed spectrogrph with M = 6104 (the exmple used in Fig. 2; see Nit et l. [7] for more detiled motivtion of this choice. Despite its lrge ccumultion length, the estimtor K still hs noticeble skewness, which needs to be properly considered in order to obtin the flse lrm probbility levels equivlent to ±3σ for norml distribution. Compred with the symmetric thresholds of 2±3 24/12208, the new thresholds would reject 71% less vlid dt t the higher end of the distribution, while the shifted lower threshold would improve the sensitivity of RFI detection t the lower end of the distribution. 4. Conclusion We hve investigted the sttisticl properties of the SK estimtor nd determined nlyticl expressions for its PDF nd CF with the gol of improving the selection of thresholds for RFI discrimintion. We lso improved the definition of ŜK (eqution [1.1] reltive to its originl definition [7] to form n unbised estimtor, nd introduced TDK unbised estimtor (eqution [2.11] to be used for RFI detection t the DC nd Nyquist frequency bins of DFT-bsed spectrogrph, or t ny frequency bin of FIR bsed spectrogrph. We hve derived closed form nlyticl expressions for the complete set of the centrl moments of the SK nd TDK estimtors, nd estblished common frmework tht llows ccurte estimtion of the RFI thresholds bsed on the first four stndrd moments of their probbility distributions (equtions [3.1] nd [3.2], which, for ny 7

8 ccumultion length M 24 nd M 46, respectively, re used to compute the four prmeters (eqution [3.5] tht completely determine the Person IV pproximtions (eqution [3.4] of their true PDFs. Bsed on these four prmeters, which depend only on the ccumultion length M, the CF nd CCF of the SK or TDK estimtors my be computed by using either the closed forms expressions provided by equtions (3.9 nd (3.7, respectively, or by direct numericl estimtion of the integrls given by eqution (3.6. Compred to the symmetricl thresholds originlly suggested in Pper I, the procedure described in this study properly tkes into ccount the intrinsic skewness of the probbility density functions of the SK nd TDK estimtors, which provides better overll RFI detection performnce for either smll or lrge ccumultion lengths. These theoreticlly estblished results re shown in Gry, Liu & Nit [2, 3] to be exctly obeyed by dt tken in the KSRBL spectrometer hrdwre implementtion of the lgorithm, where the improvement in RFI excision by use of these modified thresholds is confirmed. The modified thresholds become ever more importnt when smller number M of ccumultions is used. A simple procedure hs been written in IDL (Interctive Dt Lnguge for numericl clcultion of the thresholds for ny M, nd is vilble upon request from the uthors. References [1] Y. Dou, D. E. Gry, Z. Liu, G. M. Nit, S. -C. Bong, K. -S. Cho, Y. -D. Prk, & Y. -J. Moon, The Koren Solr Rdio Burst Loctor (KSRBL, PASP, 121, 512 [2] D. E. Gry, Z. Liu, & G. M. Nit, A Widebnd Spectrometer with RFI Detection, PASP, 122, 560 [3] D. E. Gry, Z. Liu, & G. M. Nit, Hrdwre Implementtion of n SK Spectrometer, in this proceedings, PoS(RFI [4] J. Heinrich, A Guide to the Person Type IV Distribution, Collider Detector t Fermilb internl note 6820, [5] M. G. Kendll & A. Sturt, The Advnced Theory of Sttistics Vol.1, Griffin, London 1958 [6] Y. Nghr, The PDF nd CF of Person type IV distributions nd the ML estimtion of the prmeters, Sttistics & Probbility Letters, 43, 251 [7] G. M. Nit, D. E. Gry, Z. Liu, G. J. Hurford, & S. M. White, Rdio Frequency Interference Excision Using Spectrl Domin Sttistics, PASP, 119, 805 [8] G. M. Nit & D. E. Gry, Sttistics of the Spectrl Kurtosis Estimtor, PASP, 122, 595 [9] K. Person, Contributions to the Mthemticl Theory of Evolution.-II. Skew Vrition in Homogeneous Mteril, Philos. Trns. R. Soc. London A, 186, 343 [10] E.W. Stcy, A Generliztion of the Gmm Distribution, Ann. Mth. Sttist. Assoc., 33, 1187 [11] R. Willink, A Closed-Form Expression For The Person Type IV Distribution Function, Austrl. NZ J. Stt., 50(2, 199 8