On Thresholds for Robust Goodness-of-Fit Tests

Size: px

Start display at page:

Download "On Thresholds for Robust Goodness-of-Fit Tests"

Cody Barnaby Wilkinson
6 years ago
Views:

1 O Thresholds for Robust Goodess-of-Fit Tests Jayakrisha Uikrisha, Sea Mey, ad Veugopal V. Veeravalli Dept. of Electrical ad Computer Egieerig Coordiated Sciece Laboratory Uiversity of Illiois at Urbaa-Champaig Urbaa, IL {juikr2, mey, Abstract Goodess-of-fit tests are statistical procedures used to test the hypothesis H 0 that a set of observatios were draw accordig to some give probability distributio. Decisio thresholds used i goodess-of-fit tests are typically set for guarateeig a target false-alarm probability. I may popular testig procedures results o the weak covergece of the test statistics are used for settig approimate thresholds whe eact computatio is ifeasible. I this work, we study robust procedures for goodess-of-fit where accurate models are ot available for the distributio of the observatios uder hypothesis H 0. We develop procedures for settig thresholds i two specific eamples - a robust versio of the Kolmogorov- Smirov test for cotiuous alphabets ad a robust versio of the Hoeffdig test for fiite alphabets. I. INTRODUCTION Suppose we are give a sequece of observatios Z = {Z i : i = 1,...,} ad we are iterested i determiig whether or ot the observatios were geerated i.i.d. with law π. We refer to this hypothesis as the ull hypothesis ad deote it by H 0 : Z i.i.d.(π). (1) I iformatio theory, this problem is typically referred to as a uiversal hypothesis testig problem [1], [2] while i statistics this problem is called a goodess-offit problem [3]. A uiversal hypothesis testig procedure typically outputs a biary decisio obtaied by computig some fuctio of the observatios ad comparig it to a threshold. The threshold is chose to meet some costrait o the probability of false alarm - i.e. the probability of rejectig hypothesis H 0 whe the hypothesis H 0 is true. A goodess-of-fit procedure, o the other had, computes some fuctio of the observatios ad outputs a p-value [4] which represets the cofidece with which oe ca accept the hypothesis H 0. Both these procedures are i fact similar, with the false alarm costrait used i the uiversal hypothesis test playig the role of the p-value obtaied i the goodess-of-fit approach. I several popular goodess-of-fit procedures icludig the Kolmogorov-Smirov test ad the chi-square test [4], weak covergece results of the test statistic are used for settig thresholds for a target false alarm level. For istace, i the chi-square test it is kow that the test statistic coverges to a chi-square distributio. Hece, for large, this result gives a approimate procedure for settig the threshold for a target false alarm probability. Such procedures are particularly useful, ofte ecessary, whe it is difficult to compute the eact error probabilities for large values of the legth of the observatio sequece Z. A eact computatio for large would typically require comple simulatios. I this paper we study a robust versio of the goodess-of-fit problem i which we do ot kow π eactly but do kow that π lies i some ucertaity class of probability distributios P. We are iterested i testig whether or ot some distributio from P adequately describes the distributio that geerated the observatios. Hece the ew problem of iterest ca be described by the followig composite ull hypothesis: 0 : Z i.i.d.(π) for some π P. (2) We propose robust versios of two stadard goodessof-fit tests, viz., the Kolmogorov-Smirov (KS) test ad the Hoeffdig test. We also obtai weak covergece results for the robust test statistic which ca be used to set thresholds that meet the costrait that the worst-case false alarm probability amog all distributios i P is less tha a give costrait. Such approimate methods for settig thresholds are more importat for robust problems because it is geerally ot possible to obtai thresholds by simulatios, sice there are ifiitely may possibilities for the ukow distributios i P. We first study the robust KS test i Sectio II ad the the robust Hoeffdig test i Sectio III.

2 II. ROBUST KOLMOGOROV-SMIRNOV TEST The stadard versio of the KS test is used to determie whether or ot a give sequece of observatios is draw uder a probability distributio with a cotiuous distributio fuctio F 0 defied over some closed iterval Z R. Here F 0 takes the role of π i (1) ad H 0 represets the hypothesis that the observatios Z were draw i.i.d. from F 0. The test statistic used is D = sup F () F 0 () (3) where F represets the empirical distributio fuctio defied by F () = 1 I{Z i } i=1 where I is the idicator fuctio. I the KS test based o Z, the test statistic D is compared to a threshold τ chose so that the probability of error uder hypothesis H 0 is less tha some desired level. It is geerally difficult to compute the eact distributio of the statistic D for large sequece legths. The typical practice is to use the followig result by Kolmogorov for approimatig the distributio of D for large. Whe the observatios Z are draw i.i.d. from F 0, the followig weak covergece result holds: P F 0{ D > δ} d. P{ sup K(t) > δ} (4) t [0,1] where K is the Browia bridge [5, p. 335]. The subscript of F 0 i (4) idicates that the observatios were draw from distributio F 0. Usig this result thresholds are usually set usig look-up tables cotaiig values of the distributio fuctio of sup t [0,1] K(t). While usig the KS test i practice, oe ofte faces the problem of over-fittig. If the true uderlyig distributio is ot eactly F 0 but some other distributio close to F 0 the the KS-test will evetually reject hypothesis H 0 for large eough. I this work we propose a ew robust versio of the KS test that could potetially address this issue. The idea is to elarge the set of acceptable hypotheses beyod the sigleto {F 0 }. We defie a ew class of distributios as follows: F = {G P(Z) : F () G() F + (), Z} where P(Z) is the space of all probability distributios o Z ad F ad F + are cotiuous probability distributio fuctios such that the omial distributio F 0 F. We are ow iterested i testig the followig composite hypothesis: 0 : Z i.i.d.(g), for some G F. The advatage of usig such a ucertaity class 1, as it will become clear later, is that the resultig robust test is a simple modificatio of the stadard KS test, ad also admits a straightforward cofidece approimatio i the asymptotic settig. The distributios F + ad F ca be chose to cotrol the size of the class F that are acceptable as beig close to F 0. The proposed robust test uses the followig robust test statistic: E = mi sup F F F () F() (5) where F is the empirical distributio fuctio. The robust test for goodess-of-fit compares E to a threshold τ ad accepts hypothesis H0 ROB iff E τ. Thus the biary decisio is give by φ ROB τ, (Z) = I{E > τ} (6) where I is the idicator fuctio ad a decisio φ ROB τ, = 0 represets a decisio acceptig H0 ROB ad φ ROB τ, = 1 represets a decisio rejectig H0 ROB. Let F deote the miimizer i (5) ad let deote the poit at which the supremum i (5) is achieved for F = F. It is clear that if E > 0, the F is either F + or F. By defiitio it is obvious that for ay δ > 0, the evet { E > δ} ca be writte as the uio of two evets as follows: { E > δ} = {sup (F () F + ()) > δ} {sup [ (F () F ())] > δ}. (7) The threshold used i (6) ca be set usig the followig theorem that studies the asymptotic behavior of p (δ) := ma P G{ E > δ}. Theorem II.1. For large eough, p (δ) satisfies the followig iequalities: p(δ) p (δ) 2p(δ) 1 Whether this robust class works for a particular applicatio i mid eeds to be see through eperimets. For eample, if the deviatio is because the observatios are ot completely i.i.d. the the proposed test will ot be robust to such deviatios from omial behavior.

3 with p(δ) defied by p(δ) := P{sup t [0,1] K(t) > δ} where K is the Browia bridge. Proof: We kow from (7) that for ay G F, P G {sup (F () F + ()) > δ} P G { E > δ} P G {sup (F () F + ()) > δ} +P G {sup [ (F () F ())] > δ} Hece, ma P G{sup (F () F + ()) > δ} ma P G{ E > δ} ma P G{sup (F () F + ()) > δ} + ma P G{sup [ (F () F ())] > δ} which simplifies to: P F +{sup (F () F + ()) > δ} ma P G{ E > δ} P F +{sup (F () F + ()) > δ} +P F {sup [ (F () F ())] > δ} It is kow by aother result of Kolmogorov [5, p. 335] that P F +{ sup(f () F + ()) > δ} p(δ). Similarly it also follows that P F { sup(f () F ()) > δ} p(δ). The coclusio of the theorem follows by combiig both results. Theorem II.1 suggests that if δ is chose such that 2p(δ) = α, ad the threshold τ i the robust test (6) is chose such that τ = δ, the for a large eough, the test satisfies a uiform boud o the probability of rejectig H0 ROB over all distributios i F. i.e., for large eough ma P G{φ ROB τ, (Z) = 1} α. (8) Furthermore, for large eough we also have ma P G{φ ROB τ,(z) = 1} 1 2α. (9) which meas that the maimum false alarm probability is withi a factor of 0.5 from the desired level. Thus for a large umber of give observatios, the above result ca be used to set the threshold τ for a target false alarm costrait α by lookig up values of the p-fuctio from a table. A importat questio that we are curretly studyig is how large should be to guaratee that the probability o the left had side of (8) is withi ǫ from α. III. ROBUST HOEFFDING TEST The Hoeffdig test [2] is a stadard goodess-of-fit test for fiite alphabets. The test statistic is give by the divergece D(Γ π) where π is the probability distributio of iterest defied over a fiite alphabet Z, ad Γ is the empirical distributio (type) of the first observatios defied by Γ (z) = 1 I{Z i = z}, z Z. i=1 The followig weak covergece result for the test statistic uder π is well kow [6],[1]: 2D(Γ d. π) χ2 N 1 (10) where N is the size of the support of distributio π ad χ 2 γ deotes a chi-square radom variable with γ degrees of freedom. This result eables us to set approimate thresholds for large usig tables of the chi-square distributio. A robust versio of the Hoeffdig test was defied i [7] that uses the followig robust versio of the divergece as the test statistic: if π P D(Γ π) where P is a ucertaity class of acceptable distributios. This is a uiversal hypothesis test where we are ow testig the composite ull hypothesis give by 0 : Z i.i.d.(π) for some π P. We defie a robust versio of the divergece as follows: D ROB (µ P) := if π P D(µ π). Thus the proposed test is represeted by the biary decisio φ ROB τ,(z) = I{D ROB (Γ P) > τ} (11) where τ is a threshold that must be chose to meet the costrait o the false alarm probability.

4 I the rest of this sectio we address the followig questio: Ca we obtai ay covergece results for the worst case probability of error of the robust Hoeffdig test like those implied by the weak covergece result of (10)? We evaluate the limitig value of P π {D ROB (Γ P) > η} (12) for π P. It is easy to see that whe the ifimum i (12) is achieved, the miimizig distributio is π ML, the maimum likelihood (ML) estimate of the uderlyig distributio from P [1]. Hece, we are really iterested i fidig the asymptotic behavior of P π {D(Γ π ML ) > η}. I Theorem III.1 we aswer this questio for a special choice of P defied below. We study the sceario where P is a ucertaity class defied through the followig momet costraits as cosidered i [7]: P := {π P(Z) : π(ψ i ) = 0, 1 i d}. Here P(Z) deotes the set of probability distributios over Z ad for ay fuctio f : Z R ad for ay distributio π P(Z), we use the followig abusive otatio for the epected value of fuctio f: π(f) := z Zπ(z)f(z). Note that for z Z, π(z) deotes the probability mass at z uder measure π. The meaig will be clear from cotet. Let ψ deote the vector of fuctios (ψ 1,ψ 2,...,ψ d ) T ad Z π deote the support of distributio π. We make the followig assumptios about the fuctios {ψ i : 1 i d}: (A1) There is some distributio π 0 P such that the fuctios {ψ i : 0 i d} are liearly idepedet over Z π 0, where ψ 0 1. (A2) The origi 0 R d is a iterior poit of the set of feasible momet vectors, defied as := { R d : i = µ(ψ i ),i = 1,...,d, for some µ P(Z)}. We kow from the results of [7] that the robust divergece with respect to P ca be epressed as the solutio to the followig optimizatio problem: D ROB (µ P) = sup µ(log(1 + r T ψ)) (13) r R where the supremum is take over R := {r R d : 1 + r T ψ(z) 0 for all z Z} ad r T ψ deotes the fuctio d i=1 r i ψ i. It was also show i [1, Sectio II.E] that the robust divergece ca be epressed as a mismatched divergece defied with respect to a log-liear fuctio class. I the rest of this sectio, we argue that the results o weak covergece of the mismatched divergece statistic from [1, Theorem III.3(i)] ca be used to set thresholds for the robust hypothesis testig procedure. The mai result of this sectio is the followig theorem. For ay π P, let d π deote the umber that is oe lower tha the maimal umber of fuctios i {ψ i : 0 i d} that are liearly idepedet over Z π. I other words, d π + 1 is the dimesio of the spa of the fuctios {ψ i : 0 i d} whe restricted to Z π. Theorem III.1. Suppose assumptios (A1) ad (A2) hold. The, the followig weak covergece result holds uder π: 2D ROB (Γ d. P) χ2 d π. (14) Hece we have sup lim P π{2d ROB (Γ P) > η} = 1 F(d,η) (15) π P where F(d, η) is the cumulative distributio fuctio of a chi-square distributio with d degrees of freedom evaluated at η. The secod coclusio i the theorem above follows directly from the first, usig the fact that chi-square distributios are stochastically ordered accordig to the umber of degrees of freedom. I order to prove the mai result, we eed the followig lemma: Lemma III.2. Suppose the fuctios {ψ i : 0 i d} are liearly idepedet over the support of π. If Z π deotes the support of π P, the there eists a ope eighborhood B P(Z π ) such that for all µ B, the supremum i (13) is achieved at a uique poit r(µ). Proof: We verify that the propositio i the lemma holds for B = {µ P(Z) : µ(z) π(z) < ǫ for all z Z π, µ(y) = 0 for all y Z \ Z π } where ǫ = 1 2 mi z Z π π(z). Clearly, for all µ B, the support of µ is equal to Z π ad hece 0 D ROB (µ P) D(µ π) <. Now sice the fuctios ψ i are liearly idepedet over Z π, ad the value of the the optimizatio problem i (13) is fiite, it follows that we ca restrict the costrait set i (13) to a bouded subset of the closed set R. Thus

5 we ca restrict the optimizatio i (13) to a compact set. Furthermore, the objective fuctio is strictly cocave by the liear idepedece assumptio ad hece the coclusio of the lemma follows. Proof of Theorem III.1: Suppose d π = d. We kow by the discussio i [1, Sectio II.E] that the robust divergece ca be epressed as a mismatched divergece defied with respect to a log-liear fuctio class. I.e., D ROB (µ P) = D MM (µ π) where D MM is a relaatio of the divergece with respect to the log-liear fuctio class defied by F = {log(1 + r T ψ) : r R} D MM (µ π) = sup{µ(f) log(π(e f ))}. f F From the coclusios of Lemma III.2 it follows that the coditios of [1, Theorem III.3(i)] are satisfied ad hece the coclusio of (14) follows whe d π = d. Now, if d π < d it is clear that we ca defie a ew set of d π fuctios ψ = {ψ 1,...,ψ d π } such that the spa of the fuctios i ψ over Z π is idetical to that of the fuctios i ψ over Z π, ad further guarateeig the coditio that {ψ 0,ψ } are liearly idepedet over Z π. Hece for all µ P(Z π ) the formula for the rate fuctio i (13) is uaffected by this trasformatio. Thus the result of (14) holds for a geeral π P, by the same argumets as before. The result of (15) ca be used to set thresholds for the robust Hoeffdig test. For a give, suppose we choose τ i (11) such that τ = δ 2 where δ satisfies IV. CONCLUSIONS AND EXTENSIONS We have show that weak covergece results similar to those used for settig thresholds i goodessof-fit procedures ca also be obtaied i some robust goodess-of-fit problems where eact kowledge of the distributio is ot available uder the ull hypothesis. We are curretly studyig etesios of similar ideas to other goodess-of-fit tests like the chi-square test ad for more geeral ucertaity classes. Aother etesio of iterest is to study the rate of covergece of these weak covergece results which would eable us to evaluate the accuracy of the thresholds set usig the proposed procedures for a give legth of the observatio sequece. ACKNOWLEDGEMENTS We thak Ke Duffy for iitial discussios that motivated the problem studied i this paper. REFERENCES [1] J. Uikrisha, D. Huag, S. Mey, A. Suraa, ad V. V. Veeravalli, Uiversal ad composite hypothesis testig via mismatched divergece, IEEE Trasactios o Iformatio Theory, revised Jue [Olie]. Available: [2] W. Hoeffdig, Asymptotically optimal tests for multiomial distributios, A. Math. Statist., vol. 36, pp , [3] A. W. va der Vaart, Asymptotic Statistics. Cambridge Uiversity Press, [4] E. L. Lehma, Testig Statistical Hypotheses, 2d ed. New York: Joh Wiley & Sos, [5] R. M. Dudley, Uiform Cetral Limit Theorems. Cambridge Uiversity Press, [6] S. S. Wilks, The large-sample distributio of the likelihood ratio for testig composite hypotheses, A. Math. Statistics, vol. 9, pp , [7] C. Padit ad S. P. Mey, Worst-case large-deviatios with applicatio to queueig ad iformatio theory, Stoch. Proc. ad Appls, vol. 116, o. 5, pp , May F(d,δ) = α. By (15) this choice of threshold esures that for large eough, ad for all π P, P π {φ ROB τ, (Z) = 1} α. (16) We ote that this result is weaker tha the covergece result i (8) sice we do ot have a guaratee o the covergece of sup π P P π {2D ROB (Γ P) > η}. I order to guaratee such a result, we would eed to prove (15) with the supremum ad limit sigs iterchaged. We are curretly tryig to establish such a result.

32 estimating the cumulative distribution function

32 estimating the cumulative distribution function 32 estimatig the cumulative distributio fuctio 4.6 types of cofidece itervals/bads Let F be a class of distributio fuctios F ad let θ be some quatity of iterest, such as the mea of F or the whole fuctio