Local Minimax Testing

Transcription

1 Local Minimax Testing Sivaraman Balakrishnan and Larry Wasserman Carnegie Mellon June 11, / 32

2 Hypothesis testing beyond classical regimes Example 1: Testing distribution of bacteria in gut microbiome (d n) Example 2: Testing distribution of number of α-particles emitted by a radioactive source over a time window (d = ) June 11, / 32

3 Hypothesis testing beyond classical regimes Example 1: Testing distribution of bacteria in gut microbiome (d n) Example 2: Testing distribution of number of α-particles emitted by a radioactive source over a time window (d = ) Goal: Understand fundamental limits and avoid strong assumptions: June 11, / 32

4 Hypothesis testing beyond classical regimes Example 1: Testing distribution of bacteria in gut microbiome (d n) Example 2: Testing distribution of number of α-particles emitted by a radioactive source over a time window (d = ) Goal: Understand fundamental limits and avoid strong assumptions: Uniform null Fixed-cells asymptotics June 11, / 32

5 Hypothesis testing beyond classical regimes Example: Fit a density, test goodness-of-fit June 11, / 32

6 Hypothesis testing beyond classical regimes Example: Fit a density, test goodness-of-fit Goal: Understand fundamental limits and avoid strong assumptions: Uniform null Bounded domain Unnecessary, strong smoothness assumptions June 11, / 32

7 The basic setup Goodness-of-fit testing Observe samples Z 1,..., Z n P, for some fixed P 0 want to test: H 0 : P = P 0 H 1 : TV(P, P 0 ) ɛ TV(P, Q) = sup P (A) Q(A) A June 11, / 32

8 The basic setup Goodness-of-fit testing Observe samples Z 1,..., Z n P, for some fixed P 0 want to test: H 0 : P = P 0 H 1 : TV(P, P 0 ) ɛ TV(P, Q) = sup P (A) Q(A) A TV natural metric on distributions, invariant to scale June 11, / 32

9 The basic setup Goodness-of-fit testing Observe samples Z 1,..., Z n P, for some fixed P 0 want to test: H 0 : P = P 0 H 1 : TV(P, P 0 ) ɛ TV(P, Q) = sup P (A) Q(A) A TV natural metric on distributions, invariant to scale From Le Cam (refined by Barron): no consistent tests without further structural assumptions June 11, / 32

10 Structural Assumptions Multinomials Distributions under null and alternate are multinomials on d categories. M = { p : p R d, d i=1 } p i = 1, p i 0 i {1,..., d}. June 11, / 32

11 Structural Assumptions Multinomials Distributions under null and alternate are multinomials on d categories. M = { p : p R d, d i=1 } p i = 1, p i 0 i {1,..., d}. Minimally smooth densities Densities under null and alternate are L-Lipschitz. { L = p : p(x)dx = 1, p(x) 0 x, p(x) p(y) L x y 2 x, y R d}. X June 11, / 32

12 Structural Assumptions Multinomials Distributions under null and alternate are multinomials on d categories. M = { p : p R d, Questionable assumptions: d i=1 Uniform null, d fixed, or n d Minimally smooth densities } p i = 1, p i 0 i {1,..., d}. Densities under null and alternate are L-Lipschitz. { L = p : p(x)dx = 1, p(x) 0 x, p(x) p(y) L x y 2 x, y R d}. X Questionable assumptions: Uniform null, bounded domain, L-fixed June 11, / 32

13 Risk A test φ : X n {0, 1} is level α if, P n 0 (φ = 1) α for all P 0 C. June 11, / 32

14 Risk A test φ : X n {0, 1} is level α if, Φ n = all level α tests. P n 0 (φ = 1) α for all P 0 C. June 11, / 32

15 Risk A test φ : X n {0, 1} is level α if, P n 0 (φ = 1) α for all P 0 C. Φ n = all level α tests. The risk of a level-α test is its maximum type-ii error: R n (φ, P 0, ɛ, C) = sup P n (φ = 0). P :TV(P,P 0) ɛ,p C June 11, / 32

16 Risk A test φ : X n {0, 1} is level α if, P n 0 (φ = 1) α for all P 0 C. Φ n = all level α tests. The risk of a level-α test is its maximum type-ii error: R n (φ, P 0, ɛ, C) = sup P n (φ = 0). P :TV(P,P 0) ɛ,p C Local minimix rate { ɛ n (P 0 ) = inf ɛ : inf R n (φ, P 0, ɛ, C) 1/2 φ Φ n } June 11, / 32

17 Risk A test φ : X n {0, 1} is level α if, P n 0 (φ = 1) α for all P 0 C. Φ n = all level α tests. The risk of a level-α test is its maximum type-ii error: R n (φ, P 0, ɛ, C) = sup P n (φ = 0). P :TV(P,P 0) ɛ,p C Local minimix rate { ɛ n (P 0 ) = inf ɛ : inf R n (φ, P 0, ɛ, C) 1/2 φ Φ n } Global minimax rate ɛ n = inf { ɛ : sup P 0 inf R n (φ, P 0, ɛ, C) 1/2 φ Φ n } June 11, / 32

18 Minimax Sample Complexity If you prefer sample complexity (CS literature): Global Minimax Sample Complexity The global minimax sample complexity: n(ɛ, C) = sup P 0 C inf n(φ, ɛ, P 0, C). φ June 11, / 32

19 Minimax Sample Complexity If you prefer sample complexity (CS literature): Global Minimax Sample Complexity The global minimax sample complexity: n(ɛ, C) = sup P 0 C inf n(φ, ɛ, P 0, C). φ Local Minimax Sample Complexity The local minimax sample complexity: n(p 0, ɛ, C) = inf n(φ, ɛ, P 0, C). φ June 11, / 32

20 Minimax Sample Complexity If you prefer sample complexity (CS literature): Global Minimax Sample Complexity The global minimax sample complexity: n(ɛ, C) = sup P 0 C inf n(φ, ɛ, P 0, C). φ Local Minimax Sample Complexity The local minimax sample complexity: n(p 0, ɛ, C) = inf n(φ, ɛ, P 0, C). φ Sufficiently homogenous problems: two rates are identical. Testing: vast variability in minimax rate. Local minimax rate provides a refined picture. June 11, / 32

21 MULTINOMIALS June 11, / 32

22 Multinomials Classical work in statistics: Morris, Feinberg, Barron and Read and Cressie already emphasize importance of moving beyond fixed-cells Minimax rates for uniform null: Paninski, lots of follow-up in CS June 11, / 32

23 Multinomials Classical work in statistics: Morris, Feinberg, Barron and Read and Cressie already emphasize importance of moving beyond fixed-cells Minimax rates for uniform null: Paninski, lots of follow-up in CS Let p = (p(1),..., p(d)) with p(1) p(2) p(d). June 11, / 32

24 Multinomials Classical work in statistics: Morris, Feinberg, Barron and Read and Cressie already emphasize importance of moving beyond fixed-cells Minimax rates for uniform null: Paninski, lots of follow-up in CS Let p = (p(1),..., p(d)) with p(1) p(2) p(d). Global minimax testing rate (well-known): Faster than the estimation rate. ɛ n = d1/4 n June 11, / 32

25 Multinomials Classical work in statistics: Morris, Feinberg, Barron and Read and Cressie already emphasize importance of moving beyond fixed-cells Minimax rates for uniform null: Paninski, lots of follow-up in CS Let p = (p(1),..., p(d)) with p(1) p(2) p(d). Global minimax testing rate (well-known): ɛ n = d1/4 n Faster than the estimation rate. But what about the local minimax rate? June 11, / 32

26 Local Minimax Rate (Valiant and Valiant 2014) Tail Q σ = i : d p(j) σ. j=i June 11, / 32

27 Local Minimax Rate (Valiant and Valiant 2014) Tail Bulk Q σ = B σ = i : d p(j) σ. j=i {i : i > 1, i / Q σ }. June 11, / 32

28 Local Minimax Rate (Valiant and Valiant 2014) Tail Bulk Q σ = B σ = i : d p(j) σ. j=i {i : i > 1, i / Q σ }. V-Functional V σ (p 0 ) = j B σ p 2/3 0 (j) 3/2 June 11, / 32

29 Local Minimax Rate Valiant and Valiant (2014) showed that l n ɛ n (p 0 ) u n where l n and u n solve: Vln (p 0 ) Vun/16(p 0 ) l n =, u n =. n n June 11, / 32

30 Local Minimax Rate Valiant and Valiant (2014) showed that where l n and u n solve: l n ɛ n (p 0 ) u n Roughly: Vln (p 0 ) Vun/16(p 0 ) l n =, u n =. n n Vɛn (p 0 ) ɛ n = n June 11, / 32

31 Local Minimax Rate Valiant and Valiant (2014) showed that l n ɛ n (p 0 ) u n where l n and u n solve: Vln (p 0 ) Vun/16(p 0 ) l n =, u n =. n n Roughly: Vɛn (p 0 ) ɛ n = n We can have d =. June 11, / 32

32 Local Minimax Rate Valiant and Valiant (2014) showed that l n ɛ n (p 0 ) u n where l n and u n solve: Vln (p 0 ) Vun/16(p 0 ) l n =, u n =. n n Roughly: Vɛn (p 0 ) ɛ n = n We can have d =. (sparse) 1 V σ d (uniform) June 11, / 32

33 Multinomial Examples Uniform Null: If p 0 is uniform on d categories, ɛ n (p 0 ) d1/4 n June 11, / 32

34 Multinomial Examples Uniform Null: If p 0 is uniform on d categories, ɛ n (p 0 ) d1/4 n Also the worst-case (global minimax) sample complexity. In contrast to estimation allows for n d. June 11, / 32

35 Multinomial Examples Uniform Null: If p 0 is uniform on d categories, ɛ n (p 0 ) d1/4 n Also the worst-case (global minimax) sample complexity. In contrast to estimation allows for n d. Sparse Null: If p 0 mostly concentrates on s categories: ɛ n (p 0 ) s1/4 n June 11, / 32

36 Multinomial Examples Uniform Null: If p 0 is uniform on d categories, ɛ n (p 0 ) d1/4 n Also the worst-case (global minimax) sample complexity. In contrast to estimation allows for n d. Sparse Null: If p 0 mostly concentrates on s categories: Infinite Multinomials: with tail decay e.g: Power law multinomials Poisson distributions ɛ n (p 0 ) s1/4 n Truncated 2/3-norm finite for infinite multinomials June 11, / 32

37 The VV Test - Upper Bound The local-minimax optimal test is a two-stage test: A tail test: Tests the total mass in the ɛ-tail of the multinomial. June 11, / 32

38 The VV Test - Upper Bound The local-minimax optimal test is a two-stage test: A tail test: Tests the total mass in the ɛ-tail of the multinomial. A bulk modified-χ 2 test: Let X i denote the count of the i-th category. Use the test statistic: T = (X i np 0 (i))2 X i. p i B 0 (i) 2/3 ɛ/8 Two modifications to the usual χ 2 -test statistic Analysis proceeds by studying mean and variance of the test statistic under the null and alternate Involves several difficult inequalities to deal with the 2/3-norm Some deficiencies: need to specify ɛ poorly understood limiting distribution of test statistic June 11, / 32

39 Two simulations n = 200, d = Uniform Null, Sparse Alternate 1 Power Law Null, Sparse Alternate Power Power Chi-sq. LRT 2/3rd and tail Chi-sq. 0.1 LRT 2/3rd and tail l1 Distance l1 Distance June 11, / 32

40 Why do classical tests fail? The most classical goodness-of-fit test is the χ 2 test: T = d i=1 (X i np 0 (i)) 2 np 0 (i). p 0 (i) Small entries of p 0 can dominate the variance. Classical p 0 -fixed asymptotics mask this phenomenon. Related issues plague likelihood-ratio, l 1, l 2 and other test statistics. Classical test statistics are not even globally minimax optimal. June 11, / 32

41 Why do classical tests fail? The most classical goodness-of-fit test is the χ 2 test: T = d i=1 (X i np 0 (i)) 2 np 0 (i). p 0 (i) Small entries of p 0 can dominate the variance. Classical p 0 -fixed asymptotics mask this phenomenon. Related issues plague likelihood-ratio, l 1, l 2 and other test statistics. Classical test statistics are not even globally minimax optimal. Can we directly address deficiencies of the χ 2 -statistic? June 11, / 32

42 A simple, global minimax test Use instead a truncated test statistic: T trunc = d i=1 (X i np 0 (i)) 2 X i. max{p 0 (i), 1/d} If any entry is too small, clip the denominator to limit the contribution to the variance. June 11, / 32

43 A simple, global minimax test Use instead a truncated test statistic: T trunc = d i=1 (X i np 0 (i)) 2 X i. max{p 0 (i), 1/d} If any entry is too small, clip the denominator to limit the contribution to the variance. Theorem (BW17) The test based on T trunc is globally minimax. Test is simple, minimax-optimal, analysis is straightforward Single-stage, no knowledge of ɛ necessary June 11, / 32

44 Simulations re-visited n = 200, d = Uniform Null, Sparse Alternate 1 Power Law Null, Sparse Alternate Power Power Chi-sq. Trunc Var LRT 2/3rd and tail Chi-sq. Trunc Var 0.1 LRT 2/3rd and tail l1 Distance l1 Distance June 11, / 32

45 A new (near)-locally minimax test Inspired by a closely related test in a paper by Diakonikolas and Kane. Basic Insight: Careful modifications to χ 2 are crucial away from uniform. At uniform almost all test statistics are near optimal. June 11, / 32

46 A new (near)-locally minimax test Inspired by a closely related test in a paper by Diakonikolas and Kane. Basic Insight: Careful modifications to χ 2 are crucial away from uniform. At uniform almost all test statistics are near optimal. Slice the multinomial into almost uniform pieces, use Bonferroni. Partition the entries in B ɛ/8 into sets S j for j 1, where S j = { t : p 0 (2) 2 j < p 0 (t) p } 0(2) 2 j 1. T j = t S j [(X t np 0 (t)) 2 X t ] The max test (for Bonferroni adjusted thresholds t j ) is φ max = j I(T j > t j ). June 11, / 32

47 A new (near)-locally minimax test Theorem (BW17) The max test is locally minimax (up to logarithmic factors). Max test is (near)-local minimax optimal Less practical than the modified χ 2 test Analysis is completely transparent and does not require difficult inequalities June 11, / 32

48 A new (near)-locally minimax test Theorem (BW17) The max test is locally minimax (up to logarithmic factors). Max test is (near)-local minimax optimal Less practical than the modified χ 2 test Analysis is completely transparent and does not require difficult inequalities Summary: Testing high-dimensional multinomials, interesting local phenomena Modifications of the χ 2 -test are globally minimax and locally minimax June 11, / 32

49 DENSITIES June 11, / 32

50 Density Testing Recall: Lipschitz Densities Densities under null and alternate are L-Lipschitz. { L = p : p(x)dx = 1, p(x) 0 x, p(x) p(y) L x y 2 x, y R d}. X Focus initially on d = 1 case. June 11, / 32

51 Density Testing Recall: Lipschitz Densities Densities under null and alternate are L-Lipschitz. { L = p : p(x)dx = 1, p(x) 0 x, p(x) p(y) L x y 2 x, y R d}. X Focus initially on d = 1 case. Theorem (Ingster 1984,2000) Suppose L is fixed, and the domain X = [0, 1], p 0 uniform. Then the global minimax rate scales as: ( ) 2/5 1 ɛ n n June 11, / 32

52 Density Testing Recall: Lipschitz Densities Densities under null and alternate are L-Lipschitz. { L = p : p(x)dx = 1, p(x) 0 x, p(x) p(y) L x y 2 x, y R d}. X Focus initially on d = 1 case. Theorem (Ingster 1984,2000) Suppose L is fixed, and the domain X = [0, 1], p 0 uniform. Then the global minimax rate scales as: ( ) 2/5 1 ɛ n n d-dimensional extension in Arias-Castro et al. (2016) Only considers uniform null, suggests quantile transformation Strong assumptions of fixed L, bounded domain - analogous to the fixed-cells assumption in multinomials June 11, / 32

53 Local Minimax Rate for Testing Lipschitz Densities For a density p 0 : Its bulk B ɛ is the set of smallest Lebesgue measure that contains 1 ɛ probability content. Define the truncated 1/2-norm: ( ) 2 T (p 0 ) = p0 (x)dx. B ɛ June 11, / 32

54 Local Minimax Rate for Testing Lipschitz Densities For a density p 0 : Its bulk B ɛ is the set of smallest Lebesgue measure that contains 1 ɛ probability content. Define the truncated 1/2-norm: ( ) 2 T (p 0 ) = p0 (x)dx. B ɛ Theorem For the Lipschitz class L, the local minimax rate is: ( ) 1/5 Ln T (p 0 ) ɛ n (p 0 ). Tight characterization of the local minimax rate, up to constants. No unnecessary assumptions, L n is not treated as fixed, and the domain is not assumed to be bounded n 2 June 11, / 32

55 Examples Uniform Null: If the null p 0 is uniform on [0, B]: ( LB 2 ɛ n n 2 ) 1/5 June 11, / 32

56 Examples Uniform Null: If the null p 0 is uniform on [0, B]: ( LB 2 ɛ n n 2 ) 1/5 Spiky Null: The sparsest Lipschitz density is: ( ) p 0 (x) = max L(1 x ), 0. The minimax rate is completely independent of L and the domain, ɛ n 1 n 2/5 June 11, / 32

57 Examples contd. Can derive rates for other natural testing problems: Gaussian Null: If the null p 0 is Gaussian N(µ, σ 2 ) then the minimax rate for testing is: ( ) Lσ 2 1/5 ɛ n. n 2 June 11, / 32

58 Examples contd. Can derive rates for other natural testing problems: Gaussian Null: If the null p 0 is Gaussian N(µ, σ 2 ) then the minimax rate for testing is: ( ) Lσ 2 1/5 ɛ n. n 2 Cauchy Null: rate is: Let γ denote the shape parameter of p 0. The minimax ( L log 4 (1/ɛ) ɛ n n 2 )1/5. June 11, / 32

59 Examples contd. Can derive rates for other natural testing problems: Gaussian Null: If the null p 0 is Gaussian N(µ, σ 2 ) then the minimax rate for testing is: ( ) Lσ 2 1/5 ɛ n. n 2 Cauchy Null: rate is: Let γ denote the shape parameter of p 0. The minimax ( L log 4 (1/ɛ) ɛ n n 2 )1/5. Pareto Null: p 0 (x) x α 1, for 0 < α < 1. The minimax rate is: ( ) α L 3α+2 ɛ n. n 2 The dependence of the minimax rate as a function of ɛ is non-standard, and degrades rapidly as α 0. June 11, / 32

60 High-Level Proof Ideas Upper Bound: Classical method of goodness-of-fit testing: bin and test the corresponding multinomial using (locally minimax) multinomial test. Key technical challenge: significant flexibility in how to bin p0 June 11, / 32

61 High-Level Proof Ideas Upper Bound: Classical method of goodness-of-fit testing: bin and test the corresponding multinomial using (locally minimax) multinomial test. Key technical challenge: significant flexibility in how to bin p0 Idea 1: Use fixed bin-widths. Choose the largest bin-width that adequately controls the approximation error, i.e. keeps apart p 0 from the alternate densities. Used by Ingster, achieves global minimax rate when L is fixed, and domain is bounded. Inadequate to obtain tight local minimax rate: intuitively the number and size of the bins should be adapted to the density. June 11, / 32

62 High-Level Proof Ideas Upper Bound: Classical method of goodness-of-fit testing: bin and test the corresponding multinomial using (locally minimax) multinomial test. Key technical challenge: significant flexibility in how to bin p0 Idea 1: Use fixed bin-widths. Choose the largest bin-width that adequately controls the approximation error, i.e. keeps apart p 0 from the alternate densities. Used by Ingster, achieves global minimax rate when L is fixed, and domain is bounded. Inadequate to obtain tight local minimax rate: intuitively the number and size of the bins should be adapted to the density. Idea 2: Use adaptive bin-widths: h(x) p 0 (x), where the constants are chosen to control the approximation error. Adaptive bin-widths allow us to optimally re-distribute the approximation error. June 11, / 32

63 High-Level Proof Ideas Lower Bound: Classical method: Create many small perturbations of the null Consider distinguishing p0 from a uniform mixture over these perturbations Analyze the (optimal) Likelihood Ratio Test June 11, / 32

64 High-Level Proof Ideas Lower Bound: Classical method: Create many small perturbations of the null Consider distinguishing p0 from a uniform mixture over these perturbations Analyze the (optimal) Likelihood Ratio Test June 11, / 32

65 High-Level Proof Ideas Technical Challenges: When p 0 is far from uniform: Need to perturb some parts of p0 much more than other parts Smoothness constrains the allowed perturbations significantly June 11, / 32

66 High-Level Proof Ideas Technical Challenges: When p 0 is far from uniform: Need to perturb some parts of p0 much more than other parts Smoothness constrains the allowed perturbations significantly Key Idea: Again use adaptive bin-widths: When the bin-width is large, a larger perturbation is possible without violating smoothness Same adaptive bin-widths as in the upper bound result in optimal (and matching) lower bound June 11, / 32

67 Extending to Higher-Dimensions Define, and the truncated γ-norm: γ = d, ( ) 1/γ T ɛ (p 0 ) = p 0 (x) γ. B ɛ June 11, / 32

68 Extending to Higher-Dimensions Define, and the truncated γ-norm: γ = d, ( ) 1/γ T ɛ (p 0 ) = p 0 (x) γ. B ɛ Theorem (BW17) The local minimax rate is given as: ( ) 1 LT 2 4+d ɛ n = ɛ (p 0 ). Again, obtain significant variability in the minimax rate as a function of p 0. n 2 June 11, / 32

69 High-Level Proof Ideas Upper and lower bounds are based again on an adaptive partition. Roughly, want to partition the support of p 0 into hyper-cubes of different volume, where the volume of each hyper-cube: V (x) p 0 (x) dγ. Unlike in the 1D case, not obvious if such a partition exists, and how to construct it. June 11, / 32

70 High-Level Proof Ideas Upper and lower bounds are based again on an adaptive partition. Roughly, want to partition the support of p 0 into hyper-cubes of different volume, where the volume of each hyper-cube: V (x) p 0 (x) dγ. Unlike in the 1D case, not obvious if such a partition exists, and how to construct it. We provide a proof of existence and a recursive splitting algorithm that constructs the desired partition The existence proof utilizes smoothness in an elegant way: intuitively since p 0 is smooth, the desired volumes inherit this smoothness, and a partition satisfying these volume requirements might exist June 11, / 32

71 High-Level Proof Ideas The recursive partitioning algorithm: June 11, / 32

72 Simulation June 11, / 32

73 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge June 11, / 32

74 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena June 11, / 32

75 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena Provide tight local minimax upper and lower bounds June 11, / 32

76 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena Provide tight local minimax upper and lower bounds Paper also provides extensions which adapt to unknown problem specific parameters (smoothness parameters and ɛ) June 11, / 32

77 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena Provide tight local minimax upper and lower bounds Paper also provides extensions which adapt to unknown problem specific parameters (smoothness parameters and ɛ) Need to use careful, adaptive binning procedures June 11, / 32

78 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena Provide tight local minimax upper and lower bounds Paper also provides extensions which adapt to unknown problem specific parameters (smoothness parameters and ɛ) Need to use careful, adaptive binning procedures We are currently investigating many extensions: composite null, more smoothness, two-sample, etc. June 11, / 32

79 Summary For testing Lipschitz densities, interesting local minimax phenomena emerge Typical assumptions, bounded domain, uniform null, fixed smoothness constant can mask these phenomena Provide tight local minimax upper and lower bounds Paper also provides extensions which adapt to unknown problem specific parameters (smoothness parameters and ɛ) Need to use careful, adaptive binning procedures We are currently investigating many extensions: composite null, more smoothness, two-sample, etc. THE END June 11, / 32