Hunting for significance with multiple testing

Transcription

1 Hunting for significance with multiple testing Etienne Roquain 1 1 Laboratory LPMA, Université Pierre et Marie Curie (Paris 6), France Séminaire MODAL X, 19 mai 216 Etienne Roquain Hunting for significance with multiple testing 1 / 38

2 1 Introduction 2 Classical results 3 FDP control 4 Posthoc inference Etienne Roquain Hunting for significance with multiple testing 2 / 38

3 A multiple testing story" ( Etienne Roquain Hunting for significance with multiple testing Introduction 3 / 38

4 A multiple testing story" ( Etienne Roquain Hunting for significance with multiple testing Introduction 3 / 38

5 A multiple testing story" ( Etienne Roquain Hunting for significance with multiple testing Introduction 3 / 38

6 Shadoks motto Etienne Roquain Hunting for significance with multiple testing Introduction 4 / 38

7 False positive in practice Abstract of [Giovannucci et al. (1995)] "Between 1986 and 1992, 812 new cases of prostate cancer [...] were documented. Of 46 vegetables and fruits or related products, four were significantly associated with lower prostate cancer risk; of the four tomato sauce (P for trend =.1), tomatoes (P for trend =.3), and pizza (P for trend =.5), but not strawberries were primary sources of lycopene. Combined intake of tomatoes, tomato sauce, tomato juice, and pizza (which accounted for 82% of lycopene intake) was inversely associated with risk of prostate cancer (multivariate RR =.65; 95% CI = , for consumption frequency greater than 1 versus less than 1.5 servings per week; P for trend =.1) and advanced (stages C and D) prostate cancers (multivariate RR =.47; 95% CI =.22-1.; P for trend =.3)." Should we eat pizza to avoid prostate cancer? Etienne Roquain Hunting for significance with multiple testing Introduction 5 / 38

8 Massive and complex datasets Gene expressions interesting genes? Genome-wide assoc. study interesting SNPs? Copy number alterations interesting probes? Etienne Roquain Hunting for significance with multiple testing Introduction 6 / 38

9 Massive and complex datasets Neuroimaging (FMRI) activated regions? Neuroscience periods for neuronal response? Astronomy directions with stars? Etienne Roquain Hunting for significance with multiple testing Introduction 7 / 38

10 Single testing Data: X = (X 1,..., X n ) i.i.d. N (µ, 1), µ R unknown Null hypothesis H : µ " against alt. H 1 : µ > " Test statistic: T (X) = n 1/2 n i=1 X i, p-value p(x) = Φ(T (X)) T=.5 p=.31 T=2 p= Test of level α: reject H if p(x) α Risk under H : P(p(X) α) α Etienne Roquain Hunting for significance with multiple testing Introduction 8 / 38

11 Multiple testing Pure noise, m = 48 indep. tests: At least one false positive: with prob. 1 (1.5) Etienne Roquain Hunting for significance with multiple testing Introduction 9 / 38

12 Multiple testing Pure noise, m = 192 indep. tests: At least one false positive: with prob. 1 (1.5) Etienne Roquain Hunting for significance with multiple testing Introduction 9 / 38

13 Canonical setting X i = avg group 2 - avg group 1 (rescaled) for item i, 1 i m. Gaussian model : X N (µ, Γ) R m Hypotheses : H,i : µ i " vs H 1,i : µ i > ", 1 i m. Parameter of interest : θ i = 1{µ i > }, 1 i m. p-values: p i = Φ(X i ), 1 i m. Dependence: covariance matrix Γ (known or unknown) Row Row Row Column Dimensions: 2 x 2 Column Dimensions: 2 x 2 Column Dimensions: 2 x 2 Aim : recover θ from (p i (X), 1 i m) Etienne Roquain Hunting for significance with multiple testing Introduction 1 / 38

14 General setting Data X (X, X) with X P P (model) H,i : P P,i ", 1 i m, null hypotheses for P true/false label θ = θ(p) {, 1} m such that θ i = if and only if H,i is true for P Assume p-values (p i (X), 1 i m) such that if θ i =, p i (X) U(, 1) if θ i = 1, p i (X) let arbitrary (typically "small") Aim : recover θ from (p i (X), 1 i m) Etienne Roquain Hunting for significance with multiple testing Introduction 11 / 38

15 Thresholding i p i Rejection set R = {1 i m : p i (X) t} Some false positives Etienne Roquain Hunting for significance with multiple testing Introduction 12 / 38

16 Thresholding i p i Rejection set R = {1 i m : p i (X) t} Some false positives Etienne Roquain Hunting for significance with multiple testing Introduction 12 / 38

17 Thresholding i p i Rejection set R = {1 i m : p i (X) t} Some false positives Etienne Roquain Hunting for significance with multiple testing Introduction 12 / 38

18 Ordering p-values i p i Stopping rule Reject l smallest p-values Etienne Roquain Hunting for significance with multiple testing Introduction 13 / 38

19 Ordering p-values p(l) Stopping rule Reject l smallest p-values l Etienne Roquain Hunting for significance with multiple testing Introduction 13 / 38

20 Ordering p-values p(l) Stopping rule Reject l smallest p-values l Etienne Roquain Hunting for significance with multiple testing Introduction 13 / 38

21 Ordering p-values p(ell) Stopping rule Reject l smallest p-values l Etienne Roquain Hunting for significance with multiple testing Introduction 13 / 38

22 Choosing procedure? "Quality" criterion? Etienne Roquain Hunting for significance with multiple testing Introduction 14 / 38

23 1 Introduction 2 Classical results 3 FDP control 4 Posthoc inference Etienne Roquain Hunting for significance with multiple testing Classical results 15 / 38

24 Family-wise error rate (FWER) For a threshold t, V (t) = m i=1 (1 θ i)1{p i t}, FWER(t, P) = P(V (t) 1) = P(p (1:H ) t) FWER control α given, find t = t α with P P, FWER(t, P) α, Clear interpretation Bonferroni t = α/m (union bound) Possible refinements : X N (µ, Γ) (one-sided), Γ known V (t) V (t) = m 1{Φ(Z i ) t}, Z N (, Γ) i=1 Choose t such that P Z N (,Γ) (Φ(Z ) (1:m) t) α Relaxation to k-fwer. Etienne Roquain Hunting for significance with multiple testing Classical results 16 / 38

25 Family-wise error rate (FWER) For a threshold t, V (t) = m i=1 (1 θ i)1{p i t}, FWER(t, P) = P(V (t) 1) = P(p (1:H ) t) FWER control α given, find t = t α with P P, FWER(t, P) α, Clear interpretation Bonferroni t = α/m (union bound) Possible refinements : X N (µ, Γ) (one-sided), Γ known V (t) V (t) = m 1{Φ(Z i ) t}, Z N (, Γ) i=1 Choose t such that P Z N (,Γ) (Φ(Z ) (1:m) t) α Relaxation to k-fwer. Etienne Roquain Hunting for significance with multiple testing Classical results 16 / 38

26 Family-wise error rate (FWER) For a threshold t, V (t) = m i=1 (1 θ i)1{p i t}, FWER(t, P) = P(V (t) 1) = P(p (1:H ) t) FWER control α given, find t = t α with P P, FWER(t, P) α, Clear interpretation Bonferroni t = α/m (union bound) Possible refinements : X N (µ, Γ) (one-sided), Γ known V (t) V (t) = m 1{Φ(Z i ) t}, Z N (, Γ) i=1 Choose t such that P Z N (,Γ) (Φ(Z ) (1:m) t) α Relaxation to k-fwer. Etienne Roquain Hunting for significance with multiple testing Classical results 16 / 38

27 Family-wise error rate (FWER) For a threshold t, V (t) = m i=1 (1 θ i)1{p i t}, FWER(t, P) = P(V (t) 1) = P(p (1:H ) t) FWER control α given, find t = t α with P P, FWER(t, P) α, Clear interpretation Bonferroni t = α/m (union bound) Possible refinements : X N (µ, Γ) (one-sided), Γ known V (t) V (t) = m 1{Φ(Z i ) t}, Z N (, Γ) i=1 Choose t such that P Z N (,Γ) (Φ(Z ) (1:m) t) α Relaxation to k-fwer. Etienne Roquain Hunting for significance with multiple testing Classical results 16 / 38

28 FWER(Bonf.) illustration, α =.2, m = Noncorrected Bonferroni Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

29 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

30 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

31 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

32 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

33 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

34 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

35 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

36 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

37 FWER(Bonf.) illustration, α =.2, m = Etienne Roquain Hunting for significance with multiple testing Classical results 17 / 38

38 False discovery rate (FDR) What should we choose for k? 4 false positives out of 1 positives? 7 false positives out of 1 positives? Idea : link k to the number of positives. For a threshold t, R( t) = m i=1 1{p i t}, FDR( t, P) = E[FDP( t, P)], FDP( t, P) = V (t) R(t) ( ) = FDR control α given, find t = t α with P P, FDR( t, P) α. If FDR.5: if 2 rejections, allow at most 1 false discovery (on average) if 1 rejections, allow at most 5 false discoveries (on average) Etienne Roquain Hunting for significance with multiple testing Classical results 18 / 38

39 False discovery rate (FDR) What should we choose for k? 4 false positives out of 1 positives? 7 false positives out of 1 positives? Idea : link k to the number of positives. For a threshold t, R( t) = m i=1 1{p i t}, FDR( t, P) = E[FDP( t, P)], FDP( t, P) = V (t) R(t) ( ) = FDR control α given, find t = t α with P P, FDR( t, P) α. If FDR.5: if 2 rejections, allow at most 1 false discovery (on average) if 1 rejections, allow at most 5 false discoveries (on average) Etienne Roquain Hunting for significance with multiple testing Classical results 18 / 38

40 False discovery rate (FDR) What should we choose for k? 4 false positives out of 1 positives? 7 false positives out of 1 positives? Idea : link k to the number of positives. For a threshold t, R( t) = m i=1 1{p i t}, FDR( t, P) = E[FDP( t, P)], FDP( t, P) = V (t) R(t) ( ) = FDR control α given, find t = t α with P P, FDR( t, P) α. If FDR.5: if 2 rejections, allow at most 1 false discovery (on average) if 1 rejections, allow at most 5 false discoveries (on average) Etienne Roquain Hunting for significance with multiple testing Classical results 18 / 38

41 False discovery rate (FDR) What should we choose for k? 4 false positives out of 1 positives? 7 false positives out of 1 positives? Idea : link k to the number of positives. For a threshold t, R( t) = m i=1 1{p i t}, FDR( t, P) = E[FDP( t, P)], FDP( t, P) = V (t) R(t) ( ) = FDR control α given, find t = t α with P P, FDR( t, P) α. If FDR.5: if 2 rejections, allow at most 1 false discovery (on average) if 1 rejections, allow at most 5 false discoveries (on average) Etienne Roquain Hunting for significance with multiple testing Classical results 18 / 38

42 BH procedure [Benjamini and Hochberg, 1995] t = α( l 1)/m with l = max{ l m : p(l) αl/m} Noncorrected BH Bonferroni Etienne Roquain Hunting for significance with multiple testing Classical results 19 / 38

43 FDR(BH) illustration, α =.2, m = FDP = Etienne Roquain Hunting for significance with multiple testing Classical results 2 / 38

44 FDR(BH) illustration, α =.2, m = FDP = Etienne Roquain Hunting for significance with multiple testing Classical results 2 / 38

45 FDR(BH) illustration, α =.2, m = FDP = Etienne Roquain Hunting for significance with multiple testing Classical results 2 / 38

46 FDR(BH) illustration, α =.2, m = FDP = Etienne Roquain Hunting for significance with multiple testing Classical results 2 / 38

47 FDR(BH) illustration, α =.2, m = FDP = Etienne Roquain Hunting for significance with multiple testing Classical results 2 / 38

48 FDR control for BH Theorem [Benjamini and Hochberg (1995)] Assume p i U(, 1) for θ i = (p i, 1 i m) are mutually indep for any P P. Then for all P P, FDR(BH α, P) π α α for π = m /m proportion of zeros in θ(p). Simple proof [Blanchard and R. (28, EJS)] Lemma : Let U U(, 1) and g : [, 1] [, ) nonincreasing measurable. Then ( ) 1{U c g(u)} E 1{g(U) > } c, for any c >. g(u) Etienne Roquain Hunting for significance with multiple testing Classical results 21 / 38

49 FDR control for BH Theorem [Benjamini and Hochberg (1995)] Assume p i U(, 1) for θ i = (p i, 1 i m) are mutually indep for any P P. Then for all P P, FDR(BH α, P) π α α for π = m /m proportion of zeros in θ(p). Simple proof [Blanchard and R. (28, EJS)] Lemma : Let U U(, 1) and g : [, 1] [, ) nonincreasing measurable. Then ( ) 1{U c g(u)} E 1{g(U) > } c, for any c >. g(u) Etienne Roquain Hunting for significance with multiple testing Classical results 21 / 38

50 FDR control for BH Theorem [Benjamini and Hochberg (1995)] Assume p i U(, 1) for θ i = (p i, 1 i m) are mutually indep for any P P. Then for all P P, FDR(BH α, P) π α α for π = m /m proportion of zeros in θ(p). Simple proof [Blanchard and R. (28, EJS)] Lemma : Let U U(, 1) and g : [, 1] [, ) nonincreasing measurable. Then ( ) 1{U c g(u)} E 1{g(U) > } c, for any c >. g(u) Etienne Roquain Hunting for significance with multiple testing Classical results 21 / 38

51 Extension to positive dependence Theorem [Benjamini and Yekutieli (21)] For all P P, FDR(BH α, P) π α if p i U(, 1) for θ i = p = (p i, 1 i m) are weak PRDS : P P, i D [, 1] m, u P(p D p i u) Simple proof [Blanchard and R. (28, EJS)] Lemma : Let U U(, 1); V such that v, u P(V < v U u) Then ( ) 1{U c V } E 1{V > } c, for any c >. V Etienne Roquain Hunting for significance with multiple testing Classical results 22 / 38

52 Extension to positive dependence Theorem [Benjamini and Yekutieli (21)] For all P P, FDR(BH α, P) π α if p i U(, 1) for θ i = p = (p i, 1 i m) are weak PRDS : P P, i D [, 1] m, u P(p D p i u) Simple proof [Blanchard and R. (28, EJS)] Lemma : Let U U(, 1); V such that v, u P(V < v U u) Then ( ) 1{U c V } E 1{V > } c, for any c >. V Etienne Roquain Hunting for significance with multiple testing Classical results 22 / 38

53 FDR revolution Benjamini and Hochberg (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing GENETICS HEREDITY BIOTECHNOLOGY APPLIED MICROBIOLOGY NEUROSCIENCES NEUROLOGY MATHEMATICS SCIENCE TECHNOLOGY OTHER TOPICS BIOCHEMISTRY MOLECULAR BIOLOGY OTHERS COMPUTER SCIENCE ENVIRONMENTAL SCIENCES ECOLOGY ONCOLOGY MATHEMATICAL COMPUTATIONAL BIOLOGY Figure: 13,427 papers citing this work ( ) according to "the web of science". 3, 85 citations on google scholar Etienne Roquain Hunting for significance with multiple testing Classical results 23 / 38

54 1 Introduction 2 Classical results 3 FDP control 4 Posthoc inference Etienne Roquain Hunting for significance with multiple testing FDP control 24 / 38

55 Motivation Distribution of FDP(BH) under ρ-equicorrelation density mean median 95% quantile ρ = ρ =.1 ρ = Etienne Roquain Hunting for significance with multiple testing FDP control 25 / 38

56 Aim Choose new critical values τ l, 1 l m (incorporating dep. structure) ρ = ρ =.1 ρ =.5 ζ =.5 ζ =.25 ζ = such that t = τˆl (same way as BH) satisfies P(FDP( t) α) 1 ζ. Etienne Roquain Hunting for significance with multiple testing FDP control 26 / 38

57 Method [Delattre and R. New procedures controlling the false discovery proportion via Romano-Wolf s heuristic (215, AoS)] Proposition: B(l, l ) = P(V (τ l ) αl + 1) V (t) V (t) for all t (with V (t) in t and V () = a.s.) Then P(FDP( t) > α) m ( ) ( ) B(l, l) B(l 1, l) B(l, l 1) B(l 1, l 1) l=1 Example : X N (µ, Γ) (one-sided), Γ known ( m ) B(l, l ) = P Z N (,Γ) 1{Φ(Z i ) τ l } αl + 1. i=1 Incorporate dependence, much powerful than [Guo et al. (214, AoS)] Etienne Roquain Hunting for significance with multiple testing FDP control 27 / 38

58 1 Introduction 2 Classical results 3 FDP control 4 Posthoc inference Etienne Roquain Hunting for significance with multiple testing Posthoc inference 28 / 38

59 Coming back to the pizza... Abstract of [Giovannucci et al. (1995)] "Between 1986 and 1992, 812 new cases of prostate cancer [...] were documented. Of 46 vegetables and fruits or related products, four were significantly associated with lower prostate cancer risk; of the four tomato sauce (P for trend =.1), tomatoes (P for trend =.3), and pizza (P for trend =.5), but not strawberries were primary sources of lycopene. Combined intake of tomatoes, tomato sauce, tomato juice, and pizza (which accounted for 82% of lycopene intake) was inversely associated with risk of prostate cancer (multivariate RR =.65; 95% CI = , for consumption frequency greater than 1 versus less than 1.5 servings per week; P for trend =.1) and advanced (stages C and D) prostate cancers (multivariate RR =.47; 95% CI =.22-1.; P for trend =.3)." How many false positives in that abstract? Etienne Roquain Hunting for significance with multiple testing Posthoc inference 29 / 38

60 Volcano plot [Chiaretti et al. (24)] Etienne Roquain Hunting for significance with multiple testing Posthoc inference 3 / 38

61 Bias selection what means statistic values in the selected? Simulate i.i.d. absolute values of N (, 1) Data 1 : 5 values Data2 : 1 values Etienne Roquain Hunting for significance with multiple testing Posthoc inference 31 / 38

62 Posthoc inference Selective inference = after some selection procedure. e.g., after LASSO selection [Tibshirani et al. (215), Barber and Candes (215,AoS)] In omnia paratus posthoc inference = ready for any selection. Number of false positives in R {1,..., m} : V (R) = m (1 θ i )1{i R}. i=1 Aim: posthoc bound α given, find V α ( ) N, such that for all P P, P ( R {1,..., m} : V (R) V α (R) ) 1 α Proposed in [Goeman and Solari (211,StatScience)] (with closed testing) Etienne Roquain Hunting for significance with multiple testing Posthoc inference 32 / 38

63 Our method: reference family If A H 5, B H 7 B. A =. Etienne Roquain Hunting for significance with multiple testing Posthoc inference 33 / 38

64 Our method: reference family If A H 5, B H 7 R H? R B. A =. Etienne Roquain Hunting for significance with multiple testing Posthoc inference 33 / 38

65 Our method: reference family If A H 5, B H 7 R H 1 R B. A =. Etienne Roquain Hunting for significance with multiple testing Posthoc inference 33 / 38

66 JR criterion [Blanchard, Neuvial and R. On controlled posthoc inference (ongoing work)] Define for T = {t k } 1 k m, JR(T, P) = P( k {1,..., m} : V (t k ) k) = 1 P( k {1,..., m} : {i : p i t k } H k 1) Proposition 1 JR control at level α implies posthoc bound: { } V α (R) = 1{p i (X) > t k } + k 1, R {1,..., m}. min 1 k m i R Etienne Roquain Hunting for significance with multiple testing Posthoc inference 34 / 38

67 JR criterion [Blanchard, Neuvial and R. On controlled posthoc inference (ongoing work)] Define for T = {t k } 1 k m, JR(T, P) = P( k {1,..., m} : V (t k ) k) = 1 P( k {1,..., m} : {i : p i t k } H k 1) Proposition 1 JR control at level α implies posthoc bound: { } V α (R) = 1{p i (X) > t k } + k 1, R {1,..., m}. min 1 k m i R Etienne Roquain Hunting for significance with multiple testing Posthoc inference 34 / 38

68 JR criterion [Blanchard, Neuvial and R. On controlled posthoc inference (ongoing work)] Define for T = {t k } 1 k m, JR(T, P) = P( k {1,..., m} : V (t k ) k) = 1 P( k {1,..., m} : {i : p i t k } H k 1) Proposition 1 JR control at level α implies posthoc bound: { } V α (R) = 1{p i (X) > t k } + k 1, R {1,..., m}. min 1 k m i R Etienne Roquain Hunting for significance with multiple testing Posthoc inference 34 / 38

69 Controlling JR Proposition 2 Taking t k = t k (λ) and right-continuous in λ, ( { } JR(T, P) = P min t 1 k (p (k:h )) 1 k m ) λ Example : λ-adjustement X N (µ, Γ) (one-sided), Γ known t k (λ) = λk/m λ chosen as α-quantile of ( { m D Z N (,Γ) min (k:m)} ) k Φ(Z ) 1 k m Etienne Roquain Hunting for significance with multiple testing Posthoc inference 35 / 38

70 Outlook Various criteria FWER FDR FDP JR Various challenges Surviving to dependence Incorporating known dependence Adaptating to unknown dependence Etienne Roquain Hunting for significance with multiple testing Posthoc inference 36 / 38

71 Continuous testing in a nutshell [Blanchard, Delattre and R. Testing over a continuum of null hypotheses with False Discovery Rate control (214, Bernoulli)] (H,t, t [, 1]) continuous set of null hypotheses (p t (X), t [, 1]) p-value process (jointly-measurable) continuous FDR FDR( t, P) = E[FDP( t, P)], FDP=.83 FDP( t, P) = Λ({t : θ t =, p t (X) t}) Λ({t : p t (X) t}) pvalue process correct reject. erroneous reject. threshold= Continuous BH procedure t α Finite dimensional PRDS Result : continuous FDR control Etienne Roquain Hunting for significance with multiple testing Posthoc inference 37 / 38

72 Continuous testing in a nutshell [Blanchard, Delattre and R. Testing over a continuum of null hypotheses with False Discovery Rate control (214, Bernoulli)] (H,t, t [, 1]) continuous set of null hypotheses (p t (X), t [, 1]) p-value process (jointly-measurable) continuous FDR FDR( t, P) = E[FDP( t, P)], FDP=.83 FDP( t, P) = Λ({t : θ t =, p t (X) t}) Λ({t : p t (X) t}) pvalue process correct reject. erroneous reject. threshold= Continuous BH procedure t α Finite dimensional PRDS Result : continuous FDR control Etienne Roquain Hunting for significance with multiple testing Posthoc inference 37 / 38

73 Application [Picard et al. Continuous testing for Poisson process intensities (ongoing work)] N A heterogeneous poisson process intensity λ A N B heterogeneous poisson process intensity λ B H,t : λ A = λ B sur I t " I t, t [, 1], sliding windows Basic test statistics give finite dim PRDS Continuous BH = data-driven weighted BH Pouzat, package STAR : Spike Train Analysis with R. Etienne Roquain Hunting for significance with multiple testing Posthoc inference 38 / 38