Large-Scale Multiple Testing in Medical Informatics

Transcription

1 Large-Scale Multiple Testing in Medical Informatics experiment space model space data IMA Workshop on Large Data Sets in Medical Informatics November Rob Nowak Joint work with R. Castro, J. Haupt, M. Malloy and B. Nazer

2 Motivation: Inferring Biological Pathways microwell array virus 13,071 single-gene knock-down cell strains infect each strain with fluorescing virus 72 h fruit fly approximately 100 significant genes/protiens discovered by Ahlquist and Kawaoka Labs at UW-Madison

3 Motivation: Inferring Biological Pathways microwell array virus 13,071 single-gene knock-down cell strains infect each strain with fluorescing virus 72 h fruit fly Challenges: 1. very low SNR data and huge number of experiments and tests 2. non-linear interactions

4 Challenge 1: High-Dimensionality and Low SNR Vol August 2008 doi: /nature07151 Drosophila RNAi screen identifies host genes important for influenza virus replication Linhui Hao 1,2 *, Akira Sakurai 3 *{, Tokiko Watanabe 3, Ericka Sorensen 1, Chairul A. Nidom 5,6, Michael A. Newton 4, Paul Ahlquist 1,2 & Yoshihiro Kawaoka 3,7,8,9 How do they confidently determine the ~100 out of 13K genes hijacked for virus replication from extremely noisy data? Sequential Experimental Design: Stage 1: assay all 13K strains, twice; keep all with significant fluorescence in one or both assays for 2nd stage (13K 1K) Stage 2: assay remaining 1K strains, 6-12 times; retain only those with statistically significant fluorescence (1K 100) vastly more efficient that replicating all 13K experiments many times

5 Feedback from Data Analysis to Data Collection high-throughput experiments experiment space Optimized multi-stage designs controlling the false discovery or the family-wise error rate S. Zehetmayer, P. Bauer and M. Posch, Statist. Med. 2008; 27: model space data sets of genes critical to a certain function/process microarray or assay datasets

6 Challenge 2: Sparsity & Nonlinear Effects Genome Linear Pathway Redundant Pathways Genome Knockdown Knockdown Hijacked Proteins Hijacked Proteins detectable effect Linear Sparsity little or no observable effect! Multilinear Sparsity Detecting redundant pathways requires pairwise knockdowns and brute-force examination of all such knockdowns would require millions of experiments!

7 Outline of Talk 1. Sequential Experimental Designs for High-Dimensional Testing thresholds for recovery in high-dimensional limit: non-adaptive designs sequential designs SNR log n SNR arbitrarily slowly growing function of n 2. Compressed Sensing of Sparse Multilinear Functions number of compressed sensing measurements for sparse recovery: linear sparsity K S log n multilinear sparsity K min{s 2 log n, S log 3 (S) log n, S α log α n} where α 1 depends on pattern of sparsity

8 Multiple Testing Problem virus 13,071 single-gene knock-down cell strains infect each strain with fluorescing virus 72 h fruit fly microwell array fluorescence level genes which strains have statistically significant deviations in fluorescence?

9 Idealized Example non-sequential design (n cell strains, k samples each) two-stage design (adaptively allocate kn samples) y measure each cell strain with equal precision/snr, then threshold to control false-positive error may be impossible to reliably separate signals from noise first stage has large false-positive rate, but low false-negative. larger SNR in second stage may make it easier to separate signals from noise. Under a fixed sensing/experimental budget, does this two-stage design (or some other sequential design) provide better error control than non-sequential design?

10 Model Selection Wired Science News for Your Neurons Previous post Next post Challenge: huge number of possible models, each with a different pattern of sparsity Scanning Dead Salmon in fmri Machine Highlights Risk of Red Herrings By Alexis Madrigal September 18, :37 pm Categories: Brains and Behavior

11 The Multiple Testing Problem

12

13 Single Experiment Model y i = x i + z i,i=1,..., n z i iid N (0, 1) sparsity: x i = 0 except on a small subset S {1,..., n} where x i = µ>0 y x Suppose we want to locate just one signal component: î = arg max i y i Even if no signal is present, max i y i 2 log n It is impossible to reliably detect signal components if µ< 2 log n

14 An Alternative: Sequential Experimental Design Instead of the usual non-adaptive observation model y i = x i + z i,i=1,..., n suppose we are able to sequentially collect several independent measurements of each component of x, according to where y i,j = x i + γ 1/2 i,j z i,j, i =1,..., n, j =1,..., k j indexes the measurement steps k denotes the total number of steps z i,j iid N (0, 1) γ 1/2 i,j z i,j N (0, γ 1 i,j ) γ i,j 0 controls the precision of each measurement Total precision budget is constrained, but the choice of γ i,j can depend on past observations {y i,l } l<j.

15 Experimental (Precision) Budget sequential measurement model y i,j = x i + γ 1/2 i,j z i,j, i =1,..., n, j =1,..., k The precision parameters {γ i,j } are required to satisfy k n γ i,j n j=1 i=1 For example, the usual non-adaptive, single measurement model corresponds to taking k = 1, and γ i,1 = 1, i =1,..., n. This baseline can be compared with adaptive procedures by allowing k>1 and variable {γ i,j } satisfying budget. Precision parameters control the SNR per component. SNR is increased/decreased by more/fewer repeated samples or longer/shorter observation times allocate precision sequentially and adaptively

16 Sequential Thresholding Sequential Thresholding initialize: S 0 = {1,..., n}, γ 1 i,j =2 for j =1,..., k 1) measure: y i,j N (x i, 2),i S j 1 2) threshold: S j = {i : y i,j 0} end output: S k = {i : y i,k > 0} precision = { 1 2, if measured 0, otherwise define the sparsity level s := S total precision budget: E [ i,j γ i,j = k E S j 1 j=1 k j=1 ( ) n s 2 j 1 + s n s + ks n ] (when n s) probability of error: P(S k S) = P ({S c S k } {S S c k }) P (S c S k ) + P (S S c k )

17 Idealized Example threshold at zero and re-measure only those components that survive repeat several times most of true signal components survive several thresholding steps, almost all of noise components do not

18 Low False-Negative, High False-Positive Rates 0 µ

19 False Positives P(S k S) P (S c S k ) + P (S S c k ) P (S c S k ) = P i S P i S = i S k j=1 k j=1 y i,j > 0 y i,j > 0 2 k = n s 2 k

20 False Negatives P(S k S) P (S c S k ) + P (S S c k ) P (S Sk c ) = P k j=1 i S ( µ2 ks 2 exp y i,j < 0 4 )

21 Probability of Error Bound P(S k S) P (S c S k ) + P (S Sk c ) n s 2 k + ks ) ( 2 exp µ2 4 = n s 2 k + 1 ( ) 2 exp (µ2 4 log(ks)) 4 Consider high-dimensional limit as n and take k = log 2 n 1+ɛ 0 P(S k S) n s 2 k exp ( ) (µ2 4 log(s(1 + ɛ) log 2 n)) 4 Second term tends to zero if µ c log(s log 2 n), for any c>4

22 Gains of Sequential Design non-sequential: µ 2 log n (necessary) Rui Castro (Eindhoven) Jarvis Haupt (Minnesota) Matt Malloy (Madison) sequential thresholding: (sufficient) µ c(log s + log log 2 n), with c>4 with a bit more work we can show c>2 suffices sequential lower bound: (necessary for P(S k S) ɛ) µ 2 log s/ɛ achievable by SPRT, but requires perfect knowlege of s greater sensitivity for same precision budget or lower experimental requirements for equivalent sensitivity

23 Biology Example 13,071 single-gene knock-down cell strains prob(error) 3 db log s vs. log n SNR (db) sequential thresholding is about twice as sensitive (for equal experimental budget) or requires half the number experiments (for same sensitivity)

24 Breaking the log(s) Barrier probability of error: P(one or more false discoveries) α (too conservative) Alternative: # false discoveries α Ŝ false-discovery proportion: FDP(Ŝ) := S\ b S S = # falsely discovered components # discovered components non-discovery proportion: NDP(Ŝ) := S\ b S S = # missed components # true non-zero components assumptions: Gaussian model, sublinear sparsity level: s = n 1 β, β (0, 1) To guarantee that FDP and NDP tend to zero as n non-sequential sequential thresholding µ 2β log n (Ingster, Jin & Donoho) µ arbitrarily slowly growing function of n sequential thresholding has virtually no dependence on dimension or sparsity level, effectively eliminating the curse of dimensionality altogether

25 Example n =2 14, x 0 = n = 128 non-discovery rate of single experiment gain log(16384) 10 non-discovery rate of seq. test 5% false discovery rate for both methods

26 Challenge 2: Nonlinearities Genome Genome Knockdown Knockdowns Hijacked Proteins Hijacked Proteins no effect detectable effect must knockdown both redundant genes to see an effect! ( ) 2 85, 000, 000 possible two-fold gene deletion strains!

27 Sparse Interaction Models Knockdown y = i a i x (1) i + i<j a i a j x (2) ij Approximate output (virus reproduction) with a sparse bilinear system. x (1) i x (2) ij a i non-zero iff gene is critical to pathway non-zero iff gene pair is critical to pathway 1 if gene is knocked down; 0 otherwise sparsity = most are 0

28 Sensing Sparse Interactions Linear model: y(x) = i Bilinear model: y(x) = i x i a i x (1) i a i + i<j x (2) i,j a ia j ( ) 2 85, 000, 000 possible pairwise interactions! Since most coefficients, {x i } or {x (1) i,x (2) ij }, are zero our goal is to identify critical components and interactions from using very few measurements a = {+,, +, +,,,, +} random bit sequence e.g., random selection of multiple simultaneous gene knock-downs model x y(x) collect K size(x) measurements y 1,y 2,..., y K using K random inputs (related work also by G. Giannakis U. Minnesota)

29 (Linear) Compressed Sensing This is the conventional compressed sensing problem for the linear model. y k = i a ki x i, k =1,..., K x R n, a { 1, +1} n sparse x: x 0 = S n find sparse solution to y = Ax If the measurement matrix satisfies the restricted isometry property (RIP) with δ 2S < 2 1 for all S-sparse vectors: (1 δ S ) x 2 2 Ax 2 2 (1 + δ S ) x 2 2 then x can be recovered from y by convex optimization: min z 1 subject to Az = y RIP holds with high probability if K cs log(n/s)

30 Multilinear Compressed Sensing Multilinear model: y = a i1 a i2 a id x i1 i 2 i D i 1 <i 2 < <i D x 0 = S N x R N, a { 1, 1} n N = ( ) n D y is called a Rademacher chaos of order D Compressed sensing problem for multilinear model K measurements of this form: y =[y 1 y K ] T find sparse solution to y = Ax matrix A now composed of monomials in a i,j Does it satisfy the RIP property?

31 On Average, Things Look Good RIP: (1 δ S ) x 2 2 Ax 2 2 (1 + δ S ) x 2 2 isotropic measurements: E [ Ax 2] = x 2 2 symmetric binary random inputs: P(a i,j = +1) = P(a i,j = 1) = 1/2 Linear CS: n = 3 inputs, K = 2 measurements and D = 1, A = 1 [ ] a1,1 a 1,2 a 1,3 E[A T A] = Identity 2 a 2,1 a 2,2 a 2,3 Bilinear CS: n = 3 inputs, K = 2 measurements and D = 2, A = 1 [ ] a1,1 a 1,2 a 1,1 a 1,3 a 1,2 a 1,3 E[A T A] = Identity 2 a 2,1 a 2,2 a 2,1 a 2,3 a 2,2 a 2,3

32 What about the distributions? y = Ax, E y 2 2 = x 2 2 P( y 2 x 2 2 >t) exp( poly(t)) x 2 2 x 2 2 Gaussian tails (as in linear CS) or heavy tails?

33 Tail Behavior Best case: decoupled chaos y = a 1 a 2 x 1,2 + a 3 a 4 x a 2k 1 a 2k x 2k 1,2k ã 1 x 1,2 + ã 2 x 3,4 + + ã k x 2k 1,2k equivalent to iid binary symmetric sensing subgaussian tails independent of D: P(y 2 x 2 2 >t) exp( ct) Worst case: strongly coupled chaos y = a i a j x i,j, k := 1 i<j l ( ) l 2 significant probability of large deviations from mean: if x i,j =1/ k, then P(y 2 k) =2 l =2 ck1/2 heavy tails depending on D: P(y 2 x 2 2 >t) exp( ct 1/D )

34 Combinatorial Dimension of Rademacher Chaos The combinatorial dimension 1 α D measures the level of dependence introduced by a particular pattern of sparsity. y = i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D Blei-Janson 04: A Rademacher chaos with combinatorial dimensional α satisfies ( exp c 1 t 1/α) P ( y 2 >t ) exp ( c 2 t 1/α) tails are light to heavy, depending on 1 α D

35 Dependencies Matter (in practice) 0.25 RIP: (1 δ S ) x 2 2 Ax 2 2 (1 + δ S ) x linear 1 δ S bilinear 0.05 trilinear support size S K = 5S

36 Dependencies Matter (in theory) K = number of measurements needed to recovery S-sparse multilinear forms proof technique bound on K ingredients Gershgorin Rudelson- Vershynin Rademacher chaos S 2 log N S(log 3 S)(log N) S α log α (N/S) 1 α D empirical 2nd moment bounds, union bound empirical 2nd moment bounds, no union bound tail bounds union bound compare with linear CS bound: K S log(n/s)

37 Gershgorin Bound i) Control each element of (partial) Gram matrix G T = A T T A T using Hoeffding s inequality and bound probability that G T is approximately diagonal. G R T = δ 1 SS δ SS δ SS δ 1 SS δ SS δ SS 1 ii) Gershgorin s Disc Theorem guarantees that eigenvalues lie in the range g ii j i g ij λ i (G T ) g ii + j i g ij iii) union bound over all ( N S ) sparsity patterns. RIP holds if K cs 2 log N

38 Heavy-Tailed Restricted Isometries Theorem 1 (Vershynin) Let à be a K N measurement matrix whose rows a T i are independent isotropic random vectors in R N. Let B be a number such that all entries a ij B almost surely. Then the normalized matrix A = 1 K à satisfies the following for K N, for every sparsity level S N and 0 < ɛ< 1: if the number of measurements satisfies K C ɛ 2 S log N log 3 (S) then the RIP constant δ S of A satisfies E[δ S ] ɛ. Check conditions: isotropy: G := A T A, E[G] = Identity elements of à bounded by 1.

39 Chaos Tail Bound Lemma 1 Assume that y k, k =1,..., K, are i.i.d. Rademacher variables of order D with combinatorial dimension 1 α D and Eyk 2 = 1. There exist constants c, C > 0 such that ( ) 1 K P yk 2 1 >t C exp( c min(kt 2,K 1/α t 1/α )) K k=1 proof technique: Blei-Jansen chaos tail bounds moment bound for sums of symmetric i.i.d. variables due to R. Latala apply lemma and union bound over ɛ-net for sparse vectors (technique from Baraniuk-Devore-Davenport-Wakin 08) RIP holds if K CS α log α (N/S)

40 Conclusions Compressed Sensing of Sparse Multilinear Functions number of compressed sensing measurements for sparse recovery: linear sparsity K S log n multilinear sparsity K min{s 2 log n, S log 3 (S) log n, S α log α n} where α 1 depends on pattern of sparsity Sparse Interactions: Identifying High-Dimensional. Multilinear Systems via Compressed Sensing, B. Nazer and RN, Allerton 2010 Sequential Experimental Designs for High-Dimensional Testing thresholds for recovery in high-dimensional limit: non-adaptive designs sequential designs SNR log n SNR arbitrarily slowly growing function of n Distilled Sensing: Adaptive Sampling for Sparse Detection and Estimation J. Haupt, R. Castro, and RN, arxiv: v2