Non-Parametric Combination (NPC) & classical multivariate tests

Transcription

1 Non-Parametric Combination (NPC) & classical multivariate tests Anderson M. Winkler fmrib Analysis Group 5.May.26 Winkler Non-Parametric Combination (NPC) / 55

2 Winkler Non-Parametric Combination (NPC) 2 / 55

3 Winkler Non-Parametric Combination (NPC) 3 / 55

4 Quick glm overview Model: Y = Mψ + ɛ Null hypothesis: H : C ψ = Winkler Non-Parametric Combination (NPC) 4 / 55

5 Quick glm overview. Partition the model (done by default, no need to change): Y = Mψ + ɛ Y = Xβ + Zγ + ɛ 2. Choose a permutation strategy (Freedman Lane or Dekker by default, no need to change). 3. Choose assumptions (ee and/or ise, may need changing). 4. Run! Winkler Non-Parametric Combination (NPC) 5 / 55

6 Quick glm overview: Two-sample t test Y = Mψ + ɛ = [ t = ] How likely is a value at least as large as this if there is no effect? Winkler Non-Parametric Combination (NPC) 6 / 55

7 Quick glm overview: Two-sample t test t = +.33 t =.9 t = +.25 t =.93 t = +.42 t = t = +.4 t =.45 t = 2.4 t =.47 t = +.47 t = +2.4 t = +.45 t =.4 t = 2.24 t =.42 t = +.93 t =.25 t = +.9 t =.33 Winkler Non-Parametric Combination (NPC) 7 / 55

8 Quick glm overview: Two-sample t test t = +.33 t =.9 t = +.25 t =.93 t = +.42 t = t = +.4 t =.45 t = 2.4 t =.47 t = +.47 t = +2.4 t = +.45 t =.4 t = 2.24 t =.42 t = +.93 t =.25 t = +.9 t =.33 Winkler Non-Parametric Combination (NPC) 8 / 55

9 Quick glm overview: Two-sample t test There were 3 cases of a statistic at least as large as the one observed. We have run 2 permutations. Thus: p = 3 2 =.5 Winkler Non-Parametric Combination (NPC) 9 / 55

10 Quick glm overview Permutation tests are superior: - Reasonable assumptions: data is exchangeable. - Wide variety of statistics (but needs pivotality). - Good for small datasets. - All information needed is in the dataset itself, not on an idealised population. - Resilient to outliers. Winkler Non-Parametric Combination (NPC) / 55

11 Classical Multivariate Tests (manova, mancova, cca, etc) Winkler Non-Parametric Combination (NPC) / 55

12 Multivariate tests GLM again: Y = Mψ + ɛ Y = Xβ + Zγ + ɛ (but now Y is a N K matrix.) Winkler Non-Parametric Combination (NPC) 2 / 55

13 Univariate case (e.g. fa data) Y N = Mψ + ɛ fa fa 2 fa 3 fa 4 fa 5 fa 6 = [ ψ ψ 2 ψ i =? ɛ i =? ] + ɛ ɛ 2 ɛ 3 ɛ 4 ɛ 5 ɛ 6 Estimation: ˆψ = (M M) M Y, ˆɛ = Y M ˆψ. Winkler Non-Parametric Combination (NPC) 3 / 55

14 Multivariate case (e.g. fa, md and rd) Y N K = Mψ + ɛ fa md rd fa 2 md 2 rd 2 fa 3 md 3 rd 3 fa 4 md 4 rd 4 = fa 5 md 5 rd 5 fa 6 md 6 rd 6 [ ] ψfa, ψ md, ψ rd, ψ fa,2 ψ md,2 ψ rd,2 ψ i =? ɛ i =? Estimation: ˆψ = (M M) M Y, ˆɛ = Y M ˆψ. ɛ fa, ɛ md, ɛ rd, ɛ fa,2 ɛ md,2 ɛ rd,2 + ɛ fa,3 ɛ md,3 ɛ rd,3 ɛ fa,4 ɛ md,4 ɛ rd,4 ɛ fa,5 ɛ md,5 ɛ rd,5 ɛ fa,6 ɛ md,6 ɛ rd,6 Winkler Non-Parametric Combination (NPC) 4 / 55

15 Null hypothesis Univariate: H : C ψ = Multivariate: H : C ψd = Winkler Non-Parametric Combination (NPC) 5 / 55

16 Multivariate contrast (D) - Often omitted (i.e., it s the same as I). - Weights differently each of the modalities, and can be used for complex and interesting questions. - Needs a pairing contrast of regressors (C). Example: D = [ ] Winkler Non-Parametric Combination (NPC) 6 / 55

17 Univariate test statistic In the univariate, it s all scalar, and we can use the F -statistic: F = ˆψ / C (C (M M) C) C ˆψ ˆɛ ˆɛ rank (C) N rank (M) = ˆβ / (X X) ˆβ ˆɛ ˆɛ rank (C) N rank (X) rank (Z) When rank(c) =, ˆβ is a scalar and the t statistic can be expressed as a function of F as t = sign( ˆβ) F. Winkler Non-Parametric Combination (NPC) 7 / 55

18 Univariate test statistic In the univariate, it s all scalar, and we can use the F -statistic: F = = 5.3 Winkler Non-Parametric Combination (NPC) 8 / 55

19 Multivariate test statistic But what in the multivariate case? = not possible Winkler Non-Parametric Combination (NPC) 9 / 55

20 Multivariate test statistic H = ˆψ C ( C (M M) C ) C ˆψ E = ˆɛ ˆɛ A = HE B = H (E + H) Winkler Non-Parametric Combination (NPC) 2 / 55

21 Multivariate test statistic Let λ i be the eigenvalues of A, and θ i the eigenvalues of B. Then: - Wilks Λ = i = E + λ i E + H. - Lawley Hotelling s trace: i λ i = trace (A). - Pillai s trace: i λ i + λ i = i θ i = trace (B). - Roy s largest root (ii): λ = max i (λ i ) = θ θ (analogous to F ). - Roy s largest root (iii): θ = max i (θ i ) = λ + λ (analogous to R 2 ). Winkler Non-Parametric Combination (NPC) 2 / 55

22 Multivariate tests: Hotelling s T 2 T 2 = C ˆψΣ ( 2 C (M M) C ) Σ 2 ˆψ C where Σ = ˆɛ ˆɛ/ (N rank (M)). When the p columns of Y are independent, T 2 = K k= t2 k, i.e., it s the sum of the independent, squared, Student s t statistics. Winkler Non-Parametric Combination (NPC) 22 / 55

23 Multivariate test statistic - These statistics have limiting distributions when many assumptions are met. - But we can assess with permutations. Winkler Non-Parametric Combination (NPC) 23 / 55

24 Multivariate tests How to run: palm -i modality -i modality2 -mv wilks [...] palm -i modality -i modality2 -con C.mset D.mset -mv [...] Winkler Non-Parametric Combination (NPC) 24 / 55

25 Example: Longitudinal 2 groups, 3 types of measurement per subject, 5 timepoints. (Data from Claire, Ugwechi, Heidi.) Winkler Non-Parametric Combination (NPC) 25 / 55

26 Example: Longitudinal Solution using manova: - Design with 2 groups (as in a simple, non-paired t-test). - Same design for each of the 3 measurements. - Null: H : C ψd = Various hypotheses about longitudinal effects with different C and D. Winkler Non-Parametric Combination (NPC) 26 / 55

27 Example: Longitudinal (manova) Question : Are the changes along time the same for both groups, i.e. are the curves for treatment and controls groups parallel? C = [ ] D = Winkler Non-Parametric Combination (NPC) 27 / 55

28 Example: Longitudinal (manova) Question 2: Do the curves for treatment and control groups have the same average level, i.e., do the groups have the same overall average along time? C = [ ] D = [ ] Winkler Non-Parametric Combination (NPC) 28 / 55

29 Example: Longitudinal (manova) Question 3: Are there no global effects along time for the two treatments, i.e., is the total curve horizontal? C = [ ] D = Winkler Non-Parametric Combination (NPC) 29 / 55

30 Example: Longitudinal (manova) Question 4: Are the curves parallel and have the same average, i.e., are the curves identical for treatment and control groups? C = [ ] D = Winkler Non-Parametric Combination (NPC) 3 / 55

31 Example: Longitudinal (manova) Question 5: Are the curves parallel and the average horizontal, i.e., are both curves horizontal? [ ] C = D = Winkler Non-Parametric Combination (NPC) 3 / 55

32 Non-Parametric Combination (npc) Winkler Non-Parametric Combination (NPC) 32 / 55

33 Non-Parametric Combination (npc) We may conduct multiple tests regarding a certain hypothesis, and none of these may be significant on their own right. However, on the aggregate, they be significant. Three possibilities: - Reject the null if any is significant. - Reject the null if all are significant. - Reject the null if an aggregate measure is significant. Each individual test is called partial test, and used to test a joint null hypothesis. Winkler Non-Parametric Combination (NPC) 33 / 55

34 Union-Intersection Test (uit) Global null not rejected t t 2 Global null rejected Winkler Non-Parametric Combination (NPC) 34 / 55

35 Intersection-Union Test (iut), i.e., conjunction test Conjunction null not rejected t t 2 Conjunction null rejected Winkler Non-Parametric Combination (NPC) 35 / 55

36 Combination Method Statistic Tippett min k (p k ) Fisher 2 K k= ln (p k) Stouffer K K k= Φ ( p k ) Mudholkar George π 3(5K+4) K(5K+2) ( ) K k= ln pk p k If the tests were guaranteed to be independent, a p-value could be computed using parametric formulas. Otherwise, we can use permutations. Winkler Non-Parametric Combination (NPC) 36 / 55

37 Combination Tippett Fisher Stouffer Mudholkar George p 2 p 2 p 2 p 2 p p p p Combined p-value (P) Winkler Non-Parametric Combination (NPC) 37 / 55

38 Various combining functions Method Test statistic (T ) Significance (p-value, P ) Tippett mink (pk) ( T ) K Fisher 2 P K k= ln (pk) 2 (T ; =2K) Stou er P p K k= ( pk) T ; µ =, 2 = Wilkinson K P K I (pk 6 ) P K K ( ) K k Good k= k=t k k Q K P k= pw k K k k= wk k T /w Qk k i= (wk wi) Q K i=k+ (wk wi) Lancaster P K k= wkf k ( pk) G (T ) Winer P K k= t cdf (.q pk; k) PK k= 2 = k k 2 T ; µ =, Edgington P K P bt c k= pk j= ( )j K j (T j) K K! Mudholkar George q 3(5K+4) P K K(5K+2) k= ln pk p tcdf(t ; =5K +4) k Darlington Hayes P r k= Computed through Monte Carlo methods. Tables are available. r p(k) Q Zaykin et al. (tpm) K P k= pi(p k6 ) K K k k= k )K k I T> k k + I T 6 k T P k (k ln ln T ) j j= j! Dudbridge Koeleman (rtp) Q r k= p(k) K ( r A (T,t,K)dt Dudbridge Koeleman (dtp) Qr max k= p(k), Q K k= pi(p k6 ) k r+ P r k= (r +)R r+ K ( k )K A (T,,k)+I (r<k) K r A (T,t,K)dt Taylor Tibshirani (ts) P K K k= K+ p(k) k 2 K T ; µ =, Jiang et al. (tts) P K K k= I p(k) 6 K+ p(k) k Computed through Monte Carlo methods. T is the statistic for each method and P its asymptotic significance. All methods are shown as function of the p-values for the partial tests. For certain methods, however, the test statistic for the partial tests, if available, can be used directly. K is the number of tests being combined, pk, k = {, 2,...,K} are the partial p-values, wk are positive weights assigned to the respective pk, p (r) are the pk with rank r in ascending order (most significant first), is the significance level for the partial tests, I( ) is an indicator function that evaluates as if the condition is satisfied, otherwise, b c represents the floor function, 2 is the cumulative distribution function (cdf) for is the cdf of the normal a 2 distribution, with the degrees of freedom, tcdf is the cdf of the Student s t distribution with degrees of freedom, andt cdf its inverse, distribution with mean µ and variance 2,and P its inverse, and F and G are the cdf of arbitrary, yet well chosen distributions. For the two Dudbridge Koeleman b j= (b ln a ln T )j /j!. methods, A (T,a,b)=I T>a b a b + I T 6 a b T Winkler Non-Parametric Combination (NPC) 38 / 55

39 NPC algorithm Modified NPC algorithm Modality Modality 2 Modality 3 Modality K Modality Modality 2 Modality 3 Modality K Synchronised permutations Synchronised permutations Synchronised permutations... Synchronised permutations Synchronised permutations Synchronised permutations Synchronised permutations... Synchronised permutations Phase t* Test statistic for each P j t* 2 Test statistic for each P j t* 3 Test statistic for each P j... t* K Test statistic for each P j t* Test statistic for each P j t* 2 Test statistic for each P j t* 3 Test statistic for each P j... t* K Test statistic for each P j Empirical Empirical distribution distribution p* p-value for each P j p* 2 p-value for each P j Empirical distribution p* 3 p-value for each P j... Empirical distribution... p* K p-value for each P j Parametric... Parametric distribution distribution u* u-value for each P j Parametric distribution u* 2 u-value for each P j Parametric distribution u* 3 u-value for each P j... u* K u-value for each P j Single phase Combination of p-values (for each P j) Combination of u-values (for each P j) Transformation to z & spatial statistics (for each P j) Phase 2 Combined empirical distribution Combined empirical distribution Combined empirical distribution Combined p-value Combined p-value Combined p-value (spatial) Winkler Non-Parametric Combination (NPC) 39 / 55

40 Procedure (adapted for imaging) - For each permutation, compute the statistic separately for each modality. - Convert to a u-value (simply a parametric p-value with a different name to avoid confusion). - Combine. - Repeat many times, and at the end, compute the p-value. Winkler Non-Parametric Combination (NPC) 4 / 55

41 Benefits - No need for independence. - No need to model the non-independence. - Comes with all other benefits of permutation methods. - More powerful than manova. - Needs exchangeability, like any permutation test. Winkler Non-Parametric Combination (NPC) 4 / 55

42 Example: Pain study Not combined Foot Hand Face log (p FWER ) Winkler Non-Parametric Combination (NPC) 42 / 55

43 Example: Pain study Modified NPC M G Stouffer Fisher Tippett log (p FWER ) Winkler Non-Parametric Combination (NPC) 43 / 55

44 Example: Pain study IUT (Conjunction) CMV Hotelling's T log (p FWER ) (Data from Jon and Irene.) Winkler Non-Parametric Combination (NPC) 44 / 55

45 Power of different combining functions Power (%) for each method when only one modality has signal, as a function of the number of modalities combined. Power Tippett Fisher (permutation) Fisher (parametric) Stouffer (permutation) Stouffer (parametric) Edgington (permutation) Edgington (parametric) Combining function Friston et al. Nichols et al Number of modalities Winkler Non-Parametric Combination (NPC) 45 / 55

46 Power of different combining functions Power (%) for each Power Number of modalities with signal Tippett Fisher (permutation) Fisher (parametric) Stouffer (permutation) Stouffer (parametric) Edgington (permutation) Edgington (parametric) Friston et al. Nichols et al. Combining function method when modalities are combined, as a function of the number of modalities with signal. Winkler Non-Parametric Combination (NPC) 46 / 55

47 Error rates and power of different combining functions Error rate (%) Power (%) Gaussian (normal) errors EE & ISE ISE EE Simulation A Simulation B Simulation C Simulation D Number of tests (K) Number of tests (K) Number of tests with signal (K s) Number of tests (K) and number of tests with signal (K s) Legend: Tippett Fisher Stouffer Mudholkar George Hotelling's T 2 Winkler Non-Parametric Combination (NPC) 47 / 55

48 Combination: Concordant directions favoured ( ) K K T = max 2 ln (p k ), 2 ln ( p k ) k= k= Compute the combined statistic, one for each direction, then take the best of both results. Winkler Non-Parametric Combination (NPC) 48 / 55

49 Combination: Concordant directions favoured Tippett Fisher Stouffer Mudholkar George p 2 p 2 p 2 p 2 p p p p Combined p-value (P) Winkler Non-Parametric Combination (NPC) 49 / 55

50 Combination: Two-tailed tests (direction irrelevant) Tippett Fisher Stouffer Mudholkar George p 2 p 2 p 2 p 2 p p p p Combined p-value (P) Winkler Non-Parametric Combination (NPC) 5 / 55

51 How to run Simple npc: palm -i modality -i modality2 [...] -npc fisher Concordant directions favoured: palm -i modality -i modality2 [...] -npc fisher -concordant Two-tailed: palm -i modality -i modality2 [...] -npc fisher -twotail Winkler Non-Parametric Combination (NPC) 5 / 55

52 Multiple testing correction Another type of multiple testing problem: - Multiple hypotheses in the same model (e.g., multiple contrasts). - Multiple designs (e.g., different seeds). - Multiple modalities (e.g. multiple dti measures). - Multiple pipelines (e.g., different smoothing levels). - Multiple multivariate hypotheses (e.g. profile analyses). - Imaging and non-imaging data. Let s call this other multiple testing problem as mtp-ii to distinguish it from the spatial case, thus called mtp-i. Winkler Non-Parametric Combination (NPC) 52 / 55

53 Multiple testing correction Correction over contrasts means you will never have to do an F -test again to account for multiple testing. It is dangerous anyway if rank(c) > 2. Winkler Non-Parametric Combination (NPC) 53 / 55

54 How to run Correction for multiple modalities: palm -i modality -i modality2 [...] -corrmod Correction for multiple contrasts: palm -i modality [...] -corrcon Correction for multiple modalities and contrasts: palm -i modality -i modality2 [...] -corrmod -corrcon Use fdr: palm -i modality -i modality2 [...] -corrmod -corrcon -fdr Winkler Non-Parametric Combination (NPC) 54 / 55

55 That s all folks. Blog: brainder.org Winkler Non-Parametric Combination (NPC) 55 / 55