Advanced topics in RSA. Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit Cambridge, UK

Transcription

1 Advanced topics in RSA Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit Cambridge, UK

2 Advanced topics menu How does the noise ceiling work? Why is Kendall s tau a often needed to compare RDMs? How can we weight model features to explain brain RDMs? What s the difference between dissimilarity, distance, and metric?

3 How does the noise ceiling work?

4 Correlation distance Euclidean distance 2 (for normalised patterns) d d 2 = (1 r) r 2 = 1 2r + r r 2 = 2(1 r) 1 r 1 r cos α = r Nili et al. 2014

5 highest accuracy any model can achieve accuracy of human IT dissimilarity matrix prediction [group-average rank correlation] other subjects average as model SE (stimulus bootstrap) accuracy above chance p<0.001 (subjects and stimuli as fixed effects) convolutional fully connected SVM discriminants Khaligh-Razavi & Kriegeskorte (2014), Nili et al (RSA Toolbox) weighted combination of fully connected layers and SVM discriminants

6 distance 2 Estimating the noise ceiling subject 2 subject 1 true model group mean subject 3 The group-mean RDM minimises the sum of squared Euclidean distances to singlesubject RDMs. For rank RDMs, the groupmean RDM therefore minimises the sum of correlation distances (thus maximising the average correlation). The average correlation to the group-mean of rank RDMs, thus provides a precise and hard upper bound on the average Spearman correlation any model can achieve. distance 1 Nili et al upper bound For Kendall s tau a, an iterative procedure is needed to find the hard upper bound.

7 distance 2 distance 2 Estimating the noise ceiling subject 2 subject 1 true model group mean subject 3 subject 1 (held out) leave-one-out group mean true model distance 1 Nili et al upper bound distance 1 lower bound

8 Why is Kendall s tau a often needed to compare RDMs? Kendall s tau a is needed when testing categorical models that predict tied dissimilarities.

9 General correlation coefficient difference matrix Pearson Spearman Kendall Kendall 1944

10 True model can lose to simplified step model (according to most correlation coefficients) Nili et al. 2014

11 True model can lose to simplified step model (according to most correlation coefficients) Nili et al simulated data real data

12 True model can lose to simplified step model (according to most correlation coefficients) Nili et al simulated data real data

13 ranks values True model can lose to simplified step model (according to most correlation coefficients) true model noisy data step model data points data points

14 ranks values True model can lose to simplified step model (according to most correlation coefficients) true model noisy data step model data points Pearson, tau-a data points

15 Kendall s tau a chooses the true model over a simplified model (which predicts ties in hard cases) more frequently than Pearson r, Spearman r, tau b and tau c true model always preferred proportion of cases in which the true model was preferred true model never preferred noise level in the data

16 Conclusions Rank correlation can be useful for comparing RDMs when measurement errors might distort a simple linear relationship. When categorical models (predicting tied ranks) are used, Kendall s tau a is an appropriate rank correlation coefficient. Kendall s tau b & c, Spearman correlation, and even Pearson correlation all prefer models that predict ties for difficult comparisons to the true model.

17 How can we weight model features to explain brain RDMs?

18 highest accuracy any model can achieve accuracy of human IT dissimilarity matrix prediction [group-average of Kendall s a ] other subjects average as model SE (stimulus bootstrap) accuracy above chance p<0.001 (subjects and stimuli as fixed effects) convolutional fully connected SVM discriminants Khaligh-Razavi & Kriegeskorte (2014), Nili et al (RSA Toolbox) weighted combination of fully connected layers and SVM discriminants

19 Khaligh-Razavi & Kriegeskorte (2014)

20 Khaligh-Razavi & Kriegeskorte (2014) Representational feature weighting with non-negative least-squares f 1 f 2... f k model RDM w 1 f 1 w 2 f 2... w k f k weighted-model RDM

21 predicted distance Representational feature weighting with non-negative least-squares n d i,j 2 = [wk f k i w k f k j ] 2 k=1 n = w k 2 [f k i f k j ] 2 k=1 weight k stimuli i, j model feature k The squared distance RDM of weighted model features equals a weighted sum of single-feature RDMs. w 1 2 feature-1 RDM +w 2 2 feature-2 RDM... +w k 2 feature-k RDM = weighted-model RDM w = arg min w R +n d i,j 2 d i,j 2 2 i j = arg min d 2 w 2 w R +n k RDM k i j n k=1 i,j 2 Khaligh-Razavi & Kriegeskorte (2014) w k weight given to model feature k f k (i) model feature k for stimulus i d i,j distance between stimuli i,j w is the weight vector [w 1 w 2... w k ] minimising the squared errors

22 What s the difference between dissimilarity, distance, and metric?

23 Dissimilarity measures d(x,y) = d(y,x) LD-t Distances correlation distance squared Euclidean distance Minkowski distance with p < 1 d(x,y) 0 Metrics d(x,y) = 0 x = y d(x,z) d(x,y) + d(y,z) Euclidean distance Mahalanobis distance Minkowski distance with p 1