Information Theory. Mark van Rossum. January 24, School of Informatics, University of Edinburgh 1 / 35

Transcription

1 1 / 35 Information Theory Mark van Rossum School of Informatics, University of Edinburgh January 24, Version: January 24, 2018

2 Why information theory 2 / 35 Understanding the neural code. Encoding and decoding. We imposed coding schemes, such as 2nd-order kernel, or NLP. We possibly lost information in doing so. Instead, use information: Don t need to impose encoding or decoding scheme (non-parametric). In particular important for 1) spike timing codes, 2) higher areas. Estimate how much information is coded in certain signal. Caveats: No easy decoding scheme for organism (upper bound only) Requires more data and biases are tricky

3 3 / 35 Overview Entropy, Mutual Information Entropy Maximization for a Single Neuron Maximizing Mutual Information Estimating information Reading: Dayan and Abbott ch 4, Rieke

4 4 / 35 Definition The entropy of a quantity is defined as H(X) = x P(x) log 2 P(x) This is not derived, nor fully unique, but it fulfills these criteria: Continuous If p i = 1 n, it increases monotonically with n. H = log 2 n. Parallel independent channels add. Unit : bits Entropy can be thought of as physical entropy, richness of distribution [Shannon and Weaver, 1949, Cover and Thomas, 1991, Rieke et al., 1996]

5 Entropy 5 / 35 Discrete variable H(R) = r p(r) log 2 p(r) Continuous variable at resolution r H(R) = r p(r) r log 2 (p(r) r) = r p(r) r log 2 p(r) log 2 r letting r 0 we have lim [H + log 2 r] = r 0 p(r) log 2 p(r)dr (also called differential entropy)

6 Joint, Conditional entropy 6 / 35 Joint entropy: H(S, R) = r,s P(S, R) log 2 P(S, R) Conditional entropy: H(S R) = r P(R = r)h(s R = r) = r P(r) s P(s r) log 2 P(s r) If S, R are independent = H(S, R) H(R) H(S, R) = H(S) + H(R)

7 7 / 35 Mutual information Mutual information: I m (R; S) = r,s p(r, s) log 2 p(r, s) p(r)p(s) = H(R) H(R S) = H(S) H(S R) Measures reduction in uncertainty of R by knowing S (or vice versa) I m (R; S) 0 The continuous version is the difference of two entropies, the r divergence cancels

8 Mutual Information 8 / 35 The joint histogram determines mutual information. Given P(r, s) I m.

9 Mutual Information: Examples 9 / 35 Y non smoker smoker 1 smoker Y 1 non smoker Y 2 lung cancer 1/3 0 Y 2 lung cancer 1/9 2/9 no lung cancer 0 2/3 no lung cancer 2/9 4/9

10 Mutual Information: Examples 10 / 35 Y non smoker smoker 1 smoker Y 1 non smoker Y 2 lung cancer 1/3 0 Y 2 lung cancer 1/9 2/9 no lung cancer 0 2/3 no lung cancer 2/9 4/9 Only for the left joint probability I m > 0 (how much?). On the right, knowledge about Y 1 does not inform us about Y 2.

11 11 / 35 Kullback-Leibler divergence KL-divergence measures distance between two probability distributions D KL (P Q) = P(x) log 2 P(x) Q(x) dx, or D KL(P Q) i P i log 2 P i Q i Not symmetric, but can be symmetrized I m (R; S) = D KL (p(r, s) p(r)p(s)). Often used as probabilistic cost function: D KL (data model). Other probability distances exist (e.g. earth-movers distance)

12 Mutual info between jointly Gaussian variables 12 / 35 I(Y 1 ; Y 2 ) = P(y 1, y 2 ) log 2 P(y 1, y 2 ) P(y 1 )P(y 2 ) dy 1 dy 2 = 1 2 log 2(1 ρ 2 ) ρ is (Pearson-r) correlation coefficient.

13 Populations of Neurons 13 / 35 Given H(R) = p(r) log 2 p(r)dr N log 2 r and H(R i ) = p(r i ) log 2 p(r i )dr log 2 r We have H(R) i H(R i ) (proof, consider KL divergence)

14 Mutual information in populations of Neurons 14 / 35 Reduncancy can be defined as (compare to above) n r R = I(r i ; s) I(r; s). i=1 Some codes have R > 0 (redundant code), others R < 0 (synergistic) Example of synergistic code: P(r 1, r 2, s) with P(0, 0, 1) = P(0, 1, 0) = P(1, 0, 0) = P(1, 1, 1) = 1 4

15 15 / 35 Entropy Maximization for a Single Neuron I m (R; S) = H(R) H(R S) If noise entropy H(R S) is independent of the transformation S R, we can maximize mutual information by maximizing H(R) under given constraints Possible constraint: response r is 0 < r < r max. Maximal H(R) if p(r) U(0, r max ) (U is uniform dist) If average firing rate is limited, and 0 < r < : exponential distribution is optimal p(x) = 1/ xexp( x/ x). H = log 2 e x If variance is fixed and < r < : Gaussian distribution. H = 1 2 log 2(2πeσ 2 ) (note funny units)

16 Let r = f (s) and s p(s). Which f (assumed monotonic) maximizes H(R) using max firing rate constraint? Require: P(r) = 1 r max Thus df /ds = r max p(s) and p(s) = p(r) dr ds = 1 r max f (s) = r max s df ds s min p(s )ds This strategy is known as histogram equalization in signal processing 16 / 35

17 Fly retina Evidence that the large monopolar cell in the fly visual system carries out histogram equalization Contrast response for fly large monopolar cell (points) matches environment statistics (line) [Laughlin, 1981] (but changes in high noise conditions) 17 / 35

18 V1 contrast responses 18 / 35 Similar in V1, but On and Off channels [Brady and Field, 2000]

19 Information of time varying signals 19 / 35 Single analog channel with Gaussian signal s and Gaussian noise η: r = s + η I = 1 2 log 2(1 + σ2 s ση 2 ) = 1 2 log 2(1 + SNR) For time dependent signals I = 1 2 T dω 2π log 2(1 + s(ω) n(ω) ) To maximize information, when variance of the signal is constrained, use all frequency bands such that signal+noise = constant. Whitening. Water filling analog:

20 Information of graded synapses 20 / 35 Light - (photon noise) - photoreceptor - (synaptic noise) - LMC At low light levels photon noise dominates, synaptic noise is negligible. Information rate: 1500 bits/s [de Ruyter van Steveninck and Laughlin, 1996].

21 Spiking neurons: maximal information 21 / 35 Spike train with N = T /δt bins [Mackay and McCullogh, 1952] δt time-resolution. N! pn = N 1 events, #words = N 1!(N N 1 )! Maximal entropy if all words are equally likely. H = p i log 2 p i = log 2 N! log 2 N 1! log 2 (N N 1 )! Use for large x that log x! x(log x 1) H = T δt [p log 2 p + (1 p) log 2 (1 p)] For low rates p 1, setting λ = (δt)p: H = T λ log 2 ( e λδt )

22 Spiking neurons 22 / 35 Calculation incorrect when multiple spikes per bin. Instead, for large bins maximal information for exponential distribution: P(n) = 1 1 Z exp[ n log(1 + n )] H = log 2 (1 + n ) + n log 2 (1 + 1 n ) log 2(1 + n ) + 1

23 23 / 35 Spiking neurons: rate code [Stein, 1967] Measure rate in window T, during which stimulus is constant. Periodic neuron can maximally encode [1 + (f max f min )T ] stimuli H log 2 [1 + (f max f min )T ]. Note, only log(t )

24 [Stein, 1967] Similar behaviour for Poisson : H log(t ) 24 / 35

25 Spiking neurons: dynamic stimuli 25 / 35 [de Ruyter van Steveninck et al., 1997], but see [Warzecha and Egelhaaf, 1999].

26 26 / 35 Maximizing Information Transmission: single output Single linear neuron with post-synaptic noise v = w u + η where η is an independent noise variable I m (u; v) = H(v) H(v u) Second term depends only on p(η) To maximize I m need to maximize H(v); sensible constraint is that w 2 = 1 If u N(0, Q) and η N(0, σ 2 η) then v N(0, w T Qw + σ 2 η)

27 27 / 35 For a Gaussian RV with variance σ 2 we have H = 1 2 log 2πeσ2. To maximize H(v) we need to maximize w T Qw subject to the constraint w 2 = 1 Thus w e 1 so we obtain PCA If v is non-gaussian then this calculation gives an upper bound on H(v) (as the Gaussian distribution is the maximum entropy distribution for a given mean and covariance)

28 Infomax 28 / 35 Infomax: maximize information in multiple outputs wrt weights [Linsker, 1988] v = W u + η H(v) = 1 2 log det( vvt ) Example: 2 inputs and 2 outputs. Input is correlated. w 2 k1 + w 2 k2 = 1. At low noise independent coding, at high noise joint coding.

29 Estimating information 29 / 35 Information estimation requires a lot of data. Most statistical quantities are unbiased (mean, var,...). But both entropy and noise entropy have bias. [Panzeri et al., 2007]

30 Try to fit 1/N correction [Strong et al., 1998] 30 / 35

31 Common technique for I m : shuffle correction [Panzeri et al., 2007] See also: [Paninski, 2003, Nemenman et al., 2002] 31 / 35

32 32 / 35 Summary Information theory provides non parametric framework for coding Optimal coding schemes depend strongly on noise assumptions and optimization constraints In data analysis biases can be substantial

33 References I 33 / 35 Brady, N. and Field, D. J. (2000). Local contrast in natural images: normalisation and coding efficiency. Perception, 29(9): Cover, T. M. and Thomas, J. A. (1991). Elements of information theory. Wiley, New York. de Ruyter van Steveninck, R. R. and Laughlin, S. B. (1996). The rate of information transfer at graded-potential synapses. Nature, 379: de Ruyter van Steveninck, R. R., Lewen, G. D., Strong, S. P., Koberle, R., and Bialek, W. (1997). Reproducibility and variability in neural spike trains. Science, 275: Laughlin, S. B. (1981). A simple coding procedure enhances a neuron s information capacity. Zeitschrift für Naturforschung, 36: Linsker, R. (1988). Self-organization in a perceptual network. Computer, 21(3):

34 References II 34 / 35 Mackay, D. and McCullogh, W. S. (1952). The limiting information capacity of neuronal link. Bull Math Biophys, 14: Nemenman, I., Shafee, F., and Bialek, W. (2002). Entropy and Inference, Revisited. nips, 14. Paninski, L. (2003). Estimation of Entropy and Mutual Information. Neural Comp., 15: Panzeri, S., Senatore, R., Montemurro, M. A., and Petersen, R. S. (2007). Correcting for the sampling bias problem in spike train information measures. J Neurophysiol, 98(3): Rieke, F., Warland, D., de Ruyter van Steveninck, R., and Bialek, W. (1996). Spikes: Exploring the neural code. MIT Press, Cambridge. Shannon, C. E. and Weaver, W. (1949). The mathematical theory of communication. Univeristy of Illinois Press, Illinois.

35 References III 35 / 35 Stein, R. B. (1967). The information capacity of nerve cells using a frequency code. Biophys J, 7: Strong, S. P., Koberle, R., de Ruyter van Steveninck, R. R., and Bialek, W. (1998). Entropy and Information in Neural Spike Trains. Phys Rev Lett, 80: Warzecha, A. K. and Egelhaaf, M. (1999). Variability in spike trains during constant and dynamic stimulation. Science, 283(5409):