Principles of Neural Information Theory

Transcription

1

2 Principles of Neural Information Theory Computational Neuroscience and Metabolic E ciency James V Stone

3 Title: Principles of Neural Information Theory Author: James V Stone c 2017 Sebtel Press All rights reserved. No part of this book may be reproduced or transmitted in any form without written permission from the author. The author asserts his moral right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act First Edition, Typeset in L A TEX2". First printing. File: main NeuralInfoTheory v39.tex. ISBN Cover based on photograph of Purkinje cell from mouse cerebellum injected with Lucifer Yellow. Courtesy of National Center for Microscopy and Imaging Research, University of California.

4 For Teleri

5 Contents 1. All That We See Introduction In the Light of Evolution In Search of General Principles Information Theory and Biology An Overview of Chapters Information Theory Introduction Finding a Route, Bit by Bit Information and Entropy Entropy and Uncertainty Maximum Entropy Distributions Channel Capacity Mutual Information The Gaussian Channel E ciency Summary Measuring Neural Information Introduction The Neuron Why Spikes? Neural Information Gaussian Firing Rates Information About What? Does Timing Precision Matter? Rate Codes and Timing Codes Summary Pricing Neural Information Introduction The E ciency-rate Trade O Paying With Spikes Paying With Hardware Paying With Power Optimal Axon Diameters Optimal Distribution of Axon Diameters Axon Diameter and Spike Speed Optimal Mean Firing Rate Optimal Distribution of Firing Rates Optimal Synaptic Conductance Summary

6 5. Encoding Colour Introduction The Eye How Aftere ects Occur The Problem With Colour A Neural Encoding Strategy Encoding Colour Why Aftere ects Occur Measuring Mutual Information Maximising Mutual Information Principal Component Analysis PCA and Mutual Information Evidence for E ciency Summary Encoding Time Introduction Linear Models Neurons and Wine Glasses The LNP Model Estimating LNP Parameters The Predictive Coding Model Estimating Predictive Coding Parameters Predictive Coding and Information Evidence for Predictive Coding Summary Encoding Space Introduction Spatial Frequency Do Ganglion Cells Decorrelate Images? Are Receptive Field Structures Optimal? Predictive Coding of Images Evidence For Predictive Coding Is Receptive Field Spacing Optimal? Summary Encoding Visual Contrast Introduction The Compound Eye Not Wasting Capacity Measuring the Eye s Response Maximum Entropy Encoding E ciency of Maximum Entropy Encoding Summary

7 9. Reading the Neural Code Introduction Bayes Rule Bayesian Decoding Encoding vs Decoding Nonlinear Encoding Linear Decoding Linear Decodability How Bayes Adds Bits Summary The Neural Rubicon Introduction The Darwinian Cost of Metabolic E ciency Crossing the Neural Rubicon Further Reading 179 Appendices A. Glossary 181 B. Mathematical Symbols 185 C. Correlation and Independence 189 D. A Vector Matrix Tutorial 191 E. Neural Information Methods 195 F. The Marginal Value Theorem 201 G. Key Equations 203 References 205 Index 211

8 Preface Some scientists consider the brain to be a collection of heuristics or hacks, which have accumulated over evolutionary history. Others think that the brain relies on a small number of fundamental principles, which underpin the diverse systems within the brain. This book provides a rigorous account of how Shannon s mathematical theory of information can be used to test one such principle, metabolic e ciency. From Words to Mathematics. The methods used to explore metabolic e ciency lie in the realms of mathematical modelling. Mathematical models demand a precision unattainable with purely verbal accounts of brain function. With this precision, comes an equally precise quantitative predictive power. In contrast, the predictions of purely verbal models can be vague, and this vagueness also makes them virtually indestructible, because predictive failures can often be explained away. No such luxury exists for mathematical models. In this respect, mathematical models are easy to test, and if they are weak models then they are easy to disprove. So, in the Darwinian world of mathematical modelling, survivors tend to be few, but those few tend to be supremely fit. This is not to suggest that purely verbal models are always inferior. Such models are a necessary first step in understanding. But continually refining a verbal model into ever more rarefied forms is not scientific progress. Eventually, a purely verbal model should evolve to the point where it can be reformulated as a mathematical model, with predictions that can be tested against measurable physical quantities. Happily, most branches of neuroscience reached this state of scientific maturity some time ago. Signposts. The writing style adopted here in Principles of Neural Information Theory describes the raw science of neural information theory, un-fettered by the conventions of standard textbooks. Accordingly, key concepts are introduced informally, before being described mathematically.

9 However, such an informal style can easily be mis-interpreted as poor scholarship, because informal writing is often sloppy writing. But the way we write is usually only loosely related to the way we speak, when giving a lecture, for example. A good lecture includes many asides and hints about to what is, and is not, important. In contrast, scientific writing is usually formal, bereft of sign-posts about where the main theoretical structures are to be found, and how to avoid the many pitfalls which can mislead the unwary. So, unlike most textbooks, and like the best lectures, this book is intended to be both informal and rigorous, with prominent sign-posts as to where the main insights are to be found, and many warnings about where they are not. Originality. Evidence is presented in the form of scientific papers and books, which are cited using a conventional format (e.g. Smith and Jones (1959)). When the sheer number of cited sources is large, numbered superscripts are used in order to prevent the text from becoming un-readable. Occasionally, facts are presented without evidence, either because they are reasonably self-evident, or because they can be found in standard texts. Consequently, the reader may wonder if the ideas being presented are derived from other scientists, or if they are unique to this book. In such cases, be re-assured that almost none of the ideas in this book belong to the author. Indeed, like most books, this book represents a synthesis of ideas from many sources, but the general approach is inspired principally by these texts: Vision (1981) 79 by Marr, Spikes (1997) 97 by Rieke, Warland, de Ruyter van Steveninck and Bialek, Biophysics (2012) 19 by Bialek, and Principles of Neural Design (2015) 112 by Sterling and Laughlin. In particular, Sterling and Laughlin pointed out that the amount of physiological data being published each year contributes to a growing Data Mountain, which far outstrips the ability of current theories to make sense of those data. Accordingly, whilst this account is not intended to be definitive, it is intended to provide another piton to those established by Sterling and Laughlin on Data Mountain. Who Should Read This Book? The material presented in this book is intended to be accessible to readers with a basic scientific education at undergraduate level. In essence, this book is intended

10 for those who wish to understand how the fundamental ingredients of inert matter, energy and information have been forged by evolution to produce a particularly e cient computational machine, the brain. Understanding how the elements of this triumvirate interact demands knowledge from a wide variety of academic disciplines, but principally from biology and mathematics. Accordingly, reading this book will require some patience from mathematicians to navigate the conceptual shift involved in interpreting signal processing mathematics in terms of neural computation, and it will require sustained e ort from biologists to grasp the mathematical material. Even though some of the mathematics is fairly sophisticated, key concepts are introduced using geometric interpretations and diagrams wherever possible. PowerPoint Slides of Figures. Most of the figures used in this book can be downloaded from Corrections. Please corrections to j.v.stone@she eld.ac.uk. A list of corrections can be found at Acknowledgments. Thanks to the developers of the L A TEX2" software, used to typeset this book. Shashank Vatedka deserves a special mention for checking the mathematics in a final draft of this book. Thanks to SOMEONE for meticulous copy-editing and proofreading. (NOT DONE YET). Thanks to John de Pledge, Royston Sellman, and Steve Snow for many discussions on the role of information in biology, to Patrick Keating for advice on the optics of photoreceptors, to Frederic Theunissen for advice on measuring neural information, and to Mike Land for help with disentangling neural superposition. For reading either individual or all chapters, I am very grateful to David Atwell, Horace Barlow, Ilona Box, Julian Budd, Matthew Crosby, Nikki Hunkin, Simon Laughlin, Raymond Lister, Danielle Matthews, Pasha Parpia, Jenny Read, Jung-Tsung Shen, Tom Sta ord, Eleni Vasilaki, Paul Warren and Stuart Wilson. James V Stone, She eld, England, 2017.

11 To understand life, one has to understand not just the flow of energy, but also the flow of information. W Bialek, 2012.

12 Chapter 1 All That We See When we see, we are not interpreting the pattern of light intensity that falls on our retina; we are interpreting the pattern of spikes that the million cells of our optic nerve send to the brain. Rieke, Warland, De Ruyter van Steveninck, and Bialek, Introduction All that we see begins with an image formed on the eye s retina (Figure 1.1). Initially, this image is recorded by 126 million photoreceptors within the retina. The outputs of these photoreceptors are then repackaged or encoded, via a series of intermediate connections, into a sequence of digital pulses or spikes, that travel through the one million neurons of the optic nerve which connect the eye to the brain. The fact that we see so well suggests that the retina must be extraordinarily accurate when it encodes the image into spikes, and the brain must be equally accurate when it decodes those spikes into all that we see (Figure 1.2). But the eye and brain are not only good at translating the world into spikes, and spikes into perception, they are also e cient at transmitting information from the eye to the brain. Precisely how e cient, is the subject of this book. 1

13 1 All That We See 1.2. In the Light of Evolution In 1973, the evolutionary biologist Dobzhansky famously wrote: Nothing in biology makes sense except in the light of evolution. But evolution has to operate within limits set by the laws of physics, and (as we shall see) the laws of information. Just as a bird cannot fly without obeying the laws of physics, so, a brain cannot function without obeying the laws of information. And, just as the shape of a bird s wing is determined by the laws of physics, so the structure of a neuron is determined by the laws of information. Neurons communicate information, and that is pretty much all that they do. But neurons are energetically expensive to make, maintain, and run 71. Half of a child s total energy budget, and a fifth of an adult s budget, is required just to keep the brain ticking over 108 (Figure 1.3). For children and adults, half the total energy budget of the brain is used for neuronal information processing, and the rest is used for basic maintenance 10. The high cost of using neurons probably accounts for the fact that only 2-4% of them are active at any one time 73. Given that neurons and spikes are so expensive, we should be un-surprised to find that when the visual data from the eye is encoded as a series of spikes, each neuron and each spike conveys information e ciently. In the context of the Darwin-Wallace theory of evolution, it seems self-evident that neurons should be e cient. But, in order to formalise the notion of e ciency, we need a rigorous definition of Darwinian fitness. Even though fitness can be measured in terms of the number of o spring an animal manages to raise to sexual maturity, the connection Re#na Op#c Nerve Lens Figure 1.1. Cross section of eye. Modified from Wikimedia Commons. 2

14 1.2. In the Light of Evolution between neural computation and fitness is di cult to define. In the absence of such a definition, we consider quantities which can act as a plausible proxy for fitness. One such proxy, with a long track-record in neuroscience, is information, which is measured in units of bits. The amount of information an animal can gather from its environment is related to fitness because information in the form of sight, sound and scent ultimately provides food, mates and shelter. However, information comes at a price, paid in neuronal infrastructure and energy. So animals want information, but they want information at a price that will increase their fitness. This means that animals usually want cheap information. It is often said that there is no such thing as a free lunch, which is as true in Nature as it is in New York. If an animal demands that a neuron delivers information at a high rate then the laws of information dictate that the price per bit will be high; a price that is paid in Joules. However, sometimes information is worth having even if it is expensive. For example, visually responsive neurons which alert an animal to the presence of imminent danger (e.g. by identifying a predator) require large amounts of information to be transmitted rapidly, even if the energy cost of transmitting that information through neurons in the visual system is high. Such neurons take full advantage of their potential for transmitting information, so they maximise their coding e ciency. Conversely, if an animal demands that information a Response (spikes) Encoding b Luminance Decoding Reconstructed luminance Time (ms) Figure 1.2. Encoding and decoding (schematic). A rapidly changing luminance (bold curve in b) is encoded as a spike train (a), which is decoded to estimate the luminance (thin curve in b). See Chapters 6 and 9. 3

15 1 All That We See is delivered at the lowest price per bit, with maximal metabolic efficiency 74;112, then those same laws dictate that the information rate must be low. The idea of coding efficiency has a longer history than metabolic efficiency, and has been enshrined as the efficient coding hypothesis, which posits that the way neurons encode sensory data is designed by Nature to transmit as much information as possible. It has been championed over many years by Horace Barlow (1959) 16, amongst others (e.g. Attneave (1954) 9, Atick (1992) 5 ), and has had a substantial influence on computational neuroscience. However, accumulating evidence, summarised in this text, suggests that metabolic efficiency, rather than coding efficiency, may be the dominant influence on Nature s design of neural systems. Both coding efficiency and metabolic efficiency are defined more formally in Section 2.9. We should note that there are a number of di erent computational models which collectively fall under the umbrella term efficient coding. To a first approximation, the results of applying the methods associated with these models tend to be similar 97, even though the methods are di erent. These methods include sparse coding 47, principal component analysis, independent component analysis 18;113, information maximisation (infomax) 75, redundancy reduction 7, and predictive coding 94;110. (a) (b) Figure 1.3. a) The human brain weighs 1300g, contains about 1010 neurons, and consumes 12 Watts of power. The outer surface seen here is the neocortex. b) Each neuron (plus its support structures) therefore accounts for an average of 1200pJ/s (1pJ/s = J/s). From Wikimedia Commons. 4

16 1.3. In Search of General Principles 1.3. In Search of General Principles The test of a theory is not just whether or not it accounts for a body of data, but also how complex the theory is in relation to the complexity of the data being explained. Clearly, if a theory is, in some sense, more convoluted than the phenomena it explains then it is not much of a theory. This is why we favour theories that explain a vast range of phenomena with the minimum of words or equations. A prime example of a parsimonious theory is Newton s theory of gravitation, which explains (amongst other things) how a ball falls to Earth, how atmospheric pressure varies with height above the Earth, and how the Earth orbits the Sun. In essence, we favour theories which rely on a general principle to explain a diverse range of physical phenomena. However, even a theory based on general principles is of little use if it is too vague to be tested rigorously. Accordingly, if we want to understand how the brain works then we need more than a theory which is expressed in mere words. For example, if the theory of gravitation were stated only in words then we could say that each planet has an approximately circular orbit, but we would have to use many words to prove precisely why each orbit must be elliptical, and to state exactly how elliptical each orbit is. In contrast, a few equations express these facts exactly, and without ambiguity. Thus, whereas words are required to provide theoretical context, mathematics imposes a degree of precision which is extremely di cult, if not impossible, to achieve with words alone. To quote Galileo Galilei (1623) The universe is written in this grand book, which stands continually open to our gaze, but it cannot be understood unless one first learns to comprehend the language in which it is written. It is written in the language of mathematics, without which it is humanly impossible to understand a single word of it. In the spirit of Galileo s recommendation, we begin with a quantitative definition of information, in Chapter 2. 5

17 1 All That We See 1.4. Information Theory and Biology Claude Shannon s theory of communication 103 (1948), heralded a transformation in our understanding of information. Before 1948, information was regarded as a kind of miasmic fluid. But afterwards, it became apparent that information is a well-defined and, above all, measurable quantity. Since that time, it has become increasingly apparent that information, and the energy cost of information, imposes fundamental limits on the form and function of biological mechanisms. Shannon s theory provides a mathematical definition of information, and describes precisely how much information can be communicated between di erent elements of a system. This may not sound like much, but Shannon s theory underpins our understanding of why there are definite limits to the rate at which information can be processed within any system, whether man-made or biological. Information theory does not place any conditions on what type of mechanism processes information in order to achive a given objective; in other words, on exactly how it is to be achieved. However, unless there are unlimited amounts of energy available, relatively little information will reach the brain without some form of encoding. In other words, information theory does not specify how any biological function, such as vision, is implemented, but it does set fundamental limits on what is achievable by any physical mechanisms within any visual system. The distinction between a function and the mechanism which implements that function is a cornerstone of David Marr s (1982) 79 computational theory of vision. Marr stressed the need to consider physiological findings in the context of computational models, and his approach is epitomised in a single quote: Trying to understand perception by studying only neurons is like trying to understand bird flight by studying only feathers: It just cannot be done. Even though Marr did not address the role of information theory directly, his analytic approach has served as a source of inspiration, not only for this book, but also for much of the progress made within computational neuroscience. 6

18 1.5. An Overview of Chapters 1.5. An Overview of Chapters This section contains technical terms which are explained fully in the relevant chapters, and in the Glossary (Appendix A, p181). To fully appreciate the importance of information theory for neural computation, some familiarity with the basic elements of information theory is required; these elements are presented in Chapter 2 (which can be skipped on a first reading of the book). In Chapter 3, we use information theory to estimate the amount of information in the output of a spiking neuron, and also to estimate how much of this information (i.e. mutual information) is related to the neuron s input. In Chapter 4, we discover that one of the consequences of information theory (specifically, Shannon s noisy coding theorem) is that the cost of information rises inexorably and disproportionately with information rate. We consider empirical results which suggest that this steep rise accounts for physiological values of axon diameter, the distribution of axon diameters, mean firing rate, and synaptic conductance; values which appear to be tuned to minimise the cost of information. In Chapter 5, we consider how the correlations between the outputs of photoreceptors sensitive to similar colours threaten to reduce information rates, and how this can be ameliorated by synaptic preprocessing in the retina. This pre-processing makes e cient use of the available neuronal inftrastructure to maximise information rates, which explains not only how, but also why, there is a red-green aftere ect, but no red-blue aftere ect. A more formal account involves using principal component analysis to estimate the synaptic connections which maximise neuronal information throughput. In Chapter 6, the lessons learned so far are applied to the problem of encoding time-varying visual inputs. We explore how a standard (LNP) neuron model can be used as a model of physiological neurons. We then introduce a model based on predictive coding, which yields similar results to the LNP model, and we consider how predictive coding represents a biologically plausible candidate for maximising information rates, and minimising the cost of information. In Chapter 7, we consider how information theory predicts the receptive field structures of retinal ganglion cells across a range of 7

19 1 All That We See luminance conditions. Evidence is presented that these receptive field structures are also obtained using predictive coding, and the information-theoretic analysis applied to time-varying visual inputs (in Chapter 6) is extended to the spatial domain (i.e. retinal images). Once colour, temporal and spatial structure has been encoded by a neuron, the resultant signals must pass through the neuron s non-linear input/output (transfer) function. Accordingly, Chapter 8 is based on a classic paper by Simon Laughlin (1981) 68,whichpredictstheprecise form that this transfer function should adopt in order to maximise information throughput; a form which matches the transfer function found in visual neurons. Having considered how neurons encode sensory inputs, the problem of how the brain decodes neuronal outputs is addressed in Chapter 9; where the importance of prior knowledge, or experience, is explored in the context of Bayes theorem. We also consider how often a neuron should produce a spike so that each spike conveys as much information as possible, and we discover that the answer involves a vital property of e cient communication (i.e. linear decodability). A fundamental tenet of the computational approach adopted in this text is that, within each chapter, we explore particular neuronal mechanisms, how they work, and (most importantly) why they work in the way they do. As a result, each chapter documents evidence that the design of biological mechanisms is determined largely by the need to process information e ciently. 8

20 Dr James Stone is an Honorary Reader in Vision and Computational Neuroscience at the University of Sheffield, England. Seeing The Computational Approach to Biological Vision second edition John P. Frisby and James V. Stone