Analytical estimates of limited sampling biases in different information measures

Transcription

1 Network: Computation in Neural Sytem 7 (996) Printed in the UK Analytical etimate of limited ampling biae in different information meaure Stefano Panzeri and Aleandro Treve Biophyic, SISSA, via Beirut 2 4, 3403 Triete, Italy Mathematical Phyic, SISSA, via Beirut 2 4, 3403 Triete, Italy Received 27 March 995, in final form 27 October 995 Abtract. Meauring the information carried by neuronal activity i made difficult, particularly when recording from mammalian cell, by the limited amount of data uually available, which reult in a ytematic error. While empirical ad hoc procedure have been ued to correct for uch error, we have recently propoed a direct procedure coniting of the analytical calculation of the average error, it etimation (up to ubleading term) from the data, and it ubtraction from raw information meaure to yield unbiaed meaure. We calculate here the leading correction term for both the average tranmitted information and the conditional information and, ince uually one mut firt regularize the data, we pecify the expreion appropriate to different regularization. Computer imulation indicate a broad range of validity of the analytical reult, ugget the effectivene of regularizing by imple binning and illutrate the advantage of thi over the previouly ued boottrap procedure.. Introduction The performance of network of neuron a information proceing device can only be correctly gauged by uing the appropriate information meaure a performance quantifier. Thi i a traightforward notion a far a the analyi of abtract model i concerned, but when it come to real neuron, whoe activity i recorded in vivo, extracting information meaure i o ridden with ubtletie, epecially in mammal, that in practice few meaure have been produced until now (e.g. Eckhorn and Pöpel 975, Optican and Richmond 987, McClurkin et al 99, Gawne and Richmond 993, Tovee et al 993). Apart from the pychological difficulty of accepting the validity of quantitie which are alway relative to the procedure ued to meaure them, the bigget ource of problem ha been the limited ize of data ample, which reult in meaure ditorted by a ytematic error, occaionally a large a the target quantity itelf. A a conequence, important quetion uch a the type of neuronal coding ued by different ytem in the mammalian brain, the peed of information proceing and it efficiency at the neuronal level, have been mot eaily approached qualitatively from a theoretical viewpoint, rather than quantitatively from experimental obervation. Empirical procedure to correct information meaure for limited ampling have been refined (Optican et al 99, Chee-Ort and Optican 993, Hertz et al 992), but they tefano@limbo.ia.it Although for ome ytem, a in the elegant analyi of early viion by Atick and collaborator (Atick and Redlich 990, Dong and Atick 995), it i poible to bypa the meaurement of information quantitie from the data X/96/ $9.50 c 996 IOP Publihing Ltd 87

2 88 S Panzeri and A Treve are not yet atifactory, for reaon that will be made clear in the following. We have propoed, intead, an approach baed on a direct evaluation and ubtraction, of the limited ampling bia (Treve and Panzeri 995). The idea, which had been conceived a early a 40 year ago (Miller 955), needed to be developed to be applicable to neuronal data, in particular to be adapted to the regularization procedure ued with neuronal data; thi development i the content of the preent report. Of crucial practical importance i the trade off between the information lo due to the regularization and the limited ampling error and our reult, which give unbiaed etimator of regularized quantitie and hift the balance toward chooing milder regularization. 2. Information meaure from limited ample To be concrete, we conider a ituation in which we wih to meaure the amount of information, in bit, that ome variable r, aociated with the repone of one or more neuron, convey about a timulu,, preented to the animal. We take to belong to the dicrete et S of S element. We wih to meaure both the (average) conditional information tranmitted when i preented, I() = drp(r )log 2 p( r) p() = drp(r )log 2 P(r ) P(r) and it average acro timuli, i.e. the mutual information I = P(r ) p() drp(r )log 2 P(r). (2) We aume that only N timulu repone pair (, r) are available, intead of the full probabilitie p(), P(r) and P(,r) (the lat two are, in general, probability denitie rather than probabilitie, and are thu denoted by capital letter). For N, individual (, r) pair are expected to occur with frequencie tending to match the underlying probabilitie, but for N finite, ue of the experimental frequencie p N (), P N (r) and P N (, r) directly in the formulae above lead to ytematic error. That the problem exit, can be een by conidering uncorrelated timuli and repone, uch that P(,r) = p()p(r): a finite-n evaluation of the mutual information, which i zero by definition, will almot certainly yield a poitive reult, which therefore indicate a ytematic error. The procedure uggeted by Optican et al (99) to correct for the error, and ucceively improved by Chee-Ort and Optican (993), follow from conidering the cae of uncorrelated pair: it involve generating a huffled probability ditribution by randomly pairing timuli and real repone, calculating the huffled information contained in the real repone about the randomly paired peudotimuli and finally ubtracting a fraction of the huffled information from the raw value of meaured information. Thi random huffling procedure, often called boottrap becaue it ue the data to correct the data themelve, i flawed in everal way. Firt, and mot evidently when repone are dicrete, the huffled information may in ome cae be a trong overetimation of the bia, for reaon to be clarified below and then it i wrong to ubtract from the raw etimate the correction derived from random huffling. Furthermore, the huffling procedure i applicable only to meaure of mutual information and not to meaure of conditional information, ince the random huffling mixe repone occurring to different timuli. Finally, when the repone are regularized before being ued to meaure information, the regularization can affect the raw and huffled information meaure to different degree, a will again be clear below, ometime reulting in an underetimation of the bia. ()

3 Information meaure from limited ample 89 More ophiticated i the procedure uggeted by Hertz et al (992), baed on a trong regularization of the input output ditribution by mean of a neural network ued to etimate the probability of each input given the output r. The neural network i trained o a to maximize the probability that a timulu i correctly recognized, i.e. that the timulu etimated to be mot probable i the actual one. Thi method appear to yield unbiaed etimate, in the ene that after the regularization one obtain huffled information very cloe to zero (Kjaer et al 994). The lat flaw in the huffling procedure alo applie here, although, in the ene that negligible huffled information doe not guarantee that the raw etimate i unbiaed. Moreover, while any regularization reult in information lo and in information value relative to that regularization, the regularization produced by the artificial neural network i particularly complex and data dependent and it i difficult to ae the relation between the original target information and the regularized meaure one obtain. Thi i made evident in the apparently paradoxical reult of Kjaer et al (994) where occaionally code that are by definition more rich in information (retaining more principal component of the repone) appear to carry le information, after they have been queezed through the artificial network. The limited ampling error i a tatitical problem common to many different field, whenever one trie to etimate, from a finite ample, a function of a full probability ditribution. Several author have addreed it, outide the domain and the peculiaritie of computational neurocience, e.g. focuing on probabilitie given on dicrete et. Wolpert and Wolf (995) (ee alo reference therein) propoe the calculation of the function (in our cae, e.g., I) of the true probabilitie given the experimental frequencie. Thi, which i in fact the original aim, (and which i obviouly different from calculating the function of the frequencie, our I N ) i feaible, however, only by making an aumption a to the a priori probability ditribution. It i then difficult to ee how to ue thi conceptually appealing approach in cae, uch a our of timulu repone pair, when no reaonable aumption on the prior i elf-evident. We have therefore developed an alternative approach, baed on the ue of the replica trick to compute the average error directly a an aymptotic expanion in invere power of the ample ize. In the firt application of the approach (Treve and Panzeri 995), we have required that the repone pace be dicrete (or, when the original repone pace i continuou, that it be dicretized with a traightforward binning procedure, that imply allot raw repone to the interval in which they happen to lie) and we have implicitly aumed that the conditional repone probabilitie (for each timulu) are different from zero in every bin. With thee aumption we have found that the leading contribution to the bia, dependent olely upon the number of timuli and repone bin and already calculated many year ago by Miller (955), yield mot of the error and can thu be ubtracted to correct raw etimate. Succeive term in the expanion are of little ue: either they are negligible in comparion with the firt term, or when N become very mall, they explode quickly, ignalling that data are o carce that the expanion i meaningle beyond the firt term. Regularization method different from pure dicretization, uch a convolution with a Gauian kernel, are often ued in practice to manipulate the raw data generated by recording the real repone of the cell(), which are uually continuou (poibly multidimenional) variable. In thi paper, we carry out a imilar calculation of the average ytematic error when different regularization are ued, to undertand how uch manipulation interact with the finite-ize problem and to find the correponding correction term. Moreover, we conider not only mutual information but alo conditional information (i.e. relative to a given timulu) and find the appropriate correction term, which again turn out to have different form.

4 90 S Panzeri and A Treve Our analyi alo lead u to clarify the role of the previouly ued huffled information in the correction procedure and to find imple criteria to etablih whether the ize of the data i large enough to obtain (given the regularization) a reliable and (after the correction) nearly unbiaed meaure of information. 3. The average error In thi ection we preent our evaluation of the bia, i.e. the average error, when different regularization procedure are applied to the raw data. We take the timuli to have been drawn at random (with a multinomial probability ditribution) from a dicrete et S of S element. Note that when the experimental frequency of preentation of timuli i, intead, et exactly equal to it probability and doe not fluctuate, one find lightly different correction term, a will be dicued eparately in the next ection. Let u initially conider the more general cae in which the (raw) neuronal repone i a real (poibly multidimenional) variable. It i clear from the formula for the mutual information (2), that if one i meauring a continuou output variable, in order to obtain an etimate of the tranmitted information from a finite et of N data it i neceary to regularize the raw data in ome way; otherwie, the finite number of repone will almot certainly all be different from each other, therefore each repone will uniquely identify it timulu (p N ( r) will be either or 0) and, a a reult, one will obtain a meaure of the entropy of the timulu et only and not of the tranmitted information. Moreover, the repone pace i uually quantized anyway, becaue one need to evaluate the expreion for I and I(), in practice, by performing a um, rather than an integral. Furthermore, many author, for everal reaon, prefer to ue data manipulation different from pure dicretization of the repone pace. Thee regularization can be of a imple form, uch a a convolution with a Gauian followed by dicretization, or much more complicated, like the neural network ued by Hertz et al (992). In the following ubection we hall conider four important cae of regularization: pure dicretization; convolution with a continuou ditribution and dicretization; neural network fitting of the conditional probabilitie; convolution with a continuou ditribution without dicretization of the repone pace. We hall preent in the appendix A the explicit calculation leading to our expreion for the bia in the econd cae only; but, for the ake of generality, we hall briefly dicu how to retrieve the reult preented when the other data manipulation are applied. 3.. Pure dicretization of the repone pace Let u conider in thi ection the cae in which the real repone have been binned into R different interval [m j,m j ],j =,,R, by imply aigning each repone to the interval in which it fall. In thi cae, the binning procedure atifie an independence condition, in the ene that the number of time a given bin r i occupied depend only on the underlying occupancy probability of thi bin, and not on the occupancy of other bin We write all the formulae in the manner appropriate to a one-dimenional repone pace, but the generalization to higher dimenion, a well a to the cae in which the original repone i dicrete (e.g. the number of pike of a neuron in a given time window), i traightforward. We tre that R i the total number of repone bin, independently of the underlying dimenionality, if any, of the repone pace. If, for example, the raw repone are the firing rate of two cell, which are then dicretized into R and R 2 bin, repectively, we et R = R R 2.

5 Information meaure from limited ample 9 (thi condition i violated by the prior regularization of the repone, a in the cae to follow). Within thi binning procedure, from N experimental trial available, one can obtain a raw etimate of the information: I D N () = R i= p N (i )log 2 p N (i ) p N (i) I D N = p N ()IN D (). (3) In (3) the p N are the experimental frequency-of-occupancy table, e.g. p N (i) = n(i)/n, or p N (i ) = n(i )/N, where n(i ) i the number of time repone i occurred when timulu wa preented, n(i) the number of time repone i occurred acro all timuli, and N i the number of experimental preentation of timulu. For large N the experimental frequencie p N (i) tend to the correponding probabilitie p(i), which are imply related to the original continuou underlying probability ditribution by an integration over the repone bin. Similarly, a N increae, the etimate of the tranmitted information tend to the information carried by the dicretized probabilitie: I D () = R i= p(i )log 2 p(i ) p(i) I D = p()i D (). (4) By temporarily retricting ourelve to the total tranmitted information, it i important to note that the value of the information I D obtained after quantization i le than the value of information carried by the continuou repone and, in general, information meaure are dependent upon the binning procedure adopted and, mot importantly, upon the number of bin R. There i no way of etimating the difference between I and I D from firt principle, but a good trategy for controlling thee dicrepancie can be to quantize the repone by ucceively increaing the value of R until the finite-n meaure, after the correction we are dicuing, doe not change very much. However, when the ize of the data ample i mall, a reaonable choice for R i a compromie between trying to keep the lo of information due to dicretization a mall a poible, which would require R large, and the need to control the finite-ize ditortion, which, a we hall ee below, can eventually require R mall. Of coure, the difference between IN D and I D fluctuate depending on the particular outcome of the N trial performed. One can, however, etimate the average of the difference, that i the bia, by averaging (... ) over all poible outcome of the N trial, keeping the underlying probability ditribution fixed. We have obtained an expreion for the bia a a erie expanion in invere power of the ample ize N: I D N ID = C m (5) m= where C m repreent ucceive contribution to the aymptotic expanion of the bia (the term C m i proportional to N m ). Here we report jut the leading term, whoe expreion i derived in appendix A: C D = 2N log 2 R R (S ) where R denote the number of relevant repone bin for the trial with timulu, i.e. the repone bin in which the occupancy probability p(i ) (at given ) i non-zero. In the ame way, R denote the number of repone bin where p(i) i non-zero. In the cae in } (6)

6 92 S Panzeri and A Treve which each repone bin i ha a non-zero probability of being occupied for every timulu, we recover the reult reported in Treve and Panzeri (995): C D = (S )(R )}. (7) 2N log 2 At the end, to correct for the finite-ize problem we have to evaluate the correction term in (6), which depend upon the underlying probabilitie olely through the R parameter, and thu in a much weaker way than the mutual information, which depend upon the full ditribution. Therefore, even though the parameter R, R have, in turn, to be etimated from the data, thi procedure i much more accurate than a direct etimate of the information. To undertand how one can etimate the number of relevant bin, we note that the number of relevant bin differ from the total number of bin allocated becaue ome bin may never be occupied by repone to a particular timulu. A a conequence, if R i calculated uing for each timulu the total number of bin R, then the C term, which i in thi cae equal to (7), turn out to overetimate the ytematic error, whenever there are timuli that do not pan the full repone et. On the other hand, the number of relevant bin alo differ from the number of bin actually occupied, R, for each timulu (with few trial), becaue more trial might have occupied additional bin. Again, it turn out that uing the number of actually occupied bin R for calculating C lead, when few trial are available, to an underetimate of the ytematic error (the underetimation becoming negligible for R/N becaue R tend to coincide with R for all timuli). It i clear that when N i mall, more ophiticated procedure, uch a Bayeian etimation, are needed to evaluate the quantitie R, R. A mentioned above, Wolpert and Wolf (995) how how to calculate any function of the probabilitie given the experimental frequencie, uing the Baye rule. Thi require ome knowledge, or ome aumption, on the a priori probability ditribution of the probabilitie (ee appendix B). Since we do not have any knowledge of the prior, we do not ee how to ue thi approach to etimate the mutual information itelf, which quantity depend on the full detail of the probability table. Neverthele, we how (in appendix B) how a correction to the mutual information depending only upon a few parameter, uch a R, R, can alo be well etimated with a crude hypothei about the prior probability function. The idea i to ue Baye theorem to recontruct the true probabilitie, uppoing they are non-zero into R interval, and then chooe an R uch that the expected number of occupied interval (which can be calculated a a function of the Baye etimate of the probabilitie) matche the experimentally oberved value. Thi etimation, although baed on a very imple anatz (appendix B) on the prior ditribution, i ufficient to give good reult even for relatively mall value of N, a hown in figure (a). The reaon for thi good etimation, in our opinion, lie in the fact that only the parameter R, R have to be etimated baed on the arbitrary anatz, and the information I depend upon thee only in the correction term. The obervation that the leading bia term (6) can alo, in general, be probability dependent lead to a better undertanding of the effectivene with which the huffled information can correct for limited ample. If the underlying probability i uch that each bin ha non-zero probability, then the bia hould be of the ame order for the huffled and the true probability table and we can correct the meaured information by imply ubtracting the value of the huffled information, a previouly tated by Treve and Panzeri (995). If, intead, we have many zero-probability bin, the huffling obviouly overetimate the number of occupied bin, which implie that, in thi cae, the huffled information i a (poibly high) overetimation of the bia, wherea our C D term, (6), hould continue to

7 Information meaure from limited ample 93 give a good etimate of the bia. Therefore, even when retricting to mutual information and dicrete or dicretized repone, there i no way, valid for all probability ditribution, of relating the value of huffled information to the value of the bia, a originally propoed by Optican et al (99). The huffling procedure can only be ued when all three condition are met: (i) repone are dicrete or imply dicretized; (ii) the target i the mutual information; and (iii) p(, i) i deemed to have all non-zero element. Our analyi, intead, can alo be carried out for the conditional information. Again, we can give an aymptotic expanion for the bia: I D N () ID () = m= and the leading correction term i now: C D () = [ p(i )] 2N log 2 i p N () p(i ) + 2p 2 (i ) + 2N log 2 i p(i) Cm D () (8) } p(i ) where the hat on the um over repone bin i denote that only interval of non-zero occupancy probability are to be conidered, and in calculating explicitly the average of p () the intance with p N () = 0 mut be excluded. Etimating thi expreion (9) for the bia directly from real data i likely to lead, a for the C D term, to undercounting if N i mall. However, the dependence of C D () on the probabilitie i not a imple a for C D, and therefore a Bayeian etimate of CD () i more complicated and, without ome knowledge on the prior, i not expected to work a well. Thi handicap may be to ome extent circumvented by chooing, when uch freedom exit, repone bin appropriate to the timulu being conidered, i.e. collating all bin for which no repone to occur into a ingle bin. All the analytical reult preented here for the dicrete cae are well confirmed by computer imulation preented in Treve and Panzeri (995) and Panzeri and Treve (995). New imulation with more realitic probability ditribution are preented in the next ubection and confronted with the reult obtained with different regularization. (9) 3.2. Convolution with continuou kernel and dicretization Let u now conider the cae in which the regularization of the data i performed by firt convolving the repone with a continuou kernel function and then dicretizing the output pace into R interval [m j,m j ],j =,,R. With thi data manipulation, moothing (denoted by a tilde) followed by dicretization, we obtain, from the N available timulu repone pair, a raw etimate of the information: R I N D () = i= p N (i )log 2 p N (i ) p N (i) I N D = p N () I N D () (0) where the p N ( ) are the experimental frequency table, obtained by convolving the actual experimental repone r j with ome kernel ditribution K(r,r j,σ) (e.g. a Gauian one) and then integrating out the obtained probability denity over the repone interval: N p N (i ) E i (r j ; σ) p N (i) p N () p N (i ) () N j=

8 94 S Panzeri and A Treve where E i (r j ; σ) i the integral (over the ith interval) of the kernel function centred in r j : mi E i (r j ; σ) = drk(r,r j,σ). (2) m i The um over j in () i performed over all the actual repone to timulu and the function K can depend on ome parameter σ (uch a the width in the cae of a Gauian convolution) which can be a function of the data ditribution itelf: σ = σ(,r j ). For large N the raw repone ditribution approach the underlying one and thu we can write: p(i ) = dre i (r; σ)p(r ) p(i) = p() p(i ). (3) Similarly, the etimate of the tranmitted information tend to the information carried by the moothed underlying probabilitie: R I() p(i ) = p(i )log 2 I = p() I(). (4) p(i) i= Again, information value are in general dependent upon the moothing and binning procedure adopted and, mot importantly, upon the number of bin R and, now, upon the moothing width. It i worth emphaizing that moothing produce a further lo of information on top of the lo due to the dicretization alone, and if the rationale for moothing i only to better control the finite ampling error, it i important to undertand whether much better control can indeed be achieved. For the leading term in the bia I N D I D C D IN D () I D () C D () (5) we now find the expreion C D = 2N log 2 i q(i ) p(i ) i q(i) (S ) p(i) C D () = p N log 2 i N () q(i ) } p2 (i ) + p2 (i ) q(i ) 2 p(i ) p(i) q(i) p(i ) p(i ) p 2 } (i) + (7) 2N log 2 i p 2 (i) where q( ) are evaluated from the underlying probability ditribution a follow: q(i ) drp(r )Ei 2 (r; σ) q(i) p() q(i ). (8) The correction term (6) and (7) are now dependent upon both the underlying probability and the choen regularization. The firt dependence raie, a in the dicrete cae, the problem of how to etimate the correction (6) and (7) from the data and, in particular, how to avoid undercounting the bin with non-zero probability over which to take the um in (6) and (7). If one convolve the repone with an infinite range ditribution, uch a the Gauian, no interval remain trictly empty after the convolution, and then the potential underetimation of the correction i le important than in the dicrete cae. Even with a Gauian convolution, however, ome undercounting might occur becaue In the following, in evaluating average, we aume that the regularization parameter do not fluctuate depending upon the outcome. When data-dependent parameter are ued, we uppoe that the fluctuation in information meaure due to variation in the parameter are ubleading with repect to thoe due to fluctuation of P N ( ). } (6)

9 Information meaure from limited ample 95 of numerical truncation. If we uppoe that the typical moothing width i mall compared with the typical bin length, we can take the moothing to have ignificant effect only in the nearet interval. In thi cae, an approximate form of the averaged underetimation can be worked out : C D ( C D ) N ( C D ) } [ ] N = 2N log 2 i p(i ) p(i ) p(i + ) [ ] } N 2N log 2 i p(i ) p(i) p(i + ). (9) Thi approximate form for the underetimation of C D capture jut the fact that, when the moothing width i mall with repect to the typical bin length, in a bin the moothed probability p(i ) can be conidered null only if we do not have outcome in the nearet bin. In thi cae, ( C D) can be added to C D to marginally improve the etimation of the bia. A for the validity of the boottrap procedure, the fact that the correction term (6) and (7) are now alo regularization dependent, further complicate the analyi. If the convolution width i not too large, we can expect that the procedure will tend to overetimate the repone range for ome timulu (due eentially to the ame mechanim which appear in the dicrete cae) and then to overetimate the bia in the cae in which one oberve very different repone range to different timuli. Thu, in thi ituation the huffled information might be larger than the bia. On the other hand, when the convolution width i large and data dependent (for example, determined by the tandard deviation of the repone to each timulu, a in Optican and Richmond (987)), or in general when the regularization i data dependent (and then different for the actual and the huffled repone), the huffled information might eaily underetimate the bia, becaue it might reflect an effectively tronger regularization. Thu, in thee ituation it i not afe to rely on the boottrap procedure, either to correct the raw etimate by ubtraction, or to conclude, when the huffled information i very mall, that the average bia itelf mut be mall. To upport thee analytical reult, we performed explicit numerical imulation. We choe a tet underlying probabilitie Poion ditribution, which are fair imple model of the piking activity of neuron under certain condition (Abele et al 990, Levine and Troy 986, Scobey and Gabor 989). We generated the ditribution of mean firing rate r() correponding to each timulu by electing a random variable x from a flat ditribution in the interval [0, ) and then etting ( r = log x ) if x<2a r = 0 if x>2a. (20) 2a The parameter a i, on average, the parene (Treve 990) of the firing rate ditribution. The number of pike n recorded on each trial over a period t (t = 500 m in the preent imulation) followed the Poion ditribution P(n ) = [r()t]n exp[ r()t]. (2) n! To meaure, from N trial, the information carried by the firing rate generated in thi way, we ued a regularization procedure imilar to that ued in Roll et al (995a). The range of An expreion for C D can alo be derived for the dicrete cae (in fact, a impler and exact expreion). However, in that cae, it give typically large contribution which are themelve difficult to etimate from the data, o that in the dicrete cae it i much better to ue the Bayeian algorithm to etimate R, R intead.

10 96 S Panzeri and A Treve repone wa dicretized into a preelected number R of bin, with the bin limit elected o that each bin contain the ame number of trial (within ±). A moothing procedure wa applied by convolving the individual value with a Gauian kernel. The moothing width ha an overall multiplicative parameter γ (ucceively increaed in the imulation to tet how different convolution width influence the finite ize effect) and i proportional to the quare root of each value (the proportionality factor i et uch that on average the moothing width match γσ, where σ i the tandard deviation of the firing rate of each timulu). Figure how, for different ample ize, how our correction procedure improve both on raw etimate of mutual information and on the boottrap procedure of ubtracting the huffled information; moreover, the figure illutrate the effect of moothing the repone on the accuracy of information etimate. When no moothing i applied (figure (a)), the aymptotic value of the dicretized information i only a few per cent below the true, or unregularized, value (the full line). The finite ampling bia in raw information etimate become of imilar ize to the lo due to dicretization, and roughly compenate for it, only if a many a 256 trial per timulu are available. The boottrap procedure reduce the bia to imilar level earlier, at roughly 00 trial per timulu (but note that the remaining bia i alo downward and doe not compenate for the regularization lo). Our correction procedure uing the Bayeian etimate for R, R allow the ame preciion already for N R (in thi cae R = 6). In contrat, uing correction term baed on the number of Figure. Mutual information value for the ditribution of timuli and Poion repone decribed in the text (the parene of the mean firing rate i a = 0.4), with S = 6 and R = 6 and different value of N. The three panel correpond to (a) γ = 0.0 (pure dicretization); (b) γ = 0.5; (c) γ =.0. The full line i the real value of the information in the ditribution and the dahed line i the regularized value, that could be extracted from an infinite ample of data, after the precribed regularization of the repone (thi latter value varie with N becaue the regularization moothing width i data dependent). In the firt panel, compared to thee reference value are, for each N, the raw etimate ( ), the etimate corrected by ubtracting the C term calculated by etimating the relevant bin by counting the number of actually occupied one ( ), etimating the effective bin with the Bayeian procedure decribed in the text ( ), taking all bin to be relevant ( R = R = 6) (+) and the etimate corrected by the boottrap method ( ). In the econd and third panel, we plot only the raw etimate ( ), thoe corrected by ubtracting C D + ( C D )( )and by the boottrap method ( ). Each value i plotted with the tandard deviation of the mean of 00 meaurement. Note that the N axi i on a logarithmic cale.

11 Information meaure from limited ample 97 Figure. Continuation. actually occupied repone bin, or on the total number of bin, i not much more effective than the boottrap or even raw etimate (note that, unlike in Treve and Panzeri (995), we ue here more realitic repone ditribution with many empty bin, which account for the poor performance of ubtracting either the huffled information or C calculated naively). When a weaker (figure (b)) or tronger (figure (c)) moothing i applied before dicretizing the repone, the lo of information due to regularization become much larger and more important than finite ampling error. Neverthele, the latter are till controlled effectively by our correction procedure. Although the procedure i le refined than in the dicrete cae, convergence to the aymptotic (but trongly downward biaed) value i fater (particularly in figure (c)). The concluion appear to be that moothing with a Gauian doe more damage than good, although we do note that (i) there may be other reaon for moothing (e.g. avoiding edge effect), and (ii) when it i known that the moothing width i mall with repect to the relevant difference in the repone, moothing may induce much maller lo than in our example, with poibly fater convergence with ample ize. Figure 2 how the worth of ubtracting the C D () term in the cae of conditional information. In thi cae, no huffling of the timulu repone pair would be applicable, wherea it i evident that our ubtraction yield reaonable reult, bringing the corrected

12 98 S Panzeri and A Treve Figure 2. Value for the information conditional to which of S = 20 (imulated) timuli wa preented, plotted againt the mean rate r() to each timulu (on an arbitrary cale). The firing rate are ditributed with parene a = 0.7. Again, the full curve indicate the real and the dahed one the regularized information value; and the ymbol indicate raw and ubtracted meaure, each with tandard deviation of the mean over 00 meaure. Here R = 0, N = 300, γ = value within the narrow range panned by the difference between real and regularized information value. The C-hape of the information veru rate plot and whether or not (a in thi imulation) it touche the rate axi are intereting fact, dicued by Roll et al (995b) Neural network regularization When a regularization imilar to that introduced by Hertz et al (992) i ued, the bia can be evaluated in a imilar fahion. The idea of Hertz and hi co-worker i to ue a feed-forward neural network to fit, from the real repone r j to a given timulu, the conditional probability that a timulu i recognized a the ith: where E i (r; ω) = N p N (i ) E i (r j ; ω) p N (i) p N () p N (i ) (22) N j= exp [ l (W ilh l + B i ) ] S j= exp [ l (W jlh l +B j ) ] [ ] Q H l(r) = tanh ω lm r m + b l. (23) In (23), H l depend on Q variable r m choen to decribe the raw neuronal repone, wherea W, ω, b, B are parameter for the neural network, elected according to a certain optimization procedure (ee Kjaer et al (994) for detail). After thi regularization, the output pace become an S-dimenional dicretized et, equivalent to the timulu et, which could be called the et of poited timuli, and the conditional probability p(i ) (22) can be interpreted a the conditional probability with which a repone elicited by timulu may be attributed to timulu i. It i to be noted that, in thi procedure (Hertz et al 992, Kjaer et al 994), to avoid overfitting, the (fitting) parameter W, ω, b, B entering in the neural network are adjuted m=

13 Information meaure from limited ample 99 on a et of training data and the information i calculated on a et of tet experimental data. Thu, in the context of evaluating the finite ize bia, the parameter N i the number of tet timulu repone pair. Without going further into the detail of the procedure, it i ufficient, for our purpoe, to remark that the form of the regularized probability ditribution (22) i the ame a in (), except that E i (r) i no longer evaluated imply by integrating a continuou kernel over the ith bin, but with the more complicated rule (23). Thi doe not affect the reult for the bia, which are therefore the ame a in ubection 3.2, with the only difference that E i (r) mut be computed from (23) intead of (2). In practice, the information etimate produced by the network i unlikely to require any finite ize ubtraction a, if anything, it uffer more from the lo due to regularization. A comparion of the binning-and-correcting procedure and the neural network procedure on a large et of realitic imulated data i the object of another tudy (Golomb et al 996) Convolution with continuou kernel Finally, let u conider the cae in which raw repone are manipulated by convolving them with a continuou kernel function, a before, but without a ubequent dicretization of the output pace. The raw information etimate now read I N () = dr P P N (r ) N (r )log 2 P N (r) I N = p N () I N () (24) where the P N are the experimental ditribution, obtained by convolving the experimental repone r i with a kernel ditribution K(r,r i,σ): P N (r ) = N N j= K(r,r i,σ) P N (r) = p N () P N (r ) (25) where the um over j i performed over all the N experimental repone to the timulu. ANincreae, the raw repone ditribution approach the underlying one: P(r ) = dr K(r,r, σ )P (r ) P(r) = p() P(r ) (26) and the raw etimate of information tend to: I() = dr P(r )log P(r ) 2 P(r) The expreion we find in thi cae for the bia are [ ( ) Q(r ) C = dr Q(r) ] } (S ) 2N log 2 P(r ) P(r) I = p() I(). (27) [ C () = dr p N N log 2 () Q(r ) ] P 2 (r ) + Q(r ) + P 2 (r ) 2P(r ) P(r) + dr P 2 (r) P(r ) + Q(r) P(r ) 2Nlog 2 P 2 (r) It hould be noted that the mutual information defined in Kjaer et al (994) i not fully equivalent to the mutual information carried by the regularized probabilitie (4). (28) (29)

14 00 S Panzeri and A Treve where N Q(r ) = K 2 (r, r i,σ) Q(r) = p N () Q N (r ). (30) N j= In the continuou cae, the problem of underetimation of the correction term (28) and (29) when calculated from data, i abent, ince thi problem i intrinically related to the dicretization of the output pace. Thi continuou cae i rather academic anyway, a in practice one uually perform the required integral on the computer by firt dicretizing and then taking um. It remain true, however, that one i cloe to the continuum limit, and the imple expreion above hold, whenever the dicretization i ufficiently fine with repect to the width of the kernel. 4. The bia with fixed number of trial per timulu In the previou ection we tudied the finite ize ditortion when the timuli are drawn at random from a dicrete et. Here we preent the reult valid when, intead, the experimental frequency of preentation of timuli doe not fluctuate, but it i et exactly to it probability: p N () p(). The calculation of the bia i very imilar to that preented for the previou cae, but with the obviou difference that, in evaluating average a in (A4) (A6), one ha to average over repone in the ame way a detailed in appendix A, but not, a before, over p N () with the multinomial ditribution. We report only the reult for the cae of convolution with a kernel K(r,r j,σ) and dicretization into R interval C D = 2N log 2 [ C D () = N log 2 i i p ( q(i ) p(i ) + p N() p 2 (i ) p(i) N () q(i ) p2 (i ) 2 p(i ) p(i ) q(i) + 2N log 2 i p 2 (i) ) ] } q(i) S p(i) } + p2 (i ) q(i ) p(i) p( ) p 2 (i ) p(i ) p 2 (i) where the notation i the ame a in ection 3.2. The reult correponding to the other regularization conidered in the previou ection can be derived by taking the appropriate limit, a explained in the appendix. } (3) (32) 5. How bet to chooe the number of bin? In previou ection we have dicued the poible problem ariing when convolving data with continuou ditribution and etablihed the range of validity of our correction, which in the dicrete cae work well even down to N R. Given the effectivene of the binning procedure, we recommend limiting the regularization to imple binning, unle motivated by other conideration (a mentioned in ection 3.2). The important quetion of the optimal choice of the number of bin for an experiment with S timuli and N trial per timulu remain. A reaonable anwer can be to chooe R N to be at the limit of the region where the correction procedure i expected to work and thu till be able to control finite ampling, while minimizing the downward bia produced by binning into too few bin. Thi choice hould effectively minimize the combined error due to regularization

15 Information meaure from limited ample 0 and finite ampling. In figure 3 information etimate obtained by chooing R = N are compared, for different value of N, with the full, unregularized, value of the information carried by the Poion ditribution of repone. Reult appear to be reaonable in the whole N range explored. However, the correction procedure baed on binning indicate the minimum appropriate number of trial per timulu in real experiment. The correction function reaonably down to N R, and the minimum number of repone bin which may not throw away information, if the appropriate code i ued, i jut the ame a the number of timuli, R = S. Therefore a minimum number of N = S trial per timulu i a fair demand to be made on the deign of experiment from which information etimate are to be derived. Figure 3. Mutual information value for the ditribution of timuli and Poion repone decribed in the text. Here a = 0.4, S = 6 and the repone pace i purely dicretized. The number of bin R i fixed equal to N. The ymbol have the ame meaning a in figure (a). Note that for N = 6, alo R = 6 and the reult i the ame a hown in figure (a). For higher value of N, reult approach the unregularized value of the information, wherea in figure (a) they approached the value regularized with R = 6 bin. 6. Concluion The work reported here ha no deep theoretical ignificance; conceptually, it i on a par with calculating the correct expreion for the moment of a Gauian ditribution, for example, when thee have to be etimated from the data. It i, however, of practical importance, epecially, although by no mean olely, for the analyi of neuronal activity recorded in the mammalian nervou ytem in vivo. Meauring the information carried by neuronal activity ha been avoided by many neurophyiologit becaue of the eemingly huge amount of data required to obtain reaonable tatitic, and the outcome of uch meaurement have been widely accepted only in a few intance, e.g. when performed in inect (Bialek et al 99), in which data collection i not a contraint and the reult appeared hard (jut a the nervou ytem examined appear to be hard-wired). Our procedure for evaluating the bia and correcting information etimate will not, a i evident from the figure, be of any help when data are o carce a to make the expanion meaningle; nor, obviouly, when they are abundant enough to make any correction uperfluou. The procedure i only ueful for a range of ample ize, which range i,

16 02 S Panzeri and A Treve however, roughly of the order of magnitude of that in which typical neurophyiological experiment lie. The data collected in uch experiment are, then, available for information meaurement, at the very limited cot of adding a very quick routine to tandard data analyi package (Roll et al 995a c). The diffuion of the practice of meauring (accurately) the information content of neuronal activity i likely to greatly enhance our quantitative undertanding of the proceing of information in the nervou ytem. Appendix A. We give here the derivation of the reult preented in previou ection. In the calculation we conider the cae in which the data are treated by convolving repone with a kernel ditribution and then by dicretizing the repone pace into R interval. Finally, however, we how how to recover the reult appropriate to the other data manipulation. The method ued here i different from that employed in Treve and Panzeri (995) and cloer to that of Carlton (969); reult are, in any cae, fully equivalent. We tart by calculating the average of the total amount of information (0), which can be expreed a follow: I N D = i p N() p N (i )log 2 p N (i ) p i N(i) log 2 p N (i) (A) where p( ) i defined in () and the hat on the um over repone bin i in (A) denote that we mut exclude from that um, for each term of the um over timuli, the bin in which p(i ) = 0 (in fact, in thoe bin the only permitted outcome i p N (i ) = 0 and they trivially diappear from the average). Now we can ued the following erie expanion for the logarithm: log 2 ( p N ( )) = ( p N ( )) j. (A2) log 2 j j= Thi expanion (A2) i convergent for all value of p N ( ), ince 0 < p N ( ) (note that in our calculation the configuration p N ( ) = 0 can be excluded ince it give a vanihing contribution to the average). Taking term by term expectation in (A) we find: I N D = log 2 p N () p N (i ) ( p N(i )) j i j= + log 2 = log 2 j p N (i) ( p N(i)) j j i j= + log 2 i i j= k=0 j= k=0 j ( ) k j j ( ) k j ( ) j k ( ) j k p N () p k+ N (i ) p k+ N (i) (A3) where in the lat tep we ued the binomial decompoition for ( p N ( )) j. We can now calculate the average by the following procedure. Firt we average over repone (at fixed timulu and number of preentation per timulu N Np N ()) imply by auming that the probability of obtaining a raw repone r (given the timulu ) i given We are happy to make the required routine available via the Internet.

17 Information meaure from limited ample 03 by P(r )dr and by ubtituting the um over outcome with the correponding (correctly normalized) integral in the repone pace. We are then left with an average over p N (), with a multinomial ditribution. Note that, in averaging term of the form ( p N (i)) k, ince the parameter pecifying the kernel can be timuli dependent, we mut decompoe p N (i) a p N (i) = p N() p N (i ), average firt over the repone (at fixed timulu) and finally over p N () with the multinomial ditribution. In general, we obtain the following expreion: p k N (i ) = pk (i ) + N ( k 2 ) p k 2 (i ) [ q(i ) p 2 (i ) ] ( + o N p(i ) p N () p N k (i ) =p() pk (i ) + ( ) k p k 2 (i ) [ q(i ) p 2 (i ) ] ( ) + o N 2 Np() p(i ) (A5) p N k (i) = pk (i) + ( ) k p k 2 (i) [ q(i) p 2 (i) ] ( ) + o (A6) N 2 N p(i) where q( ) i defined in (8). Ignoring the third term in each of (A4) (A6) and then ubtituting (A4) (A6) into (A3), we find an expreion for the bia which i exact up to O(/N 2 ) term and i a good approximation to the bia if in each bin N p(i ) : I N D [ ] j 2 log 2 i p(i ) p(i ) [ p(i ) ] 2 j j= + 2N + log 2 i [ q(i ) p 2 (i ) ][ j(j ) p(i ) 2j( p(i )) ]} j= [ ] j 2 p(i) p(i) [ p(i) ] 2 j + [ q(i) p 2 (i) ][ j(j ) p(i) 2j( p(i)) ]} 2N = I D + C D (A7) where I D i given in (4) and C D i the leading contribution to the bia: [ C D = ( ) ] } q(i ) q(i) (S ). (A8) 2N log 2 i p(i ) p(i) By going further in the /N expanion when conidering the average (A4) (A6), one can alo obtain the next term in the /N expanion of the bia by the ame procedure. Here we report only the reult for the econd term: C 2 D = p 2N 2 N log 2 () 2 p(i ) t(i ) + 3 q [ i 2 ] } (i ) p 3 (i ) 2 p(i) t(i) + 3 q 2 } (i) + (A9) 2N 2 log 2 i p 3 (i) where t(i ) drp(r )Ei 3 (r ) t(i) p() t(i ). (A0) ) (A4)

18 04 S Panzeri and A Treve Higher-order term (valid in the dicrete cae) are reported in Treve and Panzeri (995). In fact, (A8) i derived (a the leading term in the bia) under the condition that N p(i ) in each interval; wherea by inpecting the higher-order expanion term, one can, a mentioned in Treve and Panzeri (995), expect them to be ucceively maller (and negligible with repect to C D) under the le tringent condition that C D. Therefore, the higher-order contribution are, in any cae, cloe to negligible whenever C D i a good approximation for the bia. When thi i not the cae, becaue the condition N p(i ) i everely violated, computer imulation indicate that taking higher-order term into account (which i itelf not eay), doe not help; on the contrary, in uch a low-n regime in which C D i often already too large, the next term become huge and ignal the breakdown of the expanion procedure. If one i intereted in meauring, intead of the average tranmitted information, the conditional tranmitted information, relative to a given timulu, a imilar calculation can be performed to obtain the bia of thi quantity. The main technical tep which i different i that when calculating I() D from (0), I N D () = R p N (i )log p N (i ) R p N (i )log p N (i) (A) log 2 log 2 i= i= after uing the convergent expanion (A2) for the logarithm, one ha to calculate the average of p N (i ) p N k (i) up to the next-to-leading order p N (i ) p N k (i) = p(i ) pk (i) + ( ) k p(i ) p k (i) [ p(i) ] N 2 + k ( [ ] p(i ) p(i ) p k (i) + o N Np() p(i ) Our reult, again valid when N p(i ) in each interval, i now expreed a with C D () = N log 2 i I N D () I D () C D () + 2N log 2 } + p2 (i ) q(i ) p(i) q(i) p(i ) p(i ) p 2 (i) p N () q(i ) p2 (i ) 2 p(i ) i p 2 (i) ). (A2) (A3) }. (A4) The dicrete cae (for which the reult are fully dicued in ection 3.) can be eaily derived by chooing a Gauian a kernel function and then taking the limit of zero convolution width. In thi cae, it i eay to how from (A8) that the leading bia term take the form: C D = [ ] [ ] } 2N log 2 i p(i ) i p(i) } = R R (S ). (A5) 2N log 2 It hould be noted that in the dicrete cae the following evaluation of the bia of the mutual information wa derived by Carlton (969): ( IN log D ID i 2 + p(i) ) Np(i) 2N log 2 p(i) [ p(i)(n ) ] } (Np(i) + p(i)) 2

19 Information meaure from limited ample 05 + ( log i 2 + p(i ) ) N p(i ) p(i ) [ p(i )(N ) ] }. (A6) 2N log 2 (N p(i ) + p(i )) 2 The expreion (A6) for the bia agree with our expreion (6), up to the /N order, but i very different when going to higher order. The procedure employed by Carlton to derive the reult (A6) i imilar to that preented here, in the ene that he ue the expanion (A2) for the logarithm and take term by term expectation by truncating average of power of p( ) to the next-to-leading order, a in (A4) (A6), but with a trick (valid only in the dicrete cae) ued to obtain (without going further in /N in the evaluation of the average (A4) (A6)) a partial re-ummation (to all order in /N) of the complete expreion for the bia. Thi partial re-ummation, however, i of dubiou value from the conceptual point of view and give utterly nonenical reult when checked numerically. In fact, for example, by uing the correction term (A6) in the imulation reported in figure (a), we obtained an etimate of the bia much larger than the raw information in the N range The continuum limit, reult for which are preented in ection 3.4, can be reached when R, a follow. Let u denote ome typical ize of the repone by ϱ (taken here to be uni-dimenional) and let u introduce the following ucceion of infinite dicretization, indexed by n, into interval R i;n (i = 0, ±, ±2,... label each interval) R i;n r; i } ϱ r<i+ 2n 2 ϱ. (A7) n The dicrete probabilitie (3) have the form p n (i ) dr P(r). (A8) R i;n By introducing the function Ɣ n (r) 2n ϱ p n(i) for r R i;n (A9) we have the identity ( 2 n ) p n (i) log 2 ϱ p n(i) = Ɣ n (r) log 2 Ɣ n (r) dr (A20) R i;n from which we can derive p n (, i) p n (, i) log 2 p() p i, n (i) = Ɣ n (, r) drɣ n (, r) log 2 p()ɣ(r). (A2) Now, with the hypothei that P(r), P(r ) are bounded and continuou almot everywhere (Ihara 993), we have that in the n limit Ɣ(r ) p(r ) and in the ame limit the (infinitely) dicretized information (A2) tend to the continuou one (24), wherea the infinitely dicretized term (A8) tend to that derived in the continuou cae (28). Appendix B. In thi appendix we briefly dicu the Bayeian-like method we ue to extract the value of R, R from the data. Let u firt recall ome terminology from Baye theory (ee, for example, Wolpert and Wolf (995) and the recent review of MacKay (995) on Baye theory and data modelling). The meaning of the variou parameter i explained in ection 3..