Mathematical Equivalence of Two Common Forms of Firing Rate Models of Neural Networks

Transcription

1 NOTE Communcated by Terrence Sejnowsk Mathematcal Equvalence of Two Common Forms of Frng Rate Models of Neural Networks Kenneth D Mller ken@neurotheorycolumbaedu Center for Theoretcal Neuroscence, Dept of Neuroscence, Swartz Program n Theoretcal Neuroscence, and Kavl Insttute for Bran Scence, College of Physcans and Surgeons, Columba Unversty, New York, NY 10032, USA Francesco Fumarola fumarola@physcolumbaedu Department of Physcs, Columba Unversty, New York, NY 10027, USA We demonstrate the mathematcal equvalence of two commonly used forms of frng rate model equatons for neural networks In addton, we show that what s commonly nterpreted as the frng rate n one form of model may be better nterpreted as a low-pass-fltered frng rate, and we pont out a conductance-based frng rate model At least snce the poneerng work of Wlson & Cowan (1972), t has been common to study neural crcut behavor usng rate equatons equatons that specfy neural actvtes smply n terms of ther rates of frng acton potentals, as opposed to spkng models, n whch the actual emssons of acton potentals, or spkes, are modeled Rate models can be derved as approxmatons to spkng models n a varety of ways (Wlson & Cowan, 1972; Matta & Del Gudce, 2002; Shrk, Hansel, & Sompolnsky, 2003; Ermentrout, 1994; La Camera, Rauch, Luscher, Senn, & Fus, 2004; Avel and Gerstner, 2006; Ostojc & Brunel, 2011; revewed n Ermentrout & Terman, 2010; Gerstner & Kstler, 2002; and Dayan & Abbott, 2001) Two forms of rate model most commonly used to model neural crcuts are the followng, whch we wll refer to as the v-equaton and r-equaton respectvely: τ dv = v + Ĩ + Wf(v), (1) τ dr = r + f(wr + I) (2) Neural Computaton 24, (2012) c 2011 Massachusetts Insttute of Technology

2 26 K Mller and F Fumarola Here, v and r are each vectors representng neural actvty, wth each element representng the actvty of one neuron n the modeled crcut v s commonly thought of as representng voltage, whler s commonly thought of as representng frng rate (probablty of spkng per unt tme) f(x) s a nonlnear nput-output functon that acts element-by-element on the elements of x, that s, t has th element (f(x)) = f (x ) for some nonlnear functon of one varable f f typcally takes such forms as an exponental, a power law, or a sgmod functon, and f (v ) s typcally regarded as a statc nonlnearty convertng the voltage of the th cell v to the cell s nstantaneous frng rate W s the matrx of synaptc weghts between the neurons n the modeled crcut Ĩ and I are the vectors of external nputs to the neurons n the v or r networks, respectvely, whch may be tme dependent In the appendx, we llustrate a smple heurstc dervaton of the v-equaton, startng from the bophyscal equaton for the voltages v Along the way, we also pont to a conductance-based verson of the rate equaton When developng a rate model of a network, t can be unclear whch form of equaton to use or whether t makes a dfference Here we demonstrate that the choce between equatons 1 and 2 makes no dfference: the two models are mathematcally equvalent, and so wll dsplay the same set of behavors It has been noted prevously (Beer, 2006) that when I s constant and W s nvertble, the two equatons are equvalent under the relatonshp v = Wr + I, Ĩ = I We generalze ths result to demonstrate the equvalence of the two equatons when W s not nvertble and nputs may be tme dependent The v-equaton s defned when we specfy the nput across tme, Ĩ(t), and the ntal condton v(0); we wll call the combnaton of these and equaton 1 a v-model The r-equaton s defned when we specfy I(t) and r(0); we wll call the combnaton of these and equaton 2 an r-model We wll show that any v-model can be mapped to an r-model and any r-model can be mapped to a v-model such that the solutons to equatons 1 and 2 satsfy v = Wr + I As we wll see, the nputs n equvalent models are related by Ĩ = I + τ di, or τ di = I + ĨThats,Is a low-pass-fltered verson of Ĩ Note that there s an equvalence class of I, parameterzed by I(0), that all correspond to the same Ĩ under ths equvalence We assume that the equvalence class has been specfed, that s, Ĩ has been specfed (f I has been specfed, Ĩ can be found as Ĩ = I + τ di ) Then a v-model s defned by specfyng v(0), whle an r-model s defned by specfyng the set {r(0), I(0)}IfWs D D, then v(0) s D-dmensonal, whle {r(0), I(0)} s 2D-dmensonal, so we can guess that the map from r to v takes a D-dmensonal space of r-models to a sngle v-model, and conversely the map from v to r takes a sngle v-model back to a D-dmensonal space of r-models, and we wll show that ths s true

3 Mathematcal Equvalence of Frng Rate Models 27 We frst show that f r evolves accordng to the r-equaton, then Wr + I evolves accordng to the v-equaton Settng v = Wr + I, we fnd: τ dv dr = Wτ + τ di di = W( r + f (Wr + I)) + τ (3) = (v I) + W f (v) + τ di (4) = v + Ĩ + W f (v) (5) Therefore, f v evolves accordng to the v-equaton and r evolves accordng tothe r-equaton and v(0) = Wr(0) + I(0), then, snce thev-equaton propagates Wr + I forward n tme, v = Wr + I at all tmes t > 0 We wll thus have establshed the desred equvalence f we can solve v(0) = Wr(0) + I(0) for any v-model, specfed by v(0), or for any r-model, specfed by {r(0), I(0)} Note that, as expected, a D-dmensonal space of r-models converges on the same v-model Snce {r(0), I(0)} forms a 2D-dmensonal space, whch s constraned by the D-dmensonal equaton v(0) = Wr(0) + I(0), theddmensonal subspace of r-models {r(0), I(0)} that satsfy ths equaton all converge on the same v-model To go from an r-model to a v-model s straghtforward: we smply set v(0) = Wr(0) + I(0) Togofromav-model to an r-model, we frst defne some useful notaton: 1 N W s the null space of W, that s, the subspace of all vectors that W maps to 0 P N s the projecton operator nto N W N W s the subspace perpendcular to N W Ths s the subspace spanned by the rows of W P N s the projecton operator nto N W R W s the range of W, that s, the subspace of vectors that can be wrtten Wx for some x Ths s the subspace spanned by the columns of W P R s the projecton operator nto R W R W s the subspace perpendcular to R W, also called the left null space P R s the projecton operator nto R W For any vector x, wedefnex N P N x, x N P N x, x R P R x, x R P R x We rely on the fact that x = x N + x N = x R + x R 1 If W s normal, the egenvectors are orthogonal, so the null space s precsely the space orthogonal to the range: P N = P R and P N = P R However, f W s nonnormal, then vectors orthogonal to the null space can be mapped nto the null space; the range always has the dmenson of the full space mnus the dmenson of the null space, but t need not be orthogonal to the null space

4 28 K Mller and F Fumarola Gven a v-model, the equaton v(0) = Wr(0) + I(0) has a soluton f and only f v(0) I(0) R W, whch s true f and only f v R (0) I R (0) = 0, 2 so we must choose I R (0) = v R (0) (9) Lettng D R be the dmenson of R W and D N the dmenson of N W,the fundamental theorem of lnear algebra states that D R + D N = D SoI R (0) has dmenson D N Ths leaves unspecfed I R (0), whch has dmenson D R To solve for r N (0), wenotethattheequatonv = Wr + I can equvalently be wrtten v = Wr N + I (because Wr N = 0, so Wr = Wr N ) That s, knowledge of v specfes only r N WedefneW 1 to be the Moore-Penrose pseudo-nverse of W Ths s the matrx that gves the one-to-one mappng of R W nto N W that nverts the one-to-one mappng of N W to RW nduced by W, and that maps all vectors n R W to 03 The pseudo-nverse has the property that W 1 W = P N whle WW 1 = P R Then we can solve for r N (0) as r N (0) = W 1 (v(0) I(0)) = W 1 (v R (0) I R (0)) (10) Ths s a D R -dmensonal equaton for the 2D R -dmensonal set of unknowns {r N (0), I R (0)}, so t determnes D R of these parameters and leaves D R free For example, t could be solved by freely choosng I R (0) and then settng r N (0) = W 1 (v R (0) I R (0)), or by freely choosng r N (0) and then settng I R (0) = v R (0) Wr N (0) Equatons 10 and 9 together ensure the equalty v(0) = Wr(0) + I(0) Applyng W to both sdes of equaton 10 yelds v R (0) = Wr N (0) + I R (0) = Wr(0) + I R (0) Ths states that the equalty holds wthn the range of W; 2 Note that the condton v I R W, meanng that v = Wr + I can be solved, s true for all tme f t s true n the ntal condton We compute: d(v I) τ = v + Ĩ + W f (v) τ di (6) = v + I + W f (v) (7) Applyng P R to equaton 7 and notng that P R W = 0, we fnd τ d(v R I R ) = (v R I R ) (8) If v(0) I(0) R W,thenv R (0) I R (0) = 0, and hence v R I R = 0 at all subsequent tmes so v I R W at all subsequent tmes Note also that for any ntal condtons, the condton v(t) I(t) R W s true asymptotcally as t 3 If the sngular value decomposton of a matrx M s M = USV,whereS s the dagonal matrx of sngular values and U and V are untary matrces, then ts pseudonverse s M 1 = V SU,where S s the pseudonverse of S, obtaned by nvertng all nonzero sngular values n S

5 Mathematcal Equvalence of Frng Rate Models 29 orthogonal to the range of W, wehavep R Wr = 0andv R (0) = I R (0) Together, these yeld v(0) = Wr(0) + I(0) Fnally, we can freely choose r N (0), whch has no effect on the equaton v(0) = Wr(0) + I(0) r N (0) has D N dmensons, so we have freely chosen D R + D N = D dmensons n fndng an r-model that s equvalent to the v- model That s, we have found a D-dmensonal subspace of such r-models those that satsfy v(0) = Wr(0) + I(0) To summarze, we have establshed the equvalence between r-models and v-models For each fxed choce of W, τ,andĩ(t),anr-model s specfed by {r(0), I(0)} and equaton 2, whle a v-model s specfed by v(0) and equaton 1 The equvalence s establshed by settng v(0) = Wr(0) + I(0), whch yelds a D-dmensonal subspace of equvalent r-models for a gven v-model Under ths equvalence, v obeys equaton 1, r obeys equaton 2, and the two are related at all tmes by v = Wr + I, wth τ di = I + ĨTogo from an r-model to ts equvalent v-model, we smply set v(0) = Wr(0) + I(0)Togofromav-model to one of ts equvalent r-models, we set I R (0) = v R (0), freely choose r N (0), and freely choose {r N (0), I R (0)} from the D R - dmensonal subspace of such choces that satsfy r N (0) = W 1 (v R (0) I R (0)), where W 1 s the pseudonverse of W Fnally, note that equaton 2 can be wrtten τ dr = r + f(v) That s, f we regard v as a voltage and f (v) as a frng rate, as suggested by the dervaton n the appendx, then r s a low-pass-fltered verson of the frng rate, just as I s a low-pass-fltered verson of the nput Ĩ Appendx: Smple Dervaton of the v-equaton As an example of an unsophstcated and heurstc dervaton of these equatons (more sophstcated dervatons can be found n the references n the man text), the v-equaton can be derved as follows We start wth the equaton for the membrane voltage of the th neuron: C dv = j g j (E j v ), (A1) where C s the capactance of the th neuron and g j s the jth conductance onto the neuron, wth reversal potental E j We assume that the g j s are composed of an ntrnsc conductance, g L, wth reversal potental EL ; extrnsc nput g ext wth reversal potental E ext ; and wthn-network synaptc conductances, wth g j representng nput from neuron j wth reversal potental Ẽ j Dvdng by k g k and defnng τ (t) = C / k g k gves E ext τ (t) dv = v + gl EL g L + j + k g k g jẽj (A2)

6 30 K Mller and F Fumarola We now make a number of further smplfyng assumptons We assume that g j s proportonal to the frng rate r j of neuron j, wth proportonalty constant W j 0: g j = W j r j Ths gnores synaptc tme courses, among other thngs We assume that r j s gven by the statc nonlnearty r j = f (v j ) (see Mller & Troyer, 2002; Hansel & van Vreeswjk, 2002; Prebe, Mechler, Carandn, & Ferster, 2004, for such a relatonshp between frng rate and voltage averaged over a few tens of mllseconds) We assume synapses are ether exctatory wth reversal potental E E or nhbtory wth reversal potental E I, and lnearly transform the unts of voltage so that E E = 1and E I = 1 We defne W j = W j E j Ths s now a synaptc weght that s postve for exctatory synapses and negatve for nhbtory synapses We defne Ĩ g L EL E ext and defne g g L Ths yelds the conductancebased rate equaton, τ (t) dv = v + Ĩ + j W j f (v j ) g + k W k f (v k ), (A3) wth τ (t) = C / ( g + k W k f (v k )) Fnally, we assume that the total conductance, represented by the denomnator n the last term of equaton A3, can be taken to be constant, for example, f g L s much larger than synaptc and external conductances or f nputs tend to be push-pull, wth wthdrawal of some nputs compensatng for addton of others We absorb the constant denomnator nto the defntons of Ĩ and W j and note that ths also mples that τ s constant, to arrve fnally at the v-equaton: τ dv = v + j W j f (v j ) + Ĩ (A4) Acknowledgments Ths work was supported by R01-EY11001 from the Natonal Eye Insttute and by the Gatsby Chartable Foundaton through the Gatsby Intatve n Bran Crcutry at Columba Unversty References Avel, Y, & Gerstner, W (2006) From spkng neurons to rate models: A cascade model as an approxmaton to spkng neuron models wth refractorness Phys Rev E, 73, Beer, R D (2006) Parameter space structure of contnuous-tme recurrent neural networks Neural Comput, 18,

7 Mathematcal Equvalence of Frng Rate Models 31 Dayan, P, & Abbott, L F (2001) Theoretcal neuroscence Cambrdge, MA: MIT Press Ermentrout, B (1994) Reducton of conductance based models wth slow synapses to neural nets Neural Comput, 6, Ermentrout, G B, & Terman, D H (2010) Mathematcal foundatons of neuroscence New York: Sprnger Gerstner, W, & Kstler, W (2002) Spkng neuron models Cambrdge:Cambrdge Unversty Press Hansel, D, & van Vreeswjk, C (2002) How nose contrbutes to contrast nvarance of orentaton tunng n cat vsual cortex J Neurosc, 22, La Camera, G, Rauch, A, Luscher, H R, Senn, W, & Fus, S (2004) Mnmal models of adapted neuronal response to n vvo lke nput currents Neural Comput, 16, Matta, M, & Del Gudce, P (2002) Populaton dynamcs of nteractng spkng neurons PhysRevE, 66, Mller, K D, & Troyer, T W (2002) Neural nose can explan expansve, power-law nonlneartes n neural response functons J Neurophysol, 87, Ostojc, S, & Brunel, N (2011) From spkng neuron models to lnear-nonlnear models PLoS Comput Bol, 7, e Prebe, N, Mechler, F, Carandn, M, & Ferster, D (2004) The contrbuton of spke threshold to the dchotomy of cortcal smple and complex cells Nat Neurosc, 7(10), Shrk, O, & Hansel, D, & Sompolnsky, H (2003) Rate models for conductancebased cortcal neuronal networks Neural Comput, 15, Wlson, H R, & Cowan, J D (1972) Exctatory and nhbtory nteractons n localzed populatons of model neurons Bol Cybern, 12, 1 24 Receved July 6, 2011; accepted July 10, 2011