Research Article A Radial Basis Function Spike Model for Indirect Learning via Integrate-and-Fire Sampling and Reconstruction Techniques

Transcription

1 Advances in Arificial Neural Sysems Volume 1, Aricle ID , 16 pages doi:1.1155/1/ Research Aricle A Radial Basis Funcion Spike Model for Indirec Learning via Inegrae-and-Fire Sampling and Reconsrucion Techniques X. Zhang, 1 G. Foderaro, 1 C. Henriquez, A. M. J. VanDongen, 3 and S. Ferrari 1 1 Laboraory for Inelligen Sysems and Conrol LISC, Deparmen of Mechanical Engineering and Maerials Science, Duke Universiy, Durham, NC 778, USA Deparmen of Biomedical Engineering and Deparmen of Compuer Science, Duke Universiy Durham, NC 778, USA 3 Program in Neuroscience & Behavioral Disorders, Duke-NUS Graduae Medical School, Singapore, Singapore Correspondence should be addressed o S. Ferrari, sferrari@duke.edu Received 17 February 1; Acceped 1 May 1 Academic Edior: Olivier Basien Copyrigh 1 X. Zhang e al. This is an open access aricle disribued under he Creaive Commons Aribuion License, which permis unresriced use, disribuion, and reproducion in any medium, provided he original work is properly cied. This paper presens a deerminisic and adapive spike model derived from radial basis funcions and a leaky inegrae-andfire sampler developed for raining spiking neural neworks wihou direc weigh manipulaion. Several algorihms have been proposed for raining spiking neural neworks hrough biologically-plausible learning mechanisms, such as spike-imingdependen synapic plasiciy and Hebbian plasiciy. These algorihms ypically rely on he abiliy o updae he synapic srenghs, or weighs, direcly, hrough a weigh updae rule in which he weigh incremen can be decided and implemened based on he raining equaions. However, in several poenial applicaions of adapive spiking neural neworks, including neuroprosheic devices and CMOS/memrisor nanoscale neuromorphic chips, he weighs canno be manipulaed direcly and, insead, end o change over ime by virue of he pre- and possynapic neural aciviy. This paper presens an indirec learning mehod ha induces changes in he synapic weighs by modulaing spike-iming-dependen plasiciy by means of conrolled inpu spike rains. In place of he weighs, he algorihm manipulaes he inpu spike rains used o simulae he inpu neurons by deermining a sequence of spike imings ha minimize a desired objecive funcion and, indirecly, induce he desired synapic plasiciy in he nework. 1. Inroducion This paper presens a deerminisic and adapive spike model obained from radial basis funcions RBFs and a leaky inegrae-and-fire LIF sampler for he purpose of raining spiking neural neworks SNNs, wihou direcly manipulaing he synapic weighs. Spiking neural neworks are compuaional models of biological neurons comprised of sysems of differenial equaions ha can reproduce some of he spike paerns and dynamics observed in real neuronal neworks 1,. Recenly, SNNs have also been shown capable of simulaing sigmoidal arificial neural neworks ANNs and of solving small-dimensional nonlinear funcion approximaion problems hrough reinforcemen learning 3 5. Like all ANN learning echniques, exising SNN raining algorihms rely on he direc manipulaion of he synapic weighs 4 9. In oher words, he learning algorihms ypically include a weigh-updae rule by which he synapic weighs are updaed over several ieraions, based on he reinforcemen signal or nework performance. In many poenial SNN applicaions, including neuroprosheic devices, ligh-sensiive neuronal neworks grown in viro, and CMOS/memrisor nanoscale neuromorphic chips 1, he synapic weighs canno be updaed direcly by he learning algorihm. In several of hese applicaions, he objecive is o simulae a nework of biological or arificial spiking neurons o perform a complex funcion, such as processing an audiory signal or resoring a cogniive funcion. In neuroprosheic medical implans, for example, he arificial device may consis of a microelecrode array or inegraed circui ha simulaes biological neurons via spike rains. Therefore, he device is no capable of direcly modifying he synapic efficacies of he biological neurons, as do exising SNN raining algorihms, bu i is capable of

2 Advances in Arificial Neural Sysems simulaing a subse of neurons hrough conrolled pulses of elecrical curren. As anoher example, ligh-sensiive neuronal neworks grown in viro can be similarly simulaed hrough conrolled ligh paerns ha cause seleced neurons o fire a precise momens in ime, in an aemp o induce plasiciy, while heir oupu is being recorded in real ime using a mulielecrode array MEA 11. In his case, a digial compuer can be used o deermine he desired simulaion paerns for an in-viro neuronal nework wih random conneciviy, produced by culuring dissociaed corical neurons derived from embryonic day E18 ra brain 11, 1. As a resul, he culures may be used o verify biophysical models of he mechanisms by which biological neuronal neworks execue he conrol and sorage of informaion via emporal coding and learn o solve complex asks over ime. In hese neworks, he acual conneciviy and synapic plasiciies are ypically unknown and canno be manipulaed direcly as required by exising SNN learning algorihms. This paper presens a novel indirec learning approach and algorihm ha assume synapic weighs canno be updaed or manipulaed a any ime. The learning algorihm induces changes in he SNN weighs by modulaing spikeiming dependen plasiciy STDP hrough conrolled inpu spike rains. The algorihm adaps he inpu spike rains ha are used o simulae he inpu neurons of he SNN and, hus, are realizable hrough conrolled pulses of elecric volage or conrolled pulses of blue ligh. The main difficuly o be overcome in indirec learning is ha he algorihm aims a adaping pulse signals, such as square waves, in place of coninuous-valued weighs. While available in closed analyic form, hese signals ypically are represened by piece-wise coninuous, muli-valued or many-o-one, and nondiffereniable funcions ha are difficul o adap or updae using opimizaion or reinforcemen-learning algorihms. Furhermore, simulaion paerns ypically are generaed by spike models ha are sochasic, such as he Poisson spike model 5, Thus, even when he spike model is opimized, i does no allow for precise iming of pre- and pos-synapic firings, and as a resul, may induce undesirable changes in he synapic weighs. In his paper, a deerminisic spike model ha allows for precise iming of neuron firings is obained using adapive RBFs o model he characerisics of he spike paern hrough a coninuous and infiniely differeniable funcion ha also is one o one. The RBF model is combined wih an LIF sampling echnique originally developed in 16, 17 for he approximae reconsrucion of bandlimied funcions. I is shown ha, by his approach, he spike rains generaed by he LIF sampler display he precise characerisics specified by he RBF model. Furhermore, his deerminisic spike model can be opimized o modulae STDP hrough conrolled inpu spike rains ha bring abou he desired SNN weigh change wihou direc manipulaion. The indirec learning approach presened in his paper is applicable boh in supervised and unsupervised seings. In fac, when he desired SNN oupu is unknown, i can be replaced by a reinforcemen signal produced by a criic SNN, as shown by he adapive criic mehod reviewed in Secion 3. The paper is organized as follows. The model of spiking neural nework used o derive and demonsrae he raining equaions is presened in Secion. InSecion 3, an adapive criic approach is described o illusrae how he proposed indirec raining mehodology can be applied using reinforcemen learning, for example, o model or conrol a dynamical plan. The novel spike model and indirec raining mehodology are presened in Secion 4 and demonsraed on a benchmark problem involving a wo-node spiking neural nework. The generalized form of gradien equaions for indirec raining is presened in Secion 5 and demonsraed hrough a hree-node spiking neural nework. These gradien equaions show how he mehodology can be generalized o any spiking neural nework wih he characerisics described in he nex secion.. Spiking Neural Nework Model.1. Models of Neuron and Synapse. Various models of SNNs have been proposed in recen years, moivaed by biological sudies ha have shown complex spike paerns and dynamics o be an essenial componen of informaion processing and learning in he brain. The wo crucial consideraions involved in deermining a suiable SNN model are he range of neurocompuaional behaviors he model can reproduce, and is compuaional efficiency 18. As can be eced, he implemenaion efficiency ypically increases wih he number of feaures and behaviors ha can be accuraely reproduced 18, such ha each model offers a radeoff beween hese compeing objecives. One of he compuaional neuron models ha is mos biophysically accurae is he well-known Hodgkin-Huxley HH model. Due o is exremely low compuaional efficiency, however, using he HH model o simulae large neworks of neurons can be compuaionally prohibiive 18. Recenly, bifurcaion sudies have been used o reduce he HH dynamics from four o wo differenial equaions, referred o as he Izhikevich model, which are capable of reproducing a wide range of spiking paerns and behaviors wih much higher efficiency han he HH model 19. In 15, he auhors proposed an indirec raining mehod based on a Poison spike model and demonsraed i on a nework of Izhikevich neurons. In his work i was found ha he adapive criic archiecure described in Secion 3 could be implemened wihou modeling he neuron response in closed form. However, he effeciveness of he approach in 15 was limied in ha, due o he use of a sochasic Poison model, he Izhikevich SNN could no converge o he opimal conrol law. Therefore, in his paper, a new deerminisic spike model is proposed and implemened by deriving he raining equaions in closed form, using a leaky inegrae-and-fire LIF SNN. The LIF is he simples model of spiking neuron. I has he advanages ha i displays he highes compuaional efficiency and is amenable o mahemaical analysis 13, 14. The LIF membrane poenial, v, is governed by dv C m = I leak + I s + I inj, 1 d

3 Advances in Arificial Neural Sysems 3 where C m is he membrane capaciance, I leak is he curren due o he leak of he membrane, I s is he synapic inpu o he neuron, and I inj is he curren injeced o he neuron. The leak curren is defined as follow: I leak = C m τ m v V, where V is he resing poenial and τ m is he passivemembrane ime consan. τ m is relaed o he capaciance, C m, and he membrane resisance, R m, of he membrane poenial by τ m = R m C m. The response of he membrane poenial is obained by solving he differenial equaion 1 analyicallyforv, such ha I inj ρ v = V + e /τm e ρ/τm dρ, 3 C m where ρ is a dummy variable used for inegraion and is he ime a which he membrane poenial equals V. Whenever he membrane poenial, v,reaches a prescribed hreshold value, V h, he neuron will fire. A his poin, we have considered an isolaed neuron ha is simulaed by an exernal curren, I inj. When he LIF neuron 1isparofa larger nework, he inpu curren, referred o as he synapic curren, is generaed by he aciviy of presynapic neurons. Le he synapic curren, denoed by I s, be modeled as he sum of he currens of exciaory presynapic neurons and of inhibiory presynapic neurons, N E N I I s = C m a E,k S E,k + C m a I,k S I,k, 4 k=1 where he subscrip E denoes inpus from exciaory neurons, while he subscrip I denoes inpus from inhibiory neurons. The ampliudes, a E,k > anda I,k <, represen he change in poenial due o a single synapic even and depend on he weigh of he synapse. N E and N I are he numbers of exciaory and inhibiory curren synapses, respecively. S E,k and S I,k describe he exciaory and inhibiory synapic inpus as a series of inpu spikes o each synapse. The synapic inpus are modeled as a sum of insananeous impulse funcions, k=1 S E,k = δ E,k, 5 E,k S I,k = I,k δ I,k, 6 where E,k and I,k are he firing imes of he presynapic neurons and δ represens he Dirac dela funcion 1. In addiion o he above deerminisic properies, here are prevalen sochasic qualiies in biological neural neworks caused by effecs such as hermal noise and random variaions in neuroransmiers. For simpliciy, hese effecs are assumed negligible in his paper. However, he reader is referred o 15 for a echnique ha can be used o incorporae hese effecsin he abovesnnmodel. Δw pos pre Figure 1: STDP erm as a funcion of he ime delay beween he las spike of possynapic neuron and presynapic neuron... Model of Spike-Timing-Dependen Plasiciy STDP. A persisen learning mechanism known as spike-imingdependen plasiciy, recenly observed in biological neuronal neworks, is used in his paper o model synapic plasiciy in he LIF SNN. Synapic plasiciy refers o he mechanism by which he synapic efficacies or srenghs beween neurons are modified over ime, ypically as a resul of he neuronal aciviy. These changes are known o be driven in par by he correlaed aciviy of adjacenly conneced neurons. The direcions and magniudes of he changes are dependen on he relaive imings of he presynapic spike arrivals and possynapic firings. For simpliciy, in his paper, all changes in synapic srenghs are assumed o occur solely as a resul of he spike-iming-dependen plasiciy mechanism. The approach presened in 5 can be used o also incorporae a model of he Hebbian plasiciy. Spike-iming-dependen plasiciy STDP is known o modify he synapic srenghs according o he relaive iming of he oupu and inpu acion poenials, or spikes, of a paricular neuron. If he presynapic neuron fires shorly before he possynapic neuron, he srengh of he connecion will be increased. In conras, if he presynapic neuron fires afer he possynapic neuron, he srengh of he connecion will be decreased, as illusraed in Figure 1. Two consans and deermine he ranges of he presynapic o possynapic inerspike inervals over which synapic srenghening and weakening occur. Le he synapic efficacy or srengh be referred o as weigh and denoed by w. A is he maximum ampliude of he change of weigh due o a pair of spikes. pre and pos denoe firing imes of he presynapic neuron and he possynapic neuron, respecively. Then, for each se of neighboring spikes, he weigh adjusmen is given by, Δw = Ae pre posτ, 7 such ha he connecion weigh increases when he possynapic spike follows he presynapic spike and he weigh decreases when he opposie occurs. The ampliude of he adjusmen Δw lessens as he ime beween he spikes becomes larger, as is illusraed in Figure 1.

4 4 Advances in Arificial Neural Sysems Differen mehods can be used o idenify he spikes ha give rise o he STDP mechanism. In his paper, he nearesspike STDP model isadoped,bywhich,foreveryspike of he presynapic neuron, is neares possynapic spike is used o calculae he iming difference in 7, regardless of wheher i akes place before or afer he presynapic firing. Then, from 7, he weigh change due o one of he spikes of he ih neuron is modeled by he rule { A+ e Δi/τ+ if Δ i Δw i Δ i = 8 A e Δi/τ if Δ i >, where Δ i = 1,i argmin,k 1,i,k. Δ i is he firing ime difference beween he wo neurons, 1,i is he firing ime of he presynapic neuron, and,i is he firing ime of he possynapic neuron. The consans used in his paper are = = ms, A + =.6, and A = 1.5A +,wherea + and A deermine he maximum amouns of synapic modificaion ha occur when Δ is approximaely zero 1. From 8, he firing ime of he possynapic neuron is chosen such ha he absolue firing ime difference beween he presynapic neuron and he possynapic neuron, 1,i,k, is minimized. Therefore, he possynapic spike,k ha is neares o he presynapic firing ime 1,i is used o calculae he firing ime difference Δ i. The value of,k ha minimizes 1,i,k can be found by comparing he firing ime difference of he wo neurons. In order o obain an indirec learning mehod ha does no rely on he direc manipulaion of he synapic weighs, in his paper, i is assumed ha he weighs can only be modified according o he STDP rule 8. Also, he raining algorihm canno specify any of he erms in 8 direcly bu can only induce he firing imes in 8 by simulaing he inpu neurons, for example, hrough conrolled pulses of elecric volage or blue ligh. 3. Adapive Criic Archiecure for Indirec Reinforcemen Learning Many approaches have been proposed o rain SNNs by modifying he synapic weighs by means of reward signals or by updae rules inspired by reward-driven Hebbian learning and modulaion of STDP 5, 8. However, hese mehods are no applicable o approaches involving biological neuronal neworks e.g., neuronal culures grown in viro, because biological synapic efficacies canno be manipulaed direcly by he raining algorihm. Similarly, in nanoscale neuromorphic chips, he CMOS/memrisor synapic weighs canno be manipulaed direcly bu may be modified via conrolled programming volages ha induce a mechanism analogous o STDP, as demonsraed in 1. This paper presens an approach for raining an SNN hrough conrolled inpu pulse signals e.g., volages or blue ligh ha are generaed by a new deerminisic and adapive spike model presened in Secion The derivaion of he raining equaions used o adap he proposed spike model and subsequen pulse signals are demonsraed in Secion 4. using a wo-node LIF SNN. I is shown ha he synapic weigh can be updaed by he proposed indirec raining mehod and driven precisely o a desired value, by minimizing a funcion of he synapic weigh error. I follows ha, using a simple chain rule, he same raining equaions can be used o minimize a funcion of he decoded SNN oupu error, as is ypically required by supervised raining. In his secion, we show how he indirec raining mehod can be used for reinforcemen learning, using a criic SNN o modulae he STDP mechanisms in an acion SNN wih synapic weighs ha canno be manipulaed direcly e.g., a biological neuronal nework. Le NN a represen an acion SNN of LIF neurons, exhibiing STDP and modeled as described in Secion. I is assumed ha he weighs of he acion SNN canno be manipulaed direcly, and he only conrols available o he learning algorihm are he raining signals Figure, comprised of programming volages ha can be delivered using a square wave or he Rademacher funcion Secion 4.1. Now, le NN c denoe a criic SNN of feedforward fully conneced HH neurons. In his archiecure, schemaized in Figure, NN a is reaed as a biological nework and NN c is reaed as an arificial nework implemened on a compuer or inegraed circui. Thus, he synapic weighs of NN c, defined as w ij, are direcly adjusable, while he synapic efficacies of NN a can only be modified hrough a simulaed STDP mechanism which is modulaed by he inpu spikes from NN c. The algorihm presened in his secion rains he nework by changing he values of w ij inside NN c, which are assumed o be bounded by a posiive consan w max such ha w max w ij w max,foralli, j. Inhispaper,spikefrequenciesareusedocodeconinuous signals, and he leaky inegraor û = α e βk H k γ 1 k S i T is used o decode spike rains and conver hem ino coninuous signals as required according o he archiecure in Figure Where, α, β, andγ are user-specified consans, H is he Heaviside funcion, and S i T is he se of spiking imes of he oupu neuron. One advanage of he leaky inegraor decoder is is effeciveness of filering ineviable noise during he fligh conrol wihou influencing he speed of maching he coninuous value wih he arge funcion Adapive Criic Recurrence Relaions. Typically, adapive criic algorihms are used o updae he acor and criic weighs by compuing a synapic weigh incremen hrough an opimizaion-based algorihm, such as backpropagaion 3, page 359. Therefore a new approach is required in order o apply adapive criics o SNNs in which changes in he synapic weighs can occur only as a resul of preand possynapic neuronal aciviy. The raining approach presened in Secion4 can be combined wih he policy and value-ieraion procedures described in his secion o supervise simulaion paerns in he acion SNN such

5 Advances in Arificial Neural Sysems 5 ha he synapic srenghs, and subsequenly heir dynamic mappings, are adaped according o he STDP rule in 8. Suppose he acion SNN is being rained o conrol or model a dynamical sysem which obeys a nonlinear differenial equaion of he form ẋ = f x, u,, 11 where x X R n and u U R m denoe he dynamical sysem sae and conrol inpus, respecively, and ẋ denoes he derivaive of x wih respec o ime. The model of he dynamical sysem 11 may be unknown or imperfec and may be improved upon over ime by a sysem idenificaion ID algorihm. The macroscopic behavioral goals of he plan can be ressed by he value funcion or cos-o-go f 1 V π x k = Lx k, u k, x k +1, 1 k= where u k = πx k is he unknown conrol law or dynamic mapping o be learned by he acion SNN, NN a. Then, he cos-o-go in 1 can be used o represen he fuure performance of he acion nework and dynamical sysem, as is accrued from he presen, k, up o he final ime, f, if subjec o he presen conrol law π. The Lagrangian L represens insananeous behavioral goals as a funcion of x and u. Since he dynamic mapping π musbeadapedover ime hrough he learning algorihm and an accurae plan model may no be available, he cos-o-go ypically is unknown and is learned by he criic, NN c. Then, by Bellman s principle of opimaliy 4, a any ime k he cos-o-go can be minimized online wih respec o he presen conrol law, based on he known value of x k and predicions of x k +1.Thevalueofx k isassumed o be fully observable from he dynamical sysem a any presen ime k,andx k + 1 is prediced or esimaed from he approximaion of he sysem s dynamics 11, based on he presen values of he sae and he conrol. In 5, Howard showed ha if ieraive approximaions of he conrol law and opimal cos-o-go, denoed by π l and V l, respecively, are modified by a policy-improvemen rouine and a value-deerminaion operaion, respecively, hey evenually converge o heir opimal counerpars π and V. The policy-improvemen rouine saes ha, given a coso-go funcion V, π l corresponding o a conrol law π l,an improved conrol law π l+1 can be obained as follows: π l+1 x k =arg min u U {Lx k, u k, x k +1 +Vx k+1, π l }, 13 such ha Vx k, π l+1 Vx k, π l, for any x k. Furhermore, he sequence of funcions C = {π l l =, 1,,...} converges o he opimal conrol law π.the value-deerminaion operaion saes ha given a conrol law π, he cos-o-go can be updaed according o he following rule: V l+1 x k+1, π=lx k, u k, x k +1+V ell x k+1, π, 14 such ha he sequence of funcions V ={V l l =, 1,,...} converges o V. Then, a every ieraion cycle l of he adapive criic algorihm, he above policy and value-ieraion procedures can be used o in andem o supervise raining of he acor and criic SNN; ha is, u k NN a l, x k π l x k, 15 V π x k NN c l, x k, π V l x k, π, 16 respecively where l = Mk and M is a posiive consan chosen based on he desired frequency of he updaes. I can be seen ha, a l + 1, he desired mappings for NN a and NN c,providedby13 and14, respecively, are based on he acor and criic oupus a l,andonx k+1, which can be esimaed from an available dynamical sysem model 11. To accomplish 15 and16, wo error merics defined as a funcion of he acual decoded acor and criic oupus and of he desired acor and criic oupus 13and14mus be minimized by he chosen raining algorihm, as lained in he following subsecion. 3.. Acion and Criic Nework Training Algorihm. The acion and criic neworks implemened in Figure differ in ha while he criic nework can be rained by a convenional algorihm ha manipulaes he synapic weighs direcly e.g., see 5, 15, he acion nework mus be rained by inducing weigh changes indirecly hrough STDP 8, such ha is decoded oupu û will closely mach u in 15, for all x X. Since he weighs wihin NN a canno be adjused direcly, hey are manipulaed wih raining signals from NN c. Connecions are made beween q pairs of acion/criic neurons which serve as oupus in NN c and inpus for NN a Figure. To provide feedback signals o he criic, connecions are also creaed beween r pairs of neurons from NN a o NN c. The informaion flow hrough he neworks is illusraed in Figure. Since i is no known wha raining signals will produce he desired resuls in NN a, he criic mus also be rained o mach V π in 16 forallx X and u U. Boh updaes 15and16 can be formulaed as follows. Le SNN denoeanacionorcriicneworkcomprised of muliple spiking neurons and STDP synapses. Then, as shown in 15 and16, SNN mus represen a desired mapping beween wo vecors ξ R p and y R r : y k = g l ξ k SNNξ k, W l k ŷ k. 17 The desired mapping g l : R p R r can be assumed known and saionary during every ieraion cycle l and is updaed by he policy/value-ieraion rouine. Le W l k ={w ij k } denoe a marix conaining he SNN synapic weighs a ime k. Then, for proper values of W l, if he SNN is provided

6 6 Advances in Arificial Neural Sysems NN C Rae coded x Training signals Rae coding Dynamical sysem Sysem sae, x Conrol inpus, û NN A Hidden neurons Inpu neurons Oupu neurons Figure : Adapive criic archiecure comprised of a criic nework, NN c, and, an acion nework, NN a, o conrol a dynamical sysem. wih an observaion of he inpu, ξ l,encodedasn spike rain sequences Xi l = { i,κ : κ = 1,..., Ni l }, i = 1,..., n, overa ime inerval k, k+τ wih he firing imes a i,κ, he decoded oupu of he SNN mus mach he oupu y l = g l ξ l, where since τ k+1 k, Xi l can be used o encode insananeous values of ξ l a any k. Then, he chosen SNN raining algorihm mus modify he synapic weighs such ha a figure of meri represening disance is opimized a he oupu space of he mapping in 17. For he criic nework, NN c, he synapic weighs are adjused manually, using he reward-modulaed Hebbian approach presened in 5, 15. Because he arge funcion y is known for all values of ξ, he SNN oupu error y ŷis also known and can be used as a feedback o he criic in he form of an imiaed chemical reward, r, ha decays over ime wih ime consan τ c. The criic reward is modeled as r = b ŷ, y + r Δ e i τ c, 18 where for he criic y R. Therefore, he error funcion can be defined as bŷ, y = sgny ŷ, where sgn is he signum funcion. Thus, he value of b is posiive when he criic s oupu is oo low, and i is negaive when he criic s oupu is oo high. For he acion nework, NN a, he synapic weighs canno be manipulaed direcly; herefore he disance beween y and he decoded SNN s oupu ŷ is minimized wihrespec o he parameers of he RBF spike model described in he nex secion. From 17, we idenify ag wo ses of neurons referred o as inpu and oupu neurons Figure, where each neuron in he se provides he response for one elemen of ξ and y, respecively, in he form of a spike rain. I is assumed ha he se of inpu neurons can be induced o fire on command wih very high precision over a raining ime inerval k, k+τ, wih τ < T k+1 k. The inpu neurons firings could be implemened in pracice by using local programming volages wih conrolled pulse widh and heigh ha are easily realizable boh in CMOS/memrisor chips 1 and in neuronal neworks grown in viro 6. In order o induce STDP in a manner ha will improve he SNN represenaion of he mapping in 17, he programming volages are delivered based on an opimized spiking sequence S ι i ={ i,κ : κ = 1,..., N ι i }, i = 1,..., q, during he ih ime inerval of he raining algorihm, ι, ι+1. Exising spike models 13, 14 canno be used o generae S ι i because hey are sochasic and, as such, hey do no allow for precise iming of pre- and possynapic firings. As a resul, hey may induce undesirable changes in he synapic weighs by virue of he STDP rule 7, illusraed in Figure 1,which shows ha a small difference in he arrival ime of a pre- and possynapic spike can obain he opposie effec in erms of he weigh change Δw. Beside allowing for precise iming of neuron firings, he new deerminisic spike model presened in he nex secion can be updaed by he raining algorihm, by opimizing a corresponding coninuous signal modeled by a superposiion of Gaussian radial basis funcions RBFs, which is characerized by a se of adjusable parameers P = {w k, c k, β k k = 1,..., N}, wherew k is he heigh he RBF

7 Advances in Arificial Neural Sysems 7 s IF Inpu Iinj Neuron 1 Pre w1 Inegrae-and-fire Neuron Pos Figure 3: Model of wo-node LIF spiking neural nework. Oupu pulse, c k is he cener of RBF pulse, β k is he widh of he RBF pulse, and N is he number of RBF pulses. As shown in he nex secion, he coninuous RBF signal can be inegraed agains a suiable averaging funcion in a leaky inegraor and hen compared o a posiive hreshold, by means of a leaky inegrae-and-fire LIF sampler. Subsequenly, a precise pulse funcion wih desired widhs and inensiy can be generaed a a sequence of ime insans, i,κ, during he inerval ι, ι+1, ha correspond o he ceners of he RBF signal specified by P. Thus, a se of opimal RBF parameers P used o generae S ι i can be deermined by minimizing a measure of he disance beween he decoded SNN s oupu ŷ and he desired oupu y, compued by he policyimprovemen rouine 13. In he nex secion, he derivaion of gradien equaions ha can be used o minimize he SNN s oupu error y ŷ wih respec o P is illusraed by means of a wo-node LIF neural nework, wih STDP modeled as shown in Secion. Le P = {p lk } denoe a marix of RBF parameers o be adjused by he raining algorihm. I can be easily shown ha he minimizaion of an error funcion Ey ŷwihrespec o he elemens of P when y is a known consan can be accomplished using he gradiens ŷ/ p lk,orsomefuncion hereof, based on he chosen unconsrained minimizaion algorihm 7. As shown in 17, for a given inpu ξ l he only SNN variables o be opimized are he synapic weighs w ij. From he STDP rule 7, he weighs are a funcion of he parameers p lk, which deermine he inpu spike sequence or raining signal o he SNN Figure 3. By virue of he chain rule of differeniaion, i follows ha ŷ = ŷ w ij. 19 p lk w ij p lk Since ŷ/ w ij can be compued from he SNN model Secion, i follows ha if a raining algorihm can successfully modify he synapic weighs w ij using he proposed spike model, i also can successfully modify ŷ, simply by redefining he error funcion. In oher words, if a raining algorihm can successfully rain he synapic weighs of an SNN using he gradien w ij / p lk,ifollows ha i can also rain he SNN using he gradien ŷ/ p lk, since he componen ŷ/ w ij is known and given by he SNN equaions 1 5. Based on his propery, he novel spike model and indirec raining algorihm are illusraed by raining he synapic weigh of a wo-node LIF SNN o mee a desired value w, wihou direc manipulaion. 4. Indirec Training Mehodology Consider he wo-node LIF SNN schemaized in Figure 3, modeled using he approach described in Secion. s represens he inpu given o he LIF sampler, and S denoes he corresponding LIF sampler s oupu. Based on he approach presened in Secion 3, he SNN synapic srengh w 1, represening he synapic efficacy for a presynapic neuron labeled by i = 1 and a possynapic neuron labeled by i = canno be modified direcly by he raining algorihm bu can only change as a resul of he STDP mechanism described in Secion. Inplaceof conrolling w 1, he goal of he indirec raining algorihm is o deermine a spike rain ha can be used o simulae he inpu neuron i = 1 using I inj, hereby inducing i o spike, such ha he synapic weigh w 1 changes from an iniial random value o a desired value w. For his purpose, we inroduce a coninuous spike model comprised of a superposiion of Gaussian radial basis funcions RBFs: N s = w k β k c k, k=1 where w k deermines he heigh of he kh RBF, which will decide he consan curren inpu I inj. β k deermines he widh of he kh RBF and c k deermines he cener of he kh RBF, where k = 1,..., N and N is he number of RBFs. Thus, he se of RBF adjusable parameers is P ={w k, c k, β k k = 1,..., N}. A spike rain can be obained from he coninuous spike model by processing s using a suiable LIF sampler ha oupus a square pulse funcion S wih he same heighs, widhs, and ceners, as he RBF spike model. In his wo-node SNN model, here is no synapic curren sen o he firs neuron because here are no presynapic neurons conneced o i. Raher, he inpu for he firs neuron is he square pulse funcion provided by he oupu of he LIF sampler. Therefore, during one square pulse, he inpu, I inj, o neuron i = 1 can be viewed as a consan, which for our case is h, he heigh of he RBF. For a fixed hreshold and a consan inpu, he ime i akes for he firs neuron o fire or a spike o be generaed is T = τ m ln 1 θ hr m hr m >θ, 1 where θ is he poenial difference beween he spike generaing hreshold, V h, and he resing poenial, V ; ha is, θ = V h V. h is he heigh of inpu square pulse, R m is he resisance of he membrane, and τ m is he passivemembrane ime consan. Thus, from 1, he value of T can be conrolled by adjusing h. We allow he firs neuron o fire near he end of a square pulse by inpuing a spike sequence generaed by an RBF model wih a suiable widh and heigh. Then, he firing imes of he firs neuron can be easily adjused by alering he ceners of he RBF.

8 8 Advances in Arificial Neural Sysems S 1.5 Square pulse a V 5 V1 5 Membrane poenial of neuron Membrane poenial of neuron b c Figure 4: Membrane poenial ofhewo neuronsinhesnn in Figure 3. s RBF pulse a S Square pulse ,1 1, 1,3 1,4 1, b Figure 5: Deerminisic spike model signals. In his simple example, i is assumed ha he synapse is of he exciaory ype. Therefore, 4 canbe wrien as, k I s = C m a E δ 1,k τ d, k=k where τ d is a known consan ha represens he conducion delay of he synapic curren from neuron i = 1and 1,k are he firing imes of he firs neuron. k is he index of he spike of he firs neuron, which occurs afer he previous spike of he second neuron, and k is he index of he spike of he firs neuron, which provokes he firing ime of he second neuron. The only inpu o he second neuron is he synapic curren defined in. Therefore, subsiuing and ino 1 resuls in he following equaion for he membrane poenial, C m dv d = C k m v V + C m a E δ 1,k τ d. τ m k=k 3 Solving 3 for he membrane poenial of he second neuron provides he response of neuron i =, k v = V + a E k=k 1,k τ d τ m H 1,k τ d, 4 where H is he Heaviside funcion. Whenever he membrane poenial in 4 exceeds he hreshold V h, he second neuron fires, and he membrane poenial v is hen se insanly equal o V. I follows ha he firing imes of he second neuron,j can be wrien as a funcion of he firing imes of he firs neuron,,j = k 1,k + τ d H a E,j 1,k τ d τ k=k m H,j 1,k τ d V h V, 5 where j is he index of he firing imes. In addiion, he membrane poenials of he firs neuron and he second neuron can be calculaed using 4and1. An example of he membrane poenial ime hisory for he wo neurons is ploed in Figure 4, where he square pulse funcion S used o simulae he firs neuron is ploed along wih he neurons membrane poenials v 1 and v.the red dos denoe he firing imes and poenials for he wo neurons. I can be seen from Figure 4 ha, wih his inpu, he firs neuron fires five imes and he second neuron fires one ime. In his SNN, he inpu o he second neuron is given only by he synapic curren caused by he firing of he firs neuron. I can be seen ha only when he second and hird firing imes of he firs neuron are very close, he membrane poenial of he second neuron increases over he hreshold, causing i o fire, whereas he fourh and fifh

9 Advances in Arificial Neural Sysems 9 N1 N The spikes of he wo neurons 5 1,1 1, 1,3 1,4 1, ,3 +τ d 1,5 +τ d a The weigh change of he synapse ms b Figure 6: Opimized inpu spike rain and indirec weigh changes brough abou by STDP. S 6 RBF funcion Figure 7: Opimized RBFspikemodel forhesnn infigure 3. funcions is adoped, which convers any coninuous signal f ino a square wave funcion by inegraing i agains an averaging funcion u k,z. The inegraed resul is compared o a posiive hreshold and a negaive hreshold, such ha when eiher one of he wo hresholds is reached, a pulse is generaed a ime k = z. The value of he inegraor is hen rese and he process repeas. In his paper, he averaging funcion u k,z is chosen o be he onenial e α z X k,z, where X I is he characerisic funcion of I and α>isa consan ha models he leakage of he inegraor, as due o pracical implemenaions 8. Then, he LIF sampler firing condiion ha generaes he square wave or sequence of pulses is 6 Signal afer IF sampler k+1 ±θ = f e α k+1 d f, u k, 6 k s where k is he ime insan of he sample, k+1 is he nex ime insan of he sample, u k is he averaging funcion, θ is he hreshold value, and α is he leakage of he inegraor. The oupu of he LIF sampler can be ressed in erms of he ime insans a which he inegral reaches he hreshold, {,..., n }, and by he samples q 1,..., q n,definedas Figure 8: Square wave obained by he LIF sampler, for he RBF spike model in Figure 7. firing imes of he firs neuron are oo far apar o cause firing of he second neuron Deerminisic Spike Model. The deerminisic spike model consiss of a coninuous RBF model in he form combined wih an LIF sampler ha convers he RBF ino a square wave funcion. The LIF sampler developed in 8 for he approximae reconsrucion of bandlimied j 1 q j f xe αx j dx, for 1 j n. 7 j From 6, i follows ha q j = θ. By adjusing he parameers of he LIF sampler, i is possible o conver he RBF signal in o a square wave comprised of pulses wih he same widh, heigh, and ceners as he RBFs in. Thus, by using he RBF spike model in 6 wih suiable widhs and heighs, i is possible o induce he firs neuron o fire shorly before he end of each pulse of he square-pulse funcion also considering he refracory period of he neuron. For simpliciy, in his example, i is assumed ha he heighs w k and widhs β k are known posiive consans of equal magniudes for all k. Then, he ceners of he RBF comprise he se of adjusable parameers, P ={c k k = 1,..., N}, o be opimized. The same approach can be easily exended o a case where all of he RBF parameers are adaped.

10 1 Advances in Arificial Neural Sysems 3.6 The weigh change of he synapse.8 The error convergence of he weigh w1 3. e a b Figure 9: Indirec weigh changes brough abou by STDP and corresponding deviaion from w =.8, for he SNN in Figure 3 simulaed using he inpu spike rain in Figure 7. Neuron 1 Neuron Neuron 3 s Inpu I Synapse Synapse Oupu IF w 1 w 3 Pre Pos Pre Figure 1: Model of hree-node LIF spiking neural nework RBF funcion for neuron 1 S Signal afer IF sampler s Figure 1: Square wave obained from he LIF sampler, using he opimized RBFspike model in Figure Figure 11: Opimized RBFspikemodel forhesnn infigure 1. I can be shown using he LIF SNN model in Secion ha, under he aforemenioned assumpions, he firing imes of he firs neuron saisfy he consrains, 1,k c k + β, 1,k + Δ abs c k + β, 8 where Δ abs is an absolue refracory ime of he neuron. Then, he firing imes of he firs neuron can be wrien as a funcion of he RBF ceners, c k. By design, he firs neuron fires afer he same ime inerval, relaive o each square pulse Figure 4, due o he chosen RBF heighs and widhs. Then, he firing imes of he firs neuron can be wrien as, 1,k = c k β + T, 9 where 1,k denoes he firing imes of he firs neuron, c k are he ceners of he RBF, β is he consan widh of he RBF, and T is given by 1. The RBF spike model is demonsraed in Figure 5,where an example of RBF oupu obained from is ploed and, afer being fed o he LIF sampler, produces a corresponding square wave which can be used as conrolled pulse. I can be seen ha he ceners of he RBFs precisely deermine he imes a which a pulse occurs in he square wave and ha he square wave mainains he desired consan widh and magniude for all k. The nex subsecion illusraes how, by adaping he coninuous RBF spike model in, i is

11 Advances in Arificial Neural Sysems The weigh change of he synapse 1.4 The error convergence of he weigh w e w 1 w 3 e 1 e 3 a Figure 13: Indirec weigh changes brough abou by STDP and corresponding error funcions, for he SNN in Figure 1 simulaed using he inpu spike rain in Figure 1. b possible o minimize a desired error funcion and rain he synapic weigh of he SNN wihou direcly manipulaing i. 4.. Indirec Training Equaions. As lained in Secion., i is assumed ha he synapic weigh w 1 can only be modified by conrolling he aciviy of he SNN inpu neurons, wih i = 1, and ha i obeys he neares-spike STDP model in 8. Over ime, he synapic weigh can change repeaedly, and herefore is final value can be wrien as he sum of all incremenal changes ha have occurred over he ime inerval ι, ι+1, M w 1 = w 1+Δw i, 3 i=1 where, for he wo-neuron SNN in Figure 3, Δw 1,..., Δw M are due o M pairs of pre- and possynapic spikes ha occur a any ime ι, ι+1. Every weigh incremen Δw i is induced via STDP and, hus, obeys equaion 8. Since he firing imes in 8 depend on he ceners of he RBF inpu hrough 1 9, i follows ha every weigh incremen Δw i is a funcion of he RBF ceners. In his example, he consrains 8canbewrienas, β <c 1, c N + β < f, c k +β<c k+1, for k =1,..., N, 31 where, from Figure 5, N = 5. A raining-error funcion is defined in erms of he desired synapic weigh w and he acual value of he synapic weigh w 1. While differen forms of error funcion may be used, he chosen form deermines he complexiy of he derivaion of he raining gradiens. I was found ha he mos convenien form of raining-error funcion can be derived from he raio of he acual weigh over he desired weigh, e = w 1 w, 3 where w 1 is he weigh value a ime ι, ι+1, obained from all previous spikes. As lained in Secion 3, w can be viewed as he weigh ha leads o desired oupu ŷ for a given inpu ξ. I can be seen ha when w 1 is equal o w, e is equal o one. Plugging 3 ino 36, he weigh raio can be rewrien as, e = 1 w i=1 M w 1+Δw i, 33 and he error beween w 1 and w can be minimized by minimizing he naural logarihm of he raio 33: E lne = lnw +ln1+δw 1 + +ln1+δw M lnw. 34 As is ypical of all opimizaion problems, minimizing a quadraic form presens several advanages 9. Therefore, a any ime ι, ι+1, he indirec raining algorihm seeks o minimize he quadraic raining-error funcion: J E ={lne} ={lnw +ln1+δw 1 + +ln1+δw M lnw }. 35 Then, indirec raining can be formulaed as an unconsrained opimizaion problem in which J is o be minimized wih respec o he RBF ceners, or P ={c k : c k ι, ι+1 }. Any gradien-based numerical opimizaion algorihm can be uilized for his purpose. The analyical form of he gradien J/ c k depends on he form of he spike paerns. The following subsecion derives his gradien analyically for one example of spike paerns and demonsraes ha, using his gradien, he synapic weigh can be rained indirecly o mee he desired value w exacly. The same resuls were derived and demonsraed numerically for all oher possible

12 1 Advances in Arificial Neural Sysems spike paerns, bu hey are omied here for breviy. Anoher represenaion of error e is defined below o make he convergence of error more undersandable in figures, e = w 1 w. 36 where, denoes he absolue value Derivaion of Gradien Equaions for Indirec Training. Consider he case in which M = 5, w = 1.7, and a E > V h V >a E, which resuls in a wo-node SNN Figure 3 in which neuron i = 1 mus fire a leas wo imes in order for neuron i = o fire. An example of such a spike paern is shown in Figure 6, wheren 1 and N denoe spike rains of neuron i = 1andi =, respecively. In his case, he firs neuron fires five imes, and he second neuron fires wo imes. The firing ime of he firs neuron is conrolled by he RBF spike model described in Secion 4.1. InFigure 6, i,k denoes he kh firing ime of he ih neuron wih k I i, where I i ={k = 1,...,5} is an index se for he firing imes and i is he index labeling he neurons. In his example, he second neuron fires afer 1,3 and 1,5,because 1,, 1,3 and 1,4, 1,5 are close enough o cause he membrane poenial of he second neuron, v, o exceed he hreshold. Iniially, he synapic weigh is equal o 1.7. The weigh change is disconinuous due o he discree propery of spikes. As shown in Figure 6, he weigh increases hree imes a 1,3 +τ d, decreases a 1,4, and increases again a 1,5 +τ d.for his example, he synapic weigh changes in five incremens: Δw 1 = A + Δw = A + Δw 3 = A + Δw 4 = A Δw 5 = A + 1,1 1, τd 1,3 τd. 1,3 1,3 1,4 τd τd τd 37 When he equaions above are subsiued in 35, he raining-error funcion can be wrien as { 1,1 1,3 τd J = lnw +ln 1+A + +ln 1+A + +ln 1+A + +ln 1 A +ln 1+A + 1, τd 1,3 τd 1,3 1,4 lnw }. τd τd 38 Then, he gradiens of J wih respec o he RBF ceners are given by J c 1 = E = E c 1 J = E c c = E 1,1 = E 1, = E Δw 1 1+Δw 1 Δw 1+Δw J = E = E c 3 c 3 1,3 Δw = E 1 1+Δw 1 + Δw 1+Δw + Δw 4 1+Δw 4 J = E = E Δw = E 4 c 4 c 4 1,4 1+Δw 4 J = E = E =. c 5 c 5 1,5 39 Using he above gradiens, he opimal values of c 1,..., c 5 can be obained by minimizing J using a gradien-based numerical algorihm such as Newon s mehod. An example of indirec learning algorihm implemenaion is shown in Figure 6, where he RBF-adjusable parameers which, in his case, coincide wih he firing imes of neuron i = 1 are opimized o induce a change in he synapic weigh w 1 from an iniial value of 1.74, o a desired value w = 1.7. Anoher example, for which he gradien equaions are omied for breviy, is shown in Figures 7 9. In his case, he desired weigh value w =.8 isfar from he iniial weigh w 1 ι = 3.5, and hus N = 43 spikes are required o rain he SNN. The opimal RBF inpu and corresponding conrolled pulse used o simulae neuron i = 1 are shown in Figures 7 and 8, respecively.theime hisory of he weigh w 1 is ploed in Figure 9a, along wih he corresponding deviaion from w ploed in Figure 9b. 5. Generalized Form of Gradien Equaions for Indirec Training A more general form of he gradien equaions can be found by rewriing he response of he membrane poenial for neuron i = in4, as follows, v = V + b k=a a E 1,k τ d τ m H 1,k τ d, 4 where 1,k denoes he kh firing ime of neuron i = 1. Then, he gradiens of he raining objecive funcion 35 wih respec o he ceners of he RBF spike model for M = 5are given by he equaions in he Appendix, which are obained in erms of he wo funcions, g + 1,i, 1,j = A + g 1,i, 1,j = A 1,i 1,i 1,j 1,j τd, τd, where 1,i, 1,j are wo disinc firing imes of neuron i = 1. 41

13 Advances in Arificial Neural Sysems 13 The mehodology presened in Secion4 can also be exended o larger SNNs, alhough in his case i may be more convenien o compue he gradiens of he objecive funcion numerically. As an example, consider he hreeneuron SNN in Figure 1, modeled by he approach in Secion and wih wo synapic weighs w 1 and w 3 ha each obey he STDP mechanism in 8. Suppose ha he desired values of he synapic weighs are w1 = 4.1 and w3 =.8. Using he indirec learning mehod presened in his paper, and he gradien provided in he Appendix, a raining-error funcion formulaed in erms of he deviaions of w 1 andw 3 fromw1 and w3, respecively,canbe minimized wih respec o he parameers ceners P of he RBF spike model. Once he opimal RBF spike model, ploed in Figure 11, is fed o he LIF sampler, he conrolled pulse ploed in Figure 1 is obained and implemened via I inj. The conrolled pulse is hus used o simulae neuron i = 1a precise insans in ime ha corresponds o ceners of he opimal RBF spike model. By his approach, w 1 canbe made o converge o he desired value w1. The same mehod is implemened o simulae neuron i = o make he value of w 3 convergeow3. The ime hisories of he weigh values w 1 andw 3, obained by he indirec raining algorihm are ploed in Figure 13a. As is also shown by he corresponding raining errors, ploed in Figure 13b, he SNN weighs over ime converge o he desired values, ha is w1 = 4.1 andw3 =.8. These resuls demonsrae ha, even for larger SNNs, he indirec raining mehod presened in his paper is capable of modifying synapic weighs unil hey mee heir desired values, wihou direc manipulaion. Since his indirec raining algorihm only relies on modulaing he aciviy of he inpu neurons via conrolled inpu spike rains, i also has he poenial of being realizable in viro and in silico o rain biological neuronal neworks and CMOS/memrisor nanoscale chips, respecively. 6. Summary and Conclusion Recenly, several algorihms have been proposed for raining spiking neural neworks hrough biologically plausible learning mechanisms, such as spike-iming-dependen synapic plasiciy. These algorihms, however, rely on being able o modify he synapic weighs direcly or STDP. In oher words, hey minimize a desired objecive funcion wih respec o weigh incremens ha are assumed o be conrollable and, hus, are decided by a weigh updae rule, and hen are implemened direcly by he raining algorihm. In several poenial applicaions of spiking neural neworks, synapic weighs canno be manipulaed direcly bu change over ime by virue of pre- and possynapic neural aciviy. In hese applicaions, he aciviy of seleced inpu neurons can be conrolled via programming volages or pulses of blue ligh ha induce precise spiking of he inpu neurons, a precise momens in ime. This paper presens an indirec learning mehod ha induces changes in he synapic weighs by modulaing spikeiming-dependen plasiciy using conrolled inpu spike rains, in lieu of he weigh incremens. The key difficuly o be overcome is ha indirec learning seeks o adap a pulse signal, such as a square wave or Rademacher funcion, in place of coninuous-valued weighs. Pulse signals ha can be delivered o simulae inpu neurons and cause hem o spike, such as blue ligh paerns or programming volages, are represened by piece-wise coninuous, muli-valued or many-o-one, and nondiffereniable funcions ha are no well suied o numerical opimizaion. Furhermore, simulaion paerns ypically are generaed by spike models ha are sochasic, such as he Poisson spike model. Therefore, even when he spike model is opimized, i does no allow for precise iming of pre- and pos-synapic firings, and as a resul, may induce undesirable changes in he synapic weighs. This paper presens a deerminisic and adapive spike model derived from radial basis funcions and a leaky inegrae-and-fire sampler. This spike model can be easily opimized o deermine he sequence of spike imings ha minimizes a desired objecive funcion and, hen, used o simulae inpu neurons. The resuls demonsrae ha his mehodology is capable of inducing he desired synapic plasiciy in he nework and modify synapic weighs o mee heir desired values only by virue of conrolled inpu spike rains ha are realizable boh in biological neuronal neworks and in CMOS/memrisor nanoscale chips. Appendix The gradiens of he raining objecive funcion can be derived as follows. The membrane poenial of neuron is denoed by v i.j, where he membrane poenial can only be affeced by presynapic spikes ha occur wihin he ime inerval 1.i, 1,j. E c 1 = g + 1,1, 1,3 E if v 1,3 + τ d 1,3 1+g+ 1,1, >V h 1,3 g + 1,1, 1, E if v 1, + τ d 1, 1+g+ 1,1, >V h 1, g + 1,1, 1,4 E if v 1,4 + τ d 1,4 1+g+ 1,1, >V h 1,4 g + 1,1, 1,5 E if v 1,5 + τ d 1,5 1+g+ 1,1, >V h. 1,5

14 14 Advances in Arificial Neural Sysems E c = E c 3 = g + 1,, 1,3 E 1+g+ 1,, 1,3 g + 1,1, 1, E 1+g+ 1,1, 1, E E E g + 1,1, 1, 1+g+ 1,1, 1, + g + 1,1, 1, + 1+g+ 1,1, 1, g 1,, 1, g 1,, 1,4 g + 1,, 1,4 1+g+ 1,, 1,4 g 1,, 1,3 1+g 1,, 1,3 g 1,, 1,3 1+g 1,, 1,3 g 1,, 1,5 1+g 1,, 1,5 E g + 1,1, 1,3 g E + 1,, + 1,3 1+g+ 1,1, 1,3 1+g+ 1,, 1,3 g 1,3, 1,4 + 1+g 1,3, 1,4 E g + 1,1, 1,3 g + 1,, + 1,3 1+g+ 1,1, 1,3 1+g+ 1,, 1,3 g + 1,3, 1,5 E 1+g+ 1,3, 1,5 g 1,, 1,3 1+g 1,, 1,3 g + 1,3, 1,4 E 1+g+ 1,3, 1,4 E g + 1,1, 1,3 g + 1,, + 1,3 1+g+ 1,1, 1,3 1+g+ 1,, 1,3 + g 1,3, 1,4 g 1,3, + 1,5 1+g 1,3, 1,4 1+g 1,3, 1,5 if v 1,3 + τ d 1,3 >V h if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h c 3 > c 5 + c +τ d if v 1, + τ d 1, >V h, v 1,4 + τ d 3,4 >V h, c 3 < c 4 + c +τ d or v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 3 < c 5 + c +τ d if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 <V h if v 1,4 + τ d 1,4 >V h. if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 >V h, c 3 > c 4 c 5 τ d if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 >V h, c 3 < c 4 c 5 τ d if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 3 > c 5 + c +τ d or v 1,5 + τ d 1,5 >V h if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 3 < c 5 + c +τ d or v 1, + τ d 1, >V h, v 1,4 + τ d 3,4 >V h, c 3 < c 4 + c +τ d or v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 <V h if v 1, + τ d 1, >V h, v 1,4 + τ d 3,4 >V h, c 3 > c 4 + c +τ d or v 1,4 + τ d 1,4 >V h if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 <V h.

15 Advances in Arificial Neural Sysems 15 g 1,3, 1,4 E 1+g 1,3, 1,4 g + 1,4, 1,5 E 1+g+ 1,4, 1,5 if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 >V h, c 4 < c 3 + c 5 +τ d or v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 <V h if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 >V h, c 4 > c 3 + c 5 +τ d or v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 3 > c + c 5 +τ d or v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 3 < c + c 5 +τ d E g + 1,3, 1,4 g = E 1,4, + 1,5 if v c 1, + τ d 1, 4 1+g+ 1,3, 1,4 1+g 1,4, >V h, 1,5 v 1,4 + τ d 3,4 >V h, c 4 < c 3 c τ d g 1,4, 1,5 E if v 1, + τ d 1, 1+g 1,4, >V h, v 1,4 + τ d 3,4 >V h, 1,5 c 4 > c 3 c τ d g 1,, 1,4 E if v 1, + τ d 1, 1+g 1,, >V h, v 1,5 + τ d 3,5 <V h 1,4 g + 1,1, 1,4 g E + 1,, + 1,4 1+g+ 1,1, 1,4 1+g+ 1,, 1,4 g + 1,3, 1,4 g + 1,4, + 1,5 if v 1,4 + τ d,1,4 >V h 1+g+ 1,3, 1,4 1+g 1,4, 1,5 g + 1,4, 1,5 E if v 1,5 + τ d,1,5 >V h. 1+g+ 1,4, 1,5 if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 >V h, c 5 > c 4 c 3 τ d g + 1,4, 1,5 E if v 1,3 + τ d 1,3 1+g+ 1,4, >V h, 1,5 v 1,5 + τ d 4,5 >V h, c 5 < c 4 c 3 τ d g + 1,3, 1,5 g E + 1,4, + 1,5 if v 1, + τ d 1+g+ 1,3, 1,5 1+g+ 1,4, 1, >V h, 1,5 g + 1,4, 1,5 E 1+g+ 1,4, 1,5 E g 1,4, = E 1,5 c 1+g 1,4, 1,5 5 g 1,, 1,5 E 1+g 1,, 1,5 g 1,3, 1,5 E 1+g 1,3, 1,5 g 1,4, 1,5 E 1+g 1,4, 1,5 g + 1,1, 1,5 g E + 1,, + 1,5 1+g+ 1,1, 1,5 1+g+ 1,, 1,5 g + 1,3, 1,5 g + + 1,4, + 1,5 1+g+ 1,3, 1,5 1+g+ 1,4, 1,5 v 1,5 + τ d 3,5 >V h, c 5 < c 3 c τ d if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 >V h, c 5 > c 3 c τ d if v 1, + τ d 1, >V h, v 1,4 + τ d 3,4 >V h if v 1, + τ d 1, >V h, v 1,5 + τ d 3,5 <V h if v 1,3 + τ d 1,3 >V h, v 1,5 + τ d 4,5 <V h if v 1,4 + τ d 1,4 >V h if v 1,5 + τ d 1,5 >V h. A.1

16 16 Advances in Arificial Neural Sysems Acknowledgmen This work was suppored by he Naional Science Foundaion, under ECCS Gran References 1J.J.B.Jack,D.Nobel,andR.Tsien,Elecric Curren Flow in Exciable Cells, Oxford Universiy Press, Oxford, UK, 1s ediion, A.L.HodgkinandA.F.Huxley, Aquaniaivedescripion of membrane curren and is applicaion o conducion and exciaion in nerve, The Journal of Physiology, vol. 117, no. 4, pp , W. Maass, Noisy spiking neurons wih emporal coding have more compuaional power han sigmoidal neurons, Advances in Neural Informaion Processing Sysems, vol. 9, pp , C. M. A. Pennarz, Reinforcemen learning by Hebbian synapses wih adapive hresholds, Neuroscience, vol. 81, no., pp , S. Ferrari, B. Meha, G. Di Muro, A. M. J. VanDongen, and C. Henriquez, Biologically realizable reward-modulaed hebbian raining for spiking neural neworks, in Proceedings of he Inernaional Join Conference on Neural Neworks IJCNN 8, pp , Hong Kong, June 8. 6 R. Legensein, C. Naeger, and W. Maass, Wha can a neuron learn wih spike-iming-dependen plasiciy? Neural Compuaion, vol. 17, no. 11, pp , 5. 7 J. P. Pfiser, T. Toyoizumi, D. Barber, and W. Gersner, Opimal spike-iming-dependen plasiciy for precise acion poenial firing in supervised learning, Neural Compuaion, vol. 18, no. 6, pp , 6. 8 R. V. Florian, Reinforcemen learning hrough modulaion of spike-iming-dependen synapic plasiciy, Neural Compuaion, vol. 19, no. 6, pp , 7. 9 S. G. Wysoski, L. Benuskova, and N. Kasabov, Adapive learning procedure for a nework of spiking neurons and visual paern recogniion, in Proceedings of he 8h Inernaional Conference on Advanced Conceps for Inelligen Vision Sysems ACIVS 6, vol of Lecure Noes in Compuer Science, pp , Anwerp, Belgium, Sepember 6. 1S.H.Jo,T.Chang,I.Ebong,B.B.Bhadviya,P.Mazumder, and W. Lu, Nanoscale memrisor device as synapse in neuromorphic sysems, Nano Leers, vol. 1, no. 4, pp , A. M. VanDongen, Vandongen laboraory, hp:// 1 T. J. Van De Ven, H. M. A. VanDongen, and A. M. J. VanDongen, The nonkinase phorbol eser recepor α1- chimerin binds he NMDA recepor NRA subuni and regulaes dendriic spine densiy, Journal of Neuroscience, vol. 5, no. 41, pp , P. Dayan and L. F. Abbo, Theoreical Neuroscience: Compuaional and Mahemaical Modeling of Neural Sysems, MIT Press, Cambridge, Mass, USA, W. Gersner and W. Kisler, Spiking Neuron Models: Single Neurons, Populaions, Plasiciy, Cambridge Universiy Press, Cambridge, UK, G. Foderaro, C. Henriquez, and S. Ferrari, Indirec raining of a spiking neural nework for fligh conrol via spike-imingdependen synapic plasiciy, in Proceedings of he 49h IEEE Conference on Decision and Conrol CDC 1, pp , Alana, Ga, USA, December A. Aldroubi and K. Gröchenig, Nonuniform sampling and reconsrucion in shif-invarian spaces, SIAM Review, vol. 43, no. 4, pp , A. A. Lazar and L. T. Toh, A oepliz formulaion of a realime algorihm for ime decoding machines, in Proceedings of he Telecommunicaion Sysems, Modeling and Analysis Conference, E. M. Izhikevich, Which model o use for corical spiking neurons? IEEE Transacions on Neural Neworks, vol. 15, no. 5, pp , E. M. Izhikevich, Simple model of spiking neurons, IEEE Transacions on Neural Neworks, vol. 14, no. 6, pp , 3. A. N. Burki, A review of he inegrae-and-fire neuron model: I. Homogeneous synapic inpu, Biological Cyberneics, vol. 95, no. 1, pp. 1 19, 6. 1 S. Song, K. D. Miller, and L. F. Abbo, Compeiive Hebbian learning hrough spike-iming-dependen synapic plasiciy, Naure Neuroscience, vol. 3, no. 9, pp ,. P. Sjosrom, G. Turrigiano, and S. Nelson, Rae, iming, and cooperaiviy joinly deermine corical synapic plasiciy, Neuron, vol. 3, no. 6, pp , 1. 3 S. Ferrari and R. Sengel, Model-based adapive criic designs, in Learning and Approximae Dynamic Programming, J. Si, A. Baro, and W. Powell, Eds., John Wiley & Sons, 4. 4 R. E. Bellman, Dynamic Programming, Princeon Universiy Press, Princeon, NJ, USA, R. Howard, Dynamic Programming and Markov Processes, MIT Press, Cambridge, Mass, USA, A. M. J. VanDongen, J. Codina, J. Olae e al., Newly idenified brain poassium channels gaed by he guanine nucleoide binding proein Go, Science, vol. 4, no. 4884, pp , J. E. Dennis and R. B. Schnabel, Numerical Mehods for Unconsrained Opimizaion and Nonlinear Equaions, SIAM, Englewood Cliffs, NJ, USA, H. G. Feichinger, J. C. Príncipe, J. L. Romero, A. Singh Alvarado, and G. A. Velasco, Approximae reconsrucion of bandlimied funcions for he inegrae and fire sampler, Advances in Compuaional Mahemaics, vol. 36, no. 1, pp , 1. 9 R. F. Sengel, Opimal Conrol and Esimaion, DoverPublicaions, Inc., 1986.

17 Journal of Advances in Indusrial Engineering Mulimedia The Scienific World Journal Applied Compuaional Inelligence and Sof Compuing Inernaional Journal of Disribued Sensor Neworks Advances in Fuzzy Sysems Modelling & Simulaion in Engineering Submi your manuscrips a Journal of Compuer Neworks and Communicaions Advances in Arificial Inelligence Hindawi Publishing Corporaion Inernaional Journal of Biomedical Imaging Volume 14 Advances in Arificial Neural Sysems Inernaional Journal of Compuer Engineering Compuer Games Technology Advances in Advances in Sofware Engineering Inernaional Journal of Reconfigurable Compuing Roboics Compuaional Inelligence and Neuroscience Advances in Human-Compuer Ineracion Journal of Journal of Elecrical and Compuer Engineering