Spike timing dependent plasticity - STDP

Transcription

1 Spike timing dependent plasticity - STDP Post before Pre: LTD + ms Pre before Post: LTP - ms Markram et. al. 997

2 Spike Timing Dependent Plasticity: Temporal Hebbian Learning Synaptic cange % Pre t Pre t Post Post Pre t Pre Post t Post Pre precedes Post: Long-term Potentiation Pre follows Post: Long-term Depression Weigt-cange curve (Bi&Poo, 2)

3 Macine Learning C lassical C onditioning Synaptic Plasticity Dynamic Prog. (Bellman Eq.) Monte Carlo Control SARSA Q-Learning Overview over different metods Anticipatory Control of Actions and Prediction of Values REINFORCEMENT LEARNING TD( λ ) often λ= δ -R ule Eligibility Traces Actor/Critic tecnical & Basal Gangl. Correlation based Control (non-evaluative) ISO-Control = eample based TD() = Rescorla/ Wagner TD() Neur.TD-form alism Neur.TD-Models ( Critic ) EVALUATIVE FEEDBACK (Rewards) You are ere! supervised L. = Differential Hebb-Rule ( slow ) ISO-Learning Neuronal Reward Systems (Basal Ganglia) UN-SUPERVISED LEARNING Hebb-Rule Correlation of Signals Differential Hebb-Rule ( fast ) STDP-Models biopysical & network ISO-Model of STDP Biopys. of Syn. Plasticity Dopamine Glutamate correlation based NON-EVALUATIVE FEEDBACK (Correlations) LTP (LTD=anti) STDP

4 History of te Concept of Temporally Asymmetrical Learning: Classical Conditioning I. Pawlow

5

6 History of te Concept of Temporally Asymmetrical Learning: Classical Conditioning Correlating two stimuli wic are sifted wit respect to eac oter in time. Pavlov s Dog: Bell comes earlier tan Food Tis requires to remember te stimuli in te system. Eligibility Trace: A synapse remains eligible for modification for some time after it was active (Hull 938, ten a still abstract concept). I. Pawlow

7 Classical Conditioning: Eligibility Traces Conditioned Stimulus (Bell) X Stimulus Trace E ω + ω Σ Σ Response Unconditioned Stimulus (Food) ω = Te first stimulus needs to be remembered in te system

8 History of te Concept of Temporally Asymmetrical Learning: Classical Conditioning Eligibility Traces Note: Tere are vastly different time-scales for (Pavlov s) beavioural eperiments: Typically up to 4 seconds as compared to STDP at neurons: Typically 4-6 milliseconds (ma.) I. Pawlow

9 Macine Learning C lassical C onditioning Synaptic Plasticity Dynamic Prog. (Bellman Eq.) Monte Carlo Control SARSA Q-Learning Overview over different metods Anticipatory Control of Actions and Prediction of Values REINFORCEMENT LEARNING TD( λ ) often λ= δ -R ule Eligibility Traces Actor/Critic tecnical & Basal Gangl. Correlation based Control (non-evaluative) ISO-Control = eample based supervised L. TD() = Rescorla/ Wagner TD() Neur.TD-form alism Neur.TD-Models ( Critic ) EVALUATIVE FEEDBACK (Rewards) = Matematical formulation of learning rules is ISO-Learning UN-SUPERVISED LEARNING similar but Hebb-Rule time-scales Differential Hebb-Rule ( slow ) Neuronal Reward Systems (Basal Ganglia) Correlation of Signals are muc different. Differential Hebb-Rule ( fast ) STDP-Models biopysical & network ISO-Model of STDP Biopys. of Syn. Plasticity Dopamine Glutamate correlation based NON-EVALUATIVE FEEDBACK (Correlations) LTP (LTD=anti) STDP

10 Differential Hebb Learning Rule Simpler Notation = Input u = Traced Input d dt ω ( t) = µ u ( t) V (t) y ( i i Early: Bell X i u i ω Σ V Late: Food X u

11 Defining te Trace In general tere are many ways to do tis, but usually one cooses a trace tat looks biologically realistic and allows for some analytical calculations, too. (t) = n k (t) tõ t< EPSP-like functions: α-function: Dampened Sine wave: (t) = te àat k (t) = b sin(bt) e àat k Sows an oscillation. Double ep.: (t) = î (e àat à e àbt ) k Tis one is most easy to andle analytically and, tus, often used.

12 Defining te Traced Input u Convolution used to define te traced input, Correlation used to calculate weigt growt (see below). ) ( ) ( ) ( ) ( ) ( ) ( ) ( f g g f du u g u f = = = u ) ( ) ( ) ( ) ( ) ( ) ( ) ( g f f g du u g u f =/ = = w

13 Defining te Traced Input u u ( ) = f ( u) g( u) du = f ( ) g( ) = g( ) f ( ) Specifically (we are dealing wit causal functions!): u(t) = s (ü)(t à ü)dü If is a spike train (using te δ-function): (t) = P j= M î(tj ) Ten: u(t) = P j= M (t à tj ) For eample: (t) = î() (t) = î(t) u(t) = (t) u(t) = (t à T)

14 Differential Hebb Rules Te Basic Rule General: Two inputs only. Tus we get for te output: v = w u + w u ISO-Learning Te basic rule: ISO-Learning X ω Σ ω v v One weigt uncanging: w ==const. ISO rule dw dt = ö u v Same for all inputs. Isotropic Sequence Order Lng. (as we can also allow w to cange!)

15 Differential Hebb Rules More rules (but wy?) ICO - Learning ICO-Learning ISO3 - Learning ISO3-Learning u X ω Σ ω v X ω Σ ω v v ICO dw dt = ö u u Input correlation Learning (as we take te derivative of te uncanging input u ) ISO3 r r > dw = ö u v R k dt R Tree factor learning k Te denotes tat we are only using positive contributions

16 Stability Analysis Inputs dw = öu dt v X X X u ω Σ V 4w (t) = s dw (t) dt dt X u AC CC 4w (t) = 4w AC (t) + 4wCC (t)

17 Stability Analysis 4w (t) = 4w AC Undesired contribution (t) + 4wCC (t) Desired contribution Some problems wit tese differential equations: 4w (t) = s dw (t) dt dt ) As we are integrating to strictly we need to assume tat tere is no second pulse pair coming in ever. 2) Furtermore we sould assume tat w (ence µ small) or we get second order influences, too.

18 Stability Analysis (ISO) Under tese assumptions we can calculate w AC and w CC to find out weter te rules are stable or not. In general we assume two inputs: (t) = î(t) and (t) = î(t à T) Inputs X T X and get for ISO: dw = öu dt v 4w CC = w s (t) (t à T)dt = w 2û aàb a+b (t) à 4w AC = w e s (t) (tàt)dt à á à = w e 2 2 () à á = ISO is (only) asymptotically stable for t

19 Stability Analysis for pulse pair inputs (ISO) w Setting =.2. µ=.2 w Single pairing relaation beavior.8.6 µ=. Notice te AC contribution.4 t time [step] Te remaining upward drift is only due to te AC term influence (Instable!) Tis sows tat early arrival of a new pulse pair migt easily fall into a not fully relaed system. (Instable!)

20 Learning Window (weigt cange curve) ISO: Weigt cange curve.2 Compare to STDP. w T Te weigt cange curve plots w in dependence on te pulse pairing distance T in steps, were we define T> if te signal arrives before and T< else.

21 Stability Analysis: Compare ISO wit ICO Te basic rule: ISO-Learning ISO-Learning ICO - Learning ICO-Learning X ω Σ ω v v u X ω Σ ω v ISO rule dw dt = ö u v ICO dw dt = ö u u Notice te difference Input correlation Learning (as we take te derivative of te uncanging input u )

22 Stability Analysis: ICO 4w CC = w s (t) (t à T)dt = w 2û 4w AC ñ w aàb a+b (t) ω = µ=.2 µ=. ISO Single pulse pair (no more AC term in ICO). ICO ICO: Weigt cange curve (same as for ISO).2 t time [step] w T

23 Stability Analysis: More comparisons Te basic rule: ISO-Learning ISO-Learning ICO - Learning ICO-Learning X ω Σ ω v v u X ω Σ ω v ISO rule dw dt = ö u v ICO dw dt = ö u u Conjoint learning-controlsignal (same for all inputs!) Single input as designated learning-control-signal. Makes ICO a eterosynaptic rule of questionable biological realism.

24 Stability Analysis: More comparisons Tis difference is especially visible wen wanting to symmetrize te rules (bot weigts can cange!). ISO-Sym One control signal! ω ω ω X ω i X Σ T=8 v time [steps] v 6 X ω ω T=5 d/dt ω X ICO-Sym Two control signals! Σ time [steps] v

25 Te Effects of Symmetry Inputs X T X Synapse w grows because is before. Synapse w srinks because is after ω T= time [steps] ICO-sym is truly symmetrical, but needs two control signals. ISO-sym beaves in a difficult and unstable oscillatory way.

26 ISO3: uses like ISO a single learning-control-signal ISO3 r dw = ö u v R k dt ISO3 - Learning ISO3-Learning X ω R Σ ω r > v v Idea: Te system sould learn ONLY at tat moment in time wen tere was a relevant event r! We use a sorter trace for r, as it sould remain rater restricted in time. Same filter function but parameters a r and b r. We also define T r as te interval between and r. Many times T r =T, ence r occurs togeter wit.

27 Stability Analysis: ISO3 4w CC = w R (t) (t à T) r(t à T r )dt 4w AC = w R (t) (t) r(t à T r )dt Observations: ) Cannot be solved anymore! 2) AC term is generally NOT equal to zero. 3) Not even asymptotic convergence can be generally assured. So wat ave we gained? One can sow tat for T r =T te AC term vanises if v as its maimum at T.

28 T Stability Analysis: ISO3, grapical proof u ' u u v (t) = u (t); t < T as as not yet appened AC r Maimum at T CC lim t!t à v (t) = Contributions of AC and CC grapically depicted If we restrict learning to te moment wen occurs ten we do not ave any AC contribution.!! A questionable assumption: argma(u ) = T!!

29 Stability Analysis: ISO3 ω No more upwards drift for ISO3 = ISO ISO3 µ= -4 µ= tim e [step] w ISO.2. Single pulse pair (ISO3 is stable and relaes instantaneously). ISO3 t Weigt cange curve (no more STDP!) ω -5 5

30 A General Problem: T is usually unknown and variable Introducing a filter bank: (eample ISO) u N u X N u N ω N u u X ω ω Σ v v Spreading out te earlier input over time! Remember: A questionable assumption: argma(u ) = T

31 Stability Analysis: ISO3 wit a filter bank Wit a filter bank we get for te output: v = w u + P Original Rule was: j= k dw dt = ö u v R Single weigts develop now as: ö 4w k = R w u k u R + CC R u k P N w j j= Wit delta-function inputs at t= and t=t we get: AC (uj ) R N w j uj ð P ñ u 4w k ö = w u ()u k (T) + w j uj (T) k (T) j {z } It is possible to prove tat as a consequence of te learning! tis term becomes zero