J. Marro Institute Carlos I, University of Granada. Net-Works 08, Pamplona, 9-11 June 2008

Transcription

1 J. Marro Institute Carlos I, University of Granada Net-Works 08, Pamplona, 9-11 June 2008

2 Networks? mathematical objects Königsberg bridges + Euler graphs (s. XVIII),... models of natural systems social and economic nets, www and Internet, food webs, metabolic networks, communication and transport nets,... models of interacting particles capture essence of cooperation, importance of topology of interactions, synchronization,... antecedents: mathematics, neuro- and computer science,..., physics neuro- and computer science Santiago Ramón y Cajal (Nobel 1906), Warren McCulloch and Walter Pitts (1943), Donald Hebb (1949), Shun-Ichi Amary (1972), John Hopfield (1982) ( 5,000 cites), Daniel Amit (1989),... good (biological, hardware,...) data for (statistical) physicists (Positron Emission Tomography, Functional Magnetic Resonance Imaging, Electrophysiological Recording, Magnetoencefalography,... Computers Clusters, Farms, Parallelization,...) 2

3 Amari Hopfield Attractor Network inhomogeneous connections, w ji j each node receives a signal: i w ji h i = j w ji σ i state at time t is characterized by node activities: σ {σ i } (e.g. σ i =±1) connection weights: w {w ij є R} (i,j = 1,,N ) 3

4 Amari Hopfield Attractor Network synaptic connection weights are choosen (e.g.) by the Hebb (synaptic learning) prescription: w ji = N 1 ν ξ iν ξ j ν ( N neurons; P sets of activity patterns ξ ν = {ξ iν = ±1}, ν = 1,,P ) node activities are stochastic variables, e.g., assuming neuron at i fires (changes state) if σ i ( t + 1 ) = sign { h i (t) θ i } in practice, make change with probability depending on an inverse temperature, β, namely, p = { 1 + exp [ 2β ( h i θ i ) ] } 1 4

5 Dynamic Attractors, Associative Memory

6 This picture lacks two essential facts: Activity of systems in nature does not stay in one memory irregular wandering among stored patterns, e.g., insects olfactory processes associated with heteroclinic paths in patterns space One needs irregular time behavior & instabilities! Connections constant but fast time changes (fluctuations on the 10-3 s scale, short-term plasticity, synaptic fatigue including depression and/or facilitation,...) these changes during operation may induce dynamic instabilities and even chaos 6

7 Many different context in which connections are not only homogeneous but with fast time changes! Weighted, and time dependent communication lines transport nets: connections differ in capacity, number of flights and passengers condensed matter: spin glasses and other systems with impurities, microscopic disorder and local fields, reaction diffusion systems,... diffusion of ions, chemical reactions, local rearrangements,... constantly vary the effective interactions between the units fluxes of different and varying intensities in food webs, ecological and metabolic nets agents interchange different amounts of information or money in the internet, www, economic and other social nets complex and varying patterns of synapses in the central nervous system and the brain

8 I shall assume that the observed time variation of connections is because the weights depend on the system activity due to two main general features of nodes: nodes are excitable elements, i.e., respond to perturbation if above threshold, and then show time lag (to next response) in which they behave as refractory to further excitation effective excitability would cause a non-equilibrium steady condition in a set of interacting elements which would, e.g., impede the damping by friction of signals in forest fires, the nervous system and autocatalytic reactions in surfaces nodes may be quiet certain nodes are more active, and some may even not be engaged at a given time in a given cooperative task. there is no need for a network to maintain all the nodes informed of the activity of all the others at all times and, in fact, it has often been reported non-uniform distribution of activity, e.g., associated to working memories 8

9 Implementing these features in a model network: SILENT NODES At each time, only the state of a fraction ρ of (e.g.) randomly chosen nodes will be updated: ρ 1: all the nodes are updated at once (parallel or Little updating, as in cellular automata); ρ 0: just one node at a time (sequential or Glauber updating); Note: this mimics an observable property of excitable elements and, in any case, why not study all possibilities between these two limits? Question: what are the consequences of having nodes that do not fire at a given time (even if they receive action above threshold), or of eventually turning off a fraction of nodes or communication lines

10 Implementing these features in a model network: EFFECTIVE EXCITABILITY OF NODES Replace the constant (e.g., Hebb s) weights for noisy ones: ω ij ω ij x j where x j is stochastic variable varying on time scale much smaller than one for nodes changes, so that one considers stationary distribution, either P1 x q x, 1 q x 1 or else j j j j P2 x x 1 x 1 j j j (local versus global order influence on the weights)

11 Implementing other interesting features in a model network: WIRING TOPOLOGY Let the topology matrix: ij ij 1 if link does exist ij, where ij, ki j ij 0 if link does not exist Therefore, each node feels an effective field: h, where depends on P x eff eff eff i j ij ij j ij j which allows consider different connectivity distributions, e.g.: pk k k1 k k2, k2 k p k k, k k, k for finite N 0 m

12 Instabilities stored patterns are attractors of dynamics but noise : destabilizes the attractors and, consequently: susceptibility to external stimulus increases an efficient search in attractors space is induced search may be chaotic for ρ ρ C, where slight variations of ρ may switch search from regular to irregular J. Marro, J. Torres, J. Cortes, J. Stat. Mech. P02017 (2008) J. Torres et al., TBP, arxiv: v1 [cond-mat.dis-nn] 12

13 Instabilities Φ = 1 Bifurcation diagram with evidence of chaos and periodic windows in MC simulations (P 2, M = 20 random patterns, N = 3600 nodes, Φ = 0.5, β = 20) stationary order = sum of squares of overlaps (α=m/n) J. Marro, J. Torres, J. Cortes, J. Stat. Mech. P02017 (2008) 13

14 Instabilities Lyapunov surface λ as a function of Φ (β=25) and β (Φ=0.01) complex landscape: λ > 0 (chaos) if region is black. (there is chaos also for ρ = 1) (P 2, M = 1 + anti-pattern) For Φ = 0.005, β = 50, M = 1, analytically. J. Marro, J. Torres, J. Cortes, J. Stat. Mech. P02017 (2008) 14

15 Instabilities overlap versus time (N = 1600, M = 3 uncorrelated patterns, Φ = 0.4, β = 20) convergence and stability of attractor in Fully fact, irregular anti-pattern (positive (zero Lyapunov overlap with others) exponent) behavior ρ = 0.08 < for ρ C ρ = Regular oscillation between 0.50 attractor > ρ C and its negative for ρ = 0.65 > ρ C Onset of chaos as ρ is incresed somewhat; ρ = 0.92 Rapid and ordered periodic oscillations between pattern and its antipattern (all nodes synchronized) J. Marro, J. Torres, J. Cortes, J. Stat. Mech. P02017 (2008)] 15

16 Phase (attractors) space Irregular switching among attractors states of attention, possible role of chaos # attractors visited (and correlation distance between them) increases with ρ mean firing rate versus t n and phase space trajectories (Φ = 0.05, N = 1600, β = 167, ρ C = 0.38, and three patterns, ξ μ μ =1,2,3.) J. Marro, J. Torres, J. Cortes, J. Stat. Mech. P02017 (2008)] 16

17 IEEE Computational Intelligence Magazine, August 2007, pp Walter J. Freeman (2003, 2005) interpreted EEG signals from cortical neural activity in animals and humans over the gamma frequency band (20-80 Hz) which shows sustained quasi-stable patterns of activities for several 100 ms, and extends over spatial scales comparable to the size of the hemisphere using dynamic systems theory. Freeman explicitly assumes: the brain s basal state is a high-dimensional chaotic attractor. Under the influence of external stimuli, the dynamics is constrained to a lower-dimensional wing. The system stays in this wing intermittently and produces an amplitude modulation (AM) activity pattern. Ultimately, the system jumps to another wing as it explores the complex attractor landscape. This is in support of the chaotic itinerancy theory (Tsuda 2001) that successfully interprets EEG measurements in terms of trajectories of a dynamical system, which intermittently visits the attractor ruins as it traverses across the landscape. Behavioral and Brain Sciences 24, (2001) 17

18 Phase (attractors) space: experiment vs model Stimuli of same intensity and duration (green and red). Stimulus destabilization in the absence of chaos. (Top involves falseneighbor method with d = 5 and τ = 20.) Response to odor stimuli of certain neurons in the locust antennal lobe; Mazor & Laurent, Neuron 48 (2005) Trajectories and t variation of mean firing rate during two MC simulations for N = 1600, β = 4, Φ = 0.45, ρ = 3/64 < ρ C, M =6. J. Torres et al., TBP, arxiv: v1 [cond-mat.dis-nn] 18

19 Criticality overlap (same cases as in main graph: ρ increases from top to bottom) power spectra is rather flat (but for large fequency peaks) when ρ = 0.35, 0.55 regular behavior while becomes power lawed during the irregular, critical cases ρ = 0.4 and 0.45 power spectra of the local field h i for N = 1600, P = 5 and Φ = 0.5 S. de Franciscis et al., report. 19

20 Criticality Typical Local field h i time series just before (ρ=0.38) and at criticality (ρ=0.40) for the same system S. de Franciscis et al., report. 20

21 Criticality Probability that h i = const. (except for small fluctuations) during time interval Δτ (data for all i and several time series) Evidence for criticality when 0.4 ρ 0.5 T = 0.01, Φ = 0.5, N = 1600, P = 10. P 2 with ζ ~ μ (m μ ) 2 S. de Franciscis et al., report. 21

22 Criticality Same in the critical case ρ = 0.45 as one varies P and N but not the load α = P / N The observed critical behavior is robust S. de Franciscis et al., report. 22

23 Criticality Typical Local field h i time series just before (ρ=0.35) and at and inside criticality (ρ=0.40, 0.45, 0.50 and 0.55) for the system which is analysed next. S. de Franciscis et al., report. 23

24 Criticality The correlation function, defined as the average (over i and t but only one system in this case) of [h i (t+δτ) h i (t)] 2 showing criticality at 0.4 and, to some extent, also at 0.45 and 0.5 P 2, ζ ~ μ m μ S. de Franciscis et al., report. 24

25 Topology M = 1, large N, any connectivity p(k) and mean-field assumptions (substitute ε ij by its mean value), phase transitions: from high-t disorder to associative memory at T c = k 2 / kn at T=0, for fatigue parameter Φ Φ 0 = 1 k +1 / k +1 [ > 0 monitors wiring in j (σ,φ) ], phase with periodic hopping among attractors at finite T, the hopping becomes irregular below Φ c (T) ~ Φ 0, and no irregular behavior is observed above T ~ 0.35 T c. Confirmed in MC simulations for M > 1 as hopping among different patterns Φ c () the more heterogeneous p(k) is, the lower the amount of fatigue, and the higher T c, needed to destabilize dynamics: S. Johnson, J. Marro, J. Torres, Europhys. Lett., to appear 25

26 Topology Edge of chaos transition curve between associative memory and irregular hopping showing critical fatigue (theories and MC data, T = 2/N, k = 20, = 2) for bimodal (top) and power law p(h) s with varying Δ and. 2 : minimum fatigue (Φ ~ 1, slight depression; Φ = 1 means no change) needed to destabilize. (Functional not structural connectivity in brain is power lawed with 2 ; Eguiluz et al. PRL 2005; same with many other systems in nature) S. Johnson, J. Marro, J. Torres, EPL, arxiv: v1 [cond-mat.dis-nn] 26

27 Topology System is showed one of the M stored patterns at equal time intervals: h i > h i + δ ξ ν i Ideally (standard model) it d remain there unless strong next stimulus estimate performance from time average of overlap between current state and input pattern: better the closer to the edge of chaos (and Φ ~ 1, 2), where both retrieval and destabilization are more efficient S. Johnson, J. Marro, J. Torres, TBP, arxiv: v1 [cond-mat.dis-nn] 27

28 Generalization & applications instead of making synapses noisy by ω ij ω ij x j assume state at t (σ,μ), σ = {σ i } and μ = {ω ij αξ iμ ξ jμ }, evolves in discrete time: P t+1 (σ,μ) = T μ [σ σ] T σ [μ μ] P t (σ,μ ) i.e., according to two stochastic processes driven by independent temperatures and by the energy cost ΔH of the corresponding change, as in familiar Glauber dynamics.. J. Marro et al., TBP & AIP Proceedings Series 779, (2005) 28

29 Generalization & applications Phase Diagram (approx.) (T 0 neurons; T 1 synapses ) Phases: Ferromagnetic (better assoc. memory than standard case, e.g., no mixture states); Paramagnetic (P); Oscillatory (dynamic memory; relate ± correlated patterns) J. Marro et al., TBP & AIP Proceedings Series 779, (2005) 29

30