application to biology

Transcription

1 This article was downloaded by: [Stony Brook University] On: 28 May 2015, At: 07:45 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: Registered office: Mortimer House, Mortimer Street, London W1T 3JH, UK Click for updates Advances in Physics Publication details, including instructions for authors and subscription information: Landscape and flux theory of nonequilibrium dynamical systems with application to biology Jin Wang abc a State Key Laboratory of Electroanalytical Chemistry, Changchun Institute of Applied Chemistry, Chinese Academy of Sciences, Jilin, People's Republic of China b College of Physics, Jilin University, Jilin, People's Republic of China c Department of Chemistry, Physics & Applied Mathematics, State University of New York at Stony Brook, Stony Brook, USA Published online: 20 May To cite this article: Jin Wang (2015) Landscape and flux theory of non-equilibrium dynamical systems with application to biology, Advances in Physics, 64:1, 1-137, DOI: / To link to this article: PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content ) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

2 Conditions of access and use can be found at

3 Advances in Physics, 2015 Vol. 64, No. 1, 1 137, REVIEW ARTICLE Landscape and flux theory of non-equilibrium dynamical systems with application to biology Jin Wang a,b,c a State Key Laboratory of Electroanalytical Chemistry, Changchun Institute of Applied Chemistry, Chinese Academy of Sciences, Jilin, People s Republic of China; b College of Physics, Jilin University, Jilin, People s Republic of China; c Department of Chemistry, Physics & Applied Mathematics, State University of New York at Stony Brook, Stony Brook, USA (Received 12 September 2014; accepted 24 March 2015) We present a review of the recently developed landscape and flux theory for non-equilibrium dynamical systems. We point out that the global natures of the associated dynamics for non-equilibrium system are determined by two key factors: the underlying landscape and, importantly, a curl probability flux. The landscape (U) reflects the probability of states (P) (U = lnp) and provides a global characterization and a stability measure of the system. The curl flux term measures how much detailed balance is broken and is one of the two main driving forces for the non-equilibrium dynamics in addition to the landscape gradient. Equilibrium dynamics resembles electron motion in an electric field, while non-equilibrium dynamics resembles electron motion in both electric and magnetic fields. The landscape and flux theory has many interesting consequences including (1) the fact that irreversible kinetic paths do not necessarily pass through the landscape saddles; (2) non-equilibrium transition state theory at the new saddle on the optimal paths for small but finite fluctuations; (3) a generalized fluctuation dissipation relationship for non-equilibrium dynamical systems where the response function is not just equal to the fluctuations at the steady state alone as in the equilibrium case but there is an additional contribution from the curl flux in maintaining the steady state; (4) non-equilibrium thermodynamics where the free energy change is not just equal to the entropy production alone, as in the equilibrium case, but also there is an additional housekeeping contribution from the non-zero curl flux in maintaining the steady state; (5) gauge theory and a geometrical connection where the flux is found to be the origin of the gauge field curvature and the topological phase in analogy to the Berry phase in quantum mechanics; (6) coupled landscapes where non-adiabaticity of multiple landscapes in non-equilibrium dynamics can be analyzed using the landscape and flux theory and an eddy current emerges from the non-zero curl flux; (7) stochastic spatial dynamics where landscape and flux theory can be generalized for non-equilibrium field theory. We provide concrete examples of biological systems to demonstrate the new insights from the landscape and flux theory. These include models of (1) the cell cycle where the landscape attracts the system down to an oscillation attractor while the flux drives the coherent motion on the oscillation ring, the different phases of the cell cycle are identified as local basins on the cycle path and biological checkpoints are identified as local barriers or transition states between the local basins on the cell-cycle path; (2) stem cell differentiation where the Waddington landscape for development as well as the differentiation and reprogramming paths can be quantified; (3) cancer biology where cancer can be described as a disease of having multiple cellular states and the cancer state as well as the normal state can be quantified as basins of attractions on the underlying landscape while the transitions between normal and cancer states can be quantified as the transitions between the two attractors; (4) evolution where more general evolution dynamics beyond Wright and Fisher can be quantified using the specific example of allele frequency-dependent selection; (5) ecology where the landscape and flux as well as the global stability of predator prey, cooperation and competition are quantified; (6) neural networks where general asymmetrical connections are considered for learning and memory, gene self-regulators where non-adiabatic dynamics of gene expression * jin.wang.1@stonybrook.edu c 2015 Taylor & Francis

4 2 J. Wang can be described with the landscape and flux in expanded dimensions and analytically treated; (7) chaotic strange attractor where the flux is crucial for the chaotic dynamics; (8) development in space where spatial landscape can be used to describe the process and pattern formation. We also give the philosophical implications of the theory and the outlook for future studies. PACS: 05, statistical physics, thermodynamics, and nonlinear dynamical systems; aq, networks; Vf, systems biology; ll, models of single neurons and networks; n, ecology and evolution; d, quantum statistical mechanics Keywords: landscape; flux; non-equilibrium systems and networks; non-adiabatic; spatial; quantum Contents PAGE 1 Introduction 4 2 Landscape and flux theory of non-equilibrium dynamical complex systems Decomposition of the driving force into a landscape gradient and curl flux for non-equilibrium dynamical complex systems Global stability for non-equilibrium systems Intrinsic non-equilibrium potential landscape as a Lyapunov function for quantifying global stability Force decomposition into a non-equilibrium intrinsic potential landscape and an intrinsic curl flux for deterministic dynamics The origin of non-equilibrium flux in the dynamical systems and networks Flux and its origin in a mono-stable kinetic cycle Flux and the potential landscape in the stable limit cycle Non-equilibrium kinetic paths Non-equilibrium transition state rates Equilibrium transition state rate The exponential contribution of the non-equilibrium transition state rate The transition state rate theory for non-equilibrium systems Non-equilibrium thermodynamics FDT for intrinsic non-equilibrium systems Gauge field, FDT 37 3 Multiple landscapes, curl flux, and non-adiabaticity Introduction Theory of non-adiabatic non-equilibrium potential and flux landscape for general dynamical systems A one variable coupled landscape 44 4 Spatial fields, landscapes, and fluxes Potential and flux field landscape theory for stochastic spatial non-equilibrium systems The Lyapunov functional for spatially dependent dynamical systems Non-equilibrium thermodynamics of stochastic spatial systems 54 5 The cell cycle: limit cycle oscillations Model for the cell-cycle network Self-consistent mean field approximation Results 60 6 Cell fate decisions: stem cell differentiation and reprogramming, paths, and rates The Waddington landscape for key modules of stem cell differentiation, paths, and rates 67

5 Advances in Physics The Waddington landscape and epigenetics The Waddington landscape for large networks 72 7 Landscapes and paths of cancer Introduction Results The underlying cancer gene regulatory network The landscape of the cancer network Dominant paths among normal, cancer, and apoptosis states Changes in landscape topography of cancer 84 8 Evolution Introduction Evolutionary driving forces: selection, mutation, and genetic drift Selection Mutation Genetic drift Mean fitness as the adaptive landscape for the frequency-independent selection systems The adaptive landscape for general evolutionary dynamics Potential and flux landscape of a group-help model with frequency-dependent selection for evolution The underlying potential flux landscape for general evolutionary dynamics Generalizing Fisher s FTNS The Red Queen hypothesis explained by a generalized FTNS 94 9 Ecology Introduction The ecological dynamical models The potential landscapes and fluxes of ecosystems: predation, competition, and mutualism Discussion Landscape and fluxes of neural networks Introduction The dynamics of general neural networks Potential and flux landscape of neural networks Flux and asymmetric synaptic connections in general neural networks Potential and flux landscape for REM/non-REM cycle Chaos: Lorentz strange attractor Introduction Landscape and probabilistic flux Flux may provide a clue to the formation of chaos Multiple landscapes and the curl flux for a self-regulator Potential, circulation flux, and eddy current Results and discussions Spatial landscapes and development Introduction Master equation Results and discussions Conclusion 127 Acknowledgments 129 Notation 129

6 4 J. Wang Disclosure statement 130 Funding 130 References Introduction The world around us and even ourselves are dynamic and not at equilibrium [1 17]. The physical world, including the weather, oceans, the Earth, the Sun, the solar systems, the galaxies, and even our universe (multiverses), is dynamically evolving. The biological world, including the cells, organs, plants, animals, humans, evolution, and ecology as well as society and economics, functions in a dynamical way. These dynamical complex systems are not closed systems. In fact, they are open systems with constant exchange of energy, materials, and information with outside environments. Therefore, these systems are intrinsically at non-equilibrium. Traditionally, a successful framework and corresponding comprehensive recipes have been laid out for studying complex closed or equilibrium systems in a global and quantitative way without significant exchange of energy, materials, and information with the outside environments [1,4]. Equilibrium thermodynamics and statistical mechanics have been successfully applied in physical and biological systems [18 20]. One starts with a priori known interactions among individual units of the systems. From these, the partition function and the free energy of the system can be quantified. The phase diagram characterizing the global behavior of the system can then be explored. In other words, in equilibrium systems, the global behavior can be determined by the underlying interaction potential energy landscape, while the local dynamics is determined by the gradient of the interaction potential energy. The funneled protein folding energy landscape provides a nice illustration [20]. The predictions of folding landscape can be tested by the corresponding equilibrium thermodynamic and kinetic experiments [18 20]. Following the remarkable successes in describing near equilibrium systems, a natural question would be: Can we apply the same recipe to dynamical non-equilibrium systems? This turns out to be a huge challenge for several reasons. First, it is challenging to specify the rules governing the dynamics. Even though these dynamical rules are specified from those and the local stability and function can be quantified, the global stability and behavior for the dynamics are still often out of reach. Furthermore, since the systems are not at equilibrium, even with the dynamical rules specified, the corresponding interaction energies are not a priori known in general. In other words, the driving forces for the dynamics in general cannot be written as a gradient of the interaction potential energy as in equilibrium systems [13 15,17]. Therefore, the recipe of equilibrium statistical mechanics of exploring the partition function, free energy, and phase diagram from interaction energy cannot be applied to non-equilibrium systems. In other words, the global behavior cannot be explored in the equilibrium way. It is then challenging to see the relationship between the dynamics and the non-equilibrium nature of the system. Progress has been made in exploring non-equilibrium systems and addressing various issues such as the characterization of the non-equilibrium steady state, deterministic and stochastic dynamics [1 12,21,22], and non-equilibrium thermodynamics, the stochastic thermodynamics, the relations between non-equilibrium work and equilibrium free energy, the link between the statistics of entropy production and entropy annihilation [1 5,9 12,21,23 37]. Also transition rates and paths have been studied with different methods including the Wentzel-Kramers-Brillouin (WKB) method, and methods of characteristics and optimal weights near zero-fluctuation limit have been explored [22,38 52]. Despite the progress made in this area, there are still challenges remaining. These include theoretical issues on the origin of the underlying driving force for non-equilibrium systems and how this differs from equilibrium systems having a gradient driving force. Other issues are global stability, kinetic

7 Advances in Physics 5 paths, and rates in finite fluctuations. Also the role of the driving force and the connection between dynamics and non-equilibrium thermodynamics needs to be understood. Other issues are the fluctuation dissipation theorem (FDT), and the intrinsic symmetry of non-equilibrium systems, as well as the understanding of the underlying mechanisms for different physical and biological systems illustrating many of the above-mentioned non-equilibrium dynamic behaviors. In this review, we will address all the above-mentioned challenges, with a systematic and unified theoretical framework in terms of both a landscape and a curl flux to quantify dynamical non-equilibrium complex systems in a global way based on our recent studies. We show that the existence of the flux in addition to the landscape is the origin of many of the characteristic non-equilibrium behaviors deviating from the naive expectations based on the equilibrium perspectives. The theoretical framework has been laid down and already published in previous papers [13 17,26,53 85]. We present only a review here. For more details, see the abovementioned references. Our starting point is to realize that for general non-equilibrium dynamical systems, the driving force of the dynamics cannot be written as a pure gradient of an equilibrium like potential landscape [13 15,17,62,72]. However, the general dynamics can be decomposed into a gradient of a non-equilibrium potential landscape and another force which has a curl rotating nature (curl flux force) [13 15,17,62,72]. In an analogy, the dynamics of equilibrium systems is like that of an electron moving in an electric field going straight down the electrical potential gradient, while the dynamics of non-equilibrium systems is like an electron moving in an electric field and magnetic field spiraling down the electrical potential gradient [13 15,17,62,72]. Therefore, we can see that in non-equilibrium systems: the dynamics is determined by both the non-equilibrium potential landscape and the curl flux, rather than by a single factor of potential landscape as in the equilibrium systems. As we will show later on, the non-equilibrium potential landscape is closely linked to the steady-state probability distribution for non-equilibrium systems and therefore has a probabilistic nature. This is analogous to the equilibrium case where the equilibrium potential landscape is closely linked to the equilibrium probability of the equilibrium systems. It turns out that the non-equilibrium potential landscape topography is essential and can be used to quantify the global stability and behavior of the non-equilibrium dynamical systems [13 15,17,62,72]. Furthermore, the dynamics of non-equilibrium systems is distinctly different from those of equilibrium systems. First, the kinetic paths from one state to another do not follow the landscape gradient due to the presence of the curl flux [14,61,62,82]. As a result, at finite fluctuations, the dominant kinetic paths do not go through the landscape saddle point from one state basin to another. This is in contrast to equilibrium systems where the dominant minimum energy kinetic paths go through the saddle point of the landscape [14,61,62,82]. Moreover, the forward and backward paths are irreversible in contrast to the case of equilibrium systems where the forward and backward paths are reversible. The presence of the curl flux in non-equilibrium systems thus breaks time reversal symmetry. Importantly, the kinetic rates from one state to another for non-equilibrium systems do not follow the conventional transition state or Kramers theory for equilibrium systems [14,61,62,82]. Instead of the saddle on the landscape, a new transition state or Kramers theory for non-equilibrium systems emerges: the kinetic rates for non-equilibrium systems are determined by a new saddle of dominant paths instead of a point on the landscape [14,61,62,82]. The non-equilibrium thermodynamics can also be quantified based on the combined landscape and flux theory. In conventional equilibrium thermodynamics, entropy production arises from free energy dissipation during the relaxation to the equilibrium. In non-equilibrium systems, the entropy production is from the free energy dissipation during the relaxation to the steady state plus the additional house-keeping contribution for maintaining the non-equilibrium steady state

8 6 J. Wang from the presence of the flux due to the energy pump from outside environments [9,15,67]. The total entropy including the system and environment is never decreasing, while the system nonequilibrium free energy is never increasing. We can also consider phases and phase transition in non-equilibrium systems. Some new phases emerge only in non-equilibrium systems, while they cannot exist in equilibrium systems [67]. An example is the limit cycle. We also see that the FDT will have to be modified for non-equilibrium systems [15]. Instead of the conventional FDT in equilibrium systems where the response or relaxation to the equilibrium is closely linked to the equilibrium fluctuations, a new FDT for the non-equilibrium systems emerges. Based on this theory, the response or relaxation to the steady state is closely linked to the steady-state fluctuations and additional contributions from the presence of the flux for maintaining the non-equilibrium steady state [15]. Furthermore, we can also link the presence of the flux to the fluctuation theorem giving the statistical relationship between the forward and backward processes [15]. We find that non-equilibrium dynamics is closely related to gauge field theory. The presence of the curl flux provides the basis for the gauge field. In fact, it is closely related to gauge field curvature [15]. Therefore, there is an intimate link between the non-equilibrium statistical nature and the geometry of the gauge field (non-zero curvature). These two descriptions of the world through statistical numbers and geometry/topology are both fundamental and seem not to be related to one another. In our non-equilibrium descriptions, the non-equilibriumness is quantified by the degree to which the curl probability flux vector differs from zero. On the other hand, we also see that this is equivalent to measuring the degree of the curvature of the internal geometry (symmetry) of the corresponding gauge field being away from zero (flat). So we can link the non-equilibriumness to the internal geometry. As a result, there is a path-dependent gauge field where its integral over a closed loop is not zero due to the presence of the flux [15]. This non-zero loop integral is analogous to the Aharonov Bohm or Berry phase factor except that it is now real rather than imaginary. This generates the non-trivial global topology of the gauge field which is also linked closely to the thermodynamics of heat dissipation to the environments. Therefore, the non-equilibrium nature of systems is deeply linked to geometry and topology. We can extend landscape and flux theory to include collective dynamics in physical spaces of higher dimension. The original non-equilibrium statistical mechanics theory now becomes a nonequilibrium statistical field theory [73,75,85]. The field dynamics in space can still be determined by the key dual pair: the non-equilibrium field landscape and curl flux field. The global stability and thermodynamics for the system with non-equilibrium field can still be specified through a non-equilibrium statistical mechanical landscape and flux framework except now the dynamical variables of fields and the correlations in physical space are included [73,75,85]. Similar conclusions are nevertheless obtained. We extend the approach to include not only a single landscape but multiple landscapes. We see that a discrete number of multiple landscapes can be seen as a single landscape with more extended dimensions. This is an effective way of exploring the non-adiabatic effects when more timescales are involved in the coupling of the landscapes or from the extended dimensions [63,65,66,74]. We find additional flux coming from the extended dimensions which can contribute significantly to the dynamics. We illustrate the landscape and flux theory with a few concrete examples, the cell cycle [13,17,83], stem cell differentiation and development [60,62,66,76,77,79], chaotic Lorentz attractors [68] and spatial development [73,75,85], evolution [69] and ecology [81], neural networks [72], self-regulating genes as an example of multiple landscapes [63,74] and spatial development [58]. For the cell cycle, the resulting dynamics is a limit cycle with distinct biological phases of G0/G1, S/G2, and M and corresponding checkpoints. We point out that the underlying

9 Advances in Physics 7 non-equilibrium potential landscape has an inhomogeneous Mexican hat shape. Away from the oscillation ring, the gradient of the non-equilibrium potential landscape attracts the system down to the ring, while on the ring, it is the curl flux which drives the oscillation. The local basins along the oscillation ring quantify the cell-cycle phases while the barriers or transition states between them quantify the checkpoints [13,17,83]. Global sensitivity analysis on landscape topography and flux can be used to explore the key hot spot structure elements in the cell-cycle network responsible for the global function. We also illustrate how the presence of the flux from the nutrition supply breaks the time reversal symmetry from the three-point correlation functions [13,17,83]. For stem cell differentiation and development, we point out that stem cells and differentiated cells can be quantified as basins of attractions on the underlying non-equilibrium landscape [60,62,66,76,77,79]. The landscape and the transitions from stem cell basin to differentiated cell basins quantifies Waddington s qualitative metaphor picture for development. The kinetic paths do not go through the saddle point of the landscape due to the presence of the curl flux. As a consequence, the forward differentiation paths and backward reprogramming paths are not mirror images. Quantifying the differentiation and reprogramming paths is crucial for tissue re-engineering. The kinetic rates do not follow the equilibrium transition state rate theory. Instead, from non-equilibrium transition state theory (TST), the transition state is determined as the saddle on the path. For cancer biology, we describe cancer as a disease of states rather than being caused purely by mutation. Cancer states then naturally emerge from the gene gene interactions in an underlying gene regulatory network. We apply the non-equilibrium landscape and flux theory to cancer [84]. We show that (1) The normal states and cancer states can be quantified as basins of attractions on the cancer landscape, the depth of which represents the associated probabilities. (2) The global stability, behavior, and function of cancer through underlying regulatory networks can be quantified through the landscape topology in terms of depths of and barriers between the normal and cancer states. (3) The paths and transition rates from normal (cancer) state to cancer (normal) state representing the underlying tumor-genesis and reverse processes will be quantitatively uncovered, leading to the hope for prevention and treatment. This will be developed through a path integral approach and a non-equilibrium TST. Through global sensitivity analysis based on landscape topography, we can identify the key genes and regulations important for the global stability, behavior, and function of cancer networks. The results of global sensitivity analysis will provide multiple targets for cancer intervention. This will quantitatively uncover the link between the structure and the function of cancer networks, and provide the theoretical basis for exploring cancer as a network disease and designing new anti-cancer strategies based on this view. Our focus of study then shifts from exploring local properties (cancer as a disease of individual driver-mutation) in the conventional approach to quantitatively study the global nature (cancer as a disease of states) of cancer through the underlying regulatory networks. We establish a landscape and flux theory of evolution [69]. Conventional theory is inadequate for describing the general evolution dynamics. In Wright s adaptive landscape picture, evolution dynamics follows the gradient of a landscape. In Fisher s fundamental theorem of natural selection (FTNS), however, the fitness never decreases. For general evolution dynamics, none of these notions turns out to be correct. A good example is given by allele frequency-dependent selection where the driving force for evolution cannot be expressed as a pure gradient. This has been called the Red Queen effect where evolution can still keep running even at optimal fitness [69]. Red Queen effects exist widely in the coevolution of species and have been used to question the validity of Wright and Fisher s theory of evolution [69,86]. The combined landscape and flux theory provides a quantitative and physical basis for such general evolution. Evolution dynamics is not only determined by a landscape gradient as in the conventional evolution theory but also by the

10 8 J. Wang curl flux [69]. Even when the landscape reaches an optimum (such as the limit cycle oscillation ring), evolution still keeps on going, being driven by the curl flux. This naturally explains Red Queen effects. The source of the curl flux can be from allele frequency-dependent selection or mutation, recombination, epistasis, etc. We also discuss a landscape and flux theory for ecology [81]. We uncover the underlying intrinsic potential landscape as a global Lyapunov function. This provides a quantitative measure of the global stability of ecosystems. We can study several typical and important ecological systems involving predation, competition, and mutualism [81]. A single attractor, multiple stable attractors, and also limit cycle attractors emerge from these systems. Both the landscape gradient and curl flux determine the dynamics of these ecosystems. The theory provides a quantitative and physical framework for exploring the global stability, function, and dynamics of ecosystems [81]. We can also use landscape and flux theory for neural networks in the brain [72]. Previously, neural cognition in learning and memory has been thought as information storage and retrieval from basins of attractions. Such a conclusion has been based on the assumption that neurons are symmetrically connected in neural networks. This leads to a global energy landscape for stability and associated gradient dynamics. However, in reality, the neurons in neural networks of the brain are asymmetrically connected. In this general case, the conventional theory does not apply. From combined landscape and flux theory, we can uncover the underlying landscape and Lyapunov function for quantifying the global stability of general neural networks [72]. Furthermore, we show that the asymmetrical connections of the neurons lead to a curl flux which contributes to the driving force of the dynamics in addition to the gradient of the landscape. Many neural oscillations have been found in the brain. These cannot be explained by the conventional symmetrically connected network dynamics. Instead, landscape and flux theory can form the basis for describing the global nature of the neural oscillation dynamics. For learning and memory, the oscillation may help to store and retrieve information in a continuous way for sequential events [72]. We also map out the non-equilibrium potential for the chaotic Lorentz attractor [68]. There, we find that the underlying potential has a butterfly shape determining the global stability. The dynamics of chaotic attractor is determined primarily by the gradient of the potential away from the butterfly sheet and by the curl flux within the butterfly sheet. Therefore, the flux is closely related to the onset of the chaotic attractor behavior [68]. Using a simple example of self-regulation of genes, we explore multi-landscape dynamics where discrete transitions of a single molecule are involved [63,74]. We show that the intralandscape dynamics in protein concentrations alone with inter-landscape dynamics in gene states can be described as the dynamics in a single landscape in protein concentration and probability of gene states. We explore the non-adiabatic effect of weak coupling between the landscapes which again turns out to be equivalent to the emergence of a curl flux on the single landscape in the expanded dimensions of protein concentration and gene probability states [63,74]. This curl flux influences significantly the dynamics of the self-regulator leading to the emergence of multiple states, larger variations, time irreversibility of the kinetic process, and energy dissipation. We also extended the landscape and flux framework to systems extended in the physical spatial domain [58,73,75,85]. As a simple example of spatial development, we show that a spatially varying landscape is crucial for specifying the local patterns for development. We will give more detailed descriptions of the combined landscape and flux theory as well as the associated applications in the following sections. In the conclusion section, we will provide a brief summary. We will also give the philosophical implications of the theory and the outlook for future studies.

11 Advances in Physics 9 2. Landscape and flux theory of non-equilibrium dynamical complex systems 2.1. Decomposition of the driving force into a landscape gradient and curl flux for non-equilibrium dynamical complex systems For dynamic complex systems, the dynamics are often described in the continuous representation of dynamic equations as follows [7,87]: ẋ = F(x), (1) x is a vector representing the system variables (concentrations, densities, etc). ẋ represents the change of the system variables with respect to time. The above equation can be thought of as an over-damped limit of Newton s second law with friction αẍ +ẋ = F(x), while the F(x) is also a vector representing the driving force [7,87]. The dynamic equation above thus has the meaning that the temporal evolution of the system variables is determined by the driving force. This type of ordinary differential equations describes the dynamics of a wide range of complex systems, such as nonlinear dynamics, chaos, biological and social networks, weather, ecology and evolution, etc. The same type of equation has been the focus of study of nonlinear dynamics and chaotic behavior. Protein dynamics, protein folding, molecular recognition, cellular networks and even biological evolutions represent some biological applications. The conventional way to analyze system dynamics based on the above equation is to follow the temporal trajectories of the system variables. The local stability can be analyzed through the identification of the fixed points of the system [7,87]. The local stability around those fixed points can be quantitatively studied. However, if we aim to explore the global nature of the system, we need to quantify the global stability. If the driving force can be expressed as the gradient of a scalar potential, then the system s dynamics is described by gradient dynamics and the system s global behavior can be quantified by this scalar potential [13,16,17,62]. In fact, this scalar potential for some physical and biological systems is known directly from the interaction potential energy. In these examples, equilibrium statistical mechanics can be applied to study the global behavior, such as global phases, phase diagrams, and the corresponding global thermodynamic behavior by quantifying the corresponding partition function and associated free energies [13,16,17,62,69]. However, in general, the driving force of dynamic complex systems cannot always be quantified as a pure gradient of a scalar potential. In such cases, equilibrium statistical mechanics cannot be applied. The challenge is then how to study the global behavior and function, and what are the keys to characterizing the dynamics globally. In order to address this, we look at stochastic versions of the dynamic evolution equations. The rationale is the following: in the natural world, fluctuations and noise never vanish. So the more realistic description of dynamics is always stochastic. Furthermore, we will show a strategy of studying the stochastic dynamics first and then taking the zero fluctuation limit for reaching the original deterministic dynamics [1,2,7,88]. It turns out to be much more convenient to study the global behavior in the stochastic representation than in the original deterministic format where there is no natural probability measure to start with. The stochastic dynamic equations are given as follows [1,7]: ẋ = F(x) + η(x, t). (2) Here η(x, t) is a stochastic driving force whose amplitude of fluctuation is measured by its autocorrelation function. η(x, t)η(x, t ) =2DDδ(t t ), where D is a scale factor representing the magnitude of the fluctuations and D is the diffusion tensor or matrix [1,7]. Since the systems now follows stochastic dynamics, tracing the individual trajectories becomes unpredictable and

12 10 J. Wang less meaningful than the deterministic case. Instead, one should trace the evolution of probability distribution leading to a global characterization of the system as we will show later. The stochastic equation of motion is called the Langevin equation, while the corresponding probability evolution equation follows the Fokker Planck equation [1,7], P(x, t) t + J(x, t) = 0. (3) The meaning of the above equation is the local conservation of probability. The flux is given as J(x, t) = F(x)P(x, t) (DDP(x, t) [1,7,13,17]. The change of the local probability at any location is equal to the flux in or out of that region. For the statistical steady state expected to engage in the long time limit, the time derivative of the probability becomes zero and the probability no longer changes with respect to time. This implies that the divergence of the flux is zero from the above equation J ss (x) = 0, where ss stands for steady state. There are several possibilities for how the vanishing divergence is achieved. (1) The flux itself is zero. J ss = 0. The zero flux implies no net flux in or out of any subpart of the ensemble. Such a system obeys detailed balance and has come to a true equilibrium. The driving force in this case can then be written as a pure gradient up to a gradient with respect to the diffusion coefficient. F = DD P ss /P ss + DD = DD U + DD, where the potential function U is defined as the logarithm of the equilibrium probability U = ln P ss.here the steady-state probability is the same as the equilibrium probability due to the detailed balance condition [13,17]. Then, we can see for equilibrium systems where the detailed balance is preserved, the Boltzmann relationship is recovered (the potential landscape is related to the equilibrium probability), and the dynamics is determined by the gradient of the potential up to a gradient of the inhomogeneity of the diffusion coefficient. So for equilibrium systems, the global behavior is determined by the equilibrium probability landscape or the potential landscape which can characterize the basins and barriers as well as the global stability and phases once the potential landscape is known [13,17]. On the other hand, local dynamics is determined by the potential landscape through the gradient. The dynamics of such equilibrium systems is analogous to electrons moving straight down to the electric field gradient [13,17]. (2) When a steady state emerges, the flux itself is not necessarily zero even for the divergence free conditions [13,17]. In this case, there is a net flux into or out of the regions of the ensemble. With such a non-zero flux, the system is no longer in real equilibrium and detailed balance is broken. In fact, the degree of non-equilibriumness can be measured by how much the curl flux differs from zero. The divergence free condition of the flux signaling the statistical steady state means that there are no sources and sinks for probability to go to or to come from. Therefore, locally the flux must be rotational and be derived from a curl. The driving force then can be decomposed to the gradient of the steady-state non-equilibrium potential U, where U is defined as the logarithm of the steady-state probability distribution U = ln P ss,uptoagradient of diffusion but there is also a curl flux [13,17,67]: F = DD P ss /P ss + DD + J ss /P ss = DD U + DD + J ss /P ss. The non-equilibrium potential is still closely linked to the steadystate probability since the steady-state probability quantifies the probability of each state and the underlying probability landscape topography can be described in terms of basins and barriers. Therefore, the non-equilibrium potential still can be used to quantify the global behavior and stability of non-equilibrium systems. On the other hand, the dynamics of non-equilibrium systems is not determined by the potential gradient alone, but also by the curl flux. The non-equilibrium potential and curl flux are a dual pair for determining the non-equilibrium dynamics [13,17,67]. The dynamics of non-equilibrium systems is analogous to electrons moving in an electric and

13 Advances in Physics 11 magnetic field. The dynamics evolves spiraling down an electric field gradient due to effective Lorentz forces. There are some crucial differences between equilibrium and non-equilibrium systems: (1) For equilibrium systems, the potential landscape is given a priori, while the landscapes for strictly non-equilibrium systems are emergent. In order to determine the non-equilibrium potential, one needs at the outset to obtain the steady-state probability distribution from the dynamics. This is not known a priori. In other words, for equilibrium systems, the potential is an input that directly provides the driving force for the dynamics, while the non-equilibrium potential is the result of the dynamics. (2) The dynamics is determined by the statistical steady-state condition for nonequilibrium systems. The non-equilibrium potential is determined by the resulting steady-state probability, and the curl part of the flux is determined by the originally given driving forces and the emergent steady-state probability. In other words, the steady-state behavior still influences the temporal evolution dynamics, but it is not the whole story. (3) The force decomposition to the non-equilibrium potential gradient and curl flux is not unique. However, we always can choose a physical gauge here so that the non-equilibrium potential is represented as the logarithm of the steady-state probability. This choice has a clear physical meaning and is directly related to the probability landscape which is crucial for quantifying the global stability and behavior. This choice is also reasonable since it naturally leads to gradient dynamics at equilibrium conditions where there is precisely zero flux. (4) The non-equilibrium situation we discuss here is the intrinsic non-equilibrium with detailed balance breaking. Such steady states must be contrasted to the often studied cases where the physical systems start out in equilibrium with detailed balance, but upon perturbation the system moves away from equilibrium position and the subsequent relaxation dynamics to the equilibrium state is termed non-equilibrium dynamics. For us, non-equilibrium dynamics refers to the case of intrinsic non-equilibrium with detailed balance explicitly broken. Relaxation after the perturbation moves only toward the steady state without detailed balance rather than to a true equilibrium state with detailed balance. We will discuss this further when we study the changes in the FDT for non-equilibrium steady states. (5) The nature of the curl flux is global. In equilibrium systems, a detailed balance condition of zero flux J ss = 0 needs to be satisfied everywhere. This condition is local, since if we break the detailed balance condition locally, one only needs to fix a corresponding local part to restore the detailed balance without influencing the other parts of the state space. However, the situation for non-equilibrium systems is very different. The flux is not zero and satisfies a divergence free condition due to the steady-state condition which implies its rotational curl nature. The rotation is global in state space. Therefore, any local perturbations away from the rotational curl will require global adjustments to be restored back. Here the local adjustments are not enough for restoring the global rotational curl. It turns out that the true non-equilibrium behavior is deeply connected to the global topology of the underlying dynamical systems due to the nature of the curl flux. Recently, several other methods of decomposing the continuous equations of motion have been proposed [89 91] for stochastic systems. One method is aimed at global stochastic dynamics [89]. The numerical solutions of this decomposition seem challenging. It was argued that the decomposition might be specific due to the constraints imposed resulting orthogonality [92]. Another method suggests still another different representation and mapping [90]. A decomposition method recently proposed works only very near the zero noise limit [91] and this analysis reaches similar conclusions to an earlier general study [69] in the zero-fluctuation limit. Relationship between decomposition and conservative/non-conserved systems as well as time reversal symmetry have also been discussed [92 94]. The network thermodynamics and decomposition for discrete Markov chains has also been discussed [3,78,95 101]. The landscape and flux theory can be applied to both zero and finite fluctuations (for finite fluctuations, the free energy functional instead of the intrinsic potential becomes the

14 12 J. Wang Lyapunov function, see more details in the later sections on non-equilibrium thermodynamics) [13 15,62,69,73,75,82,85]. The landscape gradient and the curl flux have an emergent duality which measures the extent of detailed balance breaking, in analogy to how an electron moves in electric and magnetic fields at the same time. We will now turn to the discussion of how we can quantify the global stability with the concept of the non-equilibrium potential Global stability for non-equilibrium systems Intrinsic non-equilibrium potential landscape as a Lyapunov function for quantifying global stability Global stability is often critical for understanding complex dynamical systems [1 13,16,17]. Great efforts have been made to study the subject. A conventional way of exploring the stability is first to search for the fixed points of the dynamical equations governing complex system. Following this, the stability to small displacements around these fixed points can be analyzed. However, such an analysis is limited to the specific fixed points which are found and therefore can only explore local stability. For global stability, one has to search the entire space and not just specific local regions of the state space. One way of doing this is to seek a Lyapunov function for the system that monotonically changes with respect to time so that the global stability is quantifiable. We are going to follow this route to quantify the global stability of complex dynamical systems. We first look at the global stability of general dynamical systems specified by the equation: ẋ = F(x) [7,13,87]. To do so, we first add the stochastic noise to the system, making the dynamical equations stochastic. Due to the stochastic nature of the problem, no single trajectory will allow us to predict the individual outcome of an initial set of conditions. We can only obtain information statistically. Nevertheless, the evolution of the probability is deterministic and actually linear. So instead of studying individual stochastic trajectories, we focus on the probability evolution which is predictable and can be used to describe the global behavior of the stochastic dynamics. The probability evolution of the continuous observables follows the Fokker Planck equation as described earlier. Let us look at the behavior of the probability in the small noise limit mimicking the deterministic system. In the literature, the WKB method has been used in the study of stochastic processes in the zero noise limit [1,2,5 7,12,22,38 52]. We can follow a similar route by expanding the steady-state probability or non-equilibrium potential with respect to the scale of the fluctuations D 1[5 7,12,69,71,81]: U(x) = [ k=0 Dk φ k (x)]/d = φ 0 (x)/d + φ 1 (x) + Dφ 2 (x) +. Consequently, P ss (x) = (1/Z) exp[ ( k=0 Dk φ k (x))/d] and Z = exp[ ( k=0 Dk φ k (x))/d]dx. There are several motivations for doing such an expansion. (1) When the fluctuations become small and reach the zero limit, the deterministic dynamic equations are recovered. The challenge is whether we can find a potential landscape having the Lyapunov property for the corresponding deterministic dynamics. (2) For realistic systems with finite but weak fluctuations, one might be able to use the weak fluctuation approximation through expansion in the fluctuation strengths D to address the global stability issue. (3) One can study how the probability landscape changes with respect to the fluctuations this way. (4) With large fluctuations, one can solve the Fokker Planck or master equation and the corresponding probability directly without making the weak fluctuation expansion. By substituting the expression of P from the weak fluctuation expansion into the steadystate Fokker Plank equation and comparing the corresponding coefficients with the same powers of D on both sides of the equation, a set of equations in each order of 1/D about φ i (x) [7,69,71,81] can be obtained. The equation up to the leading order expansion D 1 resembles

15 the Hamilton Jacobian (HJ) equation of classical mechanics: Advances in Physics 13 F φ 0 + φ 0 D φ 0 = 0. (4) The solution at the zero order, φ 0, does not depend on the fluctuation strength D explicitly. The physical meaning of φ 0 can be seen clearly since φ 0 up to a scale factor D is closely associated with the steady-state probability P ss exp( φ 0 /D) at the low noise limit. So, φ 0 is tied up with probability which characterizes the steady state globally. In fact, the norm of φ 0 monotonously decreases along a deterministic trajectory due to the positive definite nature of the diffusion matrix D [7,69,71,81]: dφ 0 (x) dt = F φ 0 = φ 0 D φ 0 0. (5) As we can see from the above equation, the temporal evolution of the φ 0 will not stop (the temporal change is not zero, dφ 0 (x)/dt 0) until φ 0 = 0 where there are no longer changes in time (dφ 0 (x)/dt = 0). As we can see, φ 0 is thus a Lyapunov function. φ 0 monotonically decreases with respect to time. This property can be used to study the global stability. With both the probability and Lyapunov properties, the φ 0 acquires its physical meaning as the intrinsic (zero-fluctuation limit) non-equilibrium potential characterizing the global stability for general dynamical systems [7,69,71,81]. As we can see, the φ 0 = 0 provides conditions for finding the attractors on the potential landscape. For point attractors, the temporal evolution of φ 0 settles at minimum values. For continuous attractors such as the line attractors, the limit cycles or the strange attractors, the φ 0 must settle to constant values. The rationale is the following: if the values of the φ 0 on continuous attractors are different, then the right side of the above equation will not be equal to zero. This implies that the φ 0 has not settled to a final value (dφ 0 /dt 0). This is obviously not the case. We thus see that the condition dφ 0 /dt = 0 requires that the φ 0 has a constant value on continuous attractors. The numerical solutions of the HJ equation have been discussed and a self-consistent method to solve it has been proposed [39]. Another numerical level-set method has been developed for solving this equation [102]. Recently a method [50] was developed to construct landscapes self-consistently in phase space for multi-stable systems at the zero noise limit. The local and global mapping and global construction of the landscape at the zero noise limit was also recently discussed [51]. In summary, due to its monotonic decreasing nature, the intrinsic non-equilibrium potential landscape φ 0 is a Lyapunov function [7,69,71,81]. Finding the Lyapunov function is crucial for quantifying the global stability for complex dynamical systems Force decomposition into a non-equilibrium intrinsic potential landscape and an intrinsic curl flux for deterministic dynamics After addressing the importance of the non-equilibrium intrinsic potential landscape, it is natural to explore the other component of the driving force, the curl flux. In the zero-fluctuation limit, D 0, we not only can expand the steady-state probability but also the steady-state probability flux J ss in terms of the fluctuation strengths characterized by the scale factor D. The leading order results are given as follows: (J ss /P ss ) D 0 = F + D φ 0 [69,71,81]. We can define the intrinsic flux velocity, V = (J ss /P ss ) D 0. From Equation (4), we

16 14 J. Wang can see that V φ 0 = 0. (6) This implies that the gradient of the non-equilibrium intrinsic potential φ 0 is perpendicular to the intrinsic flux vector (or intrinsic flux velocity) in the zero-fluctuation limit. As we can see, in the zero-fluctuation limit for the deterministic dynamics, the driving force can be decomposed to the gradient of intrinsic potential φ 0 and the intrinsic flux velocity V [7,69,71,81]: F = D φ 0 + V. (7) Therefore, the global stability for the non-equilibrium complex dynamical systems is determined and quantified by the non-equilibrium potential landscape, while the dynamics of nonequilibrium dynamical systems is determined by both the non-equilibrium potential and the curl flux. Note that the Lyapunov function φ is valid in the zero noise limit. For finite fluctuations, the free energy function F is a Lyapunov function (see details in later sections on non-equilibrium thermodynamics). We want to point out that using the population landscape U instead of Lyapunov function in finite fluctuations has certain advantages. For example, the Lyapunov function only gives global behavior but not the local details. For limit cycle oscillations (more details in cell cycle section), the Lyapunov function gives a perfect Mexican hat with the oscillation ring with equal potential depths. On the other hand, the population landscape U reflects the differences on the oscillation path therefore giving an inhomogeneous oscillation ring with different potential depths on the ring. The population landscape thus directly reflects the inhomogeneous speed on the ring while the Lyapunov function cannot. The condition for the small fluctuation limit is that the scaled diffusion coefficient should be significantly smaller than 1 for Fokker Planck equation to work well. In our examples, the numerical values of the scaled diffusion coefficient D satisfy this criterion. When the fluctuations are large, the diffusion equation with the second-order truncation may not be a good approximation. The higher order terms have to be included. The full master equation without the truncation should be considered. In practice, stochastic simulations are often more efficient for realizing the goal than directly solving the master equation which is time and allocation consuming (an exponential size scaling bottleneck is quickly reached when the system size becomes large) The origin of non-equilibrium flux in the dynamical systems and networks In this section, we explore the origin of the non-equilibrium flux. The non-zero flux is closely related to the breaking of detailed balance [67]. For a typical model of dynamical systems, the driving force is described in a phenomenological way. Therefore, it is difficult to see explicitly the analytical expression of the two components of the driving force: gradient and flux [13 15,17]. Furthermore, it is also hard sometimes to see how precisely the detailed balance is broken and how the flux becomes non-zero mechanistically. In this section, we explore the microscopic origin of curl probability flux. We have studied the dynamics of several systems including ones exhibiting mono-stability and those having limit cycles in searching for the microscopic origin of the probability flux. The origin of the probability flux turns out to be in the energy pump provided from non-equilibrium conditions, that is, on the concentration differences in specific energy producing sources. In chemical kinetic systems, the probabilistic flux is also closely related to the steady-state deterministic chemical flux. In a mono-stable model of the kinetic cycle, the probabilistic flux is directly related to the deterministic flux which is generated by the chemical potential difference (non-equilibrium energy pump) from adenosine triphosphate (ATP) hydrol-

17 Advances in Physics 15 ysis (the energy production source or energy supply in the cell). In the reversible Schnakenberg model for limit cycles, the probabilistic flux is correlated to the chemical driving force [67]. We can see that the curl probability flux is closely associated with existence of a nonequilibrium energy pump, either in a deterministic or probabilistic way. The curl flux generated from the energy pump is essential for keeping coherent and stable limit cycle oscillations Flux and its origin in a mono-stable kinetic cycle Deterministicpump andflux of thekinetic cycle. The kinetic cycle model shown in Figure 1 is a popular one in enzyme kinetics [103,104]. For example, the substrates D and E in Figure 1(b) denote the energy providers ATP and adenosine diphosphate (ADP) in a cell. The numbers of ATP or ADP molecules are often large enough in the cell so that their concentrations are kept almost constant relative to other reactants due to their key specific biologic function. This provides a relatively stable energy pump by keeping a non-equilibrium ratio of the concentrations [D]/[E], while the internal cycle remains functional. It turns out that this open chemical cycle can be mapped onto a closed cycle in Figure 1(a), with the kinetic rate coefficients modified as follows: k 1 = k1 0[D] and k 1 = k 1 0 [E]. We can include the energy pump from outside by changing k 1 and k 1,[104]. The energy pump generating the chemical potential originates from the non-equilibrium ratio in concentrations of the substrates, [D]/[E] (such as ATP and ADP concentrations ). Maintaining such a non-equilibrium ratio in concentrations will lead to broken detailed balance. From the law of mass action, we study the deterministic chemical reactions described by the ordinary differential equations of the kinetic cycle. Since the total number of the molecules of A, B, and C is conserved, the chemical reaction equations can be reduced to two variable differential equations. x, y, and N c x y denote the concentration of species A, B, and C, respectively. dx dt = k 1y + k 3 (N c x y) (k 1 + k 3 )x, dy dt = k 1x + k 2 (N c x y) (k 1 + k 2 )y. The above equations not only describe the closed cycle but also include the open chemical cycle, with certain modifications, k 1 = k 0 1 [D] and k 1 = k 0 1 [E]. (8) (a) (b) Figure 1. Kinetic cycles of the simplified three species enzyme kinetics. Comparing to the closed system of cycle (a), cycle (b) brings in two more substrates D and E which can break the detailed balance to generate a non-equilibrium steady-state flux, with the reaction: A + D k0 1 B + E by keeping a non-equilibrium concentration ratio [D]/[E]. By neglecting the fluctuation of concentrations of D and E, cycle (b) can be represented by cycle (a) in terms of the pseudo-first-order rate constants: k 1 = k 0 1 [D] andk 1 = k 0 1 [E](from Ref. [67]). k 0 1

18 16 J. Wang For convenience, we use the substitutions below: K 1 = k 1 k 2 + k 1 k 3 + k 2 k 3, K 2 = k 1 k 2 + k 1 k 3 + k 2 k 3, K 3 = k 1 k 2 + k 1 k 3 + k 2 k 3. (9) We call the net chemical reaction flux in the steady state the deterministic flux. Taking the reactions between A and B as an example, the net flux can be expressed as J 1 = x 0 k 1 y 0 k 1, where x 0 and y 0 are the steady-state fixed point concentrations (the subscript 1 labels the deterministic flux between A and B, subscript 2 labels deterministic flux between B and C, the subscript 3 labels the deterministic flux between C and A) and the steady-state deterministic flux becomes [67] J SS1 = x 0 k 1 y 0 k 1 = N c k 1 K 1 k 1 K 2 K 1 + K 2 + K 3 = N c k 1 k 2 k 3 k 1 k 2 k 3 k 1 k 2 + k 1 k 3 + k 2 k 3 + k 1 k 2 + k 1 k 3 + k 2 k 3 + k 1 k 2 + k 1 k 3 + k 2 k 3. (10) The above expression also represents the non-equilibrium steady-state flux for the open kinetic cycle, shown in Figure 1(b) with k 1 = k1 0[D] and k 1 = k 1 0 [E]. The deterministic steadystate flux indicates how fast the chemical species can be transferred when the reactions reach the steady state. For this three node cycle, the net fluxes between each pair of species are equal, J SS = J SS1 = J SS2 = J SS3. This means that there is only one deterministic flux in the kinetic cycle [67]. In chemical thermodynamics, the chemical potential difference in terms of Gibbs free energy of the kinetic cycle is defined as the ratio between the products of the forward rates and the backward rates. ( ) ( ) k1 k 2 k 3 k1 0 G = ln = ln k 2k 3 [D] k 1 k 2 k 3 k 1 0 k. (11) 2k 3 [E] When G = 0, the whole kinetic cycle is in equilibrium without any input or output. In this case, the ratio between the products of the forward rates and backward rates is equal to one. This preserves time reversal symmetry: k 1 k 2 k 3 k 1 k 2 k 3 = 1 or [D] eq [E] eq = k0 1 k 2k 3 k 0 1 k 2k 3. (12) From Equation (10), this condition leads to J SS = J SS1 = J SS2 = J SS3 = 0 Equations (12) represents the equilibrium condition in chemical reaction kinetics. However, if the ratio of the constant energy pump concentrations [D] and [E] does not satisfy the equilibrium condition of Equations (12), detailed balance will be broken. The Gibbs free energy difference G will not be zero as in the equilibrium case. As a result, a non-zero deterministic flux J SS emerges [67]. The energy pump such as ATP/ADP is then the origin of the non-zero deterministic flux. Probabilistic flux and energy pump of the kinetic cycle. External fluctuations as well as the inherent stochasticity of molecular processes often lead to the stochastic chemical dynamics. For

19 Advances in Physics 17 stochastic chemical dynamics, we need to follow the time evolution of the probabilistic distribution of the species to describe the global behavior instead of the individual trajectories, which are unpredictable. We start with the chemical master equation describing the probability evolution: dp m,n dt = p m,n [(N m n)(k 3 + k 2 ) + n(k 1 + k 2 ) + m(k 3 + k 1 )] + p m+1,n (m + 1)k 3 + p m+1,n 1 (m + 1)k 1 + p m,n+1 (n + 1)k 2 + p m 1,n+1 (n + 1)k 1 + p m,n 1 [N (m + n 1)]k 2 + p m 1,n [N (m + n 1)]k 3. (13) Here m and n denote the number of x, y molecules, respectively. The steady-state probability arising from Equation (13) follows a multinomial distribution [67,104]. To get an intuitive feeling and quantitative description of the probability flux, we explore the large molecular number regime where the underlying discrete chemical master equation in integer molecular number reduces to the Fokker Planck equation in continuous concentrations after second-order truncation [105]. We start from Equation (13) by replacing m, n with x = m/v, y = n/v, where V is the volume of the biochemical system. For convenience, we use p(x, y) to express p({x, y}, t) below. After some algebra and Taylor expansions, we obtain the stationary probabilistic flux J 0 [67]: N c x y x J 0 = K 3 K 1 N c x y + x K 3 K 1 1 2V x K 1 K 1 + K 2 + K 3 N c J SS p 0 (x, y) N c x y K 3 + x K 1 x K 1 N c x y K 3 + x K 1 K 1 + K 2 + K 3 N c J SS p 0 (x, y). (14) From Equation (11), the relationship between the probabilistic flux and chemical potential difference becomes [67] N c x y x J 0 = k 1 k 2 k K 3 3 K 1 N c x y + x K 3 K 1 k 1k 2 k 3 2V x K 1 (1 exp( G))p 0 (x, y) N c x y K 3 + x K 1 x K 1 N c x y K 3 + x K 1 (1 exp( G))p 0 (x, y). (15) It is clear that the stationary probability flux J 0 consists of two parts. The first is related to the drift or driving force of stochastic dynamics (appeared in the Fokker Planck equation), while the second is related to the diffusion tensor due to the fluctuations. From Equations (14) and (15), we can see that the probabilistic flux is directly correlated with the deterministic flux as well as the chemical potential difference. In this example, the probability flux is directly proportional to the deterministic chemical flux. When the concentrations of the energy providers do not obey the equilibrium condition ([D]/[E] k 0 1 k 2k 3 /k 0 1 k 2k 3 ), an energy

20 18 J. Wang pump emerges, ( G 0), resulting in the chemical potential difference and the deterministic flux, as well as the broken detailed balance. We can state that the energy pump is the origin of the non-equilibrium flux [67]. The relationship between the chemical potential difference and flux is similar to that for the electric current in electrical circuits under the voltage from a battery. This analogy can give us insights into how to study non-equilibrium biological systems with physical theory Flux and the potential landscape in the stable limit cycle We will now explore the origin of the flux in an oscillation system. We prove that in an oscillation system, it is also the energy pump that generates the chemical potential difference G which then produces not only the deterministic chemical reaction flux J SS, but also the probabilistic flux. Limit cycle in the reversible Schnakenberg model. We study an extension of the Schnakenberg model [106] where all the individual reactions are reversible [67]: X k 1 A, B k2 Y, 2X + Y k3 3X. (16) k 1 k 2 k 3 From the law of mass action, the deterministic equations of the model are dx dt = k 1A k 1 x + k 3 x 2 y k 3 x 3, dy dt = k 2B k 2 y k 3 x 2 y + k 3 x 3. Here the species B and A are assumed to be kept at constant concentrations, since they act as the source of energy pump in the non-equilibrium conditions (see Figure 2). In the biological context, these are related to ATP and ADP concentrations. For convenience, we term the ratio B/A as the strength of the energy pump. This is because the non-equilibrium concentration ratio of B/A provides a quantitative measure of how strong the non-equilibrium pump is. This is quantitatively realized through the Gibbs free energy G over the reaction loop ( G = 0 is the equilibrium condition). We choose the reaction parameters such that some of the backward rates are much smaller than the forward rates. This will keep the current reversible model consistent with the original Schnakenberg model [107]. k 1 = k 2 = k 3 = k 1 = 1.0, k 2 = k 3 = 0.01, A = (17) (18) From the linear stability analysis, in the regime of 0.1 < B < 0.9, a stable limit cycle emerges. In the regime of 0 < B <= 0.1 or B >= 0.9, a mono-stable cycle appears. The details are shown in Figure 3. 2 X B(input) k 2 Y k 3 3 X k 1 k -2 k -3 k -1 A(output) Figure 2. Scheme for the reversible Schnakenberg model. The dash curve in the box uncovers the self-catalytic mechanism of species X, and the dash box shows the key part that generates oscillation, while B and A are energy input and output through the non-equilibrium concentrations (from Ref. [67]).

21 Advances in Physics Steady States y Hopf bifurcation point Energy pump (B/A) Figure 3. The stationary solution y 0 of Equations (17) in three different regions of energy pump B/A. The unstable fixed states are marked as the dash line within 0.1 < B < 0.9, that is, 2 < B/A < 18, which refers to a limit cycle around the unstable fixed point. The star on the curve is noted as the Hopf bifurcation point. All parameters are set as in Equations (18) (from Ref. [67]). The effective variant form of the flux in the oscillation model. For this non-equilibrium open system, from the law of mass action, we can define the forward and backward reaction fluxes J j+, J j for each reaction. The chemical potential difference is then defined as G j = k B T ln J j+ /J j. This reversible model leads to a simplification B A. The total chemical potential difference becomes G AB = j G j = k B T ln(k 1 k 2 k 3 B/k 1 k 2 k 3 A). When the system reaches a steady state, the deterministic flux becomes J SS = Bk 2 y 0 k 2.In Figure 4, we show the deterministic flux varying with G ( G AB ) in the mono-stability regime. We can see that the non-zero chemical potential can generate a non-zero deterministic flux. It can also create a non-zero steady-state probability flux with an exponential relationship. This is similar to what we already have discussed analytically in the mono-stable model. That is to say, the non-equilibrium chemical potential difference G leads to a non-zero deterministic flux. When we increase the energy input, the mono-stable fixed point becomes unstable and a limit cycle emerges. For the limit cycle, the deterministic flux becomes oscillatory in time. To solve this problem, we average the deterministic flux with respect to time t over one period of the oscillation. This averaged deterministic flux is termed the effective flux, analogous to the effective electric current for an alternating current. In this way, we can compare the effective flux, instead of the oscillatory flux in time, with the probabilistic flux and other dynamic/thermodynamic variables. To study the relationship between probabilistic flux and deterministic flux, we need to map the vectors of the probabilistic flux onto the deterministic trajectory. We can then compare them using two similar quantities under different chemical potential differences G. From the definition G = k B T ln(k 1 k 2 k 3 B/k 1 k 2 k 3 A), the chemical potential difference as Gibbs free energy, is closely related to B/A, the non-equilibrium concentrations giving the non-zero energy pump G. It is important to understand how the energy pump changes the flux.

22 20 J. Wang J SS G Figure 4. The non-equilibrium chemical potential difference G can generate the non-zero deterministic flux J SS. The parameters we use are all shown in Equations (18), while B is chosen within (0, 0.1), and with these parameters this model shows one mono-stable steady state and leads to a steady-state deterministic flux J SS (from Ref. [67]). Chemical energy pump drives the flux. In the parameter regions 0.1 < B < 0.9, a limit cycle emerges. We find that in non-equilibrium conditions the energy pump, quantified by B/A, is the main source to keep the limit cycle going. Analogous to the case of the voltage pump that generates the electric current and drives it in an electrical circuit, the chemical energy pump here drives both the average deterministic flux and the probabilistic flux in a similar way. We calculate the steady-state probabilistic flux from the general case Equation (A6) in [67]. We show the flux together with the landscape and deterministic trajectory to illustrate the relationship to each other (Figure 5). We can clearly see that the flux flows inside the groove of the potential landscape and the direction is always pointing toward the deterministic trajectory. L Flux IntTra = J l L l. (19) We computed the loop integral of the probabilistic flux defined by Equation (19), where we mapped the stochastic flux to the deterministic trajectory and averaged the tangential component over the whole trajectory (red solid line in Figure 5). The result is shown by the solid line in Figure 6. The dashed line denotes the deterministic effective flux. We see that the trajectory average of the stochastic flux shows behavior similar to that of the deterministic effective flux as the chemical potential G increases. There is indeed a correlation between the two fluxes. These above results support our view that the energy pump is the origin of non-equilibrium flux. The energy pump breaks the detailed balance [67]. This directly generates the chemical potential difference. From the perspectives of chemical thermodynamics, it is the chemical potential difference that drives the molecules to start to react and transforms the chemical species into each other. This leads to the non-zero deterministic flux. For non-equilibrium systems, we discussed the probabilistic flux in concentration space, which can be used to describe the global nature of the dynamic system. Now we connect two kinds of fluxes together under an energy pump. We state that the energy pump from the environments breaks the detailed balance and

23 Advances in Physics 21 (a) Flux 8 (b) Y (B=0.2) Y (B=0.4) (c) 8 (d) Y (B=0.6) (e) 8 (f) (g) Y (B=0.8) X (h) Flux direction X Figure 5. Flux and its direction flowing on the landscape and along the deterministic trajectory. The red arrows in (a), (c), (e), and (g) are vectors of probabilistic flux for different energy pump strengths, and (b), (d), (f), and (h) show their directions. The black solid line refers to the deterministic trajectory. The gradual change color from red to blue represents hill to valley of the potential landscape, which is similar as in Ref. [13] (from Ref. [67]). drives the dynamics through the flux and potential landscape. This can influence the structure and stability of the system Non-equilibrium kinetic paths One of the most crucial aspects of the dynamics of complex physical and biological systems is how to go from one state to another and furthermore how fast to go from one state to another. We are going to address and quantify these two issues with the landscape and flux theory [14,61,62,82]. The dynamics of complex systems is often realized through specific pathways. Identifying and quantifying these paths will reveal how the dynamical processes actually occur and therefore uncovers the underlying mechanisms for the dynamics. There have been various studies on the kinetic paths of equilibrium systems. However, despite the significant efforts and progress made [22,38 52, ] in the zero-fluctuation limit, there are still open questions in non-equilibrium path studies, for example, under finite fluctuations as well as the underlying driving force identification in determining the paths. In this section, we will quantify the non-equilibrium paths via the landscape and flux theory [14,61,62,82] for small but finite fluctuations including the zero-fluctuation limit. There are often many possible routes for realizing the dynamics. Different paths give different contributions to the dynamics. Therefore, not all the paths are equivalent. The challenge then is

24 22 J. Wang Figure 6. The relationship of deterministic effective flux and loop integration of probability flux. Flux Ave represents effective deterministic flux and Flux IntTra is loop averaged probability flux which can be calculated by the definition in Equation (19), and G means chemical potential difference and also Gibbs free energy. The two fluxes show the similar behavior with changing of G, while the variation range G (10.3, 12.05) is corresponding to the limit cycle range B (0.15, 0.85) (from Ref. [67]). how to identify and quantify the paths which give the dominant contributions to the dynamics. We employ a path integral formulation for the non-equilibrium dynamical complex systems to quantify the kinetic paths and identify the dominant pathways [14,61,62,82]. There are six advantages of this approach. (1) The weights of each individual path can be quantified and the dominant paths can be identified based on the weights. (2) By varying the positions in the state space and quantifying the weights of the associated dominant paths connecting the states with the reference state, one can find the relative weights or probability landscape of the state space. (3) The path integral formulation can be directly used to quantify the kinetic rates from one state to another. (4) By identifying the dominant paths, the effective degrees of freedom and therefore the associated computation times of uncovering the dynamics are reduced from the original impossible exponentials to the manageable polynomials. (5) The identification and quantification of the kinetic paths make it possible to visualize how the dynamical processes are being realized step by step. (6) One can identify the key factors and driving force (landscape and flux) in determining the dominant paths. The dynamics of non-equilibrium complex systems as discussed before can be formulated as [14,61,62,82]: dxdt = F(x) + η. The stochastic fluctuation term is related to the intensity fluctuations either from the environmental external fluctuations or intrinsic statistical fluctuations from a finite number of components. We now formulate the path dynamics for the probability of starting at the initial state x initial at t = 0 and ending at the final state of x final at time t, with the path integral [14,53 55,61,62,82, ] as P(x final, t, x initial,0) [ = DxExp ( 1 dt 2 F(x) + 1 ( ) ( ))] dx 4 dt F(x) 1 dx D(x) dt F(x)

25 = = DxExp[ S(x)] [ DxExp Advances in Physics 23 ] L(x(t)) dt. (20) The integral over Dx refers to the sum over all possible paths connecting the initial state x initial at time t = 0 to the final state x final at time t. D(x) is the diffusion coefficient matrix (or tensor). The second term of the exponent gives the weight of a specific path due to the underlying Gaussian fluctuations. The first term of the exponent gives the contribution from the variable transformation of the Gaussian fluctuations η to the path x (the Jacobian). The whole exponent represents the weight of each individual path. Therefore, the probability of the system s going from the initial state x initial to the final state x final can be expressed as the sum of all the possible paths, each with a different weight. S(x) denotes the action and L(x(t)) denotes the Lagrangian for each path (Figure 7). It is expected that not all the paths give the same contribution. Sometimes we can approximate the transition probability with the path integrals coming from the dominant paths only. Since each path is exponentially weighted by the action, the contributions from sub-leading paths to the weights are from the next order in actions in the exponential. The differences of actions between dominant and sub-leading paths transform to the differences in weights [14,61,62,82]. The weight differences are exponentially amplified from the action differences. Therefore, when there is any significant difference in the actions of the paths, there will be a huge difference in the weights due to the exponential relationship between the weight and the action. Therefore, the sub-leading path contributions to the weight or probability are often significantly smaller than the dominant path contributions. Of course, in some cases, the actions of different paths are about the same. In such situations, every path contributes equally and there are no dominant paths. In order to quantify the probability, one has to sum over contributions from all the paths [14,61,62,82]. Let us explore the paths with the optimal weights from the above discussion. Since the probability or the weight is exponentially related to the action (by a negative sign), the dominant paths should have the least action which leads to the maximum weight. We should then minimize the action to identify the dominant paths. By minimization of the action, the dominant paths should satisfy the Euler Lagrangian (E L) equation (see Figure 7): (d/dt)( L/ ẋ) L/ x = 0. If the driving force F is a gradient of a potential U, F(x) = U(x), the E L equation is simplified as follows: ẍ 1 2 ( D(x)/ x/d(x))ẋ2 = 2D(x)( V(x)/ x), Where V(x) = (1/4D)F 2 Figure 7. Possible kinetic paths versus time: the kinetic paths from the initial position to final position are illustrated. The dominant path with the highest weight is illustrated along with another paths.

26 24 J. Wang 1 F(x) is the effective potential. The equation of motion of x has the acceleration term 2 ẍ, the frictional (positive and negative) term 1 2 ( D(x)/ x/d(x))ẋ2 and the driving force term 2D(x)( V(x)/ x). The determination of the dominant paths becomes equivalent to an n- dimensional particle moving in a force field with friction (n is the number of different variables in the system) [14,61,62,82]. In general, the driving force cannot be written as a pure gradient for non-equilibrium dynamical systems. In such cases, we can obtain the equation of the dominant path as follows from the minimization of the action [14,61,62,82]: ẍ 1 2 D(x)/ x ẋ 2 = E (ẋ F) + (ẋ )F, (21) D(x) where E = 2D(x)( V(x)/ x). This is similar to the equation when the driving force is a pure gradient except for the last two terms of the right side. One can check that the last two terms are equal to zero if the input driving force is a pure gradient. To see the origin of the last two terms, we look at the cross-product terms in the action: 1 2 (1/D) F ẋ dt = 1 2 (1/D) F dx. When the force is a gradient F = D U, this part of the action is a constant. It does not contribute to the dominant path equation. However, if the driving force is not equal to a gradient, then this part of the action is not constant. Furthermore, the loop integral back to itself will not be equal to zero and this part of the action becomes path dependent. It will contribute to the dominant path and is the origin of the two additional terms formed on the right side of the dominant path equation. As discussed, the driving force F of non-equilibrium complex dynamical systems in general can be decomposed into a gradient of a potential and an additional curl flow flux (F = DD P ss /P ss + DD + J ss /P ss = DD U + DD + J ss /P ss ). With detailed balance, the gradient of potential U determines the dynamics. For non-equilibrium systems, both the gradient of potential landscape U and the curl flux of probability determine dynamics [13]. Here we see that the dominant path is determined by both the gradient and the curl flux force. In fact, the path-dependent contribution to the action from the force 1 2 (1/D) F dx = 1 [(1/D)(Jss /P 2 ss ) + (1/D) D]dx. The path-dependent contribution is mostly from the divergent-free flux J ss due to its rotational curl nature. The curl flux is the origin of there being a non-zero action for the closed loop. This contributes to a real phase in the exponentials of the probability. The real phase gives the distinct probabilities classified by different topological windings. This real phase is similar to the Aharonov Bohm effect and Berry phase in quantum mechanics except that there the phase is imaginary. As we can see, the non-equilibrium dynamics has a deep connection to the topological nature of the trajectories of the underlying system. The origin of the differences in real phase versus imaginary phase comes from the fact that the classical non-equilibrium dynamics discussed here in general comes from the curl flux. This leads to the evolution equations containing non-hermitian operators while traditional quantum mechanics has Hermitian Hamiltonians and unitary evolution [14,78,82]. The above dominant path equation is general and works in any dimension. For easy understanding and visualization, let us consider the situation in three dimensions. When D(x) is a constant, the friction term is zero, the right side of the equation can be written as follows: ẍ = E +ẋ B, where B = ( F) (this mathematical form of and can only be written in equal or below three dimensions). The divergence of B is zero, B = 0. So in general, B is a rotational curl since there is no sink or source to go to or come out from (the right side of the divergence equation for B is zero). As we see, if the driving force of the dynamics F is a gradient force, then B = 0, and there is no curl component of the driving force. If F cannot be written as a gradient force, then in general B 0, and there is a curl flux component to the driving force. Without friction, the dominant path equation in three dimensions looks exactly like an electron

27 Advances in Physics 25 moving in an electric field E and magnetic field B. Notice that in classical electrodynamics, the electric field is the gradient of the electrostatic potential, while the magnetic field has no magnetic charge and therefore is divergence free. The solution of the dominant path equations is important for understanding the kinetic mechanisms of non-equilibrium dynamical systems. We can solve the dominant path equations directly using the boundary condition specifying the initial and final states. We can then quantify the dominant kinetic paths from the initial to the final state. But it is not always easy to solve the problem with two specified boundaries numerically, especially when the system is large. On the other hand, we can evaluate the weights of the kinetic paths from the path integral formalism directly. When the action S(x) is minimized, the most probable path can be obtained. The Lagrangian can be written as follows [14,82]: L(x) = 4Dẋ2 1 V(x) 1 F(x) ẋ. (22) 2D A generalized momentum can be written as follows: P(x) = L/ ẋ = (1/2D)(ẋ F(x)). The corresponding Hamiltonian of the system can be written as follows: From the above equation, we can obtain H(x) = L(x) + P(x) ẋ = E eff. (23) ẋ 2 4D + V(x) = E eff, (24) ẋ = 4D(E eff V(x)). (25) When we substitute Equation (23) into the action, we obtain S(x) = (P(x) ẋ H(x)) dt. We see that the action quantifying the weights of the paths depends on the values of the Hamiltonians. Specific values of the Hamiltonians or effective energies E eff correspond to specific final times T. For a given Hamiltonian, there is an optimal path associated with it which minimizes the action. Computing the above action in terms of the integral S HJ = i p i(x) dx i, a dot product in multidimensional space of x is still very challenging. It is also difficult to visualize. We see that the action can be simplified further and is equivalent to a line integral along a particular onedimensional (1D) path so that S HJ = i p i(x) dx i = p l dl, where p l = (E eff V(x))/D (1/2D)F l. This transformation allows us to switch from the time-dependent description to the Hamiltonian-dependent HJ description [119,120]. The dominant pathways connecting the given initial and final coordinates can therefore be obtained by minimizing the action in the HJ representation: ( ) xf (Eeff V(x)) S HJ = 1 D 2D F l dl, (26) x i where dl is an infinitesimal displacement along the 1D path. E eff is an effective energy parameter which determines the total time elapsed during the transition, according to xf 1 t f t i = dl 4D(E eff V(x). (27) x i Here, we have adopted a simple choice E eff = V(x max ) (V(x max ) corresponding to the extreme (maximum) of effective potential V = 0. This leads to the longest kinetic time. Using the HJ formulation of the dynamics in length space is much more efficient in avoiding long time traps

28 26 J. Wang than the conventional approach. This is realized by considering intervals of fixed displacements in length space rather than fixed displacements in time. The numerical advantages of the HJ formalism for describing long-time dynamics at constant energy were applied to equilibrium systems where dynamics is determined by the gradient energy [119,120]. Here, the computational advantages can be achieved for non-equilibrium stochastic dynamical systems, when the underlying dynamics of the system cannot be written as a pure gradient of the potential landscape alone (determined by both gradient of the potential and curl flux). In the small fluctuation limit, we can identify the fixed points of the deterministic equations. The stable fixed points correspond to the basins of attraction of the underlying landscape. When the system has only one basin of attraction, we can choose the basin position as the reference state (final state). By exploring the state space through varying the initial states, we can find the effective actions S and the associated weights W of each state relative to the reference state: W(x) = exp[ S(x)] (under dominant path approximations). From this, we can quantify the generalized potential landscape from the weights for a system with mono-stability: U(x) = ln W(x) = S(x). When the system has multiple basins of attraction, we can then choose all the basin positions as the reference states (final states, x f 1, x f 2,..., x fn ). For any initial state, we can calculate the actions S relative to each basin: S 1 (x), S 2 (x),..., S N (x), and the associated weights, W: W 1 = exp[ S 1 (x)], W 2 = exp[ S 2 (x)],..., W N = exp[ S N (x)]. We can then select the least action S or the dominant (largest) weight W from this state relative to the reference states as the action or the weight of this state. By exploring the state space through varying the initial states, we can finally find the least action or dominant weight associated with each state relative to the reference states of the basins x f 1, x f 2,..., x fn. In this way, we can quantify the generalized potential landscape from the weights for the whole system with multi-stability: U(x) = ln S optimal (x). The advantage of quantifying the landscape from the dominant path approximation via the path integral is that the computational task reduces from exponential of M N to polynomial of M N, where M is number of intervals for each variable and N is the number of variables. Another advantage of quantifying the landscape from the dominant path approximation via path integral is that there is no assumption regarding the coupling among variables. Therefore, this way of quantifying the landscape can work even when the couplings among variables are strong. This is in contrast to the cases where another approximation in terms of the self-consistent mean field for reducing the large dimensionality is often used. The mean field approximation only works for the weak coupling regime Non-equilibrium transition state rates For complex physical, chemical, and biological systems, quantifying the rate of transition from one state to another is critical for understanding the behavior, function, and global stability [121,122]. Although the analytical quantification of kinetic rates from TST or Kramers rate theory has been successfully applied to equilibrium systems, the analytical quantification of kinetic rates for non-equilibrium systems is still challenging. The main efforts have been focused on the zero noise limit [22,38 52,82,108]. For equilibrium systems, the global stability can be quantitatively studied when the underlying potential landscape is known a priori. The dynamics and the corresponding dominant kinetic paths between different states follow the gradient. The transition state or Kramers rate theory for kinetic rate for equilibrium systems is determined mainly by the barrier between the stable states characterized by the basins of attraction. Here the barrier height is determined by the difference in potential energy between the stable fixed point and the saddle point on the underlying potential landscape. The kinetic rates also depend on the fluctuations around one stable basin and also

29 Advances in Physics 27 the saddle point between the basins of attractions. This theory for kinetic rates was proposed by Eyring from a chemistry perspectives and by Kramers from a physics perspective on thermally activated barrier crossing more than 70 years ago [82,123,124]. This provides a good analytical quantification of the kinetic rate from one attractor to another for equilibrium systems. However, for general non-equilibrium dynamical systems, the equilibrium TST or Kramers rate formula are expected to fail. This is because the dominant kinetic paths do not follow the gradient path of the underlying non-equilibrium potential landscape, as electrons moving in an electric field (potential landscape gradient) in the equilibrium case do. Instead, the dominant kinetic paths deviate from the gradient paths due to the presence of the curl flux force that breaks detailed balance. The non-equilibrium dynamics is like that of an electron moving in both electric (potential landscape gradient) and magnetic (flux) fields [14,61,62,82]. The equilibrium transition state rate can be analytically quantified through the path integral formulation [125]. However, the analytical quantification of the transition rate for nonequilibrium systems is still challenging. It has been argued that in the zero noise limit, the dominant kinetic path will pass through the saddle point between the basins of attraction. A corresponding analytical approximation of the escape rate from a stable basin can be found [38,43]. However, for general non-equilibrium systems, finite fluctuations are often present. The dominant kinetic paths do not necessarily go through the saddle points due to the presence of the curl flux, breaking the detailed balance as mentioned previously [14,61,62,82]. In this section, we develop an analytical TST for the kinetic rate of general non-equilibrium dynamical systems [14,61,82]. In this theory, (1) we first quantify the dominant path according to the path integral by minimizing the action. Here the starting point and the ending point for the path integral are the two stable fixed points S and S. (2) Due to the presence of the non-zero curl flux, the dominant path will not follow the gradient path of the underlying landscape. Furthermore, under finite fluctuations, the dominant paths may not go through the original saddle point Ŝ. (3) On the dominant path, we search for Ŝ, the new saddle, which will not likely be at the original saddle point Ŝ of the driving force. (4) The action of the path integral from S to S obtained in (1) is less than the action through the original saddle Ŝ along the gradient path of the underlying landscape. The action calculated from the stable fixed point to the new saddle along the dominant path provides the exponential part of the new non-equilibrium transition state rate theory. For comparison, in conventional TST for equilibrium systems, the kinetic rate is determined by the barrier at the saddle point between the basins of attraction on the underlying landscape. In the new non-equilibrium TST, the prefactor part of the transition rate reflects the fluctuations around the stable basin and the local curvature around the new saddle along the optimal path. The present non-equilibrium TST can be applied to general non-equilibrium, physical, chemical, and biological systems Equilibrium transition state rate In this section, we will first give a review of the transition state and Kramer s rate theories for the equilibrium systems as preparation of describing a TST for non-equilibrium systems. The fluctuations in complex dynamical systems are often not constant and location dependent. The corresponding stochastic dynamics can be quantified in continuous spaces by Langevin equations (in Ito s form): ẋ μ = F μ ( x) + a Ba μ ( x)ξ a (t), where x denotes the dynamical variables of the system. F μ ( x) represents the driving force, ξ a (t) denotes the Gaussian distributed white noise with unit fluctuations. B a μ ( x) denotes the strength of the location-dependent fluctuations: ξ a (t)ξ b (t ) =δ ab δ(t t ). Again, instead of following un-predicable individual stochastic trajectory, we focus on the predicable probability distribution P( x, t) obeying the Fokker Planck

30 28 J. Wang equation [2]: dp dt = μ μ ( F μ P) + μ,ν 1 2 μ ν (ε μν P) (28) with the diffusion coefficient ε μν ( x) = a,b Ba μ ( x)bb μ ( x)δab. We adopt the notation μ / x μ. For convenience, we also denote P( x) P( x, t) as the time-dependent probability distribution and P SS ( x) as the time independent steady-state probability distribution. Under the intrinsic noise from statistical number fluctuations, the resulting Fokker Planck equation, with location-dependent diffusion coefficients, can be derived from a second-order expansion in the noise of the underlying master equations in the noise [105]. The Fokker Planck equation can be rewritten in terms of probability conservation law. The change in local probability is equal to the net incoming or outgoing probability flux [14,61,82]: dp( x, t)/dt = j. When the steady-state flux: F μ ( x)p SS ( x) ν 1 2 ν[ε μν ( x)p SS ( x)] = jμ SS( x) is zero: j SS = 0, the system is in detailed balance since there is no net flux in and out of the system. In this case, the system is in equilibrium. The equilibrium probability distribution is closely linked to the underlying potential by the Bolzmann law. The associated driving force can be determined by the gradient of the equilibrium potential: U = ln P eq and F μ ( x) = 1 2 μ[u( x)] + ν 1 2 ν[ε μν ( x)]. The last term reflects the contribution from the inhomogeneity of the diffusion coefficients in x. For general non-equilibrium systems, the net flux in and out of the system is not zero, j SS 0. Detailed balance is broken. The steady-state flux satisfies the condition j SS = 0 at steady state. This divergence free condition of the steady-state flux reflects its rotational curl free nature. The curl flux quantifies how far the system departs from equilibrium. For non-equilibrium dynamical systems, the global stability and dynamics (F μ ( x) = 1 2 μ[u( x)] + ν 1 2 ν[ε μν ( x)] + j SS /P ss ) are determined by both the steady-state probability distribution defining the non-equilibrium landscape U = ln P ss and the curl probability flux: jμ SS( x). 1D systems are integrable with zero flux j SS = 0 with natural boundary condition. Transition rates from one basin to another basin based on TST turn out to be given by [82]: r eq K U (S)) U (Ŝ) = e 2[U(Ŝ) U(S)]/ε. (29) 2π U(x) is the potential energy function in the equilibrium system. The associated driving force is the gradient of this potential energy: F(x) = U (x), as shown in Figure 8. The basin of the attraction of the underlying potential energy landscape is located at S and the saddle point is located at Ŝ. The saddle point locates the barrier between the basins of attraction. In the small fluctuation limit ε 0, the transition state rate in Equation (29) can be rewritten as follows [126]: r eq K = (2π) 1 df dx (S)dF (Ŝ)e SDOM HJ, (30) dx where SHJ DOM = Ŝ S p dx is the HJ weight action from S to Ŝ along the dominant path [14,82,125]. The p denotes the canonical momentum and dx denotes the variable displacement of the system. The underlying physical picture is quite clear. The transition state rate for equilibrium process is determined by two key factors. The dominant contribution comes from the exponential of the action. The rest is a prefactor that comes from the fluctuations around stable and saddle (transition state) points of the equilibrium potential landscape. The 1D transition state rate given in Equation (29) has been generalized to the N dimensions [124,127,128] for equilibrium systems.

31 Advances in Physics 29 Figure 8. The potential barrier U for calculating the transition state or Kramers escaping rate. The basins of attractions are localized at S and S. Ŝ is the saddle point (from Ref. [82]). However, for general non-equilibrium systems, the driving force F is not a pure gradient of a potential U, F μ ( x) μ U( x). Furthermore, without the detailed balance, the curl current flux j ss is not zero. The non-zero flux contributes to the driving force. This leads to the deviation of the kinetic path from that of the pure gradient, resulting in a path-dependent HJ weight action [14,82]. Therefore, the dominant paths do not necessarily go through the saddle point or transition state. Then, the new transition state has to be specified. Therefore, finding the rate for general non-equilibrium systems demands specifying the dominant kinetic path as well as the new saddle on the path and the complete form of the weight action needs to be computed in order to quantify the rates from the new transition states. SHJ DOM The exponential contribution of the non-equilibrium transition state rate For an N-dimensional non-equilibrium system, we can develop a TST for kinetic rates that goes beyond the equilibrium expression given in Equation (30). The generalized weight action for non-equilibrium systems is [82] S = t f t i dtl with the Lagrangian L = εμν 1 2 (ẋ μ F μ )(ẋ ν F ν ) ε μχ χ (F ν εμν 1 ). (31) μν μνχ In the zero-fluctuation limit ε 0, this action S = t f t i dtl leads to the exponential part of Freidlin Wentzell s theory [38]. Furthermore, in the zero-fluctuation limit, the weight ratio of e S l 1 /e S l 2 between the two smooth paths l 1 and l 2 agrees with the Onsager Machlup function [115]. The optimal path that contributes the most to the path sum can be quantified by minimizing the weight action S with respect to the paths x μ (t) s. We then obtain the equation of motion for the dominant path through its satisfying the E L equation (d/dt)( L/ ẋ α ) = L/ x α. The dominant path approach gives the lowest order approximation of the full path integral weight action. When the fluctuations are relatively small, this starting point provides a practical way to quantify the process in the large dynamical systems. Other sub-leading terms are exponentially suppressed compared to the leading order contribution [14,82]. Rather than directly solving the E L equation of motion, the dominant kinetic path can also be evaluated by minimizing the weight action S directly. Let us define the canonical momentum

32 30 J. Wang p μ = L/ ẋ μ = ν ε 1 μν (ẋ ν F ν ), then the total energy becomes [14,82] E = H = L p μ ẋ μ. (32) This quantity is a constant along the dominant kinetic path. Then, the HJ weight action [129],the minimization of which giving the dominant paths, can be written as follows: S HJ (x i, x f ) = xf x i xf 2(E Veff )dl x i μν ε 1 μν F ν dx μ. (33) Notice that the above action is now simplified as a line integral along the dominant path dl = μν ε 1 μν dx μ dx ν in a curved space with distance measure εμν 1, where the ε μν quantifies the fluctuation or diffusion strengths The transition state rate theory for non-equilibrium systems As shown and discussed in Figures 9 and 10, the forward and backward dominant paths (lines with arrows) are irreversible and do not pass through the saddle point Ŝ on the gradient path (white lines) on the landscape [14,82]. From the effective driving force Fμ eff = ν ε 1 μν F ν of the second term on the right-hand side of Equation (33), we can always find a saddle or more accurately the global maximum along the dominant path Ŝ using the fact that the projection of the force (S ) along the path is zero, as shown in Figure 9 in two dimensions and Figure 10 in three Fl eff dimensions. The Fl eff will always change its sign moving from the neighborhood of S (pointing to S) toward the neighborhood S (pointing to S ). Quite often, we only have one saddle point along the dominant path. When there are multiple new saddles along the path, we pick the last one before the trajectory reaches the ending stable fixed point S as Ŝ. Therefore, by replacing the saddle point Ŝ for equilibrium systems with the new saddle point Ŝ for the non-equilibrium system, we can finally derive a transition state rate expression analytically for non-equilibrium Figure 9. 2D illustration of non-equilibrium landscape with the irreversible dominant transition paths between basins S and S (green lines with arrows) and the gradient path (white line). Here, Ŝ is the saddle point and Ŝ is the global maximum along the dominant path (from Ref. [82]).

33 Advances in Physics 31 Figure 10. Three-dimensional ( 3D) illustration of non-equilibrium landscape with the irreversible dominant transition paths between basins S and S (purple lines with arrows) and the gradient path (white line). Here, Ŝ is the saddle point and Ŝ is the global maximum along the dominant path (from Ref. [82]). systems as follows [14,82]: r noneq K = (Eτ) 1 = λ u(ŝ ) 2π detm (S) detm (Ŝ ) e SDOM HJ. (34) In the exponential factor, the weight action SHJ DOM = Ŝ S p dx, as defined in Equation (33) is integrated along the 1D dominant path l from the stable basin S to the new saddle Ŝ. For the prefactor part of the expression, we can give a derivation in a similar spirit as for the zero noise limit [43]. We call λ u (Ŝ ) the positive eigenvalue of the force matrix involving the force derivatives with respect to the coordinates, F μ,ν (Ŝ ) = ( F μ / x ν )(Ŝ ) at the new saddle Ŝ. This gives a quantitative measure of the width of the contributing fluctuations at the new saddle Ŝ along the dominant path. At the stable state S, we can find the stationary solution for the Fokker Planck equation. The associated fluctuation matrix M (S) satisfies an algebraic Equation (35) at the stable state S: ε ξχ M,μξ M,νχ + M,μξ F ν,ξ + M,νξ F μ,ξ = 0. (35) ξχ ξ ξ Since the new saddle Ŝ is not a fixed point of the system (force F 0), there is no stationary solution for the Fokker Planck equation at Ŝ. Therefore, the matrix M (Ŝ ) representing the second-order fluctuations satisfies the dynamic equation at Ŝ : dm μν (x) dt = 2 H M μξ M νξ p ξ p ξ 2 H x μ x ν 2 H x ν p ξ M μξ 2 H x μ p ξ M νξ. (36) From the expression in the rate formula, the detm (S) represents the second-order fluctuations in terms of frequencies of all stable modes around stable basin state (in all directions),

34 32 J. Wang while detm (S ) represents the second-order fluctuations in terms of frequencies of all stable modes and unstable modes (in all directions) around the saddle point on the dominant path Ŝ. detm (S)/ detm (Ŝ ) represents the ratio of the fluctuations around the saddle and the stable basin state along the dominant path. On the other hand, the λ u (Ŝ ) determines the fluctuations of the single unstable mode at the saddle point Ŝ. In this rate expression for non-equilibrium dynamical systems, the main contribution is from the exponential with the weight action from stable basin state to the new saddle on the dominant path. On the other hand, the non-exponential prefactor gives a second-order correction for the local fluctuations at the stable point S and the saddle on the dominant path Ŝ. In conventional TST for equilibrium systems, the kinetic rate is determined by the potential energy barrier at the saddle point between the basins of attraction (potential energy difference between the saddle point and stable basin on the landscape). On the other hand, in the transition state rate theory for non-equilibrium systems, the kinetic rate is determined by the weight action from the stable basin to the new saddle along the dominant path. It is important to realize that the non-equilibrium saddle point is path and directional dependent. In other words, the forward and backward paths do not share the same saddle point. In contrast, in the conventional equilibrium case, the forward and backward dominant paths all pass through the same saddle point on the landscape as shown in Figure Non-equilibrium thermodynamics For equilibrium systems, once the potential interaction energy or Hamiltonian is known, we can find the equilibrium probability distribution directly using the Bolzmann relationship. We can furthermore construct the partition function from which the entropy and free energy of the system can be quantified. For non-equilibrium systems, the potential function is not known a priori. The challenge then becomes: Can we construct an appropriate thermodynamics for a non-equilibrium dynamical system? Progress has been made in non-equilibrium thermodynamics [3,4,9 11,26,69,130]. We want to formulate the non-equilibrium thermodynamics with the landscape and flux perspective. As we have already seen earlier, the non-equilibrium intrinsic potential is closely related to the steady-state probability distribution at the zero-fluctuation limit, P ss (x) = P ss (x) D 0 = exp( φ 0 /D)/Z, where D = D D 0 and Z is the intrinsic partition function defined as Z = exp( φ 0 /D) dx [26,69,71,81]. From this, we can reach φ 0 = D ln(zp ss ). We note that the intrinsic partition function Z is independent of time t. Analogous to the equilibrium system, we are allowed to define the intrinsic entropy of the deterministic non-equilibrium dynamical system as follows [3,4,9 11,26,69]: S = P(x, t) ln P(x, t) dx, where P(x, t) = P(x, t) D 0. Moreover, we can define the intrinsic energy E of the nonequilibrium dynamical system as follows: E = φ 0 P(x, t) dx = D ln(zp ss )P(x, t) dx. A natural definition of the intrinsic free energy F of the non-equilibrium system becomes F = E DS = D[ P(x, t) ln(p(x, t)/p ss ) dx ln Z]. For thermodynamic reasoning, we now would like to ask whether or not the entropy of the non-equilibrium system is maximized. We will then explore the time evolution of the entropy [3,4,9 12,26,69]. The change of the entropy in time can be decomposed into two parts: Ṡ = Ṡ t Ṡ e, where the entropy production rate (EPR), Ṡ t = dx(j (DD) 1 J)/P, becomes positive or zero. Note that entropy production is closely related to the probability flux. The flux is the origin of the entropy production. On the other hand, the heat dissipation rate or entropy flow rate from the environments to the non-equilibrium system can be either positive or negative: Ṡ e = dx(j (DD) 1 F ), where the effective force is defined as F = F D D. We can interpret Ṡ as the entropy change of the non-equilibrium system. Ṡ t then has the physical meaning of the total entropy change of both the system and its

35 Advances in Physics 33 environments. It is always larger or equal to zero. This is consistent with the second law of thermodynamics. Furthermore, we can see that the entropy of the non-equilibrium system by itself can be increased or decreased due to the entropy flow from or to the environments. Therefore, the temporal change for the entropy of the system can be positive or negative. This provides the chance to create order by decreasing the system entropy. So the entropy of system for the non-equilibrium is not always increasing and therefore is not always maximized. Furthermore, We can investigate the derivative of the intrinsic free energy with respect to time [3,4,9 12,26,69,71,81]: = D 2 df dt ( ) ( ) P P ln D ln P dx 0. (37) P ss P ss The above equation indicates that the intrinsic free energy of the non-equilibrium system F always decreases in time until reaching the minimum value F = D ln Z. This can be understood as one form of the expression for the second law of thermodynamics of non-equilibrium systems. Therefore, the intrinsic free energy is a Lyapunov function and can also be used to explore the global stability of the non-equilibrium system. As we see, although the system entropy is not necessarily maximized (only the total entropy keeps increasing), the intrinsic free energy of the system does minimize itself. This might provide a design principle for the complex dynamical systems to search for their optima. We can now also investigate the non-equilibrium nature of the steady state. We can do so by the study of how the intrinsic energy, entropy, and free energy of the non-equilibrium systems change with respect to the fluctuation strengths D and the underlying system parameters. The intrinsic system entropy S ss = P ss (x) ln P ss (x) dx, the intrinsic energy E ss = φ 0 P ss (x) dx = D ln(zp ss )P ss (x) dx, and the intrinsic free energy F ss = D ln Z at the steady state can be naturally defined with the probability in time now replaced as the steady-state probability as shown above [11,26,56,69,71,81]. Therefore, the intrinsic free energy of the non-equilibrium system at steady state becomes F ss = D ln Z = E ss DS ss. We can take this as the first law of thermodynamics for non-equilibrium systems. We see that the fluctuation strength D here plays the role of temperature for the non-equilibrium systems analogous to that for the equilibrium case. We can also explore the non-equilibrium thermodynamic behavior with respective to the changes of D. We expect entropy and disorder to dominate at high fluctuations, while energy and order dominate at low fluctuations. Therefore, non-equilibrium phase transitions might occur from disorder to order as fluctuation decreases or vice versa. The non-equilibrium phase transition might also occur when the parameter changes of the systems influence significantly the energy entropy balance and therefore the behavior of the system free energy. When the fluctuations D are finite, we can also define and construct the non-equilibrium entropy, energy, and free energy of the corresponding stochastic dynamical systems in the following way. The entropy is P ln P dx; The energy is E = DUP dx, where U is the nonequilibrium population potential and is related to the steady-state probability as U = ln P ss ; The free energy is F = E DS = DUP dx D[ P ln P dx] = D[ P ln(p/p ss ) dx ln Z]. The above forms the non-equilibrium stochastic thermodynamic first law for dynamical systems with finite fluctuations [69,71,81]. We can also check and see the behavior of the total entropy production (Ṡ t = dx(j (DD) 1 J)/P0) as well as the free energy change of the stochastic dynamical systems with respect to time with finite fluctuations (df/dt = D 2 ln(p/p ss ) D ln(p/p ss )P dx 0). We can see that the total entropy production is always greater than or equal to zero, while the free energy of the system is always less than or equal to zero. This is the second law of non-equilibrium stochastic thermodynamics for dynamical systems with finite fluctuations. Therefore, the free energy is also a Lyapunov function monotonically decreasing in

36 34 J. Wang time. At the non-equilibrium steady state, the non-equilibrium free energy becomes [69,71,81] F ss = E ss DS ss = DUP ss dx D[ P ss ln P ss dx] = D ln Z. Here the partition function Z is defined as the form Z = exp( U) dx which is related to the non-equilibrium population potential U FDT for intrinsic non-equilibrium systems The FDT plays a key role for systems near equilibrium with detail balance [131,132]. The FDT connects the fluctuations of the system quantified by the correlation function at equilibrium with the response or relaxation of the system to that equilibrium quantified by the response function. Efforts have been made to extend the FDT to non-equilibrium systems [15,27,28, ]. It has been found that the FDT involves the correlation function of a specific variable conjugate to entropy [25]. Moreover, by choosing proper observables, the FDT for non-equilibrium systems can be formulated [146]. In this study, we illustrated a way to generalize FDT for non-equilibrium processes based on the landscape and flux theory, focusing on the direct observables such as x i, under stochastic dynamics [15]. The stochastic dynamics in continuous observable can be described by Langevin dynamics or Fokker Planck equations. We find that the response function can be decomposed into two parts. One contribution is from the correlation of the observables themselves representing the spontaneous relaxations to the steady state. This contribution also exists in systems with detailed balance. The only difference is that state the system relaxes to is not the equilibrium state but the non-equilibrium steady state without detailed balance. The other contribution is related to the heat dissipation. It represents the contribution from breaking detailed balance and is directly related to the curl flux part of the force. Remarkably, non-equilibrium thermodynamics [10,12,147,148] can be derived from the generalized FDT in the equal time limit [15]. Again stochastic dynamics in continuous representation can be formulated by Langevin equations [15]: ẋ i = F i (x) + B ij (x)ξ j (t), (38) where F i (x) represents the driving force and ξ i (t) represents the Gaussian white noise describing the statistics of the stochastic fluctuations from the environments to the system: ξ i (t)ξ j (t ) = δ(t t ), adopting the Einstein convention for the notations. The temporal evolution of probability distribution obeys the Fokker Planck equation [15]: Ṗ(x, t) = ˆL(x)P(x, t) (39) with the operator ˆL(x) = [ i F i (x) + i j D ij (x)] and the diffusion coefficient D ij (x) = 1 2 (BBT ) ij (x). For convenience, we adopt the notation i / x i, P(x) P(x, t) to represent the time-dependent probability distribution and P SS (x) to represent the time independent steady-state probability distribution. F i (x)p(x) + D ij (x) j P(x) = j i (x), (40) where F i = F i j D ij. Then, Fokker Planck equation again can be rewritten in the following form: dp(x, t)/dt = j. The system is in detailed balance if the steady-state flux is zero: j SS = 0: j SS i (x) = F i (x)p SS (x) + D ij (x) j P SS (x). (41) In non-equilibrium systems, detailed balance is broken: j SS 0. The steady-state flux is therefore not zero. From the steady-state conditions, the steady-state flux is a divergence free vector with j SS = 0[15]. The driving force F j (x) can then be decomposed of two contributions:

37 Advances in Physics 35 a potential gradient part D ij (x)( / x i )U(x), where U(x) = ln P ss (x) and a curl flux part jj SS (x)/p SS (x) v SS j (x). We can define the probability velocity of the steady state as v SS i (x). On other hand, we can also decompose the gradient of potential ln P SS (x) into the driving force and the curl flux [15]: i ln[p SS (x)] = D 1 ij (x)[ F j (x) v SS j (x)]. (42) FDT for equilibrium systems with detailed balance was investigated using perturbation approach [132]. Here we will extend perturbation approach to non-equilibrium systems. By applying a linear perturbation on the driving force of the system: F i (x) F i (x) = F i(x) + h(t)δf i (x), we see the change of the temporal evolution operator that follows as a consequence of the perturbations in the driving force ˆL ˆL = ˆL h(t)δ ˆL, with δ ˆL = δf i (x) i + i δf i (x). The corresponding change of the probability evolution and the average of the observed variable become [15]: [ t ] P(x, t) = exp dt(ˆl h(t)δ ˆL) P(x, t ), t δ (t) = (t) = dx (x)[p(x, t) P SS (x)]. (43) Therefore, for t t, the response function of the system measured by the change of the average of observable with respect to the perturbation applied on the force reads as [15] R (t t ) = δ (t) δh(t ) = dx (x)eˆl(t t ) ( δ ˆL)P SS (x). (44) δf=0 Applying the decomposition in Equation (42), we can obtain the response function for the stochastic non-equilibrium systems [15]: R (t t ) = dx eˆl(t t ) {δf i [ F k v SS k ]D 1 ik i δf i }P SS = (t) i δf i (t ) [ (t)δf i (t ) F k (t )D 1 ik (t ) + (t)δf i (t )v SS k (t )D 1 ik (t ) ]. (45) The above expression we obtained provides a general relation between the response functions and the correlation functions of stochastic non-equilibrium dynamical systems. Here, the correlation between the two observables Ω 1 and Ω 2 is defined as [15] C Ω 1 Ω 2(t, t) = Ω 1 (t )Ω 2 (t) Ω 1 (t ) Ω 2 (t) with Ω 1 (t )Ω 2 (t) = P SS (x i )Ω 1 x i Ω 2 x j P(x i, t x j, t). Here, P(x i, t x j, t) is the transition probability from initial state x i at time t to final state x j at time t. For a variable or coordinate x independent (constant) perturbation of particular component i: δf i = δ i i, we obtain [15] R i (t t ) = (t) i ln[p SS (x)] = [ (t) F k (t )D 1 ik (t ) + (t)v SS k (t )D 1 ik (t ) ]. (46) This is a generalized FDT for non-equilibrium systems. Notice that the generalized FDT applies to any number of dimensions not just in one dimension [147,148]. From the force decomposition in Equation (42), the response or relaxation to the steady state of the stochastic non-equilibrium system can be decomposed to two parts. The first contribution, analogous to that for equilibrium

38 36 J. Wang systems, is related to the usual correlation function of the observable with the driving force. This contribution exists in FDT even for equilibrium systems obeying the detailed balance. In equilibrium systems with detailed balance, the driving force can be explicitly expressed as the gradient of the logarithm of the steady-state probability. However, the second contribution does not occur in equilibrium systems. It is directly related to the non-zero flux which violates detailed balance. The curl flux gives a quantitative measure of the degree of the non-equilibrium-ness or how far away the system is from equilibrium. In other words, for a non-equilibrium system, the response or relaxation depends both on steady-state fluctuations and also a contribution from the curl flux. For constant diffusion coefficients D ij for simplicity when j = 0, the system is in equilibrium with detailed balance and time reversal symmetry [15]: in this case (t)f j (x(t )) = F j (x(t)) (t ). Applying the Langevin equation (38), F i (x(t)) (t ) = [ẋ i (t) ξ i (t)] (t ) = ẋ i (t) (t ), due to the fact that the stochastic random force from the environment does not correlate with the system variable of the previous times (t > t ): ξ i (t) (t ) =0. Then, we can see [ ] d R i (t t ) = D 1 ik dt x k(t) (t ). (47) For the specific observable operator (x) = x j, we find R x j i (t t ) = D 1 ik [ d dt x k(t)x j (t ) ]. (48) We see that the force-observable correlation now becomes the temporal change of the correlations of the observable variables. The equilibrium FDT then becomes that the response or relaxation to the equilibrium system is equivalent and can be measured by the equilibrium fluctuations. This is the FDT near equilibrium [132]. We should point out that the FDT in Equation (46) can also be generalized to the case where the system is not in steady state but in an arbitrary state with distribution P(x). For t t, we can then arrive at [15] R i (t t ) = dx (x)eˆl(t t ) ( δ ˆL)P(x) (49) = [ (t) F k (t )D 1 ik (t ) + (t)v k (t )D 1 ik (t ) ] with the probability velocity of the general state: v i (x) j i (x)/p(x). Now we will give a new way of deriving the non-equilibrium thermodynamics for the stochastic dynamical systems. Let us choose the observable in the generalized FDT as = v i (x) and sum over i from Equation (49), the response function in equal time limit t = t then becomes [15] R v i i (t) = dxv i (x)[ i P(x)] = dx[ j(x)]lnp(x) = d dxp(x) ln P(x) = Ṡ dt = [ v i (t) F k (t)d 1 ik (t) + v i(t)v k (t)d 1 ik (t) ]. (50) Therefore, the entropy production of the system S = d(x)p(x) ln P(x) has two contributions: Ṡ = v i i ln[p(x)] = v i D 1 ij v j + v i D 1 F j =e p Ṡ m. (51) ij

39 Advances in Physics 37 e p 0 is the average total EPR of the system and environments. TṠ m = Tṡ m is the average heat dissipation rate to the environment or medium. The heat dissipation rate for the environment is q = F i ẋ j D 1 ij = Tṡ m, where the exchanged heat q between the system and the environment can be identified with the increase in entropy s m in the environment with temperature T [147,148]. The system entropy change rate Ṡ is closely related to the gradient of the time-dependent probability distribution: i P, which is composed of two parts. One contribution is from the total bulk entropy production of the system and environment which is associated with the curl flux v. The other part is from the heat dissipation into the environment (surface) which is associated with the driving force F [147,148]. We can see that the driving force for entropy production is the curl flux. With detailed balance, the only contribution to entropy production comes from the timedependent flux. Without detailed balance, entropy production has both contributions from the time-dependent and the steady-state flux. We can separate the contribution of the time-dependent and time independent parts of the curl flux to the entropy production of the system. We can further relate these to the relaxation of time-dependent probability and steady-state flux explicitly. Let us take the observable = v i (x) v SS i (x) and sum over i, the response function in Equation (49) with equal time limit t = t to yield [15] [ P SS ] (x) v i i ln P(x) = Ḟfree T = v SS i D 1 ij v j v i D 1 ij v j = Q hk T e p. (52) This leads to Te p = Q hk Ḟ free with free energy defined as F free = T ln[p(x)/p SS (x)] =U TS, the house-keeping heat defined as Q hk = T v SS i (t)v j (t)d 1 ij (t) =T v SS i (t)v SS j (t)d 1 ij (t) = Te p + Ḟ free = TṠ m + U and total energy defined as U = T dxp(x) ln[p SS (x)][10,12,147,148]. The change of the total internal energy is therefore U = T v i (t) i ln[p SS (x)]. We can see that there are two different origins of the total entropy production e p. Ḟ free arises from spontaneous non-stationary relaxation associated with the gradient of relative potential i ln[p(x)/p SS (x)]. Q hk is the driving force necessary to sustain the non-equilibrium environment, which is associated with the steady-state flux v SS (x). For the non-equilibrium steady state, Ḟ free = 0. Q hk is equal to the environment or medium dissipated heat for maintaining the violation of detailed balance [15]: Q hk = TṠ m = T v SS i D 1 ij F j. When the system is in equilibrium with detailed balance, Q hk = 0. The total entropy production of the system is equal to the spontaneous relaxation of free energy Te p = Ḟ free. Therefore, we find that the generalized FDT in the equal time limit t = t naturally leads to non-equilibrium thermodynamics with total entropy production coming from the contribution of both non-stationary spontaneous relaxation and stationary house-keeping part [12,15] Gauge field, FDT Symmetry has played a very important role in physics and chemistry. All the current fundamental physical laws are the result of symmetry and symmetry breaking. So investigating the symmetry of a system will help us to uncover the underlying physical laws of that system. We devote this section to discuss the gauge symmetry of the non-equilibrium systems. We will explore the relationship of the non-equilibrium Fokker Planck equation with Abelian Gauge Theory and internal space geometry as in quantum electrodynamics [149]. The Fokker Planck equation can be rewritten as follows [15]: t P(x, t) = i D ij (x) j P(x, t), with the covariant derivative with respect to observable variable i = i + A i = i 1 2 D 1 ij F j and the covariant derivative with respect to time t = t + A t = t + [D ij A i A j i (D ij A j )]. Here,

40 38 J. Wang the independent coordinate components A i of the Abelian gauge field (A t, A i ) will introduce a curvature of internal charge space written as 1 2 R ij = i A j j A i = [ i, j ], (53) where [ ] is a commutator of two operators. According to Equation (41), in the equilibrium case with detailed balance: j SS = 0. Then, A i = i ln(p SS ) is a pure gradient and the curvature is zero: R ij = 0 which corresponds to a flat space. In the non-equilibrium case, A cannot be written as a pure gradient. Therefore, the curvature is not zero, R ij 0. This leads to a curved internal space. Notice that the R ij is a gauge invariant tensor: for a gauge transformation A i A i + i φ, R ij R ij = R ij, the curvature does not change. Moreover, the presence of the probability velocity v(x, t) and the curl flux j(x, t) are also closely related to the internal curvature as [15]: i (D 1 jk v k) j (D 1 ik v k) = R ij. (54) In the case of a constant diffusion coefficient D ij = Dδ ij : i v j j v i = R ij. We notice that Equation (54) is gauge invariant. This implies that if we change A i A i + φ, P(x, t), v(x, t) and j(x, t) are all changed. However, Equation (54) is always satisfied with the same curvature R ij. Moreover, while v(x, t) and j(x, t) depend on the solutions of P(x, t), they always satisfy Equation (54). Therefore, R ij represents a measurement of the geometry of the internal space of the non-equilibrium dynamics. This curvature of the internal space is associated with the heat dissipation in the environment or medium along a closed loop. Along any specific path x(t), T s m is the heat dissipation in the environment or medium [15]: T s m (x (t ), x(t)) = T = t t t t ṡ m dt D 1 ij (x(t)) F j (x(t))ẋ i dt = t t A i (x(t)) dx i (t). (55) Applying the current definition in Equation (40) and Stokes s theorem, the entropy increase in the environment or medium s m along a close loop C becomes [15] T s C m = A i (x) dx i = D 1 ij (x)v SS j (x) dx i C C = 1 dσ ij R ij, (56) 2 where is the surface formed by the closed loop C,dσ ij is the an area element on this surface. R ij is the curvature of the internal space due to the presence of the gauge field A. Both the curvature R ij and the closed-loop heat dissipation in the medium or environment T s C m are gauge invariant under gauge transformation A i A i + i φ. Therefore, we can associate the non-equilibrium nature of the dynamics with a curved internal space. Notice that the non-equilibriumness is thermodynamic in nature and measured by statistical number counting. The internal space is geometric and topological in nature and is measured by the curvature. Number counting and geometry/topology are the only two most reliable measure of the objective world. Here, we see an intimate connection between the statistical number counting and geometry/topology. The presence of the non-zero flux destroys the detailed balance. This leads to non-zero internal curvature and a global topological non-trivial phase in analogy to quantum mechanical Berry phase [13,14]. The difference lies in the fact that in quantum mechanics, the Berry phase is imaginary, while in classical non-equilibrium dynamics such a phase is real.

41 Advances in Physics 39 The medium or environmental heat dissipation s m in Equation (55) plays a key role in the time irreversibility for non-equilibrium systems [147,148, ]. We shall see it also gives an important contribution to the generalized FDT for non-equilibrium dynamics. This contribution is closely related to the gauge field and internal curvature. The gauge aspect of discrete case was also recently discussed and similar conclusion was obtained [153]. When the system is in a non-equilibrium state, there is no detailed balance and the flux is non-zero [15]: j 0. We are often more interested in the direct observable variable x i and the FDT in the form of the equilibrium case shown explicitly in Equation (48). Accordingly, we can transform the original force-observable correlation to the observable-observable correlation x k (t)x i (t ). Without detailed balance, the system becomes time irreversible: (t)f j (x(t )) F j (x(t)) (t ). According to the Fluctuation theorem [147,148,151,152], we get ln PSS (x ) P(x, t x, t ) P SS (x) P(x, t x, t ) = s m + ln PSS (x ) P SS (x). (57) Here, P(x, t x, t ) and ( P(x, t x, t ) represent the probabilities of a forward and backward paths, respectively. We can write (t)f i (x(t )) F i (x(t)) (t ) = dxdx (x)f i (x )A(x, x, t t ) with A(x, x, t t ) given as A(x, x, t t ) = P SS (x )P(x, t x, t ) P SS (x)p(x, t x, t ) = P SS (x ) D[x] P(x, t x, t ) ( 1 PSS (x) P SS (x ) e s m D[x] is the path integral from x (t ) to x(t). Then, we arrive at [15] [ ] d R i (t t ) = D 1 ik dt x k(t) (t ) D 1 ik dx dx (x)f k (x )A(x, x, t t ) ), (58) D 1 ik (t)vss k (t ). (59) When we set the operator as (x) = x j, the response function becomes [ ] R j d i (t t ) = D 1 ik dt x k(t)x j (t ) D 1 ik dx dx x j F k (x )A(x, x, t t ) D 1 ik x j(t)v SS k (t ). (60) The first part on the right-hand side of the equation is similar to that for the equilibrium case in Equation (48). The last two parts of Equation (60) are zero for the detailed balance case. These two parts are related to the non-zero curvature due to the presence of the gauge field originated from the curl flux in internal space, as shown in Equations (54) and (56). In Equation (58), the factor U(x, y) = e (T/2) s m = e P A i(x) dx i is analogous to the Wilson loop or Wilson line in Abelian gauge theory, with P representing the integral for a path from x to y [149]. This gives a quantitative description of the irreversibility from the heat dissipation in the medium or environment. The function inside the path integral of Equation (58) is [U(x, y)] 2/T (P SS (y)/p SS (x)) = e qhk/t, where q hk is the house-keeping heat along a trajectory. It was proved that e qhk/t =1

42 40 J. Wang [147,148]. Along a closed loop, e qc hk /T = [U(x, x)] (2/T). Under the gauge transformation, U(x, y) transforms as U(x, y) e φ(x) U(x, y)e φ(y). Therefore, U(x, y) acquires a phase factor. It satisfies a differential equation [15]: ẋ i i U(x, y) = 0. (61) This implies that the gradient of the phase factor (Wilson lines) originates from the heat dissipation or house keeping of the non-equilibrium systems that is perpendicular to the dynamics. This is just the same case as the circular motion where the radial motion and phase motion are perpendicular to each other. The origin of such behavior is from the non-zero curvature of internal space of the gauge field caused by the presence of the non-zero flux which breaks the detailed balance for non-equilibrium systems. 3. Multiple landscapes, curl flux, and non-adiabaticity Physical and biological systems often involve multiple degrees of freedom with widely different timescales. Take an (electronic) Hamiltonian system with known energy function, for example, the interplay with different timescales of intra-landscape dynamics (motion of the electrons) along the same and inter-landscape hopping between different (electronic) energy surfaces (for nuclear motion) is critical for electron transfer. If the dynamics on the intra-landscape is faster (slower) than the inter-landscape hopping, the process is often called non-adiabatic (adiabatic). For general non-equilibrium complex dynamics where the Hamiltonian or energy function is unknown a priori, the challenge is how to study the multiple timescale problem (adiabatic and non-adiabatic process) of the non-equilibrium system dynamics. We extend the landscape and flux theory for describing global non-equilibrium and non-adiabatic complex system dynamics with eddy current to coupled landscapes [74]. We find that through rigorous mathematical transformation, the coupled landscapes which are often technically difficult to deal with, can be studied in a continuous representation, in which they become a single landscape but with additional dimensions. Intra- and inter-landscape dynamics on the coupled landscapes becomes the dynamics along the multidimensional surface of a single landscape. On this single landscape, the dynamics of the complex system can be decomposed to two determining factors: a gradient of the potential landscape which is closely related to the steady-state probability distribution of the enlarged dimensions, and a probability flux which has a curl or eddy-current nature Introduction The world can be seen as composed complex systems. The complex system dynamics are under intensive study, but due to their own natures, many properties are not well understood, especially the global quantification of the system. Complex systems can be physical or biological. For example, in physical world, the convection in the atmosphere is important for the weather pattern, while in the biological world, molecular motors through the ATP/ADP pumping realize the function for muscle contraction. Complex systems are usually not isolated. They are in constant exchange with energy, materials, and information with their environments. So complex systems are usually activated with pumps and are not similar to the conservative systems often encountered in the bulk of the physics and chemistry literatures. In a conservative system, the energy function is often known and given a priori, the ultimate distribution of the system often follows a Boltzmann law and the dynamics is determined by the gradient of the energy function. Typical complex systems are usually not in equilibrium. There is no energy function given a priori. The global nature of the dynamics is a challenge to address. We previously have established

43 Advances in Physics 41 that the non-equilibrium dynamics can be characterized within a landscape and flux framework [13 15,17,62]. Furthermore, there is another intriguing complexity related to the multiple timescales even when the information on the underlying landscape is known (either for equilibrium or nonequilibrium systems). For example, even for a Hamiltonian system with known energy function, the interplay with different timescales of intra-landscape motion along a single surface and inter-landscape hopping between different surfaces represent non-equilibrium systems which can absorb the energy from energy pump (provided, e.g., from ATP hydrolysis often occurred in biology) and move along or jump between different chemical states for realizing the muscle contraction. If the intra-landscape motion is faster (slower) than the inter-landscape hopping, the process is often called non-adiabatic (adiabatic). For a Hamiltonian system with given energy function such as multi-electronic system in atomic physics, electron transfer, etc. much has been explored and progress has been made [154,155]. While the landscape and flux theory [13 15,17,62,69] is useful for addressing the global nature of complex systems, as it stands, it can only be applied to a single potential surface and is not directly applicable to multiple coupled landscapes. On the other hand, most existing treatments of the adiabatic and non-adiabatic dynamics of multiple energy surfaces taking into account multiple timescales apply only for Hamiltonian systems where the energy function is a prior known for each individual surfaces. It does not apply to the case where the underlying process is non-equilibrium in nature. The adiabatic and non-adiabatic non-equilibrium systems have been studied computationally in the context of electronic transitions, electron transfer, networks, motors, and nonlinear dynamical systems [45,63,66,77,154,155]. However, a global description and framework of understanding is still lacking and in demand. Furthermore, although some systems have been studied computationally, general complex systems require more intensive computation and theoretical guidance is needed to develop efficient algorithm to study their global dynamics. Finally, the ultimate goal is to uncover the organizing principles underlying complex systems so as to apply them to design and engineering. Therefore, for general non-equilibrium complex dynamics where Hamiltonian or energy function is unknown a priori, the challenge is how to study the multiple timescale problem (adiabatic and non-adiabatic process). We now discuss a theoretical framework for describing global non-equilibrium and nonadiabatic complex system dynamics that have eddy currents and require coupled landscapes [74]. We find that through a rigorous mathematical transformation, the coupled landscapes which are often technically difficult to deal with, can be studied in a continuous representation, in which they become equivalent to a single landscape but one with additional degrees of freedom of higher dimension. Intra- and inter-landscape dynamics on the coupled landscapes become dynamics along the multidimensional surface of a single landscape. On this single landscape, the dynamics of the complex system can be decomposed into two determining factors: a gradient of the potential landscape which is closely related to the steady-state probability distribution of the enlarged dimensions, and a probability flux which has a curl or eddy-current nature. We summarize the approach in Figure 11. Figure 11(a) shows the adiabatic single landscape dynamics which is determined by the potential and gradient. Figure 11(b) shows the adiabatic dynamics of the single landscape for non-equilibrium system which is determined by both a landscape gradient and a curl flux or eddy current. Figure 11(c) shows the non-adiabatic multiple landscape surfaces with the energy function of each individual surface being known a priori and the dynamics being determined by the combination of the gradient of the energy on the surface and hopping in between the surfaces. This is the traditionally focused approach to non-adiabatic dynamics. Figure 11(d) shows non-adiabatic multiple landscapes where the energy function is a priori not known. The

44 42 J. Wang (a) (b) (e) (c) (d) Figure 11. Illustrations of the equilibrium/non-equilibrium and adiabatic/non-adiabatic landscapes. (a) The adiabatic single landscape which underlies the equilibrium gradient dynamics. (b) The adiabatic single landscape which underlies the non-equilibrium dynamics determined by both the landscape gradient and curl flux. (c) The non-adiabatic multiple landscapes where the dynamics is determined by the combination of the gradient of the individual landscape and hopping between the landscapes. (d) The non-adiabatic multiple landscapes where dynamics is determined by both the landscape gradient and curl flux as well as the additional inter-landscape hopping. (e) The description equivalent to D with the single landscape for non-adiabatic non-equilibrium systems in the continuous representation, where the dynamics is determined by the gradient of the landscape and curl flux or eddy current on the expanded space (from Ref. [74]). non-adiabatic dynamics is determined by both the landscape gradient and curl flux on the landscape surface as well as by the additional interface hopping. Finally, Figure 11(e) shows the equivalent non-adiabatic single landscape for non-equilibrium systems in continuous representation (equivalent description of Figure 11(d)) where the dynamics is determined by the gradient of a non-equilibrium potential and a curl flux or eddy current in an expanded space with more dimensions Theory of non-adiabatic non-equilibrium potential and flux landscape for general dynamical systems To establish a potential and flux landscape framework for studying the non-adiabatic nonequilibrium dynamical systems, we can divide the degrees of freedom or important variables into two groups. When the timescale for the dynamics of one group of variables y is significantly faster than for the other group of variables x, then the system is in adiabatic limit where we can eliminate the fast variables y and only consider the effective evolution dynamics of the system along the x variables. We have previously discussed this adiabatic case and have shown that the driving force of the general non-equilibrium dynamics can be decomposed into a gradient of a potential and a curl flux. While the gradient force drives the system as an electric field on electrons generating motion downhill the gradient, the curl flux provides an additional contribution that acts like a magnetic field on electrons generating curly motion. It is the curl flux term that breaks detailed balance and gives the possibility of a limit cycle, while the limit cycle does not exist when the detailed balance is preserved in an equilibrium system.

45 Advances in Physics 43 If one group of variables y is comparable or slower in their motion than the other group of variables x, then the system is said to be in a non-adiabatic regime where we cannot eliminate the y variables and only consider the dynamics along the x variables. In this case, we must take both groups of variables of x and y into the consideration. If the variables y are continuous, we can again extend our previously studies and decompose the driving force of the system dynamics into a gradient and curl flux in x and y spaces. However, the y variables often can be discrete variables, in many examples that appear in physics, chemistry, and biology. Electron motions can be thought of as motion along continuous coordinates x while different electronic energy surfaces where electrons move on can be represented by variable y. Depending on the electronic states, the y is usually discrete (integers). Electron transfer and motors are other examples, where the motion is best quantified by two sets of variables, x describing the continuous motion along energy surfaces for nuclear coordinates, while y for describing which energy surface the motion is on. This set of combined continuous x and discrete y variables for complex systems is a challenge to study even with serious numerical computations because we are dealing with the motion on the multiple (labeled by y) coupled (through y) energy surfaces or landscapes instead of a single energy surface or landscape. Thus, it is harder to visualize multiple landscape motion than the pure continuous representation. One immediate question is whether even when the motion in x along each energy surface follows a gradient dynamics, does the whole system of multiple coupled energy surfaces or landscapes with x and y follow the gradient dynamics and preserve the detailed balance? The answer to this question is No, in general, otherwise the cycle motion such as muscle contraction caused by motors would not occur since there is an active pumping process with the consumption of ATP that breaks detailed balance. However, it is not so easy to quantify and visualize how a continuous curl flux emerges for detailed balance breaking with coupled landscapes using the combined continuous x and discrete variable y representation. Furthermore, for general dynamical systems, at each discrete y, the motion on x may not be driven by gradient force alone. Nevertheless, we may decompose the driving force for each discrete y along x to the gradient of a potential and a curl flux. So now we are dealing with again the cases of multiple (labeled by y) coupled (through y) potential landscapes. Therefore, for the general case of combined continuous and discrete variable systems where the timescales of x and y are not apparently separable, the challenge is how to quantify and visualize the global dynamics on such coupled landscapes. We will show that the key to resolve this issue lies in the fact that we can transform the discrete variables y into an equivalent continuous representation with variables z. The physical meaning of the corresponding continuous variables z is the occupation or probability on each discrete variable y. Once the transformation from discrete variables y to z is completed, the whole system can be described again by two sets of continuous variables x and z. Therefore, we can decompose the driving force into a gradient and a curl flux for the general case when the z variables are fast (adiabatic regime), or comparable/slow (non-adiabatic regime) compared to the x variables. In this way, the physical picture is very clear. We can see how the dynamic systems in different timescales can be decomposed into gradient of a single (not multiple coupled) potential landscape with continuous variables x and z and the curl flux defined on this single potential landscape in x and z. In other words, we have transformed a rather difficult problem of multiple (labeled by y) coupled landscape dynamics problem in x and y (finite size in y because of its discrete nature) into a single landscape problem in continuous variables x and z with expanded dimensions (infinite number of values in z, due to its continuous nature). Our aim here is to obtain a physically intuitive picture and quantification of the dynamics for non-adiabatic non-equilibrium dynamical systems. We will, in the following, develop a theoretical framework to explore the global dynamics of non-adiabatic non-equilibrium systems [74]. The deviation from the strong-adiabatic limit is represented by fluctuations in the discrete

46 44 J. Wang state space. We first explore a simple system of single self-regulating gene to illustrate the idea. The discrete state of our example of self-regulating gene is represented by the on or off binding state of the gene s DNA. The (nearly) continuous variable here is the concentration of the proteins produced by the gene A one variable coupled landscape We will use an example to show how we can transform the challenge of exploring the coupled multiple landscapes to an equivalent single landscape with extended dimensions. To begin with, we consider a 1D process of simple synthesis and degradation (decay) coupled with reactions. The reactions of synthesis and decay can occur either in a gene-on landscape or on a gene-off landscape. This is a typical non-adiabatic case mimicking many of the important physical and biological processes, for example, electron transfer, gene regulation, motor dynamics, etc. Set state 1 as the gene-on state and 0 as the gene-off state. Assuming the reaction requires a dimer, we can write down the corresponding Master equations as follows: dp 1 (n) dt dp 0 (n) dt = h 2 [(n + 2)(n + 1)]P 0(n + 2) fp 1 (n) + k[(n + 1)P 1 (n + 1) np 1 (n)] + g 1 [P 1 (n 1) P 1 (n)], (62) = h 2 [n(n 1)]P 0(n) + fp 1 (n 2) + k[(n + 1)P 0 (n + 1) np 0 (n)] + g 0 [P 0 (n 1) P 0 (n)]. (63) Here n is a discrete particle number. The physical meaning is clear. The population probability change of the reaction on (off) state is controlled positively (negatively) by the on reaction, negatively (positively) by the off reaction, its own synthesis and decay. In the continuous space, we can set x = n/v, where V is the volume of the system. In the large volume (V ) limit, where ( ) d H P1 1 f = dt P 0 f 1 2 hx2 H hx2 ( P1 P 0 ), (64) H 1 = 1 V F V D 1, (65) H 0 = 1 V F V D 0 (66) with F 1 = g 1 kx, F 0 = g 0 kx and D 1 = g 1 + kx, D 0 = g 0 + kx. We can write the above equation in an operator form from the probability. and dp dt [ P(t) = exp = HP (67) ] dth P, (68) where P is the probability and H is the operator.

47 With the coherent-state representation of the spin, Advances in Physics 45 I s = 1 π 2π sin θ dθ dφ ŝ s (69) 2π 0 0 and I p = dp p p, (70) where e iφ/2 cos 2 θ ŝ = 2 e iφ/2 sin 2 θ, s =(e iφ/2,e iφ/2 ) (71) 2 and p is the eigenstate of momentum operator ˆp = i( / x): ˆp p =p p. Inserting into P(x f, s f, t f x i, s i, t i ), we obtain ( ) i=n 1 P(x f, s f, t f x i, s i, t i ) = x f, s f ˆT exp Hdt x i, s i = x f, s f (1 + H i t) x i, s i i=1 i=n 1 = x f, s f [I s I p (1 + H i t)]i s I p x i, s i = const i=1 D[ cos θ]dφdpdx exp ( One can calculate matrix for production of s p (1 + H t)] ŝ q by using ) dtl. (72) p H 1 x = i 1 V pf 1 1 V 2 p2 D 1 p x, (73) p H 0 x = i 1 V pf 0 1 V 2 p2 D 0 p x (74) and p x =e ipx. Thus, we find the effective Lagrangian L = i dφ 2 dt 1 + cos θ cos θ f e iφ ( + + (L 1 + f ) 1 + cos θ 2 1 cos θ hx2 e iφ 2 L 0 + h ) 1 cos θ 2 x2, (75) 2 L 1 = i 1 V F 1p V 2 p2 D 1 i dx p, dt (76) L 0 = i 1 V F 0p V 2 p2 D 0 i dx p. dt (77) Defining (1 + cos θ)/2 = c 1, (1 cos θ)/2 = c 0, expanding Equation (75) with respect to the conjugate variables φ and p to the lowest order ( L cl = iφ dc 1 fc 1 + h ) ( 1 dt 2 x2 c 0 + ip V c 1g V c 0g 0 1 ) dx kx. (78) V dt

48 46 J. Wang Integrating φ and p leads to δ functions which gives the deterministic equations: dc 1 = fc 1 + h dt 2 x2 c 0, (79) dx dt = g 1 V c 1 + g 0 V c 0 k x. V (80) Taylor expands Equation (75) with respect to the conjugate variables φ and p to the second order yielding L = L cl + 1 ( 2 φ2 fc 1 + h ) 2 x2 c p2 V (g 1c g 0 c 0 + kx). (81) This leads to the coupled-langevin equations as with dc 1 = fc 1 + h dt 2 x2 c 0 + η θ, (82) dx dt = g 1 V c 1 + g 0 V c 0 k V x + η x (83) η θ (t)η θ (t ) = 1 (fc 1 + h2 ) 2 c 0 δ(t t ), (84) η x (t)η x (t ) = 1 ( g1 2V V c 1 + g 0 V c 0 + k ) V x δ(t t ). (85) Setting k/v = 1, c 1 = (1 ξ)/2, g 1 g 0, g 0 g 1, h = 2h 0 V 2, Equations (82) and (83) become 1 dξ ω dt = (1 ξ) κx2 (1 + ξ) 2 ω η θ, (86) dx dt = g 1 + g 0 2V + g 1 g 0 2V ξ x + η x = X ad V + δx V ξ x + η x. (87) We can define ω as a parameter which quantifies the degree of adiabaticity as the ratio ω = f /k, where the f is the off rate of inter-landscape hopping and k is the decay rate on a landscape representing the intro-landscape dynamics. ω being the ratio of the timescales of inter-landscape hopping and intra-landscape motion gives a quantitative measure of the relative importance of the non-adibaticity. The larger ω values indicates a stronger coupling among landscapes (x, i) (i = 0, 1), resulting in an effective landscape (x). On the other hand, the smaller ω values imply a weak coupling among landscapes, resulting in multiple landscapes (x, i), (i = 0, 1). The nonadiabatic multiple landscapes can be further transformed into the picture of single landscape with extra dimensions (x, ξ). 2 ω η θ(t) 2 ω η θ(t ) = η x (t)η x (t ) = 1 2V (fc 1 + h2 x2 c 0 ) δ(t t ) = 1 ω [(1 ξ)+ κx2 (1 + ξ)]δ(t t ), (88) ω 2 ( Xad V + δx ) V ξ + x δ(t t ). (89)

49 Advances in Physics 47 The corresponding Fokker Planck equation for p(x, c 1, t) then is as follows: [( g1 t p = x V c 1 + g 0 V c 0 k ) ] V x p + 1 [ x 2V [( c1 fc 1 + h ) ] 2 x2 c 0 p + 1 [ c 1 2 ( g1 V c 1 + g 0 (fc 1 + h2 ) x2 c 0 p V c 0 + k ) V x ] ] p. (90) Therefore, we can see that we have successfully transformed a non-adiabatic dynamic challenge with a multiple coupled landscape on (x, i) space (i = 0, 1) into dynamics on a landscape with extended dimensions (x, c) or (x, ξ). There is an important implication of this study. In the original problem, the dynamics is 1D and therefore there is always a landscape whose derivative determines the intra-landscape motion with occasional jumps to other landscapes. In this picture, if there is no coupling or jump to other landscapes, we will not have a flux and the system dynamics is driven by a potential gradient. The question is how would coupling of the discrete state change this picture. As seen, we have transformed this to the dynamics with an extra dimension. In two dimensions, the system is no longer guaranteed to have a pure gradient. In fact, the original coupled landscape dynamics with intra-landscape driving force driven only by a potential gradient now becomes a system with driving forces from both the potential gradient and flux components of a single landscape in expanded space. Introducing the extra dimensions provides the extra timescales. So the flux here comes from the landscape coupling and the timescale due to the coupling. This is the non-adiabatic origin of the flux. In a similar spirit, the above schemes can be naturally generalized to N variable case with 2 N coupled landscapes. In general, we can transform the general issues of non-adiabatic dynamics involved with the coupled landscapes to the non-equilibrium dynamics on a single composite landscape in expanded continuous dimensions. Then, we can explore the global behavior and dynamics of the system by directly applying the non-equilibrium landscape and flux theory directly [13 15,17,62, 69]. 4. Spatial fields, landscapes, and fluxes We live in a non-equilibrium world. Our Earth is a non-equilibrium system constantly receiving energy from the Sun. Our human body as a non-equilibrium system constantly consumes energy for survival. Uncovering the principles and physical mechanisms of these activated processes is vital for understanding the non-equilibrium systems. Significant efforts and progresses have been made recently on the global stability and dynamics of the non-equilibrium systems [3 6,8,9,11,13 15,69]. The current study have often been focused on the homogeneous systems where the spatial dependence is ignored or at most characterized by an averaged mean field. However, almost all of the realistic systems are spatial dependent. The famous Bernard convection flow is a typical physical example of spatially dependent non-equilibrium system [156]. The drosophila differentiation and growth is another famous biological example of spatially dependent non-equilibrium system [157,158]. These systems have been studied through the dynamics of reaction diffusion processes at the mean field level, often characterized by the partial differential equations. The mean field level description can give local dynamics and local stability analysis for these spatial-dependent non-equilibrium systems. However, the global natures of the systems such as global stability and robustness cannot be addressed using the typical mean level local description. In this section, we will generalize the non-equilibrium landscape and flux theory [13 15,17,62,69] to include quantities that vary in physical space. In other words, we will go from

50 48 J. Wang non-equilibrium statistical mechanics to non-equilibrium statistical field theory [73,75,85]. We can develop a general method to construct a Lyapunov functional to quantify the global stability and robustness of such spatially dependent non-equilibrium systems. We find the Lyapunov functional reflects an underlying intrinsic potential field landscape for the spatial non-equilibrium systems. The topography of the intrinsic potential field landscape can be characterized by the basins of attractions directly related to the global stability. In the spatially dependent equilibrium systems, the dynamics is determined by the functional gradient of an energy functional. Usually, the energy functional is known a priori as the interaction potential functional for the systems. For the spatially dependent dynamical systems, in general, an energy functional giving equilibrium gradient dynamics cannot usually be found. One has to consider the additional contributions from a curl flux. The curl flux field quantitatively characterizes the degree of the non-equilibriumness. Again the potential field landscape and curl flux field form a dual pair for characterizing the global spatial temporal non-equilibrium dynamics. In the following subsections, we will develop a potential and flux field landscape theory for spatially dependent non-equilibrium systems [73,75,85]. We will also uncover the Lyapunov functional for quantifying the global stability of the spatially dependent non-equilibrium systems [73,75,85] Potential and flux field landscape theory for stochastic spatial non-equilibrium systems To extend the potential and flux landscape theory for the non-spatially dependent (or spatially homogenous) dynamical systems to the general spatially dependent dynamical systems, we first will start with the stochastic dynamics with spatial dependence. To distinguish the problem from that with the spatially independent case, we use a different symbol φ( x, t) to represent the spatial dependence of the dynamical variables rather than C in the spatial-independent case [73,75,85]. φ( x, t) = F[ x; φ] + ξ[ x, t; φ], t φ a ( x, t) = F a [ x; φ] + ξ a [ x, t; φ], (91) t where ξ a [ x, t; φ] = b d 3 x G ab [ x, x ; φ]ζ b ( x, t) with <ζ a ( x, t) >= 0 and <ζ a ( x, t)ζ b ( x, t )>= δ ab δ (3) ( x x )δ(t t ). Notice that φ( x, t) instead of being a single dynamical variable as in the spatially independent case, now is a function of space and time and therefore becomes a vector field. F[ x; φ] becomes the deterministic driving force field which can depend on space and dynamic variables explicitly. ξ[ x, t; φ] becomes the stochastic driving force field depending on the space and field variables. Therefore, the evolution of the spatially dependent dynamical systems is determined by the deterministic and stochastic driving force fields. G ab [ x, x ; φ] gives the spatial and dynamical variable dependence of the stochastic driving force field. The deterministic part of the driving force field F[ x; φ] often contains a term which is associated with spatial diffusion of the dynamical variables Diff( x), Diff( x) φ( x, t). here refers to the spatial derivative. Therefore, the stochastic ordinary differential equation for the evolutions of the dynamical variables in the spatial-independent case becomes partial differential equation for the evolutions of the dynamical field variables in the spatial-dependent case [73,75,85]. Once the stochastic partial differential equation for the field is written up as aforementioned, we can investigate the evolution of individual trajectories for the dynamical variables. However, since the individual trajectories are stochastic, we cannot follow the trajectories to predict the outcome. The more appropriate quantity to trace is the evolution of the probability distribution rather than the individual trajectories.

51 Advances in Physics 49 The next question then is what would be the corresponding Fokker Planck probability evolution equation for the spatial dependent dynamical system. In order to do so, we must extend the derivatives in the original Fokker Planck equation in the spatial-independent case to the spatialdependent case. Since the dynamical variables themselves now are functions of space, so the derivatives with respect to them become functional derivatives. We need to define the functional derivative explicitly. A simple ansatz would be to divide the space into cubic cells of each with size of l and volume of l 3 [2]. We can then define the z i = l 3 φ( x i ) for each spatial cell with the label i. Let us consider function of the variables z ={ z i } of all the different spatial cells. The driving force F[ x; φ] becomes functions of all these cell variables z. The partial derivatives can be defined straightforwardly. The functional derivative can be defined as δ F[ φ] δ φ( x i ) = lim l 0 l 3 F[ z] z i. (92) Therefore, the corresponding stochastic partial differential equation becomes [2] z i t φ( x, t) t = i = F[ x; φ] + ξ[ x, t; φ] (93) Diff ij z j + δf[ z i ] + j δg ij ζ j with the spatial diffusion term explicitly put in. In this equation, the Diff ij are the spatial diffusion coefficients giving the discrete approximation of Diff. F[ φ( x i, t)] = lim δf[ z i ]l 3 l 0 and G( x i, t) = lim l 3 δg ij ζ j. l 0 j Assuming an explicit general variable-dependent correlation [73,75, 85]: and G( x, t)g( x, t ) =D( x, x )δ(t t ) D( x i, x j ) = lim l 0 l 6 k δg ik δg jk. Finally the Fokker Planck equation for z i becomes [2,73,75,85] P( z) t = ij {[Diff ij z j + δ ij δf( z i )]P( z)}+ 1 2 z j δg ik δg jk P( z). z i 2 z ijk i If we take the l 0 limit, then we finally obtain the functional Fokker Planck diffusion equation for the field variable φ( x). P[ φ] = d 3 δ x t δφ a a ( x) ( Diff( x)φ a( x) + F a [ x; φ]p[ φ]) + d 3 x d 3 x δ 2 δφ ab a ( x)δφ b ( x ) (D ab[ x, x ; φ]p[ φ])) (94)

52 50 J. Wang with spatial diffusion term explicitly put in. For convenience, we list a table for linking the variables in Fokker Planck diffusion equation for spatial-independent case and the field variables in functional Fokker Planck equation in spatial-dependent case. Notation correspondence J i i a, x, j b, x, k c, x, C i (t) φ a ( x, t), F i ( C) F a [ x, φ], G ij ( C) G ab [ x, x ; φ], J i ( C) J a [ x; φ], d 3 x δ δ φ, i d 3 δ x C i δφ a ( x), J i ( C) d 3 δ x C i δφ a a ( x) J a[ x; φ]. (95) Quantification of the potential field landscape and decomposition of the dynamics to a potential field landscape and a curl flux field for stochastic spatially dependent dynamical systems (1) Functional Fokker Planck Equation Here we can see the Fokker Planck equation for the evolution of the probability distribution of the spatial-independent dynamical systems and the corresponding functional Fokker Planck equation for the evolution of the probability functional of the spatial-dependent dynamical systems [73,75,85]. P = ( FP) + (DP), t that is, P = i (F i P) + i j (D ij P) t P[ φ] t = d 3 δ x δφ a a ( x) (F a[ x; φ]p[ φ]) + d 3 x d 3 x δ 2 δφ ab a ( x)δφ b ( x ) (D ab[ x, x ; φ]p[ φ])). (96) The physical meaning is clear. The change of the probability functional is determined by both the driving force field and the fluctuations characterized by the diffusion in field configurations. Here for clarity purposes, we have not explicitly written out the spatial diffusion term (Diff term aforementioned ) contained in the deterministic force field.

53 Advances in Physics 51 (2) Probability flux field Here we define the corresponding flux field J a [ x; φ] considering the spatially dependent case as compared with the flux J for the spatial-independent case. J = FP (DP), that is, J i = F i P j (D ij P) J a [ x; φ] = F a [ x; φ]p[ φ] d 3 x δ δφ b b ( x ) (D ab[ x, x ; φ]p[ φ])). The physical meaning of the flux field is that the net flux determines the evolution of the probability functional since P[ φ] t = a d 3 δ x δφ a ( x) J a[ x; φ]. The probability functional changes in time are determined by the functional divergence of the net flux field. (3) Force field decomposition Here we consider the driving force field decomposition into the functional gradient of the potential field landscape and the curl flux field how including spatial dependence as compared to the spatial-independent case where the driving force for the dynamics is decomposed to gradient of the potential and curl flux [73,75,85]. that is, F = D U + J ss P ss, F i = D ij j U + J i ss, P ss where F = F D and U = ln P ss F a [ x; φ] = b d 3 x D ab [ x, x ; φ] δu[ φ] δφ b ( x ) + J a ss[ x; φ] P ss [ φ], (97) where F a [ x; φ] = F a [ x; φ] b d 3 x (δ/δφ b ( x ))D ab [ x, x ; φ] and U[ φ] = ln P ss [ φ] and the divergent-free probability flux field J[ x; φ] satisfies d 3 δ x δφ a ( x) J a[ x; φ] = 0. a We can see clearly, U[ φ] is quantified and closely linked to the steady-state probability functional of the whole spatial-dependent dynamical systems. Since the U[ φ] directly reflects the probability or weight of each field configuration in space, U serves as the potential field landscape which can be used to characterize the global nature of the system. On the other hand, for steady state, the divergence of the flux field is equal to zero. a d 3 x(δ/δφ a ( x))j a [ x; φ] = 0.

54 52 J. Wang If J a [ x; φ] = 0, then we have the detailed balance equilibrium condition. Under this condition, the potential landscape field is related to the equilibrium probability functional U[ φ] = ln P eq [ φ], while the dynamics is determined by the functional gradient of potential F a [ x; φ] = b d 3 x D ab [ x, x ; φ](δu[ φ]/δφ b ( x )) with an addition to the diffusion-dependent force field b d 3 x (δ/δφ b ( x ))D ab [ x, x ; φ]. When the flux field itself is not equal to zero, the divergence free property of the flux at steady state a d 3 x(δ/δφ a ( x))j a [ x; φ] = 0 implies that the flux field J[ x; φ] has no sinks or sources to go into or come out of in the field configurational space φ [73,75,85]. Therefore, J[ x; φ] has to be a curl rotating around in the field configurational space. The dynamics is determined by the functional gradient of the potential field landscape, the curl flux field, and the additional diffusion-dependent force field. In this way, we realize the force field decomposition in the field configurational space for determining the non-equilibrium dynamics. While gradient dynamics attracts the system down to the underlying potential field landscape, the flux will tend to drive the spatial-dependent dynamical system curling around in the field configurational space φ. The gradient of the potential field landscape and the curl flux field are analogous to dynamics of a charged scalar field moving in both spatial-dependent electric and magnetic fields in quantum field theory The Lyapunov functional for spatially dependent dynamical systems Global stability is essential for the function and dynamics of spatially dependent systems. Here we will provide a general method to construct a Lyapunov functional monotonically decreasing for the spatially dependent dynamical systems [73,75,85]. In this way, we can quantify the global stability with this Lyapunov functional. It turns out that the Lyapunov functional is the intrinsic potential field landscape of the dynamical systems under zero fluctuations in field configurations. Furthermore, we also define the free energy functional of the spatially dependent dynamical system and find that it is a Lyapunov function for finite fluctuations in field configurations. So the free energy functional can be used to explore the global stability of spatial-dependent dynamical systems under finite fluctuations. Or equivalently, the Lyapunov functional of spatially dependent deterministic dynamical systems The corresponding deterministic field equation of the corresponding Fokker Planck probability equation (FFPE) is as follows: φ a (q, t) t = F {a,q} [φ], where F {a,q} [φ] stands for the force field with the field variable φ with spatial variable q and parameter a φ = φ(a, q). The zeroth-order potential field term as intrinsic potential field landscape (0) [φ] plays the role of a Lyapunov functional of the spatially dependent deterministic dynamical system [73,75,85]. When D is small but not yet in the zero limit, we need to keep only the zero order (0) [φ] in the expansion of and thus the steady-state probability distribution is approximated by P ss [φ] = 1 [ ] Z ss [φ] exp (0) [φ]. D For P ss [φ] to be a proper functional when D is small but not in the zero limit, P ss [φ] should have a higher bound and thus (0) [φ] should have a lower bound. Thus, we can always add a constant

55 to (0) [φ] to make it non-negative, that is, Advances in Physics 53 (0) [φ] 0. Then we calculate its time derivative [73,75,85]: d (0) [φ] dt = φa (q, t) δ {a,q} (0) [φ] t = F {a,q} [φ]δ {a,q} (0) [φ] = (δ {a,q} (0) [φ])d {a,q}{b,q } [φ](δ {b,q } (0) [φ]) 0. (98) where δ {a,q} = δ {a,q} /δφ(a, q) represents the functional derivative with respect to field φ(a, q). Therefore, the intrinsic potential field landscape (0) [φ] is a Lyapunov functional of the spatially dependent deterministic dynamical systems. We can use the intrinsic potential field landscape to directly quantify the global stability of the spatially dependent deterministic dynamical systems. Furthermore, from the relation (F {a,q} [φ] + D {a,q}{b,q } [φ]δ {b,q } (0) [φ])δ {a,q} (0) [φ] = 0 in zero-fluctuation limit and expression of flux field in zero-fluctuation limit V ss (0){a,q} [φ] = D {a,q}{b,q } [φ]δ {b,q } (0) [φ] + F {a,q} [φ], we can see that the driving force field F {a,q} [φ] in the spatially dependent deterministic dynamical systems can be decomposed of two terms, one is the gradient of the intrinsic potential field landscape D {a,q}{b,q } [φ]δ {b,q } (0) [φ] and the other is the curl flux velocity field V ss (0){a,q} [φ], F {a,q} [φ] = D {a,q}{b,q } [φ]δ {b,q } (0) [φ] + V ss (0){a,q} [φ]. Furthermore, the V ss (0){a,q} [φ]δ {a,q} (0) [φ] = 0[73,75,85]. This implies that the flux velocity field of the driving force field is perpendicular in direction to the gradient of the intrinsic potential field landscape of the driving force field for the deterministic spatially dependent dynamical systems. Lyapunovfunctional of thestochastic spatially dependent dynamical systems In the last section, we discussed about the global stability and Lyapunov functional of the deterministic spatially dependent dynamical systems. We now turn to the explorations of the Lyapunov functional for global stability for finite fluctuations. The free energy functional F of the system as a functional of the probability distribution P[φ, t] is a Lyapunov functional of the FFPE. The proof is as follows [73,75,85]: P[φ, t] F(t) = ln P ss [φ] = P[φ, t] P[φ,t] ( ln ( ( P[φ, t] ln P[φ, t] P ss [φ] ) Dφ ) ) P[φ, t] + P ss [φ] Dφ P[φ, t] = P ss [φ] ( ( P[φ, t] = P ss [φ] ln P ss [φ] = P ss [φ](r[φ, t]lnr[φ, t] R[φ, t] + 1)Dφ ) ) P[φ, t] P[φ, t] P ss [φ] P ss [φ] + 1 Dφ 0, (99)

56 54 J. Wang d dt F[P[φ, t]] = d dt = = 0. ln P[φ, t] P ss [φ] V {a,q} [φ, t] ( [δ {a,q} ln P[φ,t] [δ {a,q} ( ln P[φ, t] P ss [φ] )] P[φ, t] P ss [φ] P[φ,t] )] D {a,q}{b,q } [φ] [ δ {b,q } ( ln )] P[φ, t] P ss [φ] P[φ,t] F[P[φ, t]] = ln(p[φ, t]/p ss [φ]) P[φ,t] plays the same role for the FFPE in the case of finite fluctuations as (0) [φ] for the deterministic equation. When the probability distribution functional in time P[φ, t] differs from probability distribution functional at steady state P ss [φ], δ {a,q} ln(p[φ, t]/p ss [φ]) is non-zero and thus free energy functional F(t) will continue to decrease until P[φ, t] agrees with P ss [φ], where F(t) = 0 and (d/dt)f(t) = 0. From this we finally obtain the Lyapunov functional for the deterministic spatially dependent dynamical systems as the intrinsic potential field landscape and the Lyapunov functional for the spatially dependent dynamical systems with finite fluctuations as free energy functional. We can quantify the global stability and behavior with the Lyapunov functional [73,75,85] Non-equilibrium thermodynamics of stochastic spatial systems We consider spatially inhomogeneous dynamical systems described by a functional Fokker Planck equation. Define the following three thermodynamic quantities internal energy functional, entropy functional, and free energy functional and their total energy, entropy, and free energy counterparts (for details, see [73,75,85]): U(t) = ln P ss (t) U(t) = U(t) = P(t)( ln P ss (t)) d q, S(t) = ln P(t) S(t) = S(t) = P(t)( ln P(t)) d q, F(t) = U(t) S(t) = ln P(t) ( F(t) = F(t) = P(t) ln P(t) ) d q, P ss (t) P ss (t) where d q = M dq (M is the metric tensor), P ss (t) is the steady-state probability functional at each fixed time t and the probability functional in time P(t) is the time-dependent solution of the functional Fokker Planck equation. We demonstrate that these three definitions combined with the following equations derived from the force field or functional decomposition equations will give the non-equilibrium thermodynamic equations. {a,q} U(t) = 2V ss {a,q} (t) 2 F {a,q}, {a,q} S(t) = 2V {a,q} (t) 2 F {a,q}, where V ss {a,q} (t) and V {a,q}(t) are steady-state and time-dependent probability flux field, respectively.

57 Advances in Physics 55 The first law of thermodynamics for spatially inhomogeneous stochastic dynamical systems We derive the rate of change of the internal energy U(t) [73,75,85]: U(t) = d U(t)P(t) d q dt U(t) = P(t) d q + U(t)Ṗ(t) d q t U(t) = U(t) {a,q} J {a,q} (t) d q t U(t) = + J {a,q} (t) {a,q} U(t) d q t U(t) = + P(t)V {a,q} (t) {a,q} U(t) d q t U(t) = + V {a,q} (t) {a,q} U(t). (100) t The first term U(t)/ t is called dissipative work W d (t) which in general is due to the change of time-dependent parameters of the system. By plugging in the equation {a,q} U(t) = 2V{a,q} ss (t) 2 F {a,q}, we can see that the second term becomes [73,75,85] V {a,q} (t) {a,q} U(t) = 2V ss {a,q} (t)v{a,q} (t) 2 F {a,q} V {a,q} (t) = Q hk (t) h p (t) = Q ex (t), (101) where h p (t) = 2 F {a,q} V {a,q} (t) is the heat dissipation rate, which splits into two parts: h p (t) = Q hk (t) + Q ex (t). One part is the house-keeping heat rate Q hk (t) = 2V{a,q} ss V{a,q} (t) originated from non-zero steady-state flux velocity breaking the detailed balance and the other part is the excessive heat rate Q ex (t). The convention of the sign of heat here is that it is positive when it flows from the system to the heat bath. Thus, Q ex (t) is the excessive heat rate transferred from the environment to the system. Thus, we reach the following equation: U(t) = W d (t) Q ex (t) This is a generalization of the first law of thermodynamics for spatially dependent systems. It states that the increase in the internal energy of the system is due to the work done by the environment to the system W d (t) and the EXCESSIVE heat transferred from the environment to the system Q ex (t). The house-keeping heat does not contribute to the change of the internal energy of the system but is used to maintain the system away from equilibrium steadily (thus the internal energy of the system does not change).

58 56 J. Wang The second law of thermodynamics for spatially inhomogeneous stochastic dynamical systems Ṡ(t) = d P(t) ln P(t) d q dt = Ṗ(t) d q + ( ln P(t)) P(t) d q = d ( ) P(t) d q + S(t)( {a,q} J {a,q} (t)) d q dt = 0 S(t) {a,q} J {a,q} (t) d q = J {a,q} (t) {a,q} S(t) d q = P(t)V {a,q} (t) {a,q} S(t) d q = V {a,q} (t) {a,q} S(t). (102) By plugging in the equation {a,q} S(t) = 2V {a,q} (t) 2 F {a,q}, we obtain Ṡ(t) = 2V {a,q} (t)v {a,q} (t) 2 F {a,q} (t)v {a,q} (t) =e p (t) h p (t). The first term on the right-hand side e p (t) = 2V {a,q} (t)v {a,q} (t) the EPR which is always nonnegative and the second term h p (t) = 2 F {a,q} (t)v {a,q} (t) is the heat dissipation rate [73,75,85]. Thus, we get the entropy balance equation: Ṡ(t) = e p (t) h d (t), Ṡ(t) represents the entropy change for the system, h d (t) represents the heat dissipation and therefore the change of the entropy due to the environments. We can write down the above entropy balance equation as e p (t) = Ṡ(t) + h d (t). e p (t) is the combination of system and environmental entropy change and therefore represents the change of the total entropy. Since the definition of the entropy production guarantees it to be positive, the total entropy change of the system and environment together will never decrease. S tot = e p (t) = Ṡ(t) + h d (t), e p (t) 0. This is the generalized second law of thermodynamics for spatially dependent system. Non-equilibrium thermodynamics for spatially dependent stochastic dynamical systems in free energy functional representation We can also derive the rate of F by combining the first law and the second law [73,75,85]: Ḟ(t) = U(t) Ṡ(t) = (W d (t) Q ex (t)) (e p (t) h d (t)) = W d (t) (e p (t) Q hk (t)) = W d (t) f d (t). (103) Where we have split the EPR into two parts: e p (t) = Q hk (t) + f d (t). The first term Q hk (t) is the house-keeping heat yet here its physical meaning is a part of the EPR because this part of

59 Advances in Physics 57 heat is used to maintain the system away from equilibrium which contributes to a part of EPR. The second term f d (t) is a part of the EPR that comes from the contribution of the dissipation of the free energy F. In other words, Ḟ(t) term accounts for the usual free energy relaxation. This non-equilibrium term exists for the pure equilibrium systems away from the equilibrium state. The house-keeping term Q hk (t) is intimately linked with the flux (when flux is zero, housekeeping Q hk (t) is zero). When the detailed balance is preserved, the systems are intrinsically in equilibrium. The free energy relaxation back to the equilibrium state is equal to the dissipation or the entropy production when the W d (t) = 0 (work done from environments to the system is zero). If flux is not zero, the detailed balance is broken. The system is in non-equilibrium state. Then, the total entropy production is partitioned to the house keeping for the non-equilibrium steady state and the free energy relaxation back to the steady state when the W d (t) = 0. Non-zero flux plays an important role here in the non-equilibrium thermodynamics. If W d (t) 0, we have additional contribution to the free energy relaxation from the work done to the system from the environments. Thus, we get the equation [73,75,85]: Ḟ(t) = W d (t) f d (t). This equation or equivalently Ḟ = W d (t) + f d (t) can be understood as follows: the dissipation of free energy does two things: one is that it does work to the environment W d (t) and the other is that it (f d (t)) contributes to a part of the EPR. This is the generalization of first law of thermodynamics to the non-equilibrium regime for spatially inhomogeneous stochastic dynamical systems. We now discuss the change of the free energy in time. We have already proved in an earlier section that free energy is a Lyapunov functional. Thus, we can generalize the second law of non-equilibrium thermodynamics for spatially inhomogeneous systems by the fact that the free energy functional never increases [73,75,85]. When the probability distribution functional in time P[φ, t] differs from probability distribution functional at steady state P ss [φ], δ {a,q} ln(p[φ, t]/p ss [φ]) is non-zero and thus free energy functional F(t) will continue to decrease until P[φ, t] agrees with P ss [φ], where F(t) = 0 and (d/dt)f(t) = 0[73,75,85]. 5. The cell cycle: limit cycle oscillations The cell cycle is one of the most fundamental aspects of the cell. With sufficient nutrition, cells grow and proliferate. New copies of DNA form and segregate from their original parents. Then, the cells divide and new cells are born. This covers the whole life cycle of a cell. Understanding the cell cycle is crucial for understanding the cell function. In current biology textbooks, the cell cycle is thought of as having several major distinct phases, G1 (resting) phase, S phase (synthesis), G2 phase (interphase), and M phase (mitosis) along with their corresponding checkpoints controlling the cell-cycle dynamics. From a molecular perspective, the cell cycle has been shown to be controlled by the underlying gene regulatory networks [17,57,83, ]. The concentration dynamics of genes/proteins can be quantitatively traced and correlated with the cell cycle. However, physical understanding from the molecular network perspective and a global picture of what is going on still proves challenging. Some key questions remain to be answered: What controls the global stability of the underlying gene network as well as its different phases, the associated checkpoints, and the whole cell cycle; How will those controlling factors influence the function. Addressing these issues is not only crucial for understanding the underlying mechanism, but also for the possible medical applications. It is known that cancer cells have faster

60 58 J. Wang cell-cycle period than the normal cells. Understanding the cell-cycle mechanism will be helpful to explore the origin of the faster speed of the cancer cell cycle. Based on this, new strategies can be designed to change the cell-cycle period for cancer preventions and treatments Model for the cell-cycle network Here, we will apply landscape and flux theory to the cell cycle [17,57,83]. Progress has been made in modeling cell-cycle dynamics as determined by the underlying gene regulatory networks including the yeast cell cycle [ ] and mammalian cell cycle [ ]. We explore a simplified underlying gene regulatory network description based on cell-cycle biology [ ,167]. A detailed gene network diagram for the model of the cyclin/cdk network driving the mammalian cell cycle is shown in Figure 12. The model contains all four modules corresponding to four stages of the cell cycle sequentially, which are separately centered on cyclin D/Cdk4-6 (Module 1), cyclin E/Cdk2 (Module 2), cyclin A/Cdk2 (Module 3), and cyclin B/Cdk1 (Module 4). The network also incorporates the prb/e2f pathway, which controls progression or arrest of the cell cycle. At the beginning of the cell cycle, the growth factor (GF) promotes the synthesis of cyclin D, and Cyclin D can form a complex with the kinase subunit Cdk4-6. The active forms of cyclin D/Cdk4-6 and cyclin E/Cdk2 ensures progression in G1 and elicits the G1/S transition, by phosphorylating and inhibiting prb. The inhibition of prb ensures the activation of the transcription factor E2F that allows cell-cycle progression by promoting the synthesis of G1 cyclins. During S and G2 phases, cyclin A/Cdk2 inhibits (by phosphorylation) the Cdh1 protein that promotes the degradation of cyclin B. The negative feedback loops exerted (via Cdc20 activation) Figure 12. The diagram for the mammalian cell-cycle model (see Figure S11 in SI Appendix of Ref. [83]for a more detailed diagram). Arrows represent activation and dotted lines with short bar represent repression. The model includes four major cyclin/cdk complexes centered on cyclin D/Cdk4-6, cyclin E/Cdk2, cyclin A/Cdk2, and cyclin B/Cdk1. The opposite effects of prb and E2F control the cell-cycle progression. The combined effects of four modules determine the cell-cycle dynamics of oscillation. Red colors represent the key genes and regulations found by global sensitivity analysis. Blue colors represent the key genes found by global sensitivity analysis which are consistent with experiments (from Ref. [83]).

61 Advances in Physics 59 by cyclin B/Cdk1 on itself and cyclin A/Cdk2, and the negative feedback loop exerted by cyclin A/Cdk2 on E2F allows the reset of the cell cycle and the start of a new round of oscillations. Inhibitory phosphorylation by the kinase Wee1 and activating dephosphorylation by the Cdc25 phosphatases regulate the activity of Cdk1 and Cdk2. The activity of the cyclin/cdk complexes can also be regulated by reversible association with the protein inhibitor p21/p27. The combined effects of four modules determine the cell-cycle dynamics of oscillation. Based on mass action or Michaelias Menton kinetics, the dynamics of the model is governed by a set of nonlinear ordinary differential equations (44 ordinary differential equations with detailed rate parameter values [83], with the mathematical form as dc/dt = F(C), where C is the concentration vector and F is the underlying chemical driving force) Self-consistent mean field approximation In reality, gene regulatory network dynamics is stochastic. This is due to the fact that both intrinsic fluctuations from the finite number of molecules and extrinsic fluctuations from the environments are unavoidable. Nevertheless, probability evolution usually follows a linear equation and is predictable [17,76,83]. It is technically challenging to solve the diffusion equation for a large regulatory network due to its inherent large dimensionality. We can apply a self-consistent mean field approximation to reduce the dimensionality [8,17,76,83] from exponential to polynomials. In this way, one can follow the time evolution and obtain the steady-state probability in protein concentration space. Based on the result, one can map out the underlying potential landscape, associated with the steady-state probability distribution. We assume that the gene network state follows a probabilistic diffusive evolution equation: P(X 1, X 2,..., X n, t), where X 1,X 2,... is the concentration of genes/proteins or populations of molecules. This leads to an N-dimensional partial differential equation, which is not feasible to solve exactly. This is because every variable can have M values. Then, the dimensionality of the system becomes M N. Using a self-consistent mean field approach [8,17,56,76,83], one can split the probability into the products of individual probabilities: P(X 1, X 2,..., X n, t) n i P(X i, t) and solve the probability self-consistently. This effectively reduces the dimensionality from exponentially large M N to polynomially small M N. This makes the computational task tractable. As mentioned, for the multidimensional system, it is still challenging to solve the diffusion probability directly. We can also simplify the calculations using moment equations. We can simplify the computational task by assuming specific probability distributions based on physical arguments. This leads to specific connections between moments. In principle, once all the moments are known, one should be able to construct probability distribution. For example, the Poisson distribution has only one parameter (its associated mean), so one can calculate all the other moments from the first moment or the mean. In this study, we assume summed gaussian distributions as an approximation [1,7]. A Gaussian distribution requires two moments, mean and variance to specify. When the diffusion coefficient D is small, the moment equations can be approximated as [1,7] ẋ(t) = C[x(t)]. (104) The above first-order ordinary differential equations follow Michaelias Menton kinetics based on mass action. σ (t) = σ (t)a T (t) + A(t)σ (t) + 2D[x(t)]. (105)

62 60 J. Wang These equations quantify the fluctuations of those dynamics. Here,x, σ (t), and A(t) are vectors and tensors, and A T (t) is the transpose of A(t). The matrix elements of A are A ij = C i [X (t)]/ x j (t). According to this equations, we can solve x(t) and σ (t). We consider here only diagonal elements of σ (t) from a mean field approximation. Therefore, the evolution of the distribution for one variable can be obtained using the mean and variance by gaussian approximation [8,17,56,76,83]: 1 [x x(t)]2 P(x, t) = exp. (106) 2πσ(t) 2σ(t) The results can be extended to the multidimensional system using the same method. The probability from the above corresponds to a single fixed point or basin of attraction. If the system exhibits multi-stability, then there will be several probability distributions localized at every basin of attraction, but with different variations. Therefore, the total probability becomes the weighted sum of all these probability distributions. For instance, for a bi-stable system, P(x, t) = w 1 P a (x) + w 2 P b (x) (w 1 + w 2 = 1). Here, the weighting factors (w 1, w 2 ) reflect the relative sizes of the basins of attractions. The relative weights can be obtained by running multiple initial conditions and counting what fractions falling into each basin. For an oscillating system, the mean and variance x(t) and σ (t), are not constants at any times. They are explicit functions of times. One can obtain the steady-state results by integration of the probability in time for one period and divide by the period [8,17,56,76,83]: P oscillation = ( st+z st P o (x, t) dt)/z. Here, z is period of oscillation, and st is starting point for integration. Once we obtain the total probability, one can construct the potential landscape by U(x) = ln P ss (x) [8,17,56,76,83]. For the gene regulatory network, every parameter, node, and link contribute to the structure and dynamics of the network. Together, they determine the total probability distribution, or the underlying potential landscape. In the 44-dimensional protein concentration space, it is difficult to visualize 44-dimensional probabilistic flux [83]. We thus will explore the associated two-dimensional (2D, variables CycE and CycA) projection of the landscape and flux vector: J 1 (x 1, x 2, t) = F 1 (x 1, x 2 )P D x 1 P, J 2 (x 1, x 2, t) = F 2 (x 1, x 2 )P D x 2 P. (107) 5.3. Results By solving the dynamics of the underlying gene network, a limit cycle emerges. The limit cycle starts from the beginning of each cycle for cell growth and ends at the division of each cell cycle. This process repeats itself multiple times. From the stochastic dynamics and the corresponding diffusive probability evolution, one finds the underlying landscape and flux of the underlying cell-cycle gene regulatory network. The result is in 44-dimensional (44 species of proteins) concentration space [83]. To visualize the result, we project the landscape and flux into the two dimensions of cyc E and cyc A protein concentrations that are responsible for cell-cycle dynamics in order to present the quantitative landscape and flux picture shown in Figure 13. Onthe left-hand side of Figure 13, we show the typical text book description of the cell-cycle processes running through the distinct G1, S, G2, and M phases and the corresponding checkpoints [ ]. On the right-hand side of Figure 13, we show the landscape U as a function of cyc A and cyc E.

63 Advances in Physics 61 (a) (b) (c) Figure 13. (a) shows the four phases of cell cycle with the three checkpoints: G1,S, G2, and M phases. (b) shows the three phases ( G1, S/G2, and M ) and the two checkpoints (G1 checkpoint and S/G2 checkpoint) in our landscape view (in terms of gene CycE and CycA). (c) shows the 2D landscape, in which white arrows represent probabilistic flux, and red arrows represent the negative gradient of potential. Diffusion coefficient D = 0.05 (from Ref. [83]). We can clearly see that the global shape of the potential landscape is a Mexican hat with a central island with a high potential and a closed cell-cycle oscillation ring valley with low potential between the center island and outer higher plateaus of the landscape [83]. On the limit cycle oscillation ring valley, there are three major distinct basins. We can identify each basin according to its corresponding protein concentrations and stages of the cell cycle. We can see that the deepest local basin along the limit cycle ring valley represents the G1 phase, the next basin on the ring valley represents the S/G2 phase, and the third basin along the ring valley basin represents the M phase. Between the G1 and S/G2 basins, we see a local barrier or transition state which we can identify as the G1 checkpoint since it provides a natural point for the cell to decide whether or not to continue to go forward. Between the S/G2 and M basins, we see a local

64 62 J. Wang barrier or transition state which we can identify as the S/G2 checkpoint. Between the M and G1 basins, we see a local barrier or transition state which we can identify as the M checkpoint. In summary, we can identify the different phases (G1, S/G2, and M ) of the cell cycle as the local basins of attractions on the cycle path and checkpoints as barriers or transition states between the local basins of attractions on the cell-cycle path. The above quantitative picture of the cell cycle provides a physical foundation for the text book description of cell cycle. However, there is an issue. With the potential landscape alone as the driving force for the network dynamics, we can see that the underlying dynamics would prefer to stay in G1 basin since it has the lowest potential and highest probability. Therefore, the cell-cycle process would not continue along the limit cycle ring and it will most likely trap into the G1 phase. On the other hand, we know from landscape and flux theory that there is another driving force for general dynamical systems, the curl flux [13 15,17,62,69,83]. While outside the ring valley of the cell cycle, the curl flux is not as effective as the landscape gradient for guiding the dynamics, on the ring valley of the cell cycle, the landscape gradient tends to trap the cell into the resting G1 basin. Here the curl flux comes in for the rescue. The curl flux has rotational nature. As a driving force for dynamics, it acts along the ring valley. In fact, it is the driving force for pushing and maintaining the cell-cycle periodic motion. Therefore, the local barriers and the curl flux are the keys to push or block the cell-cycle processes. In Figure 13(c), we show the 2D contour of the landscape and flux as the driving force for the cell cycle. The white arrows represent the curl flux and the red arrows represent the negative of the potential landscape gradient. As we can see away from the basins, the landscape attracts the system down to the basin, while on the oscillation ring, the flux becomes dominant and drives the cell cycle around the oscillation ring. One may be curious about the origin of the flux as the driving force for the limit cycle. In fact, the origin of the flux is the energy input from the environment [13 15,17,57,62,67,69,83]. In the cell cycle, this energy input is provided by the nutrition supply. A signature of the nutrition supply is the GF. When the nutrition supply is intact, the GF is enhanced and cell starts to grow. The above mapping of cell cycle to a landscape and a flux gives a quantitative and physical picture of the cell-cycle process. While the landscape attracts the cell into different phases or basins, the curl flux drives the cell cycle on the oscillation ring. In the following figures, we show quantitative results to support the above physical picture. Figure 14 shows quantitatively the interplay between the barrier and flux in determining the cell-cycle dynamics. Figure 14(a) shows that as GF is enhanced, the flux increases so the driving force for the cell cycle is enhanced for completing the cycle. The barrier of the center island also becomes higher, which is important for maintaining the coherence of the cell cycle. Without the center island, the state on the one side of the cycle trajectory could jump easily to the other side of the trajectory. This would lead to the loss of the sense of the directions for the cell-cycle motion and therefore the decoherence of the oscillations. On the other hand, we can see the barrier between G2 and M phases decreases, promoting the cycle. The barrier between G1 and S/G2 increases with respect to the increase in GF. However, the magnitude of the barrier between G1 and S/G2 is much smaller than that of between G2 and M phase. Therefore, the barrier between G2 and M along with the flux is the rate limiting passage way determining the cell-cycle dynamics along the oscillation ring. When we change the regulation among the genes in the network, we uncover the correlations between the curl flux (curl flux here is quantitatively measured by the flux integrated over the cell-cycle closed-loop path) and the period of oscillation as shown in Figure 14(b). We can see that as the flux increases, the period of the cell-cycle oscillations becomes faster. We can see that the flux through nutrition supply is the driving force for the periodicity of the cell cycle. We also show that the correlation between the period of oscillation and barrier between the S/G2 and M phases in Figure 14(c). As the barrier increases, the period becomes longer. Therefore, the local barrier between S/G2 and M phases acts as a friction force for the cell-cycle oscillation.

65 Advances in Physics 63 (a) (b) (c) Figure 14. (a) shows the change (percentage) of Flux IntLoop, Barrier Center, Barrier G1/S, and Barrier G2/M when GF is changed. (b) shows the correlation between flux and period when parameters (regulation strengths or synthesis rates) are changed (correlation coefficient is 0.839). (c) shows the correlation between Barrier G2/M and period when parameters are changed (correlation coefficient is 0.879). (d) shows the correlation (correlation coefficient is 0.741) between Barrier Center ( flux for inner plot, correlation coefficient is 0.553) and coherence when parameters are changed (from Ref. [83]). This serves the function of the checkpoint. We can also see that the correlation between the coherence of the oscillation and the barrier of the center island as well as the flux in Figure 14(d). We can see that both the flux and the center island barrier height promote the coherence of the oscillations. Furthermore, we uncover in Figure 15 the correlation between the energy consumption and the quality of the cell cycle as the amount of GF changes. We can see that both the EPR and energy consumption per cycle increase with respect to the nutrition supply signatured by the increase in the GF. Since the GF promotes the flux, the energy consumption increases as the flux increases. The origin of the energy consumption is the energy input through the GF. The physical realization of the nutrition supply is through flux acting as one of the driving forces for cell cycle. On the other hand, we can see as the central barrier increases, the energy consumption increases. This is due to the fact that center barrier promotes the coherence of oscillations and the oscillations consume energy. Furthermore, we can see that faster oscillations consume more energy as expected. At last, as the barrier between G2 and M becomes smaller, the cell-cycle oscillation is faster. Therefore, the resulting energy consumption is larger. (d)

66 64 J. Wang (a) (b) (c) (d) (e) Figure 15. (a) and (b) show the change of EPR and energy per cell-cycle increase as GF is increased. (c) (f) show separately the Flux, Barrier Center, Barrier G2/M, and Period versus the energy per cell cycle with GF changed. It can be seen that the energy per cycle increases as the Flux and Barrier Center increase, while decreases as the Barrier G2/M and Period increase (from Ref. [83]). To study the functions of the cell cycle from the network structural perspective, one can carry out a global sensitivity analysis on the cell-cycle period, the landscape barrier, and the flux, to find out which key genes or regulatory wirings are responsible for cell-cycle function in Figure 16. Exploring the parameters of the synthesis rates and regulation strengths (41 parameters were selected), Figure 16(a) shows the corresponding changes in period, flux, and the central barrier (Barrier Center ) when these 41 parameters are separately changed. Figure 16(b) shows the corresponding changes of Barrier G1/S (G1 checkpoint) and Barrier G2/M (DNA replication checkpoint) when these 41 parameters were separately changed. By selecting the top genes and regulations influencing the function (those influencing the flux and period the most in Figure 16), we can identify certain key factors for the cell cycle progression. Some of the identified key genes and regulations have been confirmed by experiments. For instance, prb serves as a key gene in controlling G1 checkpoint. This is due to the fact that its activation represses the cell-cycle process and the cancer [168,169]. The results from the global sensitivity analysis (Figure 16) are consistent with the above findings. We see that the activation of prb leads to the decrease in the curl flux, the increase in the barrier between G2 and M phases (f)

67 Advances in Physics 65 (a) (b) Figure 16. Global sensitivity analysis in terms of the barrier (Barrier Center for global stability, Barrier G1/S for G1 checkpoint, Barrier G2/M for S/G2 checkpoint), period, and flux changes when parameters are changed. x coordinate (1 41) is corresponding to the 41 parameters (synthesis rate or regulation strength as follows: 1: AP1, 2: E2F, 3: prb, 4: synthesis of CDC25 acting on CycE/Cdk2, 5: Skp2, 6: Cdh1, 7: synthesis of CDC25 acting on CycA/Cdk2, 8: p27 synthesis independent of E2F, 9: p27 induced by E2F, 10: synthesis of CycB, 11: Cdc20, 12: synthesis of Cdc25 acting on CycB/cdk1, 13: synthesis of Wee1, 14: CycD induced by AP1, 15: CycD induced by E2F, 16: CycD bind Cdk4,6, 17: CycD/Cdk4,6 bind P27, 18: CycE induced by E2F, 19: CycE/Cdk2 bind p27, 20: CycE bind Cdk2, 21: CycA bind Cdk2, 22: CycA/Cdk2 bind p27, 23: CycB/Cdk1 bind p27, 24: CycB bind Cdk1, 25: inhibition of prb to CycD, 26: CycE inhibited by prb, 27: CycA inhibited by prb, 28: p27 inhibited by prb, 29: Cdc25 activate CycA/Cdk2, 30: Wee1 inhibit CycA/Cdk2, 31: CycA/Cdk2 activate Cdc25, 32: CycA/Cdk2 and CycB/Cdk1 inhibit Cdh1, 33: CycE/Cdk2 inhibit p27, 34: Cdc25 activate CycB/Cdk1, 35: Wee1 inhibit CycB/Cdk1, 36: Cdc20 (inactivate form), 37: CycB/Cdk1 activate Cdc20, 38: CycB/Cdk1 activate Cdc25, 39: CycB/Cdk1 inhibit Wee1, 40: CycE/Cdk2 activate Cdc45, 41: ATR activate Chk1 ). Here, every parameter is changed 10% (from Ref. [83]). Barrier G2/M, and eventually the elongation of the cell cycle. The analysis also provides further predictions (Ref. [83, Table 1]), which can be tested by experiments. The key sensitivity analysis results (key genes and regulations) in the wiring diagram for the cell-cycle network (Figure 12) with red color.

68 66 J. Wang It has been observed that the cell-cycle period of a cancer cell is significantly faster than that of the normal cell. This leads to uncontrollable cycling. In other words, the two driving forces of the cell-cycle dynamics, the landscape barrier and the flux are unbalanced. The curl flux dominates the effects of potential landscape barrier along the cell-cycle path. In order to restore the balance between the flux and potential landscape, one needs to either decrease the flux or increase the barriers for the cancer cells to become normal. This can be realized by changing the nutrition supply and perturbing the genes and regulatory wirings through genetic or environmental changes according to the results of our global sensitivity analysis based on the underlying landscape topography and curl flux. In summary, an underlying Mexican hat shape landscape controls the mammalian cell-cycle process. Landscape topography, quantified by the barrier heights between basins of attractions can provide a quantitative measure of the global stability and function of the cell cycle [17,57,83]. Different phases of the cell cycle (G1, S/G2, M) are identified as the local basins of attractions on the cycle path and checkpoints as barriers or transition states between the local basins of attractions on the cell-cycle path. There are two driving forces which determine the progression of the cell cycle. One driving force is from probability flux (that originates from the nutrition supply) along the cell-cycle path. The other determinant is the potential barriers along the cellcycle path, characterizing the cell-cycle checkpoints. Landscape and flux theory provides a simple physical and quantitative picture for the mechanism of cell-cycle checkpoints. 6. Cell fate decisions: stem cell differentiation and reprogramming, paths, and rates The normal cell has a few alternative fates. One is to self renew as a primary or stem cell through the cell cycle. Another is to change from stem cell to the differentiated cell. The differentiation process is the basis for the development. Therefore, stem cell differentiation is one of the most fundamental processes in biology. It is now known that the stem cell differentiation is determined by underly gene regulatory networks. Recently, there have been new developments in the field. Researchers have been able to use a few regulatory genes to control the reverse process of cell differentiation, thus reprogramming the cell. Researchers are able to take a differentiated cell from an animal s body and transform it back to a so-called induced multi-potent stem cells, a multi-potent stem cell [170,171]. This opens up the door for manipulating the cell differentiation and reprogramming process. The potential applications for regenerative medicine are obvious. The dream to use one s own cells to regenerate or repair damaged organs may become true not too long from now. Significant efforts have been made toward understanding the mechanisms and global picture of the cell differentiation and development. Most noticeably, Waddington proposed his idea for development [172]. In his original picture drawn by his artist friend, the developmental process can be viewed as a marble rolling down from the top to the bottom of the mountain valleys. Here the top of the mountain represents the stem cell while the bottom of the mountain valleys represent the fates of differentiated cells. The journeys from the stem cell to the differentiated cell on the mountain become the developmental paths. The Waddington picture for development is useful in visualizing the developmental process. It provides an intuitive understanding of development. This picture has influenced the generations of biologists in thinking about development and differentiation. However, Waddington s description of development is only at an intuitive and qualitative level. There were no physical and quantitative basis for this idea when he proposed it. This is the reason why people can only use it as a metaphor rather than a real physical theory. Recently, there have been important theoretical works on cell fate decision in the context of phenotypic transitions in gene regulations, cell development, differentiation, and metastasis

69 Advances in Physics 67 of epithelial mesenchymal transition [14,22,46 52,62,66,76,77,82,84, ,126, ]. A quantitative description of the Waddington landscape can be found by applying landscape and flux theory, yielding a global, physical and quantitative theory for development [60,62,66,79] The Waddington landscape for key modules of stem cell differentiation, paths, and rates To illustrate the quantitative Waddington picture for development, we first focus on a well-studied gene module for development. This gene module is the key one determining many of the cell fate decision-making processes in the development. The structure of this gene module involves two self-activating genes mutually repressing each other. The dynamics of this circuit is described by a set of two variable ordinary differential equations below, for the rates of the expression [60,62,79,176]: dx 1 dt dx 2 dt = a 1x n 1 S n + x n + b 1S n 1 S n + x n k 1 x 1 = F 1, 2 = a 2x n 2 S n + x n + b 2S n 2 S n + x n k 2 x 2 = F 2, (108) 1 where x 1 and x 2 are the time-dependent expression levels of the two cell-specific transcription factors X 1 and X 2 [60,62,79,176]; parameters a 1 and a 2 are the self-activation strength of the transcription factors X 1 and X 2, respectively; b 1 and b 2 are the strength of the mutual repression for transcription factors X 1 and X 2, respectively; k 1 and k 2 are the first-order degradation rates for X 1 and X 2, respectively [60,62,79,176]; S represents the threshold (inflection point) of the sigmoidal functions, that is, the minimum concentration needed for appreciable changes; and n is the Hill coefficient which represents the cooperativity of the regulatory binding and determines the steepness of the sigmoidal function. We can see that varying these parameters can lead to bi-stable states or tri-stable states and to phase transitions between different sorts of behavior. Here, the parameters for Hill function and degradation rate for X 1 and X 2 are specified as follows: S = 0.5, n = 4, and k = k 1 = k 2 = 1.0 [60,62,79,176]. For illustration, this section uses results for the symmetric situation a = a 1 = a 2 and b = b 1 = b 2. Although the values of parameters can be different in organisms under different circumstances, the mathematical model here describes a simplified gene circuit, and these values (S = 0.5, n = 4, k = k 1 = k 2 = 1.0) have been used in many previous studies [60,62,79,176]. For these fixed parameters and the above dynamical equations for the gene module with external fluctuations, one can explore the underlying stochastic dynamics of this gene circuit and map out the corresponding landscape. The landscape is shown in Figure 17. There can be three basins or two basins in general according to different self-activation regulation strengths. At high selfactivation regulation strengths, three basins of attraction emerge. Two basins of attraction emerge for low values of self-activation. The central basin is symmetric and represents a moderate concentrations of both genes. This is identified as the IPS multi-potent stem cell basin. On the other hand, the two side basins represent the differentiated cell fates (one marker gene has high expression and the other is suppressed). In this way, the cell fates of stem cell and differentiated cells can be quantified as basins of attractions on the landscape. The developmental process of differentiation or reprogramming can be seen as a cell fate decision process allowing a transition from one basin of attraction to another (stem cell to differentiated cell or vice versa). The paths between these cell fate basins represent the developmental or reprogramming paths. The barriers and the kinetic time needed to move in between the basins will quantify the degrees of difficulties of switching from one cell fate to another. The effective self-activation strengths of gene regulations change during the developmental process. Therefore, at high self-activation in the initial stage of

70 68 J. Wang Figure 17. The landscape of the development at different stages or different parameters a (from Ref. [62]). cell development, the stem cell basin dominates while at a later stage of the development, the two differentiated cell basins dominate. We can treat the self-activation gene regulation strength itself as a dynamical variable during the developmental process [60,62,79,176]. da dt = λa. (109) We can see that landscape starts to change from one for the original dominant stem cell stage to one for the final differentiated cell dominating era. Taken this together, this analysis yields a quantitative Waddington landscape for differentiation and development as shown in Figure 18. The advantage of this approach is obvious. (1) One can now quantify the Waddington picture for development and differentiation. The development and differentiation can be mapped quantitatively to a landscape. (2) At the beginning of the development, the stem cell has a stable basin of attraction. This is in contrast to the original Waddington picture of development where the stem cell was described as sitting on top of the hill or barrier. In fact, this implies that in order to transform from stem cell to differentiated cell, the stem state has to go over the barrier to reach to the final differentiated state. (3) We can quantify the dominant paths from the stem cell to the differentiated cell. This gives the major developmental pathways. On the other hand, we can also quantify the dominant paths from the differentiated cell state to the stem cell. This predicts a major reprogramming pathway. Quantification of the reprogramming pathway is critical for the tissue engineering and regenerative medicine. Notice that these two pathways are not reversals of each other. This is due to the fact that the underlying network does not satisfy the detailed balance and therefore there is a curl flux as an additional driving force going beyond the landscape gradient for the dynamics. For this concrete example of mutual repression with self-activation cell fate decision-making, we can compare the difference in predicted rates using the equilibrium TST, the zero noise limit (WKB) non-equilibrium TST predictions (optimal path going through the saddle point) [43], and the present non-equilibrium TST prediction where the barrier for determining the kinetic process is taken from the saddle on the dominant path rather than the conventional saddle on the landscape by itself [82]. We can also compare these to direct simulation results for both

71 Advances in Physics 69 Figure 18. The quantified Waddington developmental landscape and pathways (from Ref. [62]). the forward and backward directions as shown in Figure 19. We can see that the results based on the present TST are more consistent with the simulations compared to other methods in this example. The major difference of the present method to the earlier ones [22,43 52] lies in the fact that the barrier or action and prefactor is evaluated at the optimum on the dominant path rather than the saddle on the landscape. In the zero noise limit, all approximations agree that the optimal path goes through the landscape saddle point. Some differences of the present method compared to others may arise, when we are working on the path integral or action in the coordinate x space representation, but not in the phase space of x and p representation as some of the others. The optimal paths in the x and p spaces may not be the same as the optimal path in the coordinate x space by itself. For the small but finite noise case considered here, the reason why the optimal path does not follow the steepest descent path is the presence of the curl flux in addition to the force from the gradient of the non-equilibrium potential. The reason why the optimal path does not necessarily go through the saddle point making the present TST necessary (on the optimal path rather than on the landscape saddle) seems to lie in the fact that an extra term not present in zero noise limit in the path integral action kicks in (the divergence of the force term due to the variable (Jacobian) transformation from noise to observables). We want to emphasize that the present non-equilibrium TST not only takes into account of the exponential part of the rate, the barrier heights on the optimal path (not on the landscape saddle), but also the prefactor at the saddle on the optimal paths rather than at landscape saddle. This contrasts with the conventional rate studies where the barrier height is measured at the landscape saddle and the prefactor is measured as fluctuations around that saddle and stable basin.

72 70 J. Wang Figure 19. The mean first passage time (MFPT) of the reprogramming (S S ) from our theoretical predictions (non-equilibrium TST ), Langevin dynamics simulations, zero noise approximations, and equilibrium TST, for different cell volume V (different fluctuation levels) (from Ref. [82]) The Waddington landscape and epigenetics Since the cellular processes are all determined by the underlying regulatory networks, there are two components of the cell networks that can influence the cell behavior. One is the nodes, the (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 20. The potential landscape (3D view) in the n A n B plane for different self-activation strength F A and binding/unbinding speed ω (from Ref. [66]).

73 Advances in Physics 71 genes. The other is the links or regulation strengths among genes. Epigenetics plays an important role in cellular process. It provides the source of change through the regulation changes other than the genes themselves for influencing the biological function. For many gene networks or circuits, the dynamics is studied by directly following the gene products, the proteins. So implicitly people have assumed that proteins faithfully represent the genes. This is an okay assumption when the regulations of the genes through proteins are relatively fast and there is a tight coupling between proteins and genes. However, due to the epigenetics of DNA methylation and histone remodification during the gene regulations in eucaryotic cells, the regulatory proteins have to take a longer series steps in order to reach the target gene. Therefore, the coupling between the genes and proteins is less strong and we have to consider both of them through different timescales of epigenetics. Effectively, we enter the non-adiabatic weak coupling regime of gene dynamics instead of the usually assumed adiabatic dynamics with strong coupling of proteins and genes. In order to see this effect, we can explicitly take into account of the regulatory binding steps of the proteins to the DNA. The epigenetic effects are then reflected in the slow timescales (through DNA methylation and histone modification). The result is shown in the following figure. We use the ratio of the unbinding rate versus binding rate as the measure of the slow timescale from the epigenetic effect. We also use selfactivating strength to monitor the developmental process. In Figure 20, along the developmental process, we can see that gradual cell fate changes from the central stem cell basin to the differentiated cell basis. On the other hand, as we turn on the epigenetic effects of slow binding mimicking DNA methylation and histone remodifications, the epigenetics shows four effects on the development. (1) It provides another mechanism of differentiation. Through enhancing the effects of slow regulatory binding, we can see that the cell fate can transform from the stem cell (a) (b) (c) Figure 21. The MFPT of the differentiation and reprogramming for different self-activation strength F A and binding/unbinding speed ω (from Ref. [66]).

74 72 J. Wang (0.5,0.5) to differentiated cells (1,0) or (0,1). (2) New states can be generated. As we can see, some intermediate or differentiated states can be formed (0,0). (3) Furthermore, there are many more metastable basins forming around the differentiated and stem cell states. This explains the inhomogeneous distributions of the stem and differentiated cells. The inhomogeneity is often observed in the experiments but has puzzled many since it has had no good explanation. The landscape study gives a quantitative explanation of the origin of the inhomogeneous distribution of states through epigenetic effects of slow binding. (4) The rate for differentiation has an optimum speed with respect to the timescales of the regulatory binding shown in Figure 21. This suggests that one can reach optimal differentiation through the modulation of epigenetics. Future experiments should test this theoretical prediction The Waddington landscape for large networks We have also explored a more realistic gene network underlying the human embryonic developmental process [76]. This network has been constructed from collecting the experimental Figure 22. The wiring diagram for the stem cell developmental network including 52 gene nodes and their interactions (arrows represent activation and perpendicular bars represent repression). The magenta nodes represent 11 marker genes for the pluripotent stem cell state, cyan nodes represent 11 marker genes for the differentiation state, and the yellow nodes represent genes activated by the stem cell marker genes. The solid black links represent the key links found by the global sensitivity analysis, and the octagon shape nodes represent key stem cell and differentiation markers found by global sensitivity analysis (from Ref. [76]).

75 Advances in Physics 73 (a) (b) Figure 23. A bi-stable landscape picture for the stem cell network. Parameters are specified as follows: k = 1 (degradation), b = 0.5 (repression), a = 0.37 (activation), and diffusion coefficient D = (a) 3D landscape and dominant kinetic paths. The yellow line represents developmental path, and the magenta line represents reprogramming path. (b) 2D dominant kinetic path and flux on the landscape. The white arrows represent the direction of flux, and the red arrows represent the direction of the negative gradient of potential energy (from Ref. [76]).

76 74 J. Wang literatures [177]. It has 52 genes with 11 stem cell marker genes and 11 differentiated cell marker genes as shown in Figure 22. We found that even with many more genes involved, the basic picture of development remains the same as for the two gene module study. As shown in Figure 23, the resulting developmental landscape has two major basins, one is the stem cell and the other is the differentiated cell. The developmental process can be quantified as the transformation of cell fate from stem cell down to the differentiated cells. Furthermore, we can quantify the dominant pathways for differentiation and reprogramming. They are irreversible. By quantifying these pathways, we can see the detailed process of how the cell fate decision actually is made. We can see in Figure 24 that the developmental pathways starts from the embryonic stem cell stage of high nanog, low GATA6, and low cdx2 protein concentrations, through the route of low nanog, low GATA6, and low cdx2 protein concentration state, and further to the low nanog, low GATA6, and high cdx2 protein concentration state, and finally reach the differentiated state with low nanog, high GATA6, and high cdx2 protein concentrations. For the reprogramming pathways, Figure 24. Differentiation and reprogramming process represented by 313 nodes (every node denotes a cell state, characterized by expression patterns of the 22 marker genes) and 329 edges (paths). The sizes of nodes and edges are proportional to the occurrence probability of the corresponding states and paths, respectively. Red nodes represent states which are more close to stem cell states in terms of gene expression pattern, and blue nodes represent states which are more close to differentiation states. The green and magenta paths denote dominant kinetic paths from path integral separately for differentiation and reprogramming. Here, we set a probability cutoff to decrease the number of states and paths, that is, we only demonstrate the states and paths with higher probability (from Ref. [76]).

77 Advances in Physics 75 the network starts from the differentiated state with low nanog, high GATA6, and high cdx2 concentration, through the route of high nanog, high GATA6, and high cdx2 protein concentration state, and further to the high nanog, low GATA6, and high cdx2 protein concentration state, and finally reach the differentiated state with high nanog, high GATA6, and low cdx2 concentration stem cell state. These identifications and quantifications of the developmental and reprogramming pathways not only uncover the underlying mechanisms, but also quantify the route by which (a) (b) (c) (e) (d) (f) Figure 25. The barrier height and MFPT results when the activation strength a, the repression strength b as well as the noise level D changes (Langevin dynamics). (a) and (b) show that when a increases, stem cell state becomes more stable, the barrier for stem cell state U SP (or the barrier for differentiation process U differentiation ) increases, and the MFPT for differentiation process from stem cell state to differentiation state (τ differentiation ) increases. By contrast, When a increases, differentiation state becomes less stable, the barrier for differentiation state U SD (or the barrier for reprogramming process U reprogramming ) decreases, and the MFPT for reprogramming process from differentiation state to stem cell state (τ reprogramming ) declines. (c) and (d) show that when b increases, the barrier for stem cell state U SP (U differentiation ), the barrier for differentiation state U SD (U reprogramming ), the MFPT for differentiation process (τ differentiation ), and the MFPT for reprogramming process (τ reprogramming ) all increase. (e) and (f) show that when noise level D increases, the barrier for stem cell state U SP (U differentiation ), the barrier for differentiation state U SD (U reprogramming ), the MFPT for differentiation process (τ differentiation ), and the MFPT for reprogramming process (τ reprogramming ) all decrease (from Ref. [76]).

78 76 J. Wang development and reprogramming are realized. This is important and should provide a quantitative strategy for guiding reprogramming and development in regenerative medicine. The kinetics of the transformation from stem cell to differentiated cells or vice versa is controlled by the landscape topography quantified by the barrier heights between the two basins of attraction as illustrated in Figure 25 [76]. The changes in barrier heights and kinetics quantified by the MFPT are illustrated in Figure 25 upon changes in regulation activation, regulation repression, and fluctuations. By performing the global sensitivity analysis of the landscape topography through the barrier heights between the stem cell fate basin and differentiated cell fate basin, one can identify which genes and regulations are important in the developmental network and responsible for the development and reprogramming process where the degree of difficulty of the underlying cell fate decision-making process is controlled by the barrier height in between. The (a) (c) (e) (b) (d) Figure 26. Results of the global sensitivity analysis in terms of barrier height and MFPT when parameters are changed. The results in (a) are for six repression links (named, respectively, as R1,R2,..., R6, see Ref. [76, Table S2]) based on the change of barrier heights ( Barrier ). The results in (b) are for 14 activation links (named, respectively, as A1,A2,..., R14, see Table S3) based on barrier heights. Blue bars represent the change of U SP (barrier for differentiation process), red color represents the change of U SD (barrier for reprogramming process). (c) and (d) separately show the corresponding results in terms of the change of MFPT ( MFPT). Blue bars represent the MFPT change for differentiation process, and red bars represent the MFPT change for reprogramming process. (e) shows the corresponding global sensitivity for the knockdown of individual genes (from Ref. [76]).

79 Advances in Physics 77 resulting identified key genes and regulations are indicated in Figures 22 and 26. These are the hot spots for the development and reprogramming. They are critical in determining the cell fate decision-making process. Therefore, by exploring the landscape topography, we can study the underlying network structure and determine the backbone for the biological function and behavior. This will not only help us to understand the underlying mechanism, but may also help the network design and new generation of network drug discovery. 7. Landscapes and paths of cancer 7.1. Introduction Cancer presents a serious threat to human health. For example, breast cancer is the most frequent malignancy in women. Worldwide, breast cancer comprises 22.9% of all cancers (excluding nonmelanoma skin cancers) in women. About 1 in 8 US women will develop invasive breast cancer over the course of her lifetime. In 2008, breast cancer caused 458,503 deaths worldwide [ ]. Though most breast cancers are benign and curable by surgery, one-quarter have a latent and insidious character, growing slowly but metastasizing early. Current therapies delay tumor progression significantly, but recurrence is inevitable, resulting in high mortality rates [181]. About 5 10% of breast cancers can be linked to gene mutations (abnormal changes) inherited from one mother or father. Mutations of the BRCA1 and BRCA2 genes are the most common. Women with these mutations have up to an 80% risk of developing breast cancer during their lifetime [181]. There are specific genes in the cells of our bodies that normally help to prevent tumors from forming. One of these tumor-suppressor genes is P53. The function of P53 is to suppress cells from growing. When it has been damaged or altered, P53 loses its ability to block cell growth, and thus results in an increased risk of cancer. Almost 50% of all human cancer cells contain a P53 mutation. These cancers are more aggressive and more often fatal. Conventionally, the cancer has been seen as a diseases caused by mutations [182]. This has guided the thinking of the drug discovery industry for the last 50 years. Despite the efforts, cancer still presents a major threat to human health. Recently, it becomes more and more clear that cancer is not a disease solely from some single gene-mutation, but a disease of state [50,84, ,173, ]. More observations have appeared that contradict the paradigm of mutationdriven tumor-genesis. Cancer should be seen as a particular natural state that originates from cell regulation. In other words, cancer should be seen as a network disease rather than as coming from a single mutation. The cancer state is often hidden under the complex molecular networks and therefore normally is inaccessible [183]. These molecular networks form different kinds of cell types through evolution. The microenvironment can be equally important as mutations and perhaps even more for the cancer formation. While mutations can influence the network, microenvironments although not causing the genetic changes do change the network wirings or the interaction strengths between the genes in the underlying gene regulatory network. The changes of the interactions between genes can also alter the natures of the resulting states and cell types. Therefore, driver gene mutations may not play the absolute dominant role in causing a stepwise cancer cell phenotype formation. Rather, their role might be to allow the cells to have the access to these hidden cellular states depending on the microenvironments. Cancer would then be viewed as an intrinsic state and only released through a series of mutations controlled by the microenvironments. Cancer states should be naturally emerging functional entities, the result of collective action of all the gene gene interactions in the gene regulatory network. They form a basin of attractions around the states. They become robust due to the flow of the unstable states into them. This implies that with a small perturbation to a state within the basin, the system will return to the

80 78 J. Wang attractor state. Importantly, a single network can give rise to multiple attractors, leading to multistability. Thus, to learn the mechanism of cancer, we need to investigate the underlying gene regulatory networks and associated dynamics (usually represented by a wiring diagram including gene nodes and their interaction links), which govern the evolution and behavior of normal and cancer states. The cancer gene regulatory networks are dynamical systems. Intrinsic stochasticity is present due to statistical fluctuations from a finite number of molecules of the network, and external stochasticity is present due to highly dynamical and inhomogeneous environments. Thus, we should study the stochastic dynamics of cancer network dynamics in fluctuating conditions in order to model realistically the cellular inner and outer environments. There have been increasing numbers of studies on the global topological structures of the network systems, recently [190]. The underlying nature of the networks has been explored by experimental research [191,192]. The conventional way of describing the network dynamics is often in terms of either deterministic or stochastic chemical kinetics and follows the temporal trajectories of system variables. It often probes only the local natures of the network such as local stability around fixed point state [ ,183]. However global nature of the system cannot be easily revealed from such analysis. As mentioned, the biological function is often realized by gene regulatory networks at the cell level. The global stability and behavior of cellular networks are essential for performing the biological functions. However, the quantifications of the global stability, function, and behavior of cancer networks as well as the key factors determining the underlying dynamics present us a challenge in system biology. Solving this problem is crucial for understanding the functions and mechanism of cancer. For cancer cellular network with its huge state space, understanding how seemingly infinite number of genotypes can produce a finite number of functional phenotypes (i.e. normal and cancer states) is challenging. Rather than focusing on the individual trajectory dynamics, a probabilistic description provides a global quantification. Different states correspond to different probabilities of appearance. The functional state should possess a higher probability of appearance or lower potential energy, whereas nonfunctional states have lower probability or higher potential [13,17]. The dynamical system of cancer gene networks is not at equilibrium due to the constant exchange of energy, information and matter from the environments. The equilibrium landscape recipe does not apply here. Recently, there have been increasing efforts in exploring the physics of cancer network dynamics [50,84, ,173, ]. Despite the progress made, there are still challenges on the origin and underlying mechanisms, the driving forces, the quantification of landscape, the paths and speed, the key genes and regulations of cancer. Non-equilibrium landscape and flux theory can be applied to cancer to meet some of the aforementioned challenges [84]. Analysis shows the following: (1) The normal states and cancer states can be quantified as basins of attractions on the cancer landscape, the depth of which represents the associated probabilities. (2) The global stability, behavior, and function of cancer through underlying regulatory networks can be quantified through the landscape topology in terms of depths of and barriers between the normal and cancer states. (3) The paths and transition rates from the normal (cancer) state to the cancer (normal) state representing the underlying tumor-genesis and reverse processes will be quantitatively uncovered, through a path integral approach and non-equilibrium TST. (4) Global sensitivity analysis based on landscape topography, again allows one to identify the key genes and regulations important to the global stability, behavior, and function of

81 Advances in Physics 79 cancer networks. The results of global sensitivity analysis will provide multiple targets for cancer Results The underlying cancer gene regulatory network Ten hallmarks of cancer [168,169,182] have been proposed previously, characterized by some key cancer marker genes. Each hallmark is often involved in a set of functionally linked pathways. This makes the mapping of the functional modules and the mutated genes onto a cancer network possible. One can use the network to uncover the underlying relationships, insights, mechanisms, and principles of cancer [189]. Starting from cancer marker genes and certain critical tumorsuppressor genes such as P53, RB, P21, and PTEN, through an extensive literature search [193], one can construct a cancer gene regulatory network made of 32 gene nodes (Figure 27) and 111 edges (66 activation interactions and 45 repression interactions). In Figure 27, the arrows represent activation and the short bars represent repression. The network has mainly three types of marker genes: apoptosis marker genes (green nodes, including BAX, BAD, BCL2, and Caspase), cancer marker genes (magenta nodes, including AKT, MDM2, CDK2, CDK4, CDK1, NFKB, htert, VEGF, HIF1, HGF, and EGFR), and tumor repressor genes (light blue nodes, including P53, RB, P21, PTEN, ARF, and CDH1). The brown nodes represent other genes. The dynamics of this gene network can be described by corresponding ordinary differential equations and the interactions among genes in terms of Hill functions representing their activation or repression strengths and cooperativity. These equations have the mathematical form as follows: F i = k X i + a Xa i + b Sn. (110) S n + Xa i S n + Xb i In Equation (110), i = 1, 2,..., 32. S represents the threshold (inflection point) of the sigmoidal functions, representing the strength of the regulatory interaction, and n is the Hill coefficient which determines the steepness of the sigmoidal function [84]. The Hill function parameters are given as S = 0.5, n = 4. Furthermore, k represents the self-degradation constant, b represents the repression constant, and a represents the activation constant. Xa i and Xb i represent the average interaction strengths, respectively, for the activation and the repression from the other nodes to the node i. At each node i, Xa i can be defined as follows: ((Xa(1) n M (a(1), i) + X a(2) n M (a(2), i) + +X a(m1) n M (a(m1), i))/m1, and Xb i can be defined as follows: ((Xb(1) n M (b(1), i) + X b(2) n M (b(2), i) + +X b(m2) n M (b(m2), i))/m2. a(1), a(2),..., a(m1) represents a list of the nodes with the activation interactions to node i, and b(1), b(2),..., b(m1) represents the list of nodes with the repression interactions to node i. M (j, i) (i, j = 1, 2,...,32) represents the interaction matrix M with the strengths from node j to node i. The model makes the simplified assumption that the regulation from one individual gene j to the other genes has the same interaction strength The landscape of the cancer network From the above 32 ODEs, one can identify the driving force governing the cancer network dynamics by considering the corresponding stochastic dynamic [13,17,62]. By applying the self-consistent mean field approximation, one finds the steady-state probability distribution of the cancer network in 32 gene concentration variables. According to the relationship between the underlying landscape and the probability, U = ln(p ss ) [13,17,62,76,84], one can further acquire the potential landscape of the cancer gene network. Here, P ss represents the steady-state probability distribution, and U denotes the dimensionless potential energy.

82 80 J. Wang Figure 27. The diagram for the cancer network including 32 nodes (genes) and 111 edges (66 activation interactions and 45 repression interactions). Red arrows represent activation and blue filled circles represent repression. The network includes mainly three kinds of marker genes: apoptosis (green nodes), cancer marker genes (magenta nodes), and tumor repressor genes (light blue nodes). The cancer marker genes include EGFR for proliferative signal, VEGF for angiogenesis, HGF for metastasis, htert for unlimited replication, HIF1 for glycolysis, CDK2 and CDk4 for evading growth suppressors. The solid black links represent the key links found by the global sensitivity analysis, and the octagon shape nodes represent key genes for the transition between normal and cancer states found by global sensitivity analysis. The brown nodes represent other genes (from Ref. [84]). For visualization purposes, we can project the 32-dimensional landscape onto a 2D state space by integrating out the other 30 variables and leaving the 2 key genes, AKT (an oncogene) and RB (a tumor repressor gene). Figure 28 shows 3D landscape and 2D contour map for the entire cancer network in gene expression state space in AKT and RB. In Figure 28(a), three stable states or basins of attraction on the landscape (tri-stability) emerge. The landscape reflects the steadystate probability distribution. Every basin of attraction (high probability states) can represent a cell type in gene expression state space. These basins are separated by certain barriers, preventing easy conversions among different cell types. The bottom attractor denotes apoptosis state, having higher expression of tumor repressor gene RB, P21, PTEN, lower expression of oncogene AKT, EGFR, VEGF, HGF, HIF1, htert, MDM2, CDK2, CDK4, and higher expression level of apoptosis marker gene Caspase. The middle attractor denotes the normal state, having higher expression level of tumor repressor gene RB, P21, PTEN, higher expression level of oncogene AKT, EGFR, VEGF, HGF, HIF1, htert, MDM2, CDK2, CDK4, and lower expression level of apoptosis marker gene Caspase. The top attractor denotes the cancer state, having

83 Advances in Physics 81 (a) (b) Figure 28. The tri-stable landscape for the cancer network from self-consistent approximation method. The parameters are set as follows: degradation constant k = 1, activation constant a = 0.5, and repression constant b = 0.5 (see supporting information for parameter depictions). Diffusion coefficient, D = (a) shows the 3D landscape and dominant kinetic paths. The yellow path represents the path from normal state attractor to cancer state attractor, and magenta path represents the path from cancer state attractor to normal state attractor. Black paths represent the apoptosis paths for normal and cancer states. (b) shows the corresponding 2D landscape of cancer network. Red arrows represent the negative gradient of potential energy, and white arrows represent the probabilistic flux (from Ref. [84]).

84 82 J. Wang lower expression level of tumor repressor gene RB, P21, PTEN, much higher expression level of oncogene AKT, EGFR, VEGF, HGF, HIF1, htert, MDM2, CDK2, CDK4, and lower expression level of apoptosis marker gene Caspase (see Table S3 in supporting information for detailed relative gene expression level of three stable states of Ref. [84]). The three attractors or stable states are consistent with our biological understanding of normal, cancer, and apoptosis state in cancer network [168,169,182]. The tri-stable landscape exists in certain regulation ranges and provides a relatively balanced case for 3 (normal, cancer, and apoptosis) state coexistence. We can explore the transitions among these three attractors. Changing the nodes to mimic the mutations or regulation strengths mimicking the non-genetic environmental changes leads to changes of the landscape topography. The effects range from there being a single dominant basin to bi-stable basins and to tri-stable basins or vice versa Dominant paths among normal, cancer, and apoptosis states The path integral method [14,62,76,84] yields dominant paths between the normal cell state, the cancer cell state, and the apoptosis state basin. In Figure 28, the yellow path denotes the dominant path from the normal to the cancer attractor, and the magenta path denotes the dominant path from the cancer to the normal attractor. We can see that the dominant paths between normal and cancer states are irreversible. We also see the apoptosis paths for normal and cancer states (black paths) to the death. We can quantify the probabilistic flux of the cancer network, shown on the landscape (Figure 28(b)). The white and red arrows represent the direction of probabilistic flux and the negative gradient of the potential landscape, respectively. We see that the dynamics of the cancer system is determined by both the force from the gradient of potential and the force from the curl flux [13]. The force from the curl flux leads the deviation of the dominant paths from the steepest descent path calculated from gradient of potential. Therefore, we see the two dominant paths from the normal to the cancer state and from the cancer to the normal state are irreversible (yellow line and magenta line are not identical). The landscape in Figure 28 merely shows a 2D projection of the whole 32 dimensional state space. To see the paths in the whole state space, one can project the continuous expression level of the 21 major marker genes (reducing the original dimensionality of 32) to high and low binary expressions in each gene (2 21 cell states totally) (ATM, P53, P21, PTEN, CDH1, RB, ARF, AR, MYC, AKT, EGFR, VEGF, HGF, HIF1, htert, MDM2, CDK2, CDK4, CDK1, E2F1, and Caspase). The normal state can be represented by the binary number (denoting the expression level from gene 1 to gene 21, 1 for high expression, 0 for low expression), and for the cancer state, it is represented by For the apoptosis state, it is represented by Figure 29 shows the discrete cancer landscape represented by 247 cell states (nodes representing the states, characterized by expression patterns of the 21 marker genes) and 334 transition jumps (edges) between the different cell states. The sizes of the nodes and edges are proportional to the occurrence probability of the corresponding states and paths, respectively. Blue nodes denote the cell states closer to normal cell states, red nodes denote the cell states closer to cancer states, and green nodes denote the cell states closer to apoptosis states. The largest blue node (high RB/high AKT/low Caspase) shows the most significant normal state, the largest red node (low RB/high AKT/low Caspase) shows the most significant cancer state, and the largest green node (high RB/low AKT/high Caspase) shows the most significant apoptosis state. There are 21 dimensional dominant paths shown as green and magenta paths separately for the normal to cancer transformation and for the cancer to normal process, and black paths label apoptosis paths of normal and cancer states. We see again that the path from the normal to the

85 Advances in Physics 83 Figure 29. Discrete landscape for cancer network with 247 nodes (every node denotes a cell state, characterized by expression patterns of the 21 marker genes) and 334 edges (paths) at default parameter values (a = 0.5, b = 0.5, k = 1). The sizes of nodes and edges are proportional to the occurrence probability of the corresponding states and paths, respectively. Blue nodes represent states closer to normal cell states, red nodes represent states closer to cancer cell states, and green nodes represent states closer to apoptosis cell states. The green and magenta paths denote dominant kinetic paths from path integral separately for normal to cancer attractor and cancer to normal attractor, and black paths represent the apoptosis paths for normal and cancer state. Here, we only demonstrate the states and paths with higher probability by setting a probability cutoff. The largest blue node (high RB/high AKT/low Caspase) represents most major normal state, the largest red node (low RB/high AKT/low Caspase) represents most major cancer state, and the largest green node (high RB/low AKT/high Caspase) represents most major apoptosis state. Some key intermediate states along the kinetic paths also have been labeled with high or low expression level (from Ref. [84]). cancer attractor and the path from the cancer to the normal attractor are not reverses of each other. We can track the transition from the normal to the cancer state according to certain marker genes RB, MDM2, and CDK2. We see that the cancerization process proceeds (the green path from the normal to the cancer attractor in Figure 29) from MDM2 on (from 0 to 1), CDK2 on (from 0 to 1), RB off (from 1 to 0), and finally to cancer state. This leads to a possible mechanism for cancerization process. Initially, the on state of MDM2 represses the tumor repressor genes P53 and P21. This releases the genes CDK2 and CDK4 (CDK2 and CDK4 on) responsible for cell growth due to the inhibition of P21 to CDK. Then, RB is off due to the suppression of activated CDK2 and CDK4 on it, and cell gets into cancer state. This shows the importance of oncogene MDM2 to induce the tumor-genesis. For the reverse transition path from cancerization (from cancer to normal state), we can see that the cell proceeds (the magenta path from cancer to normal attractor in Figure 29) from RB on, off of CDK2 and CDK4, and finally to the off of MDM2. This shows that in the process of transition from cancer state to normal state, the cell may first switch on the key tumor repressor genes RB. As the time goes on, the growth genes CDK2 and CDK4 are gradually inactivated

86 84 J. Wang due to the repression of RB to them. Finally, the oncogene MDM2 gene is turned off. The cell then goes back to the normal state. In the experimental studies, inhibiting expression of TCTP (encoding translationally controlled tumor protein) was suggested as an important mechanism for tumor reversion. This is due to the activation of the expression of P53. Note that TCTP promotes MDM2-mediated degradation of P53 [ ]. This illustrates the role of oncogene MDM2 as an important drug target for tumor reversion. It also shows a verification for our theoretical predictions. It also illustrates the importance of restoring the function of tumor repressor gene RB as an anti-cancer tactics. The biological paths for cancerization and the apoptosis obtained from the landscape and flux theory of cancer can be tested and validated by experiments in the near future, and used to guide the design of new anti-cancer strategies Changes in landscape topography of cancer Based on the results of global sensitivity analysis, we identified certain key regulations and visualized the change of landscape when these regulation strengths are changed (Figure 30). In Figure 30, the vertical axis of every sub-figure denotes the negative probability ( P corresponding to potential energy U according to U = ln(p)). The four rows are separately associated with four specific regulations for illustration (two key activations and two key repressions). As the activations on AKT and VEGF increase, the landscape changes from tri-stability with dominant normal state to bi-stability (cancer and apoptosis coexist), and finally to mono-stability with a dominant cancer state. This shows the role of AKT and VEGF [ ] for inducing the cancer, consistent with the sensitivity analysis. When the repression on RB is increased, the landscape changes from bi-stability with dominant normal state to tri-stability, and finally to a mono-stability with dominant cancer state. This demonstrates the role of suppressing RB for inducing the cancer. As the repression on AKT is increased, the landscape changes from a cancer dominant bi-stability to tri-stability with dominant normal state, and finally to mono-stability with a dominant apoptosis state. This illustrates that repressing AKT can attenuate cancer through inducing the transition between cancer and normal state or inducing cell apoptosis [197]. The changes of landscape with regulations provide a possible explanation for the mechanisms of cancerization. At small AKT activation, the tri-stable landscape has a dominant normal state with deepest potential basin. It is difficult for the cell to switch from the normal attractor to the other two attractors (cancer and apoptosis). This shows that the cells perform normal functions and are stable against fluctuations. As the activation on AKT increases, the normal attractor becomes less stable, and the cancer attractor becomes more and more stable. This demonstrates the cancerization process from normal cells through the change of AKT activation strength. Finally, the cells show a landscape with only one dominant cancer attractor. This marks the completion of the transformation from normal cells to cancer cells. A funneled shape landscape guarantees the stability of cancer state. At this stage, it is difficult for the cells to escape from the cancer attractor. In the similar way, the landscape topography changes with respect to the changes in repression on AKT (the fourth row). This may provide a strategy for inducing the death of cancer cells, reflected by the landscape topography changes from a dominant cancer state to a dominant apoptosis state. These are some examples for regulation strength changes which can induce the landscape topography change of cancer gene network. In reality, there are many different types of combinations for changing regulation strengths in the network. This can lead to the change of landscape topography and further the change of network function. Due to the limitation of computational cost, here we only did single-factor sensitivity analysis. Ideally, a multi-factor sensitivity analysis is expected to find more realistic and interesting anti-cancer recipes.

87 Advances in Physics 85 Figure 30. The change of landscape when some key regulation strength (activation and repression parameters, M ji ) are changed. The vertical axis of every sub-figure represents P 1000 (P is probability and P is corresponding to potential energy U). From left column to right column is separately corresponding to s = 50%, s = 0, s = 50%, and s = 100% (here s represents the percentage of parameters changed). The four rows are separately corresponding to the change of four parameters. As labeled, the first row represents activation strength from VEGF to AKT, the second row represents the activation strength from AKT to VEGF, the third row represents the repression strength from CDK2 to RB, and the fourth row represents the repression strength from PTEN to AKT. The labels C, N, and A separately represent cancer attractor, normal attractor, and apoptosis attractor (from Ref. [84]). 8. Evolution 8.1. Introduction Evolution is the most fundamental idea in biology. According to Darwin, current biology is shaped by evolutionary history. Evolution is governed by the natural selection. The fittest to the environment survive. A quantitative realization of the Darwinian evolution was reached by Wright and Fisher [102, ]. Wright quantified the adaptive landscape for evolution where the evolution follows a gradient of the adaptive landscape and always searches for the better fitness. Fisher on the other hand developed his fundamental theory of natural selection that evolution is such that the fitness always increases and this increment is determined by the genetic variance of the fitness. Wright and Fisher s combined theory formed the foundation of evolutionary landscapes. However, Wright and Fisher s theory of evolution were often challenged

88 86 J. Wang by the following arguments for general evolution. It is known that in evolution, one often has so-called Red Queen effects where the evolution still continues even after reaching the optimal fitness [86]. For example, in evolution, one can often see limit cycle oscillatory cases where dynamics continues on the oscillation ring when the apparent fitness already reaches optimum. This is the most serious criticism of Wright and Fisher s quantitative theory of evolution. In other words, this clearly points out that Wright and Fisher s evolution theory only works under very restrictive conditions, for example, allele frequency-independent cases where there are no biotic interactions among species or within species. In general, almost all the realistic evolution processes do not satisfy the criteria that Wright and Fisher theory employ as assumptions. Clearly, a more general quantitative evolution theory is required and efforts have been made toward this direction [69, ]. As in our earlier examples, general evolution dynamics does not follow the gradient dynamics as Wright s theory stated. Instead, general evolution is determined by both a gradient of the potential landscape and a flux [69]. We can identify the intrinsic potential landscape as the Lyapunov function monotonically decreasing in the process to quantify the stability and function for the evolution but we must realize that this Lyapunov function is not the same as the fitness and thus defines a new fitness function due to its statistical probabilistic nature. So the evolution always searches for optimum on an intrinsic potential landscape, but not the original fitness. Furthermore, the presence of the additional curl flux will give an extra contribution to Fisher s fundamental theorem, so that the temporal change of the new fitness function with respect to time is not only equal to the genetic variance of the fitness, but also contains an extra contribution from the curl flux. The origin of the flux can come from allele frequency-dependent selection. In fact, this term can be negative or positive. Therefore, there is a chance that the new fitness is no longer changing any further. Instead, evolution still continues with non-zero genetic variance due to the presence of the flux. We can thus explain the famous Red Queen effects [86] with the limit cycle evolution. The evolution does tend to the higher fitness or lower potential. However, when it reaches an optimum such as on the limit cycle ring, the driving force for the continuation of the evolution dynamics on the ring comes not from the landscape gradient but from the curl flux arising from, for example, the biotic interactions among species or within species. This can create endless evolution. Evolution theory is based on three key ingredients: reproduction, mutation, and selection. The mathematical evolutionary theory is based on describing the changes in allele frequencies [69, ]. An allele is one of the multiple forms of a gene located at a specific position on a particular chromosome. A specific position on the chromosome is called a locus. Most of the higher eukaryotes are in diploid form with two copies of each gene. In sexual reproduction, only one allele is passed onto a single gamete. The two gametes unite together to restore the double complements of alleles. Here, we focus on a single diploid locus with multiple alleles in a randomly mating diploid population. We represent the n alleles at the given locus by A 1, A 2,..., A n and their frequencies by x 1, x 2,..., x n ( n i x i = 1), respectively. Due to the conservation of total allele frequencies, n i x i = 1, the n allele system is with n 1 degree of freedom. The state space is therefore n 1 dimension, {x k 1 k n 1} Evolutionary driving forces: selection, mutation, and genetic drift Selection Darwin proposed his evolution ideas as Survival of the fittest. It becomes a metaphor for describing natural selection. Natural selection can be expressed through the fitness defined as the average number of offsprings produced by individuals with a certain genotype. We can see how the natural selection changes the allele frequencies through the means of the fitness.

89 Advances in Physics 87 In a population with N new born diploid individuals, there are x i x j N individuals with genotype A i A j. For the next generation, the expected number of offsprings with genotype A i A j becomes w ij x i x j N. The expected total number of the population is then n ij w ijx i x j N, where w ij represents the fitness of genotype A i A j. Therefore, the proportion of A i allele for the next generation will be x i = n j x ix j w ij N/ n ij w ijx i x j N = x i w i / w, where, w i = n j=1 x jw ij and w = n i,j=1 x ix j w ij.we call w as the mean fitness of population. Therefore, the rate of change in allele frequencies under natural selection becomes [ ] dx i dt = Fi S = x i(w i w). (111) w It is worthwhile to notice that n i=1 FS i = Mutation The process of replicating a gene is not always accurate. Mutation represents any change in a new gene from the parental gene. Mutation can also result in the changes in allele frequency. The rate of change in allele frequencies under mutation becomes [ ] dx i dt = F M i = n x j m ji x i j=1 n m ij, (112) where the m ji is the rate of mutation from allele A j to A i. Due to conservation of total frequency of alleles, n i x i = 1, we notice n i=1 FM i = Genetic drift Sexual reproduction can be described as a binomial sampling process: a new generation with N individuals is formed as a result of sampling of 2N alleles from a large pool of gametes. Due to the sampling nature of the reproduction, the allele frequencies change in a random fashion. The change in allele frequencies from the random process is called genetic drift. This contributes to a stochastic evolution force. The corresponding mathematical approaches for describing genetic drift is the diffusion [ ]: P = D (GP), (113) t where G ij = x i (δ ij x j ) is from the sampling nature of the genetic drift and D = 1/(4N e ), N e is the effective population size. The state space is n 1 dimension. The operator, therefore, has n 1 components. The matrix G ij shows some distinct features. The first one is j=1 ( G) i = 1 nx i, (114) so that n i=1 ( G) i = 0. The second one is that its inverse matrix is known to have the feature [202]: where F n = n 1 i=1 F i. (G 1 F) i = F i x i F n x n, (115)

90 88 J. Wang 8.3. Mean fitness as the adaptive landscape for the frequency-independent selection systems When the fitness of every genotype is independent of the allele frequencies, the corresponding natural selection is called frequency-independent selection. From Equation (111), the frequencyindependent selection can be written in the form as F S = ( 1 2 )G ln w, and thus (G 1 F S ) = 0. Moreover, [G 1 ( G)] = [ ( n i=1 ln x i)] = 0. We can see J ss = 0forthe population system with both natural selection and genetic drift. In this situation, F S D G + DG U = ( 1 2 )G ln w D G + DG U = 0. So, φ 0 = (DU) D 0 = ( 1 ) ln w. 2 Therefore, we see φ 0 = ( ) 1 ln w (116) 2 for the frequency-independent selection population [69]. Notice we ignore a constant of integration here. From the previous discussion, we can see d w/dt = F w = 2 wf φ 0 = 2 w φ 0 G φ 0 0. This indicates that the mean fitness as a Lyapunov function can be used to quantify where the evolutionary optima are. This is the picture for Wright s adaptive fitness landscape with a gradient flow. However, Wright s adaptive fitness landscape description will break down for the more general case, that is, under frequency-dependent selection force The adaptive landscape for general evolutionary dynamics As discussed, Wright s fitness landscape will be inadequate for the general evolution dynamics such as frequency-dependent selection. Then, the challenge becomes whether there is an adaptive landscape for evolution and if so what it is? The potential and flux landscape theory meets the challenge to provide a solution [69]. We will start with studying the stochastic evolution dynamics through the probabilistic description of Fokker Planck diffusion equation including the selection, mutation, and genetic drift, P t = [(F S + F M )P D (GP)]. (117) In the steady state, the corresponding steady-state probability flux is J ss = (F S + F M )P ss D (GP ss ) = P ss [(F S + F M D G) + DG U]. When the steady-state flux is zero, the system is in detailed balance. Then, the judgment of the detailed balance becomes the judgment of whether the flux is zero. According to our previous discussion, we can check the [G 1 (F S + F M D G)] to obtain the information. Since (G 1 F M ) = 0 and [G 1 ( G)] = 0, mutation and genetic drift do not violate the detailed balance. Therefore, whether or not the detailed balance is preserved is determined by whether the natural selection via (G 1 F S ) is equal to zero. The non-equilibrium intrinsic potential φ 0 of this system satisfies (F S + F M ) φ 0 + φ 0 G φ 0 = 0. (118) Because of the deterministic equation dx/dt = F S + F M, φ 0 /dt = φ 0 G φ 0 0. Therefore, φ 0 being monotonically going down with time is a Lyapunov function always searching for optimum irrespective to whether the evolution system is in detailed balance or not. Therefore, φ 0 here defines the true adaptive landscape for evolution.

91 Advances in Physics Potential and flux landscape of a group-help model with frequency-dependent selection for evolution For the purpose of studying the general evolution dynamics, we consider the case of frequencydependent selection where the fitness is dependent on the allele frequency describing the social interactions. We investigate a group-help model (GHM) to explore the underlying evolution dynamics that includes biotic interactions among individuals in addition to the selection, mutation and genetic drift. Frequency-dependent selection emerges when the fitness is dependent on the allele frequency. A social system is a typical one with frequency-dependent selection. If the genotypes differ in their social behavior, the fitness of a genotype may depend on the population composition. For selection, we investigate a GHM to study the effect of interactions among individuals on the evolution. In the GHM, it is assumed that a single diploid locus has three alleles A 1, A 2, A 3 with frequencies x 1, x 2, x 3, respectively. The fitness of genotypes are taken as follows [69]: w 11 = α + 2βx 1 x 2, w 12 = α + x 1 x 1, w 13 = α + x 3 x 3, w 21 = α + x 1 x 1, w 22 = α + 2βx 2 x 3, w 23 = α + x 2 x 2, w 31 = α + x 3 x 3, w 32 = α + x 2 x 2, w 33 = α + 2βx 3 x 1. (119) Due to conservation of total allele frequency, we have x 3 = 1 x 1 x 2. The fitness of each genotype has two components influenced by the external physical environment described by a strength α and biotic interactions characterized by the strength β. The scheme of the GHM for evolution with frequency-dependent selection is illustrated in Figure 31. In this random mating population, three groups are formed and within each group two genotype members help each other. This is from the selection that shows that one s relatives carrying many of the same genes help it to survive and reproduce. The fitness of each Figure 31. Scheme for GHM (from Ref. [69]).

92 90 J. Wang genotype has two components influenced by external physical environment characterized by the strength parameter α and biotic interactions characterized by strength parameter β. The heterozygote helps the homozygote with the assistance factor β(β 0), while the homozygote helps the heterozygote with the constant assistance factor 1. We can obtain a frequency-dependent selection force from inserting these fitness (Equation (119)) into Equation (111). This selection has (G 1 F S ) 0. Therefore, it will induce a non-zero steady-state probability flux, J 0 which breaks detailed balance The underlying potential flux landscape for general evolutionary dynamics Wright s adaptive fitness landscape is of a gradient nature. It is invalid for the general case. An example of this inadequacy is illustrated by the frequency-dependent selection population system. We can easily prove it in a simple case. For instance, limit cycle dynamics can emerge under the frequency-dependent selection. The fitness is not monotonous along the limit cycle because of the gradient nature of Wright s landscape of being always searching for the optimum. If the fitness is a constant, then the evolution according to Wright will stop. On the other hand, if the fitness is not uniform on the limit cycle, another issue arises. When finishing the cycle, the ending point is supposed to have higher fitness from the gradient nature of Wright s landscape always searching for optimum. However, the ending point has the same value of potential as the initial point due to the full cycle back to itself. This creates a paradox. The potential and flux landscape provides a way of obtaining the true adaptive landscape for evolution and resolving this paradox. In the landscape and flux theory, there are two driving forces for the dynamics of the non-equilibrium systems. One is the potential landscape related to the steady-state probability distribution and the other is related to the steady-state probability flux. For the limit cycle, both forces are in action. While the potential landscape attracts the system to the oscillation ring, it is the flux that drives the coherent oscillation on the ring. Thus, one can have a constant intrinsic landscape on the oscillation ring with the same fitness, but the dynamics still goes on through the curl flux circulating around the limit cycle. To illustrate this, we will construct the Lyapunov function φ 0 to quantify the true intrinsic adaptive landscape for GHM. We have also considered the population landscape U for the finite population with fluctuations. We will now study the dynamics of the GHM. Four phases emerge under different parameter strengths: the mono-stability phase, β = 0.03; a limit cycle phase, β = 0.09; coexistence of a limit cycle and tri-stability phase, β = 0.11; the tri-stability phase, β = The fluctuation strength is taken as D = for quantifying the population potential U = ln P ss. To quantify the intrinsic and population potential landscape as well as the corresponding steady-state curl flux for the GHM of evolution dynamics, we need to solve the corresponding Fokker Planck equation (Equation (117)) in steady state for U = ln P ss, where P ss is the steady-state probability, and HJ equation (Equation (118)) for φ 0. We apply the zero flux boundary condition, n J = n [(F S + F M )P D (GP)] = 0, for the Fokker Planck equation (Equation (117)). n represents the unit normal vector of boundary. Such condition corresponds to the conservation of total probability. We obtain the numerical solution of the steady-state Fokker Planck equation P ss and the associated population potential U [69]. Correspondingly, the boundary condition of HJ equation (Equation (118)) is taken in the form n (F S + F M + G φ 0 ) = 0. This is the zero-fluctuation limit of the zero flux boundary condition. The HJ equation is difficult to solve analytically. It is also challenging to obtain the numerical solution. The notion of viscosity solution was introduced to help solving the HJ equation. According to this, a numerical method level-set method was developed. We can apply Mitchell s level-set toolbox to solve the HJ equation for intrinsic potential φ 0 [102].

93 Advances in Physics 91 (a) (b) (c) (d) (e) (f) (g) (h) Figure 32. Landscapes of different phases. Top row: the population potential landscape U ((a) β = 0.03, (b) β = 0.09, (c) β = 0.11, and (d) β = 0.17). In (b), the lower left corner sub-picture shows the enlarged (U is enlarged by five times) valley bottom of the population potential landscape. Purple arrows represent the flux (J ss /P ss ), while the black arrows represent the negative gradient of population potential landscape ( U). Bottom row: the adaptive landscape defined by intrinsic potential φ 0 ((e) β = 0.03, (f) β = 0.09, (g) β = 0.11, and (h) β = 0.17). In (f), the lower left corner sub-picture shows the enlarged (φ 0 is enlarged by six times) valley bottom of the intrinsic potential landscape. Purple arrows represent the intrinsic flux (V = (J ss /P ss ) D 0 ), while the black arrows represent the negative gradient of intrinsic potential ( φ 0 ) (from Ref. [69]). Figure 32 shows the population potential U(the top row) and intrinsic potential φ 0 (the bottom row), respectively. The arrows on the landscape in Figure 32 show schematically the two components of driving force: negative gradient of the landscape ( U for top row and φ 0 for bottom row)(black arrows) and the steady-state probability curl flux (J ss /P ss for the top row and intrinsic flux velocity V = (J ss /P ss ) D 0 for the bottom row)(purple arrows). The arrows at the bottom are the projections of those arrows. The flux (purple arrows) and negative gradient of U(black arrows) are almost perpendicular to each other at the bottom plane of Figure 32(a) (d). We can see the flux J ss /P ss (purple arrows) and negative gradient of φ 0 (black arrows) are perpendicular to each other at the bottom plane of Figure 32(e) ( h). The non-equilibrium intrinsic potential landscape φ 0 does not change with the fluctuation strength D. The effect of D on probability is through the ln P ss = U = φ 0 /D under weak fluctuations. Figure 32(a) and 32(e) show both the population landscape U and intrinsic landscape φ 0 for the mono-stable phase (β = 0.03). While the negative gradient of potential (black arrows) attracts the system into the minimum of the funnel (basin of attractions), around the basin the system is

94 92 J. Wang also driven by the intrinsic flux velocity (purple arrows). Without the former, the system will not be attracted to the bottom of the funneled attractor basin. Without the latter, the system will directly go down to the low potential basin. The effect of the intrinsic flux becomes more apparent when the system approaches to the bottom of the basin in a spiral fashion. It is interesting to see that the intrinsic potential φ 0 gives the essential topography (funnel) of the landscape for global stability. Figure 32(b) and 32(f) show both the population landscape U and intrinsic landscape φ 0 for limit cycle phase (β = 0.09). Under finite fluctuations, the topography of the population landscape shows a Mexican hat shape with a closed ring valley for oscillations. The forces from negative gradient of the potential are almost negligible along the closed ring. But they are more significant inside and outside ring. Therefore, away from the closed ring, the evolution is attracted by the negative landscape gradient toward the closed ring. On the Mexican hat closed ring valley, the flux becomes dominant and provides the driving force for coherent oscillation. The direction of the flux near the close ring is parallel to the oscillation path. We can also see that the underlying intrinsic landscape φ 0 has a Mexican hat like topography with constant φ 0 values on the oscillation ring valley. Notice that population landscape U is not constant along the ring since U is a direct reflection of steady-state probability U = ln P ss. The population landscape U is not a Lyapunov function in general but captures more details than the intrinsic landscape φ 0.This is particularly true for the inhomogeneity of the steady probability due to the inhomogeneous speed on the ring. The non-equilibrium intrinsic potential φ 0 characterizes the essential global topography of the oscillation landscape. Figure 32(c) and 32(g) show both the population landscape U and the intrinsic landscape φ 0 for the coexistence of a limit cycle oscillation and tri-stability phase (β = 0.11). The topography of the landscape shows three basins around a Mexican hat ring valley. The steady-state probability flux on the closed ring valley landscape of the limit cycle appears to be more significant than that of the negative gradient of the potential landscape. It drives the coherent oscillations. But the flux is less significant compared with gradient force around the three stable basins of attraction. The direction of the flux on the limit cycle ring is parallel to the oscillation path. The flux curls around near the bottom of the three stable basins of attraction. The forces from negative gradient of the potential landscape are almost negligible on the closed ring. They are more significant away from the oscillation ring and the bottom of the three basin valleys. Therefore, away from the closed ring and the bottom of the three basins of attraction, the evolution dynamics is attracted by the landscape toward them. We see again that φ 0 is constant on the limit cycle oscillation ring. Figure 32(d) and 32(h) show both the population landscape U and the intrinsic landscape φ 0 for the tri-stability phase (β = 0.17). Three basins of attraction emerge with equal depths due to the symmetry. The direction of the flux around the linking region between the two attractors is parallel to the link. The directions of the flux are curling around at the bottom of three basins of attractions. The forces from negative gradient of the potential landscape are less significant along the linking regions between the two attractors (away from the attractors) and more significant near the basins of attractions. Therefore, near the basins of attractions, evolution is attracted by the landscape toward the basins Generalizing Fisher s FTNS The Wright adaptive landscape theory of never decreasing fitness and Fisher s FTNS on the adaptive rate resulting from natural selection are in fact intimately related. The validity of Wight s adaptive fitness landscape is guaranteed by Fisher s FTNS [204]. The mathematical form in

95 constant environments of FTNS is given as follows [205]: Advances in Physics 93 d w dt = V A(w) w, (120) where V A (w) = 2 n i=1 x i(w i w) 2 is the additive genetic variance. Therefore, the change of the mean fitness is from the effects of selection favoring the most fitted individuals among variations. Because the variance is always larger or equal to zero, Fisher s fundamental theorem implies that the mean fitness never decreases for a constant environment. However, the mean fitness is not a Lyapunov function in general. Fisher s FTNS and Wright s adaptive fitness landscape are only valid in the frequency-independent case. What is the relationship between the adaptive rate and genetic variations in the general evolutionary dynamics then? We can generalize Fisher s FTNS with potential flux landscape theory. We will study the general case for evolution dynamics and quantify the adaptive rate below. From the landscape and flux theory, for any positive definite matrix D, the general selection force can be decomposed to the intrinsic potential φ 0 (D) and the corresponding intrinsic flux velocity V(D) [69]: F = D φ 0 + V. (121) Since V φ 0 = 0, we get F D 1 F = φ 0 D φ 0 + V D 1 V. Thus, the adaptive rate is given by dφ 0 = φ 0 D φ 0 dt = F D 1 F + V D 1 V. (122) In evolution dynamics, the diffusion matrix D has a special biological meaning. Namely it describes the sampling nature of the random mating. Therefore, we are interested in the φ 0 (D) and dφ 0 (D)/dt. We will see that the dφ 0 (D)/dt is related to the genetic variance. From this, we can generalize Fisher s FTNS. We can check, V A (w (i) ) ( w (i) ) 2 = 2F (i) (D 1 ) (i) F (i), (123) where V A (w (i) ) = 2 n i k (w(i) k w (i) ) 2. We take the sum with respect to i to get F D 1 F = ( 1 2 ) N i=1 V A(w (i) )/( w (i) ) 2 [69]. Finally, we have k=1 X (i) dφ 0 (D) dt = 1 V A (w) + V(D) D 1 V(D). (124) 2 w 2 The mono-species population system under frequency-independent selection is in equilibrium where the intrinsic flux velocity V(D) = 0, and the intrinsic non-equilibrium potential φ 0 = ( 1 ) ln w. So for the case of frequency-independent mono-species population system, 2 Equation (124) is reduced to Fisher s FTNS, d w/dt = V A (w)/ w. We call Equation (124) a generalized FTNS. It uncovers the underlying connection of the adaptive rate. One component of this connection is the genetic variance as originally proposed by Fisher, which is related to the non-equilibrium intrinsic potential φ 0. The other component of this connection is from the corresponding intrinsic flux velocity V, missing in Fisher s original FTNS. As we can see clearly in the general evolution dynamics, the adaptation rate is not only determined by the genetic variance, but also by the curl flux resulting from the complex biotic interactions, which breaks the detailed balance of the system [69].

96 94 J. Wang 8.8. The Red Queen hypothesis explained by a generalized FTNS We now investigate what the generalized FTNS implies. We can gain insights from the case when dφ 0 (D)/dt = 0, when the overall population system reaches its optima. For frequencyindependent selection, the system is in equilibrium with the detailed balance, that is, V(D) = 0, due to (D 1 F) = [( 1 2 ) ( N i=1 ln w(i) )] = 0. This leads the genetic variance for every species to be V A (w (i) ) = 0[69]. Thus, the natural selection cannot change the allele frequency. However, for general evolution dynamics, the system is often not in equilibrium. The detailed balance is broken and the intrinsic flux is not equal to zero, that is, for the evolution system with frequency-dependent selection. The non-zero intrinsic flux velocity, V(D) 0, gives rise to N i=1 V A(w (i) )/( w (i) ) 2 = 2V(D) D 1 V(D) 0. The non-zero genetic variance implies that the additional natural selection can have effects on some species to change the allele frequencies even if the system reaches its overall population optima. We explored further on which species has the non-zero genetic variance. Since D is a block diagonal matrix, we can decompose the selection force acted on the ith species from Equation (121) in the form of F (i) = D (i) ( φ 0 ) (i) + V (i). Inserting this into Equation (123) and using the orthogonality feature of V φ 0 = 0, we obtain V A (w (i) )/( w (i) ) 2 = 2[( φ 0 ) (i) D (i) ( φ 0 ) (i) + V (i) (D 1 ) (i) V (i) ]. Moreover, when dφ 0 (D)/dt = 0, from Equation (122), we obtain φ 0 = 0[69]. Thus, we get V A (w (i) ) = 2( w (i) ) 2 [V (i) (D) (D 1 ) (i) V (i) (D)] 0. (125) If the intrinsic flux velocity is from the ith subspace, the non-zero V (i) gives rise to the genetic variance for ith species. Consequently, the ith species will continue to evolve even when the overall population reaches its optima. Let us explore the origin of the non-zero intrinsic flux velocity. We can decompose the selection force into two components, one from external environment and the other from internal interactions among individuals. The former is the frequency-independent selection. It provides a gradient force as given in Wright s picture. The non-zero intrinsic flux velocity is originated from the internal interactions from the same species or between different species. Thus, biotic interactions can give rise to endless evolution of some species. This effect that originates from the interactions between different species was mentioned by Van Valen as the Red Queen hypothesis [86]. According to the Red Queen hypothesis, the biotic interactions between different species can lead to endless evolution for some species even when the underlying physical environments are unchanged. The coevolving systems often enter into a limit cycle or chaos phase, leading to Red Queen dynamics. Potential and flux landscape theory provides a theoretical foundation and quantitative explanation for this effect [69]. We can illustrate the Red Queen hypothesis in action using an example. In a parasite host system, the mutual interactions can sustain a genetic variance which maintains genotypic diversification in host populations through sexual reproduction. The host species benefit from the genotypic diversifications in resisting the parasites and thus the sexual reproduction can be persisted. Conventional evolutionary theory neglects the effects of biotic interactions on the evolutionary process. Consequently, the evolutionary process is to adapt to a fixed landscape (when the external physical environment is unchanged). More recently, evolutionary game theory has been proposed and developed to study the coevolving systems [206]. According to evolutionary game theory, the landscape of a biotic system is continually changing by coevolving with other biotic systems [208]. From potential and flux landscape theory of evolution, we can quantify the Lyapunov function for the system including all the biotic interactions. We can find the non-equilibrium intrinsic potential to quantify the underlying adaptive landscape. The intrinsic

97 Advances in Physics 95 potential landscape provides the information on where the evolution optima are. The intrinsic flux can drive some species into an endless cyclic evolution even when the optima is reached. 9. Ecology Ecology is the subject that is concerned with the stability and dynamic behavior of interacting species for specific time window. Potential and flux landscape theory can be applied to ecosystems [81]. Again the driving forces of ecological dynamics are determined by both a gradient of the potential landscape and a curl probability flux from the in or out flow of the energy, materials or information to the ecosystems. The underlying intrinsic potential landscape provides a global Lyapunov function giving a quantitative measure for the global stability of the ecosystems. We can study several typical and important ecological systems that involve predation, competition, and mutualism. Single attractor, multiple attractors, and limit cycle attractors emerge from these studies. Both the landscape gradient and curl flux determine the dynamics of these ecosystems. Landscape flux theory provides a way to explore the global stability, function, and dynamics of ecosystems [81] Introduction Ecosystems are the mutually interacting systems with the exchange of energy, material, information from the environments. The structure and the function of the ecosystems are determined by both cooperation and competition [211,212]. Ecosystems are capable of regulating themselves to maintain a certain stability. Global stability is one of the most fundamental aspects of ecological systems. The stability issue is directly relevant to the existence of every species. The stability is influenced by many sources, such as the structures of the interactions and the nature of the environment. Studies of the stability of ecosystems are important for uncovering the underly laws and mechanisms governing species and populations [211,212]. Ecosystems are complex and often involve many forms of interactions among their components. The underlying interactions are usually nonlinear. The nonlinear interactions can lead to complex dynamics. These systems can be mathematically described by a set of nonlinear differential equations [213,214]. There have been many studies on the stability of ecosystems [ ]. Most of these studies have been focused on the local linear stability analysis. The global stability of the ecological systems is still challenging. Moreover, the connection between the global characterization of the ecological systems and the dynamics of the components is still not clear. Past researchers have explored the dynamical system with the approach of Lyapunov function to investigate the global stability. The analytical Lyapunov function for predation model was first suggested by Volterra in 1931 [214]. Since then, significant efforts have been devoted to find the analytical Lyapunov function [ ,220,221] for a variety of ecological systems. However, it is still a great challenge to find the Lyapunov function of the more general and realistic ecological models even though a few highly simplified ones have been worked out [215,218]. Up to now, there is no general approach and recipe for finding the Lyapunov function. One has to work on a case-by-case basis. Furthermore, there is not even a guarantee that a Lyapunov function exists for a more complex system. Importantly, the Lyapunov function and the stability of a important class of ecosystems described by the predation model with a solution of a limit cycle have hardly been explored. Here we will provide a general approach to explore the Lyapunov function and therefore the global stability of ecological systems. In the earlier studies, people have focused on deterministic models. However, both external and intrinsic fluctuations of ecosystems are present and unavoidable. Environmental fluctuations can influence the global ecological behaviors. The intrinsic fluctuations are due to the finite size

98 96 J. Wang (a) (b) (c) Figure 33. (a) Predation model, (b) competition model, and (c) mutualism model (from Ref. [81]). of the ecosystem. It is widely believed that the analysis of global stability is important for a full understanding of the robustness of ecological systems under fluctuations [174, ,221,222]. As discussed, it is still challenging to explore global stability with deterministic dynamics. A probabilistic (P) description is necessary due to the presence of fluctuations in the real systems. The probabilistic description has the advantage of quantifying the weights of the whole population state space and therefore is global. The potential landscape U is linked with the probability P by U ln P and can be used to address global stability and function of ecosystems. Here we develop a potential and flux landscape theory to quantify the global stability and dynamics for ecosystems [13,16,26,57,69,71,81]. Again the underlying intrinsic potential landscape is a global Lyapunov function for ecosystem. The topography of the landscape therefore provides a quantitative measure for the global stability of the ecosystems. The dynamics of the ecosystems is determined by both the gradient of the potential landscape and curl probability flux from the environmental exchange of energy, materials, and information, breaking the detailed balance. The landscape and flux theory is applied to three important ecosystems: predation, competition, and mutualism. Lotka-Volterra model of the interactions between two species is the most famous ecological model proposed by Lotka [213] and Volterra [214], respectively. Over the years, this model has attracted significant attention for exploring the underlying dynamical process of the ecology. In ecosystems, the relationship between species can be grouped into two categories: a negative antagonism interaction ( ) and a positive mutualism interaction (+). We show the different models in Figure 33. Predation shows the activation repression relationship (+/ ) where one species S A is disfavored, while the other species S B benefits in Figure 33(a); competition shows a mutual repression relationship ( / ) where each species S A or S B is influenced negatively by the other one in Figure 33(b); mutualism shows a mutual activation relationship (+/+) where both species S A and S B benefit from interactions of the other in Figure 33(c) [212,216,221]. For predation, predators sustain their lives by the consumption of prey. Without the presence of prey, predators are not able to survive. On the other hand, the prey can sustain their lives and grow without predators. The presence of the prey controls the growth of the predators. The predator prey (predation) system emerges from such interactions [ ]. The predator prey system can have one stable state or stable limit cycle. Competition between species often happens when they rely on the same resources. Competition can promote the ecological characteristics such as species differentiation and generate certain biological community structures. The system can have multi-stable states. Mutualism means mutual activation benefiting to each individuals. The mutualism system can also have multi-stable states. These relations aforementioned show the complexity of biological communities, their stable structures, and the ecological balance [212,216,221]. These models are also important for population biology because of the applications to the real biological world The ecological dynamical models The ecosystems can be described by a set of nonlinear ordinary differential equations with species interactions. We will add some restrictions on the models to enable them to be more reasonable

99 Advances in Physics 97 and closer to the real world [212,223], such as avoidance of exponential growth and existence of lower critical bound for each species. (1) Predation model The general Holling type II responses for the prey to account the nonlinearity in interactions can be added in a predation model for two species predator prey model [212]: dn 1 dt dn 2 dt = N 1 (1 N 1 ) an 1N 2 N 1 + d = F 1(N 1, N 2 ), ( = bn 2 1 N ) 2 = F 2 (N 1, N 2 ), (126) N 1 where N 1 represents the normalized population of prey, while N 2 represents the normalized population of predator. The parameter a represents the relative death rate or the interaction strength for the prey. The parameter b represents the ratio of the linear birth rate of the predator to that of the prey. The parameter d represents the relative saturation rate of the prey. The system has two saddle points: one is at (0, 0) representing that none of the species exists and the second one at (1, 0) representing the prey at their carrying capacity in the absence of predators [212]. The second point is stable along the N 1 population axis and unstable along the N 2 population axis. There is also a critical point which is the unstable center of the limit cycle or the stable point under different parameter ranges. The system has a stable limit cycle oscillation when the parameters are set to a = 1.5, b = 0.1, d = 0.2. (2) Competition model A realistic competitive model should have a lower critical bound, which means the populations would perish once the size of the population is below this threshold. The model is shown as follows [223]: dn 1 = N 1 (N 1 L 1 )(1 N 1 ) a 1 N 1 N 2 = F 1 (N 1, N 2 ), dt dn 2 = αn 2 ((N 2 L 2 )(1 N 2 ) a 2 N 1 ) = F 2 (N 1, N 2 ), (127) dt where N 1 and N 2 represent the normalized populations of the two competitive species S A and S B. L 1, L 2 represent the lower critical bounds for species S A, S B, respectively. The ranges of L 1, L 2 are from 0 to 1. a 1, a 2 are the competitive capability for species S A, S B, respectively. α is the relative rate of increase for species S B [223]. We have performed phase analysis for the system. Both of the two populations can be at an extinct state (0, 0) (marked as O) of the system. This is because both groups have a lower critical density. Under weak competitors for other species, the states: (1,0) (marked as A, which means the species S A exists alone) and (0,1) (marked as B, which means the species S B exists alone) are locally stable. Besides the above three states, when the values of a 1 and a 2 meet certain conditions, the system can have another local stable state which corresponds to the coexistence of the two species (marked as C). Here the parameters are set to a 1 = a 2 = 0.1, L 1 = L 2 = 0.3, α = 1.0 since the system has these four states. (3) Mutualism model We consider the case of two mutualism species both having a lower critical bound. This realistic mutualism model can be described as [223]: dn 1 dt = N 1 (N 1 L 1 )(1 N 1 ) + a 1 N 1 N 2 = F 1 (N 1, N 2 ),

100 98 J. Wang dn 2 dt = αn 2 ((N 2 L 2 )(1 N 2 ) + a 2 N 1 ) = F 2 (N 1, N 2 ), (128) where N 1 and N 2 represent the normalized populations of the two mutualism species S A and S B. L 1, L 2 represent lower critical points for species S A and S B, respectively. The ranges of L 1, L 2 are from 0 to 1. a 1, a 2 are the mutualism capability for species S A, S B, respectively. α is the relative rate of increase for species S B [223]. We have performed the phase analysis of this system. Both of the two populations can be at an extinct state giving the trivial solution (0, 0) (marked as O) of the system. This is because the two groups have a lower critical density. Under no mutual helper for each species, the states: (1,0) (marked as A, which means the species S A exists alone) and (0,1) (marked as B, which means the species S B exists alone) are locally stable. Besides the above three phases, the system has another local stable state which corresponds to the coexistence of the two species (marked as C). Here, the parameters are set to a 1 = a 2 = 0.1, L 1 = L 2 = 0.5, α = 1.0 since the system has these four states The potential landscapes and fluxes of ecosystems: predation, competition, and mutualism The population potential landscape U(the top row) and intrinsic potential landscape φ 0 (the bottom row) for predation, competition, and mutualism models are shown, respectively, in Figure 34. The negative gradient of the population potential landscape U at the top row and the intrinsic potential landscape φ 0 at the bottom row are represented by the black arrows, while the curl steady-state probability flux J ss /P ss at the top row and the intrinsic flux velocity at the bottom row are represented by the purple arrows. The arrows at the bottom of each sub-figures are the projection of the associated arrows. The curl flux with purple arrows are nearly orthogonal to the negative gradient of U with black arrows shown at the bottom plane of Figure 34(a) (c). The flux velocity with purple arrows are orthogonal to the negative gradient of φ 0 with black arrows shown at the bottom plane of Figure 34(d) (f). Figure 34(a) and 34(d) show the population potential landscape U and intrinsic potential landscape φ 0 for the predation model when the parameters are set to a = 1.5, b = 0.1, d = 0.2, D = We can see when the fluctuations characterized by the diffusion coefficient are small that the underlying potential landscape has a distinct Mexican hat-shaped closed irregular and inhomogeneous closed ring valley shown in Figure 34(a). We can clearly see that the population potential landscape is not uniformly distributed along the limit cycle closed ring. The intrinsic potential landscape φ 0 has a homogeneous closed ring valley along deterministic oscillation trajectories with a constant value of φ 0 shown in Figure 34(d). The intrinsic potential landscape φ 0 as a Lyapunov function can quantify the global topography of the oscillation landscape of predation model. The figure shows that the potential is lower along the oscillation ring. The potential landscape is higher with a center island inside the oscillation ring and mountain outside the oscillation ring. The system is attracted to the closed oscillation ring by the force generated from the landscape s gradient-potential U or the φ 0. The other driving force for the system from the curl flux keeps the continuous periodical oscillation dynamics. Both landscape and flux are required to quantify the dynamics of the non-equilibrium predation ecosystems. The oscillation study here shows that when the number of predators increases, more prey will be consumed. Then due to the shortage of food, the number of the predator will be reduced. The reduction of the predators leads to the growth of the prey, then the number of predators increases again from the rich prey. This is the origin of the limit cycle predator prey dynamics. Figure 34(b) and 34(e) show the population potential landscape U and intrinsic potential landscape φ 0 for competitive model when the parameters are set to a 1 = a 2 = 0.1, L 1 = L 2 =

101 (a) (b) (c) (d) (e) (f) Figure 34. Top row: the population potential landscape U ((a) predation model, (b) competition model, and (c) mutualism model). Purple arrows represent the flux velocity (J ss /P ss ), while the black arrows represent the negative gradient of population potential ( U). Bottom row: the potential intrinsic energy landscape φ 0 ((d) predation model, (e) competition model, and (f) mutualism model). Purple arrows represent the intrinsic flux velocity (V = (J ss /P ss ) D 0 ), while the black arrows represent the negative gradient of intrinsic potential ( φ 0 )) (from Ref. [81]). Advances in Physics 99

102 100 J. Wang 0.3, α = 1.0, D = We can see both the underlying population potential landscape and intrinsic potential landscape have four distinct basins around four locally stable states. These four stable states are the survival alone state A of species S A, the survival alone state B of species S B,a coexisting state C, and a mutually extinct state O. These figures show that the potential landscape is relatively higher (and probability is relatively lower) on the extinct state (the state O) of these two species due to the small lower critical points L 1, L 2 for species. The potential landscapes of the survival alone states A and B are lower (more stable) in potential than that of the coexisting state C. This shows that the coexisting state is less stable than the species survival alone states because they have competitive relations with each other. The forces from negative gradient of the potential landscape are more significant away from the attractors and less significant near the basins. Therefore, the system is attracted by the gradient of the landscape toward the four basins. The directions of the flux are curling around among the basins. Figure 34(c) and 34(f) show the population potential landscape U and intrinsic potential landscape φ 0 for the mutualism model when the parameters are set to a 1 = a 2 = 0.1, L 1 = L 2 = 0.5, α = 1.0, D = Both the underlying population potential landscape and intrinsic potential landscape have distinct four basins. The basins are located around the four stable states. These figures show that the potential landscape is the highest (and probability is lower) on the extinct state O of these two species. The potential landscape of coexisting state C is the lowest than those of species survival alone states A and B, and the extinct state O. This shows that the coexisting state is more stable than the species alone state for the two species with mutualism relationship from each other. The directions of the flux are curling around among the four basins. The system is also attracted by the gradient of landscape toward the four basins Discussion It is interesting to compare landscape and flux theory of ecology with landscape ecology models. The landscape ecology models concentrate on spatial heterogeneity with space probabilistic methods [224,225]. These methods as well as the present theory all focus on the dynamical evolution in probability. However, the present theory concentrates on the probability landscape and flux in the population space rather than in the geographical space as in the landscape ecology models. 10. Landscape and fluxes of neural networks Introduction Understanding brain function is a grand goal for biology. The brain is a complex and dynamical system [72, ]. The individual neurons connect with each other through synapses to form the neural networks. The neural networks of the brain generate complex patterns of activity related to biological functions, such as learning, long-term associative memory, working memory, olfaction, decision-making and thinking [ ]. Many models have been suggested for understanding how neural networks function. The Hodgkin Huxley model provides a description of a single neuron [229]. However, the brain functions are realized by the neural networks rather than individual neurons. Hopfield developed a model [230,231] to explore the global features of neural networks. Hopfield showed that, for symmetric neural connections, a global energy landscape can be constructed that decreases with time. As shown in Figure 35, the neural network from the initial starting point follows a gradient path down to the nearest basin of attractor of the underlying energy landscape. The basins of attraction store the memory formed by learning from specific enhanced wiring

103 Advances in Physics 101 Figure 35. The schematic diagram of the original computational energy function landscape of Hopfield neural network (from Ref. [72]). patterns. The memory retrieval process can then be from a cue (incomplete initial information) to the corresponding memory (final complete information). This gives a clear picture of how neural networks store their memory and retrieve the functions. However, in real neural networks, the neural connections are mostly asymmetric. The original Hopfield analysis does not hold with asymmetrical connections. Therefore, there is no easy way of finding out the underlying energy function. Without an energy function, the global stability and function of the neural networks are hard to explore. In this study, we will explore the global behavior of neural networks for the general case of both symmetric and asymmetrical synaptic connections. Here, we will apply potential and flux landscape theory for neural networks. The driving force of the neural network dynamics is determined by both the gradient of the potential and a curl probability flux. The curl flux can generate coherent oscillations, which are not possible with a pure gradient force. The potential landscape still functions as a Lyapunov function crucial for quantifying the global stability even with asymmetrical connections [13,17,60,72] of the neural networks. The original Hopfield model shows a good associative memory features of the underlying neural network. The network state always goes down to certain fixed point attractors with the stored memories. However, evidences accumulate that oscillations also play important roles in cognitive processes [72,233,235], that is, the theta rhythm enhanced in neocortex during working memory [236] and an enhanced gamma rhythm related to attention [237,238]. However, the original Hopfield model does not hold for asymmetrical connections and cannot be used to describe this oscillation behavior. Landscape and flux theory provides a way to explore the global features of neural networks that include the oscillations [13,14,17,72]. First, we will study a Hopfield associative memory network including 20 model neurons. The synaptic strengths of neural connections are quantitatively represented by the connection strength parameters T ij [230,231]. We uncovered the probabilistic potential landscape and the corresponding Lyapunov function for this neural circuit, not only for symmetric connections of the original Hopfield model where the dynamics is dictated by the gradient of the potential, but also for asymmetric connections where the original Hopfield analysis does not hold. We can

104 102 J. Wang explore the effects of the connections having different degrees of asymmetry on the behaviors of the circuits and the robustness of the neural networks in terms of landscape topography through barrier heights. Neural networks with strong asymmetric connections can generate the limit cycle oscillations. However, the oscillations cannot occur for the symmetric connections due to the gradient feature of the dynamics. The corresponding potential landscape for limit cycle oscillations shows a Mexican hat closed-ring shape topology. The global stability of oscillation can be quantified through the barrier height of the center island of the hat. The dynamics of the neural networks is determined by both the gradient of the landscape and the probability flux. While the gradient force attracts the network down to the ring, the flux becomes the driving force for the coherent oscillations on the ring [13,17,59,72]. The probability flux is closely related to the asymmetric part of the neural connections. We can explore how the period and coherence of the oscillations for asymmetric neural networks and their relationships with the landscape topography. Both the flux and the landscape are crucial for the process of continuous memory retrievals in the oscillation attractors. We suggest that flux may provide driving force for the associations among various memory basins. The connections with different degree of asymmetry influence the capacity of memory [72]. Potential and flux theory can be applied to a rapid-eye-movement (REM) sleep cycle model [72,239,240]. We performed a global sensitivity analysis based on the landscape topography to explore the influences of the key factors such as release of acetylcholine and norepinephrine on the stability and function of the underlying neural network. Furthermore, we find that the flux is crucial for both the stability and the period of REM sleep rhythms The dynamics of general neural networks We start from the dynamical equations of Hopfield neural network with N neurons [231]: F i = du i = 1 N T i,j f j (u j ) u i + I i (i = 1,..., N). (129) dt C i R i j=1 The variable u i is the effective input action potential of the neuron i. The action potential u changes with respect to time in the process of charging and discharging of individual neuron. The collection of action potentials of different neurons can be used to represent the states of the neuron. C i represents the capacitance and R i represents the resistance of the neuron i. T i,j represents the strength of the connection from neuron j to neuron i. The function f i (u i ) is the firing rate of neuron i. It has a sigmoid functional form. The strength of the synaptic current into a postsynaptic neuron i due to a presynaptic neuron j is proportional to the product of the firing rate of the neuron i, f i (u i ) and the strength of the synapse connection T i,j from j to i. Therefore, the synaptic current can be represented by T i,j f j (u j ). The inputs of each neuron have contributions from three sources: postsynaptic currents from other neurons, leakage current due to the finite input resistance, and input currents I i from other neurons outside the circuit [72,231]. In the original Hopfield model, the strength of synapse T i,j must be equal to T j,i. Therefore, the connection strengths between neurons are symmetric. In this study, we will explore more general neural networks without this restriction and include the asymmetric connections between neurons. We will first explore the underlying stochastic dynamics of neural networks under fluctuations from the corresponding Fokker Planck diffusion equation for the probability evolution. We can then obtain the Lyapunov function φ 0 for a general neural network by solving the following HJ

105 Advances in Physics 103 equation from the leading order expansion in fluctuation strength characterized by the diffusion coefficient of the steady-state Fokker Planck equation [69,72]: n i=1 F i (u) φ 0(u) u i + n i=1 n j=1 D ij (u) φ 0(u) φ 0 (u) = 0. (130) u i u j For a symmetric neural network (T ij = T ji ), we can see right away that there exists a Lyapunov function which is the energy E of the system as [230,231] E = 1 2 N T ij f j (u j )f i (u i ) + i,j=1 N 1 ui R i i 0 N ξf i (ξ) dξ I i f i (u i ). (131) i For this symmetric case, it is easy to see that de N ( ) dfi (u i ) N = T ij f i (u i ) u i + I i dt dt R i = = i N i N i df i (u i ) C i u i dt j C i f i (u i) u 2 i. (132) Since the C i are always positive and the function f i increases with variable u i monotonously, the function E always decreases with time. As we can see, different from the energy defined by Hopfield, φ 0 is a Lyapunov function with respect to whether the neural network is symmetrically connected or not. In fact, φ 0 can be reduced to the energy function only when the connections of neural network are symmetric. In general, one needs to solve the HJ equation to quantify the φ 0. For a symmetric connections, the driving force of the neural network can be written as a gradient, F i = A(u) E(u), where A ij = δ ij /C i f i (u i). For a more general asymmetric connections, the driving force for the neural networks cannot be written as the form of pure gradient of the potential any more. Instead, according to the landscape and flux theory, the driving force are determined by both a gradient of a potential closely related to steady-state probability distribution and a curl flux. As we will see, complex neural behaviors such as oscillations can emerge in an asymmetric neural circuit. The oscillation behavior is impossible for Hopfield model with symmetrical neural connections. The non-zero flux J plays an important role in driving the coherent oscillations Potential and flux landscape of neural networks From the dynamics of the general neural networks, we started with the corresponding probabilistic diffusion equation and obtained the steady-state probability distributions P ss based on a self-consistent mean field method. We then quantify the underlying potential landscape (the population potential landscape U here is defined as U(x) = ln(p ss (x))) [13,14,17,60,72]. It is challenging to visualize the multidimensional state space of neural activity u. We can, however, select two state variables from the 20 in the neural network model to project the landscape by integrating out the other 18 variables.

106 104 J. Wang (a) (b) (c) (d) Figure 36. The 3D potential landscape figures from restricted symmetric circuit to totally unrestricted circuit. D = 1 for multi-stable case and D = for oscillation (from Ref. [72]). We first study a symmetric neural network from Hopfield model [231]. Figure 36(a) shows the landscape of the neural network. The symmetric network has eight basins of attractors. Each attractor represents a state that stores memory. When the neural network is cued to start with initial condition with incomplete information, it will go down to the nearest basin of attractor with a memory of complete information. This dynamical neural network guarantees the memory retrieval from a cue to the corresponding memory [72]. As we discussed earlier, we quantified a Lyapunov function φ 0 from the leading order expansion of the potential U(x) on the diffusion coefficient D. It is difficult to solve the equation of φ 0 directly due to the high dimensionality. We can apply a linear fit method for the diffusion coefficient D with respect to the DU to solve the φ 0 approximately. Figure 37 illustrates the intrinsic potential landscape φ 0 of the symmetric neural circuit. There are also eight basins of attractor. This landscape looks similar to the Hopfield energy landscape shown in Figure 35. Different from the energy defined by the Hopfield analysis which only works for the neural networks with symmetric connections, φ 0 is a Lyapunov function for neural network with either symmetric or asymmetrical connections. The landscape construction provides a general way to quantify the global features of the asymmetric neural circuits with Lyapunov function φ 0. To explore the role of asymmetry in neural connections, we chose a set of T ij randomly [72]. First, we set the symmetric connections, T ij = T ji when i j. HereT ii is set to zero indicating that neurons do not connect with themselves. Figure 36(a) shows the potential landscapes of this symmetric circuit, and we can see eight basins of attractor. We relaxed the restrictions on T ij. We set T ij = T ji for negative T ij when i j > 4 and i j > 5, and quantified the landscapes in Figure 36(b) and Figure 36(c), respectively. The landscapes show a trend that the number of stored memories decreases gradually. When we explored the original neural network without any restrictions on the connections T ij, there exists a possibility where all the stable fixed points disappear and a limit cycle emerges as shown in Figure 36(d). For this case, the potential

107 Advances in Physics 105 Figure 37. The potential landscape φ 0 of the symmetric circuit (from Ref. [72]). landscape has a Mexican hat shape. These figures show that as the neural network becomes less symmetric, the number of point attractors decreases. This result does not necessarily mean that the memory capacity of an asymmetric neural network must be smaller than a symmetric one. In fact, the limit cycle oscillation also stores the memory, but in a continuous fashion on the oscillation ring rather than in the isolated basin. As shown in Figure 36(d), oscillation can emerge for unrestricted T ij. The system cannot oscillate if it is driven only by the gradient force of the underlying landscape resulting from the symmetric neural connections. The driving force F in the general neural networks usually cannot be written as a pure gradient of a potential. As we have discussed earlier, the driving force F for a neural network is determined by both gradient of a potential landscape related to steady-state probability distribution and a curl flux [13,14,17,60,72]. In general, the neural network dynamics will stop at a point attractor only when the Lyapunov function reaches a global minimum. As shown earlier, the neural network can also oscillate where the values of Lyapunov function φ 0 on the oscillation ring are constant. The quantified Lyapunov function φ 0 provides a good quantitative description of the global intrinsic features of the underlying neural networks. The population potential U is not a Lyapunov function [69,72]. However, the population landscape potential U captures more details of the neural networks. This is because the population landscape is directly linked to steady-state probability distribution. For oscillations, U reflects the inhomogeneous probability distribution on the oscillation ring. This shows the inhomogeneous speed on the limit cycle oscillation ring. φ 0 being a global Lyapunov function does not capture this inhomogeneity information. For a symmetric circuit, the neural network cannot oscillate, since the gradient force cannot provide the vorticity needed for oscillations [231]. We can clearly see that the curl flux plays an important role as the connections between neurons become less symmetric. The curl flux is the key driving force when the neural network is attracted onto the limit cycle oscillation ring [72] Flux and asymmetric synaptic connections in general neural networks The oscillatory patterns of neural activities are widely distributed in our brain [233, ]. The oscillations play a mechanistic role in various aspects of memory including the spatial representation and memory maintenance [236,244]. Continuous attractor models have been investigated for the mechanism of the memory of eye position [ ]. However, the understanding of the sequential orders of the recall is still poor [248,249]. This is because the basins of the attractions storing the memory patterns are often isolated without any natural connections in the original symmetrical Hopfield networks. Asymmetric neural networks are capable of recalling sequences

108 106 J. Wang and the asymmetry determines the direction of flows in configuration space [250,251]. It is natural to expect that the flux may provide the driving force for associations among different memories. Synchronization is important in neuroscience. Recently, phase-locking among oscillations in different neuronal groups provides a new window to explore the cognitive functions involving communications among neuronal groups such as attention [252]. The synchronization can only occur among different groups with coherent oscillations. In the landscape and flux theory, the flux is closely related to the frequency of oscillations. We expect the flux to play an important role in modulation of rhythm synchrony Potential and flux landscape for REM/non-REM cycle We can apply potential and flux landscape theory to a more realistic model describing the REM/non-REM cycle with the human sleep data [72,239,240,253]. The REM sleep oscillations can be described by the interactions of two neural populations: REM-on neurons (mprf, LDT/PPT) and REM-off neurons (LC/DR). A limit cycle of the REM sleep system is similar to the predator prey model in ecology for describing the interactions between prey and predator populations [213,214]. The mprf neurons ( prey ) are self-activated through acetylcholine(ach). As the activities of REM-on neurons reach a certain threshold, REM sleep occurs. Being activated by Ach from the REM-on neurons, the LC/DR neurons ( predator ) inhibit REM-on neurons through serotonin and norepinephrine, then the REM episode is terminated. With less activations from REM-on neurons, the activities of LC/DR neurons decrease (a) (c) (b) (d) Figure 38. (a) The potential landscape for b = 2.0 and a = 1.0. The red arrows represent the flux. (b) The effects of parameters a and b on the barrier height. (c) The effects of parameters a and b on flux. (d) The phase coherence versus the degree of asymmetry S (from Ref. [72]).

109 Advances in Physics 107 due to self-inhibition (norepinephrine and serotonin). This leads to the release of REM-on neurons from inhibition. Another REM cycle starts right afterwards. This circuit can be mathematically described by the following dynamic equations: dx/dt = a A(x) x S 1 (x) b B(x) x y and dy/dt = c y + d x y S 2 (y). The x and y represent the activities of REM-on and REM-off neural populations, respectively. From the dynamics of REM sleep neural network, the potential landscape and the flux of the REM/non-REM cycles can be quantified [72]. As illustrated in Figure 38(a), the potential landscape U has a Mexican hat shape. The oscillations are mainly driven by the flux illustrated by the red arrows along the cycle. The flux plays a key role for the robustness of oscillations. Figure 38(b) and 38(c) show this explicitly: the barrier and the average flux along the ring is larger as a increases (the REM network is more stable and oscillations is more coherent (Figure 38(d)). Both the potential landscape and the flux are important for the stability of this oscillatory network. 11. Chaos: Lorentz strange attractor Landscape and flux theory can be used to explore the dynamics and the global stability of chaotic strange attractor with intrinsic fluctuations [68]. The Lorentz strange attractor is a typical nonlinear dynamical system exhibiting chaotic behavior, which often appears in nature and biology [254]. The underlying landscape overall has a butterfly shape. Both the landscape and curl probabilistic flux determine the dynamics even when there is chaos. The landscape attracts the system down to the chaotic attractor, while the curl flux drives the coherent motions on chaotic attractors. The curl probabilistic flux may provide us a clue to the mystery of chaotic attractor Introduction Nonlinear interactions can generate complex dynamics and patterns, such as limit cycles and chaos. Nonlinear dynamical systems have been extensively studied and applied to many fields, including physical systems, weather, chemical reactions, biological systems, information processing, etc. [68, ]. However, understanding the global stability and dynamics of the chaotic systems is a challenge. Finite systems often have the intrinsic statistical fluctuations from the number of molecules or components within and the external fluctuations from the environments [88, ]. It is important to investigate the stability of the dynamical systems under the stochastic fluctuations. For chaotic systems, being extremely sensitive to initial conditions, it is impractical to enumerate all the initial conditions to investigate the dynamical outcome of the system. Probabilistic approaches thus provide a useful route Landscape and probabilistic flux The classical Lorenz equations [254] take the following form: dx = σ(y x), dt dy = rx y xz, dt dz = xy bz. (133) dt A chemical Lorenz model can be derived from the classical Lorenz equations by a nonlinear transformation with the corresponding parameters as Ref. [68, Table 1]. The reactions of chemical

110 108 J. Wang Lorenz model can be given through the following reaction steps [255,263]: X + Y + Z > [k 1 ]X + 2Z, A1 + X + Y > [k 2 ]2X + Y, A2 + X + Y > [k 3 ]X + 2Y, X + Z > [k 4 ]X + P1, Y + Z > [k 5 ]2Y,2X > [k 6 ]P2, 2Y > [k 7 ]P3, 2Z > [k 8 ]P4, A3 + X > [k 9 ]2X, X > [k 10 ]P5, Y > [k 11 ]P6, A4 + Y > [k 12 ]2Y, A5 + Z > [k 13 ]2Z. (134) Here, k i (i = 1, 2,...,13) are chemical reaction rate constants. The concentration of species A i (i = 1, 2,...,5) and P i (i = 1, 2,...,6) is assumed to be constant. Writing x, y, and z separately for the concentrations of species X, Y, and Z, we can derive the corresponding deterministic (average kinetics) equations of the system: F 1 = dx dt = k 2xy 2k 6 x 2 + k 9 x k 10 x, F 2 = dy dt = k 1xyz + k 3 xy + k 5 yz 2k 7 y 2 k 11 y + k 12 y, F 3 = dz dt = k 1xyz k 4 xz k 5 yz 2k 8 z 2 + k 13 z. (135) where rate constants k i (i = 1, 2,...,13) have included the constant concentration of species A i. As mentioned, chemical Lorenz model can be derived from the classic Lorenz equations. They are not completely identical by comparing their solutions, although the solutions changed with parameters in a similar fashion. When the rate constants are set by k 1 = 0.001, k 2 = 0.1, k 3 = 0.81, k 4 = 1, k 5 = 1, k 6 = 0.05, k 7 = 0.005, k 8 = , k 9 = 1000, k 10 = 1000, k 11 = 8100, k 12 = 100, k 13 = (parameters when V = 1inRef.[68, Table 1], corresponding to σ = 10, r = 80, b = 8 in classical Lorenz equations), the chemical Lorenz model gives 3 a chaotic solution, jumping back and forth between the two butterfly wings. This chaotic behavior is very similar to that of the classical Lorenz equations. For the classic Lorenz equations, a parameter r was used to determine how the system transforms from stable solution into chaotic solution. In this chemical example, we also have a corresponding parameter r. We use this r as an adjustable parameter to study the different behaviors of chemical Lorenz model at different parameter regimes, although the parameter r in the chemical Lorenz model is not exactly the same as one used in the classic Lorenz equations. In the chemical dynamical systems, small numbers of molecules give rise to intrinsic fluctuations. In a system with only a finite number of molecules, intrinsic perturbations can influence the system behavior, due to the fact that fluctuation strength is proportional to 1/ N [16,68,264], where N represents the number of molecules. Therefore, the smaller the total number of molecules in a dynamical system, the larger fluctuations are expected. We define V as the effective dimensionless volume that scales the total molecular number and characterizes the intrinsic fluctuations of the system [264]. In this model, when V = 1, the total molecular number is about 10, 000. By changing V, we can explore the behavior of system at different molecular numbers. Stochastic simulation [255,263,265] and probabilistic approach [6] of the model Lorenz chaotic system gives the steady-state probability distribution of the molecular species concentrations. The steady-state probability distribution yields the potential landscape of the system

111 Advances in Physics 109 Figure 39. Four-dimensional (4D) landscape and associated probabilistic flux of chemical Lorenz model for molecular number variables X, Y, Z when the volume V is 10 (molecular number is 100,000), and r = 80. For background landscape, deep color represents lower potential energy or higher probability, and light color represents higher potential energy or lower probability. Landscape exhibits a butterfly shape with two oscillation rings coupled to each other. Magenta arrows represent the direction of probabilistic flux (from Ref. [68]). [8,11,13,17,57,64,68,266,267]: U = ln P ss, where P ss is the steady-state probability in the state space of concentrations. Figure 39 shows the three-variable (X, Y, Z) potential landscape of the chemical Lorenz chaotic system with the given parameter. The landscape has a butterfly shape with two wings connected together. Different colors show the depth of the landscape. Darker color shows a lower potential with higher probability, and lighter color shows higher potential with lower probability. The butterfly shape of the potential landscape is covered by the stochastic chaotic trajectories. Away from the butterfly landscape, the potential is higher with lower probability. On the coupled two wings around two eyes (holes) of the butterfly chaotic attractor, the potential is lower, with higher probability. The two holes have higher local potentials and lower probabilities than the surrounding butterfly wings. Therefore, the system is attracted to the chaotic attractors, and the shape of landscape guarantees the stability of the chaotic oscillator, as illustrated clearly in the 3D landscape and 2D landscape projection with two variables X, Y (Figure 40(a) and 40(b)). The Lorenz chaotic system is an open system. It can reach a non-equilibrium steady state (NESS). A non-zero flux is a distinct feature of a NESS [68]. The non-zero flux will generate the dissipation energy for sustaining the NESS. The probabilistic flux of the system in state space

112 110 J. Wang (a) (b) Figure 40. (a) shows 3D landscape and associated probabilistic flux of chemical Lorenz attractor for two variables X and Y when V = 10 (the corresponding molecular number is 100,000) and r = 80. (b) shows 2D landscape and corresponding probabilistic flux of chemical Lorenz model when V = 10. For background landscape, deep color represents lower potential energy or higher probability, and light color represents higher potential energy or lower probability. Magenta arrows represent flux, and white arrows represent negative gradient of potential energy (from Ref. [68]).

113 Advances in Physics 111 of concentrations is given by [1] J(x, t) = FP D ( P). Both the potential landscape and curl probabilistic flux determine the dynamics and global features. The dynamics of the system can be described as a spiral around the gradient (F = 1 2 D (ln P ss) + J ss /P ss DT, ss denotes the steady state). It is easier to visualize the landscape in two dimensions. We explore the two-variable (X and Y ) projection of Lorenz attractor for V = 10 and r = 80. In Figure 40(a) and Figure 40(b), the landscape in two variables X and Y space has the shape of the two joint ring valleys. Each butterfly wing has its own ring valley. In Figure 40(b), for each ring valley, the direction of the flux is along individual orbit on the ring. This drives the oscillations. In addition, by observing the directions of the fluxes, we can see that the curl flux flows along the individual orbits on the ring. Flux also flows along the outer edge of the double-ring butterfly. This illustrates that flux vector has two components. One is the individual curl flow for each ring, and the other is the flow along the whole attractor. In the original Lorenz model [254], the trajectory of Lorenz attractor transfers from one spiral to the other at irregular intervals. In our model, this corresponds to that the system makes a transition from one oscillation ring valley to the other. Based on the analysis above, this transition is driven by the outer flux flowing along outer edge of the ring. For the Lorenz system, both landscape and flux determine the dynamical behavior of the chaotic attractor. The landscape attracts system to the double-ring valleys and curl flux drives the oscillation transitions of the system between the double wings. The landscape and flux theory provides a non-local view to understand the global features of Lorenz strange attractor [68] Flux may provide a clue to the formation of chaos Lorenz system is a non-integrable dynamical system. It is not determined by the pure gradient driving force. We can easily check this by taking the curl of the driving force resulting nonzero values. In fact, there is no hamiltonian energy which gives gradient for the Lorenz system. The deviation from the pure energy function may give rise to the driving force for the chaotic attractor [68]. From Equation (8), the driving force F of chemical Lorenz system can be written as follows: F = F flux + F gradient + F diffusion, where F flux (J ss /P ss ) represents the force from flux, and F gradient ( 1 D U) represents the force from the gradient of the potential. We will mainly explore the 2 relative magnitude of these two components, due to the relatively smaller contributions from the other item related to diffusion F diffusion ( 1 2 DT ). Figure 41 gives the results of the ratio F flux /F gradient when volume V and parameter r are separately changed. In Figure 41(a), when V increases, the ratio of F flux and F gradient (the average of F flux and F gradient ) increases. From the corresponding distributions (Figure 41(e)), we can reach consistent results. When V increases, the noise decreases, implying the force from the gradient of potential decreases. The force from flux then becomes more prominent. This illustrates that the smaller the fluctuations, the larger the relative contribution of the flux to the driving force of the system. Figure 41(c) also show consistent results where F flux /F gradient increases as the coherence of the system increases. From Figure 41(b), when the parameter r increases, the ratio of F flux to F gradient also increase. Figure 41(f) shows the corresponding distribution of F flux /F gradient at different r values. Figure 41(d) illustrates that the ratio F flux /F gradient increase when the coherence of the chaotic system increases. The corresponding stability of the chaotic attractor increases (the r increases) as the flux contribution in the total driving force increases. From Figures 42(a) ( c) and 41(b)(d)(f), an upward trend is observed for both average flux, ratio of flux and gradient force, and EPR going from chaos to limit cycle as r increases. This

114 112 J. Wang (a) (b) (c) (d) (e) (f) Figure 41. (a) shows the change of K when V is increased when r = 80. (b) shows the change of K when r is increased when V = 10. Here K is defined as: F flux /F gradient, representing the relative magnitude of force from flux and force from gradient of potential. (c) and (d) show separately K versus coherence when V and r is changed. (e) and (f) show separately the distribution of K at different V and r (from Ref. [68]). (a) (b) (c) Figure 42. (a) (c) show separately coherence, barrier height, the EPR, and the height of peak of power spectrum of auto correlation function versus parameter r (from Ref. [68]). shows that as the order increases from chaos to limit cycle, the average flux, ratio of flux and gradient force, and EPR characterizing the dissipation and energy cost increases. This shows that more ordered system as limit cycle requires more flux and energy to sustain the stability compared to more disordered chaotic system. There are often chaotic systems including autonomous system such as Van der Pol oscillator and Duffing oscillator and non-autonomous system such as Lorenz system, Rossler system,

115 Advances in Physics 113 Henon Heiles system (Hamiltonian system), and Chua circuit system that can be studied with the present ideas. None of them are gradient systems. The driving force for all of them have non-zero curl giving curl flux responsible for chaotic behavior [68]. 12. Multiple landscapes and the curl flux for a self-regulator We consider a simplest circuit of a self-regulating single gene (Figure 43(a) for illustration) [74]. The self-regulating gene dynamics can be described by several key factors. One is the on and off of the gene states. The other is the protein concentrations. When a regulator protein binds to the gene, the gene is either on or off depending on whether the regulator is an activator or repressor. When the gene is activated, through transcription and translation processes, the protein is synthesized and produced. In the self-regulating gene system, the proteins produced by the gene will act back to its own gene and regulate the activity of the gene. The dynamics of such system can be described by the underlying chemical reactions for protein synthesis and degradation, and the binding/unbinding of the regulating proteins to the genes. Furthermore, in the cell, there are only finite number of molecules (typically less than 10 4 ); therefore, the statistical fluctuations in molecular numbers need to be taken into consideration. The dynamical process of (a) (c) (b) Figure 43. Illustrations of self-regulating gene dynamics. (a) Reaction scheme in the self-regulating gene circuit. (b) The multiple landscapes and dynamics of the self-regulating gene based on the view of Figure 11(d) (dimension of individual landscapes is one in B and two in Figure 11(d)). (c) The equivalent single landscape and dynamics on the expanded space of the protein concentration ρ and the gene state ξ. Dotted lines show the basin of attractor and arrows represent the curl flux (from Ref. [74]).

116 114 J. Wang the self-regulating genes can be described by the following master equation [74]: P(n) t = ( ) g1 0 [P(n 1) P(n)] + k(n + 1)P(n + 1, t) 0 g 0 ( ) h f knp(n) + P(n). (136) h f Here P(n) is a vector with two components P 0 (n) and P 1 (n), with discrete variable; 0 and 1 represent the state of the gene either in on or off state. Therefore, P(n) gives the probability of the protein concentration when the gene is on or off. Here, g 0 is the production or synthesis rate of protein at the state when a regulation protein is bound to the gene, and g 1 is the production or synthesis rate when the regulation protein is not bound to the gene. If g 1 > g 0, it corresponds to the self-repressor case. If g 1 < g 0, it corresponds to the self-activator case. k is the degradation rate of the protein. h 0 is the binding rate of the regulating protein to its gene, while f is the unbinding rate of the regulating protein from its gene. We assume that the regulation factor is a dimer of the product proteins, so that h = h 0 (n + 1)n. One can think of the self-regulating gene dynamics as a coupled dynamical systems. For a specific gene state, the protein concentration dynamics is dictated and follows the gradient dynamics due to its 1D nature on the protein concentration on the one hand and the coupling dynamics through the binding/unbinding of the proteins to the genes on the other hand. Therefore, we can think of the self-regulation genes dynamics as moving along the protein concentration space and jumping between the two potential energy landscape surfaces labeled by discrete values of 0 and 1 representing the state of the gene. The timescale of binding/unbinding of regulating proteins to the genes determining the gene states relative to the synthesis and degradation of the protein determining the protein concentrations gives the origin of the complexity of the problem (non-adiabaticity and adiabaticity). We follow the landscape and flux theory for multiple landscapes and the procedure described earlier (see the multiple landscape theory section). This leads to the coupled-langevin equations from the original continuous variable ψ for particle number and a discrete variable representing the on and off gene state to the current continuous variables ψ and ξ, where ξ can be thought of the continuous representation of discrete description of on and off state (e.g. a continuous variable ξ can be defined as the probability of finding the discrete on or off state) [74]. ψ = ψ + X ad + ξδx + η ψ, (137) 1 ω ξ = K(1 + ξ)ψ 2 + (1 ξ)+ η θ, (138) where η ψ and η θ are Gaussian random numbers with η ψ =0, η θ =0, and η ψ (t)η ψ (t ) = 1 2 (ψ + X ad + ξδx )δ(t t ), η θ (t)η θ (t ) = 1 ω (K(1 + ξ)ψ2 + (1 ξ))δ(t t ). (139) Here, X ad is the representative copy number of protein, and ω is the adiabaticity parameter measuring the unbinding rate relative to the degradation rate. When ω is large, the binding/unbinding of the proteins to the genes is much faster than the synthesis and degradation of the proteins, the system is in adiabatic limit. When ω is small, the binding/unbinding of the proteins to the genes is comparable or slower than the synthesis and degradation of the proteins, the system is

117 Advances in Physics 115 in the non-adiabatic regime. ω parameter characterizes the relative dynamic timescales of the system and will be the focus of our discussion as mentioned earlier. K is the ratio of binding versus unbinding rate of the proteins to the gene, mimicking the equilibrium binding constant. δx represents the difference in the synthesis rate between on and off states of the genes. If we rewrite Equation (137) using the system volume and defining ρ = ψ/, x ad = X ad /, δx = δx /, and κ = K 2, the Langevin equations are more symmetrical as with ρ = ρ + x ad + ξδx + ɛ ρ, (140) 1 ω ξ = κ(1 + ξ)ρ 2 + (1 ξ)+ ɛ θ (141) ξ = cos θ, ɛ ρ (t)ɛ ρ (t ) = 1 2 (ρ + xad + ξδx)δ(t t ), ɛ θ (t)ɛ θ (t ) = 1 ω (κ(1 + ξ)ρ2 + (1 ξ))δ(t t ). (142) Here, ρ can be regarded as the density of protein, so that Equation (140) is the usual equation of the volume -expansion, representing the statistical fluctuations from the finite number of molecules deviating from the large number of molecules, and we can say that Equation (141) is the corresponding equation of the adiabaticity parameter ω-expansion, representing the timescale fluctuations deviating from the adiabatic case where the binding/unbinding of proteins to the genes is significantly faster than the corresponding protein synthesis and degradation Potential, circulation flux, and eddy current Our aim is to obtain an intuitive picture of relations among potential, eddy-current (circulation), adiabatic change of states, and churn motions of non-adiabatic dynamics [74]. Let us focus our discussion on the coupled Lagevin equation. As we discussed, we transformed the original self-regulating gene dynamics along the protein concentration variable n and discrete values of on and off gene states to the problem of dynamics in the continuous variables ψ or ρ representing the concentrations and θ or ξ (continuous representation) representing the gene states. We can first discuss the adiabatic case when ω is large, where the binding/unbinding of the proteins to the gene is relatively fast compared with the synthesis and degradation rate of the proteins. This means the gene switches its on and off states frequently. In this case, from equation for temporal dynamics of ξ, both the left-hand side of the equation and the noise term tends to zero since they are inversely proportional to the ω and ω, respectively. As a result, we obtain a specific relationship between the ρ and ξ through κ(1 + ξ)ρ 2 + (1 ξ) = 0or ξ = (1 κρ 2 )/(1 + κρ 2 ). If we substitute this expression to the equation for the dynamics of ρ, we get ρ = ρ + x ad + ((1 κρ 2 )/(1 + κρ 2 ))δx + ɛ ρ or equivalently ρ = ( U/ ρ) + ɛ ρ, where U is the effective potential given as U = 1 2 ρ2 x ad ρ δx(2arctg[ κρ]/ κ ρ). Therefore, for the self-regulating gene in the adiabatic limit, the strong couplings between gene states lead to frequent flipping of the gene states. We reduce this to a 1D problem with an effective potential landscape U giving the pure gradient dynamics. The effective potential is a result of the fast binding/unbinding of the regulators to the genes. On the other hand, if the ω is not necessarily large, the binding/unbinding of the proteins to the gene is comparable or even slow compared with the synthesis and degradation rate of the

118 116 J. Wang proteins. This is the non-adiabatic regime. In this regime, there is no approximation we can use for simplifying the coupled-langevin equations further. This is because we no longer have the timescale separation any more as the adiabatic case. Therefore, the system is inherently 2D. So we can see in the non-adiabatic limit, we will have the following picture. By solving the corresponding 2D Fokker Planck equation in the steady state, we can find the steady-state probability distribution P ss. By the relationship of U = ln P ss, we characterize the potential landscape of the system [13,74]. Furthermore, we find that the driving force of our 2D problem cannot be written as a pure gradient of a potential. The probability flux can be obtained through the subtraction of gradient and divergent diffusion matrix from the deterministic driving force. The net flux is not equal to zero. This implies that the system in the non-adiabatic case is in a non-equilibrium regime with broken detailed balance. The degree of non-equilibriumness or detailed balance breaking is quantified by the strength of the curl probability flux. So in the 2D space of protein concentration ρ and gene state ξ in the non-adiabatic regime, the dynamics is determined by both the gradient of the landscape U and curl flux. Instead of straightly going down to the gradient, the motion proceeds curly spiraling down the gradient. The curl probability flux provides the origin [74] of the eddy-current motion in the dynamic trajectories [45]. Let us try to connect the new picture of non-adiabatic dynamics with some previous studies. If we see the non-adiabatic case in another angle as in previous investigations, we can imagine two landscapes each in ρ space with the discrete gene state labeling of ξ = 0 and ξ = 1 potential energy surfaces. The stochastic dynamics is mainly moving along one energy surface and occasionally jump to another energy surface. Moving along the new energy surface for a while, then the system jumps back to the original energy surface. This processes keep on iterating so the stochastic trajectory along and between the two energy surfaces forms a cyclic churn-like spiraling motion. The cause of this eddy-current motion is the jumps between the energy surfaces. By expanding our system variables from discrete to continuous, we realize that although dynamics along specific gene state can be determined by the gradient of the single potential surface, the motion in the 2D space of protein concentration and continuous gene states can no longer be seen in this way. In fact, the dynamics can be described by a single potential landscape U instead of two individual landscapes here for this self-regulating gene example, giving the gradient part of the dynamics. See Figure 11. Furthermore, the detailed balance is broken. This is characterized by the steady-state probability flux which has the divergent-free curl nature. From this continuous representation, we can quantify the origin that the churn eddy-current motion observed in the simulations by the strengths of the curl steady-state probability flux. Finally, let us point out an extreme non-adiabatic condition where the ω goes to zero [74]. In this case, from equation for dynamics of ξ, ξ = 0, then ξ = constant and the motion is fixed at a specific ξ along ρ. Again the effective dynamics is a gradient of the potential landscape along 1D protein concentration coordinates ρ, with effective potential U = 1 2 ρ2 (x ad + ξδx)ρ. There is essentially no jumping between the energy surfaces because the binding/unbinding of regulators to the genes is so slow. Therefore, the dynamics becomes very simple, following gradient of the potential along a single energy landscape surface formed by the protein synthesis and degradation similar to the adiabatic case although the effective potential is different. The dynamics does not have any curl flux component and therefore no eddy-current churn motion. Therefore, for both extreme cases of very fast or very slow binding/unbinding of regulatory proteins to the gene relative to the synthesis and degradation of the proteins, the dynamics of self-regulator is driven by the pure gradient of the underlying single potential landscape. On the other hand, when the timescale of binding/unbinding of regulatory proteins to the gene is comparable to the synthesis and degradation of the proteins, the dynamics of self-regulator follows a gradient of the underlying landscape and a curl flux giving the possibility of eddy-current and churn motion. So the timescale is the key for non-adiabatic dynamics and leads to the origin of the curl probability flux.

119 Advances in Physics 117 We must point out that our conclusion above on the decomposition of the driving force for dynamics is only valid for the self-regulator network systems [74]. For general complex dynamical systems, even in the adiabatic limit of timescale separation, the dynamics in general is not driven by a pure gradient of a single potential landscape but in addition a curl flux force. The origin of the curl is from the underlying dynamics. On the other hand, in the non-adiabatic regime, the dynamics in general is driven by both gradient of the potential landscape and the curl flux force. The curl flux now has two contributions, one from the adiabatic part and the other is the non-adiabatic part from the timescale consideration Results and discussions In this part, we will explore the consequence of taking the timescale into the consideration for the dynamics and especially the role of the curl flux leading to eddy-current and churn motion [74]. In Figure 44, we show a 2D contour map of the probability landscape U = ln P ss in protein concentration ρ and gene state ξ. The landscape has a single basin of attraction. When the ω is large, the flux is small and negligible in numerical values. So the dynamics in adiabatic regime is driven by the gradient of the landscape. The orientation of the landscape is vertical and therefore the corresponding projections to the protein concentration variables at the two gene states of on and off with ξ = 0 and ξ = 1 giving the same result at the same location. Therefore, there is only single peak for the distribution at the protein concentration space when ω is small in this adiabatic regime. On the other hand, when the ω becomes smaller, the landscape is still a single basin of attraction, but the dynamics is non-adiabatic and driven by both the gradient of the landscape and the non-zero curl flux. The orientation of the landscape is tilted from the vertical direction. The smaller ω is the more tilted is the landscape. As we can see, this orientation of the landscape is driven by the non-zero curl flux giving the eddy-current like motion in the combined protein concentration and gene state space. The tilting orientation of the landscape leads to the corresponding projections to the protein concentration variables at the two gene states of on and off with ξ = 0 and ξ = 1 giving the different results at the two different locations. Therefore, two peaks for the distribution at the protein concentration space are expected when ω is small in the non-adiabatic regime. For the self-repressor, the one peak is typically expected since for many procaryotic cells, the binding/unbinding rate is relatively fast compared with the protein synthesis and degradation. While for eukaryotic cells, the binding/unbinding can be comparable or slower than protein synthesis and degradation. In this non-adiabatic regime, the two peaks emerge which (a) (b) (c) Figure 44. Calculated single composite landscape on the expanded space of the protein concentration ρ and the gene state ξ at (a) ω = 0.01, (b) ω = 0.1, and (c) ω = 100. Superimposed on landscapes are J ss (white arrows) and dominant kinetic paths between the gene-on and gene-off states (black lines and blue lines). Red lines are distributions of ρ in the gene-off (ξ = 1) and on (ξ = 1) states (from Ref. [74]).

120 118 J. Wang Figure 45. The Fano factor as function of log ω, which is the variance divided by the mean, showing the relative width of the steady-state probability distribution of protein concentration (from Ref. [74]). are quite unexpected from the conventional point of view. We show here that the two peaks of the protein concentration distribution at different locations here originate from the curl flux which does not preserve the detailed balance and leads to the eddy current breaking the symmetry of the original one peak at the same location for both gene-on and gene-off states [74]. In Figure 45, we also show the Fano factor (variance divided by the mean) or relative width of the distribution peaks as a function of the adiabaticity parameter ω. We find that when the omega is large, binding and unbinding are frequent, therefore the gene states are strongly coupled. This leads to narrower and Poisson-like distribution (Fano factor is close to 1) of the protein concentrations. This is because the dynamics essential follows gradient of the effective landscape and motion is convergent to the same location. On the other hand, when the ω becomes smaller, the couplings between the gene states are weaker. The dynamics is driven by both the gradient of potential landscape and the curl flux. The effect of the curl flux is to form eddy-current motions. This will lead to the dispersion of the motion, leading the distribution of the protein concentration to split from one peak to two peaks in different locations. We see the broader distribution of the protein concentrations. This is deviating from Poisson and the fluctuations characterized by the Fano factor become large. Again, we now understand this is due to the non-zero curl flux [74]. In Figure 46, we quantify the kinetic paths from the gene-on state to the gene-off state and back. Kinetic paths are crucial for understanding how the evolution dynamics of the system is realized. We see that for the adiabatic case when ω is large, the forward and backward kinetic paths are almost identical and opposition in direction. This is expected because the underlying dynamics is driven by the gradient of the potential landscape. On the other hand, for the nonadiabatic case when ω is small, the forward and backward kinetic paths are significantly different from each other and they are irreversible. The irreversibility of the kinetic paths means the time reversal symmetry is violated. Again this is caused by the non-zero curl flux. The curl flux gives

121 Advances in Physics 119 (a) (b) (c) Figure 46. Calculated single composite landscape on the expanded space of the protein concentration ρ and the gene state ξ at (a) ω = 0.01, (b) ω = 1, and (c) ω = 100. Superimposed on landscapes are J ss (black arrows) and dominant kinetic paths between the gene-on and gene-off states (white lines and green lines) (from Ref. [74]). (a) (b) (c) Figure 47. The two time correlation function in protein concentration with (a) ω = 0.01, (b) ω = 0.1, and (c) ω = 100 (from Ref. [74]). the directional preference and therefore leads to the non-equivalence of forward and backward paths. The curl flux breaks the detailed balance and therefore gives the time asymmetry [74]. In Figure 47, we show the two-point correlation function of protein concentrations in time. We found that for adiabatic case when ω is large, the correlation in time follows a single exponential law. On the other hand, when ω is small in the non-adiabatic regime, the correlation function in time starts to have complex behavior ranging from multi-exponential to weak oscillations. Again the curl flux is the origin of this behavior since it can give rise to more apparent states with higher probability as shown earlier through the dispersion of the original single peak distribution in protein concentration. Furthermore, because of the curling nature of the flux, cycle oscillations become possible. So the complexity in the time correlation function deviating from single exponential is a quantitative signature of the curl flux and detailed balance breaking [74]. In Figure 48, we calculated the residence time (the waiting time before the state switching) distribution of particular gene states [74]. For adiabatic case when ω is large, the residence time distribution monotonically decays. On the other hand, when ω is small in the non-adiabatic regime, the residence time starts to form a peak of its distribution. This is another signature of the detailed balance breaking which is caused by the curl flux. The curl flux leads to the irreversible