State-Feedback Control of Partially-Observed Boolean Dynamical Systems Using RNA-Seq Time Series Data
|
|
- Jane Gregory
- 6 years ago
- Views:
Transcription
1 State-Feedback Control of Partially-Observed Boolean Dynamical Systems Using RNA-Seq Time Series Data Mahdi Imani and Ulisses Braga-Neto Department of Electrical and Computer Engineering Texas A&M University College Station TX USA Abstract External control of a genetic regulatory network is used for the purpose of avoiding undesirable states such as those associated with disease. This paper proposes a strategy for state-feedback infinite-horizon control of Partially-Observed Boolean Dynamical Systems (POBDS) using a single time series of Next-Generation Sequencing (NGS) RNA-seq data. A separation principle is assumed whereby first the optimal stationary policy is obtained offline by solving Bellman s equation and then an optimal MMSE observer the Boolean Kalman Filter is employed for online implementation of the policy using the RNAseq observations of the evolving system. Performance is investigated using a Boolean network model of the mutated mammalian cell cycle and simulated RNA-seq observations. State-Feedback Control Boolean Dynamical Systems Boolean Kalman Filter Next-Generation Sequencing. I. INTRODUCTION A fundamental problem in genomic signal processing is to design intervention strategies for gene regulatory networks to beneficially alter network dynamics. This task is usually done to reduce the steady-state mass of undesirable states such as cell proliferation states which may be associated with cancer [1]. Boolean networks [2] have emerged as an effective model of the dynamical behavior of gene regulatory networks consisting of genes in activated/ inactivated states the relationship among which is governed by logical relationships updated at discrete time intervals [3] [4]. To date different modeling approaches such as Probabilistic Boolean Networks (PBNs) [3] S-systems [5] and Bayesian networks [6] have been proposed in literature to mathematically capture the behavior of genetic regulatory networks. In addition various intervention approaches [1] [7] [9] have been developed. These all assume that the system states are directly measurable and are mostly in the framework of Probabilistic Boolean Networks (PBNs). In this paper we consider a signal model with distinct state and observation processes; thus the transcriptional state of the genes is not assumed to be observable directly but only though indirect RNA-seq measurements [4] [10] [12]. Initially we apply the theory of infinitehorizon optimal stochastic control to find the stationary policy that optimally controls the evolution of a Boolean dynamical system is applied. In the next step we apply the optimal MMSE observer for the proposed signal model known as the Boolean Kalman Filter [4] to estimate the state of the system to which is applied the previously obtained control policy. Performance is investigated using a Boolean network model of the mutated mammalian cell cycle and simulated RNA-seq observations. II. PARTIALLY-OBSERVED BOOLEAN DYNAMICAL SYSTEMS Deterministic Boolean network models are unable to cope with (1) uncertainty in state transition due to system noise and the effect of unmodeled variables; the fact that (2) the Boolean states of a system are never observed directly this calls for a stochastic approach. We describe below the stochastic signal model for partially-observed Boolean dynamical systems first proposed in [4] which will be employed here. A. State Model We assume that the system is described by a state process {X k ; k = } where X k {0 1} d is a Boolean vector of size d in the case of a gene regulatory network the components of X k represent the activation/inactivation state of the genes at time k. The state is affected by a sequence of control inputs
2 {u k ; k = } where u k U represents a purposeful intervention into the system state in the biological example this might model drug applications. The state evolution is thus specified by the following discrete-time nonlinear signal model: X k = f (X k 1 u k 1 ) n k (1) for k = where f {0 1} d U {0 1} d is a network function {n k ; k = } is a white state noise process with n k {0 1} d and indicates component-wise modulo-2 addition. The noise is white in the sense that it is uncorrelated; i.e. n k and n l are uncorrelated for k l. In addition the noise process is assumed to be uncorrelated from the state process and control input. B. Observation Model In most real-world applications the system state is only partially observable and distortion is introduced in the observations by sensor noise. Let {Y k ; k = } be the observation process which is the actual data obtained from the experiment. The observation Y k is formed from the state X k through the equation: Y k = h (X k v k ) (2) for k = where {v k ; k = } is a white observation noise process which is uncorrelated from the state process. We will consider here the specific case of RNA-seq transcriptomic data in which case Y k = (Y k1... Y kd ) contains the RNA-seq data at time k; for a single-lane NGS platform Y kj is the read count corresponding to transcript j in the single lane for j = 1... d. In this paper we choose to use a Negative Binomial model for the number of reads for each transcript: P (Y kj = m X kj ) = m Γ(m + φ j ) m! Γ(φ j ) ( λ kj φ j ) ( ) λ kj + φ j λ kj + φ j φ j (3) for m = ; where Γ denotes the Gamma function and φ j λ kj > 0 are the real-valued inverse dispersion parameter and mean read count of transcript j at time k respectively for j = 1... d. The inverse dispersion parameter models observation noise; the smaller it is the more variable the measurements are. Now recall that according to the Boolean state model there are two possible states for the abundance of transcript j: high if X kj = 1 and low if X kj = 0. Accordingly we model the parameter λ kj in log space as: log λ kj = log s + µ + δ j X kj (4) where the parameter s is the sequencing depth (which is instrument-dependent) µ > 0 is the baseline level of expression in the inactivated transcriptional state and δ j > 0 expresses the effect on the observed RNA-seq read count as gene j goes from the inactivated to the activated state for j = 1... d. Typical values for all these parameters are given in section VI. III. BOOLEAN KALMAN FILTER The optimal filtering problem consists of given the history of observations Y 1... Y k up to the present time k finding an estimator ˆX k (Y 1... Y k ) of the state X k that optimizes a given performance criterion. The criterion considered here is the (conditional) mean square error: MSE(Y 1... Y k ) = E [ ˆX k X k 2 Y k... Y 1 ] (5) The minimum mean-square error (MMSE) state estimator for the model described in the previous sections is the Boolean Kalman Filter (BKF). A recursive algorithm for the exact computation of the BKF for a general signal model was given in [4]. Briefly let (x 1... x 2d ) be an arbitrary enumeration of the possible state vectors. For each time k = define the posterior distribution vectors (PDV) Π k k and Π k k 1 of length by (Π k k ) i = P (X k = x i Y k... Y 1 ) (6) (Π k k 1 ) i = P (X k = x i Y k 1... Y 1 ) (7) for i = Let the prediction matrix M k of size be the transition matrix of the controlled Markov chain corresponding to the state process: (M k ) ij = P (X k = x i X k 1 = x j u k 1 = u) = P (n k = x i f(x j u)) (8) for i j = On the other hand assuming conditional independence among the measurements given the state the update matrix T k also of size is a diagonal matrix defined by: (T k (Y k )) ii = P (Y k X k = x i ) d Γ(Y kj + φ j ) s exp(µ + δ j (x i Y kj ) j ) = j=1 Y kj! Γ(φ j ) s exp(µ + δ j (x i ) j ) + φ j φ φ j j ( ) s exp(µ + δ j (x i ) j ) + φ j (9) 2
3 for i = For a Boolean vector v {0 1} d define the binarized vector v(i) = I v(i)>1/2 and the complement vector v c (i) = 1 v(i) for i = 1... d and the L 1 -norm v 1 = d v(i). Finally define the matrix A of size d via A = [x 1 x 2d ]. Then it can be shown that the optimal MMSE estimator ˆX k can be computed by Algorithm 1. Algorithm 1 Boolean Kalman Filter 1: Initialization Step: The initial PDV is given by (Π 0 0 ) i = P (X 0 = x i ) for i = 1... For k = do: 2: Prediction Step: Given the previous PDV Π k 1 k 1 the predicted PDV Π k k 1 is given by Π k k 1 = M k Π k 1 k 1 3: Update Step: Given the current observation Y k =y k let β k = T k (y k ) Π k k 1 The updated PDV Π k k is obtained by normalizing β k to obtain a probability measure: Π k k = β k / β k 1 4: MMSE Estimator Computation Step: The MMSE estimator is given by ˆX k = AΠ k k with optimal conditional MSE MSE(Y 1... Y k ) = min{aπ k k (AΠ k k ) c } 1. IV. DISCOUNTED COST INFINITE-HORIZON CONTROL In this section the infinite-horizon control problem when the states of the system are directly observed is formulated. We will state the problem and give its solution without proof. For more details the reader is referred to [7] [13]. The goal of control is to select the appropriate external input u ext k U ext at each time k to make the network spend the least amount of time on average in undesirable states; e.g. states associated with cell proliferation which may be associated with cancer [1]. In formal terms assuming a bounded cost of control g(x k u ext k ) that is a function of the state and the intervention at time k our goal is to find a stationary control policy µ {0 1} d U ext which minimizes the infinite-horizon cost (for a given initial state X 0 = x j 0 ): m J µ (j) = lim E γ m k g (X k µ(x k )) (10) k=0 for j = 1... where 0 < γ < 1 is a discounting factor that ensures that the limit of the finite sums converges as the horizon length m goes to infinity. The discount factor places a premium on minimizing the costs of early interventions as opposed to later ones which is sensible from a medical perspective [7]. Let Π be the set of all admissible control policies. We assume in this paper that the set of external inputs U ext is finite so that Π is also finite. The optimal infinitehorizon cost is defined by J (j) = min µ Π J µ(j) (11) for j = 1... where J µ (j) is given by (10). The following theorem which follows from classical results proved in [13] provides necessary and sufficient conditions for optimality of a stationary control policy. Here we assume that the transition matrix M k of the controlled Markov chain defined in (8) may depend on k only through the input; the dependency on the input is made explicit by writing M(u) while dropping the index k. Theorem 1. (Bellman s Equation) The optimal infinitehorizon cost J satisfies: J (j) = min g(x u U j u) + γ (M(u)) ij J (i) ext (12) for j = Equivalently J is the unique fixed point of the functional T [J] defined by T [J](j) = min g(x u U j u) + γ (M(u)) ij J(i) ext (13) for j = 1... where J {0 1} d R is an arbitrary cost function. Furthermore a stationary policy µ is optimal if and only if it attains the minimum in Bellman s equation (12) for each initial state x j that is T [J ](j) = min g(x u U j u) + γ ext = g(x j µ (x j )) + γ (M(u)) ij J (i) (M(µ (x j ))) ij J (i) (14) 3
4 for j = The previous theorem suggests a procedure to determine the optimal stationary control policy. Starting with an arbitrary initial cost function J 0 {0 1} d R run the iteration J t = T [J t 1 ] (15) until a fixed point is obtained it can be shown that the iteration will indeed converge to a fixed point [13]. This fixed point is the optimal cost J and the corresponding policy defined by (14) is the optimal stationary control policy. In practice a stopping criterion has to be defined to stop the iteration when the change between consecutive J t 1 and J t is small enough. The procedure is summarized in Algorithm 2. Algorithm 2 Optimal Infinite-Horizon Control 1: Initialization Step: Initialize J 0 arbitrarily e.g. J 0(j) = max... g(x i u) for j = Choose a small number β and discounting factor γ. 2: Let J(j) = J(j) for j = : Let J(j) = min u U ext g(x j u) + γ (M(u)) ij J(i) for j = : Compute = max j=1...( J(j) J(j) ). 5: If > β go to step 2. 6: The optimal stationary control policy is: µ (x j ) = arg min for j = u U ext g(x j u) + γ (M(u)) ij J(i) V. OPTIMAL INTERVENTION STRATEGY To be able to apply the optimal stationary policy the states of the system must be known at each time step. However in most real applications the states of the systems are not directly observable. Therefore we propose to use the Boolean Kalman Filter to estimate the states for application of the policy. While the optimal policy is determined offline the BKF is applied online and the control input dictated by the optimal control policy is applied to the state estimate at each time k (the control policy is suboptimal on the space of observed data). The proposed approach is illustrated in Figure 1. BKF Optimal Policy System Fig. 1: The proposed scheme for intervention to data observed through noise. VI. NUMERICAL EXPERIMENT In this section we present results of a numerical experiment using a Boolean network based on the mutated mammalian cell cycle network. Mammalian cell division is coordinated with the overall growth of the organism through extracellular signals that control the activation of CycD in the cell. This mutation introduces a situation where both CycD and Rb might be inactive in which case the cell cycles (i.e. divides) in the absence of any growth factor [14]. This suggests considering the logical states in which both Rb and CycD are downregulated as undesirable proliferatory states. Table I presents the Boolean functions of the mutated cell-cycle network. TABLE I: Mutated Boolean functions of mammalian cell cycle. Genes CycD Rb E2F CycE CycA Cdc20 Cdh1 UbcH10 CycB Predictors Input (CycD CycE CycA CycB) (Rb CycA CycB) (E2F Rb) (E2F Rb Cdc20 (Cdh1 UbcH10)) (CycA Rb Cdc20 (Cdh1 UbcH10)) CycB (CycA CycB) Cdc20 Cdh1 (Cdh1 UbcH10 (Cdc20 CycA CycB) (Cdc20 Cdh1) The control input consists of flipping or not flipping the state of a control gene. The cost function is defined as follows: g(x j 5 + O(u) if (CycDRb) = (00) u) = { O(u) if (CycDRb) (00) where O(u) is 0 if state of the control gene is not flipped or 1 if it is flipped. The process noise is assumed to have independent components distributed as Bernoulli(p) with intensity p = 0.01 so that all genes are perturbed with a small 4
5 Fraction of observed desirable states in long run Sum of state values of CycD and Rb genes probability except the transducer gene CycD which is perturbed by noise with intensity p = 0.5 to simulate the presence or absence of extracellular signals sensed by this gene. The parameters of the RNA-seq model are assumed as follows: the sequencing depth s is equal to which corresponds to 50K 100K reads; the baseline expression µ is set to be 0.01; the parameter δ i is assumed to be 2 for all transcripts (i = ) and finally two values of the inverse dispersion parameters φ i = 0.5 and φ = 5 are used to model the high and low dispersed measurements respectively. The simulation proceeds as follows: first the optimal stationary policy is found based on Algorithm 2 with the threshold parameter β = 10 6 and γ = Then an initial state and corresponding observed RNA-seq measurement are generated. Based on the estimated state by the BKF and the optimal stationary policy obtained initially the appropriate control function is applied to the system. Based on the current intervention the next data point is generated. The process continues in an online closed loop model. The sum of the state values of CycD and Rb genes for the system under control of Rb gene and without control are presented in Figure 2 over 80 time steps. It can be seen that the simultaneous inactivation of both CycD and Rb genes occurred several times for the uncontrolled system; however this undesirable condition (simultaneous inactivation of both Rb and CycD genes) is visited less often in the system under control of the Rb gene. Furthermore better control can be seen in the presence of low dispersed data. The reason is that the performance of state estimation is worse for high dispersed data leading to lower control performance as well. Next the results of the proposed method under control of the Rb gene observed through simulated RNA-seq data are compared with the results of the value iteration method with directly observed states. The fractions of observed desirable states over 5000 time steps for data with high and low dispersion and different process noise are shown in Figure 3. It should be noted that since the stationary policy is dependent on the process noise different policies are obtained for each value of the process noise. As we expected the value iteration method with directly observed states has higher performance than the system with partially-observed states. In addition the fractions of observed desirable states for systems observed through low dispersion data are very close to the value iteration method; however in data with high dispersion the accuracy of state estimation by BKF is Under Control of Rb Time a. Low dispersed data ( ) Without Control Time b. High dispersed data ( ) Fig. 2: Sum of state values of CycD and Rb genes over 80 time steps for system under control of the Rb gene and without control. smaller which degrades control performance. Fig. 3: Fraction of observed desirable states for the system with directly observed states (VI) the proposed method with observations through RNA-seq data (VI- BKF) and system without control (Low Dispersion: φ = 5 High Dispersion φ = 0.5). The fraction of observed desirable states using different genes as control input over 5000 time steps is shown in Figure 4. It is clear that the Rb gene is the best control gene for this system. 5
6 Fraction of observed desirable states in long run CycD Rb E2F CycE CycA Cdc20 Cdh1 UbcH10 CycB No control Fig. 4: Fraction of desirable states using different control genes for process noise p = 0.01 and φ j = 5 for j = VII. CONCLUSION In this paper an intervention strategy for partiallyobserved Boolean dynamical systems is discussed. The method contains two main steps: in the first step the optimal control policy for the system with directly observed states is obtained and in the second step the obtained control policy is employed to the system based on the estimated states by the Boolean Kalman Filter. The performance of the method is evaluated using a Boolean network model of the mutated mammalian cell cycle and simulated RNA-seq observations. ACKNOWLEDGMENT The authors acknowledge the support of the National Science Foundation through NSF award CCF REFERENCES [1] A. Datta A. Choudhary M. L. Bittner and E. R. Dougherty External control in markovian genetic regulatory networks Machine learning vol. 52 no. 1-2 pp [2] S. A. Kauffman Metabolic stability and epigenesis in randomly constructed genetic nets Journal of theoretical biology vol. 22 no. 3 pp [3] I. Shmulevich E. R. Dougherty and W. Zhang From boolean to probabilistic boolean networks as models of genetic regulatory networks Proceedings of the IEEE vol. 90 no. 11 pp [4] U. Braga-Neto Optimal state estimation for boolean dynamical systems in Signals Systems and Computers (ASILOMAR) 2011 Conference Record of the Forty Fifth Asilomar Conference on pp IEEE [5] E. O. Voit and J. Almeida Decoupling dynamical systems for pathway identification from metabolic profiles Bioinformatics vol. 20 no. 11 pp [6] N. Friedman M. Linial I. Nachman and D. Pe er Using bayesian networks to analyze expression data Journal of computational biology vol. 7 no. 3-4 pp [7] R. Pal A. Datta and E. R. Dougherty Optimal infinite-horizon control for probabilistic boolean networks Signal Processing IEEE Transactions on vol. 54 no. 6 pp [8] Á. Halász M. S. Sakar H. Rubin V. Kumar G. J. Pappas et al. Stochastic modeling and control of biological systems: the lactose regulation system of escherichia coli Automatic Control IEEE Transactions on vol. 53 no. Special Issue pp [9] G. Vahedi B. Faryabi J.-F. Chamberland A. Datta and E. R. Dougherty Optimal intervention strategies for cyclic therapeutic methods Biomedical Engineering IEEE Transactions on vol. 56 no. 2 pp [10] M. Imani and U. Braga-Neto Optimal state estimation for boolean dynamical systems using a boolean kalman smoother in 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP) pp IEEE [11] M. Imani and U. Braga-Neto Optimal gene regulatory network inference using the boolean kalman filter and multiple model adaptive estimation in th Asilomar Conference on Signals Systems and Computers pp IEEE [12] A. Bahadorinejad and U. Braga-Neto Optimal fault detection and diagnosis in transcriptional circuits using next-generation sequencing IEEE/ACM Transactions on Computational Biology and Bioinformatics [13] D. P. Bertsekas D. P. Bertsekas D. P. Bertsekas and D. P. Bertsekas Dynamic programming and optimal control vol. 1. Athena Scientific Belmont MA [14] M. R. Yousefi A. Datta and E. R. Dougherty Optimal intervention strategies for therapeutic methods with fixed-length duration of drug effectiveness Signal Processing IEEE Transactions on vol. 60 no. 9 pp
Optimal State Estimation for Boolean Dynamical Systems using a Boolean Kalman Smoother
Optimal State Estimation for Boolean Dynamical Systems using a Boolean Kalman Smoother Mahdi Imani and Ulisses Braga-Neto Department of Electrical and Computer Engineering Texas A&M University College
More informationMultiple Model Adaptive Controller for Partially-Observed Boolean Dynamical Systems
Multiple Model Adaptive Controller for Partially-Observed Boolean Dynamical Systems Mahdi Imani and Ulisses Braga-Neto Abstract This paper is concerned with developing an adaptive controller for Partially-Observed
More informationIEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 2, FEBRUARY X/$ IEEE
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL 58, NO 2, FEBRUARY 2010 879 Context-Sensitive Probabilistic Boolean Networks: Steady-State Properties, Reduction, and Steady-State Approximation Ranadip Pal,
More informationBIOINFORMATICS. Adaptive Intervention in Probabilistic Boolean Networks. Ritwik Layek 1, Aniruddha Datta 1, Ranadip Pal 2, Edward R.
BIOINFORMATICS Vol. 00 no. 00 Pages 1 8 Adaptive Intervention in Probabilistic Boolean Networks Ritwik Layek 1, Aniruddha Datta 1, Ranadip Pal 2, Edward R. Dougherty 1,3 1 Texas A & M University, Electrical
More informationIndex. FOURTH PROOFS n98-book 2009/11/4 page 261
FOURTH PROOFS n98-book 2009/11/4 page 261 Index activity, 10, 13 adaptive control, 215 adaptive controller, 215 adaptive immune system, 20 adaptive intervention strategy, 216 adjacency rule, 111 admissible
More informationOptimal control of Boolean control networks
1 Optimal control of Boolean control networks Ettore Fornasini and Maria Elena Valcher Abstract In this paper we address the optimal control problem for Boolean control networks (BCNs). We first consider
More informationGLOBEX Bioinformatics (Summer 2015) Genetic networks and gene expression data
GLOBEX Bioinformatics (Summer 2015) Genetic networks and gene expression data 1 Gene Networks Definition: A gene network is a set of molecular components, such as genes and proteins, and interactions between
More informationLecture Network analysis for biological systems
Lecture 11 2014 Network analysis for biological systems Anja Bråthen Kristoffersen Biological Networks Gene regulatory network: two genes are connected if the expression of one gene modulates expression
More informationSYSTEMS MEDICINE: AN INTEGRATED APPROACH WITH DECISION MAKING PERSPECTIVE. A Dissertation BABAK FARYABI
SYSTEMS MEDICINE: AN INTEGRATED APPROACH WITH DECISION MAKING PERSPECTIVE A Dissertation by BABAK FARYABI Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the
More informationExternal Control in Markovian Genetic Regulatory Networks: The Imperfect Information Case
Bioinformatics Advance Access published January 29, 24 External Control in Markovian Genetic Regulatory Networks: The Imperfect Information Case Aniruddha Datta Ashish Choudhary Michael L. Bittner Edward
More informationTemporal-Difference Q-learning in Active Fault Diagnosis
Temporal-Difference Q-learning in Active Fault Diagnosis Jan Škach 1 Ivo Punčochář 1 Frank L. Lewis 2 1 Identification and Decision Making Research Group (IDM) European Centre of Excellence - NTIS University
More informationCompletion Time of Fuzzy GERT-type Networks with Loops
Completion Time of Fuzzy GERT-type Networks with Loops Sina Ghaffari 1, Seyed Saeid Hashemin 2 1 Department of Industrial Engineering, Ardabil Branch, Islamic Azad university, Ardabil, Iran 2 Department
More informationIntroduction to Bioinformatics
Systems biology Introduction to Bioinformatics Systems biology: modeling biological p Study of whole biological systems p Wholeness : Organization of dynamic interactions Different behaviour of the individual
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2014
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2014 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationCONTROL OF STATIONARY BEHAVIOR IN PROBABILISTIC BOOLEAN NETWORKS BY MEANS OF STRUCTURAL INTERVENTION
Journal of Biological Systems, Vol. 0, No. 4 (2002) 43 445 c World Scientific Publishing Company CONTROL OF STATIONARY BEHAVIOR IN PROBABILISTIC BOOLEAN NETWORKS BY MEANS OF STRUCTURAL INTERVENTION ILYA
More informationBasic modeling approaches for biological systems. Mahesh Bule
Basic modeling approaches for biological systems Mahesh Bule The hierarchy of life from atoms to living organisms Modeling biological processes often requires accounting for action and feedback involving
More informationInput-State Incidence Matrix of Boolean Control Networks and Its Applications
Input-State Incidence Matrix of Boolean Control Networks and Its Applications Yin Zhao, Hongsheng Qi, Daizhan Cheng The Key Laboratory of Systems & Control Institute of Systems Science, Academy of Mathematics
More informationA linear control model for gene intervention in a genetic regulatory network
Title A linear control model for gene intervention in a genetic regulatory network Author(s) Zhang, SQ; Ng, MK; Ching, WK; Akutsu, T Citation 2005 Ieee International Conference On Granular Computing, 2005,
More informationDecision Theory: Markov Decision Processes
Decision Theory: Markov Decision Processes CPSC 322 Lecture 33 March 31, 2006 Textbook 12.5 Decision Theory: Markov Decision Processes CPSC 322 Lecture 33, Slide 1 Lecture Overview Recap Rewards and Policies
More informationA Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem
A Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem Kangkang Deng, Zheng Peng Abstract: The main task of genetic regulatory networks is to construct a
More information9 Multi-Model State Estimation
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 9 Multi-Model State
More informationNetworks in systems biology
Networks in systems biology Matthew Macauley Department of Mathematical Sciences Clemson University http://www.math.clemson.edu/~macaule/ Math 4500, Spring 2017 M. Macauley (Clemson) Networks in systems
More informationComputational Genomics. Systems biology. Putting it together: Data integration using graphical models
02-710 Computational Genomics Systems biology Putting it together: Data integration using graphical models High throughput data So far in this class we discussed several different types of high throughput
More informationProcedia Computer Science 00 (2011) 000 6
Procedia Computer Science (211) 6 Procedia Computer Science Complex Adaptive Systems, Volume 1 Cihan H. Dagli, Editor in Chief Conference Organized by Missouri University of Science and Technology 211-
More informationAn Empirical Algorithm for Relative Value Iteration for Average-cost MDPs
2015 IEEE 54th Annual Conference on Decision and Control CDC December 15-18, 2015. Osaka, Japan An Empirical Algorithm for Relative Value Iteration for Average-cost MDPs Abhishek Gupta Rahul Jain Peter
More informationElements of Reinforcement Learning
Elements of Reinforcement Learning Policy: way learning algorithm behaves (mapping from state to action) Reward function: Mapping of state action pair to reward or cost Value function: long term reward,
More informationReinforcement Learning. Introduction
Reinforcement Learning Introduction Reinforcement Learning Agent interacts and learns from a stochastic environment Science of sequential decision making Many faces of reinforcement learning Optimal control
More informationReinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
More informationA New Method to Build Gene Regulation Network Based on Fuzzy Hierarchical Clustering Methods
International Academic Institute for Science and Technology International Academic Journal of Science and Engineering Vol. 3, No. 6, 2016, pp. 169-176. ISSN 2454-3896 International Academic Journal of
More informationOn the Convergence of Optimistic Policy Iteration
Journal of Machine Learning Research 3 (2002) 59 72 Submitted 10/01; Published 7/02 On the Convergence of Optimistic Policy Iteration John N. Tsitsiklis LIDS, Room 35-209 Massachusetts Institute of Technology
More informationOn Finding Optimal Policies for Markovian Decision Processes Using Simulation
On Finding Optimal Policies for Markovian Decision Processes Using Simulation Apostolos N. Burnetas Case Western Reserve University Michael N. Katehakis Rutgers University February 1995 Abstract A simulation
More informationProbabilistic reconstruction of the tumor progression process in gene regulatory networks in the presence of uncertainty
Probabilistic reconstruction of the tumor progression process in gene regulatory networks in the presence of uncertainty Mohammad Shahrokh Esfahani, Byung-Jun Yoon, Edward R. Dougherty,2 Department of
More informationIntroduction to Bioinformatics
CSCI8980: Applied Machine Learning in Computational Biology Introduction to Bioinformatics Rui Kuang Department of Computer Science and Engineering University of Minnesota kuang@cs.umn.edu History of Bioinformatics
More informationEpistemology and the Role of Mathematics in Translational Science
Epistemology and the Role of Mathematics in Translational Science Edward R. Dougherty Abstract The terminology translational science has recently become very popular; however, there has been little effort
More informationReinforcement learning
Reinforcement learning Based on [Kaelbling et al., 1996, Bertsekas, 2000] Bert Kappen Reinforcement learning Reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error
More informationSynchronous state transition graph
Heike Siebert, FU Berlin, Molecular Networks WS10/11 2-1 Synchronous state transition graph (0, 2) (1, 2) vertex set X (state space) edges (x,f(x)) every state has only one successor attractors are fixed
More informationMDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
More informationDESIGN OF EXPERIMENTS AND BIOCHEMICAL NETWORK INFERENCE
DESIGN OF EXPERIMENTS AND BIOCHEMICAL NETWORK INFERENCE REINHARD LAUBENBACHER AND BRANDILYN STIGLER Abstract. Design of experiments is a branch of statistics that aims to identify efficient procedures
More informationReinforcement Learning and Optimal Control. ASU, CSE 691, Winter 2019
Reinforcement Learning and Optimal Control ASU, CSE 691, Winter 2019 Dimitri P. Bertsekas dimitrib@mit.edu Lecture 8 Bertsekas Reinforcement Learning 1 / 21 Outline 1 Review of Infinite Horizon Problems
More informationLearning in Bayesian Networks
Learning in Bayesian Networks Florian Markowetz Max-Planck-Institute for Molecular Genetics Computational Molecular Biology Berlin Berlin: 20.06.2002 1 Overview 1. Bayesian Networks Stochastic Networks
More informationAbstract Dynamic Programming
Abstract Dynamic Programming Dimitri P. Bertsekas Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Overview of the Research Monograph Abstract Dynamic Programming"
More informationarxiv: v1 [q-bio.mn] 7 Nov 2018
Role of self-loop in cell-cycle network of budding yeast Shu-ichi Kinoshita a, Hiroaki S. Yamada b a Department of Mathematical Engineering, Faculty of Engeneering, Musashino University, -- Ariake Koutou-ku,
More information5.3 METABOLIC NETWORKS 193. P (x i P a (x i )) (5.30) i=1
5.3 METABOLIC NETWORKS 193 5.3 Metabolic Networks 5.4 Bayesian Networks Let G = (V, E) be a directed acyclic graph. We assume that the vertices i V (1 i n) represent for example genes and correspond to
More information6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
More informationIntrinsic Noise in Nonlinear Gene Regulation Inference
Intrinsic Noise in Nonlinear Gene Regulation Inference Chao Du Department of Statistics, University of Virginia Joint Work with Wing H. Wong, Department of Statistics, Stanford University Transcription
More information56:198:582 Biological Networks Lecture 9
56:198:582 Biological Networks Lecture 9 The Feed-Forward Loop Network Motif Subgraphs in random networks We have discussed the simplest network motif, self-regulation, a pattern with one node We now consider
More informationValue and Policy Iteration
Chapter 7 Value and Policy Iteration 1 For infinite horizon problems, we need to replace our basic computational tool, the DP algorithm, which we used to compute the optimal cost and policy for finite
More information6.231 DYNAMIC PROGRAMMING LECTURE 6 LECTURE OUTLINE
6.231 DYNAMIC PROGRAMMING LECTURE 6 LECTURE OUTLINE Review of Q-factors and Bellman equations for Q-factors VI and PI for Q-factors Q-learning - Combination of VI and sampling Q-learning and cost function
More informationOptimal control and estimation
Automatic Control 2 Optimal control and estimation Prof. Alberto Bemporad University of Trento Academic year 2010-2011 Prof. Alberto Bemporad (University of Trento) Automatic Control 2 Academic year 2010-2011
More informationDNA regulatory circuits can be often described by
SUBMITTED TO IEEE/ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS 1 Statistical Detection of Boolean Regulatory Relationships Ting Chen, Student Member, IEEE, and Ulisses M. Braga-Neto, Senior
More informationCS 7180: Behavioral Modeling and Decisionmaking
CS 7180: Behavioral Modeling and Decisionmaking in AI Markov Decision Processes for Complex Decisionmaking Prof. Amy Sliva October 17, 2012 Decisions are nondeterministic In many situations, behavior and
More informationL06. LINEAR KALMAN FILTERS. NA568 Mobile Robotics: Methods & Algorithms
L06. LINEAR KALMAN FILTERS NA568 Mobile Robotics: Methods & Algorithms 2 PS2 is out! Landmark-based Localization: EKF, UKF, PF Today s Lecture Minimum Mean Square Error (MMSE) Linear Kalman Filter Gaussian
More informationAn Adaptive Clustering Method for Model-free Reinforcement Learning
An Adaptive Clustering Method for Model-free Reinforcement Learning Andreas Matt and Georg Regensburger Institute of Mathematics University of Innsbruck, Austria {andreas.matt, georg.regensburger}@uibk.ac.at
More information21 Markov Decision Processes
2 Markov Decision Processes Chapter 6 introduced Markov chains and their analysis. Most of the chapter was devoted to discrete time Markov chains, i.e., Markov chains that are observed only at discrete
More information1 Problem Formulation
Book Review Self-Learning Control of Finite Markov Chains by A. S. Poznyak, K. Najim, and E. Gómez-Ramírez Review by Benjamin Van Roy This book presents a collection of work on algorithms for learning
More informationDecision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
More informationApproximate active fault detection and control
Approximate active fault detection and control Jan Škach Ivo Punčochář Miroslav Šimandl Department of Cybernetics Faculty of Applied Sciences University of West Bohemia Pilsen, Czech Republic 11th European
More informationStochastic Shortest Path Problems
Chapter 8 Stochastic Shortest Path Problems 1 In this chapter, we study a stochastic version of the shortest path problem of chapter 2, where only probabilities of transitions along different arcs can
More informationOptimal Perturbation Control of General Topology Molecular Networks
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 7, APRIL 1, 2013 1733 Optimal Perturbation Control of General Topology Molecular Networks Nidhal Bouaynaya, Member, IEEE, Roman Shterenberg, and Dan
More informationLecture 3: Mixture Models for Microbiome data. Lecture 3: Mixture Models for Microbiome data
Lecture 3: Mixture Models for Microbiome data 1 Lecture 3: Mixture Models for Microbiome data Outline: - Mixture Models (Negative Binomial) - DESeq2 / Don t Rarefy. Ever. 2 Hypothesis Tests - reminder
More informationIntroduction to Approximate Dynamic Programming
Introduction to Approximate Dynamic Programming Dan Zhang Leeds School of Business University of Colorado at Boulder Dan Zhang, Spring 2012 Approximate Dynamic Programming 1 Key References Bertsekas, D.P.
More informationCase Studies of Logical Computation on Stochastic Bit Streams
Case Studies of Logical Computation on Stochastic Bit Streams Peng Li 1, Weikang Qian 2, David J. Lilja 1, Kia Bazargan 1, and Marc D. Riedel 1 1 Electrical and Computer Engineering, University of Minnesota,
More informationnutrients growth & division repellants movement
Network Dynamics and Cell Physiology John J. Tyson Department of Biological Sciences & Virginia Bioinformatics Institute Outline 1. Cell Signaling: Physiology 2. Cell Signaling: Molecular Biology 3. Chemical
More informationBayesian Control of Large MDPs with Unknown Dynamics in Data-Poor Environments
Bayesian Control of Large MDPs with Unknown Dynamics in Data-Poor Environments Mahdi Imani Texas A&M University College Station, TX, USA m.imani88@tamu.edu Seyede Fatemeh Ghoreishi Texas A&M University
More informationIntroduction Probabilistic Programming ProPPA Inference Results Conclusions. Embedding Machine Learning in Stochastic Process Algebra.
Embedding Machine Learning in Stochastic Process Algebra Jane Hillston Joint work with Anastasis Georgoulas and Guido Sanguinetti, School of Informatics, University of Edinburgh 16th August 2017 quan col....
More informationKalman filtering and friends: Inference in time series models. Herke van Hoof slides mostly by Michael Rubinstein
Kalman filtering and friends: Inference in time series models Herke van Hoof slides mostly by Michael Rubinstein Problem overview Goal Estimate most probable state at time k using measurement up to time
More informationLecture notes for Analysis of Algorithms : Markov decision processes
Lecture notes for Analysis of Algorithms : Markov decision processes Lecturer: Thomas Dueholm Hansen June 6, 013 Abstract We give an introduction to infinite-horizon Markov decision processes (MDPs) with
More informationA Tour of Reinforcement Learning The View from Continuous Control. Benjamin Recht University of California, Berkeley
A Tour of Reinforcement Learning The View from Continuous Control Benjamin Recht University of California, Berkeley trustable, scalable, predictable Control Theory! Reinforcement Learning is the study
More informationChapter 15 Active Reading Guide Regulation of Gene Expression
Name: AP Biology Mr. Croft Chapter 15 Active Reading Guide Regulation of Gene Expression The overview for Chapter 15 introduces the idea that while all cells of an organism have all genes in the genome,
More informationReal Time Value Iteration and the State-Action Value Function
MS&E338 Reinforcement Learning Lecture 3-4/9/18 Real Time Value Iteration and the State-Action Value Function Lecturer: Ben Van Roy Scribe: Apoorva Sharma and Tong Mu 1 Review Last time we left off discussing
More informationPrioritized Sweeping Converges to the Optimal Value Function
Technical Report DCS-TR-631 Prioritized Sweeping Converges to the Optimal Value Function Lihong Li and Michael L. Littman {lihong,mlittman}@cs.rutgers.edu RL 3 Laboratory Department of Computer Science
More informationDS-GA 1002 Lecture notes 11 Fall Bayesian statistics
DS-GA 100 Lecture notes 11 Fall 016 Bayesian statistics In the frequentist paradigm we model the data as realizations from a distribution that depends on deterministic parameters. In contrast, in Bayesian
More informationCan evolution paths be explained by chance alone? arxiv: v1 [math.pr] 12 Oct The model
Can evolution paths be explained by chance alone? Rinaldo B. Schinazi Department of Mathematics University of Colorado at Colorado Springs rinaldo.schinazi@uccs.edu arxiv:1810.05705v1 [math.pr] 12 Oct
More informationA6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2015 http://www.astro.cornell.edu/~cordes/a6523 Lecture 23:! Nonlinear least squares!! Notes Modeling2015.pdf on course
More informationChapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS
Chapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS 63 2.1 Introduction In this chapter we describe the analytical tools used in this thesis. They are Markov Decision Processes(MDP), Markov Renewal process
More information2. Mathematical descriptions. (i) the master equation (ii) Langevin theory. 3. Single cell measurements
1. Why stochastic?. Mathematical descriptions (i) the master equation (ii) Langevin theory 3. Single cell measurements 4. Consequences Any chemical reaction is stochastic. k P d φ dp dt = k d P deterministic
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More informationLecture 7: Simple genetic circuits I
Lecture 7: Simple genetic circuits I Paul C Bressloff (Fall 2018) 7.1 Transcription and translation In Fig. 20 we show the two main stages in the expression of a single gene according to the central dogma.
More informationCS 4649/7649 Robot Intelligence: Planning
CS 4649/7649 Robot Intelligence: Planning Probability Primer Sungmoon Joo School of Interactive Computing College of Computing Georgia Institute of Technology S. Joo (sungmoon.joo@cc.gatech.edu) 1 *Slides
More informationParameter estimation using simulated annealing for S- system models of biochemical networks. Orland Gonzalez
Parameter estimation using simulated annealing for S- system models of biochemical networks Orland Gonzalez Outline S-systems quick review Definition of the problem Simulated annealing Perturbation function
More informationInternet Monetization
Internet Monetization March May, 2013 Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition
More informationThe Kalman Filter ImPr Talk
The Kalman Filter ImPr Talk Ged Ridgway Centre for Medical Image Computing November, 2006 Outline What is the Kalman Filter? State Space Models Kalman Filter Overview Bayesian Updating of Estimates Kalman
More informationSig2GRN: A Software Tool Linking Signaling Pathway with Gene Regulatory Network for Dynamic Simulation
Sig2GRN: A Software Tool Linking Signaling Pathway with Gene Regulatory Network for Dynamic Simulation Authors: Fan Zhang, Runsheng Liu and Jie Zheng Presented by: Fan Wu School of Computer Science and
More informationMARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
More informationHub Gene Selection Methods for the Reconstruction of Transcription Networks
for the Reconstruction of Transcription Networks José Miguel Hernández-Lobato (1) and Tjeerd. M. H. Dijkstra (2) (1) Computer Science Department, Universidad Autónoma de Madrid, Spain (2) Institute for
More informationMathematical Biology - Lecture 1 - general formulation
Mathematical Biology - Lecture 1 - general formulation course description Learning Outcomes This course is aimed to be accessible both to masters students of biology who have a good understanding of the
More informationModels of Molecular Evolution
Models of Molecular Evolution Bret Larget Departments of Botany and of Statistics University of Wisconsin Madison September 15, 2007 Genetics 875 (Fall 2009) Molecular Evolution September 15, 2009 1 /
More informationASIGNIFICANT research effort has been devoted to the. Optimal State Estimation for Stochastic Systems: An Information Theoretic Approach
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 42, NO 6, JUNE 1997 771 Optimal State Estimation for Stochastic Systems: An Information Theoretic Approach Xiangbo Feng, Kenneth A Loparo, Senior Member, IEEE,
More informationReinforcement Learning
CS7/CS7 Fall 005 Supervised Learning: Training examples: (x,y) Direct feedback y for each input x Sequence of decisions with eventual feedback No teacher that critiques individual actions Learn to act
More informationBasics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
More informationAlgorithmisches Lernen/Machine Learning
Algorithmisches Lernen/Machine Learning Part 1: Stefan Wermter Introduction Connectionist Learning (e.g. Neural Networks) Decision-Trees, Genetic Algorithms Part 2: Norman Hendrich Support-Vector Machines
More informationCSEP 573: Artificial Intelligence
CSEP 573: Artificial Intelligence Hidden Markov Models Luke Zettlemoyer Many slides over the course adapted from either Dan Klein, Stuart Russell, Andrew Moore, Ali Farhadi, or Dan Weld 1 Outline Probabilistic
More informationMeasures for information propagation in Boolean networks
Physica D 227 (2007) 100 104 www.elsevier.com/locate/physd Measures for information propagation in Boolean networks Pauli Rämö a,, Stuart Kauffman b, Juha Kesseli a, Olli Yli-Harja a a Institute of Signal
More informationRegulation of Gene Expression
Chapter 18 Regulation of Gene Expression PowerPoint Lecture Presentations for Biology Eighth Edition Neil Campbell and Jane Reece Lectures by Chris Romero, updated by Erin Barley with contributions from
More informationNetwork Dynamics and Cell Physiology. John J. Tyson Department of Biological Sciences & Virginia Bioinformatics Institute
Network Dynamics and Cell Physiology John J. Tyson Department of Biological Sciences & Virginia Bioinformatics Institute Signal Transduction Network Hanahan & Weinberg (2000) Gene Expression Signal-Response
More informationSensitivity analysis of biological Boolean networks using information fusion based on nonadditive set functions
University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Biochemistry -- Faculty Publications Biochemistry, Department of 9-2014 Sensitivity analysis of biological Boolean networks
More informationComputation and Dynamic Programming
Computation and Dynamic Programming Huseyin Topaloglu School of Operations Research and Information Engineering, Cornell University, Ithaca, New York 14853, USA topaloglu@orie.cornell.edu June 25, 2010
More informationarxiv: v1 [cs.sy] 25 Oct 2017
Reconstruct the Logical Network from the Transition Matrix Cailu Wang, Yuegang Tao School of Control Science and Engineering, Hebei University of Technology, Tianjin, 300130, P. R. China arxiv:1710.09681v1
More informationBAYESIAN ESTIMATION OF UNKNOWN PARAMETERS OVER NETWORKS
BAYESIAN ESTIMATION OF UNKNOWN PARAMETERS OVER NETWORKS Petar M. Djurić Dept. of Electrical & Computer Engineering Stony Brook University Stony Brook, NY 11794, USA e-mail: petar.djuric@stonybrook.edu
More information