Multiple Model Adaptive Controller for Partially-Observed Boolean Dynamical Systems

Transcription

1 Multiple Model Adaptive Controller for Partially-Observed Boolean Dynamical Systems Mahdi Imani and Ulisses Braga-Neto Abstract This paper is concerned with developing an adaptive controller for Partially-Observed Boolean Dynamical Systems (POBDS). Assuming that partial nowledge about the system can be modeled by a finite number of candidate models, then simultaneous identification and control of a POBDS is achieved using the combination of a state-feedbac controller and a Multiple-Model Adaptive Estimation (MMAE) technique. The proposed method contains two main steps: first, in the offline step, the stationary control policy for the underlying Boolean dynamical system is computed for each candidate model. Then, in the online step, an optimal Bayesian estimator is modeled using a ban of Boolean Kalman Filters (BKFs), each tuned to a candidate model. The result of the offline step along with the estimated state by the ban of BKFs specify the control input that should be applied at each time point. The performance of the proposed adaptive controller is investigated using a Boolean networ model constructed from melanoma gene expression data observed through RNA-seq measurements. I. INTRODUCTION The partially-observed Boolean dynamical system (POBDS) model provides a rich framewor for modeling and control of systems containing Boolean state variables observing through noisy measurements. Examples of applications of systems with Boolean states abound, including gene-regulatory networs [1], [2], robotics [3], digital communication systems [4], and more. Several tools for this signal model have been developed in recent years, such as the optimal filter and smoother based on the minimum mean-square error (MMSE) criterion, which are called the Boolean Kalman Filter [5] and Boolean Kalman Smoother [6], respectively. In addition, particle filtering implementations of these filters, as well as schemes for handling correlated Boolean noise, simultaneous state and parameter estimation, networ inference, and fault detection for POBDSs were developed in [7] [11]. Furthermore, the software tool BoolFilter [12] is freely available under R library for estimation and inference of partially-observed Boolean dynamical systems. Decision maing under various sources of uncertainty is an issue of great interests in various fields [13] [18]. Unlie various intervention approaches [19] [22] that have been developed in the context of Probabilistic Boolean Networs (PBNs) [23], S-systems [24], and Bayesian networs [25], which assume that the Boolean states of the system are *The authors acnowledge the support of the National Science Foundation, through NSF award CCF M. Imani and U. M. Braga-Neto are with the Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA m.imani88@tamu.edu, ulisses@ece.tamu.edu directly observable, state and output feedbac controllers are designed in [26] [28] to deal with partially-observed Boolean dynamical systems. The output-feedbac controller for POBDS in [27] was developed based on the well-nown point-based value iteration (PBVI) method. This method can only be applied to a POBDS with a finite measurement space. On the other hand, the state-feedbac controller for POBDS in [26] was designed based on optimal infinite horizon control of the underlying Boolean dynamical system using the BKF as state observer and, unlie PBVI for POBDS, it can be applied to any arbitrary measurement space. Notice that both these controllers require full nowledge about the POBDS. The goal of this paper is to obtain a controller when only partial information is available about the POBDS. For example, the Boolean Networ topology (e.g., the connection between nodes) may be incompletely nown, or noise and other measurement parameters may be liewise unavailable. We build our multiple model adaptive controller based on the state feedbac controller [26] and multiple model adaptive estimator [10], assuming that the unnown system parameters are contained in a finite set. First, in an offline step, the stationary control policy for each candidate is computed and then, a ban of BKFs running in parallel, each tuned to a candidate model, provides a fully adaptive estimate of both state and parameters. The result of the offline step along with the estimated state by the ban of BKFs specify the control input that should be applied at each time point. Performance is investigated using a melanoma regulatory networ and simulated RNA-seq measurements. II. PARTIALLY-OBSERVED BOOLEAN DYNAMICAL SYSTEMS We describe below the partially-observed Boolean dynamical systems (POBDS) model, first proposed in [5]. We assume that the system is described by a state process {X ; = 0, 1,...}, where X {0, 1} d is a Boolean vector of size d in the case of a gene regulatory networ, the components of X represent the activation/inactivation state of the genes at time. The state is affected by a sequence of control inputs {u ; = 0, 1,...}, where u U represents a purposeful intervention into the system state in the biological example, this might model drug applications. The sequence of states is observed indirectly through the observation process {Y ; = 1, 2,...}, where Y is a vector of (typically non-boolean) measurements. The states are assumed to be updated and observed at each time through

2 the following nonlinear signal model: X = f (X 1, u 1 ) n Y = h (X, v ) (state model) (observation model) for = 1, 2,..., where f {0, 1} d U {0, 1} d is a networ function, {n ; = 1, 2,...} is a white state noise process with n {0, 1} d, and indicates componentwise modulo-2 addition. The noise is white in the sense that n and n l are independent for l. In addition, the noise process is assumed to be independent from the state process and control input. A. Boolean Kalman Filter The optimal filtering problem consists of, given observations Y 1 = (Y 1,..., Y ) and control input u 0 1 = (u 0,..., u 1 ), finding an estimator ˆX = h(y 1, u 0 1 ) of the state X that minimizes the conditional mean-square error (MSE): MSE( ˆX Y 1, u 0 1 ) = E [ ˆX X 2 Y 1, u 0 1 ], at each time step 1 to. For a vector v of size d, define v 1 = d i=1 v(i), v {0, 1} d via v(i) = I v(i)>1/2 for i = 1,..., d, v c {0, 1} d via v c (i) = 1 v(i), for i = 1,..., d; where I v(i)>1/2 returns 1 if v(i) > 1/2 and 0 otherwise. The optimal MMSE filter ˆX is given by [5], [9] with optimal filtering MMSE (1) (2) ˆX MS = E [X Y 1, u 0 1 ], (3) MSE( ˆX MS Y 1, u 0 1 ) = min{e[x Y 1, u 0 1 ], E[X Y 1, u 0 1 ] c } 1, (4) where the minimum is computed componentwise. Both the optimal filter and its MSE can be computed by a recursive matrix-based procedure, called the Boolean Kalman Filter (BKF) [5], which is briefly described next. Let (x 1,..., x 2d ) be an arbitrary enumeration of the possible state vectors. Define the state conditional probability distribution vectors Π and Π 1 by Π (i) = P (X = x i Y 1, u 0 1 ), Π 1 (i) = P (X = x i Y 1 1, u 0 1 ), for i = 1,..., 2 d, and = 1, 2,.... We also define Π 0 0 to be the initial (prior) distribution of the states at time zero. Let the prediction matrix M (u) of size 2 d 2 d be the transition matrix of the controlled Marov chain under control input u defined by the state model: (M (u)) ij = P (X = x i X 1 = x j, u 1 = u) = P (n = f(x j, u) x i ), for i, j = 1,..., 2 d. Additionally, given a value of the observation vector Y at time, the update matrix T (Y ) (5) (6) of size 2 d 2 d is a diagonal matrix defined by: (T (Y )) ii = p (Y X = x i ), (7) for i = 1,..., 2 d, where p( ) denotes either a probability density function or a probability mass function, in the case of continuous or discrete measurements, respectively. Finally, define the matrix A of size d 2 d via A = [x 1 x 2d ]. It can be shown that the optimal MMSE estimator ˆX MS can be computed by Algorithm 1 [5], [9]. Algorithm 1 Boolean Kalman Filter 1: Initialization: (Π 0 0 ) i = P (X 0 = x i ), for i = 1,..., 2 d. For = 1, 2,..., do: 2: Prediction: Π 1 = M (u 1 ) Π 1 1 3: Update: β = T (Y ) Π 1 4: Filtered Distribution Vector: Π = β / β 1 5: MMSE Estimator Computation: ˆX MS with optimal conditional MSE = AΠ MSE( ˆX MS Y 1, u 0 1 ) = min{aπ,(aπ ) c } 1. III. STATE-FEEDBACK CONTROLLER FOR POBDS The state-feedbac controller for POBDS first introduced in [26] contains offline and online steps. In the offline step, the stationary control policy for the underlying Boolean dynamical system, with the assumption of direct observability of states, is computed, and then in the online step the designed control policy is applied to the system based on the estimated state by the BKF. The method is described briefly in the following paragraphs. The goal of infinite-horizon control of a Boolean dynamical system is to select the appropriate external input u U at each time to mae the system spend the least amount of time, on average, in undesirable states; e.g., states associated with cell proliferation in biological systems, which may be associated with cancer [20]. In formal terms, assuming a bounded cost of control g(x, u ), our goal is to find a stationary control policy µ {0, 1} d U, which minimizes the infinite-horizon cost (for a given initial state X 0 = x j 0 ): m J µ (j) = lim E γ m g (X, µ(x )), (8) =0 for j = 1,..., 2 d, where 0 < γ < 1 is a discounting factor that ensures that the limit of the finite sums converges as the horizon length m goes to infinity. We assume that the system prediction matrix M (u) can only depend on time through the control input u. We will

3 therefore drop the index and write simply M(u). Defining a mapping T as T [J](j) =min u U g(x j, u) + γ 2 d i=1 (M(u)) ij J(i), (9) the optimal stationary control policy can be obtained by starting with an arbitrary initial cost function J 0 {0, 1} d R, and running the following iteration J t = T [J t 1 ], (10) until a fixed point is obtained it can be shown that the iteration will indeed converge to a fixed point [30]. This fixed point is the optimal cost J R 2d, and the corresponding policy µ U 2d is the optimal stationary control policy. After obtaining the offline stationary control policy for the underlying BDS by using the value iteration method, the estimated state by the Boolean Kalman Filter is used for decision maing in the online process. If µ denotes the optimal stationary policy, then the control input at time is given simply by: u VBKF = µ ( ˆX MS ). (11) where ˆX MS is the estimated state at time obtained by the BKF given measurements Y 1 and sequence of control u 0 1, as described in the previous section. For more information, the reader is referred to [26]. IV. MULTIPLE MODEL ADAPTIVE CONTROLLER FOR POBDS The state-feedbac controller introduced in the previous section requires full nowledge about the POBDS. On the other hand, suppose that the nonlinear signal model in (1) is incompletely specified. For example, the deterministic functions f and h may be only partially nown, or the statistics of the noise processes n and v may need to be estimated. We assume that the missing information can be coded into a finite-dimensional parameter vector θ Θ, where Θ = {θ 1,..., θ M } is the parameter space. Control of such a partially-nown POBDS can be achieved by a combination of the state-feedbac controller introduced in section III and the multiple model adaptive estimation method in [10]. Similar to the state-feedbac controller, the proposed method contains offline and online steps. First, the stationary control policy for each candidate is computed (e.g. running M value iteration methods) in the offline step. Then, after obtaining stationary control policy µ θ i for all i = 1,..., M, an optimal Bayesian procedure is performed to select the candidate with the largest posterior probability given the latest information up to current time in the online process. The computation maes use of probabilities computed by a ban of BKFs running in parallel, one for each candidate model. This is similar to the multiple model adaptive estimation (MMAE) method for linear systems [31]. Given the prior probability p 0 (θ i ) of candidate model i, for i = 1,..., M, the posterior probability p (θ i ) at time, given the history of observations Y 1 and sequence of control inputs u 0 1, can be obtained as: p (θ i ) = P (θ = θ i Y 1, u 0 1 ) p(y θ = θ i, Y 1 1, u 0 1 ) p 1 (θ i ) = M l=1 p(y θ = θ l, Y 1 1, u 0 1 ) p 1 (θ l ). (12) Furthermore, one can write p(y θ = θ i, Y 1 1, u 0 1 ) = = 2 d p(y X = x j, θ = θ i ) j=1 P (X = x j θ = θ i, Y 1 1, u 0 1 ) 2 d (T θi (Y )) Π θi j=1 jj 1 (j) = T θi (Y ) Π θi 1 1 = β θi 1, (13) with β θi denoting the unnormalized PDV at time, computed at the update step of the BKF tuned to θ i. Combining equations (12) and (13), we obtain the update equation for the candidate model probabilities: β θi p (θ i ) = 1 p 1 (θ i ) M l=1 β θ l, for i = 1,..., M. (14) 1 p 1 (θ l ) Model selection at time can be then accomplished by a maximum a-posteriori criterion: ˆθ = argmax p (θ). (15) θ {θ 1,...,θ M } which leads to the estimate of state at time : ˆX MS ˆX MMAE = ˆX MS (ˆθ ), (16) where (θ) denotes the optimal MMSE state estimate produced by a BKF tuned to the parameter θ. Finally, the control input is the stationary control policy of the selected model in (15) applied on the MMAE estimate of state: u = µ ˆθ ( ˆX MMAE ). (17) A schematic diagram for the multiple model adaptive controller for POBDS is presented in Figure 1. The entire procedure is displayed in Algorithm 2. V. NUMERICAL EXPERIMENT In this section, we conduct a numerical experiment using a Boolean networ for metastatic melanoma [32]. The networ contains 7 genes: WNT5A, pirin, S100P, RET1, MART1, HADHB and STC2. The regulatory relationship for this networ is presented in Table I. For each gene, the output binary string specifies the output value for each value of the input gene(s). For example, the last row of Table I specifies the value of STC2 at current time step from different pairs of (pirin,stc2) values at previous time step 1: (pirin = 0, STC2 = 0) 1 STC2 = 1 (pirin = 0, STC2 = 1) 1 STC2 = 1 (pirin = 1, STC2 = 0) 1 STC2 = 0 (pirin = 1, STC2 = 1) 1 STC2 = 1

4 Offline BKF for Model 1 BKF for Model 2 Posterior Update Offline Step (Stationary Control Policies) Model Selection BKF for Model M System Fig. 1: Schematic diagram of multiple model adaptive controller for POBDS. TABLE I: Boolean functions for the melanoma Boolean networ using output binary string notation (see text). Algorithm 2 Multiple Model Adaptive Controller for POBDS 1: OFFLINE STEP 1) Compute the stationary control policy µ θ i, for all candidates i = 1,..., M by running M parallel value iteration methods. 2: ONLINE STEP 1) Initialization: Set prior distribution of each candidate p 0(θ i), i = 1,..., M. For = 1, 2,..., do: 2) Posterior Update: Using the outputs β θ i 1 of the ban of BKFs, for i = 1,..., M, update the posterior probability of each candidate as: p (θ i) = β θ i 1 p 1(θ i) M l=1 β θ l 1 p, for i = 1,..., M. 1(θ l ) 3) Parameter Estimation: The estimate of parameter θ at time is obtained as: ˆθ = argmax p (θ). θ {θ 1,...,θ M } 4) State Estimation: The MMAE estimate of the state at time is the estimated state by a BKF tuned to the estimated parameter ˆθ : ˆX MMAE = ˆX MS (ˆθ ) 5) Control: The control input is the stationary control policy of the selected model in (15) applied on the MMAE estimate of the state: u = µ ˆθ ( ˆX MMAE ). Genes Input Gene(s) Output WNT5A HADHB 10 pirin prin, RET1,HADHB S100P S100P,RET1,STC RET1 RET1,HADHB,STC MART1 pirin,mart1,stc HADHB pirin,s100p,ret STC2 pirin,stc The goal of control is preventing WNT5A gene to be upregulated. For more information about the biological rationale for this, the reader is referred to [32]. The control input U consists of {0, 1} in which u = 1 refers to flipping the state of RET1 and u = 0 refers no action. The cost function is defined as follows: 6 if WNT5A = 1, and u = 1 g(x j 5 if WNT5A = 1, and u = 0, u) = 1 if WNT5A = 0, and u = 1 0 if WNT5A = 0, and u = 0 for j = 1,..., 2 d. The process noise is assumed to have independent components identically distributed as Bernoulli with a small intensity p, so that all genes are perturbed with a small probability. We assume the Boolean states are observed through a single-lane RNA-sequencing experiment, in which each gene is assumed to be observed independently, so that Y (j) is the read count corresponding to transcript j in the single lane, for j = 1,..., 7. Assuming a negative binomial

5 Average Cost WNT5A Activations model for each count, we have: P (Y (j) = y(j) X (j) = x(j)) = λ j Γ(y(j) + φ j ) y(j)! Γ(φ j ) ( ) λ j + φ j y(j) φ j ( ) λ j + φ j φ j, (18) where Γ denotes the Gamma function, and φ j, λ j > 0 are the real-valued inverse dispersion parameter and mean read count of transcript j, respectively, for j = 1,..., 7. According to the Boolean state model, there are two possible states for the abundance of transcript j: high, if x(j) = 1, and low, if x(j) = 0. Accordingly, we model the parameter λ j in logspace as [9], [26]: log λ j = log s + µ + δ j x(j), (19) where the parameter s is the sequencing depth (which is instrument-dependent), µ > 0 is the baseline level of expression in the inactivated transcriptional state, and δ j > 0 expresses the effect on the observed RNA-seq read count as gene j goes from the inactivated to the activated state, for j = 1,..., 7. The parameters for the simulation are as follows. The discount factor γ is assumed to be The criterion for stopping the value iteration is: exit the iteration when max j=1,...,2 d( J (j) J(j) ) becomes less then 10 12, where J and J are cost values at consecutive iterations. The parameters of observation model are set to be s = , µ = 0.1, φ i = 2, δ i = 2, for i = 1,..., 7. We consider the intensity of process noise p to be unnown, and assume it can tae values 0.01, 0.05, or 0.1, which leads to three possible candidate models. Hence, first the stationary control policy for each of the three candidate models is computed and then three BKFs are run in parallel for the simultaneous estimation and control process (Figure 1). An uniform prior is assumed for all candidates (p 0 (θ i ) = 1/3, i = 1, 2, 3). The actual simulated trajectories are generated assuming p = Figure 2 displays the estimated model for a single trajectory under control of RET1 and without control. It is clear that the correct model is selected in both cases in less than 20 measurements. In addition, the activation status of the WNT5A gene under control of RET1 gene and without control are displayed in Figure 2 for 100 time steps. It can be seen that WNT5A is mostly upregulated in the system without control, which is undesirable, as opposed to the system under control by the proposed multiple model adaptive controller. Figure 3 displays the average cost (over 100 independent runs) under control of RET1 and without control. In all cases, the system started from rest (all genes were inactivated at time 0). It is clear that the system under control by the proposed multiple model adaptive controller has significantly lower cost on average than the system without control. VI. CONCLUSION In this paper, we proposed a multiple model adaptive controller for partially-observed Boolean dynamical systems, when the system is only partially nown. The proposed mod2[1:100] A[oX, 1] Under Control of RET Index Time Index Without Control Fig. 2: Estimated model and activation status of WNT5A with and without control. cost_no[1:100] Under Control of RET Time Index Without Control Fig. 3: Average cost with and without control. method is based on a state-feedbac controller and the multiple model adaptive estimation technique. The application of the proposed method is discussed in the context of the Boolean networ of melanoma observed through RNA-seq measurements. REFERENCES [1] S. A. Kauffman, Metabolic stability and epigenesis in randomly constructed genetic nets, Journal of theoretical biology, vol. 22, no. 3, pp , [2] A. Karbalayghareh, U. Braga-Neto, J. Hua, and E. R. Dougherty, Classification of state trajectories in gene regulatory networs, IEEE/ACM Transactions on Computational Biology and Bioinformatics, [3] A. Roli, M. Manfroni, C. Pinciroli, and M. Birattari, On the design of boolean networ robots, in Applications of Evolutionary Computation, pp , Springer, [4] D. G. Messerschmitt, Synchronization in digital system design, Selected Areas in Communications, IEEE Journal on, vol. 8, no. 8, pp , [5] U. Braga-Neto, Optimal state estimation for boolean dynamical systems, in Signals, Systems and Computers (ASILOMAR), 2011 Conference Record of the Forty Fifth Asilomar Conference on, pp , IEEE, [6] M. Imani and U. Braga-Neto, Optimal state estimation for boolean dynamical systems using a boolean alman smoother, in 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp , IEEE, [7] M. Imani and U. Braga-Neto, Particle filters for partially-observed boolean dynamical systems, arxiv preprint arxiv: , 2017.

6 [8] L. D. McClenny, M. Imani, and U. Braga-Neto, Boolean alman filter with correlated observation noise, in the 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), IEEE, [9] M. Imani and U. Braga-Neto, Maximum-lielihood adaptive filter for partially-observed boolean dynamical systems, IEEE Transactions on Signal Processing, vol. 65, no. 2, pp , [10] M. Imani and U. Braga-Neto, Optimal gene regulatory networ inference using the boolean alman filter and multiple model adaptive estimation, in th Asilomar Conference on Signals, Systems and Computers, pp , IEEE, [11] A. Bahadorinejad and U. Braga-Neto, Optimal fault detection and diagnosis in transcriptional circuits using next-generation sequencing, IEEE/ACM Transactions on Computational Biology and Bioinformatics, [12] L. D. McClenny, M. Imani, and U. Braga-Neto, Boolfilter pacage vignette, [13] S. F. Ghoreishi and D. L. Allaire, Compositional uncertainty analysis via importance weighted gibbs sampling for coupled multidisciplinary systems, in 18th AIAA Non-Deterministic Approaches Conference, p. 1443, [14] E. Nozari, F. Pasqualetti, and J. Cortes, Time-varying actuator scheduling in complex networs, arxiv preprint arxiv: , [15] S. Z. Dadaneh and X. Qian, Bayesian module identification from multiple noisy networs, EURASIP Journal on Bioinformatics and Systems Biology, vol. 2016, no. 1, p. 1, [16] A. Sarrafi and Z. Mao, Probabilistic uncertainty quantification of wavelet-transform-based structural health monitoring features, in SPIE Smart Structures and Materials+ Nondestructive Evaluation and Health Monitoring, pp N 98051N, International Society for Optics and Photonics, [17] S. F. Ghoreishi, Uncertainty analysis for coupled multidisciplinary systems using sequential importance resampling, Master s thesis, Texas A&M University, [18] A. Sarrafi and Z. Mao, Statistical modeling of wavelet-transformbased features in structural health monitoring, in Model Validation and Uncertainty Quantification, Volume 3, pp , Springer, [19] R. Laye, A. Datta, R. Pal, and E. R. Dougherty, Adaptive intervention in probabilistic boolean networs, Bioinformatics, vol. 25, no. 16, pp , [20] A. Datta, A. Choudhary, M. L. Bittner, and E. R. Dougherty, External control in marovian genetic regulatory networs, Machine learning, vol. 52, no. 1-2, pp , [21] R. Pal, A. Datta, and E. R. Dougherty, Optimal infinite-horizon control for probabilistic boolean networs, Signal Processing, IEEE Transactions on, vol. 54, no. 6, pp , [22] Á. Halász, M. S. Saar, H. Rubin, V. Kumar, G. J. Pappas, et al., Stochastic modeling and control of biological systems: the lactose regulation system of escherichia coli, Automatic Control, IEEE Transactions on, vol. 53, no. Special Issue, pp , [23] I. Shmulevich, E. R. Dougherty, and W. Zhang, From boolean to probabilistic boolean networs as models of genetic regulatory networs, Proceedings of the IEEE, vol. 90, no. 11, pp , [24] E. O. Voit and J. Almeida, Decoupling dynamical systems for pathway identification from metabolic profiles, Bioinformatics, vol. 20, no. 11, pp , [25] N. Friedman, M. Linial, I. Nachman, and D. Pe er, Using bayesian networs to analyze expression data, Journal of computational biology, vol. 7, no. 3-4, pp , [26] M. Imani and U. Braga-Neto, State-feedbac control of partiallyobserved boolean dynamical systems using rna-seq time series data, in American Control Conference (ACC), 2016, pp , IEEE, [27] M. Imani and U. Braga-Neto, Point-based value iteration for partiallyobserved boolean dynamical systems with finite observation space, in Decision and Control (CDC), 2016 IEEE 55th Conference on, pp , IEEE, [28] M. Imani and U. Braga-Neto, Control of gene regulatory networs with noisy measurements and uncertain inputs, arxiv preprint arxiv: , [29] S. Friedman, S. F. Ghoreishi, and D. L. Allaire, Quantifying the impact of different model discrepancy formulations in coupled multidisciplinary systems, in 19th AIAA Non-Deterministic Approaches Conference, p. 1950, [30] D. P. Bertseas, Dynamic programming and optimal control, vol. 1. Athena Scientific Belmont, MA, [31] P. S. Maybec and P. D. Hanlon, Performance enhancement of a multiple model adaptive estimator, Aerospace and Electronic Systems, IEEE Transactions on, vol. 31, no. 4, pp , [32] E. R. Dougherty, R. Pal, X. Qian, M. L. Bittner, and A. Datta, Stationary and structural control in gene regulatory networs: basic concepts, International Journal of Systems Science, vol. 41, no. 1, pp. 5 16, 2010.