IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 1

Transcription

1 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 1 Intervention in Gene Reulatory Networks via a Stationary Mean-First-Passae-Time Control Policy Golnaz Vahedi, Student Member, IEEE, Babak Faryabi, Student Member, IEEE, Jean-Francois Chamberland, Member, IEEE, Aniruddha Datta, Senior Member, IEEE, Edward R. Douherty, Member, IEEE Abstract A prime objective of modelin enetic reulatory networks is the identification of potential tarets for therapeutic intervention. To date, optimal stochastic intervention has been studied in the context of probabilistic Boolean networks, with the control policy based on the transition probability matrix of the associated Markov chain and dynamic prorammin used to find optimal control policies. Dynamical prorammin alorithms are problematic owin to their hih computational complexity. Two additional computationally burdensome issues that arise are the potential for controllin the network and identifyin the best ene for intervention. This paper proposes an alorithm based on mean first-passae time that assins a stationary control policy for each ene candidate. It serves as an approximation to an optimal control policy and, owin to its reduced computational complexity, can be used to predict the best control ene. Once the best control ene is identified, one can derive an optimal policy or simply utilize the approximate policy for this ene when the network size precludes a direct application of dynamic prorammin alorithms. A salient point is that the proposed alorithm can be model-free. It can be directly desined from time-course data without havin to infer the transition probability matrix of the network. Index Terms Probabilistic Boolean Networks, Genetic Reulatory Networks, Stochastic Optimal Control, Dynamic Prorammin, Mean First-Passae Time. I. INTRODUCTION AN ultimate objective of modelin enetic reulatory networks is the identification of potential tarets for therapeutic intervention [1]. For instance, in cancer one can consider correlation between metastasis and the abundances of mrna for certain enes. In this vein, the abundance of mrna for the ene WNT5A has been found to be hihly discriminatin between cells with properties typically associated with hih versus low metastatic competence [2]. Appropriate alteration in the expression of WNT5A can be perceived therapeutically, and it can therefore be used to search for an optimal intervention stratey [3]. Copyriht (c) 2006 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sendin an to pubs-permissions@ieee.or. This research has been supported in part by the National Science Foundation under Grant CCF and Grant ECS G. Vahedi, B. Faryabi, J.-F. Chamberland, and A. Datta are with the Department of Electrical and Computer Enineerin, Texas A&M University, Collee Station, TX, 77843, USA. ( olnaz@tamu.edu; bfariabi@ece.tamu.edu; chmbrlnd@tamu.edu; datta@ece.tamu.edu) E.R. Douherty is with the Department of Electrical and Computer Enineerin, Texas A&M University, Collee Station, TX, 77843, USA and with Translational Genomics Research Institute, 400 North Fifth Street, Suite 1600, Phoenix, AZ 85004, USA. ( edward@ece.tamu.edu) To date, optimal reulatory intervention has been studied in the context of probabilistic Boolean networks (PBNs), in particular, with respect to the dynamics determined by the probability transition matrix of the associated Markov chain [4]. Major efforts have focused on manipulatin external (control) variables to desirably affect dynamical evolution over a finite time horizon [5] [6]. These short-term policies have been shown to chane the dynamical performance of reulatory networks over a small number of staes; however, they are not necessarily effective in chanin lon-run network behavior. To address this issue, stochastic control has been employed via dynamic prorammin alorithms to find stationary control policies that affect the steady-state distributions of PBNs [7]. Study of infinite-horizon intervention strateies poses two fundamental questions. First, is it possible to beneficially affect a network by applyin the optimal stationary control policy? This translates into assessin the controllability of the network. In practice, a physician would like to predict the effectiveness of a certain treatment at different staes of a disease and on different patients. Investiatin the effect of a certain type of control on various networks is equivalent to questionin the controllability of the network. To date, there has been no investiation on this important topic in the context of ene reulatory networks. Second, can we identify the best intervenin ene? In other words, which ene is the best potential lever point, to borrow the terminoloy from [8], in the sense of havin the reatest possible impact on the desired network behavior? In principle, solvin an optimal control problem for each candidate ene and comparin the performance of the system for these various controls would answer these questions; however, this process is a computationally demandin procedure. The complexity of dynamic prorammin alorithms is vast and increases exponentially with the number of enes [9]. In their early papers, Shmulevich et al. employ two methods for selectin a candidate ene for intervention: mean firstpassae time and influence [1], [4]. The followin bioloical example, borrowed from [10], explains the intuition behind usin mean first-passae times for selectin the best control ene. In bioloy, there are numerous cases where the (in)activation of a certain ene or protein can lead more quickly (or with hiher probability) to a particular cellular functional state or phenotype than the (in)activation of another ene or protein. For instance, in a stable cancer cell line, in the absence of intervention, the cells will keep proliferatin. This behavior can be reversed by controllin the expression

2 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 2 of certain enes. Assume that the oal of the intervention is to push the cell into prorammed cell death (apoptosis). Further assume that we can achieve this intervention with two candidate enes: p53 and telomerase. The p53 ene is the most well-known tumor suppressor ene [11] [12] [13]. The telomerase ene encodes telomerase, which maintains the interity of the end of chromosomes (telomeres) in erm cells. Germ cells are responsible for propaatin the complete enetic material to the followin eneration. Telomerase also maintains the interity of the end of chromosomes in proenitor cells. Proenitor cells are responsible for replenishin cells durin the normal cell turnover (homeostasis). In somatic cells, the telomerase ene is turned off, resultin in telomere shortenin each time the cell divides a key reason for the limited life-span of normal cells [14]. In the majority of tumor cells, telomerase is activated, which is believed to contribute to the proloned life-span of the tumor cells [15]. This worsens pronosis for cancer patients [16] [17]. Extensive experimental results indicate that when p53 is activated in the cells, for example in response to radiation, the cells undero rapid rowth inhibition and apoptosis in as short as a few hours [18]. In contrast, inhibition of the telomerase ene also leads to cell rowth inhibition, differentiation, and cell death, but only after cells o throuh a number of cell divisions (allowin telomere shortenin). Cell death takes a loner time throuh this latter process than via p53 activation. The use of mean first-passae times for findin the best control ene is intuitive; however, it focuses on a one-step control scenario. The influence method distinuishes enes that have a major impact on a predictor function from those that have only a minor impact. This method was introduced to reflect the extent to which a set of enes is capable of determinin the value of a taret ene [4]. It has been used as a criterion to select a control ene with the suestion that the ene with the larest influence on the taret ene is likely to be a ood control ene in the finite and infinite-horizon control of PBNs [6] [7]; however, no research has been done on the overall performance of this heuristic measure. Capitalizin on the bioloical intuition behind mean firstpassae time, we propose an alorithm based on mean firstpassae times that assins a stationary control policy for each candidate ene. We call this alorithm the Mean First-Passae Time (MFPT) alorithm and refer to the correspondin stationary control policy as the Mean First-Passae Time (MFPT) control policy. The proposed alorithm selects the MFPT control policy based on two heuristics: (1) it is preferable to reach desirable states as early as possible; (2) it is preferable to leave undesirable states as early as possible. The MFPT alorithm can be employed in four main applications. First, the MFPT alorithm can be used for predictin the best control ene. The MFPT alorithm enables the computation of the MFPT control policies for all the enes in the network with a manaeable complexity. The control ene with the hihest desirable effect on the lon-run behavior of the network upon the application of the correspondin MFPT control policy is likely the most effective ene for controllin the bioloical system. Second, to reduce the complexity of the optimal stochastic control problem, the MFPT control policy can be used as an approximate solution. Contrary to optimal alorithms, the MFPT alorithm finds policies with constant complexity. Third, the MFPT alorithm can be used to measure the controllability of a network. Since the MFPT control policy is an approximation for the optimal control policy, one can define a network to be controllable if the effect of the MFPT control policy is reater than a desired threshold. Finally, the MFPT alorithm can be used to desin a control policy without requirin network inference. The optimal stochastic control polices proposed thus far require perfect knowlede of the probability transition matrix overnin the network, which must be derived from the PBN or inferred directly. This procedure is prone to modelin errors and suffers from problems of computational complexity for both network inference and findin the optimal control solutions. To achieve model-free intervention, the MFPT control policy can be desined based on estimates of the mean first-passae times. The model-free intervention method has low complexity, is robust to modelin errors, and adapts to chanes in the underlyin bioloical system. Our focus in this paper is on binary PBNs; however, the MFPT alorithm applies without chane to a PBN havin any discrete rane of values. More enerally, the MFPT alorithm can be applied to any Markovian reulatory network and should be mathematically viewed in this manner. For instance, the MFPT alorithm applies to dynamic Bayesian networks (DBNs) [19]. The proposed method can actually be viewed for DBNs in the same manner as with PBNs because any DBN can be represented by a probabilistically equivalent PBN [20]. Concentratin on the latter framework, the difficulty with PBNs possessin more than two values is that the size of the state space expands dramatically. Conceptually, networks with finer quantization can be analyzed usin the same tools; indeed, the oriinal application of automatic control considered a ternary network arisin from cdna-microarray data quantized into three values: -1 (down-reulated), +1 (up-reulated), and 0 (invariant) [5]. Here let us make two points. First, from a theoretical perspective, with finer quantization, the MFPT alorithm still provides sinificant computational benefits over ordinary dynamic prorammin alorithms. Second, from a practical perspective, as with the melanoma example presented in this paper, our focus is on the up-down reulation model. In this scenario, the PBN is binary; in the case of quantization based on down-reulation, up-reulation, or invariance, the network is ternary. This paper is structured in the followin manner. Necessary definitions are provided in Section II. The MFPT alorithm and its applications are explained in Sections III and IV. We analyze the complexity of the MFPT alorithm in Section V. In Section VI, we corroborate our claims usin extensive simulations for four applications: comparison of the optimal and MFPT control policies, findin the best control ene, quantifyin controllability, and model-free control. We then compare optimal and MFPT control policies for the network obtained usin melanoma ene-expression data. Althouh our focus is on applications in the framework of PBNs, the MFPT alorithm applies to any Markovian reulatory network.

3 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 3 II. BACKGROUND A. Probabilistic Boolean Networks An instantaneous probabilistic Boolean network (PBN) consists of a sequence V = {x i } n i=1 of n nodes where x i {0,, d 1}, toether with a sequence {f l } k l=1 of vectorvalued functions called predictor functions. In the framework of ene reulation, each element x i represents the expression value of a ene. It is common to mix the terminoloy by referrin to x i as the ith ene. Each vector-valued function f l = (f l1,..., f ln ) determines a constituent network of the PBN. The function f li : {0,..., d 1} n {0,..., d 1} is the predictor of ene i, whenever network l is selected. The number of quantization levels is denoted by d. At each step, a predictor function is randomly selected accordin to probability distribution {p l } k l=1. After selectin the predictor function f l, the values of enes are updated accordinly, that is, in conformity with the network determined by f l. We consider PBNs with perturbation, in which each ene may chane its value with a small perturbation probability p at each time unit. The dynamics of a PBN can be represented via a Markov chain and, as a consequence of the perturbation, the Markov chain is erodic and possesses a steady-state distribution. Two quantization levels have thus far been used in practice. If d = 2 (binary), then the constituent networks are Boolean networks with 0 and 1 meanin OFF and ON, respectively. The case d = 3 (ternary) arises when we consider a ene to be down-reulated (-1), up-reulated (1), or invariant (0). This situation commonly occurs with cdna microarrays, where a ratio is taken between the expression values on the test channel (red) and the base channel (reen). In this paper, we will develop the methodoloy for d = 2, so that ene values are either 0 or 1; however, the methodoloy is applicable to any finite number of levels. The ene-activity profile (GAP) is an n-diit binary vector x(t) = (x 1 (t),..., x n (t)) ivin the expression values of the enes at time t, where x i (t) {0, 1}. We note that there is a natural bijection between the GAP x(t) and its decimal representation z(t), which takes values in S = {0, 1,...2 n 1}. This bijection will be used later to present data. In the presence of external controls, we suppose that the PBN has m binary inputs, c 1 (t),..., c m (t), which specify the interventions on control enes 1,..., m. A control c i (t), which can take values 0 or 1 at each step t, specifies the action on the control ene i. The decimal bijection of the control vector, u 1,..., m (t) C = {0, 1,..., 2 m 1}, describes the complete status of all the control inputs. As in previous applications, we focus on a sinle control ene, u (t) C = {0, 1}. The treatment alters the status of the control ene, which can be selected amon all the enes in the network. If the control at time step t is on, u (t) = 1, then the state of the control ene is toled; if u (t) = 0, then the state of the control ene remains unchaned. System evolution is represented by a stationary discrete-time equation z(t + 1) = f(z(t), u (t), w(t)) t = 0, 1,..., where state z(t) is an element of the state-space S. The disturbance w(t) is the manifestation of uncertainties in the PBN. It is assumed that both the ene perturbation distribution and the network switchin distribution are independent and identical for all time steps t. The state z(t) at any time step t is a GAP. Oriinatin from a state i, the successor state j is selected randomly within the set S accordin to the transition probability p ij (u) = P(z(t + 1) = j z(t) = i, u (t) = u), for all i and j in S, and for all u in C. Gene perturbation insures that all the states in the Markov chain communicate with each other. Hence, the finite-state Markov chain has a unique steady-state distribution [1]. B. Optimal intervention The problem of optimal intervention for PBNs is formulated as an optimal stochastic control problem. A cost-per-stae, r(i, u, j), is associated to each intervention in the system. In eneral, a cost-per-stae may depend on the oriin state i, the successor state j, and the control input u. We assume that the cost-per-stae is stationary and bounded for all i, j in S, and u in C. We define the expected immediate cost in state i, when control u is selected, by r(i, u) = j S p ij (u)r(i, u, j). We consider the discounted formulation of the expected total cost. The discountin factor, α (0, 1), ensures the converence of the expected total cost over the lon-run [21]. In the case of cancer therapy, the discountin factor emphasizes that obtainin treatment at an earlier stae is favored over later staes. The expected total discounted cost, iven a policy π, an initial state i, and control ene, is denoted by { N 1 J π (i) = lim E α t r(z(t), µ (z(t)), z(t + 1)) i. N t=0 (1) A policy π = {µ (0), µ (1),...} is a sequence of decision rules µ (t) : S C, for each time step t, when the control ene is. The vector J π of the expected total costs is called the value function. In a stochastic control problem, we seek an intervention stratey π amon all the admissible intervention strateies Π that minimizes the value function for each state i, i.e., π (i) = ar min J π (i), i S. (2) π Π A stationary intervention stratey for the control ene is an admissible intervention stratey of the form π = {µ, µ,...}. It is known that an optimal intervention stratey exists for the discounted optimal stochastic control problems. The optimal cost function J satisfies 2 J [ n 1 (i) = min r(i, u) + α p ij (u)j (j) ], i S. u C j=0 } (3)

4 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 4 Furthermore, J is the unique solution of this equation within the class of bounded functions. Equation (3) is known as the Bellman optimality equation. The optimal control policy attains the minimum in the riht-hand side of the Bellman optimality equation for all i. Moreover, an optimal policy determined by the Bellman optimality equation is also a stationary policy. For the proofs of the above statements and more details one can refer to [21]. Standard dynamic prorammin alorithms can be used to find a fixed point of the Bellman optimality equation. C. Influence Influence is a method for quantifyin the relative impact of enes on other enes within the context of PBNs [4]. The influence I j (f) of ene x j on the function f, with respect to the probability distribution D(x), x {0, 1} n, is defined as I j (f) = E D [ f(x) x j ], (4) where E D [ ] is the expectation operator with respect to f(x) the distribution D, x j = f(x (j,0) ) f(x (j,1) ) is the partial derivative of the Boolean function f, the symbol is addition modulo 2 (exclusive OR), and x (j,k) = (x 1,..., x j 1, k, x j+1,..., x n ) for k {0, 1}. In other words, (4) ives the influence as the probability (under the distribution D(x)) that a tole of the jth variable chanes the value of the function. In the context of PBNs, the influence of ene x k on ene x i is iven by I k (x i ) = l(i) j=1 I k(f (i) j ) p (i) j where {p (i) j }l(i) j=1 are the selection probabilities of the predictor functions of ene i and l(i) represents the number of predictor functions of ene i [4]. To quantify the lon-run influence, D(x) is the stationary distribution of the PBN. III. MFPT ALGORITHM In this section, we first elaborate on how the MFPT alorithm is desined based on the mean first-passae time. We then summarize the MFPT alorithm. Application of the MFPT alorithm requires the desination of desirable and undesirable states, and this depends upon the existence of relevant bioloical knowlede. Intervention is performed by flippin (tolin) the expression status of a particular ene from ON to OFF or vice-versa, the intent bein to externally uide the time evolution of the network towards more desirable states. If is the control ene, then applyin the control (intervention) in state x translates into flippin the value of in that state (the control ene chanes to 0 if its value is 1 and vice-versa). Consequently, the network resumes its transition from the new state x, which we call the flipped-state. In the context of therapy, the state-space of a PBN can be partitioned into desirable and undesirable states. Given a control ene, when a desirable state reaches the set of undesirable states on averae faster than its flipped-state, it is reasonable to intervene and transition into the flipped-state. Similarly, if an undesirable state reaches the set of desirable states on averae faster than its flipped-state, it is reasonable not to intervene. These insihts motivate the use of mean first-passae times for desinin intervention strateies. Without loss of enerality we can assume that the transition probability matrix P of the Markov chain (representin a PBN) is partitioned accordin to ( ) PD,D P P = D,U P U,D P, U,U where D and U are the subsets of desirable and undesirable states, respectively. The mean first-passae times are computed by solvin the followin systems of linear equations [22]: K U = e + P D,D K U, (5) K D = e + P U,U K D, (6) where e is a column vector of 1 s with appropriate lenth, K U is a vector containin the mean first-passae times from each state in the subset of desirable states D to undesirable states in set U, and K D is a vector containin the mean first-passae times from each state in the subset of undesirable states U to the desirable states in set D. A control policy µ correspondin to control ene is a vector of size 2 n, the number of states in the network. The decision rule µ : S C specifies the control action for each state x in S. Havin µ (x) = 0 for state x means that, whenever the network reaches state x, no control is applied and the system continues its transition based on the transition probabilities of state x. On the other hand, havin µ (x) = 1 implies that, whenever the network reaches state x, the control is applied and the system continues its evolution based on the transition probabilities of state x, the flipped-state of x. The oal of the MFPT alorithm is to desin the MFPT control policies {ˆµ } n =1. The objective is to choose a control value u for every state in S such that the network evolves towards more desirable states. The MFPT alorithm selects the control policy for control ene in the followin manner. Assume state x is an undesirable state. We compare the mean first-passae times from state x to D and from the flipped-state x to D. In other words, we would like to know on averae which one of these two states, x and x, hits the set of desirable states for the first time faster than the other one. The alorithm chooses ˆµ (x) = 1 if the difference between the mean firstpassae times of state x and the flipped-state x to the set of desirable states, i.e. K D (x) K D ( x), is reater than a tunin parameter γ (to be discussed). Otherwise, ˆµ (x) = 0. Analoously, if state x is desirable, then ˆµ (x) = 1 if the difference between the mean first-passae times of state x and the flipped-state x to undesirable states, i.e. K U ( x) K U (x), is reater than γ. Otherwise, ˆµ (x) = 0. These comparisons are repeated for all states. Alorithm 1 summarizes the proposed procedure. The threshold γ in the MFPT alorithm is a tunin parameter chosen based on the ratio of the cost of control to the cost of undesirable states. When the cost of applyin treatment in a state is hih compared to the cost of undesirable states, an optimal control policy is less likely to apply the control frequently. Thus, γ is set to a larer value when this ratio is hiher, the intent bein to apply control less frequently. We explain after the followin definitions how one can set this

5 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 5 Alorithm 1 MFPT alorithm Partition the state-space into undesirable U and desirable D subsets. Compute K U and K D. 1. repeat for All states x in U do x flip control ene in x. if K D (x) K D ( x) > γ then µ (x) = 1; else µ (x) = 0; end if end for for All states x in D do x flip control ene in x. if K U ( x) K U (x) > γ then µ (x) = 1; else µ (x) = 0; end if end for + 1. until > number of enes parameter. An effective control policy reduces the likelihood of visitin undesirable states compared to a network without intervention by modifyin the lon-run behavior of the network. The effectiveness of a control policy can be measured by the amount of chane (shift) in the areated probability of undesirable states before and after the intervention. As a performance measure we define i U P = π i i U π i i U π, i where π i is the probability of bein in undesirable state i in the lon-run after intervenin with control ene, and π i is the probability of bein in undesirable state i in the lon-run when there is no intervention. The ratio P measures the proportion of reduction in the total probability of undesirable states in the steady state when the control ene is selected. We denote this proportion by P opt when an optimal control policy µ is applied. In other words, in the optimal case one can shift the areated probability of undesirable states to desirable states by P opt throuh appropriately alterin the expression of the control ene. Similarly, the shift obtained by the MFPT control policy ˆµ γ MFPT(γ) is denoted by P, where γ is the tunin parameter. We define the probability of the execution of control as Γ = 2 n 1 j=0 π j I(µ (j) = 1), (7) where n is the number of enes, π j is the steady-state probability of state j S, µ (j) is the value of the control policy in state j, and I is the indicator function. The purpose of introducin this probability is to have a fair evaluation of the performance of the MFPT control policy in terms of the number of control executions, which for the optimal policy is related to the cost of control. For each control ene, one can define Γ opt as the probability of the execution of control when the optimal control policy is applied. Similarly, Γ MFPT(γ) is the probability of the execution of control when the MFPT control policy with the parameter γ is applied. We numerically find the value of the parameter γ for each control cost. We enerate random intervention problems and calculate the averaes of Γ opt and Γ MFPT(γ). These averaes are taken over random intervention problems with fixed control cost. Startin from γ = 0, we increase the value of γ. For each control cost, the desired value of γ is the minimal one for which, on averae, Γ opt > Γ MFPT(γ). This condition uarantees that on averae the MFPT control policy applies no more control than the optimal control policy. Since the values of the parameter γ are found from random intervention problems, in practice one can have a conservative approach and choose the parameter γ to be reater than the proposed value. The conservative approach can assure a hih probability that Γ opt P MFPT(γ) > Γ MFPT(γ) from P opt. On the other hand, the deviation of becomes larer. IV. APPLICATIONS OF THE MFPT ALGORITHM We devise solutions accordin to the MFPT alorithm for four intervention applications. 1) Identification of the best control ene: Recallin the example of p53 and telomerase in the Introduction, it is important to select the most effective control ene in a therapeutical intervention. The best control ene can be found by directly solvin a dynamic prorammin alorithm and computin { P opt } n =1 for all the enes in the network. In short, is iven by opt = ar max P ; (8) =1...n however, this optimal method to find the best control ene is computationally prohibitive. On the other hand, the MFPT alorithm enables the computation of the MFPT control policies {ˆµ γ } n =1 for all the enes in the network with an acceptable complexity. Takin this approach, the MFPT alorithm predicts the best control ene to be MFPT(γ) ĝ = ar max P. (9) =1...n We will show that ĝ = with hih probability, and that P opt opt P ĝ is small whenever ĝ. In this context, we are usin the MFPT alorithm to find the control ene. Once the best ene candidate is identified, an optimal control policy can be obtained usin dynamic prorammin alorithms. We will also show that the MFPT-based prediction of the best control ene sinificantly outperforms the influence method. 2) Approximation of the optimal control policy: The MFPT alorithm can devise an intervention stratey as an approximation of the optimal intervention stratey. To this end, we numerically find the value of the parameter γ for each

6 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 6 control cost so that, on averae, Γ opt > ΓMFPT. To assess the accuracy of the approximation, we show that the averae of P opt MFPT(γ) P over random intervention problems with fixed control cost is small. Note that, so as not to confound approximation accuracy with the MFPT alorithm s ability to find a control ene, we apply both the optimal and MFPT methods usin the optimal control ene. 3) Controllability: An important aspect of pronosis is quantifyin the possibility of recovery. In our framework, this amounts to quantifyin the controllability of a ene reulatory network, a concept borrowed from traditional control theory. Can the network be sufficiently controlled to provide meaninful recovery? We desire a controllability measure where the objective of the control is to reduce the likelihood of observin the undesirable states in the lon-run. An optimal control stratey takes into account the cost of control, but here we want only to focus on the possibility of sufficient control, absent concerns with costs, either medical or financial. To this end, we choose the cost of control to be zero. The zero control-cost stratey admits any number of states with active control. Our point here (one that is certainly debatable) is that we desire a measure of controllability with no restrictions on the number of times the control miht be applied. Thus, a possible approach is to set the cost of control to zero and compute P opt. To reduce the computational burden, we propose P MFPT(0) (γ = 0) as a controllability measure. Our simulations show that the P MFPT(0) is a hihly accurate approximation of P opt when the cost of control is zero. Therefore, the MFPT alorithm can be employed to determine the controllability of a network. For example if P MFPT(0) is very small, we conclude that the network is not controllable. If P MFPT(0) = 0.5, then we conclude that it is possible to shift 50% of the probability mass of the undesirable states to desirable ones in the lon-run, iven the application of the control has zero cost. 4) Model-free intervention: To date, the proposed intervention methods for PBNs are model dependent, requirin at least knowlede of the transition probability matrix. This can be derived from the PBN if the latter is known. Since in practice PBNs are not known except via system identification from observed data, one is faced with a difficult inference problem [23]. This problem can be avoided by directly inferrin the transition probability matrix; however, this is still a formidable task because the complexity of estimatin the transition probabilities of a Markov chain increases exponentially with the number of enes in the model. When time-course data are available, the MFPT alorithm can be implemented by directly estimatin the mean first-passae times. The estimated mean first-passae times are used to construct the matrices of the mean first-passae times, K U and K D. The MFPT alorithm can then be applied to the estimated matrices K U and K D to devise a model-free MFPT control policy. In the followin, we propose a procedure for estimatin the mean first-passae times from time-course measurements. Assume x is a desirable state and it is observed at time t 0. Further assume that, startin from time t 0, the first undesirable state occurs at time t 0 + k 0. In other words, it takes k 0 time points for the desirable state x to transition (reach) to an undesirable state. Similarly, assume the next observation of state x is at time t 1 and since time t 1 the first undesirable state occurs at time t 1 + k 1. In this example, the averae first passae time from state x to the subset of undesirable states is (k 0 + k 1 )/2. Likewise, one can define an example for an undesirable state y reachin the subset of desirable states. It is evident that for larer number of observations, this estimation becomes more accurate. The above procedure needs to be implemented with a low complexity. At each time point, we update the number of steps for each state to reach the opposite subset of states and store the frequency of the occurrence of each state. One needs to update the averae first passae times for a new observation. The complexity of estimatin the mean first-passae times followin our procedure is constant with respect to the number of enes for each iteration. More details reardin the implementation can be found in the supplementary materials. An advantae of the model-free approach is that the estimated matrices K U and K D can be updated whenever new time-course data become available. The possibility of updatin the estimated mean first-passae times enables the MFPT alorithm to adapt its control policy to the status of ene interactions. In other words, the model-free MFPT control method is adaptive to chanes in the network model. In contrary, the control policy devised by the existin intervention methods cannot adapt to the chanes in the status of ene interactions. Once the PBN is inferred form data, the modeldependent control policy is fixed. Throuh numerical studies, we will exhibit the effectiveness of the model-free MFPT control policy obtained by estimatin the mean first-passae times. On one hand, we will estimate the matrices K U and K D based on synthetic time-course data and use the MFPT alorithm to find the control policy; on the other hand, we will use the same time-course data to build a Markov chain representin the dynamics of the model and then find the control policy based on the estimated transition probability matrix usin dynamic prorammin. We will observe that the MFPT control policy based on the estimated mean first-passae times outperforms the control policy devised from the estimated transition probabilities of the Markov chain, iven the same set of time-course data, for feasible size data sets. V. COMPLEXITY ANALYSIS OF THE MFPT ALGORITHM The main objective of an effective intervention stratey is to reduce the likelihood of visitin undesirable states compared to a network without intervention by modifyin the lon-run behavior of the network. Given a time-course data set, there are two possible approaches for desinin a stratey for any model such that its dynamic behavior can be represented as a Markov chain (such as PBN or Dynamic Bayesian Network). In the first approach, one can estimate the transition probabilities of the states from time-course measurements. Let us call this approach model-dependent. We require all the details about the model, i.e. the transition probabilities of the Markov chain. Various methods can be employed to desin an

7 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 7 effective intervention stratey based on the estimated model. The optimal control policy can be desined via dynamic prorammin techniques [7]. In favor of lower computational complexity, an approximation of the optimal control policy can be achieved usin the MFPT alorithm. In the second approach, an effective intervention stratey can be desined directly from time-course measurements. We call this approach model-free. In contrary to the modeldependent approach where the transition probabilities of the Markov chain are needed, we do not require the details of the model. To this end, a model-free alorithm based on reinforcement learnin has recently been introduced [24]. This method bypasses the impediment of model estimation and an effective control policy can be desined with a low complexity. We propose that the MFPT alorithm can also be considered as a model-free method. In this section, we analyze the complexity of the model-based and the model-free MFPT alorithms. A. Model-dependent Approach In the previous section, we introduced the four major applications of the MFPT alorithm: identification of the best control ene, approximation of an optimal control policy, controllability, and model-free intervention. Employment of the MFPT alorithm in the first three applications is considered as a model-dependent approach since it is assumed that the transition probability matrix of the Markov chain is known. Given the model is known, let us compare the computational complexities of the dynamic prorammin and the MFPT alorithms. To find an optimal control policy usin value or policy iteration, one should iteratively find the value (cost) function until the alorithm reaches the fixed point of the Bellman optimality equation in (3). Once the optimal cost functions are computed, one must check which control value attains the minimum in the riht-hand side of the Bellman optimality equation and this procedure should be iterated for all the states. To the best of our knowlede, there does not exist a tiht upper bound on the number of iterations required to find an optimal policy usin either value or policy iteration, despite recent research initiatives [25]. Given the control ene, the policy iteration alorithm has complexity O(2 3n ) per iteration, whereas the complete complexity of the MFPT alorithm, which consists of two matrix inversions, is O(2 3n ). In eneral, it is known that the policy iteration alorithm converes, but it is not known whether the number of iterations in policy iteration can be bounded by a polynomial in the instance size [25]. Even assumin that the number of iterations can be bounded by a polynomial in the number of states, the complexity of the MFPT alorithm is lower than the policy iteration alorithm because it is computed once and does not require iteration. Reardin the value iteration alorithm, the asymptotic upper bound on the number of iterations required to find an optimal policy usin the value iteration alorithm is polynomial in the number of states [25]. The deree of the polynomial is determined to be reater than two in special cases [26], [27]. Given the complexity of each iteration in the value iteration alorithm is O(2 2n ), the complexity of the value iteration alorithm to find an optimal control policy is O(2 (2+α)n ), where α > 1. Hence, the complexity of the MFPT alorithm is also lower than the complexity of the value iteration alorithm. To find the optimal cost functions for n control enes, the complexity of a dynamic prorammin alorithm is n times the complexity of this alorithm for one control ene. In contrary, once the mean first-passae time vectors are computed, they can be used to devise MFPT control policies for all control enes. It is important to point out that for any control ene, in addition to the above complexities, the dynamic prorammin and the MFPT alorithms must loop over all the states to find their correspondin control policies. In dynamic prorammin alorithms, to obtain the optimal control policy, one must check which control value attains the minimum in the riht-hand side of the Bellman optimality equation and this procedure must be iterated for all the states. In the MFPT alorithm, one must investiate which control value leads to a more favorable mean first-passae time and this procedure must be repeated for all the states. It is evident from the above analysis that the application of our proposed method is restricted to small number of enes since the complexity of the MFPT alorithm increases exponentially with the number of enes. We should point out that in our application of interest, intervention in ene reulatory networks, the oal is not to model fine-rained molecular interactions amon a host of enes, but rather to model a limited number of enes, typically with very coarse quantization, whose reulatory activities are sinificantly related to a particular aspect of a specific disease, such as metastasis in melanoma [3]. Hence, while the asymptotic results on the complexities of optimal alorithms are encourain, they are not our main interest; rather our problem deals with networks with small numbers of states. Fi. 1 shows the averae execution time of the value and policy iteration alorithms over 1000 randomly enerated intervention problems as a function of the number of enes n, alon with the execution times of the MFPT alorithm. Per this fiure, the execution time of the MFPT alorithm is much smaller than the execution time of the two optimal alorithms. The direct comparison has been limited to 10-ene networks on account of the hih complexity of the modelin and optimal intervention alorithms. The maximum size of the intervention problem which can be solved by our MFPT method is hardwaredependent. For instance, our current hardware confiuration (sinle Xeon processor and 1-GB memory) can obtain MFPT intervention policy for a synthetic 15-ene reulatory network, which, iven the data limits of current expression measurin technoloy, is sufficient for the applications in which we are now enaed. Given more memory and processin power, intervention strateies can be desined for larer networks. Should the need arise for larer networks, we will consider implementation on our Beowulf cluster at the Translational Genomics Research Institute.

8 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 8 Averae execution time (seconds) Optimal (policy iteration) Optimal (value iteration) MFPT Number of enes Fi. 1. The averae execution time of the value and policy iteration alorithms over 1000 randomly enerated intervention problems as functions of the number of enes, alon with the execution times of the MFPT alorithm. B. Model-free Approach The model-dependent approaches yield effective solutions for lare numbers of observations. However, these approaches have major drawbacks in practice. For lower numbers of observations, which correspond better to feasible experimental conditions, estimatin the Markov chain yields poor results. Estimation errors may have a hue impact on findin an effective intervention stratey, which is often quite sensitive to chanes in the transition probabilities [28]. Furthermore, the complexity of estimatin the transition probabilities of a Markov chain increases exponentially with the number of enes in the model, O(2 2n ). This is in addition to the complexity of desinin an effective intervention stratey. Hence, a procedure that can find an effective intervention stratey without havin to know the transition probabilities is very attractive. The model-free based MFPT alorithm (fourth application) estimates the mean first-passae times from time-course measurements. The complexity of estimatin these vectors followin the proposed procedure in the previous section is constant with respect to n for each iteration, where n denotes the number of enes. In other words, we devise an effective intervention stratey by learnin about the mean first-passae times directly from the data. The hihliht of this paper is the possibility of employin the MFPT alorithm in a model-free approach. To this end, we summarize the two main benefits of our proposed model-free method: 1) The complexity of the modelin and intervention is sinificantly less than that of the model-dependent methods 2) In contrary to the optimal control problem approach, which is sensitive to chanes in the system, the MFPT alorithm needs the averae behavior of the system and is expected to be more appealin for smaller number of observations. We corroborate this claim in the result section by comparin the model-free MFPT method with the model-dependent optimal control method. VI. RESULTS AND DISCUSSION In this section, we first study the performance of the MFPT alorithm for each of the aforementioned applications throuh extensive simulations of random PBNs. We then compare the performance of the MFPT alorithm and the influence method for the network obtained from a melanoma ene-expression data set. A. Synthetic Networks We postulate the followin cost-per-stae: 0 if u = 0 and j D, 10 if u = 0 and j U, r(u, j) = c if u = 1 and j D, 10 + c if u = 1 and j U, where c is the cost of control. The taret ene is chosen to be the most sinificant ene in the GAP. We assume the up-reulation of the taret ene is undesirable. Consequently, the state-space is partitioned into desirable states, D = {0,...,2 n 1 1}, and undesirable states, U = {2 n 1,..., 2 n 1}, where n is the number of enes. The cost values have been chosen in accord with an earlier study [7]. Since our objective is to down-reulate the taret ene, a hiher cost is assined to destination states havin an upreulated taret ene. Moreover, for a iven status of the taret ene for a destination state, a hiher cost is assined when the control is applied, versus when it is not. In practice, the cost values will have to mathematically capture the benefits and costs of intervention and the relative preference of states. These cost values will eventually be set with the help of physicians in accordance with their clinical judement. Althouh this is not feasible within current medical practice, we do believe that such an approach will become feasible when enineerin approaches are interated into translational medicine. In order to investiate the effect of the cost of control in our alorithm, we vary its value from 0 to 10, which is the cost of the undesirable states. We enerate random PBNs in the followin manner. Each PBN consists of 10 constituent BNs. Each BN is randomly enerated with a specific bias b, the bias bein the probability that a randomly enerated Boolean function takes on the value 1. Since the bias affects the dynamical properties of randomly enerated BNs [29], we take it as a parameter in our simulations. We randomly select the bias b of a BN from a beta distribution. We vary the mean of the beta distribution from 0.3 to 0.7 with step-size 0.1. The variance σ 2 of the beta distribution is set to a constant value This provides random biases from low (0.3) to hih (0.7). We enerate 1000 random PBNs for each bias mean. For each PBN, the transition probabilities of the correspondin Markov chain are estimated. The above procedure is repeated for networks of 5 to 10 enes. Due to the computational complexity of the optimal stochastic control problem and the estimation of the transition probabilities of the correspondin Markov chain, the study of a lare number of networks beyond 10 enes is outside our current computational capacity.

9 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING 9 1) Identification of the best control ene: We first show the performance of the MFPT alorithm and the influence method when they are employed to predict the best control ene. It is assumed that the cost of control c is equal to 1. In Tables I, III, II, and IV, we compare the performances of the MFPT alorithm and the influence method for predictin the best control ene. First the optimal control policy for each control ene is obtained by a dynamic prorammin alorithm. The best control ene is found based on (8). Similarly, the MFPT control policy for each control ene is computed and the predicted control ene ĝ is found based on (9). The influence method is also employed to predict the best control ene. The predicted best control ene by the influence method is called ğ. We define the probability of the correct prediction of each method to be the number of PBNs for which the method correctly predicts the best control ene divided by the total number of PBNs in the experiment. The probabilities of correctly predictin the best control ene by the MFPT alorithm and the influence method are shown in Tables I and III. The averae differences between proportions of reduction in the total probability of undesirable states correspondin to the ene predicted by each method and the best control ene, i.e. ( P opt opt opt opt P ĝ ) and ( P P ğ ), are shown in Tables II and IV. In our experiments, the probability of the correct prediction by the MFPT alorithm is always reater than Table. II shows that P opt opt P ĝ on averae is less than TABLE I THE PROBABILITY OF FINDING THE BEST CONTROL GENE WITH THE MFPT ALGORITHM WHEN C = 1 FOR NETWORKS WITH DIFFERENT NUMBER OF GENES. (A) 5 enes (B) 6 enes (C) 7 enes (D) 8 enes (E) 9 enes (F) 10 enes TABLE II THE AVERAGE DIFFERENCE BETWEEN THE PROPORTIONS OF REDUCTION IN THE TOTAL PROBABILITY OF UNDESIRABLE STATES OBTAINED BY THE BEST CONTROL GENE AND THE PREDICTED CONTROL GENE OBTAINED BY THE MFPT ALGORITHM ĝ FOR NETWORKS WITH VARIOUS NUMBER OF GENES. (A) 5 enes (B) 6 enes (C) 7 enes (D) 8 enes (E) 9 enes (F) 10 enes The performance of the influence method is also shown in Tables III and IV. These tables suest that approximately 0.60 of the time the influence method s prediction is correct. In eneral, P opt opt P ğ is reater than Tables V, VI, VII, and VIII show the performance of the MFPT alorithm for TABLE III THE PROBABILITY OF FINDING THE BEST CONTROL GENE WITH THE INFLUENCE METHOD WHEN C = 1 FOR NETWORKS WITH DIFFERENT NUMBER OF GENES. (A) 5 enes (B) 6 enes (C) 7 enes (D) 8 enes (E) 9 enes (F) 10 enes TABLE IV THE AVERAGE DIFFERENCE BETWEEN THE PROPORTIONS OF REDUCTION IN THE TOTAL PROBABILITY OF UNDESIRABLE STATES OBTAINED BY THE BEST CONTROL GENE AND THE PREDICTED CONTROL GENE OBTAINED BY THE INFLUENCE METHOD ğ FOR NETWORKS WITH VARIOUS NUMBER OF GENES. (A) 5 enes (B) 6 enes (C) 7 enes (D) 8 enes (E) 9 enes (F) 10 enes TABLE V THE PROBABILITY OF FINDING THE BEST CONTROL GENE WITH THE MFPT ALGORITHM. (A) c= (B) c= TABLE VI THE AVERAGE DIFFERENCE BETWEEN THE PROPORTIONS OF REDUCTION IN THE TOTAL PROBABILITY OF UNDESIRABLE STATES OBTAINED BY THE BEST CONTROL GENE AND THE PREDICTED CONTROL GENE OBTAINED BY THE MFPT ALGORITHM ĝ WITH VARIOUS COST VALUES. (A) c= (B) c= TABLE VII THE PROBABILITY OF FINDING THE BEST CONTROL GENE WITH THE INFLUENCE METHOD. (A) c= (B) c= TABLE VIII THE AVERAGE DIFFERENCE BETWEEN THE PROPORTIONS OF REDUCTION IN THE TOTAL PROBABILITY OF UNDESIRABLE STATES OBTAINED BY THE BEST CONTROL GENE AND THE PREDICTED CONTROL GENE OBTAINED BY THE INFLUENCE METHOD ğ WITH VARIOUS COST VALUES. (A) c= (B) c=