Stochastic Game Approach for Replay Attack Detection

Transcription

1 Stochastic Game Approach or Replay Attack Detection Fei Miao Miroslav Pajic George J. Pappas. Abstract The existing tradeo between control system perormance and the detection rate or replay attacks highlights the need to provide an optimal control policy that balances the security overhead with control cost. We employ a inite horizon, zero-sum, nonstationary stochastic game approach to minimize the worst-case control and detection cost, and obtain an optimal control policy or switching between controlcost optimal (but nonsecure) and secure (but cost-suboptimal) controllers in presence o replay attacks. To ormulate the game, we quantiy game parameters using knowledge o the system dynamics, controller design and utilized statistical detector. We show that the optimal strategy or the system exists, and present a suboptimal algorithm used to calculate the system s strategy by combining robust game techniques and a inite horizon stationary stochastic game algorithm. Our approach can be generalized or any system with multiple inite cost, time-invariant linear controllers/estimators/intrusion detectors. I. INTRODUCTION Cyber Physical Systems (CPS) eature tight integration o embedded computation, networks, and controlled physical processes []. This interaction between continuous physical dynamics, and discrete communication and computation substrates have made CPS vulnerable to malicious attacks beyond the standard cyber attacks [2]. Successul attacks on CPS could hamper the critical inrastructure with undesired consequences. For example, the Maroochy Water incident and the response discussed in [3] raised attention to security challenges and requirements or secure CPS [2], []. Several attack models, including physical attack, data deception attack, data denial-o-service attack (DoS), zero dynamics attack, and replay attack are analyzed in [4]. Most o these attacks assume some knowledge o the system s dynamics. On the other hand, even without any inormation about the system (including the controller s design), with replay attacks the attacker can record and resend delayed sensor measurements to the controller, causing deterioration in control perormance (or even unstable behavior). Mo and Sinopoli designed a countermeasure method to increase the detection rate or replay attacks, which are undetectable by some statistical detector, like χ 2 detector [5]. By adding a Gaussian signal to the optimal control input, the control This material is based on research sponsored by DARPA under agreement number FA The U.S. Government is authorized to reproduce and distribute reprints or Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those o the authors and should not be interpreted as necessarily representing the oicial policies or endorsements, either expressed or implied, o DARPA or the U.S. Government. F. Miao, M. Pajic and G. J. Pappas are with the Department o Electrical and Systems Engineering, University o Pennsylvania, Philadelphia, PA, USA {miaoei, pajic, pappasg}@seas.upenn.edu perormance is sacriiced to increase the attack detection rate. Consequently, there is a need to provide a control strategy that balances the control and security requirements. Game theory application or security has raised a lot o interest in recent years. The survey [6] provides a selected set o works that use game-theoretic approaches in computer networks security and privacy problems. Zhu et al. deine a zero-sum stochastic game or the design o an Intrusion Detection System (IDS) and provide the stationary optimal strategy [7]. Robust algorithms against well-deined uncertainties are presented in [8]. A game theoretic ormulation or minimax or robust estimation in the presence o aults, and the existence conditions or optimal solutions are discussed in [9]. Yet, there exist many challenges in applying these approaches or design o secure CPS. As stated in [6], one problem with game theoretic techniques in security modeling is the diiculty o quantiying security game parameters. In this work we design a zero-sum, inite horizon, nonstationary stochastic game or minimizing the worst-case control and detection cost, and obtain an optimal control policy or switching between control-cost optimal (but nonsecure) and secure (but cost-suboptimal) controllers in presence o replay attacks. Thereore, in a system dynamic ashion we balance the control perormance and security overhead or replay attack detection. To achieve this, we quantiy game parameters using the knowledge o the system dynamics, controller design and utilized statistical detector. Our problem ormulation satisies the value existence conditions or nonstationary games [], and we show that the optimal strategy or the system exists. A suboptimal algorithm is also developed, based on robust game techniques [] and the inite horizon stationary stochastic game algorithm rom [2]. The presented approach does not have to be constrained to this scenario (i.e., replay attacks, two controllers); it is possible to generalize our analysis, especially the quantiication approach, or any system with multiple inite cost, timeinvariant linear controllers/estimators/ailure or intrusion detectors, acing dierent attacks. In this case, the system will choose rom the library o control, estimation or detection methods under the uncertainty o attacker s particular choice. This paper is organized as ollows. In Section II, we present the system and replay attack model, and describe the control problem. In Section III, we ormulate and quantiy the nonstationary, zero-sum stochastic game between the system and the attacker. The existence o the system s optimal strategy is shown, ollowed by a suboptimal value iterative algorithm in Section IV. On several examples, in Section V we illustrate perormance o the derived attack detection scheme. Finally, Section VI provides concluding remarks.

2 II. CONTROL SYSTEM AND REPLAY ATTACK MODEL Beore presenting the game ormulation, we irst introduce the system and attack models. We consider the setup rom Figure, where a Linear -Invariant (LTI) plant is controlled by an LQR controller with a Kalman ilter (acting as a state estimator), and a χ 2 detector that is used to detect any abnormal behavior. Ater we describe each component o the system, along with a model o replay attacks, we introduce the problem o balancing the control perormance and detection rate. LTI Plant: We consider LTI plants described as: x k+ =Ax k + Bu k + w k, y k =Cx k + v k, where x k R n, u k R p and y k R m denote the plant s state, input and output vectors respectively, and w k and v k denote process and measurement noise at time k. We assume that w k N (, Q), v k N (, R), and x N ( x, Σ) are independent and identically distributed (IID) Gaussian random variables, and that the plant is detectable. Kalman Filter: We assume the Kalman ilter is already in steady state, with parameter K and initial condition P = Σ. Thus, the ilter acts as a ixed-gain estimator o the orm: ˆx = x, P = Σ, K = PC T (CPC T + R), ˆx k k = ˆx k k + K(y k Cˆx k k ), ˆx k+ k = Aˆx k k + Bu k. Optimal LQG Controller: Using the state estimation ˆx k k, the controller acts as a ixed gain compensator o the orm u = u k = Lˆx k k, where L (B T SB+U) B T SA, and S is the solution o the Riccati equation: S = A T SA + W A T SB(B T SB + U) B T SA, (3) or some W, U. Thus, u k satisies: u k = arg min lim E T u k T T [ (x T k Wx k + u T k Uu k )] (4) χ 2 k= Detector: For the described system, the Kalman ilter residues z i = y i Cˆx i i are IID with Gaussian distribution N (, P), where P = CPC T + R. Thereore, at each time k, χ 2 detector takes the orm: k g k = (y i Cˆx i i ) T P (y i Cˆx i i ) α. (5) i=k τ+ a S () (2) Here, τ is the detection window size and α is the alarm triggering threshold. Both parameters are predeined to provide a desired alse alarm rate according to the distribution o g k. Attacker Model or Replay Attacks: We assume the attacker can record sensor measurements, choose the replay window size T, and at each time-step decide whether to send the correct or delayed plant outputs. Thus, data y k (k ) received by the estimator and detector can be described as: { y k y k, sensor output is not changed at k = (6) y k T, T >, replay attack occurs at k I (A + BL)(I KC) A is stable, the χ 2 detector is useless, since the expectation o detector statistics or the compromised residues z k = y k T Cˆx k k will converge to the same value as that or z k = y k Cˆx k k [5] lim k E[(z k) T P z k] = lim E[(z k) T P z k ] = m (7) k To increase the detection rate, an IID Gaussian signal u k N (, L) can be added to the optimal controller u k = u k + u k. (8) With this u k, (7) or z k will be m + 2trace(CT P CU), where U satisies U BLB T = AUA T. On the other hand, applying the additional input u would increase the quadratic cost to J = J + trace[(u + B T SB)L], where J denotes the optimal cost (or control input u k rom (4)) [5]. The above analysis o control perormance and detection rate is over ininite time horizon. To balance the security overhead and the control cost, we consider applying dierent controllers according to the system dynamics in inite time. Consequently, our goal is to design an optimal policy or switching between the (cost) optimal controller (i.e., u k ) and the controller rom (8) which allows or the detection o replay attacks, as shown in Figure 2. To achieve this, we rame the problem as a non-cooperative stochastic game between the system and the attacker, since the system has limited knowledge about the attacker s decisions, while the detector only provides a probabilistic detection rate o malicious behavior. III. STOCHASTIC GAME FORMULATION OF REPLAY DETECTION To obtain a switching control policy that minimizes the expected cost or the considered closed-loop system, we ormulate a zero-sum inite horizon stochastic game between the CAN Bus a 2 a m Plant S 2... S p CAN Bus It is worth noting here that when a replay attack occurs, values rom all sensors can be compromised. Attacker y k /y k-t record u k Controller Detector y k -Cx k k x k k Estimator replay:change y k to y k-t y' k Switch u* k u* k + Controller Controller 2 u k x k k From Estimator u k- Inormation From Detector Fig.. System diagram with only one controller, where replay attacks can compromise sensor measurements delivered to the controller. Fig. 2. Block diagram o the switching controller Controller is cost optimal, while Controller 2 provides a higher attack detection rate.

3 system and the attacker. We consider K-stage games, where each stage corresponds to n time-steps o the dynamical model rom (). For simplicity, in this section we present the case or n =, i.e., every game stage is also one time step o the physical model. We denote the attacker as the maximizer (the row player) and the system as the minimizer (the column player) in the game. While each player is able to observe the state o the game, neither player has exact inormation about the other player s previous behavior. We assume that once the detector triggers the alarm, the system stops its execution to check or malicious behavior. In this case, i the attacker has been active it will be detected and the system will remain sae, using only the optimal controller in the uture (i.e., the system wins). Otherwise, the system pays a signiicantly large penalty or triggering the alse alarm. Figure 3 illustrates the stochastic game model. The game state space S is stationary. At each stage k, the nonstationary parameters include action spaces or the attacker (A tk ) and system (A sk ), the state transition probability matrix P k, and the immediate payo matrix r k. We consider mixed strategies F k, G k or both players. Formally, we deine the game as a sequence o tuples (S, A tk, A sk, F k, G k, P k, r k ), k {,, K}. Every parameter is quantiied using the system and attack model described in Section II. ) Game State Space: S = {s, s 2, s 3 } denotes the set o the stochastic game states. Absorbing state s = sae describes that the system has already successully detected a replay attack; s 2 = no detection speciies that the alarm has not been triggered; inally, the system enters the state s 3 = alse alarm trigger when the alarm is triggered beore attacks occur. 2) Attacker s Action Space: At each stage k, the attacker has M candidate actions, i.e., A tk = {a k,, a Mk } = {y k, y k t2,, y k tm } is the attacker s action space, where y k is the real sensor value and y k ti denotes the replay attack with a window size t i. 3) System s Action Space: A sk = {u k, u 2k } = {u k, u k + u k} is the system s action space acing a replay attack at stage k. Here, u k = u k is the cost optimal input, while u 2k = u k + u k provides a higher detection rate. 2 4) Mixed Strategy: Let k i(s) and gj k (s) be the probabilities that at stage k and state s the attacker and the system 2 Note that i N > 2 controllers/estimators/detectors are used, the system s action space should be o size N and the method presented in this paper could still be used. S S 2 No S a e Detection S 3 False Alarm Trigger chose actions a ik A tk and u jk A sk, respectively. We deine F k and G k as the sets o strategies o the attacker and the system at stage k: F k := { k = [ k (s ), k (s 2 ), k (s 3 )] k (s) R M, s S, k(s) i, a ik A tk, k(s) i = }, i G k := {g k = [g k (s ), g k (s 2 ), g k (s 3 )] g k (s) R 2, s S, g j k (s), u jk A sk, j g j k (s) = }, We denote with H k = H F G F k G k the concatenation o the strategies until stage k, where H S is the initial state set, and each strategy history h k H k can be described as h k = h g k g k or h = s l. 5) State Transition Probability: P k is the state transition probability set at k, where P k P k satisies: P k (s ij h k, s) = [ P k (s h k, s) ] R M 2, s, s S, P ij k (s h k, s) =, (a ik, u jk ) A tk A sk, s S. ij Here, P k (s h k, s) is the probability provided by the detector (like χ 2 detector), that the system transits rom state s at stage k to state s at stage k +, given a history h k H k and both players actions (a ik, u jk ). 3 6) Immediate Payo Function: We deine the immediate payo matrix set at stage k as r k R M 2, and r k r k, where r k (h k, s) = [ k (h k, s) ], denotes the payo o every action pair (a ik, u jk ) or strategy history h k and state s. Given system state x k (h k ) and control input γ k (h k, a ik, u jk ), similar to the LQG cost we have k (h k, s l ) = tk (h k, s l )(attacker) = sk (h k, s l )(system): k (h k, s ) = x T k (h k)wx k (h k ) + γk T (h k, a k, u jk )Uγ k (h k, a k, u jk ), k (h k, s 2 ) = x T k (h k)wx k (h k ) + γk T (h k, a ik, u jk )Uγ k (h k, a ik, u jk ), k (ĥk, s 3 ) = alse alarm trigger penalty. Note that at state s the system wins, so the payo is determined by real sensor data a k = y k. 7) Control System Dynamics: With all the above deinition, or any strategy history h k, given the system and attack models rom Section II, we can obtain the expected system behavior under the stochastic game ormulation (or simplicity we omit the E), which is used to compute the expected payo or dierent strategies (deined in the next section). Given initial ˆx (h ) = x, x (h ) = x, at each stage k the LTI plant and Kalman Filter evolve as: x k (h k ) = Ax k (h k ) + Bu k (h k ) + w k, a k (h k ) = y k (h k ), a ik (h k ) = y k ti (h k ), i, ˆx k k (h k, a ik ) = ˆx k k (h k ) + K(a ik (h k ) Cˆx k k (h k )), ˆx k+ k (h k, a ik, u jk ) = Aˆx k k (h k, a ik ) + Bγ k (h k, a ik, u jk ). To speciy behavior o the χ 2 Detector we deine residues z k+ (h k, a ik, u jk ) = a ik (h k ) Cˆx k+ k (h k, a ik, u jk ). Fig. 3. Stochastic Game Model, s is an absorbing state. 3 Full description o the game ormulation can be ound at sites.google.com/site/miaoeiatpenn/publications

4 Then g k+ used to extract the state transition probability is k g k+ (h k, a ik, u jk ) = [z t (h t )] T P z t (h t ) (9) t=k τ+2 + [z k+ (h k, a ik, u jk )] T P z k+ (h k, a ik, u jk ) Updating the Model With Strategy At Stage k: I the strategies at stage k are k and g k, then h k+ = h k k g k. Furthermore, let p(s l k ) be the probability that the system is at state s l at stage k (note that p(s l ) is given). Then we update the ollowing parameters or stage k + : 2 M 3 u k (h k+ ) = p(s l k)k(s i l )g j k (s l)γ k (h k, a ik, u jk ), y k (h k+ ) = j= i= l= M i= l= 3 p(s l k)k(s i l )a ik (h k ), and the probability o the system being at state s l at stage k + is: 3 p(s l k+) = p(s l k)[ k (s l )] T Pk (s l h k, s l )g k (s l ). l= Thereore, when we involve the system dynamics to deine the game, the resulting ormulation utilizes a nonstationary immediate payo matrix and a transition probability matrix. IV. EXISTENCE OF AN OPTIMAL STRATEGY AND SUBOPTIMAL ALGORITHM Based on the game ormulation, in this section we discuss the existence o an optimal solution or the system, and present an algorithm to compute a suboptimal system strategy. A. Existence o the System s Optimal Strategy We deine the concatenation o strategies or K-stage game o each player ( or attacker and g or system) as = K, k F k, g = g g K, g k G k. Let the random variable ζ k describe the state o the game at stage k, and let us deine the conditional expected total payo till K or any, g, given initial state ζ = s as R K(s,, g) = K k= l= 3 p(ζ k = s l ζ = s)[ k (s l )] T r k (h k, s l )g k (s l ). Since the immediate payo satisies k (h k, s l ) <, we have that R K(s,, g) is a nonnegative real-valued, nondecreasing unction with K. Furthermore, or inite K R K (s,, g) <, s,, g. () Deinition ([]): A two-person zero-sum K-stage stochastic game is said to have a value vector v K i v K,s = v K,s = v K,s, or any s S, v K,s = sup in where R K(s,, g), v K,s = in g g sup R K (s,, g). For the inite value K-stage stochastic game, strategies g and are called optimal or player two (the system) and player one (the attacker), respectively, i or all s S v K,s = sup R K (s,, g ), v K,s = in g R K(s,, g). The existence conditions o the value and optimal strategies or a general inite horizon zero-sum nonstationary stochastic game are shown in []. The game deined in this paper has a inite state space, inite action spaces, and satisies (). Thereore, using the same approach as in the proos rom [] we can prove the ollowing theorem: Theorem : There exists the value o the considered game and an optimal strategy or the system. B. Suboptimal Algorithm For the Nonstationary Game Existing value iterative algorithms or stationary stochastic games can not be used to solve our game, since the nonstationary game parameters depend on the previous history, which is only available in the uture algorithm iterations. Hence, we design a suboptimal algorithm based on the value iteration method or inite horizon stationary stochastic game rom [2] and robust game techniques rom []. Algorithm provides an upper bound or the game value and the corresponding nonstationary suboptimal strategy or the system. The idea is to solve a robust game at each iteration step i.e., minimize the worst-case caused by extreme points o the nonstationary payo and state transition probability polyhedra, or (r k, P k ), deined or all possible histories. The value iteration algorithm or inite horizon stationary stochastic game (with ixed payo r and state transition probability P at every stage) works in the way that i a player knew how to play in the game optimally rom the next stage on, then, at the current stage, he would play with such strategies [2]. The value o K-stage game is inally provided by the last step o iteration. Similarly, the Algorithm o the nonstationary stochastic game starts rom the last stage, gets the matrix game value and the optimal strategy related to nonstationary (r k, P k ) at each stage, and returns an upper bound or the value o the total payo in K-stages. To estimate the values at each step, we consider the payo and state transition probability set (r k, P k ) as an uncertain parameter set or the one shot robust game []. To quantiy the bounded polyhedra (r k, P k ), we need the expected system dynamics x k, u k, y k, k =,, K deined in Section III-.7, which is determined by the strategy history. The extreme points or the uncertain set (r k, P k ) depend on pure strategy histories. Let H p k H k be the concatenation o pure strategies until stage k, where a pure history h p k Hp k, hp k = s p gp p k gp k satisies that all p t (s), g p t (s) have only one non-zero element (i.e., they choose the corresponding action or the pure strategy). By using any h p k in the game model o Section III-.7, we get the corresponding r p k (hp k, s l) and P p k (hp k, s l) or stages to K. Thus, the extreme points set (r kp, P kp ) or (r k, P k ) is: {(r p k (hp k, s l), P p k (hp k, s l)) h p k Hp k, l {, 2, 3}}. We deine the pure strategy backup matrix set Q kp, k =,, K as: Q kp = {Q p k (hp k, s l) = r p k (hp k, s l) + P p k (s h p k, s l)v s k+ (hp k+ )}, () where vk+ s (hp k+ ) is the robust game value resulting rom the iteration at stage k + o Algorithm, and relates

5 to matrix games deined by Q (k+)p. In addition, we deine the backup matrix set Q k or all possible h k, as Q k = { Q k (h k, s l ) = r k (h k, s l ) + P k (s h k, s l )vk+ s (hp k+ )}. Finally, or the stage K we deine Q p K (hp K, s l) = r p K (hp K, s l) and Q K (h K, s l ) = r K (h K, s l ). Now, consider the iteration or calculating v s l k (hp k ) rom all matrix games Q p k (hp k, s l) Q kp in Algorithm. We denote the non-pure history value o k as v s l k (hp k ), which is calculated rom the Q k (h k, s l ) that have the same pure strategies as h p k in all stages,, k 2, and any strategy at stage k. We use the ollowing result to show that at every k, v s l k (hp k ) is greater than or equal to vs l k (hp k ). Theorem 2: Consider the value iteration or stage k as a one shot robust game. Based on vk+ s (hp k+ ) o previous iteration, We deine the robust game value obtained at k as v s l k (hp k ) = max v [Q p Q p k (hp k,s k (hp k, s l)], (2) l) Q kp where v is the unction that yields the value o a zero-sum matrix game. Then or k = 2,, K, v s l k (hp k ) is upper bounded by v s l k (hp k ) (i.e., vs l k (hp k ) vs l k (hp k )). Proo: Since vk+ s (hp k+ ) is a nonnegative scalar, the extreme points o Q k can only come rom the extreme points o the tuple (r k, P k ), i.e., by considering the matrix game value o every Q p k (hp k, s l) deined in (), we will get the maximum value rom extreme points o Q k. Now consider the ollowing optimization problem or the system (4) and or any attacker s strategy vector min z (3) g subject to z max Q k (h k,s l ) Q k T [ Q k (h k, s l )]g. (4) As proven by Lemma 5 in [], (4) is equivalent to the ollowing constraint that considers only the extreme points z max Q p k (hp k,s l) Q kp T [Q p k (hp k, s l)]g, (5) For the worst-case, the above is also true. Hence, let v s l k (hp k ) = max min Q p k (hp k,s l) Q kp g max T [Q p k (hp k, s l)]g. (6) For optimal policies (h p k, s l) and g (h p k, s l), the above optimization (6) results in cost max v [Q p Q p k (hp k,s k (hp k, s l)]. l) Q kp However, ( (h p k, s l), g (h p k, s l)) can be non-pure strategies, meaning that when used in III-.7, they will not result in extreme points o Q k+. The non-pure history value vk+ s (hp k ) o iteration or stage k + satisies vk+ s (hp k ) vs k+ (hp k+ ). Replacing vk+ s (hp k+ ) by vs k+ (hp k ) in () will decrease every element in matrix Q p ij k, since rij k and Pk. With a similar argument in the next iteration or stage k, we have v s l k (hp k ) vs l k (hp k ). According to Theorem 2, we use Algorithm to compute an upper bound o the value and the corresponding suboptimal strategy or every step. The unction π computes the strategy and robust value as deined in (2). Note that or Algorithm : Suboptimal Algorithm or A Finite Nonstationary Stochastic Game Input: System model parameters and game parameters. Initialization: Compute the set o (r kp, P kp ) or k =,...K. Compute the game at stage K: Q p K (hp K, s l) = r p K (hp K, s l), (h p K, s l), g (h p K, s l), v s l K (hp K ) π(qp K (hp K, s l). Iteration: For k = (K ),,, get the backup matrix set Q kp or all h p k Hp k, where each matrix is deined in (), then calculate: (h p k, s l), g (h p k, s l), v s l k (hp k ) π(qp k (hp k, s l)). k = [ (h p k, s l), l =, 2, 3], gk = [g (h p k, s l), l =, 2, 3]. Return: the strategy concatenation pair a = K, g a = g gk and the value upper bound vs l, l =, 2, 3. the replay attacks considered in this paper, at state s the system wins the game and replay is harmless. Thus, the optimal strategies o s is g k (s ) = [ ] T, k (s ) = [ ] T. The nonstationary game values v s l k (hp k ) and v s l k (hp k ) that result rom value iteration o two strategies only dier at stage k (i.e., same and pure rom stages to k 2). By value iteration backward to stage, we compare the game value (or all possible strategies) and the robust game value v s l Theorem 3: o Algorithm in the ollowing theorem. Algorithm results in an upper bound v s l or the value o the K-stage game, together with suboptimal strategies a and g a. Proo: The strategies a, g a o Algorithm are possibly not pure. According to Theorem 2, v s l k (hp k ) vs l k (hp k ), and the proo holds or every k = 2,, K. Consider the value iteration or k =, with v s l 2 (hp ) vs l 2 (hp 2 ), Q ij (h, s l ) = (h, s l ) + P ij (s h, s l )v2 s (hp ) Qij (hp 2, s l), thus v [Q (h, s l )] v s l. Iterative value based on pure strategy back up matrix sets Q kp, k =,, K obtained rom Algorithm is an upper bound or the game value. V. EXAMPLE To illustrate our stochastic game approach, we consider the our input our output system examined in Section 3. o [3]. We assume that the attacker s action space (i.e., replay window size) contains t 2 =, t 3 = 2, t 4 = 3, t 5 = 4, and that the initial system state is s 2 (i.e., p(s 2) = probability o applying controller Suboptimal strategy when system being at state S Fig. 4. System s suboptimal strategy at state s 2 the probability o switching to Controller 2 at every stage.

6 System s total cost System control cost comparison with initial state s2 System always applies controller2 System applies suboptimal strategy Comparison o probability o system being at state s Probability o system being at state s System always applies controller2 System applies suboptimal strategy System s total cost System control cost comparison without replay system always applies controller2 system applies suboptimal strategy (a) Control cost comparison when the system applies dierent strategies; initial state is s 2 Fig. 5. (b) Probability o system being at state s (sae, already successully detected a replay attack). Comparison on the system control cost and detection rate under dierent scenarios (c) Control cost comparison when no replay occurs; initial state is s 2. ). Using Algorithm, we obtained a suboptimal system s strategy g a, or the K = 5-stage game. Figure 4 shows the probability o switching to Controller 2 at every step. We compare the system s control cost in two cases, when the system applies the suboptimal game strategy g a and when it only uses Controller 2 (i.e., always adds noise by applying input u k + u k at every step k); all plots presented in this section are obtained by averaging results o simulations. In Figures 5(b) and 5(a), we present the system s total cost when the attacker does not ollow the strategy a but rather replays previous sensor measurements with delay o T = 25s, starting rom time t = 26s. We compare two strategies or the system, using Controller 2 in each step and ollowing the strategy g a. Figure 5(b) shows the probability that the system is at state s (successully detected a replay attack) over time. These results illustrate that the suboptimal system strategy g a results in a lower control cost compared to the non-optimal Controller 2, although Controller 2 insures a higher detection rate. Note that in regular operation modes, when no attacks occur, it is ineicient to sacriice control perormance at every time step. By switching controllers we reduce the perormance loss, as shown in Figure 5(c), while being able to detect potential attacks. Besides inding the optimal or suboptimal strategy, we can also use the stochastic game ramework to ind the best reply strategy or a speciic attacker behavior. For example, the attacker s strategy can be to always apply a certain type o attack beore being detected. When the computational complexity o Algorithm is high and we want a aster iteration, we can calculate the best reply strategy with respect to one attack type rom the attack action space, run the algorithm or a dierent action type each time, and ind the worst-case cost. This approximation is reasonable since when we do not have expectation that a certain attack type will appear, then our goal is to keep the system perormance acceptable under the worst-case attack type. VI. CONCLUSION In this paper, we have proposed the use o noncooperative stochastic games to design a suboptimal switching control policy that balances control perormance with the intrusion detection rate or replay attacks. We have presented the detailed quantiication process or the game parameters, which utilizes knowledge o the system s dynamics. To solve the nonstationary stochastic game, we have developed a suboptimal value iteration algorithm by considering each iteration as a robust game. Note that the quantiication process and the proposed algorithm can be generalized and applied or optimal design o LTI plants with inite number, inite cost components acing attacks. In uture, we will explore the use o game theoretic methods to provide system resiliency against other types o attacks, and revise the presented algorithm to reduce its computational complexity. REFERENCES [] K.-D. Kim and P. R. Kumar, Cyber-physical systems: A perspective at the centennial, Proceedings o the IEEE, vol., no. special Centennial-Issue, 22. [2] A. Cardenas, S. Amin, and S. S. Sastry, Research challenges or the security o control systems. in Proceedings o the 3rd USENIX Workshop on Hot topics in security, 28, Article 6. [3] J. Slay and M. Miller, Lessons learned rom the maroochy water breach, in Critical Inrast. Protection, 27, pp [4] A. Teixeira, D. Pérez, H. Sandberg, and K. H. Johansson, Attack models and scenarios or networked control systems, in st international conerence on HiCoNS, 22, pp [5] Y. Mo and B. Sinopoli, Secure control against replay attacks, in 47th Annual Allerton Conerence on Communication, Contro, and Computing, 29, pp [6] M. Manshaei, Q. Zhu, T. Alpcan, T. Basar, and J.-P. Hubaux, Game theory meets network security and privacy, ACM Comput. Surv., vol. 45, no. 3, pp. 25: 25:39, 23. [7] Q. Zhu and T. Basar, Dynamic policy-based ids coniguration, in 48th IEEE Conerence on Decision and Control (CDC), 29, pp [8] A. Cardenas, S. Amin, and S. Sastry, Secure control: Towards survivable cyber-physical systems, in Workshops at the Int. Con. on Distributed Computing Systems, 28, pp [9] S. Verdu and H. Poor, On minimax robustness: A general approach and applications, IEEE Transactions on Inormation Theory, vol. 3, no. 2, pp , 984. [] A. S. Nowak, Approximation theorems or zero-sum nonstationary stochastic games, in American Mathematical Society, vol. 92, no. 3, 984. [] M. Aghassi and D. Bertsimas, Robust game theory, Math. Program., no., pp , 26. [2] M. J. Kearns, Y. Mansour, and S. P. Singh, Fast planning in stochastic games, in Proceedings o the 6th Conerence on Uncertainty in Artiicial Intelligence, 2, pp [3] Y. M. Chabukswar, R. and B. Sinopoli, Detecting integrity attacks on scada systems, in Proceedings o the 8th IFAC World Congress, vol. 8, 2.