Quantifying the Resiliency of Fail-Operational Real-Time Networked Control Systems

Transcription

1 Technical Report MPI-SWS April 2018 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem Arpan Gujarati Max Planck Intitute for Software Sytem (MPI-SWS), Kaierlautern, Germany Mitra Nari Max Planck Intitute for Software Sytem (MPI-SWS), Kaierlautern, Germany Björn B. Brandenburg Max Planck Intitute for Software Sytem (MPI-SWS), Kaierlautern, Germany Abtract In time-enitive, afety-critical ytem that mut be fail-operational, active replication i commonly ued to mitigate tranient fault that arie due to electromagnetic interference (EMI). However, deigning an effective and well-performing active replication cheme i challenging ince replication conflict with the ize, weight, power, and cot contraint of embedded application. To enable a ytematic and rigorou exploration of the reulting tradeoff, we preent an analyi to quantify the reiliency of fail-operational networked control ytem againt EMI-induced memory corruption, hot crahe, and retranmiion delay. Since control ytem are typically robut to a few failed iteration, e.g., one mied actuation doe not crah an inverted pendulum, traditional olution baed on hard real-time aumption are often too peimitic. Our analyi reduce thi peimim by modeling a control ytem inherent robutne a an (m, k)-firm pecification. A cae tudy with an active upenion workload indicate that the analytical bound cloely predict the failure rate etimate obtained through imulation, thereby enabling a meaningful deign-pace exploration, and alo demontrate the utility of the analyi in identifying non-trivial and non-obviou reliability tradeoff ACM Subject Claification Embedded and cyber-phyical ytem Keyword and phrae probabilitic analyi, reliability analyi, networked control ytem 1 Introduction Networked control ytem (NCS) where enor, controller, and actuator belonging to one or more control loop are connected by a hared network are widely deployed in contemporary cyber-phyical ytem a they offer many practical advantage over dedicated wiring olution, not the leat of which are cot and weight aving [25]. Like other embedded ytem, NCS are uceptible to both internal and external ource of electromagnetic interference (EMI), e.g., park plug, TV tower, etc. [46]. In fact, the likelihood of oft error due to EMI acro a fleet of device hould not be underetimated. For example, Mancuo [40] oberved that, auming one oft error per bit in a 1 MB SRAM every hour of operation, and a worldwide population of 0.5 billion car with an average daily operation time of 5%, about 5,000 vehicle per day are affected by a oft error. Since unmitigated oft error can reult in potentially catatrophic ytem failure, EMIinduced error cenario are anticipated in the deign of afety-critical ytem, and commonly

2 16:2 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem mitigated by mean of either active or paive replication. In the context of high-frequency control application pecifically, paive replication, i.e., the ue of hot/cold tandby, i inufficient if the failure detection and view-change latencie exceed the control frequency. Sytem engineer thu devie active replication (or tatic redundancy) cheme to enure that afety-critical NCS are fail-operational (e.g., ee [15, 22, 28]). However, coming up with a good active replication cheme i no eay tak. Engineer face many quetion, uch a which component, if made more or le reilient (e.g., by adding an extra replica, or hielding), will mot impact the overall reliability? Alternatively, which component could be replaced with cheaper conumer-grade part with the leat effect on ytem reliability? Would dual modular redundancy uffice if the control logic i robut to, ay, 10% meage lo or would triple modular redundancy be needed? In general, uch quetion (and many more like them) do not have obviou anwer, and particularly not if ize, weight, and power (SWaP) a well a cot contraint mut be taken into account, too. The challenge i further exacerbated by the fact that commercially-ued controller are typically afeguarded againt diturbance and noie uing appropriate limiting or clamping mechanim, and mot well-deigned control ytem are inherently robut to a few failed iteration, e.g., one mied actuation doe not crah an inverted pendulum. That i, requiring that all control loop iteration mut be correct and timely i.e., completely unaffected by oft error force exceively peimitic anwer relative to the true need of the workload, and conequently reult in under-utilized, cot-inefficient ytem. Thu, to appropriately dimenion a fail-operational real-time NCS, a robutne-aware reliability analyi i required. In thi paper, we preent a ound reliability analyi that evaluate a given configuration of an actively replicated NCS and quantifie it reiliency to EMI-induced tranient error, including meage omiion error due to hot crahe, incorrect computation error due to memory corruption, and deadline violation due to retranmiion delay. The objective i to provide ytem engineer with a ound method to evaluate (i.e., afely bound) the reliability of an active replication cheme (i.e., for a given number of replica for each tak in the NCS) auming peak failure rate are known from empirical meaurement and/or environmental modeling. We conider NCS that are networked uing a broadcat medium uch a CAN (or Ethernet with a reliable broadcat primitive implemented on top) and evaluate them at the granularity of meage exchange between the ditributed component. Unlike traditional olution baed on hard real-time aumption, our analyi leverage the robutne of well-deigned control ytem: ince robut control loop tolerate a limited number of tranient failure (which reult in degraded control performance, but not an unrecoverable plant tate), we characterize control loop with (m, k)-firm pecification, where out of every k conecutive control loop iteration, at leat m mut be correct and timely [26]. Blind and Allgöwer [9] have hown that the (m, k)-firm model i trictly tronger than the claical aymptotic requirement for control robutne (e.g., a recently tudied by Saha et al. [51]), which mandate that, a the number of control loop iteration approache infinity, the failure rate hould not exceed a given threhold. We thu ue thi model to bound the failure in time (FIT) of an NCS, i.e., the expected number of control failure in one billion operating hour, where control failure denote a violation of the (m, k)-firm contraint. The propoed analyi conit of three tep. Given a model of a fault-tolerant ingleinput ingle-output (FT-SISO) control loop with active replication of it critical tak ( 2), the program-viible effect of EMI are firt claified a crahe (reulting in meage omiion), memory corruption (reulting in incorrect meage), and meage retranmiion (reulting in deadline violation), and each of thee error i modeled probabilitically ( 3). Second, an intermediate analyi ( 4) then relate the probability of individual meage error to

3 A. Gujarati, M. Nari, and B. B. Brandenburg 16:3 Senor tak replica S 1 S 2 N Senor mg. tream X 1 X 2 Controller tak replica C 1 C 2 N Control Mg. tream U 1 U 2 Actuator tak A Senor Plant Actuator Figure 1 An FT-SISO control loop. Solid boxe denote hot. Each dahed box denote a tak replica et or a et of meage tream tranmitted by a tak replica et. Dahed arrow denote meage tream broadcated over the hared network N, e.g., X 1 and X 2 are received by all tak in C. that of a failed iteration of a control loop, i.e., where the controlled plant i not actuated a expected in an error-free iteration. 1 Finally, a reliability analyi upper-bound the FIT of an NCS, which may conit of one or more FT-SISO control loop, a a function of the control loop repective (m, k)-firm pecification. We have evaluated the propoed analyi with a cae tudy exploring replication option for a CAN-baed active upenion workload ( 5). Our reult how that analyi and imulation reult cloely track each other when configuration parameter are varied. We alo demontrate how the analyi can help in identifying non-obviou reliability tradeoff, and identify the underlying timing analyi of the CAN bu [16] a the ingle greatet individual ource of peimim in our analyi due to it reliance on a critical intant that occur only rarely. 2 Sytem Model We conider an FT-SISO networked control loop L deployed on hot H {H 1, H 2,...} connected by a broadcat medium N, which i hared with other traffic a well, e.g., other control loop, the clock ynchronization protocol, etc. A block diagram i hown in Fig. 1. The enor tak replica S {S 1, S 2,...} periodically generate enor output and broadcat it over N. A a convention, we let upercript denote replica ID. We let X i denote the meage tream carrying the enor value of the i th replica of the enor tak, and let X {X 1, X 2,...} denote the et of all uch meage tream. The controller tak replica C {C 1, C 2,...}, upon periodic activation, read the latet received enor meage, compute a new control command for the plant, update their local tate (e.g., in a PID controller, the integrator), and broadcat the control command. They are aigned appropriate offet to enure that, in an error-free execution, the enor meage are available before any controller tak replica are activated. The meage tream carrying control command are denoted U {U 1, U 2,...}. The actuator tak A i directly connected to the plant. Upon periodic activation, it read the latet received control command and actuate the plant accordingly. Like the controller tak, A i alo aigned an appropriate offet to enure that, in an error-free execution, all control command are received before it activation. Unlike the enor and controller tak, 1 Note the difference between a failed iteration of a control loop and control failure. A failed iteration i imply a deviation from an ideal, error-free cenario. Multiple failed iteration may lead to control failure if they violate the control loop (m, k)-firm pecification.

4 16:4 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem Algorithm 1 Voting procedure before the i th activation of any controller tak. The voting procedure for the actuator tak i defined imilarly by replacing the input et X i with U i. 1: procedure PeriodicControllerTakActivation 2: Latet i tart voting protocol 3: for all Xi k X i do 4: if Xi k not received by it deadline then 5: continue alo account for omiion 6: Latet i Latet i X k i 7: if Latet i then return omit output 8: reult i SimpleMajority(Latet i ) break tie baed on meage ID 9:... main logic of the tak tart the actuator tak A i not replicated ince it require pecial hardware in the plant actuator to handle redundant input [28]. We reviit thi iue in 7. All tak and meage in the control loop have a period of T time unit. The i th runtime activation or job of enor tak replica in S {S 1, S 2,... } and controller tak replica in C {C 1, C 2,... } are denoted S i {Si 1, S2 i,... } and C i {Ci 1, C2 i,... }, repectively; and the i th job of actuator tak A i denoted A i. Similarly, the i th meage in enor meage tream X {X 1, X 2,... } and controller meage tream U {U 1, U 2,... } are denoted X i {Xi 1, X2 i,... } and U i {Ui 1, U i 2,... }, repectively. Finally, we let U i denote the actuator command applied to the phyical plant in the i th iteration, i.e., output of job A i, and let U {U 1, U 2,... } denote the ordered et of uch command applied to the phyical plant acro all iteration. Aumption. We aume that tak reolve redundant input at the tart of every iteration through voting (Algorithm 1). We let V i {Vi 1, V i 2,... } denote the et of voter intance that reolve the redundant input for controller job C i {Ci 1, C2 i,... }, repectively, and let Vi A denote the voter intance that reolve the redundant input for the actuator job A i. Since all input are available before the tak i activated in an error-free cenario, meage tream that are delayed or omitted due to tranmiion or crah error are ignored during voting (Line 5 of Algorithm 1). In the wort cae, if no input i available on time to the voter due to error, the tak activation i kipped, i.e., the tak output for that iteration i omitted (Line 7). While computing the imple majority (Line 8), any tie in quorum ize are broken determinitically uing meage ID. We (peimitically) aume that corrupted meage replica are identical becaue it i a wort-cae cenario w.r.t. the voting protocol. In particular, if the number of corrupted meage exceed the number of correct meage, then auming identically corrupted meage implie that the voting outcome i corrupted, while in the cae of non-identically corrupted meage there i a high likelihood that correct meage till form the larget quorum. In practice though, whether or not corrupted meage are likely to be identical i highly ytem- and application-pecific. Random EMI normally doe not caue identically corrupted pattern and many ytem ue end-to-end checkum; the likelihood of identically corrupted meage i thu mall. In contrat, if the application payload i of boolean type or encoded uing only a few bit, the likelihood of identically corrupted meage i non-negligible. Furthermore, we aume that NCS hot are ynchronized uing a clock ynchronization protocol (uch a the Preciion Time Protocol [1]), and that tak and meage offet have been choen to account for the maximum clock ynchronization error. Without thi aumption, it i

5 A. Gujarati, M. Nari, and B. B. Brandenburg 16:5 much more challenging to enure replica determinim (e.g., imply aigning appropriate offet to tak and meage i inufficient) [47]. We alo require that all NCS tak are determinitic. Thu, given identical input and identical tate, any two enor (controller) tak replica produce identical enor (control) meage, unle one i affected by memory corruption. 3 Fault Model To lay the foundation for our analyi, we firt give a precie fault model. We model the EMI-induced raw tranient fault, i.e., bit-flip on the network and in hot memory, a random event following a Poion ditribution. Let P(x, δ, λ) denote the probability ma function of the Poion ditribution, i.e., the probability that x independent event occur in an interval of length δ when the arrival rate i λ. Let τ and λ i denote the peak rate of raw tranient fault affecting the network and each hot H i H, repectively. We define the probability that x raw tranient fault affect the network (repectively, hot H i ) in any interval of length δ a P(x, δ, τ) (repectively, P(x, δ, λ i )). In practice, the peak fault rate are empirically determined with meaurement or derived from environmental modeling auming wort-poible operating condition, and typically include afety margin a deemed appropriate by reliability engineer or domain expert. A a reult, a Poion proce i a good approximation of the wort-cae cenario, a previouly dicued by Broter et al. [13]. For intance, in the cae of network fault, τ i likely to exceed any tranient actual fault rate τ actual experienced in practice, which alo varie over time and/or baed on a ytem current urrounding. Thu, a per the Poion model, while the actual probability that the network experience at leat one tranient fault in any interval of length δ i given by x>0 P(x, δ, τ actual), we upper-bound thi probability in our analyi by x>0 P(x, δ, τ). That i, if τ > τ actual, then x>0 P(x, δ, τ) > x>0 P(x, δ, τ actual). 2 Raw tranient fault may manifet a program-viible retranmiion, crah, and incorrect computation error [6, 8], which are alo modeled probabilitically, a decribed below. Networking protocol incorporate explicit mechanim to mitigate the effect of tranient fault on the wire, e.g., error detection and correction in CAN [43]. Thu, we aume that network meage corruption are alway detected, but may reult in retranmiion error which may eventually lead to deadline violation. A in [12], we make the implifying (but afe) aumption that every tranient fault on the network caue a retranmiion. Thu, we define the retranmiion rate a τ, and the probability that x retranmiion occur in any interval of length δ a P(x, δ, τ). Given thi, an upper bound on the probability that a meage mie it deadline can be derived uing prior work [13, 55]. In thi work, we aume that an upper bound on the wort-cae deadline-mi probability of any meage intance belonging to any enor meage tream X x or any control meage tream U x i known and denote thi bound a B(X x ) or B(U x ), repectively. Crah error occur if the ytem uffer an EMI-induced corruption that caue an exception to be raied and the ytem to be rebooted, or that induce an unbounded hang that caue the ytem watchdog timer to trigger a reboot, e.g., ee [42]. A crahed ytem remain unavailable for ome time while it reboot and thu caue an interval in which meage are continuouly omitted. We aume that the recovery interval on each hot H i i upper-bounded by R i, which we aume alo include any delay that arie due to the need to reynchronize any application tate after a crah. 2 Thi baic fact can be proved by repreenting the cumulative denity function of the Poion ditribution in the form of an upper incomplete gamma function [5].

6 16:6 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem Prior tudie have hown that a large fraction of tranient fault have no negative effect [3, 7, 59]. We thu aume a derating factor that account for maked tranient fault, which can be determined empirically [41]. Let f i denote the derating factor for crah error on hot H i ; the peak rate of crah error on hot H i i then given by ρ i f i λ i. Uing the peak crah error rate, we model crah error like raw tranient fault a random event following a Poion ditribution. Thu, we define the probability that x crah error occur on hot H i in any interval of length δ a P(x, δ, ρ i ). Incorrect computation error may occur if a meage i corrupted before tranmiion (during preparation), before the network controller compute a checkum for ubequent error detection. Like crah error, auming a hot-pecific derating factor f i for incorrect computation error, the average error rate on hot H i i given by κ i f i λ i and the probability that x error occur in any interval of length δ i given by P(x, δ, κ i ). 3 Our notion of incorrect computation error doe not refer to oftware bug or Byzantine error. We refer to the interval during which a meage i at rik of corruption a it expoure interval. For tateful tak uch a a PID controller, the meage computation relie on both the current input and the application tate, and the latter could be affected by latent fault (i.e., tate corruption that have not yet been detected). Thu, the expoure interval of a meage depend on the mechanim in place to tolerate (or avoid) latent fault. If the hardware platform ue Error-Correcting Code (ECC) memory and proceor with locktep execution (common in afety-critical ytem), then the built-in protection uppre latent fault, and it uffice to conider the cheduling window of a meage (i.e., the duration from the meage creation to it deadline) a it expoure interval. If no uch architectural upport i available, then any relevant tate can be protected with a data integrity checker tak that periodically verifie the checkum of all relevant data tructure (and that reboot the ytem in the cae of any mimatch). The expoure interval of a meage then include it cheduling window and (in the wort cae) an entire period of the data integrity checker. We aume that the wort-cae expoure interval for each meage in X x, U y, and U i known in advance and denote it uing E(X x ), E(U y ), and E(U), repectively. Aumption. Baed on the tochatic nature of phyical EMI procee, we conider EMI-induced tranient fault, and hence baic meage error, to be independent. We do however account explicitly for correlated error that arie from the ytem architecture, e.g., determinitic replica will produce the ame wrong output if given the ame wrong input. We alo implicitly account for correlated urge in error rate acro all component ince we analyze peak rate for all component. For example, if a UAV with an FT-SISO control loop i flying through a trong radar beam, all replica of the control loop imultaneouly experience increaed rate of EMI. The propoed analyi i able to handle thi correlation becaue the derived upper bound on the failure rate i monotonic in all fault rate and applied auming peak fault rate, which in turn are determined uch that they exceed the fault rate expected in practice, epecially during uch high interference cenario. While evaluating the EMI-induced error dicued above, we aume that other ytem component are reliable, even though the NCS ubytem being analyzed may directly depend on them, e.g., the power ource, the phyical enor and the actuator, the controlled 3 The choice of Poion ditribution for modeling both crah and incorrect computation error i reaonable ince real-time tak are repeated, hort workload; thu, any generated meage i equally likely to be affected by an error, and a hot i equally likely to be crahed during any iteration of the tak (ee [36] for a mathematical bai for thi argument).

7 A. Gujarati, M. Nari, and B. B. Brandenburg 16:7 meage delayed meage omitted meage corrupted voter o/p incorrect meage delayed meage omitted voter o/p omitted meage corrupted voter o/p incorrect meage omitted meage corrupted voter o/p omitted actuation incorrect actuation omitted failed iteration Figure 2 Error probabilitie at different tage of a CAN-baed wheel control loop (ee 5 for detail). Arrow denote dependencie among error probabilitie of the different control loop tage. The error rate (per m) are τ 10 4 for the CAN bu, ρ i and κ i for each H i hoting enor and controller tak, and ρ a and κ a for the actuator tak hot H a. phyical plant, or the clock ynchronization mechanim. Thi aumption doe not imply that the propoed analyi i not ueful if a dependent component fail, rather it provide a FIT rate for one ubytem, which can then be compoed with the FIT of other dependent, dependee, or unrelated ubytem, e.g., uing a fault tree analyi. Thi i a common way of decompoing the reliability analyi of the whole ytem into manageable component. We alo aume that the network protocol guarantee atomic broadcat, i.e., meage are received conitently by either all hot, or none. While Byzantine error cenario violate thi aumption, e.g., [38], they occur with uch low likelihood that they are bet modeled a a eparate, additive failure ource and accounted for uing a eparate FIT analyi. Finally, recall from 2 that tak are aigned appropriate offet to enure equentiality, e.g., to enure that enor value are alway available (in an error-free execution) before any control tak replica i activated. In thi work, we aume that proceor cheduling on each hot i tatically checked and thu tak offet are correctly enforced. Alternatively, proceor cheduling delay due to tranient fault could be explicitly taken into account a an additional ource of failed control loop iteration, e.g., when upper-bounding the probability of a meage omiion (ee Definition 1 in 4). 4 Probabilitic Analyi We analyze the probability that the n th iteration of the control loop fail, for any n. A mentioned in 2, due to clock ynchronization and the atomic broadcat aumption, meage replica are identical in an error-free cenario, i.e., the meage in X n carry identical enor value and the meage in U n carry identical control command. However, due to incorrect computation error, one or more meage in X n may be corrupted. If the voter V n chooe a corrupted enor value, then all meage U n carrying the control command are alo corrupted. Meage in X n could alo be delayed or omitted due to tranmiion and crah failure, in which cae the voter V n work with fewer input. But if all the meage in X n are either delayed or omitted, the controller job C n have no input to work with, hence the meage U n are not prepared. Similarly, the controller to actuator information flow may alo be affected by error, reulting in A n output U n being corrupted or omitted. Thee dependencie are illutrated uing an example in Fig. 2. Baed on thi intuition, we next bound the probability that the final output U n i corrupted or omitted, in a bottom-up fahion and in mall tep of a few lemma each. We ue

8 16:8 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem P ( ) to denote exact probabilitie and Q( ) to denote upper bound on the exact probabilitie. In particular, we firt define the analyi a a function of the following exact (but unknown) probabilitie for each meage m: Definition 1. P (m omitted) denote the exact probability of an omiion. P (m delayed) denote the exact probability of a deadline violation. P (m corrupted) denote the exact probability of an incorrect computation. In addition, ince the effect of meage corruption on Algorithm 1 output alo depend on the application-pecific meage payload, the analyi initially alo aume the following exact (but unknown) probability. Definition 2. P (Majority incorrect I, C) denote the exact probability that, given a et of incorrect input I and correct input C, the SimpleMajority(I C) procedure in Algorithm 1 (Line 8) output an incorrect value. In each tep of the analyi, we enure that the derived probability i either independent of, or increaing in, thee exact error probabilitie. Thu, when intantiating the analyi uing upper bound on the exact probabilitie, we implicitly guarantee that the derived iteration failure probability upper-bound the actual iteration failure probability. Due to pace contraint, we do not give a proof of monotonicity in thi paper. We reviit the iue at relevant place where we explicitly add ome peimim to the analyi to enure monotonicity. Step 1. Analyzing the correctne of Vn y output. We evaluate the probability that a controller voter intance Vn y output an incorrect value becaue of corrupted input. Recall from 2 that X n denote the et of all enor meage replica that are input to Vn y. Let T n O n, D n, I n, C n, Z n denote a 5-tuple contrained by the following definition. Definition 3. T n O n, D n, I n, C n, Z n i valid if O n, D n, I n, C n, and Z n partition et X n : meage in O n are omitted; meage in D n are not omitted, but delayed due to retranmiion; meage in I n are neither omitted nor delayed, but are incorrectly computed; meage in C n are neither omitted, delayed, nor incorrectly computed; and meage in Z n may be omitted, delayed, or corrupted. In general, Z n denote the meage whoe fate i undecided, or in other word, each meage X y n Z n may till be omitted with probability P (X y n omitted), delayed with probability P (X y n delayed), and incorrectly computed with probability P (X y n corrupted). Thu, if meage X y n X n i guaranteed to be omitted due to hot crahe, then X y n O n. Similarly, if X y n i guaranteed to be tranmitted on time and without being incorrectly computed due to hot corruption, then X y n C n. Baed on Definition 1 3, we ue the following recurive expreion to compute the probability that V y n output an incorrect value becaue the majority of it input i corrupted. P ( V y n output incorrect O n, D n, I n, C n, Z n ) { P (Majority incorrect In, C n ) Z n Γ( O n, D n, I n, C n, Z n ) Z n where Γ( O n, D n, I n, C n, Z n ) ( P (X n omitted) P (Vn y output incorrect O n {Xn}, D n, I n, C n, Z n \ {X ( n} ) (1 P (X n omitted)) P (Xn delayed) P (Vn y output incorrect O n, D n {Xn}, I n, C n, Z n \ {Xn} ) ) + ) + (1)

9 A. Gujarati, M. Nari, and B. B. Brandenburg 16:9 ( ) (1 P (X n omitted)) (1 P (Xn delayed)) P (Xn corrupted) P (Vn y output incorrect O n, D n, I n {Xn}, C n, Z n \ {Xn} ) + ( (1 P (X n omitted)) (1 P (Xn delayed)) (1 P (Xn corrupted)) P (Vn y output incorrect O n, D n, I n, C n {Xn}, Z n \ {Xn} ) and X n denote the meage with the mallet ID in Z n (if Z n ). ) In each tep of the recurion, a ingle meage Xn Z n i either (i) omitted with probability P (Xn y omitted) and inerted into et O n ; (ii) not omitted but delayed with probability (1 P (Xn y omitted)) P (Xn y delayed) and inerted into et D n ; (iii) tranmitted on time, i.e., neither omitted nor delayed, but i incorrectly computed with probability (1 P (Xn y omitted)) (1 P (Xn y delayed)) P (Xn y corrupted) and inerted into et I n ; or (iv) tranmitted on time and without any corruption with probability (1 P (Xn y omitted)) (1 P (Xn y delayed)) (1 P (Xn y corrupted)), and thu inerted into et C n. The recurion terminate when all cae have been exhautively enumerated, i.e., when Z n and O n D n I n C n X n. Therefore, P (Vn y output incorrect,,,, X n ), a defined in Eq. 1, compute the exact probability that controller voter intance Vn y output an incorrect value. However, Eq. 1 i not monotonically increaing in the omiion and delay probabilitie, a required. It monotonicity in P (Xn omitted) and P (Xn delayed) depend on P (Xn corrupted). Thi i becaue the overall failure probability could be reduced by imply delaying or omitting a meage, if that meage i likely to be incorrectly computed and thu ha the potential to tilt the voting outcome in favor of an incorrect quorum. To remove thi dependency on P (Xn corrupted), we replace Γ( O n, D n, I n, C n, Z n ) in Eq. 1 with a lightly peimitic term Γ π ( O n, D n, I n, C n, Z n ) (notice the fifth term in the definition of Γ π ( O n, D n, I n, C n, Z n )), and define an upper bound (tated below) on the probability that controller voter intance Vn y output an incorrect value. 4 ( V y Q n output incorrect O ) { n, D n, I n, P (Majority incorrect In, C n ) Z n C n, Z n Γ π ( O n, D n, I n, C n, Z n ) Z n where Γ π ( O n, D n, I n, C n, Z n ) ( ) P (X n omitted) Q(Vn y output incorrect O n {Xn}, D n, I n, C n, Z n \ {X + n} ) ( ) (1 P (X n omitted)) P (Xn delayed) Q(Vn y output incorrect O n, D n {Xn}, I n, C n, Z n \ {X + ( n} ) ) (1 P (X n omitted)) (1 P (Xn delayed)) P (Xn corrupted) Q(Vn y output incorrect O n, D n, I n {Xn}, C n, Z n \ {Xn} ) + ( ) (1 P (X n omitted)) (1 P (Xn delayed)) (1 P (Xn corrupted)) Q(Vn y output incorrect O n, D n, I n, C n {Xn}, Z n \ {Xn} ) + ( ) (1 P (X n omitted)) P (Xn delayed) P (Xn corrupted) + P (Xn omitted) P (Xn corrupted) + Q(Vn y output incorrect O n, D n, I n {Xn}, C n, Z n \ {Xn} ) and X n denote the meage with the mallet ID in Z n (if Z n ). Q(Vn y output incorrect,,,, X n ) thu yield an upper bound on the probability that voter intance Vn y output an incorrect value. For convenience, we let Q(Vn y output incorrect) Q(Vn y output incorrect,,,, X n ) in the following. (2) 4 See the appendix for a proof of monotonicity of Eq. 2.

10 16:10 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem Step 2. Analyzing whether Vn y omit it output. We evaluate the probability that a controller voter intance Vn y omit it output becaue all it input were either delayed or omitted, i.e., the pecial cae in Algorithm 1 (Line 7). Once again, we tate a recurive expreion to compute the probability, imilar to the one ued in Step 1. ( ) ( V y P n output omitted O ) Λ O n, D n, I n, C n, Z n Z n n, D n, I n, C n, Z 1 I n C n (3) n 0 I n C n where Λ( O n, D n, I n, C n, Z n ) ( ) P (X n omitted) P (Vn y output omitted O n {Xn}, D n, I n, C n, Z n \ {X + ( n} ) ) (1 P (X n omitted)) P (Xn delayed) P (Vn y output omitted O n, D n {X n}, I n, C n, Z n \ {Xn} ) + ( ) (1 P (X n omitted)) (1 P (Xn delayed)) P (Xn corrupted) P (Vn y output omitted O n, D n, I n {Xn}, C n, Z n \ {Xn} ) + ( ) (1 P (X n omitted)) (1 P (Xn delayed)) (1 P (Xn corrupted)) P (Vn y output omitted O n, D n, I n, C n {Xn}, Z n \ {Xn} ) and X n denote the meage with the mallet ID in Z n (if Z n ). P (Vn y output omitted,,,, X n ) thu yield the exact probability that voter intance Vn y omit it output. Note that Eq. 3 doe not depend on the correctne of Vn y input, but only on the timeline of it input, unlike the imple majority procedure in Eq. 1. Hence, Eq. 3 monotonicity in P (Xn omitted) and P (Xn delayed) doe not depend on P (Xn corrupted), unlike Eq. 1. A a reult, the ue of a peimitic term uch a Γ π ( O n, D n, I n, C n, Z n ) in Eq. 2 i not required in thi cae. For convenience, we let P (Vn y output omitted) P (Vn y output omitted,,,, X n ). Step 3: Analyzing the actuator voter intance Vn A. We bound the probability that Vn A output an incorrect value, becaue the majority of it input i corrupted, or that it doe not chooe anything, becaue all it input are either omitted or delayed. Since all controller voter intance V n operate on the ame input value, if a correct voter intance Vn y output an incorrect value becaue of wrong input, it implie that all correct voter intance in V n output incorrect value. In uch a cenario, the actuator voter Vn A i guaranteed to get only incorrect control meage, ince all of the control meage will be prepared uing the corrupted enor value. A imilar property hold for the controller voter output omiion. Proper deadline and offet aignment guarantee that, in an error-free cenario, meage in X n are tranmitted before the voter intance in V n are activated. Thu, each voter intance can decide locally whether a meage wa received pat it deadline (in which cae it i dicarded, recall Algorithm 1). A a reult, if a controller voter intance Vn y doe not chooe any value becaue all it input are delayed or omitted, then all controller voter intance in V n do not chooe any value, either. Thu, no output i generated by the controller tak replica and the actuator voter omit it output, too, which reult in a kipped actuation. Let Q(Vn A output incorrect) denote an upper bound on the probability that voter intance Vn A output an incorrect value, conditioned on the aumption that the enor input of the controller voter intance V n did not reult in a corrupted output. Similarly, let P (Vn A output omitted) denote the probability that voter intance Vn A output i omitted, conditioned on the aumption that the enor input of the controller voter intance V n did

11 A. Gujarati, M. Nari, and B. B. Brandenburg 16:11 not reult in an omitted output. Both Q(V A n output incorrect) and P (V A n output omitted) can be derived uing the recurive procedure dicued in Step 1 and 2, repectively, by replacing the et of voter input X n with U n (recall from 2 that U n denote the et of all input to V A n ). The cae that the enor input of the controller voter intance V n reult in a corrupted or omitted output i accounted for in Step 5. Step 4: Analyzing the final output U n. We firt bound the probability that the actuation during the n th control loop iteration i incorrect (Lemma 4), followed by the probability that it i omitted (Lemma 5), and finally the joint probability of both event (Lemma 6). For brevity, we let φ 1 Q(Vn y output incorrect), φ 2a Q(Vn A output incorrect), φ 2b P (U n corrupted), ω 1 P (Vn y output omitted), ω 2a P (Vn A output omitted), and ω 2b P (U n omitted). Lemma 4. The probability that the actuation during the n th control loop iteration i incorrect i at mot φ 1 (1 + φ 2a φ 2b ) + φ 2a + φ 2b. Proof. We conider two cae baed on whether the enor input to any voter intance Vn y reult in corruption of the controller voter output (cae 1) or not (cae 2). The probability that cae 1 occur i φ cae1 P (Vn y output incorrect). For thi cae, ince the enor input to voter intance Vn y reult in corruption of it output, voter intance in all controller tak chooe an incorrect output. Thu, all control command tranmitted were incorrect, thu it i guaranteed that the actuation during the n th control loop iteration i incorrect. Thu, the conditional probability in thi cae i φ cond1 1. The probability that cae 2 occur i φ cae2 1 φ cae1. For thi cae, the conditional probability that the actuation during the n th control loop iteration i incorrect depend on two ource: (a) voter intance Vn A output can be incorrect, and (b) A hot can be affected by incorrect computation error. The probability for cae (a) i φ cae2a P (Vn A output incorrect). The probability for cae (b) i φ cae2b P (U n corrupted). Cae (a) and (b) are independent: (a) occur becaue input to Vn A were corrupted due to incorrect computation error on the controller tak hot, wherea (b) occur due to incorrect computation error on the actuator tak hot. Thu, uing theorem P (A 1 A 2 ) P (A 1 )+P (A 2 ) P (A 1 ) P (A 2 ) for independent event A 1 and A 2, the conditional probability for cae 2 i φ cond2 φ cae2a + φ cae2b φ cae2a φ cae2b. By the law of total probability, the probability that the actuation during the n th control loop iteration i incorrect i given by φ cae1 φ cond1 + φ cae2 φ cond2. Upon expanding φ cond1, φ cond2, and φ cae2, and then rearranging the reulting expreion w.r.t. φ cae1, we get ( ) φcae1 (1 φ φ cae1 φ cond1 + φ cae2 φ cond2 cae2a φ cae2b + φ cae2a φ cae2b ). + φ cae2a + φ cae2b φ cae2a φ cae2b Further, upon dropping any negative term for monotonicity, and ince φ cae1 φ 1, φ cae2a φ 2a, and φ cae2b φ 2b, we have the following upper bound: φ cae1 φ cond1 + φ cae2 φ cond2 φ 1 (1 + φ 2a φ 2b ) + φ 2a + φ 2b. Lemma 5. The probability that the actuation during the n th control loop iteration i delayed or omitted i at mot ω 1 (1 + ω 2a ω 2b ) + ω 2a + ω 2b. The proof of Lemma 5 i analogou to that of Lemma 4 and i thu omitted. In Lemma 6, we compoe the probabilitie derived in Lemma 4 and 5 to derive the probability that the n th control loop iteration fail, i.e., that the actuation during thi iteration i either incorrect or delayed (or omitted). We do not aume that the probabilitie derived in Lemma 4 and 5 are independent, ince it i poible that an omitted control meage tilted the majority in favor of the correct quorum, thereby reducing the probability that the actuation i incorrect.

12 16:12 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem Lemma 6. The probability that the n th control loop iteration fail i at mot Q ( n th control loop iteration fail ) ( ) φ1 (1 + φ 2a φ 2b ) + φ 2a + φ 2b +. (4) ω 1 (1 + ω 2a ω 2b ) + ω 2a + ω 2b Proof. Follow from Lemma 4 and 5. In ummary, Step 1-4 account for all direct and indirect dependencie between the individual meage error event and the final actuation of the controlled plant, and the derived Q ( n th control loop iteration fail ) automate propagation of the failure probability along thi dependency tree. Although the analyi ha exponential time complexity in the number of enor meage tream X n and the number of controller meage tream U n (due to the branching recurion in Eq. 2 and 3), ince the number of replica of any tak i likely mall, i.e., typically under ten, the analyi can be quickly performed. Upper-bounding the failure probability. Since exact meage error probabilitie are impoible to obtain, we intantiate the above analyi with upper bound on the exact probabilitie. The analyi i monotonically increaing in the meage error probabilitie, and thu remain ound depite the ue of thee upper bound. We next define upper bound on the meage error probabilitie for any enor meage Xn. y The bound for any control meage Un y and actuator tak output meage U n are analogouly defined. The probability that any enor meage Xn y i delayed beyond it deadline i bounded by P (Xn y delayed) B(X y ) (a defined in 3). Let the hot on which Xn y ender tak i deployed be denoted H a. Regarding meage omiion, uppoe Xn y i expected to be cheduled for tranmiion at the earliet by time t and at the latet by time t + J (where J denote the maximum releae jitter of the meage). Since R a i the maximum time to recover from a crah error on hot H a, if there i at leat one crah error during the interval [t R a, t+j), Xn y arrival may be kipped. Thu, P (Xn y omitted) x>0 P(x, R a +J, ρ a ). Regarding meage corruption due to incorrect computation error, recall from 2 that the expoure interval for enor meage Xn y i upper-bounded by E(X y ). Thu, Xn y may be corrupted if there i at leat one incorrect computation error in thi interval. Thu, P (Xn y corrupted) x>0 P(x, E(Xy ), κ a ). The probability P (SimpleMajority incorrect I, C) i upper-bounded by making the wort-cae aumption that incorrect input in I are identically faulty. Recall from Definition 2 that C and I denote the et of correct and incorrect input, repectively, to the SimpleMajority(I C) procedure in Algorithm 1. Auming n c C, n i I, and that 0 C I denote the meage in C I with the mallet ID, we obtain the following bound. ( ) SimpleMajority 1 (n i > n c ) (n i n c 0 0 I) Q incorrect I, C 0 n i n c 0 0 C (5) 0 n i < n c n i n c 0 Lemma 7. Eq. 5 upper-bound the probability that procedure SimpleMajority(I C) output in Algorithm 1 (Line 8) i incorrect. Proof. If n i > n c, the larget-ized quorum belong to incorrect meage, and Algorithm 1 output i incorrect with probability 1. If n i n c 0, there are two larget-ized quorum. If meage 0 with the mallet ID i incorrect ( 0 I), Algorithm 1 chooe an incorrect output with probability 1. Otherwie ( 0 C), it chooe an incorrect output with probability 0. If n i < n c, the larget-ized quorum belong to correct meage, and Algorithm 1 output i correct, i.e., incorrect with probability 0. If n i n c 0, the voter ha received no input, o the probability of chooing an incorrect output i 0.

13 A. Gujarati, M. Nari, and B. B. Brandenburg 16:13 The IID property. Since each of the upper bound defined above i independent of n, Eq. 4 can be iteratively unfolded until it conit only of term that are independent of n. The bound i thu identical for any control loop iteration. In addition, the upper bound are derived under wort-cae aumption with repect to interference from other meage on the network [13, 16]; and failure of the n th control loop iteration, defined a a deviation from an error-free execution of that iteration, i independent of whether pat iteration encountered any failure or not. Thu, the bound obtained uing Eq. 4 for any two iteration n 1 and n 2 are mutually independent a well. A a reult, when Q(n th control loop iteration fail), which i monotonic in the error rate, i intantiated with the aforementioned upper bound on the error rate, it atifie the IID property with repect to n. FIT analyi. We ue the probability of a failed control loop iteration, i.e., the reult of Lemma 6, to derive the NCS FIT rate. Firt, we derive a lower bound on the mean time to failure (MTTF) of the control loop. Recall from 1 that a control failure occur if the control loop violate it (m, k) pecification. We model thi problem in the form of a well-tudied a-within-conecutive-b-out-of-c:f ytem model [31], and leverage exiting reult [53] (which depend on the IID property of the iteration failure probability) on the reliability analyi of thi ytem model to afely lower-bound the MTTF. Given an MTTF lower bound MT T F LB in hour, the FIT rate i computed a 10 9 /MT T F LB [56]. The full derivation and evaluation of the FIT analyi i available online [24]. 5 Evaluation The objective of the evaluation i threefold. Firt, in order to undertand the accuracy of our approach, we compare the propoed analyi with imulation ( 5.1). Second, we demontrate the ability of our analyi to reveal and quantify non-obviou difference in the reliability of workload with different (m, k) requirement and ubject to error rate ( 5.2). And third, we illutrate the utility of our analyi in a deign-pace exploration context by comparing FIT of different replication cheme ( 5.3). To implement the analyi, we extended the SchedCAT [10] library to upport our ytem model for CAN-baed NCS, and implemented the propoed analyi on top. All computation related to the analyi were carried out at a preciion of 200 decimal place uing the mpmath Python library for arbitrary preciion arithmetic [30]. A the underlying timing analyi of the network, we ued Broter et al. probabilitic repone-time analyi for CAN [12]. We alo implemented a imulation of a CAN-baed NCS that mimic the ytem model decribed in 2 along with CAN network tranmiion protocol (ee [43] for a detailed decription). We ue an active upenion workload for our experiment ince it play an important role in enuring the tability of a vehicle, and ince robutne of uch control ytem under fault ha been thoroughly invetigated in the pat. For example, Li in hi thei [35] dicue related work in the context of actuator delay and fault, and propoe a fault-tolerant controller deign for guaranteeing aymptotic tability. We bae our experiment on the CAN-baed active upenion workload tudied by Anta and Tabuada [4], ince it nicely matche our SISO NCS model. However, while Anta and Tabuada aume hard contraint and vary the control loop periodicity for improved bandwidth allocation, our objective i to explore the reliability of the control loop when aigned different (m, k)-firm configuration (ynthetically choen in thi paper) and for different fault parameter. The workload conit of four control loop (L 1, L 2, L 3, and L 4 ) correponding to the control of four wheel (W 1, W 2, W 3, and W 4 ) with magnetic upenion (period 1.75 m),

14 16:14 Quantifying the Reiliency of Fail-Operational Real-Time Networked Control Sytem two hard real-time meage that report the current in the power line cable (period 4 m) and the internal temperature of the coil (period 10 m). In addition, we aumed the preence of a clock ynchronization meage with a period of 50 m [21] and a oft real-time meage reponible for logging with a period of 100 m. The logging meage carried payload of eight byte each, the control loop meage carried payload of three byte each, and the remaining meage carried one-byte payload. Conidering a bu rate of 1 mbit/, the workload reulted in a total bu utilization of 40%. The clock ynchronization meage tream had the highet priority, followed by the current and temperature monitoring meage tream, the control meage tream, and lat, the logging meage tream. The recovery time from a crah wa et to R h 1 for each hot H h H, and the expoure interval of each meage tream wa et to ten time it period to reflect the poibility of latent error. The error rate and the (m, k) pecification ued in each experiment are mentioned in the correponding graph. All error rate in the following are reported a the mean number of error per m. For context, Ferreira et al. [20] and Rufino et al. [50] reported peak tranmiion error rate range from 10 4 in aggreive environment to in lab condition, and a per Hazucha and Svenon [27], a 4 Mbit SRAM chip ha a fault rate of approximately The error rate ued in the following experiment have imilar order of magnitude. Since the actuator tak i not replicated, it hot wa aumed to be heavily hielded and thu aigned negligibly low crah and incorrect computation error rate. 5.1 Experiment 1: Simulation v. Analyi To ae the accuracy of the propoed analyi, we compared the analytically-derived iterationfailure probability bound ( 4) with an etimate of the mean iteration failure probability obtained through imulation. The peimim incurred by the FIT analyi wa already evaluated in prior work [24] and found to be acceptably mall, and i not conidered here. Recall from 4 that: (1) the analyi firt upper-bound the control loop iteration failure probability a a monotonic function of the exact meage error probabilitie; and (2) ince it i impoible to determine the exact meage error probabilitie, a afe upper bound on the iteration failure probability i then obtained by intantiating the monotonic function from (1) with upper bound on the exact meage error probabilitie (derived uing the Poion fault model in 3). To eparately evaluate the peimim incurred in tep (1) and (2), we ued two different imulator verion Sim-v1 and Sim-v2 in thi experiment. In the imple verion (Sim-v1), for each enor meage (and imilarly for each control meage), the meage error probabilitie were known to the imulator. Thu, each time any meage i activated, the imulator draw a number uniformly at random from the range [0, 1], compare it with the repective meage error probabilitie to decide whether the meage i affected by that error type, and if the meage i affected, imulate the correponding error cenario. Thu, Sim-v1 doe not actually imulate Poion procee, nor doe it imulate the CAN protocol, but it help to iolate the peimim incurred in tep (1). Sim-v2 i more complex than Sim-v1, and imulate the entire NCS along with the CAN tranmiion protocol. Separate Poion procee are ued to generate the repective fault event on each hot and on the network. Thee fault event may manifet a meage error if they coincide with the meage lifetime, e.g., a an incorrect computation error if they coincide with the meage expoure interval and a retranmiion error if they coincide with the meage network tranmiion interval. Sim-v2 evaluate the peimim incurred when upper-bounding the meage error probabilitie a a function of the raw tranient fault rate uing the Poion model, e.g., when uing the Poion-baed CAN timing analyi [13] to determine bound on deadline violation probabilitie. It alo evaluate whether thi

15 A. Gujarati, M. Nari, and B. B. Brandenburg 16:15 P(failed iteration) Analyi Sim1 Sim # enor and controller tak replica P(failed iteration) Analyi Sim1 Sim # enor and controller tak replica P(failed iteration) Analyi Sim1 Sim # enor and controller tak replica P(failed iteration) (a) τ , ρ i 10 4, κ i Analyi Sim1 Sim # enor and controller tak replica P(failed iteration) (b) τ , ρ i 10 20, κ i Analyi Sim1 Sim2 58.7% 65.9% 73.1% 80.3% 87.6% 94.8% CAN bu utilization P(failed iteration) (c) τ 3, ρ i 10 20, κ i Analyi Sim1 Sim Reboot time (m) (d) τ , ρ i 10 5, κ i 10 5 (e) τ , ρ i 10 5, κ i 10 5 (f) τ , ρ i 10 4, κ i Figure 3 Comparing the iteration failure probability bound derived from the analyi with the etimate derived from imulation verion Sim-v1 and Sim-v2. The vertical error bar along with the imulation etimate denote 99% confidence interval. Inet (a), (b), (c), and (d) illutrate the variation in the iteration failure probability when the number of enor and controller tak replica of L 1 are increaed from one to five, for different et of error rate. Inet (e) and (f) illutrate the impact of increaing CAN bu utilization and reboot time, repectively, for three replica. peimim ignificantly impact the overall iteration failure probability bound. Both Sim-v1 and Sim-v2 make the wort-cae aumption that any two faulty meage copie are identical, a in the analyi. We compared the analyi, Sim-v1, and Sim-v2 for four different et of error rate and replication factor. We ued higher error rate for thi experiment than can be realitically expected (and much higher than thoe ued in the later experiment) a otherwie the imulation would be extremely time-conuming. To undertand the effect of individual error type, we firt compared three cenario in which repectively only one of the three error type wa aigned a ignificant rate, i.e., ρ i 10 4, κ i 10 4, and τ 3, repectively, wherea the other were aigned negligible value, i.e Additionally, we evaluated a fourth cenario where all three error type have ignificant rate, i.e., ρ i 10 5, κ i 10 5, and τ Finally, to undertand the effect of bu utilization and reboot time on the analyi, we compared the analyi, Sim-v1, and Sim-v2 for different CAN bu utilization (by auming increaed meage payload ize) and for different reboot time (100 m-2000 m), with a replication factor of three. The reult are hown in Fig. 3a 3f. Several trend can be clearly een. Firt, in all evaluated cenario, the analyi reult alway track Sim-v1 extremely cloely, which indicate that any peimim introduced in tep (2) to enure monotonicity of the model with repect to the error rate i negligible. The reult hown in Fig. 3a, 3b, and 3d further how that the analyi track Sim-v2 quite cloely, too, provided that the underlying CAN timing analyi i not the bottleneck (i.e., if meage delay are not the dominant ource of failure). Specifically, we oberve that the full analyi, including tep (1), reult in le than an order of magnitude difference between