On Threshold Optimization in Fault Tolerant Systems

Transcription

1 On Tresold Optimization in Fault Tolerant Systems Fredrik Gustafsson, Jan Åslund, Erik Frisk, Mattias Kryser, Lars Nielsen Department of Electrical Engineering Linköping University, Sweden bstract: Fault tolerant systems are considered, were a nominal system is monited by a fault detection algitm, te nominal system is switced to a backup system in case of a detected fault. Conventional fault detection is in te classical setting a trade-off between detection probability false alarm probability. F te considered fault tolerant system, a system failure occurs eiter wen te nominal system gets a fault tat is detected, wen te fault detect signals an alarm te backup system breaks down. Tis means tat te trade-off f tresold setting is different depends on te overall conditions, te caracterization understing of tis trade-off is imptant. It is sown tat te probability of system failure can be expressed in a general fm based on te probability of false alarm detection power, based on tis fm te influence ratio is introduced. Tis ratio includes all infmation about te supervised system te backup system tat is needed f te tresold optimization problem. It is sown tat te influence ratio as a geometrical interpretation as te gradient of te receiver operating caracteristics (ROC) curve at te optimal point, furterme, it is te tresold f te optimal test quantity in imptant cases. Keywds: fault detection, fault tolerant control, tresold selection, system failure. INTRODUCTION Safety is of maj concern in many applications [Vil92], because of tat fault tolerance can be introduced by means of a back-up system. Ten te key mecanism is situation classification, [BKLS03], followed by a decision to switc to te back-up system. Wen designing suc a system, it is very imptant from an application point of view to underst te safety implications of design coices like tresold selection [ÅBF + 07]. Suc safety analysis is te topic of tis paper, aiming at obtaining useful explicit caracterizations. Consider te design of a fault tolerant system in Figur. fault detect monits te perfmance of a nominal system. Tis nominal system can be a certain sens, a computer tat generates a control signal, (part of) a plant. Te idea is tat it is possible to switc to a backup system wenever appropriate. Here, te backup system is used after an alarm is issued by te fault detect. Te risk after suc a decision is tat, befe maintenance is practically possible, te backup system breaks down after wic tere is no remedy. Te navigation system in te Swedis figter Gripen is designed as in Figur. stard inertial navigation system, comprising an inertial measurement unit (IMU) wit suppt senss, is te nominal system. Te steering system as its own set of IMU suppt senss, from wic an HRS (adaptive eading reference system) Tis wk as been perfmed witin te MOVIII excellence center funded by te Swedis Strategic Researc Council System 2 (Backup) u t System (Nominal) z 2 t z t yt Switc Detect z t larm Fig.. Fault tolerant system, were te nominal system is switced to te backup system 2 after te fault detect signals f an alarm. provides an independent backup navigation system. Te detect is ere called XMON (cross moniting), it compares te two navigations systems using statistical decision algitms. One first fundamental question is of course if te total system becomes safer wen a diagnosis function a backup system are introduced, if so, by ow muc? er question is ow to fmulate specification requirements on te diagnosis algitms so tat overall system safety is as good as possible [ÅBF + 07]. Tis also naturally leads to te question of ow to select internal design parameters in te diagnosis algitms, like tat of selection of a tresold tat balances te rates of missed detection false alarm.

2 To get a le on tese questions it is necessary to ave a quantitative metod, tis will also be a requirement from government, e.g. wen declaring air wtiness f aircraft. It sould be ed tat te main concern ere is te interplay between safety algitms, tat tis sould be confused wit te me studied problem on safety of software. It is ere assumed tat te software is a crect coding of te specified algitm following te procedures f implementation of safety critical systems. Te purpose of tis paper is to analyze fault tolerant systems, like te one illustrated in Figur, to obtain as muc useful insigt as possible. Tis means tat once te problem is fmulated, different pats of solution are investigated, wic gives alternative caracterizations of te optimal design. Section 2 introduces fault detection recapitulates te main perfmance measures. Section 3 gives te basic fmulation exps te probability of system failure into an expression including false alarm probability detection probability. Based on tat expression, an influence fact is defined. It is sown in Section 4 5 ow te influence fact caracterizes overall system caracteristics te tresold selection. Finally, te conclusions are drawn in Section FULT DETECTION ND PERFORMNCE MESURES In noisy environments, a typical fault detect consist of a test statistic, also called a test quantity, a cresponding tresold. Te test quantity, T(observations) is a function of te observations te detect alarms if te test statistic exceeds te tresold, i.e. if T(observations) > () Te idea is tat te test statistic is close to zero in te fault free case, but in te faulty case [Ber85]. Tus if T > te fault detect alarms a fault as been detected. Two commonly used perfmance measures f fault detects are te probabilities of false alarm P F () = P(T > no fault) (2) te detection power of a fault f of size f 0 P D () = P(T > f = f 0 ). (3) Te fault size f 0 will be a representative size of te faults causing failures. Te dependence of te fault size on te detection power will terefe be explicit in te continuation. Ideally, te false alarm probability sould be zero te probability of detection one. 3. SYSTEM FILURE ND FULT DETECTION Te objective is to analyze fault tolerant systems tat include a fault detection system te system in Figur will be used to illustrate basic concepts. discussion on me general systems is found in Section 5. F te system in Figur, te nominal system is automatically switced to te backup system wen te fault detect alarms. Te alarm may be crect false te question is ow to select te tresold of te fault detect to minimize te probability of system failure (SF). Since te detection system consists of a test quantity tat alarms if it exceeds a given tresold, te studied optimization problem is ten min P(SF) (4) Te event SF is a logical consequence of oter events. F te example system in Figur tere are two mutually exclusive cases were te system fails: Nominal system breaks down ( NOM) witout being detected ( LRM). Te fault detect alarms (LRM) after wic te backup system breaks down ( BCKUP). Tis example will be used in te following to illustrate properties tat old in a me general setting. Te probability f system failure is tus P(SF) = P( LRM NOM)+ P(LRM BCKUP) (5) Note tat all probabilities are conditioned on situation dependent facts, suc as operation time, operational regulations, maintenance programs, etc. Exping te first term in (5) gives P( LRM NOM) = P( LRM NOM)P( NOM) = ( P D )P( NOM) ssume tat alarm failure of te backup system are independent events, ten te second term in (5) is P(LRM BCKUP) = P(LRM)P( BCKUP) were P(LRM) = P(LRM NOM)P( NOM) + P(LRM NOM)P(NOM) = P D P( NOM) + P F P(NOM) Summing up, te probability f system failure in te example is given by were P(SF) = αp F βp D + γ (6) α = P(NOM)P( BCKUP) β = P( NOM)P(BCKUP) γ = P( NOM) One can e tat P(SF) is a linear function of te probabilities P F P D in te example. Tis property also olds in me general cases were te fault tolerant system switces between different configurations depending on alarm state. Tis is studied furter in Section 5. Te expression (6) will be used in te following section to analyze te optimization problem (4). Te influence ratio λ is defined as te ratio of te influence facts f te probabilities P F P D in (6), i.e. λ = α (7) β To solve te optimization problem (4), in der to find te optimal tresold, is equivalent to solving te problem min λp F () P D () Tis means tat te ratio λ includes all infmation about te supervised system, te backup system, te fault tolerant control strategy tat is needed f te tresold optimization problem.

3 4. THRESHOLD SELECTION Tis section will describe ow te parameter λ appears explicitly in two different contexts concerning optimal tresold selection. Te first is a geometric interpretation of λ in a tool, called ROC curve, f evaluating perfmance of a test. Te second concerns determining te value of an optimal tresold wen an optimal test, a likeliood ratio test, is used. 4. Te ROC curve te influence ratio λ Given a test statistic T, te false alarm probability detection power depend on te tresold value as can be seen in (2) (3). Tresold selection typically is a compromise between obtaining low false alarm probability a ig detection power. Tis compromise can be seen in a so called receiver operating caracteristics (ROC) curve, were P F () is plotted against P D (). ROC curves can also be used to compare different test statistics T i. F example, f a given false alarm probability te best test statistic is te one wic gives te igest power P D (). Tat is, te ROC curve is a convenient evaluation tool in fault detection. Te ROC curve f a test tat will be used in Section 4.2 is plotted in Figure 2. Fig. 2. Example of an ROC curve. Differentiating (6) wit respect to we get te following condition f te optimal tresold α dp F() d = β dp D() d wic gives tat dp D = α dp F β = λ Tus, f te optimal tresold it olds tat te gradient of te ROC curve equals te influence ratio λ introduced in (7). Te constant λ is dependent only on te pri probabilities of component failures. 4.2 Optimal tresold selection te influence ratio λ n optimal tresold is determined wit respect to a given test quantity. Here, a quite generic setup f te detection problem is used te specific test tat is used is an optimal detection test in te detection setup. simple but still quite generic model f te fault detection problem is to decide weter a computed residual is zero mean. Te two cresponding ypoteses are given by H 0 : r k = e k, H : r k = + e k. (8a) (8b) Here, r k, k =,...,N, represent different residuals over time / space, is a constant given by te fault response of a representative fault, e k is wite Gaussian noise wit known variance. Many applications can be recast into tis model. Te Neyman-Pearson teem [Le9, Kay98] states tat an optimal test quantity is given by te likeliood ratio T(r :N ) = P(r :N H ) P(r :N H 0 ) > (9) were is te tresold. Utilizing te witeness Gaussian property of te noise we get T(r :N ) = e (2πσ 2 ) N/2 e (2πσ 2 ) N/2 = e N 2 N k= 2r k 2σ 2 N k= (r k )2 2σ 2 N k= (r k )2 2σ 2 > Taking logaritm simplifying give te test quantity T(r :N ) = N r k > σ2 N N log + 2 (0) k= wic is distributed accding to ) N (0, σ2 under H 0 N T(r :N ) ) N (, σ2 under H N Now te optimal test quantity f te detection setup (8) as been determined. Next step is to determine te optimal tresold, i.e. te tresold tat minimizes P(SF). F tis, let Q(x) dee te nmal cumulative distribution function were Q(x) = x ϕ(x) dx ϕ(x) = 2π e x2 /2 Te probability f false alarm detection are ten ( ) P F = P(T(r :N ) > H 0 ) = Q (a) ( ) P D = P(T(r :N ) > H ) = Q (b) Wit te expressions f P F P D it is straigtfward to derive te expression f te ROC curve by ing tat

4 = σ 2 P D = Q N Q ( P F ) ( Q ( P F ) ) N 2 Te last expression sows tat P D is a function of te desired P F SNR only. Te optimal tresold is derived by substituting () into (6) differentiating wit respect to. Te condition f an optimal is ten βϕ ( ) = αϕ ( wic is equivalent to ) / ) ( )2 exp ( 2σ 2 exp ( 2 /N 2σ 2 = λ /N Taking logaritm gives 2 2 2σ 2 /N = log λ solving f gives te solution = 2 + σ2 N log λ Comparing te solution wit (0) gives tat = λ tus, te optimal tresold is in fact equal to te influence ratio λ. In te calculations, it is assumed tat λ is positive. In te example in Figur tis assumption is always fulfilled. F me general systems, tis migt always be te case. See Section 5 f furter discussions. 4.3 Discussion In Section 3 te influence ratio λ was introduced. Sections ave sown tat λ is of central imptance concerning tresold selection. Now, te example in Figur will be used to furter discuss te results. Let p p 2 dee te probabilities of failure of te nominal backup system respectively. Ten, te probability of system failure, as a function of detection tresold, is P(SF()) = ( p )p 2 P F () p ( p 2 )P D ()+p (2) Tus, te influence ratio is λ = ( p )p 2 ( p 2 )p F small probabilities p i we ave te approximation λ p 2 p Tis means tat, f any designed test quantity, at te optimal point on te ROC curve it olds tat dp D p 2 dp F p Te geometric interpretation is sown in Figure 3 were te system property gives te influence ratio te designed test te ROC curve. It can be seen in te figure tat a larger influence ratio, i.e. a relatively me unreliable backup system, leads to tat te diagnostic system sould σ 2 ) priitize false alarms compared to detection perfmance. P D p P F Fig. 3. Geometrical interpretation of te influence ratio in a ROC curve. Wen considering a me specific test quantity, like in Section 4.2, te influence ratio is equal to te optimal tresold. F te example p 2 /p wic means tat te detection test (9) becomes P(r :N H ) P(r :N H 0 ) > p 2 p Te rigt side is a ratio between te probabilities of failure f te backup nominal systems. Te left side is, similarly, a ratio between probabilities of te observations given te no fault mode te faulty mode. 5. GENERL SYSTEMS Discussions in previous sections ave mainly focused on te system in Figur. Tis section will discuss ow me general systems can be led sow tat te results apply directly. Te analysis was based on te fact tat we ad te following expression f te probability f system failure: P(SF) = αp F βp D + γ quite general structure of a fault tolerant system is tat we switc between different system configurations depending on alarm state. strategy may f example be to switc to a backup system to reconfigure te control strategy in case of an alarm. First we will illustrate te results f a strategy f fault tolerance wit a representative example. Ten we sow tat te linearity property of (6) olds in a general setting. Finally, we give some examples a discussion on cases were te fault detection system does contribute to fault tolerance. 5. nalysis of a fault tolerant system In tis section an example is studied to sow ow te analysis can be extended to me complex systems. Te system relies on tree senss, s,...s 3 is designed p 2

5 suc tat it fails if sens s sens s 2 s 3 fails. To improve system reliability, a diagnosis system is implemented tat supervises sens s. In case of a fault on s is detected, te quantity tat s measures is instead estimated by an observer tat uses sens s 2 as feedback. fault tree tat describes te complete logic f wen a system failure occurs can be seen in Figure 4. Events e i alarm activated =active bad prediction bad prediction new event = bad value alarm deactivated =active bad value system failure e 2 e 3 alarm P SF Fig. 5. Te probability P(SF) plotted as a function of te tresold. Tese two limits crespond to values 0 of te tresold in (9). observer fault e 2 F D P D 0.9 observer faulty 0.8 e 2 Fig. 4. Fault tree f te example system. represents failure of sens s i, event F false alarm, event D tat te diagnosis system detects a fault, event observer faulty means tat te observer delivers faulty estimates even toug tere is no fault in te system. Tis may be, f example, due to a po process model. Te fault tree was derived using te approac in [ÅBF + 07]. It is assumed tat all senss fail independently. To be able to make a probabilistic analysis of te system, we introduce te probabilities P(e i ) = P(sens s i fails) = p si Te fault detection system is caracterized by te parameters σ 2 =, =, N = 5 from Section 4.2. Te probabilities used in te example are set to p obs = 0 2, p s = 2 0, p s2 = 0 3, p s3 = 0 3 Now, computing te probability f system failure gives were P(SF) = αp F βp D + γ α = ( p s )(p obs p s3 + p s2 ( p obs p s3 )) β = p s p s3 ( p obs )( p s2 ) γ = p s (p s2 + p s3 p s2 p s3 ) Tus, te linearity property of P(SF) olds in tis case it will be proven below tat tis also olds in a general setting. In Figure 5, te probability P(SF) plotted as a function of te tresold were te minimum is clearly visible. Te function P(SF) as limit values as ± accding to lim P(SF) = γ, lim P(SF) = α β + γ P F Fig. 6. ROC curve f te example system. In tis case, te test is described in Section 4.2 te optimal tresold is = α/β, i.e. = ( p s)(p obs p s3 + p s2 ( p obs p s3 )) p s p s3 ( p obs )( p s2 ) = 4. Te interpretation of λ = α/β in te ROC curve tat was described in Section 4. is illustrated f tis example in Figure 6. In tis case te optimal point is caracterized by dp D dp F = λ = Linearity of te system failure probability Now, we prove tat te linearity of P(SF) olds in a me general setting tan te case in Figur. Depending on te alarm state, te system is configured differently as seen in te previous example. If we ave no alarm te iginal configuration is used, ten we use te ation C f te case were te configuration is wking properly C f te case tat it fails. Te cases C 2 C 2 are defined analogously f te backup configuration.

6 Teem. Let S be te supervised event. Ten te system failure probability is given by P(SF) = αp F βp D + γ were α = P( C 2 S) P( C S) β = P( C S) P( C 2 S) γ = P( C ) Proof. s befe, te system failure even can be exped into mutually exclusive cases as: P(SF) = P( C ) + P( C 2 ) = P( C S) + P( C S)+ P( C 2 S) + P( C 2 S) were is te alarm event. Expansion of eac of te four terms gives P( C S) = P( C S)P( C S) = = P( S)P( C S) = ( P F )P( C S) P( C S) = P( C S)P( C S) = = P( S)P( C S) = ( P D )P( C S) P( C 2 S) = P( C 2 S)P( C 2 S) = = P( S)P( C 2 S) = P D P( C 2 S) P( C 2 S) = P( C 2 S)P( C 2 S) = = P( S)P( C 2 S) = P F P( C 2 S) Collecting te expressions proves tat P(SF) is linear in P F P D, i.e. P(SF) = αp F βp D + γ were te coefficients α, β, γ are given in te teem. 5.3 Example of superfluous fault detects In te analysis in previous sections it was assumed tat bot α β were positive. Tis is always te case as will be illustrated wit two examples followed by a discussion an interpretation. First, consider te case sown in Figure 7. Let event e i (*) e 2 system failure Fig. 7. Example system. e 3 crespond to te failure of a sens s i. Here, te first configuration is tat we ave system failure if event e 2 occur. In case of alarm, te system switces to a mode were event e 3 causes system failure. Te detection F alarm D system alarms if event is detected, i.e. te supervised event S is. In tis case it olds tat β = p (p 2 p 3 ) wic is negative if p 2 < p 3. Tis means tat detecting a fault, even if te alarm is crect, will increase te probability of system failure. Te reason is tat even if we know tat sens s as failed, it is still better to switc to sens s 3 since sens s 2 is me reliable. It is terefe better to never alarm te detection system is superfluous. s a second example, consider te same system as in Figure 7 were te ND-gate indicated by (*) is replaced by an OR-gate. Ten we ave α = ( p )(p 3 p 2 ) wic is negative if p 3 < p 2. In suc a case, a false alarm as a positive influence on te system safety. Te reason f tis is tat even if we know tat sens s as failed, it is still better to switc to sens s 3, since it is me reliable tan sens s 2. lso ere te detection system is superfluous since it is always better to use sens s CONCLUSIONS In tis paper we ave considered tresold selection in fault tolerant systems. One of te fundamental questions as been to analyze te influence of fault detection properties on te overall system safety. Two imptant fault detection properties are probability of false alarm detection power it as been sown tat te probability of system failure can be expressed in te fm (6) f a general setting. Based on tis fm, te influence ratio was introduced in (7). Tis ratio includes all infmation about te supervised system te backup system tat is needed f te tresold optimization problem. It as been sown tat te influence ratio as a geometrical interpretation as te gradient of te ROC curve at te optimal point. Furterme, it was sown tat te influence ratio is te tresold f te optimal test quantity given by te likeliood ratio in te case studied in Section 4.2. Finally, two examples were given tat illustrate cases were te fault detects are superfluous. REFERENCES [ÅBF + 07] Jan Åslund, Jonas Biteus, Erik Frisk, Mattias Kryser, Lars Nielsen. Safety analysis of autonomous systems by extended fault tree analysis. International Journal of daptive Control Signal Processing, 2(2-3): , [Ber85] James O. Berger. Statistical Decision Tey Bayesian nalysis. Springer, 985. [BKLS03] M. Blanke, M. Kinnaert, J. Lunze, M. Staroswiecki. Diagnosis Fault-Tolerant Control. Springer, [Kay98] S.M. Kay. Fundamentals of signal processing detection tey. Prentice Hall, 998. [Le9] E.L. Lemann. Testing statistical ypotesis. Statistical/Probability series. Wadswt & Brooks/Cole, 99. [Vil92] lain Villemeur. Reliability, availability, maintainability safety assessment, volum & 2. Jon Wiley & sons, U.K., 992.